2019-12-05T05:43:45.956 INFO:root:teuthology version: 1.0.0-e0b5789 2019-12-05T05:43:45.956 DEBUG:teuthology.run:Teuthology command: teuthology /tmp/teuthology-worker.0dYqvX.tmp -- --verbose --lock --description fs/thrash/{begin.yaml ceph-thrash/mds.yaml clusters/1-mds-1-client-coloc.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{centos_7.yaml} tasks/cfuse_workunit_snaptests.yaml} --name pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi --block --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567683 2019-12-05T05:43:46.002 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-05T05:43:46.065 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567683 branch: wip-pdonnell-testing-20191204.231200 description: fs/thrash/{begin.yaml ceph-thrash/mds.yaml clusters/1-mds-1-client-coloc.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{centos_7.yaml} tasks/cfuse_workunit_snaptests.yaml} email: pdonnell@redhat.com first_in_suite: false job_id: '4567683' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: '7.6' overrides: admin_socket: branch: wip-pdonnell-testing-20191204.231200 ceph: cephfs_ec_profile: - m=2 - k=2 - crush-failure-domain=osd conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 global: mon client directed command retry: 5 ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: osd mds ms inject socket failures: 2500 mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 1 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 fs: xfs log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - not responding, replacing - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: c5290805033483ba0cec165c06bde07dae5f19ec ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: c5290805033483ba0cec165c06bde07dae5f19ec thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20191204.231200 sha1: c5290805033483ba0cec165c06bde07dae5f19ec owner: scheduled_pdonnell@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 sha1: c5290805033483ba0cec165c06bde07dae5f19ec suite: fs suite_branch: wip-pdonnell-testing-20191204.231200 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: c5290805033483ba0cec165c06bde07dae5f19ec tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - ceph: null - mds_thrash: null - ceph-fuse: null - workunit: clients: all: - fs/snaps teuthology_branch: master tube: smithi verbose: false worker_log: /home/teuthworker/archive/worker_logs/worker.smithi.9206 2019-12-05T05:43:46.066 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa; will attempt to use it 2019-12-05T05:43:46.068 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa/tasks 2019-12-05T05:43:46.068 INFO:teuthology.run_tasks:Running task internal.check_packages... 2019-12-05T05:43:46.070 INFO:teuthology.task.internal:Checking packages... 2019-12-05T05:43:46.098 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'c5290805033483ba0cec165c06bde07dae5f19ec' 2019-12-05T05:43:46.098 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2019-12-05T05:43:46.098 INFO:teuthology.packaging:ref: None 2019-12-05T05:43:46.098 INFO:teuthology.packaging:tag: None 2019-12-05T05:43:46.098 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20191204.231200 2019-12-05T05:43:46.098 INFO:teuthology.packaging:sha1: c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T05:43:46.099 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&ref=wip-pdonnell-testing-20191204.231200 2019-12-05T05:43:46.278 INFO:teuthology.task.internal:Found packages for ceph version 15.0.0-8014.gc529080 2019-12-05T05:43:46.278 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2019-12-05T05:43:46.290 INFO:teuthology.task.internal:no buildpackages task found 2019-12-05T05:43:46.290 INFO:teuthology.run_tasks:Running task internal.lock_machines... 2019-12-05T05:43:46.306 INFO:teuthology.task.internal.lock_machines:Locking machines... 2019-12-05T05:43:46.307 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-05T05:43:46.358 INFO:teuthology.task.internal.lock_machines:waiting for more smithi machines to be free (need 2 + 2, have 2)... 2019-12-05T05:43:56.373 DEBUG:teuthology.lock.ops:lock_many request: {'count': 2, 'locked_by': 'scheduled_pdonnell@teuthology', 'description': '/home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567683', 'machine_type': 'smithi'} 2019-12-05T05:43:56.406 DEBUG:teuthology.lock.ops:locked ubuntu@smithi016.front.sepia.ceph.com, ubuntu@smithi076.front.sepia.ceph.com 2019-12-05T05:43:56.560 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-12-05T05:43:56.773 INFO:teuthology.provision.fog.smithi076:Scheduling deploy of centos 7.6 2019-12-05T05:43:56.781 INFO:teuthology.provision.fog.smithi016:Scheduling deploy of centos 7.6 2019-12-05T05:43:57.135 INFO:teuthology.orchestra.console:Power off smithi076 2019-12-05T05:43:57.135 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-12-05T05:43:57.221 INFO:teuthology.orchestra.console:Power off smithi016 2019-12-05T05:43:57.222 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2019-12-05T05:43:57.266 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:43:57.324 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:01.366 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:01.400 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:05.506 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:05.544 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:05.772 INFO:teuthology.orchestra.console:Power off for smithi016 completed 2019-12-05T05:44:05.877 INFO:teuthology.orchestra.console:Power on smithi016 2019-12-05T05:44:05.877 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-12-05T05:44:05.921 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:09.648 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:09.884 INFO:teuthology.orchestra.console:Power off for smithi076 completed 2019-12-05T05:44:09.961 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:09.994 INFO:teuthology.orchestra.console:Power on smithi076 2019-12-05T05:44:09.994 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power on 2019-12-05T05:44:10.034 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:14.076 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:14.107 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi016.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:14.344 INFO:teuthology.orchestra.console:Power on for smithi016 completed 2019-12-05T05:44:14.948 INFO:teuthology.provision.fog.smithi016:Waiting for deploy to finish 2019-12-05T05:44:18.213 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2019-12-05T05:44:18.557 INFO:teuthology.orchestra.console:Power on for smithi076 completed 2019-12-05T05:44:18.657 INFO:teuthology.provision.fog.smithi076:Waiting for deploy to finish 2019-12-05T05:46:30.850 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:46:39.916 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:46:48.973 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:46:58.030 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:47:07.081 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:47:16.142 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:47:19.788 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:47:28.846 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:47:37.902 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:47:46.958 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:47:53.521 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:47:53.756 INFO:teuthology.orchestra.run.smithi016:> hostname 2019-12-05T05:47:54.210 INFO:teuthology.orchestra.run.smithi016.stdout:smithi068 2019-12-05T05:47:54.211 INFO:teuthology.orchestra.run.smithi016:> grep smithi068 /etc/hosts 2019-12-05T05:47:54.253 INFO:teuthology.orchestra.run.smithi016.stdout:172.21.15.68 smithi068.front.sepia.ceph.com smithi068 2019-12-05T05:47:54.253 INFO:teuthology.orchestra.run.smithi016:> sudo hostname smithi016 2019-12-05T05:47:54.420 INFO:teuthology.orchestra.run.smithi016:> sudo sed -i -e 's/smithi068/smithi016/g' /etc/hosts 2019-12-05T05:47:54.479 INFO:teuthology.orchestra.run.smithi016:> sudo sed -i -e 's/172.21.15.68/172.21.15.16/g' /etc/hosts 2019-12-05T05:47:54.533 INFO:teuthology.orchestra.run.smithi016:> sudo sed -i -e 's/smithi068/smithi016/g' /etc/hostname 2019-12-05T05:47:54.586 INFO:teuthology.orchestra.run.smithi016:> sudo hostname smithi016 2019-12-05T05:47:54.652 INFO:teuthology.provision.fog.smithi016:Deploy complete! 2019-12-05T05:47:54.654 DEBUG:teuthology.parallel:result is None 2019-12-05T05:47:56.010 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:48:05.073 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:48:14.154 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:48:23.214 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:48:32.267 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:48:41.325 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:48:50.380 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:49:56.383 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:49:56.583 INFO:teuthology.orchestra.run.smithi076:> hostname 2019-12-05T05:49:57.153 INFO:teuthology.orchestra.run.smithi076.stdout:smithi076 2019-12-05T05:49:57.153 INFO:teuthology.orchestra.run.smithi076:> grep smithi076 /etc/hosts 2019-12-05T05:49:57.238 INFO:teuthology.orchestra.run.smithi076.stdout:172.21.15.76 smithi076.front.sepia.ceph.com smithi076 2019-12-05T05:49:57.239 INFO:teuthology.orchestra.run.smithi076:> sudo hostname smithi076 2019-12-05T05:49:57.405 INFO:teuthology.orchestra.run.smithi076:> sudo sed -i -e 's/smithi076/smithi076/g' /etc/hosts 2019-12-05T05:49:57.504 INFO:teuthology.orchestra.run.smithi076:> sudo sed -i -e 's/172.21.15.76/172.21.15.76/g' /etc/hosts 2019-12-05T05:49:57.601 INFO:teuthology.orchestra.run.smithi076:> sudo sed -i -e 's/smithi076/smithi076/g' /etc/hostname 2019-12-05T05:49:57.688 INFO:teuthology.orchestra.run.smithi076:> sudo hostname smithi076 2019-12-05T05:49:57.757 INFO:teuthology.provision.fog.smithi076:Deploy complete! 2019-12-05T05:49:57.759 DEBUG:teuthology.parallel:result is None 2019-12-05T05:49:57.963 INFO:teuthology.lock.ops:Checking smithi016.front.sepia.ceph.com 2019-12-05T05:49:57.963 INFO:teuthology.lock.ops:New key found. Updating... 2019-12-05T05:49:57.987 INFO:teuthology.lock.ops:Checking smithi076.front.sepia.ceph.com 2019-12-05T05:49:57.987 INFO:teuthology.lock.ops:New key found. Updating... 2019-12-05T05:49:58.028 INFO:teuthology.lock.ops:Updating [smithi016.front.sepia.ceph.com]: set os type and version on server 2019-12-05T05:49:58.028 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:49:58.030 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:49:58.134 INFO:teuthology.orchestra.run.smithi016:> true 2019-12-05T05:49:58.561 INFO:teuthology.orchestra.run.smithi016:> uname -m 2019-12-05T05:49:58.636 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2019-12-05T05:49:58.636 INFO:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2019-12-05T05:49:58.703 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Linux" 2019-12-05T05:49:58.703 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="7 (Core)" 2019-12-05T05:49:58.703 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2019-12-05T05:49:58.704 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2019-12-05T05:49:58.704 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="7" 2019-12-05T05:49:58.704 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-05T05:49:58.704 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2019-12-05T05:49:58.707 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-05T05:49:58.707 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.centos.org/" 2019-12-05T05:49:58.708 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-05T05:49:58.708 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:49:58.708 INFO:teuthology.orchestra.run.smithi016.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-05T05:49:58.708 INFO:teuthology.orchestra.run.smithi016.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-05T05:49:58.708 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-05T05:49:58.708 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-05T05:49:58.709 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:49:58.709 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2019-12-05T05:49:58.724 INFO:teuthology.lock.ops:Updating [smithi076.front.sepia.ceph.com]: set os type and version on server 2019-12-05T05:49:58.725 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:49:58.726 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:49:58.830 INFO:teuthology.orchestra.run.smithi076:> true 2019-12-05T05:49:59.210 INFO:teuthology.orchestra.run.smithi076:> uname -m 2019-12-05T05:49:59.272 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2019-12-05T05:49:59.272 INFO:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2019-12-05T05:49:59.327 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Linux" 2019-12-05T05:49:59.327 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="7 (Core)" 2019-12-05T05:49:59.327 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2019-12-05T05:49:59.327 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2019-12-05T05:49:59.328 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="7" 2019-12-05T05:49:59.328 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-05T05:49:59.328 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2019-12-05T05:49:59.329 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-05T05:49:59.329 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://www.centos.org/" 2019-12-05T05:49:59.329 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-05T05:49:59.329 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:49:59.329 INFO:teuthology.orchestra.run.smithi076.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-05T05:49:59.329 INFO:teuthology.orchestra.run.smithi076.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-05T05:49:59.329 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-05T05:49:59.330 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-05T05:49:59.330 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:49:59.330 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2019-12-05T05:49:59.344 INFO:teuthology.task.internal.lock_machines:2 smithi machines locked this try, 2/2 locked so far 2019-12-05T05:49:59.373 INFO:teuthology.task.internal.lock_machines:Locked targets: smithi016.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzkbvNpgEekN2H5FRrrk8O9zYD3vW8yCH3uhmzawZAVKOdsmM4IeOc05s7xzUjHA0z4xaSGAVu2uwJrb99bqqTRCLGL9ZJuHe8hZaEoo5SWseu0pgiPMJLl59z+xqHQCTPHI9lxuAdpRWxiXzS7+oAWnSEAfPCr6tyPmrew6vbMely+/lMmXc+03L7Hb4MxO6lMe/w/cZmz9zl0wCiich480FZKKRygxRmqJDwpswJkxKEVA95fxYx5FLUPypTwLTi2A63lRTXxIETT3i8QfPdIx5qAZIcBQlwC7tdzu2G1GNXQxCAf75YP5FgbRJCakCb6vpl3EoWbc8HRFYuGLHD smithi076.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc4m2iSvsnpO8WMgH+idDlxTdFNO1LUOYlSE5JEJUyfOUF3+2D64dchE5K6fkXioBw9cTyXP/sIL3PTmvkfkQjZ/T/eFyMfOoYcIJZlo/yQicIh3dPRb0y7+oWDYY79l6VwjCDugCrrHbqPmf/6knofnwhL+Wxrc0Ogit57xxMevL58y+1Y9TY60VAf5XlFJa+er/8UB6ZXddirHQY61T9o3BjHNybxrjPANCFasSrwX/6z2BG4B/SwN5c0Bc0N24yygsgIjYG6j/iPZDONuNLfP5R3Gq3IannZ0X+9aI/6fvdeBaiqokwvyeqcgdQqW0a2YWxNBWsAeK6e/P5SpXl 2019-12-05T05:49:59.373 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2019-12-05T05:49:59.418 INFO:teuthology.run_tasks:Running task internal.save_config... 2019-12-05T05:49:59.424 INFO:teuthology.task.internal:Saving configuration 2019-12-05T05:49:59.441 INFO:teuthology.run_tasks:Running task internal.check_lock... 2019-12-05T05:49:59.452 INFO:teuthology.task.internal.check_lock:Checking locks... 2019-12-05T05:49:59.465 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-12-05 05:43:56.395270', u'locked_by': u'scheduled_pdonnell@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi016.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBN9pErVAviBxyAmoZu7YKAIYVKrvWYNOGYQtwWkLuvwAFMnNiLBFqkfbM12KsLTg7wGuMyo4bIPajPaSH9TlG88=', u'description': u'/home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567683'} 2019-12-05T05:49:59.478 DEBUG:teuthology.task.internal.check_lock:machine status is {u'is_vm': False, u'locked': True, u'locked_since': u'2019-12-05 05:43:56.395248', u'locked_by': u'scheduled_pdonnell@teuthology', u'up': True, u'mac_address': None, u'name': u'smithi076.front.sepia.ceph.com', u'os_version': u'7', u'machine_type': u'smithi', u'vm_host': None, u'os_type': u'centos', u'arch': u'x86_64', u'ssh_pub_key': u'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCPRhEWxMkh1aKg1eYeAsSk4ZWNBScfaLu25Wldx/kSil0rRG+jX1hUJUcmWvjeFDJ6WSfa7LQ1mapEmd1Vnbtw=', u'description': u'/home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567683'} 2019-12-05T05:49:59.478 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2019-12-05T05:49:59.490 INFO:teuthology.task.internal:roles: ubuntu@smithi016.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2019-12-05T05:49:59.490 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2019-12-05T05:49:59.490 INFO:teuthology.run_tasks:Running task console_log... 2019-12-05T05:49:59.640 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=, signals=[15]) 2019-12-05T05:49:59.640 INFO:teuthology.run_tasks:Running task internal.connect... 2019-12-05T05:49:59.651 INFO:teuthology.task.internal:Opening connections... 2019-12-05T05:49:59.651 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2019-12-05T05:49:59.652 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:49:59.757 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:49:59.758 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:49:59.857 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2019-12-05T05:49:59.868 INFO:teuthology.orchestra.run.smithi076:> uname -m 2019-12-05T05:50:00.097 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2019-12-05T05:50:00.098 INFO:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2019-12-05T05:50:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Linux" 2019-12-05T05:50:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="7 (Core)" 2019-12-05T05:50:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2019-12-05T05:50:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2019-12-05T05:50:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="7" 2019-12-05T05:50:00.169 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-05T05:50:00.170 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2019-12-05T05:50:00.170 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-05T05:50:00.170 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://www.centos.org/" 2019-12-05T05:50:00.170 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-05T05:50:00.170 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:50:00.170 INFO:teuthology.orchestra.run.smithi076.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-05T05:50:00.170 INFO:teuthology.orchestra.run.smithi076.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-05T05:50:00.171 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-05T05:50:00.171 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-05T05:50:00.171 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:50:00.171 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2019-12-05T05:50:00.183 INFO:teuthology.orchestra.run.smithi016:> uname -m 2019-12-05T05:50:00.310 INFO:teuthology.orchestra.run.smithi016.stdout:x86_64 2019-12-05T05:50:00.310 INFO:teuthology.orchestra.run.smithi016:> cat /etc/os-release 2019-12-05T05:50:00.389 INFO:teuthology.orchestra.run.smithi016.stdout:NAME="CentOS Linux" 2019-12-05T05:50:00.389 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION="7 (Core)" 2019-12-05T05:50:00.389 INFO:teuthology.orchestra.run.smithi016.stdout:ID="centos" 2019-12-05T05:50:00.389 INFO:teuthology.orchestra.run.smithi016.stdout:ID_LIKE="rhel fedora" 2019-12-05T05:50:00.389 INFO:teuthology.orchestra.run.smithi016.stdout:VERSION_ID="7" 2019-12-05T05:50:00.390 INFO:teuthology.orchestra.run.smithi016.stdout:PRETTY_NAME="CentOS Linux 7 (Core)" 2019-12-05T05:50:00.390 INFO:teuthology.orchestra.run.smithi016.stdout:ANSI_COLOR="0;31" 2019-12-05T05:50:00.390 INFO:teuthology.orchestra.run.smithi016.stdout:CPE_NAME="cpe:/o:centos:centos:7" 2019-12-05T05:50:00.390 INFO:teuthology.orchestra.run.smithi016.stdout:HOME_URL="https://www.centos.org/" 2019-12-05T05:50:00.390 INFO:teuthology.orchestra.run.smithi016.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2019-12-05T05:50:00.390 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:50:00.390 INFO:teuthology.orchestra.run.smithi016.stdout:CENTOS_MANTISBT_PROJECT="CentOS-7" 2019-12-05T05:50:00.391 INFO:teuthology.orchestra.run.smithi016.stdout:CENTOS_MANTISBT_PROJECT_VERSION="7" 2019-12-05T05:50:00.391 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2019-12-05T05:50:00.391 INFO:teuthology.orchestra.run.smithi016.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="7" 2019-12-05T05:50:00.391 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:50:00.391 INFO:teuthology.lock.ops:Updating smithi016.front.sepia.ceph.com on lock server 2019-12-05T05:50:00.404 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2019-12-05T05:50:00.448 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2019-12-05T05:50:00.458 INFO:teuthology.task.internal:Checking for old test directory... 2019-12-05T05:50:00.458 INFO:teuthology.orchestra.run.smithi016:> test '!' -e /home/ubuntu/cephtest 2019-12-05T05:50:00.463 INFO:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2019-12-05T05:50:00.549 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2019-12-05T05:50:00.588 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2019-12-05T05:50:00.588 INFO:teuthology.orchestra.run.smithi016:> test -z $(ls -A /var/lib/ceph) 2019-12-05T05:50:00.593 INFO:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2019-12-05T05:50:00.702 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2019-12-05T05:50:01.616 INFO:teuthology.run_tasks:Running task kernel... 2019-12-05T05:50:01.633 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.c': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}, 'mds.a': {'sha1': 'distro', 'kdb': True}, 'mds.b': {'sha1': 'distro', 'kdb': True}, 'osd.1': {'sha1': 'distro', 'kdb': True}, 'osd.0': {'sha1': 'distro', 'kdb': True}, 'osd.3': {'sha1': 'distro', 'kdb': True}, 'osd.2': {'sha1': 'distro', 'kdb': True}, 'osd.5': {'sha1': 'distro', 'kdb': True}, 'osd.4': {'sha1': 'distro', 'kdb': True}, 'osd.7': {'sha1': 'distro', 'kdb': True}, 'osd.6': {'sha1': 'distro', 'kdb': True}, 'mgr.y': {'sha1': 'distro', 'kdb': True}, 'mgr.x': {'sha1': 'distro', 'kdb': True}, 'client.0': {'sha1': 'distro', 'kdb': True}} 2019-12-05T05:50:01.633 INFO:teuthology.task.kernel:config {'mon.a': {'sha1': 'distro', 'kdb': True}, 'mon.b': {'sha1': 'distro', 'kdb': True}}, timeout 300 2019-12-05T05:50:01.633 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi076.front.sepia.ceph.com 2019-12-05T05:50:01.634 INFO:teuthology.orchestra.run.smithi076:> sudo package-cleanup -y --oldkernels 2019-12-05T05:50:02.504 INFO:teuthology.orchestra.run.smithi076.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-05T05:50:04.877 INFO:teuthology.orchestra.run.smithi076.stdout:No old kernels to remove 2019-12-05T05:50:04.896 INFO:teuthology.task.kernel:Removing old kernels from ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:50:04.896 INFO:teuthology.orchestra.run.smithi016:> sudo package-cleanup -y --oldkernels 2019-12-05T05:50:05.985 INFO:teuthology.orchestra.run.smithi016.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-05T05:50:08.706 INFO:teuthology.orchestra.run.smithi016.stdout:No old kernels to remove 2019-12-05T05:50:08.725 INFO:teuthology.orchestra.run.smithi016:> uname -r 2019-12-05T05:50:08.819 INFO:teuthology.orchestra.run.smithi016.stdout:3.10.0-1062.4.1.el7.x86_64 2019-12-05T05:50:08.819 INFO:teuthology.task.kernel:Running kernel on smithi016: 3.10.0-1062.4.1.el7.x86_64 2019-12-05T05:50:08.819 INFO:teuthology.orchestra.run.smithi016:> sudo yum install -y kernel 2019-12-05T05:50:09.084 INFO:teuthology.orchestra.run.smithi016.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-05T05:50:09.129 INFO:teuthology.orchestra.run.smithi016.stdout:Determining fastest mirrors 2019-12-05T05:50:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: * base: mirror.math.princeton.edu 2019-12-05T05:50:09.348 INFO:teuthology.orchestra.run.smithi016.stdout: * epel: ftp.linux.ncsu.edu 2019-12-05T05:50:09.349 INFO:teuthology.orchestra.run.smithi016.stdout: * extras: mirror.vcu.edu 2019-12-05T05:50:09.350 INFO:teuthology.orchestra.run.smithi016.stdout: * updates: mirror.vcu.edu 2019-12-05T05:50:11.635 INFO:teuthology.orchestra.run.smithi016.stdout:Resolving Dependencies 2019-12-05T05:50:11.636 INFO:teuthology.orchestra.run.smithi016.stdout:--> Running transaction check 2019-12-05T05:50:11.636 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package kernel.x86_64 0:3.10.0-1062.7.1.el7 will be installed 2019-12-05T05:50:12.913 INFO:teuthology.orchestra.run.smithi016.stdout:--> Finished Dependency Resolution 2019-12-05T05:50:13.746 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:50:13.746 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies Resolved 2019-12-05T05:50:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:50:13.761 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2019-12-05T05:50:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2019-12-05T05:50:13.761 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2019-12-05T05:50:13.761 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2019-12-05T05:50:13.761 INFO:teuthology.orchestra.run.smithi016.stdout: kernel x86_64 3.10.0-1062.7.1.el7 updates 50 M 2019-12-05T05:50:13.762 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:50:13.762 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2019-12-05T05:50:13.762 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2019-12-05T05:50:13.762 INFO:teuthology.orchestra.run.smithi016.stdout:Install 1 Package 2019-12-05T05:50:13.762 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:50:13.762 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 50 M 2019-12-05T05:50:13.762 INFO:teuthology.orchestra.run.smithi016.stdout:Installed size: 64 M 2019-12-05T05:50:13.762 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading packages: 2019-12-05T05:50:13.765 INFO:teuthology.orchestra.run.smithi016.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-05T05:50:15.007 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2019-12-05T05:50:15.357 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2019-12-05T05:50:16.228 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded 2019-12-05T05:50:16.229 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2019-12-05T05:50:53.948 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : kernel-3.10.0-1062.7.1.el7.x86_64 1/1 2019-12-05T05:50:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : kernel-3.10.0-1062.7.1.el7.x86_64 1/1 2019-12-05T05:50:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:50:54.503 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2019-12-05T05:50:54.503 INFO:teuthology.orchestra.run.smithi016.stdout: kernel.x86_64 0:3.10.0-1062.7.1.el7 2019-12-05T05:50:54.504 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:50:54.504 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2019-12-05T05:50:54.556 INFO:teuthology.orchestra.run.smithi016:> rpm -q kernel --last | head -n 1 2019-12-05T05:50:54.954 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-3.10.0-1062.7.1.el7.x86_64 Thu 05 Dec 2019 05:50:19 AM UTC 2019-12-05T05:50:54.958 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:50:54.958 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.4.1.el7.x86_64 Expected: 3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:50:54.958 INFO:teuthology.orchestra.run.smithi076:> uname -r 2019-12-05T05:50:55.078 INFO:teuthology.orchestra.run.smithi076.stdout:3.10.0-1062.4.1.el7.x86_64 2019-12-05T05:50:55.079 INFO:teuthology.task.kernel:Running kernel on smithi076: 3.10.0-1062.4.1.el7.x86_64 2019-12-05T05:50:55.079 INFO:teuthology.orchestra.run.smithi076:> sudo yum install -y kernel 2019-12-05T05:50:55.332 INFO:teuthology.orchestra.run.smithi076.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-05T05:50:55.372 INFO:teuthology.orchestra.run.smithi076.stdout:Determining fastest mirrors 2019-12-05T05:50:55.616 INFO:teuthology.orchestra.run.smithi076.stdout: * base: mirror.rackspace.com 2019-12-05T05:50:55.616 INFO:teuthology.orchestra.run.smithi076.stdout: * epel: ftp.linux.ncsu.edu 2019-12-05T05:50:55.617 INFO:teuthology.orchestra.run.smithi076.stdout: * extras: mirror.vcu.edu 2019-12-05T05:50:55.618 INFO:teuthology.orchestra.run.smithi076.stdout: * updates: mirror.vcu.edu 2019-12-05T05:50:58.009 INFO:teuthology.orchestra.run.smithi076.stdout:Resolving Dependencies 2019-12-05T05:50:58.009 INFO:teuthology.orchestra.run.smithi076.stdout:--> Running transaction check 2019-12-05T05:50:58.010 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package kernel.x86_64 0:3.10.0-1062.7.1.el7 will be installed 2019-12-05T05:50:59.363 INFO:teuthology.orchestra.run.smithi076.stdout:--> Finished Dependency Resolution 2019-12-05T05:51:00.136 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:51:00.137 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies Resolved 2019-12-05T05:51:00.142 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:51:00.143 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2019-12-05T05:51:00.143 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2019-12-05T05:51:00.143 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2019-12-05T05:51:00.143 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2019-12-05T05:51:00.143 INFO:teuthology.orchestra.run.smithi076.stdout: kernel x86_64 3.10.0-1062.7.1.el7 updates 50 M 2019-12-05T05:51:00.144 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:51:00.144 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2019-12-05T05:51:00.144 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2019-12-05T05:51:00.144 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2019-12-05T05:51:00.144 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:51:00.145 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 50 M 2019-12-05T05:51:00.145 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 64 M 2019-12-05T05:51:00.145 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading packages: 2019-12-05T05:51:00.151 INFO:teuthology.orchestra.run.smithi076.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-05T05:51:01.083 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2019-12-05T05:51:01.278 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2019-12-05T05:51:02.092 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded 2019-12-05T05:51:02.092 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2019-12-05T05:51:36.976 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-3.10.0-1062.7.1.el7.x86_64 1/1 2019-12-05T05:51:37.581 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-3.10.0-1062.7.1.el7.x86_64 1/1 2019-12-05T05:51:37.581 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:51:37.581 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2019-12-05T05:51:37.581 INFO:teuthology.orchestra.run.smithi076.stdout: kernel.x86_64 0:3.10.0-1062.7.1.el7 2019-12-05T05:51:37.581 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:51:37.582 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2019-12-05T05:51:37.635 INFO:teuthology.orchestra.run.smithi076:> rpm -q kernel --last | head -n 1 2019-12-05T05:51:37.958 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-3.10.0-1062.7.1.el7.x86_64 Thu 05 Dec 2019 05:51:04 AM UTC 2019-12-05T05:51:37.960 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:51:37.960 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 3.10.0-1062.4.1.el7.x86_64 Expected: 3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:51:37.960 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2019-12-05T05:51:37.960 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-12-05T05:51:37.960 DEBUG:teuthology.task.kernel:src is distro, skipping download 2019-12-05T05:51:37.961 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2019-12-05T05:51:37.961 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi016.front.sepia.ceph.com, path=None, version=distro) 2019-12-05T05:51:37.961 INFO:teuthology.orchestra.run.smithi016:> rpm -q kernel --last | head -n 1 2019-12-05T05:51:38.330 INFO:teuthology.orchestra.run.smithi016.stdout:kernel-3.10.0-1062.7.1.el7.x86_64 Thu 05 Dec 2019 05:50:19 AM UTC 2019-12-05T05:51:38.333 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:51:38.333 INFO:teuthology.orchestra.run.smithi016:> sudo rpm -qi grub2 2019-12-05T05:51:38.508 INFO:teuthology.orchestra.run.smithi016.stdout:Name : grub2 2019-12-05T05:51:38.508 INFO:teuthology.orchestra.run.smithi016.stdout:Epoch : 1 2019-12-05T05:51:38.508 INFO:teuthology.orchestra.run.smithi016.stdout:Version : 2.02 2019-12-05T05:51:38.508 INFO:teuthology.orchestra.run.smithi016.stdout:Release : 0.76.el7.centos 2019-12-05T05:51:38.508 INFO:teuthology.orchestra.run.smithi016.stdout:Architecture: x86_64 2019-12-05T05:51:38.509 INFO:teuthology.orchestra.run.smithi016.stdout:Install Date: Tue 05 Nov 2019 03:41:14 PM UTC 2019-12-05T05:51:38.509 INFO:teuthology.orchestra.run.smithi016.stdout:Group : System Environment/Base 2019-12-05T05:51:38.509 INFO:teuthology.orchestra.run.smithi016.stdout:Size : 0 2019-12-05T05:51:38.509 INFO:teuthology.orchestra.run.smithi016.stdout:License : GPLv3+ 2019-12-05T05:51:38.509 INFO:teuthology.orchestra.run.smithi016.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-12-05T05:51:38.509 INFO:teuthology.orchestra.run.smithi016.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-12-05T05:51:38.509 INFO:teuthology.orchestra.run.smithi016.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-12-05T05:51:38.510 INFO:teuthology.orchestra.run.smithi016.stdout:Build Host : kbuilder.bsys.centos.org 2019-12-05T05:51:38.510 INFO:teuthology.orchestra.run.smithi016.stdout:Relocations : (not relocatable) 2019-12-05T05:51:38.510 INFO:teuthology.orchestra.run.smithi016.stdout:Packager : CentOS BuildSystem 2019-12-05T05:51:38.510 INFO:teuthology.orchestra.run.smithi016.stdout:Vendor : CentOS 2019-12-05T05:51:38.510 INFO:teuthology.orchestra.run.smithi016.stdout:URL : http://www.gnu.org/software/grub/ 2019-12-05T05:51:38.510 INFO:teuthology.orchestra.run.smithi016.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-12-05T05:51:38.510 INFO:teuthology.orchestra.run.smithi016.stdout:Description : 2019-12-05T05:51:38.510 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T05:51:38.511 INFO:teuthology.orchestra.run.smithi016.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-12-05T05:51:38.511 INFO:teuthology.orchestra.run.smithi016.stdout:customizable bootloader with modular architecture. It supports a rich 2019-12-05T05:51:38.511 INFO:teuthology.orchestra.run.smithi016.stdout:variety of kernel formats, file systems, computer architectures and 2019-12-05T05:51:38.511 INFO:teuthology.orchestra.run.smithi016.stdout:hardware devices. 2019-12-05T05:51:38.512 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-12-05T05:51:38.512 INFO:teuthology.task.kernel:Updating grub on smithi016 to boot 3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:51:38.513 INFO:teuthology.orchestra.run.smithi016:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-12-05T05:51:39.009 INFO:teuthology.orchestra.run.smithi016.stderr:Generating grub configuration file ... 2019-12-05T05:51:39.156 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:51:39.172 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.7.1.el7.x86_64.img 2019-12-05T05:51:39.428 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-12-05T05:51:39.450 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-12-05T05:51:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-12-05T05:51:39.526 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-12-05T05:51:39.566 INFO:teuthology.orchestra.run.smithi016.stderr:Found linux image: /boot/vmlinuz-0-rescue-a1739c9c60b14f0891126d2e603bb003 2019-12-05T05:51:39.579 INFO:teuthology.orchestra.run.smithi016.stderr:Found initrd image: /boot/initramfs-0-rescue-a1739c9c60b14f0891126d2e603bb003.img 2019-12-05T05:51:40.632 INFO:teuthology.orchestra.run.smithi016.stderr:done 2019-12-05T05:51:40.632 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T05:51:40.686 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.QKdX6PJ5si 2019-12-05T05:51:40.686 INFO:teuthology.orchestra.run.smithi016:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QKdX6PJ5si 2019-12-05T05:51:40.759 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.QKdX6PJ5si 2019-12-05T05:51:40.901 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.QKdX6PJ5si is 6KB 2019-12-05T05:51:40.959 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.QKdX6PJ5si 2019-12-05T05:51:41.018 INFO:teuthology.orchestra.run.smithi016:> sudo grub2-set-default 0 2019-12-05T05:51:41.163 INFO:teuthology.orchestra.run.smithi016:> sudo shutdown -r now 2019-12-05T05:51:41.169 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2019-12-05T05:51:41.169 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2019-12-05T05:51:41.169 INFO:teuthology.orchestra.run.smithi076:> rpm -q kernel --last | head -n 1 2019-12-05T05:51:41.536 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-3.10.0-1062.7.1.el7.x86_64 Thu 05 Dec 2019 05:51:04 AM UTC 2019-12-05T05:51:41.539 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:51:41.539 INFO:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2 2019-12-05T05:51:41.660 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2 2019-12-05T05:51:41.660 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2019-12-05T05:51:41.660 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.02 2019-12-05T05:51:41.660 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 0.76.el7.centos 2019-12-05T05:51:41.661 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2019-12-05T05:51:41.661 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Tue 05 Nov 2019 03:41:14 PM UTC 2019-12-05T05:51:41.661 INFO:teuthology.orchestra.run.smithi076.stdout:Group : System Environment/Base 2019-12-05T05:51:41.661 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 0 2019-12-05T05:51:41.661 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2019-12-05T05:51:41.661 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Mon 12 Nov 2018 02:28:06 PM UTC, Key ID 24c6a8a7f4a80eb5 2019-12-05T05:51:41.661 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.02-0.76.el7.centos.src.rpm 2019-12-05T05:51:41.661 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Fri 09 Nov 2018 12:58:43 AM UTC 2019-12-05T05:51:41.662 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : kbuilder.bsys.centos.org 2019-12-05T05:51:41.662 INFO:teuthology.orchestra.run.smithi076.stdout:Relocations : (not relocatable) 2019-12-05T05:51:41.662 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : CentOS BuildSystem 2019-12-05T05:51:41.662 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2019-12-05T05:51:41.662 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2019-12-05T05:51:41.662 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Bootloader with support for Linux, Multiboot and more 2019-12-05T05:51:41.662 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2019-12-05T05:51:41.663 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T05:51:41.663 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2019-12-05T05:51:41.663 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2019-12-05T05:51:41.663 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2019-12-05T05:51:41.663 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2019-12-05T05:51:41.664 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2019-12-05T05:51:41.664 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:51:41.665 INFO:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2019-12-05T05:51:42.102 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2019-12-05T05:51:42.245 INFO:teuthology.orchestra.run.smithi076.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:51:42.262 INFO:teuthology.orchestra.run.smithi076.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.7.1.el7.x86_64.img 2019-12-05T05:51:42.515 INFO:teuthology.orchestra.run.smithi076.stderr:Found linux image: /boot/vmlinuz-3.10.0-1062.4.1.el7.x86_64 2019-12-05T05:51:42.536 INFO:teuthology.orchestra.run.smithi076.stderr:Found initrd image: /boot/initramfs-3.10.0-1062.4.1.el7.x86_64.img 2019-12-05T05:51:42.571 INFO:teuthology.orchestra.run.smithi076.stderr:Found linux image: /boot/vmlinuz-3.10.0-957.el7.x86_64 2019-12-05T05:51:42.589 INFO:teuthology.orchestra.run.smithi076.stderr:Found initrd image: /boot/initramfs-3.10.0-957.el7.x86_64.img 2019-12-05T05:51:42.618 INFO:teuthology.orchestra.run.smithi076.stderr:Found linux image: /boot/vmlinuz-0-rescue-a1739c9c60b14f0891126d2e603bb003 2019-12-05T05:51:42.628 INFO:teuthology.orchestra.run.smithi076.stderr:Found initrd image: /boot/initramfs-0-rescue-a1739c9c60b14f0891126d2e603bb003.img 2019-12-05T05:51:43.508 INFO:teuthology.orchestra.run.smithi076.stderr:done 2019-12-05T05:51:43.511 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T05:51:43.582 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.zeE4vnlm7f 2019-12-05T05:51:43.582 INFO:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zeE4vnlm7f 2019-12-05T05:51:43.657 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.zeE4vnlm7f 2019-12-05T05:51:43.803 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.zeE4vnlm7f is 6KB 2019-12-05T05:51:43.879 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.zeE4vnlm7f 2019-12-05T05:51:43.939 INFO:teuthology.orchestra.run.smithi076:> sudo grub2-set-default 0 2019-12-05T05:51:44.096 INFO:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2019-12-05T05:51:44.101 INFO:teuthology.misc:Re-opening connections... 2019-12-05T05:51:44.102 INFO:teuthology.misc:trying to connect to ubuntu@smithi076.front.sepia.ceph.com 2019-12-05T05:51:44.103 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:51:44.104 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:51:44.207 INFO:teuthology.orchestra.run.smithi076:> true 2019-12-05T05:51:44.570 DEBUG:teuthology.misc:waited 0.468590974808 2019-12-05T05:51:45.571 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:51:45.571 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:51:45.572 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:52:45.579 DEBUG:teuthology.orchestra.remote:timed out 2019-12-05T05:52:45.579 DEBUG:teuthology.misc:waited 61.477399826 2019-12-05T05:52:46.580 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:52:46.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:52:46.581 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:52:49.656 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2019-12-05T05:52:49.657 DEBUG:teuthology.misc:waited 65.5551269054 2019-12-05T05:52:50.658 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:52:50.658 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:52:50.659 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:52:52.728 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2019-12-05T05:52:52.729 DEBUG:teuthology.misc:waited 68.6270859241 2019-12-05T05:52:53.729 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:52:53.729 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:52:53.730 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:52:55.800 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2019-12-05T05:52:55.801 DEBUG:teuthology.misc:waited 71.6990480423 2019-12-05T05:52:56.801 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:52:56.801 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:52:56.802 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:52:58.872 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2019-12-05T05:52:58.873 DEBUG:teuthology.misc:waited 74.7710409164 2019-12-05T05:52:59.873 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:52:59.874 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:52:59.874 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:53:01.945 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2019-12-05T05:53:01.945 DEBUG:teuthology.misc:waited 77.8432888985 2019-12-05T05:53:02.946 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:53:02.946 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:53:02.947 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:53:05.016 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2019-12-05T05:53:05.017 DEBUG:teuthology.misc:waited 80.9150309563 2019-12-05T05:53:06.017 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:53:06.018 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:53:06.018 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:53:07.285 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2019-12-05T05:53:07.286 DEBUG:teuthology.misc:waited 83.1839599609 2019-12-05T05:53:08.286 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:53:08.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:53:08.287 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:53:08.288 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.16 2019-12-05T05:53:08.288 DEBUG:teuthology.misc:waited 84.1864359379 2019-12-05T05:53:09.289 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:53:09.290 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:53:09.290 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:53:09.392 INFO:teuthology.orchestra.run.smithi016:> true 2019-12-05T05:53:09.676 DEBUG:teuthology.misc:waited 85.5746908188 2019-12-05T05:53:10.794 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2019-12-05T05:53:10.794 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "3.10.0-1062.7.1.el7.x86_64"... 2019-12-05T05:53:10.794 INFO:teuthology.orchestra.run.smithi016:> uname -r 2019-12-05T05:53:10.860 INFO:teuthology.orchestra.run.smithi016.stdout:3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:53:10.860 DEBUG:teuthology.task.kernel:current kernel version is 3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:53:10.860 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-12-05T05:53:10.860 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-12-05T05:53:10.861 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "3.10.0-1062.7.1.el7.x86_64"... 2019-12-05T05:53:10.861 INFO:teuthology.orchestra.run.smithi076:> uname -r 2019-12-05T05:54:08.697 ERROR:paramiko.transport:Socket exception: Connection reset by peer (104) 2019-12-05T05:54:08.722 ERROR:teuthology.task.kernel:Saw exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 695, in wait_for_reboot assert not need_to_install(ctx, client, need_install[client]), \ File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/task/kernel.py", line 184, in need_to_install stdout=uname_fp, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/cluster.py", line 64, in run return [remote.run(**kwargs) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/remote.py", line 198, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 428, in run r.execute() File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/teuthology/orchestra/run.py", line 98, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/client.py", line 493, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py", line 806, in open_session timeout=timeout, File "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/local/lib/python2.7/site-packages/paramiko/transport.py", line 933, in open_channel raise e error: [Errno 104] Connection reset by peer 2019-12-05T05:54:09.725 INFO:teuthology.misc:Re-opening connections... 2019-12-05T05:54:09.726 INFO:teuthology.misc:trying to connect to ubuntu@smithi076.front.sepia.ceph.com 2019-12-05T05:54:09.726 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:54:09.726 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:54:09.833 INFO:teuthology.orchestra.run.smithi076:> true 2019-12-05T05:54:10.376 DEBUG:teuthology.misc:waited 0.650196075439 2019-12-05T05:54:11.377 INFO:teuthology.misc:trying to connect to ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T05:54:11.378 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T05:54:11.379 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T05:54:11.481 INFO:teuthology.orchestra.run.smithi016:> true 2019-12-05T05:54:11.909 DEBUG:teuthology.misc:waited 2.1831099987 2019-12-05T05:54:12.910 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2019-12-05T05:54:12.910 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "3.10.0-1062.7.1.el7.x86_64"... 2019-12-05T05:54:12.910 INFO:teuthology.orchestra.run.smithi076:> uname -r 2019-12-05T05:54:13.009 INFO:teuthology.orchestra.run.smithi076.stdout:3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:54:13.010 DEBUG:teuthology.task.kernel:current kernel version is 3.10.0-1062.7.1.el7.x86_64 2019-12-05T05:54:13.010 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2019-12-05T05:54:14.023 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2019-12-05T05:54:14.023 INFO:teuthology.orchestra.run.smithi016:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-12-05T05:54:14.181 INFO:teuthology.orchestra.run.smithi016.stdout:ttyS1 2019-12-05T05:54:14.193 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2019-12-05T05:54:14.193 INFO:teuthology.orchestra.run.smithi076:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2019-12-05T05:54:14.359 INFO:teuthology.orchestra.run.smithi076.stdout:ttyS1 2019-12-05T05:54:14.372 INFO:teuthology.run_tasks:Running task internal.base... 2019-12-05T05:54:14.390 INFO:teuthology.task.internal:Creating test directory... 2019-12-05T05:54:14.390 INFO:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-12-05T05:54:14.397 INFO:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2019-12-05T05:54:14.489 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2019-12-05T05:54:14.502 INFO:teuthology.run_tasks:Running task internal.archive... 2019-12-05T05:54:14.514 INFO:teuthology.task.internal:Creating archive directory... 2019-12-05T05:54:14.515 INFO:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-12-05T05:54:14.522 INFO:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2019-12-05T05:54:14.621 INFO:teuthology.run_tasks:Running task internal.coredump... 2019-12-05T05:54:14.631 INFO:teuthology.task.internal:Enabling coredump saving... 2019-12-05T05:54:14.631 INFO:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-05T05:54:14.636 INFO:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-05T05:54:14.758 INFO:teuthology.orchestra.run.smithi016.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-05T05:54:14.759 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2019-12-05T05:54:14.771 INFO:teuthology.run_tasks:Running task internal.sudo... 2019-12-05T05:54:15.155 INFO:teuthology.task.internal:Configuring sudo... 2019-12-05T05:54:15.156 INFO:teuthology.orchestra.run.smithi016:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-12-05T05:54:15.162 INFO:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2019-12-05T05:54:15.282 INFO:teuthology.run_tasks:Running task internal.syslog... 2019-12-05T05:54:15.325 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2019-12-05T05:54:15.325 INFO:teuthology.orchestra.run.smithi016:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-12-05T05:54:15.330 INFO:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2019-12-05T05:54:15.418 INFO:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-05T05:54:15.508 INFO:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-05T05:54:15.614 INFO:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-05T05:54:15.695 INFO:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-05T05:54:15.769 INFO:teuthology.orchestra.run.smithi076:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-12-05T05:54:15.835 INFO:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-05T05:54:15.946 INFO:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2019-12-05T05:54:16.066 INFO:teuthology.orchestra.run.smithi016:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-05T05:54:16.171 INFO:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2019-12-05T05:54:16.609 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /etc/rsyslog.d/80-cephtest.conf' 2019-12-05T05:54:16.710 INFO:teuthology.orchestra.run.smithi016:> sudo service rsyslog restart 2019-12-05T05:54:16.714 INFO:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2019-12-05T05:54:16.819 INFO:teuthology.orchestra.run.smithi016.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-12-05T05:54:16.881 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2019-12-05T05:54:16.909 INFO:teuthology.run_tasks:Running task internal.timer... 2019-12-05T05:54:16.940 INFO:teuthology.task.internal:Starting timer... 2019-12-05T05:54:16.940 INFO:teuthology.run_tasks:Running task pcp... 2019-12-05T05:54:16.973 INFO:teuthology.run_tasks:Running task selinux... 2019-12-05T05:54:16.985 INFO:teuthology.orchestra.run.smithi016:> sudo service auditd rotate 2019-12-05T05:54:17.167 INFO:teuthology.orchestra.run.smithi016.stdout:Rotating logs: [ OK ] 2019-12-05T05:54:17.170 INFO:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2019-12-05T05:54:17.332 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: [ OK ] 2019-12-05T05:54:17.335 DEBUG:teuthology.task.selinux:Getting current SELinux state 2019-12-05T05:54:17.335 INFO:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2019-12-05T05:54:17.421 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2019-12-05T05:54:17.528 INFO:teuthology.orchestra.run.smithi016:> /usr/sbin/getenforce 2019-12-05T05:54:17.631 INFO:teuthology.orchestra.run.smithi016.stdout:Permissive 2019-12-05T05:54:17.632 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi016.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi076.front.sepia.ceph.com': 'permissive'} 2019-12-05T05:54:17.632 INFO:teuthology.orchestra.run.smithi076:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-12-05T05:54:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T05:54:17.751 INFO:teuthology.orchestra.run.smithi016:> sudo grep 'avc: .*denied' /var/log/audit/audit.log | grep -v '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\)' 2019-12-05T05:54:17.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2019-12-05T05:54:17.876 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2019-12-05T05:54:17.876 INFO:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2019-12-05T05:54:18.014 INFO:teuthology.orchestra.run.smithi016:> sudo /usr/sbin/setenforce permissive 2019-12-05T05:54:18.154 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2019-12-05T05:54:18.165 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2019-12-05T05:54:18.191 INFO:teuthology.repo_utils:Fetching master from origin 2019-12-05T05:54:18.286 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to branch origin/master 2019-12-05T05:54:18.323 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2019-12-05T05:54:18.328 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi076.front.sepia.ceph.com,smithi016.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2019-12-05T05:54:18.766 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2019-12-05T05:54:22.725 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2019-12-05T05:54:22.751 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:22 +0000 (0:00:00.077) 0:00:00.077 ***** 2019-12-05T05:54:22.764 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:22 +0000 (0:00:00.013) 0:00:00.090 ***** 2019-12-05T05:54:25.981 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi076.front.sepia.ceph.com] 2019-12-05T05:54:26.082 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:26 +0000 (0:00:03.317) 0:00:03.408 ***** 2019-12-05T05:54:26.168 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] 2019-12-05T05:54:26.236 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:26 +0000 (0:00:00.153) 0:00:03.562 ***** 2019-12-05T05:54:26.544 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-05T05:54:26.579 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:26 +0000 (0:00:00.343) 0:00:03.905 ***** 2019-12-05T05:54:26.601 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-05T05:54:26.642 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:26 +0000 (0:00:00.062) 0:00:03.967 ***** 2019-12-05T05:54:27.248 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-12-05T05:54:27.250 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-12-05T05:54:27.287 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:27 +0000 (0:00:00.645) 0:00:04.613 ***** 2019-12-05T05:54:27.314 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2019-12-05T05:54:27.316 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:54:27.354 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:27 +0000 (0:00:00.067) 0:00:04.680 ***** 2019-12-05T05:54:27.360 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2019-12-05T05:54:27.363 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2019-12-05T05:54:27.401 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:27 +0000 (0:00:00.047) 0:00:04.727 ***** 2019-12-05T05:54:27.428 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2019-12-05T05:54:27.430 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:54:27.466 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:27 +0000 (0:00:00.065) 0:00:04.793 ***** 2019-12-05T05:54:27.799 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2019-12-05T05:54:27.801 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T05:54:27.836 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:27 +0000 (0:00:00.370) 0:00:05.163 ***** 2019-12-05T05:54:27.842 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T05:54:27.881 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:27 +0000 (0:00:00.044) 0:00:05.207 ***** 2019-12-05T05:54:29.278 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2019-12-05T05:54:29.280 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575525267.9-43679926729182/source", "state": "file", "uid": 0} 2019-12-05T05:54:29.332 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:29 +0000 (0:00:01.451) 0:00:06.658 ***** 2019-12-05T05:54:29.340 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575525267.85-139401663885459/source", "state": "file", "uid": 0} 2019-12-05T05:54:29.385 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:29 +0000 (0:00:00.052) 0:00:06.711 ***** 2019-12-05T05:54:29.980 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2019-12-05T05:54:29.981 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:30.004 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:30.308 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:30.377 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:30.606 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:30.723 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:30.957 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:31.047 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:31.235 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:31.402 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:31.549 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:31.738 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:31.885 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:32.086 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2019-12-05T05:54:32.208 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2019-12-05T05:54:32.215 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2019-12-05T05:54:32.466 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:32 +0000 (0:00:03.081) 0:00:09.792 ***** 2019-12-05T05:54:32.507 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:32 +0000 (0:00:00.041) 0:00:09.833 ***** 2019-12-05T05:54:33.424 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi016.front.sepia.ceph.com] 2019-12-05T05:54:33.480 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] 2019-12-05T05:54:33.527 INFO:teuthology.task.ansible.out: [WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2019-12-05T05:54:33.529 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:33 +0000 (0:00:01.020) 0:00:10.854 ***** 2019-12-05T05:54:33.601 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:33 +0000 (0:00:00.073) 0:00:10.927 ***** 2019-12-05T05:54:34.036 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** changed: [smithi016.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.226332", "end": "2019-12-05 05:54:34.017702", "rc": 0, "start": "2019-12-05 05:54:33.791370", "stderr": "", "stderr_lines": [], "stdout": "7b315d00bbff69e651613e1408370c0721c0c099", "stdout_lines": ["7b315d00bbff69e651613e1408370c0721c0c099"]} 2019-12-05T05:54:34.071 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:34 +0000 (0:00:00.469) 0:00:11.397 ***** 2019-12-05T05:54:34.080 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.231707", "end": "2019-12-05 05:54:34.022957", "rc": 0, "start": "2019-12-05 05:54:33.791250", "stderr": "", "stderr_lines": [], "stdout": "7b315d00bbff69e651613e1408370c0721c0c099", "stdout_lines": ["7b315d00bbff69e651613e1408370c0721c0c099"]} 2019-12-05T05:54:34.114 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:34 +0000 (0:00:00.043) 0:00:11.441 ***** 2019-12-05T05:54:34.427 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2019-12-05T05:54:34.984 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014476", "end": "2019-12-05 05:54:34.400977", "failed_when_result": false, "rc": 0, "start": "2019-12-05 05:54:34.386501", "stderr": "", "stderr_lines": [], "stdout": "7b088106a1031364e6869fc2888a0e2542579008", "stdout_lines": ["7b088106a1031364e6869fc2888a0e2542579008"]} changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.011214", "end": "2019-12-05 05:54:34.406720", "failed_when_result": false, "rc": 0, "start": "2019-12-05 05:54:34.395506", "stderr": "", "stderr_lines": [], "stdout": "7b088106a1031364e6869fc2888a0e2542579008", "stdout_lines": ["7b088106a1031364e6869fc2888a0e2542579008"]} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.352) 0:00:11.793 ***** Thursday 05 December 2019 05:54:34 +0000 (0:00:00.034) 0:00:11.827 ***** TASK [users : Determine if we can skip users and pubkeys updates] ************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.043) 0:00:11.871 ***** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.050) 0:00:11.922 ***** TASK [users : Merge extra_admin_users into managed_admin_users] **************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.043) 0:00:11.965 ***** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.043) 0:00:12.008 ***** TASK [users : Remove managed_admin_users from managed_users] ******************* skipping: [smithi016.fro 2019-12-05T05:54:34.985 INFO:teuthology.task.ansible.out:nt.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.043) 0:00:12.051 ***** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.042) 0:00:12.094 ***** TASK [users : Filter the managed_users list] *********************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.047) 0:00:12.141 ***** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.043) 0:00:12.184 ***** TASK [users : Filter the managed_admin_users list] ***************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.045) 0:00:12.229 ***** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Thursday 05 December 2019 05:54:34 +0000 (0:00:00.043) 0:00:12.273 ***** 2019-12-05T05:54:35.221 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2019-12-05T05:54:35.223 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T05:54:35.259 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:35 +0000 (0:00:00.312) 0:00:12.585 ***** 2019-12-05T05:54:35.285 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T05:54:35.321 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:54:35 +0000 (0:00:00.062) 0:00:12.648 ***** 2019-12-05T05:54:35.649 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-12-05T05:54:35.650 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-12-05T05:54:35.716 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2019-12-05T05:54:35.990 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-12-05T05:54:36.086 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2019-12-05T05:54:36.357 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-12-05T05:54:36.370 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2019-12-05T05:54:36.694 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-12-05T05:54:36.728 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/adeza", "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "move_home": false, "name": "adeza", "shell": "/bin/bash", "state": "present", "uid": 1104} 2019-12-05T05:54:37.028 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-12-05T05:54:37.051 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1105} 2019-12-05T05:54:37.384 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-12-05T05:54:37.385 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1106} 2019-12-05T05:54:37.736 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-12-05T05:54:37.744 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1107} 2019-12-05T05:54:37.991 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-12-05T05:54:38.078 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1108} 2019-12-05T05:54:38.329 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-12-05T05:54:38.424 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2019-12-05T05:54:38.696 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-12-05T05:54:38.781 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2019-12-05T05:54:39.040 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-12-05T05:54:39.135 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2019-12-05T05:54:39.355 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-12-05T05:54:39.480 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2019-12-05T05:54:39.698 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-12-05T05:54:39.813 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1114} 2019-12-05T05:54:40.012 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-12-05T05:54:40.148 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1115} 2019-12-05T05:54:40.337 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-12-05T05:54:40.505 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1116} 2019-12-05T05:54:40.663 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-12-05T05:54:40.841 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1117} 2019-12-05T05:54:40.989 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-12-05T05:54:41.185 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1118} 2019-12-05T05:54:41.314 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-12-05T05:54:41.553 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1119} 2019-12-05T05:54:41.645 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-12-05T05:54:41.933 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/gmeno", "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "move_home": false, "name": "gmeno", "shell": "/bin/bash", "state": "present", "uid": 1120} 2019-12-05T05:54:41.996 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-12-05T05:54:42.292 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1121} 2019-12-05T05:54:42.325 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-12-05T05:54:42.652 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1122} 2019-12-05T05:54:42.656 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-12-05T05:54:42.996 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-12-05T05:54:43.006 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/zyan", "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "move_home": false, "name": "zyan", "shell": "/bin/bash", "state": "present", "uid": 1123} 2019-12-05T05:54:43.319 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-12-05T05:54:43.364 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1124} 2019-12-05T05:54:43.650 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-12-05T05:54:43.715 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1125} 2019-12-05T05:54:43.978 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-12-05T05:54:44.062 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1126} 2019-12-05T05:54:44.292 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-12-05T05:54:44.387 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1127} 2019-12-05T05:54:44.629 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-12-05T05:54:44.856 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1128} 2019-12-05T05:54:44.967 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-12-05T05:54:45.074 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1129} 2019-12-05T05:54:45.291 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-12-05T05:54:45.408 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1130} 2019-12-05T05:54:45.612 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-12-05T05:54:45.763 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1131} 2019-12-05T05:54:45.928 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-12-05T05:54:46.110 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1132} 2019-12-05T05:54:46.271 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-12-05T05:54:46.462 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1133} 2019-12-05T05:54:46.604 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-12-05T05:54:46.831 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1134} 2019-12-05T05:54:46.938 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-12-05T05:54:47.184 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1136} 2019-12-05T05:54:47.281 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-12-05T05:54:47.527 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1137} 2019-12-05T05:54:47.635 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-12-05T05:54:47.871 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1138} 2019-12-05T05:54:47.981 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-12-05T05:54:48.226 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1139} 2019-12-05T05:54:48.355 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-12-05T05:54:48.568 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1140} 2019-12-05T05:54:48.695 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-05T05:54:48.897 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1141} 2019-12-05T05:54:49.054 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-12-05T05:54:49.244 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-05T05:54:49.402 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-12-05T05:54:49.591 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1143} 2019-12-05T05:54:49.731 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-12-05T05:54:49.934 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1144} 2019-12-05T05:54:50.092 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-12-05T05:54:50.263 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1146} 2019-12-05T05:54:50.454 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-12-05T05:54:50.634 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1147} 2019-12-05T05:54:50.804 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-12-05T05:54:50.994 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1148} 2019-12-05T05:54:51.142 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-12-05T05:54:51.353 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1150} 2019-12-05T05:54:51.460 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-05T05:54:51.719 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1153} 2019-12-05T05:54:51.786 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-12-05T05:54:52.068 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-05T05:54:52.112 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-12-05T05:54:52.419 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1156} 2019-12-05T05:54:52.446 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-12-05T05:54:52.759 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1157} 2019-12-05T05:54:52.781 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-12-05T05:54:53.101 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-12-05T05:54:53.113 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1158} 2019-12-05T05:54:53.435 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-12-05T05:54:53.447 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1159} 2019-12-05T05:54:53.765 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-12-05T05:54:53.790 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1161} 2019-12-05T05:54:54.104 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-12-05T05:54:54.140 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1162} 2019-12-05T05:54:54.449 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-12-05T05:54:54.494 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1163} 2019-12-05T05:54:54.788 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-12-05T05:54:54.849 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1164} 2019-12-05T05:54:55.122 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-12-05T05:54:55.219 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1165} 2019-12-05T05:54:55.447 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-12-05T05:54:55.558 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1167} 2019-12-05T05:54:55.769 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-12-05T05:54:55.891 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1168} 2019-12-05T05:54:56.093 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-12-05T05:54:56.187 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1169} 2019-12-05T05:54:56.439 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-12-05T05:54:56.535 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1170} 2019-12-05T05:54:56.777 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-12-05T05:54:56.878 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1171} 2019-12-05T05:54:57.126 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-05T05:54:57.236 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1172} 2019-12-05T05:54:57.444 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-12-05T05:54:57.599 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1174} 2019-12-05T05:54:57.764 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-12-05T05:54:57.944 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1142} 2019-12-05T05:54:58.115 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-12-05T05:54:58.291 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1177} 2019-12-05T05:54:58.444 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-12-05T05:54:58.655 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1178} 2019-12-05T05:54:58.785 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-12-05T05:54:58.986 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1179} 2019-12-05T05:54:59.115 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-12-05T05:54:59.370 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1180} 2019-12-05T05:54:59.469 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-12-05T05:54:59.737 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1181} 2019-12-05T05:54:59.835 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-12-05T05:55:00.122 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1182} 2019-12-05T05:55:00.134 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-12-05T05:55:00.461 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1184} 2019-12-05T05:55:00.473 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-12-05T05:55:00.804 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1185} 2019-12-05T05:55:00.827 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-12-05T05:55:01.124 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1186} 2019-12-05T05:55:01.147 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-12-05T05:55:01.494 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-12-05T05:55:01.507 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1187} 2019-12-05T05:55:01.856 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-12-05T05:55:01.877 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1188} 2019-12-05T05:55:02.176 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-12-05T05:55:02.241 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1189} 2019-12-05T05:55:02.523 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-12-05T05:55:02.572 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1190} 2019-12-05T05:55:02.858 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-12-05T05:55:02.924 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1191} 2019-12-05T05:55:03.189 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-12-05T05:55:03.279 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1192} 2019-12-05T05:55:03.544 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-12-05T05:55:03.620 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1193} 2019-12-05T05:55:03.872 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-12-05T05:55:03.958 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1194} 2019-12-05T05:55:04.188 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-12-05T05:55:04.315 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1195} 2019-12-05T05:55:04.514 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-12-05T05:55:04.661 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1196} 2019-12-05T05:55:04.842 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-12-05T05:55:05.010 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1197} 2019-12-05T05:55:05.179 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-12-05T05:55:05.338 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1198} 2019-12-05T05:55:05.512 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-12-05T05:55:05.676 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1199} 2019-12-05T05:55:05.845 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-12-05T05:55:06.002 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1200} 2019-12-05T05:55:06.171 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-12-05T05:55:06.361 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1201} 2019-12-05T05:55:06.545 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-12-05T05:55:06.768 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1202} 2019-12-05T05:55:06.888 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-12-05T05:55:07.131 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1203} 2019-12-05T05:55:07.226 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-12-05T05:55:07.491 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1204} 2019-12-05T05:55:07.556 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-12-05T05:55:07.844 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1205} 2019-12-05T05:55:07.883 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1209} 2019-12-05T05:55:08.200 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1206} 2019-12-05T05:55:08.201 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1210} 2019-12-05T05:55:08.524 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1211} 2019-12-05T05:55:08.558 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1207} 2019-12-05T05:55:08.859 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1212} 2019-12-05T05:55:08.913 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1208} 2019-12-05T05:55:09.208 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1213} 2019-12-05T05:55:09.256 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1209} 2019-12-05T05:55:09.493 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1214} 2019-12-05T05:55:09.600 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1210} 2019-12-05T05:55:09.819 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-05T05:55:09.927 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1211} 2019-12-05T05:55:10.189 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10102, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 10102} 2019-12-05T05:55:10.278 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1212} 2019-12-05T05:55:10.625 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1213} 2019-12-05T05:55:10.848 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "name": "rraja", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-12-05T05:55:10.980 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:10 +0000 (0:00:35.658) 0:00:48.306 ***** 2019-12-05T05:55:10.986 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1214} 2019-12-05T05:55:11.002 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-12-05T05:55:11.043 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:11 +0000 (0:00:00.063) 0:00:48.369 ***** 2019-12-05T05:55:11.112 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2019-12-05T05:55:11.116 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-12-05T05:55:11.117 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7 2019-12-05T05:55:11.119 INFO:teuthology.task.ansible.out:L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32 2019-12-05T05:55:11.120 INFO:teuthology.task.ansible.out:f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b 2019-12-05T05:55:11.122 INFO:teuthology.task.ansible.out:09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e 2019-12-05T05:55:11.123 INFO:teuthology.task.ansible.out:66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa 2019-12-05T05:55:11.125 INFO:teuthology.task.ansible.out:8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e 2019-12-05T05:55:11.126 INFO:teuthology.task.ansible.out:5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f 2019-12-05T05:55:11.128 INFO:teuthology.task.ansible.out:92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42d 2019-12-05T05:55:11.129 INFO:teuthology.task.ansible.out:a0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4i 2019-12-05T05:55:11.131 INFO:teuthology.task.ansible.out:resis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}]}, "changed": false} 2019-12-05T05:55:11.171 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:11 +0000 (0:00:00.128) 0:00:48.497 ***** 2019-12-05T05:55:11.324 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-12-05T05:55:11.324 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1155} 2019-12-05T05:55:11.659 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 10102, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 10102} 2019-12-05T05:55:12.120 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-12-05T05:55:12.122 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com -> localhost] => {"after": "7b315d00bbff69e651613e1408370c0721c0c099", "attempts": 1, "before": "7b315d00bbff69e651613e1408370c0721c0c099", "changed": false, "remote_url_changed": false} 2019-12-05T05:55:12.158 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:12 +0000 (0:00:00.986) 0:00:49.484 ***** 2019-12-05T05:55:12.269 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2019-12-05T05:55:12.269 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "name": "rraja", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2019-12-05T05:55:12.402 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:12 +0000 (0:00:00.243) 0:00:49.727 ***** 2019-12-05T05:55:12.429 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2019-12-05T05:55:12.466 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:12 +0000 (0:00:00.065) 0:00:49.793 ***** 2019-12-05T05:55:12.516 INFO:teuthology.task.ansible.out: 2019-12-05T05:55:12.517 INFO:teuthology.task.ansible.out:TASK [users : Merge managed_users and managed_admin users] ********************* 2019-12-05T05:55:12.524 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "tr 2019-12-05T05:55:12.526 INFO:teuthology.task.ansible.out:ociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7 2019-12-05T05:55:12.528 INFO:teuthology.task.ansible.out:L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32 2019-12-05T05:55:12.529 INFO:teuthology.task.ansible.out:f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b 2019-12-05T05:55:12.532 INFO:teuthology.task.ansible.out:09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e 2019-12-05T05:55:12.533 INFO:teuthology.task.ansible.out:66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa 2019-12-05T05:55:12.535 INFO:teuthology.task.ansible.out:8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 2019-12-05T05:55:12.537 INFO:teuthology.task.ansible.out:6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f8 2019-12-05T05:55:12.538 INFO:teuthology.task.ansible.out:48846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8a 2019-12-05T05:55:12.540 INFO:teuthology.task.ansible.out:a01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalho 2019-12-05T05:55:12.542 INFO:teuthology.task.ansible.out:jr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}]}, "changed": false} 2019-12-05T05:55:12.579 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:12 +0000 (0:00:00.112) 0:00:49.905 ***** 2019-12-05T05:55:12.606 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T05:55:12.607 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-12-05T05:55:12.975 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-12-05T05:55:13.264 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2019-12-05T05:55:13.266 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com -> localhost] => {"after": "7b315d00bbff69e651613e1408370c0721c0c099", "attempts": 1, "before": "7b315d00bbff69e651613e1408370c0721c0c099", "changed": false, "remote_url_changed": false} 2019-12-05T05:55:13.303 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:13 +0000 (0:00:00.724) 0:00:50.629 ***** 2019-12-05T05:55:13.340 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T05:55:13.341 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-12-05T05:55:13.661 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-12-05T05:55:13.714 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2019-12-05T05:55:13.994 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-12-05T05:55:13.996 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-12-05T05:55:14.078 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2019-12-05T05:55:14.332 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-12-05T05:55:14.435 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2019-12-05T05:55:14.693 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-12-05T05:55:14.694 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-12-05T05:55:14.785 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCWy2i6Iv6IWLh53Z44O4fQ0ctxR8odCAWIZAaSQXAxxC++RS+95Zr+BZ5wJwsBOyP1G4AAhl+9uwnn3NmDDG7fKGexPoQrTZZ27KdJ3bHpZRq5LJf9byWFwz4rDsqIqkGjI9sUkfa6nV7ci2BdEkIgLupTT2uXUzTb/njOjI7Fc+cQ7ac4QfJ+yeIcqxeje/YYDF3068V4z1yynk9q2KRXYdERKQghaWDNJlbdBhn/fmsam6okOtUnsXjE88qGPd+LVHWfK0EfYvYXiAoOaJDK8d4EXnJ4RKJceQPBk8ravuQtKVwnRIr8WtmILHuoztthfwqb2WvZ5iMDt6v7oQZ alfredo@papaya.local", "key_options": null, "keyfile": "/home/adeza/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adeza", "validate_certs": true} 2019-12-05T05:55:15.034 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-12-05T05:55:15.148 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZtIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "va 2019-12-05T05:55:15.150 INFO:teuthology.task.ansible.out:lidate_certs": true} 2019-12-05T05:55:15.317 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-12-05T05:55:15.328 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:15.475 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2019-12-05T05:55:15.652 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-12-05T05:55:15.653 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-12-05T05:55:15.663 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:15.674 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:15.685 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:15.811 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZtt 2019-12-05T05:55:15.813 INFO:teuthology.task.ansible.out:Edz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2019-12-05T05:55:15.983 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-12-05T05:55:16.160 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2019-12-05T05:55:16.317 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-12-05T05:55:16.319 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-12-05T05:55:16.505 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2019-12-05T05:55:16.517 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:16.618 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-12-05T05:55:16.855 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uT 2019-12-05T05:55:16.856 INFO:teuthology.task.ansible.out:fMorbnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2019-12-05T05:55:16.876 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:16.894 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:16.901 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f4 2019-12-05T05:55:16.907 INFO:teuthology.task.ansible.out:3d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-12-05T05:55:17.229 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2019-12-05T05:55:17.249 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-12-05T05:55:17.601 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-12-05T05:55:17.612 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+ 2019-12-05T05:55:17.613 INFO:teuthology.task.ansible.out:2vtSoAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2019-12-05T05:55:17.944 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-12-05T05:55:17.955 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2019-12-05T05:55:18.274 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-12-05T05:55:18.309 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2019-12-05T05:55:18.604 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-12-05T05:55:18.605 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-12-05T05:55:18.658 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvWnKfzTd574C4GXg1mr4pbJOInKv4qMRf0DwksTzqtXStlhT+PaO2C8p48U2Xjz5hmPK9J+iHA2xKwI6lKI194wUo3tW283euRyTCN7LlbV6tgD9k78hDaeA+53MHYmwbxjHEqAadqrLU+68LfmL7gaaUSToP/Ap+hdcDC7H9lulMcxkUeahEiDfG2s1mD8IUM6XVuqZM9WEARKpF1oSEBKhMMob9YJZPpJWipuAGkPvDA0yHK1lB58Yr2x0yaezVjQc7V2dwaY1gXR6AscU5WzlwWNNwXHqPdyyWRaGx/9Fk00yfzX4V7ZqTvKdMLKwB7EnamduTMPsaoKBUBJCh gmeno@gmeno-virtual-machine", "key_options": null, "keyfile": "/home/gmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gmeno", "validate_certs": true} 2019-12-05T05:55:18.939 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-12-05T05:55:18.940 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-12-05T05:55:19.002 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2019-12-05T05:55:19.274 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-12-05T05:55:19.358 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2019-12-05T05:55:19.609 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-12-05T05:55:19.714 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCtmxLK2rSomB14d9/fkAshmsXQfd6cJ+vEb0MZDaUT+OLmr1vAq0+H5g9w3hApJ2UEOFSaiz6ywgBAFeBeu/KmUALOhJIKKk0imoTunYclqxVPBK6tG/2b+ME8VE1afoEpMem3Bzfkrq0pDt6DLFBDlt9rRIqtO/ZlqEayWkEwTXJ7pwj+oJS03Z3Y7q0Iqw8PToKqJWwy2OvW1I9Pi0Q8s4YV9I34rSO7Wv4w43AXpFHRlvJWIcI5g5WFI5tnjel+kUAWMZft9ynFoAP5iXdByq897r4jAF74hapaxLMsHBz4MyYmPZqS76Cx2e3kGqXeetPinfcZxC3ViboR7o/1 root@zhyan-desk1", "key_options": null, "keyfile": "/home/zyan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zyan", "validate_certs": true} 2019-12-05T05:55:19.954 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-12-05T05:55:19.956 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-12-05T05:55:20.060 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1 2019-12-05T05:55:20.061 INFO:teuthology.task.ansible.out:Z9CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2019-12-05T05:55:20.280 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-12-05T05:55:20.427 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkR 2019-12-05T05:55:20.429 INFO:teuthology.task.ansible.out:hoBYB0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2019-12-05T05:55:20.620 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-12-05T05:55:20.778 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2019-12-05T05:55:20.944 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-12-05T05:55:21.101 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2019-12-05T05:55:21.217 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-12-05T05:55:21.465 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwm 2019-12-05T05:55:21.467 INFO:teuthology.task.ansible.out:ih46TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2019-12-05T05:55:21.521 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-12-05T05:55:21.830 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2019-12-05T05:55:21.872 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-12-05T05:55:21.874 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-12-05T05:55:22.166 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2019-12-05T05:55:22.219 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-12-05T05:55:22.526 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2019-12-05T05:55:22.590 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-12-05T05:55:22.861 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2019-12-05T05:55:22.935 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-12-05T05:55:23.232 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2019-12-05T05:55:23.299 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-12-05T05:55:23.574 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2 2019-12-05T05:55:23.576 INFO:teuthology.task.ansible.out:CfQvUxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2019-12-05T05:55:23.627 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-12-05T05:55:23.916 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2019-12-05T05:55:23.985 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-12-05T05:55:24.250 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2019-12-05T05:55:24.304 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-05T05:55:24.609 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2019-12-05T05:55:24.632 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-12-05T05:55:24.976 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-12-05T05:55:24.977 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2019-12-05T05:55:25.301 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-12-05T05:55:25.303 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-12-05T05:55:25.327 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2019-12-05T05:55:25.670 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-12-05T05:55:25.672 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-12-05T05:55:25.692 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2019-12-05T05:55:26.024 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl 2019-12-05T05:55:26.025 INFO:teuthology.task.ansible.out:/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-12-05T05:55:26.376 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2019-12-05T05:55:26.387 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-12-05T05:55:26.719 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDH/YT1mGZg50qDdngGQ4Ea+jfJGZ582J6pQbYkrvuGsyATMzNwk73sAbn7mgbpw7nkW/ol/vwahXHZ5heC056BFnqiZ7tjoDeotDx/1XxFtkbyIaVflUeD1vBww38v6IslLHcVhPilauPVZSfVzEHI7JB0a+9OwNrEH3+dt1B6vR+WRy2gHG/aEJffPMv4yv/0FgRWy3cEiUvGlLq0KVVTbsqqLQhw/u5k8bBIoibBtZLFAMTrjLYp9Dv8dxJROvmey6O2TuX6gCY1rLGTAArYULpXH73OLIqaksJKc3srgoJrTSZPO6bo3BQeLz4ODJ7uvTpvQIhacBpp/dq1ln7p owasserm@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCiqChxNdielMPBchs0u6bxWstqU9oeY2K9qf1PNwEIrwDYTrpDjsdoVnS750tOBjWY0wr9Gf9ATOJXi89151oao9GaUueQ1rTgmu7GtPlKsuBUVNMFzLEa0mqneqDA4AfknisFt/LnL8fLM4BddT25tmQ4oUGS7F17p4kXqwboK/DsJciCMWWc/5AT4NVwdnYFzIA6ta25/bB94kpktaZ6HDw7G8q/mK62Yza3EZfq8SP7p0sPL7oVsInQ9AutcVonmALgzPa9t1/MPGaXUIqARaWtwe4MSlNEIs4afLFyteBVeq+dgU01yeyjFivwVmuS+aPFTm6sjGGe8/gOsFDIJiVertOqi663m7q1WYGlVANfif7uL2+uJ9yrvOmrFh5s/CJMJBRG1SLfK1j/ivFwvPAQc/uMqEIRzrVZOsuNLPO4zHPWzmBAPUteb+2BqSdUFYMlBxb3lmhFFffroDqnOK1nbdAfmyKiosUizQgX2Gs2wjjtejuhS9dpYkNu+sBOujsxa2mZUouyWf4fXyWGVjthrCZaM+FyeSDpJrgwgRiSKNB8aRxB10ZNo9WZEy6eH2adp0Xf6v2KsKvryDndqBEkBI8zLbGrPVHGSEqiSBWc+NeAB06uSscJd0pHU3hDg0qW0EaxOXaial9qrBiwcFVKEr0g7LDURQ6KZsZy9w== owasserm@owasserm.redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2019-12-05T05:55:26.730 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-12-05T05:55:27.052 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6 2019-12-05T05:55:27.053 INFO:teuthology.task.ansible.out:AKY1OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2019-12-05T05:55:27.085 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2019-12-05T05:55:27.087 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-05T05:55:27.402 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2Ruq 2019-12-05T05:55:27.403 INFO:teuthology.task.ansible.out:iZyKz2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2019-12-05T05:55:27.447 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-12-05T05:55:27.449 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-12-05T05:55:27.450 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-12-05T05:55:27.748 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2019-12-05T05:55:27.792 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-12-05T05:55:28.096 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2019-12-05T05:55:28.138 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-12-05T05:55:28.447 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2019-12-05T05:55:28.478 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-12-05T05:55:28.480 INFO:teuthology.task.ansible.out: 2019-12-05T05:55:28.811 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options" 2019-12-05T05:55:28.814 INFO:teuthology.task.ansible.out:: null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-12-05T05:55:29.151 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-12-05T05:55:29.193 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+z 2019-12-05T05:55:29.199 INFO:teuthology.task.ansible.out:mfjEfdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbB 2019-12-05T05:55:29.200 INFO:teuthology.task.ansible.out:mS+dTzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2019-12-05T05:55:29.485 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-12-05T05:55:29.507 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2019-12-05T05:55:29.793 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-12-05T05:55:29.866 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2019-12-05T05:55:30.104 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-12-05T05:55:30.208 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2019-12-05T05:55:30.437 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-12-05T05:55:30.439 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-12-05T05:55:30.564 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2019-12-05T05:55:30.792 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-12-05T05:55:30.897 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2019-12-05T05:55:31.074 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-12-05T05:55:31.282 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com", "key_options": null, "keyfile": "/home/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2019-12-05T05:55:31.418 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-12-05T05:55:31.420 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-12-05T05:55:31.636 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2019-12-05T05:55:31.772 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-12-05T05:55:31.976 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEudQMGBPqfZBV8bXpdQOgVTDP6GFsr5oL4gdbHFMuJCGe0Sfq8LTaSEBQx0MH7tXc7JJ1fU51U7DSt0kfy/PFeV1Y5XnNATiZu/SzdO3jHe06/8CCr3DLnlEmzXASq+tqDJbY1eV011d/TX1jiC8Qd1VNbpdZYExEDsDzwJXnWdh6rjTL4anGjIgQWcmU0XdbUJXvRP4dGjVvZVHZnflAkxLPBbAx/8BBOuBmv6vP4hNs4poi4VLwR6S6Ha1fgMJmsn6StsCVuOvKBJFuGvaulHgHgdoDJ1ryUFikkXXCWoNx4IbwtpQIkSxWMuAovlwwk9G8ozSs+RrFzCavE0f omw@omw-dev", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2019-12-05T05:55:32.103 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-12-05T05:55:32.333 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user 2019-12-05T05:55:32.335 INFO:teuthology.task.ansible.out:": "gabrioux", "validate_certs": true} 2019-12-05T05:55:32.449 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-12-05T05:55:32.676 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2019-12-05T05:55:32.797 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-05T05:55:33.022 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2019-12-05T05:55:33.137 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-12-05T05:55:33.390 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS 2019-12-05T05:55:33.391 INFO:teuthology.task.ansible.out:33Kkke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2019-12-05T05:55:33.467 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-12-05T05:55:33.749 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2019-12-05T05:55:33.791 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2019-12-05T05:55:34.095 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2019-12-05T05:55:34.137 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-12-05T05:55:34.437 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2019-12-05T05:55:34.478 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-12-05T05:55:34.771 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2019-12-05T05:55:34.817 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-12-05T05:55:35.110 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2019-12-05T05:55:35.162 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-12-05T05:55:35.457 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2019-12-05T05:55:35.499 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-12-05T05:55:35.844 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh 2019-12-05T05:55:35.845 INFO:teuthology.task.ansible.out:", "validate_certs": true} 2019-12-05T05:55:36.162 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-12-05T05:55:36.205 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2019-12-05T05:55:36.498 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-12-05T05:55:36.541 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2019-12-05T05:55:36.838 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-12-05T05:55:36.903 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2019-12-05T05:55:37.172 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-12-05T05:55:37.236 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2019-12-05T05:55:37.512 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-12-05T05:55:37.586 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2019-12-05T05:55:37.862 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-12-05T05:55:37.915 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2019-12-05T05:55:38.196 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-12-05T05:55:38.259 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2019-12-05T05:55:38.513 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-12-05T05:55:38.600 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9/Io6M3q8yQqoPNPKAPJ6Fzq18DMfmNtxcD8GUWNo0xcMxd7XOlarvYc7INDQjkpzx6s+aqMme55pSUZuN5plW+nuFmRPqBaIxyeCnNjzCch0wbtEZe5dzT270FhzM1Tu4W0dw+WD6zrGjXayLNs2uxNp3UrWtMTBRzd0IfxJ1R2aJtiCE/HNxIZn+bhS/OmEPV1izz23MvKXD+nRaeJK6Aqrg1zbnpoNskz2fZ/AnmhnAMc4EHl/NUyW1mi4g81KcQMwE8poWmw1NpZje9gWFcISVClLvvhk/A6at/v90P7PisX+/ulTA/QYJYvGBjgbNtURYu/PWzYBOAEZQGfj shilpa@dhcp35-184.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDqRSG3i8QY7t9sp6I8vDZ7LVdk+DDrMhI1QmvU2z+YQhUGDBOvvO6YP0gUXPDyalyj0sJXiwLWPmkrOsbnR96YCUcIt7Ba6R0d0w85JxcJ1WyPcsg2UyOBwq4A+s7VAFZm6FEWzlaubixsNp04gKDqBjSjaqA1Kwx8/Tg39byay8Hn3VJeZ1SkRukvhykz2e6gl7ydfuJQVmEyO3pTrd5157ZqgvQRABT10wc3vR2m4eC65hU/v0ci7HMY9oVtGEdWVBapWMuiocrOPCRWF5zrMJwW4vzPOCHa/47hBEIWfOWJidUVzAawnM3nOe7PykvvDK/iP8B6JpxKTE74eLeh shilpa@fedora.lab.eng.blr.redhat.com", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2019-12-05T05:55:38.869 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-12-05T05:55:38.963 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2019-12-05T05:55:39.223 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-12-05T05:55:39.308 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2019-12-05T05:55:39.589 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-12-05T05:55:39.683 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC709gIxs1RgiA95ddp2uymzgV84VHJVVMj71FyTlz6pa0gtlaxF+r16p8dnNWjhyJRUXMwBKFOwHx95bcdfiRA+BfC04UnI8VvBxoED5Nt9EbKD5h78ZoNEeQJKoG1dsM/kWWlTd9WCsCyzjRz32+YNWul0LOhrfloHP+LHnfjf/uOnzd8iRHgnpWr5Z9uB0MBu4y/L6AmHnR+8pMZQq6Bfno8ZLwNCzx9v+4MqOz3jY7FYv7SJIftQH0fJygIfCz3bKyUEWk9a/JJacETuJGBnHgNAXUr/FtM1IFiMl2SsTixRgnosEz4pX8EIe5LJyw/Qudro8BPEC8aIGy0XAkerGmTt5iALMnzmeeanZCFm4rF/YiFoHLMS/B0uv42MD64hB3knrV8Bcq+mRjwVOYgecO+xoiBsU6FZeXaBk5IsjmYTwataGv3ILEmNpss0WeqHxF9nd7oTLnw51cZfcaerOnJ5IZJZTXwLNVr7K176BunHggAhjvtvwdeKhphqt9f3ilLr6gzksEoAR4JBOYJJtd8sEqHwBtti+YbMGwH9Y+Z+SujlQBQIsom60xj5OzrwjsIi5JAtOhbc1Mq2fptSXRsR8ZBT8aUFqZ8R7ixEqq2F47OAznbhSQKidzwUNkWNI3WAKi38Oi4aya1k3iqtL04bGf8R/xD5Vq/2J+UlQ== vant alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2019-12-05T05:55:39.948 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-12-05T05:55:40.045 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2019-12-05T05:55:40.320 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-12-05T05:55:40.321 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-12-05T05:55:40.414 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2019-12-05T05:55:40.680 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-12-05T05:55:40.681 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-12-05T05:55:40.774 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2019-12-05T05:55:41.035 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-12-05T05:55:41.130 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2019-12-05T05:55:41.374 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-12-05T05:55:41.490 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2019-12-05T05:55:41.710 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2019-12-05T05:55:41.850 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2019-12-05T05:55:42.062 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-12-05T05:55:42.212 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2019-12-05T05:55:42.413 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-12-05T05:55:42.573 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.lo 2019-12-05T05:55:42.575 INFO:teuthology.task.ansible.out:caldomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2019-12-05T05:55:42.766 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-12-05T05:55:42.931 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfb 2019-12-05T05:55:42.933 INFO:teuthology.task.ansible.out:smXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2019-12-05T05:55:43.071 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-12-05T05:55:43.286 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz1UnJy64hXJKl1OsqRPlZ84m/fyHE8/Hqimy5s8sWcvRs2tL5oK30zFataqv1BC7r46+9t1eGUIRAuBg6QPuJnJCnPzS8yTVJ6DqSDGE2NAnpcMK5fSb+qgQ2adHFkjzZ4eKZbnDJMjYBtpUEp1Ve2ncdq5YCy6hGLIstkGq331ULBc/Lm3ZHBwy5I51S8eldkuP975otBYSPZaaN25QDncZzrW36zb3VwoGUAANEfHC1kLgTTFjx0bdsAGYCduXzyyXls8qSv9gbzcRzZZoH5B7tj8VDBoHszPnqGjGqwtWaelifYE/Svp+8fJKLlzRUM+XGNZoNcRl1/1XVhYV/ deepika@deepika", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2019-12-05T05:55:43.403 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-12-05T05:55:43.668 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2019-12-05T05:55:43.714 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-12-05T05:55:44.032 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1q3k4RvLLCaG874GDJjja4eYqFwrMjsTU+aRzambaVuCmAJT3KQMGuwMjTuCIN7R1Fqr+/dxUClamsfsGqPMe0ASHT9X/0AyNgEkr6na1Fvk9NAwAFelwtiRIHhclzYf6KY8QHZN/eBy43prVKDXd30b15gwQyVqU0/xmny0GrYCN24k3N/R3divt2GVqptYgEwS2KEyFT5EMnJWjdIVG9mBOHrgec2VF/rc6w3BZ0s56iKdpOeCgk5ISYTcom+Xrf+fSDHirlviZv05tBP8ESmampUT/EnwF8Mdaln5XJvZPhrq2JpUisi9fVeR+zlzg6gRGubvokdA3AERrhCQ9 skoduri@rgw_vm", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "s 2019-12-05T05:55:44.034 INFO:teuthology.task.ansible.out:oumyakoduri", "validate_certs": true} 2019-12-05T05:55:44.355 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-12-05T05:55:44.405 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2019-12-05T05:55:44.639 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-12-05T05:55:44.765 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2019-12-05T05:55:44.981 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-12-05T05:55:45.116 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2019-12-05T05:55:45.302 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2019-12-05T05:55:45.473 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2019-12-05T05:55:45.617 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_key 2019-12-05T05:55:45.618 INFO:teuthology.task.ansible.out:s", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2019-12-05T05:55:45.842 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2019-12-05T05:55:45.947 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2019-12-05T05:55:46.203 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2019-12-05T05:55:46.308 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2019-12-05T05:55:46.403 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:46 +0000 (0:00:33.099) 0:01:23.729 ***** 2019-12-05T05:55:46.443 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T05:55:46.444 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:46.454 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:46.465 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:46.467 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:46.477 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:46.488 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:46.503 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:46.510 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:46.512 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:46.577 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T05:55:46.579 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2019-12-05T05:55:46.850 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T05:55:46.852 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-12-05T05:55:46.864 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:46.929 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T05:55:46.930 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2019-12-05T05:55:47.198 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T05:55:47.200 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-12-05T05:55:47.277 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T05:55:47.278 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2019-12-05T05:55:47.516 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T05:55:47.518 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-12-05T05:55:47.637 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T05:55:47.638 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2019-12-05T05:55:47.856 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-12-05T05:55:47.864 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.875 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.886 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.901 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.901 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.912 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.923 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.934 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.938 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.946 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.958 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.978 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:47.992 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.003 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.013 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.024 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.035 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.047 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.058 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.069 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.070 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.081 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.103 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.104 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.106 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.115 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.126 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.140 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.141 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.152 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.163 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.174 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.185 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.197 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.208 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.219 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.230 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.241 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.253 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.264 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.275 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.276 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.287 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.298 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.299 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.320 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.322 INFO:teuthology.task.ansible.out: 2019-12-05T05:55:48.323 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.333 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.349 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Z 2019-12-05T05:55:48.353 INFO:teuthology.task.ansible.out:x3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.365 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.372 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.382 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.383 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.399 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.405 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.406 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.427 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.438 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.456 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.461 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.461 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.474 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.486 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.486 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.504 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.504 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.517 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.528 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.540 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.540 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.551 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.565 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.566 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.578 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.589 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.590 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.601 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.612 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.613 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.624 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.635 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.646 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.647 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.658 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.669 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.670 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.681 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.717 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.719 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlnlBzRdWRqns/9VrYuF9tLILUjeGd9z8Tf/dPZmxAt3nnZqpTvI2xXemQpyt/2lsD0U24EuTc1Y/dBsgqvXtZOw51TZE8RK26hvCCaKGfcKONjvR7goHAmkYPHK9+F6m7fgYkRzA0+ZW4HcWNh/xxhhL2G1EGfz5bzIoGQJHSdzc6VeVIsY7HiYEMTxvxdsgV72Kjq97BdmHm56RBzoH24oN0UzNZS8Syx5SuWj9dA3IyFBPwjLlozSk6U2Ofd7qaXuQKl807eD9g6RhRldzHPS+i0EK3EM6Jvho5Y08k1KqV4gIEwMdzDbD3SH0xq+nUPFKcDTUFNp+tkg10GxqN zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.721 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.779 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:48 +0000 (0:00:02.375) 0:01:26.105 ***** 2019-12-05T05:55:48.806 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2019-12-05T05:55:48.808 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:48.843 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:48 +0000 (0:00:00.064) 0:01:26.169 ***** 2019-12-05T05:55:49.047 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2019-12-05T05:55:49.049 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2019-12-05T05:55:49.159 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:49 +0000 (0:00:00.315) 0:01:26.485 ***** 2019-12-05T05:55:49.208 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi076.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-12-05T05:55:49.208 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9', u'name': u'andrewschoen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:49.210 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n', u'name': u'zack'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:49.220 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd', u'name': u'kdreyer'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:49.231 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47', u'name': u'adeza'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adeza", "ovpn": "alfredo@papaya.local V5VkRT2Du88WwQrYeK196w 4dff4d9bdc4d9ab60532814c338abe4534cb45a9ca729bdbc2411ad513516b47"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:49.247 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n', u'name': u'dmick'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:49.254 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n', u'name': u'sage'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:49.255 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n', u'name': u'dgalloway'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:49.271 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045', u'name': u'akraitma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:49.277 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e', u'name': u'trociny'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:49.511 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T05:55:49.513 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-12-05T05:55:49.621 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T05:55:49.622 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007', u'name': u'smithfarm', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2019-12-05T05:55:49.637 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n', u'name': u'mbenjamin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:49.847 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi076.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-12-05T05:55:49.966 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01', u'name': u'aemerson', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemerson", "validate_certs": true} 2019-12-05T05:55:50.165 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** ok: [smithi076.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-12-05T05:55:50.328 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T05:55:50.329 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78', u'name': u'amaredia', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2019-12-05T05:55:50.487 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T05:55:50.489 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-12-05T05:55:50.686 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2019-12-05T05:55:50.688 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5', u'name': u'tserlin', u'key': u'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2019-12-05T05:55:50.697 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28', u'name': u'dis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.709 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n', u'name': u'gregf'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.720 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n', u'name': u'joshd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.732 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n', u'name': u'davidz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.743 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc', u'name': u'gmeno'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gmeno", "ovpn": "gmeno@gmeno-virtual-machine FKFu8B2pMqotpmEVAO1few 8229574e499eaf767a408909f5afdf2e2a0bb8f3e61b18d63a651f7102c68dbc"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.754 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1', u'name': u'ivancich'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.756 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n', u'name': u'wusui'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.767 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T05:55:50.769 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304', u'name': u'zyan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zyan", "ovpn": "zyan@redhat OP16IkReCatMfA4Mf3pkdQ b0262be71ef008e2f7d585e34431dc2757c1e22ac1aa844863be533bf873d304"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.780 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n', u'name': u'yuriw'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH 5ivdxgFO4eIkbXVhl8xkvw 59212d29b8b42d9fe457c1b2c43d774e1d25807be10dcc1252d4aec63b97a467\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.802 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n', u'name': u'tamil'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.803 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n', u'name': u'jowilkin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.829 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4', u'name': u'bhubbard'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.830 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n', u'name': u'yehudasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.832 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4', u'name': u'dang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.843 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb', u'name': u'branto'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.865 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6', u'name': u'xiaoxichen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.867 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e', u'name': u'ffilz'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.879 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n', u'name': u'joao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.890 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n', u'name': u'nhm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.902 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313', u'name': u'nwatkins'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.916 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22', u'name': u'mkidd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af', u'name': u'jlopez'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.925 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb', u'name': u'haomaiwang'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.938 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608', u'name': u'jdillaman'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.940 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3', u'name': u'kchai'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.951 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.963 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n', u'name': u'dfuller'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.975 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n', u'name': u'owasserm'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@owasserm.redhat.com hsdVTbVub6eRnhlO9B02rQ 7c9baf41670ff9ab612f75d4be42d0aaf0d7ecaa3c8928032b61f1be91725890\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.976 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n', u'name': u'vasu'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:50.988 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37', u'name': u'smohan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.000 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114', u'name': u'abhi'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.001 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611', u'name': u'cbodley'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.013 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195', u'name': u'mwatts'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.024 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.036 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea', u'name': u'pdonnell'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.047 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868', u'name': u'jlayton'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.058 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f', u'name': u'rzarzynski'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.070 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e', u'name': u'rdias'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.081 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953', u'name': u'vshankar'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride r39frQXXj1GUJZwq1GS7fw 1170ef4c918c5ff15334d10f666441b0dfe0bb869a5e15218fdfad2e8cc4e953"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.103 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T05:55:51.104 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2019-12-05T05:55:51.105 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58', u'name': u'akupczyk'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.107 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e', u'name': u'nojha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.118 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43', u'name': u'ifed01'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.130 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4', u'name': u'myoungwon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@omw-dev 78twXwYRU+MeH+yZ9Rw9Zg 2dd66fa122e5cf3b8bfa835cefac7c6e4e66d70643a3819813104c2057e597e4"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.141 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2', u'name': u'gabrioux'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.154 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f', u'name': u'leseb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.167 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca', u'name': u'hchen'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n', u'name': u'jcollin'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.178 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc', u'name': u'xxg'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.190 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53', u'name': u'pcuzner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.201 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2', u'name': u'mkogan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.212 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed', u'name': u'vumrao'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.215 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736', u'name': u'henrix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.226 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26', u'name': u'pbs1108'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.237 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315', u'name': u'clacroix'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.248 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006', u'name': u'epuertat'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.260 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461', u'name': u'tdehler'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.271 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305', u'name': u'laura'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.282 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce', u'name': u'yaarit'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.283 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3', u'name': u'rpavani1998'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.305 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4', u'name': u'rishabh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.306 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2', u'name': u'skrah'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.326 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8', u'name': u'smanjara'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora 5oorMoVYD3sT0nmOTBDh9w 83be007f68694c9463ef46e4ce223221d639d78f11d5b68449598de77e8e0ce8"}, "skip_reason": "Conditional result was False"} skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41', u'name': u'bengland'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.337 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d', u'name': u'pnawracay'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.348 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8', u'name': u'alfonsomthd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost eUvx+7UEx5IYGmS0lNIscQ 2f1bfd4874280b9f525a46e93e767504df80a9b09a83a2fea387dcd6e34bc0f8"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.359 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e', u'name': u'oliveiradan'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.371 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba', u'name': u'swagner'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.383 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e', u'name': u'yuvalif'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.394 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852', u'name': u'kkeithle'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.394 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130', u'name': u'emmericp'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.416 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0', u'name': u'mchangir'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.427 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286', u'name': u'sidharthanup'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.450 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31', u'name': u'varsha'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.453 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T05:55:51.455 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-12-05T05:55:51.465 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n', u'name': u'sjust'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc', u'name': u'ideepika'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@thinkpad HXRUSerxILmV0ejAD/Zt1g 29b44e741567e7a2e7b664d6533247d79851fc302051c9ba34103c63a2fcfacc"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.473 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46', u'name': u'gsalomon'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.484 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de', u'name': u'soumyakoduri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@rgw_vm fAskaxmJYtm4TDcModIkrQ 884b57c3b5d56493da361dc9f1cc4e06e766628fcb0f916090f2096edc5ce7de"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.496 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a', u'name': u'kyr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.497 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849', u'name': u'sseshasa'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.519 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189', u'name': u'rfriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.521 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b', u'name': u'lmb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.532 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc', u'name': u'ksirivad'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.532 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4', u'name': u'ofriedma'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.544 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52', u'name': u'h4iresis'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.555 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5', u'name': u'shyukri'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.569 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e', u'name': u'scarvalhojr'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.570 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b', u'name': u'rakesh'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.582 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486', u'name': u'ryg-'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat 7WbRbRXPcrQ2++Bww59JKg 1186db58c70662d55cae6d75e54022d1392708d3cf742b10ce5ff9d67dd8486"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.593 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3', u'name': u'prsrivas'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.604 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42', u'name': u'zdover'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@slim_thinkpad H/78rBqDIJ5AAjMT6sTmKg 49fb166c553d14549491280458269f987bea97295dbf525ffb7c35336175ee42"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.616 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'ovpn': u'rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b', u'name': u'rraja'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@thinkpadp50 8S3Lez75woSBdYiLqQxf2g c9d1666dfc7a3b762700cd250f775a506c397b0f0e6a86289ba555a3546e532b"}, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.674 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:51 +0000 (0:00:02.514) 0:01:29.000 ***** 2019-12-05T05:55:51.702 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2019-12-05T05:55:51.703 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:55:51.742 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:55:51 +0000 (0:00:00.068) 0:01:29.069 ***** 2019-12-05T05:55:51.796 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T05:55:51.798 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-12-05T05:55:52.113 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2019-12-05T05:55:52.115 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-12-05T05:55:52.420 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-12-05T05:55:52.461 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2019-12-05T05:55:52.745 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-12-05T05:55:52.757 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2019-12-05T05:55:53.061 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-12-05T05:55:53.125 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2019-12-05T05:55:53.376 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-12-05T05:55:53.441 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2019-12-05T05:55:53.704 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-12-05T05:55:54.482 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} ok: [smithi076.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} ok: [smithi016.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} ok: [smithi076.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} ok: [smithi016.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2019-12-05T05:55:54.672 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-12-05T05:55:54.787 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2019-12-05T05:55:55.007 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-12-05T05:55:55.124 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2019-12-05T05:55:55.325 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-12-05T05:55:55.463 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2019-12-05T05:55:55.659 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-12-05T05:55:55.794 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2019-12-05T05:55:56.140 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2019-12-05T05:55:56.277 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jj", "name": "jj", "remove": false, "state": "absent"} 2019-12-05T05:55:56.458 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2019-12-05T05:55:56.774 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2019-12-05T05:55:56.831 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "amarangone", "name": "amarangone", "remove": false, "state": "absent"} 2019-12-05T05:55:57.120 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2019-12-05T05:55:57.396 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "oprypin", "name": "oprypin", "remove": false, "state": "absent"} 2019-12-05T05:55:57.397 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2019-12-05T05:55:57.729 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2019-12-05T05:55:57.960 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adamyanova", "name": "adamyanova", "remove": false, "state": "absent"} 2019-12-05T05:55:58.061 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2019-12-05T05:55:58.386 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2019-12-05T05:55:58.542 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sbillah", "name": "sbillah", "remove": false, "state": "absent"} 2019-12-05T05:55:58.699 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2019-12-05T05:55:59.129 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "onyb", "name": "onyb", "remove": false, "state": "absent"} 2019-12-05T05:55:59.225 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jj", "name": "jj", "remove": false, "state": "absent"} 2019-12-05T05:55:59.685 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jwilliamson", "name": "jwilliamson", "remove": false, "state": "absent"} 2019-12-05T05:55:59.770 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "amarangone", "name": "amarangone", "remove": false, "state": "absent"} 2019-12-05T05:56:00.251 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "kmroz", "name": "kmroz", "remove": false, "state": "absent"} 2019-12-05T05:56:00.370 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "oprypin", "name": "oprypin", "remove": false, "state": "absent"} 2019-12-05T05:56:00.796 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "shehbazj", "name": "shehbazj", "remove": false, "state": "absent"} 2019-12-05T05:56:01.001 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "adamyanova", "name": "adamyanova", "remove": false, "state": "absent"} 2019-12-05T05:56:01.327 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "remove": false, "state": "absent"} 2019-12-05T05:56:01.591 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "sbillah", "name": "sbillah", "remove": false, "state": "absent"} 2019-12-05T05:56:01.926 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "asheplyakov", "name": "asheplyakov", "remove": false, "state": "absent"} 2019-12-05T05:56:02.179 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "onyb", "name": "onyb", "remove": false, "state": "absent"} 2019-12-05T05:56:02.476 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "liupan", "name": "liupan", "remove": false, "state": "absent"} 2019-12-05T05:56:02.753 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "jwilliamson", "name": "jwilliamson", "remove": false, "state": "absent"} 2019-12-05T05:56:02.808 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-12-05T05:56:03.133 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-12-05T05:56:03.333 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "kmroz", "name": "kmroz", "remove": false, "state": "absent"} 2019-12-05T05:56:03.450 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-12-05T05:56:03.753 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-12-05T05:56:03.863 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "shehbazj", "name": "shehbazj", "remove": false, "state": "absent"} 2019-12-05T05:56:04.107 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-12-05T05:56:04.390 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "remove": false, "state": "absent"} 2019-12-05T05:56:04.423 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-12-05T05:56:04.774 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-12-05T05:56:04.973 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "asheplyakov", "name": "asheplyakov", "remove": false, "state": "absent"} 2019-12-05T05:56:05.331 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "fche", "name": "fche", "remove": false, "state": "absent"} 2019-12-05T05:56:05.389 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:05 +0000 (0:00:13.646) 0:01:42.715 ***** 2019-12-05T05:56:05.541 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "liupan", "name": "liupan", "remove": false, "state": "absent"} 2019-12-05T05:56:05.887 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2019-12-05T05:56:06.226 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2019-12-05T05:56:06.492 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "checksum": "4d6d0da35c32595244f0557c86cc20a1ae54bb67", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "283e36f6f6840b00622b73078c099c15", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575525365.41-222351861562342/source", "state": "file", "uid": 0} 2019-12-05T05:56:06.531 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:06 +0000 (0:00:01.141) 0:01:43.857 ***** 2019-12-05T05:56:06.570 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-12-05T05:56:06.573 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => { "msg": "Host smithi076.front.sepia.ceph.com is running CentOS 7.6 (Core)" } TASK [users : Remove revoked users] ******************************************** ok: [smithi016.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2019-12-05T05:56:06.628 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:06 +0000 (0:00:00.097) 0:01:43.954 ***** 2019-12-05T05:56:06.668 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi076.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-12-05T05:56:06.707 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:06 +0000 (0:00:00.079) 0:01:44.033 ***** 2019-12-05T05:56:06.755 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-12-05T05:56:06.757 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:06.795 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:06 +0000 (0:00:00.087) 0:01:44.121 ***** 2019-12-05T05:56:06.924 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T05:56:06.926 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2019-12-05T05:56:07.217 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2019-12-05T05:56:07.218 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.099354", "end": "2019-12-05 05:56:07.194919", "rc": 0, "start": "2019-12-05 05:56:07.095565", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-12-05T05:56:07.255 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:07 +0000 (0:00:00.459) 0:01:44.581 ***** 2019-12-05T05:56:07.272 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T05:56:07.273 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2019-12-05T05:56:07.285 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-12-05T05:56:07.287 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:07.336 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:07 +0000 (0:00:00.081) 0:01:44.662 ***** 2019-12-05T05:56:07.375 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-12-05T05:56:07.376 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:07.416 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:07 +0000 (0:00:00.079) 0:01:44.742 ***** 2019-12-05T05:56:07.466 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-12-05T05:56:07.468 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:07.511 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:07 +0000 (0:00:00.095) 0:01:44.837 ***** 2019-12-05T05:56:07.548 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2019-12-05T05:56:07.550 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:07.585 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:07 +0000 (0:00:00.074) 0:01:44.911 ***** 2019-12-05T05:56:07.602 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T05:56:07.604 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2019-12-05T05:56:07.620 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:07.652 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:07 +0000 (0:00:00.066) 0:01:44.978 ***** 2019-12-05T05:56:07.680 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2019-12-05T05:56:07.681 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:07.718 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:07 +0000 (0:00:00.065) 0:01:45.044 ***** 2019-12-05T05:56:07.757 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-12-05T05:56:07.759 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T05:56:07.796 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:07 +0000 (0:00:00.078) 0:01:45.122 ***** 2019-12-05T05:56:07.823 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:07.862 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:07 +0000 (0:00:00.066) 0:01:45.189 ***** 2019-12-05T05:56:07.899 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-12-05T05:56:07.900 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:07.938 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:07 +0000 (0:00:00.075) 0:01:45.264 ***** 2019-12-05T05:56:07.954 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2019-12-05T05:56:07.956 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2019-12-05T05:56:07.967 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-12-05T05:56:07.969 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T05:56:08.007 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.068) 0:01:45.333 ***** 2019-12-05T05:56:08.009 INFO:teuthology.task.ansible.out: 2019-12-05T05:56:08.046 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-12-05T05:56:08.048 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:08.087 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.080) 0:01:45.413 ***** 2019-12-05T05:56:08.126 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2019-12-05T05:56:08.128 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T05:56:08.177 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.090) 0:01:45.503 ***** 2019-12-05T05:56:08.209 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2019-12-05T05:56:08.211 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:08.248 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.070) 0:01:45.574 ***** 2019-12-05T05:56:08.277 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:08.317 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.068) 0:01:45.643 ***** 2019-12-05T05:56:08.344 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2019-12-05T05:56:08.346 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:08.383 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.066) 0:01:45.710 ***** 2019-12-05T05:56:08.410 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2019-12-05T05:56:08.412 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:08.447 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.063) 0:01:45.773 ***** 2019-12-05T05:56:08.475 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-12-05T05:56:08.476 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T05:56:08.520 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.073) 0:01:45.846 ***** 2019-12-05T05:56:08.558 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2019-12-05T05:56:08.559 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Remove revoked users] ******************************************** changed: [smithi016.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "fche", "name": "fche", "remove": false, "state": "absent"} 2019-12-05T05:56:08.600 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.075) 0:01:45.922 ***** 2019-12-05T05:56:08.665 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.068) 0:01:45.991 ***** 2019-12-05T05:56:08.671 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-12-05T05:56:08.673 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:08.716 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.051) 0:01:46.042 ***** 2019-12-05T05:56:08.755 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:08.801 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.084) 0:01:46.127 ***** 2019-12-05T05:56:08.848 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2019-12-05T05:56:08.856 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:08.894 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.093) 0:01:46.220 ***** 2019-12-05T05:56:08.946 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:08.990 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:08 +0000 (0:00:00.095) 0:01:46.316 ***** 2019-12-05T05:56:09.018 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-12-05T05:56:09.019 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:09.056 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:09 +0000 (0:00:00.066) 0:01:46.382 ***** 2019-12-05T05:56:09.085 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-12-05T05:56:09.087 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T05:56:09.124 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:09 +0000 (0:00:00.067) 0:01:46.450 ***** 2019-12-05T05:56:09.163 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-12-05T05:56:09.191 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:09 +0000 (0:00:00.066) 0:01:46.517 ***** 2019-12-05T05:56:09.218 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2019-12-05T05:56:09.220 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:09.260 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:09 +0000 (0:00:00.068) 0:01:46.586 ***** 2019-12-05T05:56:09.623 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** ok: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T05:56:09.663 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:09 +0000 (0:00:00.403) 0:01:46.989 ***** 2019-12-05T05:56:09.862 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2019-12-05T05:56:09.865 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "checksum": "4d6d0da35c32595244f0557c86cc20a1ae54bb67", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "283e36f6f6840b00622b73078c099c15", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575525368.69-205090713652731/source", "state": "file", "uid": 0} 2019-12-05T05:56:09.923 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:09 +0000 (0:00:00.259) 0:01:47.248 ***** 2019-12-05T05:56:09.951 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2019-12-05T05:56:09.953 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => { "msg": "Host smithi016.front.sepia.ceph.com is running CentOS 7.6 (Core)" } 2019-12-05T05:56:10.009 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:10 +0000 (0:00:00.086) 0:01:47.335 ***** 2019-12-05T05:56:10.061 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi016.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2019-12-05T05:56:10.096 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:10 +0000 (0:00:00.086) 0:01:47.422 ***** 2019-12-05T05:56:10.124 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2019-12-05T05:56:10.126 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:10.168 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:10 +0000 (0:00:00.072) 0:01:47.494 ***** 2019-12-05T05:56:10.592 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.082439", "end": "2019-12-05 05:56:10.561591", "rc": 0, "start": "2019-12-05 05:56:10.479152", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2019-12-05T05:56:10.633 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:10 +0000 (0:00:00.465) 0:01:47.959 ***** 2019-12-05T05:56:10.661 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2019-12-05T05:56:10.663 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:10.701 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:10 +0000 (0:00:00.067) 0:01:48.027 ***** 2019-12-05T05:56:10.732 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2019-12-05T05:56:10.734 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:10.772 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:10 +0000 (0:00:00.070) 0:01:48.098 ***** 2019-12-05T05:56:10.809 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2019-12-05T05:56:10.811 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:10.852 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:10 +0000 (0:00:00.077) 0:01:48.176 ***** 2019-12-05T05:56:10.898 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:10.942 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:10 +0000 (0:00:00.092) 0:01:48.268 ***** 2019-12-05T05:56:10.950 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-12-05T05:56:10.952 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1d6dcd385d1b7c9c2b757c62b841c887e90d7f66", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 293, "state": "file", "uid": 0} 2019-12-05T05:56:10.987 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:11.026 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.084) 0:01:48.352 ***** 2019-12-05T05:56:11.054 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:11.094 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.068) 0:01:48.420 ***** 2019-12-05T05:56:11.131 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2019-12-05T05:56:11.133 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T05:56:11.176 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.075) 0:01:48.496 ***** 2019-12-05T05:56:11.208 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2019-12-05T05:56:11.210 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:11.247 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.077) 0:01:48.573 ***** 2019-12-05T05:56:11.284 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2019-12-05T05:56:11.291 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:11.345 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.090) 0:01:48.664 ***** 2019-12-05T05:56:11.381 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2019-12-05T05:56:11.382 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T05:56:11.427 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.089) 0:01:48.753 ***** 2019-12-05T05:56:11.455 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2019-12-05T05:56:11.457 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:11.516 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.068) 0:01:48.821 ***** 2019-12-05T05:56:11.538 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** skipping: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T05:56:11.571 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.076) 0:01:48.897 ***** 2019-12-05T05:56:11.634 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:11.676 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.104) 0:01:49.002 ***** 2019-12-05T05:56:11.715 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:11.755 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.078) 0:01:49.081 ***** 2019-12-05T05:56:11.798 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:11.845 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.089) 0:01:49.171 ***** 2019-12-05T05:56:11.876 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:11.921 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.075) 0:01:49.247 ***** 2019-12-05T05:56:11.930 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-12-05T05:56:11.934 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d9691278f687ee76cfe8a3de60fcbd4f1f79a8c6", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 327, "state": "file", "uid": 0} 2019-12-05T05:56:11.977 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:11 +0000 (0:00:00.056) 0:01:49.303 ***** 2019-12-05T05:56:11.983 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2019-12-05T05:56:11.992 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T05:56:12.025 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:12 +0000 (0:00:00.048) 0:01:49.351 ***** 2019-12-05T05:56:12.053 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:12.092 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:12 +0000 (0:00:00.066) 0:01:49.418 ***** 2019-12-05T05:56:12.140 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2019-12-05T05:56:12.142 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:12.178 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:12 +0000 (0:00:00.086) 0:01:49.504 ***** 2019-12-05T05:56:12.215 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:12.256 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:12 +0000 (0:00:00.075) 0:01:49.580 ***** 2019-12-05T05:56:12.294 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:12.326 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:12 +0000 (0:00:00.071) 0:01:49.652 ***** 2019-12-05T05:56:12.353 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:12.389 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:12 +0000 (0:00:00.063) 0:01:49.715 ***** 2019-12-05T05:56:12.426 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2019-12-05T05:56:12.437 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:12.464 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:12 +0000 (0:00:00.074) 0:01:49.790 ***** 2019-12-05T05:56:12.489 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2019-12-05T05:56:12.491 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2019-12-05T05:56:12.526 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:12 +0000 (0:00:00.062) 0:01:49.852 ***** 2019-12-05T05:56:12.552 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2019-12-05T05:56:12.591 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:12 +0000 (0:00:00.062) 0:01:49.915 ***** 2019-12-05T05:56:12.632 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T05:56:12.663 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:12 +0000 (0:00:00.074) 0:01:49.989 ***** 2019-12-05T05:56:13.011 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2019-12-05T05:56:13.013 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T05:56:13.057 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:13 +0000 (0:00:00.387) 0:01:50.376 ***** 2019-12-05T05:56:13.119 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-12-05T05:56:13.121 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "md5sum": "6467980f3cbb5aa63a3cfe239626f53e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575525372.02-13237651153889/source", "state": "file", "uid": 0} 2019-12-05T05:56:14.058 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-12-05T05:56:14.060 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "1d6dcd385d1b7c9c2b757c62b841c887e90d7f66", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 293, "state": "file", "uid": 0} 2019-12-05T05:56:14.216 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-12-05T05:56:14.217 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "md5sum": "0848bb8607c5b0c5706d8e1930fa7d55", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575525373.13-145055059156970/source", "state": "file", "uid": 0} 2019-12-05T05:56:14.254 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:14 +0000 (0:00:01.203) 0:01:51.580 ***** 2019-12-05T05:56:14.855 INFO:teuthology.task.ansible.out: TASK [common : Temporarily comment mirrorlist so we can update yum cache] ****** 2019-12-05T05:56:14.855 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made", "warnings": ["Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually"]} 2019-12-05T05:56:15.023 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2019-12-05T05:56:15.024 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "d9691278f687ee76cfe8a3de60fcbd4f1f79a8c6", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 327, "state": "file", "uid": 0} 2019-12-05T05:56:15.072 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:15 +0000 (0:00:00.817) 0:01:52.398 ***** 2019-12-05T05:56:15.165 INFO:teuthology.task.ansible.out: TASK [common : Temporarily comment mirrorlist so we can update yum cache] ****** 2019-12-05T05:56:15.169 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} [WARNING]: Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually 2019-12-05T05:56:15.211 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:15 +0000 (0:00:00.138) 0:01:52.537 ***** 2019-12-05T05:56:16.261 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2019-12-05T05:56:16.263 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "73c387451a4fa299c9cb53e25b182ae50e8b8563", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "md5sum": "6467980f3cbb5aa63a3cfe239626f53e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 418, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575525375.11-233183058830019/source", "state": "file", "uid": 0} 2019-12-05T05:56:17.396 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": true, "checksum": "9d25e11bf71dfd21a39ace708c03408897a4f263", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "md5sum": "0848bb8607c5b0c5706d8e1930fa7d55", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:system_conf_t:s0", "size": 474, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1575525376.28-230933829443408/source", "state": "file", "uid": 0} 2019-12-05T05:56:17.436 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:17 +0000 (0:00:02.225) 0:01:54.762 ***** 2019-12-05T05:56:17.768 INFO:teuthology.task.ansible.out: TASK [common : Temporarily comment mirrorlist so we can update yum cache] ****** 2019-12-05T05:56:17.768 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made", "warnings": ["Module remote_tmp /root/.ansible/tmp did not exist and was created with a mode of 0700, this may cause issues when running as another user. To avoid this, create the remote_tmp dir with the correct permissions manually"]} 2019-12-05T05:56:18.072 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-05T05:56:18.111 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 05:56:18 +0000 (0:00:00.674) 0:01:55.437 ***** 2019-12-05T06:00:15.548 INFO:teuthology.task.ansible.out: TASK [common : Update yum cache] *********************************************** 2019-12-05T06:00:15.631 INFO:teuthology.task.ansible.out: [WARNING]: Consider using the file module with state=absent rather than running 'rm'. If you need to use command because file is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rf /var/cache/yum/*; yum clean all; yum makecache", "delta": "0:04:00.049133", "end": "2019-12-05 06:00:15.515160", "rc": 0, "start": "2019-12-05 05:56:15.466027", "stderr": "http://mirrors.cmich.edu/centos/7.7.1908/extras/x86_64/repodata/e87982a7d437db5420d7a3821de1bd4a7b145d57a64a3a464cdeed198c4c1ad5-primary.sqlite.bz2: [Errno 14] curl#56 - \"Recv failure: Connection reset by peer\"\nTrying other mirror.", "stderr_lines": ["http://mirrors.cmich.edu/centos/7.7.1908/extras/x86_64/repodata/e87982a7d437db5420d7a3821de1bd4a7b145d57a64a3a464cdeed198c4c1ad5-primary.sqlite.bz2: [Errno 14] curl#56 - \"Recv failure: Connection reset by peer\"", "Trying other mirror."], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nCleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates\nLoaded plugins: fastestmirror, langpacks, priorities\nDetermining fastest mirrors\n * base: mirror.rackspace.com\n * extras: mirror.vcu.edu\n * updates: mirror.vcu.edu\nMetadata Cache Created", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Cleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates", "Loaded plugins: fastestmirror, langpacks, priorities", "Determining fastest mirrors", " * base: mirror.rackspace.com", " * extras: mirror.vcu.edu", " * updates: mirror.vcu.edu", "Metadata Cache Created"]} Thursday 05 December 2019 06:00:15 +0000 (0:03:57.477) 0:05:52.914 ***** 2019-12-05T06:00:15.965 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made"} 2019-12-05T06:00:16.294 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-05T06:00:16.352 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:16 +0000 (0:00:00.764) 0:05:53.678 ***** 2019-12-05T06:00:16.380 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-12-05T06:00:16.382 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:16.430 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:16 +0000 (0:00:00.077) 0:05:53.756 ***** 2019-12-05T06:00:16.469 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2019-12-05T06:00:16.470 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:16.524 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:16 +0000 (0:00:00.079) 0:05:53.836 ***** 2019-12-05T06:00:16.559 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-12-05T06:00:16.561 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:16.605 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:16 +0000 (0:00:00.095) 0:05:53.931 ***** 2019-12-05T06:00:16.631 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-12-05T06:00:16.633 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:16.671 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:16 +0000 (0:00:00.065) 0:05:53.997 ***** 2019-12-05T06:00:16.823 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-12-05T06:00:16.826 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:16.989 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:16 +0000 (0:00:00.318) 0:05:54.316 ***** 2019-12-05T06:00:17.016 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2019-12-05T06:00:17.021 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:17.056 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:17 +0000 (0:00:00.064) 0:05:54.380 ***** 2019-12-05T06:00:17.084 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:17.129 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:17 +0000 (0:00:00.069) 0:05:54.450 ***** 2019-12-05T06:00:17.150 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-12-05T06:00:17.152 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:17.189 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:17 +0000 (0:00:00.065) 0:05:54.515 ***** 2019-12-05T06:00:18.533 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2019-12-05T06:00:18.534 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.986752", "end": "2019-12-05 06:00:18.500479", "rc": 0, "start": "2019-12-05 06:00:17.513727", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:00:18.579 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:18 +0000 (0:00:01.388) 0:05:55.904 ***** 2019-12-05T06:00:19.307 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-12-05T06:00:19.309 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_7.2.x86_64 providing krb5-workstation is already installed"]} 2019-12-05T06:00:19.356 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:19 +0000 (0:00:00.774) 0:05:56.678 ***** 2019-12-05T06:00:19.378 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2019-12-05T06:00:19.379 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:19.414 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:19 +0000 (0:00:00.062) 0:05:56.740 ***** 2019-12-05T06:00:19.440 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2019-12-05T06:00:19.442 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:19.476 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:19 +0000 (0:00:00.061) 0:05:56.802 ***** 2019-12-05T06:00:19.504 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:19.540 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:19 +0000 (0:00:00.064) 0:05:56.866 ***** 2019-12-05T06:00:19.579 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-12-05T06:00:19.581 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:19.615 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:19 +0000 (0:00:00.074) 0:05:56.941 ***** 2019-12-05T06:00:20.575 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-12-05T06:00:20.576 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-12-05T06:00:20.627 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:20 +0000 (0:00:01.011) 0:05:57.953 ***** 2019-12-05T06:00:20.634 INFO:teuthology.task.ansible.out: TASK [common : Update yum cache] *********************************************** 2019-12-05T06:00:20.636 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": "rm -rf /var/cache/yum/*; yum clean all; yum makecache", "delta": "0:04:02.136177", "end": "2019-12-05 06:00:20.564799", "rc": 0, "start": "2019-12-05 05:56:18.428622", "stderr": "http://mirrors.cmich.edu/centos/7.7.1908/extras/x86_64/repodata/e87982a7d437db5420d7a3821de1bd4a7b145d57a64a3a464cdeed198c4c1ad5-primary.sqlite.bz2: [Errno 14] curl#56 - \"Recv failure: Connection reset by peer\"\nTrying other mirror.", "stderr_lines": ["http://mirrors.cmich.edu/centos/7.7.1908/extras/x86_64/repodata/e87982a7d437db5420d7a3821de1bd4a7b145d57a64a3a464cdeed198c4c1ad5-primary.sqlite.bz2: [Errno 14] curl#56 - \"Recv failure: Connection reset by peer\"", "Trying other mirror."], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nCleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates\nLoaded plugins: fastestmirror, langpacks, priorities\nDetermining fastest mirrors\n * base: mirror.rackspace.com\n * extras: mirror.vcu.edu\n * updates: mirror.vcu.edu\nMetadata Cache Created", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Cleaning repos: base centos7-fcgi-ceph epel extras lab-extras updates", "Loaded plugins: fastestmirror, langpacks, priorities", "Determining fastest mirrors", " * base: mirror.rackspace.com", " * extras: mirror.vcu.edu", " * updates: mirror.vcu.edu", "Metadata Cache Created"]} 2019-12-05T06:00:20.679 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:20 +0000 (0:00:00.052) 0:05:58.005 ***** 2019-12-05T06:00:21.065 INFO:teuthology.task.ansible.out: TASK [common : Uncomment mirrorlist] ******************************************* changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-mirrorlist', u'name': u'Extra Packages for Enterprise Linux', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': u'1', u'baseurl': u'http://dl.fedoraproject.org/pub/epel/7/$basearch'}, 'key': u'epel'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/7/$basearch", "enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "msg": "1 replacements made"} 2019-12-05T06:00:21.371 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'mirrorlist': u'file:///etc/yum.repos.d/epel-testing-mirrorlist', u'name': u'Extra Packages for Enterprise Linux - Testing', u'failovermethod': u'priority', u'gpgcheck': 0, u'enabled': 0, u'baseurl': u'http://dl.fedoraproject.org/pub/epel/testing/7/$basearch'}, 'key': u'epel-testing'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "epel-testing", "value": {"baseurl": "http://dl.fedoraproject.org/pub/epel/testing/7/$basearch", "enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "msg": "1 replacements made"} 2019-12-05T06:00:21.416 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:21 +0000 (0:00:00.735) 0:05:58.741 ***** 2019-12-05T06:00:21.442 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2019-12-05T06:00:21.443 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:21.482 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:21 +0000 (0:00:00.067) 0:05:58.808 ***** 2019-12-05T06:00:21.507 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:21.544 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:21 +0000 (0:00:00.062) 0:05:58.870 ***** 2019-12-05T06:00:21.572 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2019-12-05T06:00:21.573 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:21.614 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:21 +0000 (0:00:00.070) 0:05:58.940 ***** 2019-12-05T06:00:21.642 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:21.680 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:21 +0000 (0:00:00.066) 0:05:59.006 ***** 2019-12-05T06:00:21.707 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2019-12-05T06:00:21.708 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:21.746 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:21 +0000 (0:00:00.065) 0:05:59.072 ***** 2019-12-05T06:00:21.776 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:21.834 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:21 +0000 (0:00:00.088) 0:05:59.160 ***** 2019-12-05T06:00:21.864 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2019-12-05T06:00:21.865 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:21.906 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:21 +0000 (0:00:00.069) 0:05:59.230 ***** 2019-12-05T06:00:21.931 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2019-12-05T06:00:21.933 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:21.977 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:21 +0000 (0:00:00.072) 0:05:59.303 ***** 2019-12-05T06:00:23.527 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:01.231800", "end": "2019-12-05 06:00:23.500508", "rc": 0, "start": "2019-12-05 06:00:22.268708", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:00:23.564 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:23 +0000 (0:00:01.587) 0:06:00.890 ***** 2019-12-05T06:00:24.086 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2019-12-05T06:00:24.086 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["krb5-workstation-1.15.1-37.el7_7.2.x86_64 providing krb5-workstation is already installed"]} 2019-12-05T06:00:24.134 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:24 +0000 (0:00:00.570) 0:06:01.460 ***** 2019-12-05T06:00:24.163 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:24.198 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:24 +0000 (0:00:00.063) 0:06:01.524 ***** 2019-12-05T06:00:24.226 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:24.262 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:24 +0000 (0:00:00.064) 0:06:01.588 ***** 2019-12-05T06:00:24.307 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2019-12-05T06:00:24.307 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:24.344 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:24 +0000 (0:00:00.081) 0:06:01.670 ***** 2019-12-05T06:00:24.371 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2019-12-05T06:00:24.371 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:24.409 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:24 +0000 (0:00:00.065) 0:06:01.735 ***** 2019-12-05T06:00:24.480 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2019-12-05T06:00:24.482 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2019-12-05T06:00:24.526 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:24 +0000 (0:00:00.116) 0:06:01.852 ***** 2019-12-05T06:00:25.321 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2019-12-05T06:00:25.322 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2019-12-05T06:00:25.365 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:25 +0000 (0:00:00.839) 0:06:02.692 ***** 2019-12-05T06:00:25.367 INFO:teuthology.task.ansible.out: 2019-12-05T06:00:25.455 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-12-05T06:00:25.457 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2019-12-05T06:00:26.406 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2019-12-05T06:00:27.286 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2019-12-05T06:00:27.342 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:27 +0000 (0:00:01.976) 0:06:04.668 ***** 2019-12-05T06:00:27.687 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T06:00:27.754 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:27 +0000 (0:00:00.406) 0:06:05.075 ***** 2019-12-05T06:00:28.610 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-05T06:00:28.612 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2019-12-05T06:00:29.434 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2019-12-05T06:00:30.282 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2019-12-05T06:00:30.322 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:30 +0000 (0:00:02.572) 0:06:07.648 ***** 2019-12-05T06:00:30.354 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:30.386 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:30 +0000 (0:00:00.064) 0:06:07.712 ***** 2019-12-05T06:00:30.424 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-12-05T06:00:30.426 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2019-12-05T06:00:30.469 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:30 +0000 (0:00:00.079) 0:06:07.792 ***** 2019-12-05T06:00:30.775 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2019-12-05T06:00:30.777 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing smartmontools are up to date", ""]} 2019-12-05T06:00:30.823 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:30 +0000 (0:00:00.357) 0:06:08.149 ***** 2019-12-05T06:00:30.829 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-12-05T06:00:30.831 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-12-05T06:00:30.868 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:30 +0000 (0:00:00.045) 0:06:08.194 ***** 2019-12-05T06:00:30.928 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** skipping: [smithi076.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T06:00:30.950 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T06:00:30.961 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T06:00:30.982 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T06:00:31.039 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:31 +0000 (0:00:00.170) 0:06:08.365 ***** 2019-12-05T06:00:31.814 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2019-12-05T06:00:31.816 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2019-12-05T06:00:32.662 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2019-12-05T06:00:33.609 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2019-12-05T06:00:33.678 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:33 +0000 (0:00:02.634) 0:06:10.999 ***** 2019-12-05T06:00:33.995 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2019-12-05T06:00:33.997 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T06:00:34.069 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:34 +0000 (0:00:00.387) 0:06:11.387 ***** 2019-12-05T06:00:34.399 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-12-05T06:00:34.400 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2019-12-05T06:00:34.445 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:34 +0000 (0:00:00.384) 0:06:11.771 ***** 2019-12-05T06:00:34.477 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:34.517 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:34 +0000 (0:00:00.071) 0:06:11.843 ***** 2019-12-05T06:00:35.018 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-05T06:00:35.020 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "9a447e184389aadb9aaf1e77f30f0a7f8b716c4e", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "secontext": "system_u:object_r:bin_t:s0", "size": 12725, "state": "file", "uid": 0} 2019-12-05T06:00:35.415 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-12-05T06:00:35.417 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2019-12-05T06:00:35.455 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:35 +0000 (0:00:00.938) 0:06:12.781 ***** 2019-12-05T06:00:35.892 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-05T06:00:35.894 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2019-12-05T06:00:36.383 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2019-12-05T06:00:36.385 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2019-12-05T06:00:36.423 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:36 +0000 (0:00:00.965) 0:06:13.747 ***** 2019-12-05T06:00:36.449 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:36.490 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:36 +0000 (0:00:00.068) 0:06:13.816 ***** 2019-12-05T06:00:36.816 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* ok: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T06:00:36.869 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:36 +0000 (0:00:00.378) 0:06:14.195 ***** 2019-12-05T06:00:36.877 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2019-12-05T06:00:36.879 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2019-12-05T06:00:36.922 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:36 +0000 (0:00:00.053) 0:06:14.248 ***** 2019-12-05T06:00:36.952 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2019-12-05T06:00:36.954 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:36.992 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:36 +0000 (0:00:00.069) 0:06:14.318 ***** 2019-12-05T06:00:37.031 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2019-12-05T06:00:37.033 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2019-12-05T06:00:37.071 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:37 +0000 (0:00:00.079) 0:06:14.397 ***** 2019-12-05T06:00:37.235 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2019-12-05T06:00:37.236 INFO:teuthology.task.ansible.out:fatal: [smithi076.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2019-12-05T06:00:37.281 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:37 +0000 (0:00:00.209) 0:06:14.607 ***** 2019-12-05T06:00:37.328 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-12-05T06:00:37.329 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:37.378 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:37 +0000 (0:00:00.097) 0:06:14.705 ***** 2019-12-05T06:00:37.418 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2019-12-05T06:00:37.421 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2019-12-05T06:00:37.468 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:37 +0000 (0:00:00.089) 0:06:14.794 ***** 2019-12-05T06:00:37.520 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2019-12-05T06:00:37.521 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T06:00:37.531 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T06:00:37.542 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T06:00:37.553 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2019-12-05T06:00:37.601 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:37 +0000 (0:00:00.132) 0:06:14.927 ***** 2019-12-05T06:00:38.348 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2019-12-05T06:00:38.349 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2019-12-05T06:00:38.393 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:38 +0000 (0:00:00.788) 0:06:15.716 ***** 2019-12-05T06:00:39.492 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2019-12-05 05:53:53 UTC", "ActiveEnterTimestampMonotonic": "19714400", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "local-fs.target systemd-journald.socket nss-lookup.target network.target time-sync.target var-run.mount basic.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-12-05 05:53:53 UTC", "AssertTimestampMonotonic": "19713844", "Before": "xdm.service shutdown.target multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-12-05 05:53:53 UTC", "ConditionTimestampMonotonic": "19713844", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "8970", "ExecMainStartTimestamp": "Thu 2019-12-05 05:53:53 UTC", "ExecMainStartTimestampMonotonic": "19714357", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbi 2019-12-05T06:00:39.495 INFO:teuthology.task.ansible.out:n/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_errors=no ; start_time=[Thu 2019-12-05 05:53:53 UTC] ; stop_time=[n/a] ; pid=8970 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-12-05 05:53:53 UTC", "InactiveExitTimestampMonotonic": "19714400", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "8970", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target network-online.target", 2019-12-05T06:00:39.503 INFO:teuthology.task.ansible.out: "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Thu 2019-12-05 05:53:53 UTC", "WatchdogTimestampMonotonic": "19714384", "WatchdogUSec": "0"}} 2019-12-05T06:00:39.552 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:39 +0000 (0:00:01.162) 0:06:16.878 ***** 2019-12-05T06:00:39.894 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-12-05T06:00:39.896 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.004362", "end": "2019-12-05 06:00:39.852691", "rc": 0, "start": "2019-12-05 06:00:39.848329", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-12-05T06:00:39.957 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:39 +0000 (0:00:00.405) 0:06:17.283 ***** 2019-12-05T06:00:40.460 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-12-05T06:00:40.462 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-29.el7.x86_64 providing policycoreutils-python is already installed"]} 2019-12-05T06:00:40.515 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:40 +0000 (0:00:00.557) 0:06:17.841 ***** 2019-12-05T06:00:41.062 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2019-12-05T06:00:41.062 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": [], "updated": []}, "msg": "", "rc": 0, "results": ["All packages providing nagios-common are up to date", "All packages providing nrpe are up to date", "All packages providing nagios-plugins are up to date", "All packages providing nagios-plugins-load are up to date", ""]} 2019-12-05T06:00:41.101 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:41 +0000 (0:00:00.586) 0:06:18.427 ***** 2019-12-05T06:00:41.132 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu)] ***************** 2019-12-05T06:00:41.134 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:41.171 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:41 +0000 (0:00:00.069) 0:06:18.497 ***** 2019-12-05T06:00:42.115 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2019-12-05T06:00:42.116 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2019-12-05T06:00:42.152 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:42 +0000 (0:00:00.980) 0:06:19.478 ***** 2019-12-05T06:00:43.126 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2019-12-05T06:00:43.181 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:43 +0000 (0:00:01.029) 0:06:20.507 ***** 2019-12-05T06:00:43.212 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2019-12-05T06:00:43.213 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:43.254 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:43 +0000 (0:00:00.066) 0:06:20.574 ***** 2019-12-05T06:00:43.584 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2019-12-05T06:00:43.586 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T06:00:43.627 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:43 +0000 (0:00:00.378) 0:06:20.953 ***** 2019-12-05T06:00:43.989 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi016.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2019-12-05T06:00:44.022 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:44 +0000 (0:00:00.395) 0:06:21.348 ***** 2019-12-05T06:00:44.049 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2019-12-05T06:00:44.051 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:00:44.114 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:44 +0000 (0:00:00.091) 0:06:21.440 ***** 2019-12-05T06:00:45.126 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2019-12-05T06:00:45.128 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "333141ba5e85aea94f51511a52d9b115d2e0112f", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1174, "state": "file", "uid": 0} 2019-12-05T06:00:45.166 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:45 +0000 (0:00:01.051) 0:06:22.492 ***** 2019-12-05T06:00:45.648 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2019-12-05T06:00:45.649 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Thu 2019-12-05 05:53:09 UTC", "ActiveEnterTimestampMonotonic": "19067622", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount systemd-journald.socket time-sync.target basic.target local-fs.target network.target nss-lookup.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "yes", "AssertTimestamp": "Thu 2019-12-05 05:53:09 UTC", "AssertTimestampMonotonic": "19066916", "Before": "shutdown.target multi-user.target getty@tty1.service xdm.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "yes", "ConditionTimestamp": "Thu 2019-12-05 05:53:09 UTC", "ConditionTimestampMonotonic": "19066914", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Program Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "EnvironmentFile": "/etc/sysconfig/nrpe (ignore_errors=no)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "8758", "ExecMainStartTimestamp": "Thu 2019-12-05 05:53:09 UTC", "ExecMainStartTimestampMonotonic": "19067563", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_SSL_OPT ; ignore_err 2019-12-05T06:00:45.651 INFO:teuthology.task.ansible.out:ors=no ; start_time=[Thu 2019-12-05 05:53:09 UTC] ; stop_time=[n/a] ; pid=8758 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "Group": "nrpe", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Thu 2019-12-05 05:53:09 UTC", "InactiveExitTimestampMonotonic": "19067622", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127535", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127535", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "8758", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "network-online.target basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOn 2019-12-05T06:00:45.652 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestamp": "Thu 2019-12-05 05:53:09 UTC", "WatchdogTimestampMonotonic": "19067603", "WatchdogUSec": "0"}} 2019-12-05T06:00:45.691 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:45 +0000 (0:00:00.520) 0:06:23.013 ***** 2019-12-05T06:00:46.022 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2019-12-05T06:00:46.024 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.004626", "end": "2019-12-05 06:00:45.997163", "rc": 0, "start": "2019-12-05 06:00:45.992537", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2019-12-05T06:00:46.065 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:46 +0000 (0:00:00.377) 0:06:23.391 ***** 2019-12-05T06:00:46.568 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2019-12-05T06:00:46.602 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["libsemanage-python-2.5-14.el7.x86_64 providing libsemanage-python is already installed", "policycoreutils-python-2.5-29.el7.x86_64 providing policycoreutils-python is already installed"]} 2019-12-05T06:00:46.615 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:00:46 +0000 (0:00:00.550) 0:06:23.941 ***** 2019-12-05T06:01:06.935 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2019-12-05T06:01:06.936 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["selinux-policy-targeted", "3.13.1-252.el7_7.6.noarch from updates"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.rackspace.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.vcu.edu\n * updates: mirror.vcu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Running transaction check\n---> Package selinux-policy.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n selinux-policy-targeted noarch 3.13.1-252.el7_7.6 updates 7.0 M\nUpdating for dependencies:\n selinux-policy noarch 3.13.1-252.el7_7.6 updates 492 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package (+1 Dependent package)\n\nTotal download size: 7.5 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n--------------------------------------------------------------------------------\nTotal 2019-12-05T06:01:06.937 INFO:teuthology.task.ansible.out: 17 MB/s | 7.5 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Updating : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Cleanup : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Cleanup : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n Verifying : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Verifying : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n\nUpdated:\n selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 \n\nDependency Updated:\n selinux-policy.noarch 0:3.13.1-252.el7_7.6 \n\nComplete!\n"]} 2019-12-05T06:01:06.992 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:06 +0000 (0:00:20.376) 0:06:44.318 ***** 2019-12-05T06:01:07.687 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-12-05T06:01:07.689 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-12-05T06:01:07.740 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:07 +0000 (0:00:00.748) 0:06:45.066 ***** 2019-12-05T06:01:14.513 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "changes": {"installed": [], "updated": [["selinux-policy-targeted", "3.13.1-252.el7_7.6.noarch from updates"]]}, "msg": "", "rc": 0, "results": ["Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.rackspace.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.vcu.edu\n * updates: mirror.vcu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Processing Dependency: selinux-policy = 3.13.1-252.el7_7.6 for package: selinux-policy-targeted-3.13.1-252.el7_7.6.noarch\n--> Running transaction check\n---> Package selinux-policy.noarch 0:3.13.1-252.el7.1 will be updated\n---> Package selinux-policy.noarch 0:3.13.1-252.el7_7.6 will be an update\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nUpdating:\n selinux-policy-targeted noarch 3.13.1-252.el7_7.6 updates 7.0 M\nUpdating for dependencies:\n selinux-policy noarch 3.13.1-252.el7_7.6 updates 492 k\n\nTransaction Summary\n================================================================================\nUpgrade 1 Package (+1 Dependent package)\n\nTotal download size: 7.5 M\nDownloading packages:\nDelta RPMs disabled because /usr/bin/applydeltarpm not installed.\n------------------------------------------------- 2019-12-05T06:01:14.517 INFO:teuthology.task.ansible.out:-------------------------------\nTotal 15 MB/s | 7.5 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Updating : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Updating : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Cleanup : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Cleanup : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n Verifying : selinux-policy-3.13.1-252.el7_7.6.noarch 1/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7_7.6.noarch 2/4 \n Verifying : selinux-policy-targeted-3.13.1-252.el7.1.noarch 3/4 \n Verifying : selinux-policy-3.13.1-252.el7.1.noarch 4/4 \n\nUpdated:\n selinux-policy-targeted.noarch 0:3.13.1-252.el7_7.6 \n\nDependency Updated:\n selinux-policy.noarch 0:3.13.1-252.el7_7.6 \n\nComplete!\n"]} 2019-12-05T06:01:14.552 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:14 +0000 (0:00:06.809) 0:06:51.876 ***** 2019-12-05T06:01:15.239 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2019-12-05T06:01:15.240 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2019-12-05T06:01:15.283 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:15 +0000 (0:00:00.732) 0:06:52.608 ***** 2019-12-05T06:01:15.854 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-12-05T06:01:15.856 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:07.737848", "end": "2019-12-05 06:01:15.806923", "failed_when_result": false, "rc": 0, "start": "2019-12-05 06:01:08.069075", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2019-12-05T06:01:15.906 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:15 +0000 (0:00:00.623) 0:06:53.232 ***** 2019-12-05T06:01:16.844 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2019-12-05T06:01:16.881 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:16 +0000 (0:00:00.973) 0:06:54.205 ***** 2019-12-05T06:01:17.227 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-12-05T06:01:17.227 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.017011", "end": "2019-12-05 06:01:17.196124", "rc": 0, "start": "2019-12-05 06:01:17.179113", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-12-05T06:01:17.275 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:17 +0000 (0:00:00.396) 0:06:54.601 ***** 2019-12-05T06:01:17.634 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.011201", "end": "2019-12-05 06:01:17.603812", "rc": 0, "start": "2019-12-05 06:01:17.592611", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:01:17.693 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:17 +0000 (0:00:00.417) 0:06:55.019 ***** 2019-12-05T06:01:23.973 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2019-12-05T06:01:23.974 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:08.355236", "end": "2019-12-05 06:01:23.942123", "failed_when_result": false, "rc": 0, "start": "2019-12-05 06:01:15.586887", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2019-12-05T06:01:24.036 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:24 +0000 (0:00:06.342) 0:07:01.362 ***** 2019-12-05T06:01:24.986 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2019-12-05T06:01:24.989 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2019-12-05T06:01:25.051 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:25 +0000 (0:00:01.015) 0:07:02.377 ***** 2019-12-05T06:01:25.381 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2019-12-05T06:01:25.382 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.017929", "end": "2019-12-05 06:01:25.361017", "rc": 0, "start": "2019-12-05 06:01:25.343088", "stderr": "", "stderr_lines": [], "stdout": "checkmodule: loading policy configuration from /tmp/nrpe.te\ncheckmodule: policy configuration loaded\ncheckmodule: writing binary representation (version 19) to /tmp/nrpe.mod", "stdout_lines": ["checkmodule: loading policy configuration from /tmp/nrpe.te", "checkmodule: policy configuration loaded", "checkmodule: writing binary representation (version 19) to /tmp/nrpe.mod"]} 2019-12-05T06:01:25.427 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:25 +0000 (0:00:00.371) 0:07:02.749 ***** 2019-12-05T06:01:25.757 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2019-12-05T06:01:25.759 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.012845", "end": "2019-12-05 06:01:25.732528", "rc": 0, "start": "2019-12-05 06:01:25.719683", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:01:25.803 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:25 +0000 (0:00:00.379) 0:07:03.129 ***** 2019-12-05T06:01:26.066 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2019-12-05T06:01:26.067 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:08.055041", "end": "2019-12-05 06:01:26.038614", "rc": 0, "start": "2019-12-05 06:01:17.983573", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:01:26.107 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:26 +0000 (0:00:00.303) 0:07:03.433 ***** 2019-12-05T06:01:26.432 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2019-12-05T06:01:26.434 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-12-05T06:01:26.475 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:26 +0000 (0:00:00.364) 0:07:03.797 ***** 2019-12-05T06:01:26.498 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2019-12-05T06:01:26.499 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2019-12-05T06:01:26.537 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:26 +0000 (0:00:00.065) 0:07:03.863 ***** 2019-12-05T06:01:26.575 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-12-05T06:01:26.577 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-05T06:01:26.613 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:26 +0000 (0:00:00.076) 0:07:03.939 ***** 2019-12-05T06:01:26.650 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-12-05T06:01:26.651 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "podman", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2019-12-05T06:01:26.689 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:26 +0000 (0:00:00.076) 0:07:04.016 ***** 2019-12-05T06:01:26.748 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi076.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-05T06:01:26.794 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:26 +0000 (0:00:00.104) 0:07:04.120 ***** 2019-12-05T06:01:27.144 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-05T06:01:27.184 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:27 +0000 (0:00:00.390) 0:07:04.510 ***** 2019-12-05T06:01:27.507 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2019-12-05T06:01:27.557 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:27 +0000 (0:00:00.372) 0:07:04.883 ***** 2019-12-05T06:01:27.588 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2019-12-05T06:01:27.610 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:01:27.623 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:27 +0000 (0:00:00.066) 0:07:04.949 ***** 2019-12-05T06:01:27.649 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-12-05T06:01:27.651 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:01:27.693 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:27 +0000 (0:00:00.067) 0:07:05.017 ***** 2019-12-05T06:01:28.044 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-12-05T06:01:28.046 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-12-05T06:01:28.084 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:28 +0000 (0:00:00.391) 0:07:05.409 ***** 2019-12-05T06:01:29.175 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-12-05T06:01:29.222 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:29 +0000 (0:00:01.139) 0:07:06.548 ***** 2019-12-05T06:01:29.259 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2019-12-05T06:01:29.261 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi076"}, "changed": false} 2019-12-05T06:01:29.313 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:29 +0000 (0:00:00.091) 0:07:06.640 ***** 2019-12-05T06:01:30.308 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ changed: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi076.front.sepia.ceph.com", "ansible_hostname": "smithi076", "ansible_nodename": "smithi076"}, "changed": true, "name": "smithi076"} 2019-12-05T06:01:30.355 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:30 +0000 (0:00:01.041) 0:07:07.681 ***** 2019-12-05T06:01:31.298 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "28e3f1fe12efa510f1199f3ab5f2a5a0feb586ff", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1042, "state": "file", "uid": 0} 2019-12-05T06:01:31.356 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:31 +0000 (0:00:00.998) 0:07:08.680 ***** 2019-12-05T06:01:32.321 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-12-05T06:01:32.371 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:32 +0000 (0:00:01.016) 0:07:09.697 ***** 2019-12-05T06:01:33.332 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** changed: [smithi076.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-12-05T06:01:33.370 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:33 +0000 (0:00:00.999) 0:07:10.697 ***** 2019-12-05T06:01:34.317 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-12-05T06:01:34.320 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-12-05T06:01:34.363 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:34 +0000 (0:00:00.992) 0:07:11.689 ***** 2019-12-05T06:01:34.548 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:08.423151", "end": "2019-12-05 06:01:34.514474", "rc": 0, "start": "2019-12-05 06:01:26.091323", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:01:34.590 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:34 +0000 (0:00:00.227) 0:07:11.916 ***** 2019-12-05T06:01:34.734 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-05T06:01:34.790 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:34 +0000 (0:00:00.199) 0:07:12.116 ***** 2019-12-05T06:01:34.976 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2019-12-05T06:01:35.027 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:35 +0000 (0:00:00.237) 0:07:12.353 ***** 2019-12-05T06:01:35.069 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi", "iozone"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/yum_systems.yml"], "changed": false} 2019-12-05T06:01:35.132 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:35 +0000 (0:00:00.104) 0:07:12.458 ***** 2019-12-05T06:01:35.161 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 10103, "name": "kvm", "state": "present", "system": false} 2019-12-05T06:01:35.208 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:35 +0000 (0:00:00.076) 0:07:12.534 ***** 2019-12-05T06:01:35.210 INFO:teuthology.task.ansible.out: 2019-12-05T06:01:35.216 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2019-12-05T06:01:35.218 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-05T06:01:35.275 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:35 +0000 (0:00:00.066) 0:07:12.601 ***** 2019-12-05T06:01:35.324 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2019-12-05T06:01:35.326 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml) => {"ansible_facts": {"epel_packages": ["cryptopp-devel", "cryptopp", "fcgi", "dbench", "fuse-sshfs", "bonnie++", "perl-JSON-XS"], "packages": ["@core", "@base", "yum-plugin-priorities", "yum-plugin-fastestmirror", "redhat-lsb", "sysstat", "gdb", "git-all", "python-configobj", "gcc-c++", "libedit", "openssl098e", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse", "fuse-libs", "python-virtualenv", "openssl", "libuuid", "btrfs-progs", "attr", "valgrind", "python-nose", "mpich", "podman", "ant", "iozone", "libtool", "automake", "gettext", "libuuid-devel", "libacl-devel", "bc", "xfsdump", "xfsprogs-devel", "blktrace", "numpy", "python-matplotlib", "qemu-kvm", "usbredir", "genisoimage", "httpd", "httpd-devel", "httpd-tools", "mod_ssl", "mod_fastcgi-2.4.7-1.ceph.el7.centos", "libev-devel", "perl-XML-Twig", "java-1.6.0-openjdk-devel", "junit4", "nfs-utils", "ncurses-devel", "python-devel", "python-virtualenv", "perl-CPAN"], "yum_repos": {"centos7-fcgi-ceph": {"baseurl": "http://{{ gitbuilder_host }}/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}, "lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_7.yml"} 2019-12-05T06:01:35.379 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:35 +0000 (0:00:00.103) 0:07:12.705 ***** 2019-12-05T06:01:35.427 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ ok: [smithi016.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2019-12-05T06:01:35.464 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:35 +0000 (0:00:00.083) 0:07:12.789 ***** 2019-12-05T06:01:35.564 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-05T06:01:35.566 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-12-05T06:01:35.603 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:35 +0000 (0:00:00.140) 0:07:12.929 ***** 2019-12-05T06:01:35.802 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2019-12-05T06:01:35.804 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2019-12-05T06:01:35.840 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:35 +0000 (0:00:00.237) 0:07:13.166 ***** 2019-12-05T06:01:36.155 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2019-12-05T06:01:36.157 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "name": "ubuntu", "state": "present", "system": false} 2019-12-05T06:01:36.196 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:36 +0000 (0:00:00.356) 0:07:13.522 ***** 2019-12-05T06:01:36.240 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:01:36.293 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:36 +0000 (0:00:00.096) 0:07:13.619 ***** 2019-12-05T06:01:36.324 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2019-12-05T06:01:36.325 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:01:36.363 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:36 +0000 (0:00:00.070) 0:07:13.689 ***** 2019-12-05T06:01:36.571 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-12-05T06:01:36.572 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-12-05T06:01:36.609 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:36 +0000 (0:00:00.245) 0:07:13.935 ***** 2019-12-05T06:01:36.742 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2019-12-05T06:01:36.742 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2019-12-05T06:01:36.786 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:36 +0000 (0:00:00.177) 0:07:14.112 ***** 2019-12-05T06:01:37.622 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-12-05T06:01:37.668 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:37 +0000 (0:00:00.878) 0:07:14.990 ***** 2019-12-05T06:01:37.901 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2019-12-05T06:01:37.940 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:37 +0000 (0:00:00.275) 0:07:15.266 ***** 2019-12-05T06:01:37.976 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi016"}, "changed": false} 2019-12-05T06:01:38.048 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:38 +0000 (0:00:00.106) 0:07:15.373 ***** 2019-12-05T06:01:38.609 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2019-12-05T06:01:38.611 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi016.front.sepia.ceph.com", "ansible_hostname": "smithi016", "ansible_nodename": "smithi016"}, "changed": true, "name": "smithi016"} 2019-12-05T06:01:38.658 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:38 +0000 (0:00:00.611) 0:07:15.984 ***** 2019-12-05T06:01:39.611 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2019-12-05T06:01:39.611 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "28e3f1fe12efa510f1199f3ab5f2a5a0feb586ff", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1042, "state": "file", "uid": 0} 2019-12-05T06:01:39.665 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:39 +0000 (0:00:01.006) 0:07:16.991 ***** 2019-12-05T06:01:40.241 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ [WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.236579", "end": "2019-12-05 06:01:40.205261", "rc": 0, "start": "2019-12-05 06:01:37.968682", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:01:40.313 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:40 +0000 (0:00:00.647) 0:07:17.639 ***** 2019-12-05T06:01:40.342 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-12-05T06:01:40.381 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:40 +0000 (0:00:00.068) 0:07:17.707 ***** 2019-12-05T06:01:40.667 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2019-12-05T06:01:40.668 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2019-12-05T06:01:40.707 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:40 +0000 (0:00:00.325) 0:07:18.033 ***** 2019-12-05T06:01:41.341 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-12-05T06:01:41.342 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'CentOS 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/'}, 'key': u'centos7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "58f6bfe213f63251ef8c4f03709a8fcef9b9e7bf", "dest": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "centos7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 250, "state": "file", "uid": 0} 2019-12-05T06:01:41.636 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2019-12-05T06:01:41.638 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2019-12-05T06:01:41.678 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:41 +0000 (0:00:00.971) 0:07:19.004 ***** 2019-12-05T06:01:42.222 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-12-05T06:01:42.222 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "765bacbb40da31f952344b310a4be4322cbee8a4", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 209, "state": "file", "uid": 0} 2019-12-05T06:01:42.268 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:42 +0000 (0:00:00.590) 0:07:19.594 ***** 2019-12-05T06:01:42.297 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:01:42.344 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:42 +0000 (0:00:00.075) 0:07:19.670 ***** 2019-12-05T06:01:42.403 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-12-05T06:01:42.404 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-05T06:01:42.446 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:42 +0000 (0:00:00.101) 0:07:19.772 ***** 2019-12-05T06:01:42.484 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-12-05T06:01:42.485 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:01:42.524 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:42 +0000 (0:00:00.078) 0:07:19.850 ***** 2019-12-05T06:01:42.571 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T06:01:42.573 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2019-12-05T06:01:42.611 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:42 +0000 (0:00:00.087) 0:07:19.938 ***** 2019-12-05T06:01:42.693 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2019-12-05T06:01:42.694 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2019-12-05T06:01:42.734 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:42 +0000 (0:00:00.122) 0:07:20.060 ***** 2019-12-05T06:01:43.067 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-05T06:01:43.067 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-05T06:01:43.108 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:43 +0000 (0:00:00.374) 0:07:20.435 ***** 2019-12-05T06:01:43.123 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-12-05T06:01:43.124 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-50.el7.noarch providing yum-utils is already installed"]} 2019-12-05T06:01:43.166 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:43 +0000 (0:00:00.057) 0:07:20.492 ***** 2019-12-05T06:01:43.466 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-05T06:01:43.468 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 10103, "name": "kvm", "state": "present", "system": false} 2019-12-05T06:01:43.518 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:43 +0000 (0:00:00.351) 0:07:20.844 ***** 2019-12-05T06:01:43.824 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-12-05T06:01:43.825 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.324586", "end": "2019-12-05 06:01:43.787592", "rc": 0, "start": "2019-12-05 06:01:43.463006", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.rackspace.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.vcu.edu\n * updates: mirror.vcu.edu\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Loading mirror speeds from cached hostfile", " * base: mirror.rackspace.com", " * epel: download-cc-rdu01.fedoraproject.org", " * extras: mirror.vcu.edu", " * updates: mirror.vcu.edu", "No unfinished transactions left."]} 2019-12-05T06:01:43.886 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:43 +0000 (0:00:00.367) 0:07:21.211 ***** 2019-12-05T06:01:43.904 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-05T06:01:43.905 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1215, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2019-12-05T06:01:43.944 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:43 +0000 (0:00:00.057) 0:07:21.269 ***** 2019-12-05T06:01:44.245 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-12-05T06:01:44.246 INFO:teuthology.task.ansible.out:fatal: [smithi076.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.036314", "end": "2019-12-05 06:01:44.219453", "msg": "non-zero return code", "rc": 1, "start": "2019-12-05 06:01:44.183139", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2019-12-05T06:01:44.286 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:44 +0000 (0:00:00.342) 0:07:21.612 ***** 2019-12-05T06:01:44.324 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-12-05T06:01:44.326 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:01:44.364 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:44 +0000 (0:00:00.078) 0:07:21.691 ***** 2019-12-05T06:01:44.896 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-12-05T06:01:44.897 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2019-12-05T06:01:44.937 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:44 +0000 (0:00:00.572) 0:07:22.263 ***** 2019-12-05T06:01:45.883 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-12-05T06:01:45.884 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2019-12-05T06:01:45.923 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:45 +0000 (0:00:00.986) 0:07:23.249 ***** 2019-12-05T06:01:48.383 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.136221", "end": "2019-12-05 06:01:48.348802", "rc": 0, "start": "2019-12-05 06:01:46.212581", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:01:48.419 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:48 +0000 (0:00:02.496) 0:07:25.746 ***** 2019-12-05T06:01:48.456 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2019-12-05T06:01:48.493 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:48 +0000 (0:00:00.073) 0:07:25.819 ***** 2019-12-05T06:01:49.507 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2019-12-05T06:01:49.508 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'CentOS 7 Local fastcgi Repo', u'baseurl': u'http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/'}, 'key': u'centos7-fcgi-ceph'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "58f6bfe213f63251ef8c4f03709a8fcef9b9e7bf", "dest": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "gid": 0, "group": "root", "item": {"key": "centos7-fcgi-ceph", "value": {"baseurl": "http://gitbuilder.ceph.com/mod_fastcgi-rpm-centos7-x86_64-basic/ref/master/", "enabled": 1, "gpgcheck": 0, "name": "CentOS 7 Local fastcgi Repo"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/centos7-fcgi-ceph.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 250, "state": "file", "uid": 0} 2019-12-05T06:01:50.441 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={'value': {u'gpgcheck': 0, u'enabled': 1, u'name': u'lab-extras', u'baseurl': u'http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/'}, 'key': u'lab-extras'}) => {"ansible_loop_var": "item", "changed": false, "checksum": "765bacbb40da31f952344b310a4be4322cbee8a4", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/centos7/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 209, "state": "file", "uid": 0} 2019-12-05T06:01:50.510 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:50 +0000 (0:00:02.016) 0:07:27.836 ***** 2019-12-05T06:01:50.536 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2019-12-05T06:01:50.538 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:01:50.573 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:50 +0000 (0:00:00.063) 0:07:27.899 ***** 2019-12-05T06:01:50.611 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2019-12-05T06:01:50.612 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi016.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2019-12-05T06:01:50.654 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:50 +0000 (0:00:00.077) 0:07:27.977 ***** 2019-12-05T06:01:50.686 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2019-12-05T06:01:50.688 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:01:50.731 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:50 +0000 (0:00:00.079) 0:07:28.057 ***** 2019-12-05T06:01:50.769 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T06:01:50.770 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2019-12-05T06:01:50.820 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:50 +0000 (0:00:00.079) 0:07:28.137 ***** 2019-12-05T06:01:51.299 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2019-12-05T06:01:51.302 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["yum-utils-1.1.31-50.el7.noarch providing yum-utils is already installed"]} 2019-12-05T06:01:51.336 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:51 +0000 (0:00:00.525) 0:07:28.662 ***** 2019-12-05T06:01:52.043 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2019-12-05T06:01:52.045 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "cmd": ["yum-complete-transaction", "--cleanup-only"], "delta": "0:00:00.342044", "end": "2019-12-05 06:01:51.995440", "rc": 0, "start": "2019-12-05 06:01:51.653396", "stderr": "", "stderr_lines": [], "stdout": "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.rackspace.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.vcu.edu\n * updates: mirror.vcu.edu\nNo unfinished transactions left.", "stdout_lines": ["Loaded plugins: fastestmirror, langpacks, priorities", "Loading mirror speeds from cached hostfile", " * base: mirror.rackspace.com", " * epel: download-cc-rdu01.fedoraproject.org", " * extras: mirror.vcu.edu", " * updates: mirror.vcu.edu", "No unfinished transactions left."]} 2019-12-05T06:01:52.088 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:52 +0000 (0:00:00.751) 0:07:29.414 ***** 2019-12-05T06:01:52.461 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2019-12-05T06:01:52.462 INFO:teuthology.task.ansible.out:fatal: [smithi016.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.037681", "end": "2019-12-05 06:01:52.437021", "msg": "non-zero return code", "rc": 1, "start": "2019-12-05 06:01:52.399340", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2019-12-05T06:01:52.517 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:52 +0000 (0:00:00.429) 0:07:29.844 ***** 2019-12-05T06:01:52.554 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2019-12-05T06:01:52.556 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:01:52.592 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:52 +0000 (0:00:00.074) 0:07:29.918 ***** 2019-12-05T06:01:53.177 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-05T06:01:53.730 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi", "iozone"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n librados2 x86_64 1:10.2.5-4.el7 @base 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 2019-12-05T06:01:53.731 INFO:teuthology.task.ansible.out: @base 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-167.el7_7.1 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 @updates 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 4 Packages (+3 Dependent packages)\n\nInstalled size: 23 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 1/7 \n Erasing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 2/7 \n Erasing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 3/7 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/7 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 6/7 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 7/7 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/7 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/7 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/7 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/7 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 7/7 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7.centos 2019-12-05T06:01:53.733 INFO:teuthology.task.ansible.out: \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} Thursday 05 December 2019 06:01:53 +0000 (0:00:00.627) 0:07:30.546 ***** 2019-12-05T06:01:56.535 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "leveldb is not installed", "xmlstarlet is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @base 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2019-12-05T06:01:56.586 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:01:56 +0000 (0:00:03.365) 0:07:33.912 ***** 2019-12-05T06:02:00.925 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-05T06:02:00.926 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["librbd1", "librados2", "mod_fastcgi", "iozone"]}, "msg": "", "rc": 0, "results": ["ceph is not installed", "ceph-base is not installed", "ceph-selinux is not installed", "ceph-common is not installed", "ceph-debuginfo is not installed", "ceph-release is not installed", "libcephfs1 is not installed", "ceph-radosgw is not installed", "python-ceph is not installed", "python-rados is not installed", "python-rbd is not installed", "python-cephfs is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be erased\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be erased\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-img-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be erased\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be erased\n--> Running transaction check\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be erased\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n iozone x86_64 3.424-2_ceph.el7.centos @lab-extras 1.3 M\n librados2 x86_64 1:10.2.5-4.el7 @base 5.5 M\n librbd1 x86_64 1:10.2.5-4.el7 2019-12-05T06:02:00.928 INFO:teuthology.task.ansible.out: @base 7.4 M\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos @centos7-fcgi-ceph 213 k\nRemoving for dependencies:\n qemu-img x86_64 10:1.5.3-167.el7_7.1 @updates 2.3 M\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 @updates 5.7 M\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 @updates 755 k\n\nTransaction Summary\n================================================================================\nRemove 4 Packages (+3 Dependent packages)\n\nInstalled size: 23 M\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 1/7 \n Erasing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 2/7 \n Erasing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 3/7 \n Erasing : 1:librbd1-10.2.5-4.el7.x86_64 4/7 \n Erasing : 1:librados2-10.2.5-4.el7.x86_64 5/7 \n Erasing : iozone-3.424-2_ceph.el7.centos.x86_64 6/7 \n Erasing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 7/7 \nwarning: file /var/run/mod_fastcgi: remove failed: No such file or directory\n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/7 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/7 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/7 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/7 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/7 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/7 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 7/7 \n\nRemoved:\n iozone.x86_64 0:3.424-2_ceph.el7.centos 2019-12-05T06:02:00.929 INFO:teuthology.task.ansible.out: \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n\nDependency Removed:\n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-05T06:02:00.974 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:00 +0000 (0:00:04.387) 0:07:38.300 ***** 2019-12-05T06:02:04.412 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-12-05T06:02:04.413 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "changes": {"removed": ["boost-random", "boost-random"]}, "msg": "", "rc": 0, "results": ["boost-program-options is not installed", "leveldb is not installed", "xmlstarlet is not installed", "python-jinja2 is not installed", "python-ceph is not installed", "python-flask is not installed", "python-requests is not installed", "python-urllib3 is not installed", "python-babel is not installed", "hdparm is not installed", "python-markupsafe is not installed", "python-werkzeug is not installed", "python-itsdangerous is not installed", "Loaded plugins: fastestmirror, langpacks, priorities\nResolving Dependencies\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be erased\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nRemoving:\n boost-random x86_64 1.53.0-27.el7 @base 28 k\n\nTransaction Summary\n================================================================================\nRemove 1 Package\n\nInstalled size: 28 k\nDownloading packages:\nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Erasing : boost-random-1.53.0-27.el7.x86_64 1/1 \n Verifying : boost-random-1.53.0-27.el7.x86_64 1/1 \n\nRemoved:\n boost-random.x86_64 0:1.53.0-27.el7 \n\nComplete!\n"]} 2019-12-05T06:02:04.449 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:04 +0000 (0:00:03.475) 0:07:41.775 ***** 2019-12-05T06:02:14.978 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": ["@core", "@base", "iozone", "qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7.centos"]}, "msg": "Warning: Group core does not have any packages to install.\nWarning: Group base does not have any packages to install.\nGroup base does have 1 conditional packages, which may get installed.\n", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-50.el7.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-17.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-114.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-20.el7.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7.centos.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed", "gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "parted-3.1-29.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "1:openssl-1.0.2k-16.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-61.el7.x86_64 providing libuuid i 2019-12-05T06:02:14.979 INFO:teuthology.task.ansible.out:s already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is already installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed", "python-nose-1.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "podman-1.4.4-4.el7.centos.x86_64 providing podman is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-61.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already installed", "blktrace-1.0.5-8.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-90.el7.centos.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-90.el7.centos.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-90.el7.centos.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-90.el7.centos.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel 2019-12-05T06:02:14.981 INFO:teuthology.task.ansible.out:-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already installed", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.65.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed", "python-devel-2.7.5-86.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.rackspace.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.vcu.edu\n * updates: mirror.vcu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be installed\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 wil 2019-12-05T06:02:14.982 INFO:teuthology.task.ansible.out:l be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n================================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7.centos lab-extras 800 k\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos centos7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 updates 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 base 39 k\n librados2 x86_64 1:10.2.5-4.el7 base 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.1 updates 699 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 updates 435 k\n\nTransaction Summary\n================================================================================\nInstall 3 Packages (+5 Dependent packages)\n\nTotal download size: 8.1 M\nInstalled size: 23 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 15 MB/s | 8.1 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/8 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/8 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/8 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 4/8 \n Installing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/8 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 6/8 \n 2019-12-05T06:02:14.984 INFO:teuthology.task.ansible.out:Installing : iozone-3.424-2_ceph.el7.centos.x86_64 7/8 \n Installing : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 8/8 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/8 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/8 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/8 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/8 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/8 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/8 \n Verifying : boost-random-1.53.0-27.el7.x86_64 7/8 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 8/8 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-05T06:02:15.025 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:15 +0000 (0:00:10.575) 0:07:52.351 ***** 2019-12-05T06:02:15.844 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-12-05T06:02:15.846 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "bonnie++-1.97.3-1.el7.x86_64 providing bonnie++ is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} 2019-12-05T06:02:15.885 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:15 +0000 (0:00:00.860) 0:07:53.211 ***** 2019-12-05T06:02:16.418 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-12-05T06:02:16.420 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2019-12-05T06:02:16.466 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:16 +0000 (0:00:00.580) 0:07:53.792 ***** 2019-12-05T06:02:16.503 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-05T06:02:16.505 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:16.544 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:16 +0000 (0:00:00.078) 0:07:53.870 ***** 2019-12-05T06:02:16.987 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target polkit.service system.slice dbus.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "ip6tables.service shutdown.target iptables.service ipset.service ebtables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": " 2019-12-05T06:02:16.989 INFO:teuthology.task.ansible.out:yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPr 2019-12-05T06:02:16.990 INFO:teuthology.task.ansible.out:iority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "system.slice network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-05T06:02:17.025 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:17 +0000 (0:00:00.480) 0:07:54.351 ***** 2019-12-05T06:02:17.441 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi076.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-12-05T06:02:17.495 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:17 +0000 (0:00:00.468) 0:07:54.819 ***** 2019-12-05T06:02:18.332 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-12-05T06:02:18.334 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-12-05T06:02:18.374 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:18 +0000 (0:00:00.881) 0:07:55.701 ***** 2019-12-05T06:02:18.401 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-12-05T06:02:18.402 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:18.442 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:18 +0000 (0:00:00.067) 0:07:55.768 ***** 2019-12-05T06:02:18.501 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-12-05T06:02:18.542 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:18 +0000 (0:00:00.099) 0:07:55.868 ***** 2019-12-05T06:02:18.568 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-12-05T06:02:18.570 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:18.609 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:18 +0000 (0:00:00.067) 0:07:55.935 ***** 2019-12-05T06:02:18.637 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-12-05T06:02:18.639 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:18.679 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:18 +0000 (0:00:00.069) 0:07:56.005 ***** 2019-12-05T06:02:18.717 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-12-05T06:02:18.719 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:18.765 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:18 +0000 (0:00:00.085) 0:07:56.091 ***** 2019-12-05T06:02:18.792 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-12-05T06:02:18.794 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:18.840 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:18 +0000 (0:00:00.075) 0:07:56.166 ***** 2019-12-05T06:02:18.867 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-12-05T06:02:18.910 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:18 +0000 (0:00:00.070) 0:07:56.236 ***** 2019-12-05T06:02:18.947 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-12-05T06:02:18.948 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:18.985 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:18 +0000 (0:00:00.075) 0:07:56.311 ***** 2019-12-05T06:02:19.011 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-12-05T06:02:19.013 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.050 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.064) 0:07:56.376 ***** 2019-12-05T06:02:19.076 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.115 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.065) 0:07:56.441 ***** 2019-12-05T06:02:19.142 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-05T06:02:19.144 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.179 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.063) 0:07:56.505 ***** 2019-12-05T06:02:19.206 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2019-12-05T06:02:19.208 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.244 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.065) 0:07:56.570 ***** 2019-12-05T06:02:19.270 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-05T06:02:19.272 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.307 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.063) 0:07:56.633 ***** 2019-12-05T06:02:19.334 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2019-12-05T06:02:19.336 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.373 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.066) 0:07:56.700 ***** 2019-12-05T06:02:19.400 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-12-05T06:02:19.402 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.440 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.066) 0:07:56.766 ***** 2019-12-05T06:02:19.466 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-12-05T06:02:19.467 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.504 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.064) 0:07:56.830 ***** 2019-12-05T06:02:19.554 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-12-05T06:02:19.555 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.597 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.092) 0:07:56.923 ***** 2019-12-05T06:02:19.635 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2019-12-05T06:02:19.637 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.676 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.079) 0:07:57.003 ***** 2019-12-05T06:02:19.715 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-12-05T06:02:19.717 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.760 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.083) 0:07:57.086 ***** 2019-12-05T06:02:19.789 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.834 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.072) 0:07:57.159 ***** 2019-12-05T06:02:19.871 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-12-05T06:02:19.873 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.918 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.084) 0:07:57.244 ***** 2019-12-05T06:02:19.945 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-05T06:02:19.946 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:19.995 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:19 +0000 (0:00:00.077) 0:07:57.322 ***** 2019-12-05T06:02:20.029 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2019-12-05T06:02:20.030 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.070 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.074) 0:07:57.396 ***** 2019-12-05T06:02:20.096 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2019-12-05T06:02:20.098 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.134 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.064) 0:07:57.460 ***** 2019-12-05T06:02:20.171 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2019-12-05T06:02:20.173 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.208 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.073) 0:07:57.534 ***** 2019-12-05T06:02:20.234 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2019-12-05T06:02:20.236 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.272 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.064) 0:07:57.598 ***** 2019-12-05T06:02:20.298 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2019-12-05T06:02:20.300 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.335 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.063) 0:07:57.662 ***** 2019-12-05T06:02:20.363 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2019-12-05T06:02:20.365 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.401 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.065) 0:07:57.727 ***** 2019-12-05T06:02:20.427 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2019-12-05T06:02:20.429 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.466 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.064) 0:07:57.792 ***** 2019-12-05T06:02:20.492 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2019-12-05T06:02:20.494 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.530 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.063) 0:07:57.856 ***** 2019-12-05T06:02:20.556 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-05T06:02:20.557 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.593 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.063) 0:07:57.919 ***** 2019-12-05T06:02:20.626 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-05T06:02:20.628 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.665 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.072) 0:07:57.991 ***** 2019-12-05T06:02:20.691 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-05T06:02:20.693 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:20.728 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:20 +0000 (0:00:00.063) 0:07:58.054 ***** 2019-12-05T06:02:21.061 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2019-12-05T06:02:21.096 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:21 +0000 (0:00:00.367) 0:07:58.422 ***** 2019-12-05T06:02:21.452 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-05T06:02:21.454 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T06:02:21.492 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:21 +0000 (0:00:00.396) 0:07:58.818 ***** 2019-12-05T06:02:21.817 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-05T06:02:21.820 INFO:teuthology.task.ansible.out: [WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2019-12-05T06:02:21.855 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:21 +0000 (0:00:00.362) 0:07:59.181 ***** 2019-12-05T06:02:21.881 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-05T06:02:21.883 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:21.918 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:21 +0000 (0:00:00.062) 0:07:59.244 ***** 2019-12-05T06:02:21.944 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-05T06:02:21.946 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:21.982 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:21 +0000 (0:00:00.064) 0:07:59.309 ***** 2019-12-05T06:02:21.987 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:22.012 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-05T06:02:22.015 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.049 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.066) 0:07:59.375 ***** 2019-12-05T06:02:22.076 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-05T06:02:22.077 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.113 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.063) 0:07:59.439 ***** 2019-12-05T06:02:22.144 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.178 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.064) 0:07:59.504 ***** 2019-12-05T06:02:22.215 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-05T06:02:22.216 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.254 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.075) 0:07:59.580 ***** 2019-12-05T06:02:22.291 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-05T06:02:22.297 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.334 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.080) 0:07:59.660 ***** 2019-12-05T06:02:22.360 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-05T06:02:22.362 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.400 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.066) 0:07:59.726 ***** 2019-12-05T06:02:22.427 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-05T06:02:22.429 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.466 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.066) 0:07:59.792 ***** 2019-12-05T06:02:22.498 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.536 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.069) 0:07:59.862 ***** 2019-12-05T06:02:22.573 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-12-05T06:02:22.575 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.615 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.078) 0:07:59.941 ***** 2019-12-05T06:02:22.642 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.680 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.065) 0:08:00.006 ***** 2019-12-05T06:02:22.707 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-05T06:02:22.708 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.752 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.064) 0:08:00.071 ***** 2019-12-05T06:02:22.771 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2019-12-05T06:02:22.772 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.811 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.065) 0:08:00.137 ***** 2019-12-05T06:02:22.837 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2019-12-05T06:02:22.838 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.876 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.065) 0:08:00.203 ***** 2019-12-05T06:02:22.903 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:22.941 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:22 +0000 (0:00:00.064) 0:08:00.267 ***** 2019-12-05T06:02:22.968 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-12-05T06:02:22.969 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.007 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.065) 0:08:00.333 ***** 2019-12-05T06:02:23.034 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.085 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.077) 0:08:00.411 ***** 2019-12-05T06:02:23.111 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-12-05T06:02:23.112 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.153 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.067) 0:08:00.479 ***** 2019-12-05T06:02:23.158 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-05T06:02:23.160 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "changes": {"installed": ["@core", "@base", "iozone", "qemu-kvm", "mod_fastcgi-2.4.7-1.ceph.el7.centos"]}, "msg": "Warning: Group core does not have any packages to install.\nWarning: Group base does not have any packages to install.\nGroup base does have 1 conditional packages, which may get installed.\n", "rc": 0, "results": ["yum-plugin-priorities-1.1.31-52.el7.noarch providing yum-plugin-priorities is already installed", "yum-plugin-fastestmirror-1.1.31-50.el7.noarch providing yum-plugin-fastestmirror is already installed", "redhat-lsb-4.1-27.el7.centos.1.x86_64 providing redhat-lsb is already installed", "sysstat-10.1.5-17.el7.x86_64 providing sysstat is already installed", "gdb-7.6.1-114.el7.x86_64 providing gdb is already installed", "git-all-1.8.3.1-20.el7.noarch providing git-all is already installed", "python-configobj-4.7.2-7.el7.noarch providing python-configobj is already installed", "gcc-c++-4.8.5-39.el7.x86_64 providing gcc-c++ is already installed", "libedit-3.0-12.20121213cvs.el7.x86_64 providing libedit is already installed", "openssl098e-0.9.8e-29.el7.centos.3.x86_64 providing openssl098e is already installed", "boost-thread-1.53.0-27.el7.x86_64 providing boost-thread is already installed", "xfsprogs-4.5.0-20.el7.x86_64 providing xfsprogs is already installed", "gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "parted-3.1-29.el7.x86_64 providing parted is already installed", "libgcrypt-1.5.3-14.el7.x86_64 providing libgcrypt is already installed", "fuse-2.9.2-11.el7.x86_64 providing fuse is already installed", "fuse-libs-2.9.2-11.el7.x86_64 providing fuse-libs is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "1:openssl-1.0.2k-16.el7.x86_64 providing openssl is already installed", "libuuid-2.23.2-61.el7.x86_64 providing libuuid is already installed", "btrfs-progs-4.9.1-1.el7.x86_64 providing btrfs-progs is alrea 2019-12-05T06:02:23.162 INFO:teuthology.task.ansible.out:dy installed", "attr-2.4.46-13.el7.x86_64 providing attr is already installed", "1:valgrind-3.14.0-16.el7.x86_64 providing valgrind is already installed", "python-nose-1.3.7-1.el7.noarch providing python-nose is already installed", "mpich-3.0-3.0.4-10.el7.x86_64 providing mpich is already installed", "podman-1.4.4-4.el7.centos.x86_64 providing podman is already installed", "ant-1.9.4-2.el7.noarch providing ant is already installed", "libtool-2.4.2-22.el7_3.x86_64 providing libtool is already installed", "automake-1.13.4-3.el7.noarch providing automake is already installed", "gettext-0.19.8.1-2.el7.x86_64 providing gettext is already installed", "libuuid-devel-2.23.2-61.el7.x86_64 providing libuuid-devel is already installed", "libacl-devel-2.2.51-14.el7.x86_64 providing libacl-devel is already installed", "bc-1.06.95-13.el7.x86_64 providing bc is already installed", "xfsdump-3.1.7-1.el7.x86_64 providing xfsdump is already installed", "xfsprogs-devel-4.5.0-20.el7.x86_64 providing xfsprogs-devel is already installed", "blktrace-1.0.5-8.el7.x86_64 providing blktrace is already installed", "1:numpy-1.7.1-13.el7.x86_64 providing numpy is already installed", "python-matplotlib-1.2.0-16.el7.x86_64 providing python-matplotlib is already installed", "usbredir-0.7.1-3.el7.x86_64 providing usbredir is already installed", "genisoimage-1.1.11-25.el7.x86_64 providing genisoimage is already installed", "httpd-2.4.6-90.el7.centos.x86_64 providing httpd is already installed", "httpd-devel-2.4.6-90.el7.centos.x86_64 providing httpd-devel is already installed", "httpd-tools-2.4.6-90.el7.centos.x86_64 providing httpd-tools is already installed", "1:mod_ssl-2.4.6-90.el7.centos.x86_64 providing mod_ssl is already installed", "libev-devel-4.15-7.el7.x86_64 providing libev-devel is already installed", "perl-XML-Twig-3.44-2.el7.noarch providing perl-XML-Twig is already installed", "1:java-1.6.0-openjdk-devel-1.6.0.41-1.13.13.1.el7_3.x86_64 providing java-1.6.0-openjdk-devel is already insta 2019-12-05T06:02:23.164 INFO:teuthology.task.ansible.out:lled", "junit-4.11-8.el7.noarch providing junit4 is already installed", "1:nfs-utils-1.3.0-0.65.el7.x86_64 providing nfs-utils is already installed", "ncurses-devel-5.9-14.20130511.el7_4.x86_64 providing ncurses-devel is already installed", "python-devel-2.7.5-86.el7.x86_64 providing python-devel is already installed", "python-virtualenv-15.1.0-2.el7.noarch providing python-virtualenv is already installed", "perl-CPAN-1.9800-294.el7_6.noarch providing perl-CPAN is already installed", "Loaded plugins: fastestmirror, langpacks, priorities\nLoading mirror speeds from cached hostfile\n * base: mirror.rackspace.com\n * epel: download-cc-rdu01.fedoraproject.org\n * extras: mirror.vcu.edu\n * updates: mirror.vcu.edu\nResolving Dependencies\n--> Running transaction check\n---> Package iozone.x86_64 0:3.424-2_ceph.el7.centos will be installed\n---> Package mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos will be installed\n---> Package qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Processing Dependency: qemu-kvm-common = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: qemu-img = 10:1.5.3-167.el7_7.1 for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librbd.so.1()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Processing Dependency: librados.so.2()(64bit) for package: 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64\n--> Running transaction check\n---> Package librados2.x86_64 1:10.2.5-4.el7 will be installed\n--> Processing Dependency: libboost_random-mt.so.1.53.0()(64bit) for package: 1:librados2-10.2.5-4.el7.x86_64\n---> Package librbd1.x86_64 1:10.2.5-4.el7 will be installed\n---> Package qemu-img.x86_64 10:1.5.3-167.el7_7.1 will be installed\n---> Package qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 will be installed\n--> Running transaction check\n---> Package boost-random.x86_64 0:1.53.0-27.el7 will be installed\n--> Finished Dependency Resolution\n\nDependencies Resolved\n\n===== 2019-12-05T06:02:23.165 INFO:teuthology.task.ansible.out:===========================================================================\n Package Arch Version Repository Size\n================================================================================\nInstalling:\n iozone x86_64 3.424-2_ceph.el7.centos lab-extras 800 k\n mod_fastcgi x86_64 2.4.7-1.ceph.el7.centos centos7-fcgi-ceph 70 k\n qemu-kvm x86_64 10:1.5.3-167.el7_7.1 updates 1.9 M\nInstalling for dependencies:\n boost-random x86_64 1.53.0-27.el7 base 39 k\n librados2 x86_64 1:10.2.5-4.el7 base 1.8 M\n librbd1 x86_64 1:10.2.5-4.el7 base 2.4 M\n qemu-img x86_64 10:1.5.3-167.el7_7.1 updates 699 k\n qemu-kvm-common x86_64 10:1.5.3-167.el7_7.1 updates 435 k\n\nTransaction Summary\n================================================================================\nInstall 3 Packages (+5 Dependent packages)\n\nTotal download size: 8.1 M\nInstalled size: 23 M\nDownloading packages:\n--------------------------------------------------------------------------------\nTotal 20 MB/s | 8.1 MB 00:00 \nRunning transaction check\nRunning transaction test\nTransaction test succeeded\nRunning transaction\n Installing : boost-random-1.53.0-27.el7.x86_64 1/8 \n Installing : 1:librados2-10.2.5-4.el7.x86_64 2/8 \n Installing : 1:librbd1-10.2.5-4.el7.x86_64 3/8 \n Installing : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 4/8 \n Installing : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 5/8 \n Installing : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 6/8 \n Installing : iozone-3.424-2_ceph.el7.centos.x86_64 7/8 \n In 2019-12-05T06:02:23.167 INFO:teuthology.task.ansible.out:stalling : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 8/8 \n Verifying : 1:librbd1-10.2.5-4.el7.x86_64 1/8 \n Verifying : 10:qemu-kvm-common-1.5.3-167.el7_7.1.x86_64 2/8 \n Verifying : mod_fastcgi-2.4.7-1.ceph.el7.centos.x86_64 3/8 \n Verifying : 1:librados2-10.2.5-4.el7.x86_64 4/8 \n Verifying : 10:qemu-kvm-1.5.3-167.el7_7.1.x86_64 5/8 \n Verifying : 10:qemu-img-1.5.3-167.el7_7.1.x86_64 6/8 \n Verifying : boost-random-1.53.0-27.el7.x86_64 7/8 \n Verifying : iozone-3.424-2_ceph.el7.centos.x86_64 8/8 \n\nInstalled:\n iozone.x86_64 0:3.424-2_ceph.el7.centos \n mod_fastcgi.x86_64 0:2.4.7-1.ceph.el7.centos \n qemu-kvm.x86_64 10:1.5.3-167.el7_7.1 \n\nDependency Installed:\n boost-random.x86_64 0:1.53.0-27.el7 \n librados2.x86_64 1:10.2.5-4.el7 \n librbd1.x86_64 1:10.2.5-4.el7 \n qemu-img.x86_64 10:1.5.3-167.el7_7.1 \n qemu-kvm-common.x86_64 10:1.5.3-167.el7_7.1 \n\nComplete!\n"]} 2019-12-05T06:02:23.201 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.048) 0:08:00.527 ***** 2019-12-05T06:02:23.207 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-12-05T06:02:23.208 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.210 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:23.249 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.047) 0:08:00.575 ***** 2019-12-05T06:02:23.276 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-12-05T06:02:23.277 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.317 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.067) 0:08:00.643 ***** 2019-12-05T06:02:23.356 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-12-05T06:02:23.357 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.394 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.077) 0:08:00.720 ***** 2019-12-05T06:02:23.422 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T06:02:23.423 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-12-05T06:02:23.464 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.069) 0:08:00.790 ***** 2019-12-05T06:02:23.503 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-12-05T06:02:23.504 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.525 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-12-05T06:02:23.537 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.548 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.549 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.560 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.571 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.614 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.150) 0:08:00.941 ***** 2019-12-05T06:02:23.756 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-12-05T06:02:23.757 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003702", "end": "2019-12-05 06:02:23.736437", "rc": 0, "start": "2019-12-05 06:02:23.732735", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-12-05T06:02:23.800 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.185) 0:08:01.126 ***** 2019-12-05T06:02:23.839 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-12-05T06:02:23.842 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.882 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.082) 0:08:01.208 ***** 2019-12-05T06:02:23.909 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-12-05T06:02:23.910 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:23.950 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:23 +0000 (0:00:00.068) 0:08:01.277 ***** 2019-12-05T06:02:23.978 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2019-12-05T06:02:23.980 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["cryptopp-devel-5.6.2-10.el7.x86_64 providing cryptopp-devel is already installed", "cryptopp-5.6.2-10.el7.x86_64 providing cryptopp is already installed", "fcgi-2.4.0-25.el7.x86_64 providing fcgi is already installed", "dbench-4.0-10.el7.x86_64 providing dbench is already installed", "fuse-sshfs-2.10-1.el7.x86_64 providing fuse-sshfs is already installed", "bonnie++-1.97.3-1.el7.x86_64 providing bonnie++ is already installed", "1:perl-JSON-XS-3.01-2.el7.x86_64 providing perl-JSON-XS is already installed"]} TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:24.017 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:24 +0000 (0:00:00.066) 0:08:01.344 ***** 2019-12-05T06:02:24.054 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:24 +0000 (0:00:00.035) 0:08:01.380 ***** 2019-12-05T06:02:24.091 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:24.092 INFO:teuthology.task.ansible.out:TASK [testnode : Make sure apt dependencies are installed] ********************* skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:24.135 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:24 +0000 (0:00:00.080) 0:08:01.460 ***** 2019-12-05T06:02:24.565 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["device-mapper-multipath is not installed"]} 2019-12-05T06:02:24.609 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:24 +0000 (0:00:00.472) 0:08:01.932 ***** 2019-12-05T06:02:24.625 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-12-05T06:02:24.626 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.149-8.el7.x86_64 providing device-mapper is already installed"]} 2019-12-05T06:02:24.668 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:24 +0000 (0:00:00.061) 0:08:01.994 ***** 2019-12-05T06:02:24.674 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-05T06:02:24.675 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:24.725 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:24 +0000 (0:00:00.057) 0:08:02.051 ***** 2019-12-05T06:02:24.732 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-05T06:02:24.733 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={u'block_used': 13036430, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227288054, u'size_available': 930971869184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966956, u'device': u'/dev/sda1', u'inode_used': 88020, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227288054, "block_size": 4096, "block_total": 240324484, "block_used": 13036430, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966956, "inode_total": 61054976, "inode_used": 88020, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930971869184, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}} 2019-12-05T06:02:24.773 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:24 +0000 (0:00:00.048) 0:08:02.100 ***** 2019-12-05T06:02:25.095 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.012297", "end": "2019-12-05 06:02:25.071618", "rc": 0, "start": "2019-12-05 06:02:25.059321", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:25.133 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:25 +0000 (0:00:00.359) 0:08:02.459 ***** 2019-12-05T06:02:25.181 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-12-05T06:02:25.182 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={u'block_used': 13036430, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227288054, u'size_available': 930971869184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966956, u'device': u'/dev/sda1', u'inode_used': 88020, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227288054, "block_size": 4096, "block_total": 240324484, "block_used": 13036430, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966956, "inode_total": 61054976, "inode_used": 88020, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930971869184, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:25.183 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2019-12-05T06:02:25.185 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target dbus.service polkit.service system.slice", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "Conflicts": "iptables.service ip6tables.service shutdown.target ebtables.service ipset.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "EnvironmentFile": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "Igno 2019-12-05T06:02:25.187 INFO:teuthology.task.ansible.out:reOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "18446744073709551615", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127535", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127535", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", " 2019-12-05T06:02:25.188 INFO:teuthology.task.ansible.out:TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "Wants": "network-pre.target system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-05T06:02:25.225 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:25 +0000 (0:00:00.092) 0:08:02.551 ***** 2019-12-05T06:02:25.263 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:25 +0000 (0:00:00.037) 0:08:02.589 ***** 2019-12-05T06:02:25.289 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-12-05T06:02:25.332 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:25 +0000 (0:00:00.068) 0:08:02.658 ***** 2019-12-05T06:02:25.381 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-05T06:02:25.382 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QC9K', u'wwn-0x5000c50091e06051'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QC9K', u'wwn': u'0x5000c50091e06051', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QC9K-part1', u'wwn-0x5000c50091e06051-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QC9K", "wwn-0x5000c50091e06051"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QC9K-part1", "wwn-0x5000c50091e06051-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QC9K", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-05T06:02:25.384 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c50091e06051"}}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:25.684 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2019-12-05T06:02:25.685 INFO:teuthology.task.ansible.out:fatal: [smithi016.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2019-12-05T06:02:25.747 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:25 +0000 (0:00:00.415) 0:08:03.073 ***** 2019-12-05T06:02:26.151 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2019-12-05T06:02:26.153 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2019-12-05T06:02:26.192 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.444) 0:08:03.518 ***** 2019-12-05T06:02:26.219 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2019-12-05T06:02:26.221 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:26.259 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.067) 0:08:03.585 ***** 2019-12-05T06:02:26.285 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2019-12-05T06:02:26.326 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.064) 0:08:03.649 ***** 2019-12-05T06:02:26.349 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2019-12-05T06:02:26.351 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:26.392 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.068) 0:08:03.718 ***** 2019-12-05T06:02:26.420 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2019-12-05T06:02:26.420 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:26.461 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.069) 0:08:03.787 ***** 2019-12-05T06:02:26.498 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2019-12-05T06:02:26.500 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi016.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:26.538 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.077) 0:08:03.865 ***** 2019-12-05T06:02:26.565 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2019-12-05T06:02:26.566 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:26.605 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.065) 0:08:03.931 ***** 2019-12-05T06:02:26.631 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2019-12-05T06:02:26.672 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.063) 0:08:03.994 ***** 2019-12-05T06:02:26.695 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2019-12-05T06:02:26.696 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:26.733 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.064) 0:08:04.059 ***** 2019-12-05T06:02:26.759 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2019-12-05T06:02:26.761 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:26.798 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.065) 0:08:04.124 ***** 2019-12-05T06:02:26.826 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2019-12-05T06:02:26.828 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:26.867 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.068) 0:08:04.193 ***** 2019-12-05T06:02:26.893 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-05T06:02:26.895 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:26.939 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:26 +0000 (0:00:00.072) 0:08:04.265 ***** 2019-12-05T06:02:26.967 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.008 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.068) 0:08:04.334 ***** 2019-12-05T06:02:27.035 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2019-12-05T06:02:27.035 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.086 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.078) 0:08:04.412 ***** 2019-12-05T06:02:27.087 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.115 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.116 INFO:teuthology.task.ansible.out:TASK [testnode : Install non aarch64 packages] ********************************* 2019-12-05T06:02:27.118 INFO:teuthology.task.ansible.out: skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.166 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.079) 0:08:04.492 ***** 2019-12-05T06:02:27.167 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.173 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-05T06:02:27.174 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-KxSeC4-Zm7O-Owyc-aX7S-XwNI-sBIk-1M2yZT', u'nvme-INTEL_SSDPEDMD400G4_PHFT620400U4400BGN', u'nvme-nvme.8086-50484654363230343030553434303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.442879", "end": "2019-12-05 06:02:27.102093", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-KxSeC4-Zm7O-Owyc-aX7S-XwNI-sBIk-1M2yZT", "nvme-INTEL_SSDPEDMD400G4_PHFT620400U4400BGN", "nvme-nvme.8086-50484654363230343030553434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:25.659 2019-12-05T06:02:27.176 INFO:teuthology.task.ansible.out:214", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:27.187 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2019-12-05T06:02:27.187 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.225 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.059) 0:08:04.551 ***** 2019-12-05T06:02:27.252 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2019-12-05T06:02:27.253 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.289 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.063) 0:08:04.615 ***** 2019-12-05T06:02:27.326 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2019-12-05T06:02:27.327 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.364 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.075) 0:08:04.690 ***** 2019-12-05T06:02:27.418 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.419 INFO:teuthology.task.ansible.out:TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.455 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.090) 0:08:04.781 ***** 2019-12-05T06:02:27.457 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.493 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2019-12-05T06:02:27.494 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.539 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.083) 0:08:04.865 ***** 2019-12-05T06:02:27.578 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2019-12-05T06:02:27.579 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.616 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.077) 0:08:04.943 ***** 2019-12-05T06:02:27.644 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2019-12-05T06:02:27.646 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.684 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.067) 0:08:05.010 ***** 2019-12-05T06:02:27.712 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2019-12-05T06:02:27.713 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.751 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.067) 0:08:05.078 ***** 2019-12-05T06:02:27.753 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.800 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.802 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.839 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.087) 0:08:05.166 ***** 2019-12-05T06:02:27.841 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.871 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.872 INFO:teuthology.task.ansible.out:TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.909 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.068) 0:08:05.235 ***** 2019-12-05T06:02:27.910 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.940 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:27.941 INFO:teuthology.task.ansible.out:TASK [testnode : Configure /etc/sudoers.] ************************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:27.978 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:27 +0000 (0:00:00.069) 0:08:05.304 ***** 2019-12-05T06:02:27.980 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.006 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.007 INFO:teuthology.task.ansible.out:TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:28.043 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:28 +0000 (0:00:00.065) 0:08:05.370 ***** 2019-12-05T06:02:28.046 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.072 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.073 INFO:teuthology.task.ansible.out:TASK [testnode : Rebuild rpmdb] ************************************************ skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:28.110 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:28 +0000 (0:00:00.066) 0:08:05.436 ***** 2019-12-05T06:02:28.112 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.140 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.142 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure ceph packages are not present.] ************************ skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:28.177 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:28 +0000 (0:00:00.066) 0:08:05.503 ***** 2019-12-05T06:02:28.179 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.205 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.207 INFO:teuthology.task.ansible.out:TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:28.244 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:28 +0000 (0:00:00.067) 0:08:05.570 ***** 2019-12-05T06:02:28.246 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.277 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.278 INFO:teuthology.task.ansible.out:TASK [testnode : Remove packages] ********************************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:28.313 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:28 +0000 (0:00:00.069) 0:08:05.640 ***** 2019-12-05T06:02:28.315 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.341 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.343 INFO:teuthology.task.ansible.out:TASK [testnode : Install packages] ********************************************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:28.379 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:28 +0000 (0:00:00.065) 0:08:05.705 ***** 2019-12-05T06:02:28.382 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.415 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2019-12-05T06:02:28.416 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:28.455 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:28 +0000 (0:00:00.075) 0:08:05.780 ***** 2019-12-05T06:02:28.520 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.522 INFO:teuthology.task.ansible.out:TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:28.524 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOqr6lPynEfxhys8cENcKAlb1rIqdp6ZiE'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.056631", "end": "2019-12-05 06:02:28.477604", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOqr6lPynEfxhys8cENcKAlb1rIqdp6ZiE"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:27.420973", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:28.562 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:28 +0000 (0:00:00.107) 0:08:05.888 ***** 2019-12-05T06:02:28.563 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:28.897 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2019-12-05T06:02:28.935 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:28 +0000 (0:00:00.373) 0:08:06.261 ***** 2019-12-05T06:02:29.297 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T06:02:29.372 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:29 +0000 (0:00:00.436) 0:08:06.697 ***** 2019-12-05T06:02:29.721 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-05T06:02:29.722 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2019-12-05T06:02:29.762 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:29 +0000 (0:00:00.390) 0:08:07.088 ***** 2019-12-05T06:02:29.789 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-05T06:02:29.790 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:29.828 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:29 +0000 (0:00:00.066) 0:08:07.154 ***** 2019-12-05T06:02:29.858 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-05T06:02:29.860 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:29.917 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:29 +0000 (0:00:00.089) 0:08:07.243 ***** 2019-12-05T06:02:29.923 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-05T06:02:29.929 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOOr3SDkeJS9DENaDf5fGl6SKSFN21CbEe'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.076679", "end": "2019-12-05 06:02:29.868251", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOOr3SDkeJS9DENaDf5fGl6SKSFN21CbEe"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:28.791572", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:29.947 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2019-12-05T06:02:29.949 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:29.990 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:29 +0000 (0:00:00.072) 0:08:07.316 ***** 2019-12-05T06:02:30.018 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-05T06:02:30.020 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.062 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.072) 0:08:07.388 ***** 2019-12-05T06:02:30.110 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.170 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.108) 0:08:07.496 ***** 2019-12-05T06:02:30.204 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-05T06:02:30.207 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.244 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.074) 0:08:07.571 ***** 2019-12-05T06:02:30.272 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2019-12-05T06:02:30.274 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.323 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.078) 0:08:07.649 ***** 2019-12-05T06:02:30.349 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2019-12-05T06:02:30.351 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.388 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.065) 0:08:07.715 ***** 2019-12-05T06:02:30.415 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2019-12-05T06:02:30.417 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.454 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.065) 0:08:07.780 ***** 2019-12-05T06:02:30.481 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2019-12-05T06:02:30.483 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.519 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.065) 0:08:07.845 ***** 2019-12-05T06:02:30.546 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2019-12-05T06:02:30.548 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.587 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.065) 0:08:07.911 ***** 2019-12-05T06:02:30.624 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2019-12-05T06:02:30.625 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.661 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.075) 0:08:07.987 ***** 2019-12-05T06:02:30.688 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2019-12-05T06:02:30.690 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.726 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.065) 0:08:08.052 ***** 2019-12-05T06:02:30.756 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.790 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.064) 0:08:08.116 ***** 2019-12-05T06:02:30.817 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.855 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.065) 0:08:08.181 ***** 2019-12-05T06:02:30.882 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2019-12-05T06:02:30.884 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:30.920 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:30 +0000 (0:00:00.064) 0:08:08.246 ***** 2019-12-05T06:02:30.968 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2019-12-05T06:02:30.969 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.006 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.086) 0:08:08.332 ***** 2019-12-05T06:02:31.033 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2019-12-05T06:02:31.035 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.071 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.064) 0:08:08.397 ***** 2019-12-05T06:02:31.107 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2019-12-05T06:02:31.109 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.144 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.073) 0:08:08.471 ***** 2019-12-05T06:02:31.171 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2019-12-05T06:02:31.173 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.211 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.066) 0:08:08.537 ***** 2019-12-05T06:02:31.238 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2019-12-05T06:02:31.241 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.287 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.075) 0:08:08.613 ***** 2019-12-05T06:02:31.295 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-05T06:02:31.296 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOtLF2CPdkvOSBvNc3IQXDnMq8J603ylu6'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.067359", "end": "2019-12-05 06:02:31.235984", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOtLF2CPdkvOSBvNc3IQXDnMq8J603ylu6"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:30.168625", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:31.309 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2019-12-05T06:02:31.311 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.348 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.060) 0:08:08.674 ***** 2019-12-05T06:02:31.376 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T06:02:31.377 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2019-12-05T06:02:31.418 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.068) 0:08:08.743 ***** 2019-12-05T06:02:31.454 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2019-12-05T06:02:31.456 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.483 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2019-12-05T06:02:31.484 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.495 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.496 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.507 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.523 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.563 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.146) 0:08:08.889 ***** 2019-12-05T06:02:31.718 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2019-12-05T06:02:31.720 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003741", "end": "2019-12-05 06:02:31.697034", "rc": 0, "start": "2019-12-05 06:02:31.693293", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_master/virtualenv/bin/teuthology-lock"]} 2019-12-05T06:02:31.774 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.209) 0:08:09.098 ***** 2019-12-05T06:02:31.811 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2019-12-05T06:02:31.813 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.865 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.093) 0:08:09.192 ***** 2019-12-05T06:02:31.902 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2019-12-05T06:02:31.904 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:31.954 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:31 +0000 (0:00:00.074) 0:08:09.266 ***** 2019-12-05T06:02:31.966 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2019-12-05T06:02:31.968 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:32.005 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:32 +0000 (0:00:00.065) 0:08:09.331 ***** 2019-12-05T06:02:32.048 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2019-12-05T06:02:32.049 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:32.102 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:32 +0000 (0:00:00.097) 0:08:09.428 ***** 2019-12-05T06:02:32.586 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2019-12-05T06:02:32.587 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["gdisk-0.8.10-2.el7.x86_64 providing gdisk is already installed", "7:device-mapper-1.02.149-8.el7.x86_64 providing device-mapper is already installed"]} 2019-12-05T06:02:32.626 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:32 +0000 (0:00:00.524) 0:08:09.952 ***** 2019-12-05T06:02:32.666 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-05T06:02:32.668 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbObisas5Vvmtv5qpxYFL6kCVM9EtY7H3tI'], u'uuids': [u'a74ad527-9477-40a3-9c95-239912c79d7a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.071807", "end": "2019-12-05 06:02:32.627757", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbObisas5Vvmtv5qpxYFL6kCVM9EtY7H3tI"], "labels": [], "masters": [], "uuids": ["a74ad527-9477-40a3-9c95-239912c79d7a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:31.555950", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:32.685 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-05T06:02:32.687 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item={u'block_used': 13036456, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227288028, u'size_available': 930971762688, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966952, u'device': u'/dev/sda1', u'inode_used': 88024, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227288028, "block_size": 4096, "block_total": 240324484, "block_used": 13036456, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966952, "inode_total": 61054976, "inode_used": 88024, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930971762688, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}} 2019-12-05T06:02:32.726 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:32 +0000 (0:00:00.100) 0:08:10.053 ***** 2019-12-05T06:02:33.050 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2019-12-05T06:02:33.057 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.013877", "end": "2019-12-05 06:02:33.020078", "rc": 0, "start": "2019-12-05 06:02:33.006201", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:33.091 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:33 +0000 (0:00:00.364) 0:08:10.417 ***** 2019-12-05T06:02:33.139 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2019-12-05T06:02:33.141 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'block_used': 13036456, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227288028, u'size_available': 930971762688, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966952, u'device': u'/dev/sda1', u'inode_used': 88024, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227288028, "block_size": 4096, "block_total": 240324484, "block_used": 13036456, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966952, "inode_total": 61054976, "inode_used": 88024, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930971762688, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:33.186 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:33 +0000 (0:00:00.095) 0:08:10.512 ***** 2019-12-05T06:02:33.213 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2019-12-05T06:02:33.251 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:33 +0000 (0:00:00.064) 0:08:10.577 ***** 2019-12-05T06:02:33.299 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-05T06:02:33.300 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HKZ5', u'wwn-0x5000c50087198ecb'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4HKZ5', u'wwn': u'0x5000c50087198ecb', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HKZ5-part1', u'wwn-0x5000c50087198ecb-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HKZ5", "wwn-0x5000c50087198ecb"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HKZ5-part1", "wwn-0x5000c50087198ecb-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HKZ5", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-05T06:02:33.301 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c50087198ecb"}}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:34.059 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOyGeHp0e5vVS1BpJf2vMMWdP0H6hpWkoU'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.062678", "end": "2019-12-05 06:02:34.028226", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOyGeHp0e5vVS1BpJf2vMMWdP0H6hpWkoU"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:32.965548", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:34.131 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:34 +0000 (0:00:00.880) 0:08:11.457 ***** 2019-12-05T06:02:34.180 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T06:02:34.181 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QC9K', u'wwn-0x5000c50091e06051'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W5QC9K', u'wwn': u'0x5000c50091e06051', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W5QC9K-part1', u'wwn-0x5000c50091e06051-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QC9K", "wwn-0x5000c50091e06051"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QC9K-part1", "wwn-0x5000c50091e06051-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QC9K", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-05T06:02:34.183 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c50091e06051"}}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:34.557 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-KxSeC4-Zm7O-Owyc-aX7S-XwNI-sBIk-1M2yZT', u'nvme-INTEL_SSDPEDMD400G4_PHFT620400U4400BGN', u'nvme-nvme.8086-50484654363230343030553434303042474e-494e54454c205353445045444d443430304734-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'372.61 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'781422768', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPEDMD400G4', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.063208", "end": "2019-12-05 06:02:34.526156", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-KxSeC4-Zm7O-Owyc-aX7S-XwNI-sBIk-1M2yZT", "nvme-INTEL_SSDPEDMD400G4_PHFT620400U4400BGN", "nvme-nvme.8086-50484654363230343030553434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:34.462948", "stderr": 2019-12-05T06:02:34.558 INFO:teuthology.task.ansible.out: "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.0571274 s, 2.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.0571274 s, 2.0 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:35.030 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOqr6lPynEfxhys8cENcKAlb1rIqdp6ZiE'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.203253", "end": "2019-12-05 06:02:34.992443", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOqr6lPynEfxhys8cENcKAlb1rIqdp6ZiE"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:34.789190", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.196356 s, 587 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.196356 s, 587 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:35.190 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-05T06:02:35.191 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-2XiOKy-Gm6M-DRkN-bcfV-Hslo-ZUM9-wo7RTb', u'nvme-INTEL_SSDPED1D480GA_PHMB7513002C480DGN', u'nvme-nvme.8086-50484d42373531333030324334383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPED1D480GA', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.572440", "end": "2019-12-05 06:02:35.161449", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-2XiOKy-Gm6M-DRkN-bcfV-Hslo-ZUM9-wo7RTb", "nvme-INTEL_SSDPED1D480GA_PHMB7513002C480DGN", "nvme-nvme.8086-50484d42373531333030324334383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:33.589009", "stderr" 2019-12-05T06:02:35.192 INFO:teuthology.task.ansible.out:: "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:35.559 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T06:02:35.559 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOOr3SDkeJS9DENaDf5fGl6SKSFN21CbEe'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.205695", "end": "2019-12-05 06:02:35.518674", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOOr3SDkeJS9DENaDf5fGl6SKSFN21CbEe"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:35.312979", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.199324 s, 579 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.199324 s, 579 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:36.047 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOtLF2CPdkvOSBvNc3IQXDnMq8J603ylu6'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.203311", "end": "2019-12-05 06:02:36.015469", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOtLF2CPdkvOSBvNc3IQXDnMq8J603ylu6"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:35.812158", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.196413 s, 587 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.196413 s, 587 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:36.551 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR39DaldBI2O6rbr9vmneqXcKvpfIeB9xVh'], u'uuids': [u'ab2c1304-90a9-4730-aeca-f7481aedde07']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.051647", "end": "2019-12-05 06:02:36.517390", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR39DaldBI2O6rbr9vmneqXcKvpfIeB9xVh"], "labels": [], "masters": [], "uuids": ["ab2c1304-90a9-4730-aeca-f7481aedde07"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:35.465743", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition 2019-12-05T06:02:36.552 INFO:teuthology.task.ansible.out:the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:36.563 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbObisas5Vvmtv5qpxYFL6kCVM9EtY7H3tI'], u'uuids': [u'a74ad527-9477-40a3-9c95-239912c79d7a']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.193171", "end": "2019-12-05 06:02:36.525841", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbObisas5Vvmtv5qpxYFL6kCVM9EtY7H3tI"], "labels": [], "masters": [], "uuids": ["a74ad527-9477-40a3-9c95-239912c79d7a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:36.332670", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.186766 s, 618 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.186766 s, 618 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:37.080 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOyGeHp0e5vVS1BpJf2vMMWdP0H6hpWkoU'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHFT620400U4400BGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.186021", "end": "2019-12-05 06:02:37.044270", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-XfK2bN32EU9n4aDGyXk5r2Nt18YIZEbOyGeHp0e5vVS1BpJf2vMMWdP0H6hpWkoU"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHFT620400U4400BGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:36.858249", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.179112 s, 644 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.179112 s, 644 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:37.126 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:37 +0000 (0:00:02.994) 0:08:14.452 ***** 2019-12-05T06:02:37.533 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-12-05T06:02:37.535 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.078613", "end": "2019-12-05 06:02:37.495227", "rc": 0, "start": "2019-12-05 06:02:37.416614", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:37.583 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:37 +0000 (0:00:00.457) 0:08:14.909 ***** 2019-12-05T06:02:37.934 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3rtVzTw9h1FY26cHtYBu1CcjYkoIxAuT7'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.071943", "end": "2019-12-05 06:02:37.901152", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3rtVzTw9h1FY26cHtYBu1CcjYkoIxAuT7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:36.829209", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:38.001 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.066511", "end": "2019-12-05 06:02:37.971873", "rc": 0, "start": "2019-12-05 06:02:37.905362", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:38.065 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:38 +0000 (0:00:00.481) 0:08:15.391 ***** 2019-12-05T06:02:38.103 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-12-05T06:02:38.154 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:38 +0000 (0:00:00.089) 0:08:15.480 ***** 2019-12-05T06:02:38.197 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2019-12-05T06:02:38.198 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:38.248 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:38 +0000 (0:00:00.093) 0:08:15.574 ***** 2019-12-05T06:02:38.275 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2019-12-05T06:02:38.277 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:38.324 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:38 +0000 (0:00:00.075) 0:08:15.649 ***** 2019-12-05T06:02:38.370 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:38.415 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:38 +0000 (0:00:00.091) 0:08:15.741 ***** 2019-12-05T06:02:38.444 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2019-12-05T06:02:38.446 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:38.510 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:38 +0000 (0:00:00.095) 0:08:15.836 ***** 2019-12-05T06:02:38.808 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T06:02:38.852 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:38 +0000 (0:00:00.341) 0:08:16.178 ***** 2019-12-05T06:02:38.911 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ skipping: [smithi076.front.sepia.ceph.com] => (item={u'block_used': 13036430, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227288054, u'size_available': 930971869184, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966956, u'device': u'/dev/sda1', u'inode_used': 88020, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227288054, "block_size": 4096, "block_total": 240324484, "block_used": 13036430, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966956, "inode_total": 61054976, "inode_used": 88020, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930971869184, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:38.962 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:38 +0000 (0:00:00.109) 0:08:16.288 ***** 2019-12-05T06:02:39.001 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-12-05T06:02:39.002 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:39.048 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:39 +0000 (0:00:00.084) 0:08:16.372 ***** 2019-12-05T06:02:39.076 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T06:02:39.078 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:39.143 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:39 +0000 (0:00:00.093) 0:08:16.465 ***** 2019-12-05T06:02:39.166 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T06:02:39.168 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:39.208 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:39 +0000 (0:00:00.068) 0:08:16.534 ***** 2019-12-05T06:02:39.235 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-12-05T06:02:39.237 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:39.295 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:39 +0000 (0:00:00.087) 0:08:16.621 ***** 2019-12-05T06:02:39.297 INFO:teuthology.task.ansible.out: 2019-12-05T06:02:39.308 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-05T06:02:39.310 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3ea1EpF9QTbRXOsRzddDw54s70Len3ZLA'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.040834", "end": "2019-12-05 06:02:39.252017", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3ea1EpF9QTbRXOsRzddDw54s70Len3ZLA"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:38.211183", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:39.344 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-12-05T06:02:39.346 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:39.384 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:39 +0000 (0:00:00.088) 0:08:16.710 ***** 2019-12-05T06:02:39.422 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:39.466 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:39 +0000 (0:00:00.082) 0:08:16.792 ***** 2019-12-05T06:02:40.303 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2019-12-05T06:02:40.348 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:40 +0000 (0:00:00.881) 0:08:17.674 ***** 2019-12-05T06:02:40.660 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR36EapINGczo4P5BZkLkd5V6yGYPeFZsIU'], u'uuids': [u'80aca4ef-36a4-4cf8-be1c-7f17bc442f57']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.055550", "end": "2019-12-05 06:02:40.626715", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR36EapINGczo4P5BZkLkd5V6yGYPeFZsIU"], "labels": [], "masters": [], "uuids": ["80aca4ef-36a4-4cf8-be1c-7f17bc442f57"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:39.571165", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition th 2019-12-05T06:02:40.662 INFO:teuthology.task.ansible.out:e disk using fdisk or", "other utilities."]} 2019-12-05T06:02:41.161 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T06:02:41.636 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T06:02:42.061 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2019-12-05T06:02:42.062 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3UpBOqNZRMWUnFAneQVdlkUklFb0deZ9d'], u'uuids': [u'a775a883-7f92-47a1-9c68-0ae0ff5c92f1']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.049160", "end": "2019-12-05 06:02:42.027532", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3UpBOqNZRMWUnFAneQVdlkUklFb0deZ9d"], "labels": [], "masters": [], "uuids": ["a775a883-7f92-47a1-9c68-0ae0ff5c92f1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:40.978372", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot.\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2019-12-05T06:02:42.112 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:42 +0000 (0:00:01.763) 0:08:19.438 ***** 2019-12-05T06:02:42.120 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-05T06:02:42.121 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T06:02:42.176 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T06:02:42.178 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': u'ATA', u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HKZ5', u'wwn-0x5000c50087198ecb'], u'uuids': []}, u'sas_device_handle': None, u'host': u'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', u'support_discard': u'0', u'serial': u'Z1W4HKZ5', u'wwn': u'0x5000c50087198ecb', u'holders': [], u'size': u'931.51 GB', u'scheduler_mode': u'deadline', u'rotational': u'1', u'sectors': u'1953525168', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'ST1000NM0033-9ZM', u'partitions': {u'sda1': {u'start': u'2048', u'sectorsize': 512, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'sectors': u'1953522688', u'holders': [], u'links': {u'masters': [], u'labels': [], u'ids': [u'ata-ST1000NM0033-9ZM173_Z1W4HKZ5-part1', u'wwn-0x5000c50087198ecb-part1'], u'uuids': [u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67']}, u'size': u'931.51 GB'}}}, 'key': u'sda'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HKZ5", "wwn-0x5000c50087198ecb"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HKZ5-part1", "wwn-0x5000c50087198ecb-part1"], "labels": [], "masters": [], "uuids": ["60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HKZ5", "size": "931.51 GB", "support_discard": "0", "vendor": " 2019-12-05T06:02:42.179 INFO:teuthology.task.ansible.out:ATA", "virtual": 1, "wwn": "0x5000c50087198ecb"}}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:42.576 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [u'dm-0', u'dm-1', u'dm-2', u'dm-3', u'dm-4'], u'labels': [], u'ids': [u'lvm-pv-uuid-2XiOKy-Gm6M-DRkN-bcfV-Hslo-ZUM9-wo7RTb', u'nvme-INTEL_SSDPED1D480GA_PHMB7513002C480DGN', u'nvme-nvme.8086-50484d42373531333030324334383044474e-494e54454c2053534450454431443438304741-00000001'], u'uuids': []}, u'sas_device_handle': None, u'host': u'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [u'vg_nvme-lv_5', u'vg_nvme-lv_4', u'vg_nvme-lv_3', u'vg_nvme-lv_2', u'vg_nvme-lv_1'], u'size': u'447.13 GB', u'scheduler_mode': u'none', u'rotational': u'0', u'sectors': u'937703088', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': u'INTEL SSDPED1D480GA', u'partitions': {}}, 'key': u'nvme0n1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.064597", "end": "2019-12-05 06:02:42.536935", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_5", "vg_nvme-lv_4", "vg_nvme-lv_3", "vg_nvme-lv_2", "vg_nvme-lv_1"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-2XiOKy-Gm6M-DRkN-bcfV-Hslo-ZUM9-wo7RTb", "nvme-INTEL_SSDPED1D480GA_PHMB7513002C480DGN", "nvme-nvme.8086-50484d42373531333030324334383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:42.472338", "stderr": "110+0 record 2019-12-05T06:02:42.583 INFO:teuthology.task.ansible.out:s in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.058406 s, 2.0 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.058406 s, 2.0 GB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:42.587 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-05T06:02:42.589 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T06:02:43.054 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T06:02:43.055 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_1', u'dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR39DaldBI2O6rbr9vmneqXcKvpfIeB9xVh'], u'uuids': [u'ab2c1304-90a9-4730-aeca-f7481aedde07']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-4'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.153720", "end": "2019-12-05 06:02:43.017340", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR39DaldBI2O6rbr9vmneqXcKvpfIeB9xVh"], "labels": [], "masters": [], "uuids": ["ab2c1304-90a9-4730-aeca-f7481aedde07"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:42.863620", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.14723 s, 783 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.14723 s, 783 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:43.068 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T06:02:43.145 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:43 +0000 (0:00:01.032) 0:08:20.471 ***** 2019-12-05T06:02:43.505 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-05T06:02:43.507 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-12-05T06:02:43.508 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T06:02:43.512 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_3', u'dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3rtVzTw9h1FY26cHtYBu1CcjYkoIxAuT7'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-2'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.126110", "end": "2019-12-05 06:02:43.472630", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3rtVzTw9h1FY26cHtYBu1CcjYkoIxAuT7"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:43.346520", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.120051 s, 961 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.120051 s, 961 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:43.564 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:43 +0000 (0:00:00.419) 0:08:20.890 ***** 2019-12-05T06:02:43.620 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** skipping: [smithi076.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:43.956 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T06:02:43.968 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T06:02:43.968 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_2', u'dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3ea1EpF9QTbRXOsRzddDw54s70Len3ZLA'], u'uuids': []}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-3'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.125463", "end": "2019-12-05 06:02:43.937219", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3ea1EpF9QTbRXOsRzddDw54s70Len3ZLA"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:43.811756", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.119769 s, 963 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.119769 s, 963 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:44.256 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-05T06:02:44.258 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T06:02:44.413 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T06:02:44.414 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_5', u'dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR36EapINGczo4P5BZkLkd5V6yGYPeFZsIU'], u'uuids': [u'80aca4ef-36a4-4cf8-be1c-7f17bc442f57']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [], u'size': u'14.90 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'31252480', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-0'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.124588", "end": "2019-12-05 06:02:44.387093", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR36EapINGczo4P5BZkLkd5V6yGYPeFZsIU"], "labels": [], "masters": [], "uuids": ["80aca4ef-36a4-4cf8-be1c-7f17bc442f57"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:44.262505", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.118577 s, 973 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.118577 s, 973 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:44.550 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-05T06:02:44.552 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T06:02:44.835 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T06:02:44.845 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2019-12-05T06:02:44.846 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'sectorsize': u'512', u'vendor': None, u'links': {u'masters': [], u'labels': [], u'ids': [u'dm-name-vg_nvme-lv_4', u'dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3UpBOqNZRMWUnFAneQVdlkUklFb0deZ9d'], u'uuids': [u'a775a883-7f92-47a1-9c68-0ae0ff5c92f1']}, u'sas_device_handle': None, u'host': u'', u'support_discard': u'512', u'serial': u'PHMB7513002C480DGN', u'holders': [], u'size': u'89.40 GB', u'scheduler_mode': u'', u'rotational': u'0', u'sectors': u'187490304', u'sas_address': None, u'virtual': 1, u'removable': u'0', u'model': None, u'partitions': {}}, 'key': u'dm-1'}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.126142", "end": "2019-12-05 06:02:44.818340", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-v07NpmAywdgZOoRt3dZqEbe9hOsu7wR3UpBOqNZRMWUnFAneQVdlkUklFb0deZ9d"], "labels": [], "masters": [], "uuids": ["a775a883-7f92-47a1-9c68-0ae0ff5c92f1"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "serial": "PHMB7513002C480DGN", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2019-12-05 06:02:44.692198", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB) copied, 0.11988 s, 962 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB) copied, 0.11988 s, 962 MB/s"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:44.900 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:44 +0000 (0:00:01.335) 0:08:22.226 ***** 2019-12-05T06:02:44.946 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:44 +0000 (0:00:00.046) 0:08:22.273 ***** 2019-12-05T06:02:45.247 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-12-05T06:02:45.250 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T06:02:45.290 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:45 +0000 (0:00:00.343) 0:08:22.616 ***** 2019-12-05T06:02:45.298 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2019-12-05T06:02:45.299 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.077840", "end": "2019-12-05 06:02:45.257137", "rc": 0, "start": "2019-12-05 06:02:45.179297", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:45.357 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:45 +0000 (0:00:00.066) 0:08:22.683 ***** 2019-12-05T06:02:45.367 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-12-05T06:02:45.427 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:45 +0000 (0:00:00.070) 0:08:22.753 ***** 2019-12-05T06:02:45.739 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2019-12-05T06:02:45.741 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.058908", "end": "2019-12-05 06:02:45.710386", "rc": 0, "start": "2019-12-05 06:02:45.651478", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:45.809 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:45 +0000 (0:00:00.381) 0:08:23.135 ***** 2019-12-05T06:02:45.846 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2019-12-05T06:02:45.889 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:45 +0000 (0:00:00.079) 0:08:23.215 ***** 2019-12-05T06:02:45.916 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:45.958 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:45 +0000 (0:00:00.068) 0:08:23.283 ***** 2019-12-05T06:02:45.998 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:46.034 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:46 +0000 (0:00:00.076) 0:08:23.360 ***** 2019-12-05T06:02:46.061 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-05T06:02:46.062 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:46.102 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:46 +0000 (0:00:00.067) 0:08:23.428 ***** 2019-12-05T06:02:46.130 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:46.177 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:46 +0000 (0:00:00.075) 0:08:23.503 ***** 2019-12-05T06:02:46.183 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* changed: [smithi076.front.sepia.ceph.com] => {"changed": true} 2019-12-05T06:02:46.225 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:46 +0000 (0:00:00.047) 0:08:23.550 ***** 2019-12-05T06:02:46.518 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ ok: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2019-12-05T06:02:46.556 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:46 +0000 (0:00:00.332) 0:08:23.883 ***** 2019-12-05T06:02:46.615 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2019-12-05T06:02:46.616 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={u'block_used': 13036456, u'uuid': u'60ab4ba9-cdc2-4668-aaad-a5769fd8ef67', u'size_total': 984369086464, u'block_total': 240324484, u'mount': u'/', u'block_available': 227288028, u'size_available': 930971762688, u'fstype': u'ext4', u'inode_total': 61054976, u'inode_available': 60966952, u'device': u'/dev/sda1', u'inode_used': 88024, u'block_size': 4096, u'options': u'rw,seclabel,relatime,data=ordered'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227288028, "block_size": 4096, "block_total": 240324484, "block_used": 13036456, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60966952, "inode_total": 61054976, "inode_used": 88024, "mount": "/", "options": "rw,seclabel,relatime,data=ordered", "size_available": 930971762688, "size_total": 984369086464, "uuid": "60ab4ba9-cdc2-4668-aaad-a5769fd8ef67"}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:46.679 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:46 +0000 (0:00:00.117) 0:08:24.000 ***** 2019-12-05T06:02:46.725 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2019-12-05T06:02:46.727 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:46.784 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:46 +0000 (0:00:00.110) 0:08:24.110 ***** 2019-12-05T06:02:46.823 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T06:02:46.825 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:46.862 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:46 +0000 (0:00:00.077) 0:08:24.188 ***** 2019-12-05T06:02:46.894 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:46.929 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:46 +0000 (0:00:00.067) 0:08:24.255 ***** 2019-12-05T06:02:46.970 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2019-12-05T06:02:46.976 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:47.013 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.084) 0:08:24.339 ***** 2019-12-05T06:02:47.020 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2019-12-05T06:02:47.022 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-12-05T06:02:47.065 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.049) 0:08:24.389 ***** 2019-12-05T06:02:47.069 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2019-12-05T06:02:47.070 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:47.123 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.060) 0:08:24.449 ***** 2019-12-05T06:02:47.132 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-12-05T06:02:47.134 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:47.181 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.057) 0:08:24.507 ***** 2019-12-05T06:02:47.190 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2019-12-05T06:02:47.192 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:47.236 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.053) 0:08:24.561 ***** 2019-12-05T06:02:47.246 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:47.298 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.062) 0:08:24.624 ***** 2019-12-05T06:02:47.336 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-12-05T06:02:47.338 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:47.379 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.081) 0:08:24.705 ***** 2019-12-05T06:02:47.416 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-12-05T06:02:47.418 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:47.456 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.077) 0:08:24.782 ***** 2019-12-05T06:02:47.496 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-12-05T06:02:47.498 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:47.540 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.083) 0:08:24.866 ***** 2019-12-05T06:02:47.578 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-12-05T06:02:47.580 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:47.619 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.079) 0:08:24.945 ***** 2019-12-05T06:02:47.656 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-12-05T06:02:47.658 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:47.693 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.072) 0:08:25.018 ***** 2019-12-05T06:02:47.794 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2019-12-05T06:02:47.796 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'pvs': u'/dev/nvme0n1'}, 'key': u'vg_nvme'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2019-12-05T06:02:47.842 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:47 +0000 (0:00:00.142) 0:08:25.161 ***** 2019-12-05T06:02:48.184 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-12-05T06:02:48.185 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-28.el7.centos.x86_64 providing ntp is already installed"]} 2019-12-05T06:02:48.225 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:48 +0000 (0:00:00.389) 0:08:25.551 ***** 2019-12-05T06:02:48.258 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-12-05T06:02:48.260 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:48.308 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:48 +0000 (0:00:00.082) 0:08:25.634 ***** 2019-12-05T06:02:48.341 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-12-05T06:02:48.343 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:48.389 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:48 +0000 (0:00:00.081) 0:08:25.715 ***** 2019-12-05T06:02:48.396 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-05T06:02:48.398 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T06:02:48.867 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T06:02:49.330 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} 2019-12-05T06:02:49.377 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:49 +0000 (0:00:00.987) 0:08:26.703 ***** 2019-12-05T06:02:49.406 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T06:02:49.418 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:49.463 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:49 +0000 (0:00:00.086) 0:08:26.789 ***** 2019-12-05T06:02:49.840 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-05T06:02:49.842 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T06:02:49.978 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-12-05T06:02:49.979 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "ntpdate.service system.slice tmp.mount syslog.target systemd-journald.socket basic.target -.mount sntp.service", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "multi-user.target shutdown.target chronyd.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "1844674407370955161 2019-12-05T06:02:49.981 INFO:teuthology.task.ansible.out:5", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127511", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127511", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "-.mount basic.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec 2019-12-05T06:02:49.983 INFO:teuthology.task.ansible.out:": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-05T06:02:50.020 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:50 +0000 (0:00:00.556) 0:08:27.346 ***** 2019-12-05T06:02:50.318 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2019-12-05T06:02:50.320 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2019-12-05T06:02:50.369 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:50 +0000 (0:00:00.349) 0:08:27.695 ***** 2019-12-05T06:02:50.696 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2019-12-05T06:02:50.697 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2019-12-05T06:02:50.754 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:50 +0000 (0:00:00.385) 0:08:28.081 ***** 2019-12-05T06:02:50.806 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-05T06:02:50.807 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'value': {u'vg': u'vg_nvme', u'size': u'14.9G'}, 'key': u'lv_5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:50.902 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2019-12-05T06:02:50.903 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-10.el7.noarch providing python-pip is already installed"]} 2019-12-05T06:02:50.944 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:50 +0000 (0:00:00.189) 0:08:28.270 ***** 2019-12-05T06:02:50.972 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2019-12-05T06:02:50.974 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:51.016 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:51 +0000 (0:00:00.071) 0:08:28.342 ***** 2019-12-05T06:02:51.136 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_4'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T06:02:51.347 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2019-12-05T06:02:51.349 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-12-05T06:02:51.388 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:51 +0000 (0:00:00.372) 0:08:28.715 ***** 2019-12-05T06:02:51.448 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2019-12-05T06:02:51.449 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_3'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T06:02:51.760 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_2'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T06:02:52.118 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => (item={'value': {u'scratch_dev': True, u'vg': u'vg_nvme', u'size': u'89.4G'}, 'key': u'lv_1'}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2019-12-05T06:02:52.175 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:52 +0000 (0:00:00.786) 0:08:29.501 ***** 2019-12-05T06:02:52.327 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2019-12-05T06:02:52.329 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-12-05T06:02:52.374 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:52 +0000 (0:00:00.197) 0:08:29.699 ***** 2019-12-05T06:02:52.516 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2019-12-05T06:02:52.518 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2019-12-05T06:02:52.561 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:52 +0000 (0:00:00.187) 0:08:29.887 ***** 2019-12-05T06:02:52.602 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2019-12-05T06:02:52.604 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2019-12-05T06:02:52.666 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:52 +0000 (0:00:00.104) 0:08:29.992 ***** 2019-12-05T06:02:53.043 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2019-12-05T06:02:53.048 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true} 2019-12-05T06:02:53.082 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:53 +0000 (0:00:00.416) 0:08:30.408 ***** 2019-12-05T06:02:53.549 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2019-12-05T06:02:53.592 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:53 +0000 (0:00:00.509) 0:08:30.918 ***** 2019-12-05T06:02:53.622 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2019-12-05T06:02:53.623 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:53.670 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:53 +0000 (0:00:00.077) 0:08:30.996 ***** 2019-12-05T06:02:53.697 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2019-12-05T06:02:53.698 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:53.743 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:53 +0000 (0:00:00.073) 0:08:31.069 ***** 2019-12-05T06:02:53.770 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2019-12-05T06:02:53.771 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:53.815 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:53 +0000 (0:00:00.072) 0:08:31.141 ***** 2019-12-05T06:02:53.842 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2019-12-05T06:02:53.843 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:53.885 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:53 +0000 (0:00:00.070) 0:08:31.212 ***** 2019-12-05T06:02:53.924 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2019-12-05T06:02:53.925 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:53.966 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:53 +0000 (0:00:00.080) 0:08:31.292 ***** 2019-12-05T06:02:53.995 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2019-12-05T06:02:53.996 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:54.037 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.070) 0:08:31.363 ***** 2019-12-05T06:02:54.067 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2019-12-05T06:02:54.067 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:54.119 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.081) 0:08:31.445 ***** 2019-12-05T06:02:54.213 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2019-12-05T06:02:54.215 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 19.3.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 19.3.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-12-05T06:02:54.263 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.144) 0:08:31.589 ***** 2019-12-05T06:02:54.290 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-12-05T06:02:54.292 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:54.331 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.067) 0:08:31.657 ***** 2019-12-05T06:02:54.357 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-12-05T06:02:54.360 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2019-12-05T06:02:54.400 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.069) 0:08:31.726 ***** 2019-12-05T06:02:54.643 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2019-12-05T06:02:54.645 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["ntp-4.2.6p5-28.el7.centos.x86_64 providing ntp is already installed"]} 2019-12-05T06:02:54.682 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.281) 0:08:32.008 ***** 2019-12-05T06:02:54.699 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-12-05T06:02:54.700 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-12-05T06:02:54.741 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.059) 0:08:32.067 ***** 2019-12-05T06:02:54.748 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2019-12-05T06:02:54.749 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:54.804 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.058) 0:08:32.126 ***** 2019-12-05T06:02:54.806 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:54.847 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.047) 0:08:32.173 ***** 2019-12-05T06:02:54.853 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2019-12-05T06:02:54.856 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:54.894 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.046) 0:08:32.220 ***** 2019-12-05T06:02:54.900 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-12-05T06:02:54.901 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-12-05T06:02:54.947 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:54 +0000 (0:00:00.052) 0:08:32.273 ***** 2019-12-05T06:02:54.979 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:55.014 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:55 +0000 (0:00:00.066) 0:08:32.340 ***** 2019-12-05T06:02:55.338 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2019-12-05T06:02:55.339 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-12-05T06:02:55.375 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:55 +0000 (0:00:00.360) 0:08:32.701 ***** 2019-12-05T06:02:55.704 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-12-05T06:02:55.706 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-12-05T06:02:55.748 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:55 +0000 (0:00:00.372) 0:08:33.073 ***** 2019-12-05T06:02:55.790 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:55.827 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:55 +0000 (0:00:00.080) 0:08:33.153 ***** 2019-12-05T06:02:55.882 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-12-05T06:02:55.884 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:55.886 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "secontext": "system_u:object_r:net_conf_t:s0", "size": 2767, "state": "file", "uid": 0} TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* skipping: [smithi076.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:55.887 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2019-12-05T06:02:55.928 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:55 +0000 (0:00:00.100) 0:08:33.254 ***** 2019-12-05T06:02:55.963 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:55 +0000 (0:00:00.035) 0:08:33.289 ***** 2019-12-05T06:02:55.969 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2019-12-05T06:02:55.970 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.016 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.051) 0:08:33.342 ***** 2019-12-05T06:02:56.029 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** skipping: [smithi076.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.030 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.032 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.043 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'enp3s0f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.083 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.067) 0:08:33.409 ***** 2019-12-05T06:02:56.109 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T06:02:56.115 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.151 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.067) 0:08:33.477 ***** 2019-12-05T06:02:56.178 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-12-05T06:02:56.180 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.216 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.064) 0:08:33.542 ***** 2019-12-05T06:02:56.243 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-12-05T06:02:56.245 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.287 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.070) 0:08:33.613 ***** 2019-12-05T06:02:56.325 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2019-12-05T06:02:56.326 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.380 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.092) 0:08:33.706 ***** 2019-12-05T06:02:56.418 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-12-05T06:02:56.419 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.459 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.078) 0:08:33.785 ***** 2019-12-05T06:02:56.486 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-05T06:02:56.488 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.530 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.071) 0:08:33.857 ***** 2019-12-05T06:02:56.537 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2019-12-05T06:02:56.539 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "enabled": true, "name": "ntpd", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "system.slice syslog.target ntpdate.service basic.target -.mount sntp.service tmp.mount systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "0", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target multi-user.target chronyd.service", "BlockIOAccounting": "no", "BlockIOWeight": "18446744073709551615", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "18446744073709551615", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "18446744073709551615", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConflictedBy": "chronyd.service", "Conflicts": "shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "EnvironmentFile": "/etc/sysconfig/ntpd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/ntpd ; argv[]=/usr/sbin/ntpd -u ntp:ntp $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/ntpd.service", "GuessMainPID": "yes", "IOScheduling": "0", "Id": "ntpd.service", "IgnoreOnIsolate": "no", "IgnoreOnSnapshot": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobTimeoutAction": "none", "JobTimeoutUSec": "0", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "1844674407370955161 2019-12-05T06:02:56.542 INFO:teuthology.task.ansible.out:5", "LimitCORE": "18446744073709551615", "LimitCPU": "18446744073709551615", "LimitDATA": "18446744073709551615", "LimitFSIZE": "18446744073709551615", "LimitLOCKS": "18446744073709551615", "LimitMEMLOCK": "65536", "LimitMSGQUEUE": "819200", "LimitNICE": "0", "LimitNOFILE": "4096", "LimitNPROC": "127535", "LimitRSS": "18446744073709551615", "LimitRTPRIO": "0", "LimitRTTIME": "18446744073709551615", "LimitSIGPENDING": "127535", "LimitSTACK": "18446744073709551615", "LoadState": "loaded", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "18446744073709551615", "MemoryLimit": "18446744073709551615", "MountFlags": "0", "Names": "ntpd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "ProtectHome": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "Requires": "basic.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitInterval": "10000000", "StartupBlockIOWeight": "18446744073709551615", "StartupCPUShares": "18446744073709551615", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "no", "TasksCurrent": "18446744073709551615", "TasksMax": "18446744073709551615", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec 2019-12-05T06:02:56.544 INFO:teuthology.task.ansible.out:": "50000", "Transient": "no", "Type": "forking", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "WantedBy": "multi-user.target", "Wants": "system.slice", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2019-12-05T06:02:56.582 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.050) 0:08:33.908 ***** 2019-12-05T06:02:56.588 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-12-05T06:02:56.590 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.646 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.064) 0:08:33.972 ***** 2019-12-05T06:02:56.677 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-05T06:02:56.679 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:56.725 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:56 +0000 (0:00:00.078) 0:08:34.051 ***** 2019-12-05T06:02:57.077 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-12-05T06:02:57.078 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.017350", "end": "2019-12-05 06:02:57.032636", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-12-05 06:02:57.015286", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:57.378 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006126", "end": "2019-12-05 06:02:57.343085", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-12-05 06:02:57.336959", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:57.527 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "msg": "", "rc": 0, "results": ["python2-pip-8.1.2-10.el7.noarch providing python-pip is already installed"]} 2019-12-05T06:02:57.567 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:57 +0000 (0:00:00.841) 0:08:34.892 ***** 2019-12-05T06:02:57.631 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:02:57.674 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:57 +0000 (0:00:00.107) 0:08:35.000 ***** 2019-12-05T06:02:57.714 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-12-05T06:02:57.716 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.006734", "end": "2019-12-05 06:02:57.680261", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-12-05 06:02:57.673527", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:58.013 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2019-12-05T06:02:58.074 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:58 +0000 (0:00:00.400) 0:08:35.400 ***** 2019-12-05T06:02:58.097 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2019-12-05T06:02:58.100 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.008245", "end": "2019-12-05 06:02:58.005728", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-12-05 06:02:57.997483", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:02:58.145 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:58 +0000 (0:00:00.067) 0:08:35.468 ***** 2019-12-05T06:02:58.183 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-12-05T06:02:58.225 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:58 +0000 (0:00:00.079) 0:08:35.548 ***** 2019-12-05T06:02:58.535 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-12-05T06:02:58.537 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-05T06:02:59.059 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1215, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2019-12-05T06:02:59.095 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:02:59 +0000 (0:00:00.873) 0:08:36.421 ***** 2019-12-05T06:02:59.096 INFO:teuthology.task.ansible.out: 2019-12-05T06:03:00.595 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2019-12-05T06:03:00.596 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip2", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "You are using pip version 8.1.2, however version 19.3.1 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n", "stderr_lines": ["You are using pip version 8.1.2, however version 19.3.1 is available.", "You should consider upgrading via the 'pip install --upgrade pip' command."], "stdout": "Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages\nRequirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)\nRequirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied (use --upgrade to upgrade): remoto>=0.0.35 in /usr/lib/python2.7/site-packages", "Requirement already satisfied (use --upgrade to upgrade): execnet in /usr/lib/python2.7/site-packages (from remoto>=0.0.35)", "Requirement already satisfied (use --upgrade to upgrade): apipkg>=1.4 in /usr/lib/python2.7/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2019-12-05T06:03:00.646 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:00 +0000 (0:00:01.550) 0:08:37.972 ***** 2019-12-05T06:03:00.684 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2019-12-05T06:03:00.686 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:00.730 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:00 +0000 (0:00:00.083) 0:08:38.056 ***** 2019-12-05T06:03:00.763 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2019-12-05T06:03:00.764 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2019-12-05T06:03:00.807 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:00 +0000 (0:00:00.075) 0:08:38.132 ***** 2019-12-05T06:03:01.124 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2019-12-05T06:03:01.126 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2019-12-05T06:03:01.170 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:01 +0000 (0:00:00.363) 0:08:38.496 ***** 2019-12-05T06:03:01.196 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2019-12-05T06:03:01.198 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:01.235 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:01 +0000 (0:00:00.064) 0:08:38.561 ***** 2019-12-05T06:03:01.263 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2019-12-05T06:03:01.265 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2019-12-05T06:03:01.301 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:01 +0000 (0:00:00.067) 0:08:38.628 ***** 2019-12-05T06:03:01.329 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2019-12-05T06:03:01.332 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:01.375 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:01 +0000 (0:00:00.073) 0:08:38.701 ***** 2019-12-05T06:03:01.708 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** changed: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2019-12-05T06:03:01.750 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:01 +0000 (0:00:00.375) 0:08:39.077 ***** 2019-12-05T06:03:02.079 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2019-12-05T06:03:02.081 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2019-12-05T06:03:02.127 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.376) 0:08:39.453 ***** 2019-12-05T06:03:02.154 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2019-12-05T06:03:02.155 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.204 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.076) 0:08:39.530 ***** 2019-12-05T06:03:02.241 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2019-12-05T06:03:02.242 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} skipping: [smithi016.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.253 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.264 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.265 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.307 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.103) 0:08:39.633 ***** 2019-12-05T06:03:02.345 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2019-12-05T06:03:02.347 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.368 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno1', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi016.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'eno2', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.380 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'lo', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.385 INFO:teuthology.task.ansible.out: skipping: [smithi016.front.sepia.ceph.com] => (item={'skip_reason': u'Conditional result was False', 'item': u'ens1f0', 'skipped': True, 'ansible_loop_var': u'item', 'changed': False}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.423 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.116) 0:08:39.749 ***** 2019-12-05T06:03:02.453 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2019-12-05T06:03:02.455 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.497 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.073) 0:08:39.823 ***** 2019-12-05T06:03:02.545 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2019-12-05T06:03:02.547 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.585 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.088) 0:08:39.911 ***** 2019-12-05T06:03:02.616 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2019-12-05T06:03:02.618 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.661 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.075) 0:08:39.987 ***** 2019-12-05T06:03:02.689 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.736 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.074) 0:08:40.062 ***** 2019-12-05T06:03:02.765 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2019-12-05T06:03:02.767 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.805 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.069) 0:08:40.131 ***** 2019-12-05T06:03:02.842 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.900 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.095) 0:08:40.227 ***** 2019-12-05T06:03:02.939 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2019-12-05T06:03:02.941 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:02.983 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:02 +0000 (0:00:00.082) 0:08:40.309 ***** 2019-12-05T06:03:03.010 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2019-12-05T06:03:03.012 INFO:teuthology.task.ansible.out:skipping: [smithi016.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2019-12-05T06:03:03.055 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:03 +0000 (0:00:00.072) 0:08:40.382 ***** 2019-12-05T06:03:03.415 INFO:teuthology.task.ansible.out: 2019-12-05T06:03:03.417 INFO:teuthology.task.ansible.out:TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi016.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.014470", "end": "2019-12-05 06:03:03.374477", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2019-12-05 06:03:03.360007", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:03:03.748 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.007077", "end": "2019-12-05 06:03:03.709116", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2019-12-05 06:03:03.702039", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:03:04.058 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.006845", "end": "2019-12-05 06:03:04.032719", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2019-12-05 06:03:04.025874", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2019-12-05T06:03:04.385 INFO:teuthology.task.ansible.out:ok: [smithi016.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.007659", "end": "2019-12-05 06:03:04.350549", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2019-12-05 06:03:04.342890", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2019-12-05T06:03:04.423 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:04 +0000 (0:00:01.367) 0:08:41.749 ***** 2019-12-05T06:03:04.450 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2019-12-05T06:03:04.496 INFO:teuthology.task.ansible.out:Thursday 05 December 2019 06:03:04 +0000 (0:00:00.064) 0:08:41.814 ***** 2019-12-05T06:03:04.806 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2019-12-05T06:03:04.808 INFO:teuthology.task.ansible.out:changed: [smithi016.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2019-12-05T06:03:04.908 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2019-12-05T06:03:04.921 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi016.front.sepia.ceph.com : ok=121 changed=41 unreachable=0 failed=0 skipped=159 rescued=0 ignored=3 smithi076.front.sepia.ceph.com : ok=121 changed=41 unreachable=0 failed=0 skipped=159 rescued=0 ignored=3 Thursday 05 December 2019 06:03:04 +0000 (0:00:00.425) 0:08:42.239 ***** =============================================================================== common : Update yum cache --------------------------------------------- 237.48s users : Create all admin users with sudo access. ----------------------- 35.66s users : Update authorized_keys using the keys repo --------------------- 33.10s common : nrpe - Ensure SELinux policy is up to date -------------------- 20.38s users : Remove revoked users ------------------------------------------- 13.65s testnode : Install packages -------------------------------------------- 10.58s testnode : Ensure ceph dependency packages are not present. ------------- 3.48s Gathering Facts --------------------------------------------------------- 3.32s ansible-managed : Add authorized keys for the ansible user. ------------- 3.08s users : Update authorized_keys for each user with literal keys ---------- 2.51s testnode : Rebuild rpmdb ------------------------------------------------ 2.50s testnode : Configure version specific repos in /etc/yum.repos.d/ -------- 2.02s common : Make sure smartmontools is installed --------------------------- 1.98s common : Set the hardware clock ----------------------------------------- 1.59s testnode : Install packages via pip ------------------------------------- 1.55s ansible-managed : Create the cephlab_sudo sudoers.d file. ------- 2019-12-05T06:03:04.923 INFO:teuthology.task.ansible.out:-------- 1.45s testnode : List any leftover Ceph artifacts from previous jobs ---------- 1.37s common : Configure epel repos in /etc/yum.repos.d/ ---------------------- 1.20s common : Upload nagios nrpe config. ------------------------------------- 1.05s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.03s 2019-12-05T06:03:05.212 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi016.front.sepia.ceph.com')] 2019-12-05T06:03:05.214 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T06:03:05.214 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi076.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T06:03:05.316 INFO:teuthology.orchestra.run.smithi076:> true 2019-12-05T06:03:05.721 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2019-12-05T06:03:05.722 DEBUG:teuthology.orchestra.connection:{'username': 'ubuntu', 'hostname': 'smithi016.front.sepia.ceph.com', 'timeout': 60} 2019-12-05T06:03:05.857 INFO:teuthology.orchestra.run.smithi016:> true 2019-12-05T06:03:06.247 INFO:teuthology.run_tasks:Running task clock... 2019-12-05T06:03:06.291 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2019-12-05T06:03:06.291 INFO:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-12-05T06:03:06.443 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-12-05T06:03:16.536 INFO:teuthology.orchestra.run.smithi076.stdout:ntpd: time slew -0.001437s 2019-12-05T06:03:16.568 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit not found. 2019-12-05T06:03:16.669 INFO:teuthology.orchestra.run.smithi076.stdout: remote refid st t when poll reach delay offset jitter 2019-12-05T06:03:16.669 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================== 2019-12-05T06:03:16.670 INFO:teuthology.orchestra.run.smithi076.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T06:03:16.670 INFO:teuthology.orchestra.run.smithi076.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T06:03:16.670 INFO:teuthology.orchestra.run.smithi076.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T06:03:16.670 INFO:teuthology.orchestra.run.smithi076.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T06:03:16.671 INFO:teuthology.orchestra.run.smithi016:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2019-12-05T06:03:16.819 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2019-12-05T06:03:26.907 INFO:teuthology.orchestra.run.smithi016.stdout:ntpd: time slew -0.003268s 2019-12-05T06:03:26.949 INFO:teuthology.orchestra.run.smithi016.stderr:Failed to start ntp.service: Unit not found. 2019-12-05T06:03:27.025 INFO:teuthology.orchestra.run.smithi016.stdout: remote refid st t when poll reach delay offset jitter 2019-12-05T06:03:27.025 INFO:teuthology.orchestra.run.smithi016.stdout:============================================================================== 2019-12-05T06:03:27.025 INFO:teuthology.orchestra.run.smithi016.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T06:03:27.026 INFO:teuthology.orchestra.run.smithi016.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T06:03:27.026 INFO:teuthology.orchestra.run.smithi016.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T06:03:27.026 INFO:teuthology.orchestra.run.smithi016.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2019-12-05T06:03:27.027 INFO:teuthology.run_tasks:Running task install... 2019-12-05T06:03:27.054 DEBUG:teuthology.task.install:project ceph 2019-12-05T06:03:27.054 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'}} 2019-12-05T06:03:27.054 DEBUG:teuthology.task.install:config {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'} 2019-12-05T06:03:27.054 INFO:teuthology.task.install:Using flavor: basic 2019-12-05T06:03:27.063 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'ceph-daemon', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python-ceph', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-daemon', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-cloud', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-ssh', 'ceph-fuse', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python-ceph', 'rbd-fuse']} 2019-12-05T06:03:27.064 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2019-12-05T06:03:27.064 INFO:teuthology.orchestra.run.smithi076:> sudo lsb_release -is 2019-12-05T06:03:27.232 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS 2019-12-05T06:03:27.234 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-05T06:03:27.234 INFO:teuthology.orchestra.run.smithi016:> sudo lsb_release -is 2019-12-05T06:03:27.235 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'flavor': 'basic', 'extras': None} 2019-12-05T06:03:27.235 INFO:teuthology.orchestra.run.smithi076:> sudo lsb_release -is 2019-12-05T06:03:27.364 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS 2019-12-05T06:03:27.366 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-05T06:03:27.367 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T06:03:27.419 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS 2019-12-05T06:03:27.420 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-05T06:03:27.420 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec', 'exclude_packages': [], 'debuginfo': None, 'install_ceph_packages': True, 'wait_for_package': False, 'downgrade_packages': [], 'project': 'ceph', 'tag': None, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'branch': None, 'packages': {}, 'repos_only': False, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'flavor': 'basic', 'extras': None} 2019-12-05T06:03:27.421 INFO:teuthology.orchestra.run.smithi016:> sudo lsb_release -is 2019-12-05T06:03:27.545 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/centos/7/flavors/default/ 2019-12-05T06:03:27.546 INFO:teuthology.task.install.rpm:Package version is 15.0.0-8014.gc529080 2019-12-05T06:03:27.566 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS 2019-12-05T06:03:27.570 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-05T06:03:27.570 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T06:03:27.679 INFO:teuthology.orchestra.run.smithi076:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-12-05T06:03:27.733 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20191204.231200/c5290805033483ba0cec165c06bde07dae5f19ec/centos/7/flavors/default/ 2019-12-05T06:03:27.734 INFO:teuthology.task.install.rpm:Package version is 15.0.0-8014.gc529080 2019-12-05T06:03:27.808 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-daemon, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2019-12-05T06:03:27.809 INFO:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c5290805033483ba0cec165c06bde07dae5f19ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-12-05T06:03:27.866 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /etc/yum.repos.d/ceph.repo' 2019-12-05T06:03:27.918 INFO:teuthology.orchestra.run.smithi076:> test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-12-05T06:03:28.030 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-daemon, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-cloud, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-ssh, ceph-fuse, libcephfs2, libcephfs-devel, librados2, librbd1, python-ceph, rbd-fuse, python3-cephfs, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel, bison, flex, elfutils-libelf-devel, openssl-devel on remote rpm x86_64 2019-12-05T06:03:28.031 INFO:teuthology.orchestra.run.smithi016:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c5290805033483ba0cec165c06bde07dae5f19ec/;g' /etc/yum.repos.d/ceph.repo ; fi 2019-12-05T06:03:28.128 INFO:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-12-05T06:03:28.152 INFO:teuthology.orchestra.run.smithi016:> test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2019-12-05T06:03:28.229 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2019-12-05T06:03:28.231 INFO:teuthology.orchestra.run.smithi076:> sudo yum clean all 2019-12-05T06:03:28.245 INFO:teuthology.orchestra.run.smithi016:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2019-12-05T06:03:28.343 INFO:teuthology.orchestra.run.smithi016.stdout:check_obsoletes = 1 2019-12-05T06:03:28.345 INFO:teuthology.orchestra.run.smithi016:> sudo yum clean all 2019-12-05T06:03:28.415 INFO:teuthology.orchestra.run.smithi076.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-05T06:03:28.453 INFO:teuthology.orchestra.run.smithi076.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras 2019-12-05T06:03:28.454 INFO:teuthology.orchestra.run.smithi076.stdout: : lab-extras updates 2019-12-05T06:03:28.499 INFO:teuthology.orchestra.run.smithi076.stdout:Cleaning up list of fastest mirrors 2019-12-05T06:03:28.500 INFO:teuthology.orchestra.run.smithi076.stdout:Other repos take up 1.5 M of disk space (use --verbose for details) 2019-12-05T06:03:28.513 INFO:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-daemon ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2019-12-05T06:03:28.565 INFO:teuthology.orchestra.run.smithi016.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-05T06:03:28.607 INFO:teuthology.orchestra.run.smithi016.stdout:Cleaning repos: base centos7-fcgi-ceph ceph ceph-noarch ceph-source epel extras 2019-12-05T06:03:28.607 INFO:teuthology.orchestra.run.smithi016.stdout: : lab-extras updates 2019-12-05T06:03:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:Cleaning up list of fastest mirrors 2019-12-05T06:03:28.654 INFO:teuthology.orchestra.run.smithi016.stdout:Other repos take up 1.5 M of disk space (use --verbose for details) 2019-12-05T06:03:28.667 INFO:teuthology.orchestra.run.smithi016:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-daemon ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-cloud ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-ssh ceph-fuse libcephfs2 libcephfs-devel librados2 librbd1 python-ceph rbd-fuse python3-cephfs bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel bison flex elfutils-libelf-devel openssl-devel 2019-12-05T06:03:28.743 INFO:teuthology.orchestra.run.smithi076.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-05T06:03:28.787 INFO:teuthology.orchestra.run.smithi076.stdout:Determining fastest mirrors 2019-12-05T06:03:28.886 INFO:teuthology.orchestra.run.smithi016.stdout:Loaded plugins: fastestmirror, langpacks, priorities 2019-12-05T06:03:28.925 INFO:teuthology.orchestra.run.smithi016.stdout:Determining fastest mirrors 2019-12-05T06:03:29.518 INFO:teuthology.orchestra.run.smithi076.stdout: * base: mirror.rackspace.com 2019-12-05T06:03:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-05T06:03:29.520 INFO:teuthology.orchestra.run.smithi076.stdout: * extras: mirror.vcu.edu 2019-12-05T06:03:29.521 INFO:teuthology.orchestra.run.smithi076.stdout: * updates: mirror.vcu.edu 2019-12-05T06:03:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: * base: mirror.rackspace.com 2019-12-05T06:03:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: * epel: download-cc-rdu01.fedoraproject.org 2019-12-05T06:03:29.616 INFO:teuthology.orchestra.run.smithi016.stdout: * extras: mirror.vcu.edu 2019-12-05T06:03:29.617 INFO:teuthology.orchestra.run.smithi016.stdout: * updates: mirror.vcu.edu 2019-12-05T06:03:37.233 INFO:teuthology.orchestra.run.smithi016.stdout:7 packages excluded due to repository priority protections 2019-12-05T06:03:37.422 INFO:teuthology.orchestra.run.smithi076.stdout:7 packages excluded due to repository priority protections 2019-12-05T06:03:44.856 INFO:teuthology.orchestra.run.smithi076.stdout:Resolving Dependencies 2019-12-05T06:03:44.857 INFO:teuthology.orchestra.run.smithi076.stdout:--> Running transaction check 2019-12-05T06:03:44.857 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2019-12-05T06:03:44.866 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.866 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: ceph-mds = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.869 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: ceph-mon = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.869 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: ceph-osd = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.869 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-daemon.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.871 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-fuse.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.872 INFO:teuthology.orchestra.run.smithi016.stdout:Resolving Dependencies 2019-12-05T06:03:44.872 INFO:teuthology.orchestra.run.smithi016.stdout:--> Running transaction check 2019-12-05T06:03:44.873 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package bison.x86_64 0:3.0.4-2.el7 will be installed 2019-12-05T06:03:44.879 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: /usr/bin/python3 for package: 2:ceph-fuse-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.880 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.882 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: ceph-mds = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.882 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python3 for package: 2:ceph-fuse-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.883 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-mgr.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.883 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: ceph-mon = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.884 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: ceph-osd = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.885 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: ceph-base = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.886 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-daemon.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.886 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: pyOpenSSL for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.887 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-fuse.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.888 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.889 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.890 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.891 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.892 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.893 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-mgr-dashboard.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.894 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-mgr-dashboard-15.0.0-8014.gc529080.el7.noarch 2019-12-05T06:03:44.895 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-8014.gc529080.el7.noarch 2019-12-05T06:03:44.897 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: /usr/bin/python3 for package: 2:ceph-fuse-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.897 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.897 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.897 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-15.0.0-8014.gc529080.el7.noarch 2019-12-05T06:03:44.898 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-mgr-rook.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.898 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-15.0.0-8014.gc529080.el7.noarch 2019-12-05T06:03:44.899 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-mgr-ssh.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.900 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python3 for package: 2:ceph-fuse-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.900 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-mgr.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.900 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-15.0.0-8014.gc529080.el7.noarch 2019-12-05T06:03:44.901 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-radosgw.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.902 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: ceph-base = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.920 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: ceph-selinux = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.924 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: pyOpenSSL for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.924 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-requests for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.924 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-werkzeug for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.924 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-cherrypy for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.925 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-bcrypt for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.925 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-pecan for package: 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.925 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-mgr-dashboard.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.925 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: ceph-grafana-dashboards = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-mgr-dashboard-15.0.0-8014.gc529080.el7.noarch 2019-12-05T06:03:44.926 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-routes for package: 2:ceph-mgr-dashboard-15.0.0-8014.gc529080.el7.noarch 2019-12-05T06:03:44.926 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.926 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-mgr-diskprediction-local.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.926 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: scipy for package: 2:ceph-mgr-diskprediction-local-15.0.0-8014.gc529080.el7.noarch 2019-12-05T06:03:44.927 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-mgr-rook.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.927 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-kubernetes for package: 2:ceph-mgr-rook-15.0.0-8014.gc529080.el7.noarch 2019-12-05T06:03:44.927 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-mgr-ssh.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.927 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-remoto for package: 2:ceph-mgr-ssh-15.0.0-8014.gc529080.el7.noarch 2019-12-05T06:03:44.927 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-radosgw.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.927 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: ceph-selinux = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.928 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: librgw2 = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.928 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.928 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.928 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.928 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-test.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.928 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: ceph-common = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.929 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: xmlstarlet for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.929 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: jq for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.929 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: socat for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.929 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.929 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.929 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-2.el7 will be installed 2019-12-05T06:03:44.930 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: elfutils-libelf(x86-64) = 0.176-2.el7 for package: elfutils-libelf-devel-0.176-2.el7.x86_64 2019-12-05T06:03:44.930 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: librgw2 = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.930 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: librdkafka.so.1()(64bit) for package: 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.931 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: librabbitmq.so.4()(64bit) for package: 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.931 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: liboath.so.0()(64bit) for package: 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.931 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-test.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.932 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: ceph-common = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.934 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: xmlstarlet for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.934 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.el7.x86_64 2019-12-05T06:03:44.935 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: jq for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.936 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: socat for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.937 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libleveldb.so.1()(64bit) for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.938 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libradosstriper.so.1()(64bit) for package: 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.939 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package elfutils-libelf-devel.x86_64 0:0.176-2.el7 will be installed 2019-12-05T06:03:44.940 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: elfutils-libelf(x86-64) = 0.176-2.el7 for package: elfutils-libelf-devel-0.176-2.el7.x86_64 2019-12-05T06:03:44.942 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2019-12-05T06:03:44.944 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libcephfs-devel.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.944 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: librados-devel = 2:15.0.0-8014.gc529080.el7 for package: 2:libcephfs-devel-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.946 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libcephfs2.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.949 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2019-12-05T06:03:44.950 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package librados2.x86_64 2:15.0.0-8014.gc529080.el7 will be an update 2019-12-05T06:03:44.953 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: pkgconfig(zlib) for package: elfutils-libelf-devel-0.176-2.el7.x86_64 2019-12-05T06:03:44.954 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.957 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2019-12-05T06:03:44.957 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package librbd1.x86_64 2:15.0.0-8014.gc529080.el7 will be an update 2019-12-05T06:03:44.959 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-19.el7 will be installed 2019-12-05T06:03:44.960 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package flex.x86_64 0:2.5.37-6.el7 will be installed 2019-12-05T06:03:44.961 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-devel-1.0.2k-19.el7.x86_64 2019-12-05T06:03:44.962 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libcephfs-devel.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.962 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: librados-devel = 2:15.0.0-8014.gc529080.el7 for package: 2:libcephfs-devel-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.965 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libcephfs2.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.966 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64 2019-12-05T06:03:44.967 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package librados2.x86_64 1:10.2.5-4.el7 will be updated 2019-12-05T06:03:44.969 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package librados2.x86_64 2:15.0.0-8014.gc529080.el7 will be an update 2019-12-05T06:03:44.971 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-ceph-compat.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.972 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-cephfs = 2:15.0.0-8014.gc529080.el7 for package: 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.973 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: liblttng-ust.so.0()(64bit) for package: 2:librados2-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.974 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-rgw = 2:15.0.0-8014.gc529080.el7 for package: 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.975 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package librbd1.x86_64 1:10.2.5-4.el7 will be updated 2019-12-05T06:03:44.976 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package librbd1.x86_64 2:15.0.0-8014.gc529080.el7 will be an update 2019-12-05T06:03:44.977 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-rados = 2:15.0.0-8014.gc529080.el7 for package: 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.978 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package openssl-devel.x86_64 1:1.0.2k-19.el7 will be installed 2019-12-05T06:03:44.979 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-rbd = 2:15.0.0-8014.gc529080.el7 for package: 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.979 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-19.el7 for package: 1:openssl-devel-1.0.2k-19.el7.x86_64 2019-12-05T06:03:44.981 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python3-cephfs.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.983 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python3-ceph-argparse = 2:15.0.0-8014.gc529080.el7 for package: 2:python3-cephfs-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.984 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: krb5-devel(x86-64) for package: 1:openssl-devel-1.0.2k-19.el7.x86_64 2019-12-05T06:03:44.985 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python3-rados = 2:15.0.0-8014.gc529080.el7 for package: 2:python3-cephfs-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.987 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: 2:python3-cephfs-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.990 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-ceph-compat.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.991 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package rbd-fuse.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.991 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-cephfs = 2:15.0.0-8014.gc529080.el7 for package: 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.992 INFO:teuthology.orchestra.run.smithi076.stdout:--> Running transaction check 2019-12-05T06:03:44.993 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-base.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:44.993 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-rgw = 2:15.0.0-8014.gc529080.el7 for package: 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.995 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-rados = 2:15.0.0-8014.gc529080.el7 for package: 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.997 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-rbd = 2:15.0.0-8014.gc529080.el7 for package: 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:44.999 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python3-cephfs.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.002 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python3-ceph-argparse = 2:15.0.0-8014.gc529080.el7 for package: 2:python3-cephfs-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.017 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python3-rados = 2:15.0.0-8014.gc529080.el7 for package: 2:python3-cephfs-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.019 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-common.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.021 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-ceph-argparse = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.022 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-ceph-common = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.022 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.022 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.022 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.022 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-grafana-dashboards.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.023 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-mds.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.023 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-mon.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.023 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libpython3.6m.so.1.0()(64bit) for package: 2:python3-cephfs-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.023 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package rbd-fuse.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.023 INFO:teuthology.orchestra.run.smithi016.stdout:--> Running transaction check 2019-12-05T06:03:45.023 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-base.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.024 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-osd.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.024 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-common.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.025 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package ceph-selinux.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.029 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-ceph-argparse = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.029 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package elfutils-libelf.x86_64 0:0.172-2.el7 will be updated 2019-12-05T06:03:45.030 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-ceph-common = 2:15.0.0-8014.gc529080.el7 for package: 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.031 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: elfutils-libelf(x86-64) = 0.172-2.el7 for package: elfutils-0.172-2.el7.x86_64 2019-12-05T06:03:45.032 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-prettytable for package: 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.033 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: elfutils-libelf(x86-64) = 0.172-2.el7 for package: elfutils-libs-0.172-2.el7.x86_64 2019-12-05T06:03:45.033 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libbabeltrace-ctf.so.1()(64bit) for package: 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.034 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package elfutils-libelf.x86_64 0:0.176-2.el7 will be an update 2019-12-05T06:03:45.034 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package jq.x86_64 0:1.5-1.el7 will be installed 2019-12-05T06:03:45.035 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 2019-12-05T06:03:45.036 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libbabeltrace.so.1()(64bit) for package: 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 2019-12-05T06:03:45.036 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-grafana-dashboards.noarch 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.036 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-mds.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.036 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package krb5-devel.x86_64 0:1.15.1-37.el7_7.2 will be installed 2019-12-05T06:03:45.037 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-mon.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.038 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-osd.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.042 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-12-05T06:03:45.043 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package ceph-selinux.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.047 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-12-05T06:03:45.047 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package elfutils-libelf.x86_64 0:0.172-2.el7 will be updated 2019-12-05T06:03:45.049 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: elfutils-libelf(x86-64) = 0.172-2.el7 for package: elfutils-0.172-2.el7.x86_64 2019-12-05T06:03:45.050 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: elfutils-libelf(x86-64) = 0.172-2.el7 for package: elfutils-libs-0.172-2.el7.x86_64 2019-12-05T06:03:45.051 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-12-05T06:03:45.052 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package elfutils-libelf.x86_64 0:0.176-2.el7 will be an update 2019-12-05T06:03:45.052 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package jq.x86_64 0:1.5-1.el7 will be installed 2019-12-05T06:03:45.053 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libonig.so.2()(64bit) for package: jq-1.5-1.el7.x86_64 2019-12-05T06:03:45.057 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package krb5-devel.x86_64 0:1.15.1-37.el7_7.2 will be installed 2019-12-05T06:03:45.057 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-12-05T06:03:45.062 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed 2019-12-05T06:03:45.062 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2019-12-05T06:03:45.063 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2019-12-05T06:03:45.063 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package librados-devel.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.063 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libradosstriper1.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.063 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libverto-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-12-05T06:03:45.064 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2019-12-05T06:03:45.064 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libselinux-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-12-05T06:03:45.064 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package librgw2.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.066 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2019-12-05T06:03:45.067 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-05T06:03:45.068 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libcom_err-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-12-05T06:03:45.068 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-05T06:03:45.069 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-16.el7 will be updated 2019-12-05T06:03:45.071 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-16.el7 for package: 1:openssl-1.0.2k-16.el7.x86_64 2019-12-05T06:03:45.072 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be an update 2019-12-05T06:03:45.073 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: keyutils-libs-devel for package: krb5-devel-1.15.1-37.el7_7.2.x86_64 2019-12-05T06:03:45.073 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed 2019-12-05T06:03:45.074 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-cephfs.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.074 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2019-12-05T06:03:45.075 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2019-12-05T06:03:45.075 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-05T06:03:45.076 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package leveldb.x86_64 0:1.12.0-11.el7 will be installed 2019-12-05T06:03:45.078 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package liboath.x86_64 0:2.6.2-1.el7 will be installed 2019-12-05T06:03:45.078 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package librabbitmq.x86_64 0:0.8.0-2.el7 will be installed 2019-12-05T06:03:45.078 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package librados-devel.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.078 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libradosstriper1.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.079 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-05T06:03:45.079 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-05T06:03:45.080 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package librdkafka.x86_64 0:0.11.5-1.el7 will be installed 2019-12-05T06:03:45.080 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-05T06:03:45.081 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package librgw2.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.081 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2019-12-05T06:03:45.082 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-rados.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.082 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package lttng-ust.x86_64 0:2.4.1-4.el7 will be installed 2019-12-05T06:03:45.083 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-rbd.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.083 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-remoto.noarch 0:0.0.25-1.el7 will be installed 2019-12-05T06:03:45.084 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-execnet >= 1.2.0 for package: python-remoto-0.0.25-1.el7.noarch 2019-12-05T06:03:45.084 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: liburcu-bp.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-05T06:03:45.085 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-requests.noarch 0:2.6.0-7.el7_7 will be installed 2019-12-05T06:03:45.085 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: liburcu-cds.so.1()(64bit) for package: lttng-ust-2.4.1-4.el7.x86_64 2019-12-05T06:03:45.085 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-16.el7 will be updated 2019-12-05T06:03:45.086 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-7.el7_7.noarch 2019-12-05T06:03:45.087 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-rgw.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.088 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: openssl-libs(x86-64) = 1:1.0.2k-16.el7 for package: 1:openssl-1.0.2k-16.el7.x86_64 2019-12-05T06:03:45.088 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2019-12-05T06:03:45.088 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2019-12-05T06:03:45.089 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package openssl-libs.x86_64 1:1.0.2k-19.el7 will be an update 2019-12-05T06:03:45.089 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2019-12-05T06:03:45.089 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2019-12-05T06:03:45.090 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package pyOpenSSL.x86_64 0:0.13.1-4.el7 will be installed 2019-12-05T06:03:45.090 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2019-12-05T06:03:45.091 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-cephfs.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.091 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-cherrypy.noarch 0:3.2.2-4.el7 will be installed 2019-12-05T06:03:45.091 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-pecan.noarch 0:0.4.5-2.el7 will be installed 2019-12-05T06:03:45.092 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python2-kubernetes.noarch 0:10.0.1-1.el7 will be installed 2019-12-05T06:03:45.092 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-mako >= 0.4.0 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-05T06:03:45.093 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-05T06:03:45.093 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-simplegeneric >= 0.8 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-05T06:03:45.094 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-05T06:03:45.094 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-webob >= 1.2 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-05T06:03:45.095 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-05T06:03:45.095 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-webtest >= 1.3.1 for package: python-pecan-0.4.5-2.el7.noarch 2019-12-05T06:03:45.096 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-05T06:03:45.097 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-singledispatch for package: python-pecan-0.4.5-2.el7.noarch 2019-12-05T06:03:45.099 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-rados.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.099 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-rbd.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.099 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python3.x86_64 0:3.6.8-10.el7 will be installed 2019-12-05T06:03:45.099 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python3-setuptools for package: python3-3.6.8-10.el7.x86_64 2019-12-05T06:03:45.100 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-remoto.noarch 0:0.0.25-1.el7 will be installed 2019-12-05T06:03:45.100 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-execnet >= 1.2.0 for package: python-remoto-0.0.25-1.el7.noarch 2019-12-05T06:03:45.100 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python3-pip for package: python3-3.6.8-10.el7.x86_64 2019-12-05T06:03:45.101 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-requests.noarch 0:2.6.0-7.el7_7 will be installed 2019-12-05T06:03:45.101 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-urllib3 >= 1.10.2-1 for package: python-requests-2.6.0-7.el7_7.noarch 2019-12-05T06:03:45.102 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python3-ceph-argparse.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.102 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed 2019-12-05T06:03:45.102 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-rgw.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.103 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-routes.noarch 0:1.13-2.el7 will be installed 2019-12-05T06:03:45.103 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-repoze-lru for package: python-routes-1.13-2.el7.noarch 2019-12-05T06:03:45.104 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-werkzeug.noarch 0:0.9.1-2.el7 will be installed 2019-12-05T06:03:45.105 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python2-bcrypt.x86_64 0:3.1.6-2.el7 will be installed 2019-12-05T06:03:45.105 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python2-six for package: python2-bcrypt-3.1.6-2.el7.x86_64 2019-12-05T06:03:45.106 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python2-kubernetes.noarch 0:10.0.1-1.el7 will be installed 2019-12-05T06:03:45.107 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python3-rados.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.107 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2019-12-05T06:03:45.108 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-certifi for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-05T06:03:45.109 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-google-auth for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-05T06:03:45.110 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-requests-oauthlib for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-05T06:03:45.111 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2019-12-05T06:03:45.112 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-websocket-client for package: python2-kubernetes-10.0.1-1.el7.noarch 2019-12-05T06:03:45.112 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-05T06:03:45.113 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python3.x86_64 0:3.6.8-10.el7 will be installed 2019-12-05T06:03:45.114 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python3-setuptools for package: python3-3.6.8-10.el7.x86_64 2019-12-05T06:03:45.115 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-05T06:03:45.115 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2019-12-05T06:03:45.115 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python3-pip for package: python3-3.6.8-10.el7.x86_64 2019-12-05T06:03:45.116 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed 2019-12-05T06:03:45.117 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python3-ceph-argparse.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.117 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python3-libs.x86_64 0:3.6.8-10.el7 will be installed 2019-12-05T06:03:45.119 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed 2019-12-05T06:03:45.120 INFO:teuthology.orchestra.run.smithi076.stdout:--> Running transaction check 2019-12-05T06:03:45.120 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package elfutils.x86_64 0:0.172-2.el7 will be updated 2019-12-05T06:03:45.122 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package elfutils.x86_64 0:0.176-2.el7 will be an update 2019-12-05T06:03:45.122 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python3-rados.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.123 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package scipy.x86_64 0:0.12.1-6.el7 will be installed 2019-12-05T06:03:45.130 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: f2py for package: scipy-0.12.1-6.el7.x86_64 2019-12-05T06:03:45.130 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libumfpack.so.5()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-05T06:03:45.130 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package elfutils-libs.x86_64 0:0.172-2.el7 will be updated 2019-12-05T06:03:45.130 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package elfutils-libs.x86_64 0:0.176-2.el7 will be an update 2019-12-05T06:03:45.130 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2019-12-05T06:03:45.130 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2019-12-05T06:03:45.131 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-16.el7 will be installed 2019-12-05T06:03:45.131 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libamd.so.2()(64bit) for package: scipy-0.12.1-6.el7.x86_64 2019-12-05T06:03:45.132 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package socat.x86_64 0:1.7.3.2-2.el7 will be installed 2019-12-05T06:03:45.139 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package xmlstarlet.x86_64 0:1.6.1-1.el7 will be installed 2019-12-05T06:03:45.139 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libcom_err(x86-64) = 1.42.9-16.el7 for package: libcom_err-devel-1.42.9-16.el7.x86_64 2019-12-05T06:03:45.141 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package zlib-devel.x86_64 0:1.2.7-18.el7 will be installed 2019-12-05T06:03:45.141 INFO:teuthology.orchestra.run.smithi016.stdout:--> Running transaction check 2019-12-05T06:03:45.141 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package elfutils.x86_64 0:0.172-2.el7 will be updated 2019-12-05T06:03:45.141 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package elfutils.x86_64 0:0.176-2.el7 will be an update 2019-12-05T06:03:45.141 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libselinux-devel.x86_64 0:2.5-14.1.el7 will be installed 2019-12-05T06:03:45.141 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-12-05T06:03:45.142 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-12-05T06:03:45.142 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-12-05T06:03:45.142 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2019-12-05T06:03:45.142 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2019-12-05T06:03:45.142 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2019-12-05T06:03:45.142 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package openssl.x86_64 1:1.0.2k-16.el7 will be updated 2019-12-05T06:03:45.143 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package elfutils-libs.x86_64 0:0.172-2.el7 will be updated 2019-12-05T06:03:45.144 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package elfutils-libs.x86_64 0:0.176-2.el7 will be an update 2019-12-05T06:03:45.144 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package keyutils-libs-devel.x86_64 0:1.5.8-3.el7 will be installed 2019-12-05T06:03:45.144 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libbabeltrace.x86_64 0:1.2.4-3.el7 will be installed 2019-12-05T06:03:45.144 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package openssl.x86_64 1:1.0.2k-19.el7 will be an update 2019-12-05T06:03:45.144 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-ceph-argparse.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.144 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-ceph-common.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.145 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-execnet.noarch 0:1.2.0-7.el7 will be installed 2019-12-05T06:03:45.145 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2019-12-05T06:03:45.145 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libcom_err-devel.x86_64 0:1.42.9-16.el7 will be installed 2019-12-05T06:03:45.146 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libcom_err(x86-64) = 1.42.9-16.el7 for package: libcom_err-devel-1.42.9-16.el7.x86_64 2019-12-05T06:03:45.146 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2019-12-05T06:03:45.146 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2019-12-05T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2019-12-05T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2019-12-05T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2019-12-05T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2019-12-05T06:03:45.147 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2019-12-05T06:03:45.148 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libselinux-devel.x86_64 0:2.5-14.1.el7 will be installed 2019-12-05T06:03:45.149 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libsepol-devel(x86-64) >= 2.5-10 for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-12-05T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2019-12-05T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2019-12-05T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2019-12-05T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2019-12-05T06:03:45.149 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2019-12-05T06:03:45.150 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-05T06:03:45.150 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-05T06:03:45.152 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: pkgconfig(libsepol) for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-12-05T06:03:45.152 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: pkgconfig(libpcre) for package: libselinux-devel-2.5-14.1.el7.x86_64 2019-12-05T06:03:45.152 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-05T06:03:45.153 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2019-12-05T06:03:45.153 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2019-12-05T06:03:45.153 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2019-12-05T06:03:45.154 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed 2019-12-05T06:03:45.154 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed 2019-12-05T06:03:45.154 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libverto-devel.x86_64 0:0.2.5-4.el7 will be installed 2019-12-05T06:03:45.155 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package numpy-f2py.x86_64 1:1.7.1-13.el7 will be installed 2019-12-05T06:03:45.155 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2019-12-05T06:03:45.155 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-05T06:03:45.155 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package oniguruma.x86_64 0:5.9.5-3.el7 will be installed 2019-12-05T06:03:45.156 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package openssl.x86_64 1:1.0.2k-16.el7 will be updated 2019-12-05T06:03:45.156 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package openssl.x86_64 1:1.0.2k-19.el7 will be an update 2019-12-05T06:03:45.157 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-05T06:03:45.157 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2019-12-05T06:03:45.157 INFO:teuthology.orchestra.run.smithi076.stdout:--> Running transaction check 2019-12-05T06:03:45.157 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libcom_err.x86_64 0:1.42.9-13.el7 will be updated 2019-12-05T06:03:45.158 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-ceph-argparse.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.158 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-ceph-common.x86_64 2:15.0.0-8014.gc529080.el7 will be installed 2019-12-05T06:03:45.158 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-execnet.noarch 0:1.2.0-7.el7 will be installed 2019-12-05T06:03:45.159 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libcom_err(x86-64) = 1.42.9-13.el7 for package: libss-1.42.9-13.el7.x86_64 2019-12-05T06:03:45.160 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libcom_err(x86-64) = 1.42.9-13.el7 for package: e2fsprogs-libs-1.42.9-13.el7.x86_64 2019-12-05T06:03:45.160 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-mako.noarch 0:0.8.1-2.el7 will be installed 2019-12-05T06:03:45.160 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-markupsafe for package: python-mako-0.8.1-2.el7.noarch 2019-12-05T06:03:45.161 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: libcom_err(x86-64) = 1.42.9-13.el7 for package: e2fsprogs-1.42.9-13.el7.x86_64 2019-12-05T06:03:45.161 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-beaker for package: python-mako-0.8.1-2.el7.noarch 2019-12-05T06:03:45.161 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-prettytable.noarch 0:0.7.2-3.el7 will be installed 2019-12-05T06:03:45.161 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-repoze-lru.noarch 0:0.4-3.el7 will be installed 2019-12-05T06:03:45.162 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-simplegeneric.noarch 0:0.8-7.el7 will be installed 2019-12-05T06:03:45.163 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libcom_err.x86_64 0:1.42.9-16.el7 will be an update 2019-12-05T06:03:45.163 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2019-12-05T06:03:45.163 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2019-12-05T06:03:45.163 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-singledispatch.noarch 0:3.4.0.2-2.el7 will be installed 2019-12-05T06:03:45.163 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-urllib3.noarch 0:1.10.2-7.el7 will be installed 2019-12-05T06:03:45.164 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-webob.noarch 0:1.2.3-7.el7 will be installed 2019-12-05T06:03:45.164 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 will be installed 2019-12-05T06:03:45.164 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-webtest.noarch 0:1.3.4-6.el7 will be installed 2019-12-05T06:03:45.164 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python2-certifi.noarch 0:2018.10.15-5.el7 will be installed 2019-12-05T06:03:45.164 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python2-google-auth.noarch 1:1.1.1-5.el7 will be installed 2019-12-05T06:03:45.165 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2019-12-05T06:03:45.165 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2019-12-05T06:03:45.166 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2019-12-05T06:03:45.166 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2019-12-05T06:03:45.166 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2019-12-05T06:03:45.166 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2019-12-05T06:03:45.166 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-cachetools for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-05T06:03:45.166 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python2-pyasn1-modules for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-05T06:03:45.168 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2019-12-05T06:03:45.168 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2019-12-05T06:03:45.168 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2019-12-05T06:03:45.169 INFO:teuthology.orchestra.run.smithi076.stdout:--> Running transaction check 2019-12-05T06:03:45.169 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package e2fsprogs.x86_64 0:1.42.9-13.el7 will be updated 2019-12-05T06:03:45.169 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package e2fsprogs.x86_64 0:1.42.9-16.el7 will be an update 2019-12-05T06:03:45.169 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python2-rsa for package: 1:python2-google-auth-1.1.1-5.el7.noarch 2019-12-05T06:03:45.169 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python2-requests-oauthlib.noarch 0:0.8.0-5.el7 will be installed 2019-12-05T06:03:45.169 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python2-oauthlib for package: python2-requests-oauthlib-0.8.0-5.el7.noarch 2019-12-05T06:03:45.170 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python2-six.noarch 0:1.9.0-0.el7 will be installed 2019-12-05T06:03:45.170 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python3-pip.noarch 0:9.0.3-5.el7 will be installed 2019-12-05T06:03:45.172 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package e2fsprogs-libs.x86_64 0:1.42.9-13.el7 will be updated 2019-12-05T06:03:45.172 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package e2fsprogs-libs.x86_64 0:1.42.9-16.el7 will be an update 2019-12-05T06:03:45.172 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libss.x86_64 0:1.42.9-13.el7 will be updated 2019-12-05T06:03:45.172 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package libss.x86_64 0:1.42.9-16.el7 will be an update 2019-12-05T06:03:45.172 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2019-12-05T06:03:45.173 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2019-12-05T06:03:45.173 INFO:teuthology.orchestra.run.smithi076.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2019-12-05T06:03:45.173 INFO:teuthology.orchestra.run.smithi076.stdout:--> Running transaction check 2019-12-05T06:03:45.173 INFO:teuthology.orchestra.run.smithi076.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2019-12-05T06:03:45.173 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python3-setuptools.noarch 0:39.2.0-10.el7 will be installed 2019-12-05T06:03:45.173 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package suitesparse.x86_64 0:4.0.2-10.el7 will be installed 2019-12-05T06:03:45.173 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libtbbmalloc.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-05T06:03:45.174 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libtbb.so.2()(64bit) for package: suitesparse-4.0.2-10.el7.x86_64 2019-12-05T06:03:45.174 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package userspace-rcu.x86_64 0:0.7.16-1.el7 will be installed 2019-12-05T06:03:45.174 INFO:teuthology.orchestra.run.smithi016.stdout:--> Running transaction check 2019-12-05T06:03:45.174 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libcom_err.x86_64 0:1.42.9-13.el7 will be updated 2019-12-05T06:03:45.175 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libcom_err(x86-64) = 1.42.9-13.el7 for package: libss-1.42.9-13.el7.x86_64 2019-12-05T06:03:45.175 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libcom_err(x86-64) = 1.42.9-13.el7 for package: e2fsprogs-libs-1.42.9-13.el7.x86_64 2019-12-05T06:03:45.175 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: libcom_err(x86-64) = 1.42.9-13.el7 for package: e2fsprogs-1.42.9-13.el7.x86_64 2019-12-05T06:03:45.175 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libcom_err.x86_64 0:1.42.9-16.el7 will be an update 2019-12-05T06:03:45.176 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libsepol-devel.x86_64 0:2.5-10.el7 will be installed 2019-12-05T06:03:45.176 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package pcre-devel.x86_64 0:8.32-17.el7 will be installed 2019-12-05T06:03:45.177 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-beaker.noarch 0:1.5.4-10.el7 will be installed 2019-12-05T06:03:45.177 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-paste for package: python-beaker-1.5.4-10.el7.noarch 2019-12-05T06:03:45.178 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-cachetools.noarch 0:1.0.3-1.el7 will be installed 2019-12-05T06:03:45.179 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-markupsafe.x86_64 0:0.11-10.el7 will be installed 2019-12-05T06:03:45.179 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python2-oauthlib.noarch 0:2.0.1-8.el7 will be installed 2019-12-05T06:03:45.179 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-jwcrypto for package: python2-oauthlib-2.0.1-8.el7.noarch 2019-12-05T06:03:45.180 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python2-pyasn1-modules.noarch 0:0.1.9-7.el7 will be installed 2019-12-05T06:03:45.180 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python2-rsa.noarch 0:3.4.1-1.el7 will be installed 2019-12-05T06:03:45.181 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package tbb.x86_64 0:4.1-9.20130314.el7 will be installed 2019-12-05T06:03:45.181 INFO:teuthology.orchestra.run.smithi016.stdout:--> Running transaction check 2019-12-05T06:03:45.181 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package e2fsprogs.x86_64 0:1.42.9-13.el7 will be updated 2019-12-05T06:03:45.182 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package e2fsprogs.x86_64 0:1.42.9-16.el7 will be an update 2019-12-05T06:03:45.182 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package e2fsprogs-libs.x86_64 0:1.42.9-13.el7 will be updated 2019-12-05T06:03:45.182 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package e2fsprogs-libs.x86_64 0:1.42.9-16.el7 will be an update 2019-12-05T06:03:45.183 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libss.x86_64 0:1.42.9-13.el7 will be updated 2019-12-05T06:03:45.183 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package libss.x86_64 0:1.42.9-16.el7 will be an update 2019-12-05T06:03:45.184 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-jwcrypto.noarch 0:0.4.2-1.el7 will be installed 2019-12-05T06:03:45.184 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 will be installed 2019-12-05T06:03:45.184 INFO:teuthology.orchestra.run.smithi016.stdout:--> Processing Dependency: python-tempita for package: python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 2019-12-05T06:03:45.185 INFO:teuthology.orchestra.run.smithi016.stdout:--> Running transaction check 2019-12-05T06:03:45.185 INFO:teuthology.orchestra.run.smithi016.stdout:---> Package python-tempita.noarch 0:0.5.1-6.el7 will be installed 2019-12-05T06:03:46.095 INFO:teuthology.orchestra.run.smithi016.stdout:--> Finished Dependency Resolution 2019-12-05T06:03:46.096 INFO:teuthology.orchestra.run.smithi076.stdout:--> Finished Dependency Resolution 2019-12-05T06:03:46.185 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:03:46.185 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies Resolved 2019-12-05T06:03:46.188 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:03:46.188 INFO:teuthology.orchestra.run.smithi016.stdout:Dependencies Resolved 2019-12-05T06:03:46.198 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:03:46.198 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2019-12-05T06:03:46.199 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2019-12-05T06:03:46.199 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2019-12-05T06:03:46.199 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2019-12-05T06:03:46.199 INFO:teuthology.orchestra.run.smithi076.stdout: bison x86_64 3.0.4-2.el7 base 674 k 2019-12-05T06:03:46.199 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:15.0.0-8014.gc529080.el7 ceph 2.0 k 2019-12-05T06:03:46.199 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-daemon x86_64 2:15.0.0-8014.gc529080.el7 ceph 18 k 2019-12-05T06:03:46.200 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:15.0.0-8014.gc529080.el7 ceph 547 k 2019-12-05T06:03:46.200 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:15.0.0-8014.gc529080.el7 ceph 1.7 M 2019-12-05T06:03:46.200 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 3.9 M 2019-12-05T06:03:46.200 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-cloud 2019-12-05T06:03:46.200 INFO:teuthology.orchestra.run.smithi076.stdout: noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 82 k 2019-12-05T06:03:46.201 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local 2019-12-05T06:03:46.201 INFO:teuthology.orchestra.run.smithi076.stdout: noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 8.6 M 2019-12-05T06:03:46.201 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 23 k 2019-12-05T06:03:46.201 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-ssh noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 34 k 2019-12-05T06:03:46.201 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:15.0.0-8014.gc529080.el7 ceph 6.1 M 2019-12-05T06:03:46.201 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:15.0.0-8014.gc529080.el7 ceph 37 M 2019-12-05T06:03:46.201 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel x86_64 0.176-2.el7 base 39 k 2019-12-05T06:03:46.202 INFO:teuthology.orchestra.run.smithi076.stdout: flex x86_64 2.5.37-6.el7 base 293 k 2019-12-05T06:03:46.204 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:15.0.0-8014.gc529080.el7 ceph 17 k 2019-12-05T06:03:46.204 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:15.0.0-8014.gc529080.el7 ceph 483 k 2019-12-05T06:03:46.205 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel x86_64 1:1.0.2k-19.el7 base 1.5 M 2019-12-05T06:03:46.205 INFO:teuthology.orchestra.run.smithi076.stdout: python-ceph-compat x86_64 2:15.0.0-8014.gc529080.el7 ceph 2.2 k 2019-12-05T06:03:46.205 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:15.0.0-8014.gc529080.el7 ceph 140 k 2019-12-05T06:03:46.205 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:15.0.0-8014.gc529080.el7 ceph 68 k 2019-12-05T06:03:46.205 INFO:teuthology.orchestra.run.smithi076.stdout:Updating: 2019-12-05T06:03:46.205 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:15.0.0-8014.gc529080.el7 ceph 3.5 M 2019-12-05T06:03:46.206 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:15.0.0-8014.gc529080.el7 ceph 1.7 M 2019-12-05T06:03:46.206 INFO:teuthology.orchestra.run.smithi076.stdout:Installing for dependencies: 2019-12-05T06:03:46.206 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:15.0.0-8014.gc529080.el7 ceph 5.6 M 2019-12-05T06:03:46.206 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:15.0.0-8014.gc529080.el7 ceph 19 M 2019-12-05T06:03:46.206 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 18 k 2019-12-05T06:03:46.206 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:15.0.0-8014.gc529080.el7 ceph 1.9 M 2019-12-05T06:03:46.207 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:15.0.0-8014.gc529080.el7 ceph 4.3 M 2019-12-05T06:03:46.207 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:15.0.0-8014.gc529080.el7 ceph 16 M 2019-12-05T06:03:46.207 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:15.0.0-8014.gc529080.el7 ceph 20 k 2019-12-05T06:03:46.211 INFO:teuthology.orchestra.run.smithi076.stdout: jq x86_64 1.5-1.el7 epel 153 k 2019-12-05T06:03:46.211 INFO:teuthology.orchestra.run.smithi076.stdout: keyutils-libs-devel x86_64 1.5.8-3.el7 base 37 k 2019-12-05T06:03:46.211 INFO:teuthology.orchestra.run.smithi076.stdout: krb5-devel x86_64 1.15.1-37.el7_7.2 updates 272 k 2019-12-05T06:03:46.211 INFO:teuthology.orchestra.run.smithi076.stdout: leveldb x86_64 1.12.0-11.el7 epel 161 k 2019-12-05T06:03:46.211 INFO:teuthology.orchestra.run.smithi076.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2019-12-05T06:03:46.211 INFO:teuthology.orchestra.run.smithi076.stdout: libcom_err-devel x86_64 1.42.9-16.el7 base 32 k 2019-12-05T06:03:46.212 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2019-12-05T06:03:46.212 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.8.0-2.el7 base 37 k 2019-12-05T06:03:46.212 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:15.0.0-8014.gc529080.el7 ceph 153 k 2019-12-05T06:03:46.212 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:15.0.0-8014.gc529080.el7 ceph 361 k 2019-12-05T06:03:46.213 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2019-12-05T06:03:46.213 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:15.0.0-8014.gc529080.el7 ceph 5.4 M 2019-12-05T06:03:46.213 INFO:teuthology.orchestra.run.smithi076.stdout: libselinux-devel x86_64 2.5-14.1.el7 base 187 k 2019-12-05T06:03:46.213 INFO:teuthology.orchestra.run.smithi076.stdout: libsepol-devel x86_64 2.5-10.el7 base 77 k 2019-12-05T06:03:46.213 INFO:teuthology.orchestra.run.smithi076.stdout: libverto-devel x86_64 0.2.5-4.el7 base 12 k 2019-12-05T06:03:46.213 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2019-12-05T06:03:46.213 INFO:teuthology.orchestra.run.smithi076.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k 2019-12-05T06:03:46.214 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2019-12-05T06:03:46.214 INFO:teuthology.orchestra.run.smithi076.stdout: pcre-devel x86_64 8.32-17.el7 base 480 k 2019-12-05T06:03:46.214 INFO:teuthology.orchestra.run.smithi076.stdout: pyOpenSSL x86_64 0.13.1-4.el7 base 135 k 2019-12-05T06:03:46.225 INFO:teuthology.orchestra.run.smithi076.stdout: python-beaker noarch 1.5.4-10.el7 base 80 k 2019-12-05T06:03:46.225 INFO:teuthology.orchestra.run.smithi076.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2019-12-05T06:03:46.225 INFO:teuthology.orchestra.run.smithi076.stdout: python-ceph-argparse x86_64 2:15.0.0-8014.gc529080.el7 ceph 35 k 2019-12-05T06:03:46.225 INFO:teuthology.orchestra.run.smithi076.stdout: python-ceph-common x86_64 2:15.0.0-8014.gc529080.el7 ceph 42 k 2019-12-05T06:03:46.225 INFO:teuthology.orchestra.run.smithi076.stdout: python-cephfs x86_64 2:15.0.0-8014.gc529080.el7 ceph 102 k 2019-12-05T06:03:46.226 INFO:teuthology.orchestra.run.smithi076.stdout: python-cherrypy noarch 3.2.2-4.el7 base 422 k 2019-12-05T06:03:46.226 INFO:teuthology.orchestra.run.smithi076.stdout: python-execnet noarch 1.2.0-7.el7 epel 261 k 2019-12-05T06:03:46.226 INFO:teuthology.orchestra.run.smithi076.stdout: python-jwcrypto noarch 0.4.2-1.el7 base 57 k 2019-12-05T06:03:46.226 INFO:teuthology.orchestra.run.smithi076.stdout: python-mako noarch 0.8.1-2.el7 base 307 k 2019-12-05T06:03:46.226 INFO:teuthology.orchestra.run.smithi076.stdout: python-markupsafe x86_64 0.11-10.el7 base 25 k 2019-12-05T06:03:46.226 INFO:teuthology.orchestra.run.smithi076.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 base 866 k 2019-12-05T06:03:46.227 INFO:teuthology.orchestra.run.smithi076.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2019-12-05T06:03:46.227 INFO:teuthology.orchestra.run.smithi076.stdout: python-prettytable noarch 0.7.2-3.el7 base 37 k 2019-12-05T06:03:46.227 INFO:teuthology.orchestra.run.smithi076.stdout: python-rados x86_64 2:15.0.0-8014.gc529080.el7 ceph 210 k 2019-12-05T06:03:46.227 INFO:teuthology.orchestra.run.smithi076.stdout: python-rbd x86_64 2:15.0.0-8014.gc529080.el7 ceph 199 k 2019-12-05T06:03:46.227 INFO:teuthology.orchestra.run.smithi076.stdout: python-remoto noarch 0.0.25-1.el7 epel 22 k 2019-12-05T06:03:46.227 INFO:teuthology.orchestra.run.smithi076.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2019-12-05T06:03:46.227 INFO:teuthology.orchestra.run.smithi076.stdout: python-requests noarch 2.6.0-7.el7_7 updates 95 k 2019-12-05T06:03:46.228 INFO:teuthology.orchestra.run.smithi076.stdout: python-rgw x86_64 2:15.0.0-8014.gc529080.el7 ceph 77 k 2019-12-05T06:03:46.228 INFO:teuthology.orchestra.run.smithi076.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2019-12-05T06:03:46.228 INFO:teuthology.orchestra.run.smithi076.stdout: python-simplegeneric noarch 0.8-7.el7 epel 12 k 2019-12-05T06:03:46.228 INFO:teuthology.orchestra.run.smithi076.stdout: python-singledispatch noarch 3.4.0.2-2.el7 epel 18 k 2019-12-05T06:03:46.228 INFO:teuthology.orchestra.run.smithi076.stdout: python-tempita noarch 0.5.1-6.el7 base 33 k 2019-12-05T06:03:46.229 INFO:teuthology.orchestra.run.smithi076.stdout: python-urllib3 noarch 1.10.2-7.el7 base 103 k 2019-12-05T06:03:46.229 INFO:teuthology.orchestra.run.smithi076.stdout: python-webob noarch 1.2.3-7.el7 base 202 k 2019-12-05T06:03:46.229 INFO:teuthology.orchestra.run.smithi076.stdout: python-websocket-client noarch 0.56.0-3.git3c25814.el7 extras 66 k 2019-12-05T06:03:46.229 INFO:teuthology.orchestra.run.smithi076.stdout: python-webtest noarch 1.3.4-6.el7 base 102 k 2019-12-05T06:03:46.229 INFO:teuthology.orchestra.run.smithi076.stdout: python-werkzeug noarch 0.9.1-2.el7 extras 562 k 2019-12-05T06:03:46.229 INFO:teuthology.orchestra.run.smithi076.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2019-12-05T06:03:46.230 INFO:teuthology.orchestra.run.smithi076.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2019-12-05T06:03:46.230 INFO:teuthology.orchestra.run.smithi076.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2019-12-05T06:03:46.230 INFO:teuthology.orchestra.run.smithi076.stdout: python2-kubernetes noarch 10.0.1-1.el7 epel 1.1 M 2019-12-05T06:03:46.230 INFO:teuthology.orchestra.run.smithi076.stdout: python2-oauthlib noarch 2.0.1-8.el7 base 146 k 2019-12-05T06:03:46.230 INFO:teuthology.orchestra.run.smithi076.stdout: python2-pyasn1-modules noarch 0.1.9-7.el7 base 59 k 2019-12-05T06:03:46.230 INFO:teuthology.orchestra.run.smithi076.stdout: python2-requests-oauthlib 2019-12-05T06:03:46.231 INFO:teuthology.orchestra.run.smithi076.stdout: noarch 0.8.0-5.el7 base 35 k 2019-12-05T06:03:46.231 INFO:teuthology.orchestra.run.smithi076.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2019-12-05T06:03:46.231 INFO:teuthology.orchestra.run.smithi076.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2019-12-05T06:03:46.231 INFO:teuthology.orchestra.run.smithi076.stdout: python3 x86_64 3.6.8-10.el7 base 69 k 2019-12-05T06:03:46.231 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse x86_64 2:15.0.0-8014.gc529080.el7 ceph 36 k 2019-12-05T06:03:46.231 INFO:teuthology.orchestra.run.smithi076.stdout: python3-libs x86_64 3.6.8-10.el7 base 7.0 M 2019-12-05T06:03:46.231 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pip noarch 9.0.3-5.el7 base 1.8 M 2019-12-05T06:03:46.232 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:15.0.0-8014.gc529080.el7 ceph 334 k 2019-12-05T06:03:46.232 INFO:teuthology.orchestra.run.smithi076.stdout: python3-setuptools noarch 39.2.0-10.el7 base 629 k 2019-12-05T06:03:46.232 INFO:teuthology.orchestra.run.smithi076.stdout: scipy x86_64 0.12.1-6.el7 base 9.3 M 2019-12-05T06:03:46.232 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.3.2-2.el7 base 290 k 2019-12-05T06:03:46.232 INFO:teuthology.orchestra.run.smithi076.stdout: suitesparse x86_64 4.0.2-10.el7 base 928 k 2019-12-05T06:03:46.232 INFO:teuthology.orchestra.run.smithi076.stdout: tbb x86_64 4.1-9.20130314.el7 base 124 k 2019-12-05T06:03:46.233 INFO:teuthology.orchestra.run.smithi076.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2019-12-05T06:03:46.233 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-1.el7 epel 62 k 2019-12-05T06:03:46.233 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel x86_64 1.2.7-18.el7 base 50 k 2019-12-05T06:03:46.233 INFO:teuthology.orchestra.run.smithi076.stdout:Updating for dependencies: 2019-12-05T06:03:46.233 INFO:teuthology.orchestra.run.smithi076.stdout: e2fsprogs x86_64 1.42.9-16.el7 base 700 k 2019-12-05T06:03:46.233 INFO:teuthology.orchestra.run.smithi076.stdout: e2fsprogs-libs x86_64 1.42.9-16.el7 base 167 k 2019-12-05T06:03:46.233 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils x86_64 0.176-2.el7 base 305 k 2019-12-05T06:03:46.234 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf x86_64 0.176-2.el7 base 194 k 2019-12-05T06:03:46.234 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libs x86_64 0.176-2.el7 base 291 k 2019-12-05T06:03:46.234 INFO:teuthology.orchestra.run.smithi076.stdout: libcom_err x86_64 1.42.9-16.el7 base 41 k 2019-12-05T06:03:46.234 INFO:teuthology.orchestra.run.smithi076.stdout: libss x86_64 1.42.9-16.el7 base 46 k 2019-12-05T06:03:46.234 INFO:teuthology.orchestra.run.smithi076.stdout: openssl x86_64 1:1.0.2k-19.el7 base 493 k 2019-12-05T06:03:46.234 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs x86_64 1:1.0.2k-19.el7 base 1.2 M 2019-12-05T06:03:46.235 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:03:46.235 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2019-12-05T06:03:46.235 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2019-12-05T06:03:46.235 INFO:teuthology.orchestra.run.smithi076.stdout:Install 20 Packages (+77 Dependent packages) 2019-12-05T06:03:46.235 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 2 Packages (+ 9 Dependent packages) 2019-12-05T06:03:46.236 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:03:46.237 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:03:46.237 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2019-12-05T06:03:46.237 INFO:teuthology.orchestra.run.smithi016.stdout: Package Arch Version Repository Size 2019-12-05T06:03:46.237 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2019-12-05T06:03:46.237 INFO:teuthology.orchestra.run.smithi016.stdout:Installing: 2019-12-05T06:03:46.238 INFO:teuthology.orchestra.run.smithi016.stdout: bison x86_64 3.0.4-2.el7 base 674 k 2019-12-05T06:03:46.238 INFO:teuthology.orchestra.run.smithi016.stdout: ceph x86_64 2:15.0.0-8014.gc529080.el7 ceph 2.0 k 2019-12-05T06:03:46.238 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-daemon x86_64 2:15.0.0-8014.gc529080.el7 ceph 18 k 2019-12-05T06:03:46.238 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse x86_64 2:15.0.0-8014.gc529080.el7 ceph 547 k 2019-12-05T06:03:46.238 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr x86_64 2:15.0.0-8014.gc529080.el7 ceph 1.7 M 2019-12-05T06:03:46.238 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 3.9 M 2019-12-05T06:03:46.238 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-cloud 2019-12-05T06:03:46.239 INFO:teuthology.orchestra.run.smithi016.stdout: noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 82 k 2019-12-05T06:03:46.239 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local 2019-12-05T06:03:46.239 INFO:teuthology.orchestra.run.smithi016.stdout: noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 8.6 M 2019-12-05T06:03:46.239 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 23 k 2019-12-05T06:03:46.239 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-ssh noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 34 k 2019-12-05T06:03:46.245 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw x86_64 2:15.0.0-8014.gc529080.el7 ceph 6.1 M 2019-12-05T06:03:46.245 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test x86_64 2:15.0.0-8014.gc529080.el7 ceph 37 M 2019-12-05T06:03:46.245 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel x86_64 0.176-2.el7 base 39 k 2019-12-05T06:03:46.246 INFO:teuthology.orchestra.run.smithi016.stdout: flex x86_64 2.5.37-6.el7 base 293 k 2019-12-05T06:03:46.246 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel x86_64 2:15.0.0-8014.gc529080.el7 ceph 17 k 2019-12-05T06:03:46.246 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2 x86_64 2:15.0.0-8014.gc529080.el7 ceph 483 k 2019-12-05T06:03:46.246 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel x86_64 1:1.0.2k-19.el7 base 1.5 M 2019-12-05T06:03:46.246 INFO:teuthology.orchestra.run.smithi016.stdout: python-ceph-compat x86_64 2:15.0.0-8014.gc529080.el7 ceph 2.2 k 2019-12-05T06:03:46.246 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs x86_64 2:15.0.0-8014.gc529080.el7 ceph 140 k 2019-12-05T06:03:46.247 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse x86_64 2:15.0.0-8014.gc529080.el7 ceph 68 k 2019-12-05T06:03:46.247 INFO:teuthology.orchestra.run.smithi016.stdout:Updating: 2019-12-05T06:03:46.247 INFO:teuthology.orchestra.run.smithi016.stdout: librados2 x86_64 2:15.0.0-8014.gc529080.el7 ceph 3.5 M 2019-12-05T06:03:46.247 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1 x86_64 2:15.0.0-8014.gc529080.el7 ceph 1.7 M 2019-12-05T06:03:46.247 INFO:teuthology.orchestra.run.smithi016.stdout:Installing for dependencies: 2019-12-05T06:03:46.247 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base x86_64 2:15.0.0-8014.gc529080.el7 ceph 5.6 M 2019-12-05T06:03:46.247 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common x86_64 2:15.0.0-8014.gc529080.el7 ceph 19 M 2019-12-05T06:03:46.248 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards noarch 2:15.0.0-8014.gc529080.el7 ceph-noarch 18 k 2019-12-05T06:03:46.248 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds x86_64 2:15.0.0-8014.gc529080.el7 ceph 1.9 M 2019-12-05T06:03:46.248 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon x86_64 2:15.0.0-8014.gc529080.el7 ceph 4.3 M 2019-12-05T06:03:46.248 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd x86_64 2:15.0.0-8014.gc529080.el7 ceph 16 M 2019-12-05T06:03:46.248 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux x86_64 2:15.0.0-8014.gc529080.el7 ceph 20 k 2019-12-05T06:03:46.248 INFO:teuthology.orchestra.run.smithi016.stdout: jq x86_64 1.5-1.el7 epel 153 k 2019-12-05T06:03:46.249 INFO:teuthology.orchestra.run.smithi016.stdout: keyutils-libs-devel x86_64 1.5.8-3.el7 base 37 k 2019-12-05T06:03:46.249 INFO:teuthology.orchestra.run.smithi016.stdout: krb5-devel x86_64 1.15.1-37.el7_7.2 updates 272 k 2019-12-05T06:03:46.249 INFO:teuthology.orchestra.run.smithi016.stdout: leveldb x86_64 1.12.0-11.el7 epel 161 k 2019-12-05T06:03:46.249 INFO:teuthology.orchestra.run.smithi016.stdout: libbabeltrace x86_64 1.2.4-3.el7 epel 147 k 2019-12-05T06:03:46.249 INFO:teuthology.orchestra.run.smithi016.stdout: libcom_err-devel x86_64 1.42.9-16.el7 base 32 k 2019-12-05T06:03:46.249 INFO:teuthology.orchestra.run.smithi016.stdout: liboath x86_64 2.6.2-1.el7 epel 51 k 2019-12-05T06:03:46.249 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq x86_64 0.8.0-2.el7 base 37 k 2019-12-05T06:03:46.250 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel x86_64 2:15.0.0-8014.gc529080.el7 ceph 153 k 2019-12-05T06:03:46.255 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1 x86_64 2:15.0.0-8014.gc529080.el7 ceph 361 k 2019-12-05T06:03:46.256 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka x86_64 0.11.5-1.el7 epel 368 k 2019-12-05T06:03:46.256 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2 x86_64 2:15.0.0-8014.gc529080.el7 ceph 5.4 M 2019-12-05T06:03:46.256 INFO:teuthology.orchestra.run.smithi016.stdout: libselinux-devel x86_64 2.5-14.1.el7 base 187 k 2019-12-05T06:03:46.256 INFO:teuthology.orchestra.run.smithi016.stdout: libsepol-devel x86_64 2.5-10.el7 base 77 k 2019-12-05T06:03:46.256 INFO:teuthology.orchestra.run.smithi016.stdout: libverto-devel x86_64 0.2.5-4.el7 base 12 k 2019-12-05T06:03:46.256 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust x86_64 2.4.1-4.el7 epel 176 k 2019-12-05T06:03:46.257 INFO:teuthology.orchestra.run.smithi016.stdout: numpy-f2py x86_64 1:1.7.1-13.el7 base 206 k 2019-12-05T06:03:46.257 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma x86_64 5.9.5-3.el7 epel 129 k 2019-12-05T06:03:46.257 INFO:teuthology.orchestra.run.smithi016.stdout: pcre-devel x86_64 8.32-17.el7 base 480 k 2019-12-05T06:03:46.257 INFO:teuthology.orchestra.run.smithi016.stdout: pyOpenSSL x86_64 0.13.1-4.el7 base 135 k 2019-12-05T06:03:46.257 INFO:teuthology.orchestra.run.smithi016.stdout: python-beaker noarch 1.5.4-10.el7 base 80 k 2019-12-05T06:03:46.257 INFO:teuthology.orchestra.run.smithi016.stdout: python-cachetools noarch 1.0.3-1.el7 epel 22 k 2019-12-05T06:03:46.257 INFO:teuthology.orchestra.run.smithi016.stdout: python-ceph-argparse x86_64 2:15.0.0-8014.gc529080.el7 ceph 35 k 2019-12-05T06:03:46.258 INFO:teuthology.orchestra.run.smithi016.stdout: python-ceph-common x86_64 2:15.0.0-8014.gc529080.el7 ceph 42 k 2019-12-05T06:03:46.258 INFO:teuthology.orchestra.run.smithi016.stdout: python-cephfs x86_64 2:15.0.0-8014.gc529080.el7 ceph 102 k 2019-12-05T06:03:46.258 INFO:teuthology.orchestra.run.smithi016.stdout: python-cherrypy noarch 3.2.2-4.el7 base 422 k 2019-12-05T06:03:46.258 INFO:teuthology.orchestra.run.smithi016.stdout: python-execnet noarch 1.2.0-7.el7 epel 261 k 2019-12-05T06:03:46.258 INFO:teuthology.orchestra.run.smithi016.stdout: python-jwcrypto noarch 0.4.2-1.el7 base 57 k 2019-12-05T06:03:46.258 INFO:teuthology.orchestra.run.smithi016.stdout: python-mako noarch 0.8.1-2.el7 base 307 k 2019-12-05T06:03:46.259 INFO:teuthology.orchestra.run.smithi016.stdout: python-markupsafe x86_64 0.11-10.el7 base 25 k 2019-12-05T06:03:46.259 INFO:teuthology.orchestra.run.smithi016.stdout: python-paste noarch 1.7.5.1-9.20111221hg1498.el7 base 866 k 2019-12-05T06:03:46.259 INFO:teuthology.orchestra.run.smithi016.stdout: python-pecan noarch 0.4.5-2.el7 epel 255 k 2019-12-05T06:03:46.259 INFO:teuthology.orchestra.run.smithi016.stdout: python-prettytable noarch 0.7.2-3.el7 base 37 k 2019-12-05T06:03:46.259 INFO:teuthology.orchestra.run.smithi016.stdout: python-rados x86_64 2:15.0.0-8014.gc529080.el7 ceph 210 k 2019-12-05T06:03:46.259 INFO:teuthology.orchestra.run.smithi016.stdout: python-rbd x86_64 2:15.0.0-8014.gc529080.el7 ceph 199 k 2019-12-05T06:03:46.259 INFO:teuthology.orchestra.run.smithi016.stdout: python-remoto noarch 0.0.25-1.el7 epel 22 k 2019-12-05T06:03:46.260 INFO:teuthology.orchestra.run.smithi016.stdout: python-repoze-lru noarch 0.4-3.el7 epel 13 k 2019-12-05T06:03:46.260 INFO:teuthology.orchestra.run.smithi016.stdout: python-requests noarch 2.6.0-7.el7_7 updates 95 k 2019-12-05T06:03:46.260 INFO:teuthology.orchestra.run.smithi016.stdout: python-rgw x86_64 2:15.0.0-8014.gc529080.el7 ceph 77 k 2019-12-05T06:03:46.260 INFO:teuthology.orchestra.run.smithi016.stdout: python-routes noarch 1.13-2.el7 epel 640 k 2019-12-05T06:03:46.260 INFO:teuthology.orchestra.run.smithi016.stdout: python-simplegeneric noarch 0.8-7.el7 epel 12 k 2019-12-05T06:03:46.261 INFO:teuthology.orchestra.run.smithi016.stdout: python-singledispatch noarch 3.4.0.2-2.el7 epel 18 k 2019-12-05T06:03:46.261 INFO:teuthology.orchestra.run.smithi016.stdout: python-tempita noarch 0.5.1-6.el7 base 33 k 2019-12-05T06:03:46.261 INFO:teuthology.orchestra.run.smithi016.stdout: python-urllib3 noarch 1.10.2-7.el7 base 103 k 2019-12-05T06:03:46.261 INFO:teuthology.orchestra.run.smithi016.stdout: python-webob noarch 1.2.3-7.el7 base 202 k 2019-12-05T06:03:46.261 INFO:teuthology.orchestra.run.smithi016.stdout: python-websocket-client noarch 0.56.0-3.git3c25814.el7 extras 66 k 2019-12-05T06:03:46.261 INFO:teuthology.orchestra.run.smithi016.stdout: python-webtest noarch 1.3.4-6.el7 base 102 k 2019-12-05T06:03:46.262 INFO:teuthology.orchestra.run.smithi016.stdout: python-werkzeug noarch 0.9.1-2.el7 extras 562 k 2019-12-05T06:03:46.262 INFO:teuthology.orchestra.run.smithi016.stdout: python2-bcrypt x86_64 3.1.6-2.el7 epel 39 k 2019-12-05T06:03:46.262 INFO:teuthology.orchestra.run.smithi016.stdout: python2-certifi noarch 2018.10.15-5.el7 epel 12 k 2019-12-05T06:03:46.262 INFO:teuthology.orchestra.run.smithi016.stdout: python2-google-auth noarch 1:1.1.1-5.el7 epel 76 k 2019-12-05T06:03:46.262 INFO:teuthology.orchestra.run.smithi016.stdout: python2-kubernetes noarch 10.0.1-1.el7 epel 1.1 M 2019-12-05T06:03:46.262 INFO:teuthology.orchestra.run.smithi016.stdout: python2-oauthlib noarch 2.0.1-8.el7 base 146 k 2019-12-05T06:03:46.262 INFO:teuthology.orchestra.run.smithi016.stdout: python2-pyasn1-modules noarch 0.1.9-7.el7 base 59 k 2019-12-05T06:03:46.263 INFO:teuthology.orchestra.run.smithi016.stdout: python2-requests-oauthlib 2019-12-05T06:03:46.263 INFO:teuthology.orchestra.run.smithi016.stdout: noarch 0.8.0-5.el7 base 35 k 2019-12-05T06:03:46.263 INFO:teuthology.orchestra.run.smithi016.stdout: python2-rsa noarch 3.4.1-1.el7 epel 67 k 2019-12-05T06:03:46.263 INFO:teuthology.orchestra.run.smithi016.stdout: python2-six noarch 1.9.0-0.el7 epel 2.9 k 2019-12-05T06:03:46.263 INFO:teuthology.orchestra.run.smithi016.stdout: python3 x86_64 3.6.8-10.el7 base 69 k 2019-12-05T06:03:46.263 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse x86_64 2:15.0.0-8014.gc529080.el7 ceph 36 k 2019-12-05T06:03:46.263 INFO:teuthology.orchestra.run.smithi016.stdout: python3-libs x86_64 3.6.8-10.el7 base 7.0 M 2019-12-05T06:03:46.264 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pip noarch 9.0.3-5.el7 base 1.8 M 2019-12-05T06:03:46.264 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados x86_64 2:15.0.0-8014.gc529080.el7 ceph 334 k 2019-12-05T06:03:46.264 INFO:teuthology.orchestra.run.smithi016.stdout: python3-setuptools noarch 39.2.0-10.el7 base 629 k 2019-12-05T06:03:46.264 INFO:teuthology.orchestra.run.smithi016.stdout: scipy x86_64 0.12.1-6.el7 base 9.3 M 2019-12-05T06:03:46.266 INFO:teuthology.orchestra.run.smithi016.stdout: socat x86_64 1.7.3.2-2.el7 base 290 k 2019-12-05T06:03:46.266 INFO:teuthology.orchestra.run.smithi016.stdout: suitesparse x86_64 4.0.2-10.el7 base 928 k 2019-12-05T06:03:46.266 INFO:teuthology.orchestra.run.smithi016.stdout: tbb x86_64 4.1-9.20130314.el7 base 124 k 2019-12-05T06:03:46.266 INFO:teuthology.orchestra.run.smithi016.stdout: userspace-rcu x86_64 0.7.16-1.el7 epel 73 k 2019-12-05T06:03:46.267 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet x86_64 1.6.1-1.el7 epel 62 k 2019-12-05T06:03:46.267 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel x86_64 1.2.7-18.el7 base 50 k 2019-12-05T06:03:46.267 INFO:teuthology.orchestra.run.smithi016.stdout:Updating for dependencies: 2019-12-05T06:03:46.267 INFO:teuthology.orchestra.run.smithi016.stdout: e2fsprogs x86_64 1.42.9-16.el7 base 700 k 2019-12-05T06:03:46.267 INFO:teuthology.orchestra.run.smithi016.stdout: e2fsprogs-libs x86_64 1.42.9-16.el7 base 167 k 2019-12-05T06:03:46.267 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils x86_64 0.176-2.el7 base 305 k 2019-12-05T06:03:46.267 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf x86_64 0.176-2.el7 base 194 k 2019-12-05T06:03:46.268 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libs x86_64 0.176-2.el7 base 291 k 2019-12-05T06:03:46.268 INFO:teuthology.orchestra.run.smithi016.stdout: libcom_err x86_64 1.42.9-16.el7 base 41 k 2019-12-05T06:03:46.268 INFO:teuthology.orchestra.run.smithi016.stdout: libss x86_64 1.42.9-16.el7 base 46 k 2019-12-05T06:03:46.268 INFO:teuthology.orchestra.run.smithi016.stdout: openssl x86_64 1:1.0.2k-19.el7 base 493 k 2019-12-05T06:03:46.268 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs x86_64 1:1.0.2k-19.el7 base 1.2 M 2019-12-05T06:03:46.268 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:03:46.269 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction Summary 2019-12-05T06:03:46.269 INFO:teuthology.orchestra.run.smithi016.stdout:================================================================================ 2019-12-05T06:03:46.269 INFO:teuthology.orchestra.run.smithi016.stdout:Install 20 Packages (+77 Dependent packages) 2019-12-05T06:03:46.269 INFO:teuthology.orchestra.run.smithi016.stdout:Upgrade 2 Packages (+ 9 Dependent packages) 2019-12-05T06:03:46.269 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:03:46.269 INFO:teuthology.orchestra.run.smithi016.stdout:Total download size: 153 M 2019-12-05T06:03:46.269 INFO:teuthology.orchestra.run.smithi016.stdout:Downloading packages: 2019-12-05T06:03:46.270 INFO:teuthology.orchestra.run.smithi016.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-05T06:03:46.270 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 153 M 2019-12-05T06:03:46.270 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading packages: 2019-12-05T06:03:46.270 INFO:teuthology.orchestra.run.smithi076.stdout:Delta RPMs disabled because /usr/bin/applydeltarpm not installed. 2019-12-05T06:03:55.273 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2019-12-05T06:03:55.273 INFO:teuthology.orchestra.run.smithi076.stdout:Total 17 MB/s | 153 MB 00:09 2019-12-05T06:03:55.396 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2019-12-05T06:03:55.721 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2019-12-05T06:03:56.289 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded 2019-12-05T06:03:56.289 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2019-12-05T06:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout:-------------------------------------------------------------------------------- 2019-12-05T06:03:56.744 INFO:teuthology.orchestra.run.smithi016.stdout:Total 14 MB/s | 153 MB 00:10 2019-12-05T06:03:56.866 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction check 2019-12-05T06:03:57.190 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction test 2019-12-05T06:03:57.205 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : libcom_err-1.42.9-16.el7.x86_64 1/119 2019-12-05T06:03:57.585 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : 1:openssl-libs-1.0.2k-19.el7.x86_64 2/119 2019-12-05T06:03:57.756 INFO:teuthology.orchestra.run.smithi016.stdout:Transaction test succeeded 2019-12-05T06:03:57.756 INFO:teuthology.orchestra.run.smithi016.stdout:Running transaction 2019-12-05T06:03:57.906 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.2-1.el7.x86_64 3/119 2019-12-05T06:03:58.213 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : leveldb-1.12.0-11.el7.x86_64 4/119 2019-12-05T06:03:58.553 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 5/119 2019-12-05T06:03:58.704 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : libcom_err-1.42.9-16.el7.x86_64 1/119 2019-12-05T06:03:59.034 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : 1:openssl-libs-1.0.2k-19.el7.x86_64 2/119 2019-12-05T06:03:59.325 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : liboath-2.6.2-1.el7.x86_64 3/119 2019-12-05T06:03:59.608 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : leveldb-1.12.0-11.el7.x86_64 4/119 2019-12-05T06:03:59.913 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librabbitmq-0.8.0-2.el7.x86_64 5/119 2019-12-05T06:04:00.397 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 6/119 2019-12-05T06:04:00.815 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-libs-3.6.8-10.el7.x86_64 7/119 2019-12-05T06:04:01.280 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-3.6.8-10.el7.x86_64 8/119 2019-12-05T06:04:01.870 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : librdkafka-0.11.5-1.el7.x86_64 6/119 2019-12-05T06:04:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-setuptools-39.2.0-10.el7.noarch 9/119 2019-12-05T06:04:02.544 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pip-9.0.3-5.el7.noarch 10/119 2019-12-05T06:04:02.920 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : pyOpenSSL-0.13.1-4.el7.x86_64 11/119 2019-12-05T06:04:02.978 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-libs-3.6.8-10.el7.x86_64 7/119 2019-12-05T06:04:03.202 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : elfutils-libelf-0.176-2.el7.x86_64 12/119 2019-12-05T06:04:03.457 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-3.6.8-10.el7.x86_64 8/119 2019-12-05T06:04:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python2-six-1.9.0-0.el7.noarch 13/119 2019-12-05T06:04:04.060 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 14/119 2019-12-05T06:04:04.321 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-setuptools-39.2.0-10.el7.noarch 9/119 2019-12-05T06:04:04.452 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:python-ceph-argparse-15.0.0-8014.gc529080.el7.x86_64 15/119 2019-12-05T06:04:04.742 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python3-pip-9.0.3-5.el7.noarch 10/119 2019-12-05T06:04:04.848 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 16/119 2019-12-05T06:04:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-webob-1.2.3-7.el7.noarch 17/119 2019-12-05T06:04:05.193 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pyOpenSSL-0.13.1-4.el7.x86_64 11/119 2019-12-05T06:04:05.517 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : elfutils-libelf-0.176-2.el7.x86_64 12/119 2019-12-05T06:04:05.524 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : zlib-devel-1.2.7-18.el7.x86_64 18/119 2019-12-05T06:04:05.864 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 19/119 2019-12-05T06:04:05.907 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python2-six-1.9.0-0.el7.noarch 13/119 2019-12-05T06:04:06.183 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-requests-2.6.0-7.el7_7.noarch 20/119 2019-12-05T06:04:06.249 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-werkzeug-0.9.1-2.el7.noarch 14/119 2019-12-05T06:04:06.464 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 21/119 2019-12-05T06:04:06.676 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:python-ceph-argparse-15.0.0-8014.gc529080.el7.x86_64 15/119 2019-12-05T06:04:06.805 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 22/119 2019-12-05T06:04:07.054 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-cherrypy-3.2.2-4.el7.noarch 16/119 2019-12-05T06:04:07.137 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : elfutils-libs-0.176-2.el7.x86_64 23/119 2019-12-05T06:04:07.438 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-webob-1.2.3-7.el7.noarch 17/119 2019-12-05T06:04:07.469 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:python3-ceph-argparse-15.0.0-8014.gc529080.el7.x86_6 24/119 2019-12-05T06:04:07.757 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 25/119 2019-12-05T06:04:07.800 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : zlib-devel-1.2.7-18.el7.x86_64 18/119 2019-12-05T06:04:08.088 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-urllib3-1.10.2-7.el7.noarch 19/119 2019-12-05T06:04:08.115 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : libss-1.42.9-16.el7.x86_64 26/119 2019-12-05T06:04:08.392 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-requests-2.6.0-7.el7_7.noarch 20/119 2019-12-05T06:04:08.713 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-webtest-1.3.4-6.el7.noarch 21/119 2019-12-05T06:04:08.932 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcom_err-devel-1.42.9-16.el7.x86_64 27/119 2019-12-05T06:04:09.074 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python2-bcrypt-3.1.6-2.el7.x86_64 22/119 2019-12-05T06:04:09.386 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : elfutils-libs-0.176-2.el7.x86_64 23/119 2019-12-05T06:04:09.468 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : e2fsprogs-libs-1.42.9-16.el7.x86_64 28/119 2019-12-05T06:04:09.677 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:python3-ceph-argparse-15.0.0-8014.gc529080.el7.x86_6 24/119 2019-12-05T06:04:09.819 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : e2fsprogs-1.42.9-16.el7.x86_64 29/119 2019-12-05T06:04:09.952 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : socat-1.7.3.2-2.el7.x86_64 25/119 2019-12-05T06:04:10.109 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 30/119 2019-12-05T06:04:10.276 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : libss-1.42.9-16.el7.x86_64 26/119 2019-12-05T06:04:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 31/119 2019-12-05T06:04:10.637 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libcom_err-devel-1.42.9-16.el7.x86_64 27/119 2019-12-05T06:04:10.961 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 32/119 2019-12-05T06:04:11.199 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : e2fsprogs-libs-1.42.9-16.el7.x86_64 28/119 2019-12-05T06:04:11.279 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 33/119 2019-12-05T06:04:11.500 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : e2fsprogs-1.42.9-16.el7.x86_64 29/119 2019-12-05T06:04:11.551 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 34/119 2019-12-05T06:04:11.816 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python2-certifi-2018.10.15-5.el7.noarch 30/119 2019-12-05T06:04:12.041 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 35/119 2019-12-05T06:04:12.280 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-tempita-0.5.1-6.el7.noarch 31/119 2019-12-05T06:04:12.481 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 36/119 2019-12-05T06:04:12.628 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 32/119 2019-12-05T06:04:12.826 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 37/119 2019-12-05T06:04:12.887 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-beaker-1.5.4-10.el7.noarch 33/119 2019-12-05T06:04:13.197 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 38/119 2019-12-05T06:04:13.233 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-cachetools-1.0.3-1.el7.noarch 34/119 2019-12-05T06:04:13.739 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : tbb-4.1-9.20130314.el7.x86_64 35/119 2019-12-05T06:04:14.108 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 39/119 2019-12-05T06:04:14.109 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-daemon-15.0.0-8014.gc529080.el7.x86_64 40/119 2019-12-05T06:04:14.109 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : suitesparse-4.0.2-10.el7.x86_64 36/119 2019-12-05T06:04:14.489 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libbabeltrace-1.2.4-3.el7.x86_64 37/119 2019-12-05T06:04:14.633 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:python-ceph-common-15.0.0-8014.gc529080.el7.x86_64 41/119 2019-12-05T06:04:14.888 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 38/119 2019-12-05T06:04:15.063 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 42/119 2019-12-05T06:04:15.522 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python2-rsa-3.4.1-1.el7.noarch 39/119 2019-12-05T06:04:15.805 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-daemon-15.0.0-8014.gc529080.el7.x86_64 40/119 2019-12-05T06:04:16.289 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:python-ceph-common-15.0.0-8014.gc529080.el7.x86_64 41/119 2019-12-05T06:04:16.676 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : pcre-devel-8.32-17.el7.x86_64 42/119 2019-12-05T06:04:16.807 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 43/119 2019-12-05T06:04:17.251 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : scipy-0.12.1-6.el7.x86_64 44/119 2019-12-05T06:04:17.546 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 45/119 2019-12-05T06:04:17.935 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 46/119 2019-12-05T06:04:18.310 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 47/119 2019-12-05T06:04:18.455 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 1:numpy-f2py-1.7.1-13.el7.x86_64 43/119 2019-12-05T06:04:18.844 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : scipy-0.12.1-6.el7.x86_64 44/119 2019-12-05T06:04:19.133 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-prettytable-0.7.2-3.el7.noarch 45/119 2019-12-05T06:04:19.208 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 48/119 2019-12-05T06:04:19.498 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python2-pyasn1-modules-0.1.9-7.el7.noarch 46/119 2019-12-05T06:04:19.530 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 49/119 2019-12-05T06:04:19.846 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 1:python2-google-auth-1.1.1-5.el7.noarch 47/119 2019-12-05T06:04:20.257 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-jwcrypto-0.4.2-1.el7.noarch 48/119 2019-12-05T06:04:20.561 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python2-oauthlib-2.0.1-8.el7.noarch 49/119 2019-12-05T06:04:20.845 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 50/119 2019-12-05T06:04:21.259 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python2-kubernetes-10.0.1-1.el7.noarch 51/119 2019-12-05T06:04:21.555 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-1.el7.x86_64 52/119 2019-12-05T06:04:21.892 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python2-requests-oauthlib-0.8.0-5.el7.noarch 50/119 2019-12-05T06:04:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 53/119 2019-12-05T06:04:22.250 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-execnet-1.2.0-7.el7.noarch 54/119 2019-12-05T06:04:22.265 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python2-kubernetes-10.0.1-1.el7.noarch 51/119 2019-12-05T06:04:22.501 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-remoto-0.0.25-1.el7.noarch 55/119 2019-12-05T06:04:22.583 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : xmlstarlet-1.6.1-1.el7.x86_64 52/119 2019-12-05T06:04:22.811 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-grafana-dashboards-15.0.0-8014.gc529080.el7.noa 56/119 2019-12-05T06:04:22.966 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-singledispatch-3.4.0.2-2.el7.noarch 53/119 2019-12-05T06:04:23.072 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 57/119 2019-12-05T06:04:23.321 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-execnet-1.2.0-7.el7.noarch 54/119 2019-12-05T06:04:23.395 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 58/119 2019-12-05T06:04:23.571 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-remoto-0.0.25-1.el7.noarch 55/119 2019-12-05T06:04:23.886 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-grafana-dashboards-15.0.0-8014.gc529080.el7.noa 56/119 2019-12-05T06:04:24.142 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : keyutils-libs-devel-1.5.8-3.el7.x86_64 57/119 2019-12-05T06:04:24.348 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 59/119 2019-12-05T06:04:24.449 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-simplegeneric-0.8-7.el7.noarch 58/119 2019-12-05T06:04:24.697 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : jq-1.5-1.el7.x86_64 60/119 2019-12-05T06:04:24.811 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : oniguruma-5.9.5-3.el7.x86_64 59/119 2019-12-05T06:04:25.056 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 61/119 2019-12-05T06:04:25.168 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : jq-1.5-1.el7.x86_64 60/119 2019-12-05T06:04:25.515 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : userspace-rcu-0.7.16-1.el7.x86_64 61/119 2019-12-05T06:04:25.787 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 62/119 2019-12-05T06:04:26.179 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : 2:librados2-15.0.0-8014.gc529080.el7.x86_64 63/119 2019-12-05T06:04:26.241 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : lttng-ust-2.4.1-4.el7.x86_64 62/119 2019-12-05T06:04:26.537 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:libcephfs2-15.0.0-8014.gc529080.el7.x86_64 64/119 2019-12-05T06:04:26.675 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : 2:librados2-15.0.0-8014.gc529080.el7.x86_64 63/119 2019-12-05T06:04:27.047 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:python-rados-15.0.0-8014.gc529080.el7.x86_64 65/119 2019-12-05T06:04:27.047 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:libcephfs2-15.0.0-8014.gc529080.el7.x86_64 64/119 2019-12-05T06:04:27.530 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:python-rados-15.0.0-8014.gc529080.el7.x86_64 65/119 2019-12-05T06:04:27.956 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : 2:librbd1-15.0.0-8014.gc529080.el7.x86_64 66/119 2019-12-05T06:04:28.273 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:librgw2-15.0.0-8014.gc529080.el7.x86_64 67/119 2019-12-05T06:04:28.448 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : 2:librbd1-15.0.0-8014.gc529080.el7.x86_64 66/119 2019-12-05T06:04:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:python-rgw-15.0.0-8014.gc529080.el7.x86_64 68/119 2019-12-05T06:04:28.753 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:librgw2-15.0.0-8014.gc529080.el7.x86_64 67/119 2019-12-05T06:04:29.090 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:python-rgw-15.0.0-8014.gc529080.el7.x86_64 68/119 2019-12-05T06:04:29.091 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:python-rbd-15.0.0-8014.gc529080.el7.x86_64 69/119 2019-12-05T06:04:29.379 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:python-rbd-15.0.0-8014.gc529080.el7.x86_64 69/119 2019-12-05T06:04:29.455 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:python-cephfs-15.0.0-8014.gc529080.el7.x86_64 70/119 2019-12-05T06:04:29.749 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:python-cephfs-15.0.0-8014.gc529080.el7.x86_64 70/119 2019-12-05T06:04:31.869 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:libradosstriper1-15.0.0-8014.gc529080.el7.x86_64 71/119 2019-12-05T06:04:32.292 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:libradosstriper1-15.0.0-8014.gc529080.el7.x86_64 71/119 2019-12-05T06:04:32.970 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 72/119 2019-12-05T06:04:33.425 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 72/119 2019-12-05T06:04:33.522 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-base-15.0.0-8014.gc529080.el7.x86_64 73/119 2019-12-05T06:04:33.974 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-base-15.0.0-8014.gc529080.el7.x86_64 73/119 2019-12-05T06:04:44.084 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-selinux-15.0.0-8014.gc529080.el7.x86_64 74/119 2019-12-05T06:04:44.765 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-osd-15.0.0-8014.gc529080.el7.x86_64 75/119 2019-12-05T06:04:44.872 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-selinux-15.0.0-8014.gc529080.el7.x86_64 74/119 2019-12-05T06:04:45.514 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-osd-15.0.0-8014.gc529080.el7.x86_64 75/119 2019-12-05T06:04:45.664 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-mds-15.0.0-8014.gc529080.el7.x86_64 76/119 2019-12-05T06:04:46.089 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-mon-15.0.0-8014.gc529080.el7.x86_64 77/119 2019-12-05T06:04:46.347 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-mds-15.0.0-8014.gc529080.el7.x86_64 76/119 2019-12-05T06:04:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:librados-devel-15.0.0-8014.gc529080.el7.x86_64 78/119 2019-12-05T06:04:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:python3-rados-15.0.0-8014.gc529080.el7.x86_64 79/119 2019-12-05T06:04:46.733 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-mon-15.0.0-8014.gc529080.el7.x86_64 77/119 2019-12-05T06:04:47.151 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 80/119 2019-12-05T06:04:47.262 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:librados-devel-15.0.0-8014.gc529080.el7.x86_64 78/119 2019-12-05T06:04:47.481 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-routes-1.13-2.el7.noarch 81/119 2019-12-05T06:04:47.533 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:python3-rados-15.0.0-8014.gc529080.el7.x86_64 79/119 2019-12-05T06:04:47.888 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 82/119 2019-12-05T06:04:47.954 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-repoze-lru-0.4-3.el7.noarch 80/119 2019-12-05T06:04:48.313 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-routes-1.13-2.el7.noarch 81/119 2019-12-05T06:04:48.348 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-mako-0.8.1-2.el7.noarch 83/119 2019-12-05T06:04:48.679 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-markupsafe-0.11-10.el7.x86_64 82/119 2019-12-05T06:04:48.972 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 84/119 2019-12-05T06:04:49.129 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-mako-0.8.1-2.el7.noarch 83/119 2019-12-05T06:04:49.783 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : python-pecan-0.4.5-2.el7.noarch 84/119 2019-12-05T06:04:49.840 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 85/119 2019-12-05T06:04:50.135 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 86/119 2019-12-05T06:04:50.276 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 85/119 2019-12-05T06:04:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 87/119 2019-12-05T06:04:50.580 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libverto-devel-0.2.5-4.el7.x86_64 86/119 2019-12-05T06:04:50.966 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libselinux-devel-2.5-14.1.el7.x86_64 88/119 2019-12-05T06:04:51.001 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libsepol-devel-2.5-10.el7.x86_64 87/119 2019-12-05T06:04:51.373 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : libselinux-devel-2.5-14.1.el7.x86_64 88/119 2019-12-05T06:04:51.735 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : krb5-devel-1.15.1-37.el7_7.2.x86_64 89/119 2019-12-05T06:04:52.065 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 1:openssl-devel-1.0.2k-19.el7.x86_64 90/119 2019-12-05T06:04:52.139 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : krb5-devel-1.15.1-37.el7_7.2.x86_64 89/119 2019-12-05T06:04:52.805 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 1:openssl-devel-1.0.2k-19.el7.x86_64 90/119 2019-12-05T06:04:53.642 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-mgr-ssh-15.0.0-8014.gc529080.el7.noarch 91/119 2019-12-05T06:04:53.924 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-mgr-diskprediction-local-15.0.0-8014.gc529080.e 92/119 2019-12-05T06:04:54.451 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-mgr-ssh-15.0.0-8014.gc529080.el7.noarch 91/119 2019-12-05T06:04:54.788 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-mgr-diskprediction-local-15.0.0-8014.gc529080.e 92/119 2019-12-05T06:04:55.396 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-15.0.0-8014.gc529080.el7.x86_64 93/119 2019-12-05T06:04:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-mgr-dashboard-15.0.0-8014.gc529080.el7.noarch 94/119 2019-12-05T06:04:55.895 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-15.0.0-8014.gc529080.el7.x86_64 93/119 2019-12-05T06:04:56.191 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-mgr-rook-15.0.0-8014.gc529080.el7.noarch 95/119 2019-12-05T06:04:56.302 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-mgr-dashboard-15.0.0-8014.gc529080.el7.noarch 94/119 2019-12-05T06:04:56.579 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-15.0.0-8014.gc529080.e 96/119 2019-12-05T06:04:56.673 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-mgr-rook-15.0.0-8014.gc529080.el7.noarch 95/119 2019-12-05T06:04:56.883 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:python3-cephfs-15.0.0-8014.gc529080.el7.x86_64 97/119 2019-12-05T06:04:57.025 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-mgr-diskprediction-cloud-15.0.0-8014.gc529080.e 96/119 2019-12-05T06:04:57.540 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:python3-cephfs-15.0.0-8014.gc529080.el7.x86_64 97/119 2019-12-05T06:04:57.780 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:libcephfs-devel-15.0.0-8014.gc529080.el7.x86_64 98/119 2019-12-05T06:04:58.486 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:libcephfs-devel-15.0.0-8014.gc529080.el7.x86_64 98/119 2019-12-05T06:05:01.709 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 99/119 2019-12-05T06:05:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 100/119 2019-12-05T06:05:02.267 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 101/119 2019-12-05T06:05:02.613 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 99/119 2019-12-05T06:05:02.619 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:rbd-fuse-15.0.0-8014.gc529080.el7.x86_64 102/119 2019-12-05T06:05:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : 2:ceph-fuse-15.0.0-8014.gc529080.el7.x86_64 103/119 2019-12-05T06:05:03.103 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 100/119 2019-12-05T06:05:03.307 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : elfutils-0.176-2.el7.x86_64 104/119 2019-12-05T06:05:03.354 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 101/119 2019-12-05T06:05:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : elfutils-libelf-devel-0.176-2.el7.x86_64 105/119 2019-12-05T06:05:03.759 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:rbd-fuse-15.0.0-8014.gc529080.el7.x86_64 102/119 2019-12-05T06:05:04.093 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : 2:ceph-fuse-15.0.0-8014.gc529080.el7.x86_64 103/119 2019-12-05T06:05:04.127 INFO:teuthology.orchestra.run.smithi076.stdout: Updating : 1:openssl-1.0.2k-19.el7.x86_64 106/119 2019-12-05T06:05:04.414 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : elfutils-0.176-2.el7.x86_64 104/119 2019-12-05T06:05:04.794 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : elfutils-libelf-devel-0.176-2.el7.x86_64 105/119 2019-12-05T06:05:05.253 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : bison-3.0.4-2.el7.x86_64 107/119 2019-12-05T06:05:05.297 INFO:teuthology.orchestra.run.smithi016.stdout: Updating : 1:openssl-1.0.2k-19.el7.x86_64 106/119 2019-12-05T06:05:05.552 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : flex-2.5.37-6.el7.x86_64 108/119 2019-12-05T06:05:05.722 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : bison-3.0.4-2.el7.x86_64 107/119 2019-12-05T06:05:05.815 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : e2fsprogs-1.42.9-13.el7.x86_64 109/119 2019-12-05T06:05:05.990 INFO:teuthology.orchestra.run.smithi016.stdout: Installing : flex-2.5.37-6.el7.x86_64 108/119 2019-12-05T06:05:06.059 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : elfutils-0.172-2.el7.x86_64 110/119 2019-12-05T06:05:06.274 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : e2fsprogs-1.42.9-13.el7.x86_64 109/119 2019-12-05T06:05:06.325 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : 1:openssl-1.0.2k-16.el7.x86_64 111/119 2019-12-05T06:05:06.533 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-0.172-2.el7.x86_64 110/119 2019-12-05T06:05:06.618 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : 1:openssl-libs-1.0.2k-16.el7.x86_64 112/119 2019-12-05T06:05:06.816 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : 1:openssl-1.0.2k-16.el7.x86_64 111/119 2019-12-05T06:05:06.884 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : elfutils-libs-0.172-2.el7.x86_64 113/119 2019-12-05T06:05:07.107 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : 1:openssl-libs-1.0.2k-16.el7.x86_64 112/119 2019-12-05T06:05:07.142 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : e2fsprogs-libs-1.42.9-13.el7.x86_64 114/119 2019-12-05T06:05:07.402 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libss-1.42.9-13.el7.x86_64 115/119 2019-12-05T06:05:07.658 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 116/119 2019-12-05T06:05:07.678 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-libs-0.172-2.el7.x86_64 113/119 2019-12-05T06:05:07.925 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 117/119 2019-12-05T06:05:07.961 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : e2fsprogs-libs-1.42.9-13.el7.x86_64 114/119 2019-12-05T06:05:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libcom_err-1.42.9-13.el7.x86_64 118/119 2019-12-05T06:05:08.243 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libss-1.42.9-13.el7.x86_64 115/119 2019-12-05T06:05:08.466 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : 1:librbd1-10.2.5-4.el7.x86_64 116/119 2019-12-05T06:05:08.563 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : elfutils-libelf-0.172-2.el7.x86_64 119/119 2019-12-05T06:05:08.731 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:python3-ceph-argparse-15.0.0-8014.gc529080.el7.x86_6 1/119 2019-12-05T06:05:08.735 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : 1:librados2-10.2.5-4.el7.x86_64 117/119 2019-12-05T06:05:08.840 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2/119 2019-12-05T06:05:08.966 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-osd-15.0.0-8014.gc529080.el7.x86_64 3/119 2019-12-05T06:05:08.990 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : libcom_err-1.42.9-13.el7.x86_64 118/119 2019-12-05T06:05:09.083 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python2-kubernetes-10.0.1-1.el7.noarch 4/119 2019-12-05T06:05:09.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-routes-1.13-2.el7.noarch 5/119 2019-12-05T06:05:09.302 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-mgr-ssh-15.0.0-8014.gc529080.el7.noarch 6/119 2019-12-05T06:05:09.335 INFO:teuthology.orchestra.run.smithi016.stdout: Cleanup : elfutils-libelf-0.172-2.el7.x86_64 119/119 2019-12-05T06:05:09.505 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:python3-ceph-argparse-15.0.0-8014.gc529080.el7.x86_6 1/119 2019-12-05T06:05:09.606 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-mgr-15.0.0-8014.gc529080.el7.x86_64 2/119 2019-12-05T06:05:09.732 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-osd-15.0.0-8014.gc529080.el7.x86_64 3/119 2019-12-05T06:05:09.734 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 7/119 2019-12-05T06:05:09.840 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python2-kubernetes-10.0.1-1.el7.noarch 4/119 2019-12-05T06:05:09.843 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 8/119 2019-12-05T06:05:09.943 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 9/119 2019-12-05T06:05:09.949 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-routes-1.13-2.el7.noarch 5/119 2019-12-05T06:05:10.061 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 10/119 2019-12-05T06:05:10.091 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-mgr-ssh-15.0.0-8014.gc529080.el7.noarch 6/119 2019-12-05T06:05:10.178 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : zlib-devel-1.2.7-18.el7.x86_64 11/119 2019-12-05T06:05:10.208 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-urllib3-1.10.2-7.el7.noarch 7/119 2019-12-05T06:05:10.295 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:python-rados-15.0.0-8014.gc529080.el7.x86_64 12/119 2019-12-05T06:05:10.301 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libsepol-devel-2.5-10.el7.x86_64 8/119 2019-12-05T06:05:10.384 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-mako-0.8.1-2.el7.noarch 9/119 2019-12-05T06:05:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-mgr-diskprediction-local-15.0.0-8014.gc529080.e 13/119 2019-12-05T06:05:10.485 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libverto-devel-0.2.5-4.el7.x86_64 10/119 2019-12-05T06:05:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 14/119 2019-12-05T06:05:10.587 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : zlib-devel-1.2.7-18.el7.x86_64 11/119 2019-12-05T06:05:10.663 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 15/119 2019-12-05T06:05:10.736 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:python-rados-15.0.0-8014.gc529080.el7.x86_64 12/119 2019-12-05T06:05:10.822 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libss-1.42.9-16.el7.x86_64 16/119 2019-12-05T06:05:10.845 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-mgr-diskprediction-local-15.0.0-8014.gc529080.e 13/119 2019-12-05T06:05:10.945 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-markupsafe-0.11-10.el7.x86_64 14/119 2019-12-05T06:05:10.964 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcom_err-devel-1.42.9-16.el7.x86_64 17/119 2019-12-05T06:05:11.062 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:python-ceph-compat-15.0.0-8014.gc529080.el7.x86_64 15/119 2019-12-05T06:05:11.098 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 18/119 2019-12-05T06:05:11.179 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libss-1.42.9-16.el7.x86_64 16/119 2019-12-05T06:05:11.215 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 19/119 2019-12-05T06:05:11.297 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcom_err-devel-1.42.9-16.el7.x86_64 17/119 2019-12-05T06:05:11.315 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-setuptools-39.2.0-10.el7.noarch 20/119 2019-12-05T06:05:11.433 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-mds-15.0.0-8014.gc529080.el7.x86_64 21/119 2019-12-05T06:05:11.439 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-webob-1.2.3-7.el7.noarch 18/119 2019-12-05T06:05:11.539 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-repoze-lru-0.4-3.el7.noarch 19/119 2019-12-05T06:05:11.541 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 22/119 2019-12-05T06:05:11.648 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-setuptools-39.2.0-10.el7.noarch 20/119 2019-12-05T06:05:11.658 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-3.6.8-10.el7.x86_64 23/119 2019-12-05T06:05:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:python3-cephfs-15.0.0-8014.gc529080.el7.x86_64 24/119 2019-12-05T06:05:11.765 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-mds-15.0.0-8014.gc529080.el7.x86_64 21/119 2019-12-05T06:05:11.851 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-15.0.0-8014.gc529080.el7.x86_64 25/119 2019-12-05T06:05:11.883 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 1:openssl-libs-1.0.2k-19.el7.x86_64 22/119 2019-12-05T06:05:11.951 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 26/119 2019-12-05T06:05:12.050 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-3.6.8-10.el7.x86_64 23/119 2019-12-05T06:05:12.069 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 27/119 2019-12-05T06:05:12.134 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:python3-cephfs-15.0.0-8014.gc529080.el7.x86_64 24/119 2019-12-05T06:05:12.194 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-0.176-2.el7.x86_64 28/119 2019-12-05T06:05:12.226 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-15.0.0-8014.gc529080.el7.x86_64 25/119 2019-12-05T06:05:12.310 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-cherrypy-3.2.2-4.el7.noarch 26/119 2019-12-05T06:05:12.311 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 29/119 2019-12-05T06:05:12.428 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libselinux-devel-2.5-14.1.el7.x86_64 30/119 2019-12-05T06:05:12.546 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:python-ceph-argparse-15.0.0-8014.gc529080.el7.x86_64 31/119 2019-12-05T06:05:12.721 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 32/119 2019-12-05T06:05:12.808 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-beaker-1.5.4-10.el7.noarch 27/119 2019-12-05T06:05:12.847 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:librados-devel-15.0.0-8014.gc529080.el7.x86_64 33/119 2019-12-05T06:05:12.919 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-0.176-2.el7.x86_64 28/119 2019-12-05T06:05:12.981 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 34/119 2019-12-05T06:05:13.078 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : userspace-rcu-0.7.16-1.el7.x86_64 29/119 2019-12-05T06:05:13.114 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 35/119 2019-12-05T06:05:13.195 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libselinux-devel-2.5-14.1.el7.x86_64 30/119 2019-12-05T06:05:13.231 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 36/119 2019-12-05T06:05:13.312 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:python-ceph-argparse-15.0.0-8014.gc529080.el7.x86_64 31/119 2019-12-05T06:05:13.332 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-grafana-dashboards-15.0.0-8014.gc529080.el7.noa 37/119 2019-12-05T06:05:13.450 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-execnet-1.2.0-7.el7.noarch 38/119 2019-12-05T06:05:13.480 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : suitesparse-4.0.2-10.el7.x86_64 32/119 2019-12-05T06:05:13.558 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:libcephfs-devel-15.0.0-8014.gc529080.el7.x86_64 39/119 2019-12-05T06:05:13.588 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:librados-devel-15.0.0-8014.gc529080.el7.x86_64 33/119 2019-12-05T06:05:13.667 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:librbd1-15.0.0-8014.gc529080.el7.x86_64 40/119 2019-12-05T06:05:13.705 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : oniguruma-5.9.5-3.el7.x86_64 34/119 2019-12-05T06:05:13.775 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pip-9.0.3-5.el7.noarch 41/119 2019-12-05T06:05:13.814 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-simplegeneric-0.8-7.el7.noarch 35/119 2019-12-05T06:05:13.867 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 42/119 2019-12-05T06:05:13.915 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : keyutils-libs-devel-1.5.8-3.el7.x86_64 36/119 2019-12-05T06:05:13.951 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:rbd-fuse-15.0.0-8014.gc529080.el7.x86_64 43/119 2019-12-05T06:05:14.040 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-grafana-dashboards-15.0.0-8014.gc529080.el7.noa 37/119 2019-12-05T06:05:14.085 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : e2fsprogs-1.42.9-16.el7.x86_64 44/119 2019-12-05T06:05:14.149 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-execnet-1.2.0-7.el7.noarch 38/119 2019-12-05T06:05:14.202 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 45/119 2019-12-05T06:05:14.258 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:libcephfs-devel-15.0.0-8014.gc529080.el7.x86_64 39/119 2019-12-05T06:05:14.328 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-1.el7.x86_64 46/119 2019-12-05T06:05:14.359 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:librbd1-15.0.0-8014.gc529080.el7.x86_64 40/119 2019-12-05T06:05:14.437 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 47/119 2019-12-05T06:05:14.451 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-pip-9.0.3-5.el7.noarch 41/119 2019-12-05T06:05:14.535 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-common-15.0.0-8014.gc529080.el7.x86_64 42/119 2019-12-05T06:05:14.669 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:rbd-fuse-15.0.0-8014.gc529080.el7.x86_64 43/119 2019-12-05T06:05:14.794 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : e2fsprogs-1.42.9-16.el7.x86_64 44/119 2019-12-05T06:05:14.903 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-singledispatch-3.4.0.2-2.el7.noarch 45/119 2019-12-05T06:05:15.012 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : xmlstarlet-1.6.1-1.el7.x86_64 46/119 2019-12-05T06:05:15.013 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 48/119 2019-12-05T06:05:15.129 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python2-requests-oauthlib-0.8.0-5.el7.noarch 47/119 2019-12-05T06:05:15.155 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 1:openssl-1.0.2k-19.el7.x86_64 49/119 2019-12-05T06:05:15.238 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-paste-1.7.5.1-9.20111221hg1498.el7.noarch 48/119 2019-12-05T06:05:15.297 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 50/119 2019-12-05T06:05:15.389 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 1:openssl-1.0.2k-19.el7.x86_64 49/119 2019-12-05T06:05:15.448 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 51/119 2019-12-05T06:05:15.505 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : scipy-0.12.1-6.el7.x86_64 50/119 2019-12-05T06:05:15.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 52/119 2019-12-05T06:05:15.639 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-jwcrypto-0.4.2-1.el7.noarch 51/119 2019-12-05T06:05:15.665 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 53/119 2019-12-05T06:05:15.749 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-werkzeug-0.9.1-2.el7.noarch 52/119 2019-12-05T06:05:15.783 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 54/119 2019-12-05T06:05:15.857 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python2-pyasn1-modules-0.1.9-7.el7.noarch 53/119 2019-12-05T06:05:15.908 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 55/119 2019-12-05T06:05:15.991 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-prettytable-0.7.2-3.el7.noarch 54/119 2019-12-05T06:05:16.025 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libs-0.176-2.el7.x86_64 56/119 2019-12-05T06:05:16.083 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : socat-1.7.3.2-2.el7.x86_64 55/119 2019-12-05T06:05:16.117 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : e2fsprogs-libs-1.42.9-16.el7.x86_64 57/119 2019-12-05T06:05:16.192 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libs-0.176-2.el7.x86_64 56/119 2019-12-05T06:05:16.217 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:python-cephfs-15.0.0-8014.gc529080.el7.x86_64 58/119 2019-12-05T06:05:16.276 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : e2fsprogs-libs-1.42.9-16.el7.x86_64 57/119 2019-12-05T06:05:16.335 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 59/119 2019-12-05T06:05:16.368 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:python-cephfs-15.0.0-8014.gc529080.el7.x86_64 58/119 2019-12-05T06:05:16.443 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:libradosstriper1-15.0.0-8014.gc529080.el7.x86_64 60/119 2019-12-05T06:05:16.477 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librabbitmq-0.8.0-2.el7.x86_64 59/119 2019-12-05T06:05:16.569 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 61/119 2019-12-05T06:05:16.603 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:libradosstriper1-15.0.0-8014.gc529080.el7.x86_64 60/119 2019-12-05T06:05:16.703 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : leveldb-1.12.0-11.el7.x86_64 62/119 2019-12-05T06:05:16.720 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 1:numpy-f2py-1.7.1-13.el7.x86_64 61/119 2019-12-05T06:05:16.836 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : jq-1.5-1.el7.x86_64 63/119 2019-12-05T06:05:16.837 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : leveldb-1.12.0-11.el7.x86_64 62/119 2019-12-05T06:05:16.955 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : jq-1.5-1.el7.x86_64 63/119 2019-12-05T06:05:16.962 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-libs-3.6.8-10.el7.x86_64 64/119 2019-12-05T06:05:17.080 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python3-libs-3.6.8-10.el7.x86_64 64/119 2019-12-05T06:05:17.096 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 65/119 2019-12-05T06:05:17.208 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python2-oauthlib-2.0.1-8.el7.noarch 65/119 2019-12-05T06:05:17.230 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 66/119 2019-12-05T06:05:17.331 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pcre-devel-8.32-17.el7.x86_64 66/119 2019-12-05T06:05:17.364 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-fuse-15.0.0-8014.gc529080.el7.x86_64 67/119 2019-12-05T06:05:17.493 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-selinux-15.0.0-8014.gc529080.el7.x86_64 68/119 2019-12-05T06:05:17.590 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 69/119 2019-12-05T06:05:17.682 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:libcephfs2-15.0.0-8014.gc529080.el7.x86_64 70/119 2019-12-05T06:05:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-mgr-dashboard-15.0.0-8014.gc529080.el7.noarch 71/119 2019-12-05T06:05:17.883 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:python-ceph-common-15.0.0-8014.gc529080.el7.x86_64 72/119 2019-12-05T06:05:17.975 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-requests-2.6.0-7.el7_7.noarch 73/119 2019-12-05T06:05:18.048 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-fuse-15.0.0-8014.gc529080.el7.x86_64 67/119 2019-12-05T06:05:18.075 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:librgw2-15.0.0-8014.gc529080.el7.x86_64 74/119 2019-12-05T06:05:18.157 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-selinux-15.0.0-8014.gc529080.el7.x86_64 68/119 2019-12-05T06:05:18.167 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : flex-2.5.37-6.el7.x86_64 75/119 2019-12-05T06:05:18.250 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python2-six-1.9.0-0.el7.noarch 69/119 2019-12-05T06:05:18.268 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-mgr-rook-15.0.0-8014.gc529080.el7.noarch 76/119 2019-12-05T06:05:18.334 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:libcephfs2-15.0.0-8014.gc529080.el7.x86_64 70/119 2019-12-05T06:05:18.377 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : pyOpenSSL-0.13.1-4.el7.x86_64 77/119 2019-12-05T06:05:18.442 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-mgr-dashboard-15.0.0-8014.gc529080.el7.noarch 71/119 2019-12-05T06:05:18.494 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:python3-rados-15.0.0-8014.gc529080.el7.x86_64 78/119 2019-12-05T06:05:18.543 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:python-ceph-common-15.0.0-8014.gc529080.el7.x86_64 72/119 2019-12-05T06:05:18.620 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 79/119 2019-12-05T06:05:18.694 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-requests-2.6.0-7.el7_7.noarch 73/119 2019-12-05T06:05:18.753 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-15.0.0-8014.gc529080.e 80/119 2019-12-05T06:05:18.786 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:librgw2-15.0.0-8014.gc529080.el7.x86_64 74/119 2019-12-05T06:05:18.862 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 81/119 2019-12-05T06:05:18.895 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : flex-2.5.37-6.el7.x86_64 75/119 2019-12-05T06:05:18.986 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : krb5-devel-1.15.1-37.el7_7.2.x86_64 82/119 2019-12-05T06:05:19.003 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-mgr-rook-15.0.0-8014.gc529080.el7.noarch 76/119 2019-12-05T06:05:19.065 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-daemon-15.0.0-8014.gc529080.el7.x86_64 83/119 2019-12-05T06:05:19.112 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : pyOpenSSL-0.13.1-4.el7.x86_64 77/119 2019-12-05T06:05:19.166 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:librados2-15.0.0-8014.gc529080.el7.x86_64 84/119 2019-12-05T06:05:19.221 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:python3-rados-15.0.0-8014.gc529080.el7.x86_64 78/119 2019-12-05T06:05:19.258 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 85/119 2019-12-05T06:05:19.359 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 86/119 2019-12-05T06:05:19.380 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-webtest-1.3.4-6.el7.noarch 79/119 2019-12-05T06:05:19.451 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 87/119 2019-12-05T06:05:19.490 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-mgr-diskprediction-cloud-15.0.0-8014.gc529080.e 80/119 2019-12-05T06:05:19.581 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : liboath-2.6.2-1.el7.x86_64 81/119 2019-12-05T06:05:19.698 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : krb5-devel-1.15.1-37.el7_7.2.x86_64 82/119 2019-12-05T06:05:19.799 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-daemon-15.0.0-8014.gc529080.el7.x86_64 83/119 2019-12-05T06:05:19.902 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:python-rbd-15.0.0-8014.gc529080.el7.x86_64 88/119 2019-12-05T06:05:19.924 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:librados2-15.0.0-8014.gc529080.el7.x86_64 84/119 2019-12-05T06:05:20.017 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python2-rsa-3.4.1-1.el7.noarch 85/119 2019-12-05T06:05:20.028 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 89/119 2019-12-05T06:05:20.092 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python2-bcrypt-3.1.6-2.el7.x86_64 86/119 2019-12-05T06:05:20.176 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-websocket-client-0.56.0-3.git3c25814.el7.noarch 87/119 2019-12-05T06:05:20.178 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 90/119 2019-12-05T06:05:20.286 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:python-rbd-15.0.0-8014.gc529080.el7.x86_64 88/119 2019-12-05T06:05:20.288 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 91/119 2019-12-05T06:05:20.391 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : librdkafka-0.11.5-1.el7.x86_64 89/119 2019-12-05T06:05:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-base-15.0.0-8014.gc529080.el7.x86_64 92/119 2019-12-05T06:05:20.514 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-mon-15.0.0-8014.gc529080.el7.x86_64 93/119 2019-12-05T06:05:20.561 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libbabeltrace-1.2.4-3.el7.x86_64 90/119 2019-12-05T06:05:20.631 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:python-rgw-15.0.0-8014.gc529080.el7.x86_64 94/119 2019-12-05T06:05:20.678 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-test-15.0.0-8014.gc529080.el7.x86_64 91/119 2019-12-05T06:05:20.756 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 95/119 2019-12-05T06:05:20.778 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-base-15.0.0-8014.gc529080.el7.x86_64 92/119 2019-12-05T06:05:20.857 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 96/119 2019-12-05T06:05:20.887 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-mon-15.0.0-8014.gc529080.el7.x86_64 93/119 2019-12-05T06:05:20.957 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-devel-0.176-2.el7.x86_64 97/119 2019-12-05T06:05:20.996 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:python-rgw-15.0.0-8014.gc529080.el7.x86_64 94/119 2019-12-05T06:05:21.033 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 98/119 2019-12-05T06:05:21.097 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-pecan-0.4.5-2.el7.noarch 95/119 2019-12-05T06:05:21.117 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 99/119 2019-12-05T06:05:21.234 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcom_err-1.42.9-16.el7.x86_64 100/119 2019-12-05T06:05:21.239 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : tbb-4.1-9.20130314.el7.x86_64 96/119 2019-12-05T06:05:21.339 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-devel-0.176-2.el7.x86_64 97/119 2019-12-05T06:05:21.343 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-remoto-0.0.25-1.el7.noarch 101/119 2019-12-05T06:05:21.440 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : lttng-ust-2.4.1-4.el7.x86_64 98/119 2019-12-05T06:05:21.469 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : bison-3.0.4-2.el7.x86_64 102/119 2019-12-05T06:05:21.549 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 2:ceph-radosgw-15.0.0-8014.gc529080.el7.x86_64 99/119 2019-12-05T06:05:21.577 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 103/119 2019-12-05T06:05:21.658 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcom_err-1.42.9-16.el7.x86_64 100/119 2019-12-05T06:05:21.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 104/119 2019-12-05T06:05:21.759 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-remoto-0.0.25-1.el7.noarch 101/119 2019-12-05T06:05:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 1:openssl-devel-1.0.2k-19.el7.x86_64 105/119 2019-12-05T06:05:21.901 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : bison-3.0.4-2.el7.x86_64 102/119 2019-12-05T06:05:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 106/119 2019-12-05T06:05:21.985 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-cachetools-1.0.3-1.el7.noarch 103/119 2019-12-05T06:05:22.068 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python-tempita-0.5.1-6.el7.noarch 104/119 2019-12-05T06:05:22.072 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 107/119 2019-12-05T06:05:22.180 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-0.176-2.el7.x86_64 108/119 2019-12-05T06:05:22.182 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 109/119 2019-12-05T06:05:22.183 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libss-1.42.9-13.el7.x86_64 110/119 2019-12-05T06:05:22.184 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : e2fsprogs-libs-1.42.9-13.el7.x86_64 111/119 2019-12-05T06:05:22.185 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-0.172-2.el7.x86_64 112/119 2019-12-05T06:05:22.186 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 1:openssl-devel-1.0.2k-19.el7.x86_64 105/119 2019-12-05T06:05:22.186 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libelf-0.172-2.el7.x86_64 113/119 2019-12-05T06:05:22.186 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : elfutils-libs-0.172-2.el7.x86_64 114/119 2019-12-05T06:05:22.187 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcom_err-1.42.9-13.el7.x86_64 115/119 2019-12-05T06:05:22.188 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 1:openssl-1.0.2k-16.el7.x86_64 116/119 2019-12-05T06:05:22.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 1:openssl-libs-1.0.2k-16.el7.x86_64 117/119 2019-12-05T06:05:22.192 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 118/119 2019-12-05T06:05:22.294 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 1:python2-google-auth-1.1.1-5.el7.noarch 106/119 2019-12-05T06:05:22.555 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : e2fsprogs-1.42.9-13.el7.x86_64 119/119 2019-12-05T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2019-12-05T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stdout: bison.x86_64 0:3.0.4-2.el7 2019-12-05T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stdout: ceph.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-daemon.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.556 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.557 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-ssh.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libelf-devel.x86_64 0:0.176-2.el7 2019-12-05T06:05:22.558 INFO:teuthology.orchestra.run.smithi076.stdout: flex.x86_64 0:2.5.37-6.el7 2019-12-05T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-devel.x86_64 1:1.0.2k-19.el7 2019-12-05T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stdout: python-ceph-compat.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.559 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stdout:Dependency Installed: 2019-12-05T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.560 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stdout: jq.x86_64 0:1.5-1.el7 2019-12-05T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2019-12-05T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stdout: krb5-devel.x86_64 0:1.15.1-37.el7_7.2 2019-12-05T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stdout: leveldb.x86_64 0:1.12.0-11.el7 2019-12-05T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2019-12-05T06:05:22.561 INFO:teuthology.orchestra.run.smithi076.stdout: libcom_err-devel.x86_64 0:1.42.9-16.el7 2019-12-05T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stdout: liboath.x86_64 0:2.6.2-1.el7 2019-12-05T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2019-12-05T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2019-12-05T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stdout: libselinux-devel.x86_64 0:2.5-14.1.el7 2019-12-05T06:05:22.562 INFO:teuthology.orchestra.run.smithi076.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2019-12-05T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2019-12-05T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2019-12-05T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2019-12-05T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma.x86_64 0:5.9.5-3.el7 2019-12-05T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stdout: pcre-devel.x86_64 0:8.32-17.el7 2019-12-05T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stdout: pyOpenSSL.x86_64 0:0.13.1-4.el7 2019-12-05T06:05:22.563 INFO:teuthology.orchestra.run.smithi076.stdout: python-beaker.noarch 0:1.5.4-10.el7 2019-12-05T06:05:22.564 INFO:teuthology.orchestra.run.smithi076.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2019-12-05T06:05:22.564 INFO:teuthology.orchestra.run.smithi076.stdout: python-ceph-argparse.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.564 INFO:teuthology.orchestra.run.smithi076.stdout: python-ceph-common.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.564 INFO:teuthology.orchestra.run.smithi076.stdout: python-cephfs.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.564 INFO:teuthology.orchestra.run.smithi076.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2019-12-05T06:05:22.564 INFO:teuthology.orchestra.run.smithi076.stdout: python-execnet.noarch 0:1.2.0-7.el7 2019-12-05T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2019-12-05T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stdout: python-mako.noarch 0:0.8.1-2.el7 2019-12-05T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2019-12-05T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2019-12-05T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stdout: python-pecan.noarch 0:0.4.5-2.el7 2019-12-05T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2019-12-05T06:05:22.565 INFO:teuthology.orchestra.run.smithi076.stdout: python-rados.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.566 INFO:teuthology.orchestra.run.smithi076.stdout: python-rbd.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.566 INFO:teuthology.orchestra.run.smithi076.stdout: python-remoto.noarch 0:0.0.25-1.el7 2019-12-05T06:05:22.566 INFO:teuthology.orchestra.run.smithi076.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2019-12-05T06:05:22.566 INFO:teuthology.orchestra.run.smithi076.stdout: python-requests.noarch 0:2.6.0-7.el7_7 2019-12-05T06:05:22.566 INFO:teuthology.orchestra.run.smithi076.stdout: python-rgw.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.566 INFO:teuthology.orchestra.run.smithi076.stdout: python-routes.noarch 0:1.13-2.el7 2019-12-05T06:05:22.566 INFO:teuthology.orchestra.run.smithi076.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2019-12-05T06:05:22.567 INFO:teuthology.orchestra.run.smithi076.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2019-12-05T06:05:22.567 INFO:teuthology.orchestra.run.smithi076.stdout: python-tempita.noarch 0:0.5.1-6.el7 2019-12-05T06:05:22.567 INFO:teuthology.orchestra.run.smithi076.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2019-12-05T06:05:22.567 INFO:teuthology.orchestra.run.smithi076.stdout: python-webob.noarch 0:1.2.3-7.el7 2019-12-05T06:05:22.568 INFO:teuthology.orchestra.run.smithi076.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2019-12-05T06:05:22.568 INFO:teuthology.orchestra.run.smithi076.stdout: python-webtest.noarch 0:1.3.4-6.el7 2019-12-05T06:05:22.568 INFO:teuthology.orchestra.run.smithi076.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2019-12-05T06:05:22.568 INFO:teuthology.orchestra.run.smithi076.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2019-12-05T06:05:22.568 INFO:teuthology.orchestra.run.smithi076.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2019-12-05T06:05:22.568 INFO:teuthology.orchestra.run.smithi076.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2019-12-05T06:05:22.568 INFO:teuthology.orchestra.run.smithi076.stdout: python2-kubernetes.noarch 0:10.0.1-1.el7 2019-12-05T06:05:22.569 INFO:teuthology.orchestra.run.smithi076.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2019-12-05T06:05:22.569 INFO:teuthology.orchestra.run.smithi076.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2019-12-05T06:05:22.569 INFO:teuthology.orchestra.run.smithi076.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2019-12-05T06:05:22.569 INFO:teuthology.orchestra.run.smithi076.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2019-12-05T06:05:22.569 INFO:teuthology.orchestra.run.smithi076.stdout: python2-six.noarch 0:1.9.0-0.el7 2019-12-05T06:05:22.569 INFO:teuthology.orchestra.run.smithi076.stdout: python3.x86_64 0:3.6.8-10.el7 2019-12-05T06:05:22.569 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.570 INFO:teuthology.orchestra.run.smithi076.stdout: python3-libs.x86_64 0:3.6.8-10.el7 2019-12-05T06:05:22.570 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pip.noarch 0:9.0.3-5.el7 2019-12-05T06:05:22.570 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.570 INFO:teuthology.orchestra.run.smithi076.stdout: python3-setuptools.noarch 0:39.2.0-10.el7 2019-12-05T06:05:22.570 INFO:teuthology.orchestra.run.smithi076.stdout: scipy.x86_64 0:0.12.1-6.el7 2019-12-05T06:05:22.570 INFO:teuthology.orchestra.run.smithi076.stdout: socat.x86_64 0:1.7.3.2-2.el7 2019-12-05T06:05:22.570 INFO:teuthology.orchestra.run.smithi076.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2019-12-05T06:05:22.571 INFO:teuthology.orchestra.run.smithi076.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2019-12-05T06:05:22.571 INFO:teuthology.orchestra.run.smithi076.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2019-12-05T06:05:22.571 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet.x86_64 0:1.6.1-1.el7 2019-12-05T06:05:22.571 INFO:teuthology.orchestra.run.smithi076.stdout: zlib-devel.x86_64 0:1.2.7-18.el7 2019-12-05T06:05:22.571 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:05:22.571 INFO:teuthology.orchestra.run.smithi076.stdout:Updated: 2019-12-05T06:05:22.572 INFO:teuthology.orchestra.run.smithi076.stdout: librados2.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.572 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:22.572 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:05:22.572 INFO:teuthology.orchestra.run.smithi076.stdout:Dependency Updated: 2019-12-05T06:05:22.572 INFO:teuthology.orchestra.run.smithi076.stdout: e2fsprogs.x86_64 0:1.42.9-16.el7 e2fsprogs-libs.x86_64 0:1.42.9-16.el7 2019-12-05T06:05:22.572 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils.x86_64 0:0.176-2.el7 elfutils-libelf.x86_64 0:0.176-2.el7 2019-12-05T06:05:22.572 INFO:teuthology.orchestra.run.smithi076.stdout: elfutils-libs.x86_64 0:0.176-2.el7 libcom_err.x86_64 0:1.42.9-16.el7 2019-12-05T06:05:22.573 INFO:teuthology.orchestra.run.smithi076.stdout: libss.x86_64 0:1.42.9-16.el7 openssl.x86_64 1:1.0.2k-19.el7 2019-12-05T06:05:22.573 INFO:teuthology.orchestra.run.smithi076.stdout: openssl-libs.x86_64 1:1.0.2k-19.el7 2019-12-05T06:05:22.573 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:05:22.573 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2019-12-05T06:05:22.676 DEBUG:teuthology.parallel:result is None 2019-12-05T06:05:22.679 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : python2-certifi-2018.10.15-5.el7.noarch 107/119 2019-12-05T06:05:22.775 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-0.176-2.el7.x86_64 108/119 2019-12-05T06:05:22.776 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 1:librbd1-10.2.5-4.el7.x86_64 109/119 2019-12-05T06:05:22.777 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libss-1.42.9-13.el7.x86_64 110/119 2019-12-05T06:05:22.778 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : e2fsprogs-libs-1.42.9-13.el7.x86_64 111/119 2019-12-05T06:05:22.779 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-0.172-2.el7.x86_64 112/119 2019-12-05T06:05:22.780 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libelf-0.172-2.el7.x86_64 113/119 2019-12-05T06:05:22.781 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : elfutils-libs-0.172-2.el7.x86_64 114/119 2019-12-05T06:05:22.782 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : libcom_err-1.42.9-13.el7.x86_64 115/119 2019-12-05T06:05:22.783 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 1:openssl-1.0.2k-16.el7.x86_64 116/119 2019-12-05T06:05:22.784 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 1:openssl-libs-1.0.2k-16.el7.x86_64 117/119 2019-12-05T06:05:22.785 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : 1:librados2-10.2.5-4.el7.x86_64 118/119 2019-12-05T06:05:23.192 INFO:teuthology.orchestra.run.smithi016.stdout: Verifying : e2fsprogs-1.42.9-13.el7.x86_64 119/119 2019-12-05T06:05:23.192 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:05:23.192 INFO:teuthology.orchestra.run.smithi016.stdout:Installed: 2019-12-05T06:05:23.192 INFO:teuthology.orchestra.run.smithi016.stdout: bison.x86_64 0:3.0.4-2.el7 2019-12-05T06:05:23.192 INFO:teuthology.orchestra.run.smithi016.stdout: ceph.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.192 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-daemon.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.193 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-fuse.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.193 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.193 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-dashboard.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.193 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-cloud.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.193 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-diskprediction-local.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.193 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-rook.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.193 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mgr-ssh.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.194 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-radosgw.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.194 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-test.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.194 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libelf-devel.x86_64 0:0.176-2.el7 2019-12-05T06:05:23.194 INFO:teuthology.orchestra.run.smithi016.stdout: flex.x86_64 0:2.5.37-6.el7 2019-12-05T06:05:23.194 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs-devel.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.194 INFO:teuthology.orchestra.run.smithi016.stdout: libcephfs2.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.194 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-devel.x86_64 1:1.0.2k-19.el7 2019-12-05T06:05:23.195 INFO:teuthology.orchestra.run.smithi016.stdout: python-ceph-compat.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.195 INFO:teuthology.orchestra.run.smithi016.stdout: python3-cephfs.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.195 INFO:teuthology.orchestra.run.smithi016.stdout: rbd-fuse.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.195 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:05:23.195 INFO:teuthology.orchestra.run.smithi016.stdout:Dependency Installed: 2019-12-05T06:05:23.195 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-base.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.196 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-common.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.196 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-grafana-dashboards.noarch 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.197 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mds.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.197 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-mon.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.197 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-osd.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.197 INFO:teuthology.orchestra.run.smithi016.stdout: ceph-selinux.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.197 INFO:teuthology.orchestra.run.smithi016.stdout: jq.x86_64 0:1.5-1.el7 2019-12-05T06:05:23.198 INFO:teuthology.orchestra.run.smithi016.stdout: keyutils-libs-devel.x86_64 0:1.5.8-3.el7 2019-12-05T06:05:23.198 INFO:teuthology.orchestra.run.smithi016.stdout: krb5-devel.x86_64 0:1.15.1-37.el7_7.2 2019-12-05T06:05:23.198 INFO:teuthology.orchestra.run.smithi016.stdout: leveldb.x86_64 0:1.12.0-11.el7 2019-12-05T06:05:23.198 INFO:teuthology.orchestra.run.smithi016.stdout: libbabeltrace.x86_64 0:1.2.4-3.el7 2019-12-05T06:05:23.198 INFO:teuthology.orchestra.run.smithi016.stdout: libcom_err-devel.x86_64 0:1.42.9-16.el7 2019-12-05T06:05:23.198 INFO:teuthology.orchestra.run.smithi016.stdout: liboath.x86_64 0:2.6.2-1.el7 2019-12-05T06:05:23.198 INFO:teuthology.orchestra.run.smithi016.stdout: librabbitmq.x86_64 0:0.8.0-2.el7 2019-12-05T06:05:23.199 INFO:teuthology.orchestra.run.smithi016.stdout: librados-devel.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.199 INFO:teuthology.orchestra.run.smithi016.stdout: libradosstriper1.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.199 INFO:teuthology.orchestra.run.smithi016.stdout: librdkafka.x86_64 0:0.11.5-1.el7 2019-12-05T06:05:23.199 INFO:teuthology.orchestra.run.smithi016.stdout: librgw2.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.199 INFO:teuthology.orchestra.run.smithi016.stdout: libselinux-devel.x86_64 0:2.5-14.1.el7 2019-12-05T06:05:23.199 INFO:teuthology.orchestra.run.smithi016.stdout: libsepol-devel.x86_64 0:2.5-10.el7 2019-12-05T06:05:23.199 INFO:teuthology.orchestra.run.smithi016.stdout: libverto-devel.x86_64 0:0.2.5-4.el7 2019-12-05T06:05:23.200 INFO:teuthology.orchestra.run.smithi016.stdout: lttng-ust.x86_64 0:2.4.1-4.el7 2019-12-05T06:05:23.200 INFO:teuthology.orchestra.run.smithi016.stdout: numpy-f2py.x86_64 1:1.7.1-13.el7 2019-12-05T06:05:23.200 INFO:teuthology.orchestra.run.smithi016.stdout: oniguruma.x86_64 0:5.9.5-3.el7 2019-12-05T06:05:23.200 INFO:teuthology.orchestra.run.smithi016.stdout: pcre-devel.x86_64 0:8.32-17.el7 2019-12-05T06:05:23.200 INFO:teuthology.orchestra.run.smithi016.stdout: pyOpenSSL.x86_64 0:0.13.1-4.el7 2019-12-05T06:05:23.200 INFO:teuthology.orchestra.run.smithi016.stdout: python-beaker.noarch 0:1.5.4-10.el7 2019-12-05T06:05:23.200 INFO:teuthology.orchestra.run.smithi016.stdout: python-cachetools.noarch 0:1.0.3-1.el7 2019-12-05T06:05:23.200 INFO:teuthology.orchestra.run.smithi016.stdout: python-ceph-argparse.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.201 INFO:teuthology.orchestra.run.smithi016.stdout: python-ceph-common.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.201 INFO:teuthology.orchestra.run.smithi016.stdout: python-cephfs.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.201 INFO:teuthology.orchestra.run.smithi016.stdout: python-cherrypy.noarch 0:3.2.2-4.el7 2019-12-05T06:05:23.201 INFO:teuthology.orchestra.run.smithi016.stdout: python-execnet.noarch 0:1.2.0-7.el7 2019-12-05T06:05:23.201 INFO:teuthology.orchestra.run.smithi016.stdout: python-jwcrypto.noarch 0:0.4.2-1.el7 2019-12-05T06:05:23.201 INFO:teuthology.orchestra.run.smithi016.stdout: python-mako.noarch 0:0.8.1-2.el7 2019-12-05T06:05:23.201 INFO:teuthology.orchestra.run.smithi016.stdout: python-markupsafe.x86_64 0:0.11-10.el7 2019-12-05T06:05:23.202 INFO:teuthology.orchestra.run.smithi016.stdout: python-paste.noarch 0:1.7.5.1-9.20111221hg1498.el7 2019-12-05T06:05:23.202 INFO:teuthology.orchestra.run.smithi016.stdout: python-pecan.noarch 0:0.4.5-2.el7 2019-12-05T06:05:23.202 INFO:teuthology.orchestra.run.smithi016.stdout: python-prettytable.noarch 0:0.7.2-3.el7 2019-12-05T06:05:23.202 INFO:teuthology.orchestra.run.smithi016.stdout: python-rados.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.202 INFO:teuthology.orchestra.run.smithi016.stdout: python-rbd.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.202 INFO:teuthology.orchestra.run.smithi016.stdout: python-remoto.noarch 0:0.0.25-1.el7 2019-12-05T06:05:23.202 INFO:teuthology.orchestra.run.smithi016.stdout: python-repoze-lru.noarch 0:0.4-3.el7 2019-12-05T06:05:23.202 INFO:teuthology.orchestra.run.smithi016.stdout: python-requests.noarch 0:2.6.0-7.el7_7 2019-12-05T06:05:23.203 INFO:teuthology.orchestra.run.smithi016.stdout: python-rgw.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.203 INFO:teuthology.orchestra.run.smithi016.stdout: python-routes.noarch 0:1.13-2.el7 2019-12-05T06:05:23.203 INFO:teuthology.orchestra.run.smithi016.stdout: python-simplegeneric.noarch 0:0.8-7.el7 2019-12-05T06:05:23.203 INFO:teuthology.orchestra.run.smithi016.stdout: python-singledispatch.noarch 0:3.4.0.2-2.el7 2019-12-05T06:05:23.203 INFO:teuthology.orchestra.run.smithi016.stdout: python-tempita.noarch 0:0.5.1-6.el7 2019-12-05T06:05:23.203 INFO:teuthology.orchestra.run.smithi016.stdout: python-urllib3.noarch 0:1.10.2-7.el7 2019-12-05T06:05:23.203 INFO:teuthology.orchestra.run.smithi016.stdout: python-webob.noarch 0:1.2.3-7.el7 2019-12-05T06:05:23.204 INFO:teuthology.orchestra.run.smithi016.stdout: python-websocket-client.noarch 0:0.56.0-3.git3c25814.el7 2019-12-05T06:05:23.204 INFO:teuthology.orchestra.run.smithi016.stdout: python-webtest.noarch 0:1.3.4-6.el7 2019-12-05T06:05:23.204 INFO:teuthology.orchestra.run.smithi016.stdout: python-werkzeug.noarch 0:0.9.1-2.el7 2019-12-05T06:05:23.204 INFO:teuthology.orchestra.run.smithi016.stdout: python2-bcrypt.x86_64 0:3.1.6-2.el7 2019-12-05T06:05:23.204 INFO:teuthology.orchestra.run.smithi016.stdout: python2-certifi.noarch 0:2018.10.15-5.el7 2019-12-05T06:05:23.204 INFO:teuthology.orchestra.run.smithi016.stdout: python2-google-auth.noarch 1:1.1.1-5.el7 2019-12-05T06:05:23.204 INFO:teuthology.orchestra.run.smithi016.stdout: python2-kubernetes.noarch 0:10.0.1-1.el7 2019-12-05T06:05:23.205 INFO:teuthology.orchestra.run.smithi016.stdout: python2-oauthlib.noarch 0:2.0.1-8.el7 2019-12-05T06:05:23.205 INFO:teuthology.orchestra.run.smithi016.stdout: python2-pyasn1-modules.noarch 0:0.1.9-7.el7 2019-12-05T06:05:23.205 INFO:teuthology.orchestra.run.smithi016.stdout: python2-requests-oauthlib.noarch 0:0.8.0-5.el7 2019-12-05T06:05:23.205 INFO:teuthology.orchestra.run.smithi016.stdout: python2-rsa.noarch 0:3.4.1-1.el7 2019-12-05T06:05:23.205 INFO:teuthology.orchestra.run.smithi016.stdout: python2-six.noarch 0:1.9.0-0.el7 2019-12-05T06:05:23.205 INFO:teuthology.orchestra.run.smithi016.stdout: python3.x86_64 0:3.6.8-10.el7 2019-12-05T06:05:23.205 INFO:teuthology.orchestra.run.smithi016.stdout: python3-ceph-argparse.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.205 INFO:teuthology.orchestra.run.smithi016.stdout: python3-libs.x86_64 0:3.6.8-10.el7 2019-12-05T06:05:23.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-pip.noarch 0:9.0.3-5.el7 2019-12-05T06:05:23.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-rados.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.206 INFO:teuthology.orchestra.run.smithi016.stdout: python3-setuptools.noarch 0:39.2.0-10.el7 2019-12-05T06:05:23.206 INFO:teuthology.orchestra.run.smithi016.stdout: scipy.x86_64 0:0.12.1-6.el7 2019-12-05T06:05:23.207 INFO:teuthology.orchestra.run.smithi016.stdout: socat.x86_64 0:1.7.3.2-2.el7 2019-12-05T06:05:23.207 INFO:teuthology.orchestra.run.smithi016.stdout: suitesparse.x86_64 0:4.0.2-10.el7 2019-12-05T06:05:23.207 INFO:teuthology.orchestra.run.smithi016.stdout: tbb.x86_64 0:4.1-9.20130314.el7 2019-12-05T06:05:23.207 INFO:teuthology.orchestra.run.smithi016.stdout: userspace-rcu.x86_64 0:0.7.16-1.el7 2019-12-05T06:05:23.207 INFO:teuthology.orchestra.run.smithi016.stdout: xmlstarlet.x86_64 0:1.6.1-1.el7 2019-12-05T06:05:23.207 INFO:teuthology.orchestra.run.smithi016.stdout: zlib-devel.x86_64 0:1.2.7-18.el7 2019-12-05T06:05:23.207 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:05:23.208 INFO:teuthology.orchestra.run.smithi016.stdout:Updated: 2019-12-05T06:05:23.208 INFO:teuthology.orchestra.run.smithi016.stdout: librados2.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.208 INFO:teuthology.orchestra.run.smithi016.stdout: librbd1.x86_64 2:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.208 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:05:23.208 INFO:teuthology.orchestra.run.smithi016.stdout:Dependency Updated: 2019-12-05T06:05:23.208 INFO:teuthology.orchestra.run.smithi016.stdout: e2fsprogs.x86_64 0:1.42.9-16.el7 e2fsprogs-libs.x86_64 0:1.42.9-16.el7 2019-12-05T06:05:23.208 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils.x86_64 0:0.176-2.el7 elfutils-libelf.x86_64 0:0.176-2.el7 2019-12-05T06:05:23.208 INFO:teuthology.orchestra.run.smithi016.stdout: elfutils-libs.x86_64 0:0.176-2.el7 libcom_err.x86_64 0:1.42.9-16.el7 2019-12-05T06:05:23.209 INFO:teuthology.orchestra.run.smithi016.stdout: libss.x86_64 0:1.42.9-16.el7 openssl.x86_64 1:1.0.2k-19.el7 2019-12-05T06:05:23.209 INFO:teuthology.orchestra.run.smithi016.stdout: openssl-libs.x86_64 1:1.0.2k-19.el7 2019-12-05T06:05:23.209 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:05:23.209 INFO:teuthology.orchestra.run.smithi016.stdout:Complete! 2019-12-05T06:05:23.322 DEBUG:teuthology.parallel:result is None 2019-12-05T06:05:23.322 INFO:teuthology.orchestra.run.smithi076:> sudo lsb_release -is 2019-12-05T06:05:23.517 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS 2019-12-05T06:05:23.519 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-05T06:05:23.519 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T06:05:23.709 INFO:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-12-05T06:05:23.848 INFO:teuthology.orchestra.run.smithi076.stdout:15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.848 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-8014.gc529080.el7 2019-12-05T06:05:23.848 INFO:teuthology.task.install:The correct ceph version 15.0.0-8014.gc529080 is installed. 2019-12-05T06:05:23.849 INFO:teuthology.orchestra.run.smithi016:> sudo lsb_release -is 2019-12-05T06:05:24.050 INFO:teuthology.orchestra.run.smithi016.stdout:CentOS 2019-12-05T06:05:24.050 DEBUG:teuthology.misc:System to be installed: CentOS 2019-12-05T06:05:24.051 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F7%2Fx86_64&sha1=c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T06:05:24.220 INFO:teuthology.orchestra.run.smithi016:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2019-12-05T06:05:24.366 INFO:teuthology.orchestra.run.smithi016.stdout:15.0.0-8014.gc529080.el7 2019-12-05T06:05:24.367 INFO:teuthology.packaging:The installed version of ceph is 15.0.0-8014.gc529080.el7 2019-12-05T06:05:24.367 INFO:teuthology.task.install:The correct ceph version 15.0.0-8014.gc529080 is installed. 2019-12-05T06:05:24.367 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2019-12-05T06:05:24.367 INFO:teuthology.orchestra.run.smithi076:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-12-05T06:05:24.489 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /home/ubuntu/cephtest/valgrind.supp' 2019-12-05T06:05:24.617 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2019-12-05T06:05:24.617 INFO:teuthology.orchestra.run.smithi076:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-12-05T06:05:24.738 INFO:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-12-05T06:05:24.862 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /usr/bin/daemon-helper' 2019-12-05T06:05:24.988 INFO:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/daemon-helper 2019-12-05T06:05:25.102 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2019-12-05T06:05:25.102 INFO:teuthology.orchestra.run.smithi076:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-12-05T06:05:25.219 INFO:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-12-05T06:05:25.327 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /usr/bin/adjust-ulimits' 2019-12-05T06:05:25.456 INFO:teuthology.orchestra.run.smithi016:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2019-12-05T06:05:25.563 INFO:teuthology.run_tasks:Running task ceph... 2019-12-05T06:05:25.672 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2019-12-05T06:05:25.672 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 777 /var/log/ceph 2019-12-05T06:05:25.677 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 777 /var/log/ceph 2019-12-05T06:05:25.795 INFO:tasks.ceph:Disabling ceph logrotate... 2019-12-05T06:05:25.795 INFO:teuthology.orchestra.run.smithi016:> sudo rm -f -- /etc/logrotate.d/ceph 2019-12-05T06:05:25.800 INFO:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/logrotate.d/ceph 2019-12-05T06:05:25.910 INFO:tasks.ceph:Creating extra log directories... 2019-12-05T06:05:25.911 INFO:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-12-05T06:05:25.915 INFO:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2019-12-05T06:05:26.024 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2019-12-05T06:05:26.024 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2019-12-05T06:05:26.024 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2019-12-05T06:05:26.024 INFO:teuthology.orchestra.run.smithi076:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2019-12-05T06:05:26.093 INFO:teuthology.orchestra.run.smithi076:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2019-12-05T06:05:26.366 INFO:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2019-12-05T06:05:26.480 INFO:teuthology.orchestra.run.smithi016:> cat > /home/ubuntu/cephtest/logrotate.ceph-test.conf 2019-12-05T06:05:26.574 INFO:teuthology.orchestra.run.smithi016:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2019-12-05T06:05:26.743 INFO:teuthology.orchestra.run.smithi016:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2019-12-05T06:05:26.838 INFO:tasks.ceph:Creating ceph cluster ceph... 2019-12-05T06:05:26.838 INFO:tasks.ceph:config {'mon_bind_addrvec': True, 'block_journal': None, 'fs': 'xfs', 'log_whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'not responding, replacing', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'skip_mgr_daemons': False, 'cpu_profile': set([]), 'mkfs_options': None, 'tmpfs_journal': None, 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mount_options': None, 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'global': {'ms inject delay type': 'osd mds', 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'mon client directed command retry': 5, 'ms inject socket failures': 2500}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'debug osd': 25, 'bluestore compression mode': 'aggressive', 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'debug rocksdb': 10, 'debug journal': 20, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'osd op complaint time': 180}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}}} 2019-12-05T06:05:26.838 INFO:tasks.ceph:ctx.config {'kernel': {'sha1': 'distro', 'kdb': True}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.lock_machines': (2, 'smithi')}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'sha1': 'distro', 'kdb': True}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel']}, 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'}}, {'ceph': None}, {'mds_thrash': None}, {'ceph-fuse': None}, {'workunit': {'clients': {'all': ['fs/snaps']}}}], 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_wip-pdonnell-testing-20191204.231200/qa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/worker.smithi.9206', 'owner': 'scheduled_pdonnell@teuthology', 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'targets': {'smithi016.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCzkbvNpgEekN2H5FRrrk8O9zYD3vW8yCH3uhmzawZAVKOdsmM4IeOc05s7xzUjHA0z4xaSGAVu2uwJrb99bqqTRCLGL9ZJuHe8hZaEoo5SWseu0pgiPMJLl59z+xqHQCTPHI9lxuAdpRWxiXzS7+oAWnSEAfPCr6tyPmrew6vbMely+/lMmXc+03L7Hb4MxO6lMe/w/cZmz9zl0wCiich480FZKKRygxRmqJDwpswJkxKEVA95fxYx5FLUPypTwLTi2A63lRTXxIETT3i8QfPdIx5qAZIcBQlwC7tdzu2G1GNXQxCAf75YP5FgbRJCakCb6vpl3EoWbc8HRFYuGLHD', 'smithi076.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc4m2iSvsnpO8WMgH+idDlxTdFNO1LUOYlSE5JEJUyfOUF3+2D64dchE5K6fkXioBw9cTyXP/sIL3PTmvkfkQjZ/T/eFyMfOoYcIJZlo/yQicIh3dPRb0y7+oWDYY79l6VwjCDugCrrHbqPmf/6knofnwhL+Wxrc0Ogit57xxMevL58y+1Y9TY60VAf5XlFJa+er/8UB6ZXddirHQY61T9o3BjHNybxrjPANCFasSrwX/6z2BG4B/SwN5c0Bc0N24yygsgIjYG6j/iPZDONuNLfP5R3Gq3IannZ0X+9aI/6fvdeBaiqokwvyeqcgdQqW0a2YWxNBWsAeK6e/P5SpXl'}, 'job_id': '4567683', 'tube': 'smithi', 'suite_branch': 'wip-pdonnell-testing-20191204.231200', 'priority': 75, 'os_version': '7.6', 'first_in_suite': False, 'branch': 'wip-pdonnell-testing-20191204.231200', 'suite': 'fs', 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'email': 'pdonnell@redhat.com', 'archive_path': '/home/teuthworker/archive/pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi/4567683', 'description': 'fs/thrash/{begin.yaml ceph-thrash/mds.yaml clusters/1-mds-1-client-coloc.yaml conf/{client.yaml mds.yaml mon.yaml osd.yaml} mount/fuse.yaml msgr-failures/osd-mds-delay.yaml objectstore-ec/bluestore-comp-ec-root.yaml overrides/{frag_enable.yaml whitelist_health.yaml whitelist_wrongly_marked_down.yaml} supported-random-distros$/{centos_7.yaml} tasks/cfuse_workunit_snaptests.yaml}', 'nuke-on-error': True, 'repo': 'git://git.ceph.com/ceph-ci.git', 'last_in_suite': False, 'machine_type': 'smithi', 'teuthology_branch': 'master', 'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec', 'name': 'pdonnell-2019-12-05_02:12:53-fs-wip-pdonnell-testing-20191204.231200-distro-basic-smithi', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'overrides': {'ceph': {'cephfs_ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'not responding, replacing', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'conf': {'mds': {'mds debug frag': True, 'mds bal split bits': 3, 'mds bal split size': 100, 'debug mds': 20, 'mds bal merge size': 5, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds op complaint time': 180, 'mds debug scatterstat': True, 'osd op complaint time': 180, 'mds verify scatter': True}, 'global': {'ms inject delay type': 'osd mds', 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'mon client directed command retry': 5, 'ms inject socket failures': 2500}, 'osd': {'mon osd full ratio': 0.9, 'debug ms': 1, 'bluestore fsck on mount': True, 'debug osd': 25, 'bluestore compression mode': 'aggressive', 'debug bluestore': 20, 'debug bluefs': 20, 'mon osd backfillfull_ratio': 0.85, 'debug rocksdb': 10, 'debug journal': 20, 'mon osd nearfull ratio': 0.8, 'osd objectstore': 'bluestore', 'osd failsafe full ratio': 0.95, 'bluestore block size': 96636764160, 'debug filestore': 20, 'osd op complaint time': 180}, 'mon': {'debug mon': 20, 'debug paxos': 20, 'mon op complaint time': 120, 'debug ms': 1}, 'client': {'debug ms': 1, 'debug client': 20, 'client mount timeout': 600}}, 'fs': 'xfs', 'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'}}, 'admin_socket': {'branch': 'wip-pdonnell-testing-20191204.231200'}, 'thrashosds': {'bdev_inject_crash_probability': 0.5, 'bdev_inject_crash': 2}, 'workunit': {'sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec', 'branch': 'wip-pdonnell-testing-20191204.231200'}}, 'os_type': 'centos', 'suite_sha1': 'c5290805033483ba0cec165c06bde07dae5f19ec'} 2019-12-05T06:05:26.839 INFO:teuthology.orchestra.run.smithi016:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-12-05T06:05:26.845 INFO:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2019-12-05T06:05:26.945 INFO:teuthology.orchestra.run.smithi016:> sudo install -d -m0777 -- /var/run/ceph 2019-12-05T06:05:26.950 INFO:teuthology.orchestra.run.smithi076:> sudo install -d -m0777 -- /var/run/ceph 2019-12-05T06:05:27.137 DEBUG:teuthology.orchestra.remote:smithi076:/scratch_devs is 72B 2019-12-05T06:05:27.227 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-05T06:05:27.227 INFO:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-12-05T06:05:27.313 INFO:teuthology.orchestra.run.smithi076.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-1’ 2019-12-05T06:05:27.313 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T06:05:27.314 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 103112 Links: 1 2019-12-05T06:05:27.314 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T06:05:27.314 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2019-12-05T06:05:27.314 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2019-12-05 06:04:57.995359070 +0000 2019-12-05T06:05:27.314 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2019-12-05 06:02:41.605073717 +0000 2019-12-05T06:05:27.314 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2019-12-05 06:02:41.605073717 +0000 2019-12-05T06:05:27.314 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2019-12-05T06:05:27.342 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2019-12-05T06:05:27.342 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2019-12-05T06:05:27.343 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes (512 B) copied, 0.000337933 s, 1.5 MB/s 2019-12-05T06:05:27.348 INFO:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-12-05T06:05:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2019-12-05T06:05:27.415 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T06:05:27.416 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 96035 Links: 1 2019-12-05T06:05:27.416 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T06:05:27.416 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2019-12-05T06:05:27.416 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2019-12-05 06:04:57.995359070 +0000 2019-12-05T06:05:27.416 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2019-12-05 06:02:42.078058467 +0000 2019-12-05T06:05:27.416 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2019-12-05 06:02:42.078058467 +0000 2019-12-05T06:05:27.416 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2019-12-05T06:05:27.435 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2019-12-05T06:05:27.435 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2019-12-05T06:05:27.435 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes (512 B) copied, 0.000328324 s, 1.6 MB/s 2019-12-05T06:05:27.439 INFO:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-12-05T06:05:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-3’ 2019-12-05T06:05:27.492 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T06:05:27.492 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 103163 Links: 1 2019-12-05T06:05:27.492 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T06:05:27.493 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2019-12-05T06:05:27.493 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2019-12-05 06:04:57.995359070 +0000 2019-12-05T06:05:27.493 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2019-12-05 06:02:42.559042957 +0000 2019-12-05T06:05:27.493 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2019-12-05 06:02:42.559042957 +0000 2019-12-05T06:05:27.493 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2019-12-05T06:05:27.509 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2019-12-05T06:05:27.510 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2019-12-05T06:05:27.510 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes (512 B) copied, 0.00028304 s, 1.8 MB/s 2019-12-05T06:05:27.513 INFO:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-12-05T06:05:27.562 INFO:teuthology.orchestra.run.smithi076.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-4’ 2019-12-05T06:05:27.562 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T06:05:27.563 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 5h/5d Inode: 96079 Links: 1 2019-12-05T06:05:27.563 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T06:05:27.563 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2019-12-05T06:05:27.563 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2019-12-05 06:04:57.995359070 +0000 2019-12-05T06:05:27.563 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2019-12-05 06:02:43.038027512 +0000 2019-12-05T06:05:27.563 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2019-12-05 06:02:43.038027512 +0000 2019-12-05T06:05:27.563 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2019-12-05T06:05:27.577 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2019-12-05T06:05:27.578 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2019-12-05T06:05:27.578 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes (512 B) copied, 0.000298616 s, 1.7 MB/s 2019-12-05T06:05:27.581 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-12-05T06:05:27.581 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-05T06:05:27.581 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2019-12-05T06:05:27.581 INFO:tasks.ceph:dev map: {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-05T06:05:27.678 DEBUG:teuthology.orchestra.remote:smithi016:/scratch_devs is 72B 2019-12-05T06:05:27.772 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-05T06:05:27.772 INFO:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2019-12-05T06:05:27.851 INFO:teuthology.orchestra.run.smithi016.stdout: File: ‘/dev/vg_nvme/lv_4’ -> ‘../dm-1’ 2019-12-05T06:05:27.851 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T06:05:27.851 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 108605 Links: 1 2019-12-05T06:05:27.851 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T06:05:27.852 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2019-12-05T06:05:27.852 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2019-12-05 06:04:58.676073183 +0000 2019-12-05T06:05:27.852 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2019-12-05 06:02:48.832478886 +0000 2019-12-05T06:05:27.852 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2019-12-05 06:02:48.832478886 +0000 2019-12-05T06:05:27.852 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2019-12-05T06:05:27.880 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2019-12-05T06:05:27.881 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2019-12-05T06:05:27.881 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes (512 B) copied, 0.000312163 s, 1.6 MB/s 2019-12-05T06:05:27.886 INFO:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2019-12-05T06:05:27.964 INFO:teuthology.orchestra.run.smithi016.stdout: File: ‘/dev/vg_nvme/lv_3’ -> ‘../dm-2’ 2019-12-05T06:05:27.965 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T06:05:27.965 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 99950 Links: 1 2019-12-05T06:05:27.965 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T06:05:27.965 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2019-12-05T06:05:27.965 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2019-12-05 06:04:58.676073183 +0000 2019-12-05T06:05:27.965 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2019-12-05 06:02:49.315469918 +0000 2019-12-05T06:05:27.966 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2019-12-05 06:02:49.315469918 +0000 2019-12-05T06:05:27.966 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2019-12-05T06:05:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2019-12-05T06:05:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2019-12-05T06:05:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes (512 B) copied, 0.000333375 s, 1.5 MB/s 2019-12-05T06:05:27.998 INFO:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2019-12-05T06:05:28.067 INFO:teuthology.orchestra.run.smithi016.stdout: File: ‘/dev/vg_nvme/lv_2’ -> ‘../dm-3’ 2019-12-05T06:05:28.068 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T06:05:28.068 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 106115 Links: 1 2019-12-05T06:05:28.068 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T06:05:28.068 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2019-12-05T06:05:28.068 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2019-12-05 06:04:58.676073183 +0000 2019-12-05T06:05:28.069 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2019-12-05 06:02:49.802460875 +0000 2019-12-05T06:05:28.069 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2019-12-05 06:02:49.802460875 +0000 2019-12-05T06:05:28.069 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2019-12-05T06:05:28.092 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2019-12-05T06:05:28.092 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2019-12-05T06:05:28.093 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes (512 B) copied, 0.000266465 s, 1.9 MB/s 2019-12-05T06:05:28.097 INFO:teuthology.orchestra.run.smithi016:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2019-12-05T06:05:28.170 INFO:teuthology.orchestra.run.smithi016.stdout: File: ‘/dev/vg_nvme/lv_1’ -> ‘../dm-4’ 2019-12-05T06:05:28.170 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2019-12-05T06:05:28.170 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 5h/5d Inode: 93094 Links: 1 2019-12-05T06:05:28.170 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T06:05:28.171 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:device_t:s0 2019-12-05T06:05:28.171 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 2019-12-05 06:04:58.676073183 +0000 2019-12-05T06:05:28.171 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2019-12-05 06:02:50.284451926 +0000 2019-12-05T06:05:28.171 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2019-12-05 06:02:50.284451926 +0000 2019-12-05T06:05:28.171 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2019-12-05T06:05:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records in 2019-12-05T06:05:28.198 INFO:teuthology.orchestra.run.smithi016.stderr:1+0 records out 2019-12-05T06:05:28.199 INFO:teuthology.orchestra.run.smithi016.stderr:512 bytes (512 B) copied, 0.0003216 s, 1.6 MB/s 2019-12-05T06:05:28.203 INFO:tasks.ceph:fs option selected, checking for scratch devs 2019-12-05T06:05:28.203 INFO:tasks.ceph:found devs: ['/dev/vg_nvme/lv_4', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_1'] 2019-12-05T06:05:28.203 WARNING:teuthology.misc:Entering get_wwn_id_map, a deprecated function that will be removed 2019-12-05T06:05:28.203 INFO:tasks.ceph:dev map: {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-05T06:05:28.204 INFO:tasks.ceph:Generating config... 2019-12-05T06:05:28.205 INFO:tasks.ceph:[mds] mds debug frag = True 2019-12-05T06:05:28.205 INFO:tasks.ceph:[mds] mds bal split bits = 3 2019-12-05T06:05:28.205 INFO:tasks.ceph:[mds] mds bal split size = 100 2019-12-05T06:05:28.205 INFO:tasks.ceph:[mds] debug mds = 20 2019-12-05T06:05:28.205 INFO:tasks.ceph:[mds] mds bal merge size = 5 2019-12-05T06:05:28.206 INFO:tasks.ceph:[mds] debug ms = 1 2019-12-05T06:05:28.206 INFO:tasks.ceph:[mds] mds bal frag = True 2019-12-05T06:05:28.206 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2019-12-05T06:05:28.206 INFO:tasks.ceph:[mds] mds op complaint time = 180 2019-12-05T06:05:28.206 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2019-12-05T06:05:28.206 INFO:tasks.ceph:[mds] osd op complaint time = 180 2019-12-05T06:05:28.206 INFO:tasks.ceph:[mds] mds verify scatter = True 2019-12-05T06:05:28.207 INFO:tasks.ceph:[global] ms inject delay type = osd mds 2019-12-05T06:05:28.207 INFO:tasks.ceph:[global] ms inject delay max = 1 2019-12-05T06:05:28.207 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2019-12-05T06:05:28.207 INFO:tasks.ceph:[global] mon client directed command retry = 5 2019-12-05T06:05:28.207 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2019-12-05T06:05:28.207 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2019-12-05T06:05:28.207 INFO:tasks.ceph:[osd] debug ms = 1 2019-12-05T06:05:28.207 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2019-12-05T06:05:28.208 INFO:tasks.ceph:[osd] debug osd = 25 2019-12-05T06:05:28.208 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2019-12-05T06:05:28.208 INFO:tasks.ceph:[osd] debug bluestore = 20 2019-12-05T06:05:28.208 INFO:tasks.ceph:[osd] debug bluefs = 20 2019-12-05T06:05:28.209 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2019-12-05T06:05:28.209 INFO:tasks.ceph:[osd] debug rocksdb = 10 2019-12-05T06:05:28.209 INFO:tasks.ceph:[osd] debug journal = 20 2019-12-05T06:05:28.209 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2019-12-05T06:05:28.209 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2019-12-05T06:05:28.209 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2019-12-05T06:05:28.210 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2019-12-05T06:05:28.210 INFO:tasks.ceph:[osd] debug filestore = 20 2019-12-05T06:05:28.210 INFO:tasks.ceph:[osd] osd op complaint time = 180 2019-12-05T06:05:28.210 INFO:tasks.ceph:[mon] debug mon = 20 2019-12-05T06:05:28.211 INFO:tasks.ceph:[mon] debug paxos = 20 2019-12-05T06:05:28.211 INFO:tasks.ceph:[mon] mon op complaint time = 120 2019-12-05T06:05:28.211 INFO:tasks.ceph:[mon] debug ms = 1 2019-12-05T06:05:28.212 INFO:tasks.ceph:[client] debug ms = 1 2019-12-05T06:05:28.212 INFO:tasks.ceph:[client] debug client = 20 2019-12-05T06:05:28.212 INFO:tasks.ceph:[client] client mount timeout = 600 2019-12-05T06:05:28.212 INFO:tasks.ceph:Setting up mon.a... 2019-12-05T06:05:28.213 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2019-12-05T06:05:28.389 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.keyring 2019-12-05T06:05:28.395 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2019-12-05T06:05:28.561 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-05T06:05:28.682 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.16'), ('mon.c', '[v2:172.21.15.76:3301,v1:172.21.15.76:6790]'), ('mon.b', '172.21.15.76')] 2019-12-05T06:05:28.682 INFO:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.16 --addv c '[v2:172.21.15.76:3301,v1:172.21.15.76:6790]' --add b 172.21.15.76 --print /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:28.824 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:28.825 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: generated fsid df6db985-de08-463b-af1a-6ee59e0bbbe7 2019-12-05T06:05:28.825 INFO:teuthology.orchestra.run.smithi016.stdout:epoch 0 2019-12-05T06:05:28.825 INFO:teuthology.orchestra.run.smithi016.stdout:fsid df6db985-de08-463b-af1a-6ee59e0bbbe7 2019-12-05T06:05:28.825 INFO:teuthology.orchestra.run.smithi016.stdout:last_changed 2019-12-05T06:05:28.825944+0000 2019-12-05T06:05:28.825 INFO:teuthology.orchestra.run.smithi016.stdout:created 2019-12-05T06:05:28.825944+0000 2019-12-05T06:05:28.825 INFO:teuthology.orchestra.run.smithi016.stdout:min_mon_release 0 (unknown) 2019-12-05T06:05:28.826 INFO:teuthology.orchestra.run.smithi016.stdout:0: [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] mon.a 2019-12-05T06:05:28.826 INFO:teuthology.orchestra.run.smithi016.stdout:1: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.b 2019-12-05T06:05:28.826 INFO:teuthology.orchestra.run.smithi016.stdout:2: [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] mon.c 2019-12-05T06:05:28.826 INFO:teuthology.orchestra.run.smithi016.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2019-12-05T06:05:28.827 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID df6db985-de08-463b-af1a-6ee59e0bbbe7... 2019-12-05T06:05:28.828 INFO:teuthology.orchestra.run.smithi016:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-12-05T06:05:28.833 INFO:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo python -c 'import shutil, sys; shutil.copyfileobj(sys.stdin, file(sys.argv[1], "wb"))' /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf 2019-12-05T06:05:29.055 INFO:tasks.ceph:Creating admin key on mon.a... 2019-12-05T06:05:29.055 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2019-12-05T06:05:29.205 INFO:tasks.ceph:Copying monmap to all nodes... 2019-12-05T06:05:29.299 DEBUG:teuthology.orchestra.remote:smithi016:/etc/ceph/ceph.keyring is 206B 2019-12-05T06:05:29.477 DEBUG:teuthology.orchestra.remote:smithi016:/home/ubuntu/cephtest/ceph.monmap is 379B 2019-12-05T06:05:29.586 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi076.front.sepia.ceph.com 2019-12-05T06:05:29.587 INFO:teuthology.orchestra.run.smithi076:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-05T06:05:29.736 INFO:teuthology.orchestra.run.smithi076:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:29.816 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T06:05:29.817 INFO:teuthology.orchestra.run.smithi016:> sudo sh -c 'cat > /etc/ceph/ceph.keyring' && sudo chmod 0644 /etc/ceph/ceph.keyring 2019-12-05T06:05:29.969 INFO:teuthology.orchestra.run.smithi016:> cat > /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:30.043 INFO:tasks.ceph:Setting up mon nodes... 2019-12-05T06:05:30.044 INFO:tasks.ceph:Setting up mgr nodes... 2019-12-05T06:05:30.044 INFO:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2019-12-05T06:05:30.469 INFO:teuthology.orchestra.run.smithi076.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2019-12-05T06:05:30.474 INFO:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2019-12-05T06:05:30.660 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2019-12-05T06:05:30.666 INFO:tasks.ceph:Setting up mds nodes... 2019-12-05T06:05:30.667 INFO:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2019-12-05T06:05:30.851 INFO:teuthology.orchestra.run.smithi076.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2019-12-05T06:05:30.856 INFO:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2019-12-05T06:05:30.975 INFO:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2019-12-05T06:05:31.156 INFO:teuthology.orchestra.run.smithi016.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2019-12-05T06:05:31.161 INFO:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2019-12-05T06:05:31.278 INFO:tasks.ceph_client:Setting up client nodes... 2019-12-05T06:05:31.278 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2019-12-05T06:05:31.413 INFO:teuthology.orchestra.run.smithi016.stdout:creating /etc/ceph/ceph.client.0.keyring 2019-12-05T06:05:31.448 INFO:tasks.ceph:Running mkfs on osd nodes... 2019-12-05T06:05:31.448 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi076.front.sepia.ceph.com'): {'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'}, Remote(name='ubuntu@smithi016.front.sepia.ceph.com'): {'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'}} 2019-12-05T06:05:31.449 INFO:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2019-12-05T06:05:31.574 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-05T06:05:31.574 INFO:tasks.ceph:{} 2019-12-05T06:05:31.574 INFO:tasks.ceph:osd.4 2019-12-05T06:05:31.575 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi076.front.sepia.ceph.com 2019-12-05T06:05:31.575 INFO:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2019-12-05T06:05:32.148 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T06:05:32.149 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T06:05:32.149 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=0, sparse=0 2019-12-05T06:05:32.149 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T06:05:32.150 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2019-12-05T06:05:32.150 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T06:05:32.150 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T06:05:32.151 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T06:05:32.151 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T06:05:32.152 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2019-12-05T06:05:32.153 INFO:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-4 2019-12-05T06:05:32.318 INFO:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2019-12-05T06:05:32.417 INFO:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2019-12-05T06:05:32.488 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-05T06:05:32.489 INFO:tasks.ceph:{} 2019-12-05T06:05:32.489 INFO:tasks.ceph:osd.5 2019-12-05T06:05:32.489 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com 2019-12-05T06:05:32.489 INFO:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2019-12-05T06:05:33.018 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T06:05:33.018 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T06:05:33.018 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=0, sparse=0 2019-12-05T06:05:33.018 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T06:05:33.018 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2019-12-05T06:05:33.018 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T06:05:33.019 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T06:05:33.019 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T06:05:33.019 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T06:05:33.021 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2019-12-05T06:05:33.021 INFO:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-5 2019-12-05T06:05:33.178 INFO:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2019-12-05T06:05:33.303 INFO:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2019-12-05T06:05:33.391 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-05T06:05:33.391 INFO:tasks.ceph:{} 2019-12-05T06:05:33.391 INFO:tasks.ceph:osd.6 2019-12-05T06:05:33.391 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com 2019-12-05T06:05:33.391 INFO:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2019-12-05T06:05:33.961 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T06:05:33.961 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T06:05:33.961 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=0, sparse=0 2019-12-05T06:05:33.961 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T06:05:33.961 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2019-12-05T06:05:33.961 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T06:05:33.962 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T06:05:33.962 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T06:05:33.962 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T06:05:33.964 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2019-12-05T06:05:33.964 INFO:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-6 2019-12-05T06:05:34.148 INFO:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2019-12-05T06:05:34.252 INFO:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2019-12-05T06:05:34.333 INFO:tasks.ceph:{'osd.5': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_4', 'osd.7': '/dev/vg_nvme/lv_3', 'osd.6': '/dev/vg_nvme/lv_2'} 2019-12-05T06:05:34.334 INFO:tasks.ceph:{} 2019-12-05T06:05:34.334 INFO:tasks.ceph:osd.7 2019-12-05T06:05:34.334 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com 2019-12-05T06:05:34.334 INFO:teuthology.orchestra.run.smithi076:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2019-12-05T06:05:34.855 INFO:teuthology.orchestra.run.smithi076.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T06:05:34.855 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T06:05:34.855 INFO:teuthology.orchestra.run.smithi076.stdout: = crc=1 finobt=0, sparse=0 2019-12-05T06:05:34.855 INFO:teuthology.orchestra.run.smithi076.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T06:05:34.856 INFO:teuthology.orchestra.run.smithi076.stdout: = sunit=0 swidth=0 blks 2019-12-05T06:05:34.856 INFO:teuthology.orchestra.run.smithi076.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T06:05:34.856 INFO:teuthology.orchestra.run.smithi076.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T06:05:34.856 INFO:teuthology.orchestra.run.smithi076.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T06:05:34.856 INFO:teuthology.orchestra.run.smithi076.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T06:05:34.858 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi076.front.sepia.ceph.com -o noatime 2019-12-05T06:05:34.858 INFO:teuthology.orchestra.run.smithi076:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-7 2019-12-05T06:05:34.987 INFO:teuthology.orchestra.run.smithi076:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2019-12-05T06:05:35.102 INFO:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:35.273 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:35.276+0000 7f6e57df1bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2019-12-05T06:05:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:35.276+0000 7f6e57df1bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2019-12-05T06:05:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:35.276+0000 7f6e57df1bc0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-05T06:05:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:35.276+0000 7f6e57df1bc0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-05T06:05:35.274 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:35.276+0000 7f6e57df1bc0 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2019-12-05T06:05:35.275 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:35.277+0000 7f6e57df1bc0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2019-12-05T06:05:37.580 INFO:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2019-12-05T06:05:37.696 INFO:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:37.833 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:37.836+0000 7f6e17941bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2019-12-05T06:05:37.834 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:37.836+0000 7f6e17941bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2019-12-05T06:05:37.834 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:37.836+0000 7f6e17941bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-05T06:05:37.834 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:37.836+0000 7f6e17941bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-05T06:05:37.834 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:37.836+0000 7f6e17941bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2019-12-05T06:05:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:37.838+0000 7f6e17941bc0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2019-12-05T06:05:40.140 INFO:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2019-12-05T06:05:40.256 INFO:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:40.402 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:40.405+0000 7fd2617b6bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2019-12-05T06:05:40.403 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:40.405+0000 7fd2617b6bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2019-12-05T06:05:40.403 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:40.405+0000 7fd2617b6bc0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-05T06:05:40.403 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:40.405+0000 7fd2617b6bc0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-05T06:05:40.404 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:40.405+0000 7fd2617b6bc0 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2019-12-05T06:05:40.404 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:40.407+0000 7fd2617b6bc0 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2019-12-05T06:05:42.705 INFO:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2019-12-05T06:05:42.819 INFO:teuthology.orchestra.run.smithi076:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:42.932 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:42.934+0000 7f78bc74abc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2019-12-05T06:05:42.932 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:42.935+0000 7f78bc74abc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2019-12-05T06:05:42.933 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:42.935+0000 7f78bc74abc0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-05T06:05:42.933 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:42.935+0000 7f78bc74abc0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-05T06:05:42.933 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:42.935+0000 7f78bc74abc0 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2019-12-05T06:05:42.933 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:05:42.936+0000 7f78bc74abc0 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2019-12-05T06:05:45.238 INFO:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2019-12-05T06:05:45.356 INFO:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2019-12-05T06:05:45.478 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-05T06:05:45.478 INFO:tasks.ceph:{} 2019-12-05T06:05:45.478 INFO:tasks.ceph:osd.0 2019-12-05T06:05:45.479 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T06:05:45.479 INFO:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2019-12-05T06:05:45.624 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T06:05:45.624 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T06:05:45.624 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=0, sparse=0 2019-12-05T06:05:45.624 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T06:05:45.624 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2019-12-05T06:05:45.624 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T06:05:45.625 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T06:05:45.625 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T06:05:45.625 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T06:05:45.627 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2019-12-05T06:05:45.627 INFO:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-0 2019-12-05T06:05:45.745 INFO:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2019-12-05T06:05:45.849 INFO:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2019-12-05T06:05:45.930 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-05T06:05:45.931 INFO:tasks.ceph:{} 2019-12-05T06:05:45.931 INFO:tasks.ceph:osd.1 2019-12-05T06:05:45.931 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T06:05:45.931 INFO:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2019-12-05T06:05:46.058 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T06:05:46.059 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T06:05:46.059 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=0, sparse=0 2019-12-05T06:05:46.059 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T06:05:46.059 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2019-12-05T06:05:46.059 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T06:05:46.060 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T06:05:46.060 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T06:05:46.060 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T06:05:46.061 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2019-12-05T06:05:46.061 INFO:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-1 2019-12-05T06:05:46.180 INFO:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2019-12-05T06:05:46.290 INFO:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2019-12-05T06:05:46.393 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-05T06:05:46.394 INFO:tasks.ceph:{} 2019-12-05T06:05:46.394 INFO:tasks.ceph:osd.2 2019-12-05T06:05:46.394 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T06:05:46.394 INFO:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2019-12-05T06:05:46.542 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T06:05:46.542 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T06:05:46.542 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=0, sparse=0 2019-12-05T06:05:46.545 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T06:05:46.545 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2019-12-05T06:05:46.545 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T06:05:46.545 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T06:05:46.545 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T06:05:46.545 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T06:05:46.547 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2019-12-05T06:05:46.547 INFO:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2 2019-12-05T06:05:46.674 INFO:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2019-12-05T06:05:46.786 INFO:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2019-12-05T06:05:46.899 INFO:tasks.ceph:{'osd.1': '/dev/vg_nvme/lv_1', 'osd.0': '/dev/vg_nvme/lv_4', 'osd.3': '/dev/vg_nvme/lv_3', 'osd.2': '/dev/vg_nvme/lv_2'} 2019-12-05T06:05:46.899 INFO:tasks.ceph:{} 2019-12-05T06:05:46.899 INFO:tasks.ceph:osd.3 2019-12-05T06:05:46.899 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com 2019-12-05T06:05:46.899 INFO:teuthology.orchestra.run.smithi016:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2019-12-05T06:05:47.049 INFO:teuthology.orchestra.run.smithi016.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2019-12-05T06:05:47.050 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 attr=2, projid32bit=1 2019-12-05T06:05:47.051 INFO:teuthology.orchestra.run.smithi016.stdout: = crc=1 finobt=0, sparse=0 2019-12-05T06:05:47.051 INFO:teuthology.orchestra.run.smithi016.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2019-12-05T06:05:47.051 INFO:teuthology.orchestra.run.smithi016.stdout: = sunit=0 swidth=0 blks 2019-12-05T06:05:47.051 INFO:teuthology.orchestra.run.smithi016.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2019-12-05T06:05:47.051 INFO:teuthology.orchestra.run.smithi016.stdout:log =internal log bsize=4096 blocks=11443, version=2 2019-12-05T06:05:47.051 INFO:teuthology.orchestra.run.smithi016.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2019-12-05T06:05:47.052 INFO:teuthology.orchestra.run.smithi016.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2019-12-05T06:05:47.054 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi016.front.sepia.ceph.com -o noatime 2019-12-05T06:05:47.054 INFO:teuthology.orchestra.run.smithi016:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-3 2019-12-05T06:05:47.179 INFO:teuthology.orchestra.run.smithi016:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2019-12-05T06:05:47.291 INFO:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:47.465+0000 7fa7d9e65bc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2019-12-05T06:05:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:47.466+0000 7fa7d9e65bc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2019-12-05T06:05:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:47.466+0000 7fa7d9e65bc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-05T06:05:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:47.466+0000 7fa7d9e65bc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-05T06:05:47.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:47.466+0000 7fa7d9e65bc0 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2019-12-05T06:05:47.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:47.467+0000 7fa7d9e65bc0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2019-12-05T06:05:49.768 INFO:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2019-12-05T06:05:49.892 INFO:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:50.034+0000 7f669fa2abc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2019-12-05T06:05:50.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:50.034+0000 7f669fa2abc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2019-12-05T06:05:50.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:50.034+0000 7f669fa2abc0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-05T06:05:50.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:50.034+0000 7f669fa2abc0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-05T06:05:50.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:50.034+0000 7f669fa2abc0 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2019-12-05T06:05:50.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:50.035+0000 7f669fa2abc0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2019-12-05T06:05:52.338 INFO:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2019-12-05T06:05:52.455 INFO:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:52.609+0000 7f4e1348fbc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2019-12-05T06:05:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:52.609+0000 7f4e1348fbc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2019-12-05T06:05:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:52.609+0000 7f4e1348fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-05T06:05:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:52.609+0000 7f4e1348fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-05T06:05:52.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:52.609+0000 7f4e1348fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2019-12-05T06:05:52.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:52.610+0000 7f4e1348fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2019-12-05T06:05:54.911 INFO:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2019-12-05T06:05:55.033 INFO:teuthology.orchestra.run.smithi016:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:05:55.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:55.181+0000 7f338af0fbc0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2019-12-05T06:05:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:55.182+0000 7f338af0fbc0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2019-12-05T06:05:55.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:55.182+0000 7f338af0fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-05T06:05:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:55.182+0000 7f338af0fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-05T06:05:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:55.182+0000 7f338af0fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2019-12-05T06:05:55.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:05:55.183+0000 7f338af0fbc0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2019-12-05T06:05:56.841 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:05:56.846 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:05:57.488 INFO:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2019-12-05T06:05:57.606 INFO:tasks.ceph:Reading keys from all nodes... 2019-12-05T06:05:57.606 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T06:05:57.697 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.POhzAms9J6 2019-12-05T06:05:57.697 INFO:teuthology.orchestra.run.smithi076:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.POhzAms9J6 2019-12-05T06:05:57.813 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.POhzAms9J6 2019-12-05T06:05:58.004 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.POhzAms9J6 is 56B 2019-12-05T06:05:58.089 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.POhzAms9J6 2019-12-05T06:05:58.153 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T06:05:58.306 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.R1FKcWBdTf 2019-12-05T06:05:58.307 INFO:teuthology.orchestra.run.smithi076:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.R1FKcWBdTf 2019-12-05T06:05:58.411 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.R1FKcWBdTf 2019-12-05T06:05:58.553 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.R1FKcWBdTf is 56B 2019-12-05T06:05:58.613 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.R1FKcWBdTf 2019-12-05T06:05:58.670 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T06:05:58.809 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.pqIoyMKhL8 2019-12-05T06:05:58.810 INFO:teuthology.orchestra.run.smithi076:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.pqIoyMKhL8 2019-12-05T06:05:58.897 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.pqIoyMKhL8 2019-12-05T06:05:59.031 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.pqIoyMKhL8 is 56B 2019-12-05T06:05:59.089 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.pqIoyMKhL8 2019-12-05T06:05:59.139 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T06:05:59.276 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.VxB83LaJDH 2019-12-05T06:05:59.277 INFO:teuthology.orchestra.run.smithi076:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.VxB83LaJDH 2019-12-05T06:05:59.361 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.VxB83LaJDH 2019-12-05T06:05:59.491 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.VxB83LaJDH is 56B 2019-12-05T06:05:59.555 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.VxB83LaJDH 2019-12-05T06:05:59.607 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T06:05:59.757 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.mZOnVlLLpe 2019-12-05T06:05:59.758 INFO:teuthology.orchestra.run.smithi076:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.mZOnVlLLpe 2019-12-05T06:05:59.859 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.mZOnVlLLpe 2019-12-05T06:06:00.004 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.mZOnVlLLpe is 56B 2019-12-05T06:06:00.065 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.mZOnVlLLpe 2019-12-05T06:06:00.117 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T06:06:00.255 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.WtkiClmHW4 2019-12-05T06:06:00.255 INFO:teuthology.orchestra.run.smithi076:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.WtkiClmHW4 2019-12-05T06:06:00.344 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.WtkiClmHW4 2019-12-05T06:06:00.484 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.WtkiClmHW4 is 56B 2019-12-05T06:06:00.553 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.WtkiClmHW4 2019-12-05T06:06:00.605 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T06:06:00.697 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.BpIHjb1SMt 2019-12-05T06:06:00.698 INFO:teuthology.orchestra.run.smithi016:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.BpIHjb1SMt 2019-12-05T06:06:00.802 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.BpIHjb1SMt 2019-12-05T06:06:01.003 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.BpIHjb1SMt is 56B 2019-12-05T06:06:01.084 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.BpIHjb1SMt 2019-12-05T06:06:01.168 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T06:06:01.246 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.d4HXuuJQFk 2019-12-05T06:06:01.247 INFO:teuthology.orchestra.run.smithi016:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.d4HXuuJQFk 2019-12-05T06:06:01.352 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.d4HXuuJQFk 2019-12-05T06:06:01.534 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.d4HXuuJQFk is 56B 2019-12-05T06:06:01.620 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.d4HXuuJQFk 2019-12-05T06:06:01.691 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T06:06:01.756 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.wzNkLFrttU 2019-12-05T06:06:01.756 INFO:teuthology.orchestra.run.smithi016:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.wzNkLFrttU 2019-12-05T06:06:01.849 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.wzNkLFrttU 2019-12-05T06:06:02.014 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.wzNkLFrttU is 56B 2019-12-05T06:06:02.107 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.wzNkLFrttU 2019-12-05T06:06:02.187 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T06:06:02.258 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.WppIKIyUCn 2019-12-05T06:06:02.258 INFO:teuthology.orchestra.run.smithi016:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.WppIKIyUCn 2019-12-05T06:06:02.369 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.WppIKIyUCn 2019-12-05T06:06:02.565 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.WppIKIyUCn is 56B 2019-12-05T06:06:02.672 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.WppIKIyUCn 2019-12-05T06:06:02.762 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T06:06:02.917 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.aMJJaxrKlU 2019-12-05T06:06:02.918 INFO:teuthology.orchestra.run.smithi016:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.aMJJaxrKlU 2019-12-05T06:06:03.034 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.aMJJaxrKlU 2019-12-05T06:06:03.223 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.aMJJaxrKlU is 56B 2019-12-05T06:06:03.335 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.aMJJaxrKlU 2019-12-05T06:06:03.416 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T06:06:03.490 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.WBCnYbtrYJ 2019-12-05T06:06:03.491 INFO:teuthology.orchestra.run.smithi016:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.WBCnYbtrYJ 2019-12-05T06:06:03.602 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.WBCnYbtrYJ 2019-12-05T06:06:03.787 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.WBCnYbtrYJ is 56B 2019-12-05T06:06:03.872 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.WBCnYbtrYJ 2019-12-05T06:06:04.044 DEBUG:teuthology.orchestra.remote:smithi016:/etc/ceph/ceph.client.0.keyring is 59B 2019-12-05T06:06:04.123 INFO:tasks.ceph:Adding keys to all mons... 2019-12-05T06:06:04.124 INFO:teuthology.orchestra.run.smithi016:> sudo tee -a /etc/ceph/ceph.keyring 2019-12-05T06:06:04.129 INFO:teuthology.orchestra.run.smithi076:> sudo tee -a /etc/ceph/ceph.keyring 2019-12-05T06:06:04.238 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.x] 2019-12-05T06:06:04.238 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAqnuhdf4stHBAAQXV9weIO5J9Rbp5r5s0L5A== 2019-12-05T06:06:04.238 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.b] 2019-12-05T06:06:04.238 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAqnuhd2r/4MhAA/ftn+625JYsdHFf2vPxo2Q== 2019-12-05T06:06:04.239 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.4] 2019-12-05T06:06:04.239 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAvnuhd0GeEEBAAR9vHMDj89GVoT1Sz/Y8ldA== 2019-12-05T06:06:04.239 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.5] 2019-12-05T06:06:04.239 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAxnuhdev7mMRAA3kBBcQvXVa7Pp0qW5LpfwQ== 2019-12-05T06:06:04.239 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.6] 2019-12-05T06:06:04.239 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQA0nuhdPlQ1GBAAudCCepjRl6T0RRQ7JuD4Ew== 2019-12-05T06:06:04.240 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.7] 2019-12-05T06:06:04.240 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQA2nuhd1CLKNxAANSJHCqvPHVYsuMGc5qkOJw== 2019-12-05T06:06:04.240 INFO:teuthology.orchestra.run.smithi016.stdout:[mgr.y] 2019-12-05T06:06:04.240 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQAqnuhdsbduJxAApQ7oiorcuDHxdsV4k6ZSEg== 2019-12-05T06:06:04.241 INFO:teuthology.orchestra.run.smithi016.stdout:[mds.a] 2019-12-05T06:06:04.241 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQArnuhdmXRuCRAAL4QKn3lLkSLBAOGe8RCwFw== 2019-12-05T06:06:04.241 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.0] 2019-12-05T06:06:04.241 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQA7nuhdqpHJGxAAcEhBneqZtq3PqTJiKcZETw== 2019-12-05T06:06:04.241 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.1] 2019-12-05T06:06:04.242 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQA+nuhditYPAhAAU0weJS7foY47YYV4veA2Yw== 2019-12-05T06:06:04.242 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.2] 2019-12-05T06:06:04.242 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBAnuhdn25VJBAAZiUZ49YLjgMEt4np1VIgzA== 2019-12-05T06:06:04.242 INFO:teuthology.orchestra.run.smithi016.stdout:[osd.3] 2019-12-05T06:06:04.242 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQBDnuhdJ/ffChAANai+DzN+uO5fK8nPE2rEog== 2019-12-05T06:06:04.242 INFO:teuthology.orchestra.run.smithi016.stdout:[client.0] 2019-12-05T06:06:04.243 INFO:teuthology.orchestra.run.smithi016.stdout: key = AQArnuhdpyO4GBAAWUuIfeDZjOucCg09EdUIRA== 2019-12-05T06:06:04.248 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr.x] 2019-12-05T06:06:04.249 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAqnuhdf4stHBAAQXV9weIO5J9Rbp5r5s0L5A== 2019-12-05T06:06:04.249 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.b] 2019-12-05T06:06:04.249 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAqnuhd2r/4MhAA/ftn+625JYsdHFf2vPxo2Q== 2019-12-05T06:06:04.249 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.4] 2019-12-05T06:06:04.249 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAvnuhd0GeEEBAAR9vHMDj89GVoT1Sz/Y8ldA== 2019-12-05T06:06:04.249 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.5] 2019-12-05T06:06:04.249 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAxnuhdev7mMRAA3kBBcQvXVa7Pp0qW5LpfwQ== 2019-12-05T06:06:04.250 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.6] 2019-12-05T06:06:04.250 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA0nuhdPlQ1GBAAudCCepjRl6T0RRQ7JuD4Ew== 2019-12-05T06:06:04.250 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.7] 2019-12-05T06:06:04.250 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA2nuhd1CLKNxAANSJHCqvPHVYsuMGc5qkOJw== 2019-12-05T06:06:04.250 INFO:teuthology.orchestra.run.smithi076.stdout:[mgr.y] 2019-12-05T06:06:04.250 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAqnuhdsbduJxAApQ7oiorcuDHxdsV4k6ZSEg== 2019-12-05T06:06:04.250 INFO:teuthology.orchestra.run.smithi076.stdout:[mds.a] 2019-12-05T06:06:04.250 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQArnuhdmXRuCRAAL4QKn3lLkSLBAOGe8RCwFw== 2019-12-05T06:06:04.251 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.0] 2019-12-05T06:06:04.251 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA7nuhdqpHJGxAAcEhBneqZtq3PqTJiKcZETw== 2019-12-05T06:06:04.251 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.1] 2019-12-05T06:06:04.251 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQA+nuhditYPAhAAU0weJS7foY47YYV4veA2Yw== 2019-12-05T06:06:04.251 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.2] 2019-12-05T06:06:04.252 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBAnuhdn25VJBAAZiUZ49YLjgMEt4np1VIgzA== 2019-12-05T06:06:04.252 INFO:teuthology.orchestra.run.smithi076.stdout:[osd.3] 2019-12-05T06:06:04.252 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQBDnuhdJ/ffChAANai+DzN+uO5fK8nPE2rEog== 2019-12-05T06:06:04.252 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2019-12-05T06:06:04.252 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQArnuhdpyO4GBAAWUuIfeDZjOucCg09EdUIRA== 2019-12-05T06:06:04.254 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-05T06:06:04.258 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-05T06:06:04.418 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-12-05T06:06:04.423 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-12-05T06:06:04.596 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:04.601 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:04.760 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:04.765 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:04.932 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:04.937 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:05.109 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:05.115 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:05.274 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-05T06:06:05.279 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mds 'allow *' --cap osd 'allow *' --cap mon 'allow profile mgr' 2019-12-05T06:06:05.450 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-12-05T06:06:05.455 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mds allow --cap osd 'allow *' --cap mgr 'allow *' --cap mon 'allow *' 2019-12-05T06:06:05.625 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:05.630 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:05.797 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:05.808 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:05.982 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:05.987 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:06.157 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:06.169 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mgr 'allow *' --cap osd 'allow *' --cap mon 'allow *' 2019-12-05T06:06:06.336 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-05T06:06:06.341 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mds allow --cap osd 'allow rwx' --cap mgr 'allow r' --cap mon 'allow rw' 2019-12-05T06:06:06.526 INFO:tasks.ceph:Running mkfs on mon nodes... 2019-12-05T06:06:06.526 INFO:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2019-12-05T06:06:06.648 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-05T06:06:06.821 INFO:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2019-12-05T06:06:06.935 INFO:teuthology.orchestra.run.smithi076:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2019-12-05T06:06:07.023 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-05T06:06:07.143 INFO:teuthology.orchestra.run.smithi076:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2019-12-05T06:06:07.218 INFO:teuthology.orchestra.run.smithi016:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2019-12-05T06:06:07.347 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2019-12-05T06:06:07.521 INFO:teuthology.orchestra.run.smithi016:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2019-12-05T06:06:07.635 INFO:teuthology.orchestra.run.smithi016:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:06:07.640 INFO:teuthology.orchestra.run.smithi076:> rm -- /home/ubuntu/cephtest/ceph.monmap 2019-12-05T06:06:07.733 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2019-12-05T06:06:07.733 INFO:tasks.ceph.mon.b:Restarting daemon 2019-12-05T06:06:07.733 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2019-12-05T06:06:07.740 INFO:tasks.ceph.mon.b:Started 2019-12-05T06:06:07.741 INFO:tasks.ceph.mon.c:Restarting daemon 2019-12-05T06:06:07.741 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2019-12-05T06:06:07.745 INFO:tasks.ceph.mon.c:Started 2019-12-05T06:06:07.746 INFO:tasks.ceph.mon.a:Restarting daemon 2019-12-05T06:06:07.746 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2019-12-05T06:06:07.751 INFO:tasks.ceph.mon.a:Started 2019-12-05T06:06:07.751 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2019-12-05T06:06:07.751 INFO:tasks.ceph.mgr.x:Restarting daemon 2019-12-05T06:06:07.751 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2019-12-05T06:06:07.756 INFO:tasks.ceph.mgr.x:Started 2019-12-05T06:06:07.756 INFO:tasks.ceph.mgr.y:Restarting daemon 2019-12-05T06:06:07.757 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2019-12-05T06:06:07.761 INFO:tasks.ceph.mgr.y:Started 2019-12-05T06:06:07.761 INFO:tasks.ceph:Setting crush tunables to default 2019-12-05T06:06:07.761 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd crush tunables default 2019-12-05T06:06:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.950+0000 7f1699e86700 1 Processor -- start 2019-12-05T06:06:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.950+0000 7f1699e86700 1 -- start start 2019-12-05T06:06:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1699e86700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1699e86700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1694062230 con 0x7f169408c6f0 2019-12-05T06:06:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1699e86700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16940626e0 con 0x7f1694090830 2019-12-05T06:06:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1699e86700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1694096640 con 0x7f169408d2e0 2019-12-05T06:06:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1693fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f169408c6f0 0x7f169408cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58770/0 (socket says 172.21.15.16:58770) 2019-12-05T06:06:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1698c24700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1693fff700 1 -- 172.21.15.16:0/1396766633 learned_addr learned my addr 172.21.15.16:0/1396766633 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2019-12-05T06:06:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:06:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:07.951+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:06:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:08.152+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:08.152+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2019-12-05T06:06:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:08.152+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:06:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:08.152+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:08.152+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.400000 2019-12-05T06:06:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:08.553+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:08.553+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2019-12-05T06:06:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:08.553+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:06:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:08.553+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:08.553+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.800000 2019-12-05T06:06:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:09.354+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:09.355+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2019-12-05T06:06:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:09.355+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:06:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:09.355+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:09.355+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 1.600000 2019-12-05T06:06:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:10.957+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:10.957+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2019-12-05T06:06:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:10.957+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:06:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:10.957+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:13.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:10.957+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 3.200000 2019-12-05T06:06:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.159+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779847601 0 0) 0x7f1684001400 con 0x7f1694090830 2019-12-05T06:06:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.159+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1680002e80 con 0x7f1694090830 2019-12-05T06:06:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.159+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602737064 0 0) 0x7f167c001150 con 0x7f169408c6f0 2019-12-05T06:06:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.159+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16800030b0 con 0x7f169408c6f0 2019-12-05T06:06:13.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.160+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 928295437 0 0) 0x7f167c001130 con 0x7f169408c6f0 2019-12-05T06:06:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.160+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 msgr2=0x7f1694090060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:06:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.160+0000 7f16937fe700 1 --2- 172.21.15.16:0/1396766633 >> v2:172.21.15.16:3300/0 conn(0x7f169408d2e0 0x7f1694090060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.160+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 >> v1:172.21.15.76:6789/0 conn(0x7f1694090830 legacy=0x7f169408f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.160+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f169408ef20 con 0x7f169408c6f0 2019-12-05T06:06:13.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.160+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f167c001680 con 0x7f169408c6f0 2019-12-05T06:06:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.160+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 >> v1:172.21.15.16:6789/0 conn(0x7f169408c6f0 legacy=0x7f169408cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.161+0000 7f16937fe700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 0x7f1680005fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.161+0000 7f16937fe700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1680006680 0x7f1680008b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.161+0000 7f16937fe700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16800093b0 0x7f168000b880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:13.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.161+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168000c030 con 0x7f16800093b0 2019-12-05T06:06:13.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.161+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f168000c260 con 0x7f1680003b90 2019-12-05T06:06:13.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.161+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f168000c490 con 0x7f1680006680 2019-12-05T06:06:13.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.161+0000 7f1693fff700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 0x7f1680005fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.161+0000 7f1698c24700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16800093b0 0x7f168000b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.161+0000 7f168bfff700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1680006680 0x7f1680008b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.162+0000 7f1693fff700 1 -- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1680006680 msgr2=0x7f1680008b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:13.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.162+0000 7f1693fff700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1680006680 0x7f1680008b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:13.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.162+0000 7f1693fff700 1 -- 172.21.15.16:0/1396766633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16800093b0 msgr2=0x7f168000b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.162+0000 7f1693fff700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16800093b0 0x7f168000b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.162+0000 7f1693fff700 1 -- 172.21.15.16:0/1396766633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f168000c740 con 0x7f1680003b90 2019-12-05T06:06:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.162+0000 7f1698c24700 1 -- 172.21.15.16:0/1396766633 reap_dead start 2019-12-05T06:06:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.162+0000 7f1693fff700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 0x7f1680005fd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f167c00f910 tx=0x7f167c00f790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.162+0000 7f1693fff700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 0x7f1680005fd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f167c00f910 tx=0x7f167c00f790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.162+0000 7f1693fff700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 0x7f1680005fd0 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f167c00f910 tx=0x7f167c00f790).ready entity=mon.1 client_cookie=26200956ea0ae28e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.164+0000 7f1693fff700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 0x7f1680005fd0 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f167c00f910 tx=0x7f167c00f790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.164+0000 7f1693fff700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 0x7f1680005fd0 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f167c00f910 tx=0x7f167c00f790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.164+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f167c001130 con 0x7f1680003b90 2019-12-05T06:06:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.164+0000 7f1693fff700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 0x7f1680005fd0 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7f167c00f910 tx=0x7f167c00f790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.164+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f167c004430 con 0x7f1680003b90 2019-12-05T06:06:13.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.164+0000 7f16937fe700 1 -- 172.21.15.16:0/1396766633 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f167c004a80 con 0x7f1680003b90 2019-12-05T06:06:13.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.164+0000 7f1699e86700 1 -- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 msgr2=0x7f1680005fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:13.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.164+0000 7f1699e86700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 0x7f1680005fd0 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7f167c00f910 tx=0x7f167c00f790).stop 2019-12-05T06:06:13.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.166+0000 7f1699e86700 1 -- 172.21.15.16:0/1396766633 shutdown_connections 2019-12-05T06:06:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.166+0000 7f1699e86700 1 --2- 172.21.15.16:0/1396766633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1680003b90 0x7f1680005fd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.166+0000 7f1699e86700 1 -- 172.21.15.16:0/1396766633 shutdown_connections 2019-12-05T06:06:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.166+0000 7f1699e86700 1 -- 172.21.15.16:0/1396766633 wait complete. 2019-12-05T06:06:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.166+0000 7f1699e86700 1 -- 172.21.15.16:0/1396766633 >> 172.21.15.16:0/1396766633 conn(0x7f1694061960 msgr2=0x7f16940a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.172+0000 7f1699e86700 1 Processor -- start 2019-12-05T06:06:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.172+0000 7f1699e86700 1 -- start start 2019-12-05T06:06:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1699e86700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f169407c050 0x7f169407c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1699e86700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1694082db0 0x7f169407c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:13.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1699e86700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f169407f2d0 con 0x7f1694083580 2019-12-05T06:06:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1699e86700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f169407ebd0 con 0x7f1694082db0 2019-12-05T06:06:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1699e86700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1694080bf0 con 0x7f169407c050 2019-12-05T06:06:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1693fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f169407c050 0x7f169407c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1693fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f169407c050 0x7f169407c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:49412/0 (socket says 172.21.15.16:49412) 2019-12-05T06:06:13.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1693fff700 1 -- 172.21.15.16:0/2244624775 learned_addr learned my addr 172.21.15.16:0/2244624775 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> v2:172.21.15.76:3300/0 conn(0x7f1694082db0 0x7f169407c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1693fff700 1 -- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f169407c050 msgr2=0x7f169407c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:06:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1693fff700 1 -- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f169407c050 msgr2=0x7f169407c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:06:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.174+0000 7f1693fff700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f169407c050 0x7f169407c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.175+0000 7f1693fff700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f169407c050 0x7f169407c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:06:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.175+0000 7f1698c24700 1 -- 172.21.15.16:0/2244624775 >> v2:172.21.15.76:3300/0 conn(0x7f1694082db0 msgr2=0x7f169407c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2019-12-05T06:06:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.175+0000 7f1698c24700 1 -- 172.21.15.16:0/2244624775 >> v2:172.21.15.76:3300/0 conn(0x7f1694082db0 msgr2=0x7f169407c980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:06:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.175+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> v2:172.21.15.76:3300/0 conn(0x7f1694082db0 0x7f169407c980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.175+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> v2:172.21.15.76:3300/0 conn(0x7f1694082db0 0x7f169407c980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:06:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957094275 0 0) 0x7f1684006560 con 0x7f1694083580 2019-12-05T06:06:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1678002e80 con 0x7f1694083580 2019-12-05T06:06:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 <== mon.1 v1:172.21.15.76:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3300839318 0 0) 0x7f1684005910 con 0x7f1694083580 2019-12-05T06:06:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 --> v1:172.21.15.76:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16780032e0 con 0x7f1694083580 2019-12-05T06:06:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 <== mon.1 v1:172.21.15.76:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f1684005e80 con 0x7f1694083580 2019-12-05T06:06:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f169407c050 msgr2=0x7f169407c440 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:06:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f169407c050 0x7f169407c440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 >> v2:172.21.15.76:3300/0 conn(0x7f1694082db0 msgr2=0x7f169407c980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:06:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 --2- 172.21.15.16:0/2244624775 >> v2:172.21.15.76:3300/0 conn(0x7f1694082db0 0x7f169407c980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 >> v1:172.21.15.76:6789/0 conn(0x7f1694083580 legacy=0x7f16940825b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16780033a0 0x7f16780037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.177+0000 7f16917fa700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1678003cf0 0x7f16780041b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:13.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f16917fa700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167800bb60 con 0x7f1678003cf0 2019-12-05T06:06:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167800bd50 con 0x7f16780033a0 2019-12-05T06:06:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f167800e150 con 0x7f16780048c0 2019-12-05T06:06:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f168bfff700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1678003cf0 0x7f16780041b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f1693fff700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16780033a0 0x7f16780037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:13.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f1698c24700 1 -- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16780033a0 msgr2=0x7f16780037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:13.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16780033a0 0x7f16780037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:13.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f1698c24700 1 -- 172.21.15.16:0/2244624775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1678003cf0 msgr2=0x7f16780041b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:13.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1678003cf0 0x7f16780041b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:13.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.178+0000 7f1698c24700 1 -- 172.21.15.16:0/2244624775 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167800e400 con 0x7f16780048c0 2019-12-05T06:06:13.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f168bfff700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1678003cf0 0x7f16780041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:13.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f1699e86700 1 -- 172.21.15.16:0/2244624775 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f169407cec0 con 0x7f16780048c0 2019-12-05T06:06:13.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f1699e86700 1 -- 172.21.15.16:0/2244624775 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1694081cb0 con 0x7f16780048c0 2019-12-05T06:06:13.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=READY pgs=6 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:13.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f168c00a6d0 con 0x7f16780048c0 2019-12-05T06:06:13.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f168c00a900 con 0x7f16780048c0 2019-12-05T06:06:13.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f168c0014e0 con 0x7f16780048c0 2019-12-05T06:06:13.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 1) v1 ==== 349+0+0 (secure 0 0 0) 0x7f168c001970 con 0x7f16780048c0 2019-12-05T06:06:13.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.179+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f168c0023b0 con 0x7f16780048c0 2019-12-05T06:06:13.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.228+0000 7f1699e86700 1 -- 172.21.15.16:0/2244624775 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f166c002e60 con 0x7f16780048c0 2019-12-05T06:06:13.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.235+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.235+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+66539 (secure 0 0 0) 0x7f168c001760 con 0x7f16780048c0 2019-12-05T06:06:13.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.395+0000 7f1699e86700 1 -- 172.21.15.16:0/2244624775 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"profile": "default", "prefix": "osd crush tunables"} v 0) v1 -- 0x7f166c001d20 con 0x7f16780048c0 2019-12-05T06:06:13.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.405+0000 7f1698c24700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=THROTTLE_DONE pgs=6 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:13.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.406+0000 7f16917fa700 1 -- 172.21.15.16:0/2244624775 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"profile": "default", "prefix": "osd crush tunables"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f168c001760 con 0x7f16780048c0 2019-12-05T06:06:13.409 INFO:teuthology.orchestra.run.smithi016.stderr:adjusted tunables profile to default 2019-12-05T06:06:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.412+0000 7f1699e86700 1 -- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 msgr2=0x7f1678004d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.412+0000 7f1699e86700 1 --2- 172.21.15.16:0/2244624775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16780048c0 0x7f1678004d30 secure :-1 s=READY pgs=6 cs=0 l=1 rx=0x7f168c054640 tx=0x7f168c054440).stop 2019-12-05T06:06:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.412+0000 7f1698c24700 1 -- 172.21.15.16:0/2244624775 reap_dead start 2019-12-05T06:06:13.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.412+0000 7f1699e86700 1 -- 172.21.15.16:0/2244624775 shutdown_connections 2019-12-05T06:06:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.412+0000 7f1699e86700 1 -- 172.21.15.16:0/2244624775 shutdown_connections 2019-12-05T06:06:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.412+0000 7f1699e86700 1 -- 172.21.15.16:0/2244624775 wait complete. 2019-12-05T06:06:13.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:13.412+0000 7f1699e86700 1 -- 172.21.15.16:0/2244624775 >> 172.21.15.16:0/2244624775 conn(0x7f16940a00d0 msgr2=0x7f1694073100 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:13.421 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2019-12-05T06:06:13.422 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T06:06:13.498 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.4ct0Xb9ZnX 2019-12-05T06:06:13.498 INFO:teuthology.orchestra.run.smithi076:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.4ct0Xb9ZnX 2019-12-05T06:06:13.595 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.4ct0Xb9ZnX 2019-12-05T06:06:13.742 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.4ct0Xb9ZnX is 37B 2019-12-05T06:06:13.796 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.4ct0Xb9ZnX 2019-12-05T06:06:13.841 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T06:06:13.960 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.qo87KTaaEE 2019-12-05T06:06:13.961 INFO:teuthology.orchestra.run.smithi076:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.qo87KTaaEE 2019-12-05T06:06:14.021 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.qo87KTaaEE 2019-12-05T06:06:14.142 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.qo87KTaaEE is 37B 2019-12-05T06:06:14.193 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.qo87KTaaEE 2019-12-05T06:06:14.236 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T06:06:14.365 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.NszfFJs2eH 2019-12-05T06:06:14.365 INFO:teuthology.orchestra.run.smithi076:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.NszfFJs2eH 2019-12-05T06:06:14.434 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.NszfFJs2eH 2019-12-05T06:06:14.568 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.NszfFJs2eH is 37B 2019-12-05T06:06:14.617 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.NszfFJs2eH 2019-12-05T06:06:14.660 INFO:teuthology.orchestra.run.smithi076:> mktemp 2019-12-05T06:06:14.758 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.wZi9C4mKC2 2019-12-05T06:06:14.759 INFO:teuthology.orchestra.run.smithi076:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.wZi9C4mKC2 2019-12-05T06:06:14.928 INFO:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.wZi9C4mKC2 2019-12-05T06:06:15.084 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.wZi9C4mKC2 is 37B 2019-12-05T06:06:15.149 INFO:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.wZi9C4mKC2 2019-12-05T06:06:15.202 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T06:06:15.294 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.mSWdvwRnS3 2019-12-05T06:06:15.294 INFO:teuthology.orchestra.run.smithi016:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.mSWdvwRnS3 2019-12-05T06:06:15.408 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.mSWdvwRnS3 2019-12-05T06:06:15.602 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.mSWdvwRnS3 is 37B 2019-12-05T06:06:15.694 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.mSWdvwRnS3 2019-12-05T06:06:15.772 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T06:06:15.850 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.VZYCgR185k 2019-12-05T06:06:15.850 INFO:teuthology.orchestra.run.smithi016:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.VZYCgR185k 2019-12-05T06:06:15.940 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.VZYCgR185k 2019-12-05T06:06:16.073 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.VZYCgR185k is 37B 2019-12-05T06:06:16.135 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.VZYCgR185k 2019-12-05T06:06:16.203 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T06:06:16.291 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.suC79LJYXu 2019-12-05T06:06:16.291 INFO:teuthology.orchestra.run.smithi016:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.suC79LJYXu 2019-12-05T06:06:16.426 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.suC79LJYXu 2019-12-05T06:06:16.645 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.suC79LJYXu is 37B 2019-12-05T06:06:16.732 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.suC79LJYXu 2019-12-05T06:06:16.821 INFO:teuthology.orchestra.run.smithi016:> mktemp 2019-12-05T06:06:16.893 INFO:teuthology.orchestra.run.smithi016.stdout:/tmp/tmp.y3BVcsH3ag 2019-12-05T06:06:16.893 INFO:teuthology.orchestra.run.smithi016:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.y3BVcsH3ag 2019-12-05T06:06:16.982 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 0666 /tmp/tmp.y3BVcsH3ag 2019-12-05T06:06:17.126 DEBUG:teuthology.orchestra.remote:smithi016:/tmp/tmp.y3BVcsH3ag is 37B 2019-12-05T06:06:17.187 INFO:teuthology.orchestra.run.smithi016:> rm -fr /tmp/tmp.y3BVcsH3ag 2019-12-05T06:06:17.245 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 9d92c1cd-0a79-4b80-a533-62e750cbe008 0 2019-12-05T06:06:17.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.575+0000 7f569780a700 1 Processor -- start 2019-12-05T06:06:17.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.575+0000 7f569780a700 1 -- start start 2019-12-05T06:06:17.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.577+0000 7f569780a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5690090830 0x7f569008d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.577+0000 7f569780a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5690062230 con 0x7f569008c6f0 2019-12-05T06:06:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.577+0000 7f569780a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5690090480 con 0x7f569008f860 2019-12-05T06:06:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.577+0000 7f569780a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5690096640 con 0x7f5690090830 2019-12-05T06:06:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.577+0000 7f56955a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5690090830 0x7f569008d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.577+0000 7f5695da7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f569008c6f0 0x7f569008cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58842/0 (socket says 172.21.15.16:58842) 2019-12-05T06:06:17.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.577+0000 7f5695da7700 1 -- 172.21.15.16:0/2483591159 learned_addr learned my addr 172.21.15.16:0/2483591159 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.578+0000 7f5694da5700 1 -- 172.21.15.16:0/2483591159 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2261755651 0 0) 0x7f56880014d0 con 0x7f569008c6f0 2019-12-05T06:06:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.578+0000 7f5694da5700 1 -- 172.21.15.16:0/2483591159 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5678002e80 con 0x7f569008c6f0 2019-12-05T06:06:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.578+0000 7f5694da5700 1 -- 172.21.15.16:0/2483591159 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11670428 0 0) 0x7f56800009b0 con 0x7f569008f860 2019-12-05T06:06:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.578+0000 7f56955a6700 1 -- 172.21.15.16:0/2483591159 >> v1:172.21.15.76:6789/0 conn(0x7f569008f860 legacy=0x7f569008fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.578+0000 7f56955a6700 1 -- 172.21.15.16:0/2483591159 >> v1:172.21.15.16:6789/0 conn(0x7f569008c6f0 legacy=0x7f569008cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:17.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.578+0000 7f56955a6700 1 -- 172.21.15.16:0/2483591159 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f569008ef20 con 0x7f5690090830 2019-12-05T06:06:17.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.578+0000 7f56955a6700 1 --2- 172.21.15.16:0/2483591159 >> v2:172.21.15.16:3300/0 conn(0x7f5690090830 0x7f569008d2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f568400af50 tx=0x7f568400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.579+0000 7f56955a6700 1 --2- 172.21.15.16:0/2483591159 >> v2:172.21.15.16:3300/0 conn(0x7f5690090830 0x7f569008d2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f568400af50 tx=0x7f568400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.579+0000 7f56955a6700 1 --2- 172.21.15.16:0/2483591159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5690090830 0x7f569008d2e0 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f568400af50 tx=0x7f568400b760).ready entity=mon.0 client_cookie=18dca1ce16938ab3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:17.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.579+0000 7f56955a6700 1 --2- 172.21.15.16:0/2483591159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5690090830 0x7f569008d2e0 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f568400af50 tx=0x7f568400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.579+0000 7f56955a6700 1 --2- 172.21.15.16:0/2483591159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5690090830 0x7f569008d2e0 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f568400af50 tx=0x7f568400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.579+0000 7f5694da5700 1 -- 172.21.15.16:0/2483591159 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5684000960 con 0x7f5690090830 2019-12-05T06:06:17.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.579+0000 7f56955a6700 1 --2- 172.21.15.16:0/2483591159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5690090830 0x7f569008d2e0 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f568400af50 tx=0x7f568400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.579+0000 7f5694da5700 1 -- 172.21.15.16:0/2483591159 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5684009860 con 0x7f5690090830 2019-12-05T06:06:17.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.579+0000 7f5694da5700 1 -- 172.21.15.16:0/2483591159 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5684009f30 con 0x7f5690090830 2019-12-05T06:06:17.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.579+0000 7f569780a700 1 -- 172.21.15.16:0/2483591159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5690090830 msgr2=0x7f569008d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:17.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.579+0000 7f569780a700 1 --2- 172.21.15.16:0/2483591159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5690090830 0x7f569008d2e0 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f568400af50 tx=0x7f568400b760).stop 2019-12-05T06:06:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.581+0000 7f569780a700 1 -- 172.21.15.16:0/2483591159 shutdown_connections 2019-12-05T06:06:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.581+0000 7f569780a700 1 --2- 172.21.15.16:0/2483591159 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5690090830 0x7f569008d2e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.582+0000 7f569780a700 1 -- 172.21.15.16:0/2483591159 shutdown_connections 2019-12-05T06:06:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.582+0000 7f569780a700 1 -- 172.21.15.16:0/2483591159 wait complete. 2019-12-05T06:06:17.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.582+0000 7f569780a700 1 -- 172.21.15.16:0/2483591159 >> 172.21.15.16:0/2483591159 conn(0x7f5690061960 msgr2=0x7f56900a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:17.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.587+0000 7f569780a700 1 Processor -- start 2019-12-05T06:06:17.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.587+0000 7f569780a700 1 -- start start 2019-12-05T06:06:17.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.589+0000 7f569780a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f56900833a0 0x7f569014c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:17.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.589+0000 7f569780a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5690083010 con 0x7f569014c390 2019-12-05T06:06:17.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.589+0000 7f569780a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f569007c360 con 0x7f5690082bd0 2019-12-05T06:06:17.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.589+0000 7f569780a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f569007ddd0 con 0x7f56900833a0 2019-12-05T06:06:17.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.590+0000 7f5695da7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f569014c390 0x7f56900823d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58848/0 (socket says 172.21.15.16:58848) 2019-12-05T06:06:17.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.590+0000 7f5695da7700 1 -- 172.21.15.16:0/1208713673 learned_addr learned my addr 172.21.15.16:0/1208713673 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:17.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.590+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> v2:172.21.15.76:3300/0 conn(0x7f56900833a0 0x7f569014c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:17.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.590+0000 7f568e7fc700 1 -- 172.21.15.16:0/1208713673 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 627268388 0 0) 0x7f56880014d0 con 0x7f569014c390 2019-12-05T06:06:17.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.590+0000 7f568e7fc700 1 -- 172.21.15.16:0/1208713673 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5670002e80 con 0x7f569014c390 2019-12-05T06:06:17.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.591+0000 7f56955a6700 1 -- 172.21.15.16:0/1208713673 >> v1:172.21.15.76:6789/0 conn(0x7f5690082bd0 legacy=0x7f5690156080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:17.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.591+0000 7f56955a6700 1 -- 172.21.15.16:0/1208713673 >> v1:172.21.15.16:6789/0 conn(0x7f569014c390 legacy=0x7f56900823d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:17.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.591+0000 7f568e7fc700 1 -- 172.21.15.16:0/1208713673 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3354996101 0 0) 0x7f56880016d0 con 0x7f569014c390 2019-12-05T06:06:17.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.591+0000 7f56955a6700 1 -- 172.21.15.16:0/1208713673 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5690081ad0 con 0x7f56900833a0 2019-12-05T06:06:17.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.591+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> v2:172.21.15.76:3300/0 conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.591+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> v2:172.21.15.76:3300/0 conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.591+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.591+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.592+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.592+0000 7f568e7fc700 1 -- 172.21.15.16:0/1208713673 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5684000940 con 0x7f56900833a0 2019-12-05T06:06:17.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.592+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.592+0000 7f568e7fc700 1 -- 172.21.15.16:0/1208713673 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5684009980 con 0x7f56900833a0 2019-12-05T06:06:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.592+0000 7f569780a700 1 -- 172.21.15.16:0/1208713673 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5690081ad0 con 0x7f56900833a0 2019-12-05T06:06:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.592+0000 7f568e7fc700 1 -- 172.21.15.16:0/1208713673 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f568400a050 con 0x7f56900833a0 2019-12-05T06:06:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.592+0000 7f569780a700 1 -- 172.21.15.16:0/1208713673 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5690081ad0 con 0x7f56900833a0 2019-12-05T06:06:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.593+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.594+0000 7f568e7fc700 1 -- 172.21.15.16:0/1208713673 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5684009b70 con 0x7f56900833a0 2019-12-05T06:06:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.594+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.594+0000 7f568e7fc700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5670023680 0x7f5670025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.594+0000 7f568e7fc700 1 -- 172.21.15.16:0/1208713673 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5684045de0 con 0x7f56900833a0 2019-12-05T06:06:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.594+0000 7f5695da7700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5670023680 0x7f5670025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:17.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.595+0000 7f5695da7700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5670023680 0x7f5670025af0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:17.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.607+0000 7f569780a700 1 -- 172.21.15.16:0/1208713673 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5690081ad0 con 0x7f56900833a0 2019-12-05T06:06:17.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.616+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.616+0000 7f568e7fc700 1 -- 172.21.15.16:0/1208713673 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5684041470 con 0x7f56900833a0 2019-12-05T06:06:17.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.892+0000 7f569780a700 1 -- 172.21.15.16:0/1208713673 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9d92c1cd-0a79-4b80-a533-62e750cbe008", "id": 0} v 0) v1 -- 0x7f5690081ad0 con 0x7f56900833a0 2019-12-05T06:06:17.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.899+0000 7f56955a6700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:17.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.899+0000 7f568e7fc700 1 -- 172.21.15.16:0/1208713673 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9d92c1cd-0a79-4b80-a533-62e750cbe008", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5684041470 con 0x7f56900833a0 2019-12-05T06:06:17.905 INFO:teuthology.orchestra.run.smithi016.stdout:0 2019-12-05T06:06:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.910+0000 7f569780a700 1 -- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5670023680 msgr2=0x7f5670025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.910+0000 7f569780a700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5670023680 0x7f5670025af0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.910+0000 7f569780a700 1 -- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 msgr2=0x7f569014c780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.910+0000 7f569780a700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 0x7f569014c780 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7f56840103c0 tx=0x7f5684010760).stop 2019-12-05T06:06:17.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.910+0000 7f569780a700 1 -- 172.21.15.16:0/1208713673 shutdown_connections 2019-12-05T06:06:17.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.910+0000 7f569780a700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5670023680 0x7f5670025af0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:17.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.910+0000 7f569780a700 1 --2- 172.21.15.16:0/1208713673 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f56900833a0 0x7f569014c780 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:17.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.910+0000 7f569780a700 1 -- 172.21.15.16:0/1208713673 shutdown_connections 2019-12-05T06:06:17.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.910+0000 7f569780a700 1 -- 172.21.15.16:0/1208713673 wait complete. 2019-12-05T06:06:17.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:17.910+0000 7f569780a700 1 -- 172.21.15.16:0/1208713673 >> 172.21.15.16:0/1208713673 conn(0x7f5690078040 msgr2=0x7f56900a0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:17.919 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new cde687f3-983d-48c2-ad19-7332d6ffc944 1 2019-12-05T06:06:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.173+0000 7fc904f7d700 1 Processor -- start 2019-12-05T06:06:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.173+0000 7fc904f7d700 1 -- start start 2019-12-05T06:06:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc904f7d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc90008c6f0 0x7fc90008cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc904f7d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc904f7d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc90008f860 0x7fc90008fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:18.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc904f7d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc900096640 con 0x7fc90008f860 2019-12-05T06:06:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc904f7d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90008e890 con 0x7fc90008c6f0 2019-12-05T06:06:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc904f7d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc90008db90 con 0x7fc900090830 2019-12-05T06:06:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc8ff59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc90008f860 0x7fc90008fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc8fe59c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc8ff59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc90008f860 0x7fc90008fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45458/0 (socket says 172.21.15.16:45458) 2019-12-05T06:06:18.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc8fe59c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:49450/0 (socket says 172.21.15.16:49450) 2019-12-05T06:06:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc8fed9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc90008c6f0 0x7fc90008cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.175+0000 7fc8ff59e700 1 -- 172.21.15.16:0/3111914816 learned_addr learned my addr 172.21.15.16:0/3111914816 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.176+0000 7fc8fe59c700 1 -- 172.21.15.16:0/3111914816 >> v2:172.21.15.76:3300/0 conn(0x7fc90008c6f0 msgr2=0x7fc90008cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.176+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/3111914816 >> v2:172.21.15.76:3300/0 conn(0x7fc90008c6f0 0x7fc90008cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.176+0000 7fc8fe59c700 1 -- 172.21.15.16:0/3111914816 >> v2:172.21.15.16:3300/0 conn(0x7fc90008f860 msgr2=0x7fc90008fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.176+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/3111914816 >> v2:172.21.15.16:3300/0 conn(0x7fc90008f860 0x7fc90008fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.176+0000 7fc8fe59c700 1 -- 172.21.15.16:0/3111914816 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90008ef20 con 0x7fc900090830 2019-12-05T06:06:18.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.176+0000 7fc8ff59e700 1 --2- 172.21.15.16:0/3111914816 >> v2:172.21.15.16:3300/0 conn(0x7fc90008f860 0x7fc90008fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.176+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/3111914816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc8e800af50 tx=0x7fc8e800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.177+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/3111914816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc8e800af50 tx=0x7fc8e800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.177+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/3111914816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 secure :-1 s=READY pgs=7 cs=0 l=1 rx=0x7fc8e800af50 tx=0x7fc8e800b250).ready entity=mon.2 client_cookie=71f059a1ed598bf2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.177+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/3111914816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7fc8e800af50 tx=0x7fc8e800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.177+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/3111914816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7fc8e800af50 tx=0x7fc8e800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.177+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/3111914816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 secure :-1 s=THROTTLE_DONE pgs=7 cs=0 l=1 rx=0x7fc8e800af50 tx=0x7fc8e800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.177+0000 7fc8fdd9b700 1 -- 172.21.15.16:0/3111914816 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8e8007210 con 0x7fc900090830 2019-12-05T06:06:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.177+0000 7fc8fdd9b700 1 -- 172.21.15.16:0/3111914816 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e80099b0 con 0x7fc900090830 2019-12-05T06:06:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.177+0000 7fc8fdd9b700 1 -- 172.21.15.16:0/3111914816 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8e800a080 con 0x7fc900090830 2019-12-05T06:06:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.177+0000 7fc904f7d700 1 -- 172.21.15.16:0/3111914816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 msgr2=0x7fc90008d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.177+0000 7fc904f7d700 1 --2- 172.21.15.16:0/3111914816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 secure :-1 s=READY pgs=7 cs=0 l=1 rx=0x7fc8e800af50 tx=0x7fc8e800b250).stop 2019-12-05T06:06:18.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.179+0000 7fc904f7d700 1 -- 172.21.15.16:0/3111914816 shutdown_connections 2019-12-05T06:06:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.179+0000 7fc904f7d700 1 --2- 172.21.15.16:0/3111914816 >> v2:172.21.15.16:3300/0 conn(0x7fc90008f860 0x7fc90008fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.179+0000 7fc904f7d700 1 --2- 172.21.15.16:0/3111914816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc900090830 0x7fc90008d020 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.179+0000 7fc904f7d700 1 --2- 172.21.15.16:0/3111914816 >> v2:172.21.15.76:3300/0 conn(0x7fc90008c6f0 0x7fc90008cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.180+0000 7fc904f7d700 1 -- 172.21.15.16:0/3111914816 shutdown_connections 2019-12-05T06:06:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.180+0000 7fc904f7d700 1 -- 172.21.15.16:0/3111914816 wait complete. 2019-12-05T06:06:18.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.180+0000 7fc904f7d700 1 -- 172.21.15.16:0/3111914816 >> 172.21.15.16:0/3111914816 conn(0x7fc900061960 msgr2=0x7fc9000a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:18.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.185+0000 7fc904f7d700 1 Processor -- start 2019-12-05T06:06:18.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.185+0000 7fc904f7d700 1 -- start start 2019-12-05T06:06:18.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.186+0000 7fc904f7d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc90007b220 0x7fc90007b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.186+0000 7fc904f7d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc9000833a0 0x7fc90007bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.186+0000 7fc904f7d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc900083030 con 0x7fc9000823d0 2019-12-05T06:06:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.186+0000 7fc904f7d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90007ddd0 con 0x7fc90007b220 2019-12-05T06:06:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.186+0000 7fc904f7d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc900081440 con 0x7fc9000833a0 2019-12-05T06:06:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.186+0000 7fc8fed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc90007b220 0x7fc90007b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.187+0000 7fc8ff59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc9000823d0 0x7fc900082830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58864/0 (socket says 172.21.15.16:58864) 2019-12-05T06:06:18.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.187+0000 7fc8ff59e700 1 -- 172.21.15.16:0/1976461446 learned_addr learned my addr 172.21.15.16:0/1976461446 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.187+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/1976461446 >> v2:172.21.15.76:3300/0 conn(0x7fc9000833a0 0x7fc90007bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.187+0000 7fc8fed9d700 1 -- 172.21.15.16:0/1976461446 >> v2:172.21.15.76:3300/0 conn(0x7fc9000833a0 msgr2=0x7fc90007bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.187+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> v2:172.21.15.76:3300/0 conn(0x7fc9000833a0 0x7fc90007bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.187+0000 7fc8fed9d700 1 -- 172.21.15.16:0/1976461446 >> v1:172.21.15.16:6789/0 conn(0x7fc9000823d0 legacy=0x7fc900082830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.187+0000 7fc8fed9d700 1 -- 172.21.15.16:0/1976461446 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900081ad0 con 0x7fc90007b220 2019-12-05T06:06:18.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.187+0000 7fc8f77fe700 1 -- 172.21.15.16:0/1976461446 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1383918028 0 0) 0x7fc8f0007790 con 0x7fc9000823d0 2019-12-05T06:06:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.187+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> v2:172.21.15.16:3300/0 conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.188+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> v2:172.21.15.16:3300/0 conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.188+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.188+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.188+0000 7fc8f77fe700 1 -- 172.21.15.16:0/1976461446 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f8000cc0 con 0x7fc90007b220 2019-12-05T06:06:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.188+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.188+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.188+0000 7fc8f77fe700 1 -- 172.21.15.16:0/1976461446 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f8001010 con 0x7fc90007b220 2019-12-05T06:06:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.188+0000 7fc904f7d700 1 -- 172.21.15.16:0/1976461446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc900081ad0 con 0x7fc90007b220 2019-12-05T06:06:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.188+0000 7fc8f77fe700 1 -- 172.21.15.16:0/1976461446 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f80016e0 con 0x7fc90007b220 2019-12-05T06:06:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.188+0000 7fc904f7d700 1 -- 172.21.15.16:0/1976461446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc900081ad0 con 0x7fc90007b220 2019-12-05T06:06:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.189+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.190+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.190+0000 7fc8f77fe700 1 -- 172.21.15.16:0/1976461446 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc8f8001200 con 0x7fc90007b220 2019-12-05T06:06:18.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.191+0000 7fc8f77fe700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 0x7fc8e4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.191+0000 7fc8f77fe700 1 -- 172.21.15.16:0/1976461446 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7fc8f8046090 con 0x7fc90007b220 2019-12-05T06:06:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.191+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 0x7fc8e4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.192+0000 7fc8fe59c700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 0x7fc8e4025ac0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:18.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.205+0000 7fc904f7d700 1 -- 172.21.15.16:0/1976461446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900081ad0 con 0x7fc90007b220 2019-12-05T06:06:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.215+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.215+0000 7fc8f77fe700 1 -- 172.21.15.16:0/1976461446 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc8f80095d0 con 0x7fc90007b220 2019-12-05T06:06:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.494+0000 7fc904f7d700 1 -- 172.21.15.16:0/1976461446 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cde687f3-983d-48c2-ad19-7332d6ffc944", "id": 1} v 0) v1 -- 0x7fc900081ad0 con 0x7fc90007b220 2019-12-05T06:06:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.498+0000 7fc8fed9d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.498+0000 7fc8f77fe700 1 -- 172.21.15.16:0/1976461446 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cde687f3-983d-48c2-ad19-7332d6ffc944", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc8f80011e0 con 0x7fc90007b220 2019-12-05T06:06:18.499 INFO:teuthology.orchestra.run.smithi016.stdout:1 2019-12-05T06:06:18.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 -- 172.21.15.16:0/1976461446 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 msgr2=0x7fc8e4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:18.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 0x7fc8e4025ac0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:18.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 -- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 msgr2=0x7fc90007b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:18.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 0x7fc90007b610 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7fc8f80106e0 tx=0x7fc8f8010540).stop 2019-12-05T06:06:18.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 -- 172.21.15.16:0/1976461446 shutdown_connections 2019-12-05T06:06:18.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 --2- 172.21.15.16:0/1976461446 >> v2:172.21.15.76:3300/0 conn(0x7fc9000833a0 0x7fc90007bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:18.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 0x7fc8e4025ac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:18.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 --2- 172.21.15.16:0/1976461446 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90007b220 0x7fc90007b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:18.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 -- 172.21.15.16:0/1976461446 shutdown_connections 2019-12-05T06:06:18.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 -- 172.21.15.16:0/1976461446 wait complete. 2019-12-05T06:06:18.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.505+0000 7fc904f7d700 1 -- 172.21.15.16:0/1976461446 >> 172.21.15.16:0/1976461446 conn(0x7fc900078040 msgr2=0x7fc9000a0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:18.518 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new b42d078c-c688-49ea-b0d8-14fbbf31dd53 2 2019-12-05T06:06:18.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.737+0000 7f5f9708a700 1 Processor -- start 2019-12-05T06:06:18.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.737+0000 7f5f9708a700 1 -- start start 2019-12-05T06:06:18.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.739+0000 7f5f9708a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5f90090830 0x7f5f9008d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:18.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.739+0000 7f5f9708a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5f9008f860 0x7f5f9008fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:18.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.739+0000 7f5f9708a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f90062230 con 0x7f5f9008c6f0 2019-12-05T06:06:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.739+0000 7f5f9708a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f90096640 con 0x7f5f9008f860 2019-12-05T06:06:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.739+0000 7f5f9708a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9008e890 con 0x7f5f90090830 2019-12-05T06:06:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.739+0000 7f5f95627700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5f9008c6f0 0x7f5f9008cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58870/0 (socket says 172.21.15.16:58870) 2019-12-05T06:06:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.739+0000 7f5f95627700 1 -- 172.21.15.16:0/2555056368 learned_addr learned my addr 172.21.15.16:0/2555056368 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.739+0000 7f5f95e28700 1 --2- 172.21.15.16:0/2555056368 >> v2:172.21.15.76:3300/0 conn(0x7f5f9008f860 0x7f5f9008fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.739+0000 7f5f94e26700 1 --2- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5f90090830 0x7f5f9008d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.740+0000 7f5f8ffff700 1 -- 172.21.15.16:0/2555056368 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 183743271 0 0) 0x7f5f880014d0 con 0x7f5f9008c6f0 2019-12-05T06:06:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.740+0000 7f5f8ffff700 1 -- 172.21.15.16:0/2555056368 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f78002e80 con 0x7f5f9008c6f0 2019-12-05T06:06:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.740+0000 7f5f95e28700 1 -- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5f90090830 msgr2=0x7f5f9008d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.740+0000 7f5f95e28700 1 --2- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5f90090830 0x7f5f9008d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.740+0000 7f5f95e28700 1 -- 172.21.15.16:0/2555056368 >> v1:172.21.15.16:6789/0 conn(0x7f5f9008c6f0 legacy=0x7f5f9008cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.740+0000 7f5f95e28700 1 -- 172.21.15.16:0/2555056368 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9008ef20 con 0x7f5f9008f860 2019-12-05T06:06:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.740+0000 7f5f95e28700 1 --2- 172.21.15.16:0/2555056368 >> v2:172.21.15.76:3300/0 conn(0x7f5f9008f860 0x7f5f9008fc70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5f8000af80 tx=0x7f5f8000b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.740+0000 7f5f95e28700 1 --2- 172.21.15.16:0/2555056368 >> v2:172.21.15.76:3300/0 conn(0x7f5f9008f860 0x7f5f9008fc70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5f8000af80 tx=0x7f5f8000b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.740+0000 7f5f95e28700 1 --2- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f9008f860 0x7f5f9008fc70 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7f5f8000af80 tx=0x7f5f8000b280).ready entity=mon.1 client_cookie=e636ac588602d95c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:18.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.741+0000 7f5f95e28700 1 --2- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f9008f860 0x7f5f9008fc70 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f5f8000af80 tx=0x7f5f8000b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.741+0000 7f5f8ffff700 1 -- 172.21.15.16:0/2555056368 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f80007240 con 0x7f5f9008f860 2019-12-05T06:06:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.741+0000 7f5f95e28700 1 --2- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f9008f860 0x7f5f9008fc70 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f5f8000af80 tx=0x7f5f8000b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.741+0000 7f5f95e28700 1 --2- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f9008f860 0x7f5f9008fc70 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7f5f8000af80 tx=0x7f5f8000b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.741+0000 7f5f8ffff700 1 -- 172.21.15.16:0/2555056368 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f80000b10 con 0x7f5f9008f860 2019-12-05T06:06:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.741+0000 7f5f8ffff700 1 -- 172.21.15.16:0/2555056368 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f80009ea0 con 0x7f5f9008f860 2019-12-05T06:06:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.741+0000 7f5f9708a700 1 -- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f9008f860 msgr2=0x7f5f9008fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.741+0000 7f5f9708a700 1 --2- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f9008f860 0x7f5f9008fc70 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7f5f8000af80 tx=0x7f5f8000b280).stop 2019-12-05T06:06:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.742+0000 7f5f9708a700 1 -- 172.21.15.16:0/2555056368 shutdown_connections 2019-12-05T06:06:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.742+0000 7f5f9708a700 1 --2- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f9008f860 0x7f5f9008fc70 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.742+0000 7f5f9708a700 1 --2- 172.21.15.16:0/2555056368 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5f90090830 0x7f5f9008d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.743+0000 7f5f9708a700 1 -- 172.21.15.16:0/2555056368 shutdown_connections 2019-12-05T06:06:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.743+0000 7f5f9708a700 1 -- 172.21.15.16:0/2555056368 wait complete. 2019-12-05T06:06:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.743+0000 7f5f9708a700 1 -- 172.21.15.16:0/2555056368 >> 172.21.15.16:0/2555056368 conn(0x7f5f90061960 msgr2=0x7f5f900a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:18.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.748+0000 7f5f9708a700 1 Processor -- start 2019-12-05T06:06:18.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.748+0000 7f5f9708a700 1 -- start start 2019-12-05T06:06:18.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.750+0000 7f5f9708a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5f9007cc40 0x7f5f900825b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:18.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.750+0000 7f5f9708a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9007f2d0 con 0x7f5f9007c050 2019-12-05T06:06:18.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.750+0000 7f5f9708a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f9007d330 con 0x7f5f90083580 2019-12-05T06:06:18.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.750+0000 7f5f9708a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f9007ebd0 con 0x7f5f9007cc40 2019-12-05T06:06:18.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.750+0000 7f5f95627700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5f9007c050 0x7f5f9007c440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58876/0 (socket says 172.21.15.16:58876) 2019-12-05T06:06:18.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.750+0000 7f5f95627700 1 -- 172.21.15.16:0/3512165386 learned_addr learned my addr 172.21.15.16:0/3512165386 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:18.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.750+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> v2:172.21.15.16:3300/0 conn(0x7f5f9007cc40 0x7f5f900825b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.750+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4238342719 0 0) 0x7f5f880014d0 con 0x7f5f9007c050 2019-12-05T06:06:18.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.750+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f70002e80 con 0x7f5f9007c050 2019-12-05T06:06:18.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.750+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 869373265 0 0) 0x7f5f80009920 con 0x7f5f90083580 2019-12-05T06:06:18.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f94e26700 1 -- 172.21.15.16:0/3512165386 >> v1:172.21.15.76:6789/0 conn(0x7f5f90083580 legacy=0x7f5f90082af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:18.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f94e26700 1 -- 172.21.15.16:0/3512165386 >> v1:172.21.15.16:6789/0 conn(0x7f5f9007c050 legacy=0x7f5f9007c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:18.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f94e26700 1 -- 172.21.15.16:0/3512165386 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f90081cb0 con 0x7f5f9007cc40 2019-12-05T06:06:18.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> v2:172.21.15.16:3300/0 conn(0x7f5f9007cc40 0x7f5f900825b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5f84010360 tx=0x7f5f84010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> v2:172.21.15.16:3300/0 conn(0x7f5f9007cc40 0x7f5f900825b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5f84010360 tx=0x7f5f84010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f9007cc40 0x7f5f900825b0 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f5f84010360 tx=0x7f5f84010700).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:18.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f94e26700 1 -- 172.21.15.16:0/3512165386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f9007cc40 msgr2=0x7f5f900825b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:06:18.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f94e26700 1 -- 172.21.15.16:0/3512165386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f9007cc40 msgr2=0x7f5f900825b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:06:18.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f9007cc40 0x7f5f900825b0 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f5f84010360 tx=0x7f5f84010700).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:18.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f9007cc40 0x7f5f900825b0 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f5f84010360 tx=0x7f5f84010700).stop 2019-12-05T06:06:18.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f9007cc40 msgr2=0x7f5f900825b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:06:18.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f8dffb700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f9007cc40 0x7f5f900825b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:18.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f8dffb700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5f700030b0 0x7f5f70005510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:18.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f8dffb700 1 --2- 172.21.15.16:0/3512165386 >> v2:172.21.15.76:3300/0 conn(0x7f5f70005a50 0x7f5f70007ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:18.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f70008640 con 0x7f5f70005a50 2019-12-05T06:06:18.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.751+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f70008870 con 0x7f5f700030b0 2019-12-05T06:06:18.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.752+0000 7f5f95627700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5f700030b0 0x7f5f70005510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.752+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> v2:172.21.15.76:3300/0 conn(0x7f5f70005a50 0x7f5f70007ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.752+0000 7f5f94e26700 1 -- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5f700030b0 msgr2=0x7f5f70005510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.752+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5f700030b0 0x7f5f70005510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.752+0000 7f5f94e26700 1 -- 172.21.15.16:0/3512165386 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f7000ad30 con 0x7f5f70005a50 2019-12-05T06:06:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.752+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> v2:172.21.15.76:3300/0 conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.753+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> v2:172.21.15.76:3300/0 conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.753+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.753+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.753+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f84002720 con 0x7f5f70005a50 2019-12-05T06:06:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.753+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.753+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.753+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f84010760 con 0x7f5f70005a50 2019-12-05T06:06:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.753+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f84010e30 con 0x7f5f70005a50 2019-12-05T06:06:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.753+0000 7f5f9708a700 1 -- 172.21.15.16:0/3512165386 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f90081cb0 con 0x7f5f70005a50 2019-12-05T06:06:18.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.753+0000 7f5f9708a700 1 -- 172.21.15.16:0/3512165386 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f90081cb0 con 0x7f5f70005a50 2019-12-05T06:06:18.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.754+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.755+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5f84010310 con 0x7f5f70005a50 2019-12-05T06:06:18.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.755+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.755+0000 7f5f8dffb700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5f7002af20 0x7f5f7002d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:18.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.756+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 984+0+0 (secure 0 0 0) 0x7f5f84047530 con 0x7f5f70005a50 2019-12-05T06:06:18.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.756+0000 7f5f95627700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5f7002af20 0x7f5f7002d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:18.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.756+0000 7f5f95627700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5f7002af20 0x7f5f7002d370 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:18.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.768+0000 7f5f9708a700 1 -- 172.21.15.16:0/3512165386 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f90081cb0 con 0x7f5f70005a50 2019-12-05T06:06:18.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.777+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:18.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:18.777+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5f84002720 con 0x7f5f70005a50 2019-12-05T06:06:19.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.042+0000 7f5f9708a700 1 -- 172.21.15.16:0/3512165386 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b42d078c-c688-49ea-b0d8-14fbbf31dd53", "id": 2} v 0) v1 -- 0x7f5f90081cb0 con 0x7f5f70005a50 2019-12-05T06:06:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.049+0000 7f5f94e26700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.049+0000 7f5f8dffb700 1 -- 172.21.15.16:0/3512165386 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b42d078c-c688-49ea-b0d8-14fbbf31dd53", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f5f84010310 con 0x7f5f70005a50 2019-12-05T06:06:19.055 INFO:teuthology.orchestra.run.smithi016.stdout:2 2019-12-05T06:06:19.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.061+0000 7f5f9708a700 1 -- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5f7002af20 msgr2=0x7f5f7002d370 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.061+0000 7f5f9708a700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5f7002af20 0x7f5f7002d370 crc :-1 s=READY pgs=9 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.061+0000 7f5f9708a700 1 -- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 msgr2=0x7f5f70007ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.061+0000 7f5f9708a700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 0x7f5f70007ef0 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f5f84011b80 tx=0x7f5f84011f20).stop 2019-12-05T06:06:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.061+0000 7f5f95e28700 1 -- 172.21.15.16:0/3512165386 reap_dead start 2019-12-05T06:06:19.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.061+0000 7f5f95e28700 1 -- 172.21.15.16:0/3512165386 reap_dead start 2019-12-05T06:06:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.061+0000 7f5f9708a700 1 -- 172.21.15.16:0/3512165386 shutdown_connections 2019-12-05T06:06:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.061+0000 7f5f9708a700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5f70005a50 0x7f5f70007ef0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.061+0000 7f5f9708a700 1 --2- 172.21.15.16:0/3512165386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5f9007cc40 0x7f5f900825b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.062+0000 7f5f9708a700 1 -- 172.21.15.16:0/3512165386 shutdown_connections 2019-12-05T06:06:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.062+0000 7f5f9708a700 1 -- 172.21.15.16:0/3512165386 wait complete. 2019-12-05T06:06:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.062+0000 7f5f9708a700 1 -- 172.21.15.16:0/3512165386 >> 172.21.15.16:0/3512165386 conn(0x7f5f900a00d0 msgr2=0x7f5f90073100 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:19.076 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 41ff9a23-a332-455d-adb0-7bbc69c833b6 3 2019-12-05T06:06:19.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.270+0000 7fac757d7700 1 Processor -- start 2019-12-05T06:06:19.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.270+0000 7fac757d7700 1 -- start start 2019-12-05T06:06:19.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.271+0000 7fac757d7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac70090830 0x7fac7008d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:19.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.271+0000 7fac757d7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 0x7fac7008fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:19.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.271+0000 7fac757d7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac70062230 con 0x7fac7008c6f0 2019-12-05T06:06:19.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.271+0000 7fac757d7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac70096640 con 0x7fac70090830 2019-12-05T06:06:19.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.271+0000 7fac757d7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac7008e890 con 0x7fac7008f860 2019-12-05T06:06:19.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.272+0000 7fac6effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac70090830 0x7fac7008d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:19.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.272+0000 7fac6effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac70090830 0x7fac7008d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45492/0 (socket says 172.21.15.16:45492) 2019-12-05T06:06:19.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.272+0000 7fac6effd700 1 -- 172.21.15.16:0/533769466 learned_addr learned my addr 172.21.15.16:0/533769466 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:19.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.272+0000 7fac6f7fe700 1 --1- 172.21.15.16:0/533769466 >> v1:172.21.15.76:6789/0 conn(0x7fac7008c6f0 0x7fac7008cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:51958/0 (socket says 172.21.15.16:51958) 2019-12-05T06:06:19.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.272+0000 7fac6ffff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 0x7fac7008fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:19.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6ffff700 1 -- 172.21.15.16:0/533769466 >> v2:172.21.15.16:3300/0 conn(0x7fac70090830 msgr2=0x7fac7008d2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:19.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6ffff700 1 --2- 172.21.15.16:0/533769466 >> v2:172.21.15.16:3300/0 conn(0x7fac70090830 0x7fac7008d2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:19.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6e7fc700 1 -- 172.21.15.16:0/533769466 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 478290082 0 0) 0x7fac68001400 con 0x7fac7008c6f0 2019-12-05T06:06:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6ffff700 1 -- 172.21.15.16:0/533769466 >> v1:172.21.15.76:6789/0 conn(0x7fac7008c6f0 legacy=0x7fac7008cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6ffff700 1 -- 172.21.15.16:0/533769466 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac7008ef20 con 0x7fac7008f860 2019-12-05T06:06:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6effd700 1 --2- 172.21.15.16:0/533769466 >> v2:172.21.15.16:3300/0 conn(0x7fac70090830 0x7fac7008d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6ffff700 1 --2- 172.21.15.16:0/533769466 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 0x7fac7008fca0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac6000af80 tx=0x7fac6000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6ffff700 1 --2- 172.21.15.16:0/533769466 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 0x7fac7008fca0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac6000af80 tx=0x7fac6000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6ffff700 1 --2- 172.21.15.16:0/533769466 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 0x7fac7008fca0 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7fac6000af80 tx=0x7fac6000b790).ready entity=mon.2 client_cookie=1151e510e9e43d55 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6ffff700 1 --2- 172.21.15.16:0/533769466 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 0x7fac7008fca0 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7fac6000af80 tx=0x7fac6000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6ffff700 1 --2- 172.21.15.16:0/533769466 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 0x7fac7008fca0 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7fac6000af80 tx=0x7fac6000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.273+0000 7fac6e7fc700 1 -- 172.21.15.16:0/533769466 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac60000bc0 con 0x7fac7008f860 2019-12-05T06:06:19.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.274+0000 7fac6ffff700 1 --2- 172.21.15.16:0/533769466 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 0x7fac7008fca0 secure :-1 s=THROTTLE_DONE pgs=8 cs=0 l=1 rx=0x7fac6000af80 tx=0x7fac6000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.274+0000 7fac6e7fc700 1 -- 172.21.15.16:0/533769466 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac600097f0 con 0x7fac7008f860 2019-12-05T06:06:19.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.274+0000 7fac6e7fc700 1 -- 172.21.15.16:0/533769466 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac60009f60 con 0x7fac7008f860 2019-12-05T06:06:19.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.274+0000 7fac757d7700 1 -- 172.21.15.16:0/533769466 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 msgr2=0x7fac7008fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.274+0000 7fac757d7700 1 --2- 172.21.15.16:0/533769466 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 0x7fac7008fca0 secure :-1 s=READY pgs=8 cs=0 l=1 rx=0x7fac6000af80 tx=0x7fac6000b790).stop 2019-12-05T06:06:19.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.275+0000 7fac757d7700 1 -- 172.21.15.16:0/533769466 shutdown_connections 2019-12-05T06:06:19.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.275+0000 7fac757d7700 1 --2- 172.21.15.16:0/533769466 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac7008f860 0x7fac7008fca0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:19.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.275+0000 7fac757d7700 1 --2- 172.21.15.16:0/533769466 >> v2:172.21.15.16:3300/0 conn(0x7fac70090830 0x7fac7008d2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:19.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.275+0000 7fac757d7700 1 -- 172.21.15.16:0/533769466 shutdown_connections 2019-12-05T06:06:19.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.275+0000 7fac757d7700 1 -- 172.21.15.16:0/533769466 wait complete. 2019-12-05T06:06:19.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.275+0000 7fac757d7700 1 -- 172.21.15.16:0/533769466 >> 172.21.15.16:0/533769466 conn(0x7fac70061960 msgr2=0x7fac700a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:19.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.279+0000 7fac757d7700 1 Processor -- start 2019-12-05T06:06:19.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.279+0000 7fac757d7700 1 -- start start 2019-12-05T06:06:19.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.280+0000 7fac757d7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fac7012d380 0x7fac70153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:19.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.280+0000 7fac757d7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac700787a0 con 0x7fac7012cc10 2019-12-05T06:06:19.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.280+0000 7fac757d7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac700782b0 con 0x7fac70152c60 2019-12-05T06:06:19.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.280+0000 7fac757d7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac70135e60 con 0x7fac7012d380 2019-12-05T06:06:19.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.280+0000 7fac6ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fac7012cc10 0x7fac70159930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58898/0 (socket says 172.21.15.16:58898) 2019-12-05T06:06:19.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.280+0000 7fac6ffff700 1 -- 172.21.15.16:0/1048222028 learned_addr learned my addr 172.21.15.16:0/1048222028 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.281+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> v2:172.21.15.76:3300/0 conn(0x7fac7012d380 0x7fac70153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.281+0000 7fac67fff700 1 -- 172.21.15.16:0/1048222028 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469452698 0 0) 0x7fac6000aff0 con 0x7fac7012cc10 2019-12-05T06:06:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.281+0000 7fac67fff700 1 -- 172.21.15.16:0/1048222028 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac48002e80 con 0x7fac7012cc10 2019-12-05T06:06:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.281+0000 7fac6effd700 1 -- 172.21.15.16:0/1048222028 >> v1:172.21.15.76:6789/0 conn(0x7fac70152c60 legacy=0x7fac70136df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.281+0000 7fac6effd700 1 -- 172.21.15.16:0/1048222028 >> v1:172.21.15.16:6789/0 conn(0x7fac7012cc10 legacy=0x7fac70159930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.281+0000 7fac6effd700 1 -- 172.21.15.16:0/1048222028 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac701364f0 con 0x7fac7012d380 2019-12-05T06:06:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.281+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> v2:172.21.15.76:3300/0 conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.282+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> v2:172.21.15.76:3300/0 conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.282+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.282+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.282+0000 7fac67fff700 1 -- 172.21.15.16:0/1048222028 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac58000940 con 0x7fac7012d380 2019-12-05T06:06:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.282+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.282+0000 7fac757d7700 1 -- 172.21.15.16:0/1048222028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac701364f0 con 0x7fac7012d380 2019-12-05T06:06:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.282+0000 7fac67fff700 1 -- 172.21.15.16:0/1048222028 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac58001090 con 0x7fac7012d380 2019-12-05T06:06:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.282+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.282+0000 7fac67fff700 1 -- 172.21.15.16:0/1048222028 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac58001280 con 0x7fac7012d380 2019-12-05T06:06:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.282+0000 7fac757d7700 1 -- 172.21.15.16:0/1048222028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac701364f0 con 0x7fac7012d380 2019-12-05T06:06:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.283+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.283+0000 7fac67fff700 1 -- 172.21.15.16:0/1048222028 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac58000c90 con 0x7fac7012d380 2019-12-05T06:06:19.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.283+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.284+0000 7fac67fff700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac48023670 0x7fac48025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:19.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.284+0000 7fac67fff700 1 -- 172.21.15.16:0/1048222028 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1099+0+0 (secure 0 0 0) 0x7fac58045df0 con 0x7fac7012d380 2019-12-05T06:06:19.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.284+0000 7fac6f7fe700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac48023670 0x7fac48025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:19.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.285+0000 7fac6f7fe700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac48023670 0x7fac48025ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:19.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.285+0000 7fac757d7700 1 -- 172.21.15.16:0/1048222028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac701364f0 con 0x7fac7012d380 2019-12-05T06:06:19.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.295+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.295+0000 7fac67fff700 1 -- 172.21.15.16:0/1048222028 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac5802ae10 con 0x7fac7012d380 2019-12-05T06:06:19.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.525+0000 7fac757d7700 1 -- 172.21.15.16:0/1048222028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd new", "uuid": "41ff9a23-a332-455d-adb0-7bbc69c833b6", "id": 3} v 0) v1 -- 0x7fac701364f0 con 0x7fac7012d380 2019-12-05T06:06:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.536+0000 7fac6effd700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.536+0000 7fac67fff700 1 -- 172.21.15.16:0/1048222028 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "41ff9a23-a332-455d-adb0-7bbc69c833b6", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fac58000c90 con 0x7fac7012d380 2019-12-05T06:06:19.539 INFO:teuthology.orchestra.run.smithi016.stdout:3 2019-12-05T06:06:19.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.543+0000 7fac757d7700 1 -- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac48023670 msgr2=0x7fac48025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.543+0000 7fac757d7700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac48023670 0x7fac48025ae0 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:19.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.543+0000 7fac757d7700 1 -- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 msgr2=0x7fac70153050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.543+0000 7fac757d7700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 0x7fac70153050 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7fac58010380 tx=0x7fac58010720).stop 2019-12-05T06:06:19.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.543+0000 7fac757d7700 1 -- 172.21.15.16:0/1048222028 shutdown_connections 2019-12-05T06:06:19.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.543+0000 7fac757d7700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac48023670 0x7fac48025ae0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:19.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.543+0000 7fac757d7700 1 --2- 172.21.15.16:0/1048222028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fac7012d380 0x7fac70153050 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:19.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.544+0000 7fac757d7700 1 -- 172.21.15.16:0/1048222028 shutdown_connections 2019-12-05T06:06:19.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.544+0000 7fac757d7700 1 -- 172.21.15.16:0/1048222028 wait complete. 2019-12-05T06:06:19.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.544+0000 7fac757d7700 1 -- 172.21.15.16:0/1048222028 >> 172.21.15.16:0/1048222028 conn(0x7fac700738f0 msgr2=0x7fac70074ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:19.552 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 7fb2a124-c603-49d9-ad7c-44c975729335 4 2019-12-05T06:06:19.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.775+0000 7efd6b28c700 1 Processor -- start 2019-12-05T06:06:19.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.775+0000 7efd6b28c700 1 -- start start 2019-12-05T06:06:19.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.777+0000 7efd6b28c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7efd6408c6f0 0x7efd6408cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:19.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.777+0000 7efd6b28c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efd6408f860 0x7efd6408fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:19.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.777+0000 7efd6b28c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd64062230 con 0x7efd64090830 2019-12-05T06:06:19.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.777+0000 7efd6b28c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd64096640 con 0x7efd6408f860 2019-12-05T06:06:19.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.777+0000 7efd6b28c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd6408e890 con 0x7efd6408c6f0 2019-12-05T06:06:19.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.777+0000 7efd6a02a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efd6408f860 0x7efd6408fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:19.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.777+0000 7efd6a02a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efd6408f860 0x7efd6408fc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45504/0 (socket says 172.21.15.16:45504) 2019-12-05T06:06:19.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.777+0000 7efd6a02a700 1 -- 172.21.15.16:0/274292248 learned_addr learned my addr 172.21.15.16:0/274292248 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:19.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.777+0000 7efd69028700 1 --1- 172.21.15.16:0/274292248 >> v1:172.21.15.16:6789/0 conn(0x7efd64090830 0x7efd6408d020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58906/0 (socket says 172.21.15.16:58906) 2019-12-05T06:06:19.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.777+0000 7efd69829700 1 --2- 172.21.15.16:0/274292248 >> v2:172.21.15.76:3300/0 conn(0x7efd6408c6f0 0x7efd6408cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:19.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.778+0000 7efd6a02a700 1 -- 172.21.15.16:0/274292248 >> v2:172.21.15.76:3300/0 conn(0x7efd6408c6f0 msgr2=0x7efd6408cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.778+0000 7efd6a02a700 1 --2- 172.21.15.16:0/274292248 >> v2:172.21.15.76:3300/0 conn(0x7efd6408c6f0 0x7efd6408cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.778+0000 7efd6a02a700 1 -- 172.21.15.16:0/274292248 >> v1:172.21.15.16:6789/0 conn(0x7efd64090830 legacy=0x7efd6408d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.778+0000 7efd6a02a700 1 -- 172.21.15.16:0/274292248 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6408ef20 con 0x7efd6408f860 2019-12-05T06:06:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.778+0000 7efd68827700 1 -- 172.21.15.16:0/274292248 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261586560 0 0) 0x7efd58001100 con 0x7efd64090830 2019-12-05T06:06:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.778+0000 7efd6a02a700 1 --2- 172.21.15.16:0/274292248 >> v2:172.21.15.16:3300/0 conn(0x7efd6408f860 0x7efd6408fc80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efd5400f6b0 tx=0x7efd5400fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.778+0000 7efd6a02a700 1 --2- 172.21.15.16:0/274292248 >> v2:172.21.15.16:3300/0 conn(0x7efd6408f860 0x7efd6408fc80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efd5400f6b0 tx=0x7efd5400fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.778+0000 7efd6a02a700 1 --2- 172.21.15.16:0/274292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd6408f860 0x7efd6408fc80 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7efd5400f6b0 tx=0x7efd5400fa50).ready entity=mon.0 client_cookie=c167af6ff6c21b9a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.779+0000 7efd6a02a700 1 --2- 172.21.15.16:0/274292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd6408f860 0x7efd6408fc80 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7efd5400f6b0 tx=0x7efd5400fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.779+0000 7efd68827700 1 -- 172.21.15.16:0/274292248 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd54000a70 con 0x7efd6408f860 2019-12-05T06:06:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.779+0000 7efd6a02a700 1 --2- 172.21.15.16:0/274292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd6408f860 0x7efd6408fc80 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7efd5400f6b0 tx=0x7efd5400fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.779+0000 7efd6a02a700 1 --2- 172.21.15.16:0/274292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd6408f860 0x7efd6408fc80 secure :-1 s=THROTTLE_DONE pgs=21 cs=0 l=1 rx=0x7efd5400f6b0 tx=0x7efd5400fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.779+0000 7efd68827700 1 -- 172.21.15.16:0/274292248 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd540010e0 con 0x7efd6408f860 2019-12-05T06:06:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.779+0000 7efd68827700 1 -- 172.21.15.16:0/274292248 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd54001390 con 0x7efd6408f860 2019-12-05T06:06:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.779+0000 7efd6b28c700 1 -- 172.21.15.16:0/274292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd6408f860 msgr2=0x7efd6408fc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.779+0000 7efd6b28c700 1 --2- 172.21.15.16:0/274292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd6408f860 0x7efd6408fc80 secure :-1 s=READY pgs=21 cs=0 l=1 rx=0x7efd5400f6b0 tx=0x7efd5400fa50).stop 2019-12-05T06:06:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.781+0000 7efd6b28c700 1 -- 172.21.15.16:0/274292248 shutdown_connections 2019-12-05T06:06:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.781+0000 7efd6b28c700 1 --2- 172.21.15.16:0/274292248 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd6408f860 0x7efd6408fc80 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.781+0000 7efd6b28c700 1 --2- 172.21.15.16:0/274292248 >> v2:172.21.15.76:3300/0 conn(0x7efd6408c6f0 0x7efd6408cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.781+0000 7efd6b28c700 1 -- 172.21.15.16:0/274292248 shutdown_connections 2019-12-05T06:06:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.781+0000 7efd6b28c700 1 -- 172.21.15.16:0/274292248 wait complete. 2019-12-05T06:06:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.781+0000 7efd6b28c700 1 -- 172.21.15.16:0/274292248 >> 172.21.15.16:0/274292248 conn(0x7efd64061960 msgr2=0x7efd640a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.786+0000 7efd6b28c700 1 Processor -- start 2019-12-05T06:06:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.786+0000 7efd6b28c700 1 -- start start 2019-12-05T06:06:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.788+0000 7efd6b28c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efd640740e0 0x7efd640834a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.788+0000 7efd6b28c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efd64154ca0 0x7efd641570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.788+0000 7efd6b28c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd64074b90 con 0x7efd64084470 2019-12-05T06:06:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.788+0000 7efd6b28c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd64081ae0 con 0x7efd64154ca0 2019-12-05T06:06:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.788+0000 7efd6b28c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd64081db0 con 0x7efd640740e0 2019-12-05T06:06:19.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.788+0000 7efd6a02a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efd64154ca0 0x7efd641570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:19.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.788+0000 7efd6a02a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efd64154ca0 0x7efd641570a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45512/0 (socket says 172.21.15.16:45512) 2019-12-05T06:06:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.788+0000 7efd6a02a700 1 -- 172.21.15.16:0/1867423311 learned_addr learned my addr 172.21.15.16:0/1867423311 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.788+0000 7efd69829700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efd640740e0 0x7efd640834a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6a02a700 1 -- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efd640740e0 msgr2=0x7efd640834a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efd640740e0 0x7efd640834a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6a02a700 1 -- 172.21.15.16:0/1867423311 >> v1:172.21.15.76:6789/0 conn(0x7efd64084470 legacy=0x7efd640839e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:19.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6a02a700 1 -- 172.21.15.16:0/1867423311 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd6414a810 con 0x7efd64154ca0 2019-12-05T06:06:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd69829700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efd640740e0 0x7efd640834a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:06:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> v2:172.21.15.16:3300/0 conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> v2:172.21.15.16:3300/0 conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd627fc700 1 -- 172.21.15.16:0/1867423311 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd540010c0 con 0x7efd64154ca0 2019-12-05T06:06:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd627fc700 1 -- 172.21.15.16:0/1867423311 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd54000940 con 0x7efd64154ca0 2019-12-05T06:06:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd6b28c700 1 -- 172.21.15.16:0/1867423311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd64082ba0 con 0x7efd64154ca0 2019-12-05T06:06:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.789+0000 7efd627fc700 1 -- 172.21.15.16:0/1867423311 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd540016a0 con 0x7efd64154ca0 2019-12-05T06:06:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.790+0000 7efd6b28c700 1 -- 172.21.15.16:0/1867423311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd6414a810 con 0x7efd64154ca0 2019-12-05T06:06:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.790+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.792+0000 7efd627fc700 1 -- 172.21.15.16:0/1867423311 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efd540010c0 con 0x7efd64154ca0 2019-12-05T06:06:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.792+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.792+0000 7efd627fc700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efd44023630 0x7efd44025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.792+0000 7efd627fc700 1 -- 172.21.15.16:0/1867423311 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1214+0+0 (secure 0 0 0) 0x7efd54045e00 con 0x7efd64154ca0 2019-12-05T06:06:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.792+0000 7efd69829700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efd44023630 0x7efd44025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:19.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.793+0000 7efd69829700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efd44023630 0x7efd44025ac0 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:19.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.806+0000 7efd6b28c700 1 -- 172.21.15.16:0/1867423311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd64082ba0 con 0x7efd64154ca0 2019-12-05T06:06:19.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.815+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:19.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:19.815+0000 7efd627fc700 1 -- 172.21.15.16:0/1867423311 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efd54000eb0 con 0x7efd64154ca0 2019-12-05T06:06:20.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.055+0000 7efd6b28c700 1 -- 172.21.15.16:0/1867423311 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7fb2a124-c603-49d9-ad7c-44c975729335", "id": 4} v 0) v1 -- 0x7efd6414a810 con 0x7efd64154ca0 2019-12-05T06:06:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.060+0000 7efd6a02a700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=THROTTLE_DONE pgs=22 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.060+0000 7efd627fc700 1 -- 172.21.15.16:0/1867423311 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7fb2a124-c603-49d9-ad7c-44c975729335", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7efd540010c0 con 0x7efd64154ca0 2019-12-05T06:06:20.061 INFO:teuthology.orchestra.run.smithi016.stdout:4 2019-12-05T06:06:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 -- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efd44023630 msgr2=0x7efd44025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efd44023630 0x7efd44025ac0 crc :-1 s=READY pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 -- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 msgr2=0x7efd641570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 0x7efd641570a0 secure :-1 s=READY pgs=22 cs=0 l=1 rx=0x7efd54010420 tx=0x7efd540107c0).stop 2019-12-05T06:06:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 -- 172.21.15.16:0/1867423311 shutdown_connections 2019-12-05T06:06:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efd44023630 0x7efd44025ac0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efd64154ca0 0x7efd641570a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 --2- 172.21.15.16:0/1867423311 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efd640740e0 0x7efd640834a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 -- 172.21.15.16:0/1867423311 shutdown_connections 2019-12-05T06:06:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 -- 172.21.15.16:0/1867423311 wait complete. 2019-12-05T06:06:20.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.067+0000 7efd6b28c700 1 -- 172.21.15.16:0/1867423311 >> 172.21.15.16:0/1867423311 conn(0x7efd64078040 msgr2=0x7efd640a0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:20.076 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 9776ef43-83b4-4461-9d37-438d04ed68ef 5 2019-12-05T06:06:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.304+0000 7f899ec06700 1 Processor -- start 2019-12-05T06:06:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.304+0000 7f899ec06700 1 -- start start 2019-12-05T06:06:20.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.305+0000 7f899ec06700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f899808c6f0 0x7f899808cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.305+0000 7f899ec06700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998090830 0x7f899808d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.305+0000 7f899ec06700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f899808f860 0x7f899808fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.305+0000 7f899ec06700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8998096640 con 0x7f899808c6f0 2019-12-05T06:06:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.305+0000 7f899ec06700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f899808e890 con 0x7f899808f860 2019-12-05T06:06:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.305+0000 7f899ec06700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f899808db90 con 0x7f8998090830 2019-12-05T06:06:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.306+0000 7f899d1a3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f899808c6f0 0x7f899808cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:20.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.306+0000 7f899d1a3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f899808c6f0 0x7f899808cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45516/0 (socket says 172.21.15.16:45516) 2019-12-05T06:06:20.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.306+0000 7f899d1a3700 1 -- 172.21.15.16:0/995963433 learned_addr learned my addr 172.21.15.16:0/995963433 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:20.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.306+0000 7f899c9a2700 1 --2- 172.21.15.16:0/995963433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998090830 0x7f899808d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:20.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.306+0000 7f899d9a4700 1 --2- 172.21.15.16:0/995963433 >> v2:172.21.15.76:3300/0 conn(0x7f899808f860 0x7f899808fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:20.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.306+0000 7f899d9a4700 1 -- 172.21.15.16:0/995963433 >> v2:172.21.15.76:3300/0 conn(0x7f899808f860 msgr2=0x7f899808fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2019-12-05T06:06:20.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.306+0000 7f899d9a4700 1 -- 172.21.15.16:0/995963433 >> v2:172.21.15.76:3300/0 conn(0x7f899808f860 msgr2=0x7f899808fc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:06:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.306+0000 7f899d9a4700 1 --2- 172.21.15.16:0/995963433 >> v2:172.21.15.76:3300/0 conn(0x7f899808f860 0x7f899808fc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:20.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.306+0000 7f899d9a4700 1 --2- 172.21.15.16:0/995963433 >> v2:172.21.15.76:3300/0 conn(0x7f899808f860 0x7f899808fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:06:20.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.307+0000 7f899d1a3700 1 -- 172.21.15.16:0/995963433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998090830 msgr2=0x7f899808d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:20.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.307+0000 7f899d1a3700 1 --2- 172.21.15.16:0/995963433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998090830 0x7f899808d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.307+0000 7f899d1a3700 1 -- 172.21.15.16:0/995963433 >> v2:172.21.15.76:3300/0 conn(0x7f899808f860 msgr2=0x7f899808fc70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:06:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.307+0000 7f899d1a3700 1 --2- 172.21.15.16:0/995963433 >> v2:172.21.15.76:3300/0 conn(0x7f899808f860 0x7f899808fc70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.307+0000 7f899d1a3700 1 -- 172.21.15.16:0/995963433 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f899808ef20 con 0x7f899808c6f0 2019-12-05T06:06:20.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.307+0000 7f899d1a3700 1 --2- 172.21.15.16:0/995963433 >> v2:172.21.15.16:3300/0 conn(0x7f899808c6f0 0x7f899808cae0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f899000fac0 tx=0x7f899000f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.307+0000 7f899d1a3700 1 --2- 172.21.15.16:0/995963433 >> v2:172.21.15.16:3300/0 conn(0x7f899808c6f0 0x7f899808cae0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f899000fac0 tx=0x7f899000f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.307+0000 7f899d1a3700 1 --2- 172.21.15.16:0/995963433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899808c6f0 0x7f899808cae0 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7f899000fac0 tx=0x7f899000f970).ready entity=mon.0 client_cookie=f9d85711ed1b755 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.308+0000 7f899d1a3700 1 --2- 172.21.15.16:0/995963433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899808c6f0 0x7f899808cae0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f899000fac0 tx=0x7f899000f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.308+0000 7f899d1a3700 1 --2- 172.21.15.16:0/995963433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899808c6f0 0x7f899808cae0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f899000fac0 tx=0x7f899000f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.308+0000 7f8997fff700 1 -- 172.21.15.16:0/995963433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8990000ce0 con 0x7f899808c6f0 2019-12-05T06:06:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.308+0000 7f899d1a3700 1 --2- 172.21.15.16:0/995963433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899808c6f0 0x7f899808cae0 secure :-1 s=THROTTLE_DONE pgs=23 cs=0 l=1 rx=0x7f899000fac0 tx=0x7f899000f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.308+0000 7f8997fff700 1 -- 172.21.15.16:0/995963433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8990001010 con 0x7f899808c6f0 2019-12-05T06:06:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.308+0000 7f8997fff700 1 -- 172.21.15.16:0/995963433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8990001720 con 0x7f899808c6f0 2019-12-05T06:06:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.308+0000 7f899ec06700 1 -- 172.21.15.16:0/995963433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899808c6f0 msgr2=0x7f899808cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.308+0000 7f899ec06700 1 --2- 172.21.15.16:0/995963433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899808c6f0 0x7f899808cae0 secure :-1 s=READY pgs=23 cs=0 l=1 rx=0x7f899000fac0 tx=0x7f899000f970).stop 2019-12-05T06:06:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.310+0000 7f899ec06700 1 -- 172.21.15.16:0/995963433 shutdown_connections 2019-12-05T06:06:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.310+0000 7f899ec06700 1 --2- 172.21.15.16:0/995963433 >> v2:172.21.15.76:3300/0 conn(0x7f899808f860 0x7f899808fc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.310+0000 7f899ec06700 1 --2- 172.21.15.16:0/995963433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998090830 0x7f899808d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:20.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.310+0000 7f899ec06700 1 --2- 172.21.15.16:0/995963433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f899808c6f0 0x7f899808cae0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.310+0000 7f899ec06700 1 -- 172.21.15.16:0/995963433 shutdown_connections 2019-12-05T06:06:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.310+0000 7f899ec06700 1 -- 172.21.15.16:0/995963433 wait complete. 2019-12-05T06:06:20.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.310+0000 7f899ec06700 1 -- 172.21.15.16:0/995963433 >> 172.21.15.16:0/995963433 conn(0x7f8998061960 msgr2=0x7f89980a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.315+0000 7f899ec06700 1 Processor -- start 2019-12-05T06:06:20.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.315+0000 7f899ec06700 1 -- start start 2019-12-05T06:06:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.317+0000 7f899ec06700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998084460 0x7f8998083c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.317+0000 7f899ec06700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8998154ca0 0x7f89981570a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.317+0000 7f899ec06700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89980744d0 con 0x7f89980740e0 2019-12-05T06:06:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.317+0000 7f899ec06700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8998082500 con 0x7f8998154ca0 2019-12-05T06:06:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.317+0000 7f899ec06700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8998081ad0 con 0x7f8998084460 2019-12-05T06:06:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.317+0000 7f899d9a4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8998154ca0 0x7f89981570a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:20.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.317+0000 7f899d1a3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f89980740e0 0x7f8998083490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58922/0 (socket says 172.21.15.16:58922) 2019-12-05T06:06:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.317+0000 7f899d1a3700 1 -- 172.21.15.16:0/2427479376 learned_addr learned my addr 172.21.15.16:0/2427479376 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.318+0000 7f899c9a2700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998084460 0x7f8998083c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.318+0000 7f899d9a4700 1 -- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998084460 msgr2=0x7f8998083c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.318+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998084460 0x7f8998083c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.318+0000 7f899d9a4700 1 -- 172.21.15.16:0/2427479376 >> v1:172.21.15.16:6789/0 conn(0x7f89980740e0 legacy=0x7f8998083490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.318+0000 7f899d9a4700 1 -- 172.21.15.16:0/2427479376 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8998082b90 con 0x7f8998154ca0 2019-12-05T06:06:20.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.318+0000 7f8995ffb700 1 -- 172.21.15.16:0/2427479376 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 206502325 0 0) 0x7f8990007900 con 0x7f89980740e0 2019-12-05T06:06:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.318+0000 7f899c9a2700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998084460 0x7f8998083c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.318+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> v2:172.21.15.16:3300/0 conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.319+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> v2:172.21.15.16:3300/0 conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.319+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.319+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.319+0000 7f8995ffb700 1 -- 172.21.15.16:0/2427479376 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8988000a70 con 0x7f8998154ca0 2019-12-05T06:06:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.319+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.319+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.319+0000 7f8995ffb700 1 -- 172.21.15.16:0/2427479376 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8988001030 con 0x7f8998154ca0 2019-12-05T06:06:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.319+0000 7f8995ffb700 1 -- 172.21.15.16:0/2427479376 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8988001740 con 0x7f8998154ca0 2019-12-05T06:06:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.319+0000 7f899ec06700 1 -- 172.21.15.16:0/2427479376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8998082b90 con 0x7f8998154ca0 2019-12-05T06:06:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.319+0000 7f899ec06700 1 -- 172.21.15.16:0/2427479376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8998082b90 con 0x7f8998154ca0 2019-12-05T06:06:20.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.320+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.321+0000 7f8995ffb700 1 -- 172.21.15.16:0/2427479376 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8988000b10 con 0x7f8998154ca0 2019-12-05T06:06:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.321+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.321+0000 7f8995ffb700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8978023630 0x7f8978025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.321+0000 7f8995ffb700 1 -- 172.21.15.16:0/2427479376 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1329+0+0 (secure 0 0 0) 0x7f8988045d30 con 0x7f8998154ca0 2019-12-05T06:06:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.321+0000 7f899d1a3700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8978023630 0x7f8978025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.322+0000 7f899d1a3700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8978023630 0x7f8978025ac0 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:20.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.335+0000 7f899ec06700 1 -- 172.21.15.16:0/2427479376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8998082b90 con 0x7f8998154ca0 2019-12-05T06:06:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.344+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.344+0000 7f8995ffb700 1 -- 172.21.15.16:0/2427479376 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f898800a290 con 0x7f8998154ca0 2019-12-05T06:06:20.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.598+0000 7f899ec06700 1 -- 172.21.15.16:0/2427479376 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9776ef43-83b4-4461-9d37-438d04ed68ef", "id": 5} v 0) v1 -- 0x7f8998082b90 con 0x7f8998154ca0 2019-12-05T06:06:20.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.604+0000 7f899d9a4700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.604+0000 7f8995ffb700 1 -- 172.21.15.16:0/2427479376 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9776ef43-83b4-4461-9d37-438d04ed68ef", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f8988000a70 con 0x7f8998154ca0 2019-12-05T06:06:20.603 INFO:teuthology.orchestra.run.smithi016.stdout:5 2019-12-05T06:06:20.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.607+0000 7f899ec06700 1 -- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8978023630 msgr2=0x7f8978025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.607+0000 7f899ec06700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8978023630 0x7f8978025ac0 crc :-1 s=READY pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.607+0000 7f899ec06700 1 -- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 msgr2=0x7f89981570a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.607+0000 7f899ec06700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 0x7f89981570a0 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f89880103a0 tx=0x7f8988010740).stop 2019-12-05T06:06:20.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.608+0000 7f899ec06700 1 -- 172.21.15.16:0/2427479376 shutdown_connections 2019-12-05T06:06:20.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.608+0000 7f899ec06700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8978023630 0x7f8978025ac0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.608+0000 7f899ec06700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8998154ca0 0x7f89981570a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.608+0000 7f899ec06700 1 --2- 172.21.15.16:0/2427479376 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8998084460 0x7f8998083c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.608+0000 7f899ec06700 1 -- 172.21.15.16:0/2427479376 shutdown_connections 2019-12-05T06:06:20.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.608+0000 7f899ec06700 1 -- 172.21.15.16:0/2427479376 wait complete. 2019-12-05T06:06:20.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.608+0000 7f899ec06700 1 -- 172.21.15.16:0/2427479376 >> 172.21.15.16:0/2427479376 conn(0x7f8998078040 msgr2=0x7f89980a0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:20.622 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 9ed18a64-0c45-4d39-84ce-67748d102f1c 6 2019-12-05T06:06:20.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.886+0000 7f4835053700 1 Processor -- start 2019-12-05T06:06:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.886+0000 7f4835053700 1 -- start start 2019-12-05T06:06:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.887+0000 7f4835053700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4830090830 0x7f483008d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.888+0000 7f4835053700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4830062230 con 0x7f483008f860 2019-12-05T06:06:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.888+0000 7f4835053700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4830090480 con 0x7f483008c6f0 2019-12-05T06:06:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.888+0000 7f4835053700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4830096640 con 0x7f4830090830 2019-12-05T06:06:20.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.888+0000 7f482ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4830090830 0x7f483008d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.888+0000 7f482ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4830090830 0x7f483008d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45532/0 (socket says 172.21.15.16:45532) 2019-12-05T06:06:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.888+0000 7f482ed9d700 1 -- 172.21.15.16:0/289328019 learned_addr learned my addr 172.21.15.16:0/289328019 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.888+0000 7f4834852700 1 --1- 172.21.15.16:0/289328019 >> v1:172.21.15.16:6789/0 conn(0x7f483008f860 0x7f483008fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58934/0 (socket says 172.21.15.16:58934) 2019-12-05T06:06:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.888+0000 7f482f59e700 1 --1- 172.21.15.16:0/289328019 >> v1:172.21.15.76:6789/0 conn(0x7f483008c6f0 0x7f483008cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52000/0 (socket says 172.21.15.16:52000) 2019-12-05T06:06:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.889+0000 7f482e59c700 1 -- 172.21.15.16:0/289328019 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014646747 0 0) 0x7f4820001400 con 0x7f483008c6f0 2019-12-05T06:06:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.889+0000 7f482ed9d700 1 -- 172.21.15.16:0/289328019 >> v1:172.21.15.76:6789/0 conn(0x7f483008c6f0 legacy=0x7f483008cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.889+0000 7f482ed9d700 1 -- 172.21.15.16:0/289328019 >> v1:172.21.15.16:6789/0 conn(0x7f483008f860 legacy=0x7f483008fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.889+0000 7f482ed9d700 1 -- 172.21.15.16:0/289328019 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f483008ef20 con 0x7f4830090830 2019-12-05T06:06:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.889+0000 7f482ed9d700 1 --2- 172.21.15.16:0/289328019 >> v2:172.21.15.16:3300/0 conn(0x7f4830090830 0x7f483008d2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f482400f740 tx=0x7f482400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.889+0000 7f482ed9d700 1 --2- 172.21.15.16:0/289328019 >> v2:172.21.15.16:3300/0 conn(0x7f4830090830 0x7f483008d2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f482400f740 tx=0x7f482400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.889+0000 7f482ed9d700 1 --2- 172.21.15.16:0/289328019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4830090830 0x7f483008d2e0 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f482400f740 tx=0x7f482400fae0).ready entity=mon.0 client_cookie=73a3a65a2b6fe2e0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.890+0000 7f482ed9d700 1 --2- 172.21.15.16:0/289328019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4830090830 0x7f483008d2e0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f482400f740 tx=0x7f482400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.890+0000 7f482e59c700 1 -- 172.21.15.16:0/289328019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4824000960 con 0x7f4830090830 2019-12-05T06:06:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.890+0000 7f482ed9d700 1 --2- 172.21.15.16:0/289328019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4830090830 0x7f483008d2e0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f482400f740 tx=0x7f482400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.890+0000 7f482ed9d700 1 --2- 172.21.15.16:0/289328019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4830090830 0x7f483008d2e0 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f482400f740 tx=0x7f482400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.890+0000 7f482e59c700 1 -- 172.21.15.16:0/289328019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4824000c90 con 0x7f4830090830 2019-12-05T06:06:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.890+0000 7f482e59c700 1 -- 172.21.15.16:0/289328019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4824001360 con 0x7f4830090830 2019-12-05T06:06:20.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.890+0000 7f4835053700 1 -- 172.21.15.16:0/289328019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4830090830 msgr2=0x7f483008d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.890+0000 7f4835053700 1 --2- 172.21.15.16:0/289328019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4830090830 0x7f483008d2e0 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f482400f740 tx=0x7f482400fae0).stop 2019-12-05T06:06:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.892+0000 7f4835053700 1 -- 172.21.15.16:0/289328019 shutdown_connections 2019-12-05T06:06:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.892+0000 7f4835053700 1 --2- 172.21.15.16:0/289328019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4830090830 0x7f483008d2e0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.892+0000 7f4835053700 1 -- 172.21.15.16:0/289328019 shutdown_connections 2019-12-05T06:06:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.892+0000 7f4835053700 1 -- 172.21.15.16:0/289328019 wait complete. 2019-12-05T06:06:20.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.892+0000 7f4835053700 1 -- 172.21.15.16:0/289328019 >> 172.21.15.16:0/289328019 conn(0x7f4830061960 msgr2=0x7f48300a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.897+0000 7f4835053700 1 Processor -- start 2019-12-05T06:06:20.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.898+0000 7f4835053700 1 -- start start 2019-12-05T06:06:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.899+0000 7f4835053700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f483014f690 0x7f483014fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.899+0000 7f4835053700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4830089bd0 0x7f4830153370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.899+0000 7f4835053700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f483008a010 con 0x7f483008a3a0 2019-12-05T06:06:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.899+0000 7f4835053700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48300743e0 con 0x7f4830089bd0 2019-12-05T06:06:20.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.899+0000 7f4835053700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4830087a10 con 0x7f483014f690 2019-12-05T06:06:20.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.900+0000 7f482ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f483008a3a0 0x7f48300893d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58938/0 (socket says 172.21.15.16:58938) 2019-12-05T06:06:20.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.900+0000 7f482ed9d700 1 -- 172.21.15.16:0/3088234359 learned_addr learned my addr 172.21.15.16:0/3088234359 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:20.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.900+0000 7f482f59e700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f483014f690 0x7f483014fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:20.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.900+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> v2:172.21.15.76:3300/0 conn(0x7f4830089bd0 0x7f4830153370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:20.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.900+0000 7f4817fff700 1 -- 172.21.15.16:0/3088234359 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928649755 0 0) 0x7f48240079b0 con 0x7f483008a3a0 2019-12-05T06:06:20.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.900+0000 7f4817fff700 1 -- 172.21.15.16:0/3088234359 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4810002e80 con 0x7f483008a3a0 2019-12-05T06:06:20.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4834852700 1 -- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f483014f690 msgr2=0x7f483014fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f483014f690 0x7f483014fa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:20.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4834852700 1 -- 172.21.15.16:0/3088234359 >> v1:172.21.15.16:6789/0 conn(0x7f483008a3a0 legacy=0x7f48300893d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:20.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4834852700 1 -- 172.21.15.16:0/3088234359 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4830088ad0 con 0x7f4830089bd0 2019-12-05T06:06:20.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> v2:172.21.15.76:3300/0 conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> v2:172.21.15.76:3300/0 conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:20.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4817fff700 1 -- 172.21.15.16:0/3088234359 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4828001570 con 0x7f4830089bd0 2019-12-05T06:06:20.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4817fff700 1 -- 172.21.15.16:0/3088234359 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4828001190 con 0x7f4830089bd0 2019-12-05T06:06:20.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.901+0000 7f4835053700 1 -- 172.21.15.16:0/3088234359 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4830088ad0 con 0x7f4830089bd0 2019-12-05T06:06:20.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.902+0000 7f4817fff700 1 -- 172.21.15.16:0/3088234359 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4828001b50 con 0x7f4830089bd0 2019-12-05T06:06:20.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.902+0000 7f4835053700 1 -- 172.21.15.16:0/3088234359 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f483008a010 con 0x7f4830089bd0 2019-12-05T06:06:20.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.903+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.904+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.904+0000 7f4817fff700 1 -- 172.21.15.16:0/3088234359 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4828001380 con 0x7f4830089bd0 2019-12-05T06:06:20.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.904+0000 7f4817fff700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4810023670 0x7f4810025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:20.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.904+0000 7f4817fff700 1 -- 172.21.15.16:0/3088234359 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1444+0+0 (secure 0 0 0) 0x7f4828045cb0 con 0x7f4830089bd0 2019-12-05T06:06:20.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.905+0000 7f482f59e700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4810023670 0x7f4810025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:20.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.905+0000 7f482f59e700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4810023670 0x7f4810025ae0 crc :-1 s=READY pgs=13 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:20.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.917+0000 7f4835053700 1 -- 172.21.15.16:0/3088234359 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4830088ad0 con 0x7f4830089bd0 2019-12-05T06:06:20.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.927+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:20.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:20.927+0000 7f4817fff700 1 -- 172.21.15.16:0/3088234359 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f48280290f0 con 0x7f4830089bd0 2019-12-05T06:06:21.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.213+0000 7f4835053700 1 -- 172.21.15.16:0/3088234359 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9ed18a64-0c45-4d39-84ce-67748d102f1c", "id": 6} v 0) v1 -- 0x7f4830088ad0 con 0x7f4830089bd0 2019-12-05T06:06:21.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.223+0000 7f4834852700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=THROTTLE_DONE pgs=25 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.223+0000 7f4817fff700 1 -- 172.21.15.16:0/3088234359 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9ed18a64-0c45-4d39-84ce-67748d102f1c", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7f48280290f0 con 0x7f4830089bd0 2019-12-05T06:06:21.227 INFO:teuthology.orchestra.run.smithi016.stdout:6 2019-12-05T06:06:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.232+0000 7f4835053700 1 -- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4810023670 msgr2=0x7f4810025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.232+0000 7f4835053700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4810023670 0x7f4810025ae0 crc :-1 s=READY pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.232+0000 7f4835053700 1 -- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 msgr2=0x7f4830153370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.232+0000 7f4835053700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 0x7f4830153370 secure :-1 s=READY pgs=25 cs=0 l=1 rx=0x7f4828010380 tx=0x7f48280101b0).stop 2019-12-05T06:06:21.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.232+0000 7f4835053700 1 -- 172.21.15.16:0/3088234359 shutdown_connections 2019-12-05T06:06:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.232+0000 7f4835053700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4810023670 0x7f4810025ae0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.232+0000 7f4835053700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4830089bd0 0x7f4830153370 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.232+0000 7f4835053700 1 --2- 172.21.15.16:0/3088234359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f483014f690 0x7f483014fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.233+0000 7f4835053700 1 -- 172.21.15.16:0/3088234359 shutdown_connections 2019-12-05T06:06:21.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.233+0000 7f4835053700 1 -- 172.21.15.16:0/3088234359 wait complete. 2019-12-05T06:06:21.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.233+0000 7f4835053700 1 -- 172.21.15.16:0/3088234359 >> 172.21.15.16:0/3088234359 conn(0x7f48300733f0 msgr2=0x7f483007a740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:21.247 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd new 78f3fd1b-2fb0-4a3e-acd8-475d77dfac74 7 2019-12-05T06:06:21.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.501+0000 7f687f1dc700 1 Processor -- start 2019-12-05T06:06:21.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.501+0000 7f687f1dc700 1 -- start start 2019-12-05T06:06:21.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.503+0000 7f687f1dc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f687808c6f0 0x7f687808cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:21.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.503+0000 7f687f1dc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f687808f860 0x7f687808fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:21.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.503+0000 7f687f1dc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6878062230 con 0x7f6878090830 2019-12-05T06:06:21.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.503+0000 7f687f1dc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6878096640 con 0x7f687808f860 2019-12-05T06:06:21.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.503+0000 7f687f1dc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f687808db90 con 0x7f687808c6f0 2019-12-05T06:06:21.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.503+0000 7f687df7a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f687808f860 0x7f687808fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:21.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.503+0000 7f687df7a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f687808f860 0x7f687808fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45548/0 (socket says 172.21.15.16:45548) 2019-12-05T06:06:21.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.503+0000 7f687df7a700 1 -- 172.21.15.16:0/865938632 learned_addr learned my addr 172.21.15.16:0/865938632 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:21.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.503+0000 7f687d779700 1 --2- 172.21.15.16:0/865938632 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f687808c6f0 0x7f687808cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:21.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.504+0000 7f6877fff700 1 -- 172.21.15.16:0/865938632 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593096194 0 0) 0x7f686c001080 con 0x7f6878090830 2019-12-05T06:06:21.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.504+0000 7f6877fff700 1 -- 172.21.15.16:0/865938632 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6860002e80 con 0x7f6878090830 2019-12-05T06:06:21.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.504+0000 7f687df7a700 1 -- 172.21.15.16:0/865938632 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f687808c6f0 msgr2=0x7f687808cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:21.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.504+0000 7f687df7a700 1 --2- 172.21.15.16:0/865938632 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f687808c6f0 0x7f687808cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:21.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.504+0000 7f687df7a700 1 -- 172.21.15.16:0/865938632 >> v1:172.21.15.76:6789/0 conn(0x7f6878090830 legacy=0x7f687808d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.504+0000 7f687df7a700 1 -- 172.21.15.16:0/865938632 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f687808ef20 con 0x7f687808f860 2019-12-05T06:06:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.504+0000 7f687d779700 1 --2- 172.21.15.16:0/865938632 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f687808c6f0 0x7f687808cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.504+0000 7f687df7a700 1 --2- 172.21.15.16:0/865938632 >> v2:172.21.15.16:3300/0 conn(0x7f687808f860 0x7f687808fce0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f686800f730 tx=0x7f686800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.505+0000 7f687df7a700 1 --2- 172.21.15.16:0/865938632 >> v2:172.21.15.16:3300/0 conn(0x7f687808f860 0x7f687808fce0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f686800f730 tx=0x7f686800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.505+0000 7f687df7a700 1 --2- 172.21.15.16:0/865938632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687808f860 0x7f687808fce0 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f686800f730 tx=0x7f686800fad0).ready entity=mon.0 client_cookie=4342bcef97fa7752 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.505+0000 7f687df7a700 1 --2- 172.21.15.16:0/865938632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687808f860 0x7f687808fce0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f686800f730 tx=0x7f686800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.505+0000 7f687df7a700 1 --2- 172.21.15.16:0/865938632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687808f860 0x7f687808fce0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f686800f730 tx=0x7f686800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.505+0000 7f6877fff700 1 -- 172.21.15.16:0/865938632 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6868000b90 con 0x7f687808f860 2019-12-05T06:06:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.505+0000 7f687df7a700 1 --2- 172.21.15.16:0/865938632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687808f860 0x7f687808fce0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f686800f730 tx=0x7f686800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.505+0000 7f6877fff700 1 -- 172.21.15.16:0/865938632 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68680010e0 con 0x7f687808f860 2019-12-05T06:06:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.505+0000 7f6877fff700 1 -- 172.21.15.16:0/865938632 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6868001830 con 0x7f687808f860 2019-12-05T06:06:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.505+0000 7f687f1dc700 1 -- 172.21.15.16:0/865938632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687808f860 msgr2=0x7f687808fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.505+0000 7f687f1dc700 1 --2- 172.21.15.16:0/865938632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687808f860 0x7f687808fce0 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f686800f730 tx=0x7f686800fad0).stop 2019-12-05T06:06:21.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.507+0000 7f687f1dc700 1 -- 172.21.15.16:0/865938632 shutdown_connections 2019-12-05T06:06:21.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.507+0000 7f687f1dc700 1 --2- 172.21.15.16:0/865938632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687808f860 0x7f687808fce0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:21.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.507+0000 7f687f1dc700 1 --2- 172.21.15.16:0/865938632 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f687808c6f0 0x7f687808cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:21.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.507+0000 7f687f1dc700 1 -- 172.21.15.16:0/865938632 shutdown_connections 2019-12-05T06:06:21.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.507+0000 7f687f1dc700 1 -- 172.21.15.16:0/865938632 wait complete. 2019-12-05T06:06:21.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.507+0000 7f687f1dc700 1 -- 172.21.15.16:0/865938632 >> 172.21.15.16:0/865938632 conn(0x7f6878061960 msgr2=0x7f68780a37c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.513+0000 7f687f1dc700 1 Processor -- start 2019-12-05T06:06:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.513+0000 7f687f1dc700 1 -- start start 2019-12-05T06:06:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687f1dc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f687807b220 0x7f687807b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687f1dc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f68780833a0 0x7f687807bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687f1dc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f687807c090 con 0x7f68780823d0 2019-12-05T06:06:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687f1dc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f687807ddd0 con 0x7f687807b220 2019-12-05T06:06:21.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687f1dc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6878081440 con 0x7f68780833a0 2019-12-05T06:06:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687d779700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f687807b220 0x7f687807b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687d779700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f687807b220 0x7f687807b610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45552/0 (socket says 172.21.15.16:45552) 2019-12-05T06:06:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687d779700 1 -- 172.21.15.16:0/596744891 learned_addr learned my addr 172.21.15.16:0/596744891 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687cf78700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f68780833a0 0x7f687807bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687d779700 1 -- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f68780833a0 msgr2=0x7f687807bb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.516+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f68780833a0 0x7f687807bb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687d779700 1 -- 172.21.15.16:0/596744891 >> v1:172.21.15.76:6789/0 conn(0x7f68780823d0 legacy=0x7f6878082870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687d779700 1 -- 172.21.15.16:0/596744891 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6878083070 con 0x7f687807b220 2019-12-05T06:06:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687cf78700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f68780833a0 0x7f687807bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:06:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> v2:172.21.15.16:3300/0 conn(0x7f687807b220 0x7f687807b610 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> v2:172.21.15.16:3300/0 conn(0x7f687807b220 0x7f687807b610 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 0x7f687807b610 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 0x7f687807b610 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 0x7f687807b610 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f6875ffb700 1 -- 172.21.15.16:0/596744891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f687000ac40 con 0x7f687807b220 2019-12-05T06:06:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 0x7f687807b610 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f6875ffb700 1 -- 172.21.15.16:0/596744891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6870009af0 con 0x7f687807b220 2019-12-05T06:06:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f6875ffb700 1 -- 172.21.15.16:0/596744891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f687000a1c0 con 0x7f687807b220 2019-12-05T06:06:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687f1dc700 1 -- 172.21.15.16:0/596744891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6878083070 con 0x7f687807b220 2019-12-05T06:06:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.517+0000 7f687f1dc700 1 -- 172.21.15.16:0/596744891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6878083070 con 0x7f687807b220 2019-12-05T06:06:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.518+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 0x7f687807b610 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.519+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 0x7f687807b610 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.519+0000 7f6875ffb700 1 -- 172.21.15.16:0/596744891 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6870009ce0 con 0x7f687807b220 2019-12-05T06:06:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.520+0000 7f6875ffb700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6858023630 0x7f6858025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.520+0000 7f6875ffb700 1 -- 172.21.15.16:0/596744891 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1559+0+0 (secure 0 0 0) 0x7f6870046170 con 0x7f687807b220 2019-12-05T06:06:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.520+0000 7f687cf78700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6858023630 0x7f6858025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.521+0000 7f687cf78700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6858023630 0x7f6858025ac0 crc :-1 s=READY pgs=14 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:21.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.532+0000 7f687f1dc700 1 -- 172.21.15.16:0/596744891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6878081ad0 con 0x7f687807b220 2019-12-05T06:06:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.541+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 0x7f687807b610 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.541+0000 7f6875ffb700 1 -- 172.21.15.16:0/596744891 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f687000a8e0 con 0x7f687807b220 2019-12-05T06:06:21.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.819+0000 7f687f1dc700 1 -- 172.21.15.16:0/596744891 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd new", "uuid": "78f3fd1b-2fb0-4a3e-acd8-475d77dfac74", "id": 7} v 0) v1 -- 0x7f6878081ad0 con 0x7f687807b220 2019-12-05T06:06:21.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.825+0000 7f687d779700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 0x7f687807b610 secure :-1 s=THROTTLE_DONE pgs=31 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:21.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.825+0000 7f6875ffb700 1 -- 172.21.15.16:0/596744891 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "78f3fd1b-2fb0-4a3e-acd8-475d77dfac74", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f687002b190 con 0x7f687807b220 2019-12-05T06:06:21.824 INFO:teuthology.orchestra.run.smithi016.stdout:7 2019-12-05T06:06:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.828+0000 7f687f1dc700 1 -- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6858023630 msgr2=0x7f6858025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.828+0000 7f687f1dc700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6858023630 0x7f6858025ac0 crc :-1 s=READY pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.828+0000 7f687f1dc700 1 -- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 msgr2=0x7f687807b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:21.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.828+0000 7f687f1dc700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 0x7f687807b610 secure :-1 s=READY pgs=31 cs=0 l=1 rx=0x7f6870010790 tx=0x7f6870010b30).stop 2019-12-05T06:06:21.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.829+0000 7f687f1dc700 1 -- 172.21.15.16:0/596744891 shutdown_connections 2019-12-05T06:06:21.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.829+0000 7f687f1dc700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f68780833a0 0x7f687807bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:21.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.829+0000 7f687f1dc700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6858023630 0x7f6858025ac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:21.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.829+0000 7f687f1dc700 1 --2- 172.21.15.16:0/596744891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f687807b220 0x7f687807b610 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:21.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.829+0000 7f687f1dc700 1 -- 172.21.15.16:0/596744891 shutdown_connections 2019-12-05T06:06:21.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.829+0000 7f687f1dc700 1 -- 172.21.15.16:0/596744891 wait complete. 2019-12-05T06:06:21.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:21.829+0000 7f687f1dc700 1 -- 172.21.15.16:0/596744891 >> 172.21.15.16:0/596744891 conn(0x7f6878078040 msgr2=0x7f68780a0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:21.837 INFO:tasks.ceph.osd.4:Restarting daemon 2019-12-05T06:06:21.837 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2019-12-05T06:06:21.842 INFO:tasks.ceph.osd.4:Started 2019-12-05T06:06:21.842 INFO:tasks.ceph.osd.5:Restarting daemon 2019-12-05T06:06:21.842 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2019-12-05T06:06:21.846 INFO:tasks.ceph.osd.5:Started 2019-12-05T06:06:21.847 INFO:tasks.ceph.osd.6:Restarting daemon 2019-12-05T06:06:21.847 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2019-12-05T06:06:21.851 INFO:tasks.ceph.osd.6:Started 2019-12-05T06:06:21.851 INFO:tasks.ceph.osd.7:Restarting daemon 2019-12-05T06:06:21.851 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2019-12-05T06:06:21.855 INFO:tasks.ceph.osd.7:Started 2019-12-05T06:06:21.855 INFO:tasks.ceph.osd.0:Restarting daemon 2019-12-05T06:06:21.856 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2019-12-05T06:06:21.860 INFO:tasks.ceph.osd.0:Started 2019-12-05T06:06:21.860 INFO:tasks.ceph.osd.1:Restarting daemon 2019-12-05T06:06:21.860 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2019-12-05T06:06:21.864 INFO:tasks.ceph.osd.1:Started 2019-12-05T06:06:21.864 INFO:tasks.ceph.osd.2:Restarting daemon 2019-12-05T06:06:21.865 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2019-12-05T06:06:21.868 INFO:tasks.ceph.osd.2:Started 2019-12-05T06:06:21.868 INFO:tasks.ceph.osd.3:Restarting daemon 2019-12-05T06:06:21.868 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2019-12-05T06:06:21.873 INFO:tasks.ceph.osd.3:Started 2019-12-05T06:06:21.873 INFO:tasks.ceph:Waiting for OSDs to come up 2019-12-05T06:06:22.504 INFO:tasks.ceph.osd.1.smithi016.stderr:2019-12-05T06:06:22.506+0000 7f14551ddbc0 -1 Falling back to public interface 2019-12-05T06:06:22.505 INFO:tasks.ceph.osd.4.smithi076.stderr:2019-12-05T06:06:22.509+0000 7fb5603babc0 -1 Falling back to public interface 2019-12-05T06:06:22.511 INFO:tasks.ceph.osd.5.smithi076.stderr:2019-12-05T06:06:22.515+0000 7fb34c7e5bc0 -1 Falling back to public interface 2019-12-05T06:06:22.514 INFO:tasks.ceph.osd.6.smithi076.stderr:2019-12-05T06:06:22.518+0000 7f6f1f43bbc0 -1 Falling back to public interface 2019-12-05T06:06:22.515 INFO:tasks.ceph.osd.2.smithi016.stderr:2019-12-05T06:06:22.517+0000 7fbdd09bebc0 -1 Falling back to public interface 2019-12-05T06:06:22.519 INFO:tasks.ceph.osd.0.smithi016.stderr:2019-12-05T06:06:22.520+0000 7f2402b7abc0 -1 Falling back to public interface 2019-12-05T06:06:22.519 INFO:tasks.ceph.osd.7.smithi076.stderr:2019-12-05T06:06:22.523+0000 7fda2741bbc0 -1 Falling back to public interface 2019-12-05T06:06:22.520 INFO:tasks.ceph.osd.3.smithi016.stderr:2019-12-05T06:06:22.521+0000 7fd031a4dbc0 -1 Falling back to public interface 2019-12-05T06:06:22.684 INFO:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-12-05T06:06:22.930 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.917+0000 7f978ef34700 1 Processor -- start 2019-12-05T06:06:22.931 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.917+0000 7f978ef34700 1 -- start start 2019-12-05T06:06:22.931 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.918+0000 7f978ef34700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f978808fef0 0x7f978808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:22.931 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.918+0000 7f978ef34700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97880958e0 con 0x7f978808c6d0 2019-12-05T06:06:22.931 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.918+0000 7f978ef34700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97880629e0 con 0x7f978808d2c0 2019-12-05T06:06:22.931 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.918+0000 7f978ef34700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f978808df90 con 0x7f978808fef0 2019-12-05T06:06:22.931 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.918+0000 7f978d4d1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f978808c6d0 0x7f978808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:58982/0 (socket says 172.21.15.16:58982) 2019-12-05T06:06:22.932 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.918+0000 7f978d4d1700 1 -- 172.21.15.16:0/2549845057 learned_addr learned my addr 172.21.15.16:0/2549845057 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:22.932 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.918+0000 7f978dcd2700 1 --2- 172.21.15.16:0/2549845057 >> v2:172.21.15.76:3300/0 conn(0x7f978808fef0 0x7f978808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:22.932 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f9787fff700 1 -- 172.21.15.16:0/2549845057 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2039268029 0 0) 0x7f9770001150 con 0x7f978808c6d0 2019-12-05T06:06:22.932 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f9787fff700 1 -- 172.21.15.16:0/2549845057 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9774002e80 con 0x7f978808c6d0 2019-12-05T06:06:22.938 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f978dcd2700 1 -- 172.21.15.16:0/2549845057 >> v1:172.21.15.76:6789/0 conn(0x7f978808d2c0 legacy=0x7f978808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:22.938 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f9787fff700 1 -- 172.21.15.16:0/2549845057 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014428835 0 0) 0x7f9780001400 con 0x7f978808d2c0 2019-12-05T06:06:22.938 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f978dcd2700 1 -- 172.21.15.16:0/2549845057 >> v1:172.21.15.16:6789/0 conn(0x7f978808c6d0 legacy=0x7f978808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:22.939 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f978dcd2700 1 -- 172.21.15.16:0/2549845057 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f978808e620 con 0x7f978808fef0 2019-12-05T06:06:22.939 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f978dcd2700 1 --2- 172.21.15.16:0/2549845057 >> v2:172.21.15.76:3300/0 conn(0x7f978808fef0 0x7f978808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f977800af80 tx=0x7f977800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.939 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f978dcd2700 1 --2- 172.21.15.16:0/2549845057 >> v2:172.21.15.76:3300/0 conn(0x7f978808fef0 0x7f978808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f977800af80 tx=0x7f977800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.939 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f978dcd2700 1 --2- 172.21.15.16:0/2549845057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f978808fef0 0x7f978808f720 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f977800af80 tx=0x7f977800b790).ready entity=mon.1 client_cookie=6ecb36a503b5f496 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:22.940 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f978dcd2700 1 --2- 172.21.15.16:0/2549845057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f978808fef0 0x7f978808f720 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f977800af80 tx=0x7f977800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.940 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f978dcd2700 1 --2- 172.21.15.16:0/2549845057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f978808fef0 0x7f978808f720 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f977800af80 tx=0x7f977800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.940 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f9787fff700 1 -- 172.21.15.16:0/2549845057 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9778007240 con 0x7f978808fef0 2019-12-05T06:06:22.940 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f978dcd2700 1 --2- 172.21.15.16:0/2549845057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f978808fef0 0x7f978808f720 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7f977800af80 tx=0x7f977800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.940 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.919+0000 7f9787fff700 1 -- 172.21.15.16:0/2549845057 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9778000bb0 con 0x7f978808fef0 2019-12-05T06:06:22.941 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.920+0000 7f9787fff700 1 -- 172.21.15.16:0/2549845057 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9778009e60 con 0x7f978808fef0 2019-12-05T06:06:22.941 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.920+0000 7f978ef34700 1 -- 172.21.15.16:0/2549845057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f978808fef0 msgr2=0x7f978808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:22.941 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.920+0000 7f978ef34700 1 --2- 172.21.15.16:0/2549845057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f978808fef0 0x7f978808f720 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7f977800af80 tx=0x7f977800b790).stop 2019-12-05T06:06:22.941 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.920+0000 7f978ef34700 1 -- 172.21.15.16:0/2549845057 shutdown_connections 2019-12-05T06:06:22.941 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.920+0000 7f978ef34700 1 --2- 172.21.15.16:0/2549845057 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f978808fef0 0x7f978808f720 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:22.941 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.921+0000 7f978ef34700 1 -- 172.21.15.16:0/2549845057 shutdown_connections 2019-12-05T06:06:22.941 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.921+0000 7f978ef34700 1 -- 172.21.15.16:0/2549845057 wait complete. 2019-12-05T06:06:22.942 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.921+0000 7f978ef34700 1 -- 172.21.15.16:0/2549845057 >> 172.21.15.16:0/2549845057 conn(0x7f9788061940 msgr2=0x7f9788096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:22.942 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.923+0000 7f978ef34700 1 Processor -- start 2019-12-05T06:06:22.942 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.923+0000 7f978ef34700 1 -- start start 2019-12-05T06:06:22.942 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978ef34700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:22.942 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978ef34700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9788134c80 0x7f978806e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:22.942 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978ef34700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9788128af0 con 0x7f9788135450 2019-12-05T06:06:22.942 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978ef34700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9788132830 con 0x7f9788134c80 2019-12-05T06:06:22.943 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978ef34700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9788129d80 con 0x7f978806dc50 2019-12-05T06:06:22.943 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978d4d1700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:22.944 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978dcd2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9788134c80 0x7f978806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:22.944 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978d4d1700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:49578/0 (socket says 172.21.15.16:49578) 2019-12-05T06:06:22.944 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978ccd0700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9788135450 0x7f9788134480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52060/0 (socket says 172.21.15.16:52060) 2019-12-05T06:06:22.944 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978dcd2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9788134c80 0x7f978806e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:59914/0 (socket says 172.21.15.16:59914) 2019-12-05T06:06:22.944 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.924+0000 7f978d4d1700 1 -- 172.21.15.16:0/661996047 learned_addr learned my addr 172.21.15.16:0/661996047 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:22.944 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978d4d1700 1 -- 172.21.15.16:0/661996047 >> v2:172.21.15.76:3300/0 conn(0x7f9788134c80 msgr2=0x7f978806e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:22.944 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> v2:172.21.15.76:3300/0 conn(0x7f9788134c80 0x7f978806e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:22.945 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978d4d1700 1 -- 172.21.15.16:0/661996047 >> v1:172.21.15.76:6789/0 conn(0x7f9788135450 legacy=0x7f9788134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:22.945 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f9785ffb700 1 -- 172.21.15.16:0/661996047 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944538977 0 0) 0x7f9780001580 con 0x7f9788135450 2019-12-05T06:06:22.945 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978d4d1700 1 -- 172.21.15.16:0/661996047 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9788133b80 con 0x7f978806dc50 2019-12-05T06:06:22.945 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978dcd2700 1 --2- 172.21.15.16:0/661996047 >> v2:172.21.15.76:3300/0 conn(0x7f9788134c80 0x7f978806e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:22.945 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.945 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.945 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:22.946 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.946 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f9785ffb700 1 -- 172.21.15.16:0/661996047 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9770004430 con 0x7f978806dc50 2019-12-05T06:06:22.946 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.946 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.946 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f9785ffb700 1 -- 172.21.15.16:0/661996047 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9770000c90 con 0x7f978806dc50 2019-12-05T06:06:22.946 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f978ef34700 1 -- 172.21.15.16:0/661996047 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9788133b80 con 0x7f978806dc50 2019-12-05T06:06:22.946 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.925+0000 7f9785ffb700 1 -- 172.21.15.16:0/661996047 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9770004a10 con 0x7f978806dc50 2019-12-05T06:06:22.947 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.926+0000 7f978ef34700 1 -- 172.21.15.16:0/661996047 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9788133b80 con 0x7f978806dc50 2019-12-05T06:06:22.947 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.926+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.947 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.926+0000 7f9785ffb700 1 -- 172.21.15.16:0/661996047 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9770004240 con 0x7f978806dc50 2019-12-05T06:06:22.947 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.926+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.947 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.927+0000 7f9785ffb700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f976c023630 0x7f976c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:22.947 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.927+0000 7f9785ffb700 1 -- 172.21.15.16:0/661996047 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1674+0+0 (secure 0 0 0) 0x7f9770045dd0 con 0x7f978806dc50 2019-12-05T06:06:22.948 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.927+0000 7f978ccd0700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f976c023630 0x7f976c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:22.948 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.927+0000 7f978ccd0700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f976c023630 0x7f976c025ac0 crc :-1 s=READY pgs=15 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:22.948 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.933+0000 7f978ef34700 1 -- 172.21.15.16:0/661996047 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9788133b80 con 0x7f978806dc50 2019-12-05T06:06:22.949 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.937+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:22.949 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:22.937+0000 7f9785ffb700 1 -- 172.21.15.16:0/661996047 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9770001290 con 0x7f978806dc50 2019-12-05T06:06:23.134 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.136+0000 7f978ef34700 1 -- 172.21.15.16:0/661996047 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9788133b80 con 0x7f978806dc50 2019-12-05T06:06:23.135 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.137+0000 7f978d4d1700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=THROTTLE_DONE pgs=12 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:23.135 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.137+0000 7f9785ffb700 1 -- 172.21.15.16:0/661996047 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+6849 (secure 0 0 0) 0x7f9770004240 con 0x7f978806dc50 2019-12-05T06:06:23.137 INFO:teuthology.misc.health.smithi016.stdout: 2019-12-05T06:06:23.137 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":10,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:21.820257+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:06:23.140 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.141+0000 7f978ef34700 1 -- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f976c023630 msgr2=0x7f976c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:23.140 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.141+0000 7f978ef34700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f976c023630 0x7f976c025ac0 crc :-1 s=READY pgs=15 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:23.140 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.141+0000 7f978ef34700 1 -- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 msgr2=0x7f978806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:23.140 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.141+0000 7f978ef34700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 secure :-1 s=READY pgs=12 cs=0 l=1 rx=0x7f9770010420 tx=0x7f97700107c0).stop 2019-12-05T06:06:23.140 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.141+0000 7f978ef34700 1 -- 172.21.15.16:0/661996047 shutdown_connections 2019-12-05T06:06:23.141 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.141+0000 7f978ef34700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f976c023630 0x7f976c025ac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:23.141 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.141+0000 7f978ef34700 1 --2- 172.21.15.16:0/661996047 >> v2:172.21.15.76:3300/0 conn(0x7f9788134c80 0x7f978806e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:23.141 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.141+0000 7f978ef34700 1 --2- 172.21.15.16:0/661996047 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f978806dc50 0x7f978806e040 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:23.141 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.141+0000 7f978ef34700 1 -- 172.21.15.16:0/661996047 shutdown_connections 2019-12-05T06:06:23.141 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.142+0000 7f978ef34700 1 -- 172.21.15.16:0/661996047 wait complete. 2019-12-05T06:06:23.141 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:23.142+0000 7f978ef34700 1 -- 172.21.15.16:0/661996047 >> 172.21.15.16:0/661996047 conn(0x7f97880a2850 msgr2=0x7f978806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:23.149 DEBUG:teuthology.misc:0 of 8 OSDs are up 2019-12-05T06:06:23.569 INFO:tasks.ceph.osd.4.smithi076.stderr:2019-12-05T06:06:23.574+0000 7fb5603babc0 -1 osd.4 0 log_to_monitors {default=true} 2019-12-05T06:06:23.572 INFO:tasks.ceph.osd.1.smithi016.stderr:2019-12-05T06:06:23.574+0000 7f14551ddbc0 -1 osd.1 0 log_to_monitors {default=true} 2019-12-05T06:06:23.579 INFO:tasks.ceph.osd.5.smithi076.stderr:2019-12-05T06:06:23.584+0000 7fb34c7e5bc0 -1 osd.5 0 log_to_monitors {default=true} 2019-12-05T06:06:23.592 INFO:tasks.ceph.osd.2.smithi016.stderr:2019-12-05T06:06:23.592+0000 7fbdd09bebc0 -1 osd.2 0 log_to_monitors {default=true} 2019-12-05T06:06:23.593 INFO:tasks.ceph.osd.6.smithi076.stderr:2019-12-05T06:06:23.596+0000 7f6f1f43bbc0 -1 osd.6 0 log_to_monitors {default=true} 2019-12-05T06:06:23.593 INFO:tasks.ceph.osd.3.smithi016.stderr:2019-12-05T06:06:23.594+0000 7fd031a4dbc0 -1 osd.3 0 log_to_monitors {default=true} 2019-12-05T06:06:23.596 INFO:tasks.ceph.osd.7.smithi076.stderr:2019-12-05T06:06:23.601+0000 7fda2741bbc0 -1 osd.7 0 log_to_monitors {default=true} 2019-12-05T06:06:23.613 INFO:tasks.ceph.osd.0.smithi016.stderr:2019-12-05T06:06:23.603+0000 7f2402b7abc0 -1 osd.0 0 log_to_monitors {default=true} 2019-12-05T06:06:24.919 INFO:tasks.ceph.osd.2.smithi016.stderr:2019-12-05T06:06:24.921+0000 7fbdb8fc6700 -1 osd.2 0 waiting for initial osdmap 2019-12-05T06:06:24.920 INFO:tasks.ceph.osd.1.smithi016.stderr:2019-12-05T06:06:24.921+0000 7f143d7e5700 -1 osd.1 0 waiting for initial osdmap 2019-12-05T06:06:24.920 INFO:tasks.ceph.osd.0.smithi016.stderr:2019-12-05T06:06:24.921+0000 7f23eb182700 -1 osd.0 0 waiting for initial osdmap 2019-12-05T06:06:24.920 INFO:tasks.ceph.osd.6.smithi076.stderr:2019-12-05T06:06:24.924+0000 7f6f07a43700 -1 osd.6 0 waiting for initial osdmap 2019-12-05T06:06:24.929 INFO:tasks.ceph.osd.3.smithi016.stderr:2019-12-05T06:06:24.926+0000 7fd01a055700 -1 osd.3 0 waiting for initial osdmap 2019-12-05T06:06:24.929 INFO:tasks.ceph.osd.5.smithi076.stderr:2019-12-05T06:06:24.925+0000 7fb334ded700 -1 osd.5 0 waiting for initial osdmap 2019-12-05T06:06:24.930 INFO:tasks.ceph.osd.4.smithi076.stderr:2019-12-05T06:06:24.927+0000 7fb5489c2700 -1 osd.4 0 waiting for initial osdmap 2019-12-05T06:06:24.930 INFO:tasks.ceph.osd.7.smithi076.stderr:2019-12-05T06:06:24.930+0000 7fda0fa23700 -1 osd.7 0 waiting for initial osdmap 2019-12-05T06:06:27.010 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:06:27.025 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:06:29.961 INFO:teuthology.orchestra.run.smithi016:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2019-12-05T06:06:30.228 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.220+0000 7fb874352700 1 Processor -- start 2019-12-05T06:06:30.229 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.220+0000 7fb874352700 1 -- start start 2019-12-05T06:06:30.229 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.222+0000 7fb874352700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 0x7fb86c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:30.229 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.222+0000 7fb874352700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c0958e0 con 0x7fb86c08fef0 2019-12-05T06:06:30.229 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.222+0000 7fb874352700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c0629e0 con 0x7fb86c08f720 2019-12-05T06:06:30.229 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.222+0000 7fb874352700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb86c0a3f70 con 0x7fb86c08c6d0 2019-12-05T06:06:30.230 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.223+0000 7fb8720ee700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb86c08fef0 0x7fb86c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59216/0 (socket says 172.21.15.16:59216) 2019-12-05T06:06:30.230 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.223+0000 7fb8720ee700 1 -- 172.21.15.16:0/3892674853 learned_addr learned my addr 172.21.15.16:0/3892674853 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:30.230 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.223+0000 7fb8728ef700 1 --2- 172.21.15.16:0/3892674853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 0x7fb86c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:30.230 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.223+0000 7fb8718ed700 1 -- 172.21.15.16:0/3892674853 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 158231445 0 0) 0x7fb8600014d0 con 0x7fb86c08fef0 2019-12-05T06:06:30.230 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.223+0000 7fb8718ed700 1 -- 172.21.15.16:0/3892674853 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb854002e80 con 0x7fb86c08fef0 2019-12-05T06:06:30.230 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.223+0000 7fb8718ed700 1 -- 172.21.15.16:0/3892674853 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4065356918 0 0) 0x7fb8640009b0 con 0x7fb86c08f720 2019-12-05T06:06:30.230 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.223+0000 7fb8718ed700 1 -- 172.21.15.16:0/3892674853 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb854004170 con 0x7fb86c08f720 2019-12-05T06:06:30.230 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.224+0000 7fb8728ef700 1 -- 172.21.15.16:0/3892674853 >> v1:172.21.15.76:6789/0 conn(0x7fb86c08f720 legacy=0x7fb86c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.231 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.224+0000 7fb8728ef700 1 -- 172.21.15.16:0/3892674853 >> v1:172.21.15.16:6789/0 conn(0x7fb86c08fef0 legacy=0x7fb86c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.231 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.224+0000 7fb8728ef700 1 -- 172.21.15.16:0/3892674853 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c08fbc0 con 0x7fb86c08c6d0 2019-12-05T06:06:30.231 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.224+0000 7fb8728ef700 1 --2- 172.21.15.16:0/3892674853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 0x7fb86c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb85c00af50 tx=0x7fb85c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.231 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.224+0000 7fb8728ef700 1 --2- 172.21.15.16:0/3892674853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 0x7fb86c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb85c00af50 tx=0x7fb85c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.231 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.224+0000 7fb8728ef700 1 --2- 172.21.15.16:0/3892674853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 0x7fb86c08cac0 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7fb85c00af50 tx=0x7fb85c00b760).ready entity=mon.2 client_cookie=ea68e8d6717cd416 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:30.231 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.224+0000 7fb8728ef700 1 --2- 172.21.15.16:0/3892674853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 0x7fb86c08cac0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fb85c00af50 tx=0x7fb85c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.231 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.224+0000 7fb8718ed700 1 -- 172.21.15.16:0/3892674853 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb85c000960 con 0x7fb86c08c6d0 2019-12-05T06:06:30.232 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.225+0000 7fb8728ef700 1 --2- 172.21.15.16:0/3892674853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 0x7fb86c08cac0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fb85c00af50 tx=0x7fb85c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.232 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.225+0000 7fb8728ef700 1 --2- 172.21.15.16:0/3892674853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 0x7fb86c08cac0 secure :-1 s=THROTTLE_DONE pgs=15 cs=0 l=1 rx=0x7fb85c00af50 tx=0x7fb85c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.232 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.225+0000 7fb8718ed700 1 -- 172.21.15.16:0/3892674853 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb85c009860 con 0x7fb86c08c6d0 2019-12-05T06:06:30.232 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.225+0000 7fb8718ed700 1 -- 172.21.15.16:0/3892674853 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb85c009f30 con 0x7fb86c08c6d0 2019-12-05T06:06:30.232 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.225+0000 7fb874352700 1 -- 172.21.15.16:0/3892674853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 msgr2=0x7fb86c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.232 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.225+0000 7fb874352700 1 --2- 172.21.15.16:0/3892674853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 0x7fb86c08cac0 secure :-1 s=READY pgs=15 cs=0 l=1 rx=0x7fb85c00af50 tx=0x7fb85c00b760).stop 2019-12-05T06:06:30.233 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.227+0000 7fb874352700 1 -- 172.21.15.16:0/3892674853 shutdown_connections 2019-12-05T06:06:30.233 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.227+0000 7fb874352700 1 --2- 172.21.15.16:0/3892674853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb86c08c6d0 0x7fb86c08cac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:30.233 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.227+0000 7fb874352700 1 -- 172.21.15.16:0/3892674853 shutdown_connections 2019-12-05T06:06:30.233 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.227+0000 7fb874352700 1 -- 172.21.15.16:0/3892674853 wait complete. 2019-12-05T06:06:30.233 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.227+0000 7fb874352700 1 -- 172.21.15.16:0/3892674853 >> 172.21.15.16:0/3892674853 conn(0x7fb86c061940 msgr2=0x7fb86c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:30.234 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.232+0000 7fb874352700 1 Processor -- start 2019-12-05T06:06:30.234 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.232+0000 7fb874352700 1 -- start start 2019-12-05T06:06:30.234 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.234+0000 7fb874352700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb86c097780 0x7fb86c09a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:30.234 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.234+0000 7fb874352700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb86c09b840 0x7fb86c09adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:30.234 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.234+0000 7fb874352700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb86c0a2150 con 0x7fb86c096b90 2019-12-05T06:06:30.235 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.234+0000 7fb874352700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c0998e0 con 0x7fb86c097780 2019-12-05T06:06:30.235 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.234+0000 7fb874352700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb86c098eb0 con 0x7fb86c09b840 2019-12-05T06:06:30.237 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.234+0000 7fb8720ee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb86c097780 0x7fb86c09a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:30.237 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.234+0000 7fb8720ee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb86c097780 0x7fb86c09a870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45820/0 (socket says 172.21.15.16:45820) 2019-12-05T06:06:30.238 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.234+0000 7fb8720ee700 1 -- 172.21.15.16:0/3404026496 learned_addr learned my addr 172.21.15.16:0/3404026496 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:30.238 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.234+0000 7fb8730f0700 1 --2- 172.21.15.16:0/3404026496 >> v2:172.21.15.76:3300/0 conn(0x7fb86c09b840 0x7fb86c09adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:30.238 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.235+0000 7fb8720ee700 1 -- 172.21.15.16:0/3404026496 >> v2:172.21.15.76:3300/0 conn(0x7fb86c09b840 msgr2=0x7fb86c09adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.238 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.235+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> v2:172.21.15.76:3300/0 conn(0x7fb86c09b840 0x7fb86c09adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:30.238 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.235+0000 7fb8720ee700 1 -- 172.21.15.16:0/3404026496 >> v1:172.21.15.16:6789/0 conn(0x7fb86c096b90 legacy=0x7fb86c096f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.238 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.235+0000 7fb8720ee700 1 -- 172.21.15.16:0/3404026496 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c099f70 con 0x7fb86c097780 2019-12-05T06:06:30.238 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.235+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> v2:172.21.15.16:3300/0 conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.239 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.236+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> v2:172.21.15.16:3300/0 conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.239 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.236+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:30.239 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.236+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.239 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.236+0000 7fb86b7fe700 1 -- 172.21.15.16:0/3404026496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb860001010 con 0x7fb86c097780 2019-12-05T06:06:30.239 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.236+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.239 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.236+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.239 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.236+0000 7fb86b7fe700 1 -- 172.21.15.16:0/3404026496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb860001440 con 0x7fb86c097780 2019-12-05T06:06:30.239 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.236+0000 7fb874352700 1 -- 172.21.15.16:0/3404026496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb86c099f70 con 0x7fb86c097780 2019-12-05T06:06:30.240 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.236+0000 7fb86b7fe700 1 -- 172.21.15.16:0/3404026496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb860001ad0 con 0x7fb86c097780 2019-12-05T06:06:30.240 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.236+0000 7fb874352700 1 -- 172.21.15.16:0/3404026496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb86c099f70 con 0x7fb86c097780 2019-12-05T06:06:30.240 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.237+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.240 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.238+0000 7fb86b7fe700 1 -- 172.21.15.16:0/3404026496 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb860001010 con 0x7fb86c097780 2019-12-05T06:06:30.240 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.238+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.240 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.239+0000 7fb86b7fe700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb84c023630 0x7fb84c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:30.240 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.239+0000 7fb86b7fe700 1 -- 172.21.15.16:0/3404026496 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7fb860046180 con 0x7fb86c097780 2019-12-05T06:06:30.241 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.239+0000 7fb8728ef700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb84c023630 0x7fb84c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:30.241 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.240+0000 7fb8728ef700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb84c023630 0x7fb84c025ac0 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:30.250 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.252+0000 7fb874352700 1 -- 172.21.15.16:0/3404026496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb86c099f70 con 0x7fb86c097780 2019-12-05T06:06:30.258 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.260+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.258 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.260+0000 7fb86b7fe700 1 -- 172.21.15.16:0/3404026496 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb860007950 con 0x7fb86c097780 2019-12-05T06:06:30.516 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.517+0000 7fb874352700 1 -- 172.21.15.16:0/3404026496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb86c099f70 con 0x7fb86c097780 2019-12-05T06:06:30.517 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.519+0000 7fb8720ee700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.517 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.519+0000 7fb86b7fe700 1 -- 172.21.15.16:0/3404026496 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) v1 ==== 74+0+10391 (secure 0 0 0) 0x7fb860001010 con 0x7fb86c097780 2019-12-05T06:06:30.519 INFO:teuthology.misc.health.smithi016.stdout: 2019-12-05T06:06:30.519 INFO:teuthology.misc.health.smithi016.stdout:{"epoch":14,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:26.936349+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:06:30.527 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.523+0000 7fb874352700 1 -- 172.21.15.16:0/3404026496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb84c023630 msgr2=0x7fb84c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.527 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.523+0000 7fb874352700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb84c023630 0x7fb84c025ac0 crc :-1 s=READY pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:30.527 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.523+0000 7fb874352700 1 -- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 msgr2=0x7fb86c09a870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.527 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.523+0000 7fb874352700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 0x7fb86c09a870 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7fb860010760 tx=0x7fb8600105e0).stop 2019-12-05T06:06:30.527 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.523+0000 7fb874352700 1 -- 172.21.15.16:0/3404026496 shutdown_connections 2019-12-05T06:06:30.528 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.523+0000 7fb874352700 1 --2- 172.21.15.16:0/3404026496 >> v2:172.21.15.76:3300/0 conn(0x7fb86c09b840 0x7fb86c09adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:30.528 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.523+0000 7fb874352700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb84c023630 0x7fb84c025ac0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:30.528 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.523+0000 7fb874352700 1 --2- 172.21.15.16:0/3404026496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb86c097780 0x7fb86c09a870 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:30.528 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.524+0000 7fb874352700 1 -- 172.21.15.16:0/3404026496 shutdown_connections 2019-12-05T06:06:30.528 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.524+0000 7fb874352700 1 -- 172.21.15.16:0/3404026496 wait complete. 2019-12-05T06:06:30.528 INFO:teuthology.misc.health.smithi016.stderr:2019-12-05T06:06:30.524+0000 7fb874352700 1 -- 172.21.15.16:0/3404026496 >> 172.21.15.16:0/3404026496 conn(0x7fb86c071790 msgr2=0x7fb86c08a1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:30.531 DEBUG:teuthology.misc:8 of 8 OSDs are up 2019-12-05T06:06:30.531 INFO:tasks.ceph:Creating RBD pool 2019-12-05T06:06:30.531 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd pool create rbd 8 2019-12-05T06:06:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.754+0000 7f7e06b6e700 1 Processor -- start 2019-12-05T06:06:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.754+0000 7f7e06b6e700 1 -- start start 2019-12-05T06:06:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.756+0000 7f7e06b6e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7e00086d70 0x7f7e0008aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.756+0000 7f7e06b6e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 0x7f7e00064760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.756+0000 7f7e06b6e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e00061a10 con 0x7f7e00086980 2019-12-05T06:06:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.756+0000 7f7e06b6e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e00060bc0 con 0x7f7e00086d70 2019-12-05T06:06:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.756+0000 7f7e06b6e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e00060620 con 0x7f7e00064310 2019-12-05T06:06:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.756+0000 7f7e0490a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7e00086d70 0x7f7e0008aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.757+0000 7f7e0590c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 0x7f7e00064760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.757+0000 7f7e0510b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7e00086980 0x7f7e0008a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52296/0 (socket says 172.21.15.16:52296) 2019-12-05T06:06:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.757+0000 7f7e0510b700 1 -- 172.21.15.16:0/3364850994 learned_addr learned my addr 172.21.15.16:0/3364850994 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.757+0000 7f7e0590c700 1 -- 172.21.15.16:0/3364850994 >> v2:172.21.15.76:3300/0 conn(0x7f7e00086d70 msgr2=0x7f7e0008aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.757+0000 7f7e0590c700 1 --2- 172.21.15.16:0/3364850994 >> v2:172.21.15.76:3300/0 conn(0x7f7e00086d70 0x7f7e0008aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.757+0000 7f7e0590c700 1 -- 172.21.15.16:0/3364850994 >> v1:172.21.15.76:6789/0 conn(0x7f7e00086980 legacy=0x7f7e0008a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.757+0000 7f7e0590c700 1 -- 172.21.15.16:0/3364850994 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00063a10 con 0x7f7e00064310 2019-12-05T06:06:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.757+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3364850994 >> v2:172.21.15.76:3300/0 conn(0x7f7e00086d70 0x7f7e0008aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.757+0000 7f7dfffff700 1 -- 172.21.15.16:0/3364850994 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740686333 0 0) 0x7f7df8000fe0 con 0x7f7e00086980 2019-12-05T06:06:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.758+0000 7f7e0590c700 1 --2- 172.21.15.16:0/3364850994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 0x7f7e00064760 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7df000af80 tx=0x7f7df000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.758+0000 7f7e0590c700 1 --2- 172.21.15.16:0/3364850994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 0x7f7e00064760 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7df000af80 tx=0x7f7df000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.758+0000 7f7e0590c700 1 --2- 172.21.15.16:0/3364850994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 0x7f7e00064760 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7f7df000af80 tx=0x7f7df000b790).ready entity=mon.2 client_cookie=32b488d70a364a75 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.758+0000 7f7e0590c700 1 --2- 172.21.15.16:0/3364850994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 0x7f7e00064760 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f7df000af80 tx=0x7f7df000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.758+0000 7f7dfffff700 1 -- 172.21.15.16:0/3364850994 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7df0000bc0 con 0x7f7e00064310 2019-12-05T06:06:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.758+0000 7f7e0590c700 1 --2- 172.21.15.16:0/3364850994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 0x7f7e00064760 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f7df000af80 tx=0x7f7df000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.759+0000 7f7e0590c700 1 --2- 172.21.15.16:0/3364850994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 0x7f7e00064760 secure :-1 s=THROTTLE_DONE pgs=16 cs=0 l=1 rx=0x7f7df000af80 tx=0x7f7df000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.759+0000 7f7dfffff700 1 -- 172.21.15.16:0/3364850994 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df0000bc0 con 0x7f7e00064310 2019-12-05T06:06:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.759+0000 7f7dfffff700 1 -- 172.21.15.16:0/3364850994 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7df0009bd0 con 0x7f7e00064310 2019-12-05T06:06:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.759+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3364850994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 msgr2=0x7f7e00064760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.759+0000 7f7e06b6e700 1 --2- 172.21.15.16:0/3364850994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 0x7f7e00064760 secure :-1 s=READY pgs=16 cs=0 l=1 rx=0x7f7df000af80 tx=0x7f7df000b790).stop 2019-12-05T06:06:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.760+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3364850994 shutdown_connections 2019-12-05T06:06:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.760+0000 7f7e06b6e700 1 --2- 172.21.15.16:0/3364850994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e00064310 0x7f7e00064760 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.760+0000 7f7e06b6e700 1 --2- 172.21.15.16:0/3364850994 >> v2:172.21.15.76:3300/0 conn(0x7f7e00086d70 0x7f7e0008aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.761+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3364850994 shutdown_connections 2019-12-05T06:06:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.761+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3364850994 wait complete. 2019-12-05T06:06:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.761+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3364850994 >> 172.21.15.16:0/3364850994 conn(0x7f7e0005f5a0 msgr2=0x7f7e0009cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.766+0000 7f7e06b6e700 1 Processor -- start 2019-12-05T06:06:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.766+0000 7f7e06b6e700 1 -- start start 2019-12-05T06:06:30.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.768+0000 7f7e06b6e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7e000973c0 0x7f7e00096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.768+0000 7f7e06b6e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e00099de0 con 0x7f7e0014c6b0 2019-12-05T06:06:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.768+0000 7f7e06b6e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e00095e00 con 0x7f7e0014c2c0 2019-12-05T06:06:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.768+0000 7f7e06b6e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e0009b880 con 0x7f7e000973c0 2019-12-05T06:06:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.768+0000 7f7e0590c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e0014c6b0 0x7f7e00155fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59236/0 (socket says 172.21.15.16:59236) 2019-12-05T06:06:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.768+0000 7f7e0590c700 1 -- 172.21.15.16:0/3819664485 learned_addr learned my addr 172.21.15.16:0/3819664485 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.768+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> v2:172.21.15.76:3300/0 conn(0x7f7e000973c0 0x7f7e00096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.769+0000 7f7dfdffb700 1 -- 172.21.15.16:0/3819664485 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2312174521 0 0) 0x7f7df00095b0 con 0x7f7e0014c6b0 2019-12-05T06:06:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.769+0000 7f7dfdffb700 1 -- 172.21.15.16:0/3819664485 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7de0002e80 con 0x7f7e0014c6b0 2019-12-05T06:06:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.769+0000 7f7e0490a700 1 -- 172.21.15.16:0/3819664485 >> v1:172.21.15.76:6789/0 conn(0x7f7e0014c2c0 legacy=0x7f7e000964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.769+0000 7f7e0490a700 1 -- 172.21.15.16:0/3819664485 >> v1:172.21.15.16:6789/0 conn(0x7f7e0014c6b0 legacy=0x7f7e00155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.769+0000 7f7e0490a700 1 -- 172.21.15.16:0/3819664485 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e0014cad0 con 0x7f7e000973c0 2019-12-05T06:06:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.769+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> v2:172.21.15.76:3300/0 conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.769+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> v2:172.21.15.76:3300/0 conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.769+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:30.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.770+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.770+0000 7f7dfdffb700 1 -- 172.21.15.16:0/3819664485 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7df4000940 con 0x7f7e000973c0 2019-12-05T06:06:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.770+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.770+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.770+0000 7f7dfdffb700 1 -- 172.21.15.16:0/3819664485 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7df400dff0 con 0x7f7e000973c0 2019-12-05T06:06:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.770+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3819664485 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e0014cad0 con 0x7f7e000973c0 2019-12-05T06:06:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.770+0000 7f7dfdffb700 1 -- 172.21.15.16:0/3819664485 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7df400e6c0 con 0x7f7e000973c0 2019-12-05T06:06:30.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.770+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3819664485 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e0014cad0 con 0x7f7e000973c0 2019-12-05T06:06:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.771+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.772+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.772+0000 7f7dfdffb700 1 -- 172.21.15.16:0/3819664485 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7df400dc70 con 0x7f7e000973c0 2019-12-05T06:06:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.773+0000 7f7dfdffb700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7de0023670 0x7f7de0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.773+0000 7f7dfdffb700 1 -- 172.21.15.16:0/3819664485 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 4500+0+0 (secure 0 0 0) 0x7f7df4045df0 con 0x7f7e000973c0 2019-12-05T06:06:30.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.773+0000 7f7e0510b700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7de0023670 0x7f7de0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:30.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.774+0000 7f7e0510b700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7de0023670 0x7f7de0025ae0 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:30.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.785+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3819664485 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e0009bf10 con 0x7f7e000973c0 2019-12-05T06:06:30.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.794+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:30.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:30.795+0000 7f7dfdffb700 1 -- 172.21.15.16:0/3819664485 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7df400da10 con 0x7f7e000973c0 2019-12-05T06:06:31.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.021+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3819664485 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"} v 0) v1 -- 0x7f7e0009bf10 con 0x7f7e000973c0 2019-12-05T06:06:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.587+0000 7f7e0490a700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=THROTTLE_DONE pgs=51 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.587+0000 7f7dfdffb700 1 -- 172.21.15.16:0/3819664485 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "rbd"}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7f7df400dc20 con 0x7f7e000973c0 2019-12-05T06:06:31.634 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'rbd' created 2019-12-05T06:06:31.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.640+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7de0023670 msgr2=0x7f7de0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:31.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.640+0000 7f7e06b6e700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7de0023670 0x7f7de0025ae0 crc :-1 s=READY pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:31.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.640+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 msgr2=0x7f7e00096ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:31.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.640+0000 7f7e06b6e700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 0x7f7e00096ca0 secure :-1 s=READY pgs=51 cs=0 l=1 rx=0x7f7df40103a0 tx=0x7f7df4010740).stop 2019-12-05T06:06:31.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.641+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3819664485 shutdown_connections 2019-12-05T06:06:31.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.641+0000 7f7e06b6e700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7de0023670 0x7f7de0025ae0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:31.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.641+0000 7f7e06b6e700 1 --2- 172.21.15.16:0/3819664485 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e000973c0 0x7f7e00096ca0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:31.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.641+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3819664485 shutdown_connections 2019-12-05T06:06:31.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.641+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3819664485 wait complete. 2019-12-05T06:06:31.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.641+0000 7f7e06b6e700 1 -- 172.21.15.16:0/3819664485 >> 172.21.15.16:0/3819664485 conn(0x7f7e00077e10 msgr2=0x7f7e0008b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:31.656 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2019-12-05T06:06:31.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.934+0000 7fe55aae6700 1 Processor -- start 2019-12-05T06:06:31.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.935+0000 7fe55aae6700 1 -- start start 2019-12-05T06:06:31.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.937+0000 7fe55aae6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe55408c6f0 0x7fe55408cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:31.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.937+0000 7fe55aae6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe554060c00 con 0x7fe5540907c0 2019-12-05T06:06:31.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.937+0000 7fe55aae6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5540a3880 con 0x7fe554062860 2019-12-05T06:06:31.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.937+0000 7fe55aae6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe55408e860 con 0x7fe55408c6f0 2019-12-05T06:06:31.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.937+0000 7fe559884700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe5540907c0 0x7fe554153430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59242/0 (socket says 172.21.15.16:59242) 2019-12-05T06:06:31.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.937+0000 7fe559884700 1 -- 172.21.15.16:0/2722954362 learned_addr learned my addr 172.21.15.16:0/2722954362 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:31.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.937+0000 7fe559083700 1 --2- 172.21.15.16:0/2722954362 >> v2:172.21.15.76:3300/0 conn(0x7fe55408c6f0 0x7fe55408cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:31.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe553fff700 1 -- 172.21.15.16:0/2722954362 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1846252803 0 0) 0x7fe544001150 con 0x7fe5540907c0 2019-12-05T06:06:31.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe553fff700 1 -- 172.21.15.16:0/2722954362 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c002e80 con 0x7fe5540907c0 2019-12-05T06:06:31.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe553fff700 1 -- 172.21.15.16:0/2722954362 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664509800 0 0) 0x7fe548001080 con 0x7fe554062860 2019-12-05T06:06:31.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe553fff700 1 -- 172.21.15.16:0/2722954362 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c004170 con 0x7fe554062860 2019-12-05T06:06:31.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe559083700 1 -- 172.21.15.16:0/2722954362 >> v1:172.21.15.76:6789/0 conn(0x7fe554062860 legacy=0x7fe55408f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:31.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe559083700 1 -- 172.21.15.16:0/2722954362 >> v1:172.21.15.16:6789/0 conn(0x7fe5540907c0 legacy=0x7fe554153430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:31.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe559083700 1 -- 172.21.15.16:0/2722954362 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55408eef0 con 0x7fe55408c6f0 2019-12-05T06:06:31.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe559083700 1 --2- 172.21.15.16:0/2722954362 >> v2:172.21.15.76:3300/0 conn(0x7fe55408c6f0 0x7fe55408cae0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe54c00b2d0 tx=0x7fe54c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe559083700 1 --2- 172.21.15.16:0/2722954362 >> v2:172.21.15.76:3300/0 conn(0x7fe55408c6f0 0x7fe55408cae0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe54c00b2d0 tx=0x7fe54c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe559083700 1 --2- 172.21.15.16:0/2722954362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe55408c6f0 0x7fe55408cae0 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7fe54c00b2d0 tx=0x7fe54c00bae0).ready entity=mon.1 client_cookie=44fd79eb26a9cc4b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:31.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.938+0000 7fe559083700 1 --2- 172.21.15.16:0/2722954362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe55408c6f0 0x7fe55408cae0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fe54c00b2d0 tx=0x7fe54c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.939+0000 7fe559083700 1 --2- 172.21.15.16:0/2722954362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe55408c6f0 0x7fe55408cae0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fe54c00b2d0 tx=0x7fe54c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.939+0000 7fe553fff700 1 -- 172.21.15.16:0/2722954362 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe54c000ce0 con 0x7fe55408c6f0 2019-12-05T06:06:31.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.939+0000 7fe559083700 1 --2- 172.21.15.16:0/2722954362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe55408c6f0 0x7fe55408cae0 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fe54c00b2d0 tx=0x7fe54c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.939+0000 7fe553fff700 1 -- 172.21.15.16:0/2722954362 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe54c009be0 con 0x7fe55408c6f0 2019-12-05T06:06:31.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.939+0000 7fe553fff700 1 -- 172.21.15.16:0/2722954362 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe54c00a2b0 con 0x7fe55408c6f0 2019-12-05T06:06:31.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.939+0000 7fe55aae6700 1 -- 172.21.15.16:0/2722954362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe55408c6f0 msgr2=0x7fe55408cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:31.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.939+0000 7fe55aae6700 1 --2- 172.21.15.16:0/2722954362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe55408c6f0 0x7fe55408cae0 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7fe54c00b2d0 tx=0x7fe54c00bae0).stop 2019-12-05T06:06:31.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.940+0000 7fe55aae6700 1 -- 172.21.15.16:0/2722954362 shutdown_connections 2019-12-05T06:06:31.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.940+0000 7fe55aae6700 1 --2- 172.21.15.16:0/2722954362 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe55408c6f0 0x7fe55408cae0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:31.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.940+0000 7fe55aae6700 1 -- 172.21.15.16:0/2722954362 shutdown_connections 2019-12-05T06:06:31.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.941+0000 7fe55aae6700 1 -- 172.21.15.16:0/2722954362 wait complete. 2019-12-05T06:06:31.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.941+0000 7fe55aae6700 1 -- 172.21.15.16:0/2722954362 >> 172.21.15.16:0/2722954362 conn(0x7fe55405f5a0 msgr2=0x7fe554096080 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.946+0000 7fe55aae6700 1 Processor -- start 2019-12-05T06:06:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.946+0000 7fe55aae6700 1 -- start start 2019-12-05T06:06:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.948+0000 7fe55aae6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe5541286c0 0x7fe554128f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:31.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.948+0000 7fe55aae6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe554198c50 0x7fe55419b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:31.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.948+0000 7fe55aae6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe55412dc70 con 0x7fe554129ed0 2019-12-05T06:06:31.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.948+0000 7fe55aae6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe55412ef00 con 0x7fe554198c50 2019-12-05T06:06:31.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.948+0000 7fe55aae6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe55412f1d0 con 0x7fe5541286c0 2019-12-05T06:06:31.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.948+0000 7fe559884700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe554198c50 0x7fe55419b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.948+0000 7fe558882700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe554129ed0 0x7fe554129440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52318/0 (socket says 172.21.15.16:52318) 2019-12-05T06:06:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.948+0000 7fe559083700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe5541286c0 0x7fe554128f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.948+0000 7fe558882700 1 -- 172.21.15.16:0/1620416383 learned_addr learned my addr 172.21.15.16:0/1620416383 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.949+0000 7fe551ffb700 1 -- 172.21.15.16:0/1620416383 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1727916823 0 0) 0x7fe5480010f0 con 0x7fe554129ed0 2019-12-05T06:06:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.949+0000 7fe551ffb700 1 -- 172.21.15.16:0/1620416383 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe534002e80 con 0x7fe554129ed0 2019-12-05T06:06:31.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.949+0000 7fe559884700 1 -- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe5541286c0 msgr2=0x7fe554128f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.949+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe5541286c0 0x7fe554128f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.949+0000 7fe559884700 1 -- 172.21.15.16:0/1620416383 >> v1:172.21.15.76:6789/0 conn(0x7fe554129ed0 legacy=0x7fe554129440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.949+0000 7fe559884700 1 -- 172.21.15.16:0/1620416383 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe55412ffc0 con 0x7fe554198c50 2019-12-05T06:06:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.949+0000 7fe559083700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe5541286c0 0x7fe554128f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:31.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.949+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> v2:172.21.15.76:3300/0 conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.949+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> v2:172.21.15.76:3300/0 conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.949+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=READY pgs=55 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.950+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.950+0000 7fe551ffb700 1 -- 172.21.15.16:0/1620416383 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe544000ef0 con 0x7fe554198c50 2019-12-05T06:06:31.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.950+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.950+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.950+0000 7fe551ffb700 1 -- 172.21.15.16:0/1620416383 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe544004240 con 0x7fe554198c50 2019-12-05T06:06:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.950+0000 7fe55aae6700 1 -- 172.21.15.16:0/1620416383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe55412ffc0 con 0x7fe554198c50 2019-12-05T06:06:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.950+0000 7fe551ffb700 1 -- 172.21.15.16:0/1620416383 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5440048d0 con 0x7fe554198c50 2019-12-05T06:06:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.950+0000 7fe55aae6700 1 -- 172.21.15.16:0/1620416383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe55412ffc0 con 0x7fe554198c50 2019-12-05T06:06:31.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.951+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.952+0000 7fe551ffb700 1 -- 172.21.15.16:0/1620416383 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe544000940 con 0x7fe554198c50 2019-12-05T06:06:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.952+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.952+0000 7fe551ffb700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe534023670 0x7fe534025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.952+0000 7fe551ffb700 1 -- 172.21.15.16:0/1620416383 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4842+0+0 (secure 0 0 0) 0x7fe544045d20 con 0x7fe554198c50 2019-12-05T06:06:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.953+0000 7fe559083700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe534023670 0x7fe534025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.953+0000 7fe559083700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe534023670 0x7fe534025ae0 crc :-1 s=READY pgs=42 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:31.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.966+0000 7fe55aae6700 1 -- 172.21.15.16:0/1620416383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe55412ffc0 con 0x7fe554198c50 2019-12-05T06:06:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.975+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:31.975+0000 7fe551ffb700 1 -- 172.21.15.16:0/1620416383 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe544001200 con 0x7fe554198c50 2019-12-05T06:06:32.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.223+0000 7fe55aae6700 1 -- 172.21.15.16:0/1620416383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe55412ffc0 con 0x7fe554198c50 2019-12-05T06:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.590+0000 7fe559884700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.590+0000 7fe551ffb700 1 -- 172.21.15.16:0/1620416383 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "app": "rbd", "pool": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe54402af90 con 0x7fe554198c50 2019-12-05T06:06:32.636 INFO:teuthology.orchestra.run.smithi016.stderr:enabled application 'rbd' on pool 'rbd' 2019-12-05T06:06:32.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.642+0000 7fe55aae6700 1 -- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe534023670 msgr2=0x7fe534025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:32.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.642+0000 7fe55aae6700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe534023670 0x7fe534025ae0 crc :-1 s=READY pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:32.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.642+0000 7fe55aae6700 1 -- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 msgr2=0x7fe55419b050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:32.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.642+0000 7fe55aae6700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 0x7fe55419b050 secure :-1 s=READY pgs=55 cs=0 l=1 rx=0x7fe544010370 tx=0x7fe544010710).stop 2019-12-05T06:06:32.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.643+0000 7fe55aae6700 1 -- 172.21.15.16:0/1620416383 shutdown_connections 2019-12-05T06:06:32.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.643+0000 7fe55aae6700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe534023670 0x7fe534025ae0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:32.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.643+0000 7fe55aae6700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe554198c50 0x7fe55419b050 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.643+0000 7fe55aae6700 1 --2- 172.21.15.16:0/1620416383 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe5541286c0 0x7fe554128f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.643+0000 7fe55aae6700 1 -- 172.21.15.16:0/1620416383 shutdown_connections 2019-12-05T06:06:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.643+0000 7fe55aae6700 1 -- 172.21.15.16:0/1620416383 wait complete. 2019-12-05T06:06:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.643+0000 7fe55aae6700 1 -- 172.21.15.16:0/1620416383 >> 172.21.15.16:0/1620416383 conn(0x7fe55407f3f0 msgr2=0x7fe554130610 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:32.663 INFO:tasks.ceph:Setting up CephFS filesystem... 2019-12-05T06:06:32.664 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:06:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.911+0000 7fe4e1d5d700 1 Processor -- start 2019-12-05T06:06:32.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.911+0000 7fe4e1d5d700 1 -- start start 2019-12-05T06:06:32.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.913+0000 7fe4e1d5d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.913+0000 7fe4e1d5d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe4dc08fef0 0x7fe4dc08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.913+0000 7fe4e1d5d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4dc0958e0 con 0x7fe4dc08c6d0 2019-12-05T06:06:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.913+0000 7fe4e1d5d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc08df90 con 0x7fe4dc08d2c0 2019-12-05T06:06:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.913+0000 7fe4e1d5d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4dc0a3f70 con 0x7fe4dc08fef0 2019-12-05T06:06:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.913+0000 7fe4db7fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.913+0000 7fe4db7fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:60184/0 (socket says 172.21.15.16:60184) 2019-12-05T06:06:32.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.913+0000 7fe4db7fe700 1 -- 172.21.15.16:0/1636123661 learned_addr learned my addr 172.21.15.16:0/1636123661 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.913+0000 7fe4e0afb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe4dc08fef0 0x7fe4dc08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4db7fe700 1 -- 172.21.15.16:0/1636123661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe4dc08fef0 msgr2=0x7fe4dc08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4db7fe700 1 --2- 172.21.15.16:0/1636123661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe4dc08fef0 0x7fe4dc08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4db7fe700 1 -- 172.21.15.16:0/1636123661 >> v1:172.21.15.76:6789/0 conn(0x7fe4dc08c6d0 legacy=0x7fe4dc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4db7fe700 1 -- 172.21.15.16:0/1636123661 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc08e620 con 0x7fe4dc08d2c0 2019-12-05T06:06:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4db7fe700 1 --2- 172.21.15.16:0/1636123661 >> v2:172.21.15.76:3300/0 conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe4d000f740 tx=0x7fe4d000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4db7fe700 1 --2- 172.21.15.16:0/1636123661 >> v2:172.21.15.76:3300/0 conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe4d000f740 tx=0x7fe4d000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4db7fe700 1 --2- 172.21.15.16:0/1636123661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 secure :-1 s=READY pgs=57 cs=0 l=1 rx=0x7fe4d000f740 tx=0x7fe4d000f570).ready entity=mon.1 client_cookie=17b21aebfd42e62d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:32.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4db7fe700 1 --2- 172.21.15.16:0/1636123661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7fe4d000f740 tx=0x7fe4d000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4db7fe700 1 --2- 172.21.15.16:0/1636123661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7fe4d000f740 tx=0x7fe4d000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4db7fe700 1 --2- 172.21.15.16:0/1636123661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7fe4d000f740 tx=0x7fe4d000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.914+0000 7fe4daffd700 1 -- 172.21.15.16:0/1636123661 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4d0000960 con 0x7fe4dc08d2c0 2019-12-05T06:06:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.915+0000 7fe4daffd700 1 -- 172.21.15.16:0/1636123661 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d0000c90 con 0x7fe4dc08d2c0 2019-12-05T06:06:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.915+0000 7fe4daffd700 1 -- 172.21.15.16:0/1636123661 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4d00012a0 con 0x7fe4dc08d2c0 2019-12-05T06:06:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.915+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/1636123661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4dc08d2c0 msgr2=0x7fe4dc08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.915+0000 7fe4e1d5d700 1 --2- 172.21.15.16:0/1636123661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 secure :-1 s=READY pgs=57 cs=0 l=1 rx=0x7fe4d000f740 tx=0x7fe4d000f570).stop 2019-12-05T06:06:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.916+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/1636123661 shutdown_connections 2019-12-05T06:06:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.916+0000 7fe4e1d5d700 1 --2- 172.21.15.16:0/1636123661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe4dc08fef0 0x7fe4dc08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:32.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.916+0000 7fe4e1d5d700 1 --2- 172.21.15.16:0/1636123661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe4dc08d2c0 0x7fe4dc08ef20 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:32.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.916+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/1636123661 shutdown_connections 2019-12-05T06:06:32.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.916+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/1636123661 wait complete. 2019-12-05T06:06:32.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.916+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/1636123661 >> 172.21.15.16:0/1636123661 conn(0x7fe4dc061940 msgr2=0x7fe4dc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:32.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.921+0000 7fe4e1d5d700 1 Processor -- start 2019-12-05T06:06:32.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.922+0000 7fe4e1d5d700 1 -- start start 2019-12-05T06:06:32.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.923+0000 7fe4e1d5d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe4dc09e2c0 0x7fe4dc09dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:32.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.923+0000 7fe4e1d5d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4dc14d9c0 0x7fe4dc19b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:32.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.923+0000 7fe4e1d5d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4dc14de80 con 0x7fe4dc14d5d0 2019-12-05T06:06:32.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.923+0000 7fe4e1d5d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc12b3f0 con 0x7fe4dc14d9c0 2019-12-05T06:06:32.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.923+0000 7fe4e1d5d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4dc12a9c0 con 0x7fe4dc09e2c0 2019-12-05T06:06:32.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.924+0000 7fe4e0afb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4dc14d9c0 0x7fe4dc19b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:32.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.924+0000 7fe4e0afb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe4dc14d9c0 0x7fe4dc19b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45870/0 (socket says 172.21.15.16:45870) 2019-12-05T06:06:32.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.924+0000 7fe4e0afb700 1 -- 172.21.15.16:0/3241866401 learned_addr learned my addr 172.21.15.16:0/3241866401 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:32.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.924+0000 7fe4db7fe700 1 --2- 172.21.15.16:0/3241866401 >> v2:172.21.15.76:3300/0 conn(0x7fe4dc09e2c0 0x7fe4dc09dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:32.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.924+0000 7fe4e0afb700 1 -- 172.21.15.16:0/3241866401 >> v2:172.21.15.76:3300/0 conn(0x7fe4dc09e2c0 msgr2=0x7fe4dc09dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:32.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.924+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> v2:172.21.15.76:3300/0 conn(0x7fe4dc09e2c0 0x7fe4dc09dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:32.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.924+0000 7fe4e0afb700 1 -- 172.21.15.16:0/3241866401 >> v1:172.21.15.76:6789/0 conn(0x7fe4dc14d5d0 legacy=0x7fe4dc09d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.924+0000 7fe4e0afb700 1 -- 172.21.15.16:0/3241866401 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4dc12ba80 con 0x7fe4dc14d9c0 2019-12-05T06:06:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.924+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> v2:172.21.15.16:3300/0 conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.925+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> v2:172.21.15.16:3300/0 conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.925+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=READY pgs=52 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.925+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.925+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.925+0000 7fe4d8ff9700 1 -- 172.21.15.16:0/3241866401 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4d4000e90 con 0x7fe4dc14d9c0 2019-12-05T06:06:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.925+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.925+0000 7fe4d8ff9700 1 -- 172.21.15.16:0/3241866401 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4d400f2c0 con 0x7fe4dc14d9c0 2019-12-05T06:06:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.925+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/3241866401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4dc12ba80 con 0x7fe4dc14d9c0 2019-12-05T06:06:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.925+0000 7fe4d8ff9700 1 -- 172.21.15.16:0/3241866401 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4d400f950 con 0x7fe4dc14d9c0 2019-12-05T06:06:32.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.925+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/3241866401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4dc12ba80 con 0x7fe4dc14d9c0 2019-12-05T06:06:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.926+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.927+0000 7fe4d8ff9700 1 -- 172.21.15.16:0/3241866401 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe4d400ef60 con 0x7fe4dc14d9c0 2019-12-05T06:06:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.927+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.927+0000 7fe4d8ff9700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4bc023630 0x7fe4bc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.927+0000 7fe4d8ff9700 1 -- 172.21.15.16:0/3241866401 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fe4d40461c0 con 0x7fe4dc14d9c0 2019-12-05T06:06:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.927+0000 7fe4dbfff700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4bc023630 0x7fe4bc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.928+0000 7fe4dbfff700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4bc023630 0x7fe4bc025ac0 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.942+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/3241866401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4dc12ba80 con 0x7fe4dc14d9c0 2019-12-05T06:06:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.946+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:32.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:32.946+0000 7fe4d8ff9700 1 -- 172.21.15.16:0/3241866401 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe4d4000e90 con 0x7fe4dc14d9c0 2019-12-05T06:06:33.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.186+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/3241866401 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe4dc12ba80 con 0x7fe4dc14d9c0 2019-12-05T06:06:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.188+0000 7fe4e0afb700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.188+0000 7fe4d8ff9700 1 -- 172.21.15.16:0/3241866401 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+11778 (secure 0 0 0) 0x7fe4d400ee80 con 0x7fe4dc14d9c0 2019-12-05T06:06:33.188 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:33.189 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:32.580529+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:06:33.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.191+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/3241866401 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4bc023630 msgr2=0x7fe4bc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.191+0000 7fe4e1d5d700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4bc023630 0x7fe4bc025ac0 crc :-1 s=READY pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:33.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.191+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 msgr2=0x7fe4dc19b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.191+0000 7fe4e1d5d700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 0x7fe4dc19b400 secure :-1 s=READY pgs=52 cs=0 l=1 rx=0x7fe4d40107d0 tx=0x7fe4d4010600).stop 2019-12-05T06:06:33.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.192+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/3241866401 shutdown_connections 2019-12-05T06:06:33.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.192+0000 7fe4e1d5d700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4bc023630 0x7fe4bc025ac0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:33.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.192+0000 7fe4e1d5d700 1 --2- 172.21.15.16:0/3241866401 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe4dc14d9c0 0x7fe4dc19b400 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:33.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.192+0000 7fe4e1d5d700 1 --2- 172.21.15.16:0/3241866401 >> v2:172.21.15.76:3300/0 conn(0x7fe4dc09e2c0 0x7fe4dc09dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:33.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.192+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/3241866401 shutdown_connections 2019-12-05T06:06:33.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.192+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/3241866401 wait complete. 2019-12-05T06:06:33.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.192+0000 7fe4e1d5d700 1 -- 172.21.15.16:0/3241866401 >> 172.21.15.16:0/3241866401 conn(0x7fe4dc06d260 msgr2=0x7fe4dc09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:33.201 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool,creating', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:31.052682+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 32769, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'16', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T06:06:33.202 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T06:06:33.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.457+0000 7f881a568700 1 Processor -- start 2019-12-05T06:06:33.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.457+0000 7f881a568700 1 -- start start 2019-12-05T06:06:33.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.458+0000 7f881a568700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8814086d70 0x7f881408aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.458+0000 7f881a568700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8814061a10 con 0x7f8814064310 2019-12-05T06:06:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.458+0000 7f881a568700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8814064f70 con 0x7f8814086980 2019-12-05T06:06:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.458+0000 7f881a568700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8814060bc0 con 0x7f8814086d70 2019-12-05T06:06:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.459+0000 7f8819306700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8814064310 0x7f8814064770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59278/0 (socket says 172.21.15.16:59278) 2019-12-05T06:06:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.459+0000 7f8819306700 1 -- 172.21.15.16:0/3817463206 learned_addr learned my addr 172.21.15.16:0/3817463206 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:33.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.459+0000 7f8813fff700 1 --2- 172.21.15.16:0/3817463206 >> v2:172.21.15.76:3300/0 conn(0x7f8814086d70 0x7f881408aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.459+0000 7f88137fe700 1 -- 172.21.15.16:0/3817463206 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58568392 0 0) 0x7f87fc001150 con 0x7f8814064310 2019-12-05T06:06:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.459+0000 7f88137fe700 1 -- 172.21.15.16:0/3817463206 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8800002e80 con 0x7f8814064310 2019-12-05T06:06:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.460+0000 7f88137fe700 1 -- 172.21.15.16:0/3817463206 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225436332 0 0) 0x7f8804001400 con 0x7f8814086980 2019-12-05T06:06:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.460+0000 7f8813fff700 1 -- 172.21.15.16:0/3817463206 >> v1:172.21.15.76:6789/0 conn(0x7f8814086980 legacy=0x7f881408a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.460+0000 7f8813fff700 1 -- 172.21.15.16:0/3817463206 >> v1:172.21.15.16:6789/0 conn(0x7f8814064310 legacy=0x7f8814064770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.460+0000 7f8813fff700 1 -- 172.21.15.16:0/3817463206 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8814063a10 con 0x7f8814086d70 2019-12-05T06:06:33.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.460+0000 7f8813fff700 1 --2- 172.21.15.16:0/3817463206 >> v2:172.21.15.76:3300/0 conn(0x7f8814086d70 0x7f881408aeb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f880c00af50 tx=0x7f880c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.460+0000 7f8813fff700 1 --2- 172.21.15.16:0/3817463206 >> v2:172.21.15.76:3300/0 conn(0x7f8814086d70 0x7f881408aeb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f880c00af50 tx=0x7f880c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.460+0000 7f8813fff700 1 --2- 172.21.15.16:0/3817463206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8814086d70 0x7f881408aeb0 secure :-1 s=READY pgs=60 cs=0 l=1 rx=0x7f880c00af50 tx=0x7f880c00b760).ready entity=mon.1 client_cookie=cbf3f9854d389123 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.460+0000 7f8813fff700 1 --2- 172.21.15.16:0/3817463206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8814086d70 0x7f881408aeb0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f880c00af50 tx=0x7f880c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.461+0000 7f8813fff700 1 --2- 172.21.15.16:0/3817463206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8814086d70 0x7f881408aeb0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f880c00af50 tx=0x7f880c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.461+0000 7f88137fe700 1 -- 172.21.15.16:0/3817463206 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f880c000960 con 0x7f8814086d70 2019-12-05T06:06:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.461+0000 7f8813fff700 1 --2- 172.21.15.16:0/3817463206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8814086d70 0x7f881408aeb0 secure :-1 s=THROTTLE_DONE pgs=60 cs=0 l=1 rx=0x7f880c00af50 tx=0x7f880c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.461+0000 7f88137fe700 1 -- 172.21.15.16:0/3817463206 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f880c0098e0 con 0x7f8814086d70 2019-12-05T06:06:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.461+0000 7f88137fe700 1 -- 172.21.15.16:0/3817463206 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f880c00a050 con 0x7f8814086d70 2019-12-05T06:06:33.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.461+0000 7f881a568700 1 -- 172.21.15.16:0/3817463206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8814086d70 msgr2=0x7f881408aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.461+0000 7f881a568700 1 --2- 172.21.15.16:0/3817463206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8814086d70 0x7f881408aeb0 secure :-1 s=READY pgs=60 cs=0 l=1 rx=0x7f880c00af50 tx=0x7f880c00b760).stop 2019-12-05T06:06:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.462+0000 7f881a568700 1 -- 172.21.15.16:0/3817463206 shutdown_connections 2019-12-05T06:06:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.462+0000 7f881a568700 1 --2- 172.21.15.16:0/3817463206 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8814086d70 0x7f881408aeb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.462+0000 7f881a568700 1 -- 172.21.15.16:0/3817463206 shutdown_connections 2019-12-05T06:06:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.463+0000 7f881a568700 1 -- 172.21.15.16:0/3817463206 wait complete. 2019-12-05T06:06:33.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.463+0000 7f881a568700 1 -- 172.21.15.16:0/3817463206 >> 172.21.15.16:0/3817463206 conn(0x7f881405f5a0 msgr2=0x7f881409cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:33.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.466+0000 7f881a568700 1 Processor -- start 2019-12-05T06:06:33.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.466+0000 7f881a568700 1 -- start start 2019-12-05T06:06:33.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.467+0000 7f881a568700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f881409a980 0x7f881412c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:33.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.468+0000 7f881a568700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881409b350 con 0x7f881412d2e0 2019-12-05T06:06:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.468+0000 7f881a568700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881409b900 con 0x7f8814154c60 2019-12-05T06:06:33.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.468+0000 7f881a568700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8814099ec0 con 0x7f881409a980 2019-12-05T06:06:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.468+0000 7f8813fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f881412d2e0 0x7f881412c850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59284/0 (socket says 172.21.15.16:59284) 2019-12-05T06:06:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.468+0000 7f8813fff700 1 -- 172.21.15.16:0/1878310320 learned_addr learned my addr 172.21.15.16:0/1878310320 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.468+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> v2:172.21.15.76:3300/0 conn(0x7f881409a980 0x7f881412c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.468+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915245787 0 0) 0x7f880c00b3d0 con 0x7f881412d2e0 2019-12-05T06:06:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.468+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f8002e80 con 0x7f881412d2e0 2019-12-05T06:06:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.468+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178637817 0 0) 0x7f87fc0009b0 con 0x7f8814154c60 2019-12-05T06:06:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.468+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f8002ec0 con 0x7f8814154c60 2019-12-05T06:06:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f8818b05700 1 -- 172.21.15.16:0/1878310320 >> v1:172.21.15.76:6789/0 conn(0x7f8814154c60 legacy=0x7f8814157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f8818b05700 1 -- 172.21.15.16:0/1878310320 >> v1:172.21.15.16:6789/0 conn(0x7f881412d2e0 legacy=0x7f881412c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f8818b05700 1 -- 172.21.15.16:0/1878310320 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881409c330 con 0x7f881409a980 2019-12-05T06:06:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> v2:172.21.15.76:3300/0 conn(0x7f881409a980 0x7f881412c310 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> v2:172.21.15.76:3300/0 conn(0x7f881409a980 0x7f881412c310 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 0x7f881412c310 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 0x7f881412c310 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8804009ce0 con 0x7f881409a980 2019-12-05T06:06:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 0x7f881412c310 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 0x7f881412c310 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8804000df0 con 0x7f881409a980 2019-12-05T06:06:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f881a568700 1 -- 172.21.15.16:0/1878310320 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881409c330 con 0x7f881409a980 2019-12-05T06:06:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.469+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f880400a2c0 con 0x7f881409a980 2019-12-05T06:06:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.470+0000 7f881a568700 1 -- 172.21.15.16:0/1878310320 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f881409c330 con 0x7f881409a980 2019-12-05T06:06:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.470+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 0x7f881412c310 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.471+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 0x7f881412c310 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.471+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8804009980 con 0x7f881409a980 2019-12-05T06:06:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.471+0000 7f88117fa700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87f8023680 0x7f87f8025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.471+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f8804046380 con 0x7f881409a980 2019-12-05T06:06:33.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.472+0000 7f8813fff700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87f8023680 0x7f87f8025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.472+0000 7f8813fff700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87f8023680 0x7f87f8025af0 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.472+0000 7f881a568700 1 -- 172.21.15.16:0/1878310320 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f881409c330 con 0x7f881409a980 2019-12-05T06:06:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.482+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 0x7f881412c310 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.482+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8804000df0 con 0x7f881409a980 2019-12-05T06:06:33.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.720+0000 7f881a568700 1 -- 172.21.15.16:0/1878310320 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f881409c330 con 0x7f881409a980 2019-12-05T06:06:33.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.721+0000 7f8818b05700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 0x7f881412c310 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.721+0000 7f88117fa700 1 -- 172.21.15.16:0/1878310320 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v17) v1 ==== 92+0+10 (secure 0 0 0) 0x7f880402b3a0 con 0x7f881409a980 2019-12-05T06:06:33.721 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:06:33.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.727+0000 7f881a568700 1 -- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87f8023680 msgr2=0x7f87f8025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.727+0000 7f881a568700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87f8023680 0x7f87f8025af0 crc :-1 s=READY pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:33.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.727+0000 7f881a568700 1 -- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 msgr2=0x7f881412c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.727+0000 7f881a568700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 0x7f881412c310 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7f8804010720 tx=0x7f8804010ac0).stop 2019-12-05T06:06:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.727+0000 7f881a568700 1 -- 172.21.15.16:0/1878310320 shutdown_connections 2019-12-05T06:06:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.727+0000 7f881a568700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87f8023680 0x7f87f8025af0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:33.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.727+0000 7f881a568700 1 --2- 172.21.15.16:0/1878310320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f881409a980 0x7f881412c310 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:33.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.727+0000 7f881a568700 1 -- 172.21.15.16:0/1878310320 shutdown_connections 2019-12-05T06:06:33.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.727+0000 7f881a568700 1 -- 172.21.15.16:0/1878310320 wait complete. 2019-12-05T06:06:33.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.727+0000 7f881a568700 1 -- 172.21.15.16:0/1878310320 >> 172.21.15.16:0/1878310320 conn(0x7f8814095970 msgr2=0x7f88140877d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:33.741 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2019-12-05T06:06:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.980+0000 7fa8db50c700 1 Processor -- start 2019-12-05T06:06:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.981+0000 7fa8db50c700 1 -- start start 2019-12-05T06:06:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.982+0000 7fa8db50c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 0x7fa8d408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.982+0000 7fa8db50c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d40958e0 con 0x7fa8d408fef0 2019-12-05T06:06:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.982+0000 7fa8db50c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d40629e0 con 0x7fa8d408c6d0 2019-12-05T06:06:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.982+0000 7fa8db50c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d408df90 con 0x7fa8d408d2c0 2019-12-05T06:06:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.983+0000 7fa8da2aa700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa8d408fef0 0x7fa8d408f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59290/0 (socket says 172.21.15.16:59290) 2019-12-05T06:06:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.983+0000 7fa8da2aa700 1 -- 172.21.15.16:0/3997340510 learned_addr learned my addr 172.21.15.16:0/3997340510 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.983+0000 7fa8d92a8700 1 --2- 172.21.15.16:0/3997340510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 0x7fa8d408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.983+0000 7fa8d8aa7700 1 -- 172.21.15.16:0/3997340510 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343042492 0 0) 0x7fa8c4001150 con 0x7fa8d408fef0 2019-12-05T06:06:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.984+0000 7fa8d8aa7700 1 -- 172.21.15.16:0/3997340510 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8bc002e80 con 0x7fa8d408fef0 2019-12-05T06:06:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.984+0000 7fa8d8aa7700 1 -- 172.21.15.16:0/3997340510 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398523006 0 0) 0x7fa8cc001400 con 0x7fa8d408c6d0 2019-12-05T06:06:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.984+0000 7fa8d8aa7700 1 -- 172.21.15.16:0/3997340510 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8bc004170 con 0x7fa8d408c6d0 2019-12-05T06:06:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.984+0000 7fa8d92a8700 1 -- 172.21.15.16:0/3997340510 >> v1:172.21.15.76:6789/0 conn(0x7fa8d408c6d0 legacy=0x7fa8d408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.984+0000 7fa8d92a8700 1 -- 172.21.15.16:0/3997340510 >> v1:172.21.15.16:6789/0 conn(0x7fa8d408fef0 legacy=0x7fa8d408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.984+0000 7fa8d92a8700 1 -- 172.21.15.16:0/3997340510 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d408e620 con 0x7fa8d408d2c0 2019-12-05T06:06:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.984+0000 7fa8d92a8700 1 --2- 172.21.15.16:0/3997340510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 0x7fa8d408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa8c800af50 tx=0x7fa8c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.984+0000 7fa8d92a8700 1 --2- 172.21.15.16:0/3997340510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 0x7fa8d408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa8c800af50 tx=0x7fa8c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.984+0000 7fa8d92a8700 1 --2- 172.21.15.16:0/3997340510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 0x7fa8d408ef20 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7fa8c800af50 tx=0x7fa8c800b760).ready entity=mon.2 client_cookie=43ad6be04e561ac4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.984+0000 7fa8d92a8700 1 --2- 172.21.15.16:0/3997340510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 0x7fa8d408ef20 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fa8c800af50 tx=0x7fa8c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.985+0000 7fa8d8aa7700 1 -- 172.21.15.16:0/3997340510 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8c8000960 con 0x7fa8d408d2c0 2019-12-05T06:06:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.985+0000 7fa8d92a8700 1 --2- 172.21.15.16:0/3997340510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 0x7fa8d408ef20 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fa8c800af50 tx=0x7fa8c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.985+0000 7fa8d92a8700 1 --2- 172.21.15.16:0/3997340510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 0x7fa8d408ef20 secure :-1 s=THROTTLE_DONE pgs=17 cs=0 l=1 rx=0x7fa8c800af50 tx=0x7fa8c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.985+0000 7fa8d8aa7700 1 -- 172.21.15.16:0/3997340510 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c8009860 con 0x7fa8d408d2c0 2019-12-05T06:06:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.985+0000 7fa8d8aa7700 1 -- 172.21.15.16:0/3997340510 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8c8009f30 con 0x7fa8d408d2c0 2019-12-05T06:06:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.985+0000 7fa8db50c700 1 -- 172.21.15.16:0/3997340510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 msgr2=0x7fa8d408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.985+0000 7fa8db50c700 1 --2- 172.21.15.16:0/3997340510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 0x7fa8d408ef20 secure :-1 s=READY pgs=17 cs=0 l=1 rx=0x7fa8c800af50 tx=0x7fa8c800b760).stop 2019-12-05T06:06:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.986+0000 7fa8db50c700 1 -- 172.21.15.16:0/3997340510 shutdown_connections 2019-12-05T06:06:33.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.986+0000 7fa8db50c700 1 --2- 172.21.15.16:0/3997340510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d408d2c0 0x7fa8d408ef20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:33.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.987+0000 7fa8db50c700 1 -- 172.21.15.16:0/3997340510 shutdown_connections 2019-12-05T06:06:33.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.987+0000 7fa8db50c700 1 -- 172.21.15.16:0/3997340510 wait complete. 2019-12-05T06:06:33.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.987+0000 7fa8db50c700 1 -- 172.21.15.16:0/3997340510 >> 172.21.15.16:0/3997340510 conn(0x7fa8d4061940 msgr2=0x7fa8d4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:33.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.992+0000 7fa8db50c700 1 Processor -- start 2019-12-05T06:06:33.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.992+0000 7fa8db50c700 1 -- start start 2019-12-05T06:06:33.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.993+0000 7fa8db50c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa8d4128360 0x7fa8d4128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:33.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.993+0000 7fa8db50c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d4137ea0 0x7fa8d4128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:33.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.993+0000 7fa8db50c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa8d412b5d0 0x7fa8d412ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.993+0000 7fa8db50c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d412a640 con 0x7fa8d412b5d0 2019-12-05T06:06:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.993+0000 7fa8db50c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d4129c10 con 0x7fa8d4128360 2019-12-05T06:06:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.993+0000 7fa8db50c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d4129ee0 con 0x7fa8d4137ea0 2019-12-05T06:06:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.993+0000 7fa8da2aa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa8d412b5d0 0x7fa8d412ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8da2aa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa8d412b5d0 0x7fa8d412ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45896/0 (socket says 172.21.15.16:45896) 2019-12-05T06:06:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8da2aa700 1 -- 172.21.15.16:0/2060268232 learned_addr learned my addr 172.21.15.16:0/2060268232 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:33.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8d9aa9700 1 --2- 172.21.15.16:0/2060268232 >> v2:172.21.15.76:3300/0 conn(0x7fa8d4128360 0x7fa8d4128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:33.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8d92a8700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d4137ea0 0x7fa8d4128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:33.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8da2aa700 1 -- 172.21.15.16:0/2060268232 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d4137ea0 msgr2=0x7fa8d4128c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d4137ea0 0x7fa8d4128c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:33.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8da2aa700 1 -- 172.21.15.16:0/2060268232 >> v2:172.21.15.76:3300/0 conn(0x7fa8d4128360 msgr2=0x7fa8d4128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:33.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> v2:172.21.15.76:3300/0 conn(0x7fa8d4128360 0x7fa8d4128750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:34.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8da2aa700 1 -- 172.21.15.16:0/2060268232 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d412acd0 con 0x7fa8d412b5d0 2019-12-05T06:06:34.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> v2:172.21.15.16:3300/0 conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> v2:172.21.15.16:3300/0 conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.994+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:34.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.995+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.995+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.995+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.995+0000 7fa8d27fc700 1 -- 172.21.15.16:0/2060268232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8c4000940 con 0x7fa8d412b5d0 2019-12-05T06:06:34.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.995+0000 7fa8d27fc700 1 -- 172.21.15.16:0/2060268232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c4004240 con 0x7fa8d412b5d0 2019-12-05T06:06:34.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.995+0000 7fa8db50c700 1 -- 172.21.15.16:0/2060268232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d412acd0 con 0x7fa8d412b5d0 2019-12-05T06:06:34.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.995+0000 7fa8d27fc700 1 -- 172.21.15.16:0/2060268232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8c4004850 con 0x7fa8d412b5d0 2019-12-05T06:06:34.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.995+0000 7fa8db50c700 1 -- 172.21.15.16:0/2060268232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d412acd0 con 0x7fa8d412b5d0 2019-12-05T06:06:34.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.996+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.997+0000 7fa8d27fc700 1 -- 172.21.15.16:0/2060268232 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa8c4004430 con 0x7fa8d412b5d0 2019-12-05T06:06:34.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.997+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.997+0000 7fa8d27fc700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa8b4023630 0x7fa8b4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:34.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.998+0000 7fa8d27fc700 1 -- 172.21.15.16:0/2060268232 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7fa8c4047f70 con 0x7fa8d412b5d0 2019-12-05T06:06:34.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.998+0000 7fa8d9aa9700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa8b4023630 0x7fa8b4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:34.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:33.998+0000 7fa8d9aa9700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa8b4023630 0x7fa8b4025ac0 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:34.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.012+0000 7fa8db50c700 1 -- 172.21.15.16:0/2060268232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d412acd0 con 0x7fa8d412b5d0 2019-12-05T06:06:34.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.021+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.021+0000 7fa8d27fc700 1 -- 172.21.15.16:0/2060268232 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa8c40012e0 con 0x7fa8d412b5d0 2019-12-05T06:06:34.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.269+0000 7fa8db50c700 1 -- 172.21.15.16:0/2060268232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fa8d412acd0 con 0x7fa8d412b5d0 2019-12-05T06:06:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.270+0000 7fa8da2aa700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.270+0000 7fa8d27fc700 1 -- 172.21.15.16:0/2060268232 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) v1 ==== 84+0+64 (secure 0 0 0) 0x7fa8c4004430 con 0x7fa8d412b5d0 2019-12-05T06:06:34.268 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout:[ 2019-12-05T06:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: { 2019-12-05T06:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: "poolnum": 1, 2019-12-05T06:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: "poolname": "rbd" 2019-12-05T06:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout: } 2019-12-05T06:06:34.269 INFO:teuthology.orchestra.run.smithi016.stdout:] 2019-12-05T06:06:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.272+0000 7fa8db50c700 1 -- 172.21.15.16:0/2060268232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa8b4023630 msgr2=0x7fa8b4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.272+0000 7fa8db50c700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa8b4023630 0x7fa8b4025ac0 crc :-1 s=READY pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.272+0000 7fa8db50c700 1 -- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 msgr2=0x7fa8d412ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.272+0000 7fa8db50c700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 0x7fa8d412ba30 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7fa8c4012640 tx=0x7fa8c40102e0).stop 2019-12-05T06:06:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.273+0000 7fa8db50c700 1 -- 172.21.15.16:0/2060268232 shutdown_connections 2019-12-05T06:06:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.273+0000 7fa8db50c700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa8b4023630 0x7fa8b4025ac0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.273+0000 7fa8db50c700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa8d412b5d0 0x7fa8d412ba30 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.273+0000 7fa8db50c700 1 --2- 172.21.15.16:0/2060268232 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8d4137ea0 0x7fa8d4128c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.273+0000 7fa8db50c700 1 --2- 172.21.15.16:0/2060268232 >> v2:172.21.15.76:3300/0 conn(0x7fa8d4128360 0x7fa8d4128750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.273+0000 7fa8db50c700 1 -- 172.21.15.16:0/2060268232 shutdown_connections 2019-12-05T06:06:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.273+0000 7fa8db50c700 1 -- 172.21.15.16:0/2060268232 wait complete. 2019-12-05T06:06:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.273+0000 7fa8db50c700 1 -- 172.21.15.16:0/2060268232 >> 172.21.15.16:0/2060268232 conn(0x7fa8d406d260 msgr2=0x7fa8d409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:34.287 INFO:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2019-12-05T06:06:34.287 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2019-12-05T06:06:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.534+0000 7f2460d16700 1 Processor -- start 2019-12-05T06:06:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.534+0000 7f2460d16700 1 -- start start 2019-12-05T06:06:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f2460d16700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f2460d16700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f245c064310 0x7f245c064790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f2460d16700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f245c061a10 con 0x7f245c08a6b0 2019-12-05T06:06:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f2460d16700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f245c08b340 con 0x7f245c064310 2019-12-05T06:06:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f2460d16700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245c060bc0 con 0x7f245c086980 2019-12-05T06:06:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f245ad9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f245b59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f245c064310 0x7f245c064790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f245ad9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:49892/0 (socket says 172.21.15.16:49892) 2019-12-05T06:06:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f245a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f245c08a6b0 0x7f245c08ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52376/0 (socket says 172.21.15.16:52376) 2019-12-05T06:06:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f245b59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f245c064310 0x7f245c064790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:60226/0 (socket says 172.21.15.16:60226) 2019-12-05T06:06:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.536+0000 7f245ad9d700 1 -- 172.21.15.16:0/308572908 learned_addr learned my addr 172.21.15.16:0/308572908 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.537+0000 7f245ad9d700 1 -- 172.21.15.16:0/308572908 >> v2:172.21.15.76:3300/0 conn(0x7f245c064310 msgr2=0x7f245c064790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.537+0000 7f245ad9d700 1 --2- 172.21.15.16:0/308572908 >> v2:172.21.15.76:3300/0 conn(0x7f245c064310 0x7f245c064790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.537+0000 7f245ad9d700 1 -- 172.21.15.16:0/308572908 >> v1:172.21.15.76:6789/0 conn(0x7f245c08a6b0 legacy=0x7f245c08ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.537+0000 7f2459d9b700 1 -- 172.21.15.16:0/308572908 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184988337 0 0) 0x7f2444001080 con 0x7f245c08a6b0 2019-12-05T06:06:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.537+0000 7f245ad9d700 1 -- 172.21.15.16:0/308572908 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c063a10 con 0x7f245c086980 2019-12-05T06:06:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.537+0000 7f245b59e700 1 --2- 172.21.15.16:0/308572908 >> v2:172.21.15.76:3300/0 conn(0x7f245c064310 0x7f245c064790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.537+0000 7f245ad9d700 1 --2- 172.21.15.16:0/308572908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f245400fac0 tx=0x7f245400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.538+0000 7f245ad9d700 1 --2- 172.21.15.16:0/308572908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f245400fac0 tx=0x7f245400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.538+0000 7f245ad9d700 1 --2- 172.21.15.16:0/308572908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f245400fac0 tx=0x7f245400fe60).ready entity=mon.2 client_cookie=f6f61ead54db761c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.538+0000 7f245ad9d700 1 --2- 172.21.15.16:0/308572908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f245400fac0 tx=0x7f245400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.538+0000 7f245ad9d700 1 --2- 172.21.15.16:0/308572908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f245400fac0 tx=0x7f245400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.538+0000 7f2459d9b700 1 -- 172.21.15.16:0/308572908 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2454000ce0 con 0x7f245c086980 2019-12-05T06:06:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.538+0000 7f245ad9d700 1 --2- 172.21.15.16:0/308572908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 secure :-1 s=THROTTLE_DONE pgs=18 cs=0 l=1 rx=0x7f245400fac0 tx=0x7f245400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.538+0000 7f2459d9b700 1 -- 172.21.15.16:0/308572908 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2454001010 con 0x7f245c086980 2019-12-05T06:06:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.538+0000 7f2459d9b700 1 -- 172.21.15.16:0/308572908 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2454001620 con 0x7f245c086980 2019-12-05T06:06:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.538+0000 7f2460d16700 1 -- 172.21.15.16:0/308572908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 msgr2=0x7f245c086d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.538+0000 7f2460d16700 1 --2- 172.21.15.16:0/308572908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 secure :-1 s=READY pgs=18 cs=0 l=1 rx=0x7f245400fac0 tx=0x7f245400fe60).stop 2019-12-05T06:06:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.540+0000 7f2460d16700 1 -- 172.21.15.16:0/308572908 shutdown_connections 2019-12-05T06:06:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.540+0000 7f2460d16700 1 --2- 172.21.15.16:0/308572908 >> v2:172.21.15.76:3300/0 conn(0x7f245c064310 0x7f245c064790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.540+0000 7f2460d16700 1 --2- 172.21.15.16:0/308572908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c086980 0x7f245c086d70 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.540+0000 7f2460d16700 1 -- 172.21.15.16:0/308572908 shutdown_connections 2019-12-05T06:06:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.540+0000 7f2460d16700 1 -- 172.21.15.16:0/308572908 wait complete. 2019-12-05T06:06:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.540+0000 7f2460d16700 1 -- 172.21.15.16:0/308572908 >> 172.21.15.16:0/308572908 conn(0x7f245c05f5a0 msgr2=0x7f245c09cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.545+0000 7f2460d16700 1 Processor -- start 2019-12-05T06:06:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.546+0000 7f2460d16700 1 -- start start 2019-12-05T06:06:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.547+0000 7f2460d16700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.547+0000 7f2460d16700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f245c096050 con 0x7f245c073d80 2019-12-05T06:06:34.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.547+0000 7f2460d16700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f245c074c00 con 0x7f245c099990 2019-12-05T06:06:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.547+0000 7f2460d16700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f245c075e90 con 0x7f245c095c60 2019-12-05T06:06:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.547+0000 7f245ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f245c073d80 0x7f245c095460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59308/0 (socket says 172.21.15.16:59308) 2019-12-05T06:06:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.547+0000 7f245ad9d700 1 -- 172.21.15.16:0/506631154 learned_addr learned my addr 172.21.15.16:0/506631154 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.548+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.548+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853291025 0 0) 0x7f2454007950 con 0x7f245c073d80 2019-12-05T06:06:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.548+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2440002e80 con 0x7f245c073d80 2019-12-05T06:06:34.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.548+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439632467 0 0) 0x7f24440010c0 con 0x7f245c099990 2019-12-05T06:06:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.548+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24400030b0 con 0x7f245c099990 2019-12-05T06:06:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.548+0000 7f245b59e700 1 -- 172.21.15.16:0/506631154 >> v1:172.21.15.76:6789/0 conn(0x7f245c099990 legacy=0x7f245c156c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.548+0000 7f245b59e700 1 -- 172.21.15.16:0/506631154 >> v1:172.21.15.16:6789/0 conn(0x7f245c073d80 legacy=0x7f245c095460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.548+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3173555684 0 0) 0x7f2454007d60 con 0x7f245c073d80 2019-12-05T06:06:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.548+0000 7f245b59e700 1 -- 172.21.15.16:0/506631154 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f245c094b60 con 0x7f245c095c60 2019-12-05T06:06:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f244c00ab90 con 0x7f245c095c60 2019-12-05T06:06:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f244c0095b0 con 0x7f245c095c60 2019-12-05T06:06:34.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f2460d16700 1 -- 172.21.15.16:0/506631154 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f245c094b60 con 0x7f245c095c60 2019-12-05T06:06:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f244c009d00 con 0x7f245c095c60 2019-12-05T06:06:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.549+0000 7f2460d16700 1 -- 172.21.15.16:0/506631154 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f245c094b60 con 0x7f245c095c60 2019-12-05T06:06:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.550+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.551+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f244c000a70 con 0x7f245c095c60 2019-12-05T06:06:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.551+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.552+0000 7f24537fe700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2440023650 0x7f2440025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:34.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.552+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 4853+0+0 (secure 0 0 0) 0x7f244c045e20 con 0x7f245c095c60 2019-12-05T06:06:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.552+0000 7f245ad9d700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2440023650 0x7f2440025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.552+0000 7f245ad9d700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2440023650 0x7f2440025ae0 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:34.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.565+0000 7f2460d16700 1 -- 172.21.15.16:0/506631154 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f245c094b60 con 0x7f245c095c60 2019-12-05T06:06:34.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.575+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:34.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.575+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f244c00a670 con 0x7f245c095c60 2019-12-05T06:06:34.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:34.838+0000 7f2460d16700 1 -- 172.21.15.16:0/506631154 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"} v 0) v1 -- 0x7f245c094b60 con 0x7f245c095c60 2019-12-05T06:06:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.618+0000 7f245b59e700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=THROTTLE_DONE pgs=19 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.618+0000 7f24537fe700 1 -- 172.21.15.16:0/506631154 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pg_num": 8, "pool": "cephfs_metadata"}]=0 pool 'cephfs_metadata' created v18) v1 ==== 133+0+0 (secure 0 0 0) 0x7f244c000a70 con 0x7f245c095c60 2019-12-05T06:06:35.652 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_metadata' created 2019-12-05T06:06:35.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.658+0000 7f2460d16700 1 -- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2440023650 msgr2=0x7f2440025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:35.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.658+0000 7f2460d16700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2440023650 0x7f2440025ae0 crc :-1 s=READY pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:35.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.658+0000 7f2460d16700 1 -- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 msgr2=0x7f245c15a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:35.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.658+0000 7f2460d16700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 secure :-1 s=READY pgs=19 cs=0 l=1 rx=0x7f244c0103a0 tx=0x7f244c010740).stop 2019-12-05T06:06:35.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.658+0000 7f2460d16700 1 -- 172.21.15.16:0/506631154 shutdown_connections 2019-12-05T06:06:35.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.658+0000 7f2460d16700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2440023650 0x7f2440025ae0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:35.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.658+0000 7f2460d16700 1 --2- 172.21.15.16:0/506631154 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f245c095c60 0x7f245c15a4a0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:35.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.659+0000 7f2460d16700 1 -- 172.21.15.16:0/506631154 shutdown_connections 2019-12-05T06:06:35.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.659+0000 7f2460d16700 1 -- 172.21.15.16:0/506631154 wait complete. 2019-12-05T06:06:35.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.659+0000 7f2460d16700 1 -- 172.21.15.16:0/506631154 >> 172.21.15.16:0/506631154 conn(0x7f245c077e10 msgr2=0x7f245c08b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:35.677 INFO:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2019-12-05T06:06:35.677 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data m=2 k=2 crush-failure-domain=osd 2019-12-05T06:06:35.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.945+0000 7f6d7197f700 1 Processor -- start 2019-12-05T06:06:35.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.945+0000 7f6d7197f700 1 -- start start 2019-12-05T06:06:35.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.947+0000 7f6d7197f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6d6c08c6f0 0x7f6d6c08cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:35.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.947+0000 7f6d7197f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c0907c0 0x7f6d6c08fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.947+0000 7f6d7197f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c060c00 con 0x7f6d6c062860 2019-12-05T06:06:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.947+0000 7f6d7197f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6c08e860 con 0x7f6d6c08c6f0 2019-12-05T06:06:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.947+0000 7f6d7197f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c08de30 con 0x7f6d6c0907c0 2019-12-05T06:06:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.947+0000 7f6d6affd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6d6c062860 0x7f6d6c08f7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59326/0 (socket says 172.21.15.16:59326) 2019-12-05T06:06:35.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.947+0000 7f6d6affd700 1 -- 172.21.15.16:0/24622059 learned_addr learned my addr 172.21.15.16:0/24622059 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:35.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.947+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/24622059 >> v2:172.21.15.76:3300/0 conn(0x7f6d6c08c6f0 0x7f6d6c08cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:35.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.947+0000 7f6d6bfff700 1 --2- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c0907c0 0x7f6d6c08fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.948+0000 7f6d6a7fc700 1 -- 172.21.15.16:0/24622059 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2556340406 0 0) 0x7f6d54001070 con 0x7f6d6c062860 2019-12-05T06:06:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.948+0000 7f6d6a7fc700 1 -- 172.21.15.16:0/24622059 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d58002e80 con 0x7f6d6c062860 2019-12-05T06:06:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.948+0000 7f6d6b7fe700 1 -- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c0907c0 msgr2=0x7f6d6c08fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:35.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.948+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c0907c0 0x7f6d6c08fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.948+0000 7f6d6b7fe700 1 -- 172.21.15.16:0/24622059 >> v1:172.21.15.16:6789/0 conn(0x7f6d6c062860 legacy=0x7f6d6c08f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.948+0000 7f6d6b7fe700 1 -- 172.21.15.16:0/24622059 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c08eef0 con 0x7f6d6c08c6f0 2019-12-05T06:06:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.948+0000 7f6d6bfff700 1 --2- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c0907c0 0x7f6d6c08fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:35.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.948+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/24622059 >> v2:172.21.15.76:3300/0 conn(0x7f6d6c08c6f0 0x7f6d6c08cae0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6d6400b2d0 tx=0x7f6d6400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.949+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/24622059 >> v2:172.21.15.76:3300/0 conn(0x7f6d6c08c6f0 0x7f6d6c08cae0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6d6400b2d0 tx=0x7f6d6400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.949+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d6c08c6f0 0x7f6d6c08cae0 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f6d6400b2d0 tx=0x7f6d6400bae0).ready entity=mon.1 client_cookie=233e344304bc191 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:35.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.949+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d6c08c6f0 0x7f6d6c08cae0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f6d6400b2d0 tx=0x7f6d6400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.949+0000 7f6d6a7fc700 1 -- 172.21.15.16:0/24622059 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d64007590 con 0x7f6d6c08c6f0 2019-12-05T06:06:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.949+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d6c08c6f0 0x7f6d6c08cae0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f6d6400b2d0 tx=0x7f6d6400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.949+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d6c08c6f0 0x7f6d6c08cae0 secure :-1 s=THROTTLE_DONE pgs=66 cs=0 l=1 rx=0x7f6d6400b2d0 tx=0x7f6d6400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.949+0000 7f6d6a7fc700 1 -- 172.21.15.16:0/24622059 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d64009d30 con 0x7f6d6c08c6f0 2019-12-05T06:06:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.949+0000 7f6d6a7fc700 1 -- 172.21.15.16:0/24622059 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d6400a400 con 0x7f6d6c08c6f0 2019-12-05T06:06:35.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.949+0000 7f6d7197f700 1 -- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d6c08c6f0 msgr2=0x7f6d6c08cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:35.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.949+0000 7f6d7197f700 1 --2- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d6c08c6f0 0x7f6d6c08cae0 secure :-1 s=READY pgs=66 cs=0 l=1 rx=0x7f6d6400b2d0 tx=0x7f6d6400bae0).stop 2019-12-05T06:06:35.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.951+0000 7f6d7197f700 1 -- 172.21.15.16:0/24622059 shutdown_connections 2019-12-05T06:06:35.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.951+0000 7f6d7197f700 1 --2- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c0907c0 0x7f6d6c08fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:35.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.951+0000 7f6d7197f700 1 --2- 172.21.15.16:0/24622059 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6d6c08c6f0 0x7f6d6c08cae0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:35.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.951+0000 7f6d7197f700 1 -- 172.21.15.16:0/24622059 shutdown_connections 2019-12-05T06:06:35.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.951+0000 7f6d7197f700 1 -- 172.21.15.16:0/24622059 wait complete. 2019-12-05T06:06:35.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.951+0000 7f6d7197f700 1 -- 172.21.15.16:0/24622059 >> 172.21.15.16:0/24622059 conn(0x7f6d6c05f5a0 msgr2=0x7f6d6c096080 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.956+0000 7f6d7197f700 1 Processor -- start 2019-12-05T06:06:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.957+0000 7f6d7197f700 1 -- start start 2019-12-05T06:06:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.959+0000 7f6d7197f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.959+0000 7f6d7197f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6d6c082790 0x7f6d6c081d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.959+0000 7f6d7197f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c072e20 con 0x7f6d6c082240 2019-12-05T06:06:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.959+0000 7f6d7197f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6c0740b0 con 0x7f6d6c082790 2019-12-05T06:06:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.959+0000 7f6d7197f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c074380 con 0x7f6d6c14c8a0 2019-12-05T06:06:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.959+0000 7f6d6b7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.959+0000 7f6d6affd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6d6c082790 0x7f6d6c081d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.959+0000 7f6d6bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6d6c082240 0x7f6d6c070cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52402/0 (socket says 172.21.15.16:52402) 2019-12-05T06:06:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.959+0000 7f6d6bfff700 1 -- 172.21.15.16:0/2674316378 learned_addr learned my addr 172.21.15.16:0/2674316378 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:35.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.960+0000 7f6d6b7fe700 1 -- 172.21.15.16:0/2674316378 >> v2:172.21.15.76:3300/0 conn(0x7f6d6c082790 msgr2=0x7f6d6c081d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:35.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.960+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> v2:172.21.15.76:3300/0 conn(0x7f6d6c082790 0x7f6d6c081d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:35.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.960+0000 7f6d6b7fe700 1 -- 172.21.15.16:0/2674316378 >> v1:172.21.15.76:6789/0 conn(0x7f6d6c082240 legacy=0x7f6d6c070cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:35.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.960+0000 7f6d6b7fe700 1 -- 172.21.15.16:0/2674316378 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c080ec0 con 0x7f6d6c14c8a0 2019-12-05T06:06:35.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.960+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.960+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.960+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:35.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.961+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.961+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.961+0000 7f6d63fff700 1 -- 172.21.15.16:0/2674316378 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d64000cc0 con 0x7f6d6c14c8a0 2019-12-05T06:06:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.961+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.961+0000 7f6d63fff700 1 -- 172.21.15.16:0/2674316378 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d64009b90 con 0x7f6d6c14c8a0 2019-12-05T06:06:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.961+0000 7f6d7197f700 1 -- 172.21.15.16:0/2674316378 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6c080ec0 con 0x7f6d6c14c8a0 2019-12-05T06:06:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.961+0000 7f6d63fff700 1 -- 172.21.15.16:0/2674316378 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d6400a220 con 0x7f6d6c14c8a0 2019-12-05T06:06:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.961+0000 7f6d7197f700 1 -- 172.21.15.16:0/2674316378 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6c080ec0 con 0x7f6d6c14c8a0 2019-12-05T06:06:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.962+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.963+0000 7f6d63fff700 1 -- 172.21.15.16:0/2674316378 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6d64009d80 con 0x7f6d6c14c8a0 2019-12-05T06:06:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.963+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.963+0000 7f6d63fff700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d50023630 0x7f6d50025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.964+0000 7f6d63fff700 1 -- 172.21.15.16:0/2674316378 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5207+0+0 (secure 0 0 0) 0x7f6d64046040 con 0x7f6d6c14c8a0 2019-12-05T06:06:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.964+0000 7f6d6affd700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d50023630 0x7f6d50025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.964+0000 7f6d6affd700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d50023630 0x7f6d50025ac0 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.977+0000 7f6d7197f700 1 -- 172.21.15.16:0/2674316378 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6c080ec0 con 0x7f6d6c14c8a0 2019-12-05T06:06:35.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.987+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:35.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:35.987+0000 7f6d63fff700 1 -- 172.21.15.16:0/2674316378 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6d6400b750 con 0x7f6d6c14c8a0 2019-12-05T06:06:36.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.249+0000 7f6d7197f700 1 -- 172.21.15.16:0/2674316378 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"profile": ["m=2", "k=2", "crush-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "cephfs_data"} v 0) v1 -- 0x7f6d6c080ec0 con 0x7f6d6c14c8a0 2019-12-05T06:06:36.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.620+0000 7f6d6b7fe700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=THROTTLE_DONE pgs=20 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:36.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.620+0000 7f6d63fff700 1 -- 172.21.15.16:0/2674316378 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"profile": ["m=2", "k=2", "crush-failure-domain=osd"], "prefix": "osd erasure-code-profile set", "name": "cephfs_data"}]=0 v19) v1 ==== 154+0+0 (secure 0 0 0) 0x7f6d6402b170 con 0x7f6d6c14c8a0 2019-12-05T06:06:36.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.667+0000 7f6d7197f700 1 -- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d50023630 msgr2=0x7f6d50025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.667+0000 7f6d7197f700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d50023630 0x7f6d50025ac0 crc :-1 s=READY pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.668+0000 7f6d7197f700 1 -- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 msgr2=0x7f6d6c0817c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.668+0000 7f6d7197f700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 secure :-1 s=READY pgs=20 cs=0 l=1 rx=0x7f6d640106f0 tx=0x7f6d64010570).stop 2019-12-05T06:06:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.668+0000 7f6d7197f700 1 -- 172.21.15.16:0/2674316378 shutdown_connections 2019-12-05T06:06:36.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.668+0000 7f6d7197f700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d50023630 0x7f6d50025ac0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:36.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.668+0000 7f6d7197f700 1 --2- 172.21.15.16:0/2674316378 >> v2:172.21.15.76:3300/0 conn(0x7f6d6c082790 0x7f6d6c081d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:36.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.668+0000 7f6d7197f700 1 --2- 172.21.15.16:0/2674316378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d6c14c8a0 0x7f6d6c0817c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:36.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.668+0000 7f6d7197f700 1 -- 172.21.15.16:0/2674316378 shutdown_connections 2019-12-05T06:06:36.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.668+0000 7f6d7197f700 1 -- 172.21.15.16:0/2674316378 wait complete. 2019-12-05T06:06:36.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.668+0000 7f6d7197f700 1 -- 172.21.15.16:0/2674316378 >> 172.21.15.16:0/2674316378 conn(0x7f6d6c06c390 msgr2=0x7f6d6c06c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:36.687 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 erasure cephfs_data 2019-12-05T06:06:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.988+0000 7f2f3f6dc700 1 Processor -- start 2019-12-05T06:06:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.989+0000 7f2f3f6dc700 1 -- start start 2019-12-05T06:06:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.990+0000 7f2f3f6dc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2f38062860 0x7f2f3808f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.990+0000 7f2f3f6dc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f3808d2e0 0x7f2f3808fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:36.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.991+0000 7f2f3f6dc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f38060c00 con 0x7f2f3808c6f0 2019-12-05T06:06:36.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.991+0000 7f2f3f6dc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3808e860 con 0x7f2f3808d2e0 2019-12-05T06:06:36.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.991+0000 7f2f3f6dc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f3808de30 con 0x7f2f38062860 2019-12-05T06:06:36.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.991+0000 7f2f3e47a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f3808d2e0 0x7f2f3808fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:36.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.991+0000 7f2f3dc79700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2f3808c6f0 0x7f2f3808cae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59338/0 (socket says 172.21.15.16:59338) 2019-12-05T06:06:36.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.991+0000 7f2f3dc79700 1 -- 172.21.15.16:0/3336452730 learned_addr learned my addr 172.21.15.16:0/3336452730 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:36.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.991+0000 7f2f3d478700 1 --2- 172.21.15.16:0/3336452730 >> v2:172.21.15.76:3300/0 conn(0x7f2f38062860 0x7f2f3808f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:36.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.991+0000 7f2f3cc77700 1 -- 172.21.15.16:0/3336452730 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524382267 0 0) 0x7f2f34001150 con 0x7f2f3808c6f0 2019-12-05T06:06:36.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3e47a700 1 -- 172.21.15.16:0/3336452730 >> v2:172.21.15.76:3300/0 conn(0x7f2f38062860 msgr2=0x7f2f3808f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:36.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/3336452730 >> v2:172.21.15.76:3300/0 conn(0x7f2f38062860 0x7f2f3808f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:36.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3e47a700 1 -- 172.21.15.16:0/3336452730 >> v1:172.21.15.16:6789/0 conn(0x7f2f3808c6f0 legacy=0x7f2f3808cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:36.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3e47a700 1 -- 172.21.15.16:0/3336452730 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f3808eef0 con 0x7f2f3808d2e0 2019-12-05T06:06:36.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/3336452730 >> v2:172.21.15.16:3300/0 conn(0x7f2f3808d2e0 0x7f2f3808fd30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2f3000b2d0 tx=0x7f2f3000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:36.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/3336452730 >> v2:172.21.15.16:3300/0 conn(0x7f2f3808d2e0 0x7f2f3808fd30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2f3000b2d0 tx=0x7f2f3000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/3336452730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3808d2e0 0x7f2f3808fd30 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7f2f3000b2d0 tx=0x7f2f3000b5d0).ready entity=mon.0 client_cookie=cc60667de8537c8d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:37.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/3336452730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3808d2e0 0x7f2f3808fd30 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f2f3000b2d0 tx=0x7f2f3000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3cc77700 1 -- 172.21.15.16:0/3336452730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f30009f20 con 0x7f2f3808d2e0 2019-12-05T06:06:37.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/3336452730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3808d2e0 0x7f2f3808fd30 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f2f3000b2d0 tx=0x7f2f3000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.992+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/3336452730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3808d2e0 0x7f2f3808fd30 secure :-1 s=THROTTLE_DONE pgs=62 cs=0 l=1 rx=0x7f2f3000b2d0 tx=0x7f2f3000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.993+0000 7f2f3cc77700 1 -- 172.21.15.16:0/3336452730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f30009b40 con 0x7f2f3808d2e0 2019-12-05T06:06:37.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.993+0000 7f2f3cc77700 1 -- 172.21.15.16:0/3336452730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f30009d30 con 0x7f2f3808d2e0 2019-12-05T06:06:37.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.993+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/3336452730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3808d2e0 msgr2=0x7f2f3808fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:37.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.993+0000 7f2f3f6dc700 1 --2- 172.21.15.16:0/3336452730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3808d2e0 0x7f2f3808fd30 secure :-1 s=READY pgs=62 cs=0 l=1 rx=0x7f2f3000b2d0 tx=0x7f2f3000b5d0).stop 2019-12-05T06:06:37.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.994+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/3336452730 shutdown_connections 2019-12-05T06:06:37.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.994+0000 7f2f3f6dc700 1 --2- 172.21.15.16:0/3336452730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f3808d2e0 0x7f2f3808fd30 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:37.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.994+0000 7f2f3f6dc700 1 --2- 172.21.15.16:0/3336452730 >> v2:172.21.15.76:3300/0 conn(0x7f2f38062860 0x7f2f3808f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:37.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.995+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/3336452730 shutdown_connections 2019-12-05T06:06:37.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.995+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/3336452730 wait complete. 2019-12-05T06:06:37.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:36.995+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/3336452730 >> 172.21.15.16:0/3336452730 conn(0x7f2f3805f5a0 msgr2=0x7f2f38096080 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:37.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.000+0000 7f2f3f6dc700 1 Processor -- start 2019-12-05T06:06:37.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.000+0000 7f2f3f6dc700 1 -- start start 2019-12-05T06:06:37.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.002+0000 7f2f3f6dc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f38082790 0x7f2f38081fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:37.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.002+0000 7f2f3f6dc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2f38071cd0 0x7f2f38155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:37.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.002+0000 7f2f3f6dc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f38072e20 con 0x7f2f3814c8a0 2019-12-05T06:06:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.002+0000 7f2f3f6dc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f38074ae0 con 0x7f2f38082790 2019-12-05T06:06:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.002+0000 7f2f3f6dc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f380740b0 con 0x7f2f38071cd0 2019-12-05T06:06:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.002+0000 7f2f3dc79700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2f3814c8a0 0x7f2f380817c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59344/0 (socket says 172.21.15.16:59344) 2019-12-05T06:06:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.002+0000 7f2f3dc79700 1 -- 172.21.15.16:0/1612568996 learned_addr learned my addr 172.21.15.16:0/1612568996 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.002+0000 7f2f3d478700 1 --2- 172.21.15.16:0/1612568996 >> v2:172.21.15.16:3300/0 conn(0x7f2f38082790 0x7f2f38081fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.002+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> v2:172.21.15.76:3300/0 conn(0x7f2f38071cd0 0x7f2f38155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:37.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.003+0000 7f2f3e47a700 1 -- 172.21.15.16:0/1612568996 >> v2:172.21.15.16:3300/0 conn(0x7f2f38082790 msgr2=0x7f2f38081fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.003+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> v2:172.21.15.16:3300/0 conn(0x7f2f38082790 0x7f2f38081fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.003+0000 7f2f3e47a700 1 -- 172.21.15.16:0/1612568996 >> v1:172.21.15.16:6789/0 conn(0x7f2f3814c8a0 legacy=0x7f2f380817c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.003+0000 7f2f3e47a700 1 -- 172.21.15.16:0/1612568996 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f38080ec0 con 0x7f2f38071cd0 2019-12-05T06:06:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.003+0000 7f2f3d478700 1 --2- 172.21.15.16:0/1612568996 >> v2:172.21.15.16:3300/0 conn(0x7f2f38082790 0x7f2f38081fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:37.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.003+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> v2:172.21.15.76:3300/0 conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.003+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> v2:172.21.15.76:3300/0 conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.003+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:37.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.004+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.004+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.004+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.004+0000 7f2f2e7fc700 1 -- 172.21.15.16:0/1612568996 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f30009ef0 con 0x7f2f38071cd0 2019-12-05T06:06:37.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.004+0000 7f2f2e7fc700 1 -- 172.21.15.16:0/1612568996 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f30009b10 con 0x7f2f38071cd0 2019-12-05T06:06:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.004+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/1612568996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f38080ec0 con 0x7f2f38071cd0 2019-12-05T06:06:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.004+0000 7f2f2e7fc700 1 -- 172.21.15.16:0/1612568996 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f3000a4d0 con 0x7f2f38071cd0 2019-12-05T06:06:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.004+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/1612568996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f38080ec0 con 0x7f2f38071cd0 2019-12-05T06:06:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.005+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.006+0000 7f2f2e7fc700 1 -- 172.21.15.16:0/1612568996 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2f30009d00 con 0x7f2f38071cd0 2019-12-05T06:06:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.006+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.006+0000 7f2f2e7fc700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f20023630 0x7f2f20025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:37.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.006+0000 7f2f2e7fc700 1 -- 172.21.15.16:0/1612568996 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5463+0+0 (secure 0 0 0) 0x7f2f3008a0f0 con 0x7f2f38071cd0 2019-12-05T06:06:37.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.007+0000 7f2f3dc79700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f20023630 0x7f2f20025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:37.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.007+0000 7f2f3dc79700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f20023630 0x7f2f20025ac0 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:37.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.020+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/1612568996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f38080ec0 con 0x7f2f38071cd0 2019-12-05T06:06:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.029+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:37.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.030+0000 7f2f2e7fc700 1 -- 172.21.15.16:0/1612568996 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2f30009900 con 0x7f2f38071cd0 2019-12-05T06:06:37.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:37.287+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/1612568996 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 8, "erasure_code_profile": "cephfs_data", "pool": "cephfs_data"} v 0) v1 -- 0x7f2f38080ec0 con 0x7f2f38071cd0 2019-12-05T06:06:38.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.635+0000 7f2f3e47a700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.635+0000 7f2f2e7fc700 1 -- 172.21.15.16:0/1612568996 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"pool_type": "erasure", "prefix": "osd pool create", "pg_num": 8, "erasure_code_profile": "cephfs_data", "pool": "cephfs_data"}]=0 pool 'cephfs_data' created v21) v1 ==== 188+0+0 (secure 0 0 0) 0x7f2f30009e30 con 0x7f2f38071cd0 2019-12-05T06:06:38.652 INFO:teuthology.orchestra.run.smithi016.stderr:pool 'cephfs_data' created 2019-12-05T06:06:38.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f20023630 msgr2=0x7f2f20025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:38.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f20023630 0x7f2f20025ac0 crc :-1 s=READY pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:38.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 msgr2=0x7f2f38155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:38.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 0x7f2f38155f90 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f2f30054700 tx=0x7f2f30054aa0).stop 2019-12-05T06:06:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/1612568996 shutdown_connections 2019-12-05T06:06:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f38071cd0 0x7f2f38155f90 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 --2- 172.21.15.16:0/1612568996 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f20023630 0x7f2f20025ac0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 --2- 172.21.15.16:0/1612568996 >> v2:172.21.15.16:3300/0 conn(0x7f2f38082790 0x7f2f38081fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/1612568996 shutdown_connections 2019-12-05T06:06:38.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/1612568996 wait complete. 2019-12-05T06:06:38.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.659+0000 7f2f3f6dc700 1 -- 172.21.15.16:0/1612568996 >> 172.21.15.16:0/1612568996 conn(0x7f2f3806c390 msgr2=0x7f2f3806c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:38.674 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data allow_ec_overwrites true 2019-12-05T06:06:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.901+0000 7f0465b49700 1 Processor -- start 2019-12-05T06:06:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.901+0000 7f0465b49700 1 -- start start 2019-12-05T06:06:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.903+0000 7f0465b49700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f046008c6f0 0x7f046008cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:38.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.903+0000 7f0465b49700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f04600907c0 0x7f046008fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.903+0000 7f0465b49700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0460060c00 con 0x7f0460062860 2019-12-05T06:06:38.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.903+0000 7f0465b49700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046008e860 con 0x7f046008c6f0 2019-12-05T06:06:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.903+0000 7f0465b49700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f046008de30 con 0x7f04600907c0 2019-12-05T06:06:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.903+0000 7f045ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f046008c6f0 0x7f046008cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.903+0000 7f045ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f046008c6f0 0x7f046008cae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45958/0 (socket says 172.21.15.16:45958) 2019-12-05T06:06:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.903+0000 7f045ffff700 1 -- 172.21.15.16:0/420685239 learned_addr learned my addr 172.21.15.16:0/420685239 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.903+0000 7f04648e7700 1 --2- 172.21.15.16:0/420685239 >> v2:172.21.15.76:3300/0 conn(0x7f04600907c0 0x7f046008fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.904+0000 7f045ffff700 1 -- 172.21.15.16:0/420685239 >> v2:172.21.15.76:3300/0 conn(0x7f04600907c0 msgr2=0x7f046008fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:38.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.904+0000 7f045ffff700 1 --2- 172.21.15.16:0/420685239 >> v2:172.21.15.76:3300/0 conn(0x7f04600907c0 0x7f046008fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.904+0000 7f045ffff700 1 -- 172.21.15.16:0/420685239 >> v1:172.21.15.16:6789/0 conn(0x7f0460062860 legacy=0x7f046008f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.904+0000 7f045effd700 1 -- 172.21.15.16:0/420685239 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81062470 0 0) 0x7f0448001100 con 0x7f0460062860 2019-12-05T06:06:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.904+0000 7f045ffff700 1 -- 172.21.15.16:0/420685239 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046008eef0 con 0x7f046008c6f0 2019-12-05T06:06:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.904+0000 7f045ffff700 1 --2- 172.21.15.16:0/420685239 >> v2:172.21.15.16:3300/0 conn(0x7f046008c6f0 0x7f046008cae0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f045000fa40 tx=0x7f045000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.904+0000 7f045ffff700 1 --2- 172.21.15.16:0/420685239 >> v2:172.21.15.16:3300/0 conn(0x7f046008c6f0 0x7f046008cae0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f045000fa40 tx=0x7f045000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.904+0000 7f045ffff700 1 --2- 172.21.15.16:0/420685239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f046008c6f0 0x7f046008cae0 secure :-1 s=READY pgs=65 cs=0 l=1 rx=0x7f045000fa40 tx=0x7f045000fde0).ready entity=mon.0 client_cookie=790878f7525b6270 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:38.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.905+0000 7f045ffff700 1 --2- 172.21.15.16:0/420685239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f046008c6f0 0x7f046008cae0 secure :-1 s=THROTTLE_DONE pgs=65 cs=0 l=1 rx=0x7f045000fa40 tx=0x7f045000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.905+0000 7f045ffff700 1 --2- 172.21.15.16:0/420685239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f046008c6f0 0x7f046008cae0 secure :-1 s=THROTTLE_DONE pgs=65 cs=0 l=1 rx=0x7f045000fa40 tx=0x7f045000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.905+0000 7f045effd700 1 -- 172.21.15.16:0/420685239 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0450000ce0 con 0x7f046008c6f0 2019-12-05T06:06:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.905+0000 7f045ffff700 1 --2- 172.21.15.16:0/420685239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f046008c6f0 0x7f046008cae0 secure :-1 s=THROTTLE_DONE pgs=65 cs=0 l=1 rx=0x7f045000fa40 tx=0x7f045000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.905+0000 7f045effd700 1 -- 172.21.15.16:0/420685239 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0450001010 con 0x7f046008c6f0 2019-12-05T06:06:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.905+0000 7f045effd700 1 -- 172.21.15.16:0/420685239 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0450001720 con 0x7f046008c6f0 2019-12-05T06:06:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.905+0000 7f0465b49700 1 -- 172.21.15.16:0/420685239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f046008c6f0 msgr2=0x7f046008cae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:38.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.905+0000 7f0465b49700 1 --2- 172.21.15.16:0/420685239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f046008c6f0 0x7f046008cae0 secure :-1 s=READY pgs=65 cs=0 l=1 rx=0x7f045000fa40 tx=0x7f045000fde0).stop 2019-12-05T06:06:38.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.906+0000 7f0465b49700 1 -- 172.21.15.16:0/420685239 shutdown_connections 2019-12-05T06:06:38.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.906+0000 7f0465b49700 1 --2- 172.21.15.16:0/420685239 >> v2:172.21.15.76:3300/0 conn(0x7f04600907c0 0x7f046008fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:38.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.906+0000 7f0465b49700 1 --2- 172.21.15.16:0/420685239 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f046008c6f0 0x7f046008cae0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:38.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.906+0000 7f0465b49700 1 -- 172.21.15.16:0/420685239 shutdown_connections 2019-12-05T06:06:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.906+0000 7f0465b49700 1 -- 172.21.15.16:0/420685239 wait complete. 2019-12-05T06:06:38.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.906+0000 7f0465b49700 1 -- 172.21.15.16:0/420685239 >> 172.21.15.16:0/420685239 conn(0x7f046005f5a0 msgr2=0x7f0460096080 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:38.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.911+0000 7f0465b49700 1 Processor -- start 2019-12-05T06:06:38.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.911+0000 7f0465b49700 1 -- start start 2019-12-05T06:06:38.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.913+0000 7f0465b49700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:38.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.913+0000 7f0465b49700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04600741e0 con 0x7f04600a0b50 2019-12-05T06:06:38.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.913+0000 7f0465b49700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04600a13f0 con 0x7f0460081470 2019-12-05T06:06:38.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.913+0000 7f0465b49700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04600a1ee0 con 0x7f046009ff60 2019-12-05T06:06:38.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.913+0000 7f04648e7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04600a0b50 0x7f0460159990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59368/0 (socket says 172.21.15.16:59368) 2019-12-05T06:06:38.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.913+0000 7f04648e7700 1 -- 172.21.15.16:0/2198288011 learned_addr learned my addr 172.21.15.16:0/2198288011 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:38.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.913+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.913+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15006464 0 0) 0x7f04580095b0 con 0x7f04600a0b50 2019-12-05T06:06:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.913+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0444002e80 con 0x7f04600a0b50 2019-12-05T06:06:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.914+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819803086 0 0) 0x7f0448000f40 con 0x7f0460081470 2019-12-05T06:06:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.914+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04440030b0 con 0x7f0460081470 2019-12-05T06:06:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.914+0000 7f045ffff700 1 -- 172.21.15.16:0/2198288011 >> v1:172.21.15.76:6789/0 conn(0x7f0460081470 legacy=0x7f04600a0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.914+0000 7f045ffff700 1 -- 172.21.15.16:0/2198288011 >> v1:172.21.15.16:6789/0 conn(0x7f04600a0b50 legacy=0x7f0460159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:38.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.914+0000 7f045ffff700 1 -- 172.21.15.16:0/2198288011 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f046012a3f0 con 0x7f046009ff60 2019-12-05T06:06:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.914+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.914+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.914+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.914+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.915+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0450007c60 con 0x7f046009ff60 2019-12-05T06:06:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.915+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.915+0000 7f0465b49700 1 -- 172.21.15.16:0/2198288011 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f046012a3f0 con 0x7f046009ff60 2019-12-05T06:06:38.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.915+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0450007c60 con 0x7f046009ff60 2019-12-05T06:06:38.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.915+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.915+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04500015c0 con 0x7f046009ff60 2019-12-05T06:06:38.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.915+0000 7f0465b49700 1 -- 172.21.15.16:0/2198288011 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f046012a3f0 con 0x7f046009ff60 2019-12-05T06:06:38.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.916+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.917+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f04500011e0 con 0x7f046009ff60 2019-12-05T06:06:38.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.917+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.917+0000 7f045cff9700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0444023650 0x7f0444025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:38.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.917+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f0450046110 con 0x7f046009ff60 2019-12-05T06:06:38.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.918+0000 7f045f7fe700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0444023650 0x7f0444025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:38.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.918+0000 7f045f7fe700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0444023650 0x7f0444025ae0 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:38.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.933+0000 7f0465b49700 1 -- 172.21.15.16:0/2198288011 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f046012a3f0 con 0x7f046009ff60 2019-12-05T06:06:38.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.942+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:38.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:38.942+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0450000fd0 con 0x7f046009ff60 2019-12-05T06:06:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.183+0000 7f0465b49700 1 -- 172.21.15.16:0/2198288011 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"var": "allow_ec_overwrites", "prefix": "osd pool set", "pool": "cephfs_data", "val": "true"} v 0) v1 -- 0x7f046012a3f0 con 0x7f046009ff60 2019-12-05T06:06:39.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.638+0000 7f045ffff700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=THROTTLE_DONE pgs=24 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.638+0000 7f045cff9700 1 -- 172.21.15.16:0/2198288011 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"var": "allow_ec_overwrites", "prefix": "osd pool set", "pool": "cephfs_data", "val": "true"}]=0 set pool 3 allow_ec_overwrites to true v22) v1 ==== 166+0+0 (secure 0 0 0) 0x7f045002b130 con 0x7f046009ff60 2019-12-05T06:06:39.646 INFO:teuthology.orchestra.run.smithi016.stderr:set pool 3 allow_ec_overwrites to true 2019-12-05T06:06:39.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.653+0000 7f0465b49700 1 -- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0444023650 msgr2=0x7f0444025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.653+0000 7f0465b49700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0444023650 0x7f0444025ae0 crc :-1 s=READY pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.653+0000 7f0465b49700 1 -- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 msgr2=0x7f046012acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.653+0000 7f0465b49700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 secure :-1 s=READY pgs=24 cs=0 l=1 rx=0x7f0450010720 tx=0x7f0450010ac0).stop 2019-12-05T06:06:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.653+0000 7f0465b49700 1 -- 172.21.15.16:0/2198288011 shutdown_connections 2019-12-05T06:06:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.653+0000 7f0465b49700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0444023650 0x7f0444025ae0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:39.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.653+0000 7f0465b49700 1 --2- 172.21.15.16:0/2198288011 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f046009ff60 0x7f046012acf0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.654+0000 7f0465b49700 1 -- 172.21.15.16:0/2198288011 shutdown_connections 2019-12-05T06:06:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.654+0000 7f0465b49700 1 -- 172.21.15.16:0/2198288011 wait complete. 2019-12-05T06:06:39.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.654+0000 7f0465b49700 1 -- 172.21.15.16:0/2198288011 >> 172.21.15.16:0/2198288011 conn(0x7f046006f5a0 msgr2=0x7f046012e640 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:39.668 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2019-12-05T06:06:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.952+0000 7f41eb615700 1 Processor -- start 2019-12-05T06:06:39.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.952+0000 7f41eb615700 1 -- start start 2019-12-05T06:06:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.954+0000 7f41eb615700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f41e4086980 0x7f41e4086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.954+0000 7f41eb615700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41e4064310 0x7f41e4064760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.954+0000 7f41eb615700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e4061a10 con 0x7f41e408a6b0 2019-12-05T06:06:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.954+0000 7f41eb615700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e408b340 con 0x7f41e4086980 2019-12-05T06:06:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.954+0000 7f41eb615700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41e4060bc0 con 0x7f41e4064310 2019-12-05T06:06:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.954+0000 7f41e9bb2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f41e4086980 0x7f41e4086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:39.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.955+0000 7f41e93b1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41e408a6b0 0x7f41e408ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59386/0 (socket says 172.21.15.16:59386) 2019-12-05T06:06:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.955+0000 7f41e93b1700 1 -- 172.21.15.16:0/2809312439 learned_addr learned my addr 172.21.15.16:0/2809312439 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.955+0000 7f41ea3b3700 1 --2- 172.21.15.16:0/2809312439 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41e4064310 0x7f41e4064760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.955+0000 7f41e9bb2700 1 -- 172.21.15.16:0/2809312439 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41e4064310 msgr2=0x7f41e4064760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.955+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/2809312439 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41e4064310 0x7f41e4064760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.955+0000 7f41e9bb2700 1 -- 172.21.15.16:0/2809312439 >> v1:172.21.15.16:6789/0 conn(0x7f41e408a6b0 legacy=0x7f41e408ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:39.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.955+0000 7f41e9bb2700 1 -- 172.21.15.16:0/2809312439 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e4063a10 con 0x7f41e4086980 2019-12-05T06:06:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.955+0000 7f41e8bb0700 1 -- 172.21.15.16:0/2809312439 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029314175 0 0) 0x7f41d8001150 con 0x7f41e408a6b0 2019-12-05T06:06:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/2809312439 >> v2:172.21.15.16:3300/0 conn(0x7f41e4086980 0x7f41e4086d70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f41d400af50 tx=0x7f41d400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/2809312439 >> v2:172.21.15.16:3300/0 conn(0x7f41e4086980 0x7f41e4086d70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f41d400af50 tx=0x7f41d400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/2809312439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4086980 0x7f41e4086d70 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f41d400af50 tx=0x7f41d400b6e0).ready entity=mon.0 client_cookie=da237b0236374678 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/2809312439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4086980 0x7f41e4086d70 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f41d400af50 tx=0x7f41d400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41e8bb0700 1 -- 172.21.15.16:0/2809312439 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41d4007210 con 0x7f41e4086980 2019-12-05T06:06:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/2809312439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4086980 0x7f41e4086d70 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f41d400af50 tx=0x7f41d400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/2809312439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4086980 0x7f41e4086d70 secure :-1 s=THROTTLE_DONE pgs=68 cs=0 l=1 rx=0x7f41d400af50 tx=0x7f41d400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41e8bb0700 1 -- 172.21.15.16:0/2809312439 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d40099b0 con 0x7f41e4086980 2019-12-05T06:06:39.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41e8bb0700 1 -- 172.21.15.16:0/2809312439 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41d400a080 con 0x7f41e4086980 2019-12-05T06:06:39.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41eb615700 1 -- 172.21.15.16:0/2809312439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4086980 msgr2=0x7f41e4086d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:39.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.956+0000 7f41eb615700 1 --2- 172.21.15.16:0/2809312439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4086980 0x7f41e4086d70 secure :-1 s=READY pgs=68 cs=0 l=1 rx=0x7f41d400af50 tx=0x7f41d400b6e0).stop 2019-12-05T06:06:39.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.958+0000 7f41eb615700 1 -- 172.21.15.16:0/2809312439 shutdown_connections 2019-12-05T06:06:39.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.958+0000 7f41eb615700 1 --2- 172.21.15.16:0/2809312439 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41e4064310 0x7f41e4064760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:39.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.958+0000 7f41eb615700 1 --2- 172.21.15.16:0/2809312439 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4086980 0x7f41e4086d70 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:39.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.959+0000 7f41eb615700 1 -- 172.21.15.16:0/2809312439 shutdown_connections 2019-12-05T06:06:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.959+0000 7f41eb615700 1 -- 172.21.15.16:0/2809312439 wait complete. 2019-12-05T06:06:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.959+0000 7f41eb615700 1 -- 172.21.15.16:0/2809312439 >> 172.21.15.16:0/2809312439 conn(0x7f41e405f5a0 msgr2=0x7f41e409cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.964+0000 7f41eb615700 1 Processor -- start 2019-12-05T06:06:39.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.964+0000 7f41eb615700 1 -- start start 2019-12-05T06:06:39.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.966+0000 7f41eb615700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f41e4073d80 0x7f41e4095460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:39.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.966+0000 7f41eb615700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f41e4099990 0x7f41e40959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:39.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.966+0000 7f41eb615700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41e4074c00 con 0x7f41e4087680 2019-12-05T06:06:39.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.966+0000 7f41eb615700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e4075e90 con 0x7f41e4073d80 2019-12-05T06:06:39.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.966+0000 7f41eb615700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41e4076160 con 0x7f41e4099990 2019-12-05T06:06:39.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.966+0000 7f41e9bb2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f41e4073d80 0x7f41e4095460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:39.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.966+0000 7f41e9bb2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f41e4073d80 0x7f41e4095460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:45990/0 (socket says 172.21.15.16:45990) 2019-12-05T06:06:39.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.966+0000 7f41e9bb2700 1 -- 172.21.15.16:0/3328268433 learned_addr learned my addr 172.21.15.16:0/3328268433 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:39.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.966+0000 7f41e93b1700 1 --2- 172.21.15.16:0/3328268433 >> v2:172.21.15.76:3300/0 conn(0x7f41e4099990 0x7f41e40959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e9bb2700 1 -- 172.21.15.16:0/3328268433 >> v2:172.21.15.76:3300/0 conn(0x7f41e4099990 msgr2=0x7f41e40959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> v2:172.21.15.76:3300/0 conn(0x7f41e4099990 0x7f41e40959a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e9bb2700 1 -- 172.21.15.16:0/3328268433 >> v1:172.21.15.76:6789/0 conn(0x7f41e4087680 legacy=0x7f41e4087aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e9bb2700 1 -- 172.21.15.16:0/3328268433 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41e4094b60 con 0x7f41e4073d80 2019-12-05T06:06:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> v2:172.21.15.16:3300/0 conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> v2:172.21.15.16:3300/0 conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:39.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e27fc700 1 -- 172.21.15.16:0/3328268433 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41d400a9a0 con 0x7f41e4073d80 2019-12-05T06:06:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41e27fc700 1 -- 172.21.15.16:0/3328268433 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41d4009980 con 0x7f41e4073d80 2019-12-05T06:06:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.967+0000 7f41eb615700 1 -- 172.21.15.16:0/3328268433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41e4094b60 con 0x7f41e4073d80 2019-12-05T06:06:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.968+0000 7f41e27fc700 1 -- 172.21.15.16:0/3328268433 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41d400a010 con 0x7f41e4073d80 2019-12-05T06:06:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.968+0000 7f41eb615700 1 -- 172.21.15.16:0/3328268433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41e4094b60 con 0x7f41e4073d80 2019-12-05T06:06:39.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.969+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.970+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.970+0000 7f41e27fc700 1 -- 172.21.15.16:0/3328268433 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f41d4009580 con 0x7f41e4073d80 2019-12-05T06:06:39.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.970+0000 7f41e27fc700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f41c4023630 0x7f41c4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:39.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.970+0000 7f41e27fc700 1 -- 172.21.15.16:0/3328268433 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 5915+0+0 (secure 0 0 0) 0x7f41d4045e30 con 0x7f41e4073d80 2019-12-05T06:06:39.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.970+0000 7f41e93b1700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f41c4023630 0x7f41c4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:39.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.971+0000 7f41e93b1700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f41c4023630 0x7f41c4025ac0 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:39.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.984+0000 7f41eb615700 1 -- 172.21.15.16:0/3328268433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41e4094b60 con 0x7f41e4073d80 2019-12-05T06:06:39.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.994+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:39.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:39.994+0000 7f41e27fc700 1 -- 172.21.15.16:0/3328268433 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f41d400a9a0 con 0x7f41e4073d80 2019-12-05T06:06:40.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.255+0000 7f41eb615700 1 -- 172.21.15.16:0/3328268433 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"} v 0) v1 -- 0x7f41e4094b60 con 0x7f41e4073d80 2019-12-05T06:06:40.254 INFO:tasks.ceph.mon.a.smithi016.stderr:2019-12-05T06:06:40.256+0000 7f8ef39f4700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2019-12-05T06:06:40.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.262+0000 7f41e9bb2700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.262+0000 7f41e27fc700 1 -- 172.21.15.16:0/3328268433 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "data": "cephfs_data", "fs_name": "cephfs", "metadata": "cephfs_metadata"}]=0 new fs with metadata pool 2 and data pool 3 v2) v1 ==== 172+0+0 (secure 0 0 0) 0x7f41d4009580 con 0x7f41e4073d80 2019-12-05T06:06:40.268 INFO:teuthology.orchestra.run.smithi016.stderr:new fs with metadata pool 2 and data pool 3 2019-12-05T06:06:40.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 -- 172.21.15.16:0/3328268433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f41c4023630 msgr2=0x7f41c4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:40.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f41c4023630 0x7f41c4025ac0 crc :-1 s=READY pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:40.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 -- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 msgr2=0x7f41e4095460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:40.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 0x7f41e4095460 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7f41d4010440 tx=0x7f41d40102f0).stop 2019-12-05T06:06:40.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 -- 172.21.15.16:0/3328268433 shutdown_connections 2019-12-05T06:06:40.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 --2- 172.21.15.16:0/3328268433 >> v2:172.21.15.76:3300/0 conn(0x7f41e4099990 0x7f41e40959a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:40.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f41c4023630 0x7f41c4025ac0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:40.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 --2- 172.21.15.16:0/3328268433 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41e4073d80 0x7f41e4095460 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:40.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 -- 172.21.15.16:0/3328268433 shutdown_connections 2019-12-05T06:06:40.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 -- 172.21.15.16:0/3328268433 wait complete. 2019-12-05T06:06:40.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.273+0000 7f41eb615700 1 -- 172.21.15.16:0/3328268433 >> 172.21.15.16:0/3328268433 conn(0x7f41e4077e10 msgr2=0x7f41e408b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:40.284 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:06:40.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.501+0000 7f1c9dc39700 1 Processor -- start 2019-12-05T06:06:40.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.501+0000 7f1c9dc39700 1 -- start start 2019-12-05T06:06:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.502+0000 7f1c9dc39700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c9808c6d0 0x7f1c9808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:40.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.502+0000 7f1c9dc39700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c9808fef0 0x7f1c9808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.502+0000 7f1c9dc39700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1c9808ef20 0x7f1c9808f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.502+0000 7f1c9dc39700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c980a3f70 con 0x7f1c9808fef0 2019-12-05T06:06:40.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.502+0000 7f1c9dc39700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c9808d830 con 0x7f1c9808ef20 2019-12-05T06:06:40.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.502+0000 7f1c9dc39700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c9808d540 con 0x7f1c9808c6d0 2019-12-05T06:06:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.502+0000 7f1c977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c9808fef0 0x7f1c9808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.503+0000 7f1c977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c9808fef0 0x7f1c9808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46010/0 (socket says 172.21.15.16:46010) 2019-12-05T06:06:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.503+0000 7f1c977fe700 1 -- 172.21.15.16:0/164804041 learned_addr learned my addr 172.21.15.16:0/164804041 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.503+0000 7f1c97fff700 1 --2- 172.21.15.16:0/164804041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c9808c6d0 0x7f1c9808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.503+0000 7f1c9c9d7700 1 --2- 172.21.15.16:0/164804041 >> v2:172.21.15.76:3300/0 conn(0x7f1c9808ef20 0x7f1c9808f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.503+0000 7f1c977fe700 1 -- 172.21.15.16:0/164804041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c9808c6d0 msgr2=0x7f1c9808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:40.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.503+0000 7f1c977fe700 1 --2- 172.21.15.16:0/164804041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c9808c6d0 0x7f1c9808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.503+0000 7f1c977fe700 1 -- 172.21.15.16:0/164804041 >> v2:172.21.15.76:3300/0 conn(0x7f1c9808ef20 msgr2=0x7f1c9808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.503+0000 7f1c977fe700 1 --2- 172.21.15.16:0/164804041 >> v2:172.21.15.76:3300/0 conn(0x7f1c9808ef20 0x7f1c9808f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.503+0000 7f1c977fe700 1 -- 172.21.15.16:0/164804041 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c9808e620 con 0x7f1c9808fef0 2019-12-05T06:06:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.503+0000 7f1c977fe700 1 --2- 172.21.15.16:0/164804041 >> v2:172.21.15.16:3300/0 conn(0x7f1c9808fef0 0x7f1c9808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1c8000f6c0 tx=0x7f1c8000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.504+0000 7f1c977fe700 1 --2- 172.21.15.16:0/164804041 >> v2:172.21.15.16:3300/0 conn(0x7f1c9808fef0 0x7f1c9808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1c8000f6c0 tx=0x7f1c8000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.504+0000 7f1c977fe700 1 --2- 172.21.15.16:0/164804041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9808fef0 0x7f1c9808d000 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7f1c8000f6c0 tx=0x7f1c8000f570).ready entity=mon.0 client_cookie=f8967c9fdd49b760 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:40.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.504+0000 7f1c977fe700 1 --2- 172.21.15.16:0/164804041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9808fef0 0x7f1c9808d000 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f1c8000f6c0 tx=0x7f1c8000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.504+0000 7f1c977fe700 1 --2- 172.21.15.16:0/164804041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9808fef0 0x7f1c9808d000 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f1c8000f6c0 tx=0x7f1c8000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.504+0000 7f1c96ffd700 1 -- 172.21.15.16:0/164804041 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c80000960 con 0x7f1c9808fef0 2019-12-05T06:06:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.504+0000 7f1c977fe700 1 --2- 172.21.15.16:0/164804041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9808fef0 0x7f1c9808d000 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7f1c8000f6c0 tx=0x7f1c8000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.504+0000 7f1c96ffd700 1 -- 172.21.15.16:0/164804041 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c80000c90 con 0x7f1c9808fef0 2019-12-05T06:06:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.504+0000 7f1c96ffd700 1 -- 172.21.15.16:0/164804041 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c800013a0 con 0x7f1c9808fef0 2019-12-05T06:06:40.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.504+0000 7f1c9dc39700 1 -- 172.21.15.16:0/164804041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9808fef0 msgr2=0x7f1c9808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.504+0000 7f1c9dc39700 1 --2- 172.21.15.16:0/164804041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9808fef0 0x7f1c9808d000 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7f1c8000f6c0 tx=0x7f1c8000f570).stop 2019-12-05T06:06:40.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.505+0000 7f1c9dc39700 1 -- 172.21.15.16:0/164804041 shutdown_connections 2019-12-05T06:06:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.505+0000 7f1c9dc39700 1 --2- 172.21.15.16:0/164804041 >> v2:172.21.15.76:3300/0 conn(0x7f1c9808ef20 0x7f1c9808f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.505+0000 7f1c9dc39700 1 --2- 172.21.15.16:0/164804041 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c9808fef0 0x7f1c9808d000 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.505+0000 7f1c9dc39700 1 --2- 172.21.15.16:0/164804041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c9808c6d0 0x7f1c9808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.505+0000 7f1c9dc39700 1 -- 172.21.15.16:0/164804041 shutdown_connections 2019-12-05T06:06:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.505+0000 7f1c9dc39700 1 -- 172.21.15.16:0/164804041 wait complete. 2019-12-05T06:06:40.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.505+0000 7f1c9dc39700 1 -- 172.21.15.16:0/164804041 >> 172.21.15.16:0/164804041 conn(0x7f1c98061940 msgr2=0x7f1c98096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:40.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.508+0000 7f1c9dc39700 1 Processor -- start 2019-12-05T06:06:40.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.509+0000 7f1c9dc39700 1 -- start start 2019-12-05T06:06:40.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.509+0000 7f1c9dc39700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c98096e90 0x7f1c9809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:40.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.509+0000 7f1c9dc39700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c98084b90 con 0x7f1c9809b920 2019-12-05T06:06:40.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.509+0000 7f1c9dc39700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c98098790 con 0x7f1c98084f30 2019-12-05T06:06:40.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.509+0000 7f1c9dc39700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c98099a20 con 0x7f1c98096e90 2019-12-05T06:06:40.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c9c9d7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1c9809b920 0x7f1c98097a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59416/0 (socket says 172.21.15.16:59416) 2019-12-05T06:06:40.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c9c9d7700 1 -- 172.21.15.16:0/3743586147 learned_addr learned my addr 172.21.15.16:0/3743586147 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:40.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c97fff700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c98096e90 0x7f1c9809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211334169 0 0) 0x7f1c88001150 con 0x7f1c9809b920 2019-12-05T06:06:40.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c7c002e80 con 0x7f1c9809b920 2019-12-05T06:06:40.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2975082242 0 0) 0x7f1c80007580 con 0x7f1c98084f30 2019-12-05T06:06:40.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c7c0030b0 con 0x7f1c98084f30 2019-12-05T06:06:40.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1338824582 0 0) 0x7f1c88001280 con 0x7f1c9809b920 2019-12-05T06:06:40.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c7c0033c0 con 0x7f1c9809b920 2019-12-05T06:06:40.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f1c88001850 con 0x7f1c9809b920 2019-12-05T06:06:40.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c98096e90 msgr2=0x7f1c9809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:40.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c98096e90 0x7f1c9809b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:40.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 >> v1:172.21.15.76:6789/0 conn(0x7f1c98084f30 legacy=0x7f1c98097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:40.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 >> v1:172.21.15.16:6789/0 conn(0x7f1c9809b920 legacy=0x7f1c98097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:40.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c7c005d40 0x7f1c7c006130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:40.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:40.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c7c0051e0 0x7f1c7c0032e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:40.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c003a90 con 0x7f1c7c0046f0 2019-12-05T06:06:40.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c003cc0 con 0x7f1c7c0051e0 2019-12-05T06:06:40.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.510+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c7c003ef0 con 0x7f1c7c005d40 2019-12-05T06:06:40.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c97fff700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c98096e90 0x7f1c9809b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:40.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c9c9d7700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c7c0051e0 0x7f1c7c0032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c97fff700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c7c005d40 0x7f1c7c006130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c977fe700 1 -- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c7c005d40 msgr2=0x7f1c7c006130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:40.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c7c005d40 0x7f1c7c006130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:40.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c977fe700 1 -- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c7c0051e0 msgr2=0x7f1c7c0032e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:40.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1c7c0051e0 0x7f1c7c0032e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:40.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c977fe700 1 -- 172.21.15.16:0/3743586147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c7c0041a0 con 0x7f1c7c0046f0 2019-12-05T06:06:40.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c9dc39700 1 -- 172.21.15.16:0/3743586147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c9809aae0 con 0x7f1c7c0046f0 2019-12-05T06:06:40.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c97fff700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c7c005d40 0x7f1c7c006130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:06:40.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c9dc39700 1 -- 172.21.15.16:0/3743586147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c98084b90 con 0x7f1c7c0046f0 2019-12-05T06:06:40.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.511+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.512+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=READY pgs=72 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:40.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.512+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.512+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c80007d50 con 0x7f1c7c0046f0 2019-12-05T06:06:40.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.512+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.512+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.512+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c800081d0 con 0x7f1c7c0046f0 2019-12-05T06:06:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.512+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c80007d50 con 0x7f1c7c0046f0 2019-12-05T06:06:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.512+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.514+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1c80005390 con 0x7f1c7c0046f0 2019-12-05T06:06:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.514+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.514+0000 7f1c94ff9700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c7c02dc80 0x7f1c7c0300b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:40.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.514+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f1c80089d10 con 0x7f1c7c0046f0 2019-12-05T06:06:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.514+0000 7f1c97fff700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c7c02dc80 0x7f1c7c0300b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.515+0000 7f1c97fff700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c7c02dc80 0x7f1c7c0300b0 crc :-1 s=READY pgs=52 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:40.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.517+0000 7f1c9dc39700 1 -- 172.21.15.16:0/3743586147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c70002e60 con 0x7f1c7c0046f0 2019-12-05T06:06:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.521+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.521+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1c80005180 con 0x7f1c7c0046f0 2019-12-05T06:06:40.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.729+0000 7f1c9dc39700 1 -- 172.21.15.16:0/3743586147 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c70001d10 con 0x7f1c7c0046f0 2019-12-05T06:06:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.731+0000 7f1c977fe700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.731+0000 7f1c94ff9700 1 -- 172.21.15.16:0/3743586147 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+14898 (secure 0 0 0) 0x7f1c8006ef90 con 0x7f1c7c0046f0 2019-12-05T06:06:40.731 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:40.731 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:40.256173+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":32773,"flags_names":"hashpspool,ec_overwrites,creating","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:06:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.735+0000 7f1c9dc39700 1 -- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c7c02dc80 msgr2=0x7f1c7c0300b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.735+0000 7f1c9dc39700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c7c02dc80 0x7f1c7c0300b0 crc :-1 s=READY pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.735+0000 7f1c9dc39700 1 -- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 msgr2=0x7f1c7c004b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.735+0000 7f1c9dc39700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 secure :-1 s=READY pgs=72 cs=0 l=1 rx=0x7f1c800543c0 tx=0x7f1c80054270).stop 2019-12-05T06:06:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.735+0000 7f1c9c9d7700 1 -- 172.21.15.16:0/3743586147 reap_dead start 2019-12-05T06:06:40.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.735+0000 7f1c9dc39700 1 -- 172.21.15.16:0/3743586147 shutdown_connections 2019-12-05T06:06:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.735+0000 7f1c9dc39700 1 --2- 172.21.15.16:0/3743586147 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c7c0046f0 0x7f1c7c004b90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.735+0000 7f1c9dc39700 1 -- 172.21.15.16:0/3743586147 shutdown_connections 2019-12-05T06:06:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.735+0000 7f1c9dc39700 1 -- 172.21.15.16:0/3743586147 wait complete. 2019-12-05T06:06:40.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.735+0000 7f1c9dc39700 1 -- 172.21.15.16:0/3743586147 >> 172.21.15.16:0/3743586147 conn(0x7f1c980714f0 msgr2=0x7f1c9809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:40.747 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:06:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.973+0000 7fbadafc1700 1 Processor -- start 2019-12-05T06:06:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.973+0000 7fbadafc1700 1 -- start start 2019-12-05T06:06:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbadafc1700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbad408c6d0 0x7fbad408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:40.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbadafc1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbad408fef0 0x7fbad408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:40.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbadafc1700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbad408ef20 0x7fbad408f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:40.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbadafc1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad40a3f70 con 0x7fbad408fef0 2019-12-05T06:06:40.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbadafc1700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad408d830 con 0x7fbad408ef20 2019-12-05T06:06:40.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbadafc1700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbad408d540 con 0x7fbad408c6d0 2019-12-05T06:06:40.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbad8d5d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbad408fef0 0x7fbad408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbad8d5d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbad408fef0 0x7fbad408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46030/0 (socket says 172.21.15.16:46030) 2019-12-05T06:06:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbad8d5d700 1 -- 172.21.15.16:0/3480570200 learned_addr learned my addr 172.21.15.16:0/3480570200 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbad955e700 1 --2- 172.21.15.16:0/3480570200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbad408c6d0 0x7fbad408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.975+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/3480570200 >> v2:172.21.15.76:3300/0 conn(0x7fbad408ef20 0x7fbad408f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.976+0000 7fbad8d5d700 1 -- 172.21.15.16:0/3480570200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbad408c6d0 msgr2=0x7fbad408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.976+0000 7fbad8d5d700 1 --2- 172.21.15.16:0/3480570200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbad408c6d0 0x7fbad408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.976+0000 7fbad8d5d700 1 -- 172.21.15.16:0/3480570200 >> v2:172.21.15.76:3300/0 conn(0x7fbad408ef20 msgr2=0x7fbad408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.976+0000 7fbad8d5d700 1 --2- 172.21.15.16:0/3480570200 >> v2:172.21.15.76:3300/0 conn(0x7fbad408ef20 0x7fbad408f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:40.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.976+0000 7fbad8d5d700 1 -- 172.21.15.16:0/3480570200 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad408e620 con 0x7fbad408fef0 2019-12-05T06:06:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.976+0000 7fbad8d5d700 1 --2- 172.21.15.16:0/3480570200 >> v2:172.21.15.16:3300/0 conn(0x7fbad408fef0 0x7fbad408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbac400f6c0 tx=0x7fbac400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:40.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.976+0000 7fbad8d5d700 1 --2- 172.21.15.16:0/3480570200 >> v2:172.21.15.16:3300/0 conn(0x7fbad408fef0 0x7fbad408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbac400f6c0 tx=0x7fbac400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.976+0000 7fbad8d5d700 1 --2- 172.21.15.16:0/3480570200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad408fef0 0x7fbad408d000 secure :-1 s=READY pgs=73 cs=0 l=1 rx=0x7fbac400f6c0 tx=0x7fbac400f570).ready entity=mon.0 client_cookie=dddeae04956e5085 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.977+0000 7fbad8d5d700 1 --2- 172.21.15.16:0/3480570200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad408fef0 0x7fbad408d000 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7fbac400f6c0 tx=0x7fbac400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.977+0000 7fbad8d5d700 1 --2- 172.21.15.16:0/3480570200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad408fef0 0x7fbad408d000 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7fbac400f6c0 tx=0x7fbac400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.977+0000 7fbad8d5d700 1 --2- 172.21.15.16:0/3480570200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad408fef0 0x7fbad408d000 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7fbac400f6c0 tx=0x7fbac400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.977+0000 7fbacbfff700 1 -- 172.21.15.16:0/3480570200 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbac4000960 con 0x7fbad408fef0 2019-12-05T06:06:41.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.977+0000 7fbacbfff700 1 -- 172.21.15.16:0/3480570200 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbac4000c90 con 0x7fbad408fef0 2019-12-05T06:06:41.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.977+0000 7fbacbfff700 1 -- 172.21.15.16:0/3480570200 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbac40013a0 con 0x7fbad408fef0 2019-12-05T06:06:41.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.977+0000 7fbadafc1700 1 -- 172.21.15.16:0/3480570200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad408fef0 msgr2=0x7fbad408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.977+0000 7fbadafc1700 1 --2- 172.21.15.16:0/3480570200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad408fef0 0x7fbad408d000 secure :-1 s=READY pgs=73 cs=0 l=1 rx=0x7fbac400f6c0 tx=0x7fbac400f570).stop 2019-12-05T06:06:41.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.978+0000 7fbadafc1700 1 -- 172.21.15.16:0/3480570200 shutdown_connections 2019-12-05T06:06:41.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.978+0000 7fbadafc1700 1 --2- 172.21.15.16:0/3480570200 >> v2:172.21.15.76:3300/0 conn(0x7fbad408ef20 0x7fbad408f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:41.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.978+0000 7fbadafc1700 1 --2- 172.21.15.16:0/3480570200 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad408fef0 0x7fbad408d000 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.978+0000 7fbadafc1700 1 --2- 172.21.15.16:0/3480570200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbad408c6d0 0x7fbad408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:41.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.979+0000 7fbadafc1700 1 -- 172.21.15.16:0/3480570200 shutdown_connections 2019-12-05T06:06:41.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.979+0000 7fbadafc1700 1 -- 172.21.15.16:0/3480570200 wait complete. 2019-12-05T06:06:41.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.979+0000 7fbadafc1700 1 -- 172.21.15.16:0/3480570200 >> 172.21.15.16:0/3480570200 conn(0x7fbad4061940 msgr2=0x7fbad4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:41.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.985+0000 7fbadafc1700 1 Processor -- start 2019-12-05T06:06:41.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.985+0000 7fbadafc1700 1 -- start start 2019-12-05T06:06:41.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.987+0000 7fbadafc1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbad4133220 0x7fbad409a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.987+0000 7fbadafc1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad406e7c0 con 0x7fbad409ae80 2019-12-05T06:06:41.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.987+0000 7fbadafc1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbad4133b30 con 0x7fbad4132630 2019-12-05T06:06:41.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.987+0000 7fbadafc1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbad4098f20 con 0x7fbad4133220 2019-12-05T06:06:41.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.987+0000 7fbad9d5f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbad4133220 0x7fbad409a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.987+0000 7fbad8d5d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbad409ae80 0x7fbad4099eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59436/0 (socket says 172.21.15.16:59436) 2019-12-05T06:06:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.987+0000 7fbad8d5d700 1 -- 172.21.15.16:0/1846404949 learned_addr learned my addr 172.21.15.16:0/1846404949 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:41.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbad9d5f700 1 -- 172.21.15.16:0/1846404949 >> v1:172.21.15.76:6789/0 conn(0x7fbad4132630 legacy=0x7fbad4132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbac9ffb700 1 -- 172.21.15.16:0/1846404949 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4240655151 0 0) 0x7fbad0000f30 con 0x7fbad4132630 2019-12-05T06:06:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbad9d5f700 1 -- 172.21.15.16:0/1846404949 >> v1:172.21.15.16:6789/0 conn(0x7fbad409ae80 legacy=0x7fbad4099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbad9d5f700 1 -- 172.21.15.16:0/1846404949 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbad40995b0 con 0x7fbad4133220 2019-12-05T06:06:41.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/1846404949 >> v2:172.21.15.16:3300/0 conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/1846404949 >> v2:172.21.15.16:3300/0 conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=READY pgs=75 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbac9ffb700 1 -- 172.21.15.16:0/1846404949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbacc00f200 con 0x7fbad4133220 2019-12-05T06:06:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbac9ffb700 1 -- 172.21.15.16:0/1846404949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbacc00f400 con 0x7fbad4133220 2019-12-05T06:06:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbac9ffb700 1 -- 172.21.15.16:0/1846404949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbacc00fad0 con 0x7fbad4133220 2019-12-05T06:06:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.988+0000 7fbadafc1700 1 -- 172.21.15.16:0/1846404949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbad4133680 con 0x7fbad4133220 2019-12-05T06:06:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.989+0000 7fbadafc1700 1 -- 172.21.15.16:0/1846404949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbad40995b0 con 0x7fbad4133220 2019-12-05T06:06:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.989+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.990+0000 7fbac9ffb700 1 -- 172.21.15.16:0/1846404949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbacc00ef60 con 0x7fbad4133220 2019-12-05T06:06:41.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.990+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.991+0000 7fbac9ffb700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbab4023630 0x7fbab4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.991+0000 7fbac9ffb700 1 -- 172.21.15.16:0/1846404949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fbacc046110 con 0x7fbad4133220 2019-12-05T06:06:41.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.991+0000 7fbad955e700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbab4023630 0x7fbab4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:40.992+0000 7fbad955e700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbab4023630 0x7fbab4025ac0 crc :-1 s=READY pgs=53 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.004+0000 7fbadafc1700 1 -- 172.21.15.16:0/1846404949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbad4133680 con 0x7fbad4133220 2019-12-05T06:06:41.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.012+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.012+0000 7fbac9ffb700 1 -- 172.21.15.16:0/1846404949 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbacc000e50 con 0x7fbad4133220 2019-12-05T06:06:41.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.267+0000 7fbadafc1700 1 -- 172.21.15.16:0/1846404949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbad40995b0 con 0x7fbad4133220 2019-12-05T06:06:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.269+0000 7fbad9d5f700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.269+0000 7fbac9ffb700 1 -- 172.21.15.16:0/1846404949 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7fbacc00eec0 con 0x7fbad4133220 2019-12-05T06:06:41.269 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:41.269 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:06:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.273+0000 7fbadafc1700 1 -- 172.21.15.16:0/1846404949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbab4023630 msgr2=0x7fbab4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.273+0000 7fbadafc1700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbab4023630 0x7fbab4025ac0 crc :-1 s=READY pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.273+0000 7fbadafc1700 1 -- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 msgr2=0x7fbad409a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.273+0000 7fbadafc1700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 0x7fbad409a6b0 secure :-1 s=READY pgs=75 cs=0 l=1 rx=0x7fbacc00edc0 tx=0x7fbacc010a70).stop 2019-12-05T06:06:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.273+0000 7fbadafc1700 1 -- 172.21.15.16:0/1846404949 shutdown_connections 2019-12-05T06:06:41.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.273+0000 7fbadafc1700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbab4023630 0x7fbab4025ac0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.273+0000 7fbadafc1700 1 --2- 172.21.15.16:0/1846404949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbad4133220 0x7fbad409a6b0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.274+0000 7fbadafc1700 1 -- 172.21.15.16:0/1846404949 shutdown_connections 2019-12-05T06:06:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.274+0000 7fbadafc1700 1 -- 172.21.15.16:0/1846404949 wait complete. 2019-12-05T06:06:41.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.274+0000 7fbadafc1700 1 -- 172.21.15.16:0/1846404949 >> 172.21.15.16:0/1846404949 conn(0x7fbad40714f0 msgr2=0x7fbad409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:41.294 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2019-12-05T06:06:41.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.540+0000 7f55ab942700 1 Processor -- start 2019-12-05T06:06:41.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.541+0000 7f55ab942700 1 -- start start 2019-12-05T06:06:41.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.542+0000 7f55ab942700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55a408aeb0 0x7f55a4064b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:41.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.542+0000 7f55ab942700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a4061a10 con 0x7f55a4086980 2019-12-05T06:06:41.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.542+0000 7f55ab942700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a4092a70 con 0x7f55a4086d70 2019-12-05T06:06:41.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.542+0000 7f55ab942700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a4060bc0 con 0x7f55a408aeb0 2019-12-05T06:06:41.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.543+0000 7f55aa6e0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55a408aeb0 0x7f55a4064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.543+0000 7f55aa6e0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55a408aeb0 0x7f55a4064b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46046/0 (socket says 172.21.15.16:46046) 2019-12-05T06:06:41.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.543+0000 7f55a9edf700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f55a4086980 0x7f55a408a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59444/0 (socket says 172.21.15.16:59444) 2019-12-05T06:06:41.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.543+0000 7f55aa6e0700 1 -- 172.21.15.16:0/3114560034 learned_addr learned my addr 172.21.15.16:0/3114560034 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:41.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.543+0000 7f55a8edd700 1 -- 172.21.15.16:0/3114560034 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081442194 0 0) 0x7f5598001080 con 0x7f55a4086d70 2019-12-05T06:06:41.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.543+0000 7f55aa6e0700 1 -- 172.21.15.16:0/3114560034 >> v1:172.21.15.76:6789/0 conn(0x7f55a4086d70 legacy=0x7f55a4064310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.543+0000 7f55aa6e0700 1 -- 172.21.15.16:0/3114560034 >> v1:172.21.15.16:6789/0 conn(0x7f55a4086980 legacy=0x7f55a408a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.543+0000 7f55aa6e0700 1 -- 172.21.15.16:0/3114560034 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a408b310 con 0x7f55a408aeb0 2019-12-05T06:06:41.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.544+0000 7f55aa6e0700 1 --2- 172.21.15.16:0/3114560034 >> v2:172.21.15.16:3300/0 conn(0x7f55a408aeb0 0x7f55a4064b10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f559400f730 tx=0x7f559400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.544+0000 7f55aa6e0700 1 --2- 172.21.15.16:0/3114560034 >> v2:172.21.15.16:3300/0 conn(0x7f55a408aeb0 0x7f55a4064b10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f559400f730 tx=0x7f559400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.544+0000 7f55aa6e0700 1 --2- 172.21.15.16:0/3114560034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a408aeb0 0x7f55a4064b10 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7f559400f730 tx=0x7f559400fad0).ready entity=mon.0 client_cookie=503bd317ce51a5d6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.544+0000 7f55aa6e0700 1 --2- 172.21.15.16:0/3114560034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a408aeb0 0x7f55a4064b10 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f559400f730 tx=0x7f559400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.544+0000 7f55a8edd700 1 -- 172.21.15.16:0/3114560034 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55940010e0 con 0x7f55a408aeb0 2019-12-05T06:06:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.544+0000 7f55aa6e0700 1 --2- 172.21.15.16:0/3114560034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a408aeb0 0x7f55a4064b10 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f559400f730 tx=0x7f559400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.544+0000 7f55aa6e0700 1 --2- 172.21.15.16:0/3114560034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a408aeb0 0x7f55a4064b10 secure :-1 s=THROTTLE_DONE pgs=79 cs=0 l=1 rx=0x7f559400f730 tx=0x7f559400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.544+0000 7f55a8edd700 1 -- 172.21.15.16:0/3114560034 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5594000a70 con 0x7f55a408aeb0 2019-12-05T06:06:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.544+0000 7f55a8edd700 1 -- 172.21.15.16:0/3114560034 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f55940016c0 con 0x7f55a408aeb0 2019-12-05T06:06:41.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.544+0000 7f55ab942700 1 -- 172.21.15.16:0/3114560034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a408aeb0 msgr2=0x7f55a4064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.545+0000 7f55ab942700 1 --2- 172.21.15.16:0/3114560034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a408aeb0 0x7f55a4064b10 secure :-1 s=READY pgs=79 cs=0 l=1 rx=0x7f559400f730 tx=0x7f559400fad0).stop 2019-12-05T06:06:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.546+0000 7f55ab942700 1 -- 172.21.15.16:0/3114560034 shutdown_connections 2019-12-05T06:06:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.546+0000 7f55ab942700 1 --2- 172.21.15.16:0/3114560034 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a408aeb0 0x7f55a4064b10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.547+0000 7f55ab942700 1 -- 172.21.15.16:0/3114560034 shutdown_connections 2019-12-05T06:06:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.547+0000 7f55ab942700 1 -- 172.21.15.16:0/3114560034 wait complete. 2019-12-05T06:06:41.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.547+0000 7f55ab942700 1 -- 172.21.15.16:0/3114560034 >> 172.21.15.16:0/3114560034 conn(0x7f55a405f5a0 msgr2=0x7f55a409cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:41.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.551+0000 7f55ab942700 1 Processor -- start 2019-12-05T06:06:41.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.551+0000 7f55ab942700 1 -- start start 2019-12-05T06:06:41.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.553+0000 7f55ab942700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55a4087680 0x7f55a412bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.553+0000 7f55ab942700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f55a412c830 0x7f55a412c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.553+0000 7f55ab942700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55a40994e0 con 0x7f55a4087a70 2019-12-05T06:06:41.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.553+0000 7f55ab942700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55a4077260 con 0x7f55a4087680 2019-12-05T06:06:41.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.553+0000 7f55ab942700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55a409b1a0 con 0x7f55a412c830 2019-12-05T06:06:41.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.553+0000 7f55a9edf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55a4087680 0x7f55a412bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.553+0000 7f55a9edf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55a4087680 0x7f55a412bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46050/0 (socket says 172.21.15.16:46050) 2019-12-05T06:06:41.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.553+0000 7f55a9edf700 1 -- 172.21.15.16:0/2070007502 learned_addr learned my addr 172.21.15.16:0/2070007502 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:41.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.553+0000 7f55a96de700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f55a412c830 0x7f55a412c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.554+0000 7f55a27fc700 1 -- 172.21.15.16:0/2070007502 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1267162500 0 0) 0x7f5594000940 con 0x7f55a4087a70 2019-12-05T06:06:41.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.554+0000 7f55a9edf700 1 -- 172.21.15.16:0/2070007502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f55a412c830 msgr2=0x7f55a412c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.554+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f55a412c830 0x7f55a412c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.554+0000 7f55a9edf700 1 -- 172.21.15.16:0/2070007502 >> v1:172.21.15.76:6789/0 conn(0x7f55a4087a70 legacy=0x7f55a4150450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.554+0000 7f55a9edf700 1 -- 172.21.15.16:0/2070007502 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55a409b830 con 0x7f55a4087680 2019-12-05T06:06:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.554+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> v2:172.21.15.16:3300/0 conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.554+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> v2:172.21.15.16:3300/0 conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.554+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.554+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.555+0000 7f55a27fc700 1 -- 172.21.15.16:0/2070007502 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f559c001010 con 0x7f55a4087680 2019-12-05T06:06:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.555+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.555+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.555+0000 7f55a27fc700 1 -- 172.21.15.16:0/2070007502 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f559c0045c0 con 0x7f55a4087680 2019-12-05T06:06:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.555+0000 7f55ab942700 1 -- 172.21.15.16:0/2070007502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f55a409b830 con 0x7f55a4087680 2019-12-05T06:06:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.555+0000 7f55a27fc700 1 -- 172.21.15.16:0/2070007502 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f559c004c10 con 0x7f55a4087680 2019-12-05T06:06:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.555+0000 7f55ab942700 1 -- 172.21.15.16:0/2070007502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f55a409b830 con 0x7f55a4087680 2019-12-05T06:06:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.555+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.556+0000 7f55a27fc700 1 -- 172.21.15.16:0/2070007502 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f559c0047b0 con 0x7f55a4087680 2019-12-05T06:06:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.556+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.557+0000 7f55a27fc700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5584023630 0x7f5584025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.557+0000 7f55a27fc700 1 -- 172.21.15.16:0/2070007502 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f559c0463f0 con 0x7f55a4087680 2019-12-05T06:06:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.557+0000 7f55a96de700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5584023630 0x7f5584025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:41.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.558+0000 7f55a96de700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5584023630 0x7f5584025ae0 crc :-1 s=READY pgs=54 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:41.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.572+0000 7f55ab942700 1 -- 172.21.15.16:0/2070007502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f55a409b830 con 0x7f55a4087680 2019-12-05T06:06:41.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.579+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:41.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.579+0000 7f55a27fc700 1 -- 172.21.15.16:0/2070007502 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f559c001270 con 0x7f55a4087680 2019-12-05T06:06:41.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:41.801+0000 7f55ab942700 1 -- 172.21.15.16:0/2070007502 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"} v 0) v1 -- 0x7f55a409b830 con 0x7f55a4087680 2019-12-05T06:06:42.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.680+0000 7f55a9edf700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:42.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.680+0000 7f55a27fc700 1 -- 172.21.15.16:0/2070007502 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "standby_count_wanted", "prefix": "fs set", "fs_name": "cephfs", "val": "0"}]=0 v3) v1 ==== 118+0+0 (secure 0 0 0) 0x7f559c02b410 con 0x7f55a4087680 2019-12-05T06:06:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.722+0000 7f55ab942700 1 -- 172.21.15.16:0/2070007502 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5584023630 msgr2=0x7f5584025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.722+0000 7f55ab942700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5584023630 0x7f5584025ae0 crc :-1 s=READY pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.722+0000 7f55ab942700 1 -- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 msgr2=0x7f55a412bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.722+0000 7f55ab942700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 0x7f55a412bb00 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7f559c0107b0 tx=0x7f559c0105e0).stop 2019-12-05T06:06:42.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.722+0000 7f55ab942700 1 -- 172.21.15.16:0/2070007502 shutdown_connections 2019-12-05T06:06:42.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.723+0000 7f55ab942700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f55a412c830 0x7f55a412c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:42.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.723+0000 7f55ab942700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5584023630 0x7f5584025ae0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:42.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.723+0000 7f55ab942700 1 --2- 172.21.15.16:0/2070007502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55a4087680 0x7f55a412bb00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:42.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.723+0000 7f55ab942700 1 -- 172.21.15.16:0/2070007502 shutdown_connections 2019-12-05T06:06:42.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.723+0000 7f55ab942700 1 -- 172.21.15.16:0/2070007502 wait complete. 2019-12-05T06:06:42.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.723+0000 7f55ab942700 1 -- 172.21.15.16:0/2070007502 >> 172.21.15.16:0/2070007502 conn(0x7f55a4088790 msgr2=0x7f55a4088c20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:42.739 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.984+0000 7ff9b00d2700 1 Processor -- start 2019-12-05T06:06:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.984+0000 7ff9b00d2700 1 -- start start 2019-12-05T06:06:42.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.985+0000 7ff9b00d2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff9a808d2c0 0x7ff9a808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.985+0000 7ff9b00d2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9a808fef0 0x7ff9a808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.985+0000 7ff9b00d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a80958e0 con 0x7ff9a808c6d0 2019-12-05T06:06:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.985+0000 7ff9b00d2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a808df90 con 0x7ff9a808fef0 2019-12-05T06:06:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.985+0000 7ff9b00d2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a80a3f70 con 0x7ff9a808d2c0 2019-12-05T06:06:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.985+0000 7ff9aee70700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9a808fef0 0x7ff9a808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.986+0000 7ff9aee70700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff9a808fef0 0x7ff9a808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46062/0 (socket says 172.21.15.16:46062) 2019-12-05T06:06:42.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.986+0000 7ff9aee70700 1 -- 172.21.15.16:0/1989379119 learned_addr learned my addr 172.21.15.16:0/1989379119 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.986+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/1989379119 >> v2:172.21.15.76:3300/0 conn(0x7ff9a808d2c0 0x7ff9a808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.986+0000 7ff9aee70700 1 -- 172.21.15.16:0/1989379119 >> v2:172.21.15.76:3300/0 conn(0x7ff9a808d2c0 msgr2=0x7ff9a808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.986+0000 7ff9aee70700 1 --2- 172.21.15.16:0/1989379119 >> v2:172.21.15.76:3300/0 conn(0x7ff9a808d2c0 0x7ff9a808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.986+0000 7ff9aee70700 1 -- 172.21.15.16:0/1989379119 >> v1:172.21.15.76:6789/0 conn(0x7ff9a808c6d0 legacy=0x7ff9a808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.986+0000 7ff9aee70700 1 -- 172.21.15.16:0/1989379119 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9a808e620 con 0x7ff9a808fef0 2019-12-05T06:06:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.986+0000 7ff9aee70700 1 --2- 172.21.15.16:0/1989379119 >> v2:172.21.15.16:3300/0 conn(0x7ff9a808fef0 0x7ff9a808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff99800f730 tx=0x7ff998007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:42.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.986+0000 7ff9aee70700 1 --2- 172.21.15.16:0/1989379119 >> v2:172.21.15.16:3300/0 conn(0x7ff9a808fef0 0x7ff9a808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff99800f730 tx=0x7ff998007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.986+0000 7ff9aee70700 1 --2- 172.21.15.16:0/1989379119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9a808fef0 0x7ff9a808f460 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7ff99800f730 tx=0x7ff998007d50).ready entity=mon.0 client_cookie=125bae4ab8b1172a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.987+0000 7ff9aee70700 1 --2- 172.21.15.16:0/1989379119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9a808fef0 0x7ff9a808f460 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7ff99800f730 tx=0x7ff998007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.987+0000 7ff9aee70700 1 --2- 172.21.15.16:0/1989379119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9a808fef0 0x7ff9a808f460 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7ff99800f730 tx=0x7ff998007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.987+0000 7ff9ad66d700 1 -- 172.21.15.16:0/1989379119 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff998000a70 con 0x7ff9a808fef0 2019-12-05T06:06:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.987+0000 7ff9aee70700 1 --2- 172.21.15.16:0/1989379119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9a808fef0 0x7ff9a808f460 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7ff99800f730 tx=0x7ff998007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.987+0000 7ff9ad66d700 1 -- 172.21.15.16:0/1989379119 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff9980010e0 con 0x7ff9a808fef0 2019-12-05T06:06:42.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.987+0000 7ff9ad66d700 1 -- 172.21.15.16:0/1989379119 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9980017b0 con 0x7ff9a808fef0 2019-12-05T06:06:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.987+0000 7ff9b00d2700 1 -- 172.21.15.16:0/1989379119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9a808fef0 msgr2=0x7ff9a808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.987+0000 7ff9b00d2700 1 --2- 172.21.15.16:0/1989379119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9a808fef0 0x7ff9a808f460 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7ff99800f730 tx=0x7ff998007d50).stop 2019-12-05T06:06:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.989+0000 7ff9b00d2700 1 -- 172.21.15.16:0/1989379119 shutdown_connections 2019-12-05T06:06:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.989+0000 7ff9b00d2700 1 --2- 172.21.15.16:0/1989379119 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff9a808fef0 0x7ff9a808f460 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.989+0000 7ff9b00d2700 1 --2- 172.21.15.16:0/1989379119 >> v2:172.21.15.76:3300/0 conn(0x7ff9a808d2c0 0x7ff9a808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:42.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.989+0000 7ff9b00d2700 1 -- 172.21.15.16:0/1989379119 shutdown_connections 2019-12-05T06:06:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.989+0000 7ff9b00d2700 1 -- 172.21.15.16:0/1989379119 wait complete. 2019-12-05T06:06:43.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.989+0000 7ff9b00d2700 1 -- 172.21.15.16:0/1989379119 >> 172.21.15.16:0/1989379119 conn(0x7ff9a8061940 msgr2=0x7ff9a8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.994+0000 7ff9b00d2700 1 Processor -- start 2019-12-05T06:06:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.994+0000 7ff9b00d2700 1 -- start start 2019-12-05T06:06:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.996+0000 7ff9b00d2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff9a8096e90 0x7ff9a809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.996+0000 7ff9b00d2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a8084b90 con 0x7ff9a8084f30 2019-12-05T06:06:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.996+0000 7ff9b00d2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9a8098790 con 0x7ff9a809b920 2019-12-05T06:06:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.996+0000 7ff9b00d2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff9a809a450 con 0x7ff9a8096e90 2019-12-05T06:06:43.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.996+0000 7ff9ade6e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff9a8084f30 0x7ff9a8097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59468/0 (socket says 172.21.15.16:59468) 2019-12-05T06:06:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.996+0000 7ff9ade6e700 1 -- 172.21.15.16:0/4012375219 learned_addr learned my addr 172.21.15.16:0/4012375219 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.996+0000 7ff9ae66f700 1 --2- 172.21.15.16:0/4012375219 >> v2:172.21.15.76:3300/0 conn(0x7ff9a8096e90 0x7ff9a809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.997+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 492710824 0 0) 0x7ff99c0010b0 con 0x7ff9a8084f30 2019-12-05T06:06:43.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.997+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff988002e80 con 0x7ff9a8084f30 2019-12-05T06:06:43.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.997+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 700268771 0 0) 0x7ff99c000940 con 0x7ff9a8084f30 2019-12-05T06:06:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.997+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff9880032e0 con 0x7ff9a8084f30 2019-12-05T06:06:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.997+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7ff99c001540 con 0x7ff9a8084f30 2019-12-05T06:06:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.997+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 >> v2:172.21.15.76:3300/0 conn(0x7ff9a8096e90 msgr2=0x7ff9a809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.997+0000 7ff9a6ffd700 1 --2- 172.21.15.16:0/4012375219 >> v2:172.21.15.76:3300/0 conn(0x7ff9a8096e90 0x7ff9a809b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.997+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 >> v1:172.21.15.76:6789/0 conn(0x7ff9a809b920 legacy=0x7ff9a8097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.997+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 >> v1:172.21.15.16:6789/0 conn(0x7ff9a8084f30 legacy=0x7ff9a8097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.997+0000 7ff9a6ffd700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff9880033b0 0x7ff9880037e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9a6ffd700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9a6ffd700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9880049b0 0x7ff988004e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:43.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98800bc70 con 0x7ff988003d20 2019-12-05T06:06:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98800bea0 con 0x7ff9880049b0 2019-12-05T06:06:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff98800e2a0 con 0x7ff9880033b0 2019-12-05T06:06:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9ae66f700 1 --2- 172.21.15.16:0/4012375219 >> v2:172.21.15.76:3300/0 conn(0x7ff9a8096e90 0x7ff9a809b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9aee70700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9880049b0 0x7ff988004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9ae66f700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff9880033b0 0x7ff9880037e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:43.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9ade6e700 1 -- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff9880033b0 msgr2=0x7ff9880037e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff9880033b0 0x7ff9880037e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9ade6e700 1 -- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9880049b0 msgr2=0x7ff988004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff9880049b0 0x7ff988004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9aee70700 1 -- 172.21.15.16:0/4012375219 reap_dead start 2019-12-05T06:06:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9ade6e700 1 -- 172.21.15.16:0/4012375219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff98800e550 con 0x7ff988003d20 2019-12-05T06:06:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9b00d2700 1 -- 172.21.15.16:0/4012375219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9a809aae0 con 0x7ff988003d20 2019-12-05T06:06:43.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.998+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.999+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.999+0000 7ff9b00d2700 1 -- 172.21.15.16:0/4012375219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9a8084b90 con 0x7ff988003d20 2019-12-05T06:06:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.999+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=READY pgs=85 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.999+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.999+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff99c000ea0 con 0x7ff988003d20 2019-12-05T06:06:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.999+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.999+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.999+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff99c0012c0 con 0x7ff988003d20 2019-12-05T06:06:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:42.999+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff99c000ea0 con 0x7ff988003d20 2019-12-05T06:06:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.000+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.000+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff99c0014b0 con 0x7ff988003d20 2019-12-05T06:06:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.000+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.000+0000 7ff9a6ffd700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff9880285c0 0x7ff98802a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.001+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff99c045ca0 con 0x7ff988003d20 2019-12-05T06:06:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.001+0000 7ff9ae66f700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff9880285c0 0x7ff98802a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:43.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.001+0000 7ff9ae66f700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff9880285c0 0x7ff98802a9f0 crc :-1 s=READY pgs=55 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:43.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.016+0000 7ff9b00d2700 1 -- 172.21.15.16:0/4012375219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9a809aae0 con 0x7ff988003d20 2019-12-05T06:06:43.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.023+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.023+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff99c0010b0 con 0x7ff988003d20 2019-12-05T06:06:43.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.229+0000 7ff9b00d2700 1 -- 172.21.15.16:0/4012375219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff9a809aae0 con 0x7ff988003d20 2019-12-05T06:06:43.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.230+0000 7ff9ade6e700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=THROTTLE_DONE pgs=85 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.230+0000 7ff9a6ffd700 1 -- 172.21.15.16:0/4012375219 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+1420 (secure 0 0 0) 0x7ff99c0014b0 con 0x7ff988003d20 2019-12-05T06:06:43.228 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 3 2019-12-05T06:06:43.229 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:43.229 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":3,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:06:42.668095+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:43.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.233+0000 7ff9b00d2700 1 -- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff9880285c0 msgr2=0x7ff98802a9f0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.233+0000 7ff9b00d2700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff9880285c0 0x7ff98802a9f0 crc :-1 s=READY pgs=55 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.233+0000 7ff9b00d2700 1 -- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 msgr2=0x7ff9880041a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.233+0000 7ff9b00d2700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 secure :-1 s=READY pgs=85 cs=0 l=1 rx=0x7ff99c0103a0 tx=0x7ff99c010280).stop 2019-12-05T06:06:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.233+0000 7ff9b00d2700 1 -- 172.21.15.16:0/4012375219 shutdown_connections 2019-12-05T06:06:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.233+0000 7ff9b00d2700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff9880285c0 0x7ff98802a9f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.233+0000 7ff9b00d2700 1 --2- 172.21.15.16:0/4012375219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff988003d20 0x7ff9880041a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.233+0000 7ff9b00d2700 1 -- 172.21.15.16:0/4012375219 shutdown_connections 2019-12-05T06:06:43.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.233+0000 7ff9b00d2700 1 -- 172.21.15.16:0/4012375219 wait complete. 2019-12-05T06:06:43.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.233+0000 7ff9b00d2700 1 -- 172.21.15.16:0/4012375219 >> 172.21.15.16:0/4012375219 conn(0x7ff9a80714f0 msgr2=0x7ff9a809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:43.245 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:06:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.497+0000 7fb73d4ff700 1 Processor -- start 2019-12-05T06:06:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.497+0000 7fb73d4ff700 1 -- start start 2019-12-05T06:06:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.499+0000 7fb73d4ff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb73808c6d0 0x7fb73808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.499+0000 7fb73d4ff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb73808f720 0x7fb73808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.499+0000 7fb73d4ff700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73808fb80 con 0x7fb73808fef0 2019-12-05T06:06:43.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.499+0000 7fb73d4ff700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73808df90 con 0x7fb73808c6d0 2019-12-05T06:06:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.499+0000 7fb73d4ff700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7380a3f70 con 0x7fb73808f720 2019-12-05T06:06:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.500+0000 7fb7377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb73808c6d0 0x7fb73808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.500+0000 7fb736ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb73808fef0 0x7fb73808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59482/0 (socket says 172.21.15.16:59482) 2019-12-05T06:06:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.500+0000 7fb736ffd700 1 -- 172.21.15.16:0/719243094 learned_addr learned my addr 172.21.15.16:0/719243094 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.500+0000 7fb737fff700 1 --2- 172.21.15.16:0/719243094 >> v2:172.21.15.76:3300/0 conn(0x7fb73808f720 0x7fb73808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.500+0000 7fb7367fc700 1 -- 172.21.15.16:0/719243094 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736247682 0 0) 0x7fb720001150 con 0x7fb73808fef0 2019-12-05T06:06:43.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.500+0000 7fb7367fc700 1 -- 172.21.15.16:0/719243094 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb724002e80 con 0x7fb73808fef0 2019-12-05T06:06:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.500+0000 7fb737fff700 1 -- 172.21.15.16:0/719243094 >> v2:172.21.15.16:3300/0 conn(0x7fb73808c6d0 msgr2=0x7fb73808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.500+0000 7fb737fff700 1 --2- 172.21.15.16:0/719243094 >> v2:172.21.15.16:3300/0 conn(0x7fb73808c6d0 0x7fb73808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.500+0000 7fb737fff700 1 -- 172.21.15.16:0/719243094 >> v1:172.21.15.16:6789/0 conn(0x7fb73808fef0 legacy=0x7fb73808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.500+0000 7fb737fff700 1 -- 172.21.15.16:0/719243094 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73808d540 con 0x7fb73808f720 2019-12-05T06:06:43.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.501+0000 7fb7377fe700 1 --2- 172.21.15.16:0/719243094 >> v2:172.21.15.16:3300/0 conn(0x7fb73808c6d0 0x7fb73808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.501+0000 7fb737fff700 1 --2- 172.21.15.16:0/719243094 >> v2:172.21.15.76:3300/0 conn(0x7fb73808f720 0x7fb73808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb72800af80 tx=0x7fb72800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.501+0000 7fb737fff700 1 --2- 172.21.15.16:0/719243094 >> v2:172.21.15.76:3300/0 conn(0x7fb73808f720 0x7fb73808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb72800af80 tx=0x7fb72800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.501+0000 7fb737fff700 1 --2- 172.21.15.16:0/719243094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73808f720 0x7fb73808d000 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7fb72800af80 tx=0x7fb72800b790).ready entity=mon.1 client_cookie=31ead9051af100f2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.501+0000 7fb737fff700 1 --2- 172.21.15.16:0/719243094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73808f720 0x7fb73808d000 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fb72800af80 tx=0x7fb72800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.501+0000 7fb7367fc700 1 -- 172.21.15.16:0/719243094 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb728000bc0 con 0x7fb73808f720 2019-12-05T06:06:43.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.501+0000 7fb737fff700 1 --2- 172.21.15.16:0/719243094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73808f720 0x7fb73808d000 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fb72800af80 tx=0x7fb72800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.501+0000 7fb737fff700 1 --2- 172.21.15.16:0/719243094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73808f720 0x7fb73808d000 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7fb72800af80 tx=0x7fb72800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.502+0000 7fb7367fc700 1 -- 172.21.15.16:0/719243094 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7280097f0 con 0x7fb73808f720 2019-12-05T06:06:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.502+0000 7fb7367fc700 1 -- 172.21.15.16:0/719243094 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb728009ec0 con 0x7fb73808f720 2019-12-05T06:06:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.502+0000 7fb73d4ff700 1 -- 172.21.15.16:0/719243094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73808f720 msgr2=0x7fb73808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.502+0000 7fb73d4ff700 1 --2- 172.21.15.16:0/719243094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73808f720 0x7fb73808d000 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7fb72800af80 tx=0x7fb72800b790).stop 2019-12-05T06:06:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.503+0000 7fb73d4ff700 1 -- 172.21.15.16:0/719243094 shutdown_connections 2019-12-05T06:06:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.503+0000 7fb73d4ff700 1 --2- 172.21.15.16:0/719243094 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73808f720 0x7fb73808d000 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.503+0000 7fb73d4ff700 1 --2- 172.21.15.16:0/719243094 >> v2:172.21.15.16:3300/0 conn(0x7fb73808c6d0 0x7fb73808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.503+0000 7fb73d4ff700 1 -- 172.21.15.16:0/719243094 shutdown_connections 2019-12-05T06:06:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.503+0000 7fb73d4ff700 1 -- 172.21.15.16:0/719243094 wait complete. 2019-12-05T06:06:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.503+0000 7fb73d4ff700 1 -- 172.21.15.16:0/719243094 >> 172.21.15.16:0/719243094 conn(0x7fb738061940 msgr2=0x7fb738096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.509+0000 7fb73d4ff700 1 Processor -- start 2019-12-05T06:06:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.509+0000 7fb73d4ff700 1 -- start start 2019-12-05T06:06:43.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.511+0000 7fb73d4ff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb73814d400 0x7fb73814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:43.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.511+0000 7fb73d4ff700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb738098da0 con 0x7fb738132530 2019-12-05T06:06:43.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.511+0000 7fb73d4ff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73809f0a0 con 0x7fb738132d00 2019-12-05T06:06:43.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.511+0000 7fb73d4ff700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb73809bb50 con 0x7fb73814d400 2019-12-05T06:06:43.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.511+0000 7fb737fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb738132530 0x7fb738157100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59488/0 (socket says 172.21.15.16:59488) 2019-12-05T06:06:43.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.511+0000 7fb737fff700 1 -- 172.21.15.16:0/525683903 learned_addr learned my addr 172.21.15.16:0/525683903 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:43.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.511+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> v2:172.21.15.76:3300/0 conn(0x7fb73814d400 0x7fb73814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:43.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.511+0000 7fb72ffff700 1 -- 172.21.15.16:0/525683903 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 646740899 0 0) 0x7fb7280095b0 con 0x7fb738132530 2019-12-05T06:06:43.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.512+0000 7fb72ffff700 1 -- 172.21.15.16:0/525683903 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb71c002e80 con 0x7fb738132530 2019-12-05T06:06:43.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.512+0000 7fb7377fe700 1 -- 172.21.15.16:0/525683903 >> v1:172.21.15.76:6789/0 conn(0x7fb738132d00 legacy=0x7fb738131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.512+0000 7fb7377fe700 1 -- 172.21.15.16:0/525683903 >> v1:172.21.15.16:6789/0 conn(0x7fb738132530 legacy=0x7fb738157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.512+0000 7fb7377fe700 1 -- 172.21.15.16:0/525683903 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73809b0f0 con 0x7fb73814d400 2019-12-05T06:06:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.512+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> v2:172.21.15.76:3300/0 conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.512+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> v2:172.21.15.76:3300/0 conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.512+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:43.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.513+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.513+0000 7fb72ffff700 1 -- 172.21.15.16:0/525683903 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb730000cc0 con 0x7fb73814d400 2019-12-05T06:06:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.513+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.513+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.513+0000 7fb72ffff700 1 -- 172.21.15.16:0/525683903 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb73000e370 con 0x7fb73814d400 2019-12-05T06:06:43.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.513+0000 7fb73d4ff700 1 -- 172.21.15.16:0/525683903 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73809b0f0 con 0x7fb73814d400 2019-12-05T06:06:43.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.513+0000 7fb72ffff700 1 -- 172.21.15.16:0/525683903 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb73000ea40 con 0x7fb73814d400 2019-12-05T06:06:43.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.513+0000 7fb73d4ff700 1 -- 172.21.15.16:0/525683903 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73809b0f0 con 0x7fb73814d400 2019-12-05T06:06:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.514+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.515+0000 7fb72ffff700 1 -- 172.21.15.16:0/525683903 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb73000de80 con 0x7fb73814d400 2019-12-05T06:06:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.515+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.515+0000 7fb72ffff700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb71c023670 0x7fb71c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.515+0000 7fb72ffff700 1 -- 172.21.15.16:0/525683903 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fb730046160 con 0x7fb73814d400 2019-12-05T06:06:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.516+0000 7fb736ffd700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb71c023670 0x7fb71c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:43.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.516+0000 7fb736ffd700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb71c023670 0x7fb71c025ae0 crc :-1 s=READY pgs=56 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:43.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.528+0000 7fb73d4ff700 1 -- 172.21.15.16:0/525683903 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73809b0f0 con 0x7fb73814d400 2019-12-05T06:06:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.538+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.538+0000 7fb72ffff700 1 -- 172.21.15.16:0/525683903 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb730033660 con 0x7fb73814d400 2019-12-05T06:06:43.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.789+0000 7fb73d4ff700 1 -- 172.21.15.16:0/525683903 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb73809b0f0 con 0x7fb73814d400 2019-12-05T06:06:43.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.791+0000 7fb7377fe700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=THROTTLE_DONE pgs=80 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:43.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.791+0000 7fb72ffff700 1 -- 172.21.15.16:0/525683903 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7fb730033660 con 0x7fb73814d400 2019-12-05T06:06:43.790 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:43.791 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:06:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.794+0000 7fb73d4ff700 1 -- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb71c023670 msgr2=0x7fb71c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.794+0000 7fb73d4ff700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb71c023670 0x7fb71c025ae0 crc :-1 s=READY pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.794+0000 7fb73d4ff700 1 -- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 msgr2=0x7fb73814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.795+0000 7fb73d4ff700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 0x7fb73814d7f0 secure :-1 s=READY pgs=80 cs=0 l=1 rx=0x7fb730010720 tx=0x7fb730010ac0).stop 2019-12-05T06:06:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.795+0000 7fb73d4ff700 1 -- 172.21.15.16:0/525683903 shutdown_connections 2019-12-05T06:06:43.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.795+0000 7fb73d4ff700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb71c023670 0x7fb71c025ae0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:43.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.795+0000 7fb73d4ff700 1 --2- 172.21.15.16:0/525683903 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb73814d400 0x7fb73814d7f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.795+0000 7fb73d4ff700 1 -- 172.21.15.16:0/525683903 shutdown_connections 2019-12-05T06:06:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.795+0000 7fb73d4ff700 1 -- 172.21.15.16:0/525683903 wait complete. 2019-12-05T06:06:43.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:43.795+0000 7fb73d4ff700 1 -- 172.21.15.16:0/525683903 >> 172.21.15.16:0/525683903 conn(0x7fb7380714f0 msgr2=0x7fb73809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:43.812 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2019-12-05T06:06:43.812 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:06:43.813 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:06:43.821 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:06:43.821 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:06:43.821 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:06:43.826 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:06:43.827 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:06:43.828 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2019-12-05T06:06:43.965 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:06:44.002 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:06:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.039+0000 7f3ef414f700 1 Processor -- start 2019-12-05T06:06:44.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.039+0000 7f3ef414f700 1 -- start start 2019-12-05T06:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef414f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3eec08fef0 0x7f3eec08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef414f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec0958e0 con 0x7f3eec08d2c0 2019-12-05T06:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef414f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec0629e0 con 0x7f3eec08c6d0 2019-12-05T06:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef414f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3eec08df90 con 0x7f3eec08fef0 2019-12-05T06:06:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef2eed700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3eec08fef0 0x7f3eec08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef2eed700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3eec08fef0 0x7f3eec08f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46104/0 (socket says 172.21.15.16:46104) 2019-12-05T06:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef2eed700 1 -- 172.21.15.16:0/1365360967 learned_addr learned my addr 172.21.15.16:0/1365360967 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef1eeb700 1 --1- 172.21.15.16:0/1365360967 >> v1:172.21.15.16:6789/0 conn(0x7f3eec08d2c0 0x7f3eec08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59500/0 (socket says 172.21.15.16:59500) 2019-12-05T06:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef2eed700 1 -- 172.21.15.16:0/1365360967 >> v1:172.21.15.76:6789/0 conn(0x7f3eec08c6d0 legacy=0x7f3eec08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef2eed700 1 -- 172.21.15.16:0/1365360967 >> v1:172.21.15.16:6789/0 conn(0x7f3eec08d2c0 legacy=0x7f3eec08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.041+0000 7f3ef2eed700 1 -- 172.21.15.16:0/1365360967 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec08e620 con 0x7f3eec08fef0 2019-12-05T06:06:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef2eed700 1 --2- 172.21.15.16:0/1365360967 >> v2:172.21.15.16:3300/0 conn(0x7f3eec08fef0 0x7f3eec08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3ee400fab0 tx=0x7f3ee400fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef2eed700 1 --2- 172.21.15.16:0/1365360967 >> v2:172.21.15.16:3300/0 conn(0x7f3eec08fef0 0x7f3eec08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3ee400fab0 tx=0x7f3ee400fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef2eed700 1 --2- 172.21.15.16:0/1365360967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eec08fef0 0x7f3eec08f720 secure :-1 s=READY pgs=94 cs=0 l=1 rx=0x7f3ee400fab0 tx=0x7f3ee400fe50).ready entity=mon.0 client_cookie=9997ee7fa4f8da18 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:44.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef2eed700 1 --2- 172.21.15.16:0/1365360967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eec08fef0 0x7f3eec08f720 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f3ee400fab0 tx=0x7f3ee400fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef2eed700 1 --2- 172.21.15.16:0/1365360967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eec08fef0 0x7f3eec08f720 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f3ee400fab0 tx=0x7f3ee400fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef2eed700 1 --2- 172.21.15.16:0/1365360967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eec08fef0 0x7f3eec08f720 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7f3ee400fab0 tx=0x7f3ee400fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef16ea700 1 -- 172.21.15.16:0/1365360967 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ee4001460 con 0x7f3eec08fef0 2019-12-05T06:06:44.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef16ea700 1 -- 172.21.15.16:0/1365360967 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ee4000df0 con 0x7f3eec08fef0 2019-12-05T06:06:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef16ea700 1 -- 172.21.15.16:0/1365360967 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ee4001b40 con 0x7f3eec08fef0 2019-12-05T06:06:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef414f700 1 -- 172.21.15.16:0/1365360967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eec08fef0 msgr2=0x7f3eec08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.042+0000 7f3ef414f700 1 --2- 172.21.15.16:0/1365360967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eec08fef0 0x7f3eec08f720 secure :-1 s=READY pgs=94 cs=0 l=1 rx=0x7f3ee400fab0 tx=0x7f3ee400fe50).stop 2019-12-05T06:06:44.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.043+0000 7f3ef414f700 1 -- 172.21.15.16:0/1365360967 shutdown_connections 2019-12-05T06:06:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.043+0000 7f3ef414f700 1 --2- 172.21.15.16:0/1365360967 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3eec08fef0 0x7f3eec08f720 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.044+0000 7f3ef414f700 1 -- 172.21.15.16:0/1365360967 shutdown_connections 2019-12-05T06:06:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.044+0000 7f3ef414f700 1 -- 172.21.15.16:0/1365360967 wait complete. 2019-12-05T06:06:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.044+0000 7f3ef414f700 1 -- 172.21.15.16:0/1365360967 >> 172.21.15.16:0/1365360967 conn(0x7f3eec061940 msgr2=0x7f3eec096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.047+0000 7f3ef414f700 1 Processor -- start 2019-12-05T06:06:44.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.048+0000 7f3ef414f700 1 -- start start 2019-12-05T06:06:44.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.049+0000 7f3ef414f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:44.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.049+0000 7f3ef414f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec06d980 con 0x7f3eec1539a0 2019-12-05T06:06:44.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.049+0000 7f3ef414f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3eec09d3a0 con 0x7f3eec1526c0 2019-12-05T06:06:44.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.049+0000 7f3ef414f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3eec154220 con 0x7f3eec153d90 2019-12-05T06:06:44.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.049+0000 7f3ef26ec700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3eec1539a0 0x7f3eec097a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59506/0 (socket says 172.21.15.16:59506) 2019-12-05T06:06:44.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.049+0000 7f3ef26ec700 1 -- 172.21.15.16:0/767601587 learned_addr learned my addr 172.21.15.16:0/767601587 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:44.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.049+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.049+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297740485 0 0) 0x7f3ee8001150 con 0x7f3eec1539a0 2019-12-05T06:06:44.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.049+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed4002e80 con 0x7f3eec1539a0 2019-12-05T06:06:44.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.049+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 270792887 0 0) 0x7f3ee4007900 con 0x7f3eec1526c0 2019-12-05T06:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.050+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ed4004170 con 0x7f3eec1526c0 2019-12-05T06:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.050+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2426800845 0 0) 0x7f3ee80012c0 con 0x7f3eec1539a0 2019-12-05T06:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.050+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3ed4003140 con 0x7f3eec1539a0 2019-12-05T06:06:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.050+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f3ee8001870 con 0x7f3eec1539a0 2019-12-05T06:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.050+0000 7f3ef1eeb700 1 -- 172.21.15.16:0/767601587 >> v1:172.21.15.76:6789/0 conn(0x7f3eec1526c0 legacy=0x7f3eec152b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.050+0000 7f3ef1eeb700 1 -- 172.21.15.16:0/767601587 >> v1:172.21.15.16:6789/0 conn(0x7f3eec1539a0 legacy=0x7f3eec097a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.050+0000 7f3ef1eeb700 1 -- 172.21.15.16:0/767601587 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eec097120 con 0x7f3eec153d90 2019-12-05T06:06:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.050+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.050+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.050+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.051+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.051+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3edc004240 con 0x7f3eec153d90 2019-12-05T06:06:44.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.051+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.051+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3edc000c90 con 0x7f3eec153d90 2019-12-05T06:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.051+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.051+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3edc004430 con 0x7f3eec153d90 2019-12-05T06:06:44.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.051+0000 7f3ef414f700 1 -- 172.21.15.16:0/767601587 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3eec097120 con 0x7f3eec153d90 2019-12-05T06:06:44.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.051+0000 7f3ef414f700 1 -- 172.21.15.16:0/767601587 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3eec097120 con 0x7f3eec153d90 2019-12-05T06:06:44.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.052+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.052+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3edc000c90 con 0x7f3eec153d90 2019-12-05T06:06:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.052+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.053+0000 7f3ee2ffd700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 0x7f3ed4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:44.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.053+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f3edc045da0 con 0x7f3eec153d90 2019-12-05T06:06:44.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.053+0000 7f3ef26ec700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 0x7f3ed4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.053+0000 7f3ef26ec700 1 -- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 msgr2=0x7f3ed4025ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:06:44.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.053+0000 7f3ef26ec700 1 -- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 msgr2=0x7f3ed4025ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:06:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.053+0000 7f3ef26ec700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 0x7f3ed4025ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.053+0000 7f3ef26ec700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 0x7f3ed4025ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:06:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.054+0000 7f3ef414f700 1 -- 172.21.15.16:0/767601587 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3eec097120 con 0x7f3eec153d90 2019-12-05T06:06:44.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.064+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.064+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3edc001220 con 0x7f3eec153d90 2019-12-05T06:06:44.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.254+0000 7f3ef26ec700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 0x7f3ed4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.254+0000 7f3ef26ec700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 0x7f3ed4025ae0 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:44.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.282+0000 7f3ef414f700 1 -- 172.21.15.16:0/767601587 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3eec097120 con 0x7f3eec153d90 2019-12-05T06:06:44.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.285+0000 7f3ef1eeb700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=THROTTLE_DONE pgs=27 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.285+0000 7f3ee2ffd700 1 -- 172.21.15.16:0/767601587 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f3edc000c90 con 0x7f3eec153d90 2019-12-05T06:06:44.284 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:44.285 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:06:44.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.288+0000 7f3ef414f700 1 -- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 msgr2=0x7f3ed4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.288+0000 7f3ef414f700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 0x7f3ed4025ae0 crc :-1 s=READY pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.288+0000 7f3ef414f700 1 -- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 msgr2=0x7f3eec098220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.288+0000 7f3ef414f700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 secure :-1 s=READY pgs=27 cs=0 l=1 rx=0x7f3edc010370 tx=0x7f3edc010710).stop 2019-12-05T06:06:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.288+0000 7f3ef414f700 1 -- 172.21.15.16:0/767601587 shutdown_connections 2019-12-05T06:06:44.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.288+0000 7f3ef414f700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3ed4023670 0x7f3ed4025ae0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.288+0000 7f3ef414f700 1 --2- 172.21.15.16:0/767601587 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3eec153d90 0x7f3eec098220 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.288+0000 7f3ef414f700 1 -- 172.21.15.16:0/767601587 shutdown_connections 2019-12-05T06:06:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.289+0000 7f3ef414f700 1 -- 172.21.15.16:0/767601587 wait complete. 2019-12-05T06:06:44.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.289+0000 7f3ef414f700 1 -- 172.21.15.16:0/767601587 >> 172.21.15.16:0/767601587 conn(0x7f3eec06c820 msgr2=0x7f3eec06dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:44.299 INFO:tasks.ceph.ceph_manager.ceph:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:31.052682+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:34.862662+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:37.644552+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T06:06:44.300 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T06:06:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.507+0000 7fe90712d700 1 Processor -- start 2019-12-05T06:06:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.508+0000 7fe90712d700 1 -- start start 2019-12-05T06:06:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.509+0000 7fe90712d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe900086980 0x7fe900086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:44.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.509+0000 7fe90712d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 0x7fe90008ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.509+0000 7fe90712d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe900061a10 con 0x7fe90008b060 2019-12-05T06:06:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.509+0000 7fe90712d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe900060bc0 con 0x7fe900086980 2019-12-05T06:06:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.509+0000 7fe90712d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe900060620 con 0x7fe90008a6b0 2019-12-05T06:06:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.510+0000 7fe904ec9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 0x7fe90008ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.510+0000 7fe9056ca700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe900086980 0x7fe900086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.510+0000 7fe905ecb700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe90008b060 0x7fe900064310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52592/0 (socket says 172.21.15.16:52592) 2019-12-05T06:06:44.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.510+0000 7fe905ecb700 1 -- 172.21.15.16:0/2063736603 learned_addr learned my addr 172.21.15.16:0/2063736603 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:44.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.511+0000 7fe904ec9700 1 -- 172.21.15.16:0/2063736603 >> v2:172.21.15.76:3300/0 conn(0x7fe900086980 msgr2=0x7fe900086d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:44.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.511+0000 7fe904ec9700 1 --2- 172.21.15.16:0/2063736603 >> v2:172.21.15.76:3300/0 conn(0x7fe900086980 0x7fe900086d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:44.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.511+0000 7fe904ec9700 1 -- 172.21.15.16:0/2063736603 >> v1:172.21.15.76:6789/0 conn(0x7fe90008b060 legacy=0x7fe900064310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.511+0000 7fe904ec9700 1 -- 172.21.15.16:0/2063736603 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe900063a10 con 0x7fe90008a6b0 2019-12-05T06:06:44.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.511+0000 7fe8fffff700 1 -- 172.21.15.16:0/2063736603 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653534035 0 0) 0x7fe8f8001100 con 0x7fe90008b060 2019-12-05T06:06:44.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.511+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2063736603 >> v2:172.21.15.76:3300/0 conn(0x7fe900086980 0x7fe900086d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:44.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.511+0000 7fe904ec9700 1 --2- 172.21.15.16:0/2063736603 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 0x7fe90008ab20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe8f400af50 tx=0x7fe8f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.511+0000 7fe904ec9700 1 --2- 172.21.15.16:0/2063736603 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 0x7fe90008ab20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe8f400af50 tx=0x7fe8f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.511+0000 7fe904ec9700 1 --2- 172.21.15.16:0/2063736603 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 0x7fe90008ab20 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7fe8f400af50 tx=0x7fe8f400b760).ready entity=mon.2 client_cookie=3eacafeefbf2ebac server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:44.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.512+0000 7fe904ec9700 1 --2- 172.21.15.16:0/2063736603 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 0x7fe90008ab20 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fe8f400af50 tx=0x7fe8f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.512+0000 7fe8fffff700 1 -- 172.21.15.16:0/2063736603 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8f4007210 con 0x7fe90008a6b0 2019-12-05T06:06:44.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.512+0000 7fe904ec9700 1 --2- 172.21.15.16:0/2063736603 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 0x7fe90008ab20 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fe8f400af50 tx=0x7fe8f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.512+0000 7fe8fffff700 1 -- 172.21.15.16:0/2063736603 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f40097c0 con 0x7fe90008a6b0 2019-12-05T06:06:44.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.512+0000 7fe904ec9700 1 --2- 172.21.15.16:0/2063736603 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 0x7fe90008ab20 secure :-1 s=THROTTLE_DONE pgs=28 cs=0 l=1 rx=0x7fe8f400af50 tx=0x7fe8f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.512+0000 7fe8fffff700 1 -- 172.21.15.16:0/2063736603 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8f40099b0 con 0x7fe90008a6b0 2019-12-05T06:06:44.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.512+0000 7fe90712d700 1 -- 172.21.15.16:0/2063736603 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 msgr2=0x7fe90008ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.512+0000 7fe90712d700 1 --2- 172.21.15.16:0/2063736603 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 0x7fe90008ab20 secure :-1 s=READY pgs=28 cs=0 l=1 rx=0x7fe8f400af50 tx=0x7fe8f400b760).stop 2019-12-05T06:06:44.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.513+0000 7fe90712d700 1 -- 172.21.15.16:0/2063736603 shutdown_connections 2019-12-05T06:06:44.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.513+0000 7fe90712d700 1 --2- 172.21.15.16:0/2063736603 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe90008a6b0 0x7fe90008ab20 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.513+0000 7fe90712d700 1 --2- 172.21.15.16:0/2063736603 >> v2:172.21.15.76:3300/0 conn(0x7fe900086980 0x7fe900086d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:44.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.514+0000 7fe90712d700 1 -- 172.21.15.16:0/2063736603 shutdown_connections 2019-12-05T06:06:44.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.514+0000 7fe90712d700 1 -- 172.21.15.16:0/2063736603 wait complete. 2019-12-05T06:06:44.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.514+0000 7fe90712d700 1 -- 172.21.15.16:0/2063736603 >> 172.21.15.16:0/2063736603 conn(0x7fe90005f5a0 msgr2=0x7fe90009cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:44.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.519+0000 7fe90712d700 1 Processor -- start 2019-12-05T06:06:44.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.519+0000 7fe90712d700 1 -- start start 2019-12-05T06:06:44.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.521+0000 7fe90712d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe90014f6a0 0x7fe90014fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:44.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.521+0000 7fe90712d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe90013c760 0x7fe9001592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:44.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.521+0000 7fe90712d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe90013cbc0 con 0x7fe90013cf30 2019-12-05T06:06:44.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.521+0000 7fe90712d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe900087c80 con 0x7fe90013c760 2019-12-05T06:06:44.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.521+0000 7fe90712d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe90013afd0 con 0x7fe90014f6a0 2019-12-05T06:06:44.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.521+0000 7fe905ecb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe90013c760 0x7fe9001592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.521+0000 7fe9056ca700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe90014f6a0 0x7fe90014fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.521+0000 7fe905ecb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe90013c760 0x7fe9001592f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46130/0 (socket says 172.21.15.16:46130) 2019-12-05T06:06:44.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.521+0000 7fe905ecb700 1 -- 172.21.15.16:0/2823486742 learned_addr learned my addr 172.21.15.16:0/2823486742 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:44.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe9056ca700 1 -- 172.21.15.16:0/2823486742 >> v2:172.21.15.16:3300/0 conn(0x7fe90013c760 msgr2=0x7fe9001592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> v2:172.21.15.16:3300/0 conn(0x7fe90013c760 0x7fe9001592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe9056ca700 1 -- 172.21.15.16:0/2823486742 >> v1:172.21.15.16:6789/0 conn(0x7fe90013cf30 legacy=0x7fe90013bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe9056ca700 1 -- 172.21.15.16:0/2823486742 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe90013b660 con 0x7fe90014f6a0 2019-12-05T06:06:44.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe905ecb700 1 --2- 172.21.15.16:0/2823486742 >> v2:172.21.15.16:3300/0 conn(0x7fe90013c760 0x7fe9001592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:44.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> v2:172.21.15.76:3300/0 conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> v2:172.21.15.76:3300/0 conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=READY pgs=87 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:44.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe8fdffb700 1 -- 172.21.15.16:0/2823486742 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8f0000cc0 con 0x7fe90014f6a0 2019-12-05T06:06:44.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe8fdffb700 1 -- 172.21.15.16:0/2823486742 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8f000e370 con 0x7fe90014f6a0 2019-12-05T06:06:44.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.522+0000 7fe8fdffb700 1 -- 172.21.15.16:0/2823486742 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe8f000ea00 con 0x7fe90014f6a0 2019-12-05T06:06:44.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.523+0000 7fe90712d700 1 -- 172.21.15.16:0/2823486742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe90013b660 con 0x7fe90014f6a0 2019-12-05T06:06:44.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.523+0000 7fe90712d700 1 -- 172.21.15.16:0/2823486742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe90013cbc0 con 0x7fe90014f6a0 2019-12-05T06:06:44.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.524+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.525+0000 7fe8fdffb700 1 -- 172.21.15.16:0/2823486742 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe8f0009950 con 0x7fe90014f6a0 2019-12-05T06:06:44.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.525+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.525+0000 7fe8fdffb700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe8e0023630 0x7fe8e0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:44.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.525+0000 7fe8fdffb700 1 -- 172.21.15.16:0/2823486742 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fe8f0046100 con 0x7fe90014f6a0 2019-12-05T06:06:44.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.525+0000 7fe904ec9700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe8e0023630 0x7fe8e0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:44.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.526+0000 7fe904ec9700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe8e0023630 0x7fe8e0025ac0 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:44.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.539+0000 7fe90712d700 1 -- 172.21.15.16:0/2823486742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe90013b660 con 0x7fe90014f6a0 2019-12-05T06:06:44.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.549+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.549+0000 7fe8fdffb700 1 -- 172.21.15.16:0/2823486742 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe8f0000cc0 con 0x7fe90014f6a0 2019-12-05T06:06:44.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.796+0000 7fe90712d700 1 -- 172.21.15.16:0/2823486742 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fe90013b660 con 0x7fe90014f6a0 2019-12-05T06:06:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.797+0000 7fe9056ca700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=THROTTLE_DONE pgs=87 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:44.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.797+0000 7fe8fdffb700 1 -- 172.21.15.16:0/2823486742 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe8f0009950 con 0x7fe90014f6a0 2019-12-05T06:06:44.798 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:06:44.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 -- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe8e0023630 msgr2=0x7fe8e0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe8e0023630 0x7fe8e0025ac0 crc :-1 s=READY pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 -- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 msgr2=0x7fe90014fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 0x7fe90014fa90 secure :-1 s=READY pgs=87 cs=0 l=1 rx=0x7fe8f0010720 tx=0x7fe8f0010ac0).stop 2019-12-05T06:06:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 -- 172.21.15.16:0/2823486742 shutdown_connections 2019-12-05T06:06:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe8e0023630 0x7fe8e0025ac0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 --2- 172.21.15.16:0/2823486742 >> v2:172.21.15.16:3300/0 conn(0x7fe90013c760 0x7fe9001592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 --2- 172.21.15.16:0/2823486742 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe90014f6a0 0x7fe90014fa90 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 -- 172.21.15.16:0/2823486742 shutdown_connections 2019-12-05T06:06:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 -- 172.21.15.16:0/2823486742 wait complete. 2019-12-05T06:06:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:44.804+0000 7fe90712d700 1 -- 172.21.15.16:0/2823486742 >> 172.21.15.16:0/2823486742 conn(0x7fe900088790 msgr2=0x7fe900088c20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:44.818 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T06:06:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.033+0000 7f2854fb2700 1 Processor -- start 2019-12-05T06:06:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.034+0000 7f2854fb2700 1 -- start start 2019-12-05T06:06:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.035+0000 7f2854fb2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2850086d70 0x7f285008aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.035+0000 7f2854fb2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2850061a10 con 0x7f2850086980 2019-12-05T06:06:45.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.035+0000 7f2854fb2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2850064f70 con 0x7f2850064310 2019-12-05T06:06:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.035+0000 7f2854fb2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2850060bc0 con 0x7f2850086d70 2019-12-05T06:06:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.035+0000 7f284ed9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2850086980 0x7f285008a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59548/0 (socket says 172.21.15.16:59548) 2019-12-05T06:06:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.035+0000 7f284ed9d700 1 -- 172.21.15.16:0/3796759731 learned_addr learned my addr 172.21.15.16:0/3796759731 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.036+0000 7f284e59c700 1 --2- 172.21.15.16:0/3796759731 >> v2:172.21.15.76:3300/0 conn(0x7f2850086d70 0x7f285008aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.036+0000 7f284dd9b700 1 -- 172.21.15.16:0/3796759731 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3143258344 0 0) 0x7f28480014d0 con 0x7f2850086980 2019-12-05T06:06:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.036+0000 7f284dd9b700 1 -- 172.21.15.16:0/3796759731 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f283c002e80 con 0x7f2850086980 2019-12-05T06:06:45.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.036+0000 7f284e59c700 1 -- 172.21.15.16:0/3796759731 >> v1:172.21.15.76:6789/0 conn(0x7f2850064310 legacy=0x7f2850064770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.036+0000 7f284dd9b700 1 -- 172.21.15.16:0/3796759731 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4181771438 0 0) 0x7f28400009b0 con 0x7f2850064310 2019-12-05T06:06:45.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.036+0000 7f284e59c700 1 -- 172.21.15.16:0/3796759731 >> v1:172.21.15.16:6789/0 conn(0x7f2850086980 legacy=0x7f285008a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.036+0000 7f284e59c700 1 -- 172.21.15.16:0/3796759731 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850063a10 con 0x7f2850086d70 2019-12-05T06:06:45.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f284e59c700 1 --2- 172.21.15.16:0/3796759731 >> v2:172.21.15.76:3300/0 conn(0x7f2850086d70 0x7f285008aeb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f283800af50 tx=0x7f283800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f284e59c700 1 --2- 172.21.15.16:0/3796759731 >> v2:172.21.15.76:3300/0 conn(0x7f2850086d70 0x7f285008aeb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f283800af50 tx=0x7f283800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f284e59c700 1 --2- 172.21.15.16:0/3796759731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2850086d70 0x7f285008aeb0 secure :-1 s=READY pgs=89 cs=0 l=1 rx=0x7f283800af50 tx=0x7f283800b760).ready entity=mon.1 client_cookie=844f489ad1145f6b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f284e59c700 1 --2- 172.21.15.16:0/3796759731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2850086d70 0x7f285008aeb0 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f283800af50 tx=0x7f283800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f284dd9b700 1 -- 172.21.15.16:0/3796759731 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2838000960 con 0x7f2850086d70 2019-12-05T06:06:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f284e59c700 1 --2- 172.21.15.16:0/3796759731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2850086d70 0x7f285008aeb0 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f283800af50 tx=0x7f283800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f284e59c700 1 --2- 172.21.15.16:0/3796759731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2850086d70 0x7f285008aeb0 secure :-1 s=THROTTLE_DONE pgs=89 cs=0 l=1 rx=0x7f283800af50 tx=0x7f283800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f284dd9b700 1 -- 172.21.15.16:0/3796759731 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2838009860 con 0x7f2850086d70 2019-12-05T06:06:45.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f284dd9b700 1 -- 172.21.15.16:0/3796759731 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2838000960 con 0x7f2850086d70 2019-12-05T06:06:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f2854fb2700 1 -- 172.21.15.16:0/3796759731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2850086d70 msgr2=0x7f285008aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.037+0000 7f2854fb2700 1 --2- 172.21.15.16:0/3796759731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2850086d70 0x7f285008aeb0 secure :-1 s=READY pgs=89 cs=0 l=1 rx=0x7f283800af50 tx=0x7f283800b760).stop 2019-12-05T06:06:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.039+0000 7f2854fb2700 1 -- 172.21.15.16:0/3796759731 shutdown_connections 2019-12-05T06:06:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.039+0000 7f2854fb2700 1 --2- 172.21.15.16:0/3796759731 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2850086d70 0x7f285008aeb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.039+0000 7f2854fb2700 1 -- 172.21.15.16:0/3796759731 shutdown_connections 2019-12-05T06:06:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.039+0000 7f2854fb2700 1 -- 172.21.15.16:0/3796759731 wait complete. 2019-12-05T06:06:45.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.039+0000 7f2854fb2700 1 -- 172.21.15.16:0/3796759731 >> 172.21.15.16:0/3796759731 conn(0x7f285005f5a0 msgr2=0x7f285009cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:45.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.043+0000 7f2854fb2700 1 Processor -- start 2019-12-05T06:06:45.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.043+0000 7f2854fb2700 1 -- start start 2019-12-05T06:06:45.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.044+0000 7f2854fb2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f285014d580 0x7f285014d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:45.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.044+0000 7f2854fb2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f28501346f0 0x7f28501572d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:45.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.044+0000 7f2854fb2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f285009a2c0 con 0x7f2850134ec0 2019-12-05T06:06:45.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.044+0000 7f2854fb2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2850087c80 con 0x7f285014d580 2019-12-05T06:06:45.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.044+0000 7f2854fb2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f285009bee0 con 0x7f28501346f0 2019-12-05T06:06:45.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f284ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f285014d580 0x7f285014d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:45.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f284e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2850134ec0 0x7f2850133ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59552/0 (socket says 172.21.15.16:59552) 2019-12-05T06:06:45.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f284e59c700 1 -- 172.21.15.16:0/1706261895 learned_addr learned my addr 172.21.15.16:0/1706261895 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:45.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f284f59e700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f28501346f0 0x7f28501572d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:45.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f28477fe700 1 -- 172.21.15.16:0/1706261895 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3993356032 0 0) 0x7f283800b3d0 con 0x7f2850134ec0 2019-12-05T06:06:45.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f28477fe700 1 -- 172.21.15.16:0/1706261895 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2834002e80 con 0x7f2850134ec0 2019-12-05T06:06:45.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f284ed9d700 1 -- 172.21.15.16:0/1706261895 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f28501346f0 msgr2=0x7f28501572d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f28501346f0 0x7f28501572d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f284ed9d700 1 -- 172.21.15.16:0/1706261895 >> v1:172.21.15.16:6789/0 conn(0x7f2850134ec0 legacy=0x7f2850133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f284ed9d700 1 -- 172.21.15.16:0/1706261895 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28501335f0 con 0x7f285014d580 2019-12-05T06:06:45.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.045+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> v2:172.21.15.16:3300/0 conn(0x7f285014d580 0x7f285014d970 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.046+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> v2:172.21.15.16:3300/0 conn(0x7f285014d580 0x7f285014d970 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.046+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 0x7f285014d970 secure :-1 s=READY pgs=99 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:45.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.046+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 0x7f285014d970 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.046+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 0x7f285014d970 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.046+0000 7f28477fe700 1 -- 172.21.15.16:0/1706261895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2848001100 con 0x7f285014d580 2019-12-05T06:06:45.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.046+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 0x7f285014d970 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.046+0000 7f28477fe700 1 -- 172.21.15.16:0/1706261895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2848001700 con 0x7f285014d580 2019-12-05T06:06:45.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.046+0000 7f2854fb2700 1 -- 172.21.15.16:0/1706261895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28501335f0 con 0x7f285014d580 2019-12-05T06:06:45.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.046+0000 7f28477fe700 1 -- 172.21.15.16:0/1706261895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2848001d90 con 0x7f285014d580 2019-12-05T06:06:45.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.046+0000 7f2854fb2700 1 -- 172.21.15.16:0/1706261895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28501335f0 con 0x7f285014d580 2019-12-05T06:06:45.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.047+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 0x7f285014d970 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.048+0000 7f28477fe700 1 -- 172.21.15.16:0/1706261895 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2848001c20 con 0x7f285014d580 2019-12-05T06:06:45.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.048+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 0x7f285014d970 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.048+0000 7f28477fe700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2834023670 0x7f2834025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.049+0000 7f28477fe700 1 -- 172.21.15.16:0/1706261895 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f28480460c0 con 0x7f285014d580 2019-12-05T06:06:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.049+0000 7f284e59c700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2834023670 0x7f2834025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.049+0000 7f284e59c700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2834023670 0x7f2834025ae0 crc :-1 s=READY pgs=60 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:45.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.050+0000 7f2854fb2700 1 -- 172.21.15.16:0/1706261895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28501335f0 con 0x7f285014d580 2019-12-05T06:06:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.057+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 0x7f285014d970 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.057+0000 7f28477fe700 1 -- 172.21.15.16:0/1706261895 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f284802ae10 con 0x7f285014d580 2019-12-05T06:06:45.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.271+0000 7f2854fb2700 1 -- 172.21.15.16:0/1706261895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f28501335f0 con 0x7f285014d580 2019-12-05T06:06:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.272+0000 7f284ed9d700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 0x7f285014d970 secure :-1 s=THROTTLE_DONE pgs=99 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.272+0000 7f28477fe700 1 -- 172.21.15.16:0/1706261895 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f284802ae10 con 0x7f285014d580 2019-12-05T06:06:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.274+0000 7f2854fb2700 1 -- 172.21.15.16:0/1706261895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2834023670 msgr2=0x7f2834025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.274+0000 7f2854fb2700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2834023670 0x7f2834025ae0 crc :-1 s=READY pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.274+0000 7f2854fb2700 1 -- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 msgr2=0x7f285014d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.274+0000 7f2854fb2700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 0x7f285014d970 secure :-1 s=READY pgs=99 cs=0 l=1 rx=0x7f2848010700 tx=0x7f2848010530).stop 2019-12-05T06:06:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.275+0000 7f2854fb2700 1 -- 172.21.15.16:0/1706261895 shutdown_connections 2019-12-05T06:06:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.275+0000 7f2854fb2700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f28501346f0 0x7f28501572d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.275+0000 7f2854fb2700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2834023670 0x7f2834025ae0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.275+0000 7f2854fb2700 1 --2- 172.21.15.16:0/1706261895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f285014d580 0x7f285014d970 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.275+0000 7f2854fb2700 1 -- 172.21.15.16:0/1706261895 shutdown_connections 2019-12-05T06:06:45.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.275+0000 7f2854fb2700 1 -- 172.21.15.16:0/1706261895 wait complete. 2019-12-05T06:06:45.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.275+0000 7f2854fb2700 1 -- 172.21.15.16:0/1706261895 >> 172.21.15.16:0/1706261895 conn(0x7f2850088790 msgr2=0x7f2850088c20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:45.284 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:06:45.287 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T06:06:45.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.521+0000 7fb2f0c66700 1 Processor -- start 2019-12-05T06:06:45.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.521+0000 7fb2f0c66700 1 -- start start 2019-12-05T06:06:45.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.523+0000 7fb2f0c66700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec086980 0x7fb2ec086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:45.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.523+0000 7fb2f0c66700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:45.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.523+0000 7fb2f0c66700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec061a10 con 0x7fb2ec08b080 2019-12-05T06:06:45.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.524+0000 7fb2f0c66700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2ec060bc0 con 0x7fb2ec08a6b0 2019-12-05T06:06:45.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.524+0000 7fb2f0c66700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec060620 con 0x7fb2ec086980 2019-12-05T06:06:45.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.524+0000 7fb2ea59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:45.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.524+0000 7fb2ea59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46162/0 (socket says 172.21.15.16:46162) 2019-12-05T06:06:45.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.524+0000 7fb2ea59c700 1 -- 172.21.15.16:0/2273133873 learned_addr learned my addr 172.21.15.16:0/2273133873 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:45.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.524+0000 7fb2ead9d700 1 --2- 172.21.15.16:0/2273133873 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec086980 0x7fb2ec086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:45.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2ea59c700 1 -- 172.21.15.16:0/2273133873 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec086980 msgr2=0x7fb2ec086d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:45.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/2273133873 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec086980 0x7fb2ec086d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:45.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2ea59c700 1 -- 172.21.15.16:0/2273133873 >> v1:172.21.15.76:6789/0 conn(0x7fb2ec08b080 legacy=0x7fb2ec064310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2ea59c700 1 -- 172.21.15.16:0/2273133873 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec063a10 con 0x7fb2ec08a6b0 2019-12-05T06:06:45.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2e9d9b700 1 -- 172.21.15.16:0/2273133873 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3504953028 0 0) 0x7fb2e40009b0 con 0x7fb2ec08b080 2019-12-05T06:06:45.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/2273133873 >> v2:172.21.15.16:3300/0 conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb2d400fa40 tx=0x7fb2d400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/2273133873 >> v2:172.21.15.16:3300/0 conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb2d400fa40 tx=0x7fb2d400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/2273133873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7fb2d400fa40 tx=0x7fb2d400fde0).ready entity=mon.0 client_cookie=8b6dfc7115fb263a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:45.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/2273133873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7fb2d400fa40 tx=0x7fb2d400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2e9d9b700 1 -- 172.21.15.16:0/2273133873 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d4000ce0 con 0x7fb2ec08a6b0 2019-12-05T06:06:45.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/2273133873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7fb2d400fa40 tx=0x7fb2d400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.525+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/2273133873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7fb2d400fa40 tx=0x7fb2d400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.526+0000 7fb2e9d9b700 1 -- 172.21.15.16:0/2273133873 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d4001010 con 0x7fb2ec08a6b0 2019-12-05T06:06:45.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.526+0000 7fb2e9d9b700 1 -- 172.21.15.16:0/2273133873 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d4001720 con 0x7fb2ec08a6b0 2019-12-05T06:06:45.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.526+0000 7fb2f0c66700 1 -- 172.21.15.16:0/2273133873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2ec08a6b0 msgr2=0x7fb2ec08ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.526+0000 7fb2f0c66700 1 --2- 172.21.15.16:0/2273133873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7fb2d400fa40 tx=0x7fb2d400fde0).stop 2019-12-05T06:06:45.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.528+0000 7fb2f0c66700 1 -- 172.21.15.16:0/2273133873 shutdown_connections 2019-12-05T06:06:45.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.528+0000 7fb2f0c66700 1 --2- 172.21.15.16:0/2273133873 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2ec08a6b0 0x7fb2ec08ab40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.528+0000 7fb2f0c66700 1 --2- 172.21.15.16:0/2273133873 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec086980 0x7fb2ec086d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.528+0000 7fb2f0c66700 1 -- 172.21.15.16:0/2273133873 shutdown_connections 2019-12-05T06:06:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.528+0000 7fb2f0c66700 1 -- 172.21.15.16:0/2273133873 wait complete. 2019-12-05T06:06:45.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.528+0000 7fb2f0c66700 1 -- 172.21.15.16:0/2273133873 >> 172.21.15.16:0/2273133873 conn(0x7fb2ec05f5a0 msgr2=0x7fb2ec09cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:45.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.534+0000 7fb2f0c66700 1 Processor -- start 2019-12-05T06:06:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.534+0000 7fb2f0c66700 1 -- start start 2019-12-05T06:06:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.536+0000 7fb2f0c66700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb2ec073d80 0x7fb2ec095460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.536+0000 7fb2f0c66700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:45.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.536+0000 7fb2f0c66700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2ec074c00 con 0x7fb2ec087680 2019-12-05T06:06:45.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.536+0000 7fb2f0c66700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2ec075e90 con 0x7fb2ec073d80 2019-12-05T06:06:45.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.536+0000 7fb2f0c66700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ec076160 con 0x7fb2ec099990 2019-12-05T06:06:45.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.536+0000 7fb2eb59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb2ec087680 0x7fb2ec087ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59568/0 (socket says 172.21.15.16:59568) 2019-12-05T06:06:45.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.536+0000 7fb2eb59e700 1 -- 172.21.15.16:0/502066174 learned_addr learned my addr 172.21.15.16:0/502066174 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:45.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.536+0000 7fb2ead9d700 1 --2- 172.21.15.16:0/502066174 >> v2:172.21.15.76:3300/0 conn(0x7fb2ec073d80 0x7fb2ec095460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:45.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.536+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:45.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.537+0000 7fb2e37fe700 1 -- 172.21.15.16:0/502066174 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303363289 0 0) 0x7fb2e4001240 con 0x7fb2ec087680 2019-12-05T06:06:45.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.537+0000 7fb2e37fe700 1 -- 172.21.15.16:0/502066174 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2d0002e80 con 0x7fb2ec087680 2019-12-05T06:06:45.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.537+0000 7fb2ea59c700 1 -- 172.21.15.16:0/502066174 >> v2:172.21.15.76:3300/0 conn(0x7fb2ec073d80 msgr2=0x7fb2ec095460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.537+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> v2:172.21.15.76:3300/0 conn(0x7fb2ec073d80 0x7fb2ec095460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.537+0000 7fb2ea59c700 1 -- 172.21.15.16:0/502066174 >> v1:172.21.15.16:6789/0 conn(0x7fb2ec087680 legacy=0x7fb2ec087ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.537+0000 7fb2ea59c700 1 -- 172.21.15.16:0/502066174 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ec094b60 con 0x7fb2ec099990 2019-12-05T06:06:45.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.537+0000 7fb2ead9d700 1 --2- 172.21.15.16:0/502066174 >> v2:172.21.15.76:3300/0 conn(0x7fb2ec073d80 0x7fb2ec095460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:45.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.537+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.537+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.537+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=READY pgs=29 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.538+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.538+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.538+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.538+0000 7fb2e37fe700 1 -- 172.21.15.16:0/502066174 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d40058e0 con 0x7fb2ec099990 2019-12-05T06:06:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.538+0000 7fb2e37fe700 1 -- 172.21.15.16:0/502066174 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d4005500 con 0x7fb2ec099990 2019-12-05T06:06:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.538+0000 7fb2e37fe700 1 -- 172.21.15.16:0/502066174 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d4005b60 con 0x7fb2ec099990 2019-12-05T06:06:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.538+0000 7fb2f0c66700 1 -- 172.21.15.16:0/502066174 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec094b60 con 0x7fb2ec099990 2019-12-05T06:06:45.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.538+0000 7fb2f0c66700 1 -- 172.21.15.16:0/502066174 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ec094b60 con 0x7fb2ec099990 2019-12-05T06:06:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.539+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.540+0000 7fb2e37fe700 1 -- 172.21.15.16:0/502066174 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb2d40056f0 con 0x7fb2ec099990 2019-12-05T06:06:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.540+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.540+0000 7fb2e37fe700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2d0023670 0x7fb2d0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.541+0000 7fb2e37fe700 1 -- 172.21.15.16:0/502066174 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fb2d40460d0 con 0x7fb2ec099990 2019-12-05T06:06:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.541+0000 7fb2ead9d700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2d0023670 0x7fb2d0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.541+0000 7fb2ead9d700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2d0023670 0x7fb2d0025ae0 crc :-1 s=READY pgs=61 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:45.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.553+0000 7fb2f0c66700 1 -- 172.21.15.16:0/502066174 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ec094b60 con 0x7fb2ec099990 2019-12-05T06:06:45.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.563+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.563+0000 7fb2e37fe700 1 -- 172.21.15.16:0/502066174 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb2d4007690 con 0x7fb2ec099990 2019-12-05T06:06:45.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.817+0000 7fb2f0c66700 1 -- 172.21.15.16:0/502066174 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fb2ec094b60 con 0x7fb2ec099990 2019-12-05T06:06:45.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.818+0000 7fb2ea59c700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=THROTTLE_DONE pgs=29 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:45.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.818+0000 7fb2e37fe700 1 -- 172.21.15.16:0/502066174 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb2d4007690 con 0x7fb2ec099990 2019-12-05T06:06:45.819 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:06:45.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.824+0000 7fb2f0c66700 1 -- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2d0023670 msgr2=0x7fb2d0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.824+0000 7fb2f0c66700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2d0023670 0x7fb2d0025ae0 crc :-1 s=READY pgs=61 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.824+0000 7fb2f0c66700 1 -- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 msgr2=0x7fb2ec0959a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:45.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.824+0000 7fb2f0c66700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 secure :-1 s=READY pgs=29 cs=0 l=1 rx=0x7fb2d4010720 tx=0x7fb2d4010ac0).stop 2019-12-05T06:06:45.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.824+0000 7fb2f0c66700 1 -- 172.21.15.16:0/502066174 shutdown_connections 2019-12-05T06:06:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.824+0000 7fb2f0c66700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2d0023670 0x7fb2d0025ae0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.824+0000 7fb2f0c66700 1 --2- 172.21.15.16:0/502066174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2ec099990 0x7fb2ec0959a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.824+0000 7fb2f0c66700 1 --2- 172.21.15.16:0/502066174 >> v2:172.21.15.76:3300/0 conn(0x7fb2ec073d80 0x7fb2ec095460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.824+0000 7fb2f0c66700 1 -- 172.21.15.16:0/502066174 shutdown_connections 2019-12-05T06:06:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.824+0000 7fb2f0c66700 1 -- 172.21.15.16:0/502066174 wait complete. 2019-12-05T06:06:45.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:45.825+0000 7fb2f0c66700 1 -- 172.21.15.16:0/502066174 >> 172.21.15.16:0/502066174 conn(0x7fb2ec077e10 msgr2=0x7fb2ec08b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:45.833 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2019-12-05T06:06:45.834 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2019-12-05T06:06:45.834 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2019-12-05T06:06:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.057+0000 7f7d6df08700 1 Processor -- start 2019-12-05T06:06:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.057+0000 7f7d6df08700 1 -- start start 2019-12-05T06:06:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.059+0000 7f7d6df08700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 0x7f7d6808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.059+0000 7f7d6df08700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7d6808f720 0x7f7d6808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:46.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.059+0000 7f7d6df08700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d680958e0 con 0x7f7d6808fef0 2019-12-05T06:06:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.059+0000 7f7d6df08700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d680a3f70 con 0x7f7d6808f720 2019-12-05T06:06:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.059+0000 7f7d6df08700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6808d830 con 0x7f7d6808c6d0 2019-12-05T06:06:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.059+0000 7f7d677fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7d6808fef0 0x7f7d6808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59572/0 (socket says 172.21.15.16:59572) 2019-12-05T06:06:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.059+0000 7f7d677fe700 1 -- 172.21.15.16:0/1167492911 learned_addr learned my addr 172.21.15.16:0/1167492911 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.059+0000 7f7d6cca6700 1 --2- 172.21.15.16:0/1167492911 >> v2:172.21.15.76:3300/0 conn(0x7f7d6808f720 0x7f7d6808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.059+0000 7f7d67fff700 1 --2- 172.21.15.16:0/1167492911 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 0x7f7d6808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.060+0000 7f7d66ffd700 1 -- 172.21.15.16:0/1167492911 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910380228 0 0) 0x7f7d50001150 con 0x7f7d6808fef0 2019-12-05T06:06:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.060+0000 7f7d66ffd700 1 -- 172.21.15.16:0/1167492911 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d54002e80 con 0x7f7d6808fef0 2019-12-05T06:06:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.060+0000 7f7d67fff700 1 -- 172.21.15.16:0/1167492911 >> v2:172.21.15.76:3300/0 conn(0x7f7d6808f720 msgr2=0x7f7d6808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.060+0000 7f7d67fff700 1 --2- 172.21.15.16:0/1167492911 >> v2:172.21.15.76:3300/0 conn(0x7f7d6808f720 0x7f7d6808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.060+0000 7f7d67fff700 1 -- 172.21.15.16:0/1167492911 >> v1:172.21.15.16:6789/0 conn(0x7f7d6808fef0 legacy=0x7f7d6808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.060+0000 7f7d67fff700 1 -- 172.21.15.16:0/1167492911 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6808fbc0 con 0x7f7d6808c6d0 2019-12-05T06:06:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.060+0000 7f7d6cca6700 1 --2- 172.21.15.16:0/1167492911 >> v2:172.21.15.76:3300/0 conn(0x7f7d6808f720 0x7f7d6808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.060+0000 7f7d67fff700 1 --2- 172.21.15.16:0/1167492911 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 0x7f7d6808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7d5800b2d0 tx=0x7f7d5800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.061+0000 7f7d67fff700 1 --2- 172.21.15.16:0/1167492911 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 0x7f7d6808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7d5800b2d0 tx=0x7f7d5800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.061+0000 7f7d67fff700 1 --2- 172.21.15.16:0/1167492911 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 0x7f7d6808cac0 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f7d5800b2d0 tx=0x7f7d5800bae0).ready entity=mon.2 client_cookie=53a7101fab8c2402 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:46.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.061+0000 7f7d67fff700 1 --2- 172.21.15.16:0/1167492911 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 0x7f7d6808cac0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f7d5800b2d0 tx=0x7f7d5800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.061+0000 7f7d67fff700 1 --2- 172.21.15.16:0/1167492911 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 0x7f7d6808cac0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f7d5800b2d0 tx=0x7f7d5800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.061+0000 7f7d66ffd700 1 -- 172.21.15.16:0/1167492911 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d58007590 con 0x7f7d6808c6d0 2019-12-05T06:06:46.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.061+0000 7f7d67fff700 1 --2- 172.21.15.16:0/1167492911 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 0x7f7d6808cac0 secure :-1 s=THROTTLE_DONE pgs=30 cs=0 l=1 rx=0x7f7d5800b2d0 tx=0x7f7d5800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.061+0000 7f7d66ffd700 1 -- 172.21.15.16:0/1167492911 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d58009d30 con 0x7f7d6808c6d0 2019-12-05T06:06:46.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.061+0000 7f7d66ffd700 1 -- 172.21.15.16:0/1167492911 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d5800a300 con 0x7f7d6808c6d0 2019-12-05T06:06:46.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.061+0000 7f7d6df08700 1 -- 172.21.15.16:0/1167492911 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 msgr2=0x7f7d6808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.061+0000 7f7d6df08700 1 --2- 172.21.15.16:0/1167492911 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 0x7f7d6808cac0 secure :-1 s=READY pgs=30 cs=0 l=1 rx=0x7f7d5800b2d0 tx=0x7f7d5800bae0).stop 2019-12-05T06:06:46.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.063+0000 7f7d6df08700 1 -- 172.21.15.16:0/1167492911 shutdown_connections 2019-12-05T06:06:46.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.063+0000 7f7d6df08700 1 --2- 172.21.15.16:0/1167492911 >> v2:172.21.15.76:3300/0 conn(0x7f7d6808f720 0x7f7d6808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:46.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.063+0000 7f7d6df08700 1 --2- 172.21.15.16:0/1167492911 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d6808c6d0 0x7f7d6808cac0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.063+0000 7f7d6df08700 1 -- 172.21.15.16:0/1167492911 shutdown_connections 2019-12-05T06:06:46.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.063+0000 7f7d6df08700 1 -- 172.21.15.16:0/1167492911 wait complete. 2019-12-05T06:06:46.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.063+0000 7f7d6df08700 1 -- 172.21.15.16:0/1167492911 >> 172.21.15.16:0/1167492911 conn(0x7f7d68061940 msgr2=0x7f7d68096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:46.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.068+0000 7f7d6df08700 1 Processor -- start 2019-12-05T06:06:46.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.068+0000 7f7d6df08700 1 -- start start 2019-12-05T06:06:46.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.070+0000 7f7d6df08700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d6806dc50 0x7f7d6806e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:46.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.070+0000 7f7d6df08700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d68134c80 0x7f7d6806e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:46.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.070+0000 7f7d6df08700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d681350d0 con 0x7f7d68135450 2019-12-05T06:06:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.070+0000 7f7d67fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d6806dc50 0x7f7d6806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.070+0000 7f7d677fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7d68135450 0x7f7d68134480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59576/0 (socket says 172.21.15.16:59576) 2019-12-05T06:06:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.070+0000 7f7d677fe700 1 -- 172.21.15.16:0/3480038131 learned_addr learned my addr 172.21.15.16:0/3480038131 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.070+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d68132830 con 0x7f7d6806dc50 2019-12-05T06:06:46.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.070+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d681334f0 con 0x7f7d68134c80 2019-12-05T06:06:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.070+0000 7f7d6cca6700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d68134c80 0x7f7d6806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.070+0000 7f7d64ff9700 1 -- 172.21.15.16:0/3480038131 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440433637 0 0) 0x7f7d50001150 con 0x7f7d68135450 2019-12-05T06:06:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d64ff9700 1 -- 172.21.15.16:0/3480038131 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c002e80 con 0x7f7d68135450 2019-12-05T06:06:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d67fff700 1 -- 172.21.15.16:0/3480038131 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d68134c80 msgr2=0x7f7d6806e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d68134c80 0x7f7d6806e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d67fff700 1 -- 172.21.15.16:0/3480038131 >> v1:172.21.15.16:6789/0 conn(0x7f7d68135450 legacy=0x7f7d68134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d67fff700 1 -- 172.21.15.16:0/3480038131 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d68133b80 con 0x7f7d6806dc50 2019-12-05T06:06:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> v2:172.21.15.16:3300/0 conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> v2:172.21.15.16:3300/0 conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:46.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d64ff9700 1 -- 172.21.15.16:0/3480038131 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d58009b10 con 0x7f7d6806dc50 2019-12-05T06:06:46.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d64ff9700 1 -- 172.21.15.16:0/3480038131 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5800af30 con 0x7f7d6806dc50 2019-12-05T06:06:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d68133b80 con 0x7f7d6806dc50 2019-12-05T06:06:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d64ff9700 1 -- 172.21.15.16:0/3480038131 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d5800b120 con 0x7f7d6806dc50 2019-12-05T06:06:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.071+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d68133b80 con 0x7f7d6806dc50 2019-12-05T06:06:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.072+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.073+0000 7f7d64ff9700 1 -- 172.21.15.16:0/3480038131 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7d58009b10 con 0x7f7d6806dc50 2019-12-05T06:06:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.073+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.074+0000 7f7d64ff9700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d4c023670 0x7f7d4c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.074+0000 7f7d64ff9700 1 -- 172.21.15.16:0/3480038131 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f7d5808a0f0 con 0x7f7d6806dc50 2019-12-05T06:06:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.074+0000 7f7d677fe700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d4c023670 0x7f7d4c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.074+0000 7f7d677fe700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d4c023670 0x7f7d4c025ae0 crc :-1 s=READY pgs=62 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:46.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.088+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d68133b80 con 0x7f7d6806dc50 2019-12-05T06:06:46.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.096+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.096+0000 7f7d64ff9700 1 -- 172.21.15.16:0/3480038131 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7d58009900 con 0x7f7d6806dc50 2019-12-05T06:06:46.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.404+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f7d68133b80 con 0x7f7d6806dc50 2019-12-05T06:06:46.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.409+0000 7f7d67fff700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.409+0000 7f7d64ff9700 1 -- 172.21.15.16:0/3480038131 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v3) v1 ==== 74+0+94021 (secure 0 0 0) 0x7f7d58009b10 con 0x7f7d6806dc50 2019-12-05T06:06:46.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.414+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d4c023670 msgr2=0x7f7d4c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.414+0000 7f7d6df08700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d4c023670 0x7f7d4c025ae0 crc :-1 s=READY pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.414+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 msgr2=0x7f7d6806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.414+0000 7f7d6df08700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 0x7f7d6806e040 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f7d58054760 tx=0x7f7d580100c0).stop 2019-12-05T06:06:46.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.414+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 shutdown_connections 2019-12-05T06:06:46.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.414+0000 7f7d6df08700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d68134c80 0x7f7d6806e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.414+0000 7f7d6df08700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d4c023670 0x7f7d4c025ae0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.414+0000 7f7d6df08700 1 --2- 172.21.15.16:0/3480038131 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d6806dc50 0x7f7d6806e040 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.414+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 shutdown_connections 2019-12-05T06:06:46.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.414+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 wait complete. 2019-12-05T06:06:46.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.415+0000 7f7d6df08700 1 -- 172.21.15.16:0/3480038131 >> 172.21.15.16:0/3480038131 conn(0x7f7d680a2850 msgr2=0x7f7d6806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:46.443 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:46.444 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":3,"active_gid":4100,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":9546},{"type":"v1","addr":"172.21.15.76:6801","nonce":9546}]},"active_addr":"172.21.15.76:6801/9546","active_change":"2019-12-05T06:06:14.752380+0000","active_mgr_features":4540138292836696063,"available":true,"standbys":[{"gid":4108,"name":"y","mgr_features":4540138292836696063,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"1","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["ceph-daemon-package","root"],"desc":"mode for remote execution of ceph-daemon","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"str","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"ansible","can_run":true,"error_string":"","module_options":{"ca_bundle":{"name":"ca_bundle","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_location":{"name":"server_location","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_server":{"name":"verify_server","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"1","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"str","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"str","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"str","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"deepsea","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_eauth":{"name":"salt_api_eauth","type":"str","level":"advanced","flags":0,"default_value":"sharedsecret","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_password":{"name":"salt_api_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_url":{"name":"salt_api_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"salt_api_username":{"name":"salt_api_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator_cli","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"ssh","can_run":true,"error_string":"","module_options":{"inventory_cache_timeout":{"name":"inventory_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["ceph-daemon-package","root"],"desc":"mode for remote execution of ceph-daemon","long_desc":"","tags":[],"see_also":[]},"service_cache_timeout":{"name":"service_cache_timeout","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator_cli","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[]}} 2019-12-05T06:06:46.450 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2019-12-05T06:06:46.451 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2019-12-05T06:06:46.451 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:06:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.724+0000 7f75f6c59700 1 Processor -- start 2019-12-05T06:06:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.725+0000 7f75f6c59700 1 -- start start 2019-12-05T06:06:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.726+0000 7f75f6c59700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75f008d2c0 0x7f75f008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.726+0000 7f75f6c59700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f75f008fef0 0x7f75f008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.726+0000 7f75f6c59700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f00958e0 con 0x7f75f008c6d0 2019-12-05T06:06:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.726+0000 7f75f6c59700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f008df90 con 0x7f75f008d2c0 2019-12-05T06:06:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.726+0000 7f75f6c59700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f00a3f70 con 0x7f75f008fef0 2019-12-05T06:06:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.727+0000 7f75f51f6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f75f008c6d0 0x7f75f008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59584/0 (socket says 172.21.15.16:59584) 2019-12-05T06:06:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.727+0000 7f75f49f5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75f008d2c0 0x7f75f008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.727+0000 7f75f51f6700 1 -- 172.21.15.16:0/1972575040 learned_addr learned my addr 172.21.15.16:0/1972575040 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.727+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1972575040 >> v2:172.21.15.76:3300/0 conn(0x7f75f008fef0 0x7f75f008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.727+0000 7f75effff700 1 -- 172.21.15.16:0/1972575040 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2881247208 0 0) 0x7f75e0001150 con 0x7f75f008c6d0 2019-12-05T06:06:46.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.727+0000 7f75effff700 1 -- 172.21.15.16:0/1972575040 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75d8002e80 con 0x7f75f008c6d0 2019-12-05T06:06:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75f49f5700 1 -- 172.21.15.16:0/1972575040 >> v2:172.21.15.76:3300/0 conn(0x7f75f008fef0 msgr2=0x7f75f008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75f49f5700 1 --2- 172.21.15.16:0/1972575040 >> v2:172.21.15.76:3300/0 conn(0x7f75f008fef0 0x7f75f008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75f49f5700 1 -- 172.21.15.16:0/1972575040 >> v1:172.21.15.16:6789/0 conn(0x7f75f008c6d0 legacy=0x7f75f008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75f49f5700 1 -- 172.21.15.16:0/1972575040 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f008e620 con 0x7f75f008d2c0 2019-12-05T06:06:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75f49f5700 1 --2- 172.21.15.16:0/1972575040 >> v2:172.21.15.16:3300/0 conn(0x7f75f008d2c0 0x7f75f008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f75e400b2d0 tx=0x7f75e400b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75f49f5700 1 --2- 172.21.15.16:0/1972575040 >> v2:172.21.15.16:3300/0 conn(0x7f75f008d2c0 0x7f75f008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f75e400b2d0 tx=0x7f75e400b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75f49f5700 1 --2- 172.21.15.16:0/1972575040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f008d2c0 0x7f75f008ef20 secure :-1 s=READY pgs=106 cs=0 l=1 rx=0x7f75e400b2d0 tx=0x7f75e400b5d0).ready entity=mon.0 client_cookie=4dba4744ca523f46 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75f49f5700 1 --2- 172.21.15.16:0/1972575040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f008d2c0 0x7f75f008ef20 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f75e400b2d0 tx=0x7f75e400b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75f49f5700 1 --2- 172.21.15.16:0/1972575040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f008d2c0 0x7f75f008ef20 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f75e400b2d0 tx=0x7f75e400b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75effff700 1 -- 172.21.15.16:0/1972575040 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e4007590 con 0x7f75f008d2c0 2019-12-05T06:06:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.728+0000 7f75f49f5700 1 --2- 172.21.15.16:0/1972575040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f008d2c0 0x7f75f008ef20 secure :-1 s=THROTTLE_DONE pgs=106 cs=0 l=1 rx=0x7f75e400b2d0 tx=0x7f75e400b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.729+0000 7f75effff700 1 -- 172.21.15.16:0/1972575040 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e4009d30 con 0x7f75f008d2c0 2019-12-05T06:06:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.729+0000 7f75effff700 1 -- 172.21.15.16:0/1972575040 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e400a400 con 0x7f75f008d2c0 2019-12-05T06:06:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.729+0000 7f75f6c59700 1 -- 172.21.15.16:0/1972575040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f008d2c0 msgr2=0x7f75f008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.729+0000 7f75f6c59700 1 --2- 172.21.15.16:0/1972575040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f008d2c0 0x7f75f008ef20 secure :-1 s=READY pgs=106 cs=0 l=1 rx=0x7f75e400b2d0 tx=0x7f75e400b5d0).stop 2019-12-05T06:06:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.730+0000 7f75f6c59700 1 -- 172.21.15.16:0/1972575040 shutdown_connections 2019-12-05T06:06:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.730+0000 7f75f6c59700 1 --2- 172.21.15.16:0/1972575040 >> v2:172.21.15.76:3300/0 conn(0x7f75f008fef0 0x7f75f008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.730+0000 7f75f6c59700 1 --2- 172.21.15.16:0/1972575040 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f008d2c0 0x7f75f008ef20 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.731+0000 7f75f6c59700 1 -- 172.21.15.16:0/1972575040 shutdown_connections 2019-12-05T06:06:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.731+0000 7f75f6c59700 1 -- 172.21.15.16:0/1972575040 wait complete. 2019-12-05T06:06:46.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.731+0000 7f75f6c59700 1 -- 172.21.15.16:0/1972575040 >> 172.21.15.16:0/1972575040 conn(0x7f75f0061940 msgr2=0x7f75f0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.736+0000 7f75f6c59700 1 Processor -- start 2019-12-05T06:06:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.737+0000 7f75f6c59700 1 -- start start 2019-12-05T06:06:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.738+0000 7f75f6c59700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f75f0096e90 0x7f75f009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.738+0000 7f75f6c59700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75f009b920 0x7f75f0097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.738+0000 7f75f6c59700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f0084b90 con 0x7f75f0084f30 2019-12-05T06:06:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.738+0000 7f75f6c59700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f009a450 con 0x7f75f009b920 2019-12-05T06:06:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.738+0000 7f75f6c59700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f0099a20 con 0x7f75f0096e90 2019-12-05T06:06:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.739+0000 7f75f59f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75f009b920 0x7f75f0097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.739+0000 7f75f59f7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75f009b920 0x7f75f0097a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46194/0 (socket says 172.21.15.16:46194) 2019-12-05T06:06:46.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.739+0000 7f75f59f7700 1 -- 172.21.15.16:0/1411950391 learned_addr learned my addr 172.21.15.16:0/1411950391 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.739+0000 7f75f51f6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f75f0096e90 0x7f75f009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.739+0000 7f75f59f7700 1 -- 172.21.15.16:0/1411950391 >> v2:172.21.15.76:3300/0 conn(0x7f75f0096e90 msgr2=0x7f75f009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.739+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> v2:172.21.15.76:3300/0 conn(0x7f75f0096e90 0x7f75f009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.739+0000 7f75f59f7700 1 -- 172.21.15.16:0/1411950391 >> v1:172.21.15.76:6789/0 conn(0x7f75f0084f30 legacy=0x7f75f0097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.739+0000 7f75f59f7700 1 -- 172.21.15.16:0/1411950391 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f009aae0 con 0x7f75f009b920 2019-12-05T06:06:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.739+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> v2:172.21.15.16:3300/0 conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.740+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> v2:172.21.15.16:3300/0 conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.740+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.740+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.740+0000 7f75edffb700 1 -- 172.21.15.16:0/1411950391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e800f090 con 0x7f75f009b920 2019-12-05T06:06:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.740+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.740+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.740+0000 7f75edffb700 1 -- 172.21.15.16:0/1411950391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e8000a70 con 0x7f75f009b920 2019-12-05T06:06:46.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.740+0000 7f75f6c59700 1 -- 172.21.15.16:0/1411950391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f009aae0 con 0x7f75f009b920 2019-12-05T06:06:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.740+0000 7f75edffb700 1 -- 172.21.15.16:0/1411950391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e800f670 con 0x7f75f009b920 2019-12-05T06:06:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.740+0000 7f75f6c59700 1 -- 172.21.15.16:0/1411950391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f009aae0 con 0x7f75f009b920 2019-12-05T06:06:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.741+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.742+0000 7f75edffb700 1 -- 172.21.15.16:0/1411950391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f75e800edf0 con 0x7f75f009b920 2019-12-05T06:06:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.742+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.742+0000 7f75edffb700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d0023630 0x7f75d0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:46.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.742+0000 7f75edffb700 1 -- 172.21.15.16:0/1411950391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f75e8046060 con 0x7f75f009b920 2019-12-05T06:06:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.743+0000 7f75f51f6700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d0023630 0x7f75d0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.743+0000 7f75f51f6700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d0023630 0x7f75d0025ac0 crc :-1 s=READY pgs=63 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:46.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.756+0000 7f75f6c59700 1 -- 172.21.15.16:0/1411950391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f009aae0 con 0x7f75f009b920 2019-12-05T06:06:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.763+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:46.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:46.763+0000 7f75edffb700 1 -- 172.21.15.16:0/1411950391 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f75e800eaa0 con 0x7f75f009b920 2019-12-05T06:06:47.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.011+0000 7f75f6c59700 1 -- 172.21.15.16:0/1411950391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75f009aae0 con 0x7f75f009b920 2019-12-05T06:06:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.014+0000 7f75f59f7700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.014+0000 7f75edffb700 1 -- 172.21.15.16:0/1411950391 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f75e8000a70 con 0x7f75f009b920 2019-12-05T06:06:47.013 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:47.013 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:06:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.017+0000 7f75f6c59700 1 -- 172.21.15.16:0/1411950391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d0023630 msgr2=0x7f75d0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.017+0000 7f75f6c59700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d0023630 0x7f75d0025ac0 crc :-1 s=READY pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.017+0000 7f75f6c59700 1 -- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 msgr2=0x7f75f0097a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.017+0000 7f75f6c59700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 0x7f75f0097a80 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7f75e8010450 tx=0x7f75e8010280).stop 2019-12-05T06:06:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.017+0000 7f75f6c59700 1 -- 172.21.15.16:0/1411950391 shutdown_connections 2019-12-05T06:06:47.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.018+0000 7f75f6c59700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d0023630 0x7f75d0025ac0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.018+0000 7f75f6c59700 1 --2- 172.21.15.16:0/1411950391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f009b920 0x7f75f0097a80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.018+0000 7f75f6c59700 1 --2- 172.21.15.16:0/1411950391 >> v2:172.21.15.76:3300/0 conn(0x7f75f0096e90 0x7f75f009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.018+0000 7f75f6c59700 1 -- 172.21.15.16:0/1411950391 shutdown_connections 2019-12-05T06:06:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.018+0000 7f75f6c59700 1 -- 172.21.15.16:0/1411950391 wait complete. 2019-12-05T06:06:47.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.018+0000 7f75f6c59700 1 -- 172.21.15.16:0/1411950391 >> 172.21.15.16:0/1411950391 conn(0x7f75f00714f0 msgr2=0x7f75f009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:47.034 INFO:tasks.ceph.ceph_manager.ceph:all up! 2019-12-05T06:06:47.034 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:06:47.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.311+0000 7fdde92cf700 1 Processor -- start 2019-12-05T06:06:47.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.312+0000 7fdde92cf700 1 -- start start 2019-12-05T06:06:47.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.313+0000 7fdde92cf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdde408c6d0 0x7fdde408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:47.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.313+0000 7fdde92cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde408fef0 0x7fdde408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:47.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.313+0000 7fdde92cf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde408fb80 con 0x7fdde408ef20 2019-12-05T06:06:47.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.313+0000 7fdde92cf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde408df90 con 0x7fdde408fef0 2019-12-05T06:06:47.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.313+0000 7fdde92cf700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde40a3f70 con 0x7fdde408c6d0 2019-12-05T06:06:47.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.314+0000 7fdde2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde408fef0 0x7fdde408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.314+0000 7fdde3fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdde408ef20 0x7fdde408f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59600/0 (socket says 172.21.15.16:59600) 2019-12-05T06:06:47.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.314+0000 7fdde3fff700 1 -- 172.21.15.16:0/371489519 learned_addr learned my addr 172.21.15.16:0/371489519 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:47.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.314+0000 7fdde37fe700 1 --2- 172.21.15.16:0/371489519 >> v2:172.21.15.76:3300/0 conn(0x7fdde408c6d0 0x7fdde408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.314+0000 7fdde27fc700 1 -- 172.21.15.16:0/371489519 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 932942973 0 0) 0x7fddd4001150 con 0x7fdde408ef20 2019-12-05T06:06:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.314+0000 7fdde27fc700 1 -- 172.21.15.16:0/371489519 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd0002e80 con 0x7fdde408ef20 2019-12-05T06:06:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.314+0000 7fdde2ffd700 1 -- 172.21.15.16:0/371489519 >> v2:172.21.15.76:3300/0 conn(0x7fdde408c6d0 msgr2=0x7fdde408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.314+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/371489519 >> v2:172.21.15.76:3300/0 conn(0x7fdde408c6d0 0x7fdde408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.314+0000 7fdde2ffd700 1 -- 172.21.15.16:0/371489519 >> v1:172.21.15.16:6789/0 conn(0x7fdde408ef20 legacy=0x7fdde408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.315+0000 7fdde2ffd700 1 -- 172.21.15.16:0/371489519 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde408d540 con 0x7fdde408fef0 2019-12-05T06:06:47.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.315+0000 7fdde37fe700 1 --2- 172.21.15.16:0/371489519 >> v2:172.21.15.76:3300/0 conn(0x7fdde408c6d0 0x7fdde408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.315+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/371489519 >> v2:172.21.15.16:3300/0 conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fddcc00b2d0 tx=0x7fddcc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.315+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/371489519 >> v2:172.21.15.16:3300/0 conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fddcc00b2d0 tx=0x7fddcc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.315+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/371489519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7fddcc00b2d0 tx=0x7fddcc00bae0).ready entity=mon.0 client_cookie=82b99ed7c5f2826d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.315+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/371489519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fddcc00b2d0 tx=0x7fddcc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.315+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/371489519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fddcc00b2d0 tx=0x7fddcc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.315+0000 7fdde27fc700 1 -- 172.21.15.16:0/371489519 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddcc007590 con 0x7fdde408fef0 2019-12-05T06:06:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.315+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/371489519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=THROTTLE_DONE pgs=109 cs=0 l=1 rx=0x7fddcc00b2d0 tx=0x7fddcc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.316+0000 7fdde27fc700 1 -- 172.21.15.16:0/371489519 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc009d30 con 0x7fdde408fef0 2019-12-05T06:06:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.316+0000 7fdde27fc700 1 -- 172.21.15.16:0/371489519 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddcc00a400 con 0x7fdde408fef0 2019-12-05T06:06:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.316+0000 7fdde92cf700 1 -- 172.21.15.16:0/371489519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 msgr2=0x7fdde408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.316+0000 7fdde92cf700 1 --2- 172.21.15.16:0/371489519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=READY pgs=109 cs=0 l=1 rx=0x7fddcc00b2d0 tx=0x7fddcc00bae0).stop 2019-12-05T06:06:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.317+0000 7fdde92cf700 1 -- 172.21.15.16:0/371489519 shutdown_connections 2019-12-05T06:06:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.317+0000 7fdde92cf700 1 --2- 172.21.15.16:0/371489519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.317+0000 7fdde92cf700 1 --2- 172.21.15.16:0/371489519 >> v2:172.21.15.76:3300/0 conn(0x7fdde408c6d0 0x7fdde408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.318+0000 7fdde92cf700 1 -- 172.21.15.16:0/371489519 shutdown_connections 2019-12-05T06:06:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.318+0000 7fdde92cf700 1 -- 172.21.15.16:0/371489519 wait complete. 2019-12-05T06:06:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.318+0000 7fdde92cf700 1 -- 172.21.15.16:0/371489519 >> 172.21.15.16:0/371489519 conn(0x7fdde4061940 msgr2=0x7fdde4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.322+0000 7fdde92cf700 1 Processor -- start 2019-12-05T06:06:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.323+0000 7fdde92cf700 1 -- start start 2019-12-05T06:06:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.324+0000 7fdde92cf700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde406df10 0x7fdde4098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.324+0000 7fdde92cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde4156cb0 0x7fdde41590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.324+0000 7fdde92cf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde4089fb0 con 0x7fdde40992d0 2019-12-05T06:06:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.324+0000 7fdde92cf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde408ad00 con 0x7fdde4156cb0 2019-12-05T06:06:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.324+0000 7fdde92cf700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde4096c10 con 0x7fdde406df10 2019-12-05T06:06:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.325+0000 7fdde3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde4156cb0 0x7fdde41590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.325+0000 7fdde3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde4156cb0 0x7fdde41590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46208/0 (socket says 172.21.15.16:46208) 2019-12-05T06:06:47.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.325+0000 7fdde3fff700 1 -- 172.21.15.16:0/300690021 learned_addr learned my addr 172.21.15.16:0/300690021 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.325+0000 7fdde37fe700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde406df10 0x7fdde4098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.325+0000 7fdddbfff700 1 -- 172.21.15.16:0/300690021 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216913492 0 0) 0x7fddcc00af30 con 0x7fdde40992d0 2019-12-05T06:06:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.325+0000 7fdddbfff700 1 -- 172.21.15.16:0/300690021 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc8002e80 con 0x7fdde40992d0 2019-12-05T06:06:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.325+0000 7fdde3fff700 1 -- 172.21.15.16:0/300690021 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde406df10 msgr2=0x7fdde4098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.325+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde406df10 0x7fdde4098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.325+0000 7fdde3fff700 1 -- 172.21.15.16:0/300690021 >> v1:172.21.15.76:6789/0 conn(0x7fdde40992d0 legacy=0x7fdde4098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.325+0000 7fdde3fff700 1 -- 172.21.15.16:0/300690021 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4097a00 con 0x7fdde4156cb0 2019-12-05T06:06:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> v2:172.21.15.16:3300/0 conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> v2:172.21.15.16:3300/0 conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdddbfff700 1 -- 172.21.15.16:0/300690021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddd4000ef0 con 0x7fdde4156cb0 2019-12-05T06:06:47.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdddbfff700 1 -- 172.21.15.16:0/300690021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd4004430 con 0x7fdde4156cb0 2019-12-05T06:06:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdde92cf700 1 -- 172.21.15.16:0/300690021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde4097a00 con 0x7fdde4156cb0 2019-12-05T06:06:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdddbfff700 1 -- 172.21.15.16:0/300690021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddd4000ef0 con 0x7fdde4156cb0 2019-12-05T06:06:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.326+0000 7fdde92cf700 1 -- 172.21.15.16:0/300690021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde4097a00 con 0x7fdde4156cb0 2019-12-05T06:06:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.328+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.329+0000 7fdddbfff700 1 -- 172.21.15.16:0/300690021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fddd4004620 con 0x7fdde4156cb0 2019-12-05T06:06:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.329+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.329+0000 7fdddbfff700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023670 0x7fddc8025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.329+0000 7fdddbfff700 1 -- 172.21.15.16:0/300690021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fddd4045e30 con 0x7fdde4156cb0 2019-12-05T06:06:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.329+0000 7fdde37fe700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023670 0x7fddc8025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.330+0000 7fdde37fe700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023670 0x7fddc8025ae0 crc :-1 s=READY pgs=64 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:47.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.343+0000 7fdde92cf700 1 -- 172.21.15.16:0/300690021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde4097a00 con 0x7fdde4156cb0 2019-12-05T06:06:47.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.352+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.352+0000 7fdddbfff700 1 -- 172.21.15.16:0/300690021 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fddd40012e0 con 0x7fdde4156cb0 2019-12-05T06:06:47.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.614+0000 7fdde92cf700 1 -- 172.21.15.16:0/300690021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdde4097a00 con 0x7fdde4156cb0 2019-12-05T06:06:47.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.616+0000 7fdde3fff700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.616+0000 7fdddbfff700 1 -- 172.21.15.16:0/300690021 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7fddd4004620 con 0x7fdde4156cb0 2019-12-05T06:06:47.615 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:47.615 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:06:47.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.619+0000 7fdde92cf700 1 -- 172.21.15.16:0/300690021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023670 msgr2=0x7fddc8025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.619+0000 7fdde92cf700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023670 0x7fddc8025ae0 crc :-1 s=READY pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.619+0000 7fdde92cf700 1 -- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 msgr2=0x7fdde41590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.619+0000 7fdde92cf700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7fddd4010430 tx=0x7fddd4010260).stop 2019-12-05T06:06:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.619+0000 7fdde92cf700 1 -- 172.21.15.16:0/300690021 shutdown_connections 2019-12-05T06:06:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.619+0000 7fdde92cf700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023670 0x7fddc8025ae0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.619+0000 7fdde92cf700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde4156cb0 0x7fdde41590b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.619+0000 7fdde92cf700 1 --2- 172.21.15.16:0/300690021 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde406df10 0x7fdde4098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.620+0000 7fdde92cf700 1 -- 172.21.15.16:0/300690021 shutdown_connections 2019-12-05T06:06:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.620+0000 7fdde92cf700 1 -- 172.21.15.16:0/300690021 wait complete. 2019-12-05T06:06:47.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.620+0000 7fdde92cf700 1 -- 172.21.15.16:0/300690021 >> 172.21.15.16:0/300690021 conn(0x7fdde40714f0 msgr2=0x7fdde409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:47.629 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2019-12-05T06:06:47.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.864+0000 7f4ea77a6700 1 Processor -- start 2019-12-05T06:06:47.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.864+0000 7f4ea77a6700 1 -- start start 2019-12-05T06:06:47.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.866+0000 7f4ea77a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ea008c6d0 0x7f4ea008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.866+0000 7f4ea77a6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4ea008fef0 0x7f4ea008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.866+0000 7f4ea77a6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea008fb80 con 0x7f4ea008ef20 2019-12-05T06:06:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.866+0000 7f4ea77a6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea008df90 con 0x7f4ea008c6d0 2019-12-05T06:06:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.866+0000 7f4ea77a6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea00a3f70 con 0x7f4ea008fef0 2019-12-05T06:06:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.866+0000 7f4ea5d43700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ea008c6d0 0x7f4ea008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.866+0000 7f4ea5d43700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ea008c6d0 0x7f4ea008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46216/0 (socket says 172.21.15.16:46216) 2019-12-05T06:06:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.866+0000 7f4ea5d43700 1 -- 172.21.15.16:0/3667167710 learned_addr learned my addr 172.21.15.16:0/3667167710 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.867+0000 7f4ea5542700 1 --2- 172.21.15.16:0/3667167710 >> v2:172.21.15.76:3300/0 conn(0x7f4ea008fef0 0x7f4ea008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.867+0000 7f4ea5d43700 1 -- 172.21.15.16:0/3667167710 >> v2:172.21.15.76:3300/0 conn(0x7f4ea008fef0 msgr2=0x7f4ea008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.867+0000 7f4ea5d43700 1 --2- 172.21.15.16:0/3667167710 >> v2:172.21.15.76:3300/0 conn(0x7f4ea008fef0 0x7f4ea008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:47.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.867+0000 7f4ea5d43700 1 -- 172.21.15.16:0/3667167710 >> v1:172.21.15.76:6789/0 conn(0x7f4ea008ef20 legacy=0x7f4ea008f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.867+0000 7f4ea5d43700 1 -- 172.21.15.16:0/3667167710 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea008d540 con 0x7f4ea008c6d0 2019-12-05T06:06:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.867+0000 7f4ea5d43700 1 --2- 172.21.15.16:0/3667167710 >> v2:172.21.15.16:3300/0 conn(0x7f4ea008c6d0 0x7f4ea008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4e9800f740 tx=0x7f4e9800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.867+0000 7f4ea5d43700 1 --2- 172.21.15.16:0/3667167710 >> v2:172.21.15.16:3300/0 conn(0x7f4ea008c6d0 0x7f4ea008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4e9800f740 tx=0x7f4e9800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.867+0000 7f4ea5d43700 1 --2- 172.21.15.16:0/3667167710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea008c6d0 0x7f4ea008cac0 secure :-1 s=READY pgs=111 cs=0 l=1 rx=0x7f4e9800f740 tx=0x7f4e9800f570).ready entity=mon.0 client_cookie=750293a4244d59ac server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.868+0000 7f4ea5d43700 1 --2- 172.21.15.16:0/3667167710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea008c6d0 0x7f4ea008cac0 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f4e9800f740 tx=0x7f4e9800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.868+0000 7f4ea5d43700 1 --2- 172.21.15.16:0/3667167710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea008c6d0 0x7f4ea008cac0 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f4e9800f740 tx=0x7f4e9800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.868+0000 7f4ea4d41700 1 -- 172.21.15.16:0/3667167710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e98000960 con 0x7f4ea008c6d0 2019-12-05T06:06:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.868+0000 7f4ea5d43700 1 --2- 172.21.15.16:0/3667167710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea008c6d0 0x7f4ea008cac0 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f4e9800f740 tx=0x7f4e9800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.868+0000 7f4ea4d41700 1 -- 172.21.15.16:0/3667167710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e98000c90 con 0x7f4ea008c6d0 2019-12-05T06:06:47.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.868+0000 7f4ea4d41700 1 -- 172.21.15.16:0/3667167710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e980013a0 con 0x7f4ea008c6d0 2019-12-05T06:06:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.868+0000 7f4ea77a6700 1 -- 172.21.15.16:0/3667167710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea008c6d0 msgr2=0x7f4ea008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.868+0000 7f4ea77a6700 1 --2- 172.21.15.16:0/3667167710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea008c6d0 0x7f4ea008cac0 secure :-1 s=READY pgs=111 cs=0 l=1 rx=0x7f4e9800f740 tx=0x7f4e9800f570).stop 2019-12-05T06:06:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.869+0000 7f4ea77a6700 1 -- 172.21.15.16:0/3667167710 shutdown_connections 2019-12-05T06:06:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.869+0000 7f4ea77a6700 1 --2- 172.21.15.16:0/3667167710 >> v2:172.21.15.76:3300/0 conn(0x7f4ea008fef0 0x7f4ea008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.869+0000 7f4ea77a6700 1 --2- 172.21.15.16:0/3667167710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea008c6d0 0x7f4ea008cac0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.869+0000 7f4ea77a6700 1 -- 172.21.15.16:0/3667167710 shutdown_connections 2019-12-05T06:06:47.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.870+0000 7f4ea77a6700 1 -- 172.21.15.16:0/3667167710 wait complete. 2019-12-05T06:06:47.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.870+0000 7f4ea77a6700 1 -- 172.21.15.16:0/3667167710 >> 172.21.15.16:0/3667167710 conn(0x7f4ea0061940 msgr2=0x7f4ea0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:47.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.875+0000 7f4ea77a6700 1 Processor -- start 2019-12-05T06:06:47.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.875+0000 7f4ea77a6700 1 -- start start 2019-12-05T06:06:47.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.876+0000 7f4ea77a6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ea0132630 0x7f4ea0132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:47.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.876+0000 7f4ea77a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ea0099eb0 0x7f4ea009a330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:47.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.876+0000 7f4ea77a6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ea006e7c0 con 0x7f4ea009ae80 2019-12-05T06:06:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.876+0000 7f4ea77a6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ea0134390 con 0x7f4ea0099eb0 2019-12-05T06:06:47.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.876+0000 7f4ea77a6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ea0134660 con 0x7f4ea0132630 2019-12-05T06:06:47.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.876+0000 7f4ea5542700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4ea009ae80 0x7f4ea0132f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59620/0 (socket says 172.21.15.16:59620) 2019-12-05T06:06:47.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.876+0000 7f4ea5542700 1 -- 172.21.15.16:0/2223399107 learned_addr learned my addr 172.21.15.16:0/2223399107 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:47.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.876+0000 7f4ea6544700 1 --2- 172.21.15.16:0/2223399107 >> v2:172.21.15.16:3300/0 conn(0x7f4ea0099eb0 0x7f4ea009a330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.877+0000 7f4ea5d43700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ea0132630 0x7f4ea0132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.877+0000 7f4e9e7fc700 1 -- 172.21.15.16:0/2223399107 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 582373143 0 0) 0x7f4e940010b0 con 0x7f4ea009ae80 2019-12-05T06:06:47.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.877+0000 7f4e9e7fc700 1 -- 172.21.15.16:0/2223399107 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e80002e80 con 0x7f4ea009ae80 2019-12-05T06:06:47.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.877+0000 7f4ea6544700 1 -- 172.21.15.16:0/2223399107 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ea0132630 msgr2=0x7f4ea0132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.877+0000 7f4ea6544700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ea0132630 0x7f4ea0132a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.877+0000 7f4ea6544700 1 -- 172.21.15.16:0/2223399107 >> v1:172.21.15.16:6789/0 conn(0x7f4ea009ae80 legacy=0x7f4ea0132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.877+0000 7f4ea6544700 1 -- 172.21.15.16:0/2223399107 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea00995b0 con 0x7f4ea0099eb0 2019-12-05T06:06:47.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.877+0000 7f4ea6544700 1 --2- 172.21.15.16:0/2223399107 >> v2:172.21.15.16:3300/0 conn(0x7f4ea0099eb0 0x7f4ea009a330 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4e90010730 tx=0x7f4e9000b670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.877+0000 7f4ea6544700 1 --2- 172.21.15.16:0/2223399107 >> v2:172.21.15.16:3300/0 conn(0x7f4ea0099eb0 0x7f4ea009a330 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4e90010730 tx=0x7f4e9000b670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.878+0000 7f4ea6544700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea0099eb0 0x7f4ea009a330 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7f4e90010730 tx=0x7f4e9000b670).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:47.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.878+0000 7f4ea6544700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea0099eb0 0x7f4ea009a330 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f4e90010730 tx=0x7f4e9000b670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.878+0000 7f4e9e7fc700 1 -- 172.21.15.16:0/2223399107 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e90000dd0 con 0x7f4ea0099eb0 2019-12-05T06:06:47.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.878+0000 7f4ea6544700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea0099eb0 0x7f4ea009a330 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f4e90010730 tx=0x7f4e9000b670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.878+0000 7f4ea6544700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea0099eb0 0x7f4ea009a330 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f4e90010730 tx=0x7f4e9000b670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.878+0000 7f4e9e7fc700 1 -- 172.21.15.16:0/2223399107 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e9000f160 con 0x7f4ea0099eb0 2019-12-05T06:06:47.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.878+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ea00995b0 con 0x7f4ea0099eb0 2019-12-05T06:06:47.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.878+0000 7f4e9e7fc700 1 -- 172.21.15.16:0/2223399107 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e9000f870 con 0x7f4ea0099eb0 2019-12-05T06:06:47.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.878+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ea00995b0 con 0x7f4ea0099eb0 2019-12-05T06:06:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.879+0000 7f4ea6544700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea0099eb0 0x7f4ea009a330 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f4e90010730 tx=0x7f4e9000b670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.880+0000 7f4e9e7fc700 1 -- 172.21.15.16:0/2223399107 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4e9000f350 con 0x7f4ea0099eb0 2019-12-05T06:06:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.880+0000 7f4ea6544700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea0099eb0 0x7f4ea009a330 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f4e90010730 tx=0x7f4e9000b670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.881+0000 7f4e9e7fc700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4e80023670 0x7f4e80025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.881+0000 7f4e9e7fc700 1 -- 172.21.15.16:0/2223399107 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f4e90046130 con 0x7f4ea0099eb0 2019-12-05T06:06:47.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.881+0000 7f4ea5d43700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4e80023670 0x7f4e80025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.882+0000 7f4ea5d43700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4e80023670 0x7f4e80025ae0 crc :-1 s=READY pgs=65 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:47.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.895+0000 7f4ea77a6700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:6813/10292,v1:172.21.15.16:6815/10292] conn(0x7f4ea00536c0 0x7f4ea0053ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:47.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.895+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 --> [v2:172.21.15.16:6813/10292,v1:172.21.15.16:6815/10292] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ea00958e0 con 0x7f4ea00536c0 2019-12-05T06:06:47.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.895+0000 7f4ea5542700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:6813/10292,v1:172.21.15.16:6815/10292] conn(0x7f4ea00536c0 0x7f4ea0053ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:47.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.896+0000 7f4ea5542700 1 -- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:6813/10292,v1:172.21.15.16:6815/10292] conn(0x7f4ea00536c0 msgr2=0x7f4ea0053ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:06:47.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.896+0000 7f4ea5542700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:6813/10292,v1:172.21.15.16:6815/10292] conn(0x7f4ea00536c0 0x7f4ea0053ab0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:47.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.898+0000 7f4e9e7fc700 1 -- 172.21.15.16:0/2223399107 <== osd.0 v2:172.21.15.16:6813/10292 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f4e94000c90 con 0x7f4ea00536c0 2019-12-05T06:06:47.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.936+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 --> [v2:172.21.15.16:6813/10292,v1:172.21.15.16:6815/10292] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4ea00958e0 con 0x7f4ea00536c0 2019-12-05T06:06:47.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.936+0000 7f4e9e7fc700 1 -- 172.21.15.16:0/2223399107 <== osd.0 v2:172.21.15.16:6813/10292 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4e94002020 con 0x7f4ea00536c0 2019-12-05T06:06:47.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.937+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:6813/10292,v1:172.21.15.16:6815/10292] conn(0x7f4ea00536c0 msgr2=0x7f4ea0053ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.937+0000 7f4ea77a6700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:6813/10292,v1:172.21.15.16:6815/10292] conn(0x7f4ea00536c0 0x7f4ea0053ab0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.938+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4e80023670 msgr2=0x7f4e80025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.938+0000 7f4ea77a6700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4e80023670 0x7f4e80025ae0 crc :-1 s=READY pgs=65 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.938+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea0099eb0 msgr2=0x7f4ea009a330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.938+0000 7f4ea77a6700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea0099eb0 0x7f4ea009a330 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7f4e90010730 tx=0x7f4e9000b670).stop 2019-12-05T06:06:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.938+0000 7f4ea6544700 1 -- 172.21.15.16:0/2223399107 reap_dead start 2019-12-05T06:06:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.938+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 shutdown_connections 2019-12-05T06:06:47.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.938+0000 7f4ea77a6700 1 --2- 172.21.15.16:0/2223399107 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ea0099eb0 0x7f4ea009a330 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:47.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.938+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 shutdown_connections 2019-12-05T06:06:47.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.938+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 wait complete. 2019-12-05T06:06:47.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:47.938+0000 7f4ea77a6700 1 -- 172.21.15.16:0/2223399107 >> 172.21.15.16:0/2223399107 conn(0x7f4ea00714f0 msgr2=0x7f4ea009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:47.949 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2019-12-05T06:06:47.950 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2019-12-05T06:06:48.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.210+0000 7f3fc2368700 1 Processor -- start 2019-12-05T06:06:48.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.210+0000 7f3fc2368700 1 -- start start 2019-12-05T06:06:48.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.211+0000 7f3fc2368700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3fbc08c6d0 0x7f3fbc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:48.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.211+0000 7f3fc2368700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc08fb80 con 0x7f3fbc08fef0 2019-12-05T06:06:48.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.211+0000 7f3fc2368700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0958e0 con 0x7f3fbc08f720 2019-12-05T06:06:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.211+0000 7f3fc2368700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc08df90 con 0x7f3fbc08c6d0 2019-12-05T06:06:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.211+0000 7f3fbbfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3fbc08fef0 0x7f3fbc08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59630/0 (socket says 172.21.15.16:59630) 2019-12-05T06:06:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.211+0000 7f3fbbfff700 1 -- 172.21.15.16:0/3944191513 learned_addr learned my addr 172.21.15.16:0/3944191513 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.211+0000 7f3fc0905700 1 --2- 172.21.15.16:0/3944191513 >> v2:172.21.15.76:3300/0 conn(0x7f3fbc08c6d0 0x7f3fbc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.212+0000 7f3fbb7fe700 1 -- 172.21.15.16:0/3944191513 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2234891213 0 0) 0x7f3fac0014d0 con 0x7f3fbc08fef0 2019-12-05T06:06:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.212+0000 7f3fbb7fe700 1 -- 172.21.15.16:0/3944191513 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fa8002e80 con 0x7f3fbc08fef0 2019-12-05T06:06:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.212+0000 7f3fc0905700 1 -- 172.21.15.16:0/3944191513 >> v1:172.21.15.76:6789/0 conn(0x7f3fbc08f720 legacy=0x7f3fbc1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.212+0000 7f3fbb7fe700 1 -- 172.21.15.16:0/3944191513 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076795204 0 0) 0x7f3fb40009b0 con 0x7f3fbc08f720 2019-12-05T06:06:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.212+0000 7f3fc0905700 1 -- 172.21.15.16:0/3944191513 >> v1:172.21.15.16:6789/0 conn(0x7f3fbc08fef0 legacy=0x7f3fbc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.212+0000 7f3fc0905700 1 -- 172.21.15.16:0/3944191513 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc08e620 con 0x7f3fbc08c6d0 2019-12-05T06:06:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.212+0000 7f3fc0905700 1 --2- 172.21.15.16:0/3944191513 >> v2:172.21.15.76:3300/0 conn(0x7f3fbc08c6d0 0x7f3fbc08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3fa400af50 tx=0x7f3fa400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.212+0000 7f3fc0905700 1 --2- 172.21.15.16:0/3944191513 >> v2:172.21.15.76:3300/0 conn(0x7f3fbc08c6d0 0x7f3fbc08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3fa400af50 tx=0x7f3fa400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.212+0000 7f3fc0905700 1 --2- 172.21.15.16:0/3944191513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc08c6d0 0x7f3fbc08cac0 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7f3fa400af50 tx=0x7f3fa400b760).ready entity=mon.1 client_cookie=4579c7c3666dd53a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.213+0000 7f3fc0905700 1 --2- 172.21.15.16:0/3944191513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc08c6d0 0x7f3fbc08cac0 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f3fa400af50 tx=0x7f3fa400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.213+0000 7f3fc0905700 1 --2- 172.21.15.16:0/3944191513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc08c6d0 0x7f3fbc08cac0 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f3fa400af50 tx=0x7f3fa400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.213+0000 7f3fbb7fe700 1 -- 172.21.15.16:0/3944191513 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fa4000960 con 0x7f3fbc08c6d0 2019-12-05T06:06:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.213+0000 7f3fc0905700 1 --2- 172.21.15.16:0/3944191513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc08c6d0 0x7f3fbc08cac0 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7f3fa400af50 tx=0x7f3fa400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.213+0000 7f3fbb7fe700 1 -- 172.21.15.16:0/3944191513 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa40098e0 con 0x7f3fbc08c6d0 2019-12-05T06:06:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.213+0000 7f3fbb7fe700 1 -- 172.21.15.16:0/3944191513 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fa400a050 con 0x7f3fbc08c6d0 2019-12-05T06:06:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.213+0000 7f3fc2368700 1 -- 172.21.15.16:0/3944191513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc08c6d0 msgr2=0x7f3fbc08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.213+0000 7f3fc2368700 1 --2- 172.21.15.16:0/3944191513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc08c6d0 0x7f3fbc08cac0 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7f3fa400af50 tx=0x7f3fa400b760).stop 2019-12-05T06:06:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.215+0000 7f3fc2368700 1 -- 172.21.15.16:0/3944191513 shutdown_connections 2019-12-05T06:06:48.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.215+0000 7f3fc2368700 1 --2- 172.21.15.16:0/3944191513 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3fbc08c6d0 0x7f3fbc08cac0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.215+0000 7f3fc2368700 1 -- 172.21.15.16:0/3944191513 shutdown_connections 2019-12-05T06:06:48.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.215+0000 7f3fc2368700 1 -- 172.21.15.16:0/3944191513 wait complete. 2019-12-05T06:06:48.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.215+0000 7f3fc2368700 1 -- 172.21.15.16:0/3944191513 >> 172.21.15.16:0/3944191513 conn(0x7f3fbc061940 msgr2=0x7f3fbc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:48.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.218+0000 7f3fc2368700 1 Processor -- start 2019-12-05T06:06:48.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.218+0000 7f3fc2368700 1 -- start start 2019-12-05T06:06:48.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.219+0000 7f3fc2368700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.219+0000 7f3fc2368700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc0a0db0 con 0x7f3fbc13b130 2019-12-05T06:06:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.219+0000 7f3fc2368700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3fbc070800 con 0x7f3fbc06fd70 2019-12-05T06:06:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.219+0000 7f3fc2368700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3fbc09f800 con 0x7f3fbc14fd10 2019-12-05T06:06:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fc0905700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fc0905700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46238/0 (socket says 172.21.15.16:46238) 2019-12-05T06:06:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fc0905700 1 -- 172.21.15.16:0/2006690347 learned_addr learned my addr 172.21.15.16:0/2006690347 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fbbfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3fbc13b130 0x7f3fbc13a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59636/0 (socket says 172.21.15.16:59636) 2019-12-05T06:06:48.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fc0905700 1 -- 172.21.15.16:0/2006690347 >> v1:172.21.15.76:6789/0 conn(0x7f3fbc06fd70 legacy=0x7f3fbc159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fc0905700 1 -- 172.21.15.16:0/2006690347 >> v1:172.21.15.16:6789/0 conn(0x7f3fbc13b130 legacy=0x7f3fbc13a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fc0905700 1 -- 172.21.15.16:0/2006690347 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fbc09fe90 con 0x7f3fbc14fd10 2019-12-05T06:06:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fb97fa700 1 -- 172.21.15.16:0/2006690347 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 958174209 0 0) 0x7f3fac0014d0 con 0x7f3fbc13b130 2019-12-05T06:06:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fc0905700 1 --2- 172.21.15.16:0/2006690347 >> v2:172.21.15.16:3300/0 conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3fa4010450 tx=0x7f3fa40107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fc0905700 1 --2- 172.21.15.16:0/2006690347 >> v2:172.21.15.16:3300/0 conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3fa4010450 tx=0x7f3fa40107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.220+0000 7f3fc0905700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7f3fa4010450 tx=0x7f3fa40107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.221+0000 7f3fc0905700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3fa4010450 tx=0x7f3fa40107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.221+0000 7f3fb97fa700 1 -- 172.21.15.16:0/2006690347 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fa4000940 con 0x7f3fbc14fd10 2019-12-05T06:06:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.221+0000 7f3fc0905700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3fa4010450 tx=0x7f3fa40107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.221+0000 7f3fc0905700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3fa4010450 tx=0x7f3fa40107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.221+0000 7f3fb97fa700 1 -- 172.21.15.16:0/2006690347 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fa4009980 con 0x7f3fbc14fd10 2019-12-05T06:06:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.221+0000 7f3fb97fa700 1 -- 172.21.15.16:0/2006690347 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fa400a0d0 con 0x7f3fbc14fd10 2019-12-05T06:06:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.221+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc09fe90 con 0x7f3fbc14fd10 2019-12-05T06:06:48.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.221+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc0a0db0 con 0x7f3fbc14fd10 2019-12-05T06:06:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.222+0000 7f3fc0905700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3fa4010450 tx=0x7f3fa40107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.223+0000 7f3fb97fa700 1 -- 172.21.15.16:0/2006690347 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3fa4009580 con 0x7f3fbc14fd10 2019-12-05T06:06:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.223+0000 7f3fc0905700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3fa4010450 tx=0x7f3fa40107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.223+0000 7f3fb97fa700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fa0023630 0x7f3fa0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.223+0000 7f3fb97fa700 1 -- 172.21.15.16:0/2006690347 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f3fa4045f20 con 0x7f3fbc14fd10 2019-12-05T06:06:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.223+0000 7f3fbbfff700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fa0023630 0x7f3fa0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.224+0000 7f3fbbfff700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fa0023630 0x7f3fa0025ac0 crc :-1 s=READY pgs=66 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.225+0000 7f3fc2368700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:6800/10289,v1:172.21.15.16:6801/10289] conn(0x7f3fbc0536c0 0x7f3fbc053ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.225+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 --> [v2:172.21.15.16:6800/10289,v1:172.21.15.16:6801/10289] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3fbc09fe90 con 0x7f3fbc0536c0 2019-12-05T06:06:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.225+0000 7f3fc1106700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:6800/10289,v1:172.21.15.16:6801/10289] conn(0x7f3fbc0536c0 0x7f3fbc053ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.226+0000 7f3fc1106700 1 -- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:6800/10289,v1:172.21.15.16:6801/10289] conn(0x7f3fbc0536c0 msgr2=0x7f3fbc053ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:06:48.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.226+0000 7f3fc1106700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:6800/10289,v1:172.21.15.16:6801/10289] conn(0x7f3fbc0536c0 0x7f3fbc053ab0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.227+0000 7f3fb97fa700 1 -- 172.21.15.16:0/2006690347 <== osd.1 v2:172.21.15.16:6800/10289 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f3fb4000a00 con 0x7f3fbc0536c0 2019-12-05T06:06:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.270+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 --> [v2:172.21.15.16:6800/10289,v1:172.21.15.16:6801/10289] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f3fbc09fe90 con 0x7f3fbc0536c0 2019-12-05T06:06:48.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.270+0000 7f3fb97fa700 1 -- 172.21.15.16:0/2006690347 <== osd.1 v2:172.21.15.16:6800/10289 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f3fb4005390 con 0x7f3fbc0536c0 2019-12-05T06:06:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.271+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:6800/10289,v1:172.21.15.16:6801/10289] conn(0x7f3fbc0536c0 msgr2=0x7f3fbc053ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.271+0000 7f3fc2368700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:6800/10289,v1:172.21.15.16:6801/10289] conn(0x7f3fbc0536c0 0x7f3fbc053ab0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.272+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fa0023630 msgr2=0x7f3fa0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.272+0000 7f3fc2368700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fa0023630 0x7f3fa0025ac0 crc :-1 s=READY pgs=66 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.272+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fbc14fd10 msgr2=0x7f3fbc13a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.272+0000 7f3fc2368700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7f3fa4010450 tx=0x7f3fa40107f0).stop 2019-12-05T06:06:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.272+0000 7f3fc1106700 1 -- 172.21.15.16:0/2006690347 reap_dead start 2019-12-05T06:06:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.272+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 shutdown_connections 2019-12-05T06:06:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.272+0000 7f3fc2368700 1 --2- 172.21.15.16:0/2006690347 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3fbc14fd10 0x7f3fbc13a1d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.272+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 shutdown_connections 2019-12-05T06:06:48.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.272+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 wait complete. 2019-12-05T06:06:48.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.272+0000 7f3fc2368700 1 -- 172.21.15.16:0/2006690347 >> 172.21.15.16:0/2006690347 conn(0x7f3fbc130a60 msgr2=0x7f3fbc086470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:48.284 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2019-12-05T06:06:48.285 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2019-12-05T06:06:48.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.568+0000 7f6680ca0700 1 Processor -- start 2019-12-05T06:06:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.568+0000 7f6680ca0700 1 -- start start 2019-12-05T06:06:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.570+0000 7f6680ca0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f667c08d2c0 0x7f667c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.570+0000 7f6680ca0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c08fef0 0x7f667c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.570+0000 7f6680ca0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667c0958e0 con 0x7f667c08c6d0 2019-12-05T06:06:48.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.570+0000 7f6680ca0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667c08df90 con 0x7f667c08d2c0 2019-12-05T06:06:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.570+0000 7f6680ca0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667c0a3f70 con 0x7f667c08fef0 2019-12-05T06:06:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.571+0000 7f667b59e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c08fef0 0x7f667c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.571+0000 7f667a59c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f667c08d2c0 0x7f667c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.571+0000 7f667ad9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f667c08c6d0 0x7f667c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52716/0 (socket says 172.21.15.16:52716) 2019-12-05T06:06:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.571+0000 7f667ad9d700 1 -- 172.21.15.16:0/1598525791 learned_addr learned my addr 172.21.15.16:0/1598525791 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.572+0000 7f667a59c700 1 -- 172.21.15.16:0/1598525791 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c08fef0 msgr2=0x7f667c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:48.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.572+0000 7f667a59c700 1 --2- 172.21.15.16:0/1598525791 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c08fef0 0x7f667c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.572+0000 7f667a59c700 1 -- 172.21.15.16:0/1598525791 >> v1:172.21.15.76:6789/0 conn(0x7f667c08c6d0 legacy=0x7f667c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.572+0000 7f667a59c700 1 -- 172.21.15.16:0/1598525791 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c08e620 con 0x7f667c08d2c0 2019-12-05T06:06:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.572+0000 7f6679d9b700 1 -- 172.21.15.16:0/1598525791 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300971976 0 0) 0x7f6674001480 con 0x7f667c08c6d0 2019-12-05T06:06:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.572+0000 7f667a59c700 1 --2- 172.21.15.16:0/1598525791 >> v2:172.21.15.76:3300/0 conn(0x7f667c08d2c0 0x7f667c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f666400af50 tx=0x7f666400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.572+0000 7f667a59c700 1 --2- 172.21.15.16:0/1598525791 >> v2:172.21.15.76:3300/0 conn(0x7f667c08d2c0 0x7f667c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f666400af50 tx=0x7f666400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.572+0000 7f667a59c700 1 --2- 172.21.15.16:0/1598525791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f667c08d2c0 0x7f667c08ef20 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f666400af50 tx=0x7f666400b6e0).ready entity=mon.1 client_cookie=116b7be389778dc8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.572+0000 7f667a59c700 1 --2- 172.21.15.16:0/1598525791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f667c08d2c0 0x7f667c08ef20 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f666400af50 tx=0x7f666400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.572+0000 7f667a59c700 1 --2- 172.21.15.16:0/1598525791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f667c08d2c0 0x7f667c08ef20 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f666400af50 tx=0x7f666400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.573+0000 7f667a59c700 1 --2- 172.21.15.16:0/1598525791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f667c08d2c0 0x7f667c08ef20 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f666400af50 tx=0x7f666400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.573+0000 7f6679d9b700 1 -- 172.21.15.16:0/1598525791 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6664007210 con 0x7f667c08d2c0 2019-12-05T06:06:48.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.573+0000 7f6679d9b700 1 -- 172.21.15.16:0/1598525791 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66640099b0 con 0x7f667c08d2c0 2019-12-05T06:06:48.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.573+0000 7f6679d9b700 1 -- 172.21.15.16:0/1598525791 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666400a040 con 0x7f667c08d2c0 2019-12-05T06:06:48.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.573+0000 7f6680ca0700 1 -- 172.21.15.16:0/1598525791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f667c08d2c0 msgr2=0x7f667c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.573+0000 7f6680ca0700 1 --2- 172.21.15.16:0/1598525791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f667c08d2c0 0x7f667c08ef20 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f666400af50 tx=0x7f666400b6e0).stop 2019-12-05T06:06:48.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.575+0000 7f6680ca0700 1 -- 172.21.15.16:0/1598525791 shutdown_connections 2019-12-05T06:06:48.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.575+0000 7f6680ca0700 1 --2- 172.21.15.16:0/1598525791 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c08fef0 0x7f667c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:48.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.575+0000 7f6680ca0700 1 --2- 172.21.15.16:0/1598525791 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f667c08d2c0 0x7f667c08ef20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.575+0000 7f6680ca0700 1 -- 172.21.15.16:0/1598525791 shutdown_connections 2019-12-05T06:06:48.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.575+0000 7f6680ca0700 1 -- 172.21.15.16:0/1598525791 wait complete. 2019-12-05T06:06:48.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.575+0000 7f6680ca0700 1 -- 172.21.15.16:0/1598525791 >> 172.21.15.16:0/1598525791 conn(0x7f667c061940 msgr2=0x7f667c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.580+0000 7f6680ca0700 1 Processor -- start 2019-12-05T06:06:48.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.580+0000 7f6680ca0700 1 -- start start 2019-12-05T06:06:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.582+0000 7f6680ca0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.582+0000 7f6680ca0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667c131bf0 con 0x7f667c1302f0 2019-12-05T06:06:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.582+0000 7f6680ca0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667c1316f0 con 0x7f667c153b50 2019-12-05T06:06:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.582+0000 7f6680ca0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f667c1296e0 con 0x7f667c12b640 2019-12-05T06:06:48.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.582+0000 7f667ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f667c1302f0 0x7f667c12a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59652/0 (socket says 172.21.15.16:59652) 2019-12-05T06:06:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.582+0000 7f667ad9d700 1 -- 172.21.15.16:0/1743742582 learned_addr learned my addr 172.21.15.16:0/1743742582 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.583+0000 7f667a59c700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.583+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112900829 0 0) 0x7f6674000fe0 con 0x7f667c1302f0 2019-12-05T06:06:48.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.583+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6660002e80 con 0x7f667c1302f0 2019-12-05T06:06:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.583+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4195591870 0 0) 0x7f666c001080 con 0x7f667c153b50 2019-12-05T06:06:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.583+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6660002ec0 con 0x7f667c153b50 2019-12-05T06:06:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.583+0000 7f667a59c700 1 -- 172.21.15.16:0/1743742582 >> v1:172.21.15.76:6789/0 conn(0x7f667c153b50 legacy=0x7f667c155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.583+0000 7f667a59c700 1 -- 172.21.15.16:0/1743742582 >> v1:172.21.15.16:6789/0 conn(0x7f667c1302f0 legacy=0x7f667c12a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.583+0000 7f667a59c700 1 -- 172.21.15.16:0/1743742582 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667c129d70 con 0x7f667c12b640 2019-12-05T06:06:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.583+0000 7f667a59c700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f66640103f0 tx=0x7f6664010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.584+0000 7f667a59c700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f66640103f0 tx=0x7f6664010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.584+0000 7f667a59c700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 secure :-1 s=READY pgs=33 cs=0 l=1 rx=0x7f66640103f0 tx=0x7f6664010790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.584+0000 7f667a59c700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7f66640103f0 tx=0x7f6664010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.584+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666400ac20 con 0x7f667c12b640 2019-12-05T06:06:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.584+0000 7f667a59c700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7f66640103f0 tx=0x7f6664010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.584+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666400ac20 con 0x7f667c12b640 2019-12-05T06:06:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.584+0000 7f667a59c700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7f66640103f0 tx=0x7f6664010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.584+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f667c129d70 con 0x7f667c12b640 2019-12-05T06:06:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.584+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6664009960 con 0x7f667c12b640 2019-12-05T06:06:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.584+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f667c129d70 con 0x7f667c12b640 2019-12-05T06:06:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.585+0000 7f667a59c700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7f66640103f0 tx=0x7f6664010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.586+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6664009580 con 0x7f667c12b640 2019-12-05T06:06:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.586+0000 7f667a59c700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 secure :-1 s=THROTTLE_DONE pgs=33 cs=0 l=1 rx=0x7f66640103f0 tx=0x7f6664010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.587+0000 7f66737fe700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6660023680 0x7f6660025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.587+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f6664045db0 con 0x7f667c12b640 2019-12-05T06:06:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.587+0000 7f667ad9d700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6660023680 0x7f6660025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.588+0000 7f667ad9d700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6660023680 0x7f6660025af0 crc :-1 s=READY pgs=67 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.599+0000 7f6680ca0700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.16:6808/10291,v1:172.21.15.16:6809/10291] conn(0x7f667c06d210 0x7f667c0536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.600+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 --> [v2:172.21.15.16:6808/10291,v1:172.21.15.16:6809/10291] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f667c131bf0 con 0x7f667c06d210 2019-12-05T06:06:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.600+0000 7f667b59e700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.16:6808/10291,v1:172.21.15.16:6809/10291] conn(0x7f667c06d210 0x7f667c0536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.600+0000 7f667b59e700 1 -- 172.21.15.16:0/1743742582 >> [v2:172.21.15.16:6808/10291,v1:172.21.15.16:6809/10291] conn(0x7f667c06d210 msgr2=0x7f667c0536c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:06:48.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.600+0000 7f667b59e700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.16:6808/10291,v1:172.21.15.16:6809/10291] conn(0x7f667c06d210 0x7f667c0536c0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.602+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 <== osd.2 v2:172.21.15.16:6808/10291 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f666c000b30 con 0x7f667c06d210 2019-12-05T06:06:48.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.645+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 --> [v2:172.21.15.16:6808/10291,v1:172.21.15.16:6809/10291] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f667c131bf0 con 0x7f667c06d210 2019-12-05T06:06:48.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.646+0000 7f66737fe700 1 -- 172.21.15.16:0/1743742582 <== osd.2 v2:172.21.15.16:6808/10291 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f666c002430 con 0x7f667c06d210 2019-12-05T06:06:48.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 >> [v2:172.21.15.16:6808/10291,v1:172.21.15.16:6809/10291] conn(0x7f667c06d210 msgr2=0x7f667c0536c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f6680ca0700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.16:6808/10291,v1:172.21.15.16:6809/10291] conn(0x7f667c06d210 0x7f667c0536c0 crc :-1 s=READY pgs=7 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6660023680 msgr2=0x7f6660025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f6680ca0700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6660023680 0x7f6660025af0 crc :-1 s=READY pgs=67 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 msgr2=0x7f667c12ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f6680ca0700 1 --2- 172.21.15.16:0/1743742582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f667c12b640 0x7f667c12ae70 secure :-1 s=READY pgs=33 cs=0 l=1 rx=0x7f66640103f0 tx=0x7f6664010790).stop 2019-12-05T06:06:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f667b59e700 1 -- 172.21.15.16:0/1743742582 reap_dead start 2019-12-05T06:06:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 shutdown_connections 2019-12-05T06:06:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 shutdown_connections 2019-12-05T06:06:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 wait complete. 2019-12-05T06:06:48.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.648+0000 7f6680ca0700 1 -- 172.21.15.16:0/1743742582 >> 172.21.15.16:0/1743742582 conn(0x7f667c0a2850 msgr2=0x7f667c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:48.661 INFO:teuthology.orchestra.run.smithi016.stdout:55834574854 2019-12-05T06:06:48.662 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2019-12-05T06:06:48.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.873+0000 7f477861d700 1 Processor -- start 2019-12-05T06:06:48.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.873+0000 7f477861d700 1 -- start start 2019-12-05T06:06:48.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.874+0000 7f477861d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f477008c6d0 0x7f477008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:48.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.874+0000 7f477861d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f477008f720 0x7f477008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:48.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.874+0000 7f477861d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f477008fb80 con 0x7f477008fef0 2019-12-05T06:06:48.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.874+0000 7f477861d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477008df90 con 0x7f477008c6d0 2019-12-05T06:06:48.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.874+0000 7f477861d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47700a3f70 con 0x7f477008f720 2019-12-05T06:06:48.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f4776bba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f477008c6d0 0x7f477008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f4776bba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f477008c6d0 0x7f477008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46262/0 (socket says 172.21.15.16:46262) 2019-12-05T06:06:48.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f4776bba700 1 -- 172.21.15.16:0/2229741068 learned_addr learned my addr 172.21.15.16:0/2229741068 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:48.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f47773bb700 1 --2- 172.21.15.16:0/2229741068 >> v2:172.21.15.76:3300/0 conn(0x7f477008f720 0x7f477008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f4776bba700 1 -- 172.21.15.16:0/2229741068 >> v2:172.21.15.76:3300/0 conn(0x7f477008f720 msgr2=0x7f477008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:48.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f4776bba700 1 --2- 172.21.15.16:0/2229741068 >> v2:172.21.15.76:3300/0 conn(0x7f477008f720 0x7f477008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:48.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f4776bba700 1 -- 172.21.15.16:0/2229741068 >> v1:172.21.15.16:6789/0 conn(0x7f477008fef0 legacy=0x7f477008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f4776bba700 1 -- 172.21.15.16:0/2229741068 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477008d540 con 0x7f477008c6d0 2019-12-05T06:06:48.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f47773bb700 1 --2- 172.21.15.16:0/2229741068 >> v2:172.21.15.76:3300/0 conn(0x7f477008f720 0x7f477008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:06:48.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f4776bba700 1 --2- 172.21.15.16:0/2229741068 >> v2:172.21.15.16:3300/0 conn(0x7f477008c6d0 0x7f477008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f476000f740 tx=0x7f476000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f4776bba700 1 --2- 172.21.15.16:0/2229741068 >> v2:172.21.15.16:3300/0 conn(0x7f477008c6d0 0x7f477008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f476000f740 tx=0x7f476000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.875+0000 7f4776bba700 1 --2- 172.21.15.16:0/2229741068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f477008c6d0 0x7f477008cac0 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7f476000f740 tx=0x7f476000fae0).ready entity=mon.0 client_cookie=b28cf72fbf3fc837 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.876+0000 7f4776bba700 1 --2- 172.21.15.16:0/2229741068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f477008c6d0 0x7f477008cac0 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f476000f740 tx=0x7f476000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.876+0000 7f4776bba700 1 --2- 172.21.15.16:0/2229741068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f477008c6d0 0x7f477008cac0 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f476000f740 tx=0x7f476000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.876+0000 7f4775bb8700 1 -- 172.21.15.16:0/2229741068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4760000960 con 0x7f477008c6d0 2019-12-05T06:06:48.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.876+0000 7f4776bba700 1 --2- 172.21.15.16:0/2229741068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f477008c6d0 0x7f477008cac0 secure :-1 s=THROTTLE_DONE pgs=119 cs=0 l=1 rx=0x7f476000f740 tx=0x7f476000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.876+0000 7f4775bb8700 1 -- 172.21.15.16:0/2229741068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4760000c90 con 0x7f477008c6d0 2019-12-05T06:06:48.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.876+0000 7f4775bb8700 1 -- 172.21.15.16:0/2229741068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47600013a0 con 0x7f477008c6d0 2019-12-05T06:06:48.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.876+0000 7f477861d700 1 -- 172.21.15.16:0/2229741068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f477008c6d0 msgr2=0x7f477008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.876+0000 7f477861d700 1 --2- 172.21.15.16:0/2229741068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f477008c6d0 0x7f477008cac0 secure :-1 s=READY pgs=119 cs=0 l=1 rx=0x7f476000f740 tx=0x7f476000fae0).stop 2019-12-05T06:06:48.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.877+0000 7f477861d700 1 -- 172.21.15.16:0/2229741068 shutdown_connections 2019-12-05T06:06:48.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.877+0000 7f477861d700 1 --2- 172.21.15.16:0/2229741068 >> v2:172.21.15.76:3300/0 conn(0x7f477008f720 0x7f477008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:48.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.877+0000 7f477861d700 1 --2- 172.21.15.16:0/2229741068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f477008c6d0 0x7f477008cac0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.877+0000 7f477861d700 1 -- 172.21.15.16:0/2229741068 shutdown_connections 2019-12-05T06:06:48.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.877+0000 7f477861d700 1 -- 172.21.15.16:0/2229741068 wait complete. 2019-12-05T06:06:48.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.877+0000 7f477861d700 1 -- 172.21.15.16:0/2229741068 >> 172.21.15.16:0/2229741068 conn(0x7f4770061940 msgr2=0x7f4770096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:48.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.881+0000 7f477861d700 1 Processor -- start 2019-12-05T06:06:48.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.881+0000 7f477861d700 1 -- start start 2019-12-05T06:06:48.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.883+0000 7f477861d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f477014c3a0 0x7f477009d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.883+0000 7f477861d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f477009e580 0x7f477019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.883+0000 7f477861d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4770153160 con 0x7f477006e210 2019-12-05T06:06:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.883+0000 7f477861d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f477009c620 con 0x7f477014c3a0 2019-12-05T06:06:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.883+0000 7f477861d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f477008ada0 con 0x7f477009e580 2019-12-05T06:06:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.883+0000 7f47763b9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f477006e210 0x7f477009daf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59672/0 (socket says 172.21.15.16:59672) 2019-12-05T06:06:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.883+0000 7f47763b9700 1 -- 172.21.15.16:0/1163043378 learned_addr learned my addr 172.21.15.16:0/1163043378 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:48.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.883+0000 7f4776bba700 1 --2- 172.21.15.16:0/1163043378 >> v2:172.21.15.76:3300/0 conn(0x7f477014c3a0 0x7f477009d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.883+0000 7f47773bb700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f477009e580 0x7f477019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f476f7fe700 1 -- 172.21.15.16:0/1163043378 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083048562 0 0) 0x7f4764000ff0 con 0x7f477006e210 2019-12-05T06:06:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f476f7fe700 1 -- 172.21.15.16:0/1163043378 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4750002e80 con 0x7f477006e210 2019-12-05T06:06:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f4776bba700 1 -- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f477009e580 msgr2=0x7f477019d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f4776bba700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f477009e580 0x7f477019d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f4776bba700 1 -- 172.21.15.16:0/1163043378 >> v1:172.21.15.16:6789/0 conn(0x7f477006e210 legacy=0x7f477009daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f4776bba700 1 -- 172.21.15.16:0/1163043378 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f477009ccb0 con 0x7f477014c3a0 2019-12-05T06:06:48.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f4776bba700 1 --2- 172.21.15.16:0/1163043378 >> v2:172.21.15.76:3300/0 conn(0x7f477014c3a0 0x7f477009d5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f47600103a0 tx=0x7f47600101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f4776bba700 1 --2- 172.21.15.16:0/1163043378 >> v2:172.21.15.76:3300/0 conn(0x7f477014c3a0 0x7f477009d5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f47600103a0 tx=0x7f47600101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f4776bba700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f477014c3a0 0x7f477009d5b0 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f47600103a0 tx=0x7f47600101d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f4776bba700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f477014c3a0 0x7f477009d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f47600103a0 tx=0x7f47600101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.884+0000 7f476f7fe700 1 -- 172.21.15.16:0/1163043378 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4760005560 con 0x7f477014c3a0 2019-12-05T06:06:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.885+0000 7f4776bba700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f477014c3a0 0x7f477009d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f47600103a0 tx=0x7f47600101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.885+0000 7f4776bba700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f477014c3a0 0x7f477009d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f47600103a0 tx=0x7f47600101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.885+0000 7f476f7fe700 1 -- 172.21.15.16:0/1163043378 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4760005180 con 0x7f477014c3a0 2019-12-05T06:06:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.885+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f477009ccb0 con 0x7f477014c3a0 2019-12-05T06:06:48.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.885+0000 7f476f7fe700 1 -- 172.21.15.16:0/1163043378 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4760005b40 con 0x7f477014c3a0 2019-12-05T06:06:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.885+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f477009ccb0 con 0x7f477014c3a0 2019-12-05T06:06:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.886+0000 7f4776bba700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f477014c3a0 0x7f477009d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f47600103a0 tx=0x7f47600101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.887+0000 7f476f7fe700 1 -- 172.21.15.16:0/1163043378 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4760005370 con 0x7f477014c3a0 2019-12-05T06:06:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.887+0000 7f4776bba700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f477014c3a0 0x7f477009d5b0 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f47600103a0 tx=0x7f47600101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.887+0000 7f476f7fe700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4750023670 0x7f4750025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.887+0000 7f476f7fe700 1 -- 172.21.15.16:0/1163043378 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f4760045cd0 con 0x7f477014c3a0 2019-12-05T06:06:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.887+0000 7f477861d700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.16:6816/10293,v1:172.21.15.16:6819/10293] conn(0x7f47700536c0 0x7f4770053ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.887+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 --> [v2:172.21.15.16:6816/10293,v1:172.21.15.16:6819/10293] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f47700958e0 con 0x7f47700536c0 2019-12-05T06:06:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.887+0000 7f47763b9700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4750023670 0x7f4750025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.888+0000 7f47773bb700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.16:6816/10293,v1:172.21.15.16:6819/10293] conn(0x7f47700536c0 0x7f4770053ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.888+0000 7f47773bb700 1 -- 172.21.15.16:0/1163043378 >> [v2:172.21.15.16:6816/10293,v1:172.21.15.16:6819/10293] conn(0x7f47700536c0 msgr2=0x7f4770053ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:06:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.888+0000 7f47773bb700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.16:6816/10293,v1:172.21.15.16:6819/10293] conn(0x7f47700536c0 0x7f4770053ab0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.888+0000 7f47763b9700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4750023670 0x7f4750025ae0 crc :-1 s=READY pgs=68 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:48.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.889+0000 7f476f7fe700 1 -- 172.21.15.16:0/1163043378 <== osd.3 v2:172.21.15.16:6816/10293 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f4768006d00 con 0x7f47700536c0 2019-12-05T06:06:48.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.919+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 --> [v2:172.21.15.16:6816/10293,v1:172.21.15.16:6819/10293] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f47700958e0 con 0x7f47700536c0 2019-12-05T06:06:48.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.919+0000 7f476f7fe700 1 -- 172.21.15.16:0/1163043378 <== osd.3 v2:172.21.15.16:6816/10293 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4768005890 con 0x7f47700536c0 2019-12-05T06:06:48.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.920+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 >> [v2:172.21.15.16:6816/10293,v1:172.21.15.16:6819/10293] conn(0x7f47700536c0 msgr2=0x7f4770053ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.920+0000 7f477861d700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.16:6816/10293,v1:172.21.15.16:6819/10293] conn(0x7f47700536c0 0x7f4770053ab0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.921+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4750023670 msgr2=0x7f4750025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.921+0000 7f477861d700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4750023670 0x7f4750025ae0 crc :-1 s=READY pgs=68 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.921+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f477014c3a0 msgr2=0x7f477009d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.921+0000 7f477861d700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f477014c3a0 0x7f477009d5b0 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f47600103a0 tx=0x7f47600101d0).stop 2019-12-05T06:06:48.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.921+0000 7f47773bb700 1 -- 172.21.15.16:0/1163043378 reap_dead start 2019-12-05T06:06:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.921+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 shutdown_connections 2019-12-05T06:06:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.921+0000 7f477861d700 1 --2- 172.21.15.16:0/1163043378 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f477014c3a0 0x7f477009d5b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.921+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 shutdown_connections 2019-12-05T06:06:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.921+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 wait complete. 2019-12-05T06:06:48.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:48.921+0000 7f477861d700 1 -- 172.21.15.16:0/1163043378 >> 172.21.15.16:0/1163043378 conn(0x7f4770098480 msgr2=0x7f4770071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:48.930 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2019-12-05T06:06:48.931 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2019-12-05T06:06:49.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.142+0000 7f5726122700 1 Processor -- start 2019-12-05T06:06:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.142+0000 7f5726122700 1 -- start start 2019-12-05T06:06:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.144+0000 7f5726122700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f572008c6d0 0x7f572008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.144+0000 7f5726122700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:49.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.144+0000 7f5726122700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57200958e0 con 0x7f572008ef20 2019-12-05T06:06:49.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.144+0000 7f5726122700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f572008df90 con 0x7f572008c6d0 2019-12-05T06:06:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.144+0000 7f5726122700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57200a3f70 con 0x7f572008fef0 2019-12-05T06:06:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.144+0000 7f571f7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.144+0000 7f571ffff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f572008c6d0 0x7f572008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.144+0000 7f571f7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:50270/0 (socket says 172.21.15.16:50270) 2019-12-05T06:06:49.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.144+0000 7f571f7fe700 1 -- 172.21.15.16:0/929380797 learned_addr learned my addr 172.21.15.16:0/929380797 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:49.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.144+0000 7f5724ec0700 1 --1- 172.21.15.16:0/929380797 >> v1:172.21.15.76:6789/0 conn(0x7f572008ef20 0x7f572008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52750/0 (socket says 172.21.15.16:52750) 2019-12-05T06:06:49.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.145+0000 7f571f7fe700 1 -- 172.21.15.16:0/929380797 >> v2:172.21.15.76:3300/0 conn(0x7f572008c6d0 msgr2=0x7f572008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:49.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.145+0000 7f571f7fe700 1 --2- 172.21.15.16:0/929380797 >> v2:172.21.15.76:3300/0 conn(0x7f572008c6d0 0x7f572008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.145+0000 7f571f7fe700 1 -- 172.21.15.16:0/929380797 >> v1:172.21.15.76:6789/0 conn(0x7f572008ef20 legacy=0x7f572008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.145+0000 7f571f7fe700 1 -- 172.21.15.16:0/929380797 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572008d540 con 0x7f572008fef0 2019-12-05T06:06:49.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.145+0000 7f571ffff700 1 --2- 172.21.15.16:0/929380797 >> v2:172.21.15.76:3300/0 conn(0x7f572008c6d0 0x7f572008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.145+0000 7f571f7fe700 1 --2- 172.21.15.16:0/929380797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f570800f740 tx=0x7f570800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.145+0000 7f571f7fe700 1 --2- 172.21.15.16:0/929380797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f570800f740 tx=0x7f570800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.145+0000 7f571f7fe700 1 --2- 172.21.15.16:0/929380797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 secure :-1 s=READY pgs=34 cs=0 l=1 rx=0x7f570800f740 tx=0x7f570800fae0).ready entity=mon.2 client_cookie=9567e5c45a9e5d19 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:49.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.146+0000 7f571f7fe700 1 --2- 172.21.15.16:0/929380797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f570800f740 tx=0x7f570800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.146+0000 7f571f7fe700 1 --2- 172.21.15.16:0/929380797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f570800f740 tx=0x7f570800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.146+0000 7f571f7fe700 1 --2- 172.21.15.16:0/929380797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 secure :-1 s=THROTTLE_DONE pgs=34 cs=0 l=1 rx=0x7f570800f740 tx=0x7f570800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.146+0000 7f571effd700 1 -- 172.21.15.16:0/929380797 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5708000960 con 0x7f572008fef0 2019-12-05T06:06:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.146+0000 7f571effd700 1 -- 172.21.15.16:0/929380797 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5708000c90 con 0x7f572008fef0 2019-12-05T06:06:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.146+0000 7f571effd700 1 -- 172.21.15.16:0/929380797 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57080012a0 con 0x7f572008fef0 2019-12-05T06:06:49.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.146+0000 7f5726122700 1 -- 172.21.15.16:0/929380797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 msgr2=0x7f572008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.146+0000 7f5726122700 1 --2- 172.21.15.16:0/929380797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 secure :-1 s=READY pgs=34 cs=0 l=1 rx=0x7f570800f740 tx=0x7f570800fae0).stop 2019-12-05T06:06:49.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.148+0000 7f5726122700 1 -- 172.21.15.16:0/929380797 shutdown_connections 2019-12-05T06:06:49.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.148+0000 7f5726122700 1 --2- 172.21.15.16:0/929380797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572008fef0 0x7f572008d000 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.148+0000 7f5726122700 1 --2- 172.21.15.16:0/929380797 >> v2:172.21.15.76:3300/0 conn(0x7f572008c6d0 0x7f572008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:49.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.148+0000 7f5726122700 1 -- 172.21.15.16:0/929380797 shutdown_connections 2019-12-05T06:06:49.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.148+0000 7f5726122700 1 -- 172.21.15.16:0/929380797 wait complete. 2019-12-05T06:06:49.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.148+0000 7f5726122700 1 -- 172.21.15.16:0/929380797 >> 172.21.15.16:0/929380797 conn(0x7f5720061940 msgr2=0x7f5720096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:49.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.152+0000 7f5726122700 1 Processor -- start 2019-12-05T06:06:49.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.153+0000 7f5726122700 1 -- start start 2019-12-05T06:06:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.154+0000 7f5726122700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57201302f0 0x7f572009d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.154+0000 7f5726122700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572009e7d0 0x7f572009dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.154+0000 7f5726122700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f572012cb90 con 0x7f572009e280 2019-12-05T06:06:49.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.154+0000 7f5726122700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5720131ff0 con 0x7f57201302f0 2019-12-05T06:06:49.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.154+0000 7f5726122700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57201315c0 con 0x7f572009e7d0 2019-12-05T06:06:49.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.154+0000 7f571ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57201302f0 0x7f572009d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.154+0000 7f571ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57201302f0 0x7f572009d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46282/0 (socket says 172.21.15.16:46282) 2019-12-05T06:06:49.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.154+0000 7f571ffff700 1 -- 172.21.15.16:0/2105795110 learned_addr learned my addr 172.21.15.16:0/2105795110 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:49.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.154+0000 7f571f7fe700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572009e7d0 0x7f572009dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.154+0000 7f571ffff700 1 -- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572009e7d0 msgr2=0x7f572009dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571ffff700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f572009e7d0 0x7f572009dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571ffff700 1 -- 172.21.15.16:0/2105795110 >> v1:172.21.15.76:6789/0 conn(0x7f572009e280 legacy=0x7f572006dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571ffff700 1 -- 172.21.15.16:0/2105795110 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5720132680 con 0x7f57201302f0 2019-12-05T06:06:49.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571ffff700 1 --2- 172.21.15.16:0/2105795110 >> v2:172.21.15.16:3300/0 conn(0x7f57201302f0 0x7f572009d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5710010790 tx=0x7f5710010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571ffff700 1 --2- 172.21.15.16:0/2105795110 >> v2:172.21.15.16:3300/0 conn(0x7f57201302f0 0x7f572009d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5710010790 tx=0x7f5710010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571ffff700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57201302f0 0x7f572009d800 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7f5710010790 tx=0x7f5710010610).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:49.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571ffff700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57201302f0 0x7f572009d800 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f5710010790 tx=0x7f5710010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571cff9700 1 -- 172.21.15.16:0/2105795110 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5710000ce0 con 0x7f57201302f0 2019-12-05T06:06:49.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571ffff700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57201302f0 0x7f572009d800 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f5710010790 tx=0x7f5710010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5720132680 con 0x7f57201302f0 2019-12-05T06:06:49.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571ffff700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57201302f0 0x7f572009d800 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f5710010790 tx=0x7f5710010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571cff9700 1 -- 172.21.15.16:0/2105795110 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5710000cc0 con 0x7f57201302f0 2019-12-05T06:06:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f571cff9700 1 -- 172.21.15.16:0/2105795110 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5710009ce0 con 0x7f57201302f0 2019-12-05T06:06:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.155+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5720132680 con 0x7f57201302f0 2019-12-05T06:06:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.156+0000 7f571ffff700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57201302f0 0x7f572009d800 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f5710010790 tx=0x7f5710010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.157+0000 7f571cff9700 1 -- 172.21.15.16:0/2105795110 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5710009900 con 0x7f57201302f0 2019-12-05T06:06:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.157+0000 7f571ffff700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57201302f0 0x7f572009d800 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f5710010790 tx=0x7f5710010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.157+0000 7f571cff9700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5704023630 0x7f5704025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.157+0000 7f571cff9700 1 -- 172.21.15.16:0/2105795110 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f57100461a0 con 0x7f57201302f0 2019-12-05T06:06:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.158+0000 7f571f7fe700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5704023630 0x7f5704025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.158+0000 7f571f7fe700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5704023630 0x7f5704025ac0 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.173+0000 7f5726122700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] conn(0x7f572006d210 0x7f57200536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.173+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 --> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5720132680 con 0x7f572006d210 2019-12-05T06:06:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.174+0000 7f5724ec0700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] conn(0x7f572006d210 0x7f57200536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.174+0000 7f5724ec0700 1 -- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] conn(0x7f572006d210 msgr2=0x7f57200536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send send error: (32) Broken pipe 2019-12-05T06:06:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.174+0000 7f5724ec0700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] conn(0x7f572006d210 0x7f57200536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).write auth request write failed r=-32 ((32) Broken pipe) 2019-12-05T06:06:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.174+0000 7f5724ec0700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] conn(0x7f572006d210 0x7f57200536c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:06:49.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.375+0000 7f5724ec0700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] conn(0x7f572006d210 0x7f57200536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.376+0000 7f5724ec0700 1 -- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] conn(0x7f572006d210 msgr2=0x7f57200536c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:06:49.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.376+0000 7f5724ec0700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] conn(0x7f572006d210 0x7f57200536c0 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:49.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.378+0000 7f571cff9700 1 -- 172.21.15.16:0/2105795110 <== osd.4 v2:172.21.15.76:6802/10472 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f5718001080 con 0x7f572006d210 2019-12-05T06:06:49.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.435+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 --> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f5720132680 con 0x7f572006d210 2019-12-05T06:06:49.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.436+0000 7f571cff9700 1 -- 172.21.15.16:0/2105795110 <== osd.4 v2:172.21.15.76:6802/10472 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f5718000e70 con 0x7f572006d210 2019-12-05T06:06:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.436+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] conn(0x7f572006d210 msgr2=0x7f57200536c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.436+0000 7f5726122700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6802/10472,v1:172.21.15.76:6803/10472] conn(0x7f572006d210 0x7f57200536c0 crc :-1 s=READY pgs=10 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.437+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5704023630 msgr2=0x7f5704025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.437+0000 7f5726122700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5704023630 0x7f5704025ac0 crc :-1 s=READY pgs=70 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.437+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57201302f0 msgr2=0x7f572009d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.437+0000 7f5726122700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57201302f0 0x7f572009d800 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7f5710010790 tx=0x7f5710010610).stop 2019-12-05T06:06:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.437+0000 7f5724ec0700 1 -- 172.21.15.16:0/2105795110 reap_dead start 2019-12-05T06:06:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.437+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 shutdown_connections 2019-12-05T06:06:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.437+0000 7f5726122700 1 --2- 172.21.15.16:0/2105795110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57201302f0 0x7f572009d800 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.437+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 shutdown_connections 2019-12-05T06:06:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.437+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 wait complete. 2019-12-05T06:06:49.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.437+0000 7f5726122700 1 -- 172.21.15.16:0/2105795110 >> 172.21.15.16:0/2105795110 conn(0x7f57200a2850 msgr2=0x7f572006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:49.449 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2019-12-05T06:06:49.450 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2019-12-05T06:06:49.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.743+0000 7f0b35088700 1 Processor -- start 2019-12-05T06:06:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.743+0000 7f0b35088700 1 -- start start 2019-12-05T06:06:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.745+0000 7f0b35088700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0b3008c6d0 0x7f0b3008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.745+0000 7f0b35088700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b3008f720 0x7f0b3008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.745+0000 7f0b35088700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3008fb80 con 0x7f0b3008fef0 2019-12-05T06:06:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.745+0000 7f0b35088700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3008df90 con 0x7f0b3008f720 2019-12-05T06:06:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.745+0000 7f0b35088700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b300a3f70 con 0x7f0b3008c6d0 2019-12-05T06:06:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.745+0000 7f0b34887700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b3008f720 0x7f0b3008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.745+0000 7f0b34887700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b3008f720 0x7f0b3008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46296/0 (socket says 172.21.15.16:46296) 2019-12-05T06:06:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.745+0000 7f0b34887700 1 -- 172.21.15.16:0/1219174773 learned_addr learned my addr 172.21.15.16:0/1219174773 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.745+0000 7f0b2ed9d700 1 --1- 172.21.15.16:0/1219174773 >> v1:172.21.15.16:6789/0 conn(0x7f0b3008fef0 0x7f0b3008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59698/0 (socket says 172.21.15.16:59698) 2019-12-05T06:06:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.745+0000 7f0b2f59e700 1 --2- 172.21.15.16:0/1219174773 >> v2:172.21.15.76:3300/0 conn(0x7f0b3008c6d0 0x7f0b3008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.746+0000 7f0b34887700 1 -- 172.21.15.16:0/1219174773 >> v2:172.21.15.76:3300/0 conn(0x7f0b3008c6d0 msgr2=0x7f0b3008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.746+0000 7f0b34887700 1 --2- 172.21.15.16:0/1219174773 >> v2:172.21.15.76:3300/0 conn(0x7f0b3008c6d0 0x7f0b3008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.746+0000 7f0b34887700 1 -- 172.21.15.16:0/1219174773 >> v1:172.21.15.16:6789/0 conn(0x7f0b3008fef0 legacy=0x7f0b3008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.746+0000 7f0b2e59c700 1 -- 172.21.15.16:0/1219174773 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235053626 0 0) 0x7f0b18001480 con 0x7f0b3008fef0 2019-12-05T06:06:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.746+0000 7f0b34887700 1 -- 172.21.15.16:0/1219174773 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3008d540 con 0x7f0b3008f720 2019-12-05T06:06:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.746+0000 7f0b2f59e700 1 --2- 172.21.15.16:0/1219174773 >> v2:172.21.15.76:3300/0 conn(0x7f0b3008c6d0 0x7f0b3008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.746+0000 7f0b34887700 1 --2- 172.21.15.16:0/1219174773 >> v2:172.21.15.16:3300/0 conn(0x7f0b3008f720 0x7f0b3008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0b2800f730 tx=0x7f0b2800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.746+0000 7f0b34887700 1 --2- 172.21.15.16:0/1219174773 >> v2:172.21.15.16:3300/0 conn(0x7f0b3008f720 0x7f0b3008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0b2800f730 tx=0x7f0b2800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.746+0000 7f0b34887700 1 --2- 172.21.15.16:0/1219174773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3008f720 0x7f0b3008d000 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f0b2800f730 tx=0x7f0b2800fad0).ready entity=mon.0 client_cookie=8a1b9ac10c2dd7ff server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.747+0000 7f0b34887700 1 --2- 172.21.15.16:0/1219174773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3008f720 0x7f0b3008d000 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f0b2800f730 tx=0x7f0b2800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.747+0000 7f0b2e59c700 1 -- 172.21.15.16:0/1219174773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b28000b90 con 0x7f0b3008f720 2019-12-05T06:06:49.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.747+0000 7f0b34887700 1 --2- 172.21.15.16:0/1219174773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3008f720 0x7f0b3008d000 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f0b2800f730 tx=0x7f0b2800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.747+0000 7f0b34887700 1 --2- 172.21.15.16:0/1219174773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3008f720 0x7f0b3008d000 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f0b2800f730 tx=0x7f0b2800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.747+0000 7f0b2e59c700 1 -- 172.21.15.16:0/1219174773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b280010e0 con 0x7f0b3008f720 2019-12-05T06:06:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.747+0000 7f0b2e59c700 1 -- 172.21.15.16:0/1219174773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b28001830 con 0x7f0b3008f720 2019-12-05T06:06:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.747+0000 7f0b35088700 1 -- 172.21.15.16:0/1219174773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3008f720 msgr2=0x7f0b3008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.747+0000 7f0b35088700 1 --2- 172.21.15.16:0/1219174773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3008f720 0x7f0b3008d000 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f0b2800f730 tx=0x7f0b2800fad0).stop 2019-12-05T06:06:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.749+0000 7f0b35088700 1 -- 172.21.15.16:0/1219174773 shutdown_connections 2019-12-05T06:06:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.749+0000 7f0b35088700 1 --2- 172.21.15.16:0/1219174773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3008f720 0x7f0b3008d000 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.749+0000 7f0b35088700 1 --2- 172.21.15.16:0/1219174773 >> v2:172.21.15.76:3300/0 conn(0x7f0b3008c6d0 0x7f0b3008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.749+0000 7f0b35088700 1 -- 172.21.15.16:0/1219174773 shutdown_connections 2019-12-05T06:06:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.749+0000 7f0b35088700 1 -- 172.21.15.16:0/1219174773 wait complete. 2019-12-05T06:06:49.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.749+0000 7f0b35088700 1 -- 172.21.15.16:0/1219174773 >> 172.21.15.16:0/1219174773 conn(0x7f0b30061940 msgr2=0x7f0b30096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:49.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.754+0000 7f0b35088700 1 Processor -- start 2019-12-05T06:06:49.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.755+0000 7f0b35088700 1 -- start start 2019-12-05T06:06:49.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.757+0000 7f0b35088700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3006e800 0x7f0b301336f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:49.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.757+0000 7f0b35088700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0b301346c0 0x7f0b30133c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:49.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.757+0000 7f0b35088700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b3009ab90 con 0x7f0b3006dc10 2019-12-05T06:06:49.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.757+0000 7f0b35088700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3009a7d0 con 0x7f0b301346c0 2019-12-05T06:06:49.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.757+0000 7f0b35088700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3009fc20 con 0x7f0b3006e800 2019-12-05T06:06:49.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.757+0000 7f0b2f59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0b3006dc10 0x7f0b3006e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59700/0 (socket says 172.21.15.16:59700) 2019-12-05T06:06:49.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.757+0000 7f0b2f59e700 1 -- 172.21.15.16:0/1875566920 learned_addr learned my addr 172.21.15.16:0/1875566920 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:49.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.757+0000 7f0b2ed9d700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3006e800 0x7f0b301336f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.757+0000 7f0b34887700 1 --2- 172.21.15.16:0/1875566920 >> v2:172.21.15.76:3300/0 conn(0x7f0b301346c0 0x7f0b30133c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.757+0000 7f0b27fff700 1 -- 172.21.15.16:0/1875566920 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 630127439 0 0) 0x7f0b2000dd90 con 0x7f0b3006dc10 2019-12-05T06:06:49.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.758+0000 7f0b27fff700 1 -- 172.21.15.16:0/1875566920 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b14002e80 con 0x7f0b3006dc10 2019-12-05T06:06:49.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.758+0000 7f0b34887700 1 -- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3006e800 msgr2=0x7f0b301336f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.758+0000 7f0b34887700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3006e800 0x7f0b301336f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.758+0000 7f0b34887700 1 -- 172.21.15.16:0/1875566920 >> v1:172.21.15.16:6789/0 conn(0x7f0b3006dc10 legacy=0x7f0b3006e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.758+0000 7f0b34887700 1 -- 172.21.15.16:0/1875566920 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b30132df0 con 0x7f0b301346c0 2019-12-05T06:06:49.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.758+0000 7f0b2ed9d700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3006e800 0x7f0b301336f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:49.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.758+0000 7f0b34887700 1 --2- 172.21.15.16:0/1875566920 >> v2:172.21.15.76:3300/0 conn(0x7f0b301346c0 0x7f0b30133c30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0b280103a0 tx=0x7f0b28010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.758+0000 7f0b34887700 1 --2- 172.21.15.16:0/1875566920 >> v2:172.21.15.76:3300/0 conn(0x7f0b301346c0 0x7f0b30133c30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0b280103a0 tx=0x7f0b28010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.758+0000 7f0b34887700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b301346c0 0x7f0b30133c30 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7f0b280103a0 tx=0x7f0b28010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:49.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.759+0000 7f0b34887700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b301346c0 0x7f0b30133c30 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f0b280103a0 tx=0x7f0b28010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.759+0000 7f0b27fff700 1 -- 172.21.15.16:0/1875566920 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b28001290 con 0x7f0b301346c0 2019-12-05T06:06:49.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.759+0000 7f0b34887700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b301346c0 0x7f0b30133c30 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f0b280103a0 tx=0x7f0b28010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.759+0000 7f0b34887700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b301346c0 0x7f0b30133c30 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f0b280103a0 tx=0x7f0b28010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.759+0000 7f0b27fff700 1 -- 172.21.15.16:0/1875566920 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b28000eb0 con 0x7f0b301346c0 2019-12-05T06:06:49.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.759+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b30132df0 con 0x7f0b301346c0 2019-12-05T06:06:49.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.759+0000 7f0b27fff700 1 -- 172.21.15.16:0/1875566920 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b28001870 con 0x7f0b301346c0 2019-12-05T06:06:49.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.759+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b30132df0 con 0x7f0b301346c0 2019-12-05T06:06:49.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.760+0000 7f0b34887700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b301346c0 0x7f0b30133c30 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f0b280103a0 tx=0x7f0b28010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.761+0000 7f0b27fff700 1 -- 172.21.15.16:0/1875566920 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0b280010a0 con 0x7f0b301346c0 2019-12-05T06:06:49.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.761+0000 7f0b34887700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b301346c0 0x7f0b30133c30 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f0b280103a0 tx=0x7f0b28010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:49.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.761+0000 7f0b27fff700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b14023670 0x7f0b14025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:49.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.761+0000 7f0b27fff700 1 -- 172.21.15.16:0/1875566920 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f0b28045d50 con 0x7f0b301346c0 2019-12-05T06:06:49.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.761+0000 7f0b2f59e700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b14023670 0x7f0b14025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.762+0000 7f0b2f59e700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b14023670 0x7f0b14025ae0 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:49.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.774+0000 7f0b35088700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6810/10473,v1:172.21.15.76:6811/10473] conn(0x7f0b300536c0 0x7f0b30053ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:49.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.774+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 --> [v2:172.21.15.76:6810/10473,v1:172.21.15.76:6811/10473] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0b30132df0 con 0x7f0b300536c0 2019-12-05T06:06:49.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.775+0000 7f0b2ed9d700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6810/10473,v1:172.21.15.76:6811/10473] conn(0x7f0b300536c0 0x7f0b30053ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:49.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.775+0000 7f0b2ed9d700 1 -- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6810/10473,v1:172.21.15.76:6811/10473] conn(0x7f0b300536c0 msgr2=0x7f0b30053ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:06:49.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.775+0000 7f0b2ed9d700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6810/10473,v1:172.21.15.76:6811/10473] conn(0x7f0b300536c0 0x7f0b30053ab0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:49.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.777+0000 7f0b27fff700 1 -- 172.21.15.16:0/1875566920 <== osd.5 v2:172.21.15.76:6810/10473 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f0b18001230 con 0x7f0b300536c0 2019-12-05T06:06:49.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.827+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 --> [v2:172.21.15.76:6810/10473,v1:172.21.15.76:6811/10473] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0b30132df0 con 0x7f0b300536c0 2019-12-05T06:06:49.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.828+0000 7f0b27fff700 1 -- 172.21.15.16:0/1875566920 <== osd.5 v2:172.21.15.76:6810/10473 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0b180045c0 con 0x7f0b300536c0 2019-12-05T06:06:49.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.829+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6810/10473,v1:172.21.15.76:6811/10473] conn(0x7f0b300536c0 msgr2=0x7f0b30053ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.829+0000 7f0b35088700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6810/10473,v1:172.21.15.76:6811/10473] conn(0x7f0b300536c0 0x7f0b30053ab0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.829+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b14023670 msgr2=0x7f0b14025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.829+0000 7f0b35088700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b14023670 0x7f0b14025ae0 crc :-1 s=READY pgs=71 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.829+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b301346c0 msgr2=0x7f0b30133c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.829+0000 7f0b35088700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b301346c0 0x7f0b30133c30 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7f0b280103a0 tx=0x7f0b28010740).stop 2019-12-05T06:06:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.829+0000 7f0b34887700 1 -- 172.21.15.16:0/1875566920 reap_dead start 2019-12-05T06:06:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.829+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 shutdown_connections 2019-12-05T06:06:49.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.829+0000 7f0b35088700 1 --2- 172.21.15.16:0/1875566920 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b301346c0 0x7f0b30133c30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.830+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 shutdown_connections 2019-12-05T06:06:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.830+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 wait complete. 2019-12-05T06:06:49.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:49.830+0000 7f0b35088700 1 -- 172.21.15.16:0/1875566920 >> 172.21.15.16:0/1875566920 conn(0x7f0b3009b060 msgr2=0x7f0b3006d380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:49.852 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2019-12-05T06:06:49.854 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2019-12-05T06:06:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.106+0000 7f66a0581700 1 Processor -- start 2019-12-05T06:06:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.106+0000 7f66a0581700 1 -- start start 2019-12-05T06:06:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.107+0000 7f66a0581700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f669808c6d0 0x7f669808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:50.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.108+0000 7f66a0581700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f669808f720 0x7f669808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.108+0000 7f66a0581700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669808fb80 con 0x7f669808fef0 2019-12-05T06:06:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.108+0000 7f66a0581700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f669808df90 con 0x7f669808c6d0 2019-12-05T06:06:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.108+0000 7f66a0581700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66980a3f70 con 0x7f669808f720 2019-12-05T06:06:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.108+0000 7f669eb1e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f669808c6d0 0x7f669808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.108+0000 7f669e31d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f669808fef0 0x7f669808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59712/0 (socket says 172.21.15.16:59712) 2019-12-05T06:06:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.108+0000 7f669e31d700 1 -- 172.21.15.16:0/3035706112 learned_addr learned my addr 172.21.15.16:0/3035706112 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.108+0000 7f669eb1e700 1 --2- 172.21.15.16:0/3035706112 >> v2:172.21.15.16:3300/0 conn(0x7f669808c6d0 0x7f669808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46310/0 (socket says 172.21.15.16:46310) 2019-12-05T06:06:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.108+0000 7f669f31f700 1 --2- 172.21.15.16:0/3035706112 >> v2:172.21.15.76:3300/0 conn(0x7f669808f720 0x7f669808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.109+0000 7f669eb1e700 1 -- 172.21.15.16:0/3035706112 >> v2:172.21.15.76:3300/0 conn(0x7f669808f720 msgr2=0x7f669808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:50.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.109+0000 7f669eb1e700 1 --2- 172.21.15.16:0/3035706112 >> v2:172.21.15.76:3300/0 conn(0x7f669808f720 0x7f669808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.109+0000 7f669eb1e700 1 -- 172.21.15.16:0/3035706112 >> v1:172.21.15.16:6789/0 conn(0x7f669808fef0 legacy=0x7f669808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.109+0000 7f669db1c700 1 -- 172.21.15.16:0/3035706112 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1270800997 0 0) 0x7f668c0013f0 con 0x7f669808fef0 2019-12-05T06:06:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.109+0000 7f669eb1e700 1 -- 172.21.15.16:0/3035706112 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669808d540 con 0x7f669808c6d0 2019-12-05T06:06:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.109+0000 7f669f31f700 1 --2- 172.21.15.16:0/3035706112 >> v2:172.21.15.76:3300/0 conn(0x7f669808f720 0x7f669808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:50.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.109+0000 7f669eb1e700 1 --2- 172.21.15.16:0/3035706112 >> v2:172.21.15.16:3300/0 conn(0x7f669808c6d0 0x7f669808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f668800af50 tx=0x7f668800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.109+0000 7f669eb1e700 1 --2- 172.21.15.16:0/3035706112 >> v2:172.21.15.16:3300/0 conn(0x7f669808c6d0 0x7f669808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f668800af50 tx=0x7f668800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.109+0000 7f669eb1e700 1 --2- 172.21.15.16:0/3035706112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669808c6d0 0x7f669808cac0 secure :-1 s=READY pgs=126 cs=0 l=1 rx=0x7f668800af50 tx=0x7f668800b760).ready entity=mon.0 client_cookie=dd8c6039dfd4f282 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.110+0000 7f669eb1e700 1 --2- 172.21.15.16:0/3035706112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669808c6d0 0x7f669808cac0 secure :-1 s=THROTTLE_DONE pgs=126 cs=0 l=1 rx=0x7f668800af50 tx=0x7f668800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.110+0000 7f669eb1e700 1 --2- 172.21.15.16:0/3035706112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669808c6d0 0x7f669808cac0 secure :-1 s=THROTTLE_DONE pgs=126 cs=0 l=1 rx=0x7f668800af50 tx=0x7f668800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.110+0000 7f669db1c700 1 -- 172.21.15.16:0/3035706112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6688007210 con 0x7f669808c6d0 2019-12-05T06:06:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.110+0000 7f669eb1e700 1 --2- 172.21.15.16:0/3035706112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669808c6d0 0x7f669808cac0 secure :-1 s=THROTTLE_DONE pgs=126 cs=0 l=1 rx=0x7f668800af50 tx=0x7f668800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.110+0000 7f669db1c700 1 -- 172.21.15.16:0/3035706112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66880099b0 con 0x7f669808c6d0 2019-12-05T06:06:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.110+0000 7f669db1c700 1 -- 172.21.15.16:0/3035706112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f668800a080 con 0x7f669808c6d0 2019-12-05T06:06:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.110+0000 7f66a0581700 1 -- 172.21.15.16:0/3035706112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669808c6d0 msgr2=0x7f669808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.110+0000 7f66a0581700 1 --2- 172.21.15.16:0/3035706112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669808c6d0 0x7f669808cac0 secure :-1 s=READY pgs=126 cs=0 l=1 rx=0x7f668800af50 tx=0x7f668800b760).stop 2019-12-05T06:06:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.112+0000 7f66a0581700 1 -- 172.21.15.16:0/3035706112 shutdown_connections 2019-12-05T06:06:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.112+0000 7f66a0581700 1 --2- 172.21.15.16:0/3035706112 >> v2:172.21.15.76:3300/0 conn(0x7f669808f720 0x7f669808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.112+0000 7f66a0581700 1 --2- 172.21.15.16:0/3035706112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f669808c6d0 0x7f669808cac0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.112+0000 7f66a0581700 1 -- 172.21.15.16:0/3035706112 shutdown_connections 2019-12-05T06:06:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.112+0000 7f66a0581700 1 -- 172.21.15.16:0/3035706112 wait complete. 2019-12-05T06:06:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.112+0000 7f66a0581700 1 -- 172.21.15.16:0/3035706112 >> 172.21.15.16:0/3035706112 conn(0x7f6698061940 msgr2=0x7f6698096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:50.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.118+0000 7f66a0581700 1 Processor -- start 2019-12-05T06:06:50.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.118+0000 7f66a0581700 1 -- start start 2019-12-05T06:06:50.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.120+0000 7f66a0581700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.120+0000 7f66a0581700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6698132970 con 0x7f6698132530 2019-12-05T06:06:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.120+0000 7f66a0581700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6698098da0 con 0x7f669814d400 2019-12-05T06:06:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.120+0000 7f66a0581700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f669809bb50 con 0x7f6698132d00 2019-12-05T06:06:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.120+0000 7f669f31f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6698132530 0x7f6698157100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59718/0 (socket says 172.21.15.16:59718) 2019-12-05T06:06:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.120+0000 7f669f31f700 1 -- 172.21.15.16:0/625214959 learned_addr learned my addr 172.21.15.16:0/625214959 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.120+0000 7f669e31d700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.121+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215660334 0 0) 0x7f6690009890 con 0x7f6698132530 2019-12-05T06:06:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.121+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6678002e80 con 0x7f6698132530 2019-12-05T06:06:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.121+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2947221424 0 0) 0x7f668800adc0 con 0x7f669814d400 2019-12-05T06:06:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.121+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6678004170 con 0x7f669814d400 2019-12-05T06:06:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.121+0000 7f669e31d700 1 -- 172.21.15.16:0/625214959 >> v1:172.21.15.76:6789/0 conn(0x7f669814d400 legacy=0x7f6698131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.121+0000 7f669e31d700 1 -- 172.21.15.16:0/625214959 >> v1:172.21.15.16:6789/0 conn(0x7f6698132530 legacy=0x7f6698157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.121+0000 7f669e31d700 1 -- 172.21.15.16:0/625214959 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669809b0f0 con 0x7f6698132d00 2019-12-05T06:06:50.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.121+0000 7f669e31d700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f668c0106b0 tx=0x7f668c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.121+0000 7f669e31d700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f668c0106b0 tx=0x7f668c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.121+0000 7f669e31d700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7f668c0106b0 tx=0x7f668c010a50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.122+0000 7f669e31d700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f668c0106b0 tx=0x7f668c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.122+0000 7f669e31d700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f668c0106b0 tx=0x7f668c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.122+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f668c004550 con 0x7f6698132d00 2019-12-05T06:06:50.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.122+0000 7f669e31d700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f668c0106b0 tx=0x7f668c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.122+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f668c0010d0 con 0x7f6698132d00 2019-12-05T06:06:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.122+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f669809b0f0 con 0x7f6698132d00 2019-12-05T06:06:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.122+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f668c004b30 con 0x7f6698132d00 2019-12-05T06:06:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.122+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f669809b0f0 con 0x7f6698132d00 2019-12-05T06:06:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.123+0000 7f669e31d700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f668c0106b0 tx=0x7f668c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.124+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f668c004360 con 0x7f6698132d00 2019-12-05T06:06:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.124+0000 7f669e31d700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 secure :-1 s=THROTTLE_DONE pgs=35 cs=0 l=1 rx=0x7f668c0106b0 tx=0x7f668c010a50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.124+0000 7f66977fe700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6678023650 0x7f6678025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.124+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f668c0460e0 con 0x7f6698132d00 2019-12-05T06:06:50.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.125+0000 7f669eb1e700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6678023650 0x7f6678025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.125+0000 7f669eb1e700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6678023650 0x7f6678025ae0 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.137+0000 7f66a0581700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6815/10474,v1:172.21.15.76:6817/10474] conn(0x7f66980536c0 0x7f6698053ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.137+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 --> [v2:172.21.15.76:6815/10474,v1:172.21.15.76:6817/10474] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f66980958e0 con 0x7f66980536c0 2019-12-05T06:06:50.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.138+0000 7f669f31f700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6815/10474,v1:172.21.15.76:6817/10474] conn(0x7f66980536c0 0x7f6698053ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.138+0000 7f669f31f700 1 -- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6815/10474,v1:172.21.15.76:6817/10474] conn(0x7f66980536c0 msgr2=0x7f6698053ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:06:50.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.138+0000 7f669f31f700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6815/10474,v1:172.21.15.76:6817/10474] conn(0x7f66980536c0 0x7f6698053ab0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.140+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 <== osd.6 v2:172.21.15.76:6815/10474 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7f6690000c40 con 0x7f66980536c0 2019-12-05T06:06:50.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.184+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 --> [v2:172.21.15.76:6815/10474,v1:172.21.15.76:6817/10474] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f66980958e0 con 0x7f66980536c0 2019-12-05T06:06:50.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.185+0000 7f66977fe700 1 -- 172.21.15.16:0/625214959 <== osd.6 v2:172.21.15.76:6815/10474 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f6690000bc0 con 0x7f66980536c0 2019-12-05T06:06:50.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.186+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6815/10474,v1:172.21.15.76:6817/10474] conn(0x7f66980536c0 msgr2=0x7f6698053ab0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.186+0000 7f66a0581700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6815/10474,v1:172.21.15.76:6817/10474] conn(0x7f66980536c0 0x7f6698053ab0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.186+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6678023650 msgr2=0x7f6678025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.186+0000 7f66a0581700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6678023650 0x7f6678025ae0 crc :-1 s=READY pgs=72 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.186+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 msgr2=0x7f669814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.186+0000 7f66a0581700 1 --2- 172.21.15.16:0/625214959 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6698132d00 0x7f669814d7f0 secure :-1 s=READY pgs=35 cs=0 l=1 rx=0x7f668c0106b0 tx=0x7f668c010a50).stop 2019-12-05T06:06:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.186+0000 7f669f31f700 1 -- 172.21.15.16:0/625214959 reap_dead start 2019-12-05T06:06:50.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.187+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 shutdown_connections 2019-12-05T06:06:50.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.187+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 shutdown_connections 2019-12-05T06:06:50.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.187+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 wait complete. 2019-12-05T06:06:50.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.187+0000 7f66a0581700 1 -- 172.21.15.16:0/625214959 >> 172.21.15.16:0/625214959 conn(0x7f66980714f0 msgr2=0x7f669809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:50.202 INFO:teuthology.orchestra.run.smithi016.stdout:55834574854 2019-12-05T06:06:50.203 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2019-12-05T06:06:50.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.493+0000 7ff497d02700 1 Processor -- start 2019-12-05T06:06:50.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.493+0000 7ff497d02700 1 -- start start 2019-12-05T06:06:50.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.495+0000 7ff497d02700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff49008c6d0 0x7ff49008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:50.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.495+0000 7ff497d02700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff49008fef0 0x7ff49008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:50.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.495+0000 7ff497d02700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49008fb80 con 0x7ff49008ef20 2019-12-05T06:06:50.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.495+0000 7ff497d02700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49008df90 con 0x7ff49008fef0 2019-12-05T06:06:50.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.495+0000 7ff497d02700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4900a3f70 con 0x7ff49008c6d0 2019-12-05T06:06:50.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.496+0000 7ff495a9e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff49008fef0 0x7ff49008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.496+0000 7ff495a9e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff49008fef0 0x7ff49008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46328/0 (socket says 172.21.15.16:46328) 2019-12-05T06:06:50.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.496+0000 7ff495a9e700 1 -- 172.21.15.16:0/1275176206 learned_addr learned my addr 172.21.15.16:0/1275176206 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:50.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.496+0000 7ff49629f700 1 --2- 172.21.15.16:0/1275176206 >> v2:172.21.15.76:3300/0 conn(0x7ff49008c6d0 0x7ff49008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff495a9e700 1 -- 172.21.15.16:0/1275176206 >> v2:172.21.15.76:3300/0 conn(0x7ff49008c6d0 msgr2=0x7ff49008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:50.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff495a9e700 1 --2- 172.21.15.16:0/1275176206 >> v2:172.21.15.76:3300/0 conn(0x7ff49008c6d0 0x7ff49008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:50.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff495a9e700 1 -- 172.21.15.16:0/1275176206 >> v1:172.21.15.76:6789/0 conn(0x7ff49008ef20 legacy=0x7ff49008f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff495a9e700 1 -- 172.21.15.16:0/1275176206 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff49008d540 con 0x7ff49008fef0 2019-12-05T06:06:50.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff495a9e700 1 --2- 172.21.15.16:0/1275176206 >> v2:172.21.15.16:3300/0 conn(0x7ff49008fef0 0x7ff49008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff48400f740 tx=0x7ff48400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff495a9e700 1 --2- 172.21.15.16:0/1275176206 >> v2:172.21.15.16:3300/0 conn(0x7ff49008fef0 0x7ff49008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff48400f740 tx=0x7ff48400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff495a9e700 1 --2- 172.21.15.16:0/1275176206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff49008fef0 0x7ff49008d000 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7ff48400f740 tx=0x7ff48400f570).ready entity=mon.0 client_cookie=558ac2e63bd9a879 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff495a9e700 1 --2- 172.21.15.16:0/1275176206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff49008fef0 0x7ff49008d000 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7ff48400f740 tx=0x7ff48400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff495a9e700 1 --2- 172.21.15.16:0/1275176206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff49008fef0 0x7ff49008d000 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7ff48400f740 tx=0x7ff48400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff49529d700 1 -- 172.21.15.16:0/1275176206 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff484000960 con 0x7ff49008fef0 2019-12-05T06:06:50.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.497+0000 7ff495a9e700 1 --2- 172.21.15.16:0/1275176206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff49008fef0 0x7ff49008d000 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7ff48400f740 tx=0x7ff48400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.498+0000 7ff49529d700 1 -- 172.21.15.16:0/1275176206 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff484000c90 con 0x7ff49008fef0 2019-12-05T06:06:50.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.498+0000 7ff49529d700 1 -- 172.21.15.16:0/1275176206 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff4840013a0 con 0x7ff49008fef0 2019-12-05T06:06:50.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.498+0000 7ff497d02700 1 -- 172.21.15.16:0/1275176206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff49008fef0 msgr2=0x7ff49008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.498+0000 7ff497d02700 1 --2- 172.21.15.16:0/1275176206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff49008fef0 0x7ff49008d000 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7ff48400f740 tx=0x7ff48400f570).stop 2019-12-05T06:06:50.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.500+0000 7ff497d02700 1 -- 172.21.15.16:0/1275176206 shutdown_connections 2019-12-05T06:06:50.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.500+0000 7ff497d02700 1 --2- 172.21.15.16:0/1275176206 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff49008fef0 0x7ff49008d000 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:50.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.500+0000 7ff497d02700 1 --2- 172.21.15.16:0/1275176206 >> v2:172.21.15.76:3300/0 conn(0x7ff49008c6d0 0x7ff49008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:50.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.500+0000 7ff497d02700 1 -- 172.21.15.16:0/1275176206 shutdown_connections 2019-12-05T06:06:50.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.500+0000 7ff497d02700 1 -- 172.21.15.16:0/1275176206 wait complete. 2019-12-05T06:06:50.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.500+0000 7ff497d02700 1 -- 172.21.15.16:0/1275176206 >> 172.21.15.16:0/1275176206 conn(0x7ff490061940 msgr2=0x7ff490096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:50.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.505+0000 7ff497d02700 1 Processor -- start 2019-12-05T06:06:50.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.505+0000 7ff497d02700 1 -- start start 2019-12-05T06:06:50.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.507+0000 7ff497d02700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:50.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.507+0000 7ff497d02700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff4900992d0 0x7ff490098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:50.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.507+0000 7ff497d02700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff49006e300 con 0x7ff490098d80 2019-12-05T06:06:50.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.507+0000 7ff497d02700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff49008ad00 con 0x7ff4900992d0 2019-12-05T06:06:50.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.507+0000 7ff497d02700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff490096c10 con 0x7ff49006df10 2019-12-05T06:06:50.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.507+0000 7ff496aa0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff490098d80 0x7ff490088fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59734/0 (socket says 172.21.15.16:59734) 2019-12-05T06:06:50.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.507+0000 7ff496aa0700 1 -- 172.21.15.16:0/980193430 learned_addr learned my addr 172.21.15.16:0/980193430 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:50.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.507+0000 7ff495a9e700 1 --2- 172.21.15.16:0/980193430 >> v2:172.21.15.76:3300/0 conn(0x7ff4900992d0 0x7ff490098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.508+0000 7ff49629f700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.508+0000 7ff48effd700 1 -- 172.21.15.16:0/980193430 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 975528822 0 0) 0x7ff480000b90 con 0x7ff490098d80 2019-12-05T06:06:50.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.508+0000 7ff48effd700 1 -- 172.21.15.16:0/980193430 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff470002e80 con 0x7ff490098d80 2019-12-05T06:06:50.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.508+0000 7ff49629f700 1 -- 172.21.15.16:0/980193430 >> v2:172.21.15.76:3300/0 conn(0x7ff4900992d0 msgr2=0x7ff490098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.508+0000 7ff49629f700 1 --2- 172.21.15.16:0/980193430 >> v2:172.21.15.76:3300/0 conn(0x7ff4900992d0 0x7ff490098840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:50.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.508+0000 7ff49629f700 1 -- 172.21.15.16:0/980193430 >> v1:172.21.15.16:6789/0 conn(0x7ff490098d80 legacy=0x7ff490088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.508+0000 7ff48effd700 1 -- 172.21.15.16:0/980193430 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3531219430 0 0) 0x7ff4800012b0 con 0x7ff490098d80 2019-12-05T06:06:50.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff49629f700 1 -- 172.21.15.16:0/980193430 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff490097a00 con 0x7ff49006df10 2019-12-05T06:06:50.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff495a9e700 1 --2- 172.21.15.16:0/980193430 >> v2:172.21.15.76:3300/0 conn(0x7ff4900992d0 0x7ff490098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:50.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff49629f700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff488010710 tx=0x7ff488010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff49629f700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff488010710 tx=0x7ff488010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff49629f700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7ff488010710 tx=0x7ff488010ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff49629f700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7ff488010710 tx=0x7ff488010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff48effd700 1 -- 172.21.15.16:0/980193430 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff488000cc0 con 0x7ff49006df10 2019-12-05T06:06:50.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff49629f700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7ff488010710 tx=0x7ff488010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff49629f700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7ff488010710 tx=0x7ff488010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff48effd700 1 -- 172.21.15.16:0/980193430 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff488009f50 con 0x7ff49006df10 2019-12-05T06:06:50.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff48effd700 1 -- 172.21.15.16:0/980193430 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff48800a520 con 0x7ff49006df10 2019-12-05T06:06:50.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.509+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff490097a00 con 0x7ff49006df10 2019-12-05T06:06:50.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.510+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff490097a00 con 0x7ff49006df10 2019-12-05T06:06:50.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.510+0000 7ff49629f700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7ff488010710 tx=0x7ff488010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.511+0000 7ff49629f700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 secure :-1 s=THROTTLE_DONE pgs=36 cs=0 l=1 rx=0x7ff488010710 tx=0x7ff488010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.511+0000 7ff48effd700 1 -- 172.21.15.16:0/980193430 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff4880099e0 con 0x7ff49006df10 2019-12-05T06:06:50.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.512+0000 7ff48effd700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff470023680 0x7ff470025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:50.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.512+0000 7ff48effd700 1 -- 172.21.15.16:0/980193430 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff4880460f0 con 0x7ff49006df10 2019-12-05T06:06:50.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.512+0000 7ff495a9e700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff470023680 0x7ff470025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.513+0000 7ff495a9e700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff470023680 0x7ff470025af0 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.524+0000 7ff497d02700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6824/10475,v1:172.21.15.76:6826/10475] conn(0x7ff490089da0 0x7ff4900536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:50.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.524+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 --> [v2:172.21.15.76:6824/10475,v1:172.21.15.76:6826/10475] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff4900958e0 con 0x7ff490089da0 2019-12-05T06:06:50.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.525+0000 7ff496aa0700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6824/10475,v1:172.21.15.76:6826/10475] conn(0x7ff490089da0 0x7ff4900536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.525+0000 7ff496aa0700 1 -- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6824/10475,v1:172.21.15.76:6826/10475] conn(0x7ff490089da0 msgr2=0x7ff4900536c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2019-12-05T06:06:50.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.525+0000 7ff496aa0700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6824/10475,v1:172.21.15.76:6826/10475] conn(0x7ff490089da0 0x7ff4900536c0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.527+0000 7ff48effd700 1 -- 172.21.15.16:0/980193430 <== osd.7 v2:172.21.15.76:6824/10475 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23014 (crc 0 0 0) 0x7ff480001b40 con 0x7ff490089da0 2019-12-05T06:06:50.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.576+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 --> [v2:172.21.15.76:6824/10475,v1:172.21.15.76:6826/10475] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff4900958e0 con 0x7ff490089da0 2019-12-05T06:06:50.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.577+0000 7ff48effd700 1 -- 172.21.15.16:0/980193430 <== osd.7 v2:172.21.15.76:6824/10475 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7ff4800010b0 con 0x7ff490089da0 2019-12-05T06:06:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.578+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6824/10475,v1:172.21.15.76:6826/10475] conn(0x7ff490089da0 msgr2=0x7ff4900536c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.578+0000 7ff497d02700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6824/10475,v1:172.21.15.76:6826/10475] conn(0x7ff490089da0 0x7ff4900536c0 crc :-1 s=READY pgs=8 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.579+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff470023680 msgr2=0x7ff470025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.579+0000 7ff497d02700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff470023680 0x7ff470025af0 crc :-1 s=READY pgs=73 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.579+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 msgr2=0x7ff490098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.579+0000 7ff497d02700 1 --2- 172.21.15.16:0/980193430 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff49006df10 0x7ff490098300 secure :-1 s=READY pgs=36 cs=0 l=1 rx=0x7ff488010710 tx=0x7ff488010ab0).stop 2019-12-05T06:06:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.579+0000 7ff496aa0700 1 -- 172.21.15.16:0/980193430 reap_dead start 2019-12-05T06:06:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.579+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 shutdown_connections 2019-12-05T06:06:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.579+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 shutdown_connections 2019-12-05T06:06:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.579+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 wait complete. 2019-12-05T06:06:50.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.579+0000 7ff497d02700 1 -- 172.21.15.16:0/980193430 >> 172.21.15.16:0/980193430 conn(0x7ff4900714f0 msgr2=0x7ff49009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:50.594 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2019-12-05T06:06:50.595 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2019-12-05T06:06:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.853+0000 7f3e4e892700 1 Processor -- start 2019-12-05T06:06:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.853+0000 7f3e4e892700 1 -- start start 2019-12-05T06:06:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.854+0000 7f3e4e892700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3e4808c6d0 0x7f3e4808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.855+0000 7f3e4e892700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e4808f720 0x7f3e4808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:50.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.855+0000 7f3e4e892700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4808fb80 con 0x7f3e4808fef0 2019-12-05T06:06:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.855+0000 7f3e4e892700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4808df90 con 0x7f3e4808f720 2019-12-05T06:06:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.855+0000 7f3e4e892700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e480a3f70 con 0x7f3e4808c6d0 2019-12-05T06:06:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.855+0000 7f3e4d630700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e4808f720 0x7f3e4808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.855+0000 7f3e4d630700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3e4808f720 0x7f3e4808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46344/0 (socket says 172.21.15.16:46344) 2019-12-05T06:06:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.855+0000 7f3e4d630700 1 -- 172.21.15.16:0/3734678582 learned_addr learned my addr 172.21.15.16:0/3734678582 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.855+0000 7f3e4ce2f700 1 --2- 172.21.15.16:0/3734678582 >> v2:172.21.15.76:3300/0 conn(0x7f3e4808c6d0 0x7f3e4808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.855+0000 7f3e4d630700 1 -- 172.21.15.16:0/3734678582 >> v2:172.21.15.76:3300/0 conn(0x7f3e4808c6d0 msgr2=0x7f3e4808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:50.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e4d630700 1 --2- 172.21.15.16:0/3734678582 >> v2:172.21.15.76:3300/0 conn(0x7f3e4808c6d0 0x7f3e4808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:50.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e4d630700 1 -- 172.21.15.16:0/3734678582 >> v1:172.21.15.76:6789/0 conn(0x7f3e4808fef0 legacy=0x7f3e4808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e4d630700 1 -- 172.21.15.16:0/3734678582 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4808d540 con 0x7f3e4808f720 2019-12-05T06:06:50.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e4d630700 1 --2- 172.21.15.16:0/3734678582 >> v2:172.21.15.16:3300/0 conn(0x7f3e4808f720 0x7f3e4808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3e3000f730 tx=0x7f3e30007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e4d630700 1 --2- 172.21.15.16:0/3734678582 >> v2:172.21.15.16:3300/0 conn(0x7f3e4808f720 0x7f3e4808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3e3000f730 tx=0x7f3e30007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e4d630700 1 --2- 172.21.15.16:0/3734678582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4808f720 0x7f3e4808d000 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7f3e3000f730 tx=0x7f3e30007d50).ready entity=mon.0 client_cookie=f0bbee8ec169eb68 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e4d630700 1 --2- 172.21.15.16:0/3734678582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4808f720 0x7f3e4808d000 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f3e3000f730 tx=0x7f3e30007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e4d630700 1 --2- 172.21.15.16:0/3734678582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4808f720 0x7f3e4808d000 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f3e3000f730 tx=0x7f3e30007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e477fe700 1 -- 172.21.15.16:0/3734678582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e30000a70 con 0x7f3e4808f720 2019-12-05T06:06:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e4d630700 1 --2- 172.21.15.16:0/3734678582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4808f720 0x7f3e4808d000 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7f3e3000f730 tx=0x7f3e30007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.856+0000 7f3e477fe700 1 -- 172.21.15.16:0/3734678582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e300010e0 con 0x7f3e4808f720 2019-12-05T06:06:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.857+0000 7f3e477fe700 1 -- 172.21.15.16:0/3734678582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e300017b0 con 0x7f3e4808f720 2019-12-05T06:06:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.857+0000 7f3e4e892700 1 -- 172.21.15.16:0/3734678582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4808f720 msgr2=0x7f3e4808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.857+0000 7f3e4e892700 1 --2- 172.21.15.16:0/3734678582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4808f720 0x7f3e4808d000 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7f3e3000f730 tx=0x7f3e30007d50).stop 2019-12-05T06:06:50.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.858+0000 7f3e4e892700 1 -- 172.21.15.16:0/3734678582 shutdown_connections 2019-12-05T06:06:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.858+0000 7f3e4e892700 1 --2- 172.21.15.16:0/3734678582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e4808f720 0x7f3e4808d000 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.858+0000 7f3e4e892700 1 --2- 172.21.15.16:0/3734678582 >> v2:172.21.15.76:3300/0 conn(0x7f3e4808c6d0 0x7f3e4808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.858+0000 7f3e4e892700 1 -- 172.21.15.16:0/3734678582 shutdown_connections 2019-12-05T06:06:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.858+0000 7f3e4e892700 1 -- 172.21.15.16:0/3734678582 wait complete. 2019-12-05T06:06:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.858+0000 7f3e4e892700 1 -- 172.21.15.16:0/3734678582 >> 172.21.15.16:0/3734678582 conn(0x7f3e48061940 msgr2=0x7f3e48096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.863+0000 7f3e4e892700 1 Processor -- start 2019-12-05T06:06:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.863+0000 7f3e4e892700 1 -- start start 2019-12-05T06:06:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.865+0000 7f3e4e892700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3e4814d7f0 0x7f3e48132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:50.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.865+0000 7f3e4e892700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e48098da0 con 0x7f3e4814d400 2019-12-05T06:06:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.865+0000 7f3e4e892700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e4809f0a0 con 0x7f3e48132d00 2019-12-05T06:06:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.865+0000 7f3e4e892700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e4809bb50 con 0x7f3e4814d7f0 2019-12-05T06:06:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.866+0000 7f3e4ce2f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e4814d400 0x7f3e48131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59746/0 (socket says 172.21.15.16:59746) 2019-12-05T06:06:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.866+0000 7f3e4ce2f700 1 -- 172.21.15.16:0/2485314320 learned_addr learned my addr 172.21.15.16:0/2485314320 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.866+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> v2:172.21.15.76:3300/0 conn(0x7f3e4814d7f0 0x7f3e48132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.866+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1713569721 0 0) 0x7f3e40001430 con 0x7f3e4814d400 2019-12-05T06:06:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.866+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e2c002e80 con 0x7f3e4814d400 2019-12-05T06:06:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.866+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41821115 0 0) 0x7f3e38001080 con 0x7f3e48132d00 2019-12-05T06:06:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.866+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e2c0030b0 con 0x7f3e48132d00 2019-12-05T06:06:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.866+0000 7f3e4d630700 1 -- 172.21.15.16:0/2485314320 >> v1:172.21.15.76:6789/0 conn(0x7f3e48132d00 legacy=0x7f3e481550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e4d630700 1 -- 172.21.15.16:0/2485314320 >> v1:172.21.15.16:6789/0 conn(0x7f3e4814d400 legacy=0x7f3e48131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e4d630700 1 -- 172.21.15.16:0/2485314320 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4809b0f0 con 0x7f3e4814d7f0 2019-12-05T06:06:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> v2:172.21.15.76:3300/0 conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> v2:172.21.15.76:3300/0 conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=READY pgs=108 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e300014a0 con 0x7f3e4814d7f0 2019-12-05T06:06:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e300010c0 con 0x7f3e4814d7f0 2019-12-05T06:06:50.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e30001a80 con 0x7f3e4814d7f0 2019-12-05T06:06:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.867+0000 7f3e4e892700 1 -- 172.21.15.16:0/2485314320 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4809b0f0 con 0x7f3e4814d7f0 2019-12-05T06:06:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.868+0000 7f3e4e892700 1 -- 172.21.15.16:0/2485314320 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4809b0f0 con 0x7f3e4814d7f0 2019-12-05T06:06:50.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.869+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.870+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3e300010c0 con 0x7f3e4814d7f0 2019-12-05T06:06:50.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.870+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.870+0000 7f3e457fa700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e2c023650 0x7f3e2c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:50.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.870+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f3e30045d90 con 0x7f3e4814d7f0 2019-12-05T06:06:50.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.870+0000 7f3e4ce2f700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e2c023650 0x7f3e2c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:50.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.871+0000 7f3e4ce2f700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e2c023650 0x7f3e2c025ae0 crc :-1 s=READY pgs=74 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:50.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.884+0000 7f3e4e892700 1 -- 172.21.15.16:0/2485314320 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e4809b0f0 con 0x7f3e4814d7f0 2019-12-05T06:06:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.893+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:50.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:50.893+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3e30000eb0 con 0x7f3e4814d7f0 2019-12-05T06:06:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.155+0000 7f3e4e892700 1 -- 172.21.15.16:0/2485314320 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f3e4809b0f0 con 0x7f3e4814d7f0 2019-12-05T06:06:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.156+0000 7f3e4d630700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=THROTTLE_DONE pgs=108 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.156+0000 7f3e457fa700 1 -- 172.21.15.16:0/2485314320 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3e300010c0 con 0x7f3e4814d7f0 2019-12-05T06:06:51.154 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2019-12-05T06:06:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.158+0000 7f3e4e892700 1 -- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e2c023650 msgr2=0x7f3e2c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.158+0000 7f3e4e892700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e2c023650 0x7f3e2c025ae0 crc :-1 s=READY pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.158+0000 7f3e4e892700 1 -- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 msgr2=0x7f3e48132530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.158+0000 7f3e4e892700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 0x7f3e48132530 secure :-1 s=READY pgs=108 cs=0 l=1 rx=0x7f3e30010350 tx=0x7f3e300106f0).stop 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.158+0000 7f3e4e892700 1 -- 172.21.15.16:0/2485314320 shutdown_connections 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.158+0000 7f3e4e892700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e2c023650 0x7f3e2c025ae0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.158+0000 7f3e4e892700 1 --2- 172.21.15.16:0/2485314320 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e4814d7f0 0x7f3e48132530 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.158+0000 7f3e4e892700 1 -- 172.21.15.16:0/2485314320 shutdown_connections 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.159+0000 7f3e4e892700 1 -- 172.21.15.16:0/2485314320 wait complete. 2019-12-05T06:06:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.159+0000 7f3e4e892700 1 -- 172.21.15.16:0/2485314320 >> 172.21.15.16:0/2485314320 conn(0x7f3e480714f0 msgr2=0x7f3e4809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:51.167 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2019-12-05T06:06:51.167 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2019-12-05T06:06:51.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.447+0000 7f97afe3a700 1 Processor -- start 2019-12-05T06:06:51.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.447+0000 7f97afe3a700 1 -- start start 2019-12-05T06:06:51.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.449+0000 7f97afe3a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 0x7f97a808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:51.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.449+0000 7f97afe3a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80958e0 con 0x7f97a808fef0 2019-12-05T06:06:51.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.449+0000 7f97afe3a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a80629e0 con 0x7f97a808c6d0 2019-12-05T06:06:51.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.449+0000 7f97afe3a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a808df90 con 0x7f97a808d2c0 2019-12-05T06:06:51.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.449+0000 7f97aebd8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f97a808fef0 0x7f97a808f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59756/0 (socket says 172.21.15.16:59756) 2019-12-05T06:06:51.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.449+0000 7f97aebd8700 1 -- 172.21.15.16:0/3951865909 learned_addr learned my addr 172.21.15.16:0/3951865909 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:51.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.449+0000 7f97adbd6700 1 --2- 172.21.15.16:0/3951865909 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 0x7f97a808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.450+0000 7f97ad3d5700 1 -- 172.21.15.16:0/3951865909 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228698623 0 0) 0x7f97a0001150 con 0x7f97a808fef0 2019-12-05T06:06:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.450+0000 7f97ad3d5700 1 -- 172.21.15.16:0/3951865909 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9790002e80 con 0x7f97a808fef0 2019-12-05T06:06:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.450+0000 7f97ad3d5700 1 -- 172.21.15.16:0/3951865909 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3148530092 0 0) 0x7f9798001080 con 0x7f97a808c6d0 2019-12-05T06:06:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.450+0000 7f97ad3d5700 1 -- 172.21.15.16:0/3951865909 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9790004170 con 0x7f97a808c6d0 2019-12-05T06:06:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.450+0000 7f97adbd6700 1 -- 172.21.15.16:0/3951865909 >> v1:172.21.15.76:6789/0 conn(0x7f97a808c6d0 legacy=0x7f97a808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.450+0000 7f97adbd6700 1 -- 172.21.15.16:0/3951865909 >> v1:172.21.15.16:6789/0 conn(0x7f97a808fef0 legacy=0x7f97a808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.450+0000 7f97adbd6700 1 -- 172.21.15.16:0/3951865909 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a808e620 con 0x7f97a808d2c0 2019-12-05T06:06:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97adbd6700 1 --2- 172.21.15.16:0/3951865909 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 0x7f97a808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f979c00b2d0 tx=0x7f979c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97adbd6700 1 --2- 172.21.15.16:0/3951865909 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 0x7f97a808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f979c00b2d0 tx=0x7f979c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97adbd6700 1 --2- 172.21.15.16:0/3951865909 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 0x7f97a808ef20 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f979c00b2d0 tx=0x7f979c00bae0).ready entity=mon.2 client_cookie=93cb70fbd3a241b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97adbd6700 1 --2- 172.21.15.16:0/3951865909 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 0x7f97a808ef20 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f979c00b2d0 tx=0x7f979c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97adbd6700 1 --2- 172.21.15.16:0/3951865909 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 0x7f97a808ef20 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f979c00b2d0 tx=0x7f979c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97ad3d5700 1 -- 172.21.15.16:0/3951865909 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f979c000ce0 con 0x7f97a808d2c0 2019-12-05T06:06:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97adbd6700 1 --2- 172.21.15.16:0/3951865909 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 0x7f97a808ef20 secure :-1 s=THROTTLE_DONE pgs=37 cs=0 l=1 rx=0x7f979c00b2d0 tx=0x7f979c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97ad3d5700 1 -- 172.21.15.16:0/3951865909 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979c009c60 con 0x7f97a808d2c0 2019-12-05T06:06:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97ad3d5700 1 -- 172.21.15.16:0/3951865909 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f979c00a3d0 con 0x7f97a808d2c0 2019-12-05T06:06:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97afe3a700 1 -- 172.21.15.16:0/3951865909 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 msgr2=0x7f97a808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.451+0000 7f97afe3a700 1 --2- 172.21.15.16:0/3951865909 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 0x7f97a808ef20 secure :-1 s=READY pgs=37 cs=0 l=1 rx=0x7f979c00b2d0 tx=0x7f979c00bae0).stop 2019-12-05T06:06:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.454+0000 7f97afe3a700 1 -- 172.21.15.16:0/3951865909 shutdown_connections 2019-12-05T06:06:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.454+0000 7f97afe3a700 1 --2- 172.21.15.16:0/3951865909 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a808d2c0 0x7f97a808ef20 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.454+0000 7f97afe3a700 1 -- 172.21.15.16:0/3951865909 shutdown_connections 2019-12-05T06:06:51.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.454+0000 7f97afe3a700 1 -- 172.21.15.16:0/3951865909 wait complete. 2019-12-05T06:06:51.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.454+0000 7f97afe3a700 1 -- 172.21.15.16:0/3951865909 >> 172.21.15.16:0/3951865909 conn(0x7f97a8061940 msgr2=0x7f97a8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.459+0000 7f97afe3a700 1 Processor -- start 2019-12-05T06:06:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.459+0000 7f97afe3a700 1 -- start start 2019-12-05T06:06:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.461+0000 7f97afe3a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f97a807bf00 0x7f97a814d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.461+0000 7f97afe3a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a807b730 0x7f97a8153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.461+0000 7f97afe3a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97a808ab00 con 0x7f97a814d410 2019-12-05T06:06:51.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.461+0000 7f97afe3a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97a8089f00 con 0x7f97a807bf00 2019-12-05T06:06:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.461+0000 7f97afe3a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97a8079fa0 con 0x7f97a807b730 2019-12-05T06:06:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.461+0000 7f97ae3d7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f97a814d410 0x7f97a807af30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59760/0 (socket says 172.21.15.16:59760) 2019-12-05T06:06:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.461+0000 7f97ae3d7700 1 -- 172.21.15.16:0/899992309 learned_addr learned my addr 172.21.15.16:0/899992309 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.461+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> v2:172.21.15.76:3300/0 conn(0x7f97a807bf00 0x7f97a814d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:51.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.461+0000 7f97aebd8700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a807b730 0x7f97a8153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.462+0000 7f97a6ffd700 1 -- 172.21.15.16:0/899992309 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079683440 0 0) 0x7f9798000d90 con 0x7f97a814d410 2019-12-05T06:06:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.462+0000 7f97a6ffd700 1 -- 172.21.15.16:0/899992309 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9788002e80 con 0x7f97a814d410 2019-12-05T06:06:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.462+0000 7f97adbd6700 1 -- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a807b730 msgr2=0x7f97a8153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.462+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a807b730 0x7f97a8153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.462+0000 7f97adbd6700 1 -- 172.21.15.16:0/899992309 >> v1:172.21.15.16:6789/0 conn(0x7f97a814d410 legacy=0x7f97a807af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.462+0000 7f97adbd6700 1 -- 172.21.15.16:0/899992309 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97a807a630 con 0x7f97a807bf00 2019-12-05T06:06:51.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.462+0000 7f97aebd8700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a807b730 0x7f97a8153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:51.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.462+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> v2:172.21.15.76:3300/0 conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.462+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> v2:172.21.15.76:3300/0 conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.462+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:51.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.463+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.463+0000 7f97a6ffd700 1 -- 172.21.15.16:0/899992309 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f979c000cc0 con 0x7f97a807bf00 2019-12-05T06:06:51.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.463+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.463+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.463+0000 7f97a6ffd700 1 -- 172.21.15.16:0/899992309 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979c009d00 con 0x7f97a807bf00 2019-12-05T06:06:51.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.463+0000 7f97a6ffd700 1 -- 172.21.15.16:0/899992309 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f979c00a390 con 0x7f97a807bf00 2019-12-05T06:06:51.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.463+0000 7f97afe3a700 1 -- 172.21.15.16:0/899992309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97a807a630 con 0x7f97a807bf00 2019-12-05T06:06:51.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.463+0000 7f97afe3a700 1 -- 172.21.15.16:0/899992309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97a807a630 con 0x7f97a807bf00 2019-12-05T06:06:51.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.464+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.464+0000 7f97a6ffd700 1 -- 172.21.15.16:0/899992309 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f979c009ef0 con 0x7f97a807bf00 2019-12-05T06:06:51.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.465+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.465+0000 7f97a6ffd700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9788023670 0x7f9788025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:51.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.465+0000 7f97a6ffd700 1 -- 172.21.15.16:0/899992309 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f979c0460c0 con 0x7f97a807bf00 2019-12-05T06:06:51.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.465+0000 7f97ae3d7700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9788023670 0x7f9788025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:51.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.466+0000 7f97ae3d7700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9788023670 0x7f9788025ae0 crc :-1 s=READY pgs=75 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:51.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.478+0000 7f97afe3a700 1 -- 172.21.15.16:0/899992309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97a807a630 con 0x7f97a807bf00 2019-12-05T06:06:51.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.487+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.487+0000 7f97a6ffd700 1 -- 172.21.15.16:0/899992309 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f979c009900 con 0x7f97a807bf00 2019-12-05T06:06:51.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.752+0000 7f97afe3a700 1 -- 172.21.15.16:0/899992309 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f97a807a630 con 0x7f97a807bf00 2019-12-05T06:06:51.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.753+0000 7f97adbd6700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=THROTTLE_DONE pgs=110 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:51.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.753+0000 7f97a6ffd700 1 -- 172.21.15.16:0/899992309 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f979c009ef0 con 0x7f97a807bf00 2019-12-05T06:06:51.752 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2019-12-05T06:06:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 -- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9788023670 msgr2=0x7f9788025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9788023670 0x7f9788025ae0 crc :-1 s=READY pgs=75 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 -- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 msgr2=0x7f97a814d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 0x7f97a814d800 secure :-1 s=READY pgs=110 cs=0 l=1 rx=0x7f979c010700 tx=0x7f979c010aa0).stop 2019-12-05T06:06:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 -- 172.21.15.16:0/899992309 shutdown_connections 2019-12-05T06:06:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9788023670 0x7f9788025ae0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f97a807bf00 0x7f97a814d800 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 --2- 172.21.15.16:0/899992309 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f97a807b730 0x7f97a8153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 -- 172.21.15.16:0/899992309 shutdown_connections 2019-12-05T06:06:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 -- 172.21.15.16:0/899992309 wait complete. 2019-12-05T06:06:51.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:51.757+0000 7f97afe3a700 1 -- 172.21.15.16:0/899992309 >> 172.21.15.16:0/899992309 conn(0x7f97a806dfd0 msgr2=0x7f97a8070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:51.772 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.1 2019-12-05T06:06:51.772 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2019-12-05T06:06:52.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.005+0000 7f57aadd6700 1 Processor -- start 2019-12-05T06:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.005+0000 7f57aadd6700 1 -- start start 2019-12-05T06:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.006+0000 7f57aadd6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 0x7f57a408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.007+0000 7f57aadd6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f57a408f720 0x7f57a408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.007+0000 7f57aadd6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57a40958e0 con 0x7f57a408fef0 2019-12-05T06:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.007+0000 7f57aadd6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a40a3f70 con 0x7f57a408f720 2019-12-05T06:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.007+0000 7f57aadd6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a408d830 con 0x7f57a408c6d0 2019-12-05T06:06:52.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.007+0000 7f57a8b72700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f57a408fef0 0x7f57a408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52840/0 (socket says 172.21.15.16:52840) 2019-12-05T06:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.007+0000 7f57a8b72700 1 -- 172.21.15.16:0/2524799032 learned_addr learned my addr 172.21.15.16:0/2524799032 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.007+0000 7f57a9b74700 1 --2- 172.21.15.16:0/2524799032 >> v2:172.21.15.76:3300/0 conn(0x7f57a408f720 0x7f57a408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.007+0000 7f57a9373700 1 --2- 172.21.15.16:0/2524799032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 0x7f57a408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f5797fff700 1 -- 172.21.15.16:0/2524799032 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 447604622 0 0) 0x7f57a0001190 con 0x7f57a408fef0 2019-12-05T06:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f5797fff700 1 -- 172.21.15.16:0/2524799032 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5790002e80 con 0x7f57a408fef0 2019-12-05T06:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f57a9373700 1 -- 172.21.15.16:0/2524799032 >> v2:172.21.15.76:3300/0 conn(0x7f57a408f720 msgr2=0x7f57a408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:52.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f57a9373700 1 --2- 172.21.15.16:0/2524799032 >> v2:172.21.15.76:3300/0 conn(0x7f57a408f720 0x7f57a408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f57a9373700 1 -- 172.21.15.16:0/2524799032 >> v1:172.21.15.76:6789/0 conn(0x7f57a408fef0 legacy=0x7f57a408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f57a9373700 1 -- 172.21.15.16:0/2524799032 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a408fbc0 con 0x7f57a408c6d0 2019-12-05T06:06:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f57a9b74700 1 --2- 172.21.15.16:0/2524799032 >> v2:172.21.15.76:3300/0 conn(0x7f57a408f720 0x7f57a408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f57a9373700 1 --2- 172.21.15.16:0/2524799032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 0x7f57a408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f579800af50 tx=0x7f579800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f57a9373700 1 --2- 172.21.15.16:0/2524799032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 0x7f57a408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f579800af50 tx=0x7f579800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f57a9373700 1 --2- 172.21.15.16:0/2524799032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 0x7f57a408cac0 secure :-1 s=READY pgs=38 cs=0 l=1 rx=0x7f579800af50 tx=0x7f579800b760).ready entity=mon.2 client_cookie=730f6effa844929f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:52.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.008+0000 7f57a9373700 1 --2- 172.21.15.16:0/2524799032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 0x7f57a408cac0 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f579800af50 tx=0x7f579800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.009+0000 7f57a9373700 1 --2- 172.21.15.16:0/2524799032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 0x7f57a408cac0 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f579800af50 tx=0x7f579800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.009+0000 7f57a9373700 1 --2- 172.21.15.16:0/2524799032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 0x7f57a408cac0 secure :-1 s=THROTTLE_DONE pgs=38 cs=0 l=1 rx=0x7f579800af50 tx=0x7f579800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.009+0000 7f5797fff700 1 -- 172.21.15.16:0/2524799032 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5798007210 con 0x7f57a408c6d0 2019-12-05T06:06:52.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.009+0000 7f5797fff700 1 -- 172.21.15.16:0/2524799032 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57980099b0 con 0x7f57a408c6d0 2019-12-05T06:06:52.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.009+0000 7f5797fff700 1 -- 172.21.15.16:0/2524799032 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f579800a040 con 0x7f57a408c6d0 2019-12-05T06:06:52.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.009+0000 7f57aadd6700 1 -- 172.21.15.16:0/2524799032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 msgr2=0x7f57a408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.009+0000 7f57aadd6700 1 --2- 172.21.15.16:0/2524799032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 0x7f57a408cac0 secure :-1 s=READY pgs=38 cs=0 l=1 rx=0x7f579800af50 tx=0x7f579800b760).stop 2019-12-05T06:06:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.010+0000 7f57aadd6700 1 -- 172.21.15.16:0/2524799032 shutdown_connections 2019-12-05T06:06:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.010+0000 7f57aadd6700 1 --2- 172.21.15.16:0/2524799032 >> v2:172.21.15.76:3300/0 conn(0x7f57a408f720 0x7f57a408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.010+0000 7f57aadd6700 1 --2- 172.21.15.16:0/2524799032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a408c6d0 0x7f57a408cac0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.011+0000 7f57aadd6700 1 -- 172.21.15.16:0/2524799032 shutdown_connections 2019-12-05T06:06:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.011+0000 7f57aadd6700 1 -- 172.21.15.16:0/2524799032 wait complete. 2019-12-05T06:06:52.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.011+0000 7f57aadd6700 1 -- 172.21.15.16:0/2524799032 >> 172.21.15.16:0/2524799032 conn(0x7f57a4061940 msgr2=0x7f57a4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:52.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.016+0000 7f57aadd6700 1 Processor -- start 2019-12-05T06:06:52.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.016+0000 7f57aadd6700 1 -- start start 2019-12-05T06:06:52.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.017+0000 7f57aadd6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57a4096e90 0x7f57a409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:52.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.017+0000 7f57aadd6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a4084f30 0x7f57a409b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:52.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.017+0000 7f57aadd6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f57a4097280 0x7f57a40976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:52.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.017+0000 7f57aadd6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a409a450 con 0x7f57a4096e90 2019-12-05T06:06:52.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.017+0000 7f57aadd6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57a4099a20 con 0x7f57a4097280 2019-12-05T06:06:52.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.017+0000 7f57aadd6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57a4099cf0 con 0x7f57a4084f30 2019-12-05T06:06:52.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a9373700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57a4096e90 0x7f57a409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a9373700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57a4096e90 0x7f57a409b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46374/0 (socket says 172.21.15.16:46374) 2019-12-05T06:06:52.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a9373700 1 -- 172.21.15.16:0/1161482355 learned_addr learned my addr 172.21.15.16:0/1161482355 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:52.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a9b74700 1 --2- 172.21.15.16:0/1161482355 >> v2:172.21.15.76:3300/0 conn(0x7f57a4097280 0x7f57a40976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a8b72700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a4084f30 0x7f57a409b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a9373700 1 -- 172.21.15.16:0/1161482355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a4084f30 msgr2=0x7f57a409b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a4084f30 0x7f57a409b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a9373700 1 -- 172.21.15.16:0/1161482355 >> v2:172.21.15.76:3300/0 conn(0x7f57a4097280 msgr2=0x7f57a40976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> v2:172.21.15.76:3300/0 conn(0x7f57a4097280 0x7f57a40976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a9373700 1 -- 172.21.15.16:0/1161482355 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a409aae0 con 0x7f57a4096e90 2019-12-05T06:06:52.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.018+0000 7f57a9b74700 1 --2- 172.21.15.16:0/1161482355 >> v2:172.21.15.76:3300/0 conn(0x7f57a4097280 0x7f57a40976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:52.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> v2:172.21.15.16:3300/0 conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> v2:172.21.15.16:3300/0 conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=READY pgs=134 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:52.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f5795ffb700 1 -- 172.21.15.16:0/1161482355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f579800abb0 con 0x7f57a4096e90 2019-12-05T06:06:52.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f5795ffb700 1 -- 172.21.15.16:0/1161482355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5798009580 con 0x7f57a4096e90 2019-12-05T06:06:52.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f57aadd6700 1 -- 172.21.15.16:0/1161482355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57a409aae0 con 0x7f57a4096e90 2019-12-05T06:06:52.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f5795ffb700 1 -- 172.21.15.16:0/1161482355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5798009c10 con 0x7f57a4096e90 2019-12-05T06:06:52.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.019+0000 7f57aadd6700 1 -- 172.21.15.16:0/1161482355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57a409aae0 con 0x7f57a4096e90 2019-12-05T06:06:52.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.020+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.021+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.021+0000 7f5795ffb700 1 -- 172.21.15.16:0/1161482355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5798009770 con 0x7f57a4096e90 2019-12-05T06:06:52.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.021+0000 7f5795ffb700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5784023630 0x7f5784025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:52.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.021+0000 7f5795ffb700 1 -- 172.21.15.16:0/1161482355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f5798045fa0 con 0x7f57a4096e90 2019-12-05T06:06:52.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.022+0000 7f57a8b72700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5784023630 0x7f5784025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.022+0000 7f57a8b72700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5784023630 0x7f5784025ac0 crc :-1 s=READY pgs=76 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:52.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.036+0000 7f57aadd6700 1 -- 172.21.15.16:0/1161482355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a409aae0 con 0x7f57a4096e90 2019-12-05T06:06:52.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.044+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.044+0000 7f5795ffb700 1 -- 172.21.15.16:0/1161482355 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f579800b4f0 con 0x7f57a4096e90 2019-12-05T06:06:52.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.276+0000 7f57aadd6700 1 -- 172.21.15.16:0/1161482355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f57a409aae0 con 0x7f57a4096e90 2019-12-05T06:06:52.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.277+0000 7f57a9373700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.277+0000 7f5795ffb700 1 -- 172.21.15.16:0/1161482355 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5798009580 con 0x7f57a4096e90 2019-12-05T06:06:52.276 INFO:teuthology.orchestra.run.smithi016.stdout:55834574854 2019-12-05T06:06:52.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.279+0000 7f57aadd6700 1 -- 172.21.15.16:0/1161482355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5784023630 msgr2=0x7f5784025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.279+0000 7f57aadd6700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5784023630 0x7f5784025ac0 crc :-1 s=READY pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.279+0000 7f57aadd6700 1 -- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 msgr2=0x7f57a409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.279+0000 7f57aadd6700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 0x7f57a409b3e0 secure :-1 s=READY pgs=134 cs=0 l=1 rx=0x7f57980103a0 tx=0x7f5798010740).stop 2019-12-05T06:06:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.280+0000 7f57aadd6700 1 -- 172.21.15.16:0/1161482355 shutdown_connections 2019-12-05T06:06:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.280+0000 7f57aadd6700 1 --2- 172.21.15.16:0/1161482355 >> v2:172.21.15.76:3300/0 conn(0x7f57a4097280 0x7f57a40976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.280+0000 7f57aadd6700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57a4084f30 0x7f57a409b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.280+0000 7f57aadd6700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5784023630 0x7f5784025ac0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.280+0000 7f57aadd6700 1 --2- 172.21.15.16:0/1161482355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57a4096e90 0x7f57a409b3e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.280+0000 7f57aadd6700 1 -- 172.21.15.16:0/1161482355 shutdown_connections 2019-12-05T06:06:52.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.280+0000 7f57aadd6700 1 -- 172.21.15.16:0/1161482355 wait complete. 2019-12-05T06:06:52.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.280+0000 7f57aadd6700 1 -- 172.21.15.16:0/1161482355 >> 172.21.15.16:0/1161482355 conn(0x7f57a40714f0 msgr2=0x7f57a409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:52.288 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.2 2019-12-05T06:06:52.288 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2019-12-05T06:06:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.567+0000 7f6c81ec8700 1 Processor -- start 2019-12-05T06:06:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.567+0000 7f6c81ec8700 1 -- start start 2019-12-05T06:06:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.569+0000 7f6c81ec8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6c7c08d2c0 0x7f6c7c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:52.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.569+0000 7f6c81ec8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c08fef0 0x7f6c7c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.569+0000 7f6c81ec8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7c0958e0 con 0x7f6c7c08c6d0 2019-12-05T06:06:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.569+0000 7f6c81ec8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7c08df90 con 0x7f6c7c08d2c0 2019-12-05T06:06:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.569+0000 7f6c81ec8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c0a3f70 con 0x7f6c7c08fef0 2019-12-05T06:06:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.569+0000 7f6c7bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6c7c08c6d0 0x7f6c7c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59782/0 (socket says 172.21.15.16:59782) 2019-12-05T06:06:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.569+0000 7f6c7bfff700 1 -- 172.21.15.16:0/167051634 learned_addr learned my addr 172.21.15.16:0/167051634 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.569+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/167051634 >> v2:172.21.15.76:3300/0 conn(0x7f6c7c08d2c0 0x7f6c7c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.569+0000 7f6c80c66700 1 --2- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c08fef0 0x7f6c7c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.570+0000 7f6c7affd700 1 -- 172.21.15.16:0/167051634 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141000056 0 0) 0x7f6c6c0014d0 con 0x7f6c7c08c6d0 2019-12-05T06:06:52.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.570+0000 7f6c7affd700 1 -- 172.21.15.16:0/167051634 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c68002e80 con 0x7f6c7c08c6d0 2019-12-05T06:06:52.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.570+0000 7f6c7b7fe700 1 -- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c08fef0 msgr2=0x7f6c7c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:52.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.570+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c08fef0 0x7f6c7c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.570+0000 7f6c7b7fe700 1 -- 172.21.15.16:0/167051634 >> v1:172.21.15.16:6789/0 conn(0x7f6c7c08c6d0 legacy=0x7f6c7c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.570+0000 7f6c7b7fe700 1 -- 172.21.15.16:0/167051634 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c08e620 con 0x7f6c7c08d2c0 2019-12-05T06:06:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.570+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/167051634 >> v2:172.21.15.76:3300/0 conn(0x7f6c7c08d2c0 0x7f6c7c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6c6400af50 tx=0x7f6c6400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.571+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/167051634 >> v2:172.21.15.76:3300/0 conn(0x7f6c7c08d2c0 0x7f6c7c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6c6400af50 tx=0x7f6c6400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.571+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c7c08d2c0 0x7f6c7c08ef20 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7f6c6400af50 tx=0x7f6c6400b250).ready entity=mon.1 client_cookie=2de63af394b85744 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.571+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c7c08d2c0 0x7f6c7c08ef20 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f6c6400af50 tx=0x7f6c6400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.571+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c7c08d2c0 0x7f6c7c08ef20 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f6c6400af50 tx=0x7f6c6400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.571+0000 7f6c7affd700 1 -- 172.21.15.16:0/167051634 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c64007210 con 0x7f6c7c08d2c0 2019-12-05T06:06:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.571+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c7c08d2c0 0x7f6c7c08ef20 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f6c6400af50 tx=0x7f6c6400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.571+0000 7f6c7affd700 1 -- 172.21.15.16:0/167051634 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c640099b0 con 0x7f6c7c08d2c0 2019-12-05T06:06:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.571+0000 7f6c7affd700 1 -- 172.21.15.16:0/167051634 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c6400a080 con 0x7f6c7c08d2c0 2019-12-05T06:06:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.571+0000 7f6c81ec8700 1 -- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c7c08d2c0 msgr2=0x7f6c7c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.571+0000 7f6c81ec8700 1 --2- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c7c08d2c0 0x7f6c7c08ef20 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7f6c6400af50 tx=0x7f6c6400b250).stop 2019-12-05T06:06:52.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.573+0000 7f6c81ec8700 1 -- 172.21.15.16:0/167051634 shutdown_connections 2019-12-05T06:06:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.573+0000 7f6c81ec8700 1 --2- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6c7c08d2c0 0x7f6c7c08ef20 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.573+0000 7f6c81ec8700 1 --2- 172.21.15.16:0/167051634 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c08fef0 0x7f6c7c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.573+0000 7f6c81ec8700 1 -- 172.21.15.16:0/167051634 shutdown_connections 2019-12-05T06:06:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.573+0000 7f6c81ec8700 1 -- 172.21.15.16:0/167051634 wait complete. 2019-12-05T06:06:52.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.573+0000 7f6c81ec8700 1 -- 172.21.15.16:0/167051634 >> 172.21.15.16:0/167051634 conn(0x7f6c7c061940 msgr2=0x7f6c7c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.579+0000 7f6c81ec8700 1 Processor -- start 2019-12-05T06:06:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.579+0000 7f6c81ec8700 1 -- start start 2019-12-05T06:06:52.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.581+0000 7f6c81ec8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c12ca70 0x7f6c7c09d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:52.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.581+0000 7f6c81ec8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:52.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.581+0000 7f6c81ec8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7c130410 con 0x7f6c7c09e270 2019-12-05T06:06:52.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.581+0000 7f6c81ec8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7c1311f0 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.581+0000 7f6c81ec8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c1314c0 con 0x7f6c7c12ca70 2019-12-05T06:06:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.581+0000 7f6c7b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.581+0000 7f6c7b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46390/0 (socket says 172.21.15.16:46390) 2019-12-05T06:06:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.581+0000 7f6c7b7fe700 1 -- 172.21.15.16:0/3654047870 learned_addr learned my addr 172.21.15.16:0/3654047870 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.581+0000 7f6c7bfff700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c12ca70 0x7f6c7c09d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.582+0000 7f6c7b7fe700 1 -- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c12ca70 msgr2=0x7f6c7c09d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.582+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c12ca70 0x7f6c7c09d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.582+0000 7f6c7b7fe700 1 -- 172.21.15.16:0/3654047870 >> v1:172.21.15.76:6789/0 conn(0x7f6c7c09e270 legacy=0x7f6c7c06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.582+0000 7f6c78ff9700 1 -- 172.21.15.16:0/3654047870 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3842406390 0 0) 0x7f6c7400a8a0 con 0x7f6c7c09e270 2019-12-05T06:06:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.582+0000 7f6c7b7fe700 1 -- 172.21.15.16:0/3654047870 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c1322b0 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.582+0000 7f6c7bfff700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c12ca70 0x7f6c7c09d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.582+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> v2:172.21.15.16:3300/0 conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.582+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> v2:172.21.15.16:3300/0 conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.582+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.583+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.583+0000 7f6c78ff9700 1 -- 172.21.15.16:0/3654047870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c6400abb0 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.583+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.583+0000 7f6c78ff9700 1 -- 172.21.15.16:0/3654047870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c64000940 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.583+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.583+0000 7f6c81ec8700 1 -- 172.21.15.16:0/3654047870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c1322b0 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.583+0000 7f6c78ff9700 1 -- 172.21.15.16:0/3654047870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c6400abb0 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.583+0000 7f6c81ec8700 1 -- 172.21.15.16:0/3654047870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c1322b0 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.584+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.585+0000 7f6c78ff9700 1 -- 172.21.15.16:0/3654047870 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6c64009580 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.585+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.585+0000 7f6c78ff9700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c60023630 0x7f6c60025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.585+0000 7f6c78ff9700 1 -- 172.21.15.16:0/3654047870 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f6c64045e20 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.585+0000 7f6c7bfff700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c60023630 0x7f6c60025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.586+0000 7f6c7bfff700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c60023630 0x7f6c60025ac0 crc :-1 s=READY pgs=77 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:52.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.598+0000 7f6c81ec8700 1 -- 172.21.15.16:0/3654047870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c1322b0 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.607+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.607+0000 7f6c78ff9700 1 -- 172.21.15.16:0/3654047870 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6c6400d5e0 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.884+0000 7f6c81ec8700 1 -- 172.21.15.16:0/3654047870 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6c7c1322b0 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.884+0000 7f6c7b7fe700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:52.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.884+0000 7f6c78ff9700 1 -- 172.21.15.16:0/3654047870 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6c64009580 con 0x7f6c7c09e7c0 2019-12-05T06:06:52.883 INFO:teuthology.orchestra.run.smithi016.stdout:55834574857 2019-12-05T06:06:52.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.887+0000 7f6c81ec8700 1 -- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c60023630 msgr2=0x7f6c60025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.887+0000 7f6c81ec8700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c60023630 0x7f6c60025ac0 crc :-1 s=READY pgs=77 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.887+0000 7f6c81ec8700 1 -- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 msgr2=0x7f6c7c09dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:52.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.887+0000 7f6c81ec8700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7f6c64010460 tx=0x7f6c64010800).stop 2019-12-05T06:06:52.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.887+0000 7f6c81ec8700 1 -- 172.21.15.16:0/3654047870 shutdown_connections 2019-12-05T06:06:52.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.887+0000 7f6c81ec8700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c60023630 0x7f6c60025ac0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.887+0000 7f6c81ec8700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c7c09e7c0 0x7f6c7c09dd30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.887+0000 7f6c81ec8700 1 --2- 172.21.15.16:0/3654047870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c7c12ca70 0x7f6c7c09d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:52.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.888+0000 7f6c81ec8700 1 -- 172.21.15.16:0/3654047870 shutdown_connections 2019-12-05T06:06:52.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.888+0000 7f6c81ec8700 1 -- 172.21.15.16:0/3654047870 wait complete. 2019-12-05T06:06:52.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:52.888+0000 7f6c81ec8700 1 -- 172.21.15.16:0/3654047870 >> 172.21.15.16:0/3654047870 conn(0x7f6c7c0a2850 msgr2=0x7f6c7c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:52.900 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574857 got 55834574857 for osd.3 2019-12-05T06:06:52.901 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2019-12-05T06:06:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.099+0000 7fa7ffb83700 1 Processor -- start 2019-12-05T06:06:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.099+0000 7fa7ffb83700 1 -- start start 2019-12-05T06:06:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.100+0000 7fa7ffb83700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa7f808fef0 0x7fa7f808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.100+0000 7fa7ffb83700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f80958e0 con 0x7fa7f808d2c0 2019-12-05T06:06:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.101+0000 7fa7ffb83700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f80629e0 con 0x7fa7f808c6d0 2019-12-05T06:06:53.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.101+0000 7fa7ffb83700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f808df90 con 0x7fa7f808fef0 2019-12-05T06:06:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.101+0000 7fa7fd91f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa7f808d2c0 0x7fa7f808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59800/0 (socket says 172.21.15.16:59800) 2019-12-05T06:06:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.101+0000 7fa7fd91f700 1 -- 172.21.15.16:0/3184456580 learned_addr learned my addr 172.21.15.16:0/3184456580 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.101+0000 7fa7fe921700 1 --2- 172.21.15.16:0/3184456580 >> v2:172.21.15.76:3300/0 conn(0x7fa7f808fef0 0x7fa7f808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.101+0000 7fa7fd11e700 1 -- 172.21.15.16:0/3184456580 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111834592 0 0) 0x7fa7ec001100 con 0x7fa7f808d2c0 2019-12-05T06:06:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.101+0000 7fa7fd11e700 1 -- 172.21.15.16:0/3184456580 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7e0002e80 con 0x7fa7f808d2c0 2019-12-05T06:06:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.102+0000 7fa7fe921700 1 -- 172.21.15.16:0/3184456580 >> v1:172.21.15.76:6789/0 conn(0x7fa7f808c6d0 legacy=0x7fa7f808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.102+0000 7fa7fd11e700 1 -- 172.21.15.16:0/3184456580 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2019799809 0 0) 0x7fa7f0001400 con 0x7fa7f808c6d0 2019-12-05T06:06:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.102+0000 7fa7fe921700 1 -- 172.21.15.16:0/3184456580 >> v1:172.21.15.16:6789/0 conn(0x7fa7f808d2c0 legacy=0x7fa7f808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.102+0000 7fa7fe921700 1 -- 172.21.15.16:0/3184456580 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f808e620 con 0x7fa7f808fef0 2019-12-05T06:06:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.102+0000 7fa7fe921700 1 --2- 172.21.15.16:0/3184456580 >> v2:172.21.15.76:3300/0 conn(0x7fa7f808fef0 0x7fa7f808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa7e800af80 tx=0x7fa7e800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.102+0000 7fa7fe921700 1 --2- 172.21.15.16:0/3184456580 >> v2:172.21.15.76:3300/0 conn(0x7fa7f808fef0 0x7fa7f808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa7e800af80 tx=0x7fa7e800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.102+0000 7fa7fe921700 1 --2- 172.21.15.16:0/3184456580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f808fef0 0x7fa7f808f720 secure :-1 s=READY pgs=115 cs=0 l=1 rx=0x7fa7e800af80 tx=0x7fa7e800b790).ready entity=mon.1 client_cookie=b07a8cf0b98e913b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.102+0000 7fa7fe921700 1 --2- 172.21.15.16:0/3184456580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f808fef0 0x7fa7f808f720 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7fa7e800af80 tx=0x7fa7e800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.103+0000 7fa7fd11e700 1 -- 172.21.15.16:0/3184456580 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7e8007240 con 0x7fa7f808fef0 2019-12-05T06:06:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.103+0000 7fa7fe921700 1 --2- 172.21.15.16:0/3184456580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f808fef0 0x7fa7f808f720 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7fa7e800af80 tx=0x7fa7e800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.103+0000 7fa7fe921700 1 --2- 172.21.15.16:0/3184456580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f808fef0 0x7fa7f808f720 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7fa7e800af80 tx=0x7fa7e800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.103+0000 7fa7fd11e700 1 -- 172.21.15.16:0/3184456580 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e8000bb0 con 0x7fa7f808fef0 2019-12-05T06:06:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.103+0000 7fa7fd11e700 1 -- 172.21.15.16:0/3184456580 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7e8009e60 con 0x7fa7f808fef0 2019-12-05T06:06:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.103+0000 7fa7ffb83700 1 -- 172.21.15.16:0/3184456580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f808fef0 msgr2=0x7fa7f808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.103+0000 7fa7ffb83700 1 --2- 172.21.15.16:0/3184456580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f808fef0 0x7fa7f808f720 secure :-1 s=READY pgs=115 cs=0 l=1 rx=0x7fa7e800af80 tx=0x7fa7e800b790).stop 2019-12-05T06:06:53.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.105+0000 7fa7ffb83700 1 -- 172.21.15.16:0/3184456580 shutdown_connections 2019-12-05T06:06:53.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.105+0000 7fa7ffb83700 1 --2- 172.21.15.16:0/3184456580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa7f808fef0 0x7fa7f808f720 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.105+0000 7fa7ffb83700 1 -- 172.21.15.16:0/3184456580 shutdown_connections 2019-12-05T06:06:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.105+0000 7fa7ffb83700 1 -- 172.21.15.16:0/3184456580 wait complete. 2019-12-05T06:06:53.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.105+0000 7fa7ffb83700 1 -- 172.21.15.16:0/3184456580 >> 172.21.15.16:0/3184456580 conn(0x7fa7f8061940 msgr2=0x7fa7f8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:53.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.109+0000 7fa7ffb83700 1 Processor -- start 2019-12-05T06:06:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.109+0000 7fa7ffb83700 1 -- start start 2019-12-05T06:06:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.110+0000 7fa7ffb83700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.110+0000 7fa7ffb83700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f809c360 con 0x7fa7f809c700 2019-12-05T06:06:53.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.110+0000 7fa7ffb83700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f813d490 con 0x7fa7f8082680 2019-12-05T06:06:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.110+0000 7fa7ffb83700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f813f150 con 0x7fa7f809bf30 2019-12-05T06:06:53.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.110+0000 7fa7fd91f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa7f809c700 0x7fa7f8156d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59802/0 (socket says 172.21.15.16:59802) 2019-12-05T06:06:53.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.110+0000 7fa7fd91f700 1 -- 172.21.15.16:0/1546338495 learned_addr learned my addr 172.21.15.16:0/1546338495 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:53.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.110+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:53.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.111+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3440375253 0 0) 0x7fa7ec000c90 con 0x7fa7f809c700 2019-12-05T06:06:53.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.111+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d8002e80 con 0x7fa7f809c700 2019-12-05T06:06:53.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.111+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 228757373 0 0) 0x7fa7f00015c0 con 0x7fa7f8082680 2019-12-05T06:06:53.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.111+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7d80030b0 con 0x7fa7f8082680 2019-12-05T06:06:53.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.111+0000 7fa7fe921700 1 -- 172.21.15.16:0/1546338495 >> v1:172.21.15.76:6789/0 conn(0x7fa7f8082680 legacy=0x7fa7f809b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.111+0000 7fa7fe921700 1 -- 172.21.15.16:0/1546338495 >> v1:172.21.15.16:6789/0 conn(0x7fa7f809c700 legacy=0x7fa7f8156d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.111+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1138421530 0 0) 0x7fa7ec001820 con 0x7fa7f809c700 2019-12-05T06:06:53.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.111+0000 7fa7fe921700 1 -- 172.21.15.16:0/1546338495 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f813f7e0 con 0x7fa7f809bf30 2019-12-05T06:06:53.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.111+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.112+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.112+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.112+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.112+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7e800aee0 con 0x7fa7f809bf30 2019-12-05T06:06:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.112+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.112+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.112+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7e800b0e0 con 0x7fa7f809bf30 2019-12-05T06:06:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.112+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa7e8009d90 con 0x7fa7f809bf30 2019-12-05T06:06:53.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.112+0000 7fa7ffb83700 1 -- 172.21.15.16:0/1546338495 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f813f7e0 con 0x7fa7f809bf30 2019-12-05T06:06:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.112+0000 7fa7ffb83700 1 -- 172.21.15.16:0/1546338495 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f813f7e0 con 0x7fa7f809bf30 2019-12-05T06:06:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.113+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.114+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa7e80097c0 con 0x7fa7f809bf30 2019-12-05T06:06:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.114+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.114+0000 7fa7f6ffd700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7d8023650 0x7fa7d8025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:53.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.115+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa7e8045d90 con 0x7fa7f809bf30 2019-12-05T06:06:53.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.115+0000 7fa7fe120700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7d8023650 0x7fa7d8025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:53.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.115+0000 7fa7fe120700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7d8023650 0x7fa7d8025ae0 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:53.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.131+0000 7fa7ffb83700 1 -- 172.21.15.16:0/1546338495 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f813f7e0 con 0x7fa7f809bf30 2019-12-05T06:06:53.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.141+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.141+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa7e8029fa0 con 0x7fa7f809bf30 2019-12-05T06:06:53.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.426+0000 7fa7ffb83700 1 -- 172.21.15.16:0/1546338495 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa7f813f7e0 con 0x7fa7f809bf30 2019-12-05T06:06:53.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.426+0000 7fa7fe921700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=THROTTLE_DONE pgs=39 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.426+0000 7fa7f6ffd700 1 -- 172.21.15.16:0/1546338495 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa7e8029fa0 con 0x7fa7f809bf30 2019-12-05T06:06:53.425 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2019-12-05T06:06:53.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.429+0000 7fa7ffb83700 1 -- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7d8023650 msgr2=0x7fa7d8025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.429+0000 7fa7ffb83700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7d8023650 0x7fa7d8025ae0 crc :-1 s=READY pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.429+0000 7fa7ffb83700 1 -- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 msgr2=0x7fa7f815a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.429+0000 7fa7ffb83700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 secure :-1 s=READY pgs=39 cs=0 l=1 rx=0x7fa7e8010390 tx=0x7fa7e8010730).stop 2019-12-05T06:06:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.429+0000 7fa7ffb83700 1 -- 172.21.15.16:0/1546338495 shutdown_connections 2019-12-05T06:06:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.429+0000 7fa7ffb83700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7d8023650 0x7fa7d8025ae0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:53.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.429+0000 7fa7ffb83700 1 --2- 172.21.15.16:0/1546338495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa7f809bf30 0x7fa7f815a590 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:53.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.429+0000 7fa7ffb83700 1 -- 172.21.15.16:0/1546338495 shutdown_connections 2019-12-05T06:06:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.429+0000 7fa7ffb83700 1 -- 172.21.15.16:0/1546338495 wait complete. 2019-12-05T06:06:53.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.429+0000 7fa7ffb83700 1 -- 172.21.15.16:0/1546338495 >> 172.21.15.16:0/1546338495 conn(0x7fa7f809ab40 msgr2=0x7fa7f8099f80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:53.445 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.4 2019-12-05T06:06:53.445 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2019-12-05T06:06:53.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.700+0000 7faef4ac8700 1 Processor -- start 2019-12-05T06:06:53.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.700+0000 7faef4ac8700 1 -- start start 2019-12-05T06:06:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.703+0000 7faef4ac8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faef008c6d0 0x7faef008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.703+0000 7faef4ac8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef008fb80 con 0x7faef008f720 2019-12-05T06:06:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.703+0000 7faef4ac8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef00958e0 con 0x7faef008fef0 2019-12-05T06:06:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.703+0000 7faef4ac8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef008df90 con 0x7faef008c6d0 2019-12-05T06:06:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.703+0000 7faeeed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faef008c6d0 0x7faef008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.703+0000 7faeef59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faef008f720 0x7faef01533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59814/0 (socket says 172.21.15.16:59814) 2019-12-05T06:06:53.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.703+0000 7faeeed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faef008c6d0 0x7faef008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46410/0 (socket says 172.21.15.16:46410) 2019-12-05T06:06:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.703+0000 7faeef59e700 1 -- 172.21.15.16:0/1762995280 learned_addr learned my addr 172.21.15.16:0/1762995280 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.704+0000 7faeeed9d700 1 -- 172.21.15.16:0/1762995280 >> v1:172.21.15.76:6789/0 conn(0x7faef008fef0 legacy=0x7faef008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.704+0000 7faeedd9b700 1 -- 172.21.15.16:0/1762995280 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 848635870 0 0) 0x7faed8001400 con 0x7faef008fef0 2019-12-05T06:06:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.704+0000 7faeeed9d700 1 -- 172.21.15.16:0/1762995280 >> v1:172.21.15.16:6789/0 conn(0x7faef008f720 legacy=0x7faef01533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.704+0000 7faeeed9d700 1 -- 172.21.15.16:0/1762995280 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef008e620 con 0x7faef008c6d0 2019-12-05T06:06:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.704+0000 7faeeed9d700 1 --2- 172.21.15.16:0/1762995280 >> v2:172.21.15.16:3300/0 conn(0x7faef008c6d0 0x7faef008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faee000af50 tx=0x7faee000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.704+0000 7faeeed9d700 1 --2- 172.21.15.16:0/1762995280 >> v2:172.21.15.16:3300/0 conn(0x7faef008c6d0 0x7faef008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faee000af50 tx=0x7faee000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.704+0000 7faeeed9d700 1 --2- 172.21.15.16:0/1762995280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef008c6d0 0x7faef008cac0 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7faee000af50 tx=0x7faee000b760).ready entity=mon.0 client_cookie=f7e4271967f9ada server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.704+0000 7faeeed9d700 1 --2- 172.21.15.16:0/1762995280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef008c6d0 0x7faef008cac0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7faee000af50 tx=0x7faee000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.705+0000 7faeedd9b700 1 -- 172.21.15.16:0/1762995280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faee0000960 con 0x7faef008c6d0 2019-12-05T06:06:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.705+0000 7faeeed9d700 1 --2- 172.21.15.16:0/1762995280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef008c6d0 0x7faef008cac0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7faee000af50 tx=0x7faee000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.705+0000 7faeeed9d700 1 --2- 172.21.15.16:0/1762995280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef008c6d0 0x7faef008cac0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7faee000af50 tx=0x7faee000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.705+0000 7faeedd9b700 1 -- 172.21.15.16:0/1762995280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee0009860 con 0x7faef008c6d0 2019-12-05T06:06:53.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.705+0000 7faeedd9b700 1 -- 172.21.15.16:0/1762995280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faee0009f30 con 0x7faef008c6d0 2019-12-05T06:06:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.705+0000 7faef4ac8700 1 -- 172.21.15.16:0/1762995280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef008c6d0 msgr2=0x7faef008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.705+0000 7faef4ac8700 1 --2- 172.21.15.16:0/1762995280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef008c6d0 0x7faef008cac0 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7faee000af50 tx=0x7faee000b760).stop 2019-12-05T06:06:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.707+0000 7faef4ac8700 1 -- 172.21.15.16:0/1762995280 shutdown_connections 2019-12-05T06:06:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.707+0000 7faef4ac8700 1 --2- 172.21.15.16:0/1762995280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faef008c6d0 0x7faef008cac0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.707+0000 7faef4ac8700 1 -- 172.21.15.16:0/1762995280 shutdown_connections 2019-12-05T06:06:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.707+0000 7faef4ac8700 1 -- 172.21.15.16:0/1762995280 wait complete. 2019-12-05T06:06:53.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.707+0000 7faef4ac8700 1 -- 172.21.15.16:0/1762995280 >> 172.21.15.16:0/1762995280 conn(0x7faef0061940 msgr2=0x7faef0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.713+0000 7faef4ac8700 1 Processor -- start 2019-12-05T06:06:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.713+0000 7faef4ac8700 1 -- start start 2019-12-05T06:06:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.714+0000 7faef4ac8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.715+0000 7faef4ac8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7faef009b920 0x7faef0097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.715+0000 7faef4ac8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faef0084b90 con 0x7faef0084f30 2019-12-05T06:06:53.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.715+0000 7faef4ac8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faef0099a20 con 0x7faef009b920 2019-12-05T06:06:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.715+0000 7faef4ac8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faef0099cf0 con 0x7faef0096e90 2019-12-05T06:06:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.715+0000 7faeeed9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.715+0000 7faeef59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7faef009b920 0x7faef0097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.715+0000 7faeeed9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:50406/0 (socket says 172.21.15.16:50406) 2019-12-05T06:06:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.715+0000 7faeef59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7faef009b920 0x7faef0097a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:60742/0 (socket says 172.21.15.16:60742) 2019-12-05T06:06:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.715+0000 7faeeed9d700 1 -- 172.21.15.16:0/2708114918 learned_addr learned my addr 172.21.15.16:0/2708114918 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:53.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faeeed9d700 1 -- 172.21.15.16:0/2708114918 >> v2:172.21.15.76:3300/0 conn(0x7faef009b920 msgr2=0x7faef0097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> v2:172.21.15.76:3300/0 conn(0x7faef009b920 0x7faef0097a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faeeed9d700 1 -- 172.21.15.16:0/2708114918 >> v1:172.21.15.76:6789/0 conn(0x7faef0084f30 legacy=0x7faef0097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faeeed9d700 1 -- 172.21.15.16:0/2708114918 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef009aae0 con 0x7faef0096e90 2019-12-05T06:06:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:53.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faee77fe700 1 -- 172.21.15.16:0/2708114918 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faee0000940 con 0x7faef0096e90 2019-12-05T06:06:53.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.716+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.717+0000 7faee77fe700 1 -- 172.21.15.16:0/2708114918 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faee0009980 con 0x7faef0096e90 2019-12-05T06:06:53.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.717+0000 7faef4ac8700 1 -- 172.21.15.16:0/2708114918 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faef009aae0 con 0x7faef0096e90 2019-12-05T06:06:53.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.717+0000 7faee77fe700 1 -- 172.21.15.16:0/2708114918 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faee000a010 con 0x7faef0096e90 2019-12-05T06:06:53.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.717+0000 7faef4ac8700 1 -- 172.21.15.16:0/2708114918 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faef009aae0 con 0x7faef0096e90 2019-12-05T06:06:53.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.718+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.719+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.719+0000 7faee77fe700 1 -- 172.21.15.16:0/2708114918 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faee0009b70 con 0x7faef0096e90 2019-12-05T06:06:53.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.719+0000 7faee77fe700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faed4023630 0x7faed4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:53.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.720+0000 7faee77fe700 1 -- 172.21.15.16:0/2708114918 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7faee0045dd0 con 0x7faef0096e90 2019-12-05T06:06:53.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.720+0000 7faeee59c700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faed4023630 0x7faed4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:53.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.720+0000 7faeee59c700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faed4023630 0x7faed4025ac0 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:53.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.732+0000 7faef4ac8700 1 -- 172.21.15.16:0/2708114918 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faef009aae0 con 0x7faef0096e90 2019-12-05T06:06:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.741+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:53.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:53.741+0000 7faee77fe700 1 -- 172.21.15.16:0/2708114918 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faee0009580 con 0x7faef0096e90 2019-12-05T06:06:54.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.013+0000 7faef4ac8700 1 -- 172.21.15.16:0/2708114918 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7faef009aae0 con 0x7faef0096e90 2019-12-05T06:06:54.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.014+0000 7faeeed9d700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=THROTTLE_DONE pgs=40 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.014+0000 7faee77fe700 1 -- 172.21.15.16:0/2708114918 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faee0009b70 con 0x7faef0096e90 2019-12-05T06:06:54.014 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2019-12-05T06:06:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 -- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faed4023630 msgr2=0x7faed4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faed4023630 0x7faed4025ac0 crc :-1 s=READY pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 -- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 msgr2=0x7faef009b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 secure :-1 s=READY pgs=40 cs=0 l=1 rx=0x7faee0010430 tx=0x7faee00102b0).stop 2019-12-05T06:06:54.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 -- 172.21.15.16:0/2708114918 shutdown_connections 2019-12-05T06:06:54.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faed4023630 0x7faed4025ac0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 --2- 172.21.15.16:0/2708114918 >> v2:172.21.15.76:3300/0 conn(0x7faef009b920 0x7faef0097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 --2- 172.21.15.16:0/2708114918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faef0096e90 0x7faef009b3e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 -- 172.21.15.16:0/2708114918 shutdown_connections 2019-12-05T06:06:54.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 -- 172.21.15.16:0/2708114918 wait complete. 2019-12-05T06:06:54.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.019+0000 7faef4ac8700 1 -- 172.21.15.16:0/2708114918 >> 172.21.15.16:0/2708114918 conn(0x7faef00714f0 msgr2=0x7faef009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:54.030 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.5 2019-12-05T06:06:54.030 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2019-12-05T06:06:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.259+0000 7f91e80e9700 1 Processor -- start 2019-12-05T06:06:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.259+0000 7f91e80e9700 1 -- start start 2019-12-05T06:06:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.261+0000 7f91e80e9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f91e008c6d0 0x7f91e008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.261+0000 7f91e80e9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.261+0000 7f91e80e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e008fb70 con 0x7f91e008fef0 2019-12-05T06:06:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.261+0000 7f91e80e9700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e008df90 con 0x7f91e008c6d0 2019-12-05T06:06:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.261+0000 7f91e80e9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91e00a3f70 con 0x7f91e008f720 2019-12-05T06:06:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.262+0000 7f91e6e87700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.262+0000 7f91e6686700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f91e008c6d0 0x7f91e008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.262+0000 7f91e6e87700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:50420/0 (socket says 172.21.15.16:50420) 2019-12-05T06:06:54.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.262+0000 7f91e6e87700 1 -- 172.21.15.16:0/236117859 learned_addr learned my addr 172.21.15.16:0/236117859 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:54.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.262+0000 7f91e5e85700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f91e008fef0 0x7f91e008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:52902/0 (socket says 172.21.15.16:52902) 2019-12-05T06:06:54.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.263+0000 7f91e5684700 1 -- 172.21.15.16:0/236117859 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1230925439 0 0) 0x7f91d4001080 con 0x7f91e008fef0 2019-12-05T06:06:54.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.263+0000 7f91e5684700 1 -- 172.21.15.16:0/236117859 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91c8002e80 con 0x7f91e008fef0 2019-12-05T06:06:54.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.263+0000 7f91e6e87700 1 -- 172.21.15.16:0/236117859 >> v2:172.21.15.76:3300/0 conn(0x7f91e008c6d0 msgr2=0x7f91e008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:54.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.263+0000 7f91e6e87700 1 --2- 172.21.15.16:0/236117859 >> v2:172.21.15.76:3300/0 conn(0x7f91e008c6d0 0x7f91e008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:54.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.263+0000 7f91e6e87700 1 -- 172.21.15.16:0/236117859 >> v1:172.21.15.76:6789/0 conn(0x7f91e008fef0 legacy=0x7f91e008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.263+0000 7f91e6e87700 1 -- 172.21.15.16:0/236117859 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e008d540 con 0x7f91e008f720 2019-12-05T06:06:54.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.263+0000 7f91e6686700 1 --2- 172.21.15.16:0/236117859 >> v2:172.21.15.76:3300/0 conn(0x7f91e008c6d0 0x7f91e008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:54.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.263+0000 7f91e6e87700 1 --2- 172.21.15.16:0/236117859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f91d000f760 tx=0x7f91d000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.263+0000 7f91e6e87700 1 --2- 172.21.15.16:0/236117859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f91d000f760 tx=0x7f91d000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.263+0000 7f91e6e87700 1 --2- 172.21.15.16:0/236117859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 secure :-1 s=READY pgs=41 cs=0 l=1 rx=0x7f91d000f760 tx=0x7f91d000fb00).ready entity=mon.2 client_cookie=73eff26a4c6a5174 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:54.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.264+0000 7f91e6e87700 1 --2- 172.21.15.16:0/236117859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f91d000f760 tx=0x7f91d000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.264+0000 7f91e6e87700 1 --2- 172.21.15.16:0/236117859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f91d000f760 tx=0x7f91d000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.264+0000 7f91e6e87700 1 --2- 172.21.15.16:0/236117859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 secure :-1 s=THROTTLE_DONE pgs=41 cs=0 l=1 rx=0x7f91d000f760 tx=0x7f91d000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.264+0000 7f91e5684700 1 -- 172.21.15.16:0/236117859 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d0000a70 con 0x7f91e008f720 2019-12-05T06:06:54.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.264+0000 7f91e5684700 1 -- 172.21.15.16:0/236117859 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d0000e70 con 0x7f91e008f720 2019-12-05T06:06:54.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.264+0000 7f91e5684700 1 -- 172.21.15.16:0/236117859 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d0001500 con 0x7f91e008f720 2019-12-05T06:06:54.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.264+0000 7f91e80e9700 1 -- 172.21.15.16:0/236117859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 msgr2=0x7f91e008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.264+0000 7f91e80e9700 1 --2- 172.21.15.16:0/236117859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 secure :-1 s=READY pgs=41 cs=0 l=1 rx=0x7f91d000f760 tx=0x7f91d000fb00).stop 2019-12-05T06:06:54.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.266+0000 7f91e80e9700 1 -- 172.21.15.16:0/236117859 shutdown_connections 2019-12-05T06:06:54.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.266+0000 7f91e80e9700 1 --2- 172.21.15.16:0/236117859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f91e008f720 0x7f91e008d000 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.266+0000 7f91e80e9700 1 --2- 172.21.15.16:0/236117859 >> v2:172.21.15.76:3300/0 conn(0x7f91e008c6d0 0x7f91e008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:54.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.266+0000 7f91e80e9700 1 -- 172.21.15.16:0/236117859 shutdown_connections 2019-12-05T06:06:54.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.266+0000 7f91e80e9700 1 -- 172.21.15.16:0/236117859 wait complete. 2019-12-05T06:06:54.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.266+0000 7f91e80e9700 1 -- 172.21.15.16:0/236117859 >> 172.21.15.16:0/236117859 conn(0x7f91e0061940 msgr2=0x7f91e0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:54.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.271+0000 7f91e80e9700 1 Processor -- start 2019-12-05T06:06:54.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.272+0000 7f91e80e9700 1 -- start start 2019-12-05T06:06:54.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.273+0000 7f91e80e9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f91e006dc30 0x7f91e006e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:54.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.273+0000 7f91e80e9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91e0087230 0x7f91e006e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:54.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.273+0000 7f91e80e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91e0087670 con 0x7f91e0087a00 2019-12-05T06:06:54.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.273+0000 7f91e80e9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e0084810 con 0x7f91e0087230 2019-12-05T06:06:54.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.273+0000 7f91e80e9700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91e0085aa0 con 0x7f91e006dc30 2019-12-05T06:06:54.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.273+0000 7f91e6e87700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91e0087230 0x7f91e006e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.273+0000 7f91e6e87700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f91e0087230 0x7f91e006e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46436/0 (socket says 172.21.15.16:46436) 2019-12-05T06:06:54.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.273+0000 7f91e6e87700 1 -- 172.21.15.16:0/793802741 learned_addr learned my addr 172.21.15.16:0/793802741 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:54.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.274+0000 7f91e6686700 1 --2- 172.21.15.16:0/793802741 >> v2:172.21.15.76:3300/0 conn(0x7f91e006dc30 0x7f91e006e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.274+0000 7f91e6e87700 1 -- 172.21.15.16:0/793802741 >> v2:172.21.15.76:3300/0 conn(0x7f91e006dc30 msgr2=0x7f91e006e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.274+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> v2:172.21.15.76:3300/0 conn(0x7f91e006dc30 0x7f91e006e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.274+0000 7f91e6e87700 1 -- 172.21.15.16:0/793802741 >> v1:172.21.15.76:6789/0 conn(0x7f91e0087a00 legacy=0x7f91e0086a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.274+0000 7f91e6e87700 1 -- 172.21.15.16:0/793802741 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91e0086130 con 0x7f91e0087230 2019-12-05T06:06:54.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.274+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> v2:172.21.15.16:3300/0 conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.274+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> v2:172.21.15.16:3300/0 conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.274+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=READY pgs=141 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:54.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.275+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.275+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.275+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.275+0000 7f91deffd700 1 -- 172.21.15.16:0/793802741 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d00081b0 con 0x7f91e0087230 2019-12-05T06:06:54.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.275+0000 7f91deffd700 1 -- 172.21.15.16:0/793802741 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91d0000c60 con 0x7f91e0087230 2019-12-05T06:06:54.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.275+0000 7f91deffd700 1 -- 172.21.15.16:0/793802741 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91d00012f0 con 0x7f91e0087230 2019-12-05T06:06:54.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.275+0000 7f91e80e9700 1 -- 172.21.15.16:0/793802741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91e0086130 con 0x7f91e0087230 2019-12-05T06:06:54.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.275+0000 7f91e80e9700 1 -- 172.21.15.16:0/793802741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91e0086130 con 0x7f91e0087230 2019-12-05T06:06:54.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.276+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.277+0000 7f91deffd700 1 -- 172.21.15.16:0/793802741 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f91d0000e50 con 0x7f91e0087230 2019-12-05T06:06:54.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.277+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.277+0000 7f91deffd700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 0x7f91c0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:54.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.278+0000 7f91deffd700 1 -- 172.21.15.16:0/793802741 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f91d0045e60 con 0x7f91e0087230 2019-12-05T06:06:54.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.278+0000 7f91e6686700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 0x7f91c0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.278+0000 7f91e6686700 1 -- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 msgr2=0x7f91c0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:06:54.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.278+0000 7f91e6686700 1 -- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 msgr2=0x7f91c0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:06:54.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.278+0000 7f91e6686700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 0x7f91c0025ac0 crc :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:06:54.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.278+0000 7f91e6686700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 0x7f91c0025ac0 crc :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:06:54.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.291+0000 7f91e80e9700 1 -- 172.21.15.16:0/793802741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91e0086130 con 0x7f91e0087230 2019-12-05T06:06:54.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.300+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.300+0000 7f91deffd700 1 -- 172.21.15.16:0/793802741 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f91d0007350 con 0x7f91e0087230 2019-12-05T06:06:54.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.479+0000 7f91e6686700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 0x7f91c0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.479+0000 7f91e6686700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 0x7f91c0025ac0 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:54.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.578+0000 7f91e80e9700 1 -- 172.21.15.16:0/793802741 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f91e0086130 con 0x7f91e0087230 2019-12-05T06:06:54.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.579+0000 7f91e6e87700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.579+0000 7f91deffd700 1 -- 172.21.15.16:0/793802741 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f91d0000e50 con 0x7f91e0087230 2019-12-05T06:06:54.580 INFO:teuthology.orchestra.run.smithi016.stdout:55834574855 2019-12-05T06:06:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.581+0000 7f91e80e9700 1 -- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 msgr2=0x7f91c0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.581+0000 7f91e80e9700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 0x7f91c0025ac0 crc :-1 s=READY pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.582+0000 7f91e80e9700 1 -- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 msgr2=0x7f91e006e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.582+0000 7f91e80e9700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 0x7f91e006e560 secure :-1 s=READY pgs=141 cs=0 l=1 rx=0x7f91d0010470 tx=0x7f91d00102a0).stop 2019-12-05T06:06:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.582+0000 7f91e80e9700 1 -- 172.21.15.16:0/793802741 shutdown_connections 2019-12-05T06:06:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.582+0000 7f91e80e9700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91c0023630 0x7f91c0025ac0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.582+0000 7f91e80e9700 1 --2- 172.21.15.16:0/793802741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f91e0087230 0x7f91e006e560 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.582+0000 7f91e80e9700 1 --2- 172.21.15.16:0/793802741 >> v2:172.21.15.76:3300/0 conn(0x7f91e006dc30 0x7f91e006e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.582+0000 7f91e80e9700 1 -- 172.21.15.16:0/793802741 shutdown_connections 2019-12-05T06:06:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.582+0000 7f91e80e9700 1 -- 172.21.15.16:0/793802741 wait complete. 2019-12-05T06:06:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.582+0000 7f91e80e9700 1 -- 172.21.15.16:0/793802741 >> 172.21.15.16:0/793802741 conn(0x7f91e012f780 msgr2=0x7f91e0096b90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:54.592 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.6 2019-12-05T06:06:54.592 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2019-12-05T06:06:54.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.816+0000 7f4d81fe3700 1 Processor -- start 2019-12-05T06:06:54.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.817+0000 7f4d81fe3700 1 -- start start 2019-12-05T06:06:54.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.818+0000 7f4d81fe3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d7c08d2c0 0x7f4d7c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:54.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.818+0000 7f4d81fe3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d7c08fef0 0x7f4d7c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:54.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.818+0000 7f4d81fe3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d7c0958e0 con 0x7f4d7c08c6d0 2019-12-05T06:06:54.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.818+0000 7f4d81fe3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d7c08df90 con 0x7f4d7c08fef0 2019-12-05T06:06:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.818+0000 7f4d81fe3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d7c0a3f70 con 0x7f4d7c08d2c0 2019-12-05T06:06:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.819+0000 7f4d7bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4d7c08c6d0 0x7f4d7c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59844/0 (socket says 172.21.15.16:59844) 2019-12-05T06:06:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.819+0000 7f4d7bfff700 1 -- 172.21.15.16:0/3386237468 learned_addr learned my addr 172.21.15.16:0/3386237468 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:54.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.819+0000 7f4d80d81700 1 --2- 172.21.15.16:0/3386237468 >> v2:172.21.15.16:3300/0 conn(0x7f4d7c08fef0 0x7f4d7c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.819+0000 7f4d7b7fe700 1 --2- 172.21.15.16:0/3386237468 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d7c08d2c0 0x7f4d7c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.819+0000 7f4d7affd700 1 -- 172.21.15.16:0/3386237468 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605025828 0 0) 0x7f4d6c0014d0 con 0x7f4d7c08c6d0 2019-12-05T06:06:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.819+0000 7f4d7affd700 1 -- 172.21.15.16:0/3386237468 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d68002e80 con 0x7f4d7c08c6d0 2019-12-05T06:06:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.819+0000 7f4d80d81700 1 -- 172.21.15.16:0/3386237468 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d7c08d2c0 msgr2=0x7f4d7c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:54.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.819+0000 7f4d80d81700 1 --2- 172.21.15.16:0/3386237468 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d7c08d2c0 0x7f4d7c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:54.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.819+0000 7f4d80d81700 1 -- 172.21.15.16:0/3386237468 >> v1:172.21.15.16:6789/0 conn(0x7f4d7c08c6d0 legacy=0x7f4d7c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.820+0000 7f4d80d81700 1 -- 172.21.15.16:0/3386237468 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c08e620 con 0x7f4d7c08fef0 2019-12-05T06:06:54.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.820+0000 7f4d80d81700 1 --2- 172.21.15.16:0/3386237468 >> v2:172.21.15.16:3300/0 conn(0x7f4d7c08fef0 0x7f4d7c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4d7400af50 tx=0x7f4d7400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.820+0000 7f4d80d81700 1 --2- 172.21.15.16:0/3386237468 >> v2:172.21.15.16:3300/0 conn(0x7f4d7c08fef0 0x7f4d7c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4d7400af50 tx=0x7f4d7400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.820+0000 7f4d80d81700 1 --2- 172.21.15.16:0/3386237468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c08fef0 0x7f4d7c08f460 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7f4d7400af50 tx=0x7f4d7400b6e0).ready entity=mon.0 client_cookie=49bb1a718e9f0b89 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:54.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.820+0000 7f4d80d81700 1 --2- 172.21.15.16:0/3386237468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c08fef0 0x7f4d7c08f460 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7f4d7400af50 tx=0x7f4d7400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.820+0000 7f4d7affd700 1 -- 172.21.15.16:0/3386237468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d74009ba0 con 0x7f4d7c08fef0 2019-12-05T06:06:54.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.820+0000 7f4d80d81700 1 --2- 172.21.15.16:0/3386237468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c08fef0 0x7f4d7c08f460 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7f4d7400af50 tx=0x7f4d7400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.820+0000 7f4d80d81700 1 --2- 172.21.15.16:0/3386237468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c08fef0 0x7f4d7c08f460 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7f4d7400af50 tx=0x7f4d7400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.821+0000 7f4d7affd700 1 -- 172.21.15.16:0/3386237468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d740097c0 con 0x7f4d7c08fef0 2019-12-05T06:06:54.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.821+0000 7f4d7affd700 1 -- 172.21.15.16:0/3386237468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d7400a180 con 0x7f4d7c08fef0 2019-12-05T06:06:54.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.821+0000 7f4d81fe3700 1 -- 172.21.15.16:0/3386237468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c08fef0 msgr2=0x7f4d7c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.821+0000 7f4d81fe3700 1 --2- 172.21.15.16:0/3386237468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c08fef0 0x7f4d7c08f460 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7f4d7400af50 tx=0x7f4d7400b6e0).stop 2019-12-05T06:06:54.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.822+0000 7f4d81fe3700 1 -- 172.21.15.16:0/3386237468 shutdown_connections 2019-12-05T06:06:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.822+0000 7f4d81fe3700 1 --2- 172.21.15.16:0/3386237468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c08fef0 0x7f4d7c08f460 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.822+0000 7f4d81fe3700 1 --2- 172.21.15.16:0/3386237468 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d7c08d2c0 0x7f4d7c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.822+0000 7f4d81fe3700 1 -- 172.21.15.16:0/3386237468 shutdown_connections 2019-12-05T06:06:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.822+0000 7f4d81fe3700 1 -- 172.21.15.16:0/3386237468 wait complete. 2019-12-05T06:06:54.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.822+0000 7f4d81fe3700 1 -- 172.21.15.16:0/3386237468 >> 172.21.15.16:0/3386237468 conn(0x7f4d7c061940 msgr2=0x7f4d7c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.827+0000 7f4d81fe3700 1 Processor -- start 2019-12-05T06:06:54.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.828+0000 7f4d81fe3700 1 -- start start 2019-12-05T06:06:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.829+0000 7f4d81fe3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d7c096e90 0x7f4d7c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.829+0000 7f4d81fe3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4d7c084f30 0x7f4d7c09b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.829+0000 7f4d81fe3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d7c084b90 con 0x7f4d7c097280 2019-12-05T06:06:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.829+0000 7f4d81fe3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d7c09a450 con 0x7f4d7c096e90 2019-12-05T06:06:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.829+0000 7f4d81fe3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d7c099a20 con 0x7f4d7c084f30 2019-12-05T06:06:54.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.829+0000 7f4d7bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d7c096e90 0x7f4d7c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.829+0000 7f4d7bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d7c096e90 0x7f4d7c09b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46450/0 (socket says 172.21.15.16:46450) 2019-12-05T06:06:54.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.829+0000 7f4d7bfff700 1 -- 172.21.15.16:0/1968962960 learned_addr learned my addr 172.21.15.16:0/1968962960 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:54.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.829+0000 7f4d7b7fe700 1 --2- 172.21.15.16:0/1968962960 >> v2:172.21.15.76:3300/0 conn(0x7f4d7c084f30 0x7f4d7c09b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.830+0000 7f4d78ff9700 1 -- 172.21.15.16:0/1968962960 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1207787995 0 0) 0x7f4d74000940 con 0x7f4d7c097280 2019-12-05T06:06:54.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.830+0000 7f4d7bfff700 1 -- 172.21.15.16:0/1968962960 >> v2:172.21.15.76:3300/0 conn(0x7f4d7c084f30 msgr2=0x7f4d7c09b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.830+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> v2:172.21.15.76:3300/0 conn(0x7f4d7c084f30 0x7f4d7c09b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:54.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.830+0000 7f4d7bfff700 1 -- 172.21.15.16:0/1968962960 >> v1:172.21.15.76:6789/0 conn(0x7f4d7c097280 legacy=0x7f4d7c0976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:54.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.830+0000 7f4d7bfff700 1 -- 172.21.15.16:0/1968962960 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c09aae0 con 0x7f4d7c096e90 2019-12-05T06:06:54.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.830+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> v2:172.21.15.16:3300/0 conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.830+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> v2:172.21.15.16:3300/0 conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.830+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:54.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.831+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.831+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.831+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.831+0000 7f4d78ff9700 1 -- 172.21.15.16:0/1968962960 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d6c001010 con 0x7f4d7c096e90 2019-12-05T06:06:54.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.831+0000 7f4d78ff9700 1 -- 172.21.15.16:0/1968962960 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d6c001440 con 0x7f4d7c096e90 2019-12-05T06:06:54.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.831+0000 7f4d81fe3700 1 -- 172.21.15.16:0/1968962960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d7c09aae0 con 0x7f4d7c096e90 2019-12-05T06:06:54.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.831+0000 7f4d78ff9700 1 -- 172.21.15.16:0/1968962960 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d6c001a10 con 0x7f4d7c096e90 2019-12-05T06:06:54.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.831+0000 7f4d81fe3700 1 -- 172.21.15.16:0/1968962960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d7c09aae0 con 0x7f4d7c096e90 2019-12-05T06:06:54.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.832+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.832+0000 7f4d78ff9700 1 -- 172.21.15.16:0/1968962960 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4d6c001010 con 0x7f4d7c096e90 2019-12-05T06:06:54.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.833+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.833+0000 7f4d78ff9700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d60023630 0x7f4d60025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:54.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.833+0000 7f4d78ff9700 1 -- 172.21.15.16:0/1968962960 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f4d6c0460c0 con 0x7f4d7c096e90 2019-12-05T06:06:54.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.833+0000 7f4d7b7fe700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d60023630 0x7f4d60025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:54.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.834+0000 7f4d7b7fe700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d60023630 0x7f4d60025ac0 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:54.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.848+0000 7f4d81fe3700 1 -- 172.21.15.16:0/1968962960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d7c09aae0 con 0x7f4d7c096e90 2019-12-05T06:06:54.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.857+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:54.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:54.857+0000 7f4d78ff9700 1 -- 172.21.15.16:0/1968962960 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4d6c007950 con 0x7f4d7c096e90 2019-12-05T06:06:55.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.107+0000 7f4d81fe3700 1 -- 172.21.15.16:0/1968962960 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f4d7c09aae0 con 0x7f4d7c096e90 2019-12-05T06:06:55.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.108+0000 7f4d7bfff700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.108+0000 7f4d78ff9700 1 -- 172.21.15.16:0/1968962960 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4d6c001010 con 0x7f4d7c096e90 2019-12-05T06:06:55.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.110+0000 7f4d81fe3700 1 -- 172.21.15.16:0/1968962960 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d60023630 msgr2=0x7f4d60025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.110+0000 7f4d81fe3700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d60023630 0x7f4d60025ac0 crc :-1 s=READY pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.110+0000 7f4d81fe3700 1 -- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 msgr2=0x7f4d7c09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.110+0000 7f4d81fe3700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 0x7f4d7c09b3e0 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7f4d6c010760 tx=0x7f4d6c0105e0).stop 2019-12-05T06:06:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.111+0000 7f4d81fe3700 1 -- 172.21.15.16:0/1968962960 shutdown_connections 2019-12-05T06:06:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.111+0000 7f4d81fe3700 1 --2- 172.21.15.16:0/1968962960 >> v2:172.21.15.76:3300/0 conn(0x7f4d7c084f30 0x7f4d7c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.111+0000 7f4d81fe3700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d60023630 0x7f4d60025ac0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.111+0000 7f4d81fe3700 1 --2- 172.21.15.16:0/1968962960 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4d7c096e90 0x7f4d7c09b3e0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.111+0000 7f4d81fe3700 1 -- 172.21.15.16:0/1968962960 shutdown_connections 2019-12-05T06:06:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.111+0000 7f4d81fe3700 1 -- 172.21.15.16:0/1968962960 wait complete. 2019-12-05T06:06:55.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.111+0000 7f4d81fe3700 1 -- 172.21.15.16:0/1968962960 >> 172.21.15.16:0/1968962960 conn(0x7f4d7c0714f0 msgr2=0x7f4d7c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:55.117 INFO:teuthology.orchestra.run.smithi016.stdout:55834574856 2019-12-05T06:06:55.126 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574856 for osd.7 2019-12-05T06:06:55.126 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2019-12-05T06:06:55.126 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T06:06:55.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.386+0000 7f8303b3d700 1 Processor -- start 2019-12-05T06:06:55.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.386+0000 7f8303b3d700 1 -- start start 2019-12-05T06:06:55.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f8303b3d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82fc08c6d0 0x7f82fc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:55.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f8303b3d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f82fc08fef0 0x7f82fc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f8303b3d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82fc08ef20 0x7f82fc08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f8303b3d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82fc0a3f70 con 0x7f82fc08ef20 2019-12-05T06:06:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f8303b3d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82fc08d830 con 0x7f82fc08fef0 2019-12-05T06:06:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f8303b3d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc08d540 con 0x7f82fc08c6d0 2019-12-05T06:06:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f83028db700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82fc08ef20 0x7f82fc08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f83028db700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82fc08ef20 0x7f82fc08f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46460/0 (socket says 172.21.15.16:46460) 2019-12-05T06:06:55.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f83018d9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f82fc08fef0 0x7f82fc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f83028db700 1 -- 172.21.15.16:0/1813775544 learned_addr learned my addr 172.21.15.16:0/1813775544 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:55.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.389+0000 7f83020da700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82fc08c6d0 0x7f82fc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.390+0000 7f83028db700 1 -- 172.21.15.16:0/1813775544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82fc08c6d0 msgr2=0x7f82fc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:55.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.390+0000 7f83028db700 1 --2- 172.21.15.16:0/1813775544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82fc08c6d0 0x7f82fc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:55.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.390+0000 7f83028db700 1 -- 172.21.15.16:0/1813775544 >> v2:172.21.15.76:3300/0 conn(0x7f82fc08fef0 msgr2=0x7f82fc08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:55.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.390+0000 7f83028db700 1 --2- 172.21.15.16:0/1813775544 >> v2:172.21.15.76:3300/0 conn(0x7f82fc08fef0 0x7f82fc08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:55.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.390+0000 7f83028db700 1 -- 172.21.15.16:0/1813775544 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc08e620 con 0x7f82fc08ef20 2019-12-05T06:06:55.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.390+0000 7f83018d9700 1 --2- 172.21.15.16:0/1813775544 >> v2:172.21.15.76:3300/0 conn(0x7f82fc08fef0 0x7f82fc08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:55.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.390+0000 7f83020da700 1 --2- 172.21.15.16:0/1813775544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82fc08c6d0 0x7f82fc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:55.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.390+0000 7f83028db700 1 --2- 172.21.15.16:0/1813775544 >> v2:172.21.15.16:3300/0 conn(0x7f82fc08ef20 0x7f82fc08f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f82ec00f730 tx=0x7f82ec00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.391+0000 7f83028db700 1 --2- 172.21.15.16:0/1813775544 >> v2:172.21.15.16:3300/0 conn(0x7f82fc08ef20 0x7f82fc08f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f82ec00f730 tx=0x7f82ec00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.391+0000 7f83028db700 1 --2- 172.21.15.16:0/1813775544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc08ef20 0x7f82fc08f3c0 secure :-1 s=READY pgs=145 cs=0 l=1 rx=0x7f82ec00f730 tx=0x7f82ec00fad0).ready entity=mon.0 client_cookie=4121c4d71b81cbb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.391+0000 7f83028db700 1 --2- 172.21.15.16:0/1813775544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc08ef20 0x7f82fc08f3c0 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f82ec00f730 tx=0x7f82ec00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.391+0000 7f83028db700 1 --2- 172.21.15.16:0/1813775544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc08ef20 0x7f82fc08f3c0 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f82ec00f730 tx=0x7f82ec00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.391+0000 7f83010d8700 1 -- 172.21.15.16:0/1813775544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82ec0010e0 con 0x7f82fc08ef20 2019-12-05T06:06:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.391+0000 7f83028db700 1 --2- 172.21.15.16:0/1813775544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc08ef20 0x7f82fc08f3c0 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f82ec00f730 tx=0x7f82ec00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.391+0000 7f83010d8700 1 -- 172.21.15.16:0/1813775544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec000b90 con 0x7f82fc08ef20 2019-12-05T06:06:55.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.391+0000 7f83010d8700 1 -- 172.21.15.16:0/1813775544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82ec0016c0 con 0x7f82fc08ef20 2019-12-05T06:06:55.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.391+0000 7f8303b3d700 1 -- 172.21.15.16:0/1813775544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc08ef20 msgr2=0x7f82fc08f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.391+0000 7f8303b3d700 1 --2- 172.21.15.16:0/1813775544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc08ef20 0x7f82fc08f3c0 secure :-1 s=READY pgs=145 cs=0 l=1 rx=0x7f82ec00f730 tx=0x7f82ec00fad0).stop 2019-12-05T06:06:55.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.393+0000 7f8303b3d700 1 -- 172.21.15.16:0/1813775544 shutdown_connections 2019-12-05T06:06:55.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.393+0000 7f8303b3d700 1 --2- 172.21.15.16:0/1813775544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc08ef20 0x7f82fc08f3c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.393+0000 7f8303b3d700 1 --2- 172.21.15.16:0/1813775544 >> v2:172.21.15.76:3300/0 conn(0x7f82fc08fef0 0x7f82fc08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:55.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.393+0000 7f8303b3d700 1 --2- 172.21.15.16:0/1813775544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82fc08c6d0 0x7f82fc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:55.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.394+0000 7f8303b3d700 1 -- 172.21.15.16:0/1813775544 shutdown_connections 2019-12-05T06:06:55.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.394+0000 7f8303b3d700 1 -- 172.21.15.16:0/1813775544 wait complete. 2019-12-05T06:06:55.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.394+0000 7f8303b3d700 1 -- 172.21.15.16:0/1813775544 >> 172.21.15.16:0/1813775544 conn(0x7f82fc061940 msgr2=0x7f82fc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:55.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.399+0000 7f8303b3d700 1 Processor -- start 2019-12-05T06:06:55.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.399+0000 7f8303b3d700 1 -- start start 2019-12-05T06:06:55.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.401+0000 7f8303b3d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82fc098b00 0x7f82fc15a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.401+0000 7f8303b3d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82fc089fb0 con 0x7f82fc06df10 2019-12-05T06:06:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.401+0000 7f8303b3d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82fc06e300 con 0x7f82fc0992d0 2019-12-05T06:06:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.401+0000 7f8303b3d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82fc097370 con 0x7f82fc098b00 2019-12-05T06:06:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.401+0000 7f83020da700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f82fc06df10 0x7f82fc098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59864/0 (socket says 172.21.15.16:59864) 2019-12-05T06:06:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.401+0000 7f83020da700 1 -- 172.21.15.16:0/3249451496 learned_addr learned my addr 172.21.15.16:0/3249451496 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.401+0000 7f83028db700 1 --2- 172.21.15.16:0/3249451496 >> v2:172.21.15.16:3300/0 conn(0x7f82fc098b00 0x7f82fc15a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f82faffd700 1 -- 172.21.15.16:0/3249451496 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642486943 0 0) 0x7f82f000db80 con 0x7f82fc0992d0 2019-12-05T06:06:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f83028db700 1 -- 172.21.15.16:0/3249451496 >> v1:172.21.15.76:6789/0 conn(0x7f82fc0992d0 legacy=0x7f82fc156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f83028db700 1 -- 172.21.15.16:0/3249451496 >> v1:172.21.15.16:6789/0 conn(0x7f82fc06df10 legacy=0x7f82fc098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f83028db700 1 -- 172.21.15.16:0/3249451496 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc097a00 con 0x7f82fc098b00 2019-12-05T06:06:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f83028db700 1 --2- 172.21.15.16:0/3249451496 >> v2:172.21.15.16:3300/0 conn(0x7f82fc098b00 0x7f82fc15a4d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f82ec0543e0 tx=0x7f82ec054780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f83028db700 1 --2- 172.21.15.16:0/3249451496 >> v2:172.21.15.16:3300/0 conn(0x7f82fc098b00 0x7f82fc15a4d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f82ec0543e0 tx=0x7f82ec054780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f83028db700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc098b00 0x7f82fc15a4d0 secure :-1 s=READY pgs=147 cs=0 l=1 rx=0x7f82ec0543e0 tx=0x7f82ec054780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:55.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f83028db700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc098b00 0x7f82fc15a4d0 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f82ec0543e0 tx=0x7f82ec054780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f82faffd700 1 -- 172.21.15.16:0/3249451496 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82ec0012b0 con 0x7f82fc098b00 2019-12-05T06:06:55.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f83028db700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc098b00 0x7f82fc15a4d0 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f82ec0543e0 tx=0x7f82ec054780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f83028db700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc098b00 0x7f82fc15a4d0 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f82ec0543e0 tx=0x7f82ec054780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.402+0000 7f82faffd700 1 -- 172.21.15.16:0/3249451496 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82ec000940 con 0x7f82fc098b00 2019-12-05T06:06:55.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.403+0000 7f8303b3d700 1 -- 172.21.15.16:0/3249451496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82fc097a00 con 0x7f82fc098b00 2019-12-05T06:06:55.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.403+0000 7f82faffd700 1 -- 172.21.15.16:0/3249451496 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82ec001890 con 0x7f82fc098b00 2019-12-05T06:06:55.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.403+0000 7f8303b3d700 1 -- 172.21.15.16:0/3249451496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82fc097a00 con 0x7f82fc098b00 2019-12-05T06:06:55.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.404+0000 7f83028db700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc098b00 0x7f82fc15a4d0 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f82ec0543e0 tx=0x7f82ec054780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.405+0000 7f83028db700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc098b00 0x7f82fc15a4d0 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f82ec0543e0 tx=0x7f82ec054780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.405+0000 7f82faffd700 1 -- 172.21.15.16:0/3249451496 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f82ec0010c0 con 0x7f82fc098b00 2019-12-05T06:06:55.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.405+0000 7f82faffd700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f82dc023630 0x7f82dc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:55.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.405+0000 7f82faffd700 1 -- 172.21.15.16:0/3249451496 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f82ec089e20 con 0x7f82fc098b00 2019-12-05T06:06:55.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.406+0000 7f83020da700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f82dc023630 0x7f82dc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.406+0000 7f83020da700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f82dc023630 0x7f82dc025ac0 crc :-1 s=READY pgs=82 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:55.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.418+0000 7f8303b3d700 1 -- 172.21.15.16:0/3249451496 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82fc097a00 con 0x7f82fc098b00 2019-12-05T06:06:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.426+0000 7f83028db700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc098b00 0x7f82fc15a4d0 secure :-1 s=THROTTLE_DONE pgs=147 cs=0 l=1 rx=0x7f82ec0543e0 tx=0x7f82ec054780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.426+0000 7f82faffd700 1 -- 172.21.15.16:0/3249451496 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f82ec000eb0 con 0x7f82fc098b00 2019-12-05T06:06:55.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.679+0000 7f8303b3d700 1 -- 172.21.15.16:0/3249451496 --> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82fc097a00 con 0x7f82dc023630 2019-12-05T06:06:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.686+0000 7f82faffd700 1 -- 172.21.15.16:0/3249451496 <== mgr.4100 v2:172.21.15.76:6800/9546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80416 (crc 0 0 0) 0x7f82f4009a60 con 0x7f82dc023630 2019-12-05T06:06:55.703 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:55.704 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2019-12-05T06:06:55.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.697+0000 7f8303b3d700 1 -- 172.21.15.16:0/3249451496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f82dc023630 msgr2=0x7f82dc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.697+0000 7f8303b3d700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f82dc023630 0x7f82dc025ac0 crc :-1 s=READY pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.697+0000 7f8303b3d700 1 -- 172.21.15.16:0/3249451496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc098b00 msgr2=0x7f82fc15a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.697+0000 7f8303b3d700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc098b00 0x7f82fc15a4d0 secure :-1 s=READY pgs=147 cs=0 l=1 rx=0x7f82ec0543e0 tx=0x7f82ec054780).stop 2019-12-05T06:06:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.697+0000 7f8303b3d700 1 -- 172.21.15.16:0/3249451496 shutdown_connections 2019-12-05T06:06:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.697+0000 7f8303b3d700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f82dc023630 0x7f82dc025ac0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.697+0000 7f8303b3d700 1 --2- 172.21.15.16:0/3249451496 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82fc098b00 0x7f82fc15a4d0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.697+0000 7f8303b3d700 1 -- 172.21.15.16:0/3249451496 shutdown_connections 2019-12-05T06:06:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.697+0000 7f8303b3d700 1 -- 172.21.15.16:0/3249451496 wait complete. 2019-12-05T06:06:55.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.697+0000 7f8303b3d700 1 -- 172.21.15.16:0/3249451496 >> 172.21.15.16:0/3249451496 conn(0x7f82fc0714f0 msgr2=0x7f82fc09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:55.710 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2019-12-05T06:06:54.768321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390720,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584000,"statfs":{"total":773094113280,"available":764502016000,"internally_reserved":8589934592,"allocated":2097152,"data_stored":772252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002011"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.271382+0000","last_change":"2019-12-05T06:06:32.598114+0000","last_active":"2019-12-05T06:06:40.271382+0000","last_peered":"2019-12-05T06:06:40.271382+0000","last_clean":"2019-12-05T06:06:40.271382+0000","last_became_active":"2019-12-05T06:06:32.597387+0000","last_became_peered":"2019-12-05T06:06:32.597387+0000","last_unstale":"2019-12-05T06:06:40.271382+0000","last_undegraded":"2019-12-05T06:06:40.271382+0000","last_fullsized":"2019-12-05T06:06:40.271382+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"24'8","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.715381+0000","last_change":"2019-12-05T06:06:36.627910+0000","last_active":"2019-12-05T06:06:44.715381+0000","last_peered":"2019-12-05T06:06:44.715381+0000","last_clean":"2019-12-05T06:06:44.715381+0000","last_became_active":"2019-12-05T06:06:36.626897+0000","last_became_peered":"2019-12-05T06:06:36.626897+0000","last_unstale":"2019-12-05T06:06:44.715381+0000","last_undegraded":"2019-12-05T06:06:44.715381+0000","last_fullsized":"2019-12-05T06:06:44.715381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.490878+0000","last_change":"2019-12-05T06:06:40.490139+0000","last_active":"2019-12-05T06:06:40.490878+0000","last_peered":"2019-12-05T06:06:40.490878+0000","last_clean":"2019-12-05T06:06:40.490878+0000","last_became_active":"2019-12-05T06:06:40.488804+0000","last_became_peered":"2019-12-05T06:06:40.488804+0000","last_unstale":"2019-12-05T06:06:40.490878+0000","last_undegraded":"2019-12-05T06:06:40.490878+0000","last_fullsized":"2019-12-05T06:06:40.490878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.271284+0000","last_change":"2019-12-05T06:06:32.887464+0000","last_active":"2019-12-05T06:06:40.271284+0000","last_peered":"2019-12-05T06:06:40.271284+0000","last_clean":"2019-12-05T06:06:40.271284+0000","last_became_active":"2019-12-05T06:06:32.885955+0000","last_became_peered":"2019-12-05T06:06:32.885955+0000","last_unstale":"2019-12-05T06:06:40.271284+0000","last_undegraded":"2019-12-05T06:06:40.271284+0000","last_fullsized":"2019-12-05T06:06:40.271284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720231+0000","last_change":"2019-12-05T06:06:36.631805+0000","last_active":"2019-12-05T06:06:44.720231+0000","last_peered":"2019-12-05T06:06:44.720231+0000","last_clean":"2019-12-05T06:06:44.720231+0000","last_became_active":"2019-12-05T06:06:36.631077+0000","last_became_peered":"2019-12-05T06:06:36.631077+0000","last_unstale":"2019-12-05T06:06:44.720231+0000","last_undegraded":"2019-12-05T06:06:44.720231+0000","last_fullsized":"2019-12-05T06:06:44.720231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.485950+0000","last_change":"2019-12-05T06:06:40.485444+0000","last_active":"2019-12-05T06:06:40.485950+0000","last_peered":"2019-12-05T06:06:40.485950+0000","last_clean":"2019-12-05T06:06:40.485950+0000","last_became_active":"2019-12-05T06:06:40.484465+0000","last_became_peered":"2019-12-05T06:06:40.484465+0000","last_unstale":"2019-12-05T06:06:40.485950+0000","last_undegraded":"2019-12-05T06:06:40.485950+0000","last_fullsized":"2019-12-05T06:06:40.485950+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576769+0000","last_change":"2019-12-05T06:06:40.488691+0000","last_active":"2019-12-05T06:06:40.576769+0000","last_peered":"2019-12-05T06:06:40.576769+0000","last_clean":"2019-12-05T06:06:40.576769+0000","last_became_active":"2019-12-05T06:06:40.487506+0000","last_became_peered":"2019-12-05T06:06:40.487506+0000","last_unstale":"2019-12-05T06:06:40.576769+0000","last_undegraded":"2019-12-05T06:06:40.576769+0000","last_fullsized":"2019-12-05T06:06:40.576769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.268843+0000","last_change":"2019-12-05T06:06:32.597542+0000","last_active":"2019-12-05T06:06:40.268843+0000","last_peered":"2019-12-05T06:06:40.268843+0000","last_clean":"2019-12-05T06:06:40.268843+0000","last_became_active":"2019-12-05T06:06:32.596579+0000","last_became_peered":"2019-12-05T06:06:32.596579+0000","last_unstale":"2019-12-05T06:06:40.268843+0000","last_undegraded":"2019-12-05T06:06:40.268843+0000","last_fullsized":"2019-12-05T06:06:40.268843+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'4","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.710303+0000","last_change":"2019-12-05T06:06:36.626828+0000","last_active":"2019-12-05T06:06:44.710303+0000","last_peered":"2019-12-05T06:06:44.710303+0000","last_clean":"2019-12-05T06:06:44.710303+0000","last_became_active":"2019-12-05T06:06:36.626005+0000","last_became_peered":"2019-12-05T06:06:36.626005+0000","last_unstale":"2019-12-05T06:06:44.710303+0000","last_undegraded":"2019-12-05T06:06:44.710303+0000","last_fullsized":"2019-12-05T06:06:44.710303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.015035+0000","last_change":"2019-12-05T06:06:40.487696+0000","last_active":"2019-12-05T06:06:41.015035+0000","last_peered":"2019-12-05T06:06:41.015035+0000","last_clean":"2019-12-05T06:06:41.015035+0000","last_became_active":"2019-12-05T06:06:40.486899+0000","last_became_peered":"2019-12-05T06:06:40.486899+0000","last_unstale":"2019-12-05T06:06:41.015035+0000","last_undegraded":"2019-12-05T06:06:41.015035+0000","last_fullsized":"2019-12-05T06:06:41.015035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576888+0000","last_change":"2019-12-05T06:06:32.597098+0000","last_active":"2019-12-05T06:06:40.576888+0000","last_peered":"2019-12-05T06:06:40.576888+0000","last_clean":"2019-12-05T06:06:40.576888+0000","last_became_active":"2019-12-05T06:06:32.596276+0000","last_became_peered":"2019-12-05T06:06:32.596276+0000","last_unstale":"2019-12-05T06:06:40.576888+0000","last_undegraded":"2019-12-05T06:06:40.576888+0000","last_fullsized":"2019-12-05T06:06:40.576888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"24'7","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720981+0000","last_change":"2019-12-05T06:06:36.629032+0000","last_active":"2019-12-05T06:06:44.720981+0000","last_peered":"2019-12-05T06:06:44.720981+0000","last_clean":"2019-12-05T06:06:44.720981+0000","last_became_active":"2019-12-05T06:06:36.628405+0000","last_became_peered":"2019-12-05T06:06:36.628405+0000","last_unstale":"2019-12-05T06:06:44.720981+0000","last_undegraded":"2019-12-05T06:06:44.720981+0000","last_fullsized":"2019-12-05T06:06:44.720981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.084640+0000","last_change":"2019-12-05T06:06:32.886929+0000","last_active":"2019-12-05T06:06:41.084640+0000","last_peered":"2019-12-05T06:06:41.084640+0000","last_clean":"2019-12-05T06:06:41.084640+0000","last_became_active":"2019-12-05T06:06:32.886095+0000","last_became_peered":"2019-12-05T06:06:32.886095+0000","last_unstale":"2019-12-05T06:06:41.084640+0000","last_undegraded":"2019-12-05T06:06:41.084640+0000","last_fullsized":"2019-12-05T06:06:41.084640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"24'5","reported_seq":"17","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720647+0000","last_change":"2019-12-05T06:06:36.632241+0000","last_active":"2019-12-05T06:06:44.720647+0000","last_peered":"2019-12-05T06:06:44.720647+0000","last_clean":"2019-12-05T06:06:44.720647+0000","last_became_active":"2019-12-05T06:06:36.631418+0000","last_became_peered":"2019-12-05T06:06:36.631418+0000","last_unstale":"2019-12-05T06:06:44.720647+0000","last_undegraded":"2019-12-05T06:06:44.720647+0000","last_fullsized":"2019-12-05T06:06:44.720647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576879+0000","last_change":"2019-12-05T06:06:39.649718+0000","last_active":"2019-12-05T06:06:40.576879+0000","last_peered":"2019-12-05T06:06:40.576879+0000","last_clean":"2019-12-05T06:06:40.576879+0000","last_became_active":"2019-12-05T06:06:39.648680+0000","last_became_peered":"2019-12-05T06:06:39.648680+0000","last_unstale":"2019-12-05T06:06:40.576879+0000","last_undegraded":"2019-12-05T06:06:40.576879+0000","last_fullsized":"2019-12-05T06:06:40.576879+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.269507+0000","last_change":"2019-12-05T06:06:32.886427+0000","last_active":"2019-12-05T06:06:40.269507+0000","last_peered":"2019-12-05T06:06:40.269507+0000","last_clean":"2019-12-05T06:06:40.269507+0000","last_became_active":"2019-12-05T06:06:32.885509+0000","last_became_peered":"2019-12-05T06:06:32.885509+0000","last_unstale":"2019-12-05T06:06:40.269507+0000","last_undegraded":"2019-12-05T06:06:40.269507+0000","last_fullsized":"2019-12-05T06:06:40.269507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"24'1","reported_seq":"13","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:41.269608+0000","last_change":"2019-12-05T06:06:36.632232+0000","last_active":"2019-12-05T06:06:41.269608+0000","last_peered":"2019-12-05T06:06:41.269608+0000","last_clean":"2019-12-05T06:06:41.269608+0000","last_became_active":"2019-12-05T06:06:36.630785+0000","last_became_peered":"2019-12-05T06:06:36.630785+0000","last_unstale":"2019-12-05T06:06:41.269608+0000","last_undegraded":"2019-12-05T06:06:41.269608+0000","last_fullsized":"2019-12-05T06:06:41.269608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.268862+0000","last_change":"2019-12-05T06:06:39.647777+0000","last_active":"2019-12-05T06:06:40.268862+0000","last_peered":"2019-12-05T06:06:40.268862+0000","last_clean":"2019-12-05T06:06:40.268862+0000","last_became_active":"2019-12-05T06:06:39.646655+0000","last_became_peered":"2019-12-05T06:06:39.646655+0000","last_unstale":"2019-12-05T06:06:40.268862+0000","last_undegraded":"2019-12-05T06:06:40.268862+0000","last_fullsized":"2019-12-05T06:06:40.268862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.547093+0000","last_change":"2019-12-05T06:06:36.628444+0000","last_active":"2019-12-05T06:06:40.547093+0000","last_peered":"2019-12-05T06:06:40.547093+0000","last_clean":"2019-12-05T06:06:40.547093+0000","last_became_active":"2019-12-05T06:06:36.627555+0000","last_became_peered":"2019-12-05T06:06:36.627555+0000","last_unstale":"2019-12-05T06:06:40.547093+0000","last_undegraded":"2019-12-05T06:06:40.547093+0000","last_fullsized":"2019-12-05T06:06:40.547093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.084584+0000","last_change":"2019-12-05T06:06:32.886895+0000","last_active":"2019-12-05T06:06:41.084584+0000","last_peered":"2019-12-05T06:06:41.084584+0000","last_clean":"2019-12-05T06:06:41.084584+0000","last_became_active":"2019-12-05T06:06:32.886123+0000","last_became_peered":"2019-12-05T06:06:32.886123+0000","last_unstale":"2019-12-05T06:06:41.084584+0000","last_undegraded":"2019-12-05T06:06:41.084584+0000","last_fullsized":"2019-12-05T06:06:41.084584+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.486413+0000","last_change":"2019-12-05T06:06:40.486073+0000","last_active":"2019-12-05T06:06:40.486413+0000","last_peered":"2019-12-05T06:06:40.486413+0000","last_clean":"2019-12-05T06:06:40.486413+0000","last_became_active":"2019-12-05T06:06:40.484954+0000","last_became_peered":"2019-12-05T06:06:40.484954+0000","last_unstale":"2019-12-05T06:06:40.486413+0000","last_undegraded":"2019-12-05T06:06:40.486413+0000","last_fullsized":"2019-12-05T06:06:40.486413+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.723134+0000","last_change":"2019-12-05T06:06:36.630520+0000","last_active":"2019-12-05T06:06:44.723134+0000","last_peered":"2019-12-05T06:06:44.723134+0000","last_clean":"2019-12-05T06:06:44.723134+0000","last_became_active":"2019-12-05T06:06:36.629682+0000","last_became_peered":"2019-12-05T06:06:36.629682+0000","last_unstale":"2019-12-05T06:06:44.723134+0000","last_undegraded":"2019-12-05T06:06:44.723134+0000","last_fullsized":"2019-12-05T06:06:44.723134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.484464+0000","last_change":"2019-12-05T06:06:32.595548+0000","last_active":"2019-12-05T06:06:40.484464+0000","last_peered":"2019-12-05T06:06:40.484464+0000","last_clean":"2019-12-05T06:06:40.484464+0000","last_became_active":"2019-12-05T06:06:32.594119+0000","last_became_peered":"2019-12-05T06:06:32.594119+0000","last_unstale":"2019-12-05T06:06:40.484464+0000","last_undegraded":"2019-12-05T06:06:40.484464+0000","last_fullsized":"2019-12-05T06:06:40.484464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.014770+0000","last_change":"2019-12-05T06:06:39.649718+0000","last_active":"2019-12-05T06:06:41.014770+0000","last_peered":"2019-12-05T06:06:41.014770+0000","last_clean":"2019-12-05T06:06:41.014770+0000","last_became_active":"2019-12-05T06:06:39.648278+0000","last_became_peered":"2019-12-05T06:06:39.648278+0000","last_unstale":"2019-12-05T06:06:41.014770+0000","last_undegraded":"2019-12-05T06:06:41.014770+0000","last_fullsized":"2019-12-05T06:06:41.014770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":96686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.712}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.670}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.789}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.881}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.611}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.526}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.924}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":96526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.720}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.824}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.679}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.214}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.638}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.139}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.635}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.831}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.120}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.877}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.910}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.092}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.064}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.842}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.928}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.796}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":53.672}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.693}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.967}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":97646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.483}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.353}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.664}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.578}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.710}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.547}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.635}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":97486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":153.773}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.765}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.470}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.963}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.827}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.021}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.291}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.570}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.419}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.936}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.717}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.755}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.859}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.949}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.559}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.654}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.906}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.077}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.873}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.774}]}]}]}} 2019-12-05T06:06:55.714 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T06:06:55.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.957+0000 7f1ef1d62700 1 Processor -- start 2019-12-05T06:06:55.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.957+0000 7f1ef1d62700 1 -- start start 2019-12-05T06:06:55.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.959+0000 7f1ef1d62700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1eec08c6d0 0x7f1eec08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:55.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.959+0000 7f1ef1d62700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1eec08f720 0x7f1eec08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:55.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.959+0000 7f1ef1d62700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec08fb80 con 0x7f1eec08fef0 2019-12-05T06:06:55.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.959+0000 7f1ef1d62700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec08df90 con 0x7f1eec08c6d0 2019-12-05T06:06:55.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.959+0000 7f1ef1d62700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec0a3f70 con 0x7f1eec08f720 2019-12-05T06:06:55.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.959+0000 7f1eebfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1eec08c6d0 0x7f1eec08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.959+0000 7f1eebfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1eec08c6d0 0x7f1eec08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46472/0 (socket says 172.21.15.16:46472) 2019-12-05T06:06:55.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.959+0000 7f1eebfff700 1 -- 172.21.15.16:0/1591475260 learned_addr learned my addr 172.21.15.16:0/1591475260 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:55.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.959+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/1591475260 >> v2:172.21.15.76:3300/0 conn(0x7f1eec08f720 0x7f1eec08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:55.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.960+0000 7f1eebfff700 1 -- 172.21.15.16:0/1591475260 >> v2:172.21.15.76:3300/0 conn(0x7f1eec08f720 msgr2=0x7f1eec08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:55.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.960+0000 7f1eebfff700 1 --2- 172.21.15.16:0/1591475260 >> v2:172.21.15.76:3300/0 conn(0x7f1eec08f720 0x7f1eec08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:55.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.960+0000 7f1eebfff700 1 -- 172.21.15.16:0/1591475260 >> v1:172.21.15.16:6789/0 conn(0x7f1eec08fef0 legacy=0x7f1eec08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.960+0000 7f1eebfff700 1 -- 172.21.15.16:0/1591475260 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec08d540 con 0x7f1eec08c6d0 2019-12-05T06:06:55.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.960+0000 7f1eebfff700 1 --2- 172.21.15.16:0/1591475260 >> v2:172.21.15.16:3300/0 conn(0x7f1eec08c6d0 0x7f1eec08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1edc00fac0 tx=0x7f1edc00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.960+0000 7f1eebfff700 1 --2- 172.21.15.16:0/1591475260 >> v2:172.21.15.16:3300/0 conn(0x7f1eec08c6d0 0x7f1eec08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1edc00fac0 tx=0x7f1edc00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.960+0000 7f1eebfff700 1 --2- 172.21.15.16:0/1591475260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec08c6d0 0x7f1eec08cac0 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7f1edc00fac0 tx=0x7f1edc00f8f0).ready entity=mon.0 client_cookie=1caaaafa1ed3faf8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:55.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.960+0000 7f1eebfff700 1 --2- 172.21.15.16:0/1591475260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec08c6d0 0x7f1eec08cac0 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f1edc00fac0 tx=0x7f1edc00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.960+0000 7f1eebfff700 1 --2- 172.21.15.16:0/1591475260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec08c6d0 0x7f1eec08cac0 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f1edc00fac0 tx=0x7f1edc00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.960+0000 7f1eeaffd700 1 -- 172.21.15.16:0/1591475260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1edc000ce0 con 0x7f1eec08c6d0 2019-12-05T06:06:55.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.961+0000 7f1eebfff700 1 --2- 172.21.15.16:0/1591475260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec08c6d0 0x7f1eec08cac0 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f1edc00fac0 tx=0x7f1edc00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:55.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.961+0000 7f1eeaffd700 1 -- 172.21.15.16:0/1591475260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1edc001010 con 0x7f1eec08c6d0 2019-12-05T06:06:55.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.961+0000 7f1eeaffd700 1 -- 172.21.15.16:0/1591475260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1edc001720 con 0x7f1eec08c6d0 2019-12-05T06:06:55.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.961+0000 7f1ef1d62700 1 -- 172.21.15.16:0/1591475260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec08c6d0 msgr2=0x7f1eec08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:55.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.961+0000 7f1ef1d62700 1 --2- 172.21.15.16:0/1591475260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec08c6d0 0x7f1eec08cac0 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7f1edc00fac0 tx=0x7f1edc00f8f0).stop 2019-12-05T06:06:55.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.962+0000 7f1ef1d62700 1 -- 172.21.15.16:0/1591475260 shutdown_connections 2019-12-05T06:06:55.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.962+0000 7f1ef1d62700 1 --2- 172.21.15.16:0/1591475260 >> v2:172.21.15.76:3300/0 conn(0x7f1eec08f720 0x7f1eec08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:55.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.962+0000 7f1ef1d62700 1 --2- 172.21.15.16:0/1591475260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eec08c6d0 0x7f1eec08cac0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:55.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.962+0000 7f1ef1d62700 1 -- 172.21.15.16:0/1591475260 shutdown_connections 2019-12-05T06:06:55.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.962+0000 7f1ef1d62700 1 -- 172.21.15.16:0/1591475260 wait complete. 2019-12-05T06:06:55.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.962+0000 7f1ef1d62700 1 -- 172.21.15.16:0/1591475260 >> 172.21.15.16:0/1591475260 conn(0x7f1eec061940 msgr2=0x7f1eec096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:55.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.967+0000 7f1ef1d62700 1 Processor -- start 2019-12-05T06:06:55.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.967+0000 7f1ef1d62700 1 -- start start 2019-12-05T06:06:55.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.969+0000 7f1ef1d62700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1eec084f30 0x7f1eec157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:55.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.969+0000 7f1ef1d62700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec084b90 con 0x7f1eec096e90 2019-12-05T06:06:55.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.969+0000 7f1ef1d62700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eec098790 con 0x7f1eec09bbe0 2019-12-05T06:06:55.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.969+0000 7f1ef1d62700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eec09a450 con 0x7f1eec084f30 2019-12-05T06:06:55.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.969+0000 7f1eebfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1eec096e90 0x7f1eec09b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59878/0 (socket says 172.21.15.16:59878) 2019-12-05T06:06:55.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.969+0000 7f1eebfff700 1 -- 172.21.15.16:0/662470845 learned_addr learned my addr 172.21.15.16:0/662470845 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:55.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.969+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/662470845 >> v2:172.21.15.76:3300/0 conn(0x7f1eec084f30 0x7f1eec157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:56.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.970+0000 7f1ee8ff9700 1 -- 172.21.15.16:0/662470845 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680504083 0 0) 0x7f1edc007d20 con 0x7f1eec096e90 2019-12-05T06:06:56.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.970+0000 7f1ee8ff9700 1 -- 172.21.15.16:0/662470845 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ed0002e80 con 0x7f1eec096e90 2019-12-05T06:06:56.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.970+0000 7f1ef0b00700 1 -- 172.21.15.16:0/662470845 >> v1:172.21.15.76:6789/0 conn(0x7f1eec09bbe0 legacy=0x7f1eec1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:56.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.970+0000 7f1ee8ff9700 1 -- 172.21.15.16:0/662470845 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1786027790 0 0) 0x7f1ed4001100 con 0x7f1eec09bbe0 2019-12-05T06:06:56.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.970+0000 7f1ef0b00700 1 -- 172.21.15.16:0/662470845 >> v1:172.21.15.16:6789/0 conn(0x7f1eec096e90 legacy=0x7f1eec09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:56.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.970+0000 7f1ef0b00700 1 -- 172.21.15.16:0/662470845 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eec09aae0 con 0x7f1eec084f30 2019-12-05T06:06:56.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.970+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/662470845 >> v2:172.21.15.76:3300/0 conn(0x7f1eec084f30 0x7f1eec157200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1ee4010390 tx=0x7f1ee4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.970+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/662470845 >> v2:172.21.15.76:3300/0 conn(0x7f1eec084f30 0x7f1eec157200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1ee4010390 tx=0x7f1ee4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.970+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1eec084f30 0x7f1eec157200 secure :-1 s=READY pgs=124 cs=0 l=1 rx=0x7f1ee4010390 tx=0x7f1ee4010730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:56.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.971+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1eec084f30 0x7f1eec157200 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f1ee4010390 tx=0x7f1ee4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.971+0000 7f1ee8ff9700 1 -- 172.21.15.16:0/662470845 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ee4009990 con 0x7f1eec084f30 2019-12-05T06:06:56.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.971+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1eec084f30 0x7f1eec157200 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f1ee4010390 tx=0x7f1ee4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.971+0000 7f1ee8ff9700 1 -- 172.21.15.16:0/662470845 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ee40095b0 con 0x7f1eec084f30 2019-12-05T06:06:56.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.971+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1eec084f30 0x7f1eec157200 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f1ee4010390 tx=0x7f1ee4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.971+0000 7f1ef1d62700 1 -- 172.21.15.16:0/662470845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eec09aae0 con 0x7f1eec084f30 2019-12-05T06:06:56.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.971+0000 7f1ee8ff9700 1 -- 172.21.15.16:0/662470845 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ee4009c20 con 0x7f1eec084f30 2019-12-05T06:06:56.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.971+0000 7f1ef1d62700 1 -- 172.21.15.16:0/662470845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eec09aae0 con 0x7f1eec084f30 2019-12-05T06:06:56.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.972+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1eec084f30 0x7f1eec157200 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f1ee4010390 tx=0x7f1ee4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.972+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1eec084f30 0x7f1eec157200 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f1ee4010390 tx=0x7f1ee4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.972+0000 7f1ee8ff9700 1 -- 172.21.15.16:0/662470845 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1ee4009c40 con 0x7f1eec084f30 2019-12-05T06:06:56.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.973+0000 7f1ee8ff9700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1ed0023680 0x7f1ed0025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:56.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.973+0000 7f1ee8ff9700 1 -- 172.21.15.16:0/662470845 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f1ee4045d90 con 0x7f1eec084f30 2019-12-05T06:06:56.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.973+0000 7f1eebfff700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1ed0023680 0x7f1ed0025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:56.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.973+0000 7f1eebfff700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1ed0023680 0x7f1ed0025af0 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:56.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.989+0000 7f1ef1d62700 1 -- 172.21.15.16:0/662470845 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eec09aae0 con 0x7f1eec084f30 2019-12-05T06:06:56.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.999+0000 7f1ef0b00700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1eec084f30 0x7f1eec157200 secure :-1 s=THROTTLE_DONE pgs=124 cs=0 l=1 rx=0x7f1ee4010390 tx=0x7f1ee4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:55.999+0000 7f1ee8ff9700 1 -- 172.21.15.16:0/662470845 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1ee4042b60 con 0x7f1eec084f30 2019-12-05T06:06:56.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.268+0000 7f1ef1d62700 1 -- 172.21.15.16:0/662470845 --> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f1eec09aae0 con 0x7f1ed0023680 2019-12-05T06:06:56.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.276+0000 7f1ee8ff9700 1 -- 172.21.15.16:0/662470845 <== mgr.4100 v2:172.21.15.76:6800/9546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80416 (crc 0 0 0) 0x7f1edc007900 con 0x7f1ed0023680 2019-12-05T06:06:56.284 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:56.284 INFO:teuthology.orchestra.run.smithi016.stderr:dumped all 2019-12-05T06:06:56.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.286+0000 7f1ef1d62700 1 -- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1ed0023680 msgr2=0x7f1ed0025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.286+0000 7f1ef1d62700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1ed0023680 0x7f1ed0025af0 crc :-1 s=READY pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.286+0000 7f1ef1d62700 1 -- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1eec084f30 msgr2=0x7f1eec157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.286+0000 7f1ef1d62700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1eec084f30 0x7f1eec157200 secure :-1 s=READY pgs=124 cs=0 l=1 rx=0x7f1ee4010390 tx=0x7f1ee4010730).stop 2019-12-05T06:06:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.287+0000 7f1ef1d62700 1 -- 172.21.15.16:0/662470845 shutdown_connections 2019-12-05T06:06:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.287+0000 7f1ef1d62700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1ed0023680 0x7f1ed0025af0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:56.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.287+0000 7f1ef1d62700 1 --2- 172.21.15.16:0/662470845 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1eec084f30 0x7f1eec157200 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.287+0000 7f1ef1d62700 1 -- 172.21.15.16:0/662470845 shutdown_connections 2019-12-05T06:06:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.287+0000 7f1ef1d62700 1 -- 172.21.15.16:0/662470845 wait complete. 2019-12-05T06:06:56.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.287+0000 7f1ef1d62700 1 -- 172.21.15.16:0/662470845 >> 172.21.15.16:0/662470845 conn(0x7f1eec0714f0 msgr2=0x7f1eec09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:56.305 INFO:teuthology.orchestra.run.smithi016.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2019-12-05T06:06:54.768321+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390720,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584000,"statfs":{"total":773094113280,"available":764502016000,"internally_reserved":8589934592,"allocated":2097152,"data_stored":772252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.002011"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.271382+0000","last_change":"2019-12-05T06:06:32.598114+0000","last_active":"2019-12-05T06:06:40.271382+0000","last_peered":"2019-12-05T06:06:40.271382+0000","last_clean":"2019-12-05T06:06:40.271382+0000","last_became_active":"2019-12-05T06:06:32.597387+0000","last_became_peered":"2019-12-05T06:06:32.597387+0000","last_unstale":"2019-12-05T06:06:40.271382+0000","last_undegraded":"2019-12-05T06:06:40.271382+0000","last_fullsized":"2019-12-05T06:06:40.271382+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"24'8","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.715381+0000","last_change":"2019-12-05T06:06:36.627910+0000","last_active":"2019-12-05T06:06:44.715381+0000","last_peered":"2019-12-05T06:06:44.715381+0000","last_clean":"2019-12-05T06:06:44.715381+0000","last_became_active":"2019-12-05T06:06:36.626897+0000","last_became_peered":"2019-12-05T06:06:36.626897+0000","last_unstale":"2019-12-05T06:06:44.715381+0000","last_undegraded":"2019-12-05T06:06:44.715381+0000","last_fullsized":"2019-12-05T06:06:44.715381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.490878+0000","last_change":"2019-12-05T06:06:40.490139+0000","last_active":"2019-12-05T06:06:40.490878+0000","last_peered":"2019-12-05T06:06:40.490878+0000","last_clean":"2019-12-05T06:06:40.490878+0000","last_became_active":"2019-12-05T06:06:40.488804+0000","last_became_peered":"2019-12-05T06:06:40.488804+0000","last_unstale":"2019-12-05T06:06:40.490878+0000","last_undegraded":"2019-12-05T06:06:40.490878+0000","last_fullsized":"2019-12-05T06:06:40.490878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.271284+0000","last_change":"2019-12-05T06:06:32.887464+0000","last_active":"2019-12-05T06:06:40.271284+0000","last_peered":"2019-12-05T06:06:40.271284+0000","last_clean":"2019-12-05T06:06:40.271284+0000","last_became_active":"2019-12-05T06:06:32.885955+0000","last_became_peered":"2019-12-05T06:06:32.885955+0000","last_unstale":"2019-12-05T06:06:40.271284+0000","last_undegraded":"2019-12-05T06:06:40.271284+0000","last_fullsized":"2019-12-05T06:06:40.271284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720231+0000","last_change":"2019-12-05T06:06:36.631805+0000","last_active":"2019-12-05T06:06:44.720231+0000","last_peered":"2019-12-05T06:06:44.720231+0000","last_clean":"2019-12-05T06:06:44.720231+0000","last_became_active":"2019-12-05T06:06:36.631077+0000","last_became_peered":"2019-12-05T06:06:36.631077+0000","last_unstale":"2019-12-05T06:06:44.720231+0000","last_undegraded":"2019-12-05T06:06:44.720231+0000","last_fullsized":"2019-12-05T06:06:44.720231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.485950+0000","last_change":"2019-12-05T06:06:40.485444+0000","last_active":"2019-12-05T06:06:40.485950+0000","last_peered":"2019-12-05T06:06:40.485950+0000","last_clean":"2019-12-05T06:06:40.485950+0000","last_became_active":"2019-12-05T06:06:40.484465+0000","last_became_peered":"2019-12-05T06:06:40.484465+0000","last_unstale":"2019-12-05T06:06:40.485950+0000","last_undegraded":"2019-12-05T06:06:40.485950+0000","last_fullsized":"2019-12-05T06:06:40.485950+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576769+0000","last_change":"2019-12-05T06:06:40.488691+0000","last_active":"2019-12-05T06:06:40.576769+0000","last_peered":"2019-12-05T06:06:40.576769+0000","last_clean":"2019-12-05T06:06:40.576769+0000","last_became_active":"2019-12-05T06:06:40.487506+0000","last_became_peered":"2019-12-05T06:06:40.487506+0000","last_unstale":"2019-12-05T06:06:40.576769+0000","last_undegraded":"2019-12-05T06:06:40.576769+0000","last_fullsized":"2019-12-05T06:06:40.576769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.268843+0000","last_change":"2019-12-05T06:06:32.597542+0000","last_active":"2019-12-05T06:06:40.268843+0000","last_peered":"2019-12-05T06:06:40.268843+0000","last_clean":"2019-12-05T06:06:40.268843+0000","last_became_active":"2019-12-05T06:06:32.596579+0000","last_became_peered":"2019-12-05T06:06:32.596579+0000","last_unstale":"2019-12-05T06:06:40.268843+0000","last_undegraded":"2019-12-05T06:06:40.268843+0000","last_fullsized":"2019-12-05T06:06:40.268843+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'4","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.710303+0000","last_change":"2019-12-05T06:06:36.626828+0000","last_active":"2019-12-05T06:06:44.710303+0000","last_peered":"2019-12-05T06:06:44.710303+0000","last_clean":"2019-12-05T06:06:44.710303+0000","last_became_active":"2019-12-05T06:06:36.626005+0000","last_became_peered":"2019-12-05T06:06:36.626005+0000","last_unstale":"2019-12-05T06:06:44.710303+0000","last_undegraded":"2019-12-05T06:06:44.710303+0000","last_fullsized":"2019-12-05T06:06:44.710303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.015035+0000","last_change":"2019-12-05T06:06:40.487696+0000","last_active":"2019-12-05T06:06:41.015035+0000","last_peered":"2019-12-05T06:06:41.015035+0000","last_clean":"2019-12-05T06:06:41.015035+0000","last_became_active":"2019-12-05T06:06:40.486899+0000","last_became_peered":"2019-12-05T06:06:40.486899+0000","last_unstale":"2019-12-05T06:06:41.015035+0000","last_undegraded":"2019-12-05T06:06:41.015035+0000","last_fullsized":"2019-12-05T06:06:41.015035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576888+0000","last_change":"2019-12-05T06:06:32.597098+0000","last_active":"2019-12-05T06:06:40.576888+0000","last_peered":"2019-12-05T06:06:40.576888+0000","last_clean":"2019-12-05T06:06:40.576888+0000","last_became_active":"2019-12-05T06:06:32.596276+0000","last_became_peered":"2019-12-05T06:06:32.596276+0000","last_unstale":"2019-12-05T06:06:40.576888+0000","last_undegraded":"2019-12-05T06:06:40.576888+0000","last_fullsized":"2019-12-05T06:06:40.576888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"24'7","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720981+0000","last_change":"2019-12-05T06:06:36.629032+0000","last_active":"2019-12-05T06:06:44.720981+0000","last_peered":"2019-12-05T06:06:44.720981+0000","last_clean":"2019-12-05T06:06:44.720981+0000","last_became_active":"2019-12-05T06:06:36.628405+0000","last_became_peered":"2019-12-05T06:06:36.628405+0000","last_unstale":"2019-12-05T06:06:44.720981+0000","last_undegraded":"2019-12-05T06:06:44.720981+0000","last_fullsized":"2019-12-05T06:06:44.720981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.084640+0000","last_change":"2019-12-05T06:06:32.886929+0000","last_active":"2019-12-05T06:06:41.084640+0000","last_peered":"2019-12-05T06:06:41.084640+0000","last_clean":"2019-12-05T06:06:41.084640+0000","last_became_active":"2019-12-05T06:06:32.886095+0000","last_became_peered":"2019-12-05T06:06:32.886095+0000","last_unstale":"2019-12-05T06:06:41.084640+0000","last_undegraded":"2019-12-05T06:06:41.084640+0000","last_fullsized":"2019-12-05T06:06:41.084640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"24'5","reported_seq":"17","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720647+0000","last_change":"2019-12-05T06:06:36.632241+0000","last_active":"2019-12-05T06:06:44.720647+0000","last_peered":"2019-12-05T06:06:44.720647+0000","last_clean":"2019-12-05T06:06:44.720647+0000","last_became_active":"2019-12-05T06:06:36.631418+0000","last_became_peered":"2019-12-05T06:06:36.631418+0000","last_unstale":"2019-12-05T06:06:44.720647+0000","last_undegraded":"2019-12-05T06:06:44.720647+0000","last_fullsized":"2019-12-05T06:06:44.720647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576879+0000","last_change":"2019-12-05T06:06:39.649718+0000","last_active":"2019-12-05T06:06:40.576879+0000","last_peered":"2019-12-05T06:06:40.576879+0000","last_clean":"2019-12-05T06:06:40.576879+0000","last_became_active":"2019-12-05T06:06:39.648680+0000","last_became_peered":"2019-12-05T06:06:39.648680+0000","last_unstale":"2019-12-05T06:06:40.576879+0000","last_undegraded":"2019-12-05T06:06:40.576879+0000","last_fullsized":"2019-12-05T06:06:40.576879+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.269507+0000","last_change":"2019-12-05T06:06:32.886427+0000","last_active":"2019-12-05T06:06:40.269507+0000","last_peered":"2019-12-05T06:06:40.269507+0000","last_clean":"2019-12-05T06:06:40.269507+0000","last_became_active":"2019-12-05T06:06:32.885509+0000","last_became_peered":"2019-12-05T06:06:32.885509+0000","last_unstale":"2019-12-05T06:06:40.269507+0000","last_undegraded":"2019-12-05T06:06:40.269507+0000","last_fullsized":"2019-12-05T06:06:40.269507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"24'1","reported_seq":"13","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:41.269608+0000","last_change":"2019-12-05T06:06:36.632232+0000","last_active":"2019-12-05T06:06:41.269608+0000","last_peered":"2019-12-05T06:06:41.269608+0000","last_clean":"2019-12-05T06:06:41.269608+0000","last_became_active":"2019-12-05T06:06:36.630785+0000","last_became_peered":"2019-12-05T06:06:36.630785+0000","last_unstale":"2019-12-05T06:06:41.269608+0000","last_undegraded":"2019-12-05T06:06:41.269608+0000","last_fullsized":"2019-12-05T06:06:41.269608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.268862+0000","last_change":"2019-12-05T06:06:39.647777+0000","last_active":"2019-12-05T06:06:40.268862+0000","last_peered":"2019-12-05T06:06:40.268862+0000","last_clean":"2019-12-05T06:06:40.268862+0000","last_became_active":"2019-12-05T06:06:39.646655+0000","last_became_peered":"2019-12-05T06:06:39.646655+0000","last_unstale":"2019-12-05T06:06:40.268862+0000","last_undegraded":"2019-12-05T06:06:40.268862+0000","last_fullsized":"2019-12-05T06:06:40.268862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.547093+0000","last_change":"2019-12-05T06:06:36.628444+0000","last_active":"2019-12-05T06:06:40.547093+0000","last_peered":"2019-12-05T06:06:40.547093+0000","last_clean":"2019-12-05T06:06:40.547093+0000","last_became_active":"2019-12-05T06:06:36.627555+0000","last_became_peered":"2019-12-05T06:06:36.627555+0000","last_unstale":"2019-12-05T06:06:40.547093+0000","last_undegraded":"2019-12-05T06:06:40.547093+0000","last_fullsized":"2019-12-05T06:06:40.547093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.084584+0000","last_change":"2019-12-05T06:06:32.886895+0000","last_active":"2019-12-05T06:06:41.084584+0000","last_peered":"2019-12-05T06:06:41.084584+0000","last_clean":"2019-12-05T06:06:41.084584+0000","last_became_active":"2019-12-05T06:06:32.886123+0000","last_became_peered":"2019-12-05T06:06:32.886123+0000","last_unstale":"2019-12-05T06:06:41.084584+0000","last_undegraded":"2019-12-05T06:06:41.084584+0000","last_fullsized":"2019-12-05T06:06:41.084584+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.486413+0000","last_change":"2019-12-05T06:06:40.486073+0000","last_active":"2019-12-05T06:06:40.486413+0000","last_peered":"2019-12-05T06:06:40.486413+0000","last_clean":"2019-12-05T06:06:40.486413+0000","last_became_active":"2019-12-05T06:06:40.484954+0000","last_became_peered":"2019-12-05T06:06:40.484954+0000","last_unstale":"2019-12-05T06:06:40.486413+0000","last_undegraded":"2019-12-05T06:06:40.486413+0000","last_fullsized":"2019-12-05T06:06:40.486413+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.723134+0000","last_change":"2019-12-05T06:06:36.630520+0000","last_active":"2019-12-05T06:06:44.723134+0000","last_peered":"2019-12-05T06:06:44.723134+0000","last_clean":"2019-12-05T06:06:44.723134+0000","last_became_active":"2019-12-05T06:06:36.629682+0000","last_became_peered":"2019-12-05T06:06:36.629682+0000","last_unstale":"2019-12-05T06:06:44.723134+0000","last_undegraded":"2019-12-05T06:06:44.723134+0000","last_fullsized":"2019-12-05T06:06:44.723134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.484464+0000","last_change":"2019-12-05T06:06:32.595548+0000","last_active":"2019-12-05T06:06:40.484464+0000","last_peered":"2019-12-05T06:06:40.484464+0000","last_clean":"2019-12-05T06:06:40.484464+0000","last_became_active":"2019-12-05T06:06:32.594119+0000","last_became_peered":"2019-12-05T06:06:32.594119+0000","last_unstale":"2019-12-05T06:06:40.484464+0000","last_undegraded":"2019-12-05T06:06:40.484464+0000","last_fullsized":"2019-12-05T06:06:40.484464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.014770+0000","last_change":"2019-12-05T06:06:39.649718+0000","last_active":"2019-12-05T06:06:41.014770+0000","last_peered":"2019-12-05T06:06:41.014770+0000","last_clean":"2019-12-05T06:06:41.014770+0000","last_became_active":"2019-12-05T06:06:39.648278+0000","last_became_peered":"2019-12-05T06:06:39.648278+0000","last_unstale":"2019-12-05T06:06:41.014770+0000","last_undegraded":"2019-12-05T06:06:41.014770+0000","last_fullsized":"2019-12-05T06:06:41.014770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":96686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.712}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.670}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.789}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.881}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.611}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.526}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.924}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":96526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.720}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.824}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.679}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.214}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.638}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.085}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.139}]}]},{"osd":5,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.635}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.831}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.120}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.877}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.910}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.092}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.064}]}]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.842}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.928}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.796}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":53.672}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.693}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.967}]}]},{"osd":0,"up_from":13,"seq":55834574856,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":97646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.483}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.353}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.664}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.578}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.710}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.547}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.635}]}]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":97486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":153.773}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.765}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.470}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.963}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.827}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.021}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.291}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.495}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.570}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.419}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.936}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.717}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.755}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.859}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.949}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.559}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.654}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.906}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.077}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.873}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.774}]}]}]}} 2019-12-05T06:06:56.309 INFO:tasks.ceph.ceph_manager.ceph:clean! 2019-12-05T06:06:56.309 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2019-12-05T06:06:56.310 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2019-12-05T06:06:56.310 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2019-12-05T06:06:56.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.593+0000 7fd8991f4700 1 Processor -- start 2019-12-05T06:06:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.593+0000 7fd8991f4700 1 -- start start 2019-12-05T06:06:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.595+0000 7fd8991f4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd89408d2c0 0x7fd89408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.595+0000 7fd8991f4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd89408fef0 0x7fd89408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.595+0000 7fd8991f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8940958e0 con 0x7fd89408c6d0 2019-12-05T06:06:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.595+0000 7fd8991f4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89408df90 con 0x7fd89408d2c0 2019-12-05T06:06:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.595+0000 7fd8991f4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8940a3f70 con 0x7fd89408fef0 2019-12-05T06:06:56.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.595+0000 7fd892d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd89408d2c0 0x7fd89408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.595+0000 7fd892d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd89408d2c0 0x7fd89408ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46494/0 (socket says 172.21.15.16:46494) 2019-12-05T06:06:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.595+0000 7fd892d9d700 1 -- 172.21.15.16:0/2005271512 learned_addr learned my addr 172.21.15.16:0/2005271512 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.595+0000 7fd89359e700 1 --1- 172.21.15.16:0/2005271512 >> v1:172.21.15.16:6789/0 conn(0x7fd89408c6d0 0x7fd89408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59892/0 (socket says 172.21.15.16:59892) 2019-12-05T06:06:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.595+0000 7fd8989f3700 1 --2- 172.21.15.16:0/2005271512 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd89408fef0 0x7fd89408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.596+0000 7fd892d9d700 1 -- 172.21.15.16:0/2005271512 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd89408fef0 msgr2=0x7fd89408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.596+0000 7fd892d9d700 1 --2- 172.21.15.16:0/2005271512 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd89408fef0 0x7fd89408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:56.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.596+0000 7fd892d9d700 1 -- 172.21.15.16:0/2005271512 >> v1:172.21.15.16:6789/0 conn(0x7fd89408c6d0 legacy=0x7fd89408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.596+0000 7fd89259c700 1 -- 172.21.15.16:0/2005271512 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650027617 0 0) 0x7fd884001480 con 0x7fd89408c6d0 2019-12-05T06:06:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.596+0000 7fd892d9d700 1 -- 172.21.15.16:0/2005271512 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89408e620 con 0x7fd89408d2c0 2019-12-05T06:06:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.596+0000 7fd892d9d700 1 --2- 172.21.15.16:0/2005271512 >> v2:172.21.15.16:3300/0 conn(0x7fd89408d2c0 0x7fd89408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd87c00f6c0 tx=0x7fd87c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.596+0000 7fd892d9d700 1 --2- 172.21.15.16:0/2005271512 >> v2:172.21.15.16:3300/0 conn(0x7fd89408d2c0 0x7fd89408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd87c00f6c0 tx=0x7fd87c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.596+0000 7fd892d9d700 1 --2- 172.21.15.16:0/2005271512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89408d2c0 0x7fd89408ef20 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7fd87c00f6c0 tx=0x7fd87c00fa60).ready entity=mon.0 client_cookie=ed5c7dcdf1208d28 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:56.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.597+0000 7fd892d9d700 1 --2- 172.21.15.16:0/2005271512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89408d2c0 0x7fd89408ef20 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7fd87c00f6c0 tx=0x7fd87c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.597+0000 7fd89259c700 1 -- 172.21.15.16:0/2005271512 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd87c000960 con 0x7fd89408d2c0 2019-12-05T06:06:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.597+0000 7fd892d9d700 1 --2- 172.21.15.16:0/2005271512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89408d2c0 0x7fd89408ef20 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7fd87c00f6c0 tx=0x7fd87c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.597+0000 7fd892d9d700 1 --2- 172.21.15.16:0/2005271512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89408d2c0 0x7fd89408ef20 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7fd87c00f6c0 tx=0x7fd87c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.597+0000 7fd89259c700 1 -- 172.21.15.16:0/2005271512 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd87c000c90 con 0x7fd89408d2c0 2019-12-05T06:06:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.597+0000 7fd89259c700 1 -- 172.21.15.16:0/2005271512 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd87c0013a0 con 0x7fd89408d2c0 2019-12-05T06:06:56.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.597+0000 7fd8991f4700 1 -- 172.21.15.16:0/2005271512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89408d2c0 msgr2=0x7fd89408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:56.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.597+0000 7fd8991f4700 1 --2- 172.21.15.16:0/2005271512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89408d2c0 0x7fd89408ef20 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7fd87c00f6c0 tx=0x7fd87c00fa60).stop 2019-12-05T06:06:56.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.599+0000 7fd8991f4700 1 -- 172.21.15.16:0/2005271512 shutdown_connections 2019-12-05T06:06:56.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.599+0000 7fd8991f4700 1 --2- 172.21.15.16:0/2005271512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89408d2c0 0x7fd89408ef20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:56.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.599+0000 7fd8991f4700 1 --2- 172.21.15.16:0/2005271512 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd89408fef0 0x7fd89408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:56.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.599+0000 7fd8991f4700 1 -- 172.21.15.16:0/2005271512 shutdown_connections 2019-12-05T06:06:56.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.599+0000 7fd8991f4700 1 -- 172.21.15.16:0/2005271512 wait complete. 2019-12-05T06:06:56.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.599+0000 7fd8991f4700 1 -- 172.21.15.16:0/2005271512 >> 172.21.15.16:0/2005271512 conn(0x7fd894061940 msgr2=0x7fd894096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:56.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.604+0000 7fd8991f4700 1 Processor -- start 2019-12-05T06:06:56.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.605+0000 7fd8991f4700 1 -- start start 2019-12-05T06:06:56.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.607+0000 7fd8991f4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd89414f390 0x7fd89414f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:56.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.607+0000 7fd8991f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd89409fcd0 con 0x7fd894086140 2019-12-05T06:06:56.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.607+0000 7fd8991f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8940865a0 con 0x7fd894086910 2019-12-05T06:06:56.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.607+0000 7fd8991f4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd89409f630 con 0x7fd89414f390 2019-12-05T06:06:56.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.607+0000 7fd89359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd89414f390 0x7fd89414f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.607+0000 7fd8989f3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd894086140 0x7fd894156660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59900/0 (socket says 172.21.15.16:59900) 2019-12-05T06:06:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.607+0000 7fd8989f3700 1 -- 172.21.15.16:0/1291975338 learned_addr learned my addr 172.21.15.16:0/1291975338 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.608+0000 7fd89359e700 1 -- 172.21.15.16:0/1291975338 >> v1:172.21.15.76:6789/0 conn(0x7fd894086910 legacy=0x7fd894085940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.608+0000 7fd88bfff700 1 -- 172.21.15.16:0/1291975338 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3012117156 0 0) 0x7fd87c007900 con 0x7fd894086910 2019-12-05T06:06:56.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.608+0000 7fd89359e700 1 -- 172.21.15.16:0/1291975338 >> v1:172.21.15.16:6789/0 conn(0x7fd894086140 legacy=0x7fd894156660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:56.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.608+0000 7fd89359e700 1 -- 172.21.15.16:0/1291975338 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd894085040 con 0x7fd89414f390 2019-12-05T06:06:56.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.608+0000 7fd89359e700 1 --2- 172.21.15.16:0/1291975338 >> v2:172.21.15.16:3300/0 conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.608+0000 7fd89359e700 1 --2- 172.21.15.16:0/1291975338 >> v2:172.21.15.16:3300/0 conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.608+0000 7fd89359e700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=READY pgs=154 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:56.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.609+0000 7fd89359e700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.609+0000 7fd88bfff700 1 -- 172.21.15.16:0/1291975338 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8840045c0 con 0x7fd89414f390 2019-12-05T06:06:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.609+0000 7fd89359e700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.609+0000 7fd8991f4700 1 -- 172.21.15.16:0/1291975338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd894085040 con 0x7fd89414f390 2019-12-05T06:06:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.609+0000 7fd88bfff700 1 -- 172.21.15.16:0/1291975338 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd884001010 con 0x7fd89414f390 2019-12-05T06:06:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.609+0000 7fd89359e700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.609+0000 7fd88bfff700 1 -- 172.21.15.16:0/1291975338 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8840049a0 con 0x7fd89414f390 2019-12-05T06:06:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.609+0000 7fd8991f4700 1 -- 172.21.15.16:0/1291975338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd894085040 con 0x7fd89414f390 2019-12-05T06:06:56.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.610+0000 7fd89359e700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.611+0000 7fd89359e700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.611+0000 7fd88bfff700 1 -- 172.21.15.16:0/1291975338 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd8840045c0 con 0x7fd89414f390 2019-12-05T06:06:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.611+0000 7fd88bfff700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd878023630 0x7fd878025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.611+0000 7fd88bfff700 1 -- 172.21.15.16:0/1291975338 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fd884046120 con 0x7fd89414f390 2019-12-05T06:06:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.612+0000 7fd892d9d700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd878023630 0x7fd878025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:56.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.612+0000 7fd892d9d700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd878023630 0x7fd878025ac0 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:56.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.624+0000 7fd8991f4700 1 -- 172.21.15.16:0/1291975338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd894085040 con 0x7fd89414f390 2019-12-05T06:06:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.632+0000 7fd89359e700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.632+0000 7fd88bfff700 1 -- 172.21.15.16:0/1291975338 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd8840015a0 con 0x7fd89414f390 2019-12-05T06:06:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.872+0000 7fd8991f4700 1 -- 172.21.15.16:0/1291975338 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd894085040 con 0x7fd89414f390 2019-12-05T06:06:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.872+0000 7fd89359e700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=THROTTLE_DONE pgs=154 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:56.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.872+0000 7fd88bfff700 1 -- 172.21.15.16:0/1291975338 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7fd8840045c0 con 0x7fd89414f390 2019-12-05T06:06:56.872 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:56.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.875+0000 7fd8991f4700 1 -- 172.21.15.16:0/1291975338 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd878023630 msgr2=0x7fd878025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.875+0000 7fd8991f4700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd878023630 0x7fd878025ac0 crc :-1 s=READY pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.875+0000 7fd8991f4700 1 -- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 msgr2=0x7fd89414f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.875+0000 7fd8991f4700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 0x7fd89414f780 secure :-1 s=READY pgs=154 cs=0 l=1 rx=0x7fd8840106f0 tx=0x7fd884010a90).stop 2019-12-05T06:06:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.875+0000 7fd8991f4700 1 -- 172.21.15.16:0/1291975338 shutdown_connections 2019-12-05T06:06:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.875+0000 7fd8991f4700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd878023630 0x7fd878025ac0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.875+0000 7fd8991f4700 1 --2- 172.21.15.16:0/1291975338 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd89414f390 0x7fd89414f780 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:56.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.875+0000 7fd8991f4700 1 -- 172.21.15.16:0/1291975338 shutdown_connections 2019-12-05T06:06:56.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.875+0000 7fd8991f4700 1 -- 172.21.15.16:0/1291975338 wait complete. 2019-12-05T06:06:56.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:56.875+0000 7fd8991f4700 1 -- 172.21.15.16:0/1291975338 >> 172.21.15.16:0/1291975338 conn(0x7fd89412f780 msgr2=0x7fd894096b90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:56.902 INFO:teuthology.orchestra.run.smithi016.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2019-12-05T06:06:56.908 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2019-12-05T06:06:56.909 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:06:57.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.124+0000 7f0f1b923700 1 Processor -- start 2019-12-05T06:06:57.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.124+0000 7f0f1b923700 1 -- start start 2019-12-05T06:06:57.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.126+0000 7f0f1b923700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0f1408d2c0 0x7f0f1408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:57.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.126+0000 7f0f1b923700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 0x7f0f1408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.126+0000 7f0f1b923700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f140958e0 con 0x7f0f1408c6d0 2019-12-05T06:06:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.126+0000 7f0f1b923700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f1408df90 con 0x7f0f1408d2c0 2019-12-05T06:06:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.126+0000 7f0f1b923700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f140a3f70 con 0x7f0f1408fef0 2019-12-05T06:06:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.126+0000 7f0f19ec0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f1408c6d0 0x7f0f1408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59906/0 (socket says 172.21.15.16:59906) 2019-12-05T06:06:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.126+0000 7f0f19ec0700 1 -- 172.21.15.16:0/2937949208 learned_addr learned my addr 172.21.15.16:0/2937949208 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:57.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.126+0000 7f0f196bf700 1 --2- 172.21.15.16:0/2937949208 >> v2:172.21.15.76:3300/0 conn(0x7f0f1408d2c0 0x7f0f1408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.126+0000 7f0f1a6c1700 1 --2- 172.21.15.16:0/2937949208 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 0x7f0f1408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.127+0000 7f0f18ebe700 1 -- 172.21.15.16:0/2937949208 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179544069 0 0) 0x7f0f04001150 con 0x7f0f1408c6d0 2019-12-05T06:06:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.127+0000 7f0f18ebe700 1 -- 172.21.15.16:0/2937949208 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0efc002e80 con 0x7f0f1408c6d0 2019-12-05T06:06:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.127+0000 7f0f1a6c1700 1 -- 172.21.15.16:0/2937949208 >> v2:172.21.15.76:3300/0 conn(0x7f0f1408d2c0 msgr2=0x7f0f1408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.127+0000 7f0f1a6c1700 1 --2- 172.21.15.16:0/2937949208 >> v2:172.21.15.76:3300/0 conn(0x7f0f1408d2c0 0x7f0f1408ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:57.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.127+0000 7f0f1a6c1700 1 -- 172.21.15.16:0/2937949208 >> v1:172.21.15.16:6789/0 conn(0x7f0f1408c6d0 legacy=0x7f0f1408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:57.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.127+0000 7f0f1a6c1700 1 -- 172.21.15.16:0/2937949208 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1408e620 con 0x7f0f1408fef0 2019-12-05T06:06:57.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.127+0000 7f0f196bf700 1 --2- 172.21.15.16:0/2937949208 >> v2:172.21.15.76:3300/0 conn(0x7f0f1408d2c0 0x7f0f1408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:57.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.127+0000 7f0f1a6c1700 1 --2- 172.21.15.16:0/2937949208 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 0x7f0f1408f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0f0c00b300 tx=0x7f0f0c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.127+0000 7f0f1a6c1700 1 --2- 172.21.15.16:0/2937949208 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 0x7f0f1408f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0f0c00b300 tx=0x7f0f0c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.128+0000 7f0f1a6c1700 1 --2- 172.21.15.16:0/2937949208 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 0x7f0f1408f460 secure :-1 s=READY pgs=42 cs=0 l=1 rx=0x7f0f0c00b300 tx=0x7f0f0c00bb10).ready entity=mon.2 client_cookie=d6a7fc2c029cd564 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.128+0000 7f0f1a6c1700 1 --2- 172.21.15.16:0/2937949208 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 0x7f0f1408f460 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f0f0c00b300 tx=0x7f0f0c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.128+0000 7f0f1a6c1700 1 --2- 172.21.15.16:0/2937949208 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 0x7f0f1408f460 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f0f0c00b300 tx=0x7f0f0c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.128+0000 7f0f18ebe700 1 -- 172.21.15.16:0/2937949208 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f0c000f40 con 0x7f0f1408fef0 2019-12-05T06:06:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.128+0000 7f0f1a6c1700 1 --2- 172.21.15.16:0/2937949208 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 0x7f0f1408f460 secure :-1 s=THROTTLE_DONE pgs=42 cs=0 l=1 rx=0x7f0f0c00b300 tx=0x7f0f0c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.128+0000 7f0f18ebe700 1 -- 172.21.15.16:0/2937949208 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0c009b70 con 0x7f0f1408fef0 2019-12-05T06:06:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.128+0000 7f0f18ebe700 1 -- 172.21.15.16:0/2937949208 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f0c00a1e0 con 0x7f0f1408fef0 2019-12-05T06:06:57.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.128+0000 7f0f1b923700 1 -- 172.21.15.16:0/2937949208 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 msgr2=0x7f0f1408f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.128+0000 7f0f1b923700 1 --2- 172.21.15.16:0/2937949208 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 0x7f0f1408f460 secure :-1 s=READY pgs=42 cs=0 l=1 rx=0x7f0f0c00b300 tx=0x7f0f0c00bb10).stop 2019-12-05T06:06:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.130+0000 7f0f1b923700 1 -- 172.21.15.16:0/2937949208 shutdown_connections 2019-12-05T06:06:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.130+0000 7f0f1b923700 1 --2- 172.21.15.16:0/2937949208 >> v2:172.21.15.76:3300/0 conn(0x7f0f1408d2c0 0x7f0f1408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.130+0000 7f0f1b923700 1 --2- 172.21.15.16:0/2937949208 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f1408fef0 0x7f0f1408f460 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.130+0000 7f0f1b923700 1 -- 172.21.15.16:0/2937949208 shutdown_connections 2019-12-05T06:06:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.131+0000 7f0f1b923700 1 -- 172.21.15.16:0/2937949208 wait complete. 2019-12-05T06:06:57.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.131+0000 7f0f1b923700 1 -- 172.21.15.16:0/2937949208 >> 172.21.15.16:0/2937949208 conn(0x7f0f14061940 msgr2=0x7f0f14096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:57.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.135+0000 7f0f1b923700 1 Processor -- start 2019-12-05T06:06:57.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.135+0000 7f0f1b923700 1 -- start start 2019-12-05T06:06:57.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.136+0000 7f0f1b923700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0f140992d0 0x7f0f14098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:57.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.137+0000 7f0f1b923700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f14089fb0 con 0x7f0f1406df10 2019-12-05T06:06:57.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.137+0000 7f0f1b923700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f1406e300 con 0x7f0f14156cb0 2019-12-05T06:06:57.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.137+0000 7f0f1b923700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f14097370 con 0x7f0f140992d0 2019-12-05T06:06:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.137+0000 7f0f19ec0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f1406df10 0x7f0f14098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59912/0 (socket says 172.21.15.16:59912) 2019-12-05T06:06:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.137+0000 7f0f19ec0700 1 -- 172.21.15.16:0/1599207898 learned_addr learned my addr 172.21.15.16:0/1599207898 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:57.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.137+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> v2:172.21.15.76:3300/0 conn(0x7f0f140992d0 0x7f0f14098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:57.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.137+0000 7f0f127fc700 1 -- 172.21.15.16:0/1599207898 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1125469814 0 0) 0x7f0f04000f40 con 0x7f0f1406df10 2019-12-05T06:06:57.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.137+0000 7f0f127fc700 1 -- 172.21.15.16:0/1599207898 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ef4002e80 con 0x7f0f1406df10 2019-12-05T06:06:57.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f196bf700 1 -- 172.21.15.16:0/1599207898 >> v1:172.21.15.76:6789/0 conn(0x7f0f14156cb0 legacy=0x7f0f141590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:57.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f127fc700 1 -- 172.21.15.16:0/1599207898 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1265080931 0 0) 0x7f0f0c00b280 con 0x7f0f14156cb0 2019-12-05T06:06:57.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f196bf700 1 -- 172.21.15.16:0/1599207898 >> v1:172.21.15.16:6789/0 conn(0x7f0f1406df10 legacy=0x7f0f14098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:57.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f196bf700 1 -- 172.21.15.16:0/1599207898 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f14097a00 con 0x7f0f140992d0 2019-12-05T06:06:57.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> v2:172.21.15.76:3300/0 conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> v2:172.21.15.76:3300/0 conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:57.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f127fc700 1 -- 172.21.15.16:0/1599207898 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f08000940 con 0x7f0f140992d0 2019-12-05T06:06:57.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f127fc700 1 -- 172.21.15.16:0/1599207898 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f0800dff0 con 0x7f0f140992d0 2019-12-05T06:06:57.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f1b923700 1 -- 172.21.15.16:0/1599207898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f14097a00 con 0x7f0f140992d0 2019-12-05T06:06:57.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.138+0000 7f0f127fc700 1 -- 172.21.15.16:0/1599207898 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f0800e6c0 con 0x7f0f140992d0 2019-12-05T06:06:57.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.139+0000 7f0f1b923700 1 -- 172.21.15.16:0/1599207898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f14097a00 con 0x7f0f140992d0 2019-12-05T06:06:57.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.139+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.140+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.140+0000 7f0f127fc700 1 -- 172.21.15.16:0/1599207898 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0f0800dc70 con 0x7f0f140992d0 2019-12-05T06:06:57.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.140+0000 7f0f127fc700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0ef4023680 0x7f0ef4025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:57.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.141+0000 7f0f127fc700 1 -- 172.21.15.16:0/1599207898 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f0f08046060 con 0x7f0f140992d0 2019-12-05T06:06:57.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.141+0000 7f0f19ec0700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0ef4023680 0x7f0ef4025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:57.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.141+0000 7f0f19ec0700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0ef4023680 0x7f0ef4025af0 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:57.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.155+0000 7f0f1b923700 1 -- 172.21.15.16:0/1599207898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f14097a00 con 0x7f0f140992d0 2019-12-05T06:06:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.165+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.165+0000 7f0f127fc700 1 -- 172.21.15.16:0/1599207898 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0f0800d980 con 0x7f0f140992d0 2019-12-05T06:06:57.176 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:06:57.181 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:06:57.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.433+0000 7f0f1b923700 1 -- 172.21.15.16:0/1599207898 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f14097a00 con 0x7f0f140992d0 2019-12-05T06:06:57.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.436+0000 7f0f196bf700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.436+0000 7f0f127fc700 1 -- 172.21.15.16:0/1599207898 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f0f0800dff0 con 0x7f0f140992d0 2019-12-05T06:06:57.435 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:57.436 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:06:57.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.438+0000 7f0f1b923700 1 -- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0ef4023680 msgr2=0x7f0ef4025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.438+0000 7f0f1b923700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0ef4023680 0x7f0ef4025af0 crc :-1 s=READY pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.438+0000 7f0f1b923700 1 -- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 msgr2=0x7f0f14098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.438+0000 7f0f1b923700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 0x7f0f14098b00 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7f0f080103a0 tx=0x7f0f08010740).stop 2019-12-05T06:06:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.439+0000 7f0f1b923700 1 -- 172.21.15.16:0/1599207898 shutdown_connections 2019-12-05T06:06:57.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.439+0000 7f0f1b923700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0ef4023680 0x7f0ef4025af0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.439+0000 7f0f1b923700 1 --2- 172.21.15.16:0/1599207898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f140992d0 0x7f0f14098b00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.439+0000 7f0f1b923700 1 -- 172.21.15.16:0/1599207898 shutdown_connections 2019-12-05T06:06:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.439+0000 7f0f1b923700 1 -- 172.21.15.16:0/1599207898 wait complete. 2019-12-05T06:06:57.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.439+0000 7f0f1b923700 1 -- 172.21.15.16:0/1599207898 >> 172.21.15.16:0/1599207898 conn(0x7f0f140714f0 msgr2=0x7f0f1409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:57.453 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:31.052682+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:34.862662+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:37.644552+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T06:06:57.453 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T06:06:57.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.709+0000 7f5572351700 1 Processor -- start 2019-12-05T06:06:57.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.709+0000 7f5572351700 1 -- start start 2019-12-05T06:06:57.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.711+0000 7f5572351700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f556c086d70 0x7f556c08aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:57.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.711+0000 7f5572351700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c064310 0x7f556c064760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.711+0000 7f5572351700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c061a10 con 0x7f556c086980 2019-12-05T06:06:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.711+0000 7f5572351700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c060bc0 con 0x7f556c086d70 2019-12-05T06:06:57.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.711+0000 7f5572351700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c060620 con 0x7f556c064310 2019-12-05T06:06:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.711+0000 7f55708ee700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f556c086980 0x7f556c08a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59924/0 (socket says 172.21.15.16:59924) 2019-12-05T06:06:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.711+0000 7f55708ee700 1 -- 172.21.15.16:0/183494460 learned_addr learned my addr 172.21.15.16:0/183494460 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.711+0000 7f556bfff700 1 --2- 172.21.15.16:0/183494460 >> v2:172.21.15.76:3300/0 conn(0x7f556c086d70 0x7f556c08aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.711+0000 7f55710ef700 1 --2- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c064310 0x7f556c064760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.712+0000 7f556b7fe700 1 -- 172.21.15.16:0/183494460 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536263278 0 0) 0x7f55640014d0 con 0x7f556c086980 2019-12-05T06:06:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.712+0000 7f556b7fe700 1 -- 172.21.15.16:0/183494460 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5554002e80 con 0x7f556c086980 2019-12-05T06:06:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.712+0000 7f556bfff700 1 -- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c064310 msgr2=0x7f556c064760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.712+0000 7f556bfff700 1 --2- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c064310 0x7f556c064760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.712+0000 7f556bfff700 1 -- 172.21.15.16:0/183494460 >> v1:172.21.15.16:6789/0 conn(0x7f556c086980 legacy=0x7f556c08a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.712+0000 7f556bfff700 1 -- 172.21.15.16:0/183494460 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c063a10 con 0x7f556c086d70 2019-12-05T06:06:57.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.712+0000 7f55710ef700 1 --2- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c064310 0x7f556c064760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.712+0000 7f556bfff700 1 --2- 172.21.15.16:0/183494460 >> v2:172.21.15.76:3300/0 conn(0x7f556c086d70 0x7f556c08aeb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f556000af50 tx=0x7f556000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.713+0000 7f556bfff700 1 --2- 172.21.15.16:0/183494460 >> v2:172.21.15.76:3300/0 conn(0x7f556c086d70 0x7f556c08aeb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f556000af50 tx=0x7f556000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.713+0000 7f556bfff700 1 --2- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f556c086d70 0x7f556c08aeb0 secure :-1 s=READY pgs=129 cs=0 l=1 rx=0x7f556000af50 tx=0x7f556000b760).ready entity=mon.1 client_cookie=2dff9981177014c5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.713+0000 7f556bfff700 1 --2- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f556c086d70 0x7f556c08aeb0 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7f556000af50 tx=0x7f556000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.713+0000 7f556b7fe700 1 -- 172.21.15.16:0/183494460 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5560007210 con 0x7f556c086d70 2019-12-05T06:06:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.713+0000 7f556bfff700 1 --2- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f556c086d70 0x7f556c08aeb0 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7f556000af50 tx=0x7f556000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.713+0000 7f556bfff700 1 --2- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f556c086d70 0x7f556c08aeb0 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7f556000af50 tx=0x7f556000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.713+0000 7f556b7fe700 1 -- 172.21.15.16:0/183494460 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f55600099b0 con 0x7f556c086d70 2019-12-05T06:06:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.713+0000 7f556b7fe700 1 -- 172.21.15.16:0/183494460 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f556000a080 con 0x7f556c086d70 2019-12-05T06:06:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.713+0000 7f5572351700 1 -- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f556c086d70 msgr2=0x7f556c08aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.713+0000 7f5572351700 1 --2- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f556c086d70 0x7f556c08aeb0 secure :-1 s=READY pgs=129 cs=0 l=1 rx=0x7f556000af50 tx=0x7f556000b760).stop 2019-12-05T06:06:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.715+0000 7f5572351700 1 -- 172.21.15.16:0/183494460 shutdown_connections 2019-12-05T06:06:57.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.715+0000 7f5572351700 1 --2- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f556c086d70 0x7f556c08aeb0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:57.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.715+0000 7f5572351700 1 --2- 172.21.15.16:0/183494460 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c064310 0x7f556c064760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:57.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.715+0000 7f5572351700 1 -- 172.21.15.16:0/183494460 shutdown_connections 2019-12-05T06:06:57.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.715+0000 7f5572351700 1 -- 172.21.15.16:0/183494460 wait complete. 2019-12-05T06:06:57.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.715+0000 7f5572351700 1 -- 172.21.15.16:0/183494460 >> 172.21.15.16:0/183494460 conn(0x7f556c05f5a0 msgr2=0x7f556c09cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:57.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.721+0000 7f5572351700 1 Processor -- start 2019-12-05T06:06:57.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.721+0000 7f5572351700 1 -- start start 2019-12-05T06:06:57.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.723+0000 7f5572351700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f556c14c2c0 0x7f556c14c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.723+0000 7f5572351700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c0973c0 0x7f556c0964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.723+0000 7f5572351700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f556c099de0 con 0x7f556c0969e0 2019-12-05T06:06:57.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.723+0000 7f5572351700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f556c09b880 con 0x7f556c14c2c0 2019-12-05T06:06:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.723+0000 7f5572351700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f556c09ae50 con 0x7f556c0973c0 2019-12-05T06:06:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.724+0000 7f55708ee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f556c14c2c0 0x7f556c14c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.724+0000 7f55708ee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f556c14c2c0 0x7f556c14c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46532/0 (socket says 172.21.15.16:46532) 2019-12-05T06:06:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.724+0000 7f55708ee700 1 -- 172.21.15.16:0/3154875345 learned_addr learned my addr 172.21.15.16:0/3154875345 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.724+0000 7f556bfff700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c0973c0 0x7f556c0964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:57.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.724+0000 7f55708ee700 1 -- 172.21.15.16:0/3154875345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c0973c0 msgr2=0x7f556c0964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:57.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.724+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c0973c0 0x7f556c0964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:57.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.724+0000 7f55697fa700 1 -- 172.21.15.16:0/3154875345 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3601845346 0 0) 0x7f555c0097c0 con 0x7f556c0969e0 2019-12-05T06:06:57.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.724+0000 7f55708ee700 1 -- 172.21.15.16:0/3154875345 >> v1:172.21.15.76:6789/0 conn(0x7f556c0969e0 legacy=0x7f556c094440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:57.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f55708ee700 1 -- 172.21.15.16:0/3154875345 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f556c09bf10 con 0x7f556c14c2c0 2019-12-05T06:06:57.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> v2:172.21.15.16:3300/0 conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> v2:172.21.15.16:3300/0 conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=READY pgs=158 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f55697fa700 1 -- 172.21.15.16:0/3154875345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5564001010 con 0x7f556c14c2c0 2019-12-05T06:06:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f55697fa700 1 -- 172.21.15.16:0/3154875345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5564001440 con 0x7f556c14c2c0 2019-12-05T06:06:57.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f5572351700 1 -- 172.21.15.16:0/3154875345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f556c09bf10 con 0x7f556c14c2c0 2019-12-05T06:06:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f55697fa700 1 -- 172.21.15.16:0/3154875345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5564001ad0 con 0x7f556c14c2c0 2019-12-05T06:06:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.725+0000 7f5572351700 1 -- 172.21.15.16:0/3154875345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f556c09bf10 con 0x7f556c14c2c0 2019-12-05T06:06:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.726+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.727+0000 7f55697fa700 1 -- 172.21.15.16:0/3154875345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5564001010 con 0x7f556c14c2c0 2019-12-05T06:06:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.727+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.728+0000 7f55697fa700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f554c023630 0x7f554c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:57.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.728+0000 7f55697fa700 1 -- 172.21.15.16:0/3154875345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f5564046140 con 0x7f556c14c2c0 2019-12-05T06:06:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.728+0000 7f556bfff700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f554c023630 0x7f554c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.729+0000 7f556bfff700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f554c023630 0x7f554c025ac0 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.740+0000 7f5572351700 1 -- 172.21.15.16:0/3154875345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f556c09bf10 con 0x7f556c14c2c0 2019-12-05T06:06:57.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.749+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:57.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:57.749+0000 7f55697fa700 1 -- 172.21.15.16:0/3154875345 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5564007950 con 0x7f556c14c2c0 2019-12-05T06:06:58.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.002+0000 7f5572351700 1 -- 172.21.15.16:0/3154875345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7f556c09bf10 con 0x7f556c14c2c0 2019-12-05T06:06:58.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.002+0000 7f55708ee700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.002+0000 7f55697fa700 1 -- 172.21.15.16:0/3154875345 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7f5564001010 con 0x7f556c14c2c0 2019-12-05T06:06:58.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.005+0000 7f5572351700 1 -- 172.21.15.16:0/3154875345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f554c023630 msgr2=0x7f554c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.005+0000 7f5572351700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f554c023630 0x7f554c025ac0 crc :-1 s=READY pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.005+0000 7f5572351700 1 -- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 msgr2=0x7f556c14c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.005+0000 7f5572351700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 0x7f556c14c6b0 secure :-1 s=READY pgs=158 cs=0 l=1 rx=0x7f5564010760 tx=0x7f55640105e0).stop 2019-12-05T06:06:58.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.006+0000 7f5572351700 1 -- 172.21.15.16:0/3154875345 shutdown_connections 2019-12-05T06:06:58.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.006+0000 7f5572351700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f556c0973c0 0x7f556c0964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.006+0000 7f5572351700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f554c023630 0x7f554c025ac0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.006+0000 7f5572351700 1 --2- 172.21.15.16:0/3154875345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f556c14c2c0 0x7f556c14c6b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.006+0000 7f5572351700 1 -- 172.21.15.16:0/3154875345 shutdown_connections 2019-12-05T06:06:58.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.006+0000 7f5572351700 1 -- 172.21.15.16:0/3154875345 wait complete. 2019-12-05T06:06:58.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.006+0000 7f5572351700 1 -- 172.21.15.16:0/3154875345 >> 172.21.15.16:0/3154875345 conn(0x7f556c077e10 msgr2=0x7f556c08b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:58.007 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:06:58.035 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T06:06:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.320+0000 7f538a909700 1 Processor -- start 2019-12-05T06:06:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.320+0000 7f538a909700 1 -- start start 2019-12-05T06:06:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.322+0000 7f538a909700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 0x7f5384064b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:58.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.322+0000 7f538a909700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384061a10 con 0x7f5384086980 2019-12-05T06:06:58.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.322+0000 7f538a909700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384092a70 con 0x7f5384086d70 2019-12-05T06:06:58.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.322+0000 7f538a909700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5384060bc0 con 0x7f538408aeb0 2019-12-05T06:06:58.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.322+0000 7f5388ea6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5384086980 0x7f538408a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59942/0 (socket says 172.21.15.16:59942) 2019-12-05T06:06:58.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.322+0000 7f5388ea6700 1 -- 172.21.15.16:0/2321255618 learned_addr learned my addr 172.21.15.16:0/2321255618 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:58.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.322+0000 7f53896a7700 1 --2- 172.21.15.16:0/2321255618 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 0x7f5384064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.323+0000 7f53837fe700 1 -- 172.21.15.16:0/2321255618 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2730581940 0 0) 0x7f537c001150 con 0x7f5384086980 2019-12-05T06:06:58.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.323+0000 7f53837fe700 1 -- 172.21.15.16:0/2321255618 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5370002e80 con 0x7f5384086980 2019-12-05T06:06:58.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.323+0000 7f53837fe700 1 -- 172.21.15.16:0/2321255618 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2921327335 0 0) 0x7f5374001400 con 0x7f5384086d70 2019-12-05T06:06:58.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.323+0000 7f53837fe700 1 -- 172.21.15.16:0/2321255618 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5370004170 con 0x7f5384086d70 2019-12-05T06:06:58.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.323+0000 7f53896a7700 1 -- 172.21.15.16:0/2321255618 >> v1:172.21.15.76:6789/0 conn(0x7f5384086d70 legacy=0x7f5384064310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.323+0000 7f53896a7700 1 -- 172.21.15.16:0/2321255618 >> v1:172.21.15.16:6789/0 conn(0x7f5384086980 legacy=0x7f538408a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.323+0000 7f53896a7700 1 -- 172.21.15.16:0/2321255618 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5384063a10 con 0x7f538408aeb0 2019-12-05T06:06:58.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.323+0000 7f53896a7700 1 --2- 172.21.15.16:0/2321255618 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 0x7f5384064b10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f536c00af80 tx=0x7f536c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.323+0000 7f53896a7700 1 --2- 172.21.15.16:0/2321255618 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 0x7f5384064b10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f536c00af80 tx=0x7f536c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.323+0000 7f53896a7700 1 --2- 172.21.15.16:0/2321255618 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 0x7f5384064b10 secure :-1 s=READY pgs=43 cs=0 l=1 rx=0x7f536c00af80 tx=0x7f536c00b790).ready entity=mon.2 client_cookie=1778143bbb10e712 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:58.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.324+0000 7f53896a7700 1 --2- 172.21.15.16:0/2321255618 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 0x7f5384064b10 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f536c00af80 tx=0x7f536c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.324+0000 7f53896a7700 1 --2- 172.21.15.16:0/2321255618 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 0x7f5384064b10 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f536c00af80 tx=0x7f536c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.324+0000 7f53837fe700 1 -- 172.21.15.16:0/2321255618 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f536c007240 con 0x7f538408aeb0 2019-12-05T06:06:58.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.324+0000 7f53896a7700 1 --2- 172.21.15.16:0/2321255618 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 0x7f5384064b10 secure :-1 s=THROTTLE_DONE pgs=43 cs=0 l=1 rx=0x7f536c00af80 tx=0x7f536c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.324+0000 7f53837fe700 1 -- 172.21.15.16:0/2321255618 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f536c000c30 con 0x7f538408aeb0 2019-12-05T06:06:58.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.324+0000 7f53837fe700 1 -- 172.21.15.16:0/2321255618 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f536c009f10 con 0x7f538408aeb0 2019-12-05T06:06:58.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.324+0000 7f538a909700 1 -- 172.21.15.16:0/2321255618 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 msgr2=0x7f5384064b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.324+0000 7f538a909700 1 --2- 172.21.15.16:0/2321255618 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 0x7f5384064b10 secure :-1 s=READY pgs=43 cs=0 l=1 rx=0x7f536c00af80 tx=0x7f536c00b790).stop 2019-12-05T06:06:58.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.326+0000 7f538a909700 1 -- 172.21.15.16:0/2321255618 shutdown_connections 2019-12-05T06:06:58.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.326+0000 7f538a909700 1 --2- 172.21.15.16:0/2321255618 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f538408aeb0 0x7f5384064b10 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.326+0000 7f538a909700 1 -- 172.21.15.16:0/2321255618 shutdown_connections 2019-12-05T06:06:58.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.326+0000 7f538a909700 1 -- 172.21.15.16:0/2321255618 wait complete. 2019-12-05T06:06:58.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.326+0000 7f538a909700 1 -- 172.21.15.16:0/2321255618 >> 172.21.15.16:0/2321255618 conn(0x7f538405f5a0 msgr2=0x7f538409cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:58.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.331+0000 7f538a909700 1 Processor -- start 2019-12-05T06:06:58.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.331+0000 7f538a909700 1 -- start start 2019-12-05T06:06:58.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.332+0000 7f538a909700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:58.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.332+0000 7f538a909700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5384130cc0 con 0x7f5384087680 2019-12-05T06:06:58.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.332+0000 7f538a909700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53841310f0 con 0x7f53840972f0 2019-12-05T06:06:58.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.332+0000 7f538a909700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5384093880 con 0x7f5384088270 2019-12-05T06:06:58.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.333+0000 7f5388ea6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5384087680 0x7f5384087a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59948/0 (socket says 172.21.15.16:59948) 2019-12-05T06:06:58.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.333+0000 7f5388ea6700 1 -- 172.21.15.16:0/1770090091 learned_addr learned my addr 172.21.15.16:0/1770090091 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:58.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.333+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.333+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214215749 0 0) 0x7f537c001170 con 0x7f5384087680 2019-12-05T06:06:58.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.333+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5368002e80 con 0x7f5384087680 2019-12-05T06:06:58.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.333+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682758743 0 0) 0x7f536c00adf0 con 0x7f53840972f0 2019-12-05T06:06:58.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.333+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53680030b0 con 0x7f53840972f0 2019-12-05T06:06:58.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f5383fff700 1 -- 172.21.15.16:0/1770090091 >> v1:172.21.15.76:6789/0 conn(0x7f53840972f0 legacy=0x7f5384096860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f5383fff700 1 -- 172.21.15.16:0/1770090091 >> v1:172.21.15.16:6789/0 conn(0x7f5384087680 legacy=0x7f5384087a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f5383fff700 1 -- 172.21.15.16:0/1770090091 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5384095a20 con 0x7f5384088270 2019-12-05T06:06:58.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=READY pgs=44 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:58.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=THROTTLE_DONE pgs=44 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=THROTTLE_DONE pgs=44 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5374007590 con 0x7f5384088270 2019-12-05T06:06:58.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=THROTTLE_DONE pgs=44 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5374009d00 con 0x7f5384088270 2019-12-05T06:06:58.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.334+0000 7f538a909700 1 -- 172.21.15.16:0/1770090091 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5384095a20 con 0x7f5384088270 2019-12-05T06:06:58.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.335+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f537400a410 con 0x7f5384088270 2019-12-05T06:06:58.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.335+0000 7f538a909700 1 -- 172.21.15.16:0/1770090091 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5384095a20 con 0x7f5384088270 2019-12-05T06:06:58.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.336+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=THROTTLE_DONE pgs=44 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.337+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5374009ef0 con 0x7f5384088270 2019-12-05T06:06:58.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.337+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=THROTTLE_DONE pgs=44 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.337+0000 7f53817fa700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5368023650 0x7f5368025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:58.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.337+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f5374046160 con 0x7f5384088270 2019-12-05T06:06:58.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.337+0000 7f5388ea6700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5368023650 0x7f5368025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.338+0000 7f5388ea6700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5368023650 0x7f5368025ae0 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:58.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.352+0000 7f538a909700 1 -- 172.21.15.16:0/1770090091 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5384095a20 con 0x7f5384088270 2019-12-05T06:06:58.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.360+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=THROTTLE_DONE pgs=44 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.360+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5374000e80 con 0x7f5384088270 2019-12-05T06:06:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.607+0000 7f538a909700 1 -- 172.21.15.16:0/1770090091 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f5384095a20 con 0x7f5384088270 2019-12-05T06:06:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.608+0000 7f5383fff700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=THROTTLE_DONE pgs=44 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.608+0000 7f53817fa700 1 -- 172.21.15.16:0/1770090091 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5374009ef0 con 0x7f5384088270 2019-12-05T06:06:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.611+0000 7f538a909700 1 -- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5368023650 msgr2=0x7f5368025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.611+0000 7f538a909700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5368023650 0x7f5368025ae0 crc :-1 s=READY pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.611+0000 7f538a909700 1 -- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 msgr2=0x7f5384096320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.611+0000 7f538a909700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 secure :-1 s=READY pgs=44 cs=0 l=1 rx=0x7f5374010720 tx=0x7f5374010ac0).stop 2019-12-05T06:06:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.611+0000 7f538a909700 1 -- 172.21.15.16:0/1770090091 shutdown_connections 2019-12-05T06:06:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.611+0000 7f538a909700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5368023650 0x7f5368025ae0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.611+0000 7f538a909700 1 --2- 172.21.15.16:0/1770090091 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5384088270 0x7f5384096320 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.611+0000 7f538a909700 1 -- 172.21.15.16:0/1770090091 shutdown_connections 2019-12-05T06:06:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.612+0000 7f538a909700 1 -- 172.21.15.16:0/1770090091 wait complete. 2019-12-05T06:06:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.612+0000 7f538a909700 1 -- 172.21.15.16:0/1770090091 >> 172.21.15.16:0/1770090091 conn(0x7f538408db10 msgr2=0x7f538408a820 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:58.623 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:06:58.632 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T06:06:58.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.856+0000 7fc09f682700 1 Processor -- start 2019-12-05T06:06:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.856+0000 7fc09f682700 1 -- start start 2019-12-05T06:06:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.858+0000 7fc09f682700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc098086980 0x7fc098086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:58.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.858+0000 7fc09f682700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc098064310 0x7fc098064770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.858+0000 7fc09f682700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc098061a10 con 0x7fc09808a6b0 2019-12-05T06:06:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.858+0000 7fc09f682700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc09808b340 con 0x7fc098086980 2019-12-05T06:06:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.858+0000 7fc09f682700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc098060bc0 con 0x7fc098064310 2019-12-05T06:06:58.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.858+0000 7fc09dc1f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc098086980 0x7fc098086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.858+0000 7fc09dc1f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc098086980 0x7fc098086d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46556/0 (socket says 172.21.15.16:46556) 2019-12-05T06:06:58.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.858+0000 7fc09dc1f700 1 -- 172.21.15.16:0/2686014162 learned_addr learned my addr 172.21.15.16:0/2686014162 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:06:58.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.858+0000 7fc09e420700 1 --2- 172.21.15.16:0/2686014162 >> v2:172.21.15.76:3300/0 conn(0x7fc098064310 0x7fc098064770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09dc1f700 1 -- 172.21.15.16:0/2686014162 >> v2:172.21.15.76:3300/0 conn(0x7fc098064310 msgr2=0x7fc098064770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:58.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/2686014162 >> v2:172.21.15.76:3300/0 conn(0x7fc098064310 0x7fc098064770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09dc1f700 1 -- 172.21.15.16:0/2686014162 >> v1:172.21.15.76:6789/0 conn(0x7fc09808a6b0 legacy=0x7fc09808ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09dc1f700 1 -- 172.21.15.16:0/2686014162 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc098063a10 con 0x7fc098086980 2019-12-05T06:06:58.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/2686014162 >> v2:172.21.15.16:3300/0 conn(0x7fc098086980 0x7fc098086d70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc09000fa40 tx=0x7fc09000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/2686014162 >> v2:172.21.15.16:3300/0 conn(0x7fc098086980 0x7fc098086d70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc09000fa40 tx=0x7fc09000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/2686014162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc098086980 0x7fc098086d70 secure :-1 s=READY pgs=161 cs=0 l=1 rx=0x7fc09000fa40 tx=0x7fc09000fde0).ready entity=mon.0 client_cookie=969cbaad3cd724b0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/2686014162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc098086980 0x7fc098086d70 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7fc09000fa40 tx=0x7fc09000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/2686014162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc098086980 0x7fc098086d70 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7fc09000fa40 tx=0x7fc09000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/2686014162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc098086980 0x7fc098086d70 secure :-1 s=THROTTLE_DONE pgs=161 cs=0 l=1 rx=0x7fc09000fa40 tx=0x7fc09000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09cc1d700 1 -- 172.21.15.16:0/2686014162 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc090000ce0 con 0x7fc098086980 2019-12-05T06:06:58.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09cc1d700 1 -- 172.21.15.16:0/2686014162 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc090001010 con 0x7fc098086980 2019-12-05T06:06:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09cc1d700 1 -- 172.21.15.16:0/2686014162 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc090001760 con 0x7fc098086980 2019-12-05T06:06:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09f682700 1 -- 172.21.15.16:0/2686014162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc098086980 msgr2=0x7fc098086d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.859+0000 7fc09f682700 1 --2- 172.21.15.16:0/2686014162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc098086980 0x7fc098086d70 secure :-1 s=READY pgs=161 cs=0 l=1 rx=0x7fc09000fa40 tx=0x7fc09000fde0).stop 2019-12-05T06:06:58.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.861+0000 7fc09f682700 1 -- 172.21.15.16:0/2686014162 shutdown_connections 2019-12-05T06:06:58.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.861+0000 7fc09f682700 1 --2- 172.21.15.16:0/2686014162 >> v2:172.21.15.76:3300/0 conn(0x7fc098064310 0x7fc098064770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:58.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.861+0000 7fc09f682700 1 --2- 172.21.15.16:0/2686014162 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc098086980 0x7fc098086d70 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.861+0000 7fc09f682700 1 -- 172.21.15.16:0/2686014162 shutdown_connections 2019-12-05T06:06:58.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.861+0000 7fc09f682700 1 -- 172.21.15.16:0/2686014162 wait complete. 2019-12-05T06:06:58.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.861+0000 7fc09f682700 1 -- 172.21.15.16:0/2686014162 >> 172.21.15.16:0/2686014162 conn(0x7fc09805f5a0 msgr2=0x7fc09809cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:58.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.865+0000 7fc09f682700 1 Processor -- start 2019-12-05T06:06:58.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.865+0000 7fc09f682700 1 -- start start 2019-12-05T06:06:58.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.866+0000 7fc09f682700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc098089e00 0x7fc098098ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:58.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.866+0000 7fc09f682700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc098156c60 0x7fc098159060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:58.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.866+0000 7fc09f682700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc09808a370 con 0x7fc098099aa0 2019-12-05T06:06:58.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.866+0000 7fc09f682700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0980950c0 con 0x7fc098089e00 2019-12-05T06:06:58.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.866+0000 7fc09f682700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc098096c50 con 0x7fc098156c60 2019-12-05T06:06:58.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.866+0000 7fc09d41e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc098099aa0 0x7fc098099010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59964/0 (socket says 172.21.15.16:59964) 2019-12-05T06:06:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.866+0000 7fc09d41e700 1 -- 172.21.15.16:0/67895295 learned_addr learned my addr 172.21.15.16:0/67895295 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.867+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> v2:172.21.15.76:3300/0 conn(0x7fc098089e00 0x7fc098098ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.867+0000 7fc09e420700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc098156c60 0x7fc098159060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.867+0000 7fc0967fc700 1 -- 172.21.15.16:0/67895295 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4253135929 0 0) 0x7fc08c0009b0 con 0x7fc098099aa0 2019-12-05T06:06:58.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.867+0000 7fc0967fc700 1 -- 172.21.15.16:0/67895295 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc078002e80 con 0x7fc098099aa0 2019-12-05T06:06:58.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.867+0000 7fc09dc1f700 1 -- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc098156c60 msgr2=0x7fc098159060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.867+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc098156c60 0x7fc098159060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:58.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.867+0000 7fc09dc1f700 1 -- 172.21.15.16:0/67895295 >> v1:172.21.15.16:6789/0 conn(0x7fc098099aa0 legacy=0x7fc098099010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:58.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.867+0000 7fc09dc1f700 1 -- 172.21.15.16:0/67895295 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0980981d0 con 0x7fc098089e00 2019-12-05T06:06:58.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.867+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> v2:172.21.15.76:3300/0 conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.867+0000 7fc09e420700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc098156c60 0x7fc098159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:58.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.868+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> v2:172.21.15.76:3300/0 conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.868+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=READY pgs=134 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:58.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.868+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.868+0000 7fc0967fc700 1 -- 172.21.15.16:0/67895295 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc090000cc0 con 0x7fc098089e00 2019-12-05T06:06:58.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.868+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.868+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.868+0000 7fc0967fc700 1 -- 172.21.15.16:0/67895295 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0900012d0 con 0x7fc098089e00 2019-12-05T06:06:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.868+0000 7fc0967fc700 1 -- 172.21.15.16:0/67895295 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0900019e0 con 0x7fc098089e00 2019-12-05T06:06:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.868+0000 7fc09f682700 1 -- 172.21.15.16:0/67895295 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0980981d0 con 0x7fc098089e00 2019-12-05T06:06:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.868+0000 7fc09f682700 1 -- 172.21.15.16:0/67895295 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0980981d0 con 0x7fc098089e00 2019-12-05T06:06:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.869+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.870+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.870+0000 7fc0967fc700 1 -- 172.21.15.16:0/67895295 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc0900014c0 con 0x7fc098089e00 2019-12-05T06:06:58.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.870+0000 7fc0967fc700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc078023670 0x7fc078025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.870+0000 7fc0967fc700 1 -- 172.21.15.16:0/67895295 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fc0900460f0 con 0x7fc098089e00 2019-12-05T06:06:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.870+0000 7fc09d41e700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc078023670 0x7fc078025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.871+0000 7fc09d41e700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc078023670 0x7fc078025ae0 crc :-1 s=READY pgs=88 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:58.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.872+0000 7fc09f682700 1 -- 172.21.15.16:0/67895295 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0980981d0 con 0x7fc098089e00 2019-12-05T06:06:58.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.881+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:58.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:58.881+0000 7fc0967fc700 1 -- 172.21.15.16:0/67895295 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc090000eb0 con 0x7fc098089e00 2019-12-05T06:06:59.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.116+0000 7fc09f682700 1 -- 172.21.15.16:0/67895295 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fc0980981d0 con 0x7fc098089e00 2019-12-05T06:06:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.117+0000 7fc09dc1f700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.117+0000 7fc0967fc700 1 -- 172.21.15.16:0/67895295 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc09002b110 con 0x7fc098089e00 2019-12-05T06:06:59.116 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:06:59.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.121+0000 7fc09f682700 1 -- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc078023670 msgr2=0x7fc078025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:59.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.121+0000 7fc09f682700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc078023670 0x7fc078025ae0 crc :-1 s=READY pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:59.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.121+0000 7fc09f682700 1 -- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 msgr2=0x7fc098098ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:59.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.121+0000 7fc09f682700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 0x7fc098098ad0 secure :-1 s=READY pgs=134 cs=0 l=1 rx=0x7fc090010710 tx=0x7fc090010ab0).stop 2019-12-05T06:06:59.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.122+0000 7fc09f682700 1 -- 172.21.15.16:0/67895295 shutdown_connections 2019-12-05T06:06:59.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.122+0000 7fc09f682700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc078023670 0x7fc078025ae0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:59.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.122+0000 7fc09f682700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc098156c60 0x7fc098159060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:59.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.122+0000 7fc09f682700 1 --2- 172.21.15.16:0/67895295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc098089e00 0x7fc098098ad0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:59.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.122+0000 7fc09f682700 1 -- 172.21.15.16:0/67895295 shutdown_connections 2019-12-05T06:06:59.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.122+0000 7fc09f682700 1 -- 172.21.15.16:0/67895295 wait complete. 2019-12-05T06:06:59.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.122+0000 7fc09f682700 1 -- 172.21.15.16:0/67895295 >> 172.21.15.16:0/67895295 conn(0x7fc09809c100 msgr2=0x7fc09808c6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:59.138 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:06:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.393+0000 7f6567b42700 1 Processor -- start 2019-12-05T06:06:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.393+0000 7f6567b42700 1 -- start start 2019-12-05T06:06:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.395+0000 7f6567b42700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f656008d2c0 0x7f656008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.395+0000 7f6567b42700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f656008fef0 0x7f656008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:06:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.395+0000 7f6567b42700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600958e0 con 0x7f656008c6d0 2019-12-05T06:06:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.395+0000 7f6567b42700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f656008df90 con 0x7f656008fef0 2019-12-05T06:06:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.395+0000 7f6567b42700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65600a3f70 con 0x7f656008d2c0 2019-12-05T06:06:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.395+0000 7f65660df700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f656008c6d0 0x7f656008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59970/0 (socket says 172.21.15.16:59970) 2019-12-05T06:06:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.395+0000 7f65660df700 1 -- 172.21.15.16:0/3949580169 learned_addr learned my addr 172.21.15.16:0/3949580169 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.395+0000 7f65668e0700 1 --2- 172.21.15.16:0/3949580169 >> v2:172.21.15.16:3300/0 conn(0x7f656008fef0 0x7f656008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.395+0000 7f65658de700 1 --2- 172.21.15.16:0/3949580169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f656008d2c0 0x7f656008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.396+0000 7f65650dd700 1 -- 172.21.15.16:0/3949580169 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1172366575 0 0) 0x7f65580014d0 con 0x7f656008c6d0 2019-12-05T06:06:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.396+0000 7f65650dd700 1 -- 172.21.15.16:0/3949580169 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6548002e80 con 0x7f656008c6d0 2019-12-05T06:06:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.396+0000 7f65668e0700 1 -- 172.21.15.16:0/3949580169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f656008d2c0 msgr2=0x7f656008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:06:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.396+0000 7f65668e0700 1 --2- 172.21.15.16:0/3949580169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f656008d2c0 0x7f656008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.396+0000 7f65668e0700 1 -- 172.21.15.16:0/3949580169 >> v1:172.21.15.16:6789/0 conn(0x7f656008c6d0 legacy=0x7f656008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.396+0000 7f65668e0700 1 -- 172.21.15.16:0/3949580169 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656008e620 con 0x7f656008fef0 2019-12-05T06:06:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.396+0000 7f65658de700 1 --2- 172.21.15.16:0/3949580169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f656008d2c0 0x7f656008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:06:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.396+0000 7f65668e0700 1 --2- 172.21.15.16:0/3949580169 >> v2:172.21.15.16:3300/0 conn(0x7f656008fef0 0x7f656008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f655000af50 tx=0x7f655000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.397+0000 7f65668e0700 1 --2- 172.21.15.16:0/3949580169 >> v2:172.21.15.16:3300/0 conn(0x7f656008fef0 0x7f656008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f655000af50 tx=0x7f655000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.397+0000 7f65668e0700 1 --2- 172.21.15.16:0/3949580169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656008fef0 0x7f656008f460 secure :-1 s=READY pgs=164 cs=0 l=1 rx=0x7f655000af50 tx=0x7f655000b760).ready entity=mon.0 client_cookie=636b13c708143abf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.397+0000 7f65668e0700 1 --2- 172.21.15.16:0/3949580169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656008fef0 0x7f656008f460 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7f655000af50 tx=0x7f655000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.397+0000 7f65650dd700 1 -- 172.21.15.16:0/3949580169 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6550009ba0 con 0x7f656008fef0 2019-12-05T06:06:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.397+0000 7f65668e0700 1 --2- 172.21.15.16:0/3949580169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656008fef0 0x7f656008f460 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7f655000af50 tx=0x7f655000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.397+0000 7f65668e0700 1 --2- 172.21.15.16:0/3949580169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656008fef0 0x7f656008f460 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7f655000af50 tx=0x7f655000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.397+0000 7f65650dd700 1 -- 172.21.15.16:0/3949580169 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65500097c0 con 0x7f656008fef0 2019-12-05T06:06:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.397+0000 7f65650dd700 1 -- 172.21.15.16:0/3949580169 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6550000bc0 con 0x7f656008fef0 2019-12-05T06:06:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.397+0000 7f6567b42700 1 -- 172.21.15.16:0/3949580169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656008fef0 msgr2=0x7f656008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.397+0000 7f6567b42700 1 --2- 172.21.15.16:0/3949580169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656008fef0 0x7f656008f460 secure :-1 s=READY pgs=164 cs=0 l=1 rx=0x7f655000af50 tx=0x7f655000b760).stop 2019-12-05T06:06:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.400+0000 7f6567b42700 1 -- 172.21.15.16:0/3949580169 shutdown_connections 2019-12-05T06:06:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.400+0000 7f6567b42700 1 --2- 172.21.15.16:0/3949580169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f656008fef0 0x7f656008f460 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:59.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.400+0000 7f6567b42700 1 --2- 172.21.15.16:0/3949580169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f656008d2c0 0x7f656008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:06:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.400+0000 7f6567b42700 1 -- 172.21.15.16:0/3949580169 shutdown_connections 2019-12-05T06:06:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.400+0000 7f6567b42700 1 -- 172.21.15.16:0/3949580169 wait complete. 2019-12-05T06:06:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.400+0000 7f6567b42700 1 -- 172.21.15.16:0/3949580169 >> 172.21.15.16:0/3949580169 conn(0x7f6560061940 msgr2=0x7f6560096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.406+0000 7f6567b42700 1 Processor -- start 2019-12-05T06:06:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.406+0000 7f6567b42700 1 -- start start 2019-12-05T06:06:59.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.407+0000 7f6567b42700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.408+0000 7f6567b42700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656006e7c0 con 0x7f656009ae80 2019-12-05T06:06:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.408+0000 7f6567b42700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6560133b30 con 0x7f6560099eb0 2019-12-05T06:06:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.408+0000 7f6567b42700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6560134390 con 0x7f6560132630 2019-12-05T06:06:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.408+0000 7f65658de700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f656009ae80 0x7f6560132f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59978/0 (socket says 172.21.15.16:59978) 2019-12-05T06:06:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.408+0000 7f65658de700 1 -- 172.21.15.16:0/3958073357 learned_addr learned my addr 172.21.15.16:0/3958073357 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:06:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.408+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.408+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335674790 0 0) 0x7f655400dd90 con 0x7f656009ae80 2019-12-05T06:06:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.408+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6540002e80 con 0x7f656009ae80 2019-12-05T06:06:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.409+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 823015568 0 0) 0x7f6550000940 con 0x7f6560099eb0 2019-12-05T06:06:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.409+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65400030b0 con 0x7f6560099eb0 2019-12-05T06:06:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.409+0000 7f65660df700 1 -- 172.21.15.16:0/3958073357 >> v1:172.21.15.76:6789/0 conn(0x7f6560099eb0 legacy=0x7f656009a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.409+0000 7f65660df700 1 -- 172.21.15.16:0/3958073357 >> v1:172.21.15.16:6789/0 conn(0x7f656009ae80 legacy=0x7f6560132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.409+0000 7f65660df700 1 -- 172.21.15.16:0/3958073357 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65600995b0 con 0x7f6560132630 2019-12-05T06:06:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.409+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.409+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.409+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=READY pgs=45 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.409+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.410+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f655800f550 con 0x7f6560132630 2019-12-05T06:06:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.410+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.410+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.410+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f655800f0d0 con 0x7f6560132630 2019-12-05T06:06:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.410+0000 7f6567b42700 1 -- 172.21.15.16:0/3958073357 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65600995b0 con 0x7f6560132630 2019-12-05T06:06:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.410+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6558001010 con 0x7f6560132630 2019-12-05T06:06:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.410+0000 7f6567b42700 1 -- 172.21.15.16:0/3958073357 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65600995b0 con 0x7f6560132630 2019-12-05T06:06:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.411+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.412+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f655800f570 con 0x7f6560132630 2019-12-05T06:06:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.412+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.412+0000 7f655effd700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6540023650 0x7f6540025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:06:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.412+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f65580460e0 con 0x7f6560132630 2019-12-05T06:06:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.412+0000 7f65658de700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6540023650 0x7f6540025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:06:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.413+0000 7f65658de700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6540023650 0x7f6540025ae0 crc :-1 s=READY pgs=89 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:06:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.425+0000 7f6567b42700 1 -- 172.21.15.16:0/3958073357 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65600995b0 con 0x7f6560132630 2019-12-05T06:06:59.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.434+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.435+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f655800f360 con 0x7f6560132630 2019-12-05T06:06:59.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.701+0000 7f6567b42700 1 -- 172.21.15.16:0/3958073357 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f65600995b0 con 0x7f6560132630 2019-12-05T06:06:59.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.702+0000 7f65660df700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=THROTTLE_DONE pgs=45 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:06:59.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.702+0000 7f655effd700 1 -- 172.21.15.16:0/3958073357 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f655800f570 con 0x7f6560132630 2019-12-05T06:06:59.703 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2019-12-05T06:06:59.703 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:06:59.703 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:06:45.694298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:06:59.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.707+0000 7f6567b42700 1 -- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6540023650 msgr2=0x7f6540025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.707+0000 7f6567b42700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6540023650 0x7f6540025ae0 crc :-1 s=READY pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.707+0000 7f6567b42700 1 -- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 msgr2=0x7f6560132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:06:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.707+0000 7f6567b42700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 secure :-1 s=READY pgs=45 cs=0 l=1 rx=0x7f6558010700 tx=0x7f6558010aa0).stop 2019-12-05T06:06:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.707+0000 7f6567b42700 1 -- 172.21.15.16:0/3958073357 shutdown_connections 2019-12-05T06:06:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.707+0000 7f6567b42700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6540023650 0x7f6540025ae0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.707+0000 7f6567b42700 1 --2- 172.21.15.16:0/3958073357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6560132630 0x7f6560132a20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:06:59.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.707+0000 7f6567b42700 1 -- 172.21.15.16:0/3958073357 shutdown_connections 2019-12-05T06:06:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.707+0000 7f6567b42700 1 -- 172.21.15.16:0/3958073357 wait complete. 2019-12-05T06:06:59.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:06:59.707+0000 7f6567b42700 1 -- 172.21.15.16:0/3958073357 >> 172.21.15.16:0/3958073357 conn(0x7f65600714f0 msgr2=0x7f656009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:06:59.919 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4341}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 6, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4341': {u'export_targets': [], u'name': u'a', u'incarnation': 5, u'state_seq': 2, u'state': u'up:active', u'gid': 4341, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 3750563081, u'type': u'v2', u'addr': u'172.21.15.16:6832'}, {u'nonce': 3750563081, u'type': u'v1', u'addr': u'172.21.15.16:6833'}]}, u'addr': u'172.21.15.16:6833/3750563081'}}, u'fs_name': u'cephfs', u'created': u'2019-12-05T06:06:40.256556+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2019-12-05T06:06:45.694298+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2019-12-05T06:06:59.919 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2019-12-05T06:06:59.920 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2019-12-05T06:07:00.152 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2019-12-05T06:07:00.153 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "df6db985-de08-463b-af1a-6ee59e0bbbe7", 2019-12-05T06:07:00.154 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2019-12-05T06:07:00.154 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4341, 2019-12-05T06:07:00.154 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2019-12-05T06:07:00.154 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2019-12-05T06:07:00.154 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 15.452284783, 2019-12-05T06:07:00.154 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 6, 2019-12-05T06:07:00.155 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 24, 2019-12-05T06:07:00.155 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 24, 2019-12-05T06:07:00.155 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 16.181394848 2019-12-05T06:07:00.156 INFO:teuthology.orchestra.run.smithi016.stdout:} 2019-12-05T06:07:00.157 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "df6db985-de08-463b-af1a-6ee59e0bbbe7", "whoami": 0, "id": 4341, "want_state": "up:active", "state": "up:active", "rank_uptime": 15.452284783, "mdsmap_epoch": 6, "osdmap_epoch": 24, "osdmap_epoch_barrier": 24, "uptime": 16.181394848 } 2019-12-05T06:07:00.157 INFO:teuthology.run_tasks:Running task mds_thrash... 2019-12-05T06:07:00.270 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:07:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.532+0000 7f03953a9700 1 Processor -- start 2019-12-05T06:07:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.532+0000 7f03953a9700 1 -- start start 2019-12-05T06:07:00.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.534+0000 7f03953a9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f039008fef0 0x7f039008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.534+0000 7f03953a9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03900958e0 con 0x7f039008c6d0 2019-12-05T06:07:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.534+0000 7f03953a9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03900629e0 con 0x7f039008d2c0 2019-12-05T06:07:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.534+0000 7f03953a9700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f039008df90 con 0x7f039008fef0 2019-12-05T06:07:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.535+0000 7f038f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f039008c6d0 0x7f039008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:59984/0 (socket says 172.21.15.16:59984) 2019-12-05T06:07:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.535+0000 7f038f7fe700 1 -- 172.21.15.16:0/688661759 learned_addr learned my addr 172.21.15.16:0/688661759 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:00.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.535+0000 7f038ffff700 1 --2- 172.21.15.16:0/688661759 >> v2:172.21.15.76:3300/0 conn(0x7f039008fef0 0x7f039008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.535+0000 7f038e7fc700 1 -- 172.21.15.16:0/688661759 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 430655767 0 0) 0x7f03880014d0 con 0x7f039008c6d0 2019-12-05T06:07:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.535+0000 7f038e7fc700 1 -- 172.21.15.16:0/688661759 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f037c002e80 con 0x7f039008c6d0 2019-12-05T06:07:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038ffff700 1 -- 172.21.15.16:0/688661759 >> v1:172.21.15.76:6789/0 conn(0x7f039008d2c0 legacy=0x7f039008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038ffff700 1 -- 172.21.15.16:0/688661759 >> v1:172.21.15.16:6789/0 conn(0x7f039008c6d0 legacy=0x7f039008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038ffff700 1 -- 172.21.15.16:0/688661759 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039008e620 con 0x7f039008fef0 2019-12-05T06:07:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038ffff700 1 --2- 172.21.15.16:0/688661759 >> v2:172.21.15.76:3300/0 conn(0x7f039008fef0 0x7f039008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f038000af80 tx=0x7f038000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038ffff700 1 --2- 172.21.15.16:0/688661759 >> v2:172.21.15.76:3300/0 conn(0x7f039008fef0 0x7f039008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f038000af80 tx=0x7f038000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038ffff700 1 --2- 172.21.15.16:0/688661759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f039008fef0 0x7f039008f720 secure :-1 s=READY pgs=137 cs=0 l=1 rx=0x7f038000af80 tx=0x7f038000b790).ready entity=mon.1 client_cookie=1d3bf9f1243d33f3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038ffff700 1 --2- 172.21.15.16:0/688661759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f039008fef0 0x7f039008f720 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f038000af80 tx=0x7f038000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038e7fc700 1 -- 172.21.15.16:0/688661759 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0380007240 con 0x7f039008fef0 2019-12-05T06:07:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038ffff700 1 --2- 172.21.15.16:0/688661759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f039008fef0 0x7f039008f720 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f038000af80 tx=0x7f038000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038ffff700 1 --2- 172.21.15.16:0/688661759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f039008fef0 0x7f039008f720 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f038000af80 tx=0x7f038000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038e7fc700 1 -- 172.21.15.16:0/688661759 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0380000bb0 con 0x7f039008fef0 2019-12-05T06:07:00.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.536+0000 7f038e7fc700 1 -- 172.21.15.16:0/688661759 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0380009e60 con 0x7f039008fef0 2019-12-05T06:07:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.537+0000 7f03953a9700 1 -- 172.21.15.16:0/688661759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f039008fef0 msgr2=0x7f039008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.537+0000 7f03953a9700 1 --2- 172.21.15.16:0/688661759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f039008fef0 0x7f039008f720 secure :-1 s=READY pgs=137 cs=0 l=1 rx=0x7f038000af80 tx=0x7f038000b790).stop 2019-12-05T06:07:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.538+0000 7f03953a9700 1 -- 172.21.15.16:0/688661759 shutdown_connections 2019-12-05T06:07:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.538+0000 7f03953a9700 1 --2- 172.21.15.16:0/688661759 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f039008fef0 0x7f039008f720 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.538+0000 7f03953a9700 1 -- 172.21.15.16:0/688661759 shutdown_connections 2019-12-05T06:07:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.538+0000 7f03953a9700 1 -- 172.21.15.16:0/688661759 wait complete. 2019-12-05T06:07:00.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.538+0000 7f03953a9700 1 -- 172.21.15.16:0/688661759 >> 172.21.15.16:0/688661759 conn(0x7f0390061940 msgr2=0x7f0390096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.543+0000 7f03953a9700 1 Processor -- start 2019-12-05T06:07:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.544+0000 7f03953a9700 1 -- start start 2019-12-05T06:07:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.545+0000 7f03953a9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f039014f630 0x7f039014fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:00.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.545+0000 7f03953a9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f039009e7c0 0x7f039009d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.545+0000 7f03953a9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f039009dd30 0x7f039006dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.545+0000 7f03953a9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f039012cd70 con 0x7f039009dd30 2019-12-05T06:07:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.545+0000 7f03953a9700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0390131c20 con 0x7f039014f630 2019-12-05T06:07:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.545+0000 7f03953a9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03901311f0 con 0x7f039009e7c0 2019-12-05T06:07:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.546+0000 7f038ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f039009dd30 0x7f039006dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.546+0000 7f038ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f039009dd30 0x7f039006dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46592/0 (socket says 172.21.15.16:46592) 2019-12-05T06:07:00.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.546+0000 7f038f7fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f039014f630 0x7f039014fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.546+0000 7f038ffff700 1 -- 172.21.15.16:0/3729191481 learned_addr learned my addr 172.21.15.16:0/3729191481 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.546+0000 7f038effd700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f039009e7c0 0x7f039009d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.546+0000 7f038ffff700 1 -- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f039009e7c0 msgr2=0x7f039009d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.546+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f039009e7c0 0x7f039009d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.546+0000 7f038ffff700 1 -- 172.21.15.16:0/3729191481 >> v2:172.21.15.76:3300/0 conn(0x7f039014f630 msgr2=0x7f039014fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.546+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> v2:172.21.15.76:3300/0 conn(0x7f039014f630 0x7f039014fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f038ffff700 1 -- 172.21.15.16:0/3729191481 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03901322b0 con 0x7f039009dd30 2019-12-05T06:07:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f038f7fe700 1 --2- 172.21.15.16:0/3729191481 >> v2:172.21.15.76:3300/0 conn(0x7f039014f630 0x7f039014fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f038effd700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f039009e7c0 0x7f039009d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> v2:172.21.15.16:3300/0 conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> v2:172.21.15.16:3300/0 conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=READY pgs=167 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f0387fff700 1 -- 172.21.15.16:0/3729191481 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f038000afc0 con 0x7f039009dd30 2019-12-05T06:07:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f0387fff700 1 -- 172.21.15.16:0/3729191481 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03800099b0 con 0x7f039009dd30 2019-12-05T06:07:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f0387fff700 1 -- 172.21.15.16:0/3729191481 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f038000a0c0 con 0x7f039009dd30 2019-12-05T06:07:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f03953a9700 1 -- 172.21.15.16:0/3729191481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03901322b0 con 0x7f039009dd30 2019-12-05T06:07:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.547+0000 7f03953a9700 1 -- 172.21.15.16:0/3729191481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03901322b0 con 0x7f039009dd30 2019-12-05T06:07:00.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.548+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.550+0000 7f0387fff700 1 -- 172.21.15.16:0/3729191481 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f03800095b0 con 0x7f039009dd30 2019-12-05T06:07:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.550+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.550+0000 7f0387fff700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0374023630 0x7f0374025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.550+0000 7f0387fff700 1 -- 172.21.15.16:0/3729191481 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f0380045dc0 con 0x7f039009dd30 2019-12-05T06:07:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.550+0000 7f038f7fe700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0374023630 0x7f0374025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:00.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.551+0000 7f038f7fe700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0374023630 0x7f0374025ac0 crc :-1 s=READY pgs=90 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:00.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.564+0000 7f03953a9700 1 -- 172.21.15.16:0/3729191481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03901322b0 con 0x7f039009dd30 2019-12-05T06:07:00.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.572+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.572+0000 7f0387fff700 1 -- 172.21.15.16:0/3729191481 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f038000aee0 con 0x7f039009dd30 2019-12-05T06:07:00.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.830+0000 7f03953a9700 1 -- 172.21.15.16:0/3729191481 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f03901322b0 con 0x7f039009dd30 2019-12-05T06:07:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.831+0000 7f038ffff700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=THROTTLE_DONE pgs=167 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:00.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.831+0000 7f0387fff700 1 -- 172.21.15.16:0/3729191481 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f03800095b0 con 0x7f039009dd30 2019-12-05T06:07:00.832 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:07:00.832 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:07:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 -- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0374023630 msgr2=0x7f0374025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0374023630 0x7f0374025ac0 crc :-1 s=READY pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 -- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 msgr2=0x7f039006dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 0x7f039006dc50 secure :-1 s=READY pgs=167 cs=0 l=1 rx=0x7f0380010440 tx=0x7f03800107e0).stop 2019-12-05T06:07:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 -- 172.21.15.16:0/3729191481 shutdown_connections 2019-12-05T06:07:00.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0374023630 0x7f0374025ac0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f039009dd30 0x7f039006dc50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 --2- 172.21.15.16:0/3729191481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f039009e7c0 0x7f039009d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 --2- 172.21.15.16:0/3729191481 >> v2:172.21.15.76:3300/0 conn(0x7f039014f630 0x7f039014fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 -- 172.21.15.16:0/3729191481 shutdown_connections 2019-12-05T06:07:00.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.834+0000 7f03953a9700 1 -- 172.21.15.16:0/3729191481 wait complete. 2019-12-05T06:07:00.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:00.835+0000 7f03953a9700 1 -- 172.21.15.16:0/3729191481 >> 172.21.15.16:0/3729191481 conn(0x7f03900a2850 msgr2=0x7f039006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:00.845 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:31.052682+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:34.862662+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:37.644552+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T06:07:00.846 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T06:07:01.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.047+0000 7ff563e70700 1 Processor -- start 2019-12-05T06:07:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.047+0000 7ff563e70700 1 -- start start 2019-12-05T06:07:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.049+0000 7ff563e70700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff55c086980 0x7ff55c086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.049+0000 7ff563e70700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff55c064310 0x7ff55c064770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:01.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.049+0000 7ff563e70700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55c061a10 con 0x7ff55c08a6b0 2019-12-05T06:07:01.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.049+0000 7ff563e70700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55c08b340 con 0x7ff55c064310 2019-12-05T06:07:01.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.049+0000 7ff563e70700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55c060bc0 con 0x7ff55c086980 2019-12-05T06:07:01.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.049+0000 7ff562c0e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff55c064310 0x7ff55c064770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:01.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.049+0000 7ff562c0e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff55c064310 0x7ff55c064770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46598/0 (socket says 172.21.15.16:46598) 2019-12-05T06:07:01.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.049+0000 7ff56240d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff55c086980 0x7ff55c086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.049+0000 7ff562c0e700 1 -- 172.21.15.16:0/943139793 learned_addr learned my addr 172.21.15.16:0/943139793 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.050+0000 7ff562c0e700 1 -- 172.21.15.16:0/943139793 >> v2:172.21.15.76:3300/0 conn(0x7ff55c086980 msgr2=0x7ff55c086d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.050+0000 7ff562c0e700 1 --2- 172.21.15.16:0/943139793 >> v2:172.21.15.76:3300/0 conn(0x7ff55c086980 0x7ff55c086d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.050+0000 7ff562c0e700 1 -- 172.21.15.16:0/943139793 >> v1:172.21.15.76:6789/0 conn(0x7ff55c08a6b0 legacy=0x7ff55c08ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.050+0000 7ff562c0e700 1 -- 172.21.15.16:0/943139793 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c063a10 con 0x7ff55c064310 2019-12-05T06:07:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.050+0000 7ff562c0e700 1 --2- 172.21.15.16:0/943139793 >> v2:172.21.15.16:3300/0 conn(0x7ff55c064310 0x7ff55c064770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff54c00f6b0 tx=0x7ff54c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.050+0000 7ff562c0e700 1 --2- 172.21.15.16:0/943139793 >> v2:172.21.15.16:3300/0 conn(0x7ff55c064310 0x7ff55c064770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff54c00f6b0 tx=0x7ff54c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.050+0000 7ff562c0e700 1 --2- 172.21.15.16:0/943139793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c064310 0x7ff55c064770 secure :-1 s=READY pgs=168 cs=0 l=1 rx=0x7ff54c00f6b0 tx=0x7ff54c00fa50).ready entity=mon.0 client_cookie=c4affe46ea033d0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:01.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.051+0000 7ff562c0e700 1 --2- 172.21.15.16:0/943139793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c064310 0x7ff55c064770 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7ff54c00f6b0 tx=0x7ff54c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.051+0000 7ff562c0e700 1 --2- 172.21.15.16:0/943139793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c064310 0x7ff55c064770 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7ff54c00f6b0 tx=0x7ff54c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.051+0000 7ff562c0e700 1 --2- 172.21.15.16:0/943139793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c064310 0x7ff55c064770 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7ff54c00f6b0 tx=0x7ff54c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.051+0000 7ff56140b700 1 -- 172.21.15.16:0/943139793 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff54c000a70 con 0x7ff55c064310 2019-12-05T06:07:01.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.051+0000 7ff56140b700 1 -- 172.21.15.16:0/943139793 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c0010e0 con 0x7ff55c064310 2019-12-05T06:07:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.051+0000 7ff56140b700 1 -- 172.21.15.16:0/943139793 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff54c0017b0 con 0x7ff55c064310 2019-12-05T06:07:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.051+0000 7ff563e70700 1 -- 172.21.15.16:0/943139793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c064310 msgr2=0x7ff55c064770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.051+0000 7ff563e70700 1 --2- 172.21.15.16:0/943139793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c064310 0x7ff55c064770 secure :-1 s=READY pgs=168 cs=0 l=1 rx=0x7ff54c00f6b0 tx=0x7ff54c00fa50).stop 2019-12-05T06:07:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.052+0000 7ff563e70700 1 -- 172.21.15.16:0/943139793 shutdown_connections 2019-12-05T06:07:01.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.052+0000 7ff563e70700 1 --2- 172.21.15.16:0/943139793 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c064310 0x7ff55c064770 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.052+0000 7ff563e70700 1 --2- 172.21.15.16:0/943139793 >> v2:172.21.15.76:3300/0 conn(0x7ff55c086980 0x7ff55c086d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:01.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.052+0000 7ff563e70700 1 -- 172.21.15.16:0/943139793 shutdown_connections 2019-12-05T06:07:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.052+0000 7ff563e70700 1 -- 172.21.15.16:0/943139793 wait complete. 2019-12-05T06:07:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.052+0000 7ff563e70700 1 -- 172.21.15.16:0/943139793 >> 172.21.15.16:0/943139793 conn(0x7ff55c05f5a0 msgr2=0x7ff55c09cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.057+0000 7ff563e70700 1 Processor -- start 2019-12-05T06:07:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.057+0000 7ff563e70700 1 -- start start 2019-12-05T06:07:01.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.058+0000 7ff563e70700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff55c09b1a0 0x7ff55c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.058+0000 7ff563e70700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff55c198090 0x7ff55c19a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.058+0000 7ff563e70700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff55c12bb00 con 0x7ff55c13af90 2019-12-05T06:07:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.058+0000 7ff563e70700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff55c099a10 con 0x7ff55c198090 2019-12-05T06:07:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.058+0000 7ff563e70700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff55c098fe0 con 0x7ff55c09b1a0 2019-12-05T06:07:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.058+0000 7ff56240d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff55c13af90 0x7ff55c09a9a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60002/0 (socket says 172.21.15.16:60002) 2019-12-05T06:07:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.058+0000 7ff56240d700 1 -- 172.21.15.16:0/1355255492 learned_addr learned my addr 172.21.15.16:0/1355255492 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:01.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.059+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> v2:172.21.15.16:3300/0 conn(0x7ff55c198090 0x7ff55c19a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.059+0000 7ff561c0c700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff55c09b1a0 0x7ff55c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.059+0000 7ff55affd700 1 -- 172.21.15.16:0/1355255492 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328201099 0 0) 0x7ff554001430 con 0x7ff55c13af90 2019-12-05T06:07:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.059+0000 7ff55affd700 1 -- 172.21.15.16:0/1355255492 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff53c002e80 con 0x7ff55c13af90 2019-12-05T06:07:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.059+0000 7ff562c0e700 1 -- 172.21.15.16:0/1355255492 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff55c09b1a0 msgr2=0x7ff55c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.059+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff55c09b1a0 0x7ff55c197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.059+0000 7ff562c0e700 1 -- 172.21.15.16:0/1355255492 >> v1:172.21.15.16:6789/0 conn(0x7ff55c13af90 legacy=0x7ff55c09a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.059+0000 7ff562c0e700 1 -- 172.21.15.16:0/1355255492 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff55c09a0a0 con 0x7ff55c198090 2019-12-05T06:07:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.059+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> v2:172.21.15.16:3300/0 conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.060+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> v2:172.21.15.16:3300/0 conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.060+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=READY pgs=170 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.060+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.060+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.060+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.060+0000 7ff55affd700 1 -- 172.21.15.16:0/1355255492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff54c005930 con 0x7ff55c198090 2019-12-05T06:07:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.060+0000 7ff55affd700 1 -- 172.21.15.16:0/1355255492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff54c005550 con 0x7ff55c198090 2019-12-05T06:07:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.060+0000 7ff563e70700 1 -- 172.21.15.16:0/1355255492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff55c09a0a0 con 0x7ff55c198090 2019-12-05T06:07:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.060+0000 7ff55affd700 1 -- 172.21.15.16:0/1355255492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff54c005f10 con 0x7ff55c198090 2019-12-05T06:07:01.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.060+0000 7ff563e70700 1 -- 172.21.15.16:0/1355255492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff55c09a0a0 con 0x7ff55c198090 2019-12-05T06:07:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.061+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.062+0000 7ff55affd700 1 -- 172.21.15.16:0/1355255492 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff54c005740 con 0x7ff55c198090 2019-12-05T06:07:01.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.062+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.062+0000 7ff55affd700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff53c023670 0x7ff53c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:01.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.062+0000 7ff55affd700 1 -- 172.21.15.16:0/1355255492 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff54c045d30 con 0x7ff55c198090 2019-12-05T06:07:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.062+0000 7ff56240d700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff53c023670 0x7ff53c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.063+0000 7ff56240d700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff53c023670 0x7ff53c025ae0 crc :-1 s=READY pgs=91 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.079+0000 7ff563e70700 1 -- 172.21.15.16:0/1355255492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff55c09a0a0 con 0x7ff55c198090 2019-12-05T06:07:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.084+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.084+0000 7ff55affd700 1 -- 172.21.15.16:0/1355255492 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff54c005340 con 0x7ff55c198090 2019-12-05T06:07:01.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.313+0000 7ff563e70700 1 -- 172.21.15.16:0/1355255492 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ff55c09a0a0 con 0x7ff55c198090 2019-12-05T06:07:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.314+0000 7ff562c0e700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.314+0000 7ff55affd700 1 -- 172.21.15.16:0/1355255492 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7ff54c005700 con 0x7ff55c198090 2019-12-05T06:07:01.313 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:07:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 -- 172.21.15.16:0/1355255492 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff53c023670 msgr2=0x7ff53c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff53c023670 0x7ff53c025ae0 crc :-1 s=READY pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 -- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 msgr2=0x7ff55c19a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 0x7ff55c19a490 secure :-1 s=READY pgs=170 cs=0 l=1 rx=0x7ff54c0103b0 tx=0x7ff54c010750).stop 2019-12-05T06:07:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 -- 172.21.15.16:0/1355255492 shutdown_connections 2019-12-05T06:07:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff53c023670 0x7ff53c025ae0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff55c198090 0x7ff55c19a490 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 --2- 172.21.15.16:0/1355255492 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff55c09b1a0 0x7ff55c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 -- 172.21.15.16:0/1355255492 shutdown_connections 2019-12-05T06:07:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 -- 172.21.15.16:0/1355255492 wait complete. 2019-12-05T06:07:01.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.317+0000 7ff563e70700 1 -- 172.21.15.16:0/1355255492 >> 172.21.15.16:0/1355255492 conn(0x7ff55c08b790 msgr2=0x7ff55c08a210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:01.326 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T06:07:01.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.546+0000 7f22277db700 1 Processor -- start 2019-12-05T06:07:01.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.546+0000 7f22277db700 1 -- start start 2019-12-05T06:07:01.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.548+0000 7f22277db700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2220086d70 0x7f222008aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:01.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.548+0000 7f22277db700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2220064310 0x7f2220064760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:01.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.548+0000 7f22277db700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2220061a10 con 0x7f2220086980 2019-12-05T06:07:01.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.548+0000 7f22277db700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2220060bc0 con 0x7f2220086d70 2019-12-05T06:07:01.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.548+0000 7f22277db700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2220060620 con 0x7f2220064310 2019-12-05T06:07:01.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.548+0000 7f2225d78700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2220086980 0x7f222008a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60010/0 (socket says 172.21.15.16:60010) 2019-12-05T06:07:01.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.548+0000 7f2225d78700 1 -- 172.21.15.16:0/3669197546 learned_addr learned my addr 172.21.15.16:0/3669197546 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:01.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.548+0000 7f2225577700 1 --2- 172.21.15.16:0/3669197546 >> v2:172.21.15.76:3300/0 conn(0x7f2220086d70 0x7f222008aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:01.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.549+0000 7f2226579700 1 --2- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2220064310 0x7f2220064760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:01.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.549+0000 7f2224d76700 1 -- 172.21.15.16:0/3669197546 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891451928 0 0) 0x7f22180014d0 con 0x7f2220086980 2019-12-05T06:07:01.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.549+0000 7f2224d76700 1 -- 172.21.15.16:0/3669197546 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2208002e80 con 0x7f2220086980 2019-12-05T06:07:01.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.549+0000 7f2225577700 1 -- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2220064310 msgr2=0x7f2220064760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:01.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.549+0000 7f2225577700 1 --2- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2220064310 0x7f2220064760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.549+0000 7f2225577700 1 -- 172.21.15.16:0/3669197546 >> v1:172.21.15.16:6789/0 conn(0x7f2220086980 legacy=0x7f222008a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.549+0000 7f2225577700 1 -- 172.21.15.16:0/3669197546 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2220063a10 con 0x7f2220086d70 2019-12-05T06:07:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.549+0000 7f2225577700 1 --2- 172.21.15.16:0/3669197546 >> v2:172.21.15.76:3300/0 conn(0x7f2220086d70 0x7f222008aeb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f221400af50 tx=0x7f221400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.550+0000 7f2225577700 1 --2- 172.21.15.16:0/3669197546 >> v2:172.21.15.76:3300/0 conn(0x7f2220086d70 0x7f222008aeb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f221400af50 tx=0x7f221400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.550+0000 7f2225577700 1 --2- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2220086d70 0x7f222008aeb0 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7f221400af50 tx=0x7f221400b250).ready entity=mon.1 client_cookie=98775956a539c016 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.550+0000 7f2225577700 1 --2- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2220086d70 0x7f222008aeb0 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f221400af50 tx=0x7f221400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.550+0000 7f2224d76700 1 -- 172.21.15.16:0/3669197546 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2214007210 con 0x7f2220086d70 2019-12-05T06:07:01.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.550+0000 7f2225577700 1 --2- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2220086d70 0x7f222008aeb0 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f221400af50 tx=0x7f221400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.550+0000 7f2225577700 1 --2- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2220086d70 0x7f222008aeb0 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7f221400af50 tx=0x7f221400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.550+0000 7f2224d76700 1 -- 172.21.15.16:0/3669197546 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22140099b0 con 0x7f2220086d70 2019-12-05T06:07:01.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.550+0000 7f2224d76700 1 -- 172.21.15.16:0/3669197546 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f221400a080 con 0x7f2220086d70 2019-12-05T06:07:01.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.550+0000 7f22277db700 1 -- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2220086d70 msgr2=0x7f222008aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.550+0000 7f22277db700 1 --2- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2220086d70 0x7f222008aeb0 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7f221400af50 tx=0x7f221400b250).stop 2019-12-05T06:07:01.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.551+0000 7f22277db700 1 -- 172.21.15.16:0/3669197546 shutdown_connections 2019-12-05T06:07:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.551+0000 7f22277db700 1 --2- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2220086d70 0x7f222008aeb0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.551+0000 7f22277db700 1 --2- 172.21.15.16:0/3669197546 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2220064310 0x7f2220064760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:01.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.552+0000 7f22277db700 1 -- 172.21.15.16:0/3669197546 shutdown_connections 2019-12-05T06:07:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.552+0000 7f22277db700 1 -- 172.21.15.16:0/3669197546 wait complete. 2019-12-05T06:07:01.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.552+0000 7f22277db700 1 -- 172.21.15.16:0/3669197546 >> 172.21.15.16:0/3669197546 conn(0x7f222005f5a0 msgr2=0x7f222009cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.556+0000 7f22277db700 1 Processor -- start 2019-12-05T06:07:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.557+0000 7f22277db700 1 -- start start 2019-12-05T06:07:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.558+0000 7f22277db700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f222014c2c0 0x7f222014c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.558+0000 7f22277db700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22200973c0 0x7f22200964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.558+0000 7f22277db700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2220099de0 con 0x7f22200969e0 2019-12-05T06:07:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.558+0000 7f22277db700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222009b880 con 0x7f22200973c0 2019-12-05T06:07:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.558+0000 7f22277db700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f222009ae50 con 0x7f222014c2c0 2019-12-05T06:07:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.558+0000 7f2225577700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22200973c0 0x7f22200964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.558+0000 7f2226579700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f22200969e0 0x7f2220094440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60020/0 (socket says 172.21.15.16:60020) 2019-12-05T06:07:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.558+0000 7f2226579700 1 -- 172.21.15.16:0/1831524590 learned_addr learned my addr 172.21.15.16:0/1831524590 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.558+0000 7f2225d78700 1 --2- 172.21.15.16:0/1831524590 >> v2:172.21.15.76:3300/0 conn(0x7f222014c2c0 0x7f222014c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f2225577700 1 -- 172.21.15.16:0/1831524590 >> v2:172.21.15.76:3300/0 conn(0x7f222014c2c0 msgr2=0x7f222014c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> v2:172.21.15.76:3300/0 conn(0x7f222014c2c0 0x7f222014c6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f2225577700 1 -- 172.21.15.16:0/1831524590 >> v1:172.21.15.16:6789/0 conn(0x7f22200969e0 legacy=0x7f2220094440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f2225577700 1 -- 172.21.15.16:0/1831524590 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222009bf10 con 0x7f22200973c0 2019-12-05T06:07:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f221e7fc700 1 -- 172.21.15.16:0/1831524590 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2880840039 0 0) 0x7f2210002180 con 0x7f22200969e0 2019-12-05T06:07:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> v2:172.21.15.16:3300/0 conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> v2:172.21.15.16:3300/0 conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f221e7fc700 1 -- 172.21.15.16:0/1831524590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2214000940 con 0x7f22200973c0 2019-12-05T06:07:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.559+0000 7f221e7fc700 1 -- 172.21.15.16:0/1831524590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f221400a9c0 con 0x7f22200973c0 2019-12-05T06:07:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.560+0000 7f221e7fc700 1 -- 172.21.15.16:0/1831524590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2214009e00 con 0x7f22200973c0 2019-12-05T06:07:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.560+0000 7f22277db700 1 -- 172.21.15.16:0/1831524590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f222009bf10 con 0x7f22200973c0 2019-12-05T06:07:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.560+0000 7f22277db700 1 -- 172.21.15.16:0/1831524590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f222009bf10 con 0x7f22200973c0 2019-12-05T06:07:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.560+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.562+0000 7f221e7fc700 1 -- 172.21.15.16:0/1831524590 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2214009790 con 0x7f22200973c0 2019-12-05T06:07:01.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.562+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.562+0000 7f221e7fc700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2200023630 0x7f2200025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:01.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.562+0000 7f221e7fc700 1 -- 172.21.15.16:0/1831524590 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f2214045de0 con 0x7f22200973c0 2019-12-05T06:07:01.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.562+0000 7f2225d78700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2200023630 0x7f2200025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:01.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.563+0000 7f2225d78700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2200023630 0x7f2200025ac0 crc :-1 s=READY pgs=92 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:01.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.578+0000 7f22277db700 1 -- 172.21.15.16:0/1831524590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f222009bf10 con 0x7f22200973c0 2019-12-05T06:07:01.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.586+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.586+0000 7f221e7fc700 1 -- 172.21.15.16:0/1831524590 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2214009580 con 0x7f22200973c0 2019-12-05T06:07:01.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.840+0000 7f22277db700 1 -- 172.21.15.16:0/1831524590 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f222009bf10 con 0x7f22200973c0 2019-12-05T06:07:01.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.840+0000 7f2225577700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:01.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.840+0000 7f221e7fc700 1 -- 172.21.15.16:0/1831524590 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f2214009790 con 0x7f22200973c0 2019-12-05T06:07:01.840 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:07:01.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 -- 172.21.15.16:0/1831524590 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2200023630 msgr2=0x7f2200025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2200023630 0x7f2200025ac0 crc :-1 s=READY pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 -- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 msgr2=0x7f22200964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:01.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 0x7f22200964a0 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7f22140103c0 tx=0x7f22140101f0).stop 2019-12-05T06:07:01.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 -- 172.21.15.16:0/1831524590 shutdown_connections 2019-12-05T06:07:01.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2200023630 0x7f2200025ac0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 --2- 172.21.15.16:0/1831524590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22200973c0 0x7f22200964a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 --2- 172.21.15.16:0/1831524590 >> v2:172.21.15.76:3300/0 conn(0x7f222014c2c0 0x7f222014c6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:01.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 -- 172.21.15.16:0/1831524590 shutdown_connections 2019-12-05T06:07:01.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 -- 172.21.15.16:0/1831524590 wait complete. 2019-12-05T06:07:01.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:01.843+0000 7f22277db700 1 -- 172.21.15.16:0/1831524590 >> 172.21.15.16:0/1831524590 conn(0x7f2220077e10 msgr2=0x7f222008b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:01.856 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T06:07:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.061+0000 7ff3ab1f8700 1 Processor -- start 2019-12-05T06:07:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.062+0000 7ff3ab1f8700 1 -- start start 2019-12-05T06:07:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3ab1f8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff3a4086d70 0x7ff3a408aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3ab1f8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3ab1f8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a4061a10 con 0x7ff3a4086980 2019-12-05T06:07:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3ab1f8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a4060bc0 con 0x7ff3a4086d70 2019-12-05T06:07:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3ab1f8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a4060620 con 0x7ff3a4064310 2019-12-05T06:07:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3a9f96700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3a8f94700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff3a4086d70 0x7ff3a408aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3a9f96700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:50616/0 (socket says 172.21.15.16:50616) 2019-12-05T06:07:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3a8f94700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff3a4086d70 0x7ff3a408aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:60950/0 (socket says 172.21.15.16:60950) 2019-12-05T06:07:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3a9f96700 1 -- 172.21.15.16:0/3958601896 learned_addr learned my addr 172.21.15.16:0/3958601896 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.063+0000 7ff3a9795700 1 --1- 172.21.15.16:0/3958601896 >> v1:172.21.15.76:6789/0 conn(0x7ff3a4086980 0x7ff3a408a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53094/0 (socket says 172.21.15.16:53094) 2019-12-05T06:07:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.064+0000 7ff3a9f96700 1 -- 172.21.15.16:0/3958601896 >> v2:172.21.15.76:3300/0 conn(0x7ff3a4086d70 msgr2=0x7ff3a408aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.064+0000 7ff3a9f96700 1 --2- 172.21.15.16:0/3958601896 >> v2:172.21.15.76:3300/0 conn(0x7ff3a4086d70 0x7ff3a408aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.064+0000 7ff3a9f96700 1 -- 172.21.15.16:0/3958601896 >> v1:172.21.15.76:6789/0 conn(0x7ff3a4086980 legacy=0x7ff3a408a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.064+0000 7ff3a9f96700 1 -- 172.21.15.16:0/3958601896 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a4063a10 con 0x7ff3a4064310 2019-12-05T06:07:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.064+0000 7ff3a8f94700 1 --2- 172.21.15.16:0/3958601896 >> v2:172.21.15.76:3300/0 conn(0x7ff3a4086d70 0x7ff3a408aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.064+0000 7ff3a9f96700 1 --2- 172.21.15.16:0/3958601896 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff39400f760 tx=0x7ff39400fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.064+0000 7ff3a9f96700 1 --2- 172.21.15.16:0/3958601896 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff39400f760 tx=0x7ff39400fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.064+0000 7ff3a9f96700 1 --2- 172.21.15.16:0/3958601896 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 secure :-1 s=READY pgs=46 cs=0 l=1 rx=0x7ff39400f760 tx=0x7ff39400fb00).ready entity=mon.2 client_cookie=b1410a9f853f739d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:02.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.065+0000 7ff3a9f96700 1 --2- 172.21.15.16:0/3958601896 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7ff39400f760 tx=0x7ff39400fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.065+0000 7ff3a9f96700 1 --2- 172.21.15.16:0/3958601896 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7ff39400f760 tx=0x7ff39400fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.065+0000 7ff3a9f96700 1 --2- 172.21.15.16:0/3958601896 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 secure :-1 s=THROTTLE_DONE pgs=46 cs=0 l=1 rx=0x7ff39400f760 tx=0x7ff39400fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.065+0000 7ff3a3fff700 1 -- 172.21.15.16:0/3958601896 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff394000a70 con 0x7ff3a4064310 2019-12-05T06:07:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.065+0000 7ff3a3fff700 1 -- 172.21.15.16:0/3958601896 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff394000e70 con 0x7ff3a4064310 2019-12-05T06:07:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.065+0000 7ff3a3fff700 1 -- 172.21.15.16:0/3958601896 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff394001500 con 0x7ff3a4064310 2019-12-05T06:07:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.065+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3958601896 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 msgr2=0x7ff3a4064760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.065+0000 7ff3ab1f8700 1 --2- 172.21.15.16:0/3958601896 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 secure :-1 s=READY pgs=46 cs=0 l=1 rx=0x7ff39400f760 tx=0x7ff39400fb00).stop 2019-12-05T06:07:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.066+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3958601896 shutdown_connections 2019-12-05T06:07:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.066+0000 7ff3ab1f8700 1 --2- 172.21.15.16:0/3958601896 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4064310 0x7ff3a4064760 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.066+0000 7ff3ab1f8700 1 --2- 172.21.15.16:0/3958601896 >> v2:172.21.15.76:3300/0 conn(0x7ff3a4086d70 0x7ff3a408aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.067+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3958601896 shutdown_connections 2019-12-05T06:07:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.067+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3958601896 wait complete. 2019-12-05T06:07:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.067+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3958601896 >> 172.21.15.16:0/3958601896 conn(0x7ff3a405f5a0 msgr2=0x7ff3a409cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.071+0000 7ff3ab1f8700 1 Processor -- start 2019-12-05T06:07:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.071+0000 7ff3ab1f8700 1 -- start start 2019-12-05T06:07:02.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.072+0000 7ff3ab1f8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff3a4073d80 0x7ff3a4095460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.072+0000 7ff3ab1f8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4156c80 0x7ff3a4159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.072+0000 7ff3ab1f8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3a4074c00 con 0x7ff3a4099990 2019-12-05T06:07:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.072+0000 7ff3ab1f8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3a4075e90 con 0x7ff3a4073d80 2019-12-05T06:07:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.072+0000 7ff3ab1f8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a4076160 con 0x7ff3a4156c80 2019-12-05T06:07:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.072+0000 7ff3a9795700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff3a4073d80 0x7ff3a4095460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:02.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.072+0000 7ff3a9f96700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4156c80 0x7ff3a4159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:02.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.072+0000 7ff3a8f94700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff3a4099990 0x7ff3a40959a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53102/0 (socket says 172.21.15.16:53102) 2019-12-05T06:07:02.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.072+0000 7ff3a9795700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff3a4073d80 0x7ff3a4095460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:60952/0 (socket says 172.21.15.16:60952) 2019-12-05T06:07:02.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.072+0000 7ff3a8f94700 1 -- 172.21.15.16:0/3821902771 learned_addr learned my addr 172.21.15.16:0/3821902771 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:02.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.073+0000 7ff3a9795700 1 -- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4156c80 msgr2=0x7ff3a4159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.073+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4156c80 0x7ff3a4159080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.073+0000 7ff3a9795700 1 -- 172.21.15.16:0/3821902771 >> v1:172.21.15.76:6789/0 conn(0x7ff3a4099990 legacy=0x7ff3a40959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.073+0000 7ff3a9795700 1 -- 172.21.15.16:0/3821902771 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a4094b60 con 0x7ff3a4073d80 2019-12-05T06:07:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.073+0000 7ff3a9f96700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4156c80 0x7ff3a4159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.073+0000 7ff3a1ffb700 1 -- 172.21.15.16:0/3821902771 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791000572 0 0) 0x7ff39800db80 con 0x7ff3a4099990 2019-12-05T06:07:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.073+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> v2:172.21.15.76:3300/0 conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.073+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> v2:172.21.15.76:3300/0 conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.073+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.074+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.074+0000 7ff3a1ffb700 1 -- 172.21.15.16:0/3821902771 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff39c004540 con 0x7ff3a4073d80 2019-12-05T06:07:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.074+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.074+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.074+0000 7ff3a1ffb700 1 -- 172.21.15.16:0/3821902771 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff39c000f90 con 0x7ff3a4073d80 2019-12-05T06:07:02.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.074+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3821902771 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a4094b60 con 0x7ff3a4073d80 2019-12-05T06:07:02.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.074+0000 7ff3a1ffb700 1 -- 172.21.15.16:0/3821902771 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff39c004b20 con 0x7ff3a4073d80 2019-12-05T06:07:02.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.074+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3821902771 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a4094b60 con 0x7ff3a4073d80 2019-12-05T06:07:02.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.075+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.075+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.075+0000 7ff3a1ffb700 1 -- 172.21.15.16:0/3821902771 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff39c004540 con 0x7ff3a4073d80 2019-12-05T06:07:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.075+0000 7ff3a1ffb700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff384023630 0x7ff384025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.076+0000 7ff3a1ffb700 1 -- 172.21.15.16:0/3821902771 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ff39c046320 con 0x7ff3a4073d80 2019-12-05T06:07:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.076+0000 7ff3a8f94700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff384023630 0x7ff384025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.076+0000 7ff3a8f94700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff384023630 0x7ff384025ac0 crc :-1 s=READY pgs=93 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.077+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3821902771 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a4094b60 con 0x7ff3a4073d80 2019-12-05T06:07:02.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.086+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.086+0000 7ff3a1ffb700 1 -- 172.21.15.16:0/3821902771 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff39c0015c0 con 0x7ff3a4073d80 2019-12-05T06:07:02.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.287+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3821902771 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7ff3a4094b60 con 0x7ff3a4073d80 2019-12-05T06:07:02.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.288+0000 7ff3a9795700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.288+0000 7ff3a1ffb700 1 -- 172.21.15.16:0/3821902771 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7ff39c000f90 con 0x7ff3a4073d80 2019-12-05T06:07:02.287 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:07:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.290+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff384023630 msgr2=0x7ff384025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.290+0000 7ff3ab1f8700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff384023630 0x7ff384025ac0 crc :-1 s=READY pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.290+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 msgr2=0x7ff3a4095460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.290+0000 7ff3ab1f8700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 0x7ff3a4095460 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7ff39c0106e0 tx=0x7ff39c010a80).stop 2019-12-05T06:07:02.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.291+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3821902771 shutdown_connections 2019-12-05T06:07:02.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.291+0000 7ff3ab1f8700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff384023630 0x7ff384025ac0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:02.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.291+0000 7ff3ab1f8700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff3a4156c80 0x7ff3a4159080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:02.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.291+0000 7ff3ab1f8700 1 --2- 172.21.15.16:0/3821902771 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff3a4073d80 0x7ff3a4095460 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:02.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.291+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3821902771 shutdown_connections 2019-12-05T06:07:02.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.291+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3821902771 wait complete. 2019-12-05T06:07:02.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:02.291+0000 7ff3ab1f8700 1 -- 172.21.15.16:0/3821902771 >> 172.21.15.16:0/3821902771 conn(0x7ff3a4077e10 msgr2=0x7ff3a408b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:02.300 INFO:tasks.mds_thrash:mds thrasher using random seed: 1575526022 2019-12-05T06:07:02.300 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:07:02.581 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.576+0000 7f40f1215700 1 Processor -- start 2019-12-05T06:07:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.576+0000 7f40f1215700 1 -- start start 2019-12-05T06:07:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.578+0000 7f40f1215700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40ec08fef0 0x7f40ec08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.578+0000 7f40f1215700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec0958e0 con 0x7f40ec08c6d0 2019-12-05T06:07:02.582 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.578+0000 7f40f1215700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec0629e0 con 0x7f40ec08d2c0 2019-12-05T06:07:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.578+0000 7f40f1215700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ec08df90 con 0x7f40ec08fef0 2019-12-05T06:07:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.578+0000 7f40ead9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f40ec08d2c0 0x7f40ec08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34434/0 (socket says 172.21.15.76:34434) 2019-12-05T06:07:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.579+0000 7f40ead9d700 1 -- 172.21.15.76:0/2827110587 learned_addr learned my addr 172.21.15.76:0/2827110587 (peer_addr_for_me v1:172.21.15.76:0/0) 2019-12-05T06:07:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.579+0000 7f40f0a14700 1 --2- 172.21.15.76:0/2827110587 >> v2:172.21.15.16:3300/0 conn(0x7f40ec08fef0 0x7f40ec08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.579+0000 7f40ea59c700 1 -- 172.21.15.76:0/2827110587 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2033703130 0 0) 0x7f40d40014d0 con 0x7f40ec08d2c0 2019-12-05T06:07:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.579+0000 7f40ea59c700 1 -- 172.21.15.76:0/2827110587 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d8002e80 con 0x7f40ec08d2c0 2019-12-05T06:07:02.583 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.579+0000 7f40f0a14700 1 -- 172.21.15.76:0/2827110587 >> v1:172.21.15.76:6789/0 conn(0x7f40ec08d2c0 legacy=0x7f40ec08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.584 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.579+0000 7f40f0a14700 1 -- 172.21.15.76:0/2827110587 >> v1:172.21.15.16:6789/0 conn(0x7f40ec08c6d0 legacy=0x7f40ec08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.584 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.579+0000 7f40ea59c700 1 -- 172.21.15.76:0/2827110587 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113721264 0 0) 0x7f40dc000ff0 con 0x7f40ec08c6d0 2019-12-05T06:07:02.584 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.579+0000 7f40f0a14700 1 -- 172.21.15.76:0/2827110587 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec08e620 con 0x7f40ec08fef0 2019-12-05T06:07:02.584 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.580+0000 7f40f0a14700 1 --2- 172.21.15.76:0/2827110587 >> v2:172.21.15.16:3300/0 conn(0x7f40ec08fef0 0x7f40ec08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f40e400af80 tx=0x7f40e400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.584 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.580+0000 7f40f0a14700 1 --2- 172.21.15.76:0/2827110587 >> v2:172.21.15.16:3300/0 conn(0x7f40ec08fef0 0x7f40ec08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f40e400af80 tx=0x7f40e400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.584 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.580+0000 7f40f0a14700 1 --2- 172.21.15.76:0/2827110587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08fef0 0x7f40ec08f720 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7f40e400af80 tx=0x7f40e400b790).ready entity=mon.0 client_cookie=75fcf0410a10770f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:02.585 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.580+0000 7f40f0a14700 1 --2- 172.21.15.76:0/2827110587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08fef0 0x7f40ec08f720 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f40e400af80 tx=0x7f40e400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.585 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.580+0000 7f40ea59c700 1 -- 172.21.15.76:0/2827110587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e4007240 con 0x7f40ec08fef0 2019-12-05T06:07:02.597 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.580+0000 7f40f0a14700 1 --2- 172.21.15.76:0/2827110587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08fef0 0x7f40ec08f720 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f40e400af80 tx=0x7f40e400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.597 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.580+0000 7f40f0a14700 1 --2- 172.21.15.76:0/2827110587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08fef0 0x7f40ec08f720 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7f40e400af80 tx=0x7f40e400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.598 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.580+0000 7f40ea59c700 1 -- 172.21.15.76:0/2827110587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e4000bb0 con 0x7f40ec08fef0 2019-12-05T06:07:02.598 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.580+0000 7f40ea59c700 1 -- 172.21.15.76:0/2827110587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e4009e60 con 0x7f40ec08fef0 2019-12-05T06:07:02.598 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.581+0000 7f40f1215700 1 -- 172.21.15.76:0/2827110587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08fef0 msgr2=0x7f40ec08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.598 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.581+0000 7f40f1215700 1 --2- 172.21.15.76:0/2827110587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08fef0 0x7f40ec08f720 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7f40e400af80 tx=0x7f40e400b790).stop 2019-12-05T06:07:02.598 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.582+0000 7f40f1215700 1 -- 172.21.15.76:0/2827110587 shutdown_connections 2019-12-05T06:07:02.598 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.582+0000 7f40f1215700 1 --2- 172.21.15.76:0/2827110587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08fef0 0x7f40ec08f720 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:02.599 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.582+0000 7f40f1215700 1 -- 172.21.15.76:0/2827110587 shutdown_connections 2019-12-05T06:07:02.599 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.582+0000 7f40f1215700 1 -- 172.21.15.76:0/2827110587 wait complete. 2019-12-05T06:07:02.599 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.582+0000 7f40f1215700 1 -- 172.21.15.76:0/2827110587 >> 172.21.15.76:0/2827110587 conn(0x7f40ec061940 msgr2=0x7f40ec096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:02.604 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.587+0000 7f40f1215700 1 Processor -- start 2019-12-05T06:07:02.604 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.588+0000 7f40f1215700 1 -- start start 2019-12-05T06:07:02.604 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.589+0000 7f40f1215700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:02.604 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.589+0000 7f40f1215700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec098da0 con 0x7f40ec09b850 2019-12-05T06:07:02.604 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.589+0000 7f40f1215700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec09f0a0 con 0x7f40ec132d00 2019-12-05T06:07:02.604 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.589+0000 7f40f1215700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec09aa60 con 0x7f40ec132530 2019-12-05T06:07:02.605 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.590+0000 7f40ead9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f40ec132d00 0x7f40ec154b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34440/0 (socket says 172.21.15.76:34440) 2019-12-05T06:07:02.605 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.590+0000 7f40ead9d700 1 -- 172.21.15.76:0/4040360270 learned_addr learned my addr 172.21.15.76:0/4040360270 (peer_addr_for_me v1:172.21.15.76:0/0) 2019-12-05T06:07:02.605 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.590+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:02.605 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.590+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838508945 0 0) 0x7f40d40014f0 con 0x7f40ec132d00 2019-12-05T06:07:02.605 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.590+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d0002e80 con 0x7f40ec132d00 2019-12-05T06:07:02.605 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.590+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943293799 0 0) 0x7f40dc001030 con 0x7f40ec09b850 2019-12-05T06:07:02.605 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.590+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40d0004220 con 0x7f40ec09b850 2019-12-05T06:07:02.606 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.590+0000 7f40f0a14700 1 -- 172.21.15.76:0/4040360270 >> v1:172.21.15.76:6789/0 conn(0x7f40ec132d00 legacy=0x7f40ec154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.606 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.590+0000 7f40f0a14700 1 -- 172.21.15.76:0/4040360270 >> v1:172.21.15.16:6789/0 conn(0x7f40ec09b850 legacy=0x7f40ec131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.606 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.590+0000 7f40f0a14700 1 -- 172.21.15.76:0/4040360270 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec09b0f0 con 0x7f40ec132530 2019-12-05T06:07:02.606 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.606 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.606 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=READY pgs=47 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:02.606 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.607 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e40097a0 con 0x7f40ec132530 2019-12-05T06:07:02.607 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.607 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e40099a0 con 0x7f40ec132530 2019-12-05T06:07:02.607 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.607 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40f1215700 1 -- 172.21.15.76:0/4040360270 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ec09b0f0 con 0x7f40ec132530 2019-12-05T06:07:02.607 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e4009b90 con 0x7f40ec132530 2019-12-05T06:07:02.607 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.591+0000 7f40f1215700 1 -- 172.21.15.76:0/4040360270 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec09b0f0 con 0x7f40ec132530 2019-12-05T06:07:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.592+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.593+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f40e4009b90 con 0x7f40ec132530 2019-12-05T06:07:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.593+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.593+0000 7f40e3fff700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023650 0x7f40d0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.593+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f40e4045dc0 con 0x7f40ec132530 2019-12-05T06:07:02.608 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.593+0000 7f40eb59e700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023650 0x7f40d0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:02.609 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.594+0000 7f40eb59e700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023650 0x7f40d0025ae0 crc :-1 s=READY pgs=94 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:02.609 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.608+0000 7f40f1215700 1 -- 172.21.15.76:0/4040360270 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ec09b0f0 con 0x7f40ec132530 2019-12-05T06:07:02.610 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.615+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.611 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.615+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f40e40095b0 con 0x7f40ec132530 2019-12-05T06:07:02.885 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.890+0000 7f40f1215700 1 -- 172.21.15.76:0/4040360270 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40ec09b0f0 con 0x7f40ec132530 2019-12-05T06:07:02.887 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.892+0000 7f40f0a14700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=THROTTLE_DONE pgs=47 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:02.887 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.892+0000 7f40e3fff700 1 -- 172.21.15.76:0/4040360270 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f40e4009b40 con 0x7f40ec132530 2019-12-05T06:07:02.888 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:07:02.889 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:07:02.890 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.895+0000 7f40f1215700 1 -- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023650 msgr2=0x7f40d0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.891 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.895+0000 7f40f1215700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023650 0x7f40d0025ae0 crc :-1 s=READY pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:02.891 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.895+0000 7f40f1215700 1 -- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 msgr2=0x7f40ec158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:02.891 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.895+0000 7f40f1215700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 secure :-1 s=READY pgs=47 cs=0 l=1 rx=0x7f40e40103b0 tx=0x7f40e4010750).stop 2019-12-05T06:07:02.891 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.895+0000 7f40f1215700 1 -- 172.21.15.76:0/4040360270 shutdown_connections 2019-12-05T06:07:02.891 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.895+0000 7f40f1215700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023650 0x7f40d0025ae0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:02.891 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.895+0000 7f40f1215700 1 --2- 172.21.15.76:0/4040360270 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec132530 0x7f40ec158320 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:02.892 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.896+0000 7f40f1215700 1 -- 172.21.15.76:0/4040360270 shutdown_connections 2019-12-05T06:07:02.893 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.896+0000 7f40f1215700 1 -- 172.21.15.76:0/4040360270 wait complete. 2019-12-05T06:07:02.893 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:02.896+0000 7f40f1215700 1 -- 172.21.15.76:0/4040360270 >> 172.21.15.76:0/4040360270 conn(0x7f40ec0714f0 msgr2=0x7f40ec09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:02.902 INFO:tasks.mds_thrash.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:31.052682+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:34.862662+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:37.644552+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T06:07:02.903 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T06:07:03.137 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.133+0000 7ffb8b7f3700 1 Processor -- start 2019-12-05T06:07:03.137 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.133+0000 7ffb8b7f3700 1 -- start start 2019-12-05T06:07:03.137 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.134+0000 7ffb8b7f3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb84086d70 0x7ffb8408aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:03.138 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.135+0000 7ffb8b7f3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 0x7ffb84064760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:03.138 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.135+0000 7ffb8b7f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb84061a10 con 0x7ffb84086980 2019-12-05T06:07:03.138 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.135+0000 7ffb8b7f3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb84060bc0 con 0x7ffb84086d70 2019-12-05T06:07:03.138 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.135+0000 7ffb8b7f3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb84060620 con 0x7ffb84064310 2019-12-05T06:07:03.138 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.135+0000 7ffb89d90700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ffb84086980 0x7ffb8408a6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34446/0 (socket says 172.21.15.76:34446) 2019-12-05T06:07:03.138 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.135+0000 7ffb89d90700 1 -- 172.21.15.76:0/1110363597 learned_addr learned my addr 172.21.15.76:0/1110363597 (peer_addr_for_me v1:172.21.15.76:0/0) 2019-12-05T06:07:03.139 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.135+0000 7ffb8a591700 1 --2- 172.21.15.76:0/1110363597 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 0x7ffb84064760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.139 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.135+0000 7ffb8958f700 1 --2- 172.21.15.76:0/1110363597 >> v2:172.21.15.16:3300/0 conn(0x7ffb84086d70 0x7ffb8408aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.139 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.135+0000 7ffb88d8e700 1 -- 172.21.15.76:0/1110363597 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575027051 0 0) 0x7ffb7c0014d0 con 0x7ffb84086980 2019-12-05T06:07:03.139 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.135+0000 7ffb88d8e700 1 -- 172.21.15.76:0/1110363597 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb6c002e80 con 0x7ffb84086980 2019-12-05T06:07:03.139 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8a591700 1 -- 172.21.15.76:0/1110363597 >> v2:172.21.15.16:3300/0 conn(0x7ffb84086d70 msgr2=0x7ffb8408aeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:03.139 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8a591700 1 --2- 172.21.15.76:0/1110363597 >> v2:172.21.15.16:3300/0 conn(0x7ffb84086d70 0x7ffb8408aeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:03.140 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8a591700 1 -- 172.21.15.76:0/1110363597 >> v1:172.21.15.76:6789/0 conn(0x7ffb84086980 legacy=0x7ffb8408a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.140 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8a591700 1 -- 172.21.15.76:0/1110363597 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb84063a10 con 0x7ffb84064310 2019-12-05T06:07:03.140 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8958f700 1 --2- 172.21.15.76:0/1110363597 >> v2:172.21.15.16:3300/0 conn(0x7ffb84086d70 0x7ffb8408aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:03.140 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8a591700 1 --2- 172.21.15.76:0/1110363597 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 0x7ffb84064760 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffb7400b0d0 tx=0x7ffb7400b880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.140 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8a591700 1 --2- 172.21.15.76:0/1110363597 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 0x7ffb84064760 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffb7400b0d0 tx=0x7ffb7400b880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.140 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8a591700 1 --2- 172.21.15.76:0/1110363597 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 0x7ffb84064760 secure :-1 s=READY pgs=48 cs=0 l=1 rx=0x7ffb7400b0d0 tx=0x7ffb7400b880).ready entity=mon.2 client_cookie=6b331c42a43de8ed server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:03.141 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8a591700 1 --2- 172.21.15.76:0/1110363597 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 0x7ffb84064760 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7ffb7400b0d0 tx=0x7ffb7400b880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.141 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb88d8e700 1 -- 172.21.15.76:0/1110363597 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb74000bc0 con 0x7ffb84064310 2019-12-05T06:07:03.141 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8a591700 1 --2- 172.21.15.76:0/1110363597 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 0x7ffb84064760 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7ffb7400b0d0 tx=0x7ffb7400b880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.141 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.136+0000 7ffb8a591700 1 --2- 172.21.15.76:0/1110363597 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 0x7ffb84064760 secure :-1 s=THROTTLE_DONE pgs=48 cs=0 l=1 rx=0x7ffb7400b0d0 tx=0x7ffb7400b880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.141 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.137+0000 7ffb88d8e700 1 -- 172.21.15.76:0/1110363597 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb740010e0 con 0x7ffb84064310 2019-12-05T06:07:03.142 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.137+0000 7ffb88d8e700 1 -- 172.21.15.76:0/1110363597 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb74001850 con 0x7ffb84064310 2019-12-05T06:07:03.142 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.137+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/1110363597 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 msgr2=0x7ffb84064760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.142 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.137+0000 7ffb8b7f3700 1 --2- 172.21.15.76:0/1110363597 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 0x7ffb84064760 secure :-1 s=READY pgs=48 cs=0 l=1 rx=0x7ffb7400b0d0 tx=0x7ffb7400b880).stop 2019-12-05T06:07:03.142 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.138+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/1110363597 shutdown_connections 2019-12-05T06:07:03.142 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.138+0000 7ffb8b7f3700 1 --2- 172.21.15.76:0/1110363597 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb84064310 0x7ffb84064760 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.142 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.138+0000 7ffb8b7f3700 1 --2- 172.21.15.76:0/1110363597 >> v2:172.21.15.16:3300/0 conn(0x7ffb84086d70 0x7ffb8408aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:03.142 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.139+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/1110363597 shutdown_connections 2019-12-05T06:07:03.143 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.139+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/1110363597 wait complete. 2019-12-05T06:07:03.143 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.139+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/1110363597 >> 172.21.15.76:0/1110363597 conn(0x7ffb8405f5a0 msgr2=0x7ffb8409cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:03.147 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.143+0000 7ffb8b7f3700 1 Processor -- start 2019-12-05T06:07:03.147 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.143+0000 7ffb8b7f3700 1 -- start start 2019-12-05T06:07:03.148 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.144+0000 7ffb8b7f3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ffb8414c2c0 0x7ffb8414c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:03.148 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.144+0000 7ffb8b7f3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb840973c0 0x7ffb840964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:03.148 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.144+0000 7ffb8b7f3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb84099de0 con 0x7ffb840969e0 2019-12-05T06:07:03.148 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.144+0000 7ffb8b7f3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb8409b880 con 0x7ffb840973c0 2019-12-05T06:07:03.148 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.144+0000 7ffb8b7f3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb8409ae50 con 0x7ffb8414c2c0 2019-12-05T06:07:03.149 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.144+0000 7ffb89d90700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ffb8414c2c0 0x7ffb8414c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.149 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.144+0000 7ffb89d90700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ffb8414c2c0 0x7ffb8414c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51374/0 (socket says 172.21.15.76:51374) 2019-12-05T06:07:03.149 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.144+0000 7ffb89d90700 1 -- 172.21.15.76:0/2572696292 learned_addr learned my addr 172.21.15.76:0/2572696292 (peer_addr_for_me v2:172.21.15.76:0/0) 2019-12-05T06:07:03.149 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.144+0000 7ffb8958f700 1 --2- 172.21.15.76:0/2572696292 >> v2:172.21.15.16:3300/0 conn(0x7ffb840973c0 0x7ffb840964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.149 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.145+0000 7ffb827fc700 1 -- 172.21.15.76:0/2572696292 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77301510 0 0) 0x7ffb7400b240 con 0x7ffb840969e0 2019-12-05T06:07:03.150 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.145+0000 7ffb827fc700 1 -- 172.21.15.76:0/2572696292 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb64002e80 con 0x7ffb840969e0 2019-12-05T06:07:03.150 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.145+0000 7ffb89d90700 1 -- 172.21.15.76:0/2572696292 >> v2:172.21.15.16:3300/0 conn(0x7ffb840973c0 msgr2=0x7ffb840964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.150 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.145+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> v2:172.21.15.16:3300/0 conn(0x7ffb840973c0 0x7ffb840964a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.150 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.145+0000 7ffb89d90700 1 -- 172.21.15.76:0/2572696292 >> v1:172.21.15.16:6789/0 conn(0x7ffb840969e0 legacy=0x7ffb84094440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.150 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.145+0000 7ffb89d90700 1 -- 172.21.15.76:0/2572696292 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8409bf10 con 0x7ffb8414c2c0 2019-12-05T06:07:03.150 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.145+0000 7ffb8958f700 1 --2- 172.21.15.76:0/2572696292 >> v2:172.21.15.16:3300/0 conn(0x7ffb840973c0 0x7ffb840964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:03.150 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.145+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> v2:172.21.15.76:3300/0 conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.151 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.145+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> v2:172.21.15.76:3300/0 conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.151 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.145+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=READY pgs=146 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:03.151 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.146+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.151 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.146+0000 7ffb827fc700 1 -- 172.21.15.76:0/2572696292 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb7c001010 con 0x7ffb8414c2c0 2019-12-05T06:07:03.151 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.146+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/2572696292 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8409bf10 con 0x7ffb8414c2c0 2019-12-05T06:07:03.151 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.146+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.151 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.146+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.152 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.146+0000 7ffb827fc700 1 -- 172.21.15.76:0/2572696292 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb7c007950 con 0x7ffb8414c2c0 2019-12-05T06:07:03.152 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.146+0000 7ffb827fc700 1 -- 172.21.15.76:0/2572696292 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb7c001010 con 0x7ffb8414c2c0 2019-12-05T06:07:03.152 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.146+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/2572696292 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8409bf10 con 0x7ffb8414c2c0 2019-12-05T06:07:03.152 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.147+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.152 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.148+0000 7ffb827fc700 1 -- 172.21.15.76:0/2572696292 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ffb7c001010 con 0x7ffb8414c2c0 2019-12-05T06:07:03.152 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.148+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.152 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.148+0000 7ffb827fc700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb64023670 0x7ffb64025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:03.153 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.148+0000 7ffb827fc700 1 -- 172.21.15.76:0/2572696292 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7ffb7c046140 con 0x7ffb8414c2c0 2019-12-05T06:07:03.153 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.149+0000 7ffb8958f700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb64023670 0x7ffb64025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.153 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.149+0000 7ffb8958f700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb64023670 0x7ffb64025ae0 crc :-1 s=READY pgs=95 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:03.153 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.150+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/2572696292 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8409bf10 con 0x7ffb8414c2c0 2019-12-05T06:07:03.154 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.158+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.154 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.158+0000 7ffb827fc700 1 -- 172.21.15.76:0/2572696292 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ffb7c02b3b0 con 0x7ffb8414c2c0 2019-12-05T06:07:03.402 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.404+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/2572696292 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7ffb8409bf10 con 0x7ffb8414c2c0 2019-12-05T06:07:03.402 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.405+0000 7ffb89d90700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.403 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.405+0000 7ffb827fc700 1 -- 172.21.15.76:0/2572696292 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7ffb7c001010 con 0x7ffb8414c2c0 2019-12-05T06:07:03.403 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 8 2019-12-05T06:07:03.404 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb64023670 msgr2=0x7ffb64025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.404 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb64023670 0x7ffb64025ae0 crc :-1 s=READY pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.404 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 msgr2=0x7ffb8414c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.404 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 0x7ffb8414c6b0 secure :-1 s=READY pgs=146 cs=0 l=1 rx=0x7ffb7c010760 tx=0x7ffb7c010b00).stop 2019-12-05T06:07:03.404 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/2572696292 shutdown_connections 2019-12-05T06:07:03.404 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb64023670 0x7ffb64025ae0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.405 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 --2- 172.21.15.76:0/2572696292 >> v2:172.21.15.16:3300/0 conn(0x7ffb840973c0 0x7ffb840964a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.405 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 --2- 172.21.15.76:0/2572696292 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffb8414c2c0 0x7ffb8414c6b0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.405 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/2572696292 shutdown_connections 2019-12-05T06:07:03.405 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/2572696292 wait complete. 2019-12-05T06:07:03.405 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.408+0000 7ffb8b7f3700 1 -- 172.21.15.76:0/2572696292 >> 172.21.15.76:0/2572696292 conn(0x7ffb84077e10 msgr2=0x7ffb8408b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:03.413 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T06:07:03.624 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.619+0000 7f71e8f80700 1 Processor -- start 2019-12-05T06:07:03.624 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.619+0000 7f71e8f80700 1 -- start start 2019-12-05T06:07:03.624 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.621+0000 7f71e8f80700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:03.624 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.621+0000 7f71e8f80700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f71e408a6b0 0x7f71e408ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:03.624 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.621+0000 7f71e8f80700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4061a10 con 0x7f71e408b080 2019-12-05T06:07:03.624 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.621+0000 7f71e8f80700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e4060bc0 con 0x7f71e408a6b0 2019-12-05T06:07:03.625 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.621+0000 7f71e8f80700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e4060620 con 0x7f71e4086980 2019-12-05T06:07:03.625 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.621+0000 7f71e2d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.625 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.621+0000 7f71e259c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f71e408a6b0 0x7f71e408ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.625 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.621+0000 7f71e259c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f71e408a6b0 0x7f71e408ab40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51380/0 (socket says 172.21.15.76:51380) 2019-12-05T06:07:03.625 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.621+0000 7f71e2d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.76:48320/0 (socket says 172.21.15.76:48320) 2019-12-05T06:07:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.621+0000 7f71e2d9d700 1 -- 172.21.15.76:0/2091563747 learned_addr learned my addr 172.21.15.76:0/2091563747 (peer_addr_for_me v2:172.21.15.76:0/0) 2019-12-05T06:07:03.632 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e2d9d700 1 -- 172.21.15.76:0/2091563747 >> v2:172.21.15.76:3300/0 conn(0x7f71e408a6b0 msgr2=0x7f71e408ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e2d9d700 1 --2- 172.21.15.76:0/2091563747 >> v2:172.21.15.76:3300/0 conn(0x7f71e408a6b0 0x7f71e408ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e2d9d700 1 -- 172.21.15.76:0/2091563747 >> v1:172.21.15.16:6789/0 conn(0x7f71e408b080 legacy=0x7f71e4064310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e2d9d700 1 -- 172.21.15.76:0/2091563747 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e4063a10 con 0x7f71e4086980 2019-12-05T06:07:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e259c700 1 --2- 172.21.15.76:0/2091563747 >> v2:172.21.15.76:3300/0 conn(0x7f71e408a6b0 0x7f71e408ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e2d9d700 1 --2- 172.21.15.76:0/2091563747 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f71cc00f740 tx=0x7f71cc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.633 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e2d9d700 1 --2- 172.21.15.76:0/2091563747 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f71cc00f740 tx=0x7f71cc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e2d9d700 1 --2- 172.21.15.76:0/2091563747 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7f71cc00f740 tx=0x7f71cc00fae0).ready entity=mon.2 client_cookie=85453fd9f5b35743 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e2d9d700 1 --2- 172.21.15.76:0/2091563747 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f71cc00f740 tx=0x7f71cc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e2d9d700 1 --2- 172.21.15.76:0/2091563747 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f71cc00f740 tx=0x7f71cc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e2d9d700 1 --2- 172.21.15.76:0/2091563747 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 secure :-1 s=THROTTLE_DONE pgs=49 cs=0 l=1 rx=0x7f71cc00f740 tx=0x7f71cc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e1d9b700 1 -- 172.21.15.76:0/2091563747 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71cc000960 con 0x7f71e4086980 2019-12-05T06:07:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e1d9b700 1 -- 172.21.15.76:0/2091563747 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71cc000c90 con 0x7f71e4086980 2019-12-05T06:07:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e1d9b700 1 -- 172.21.15.76:0/2091563747 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71cc0013e0 con 0x7f71e4086980 2019-12-05T06:07:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e8f80700 1 -- 172.21.15.76:0/2091563747 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 msgr2=0x7f71e4086d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.622+0000 7f71e8f80700 1 --2- 172.21.15.76:0/2091563747 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 secure :-1 s=READY pgs=49 cs=0 l=1 rx=0x7f71cc00f740 tx=0x7f71cc00fae0).stop 2019-12-05T06:07:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.625+0000 7f71e8f80700 1 -- 172.21.15.76:0/2091563747 shutdown_connections 2019-12-05T06:07:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.625+0000 7f71e8f80700 1 --2- 172.21.15.76:0/2091563747 >> v2:172.21.15.76:3300/0 conn(0x7f71e408a6b0 0x7f71e408ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.625+0000 7f71e8f80700 1 --2- 172.21.15.76:0/2091563747 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e4086980 0x7f71e4086d70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.625+0000 7f71e8f80700 1 -- 172.21.15.76:0/2091563747 shutdown_connections 2019-12-05T06:07:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.625+0000 7f71e8f80700 1 -- 172.21.15.76:0/2091563747 wait complete. 2019-12-05T06:07:03.636 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.625+0000 7f71e8f80700 1 -- 172.21.15.76:0/2091563747 >> 172.21.15.76:0/2091563747 conn(0x7f71e405f5a0 msgr2=0x7f71e409cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:03.649 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.630+0000 7f71e8f80700 1 Processor -- start 2019-12-05T06:07:03.650 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.630+0000 7f71e8f80700 1 -- start start 2019-12-05T06:07:03.650 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.631+0000 7f71e8f80700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f71e40984d0 0x7f71e4097d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:03.650 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.631+0000 7f71e8f80700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71e4154c60 0x7f71e4157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:03.650 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.631+0000 7f71e8f80700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e414a760 con 0x7f71e4094740 2019-12-05T06:07:03.650 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.631+0000 7f71e8f80700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e4095950 con 0x7f71e4154c60 2019-12-05T06:07:03.650 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.631+0000 7f71e8f80700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e4095c20 con 0x7f71e40984d0 2019-12-05T06:07:03.651 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.631+0000 7f71e2d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f71e4094740 0x7f71e4097500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34466/0 (socket says 172.21.15.76:34466) 2019-12-05T06:07:03.651 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.631+0000 7f71e2d9d700 1 -- 172.21.15.76:0/3600406661 learned_addr learned my addr 172.21.15.76:0/3600406661 (peer_addr_for_me v1:172.21.15.76:0/0) 2019-12-05T06:07:03.651 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.632+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> v2:172.21.15.76:3300/0 conn(0x7f71e40984d0 0x7f71e4097d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.651 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.632+0000 7f71e359e700 1 --2- 172.21.15.76:0/3600406661 >> v2:172.21.15.16:3300/0 conn(0x7f71e4154c60 0x7f71e4157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.651 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.632+0000 7f71db7fe700 1 -- 172.21.15.76:0/3600406661 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748552776 0 0) 0x7f71cc0080f0 con 0x7f71e4094740 2019-12-05T06:07:03.651 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.632+0000 7f71db7fe700 1 -- 172.21.15.76:0/3600406661 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71c8002e80 con 0x7f71e4094740 2019-12-05T06:07:03.651 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.632+0000 7f71e259c700 1 -- 172.21.15.76:0/3600406661 >> v2:172.21.15.16:3300/0 conn(0x7f71e4154c60 msgr2=0x7f71e4157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.652 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.632+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> v2:172.21.15.16:3300/0 conn(0x7f71e4154c60 0x7f71e4157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.652 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.632+0000 7f71e259c700 1 -- 172.21.15.76:0/3600406661 >> v1:172.21.15.76:6789/0 conn(0x7f71e4094740 legacy=0x7f71e4097500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.652 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.632+0000 7f71e259c700 1 -- 172.21.15.76:0/3600406661 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e4096c00 con 0x7f71e40984d0 2019-12-05T06:07:03.652 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.632+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> v2:172.21.15.76:3300/0 conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.652 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.632+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> v2:172.21.15.76:3300/0 conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.652 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.633+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=READY pgs=148 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:03.653 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.633+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.653 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.633+0000 7f71db7fe700 1 -- 172.21.15.76:0/3600406661 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71d4000940 con 0x7f71e40984d0 2019-12-05T06:07:03.653 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.633+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.653 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.633+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.653 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.633+0000 7f71db7fe700 1 -- 172.21.15.76:0/3600406661 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d4009770 con 0x7f71e40984d0 2019-12-05T06:07:03.653 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.633+0000 7f71e8f80700 1 -- 172.21.15.76:0/3600406661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e4096c00 con 0x7f71e40984d0 2019-12-05T06:07:03.654 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.633+0000 7f71db7fe700 1 -- 172.21.15.76:0/3600406661 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71d4009e00 con 0x7f71e40984d0 2019-12-05T06:07:03.654 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.633+0000 7f71e8f80700 1 -- 172.21.15.76:0/3600406661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e4096c00 con 0x7f71e40984d0 2019-12-05T06:07:03.654 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.634+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.654 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.634+0000 7f71db7fe700 1 -- 172.21.15.76:0/3600406661 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f71d4009960 con 0x7f71e40984d0 2019-12-05T06:07:03.654 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.634+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.654 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.635+0000 7f71db7fe700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c8023670 0x7f71c8025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.635+0000 7f71db7fe700 1 -- 172.21.15.76:0/3600406661 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f71d4045d20 con 0x7f71e40984d0 2019-12-05T06:07:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.635+0000 7f71e2d9d700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c8023670 0x7f71c8025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.635+0000 7f71e2d9d700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c8023670 0x7f71c8025ae0 crc :-1 s=READY pgs=96 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.650+0000 7f71e8f80700 1 -- 172.21.15.76:0/3600406661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e4096c00 con 0x7f71e40984d0 2019-12-05T06:07:03.655 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.655+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.656 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.655+0000 7f71db7fe700 1 -- 172.21.15.76:0/3600406661 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f71d4005180 con 0x7f71e40984d0 2019-12-05T06:07:03.881 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.884+0000 7f71e8f80700 1 -- 172.21.15.76:0/3600406661 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f71e4096c00 con 0x7f71e40984d0 2019-12-05T06:07:03.881 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.885+0000 7f71e259c700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=THROTTLE_DONE pgs=148 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:03.881 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.885+0000 7f71db7fe700 1 -- 172.21.15.76:0/3600406661 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f71d402afc0 con 0x7f71e40984d0 2019-12-05T06:07:03.881 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 8 2019-12-05T06:07:03.884 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 -- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c8023670 msgr2=0x7f71c8025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.884 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c8023670 0x7f71c8025ae0 crc :-1 s=READY pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.885 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 -- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 msgr2=0x7f71e4097d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:03.885 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 0x7f71e4097d00 secure :-1 s=READY pgs=148 cs=0 l=1 rx=0x7f71d4010390 tx=0x7f71d4005ee0).stop 2019-12-05T06:07:03.885 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 -- 172.21.15.76:0/3600406661 shutdown_connections 2019-12-05T06:07:03.886 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c8023670 0x7f71c8025ae0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.886 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 --2- 172.21.15.76:0/3600406661 >> v2:172.21.15.16:3300/0 conn(0x7f71e4154c60 0x7f71e4157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.886 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 --2- 172.21.15.76:0/3600406661 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f71e40984d0 0x7f71e4097d00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:03.886 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 -- 172.21.15.76:0/3600406661 shutdown_connections 2019-12-05T06:07:03.886 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 -- 172.21.15.76:0/3600406661 wait complete. 2019-12-05T06:07:03.886 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:03.888+0000 7f71e8f80700 1 -- 172.21.15.76:0/3600406661 >> 172.21.15.76:0/3600406661 conn(0x7f71e4077e10 msgr2=0x7f71e408b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:03.900 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T06:07:04.162 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.158+0000 7f573dcba700 1 Processor -- start 2019-12-05T06:07:04.162 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.158+0000 7f573dcba700 1 -- start start 2019-12-05T06:07:04.162 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.159+0000 7f573dcba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5738086980 0x7f5738086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:04.162 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.159+0000 7f573dcba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5738061a10 con 0x7f573808a6b0 2019-12-05T06:07:04.162 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.159+0000 7f573dcba700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5738064f70 con 0x7f5738064310 2019-12-05T06:07:04.162 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.159+0000 7f573dcba700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573808b340 con 0x7f5738086980 2019-12-05T06:07:04.163 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.160+0000 7f573ca58700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5738064310 0x7f5738064770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34476/0 (socket says 172.21.15.76:34476) 2019-12-05T06:07:04.163 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.160+0000 7f573ca58700 1 -- 172.21.15.76:0/4251893455 learned_addr learned my addr 172.21.15.76:0/4251893455 (peer_addr_for_me v1:172.21.15.76:0/0) 2019-12-05T06:07:04.163 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.160+0000 7f5737fff700 1 --2- 172.21.15.76:0/4251893455 >> v2:172.21.15.16:3300/0 conn(0x7f5738086980 0x7f5738086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:04.163 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.160+0000 7f5736ffd700 1 -- 172.21.15.76:0/4251893455 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515687761 0 0) 0x7f5730001150 con 0x7f5738064310 2019-12-05T06:07:04.163 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.160+0000 7f5736ffd700 1 -- 172.21.15.76:0/4251893455 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5724002e80 con 0x7f5738064310 2019-12-05T06:07:04.163 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.160+0000 7f5736ffd700 1 -- 172.21.15.76:0/4251893455 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3976235247 0 0) 0x7f5720001080 con 0x7f573808a6b0 2019-12-05T06:07:04.164 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.160+0000 7f5736ffd700 1 -- 172.21.15.76:0/4251893455 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57240030b0 con 0x7f573808a6b0 2019-12-05T06:07:04.164 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.160+0000 7f5737fff700 1 -- 172.21.15.76:0/4251893455 >> v1:172.21.15.76:6789/0 conn(0x7f5738064310 legacy=0x7f5738064770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.164 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5737fff700 1 -- 172.21.15.76:0/4251893455 >> v1:172.21.15.16:6789/0 conn(0x7f573808a6b0 legacy=0x7f573808ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.164 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5737fff700 1 -- 172.21.15.76:0/4251893455 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5738063a10 con 0x7f5738086980 2019-12-05T06:07:04.164 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5737fff700 1 --2- 172.21.15.76:0/4251893455 >> v2:172.21.15.16:3300/0 conn(0x7f5738086980 0x7f5738086d70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f572800b2d0 tx=0x7f572800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.176 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5737fff700 1 --2- 172.21.15.76:0/4251893455 >> v2:172.21.15.16:3300/0 conn(0x7f5738086980 0x7f5738086d70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f572800b2d0 tx=0x7f572800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.177 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5737fff700 1 --2- 172.21.15.76:0/4251893455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5738086980 0x7f5738086d70 secure :-1 s=READY pgs=180 cs=0 l=1 rx=0x7f572800b2d0 tx=0x7f572800bae0).ready entity=mon.0 client_cookie=7a1e0cfe2d57c293 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:04.177 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5737fff700 1 --2- 172.21.15.76:0/4251893455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5738086980 0x7f5738086d70 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f572800b2d0 tx=0x7f572800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.177 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5736ffd700 1 -- 172.21.15.76:0/4251893455 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5728000ce0 con 0x7f5738086980 2019-12-05T06:07:04.177 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5737fff700 1 --2- 172.21.15.76:0/4251893455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5738086980 0x7f5738086d70 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f572800b2d0 tx=0x7f572800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.177 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5737fff700 1 --2- 172.21.15.76:0/4251893455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5738086980 0x7f5738086d70 secure :-1 s=THROTTLE_DONE pgs=180 cs=0 l=1 rx=0x7f572800b2d0 tx=0x7f572800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.177 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5736ffd700 1 -- 172.21.15.76:0/4251893455 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5728009be0 con 0x7f5738086980 2019-12-05T06:07:04.177 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.161+0000 7f5736ffd700 1 -- 172.21.15.76:0/4251893455 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f572800a2b0 con 0x7f5738086980 2019-12-05T06:07:04.178 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.162+0000 7f573dcba700 1 -- 172.21.15.76:0/4251893455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5738086980 msgr2=0x7f5738086d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.178 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.162+0000 7f573dcba700 1 --2- 172.21.15.76:0/4251893455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5738086980 0x7f5738086d70 secure :-1 s=READY pgs=180 cs=0 l=1 rx=0x7f572800b2d0 tx=0x7f572800bae0).stop 2019-12-05T06:07:04.178 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.163+0000 7f573dcba700 1 -- 172.21.15.76:0/4251893455 shutdown_connections 2019-12-05T06:07:04.178 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.163+0000 7f573dcba700 1 --2- 172.21.15.76:0/4251893455 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5738086980 0x7f5738086d70 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:04.178 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.163+0000 7f573dcba700 1 -- 172.21.15.76:0/4251893455 shutdown_connections 2019-12-05T06:07:04.178 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.163+0000 7f573dcba700 1 -- 172.21.15.76:0/4251893455 wait complete. 2019-12-05T06:07:04.179 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.163+0000 7f573dcba700 1 -- 172.21.15.76:0/4251893455 >> 172.21.15.76:0/4251893455 conn(0x7f573805f5a0 msgr2=0x7f573809cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:04.183 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.168+0000 7f573dcba700 1 Processor -- start 2019-12-05T06:07:04.183 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.168+0000 7f573dcba700 1 -- start start 2019-12-05T06:07:04.183 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.170+0000 7f573dcba700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5738096890 0x7f57380974f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:04.183 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.170+0000 7f573dcba700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5738098360 0x7f5738097a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:04.183 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.170+0000 7f573dcba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5738096ec0 con 0x7f5738087680 2019-12-05T06:07:04.183 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.170+0000 7f573dcba700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5738131ae0 con 0x7f5738096890 2019-12-05T06:07:04.183 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.170+0000 7f573dcba700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57381310b0 con 0x7f5738098360 2019-12-05T06:07:04.184 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.170+0000 7f5737fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5738096890 0x7f57380974f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:04.184 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.170+0000 7f57377fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5738098360 0x7f5738097a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:04.184 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.170+0000 7f5737fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5738096890 0x7f57380974f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51400/0 (socket says 172.21.15.76:51400) 2019-12-05T06:07:04.184 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.170+0000 7f5737fff700 1 -- 172.21.15.76:0/1260963694 learned_addr learned my addr 172.21.15.76:0/1260963694 (peer_addr_for_me v2:172.21.15.76:0/0) 2019-12-05T06:07:04.184 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5737fff700 1 -- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5738098360 msgr2=0x7f5738097a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.184 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5738098360 0x7f5738097a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:04.185 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5737fff700 1 -- 172.21.15.76:0/1260963694 >> v1:172.21.15.16:6789/0 conn(0x7f5738087680 legacy=0x7f5738087b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.185 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5737fff700 1 -- 172.21.15.76:0/1260963694 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5738132170 con 0x7f5738096890 2019-12-05T06:07:04.185 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f57377fe700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5738098360 0x7f5738097a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:04.186 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> v2:172.21.15.76:3300/0 conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.186 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> v2:172.21.15.76:3300/0 conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.186 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:04.186 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.187 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.187 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.187 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5734ff9700 1 -- 172.21.15.76:0/1260963694 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5728000cc0 con 0x7f5738096890 2019-12-05T06:07:04.187 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5734ff9700 1 -- 172.21.15.76:0/1260963694 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5728009d00 con 0x7f5738096890 2019-12-05T06:07:04.187 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f5734ff9700 1 -- 172.21.15.76:0/1260963694 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f572800a310 con 0x7f5738096890 2019-12-05T06:07:04.187 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f573dcba700 1 -- 172.21.15.76:0/1260963694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5738132170 con 0x7f5738096890 2019-12-05T06:07:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.171+0000 7f573dcba700 1 -- 172.21.15.76:0/1260963694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5738132170 con 0x7f5738096890 2019-12-05T06:07:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.172+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.173+0000 7f5734ff9700 1 -- 172.21.15.76:0/1260963694 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5728009ef0 con 0x7f5738096890 2019-12-05T06:07:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.173+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.174+0000 7f5734ff9700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f571c023630 0x7f571c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.174+0000 7f5734ff9700 1 -- 172.21.15.76:0/1260963694 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f57280461d0 con 0x7f5738096890 2019-12-05T06:07:04.189 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.174+0000 7f57377fe700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f571c023630 0x7f571c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:04.189 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.174+0000 7f57377fe700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f571c023630 0x7f571c025ac0 crc :-1 s=READY pgs=97 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:04.189 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.189+0000 7f573dcba700 1 -- 172.21.15.76:0/1260963694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5738132170 con 0x7f5738096890 2019-12-05T06:07:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.197+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.192 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.197+0000 7f5734ff9700 1 -- 172.21.15.76:0/1260963694 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5728009900 con 0x7f5738096890 2019-12-05T06:07:04.442 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.447+0000 7f573dcba700 1 -- 172.21.15.76:0/1260963694 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f5738132170 con 0x7f5738096890 2019-12-05T06:07:04.443 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.448+0000 7f5737fff700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.443 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.448+0000 7f5734ff9700 1 -- 172.21.15.76:0/1260963694 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5728009ef0 con 0x7f5738096890 2019-12-05T06:07:04.443 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 8 2019-12-05T06:07:04.446 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.450+0000 7f573dcba700 1 -- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f571c023630 msgr2=0x7f571c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.446 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.450+0000 7f573dcba700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f571c023630 0x7f571c025ac0 crc :-1 s=READY pgs=97 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:04.446 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.450+0000 7f573dcba700 1 -- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 msgr2=0x7f57380974f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.446 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.450+0000 7f573dcba700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 0x7f57380974f0 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7f57280107f0 tx=0x7f5728010b90).stop 2019-12-05T06:07:04.446 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.451+0000 7f573dcba700 1 -- 172.21.15.76:0/1260963694 shutdown_connections 2019-12-05T06:07:04.447 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.451+0000 7f573dcba700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f571c023630 0x7f571c025ac0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:04.447 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.451+0000 7f573dcba700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5738098360 0x7f5738097a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:04.447 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.451+0000 7f573dcba700 1 --2- 172.21.15.76:0/1260963694 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5738096890 0x7f57380974f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:04.447 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.451+0000 7f573dcba700 1 -- 172.21.15.76:0/1260963694 shutdown_connections 2019-12-05T06:07:04.447 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.451+0000 7f573dcba700 1 -- 172.21.15.76:0/1260963694 wait complete. 2019-12-05T06:07:04.457 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:04.451+0000 7f573dcba700 1 -- 172.21.15.76:0/1260963694 >> 172.21.15.76:0/1260963694 conn(0x7f573808b790 msgr2=0x7f573808a210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:04.460 INFO:tasks.mds_thrash:Wait for all MDSs to reach steady state... 2019-12-05T06:07:04.461 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.715+0000 7f7ebc02a700 1 Processor -- start 2019-12-05T06:07:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.715+0000 7f7ebc02a700 1 -- start start 2019-12-05T06:07:04.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.717+0000 7f7ebc02a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7eb408c6d0 0x7f7eb408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.717+0000 7f7ebc02a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb408fb80 con 0x7f7eb408f720 2019-12-05T06:07:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.717+0000 7f7ebc02a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb40958e0 con 0x7f7eb408fef0 2019-12-05T06:07:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.717+0000 7f7ebc02a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb408df90 con 0x7f7eb408c6d0 2019-12-05T06:07:04.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.717+0000 7f7ebadc8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7eb408f720 0x7f7eb41533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60040/0 (socket says 172.21.15.16:60040) 2019-12-05T06:07:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.717+0000 7f7ebadc8700 1 -- 172.21.15.16:0/54520379 learned_addr learned my addr 172.21.15.16:0/54520379 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.717+0000 7f7eba5c7700 1 --2- 172.21.15.16:0/54520379 >> v2:172.21.15.76:3300/0 conn(0x7f7eb408c6d0 0x7f7eb408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.718+0000 7f7eb95c5700 1 -- 172.21.15.16:0/54520379 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215206813 0 0) 0x7f7eac001480 con 0x7f7eb408f720 2019-12-05T06:07:04.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.718+0000 7f7eb95c5700 1 -- 172.21.15.16:0/54520379 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ea0002e80 con 0x7f7eb408f720 2019-12-05T06:07:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.718+0000 7f7eb95c5700 1 -- 172.21.15.16:0/54520379 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380237650 0 0) 0x7f7ea4001080 con 0x7f7eb408fef0 2019-12-05T06:07:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.718+0000 7f7eba5c7700 1 -- 172.21.15.16:0/54520379 >> v1:172.21.15.76:6789/0 conn(0x7f7eb408fef0 legacy=0x7f7eb408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.718+0000 7f7eba5c7700 1 -- 172.21.15.16:0/54520379 >> v1:172.21.15.16:6789/0 conn(0x7f7eb408f720 legacy=0x7f7eb41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.718+0000 7f7eba5c7700 1 -- 172.21.15.16:0/54520379 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb408e620 con 0x7f7eb408c6d0 2019-12-05T06:07:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.719+0000 7f7eba5c7700 1 --2- 172.21.15.16:0/54520379 >> v2:172.21.15.76:3300/0 conn(0x7f7eb408c6d0 0x7f7eb408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7eb000af50 tx=0x7f7eb000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.719+0000 7f7eba5c7700 1 --2- 172.21.15.16:0/54520379 >> v2:172.21.15.76:3300/0 conn(0x7f7eb408c6d0 0x7f7eb408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7eb000af50 tx=0x7f7eb000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.719+0000 7f7eba5c7700 1 --2- 172.21.15.16:0/54520379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb408c6d0 0x7f7eb408cac0 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7f7eb000af50 tx=0x7f7eb000b760).ready entity=mon.1 client_cookie=73bc04c0aac9a01a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.719+0000 7f7eba5c7700 1 --2- 172.21.15.16:0/54520379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb408c6d0 0x7f7eb408cac0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f7eb000af50 tx=0x7f7eb000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.719+0000 7f7eba5c7700 1 --2- 172.21.15.16:0/54520379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb408c6d0 0x7f7eb408cac0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f7eb000af50 tx=0x7f7eb000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.719+0000 7f7eb95c5700 1 -- 172.21.15.16:0/54520379 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eb0000960 con 0x7f7eb408c6d0 2019-12-05T06:07:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.719+0000 7f7eba5c7700 1 --2- 172.21.15.16:0/54520379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb408c6d0 0x7f7eb408cac0 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f7eb000af50 tx=0x7f7eb000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.719+0000 7f7eb95c5700 1 -- 172.21.15.16:0/54520379 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb00098e0 con 0x7f7eb408c6d0 2019-12-05T06:07:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.719+0000 7f7eb95c5700 1 -- 172.21.15.16:0/54520379 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eb000a050 con 0x7f7eb408c6d0 2019-12-05T06:07:04.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.720+0000 7f7ebc02a700 1 -- 172.21.15.16:0/54520379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb408c6d0 msgr2=0x7f7eb408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.720+0000 7f7ebc02a700 1 --2- 172.21.15.16:0/54520379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb408c6d0 0x7f7eb408cac0 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7f7eb000af50 tx=0x7f7eb000b760).stop 2019-12-05T06:07:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.721+0000 7f7ebc02a700 1 -- 172.21.15.16:0/54520379 shutdown_connections 2019-12-05T06:07:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.721+0000 7f7ebc02a700 1 --2- 172.21.15.16:0/54520379 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb408c6d0 0x7f7eb408cac0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:04.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.721+0000 7f7ebc02a700 1 -- 172.21.15.16:0/54520379 shutdown_connections 2019-12-05T06:07:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.721+0000 7f7ebc02a700 1 -- 172.21.15.16:0/54520379 wait complete. 2019-12-05T06:07:04.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.721+0000 7f7ebc02a700 1 -- 172.21.15.16:0/54520379 >> 172.21.15.16:0/54520379 conn(0x7f7eb4061940 msgr2=0x7f7eb4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:04.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.727+0000 7f7ebc02a700 1 Processor -- start 2019-12-05T06:07:04.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.727+0000 7f7ebc02a700 1 -- start start 2019-12-05T06:07:04.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.729+0000 7f7ebc02a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7eb406e840 0x7f7eb4134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:04.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.729+0000 7f7eba5c7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7eb406dc50 0x7f7eb406e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60046/0 (socket says 172.21.15.16:60046) 2019-12-05T06:07:04.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.729+0000 7f7ebc02a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7eb4135450 0x7f7eb41349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:04.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.729+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7eb4128af0 con 0x7f7eb406dc50 2019-12-05T06:07:04.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.729+0000 7f7eba5c7700 1 -- 172.21.15.16:0/2299311287 learned_addr learned my addr 172.21.15.16:0/2299311287 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:04.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.729+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb4132830 con 0x7f7eb4135450 2019-12-05T06:07:04.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.729+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb41334f0 con 0x7f7eb406e840 2019-12-05T06:07:04.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.729+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> v2:172.21.15.76:3300/0 conn(0x7f7eb406e840 0x7f7eb4134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:04.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.729+0000 7f7ebadc8700 1 --2- 172.21.15.16:0/2299311287 >> v2:172.21.15.16:3300/0 conn(0x7f7eb4135450 0x7f7eb41349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.729+0000 7f7eaaffd700 1 -- 172.21.15.16:0/2299311287 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615587572 0 0) 0x7f7eb0009580 con 0x7f7eb406dc50 2019-12-05T06:07:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eaaffd700 1 -- 172.21.15.16:0/2299311287 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e9c002e80 con 0x7f7eb406dc50 2019-12-05T06:07:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eb9dc6700 1 -- 172.21.15.16:0/2299311287 >> v2:172.21.15.16:3300/0 conn(0x7f7eb4135450 msgr2=0x7f7eb41349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> v2:172.21.15.16:3300/0 conn(0x7f7eb4135450 0x7f7eb41349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:04.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eb9dc6700 1 -- 172.21.15.16:0/2299311287 >> v1:172.21.15.16:6789/0 conn(0x7f7eb406dc50 legacy=0x7f7eb406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eb9dc6700 1 -- 172.21.15.16:0/2299311287 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4133b80 con 0x7f7eb406e840 2019-12-05T06:07:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> v2:172.21.15.76:3300/0 conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> v2:172.21.15.76:3300/0 conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=READY pgs=153 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eaaffd700 1 -- 172.21.15.16:0/2299311287 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ea4000a70 con 0x7f7eb406e840 2019-12-05T06:07:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7eaaffd700 1 -- 172.21.15.16:0/2299311287 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ea4009bf0 con 0x7f7eb406e840 2019-12-05T06:07:04.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.730+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb4133b80 con 0x7f7eb406e840 2019-12-05T06:07:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.731+0000 7f7eaaffd700 1 -- 172.21.15.16:0/2299311287 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ea4000a70 con 0x7f7eb406e840 2019-12-05T06:07:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.731+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb4133b80 con 0x7f7eb406e840 2019-12-05T06:07:04.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.732+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.732+0000 7f7eaaffd700 1 -- 172.21.15.16:0/2299311287 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7ea40096c0 con 0x7f7eb406e840 2019-12-05T06:07:04.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.732+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.732+0000 7f7eaaffd700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e9c023670 0x7f7e9c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:04.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.733+0000 7f7eaaffd700 1 -- 172.21.15.16:0/2299311287 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f7ea4045d80 con 0x7f7eb406e840 2019-12-05T06:07:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.733+0000 7f7eba5c7700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e9c023670 0x7f7e9c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.733+0000 7f7eba5c7700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e9c023670 0x7f7e9c025ae0 crc :-1 s=READY pgs=98 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:04.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.746+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb4133b80 con 0x7f7eb406e840 2019-12-05T06:07:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.755+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:04.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:04.755+0000 7f7eaaffd700 1 -- 172.21.15.16:0/2299311287 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7ea4000a50 con 0x7f7eb406e840 2019-12-05T06:07:05.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.008+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7eb4133b80 con 0x7f7eb406e840 2019-12-05T06:07:05.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.009+0000 7f7eb9dc6700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.009+0000 7f7eaaffd700 1 -- 172.21.15.16:0/2299311287 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f7ea40095e0 con 0x7f7eb406e840 2019-12-05T06:07:05.010 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2019-12-05T06:07:05.010 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:07:05.010 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:06:45.694298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:05.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.013+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e9c023670 msgr2=0x7f7e9c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.013+0000 7f7ebc02a700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e9c023670 0x7f7e9c025ae0 crc :-1 s=READY pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.013+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 msgr2=0x7f7eb4134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.013+0000 7f7ebc02a700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 0x7f7eb4134480 secure :-1 s=READY pgs=153 cs=0 l=1 rx=0x7f7ea40103a0 tx=0x7f7ea400b2e0).stop 2019-12-05T06:07:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.014+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 shutdown_connections 2019-12-05T06:07:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.014+0000 7f7ebc02a700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e9c023670 0x7f7e9c025ae0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.014+0000 7f7ebc02a700 1 --2- 172.21.15.16:0/2299311287 >> v2:172.21.15.16:3300/0 conn(0x7f7eb4135450 0x7f7eb41349c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.014+0000 7f7ebc02a700 1 --2- 172.21.15.16:0/2299311287 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7eb406e840 0x7f7eb4134480 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.014+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 shutdown_connections 2019-12-05T06:07:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.014+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 wait complete. 2019-12-05T06:07:05.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:05.014+0000 7f7ebc02a700 1 -- 172.21.15.16:0/2299311287 >> 172.21.15.16:0/2299311287 conn(0x7f7eb40a2850 msgr2=0x7f7eb406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:05.025 INFO:tasks.mds_thrash:Ready to start thrashing 2019-12-05T06:07:05.025 INFO:tasks.mds_thrash.ceph_manager:waiting for clean 2019-12-05T06:07:05.025 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T06:07:05.324 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.310+0000 7f50a8e47700 1 Processor -- start 2019-12-05T06:07:05.324 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.310+0000 7f50a8e47700 1 -- start start 2019-12-05T06:07:05.324 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.312+0000 7f50a8e47700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50a408c6d0 0x7f50a408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:05.325 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.312+0000 7f50a8e47700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f50a408f720 0x7f50a408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:05.325 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.312+0000 7f50a8e47700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a408fb80 con 0x7f50a408fef0 2019-12-05T06:07:05.325 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.312+0000 7f50a8e47700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a408df90 con 0x7f50a408c6d0 2019-12-05T06:07:05.325 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.312+0000 7f50a8e47700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a40a3f70 con 0x7f50a408f720 2019-12-05T06:07:05.325 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.313+0000 7f50a359e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f50a408f720 0x7f50a408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.325 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.313+0000 7f50a359e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f50a408f720 0x7f50a408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51410/0 (socket says 172.21.15.76:51410) 2019-12-05T06:07:05.325 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.313+0000 7f50a359e700 1 -- 172.21.15.76:0/2672058099 learned_addr learned my addr 172.21.15.76:0/2672058099 (peer_addr_for_me v2:172.21.15.76:0/0) 2019-12-05T06:07:05.326 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.313+0000 7f50a259c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f50a408fef0 0x7f50a408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34492/0 (socket says 172.21.15.76:34492) 2019-12-05T06:07:05.326 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.313+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/2672058099 >> v2:172.21.15.16:3300/0 conn(0x7f50a408c6d0 0x7f50a408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.326 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a359e700 1 -- 172.21.15.76:0/2672058099 >> v2:172.21.15.16:3300/0 conn(0x7f50a408c6d0 msgr2=0x7f50a408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:05.326 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a359e700 1 --2- 172.21.15.76:0/2672058099 >> v2:172.21.15.16:3300/0 conn(0x7f50a408c6d0 0x7f50a408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:05.326 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a359e700 1 -- 172.21.15.76:0/2672058099 >> v1:172.21.15.76:6789/0 conn(0x7f50a408fef0 legacy=0x7f50a408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.327 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a1d9b700 1 -- 172.21.15.76:0/2672058099 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293334774 0 0) 0x7f508c001480 con 0x7f50a408fef0 2019-12-05T06:07:05.327 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a359e700 1 -- 172.21.15.76:0/2672058099 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a408d540 con 0x7f50a408f720 2019-12-05T06:07:05.328 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a359e700 1 --2- 172.21.15.76:0/2672058099 >> v2:172.21.15.76:3300/0 conn(0x7f50a408f720 0x7f50a408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f509c00f6b0 tx=0x7f509c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.328 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a359e700 1 --2- 172.21.15.76:0/2672058099 >> v2:172.21.15.76:3300/0 conn(0x7f50a408f720 0x7f50a408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f509c00f6b0 tx=0x7f509c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.328 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a359e700 1 --2- 172.21.15.76:0/2672058099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a408f720 0x7f50a408d000 secure :-1 s=READY pgs=155 cs=0 l=1 rx=0x7f509c00f6b0 tx=0x7f509c00fa50).ready entity=mon.1 client_cookie=1590dbdbdc61795e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:05.328 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a359e700 1 --2- 172.21.15.76:0/2672058099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a408f720 0x7f50a408d000 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f509c00f6b0 tx=0x7f509c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.328 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a1d9b700 1 -- 172.21.15.76:0/2672058099 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f509c000a70 con 0x7f50a408f720 2019-12-05T06:07:05.328 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a359e700 1 --2- 172.21.15.76:0/2672058099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a408f720 0x7f50a408d000 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f509c00f6b0 tx=0x7f509c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.328 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.314+0000 7f50a359e700 1 --2- 172.21.15.76:0/2672058099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a408f720 0x7f50a408d000 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7f509c00f6b0 tx=0x7f509c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.329 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.315+0000 7f50a1d9b700 1 -- 172.21.15.76:0/2672058099 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c0010e0 con 0x7f50a408f720 2019-12-05T06:07:05.329 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.315+0000 7f50a1d9b700 1 -- 172.21.15.76:0/2672058099 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f509c0017b0 con 0x7f50a408f720 2019-12-05T06:07:05.329 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.315+0000 7f50a8e47700 1 -- 172.21.15.76:0/2672058099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a408f720 msgr2=0x7f50a408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.329 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.315+0000 7f50a8e47700 1 --2- 172.21.15.76:0/2672058099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a408f720 0x7f50a408d000 secure :-1 s=READY pgs=155 cs=0 l=1 rx=0x7f509c00f6b0 tx=0x7f509c00fa50).stop 2019-12-05T06:07:05.329 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.317+0000 7f50a8e47700 1 -- 172.21.15.76:0/2672058099 shutdown_connections 2019-12-05T06:07:05.329 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.317+0000 7f50a8e47700 1 --2- 172.21.15.76:0/2672058099 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50a408f720 0x7f50a408d000 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.329 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.317+0000 7f50a8e47700 1 --2- 172.21.15.76:0/2672058099 >> v2:172.21.15.16:3300/0 conn(0x7f50a408c6d0 0x7f50a408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:05.330 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.317+0000 7f50a8e47700 1 -- 172.21.15.76:0/2672058099 shutdown_connections 2019-12-05T06:07:05.330 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.317+0000 7f50a8e47700 1 -- 172.21.15.76:0/2672058099 wait complete. 2019-12-05T06:07:05.330 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.317+0000 7f50a8e47700 1 -- 172.21.15.76:0/2672058099 >> 172.21.15.76:0/2672058099 conn(0x7f50a4061940 msgr2=0x7f50a4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:05.330 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.322+0000 7f50a8e47700 1 Processor -- start 2019-12-05T06:07:05.330 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.322+0000 7f50a8e47700 1 -- start start 2019-12-05T06:07:05.330 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.323+0000 7f50a8e47700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50a4096e90 0x7f50a409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:05.330 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.323+0000 7f50a8e47700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a4084b90 con 0x7f50a4084f30 2019-12-05T06:07:05.331 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.323+0000 7f50a8e47700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a4098790 con 0x7f50a409b920 2019-12-05T06:07:05.331 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.323+0000 7f50a8e47700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a409a450 con 0x7f50a4096e90 2019-12-05T06:07:05.331 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.324+0000 7f50a359e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f50a409b920 0x7f50a4097a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34498/0 (socket says 172.21.15.76:34498) 2019-12-05T06:07:05.331 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.324+0000 7f50a359e700 1 -- 172.21.15.76:0/3188289868 learned_addr learned my addr 172.21.15.76:0/3188289868 (peer_addr_for_me v1:172.21.15.76:0/0) 2019-12-05T06:07:05.331 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.324+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/3188289868 >> v2:172.21.15.16:3300/0 conn(0x7f50a4096e90 0x7f50a409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.331 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.324+0000 7f509b7fe700 1 -- 172.21.15.76:0/3188289868 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 876398392 0 0) 0x7f509c0079e0 con 0x7f50a409b920 2019-12-05T06:07:05.331 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.324+0000 7f509b7fe700 1 -- 172.21.15.76:0/3188289868 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5088002e80 con 0x7f50a409b920 2019-12-05T06:07:05.332 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f509b7fe700 1 -- 172.21.15.76:0/3188289868 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785701472 0 0) 0x7f508c001480 con 0x7f50a4084f30 2019-12-05T06:07:05.332 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f50a2d9d700 1 -- 172.21.15.76:0/3188289868 >> v1:172.21.15.76:6789/0 conn(0x7f50a409b920 legacy=0x7f50a4097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.332 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f50a2d9d700 1 -- 172.21.15.76:0/3188289868 >> v1:172.21.15.16:6789/0 conn(0x7f50a4084f30 legacy=0x7f50a4097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.332 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f50a2d9d700 1 -- 172.21.15.76:0/3188289868 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a409aae0 con 0x7f50a4096e90 2019-12-05T06:07:05.332 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/3188289868 >> v2:172.21.15.16:3300/0 conn(0x7f50a4096e90 0x7f50a409b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5094010390 tx=0x7f5094010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.333 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/3188289868 >> v2:172.21.15.16:3300/0 conn(0x7f50a4096e90 0x7f50a409b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5094010390 tx=0x7f5094010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.333 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4096e90 0x7f50a409b3e0 secure :-1 s=READY pgs=185 cs=0 l=1 rx=0x7f5094010390 tx=0x7f5094010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:05.333 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4096e90 0x7f50a409b3e0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f5094010390 tx=0x7f5094010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.333 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f509b7fe700 1 -- 172.21.15.76:0/3188289868 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5094009640 con 0x7f50a4096e90 2019-12-05T06:07:05.333 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4096e90 0x7f50a409b3e0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f5094010390 tx=0x7f5094010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.333 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4096e90 0x7f50a409b3e0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f5094010390 tx=0x7f5094010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.333 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.325+0000 7f509b7fe700 1 -- 172.21.15.76:0/3188289868 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5094000940 con 0x7f50a4096e90 2019-12-05T06:07:05.334 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.326+0000 7f50a8e47700 1 -- 172.21.15.76:0/3188289868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a409aae0 con 0x7f50a4096e90 2019-12-05T06:07:05.334 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.326+0000 7f509b7fe700 1 -- 172.21.15.76:0/3188289868 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f509400a0e0 con 0x7f50a4096e90 2019-12-05T06:07:05.334 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.326+0000 7f50a8e47700 1 -- 172.21.15.76:0/3188289868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a409aae0 con 0x7f50a4096e90 2019-12-05T06:07:05.334 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.327+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4096e90 0x7f50a409b3e0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f5094010390 tx=0x7f5094010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.334 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.328+0000 7f509b7fe700 1 -- 172.21.15.76:0/3188289868 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5094009a70 con 0x7f50a4096e90 2019-12-05T06:07:05.334 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.328+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4096e90 0x7f50a409b3e0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f5094010390 tx=0x7f5094010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.334 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.328+0000 7f509b7fe700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5088023680 0x7f5088025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:05.334 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.328+0000 7f509b7fe700 1 -- 172.21.15.76:0/3188289868 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f5094045de0 con 0x7f50a4096e90 2019-12-05T06:07:05.335 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.328+0000 7f50a259c700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5088023680 0x7f5088025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.335 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.329+0000 7f50a259c700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5088023680 0x7f5088025af0 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:05.337 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.342+0000 7f50a8e47700 1 -- 172.21.15.76:0/3188289868 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a409aae0 con 0x7f50a4096e90 2019-12-05T06:07:05.347 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.352+0000 7f50a2d9d700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4096e90 0x7f50a409b3e0 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f5094010390 tx=0x7f5094010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.348 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.352+0000 7f509b7fe700 1 -- 172.21.15.76:0/3188289868 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5094009810 con 0x7f50a4096e90 2019-12-05T06:07:05.621 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.626+0000 7f50a8e47700 1 -- 172.21.15.76:0/3188289868 --> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f50a409aae0 con 0x7f5088023680 2019-12-05T06:07:05.633 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.634+0000 7f509b7fe700 1 -- 172.21.15.76:0/3188289868 <== mgr.4100 v2:172.21.15.76:6800/9546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80414 (crc 0 0 0) 0x7f508c001940 con 0x7f5088023680 2019-12-05T06:07:05.642 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:07:05.642 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2019-12-05T06:07:05.643 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.644+0000 7f50a8e47700 1 -- 172.21.15.76:0/3188289868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5088023680 msgr2=0x7f5088025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.643 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.644+0000 7f50a8e47700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5088023680 0x7f5088025af0 crc :-1 s=READY pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.643 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.644+0000 7f50a8e47700 1 -- 172.21.15.76:0/3188289868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4096e90 msgr2=0x7f50a409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.643 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.644+0000 7f50a8e47700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4096e90 0x7f50a409b3e0 secure :-1 s=READY pgs=185 cs=0 l=1 rx=0x7f5094010390 tx=0x7f5094010730).stop 2019-12-05T06:07:05.644 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.644+0000 7f50a8e47700 1 -- 172.21.15.76:0/3188289868 shutdown_connections 2019-12-05T06:07:05.644 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.644+0000 7f50a8e47700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5088023680 0x7f5088025af0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.644 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.644+0000 7f50a8e47700 1 --2- 172.21.15.76:0/3188289868 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4096e90 0x7f50a409b3e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.644 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.644+0000 7f50a8e47700 1 -- 172.21.15.76:0/3188289868 shutdown_connections 2019-12-05T06:07:05.644 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.644+0000 7f50a8e47700 1 -- 172.21.15.76:0/3188289868 wait complete. 2019-12-05T06:07:05.644 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.644+0000 7f50a8e47700 1 -- 172.21.15.76:0/3188289868 >> 172.21.15.76:0/3188289868 conn(0x7f50a40714f0 msgr2=0x7f50a409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:05.678 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2019-12-05T06:07:04.771530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390720,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584000,"statfs":{"total":773094113280,"available":764502016000,"internally_reserved":8589934592,"allocated":2097152,"data_stored":772252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003956"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.271382+0000","last_change":"2019-12-05T06:06:32.598114+0000","last_active":"2019-12-05T06:06:40.271382+0000","last_peered":"2019-12-05T06:06:40.271382+0000","last_clean":"2019-12-05T06:06:40.271382+0000","last_became_active":"2019-12-05T06:06:32.597387+0000","last_became_peered":"2019-12-05T06:06:32.597387+0000","last_unstale":"2019-12-05T06:06:40.271382+0000","last_undegraded":"2019-12-05T06:06:40.271382+0000","last_fullsized":"2019-12-05T06:06:40.271382+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"24'8","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.715381+0000","last_change":"2019-12-05T06:06:36.627910+0000","last_active":"2019-12-05T06:06:44.715381+0000","last_peered":"2019-12-05T06:06:44.715381+0000","last_clean":"2019-12-05T06:06:44.715381+0000","last_became_active":"2019-12-05T06:06:36.626897+0000","last_became_peered":"2019-12-05T06:06:36.626897+0000","last_unstale":"2019-12-05T06:06:44.715381+0000","last_undegraded":"2019-12-05T06:06:44.715381+0000","last_fullsized":"2019-12-05T06:06:44.715381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.490878+0000","last_change":"2019-12-05T06:06:40.490139+0000","last_active":"2019-12-05T06:06:40.490878+0000","last_peered":"2019-12-05T06:06:40.490878+0000","last_clean":"2019-12-05T06:06:40.490878+0000","last_became_active":"2019-12-05T06:06:40.488804+0000","last_became_peered":"2019-12-05T06:06:40.488804+0000","last_unstale":"2019-12-05T06:06:40.490878+0000","last_undegraded":"2019-12-05T06:06:40.490878+0000","last_fullsized":"2019-12-05T06:06:40.490878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.271284+0000","last_change":"2019-12-05T06:06:32.887464+0000","last_active":"2019-12-05T06:06:40.271284+0000","last_peered":"2019-12-05T06:06:40.271284+0000","last_clean":"2019-12-05T06:06:40.271284+0000","last_became_active":"2019-12-05T06:06:32.885955+0000","last_became_peered":"2019-12-05T06:06:32.885955+0000","last_unstale":"2019-12-05T06:06:40.271284+0000","last_undegraded":"2019-12-05T06:06:40.271284+0000","last_fullsized":"2019-12-05T06:06:40.271284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720231+0000","last_change":"2019-12-05T06:06:36.631805+0000","last_active":"2019-12-05T06:06:44.720231+0000","last_peered":"2019-12-05T06:06:44.720231+0000","last_clean":"2019-12-05T06:06:44.720231+0000","last_became_active":"2019-12-05T06:06:36.631077+0000","last_became_peered":"2019-12-05T06:06:36.631077+0000","last_unstale":"2019-12-05T06:06:44.720231+0000","last_undegraded":"2019-12-05T06:06:44.720231+0000","last_fullsized":"2019-12-05T06:06:44.720231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.485950+0000","last_change":"2019-12-05T06:06:40.485444+0000","last_active":"2019-12-05T06:06:40.485950+0000","last_peered":"2019-12-05T06:06:40.485950+0000","last_clean":"2019-12-05T06:06:40.485950+0000","last_became_active":"2019-12-05T06:06:40.484465+0000","last_became_peered":"2019-12-05T06:06:40.484465+0000","last_unstale":"2019-12-05T06:06:40.485950+0000","last_undegraded":"2019-12-05T06:06:40.485950+0000","last_fullsized":"2019-12-05T06:06:40.485950+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576769+0000","last_change":"2019-12-05T06:06:40.488691+0000","last_active":"2019-12-05T06:06:40.576769+0000","last_peered":"2019-12-05T06:06:40.576769+0000","last_clean":"2019-12-05T06:06:40.576769+0000","last_became_active":"2019-12-05T06:06:40.487506+0000","last_became_peered":"2019-12-05T06:06:40.487506+0000","last_unstale":"2019-12-05T06:06:40.576769+0000","last_undegraded":"2019-12-05T06:06:40.576769+0000","last_fullsized":"2019-12-05T06:06:40.576769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.268843+0000","last_change":"2019-12-05T06:06:32.597542+0000","last_active":"2019-12-05T06:06:40.268843+0000","last_peered":"2019-12-05T06:06:40.268843+0000","last_clean":"2019-12-05T06:06:40.268843+0000","last_became_active":"2019-12-05T06:06:32.596579+0000","last_became_peered":"2019-12-05T06:06:32.596579+0000","last_unstale":"2019-12-05T06:06:40.268843+0000","last_undegraded":"2019-12-05T06:06:40.268843+0000","last_fullsized":"2019-12-05T06:06:40.268843+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'4","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.710303+0000","last_change":"2019-12-05T06:06:36.626828+0000","last_active":"2019-12-05T06:06:44.710303+0000","last_peered":"2019-12-05T06:06:44.710303+0000","last_clean":"2019-12-05T06:06:44.710303+0000","last_became_active":"2019-12-05T06:06:36.626005+0000","last_became_peered":"2019-12-05T06:06:36.626005+0000","last_unstale":"2019-12-05T06:06:44.710303+0000","last_undegraded":"2019-12-05T06:06:44.710303+0000","last_fullsized":"2019-12-05T06:06:44.710303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.015035+0000","last_change":"2019-12-05T06:06:40.487696+0000","last_active":"2019-12-05T06:06:41.015035+0000","last_peered":"2019-12-05T06:06:41.015035+0000","last_clean":"2019-12-05T06:06:41.015035+0000","last_became_active":"2019-12-05T06:06:40.486899+0000","last_became_peered":"2019-12-05T06:06:40.486899+0000","last_unstale":"2019-12-05T06:06:41.015035+0000","last_undegraded":"2019-12-05T06:06:41.015035+0000","last_fullsized":"2019-12-05T06:06:41.015035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576888+0000","last_change":"2019-12-05T06:06:32.597098+0000","last_active":"2019-12-05T06:06:40.576888+0000","last_peered":"2019-12-05T06:06:40.576888+0000","last_clean":"2019-12-05T06:06:40.576888+0000","last_became_active":"2019-12-05T06:06:32.596276+0000","last_became_peered":"2019-12-05T06:06:32.596276+0000","last_unstale":"2019-12-05T06:06:40.576888+0000","last_undegraded":"2019-12-05T06:06:40.576888+0000","last_fullsized":"2019-12-05T06:06:40.576888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"24'7","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720981+0000","last_change":"2019-12-05T06:06:36.629032+0000","last_active":"2019-12-05T06:06:44.720981+0000","last_peered":"2019-12-05T06:06:44.720981+0000","last_clean":"2019-12-05T06:06:44.720981+0000","last_became_active":"2019-12-05T06:06:36.628405+0000","last_became_peered":"2019-12-05T06:06:36.628405+0000","last_unstale":"2019-12-05T06:06:44.720981+0000","last_undegraded":"2019-12-05T06:06:44.720981+0000","last_fullsized":"2019-12-05T06:06:44.720981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.084640+0000","last_change":"2019-12-05T06:06:32.886929+0000","last_active":"2019-12-05T06:06:41.084640+0000","last_peered":"2019-12-05T06:06:41.084640+0000","last_clean":"2019-12-05T06:06:41.084640+0000","last_became_active":"2019-12-05T06:06:32.886095+0000","last_became_peered":"2019-12-05T06:06:32.886095+0000","last_unstale":"2019-12-05T06:06:41.084640+0000","last_undegraded":"2019-12-05T06:06:41.084640+0000","last_fullsized":"2019-12-05T06:06:41.084640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"24'5","reported_seq":"17","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720647+0000","last_change":"2019-12-05T06:06:36.632241+0000","last_active":"2019-12-05T06:06:44.720647+0000","last_peered":"2019-12-05T06:06:44.720647+0000","last_clean":"2019-12-05T06:06:44.720647+0000","last_became_active":"2019-12-05T06:06:36.631418+0000","last_became_peered":"2019-12-05T06:06:36.631418+0000","last_unstale":"2019-12-05T06:06:44.720647+0000","last_undegraded":"2019-12-05T06:06:44.720647+0000","last_fullsized":"2019-12-05T06:06:44.720647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576879+0000","last_change":"2019-12-05T06:06:39.649718+0000","last_active":"2019-12-05T06:06:40.576879+0000","last_peered":"2019-12-05T06:06:40.576879+0000","last_clean":"2019-12-05T06:06:40.576879+0000","last_became_active":"2019-12-05T06:06:39.648680+0000","last_became_peered":"2019-12-05T06:06:39.648680+0000","last_unstale":"2019-12-05T06:06:40.576879+0000","last_undegraded":"2019-12-05T06:06:40.576879+0000","last_fullsized":"2019-12-05T06:06:40.576879+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.269507+0000","last_change":"2019-12-05T06:06:32.886427+0000","last_active":"2019-12-05T06:06:40.269507+0000","last_peered":"2019-12-05T06:06:40.269507+0000","last_clean":"2019-12-05T06:06:40.269507+0000","last_became_active":"2019-12-05T06:06:32.885509+0000","last_became_peered":"2019-12-05T06:06:32.885509+0000","last_unstale":"2019-12-05T06:06:40.269507+0000","last_undegraded":"2019-12-05T06:06:40.269507+0000","last_fullsized":"2019-12-05T06:06:40.269507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"24'1","reported_seq":"13","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:41.269608+0000","last_change":"2019-12-05T06:06:36.632232+0000","last_active":"2019-12-05T06:06:41.269608+0000","last_peered":"2019-12-05T06:06:41.269608+0000","last_clean":"2019-12-05T06:06:41.269608+0000","last_became_active":"2019-12-05T06:06:36.630785+0000","last_became_peered":"2019-12-05T06:06:36.630785+0000","last_unstale":"2019-12-05T06:06:41.269608+0000","last_undegraded":"2019-12-05T06:06:41.269608+0000","last_fullsized":"2019-12-05T06:06:41.269608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.268862+0000","last_change":"2019-12-05T06:06:39.647777+0000","last_active":"2019-12-05T06:06:40.268862+0000","last_peered":"2019-12-05T06:06:40.268862+0000","last_clean":"2019-12-05T06:06:40.268862+0000","last_became_active":"2019-12-05T06:06:39.646655+0000","last_became_peered":"2019-12-05T06:06:39.646655+0000","last_unstale":"2019-12-05T06:06:40.268862+0000","last_undegraded":"2019-12-05T06:06:40.268862+0000","last_fullsized":"2019-12-05T06:06:40.268862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.547093+0000","last_change":"2019-12-05T06:06:36.628444+0000","last_active":"2019-12-05T06:06:40.547093+0000","last_peered":"2019-12-05T06:06:40.547093+0000","last_clean":"2019-12-05T06:06:40.547093+0000","last_became_active":"2019-12-05T06:06:36.627555+0000","last_became_peered":"2019-12-05T06:06:36.627555+0000","last_unstale":"2019-12-05T06:06:40.547093+0000","last_undegraded":"2019-12-05T06:06:40.547093+0000","last_fullsized":"2019-12-05T06:06:40.547093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.084584+0000","last_change":"2019-12-05T06:06:32.886895+0000","last_active":"2019-12-05T06:06:41.084584+0000","last_peered":"2019-12-05T06:06:41.084584+0000","last_clean":"2019-12-05T06:06:41.084584+0000","last_became_active":"2019-12-05T06:06:32.886123+0000","last_became_peered":"2019-12-05T06:06:32.886123+0000","last_unstale":"2019-12-05T06:06:41.084584+0000","last_undegraded":"2019-12-05T06:06:41.084584+0000","last_fullsized":"2019-12-05T06:06:41.084584+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.486413+0000","last_change":"2019-12-05T06:06:40.486073+0000","last_active":"2019-12-05T06:06:40.486413+0000","last_peered":"2019-12-05T06:06:40.486413+0000","last_clean":"2019-12-05T06:06:40.486413+0000","last_became_active":"2019-12-05T06:06:40.484954+0000","last_became_peered":"2019-12-05T06:06:40.484954+0000","last_unstale":"2019-12-05T06:06:40.486413+0000","last_undegraded":"2019-12-05T06:06:40.486413+0000","last_fullsized":"2019-12-05T06:06:40.486413+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.723134+0000","last_change":"2019-12-05T06:06:36.630520+0000","last_active":"2019-12-05T06:06:44.723134+0000","last_peered":"2019-12-05T06:06:44.723134+0000","last_clean":"2019-12-05T06:06:44.723134+0000","last_became_active":"2019-12-05T06:06:36.629682+0000","last_became_peered":"2019-12-05T06:06:36.629682+0000","last_unstale":"2019-12-05T06:06:44.723134+0000","last_undegraded":"2019-12-05T06:06:44.723134+0000","last_fullsized":"2019-12-05T06:06:44.723134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.484464+0000","last_change":"2019-12-05T06:06:32.595548+0000","last_active":"2019-12-05T06:06:40.484464+0000","last_peered":"2019-12-05T06:06:40.484464+0000","last_clean":"2019-12-05T06:06:40.484464+0000","last_became_active":"2019-12-05T06:06:32.594119+0000","last_became_peered":"2019-12-05T06:06:32.594119+0000","last_unstale":"2019-12-05T06:06:40.484464+0000","last_undegraded":"2019-12-05T06:06:40.484464+0000","last_fullsized":"2019-12-05T06:06:40.484464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.014770+0000","last_change":"2019-12-05T06:06:39.649718+0000","last_active":"2019-12-05T06:06:41.014770+0000","last_peered":"2019-12-05T06:06:41.014770+0000","last_clean":"2019-12-05T06:06:41.014770+0000","last_became_active":"2019-12-05T06:06:39.648278+0000","last_became_peered":"2019-12-05T06:06:39.648278+0000","last_unstale":"2019-12-05T06:06:41.014770+0000","last_undegraded":"2019-12-05T06:06:41.014770+0000","last_fullsized":"2019-12-05T06:06:41.014770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":96686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.758}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.188}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.835}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.052}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.642}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.228}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.789}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":96526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.639}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.918}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.887}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.811}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.049}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.142}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.090}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.562}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.734}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.846}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.902}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.795}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.056}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.126}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.763}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.516}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.980}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.106}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.808}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.049}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.187}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":97646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.703}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.589}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.532}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.942}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.099}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.821}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.017}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":97486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.722}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.685}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.814}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.878}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.234}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.028}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.283}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.075}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.843}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.586}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.132}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.758}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.346}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.300}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.862}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.632}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.662}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.843}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.974}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.942}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.075}]}]}]}} 2019-12-05T06:07:05.688 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T06:07:05.950 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.946+0000 7f573990e700 1 Processor -- start 2019-12-05T06:07:05.951 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.946+0000 7f573990e700 1 -- start start 2019-12-05T06:07:05.951 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.947+0000 7f573990e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 0x7f573408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:05.951 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.947+0000 7f573990e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57340958e0 con 0x7f573408c6d0 2019-12-05T06:07:05.951 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.947+0000 7f573990e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57340629e0 con 0x7f573408d2c0 2019-12-05T06:07:05.952 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.947+0000 7f573990e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f573408df90 con 0x7f573408fef0 2019-12-05T06:07:05.952 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.948+0000 7f5732ffd700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f573408d2c0 0x7f573408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34504/0 (socket says 172.21.15.76:34504) 2019-12-05T06:07:05.952 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.948+0000 7f5733fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 0x7f573408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.953 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.948+0000 7f5732ffd700 1 -- 172.21.15.76:0/3849318902 learned_addr learned my addr 172.21.15.76:0/3849318902 (peer_addr_for_me v1:172.21.15.76:0/0) 2019-12-05T06:07:05.953 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.948+0000 7f57327fc700 1 -- 172.21.15.76:0/3849318902 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 621772751 0 0) 0x7f571c001150 con 0x7f573408d2c0 2019-12-05T06:07:05.953 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.948+0000 7f57327fc700 1 -- 172.21.15.76:0/3849318902 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5720002e80 con 0x7f573408d2c0 2019-12-05T06:07:05.953 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.948+0000 7f57327fc700 1 -- 172.21.15.76:0/3849318902 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608787715 0 0) 0x7f572c001400 con 0x7f573408c6d0 2019-12-05T06:07:05.953 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.948+0000 7f5733fff700 1 -- 172.21.15.76:0/3849318902 >> v1:172.21.15.76:6789/0 conn(0x7f573408d2c0 legacy=0x7f573408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.953 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.948+0000 7f5733fff700 1 -- 172.21.15.76:0/3849318902 >> v1:172.21.15.16:6789/0 conn(0x7f573408c6d0 legacy=0x7f573408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.954 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.949+0000 7f5733fff700 1 -- 172.21.15.76:0/3849318902 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573408e620 con 0x7f573408fef0 2019-12-05T06:07:05.954 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.949+0000 7f5733fff700 1 --2- 172.21.15.76:0/3849318902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 0x7f573408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f572400af50 tx=0x7f572400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.954 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.949+0000 7f5733fff700 1 --2- 172.21.15.76:0/3849318902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 0x7f573408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f572400af50 tx=0x7f572400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.954 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.949+0000 7f5733fff700 1 --2- 172.21.15.76:0/3849318902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 0x7f573408f720 secure :-1 s=READY pgs=50 cs=0 l=1 rx=0x7f572400af50 tx=0x7f572400b760).ready entity=mon.2 client_cookie=569531715f6ebeb8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:05.954 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.949+0000 7f5733fff700 1 --2- 172.21.15.76:0/3849318902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 0x7f573408f720 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f572400af50 tx=0x7f572400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.954 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.949+0000 7f57327fc700 1 -- 172.21.15.76:0/3849318902 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5724009ba0 con 0x7f573408fef0 2019-12-05T06:07:05.955 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.949+0000 7f5733fff700 1 --2- 172.21.15.76:0/3849318902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 0x7f573408f720 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f572400af50 tx=0x7f572400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.955 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.949+0000 7f5733fff700 1 --2- 172.21.15.76:0/3849318902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 0x7f573408f720 secure :-1 s=THROTTLE_DONE pgs=50 cs=0 l=1 rx=0x7f572400af50 tx=0x7f572400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.955 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.949+0000 7f57327fc700 1 -- 172.21.15.76:0/3849318902 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5724000a70 con 0x7f573408fef0 2019-12-05T06:07:05.955 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.949+0000 7f57327fc700 1 -- 172.21.15.76:0/3849318902 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f572400a180 con 0x7f573408fef0 2019-12-05T06:07:05.955 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.950+0000 7f573990e700 1 -- 172.21.15.76:0/3849318902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 msgr2=0x7f573408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.955 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.950+0000 7f573990e700 1 --2- 172.21.15.76:0/3849318902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 0x7f573408f720 secure :-1 s=READY pgs=50 cs=0 l=1 rx=0x7f572400af50 tx=0x7f572400b760).stop 2019-12-05T06:07:05.955 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.951+0000 7f573990e700 1 -- 172.21.15.76:0/3849318902 shutdown_connections 2019-12-05T06:07:05.956 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.951+0000 7f573990e700 1 --2- 172.21.15.76:0/3849318902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f573408fef0 0x7f573408f720 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.956 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.951+0000 7f573990e700 1 -- 172.21.15.76:0/3849318902 shutdown_connections 2019-12-05T06:07:05.956 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.951+0000 7f573990e700 1 -- 172.21.15.76:0/3849318902 wait complete. 2019-12-05T06:07:05.956 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.951+0000 7f573990e700 1 -- 172.21.15.76:0/3849318902 >> 172.21.15.76:0/3849318902 conn(0x7f5734061940 msgr2=0x7f5734096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:05.960 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.956+0000 7f573990e700 1 Processor -- start 2019-12-05T06:07:05.960 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.956+0000 7f573990e700 1 -- start start 2019-12-05T06:07:05.960 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.958+0000 7f573990e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f573409e7c0 0x7f573414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:05.960 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.958+0000 7f573990e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f573409dff0 0x7f5734159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:05.960 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.958+0000 7f573990e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f573409e430 con 0x7f573414f630 2019-12-05T06:07:05.961 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.958+0000 7f573990e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f573412cd70 con 0x7f573409dff0 2019-12-05T06:07:05.961 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.958+0000 7f573990e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5734131c20 con 0x7f573409e7c0 2019-12-05T06:07:05.961 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.958+0000 7f5732ffd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f573409e7c0 0x7f573414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.961 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.958+0000 7f5732ffd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f573409e7c0 0x7f573414fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51430/0 (socket says 172.21.15.76:51430) 2019-12-05T06:07:05.961 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.958+0000 7f5732ffd700 1 -- 172.21.15.76:0/2335294801 learned_addr learned my addr 172.21.15.76:0/2335294801 (peer_addr_for_me v2:172.21.15.76:0/0) 2019-12-05T06:07:05.961 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.958+0000 7f5733fff700 1 --2- 172.21.15.76:0/2335294801 >> v2:172.21.15.16:3300/0 conn(0x7f573409dff0 0x7f5734159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.962 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5732ffd700 1 -- 172.21.15.76:0/2335294801 >> v2:172.21.15.16:3300/0 conn(0x7f573409dff0 msgr2=0x7f5734159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.962 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5732ffd700 1 --2- 172.21.15.76:0/2335294801 >> v2:172.21.15.16:3300/0 conn(0x7f573409dff0 0x7f5734159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:05.962 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5732ffd700 1 -- 172.21.15.76:0/2335294801 >> v1:172.21.15.16:6789/0 conn(0x7f573414f630 legacy=0x7f573409d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:05.962 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f572bfff700 1 -- 172.21.15.76:0/2335294801 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208112608 0 0) 0x7f572c001400 con 0x7f573414f630 2019-12-05T06:07:05.962 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5732ffd700 1 -- 172.21.15.76:0/2335294801 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57341322b0 con 0x7f573409e7c0 2019-12-05T06:07:05.963 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5733fff700 1 --2- 172.21.15.76:0/2335294801 >> v2:172.21.15.16:3300/0 conn(0x7f573409dff0 0x7f5734159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:07:05.963 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5732ffd700 1 --2- 172.21.15.76:0/2335294801 >> v2:172.21.15.76:3300/0 conn(0x7f573409e7c0 0x7f573414fa20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f571c0103e0 tx=0x7f571c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.963 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5732ffd700 1 --2- 172.21.15.76:0/2335294801 >> v2:172.21.15.76:3300/0 conn(0x7f573409e7c0 0x7f573414fa20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f571c0103e0 tx=0x7f571c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.963 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5732ffd700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f573409e7c0 0x7f573414fa20 secure :-1 s=READY pgs=158 cs=0 l=1 rx=0x7f571c0103e0 tx=0x7f571c010780).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:05.963 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5732ffd700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f573409e7c0 0x7f573414fa20 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f571c0103e0 tx=0x7f571c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.963 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f572bfff700 1 -- 172.21.15.76:0/2335294801 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571c000c90 con 0x7f573409e7c0 2019-12-05T06:07:05.964 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5732ffd700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f573409e7c0 0x7f573414fa20 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f571c0103e0 tx=0x7f571c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.964 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f572bfff700 1 -- 172.21.15.76:0/2335294801 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f571c0010c0 con 0x7f573409e7c0 2019-12-05T06:07:05.964 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.959+0000 7f5732ffd700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f573409e7c0 0x7f573414fa20 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f571c0103e0 tx=0x7f571c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.964 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.960+0000 7f572bfff700 1 -- 172.21.15.76:0/2335294801 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571c0012b0 con 0x7f573409e7c0 2019-12-05T06:07:05.964 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.960+0000 7f573990e700 1 -- 172.21.15.76:0/2335294801 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57341322b0 con 0x7f573409e7c0 2019-12-05T06:07:05.964 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.960+0000 7f573990e700 1 -- 172.21.15.76:0/2335294801 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57341322b0 con 0x7f573409e7c0 2019-12-05T06:07:05.964 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.961+0000 7f5732ffd700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f573409e7c0 0x7f573414fa20 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f571c0103e0 tx=0x7f571c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.965 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.961+0000 7f572bfff700 1 -- 172.21.15.76:0/2335294801 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f571c000c90 con 0x7f573409e7c0 2019-12-05T06:07:05.965 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.961+0000 7f5732ffd700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f573409e7c0 0x7f573414fa20 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f571c0103e0 tx=0x7f571c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.965 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.961+0000 7f572bfff700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5718023630 0x7f5718025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:05.965 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.962+0000 7f572bfff700 1 -- 172.21.15.76:0/2335294801 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f571c045dc0 con 0x7f573409e7c0 2019-12-05T06:07:05.965 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.962+0000 7f57337fe700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5718023630 0x7f5718025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:05.965 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.962+0000 7f57337fe700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5718023630 0x7f5718025ac0 crc :-1 s=READY pgs=100 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:05.973 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.977+0000 7f573990e700 1 -- 172.21.15.76:0/2335294801 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57341322b0 con 0x7f573409e7c0 2019-12-05T06:07:05.980 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.985+0000 7f5732ffd700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f573409e7c0 0x7f573414fa20 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f571c0103e0 tx=0x7f571c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:05.980 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:05.985+0000 7f572bfff700 1 -- 172.21.15.76:0/2335294801 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f571c0075a0 con 0x7f573409e7c0 2019-12-05T06:07:06.253 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.257+0000 7f573990e700 1 -- 172.21.15.76:0/2335294801 --> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f573412fff0 con 0x7f5718023630 2019-12-05T06:07:06.260 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.265+0000 7f572bfff700 1 -- 172.21.15.76:0/2335294801 <== mgr.4100 v2:172.21.15.76:6800/9546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80414 (crc 0 0 0) 0x7f572c001220 con 0x7f5718023630 2019-12-05T06:07:06.269 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:07:06.270 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2019-12-05T06:07:06.271 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.275+0000 7f573990e700 1 -- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5718023630 msgr2=0x7f5718025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:06.271 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.275+0000 7f573990e700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5718023630 0x7f5718025ac0 crc :-1 s=READY pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:06.271 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.275+0000 7f573990e700 1 -- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f573409e7c0 msgr2=0x7f573414fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:06.271 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.275+0000 7f573990e700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f573409e7c0 0x7f573414fa20 secure :-1 s=READY pgs=158 cs=0 l=1 rx=0x7f571c0103e0 tx=0x7f571c010780).stop 2019-12-05T06:07:06.272 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.275+0000 7f573990e700 1 -- 172.21.15.76:0/2335294801 shutdown_connections 2019-12-05T06:07:06.272 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.275+0000 7f573990e700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5718023630 0x7f5718025ac0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:06.272 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.275+0000 7f573990e700 1 --2- 172.21.15.76:0/2335294801 >> v2:172.21.15.16:3300/0 conn(0x7f573409dff0 0x7f5734159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:06.272 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.275+0000 7f573990e700 1 --2- 172.21.15.76:0/2335294801 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f573409e7c0 0x7f573414fa20 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:06.274 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.276+0000 7f573990e700 1 -- 172.21.15.76:0/2335294801 shutdown_connections 2019-12-05T06:07:06.274 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.276+0000 7f573990e700 1 -- 172.21.15.76:0/2335294801 wait complete. 2019-12-05T06:07:06.274 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.276+0000 7f573990e700 1 -- 172.21.15.76:0/2335294801 >> 172.21.15.76:0/2335294801 conn(0x7f57340a2850 msgr2=0x7f573406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:06.286 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2019-12-05T06:07:04.771530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390720,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584000,"statfs":{"total":773094113280,"available":764502016000,"internally_reserved":8589934592,"allocated":2097152,"data_stored":772252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003956"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.271382+0000","last_change":"2019-12-05T06:06:32.598114+0000","last_active":"2019-12-05T06:06:40.271382+0000","last_peered":"2019-12-05T06:06:40.271382+0000","last_clean":"2019-12-05T06:06:40.271382+0000","last_became_active":"2019-12-05T06:06:32.597387+0000","last_became_peered":"2019-12-05T06:06:32.597387+0000","last_unstale":"2019-12-05T06:06:40.271382+0000","last_undegraded":"2019-12-05T06:06:40.271382+0000","last_fullsized":"2019-12-05T06:06:40.271382+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"24'8","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.715381+0000","last_change":"2019-12-05T06:06:36.627910+0000","last_active":"2019-12-05T06:06:44.715381+0000","last_peered":"2019-12-05T06:06:44.715381+0000","last_clean":"2019-12-05T06:06:44.715381+0000","last_became_active":"2019-12-05T06:06:36.626897+0000","last_became_peered":"2019-12-05T06:06:36.626897+0000","last_unstale":"2019-12-05T06:06:44.715381+0000","last_undegraded":"2019-12-05T06:06:44.715381+0000","last_fullsized":"2019-12-05T06:06:44.715381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.490878+0000","last_change":"2019-12-05T06:06:40.490139+0000","last_active":"2019-12-05T06:06:40.490878+0000","last_peered":"2019-12-05T06:06:40.490878+0000","last_clean":"2019-12-05T06:06:40.490878+0000","last_became_active":"2019-12-05T06:06:40.488804+0000","last_became_peered":"2019-12-05T06:06:40.488804+0000","last_unstale":"2019-12-05T06:06:40.490878+0000","last_undegraded":"2019-12-05T06:06:40.490878+0000","last_fullsized":"2019-12-05T06:06:40.490878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.271284+0000","last_change":"2019-12-05T06:06:32.887464+0000","last_active":"2019-12-05T06:06:40.271284+0000","last_peered":"2019-12-05T06:06:40.271284+0000","last_clean":"2019-12-05T06:06:40.271284+0000","last_became_active":"2019-12-05T06:06:32.885955+0000","last_became_peered":"2019-12-05T06:06:32.885955+0000","last_unstale":"2019-12-05T06:06:40.271284+0000","last_undegraded":"2019-12-05T06:06:40.271284+0000","last_fullsized":"2019-12-05T06:06:40.271284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720231+0000","last_change":"2019-12-05T06:06:36.631805+0000","last_active":"2019-12-05T06:06:44.720231+0000","last_peered":"2019-12-05T06:06:44.720231+0000","last_clean":"2019-12-05T06:06:44.720231+0000","last_became_active":"2019-12-05T06:06:36.631077+0000","last_became_peered":"2019-12-05T06:06:36.631077+0000","last_unstale":"2019-12-05T06:06:44.720231+0000","last_undegraded":"2019-12-05T06:06:44.720231+0000","last_fullsized":"2019-12-05T06:06:44.720231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.485950+0000","last_change":"2019-12-05T06:06:40.485444+0000","last_active":"2019-12-05T06:06:40.485950+0000","last_peered":"2019-12-05T06:06:40.485950+0000","last_clean":"2019-12-05T06:06:40.485950+0000","last_became_active":"2019-12-05T06:06:40.484465+0000","last_became_peered":"2019-12-05T06:06:40.484465+0000","last_unstale":"2019-12-05T06:06:40.485950+0000","last_undegraded":"2019-12-05T06:06:40.485950+0000","last_fullsized":"2019-12-05T06:06:40.485950+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576769+0000","last_change":"2019-12-05T06:06:40.488691+0000","last_active":"2019-12-05T06:06:40.576769+0000","last_peered":"2019-12-05T06:06:40.576769+0000","last_clean":"2019-12-05T06:06:40.576769+0000","last_became_active":"2019-12-05T06:06:40.487506+0000","last_became_peered":"2019-12-05T06:06:40.487506+0000","last_unstale":"2019-12-05T06:06:40.576769+0000","last_undegraded":"2019-12-05T06:06:40.576769+0000","last_fullsized":"2019-12-05T06:06:40.576769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.268843+0000","last_change":"2019-12-05T06:06:32.597542+0000","last_active":"2019-12-05T06:06:40.268843+0000","last_peered":"2019-12-05T06:06:40.268843+0000","last_clean":"2019-12-05T06:06:40.268843+0000","last_became_active":"2019-12-05T06:06:32.596579+0000","last_became_peered":"2019-12-05T06:06:32.596579+0000","last_unstale":"2019-12-05T06:06:40.268843+0000","last_undegraded":"2019-12-05T06:06:40.268843+0000","last_fullsized":"2019-12-05T06:06:40.268843+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'4","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.710303+0000","last_change":"2019-12-05T06:06:36.626828+0000","last_active":"2019-12-05T06:06:44.710303+0000","last_peered":"2019-12-05T06:06:44.710303+0000","last_clean":"2019-12-05T06:06:44.710303+0000","last_became_active":"2019-12-05T06:06:36.626005+0000","last_became_peered":"2019-12-05T06:06:36.626005+0000","last_unstale":"2019-12-05T06:06:44.710303+0000","last_undegraded":"2019-12-05T06:06:44.710303+0000","last_fullsized":"2019-12-05T06:06:44.710303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.015035+0000","last_change":"2019-12-05T06:06:40.487696+0000","last_active":"2019-12-05T06:06:41.015035+0000","last_peered":"2019-12-05T06:06:41.015035+0000","last_clean":"2019-12-05T06:06:41.015035+0000","last_became_active":"2019-12-05T06:06:40.486899+0000","last_became_peered":"2019-12-05T06:06:40.486899+0000","last_unstale":"2019-12-05T06:06:41.015035+0000","last_undegraded":"2019-12-05T06:06:41.015035+0000","last_fullsized":"2019-12-05T06:06:41.015035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576888+0000","last_change":"2019-12-05T06:06:32.597098+0000","last_active":"2019-12-05T06:06:40.576888+0000","last_peered":"2019-12-05T06:06:40.576888+0000","last_clean":"2019-12-05T06:06:40.576888+0000","last_became_active":"2019-12-05T06:06:32.596276+0000","last_became_peered":"2019-12-05T06:06:32.596276+0000","last_unstale":"2019-12-05T06:06:40.576888+0000","last_undegraded":"2019-12-05T06:06:40.576888+0000","last_fullsized":"2019-12-05T06:06:40.576888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"24'7","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720981+0000","last_change":"2019-12-05T06:06:36.629032+0000","last_active":"2019-12-05T06:06:44.720981+0000","last_peered":"2019-12-05T06:06:44.720981+0000","last_clean":"2019-12-05T06:06:44.720981+0000","last_became_active":"2019-12-05T06:06:36.628405+0000","last_became_peered":"2019-12-05T06:06:36.628405+0000","last_unstale":"2019-12-05T06:06:44.720981+0000","last_undegraded":"2019-12-05T06:06:44.720981+0000","last_fullsized":"2019-12-05T06:06:44.720981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.084640+0000","last_change":"2019-12-05T06:06:32.886929+0000","last_active":"2019-12-05T06:06:41.084640+0000","last_peered":"2019-12-05T06:06:41.084640+0000","last_clean":"2019-12-05T06:06:41.084640+0000","last_became_active":"2019-12-05T06:06:32.886095+0000","last_became_peered":"2019-12-05T06:06:32.886095+0000","last_unstale":"2019-12-05T06:06:41.084640+0000","last_undegraded":"2019-12-05T06:06:41.084640+0000","last_fullsized":"2019-12-05T06:06:41.084640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"24'5","reported_seq":"17","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720647+0000","last_change":"2019-12-05T06:06:36.632241+0000","last_active":"2019-12-05T06:06:44.720647+0000","last_peered":"2019-12-05T06:06:44.720647+0000","last_clean":"2019-12-05T06:06:44.720647+0000","last_became_active":"2019-12-05T06:06:36.631418+0000","last_became_peered":"2019-12-05T06:06:36.631418+0000","last_unstale":"2019-12-05T06:06:44.720647+0000","last_undegraded":"2019-12-05T06:06:44.720647+0000","last_fullsized":"2019-12-05T06:06:44.720647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576879+0000","last_change":"2019-12-05T06:06:39.649718+0000","last_active":"2019-12-05T06:06:40.576879+0000","last_peered":"2019-12-05T06:06:40.576879+0000","last_clean":"2019-12-05T06:06:40.576879+0000","last_became_active":"2019-12-05T06:06:39.648680+0000","last_became_peered":"2019-12-05T06:06:39.648680+0000","last_unstale":"2019-12-05T06:06:40.576879+0000","last_undegraded":"2019-12-05T06:06:40.576879+0000","last_fullsized":"2019-12-05T06:06:40.576879+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.269507+0000","last_change":"2019-12-05T06:06:32.886427+0000","last_active":"2019-12-05T06:06:40.269507+0000","last_peered":"2019-12-05T06:06:40.269507+0000","last_clean":"2019-12-05T06:06:40.269507+0000","last_became_active":"2019-12-05T06:06:32.885509+0000","last_became_peered":"2019-12-05T06:06:32.885509+0000","last_unstale":"2019-12-05T06:06:40.269507+0000","last_undegraded":"2019-12-05T06:06:40.269507+0000","last_fullsized":"2019-12-05T06:06:40.269507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"24'1","reported_seq":"13","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:41.269608+0000","last_change":"2019-12-05T06:06:36.632232+0000","last_active":"2019-12-05T06:06:41.269608+0000","last_peered":"2019-12-05T06:06:41.269608+0000","last_clean":"2019-12-05T06:06:41.269608+0000","last_became_active":"2019-12-05T06:06:36.630785+0000","last_became_peered":"2019-12-05T06:06:36.630785+0000","last_unstale":"2019-12-05T06:06:41.269608+0000","last_undegraded":"2019-12-05T06:06:41.269608+0000","last_fullsized":"2019-12-05T06:06:41.269608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.268862+0000","last_change":"2019-12-05T06:06:39.647777+0000","last_active":"2019-12-05T06:06:40.268862+0000","last_peered":"2019-12-05T06:06:40.268862+0000","last_clean":"2019-12-05T06:06:40.268862+0000","last_became_active":"2019-12-05T06:06:39.646655+0000","last_became_peered":"2019-12-05T06:06:39.646655+0000","last_unstale":"2019-12-05T06:06:40.268862+0000","last_undegraded":"2019-12-05T06:06:40.268862+0000","last_fullsized":"2019-12-05T06:06:40.268862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.547093+0000","last_change":"2019-12-05T06:06:36.628444+0000","last_active":"2019-12-05T06:06:40.547093+0000","last_peered":"2019-12-05T06:06:40.547093+0000","last_clean":"2019-12-05T06:06:40.547093+0000","last_became_active":"2019-12-05T06:06:36.627555+0000","last_became_peered":"2019-12-05T06:06:36.627555+0000","last_unstale":"2019-12-05T06:06:40.547093+0000","last_undegraded":"2019-12-05T06:06:40.547093+0000","last_fullsized":"2019-12-05T06:06:40.547093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.084584+0000","last_change":"2019-12-05T06:06:32.886895+0000","last_active":"2019-12-05T06:06:41.084584+0000","last_peered":"2019-12-05T06:06:41.084584+0000","last_clean":"2019-12-05T06:06:41.084584+0000","last_became_active":"2019-12-05T06:06:32.886123+0000","last_became_peered":"2019-12-05T06:06:32.886123+0000","last_unstale":"2019-12-05T06:06:41.084584+0000","last_undegraded":"2019-12-05T06:06:41.084584+0000","last_fullsized":"2019-12-05T06:06:41.084584+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.486413+0000","last_change":"2019-12-05T06:06:40.486073+0000","last_active":"2019-12-05T06:06:40.486413+0000","last_peered":"2019-12-05T06:06:40.486413+0000","last_clean":"2019-12-05T06:06:40.486413+0000","last_became_active":"2019-12-05T06:06:40.484954+0000","last_became_peered":"2019-12-05T06:06:40.484954+0000","last_unstale":"2019-12-05T06:06:40.486413+0000","last_undegraded":"2019-12-05T06:06:40.486413+0000","last_fullsized":"2019-12-05T06:06:40.486413+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.723134+0000","last_change":"2019-12-05T06:06:36.630520+0000","last_active":"2019-12-05T06:06:44.723134+0000","last_peered":"2019-12-05T06:06:44.723134+0000","last_clean":"2019-12-05T06:06:44.723134+0000","last_became_active":"2019-12-05T06:06:36.629682+0000","last_became_peered":"2019-12-05T06:06:36.629682+0000","last_unstale":"2019-12-05T06:06:44.723134+0000","last_undegraded":"2019-12-05T06:06:44.723134+0000","last_fullsized":"2019-12-05T06:06:44.723134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.484464+0000","last_change":"2019-12-05T06:06:32.595548+0000","last_active":"2019-12-05T06:06:40.484464+0000","last_peered":"2019-12-05T06:06:40.484464+0000","last_clean":"2019-12-05T06:06:40.484464+0000","last_became_active":"2019-12-05T06:06:32.594119+0000","last_became_peered":"2019-12-05T06:06:32.594119+0000","last_unstale":"2019-12-05T06:06:40.484464+0000","last_undegraded":"2019-12-05T06:06:40.484464+0000","last_fullsized":"2019-12-05T06:06:40.484464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.014770+0000","last_change":"2019-12-05T06:06:39.649718+0000","last_active":"2019-12-05T06:06:41.014770+0000","last_peered":"2019-12-05T06:06:41.014770+0000","last_clean":"2019-12-05T06:06:41.014770+0000","last_became_active":"2019-12-05T06:06:39.648278+0000","last_became_peered":"2019-12-05T06:06:39.648278+0000","last_unstale":"2019-12-05T06:06:41.014770+0000","last_undegraded":"2019-12-05T06:06:41.014770+0000","last_fullsized":"2019-12-05T06:06:41.014770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":96686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.758}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.188}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.835}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.052}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.642}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.228}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.789}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":96526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.639}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.918}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.887}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.811}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.049}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.142}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.090}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.562}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.734}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.846}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.902}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.795}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.056}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.126}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.763}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.516}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.980}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.106}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.808}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.049}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.187}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":97646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.703}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.589}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.532}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.942}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.099}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.821}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.017}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":97486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.722}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.685}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.814}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.878}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.234}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.028}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.283}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.075}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.843}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.586}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.132}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.758}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.346}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.300}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.862}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.632}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.662}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.843}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.974}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.942}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.075}]}]}]}} 2019-12-05T06:07:06.290 INFO:tasks.mds_thrash.ceph_manager:clean! 2019-12-05T06:07:06.290 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2019-12-05T06:07:06.570 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.554+0000 7f955b475700 1 Processor -- start 2019-12-05T06:07:06.570 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.554+0000 7f955b475700 1 -- start start 2019-12-05T06:07:06.570 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.556+0000 7f955b475700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 0x7f955408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:06.570 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.556+0000 7f955b475700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95540958e0 con 0x7f955408fef0 2019-12-05T06:07:06.571 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.556+0000 7f955b475700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95540629e0 con 0x7f955408c6d0 2019-12-05T06:07:06.571 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.556+0000 7f955b475700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f955408df90 con 0x7f955408d2c0 2019-12-05T06:07:06.571 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.556+0000 7f9559a12700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f955408c6d0 0x7f955408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.76:34518/0 (socket says 172.21.15.76:34518) 2019-12-05T06:07:06.571 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.556+0000 7f9559211700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 0x7f955408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:06.571 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.556+0000 7f9559a12700 1 -- 172.21.15.76:0/2013188202 learned_addr learned my addr 172.21.15.76:0/2013188202 (peer_addr_for_me v1:172.21.15.76:0/0) 2019-12-05T06:07:06.571 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.557+0000 7f9558a10700 1 -- 172.21.15.76:0/2013188202 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 575648711 0 0) 0x7f95500014d0 con 0x7f955408c6d0 2019-12-05T06:07:06.571 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.557+0000 7f9558a10700 1 -- 172.21.15.76:0/2013188202 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9540002e80 con 0x7f955408c6d0 2019-12-05T06:07:06.572 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.557+0000 7f9558a10700 1 -- 172.21.15.76:0/2013188202 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2983704134 0 0) 0x7f954c0009b0 con 0x7f955408fef0 2019-12-05T06:07:06.572 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.557+0000 7f9559211700 1 -- 172.21.15.76:0/2013188202 >> v1:172.21.15.76:6789/0 conn(0x7f955408c6d0 legacy=0x7f955408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:06.572 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.557+0000 7f9559211700 1 -- 172.21.15.76:0/2013188202 >> v1:172.21.15.16:6789/0 conn(0x7f955408fef0 legacy=0x7f955408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:06.572 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.557+0000 7f9559211700 1 -- 172.21.15.76:0/2013188202 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f955408e620 con 0x7f955408d2c0 2019-12-05T06:07:06.572 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.557+0000 7f9559211700 1 --2- 172.21.15.76:0/2013188202 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 0x7f955408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f954400af50 tx=0x7f954400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.573 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.557+0000 7f9559211700 1 --2- 172.21.15.76:0/2013188202 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 0x7f955408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f954400af50 tx=0x7f954400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.573 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.557+0000 7f9559211700 1 --2- 172.21.15.76:0/2013188202 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 0x7f955408ef20 secure :-1 s=READY pgs=52 cs=0 l=1 rx=0x7f954400af50 tx=0x7f954400b760).ready entity=mon.2 client_cookie=c8c82e7779a7f37c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.558+0000 7f9559211700 1 --2- 172.21.15.76:0/2013188202 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 0x7f955408ef20 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f954400af50 tx=0x7f954400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.558+0000 7f9558a10700 1 -- 172.21.15.76:0/2013188202 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9544000960 con 0x7f955408d2c0 2019-12-05T06:07:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.558+0000 7f9559211700 1 --2- 172.21.15.76:0/2013188202 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 0x7f955408ef20 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f954400af50 tx=0x7f954400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.558+0000 7f9559211700 1 --2- 172.21.15.76:0/2013188202 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 0x7f955408ef20 secure :-1 s=THROTTLE_DONE pgs=52 cs=0 l=1 rx=0x7f954400af50 tx=0x7f954400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.558+0000 7f9558a10700 1 -- 172.21.15.76:0/2013188202 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9544009860 con 0x7f955408d2c0 2019-12-05T06:07:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.558+0000 7f9558a10700 1 -- 172.21.15.76:0/2013188202 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9544009f30 con 0x7f955408d2c0 2019-12-05T06:07:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.558+0000 7f955b475700 1 -- 172.21.15.76:0/2013188202 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 msgr2=0x7f955408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.558+0000 7f955b475700 1 --2- 172.21.15.76:0/2013188202 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 0x7f955408ef20 secure :-1 s=READY pgs=52 cs=0 l=1 rx=0x7f954400af50 tx=0x7f954400b760).stop 2019-12-05T06:07:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.559+0000 7f955b475700 1 -- 172.21.15.76:0/2013188202 shutdown_connections 2019-12-05T06:07:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.559+0000 7f955b475700 1 --2- 172.21.15.76:0/2013188202 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955408d2c0 0x7f955408ef20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.559+0000 7f955b475700 1 -- 172.21.15.76:0/2013188202 shutdown_connections 2019-12-05T06:07:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.560+0000 7f955b475700 1 -- 172.21.15.76:0/2013188202 wait complete. 2019-12-05T06:07:06.575 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.560+0000 7f955b475700 1 -- 172.21.15.76:0/2013188202 >> 172.21.15.76:0/2013188202 conn(0x7f9554061940 msgr2=0x7f9554096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.565+0000 7f955b475700 1 Processor -- start 2019-12-05T06:07:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.565+0000 7f955b475700 1 -- start start 2019-12-05T06:07:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f955b475700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f95541302f0 0x7f955412a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f955b475700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955412b640 0x7f955412abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f955b475700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9554131bf0 con 0x7f955412b0f0 2019-12-05T06:07:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f955b475700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95541296e0 con 0x7f95541302f0 2019-12-05T06:07:06.576 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f955b475700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9554128cb0 con 0x7f955412b640 2019-12-05T06:07:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f9559a12700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f95541302f0 0x7f955412a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f9559211700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955412b640 0x7f955412abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f9559211700 1 -- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955412b640 msgr2=0x7f955412abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:07:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f9559211700 1 -- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955412b640 msgr2=0x7f955412abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:07:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f9559a12700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f95541302f0 0x7f955412a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.76:51444/0 (socket says 172.21.15.76:51444) 2019-12-05T06:07:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f9559211700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955412b640 0x7f955412abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:07:06.577 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f9559a12700 1 -- 172.21.15.76:0/3077290651 learned_addr learned my addr 172.21.15.76:0/3077290651 (peer_addr_for_me v2:172.21.15.76:0/0) 2019-12-05T06:07:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.567+0000 7f9559211700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955412b640 0x7f955412abb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:07:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f9559a12700 1 -- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955412b640 msgr2=0x7f955412abb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:07:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f9559a12700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955412b640 0x7f955412abb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f9559a12700 1 -- 172.21.15.76:0/3077290651 >> v1:172.21.15.16:6789/0 conn(0x7f955412b0f0 legacy=0x7f95541306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f9559a12700 1 -- 172.21.15.76:0/3077290651 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9554129d70 con 0x7f95541302f0 2019-12-05T06:07:06.578 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f9559a12700 1 --2- 172.21.15.76:0/3077290651 >> v2:172.21.15.76:3300/0 conn(0x7f95541302f0 0x7f955412a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9550012970 tx=0x7f95500127f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f9559a12700 1 --2- 172.21.15.76:0/3077290651 >> v2:172.21.15.76:3300/0 conn(0x7f95541302f0 0x7f955412a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9550012970 tx=0x7f95500127f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f9559a12700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95541302f0 0x7f955412a670 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7f9550012970 tx=0x7f95500127f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f9559a12700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95541302f0 0x7f955412a670 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f9550012970 tx=0x7f95500127f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f9559a12700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95541302f0 0x7f955412a670 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f9550012970 tx=0x7f95500127f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f9559a12700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95541302f0 0x7f955412a670 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f9550012970 tx=0x7f95500127f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.579 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f954a7fc700 1 -- 172.21.15.76:0/3077290651 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9550001010 con 0x7f95541302f0 2019-12-05T06:07:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f954a7fc700 1 -- 172.21.15.76:0/3077290651 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9550001440 con 0x7f95541302f0 2019-12-05T06:07:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f955b475700 1 -- 172.21.15.76:0/3077290651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9554129d70 con 0x7f95541302f0 2019-12-05T06:07:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.568+0000 7f954a7fc700 1 -- 172.21.15.76:0/3077290651 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9550001b10 con 0x7f95541302f0 2019-12-05T06:07:06.580 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.569+0000 7f955b475700 1 -- 172.21.15.76:0/3077290651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9554129d70 con 0x7f95541302f0 2019-12-05T06:07:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.569+0000 7f9559a12700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95541302f0 0x7f955412a670 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f9550012970 tx=0x7f95500127f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.570+0000 7f954a7fc700 1 -- 172.21.15.76:0/3077290651 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9550001010 con 0x7f95541302f0 2019-12-05T06:07:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.570+0000 7f9559a12700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95541302f0 0x7f955412a670 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f9550012970 tx=0x7f95500127f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.570+0000 7f954a7fc700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f953c023630 0x7f953c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:06.581 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.570+0000 7f954a7fc700 1 -- 172.21.15.76:0/3077290651 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f9550048320 con 0x7f95541302f0 2019-12-05T06:07:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.570+0000 7f9559211700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f953c023630 0x7f953c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:06.582 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.571+0000 7f9559211700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f953c023630 0x7f953c025ac0 crc :-1 s=READY pgs=101 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:06.583 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.585+0000 7f955b475700 1 -- 172.21.15.76:0/3077290651 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9554129d70 con 0x7f95541302f0 2019-12-05T06:07:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.591+0000 7f9559a12700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95541302f0 0x7f955412a670 secure :-1 s=THROTTLE_DONE pgs=160 cs=0 l=1 rx=0x7f9550012970 tx=0x7f95500127f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:06.586 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.591+0000 7f954a7fc700 1 -- 172.21.15.76:0/3077290651 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9550007920 con 0x7f95541302f0 2019-12-05T06:07:06.846 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.851+0000 7f955b475700 1 -- 172.21.15.76:0/3077290651 --> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9554129d70 con 0x7f953c023630 2019-12-05T06:07:06.853 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.857+0000 7f954a7fc700 1 -- 172.21.15.76:0/3077290651 <== mgr.4100 v2:172.21.15.76:6800/9546 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+80414 (crc 0 0 0) 0x7f954400b4e0 con 0x7f953c023630 2019-12-05T06:07:06.858 INFO:teuthology.orchestra.run.smithi076.stdout: 2019-12-05T06:07:06.859 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2019-12-05T06:07:06.859 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 -- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f953c023630 msgr2=0x7f953c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:06.859 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f953c023630 0x7f953c025ac0 crc :-1 s=READY pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:06.859 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 -- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95541302f0 msgr2=0x7f955412a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:06.860 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95541302f0 0x7f955412a670 secure :-1 s=READY pgs=160 cs=0 l=1 rx=0x7f9550012970 tx=0x7f95500127f0).stop 2019-12-05T06:07:06.860 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 -- 172.21.15.76:0/3077290651 shutdown_connections 2019-12-05T06:07:06.860 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f953c023630 0x7f953c025ac0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:06.860 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f955412b640 0x7f955412abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:06.861 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 --2- 172.21.15.76:0/3077290651 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f95541302f0 0x7f955412a670 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:06.861 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 -- 172.21.15.76:0/3077290651 shutdown_connections 2019-12-05T06:07:06.861 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 -- 172.21.15.76:0/3077290651 wait complete. 2019-12-05T06:07:06.861 INFO:teuthology.orchestra.run.smithi076.stderr:2019-12-05T06:07:06.861+0000 7f955b475700 1 -- 172.21.15.76:0/3077290651 >> 172.21.15.76:0/3077290651 conn(0x7f95540a2850 msgr2=0x7f955406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:06.867 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":45,"stamp":"2019-12-05T06:07:06.772091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":64,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":8390720,"kb_used_data":2048,"kb_used_omap":0,"kb_used_meta":8388608,"kb_avail":746584000,"statfs":{"total":773094113280,"available":764502016000,"internally_reserved":8589934592,"allocated":2097152,"data_stored":772252,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":8589934592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003769"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.271382+0000","last_change":"2019-12-05T06:06:32.598114+0000","last_active":"2019-12-05T06:06:40.271382+0000","last_peered":"2019-12-05T06:06:40.271382+0000","last_clean":"2019-12-05T06:06:40.271382+0000","last_became_active":"2019-12-05T06:06:32.597387+0000","last_became_peered":"2019-12-05T06:06:32.597387+0000","last_unstale":"2019-12-05T06:06:40.271382+0000","last_undegraded":"2019-12-05T06:06:40.271382+0000","last_fullsized":"2019-12-05T06:06:40.271382+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.4","version":"24'8","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.715381+0000","last_change":"2019-12-05T06:06:36.627910+0000","last_active":"2019-12-05T06:06:44.715381+0000","last_peered":"2019-12-05T06:06:44.715381+0000","last_clean":"2019-12-05T06:06:44.715381+0000","last_became_active":"2019-12-05T06:06:36.626897+0000","last_became_peered":"2019-12-05T06:06:36.626897+0000","last_unstale":"2019-12-05T06:06:44.715381+0000","last_undegraded":"2019-12-05T06:06:44.715381+0000","last_fullsized":"2019-12-05T06:06:44.715381+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1526,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.490878+0000","last_change":"2019-12-05T06:06:40.490139+0000","last_active":"2019-12-05T06:06:40.490878+0000","last_peered":"2019-12-05T06:06:40.490878+0000","last_clean":"2019-12-05T06:06:40.490878+0000","last_became_active":"2019-12-05T06:06:40.488804+0000","last_became_peered":"2019-12-05T06:06:40.488804+0000","last_unstale":"2019-12-05T06:06:40.490878+0000","last_undegraded":"2019-12-05T06:06:40.490878+0000","last_fullsized":"2019-12-05T06:06:40.490878+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2,6],"acting":[5,3,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.271284+0000","last_change":"2019-12-05T06:06:32.887464+0000","last_active":"2019-12-05T06:06:40.271284+0000","last_peered":"2019-12-05T06:06:40.271284+0000","last_clean":"2019-12-05T06:06:40.271284+0000","last_became_active":"2019-12-05T06:06:32.885955+0000","last_became_peered":"2019-12-05T06:06:32.885955+0000","last_unstale":"2019-12-05T06:06:40.271284+0000","last_undegraded":"2019-12-05T06:06:40.271284+0000","last_fullsized":"2019-12-05T06:06:40.271284+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.5","version":"24'6","reported_seq":"19","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720231+0000","last_change":"2019-12-05T06:06:36.631805+0000","last_active":"2019-12-05T06:06:44.720231+0000","last_peered":"2019-12-05T06:06:44.720231+0000","last_clean":"2019-12-05T06:06:44.720231+0000","last_became_active":"2019-12-05T06:06:36.631077+0000","last_became_peered":"2019-12-05T06:06:36.631077+0000","last_unstale":"2019-12-05T06:06:44.720231+0000","last_undegraded":"2019-12-05T06:06:44.720231+0000","last_fullsized":"2019-12-05T06:06:44.720231+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.485950+0000","last_change":"2019-12-05T06:06:40.485444+0000","last_active":"2019-12-05T06:06:40.485950+0000","last_peered":"2019-12-05T06:06:40.485950+0000","last_clean":"2019-12-05T06:06:40.485950+0000","last_became_active":"2019-12-05T06:06:40.484465+0000","last_became_peered":"2019-12-05T06:06:40.484465+0000","last_unstale":"2019-12-05T06:06:40.485950+0000","last_undegraded":"2019-12-05T06:06:40.485950+0000","last_fullsized":"2019-12-05T06:06:40.485950+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0,5],"acting":[1,2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576769+0000","last_change":"2019-12-05T06:06:40.488691+0000","last_active":"2019-12-05T06:06:40.576769+0000","last_peered":"2019-12-05T06:06:40.576769+0000","last_clean":"2019-12-05T06:06:40.576769+0000","last_became_active":"2019-12-05T06:06:40.487506+0000","last_became_peered":"2019-12-05T06:06:40.487506+0000","last_unstale":"2019-12-05T06:06:40.576769+0000","last_undegraded":"2019-12-05T06:06:40.576769+0000","last_fullsized":"2019-12-05T06:06:40.576769+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7,0],"acting":[3,5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"16","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.268843+0000","last_change":"2019-12-05T06:06:32.597542+0000","last_active":"2019-12-05T06:06:40.268843+0000","last_peered":"2019-12-05T06:06:40.268843+0000","last_clean":"2019-12-05T06:06:40.268843+0000","last_became_active":"2019-12-05T06:06:32.596579+0000","last_became_peered":"2019-12-05T06:06:32.596579+0000","last_unstale":"2019-12-05T06:06:40.268843+0000","last_undegraded":"2019-12-05T06:06:40.268843+0000","last_fullsized":"2019-12-05T06:06:40.268843+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.6","version":"24'4","reported_seq":"16","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.710303+0000","last_change":"2019-12-05T06:06:36.626828+0000","last_active":"2019-12-05T06:06:44.710303+0000","last_peered":"2019-12-05T06:06:44.710303+0000","last_clean":"2019-12-05T06:06:44.710303+0000","last_became_active":"2019-12-05T06:06:36.626005+0000","last_became_peered":"2019-12-05T06:06:36.626005+0000","last_unstale":"2019-12-05T06:06:44.710303+0000","last_undegraded":"2019-12-05T06:06:44.710303+0000","last_fullsized":"2019-12-05T06:06:44.710303+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.015035+0000","last_change":"2019-12-05T06:06:40.487696+0000","last_active":"2019-12-05T06:06:41.015035+0000","last_peered":"2019-12-05T06:06:41.015035+0000","last_clean":"2019-12-05T06:06:41.015035+0000","last_became_active":"2019-12-05T06:06:40.486899+0000","last_became_peered":"2019-12-05T06:06:40.486899+0000","last_unstale":"2019-12-05T06:06:41.015035+0000","last_undegraded":"2019-12-05T06:06:41.015035+0000","last_fullsized":"2019-12-05T06:06:41.015035+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4,7],"acting":[0,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576888+0000","last_change":"2019-12-05T06:06:32.597098+0000","last_active":"2019-12-05T06:06:40.576888+0000","last_peered":"2019-12-05T06:06:40.576888+0000","last_clean":"2019-12-05T06:06:40.576888+0000","last_became_active":"2019-12-05T06:06:32.596276+0000","last_became_peered":"2019-12-05T06:06:32.596276+0000","last_unstale":"2019-12-05T06:06:40.576888+0000","last_undegraded":"2019-12-05T06:06:40.576888+0000","last_fullsized":"2019-12-05T06:06:40.576888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"24'7","reported_seq":"20","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720981+0000","last_change":"2019-12-05T06:06:36.629032+0000","last_active":"2019-12-05T06:06:44.720981+0000","last_peered":"2019-12-05T06:06:44.720981+0000","last_clean":"2019-12-05T06:06:44.720981+0000","last_became_active":"2019-12-05T06:06:36.628405+0000","last_became_peered":"2019-12-05T06:06:36.628405+0000","last_unstale":"2019-12-05T06:06:44.720981+0000","last_undegraded":"2019-12-05T06:06:44.720981+0000","last_fullsized":"2019-12-05T06:06:44.720981+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":566,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.084640+0000","last_change":"2019-12-05T06:06:32.886929+0000","last_active":"2019-12-05T06:06:41.084640+0000","last_peered":"2019-12-05T06:06:41.084640+0000","last_clean":"2019-12-05T06:06:41.084640+0000","last_became_active":"2019-12-05T06:06:32.886095+0000","last_became_peered":"2019-12-05T06:06:32.886095+0000","last_unstale":"2019-12-05T06:06:41.084640+0000","last_undegraded":"2019-12-05T06:06:41.084640+0000","last_fullsized":"2019-12-05T06:06:41.084640+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"24'5","reported_seq":"17","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.720647+0000","last_change":"2019-12-05T06:06:36.632241+0000","last_active":"2019-12-05T06:06:44.720647+0000","last_peered":"2019-12-05T06:06:44.720647+0000","last_clean":"2019-12-05T06:06:44.720647+0000","last_became_active":"2019-12-05T06:06:36.631418+0000","last_became_peered":"2019-12-05T06:06:36.631418+0000","last_unstale":"2019-12-05T06:06:44.720647+0000","last_undegraded":"2019-12-05T06:06:44.720647+0000","last_fullsized":"2019-12-05T06:06:44.720647+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.576879+0000","last_change":"2019-12-05T06:06:39.649718+0000","last_active":"2019-12-05T06:06:40.576879+0000","last_peered":"2019-12-05T06:06:40.576879+0000","last_clean":"2019-12-05T06:06:40.576879+0000","last_became_active":"2019-12-05T06:06:39.648680+0000","last_became_peered":"2019-12-05T06:06:39.648680+0000","last_unstale":"2019-12-05T06:06:40.576879+0000","last_undegraded":"2019-12-05T06:06:40.576879+0000","last_fullsized":"2019-12-05T06:06:40.576879+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6,4],"acting":[3,5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.269507+0000","last_change":"2019-12-05T06:06:32.886427+0000","last_active":"2019-12-05T06:06:40.269507+0000","last_peered":"2019-12-05T06:06:40.269507+0000","last_clean":"2019-12-05T06:06:40.269507+0000","last_became_active":"2019-12-05T06:06:32.885509+0000","last_became_peered":"2019-12-05T06:06:32.885509+0000","last_unstale":"2019-12-05T06:06:40.269507+0000","last_undegraded":"2019-12-05T06:06:40.269507+0000","last_fullsized":"2019-12-05T06:06:40.269507+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.2","version":"24'1","reported_seq":"13","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:41.269608+0000","last_change":"2019-12-05T06:06:36.632232+0000","last_active":"2019-12-05T06:06:41.269608+0000","last_peered":"2019-12-05T06:06:41.269608+0000","last_clean":"2019-12-05T06:06:41.269608+0000","last_became_active":"2019-12-05T06:06:36.630785+0000","last_became_peered":"2019-12-05T06:06:36.630785+0000","last_unstale":"2019-12-05T06:06:41.269608+0000","last_undegraded":"2019-12-05T06:06:41.269608+0000","last_fullsized":"2019-12-05T06:06:41.269608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.268862+0000","last_change":"2019-12-05T06:06:39.647777+0000","last_active":"2019-12-05T06:06:40.268862+0000","last_peered":"2019-12-05T06:06:40.268862+0000","last_clean":"2019-12-05T06:06:40.268862+0000","last_became_active":"2019-12-05T06:06:39.646655+0000","last_became_peered":"2019-12-05T06:06:39.646655+0000","last_unstale":"2019-12-05T06:06:40.268862+0000","last_undegraded":"2019-12-05T06:06:40.268862+0000","last_fullsized":"2019-12-05T06:06:40.268862+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0,6],"acting":[4,5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.547093+0000","last_change":"2019-12-05T06:06:36.628444+0000","last_active":"2019-12-05T06:06:40.547093+0000","last_peered":"2019-12-05T06:06:40.547093+0000","last_clean":"2019-12-05T06:06:40.547093+0000","last_became_active":"2019-12-05T06:06:36.627555+0000","last_became_peered":"2019-12-05T06:06:36.627555+0000","last_unstale":"2019-12-05T06:06:40.547093+0000","last_undegraded":"2019-12-05T06:06:40.547093+0000","last_fullsized":"2019-12-05T06:06:40.547093+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.084584+0000","last_change":"2019-12-05T06:06:32.886895+0000","last_active":"2019-12-05T06:06:41.084584+0000","last_peered":"2019-12-05T06:06:41.084584+0000","last_clean":"2019-12-05T06:06:41.084584+0000","last_became_active":"2019-12-05T06:06:32.886123+0000","last_became_peered":"2019-12-05T06:06:32.886123+0000","last_unstale":"2019-12-05T06:06:41.084584+0000","last_undegraded":"2019-12-05T06:06:41.084584+0000","last_fullsized":"2019-12-05T06:06:41.084584+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.486413+0000","last_change":"2019-12-05T06:06:40.486073+0000","last_active":"2019-12-05T06:06:40.486413+0000","last_peered":"2019-12-05T06:06:40.486413+0000","last_clean":"2019-12-05T06:06:40.486413+0000","last_became_active":"2019-12-05T06:06:40.484954+0000","last_became_peered":"2019-12-05T06:06:40.484954+0000","last_unstale":"2019-12-05T06:06:40.486413+0000","last_undegraded":"2019-12-05T06:06:40.486413+0000","last_fullsized":"2019-12-05T06:06:40.486413+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6,5],"acting":[1,2,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"24'8","reported_seq":"21","reported_epoch":"24","state":"active+clean","last_fresh":"2019-12-05T06:06:44.723134+0000","last_change":"2019-12-05T06:06:36.630520+0000","last_active":"2019-12-05T06:06:44.723134+0000","last_peered":"2019-12-05T06:06:44.723134+0000","last_clean":"2019-12-05T06:06:44.723134+0000","last_became_active":"2019-12-05T06:06:36.629682+0000","last_became_peered":"2019-12-05T06:06:36.629682+0000","last_unstale":"2019-12-05T06:06:44.723134+0000","last_undegraded":"2019-12-05T06:06:44.723134+0000","last_fullsized":"2019-12-05T06:06:44.723134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:35.604913+0000","last_clean_scrub_stamp":"2019-12-05T06:06:35.604913+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"15","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:40.484464+0000","last_change":"2019-12-05T06:06:32.595548+0000","last_active":"2019-12-05T06:06:40.484464+0000","last_peered":"2019-12-05T06:06:40.484464+0000","last_clean":"2019-12-05T06:06:40.484464+0000","last_became_active":"2019-12-05T06:06:32.594119+0000","last_became_peered":"2019-12-05T06:06:32.594119+0000","last_unstale":"2019-12-05T06:06:40.484464+0000","last_undegraded":"2019-12-05T06:06:40.484464+0000","last_fullsized":"2019-12-05T06:06:40.484464+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:31.572954+0000","last_clean_scrub_stamp":"2019-12-05T06:06:31.572954+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"12","reported_epoch":"23","state":"active+clean","last_fresh":"2019-12-05T06:06:41.014770+0000","last_change":"2019-12-05T06:06:39.649718+0000","last_active":"2019-12-05T06:06:41.014770+0000","last_peered":"2019-12-05T06:06:41.014770+0000","last_clean":"2019-12-05T06:06:41.014770+0000","last_became_active":"2019-12-05T06:06:39.648278+0000","last_became_peered":"2019-12-05T06:06:39.648278+0000","last_unstale":"2019-12-05T06:06:41.014770+0000","last_undegraded":"2019-12-05T06:06:41.014770+0000","last_fullsized":"2019-12-05T06:06:41.014770+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2019-12-05T06:06:38.627113+0000","last_clean_scrub_stamp":"2019-12-05T06:06:38.627113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3,5],"acting":[0,4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":32,"acting":32,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":2286,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574858,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048848,"kb_used_data":264,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322992,"statfs":{"total":96636764160,"available":95562743808,"internally_reserved":1073741824,"allocated":270336,"data_stored":96686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.758}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.188}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.835}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.052}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.642}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.228}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.789}]}]},{"osd":6,"up_from":13,"seq":55834574857,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048840,"kb_used_data":256,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323000,"statfs":{"total":96636764160,"available":95562752000,"internally_reserved":1073741824,"allocated":262144,"data_stored":96526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.639}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.918}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.887}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.811}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.049}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.142}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.090}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.562}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.734}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.846}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.902}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.795}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.056}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.126}]}]},{"osd":4,"up_from":13,"seq":55834574858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.763}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.516}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.980}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.106}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.808}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.049}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.187}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048852,"kb_used_data":268,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322988,"statfs":{"total":96636764160,"available":95562739712,"internally_reserved":1073741824,"allocated":274432,"data_stored":97646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.703}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.589}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.532}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.942}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.099}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.821}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.017}]}]},{"osd":1,"up_from":13,"seq":55834574858,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048844,"kb_used_data":260,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93322996,"statfs":{"total":96636764160,"available":95562747904,"internally_reserved":1073741824,"allocated":266240,"data_stored":97486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.722}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.685}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.814}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.878}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.234}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.028}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.283}]}]},{"osd":2,"up_from":13,"seq":55834574857,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048836,"kb_used_data":252,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323004,"statfs":{"total":96636764160,"available":95562756096,"internally_reserved":1073741824,"allocated":258048,"data_stored":95994,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.075}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.843}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.586}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.132}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.758}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.346}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.300}]}]},{"osd":3,"up_from":13,"seq":55834574860,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1048832,"kb_used_data":248,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":93323008,"statfs":{"total":96636764160,"available":95562760192,"internally_reserved":1073741824,"allocated":253952,"data_stored":95960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.862}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.632}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.662}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.843}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.974}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":0.942}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0.000,"5min":0.000,"15min":0.000},"min":{"1min":0.000,"5min":0.000,"15min":0.000},"max":{"1min":0.000,"5min":0.000,"15min":0.000},"last":1.075}]}]}]}} 2019-12-05T06:07:06.871 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:07:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.145+0000 7f9446923700 1 Processor -- start 2019-12-05T06:07:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.145+0000 7f9446923700 1 -- start start 2019-12-05T06:07:07.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.147+0000 7f9446923700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f944008d2c0 0x7f944008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:07.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.147+0000 7f9446923700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f944008fef0 0x7f944008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:07.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.147+0000 7f9446923700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94400958e0 con 0x7f944008c6d0 2019-12-05T06:07:07.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.147+0000 7f9446923700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944008df90 con 0x7f944008fef0 2019-12-05T06:07:07.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.147+0000 7f9446923700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94400a3f70 con 0x7f944008d2c0 2019-12-05T06:07:07.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.147+0000 7f94456c1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f944008fef0 0x7f944008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:07.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.147+0000 7f94456c1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f944008fef0 0x7f944008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46658/0 (socket says 172.21.15.16:46658) 2019-12-05T06:07:07.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.147+0000 7f943ffff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f944008d2c0 0x7f944008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:07.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.147+0000 7f94456c1700 1 -- 172.21.15.16:0/3085834844 learned_addr learned my addr 172.21.15.16:0/3085834844 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:07.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.148+0000 7f943f7fe700 1 -- 172.21.15.16:0/3085834844 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157043097 0 0) 0x7f9438001400 con 0x7f944008c6d0 2019-12-05T06:07:07.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.148+0000 7f94456c1700 1 -- 172.21.15.16:0/3085834844 >> v2:172.21.15.76:3300/0 conn(0x7f944008d2c0 msgr2=0x7f944008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:07.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.148+0000 7f94456c1700 1 --2- 172.21.15.16:0/3085834844 >> v2:172.21.15.76:3300/0 conn(0x7f944008d2c0 0x7f944008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:07.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.148+0000 7f94456c1700 1 -- 172.21.15.16:0/3085834844 >> v1:172.21.15.76:6789/0 conn(0x7f944008c6d0 legacy=0x7f944008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:07.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.148+0000 7f94456c1700 1 -- 172.21.15.16:0/3085834844 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944008e620 con 0x7f944008fef0 2019-12-05T06:07:07.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.148+0000 7f94456c1700 1 --2- 172.21.15.16:0/3085834844 >> v2:172.21.15.16:3300/0 conn(0x7f944008fef0 0x7f944008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f942800f730 tx=0x7f9428007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.148+0000 7f94456c1700 1 --2- 172.21.15.16:0/3085834844 >> v2:172.21.15.16:3300/0 conn(0x7f944008fef0 0x7f944008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f942800f730 tx=0x7f9428007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.148+0000 7f94456c1700 1 --2- 172.21.15.16:0/3085834844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f944008fef0 0x7f944008f460 secure :-1 s=READY pgs=192 cs=0 l=1 rx=0x7f942800f730 tx=0x7f9428007d50).ready entity=mon.0 client_cookie=c9cec97ca6e452fb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:07.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.149+0000 7f94456c1700 1 --2- 172.21.15.16:0/3085834844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f944008fef0 0x7f944008f460 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f942800f730 tx=0x7f9428007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.149+0000 7f943f7fe700 1 -- 172.21.15.16:0/3085834844 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9428000a70 con 0x7f944008fef0 2019-12-05T06:07:07.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.149+0000 7f94456c1700 1 --2- 172.21.15.16:0/3085834844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f944008fef0 0x7f944008f460 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f942800f730 tx=0x7f9428007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.149+0000 7f94456c1700 1 --2- 172.21.15.16:0/3085834844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f944008fef0 0x7f944008f460 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f942800f730 tx=0x7f9428007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.149+0000 7f943f7fe700 1 -- 172.21.15.16:0/3085834844 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94280010e0 con 0x7f944008fef0 2019-12-05T06:07:07.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.149+0000 7f943f7fe700 1 -- 172.21.15.16:0/3085834844 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94280012d0 con 0x7f944008fef0 2019-12-05T06:07:07.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.149+0000 7f9446923700 1 -- 172.21.15.16:0/3085834844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f944008fef0 msgr2=0x7f944008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:07.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.149+0000 7f9446923700 1 --2- 172.21.15.16:0/3085834844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f944008fef0 0x7f944008f460 secure :-1 s=READY pgs=192 cs=0 l=1 rx=0x7f942800f730 tx=0x7f9428007d50).stop 2019-12-05T06:07:07.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.150+0000 7f9446923700 1 -- 172.21.15.16:0/3085834844 shutdown_connections 2019-12-05T06:07:07.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.150+0000 7f9446923700 1 --2- 172.21.15.16:0/3085834844 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f944008fef0 0x7f944008f460 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:07.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.150+0000 7f9446923700 1 --2- 172.21.15.16:0/3085834844 >> v2:172.21.15.76:3300/0 conn(0x7f944008d2c0 0x7f944008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:07.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.150+0000 7f9446923700 1 -- 172.21.15.16:0/3085834844 shutdown_connections 2019-12-05T06:07:07.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.150+0000 7f9446923700 1 -- 172.21.15.16:0/3085834844 wait complete. 2019-12-05T06:07:07.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.150+0000 7f9446923700 1 -- 172.21.15.16:0/3085834844 >> 172.21.15.16:0/3085834844 conn(0x7f9440061940 msgr2=0x7f9440096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:07.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.156+0000 7f9446923700 1 Processor -- start 2019-12-05T06:07:07.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.156+0000 7f9446923700 1 -- start start 2019-12-05T06:07:07.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.158+0000 7f9446923700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9440132d00 0x7f944014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:07.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.158+0000 7f9446923700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9440098da0 con 0x7f944014d400 2019-12-05T06:07:07.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.158+0000 7f9444ec0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f944014d400 0x7f9440131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60060/0 (socket says 172.21.15.16:60060) 2019-12-05T06:07:07.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.158+0000 7f9444ec0700 1 -- 172.21.15.16:0/2640592178 learned_addr learned my addr 172.21.15.16:0/2640592178 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:07.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.158+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f944009f0a0 con 0x7f9440132530 2019-12-05T06:07:07.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.158+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f944009bb50 con 0x7f9440132d00 2019-12-05T06:07:07.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.158+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> v2:172.21.15.76:3300/0 conn(0x7f9440132d00 0x7f944014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.158+0000 7f943d7fa700 1 -- 172.21.15.16:0/2640592178 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341774993 0 0) 0x7f94380014b0 con 0x7f944014d400 2019-12-05T06:07:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.158+0000 7f943d7fa700 1 -- 172.21.15.16:0/2640592178 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9424002e80 con 0x7f944014d400 2019-12-05T06:07:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.159+0000 7f943ffff700 1 -- 172.21.15.16:0/2640592178 >> v1:172.21.15.76:6789/0 conn(0x7f9440132530 legacy=0x7f9440157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.159+0000 7f943ffff700 1 -- 172.21.15.16:0/2640592178 >> v1:172.21.15.16:6789/0 conn(0x7f944014d400 legacy=0x7f9440131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.159+0000 7f943d7fa700 1 -- 172.21.15.16:0/2640592178 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1828285183 0 0) 0x7f9428007d20 con 0x7f9440132530 2019-12-05T06:07:07.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.159+0000 7f943ffff700 1 -- 172.21.15.16:0/2640592178 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f944009b0f0 con 0x7f9440132d00 2019-12-05T06:07:07.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.159+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> v2:172.21.15.76:3300/0 conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.159+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> v2:172.21.15.76:3300/0 conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.159+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:07.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.159+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.160+0000 7f943d7fa700 1 -- 172.21.15.16:0/2640592178 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9430009640 con 0x7f9440132d00 2019-12-05T06:07:07.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.160+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.160+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.160+0000 7f943d7fa700 1 -- 172.21.15.16:0/2640592178 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9430000940 con 0x7f9440132d00 2019-12-05T06:07:07.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.160+0000 7f943d7fa700 1 -- 172.21.15.16:0/2640592178 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f943000a0e0 con 0x7f9440132d00 2019-12-05T06:07:07.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.160+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f944009b0f0 con 0x7f9440132d00 2019-12-05T06:07:07.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.160+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f944009b0f0 con 0x7f9440132d00 2019-12-05T06:07:07.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.161+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.162+0000 7f943d7fa700 1 -- 172.21.15.16:0/2640592178 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9430009a70 con 0x7f9440132d00 2019-12-05T06:07:07.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.162+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.162+0000 7f943d7fa700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9424023680 0x7f9424025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:07.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.162+0000 7f943d7fa700 1 -- 172.21.15.16:0/2640592178 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f9430045e00 con 0x7f9440132d00 2019-12-05T06:07:07.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.162+0000 7f9444ec0700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9424023680 0x7f9424025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:07.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.163+0000 7f9444ec0700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9424023680 0x7f9424025af0 crc :-1 s=READY pgs=102 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:07.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.177+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f944009b0f0 con 0x7f9440132d00 2019-12-05T06:07:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.186+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.186+0000 7f943d7fa700 1 -- 172.21.15.16:0/2640592178 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9430009830 con 0x7f9440132d00 2019-12-05T06:07:07.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.450+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f944009b0f0 con 0x7f9440132d00 2019-12-05T06:07:07.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.452+0000 7f943ffff700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.452+0000 7f943d7fa700 1 -- 172.21.15.16:0/2640592178 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f9430009a40 con 0x7f9440132d00 2019-12-05T06:07:07.453 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:07:07.453 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:07:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.455+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9424023680 msgr2=0x7f9424025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:07.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.455+0000 7f9446923700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9424023680 0x7f9424025af0 crc :-1 s=READY pgs=102 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:07.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.455+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 msgr2=0x7f944014d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:07.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.456+0000 7f9446923700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 0x7f944014d7f0 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7f9430010390 tx=0x7f9430010730).stop 2019-12-05T06:07:07.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.456+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 shutdown_connections 2019-12-05T06:07:07.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.456+0000 7f9446923700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9424023680 0x7f9424025af0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:07.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.456+0000 7f9446923700 1 --2- 172.21.15.16:0/2640592178 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9440132d00 0x7f944014d7f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:07.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.456+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 shutdown_connections 2019-12-05T06:07:07.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.456+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 wait complete. 2019-12-05T06:07:07.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.456+0000 7f9446923700 1 -- 172.21.15.16:0/2640592178 >> 172.21.15.16:0/2640592178 conn(0x7f94400714f0 msgr2=0x7f944009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:07.465 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:31.052682+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:34.862662+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:37.644552+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T06:07:07.466 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T06:07:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.746+0000 7fb16d3e4700 1 Processor -- start 2019-12-05T06:07:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.746+0000 7fb16d3e4700 1 -- start start 2019-12-05T06:07:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.747+0000 7fb16d3e4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb168086980 0x7fb168086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.748+0000 7fb16d3e4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 0x7fb16808ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.748+0000 7fb16d3e4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb168061a10 con 0x7fb16808b060 2019-12-05T06:07:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.748+0000 7fb16d3e4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb168060bc0 con 0x7fb168086980 2019-12-05T06:07:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.748+0000 7fb16d3e4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb168060620 con 0x7fb16808a6b0 2019-12-05T06:07:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.748+0000 7fb1677fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb168086980 0x7fb168086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.748+0000 7fb166ffd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 0x7fb16808ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.748+0000 7fb167fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb16808b060 0x7fb168064310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53140/0 (socket says 172.21.15.16:53140) 2019-12-05T06:07:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.748+0000 7fb167fff700 1 -- 172.21.15.16:0/3587932014 learned_addr learned my addr 172.21.15.16:0/3587932014 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.749+0000 7fb1667fc700 1 -- 172.21.15.16:0/3587932014 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3941281529 0 0) 0x7fb160001100 con 0x7fb16808b060 2019-12-05T06:07:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.749+0000 7fb1667fc700 1 -- 172.21.15.16:0/3587932014 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb150002e80 con 0x7fb16808b060 2019-12-05T06:07:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.749+0000 7fb166ffd700 1 -- 172.21.15.16:0/3587932014 >> v2:172.21.15.76:3300/0 conn(0x7fb168086980 msgr2=0x7fb168086d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.749+0000 7fb166ffd700 1 --2- 172.21.15.16:0/3587932014 >> v2:172.21.15.76:3300/0 conn(0x7fb168086980 0x7fb168086d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.749+0000 7fb166ffd700 1 -- 172.21.15.16:0/3587932014 >> v1:172.21.15.76:6789/0 conn(0x7fb16808b060 legacy=0x7fb168064310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.749+0000 7fb166ffd700 1 -- 172.21.15.16:0/3587932014 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb168063a10 con 0x7fb16808a6b0 2019-12-05T06:07:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.749+0000 7fb1677fe700 1 --2- 172.21.15.16:0/3587932014 >> v2:172.21.15.76:3300/0 conn(0x7fb168086980 0x7fb168086d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.749+0000 7fb166ffd700 1 --2- 172.21.15.16:0/3587932014 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 0x7fb16808ab20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb15c00b2d0 tx=0x7fb15c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.750+0000 7fb166ffd700 1 --2- 172.21.15.16:0/3587932014 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 0x7fb16808ab20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb15c00b2d0 tx=0x7fb15c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.750+0000 7fb166ffd700 1 --2- 172.21.15.16:0/3587932014 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 0x7fb16808ab20 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7fb15c00b2d0 tx=0x7fb15c00bae0).ready entity=mon.2 client_cookie=8ffa158c97537f57 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.750+0000 7fb166ffd700 1 --2- 172.21.15.16:0/3587932014 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 0x7fb16808ab20 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fb15c00b2d0 tx=0x7fb15c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.750+0000 7fb166ffd700 1 --2- 172.21.15.16:0/3587932014 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 0x7fb16808ab20 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fb15c00b2d0 tx=0x7fb15c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.750+0000 7fb166ffd700 1 --2- 172.21.15.16:0/3587932014 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 0x7fb16808ab20 secure :-1 s=THROTTLE_DONE pgs=53 cs=0 l=1 rx=0x7fb15c00b2d0 tx=0x7fb15c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.750+0000 7fb1667fc700 1 -- 172.21.15.16:0/3587932014 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb15c007590 con 0x7fb16808a6b0 2019-12-05T06:07:07.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.750+0000 7fb1667fc700 1 -- 172.21.15.16:0/3587932014 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb15c009d30 con 0x7fb16808a6b0 2019-12-05T06:07:07.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.750+0000 7fb1667fc700 1 -- 172.21.15.16:0/3587932014 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb15c00a3c0 con 0x7fb16808a6b0 2019-12-05T06:07:07.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.751+0000 7fb16d3e4700 1 -- 172.21.15.16:0/3587932014 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 msgr2=0x7fb16808ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:07.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.751+0000 7fb16d3e4700 1 --2- 172.21.15.16:0/3587932014 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 0x7fb16808ab20 secure :-1 s=READY pgs=53 cs=0 l=1 rx=0x7fb15c00b2d0 tx=0x7fb15c00bae0).stop 2019-12-05T06:07:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.752+0000 7fb16d3e4700 1 -- 172.21.15.16:0/3587932014 shutdown_connections 2019-12-05T06:07:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.752+0000 7fb16d3e4700 1 --2- 172.21.15.16:0/3587932014 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb16808a6b0 0x7fb16808ab20 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:07.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.752+0000 7fb16d3e4700 1 --2- 172.21.15.16:0/3587932014 >> v2:172.21.15.76:3300/0 conn(0x7fb168086980 0x7fb168086d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:07.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.752+0000 7fb16d3e4700 1 -- 172.21.15.16:0/3587932014 shutdown_connections 2019-12-05T06:07:07.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.752+0000 7fb16d3e4700 1 -- 172.21.15.16:0/3587932014 wait complete. 2019-12-05T06:07:07.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.752+0000 7fb16d3e4700 1 -- 172.21.15.16:0/3587932014 >> 172.21.15.16:0/3587932014 conn(0x7fb16805f5a0 msgr2=0x7fb16809cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:07.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.758+0000 7fb16d3e4700 1 Processor -- start 2019-12-05T06:07:07.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.758+0000 7fb16d3e4700 1 -- start start 2019-12-05T06:07:07.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.759+0000 7fb16d3e4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb168073d80 0x7fb168095460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:07.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.760+0000 7fb16d3e4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb168156c80 0x7fb168159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:07.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.760+0000 7fb16d3e4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb168074c00 con 0x7fb168099990 2019-12-05T06:07:07.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.760+0000 7fb16d3e4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb168075e90 con 0x7fb168156c80 2019-12-05T06:07:07.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.760+0000 7fb16d3e4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb168076160 con 0x7fb168073d80 2019-12-05T06:07:07.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.760+0000 7fb167fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb168156c80 0x7fb168159080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:07.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.760+0000 7fb167fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb168156c80 0x7fb168159080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46678/0 (socket says 172.21.15.16:46678) 2019-12-05T06:07:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.760+0000 7fb167fff700 1 -- 172.21.15.16:0/253971375 learned_addr learned my addr 172.21.15.16:0/253971375 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.760+0000 7fb1677fe700 1 --2- 172.21.15.16:0/253971375 >> v2:172.21.15.76:3300/0 conn(0x7fb168073d80 0x7fb168095460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb167fff700 1 -- 172.21.15.16:0/253971375 >> v2:172.21.15.76:3300/0 conn(0x7fb168073d80 msgr2=0x7fb168095460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:07.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> v2:172.21.15.76:3300/0 conn(0x7fb168073d80 0x7fb168095460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb167fff700 1 -- 172.21.15.16:0/253971375 >> v1:172.21.15.76:6789/0 conn(0x7fb168099990 legacy=0x7fb1680959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb167fff700 1 -- 172.21.15.16:0/253971375 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb168094b60 con 0x7fb168156c80 2019-12-05T06:07:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> v2:172.21.15.16:3300/0 conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> v2:172.21.15.16:3300/0 conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb14ffff700 1 -- 172.21.15.16:0/253971375 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb160000ef0 con 0x7fb168156c80 2019-12-05T06:07:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb14ffff700 1 -- 172.21.15.16:0/253971375 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1600041c0 con 0x7fb168156c80 2019-12-05T06:07:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb16d3e4700 1 -- 172.21.15.16:0/253971375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb168094b60 con 0x7fb168156c80 2019-12-05T06:07:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.761+0000 7fb14ffff700 1 -- 172.21.15.16:0/253971375 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb160004850 con 0x7fb168156c80 2019-12-05T06:07:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.762+0000 7fb16d3e4700 1 -- 172.21.15.16:0/253971375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb168094b60 con 0x7fb168156c80 2019-12-05T06:07:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.763+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.764+0000 7fb14ffff700 1 -- 172.21.15.16:0/253971375 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb1600043b0 con 0x7fb168156c80 2019-12-05T06:07:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.764+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.764+0000 7fb14ffff700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb148023630 0x7fb148025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.764+0000 7fb14ffff700 1 -- 172.21.15.16:0/253971375 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fb160045df0 con 0x7fb168156c80 2019-12-05T06:07:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.765+0000 7fb1677fe700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb148023630 0x7fb148025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.765+0000 7fb1677fe700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb148023630 0x7fb148025ac0 crc :-1 s=READY pgs=103 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.777+0000 7fb16d3e4700 1 -- 172.21.15.16:0/253971375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb168094b60 con 0x7fb168156c80 2019-12-05T06:07:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.786+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:07.786+0000 7fb14ffff700 1 -- 172.21.15.16:0/253971375 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb160000940 con 0x7fb168156c80 2019-12-05T06:07:08.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.042+0000 7fb16d3e4700 1 -- 172.21.15.16:0/253971375 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fb168094b60 con 0x7fb168156c80 2019-12-05T06:07:08.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.042+0000 7fb167fff700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.042+0000 7fb14ffff700 1 -- 172.21.15.16:0/253971375 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb16002b080 con 0x7fb168156c80 2019-12-05T06:07:08.042 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:07:08.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.045+0000 7fb16d3e4700 1 -- 172.21.15.16:0/253971375 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb148023630 msgr2=0x7fb148025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.045+0000 7fb16d3e4700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb148023630 0x7fb148025ac0 crc :-1 s=READY pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.045+0000 7fb16d3e4700 1 -- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 msgr2=0x7fb168159080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.045+0000 7fb16d3e4700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 0x7fb168159080 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7fb160010420 tx=0x7fb160010250).stop 2019-12-05T06:07:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.045+0000 7fb16d3e4700 1 -- 172.21.15.16:0/253971375 shutdown_connections 2019-12-05T06:07:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.045+0000 7fb16d3e4700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb148023630 0x7fb148025ac0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.045+0000 7fb16d3e4700 1 --2- 172.21.15.16:0/253971375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb168156c80 0x7fb168159080 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.045+0000 7fb16d3e4700 1 --2- 172.21.15.16:0/253971375 >> v2:172.21.15.76:3300/0 conn(0x7fb168073d80 0x7fb168095460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.046+0000 7fb16d3e4700 1 -- 172.21.15.16:0/253971375 shutdown_connections 2019-12-05T06:07:08.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.046+0000 7fb16d3e4700 1 -- 172.21.15.16:0/253971375 wait complete. 2019-12-05T06:07:08.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.046+0000 7fb16d3e4700 1 -- 172.21.15.16:0/253971375 >> 172.21.15.16:0/253971375 conn(0x7fb168077e10 msgr2=0x7fb16808b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:08.056 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T06:07:08.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.330+0000 7f907daf0700 1 Processor -- start 2019-12-05T06:07:08.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.330+0000 7f907daf0700 1 -- start start 2019-12-05T06:07:08.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.332+0000 7f907daf0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9078086980 0x7f9078086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:08.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.332+0000 7f907daf0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f907808a6b0 0x7f907808ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.332+0000 7f907daf0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9078061a10 con 0x7f907808b060 2019-12-05T06:07:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.332+0000 7f907daf0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9078060bc0 con 0x7f9078086980 2019-12-05T06:07:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.332+0000 7f907daf0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9078060620 con 0x7f907808a6b0 2019-12-05T06:07:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.333+0000 7f9077fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9078086980 0x7f9078086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.333+0000 7f907c88e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f907808b060 0x7f9078064310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60086/0 (socket says 172.21.15.16:60086) 2019-12-05T06:07:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.333+0000 7f907c88e700 1 -- 172.21.15.16:0/3529305798 learned_addr learned my addr 172.21.15.16:0/3529305798 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:08.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.333+0000 7f90777fe700 1 --2- 172.21.15.16:0/3529305798 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f907808a6b0 0x7f907808ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.333+0000 7f9077fff700 1 -- 172.21.15.16:0/3529305798 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f907808a6b0 msgr2=0x7f907808ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.333+0000 7f9077fff700 1 --2- 172.21.15.16:0/3529305798 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f907808a6b0 0x7f907808ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.333+0000 7f9077fff700 1 -- 172.21.15.16:0/3529305798 >> v1:172.21.15.16:6789/0 conn(0x7f907808b060 legacy=0x7f9078064310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.333+0000 7f9076ffd700 1 -- 172.21.15.16:0/3529305798 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2214332429 0 0) 0x7f9070001150 con 0x7f907808b060 2019-12-05T06:07:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f9077fff700 1 -- 172.21.15.16:0/3529305798 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9078063a10 con 0x7f9078086980 2019-12-05T06:07:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f90777fe700 1 --2- 172.21.15.16:0/3529305798 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f907808a6b0 0x7f907808ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:08.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f9077fff700 1 --2- 172.21.15.16:0/3529305798 >> v2:172.21.15.16:3300/0 conn(0x7f9078086980 0x7f9078086d70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f906800b2d0 tx=0x7f906800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f9077fff700 1 --2- 172.21.15.16:0/3529305798 >> v2:172.21.15.16:3300/0 conn(0x7f9078086980 0x7f9078086d70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f906800b2d0 tx=0x7f906800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f9077fff700 1 --2- 172.21.15.16:0/3529305798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9078086980 0x7f9078086d70 secure :-1 s=READY pgs=196 cs=0 l=1 rx=0x7f906800b2d0 tx=0x7f906800bae0).ready entity=mon.0 client_cookie=b01550a5ce7103cb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f9077fff700 1 --2- 172.21.15.16:0/3529305798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9078086980 0x7f9078086d70 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f906800b2d0 tx=0x7f906800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f9077fff700 1 --2- 172.21.15.16:0/3529305798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9078086980 0x7f9078086d70 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f906800b2d0 tx=0x7f906800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f9076ffd700 1 -- 172.21.15.16:0/3529305798 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9068007590 con 0x7f9078086980 2019-12-05T06:07:08.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f9077fff700 1 --2- 172.21.15.16:0/3529305798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9078086980 0x7f9078086d70 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f906800b2d0 tx=0x7f906800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f9076ffd700 1 -- 172.21.15.16:0/3529305798 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9068009d30 con 0x7f9078086980 2019-12-05T06:07:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.334+0000 7f9076ffd700 1 -- 172.21.15.16:0/3529305798 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f906800a400 con 0x7f9078086980 2019-12-05T06:07:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.335+0000 7f907daf0700 1 -- 172.21.15.16:0/3529305798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9078086980 msgr2=0x7f9078086d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.335+0000 7f907daf0700 1 --2- 172.21.15.16:0/3529305798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9078086980 0x7f9078086d70 secure :-1 s=READY pgs=196 cs=0 l=1 rx=0x7f906800b2d0 tx=0x7f906800bae0).stop 2019-12-05T06:07:08.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.336+0000 7f907daf0700 1 -- 172.21.15.16:0/3529305798 shutdown_connections 2019-12-05T06:07:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.336+0000 7f907daf0700 1 --2- 172.21.15.16:0/3529305798 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f907808a6b0 0x7f907808ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.336+0000 7f907daf0700 1 --2- 172.21.15.16:0/3529305798 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9078086980 0x7f9078086d70 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.336+0000 7f907daf0700 1 -- 172.21.15.16:0/3529305798 shutdown_connections 2019-12-05T06:07:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.336+0000 7f907daf0700 1 -- 172.21.15.16:0/3529305798 wait complete. 2019-12-05T06:07:08.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.336+0000 7f907daf0700 1 -- 172.21.15.16:0/3529305798 >> 172.21.15.16:0/3529305798 conn(0x7f907805f5a0 msgr2=0x7f907809cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.343+0000 7f907daf0700 1 Processor -- start 2019-12-05T06:07:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.343+0000 7f907daf0700 1 -- start start 2019-12-05T06:07:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.345+0000 7f907daf0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90780973c0 0x7f9078096ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:08.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.345+0000 7f907daf0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9078099de0 con 0x7f907814c2c0 2019-12-05T06:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.345+0000 7f907daf0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9078095e00 con 0x7f907814c6b0 2019-12-05T06:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.345+0000 7f907daf0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f907809b880 con 0x7f90780973c0 2019-12-05T06:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.346+0000 7f90777fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90780973c0 0x7f9078096ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.346+0000 7f9077fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f907814c2c0 0x7f90780964a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60088/0 (socket says 172.21.15.16:60088) 2019-12-05T06:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.346+0000 7f90777fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90780973c0 0x7f9078096ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46688/0 (socket says 172.21.15.16:46688) 2019-12-05T06:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.346+0000 7f9077fff700 1 -- 172.21.15.16:0/2338517189 learned_addr learned my addr 172.21.15.16:0/2338517189 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:08.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.346+0000 7f9074ff9700 1 -- 172.21.15.16:0/2338517189 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73787500 0 0) 0x7f9070000f40 con 0x7f907814c6b0 2019-12-05T06:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.346+0000 7f9074ff9700 1 -- 172.21.15.16:0/2338517189 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f905c002e80 con 0x7f907814c6b0 2019-12-05T06:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.346+0000 7f9074ff9700 1 -- 172.21.15.16:0/2338517189 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1966028996 0 0) 0x7f906800af30 con 0x7f907814c2c0 2019-12-05T06:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.346+0000 7f90777fe700 1 -- 172.21.15.16:0/2338517189 >> v1:172.21.15.76:6789/0 conn(0x7f907814c6b0 legacy=0x7f9078155fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.346+0000 7f90777fe700 1 -- 172.21.15.16:0/2338517189 >> v1:172.21.15.16:6789/0 conn(0x7f907814c2c0 legacy=0x7f90780964a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.346+0000 7f90777fe700 1 -- 172.21.15.16:0/2338517189 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f907814cad0 con 0x7f90780973c0 2019-12-05T06:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f90777fe700 1 --2- 172.21.15.16:0/2338517189 >> v2:172.21.15.16:3300/0 conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f90777fe700 1 --2- 172.21.15.16:0/2338517189 >> v2:172.21.15.16:3300/0 conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f90777fe700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=READY pgs=198 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f90777fe700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f9074ff9700 1 -- 172.21.15.16:0/2338517189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9060000940 con 0x7f90780973c0 2019-12-05T06:07:08.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f90777fe700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f90777fe700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f9074ff9700 1 -- 172.21.15.16:0/2338517189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9060009770 con 0x7f90780973c0 2019-12-05T06:07:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f9074ff9700 1 -- 172.21.15.16:0/2338517189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9060009e00 con 0x7f90780973c0 2019-12-05T06:07:08.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f907daf0700 1 -- 172.21.15.16:0/2338517189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f907814cad0 con 0x7f90780973c0 2019-12-05T06:07:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.347+0000 7f907daf0700 1 -- 172.21.15.16:0/2338517189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f907814cad0 con 0x7f90780973c0 2019-12-05T06:07:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.348+0000 7f90777fe700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.349+0000 7f9074ff9700 1 -- 172.21.15.16:0/2338517189 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9060009960 con 0x7f90780973c0 2019-12-05T06:07:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.349+0000 7f90777fe700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.350+0000 7f9074ff9700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f905c023680 0x7f905c025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.350+0000 7f9074ff9700 1 -- 172.21.15.16:0/2338517189 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f9060045da0 con 0x7f90780973c0 2019-12-05T06:07:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.350+0000 7f9077fff700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f905c023680 0x7f905c025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.351+0000 7f9077fff700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f905c023680 0x7f905c025af0 crc :-1 s=READY pgs=104 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:08.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.361+0000 7f907daf0700 1 -- 172.21.15.16:0/2338517189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f907809bf10 con 0x7f90780973c0 2019-12-05T06:07:08.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.370+0000 7f90777fe700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.370+0000 7f9074ff9700 1 -- 172.21.15.16:0/2338517189 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9060005180 con 0x7f90780973c0 2019-12-05T06:07:08.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.618+0000 7f907daf0700 1 -- 172.21.15.16:0/2338517189 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f907809bf10 con 0x7f90780973c0 2019-12-05T06:07:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.619+0000 7f90777fe700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.619+0000 7f9074ff9700 1 -- 172.21.15.16:0/2338517189 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f906002adc0 con 0x7f90780973c0 2019-12-05T06:07:08.618 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:07:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.621+0000 7f907daf0700 1 -- 172.21.15.16:0/2338517189 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f905c023680 msgr2=0x7f905c025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.621+0000 7f907daf0700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f905c023680 0x7f905c025af0 crc :-1 s=READY pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.621+0000 7f907daf0700 1 -- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 msgr2=0x7f9078096ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.621+0000 7f907daf0700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 0x7f9078096ca0 secure :-1 s=READY pgs=198 cs=0 l=1 rx=0x7f9060010390 tx=0x7f9060010730).stop 2019-12-05T06:07:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.621+0000 7f907daf0700 1 -- 172.21.15.16:0/2338517189 shutdown_connections 2019-12-05T06:07:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.621+0000 7f907daf0700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f905c023680 0x7f905c025af0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.621+0000 7f907daf0700 1 --2- 172.21.15.16:0/2338517189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f90780973c0 0x7f9078096ca0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.622+0000 7f907daf0700 1 -- 172.21.15.16:0/2338517189 shutdown_connections 2019-12-05T06:07:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.622+0000 7f907daf0700 1 -- 172.21.15.16:0/2338517189 wait complete. 2019-12-05T06:07:08.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.622+0000 7f907daf0700 1 -- 172.21.15.16:0/2338517189 >> 172.21.15.16:0/2338517189 conn(0x7f9078077e10 msgr2=0x7f907808b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:08.635 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T06:07:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.913+0000 7f50d22f4700 1 Processor -- start 2019-12-05T06:07:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.914+0000 7f50d22f4700 1 -- start start 2019-12-05T06:07:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.915+0000 7f50d22f4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50cc086d70 0x7f50cc08aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.915+0000 7f50d22f4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f50cc064310 0x7f50cc064770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.915+0000 7f50d22f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc061a10 con 0x7f50cc086980 2019-12-05T06:07:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.915+0000 7f50d22f4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50cc060bc0 con 0x7f50cc086d70 2019-12-05T06:07:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.915+0000 7f50d22f4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50cc060620 con 0x7f50cc064310 2019-12-05T06:07:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.916+0000 7f50cbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50cc086d70 0x7f50cc08aeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.916+0000 7f50cbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50cc086d70 0x7f50cc08aeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46700/0 (socket says 172.21.15.16:46700) 2019-12-05T06:07:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.916+0000 7f50cbfff700 1 -- 172.21.15.16:0/1567144978 learned_addr learned my addr 172.21.15.16:0/1567144978 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.916+0000 7f50d1092700 1 --2- 172.21.15.16:0/1567144978 >> v2:172.21.15.76:3300/0 conn(0x7f50cc064310 0x7f50cc064770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.916+0000 7f50cbfff700 1 -- 172.21.15.16:0/1567144978 >> v2:172.21.15.76:3300/0 conn(0x7f50cc064310 msgr2=0x7f50cc064770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.917+0000 7f50cbfff700 1 --2- 172.21.15.16:0/1567144978 >> v2:172.21.15.76:3300/0 conn(0x7f50cc064310 0x7f50cc064770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.917+0000 7f50cbfff700 1 -- 172.21.15.16:0/1567144978 >> v1:172.21.15.16:6789/0 conn(0x7f50cc086980 legacy=0x7f50cc08a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.917+0000 7f50cbfff700 1 -- 172.21.15.16:0/1567144978 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc063a10 con 0x7f50cc086d70 2019-12-05T06:07:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.917+0000 7f50d1092700 1 --2- 172.21.15.16:0/1567144978 >> v2:172.21.15.76:3300/0 conn(0x7f50cc064310 0x7f50cc064770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.917+0000 7f50cbfff700 1 --2- 172.21.15.16:0/1567144978 >> v2:172.21.15.16:3300/0 conn(0x7f50cc086d70 0x7f50cc08aeb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f50bc00f740 tx=0x7f50bc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.917+0000 7f50cbfff700 1 --2- 172.21.15.16:0/1567144978 >> v2:172.21.15.16:3300/0 conn(0x7f50cc086d70 0x7f50cc08aeb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f50bc00f740 tx=0x7f50bc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.917+0000 7f50cbfff700 1 --2- 172.21.15.16:0/1567144978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc086d70 0x7f50cc08aeb0 secure :-1 s=READY pgs=200 cs=0 l=1 rx=0x7f50bc00f740 tx=0x7f50bc00fae0).ready entity=mon.0 client_cookie=c7e7e5507363d850 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.917+0000 7f50cbfff700 1 --2- 172.21.15.16:0/1567144978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc086d70 0x7f50cc08aeb0 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f50bc00f740 tx=0x7f50bc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.917+0000 7f50cbfff700 1 --2- 172.21.15.16:0/1567144978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc086d70 0x7f50cc08aeb0 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f50bc00f740 tx=0x7f50bc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.918+0000 7f50cbfff700 1 --2- 172.21.15.16:0/1567144978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc086d70 0x7f50cc08aeb0 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f50bc00f740 tx=0x7f50bc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.917+0000 7f50cb7fe700 1 -- 172.21.15.16:0/1567144978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50bc000960 con 0x7f50cc086d70 2019-12-05T06:07:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.918+0000 7f50cb7fe700 1 -- 172.21.15.16:0/1567144978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc000c90 con 0x7f50cc086d70 2019-12-05T06:07:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.918+0000 7f50cb7fe700 1 -- 172.21.15.16:0/1567144978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50bc0012a0 con 0x7f50cc086d70 2019-12-05T06:07:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.918+0000 7f50d22f4700 1 -- 172.21.15.16:0/1567144978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc086d70 msgr2=0x7f50cc08aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.918+0000 7f50d22f4700 1 --2- 172.21.15.16:0/1567144978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc086d70 0x7f50cc08aeb0 secure :-1 s=READY pgs=200 cs=0 l=1 rx=0x7f50bc00f740 tx=0x7f50bc00fae0).stop 2019-12-05T06:07:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.920+0000 7f50d22f4700 1 -- 172.21.15.16:0/1567144978 shutdown_connections 2019-12-05T06:07:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.920+0000 7f50d22f4700 1 --2- 172.21.15.16:0/1567144978 >> v2:172.21.15.76:3300/0 conn(0x7f50cc064310 0x7f50cc064770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.920+0000 7f50d22f4700 1 --2- 172.21.15.16:0/1567144978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50cc086d70 0x7f50cc08aeb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.920+0000 7f50d22f4700 1 -- 172.21.15.16:0/1567144978 shutdown_connections 2019-12-05T06:07:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.920+0000 7f50d22f4700 1 -- 172.21.15.16:0/1567144978 wait complete. 2019-12-05T06:07:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.920+0000 7f50d22f4700 1 -- 172.21.15.16:0/1567144978 >> 172.21.15.16:0/1567144978 conn(0x7f50cc05f5a0 msgr2=0x7f50cc09cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.925+0000 7f50d22f4700 1 Processor -- start 2019-12-05T06:07:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.925+0000 7f50d22f4700 1 -- start start 2019-12-05T06:07:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.927+0000 7f50d22f4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f50cc09b1a0 0x7f50cc197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.927+0000 7f50d22f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc12bb00 con 0x7f50cc198090 2019-12-05T06:07:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.927+0000 7f50d22f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50cc096d00 con 0x7f50cc13af90 2019-12-05T06:07:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.927+0000 7f50d22f4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50cc099a10 con 0x7f50cc09b1a0 2019-12-05T06:07:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.927+0000 7f50d1092700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f50cc198090 0x7f50cc19a490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60108/0 (socket says 172.21.15.16:60108) 2019-12-05T06:07:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.927+0000 7f50d1092700 1 -- 172.21.15.16:0/3209170202 learned_addr learned my addr 172.21.15.16:0/3209170202 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.927+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> v2:172.21.15.76:3300/0 conn(0x7f50cc09b1a0 0x7f50cc197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.928+0000 7f50c97fa700 1 -- 172.21.15.16:0/3209170202 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3379496306 0 0) 0x7f50b40095b0 con 0x7f50cc198090 2019-12-05T06:07:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.928+0000 7f50c97fa700 1 -- 172.21.15.16:0/3209170202 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b0002e80 con 0x7f50cc198090 2019-12-05T06:07:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.928+0000 7f50cbfff700 1 -- 172.21.15.16:0/3209170202 >> v1:172.21.15.76:6789/0 conn(0x7f50cc13af90 legacy=0x7f50cc09a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.928+0000 7f50cbfff700 1 -- 172.21.15.16:0/3209170202 >> v1:172.21.15.16:6789/0 conn(0x7f50cc198090 legacy=0x7f50cc19a490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.928+0000 7f50cbfff700 1 -- 172.21.15.16:0/3209170202 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50cc09a0a0 con 0x7f50cc09b1a0 2019-12-05T06:07:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.928+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> v2:172.21.15.76:3300/0 conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.929+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> v2:172.21.15.76:3300/0 conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.929+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=READY pgs=168 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.929+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.929+0000 7f50c97fa700 1 -- 172.21.15.16:0/3209170202 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50bc000940 con 0x7f50cc09b1a0 2019-12-05T06:07:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.929+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.929+0000 7f50c97fa700 1 -- 172.21.15.16:0/3209170202 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50bc000940 con 0x7f50cc09b1a0 2019-12-05T06:07:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.929+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.929+0000 7f50d22f4700 1 -- 172.21.15.16:0/3209170202 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50cc09a0a0 con 0x7f50cc09b1a0 2019-12-05T06:07:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.929+0000 7f50c97fa700 1 -- 172.21.15.16:0/3209170202 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50bc001140 con 0x7f50cc09b1a0 2019-12-05T06:07:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.929+0000 7f50d22f4700 1 -- 172.21.15.16:0/3209170202 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50cc09a0a0 con 0x7f50cc09b1a0 2019-12-05T06:07:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.930+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.932+0000 7f50c97fa700 1 -- 172.21.15.16:0/3209170202 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f50bc000d60 con 0x7f50cc09b1a0 2019-12-05T06:07:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.932+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.932+0000 7f50c97fa700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f50b0023670 0x7f50b0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:08.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.932+0000 7f50c97fa700 1 -- 172.21.15.16:0/3209170202 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f50bc045e30 con 0x7f50cc09b1a0 2019-12-05T06:07:08.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.932+0000 7f50d0891700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f50b0023670 0x7f50b0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:08.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.933+0000 7f50d0891700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f50b0023670 0x7f50b0025ae0 crc :-1 s=READY pgs=105 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:08.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.945+0000 7f50d22f4700 1 -- 172.21.15.16:0/3209170202 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50cc09a0a0 con 0x7f50cc09b1a0 2019-12-05T06:07:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.955+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:08.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:08.955+0000 7f50c97fa700 1 -- 172.21.15.16:0/3209170202 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f50bc028f20 con 0x7f50cc09b1a0 2019-12-05T06:07:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.208+0000 7f50d22f4700 1 -- 172.21.15.16:0/3209170202 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7f50cc09a0a0 con 0x7f50cc09b1a0 2019-12-05T06:07:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.209+0000 7f50cbfff700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=THROTTLE_DONE pgs=168 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.209+0000 7f50c97fa700 1 -- 172.21.15.16:0/3209170202 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7f50bc028f20 con 0x7f50cc09b1a0 2019-12-05T06:07:09.210 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:07:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.214+0000 7f50d22f4700 1 -- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f50b0023670 msgr2=0x7f50b0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.214+0000 7f50d22f4700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f50b0023670 0x7f50b0025ae0 crc :-1 s=READY pgs=105 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.214+0000 7f50d22f4700 1 -- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 msgr2=0x7f50cc197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.214+0000 7f50d22f4700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 0x7f50cc197b50 secure :-1 s=READY pgs=168 cs=0 l=1 rx=0x7f50bc0103a0 tx=0x7f50bc010740).stop 2019-12-05T06:07:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.215+0000 7f50d22f4700 1 -- 172.21.15.16:0/3209170202 shutdown_connections 2019-12-05T06:07:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.215+0000 7f50d22f4700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f50b0023670 0x7f50b0025ae0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.215+0000 7f50d22f4700 1 --2- 172.21.15.16:0/3209170202 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f50cc09b1a0 0x7f50cc197b50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.215+0000 7f50d22f4700 1 -- 172.21.15.16:0/3209170202 shutdown_connections 2019-12-05T06:07:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.215+0000 7f50d22f4700 1 -- 172.21.15.16:0/3209170202 wait complete. 2019-12-05T06:07:09.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.215+0000 7f50d22f4700 1 -- 172.21.15.16:0/3209170202 >> 172.21.15.16:0/3209170202 conn(0x7f50cc08b790 msgr2=0x7f50cc08a210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:09.224 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:09.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.458+0000 7f7a4d1dd700 1 Processor -- start 2019-12-05T06:07:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.458+0000 7f7a4d1dd700 1 -- start start 2019-12-05T06:07:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.459+0000 7f7a4d1dd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 0x7f7a4808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.459+0000 7f7a4d1dd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7a4808fef0 0x7f7a4808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.459+0000 7f7a4d1dd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a480958e0 con 0x7f7a4808ef20 2019-12-05T06:07:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.459+0000 7f7a4d1dd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a480a3f70 con 0x7f7a4808fef0 2019-12-05T06:07:09.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.459+0000 7f7a4d1dd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4808d830 con 0x7f7a4808c6d0 2019-12-05T06:07:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.460+0000 7f7a4c9dc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a4808ef20 0x7f7a4808f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60116/0 (socket says 172.21.15.16:60116) 2019-12-05T06:07:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.460+0000 7f7a4c9dc700 1 -- 172.21.15.16:0/3839097832 learned_addr learned my addr 172.21.15.16:0/3839097832 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.460+0000 7f7a46d9d700 1 --2- 172.21.15.16:0/3839097832 >> v2:172.21.15.76:3300/0 conn(0x7f7a4808fef0 0x7f7a4808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.460+0000 7f7a4759e700 1 --2- 172.21.15.16:0/3839097832 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 0x7f7a4808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.460+0000 7f7a4659c700 1 -- 172.21.15.16:0/3839097832 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126974535 0 0) 0x7f7a40001150 con 0x7f7a4808ef20 2019-12-05T06:07:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.460+0000 7f7a4659c700 1 -- 172.21.15.16:0/3839097832 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a28002e80 con 0x7f7a4808ef20 2019-12-05T06:07:09.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.461+0000 7f7a4759e700 1 -- 172.21.15.16:0/3839097832 >> v2:172.21.15.76:3300/0 conn(0x7f7a4808fef0 msgr2=0x7f7a4808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.461+0000 7f7a4759e700 1 --2- 172.21.15.16:0/3839097832 >> v2:172.21.15.76:3300/0 conn(0x7f7a4808fef0 0x7f7a4808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.461+0000 7f7a4759e700 1 -- 172.21.15.16:0/3839097832 >> v1:172.21.15.16:6789/0 conn(0x7f7a4808ef20 legacy=0x7f7a4808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.461+0000 7f7a4759e700 1 -- 172.21.15.16:0/3839097832 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4808d540 con 0x7f7a4808c6d0 2019-12-05T06:07:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.461+0000 7f7a46d9d700 1 --2- 172.21.15.16:0/3839097832 >> v2:172.21.15.76:3300/0 conn(0x7f7a4808fef0 0x7f7a4808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.461+0000 7f7a4759e700 1 --2- 172.21.15.16:0/3839097832 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 0x7f7a4808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7a3000b2d0 tx=0x7f7a3000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.461+0000 7f7a4759e700 1 --2- 172.21.15.16:0/3839097832 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 0x7f7a4808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7a3000b2d0 tx=0x7f7a3000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.461+0000 7f7a4759e700 1 --2- 172.21.15.16:0/3839097832 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 0x7f7a4808cac0 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7f7a3000b2d0 tx=0x7f7a3000bae0).ready entity=mon.2 client_cookie=2c683d70ad9304de server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.461+0000 7f7a4759e700 1 --2- 172.21.15.16:0/3839097832 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 0x7f7a4808cac0 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f7a3000b2d0 tx=0x7f7a3000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.461+0000 7f7a4659c700 1 -- 172.21.15.16:0/3839097832 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a30007590 con 0x7f7a4808c6d0 2019-12-05T06:07:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.462+0000 7f7a4759e700 1 --2- 172.21.15.16:0/3839097832 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 0x7f7a4808cac0 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f7a3000b2d0 tx=0x7f7a3000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.462+0000 7f7a4759e700 1 --2- 172.21.15.16:0/3839097832 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 0x7f7a4808cac0 secure :-1 s=THROTTLE_DONE pgs=54 cs=0 l=1 rx=0x7f7a3000b2d0 tx=0x7f7a3000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.462+0000 7f7a4659c700 1 -- 172.21.15.16:0/3839097832 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a30009d30 con 0x7f7a4808c6d0 2019-12-05T06:07:09.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.462+0000 7f7a4659c700 1 -- 172.21.15.16:0/3839097832 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a3000a400 con 0x7f7a4808c6d0 2019-12-05T06:07:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.462+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/3839097832 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 msgr2=0x7f7a4808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.462+0000 7f7a4d1dd700 1 --2- 172.21.15.16:0/3839097832 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 0x7f7a4808cac0 secure :-1 s=READY pgs=54 cs=0 l=1 rx=0x7f7a3000b2d0 tx=0x7f7a3000bae0).stop 2019-12-05T06:07:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.463+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/3839097832 shutdown_connections 2019-12-05T06:07:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.463+0000 7f7a4d1dd700 1 --2- 172.21.15.16:0/3839097832 >> v2:172.21.15.76:3300/0 conn(0x7f7a4808fef0 0x7f7a4808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:09.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.463+0000 7f7a4d1dd700 1 --2- 172.21.15.16:0/3839097832 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4808c6d0 0x7f7a4808cac0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.464+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/3839097832 shutdown_connections 2019-12-05T06:07:09.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.464+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/3839097832 wait complete. 2019-12-05T06:07:09.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.464+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/3839097832 >> 172.21.15.16:0/3839097832 conn(0x7f7a48061940 msgr2=0x7f7a48096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:09.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.467+0000 7f7a4d1dd700 1 Processor -- start 2019-12-05T06:07:09.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.467+0000 7f7a4d1dd700 1 -- start start 2019-12-05T06:07:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.468+0000 7f7a4d1dd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4809c700 0x7f7a4809bf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:09.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.468+0000 7f7a4d1dd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7a48156d70 0x7f7a48159170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.468+0000 7f7a4d1dd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4813d490 con 0x7f7a48082680 2019-12-05T06:07:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.468+0000 7f7a4d1dd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4813f150 con 0x7f7a48156d70 2019-12-05T06:07:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.468+0000 7f7a4d1dd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4813e720 con 0x7f7a4809c700 2019-12-05T06:07:09.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.469+0000 7f7a46d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4809c700 0x7f7a4809bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.469+0000 7f7a4c9dc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7a48156d70 0x7f7a48159170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.469+0000 7f7a46d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4809c700 0x7f7a4809bf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:50710/0 (socket says 172.21.15.16:50710) 2019-12-05T06:07:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.469+0000 7f7a46d9d700 1 -- 172.21.15.16:0/2093699932 learned_addr learned my addr 172.21.15.16:0/2093699932 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:09.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a4c9dc700 1 -- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4809c700 msgr2=0x7f7a4809bf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4809c700 0x7f7a4809bf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a4c9dc700 1 -- 172.21.15.16:0/2093699932 >> v1:172.21.15.76:6789/0 conn(0x7f7a48082680 legacy=0x7f7a4809b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a3ffff700 1 -- 172.21.15.16:0/2093699932 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801135925 0 0) 0x7f7a3000b140 con 0x7f7a48082680 2019-12-05T06:07:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a4c9dc700 1 -- 172.21.15.16:0/2093699932 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4813f7e0 con 0x7f7a48156d70 2019-12-05T06:07:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a46d9d700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4809c700 0x7f7a4809bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:09.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> v2:172.21.15.76:3300/0 conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> v2:172.21.15.76:3300/0 conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=READY pgs=170 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a3ffff700 1 -- 172.21.15.16:0/2093699932 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a40000ef0 con 0x7f7a48156d70 2019-12-05T06:07:09.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.470+0000 7f7a3ffff700 1 -- 172.21.15.16:0/2093699932 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a40004430 con 0x7f7a48156d70 2019-12-05T06:07:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.471+0000 7f7a3ffff700 1 -- 172.21.15.16:0/2093699932 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a40004b80 con 0x7f7a48156d70 2019-12-05T06:07:09.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.471+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/2093699932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4813f7e0 con 0x7f7a48156d70 2019-12-05T06:07:09.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.471+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/2093699932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4813f7e0 con 0x7f7a48156d70 2019-12-05T06:07:09.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.472+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.473+0000 7f7a3ffff700 1 -- 172.21.15.16:0/2093699932 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7a40004620 con 0x7f7a48156d70 2019-12-05T06:07:09.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.473+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.473+0000 7f7a3ffff700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a20023630 0x7f7a20025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:09.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.473+0000 7f7a3ffff700 1 -- 172.21.15.16:0/2093699932 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f7a40045d20 con 0x7f7a48156d70 2019-12-05T06:07:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.473+0000 7f7a4759e700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a20023630 0x7f7a20025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.474+0000 7f7a4759e700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a20023630 0x7f7a20025ac0 crc :-1 s=READY pgs=106 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.475+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/2093699932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4813f7e0 con 0x7f7a48156d70 2019-12-05T06:07:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.484+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.485+0000 7f7a3ffff700 1 -- 172.21.15.16:0/2093699932 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7a40004330 con 0x7f7a48156d70 2019-12-05T06:07:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.738+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/2093699932 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a4813f7e0 con 0x7f7a48156d70 2019-12-05T06:07:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.739+0000 7f7a4c9dc700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.739+0000 7f7a3ffff700 1 -- 172.21.15.16:0/2093699932 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2102 (secure 0 0 0) 0x7f7a400144c0 con 0x7f7a48156d70 2019-12-05T06:07:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2019-12-05T06:07:09.740 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:07:09.740 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:06:45.694298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.743+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a20023630 msgr2=0x7f7a20025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.743+0000 7f7a4d1dd700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a20023630 0x7f7a20025ac0 crc :-1 s=READY pgs=106 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.743+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 msgr2=0x7f7a48159170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.743+0000 7f7a4d1dd700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 0x7f7a48159170 secure :-1 s=READY pgs=170 cs=0 l=1 rx=0x7f7a40010370 tx=0x7f7a40010710).stop 2019-12-05T06:07:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.743+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/2093699932 shutdown_connections 2019-12-05T06:07:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.743+0000 7f7a4d1dd700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a20023630 0x7f7a20025ac0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.743+0000 7f7a4d1dd700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a48156d70 0x7f7a48159170 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.743+0000 7f7a4d1dd700 1 --2- 172.21.15.16:0/2093699932 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4809c700 0x7f7a4809bf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.744+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/2093699932 shutdown_connections 2019-12-05T06:07:09.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.744+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/2093699932 wait complete. 2019-12-05T06:07:09.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.744+0000 7f7a4d1dd700 1 -- 172.21.15.16:0/2093699932 >> 172.21.15.16:0/2093699932 conn(0x7f7a4809ab40 msgr2=0x7f7a48099f80 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:09.753 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:07:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.979+0000 7f3e6c10c700 1 Processor -- start 2019-12-05T06:07:09.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.979+0000 7f3e6c10c700 1 -- start start 2019-12-05T06:07:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.981+0000 7f3e6c10c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3e6408d2c0 0x7f3e6408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.981+0000 7f3e6c10c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e640958e0 con 0x7f3e6408c6d0 2019-12-05T06:07:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.981+0000 7f3e6c10c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e640629e0 con 0x7f3e6408fef0 2019-12-05T06:07:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.981+0000 7f3e6c10c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e6408df90 con 0x7f3e6408d2c0 2019-12-05T06:07:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.981+0000 7f3e6a6a9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e6408c6d0 0x7f3e6408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60126/0 (socket says 172.21.15.16:60126) 2019-12-05T06:07:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.981+0000 7f3e6a6a9700 1 -- 172.21.15.16:0/3614292848 learned_addr learned my addr 172.21.15.16:0/3614292848 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:09.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.981+0000 7f3e69ea8700 1 --2- 172.21.15.16:0/3614292848 >> v2:172.21.15.76:3300/0 conn(0x7f3e6408d2c0 0x7f3e6408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.981+0000 7f3e696a7700 1 -- 172.21.15.16:0/3614292848 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2666789593 0 0) 0x7f3e5c0014d0 con 0x7f3e6408c6d0 2019-12-05T06:07:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e696a7700 1 -- 172.21.15.16:0/3614292848 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e4c002e80 con 0x7f3e6408c6d0 2019-12-05T06:07:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e69ea8700 1 -- 172.21.15.16:0/3614292848 >> v1:172.21.15.76:6789/0 conn(0x7f3e6408fef0 legacy=0x7f3e6408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e69ea8700 1 -- 172.21.15.16:0/3614292848 >> v1:172.21.15.16:6789/0 conn(0x7f3e6408c6d0 legacy=0x7f3e6408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e69ea8700 1 -- 172.21.15.16:0/3614292848 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6408e620 con 0x7f3e6408d2c0 2019-12-05T06:07:09.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e69ea8700 1 --2- 172.21.15.16:0/3614292848 >> v2:172.21.15.76:3300/0 conn(0x7f3e6408d2c0 0x7f3e6408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3e5800af50 tx=0x7f3e5800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e69ea8700 1 --2- 172.21.15.16:0/3614292848 >> v2:172.21.15.76:3300/0 conn(0x7f3e6408d2c0 0x7f3e6408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3e5800af50 tx=0x7f3e5800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e69ea8700 1 --2- 172.21.15.16:0/3614292848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6408d2c0 0x7f3e6408ef20 secure :-1 s=READY pgs=172 cs=0 l=1 rx=0x7f3e5800af50 tx=0x7f3e5800b760).ready entity=mon.1 client_cookie=abce1ebd2352570d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e69ea8700 1 --2- 172.21.15.16:0/3614292848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6408d2c0 0x7f3e6408ef20 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7f3e5800af50 tx=0x7f3e5800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e69ea8700 1 --2- 172.21.15.16:0/3614292848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6408d2c0 0x7f3e6408ef20 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7f3e5800af50 tx=0x7f3e5800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e69ea8700 1 --2- 172.21.15.16:0/3614292848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6408d2c0 0x7f3e6408ef20 secure :-1 s=THROTTLE_DONE pgs=172 cs=0 l=1 rx=0x7f3e5800af50 tx=0x7f3e5800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.982+0000 7f3e696a7700 1 -- 172.21.15.16:0/3614292848 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e58007210 con 0x7f3e6408d2c0 2019-12-05T06:07:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.983+0000 7f3e696a7700 1 -- 172.21.15.16:0/3614292848 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e580099b0 con 0x7f3e6408d2c0 2019-12-05T06:07:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.983+0000 7f3e696a7700 1 -- 172.21.15.16:0/3614292848 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e58009fc0 con 0x7f3e6408d2c0 2019-12-05T06:07:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.983+0000 7f3e6c10c700 1 -- 172.21.15.16:0/3614292848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6408d2c0 msgr2=0x7f3e6408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.983+0000 7f3e6c10c700 1 --2- 172.21.15.16:0/3614292848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6408d2c0 0x7f3e6408ef20 secure :-1 s=READY pgs=172 cs=0 l=1 rx=0x7f3e5800af50 tx=0x7f3e5800b760).stop 2019-12-05T06:07:09.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.984+0000 7f3e6c10c700 1 -- 172.21.15.16:0/3614292848 shutdown_connections 2019-12-05T06:07:09.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.984+0000 7f3e6c10c700 1 --2- 172.21.15.16:0/3614292848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6408d2c0 0x7f3e6408ef20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.984+0000 7f3e6c10c700 1 -- 172.21.15.16:0/3614292848 shutdown_connections 2019-12-05T06:07:09.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.984+0000 7f3e6c10c700 1 -- 172.21.15.16:0/3614292848 wait complete. 2019-12-05T06:07:09.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.984+0000 7f3e6c10c700 1 -- 172.21.15.16:0/3614292848 >> 172.21.15.16:0/3614292848 conn(0x7f3e64061940 msgr2=0x7f3e64096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.987+0000 7f3e6c10c700 1 Processor -- start 2019-12-05T06:07:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.987+0000 7f3e6c10c700 1 -- start start 2019-12-05T06:07:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.988+0000 7f3e6c10c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3e6406dc50 0x7f3e6406e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.988+0000 7f3e6c10c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e64135450 0x7f3e6406e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.988+0000 7f3e6c10c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e641350e0 con 0x7f3e64134480 2019-12-05T06:07:09.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.988+0000 7f3e6c10c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e64132830 con 0x7f3e6406dc50 2019-12-05T06:07:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.988+0000 7f3e6c10c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e641334f0 con 0x7f3e64135450 2019-12-05T06:07:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.988+0000 7f3e6aeaa700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e64134480 0x7f3e641348e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60132/0 (socket says 172.21.15.16:60132) 2019-12-05T06:07:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.989+0000 7f3e6aeaa700 1 -- 172.21.15.16:0/492245201 learned_addr learned my addr 172.21.15.16:0/492245201 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.989+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> v2:172.21.15.76:3300/0 conn(0x7f3e6406dc50 0x7f3e6406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.989+0000 7f3e69ea8700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e64135450 0x7f3e6406e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.989+0000 7f3e62ffd700 1 -- 172.21.15.16:0/492245201 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882388177 0 0) 0x7f3e54000b90 con 0x7f3e64134480 2019-12-05T06:07:09.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.989+0000 7f3e62ffd700 1 -- 172.21.15.16:0/492245201 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e44002e80 con 0x7f3e64134480 2019-12-05T06:07:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.989+0000 7f3e6a6a9700 1 -- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e64135450 msgr2=0x7f3e6406e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.989+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e64135450 0x7f3e6406e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.989+0000 7f3e6a6a9700 1 -- 172.21.15.16:0/492245201 >> v1:172.21.15.16:6789/0 conn(0x7f3e64134480 legacy=0x7f3e641348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.989+0000 7f3e6a6a9700 1 -- 172.21.15.16:0/492245201 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e64133b80 con 0x7f3e6406dc50 2019-12-05T06:07:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.989+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> v2:172.21.15.76:3300/0 conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.990+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> v2:172.21.15.76:3300/0 conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.990+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.990+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.990+0000 7f3e62ffd700 1 -- 172.21.15.16:0/492245201 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e5c001010 con 0x7f3e6406dc50 2019-12-05T06:07:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.990+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.990+0000 7f3e62ffd700 1 -- 172.21.15.16:0/492245201 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5c001760 con 0x7f3e6406dc50 2019-12-05T06:07:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.990+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.990+0000 7f3e6c10c700 1 -- 172.21.15.16:0/492245201 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e64133b80 con 0x7f3e6406dc50 2019-12-05T06:07:09.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.990+0000 7f3e62ffd700 1 -- 172.21.15.16:0/492245201 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e5c001950 con 0x7f3e6406dc50 2019-12-05T06:07:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.990+0000 7f3e6c10c700 1 -- 172.21.15.16:0/492245201 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e64133b80 con 0x7f3e6406dc50 2019-12-05T06:07:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.991+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.992+0000 7f3e62ffd700 1 -- 172.21.15.16:0/492245201 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3e5c001b60 con 0x7f3e6406dc50 2019-12-05T06:07:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.992+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.992+0000 7f3e62ffd700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e44023670 0x7f3e44025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.992+0000 7f3e62ffd700 1 -- 172.21.15.16:0/492245201 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f3e5c046070 con 0x7f3e6406dc50 2019-12-05T06:07:09.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.992+0000 7f3e69ea8700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e44023670 0x7f3e44025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.993+0000 7f3e69ea8700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e44023670 0x7f3e44025ae0 crc :-1 s=READY pgs=107 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:09.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:09.995+0000 7f3e6c10c700 1 -- 172.21.15.16:0/492245201 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e64133b80 con 0x7f3e6406dc50 2019-12-05T06:07:10.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.005+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.005+0000 7f3e62ffd700 1 -- 172.21.15.16:0/492245201 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3e5c001950 con 0x7f3e6406dc50 2019-12-05T06:07:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.256+0000 7f3e6c10c700 1 -- 172.21.15.16:0/492245201 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3e64133b80 con 0x7f3e6406dc50 2019-12-05T06:07:10.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.258+0000 7f3e6a6a9700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=THROTTLE_DONE pgs=173 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.258+0000 7f3e62ffd700 1 -- 172.21.15.16:0/492245201 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7f3e5c001b60 con 0x7f3e6406dc50 2019-12-05T06:07:10.258 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:07:10.258 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:07:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.261+0000 7f3e6c10c700 1 -- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e44023670 msgr2=0x7f3e44025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.261+0000 7f3e6c10c700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e44023670 0x7f3e44025ae0 crc :-1 s=READY pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.261+0000 7f3e6c10c700 1 -- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 msgr2=0x7f3e6406e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.261+0000 7f3e6c10c700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 0x7f3e6406e040 secure :-1 s=READY pgs=173 cs=0 l=1 rx=0x7f3e5c010700 tx=0x7f3e5c010aa0).stop 2019-12-05T06:07:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.262+0000 7f3e6c10c700 1 -- 172.21.15.16:0/492245201 shutdown_connections 2019-12-05T06:07:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.262+0000 7f3e6c10c700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e44023670 0x7f3e44025ae0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.262+0000 7f3e6c10c700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e64135450 0x7f3e6406e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.262+0000 7f3e6c10c700 1 --2- 172.21.15.16:0/492245201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e6406dc50 0x7f3e6406e040 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.262+0000 7f3e6c10c700 1 -- 172.21.15.16:0/492245201 shutdown_connections 2019-12-05T06:07:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.262+0000 7f3e6c10c700 1 -- 172.21.15.16:0/492245201 wait complete. 2019-12-05T06:07:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.262+0000 7f3e6c10c700 1 -- 172.21.15.16:0/492245201 >> 172.21.15.16:0/492245201 conn(0x7f3e640a2850 msgr2=0x7f3e6406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:10.273 DEBUG:tasks.mds_thrash:Yielding 2019-12-05T06:07:10.273 INFO:teuthology.run_tasks:Running task ceph-fuse... 2019-12-05T06:07:10.312 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2019-12-05T06:07:10.312 INFO:tasks.ceph_fuse:config is None 2019-12-05T06:07:10.313 INFO:tasks.ceph_fuse:new config is {'client.0': None} 2019-12-05T06:07:10.313 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2019-12-05T06:07:10.315 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2019-12-05T06:07:10.315 INFO:tasks.mds_thrash.fs.[cephfs]:starting mds_do_thrash for fs cephfs 2019-12-05T06:07:10.316 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 80 secs before thrashing 2019-12-05T06:07:10.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.565+0000 7fc6f88e9700 1 Processor -- start 2019-12-05T06:07:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.565+0000 7fc6f88e9700 1 -- start start 2019-12-05T06:07:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.567+0000 7fc6f88e9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f408c6d0 0x7fc6f408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.567+0000 7fc6f88e9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc6f408f720 0x7fc6f408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.567+0000 7fc6f88e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f40958e0 con 0x7fc6f408fef0 2019-12-05T06:07:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.567+0000 7fc6f88e9700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6f40a3f70 con 0x7fc6f408f720 2019-12-05T06:07:10.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.567+0000 7fc6f88e9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f408d830 con 0x7fc6f408c6d0 2019-12-05T06:07:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.567+0000 7fc6f2d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f408c6d0 0x7fc6f408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.567+0000 7fc6f359e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc6f408f720 0x7fc6f408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.567+0000 7fc6f2d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f408c6d0 0x7fc6f408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:50730/0 (socket says 172.21.15.16:50730) 2019-12-05T06:07:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.567+0000 7fc6f259c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc6f408fef0 0x7fc6f408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53212/0 (socket says 172.21.15.16:53212) 2019-12-05T06:07:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.567+0000 7fc6f2d9d700 1 -- 172.21.15.16:0/1023819880 learned_addr learned my addr 172.21.15.16:0/1023819880 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.568+0000 7fc6f1d9b700 1 -- 172.21.15.16:0/1023819880 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635812314 0 0) 0x7fc6dc001080 con 0x7fc6f408fef0 2019-12-05T06:07:10.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.568+0000 7fc6f1d9b700 1 -- 172.21.15.16:0/1023819880 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6e0002e80 con 0x7fc6f408fef0 2019-12-05T06:07:10.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.568+0000 7fc6f359e700 1 -- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f408c6d0 msgr2=0x7fc6f408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:10.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.568+0000 7fc6f359e700 1 --2- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f408c6d0 0x7fc6f408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:10.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.568+0000 7fc6f359e700 1 -- 172.21.15.16:0/1023819880 >> v1:172.21.15.76:6789/0 conn(0x7fc6f408fef0 legacy=0x7fc6f408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:10.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.568+0000 7fc6f359e700 1 -- 172.21.15.16:0/1023819880 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f408fbc0 con 0x7fc6f408f720 2019-12-05T06:07:10.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.568+0000 7fc6f2d9d700 1 --2- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f408c6d0 0x7fc6f408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:10.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.568+0000 7fc6f359e700 1 --2- 172.21.15.16:0/1023819880 >> v2:172.21.15.76:3300/0 conn(0x7fc6f408f720 0x7fc6f408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc6e400af80 tx=0x7fc6e400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.569+0000 7fc6f359e700 1 --2- 172.21.15.16:0/1023819880 >> v2:172.21.15.76:3300/0 conn(0x7fc6f408f720 0x7fc6f408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc6e400af80 tx=0x7fc6e400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.569+0000 7fc6f359e700 1 --2- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6f408f720 0x7fc6f408d000 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7fc6e400af80 tx=0x7fc6e400b790).ready entity=mon.1 client_cookie=85322ec22e711e3f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.569+0000 7fc6f359e700 1 --2- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6f408f720 0x7fc6f408d000 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7fc6e400af80 tx=0x7fc6e400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.569+0000 7fc6f1d9b700 1 -- 172.21.15.16:0/1023819880 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6e4000bc0 con 0x7fc6f408f720 2019-12-05T06:07:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.569+0000 7fc6f359e700 1 --2- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6f408f720 0x7fc6f408d000 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7fc6e400af80 tx=0x7fc6e400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.569+0000 7fc6f359e700 1 --2- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6f408f720 0x7fc6f408d000 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7fc6e400af80 tx=0x7fc6e400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.569+0000 7fc6f1d9b700 1 -- 172.21.15.16:0/1023819880 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6e40097f0 con 0x7fc6f408f720 2019-12-05T06:07:10.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.569+0000 7fc6f1d9b700 1 -- 172.21.15.16:0/1023819880 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6e4000bc0 con 0x7fc6f408f720 2019-12-05T06:07:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.569+0000 7fc6f88e9700 1 -- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6f408f720 msgr2=0x7fc6f408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.569+0000 7fc6f88e9700 1 --2- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6f408f720 0x7fc6f408d000 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7fc6e400af80 tx=0x7fc6e400b790).stop 2019-12-05T06:07:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.571+0000 7fc6f88e9700 1 -- 172.21.15.16:0/1023819880 shutdown_connections 2019-12-05T06:07:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.571+0000 7fc6f88e9700 1 --2- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6f408f720 0x7fc6f408d000 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.571+0000 7fc6f88e9700 1 --2- 172.21.15.16:0/1023819880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f408c6d0 0x7fc6f408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.572+0000 7fc6f88e9700 1 -- 172.21.15.16:0/1023819880 shutdown_connections 2019-12-05T06:07:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.572+0000 7fc6f88e9700 1 -- 172.21.15.16:0/1023819880 wait complete. 2019-12-05T06:07:10.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.572+0000 7fc6f88e9700 1 -- 172.21.15.16:0/1023819880 >> 172.21.15.16:0/1023819880 conn(0x7fc6f4061940 msgr2=0x7fc6f4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:10.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.575+0000 7fc6f88e9700 1 Processor -- start 2019-12-05T06:07:10.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.575+0000 7fc6f88e9700 1 -- start start 2019-12-05T06:07:10.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.576+0000 7fc6f88e9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:10.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.576+0000 7fc6f88e9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f413ba10 con 0x7fc6f40a0920 2019-12-05T06:07:10.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.576+0000 7fc6f88e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6f413b510 con 0x7fc6f4154b00 2019-12-05T06:07:10.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.576+0000 7fc6f88e9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6f413d6d0 con 0x7fc6f409d8b0 2019-12-05T06:07:10.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.577+0000 7fc6f2d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc6f40a0920 0x7fc6f409c8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60146/0 (socket says 172.21.15.16:60146) 2019-12-05T06:07:10.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.577+0000 7fc6f2d9d700 1 -- 172.21.15.16:0/3262142867 learned_addr learned my addr 172.21.15.16:0/3262142867 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:10.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.577+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:10.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.577+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2716849561 0 0) 0x7fc6ec007d20 con 0x7fc6f40a0920 2019-12-05T06:07:10.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.577+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d8002e80 con 0x7fc6f40a0920 2019-12-05T06:07:10.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.577+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812337338 0 0) 0x7fc6e40095b0 con 0x7fc6f4154b00 2019-12-05T06:07:10.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.577+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6d80030b0 con 0x7fc6f4154b00 2019-12-05T06:07:10.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.578+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1548123873 0 0) 0x7fc6ec000cc0 con 0x7fc6f40a0920 2019-12-05T06:07:10.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.578+0000 7fc6f259c700 1 -- 172.21.15.16:0/3262142867 >> v1:172.21.15.76:6789/0 conn(0x7fc6f4154b00 legacy=0x7fc6f4156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:10.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.578+0000 7fc6f259c700 1 -- 172.21.15.16:0/3262142867 >> v1:172.21.15.16:6789/0 conn(0x7fc6f40a0920 legacy=0x7fc6f409c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:10.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.578+0000 7fc6f259c700 1 -- 172.21.15.16:0/3262142867 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6f413bec0 con 0x7fc6f409d8b0 2019-12-05T06:07:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.578+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.578+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.578+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.579+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.579+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6dc007210 con 0x7fc6f409d8b0 2019-12-05T06:07:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.579+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.579+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6dc001270 con 0x7fc6f409d8b0 2019-12-05T06:07:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.579+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.579+0000 7fc6f88e9700 1 -- 172.21.15.16:0/3262142867 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6f413bec0 con 0x7fc6f409d8b0 2019-12-05T06:07:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.579+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6dc001460 con 0x7fc6f409d8b0 2019-12-05T06:07:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.579+0000 7fc6f88e9700 1 -- 172.21.15.16:0/3262142867 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6f413bec0 con 0x7fc6f409d8b0 2019-12-05T06:07:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.580+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.581+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc6dc000ce0 con 0x7fc6f409d8b0 2019-12-05T06:07:10.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.581+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.581+0000 7fc6eb7fe700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6d8023650 0x7fc6d8025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:10.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.581+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fc6dc045db0 con 0x7fc6f409d8b0 2019-12-05T06:07:10.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.581+0000 7fc6f2d9d700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6d8023650 0x7fc6d8025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:10.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.582+0000 7fc6f2d9d700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6d8023650 0x7fc6d8025ae0 crc :-1 s=READY pgs=108 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:10.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.598+0000 7fc6f88e9700 1 -- 172.21.15.16:0/3262142867 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6f4059ec0 con 0x7fc6f409d8b0 2019-12-05T06:07:10.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.607+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.607+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc6dc026f50 con 0x7fc6f409d8b0 2019-12-05T06:07:10.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.865+0000 7fc6f88e9700 1 -- 172.21.15.16:0/3262142867 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc6f4059ec0 con 0x7fc6f409d8b0 2019-12-05T06:07:10.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.868+0000 7fc6f259c700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=THROTTLE_DONE pgs=55 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:10.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.868+0000 7fc6eb7fe700 1 -- 172.21.15.16:0/3262142867 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v24) v1 ==== 74+0+14885 (secure 0 0 0) 0x7fc6dc026f50 con 0x7fc6f409d8b0 2019-12-05T06:07:10.868 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:07:10.869 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"fsid":"df6db985-de08-463b-af1a-6ee59e0bbbe7","created":"2019-12-05T06:06:13.137224+0000","modified":"2019-12-05T06:06:41.258992+0000","last_up_change":"2019-12-05T06:06:25.924548+0000","last_in_change":"2019-12-05T06:06:25.924548+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"rbd","create_time":"2019-12-05T06:06:31.052682+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2019-12-05T06:06:34.862662+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs_data","create_time":"2019-12-05T06:06:37.644552+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"24","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"9d92c1cd-0a79-4b80-a533-62e750cbe008","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6813","nonce":10292},{"type":"v1","addr":"172.21.15.16:6815","nonce":10292}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6818","nonce":10292},{"type":"v1","addr":"172.21.15.16:6821","nonce":10292}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6826","nonce":10292},{"type":"v1","addr":"172.21.15.16:6828","nonce":10292}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6823","nonce":10292},{"type":"v1","addr":"172.21.15.16:6824","nonce":10292}]},"public_addr":"172.21.15.16:6815/10292","cluster_addr":"172.21.15.16:6821/10292","heartbeat_back_addr":"172.21.15.16:6828/10292","heartbeat_front_addr":"172.21.15.16:6824/10292","state":["exists","up"]},{"osd":1,"uuid":"cde687f3-983d-48c2-ad19-7332d6ffc944","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6800","nonce":10289},{"type":"v1","addr":"172.21.15.16:6801","nonce":10289}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6802","nonce":10289},{"type":"v1","addr":"172.21.15.16:6803","nonce":10289}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6806","nonce":10289},{"type":"v1","addr":"172.21.15.16:6807","nonce":10289}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6804","nonce":10289},{"type":"v1","addr":"172.21.15.16:6805","nonce":10289}]},"public_addr":"172.21.15.16:6801/10289","cluster_addr":"172.21.15.16:6803/10289","heartbeat_back_addr":"172.21.15.16:6807/10289","heartbeat_front_addr":"172.21.15.16:6805/10289","state":["exists","up"]},{"osd":2,"uuid":"b42d078c-c688-49ea-b0d8-14fbbf31dd53","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6808","nonce":10291},{"type":"v1","addr":"172.21.15.16:6809","nonce":10291}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6810","nonce":10291},{"type":"v1","addr":"172.21.15.16:6811","nonce":10291}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6817","nonce":10291},{"type":"v1","addr":"172.21.15.16:6820","nonce":10291}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6812","nonce":10291},{"type":"v1","addr":"172.21.15.16:6814","nonce":10291}]},"public_addr":"172.21.15.16:6809/10291","cluster_addr":"172.21.15.16:6811/10291","heartbeat_back_addr":"172.21.15.16:6820/10291","heartbeat_front_addr":"172.21.15.16:6814/10291","state":["exists","up"]},{"osd":3,"uuid":"41ff9a23-a332-455d-adb0-7bbc69c833b6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6816","nonce":10293},{"type":"v1","addr":"172.21.15.16:6819","nonce":10293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6822","nonce":10293},{"type":"v1","addr":"172.21.15.16:6825","nonce":10293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6830","nonce":10293},{"type":"v1","addr":"172.21.15.16:6831","nonce":10293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6827","nonce":10293},{"type":"v1","addr":"172.21.15.16:6829","nonce":10293}]},"public_addr":"172.21.15.16:6819/10293","cluster_addr":"172.21.15.16:6825/10293","heartbeat_back_addr":"172.21.15.16:6831/10293","heartbeat_front_addr":"172.21.15.16:6829/10293","state":["exists","up"]},{"osd":4,"uuid":"7fb2a124-c603-49d9-ad7c-44c975729335","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":10472},{"type":"v1","addr":"172.21.15.76:6803","nonce":10472}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":10472},{"type":"v1","addr":"172.21.15.76:6805","nonce":10472}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":10472},{"type":"v1","addr":"172.21.15.76:6809","nonce":10472}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":10472},{"type":"v1","addr":"172.21.15.76:6807","nonce":10472}]},"public_addr":"172.21.15.76:6803/10472","cluster_addr":"172.21.15.76:6805/10472","heartbeat_back_addr":"172.21.15.76:6809/10472","heartbeat_front_addr":"172.21.15.76:6807/10472","state":["exists","up"]},{"osd":5,"uuid":"9776ef43-83b4-4461-9d37-438d04ed68ef","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":10473},{"type":"v1","addr":"172.21.15.76:6811","nonce":10473}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":10473},{"type":"v1","addr":"172.21.15.76:6813","nonce":10473}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":10473},{"type":"v1","addr":"172.21.15.76:6820","nonce":10473}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":10473},{"type":"v1","addr":"172.21.15.76:6816","nonce":10473}]},"public_addr":"172.21.15.76:6811/10473","cluster_addr":"172.21.15.76:6813/10473","heartbeat_back_addr":"172.21.15.76:6820/10473","heartbeat_front_addr":"172.21.15.76:6816/10473","state":["exists","up"]},{"osd":6,"uuid":"9ed18a64-0c45-4d39-84ce-67748d102f1c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6815","nonce":10474},{"type":"v1","addr":"172.21.15.76:6817","nonce":10474}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6819","nonce":10474},{"type":"v1","addr":"172.21.15.76:6821","nonce":10474}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6825","nonce":10474},{"type":"v1","addr":"172.21.15.76:6827","nonce":10474}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":10474},{"type":"v1","addr":"172.21.15.76:6823","nonce":10474}]},"public_addr":"172.21.15.76:6817/10474","cluster_addr":"172.21.15.76:6821/10474","heartbeat_back_addr":"172.21.15.76:6827/10474","heartbeat_front_addr":"172.21.15.76:6823/10474","state":["exists","up"]},{"osd":7,"uuid":"78f3fd1b-2fb0-4a3e-acd8-475d77dfac74","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":10475},{"type":"v1","addr":"172.21.15.76:6826","nonce":10475}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":10475},{"type":"v1","addr":"172.21.15.76:6829","nonce":10475}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":10475},{"type":"v1","addr":"172.21.15.76:6833","nonce":10475}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":10475},{"type":"v1","addr":"172.21.15.76:6831","nonce":10475}]},"public_addr":"172.21.15.76:6826/10475","cluster_addr":"172.21.15.76:6829/10475","heartbeat_back_addr":"172.21.15.76:6833/10475","heartbeat_front_addr":"172.21.15.76:6831/10475","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.589393+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.608819+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.605503+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.571300+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.531995+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.541622+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.612257+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292836696063,"old_weight":0,"last_purged_snaps_scrub":"2019-12-05T06:06:24.615292+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"cephfs_data":{"crush-device-class":"","crush-failure-domain":"osd","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2019-12-05T06:07:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.871+0000 7fc6f88e9700 1 -- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6d8023650 msgr2=0x7fc6d8025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:10.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.871+0000 7fc6f88e9700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6d8023650 0x7fc6d8025ae0 crc :-1 s=READY pgs=108 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.871+0000 7fc6f88e9700 1 -- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 msgr2=0x7fc6f409d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.871+0000 7fc6f88e9700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rx=0x7fc6dc0103a0 tx=0x7fc6dc010740).stop 2019-12-05T06:07:10.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.871+0000 7fc6f88e9700 1 -- 172.21.15.16:0/3262142867 shutdown_connections 2019-12-05T06:07:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.871+0000 7fc6f88e9700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6d8023650 0x7fc6d8025ae0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.871+0000 7fc6f88e9700 1 --2- 172.21.15.16:0/3262142867 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc6f409d8b0 0x7fc6f409d0e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.871+0000 7fc6f88e9700 1 -- 172.21.15.16:0/3262142867 shutdown_connections 2019-12-05T06:07:10.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.871+0000 7fc6f88e9700 1 -- 172.21.15.16:0/3262142867 wait complete. 2019-12-05T06:07:10.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:10.871+0000 7fc6f88e9700 1 -- 172.21.15.16:0/3262142867 >> 172.21.15.16:0/3262142867 conn(0x7fc6f4130a60 msgr2=0x7fc6f4086470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:10.882 INFO:tasks.cephfs.filesystem.ceph_manager:[{u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:31.052682+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'rbd', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 1, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'rbd': {}}, u'options': {}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'17', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 0, u'flags_names': u'hashpspool', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:34.862662+0000', u'snap_epoch': 0, u'erasure_code_profile': u'', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 2, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 1, u'grade_table': [], u'pool_name': u'cephfs_metadata', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 1, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 2, u'removed_snaps': u'[]', u'crush_rule': 0, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'metadata': u'cephfs'}}, u'options': {u'pg_autoscale_bias': 4, u'recovery_priority': 5, u'pg_num_min': 16}, u'hit_set_count': 0, u'flags': 1, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'23', u'min_write_recency_for_promote': 0, u'read_tier': -1}, {u'cache_target_full_ratio_micro': 800000, u'fast_read': False, u'stripe_width': 8192, u'flags_names': u'hashpspool,ec_overwrites', u'tier_of': -1, u'hit_set_grade_decay_rate': 0, u'pg_placement_num': 8, u'use_gmt_hitset': True, u'cache_target_dirty_ratio_micro': 400000, u'create_time': u'2019-12-05T06:06:37.644552+0000', u'snap_epoch': 0, u'erasure_code_profile': u'cephfs_data', u'pg_autoscale_mode': u'off', u'snap_seq': 0, u'expected_num_objects': 0, u'size': 4, u'pg_num_pending': 8, u'auid': 0, u'cache_min_flush_age': 0, u'hit_set_period': 0, u'min_read_recency_for_promote': 0, u'target_max_objects': 0, u'pg_placement_num_target': 8, u'pg_num': 8, u'type': 3, u'grade_table': [], u'pool_name': u'cephfs_data', u'cache_min_evict_age': 0, u'snap_mode': u'selfmanaged', u'pg_num_target': 8, u'cache_mode': u'none', u'min_size': 3, u'cache_target_dirty_high_ratio_micro': 600000, u'object_hash': 2, u'last_pg_merge_meta': {u'ready_epoch': 0, u'source_version': u"0'0", u'source_pgid': u'0.0', u'last_epoch_clean': 0, u'target_version': u"0'0", u'last_epoch_started': 0}, u'write_tier': -1, u'last_force_op_resend_preluminous': u'0', u'pool': 3, u'removed_snaps': u'[]', u'crush_rule': 1, u'tiers': [], u'hit_set_params': {u'type': u'none'}, u'last_force_op_resend': u'0', u'pool_snaps': [], u'quota_max_objects': 0, u'last_force_op_resend_prenautilus': u'0', u'application_metadata': {u'cephfs': {u'data': u'cephfs'}}, u'options': {}, u'hit_set_count': 0, u'flags': 5, u'target_max_bytes': 0, u'quota_max_bytes': 0, u'hit_set_search_last_n': 0, u'last_change': u'24', u'min_write_recency_for_promote': 0, u'read_tier': -1}] 2019-12-05T06:07:10.882 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2019-12-05T06:07:11.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.082+0000 7fbd60bfb700 1 Processor -- start 2019-12-05T06:07:11.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.082+0000 7fbd60bfb700 1 -- start start 2019-12-05T06:07:11.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.083+0000 7fbd60bfb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd5c086980 0x7fbd5c086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:11.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.083+0000 7fbd60bfb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd5c064310 0x7fbd5c064790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:11.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.083+0000 7fbd60bfb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5c061a10 con 0x7fbd5c08a6b0 2019-12-05T06:07:11.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.083+0000 7fbd60bfb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5c08b340 con 0x7fbd5c064310 2019-12-05T06:07:11.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.083+0000 7fbd60bfb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd5c060bc0 con 0x7fbd5c086980 2019-12-05T06:07:11.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.083+0000 7fbd5b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd5c064310 0x7fbd5c064790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.084+0000 7fbd5b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd5c064310 0x7fbd5c064790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46754/0 (socket says 172.21.15.16:46754) 2019-12-05T06:07:11.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.084+0000 7fbd5b59e700 1 -- 172.21.15.16:0/1289530465 learned_addr learned my addr 172.21.15.16:0/1289530465 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:11.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.084+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/1289530465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd5c086980 0x7fbd5c086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.084+0000 7fbd5b59e700 1 -- 172.21.15.16:0/1289530465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd5c086980 msgr2=0x7fbd5c086d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:11.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.084+0000 7fbd5b59e700 1 --2- 172.21.15.16:0/1289530465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd5c086980 0x7fbd5c086d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:11.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.084+0000 7fbd5b59e700 1 -- 172.21.15.16:0/1289530465 >> v1:172.21.15.76:6789/0 conn(0x7fbd5c08a6b0 legacy=0x7fbd5c08ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.084+0000 7fbd5b59e700 1 -- 172.21.15.16:0/1289530465 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c063a10 con 0x7fbd5c064310 2019-12-05T06:07:11.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.084+0000 7fbd59d9b700 1 -- 172.21.15.16:0/1289530465 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526758758 0 0) 0x7fbd44001030 con 0x7fbd5c08a6b0 2019-12-05T06:07:11.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd5b59e700 1 --2- 172.21.15.16:0/1289530465 >> v2:172.21.15.16:3300/0 conn(0x7fbd5c064310 0x7fbd5c064790 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbd4c00f6b0 tx=0x7fbd4c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd5b59e700 1 --2- 172.21.15.16:0/1289530465 >> v2:172.21.15.16:3300/0 conn(0x7fbd5c064310 0x7fbd5c064790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbd4c00f6b0 tx=0x7fbd4c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd5b59e700 1 --2- 172.21.15.16:0/1289530465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd5c064310 0x7fbd5c064790 secure :-1 s=READY pgs=206 cs=0 l=1 rx=0x7fbd4c00f6b0 tx=0x7fbd4c00fa50).ready entity=mon.0 client_cookie=7c262186e62f8d9d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:11.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd5b59e700 1 --2- 172.21.15.16:0/1289530465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd5c064310 0x7fbd5c064790 secure :-1 s=THROTTLE_DONE pgs=206 cs=0 l=1 rx=0x7fbd4c00f6b0 tx=0x7fbd4c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd59d9b700 1 -- 172.21.15.16:0/1289530465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd4c000a70 con 0x7fbd5c064310 2019-12-05T06:07:11.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd5b59e700 1 --2- 172.21.15.16:0/1289530465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd5c064310 0x7fbd5c064790 secure :-1 s=THROTTLE_DONE pgs=206 cs=0 l=1 rx=0x7fbd4c00f6b0 tx=0x7fbd4c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd5b59e700 1 --2- 172.21.15.16:0/1289530465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd5c064310 0x7fbd5c064790 secure :-1 s=THROTTLE_DONE pgs=206 cs=0 l=1 rx=0x7fbd4c00f6b0 tx=0x7fbd4c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd59d9b700 1 -- 172.21.15.16:0/1289530465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd4c0010e0 con 0x7fbd5c064310 2019-12-05T06:07:11.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd59d9b700 1 -- 172.21.15.16:0/1289530465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd4c0017d0 con 0x7fbd5c064310 2019-12-05T06:07:11.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd60bfb700 1 -- 172.21.15.16:0/1289530465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd5c064310 msgr2=0x7fbd5c064790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.085+0000 7fbd60bfb700 1 --2- 172.21.15.16:0/1289530465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd5c064310 0x7fbd5c064790 secure :-1 s=READY pgs=206 cs=0 l=1 rx=0x7fbd4c00f6b0 tx=0x7fbd4c00fa50).stop 2019-12-05T06:07:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.088+0000 7fbd60bfb700 1 -- 172.21.15.16:0/1289530465 shutdown_connections 2019-12-05T06:07:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.088+0000 7fbd60bfb700 1 --2- 172.21.15.16:0/1289530465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd5c064310 0x7fbd5c064790 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.088+0000 7fbd60bfb700 1 --2- 172.21.15.16:0/1289530465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd5c086980 0x7fbd5c086d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.088+0000 7fbd60bfb700 1 -- 172.21.15.16:0/1289530465 shutdown_connections 2019-12-05T06:07:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.088+0000 7fbd60bfb700 1 -- 172.21.15.16:0/1289530465 wait complete. 2019-12-05T06:07:11.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.088+0000 7fbd60bfb700 1 -- 172.21.15.16:0/1289530465 >> 172.21.15.16:0/1289530465 conn(0x7fbd5c05f5a0 msgr2=0x7fbd5c09cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.091+0000 7fbd60bfb700 1 Processor -- start 2019-12-05T06:07:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.091+0000 7fbd60bfb700 1 -- start start 2019-12-05T06:07:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.092+0000 7fbd60bfb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbd5c073a80 0x7fbd5c073e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.093+0000 7fbd60bfb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd5c098810 0x7fbd5c0743b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.093+0000 7fbd60bfb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd5c0984a0 con 0x7fbd5c097840 2019-12-05T06:07:11.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.093+0000 7fbd60bfb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5c096f10 con 0x7fbd5c098810 2019-12-05T06:07:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.093+0000 7fbd60bfb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd5c089290 con 0x7fbd5c073a80 2019-12-05T06:07:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.093+0000 7fbd5b59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbd5c097840 0x7fbd5c097ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60160/0 (socket says 172.21.15.16:60160) 2019-12-05T06:07:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.093+0000 7fbd5b59e700 1 -- 172.21.15.16:0/870528489 learned_addr learned my addr 172.21.15.16:0/870528489 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.093+0000 7fbd5a59c700 1 --2- 172.21.15.16:0/870528489 >> v2:172.21.15.16:3300/0 conn(0x7fbd5c098810 0x7fbd5c0743b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.093+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> v2:172.21.15.76:3300/0 conn(0x7fbd5c073a80 0x7fbd5c073e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.093+0000 7fbd537fe700 1 -- 172.21.15.16:0/870528489 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4007500894 0 0) 0x7fbd4c0079c0 con 0x7fbd5c097840 2019-12-05T06:07:11.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.093+0000 7fbd537fe700 1 -- 172.21.15.16:0/870528489 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbd40002e80 con 0x7fbd5c097840 2019-12-05T06:07:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd537fe700 1 -- 172.21.15.16:0/870528489 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3944347810 0 0) 0x7fbd4c007580 con 0x7fbd5c097840 2019-12-05T06:07:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd5ad9d700 1 -- 172.21.15.16:0/870528489 >> v2:172.21.15.16:3300/0 conn(0x7fbd5c098810 msgr2=0x7fbd5c0743b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> v2:172.21.15.16:3300/0 conn(0x7fbd5c098810 0x7fbd5c0743b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd5ad9d700 1 -- 172.21.15.16:0/870528489 >> v1:172.21.15.16:6789/0 conn(0x7fbd5c097840 legacy=0x7fbd5c097ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd5ad9d700 1 -- 172.21.15.16:0/870528489 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd5c089920 con 0x7fbd5c073a80 2019-12-05T06:07:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd5a59c700 1 --2- 172.21.15.16:0/870528489 >> v2:172.21.15.16:3300/0 conn(0x7fbd5c098810 0x7fbd5c0743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:11.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> v2:172.21.15.76:3300/0 conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> v2:172.21.15.76:3300/0 conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=READY pgs=178 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:11.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd537fe700 1 -- 172.21.15.16:0/870528489 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd54000cc0 con 0x7fbd5c073a80 2019-12-05T06:07:11.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.094+0000 7fbd537fe700 1 -- 172.21.15.16:0/870528489 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd54009f50 con 0x7fbd5c073a80 2019-12-05T06:07:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.095+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.095+0000 7fbd537fe700 1 -- 172.21.15.16:0/870528489 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd5400a140 con 0x7fbd5c073a80 2019-12-05T06:07:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.095+0000 7fbd60bfb700 1 -- 172.21.15.16:0/870528489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd5c089920 con 0x7fbd5c073a80 2019-12-05T06:07:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.095+0000 7fbd60bfb700 1 -- 172.21.15.16:0/870528489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd5c089920 con 0x7fbd5c073a80 2019-12-05T06:07:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.096+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.096+0000 7fbd537fe700 1 -- 172.21.15.16:0/870528489 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbd54009bd0 con 0x7fbd5c073a80 2019-12-05T06:07:11.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.097+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.097+0000 7fbd537fe700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd40023680 0x7fbd40025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:11.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.097+0000 7fbd537fe700 1 -- 172.21.15.16:0/870528489 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fbd540460f0 con 0x7fbd5c073a80 2019-12-05T06:07:11.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.097+0000 7fbd5a59c700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd40023680 0x7fbd40025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.098+0000 7fbd5a59c700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd40023680 0x7fbd40025af0 crc :-1 s=READY pgs=109 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:11.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.098+0000 7fbd60bfb700 1 -- 172.21.15.16:0/870528489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd5c089920 con 0x7fbd5c073a80 2019-12-05T06:07:11.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.107+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.107+0000 7fbd537fe700 1 -- 172.21.15.16:0/870528489 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbd540099c0 con 0x7fbd5c073a80 2019-12-05T06:07:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.301+0000 7fbd60bfb700 1 -- 172.21.15.16:0/870528489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"} v 0) v1 -- 0x7fbd5c089920 con 0x7fbd5c073a80 2019-12-05T06:07:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.303+0000 7fbd5ad9d700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=THROTTLE_DONE pgs=178 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.303+0000 7fbd537fe700 1 -- 172.21.15.16:0/870528489 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "rbd"}]=0 v24) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbd54009bd0 con 0x7fbd5c073a80 2019-12-05T06:07:11.303 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:07:11.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.307+0000 7fbd60bfb700 1 -- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd40023680 msgr2=0x7fbd40025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.307+0000 7fbd60bfb700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd40023680 0x7fbd40025af0 crc :-1 s=READY pgs=109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.307+0000 7fbd60bfb700 1 -- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 msgr2=0x7fbd5c073e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.307+0000 7fbd60bfb700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 0x7fbd5c073e70 secure :-1 s=READY pgs=178 cs=0 l=1 rx=0x7fbd54010710 tx=0x7fbd54010ab0).stop 2019-12-05T06:07:11.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.307+0000 7fbd60bfb700 1 -- 172.21.15.16:0/870528489 shutdown_connections 2019-12-05T06:07:11.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.307+0000 7fbd60bfb700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd40023680 0x7fbd40025af0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.307+0000 7fbd60bfb700 1 --2- 172.21.15.16:0/870528489 >> v2:172.21.15.16:3300/0 conn(0x7fbd5c098810 0x7fbd5c0743b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.307+0000 7fbd60bfb700 1 --2- 172.21.15.16:0/870528489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd5c073a80 0x7fbd5c073e70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.308+0000 7fbd60bfb700 1 -- 172.21.15.16:0/870528489 shutdown_connections 2019-12-05T06:07:11.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.308+0000 7fbd60bfb700 1 -- 172.21.15.16:0/870528489 wait complete. 2019-12-05T06:07:11.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.308+0000 7fbd60bfb700 1 -- 172.21.15.16:0/870528489 >> 172.21.15.16:0/870528489 conn(0x7fbd5c08d5e0 msgr2=0x7fbd5c075f40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:11.317 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2019-12-05T06:07:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.582+0000 7f0153ce4700 1 Processor -- start 2019-12-05T06:07:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.582+0000 7f0153ce4700 1 -- start start 2019-12-05T06:07:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.584+0000 7f0153ce4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 0x7f014c086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.584+0000 7f0153ce4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f014c064310 0x7f014c064790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.584+0000 7f0153ce4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c061a10 con 0x7f014c08a6b0 2019-12-05T06:07:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.584+0000 7f0153ce4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f014c08b340 con 0x7f014c064310 2019-12-05T06:07:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.584+0000 7f0153ce4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014c060bc0 con 0x7f014c086980 2019-12-05T06:07:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.584+0000 7f0152281700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 0x7f014c086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.584+0000 7f0152a82700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f014c064310 0x7f014c064790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.584+0000 7f0151a80700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f014c08a6b0 0x7f014c08ab40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53236/0 (socket says 172.21.15.16:53236) 2019-12-05T06:07:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.584+0000 7f0151a80700 1 -- 172.21.15.16:0/2425920445 learned_addr learned my addr 172.21.15.16:0/2425920445 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.585+0000 7f0152281700 1 -- 172.21.15.16:0/2425920445 >> v2:172.21.15.76:3300/0 conn(0x7f014c064310 msgr2=0x7f014c064790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.585+0000 7f0152281700 1 --2- 172.21.15.16:0/2425920445 >> v2:172.21.15.76:3300/0 conn(0x7f014c064310 0x7f014c064790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.585+0000 7f0152281700 1 -- 172.21.15.16:0/2425920445 >> v1:172.21.15.76:6789/0 conn(0x7f014c08a6b0 legacy=0x7f014c08ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.585+0000 7f0152281700 1 -- 172.21.15.16:0/2425920445 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c063a10 con 0x7f014c086980 2019-12-05T06:07:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.585+0000 7f0152281700 1 --2- 172.21.15.16:0/2425920445 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 0x7f014c086d70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f014400b2d0 tx=0x7f014400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.585+0000 7f0152281700 1 --2- 172.21.15.16:0/2425920445 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 0x7f014c086d70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f014400b2d0 tx=0x7f014400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.585+0000 7f0152281700 1 --2- 172.21.15.16:0/2425920445 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 0x7f014c086d70 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7f014400b2d0 tx=0x7f014400ba60).ready entity=mon.2 client_cookie=af2e71c867a93e57 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.586+0000 7f0152281700 1 --2- 172.21.15.16:0/2425920445 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 0x7f014c086d70 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f014400b2d0 tx=0x7f014400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.586+0000 7f0152281700 1 --2- 172.21.15.16:0/2425920445 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 0x7f014c086d70 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f014400b2d0 tx=0x7f014400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.586+0000 7f0152281700 1 --2- 172.21.15.16:0/2425920445 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 0x7f014c086d70 secure :-1 s=THROTTLE_DONE pgs=56 cs=0 l=1 rx=0x7f014400b2d0 tx=0x7f014400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.586+0000 7f015127f700 1 -- 172.21.15.16:0/2425920445 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0144007590 con 0x7f014c086980 2019-12-05T06:07:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.586+0000 7f015127f700 1 -- 172.21.15.16:0/2425920445 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0144009d30 con 0x7f014c086980 2019-12-05T06:07:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.586+0000 7f015127f700 1 -- 172.21.15.16:0/2425920445 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f014400a400 con 0x7f014c086980 2019-12-05T06:07:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.586+0000 7f0153ce4700 1 -- 172.21.15.16:0/2425920445 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 msgr2=0x7f014c086d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.586+0000 7f0153ce4700 1 --2- 172.21.15.16:0/2425920445 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 0x7f014c086d70 secure :-1 s=READY pgs=56 cs=0 l=1 rx=0x7f014400b2d0 tx=0x7f014400ba60).stop 2019-12-05T06:07:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.588+0000 7f0153ce4700 1 -- 172.21.15.16:0/2425920445 shutdown_connections 2019-12-05T06:07:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.588+0000 7f0153ce4700 1 --2- 172.21.15.16:0/2425920445 >> v2:172.21.15.76:3300/0 conn(0x7f014c064310 0x7f014c064790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.588+0000 7f0153ce4700 1 --2- 172.21.15.16:0/2425920445 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c086980 0x7f014c086d70 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.588+0000 7f0153ce4700 1 -- 172.21.15.16:0/2425920445 shutdown_connections 2019-12-05T06:07:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.588+0000 7f0153ce4700 1 -- 172.21.15.16:0/2425920445 wait complete. 2019-12-05T06:07:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.588+0000 7f0153ce4700 1 -- 172.21.15.16:0/2425920445 >> 172.21.15.16:0/2425920445 conn(0x7f014c05f5a0 msgr2=0x7f014c09cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.592+0000 7f0153ce4700 1 Processor -- start 2019-12-05T06:07:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.592+0000 7f0153ce4700 1 -- start start 2019-12-05T06:07:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.593+0000 7f0153ce4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.593+0000 7f0153ce4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c074c00 con 0x7f014c156c80 2019-12-05T06:07:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.593+0000 7f0153ce4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f014c0746c0 con 0x7f014c099990 2019-12-05T06:07:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.593+0000 7f0153ce4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f014c075e90 con 0x7f014c073d80 2019-12-05T06:07:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.593+0000 7f0152a82700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f014c156c80 0x7f014c159080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60176/0 (socket says 172.21.15.16:60176) 2019-12-05T06:07:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.594+0000 7f0152a82700 1 -- 172.21.15.16:0/326599711 learned_addr learned my addr 172.21.15.16:0/326599711 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.594+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.594+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961174199 0 0) 0x7f013c002180 con 0x7f014c156c80 2019-12-05T06:07:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.594+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c002e80 con 0x7f014c156c80 2019-12-05T06:07:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.594+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531478333 0 0) 0x7f0140001080 con 0x7f014c099990 2019-12-05T06:07:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.594+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f012c004170 con 0x7f014c099990 2019-12-05T06:07:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.594+0000 7f0152281700 1 -- 172.21.15.16:0/326599711 >> v1:172.21.15.76:6789/0 conn(0x7f014c099990 legacy=0x7f014c0959a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f0152281700 1 -- 172.21.15.16:0/326599711 >> v1:172.21.15.16:6789/0 conn(0x7f014c156c80 legacy=0x7f014c159080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f0152281700 1 -- 172.21.15.16:0/326599711 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f014c094b60 con 0x7f014c073d80 2019-12-05T06:07:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=READY pgs=57 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0144000cc0 con 0x7f014c073d80 2019-12-05T06:07:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f014400ad00 con 0x7f014c073d80 2019-12-05T06:07:11.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f014400a180 con 0x7f014c073d80 2019-12-05T06:07:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f0153ce4700 1 -- 172.21.15.16:0/326599711 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f014c094b60 con 0x7f014c073d80 2019-12-05T06:07:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.595+0000 7f0153ce4700 1 -- 172.21.15.16:0/326599711 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f014c094b60 con 0x7f014c073d80 2019-12-05T06:07:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.596+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.597+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0144009b10 con 0x7f014c073d80 2019-12-05T06:07:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.598+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.598+0000 7f014affd700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f012c023680 0x7f012c025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.598+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f0144046150 con 0x7f014c073d80 2019-12-05T06:07:11.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.598+0000 7f0151a80700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f012c023680 0x7f012c025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.599+0000 7f0151a80700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f012c023680 0x7f012c025af0 crc :-1 s=READY pgs=110 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.599+0000 7f0153ce4700 1 -- 172.21.15.16:0/326599711 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f014c094b60 con 0x7f014c073d80 2019-12-05T06:07:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.608+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.608+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0144009900 con 0x7f014c073d80 2019-12-05T06:07:11.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.844+0000 7f0153ce4700 1 -- 172.21.15.16:0/326599711 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"} v 0) v1 -- 0x7f014c094b60 con 0x7f014c073d80 2019-12-05T06:07:11.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.844+0000 7f0152281700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=THROTTLE_DONE pgs=57 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:11.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.845+0000 7f014affd700 1 -- 172.21.15.16:0/326599711 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_metadata"}]=0 v24) v1 ==== 104+0+10 (secure 0 0 0) 0x7f0144009b10 con 0x7f014c073d80 2019-12-05T06:07:11.846 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:07:11.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.851+0000 7f0153ce4700 1 -- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f012c023680 msgr2=0x7f012c025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.851+0000 7f0153ce4700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f012c023680 0x7f012c025af0 crc :-1 s=READY pgs=110 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.851+0000 7f0153ce4700 1 -- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 msgr2=0x7f014c095460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:11.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.851+0000 7f0153ce4700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 secure :-1 s=READY pgs=57 cs=0 l=1 rx=0x7f0144010710 tx=0x7f0144010ab0).stop 2019-12-05T06:07:11.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.851+0000 7f0153ce4700 1 -- 172.21.15.16:0/326599711 shutdown_connections 2019-12-05T06:07:11.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.851+0000 7f0153ce4700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f012c023680 0x7f012c025af0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.851+0000 7f0153ce4700 1 --2- 172.21.15.16:0/326599711 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f014c073d80 0x7f014c095460 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:11.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.851+0000 7f0153ce4700 1 -- 172.21.15.16:0/326599711 shutdown_connections 2019-12-05T06:07:11.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.851+0000 7f0153ce4700 1 -- 172.21.15.16:0/326599711 wait complete. 2019-12-05T06:07:11.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:11.851+0000 7f0153ce4700 1 -- 172.21.15.16:0/326599711 >> 172.21.15.16:0/326599711 conn(0x7f014c077e10 msgr2=0x7f014c08b740 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:11.866 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2019-12-05T06:07:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.076+0000 7fa6da65d700 1 Processor -- start 2019-12-05T06:07:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.076+0000 7fa6da65d700 1 -- start start 2019-12-05T06:07:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6da65d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6d4086980 0x7fa6d4086d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6da65d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa6d408a6b0 0x7fa6d408ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6da65d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6d408b080 0x7fa6d4064310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6da65d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d4060bc0 con 0x7fa6d408b080 2019-12-05T06:07:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6da65d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d4060620 con 0x7fa6d408a6b0 2019-12-05T06:07:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6da65d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6d40608f0 con 0x7fa6d4086980 2019-12-05T06:07:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6d93fb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6d408b080 0x7fa6d4064310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6d93fb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6d408b080 0x7fa6d4064310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46784/0 (socket says 172.21.15.16:46784) 2019-12-05T06:07:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6d93fb700 1 -- 172.21.15.16:0/318426859 learned_addr learned my addr 172.21.15.16:0/318426859 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6d3fff700 1 --2- 172.21.15.16:0/318426859 >> v2:172.21.15.76:3300/0 conn(0x7fa6d408a6b0 0x7fa6d408ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.077+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/318426859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6d4086980 0x7fa6d4086d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 -- 172.21.15.16:0/318426859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6d4086980 msgr2=0x7fa6d4086d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 --2- 172.21.15.16:0/318426859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6d4086980 0x7fa6d4086d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 -- 172.21.15.16:0/318426859 >> v2:172.21.15.76:3300/0 conn(0x7fa6d408a6b0 msgr2=0x7fa6d408ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 --2- 172.21.15.16:0/318426859 >> v2:172.21.15.76:3300/0 conn(0x7fa6d408a6b0 0x7fa6d408ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 -- 172.21.15.16:0/318426859 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d4063a10 con 0x7fa6d408b080 2019-12-05T06:07:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 --2- 172.21.15.16:0/318426859 >> v2:172.21.15.16:3300/0 conn(0x7fa6d408b080 0x7fa6d4064310 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa6bc00f6b0 tx=0x7fa6bc007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 --2- 172.21.15.16:0/318426859 >> v2:172.21.15.16:3300/0 conn(0x7fa6d408b080 0x7fa6d4064310 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa6bc00f6b0 tx=0x7fa6bc007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 --2- 172.21.15.16:0/318426859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d408b080 0x7fa6d4064310 secure :-1 s=READY pgs=209 cs=0 l=1 rx=0x7fa6bc00f6b0 tx=0x7fa6bc007d50).ready entity=mon.0 client_cookie=91f93a6cbaf30bc9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 --2- 172.21.15.16:0/318426859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d408b080 0x7fa6d4064310 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7fa6bc00f6b0 tx=0x7fa6bc007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 --2- 172.21.15.16:0/318426859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d408b080 0x7fa6d4064310 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7fa6bc00f6b0 tx=0x7fa6bc007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d37fe700 1 -- 172.21.15.16:0/318426859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6bc000a70 con 0x7fa6d408b080 2019-12-05T06:07:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d93fb700 1 --2- 172.21.15.16:0/318426859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d408b080 0x7fa6d4064310 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7fa6bc00f6b0 tx=0x7fa6bc007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d37fe700 1 -- 172.21.15.16:0/318426859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc0010e0 con 0x7fa6d408b080 2019-12-05T06:07:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6d37fe700 1 -- 172.21.15.16:0/318426859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6bc001830 con 0x7fa6d408b080 2019-12-05T06:07:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.078+0000 7fa6da65d700 1 -- 172.21.15.16:0/318426859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d408b080 msgr2=0x7fa6d4064310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.079+0000 7fa6da65d700 1 --2- 172.21.15.16:0/318426859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d408b080 0x7fa6d4064310 secure :-1 s=READY pgs=209 cs=0 l=1 rx=0x7fa6bc00f6b0 tx=0x7fa6bc007d50).stop 2019-12-05T06:07:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.080+0000 7fa6da65d700 1 -- 172.21.15.16:0/318426859 shutdown_connections 2019-12-05T06:07:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.080+0000 7fa6da65d700 1 --2- 172.21.15.16:0/318426859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d408b080 0x7fa6d4064310 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.080+0000 7fa6da65d700 1 --2- 172.21.15.16:0/318426859 >> v2:172.21.15.76:3300/0 conn(0x7fa6d408a6b0 0x7fa6d408ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.080+0000 7fa6da65d700 1 --2- 172.21.15.16:0/318426859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6d4086980 0x7fa6d4086d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.080+0000 7fa6da65d700 1 -- 172.21.15.16:0/318426859 shutdown_connections 2019-12-05T06:07:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.080+0000 7fa6da65d700 1 -- 172.21.15.16:0/318426859 wait complete. 2019-12-05T06:07:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.080+0000 7fa6da65d700 1 -- 172.21.15.16:0/318426859 >> 172.21.15.16:0/318426859 conn(0x7fa6d405f5a0 msgr2=0x7fa6d409cba0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.084+0000 7fa6da65d700 1 Processor -- start 2019-12-05T06:07:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.084+0000 7fa6da65d700 1 -- start start 2019-12-05T06:07:12.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.085+0000 7fa6da65d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6d414f6a0 0x7fa6d414fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.085+0000 7fa6da65d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa6d413c760 0x7fa6d41592f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6da65d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6d413cbc0 con 0x7fa6d413cf30 2019-12-05T06:07:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6da65d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d4087c80 con 0x7fa6d414f6a0 2019-12-05T06:07:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6da65d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6d413afd0 con 0x7fa6d413c760 2019-12-05T06:07:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d8bfa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6d414f6a0 0x7fa6d414fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d8bfa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6d414f6a0 0x7fa6d414fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46786/0 (socket says 172.21.15.16:46786) 2019-12-05T06:07:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d8bfa700 1 -- 172.21.15.16:0/413088231 learned_addr learned my addr 172.21.15.16:0/413088231 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d93fb700 1 --2- 172.21.15.16:0/413088231 >> v2:172.21.15.76:3300/0 conn(0x7fa6d413c760 0x7fa6d41592f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d8bfa700 1 -- 172.21.15.16:0/413088231 >> v2:172.21.15.76:3300/0 conn(0x7fa6d413c760 msgr2=0x7fa6d41592f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> v2:172.21.15.76:3300/0 conn(0x7fa6d413c760 0x7fa6d41592f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d8bfa700 1 -- 172.21.15.16:0/413088231 >> v1:172.21.15.76:6789/0 conn(0x7fa6d413cf30 legacy=0x7fa6d413bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d8bfa700 1 -- 172.21.15.16:0/413088231 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6d413b660 con 0x7fa6d414f6a0 2019-12-05T06:07:12.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d93fb700 1 --2- 172.21.15.16:0/413088231 >> v2:172.21.15.76:3300/0 conn(0x7fa6d413c760 0x7fa6d41592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:07:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> v2:172.21.15.16:3300/0 conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> v2:172.21.15.16:3300/0 conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.086+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=READY pgs=210 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.087+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.087+0000 7fa6d17fa700 1 -- 172.21.15.16:0/413088231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6cc000cc0 con 0x7fa6d414f6a0 2019-12-05T06:07:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.087+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.087+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.087+0000 7fa6d17fa700 1 -- 172.21.15.16:0/413088231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6cc0067e0 con 0x7fa6d414f6a0 2019-12-05T06:07:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.087+0000 7fa6d17fa700 1 -- 172.21.15.16:0/413088231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6cc000cc0 con 0x7fa6d414f6a0 2019-12-05T06:07:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.087+0000 7fa6da65d700 1 -- 172.21.15.16:0/413088231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6d413b660 con 0x7fa6d414f6a0 2019-12-05T06:07:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.087+0000 7fa6da65d700 1 -- 172.21.15.16:0/413088231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6d413b660 con 0x7fa6d414f6a0 2019-12-05T06:07:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.088+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.088+0000 7fa6d17fa700 1 -- 172.21.15.16:0/413088231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa6cc005710 con 0x7fa6d414f6a0 2019-12-05T06:07:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.089+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.089+0000 7fa6d17fa700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6b8023630 0x7fa6b8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.089+0000 7fa6d17fa700 1 -- 172.21.15.16:0/413088231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7fa6cc0461c0 con 0x7fa6d414f6a0 2019-12-05T06:07:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.089+0000 7fa6d3fff700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6b8023630 0x7fa6b8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.089+0000 7fa6d3fff700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6b8023630 0x7fa6b8025ac0 crc :-1 s=READY pgs=111 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.091+0000 7fa6da65d700 1 -- 172.21.15.16:0/413088231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6d413b660 con 0x7fa6d414f6a0 2019-12-05T06:07:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.097+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.097+0000 7fa6d17fa700 1 -- 172.21.15.16:0/413088231 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa6cc005500 con 0x7fa6d414f6a0 2019-12-05T06:07:12.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.297+0000 7fa6da65d700 1 -- 172.21.15.16:0/413088231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"} v 0) v1 -- 0x7fa6d413b660 con 0x7fa6d414f6a0 2019-12-05T06:07:12.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.298+0000 7fa6d8bfa700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.298+0000 7fa6d17fa700 1 -- 172.21.15.16:0/413088231 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "pg_num", "prefix": "osd pool get", "pool": "cephfs_data"}]=0 v24) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa6cc005710 con 0x7fa6d414f6a0 2019-12-05T06:07:12.300 INFO:teuthology.orchestra.run.smithi016.stdout:pg_num: 8 2019-12-05T06:07:12.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.300+0000 7fa6da65d700 1 -- 172.21.15.16:0/413088231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6b8023630 msgr2=0x7fa6b8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.300+0000 7fa6da65d700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6b8023630 0x7fa6b8025ac0 crc :-1 s=READY pgs=111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.301+0000 7fa6da65d700 1 -- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 msgr2=0x7fa6d414fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.301+0000 7fa6da65d700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 0x7fa6d414fa90 secure :-1 s=READY pgs=210 cs=0 l=1 rx=0x7fa6cc0107d0 tx=0x7fa6cc010b70).stop 2019-12-05T06:07:12.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.301+0000 7fa6da65d700 1 -- 172.21.15.16:0/413088231 shutdown_connections 2019-12-05T06:07:12.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.301+0000 7fa6da65d700 1 --2- 172.21.15.16:0/413088231 >> v2:172.21.15.76:3300/0 conn(0x7fa6d413c760 0x7fa6d41592f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.301+0000 7fa6da65d700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6b8023630 0x7fa6b8025ac0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.301+0000 7fa6da65d700 1 --2- 172.21.15.16:0/413088231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6d414f6a0 0x7fa6d414fa90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.301+0000 7fa6da65d700 1 -- 172.21.15.16:0/413088231 shutdown_connections 2019-12-05T06:07:12.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.301+0000 7fa6da65d700 1 -- 172.21.15.16:0/413088231 wait complete. 2019-12-05T06:07:12.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.301+0000 7fa6da65d700 1 -- 172.21.15.16:0/413088231 >> 172.21.15.16:0/413088231 conn(0x7fa6d4088790 msgr2=0x7fa6d4088c20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:12.312 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2019-12-05T06:07:12.312 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:07:12.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.564+0000 7efce5b95700 1 Processor -- start 2019-12-05T06:07:12.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.564+0000 7efce5b95700 1 -- start start 2019-12-05T06:07:12.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.566+0000 7efce5b95700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 0x7efce008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:12.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.566+0000 7efce5b95700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efce008f720 0x7efce008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:07:12.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.566+0000 7efce5b95700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce00958e0 con 0x7efce008fef0 2019-12-05T06:07:12.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.566+0000 7efce5b95700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce00a3f70 con 0x7efce008f720 2019-12-05T06:07:12.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.566+0000 7efce5b95700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce008d830 con 0x7efce008c6d0 2019-12-05T06:07:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.567+0000 7efce4933700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efce008f720 0x7efce008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.567+0000 7efcdffff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 0x7efce008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.567+0000 7efcdf7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efce008fef0 0x7efce008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60196/0 (socket says 172.21.15.16:60196) 2019-12-05T06:07:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.567+0000 7efcdf7fe700 1 -- 172.21.15.16:0/3914386934 learned_addr learned my addr 172.21.15.16:0/3914386934 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:07:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.567+0000 7efcdffff700 1 -- 172.21.15.16:0/3914386934 >> v2:172.21.15.16:3300/0 conn(0x7efce008f720 msgr2=0x7efce008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:07:12.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.567+0000 7efcdffff700 1 --2- 172.21.15.16:0/3914386934 >> v2:172.21.15.16:3300/0 conn(0x7efce008f720 0x7efce008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.567+0000 7efcdffff700 1 -- 172.21.15.16:0/3914386934 >> v1:172.21.15.16:6789/0 conn(0x7efce008fef0 legacy=0x7efce008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.567+0000 7efcdffff700 1 -- 172.21.15.16:0/3914386934 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce008fbc0 con 0x7efce008c6d0 2019-12-05T06:07:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.567+0000 7efcdeffd700 1 -- 172.21.15.16:0/3914386934 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471730039 0 0) 0x7efcc8001150 con 0x7efce008fef0 2019-12-05T06:07:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.568+0000 7efce4933700 1 --2- 172.21.15.16:0/3914386934 >> v2:172.21.15.16:3300/0 conn(0x7efce008f720 0x7efce008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:07:12.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.568+0000 7efcdffff700 1 --2- 172.21.15.16:0/3914386934 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 0x7efce008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efcd000b2d0 tx=0x7efcd000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.568+0000 7efcdffff700 1 --2- 172.21.15.16:0/3914386934 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 0x7efce008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efcd000b2d0 tx=0x7efcd000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.568+0000 7efcdffff700 1 --2- 172.21.15.16:0/3914386934 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 0x7efce008cac0 secure :-1 s=READY pgs=58 cs=0 l=1 rx=0x7efcd000b2d0 tx=0x7efcd000bae0).ready entity=mon.2 client_cookie=cc9c7c9c39033b76 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:12.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.568+0000 7efcdffff700 1 --2- 172.21.15.16:0/3914386934 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 0x7efce008cac0 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7efcd000b2d0 tx=0x7efcd000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.568+0000 7efcdffff700 1 --2- 172.21.15.16:0/3914386934 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 0x7efce008cac0 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7efcd000b2d0 tx=0x7efcd000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.568+0000 7efcdeffd700 1 -- 172.21.15.16:0/3914386934 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd0007590 con 0x7efce008c6d0 2019-12-05T06:07:12.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.568+0000 7efcdffff700 1 --2- 172.21.15.16:0/3914386934 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 0x7efce008cac0 secure :-1 s=THROTTLE_DONE pgs=58 cs=0 l=1 rx=0x7efcd000b2d0 tx=0x7efcd000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.568+0000 7efcdeffd700 1 -- 172.21.15.16:0/3914386934 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd0009d30 con 0x7efce008c6d0 2019-12-05T06:07:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.569+0000 7efcdeffd700 1 -- 172.21.15.16:0/3914386934 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd000a400 con 0x7efce008c6d0 2019-12-05T06:07:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.569+0000 7efce5b95700 1 -- 172.21.15.16:0/3914386934 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 msgr2=0x7efce008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.569+0000 7efce5b95700 1 --2- 172.21.15.16:0/3914386934 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 0x7efce008cac0 secure :-1 s=READY pgs=58 cs=0 l=1 rx=0x7efcd000b2d0 tx=0x7efcd000bae0).stop 2019-12-05T06:07:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.570+0000 7efce5b95700 1 -- 172.21.15.16:0/3914386934 shutdown_connections 2019-12-05T06:07:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.570+0000 7efce5b95700 1 --2- 172.21.15.16:0/3914386934 >> v2:172.21.15.16:3300/0 conn(0x7efce008f720 0x7efce008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:07:12.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.570+0000 7efce5b95700 1 --2- 172.21.15.16:0/3914386934 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce008c6d0 0x7efce008cac0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.570+0000 7efce5b95700 1 -- 172.21.15.16:0/3914386934 shutdown_connections 2019-12-05T06:07:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.570+0000 7efce5b95700 1 -- 172.21.15.16:0/3914386934 wait complete. 2019-12-05T06:07:12.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.570+0000 7efce5b95700 1 -- 172.21.15.16:0/3914386934 >> 172.21.15.16:0/3914386934 conn(0x7efce0061940 msgr2=0x7efce0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:12.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.575+0000 7efce5b95700 1 Processor -- start 2019-12-05T06:07:12.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.575+0000 7efce5b95700 1 -- start start 2019-12-05T06:07:12.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.577+0000 7efce5b95700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7efce01325e0 0x7efce012b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:12.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.577+0000 7efce5b95700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:12.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.577+0000 7efce5b95700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efce0128b20 con 0x7efce012c740 2019-12-05T06:07:12.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.577+0000 7efce5b95700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efce012a7e0 con 0x7efce01325e0 2019-12-05T06:07:12.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.577+0000 7efce5b95700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efce0129db0 con 0x7efce0154c60 2019-12-05T06:07:12.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.577+0000 7efce4933700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.577+0000 7efce4933700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:50794/0 (socket says 172.21.15.16:50794) 2019-12-05T06:07:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.577+0000 7efce4933700 1 -- 172.21.15.16:0/845752942 learned_addr learned my addr 172.21.15.16:0/845752942 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:07:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.577+0000 7efcdffff700 1 --2- 172.21.15.16:0/845752942 >> v2:172.21.15.76:3300/0 conn(0x7efce01325e0 0x7efce012b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce4933700 1 -- 172.21.15.16:0/845752942 >> v2:172.21.15.76:3300/0 conn(0x7efce01325e0 msgr2=0x7efce012b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> v2:172.21.15.76:3300/0 conn(0x7efce01325e0 0x7efce012b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce4933700 1 -- 172.21.15.16:0/845752942 >> v1:172.21.15.76:6789/0 conn(0x7efce012c740 legacy=0x7efce012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce4933700 1 -- 172.21.15.16:0/845752942 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efce012ae70 con 0x7efce0154c60 2019-12-05T06:07:12.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:12.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efcdcff9700 1 -- 172.21.15.16:0/845752942 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd8009960 con 0x7efce0154c60 2019-12-05T06:07:12.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efcdcff9700 1 -- 172.21.15.16:0/845752942 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd800a600 con 0x7efce0154c60 2019-12-05T06:07:12.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efcdcff9700 1 -- 172.21.15.16:0/845752942 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd8009f40 con 0x7efce0154c60 2019-12-05T06:07:12.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.578+0000 7efce5b95700 1 -- 172.21.15.16:0/845752942 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efce012ae70 con 0x7efce0154c60 2019-12-05T06:07:12.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.579+0000 7efce5b95700 1 -- 172.21.15.16:0/845752942 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efce012ae70 con 0x7efce0154c60 2019-12-05T06:07:12.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.580+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.580+0000 7efcdcff9700 1 -- 172.21.15.16:0/845752942 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efcd8009620 con 0x7efce0154c60 2019-12-05T06:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.580+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.581+0000 7efcdcff9700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcc4023630 0x7efcc4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.581+0000 7efcdcff9700 1 -- 172.21.15.16:0/845752942 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7efcd8045de0 con 0x7efce0154c60 2019-12-05T06:07:12.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.581+0000 7efcdffff700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcc4023630 0x7efcc4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:07:12.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.582+0000 7efcdffff700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcc4023630 0x7efcc4025ac0 crc :-1 s=READY pgs=112 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:07:12.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.595+0000 7efce5b95700 1 -- 172.21.15.16:0/845752942 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efce012ae70 con 0x7efce0154c60 2019-12-05T06:07:12.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.604+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.604+0000 7efcdcff9700 1 -- 172.21.15.16:0/845752942 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efcd80009d0 con 0x7efce0154c60 2019-12-05T06:07:12.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.870+0000 7efce5b95700 1 -- 172.21.15.16:0/845752942 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efce012ae70 con 0x7efce0154c60 2019-12-05T06:07:12.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.871+0000 7efce4933700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=THROTTLE_DONE pgs=59 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:07:12.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.871+0000 7efcdcff9700 1 -- 172.21.15.16:0/845752942 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2102 (secure 0 0 0) 0x7efcd8009580 con 0x7efce0154c60 2019-12-05T06:07:12.874 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 7 2019-12-05T06:07:12.875 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:07:12.875 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:06:45.694298+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:07:12.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.875+0000 7efce5b95700 1 -- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcc4023630 msgr2=0x7efcc4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.875+0000 7efce5b95700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcc4023630 0x7efcc4025ac0 crc :-1 s=READY pgs=112 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.875+0000 7efce5b95700 1 -- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 msgr2=0x7efce0157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:07:12.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.875+0000 7efce5b95700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 secure :-1 s=READY pgs=59 cs=0 l=1 rx=0x7efcd8010450 tx=0x7efcd80102d0).stop 2019-12-05T06:07:12.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.876+0000 7efce5b95700 1 -- 172.21.15.16:0/845752942 shutdown_connections 2019-12-05T06:07:12.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.876+0000 7efce5b95700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcc4023630 0x7efcc4025ac0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.876+0000 7efce5b95700 1 --2- 172.21.15.16:0/845752942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efce0154c60 0x7efce0157060 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.876+0000 7efce5b95700 1 --2- 172.21.15.16:0/845752942 >> v2:172.21.15.76:3300/0 conn(0x7efce01325e0 0x7efce012b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:07:12.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.876+0000 7efce5b95700 1 -- 172.21.15.16:0/845752942 shutdown_connections 2019-12-05T06:07:12.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.876+0000 7efce5b95700 1 -- 172.21.15.16:0/845752942 wait complete. 2019-12-05T06:07:12.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:07:12.876+0000 7efce5b95700 1 -- 172.21.15.16:0/845752942 >> 172.21.15.16:0/845752942 conn(0x7efce00a2850 msgr2=0x7efce006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:07:13.088 INFO:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {u'session_autoclose': 300, u'balancer': u'', u'up': {u'mds_0': 4341}, u'last_failure_osd_epoch': 0, u'in': [0], u'last_failure': 0, u'max_file_size': 1099511627776, u'explicitly_allowed_features': 0, u'damaged': [], u'tableserver': 0, u'metadata_pool': 2, u'failed': [], u'epoch': 6, u'flags': 18, u'max_mds': 1, u'compat': {u'compat': {}, u'ro_compat': {}, u'incompat': {u'feature_10': u'snaprealm v2', u'feature_8': u'no anchor table', u'feature_9': u'file layout v2', u'feature_2': u'client writeable ranges', u'feature_3': u'default file layouts on dirs', u'feature_1': u'base v0.20', u'feature_6': u'dirfrag is stored in omap', u'feature_4': u'dir inode in separate object', u'feature_5': u'mds uses versioned encoding'}}, u'min_compat_client': u'0 (unknown)', u'data_pools': [3], u'info': {u'gid_4341': {u'export_targets': [], u'name': u'a', u'incarnation': 5, u'state_seq': 2, u'state': u'up:active', u'gid': 4341, u'features': 4540138292836696063, u'rank': 0, u'flags': 0, u'addrs': {u'addrvec': [{u'nonce': 3750563081, u'type': u'v2', u'addr': u'172.21.15.16:6832'}, {u'nonce': 3750563081, u'type': u'v1', u'addr': u'172.21.15.16:6833'}]}, u'addr': u'172.21.15.16:6833/3750563081'}}, u'fs_name': u'cephfs', u'created': u'2019-12-05T06:06:40.256556+0000', u'standby_count_wanted': 0, u'enabled': True, u'modified': u'2019-12-05T06:06:45.694298+0000', u'session_timeout': 60, u'stopped': [], u'ever_allowed_features': 0, u'root': 0} 2019-12-05T06:07:13.089 INFO:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2019-12-05T06:07:13.090 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok status 2019-12-05T06:07:13.345 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2019-12-05T06:07:13.345 INFO:teuthology.orchestra.run.smithi016.stdout: "cluster_fsid": "df6db985-de08-463b-af1a-6ee59e0bbbe7", 2019-12-05T06:07:13.345 INFO:teuthology.orchestra.run.smithi016.stdout: "whoami": 0, 2019-12-05T06:07:13.345 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4341, 2019-12-05T06:07:13.345 INFO:teuthology.orchestra.run.smithi016.stdout: "want_state": "up:active", 2019-12-05T06:07:13.346 INFO:teuthology.orchestra.run.smithi016.stdout: "state": "up:active", 2019-12-05T06:07:13.346 INFO:teuthology.orchestra.run.smithi016.stdout: "rank_uptime": 28.650234727000001, 2019-12-05T06:07:13.346 INFO:teuthology.orchestra.run.smithi016.stdout: "mdsmap_epoch": 6, 2019-12-05T06:07:13.346 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch": 24, 2019-12-05T06:07:13.346 INFO:teuthology.orchestra.run.smithi016.stdout: "osdmap_epoch_barrier": 24, 2019-12-05T06:07:13.346 INFO:teuthology.orchestra.run.smithi016.stdout: "uptime": 29.379340835000001 2019-12-05T06:07:13.347 INFO:teuthology.orchestra.run.smithi016.stdout:} 2019-12-05T06:07:13.356 INFO:tasks.cephfs.filesystem:_json_asok output: { "cluster_fsid": "df6db985-de08-463b-af1a-6ee59e0bbbe7", "whoami": 0, "id": 4341, "want_state": "up:active", "state": "up:active", "rank_uptime": 28.650234727000001, "mdsmap_epoch": 6, "osdmap_epoch": 24, "osdmap_epoch_barrier": 24, "uptime": 29.379340835000001 } 2019-12-05T06:07:13.356 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2019-12-05T06:07:13.356 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2019-12-05T06:07:13.356 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi016.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2019-12-05T06:07:13.357 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:13.357 INFO:teuthology.orchestra.run.smithi016:> mkdir -- /home/ubuntu/cephtest/mnt.0 2019-12-05T06:07:13.491 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:13.491 INFO:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-12-05T06:07:13.612 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-12-05T06:07:13.619 INFO:teuthology.orchestra.run.smithi016.stderr:mount: mount point /sys/fs/fuse/connections does not exist 2019-12-05T06:07:13.620 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:13.620 INFO:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2019-12-05T06:07:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 2 2019-12-05T06:07:13.743 INFO:teuthology.orchestra.run.smithi016.stderr:ls: cannot access /sys/fs/fuse/connections: No such file or directory 2019-12-05T06:07:13.743 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2019-12-05T06:07:13.743 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0 2019-12-05T06:07:13.748 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:13.749 INFO:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-12-05T06:07:13.870 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-12-05T06:07:13.921 INFO:teuthology.orchestra.run.smithi016.stderr:mount: mount point /sys/fs/fuse/connections does not exist 2019-12-05T06:07:13.921 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:13.921 INFO:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2019-12-05T06:07:13.941 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:2019-12-05T06:07:13.942+0000 7fa327154e00 -1 init, newargv = 0x562a1396b480 newargc=7 2019-12-05T06:07:13.942 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[14775]: starting ceph client 2019-12-05T06:07:13.993 DEBUG:teuthology.orchestra.run:got remote process result: 2 2019-12-05T06:07:14.043 INFO:teuthology.orchestra.run.smithi016.stderr:ls: cannot access /sys/fs/fuse/connections: No such file or directory 2019-12-05T06:07:14.197 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi016.stderr:ceph-fuse[14775]: starting fuse 2019-12-05T06:07:15.145 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:15.146 INFO:teuthology.orchestra.run.smithi016:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2019-12-05T06:07:15.289 DEBUG:teuthology.orchestra.run:got remote process result: 32 2019-12-05T06:07:15.321 INFO:teuthology.orchestra.run.smithi016.stderr:mount: /sys/fs/fuse/connections is already mounted or /sys/fs/fuse/connections busy 2019-12-05T06:07:15.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:15.322 INFO:teuthology.orchestra.run.smithi016:> ls /sys/fs/fuse/connections 2019-12-05T06:07:15.456 INFO:teuthology.orchestra.run.smithi016.stdout:41 2019-12-05T06:07:15.456 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2019-12-05T06:07:15.456 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:15.457 INFO:teuthology.orchestra.run.smithi016:> sudo python2 -c ' 2019-12-05T06:07:15.457 INFO:teuthology.orchestra.run.smithi016:> import glob 2019-12-05T06:07:15.457 INFO:teuthology.orchestra.run.smithi016:> import re 2019-12-05T06:07:15.457 INFO:teuthology.orchestra.run.smithi016:> import os 2019-12-05T06:07:15.457 INFO:teuthology.orchestra.run.smithi016:> import subprocess 2019-12-05T06:07:15.457 INFO:teuthology.orchestra.run.smithi016:> 2019-12-05T06:07:15.457 INFO:teuthology.orchestra.run.smithi016:> def find_socket(client_name): 2019-12-05T06:07:15.458 INFO:teuthology.orchestra.run.smithi016:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2019-12-05T06:07:15.458 INFO:teuthology.orchestra.run.smithi016:> files = glob.glob(asok_path) 2019-12-05T06:07:15.458 INFO:teuthology.orchestra.run.smithi016:> 2019-12-05T06:07:15.458 INFO:teuthology.orchestra.run.smithi016:> # Given a non-glob path, it better be there 2019-12-05T06:07:15.458 INFO:teuthology.orchestra.run.smithi016:> if "*" not in asok_path: 2019-12-05T06:07:15.458 INFO:teuthology.orchestra.run.smithi016:> assert(len(files) == 1) 2019-12-05T06:07:15.458 INFO:teuthology.orchestra.run.smithi016:> return files[0] 2019-12-05T06:07:15.458 INFO:teuthology.orchestra.run.smithi016:> 2019-12-05T06:07:15.459 INFO:teuthology.orchestra.run.smithi016:> for f in files: 2019-12-05T06:07:15.459 INFO:teuthology.orchestra.run.smithi016:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2019-12-05T06:07:15.459 INFO:teuthology.orchestra.run.smithi016:> if os.path.exists("/proc/{0}".format(pid)): 2019-12-05T06:07:15.459 INFO:teuthology.orchestra.run.smithi016:> return f 2019-12-05T06:07:15.459 INFO:teuthology.orchestra.run.smithi016:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2019-12-05T06:07:15.459 INFO:teuthology.orchestra.run.smithi016:> 2019-12-05T06:07:15.459 INFO:teuthology.orchestra.run.smithi016:> print(find_socket("client.0")) 2019-12-05T06:07:15.459 INFO:teuthology.orchestra.run.smithi016:> ' 2019-12-05T06:07:15.640 INFO:teuthology.orchestra.run.smithi016.stdout:/var/run/ceph/ceph-client.0.14775.asok 2019-12-05T06:07:15.640 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.14775.asok 2019-12-05T06:07:15.641 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:15.641 INFO:teuthology.orchestra.run.smithi016:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.14775.asok status 2019-12-05T06:07:15.918 INFO:teuthology.orchestra.run.smithi016.stdout:{ 2019-12-05T06:07:15.918 INFO:teuthology.orchestra.run.smithi016.stdout: "metadata": { 2019-12-05T06:07:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_sha1": "c5290805033483ba0cec165c06bde07dae5f19ec", 2019-12-05T06:07:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "ceph_version": "ceph version 15.0.0-8014-gc529080 (c5290805033483ba0cec165c06bde07dae5f19ec) octopus (dev)", 2019-12-05T06:07:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "entity_id": "0", 2019-12-05T06:07:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "hostname": "smithi016", 2019-12-05T06:07:15.919 INFO:teuthology.orchestra.run.smithi016.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2019-12-05T06:07:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "pid": "14775", 2019-12-05T06:07:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "root": "/" 2019-12-05T06:07:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2019-12-05T06:07:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_count": 0, 2019-12-05T06:07:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "dentry_pinned_count": 0, 2019-12-05T06:07:15.920 INFO:teuthology.orchestra.run.smithi016.stdout: "id": 4786, 2019-12-05T06:07:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "inst": { 2019-12-05T06:07:15.929 INFO:teuthology.orchestra.run.smithi016.stdout: "name": { 2019-12-05T06:07:15.930 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "client", 2019-12-05T06:07:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "num": 4786 2019-12-05T06:07:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2019-12-05T06:07:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2019-12-05T06:07:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2019-12-05T06:07:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2019-12-05T06:07:15.931 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 959759099 2019-12-05T06:07:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: } 2019-12-05T06:07:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2019-12-05T06:07:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": { 2019-12-05T06:07:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "type": "v1", 2019-12-05T06:07:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "addr": "172.21.15.16:0", 2019-12-05T06:07:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: "nonce": 959759099 2019-12-05T06:07:15.932 INFO:teuthology.orchestra.run.smithi016.stdout: }, 2019-12-05T06:07:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "inst_str": "client.4786 172.21.15.16:0/959759099", 2019-12-05T06:07:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "addr_str": "172.21.15.16:0/959759099", 2019-12-05T06:07:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "inode_count": 1, 2019-12-05T06:07:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "mds_epoch": 6, 2019-12-05T06:07:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch": 24, 2019-12-05T06:07:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "osd_epoch_barrier": 0, 2019-12-05T06:07:15.933 INFO:teuthology.orchestra.run.smithi016.stdout: "blacklisted": false 2019-12-05T06:07:15.934 INFO:teuthology.orchestra.run.smithi016.stdout:} 2019-12-05T06:07:15.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:15.934 INFO:teuthology.orchestra.run.smithi016:> stat --file-system '--printf=%T 2019-12-05T06:07:15.934 INFO:teuthology.orchestra.run.smithi016:> ' -- /home/ubuntu/cephtest/mnt.0 2019-12-05T06:07:16.056 INFO:teuthology.orchestra.run.smithi016.stdout:fuseblk 2019-12-05T06:07:16.056 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2019-12-05T06:07:16.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2019-12-05T06:07:16.057 INFO:teuthology.orchestra.run.smithi016:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2019-12-05T06:07:16.191 INFO:teuthology.run_tasks:Running task workunit... 2019-12-05T06:07:16.215 INFO:tasks.workunit:Pulling workunits from ref c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T06:07:16.216 INFO:tasks.workunit:Making a separate scratch dir for every client... 2019-12-05T06:07:16.216 INFO:tasks.workunit:timeout=3h 2019-12-05T06:07:16.216 INFO:tasks.workunit:cleanup=True 2019-12-05T06:07:16.216 INFO:teuthology.orchestra.run.smithi016:> stat -- /home/ubuntu/cephtest/mnt.0 2019-12-05T06:07:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: File: ‘/home/ubuntu/cephtest/mnt.0’ 2019-12-05T06:07:16.314 INFO:teuthology.orchestra.run.smithi016.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2019-12-05T06:07:16.314 INFO:teuthology.orchestra.run.smithi016.stdout:Device: 29h/41d Inode: 1 Links: 2 2019-12-05T06:07:16.314 INFO:teuthology.orchestra.run.smithi016.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2019-12-05T06:07:16.314 INFO:teuthology.orchestra.run.smithi016.stdout:Context: system_u:object_r:fusefs_t:s0 2019-12-05T06:07:16.315 INFO:teuthology.orchestra.run.smithi016.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2019-12-05T06:07:16.315 INFO:teuthology.orchestra.run.smithi016.stdout:Modify: 2019-12-05 06:06:44.698630749 +0000 2019-12-05T06:07:16.315 INFO:teuthology.orchestra.run.smithi016.stdout:Change: 2019-12-05 06:07:16.185100088 +0000 2019-12-05T06:07:16.315 INFO:teuthology.orchestra.run.smithi016.stdout: Birth: - 2019-12-05T06:07:16.315 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2019-12-05T06:07:16.316 INFO:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2019-12-05T06:07:16.431 INFO:teuthology.orchestra.run.smithi016:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone git://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c5290805033483ba0cec165c06bde07dae5f19ec 2019-12-05T06:07:16.752 INFO:tasks.workunit.client.0.smithi016.stdout:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2019-12-05T06:07:27.356 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:07:27.369 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:07:57.518 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:07:57.523 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:08:18.286 INFO:tasks.workunit.client.0.smithi016.stderr:Note: checking out 'c5290805033483ba0cec165c06bde07dae5f19ec'. 2019-12-05T06:08:18.287 INFO:tasks.workunit.client.0.smithi016.stderr: 2019-12-05T06:08:18.287 INFO:tasks.workunit.client.0.smithi016.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2019-12-05T06:08:18.287 INFO:tasks.workunit.client.0.smithi016.stderr:changes and commit them, and you can discard any commits you make in this 2019-12-05T06:08:18.287 INFO:tasks.workunit.client.0.smithi016.stderr:state without impacting any branches by performing another checkout. 2019-12-05T06:08:18.287 INFO:tasks.workunit.client.0.smithi016.stderr: 2019-12-05T06:08:18.287 INFO:tasks.workunit.client.0.smithi016.stderr:If you want to create a new branch to retain commits you create, you may 2019-12-05T06:08:18.288 INFO:tasks.workunit.client.0.smithi016.stderr:do so (now or later) by using -b with the checkout command again. Example: 2019-12-05T06:08:18.288 INFO:tasks.workunit.client.0.smithi016.stderr: 2019-12-05T06:08:18.288 INFO:tasks.workunit.client.0.smithi016.stderr: git checkout -b new_branch_name 2019-12-05T06:08:18.288 INFO:tasks.workunit.client.0.smithi016.stderr: 2019-12-05T06:08:18.288 INFO:tasks.workunit.client.0.smithi016.stderr:HEAD is now at c529080... Merge PR #27866 into wip-pdonnell-testing-20191204.231200 2019-12-05T06:08:18.289 INFO:teuthology.orchestra.run.smithi016:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2019-12-05T06:08:18.413 INFO:tasks.workunit.client.0.smithi016.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2019-12-05T06:08:18.413 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-12-05T06:08:18.413 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2019-12-05T06:08:18.963 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2019-12-05T06:08:19.072 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2019-12-05T06:08:19.150 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2019-12-05T06:08:19.154 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Entering directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-12-05T06:08:19.154 INFO:tasks.workunit.client.0.smithi016.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2019-12-05T06:08:19.226 INFO:tasks.workunit.client.0.smithi016.stdout:make[1]: Leaving directory `/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2019-12-05T06:08:19.320 DEBUG:teuthology.orchestra.remote:smithi016:/home/ubuntu/cephtest/workunits.list.client.0 is 4KB 2019-12-05T06:08:19.395 INFO:tasks.workunit:Running workunits matching fs/snaps on client.0... 2019-12-05T06:08:19.395 INFO:tasks.workunit:Running workunit fs/snaps/snap-rm-diff.sh... 2019-12-05T06:08:19.395 INFO:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snap-rm-diff.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snap-rm-diff.sh 2019-12-05T06:08:19.594 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:08:19.780 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.771+0000 7f5bdbb4a700 1 Processor -- start 2019-12-05T06:08:19.780 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.771+0000 7f5bdbb4a700 1 -- start start 2019-12-05T06:08:19.780 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.773+0000 7f5bdbb4a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5bd41490a0 0x7f5bd4149490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:19.780 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.773+0000 7f5bdbb4a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5bd409e9f0 0x7f5bd4152df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:19.780 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.773+0000 7f5bdbb4a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd409ee50 con 0x7f5bd409f1c0 2019-12-05T06:08:19.781 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.773+0000 7f5bdbb4a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd40a3f40 con 0x7f5bd41490a0 2019-12-05T06:08:19.781 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.773+0000 7f5bdbb4a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd409d260 con 0x7f5bd409e9f0 2019-12-05T06:08:19.781 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.774+0000 7f5bda0e7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5bd41490a0 0x7f5bd4149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:19.781 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.774+0000 7f5bda0e7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5bd41490a0 0x7f5bd4149490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46838/0 (socket says 172.21.15.16:46838) 2019-12-05T06:08:19.781 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.774+0000 7f5bda0e7700 1 -- 172.21.15.16:0/568680294 learned_addr learned my addr 172.21.15.16:0/568680294 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:19.781 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.774+0000 7f5bda8e8700 1 --2- 172.21.15.16:0/568680294 >> v2:172.21.15.76:3300/0 conn(0x7f5bd409e9f0 0x7f5bd4152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:19.781 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.774+0000 7f5bda0e7700 1 -- 172.21.15.16:0/568680294 >> v2:172.21.15.76:3300/0 conn(0x7f5bd409e9f0 msgr2=0x7f5bd4152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:19.782 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.774+0000 7f5bda0e7700 1 --2- 172.21.15.16:0/568680294 >> v2:172.21.15.76:3300/0 conn(0x7f5bd409e9f0 0x7f5bd4152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:19.782 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.775+0000 7f5bda0e7700 1 -- 172.21.15.16:0/568680294 >> v1:172.21.15.76:6789/0 conn(0x7f5bd409f1c0 legacy=0x7f5bd409e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.782 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.775+0000 7f5bda0e7700 1 -- 172.21.15.16:0/568680294 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd409d8f0 con 0x7f5bd41490a0 2019-12-05T06:08:19.782 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.775+0000 7f5bda0e7700 1 --2- 172.21.15.16:0/568680294 >> v2:172.21.15.16:3300/0 conn(0x7f5bd41490a0 0x7f5bd4149490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5bc400f6c0 tx=0x7f5bc400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.782 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.775+0000 7f5bda0e7700 1 --2- 172.21.15.16:0/568680294 >> v2:172.21.15.16:3300/0 conn(0x7f5bd41490a0 0x7f5bd4149490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5bc400f6c0 tx=0x7f5bc400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.782 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.775+0000 7f5bda0e7700 1 --2- 172.21.15.16:0/568680294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd41490a0 0x7f5bd4149490 secure :-1 s=READY pgs=214 cs=0 l=1 rx=0x7f5bc400f6c0 tx=0x7f5bc400fa60).ready entity=mon.0 client_cookie=9097606bc21b8314 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:19.782 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.775+0000 7f5bda0e7700 1 --2- 172.21.15.16:0/568680294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd41490a0 0x7f5bd4149490 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f5bc400f6c0 tx=0x7f5bc400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.783 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.775+0000 7f5bda0e7700 1 --2- 172.21.15.16:0/568680294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd41490a0 0x7f5bd4149490 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f5bc400f6c0 tx=0x7f5bc400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.783 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.775+0000 7f5bda0e7700 1 --2- 172.21.15.16:0/568680294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd41490a0 0x7f5bd4149490 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7f5bc400f6c0 tx=0x7f5bc400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.783 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.775+0000 7f5bd90e5700 1 -- 172.21.15.16:0/568680294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc4000960 con 0x7f5bd41490a0 2019-12-05T06:08:19.783 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.775+0000 7f5bd90e5700 1 -- 172.21.15.16:0/568680294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc4000c90 con 0x7f5bd41490a0 2019-12-05T06:08:19.783 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.776+0000 7f5bd90e5700 1 -- 172.21.15.16:0/568680294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc40013a0 con 0x7f5bd41490a0 2019-12-05T06:08:19.783 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.776+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/568680294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd41490a0 msgr2=0x7f5bd4149490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.783 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.776+0000 7f5bdbb4a700 1 --2- 172.21.15.16:0/568680294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd41490a0 0x7f5bd4149490 secure :-1 s=READY pgs=214 cs=0 l=1 rx=0x7f5bc400f6c0 tx=0x7f5bc400fa60).stop 2019-12-05T06:08:19.784 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.778+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/568680294 shutdown_connections 2019-12-05T06:08:19.784 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.778+0000 7f5bdbb4a700 1 --2- 172.21.15.16:0/568680294 >> v2:172.21.15.76:3300/0 conn(0x7f5bd409e9f0 0x7f5bd4152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:19.784 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.778+0000 7f5bdbb4a700 1 --2- 172.21.15.16:0/568680294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd41490a0 0x7f5bd4149490 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:19.784 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.778+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/568680294 shutdown_connections 2019-12-05T06:08:19.797 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.778+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/568680294 wait complete. 2019-12-05T06:08:19.797 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.778+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/568680294 >> 172.21.15.16:0/568680294 conn(0x7f5bd4062220 msgr2=0x7f5bd40a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:19.809 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.783+0000 7f5bdbb4a700 1 Processor -- start 2019-12-05T06:08:19.809 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.783+0000 7f5bdbb4a700 1 -- start start 2019-12-05T06:08:19.809 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.784+0000 7f5bdbb4a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5bd411b460 0x7f5bd414fc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:19.809 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.784+0000 7f5bdbb4a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd4089bd0 con 0x7f5bd4088f00 2019-12-05T06:08:19.809 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.784+0000 7f5bdbb4a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bd412dd30 con 0x7f5bd414f860 2019-12-05T06:08:19.810 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.784+0000 7f5bdbb4a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bd412f540 con 0x7f5bd411b460 2019-12-05T06:08:19.810 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.785+0000 7f5bd98e6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5bd411b460 0x7f5bd414fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:19.810 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.785+0000 7f5bda8e8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5bd4088f00 0x7f5bd4156530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60244/0 (socket says 172.21.15.16:60244) 2019-12-05T06:08:19.810 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.785+0000 7f5bda8e8700 1 -- 172.21.15.16:0/39150558 learned_addr learned my addr 172.21.15.16:0/39150558 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:19.810 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.785+0000 7f5bd98e6700 1 -- 172.21.15.16:0/39150558 >> v1:172.21.15.76:6789/0 conn(0x7f5bd414f860 legacy=0x7f5bd41304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.810 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.785+0000 7f5bd98e6700 1 -- 172.21.15.16:0/39150558 >> v1:172.21.15.16:6789/0 conn(0x7f5bd4088f00 legacy=0x7f5bd4156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:19.811 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.785+0000 7f5bd98e6700 1 -- 172.21.15.16:0/39150558 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bd408a0d0 con 0x7f5bd411b460 2019-12-05T06:08:19.811 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.785+0000 7f5bd2ffd700 1 -- 172.21.15.16:0/39150558 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4236888684 0 0) 0x7f5bcc0095b0 con 0x7f5bd4088f00 2019-12-05T06:08:19.811 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> v2:172.21.15.16:3300/0 conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.811 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> v2:172.21.15.16:3300/0 conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.811 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=READY pgs=216 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:19.811 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.811 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bd2ffd700 1 -- 172.21.15.16:0/39150558 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc8009b20 con 0x7f5bd411b460 2019-12-05T06:08:19.811 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.812 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.812 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bd2ffd700 1 -- 172.21.15.16:0/39150558 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bc800aa40 con 0x7f5bd411b460 2019-12-05T06:08:19.812 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bd408a0d0 con 0x7f5bd411b460 2019-12-05T06:08:19.812 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bd2ffd700 1 -- 172.21.15.16:0/39150558 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bc800a1a0 con 0x7f5bd411b460 2019-12-05T06:08:19.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.786+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bd408a0d0 con 0x7f5bd411b460 2019-12-05T06:08:19.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.787+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.788+0000 7f5bd2ffd700 1 -- 172.21.15.16:0/39150558 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5bc8000e20 con 0x7f5bd411b460 2019-12-05T06:08:19.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.788+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.788+0000 7f5bd2ffd700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5bb4023630 0x7f5bb4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:19.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.789+0000 7f5bd2ffd700 1 -- 172.21.15.16:0/39150558 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f5bc80460e0 con 0x7f5bd411b460 2019-12-05T06:08:19.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.789+0000 7f5bda0e7700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5bb4023630 0x7f5bb4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:19.815 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.789+0000 7f5bda0e7700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5bb4023630 0x7f5bb4025ac0 crc :-1 s=READY pgs=113 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:19.815 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.804+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bd412fbd0 con 0x7f5bd411b460 2019-12-05T06:08:19.815 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.812+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:19.816 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:19.813+0000 7f5bd2ffd700 1 -- 172.21.15.16:0/39150558 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5bc802b160 con 0x7f5bd411b460 2019-12-05T06:08:20.065 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:20.067+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5bd412fbd0 con 0x7f5bd411b460 2019-12-05T06:08:20.121 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:20.124+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:20.122 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:20.124+0000 7f5bd2ffd700 1 -- 172.21.15.16:0/39150558 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v8) v1 ==== 167+0+0 (secure 0 0 0) 0x7f5bc8000f00 con 0x7f5bd411b460 2019-12-05T06:08:20.334 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:20.337+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f5bd412fbd0 con 0x7f5bd411b460 2019-12-05T06:08:21.134 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.137+0000 7f5bd98e6700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:21.135 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.137+0000 7f5bd2ffd700 1 -- 172.21.15.16:0/39150558 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v9) v1 ==== 167+0+0 (secure 0 0 0) 0x7f5bc802b160 con 0x7f5bd411b460 2019-12-05T06:08:21.150 INFO:tasks.workunit.client.0.smithi016.stderr:enabled new snapshots 2019-12-05T06:08:21.155 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.157+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5bb4023630 msgr2=0x7f5bb4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:21.155 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.157+0000 7f5bdbb4a700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5bb4023630 0x7f5bb4025ac0 crc :-1 s=READY pgs=113 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:21.155 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.157+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 msgr2=0x7f5bd414fc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:21.155 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.157+0000 7f5bdbb4a700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 secure :-1 s=READY pgs=216 cs=0 l=1 rx=0x7f5bc8010730 tx=0x7f5bc8010ad0).stop 2019-12-05T06:08:21.156 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.157+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 shutdown_connections 2019-12-05T06:08:21.156 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.157+0000 7f5bdbb4a700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5bb4023630 0x7f5bb4025ac0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:21.156 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.157+0000 7f5bdbb4a700 1 --2- 172.21.15.16:0/39150558 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5bd411b460 0x7f5bd414fc50 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:21.156 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.157+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 shutdown_connections 2019-12-05T06:08:21.157 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.157+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 wait complete. 2019-12-05T06:08:21.157 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:21.157+0000 7f5bdbb4a700 1 -- 172.21.15.16:0/39150558 >> 172.21.15.16:0/39150558 conn(0x7f5bd407ac10 msgr2=0x7f5bd40789c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:21.168 INFO:tasks.workunit.client.0.smithi016.stderr:+ wget -q http://download.ceph.com/qa/linux-2.6.33.tar.bz2 2019-12-05T06:08:23.909 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir foo 2019-12-05T06:08:23.915 INFO:tasks.workunit.client.0.smithi016.stderr:+ cp linux-2.6.33.tar.bz2 foo 2019-12-05T06:08:25.191 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir foo/.snap/barsnap 2019-12-05T06:08:26.110 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm foo/linux-2.6.33.tar.bz2 2019-12-05T06:08:26.143 INFO:tasks.workunit.client.0.smithi016.stderr:+ diff -q foo/.snap/barsnap/linux-2.6.33.tar.bz2 linux-2.6.33.tar.bz2 2019-12-05T06:08:26.375 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'passed: files are identical' 2019-12-05T06:08:26.376 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir foo/.snap/barsnap 2019-12-05T06:08:26.376 INFO:tasks.workunit.client.0.smithi016.stdout:passed: files are identical 2019-12-05T06:08:26.399 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2019-12-05T06:08:26.399 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2019-12-05T06:08:26.399 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:08:26.399 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:08:26.789 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-0.sh... 2019-12-05T06:08:26.790 INFO:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snaptest-0.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-0.sh 2019-12-05T06:08:26.916 INFO:tasks.workunit.client.0.smithi016.stderr:+ set -e 2019-12-05T06:08:26.916 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps false 2019-12-05T06:08:27.109 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.099+0000 7f730287d700 1 Processor -- start 2019-12-05T06:08:27.109 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.100+0000 7f730287d700 1 -- start start 2019-12-05T06:08:27.109 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.102+0000 7f730287d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72fc08c760 0x7f72fc08cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:27.109 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.102+0000 7f730287d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f72fc0908d0 0x7f72fc08d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:27.109 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.102+0000 7f730287d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc05a480 con 0x7f72fc08f900 2019-12-05T06:08:27.110 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.102+0000 7f730287d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc0627d0 con 0x7f72fc08c760 2019-12-05T06:08:27.110 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.102+0000 7f730287d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc08e970 con 0x7f72fc0908d0 2019-12-05T06:08:27.110 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.102+0000 7f7300e1a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72fc08c760 0x7f72fc08cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:27.110 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.102+0000 7f7300e1a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72fc08c760 0x7f72fc08cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46892/0 (socket says 172.21.15.16:46892) 2019-12-05T06:08:27.110 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.102+0000 7f7300e1a700 1 -- 172.21.15.16:0/2043865757 learned_addr learned my addr 172.21.15.16:0/2043865757 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:27.111 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.102+0000 7f72fbfff700 1 --2- 172.21.15.16:0/2043865757 >> v2:172.21.15.76:3300/0 conn(0x7f72fc0908d0 0x7f72fc08d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:27.111 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.103+0000 7f7300e1a700 1 -- 172.21.15.16:0/2043865757 >> v2:172.21.15.76:3300/0 conn(0x7f72fc0908d0 msgr2=0x7f72fc08d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:27.111 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.103+0000 7f7300e1a700 1 --2- 172.21.15.16:0/2043865757 >> v2:172.21.15.76:3300/0 conn(0x7f72fc0908d0 0x7f72fc08d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:27.111 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.103+0000 7f7300e1a700 1 -- 172.21.15.16:0/2043865757 >> v1:172.21.15.76:6789/0 conn(0x7f72fc08f900 legacy=0x7f72fc08fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:27.111 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.103+0000 7f7300e1a700 1 -- 172.21.15.16:0/2043865757 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc08f000 con 0x7f72fc08c760 2019-12-05T06:08:27.111 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.103+0000 7f72fbfff700 1 --2- 172.21.15.16:0/2043865757 >> v2:172.21.15.76:3300/0 conn(0x7f72fc0908d0 0x7f72fc08d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:27.111 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.103+0000 7f7300e1a700 1 --2- 172.21.15.16:0/2043865757 >> v2:172.21.15.16:3300/0 conn(0x7f72fc08c760 0x7f72fc08cb50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f72e400f740 tx=0x7f72e400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.112 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.103+0000 7f7300e1a700 1 --2- 172.21.15.16:0/2043865757 >> v2:172.21.15.16:3300/0 conn(0x7f72fc08c760 0x7f72fc08cb50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f72e400f740 tx=0x7f72e400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.112 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.103+0000 7f7300e1a700 1 --2- 172.21.15.16:0/2043865757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc08c760 0x7f72fc08cb50 secure :-1 s=READY pgs=219 cs=0 l=1 rx=0x7f72e400f740 tx=0x7f72e400fae0).ready entity=mon.0 client_cookie=917b6369318bd81f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:27.112 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.104+0000 7f7300e1a700 1 --2- 172.21.15.16:0/2043865757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc08c760 0x7f72fc08cb50 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f72e400f740 tx=0x7f72e400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.112 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.104+0000 7f7300e1a700 1 --2- 172.21.15.16:0/2043865757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc08c760 0x7f72fc08cb50 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f72e400f740 tx=0x7f72e400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.112 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.104+0000 7f7300e1a700 1 --2- 172.21.15.16:0/2043865757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc08c760 0x7f72fc08cb50 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7f72e400f740 tx=0x7f72e400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.112 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.104+0000 7f72fb7fe700 1 -- 172.21.15.16:0/2043865757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72e4000960 con 0x7f72fc08c760 2019-12-05T06:08:27.113 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.104+0000 7f72fb7fe700 1 -- 172.21.15.16:0/2043865757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72e4000c90 con 0x7f72fc08c760 2019-12-05T06:08:27.113 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.104+0000 7f72fb7fe700 1 -- 172.21.15.16:0/2043865757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72e40013a0 con 0x7f72fc08c760 2019-12-05T06:08:27.113 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.104+0000 7f730287d700 1 -- 172.21.15.16:0/2043865757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc08c760 msgr2=0x7f72fc08cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:27.113 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.104+0000 7f730287d700 1 --2- 172.21.15.16:0/2043865757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc08c760 0x7f72fc08cb50 secure :-1 s=READY pgs=219 cs=0 l=1 rx=0x7f72e400f740 tx=0x7f72e400fae0).stop 2019-12-05T06:08:27.113 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.106+0000 7f730287d700 1 -- 172.21.15.16:0/2043865757 shutdown_connections 2019-12-05T06:08:27.113 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.106+0000 7f730287d700 1 --2- 172.21.15.16:0/2043865757 >> v2:172.21.15.76:3300/0 conn(0x7f72fc0908d0 0x7f72fc08d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:27.113 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.106+0000 7f730287d700 1 --2- 172.21.15.16:0/2043865757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72fc08c760 0x7f72fc08cb50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:27.114 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.106+0000 7f730287d700 1 -- 172.21.15.16:0/2043865757 shutdown_connections 2019-12-05T06:08:27.114 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.106+0000 7f730287d700 1 -- 172.21.15.16:0/2043865757 wait complete. 2019-12-05T06:08:27.114 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.106+0000 7f730287d700 1 -- 172.21.15.16:0/2043865757 >> 172.21.15.16:0/2043865757 conn(0x7f72fc0a3870 msgr2=0x7f72fc0960f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:27.126 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.112+0000 7f730287d700 1 Processor -- start 2019-12-05T06:08:27.127 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.112+0000 7f730287d700 1 -- start start 2019-12-05T06:08:27.127 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.114+0000 7f730287d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f72fc14c570 0x7f72fc14c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:27.127 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.114+0000 7f730287d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f72fc086410 0x7f72fc085440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:27.127 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.114+0000 7f730287d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72fc07efc0 con 0x7f72fc085980 2019-12-05T06:08:27.127 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.115+0000 7f730287d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72fc083a80 con 0x7f72fc086410 2019-12-05T06:08:27.128 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.115+0000 7f730287d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72fc083d50 con 0x7f72fc14c570 2019-12-05T06:08:27.128 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.115+0000 7f7300e1a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f72fc14c570 0x7f72fc14c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:27.128 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.115+0000 7f730161b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f72fc085980 0x7f72fc080cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53372/0 (socket says 172.21.15.16:53372) 2019-12-05T06:08:27.128 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.115+0000 7f72fbfff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f72fc086410 0x7f72fc085440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:27.128 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.115+0000 7f730161b700 1 -- 172.21.15.16:0/3236903523 learned_addr learned my addr 172.21.15.16:0/3236903523 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:27.128 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.116+0000 7f72fbfff700 1 -- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f72fc14c570 msgr2=0x7f72fc14c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:27.129 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.116+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f72fc14c570 0x7f72fc14c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:27.129 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.116+0000 7f72fbfff700 1 -- 172.21.15.16:0/3236903523 >> v1:172.21.15.76:6789/0 conn(0x7f72fc085980 legacy=0x7f72fc080cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:27.129 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.116+0000 7f72fbfff700 1 -- 172.21.15.16:0/3236903523 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72fc084b40 con 0x7f72fc086410 2019-12-05T06:08:27.129 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.116+0000 7f72f97fa700 1 -- 172.21.15.16:0/3236903523 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005403017 0 0) 0x7f72f4000b10 con 0x7f72fc085980 2019-12-05T06:08:27.129 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.116+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> v2:172.21.15.76:3300/0 conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.129 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.116+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> v2:172.21.15.76:3300/0 conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.130 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.116+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=READY pgs=189 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:27.131 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.116+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=THROTTLE_DONE pgs=189 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.131 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.116+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=THROTTLE_DONE pgs=189 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.131 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.117+0000 7f72f97fa700 1 -- 172.21.15.16:0/3236903523 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72ec000cc0 con 0x7f72fc086410 2019-12-05T06:08:27.131 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.117+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=THROTTLE_DONE pgs=189 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.131 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.117+0000 7f72f97fa700 1 -- 172.21.15.16:0/3236903523 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ec00e370 con 0x7f72fc086410 2019-12-05T06:08:27.132 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.117+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72fc084b40 con 0x7f72fc086410 2019-12-05T06:08:27.132 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.117+0000 7f72f97fa700 1 -- 172.21.15.16:0/3236903523 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72ec00ea40 con 0x7f72fc086410 2019-12-05T06:08:27.132 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.117+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72fc084b40 con 0x7f72fc086410 2019-12-05T06:08:27.133 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.118+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=THROTTLE_DONE pgs=189 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.133 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.119+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=THROTTLE_DONE pgs=189 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.133 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.119+0000 7f72f97fa700 1 -- 172.21.15.16:0/3236903523 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f72ec00de80 con 0x7f72fc086410 2019-12-05T06:08:27.133 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.120+0000 7f72f97fa700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72e0023630 0x7f72e0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:27.133 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.120+0000 7f72f97fa700 1 -- 172.21.15.16:0/3236903523 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f72ec046080 con 0x7f72fc086410 2019-12-05T06:08:27.133 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.120+0000 7f7300e1a700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72e0023630 0x7f72e0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:27.133 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.121+0000 7f7300e1a700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72e0023630 0x7f72e0025ac0 crc :-1 s=READY pgs=115 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:27.134 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.131+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72fc084b40 con 0x7f72fc086410 2019-12-05T06:08:27.140 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.140+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=THROTTLE_DONE pgs=189 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.140 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.140+0000 7f72f97fa700 1 -- 172.21.15.16:0/3236903523 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f72ec000cc0 con 0x7f72fc086410 2019-12-05T06:08:27.415 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.416+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7f72fc084b40 con 0x7f72fc086410 2019-12-05T06:08:27.659 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:08:27.664 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:08:27.856 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.854+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=THROTTLE_DONE pgs=189 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:27.856 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:27.854+0000 7f72f97fa700 1 -- 172.21.15.16:0/3236903523 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 disabled new snapshots v10) v1 ==== 139+0+0 (secure 0 0 0) 0x7f72ec00dd90 con 0x7f72fc086410 2019-12-05T06:08:28.037 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.036+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7f72fc084b40 con 0x7f72fc086410 2019-12-05T06:08:28.936 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.937+0000 7f72fbfff700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=THROTTLE_DONE pgs=189 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:28.936 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.937+0000 7f72f97fa700 1 -- 172.21.15.16:0/3236903523 <== mon.1 v2:172.21.15.76:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 disabled new snapshots v11) v1 ==== 139+0+0 (secure 0 0 0) 0x7f72ec000cc0 con 0x7f72fc086410 2019-12-05T06:08:28.950 INFO:tasks.workunit.client.0.smithi016.stderr:disabled new snapshots 2019-12-05T06:08:28.955 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.956+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72e0023630 msgr2=0x7f72e0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:28.955 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.956+0000 7f730287d700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72e0023630 0x7f72e0025ac0 crc :-1 s=READY pgs=115 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:28.955 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.956+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 msgr2=0x7f72fc085440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:28.955 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.956+0000 7f730287d700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 secure :-1 s=READY pgs=189 cs=0 l=1 rx=0x7f72ec010720 tx=0x7f72ec010550).stop 2019-12-05T06:08:28.956 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.957+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 shutdown_connections 2019-12-05T06:08:28.956 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.957+0000 7f730287d700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72e0023630 0x7f72e0025ac0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:28.956 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.957+0000 7f730287d700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f72fc086410 0x7f72fc085440 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:28.957 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.957+0000 7f730287d700 1 --2- 172.21.15.16:0/3236903523 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f72fc14c570 0x7f72fc14c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:28.957 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.957+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 shutdown_connections 2019-12-05T06:08:28.957 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.957+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 wait complete. 2019-12-05T06:08:28.957 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:28.957+0000 7f730287d700 1 -- 172.21.15.16:0/3236903523 >> 172.21.15.16:0/3236903523 conn(0x7f72fc0a0120 msgr2=0x7f72fc073270 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:28.974 INFO:tasks.workunit.client.0.smithi016.stderr:+ expect_failure mkdir .snap/foo 2019-12-05T06:08:28.975 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir .snap/foo 2019-12-05T06:08:28.983 INFO:tasks.workunit.client.0.smithi016.stderr:mkdir: cannot create directory ‘.snap/foo’: Operation not permitted 2019-12-05T06:08:28.983 INFO:tasks.workunit.client.0.smithi016.stderr:+ return 0 2019-12-05T06:08:28.983 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:08:29.173 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.165+0000 7f3a5117f700 1 Processor -- start 2019-12-05T06:08:29.173 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.165+0000 7f3a5117f700 1 -- start start 2019-12-05T06:08:29.174 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.167+0000 7f3a5117f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 0x7f3a4c09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:29.174 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.167+0000 7f3a5117f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c149930 con 0x7f3a4c149490 2019-12-05T06:08:29.174 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.167+0000 7f3a5117f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c05f670 con 0x7f3a4c1490a0 2019-12-05T06:08:29.174 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.167+0000 7f3a5117f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c0a3f40 con 0x7f3a4c09f1c0 2019-12-05T06:08:29.174 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.167+0000 7f3a4ad9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 0x7f3a4c09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:29.174 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.167+0000 7f3a5097e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3a4c149490 0x7f3a4c152df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60322/0 (socket says 172.21.15.16:60322) 2019-12-05T06:08:29.175 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.167+0000 7f3a5097e700 1 -- 172.21.15.16:0/1706008767 learned_addr learned my addr 172.21.15.16:0/1706008767 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:29.175 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.167+0000 7f3a4b59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3a4c1490a0 0x7f3a4c09e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53390/0 (socket says 172.21.15.16:53390) 2019-12-05T06:08:29.175 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.168+0000 7f3a4ad9d700 1 -- 172.21.15.16:0/1706008767 >> v1:172.21.15.76:6789/0 conn(0x7f3a4c1490a0 legacy=0x7f3a4c09e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:29.175 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.168+0000 7f3a4ad9d700 1 -- 172.21.15.16:0/1706008767 >> v1:172.21.15.16:6789/0 conn(0x7f3a4c149490 legacy=0x7f3a4c152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:29.175 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.168+0000 7f3a4ad9d700 1 -- 172.21.15.16:0/1706008767 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c09d8f0 con 0x7f3a4c09f1c0 2019-12-05T06:08:29.175 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.168+0000 7f3a4a59c700 1 -- 172.21.15.16:0/1706008767 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 548944922 0 0) 0x7f3a44001480 con 0x7f3a4c149490 2019-12-05T06:08:29.175 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.168+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/1706008767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 0x7f3a4c09e9f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3a4000af50 tx=0x7f3a4000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.176 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.169+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/1706008767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 0x7f3a4c09e9f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3a4000af50 tx=0x7f3a4000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.176 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.169+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/1706008767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 0x7f3a4c09e9f0 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7f3a4000af50 tx=0x7f3a4000b760).ready entity=mon.2 client_cookie=61f777cdf3d876a2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:29.176 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.169+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/1706008767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 0x7f3a4c09e9f0 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f3a4000af50 tx=0x7f3a4000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.176 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.169+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/1706008767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 0x7f3a4c09e9f0 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f3a4000af50 tx=0x7f3a4000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.176 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.169+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/1706008767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 0x7f3a4c09e9f0 secure :-1 s=THROTTLE_DONE pgs=69 cs=0 l=1 rx=0x7f3a4000af50 tx=0x7f3a4000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.183 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.169+0000 7f3a4a59c700 1 -- 172.21.15.16:0/1706008767 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a40000960 con 0x7f3a4c09f1c0 2019-12-05T06:08:29.183 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.169+0000 7f3a4a59c700 1 -- 172.21.15.16:0/1706008767 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a400098e0 con 0x7f3a4c09f1c0 2019-12-05T06:08:29.183 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.169+0000 7f3a4a59c700 1 -- 172.21.15.16:0/1706008767 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a40009eb0 con 0x7f3a4c09f1c0 2019-12-05T06:08:29.183 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.169+0000 7f3a5117f700 1 -- 172.21.15.16:0/1706008767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 msgr2=0x7f3a4c09e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:29.183 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.169+0000 7f3a5117f700 1 --2- 172.21.15.16:0/1706008767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 0x7f3a4c09e9f0 secure :-1 s=READY pgs=69 cs=0 l=1 rx=0x7f3a4000af50 tx=0x7f3a4000b760).stop 2019-12-05T06:08:29.184 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.171+0000 7f3a5117f700 1 -- 172.21.15.16:0/1706008767 shutdown_connections 2019-12-05T06:08:29.184 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.171+0000 7f3a5117f700 1 --2- 172.21.15.16:0/1706008767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c09f1c0 0x7f3a4c09e9f0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:29.184 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.171+0000 7f3a5117f700 1 -- 172.21.15.16:0/1706008767 shutdown_connections 2019-12-05T06:08:29.184 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.171+0000 7f3a5117f700 1 -- 172.21.15.16:0/1706008767 wait complete. 2019-12-05T06:08:29.184 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.171+0000 7f3a5117f700 1 -- 172.21.15.16:0/1706008767 >> 172.21.15.16:0/1706008767 conn(0x7f3a4c062220 msgr2=0x7f3a4c0a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:29.187 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.176+0000 7f3a5117f700 1 Processor -- start 2019-12-05T06:08:29.187 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.177+0000 7f3a5117f700 1 -- start start 2019-12-05T06:08:29.188 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.178+0000 7f3a5117f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3a4c11b880 0x7f3a4c11a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:29.188 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.178+0000 7f3a5117f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c134c80 0x7f3a4c11adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:29.188 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.178+0000 7f3a5117f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a4c0799f0 con 0x7f3a4c134090 2019-12-05T06:08:29.188 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.178+0000 7f3a5117f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a4c136ec0 con 0x7f3a4c11b880 2019-12-05T06:08:29.188 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.178+0000 7f3a5117f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a4c136490 con 0x7f3a4c134c80 2019-12-05T06:08:29.188 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.178+0000 7f3a4ad9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3a4c11b880 0x7f3a4c11a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:29.188 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.178+0000 7f3a5097e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c134c80 0x7f3a4c11adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:29.189 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.178+0000 7f3a4b59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3a4c134090 0x7f3a4c134480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53396/0 (socket says 172.21.15.16:53396) 2019-12-05T06:08:29.189 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.178+0000 7f3a4b59e700 1 -- 172.21.15.16:0/3747203981 learned_addr learned my addr 172.21.15.16:0/3747203981 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:29.189 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.179+0000 7f3a4ad9d700 1 -- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c134c80 msgr2=0x7f3a4c11adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:29.189 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.179+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c134c80 0x7f3a4c11adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:29.189 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.179+0000 7f3a4ad9d700 1 -- 172.21.15.16:0/3747203981 >> v1:172.21.15.76:6789/0 conn(0x7f3a4c134090 legacy=0x7f3a4c134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:29.189 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.179+0000 7f3a4ad9d700 1 -- 172.21.15.16:0/3747203981 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a4c137550 con 0x7f3a4c11b880 2019-12-05T06:08:29.189 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.179+0000 7f3a5097e700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c134c80 0x7f3a4c11adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:29.190 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.179+0000 7f3a33fff700 1 -- 172.21.15.16:0/3747203981 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 906518278 0 0) 0x7f3a3c000da0 con 0x7f3a4c134090 2019-12-05T06:08:29.190 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.179+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> v2:172.21.15.76:3300/0 conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.190 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.180+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> v2:172.21.15.76:3300/0 conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.190 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.180+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=READY pgs=192 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:29.190 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.180+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.190 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.180+0000 7f3a33fff700 1 -- 172.21.15.16:0/3747203981 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a40000940 con 0x7f3a4c11b880 2019-12-05T06:08:29.190 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.180+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.191 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.180+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.191 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.180+0000 7f3a33fff700 1 -- 172.21.15.16:0/3747203981 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a40009980 con 0x7f3a4c11b880 2019-12-05T06:08:29.191 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.180+0000 7f3a33fff700 1 -- 172.21.15.16:0/3747203981 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a40000940 con 0x7f3a4c11b880 2019-12-05T06:08:29.191 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.180+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a4c137550 con 0x7f3a4c11b880 2019-12-05T06:08:29.191 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.180+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a4c137550 con 0x7f3a4c11b880 2019-12-05T06:08:29.191 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.181+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.191 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.182+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.192 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.182+0000 7f3a33fff700 1 -- 172.21.15.16:0/3747203981 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3a40009580 con 0x7f3a4c11b880 2019-12-05T06:08:29.192 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.182+0000 7f3a33fff700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a2c023630 0x7f3a2c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:29.192 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.182+0000 7f3a33fff700 1 -- 172.21.15.16:0/3747203981 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6031+0+0 (secure 0 0 0) 0x7f3a40045d50 con 0x7f3a4c11b880 2019-12-05T06:08:29.192 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.182+0000 7f3a4b59e700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a2c023630 0x7f3a2c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:29.192 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.183+0000 7f3a4b59e700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a2c023630 0x7f3a2c025ac0 crc :-1 s=READY pgs=116 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:29.195 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.197+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a4c137550 con 0x7f3a4c11b880 2019-12-05T06:08:29.204 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.205+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.204 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.205+0000 7f3a33fff700 1 -- 172.21.15.16:0/3747203981 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3a4003ffe0 con 0x7f3a4c11b880 2019-12-05T06:08:29.444 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.445+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3a4c137550 con 0x7f3a4c11b880 2019-12-05T06:08:29.947 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.948+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:29.947 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:29.948+0000 7f3a33fff700 1 -- 172.21.15.16:0/3747203981 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v12) v1 ==== 167+0+0 (secure 0 0 0) 0x7f3a4003ffe0 con 0x7f3a4c11b880 2019-12-05T06:08:30.152 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.152+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3a4c137550 con 0x7f3a4c11b880 2019-12-05T06:08:30.316 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:30.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.590+0000 7fc797fbd700 1 Processor -- start 2019-12-05T06:08:30.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.591+0000 7fc797fbd700 1 -- start start 2019-12-05T06:08:30.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.593+0000 7fc797fbd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc79008fef0 0x7fc79008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:30.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.593+0000 7fc797fbd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7900958e0 con 0x7fc79008c6d0 2019-12-05T06:08:30.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.593+0000 7fc797fbd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7900629e0 con 0x7fc79008d2c0 2019-12-05T06:08:30.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.593+0000 7fc797fbd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc79008df90 con 0x7fc79008fef0 2019-12-05T06:08:30.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.593+0000 7fc79655a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc79008c6d0 0x7fc79008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60336/0 (socket says 172.21.15.16:60336) 2019-12-05T06:08:30.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.593+0000 7fc79655a700 1 -- 172.21.15.16:0/3737615936 learned_addr learned my addr 172.21.15.16:0/3737615936 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:30.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.593+0000 7fc796d5b700 1 --2- 172.21.15.16:0/3737615936 >> v2:172.21.15.76:3300/0 conn(0x7fc79008fef0 0x7fc79008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:30.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.594+0000 7fc795558700 1 -- 172.21.15.16:0/3737615936 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3042568315 0 0) 0x7fc780001150 con 0x7fc79008c6d0 2019-12-05T06:08:30.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.594+0000 7fc795558700 1 -- 172.21.15.16:0/3737615936 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc778002e80 con 0x7fc79008c6d0 2019-12-05T06:08:30.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.594+0000 7fc796d5b700 1 -- 172.21.15.16:0/3737615936 >> v1:172.21.15.76:6789/0 conn(0x7fc79008d2c0 legacy=0x7fc79008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:30.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.594+0000 7fc795558700 1 -- 172.21.15.16:0/3737615936 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3800111001 0 0) 0x7fc784001400 con 0x7fc79008d2c0 2019-12-05T06:08:30.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.594+0000 7fc796d5b700 1 -- 172.21.15.16:0/3737615936 >> v1:172.21.15.16:6789/0 conn(0x7fc79008c6d0 legacy=0x7fc79008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:30.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.594+0000 7fc796d5b700 1 -- 172.21.15.16:0/3737615936 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc79008e620 con 0x7fc79008fef0 2019-12-05T06:08:30.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.594+0000 7fc796d5b700 1 --2- 172.21.15.16:0/3737615936 >> v2:172.21.15.76:3300/0 conn(0x7fc79008fef0 0x7fc79008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc78800af80 tx=0x7fc78800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.595+0000 7fc796d5b700 1 --2- 172.21.15.16:0/3737615936 >> v2:172.21.15.76:3300/0 conn(0x7fc79008fef0 0x7fc79008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc78800af80 tx=0x7fc78800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.595+0000 7fc796d5b700 1 --2- 172.21.15.16:0/3737615936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc79008fef0 0x7fc79008f720 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7fc78800af80 tx=0x7fc78800b790).ready entity=mon.1 client_cookie=295c604639f853f5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:30.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.595+0000 7fc796d5b700 1 --2- 172.21.15.16:0/3737615936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc79008fef0 0x7fc79008f720 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fc78800af80 tx=0x7fc78800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.595+0000 7fc795558700 1 -- 172.21.15.16:0/3737615936 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc788007240 con 0x7fc79008fef0 2019-12-05T06:08:30.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.595+0000 7fc796d5b700 1 --2- 172.21.15.16:0/3737615936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc79008fef0 0x7fc79008f720 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fc78800af80 tx=0x7fc78800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.595+0000 7fc796d5b700 1 --2- 172.21.15.16:0/3737615936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc79008fef0 0x7fc79008f720 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7fc78800af80 tx=0x7fc78800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.595+0000 7fc795558700 1 -- 172.21.15.16:0/3737615936 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc788000bb0 con 0x7fc79008fef0 2019-12-05T06:08:30.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.595+0000 7fc795558700 1 -- 172.21.15.16:0/3737615936 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc788009e60 con 0x7fc79008fef0 2019-12-05T06:08:30.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.595+0000 7fc797fbd700 1 -- 172.21.15.16:0/3737615936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc79008fef0 msgr2=0x7fc79008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:30.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.595+0000 7fc797fbd700 1 --2- 172.21.15.16:0/3737615936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc79008fef0 0x7fc79008f720 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7fc78800af80 tx=0x7fc78800b790).stop 2019-12-05T06:08:30.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.597+0000 7fc797fbd700 1 -- 172.21.15.16:0/3737615936 shutdown_connections 2019-12-05T06:08:30.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.597+0000 7fc797fbd700 1 --2- 172.21.15.16:0/3737615936 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc79008fef0 0x7fc79008f720 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:30.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.598+0000 7fc797fbd700 1 -- 172.21.15.16:0/3737615936 shutdown_connections 2019-12-05T06:08:30.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.598+0000 7fc797fbd700 1 -- 172.21.15.16:0/3737615936 wait complete. 2019-12-05T06:08:30.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.598+0000 7fc797fbd700 1 -- 172.21.15.16:0/3737615936 >> 172.21.15.16:0/3737615936 conn(0x7fc790061940 msgr2=0x7fc790096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:30.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.603+0000 7fc797fbd700 1 Processor -- start 2019-12-05T06:08:30.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.603+0000 7fc797fbd700 1 -- start start 2019-12-05T06:08:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.604+0000 7fc797fbd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc79006dc50 0x7fc79006e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:30.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.604+0000 7fc797fbd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.605+0000 7fc797fbd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc7901350d0 con 0x7fc790135450 2019-12-05T06:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.605+0000 7fc797fbd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc790132830 con 0x7fc79006dc50 2019-12-05T06:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.605+0000 7fc797fbd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc7901334f0 con 0x7fc790134c80 2019-12-05T06:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.605+0000 7fc79655a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc79006dc50 0x7fc79006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.605+0000 7fc796d5b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.605+0000 7fc79655a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc79006dc50 0x7fc79006e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:33030/0 (socket says 172.21.15.16:33030) 2019-12-05T06:08:30.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.605+0000 7fc79655a700 1 -- 172.21.15.16:0/2686313391 learned_addr learned my addr 172.21.15.16:0/2686313391 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc796d5b700 1 -- 172.21.15.16:0/2686313391 >> v2:172.21.15.76:3300/0 conn(0x7fc79006dc50 msgr2=0x7fc79006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> v2:172.21.15.76:3300/0 conn(0x7fc79006dc50 0x7fc79006e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc796d5b700 1 -- 172.21.15.16:0/2686313391 >> v1:172.21.15.76:6789/0 conn(0x7fc790135450 legacy=0x7fc790134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc796d5b700 1 -- 172.21.15.16:0/2686313391 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc790133b80 con 0x7fc790134c80 2019-12-05T06:08:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.606+0000 7fc78effd700 1 -- 172.21.15.16:0/2686313391 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc78800aeb0 con 0x7fc790134c80 2019-12-05T06:08:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.607+0000 7fc78effd700 1 -- 172.21.15.16:0/2686313391 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc7880099b0 con 0x7fc790134c80 2019-12-05T06:08:30.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.607+0000 7fc797fbd700 1 -- 172.21.15.16:0/2686313391 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc790133b80 con 0x7fc790134c80 2019-12-05T06:08:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.607+0000 7fc78effd700 1 -- 172.21.15.16:0/2686313391 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc78800a040 con 0x7fc790134c80 2019-12-05T06:08:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.607+0000 7fc797fbd700 1 -- 172.21.15.16:0/2686313391 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc790133b80 con 0x7fc790134c80 2019-12-05T06:08:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.608+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.609+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.609+0000 7fc78effd700 1 -- 172.21.15.16:0/2686313391 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc788009ba0 con 0x7fc790134c80 2019-12-05T06:08:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.609+0000 7fc78effd700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc770023630 0x7fc770025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.609+0000 7fc78effd700 1 -- 172.21.15.16:0/2686313391 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6087+0+0 (secure 0 0 0) 0x7fc788045d20 con 0x7fc790134c80 2019-12-05T06:08:30.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.610+0000 7fc79655a700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc770023630 0x7fc770025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.610+0000 7fc79655a700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc770023630 0x7fc770025ac0 crc :-1 s=READY pgs=117 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.622+0000 7fc797fbd700 1 -- 172.21.15.16:0/2686313391 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc790133b80 con 0x7fc790134c80 2019-12-05T06:08:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.632+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.632+0000 7fc78effd700 1 -- 172.21.15.16:0/2686313391 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc7880095b0 con 0x7fc790134c80 2019-12-05T06:08:30.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.900+0000 7fc797fbd700 1 -- 172.21.15.16:0/2686313391 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc790133b80 con 0x7fc790134c80 2019-12-05T06:08:30.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.901+0000 7fc796d5b700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=THROTTLE_DONE pgs=70 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.901+0000 7fc78effd700 1 -- 172.21.15.16:0/2686313391 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+2104 (secure 0 0 0) 0x7fc788009ba0 con 0x7fc790134c80 2019-12-05T06:08:30.904 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 12 2019-12-05T06:08:30.905 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:30.905 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":12,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":12,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:29.941662+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:30.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 -- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc770023630 msgr2=0x7fc770025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:30.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc770023630 0x7fc770025ac0 crc :-1 s=READY pgs=117 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:30.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 -- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 msgr2=0x7fc79006e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:30.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 secure :-1 s=READY pgs=70 cs=0 l=1 rx=0x7fc7880103b0 tx=0x7fc788010750).stop 2019-12-05T06:08:30.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 -- 172.21.15.16:0/2686313391 shutdown_connections 2019-12-05T06:08:30.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc770023630 0x7fc770025ac0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:30.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 --2- 172.21.15.16:0/2686313391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc790134c80 0x7fc79006e580 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:30.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 --2- 172.21.15.16:0/2686313391 >> v2:172.21.15.76:3300/0 conn(0x7fc79006dc50 0x7fc79006e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:30.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 -- 172.21.15.16:0/2686313391 shutdown_connections 2019-12-05T06:08:30.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 -- 172.21.15.16:0/2686313391 wait complete. 2019-12-05T06:08:30.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:30.907+0000 7fc797fbd700 1 -- 172.21.15.16:0/2686313391 >> 172.21.15.16:0/2686313391 conn(0x7fc7900a2850 msgr2=0x7fc79006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:30.923 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:08:30.924 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:08:30.924 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:08:30.936 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:08:30.937 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:30.960 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.961+0000 7f3a4ad9d700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:30.960 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.961+0000 7f3a33fff700 1 -- 172.21.15.16:0/3747203981 <== mon.1 v2:172.21.15.76:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v13) v1 ==== 167+0+0 (secure 0 0 0) 0x7f3a40000940 con 0x7f3a4c11b880 2019-12-05T06:08:30.966 INFO:tasks.workunit.client.0.smithi016.stderr:enabled new snapshots 2019-12-05T06:08:30.970 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.970+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a2c023630 msgr2=0x7f3a2c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:30.971 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.970+0000 7f3a5117f700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a2c023630 0x7f3a2c025ac0 crc :-1 s=READY pgs=116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:30.971 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.970+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 msgr2=0x7f3a4c11a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:30.971 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.970+0000 7f3a5117f700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 secure :-1 s=READY pgs=192 cs=0 l=1 rx=0x7f3a400103a0 tx=0x7f3a40010740).stop 2019-12-05T06:08:30.971 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.970+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 shutdown_connections 2019-12-05T06:08:30.971 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.970+0000 7f3a5117f700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a2c023630 0x7f3a2c025ac0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:30.972 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.970+0000 7f3a5117f700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a4c134c80 0x7f3a4c11adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:30.972 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.970+0000 7f3a5117f700 1 --2- 172.21.15.16:0/3747203981 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3a4c11b880 0x7f3a4c11a8b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:30.972 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.970+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 shutdown_connections 2019-12-05T06:08:30.972 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.971+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 wait complete. 2019-12-05T06:08:30.972 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:30.971+0000 7f3a5117f700 1 -- 172.21.15.16:0/3747203981 >> 172.21.15.16:0/3747203981 conn(0x7f3a4c07c510 msgr2=0x7f3a4c11f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:30.980 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo asdf 2019-12-05T06:08:31.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.183+0000 7f2e7e858700 1 Processor -- start 2019-12-05T06:08:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.183+0000 7f2e7e858700 1 -- start start 2019-12-05T06:08:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.185+0000 7f2e7e858700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7808c6d0 0x7f2e7808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.185+0000 7f2e7e858700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e7808fef0 0x7f2e7808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.185+0000 7f2e7e858700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e780958e0 con 0x7f2e7808ef20 2019-12-05T06:08:31.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.185+0000 7f2e7e858700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e780a3f70 con 0x7f2e7808fef0 2019-12-05T06:08:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.185+0000 7f2e7e858700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e7808d830 con 0x7f2e7808c6d0 2019-12-05T06:08:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.185+0000 7f2e77fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e7808fef0 0x7f2e7808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.186+0000 7f2e77fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e7808fef0 0x7f2e7808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46954/0 (socket says 172.21.15.16:46954) 2019-12-05T06:08:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.186+0000 7f2e77fff700 1 -- 172.21.15.16:0/352086110 learned_addr learned my addr 172.21.15.16:0/352086110 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.186+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/352086110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7808c6d0 0x7f2e7808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.186+0000 7f2e777fe700 1 -- 172.21.15.16:0/352086110 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 899967427 0 0) 0x7f2e60001080 con 0x7f2e7808ef20 2019-12-05T06:08:31.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.186+0000 7f2e77fff700 1 -- 172.21.15.16:0/352086110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7808c6d0 msgr2=0x7f2e7808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.186+0000 7f2e77fff700 1 --2- 172.21.15.16:0/352086110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7808c6d0 0x7f2e7808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.186+0000 7f2e77fff700 1 -- 172.21.15.16:0/352086110 >> v1:172.21.15.76:6789/0 conn(0x7f2e7808ef20 legacy=0x7f2e7808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.186+0000 7f2e77fff700 1 -- 172.21.15.16:0/352086110 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7808d540 con 0x7f2e7808fef0 2019-12-05T06:08:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.186+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/352086110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7808c6d0 0x7f2e7808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.186+0000 7f2e77fff700 1 --2- 172.21.15.16:0/352086110 >> v2:172.21.15.16:3300/0 conn(0x7f2e7808fef0 0x7f2e7808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2e6800f740 tx=0x7f2e6800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.187+0000 7f2e77fff700 1 --2- 172.21.15.16:0/352086110 >> v2:172.21.15.16:3300/0 conn(0x7f2e7808fef0 0x7f2e7808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2e6800f740 tx=0x7f2e6800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.187+0000 7f2e77fff700 1 --2- 172.21.15.16:0/352086110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808d000 secure :-1 s=READY pgs=224 cs=0 l=1 rx=0x7f2e6800f740 tx=0x7f2e6800fae0).ready entity=mon.0 client_cookie=6c404cddb1ed47cf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.187+0000 7f2e77fff700 1 --2- 172.21.15.16:0/352086110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808d000 secure :-1 s=THROTTLE_DONE pgs=224 cs=0 l=1 rx=0x7f2e6800f740 tx=0x7f2e6800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.187+0000 7f2e777fe700 1 -- 172.21.15.16:0/352086110 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e68000960 con 0x7f2e7808fef0 2019-12-05T06:08:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.187+0000 7f2e77fff700 1 --2- 172.21.15.16:0/352086110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808d000 secure :-1 s=THROTTLE_DONE pgs=224 cs=0 l=1 rx=0x7f2e6800f740 tx=0x7f2e6800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.187+0000 7f2e77fff700 1 --2- 172.21.15.16:0/352086110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808d000 secure :-1 s=THROTTLE_DONE pgs=224 cs=0 l=1 rx=0x7f2e6800f740 tx=0x7f2e6800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.187+0000 7f2e777fe700 1 -- 172.21.15.16:0/352086110 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e68000c90 con 0x7f2e7808fef0 2019-12-05T06:08:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.187+0000 7f2e777fe700 1 -- 172.21.15.16:0/352086110 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e680013a0 con 0x7f2e7808fef0 2019-12-05T06:08:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.187+0000 7f2e7e858700 1 -- 172.21.15.16:0/352086110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 msgr2=0x7f2e7808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:31.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.187+0000 7f2e7e858700 1 --2- 172.21.15.16:0/352086110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808d000 secure :-1 s=READY pgs=224 cs=0 l=1 rx=0x7f2e6800f740 tx=0x7f2e6800fae0).stop 2019-12-05T06:08:31.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.188+0000 7f2e7e858700 1 -- 172.21.15.16:0/352086110 shutdown_connections 2019-12-05T06:08:31.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.188+0000 7f2e7e858700 1 --2- 172.21.15.16:0/352086110 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808d000 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:31.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.188+0000 7f2e7e858700 1 --2- 172.21.15.16:0/352086110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7808c6d0 0x7f2e7808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:31.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.189+0000 7f2e7e858700 1 -- 172.21.15.16:0/352086110 shutdown_connections 2019-12-05T06:08:31.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.189+0000 7f2e7e858700 1 -- 172.21.15.16:0/352086110 wait complete. 2019-12-05T06:08:31.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.189+0000 7f2e7e858700 1 -- 172.21.15.16:0/352086110 >> 172.21.15.16:0/352086110 conn(0x7f2e78061940 msgr2=0x7f2e78096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:31.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.194+0000 7f2e7e858700 1 Processor -- start 2019-12-05T06:08:31.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.194+0000 7f2e7e858700 1 -- start start 2019-12-05T06:08:31.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7e858700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2e78096e90 0x7f2e7809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:31.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7e858700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e78084f30 0x7f2e7809b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:31.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7e858700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e78097280 0x7f2e780976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:31.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7e858700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7809a450 con 0x7f2e78097280 2019-12-05T06:08:31.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7e858700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e78099a20 con 0x7f2e78096e90 2019-12-05T06:08:31.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7e858700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e78099cf0 con 0x7f2e78084f30 2019-12-05T06:08:31.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7d5f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e78097280 0x7f2e780976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:31.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7cdf5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2e78096e90 0x7f2e7809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:31.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e77fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e78084f30 0x7f2e7809b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:31.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7d5f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e78097280 0x7f2e780976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46962/0 (socket says 172.21.15.16:46962) 2019-12-05T06:08:31.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7cdf5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2e78096e90 0x7f2e7809b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:33050/0 (socket says 172.21.15.16:33050) 2019-12-05T06:08:31.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e77fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e78084f30 0x7f2e7809b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:50948/0 (socket says 172.21.15.16:50948) 2019-12-05T06:08:31.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.196+0000 7f2e7cdf5700 1 -- 172.21.15.16:0/3884624625 learned_addr learned my addr 172.21.15.16:0/3884624625 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:31.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.197+0000 7f2e7cdf5700 1 -- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e78084f30 msgr2=0x7f2e7809b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:31.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.197+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e78084f30 0x7f2e7809b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:31.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.197+0000 7f2e7cdf5700 1 -- 172.21.15.16:0/3884624625 >> v2:172.21.15.16:3300/0 conn(0x7f2e78097280 msgr2=0x7f2e780976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:31.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.197+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> v2:172.21.15.16:3300/0 conn(0x7f2e78097280 0x7f2e780976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:31.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.197+0000 7f2e7cdf5700 1 -- 172.21.15.16:0/3884624625 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7809aae0 con 0x7f2e78096e90 2019-12-05T06:08:31.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.197+0000 7f2e7d5f6700 1 --2- 172.21.15.16:0/3884624625 >> v2:172.21.15.16:3300/0 conn(0x7f2e78097280 0x7f2e780976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:31.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.197+0000 7f2e77fff700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e78084f30 0x7f2e7809b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:31.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.197+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> v2:172.21.15.76:3300/0 conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.197+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> v2:172.21.15.76:3300/0 conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.197+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=READY pgs=197 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:31.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.198+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.198+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.198+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.198+0000 7f2e757fa700 1 -- 172.21.15.16:0/3884624625 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e70000cc0 con 0x7f2e78096e90 2019-12-05T06:08:31.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.198+0000 7f2e757fa700 1 -- 172.21.15.16:0/3884624625 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e7000f000 con 0x7f2e78096e90 2019-12-05T06:08:31.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.198+0000 7f2e7e858700 1 -- 172.21.15.16:0/3884624625 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e7809aae0 con 0x7f2e78096e90 2019-12-05T06:08:31.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.198+0000 7f2e757fa700 1 -- 172.21.15.16:0/3884624625 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e7000f6d0 con 0x7f2e78096e90 2019-12-05T06:08:31.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.198+0000 7f2e7e858700 1 -- 172.21.15.16:0/3884624625 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e7809aae0 con 0x7f2e78096e90 2019-12-05T06:08:31.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.199+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.200+0000 7f2e757fa700 1 -- 172.21.15.16:0/3884624625 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2e7000ef60 con 0x7f2e78096e90 2019-12-05T06:08:31.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.200+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.200+0000 7f2e757fa700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 0x7f2e5c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:31.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.200+0000 7f2e757fa700 1 -- 172.21.15.16:0/3884624625 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6087+0+0 (secure 0 0 0) 0x7f2e700460f0 con 0x7f2e78096e90 2019-12-05T06:08:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.201+0000 7f2e77fff700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 0x7f2e5c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.201+0000 7f2e77fff700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 0x7f2e5c025ac0 crc :-1 s=READY pgs=118 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:31.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.215+0000 7f2e7e858700 1 -- 172.21.15.16:0/3884624625 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7809aae0 con 0x7f2e78096e90 2019-12-05T06:08:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.224+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.224+0000 7f2e757fa700 1 -- 172.21.15.16:0/3884624625 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2e7000ed50 con 0x7f2e78096e90 2019-12-05T06:08:31.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.471+0000 7f2e7e858700 1 -- 172.21.15.16:0/3884624625 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e7809aae0 con 0x7f2e78096e90 2019-12-05T06:08:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.473+0000 7f2e7cdf5700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:31.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.473+0000 7f2e757fa700 1 -- 172.21.15.16:0/3884624625 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f2e7000ef40 con 0x7f2e78096e90 2019-12-05T06:08:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2019-12-05T06:08:31.473 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:31.473 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:30.953902+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:31.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 -- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 msgr2=0x7f2e5c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:31.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 0x7f2e5c025ac0 crc :-1 s=READY pgs=118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:31.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 -- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 msgr2=0x7f2e7809b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:31.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 0x7f2e7809b3e0 secure :-1 s=READY pgs=197 cs=0 l=1 rx=0x7f2e700107b0 tx=0x7f2e70010b50).stop 2019-12-05T06:08:31.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 -- 172.21.15.16:0/3884624625 shutdown_connections 2019-12-05T06:08:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 0x7f2e5c025ac0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 --2- 172.21.15.16:0/3884624625 >> v2:172.21.15.16:3300/0 conn(0x7f2e78097280 0x7f2e780976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e78084f30 0x7f2e7809b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 --2- 172.21.15.16:0/3884624625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2e78096e90 0x7f2e7809b3e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 -- 172.21.15.16:0/3884624625 shutdown_connections 2019-12-05T06:08:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 -- 172.21.15.16:0/3884624625 wait complete. 2019-12-05T06:08:31.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:31.477+0000 7f2e7e858700 1 -- 172.21.15.16:0/3884624625 >> 172.21.15.16:0/3884624625 conn(0x7f2e780714f0 msgr2=0x7f2e7809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:31.488 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:08:33.501 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.772+0000 7fa661876700 1 Processor -- start 2019-12-05T06:08:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.773+0000 7fa661876700 1 -- start start 2019-12-05T06:08:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.774+0000 7fa661876700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa65c08c6d0 0x7fa65c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.774+0000 7fa661876700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa65c08fef0 0x7fa65c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.774+0000 7fa661876700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65c0958e0 con 0x7fa65c08ef20 2019-12-05T06:08:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.774+0000 7fa661876700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c0a3f70 con 0x7fa65c08fef0 2019-12-05T06:08:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.774+0000 7fa661876700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c08d830 con 0x7fa65c08c6d0 2019-12-05T06:08:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.775+0000 7fa65affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa65c08fef0 0x7fa65c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.775+0000 7fa65affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa65c08fef0 0x7fa65c08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46974/0 (socket says 172.21.15.16:46974) 2019-12-05T06:08:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.775+0000 7fa65bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa65c08ef20 0x7fa65c08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60372/0 (socket says 172.21.15.16:60372) 2019-12-05T06:08:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.775+0000 7fa65b7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa65c08c6d0 0x7fa65c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.775+0000 7fa65affd700 1 -- 172.21.15.16:0/1886306055 learned_addr learned my addr 172.21.15.16:0/1886306055 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.775+0000 7fa65affd700 1 -- 172.21.15.16:0/1886306055 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa65c08c6d0 msgr2=0x7fa65c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.775+0000 7fa65affd700 1 --2- 172.21.15.16:0/1886306055 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa65c08c6d0 0x7fa65c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.775+0000 7fa65affd700 1 -- 172.21.15.16:0/1886306055 >> v1:172.21.15.16:6789/0 conn(0x7fa65c08ef20 legacy=0x7fa65c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.776+0000 7fa65affd700 1 -- 172.21.15.16:0/1886306055 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c08d540 con 0x7fa65c08fef0 2019-12-05T06:08:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.775+0000 7fa65a7fc700 1 -- 172.21.15.16:0/1886306055 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 481385405 0 0) 0x7fa654000d30 con 0x7fa65c08ef20 2019-12-05T06:08:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.776+0000 7fa65affd700 1 --2- 172.21.15.16:0/1886306055 >> v2:172.21.15.16:3300/0 conn(0x7fa65c08fef0 0x7fa65c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa65000f6c0 tx=0x7fa65000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.776+0000 7fa65affd700 1 --2- 172.21.15.16:0/1886306055 >> v2:172.21.15.16:3300/0 conn(0x7fa65c08fef0 0x7fa65c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa65000f6c0 tx=0x7fa65000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.776+0000 7fa65affd700 1 --2- 172.21.15.16:0/1886306055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c08fef0 0x7fa65c08d000 secure :-1 s=READY pgs=226 cs=0 l=1 rx=0x7fa65000f6c0 tx=0x7fa65000fa60).ready entity=mon.0 client_cookie=3e455491fa51eb60 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.776+0000 7fa65affd700 1 --2- 172.21.15.16:0/1886306055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c08fef0 0x7fa65c08d000 secure :-1 s=THROTTLE_DONE pgs=226 cs=0 l=1 rx=0x7fa65000f6c0 tx=0x7fa65000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.776+0000 7fa65a7fc700 1 -- 172.21.15.16:0/1886306055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa650000960 con 0x7fa65c08fef0 2019-12-05T06:08:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.776+0000 7fa65affd700 1 --2- 172.21.15.16:0/1886306055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c08fef0 0x7fa65c08d000 secure :-1 s=THROTTLE_DONE pgs=226 cs=0 l=1 rx=0x7fa65000f6c0 tx=0x7fa65000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.776+0000 7fa65affd700 1 --2- 172.21.15.16:0/1886306055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c08fef0 0x7fa65c08d000 secure :-1 s=THROTTLE_DONE pgs=226 cs=0 l=1 rx=0x7fa65000f6c0 tx=0x7fa65000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.777+0000 7fa65a7fc700 1 -- 172.21.15.16:0/1886306055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa650000c90 con 0x7fa65c08fef0 2019-12-05T06:08:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.777+0000 7fa65a7fc700 1 -- 172.21.15.16:0/1886306055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6500013a0 con 0x7fa65c08fef0 2019-12-05T06:08:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.777+0000 7fa661876700 1 -- 172.21.15.16:0/1886306055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c08fef0 msgr2=0x7fa65c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.777+0000 7fa661876700 1 --2- 172.21.15.16:0/1886306055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c08fef0 0x7fa65c08d000 secure :-1 s=READY pgs=226 cs=0 l=1 rx=0x7fa65000f6c0 tx=0x7fa65000fa60).stop 2019-12-05T06:08:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.778+0000 7fa661876700 1 -- 172.21.15.16:0/1886306055 shutdown_connections 2019-12-05T06:08:33.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.778+0000 7fa661876700 1 --2- 172.21.15.16:0/1886306055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c08fef0 0x7fa65c08d000 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.778+0000 7fa661876700 1 --2- 172.21.15.16:0/1886306055 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa65c08c6d0 0x7fa65c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.778+0000 7fa661876700 1 -- 172.21.15.16:0/1886306055 shutdown_connections 2019-12-05T06:08:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.778+0000 7fa661876700 1 -- 172.21.15.16:0/1886306055 wait complete. 2019-12-05T06:08:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.779+0000 7fa661876700 1 -- 172.21.15.16:0/1886306055 >> 172.21.15.16:0/1886306055 conn(0x7fa65c061940 msgr2=0x7fa65c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:33.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.784+0000 7fa661876700 1 Processor -- start 2019-12-05T06:08:33.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.784+0000 7fa661876700 1 -- start start 2019-12-05T06:08:33.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa661876700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa65c1333a0 0x7fa65c128360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:33.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa661876700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa65c128fe0 0x7fa65c1288a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:33.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa661876700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa65c06ed20 0x7fa65c06f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa661876700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c088730 con 0x7fa65c1333a0 2019-12-05T06:08:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa661876700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c089810 con 0x7fa65c06ed20 2019-12-05T06:08:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa661876700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c088de0 con 0x7fa65c128fe0 2019-12-05T06:08:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa65b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa65c1333a0 0x7fa65c128360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa65b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa65c1333a0 0x7fa65c128360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:46976/0 (socket says 172.21.15.16:46976) 2019-12-05T06:08:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa65b7fe700 1 -- 172.21.15.16:0/2008984755 learned_addr learned my addr 172.21.15.16:0/2008984755 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:33.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa65affd700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa65c128fe0 0x7fa65c1288a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.786+0000 7fa65bfff700 1 --2- 172.21.15.16:0/2008984755 >> v2:172.21.15.76:3300/0 conn(0x7fa65c06ed20 0x7fa65c06f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.787+0000 7fa65b7fe700 1 -- 172.21.15.16:0/2008984755 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa65c128fe0 msgr2=0x7fa65c1288a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.787+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa65c128fe0 0x7fa65c1288a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.787+0000 7fa65b7fe700 1 -- 172.21.15.16:0/2008984755 >> v2:172.21.15.76:3300/0 conn(0x7fa65c06ed20 msgr2=0x7fa65c06f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.787+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> v2:172.21.15.76:3300/0 conn(0x7fa65c06ed20 0x7fa65c06f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:33.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.787+0000 7fa65b7fe700 1 -- 172.21.15.16:0/2008984755 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c089ea0 con 0x7fa65c1333a0 2019-12-05T06:08:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.787+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> v2:172.21.15.16:3300/0 conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.787+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> v2:172.21.15.16:3300/0 conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.787+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=READY pgs=227 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.787+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.788+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.787+0000 7fa643fff700 1 -- 172.21.15.16:0/2008984755 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa64c000940 con 0x7fa65c1333a0 2019-12-05T06:08:33.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.788+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.788+0000 7fa643fff700 1 -- 172.21.15.16:0/2008984755 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c000e80 con 0x7fa65c1333a0 2019-12-05T06:08:33.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.788+0000 7fa661876700 1 -- 172.21.15.16:0/2008984755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65c089ea0 con 0x7fa65c1333a0 2019-12-05T06:08:33.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.788+0000 7fa643fff700 1 -- 172.21.15.16:0/2008984755 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa64c001490 con 0x7fa65c1333a0 2019-12-05T06:08:33.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.788+0000 7fa661876700 1 -- 172.21.15.16:0/2008984755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65c089ea0 con 0x7fa65c1333a0 2019-12-05T06:08:33.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.789+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.790+0000 7fa643fff700 1 -- 172.21.15.16:0/2008984755 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa64c001070 con 0x7fa65c1333a0 2019-12-05T06:08:33.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.790+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.790+0000 7fa643fff700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa63c023630 0x7fa63c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:33.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.790+0000 7fa643fff700 1 -- 172.21.15.16:0/2008984755 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6087+0+0 (secure 0 0 0) 0x7fa64c045d50 con 0x7fa65c1333a0 2019-12-05T06:08:33.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.790+0000 7fa65affd700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa63c023630 0x7fa63c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:33.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.791+0000 7fa65affd700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa63c023630 0x7fa63c025ac0 crc :-1 s=READY pgs=119 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:33.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.804+0000 7fa661876700 1 -- 172.21.15.16:0/2008984755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65c089ea0 con 0x7fa65c1333a0 2019-12-05T06:08:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.813+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:33.813+0000 7fa643fff700 1 -- 172.21.15.16:0/2008984755 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa64c0092e0 con 0x7fa65c1333a0 2019-12-05T06:08:34.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.056+0000 7fa661876700 1 -- 172.21.15.16:0/2008984755 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa65c089ea0 con 0x7fa65c1333a0 2019-12-05T06:08:34.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.057+0000 7fa65b7fe700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=THROTTLE_DONE pgs=227 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:34.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.057+0000 7fa643fff700 1 -- 172.21.15.16:0/2008984755 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2104 (secure 0 0 0) 0x7fa64c001920 con 0x7fa65c1333a0 2019-12-05T06:08:34.058 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2019-12-05T06:08:34.059 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:34.059 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:30.953902+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:34.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 -- 172.21.15.16:0/2008984755 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa63c023630 msgr2=0x7fa63c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:34.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa63c023630 0x7fa63c025ac0 crc :-1 s=READY pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:34.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 -- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 msgr2=0x7fa65c128360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 0x7fa65c128360 secure :-1 s=READY pgs=227 cs=0 l=1 rx=0x7fa64c010450 tx=0x7fa64c010300).stop 2019-12-05T06:08:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 -- 172.21.15.16:0/2008984755 shutdown_connections 2019-12-05T06:08:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 --2- 172.21.15.16:0/2008984755 >> v2:172.21.15.76:3300/0 conn(0x7fa65c06ed20 0x7fa65c06f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa65c128fe0 0x7fa65c1288a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa63c023630 0x7fa63c025ac0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 --2- 172.21.15.16:0/2008984755 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa65c1333a0 0x7fa65c128360 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:34.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 -- 172.21.15.16:0/2008984755 shutdown_connections 2019-12-05T06:08:34.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 -- 172.21.15.16:0/2008984755 wait complete. 2019-12-05T06:08:34.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:34.060+0000 7fa661876700 1 -- 172.21.15.16:0/2008984755 >> 172.21.15.16:0/2008984755 conn(0x7fa65c09de30 msgr2=0x7fa65c070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:34.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:08:36.073 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.342+0000 7f4866f18700 1 Processor -- start 2019-12-05T06:08:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.342+0000 7f4866f18700 1 -- start start 2019-12-05T06:08:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.344+0000 7f4866f18700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 0x7f486008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.344+0000 7f4866f18700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48600958e0 con 0x7f486008c6d0 2019-12-05T06:08:36.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.344+0000 7f4866f18700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48600629e0 con 0x7f486008fef0 2019-12-05T06:08:36.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.344+0000 7f4866f18700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486008df90 con 0x7f486008d2c0 2019-12-05T06:08:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.344+0000 7f48654b5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f486008c6d0 0x7f486008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60388/0 (socket says 172.21.15.16:60388) 2019-12-05T06:08:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.345+0000 7f48654b5700 1 -- 172.21.15.16:0/3878656084 learned_addr learned my addr 172.21.15.16:0/3878656084 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.345+0000 7f4864cb4700 1 --2- 172.21.15.16:0/3878656084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 0x7f486008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:36.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.345+0000 7f485ffff700 1 -- 172.21.15.16:0/3878656084 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582505554 0 0) 0x7f48580014d0 con 0x7f486008c6d0 2019-12-05T06:08:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.345+0000 7f485ffff700 1 -- 172.21.15.16:0/3878656084 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848002e80 con 0x7f486008c6d0 2019-12-05T06:08:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.345+0000 7f485ffff700 1 -- 172.21.15.16:0/3878656084 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718838910 0 0) 0x7f48500009b0 con 0x7f486008fef0 2019-12-05T06:08:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.345+0000 7f485ffff700 1 -- 172.21.15.16:0/3878656084 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4848004170 con 0x7f486008fef0 2019-12-05T06:08:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.345+0000 7f4864cb4700 1 -- 172.21.15.16:0/3878656084 >> v1:172.21.15.76:6789/0 conn(0x7f486008fef0 legacy=0x7f486008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:36.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.345+0000 7f4864cb4700 1 -- 172.21.15.16:0/3878656084 >> v1:172.21.15.16:6789/0 conn(0x7f486008c6d0 legacy=0x7f486008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.345+0000 7f4864cb4700 1 -- 172.21.15.16:0/3878656084 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486008e620 con 0x7f486008d2c0 2019-12-05T06:08:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.346+0000 7f4864cb4700 1 --2- 172.21.15.16:0/3878656084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 0x7f486008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f485400af50 tx=0x7f485400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.346+0000 7f4864cb4700 1 --2- 172.21.15.16:0/3878656084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 0x7f486008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f485400af50 tx=0x7f485400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.346+0000 7f4864cb4700 1 --2- 172.21.15.16:0/3878656084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 0x7f486008ef20 secure :-1 s=READY pgs=71 cs=0 l=1 rx=0x7f485400af50 tx=0x7f485400b760).ready entity=mon.2 client_cookie=7a9f50eef879781c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.346+0000 7f4864cb4700 1 --2- 172.21.15.16:0/3878656084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 0x7f486008ef20 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7f485400af50 tx=0x7f485400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.346+0000 7f485ffff700 1 -- 172.21.15.16:0/3878656084 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4854000960 con 0x7f486008d2c0 2019-12-05T06:08:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.346+0000 7f4864cb4700 1 --2- 172.21.15.16:0/3878656084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 0x7f486008ef20 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7f485400af50 tx=0x7f485400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.346+0000 7f4864cb4700 1 --2- 172.21.15.16:0/3878656084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 0x7f486008ef20 secure :-1 s=THROTTLE_DONE pgs=71 cs=0 l=1 rx=0x7f485400af50 tx=0x7f485400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.346+0000 7f485ffff700 1 -- 172.21.15.16:0/3878656084 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4854009860 con 0x7f486008d2c0 2019-12-05T06:08:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.346+0000 7f485ffff700 1 -- 172.21.15.16:0/3878656084 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4854009f30 con 0x7f486008d2c0 2019-12-05T06:08:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.347+0000 7f4866f18700 1 -- 172.21.15.16:0/3878656084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 msgr2=0x7f486008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.347+0000 7f4866f18700 1 --2- 172.21.15.16:0/3878656084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 0x7f486008ef20 secure :-1 s=READY pgs=71 cs=0 l=1 rx=0x7f485400af50 tx=0x7f485400b760).stop 2019-12-05T06:08:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.348+0000 7f4866f18700 1 -- 172.21.15.16:0/3878656084 shutdown_connections 2019-12-05T06:08:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.348+0000 7f4866f18700 1 --2- 172.21.15.16:0/3878656084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f486008d2c0 0x7f486008ef20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:36.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.349+0000 7f4866f18700 1 -- 172.21.15.16:0/3878656084 shutdown_connections 2019-12-05T06:08:36.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.349+0000 7f4866f18700 1 -- 172.21.15.16:0/3878656084 wait complete. 2019-12-05T06:08:36.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.349+0000 7f4866f18700 1 -- 172.21.15.16:0/3878656084 >> 172.21.15.16:0/3878656084 conn(0x7f4860061940 msgr2=0x7f4860096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.354+0000 7f4866f18700 1 Processor -- start 2019-12-05T06:08:36.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.354+0000 7f4866f18700 1 -- start start 2019-12-05T06:08:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f4866f18700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:36.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f4866f18700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f486012bdd0 0x7f4860128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f4866f18700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4860099720 con 0x7f4860137ea0 2019-12-05T06:08:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f4866f18700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4860129c10 con 0x7f486012bdd0 2019-12-05T06:08:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f4866f18700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4860129ee0 con 0x7f4860128360 2019-12-05T06:08:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f48654b5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f48654b5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:50984/0 (socket says 172.21.15.16:50984) 2019-12-05T06:08:36.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f48654b5700 1 -- 172.21.15.16:0/3796171048 learned_addr learned my addr 172.21.15.16:0/3796171048 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f4865cb6700 1 --2- 172.21.15.16:0/3796171048 >> v2:172.21.15.76:3300/0 conn(0x7f486012bdd0 0x7f4860128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f4865cb6700 1 -- 172.21.15.16:0/3796171048 >> v2:172.21.15.76:3300/0 conn(0x7f486012bdd0 msgr2=0x7f4860128c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2019-12-05T06:08:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f4865cb6700 1 -- 172.21.15.16:0/3796171048 >> v2:172.21.15.76:3300/0 conn(0x7f486012bdd0 msgr2=0x7f4860128c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:08:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.356+0000 7f4865cb6700 1 --2- 172.21.15.16:0/3796171048 >> v2:172.21.15.76:3300/0 conn(0x7f486012bdd0 0x7f4860128c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:08:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f4865cb6700 1 --2- 172.21.15.16:0/3796171048 >> v2:172.21.15.76:3300/0 conn(0x7f486012bdd0 0x7f4860128c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:08:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f48654b5700 1 -- 172.21.15.16:0/3796171048 >> v2:172.21.15.76:3300/0 conn(0x7f486012bdd0 msgr2=0x7f4860128c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:08:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> v2:172.21.15.76:3300/0 conn(0x7f486012bdd0 0x7f4860128c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f48654b5700 1 -- 172.21.15.16:0/3796171048 >> v1:172.21.15.76:6789/0 conn(0x7f4860137ea0 legacy=0x7f486012b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f48654b5700 1 -- 172.21.15.16:0/3796171048 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486012acd0 con 0x7f4860128360 2019-12-05T06:08:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=READY pgs=72 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:36.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f485dffb700 1 -- 172.21.15.16:0/3796171048 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4858001010 con 0x7f4860128360 2019-12-05T06:08:36.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.357+0000 7f485dffb700 1 -- 172.21.15.16:0/3796171048 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4858001440 con 0x7f4860128360 2019-12-05T06:08:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.358+0000 7f485dffb700 1 -- 172.21.15.16:0/3796171048 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4858001b10 con 0x7f4860128360 2019-12-05T06:08:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.358+0000 7f4866f18700 1 -- 172.21.15.16:0/3796171048 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f486012acd0 con 0x7f4860128360 2019-12-05T06:08:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.358+0000 7f4866f18700 1 -- 172.21.15.16:0/3796171048 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f486012acd0 con 0x7f4860128360 2019-12-05T06:08:36.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.358+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.359+0000 7f485dffb700 1 -- 172.21.15.16:0/3796171048 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4858001010 con 0x7f4860128360 2019-12-05T06:08:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.359+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.359+0000 7f485dffb700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4840023630 0x7f4840025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.360+0000 7f485dffb700 1 -- 172.21.15.16:0/3796171048 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 6087+0+0 (secure 0 0 0) 0x7f4858046120 con 0x7f4860128360 2019-12-05T06:08:36.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.360+0000 7f4864cb4700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4840023630 0x7f4840025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.360+0000 7f4864cb4700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4840023630 0x7f4840025ac0 crc :-1 s=READY pgs=120 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:36.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.373+0000 7f4866f18700 1 -- 172.21.15.16:0/3796171048 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486012acd0 con 0x7f4860128360 2019-12-05T06:08:36.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.383+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.383+0000 7f485dffb700 1 -- 172.21.15.16:0/3796171048 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4858007920 con 0x7f4860128360 2019-12-05T06:08:36.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.612+0000 7f4866f18700 1 -- 172.21.15.16:0/3796171048 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f486012c1f0 con 0x7f4860128360 2019-12-05T06:08:36.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.613+0000 7f48654b5700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=THROTTLE_DONE pgs=72 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:36.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.614+0000 7f485dffb700 1 -- 172.21.15.16:0/3796171048 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f4858001010 con 0x7f4860128360 2019-12-05T06:08:36.614 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2019-12-05T06:08:36.615 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:36.615 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:30.953902+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:36.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.618+0000 7f4866f18700 1 -- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4840023630 msgr2=0x7f4840025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.618+0000 7f4866f18700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4840023630 0x7f4840025ac0 crc :-1 s=READY pgs=120 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.618+0000 7f4866f18700 1 -- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 msgr2=0x7f4860128750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.618+0000 7f4866f18700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 secure :-1 s=READY pgs=72 cs=0 l=1 rx=0x7f4858010760 tx=0x7f48580105e0).stop 2019-12-05T06:08:36.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.618+0000 7f4866f18700 1 -- 172.21.15.16:0/3796171048 shutdown_connections 2019-12-05T06:08:36.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.618+0000 7f4866f18700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4840023630 0x7f4840025ac0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:36.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.618+0000 7f4866f18700 1 --2- 172.21.15.16:0/3796171048 >> v2:172.21.15.76:3300/0 conn(0x7f486012bdd0 0x7f4860128c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:36.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.618+0000 7f4866f18700 1 --2- 172.21.15.16:0/3796171048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4860128360 0x7f4860128750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:36.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.619+0000 7f4866f18700 1 -- 172.21.15.16:0/3796171048 shutdown_connections 2019-12-05T06:08:36.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.619+0000 7f4866f18700 1 -- 172.21.15.16:0/3796171048 wait complete. 2019-12-05T06:08:36.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:36.619+0000 7f4866f18700 1 -- 172.21.15.16:0/3796171048 >> 172.21.15.16:0/3796171048 conn(0x7f486006d260 msgr2=0x7f486009aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:36.631 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:08:38.631 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:38.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.870+0000 7f350955f700 1 Processor -- start 2019-12-05T06:08:38.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.870+0000 7f350955f700 1 -- start start 2019-12-05T06:08:38.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.871+0000 7f350955f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f350408d2c0 0x7f350408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:38.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.871+0000 7f350955f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f350408fef0 0x7f350408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:38.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f350955f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35040958e0 con 0x7f350408c6d0 2019-12-05T06:08:38.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f350955f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350408df90 con 0x7f350408fef0 2019-12-05T06:08:38.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f350955f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35040a3f70 con 0x7f350408d2c0 2019-12-05T06:08:38.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f3503fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f350408fef0 0x7f350408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:38.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f3503fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f350408fef0 0x7f350408f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47010/0 (socket says 172.21.15.16:47010) 2019-12-05T06:08:38.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f3503fff700 1 -- 172.21.15.16:0/748963746 learned_addr learned my addr 172.21.15.16:0/748963746 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:38.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f3502ffd700 1 --2- 172.21.15.16:0/748963746 >> v2:172.21.15.76:3300/0 conn(0x7f350408d2c0 0x7f350408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:38.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f3503fff700 1 -- 172.21.15.16:0/748963746 >> v2:172.21.15.76:3300/0 conn(0x7f350408d2c0 msgr2=0x7f350408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f3503fff700 1 --2- 172.21.15.16:0/748963746 >> v2:172.21.15.76:3300/0 conn(0x7f350408d2c0 0x7f350408ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f3503fff700 1 -- 172.21.15.16:0/748963746 >> v1:172.21.15.76:6789/0 conn(0x7f350408c6d0 legacy=0x7f350408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.872+0000 7f3503fff700 1 -- 172.21.15.16:0/748963746 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350408e620 con 0x7f350408fef0 2019-12-05T06:08:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.873+0000 7f3503fff700 1 --2- 172.21.15.16:0/748963746 >> v2:172.21.15.16:3300/0 conn(0x7f350408fef0 0x7f350408f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f34f400fa30 tx=0x7f34f400fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.873+0000 7f3503fff700 1 --2- 172.21.15.16:0/748963746 >> v2:172.21.15.16:3300/0 conn(0x7f350408fef0 0x7f350408f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f34f400fa30 tx=0x7f34f400fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.873+0000 7f3503fff700 1 --2- 172.21.15.16:0/748963746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f350408fef0 0x7f350408f460 secure :-1 s=READY pgs=232 cs=0 l=1 rx=0x7f34f400fa30 tx=0x7f34f400fdd0).ready entity=mon.0 client_cookie=b013cdf3881fe4b0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.873+0000 7f3503fff700 1 --2- 172.21.15.16:0/748963746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f350408fef0 0x7f350408f460 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f34f400fa30 tx=0x7f34f400fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.873+0000 7f3503fff700 1 --2- 172.21.15.16:0/748963746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f350408fef0 0x7f350408f460 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f34f400fa30 tx=0x7f34f400fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.873+0000 7f3503fff700 1 --2- 172.21.15.16:0/748963746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f350408fef0 0x7f350408f460 secure :-1 s=THROTTLE_DONE pgs=232 cs=0 l=1 rx=0x7f34f400fa30 tx=0x7f34f400fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.873+0000 7f35027fc700 1 -- 172.21.15.16:0/748963746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34f4000df0 con 0x7f350408fef0 2019-12-05T06:08:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.873+0000 7f35027fc700 1 -- 172.21.15.16:0/748963746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34f4001460 con 0x7f350408fef0 2019-12-05T06:08:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.873+0000 7f35027fc700 1 -- 172.21.15.16:0/748963746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34f4001b30 con 0x7f350408fef0 2019-12-05T06:08:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.874+0000 7f350955f700 1 -- 172.21.15.16:0/748963746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f350408fef0 msgr2=0x7f350408f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.874+0000 7f350955f700 1 --2- 172.21.15.16:0/748963746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f350408fef0 0x7f350408f460 secure :-1 s=READY pgs=232 cs=0 l=1 rx=0x7f34f400fa30 tx=0x7f34f400fdd0).stop 2019-12-05T06:08:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.875+0000 7f350955f700 1 -- 172.21.15.16:0/748963746 shutdown_connections 2019-12-05T06:08:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.875+0000 7f350955f700 1 --2- 172.21.15.16:0/748963746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f350408fef0 0x7f350408f460 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.875+0000 7f350955f700 1 --2- 172.21.15.16:0/748963746 >> v2:172.21.15.76:3300/0 conn(0x7f350408d2c0 0x7f350408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.875+0000 7f350955f700 1 -- 172.21.15.16:0/748963746 shutdown_connections 2019-12-05T06:08:38.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.875+0000 7f350955f700 1 -- 172.21.15.16:0/748963746 wait complete. 2019-12-05T06:08:38.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.875+0000 7f350955f700 1 -- 172.21.15.16:0/748963746 >> 172.21.15.16:0/748963746 conn(0x7f3504061940 msgr2=0x7f3504096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:38.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.879+0000 7f350955f700 1 Processor -- start 2019-12-05T06:08:38.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.880+0000 7f350955f700 1 -- start start 2019-12-05T06:08:38.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f350955f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3504132d00 0x7f350414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:38.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f350955f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3504132530 0x7f3504157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:38.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f350955f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3504132980 con 0x7f350414d400 2019-12-05T06:08:38.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f350955f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f350409bb50 con 0x7f3504132d00 2019-12-05T06:08:38.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f350955f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f350409aa60 con 0x7f3504132530 2019-12-05T06:08:38.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3502ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3504132d00 0x7f350414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:38.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3502ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3504132d00 0x7f350414d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47014/0 (socket says 172.21.15.16:47014) 2019-12-05T06:08:38.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3502ffd700 1 -- 172.21.15.16:0/209419144 learned_addr learned my addr 172.21.15.16:0/209419144 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:38.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f35037fe700 1 --1- 172.21.15.16:0/209419144 >> v1:172.21.15.16:6789/0 conn(0x7f350414d400 0x7f3504131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60412/0 (socket says 172.21.15.16:60412) 2019-12-05T06:08:38.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3503fff700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3504132530 0x7f3504157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:38.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3502ffd700 1 -- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3504132530 msgr2=0x7f3504157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:38.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3504132530 0x7f3504157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:38.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3502ffd700 1 -- 172.21.15.16:0/209419144 >> v1:172.21.15.16:6789/0 conn(0x7f350414d400 legacy=0x7f3504131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:38.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3502ffd700 1 -- 172.21.15.16:0/209419144 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f350409b0f0 con 0x7f3504132d00 2019-12-05T06:08:38.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3503fff700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3504132530 0x7f3504157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:08:38.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> v2:172.21.15.16:3300/0 conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> v2:172.21.15.16:3300/0 conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.881+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=READY pgs=234 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:38.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.882+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.882+0000 7f34fbfff700 1 -- 172.21.15.16:0/209419144 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34ec000940 con 0x7f3504132d00 2019-12-05T06:08:38.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.882+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.882+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.882+0000 7f34fbfff700 1 -- 172.21.15.16:0/209419144 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f34ec000e80 con 0x7f3504132d00 2019-12-05T06:08:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.882+0000 7f350955f700 1 -- 172.21.15.16:0/209419144 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f350409b0f0 con 0x7f3504132d00 2019-12-05T06:08:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.882+0000 7f34fbfff700 1 -- 172.21.15.16:0/209419144 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f34ec0014d0 con 0x7f3504132d00 2019-12-05T06:08:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.882+0000 7f350955f700 1 -- 172.21.15.16:0/209419144 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f350409b0f0 con 0x7f3504132d00 2019-12-05T06:08:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.883+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.883+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.883+0000 7f34fbfff700 1 -- 172.21.15.16:0/209419144 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f34ec001070 con 0x7f3504132d00 2019-12-05T06:08:38.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.883+0000 7f34fbfff700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34dc023630 0x7f34dc025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:38.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.883+0000 7f34fbfff700 1 -- 172.21.15.16:0/209419144 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6059+0+0 (secure 0 0 0) 0x7f34ec046020 con 0x7f3504132d00 2019-12-05T06:08:38.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.884+0000 7f35037fe700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34dc023630 0x7f34dc025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.884+0000 7f35037fe700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34dc023630 0x7f34dc025ae0 crc :-1 s=READY pgs=121 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.885+0000 7f350955f700 1 -- 172.21.15.16:0/209419144 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f350409b0f0 con 0x7f3504132d00 2019-12-05T06:08:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.893+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:38.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:38.893+0000 7f34fbfff700 1 -- 172.21.15.16:0/209419144 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f34ec0092b0 con 0x7f3504132d00 2019-12-05T06:08:39.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.099+0000 7f350955f700 1 -- 172.21.15.16:0/209419144 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f350409b0f0 con 0x7f3504132d00 2019-12-05T06:08:39.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.100+0000 7f3502ffd700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=THROTTLE_DONE pgs=234 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:39.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.100+0000 7f34fbfff700 1 -- 172.21.15.16:0/209419144 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2104 (secure 0 0 0) 0x7f34ec000e80 con 0x7f3504132d00 2019-12-05T06:08:39.099 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2019-12-05T06:08:39.099 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:39.100 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:30.953902+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:39.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.103+0000 7f350955f700 1 -- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34dc023630 msgr2=0x7f34dc025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:39.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.103+0000 7f350955f700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34dc023630 0x7f34dc025ae0 crc :-1 s=READY pgs=121 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:39.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.103+0000 7f350955f700 1 -- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 msgr2=0x7f350414d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:39.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.103+0000 7f350955f700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 0x7f350414d7f0 secure :-1 s=READY pgs=234 cs=0 l=1 rx=0x7f34ec010450 tx=0x7f34ec0107f0).stop 2019-12-05T06:08:39.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.103+0000 7f350955f700 1 -- 172.21.15.16:0/209419144 shutdown_connections 2019-12-05T06:08:39.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.103+0000 7f350955f700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34dc023630 0x7f34dc025ae0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:39.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.103+0000 7f350955f700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3504132d00 0x7f350414d7f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:39.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.103+0000 7f350955f700 1 --2- 172.21.15.16:0/209419144 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3504132530 0x7f3504157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:39.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.104+0000 7f350955f700 1 -- 172.21.15.16:0/209419144 shutdown_connections 2019-12-05T06:08:39.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.104+0000 7f350955f700 1 -- 172.21.15.16:0/209419144 wait complete. 2019-12-05T06:08:39.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:39.104+0000 7f350955f700 1 -- 172.21.15.16:0/209419144 >> 172.21.15.16:0/209419144 conn(0x7f35040714f0 msgr2=0x7f350409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:39.117 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:08:41.118 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:41.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.398+0000 7fe012ecb700 1 Processor -- start 2019-12-05T06:08:41.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.398+0000 7fe012ecb700 1 -- start start 2019-12-05T06:08:41.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.400+0000 7fe012ecb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe00c08c6d0 0x7fe00c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:41.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.400+0000 7fe012ecb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe00c08f720 0x7fe00c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:41.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.400+0000 7fe012ecb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00c08fb70 con 0x7fe00c08fef0 2019-12-05T06:08:41.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.400+0000 7fe012ecb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c08df90 con 0x7fe00c08c6d0 2019-12-05T06:08:41.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.400+0000 7fe012ecb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c0a3f70 con 0x7fe00c08f720 2019-12-05T06:08:41.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.400+0000 7fe011468700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe00c08c6d0 0x7fe00c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:41.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.400+0000 7fe011468700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe00c08c6d0 0x7fe00c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47024/0 (socket says 172.21.15.16:47024) 2019-12-05T06:08:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.400+0000 7fe011468700 1 -- 172.21.15.16:0/397712546 learned_addr learned my addr 172.21.15.16:0/397712546 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.400+0000 7fe010c67700 1 --1- 172.21.15.16:0/397712546 >> v1:172.21.15.76:6789/0 conn(0x7fe00c08fef0 0x7fe00c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53496/0 (socket says 172.21.15.16:53496) 2019-12-05T06:08:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.400+0000 7fe011c69700 1 --2- 172.21.15.16:0/397712546 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe00c08f720 0x7fe00c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.401+0000 7fe011468700 1 -- 172.21.15.16:0/397712546 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe00c08f720 msgr2=0x7fe00c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.401+0000 7fe011468700 1 --2- 172.21.15.16:0/397712546 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe00c08f720 0x7fe00c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:41.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.401+0000 7fe011468700 1 -- 172.21.15.16:0/397712546 >> v1:172.21.15.76:6789/0 conn(0x7fe00c08fef0 legacy=0x7fe00c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:41.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.401+0000 7fe011468700 1 -- 172.21.15.16:0/397712546 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c08d540 con 0x7fe00c08c6d0 2019-12-05T06:08:41.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.401+0000 7fe003fff700 1 -- 172.21.15.16:0/397712546 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699071623 0 0) 0x7fe008001400 con 0x7fe00c08fef0 2019-12-05T06:08:41.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.401+0000 7fe011468700 1 --2- 172.21.15.16:0/397712546 >> v2:172.21.15.16:3300/0 conn(0x7fe00c08c6d0 0x7fe00c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe00400f6c0 tx=0x7fe00400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.401+0000 7fe011468700 1 --2- 172.21.15.16:0/397712546 >> v2:172.21.15.16:3300/0 conn(0x7fe00c08c6d0 0x7fe00c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe00400f6c0 tx=0x7fe00400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.401+0000 7fe011468700 1 --2- 172.21.15.16:0/397712546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c08c6d0 0x7fe00c08cac0 secure :-1 s=READY pgs=235 cs=0 l=1 rx=0x7fe00400f6c0 tx=0x7fe00400fa60).ready entity=mon.0 client_cookie=955cba56202a439c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:41.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.402+0000 7fe011468700 1 --2- 172.21.15.16:0/397712546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c08c6d0 0x7fe00c08cac0 secure :-1 s=THROTTLE_DONE pgs=235 cs=0 l=1 rx=0x7fe00400f6c0 tx=0x7fe00400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.402+0000 7fe011468700 1 --2- 172.21.15.16:0/397712546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c08c6d0 0x7fe00c08cac0 secure :-1 s=THROTTLE_DONE pgs=235 cs=0 l=1 rx=0x7fe00400f6c0 tx=0x7fe00400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.402+0000 7fe003fff700 1 -- 172.21.15.16:0/397712546 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe004000960 con 0x7fe00c08c6d0 2019-12-05T06:08:41.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.402+0000 7fe011468700 1 --2- 172.21.15.16:0/397712546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c08c6d0 0x7fe00c08cac0 secure :-1 s=THROTTLE_DONE pgs=235 cs=0 l=1 rx=0x7fe00400f6c0 tx=0x7fe00400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.402+0000 7fe003fff700 1 -- 172.21.15.16:0/397712546 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe004000c90 con 0x7fe00c08c6d0 2019-12-05T06:08:41.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.402+0000 7fe003fff700 1 -- 172.21.15.16:0/397712546 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0040013a0 con 0x7fe00c08c6d0 2019-12-05T06:08:41.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.402+0000 7fe012ecb700 1 -- 172.21.15.16:0/397712546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c08c6d0 msgr2=0x7fe00c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:41.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.402+0000 7fe012ecb700 1 --2- 172.21.15.16:0/397712546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c08c6d0 0x7fe00c08cac0 secure :-1 s=READY pgs=235 cs=0 l=1 rx=0x7fe00400f6c0 tx=0x7fe00400fa60).stop 2019-12-05T06:08:41.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.404+0000 7fe012ecb700 1 -- 172.21.15.16:0/397712546 shutdown_connections 2019-12-05T06:08:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.404+0000 7fe012ecb700 1 --2- 172.21.15.16:0/397712546 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe00c08f720 0x7fe00c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.404+0000 7fe012ecb700 1 --2- 172.21.15.16:0/397712546 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c08c6d0 0x7fe00c08cac0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.404+0000 7fe012ecb700 1 -- 172.21.15.16:0/397712546 shutdown_connections 2019-12-05T06:08:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.404+0000 7fe012ecb700 1 -- 172.21.15.16:0/397712546 wait complete. 2019-12-05T06:08:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.404+0000 7fe012ecb700 1 -- 172.21.15.16:0/397712546 >> 172.21.15.16:0/397712546 conn(0x7fe00c061940 msgr2=0x7fe00c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:41.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.409+0000 7fe012ecb700 1 Processor -- start 2019-12-05T06:08:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.409+0000 7fe012ecb700 1 -- start start 2019-12-05T06:08:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.411+0000 7fe012ecb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe00c096e90 0x7fe00c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.411+0000 7fe012ecb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00c084b90 con 0x7fe00c09b920 2019-12-05T06:08:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.411+0000 7fe012ecb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe00c098790 con 0x7fe00c084f30 2019-12-05T06:08:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.411+0000 7fe012ecb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe00c09a450 con 0x7fe00c096e90 2019-12-05T06:08:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.412+0000 7fe011468700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe00c096e90 0x7fe00c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:41.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.412+0000 7fe011468700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe00c096e90 0x7fe00c09b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47030/0 (socket says 172.21.15.16:47030) 2019-12-05T06:08:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.412+0000 7fe011c69700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe00c09b920 0x7fe00c097a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60434/0 (socket says 172.21.15.16:60434) 2019-12-05T06:08:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.412+0000 7fe011468700 1 -- 172.21.15.16:0/3614881486 learned_addr learned my addr 172.21.15.16:0/3614881486 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.412+0000 7fe011468700 1 -- 172.21.15.16:0/3614881486 >> v1:172.21.15.76:6789/0 conn(0x7fe00c084f30 legacy=0x7fe00c097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.412+0000 7fe011468700 1 -- 172.21.15.16:0/3614881486 >> v1:172.21.15.16:6789/0 conn(0x7fe00c09b920 legacy=0x7fe00c097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.412+0000 7fe001ffb700 1 -- 172.21.15.16:0/3614881486 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860218249 0 0) 0x7fe008001400 con 0x7fe00c084f30 2019-12-05T06:08:41.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.412+0000 7fe011468700 1 -- 172.21.15.16:0/3614881486 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c09aae0 con 0x7fe00c096e90 2019-12-05T06:08:41.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe011468700 1 --2- 172.21.15.16:0/3614881486 >> v2:172.21.15.16:3300/0 conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe011468700 1 --2- 172.21.15.16:0/3614881486 >> v2:172.21.15.16:3300/0 conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe011468700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=READY pgs=237 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:41.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe011468700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe001ffb700 1 -- 172.21.15.16:0/3614881486 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe004007a90 con 0x7fe00c096e90 2019-12-05T06:08:41.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe011468700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe011468700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe001ffb700 1 -- 172.21.15.16:0/3614881486 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe004005370 con 0x7fe00c096e90 2019-12-05T06:08:41.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe001ffb700 1 -- 172.21.15.16:0/3614881486 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe004005a00 con 0x7fe00c096e90 2019-12-05T06:08:41.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe012ecb700 1 -- 172.21.15.16:0/3614881486 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe00c09aae0 con 0x7fe00c096e90 2019-12-05T06:08:41.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.413+0000 7fe012ecb700 1 -- 172.21.15.16:0/3614881486 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe00c09aae0 con 0x7fe00c096e90 2019-12-05T06:08:41.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.414+0000 7fe011468700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.415+0000 7fe001ffb700 1 -- 172.21.15.16:0/3614881486 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe004005560 con 0x7fe00c096e90 2019-12-05T06:08:41.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.415+0000 7fe011468700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.416+0000 7fe001ffb700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfec023630 0x7fdfec025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:41.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.416+0000 7fe001ffb700 1 -- 172.21.15.16:0/3614881486 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6059+0+0 (secure 0 0 0) 0x7fe004045f10 con 0x7fe00c096e90 2019-12-05T06:08:41.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.416+0000 7fe010c67700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfec023630 0x7fdfec025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:41.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.416+0000 7fe010c67700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfec023630 0x7fdfec025ac0 crc :-1 s=READY pgs=122 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:41.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.429+0000 7fe012ecb700 1 -- 172.21.15.16:0/3614881486 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe00c09aae0 con 0x7fe00c096e90 2019-12-05T06:08:41.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.438+0000 7fe011468700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.438+0000 7fe001ffb700 1 -- 172.21.15.16:0/3614881486 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe0040072c0 con 0x7fe00c096e90 2019-12-05T06:08:41.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.680+0000 7fe012ecb700 1 -- 172.21.15.16:0/3614881486 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe00c09aae0 con 0x7fe00c096e90 2019-12-05T06:08:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.681+0000 7fe011468700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.681+0000 7fe001ffb700 1 -- 172.21.15.16:0/3614881486 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2104 (secure 0 0 0) 0x7fe004005560 con 0x7fe00c096e90 2019-12-05T06:08:41.680 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2019-12-05T06:08:41.680 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:41.680 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:30.953902+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:41.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.683+0000 7fe012ecb700 1 -- 172.21.15.16:0/3614881486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfec023630 msgr2=0x7fdfec025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.684+0000 7fe012ecb700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfec023630 0x7fdfec025ac0 crc :-1 s=READY pgs=122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.684+0000 7fe012ecb700 1 -- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 msgr2=0x7fe00c09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.684+0000 7fe012ecb700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 0x7fe00c09b3e0 secure :-1 s=READY pgs=237 cs=0 l=1 rx=0x7fe004010440 tx=0x7fe0040107e0).stop 2019-12-05T06:08:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.684+0000 7fe012ecb700 1 -- 172.21.15.16:0/3614881486 shutdown_connections 2019-12-05T06:08:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.684+0000 7fe012ecb700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfec023630 0x7fdfec025ac0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:41.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.684+0000 7fe012ecb700 1 --2- 172.21.15.16:0/3614881486 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe00c096e90 0x7fe00c09b3e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.684+0000 7fe012ecb700 1 -- 172.21.15.16:0/3614881486 shutdown_connections 2019-12-05T06:08:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.684+0000 7fe012ecb700 1 -- 172.21.15.16:0/3614881486 wait complete. 2019-12-05T06:08:41.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:41.684+0000 7fe012ecb700 1 -- 172.21.15.16:0/3614881486 >> 172.21.15.16:0/3614881486 conn(0x7fe00c0714f0 msgr2=0x7fe00c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:41.718 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:08:43.719 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.994+0000 7ff86501c700 1 Processor -- start 2019-12-05T06:08:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.994+0000 7ff86501c700 1 -- start start 2019-12-05T06:08:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.995+0000 7ff86501c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff86008d2c0 0x7ff86008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:44.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.996+0000 7ff86501c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8600958e0 con 0x7ff86008fef0 2019-12-05T06:08:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.996+0000 7ff86501c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8600629e0 con 0x7ff86008c6d0 2019-12-05T06:08:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.996+0000 7ff86501c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86008df90 con 0x7ff86008d2c0 2019-12-05T06:08:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.996+0000 7ff85ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff86008d2c0 0x7ff86008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.996+0000 7ff85ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff86008d2c0 0x7ff86008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47040/0 (socket says 172.21.15.16:47040) 2019-12-05T06:08:44.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.996+0000 7ff85ed9d700 1 -- 172.21.15.16:0/4267533882 learned_addr learned my addr 172.21.15.16:0/4267533882 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.996+0000 7ff86481b700 1 --1- 172.21.15.16:0/4267533882 >> v1:172.21.15.16:6789/0 conn(0x7ff86008fef0 0x7ff86008f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60444/0 (socket says 172.21.15.16:60444) 2019-12-05T06:08:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.996+0000 7ff85f59e700 1 --1- 172.21.15.16:0/4267533882 >> v1:172.21.15.76:6789/0 conn(0x7ff86008c6d0 0x7ff86008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53510/0 (socket says 172.21.15.16:53510) 2019-12-05T06:08:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.996+0000 7ff85ed9d700 1 -- 172.21.15.16:0/4267533882 >> v1:172.21.15.76:6789/0 conn(0x7ff86008c6d0 legacy=0x7ff86008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.996+0000 7ff85ed9d700 1 -- 172.21.15.16:0/4267533882 >> v1:172.21.15.16:6789/0 conn(0x7ff86008fef0 legacy=0x7ff86008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff85ed9d700 1 -- 172.21.15.16:0/4267533882 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff86008e620 con 0x7ff86008d2c0 2019-12-05T06:08:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff85ed9d700 1 --2- 172.21.15.16:0/4267533882 >> v2:172.21.15.16:3300/0 conn(0x7ff86008d2c0 0x7ff86008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff85000f740 tx=0x7ff85000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff85ed9d700 1 --2- 172.21.15.16:0/4267533882 >> v2:172.21.15.16:3300/0 conn(0x7ff86008d2c0 0x7ff86008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff85000f740 tx=0x7ff85000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff85ed9d700 1 --2- 172.21.15.16:0/4267533882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff86008d2c0 0x7ff86008ef20 secure :-1 s=READY pgs=239 cs=0 l=1 rx=0x7ff85000f740 tx=0x7ff85000fae0).ready entity=mon.0 client_cookie=4ab429e449851b21 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff85ed9d700 1 --2- 172.21.15.16:0/4267533882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff86008d2c0 0x7ff86008ef20 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7ff85000f740 tx=0x7ff85000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff85e59c700 1 -- 172.21.15.16:0/4267533882 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff850000960 con 0x7ff86008d2c0 2019-12-05T06:08:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff85ed9d700 1 --2- 172.21.15.16:0/4267533882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff86008d2c0 0x7ff86008ef20 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7ff85000f740 tx=0x7ff85000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff85ed9d700 1 --2- 172.21.15.16:0/4267533882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff86008d2c0 0x7ff86008ef20 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7ff85000f740 tx=0x7ff85000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff85e59c700 1 -- 172.21.15.16:0/4267533882 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff850000c90 con 0x7ff86008d2c0 2019-12-05T06:08:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff85e59c700 1 -- 172.21.15.16:0/4267533882 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff850001360 con 0x7ff86008d2c0 2019-12-05T06:08:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff86501c700 1 -- 172.21.15.16:0/4267533882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff86008d2c0 msgr2=0x7ff86008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.997+0000 7ff86501c700 1 --2- 172.21.15.16:0/4267533882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff86008d2c0 0x7ff86008ef20 secure :-1 s=READY pgs=239 cs=0 l=1 rx=0x7ff85000f740 tx=0x7ff85000fae0).stop 2019-12-05T06:08:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.999+0000 7ff86501c700 1 -- 172.21.15.16:0/4267533882 shutdown_connections 2019-12-05T06:08:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.999+0000 7ff86501c700 1 --2- 172.21.15.16:0/4267533882 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff86008d2c0 0x7ff86008ef20 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.999+0000 7ff86501c700 1 -- 172.21.15.16:0/4267533882 shutdown_connections 2019-12-05T06:08:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.999+0000 7ff86501c700 1 -- 172.21.15.16:0/4267533882 wait complete. 2019-12-05T06:08:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:43.999+0000 7ff86501c700 1 -- 172.21.15.16:0/4267533882 >> 172.21.15.16:0/4267533882 conn(0x7ff860061940 msgr2=0x7ff860096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.003+0000 7ff86501c700 1 Processor -- start 2019-12-05T06:08:44.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.003+0000 7ff86501c700 1 -- start start 2019-12-05T06:08:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.004+0000 7ff86501c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.004+0000 7ff86501c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff860152c20 0x7ff8601530a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.005+0000 7ff86501c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff860084ed0 con 0x7ff860088b50 2019-12-05T06:08:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.005+0000 7ff86501c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff86014a7b0 con 0x7ff860152c20 2019-12-05T06:08:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.005+0000 7ff86501c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8600861c0 con 0x7ff86009e440 2019-12-05T06:08:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.005+0000 7ff85ed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff860088b50 0x7ff8600880c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53516/0 (socket says 172.21.15.16:53516) 2019-12-05T06:08:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.005+0000 7ff85ed9d700 1 -- 172.21.15.16:0/1187023912 learned_addr learned my addr 172.21.15.16:0/1187023912 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.005+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.005+0000 7ff86481b700 1 --2- 172.21.15.16:0/1187023912 >> v2:172.21.15.76:3300/0 conn(0x7ff860152c20 0x7ff8601530a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.005+0000 7ff857fff700 1 -- 172.21.15.16:0/1187023912 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215541118 0 0) 0x7ff8500078e0 con 0x7ff860088b50 2019-12-05T06:08:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.005+0000 7ff857fff700 1 -- 172.21.15.16:0/1187023912 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff844002e80 con 0x7ff860088b50 2019-12-05T06:08:44.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.006+0000 7ff85f59e700 1 -- 172.21.15.16:0/1187023912 >> v2:172.21.15.76:3300/0 conn(0x7ff860152c20 msgr2=0x7ff8601530a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:44.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.006+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> v2:172.21.15.76:3300/0 conn(0x7ff860152c20 0x7ff8601530a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:44.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.006+0000 7ff85f59e700 1 -- 172.21.15.16:0/1187023912 >> v1:172.21.15.76:6789/0 conn(0x7ff860088b50 legacy=0x7ff8600880c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:44.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.006+0000 7ff85f59e700 1 -- 172.21.15.16:0/1187023912 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff860087280 con 0x7ff86009e440 2019-12-05T06:08:44.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.006+0000 7ff86481b700 1 --2- 172.21.15.16:0/1187023912 >> v2:172.21.15.76:3300/0 conn(0x7ff860152c20 0x7ff8601530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:44.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.006+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.006+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.006+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=READY pgs=73 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.006+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.007+0000 7ff857fff700 1 -- 172.21.15.16:0/1187023912 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff848000e50 con 0x7ff86009e440 2019-12-05T06:08:44.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.007+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.007+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.007+0000 7ff857fff700 1 -- 172.21.15.16:0/1187023912 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8480041c0 con 0x7ff86009e440 2019-12-05T06:08:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.007+0000 7ff86501c700 1 -- 172.21.15.16:0/1187023912 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff860087280 con 0x7ff86009e440 2019-12-05T06:08:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.007+0000 7ff857fff700 1 -- 172.21.15.16:0/1187023912 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff848004850 con 0x7ff86009e440 2019-12-05T06:08:44.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.007+0000 7ff86501c700 1 -- 172.21.15.16:0/1187023912 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff860087280 con 0x7ff86009e440 2019-12-05T06:08:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.008+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.008+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.008+0000 7ff857fff700 1 -- 172.21.15.16:0/1187023912 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff8480043b0 con 0x7ff86009e440 2019-12-05T06:08:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.009+0000 7ff857fff700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff844023670 0x7ff844025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.009+0000 7ff857fff700 1 -- 172.21.15.16:0/1187023912 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6059+0+0 (secure 0 0 0) 0x7ff848045d20 con 0x7ff86009e440 2019-12-05T06:08:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.009+0000 7ff85ed9d700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff844023670 0x7ff844025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.010+0000 7ff85ed9d700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff844023670 0x7ff844025ae0 crc :-1 s=READY pgs=123 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.025+0000 7ff86501c700 1 -- 172.21.15.16:0/1187023912 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff860087280 con 0x7ff86009e440 2019-12-05T06:08:44.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.033+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.033+0000 7ff857fff700 1 -- 172.21.15.16:0/1187023912 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff848000c10 con 0x7ff86009e440 2019-12-05T06:08:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.297+0000 7ff86501c700 1 -- 172.21.15.16:0/1187023912 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff860087280 con 0x7ff86009e440 2019-12-05T06:08:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.299+0000 7ff85f59e700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=THROTTLE_DONE pgs=73 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:44.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.299+0000 7ff857fff700 1 -- 172.21.15.16:0/1187023912 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2104 (secure 0 0 0) 0x7ff8480394b0 con 0x7ff86009e440 2019-12-05T06:08:44.299 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:44.299 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:30.953902+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:44.299 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2019-12-05T06:08:44.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.302+0000 7ff86501c700 1 -- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff844023670 msgr2=0x7ff844025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:44.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.302+0000 7ff86501c700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff844023670 0x7ff844025ae0 crc :-1 s=READY pgs=123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.302+0000 7ff86501c700 1 -- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 msgr2=0x7ff860087b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.302+0000 7ff86501c700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 secure :-1 s=READY pgs=73 cs=0 l=1 rx=0x7ff848010370 tx=0x7ff848010710).stop 2019-12-05T06:08:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.302+0000 7ff86501c700 1 -- 172.21.15.16:0/1187023912 shutdown_connections 2019-12-05T06:08:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.302+0000 7ff86501c700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff844023670 0x7ff844025ae0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.302+0000 7ff86501c700 1 --2- 172.21.15.16:0/1187023912 >> v2:172.21.15.76:3300/0 conn(0x7ff860152c20 0x7ff8601530a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.302+0000 7ff86501c700 1 --2- 172.21.15.16:0/1187023912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff86009e440 0x7ff860087b80 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:44.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.303+0000 7ff86501c700 1 -- 172.21.15.16:0/1187023912 shutdown_connections 2019-12-05T06:08:44.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.303+0000 7ff86501c700 1 -- 172.21.15.16:0/1187023912 wait complete. 2019-12-05T06:08:44.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:44.303+0000 7ff86501c700 1 -- 172.21.15.16:0/1187023912 >> 172.21.15.16:0/1187023912 conn(0x7ff860098480 msgr2=0x7ff860071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:44.312 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:08:46.315 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.575+0000 7ff444ba7700 1 Processor -- start 2019-12-05T06:08:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.576+0000 7ff444ba7700 1 -- start start 2019-12-05T06:08:46.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.577+0000 7ff444ba7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff44008c6d0 0x7ff44008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.577+0000 7ff444ba7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff44008fef0 0x7ff44008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.577+0000 7ff444ba7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4400958e0 con 0x7ff44008ef20 2019-12-05T06:08:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.577+0000 7ff444ba7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4400a3f70 con 0x7ff44008fef0 2019-12-05T06:08:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.577+0000 7ff444ba7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff44008d830 con 0x7ff44008c6d0 2019-12-05T06:08:46.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.578+0000 7ff43e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff44008fef0 0x7ff44008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:46.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.578+0000 7ff43f59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff44008ef20 0x7ff44008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60458/0 (socket says 172.21.15.16:60458) 2019-12-05T06:08:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.578+0000 7ff43e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff44008fef0 0x7ff44008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47056/0 (socket says 172.21.15.16:47056) 2019-12-05T06:08:46.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.578+0000 7ff43f59e700 1 -- 172.21.15.16:0/718707195 learned_addr learned my addr 172.21.15.16:0/718707195 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.578+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/718707195 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff44008c6d0 0x7ff44008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.578+0000 7ff43e59c700 1 -- 172.21.15.16:0/718707195 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff44008c6d0 msgr2=0x7ff44008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.578+0000 7ff43e59c700 1 --2- 172.21.15.16:0/718707195 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff44008c6d0 0x7ff44008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.578+0000 7ff43e59c700 1 -- 172.21.15.16:0/718707195 >> v1:172.21.15.16:6789/0 conn(0x7ff44008ef20 legacy=0x7ff44008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.578+0000 7ff43dd9b700 1 -- 172.21.15.16:0/718707195 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1765967821 0 0) 0x7ff430001150 con 0x7ff44008ef20 2019-12-05T06:08:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.578+0000 7ff43e59c700 1 -- 172.21.15.16:0/718707195 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44008d540 con 0x7ff44008fef0 2019-12-05T06:08:46.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff43e59c700 1 --2- 172.21.15.16:0/718707195 >> v2:172.21.15.16:3300/0 conn(0x7ff44008fef0 0x7ff44008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff42800af50 tx=0x7ff42800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff43e59c700 1 --2- 172.21.15.16:0/718707195 >> v2:172.21.15.16:3300/0 conn(0x7ff44008fef0 0x7ff44008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff42800af50 tx=0x7ff42800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff43e59c700 1 --2- 172.21.15.16:0/718707195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff44008fef0 0x7ff44008d000 secure :-1 s=READY pgs=241 cs=0 l=1 rx=0x7ff42800af50 tx=0x7ff42800b6e0).ready entity=mon.0 client_cookie=879da9646143798e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff43e59c700 1 --2- 172.21.15.16:0/718707195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff44008fef0 0x7ff44008d000 secure :-1 s=THROTTLE_DONE pgs=241 cs=0 l=1 rx=0x7ff42800af50 tx=0x7ff42800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff43e59c700 1 --2- 172.21.15.16:0/718707195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff44008fef0 0x7ff44008d000 secure :-1 s=THROTTLE_DONE pgs=241 cs=0 l=1 rx=0x7ff42800af50 tx=0x7ff42800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff43dd9b700 1 -- 172.21.15.16:0/718707195 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff428007210 con 0x7ff44008fef0 2019-12-05T06:08:46.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff43e59c700 1 --2- 172.21.15.16:0/718707195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff44008fef0 0x7ff44008d000 secure :-1 s=THROTTLE_DONE pgs=241 cs=0 l=1 rx=0x7ff42800af50 tx=0x7ff42800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff43dd9b700 1 -- 172.21.15.16:0/718707195 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4280099b0 con 0x7ff44008fef0 2019-12-05T06:08:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff43dd9b700 1 -- 172.21.15.16:0/718707195 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff42800a080 con 0x7ff44008fef0 2019-12-05T06:08:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff444ba7700 1 -- 172.21.15.16:0/718707195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff44008fef0 msgr2=0x7ff44008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.579+0000 7ff444ba7700 1 --2- 172.21.15.16:0/718707195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff44008fef0 0x7ff44008d000 secure :-1 s=READY pgs=241 cs=0 l=1 rx=0x7ff42800af50 tx=0x7ff42800b6e0).stop 2019-12-05T06:08:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.581+0000 7ff444ba7700 1 -- 172.21.15.16:0/718707195 shutdown_connections 2019-12-05T06:08:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.581+0000 7ff444ba7700 1 --2- 172.21.15.16:0/718707195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff44008fef0 0x7ff44008d000 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.581+0000 7ff444ba7700 1 --2- 172.21.15.16:0/718707195 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff44008c6d0 0x7ff44008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.581+0000 7ff444ba7700 1 -- 172.21.15.16:0/718707195 shutdown_connections 2019-12-05T06:08:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.581+0000 7ff444ba7700 1 -- 172.21.15.16:0/718707195 wait complete. 2019-12-05T06:08:46.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.581+0000 7ff444ba7700 1 -- 172.21.15.16:0/718707195 >> 172.21.15.16:0/718707195 conn(0x7ff440061940 msgr2=0x7ff440096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.586+0000 7ff444ba7700 1 Processor -- start 2019-12-05T06:08:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.586+0000 7ff444ba7700 1 -- start start 2019-12-05T06:08:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.588+0000 7ff444ba7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff44014d400 0x7ff44014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:46.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.588+0000 7ff444ba7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff440098da0 con 0x7ff440132530 2019-12-05T06:08:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.588+0000 7ff444ba7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff44009f0a0 con 0x7ff440132d00 2019-12-05T06:08:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.588+0000 7ff444ba7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff44009bb50 con 0x7ff44014d400 2019-12-05T06:08:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.588+0000 7ff43f59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff440132530 0x7ff440157100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60464/0 (socket says 172.21.15.16:60464) 2019-12-05T06:08:46.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.588+0000 7ff43f59e700 1 -- 172.21.15.16:0/2856593551 learned_addr learned my addr 172.21.15.16:0/2856593551 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.588+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> v2:172.21.15.76:3300/0 conn(0x7ff44014d400 0x7ff44014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.589+0000 7ff4377fe700 1 -- 172.21.15.16:0/2856593551 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1218935761 0 0) 0x7ff430000f40 con 0x7ff440132530 2019-12-05T06:08:46.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.589+0000 7ff4377fe700 1 -- 172.21.15.16:0/2856593551 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff424002e80 con 0x7ff440132530 2019-12-05T06:08:46.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.589+0000 7ff43ed9d700 1 -- 172.21.15.16:0/2856593551 >> v1:172.21.15.76:6789/0 conn(0x7ff440132d00 legacy=0x7ff440131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:46.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.589+0000 7ff43ed9d700 1 -- 172.21.15.16:0/2856593551 >> v1:172.21.15.16:6789/0 conn(0x7ff440132530 legacy=0x7ff440157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:46.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.589+0000 7ff43ed9d700 1 -- 172.21.15.16:0/2856593551 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44009b0f0 con 0x7ff44014d400 2019-12-05T06:08:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.589+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> v2:172.21.15.76:3300/0 conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.589+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> v2:172.21.15.76:3300/0 conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.589+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=READY pgs=208 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.590+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=THROTTLE_DONE pgs=208 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.590+0000 7ff4377fe700 1 -- 172.21.15.16:0/2856593551 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff438000cc0 con 0x7ff44014d400 2019-12-05T06:08:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.590+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=THROTTLE_DONE pgs=208 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.590+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=THROTTLE_DONE pgs=208 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.590+0000 7ff4377fe700 1 -- 172.21.15.16:0/2856593551 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff438000cc0 con 0x7ff44014d400 2019-12-05T06:08:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.590+0000 7ff444ba7700 1 -- 172.21.15.16:0/2856593551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff44009b0f0 con 0x7ff44014d400 2019-12-05T06:08:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.590+0000 7ff4377fe700 1 -- 172.21.15.16:0/2856593551 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff438005af0 con 0x7ff44014d400 2019-12-05T06:08:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.590+0000 7ff444ba7700 1 -- 172.21.15.16:0/2856593551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff44009b0f0 con 0x7ff44014d400 2019-12-05T06:08:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.591+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=THROTTLE_DONE pgs=208 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.592+0000 7ff4377fe700 1 -- 172.21.15.16:0/2856593551 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff4380061d0 con 0x7ff44014d400 2019-12-05T06:08:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.592+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=THROTTLE_DONE pgs=208 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.592+0000 7ff4377fe700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff424023670 0x7ff424025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.592+0000 7ff4377fe700 1 -- 172.21.15.16:0/2856593551 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(26..26 src has 1..26) v4 ==== 6059+0+0 (secure 0 0 0) 0x7ff438046390 con 0x7ff44014d400 2019-12-05T06:08:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.592+0000 7ff43e59c700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff424023670 0x7ff424025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.593+0000 7ff43e59c700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff424023670 0x7ff424025ae0 crc :-1 s=READY pgs=124 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.607+0000 7ff444ba7700 1 -- 172.21.15.16:0/2856593551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff44009b0f0 con 0x7ff44014d400 2019-12-05T06:08:46.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.616+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=THROTTLE_DONE pgs=208 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.616+0000 7ff4377fe700 1 -- 172.21.15.16:0/2856593551 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff438005fc0 con 0x7ff44014d400 2019-12-05T06:08:46.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.837+0000 7ff444ba7700 1 -- 172.21.15.16:0/2856593551 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff44009b0f0 con 0x7ff44014d400 2019-12-05T06:08:46.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.838+0000 7ff43ed9d700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=THROTTLE_DONE pgs=208 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:46.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.838+0000 7ff4377fe700 1 -- 172.21.15.16:0/2856593551 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+2104 (secure 0 0 0) 0x7ff438000cc0 con 0x7ff44014d400 2019-12-05T06:08:46.839 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:46.839 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":13,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4229,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":7}],"filesystems":[{"mdsmap":{"epoch":13,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:30.953902+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4341},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4341":{"gid":4341,"name":"a","rank":0,"incarnation":5,"state":"up:active","state_seq":2,"addr":"172.21.15.16:6833/3750563081","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3750563081},{"type":"v1","addr":"172.21.15.16:6833","nonce":3750563081}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 13 2019-12-05T06:08:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.842+0000 7ff444ba7700 1 -- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff424023670 msgr2=0x7ff424025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.842+0000 7ff444ba7700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff424023670 0x7ff424025ae0 crc :-1 s=READY pgs=124 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.842+0000 7ff444ba7700 1 -- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 msgr2=0x7ff44014d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.842+0000 7ff444ba7700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 0x7ff44014d7f0 secure :-1 s=READY pgs=208 cs=0 l=1 rx=0x7ff438010730 tx=0x7ff438010ad0).stop 2019-12-05T06:08:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.842+0000 7ff444ba7700 1 -- 172.21.15.16:0/2856593551 shutdown_connections 2019-12-05T06:08:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.842+0000 7ff444ba7700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff424023670 0x7ff424025ae0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:46.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.842+0000 7ff444ba7700 1 --2- 172.21.15.16:0/2856593551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff44014d400 0x7ff44014d7f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.842+0000 7ff444ba7700 1 -- 172.21.15.16:0/2856593551 shutdown_connections 2019-12-05T06:08:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.842+0000 7ff444ba7700 1 -- 172.21.15.16:0/2856593551 wait complete. 2019-12-05T06:08:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:46.843+0000 7ff444ba7700 1 -- 172.21.15.16:0/2856593551 >> 172.21.15.16:0/2856593551 conn(0x7ff4400714f0 msgr2=0x7ff44009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:46.852 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:08:48.853 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.113+0000 7fae0642f700 1 Processor -- start 2019-12-05T06:08:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.113+0000 7fae0642f700 1 -- start start 2019-12-05T06:08:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.114+0000 7fae0642f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae0008c6d0 0x7fae0008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.114+0000 7fae0642f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fae0008f720 0x7fae0008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.114+0000 7fae0642f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0008fb80 con 0x7fae0008fef0 2019-12-05T06:08:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.114+0000 7fae0642f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae0008df90 con 0x7fae0008c6d0 2019-12-05T06:08:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.114+0000 7fae0642f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae000a3f70 con 0x7fae0008f720 2019-12-05T06:08:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.114+0000 7fae049cc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae0008c6d0 0x7fae0008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.114+0000 7fadfffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fae0008fef0 0x7fae0008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60472/0 (socket says 172.21.15.16:60472) 2019-12-05T06:08:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.115+0000 7fadfffff700 1 -- 172.21.15.16:0/460795550 learned_addr learned my addr 172.21.15.16:0/460795550 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.115+0000 7fae051cd700 1 --2- 172.21.15.16:0/460795550 >> v2:172.21.15.76:3300/0 conn(0x7fae0008f720 0x7fae0008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.115+0000 7fae049cc700 1 -- 172.21.15.16:0/460795550 >> v2:172.21.15.76:3300/0 conn(0x7fae0008f720 msgr2=0x7fae0008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.115+0000 7fae049cc700 1 --2- 172.21.15.16:0/460795550 >> v2:172.21.15.76:3300/0 conn(0x7fae0008f720 0x7fae0008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.115+0000 7fae049cc700 1 -- 172.21.15.16:0/460795550 >> v1:172.21.15.16:6789/0 conn(0x7fae0008fef0 legacy=0x7fae0008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.115+0000 7fadff7fe700 1 -- 172.21.15.16:0/460795550 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147232422 0 0) 0x7fadf0001150 con 0x7fae0008fef0 2019-12-05T06:08:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.115+0000 7fae049cc700 1 -- 172.21.15.16:0/460795550 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0008d540 con 0x7fae0008c6d0 2019-12-05T06:08:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.115+0000 7fae051cd700 1 --2- 172.21.15.16:0/460795550 >> v2:172.21.15.76:3300/0 conn(0x7fae0008f720 0x7fae0008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.115+0000 7fae049cc700 1 --2- 172.21.15.16:0/460795550 >> v2:172.21.15.16:3300/0 conn(0x7fae0008c6d0 0x7fae0008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fade800af50 tx=0x7fade800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.116+0000 7fae049cc700 1 --2- 172.21.15.16:0/460795550 >> v2:172.21.15.16:3300/0 conn(0x7fae0008c6d0 0x7fae0008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fade800af50 tx=0x7fade800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.116+0000 7fae049cc700 1 --2- 172.21.15.16:0/460795550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae0008c6d0 0x7fae0008cac0 secure :-1 s=READY pgs=244 cs=0 l=1 rx=0x7fade800af50 tx=0x7fade800b760).ready entity=mon.0 client_cookie=4535b2857be138ae server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.116+0000 7fae049cc700 1 --2- 172.21.15.16:0/460795550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae0008c6d0 0x7fae0008cac0 secure :-1 s=THROTTLE_DONE pgs=244 cs=0 l=1 rx=0x7fade800af50 tx=0x7fade800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.116+0000 7fadff7fe700 1 -- 172.21.15.16:0/460795550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fade8007210 con 0x7fae0008c6d0 2019-12-05T06:08:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.116+0000 7fae049cc700 1 --2- 172.21.15.16:0/460795550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae0008c6d0 0x7fae0008cac0 secure :-1 s=THROTTLE_DONE pgs=244 cs=0 l=1 rx=0x7fade800af50 tx=0x7fade800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.116+0000 7fae049cc700 1 --2- 172.21.15.16:0/460795550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae0008c6d0 0x7fae0008cac0 secure :-1 s=THROTTLE_DONE pgs=244 cs=0 l=1 rx=0x7fade800af50 tx=0x7fade800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.116+0000 7fadff7fe700 1 -- 172.21.15.16:0/460795550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fade80099b0 con 0x7fae0008c6d0 2019-12-05T06:08:49.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.116+0000 7fadff7fe700 1 -- 172.21.15.16:0/460795550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fade8009c60 con 0x7fae0008c6d0 2019-12-05T06:08:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.116+0000 7fae0642f700 1 -- 172.21.15.16:0/460795550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae0008c6d0 msgr2=0x7fae0008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.116+0000 7fae0642f700 1 --2- 172.21.15.16:0/460795550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae0008c6d0 0x7fae0008cac0 secure :-1 s=READY pgs=244 cs=0 l=1 rx=0x7fade800af50 tx=0x7fade800b760).stop 2019-12-05T06:08:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.118+0000 7fae0642f700 1 -- 172.21.15.16:0/460795550 shutdown_connections 2019-12-05T06:08:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.118+0000 7fae0642f700 1 --2- 172.21.15.16:0/460795550 >> v2:172.21.15.76:3300/0 conn(0x7fae0008f720 0x7fae0008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.118+0000 7fae0642f700 1 --2- 172.21.15.16:0/460795550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae0008c6d0 0x7fae0008cac0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.118+0000 7fae0642f700 1 -- 172.21.15.16:0/460795550 shutdown_connections 2019-12-05T06:08:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.118+0000 7fae0642f700 1 -- 172.21.15.16:0/460795550 wait complete. 2019-12-05T06:08:49.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.118+0000 7fae0642f700 1 -- 172.21.15.16:0/460795550 >> 172.21.15.16:0/460795550 conn(0x7fae00061940 msgr2=0x7fae00096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.123+0000 7fae0642f700 1 Processor -- start 2019-12-05T06:08:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.123+0000 7fae0642f700 1 -- start start 2019-12-05T06:08:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.125+0000 7fae0642f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae0009bbe0 0x7fae001539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.125+0000 7fae0642f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fae00084f30 0x7fae00155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.125+0000 7fae0642f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae00084b90 con 0x7fae00096e90 2019-12-05T06:08:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.125+0000 7fae0642f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae0009a450 con 0x7fae00084f30 2019-12-05T06:08:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.125+0000 7fae0642f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae00099a20 con 0x7fae0009bbe0 2019-12-05T06:08:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.125+0000 7fae051cd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fae00084f30 0x7fae00155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.125+0000 7fadfffff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae0009bbe0 0x7fae001539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.125+0000 7fae051cd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fae00084f30 0x7fae00155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:33170/0 (socket says 172.21.15.16:33170) 2019-12-05T06:08:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.125+0000 7fae051cd700 1 -- 172.21.15.16:0/2457727231 learned_addr learned my addr 172.21.15.16:0/2457727231 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.126+0000 7fae051cd700 1 -- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae0009bbe0 msgr2=0x7fae001539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.126+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae0009bbe0 0x7fae001539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.126+0000 7fadfd7fa700 1 -- 172.21.15.16:0/2457727231 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2786377565 0 0) 0x7fade800b250 con 0x7fae00096e90 2019-12-05T06:08:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.126+0000 7fae051cd700 1 -- 172.21.15.16:0/2457727231 >> v1:172.21.15.76:6789/0 conn(0x7fae00096e90 legacy=0x7fae0009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.126+0000 7fae051cd700 1 -- 172.21.15.16:0/2457727231 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0009aae0 con 0x7fae00084f30 2019-12-05T06:08:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.126+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> v2:172.21.15.76:3300/0 conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.126+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> v2:172.21.15.76:3300/0 conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.126+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=READY pgs=212 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.127+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.127+0000 7fadfd7fa700 1 -- 172.21.15.16:0/2457727231 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadf800add0 con 0x7fae00084f30 2019-12-05T06:08:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.127+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.127+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.127+0000 7fadfd7fa700 1 -- 172.21.15.16:0/2457727231 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadf8009b20 con 0x7fae00084f30 2019-12-05T06:08:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.127+0000 7fae0642f700 1 -- 172.21.15.16:0/2457727231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0009aae0 con 0x7fae00084f30 2019-12-05T06:08:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.127+0000 7fadfd7fa700 1 -- 172.21.15.16:0/2457727231 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadf800aac0 con 0x7fae00084f30 2019-12-05T06:08:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.127+0000 7fae0642f700 1 -- 172.21.15.16:0/2457727231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0009aae0 con 0x7fae00084f30 2019-12-05T06:08:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.128+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.129+0000 7fadfd7fa700 1 -- 172.21.15.16:0/2457727231 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fadf8009d10 con 0x7fae00084f30 2019-12-05T06:08:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.129+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.129+0000 7fadfd7fa700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fade4023630 0x7fade4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.129+0000 7fadfd7fa700 1 -- 172.21.15.16:0/2457727231 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6117+0+0 (secure 0 0 0) 0x7fadf80460e0 con 0x7fae00084f30 2019-12-05T06:08:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.130+0000 7fae049cc700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fade4023630 0x7fade4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:49.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.130+0000 7fae049cc700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fade4023630 0x7fade4025ac0 crc :-1 s=READY pgs=125 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:49.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.144+0000 7fae0642f700 1 -- 172.21.15.16:0/2457727231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae0009aae0 con 0x7fae00084f30 2019-12-05T06:08:49.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.153+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.153+0000 7fadfd7fa700 1 -- 172.21.15.16:0/2457727231 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fadf8000df0 con 0x7fae00084f30 2019-12-05T06:08:49.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.409+0000 7fae0642f700 1 -- 172.21.15.16:0/2457727231 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae0009aae0 con 0x7fae00084f30 2019-12-05T06:08:49.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.410+0000 7fae051cd700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:49.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.410+0000 7fadfd7fa700 1 -- 172.21.15.16:0/2457727231 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+1775 (secure 0 0 0) 0x7fadf8009c10 con 0x7fae00084f30 2019-12-05T06:08:49.411 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 15 2019-12-05T06:08:49.411 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:49.411 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:48.948270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:reconnect","state_seq":32,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:49.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 -- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fade4023630 msgr2=0x7fade4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:49.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fade4023630 0x7fade4025ac0 crc :-1 s=READY pgs=125 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:49.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 -- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 msgr2=0x7fae00155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:49.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 0x7fae00155f30 secure :-1 s=READY pgs=212 cs=0 l=1 rx=0x7fadf80107b0 tx=0x7fadf8010630).stop 2019-12-05T06:08:49.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 -- 172.21.15.16:0/2457727231 shutdown_connections 2019-12-05T06:08:49.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fade4023630 0x7fade4025ac0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:49.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae00084f30 0x7fae00155f30 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:49.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 --2- 172.21.15.16:0/2457727231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae0009bbe0 0x7fae001539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:49.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 -- 172.21.15.16:0/2457727231 shutdown_connections 2019-12-05T06:08:49.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 -- 172.21.15.16:0/2457727231 wait complete. 2019-12-05T06:08:49.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.415+0000 7fae0642f700 1 -- 172.21.15.16:0/2457727231 >> 172.21.15.16:0/2457727231 conn(0x7fae000714f0 msgr2=0x7fae0009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:49.425 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:08:49.425 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:08:49.426 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.624+0000 7fa040adb700 1 Processor -- start 2019-12-05T06:08:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.624+0000 7fa040adb700 1 -- start start 2019-12-05T06:08:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.625+0000 7fa040adb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa03c08d2c0 0x7fa03c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.625+0000 7fa040adb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.626+0000 7fa040adb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c0958e0 con 0x7fa03c08c6d0 2019-12-05T06:08:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.626+0000 7fa040adb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa03c08df90 con 0x7fa03c08d2c0 2019-12-05T06:08:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.626+0000 7fa040adb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c0a3f70 con 0x7fa03c08fef0 2019-12-05T06:08:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.626+0000 7fa03b59e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:50.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.626+0000 7fa03b59e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:51076/0 (socket says 172.21.15.16:51076) 2019-12-05T06:08:50.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.626+0000 7fa03b59e700 1 -- 172.21.15.16:0/129265541 learned_addr learned my addr 172.21.15.16:0/129265541 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.626+0000 7fa03a59c700 1 --2- 172.21.15.16:0/129265541 >> v2:172.21.15.76:3300/0 conn(0x7fa03c08d2c0 0x7fa03c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.627+0000 7fa03b59e700 1 -- 172.21.15.16:0/129265541 >> v2:172.21.15.76:3300/0 conn(0x7fa03c08d2c0 msgr2=0x7fa03c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.627+0000 7fa03b59e700 1 --2- 172.21.15.16:0/129265541 >> v2:172.21.15.76:3300/0 conn(0x7fa03c08d2c0 0x7fa03c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.627+0000 7fa03b59e700 1 -- 172.21.15.16:0/129265541 >> v1:172.21.15.76:6789/0 conn(0x7fa03c08c6d0 legacy=0x7fa03c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.627+0000 7fa03b59e700 1 -- 172.21.15.16:0/129265541 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c08e620 con 0x7fa03c08fef0 2019-12-05T06:08:50.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.627+0000 7fa03b59e700 1 --2- 172.21.15.16:0/129265541 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa02c00f760 tx=0x7fa02c00f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.627+0000 7fa03b59e700 1 --2- 172.21.15.16:0/129265541 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa02c00f760 tx=0x7fa02c00f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.627+0000 7fa03b59e700 1 --2- 172.21.15.16:0/129265541 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7fa02c00f760 tx=0x7fa02c00f590).ready entity=mon.2 client_cookie=9ec486c6f5c938ec server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.627+0000 7fa03b59e700 1 --2- 172.21.15.16:0/129265541 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7fa02c00f760 tx=0x7fa02c00f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.628+0000 7fa03b59e700 1 --2- 172.21.15.16:0/129265541 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7fa02c00f760 tx=0x7fa02c00f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.628+0000 7fa039d9b700 1 -- 172.21.15.16:0/129265541 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa02c000a70 con 0x7fa03c08fef0 2019-12-05T06:08:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.628+0000 7fa03b59e700 1 --2- 172.21.15.16:0/129265541 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 secure :-1 s=THROTTLE_DONE pgs=74 cs=0 l=1 rx=0x7fa02c00f760 tx=0x7fa02c00f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.628+0000 7fa039d9b700 1 -- 172.21.15.16:0/129265541 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa02c000e70 con 0x7fa03c08fef0 2019-12-05T06:08:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.628+0000 7fa039d9b700 1 -- 172.21.15.16:0/129265541 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa02c001500 con 0x7fa03c08fef0 2019-12-05T06:08:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.628+0000 7fa040adb700 1 -- 172.21.15.16:0/129265541 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 msgr2=0x7fa03c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.628+0000 7fa040adb700 1 --2- 172.21.15.16:0/129265541 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 secure :-1 s=READY pgs=74 cs=0 l=1 rx=0x7fa02c00f760 tx=0x7fa02c00f590).stop 2019-12-05T06:08:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.630+0000 7fa040adb700 1 -- 172.21.15.16:0/129265541 shutdown_connections 2019-12-05T06:08:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.630+0000 7fa040adb700 1 --2- 172.21.15.16:0/129265541 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c08fef0 0x7fa03c08f460 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:50.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.630+0000 7fa040adb700 1 --2- 172.21.15.16:0/129265541 >> v2:172.21.15.76:3300/0 conn(0x7fa03c08d2c0 0x7fa03c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:50.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.630+0000 7fa040adb700 1 -- 172.21.15.16:0/129265541 shutdown_connections 2019-12-05T06:08:50.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.630+0000 7fa040adb700 1 -- 172.21.15.16:0/129265541 wait complete. 2019-12-05T06:08:50.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:49.630+0000 7fa040adb700 1 -- 172.21.15.16:0/129265541 >> 172.21.15.16:0/129265541 conn(0x7fa03c061940 msgr2=0x7fa03c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:50.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.193+0000 7fa040adb700 1 Processor -- start 2019-12-05T06:08:50.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.193+0000 7fa040adb700 1 -- start start 2019-12-05T06:08:50.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.195+0000 7fa040adb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa03c09f7c0 0x7fa03c09eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:50.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.195+0000 7fa040adb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c153b00 0x7fa03c155f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:50.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.195+0000 7fa040adb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa03c09ca80 con 0x7fa03c09c110 2019-12-05T06:08:50.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.195+0000 7fa040adb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa03c12d280 con 0x7fa03c09f7c0 2019-12-05T06:08:50.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.195+0000 7fa040adb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa03c12d550 con 0x7fa03c153b00 2019-12-05T06:08:50.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.195+0000 7fa03a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa03c09f7c0 0x7fa03c09eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:50.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.196+0000 7fa03a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa03c09f7c0 0x7fa03c09eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47090/0 (socket says 172.21.15.16:47090) 2019-12-05T06:08:50.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.196+0000 7fa03a59c700 1 -- 172.21.15.16:0/2528760289 learned_addr learned my addr 172.21.15.16:0/2528760289 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:50.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.196+0000 7fa03b59e700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c153b00 0x7fa03c155f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:50.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.196+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2444308535 0 0) 0x7fa034001400 con 0x7fa03c09c110 2019-12-05T06:08:50.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.196+0000 7fa03a59c700 1 -- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c153b00 msgr2=0x7fa03c155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:50.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.196+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c153b00 0x7fa03c155f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:50.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.196+0000 7fa03a59c700 1 -- 172.21.15.16:0/2528760289 >> v1:172.21.15.76:6789/0 conn(0x7fa03c09c110 legacy=0x7fa03c09e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:50.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.196+0000 7fa03a59c700 1 -- 172.21.15.16:0/2528760289 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa03c09def0 con 0x7fa03c09f7c0 2019-12-05T06:08:50.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.196+0000 7fa03b59e700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa03c153b00 0x7fa03c155f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:50.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.196+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> v2:172.21.15.16:3300/0 conn(0x7fa03c09f7c0 0x7fa03c09eff0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa024010450 tx=0x7fa0240107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.197+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> v2:172.21.15.16:3300/0 conn(0x7fa03c09f7c0 0x7fa03c09eff0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa024010450 tx=0x7fa0240107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.197+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 0x7fa03c09eff0 secure :-1 s=READY pgs=245 cs=0 l=1 rx=0x7fa024010450 tx=0x7fa0240107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:50.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.197+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 0x7fa03c09eff0 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7fa024010450 tx=0x7fa0240107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.197+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa024000940 con 0x7fa03c09f7c0 2019-12-05T06:08:50.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.197+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 0x7fa03c09eff0 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7fa024010450 tx=0x7fa0240107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.197+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 0x7fa03c09eff0 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7fa024010450 tx=0x7fa0240107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.197+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa024000e80 con 0x7fa03c09f7c0 2019-12-05T06:08:50.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.197+0000 7fa040adb700 1 -- 172.21.15.16:0/2528760289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa03c09def0 con 0x7fa03c09f7c0 2019-12-05T06:08:50.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.197+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0240014d0 con 0x7fa03c09f7c0 2019-12-05T06:08:50.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.197+0000 7fa040adb700 1 -- 172.21.15.16:0/2528760289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa03c09def0 con 0x7fa03c09f7c0 2019-12-05T06:08:50.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.198+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 0x7fa03c09eff0 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7fa024010450 tx=0x7fa0240107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.199+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa024001070 con 0x7fa03c09f7c0 2019-12-05T06:08:50.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.199+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 0x7fa03c09eff0 secure :-1 s=THROTTLE_DONE pgs=245 cs=0 l=1 rx=0x7fa024010450 tx=0x7fa0240107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.200+0000 7fa0337fe700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa020023630 0x7fa020025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:50.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.200+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6117+0+0 (secure 0 0 0) 0x7fa024045db0 con 0x7fa03c09f7c0 2019-12-05T06:08:50.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.200+0000 7fa03ad9d700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa020023630 0x7fa020025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:50.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.201+0000 7fa03ad9d700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa020023630 0x7fa020025ac0 crc :-1 s=READY pgs=126 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:50.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.239+0000 7fa040adb700 1 -- 172.21.15.16:0/2528760289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa03c09def0 con 0x7fa03c09f7c0 2019-12-05T06:08:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa03a59c700 1 -- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 msgr2=0x7fa03c09eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:08:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa03a59c700 1 -- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 msgr2=0x7fa03c09eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:08:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 0x7fa03c09eff0 secure :-1 s=READY pgs=245 cs=0 l=1 rx=0x7fa024010450 tx=0x7fa0240107f0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:08:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 0x7fa03c09eff0 secure :-1 s=READY pgs=245 cs=0 l=1 rx=0x7fa024010450 tx=0x7fa0240107f0).stop 2019-12-05T06:08:50.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 msgr2=0x7fa03c09eff0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:08:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa0337fe700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 0x7fa03c09eff0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa0337fe700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:50.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa0337fe700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa02002db00 0x7fa02002ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa0337fe700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0200307d0 0x7fa020032ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa020033450 con 0x7fa02002aea0 2019-12-05T06:08:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa020033680 con 0x7fa0200307d0 2019-12-05T06:08:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0200338b0 con 0x7fa02002db00 2019-12-05T06:08:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.240+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:50.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.241+0000 7fa03b59e700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa02002db00 0x7fa02002ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:50.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.241+0000 7fa03ad9d700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0200307d0 0x7fa020032ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:50.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.241+0000 7fa03a59c700 1 -- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa02002db00 msgr2=0x7fa02002ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:50.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.241+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa02002db00 0x7fa02002ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:50.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.241+0000 7fa03a59c700 1 -- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0200307d0 msgr2=0x7fa020032ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:50.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.241+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa0200307d0 0x7fa020032ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:50.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.241+0000 7fa03a59c700 1 -- 172.21.15.16:0/2528760289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7fa020033ba0 con 0x7fa02002aea0 2019-12-05T06:08:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.241+0000 7fa03a59c700 1 -- 172.21.15.16:0/2528760289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa024000b90 con 0x7fa02002aea0 2019-12-05T06:08:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.241+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa02402d490 tx=0x7fa02402f130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.241+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa02402d490 tx=0x7fa02402f130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.242+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 secure :-1 s=READY pgs=246 cs=0 l=1 rx=0x7fa02402d490 tx=0x7fa02402f130).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.250+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7fa02402d490 tx=0x7fa02402f130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.250+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7fa02402d490 tx=0x7fa02402f130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.250+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa024000b90 con 0x7fa02002aea0 2019-12-05T06:08:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.250+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa024001850 con 0x7fa02002aea0 2019-12-05T06:08:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.251+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7fa02402d490 tx=0x7fa02402f130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.252+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa0240027e0 con 0x7fa02002aea0 2019-12-05T06:08:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.260+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7fa02402d490 tx=0x7fa02402f130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.260+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa024019530 con 0x7fa02002aea0 2019-12-05T06:08:50.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.534+0000 7fa040adb700 1 -- 172.21.15.16:0/2528760289 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa03c09def0 con 0x7fa02002aea0 2019-12-05T06:08:50.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.535+0000 7fa03a59c700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 secure :-1 s=THROTTLE_DONE pgs=246 cs=0 l=1 rx=0x7fa02402d490 tx=0x7fa02402f130).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.535+0000 7fa0337fe700 1 -- 172.21.15.16:0/2528760289 <== mon.0 v2:172.21.15.16:3300/0 5 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fa02401a6a0 con 0x7fa02002aea0 2019-12-05T06:08:50.534 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 16 2019-12-05T06:08:50.534 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:50.534 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:49.950174+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:50.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.537+0000 7fa040adb700 1 -- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa020023630 msgr2=0x7fa020025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.537+0000 7fa040adb700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa020023630 0x7fa020025ac0 crc :-1 s=READY pgs=126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.537+0000 7fa040adb700 1 -- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 msgr2=0x7fa02002d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.537+0000 7fa040adb700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 secure :-1 s=READY pgs=246 cs=0 l=1 rx=0x7fa02402d490 tx=0x7fa02402f130).stop 2019-12-05T06:08:50.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.537+0000 7fa03b59e700 1 -- 172.21.15.16:0/2528760289 reap_dead start 2019-12-05T06:08:50.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.538+0000 7fa040adb700 1 -- 172.21.15.16:0/2528760289 shutdown_connections 2019-12-05T06:08:50.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.538+0000 7fa040adb700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa02002aea0 0x7fa02002d2a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:50.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.538+0000 7fa040adb700 1 --2- 172.21.15.16:0/2528760289 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa03c09f7c0 0x7fa03c09eff0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:50.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.538+0000 7fa040adb700 1 -- 172.21.15.16:0/2528760289 shutdown_connections 2019-12-05T06:08:50.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.538+0000 7fa040adb700 1 -- 172.21.15.16:0/2528760289 wait complete. 2019-12-05T06:08:50.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:50.538+0000 7fa040adb700 1 -- 172.21.15.16:0/2528760289 >> 172.21.15.16:0/2528760289 conn(0x7fa03c08acd0 msgr2=0x7fa03c099650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:50.546 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 16, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 16, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4229": { "addr": "172.21.15.76:6835/3164208461", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 3164208461, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 3164208461, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 4229, "incarnation": 14, "name": "b", "rank": 0, "state": "up:rejoin", "state_seq": 33 } }, "last_failure": 0, "last_failure_osd_epoch": 27, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:08:49.950174+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4229 } } } ], "standbys": [] } 2019-12-05T06:08:50.987 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir .snap/foo 2019-12-05T06:08:51.007 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep asdf .snap/foo/foo 2019-12-05T06:08:51.028 INFO:tasks.workunit.client.0.smithi016.stdout:asdf 2019-12-05T06:08:51.028 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir .snap/foo 2019-12-05T06:08:51.038 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo asdf 2019-12-05T06:08:51.044 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir .snap/bar 2019-12-05T06:08:51.056 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm bar 2019-12-05T06:08:51.061 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep asdf .snap/bar/bar 2019-12-05T06:08:51.074 INFO:tasks.workunit.client.0.smithi016.stdout:asdf 2019-12-05T06:08:51.074 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir .snap/bar 2019-12-05T06:08:51.085 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm foo 2019-12-05T06:08:51.089 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps false 2019-12-05T06:08:51.247 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.238+0000 7fa23cb56700 1 Processor -- start 2019-12-05T06:08:51.248 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.238+0000 7fa23cb56700 1 -- start start 2019-12-05T06:08:51.248 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.240+0000 7fa23cb56700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa23808d350 0x7fa23808f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:51.248 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.240+0000 7fa23cb56700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:51.248 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.240+0000 7fa23cb56700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa23805a480 con 0x7fa23808c760 2019-12-05T06:08:51.249 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.240+0000 7fa23cb56700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2380627d0 con 0x7fa23808d350 2019-12-05T06:08:51.249 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.240+0000 7fa23cb56700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa23808e970 con 0x7fa2380908d0 2019-12-05T06:08:51.249 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.241+0000 7fa23759e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:51.249 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.241+0000 7fa23659c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa23808d350 0x7fa23808f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:51.250 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.241+0000 7fa23659c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa23808d350 0x7fa23808f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:33198/0 (socket says 172.21.15.16:33198) 2019-12-05T06:08:51.250 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.241+0000 7fa23659c700 1 -- 172.21.15.16:0/3065927908 learned_addr learned my addr 172.21.15.16:0/3065927908 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:51.250 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.241+0000 7fa23759e700 1 --2- 172.21.15.16:0/3065927908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:51098/0 (socket says 172.21.15.16:51098) 2019-12-05T06:08:51.250 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.241+0000 7fa236d9d700 1 --1- 172.21.15.16:0/3065927908 >> v1:172.21.15.76:6789/0 conn(0x7fa23808c760 0x7fa23808cb50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53576/0 (socket says 172.21.15.16:53576) 2019-12-05T06:08:51.251 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.242+0000 7fa23759e700 1 -- 172.21.15.16:0/3065927908 >> v2:172.21.15.76:3300/0 conn(0x7fa23808d350 msgr2=0x7fa23808f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:51.251 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.242+0000 7fa23759e700 1 --2- 172.21.15.16:0/3065927908 >> v2:172.21.15.76:3300/0 conn(0x7fa23808d350 0x7fa23808f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:51.251 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.242+0000 7fa23759e700 1 -- 172.21.15.16:0/3065927908 >> v1:172.21.15.76:6789/0 conn(0x7fa23808c760 legacy=0x7fa23808cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:51.251 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.242+0000 7fa23759e700 1 -- 172.21.15.16:0/3065927908 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa23808f000 con 0x7fa2380908d0 2019-12-05T06:08:51.252 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.242+0000 7fa23659c700 1 --2- 172.21.15.16:0/3065927908 >> v2:172.21.15.76:3300/0 conn(0x7fa23808d350 0x7fa23808f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:51.252 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.242+0000 7fa23759e700 1 --2- 172.21.15.16:0/3065927908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa22800af80 tx=0x7fa22800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.252 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.242+0000 7fa23759e700 1 --2- 172.21.15.16:0/3065927908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa22800af80 tx=0x7fa22800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.252 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.242+0000 7fa23759e700 1 --2- 172.21.15.16:0/3065927908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 secure :-1 s=READY pgs=75 cs=0 l=1 rx=0x7fa22800af80 tx=0x7fa22800b790).ready entity=mon.2 client_cookie=3100519ad3da9d5e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:51.253 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.243+0000 7fa23759e700 1 --2- 172.21.15.16:0/3065927908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fa22800af80 tx=0x7fa22800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.253 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.243+0000 7fa23759e700 1 --2- 172.21.15.16:0/3065927908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fa22800af80 tx=0x7fa22800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.253 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.243+0000 7fa235d9b700 1 -- 172.21.15.16:0/3065927908 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa228000bc0 con 0x7fa2380908d0 2019-12-05T06:08:51.253 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.243+0000 7fa23759e700 1 --2- 172.21.15.16:0/3065927908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 secure :-1 s=THROTTLE_DONE pgs=75 cs=0 l=1 rx=0x7fa22800af80 tx=0x7fa22800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.254 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.243+0000 7fa235d9b700 1 -- 172.21.15.16:0/3065927908 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2280097f0 con 0x7fa2380908d0 2019-12-05T06:08:51.254 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.243+0000 7fa235d9b700 1 -- 172.21.15.16:0/3065927908 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa228009ec0 con 0x7fa2380908d0 2019-12-05T06:08:51.254 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.243+0000 7fa23cb56700 1 -- 172.21.15.16:0/3065927908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 msgr2=0x7fa23808fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:51.254 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.243+0000 7fa23cb56700 1 --2- 172.21.15.16:0/3065927908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 secure :-1 s=READY pgs=75 cs=0 l=1 rx=0x7fa22800af80 tx=0x7fa22800b790).stop 2019-12-05T06:08:51.255 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.245+0000 7fa23cb56700 1 -- 172.21.15.16:0/3065927908 shutdown_connections 2019-12-05T06:08:51.255 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.245+0000 7fa23cb56700 1 --2- 172.21.15.16:0/3065927908 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2380908d0 0x7fa23808fe40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:51.255 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.245+0000 7fa23cb56700 1 --2- 172.21.15.16:0/3065927908 >> v2:172.21.15.76:3300/0 conn(0x7fa23808d350 0x7fa23808f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:51.255 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.245+0000 7fa23cb56700 1 -- 172.21.15.16:0/3065927908 shutdown_connections 2019-12-05T06:08:51.256 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.245+0000 7fa23cb56700 1 -- 172.21.15.16:0/3065927908 wait complete. 2019-12-05T06:08:51.256 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.245+0000 7fa23cb56700 1 -- 172.21.15.16:0/3065927908 >> 172.21.15.16:0/3065927908 conn(0x7fa2380a3870 msgr2=0x7fa2380960f0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:51.261 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.250+0000 7fa23cb56700 1 Processor -- start 2019-12-05T06:08:51.261 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.250+0000 7fa23cb56700 1 -- start start 2019-12-05T06:08:51.262 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.251+0000 7fa23cb56700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa23807db10 0x7fa23807df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:51.262 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.251+0000 7fa23cb56700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa238084830 0x7fa23807e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:51.262 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.251+0000 7fa23cb56700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:51.262 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.251+0000 7fa23cb56700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa238074570 con 0x7fa238084830 2019-12-05T06:08:51.263 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.251+0000 7fa23cb56700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2380828d0 con 0x7fa23807db10 2019-12-05T06:08:51.263 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.251+0000 7fa23cb56700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa238081ea0 con 0x7fa238083860 2019-12-05T06:08:51.263 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa238084830 0x7fa23807e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:51.263 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23759e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:51.264 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa236d9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa23807db10 0x7fa23807df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:51.264 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23759e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:51100/0 (socket says 172.21.15.16:51100) 2019-12-05T06:08:51.264 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23759e700 1 -- 172.21.15.16:0/3187441749 learned_addr learned my addr 172.21.15.16:0/3187441749 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:51.264 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23659c700 1 --2- 172.21.15.16:0/3187441749 >> v2:172.21.15.16:3300/0 conn(0x7fa238084830 0x7fa23807e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47110/0 (socket says 172.21.15.16:47110) 2019-12-05T06:08:51.265 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23759e700 1 -- 172.21.15.16:0/3187441749 >> v2:172.21.15.76:3300/0 conn(0x7fa23807db10 msgr2=0x7fa23807df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:51.265 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> v2:172.21.15.76:3300/0 conn(0x7fa23807db10 0x7fa23807df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:51.265 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23759e700 1 -- 172.21.15.16:0/3187441749 >> v2:172.21.15.16:3300/0 conn(0x7fa238084830 msgr2=0x7fa23807e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:51.265 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> v2:172.21.15.16:3300/0 conn(0x7fa238084830 0x7fa23807e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:51.266 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23759e700 1 -- 172.21.15.16:0/3187441749 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa238080cc0 con 0x7fa238083860 2019-12-05T06:08:51.266 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa23659c700 1 --2- 172.21.15.16:0/3187441749 >> v2:172.21.15.16:3300/0 conn(0x7fa238084830 0x7fa23807e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:51.266 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.252+0000 7fa236d9d700 1 --2- 172.21.15.16:0/3187441749 >> v2:172.21.15.76:3300/0 conn(0x7fa23807db10 0x7fa23807df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:08:51.266 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.267 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.267 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:51.267 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.267 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.268 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.268 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa22f7fe700 1 -- 172.21.15.16:0/3187441749 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa228002c60 con 0x7fa238083860 2019-12-05T06:08:51.268 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa22f7fe700 1 -- 172.21.15.16:0/3187441749 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2280099b0 con 0x7fa238083860 2019-12-05T06:08:51.268 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa238080cc0 con 0x7fa238083860 2019-12-05T06:08:51.269 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa22f7fe700 1 -- 172.21.15.16:0/3187441749 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa22800a0a0 con 0x7fa238083860 2019-12-05T06:08:51.269 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.253+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa238080cc0 con 0x7fa238083860 2019-12-05T06:08:51.269 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.254+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.269 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.256+0000 7fa22f7fe700 1 -- 172.21.15.16:0/3187441749 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa228009ba0 con 0x7fa238083860 2019-12-05T06:08:51.270 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.256+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.270 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.256+0000 7fa22f7fe700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa21c023630 0x7fa21c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:51.270 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.256+0000 7fa22f7fe700 1 -- 172.21.15.16:0/3187441749 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6117+0+0 (secure 0 0 0) 0x7fa228045dc0 con 0x7fa238083860 2019-12-05T06:08:51.270 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.256+0000 7fa236d9d700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa21c023630 0x7fa21c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:51.271 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.257+0000 7fa236d9d700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa21c023630 0x7fa21c025ac0 crc :-1 s=READY pgs=127 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:51.271 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.270+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa238082f60 con 0x7fa238083860 2019-12-05T06:08:51.279 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.280+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.279 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.280+0000 7fa22f7fe700 1 -- 172.21.15.16:0/3187441749 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa2280095b0 con 0x7fa238083860 2019-12-05T06:08:51.542 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.544+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7fa238082f60 con 0x7fa238083860 2019-12-05T06:08:51.997 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.999+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:51.997 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:51.999+0000 7fa22f7fe700 1 -- 172.21.15.16:0/3187441749 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 disabled new snapshots v18) v1 ==== 139+0+0 (secure 0 0 0) 0x7fa228009b60 con 0x7fa238083860 2019-12-05T06:08:52.194 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:52.195+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"} v 0) v1 -- 0x7fa238082f60 con 0x7fa238083860 2019-12-05T06:08:52.547 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:52.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.821+0000 7f1e47e2c700 1 Processor -- start 2019-12-05T06:08:52.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.821+0000 7f1e47e2c700 1 -- start start 2019-12-05T06:08:52.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.823+0000 7f1e47e2c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1e4008c6d0 0x7f1e4008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:52.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.823+0000 7f1e47e2c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e4008fb80 con 0x7f1e4008f720 2019-12-05T06:08:52.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.823+0000 7f1e47e2c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e400958e0 con 0x7f1e4008fef0 2019-12-05T06:08:52.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.823+0000 7f1e47e2c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4008df90 con 0x7f1e4008c6d0 2019-12-05T06:08:52.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.823+0000 7f1e46bca700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e4008f720 0x7f1e401533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60528/0 (socket says 172.21.15.16:60528) 2019-12-05T06:08:52.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.823+0000 7f1e46bca700 1 -- 172.21.15.16:0/3886243848 learned_addr learned my addr 172.21.15.16:0/3886243848 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:52.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.823+0000 7f1e463c9700 1 --2- 172.21.15.16:0/3886243848 >> v2:172.21.15.76:3300/0 conn(0x7f1e4008c6d0 0x7f1e4008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:52.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.824+0000 7f1e453c7700 1 -- 172.21.15.16:0/3886243848 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597079651 0 0) 0x7f1e380014d0 con 0x7f1e4008f720 2019-12-05T06:08:52.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.824+0000 7f1e453c7700 1 -- 172.21.15.16:0/3886243848 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e28002e80 con 0x7f1e4008f720 2019-12-05T06:08:52.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.824+0000 7f1e463c9700 1 -- 172.21.15.16:0/3886243848 >> v1:172.21.15.76:6789/0 conn(0x7f1e4008fef0 legacy=0x7f1e4008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:52.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.824+0000 7f1e453c7700 1 -- 172.21.15.16:0/3886243848 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1288702947 0 0) 0x7f1e34001080 con 0x7f1e4008fef0 2019-12-05T06:08:52.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.824+0000 7f1e463c9700 1 -- 172.21.15.16:0/3886243848 >> v1:172.21.15.16:6789/0 conn(0x7f1e4008f720 legacy=0x7f1e401533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:52.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.824+0000 7f1e463c9700 1 -- 172.21.15.16:0/3886243848 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e4008e620 con 0x7f1e4008c6d0 2019-12-05T06:08:52.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.824+0000 7f1e463c9700 1 --2- 172.21.15.16:0/3886243848 >> v2:172.21.15.76:3300/0 conn(0x7f1e4008c6d0 0x7f1e4008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1e3000af50 tx=0x7f1e3000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.824+0000 7f1e463c9700 1 --2- 172.21.15.16:0/3886243848 >> v2:172.21.15.76:3300/0 conn(0x7f1e4008c6d0 0x7f1e4008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1e3000af50 tx=0x7f1e3000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.824+0000 7f1e463c9700 1 --2- 172.21.15.16:0/3886243848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e4008c6d0 0x7f1e4008cac0 secure :-1 s=READY pgs=217 cs=0 l=1 rx=0x7f1e3000af50 tx=0x7f1e3000b760).ready entity=mon.1 client_cookie=fd0b3e51e627cf66 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:52.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.825+0000 7f1e463c9700 1 --2- 172.21.15.16:0/3886243848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e4008c6d0 0x7f1e4008cac0 secure :-1 s=THROTTLE_DONE pgs=217 cs=0 l=1 rx=0x7f1e3000af50 tx=0x7f1e3000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.825+0000 7f1e453c7700 1 -- 172.21.15.16:0/3886243848 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e30000960 con 0x7f1e4008c6d0 2019-12-05T06:08:52.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.825+0000 7f1e463c9700 1 --2- 172.21.15.16:0/3886243848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e4008c6d0 0x7f1e4008cac0 secure :-1 s=THROTTLE_DONE pgs=217 cs=0 l=1 rx=0x7f1e3000af50 tx=0x7f1e3000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.825+0000 7f1e463c9700 1 --2- 172.21.15.16:0/3886243848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e4008c6d0 0x7f1e4008cac0 secure :-1 s=THROTTLE_DONE pgs=217 cs=0 l=1 rx=0x7f1e3000af50 tx=0x7f1e3000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.825+0000 7f1e453c7700 1 -- 172.21.15.16:0/3886243848 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e30009860 con 0x7f1e4008c6d0 2019-12-05T06:08:52.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.825+0000 7f1e453c7700 1 -- 172.21.15.16:0/3886243848 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e30009f30 con 0x7f1e4008c6d0 2019-12-05T06:08:52.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.825+0000 7f1e47e2c700 1 -- 172.21.15.16:0/3886243848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e4008c6d0 msgr2=0x7f1e4008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:52.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.825+0000 7f1e47e2c700 1 --2- 172.21.15.16:0/3886243848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e4008c6d0 0x7f1e4008cac0 secure :-1 s=READY pgs=217 cs=0 l=1 rx=0x7f1e3000af50 tx=0x7f1e3000b760).stop 2019-12-05T06:08:52.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.826+0000 7f1e47e2c700 1 -- 172.21.15.16:0/3886243848 shutdown_connections 2019-12-05T06:08:52.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.826+0000 7f1e47e2c700 1 --2- 172.21.15.16:0/3886243848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e4008c6d0 0x7f1e4008cac0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:52.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.827+0000 7f1e47e2c700 1 -- 172.21.15.16:0/3886243848 shutdown_connections 2019-12-05T06:08:52.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.827+0000 7f1e47e2c700 1 -- 172.21.15.16:0/3886243848 wait complete. 2019-12-05T06:08:52.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.827+0000 7f1e47e2c700 1 -- 172.21.15.16:0/3886243848 >> 172.21.15.16:0/3886243848 conn(0x7f1e40061940 msgr2=0x7f1e40096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:52.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.831+0000 7f1e47e2c700 1 Processor -- start 2019-12-05T06:08:52.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.831+0000 7f1e47e2c700 1 -- start start 2019-12-05T06:08:52.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.833+0000 7f1e47e2c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1e4012b640 0x7f1e4012ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.833+0000 7f1e47e2c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e40153ba0 0x7f1e40155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.833+0000 7f1e47e2c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e40131bf0 con 0x7f1e401302f0 2019-12-05T06:08:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.833+0000 7f1e47e2c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e401296e0 con 0x7f1e40153ba0 2019-12-05T06:08:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.833+0000 7f1e47e2c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e40128cb0 con 0x7f1e4012b640 2019-12-05T06:08:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.833+0000 7f1e46bca700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e40153ba0 0x7f1e40155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.833+0000 7f1e463c9700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1e401302f0 0x7f1e4012a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:53602/0 (socket says 172.21.15.16:53602) 2019-12-05T06:08:52.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.833+0000 7f1e463c9700 1 -- 172.21.15.16:0/1255949789 learned_addr learned my addr 172.21.15.16:0/1255949789 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.833+0000 7f1e45bc8700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1e4012b640 0x7f1e4012ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.833+0000 7f1e3effd700 1 -- 172.21.15.16:0/1255949789 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4059527422 0 0) 0x7f1e30000940 con 0x7f1e401302f0 2019-12-05T06:08:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e3effd700 1 -- 172.21.15.16:0/1255949789 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e20002e80 con 0x7f1e401302f0 2019-12-05T06:08:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e46bca700 1 -- 172.21.15.16:0/1255949789 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1e4012b640 msgr2=0x7f1e4012ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1e4012b640 0x7f1e4012ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e46bca700 1 -- 172.21.15.16:0/1255949789 >> v1:172.21.15.76:6789/0 conn(0x7f1e401302f0 legacy=0x7f1e4012a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:52.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e46bca700 1 -- 172.21.15.16:0/1255949789 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e40129d70 con 0x7f1e40153ba0 2019-12-05T06:08:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> v2:172.21.15.16:3300/0 conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> v2:172.21.15.16:3300/0 conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=READY pgs=248 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e3effd700 1 -- 172.21.15.16:0/1255949789 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e38001270 con 0x7f1e40153ba0 2019-12-05T06:08:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e47e2c700 1 -- 172.21.15.16:0/1255949789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e40129d70 con 0x7f1e40153ba0 2019-12-05T06:08:52.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e3effd700 1 -- 172.21.15.16:0/1255949789 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e38001580 con 0x7f1e40153ba0 2019-12-05T06:08:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e3effd700 1 -- 172.21.15.16:0/1255949789 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e38001270 con 0x7f1e40153ba0 2019-12-05T06:08:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.834+0000 7f1e47e2c700 1 -- 172.21.15.16:0/1255949789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e40129d70 con 0x7f1e40153ba0 2019-12-05T06:08:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.835+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.836+0000 7f1e3effd700 1 -- 172.21.15.16:0/1255949789 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1e380045c0 con 0x7f1e40153ba0 2019-12-05T06:08:52.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.836+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.836+0000 7f1e3effd700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e20023670 0x7f1e20025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:52.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.836+0000 7f1e3effd700 1 -- 172.21.15.16:0/1255949789 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6117+0+0 (secure 0 0 0) 0x7f1e380460d0 con 0x7f1e40153ba0 2019-12-05T06:08:52.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.837+0000 7f1e463c9700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e20023670 0x7f1e20025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:52.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.837+0000 7f1e463c9700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e20023670 0x7f1e20025ae0 crc :-1 s=READY pgs=128 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:52.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.852+0000 7f1e47e2c700 1 -- 172.21.15.16:0/1255949789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e40129d70 con 0x7f1e40153ba0 2019-12-05T06:08:52.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.861+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:52.861+0000 7f1e3effd700 1 -- 172.21.15.16:0/1255949789 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1e38000cc0 con 0x7f1e40153ba0 2019-12-05T06:08:52.994 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:52.995+0000 7fa23759e700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=THROTTLE_DONE pgs=76 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:52.994 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:52.995+0000 7fa22f7fe700 1 -- 172.21.15.16:0/3187441749 <== mon.2 v2:172.21.15.76:3301/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "false"}]=0 disabled new snapshots v19) v1 ==== 139+0+0 (secure 0 0 0) 0x7fa228009b60 con 0x7fa238083860 2019-12-05T06:08:53.010 INFO:tasks.workunit.client.0.smithi016.stderr:disabled new snapshots 2019-12-05T06:08:53.014 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa21c023630 msgr2=0x7fa21c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:53.014 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa21c023630 0x7fa21c025ac0 crc :-1 s=READY pgs=127 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.014 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 msgr2=0x7fa238083cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:53.014 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 secure :-1 s=READY pgs=76 cs=0 l=1 rx=0x7fa228010450 tx=0x7fa2280107f0).stop 2019-12-05T06:08:53.015 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 shutdown_connections 2019-12-05T06:08:53.015 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa238083860 0x7fa238083cd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.015 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 --2- 172.21.15.16:0/3187441749 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa21c023630 0x7fa21c025ac0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.015 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 --2- 172.21.15.16:0/3187441749 >> v2:172.21.15.16:3300/0 conn(0x7fa238084830 0x7fa23807e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.015 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 --2- 172.21.15.16:0/3187441749 >> v2:172.21.15.76:3300/0 conn(0x7fa23807db10 0x7fa23807df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.015 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 shutdown_connections 2019-12-05T06:08:53.016 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.015+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 wait complete. 2019-12-05T06:08:53.016 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.016+0000 7fa23cb56700 1 -- 172.21.15.16:0/3187441749 >> 172.21.15.16:0/3187441749 conn(0x7fa23808a470 msgr2=0x7fa238077fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:53.027 INFO:tasks.workunit.client.0.smithi016.stderr:+ expect_failure mkdir .snap/baz 2019-12-05T06:08:53.027 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir .snap/baz 2019-12-05T06:08:53.033 INFO:tasks.workunit.client.0.smithi016.stderr:mkdir: cannot create directory ‘.snap/baz’: Operation not permitted 2019-12-05T06:08:53.034 INFO:tasks.workunit.client.0.smithi016.stderr:+ return 0 2019-12-05T06:08:53.034 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2019-12-05T06:08:53.034 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2019-12-05T06:08:53.035 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:08:53.036 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:08:53.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.088+0000 7f1e47e2c700 1 -- 172.21.15.16:0/1255949789 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e40129d70 con 0x7f1e40153ba0 2019-12-05T06:08:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.088+0000 7f1e46bca700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.088+0000 7f1e3effd700 1 -- 172.21.15.16:0/1255949789 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 19 v19) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f1e380045c0 con 0x7f1e40153ba0 2019-12-05T06:08:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 19 2019-12-05T06:08:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.091+0000 7f1e47e2c700 1 -- 172.21.15.16:0/1255949789 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e20023670 msgr2=0x7f1e20025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.091+0000 7f1e47e2c700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e20023670 0x7f1e20025ae0 crc :-1 s=READY pgs=128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.091+0000 7f1e47e2c700 1 -- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 msgr2=0x7f1e40155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:53.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.091+0000 7f1e47e2c700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 0x7f1e40155fa0 secure :-1 s=READY pgs=248 cs=0 l=1 rx=0x7f1e380106f0 tx=0x7f1e38010520).stop 2019-12-05T06:08:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.091+0000 7f1e47e2c700 1 -- 172.21.15.16:0/1255949789 shutdown_connections 2019-12-05T06:08:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.091+0000 7f1e47e2c700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e20023670 0x7f1e20025ae0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.091+0000 7f1e47e2c700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e40153ba0 0x7f1e40155fa0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.091+0000 7f1e47e2c700 1 --2- 172.21.15.16:0/1255949789 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1e4012b640 0x7f1e4012ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.092+0000 7f1e47e2c700 1 -- 172.21.15.16:0/1255949789 shutdown_connections 2019-12-05T06:08:53.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.092+0000 7f1e47e2c700 1 -- 172.21.15.16:0/1255949789 wait complete. 2019-12-05T06:08:53.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:53.092+0000 7f1e47e2c700 1 -- 172.21.15.16:0/1255949789 >> 172.21.15.16:0/1255949789 conn(0x7f1e400a2850 msgr2=0x7f1e4006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:53.107 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:53.107 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":19,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":19,"flags":16,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:52.985948+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":34,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:53.107 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4341 2019-12-05T06:08:53.108 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.a 2019-12-05T06:08:53.160 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-1.sh... 2019-12-05T06:08:53.161 INFO:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snaptest-1.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-1.sh 2019-12-05T06:08:53.247 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:08:53.397 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.390+0000 7f2e331cb700 1 Processor -- start 2019-12-05T06:08:53.398 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.390+0000 7f2e331cb700 1 -- start start 2019-12-05T06:08:53.398 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.391+0000 7f2e331cb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e2c1490a0 0x7f2e2c149490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:53.398 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.391+0000 7f2e331cb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c09e9f0 0x7f2e2c152df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:53.398 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.391+0000 7f2e331cb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c09ee50 con 0x7f2e2c09f1c0 2019-12-05T06:08:53.398 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.391+0000 7f2e331cb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2c0a3f40 con 0x7f2e2c1490a0 2019-12-05T06:08:53.398 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.391+0000 7f2e331cb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c09d260 con 0x7f2e2c09e9f0 2019-12-05T06:08:53.399 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.391+0000 7f2e31768700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e2c1490a0 0x7f2e2c149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:53.399 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.391+0000 7f2e30f67700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2e2c09f1c0 0x7f2e2c09e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60544/0 (socket says 172.21.15.16:60544) 2019-12-05T06:08:53.399 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.391+0000 7f2e30f67700 1 -- 172.21.15.16:0/3346414023 learned_addr learned my addr 172.21.15.16:0/3346414023 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:53.399 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.392+0000 7f2e31f69700 1 --2- 172.21.15.16:0/3346414023 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c09e9f0 0x7f2e2c152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:53.399 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.392+0000 7f2e31768700 1 -- 172.21.15.16:0/3346414023 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c09e9f0 msgr2=0x7f2e2c152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:53.399 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.392+0000 7f2e31768700 1 --2- 172.21.15.16:0/3346414023 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c09e9f0 0x7f2e2c152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:53.399 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.392+0000 7f2e31768700 1 -- 172.21.15.16:0/3346414023 >> v1:172.21.15.16:6789/0 conn(0x7f2e2c09f1c0 legacy=0x7f2e2c09e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:53.400 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.392+0000 7f2e31768700 1 -- 172.21.15.16:0/3346414023 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c09d8f0 con 0x7f2e2c1490a0 2019-12-05T06:08:53.400 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.392+0000 7f2e23fff700 1 -- 172.21.15.16:0/3346414023 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161968240 0 0) 0x7f2e280013f0 con 0x7f2e2c09f1c0 2019-12-05T06:08:53.400 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.392+0000 7f2e31768700 1 --2- 172.21.15.16:0/3346414023 >> v2:172.21.15.16:3300/0 conn(0x7f2e2c1490a0 0x7f2e2c149490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2e2400af50 tx=0x7f2e2400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.401 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.392+0000 7f2e31768700 1 --2- 172.21.15.16:0/3346414023 >> v2:172.21.15.16:3300/0 conn(0x7f2e2c1490a0 0x7f2e2c149490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2e2400af50 tx=0x7f2e2400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.402 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.392+0000 7f2e31768700 1 --2- 172.21.15.16:0/3346414023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c1490a0 0x7f2e2c149490 secure :-1 s=READY pgs=250 cs=0 l=1 rx=0x7f2e2400af50 tx=0x7f2e2400b6e0).ready entity=mon.0 client_cookie=61b49f8c2bdc6eef server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:53.402 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.393+0000 7f2e31768700 1 --2- 172.21.15.16:0/3346414023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c1490a0 0x7f2e2c149490 secure :-1 s=THROTTLE_DONE pgs=250 cs=0 l=1 rx=0x7f2e2400af50 tx=0x7f2e2400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.402 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.393+0000 7f2e23fff700 1 -- 172.21.15.16:0/3346414023 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e24007210 con 0x7f2e2c1490a0 2019-12-05T06:08:53.403 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.393+0000 7f2e31768700 1 --2- 172.21.15.16:0/3346414023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c1490a0 0x7f2e2c149490 secure :-1 s=THROTTLE_DONE pgs=250 cs=0 l=1 rx=0x7f2e2400af50 tx=0x7f2e2400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.403 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.393+0000 7f2e31768700 1 --2- 172.21.15.16:0/3346414023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c1490a0 0x7f2e2c149490 secure :-1 s=THROTTLE_DONE pgs=250 cs=0 l=1 rx=0x7f2e2400af50 tx=0x7f2e2400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.403 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.393+0000 7f2e23fff700 1 -- 172.21.15.16:0/3346414023 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e240099b0 con 0x7f2e2c1490a0 2019-12-05T06:08:53.404 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.393+0000 7f2e23fff700 1 -- 172.21.15.16:0/3346414023 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e2400a080 con 0x7f2e2c1490a0 2019-12-05T06:08:53.404 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.393+0000 7f2e331cb700 1 -- 172.21.15.16:0/3346414023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c1490a0 msgr2=0x7f2e2c149490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:53.404 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.393+0000 7f2e331cb700 1 --2- 172.21.15.16:0/3346414023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c1490a0 0x7f2e2c149490 secure :-1 s=READY pgs=250 cs=0 l=1 rx=0x7f2e2400af50 tx=0x7f2e2400b6e0).stop 2019-12-05T06:08:53.404 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.394+0000 7f2e331cb700 1 -- 172.21.15.16:0/3346414023 shutdown_connections 2019-12-05T06:08:53.404 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.394+0000 7f2e331cb700 1 --2- 172.21.15.16:0/3346414023 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c09e9f0 0x7f2e2c152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:53.404 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.394+0000 7f2e331cb700 1 --2- 172.21.15.16:0/3346414023 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e2c1490a0 0x7f2e2c149490 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.405 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.395+0000 7f2e331cb700 1 -- 172.21.15.16:0/3346414023 shutdown_connections 2019-12-05T06:08:53.405 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.395+0000 7f2e331cb700 1 -- 172.21.15.16:0/3346414023 wait complete. 2019-12-05T06:08:53.405 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.395+0000 7f2e331cb700 1 -- 172.21.15.16:0/3346414023 >> 172.21.15.16:0/3346414023 conn(0x7f2e2c062220 msgr2=0x7f2e2c0a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:53.405 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.397+0000 7f2e331cb700 1 Processor -- start 2019-12-05T06:08:53.405 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.398+0000 7f2e331cb700 1 -- start start 2019-12-05T06:08:53.405 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.399+0000 7f2e331cb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:53.406 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.399+0000 7f2e331cb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2e2c12f4c0 0x7f2e2c133eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:53.406 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.399+0000 7f2e331cb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e2c1339c0 con 0x7f2e2c134a30 2019-12-05T06:08:53.406 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.399+0000 7f2e331cb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e2c131d20 con 0x7f2e2c12f4c0 2019-12-05T06:08:53.406 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.399+0000 7f2e331cb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c12db00 con 0x7f2e2c14d330 2019-12-05T06:08:53.406 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.399+0000 7f2e31768700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:53.406 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.399+0000 7f2e30f67700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2e2c12f4c0 0x7f2e2c133eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:53.406 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.399+0000 7f2e31768700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:51136/0 (socket says 172.21.15.16:51136) 2019-12-05T06:08:53.407 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.399+0000 7f2e30f67700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2e2c12f4c0 0x7f2e2c133eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:33240/0 (socket says 172.21.15.16:33240) 2019-12-05T06:08:53.407 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.399+0000 7f2e31768700 1 -- 172.21.15.16:0/3153403261 learned_addr learned my addr 172.21.15.16:0/3153403261 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:53.409 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.400+0000 7f2e31768700 1 -- 172.21.15.16:0/3153403261 >> v2:172.21.15.76:3300/0 conn(0x7f2e2c12f4c0 msgr2=0x7f2e2c133eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:53.409 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.400+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> v2:172.21.15.76:3300/0 conn(0x7f2e2c12f4c0 0x7f2e2c133eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:53.410 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.400+0000 7f2e31768700 1 -- 172.21.15.16:0/3153403261 >> v1:172.21.15.76:6789/0 conn(0x7f2e2c134a30 legacy=0x7f2e2c132420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:53.410 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.400+0000 7f2e31768700 1 -- 172.21.15.16:0/3153403261 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e2c12ebc0 con 0x7f2e2c14d330 2019-12-05T06:08:53.410 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.400+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.410 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.400+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.410 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.400+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=READY pgs=77 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:53.410 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.400+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.410 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.401+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.411 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.401+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.411 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.401+0000 7f2e21ffb700 1 -- 172.21.15.16:0/3153403261 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e2400a9f0 con 0x7f2e2c14d330 2019-12-05T06:08:53.411 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.401+0000 7f2e21ffb700 1 -- 172.21.15.16:0/3153403261 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e24000940 con 0x7f2e2c14d330 2019-12-05T06:08:53.411 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.401+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c12ebc0 con 0x7f2e2c14d330 2019-12-05T06:08:53.412 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.401+0000 7f2e21ffb700 1 -- 172.21.15.16:0/3153403261 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e24009e00 con 0x7f2e2c14d330 2019-12-05T06:08:53.412 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.401+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c12ebc0 con 0x7f2e2c14d330 2019-12-05T06:08:53.412 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.402+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.412 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.403+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.412 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.403+0000 7f2e21ffb700 1 -- 172.21.15.16:0/3153403261 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2e24009790 con 0x7f2e2c14d330 2019-12-05T06:08:53.413 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.403+0000 7f2e21ffb700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e0c023630 0x7f2e0c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:53.413 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.403+0000 7f2e21ffb700 1 -- 172.21.15.16:0/3153403261 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 6117+0+0 (secure 0 0 0) 0x7f2e24045df0 con 0x7f2e2c14d330 2019-12-05T06:08:53.413 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.403+0000 7f2e30f67700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e0c023630 0x7f2e0c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:53.413 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.404+0000 7f2e30f67700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e0c023630 0x7f2e0c025ac0 crc :-1 s=READY pgs=129 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:53.413 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.406+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c1346f0 con 0x7f2e2c14d330 2019-12-05T06:08:53.414 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.416+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:53.414 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.416+0000 7f2e21ffb700 1 -- 172.21.15.16:0/3153403261 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2e24009580 con 0x7f2e2c14d330 2019-12-05T06:08:53.669 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:53.670+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2e2c12ebc0 con 0x7f2e2c14d330 2019-12-05T06:08:54.013 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:54.005+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:54.013 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:54.005+0000 7f2e21ffb700 1 -- 172.21.15.16:0/3153403261 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v20) v1 ==== 167+0+0 (secure 0 0 0) 0x7f2e2402ae10 con 0x7f2e2c14d330 2019-12-05T06:08:54.228 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:54.229+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2e2c1346f0 con 0x7f2e2c14d330 2019-12-05T06:08:55.016 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.017+0000 7f2e31768700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=THROTTLE_DONE pgs=77 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.016 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.017+0000 7f2e21ffb700 1 -- 172.21.15.16:0/3153403261 <== mon.2 v2:172.21.15.76:3301/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v21) v1 ==== 167+0+0 (secure 0 0 0) 0x7f2e2402ae10 con 0x7f2e2c14d330 2019-12-05T06:08:55.043 INFO:tasks.workunit.client.0.smithi016.stderr:enabled new snapshots 2019-12-05T06:08:55.050 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.049+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e0c023630 msgr2=0x7f2e0c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:55.051 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.049+0000 7f2e331cb700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e0c023630 0x7f2e0c025ac0 crc :-1 s=READY pgs=129 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:55.051 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.049+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 msgr2=0x7f2e2c14d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:55.051 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.049+0000 7f2e331cb700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 secure :-1 s=READY pgs=77 cs=0 l=1 rx=0x7f2e24010420 tx=0x7f2e240102a0).stop 2019-12-05T06:08:55.051 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.049+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 shutdown_connections 2019-12-05T06:08:55.051 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.049+0000 7f2e331cb700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e0c023630 0x7f2e0c025ac0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:55.051 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.050+0000 7f2e331cb700 1 --2- 172.21.15.16:0/3153403261 >> v2:172.21.15.76:3300/0 conn(0x7f2e2c12f4c0 0x7f2e2c133eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:55.051 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.050+0000 7f2e331cb700 1 --2- 172.21.15.16:0/3153403261 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e2c14d330 0x7f2e2c14d720 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:55.052 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.050+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 shutdown_connections 2019-12-05T06:08:55.052 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.050+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 wait complete. 2019-12-05T06:08:55.052 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.050+0000 7f2e331cb700 1 -- 172.21.15.16:0/3153403261 >> 172.21.15.16:0/3153403261 conn(0x7f2e2c079e60 msgr2=0x7f2e2c07cf90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:55.062 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 1 2019-12-05T06:08:55.068 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 2 2019-12-05T06:08:55.073 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 3 2019-12-05T06:08:55.083 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -e file4 ']' 2019-12-05T06:08:55.083 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir .snap/snap1 2019-12-05T06:08:55.102 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 4 2019-12-05T06:08:55.105 INFO:tasks.workunit.client.0.smithi016.stderr:++ ls 2019-12-05T06:08:55.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ now='file1 2019-12-05T06:08:55.120 INFO:tasks.workunit.client.0.smithi016.stderr:file2 2019-12-05T06:08:55.120 INFO:tasks.workunit.client.0.smithi016.stderr:file3 2019-12-05T06:08:55.120 INFO:tasks.workunit.client.0.smithi016.stderr:file4' 2019-12-05T06:08:55.121 INFO:tasks.workunit.client.0.smithi016.stderr:++ ls .snap/snap1 2019-12-05T06:08:55.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ then='file1 2019-12-05T06:08:55.125 INFO:tasks.workunit.client.0.smithi016.stderr:file2 2019-12-05T06:08:55.126 INFO:tasks.workunit.client.0.smithi016.stderr:file3' 2019-12-05T06:08:55.126 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir .snap/snap1 2019-12-05T06:08:55.137 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' 'file1 2019-12-05T06:08:55.138 INFO:tasks.workunit.client.0.smithi016.stderr:file2 2019-12-05T06:08:55.138 INFO:tasks.workunit.client.0.smithi016.stderr:file3 2019-12-05T06:08:55.138 INFO:tasks.workunit.client.0.smithi016.stderr:file4' = 'file1 2019-12-05T06:08:55.138 INFO:tasks.workunit.client.0.smithi016.stderr:file2 2019-12-05T06:08:55.138 INFO:tasks.workunit.client.0.smithi016.stderr:file3' ']' 2019-12-05T06:08:55.138 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 1 2019-12-05T06:08:55.147 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 2 2019-12-05T06:08:55.158 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 3 2019-12-05T06:08:55.182 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir .snap/snap1 2019-12-05T06:08:55.183 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 4 2019-12-05T06:08:55.189 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir .snap/snap1 2019-12-05T06:08:55.200 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm file1 file2 file3 file4 2019-12-05T06:08:55.212 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2019-12-05T06:08:55.213 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2019-12-05T06:08:55.213 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:08:55.213 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:08:55.336 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-2.sh... 2019-12-05T06:08:55.336 INFO:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snaptest-2.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-2.sh 2019-12-05T06:08:55.627 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.609+0000 7f43c7ac2700 1 Processor -- start 2019-12-05T06:08:55.628 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.609+0000 7f43c7ac2700 1 -- start start 2019-12-05T06:08:55.628 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.611+0000 7f43c7ac2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f43c009e9f0 0x7f43c0149490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:55.628 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.611+0000 7f43c7ac2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c009ee50 con 0x7f43c01490a0 2019-12-05T06:08:55.628 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.611+0000 7f43c7ac2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c005f670 con 0x7f43c009f1c0 2019-12-05T06:08:55.628 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.611+0000 7f43c7ac2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c00a3f40 con 0x7f43c009e9f0 2019-12-05T06:08:55.628 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.611+0000 7f43c605f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f43c01490a0 0x7f43c009e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60570/0 (socket says 172.21.15.16:60570) 2019-12-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.611+0000 7f43c605f700 1 -- 172.21.15.16:0/1155378354 learned_addr learned my addr 172.21.15.16:0/1155378354 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.612+0000 7f43c6860700 1 --2- 172.21.15.16:0/1155378354 >> v2:172.21.15.76:3300/0 conn(0x7f43c009e9f0 0x7f43c0149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.612+0000 7f43c505d700 1 -- 172.21.15.16:0/1155378354 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1635090446 0 0) 0x7f43b80014d0 con 0x7f43c01490a0 2019-12-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.612+0000 7f43c505d700 1 -- 172.21.15.16:0/1155378354 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43a8002e80 con 0x7f43c01490a0 2019-12-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.612+0000 7f43c505d700 1 -- 172.21.15.16:0/1155378354 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 45505679 0 0) 0x7f43b4001080 con 0x7f43c009f1c0 2019-12-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.612+0000 7f43c505d700 1 -- 172.21.15.16:0/1155378354 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43a80041b0 con 0x7f43c009f1c0 2019-12-05T06:08:55.629 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.612+0000 7f43c6860700 1 -- 172.21.15.16:0/1155378354 >> v1:172.21.15.76:6789/0 conn(0x7f43c009f1c0 legacy=0x7f43c0150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:55.630 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.613+0000 7f43c6860700 1 -- 172.21.15.16:0/1155378354 >> v1:172.21.15.16:6789/0 conn(0x7f43c01490a0 legacy=0x7f43c009e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:55.630 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.613+0000 7f43c6860700 1 -- 172.21.15.16:0/1155378354 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c009d8f0 con 0x7f43c009e9f0 2019-12-05T06:08:55.630 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.613+0000 7f43c6860700 1 --2- 172.21.15.16:0/1155378354 >> v2:172.21.15.76:3300/0 conn(0x7f43c009e9f0 0x7f43c0149490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f43b000af80 tx=0x7f43b000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.630 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.613+0000 7f43c6860700 1 --2- 172.21.15.16:0/1155378354 >> v2:172.21.15.76:3300/0 conn(0x7f43c009e9f0 0x7f43c0149490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f43b000af80 tx=0x7f43b000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.630 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.613+0000 7f43c6860700 1 --2- 172.21.15.16:0/1155378354 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c009e9f0 0x7f43c0149490 secure :-1 s=READY pgs=222 cs=0 l=1 rx=0x7f43b000af80 tx=0x7f43b000b790).ready entity=mon.1 client_cookie=c78f2033ecdd4838 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:55.630 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.613+0000 7f43c6860700 1 --2- 172.21.15.16:0/1155378354 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c009e9f0 0x7f43c0149490 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rx=0x7f43b000af80 tx=0x7f43b000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.630 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.613+0000 7f43c6860700 1 --2- 172.21.15.16:0/1155378354 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c009e9f0 0x7f43c0149490 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rx=0x7f43b000af80 tx=0x7f43b000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.631 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.613+0000 7f43c505d700 1 -- 172.21.15.16:0/1155378354 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b0007240 con 0x7f43c009e9f0 2019-12-05T06:08:55.631 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.613+0000 7f43c6860700 1 --2- 172.21.15.16:0/1155378354 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c009e9f0 0x7f43c0149490 secure :-1 s=THROTTLE_DONE pgs=222 cs=0 l=1 rx=0x7f43b000af80 tx=0x7f43b000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.631 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.614+0000 7f43c505d700 1 -- 172.21.15.16:0/1155378354 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b0000c30 con 0x7f43c009e9f0 2019-12-05T06:08:55.631 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.614+0000 7f43c505d700 1 -- 172.21.15.16:0/1155378354 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b0009e60 con 0x7f43c009e9f0 2019-12-05T06:08:55.631 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.614+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1155378354 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c009e9f0 msgr2=0x7f43c0149490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:55.631 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.614+0000 7f43c7ac2700 1 --2- 172.21.15.16:0/1155378354 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c009e9f0 0x7f43c0149490 secure :-1 s=READY pgs=222 cs=0 l=1 rx=0x7f43b000af80 tx=0x7f43b000b790).stop 2019-12-05T06:08:55.631 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.616+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1155378354 shutdown_connections 2019-12-05T06:08:55.632 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.616+0000 7f43c7ac2700 1 --2- 172.21.15.16:0/1155378354 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f43c009e9f0 0x7f43c0149490 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:55.632 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.616+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1155378354 shutdown_connections 2019-12-05T06:08:55.632 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.616+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1155378354 wait complete. 2019-12-05T06:08:55.632 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.616+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1155378354 >> 172.21.15.16:0/1155378354 conn(0x7f43c0062220 msgr2=0x7f43c00a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:55.632 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.621+0000 7f43c7ac2700 1 Processor -- start 2019-12-05T06:08:55.641 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.621+0000 7f43c7ac2700 1 -- start start 2019-12-05T06:08:55.641 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.623+0000 7f43c7ac2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f43c0134730 0x7f43c0157240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:55.641 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.623+0000 7f43c7ac2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f43c0157780 0x7f43c0159b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:55.641 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.623+0000 7f43c7ac2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43c01312e0 con 0x7f43c0119880 2019-12-05T06:08:55.645 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.623+0000 7f43c7ac2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c0132fa0 con 0x7f43c0157780 2019-12-05T06:08:55.645 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.623+0000 7f43c7ac2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43c0132570 con 0x7f43c0134730 2019-12-05T06:08:55.645 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.623+0000 7f43c6860700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f43c0157780 0x7f43c0159b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:55.645 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.623+0000 7f43c6860700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f43c0157780 0x7f43c0159b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47180/0 (socket says 172.21.15.16:47180) 2019-12-05T06:08:55.645 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.623+0000 7f43c6860700 1 -- 172.21.15.16:0/1975956646 learned_addr learned my addr 172.21.15.16:0/1975956646 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:55.646 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.623+0000 7f43c585e700 1 --2- 172.21.15.16:0/1975956646 >> v2:172.21.15.76:3300/0 conn(0x7f43c0134730 0x7f43c0157240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:55.646 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43c6860700 1 -- 172.21.15.16:0/1975956646 >> v2:172.21.15.76:3300/0 conn(0x7f43c0134730 msgr2=0x7f43c0157240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:55.646 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43beffd700 1 -- 172.21.15.16:0/1975956646 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919687995 0 0) 0x7f43b8001480 con 0x7f43c0119880 2019-12-05T06:08:55.646 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> v2:172.21.15.76:3300/0 conn(0x7f43c0134730 0x7f43c0157240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:55.646 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43c6860700 1 -- 172.21.15.16:0/1975956646 >> v1:172.21.15.76:6789/0 conn(0x7f43c0119880 legacy=0x7f43c0133f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:55.647 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43c6860700 1 -- 172.21.15.16:0/1975956646 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43c0133630 con 0x7f43c0157780 2019-12-05T06:08:55.647 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> v2:172.21.15.16:3300/0 conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.647 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> v2:172.21.15.16:3300/0 conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.647 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=READY pgs=252 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:55.647 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=THROTTLE_DONE pgs=252 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.648 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43beffd700 1 -- 172.21.15.16:0/1975956646 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b000afe0 con 0x7f43c0157780 2019-12-05T06:08:55.648 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.624+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=THROTTLE_DONE pgs=252 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.648 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.625+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=THROTTLE_DONE pgs=252 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.648 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.625+0000 7f43beffd700 1 -- 172.21.15.16:0/1975956646 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43b00099b0 con 0x7f43c0157780 2019-12-05T06:08:55.648 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.625+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43c0133630 con 0x7f43c0157780 2019-12-05T06:08:55.648 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.625+0000 7f43beffd700 1 -- 172.21.15.16:0/1975956646 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43b000a080 con 0x7f43c0157780 2019-12-05T06:08:55.649 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.625+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43c0133630 con 0x7f43c0157780 2019-12-05T06:08:55.649 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.626+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=THROTTLE_DONE pgs=252 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.649 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.626+0000 7f43beffd700 1 -- 172.21.15.16:0/1975956646 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f43b0009ba0 con 0x7f43c0157780 2019-12-05T06:08:55.649 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.626+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=THROTTLE_DONE pgs=252 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.649 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.627+0000 7f43beffd700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f43a0023630 0x7f43a0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:55.649 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.627+0000 7f43beffd700 1 -- 172.21.15.16:0/1975956646 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f43b0045da0 con 0x7f43c0157780 2019-12-05T06:08:55.650 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.627+0000 7f43c605f700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f43a0023630 0x7f43a0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:55.650 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.628+0000 7f43c605f700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f43a0023630 0x7f43a0025ac0 crc :-1 s=READY pgs=130 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:55.651 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.641+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43c0133630 con 0x7f43c0157780 2019-12-05T06:08:55.651 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.648+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=THROTTLE_DONE pgs=252 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:55.651 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.648+0000 7f43beffd700 1 -- 172.21.15.16:0/1975956646 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f43b00095b0 con 0x7f43c0157780 2019-12-05T06:08:55.890 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:55.892+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f43c0133630 con 0x7f43c0157780 2019-12-05T06:08:56.011 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:56.012+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=THROTTLE_DONE pgs=252 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.011 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:56.012+0000 7f43beffd700 1 -- 172.21.15.16:0/1975956646 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v22) v1 ==== 167+0+0 (secure 0 0 0) 0x7f43b0009cc0 con 0x7f43c0157780 2019-12-05T06:08:56.108 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:08:56.109 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:08:56.109 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:08:56.116 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:08:56.116 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:56.244 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:56.245+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f43c0133630 con 0x7f43c0157780 2019-12-05T06:08:56.273 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:08:56.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.312+0000 7fb765e13700 1 Processor -- start 2019-12-05T06:08:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.312+0000 7fb765e13700 1 -- start start 2019-12-05T06:08:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb765e13700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb76008d2c0 0x7fb76008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb765e13700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb76008fef0 0x7fb76008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb765e13700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7600958e0 con 0x7fb76008c6d0 2019-12-05T06:08:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb765e13700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb76008df90 con 0x7fb76008d2c0 2019-12-05T06:08:56.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb765e13700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7600a3f70 con 0x7fb76008fef0 2019-12-05T06:08:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb757fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb76008d2c0 0x7fb76008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb757fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb76008d2c0 0x7fb76008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47202/0 (socket says 172.21.15.16:47202) 2019-12-05T06:08:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb757fff700 1 -- 172.21.15.16:0/2741832197 learned_addr learned my addr 172.21.15.16:0/2741832197 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb75ffff700 1 --1- 172.21.15.16:0/2741832197 >> v1:172.21.15.16:6789/0 conn(0x7fb76008c6d0 0x7fb76008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60600/0 (socket says 172.21.15.16:60600) 2019-12-05T06:08:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2741832197 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb76008fef0 0x7fb76008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb757fff700 1 -- 172.21.15.16:0/2741832197 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb76008fef0 msgr2=0x7fb76008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:56.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb757fff700 1 --2- 172.21.15.16:0/2741832197 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb76008fef0 0x7fb76008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb757fff700 1 -- 172.21.15.16:0/2741832197 >> v1:172.21.15.16:6789/0 conn(0x7fb76008c6d0 legacy=0x7fb76008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb757fff700 1 -- 172.21.15.16:0/2741832197 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76008e620 con 0x7fb76008d2c0 2019-12-05T06:08:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2741832197 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb76008fef0 0x7fb76008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:08:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb757fff700 1 --2- 172.21.15.16:0/2741832197 >> v2:172.21.15.16:3300/0 conn(0x7fb76008d2c0 0x7fb76008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb75000f740 tx=0x7fb75000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb757fff700 1 --2- 172.21.15.16:0/2741832197 >> v2:172.21.15.16:3300/0 conn(0x7fb76008d2c0 0x7fb76008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb75000f740 tx=0x7fb75000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.313+0000 7fb757fff700 1 --2- 172.21.15.16:0/2741832197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb76008d2c0 0x7fb76008ef20 secure :-1 s=READY pgs=258 cs=0 l=1 rx=0x7fb75000f740 tx=0x7fb75000fae0).ready entity=mon.0 client_cookie=bd91d1216a2d67d3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:56.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb757fff700 1 --2- 172.21.15.16:0/2741832197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb76008d2c0 0x7fb76008ef20 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7fb75000f740 tx=0x7fb75000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb757fff700 1 --2- 172.21.15.16:0/2741832197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb76008d2c0 0x7fb76008ef20 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7fb75000f740 tx=0x7fb75000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb757fff700 1 --2- 172.21.15.16:0/2741832197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb76008d2c0 0x7fb76008ef20 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7fb75000f740 tx=0x7fb75000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb75f7fe700 1 -- 172.21.15.16:0/2741832197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb750000960 con 0x7fb76008d2c0 2019-12-05T06:08:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb75f7fe700 1 -- 172.21.15.16:0/2741832197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb750000c90 con 0x7fb76008d2c0 2019-12-05T06:08:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb75f7fe700 1 -- 172.21.15.16:0/2741832197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7500013e0 con 0x7fb76008d2c0 2019-12-05T06:08:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb765e13700 1 -- 172.21.15.16:0/2741832197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb76008d2c0 msgr2=0x7fb76008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb765e13700 1 --2- 172.21.15.16:0/2741832197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb76008d2c0 0x7fb76008ef20 secure :-1 s=READY pgs=258 cs=0 l=1 rx=0x7fb75000f740 tx=0x7fb75000fae0).stop 2019-12-05T06:08:56.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb765e13700 1 -- 172.21.15.16:0/2741832197 shutdown_connections 2019-12-05T06:08:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb765e13700 1 --2- 172.21.15.16:0/2741832197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb76008d2c0 0x7fb76008ef20 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb765e13700 1 --2- 172.21.15.16:0/2741832197 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb76008fef0 0x7fb76008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb765e13700 1 -- 172.21.15.16:0/2741832197 shutdown_connections 2019-12-05T06:08:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb765e13700 1 -- 172.21.15.16:0/2741832197 wait complete. 2019-12-05T06:08:56.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.314+0000 7fb765e13700 1 -- 172.21.15.16:0/2741832197 >> 172.21.15.16:0/2741832197 conn(0x7fb760061940 msgr2=0x7fb760096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.317+0000 7fb765e13700 1 Processor -- start 2019-12-05T06:08:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.317+0000 7fb765e13700 1 -- start start 2019-12-05T06:08:56.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb765e13700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7601323e0 0x7fb760151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb765e13700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76009c0e0 con 0x7fb760150c60 2019-12-05T06:08:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb765e13700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76009bbe0 con 0x7fb7601329e0 2019-12-05T06:08:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb765e13700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb76009dde0 con 0x7fb7601323e0 2019-12-05T06:08:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb75ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb760150c60 0x7fb76009ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60604/0 (socket says 172.21.15.16:60604) 2019-12-05T06:08:56.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb764bb1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7601323e0 0x7fb760151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb75ffff700 1 -- 172.21.15.16:0/2855430768 learned_addr learned my addr 172.21.15.16:0/2855430768 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb764bb1700 1 -- 172.21.15.16:0/2855430768 >> v1:172.21.15.76:6789/0 conn(0x7fb7601329e0 legacy=0x7fb7601558a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb764bb1700 1 -- 172.21.15.16:0/2855430768 >> v1:172.21.15.16:6789/0 conn(0x7fb760150c60 legacy=0x7fb76009ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb764bb1700 1 -- 172.21.15.16:0/2855430768 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76009e470 con 0x7fb7601323e0 2019-12-05T06:08:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb75d7fa700 1 -- 172.21.15.16:0/2855430768 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1953809984 0 0) 0x7fb748001150 con 0x7fb760150c60 2019-12-05T06:08:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.318+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2855430768 >> v2:172.21.15.16:3300/0 conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2855430768 >> v2:172.21.15.16:3300/0 conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=READY pgs=260 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb75d7fa700 1 -- 172.21.15.16:0/2855430768 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb758009f50 con 0x7fb7601323e0 2019-12-05T06:08:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb75d7fa700 1 -- 172.21.15.16:0/2855430768 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb758008800 con 0x7fb7601323e0 2019-12-05T06:08:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb765e13700 1 -- 172.21.15.16:0/2855430768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76009e470 con 0x7fb7601323e0 2019-12-05T06:08:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb75d7fa700 1 -- 172.21.15.16:0/2855430768 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75800a530 con 0x7fb7601323e0 2019-12-05T06:08:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb765e13700 1 -- 172.21.15.16:0/2855430768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76009e470 con 0x7fb7601323e0 2019-12-05T06:08:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.319+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.320+0000 7fb75d7fa700 1 -- 172.21.15.16:0/2855430768 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb758009cb0 con 0x7fb7601323e0 2019-12-05T06:08:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.320+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.320+0000 7fb75d7fa700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb744023630 0x7fb744025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.320+0000 7fb75d7fa700 1 -- 172.21.15.16:0/2855430768 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6173+0+0 (secure 0 0 0) 0x7fb7580461b0 con 0x7fb7601323e0 2019-12-05T06:08:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.320+0000 7fb75ffff700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb744023630 0x7fb744025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:56.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.321+0000 7fb75ffff700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb744023630 0x7fb744025ac0 crc :-1 s=READY pgs=131 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:56.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.327+0000 7fb765e13700 1 -- 172.21.15.16:0/2855430768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76009e470 con 0x7fb7601323e0 2019-12-05T06:08:56.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.330+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.330+0000 7fb75d7fa700 1 -- 172.21.15.16:0/2855430768 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb758009960 con 0x7fb7601323e0 2019-12-05T06:08:56.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.525+0000 7fb765e13700 1 -- 172.21.15.16:0/2855430768 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb76009e470 con 0x7fb7601323e0 2019-12-05T06:08:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.526+0000 7fb764bb1700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:56.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.526+0000 7fb75d7fa700 1 -- 172.21.15.16:0/2855430768 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 22 v22) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fb758009c10 con 0x7fb7601323e0 2019-12-05T06:08:56.525 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 22 2019-12-05T06:08:56.525 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:56.525 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":22,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":22,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:56.008698+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":34,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.528+0000 7fb765e13700 1 -- 172.21.15.16:0/2855430768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb744023630 msgr2=0x7fb744025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.528+0000 7fb765e13700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb744023630 0x7fb744025ac0 crc :-1 s=READY pgs=131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.528+0000 7fb765e13700 1 -- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 msgr2=0x7fb760151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.528+0000 7fb765e13700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 0x7fb760151050 secure :-1 s=READY pgs=260 cs=0 l=1 rx=0x7fb758010740 tx=0x7fb758010ae0).stop 2019-12-05T06:08:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.528+0000 7fb765e13700 1 -- 172.21.15.16:0/2855430768 shutdown_connections 2019-12-05T06:08:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.528+0000 7fb765e13700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb744023630 0x7fb744025ac0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.528+0000 7fb765e13700 1 --2- 172.21.15.16:0/2855430768 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7601323e0 0x7fb760151050 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.529+0000 7fb765e13700 1 -- 172.21.15.16:0/2855430768 shutdown_connections 2019-12-05T06:08:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.529+0000 7fb765e13700 1 -- 172.21.15.16:0/2855430768 wait complete. 2019-12-05T06:08:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:56.529+0000 7fb765e13700 1 -- 172.21.15.16:0/2855430768 >> 172.21.15.16:0/2855430768 conn(0x7fb76009f840 msgr2=0x7fb7600706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:56.542 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:08:57.021 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.022+0000 7f43c6860700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=THROTTLE_DONE pgs=252 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:57.021 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.023+0000 7f43beffd700 1 -- 172.21.15.16:0/1975956646 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v23) v1 ==== 167+0+0 (secure 0 0 0) 0x7f43b00096d0 con 0x7f43c0157780 2019-12-05T06:08:57.059 INFO:tasks.workunit.client.0.smithi016.stderr:enabled new snapshots 2019-12-05T06:08:57.064 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.065+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f43a0023630 msgr2=0x7f43a0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:57.065 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.065+0000 7f43c7ac2700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f43a0023630 0x7f43a0025ac0 crc :-1 s=READY pgs=130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:57.065 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.065+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 msgr2=0x7f43c0159b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:57.065 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.065+0000 7f43c7ac2700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 secure :-1 s=READY pgs=252 cs=0 l=1 rx=0x7f43b0010440 tx=0x7f43b00102c0).stop 2019-12-05T06:08:57.065 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.066+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 shutdown_connections 2019-12-05T06:08:57.065 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.066+0000 7f43c7ac2700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f43a0023630 0x7f43a0025ac0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:57.065 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.066+0000 7f43c7ac2700 1 --2- 172.21.15.16:0/1975956646 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f43c0157780 0x7f43c0159b80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:57.066 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.066+0000 7f43c7ac2700 1 --2- 172.21.15.16:0/1975956646 >> v2:172.21.15.76:3300/0 conn(0x7f43c0134730 0x7f43c0157240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:57.066 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.066+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 shutdown_connections 2019-12-05T06:08:57.066 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.066+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 wait complete. 2019-12-05T06:08:57.066 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:08:57.066+0000 7f43c7ac2700 1 -- 172.21.15.16:0/1975956646 >> 172.21.15.16:0/1975956646 conn(0x7f43c007b9e0 msgr2=0x7f43c007ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:57.078 INFO:tasks.workunit.client.0.smithi016.stdout:Create dir 100 to 199 ... 2019-12-05T06:08:57.096 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 100 2019-12-05T06:08:57.150 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 101 2019-12-05T06:08:57.195 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 102 2019-12-05T06:08:57.238 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 103 2019-12-05T06:08:57.271 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 104 2019-12-05T06:08:57.304 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 105 2019-12-05T06:08:57.335 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 106 2019-12-05T06:08:57.362 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 107 2019-12-05T06:08:57.391 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 108 2019-12-05T06:08:57.419 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 109 2019-12-05T06:08:57.446 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 110 2019-12-05T06:08:57.481 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 111 2019-12-05T06:08:57.502 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 112 2019-12-05T06:08:57.530 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 113 2019-12-05T06:08:57.557 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 114 2019-12-05T06:08:57.787 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 115 2019-12-05T06:08:57.834 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:08:57.839 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 116 2019-12-05T06:08:57.841 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:08:57.885 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 117 2019-12-05T06:08:57.921 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 118 2019-12-05T06:08:57.954 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 119 2019-12-05T06:08:58.016 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 120 2019-12-05T06:08:58.046 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 121 2019-12-05T06:08:58.075 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 122 2019-12-05T06:08:58.544 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:08:58.727 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 123 2019-12-05T06:08:58.770 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 124 2019-12-05T06:08:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.788+0000 7f48ad73c700 1 Processor -- start 2019-12-05T06:08:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.788+0000 7f48ad73c700 1 -- start start 2019-12-05T06:08:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.789+0000 7f48ad73c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48a808d2c0 0x7f48a808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.789+0000 7f48ad73c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a808fef0 0x7f48a808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:08:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.789+0000 7f48ad73c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a80958e0 con 0x7f48a808c6d0 2019-12-05T06:08:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.789+0000 7f48ad73c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48a808df90 con 0x7f48a808d2c0 2019-12-05T06:08:58.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.789+0000 7f48ad73c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a80a3f70 con 0x7f48a808fef0 2019-12-05T06:08:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.789+0000 7f48a6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48a808d2c0 0x7f48a808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.789+0000 7f48a6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f48a808d2c0 0x7f48a808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:47232/0 (socket says 172.21.15.16:47232) 2019-12-05T06:08:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.789+0000 7f48a6ffd700 1 -- 172.21.15.16:0/1690080686 learned_addr learned my addr 172.21.15.16:0/1690080686 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:08:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.789+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1690080686 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a808fef0 0x7f48a808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a6ffd700 1 -- 172.21.15.16:0/1690080686 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a808fef0 msgr2=0x7f48a808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:08:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a6ffd700 1 --2- 172.21.15.16:0/1690080686 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a808fef0 0x7f48a808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:58.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a6ffd700 1 -- 172.21.15.16:0/1690080686 >> v1:172.21.15.76:6789/0 conn(0x7f48a808c6d0 legacy=0x7f48a808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a6ffd700 1 -- 172.21.15.16:0/1690080686 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a808e620 con 0x7f48a808d2c0 2019-12-05T06:08:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a6ffd700 1 --2- 172.21.15.16:0/1690080686 >> v2:172.21.15.16:3300/0 conn(0x7f48a808d2c0 0x7f48a808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f489000f740 tx=0x7f489000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a6ffd700 1 --2- 172.21.15.16:0/1690080686 >> v2:172.21.15.16:3300/0 conn(0x7f48a808d2c0 0x7f48a808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f489000f740 tx=0x7f489000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a6ffd700 1 --2- 172.21.15.16:0/1690080686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a808d2c0 0x7f48a808ef20 secure :-1 s=READY pgs=261 cs=0 l=1 rx=0x7f489000f740 tx=0x7f489000f570).ready entity=mon.0 client_cookie=f607c77299530d19 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a6ffd700 1 --2- 172.21.15.16:0/1690080686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a808d2c0 0x7f48a808ef20 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f489000f740 tx=0x7f489000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a6ffd700 1 --2- 172.21.15.16:0/1690080686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a808d2c0 0x7f48a808ef20 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f489000f740 tx=0x7f489000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a6ffd700 1 --2- 172.21.15.16:0/1690080686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a808d2c0 0x7f48a808ef20 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f489000f740 tx=0x7f489000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a67fc700 1 -- 172.21.15.16:0/1690080686 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4890000960 con 0x7f48a808d2c0 2019-12-05T06:08:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a67fc700 1 -- 172.21.15.16:0/1690080686 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4890000c90 con 0x7f48a808d2c0 2019-12-05T06:08:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48a67fc700 1 -- 172.21.15.16:0/1690080686 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4890001320 con 0x7f48a808d2c0 2019-12-05T06:08:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48ad73c700 1 -- 172.21.15.16:0/1690080686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a808d2c0 msgr2=0x7f48a808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.790+0000 7f48ad73c700 1 --2- 172.21.15.16:0/1690080686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a808d2c0 0x7f48a808ef20 secure :-1 s=READY pgs=261 cs=0 l=1 rx=0x7f489000f740 tx=0x7f489000f570).stop 2019-12-05T06:08:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.792+0000 7f48ad73c700 1 -- 172.21.15.16:0/1690080686 shutdown_connections 2019-12-05T06:08:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.792+0000 7f48ad73c700 1 --2- 172.21.15.16:0/1690080686 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a808fef0 0x7f48a808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:08:58.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.792+0000 7f48ad73c700 1 --2- 172.21.15.16:0/1690080686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f48a808d2c0 0x7f48a808ef20 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:58.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.792+0000 7f48ad73c700 1 -- 172.21.15.16:0/1690080686 shutdown_connections 2019-12-05T06:08:58.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.792+0000 7f48ad73c700 1 -- 172.21.15.16:0/1690080686 wait complete. 2019-12-05T06:08:58.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.792+0000 7f48ad73c700 1 -- 172.21.15.16:0/1690080686 >> 172.21.15.16:0/1690080686 conn(0x7f48a8061940 msgr2=0x7f48a8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.795+0000 7f48ad73c700 1 Processor -- start 2019-12-05T06:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.795+0000 7f48ad73c700 1 -- start start 2019-12-05T06:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f48ad73c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:58.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f48ad73c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a8098da0 con 0x7f48a814d400 2019-12-05T06:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f48ad73c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48a809f0a0 con 0x7f48a8132d00 2019-12-05T06:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f48ad73c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48a809bb50 con 0x7f48a814d7f0 2019-12-05T06:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f48a77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f48a814d400 0x7f48a8131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:60636/0 (socket says 172.21.15.16:60636) 2019-12-05T06:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f48a77fe700 1 -- 172.21.15.16:0/1604172110 learned_addr learned my addr 172.21.15.16:0/1604172110 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:08:58.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f489ffff700 1 -- 172.21.15.16:0/1604172110 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673513890 0 0) 0x7f4890007900 con 0x7f48a8132d00 2019-12-05T06:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f489ffff700 1 -- 172.21.15.16:0/1604172110 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f488c002e80 con 0x7f48a8132d00 2019-12-05T06:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f48a7fff700 1 -- 172.21.15.16:0/1604172110 >> v1:172.21.15.76:6789/0 conn(0x7f48a8132d00 legacy=0x7f48a81550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f48a7fff700 1 -- 172.21.15.16:0/1604172110 >> v1:172.21.15.16:6789/0 conn(0x7f48a814d400 legacy=0x7f48a8131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.797+0000 7f48a7fff700 1 -- 172.21.15.16:0/1604172110 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48a809b0f0 con 0x7f48a814d7f0 2019-12-05T06:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f489ffff700 1 -- 172.21.15.16:0/1604172110 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48a0009990 con 0x7f48a814d7f0 2019-12-05T06:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f48ad73c700 1 -- 172.21.15.16:0/1604172110 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48a809b0f0 con 0x7f48a814d7f0 2019-12-05T06:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f48ad73c700 1 -- 172.21.15.16:0/1604172110 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48a8098da0 con 0x7f48a814d7f0 2019-12-05T06:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f489ffff700 1 -- 172.21.15.16:0/1604172110 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48a00095b0 con 0x7f48a814d7f0 2019-12-05T06:08:58.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.798+0000 7f489ffff700 1 -- 172.21.15.16:0/1604172110 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48a00097a0 con 0x7f48a814d7f0 2019-12-05T06:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.799+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.800+0000 7f489ffff700 1 -- 172.21.15.16:0/1604172110 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f48a00097a0 con 0x7f48a814d7f0 2019-12-05T06:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.800+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.800+0000 7f489ffff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f488c023670 0x7f488c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.800+0000 7f489ffff700 1 -- 172.21.15.16:0/1604172110 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 6173+0+0 (secure 0 0 0) 0x7f48a0045da0 con 0x7f48a814d7f0 2019-12-05T06:08:58.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.800+0000 7f48a77fe700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f488c023670 0x7f488c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:08:58.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.801+0000 7f48a77fe700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f488c023670 0x7f488c025ae0 crc :-1 s=READY pgs=133 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:08:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.803+0000 7f48ad73c700 1 -- 172.21.15.16:0/1604172110 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48a809b0f0 con 0x7f48a814d7f0 2019-12-05T06:08:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.810+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:58.810+0000 7f489ffff700 1 -- 172.21.15.16:0/1604172110 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f48a0009c00 con 0x7f48a814d7f0 2019-12-05T06:08:58.809 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 125 2019-12-05T06:08:58.841 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 126 2019-12-05T06:08:58.870 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 127 2019-12-05T06:08:58.900 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 128 2019-12-05T06:08:58.928 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 129 2019-12-05T06:08:58.957 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 130 2019-12-05T06:08:58.988 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 131 2019-12-05T06:08:59.018 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 132 2019-12-05T06:08:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.040+0000 7f48ad73c700 1 -- 172.21.15.16:0/1604172110 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f48a809b0f0 con 0x7f48a814d7f0 2019-12-05T06:08:59.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.041+0000 7f48a7fff700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=THROTTLE_DONE pgs=78 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:08:59.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.041+0000 7f489ffff700 1 -- 172.21.15.16:0/1604172110 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f48a0009780 con 0x7f48a814d7f0 2019-12-05T06:08:59.042 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2019-12-05T06:08:59.042 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:08:59.042 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:57.018632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":34,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:08:59.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.046+0000 7f48ad73c700 1 -- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f488c023670 msgr2=0x7f488c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:59.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.046+0000 7f48ad73c700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f488c023670 0x7f488c025ae0 crc :-1 s=READY pgs=133 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:59.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.046+0000 7f48ad73c700 1 -- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 msgr2=0x7f48a8132530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:08:59.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.046+0000 7f48ad73c700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 secure :-1 s=READY pgs=78 cs=0 l=1 rx=0x7f48a0010390 tx=0x7f48a0010730).stop 2019-12-05T06:08:59.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.046+0000 7f48ad73c700 1 -- 172.21.15.16:0/1604172110 shutdown_connections 2019-12-05T06:08:59.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.046+0000 7f48ad73c700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f488c023670 0x7f488c025ae0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:59.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.047+0000 7f48ad73c700 1 --2- 172.21.15.16:0/1604172110 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f48a814d7f0 0x7f48a8132530 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:08:59.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.047+0000 7f48ad73c700 1 -- 172.21.15.16:0/1604172110 shutdown_connections 2019-12-05T06:08:59.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.047+0000 7f48ad73c700 1 -- 172.21.15.16:0/1604172110 wait complete. 2019-12-05T06:08:59.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:08:59.047+0000 7f48ad73c700 1 -- 172.21.15.16:0/1604172110 >> 172.21.15.16:0/1604172110 conn(0x7f48a80714f0 msgr2=0x7f48a809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:08:59.057 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:08:59.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 109 secs before thrashing 2019-12-05T06:08:59.059 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 133 2019-12-05T06:08:59.368 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 134 2019-12-05T06:08:59.408 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 135 2019-12-05T06:08:59.453 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 136 2019-12-05T06:08:59.492 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 137 2019-12-05T06:08:59.528 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 138 2019-12-05T06:08:59.560 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 139 2019-12-05T06:08:59.586 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 140 2019-12-05T06:08:59.615 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 141 2019-12-05T06:08:59.643 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 142 2019-12-05T06:08:59.671 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 143 2019-12-05T06:08:59.699 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 144 2019-12-05T06:08:59.727 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 145 2019-12-05T06:08:59.754 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 146 2019-12-05T06:08:59.782 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 147 2019-12-05T06:08:59.811 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 148 2019-12-05T06:08:59.840 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 149 2019-12-05T06:08:59.869 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 150 2019-12-05T06:08:59.897 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 151 2019-12-05T06:08:59.922 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 152 2019-12-05T06:08:59.944 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 153 2019-12-05T06:08:59.971 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 154 2019-12-05T06:09:00.000 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 155 2019-12-05T06:09:00.598 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 156 2019-12-05T06:09:00.651 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 157 2019-12-05T06:09:00.693 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 158 2019-12-05T06:09:00.730 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 159 2019-12-05T06:09:00.762 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 160 2019-12-05T06:09:00.794 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 161 2019-12-05T06:09:00.822 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 162 2019-12-05T06:09:00.852 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 163 2019-12-05T06:09:00.882 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 164 2019-12-05T06:09:00.907 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 165 2019-12-05T06:09:00.941 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 166 2019-12-05T06:09:00.966 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 167 2019-12-05T06:09:00.995 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 168 2019-12-05T06:09:01.027 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 169 2019-12-05T06:09:01.051 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 170 2019-12-05T06:09:01.079 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 171 2019-12-05T06:09:01.108 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 172 2019-12-05T06:09:01.135 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 173 2019-12-05T06:09:01.169 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 174 2019-12-05T06:09:01.193 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 175 2019-12-05T06:09:01.222 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 176 2019-12-05T06:09:01.248 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 177 2019-12-05T06:09:01.280 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 178 2019-12-05T06:09:01.303 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 179 2019-12-05T06:09:01.331 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 180 2019-12-05T06:09:01.359 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 181 2019-12-05T06:09:01.389 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 182 2019-12-05T06:09:02.356 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 183 2019-12-05T06:09:02.404 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 184 2019-12-05T06:09:02.445 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 185 2019-12-05T06:09:02.482 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 186 2019-12-05T06:09:02.514 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 187 2019-12-05T06:09:03.117 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 188 2019-12-05T06:09:03.168 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 189 2019-12-05T06:09:03.209 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 190 2019-12-05T06:09:03.246 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 191 2019-12-05T06:09:03.279 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 192 2019-12-05T06:09:03.313 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 193 2019-12-05T06:09:03.344 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 194 2019-12-05T06:09:03.373 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 195 2019-12-05T06:09:03.400 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 196 2019-12-05T06:09:03.429 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 197 2019-12-05T06:09:03.456 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 198 2019-12-05T06:09:03.484 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 199 2019-12-05T06:09:03.512 INFO:tasks.workunit.client.0.smithi016.stdout:Take first snapshot .snap/test1 2019-12-05T06:09:03.686 INFO:tasks.workunit.client.0.smithi016.stdout:Create dir 200 to 299 ... 2019-12-05T06:09:03.687 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 200 2019-12-05T06:09:04.102 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 201 2019-12-05T06:09:04.170 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 202 2019-12-05T06:09:04.199 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 203 2019-12-05T06:09:04.226 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 204 2019-12-05T06:09:05.004 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 205 2019-12-05T06:09:05.169 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 206 2019-12-05T06:09:05.212 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 207 2019-12-05T06:09:05.248 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 208 2019-12-05T06:09:05.283 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 209 2019-12-05T06:09:05.309 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 210 2019-12-05T06:09:05.338 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 211 2019-12-05T06:09:05.365 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 212 2019-12-05T06:09:05.835 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 213 2019-12-05T06:09:05.887 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 214 2019-12-05T06:09:06.160 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 215 2019-12-05T06:09:06.201 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 216 2019-12-05T06:09:06.240 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 217 2019-12-05T06:09:06.275 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 218 2019-12-05T06:09:06.305 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 219 2019-12-05T06:09:06.335 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 220 2019-12-05T06:09:06.362 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 221 2019-12-05T06:09:06.389 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 222 2019-12-05T06:09:06.415 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 223 2019-12-05T06:09:06.441 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 224 2019-12-05T06:09:06.467 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 225 2019-12-05T06:09:06.493 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 226 2019-12-05T06:09:06.518 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 227 2019-12-05T06:09:06.545 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 228 2019-12-05T06:09:06.573 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 229 2019-12-05T06:09:06.596 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 230 2019-12-05T06:09:06.624 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 231 2019-12-05T06:09:06.653 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 232 2019-12-05T06:09:06.676 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 233 2019-12-05T06:09:06.917 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 234 2019-12-05T06:09:07.072 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 235 2019-12-05T06:09:07.111 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 236 2019-12-05T06:09:07.162 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 237 2019-12-05T06:09:07.183 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 238 2019-12-05T06:09:07.215 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 239 2019-12-05T06:09:07.246 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 240 2019-12-05T06:09:07.275 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 241 2019-12-05T06:09:07.304 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 242 2019-12-05T06:09:07.343 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 243 2019-12-05T06:09:07.363 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 244 2019-12-05T06:09:07.383 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 245 2019-12-05T06:09:07.408 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 246 2019-12-05T06:09:07.440 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 247 2019-12-05T06:09:07.462 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 248 2019-12-05T06:09:07.488 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 249 2019-12-05T06:09:07.517 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 250 2019-12-05T06:09:07.542 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 251 2019-12-05T06:09:09.128 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 252 2019-12-05T06:09:09.165 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 253 2019-12-05T06:09:09.198 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 254 2019-12-05T06:09:09.229 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 255 2019-12-05T06:09:09.260 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 256 2019-12-05T06:09:09.293 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 257 2019-12-05T06:09:09.313 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 258 2019-12-05T06:09:09.339 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 259 2019-12-05T06:09:09.644 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 260 2019-12-05T06:09:09.697 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 261 2019-12-05T06:09:09.738 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 262 2019-12-05T06:09:09.774 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 263 2019-12-05T06:09:09.807 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 264 2019-12-05T06:09:09.836 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 265 2019-12-05T06:09:09.863 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 266 2019-12-05T06:09:09.893 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 267 2019-12-05T06:09:09.919 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 268 2019-12-05T06:09:09.943 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 269 2019-12-05T06:09:09.990 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 270 2019-12-05T06:09:10.017 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 271 2019-12-05T06:09:10.044 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 272 2019-12-05T06:09:10.070 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 273 2019-12-05T06:09:10.097 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 274 2019-12-05T06:09:10.122 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 275 2019-12-05T06:09:10.148 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 276 2019-12-05T06:09:10.174 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 277 2019-12-05T06:09:10.199 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 278 2019-12-05T06:09:10.224 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 279 2019-12-05T06:09:10.252 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 280 2019-12-05T06:09:10.286 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 281 2019-12-05T06:09:10.304 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 282 2019-12-05T06:09:10.327 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 283 2019-12-05T06:09:11.025 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 284 2019-12-05T06:09:11.074 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 285 2019-12-05T06:09:11.117 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 286 2019-12-05T06:09:11.156 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 287 2019-12-05T06:09:11.193 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 288 2019-12-05T06:09:11.219 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 289 2019-12-05T06:09:11.248 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 290 2019-12-05T06:09:11.277 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 291 2019-12-05T06:09:11.745 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 292 2019-12-05T06:09:11.789 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 293 2019-12-05T06:09:11.826 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 294 2019-12-05T06:09:12.119 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 295 2019-12-05T06:09:12.164 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 296 2019-12-05T06:09:12.203 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 297 2019-12-05T06:09:12.238 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 298 2019-12-05T06:09:12.277 INFO:tasks.workunit.client.0.smithi016.stdout: create dir 299 2019-12-05T06:09:12.302 INFO:tasks.workunit.client.0.smithi016.stdout:Create a snapshot in every first level dir ... 2019-12-05T06:09:12.531 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/.snap/snap-subdir-test 2019-12-05T06:09:12.556 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/30 file after the snapshot 2019-12-05T06:09:12.558 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/31 file after the snapshot 2019-12-05T06:09:12.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/32 file after the snapshot 2019-12-05T06:09:12.564 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/33 file after the snapshot 2019-12-05T06:09:12.567 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/34 file after the snapshot 2019-12-05T06:09:12.569 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/35 file after the snapshot 2019-12-05T06:09:12.572 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/36 file after the snapshot 2019-12-05T06:09:12.574 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/37 file after the snapshot 2019-12-05T06:09:12.576 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/38 file after the snapshot 2019-12-05T06:09:12.593 INFO:tasks.workunit.client.0.smithi016.stdout: create 100/39 file after the snapshot 2019-12-05T06:09:12.593 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/.snap/snap-subdir-test 2019-12-05T06:09:12.603 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/30 file after the snapshot 2019-12-05T06:09:12.605 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/31 file after the snapshot 2019-12-05T06:09:12.608 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/32 file after the snapshot 2019-12-05T06:09:12.610 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/33 file after the snapshot 2019-12-05T06:09:12.613 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/34 file after the snapshot 2019-12-05T06:09:12.616 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/35 file after the snapshot 2019-12-05T06:09:12.618 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/36 file after the snapshot 2019-12-05T06:09:12.621 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/37 file after the snapshot 2019-12-05T06:09:12.623 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/38 file after the snapshot 2019-12-05T06:09:12.626 INFO:tasks.workunit.client.0.smithi016.stdout: create 101/39 file after the snapshot 2019-12-05T06:09:12.628 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/.snap/snap-subdir-test 2019-12-05T06:09:12.652 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/30 file after the snapshot 2019-12-05T06:09:12.652 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/31 file after the snapshot 2019-12-05T06:09:12.656 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/32 file after the snapshot 2019-12-05T06:09:12.661 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/33 file after the snapshot 2019-12-05T06:09:12.661 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/34 file after the snapshot 2019-12-05T06:09:12.665 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/35 file after the snapshot 2019-12-05T06:09:12.665 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/36 file after the snapshot 2019-12-05T06:09:12.668 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/37 file after the snapshot 2019-12-05T06:09:12.673 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/38 file after the snapshot 2019-12-05T06:09:12.673 INFO:tasks.workunit.client.0.smithi016.stdout: create 102/39 file after the snapshot 2019-12-05T06:09:12.679 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/.snap/snap-subdir-test 2019-12-05T06:09:12.699 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/30 file after the snapshot 2019-12-05T06:09:12.700 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/31 file after the snapshot 2019-12-05T06:09:12.703 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/32 file after the snapshot 2019-12-05T06:09:12.706 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/33 file after the snapshot 2019-12-05T06:09:12.708 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/34 file after the snapshot 2019-12-05T06:09:12.710 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/35 file after the snapshot 2019-12-05T06:09:12.713 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/36 file after the snapshot 2019-12-05T06:09:12.715 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/37 file after the snapshot 2019-12-05T06:09:12.972 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/38 file after the snapshot 2019-12-05T06:09:13.079 INFO:tasks.workunit.client.0.smithi016.stdout: create 103/39 file after the snapshot 2019-12-05T06:09:13.083 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/.snap/snap-subdir-test 2019-12-05T06:09:13.175 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/30 file after the snapshot 2019-12-05T06:09:13.179 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/31 file after the snapshot 2019-12-05T06:09:13.185 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/32 file after the snapshot 2019-12-05T06:09:13.186 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/33 file after the snapshot 2019-12-05T06:09:13.191 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/34 file after the snapshot 2019-12-05T06:09:13.194 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/35 file after the snapshot 2019-12-05T06:09:13.198 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/36 file after the snapshot 2019-12-05T06:09:13.201 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/37 file after the snapshot 2019-12-05T06:09:13.205 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/38 file after the snapshot 2019-12-05T06:09:13.209 INFO:tasks.workunit.client.0.smithi016.stdout: create 104/39 file after the snapshot 2019-12-05T06:09:13.211 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/.snap/snap-subdir-test 2019-12-05T06:09:13.801 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/30 file after the snapshot 2019-12-05T06:09:13.801 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/31 file after the snapshot 2019-12-05T06:09:13.805 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/32 file after the snapshot 2019-12-05T06:09:13.811 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/33 file after the snapshot 2019-12-05T06:09:13.816 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/34 file after the snapshot 2019-12-05T06:09:13.820 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/35 file after the snapshot 2019-12-05T06:09:13.826 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/36 file after the snapshot 2019-12-05T06:09:13.831 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/37 file after the snapshot 2019-12-05T06:09:13.835 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/38 file after the snapshot 2019-12-05T06:09:13.840 INFO:tasks.workunit.client.0.smithi016.stdout: create 105/39 file after the snapshot 2019-12-05T06:09:13.844 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/.snap/snap-subdir-test 2019-12-05T06:09:13.885 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/30 file after the snapshot 2019-12-05T06:09:13.889 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/31 file after the snapshot 2019-12-05T06:09:13.894 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/32 file after the snapshot 2019-12-05T06:09:13.897 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/33 file after the snapshot 2019-12-05T06:09:13.902 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/34 file after the snapshot 2019-12-05T06:09:13.906 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/35 file after the snapshot 2019-12-05T06:09:13.909 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/36 file after the snapshot 2019-12-05T06:09:13.914 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/37 file after the snapshot 2019-12-05T06:09:13.917 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/38 file after the snapshot 2019-12-05T06:09:13.922 INFO:tasks.workunit.client.0.smithi016.stdout: create 106/39 file after the snapshot 2019-12-05T06:09:13.926 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/.snap/snap-subdir-test 2019-12-05T06:09:14.428 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/30 file after the snapshot 2019-12-05T06:09:14.435 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/31 file after the snapshot 2019-12-05T06:09:14.440 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/32 file after the snapshot 2019-12-05T06:09:14.445 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/33 file after the snapshot 2019-12-05T06:09:14.451 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/34 file after the snapshot 2019-12-05T06:09:14.456 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/35 file after the snapshot 2019-12-05T06:09:14.461 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/36 file after the snapshot 2019-12-05T06:09:14.466 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/37 file after the snapshot 2019-12-05T06:09:14.471 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/38 file after the snapshot 2019-12-05T06:09:14.475 INFO:tasks.workunit.client.0.smithi016.stdout: create 107/39 file after the snapshot 2019-12-05T06:09:14.479 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/.snap/snap-subdir-test 2019-12-05T06:09:14.926 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/30 file after the snapshot 2019-12-05T06:09:14.946 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/31 file after the snapshot 2019-12-05T06:09:14.947 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/32 file after the snapshot 2019-12-05T06:09:14.947 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/33 file after the snapshot 2019-12-05T06:09:14.947 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/34 file after the snapshot 2019-12-05T06:09:14.951 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/35 file after the snapshot 2019-12-05T06:09:14.996 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/36 file after the snapshot 2019-12-05T06:09:15.039 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/37 file after the snapshot 2019-12-05T06:09:15.044 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/38 file after the snapshot 2019-12-05T06:09:15.048 INFO:tasks.workunit.client.0.smithi016.stdout: create 108/39 file after the snapshot 2019-12-05T06:09:15.051 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/.snap/snap-subdir-test 2019-12-05T06:09:15.155 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/30 file after the snapshot 2019-12-05T06:09:15.159 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/31 file after the snapshot 2019-12-05T06:09:15.162 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/32 file after the snapshot 2019-12-05T06:09:15.166 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/33 file after the snapshot 2019-12-05T06:09:15.169 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/34 file after the snapshot 2019-12-05T06:09:15.173 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/35 file after the snapshot 2019-12-05T06:09:15.177 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/36 file after the snapshot 2019-12-05T06:09:15.181 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/37 file after the snapshot 2019-12-05T06:09:15.185 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/38 file after the snapshot 2019-12-05T06:09:15.189 INFO:tasks.workunit.client.0.smithi016.stdout: create 109/39 file after the snapshot 2019-12-05T06:09:15.193 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/.snap/snap-subdir-test 2019-12-05T06:09:15.420 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/30 file after the snapshot 2019-12-05T06:09:15.423 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/31 file after the snapshot 2019-12-05T06:09:15.427 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/32 file after the snapshot 2019-12-05T06:09:15.431 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/33 file after the snapshot 2019-12-05T06:09:15.448 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/34 file after the snapshot 2019-12-05T06:09:15.448 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/35 file after the snapshot 2019-12-05T06:09:15.449 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/36 file after the snapshot 2019-12-05T06:09:15.449 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/37 file after the snapshot 2019-12-05T06:09:15.454 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/38 file after the snapshot 2019-12-05T06:09:15.458 INFO:tasks.workunit.client.0.smithi016.stdout: create 110/39 file after the snapshot 2019-12-05T06:09:15.463 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/.snap/snap-subdir-test 2019-12-05T06:09:15.502 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/30 file after the snapshot 2019-12-05T06:09:15.505 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/31 file after the snapshot 2019-12-05T06:09:15.509 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/32 file after the snapshot 2019-12-05T06:09:15.513 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/33 file after the snapshot 2019-12-05T06:09:15.517 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/34 file after the snapshot 2019-12-05T06:09:15.521 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/35 file after the snapshot 2019-12-05T06:09:15.524 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/36 file after the snapshot 2019-12-05T06:09:15.528 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/37 file after the snapshot 2019-12-05T06:09:15.532 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/38 file after the snapshot 2019-12-05T06:09:15.535 INFO:tasks.workunit.client.0.smithi016.stdout: create 111/39 file after the snapshot 2019-12-05T06:09:15.538 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/.snap/snap-subdir-test 2019-12-05T06:09:15.573 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/30 file after the snapshot 2019-12-05T06:09:15.573 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/31 file after the snapshot 2019-12-05T06:09:15.575 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/32 file after the snapshot 2019-12-05T06:09:15.579 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/33 file after the snapshot 2019-12-05T06:09:15.583 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/34 file after the snapshot 2019-12-05T06:09:15.585 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/35 file after the snapshot 2019-12-05T06:09:15.588 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/36 file after the snapshot 2019-12-05T06:09:15.590 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/37 file after the snapshot 2019-12-05T06:09:15.593 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/38 file after the snapshot 2019-12-05T06:09:15.596 INFO:tasks.workunit.client.0.smithi016.stdout: create 112/39 file after the snapshot 2019-12-05T06:09:15.599 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/.snap/snap-subdir-test 2019-12-05T06:09:15.626 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/30 file after the snapshot 2019-12-05T06:09:15.633 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/31 file after the snapshot 2019-12-05T06:09:15.633 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/32 file after the snapshot 2019-12-05T06:09:15.633 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/33 file after the snapshot 2019-12-05T06:09:15.635 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/34 file after the snapshot 2019-12-05T06:09:15.644 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/35 file after the snapshot 2019-12-05T06:09:15.644 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/36 file after the snapshot 2019-12-05T06:09:15.644 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/37 file after the snapshot 2019-12-05T06:09:15.647 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/38 file after the snapshot 2019-12-05T06:09:15.649 INFO:tasks.workunit.client.0.smithi016.stdout: create 113/39 file after the snapshot 2019-12-05T06:09:15.652 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/.snap/snap-subdir-test 2019-12-05T06:09:15.678 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/30 file after the snapshot 2019-12-05T06:09:15.679 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/31 file after the snapshot 2019-12-05T06:09:15.682 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/32 file after the snapshot 2019-12-05T06:09:15.686 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/33 file after the snapshot 2019-12-05T06:09:15.688 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/34 file after the snapshot 2019-12-05T06:09:15.691 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/35 file after the snapshot 2019-12-05T06:09:15.693 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/36 file after the snapshot 2019-12-05T06:09:15.696 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/37 file after the snapshot 2019-12-05T06:09:15.698 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/38 file after the snapshot 2019-12-05T06:09:15.701 INFO:tasks.workunit.client.0.smithi016.stdout: create 114/39 file after the snapshot 2019-12-05T06:09:15.704 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/.snap/snap-subdir-test 2019-12-05T06:09:15.728 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/30 file after the snapshot 2019-12-05T06:09:15.730 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/31 file after the snapshot 2019-12-05T06:09:15.732 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/32 file after the snapshot 2019-12-05T06:09:15.734 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/33 file after the snapshot 2019-12-05T06:09:15.739 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/34 file after the snapshot 2019-12-05T06:09:15.741 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/35 file after the snapshot 2019-12-05T06:09:15.744 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/36 file after the snapshot 2019-12-05T06:09:15.745 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/37 file after the snapshot 2019-12-05T06:09:15.748 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/38 file after the snapshot 2019-12-05T06:09:15.751 INFO:tasks.workunit.client.0.smithi016.stdout: create 115/39 file after the snapshot 2019-12-05T06:09:15.754 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/.snap/snap-subdir-test 2019-12-05T06:09:15.777 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/30 file after the snapshot 2019-12-05T06:09:15.780 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/31 file after the snapshot 2019-12-05T06:09:15.782 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/32 file after the snapshot 2019-12-05T06:09:15.785 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/33 file after the snapshot 2019-12-05T06:09:15.802 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/34 file after the snapshot 2019-12-05T06:09:15.802 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/35 file after the snapshot 2019-12-05T06:09:15.802 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/36 file after the snapshot 2019-12-05T06:09:15.803 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/37 file after the snapshot 2019-12-05T06:09:15.803 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/38 file after the snapshot 2019-12-05T06:09:15.803 INFO:tasks.workunit.client.0.smithi016.stdout: create 116/39 file after the snapshot 2019-12-05T06:09:15.803 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/.snap/snap-subdir-test 2019-12-05T06:09:15.836 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/30 file after the snapshot 2019-12-05T06:09:15.839 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/31 file after the snapshot 2019-12-05T06:09:15.843 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/32 file after the snapshot 2019-12-05T06:09:15.845 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/33 file after the snapshot 2019-12-05T06:09:15.848 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/34 file after the snapshot 2019-12-05T06:09:15.850 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/35 file after the snapshot 2019-12-05T06:09:15.853 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/36 file after the snapshot 2019-12-05T06:09:15.855 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/37 file after the snapshot 2019-12-05T06:09:15.857 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/38 file after the snapshot 2019-12-05T06:09:15.860 INFO:tasks.workunit.client.0.smithi016.stdout: create 117/39 file after the snapshot 2019-12-05T06:09:15.863 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/.snap/snap-subdir-test 2019-12-05T06:09:15.888 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/30 file after the snapshot 2019-12-05T06:09:15.890 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/31 file after the snapshot 2019-12-05T06:09:15.892 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/32 file after the snapshot 2019-12-05T06:09:15.896 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/33 file after the snapshot 2019-12-05T06:09:15.899 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/34 file after the snapshot 2019-12-05T06:09:15.901 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/35 file after the snapshot 2019-12-05T06:09:15.903 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/36 file after the snapshot 2019-12-05T06:09:15.906 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/37 file after the snapshot 2019-12-05T06:09:15.909 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/38 file after the snapshot 2019-12-05T06:09:15.912 INFO:tasks.workunit.client.0.smithi016.stdout: create 118/39 file after the snapshot 2019-12-05T06:09:15.914 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/.snap/snap-subdir-test 2019-12-05T06:09:15.938 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/30 file after the snapshot 2019-12-05T06:09:15.940 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/31 file after the snapshot 2019-12-05T06:09:15.947 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/32 file after the snapshot 2019-12-05T06:09:15.947 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/33 file after the snapshot 2019-12-05T06:09:15.948 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/34 file after the snapshot 2019-12-05T06:09:15.950 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/35 file after the snapshot 2019-12-05T06:09:15.954 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/36 file after the snapshot 2019-12-05T06:09:16.017 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/37 file after the snapshot 2019-12-05T06:09:16.020 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/38 file after the snapshot 2019-12-05T06:09:16.024 INFO:tasks.workunit.client.0.smithi016.stdout: create 119/39 file after the snapshot 2019-12-05T06:09:16.027 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/.snap/snap-subdir-test 2019-12-05T06:09:16.219 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/30 file after the snapshot 2019-12-05T06:09:16.224 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/31 file after the snapshot 2019-12-05T06:09:16.230 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/32 file after the snapshot 2019-12-05T06:09:16.235 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/33 file after the snapshot 2019-12-05T06:09:16.240 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/34 file after the snapshot 2019-12-05T06:09:16.248 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/35 file after the snapshot 2019-12-05T06:09:16.253 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/36 file after the snapshot 2019-12-05T06:09:16.255 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/37 file after the snapshot 2019-12-05T06:09:16.260 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/38 file after the snapshot 2019-12-05T06:09:16.264 INFO:tasks.workunit.client.0.smithi016.stdout: create 120/39 file after the snapshot 2019-12-05T06:09:16.271 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/.snap/snap-subdir-test 2019-12-05T06:09:16.299 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/30 file after the snapshot 2019-12-05T06:09:16.303 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/31 file after the snapshot 2019-12-05T06:09:16.307 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/32 file after the snapshot 2019-12-05T06:09:16.311 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/33 file after the snapshot 2019-12-05T06:09:16.316 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/34 file after the snapshot 2019-12-05T06:09:16.318 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/35 file after the snapshot 2019-12-05T06:09:16.322 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/36 file after the snapshot 2019-12-05T06:09:16.324 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/37 file after the snapshot 2019-12-05T06:09:16.328 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/38 file after the snapshot 2019-12-05T06:09:16.331 INFO:tasks.workunit.client.0.smithi016.stdout: create 121/39 file after the snapshot 2019-12-05T06:09:16.334 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/.snap/snap-subdir-test 2019-12-05T06:09:16.362 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/30 file after the snapshot 2019-12-05T06:09:16.365 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/31 file after the snapshot 2019-12-05T06:09:16.368 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/32 file after the snapshot 2019-12-05T06:09:16.371 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/33 file after the snapshot 2019-12-05T06:09:16.375 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/34 file after the snapshot 2019-12-05T06:09:16.378 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/35 file after the snapshot 2019-12-05T06:09:16.381 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/36 file after the snapshot 2019-12-05T06:09:16.383 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/37 file after the snapshot 2019-12-05T06:09:16.386 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/38 file after the snapshot 2019-12-05T06:09:16.389 INFO:tasks.workunit.client.0.smithi016.stdout: create 122/39 file after the snapshot 2019-12-05T06:09:16.392 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/.snap/snap-subdir-test 2019-12-05T06:09:16.420 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/30 file after the snapshot 2019-12-05T06:09:16.422 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/31 file after the snapshot 2019-12-05T06:09:16.425 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/32 file after the snapshot 2019-12-05T06:09:16.429 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/33 file after the snapshot 2019-12-05T06:09:16.432 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/34 file after the snapshot 2019-12-05T06:09:16.434 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/35 file after the snapshot 2019-12-05T06:09:16.437 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/36 file after the snapshot 2019-12-05T06:09:16.440 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/37 file after the snapshot 2019-12-05T06:09:16.443 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/38 file after the snapshot 2019-12-05T06:09:16.445 INFO:tasks.workunit.client.0.smithi016.stdout: create 123/39 file after the snapshot 2019-12-05T06:09:16.448 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/.snap/snap-subdir-test 2019-12-05T06:09:16.476 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/30 file after the snapshot 2019-12-05T06:09:16.478 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/31 file after the snapshot 2019-12-05T06:09:16.480 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/32 file after the snapshot 2019-12-05T06:09:16.483 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/33 file after the snapshot 2019-12-05T06:09:16.488 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/34 file after the snapshot 2019-12-05T06:09:16.493 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/35 file after the snapshot 2019-12-05T06:09:16.493 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/36 file after the snapshot 2019-12-05T06:09:16.494 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/37 file after the snapshot 2019-12-05T06:09:16.497 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/38 file after the snapshot 2019-12-05T06:09:16.499 INFO:tasks.workunit.client.0.smithi016.stdout: create 124/39 file after the snapshot 2019-12-05T06:09:16.502 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/.snap/snap-subdir-test 2019-12-05T06:09:16.536 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/30 file after the snapshot 2019-12-05T06:09:16.539 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/31 file after the snapshot 2019-12-05T06:09:16.541 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/32 file after the snapshot 2019-12-05T06:09:16.545 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/33 file after the snapshot 2019-12-05T06:09:16.547 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/34 file after the snapshot 2019-12-05T06:09:16.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/35 file after the snapshot 2019-12-05T06:09:16.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/36 file after the snapshot 2019-12-05T06:09:16.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/37 file after the snapshot 2019-12-05T06:09:16.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/38 file after the snapshot 2019-12-05T06:09:16.562 INFO:tasks.workunit.client.0.smithi016.stdout: create 125/39 file after the snapshot 2019-12-05T06:09:16.563 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/.snap/snap-subdir-test 2019-12-05T06:09:16.587 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/30 file after the snapshot 2019-12-05T06:09:16.590 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/31 file after the snapshot 2019-12-05T06:09:16.593 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/32 file after the snapshot 2019-12-05T06:09:16.601 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/33 file after the snapshot 2019-12-05T06:09:16.602 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/34 file after the snapshot 2019-12-05T06:09:16.603 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/35 file after the snapshot 2019-12-05T06:09:16.605 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/36 file after the snapshot 2019-12-05T06:09:16.609 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/37 file after the snapshot 2019-12-05T06:09:16.610 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/38 file after the snapshot 2019-12-05T06:09:16.614 INFO:tasks.workunit.client.0.smithi016.stdout: create 126/39 file after the snapshot 2019-12-05T06:09:16.615 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/.snap/snap-subdir-test 2019-12-05T06:09:16.639 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/30 file after the snapshot 2019-12-05T06:09:16.641 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/31 file after the snapshot 2019-12-05T06:09:16.642 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/32 file after the snapshot 2019-12-05T06:09:16.645 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/33 file after the snapshot 2019-12-05T06:09:16.647 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/34 file after the snapshot 2019-12-05T06:09:16.649 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/35 file after the snapshot 2019-12-05T06:09:16.652 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/36 file after the snapshot 2019-12-05T06:09:16.654 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/37 file after the snapshot 2019-12-05T06:09:16.656 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/38 file after the snapshot 2019-12-05T06:09:16.659 INFO:tasks.workunit.client.0.smithi016.stdout: create 127/39 file after the snapshot 2019-12-05T06:09:16.662 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/.snap/snap-subdir-test 2019-12-05T06:09:16.969 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/30 file after the snapshot 2019-12-05T06:09:17.038 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/31 file after the snapshot 2019-12-05T06:09:17.041 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/32 file after the snapshot 2019-12-05T06:09:17.046 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/33 file after the snapshot 2019-12-05T06:09:17.050 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/34 file after the snapshot 2019-12-05T06:09:17.055 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/35 file after the snapshot 2019-12-05T06:09:17.059 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/36 file after the snapshot 2019-12-05T06:09:17.063 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/37 file after the snapshot 2019-12-05T06:09:17.067 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/38 file after the snapshot 2019-12-05T06:09:17.075 INFO:tasks.workunit.client.0.smithi016.stdout: create 128/39 file after the snapshot 2019-12-05T06:09:17.076 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/.snap/snap-subdir-test 2019-12-05T06:09:17.855 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/30 file after the snapshot 2019-12-05T06:09:17.860 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/31 file after the snapshot 2019-12-05T06:09:17.866 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/32 file after the snapshot 2019-12-05T06:09:17.872 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/33 file after the snapshot 2019-12-05T06:09:17.879 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/34 file after the snapshot 2019-12-05T06:09:17.883 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/35 file after the snapshot 2019-12-05T06:09:17.889 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/36 file after the snapshot 2019-12-05T06:09:17.893 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/37 file after the snapshot 2019-12-05T06:09:17.897 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/38 file after the snapshot 2019-12-05T06:09:17.902 INFO:tasks.workunit.client.0.smithi016.stdout: create 129/39 file after the snapshot 2019-12-05T06:09:17.906 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/.snap/snap-subdir-test 2019-12-05T06:09:18.155 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/30 file after the snapshot 2019-12-05T06:09:18.230 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/31 file after the snapshot 2019-12-05T06:09:18.235 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/32 file after the snapshot 2019-12-05T06:09:18.240 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/33 file after the snapshot 2019-12-05T06:09:18.245 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/34 file after the snapshot 2019-12-05T06:09:18.249 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/35 file after the snapshot 2019-12-05T06:09:18.253 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/36 file after the snapshot 2019-12-05T06:09:18.256 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/37 file after the snapshot 2019-12-05T06:09:18.260 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/38 file after the snapshot 2019-12-05T06:09:18.264 INFO:tasks.workunit.client.0.smithi016.stdout: create 130/39 file after the snapshot 2019-12-05T06:09:18.268 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/.snap/snap-subdir-test 2019-12-05T06:09:18.344 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/30 file after the snapshot 2019-12-05T06:09:18.348 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/31 file after the snapshot 2019-12-05T06:09:18.351 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/32 file after the snapshot 2019-12-05T06:09:18.355 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/33 file after the snapshot 2019-12-05T06:09:18.358 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/34 file after the snapshot 2019-12-05T06:09:18.362 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/35 file after the snapshot 2019-12-05T06:09:18.364 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/36 file after the snapshot 2019-12-05T06:09:18.368 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/37 file after the snapshot 2019-12-05T06:09:18.371 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/38 file after the snapshot 2019-12-05T06:09:18.374 INFO:tasks.workunit.client.0.smithi016.stdout: create 131/39 file after the snapshot 2019-12-05T06:09:18.377 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/.snap/snap-subdir-test 2019-12-05T06:09:18.403 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/30 file after the snapshot 2019-12-05T06:09:18.406 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/31 file after the snapshot 2019-12-05T06:09:18.410 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/32 file after the snapshot 2019-12-05T06:09:18.414 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/33 file after the snapshot 2019-12-05T06:09:18.417 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/34 file after the snapshot 2019-12-05T06:09:18.420 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/35 file after the snapshot 2019-12-05T06:09:18.423 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/36 file after the snapshot 2019-12-05T06:09:18.670 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/37 file after the snapshot 2019-12-05T06:09:18.672 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/38 file after the snapshot 2019-12-05T06:09:18.677 INFO:tasks.workunit.client.0.smithi016.stdout: create 132/39 file after the snapshot 2019-12-05T06:09:18.683 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/.snap/snap-subdir-test 2019-12-05T06:09:18.722 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/30 file after the snapshot 2019-12-05T06:09:18.727 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/31 file after the snapshot 2019-12-05T06:09:18.731 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/32 file after the snapshot 2019-12-05T06:09:18.736 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/33 file after the snapshot 2019-12-05T06:09:18.740 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/34 file after the snapshot 2019-12-05T06:09:18.744 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/35 file after the snapshot 2019-12-05T06:09:18.749 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/36 file after the snapshot 2019-12-05T06:09:18.754 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/37 file after the snapshot 2019-12-05T06:09:18.757 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/38 file after the snapshot 2019-12-05T06:09:18.761 INFO:tasks.workunit.client.0.smithi016.stdout: create 133/39 file after the snapshot 2019-12-05T06:09:18.766 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/.snap/snap-subdir-test 2019-12-05T06:09:18.796 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/30 file after the snapshot 2019-12-05T06:09:18.800 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/31 file after the snapshot 2019-12-05T06:09:18.804 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/32 file after the snapshot 2019-12-05T06:09:18.808 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/33 file after the snapshot 2019-12-05T06:09:18.812 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/34 file after the snapshot 2019-12-05T06:09:18.816 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/35 file after the snapshot 2019-12-05T06:09:18.819 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/36 file after the snapshot 2019-12-05T06:09:18.822 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/37 file after the snapshot 2019-12-05T06:09:18.827 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/38 file after the snapshot 2019-12-05T06:09:18.829 INFO:tasks.workunit.client.0.smithi016.stdout: create 134/39 file after the snapshot 2019-12-05T06:09:18.832 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/.snap/snap-subdir-test 2019-12-05T06:09:19.021 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/30 file after the snapshot 2019-12-05T06:09:19.029 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/31 file after the snapshot 2019-12-05T06:09:19.320 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/32 file after the snapshot 2019-12-05T06:09:19.341 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/33 file after the snapshot 2019-12-05T06:09:19.342 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/34 file after the snapshot 2019-12-05T06:09:19.344 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/35 file after the snapshot 2019-12-05T06:09:19.344 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/36 file after the snapshot 2019-12-05T06:09:19.347 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/37 file after the snapshot 2019-12-05T06:09:19.351 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/38 file after the snapshot 2019-12-05T06:09:19.356 INFO:tasks.workunit.client.0.smithi016.stdout: create 135/39 file after the snapshot 2019-12-05T06:09:19.361 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/.snap/snap-subdir-test 2019-12-05T06:09:19.403 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/30 file after the snapshot 2019-12-05T06:09:19.406 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/31 file after the snapshot 2019-12-05T06:09:19.410 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/32 file after the snapshot 2019-12-05T06:09:19.413 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/33 file after the snapshot 2019-12-05T06:09:19.418 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/34 file after the snapshot 2019-12-05T06:09:19.421 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/35 file after the snapshot 2019-12-05T06:09:19.425 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/36 file after the snapshot 2019-12-05T06:09:19.429 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/37 file after the snapshot 2019-12-05T06:09:19.432 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/38 file after the snapshot 2019-12-05T06:09:19.435 INFO:tasks.workunit.client.0.smithi016.stdout: create 136/39 file after the snapshot 2019-12-05T06:09:19.439 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/.snap/snap-subdir-test 2019-12-05T06:09:19.485 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/30 file after the snapshot 2019-12-05T06:09:19.488 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/31 file after the snapshot 2019-12-05T06:09:19.493 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/32 file after the snapshot 2019-12-05T06:09:19.497 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/33 file after the snapshot 2019-12-05T06:09:19.501 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/34 file after the snapshot 2019-12-05T06:09:19.506 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/35 file after the snapshot 2019-12-05T06:09:19.512 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/36 file after the snapshot 2019-12-05T06:09:19.516 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/37 file after the snapshot 2019-12-05T06:09:19.521 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/38 file after the snapshot 2019-12-05T06:09:19.527 INFO:tasks.workunit.client.0.smithi016.stdout: create 137/39 file after the snapshot 2019-12-05T06:09:19.534 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/.snap/snap-subdir-test 2019-12-05T06:09:19.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/30 file after the snapshot 2019-12-05T06:09:19.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/31 file after the snapshot 2019-12-05T06:09:19.566 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/32 file after the snapshot 2019-12-05T06:09:19.568 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/33 file after the snapshot 2019-12-05T06:09:19.572 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/34 file after the snapshot 2019-12-05T06:09:19.575 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/35 file after the snapshot 2019-12-05T06:09:19.578 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/36 file after the snapshot 2019-12-05T06:09:19.581 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/37 file after the snapshot 2019-12-05T06:09:19.584 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/38 file after the snapshot 2019-12-05T06:09:19.588 INFO:tasks.workunit.client.0.smithi016.stdout: create 138/39 file after the snapshot 2019-12-05T06:09:19.591 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/.snap/snap-subdir-test 2019-12-05T06:09:20.430 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/30 file after the snapshot 2019-12-05T06:09:20.435 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/31 file after the snapshot 2019-12-05T06:09:20.441 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/32 file after the snapshot 2019-12-05T06:09:20.447 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/33 file after the snapshot 2019-12-05T06:09:20.451 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/34 file after the snapshot 2019-12-05T06:09:20.461 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/35 file after the snapshot 2019-12-05T06:09:20.464 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/36 file after the snapshot 2019-12-05T06:09:20.466 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/37 file after the snapshot 2019-12-05T06:09:20.471 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/38 file after the snapshot 2019-12-05T06:09:20.476 INFO:tasks.workunit.client.0.smithi016.stdout: create 139/39 file after the snapshot 2019-12-05T06:09:20.481 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/.snap/snap-subdir-test 2019-12-05T06:09:20.523 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/30 file after the snapshot 2019-12-05T06:09:20.527 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/31 file after the snapshot 2019-12-05T06:09:20.532 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/32 file after the snapshot 2019-12-05T06:09:20.536 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/33 file after the snapshot 2019-12-05T06:09:20.540 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/34 file after the snapshot 2019-12-05T06:09:20.545 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/35 file after the snapshot 2019-12-05T06:09:20.546 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/36 file after the snapshot 2019-12-05T06:09:20.551 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/37 file after the snapshot 2019-12-05T06:09:20.556 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/38 file after the snapshot 2019-12-05T06:09:20.559 INFO:tasks.workunit.client.0.smithi016.stdout: create 140/39 file after the snapshot 2019-12-05T06:09:20.563 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/.snap/snap-subdir-test 2019-12-05T06:09:20.595 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/30 file after the snapshot 2019-12-05T06:09:20.599 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/31 file after the snapshot 2019-12-05T06:09:20.601 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/32 file after the snapshot 2019-12-05T06:09:20.606 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/33 file after the snapshot 2019-12-05T06:09:20.610 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/34 file after the snapshot 2019-12-05T06:09:20.612 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/35 file after the snapshot 2019-12-05T06:09:20.615 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/36 file after the snapshot 2019-12-05T06:09:20.618 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/37 file after the snapshot 2019-12-05T06:09:20.621 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/38 file after the snapshot 2019-12-05T06:09:20.625 INFO:tasks.workunit.client.0.smithi016.stdout: create 141/39 file after the snapshot 2019-12-05T06:09:20.628 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/.snap/snap-subdir-test 2019-12-05T06:09:20.654 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/30 file after the snapshot 2019-12-05T06:09:20.657 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/31 file after the snapshot 2019-12-05T06:09:20.660 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/32 file after the snapshot 2019-12-05T06:09:20.663 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/33 file after the snapshot 2019-12-05T06:09:20.667 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/34 file after the snapshot 2019-12-05T06:09:20.670 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/35 file after the snapshot 2019-12-05T06:09:20.673 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/36 file after the snapshot 2019-12-05T06:09:20.676 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/37 file after the snapshot 2019-12-05T06:09:20.679 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/38 file after the snapshot 2019-12-05T06:09:20.682 INFO:tasks.workunit.client.0.smithi016.stdout: create 142/39 file after the snapshot 2019-12-05T06:09:20.692 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/.snap/snap-subdir-test 2019-12-05T06:09:20.711 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/30 file after the snapshot 2019-12-05T06:09:20.713 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/31 file after the snapshot 2019-12-05T06:09:20.716 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/32 file after the snapshot 2019-12-05T06:09:20.720 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/33 file after the snapshot 2019-12-05T06:09:20.723 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/34 file after the snapshot 2019-12-05T06:09:20.726 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/35 file after the snapshot 2019-12-05T06:09:20.729 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/36 file after the snapshot 2019-12-05T06:09:20.732 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/37 file after the snapshot 2019-12-05T06:09:20.736 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/38 file after the snapshot 2019-12-05T06:09:20.737 INFO:tasks.workunit.client.0.smithi016.stdout: create 143/39 file after the snapshot 2019-12-05T06:09:20.740 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/.snap/snap-subdir-test 2019-12-05T06:09:20.765 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/30 file after the snapshot 2019-12-05T06:09:20.767 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/31 file after the snapshot 2019-12-05T06:09:20.769 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/32 file after the snapshot 2019-12-05T06:09:20.772 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/33 file after the snapshot 2019-12-05T06:09:20.774 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/34 file after the snapshot 2019-12-05T06:09:20.777 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/35 file after the snapshot 2019-12-05T06:09:20.779 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/36 file after the snapshot 2019-12-05T06:09:20.782 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/37 file after the snapshot 2019-12-05T06:09:20.889 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/38 file after the snapshot 2019-12-05T06:09:20.894 INFO:tasks.workunit.client.0.smithi016.stdout: create 144/39 file after the snapshot 2019-12-05T06:09:20.899 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/.snap/snap-subdir-test 2019-12-05T06:09:20.942 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/30 file after the snapshot 2019-12-05T06:09:20.947 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/31 file after the snapshot 2019-12-05T06:09:20.951 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/32 file after the snapshot 2019-12-05T06:09:20.966 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/33 file after the snapshot 2019-12-05T06:09:21.035 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/34 file after the snapshot 2019-12-05T06:09:21.050 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/35 file after the snapshot 2019-12-05T06:09:21.051 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/36 file after the snapshot 2019-12-05T06:09:21.051 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/37 file after the snapshot 2019-12-05T06:09:21.054 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/38 file after the snapshot 2019-12-05T06:09:21.058 INFO:tasks.workunit.client.0.smithi016.stdout: create 145/39 file after the snapshot 2019-12-05T06:09:21.063 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/.snap/snap-subdir-test 2019-12-05T06:09:21.113 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/30 file after the snapshot 2019-12-05T06:09:21.117 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/31 file after the snapshot 2019-12-05T06:09:21.120 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/32 file after the snapshot 2019-12-05T06:09:21.123 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/33 file after the snapshot 2019-12-05T06:09:21.128 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/34 file after the snapshot 2019-12-05T06:09:21.131 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/35 file after the snapshot 2019-12-05T06:09:21.135 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/36 file after the snapshot 2019-12-05T06:09:21.138 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/37 file after the snapshot 2019-12-05T06:09:21.143 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/38 file after the snapshot 2019-12-05T06:09:21.145 INFO:tasks.workunit.client.0.smithi016.stdout: create 146/39 file after the snapshot 2019-12-05T06:09:21.148 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/.snap/snap-subdir-test 2019-12-05T06:09:21.175 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/30 file after the snapshot 2019-12-05T06:09:21.176 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/31 file after the snapshot 2019-12-05T06:09:21.179 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/32 file after the snapshot 2019-12-05T06:09:21.181 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/33 file after the snapshot 2019-12-05T06:09:21.184 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/34 file after the snapshot 2019-12-05T06:09:21.192 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/35 file after the snapshot 2019-12-05T06:09:21.193 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/36 file after the snapshot 2019-12-05T06:09:21.195 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/37 file after the snapshot 2019-12-05T06:09:21.198 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/38 file after the snapshot 2019-12-05T06:09:21.201 INFO:tasks.workunit.client.0.smithi016.stdout: create 147/39 file after the snapshot 2019-12-05T06:09:21.205 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/.snap/snap-subdir-test 2019-12-05T06:09:21.232 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/30 file after the snapshot 2019-12-05T06:09:21.235 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/31 file after the snapshot 2019-12-05T06:09:21.238 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/32 file after the snapshot 2019-12-05T06:09:21.244 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/33 file after the snapshot 2019-12-05T06:09:21.245 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/34 file after the snapshot 2019-12-05T06:09:21.247 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/35 file after the snapshot 2019-12-05T06:09:21.250 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/36 file after the snapshot 2019-12-05T06:09:21.253 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/37 file after the snapshot 2019-12-05T06:09:21.256 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/38 file after the snapshot 2019-12-05T06:09:21.258 INFO:tasks.workunit.client.0.smithi016.stdout: create 148/39 file after the snapshot 2019-12-05T06:09:21.261 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/.snap/snap-subdir-test 2019-12-05T06:09:21.285 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/30 file after the snapshot 2019-12-05T06:09:21.291 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/31 file after the snapshot 2019-12-05T06:09:21.291 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/32 file after the snapshot 2019-12-05T06:09:21.297 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/33 file after the snapshot 2019-12-05T06:09:21.297 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/34 file after the snapshot 2019-12-05T06:09:21.298 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/35 file after the snapshot 2019-12-05T06:09:21.301 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/36 file after the snapshot 2019-12-05T06:09:21.306 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/37 file after the snapshot 2019-12-05T06:09:21.307 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/38 file after the snapshot 2019-12-05T06:09:21.308 INFO:tasks.workunit.client.0.smithi016.stdout: create 149/39 file after the snapshot 2019-12-05T06:09:21.310 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/.snap/snap-subdir-test 2019-12-05T06:09:21.334 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/30 file after the snapshot 2019-12-05T06:09:21.336 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/31 file after the snapshot 2019-12-05T06:09:21.338 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/32 file after the snapshot 2019-12-05T06:09:21.341 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/33 file after the snapshot 2019-12-05T06:09:21.343 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/34 file after the snapshot 2019-12-05T06:09:21.345 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/35 file after the snapshot 2019-12-05T06:09:21.347 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/36 file after the snapshot 2019-12-05T06:09:21.352 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/37 file after the snapshot 2019-12-05T06:09:21.352 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/38 file after the snapshot 2019-12-05T06:09:21.355 INFO:tasks.workunit.client.0.smithi016.stdout: create 150/39 file after the snapshot 2019-12-05T06:09:21.357 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/.snap/snap-subdir-test 2019-12-05T06:09:21.386 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/30 file after the snapshot 2019-12-05T06:09:21.386 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/31 file after the snapshot 2019-12-05T06:09:21.386 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/32 file after the snapshot 2019-12-05T06:09:21.396 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/33 file after the snapshot 2019-12-05T06:09:21.397 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/34 file after the snapshot 2019-12-05T06:09:21.397 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/35 file after the snapshot 2019-12-05T06:09:21.397 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/36 file after the snapshot 2019-12-05T06:09:21.397 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/37 file after the snapshot 2019-12-05T06:09:21.406 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/38 file after the snapshot 2019-12-05T06:09:21.406 INFO:tasks.workunit.client.0.smithi016.stdout: create 151/39 file after the snapshot 2019-12-05T06:09:21.407 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/.snap/snap-subdir-test 2019-12-05T06:09:21.427 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/30 file after the snapshot 2019-12-05T06:09:21.429 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/31 file after the snapshot 2019-12-05T06:09:21.431 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/32 file after the snapshot 2019-12-05T06:09:21.434 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/33 file after the snapshot 2019-12-05T06:09:21.436 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/34 file after the snapshot 2019-12-05T06:09:21.439 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/35 file after the snapshot 2019-12-05T06:09:21.441 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/36 file after the snapshot 2019-12-05T06:09:21.445 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/37 file after the snapshot 2019-12-05T06:09:21.445 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/38 file after the snapshot 2019-12-05T06:09:21.447 INFO:tasks.workunit.client.0.smithi016.stdout: create 152/39 file after the snapshot 2019-12-05T06:09:21.450 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/.snap/snap-subdir-test 2019-12-05T06:09:21.472 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/30 file after the snapshot 2019-12-05T06:09:21.474 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/31 file after the snapshot 2019-12-05T06:09:21.476 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/32 file after the snapshot 2019-12-05T06:09:21.479 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/33 file after the snapshot 2019-12-05T06:09:21.481 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/34 file after the snapshot 2019-12-05T06:09:21.483 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/35 file after the snapshot 2019-12-05T06:09:21.485 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/36 file after the snapshot 2019-12-05T06:09:21.488 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/37 file after the snapshot 2019-12-05T06:09:21.490 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/38 file after the snapshot 2019-12-05T06:09:21.492 INFO:tasks.workunit.client.0.smithi016.stdout: create 153/39 file after the snapshot 2019-12-05T06:09:21.495 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/.snap/snap-subdir-test 2019-12-05T06:09:21.516 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/30 file after the snapshot 2019-12-05T06:09:21.519 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/31 file after the snapshot 2019-12-05T06:09:21.521 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/32 file after the snapshot 2019-12-05T06:09:21.525 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/33 file after the snapshot 2019-12-05T06:09:21.527 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/34 file after the snapshot 2019-12-05T06:09:21.530 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/35 file after the snapshot 2019-12-05T06:09:21.533 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/36 file after the snapshot 2019-12-05T06:09:21.536 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/37 file after the snapshot 2019-12-05T06:09:21.538 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/38 file after the snapshot 2019-12-05T06:09:21.541 INFO:tasks.workunit.client.0.smithi016.stdout: create 154/39 file after the snapshot 2019-12-05T06:09:21.543 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/.snap/snap-subdir-test 2019-12-05T06:09:21.566 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/30 file after the snapshot 2019-12-05T06:09:21.568 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/31 file after the snapshot 2019-12-05T06:09:21.570 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/32 file after the snapshot 2019-12-05T06:09:21.573 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/33 file after the snapshot 2019-12-05T06:09:21.577 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/34 file after the snapshot 2019-12-05T06:09:21.588 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/35 file after the snapshot 2019-12-05T06:09:21.588 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/36 file after the snapshot 2019-12-05T06:09:21.588 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/37 file after the snapshot 2019-12-05T06:09:21.588 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/38 file after the snapshot 2019-12-05T06:09:21.589 INFO:tasks.workunit.client.0.smithi016.stdout: create 155/39 file after the snapshot 2019-12-05T06:09:21.591 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/.snap/snap-subdir-test 2019-12-05T06:09:22.434 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/30 file after the snapshot 2019-12-05T06:09:22.438 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/31 file after the snapshot 2019-12-05T06:09:22.443 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/32 file after the snapshot 2019-12-05T06:09:22.449 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/33 file after the snapshot 2019-12-05T06:09:22.455 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/34 file after the snapshot 2019-12-05T06:09:22.460 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/35 file after the snapshot 2019-12-05T06:09:22.465 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/36 file after the snapshot 2019-12-05T06:09:22.471 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/37 file after the snapshot 2019-12-05T06:09:22.475 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/38 file after the snapshot 2019-12-05T06:09:22.480 INFO:tasks.workunit.client.0.smithi016.stdout: create 156/39 file after the snapshot 2019-12-05T06:09:22.489 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/.snap/snap-subdir-test 2019-12-05T06:09:22.520 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/30 file after the snapshot 2019-12-05T06:09:22.524 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/31 file after the snapshot 2019-12-05T06:09:22.529 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/32 file after the snapshot 2019-12-05T06:09:22.534 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/33 file after the snapshot 2019-12-05T06:09:22.544 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/34 file after the snapshot 2019-12-05T06:09:22.544 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/35 file after the snapshot 2019-12-05T06:09:22.546 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/36 file after the snapshot 2019-12-05T06:09:22.550 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/37 file after the snapshot 2019-12-05T06:09:22.555 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/38 file after the snapshot 2019-12-05T06:09:22.558 INFO:tasks.workunit.client.0.smithi016.stdout: create 157/39 file after the snapshot 2019-12-05T06:09:22.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/.snap/snap-subdir-test 2019-12-05T06:09:22.885 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/30 file after the snapshot 2019-12-05T06:09:22.890 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/31 file after the snapshot 2019-12-05T06:09:22.895 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/32 file after the snapshot 2019-12-05T06:09:22.901 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/33 file after the snapshot 2019-12-05T06:09:22.905 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/34 file after the snapshot 2019-12-05T06:09:23.701 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/35 file after the snapshot 2019-12-05T06:09:23.709 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/36 file after the snapshot 2019-12-05T06:09:23.713 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/37 file after the snapshot 2019-12-05T06:09:23.721 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/38 file after the snapshot 2019-12-05T06:09:23.724 INFO:tasks.workunit.client.0.smithi016.stdout: create 158/39 file after the snapshot 2019-12-05T06:09:23.730 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/.snap/snap-subdir-test 2019-12-05T06:09:23.796 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/30 file after the snapshot 2019-12-05T06:09:23.801 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/31 file after the snapshot 2019-12-05T06:09:23.806 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/32 file after the snapshot 2019-12-05T06:09:23.812 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/33 file after the snapshot 2019-12-05T06:09:23.817 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/34 file after the snapshot 2019-12-05T06:09:23.822 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/35 file after the snapshot 2019-12-05T06:09:23.827 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/36 file after the snapshot 2019-12-05T06:09:23.831 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/37 file after the snapshot 2019-12-05T06:09:23.837 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/38 file after the snapshot 2019-12-05T06:09:23.840 INFO:tasks.workunit.client.0.smithi016.stdout: create 159/39 file after the snapshot 2019-12-05T06:09:23.844 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/.snap/snap-subdir-test 2019-12-05T06:09:23.873 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/30 file after the snapshot 2019-12-05T06:09:23.877 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/31 file after the snapshot 2019-12-05T06:09:23.881 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/32 file after the snapshot 2019-12-05T06:09:23.885 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/33 file after the snapshot 2019-12-05T06:09:23.890 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/34 file after the snapshot 2019-12-05T06:09:23.892 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/35 file after the snapshot 2019-12-05T06:09:23.897 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/36 file after the snapshot 2019-12-05T06:09:23.902 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/37 file after the snapshot 2019-12-05T06:09:23.904 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/38 file after the snapshot 2019-12-05T06:09:23.907 INFO:tasks.workunit.client.0.smithi016.stdout: create 160/39 file after the snapshot 2019-12-05T06:09:23.913 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/.snap/snap-subdir-test 2019-12-05T06:09:23.940 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/30 file after the snapshot 2019-12-05T06:09:23.943 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/31 file after the snapshot 2019-12-05T06:09:23.946 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/32 file after the snapshot 2019-12-05T06:09:23.950 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/33 file after the snapshot 2019-12-05T06:09:23.955 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/34 file after the snapshot 2019-12-05T06:09:23.964 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/35 file after the snapshot 2019-12-05T06:09:23.983 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/36 file after the snapshot 2019-12-05T06:09:23.987 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/37 file after the snapshot 2019-12-05T06:09:23.988 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/38 file after the snapshot 2019-12-05T06:09:23.991 INFO:tasks.workunit.client.0.smithi016.stdout: create 161/39 file after the snapshot 2019-12-05T06:09:23.993 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/.snap/snap-subdir-test 2019-12-05T06:09:24.098 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/30 file after the snapshot 2019-12-05T06:09:24.099 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/31 file after the snapshot 2019-12-05T06:09:24.108 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/32 file after the snapshot 2019-12-05T06:09:24.109 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/33 file after the snapshot 2019-12-05T06:09:24.120 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/34 file after the snapshot 2019-12-05T06:09:24.120 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/35 file after the snapshot 2019-12-05T06:09:24.125 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/36 file after the snapshot 2019-12-05T06:09:24.129 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/37 file after the snapshot 2019-12-05T06:09:24.133 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/38 file after the snapshot 2019-12-05T06:09:24.139 INFO:tasks.workunit.client.0.smithi016.stdout: create 162/39 file after the snapshot 2019-12-05T06:09:24.141 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/.snap/snap-subdir-test 2019-12-05T06:09:24.412 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/30 file after the snapshot 2019-12-05T06:09:24.416 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/31 file after the snapshot 2019-12-05T06:09:24.421 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/32 file after the snapshot 2019-12-05T06:09:24.426 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/33 file after the snapshot 2019-12-05T06:09:24.431 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/34 file after the snapshot 2019-12-05T06:09:24.438 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/35 file after the snapshot 2019-12-05T06:09:24.440 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/36 file after the snapshot 2019-12-05T06:09:24.453 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/37 file after the snapshot 2019-12-05T06:09:24.453 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/38 file after the snapshot 2019-12-05T06:09:24.454 INFO:tasks.workunit.client.0.smithi016.stdout: create 163/39 file after the snapshot 2019-12-05T06:09:24.458 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/.snap/snap-subdir-test 2019-12-05T06:09:24.493 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/30 file after the snapshot 2019-12-05T06:09:24.497 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/31 file after the snapshot 2019-12-05T06:09:24.501 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/32 file after the snapshot 2019-12-05T06:09:24.506 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/33 file after the snapshot 2019-12-05T06:09:24.510 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/34 file after the snapshot 2019-12-05T06:09:24.513 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/35 file after the snapshot 2019-12-05T06:09:24.517 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/36 file after the snapshot 2019-12-05T06:09:24.520 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/37 file after the snapshot 2019-12-05T06:09:24.526 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/38 file after the snapshot 2019-12-05T06:09:24.528 INFO:tasks.workunit.client.0.smithi016.stdout: create 164/39 file after the snapshot 2019-12-05T06:09:24.533 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/.snap/snap-subdir-test 2019-12-05T06:09:24.565 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/30 file after the snapshot 2019-12-05T06:09:24.565 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/31 file after the snapshot 2019-12-05T06:09:24.573 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/32 file after the snapshot 2019-12-05T06:09:24.573 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/33 file after the snapshot 2019-12-05T06:09:24.577 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/34 file after the snapshot 2019-12-05T06:09:24.578 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/35 file after the snapshot 2019-12-05T06:09:24.581 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/36 file after the snapshot 2019-12-05T06:09:24.584 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/37 file after the snapshot 2019-12-05T06:09:24.589 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/38 file after the snapshot 2019-12-05T06:09:24.592 INFO:tasks.workunit.client.0.smithi016.stdout: create 165/39 file after the snapshot 2019-12-05T06:09:24.595 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/.snap/snap-subdir-test 2019-12-05T06:09:24.627 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/30 file after the snapshot 2019-12-05T06:09:24.629 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/31 file after the snapshot 2019-12-05T06:09:24.633 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/32 file after the snapshot 2019-12-05T06:09:24.635 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/33 file after the snapshot 2019-12-05T06:09:24.637 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/34 file after the snapshot 2019-12-05T06:09:24.642 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/35 file after the snapshot 2019-12-05T06:09:24.643 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/36 file after the snapshot 2019-12-05T06:09:24.646 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/37 file after the snapshot 2019-12-05T06:09:24.648 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/38 file after the snapshot 2019-12-05T06:09:24.651 INFO:tasks.workunit.client.0.smithi016.stdout: create 166/39 file after the snapshot 2019-12-05T06:09:24.655 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/.snap/snap-subdir-test 2019-12-05T06:09:24.676 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/30 file after the snapshot 2019-12-05T06:09:24.679 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/31 file after the snapshot 2019-12-05T06:09:24.680 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/32 file after the snapshot 2019-12-05T06:09:24.683 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/33 file after the snapshot 2019-12-05T06:09:24.686 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/34 file after the snapshot 2019-12-05T06:09:24.688 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/35 file after the snapshot 2019-12-05T06:09:24.691 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/36 file after the snapshot 2019-12-05T06:09:24.693 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/37 file after the snapshot 2019-12-05T06:09:24.696 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/38 file after the snapshot 2019-12-05T06:09:24.699 INFO:tasks.workunit.client.0.smithi016.stdout: create 167/39 file after the snapshot 2019-12-05T06:09:24.701 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/.snap/snap-subdir-test 2019-12-05T06:09:24.741 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/30 file after the snapshot 2019-12-05T06:09:24.741 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/31 file after the snapshot 2019-12-05T06:09:24.742 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/32 file after the snapshot 2019-12-05T06:09:24.742 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/33 file after the snapshot 2019-12-05T06:09:24.742 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/34 file after the snapshot 2019-12-05T06:09:24.743 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/35 file after the snapshot 2019-12-05T06:09:24.744 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/36 file after the snapshot 2019-12-05T06:09:24.744 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/37 file after the snapshot 2019-12-05T06:09:24.745 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/38 file after the snapshot 2019-12-05T06:09:24.745 INFO:tasks.workunit.client.0.smithi016.stdout: create 168/39 file after the snapshot 2019-12-05T06:09:24.748 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/.snap/snap-subdir-test 2019-12-05T06:09:24.768 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/30 file after the snapshot 2019-12-05T06:09:24.770 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/31 file after the snapshot 2019-12-05T06:09:24.772 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/32 file after the snapshot 2019-12-05T06:09:24.775 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/33 file after the snapshot 2019-12-05T06:09:24.778 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/34 file after the snapshot 2019-12-05T06:09:24.780 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/35 file after the snapshot 2019-12-05T06:09:24.783 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/36 file after the snapshot 2019-12-05T06:09:24.785 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/37 file after the snapshot 2019-12-05T06:09:24.788 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/38 file after the snapshot 2019-12-05T06:09:24.791 INFO:tasks.workunit.client.0.smithi016.stdout: create 169/39 file after the snapshot 2019-12-05T06:09:24.793 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/.snap/snap-subdir-test 2019-12-05T06:09:24.815 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/30 file after the snapshot 2019-12-05T06:09:24.817 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/31 file after the snapshot 2019-12-05T06:09:24.819 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/32 file after the snapshot 2019-12-05T06:09:24.822 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/33 file after the snapshot 2019-12-05T06:09:24.824 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/34 file after the snapshot 2019-12-05T06:09:24.826 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/35 file after the snapshot 2019-12-05T06:09:24.829 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/36 file after the snapshot 2019-12-05T06:09:24.831 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/37 file after the snapshot 2019-12-05T06:09:24.834 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/38 file after the snapshot 2019-12-05T06:09:24.836 INFO:tasks.workunit.client.0.smithi016.stdout: create 170/39 file after the snapshot 2019-12-05T06:09:24.839 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/.snap/snap-subdir-test 2019-12-05T06:09:24.860 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/30 file after the snapshot 2019-12-05T06:09:24.862 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/31 file after the snapshot 2019-12-05T06:09:24.865 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/32 file after the snapshot 2019-12-05T06:09:24.867 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/33 file after the snapshot 2019-12-05T06:09:24.869 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/34 file after the snapshot 2019-12-05T06:09:24.872 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/35 file after the snapshot 2019-12-05T06:09:24.874 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/36 file after the snapshot 2019-12-05T06:09:24.876 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/37 file after the snapshot 2019-12-05T06:09:24.879 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/38 file after the snapshot 2019-12-05T06:09:24.881 INFO:tasks.workunit.client.0.smithi016.stdout: create 171/39 file after the snapshot 2019-12-05T06:09:24.884 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/.snap/snap-subdir-test 2019-12-05T06:09:24.905 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/30 file after the snapshot 2019-12-05T06:09:24.908 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/31 file after the snapshot 2019-12-05T06:09:24.909 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/32 file after the snapshot 2019-12-05T06:09:24.912 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/33 file after the snapshot 2019-12-05T06:09:24.914 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/34 file after the snapshot 2019-12-05T06:09:24.916 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/35 file after the snapshot 2019-12-05T06:09:24.925 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/36 file after the snapshot 2019-12-05T06:09:24.926 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/37 file after the snapshot 2019-12-05T06:09:24.926 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/38 file after the snapshot 2019-12-05T06:09:24.926 INFO:tasks.workunit.client.0.smithi016.stdout: create 172/39 file after the snapshot 2019-12-05T06:09:24.929 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/.snap/snap-subdir-test 2019-12-05T06:09:24.949 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/30 file after the snapshot 2019-12-05T06:09:24.951 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/31 file after the snapshot 2019-12-05T06:09:24.954 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/32 file after the snapshot 2019-12-05T06:09:24.958 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/33 file after the snapshot 2019-12-05T06:09:24.975 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/34 file after the snapshot 2019-12-05T06:09:24.977 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/35 file after the snapshot 2019-12-05T06:09:24.979 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/36 file after the snapshot 2019-12-05T06:09:24.982 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/37 file after the snapshot 2019-12-05T06:09:24.984 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/38 file after the snapshot 2019-12-05T06:09:24.986 INFO:tasks.workunit.client.0.smithi016.stdout: create 173/39 file after the snapshot 2019-12-05T06:09:24.989 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/.snap/snap-subdir-test 2019-12-05T06:09:25.024 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/30 file after the snapshot 2019-12-05T06:09:25.024 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/31 file after the snapshot 2019-12-05T06:09:25.025 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/32 file after the snapshot 2019-12-05T06:09:25.026 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/33 file after the snapshot 2019-12-05T06:09:25.028 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/34 file after the snapshot 2019-12-05T06:09:25.030 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/35 file after the snapshot 2019-12-05T06:09:25.033 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/36 file after the snapshot 2019-12-05T06:09:25.035 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/37 file after the snapshot 2019-12-05T06:09:25.037 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/38 file after the snapshot 2019-12-05T06:09:25.040 INFO:tasks.workunit.client.0.smithi016.stdout: create 174/39 file after the snapshot 2019-12-05T06:09:25.043 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/.snap/snap-subdir-test 2019-12-05T06:09:25.063 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/30 file after the snapshot 2019-12-05T06:09:25.065 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/31 file after the snapshot 2019-12-05T06:09:25.068 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/32 file after the snapshot 2019-12-05T06:09:25.069 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/33 file after the snapshot 2019-12-05T06:09:25.071 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/34 file after the snapshot 2019-12-05T06:09:25.074 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/35 file after the snapshot 2019-12-05T06:09:25.076 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/36 file after the snapshot 2019-12-05T06:09:25.079 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/37 file after the snapshot 2019-12-05T06:09:25.082 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/38 file after the snapshot 2019-12-05T06:09:25.084 INFO:tasks.workunit.client.0.smithi016.stdout: create 175/39 file after the snapshot 2019-12-05T06:09:25.086 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/.snap/snap-subdir-test 2019-12-05T06:09:25.419 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/30 file after the snapshot 2019-12-05T06:09:25.428 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/31 file after the snapshot 2019-12-05T06:09:25.437 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/32 file after the snapshot 2019-12-05T06:09:25.437 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/33 file after the snapshot 2019-12-05T06:09:25.440 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/34 file after the snapshot 2019-12-05T06:09:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/35 file after the snapshot 2019-12-05T06:09:25.451 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/36 file after the snapshot 2019-12-05T06:09:25.456 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/37 file after the snapshot 2019-12-05T06:09:25.459 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/38 file after the snapshot 2019-12-05T06:09:25.466 INFO:tasks.workunit.client.0.smithi016.stdout: create 176/39 file after the snapshot 2019-12-05T06:09:25.467 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/.snap/snap-subdir-test 2019-12-05T06:09:25.500 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/30 file after the snapshot 2019-12-05T06:09:25.504 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/31 file after the snapshot 2019-12-05T06:09:25.507 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/32 file after the snapshot 2019-12-05T06:09:25.511 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/33 file after the snapshot 2019-12-05T06:09:25.515 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/34 file after the snapshot 2019-12-05T06:09:25.518 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/35 file after the snapshot 2019-12-05T06:09:25.522 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/36 file after the snapshot 2019-12-05T06:09:25.525 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/37 file after the snapshot 2019-12-05T06:09:25.529 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/38 file after the snapshot 2019-12-05T06:09:25.532 INFO:tasks.workunit.client.0.smithi016.stdout: create 177/39 file after the snapshot 2019-12-05T06:09:25.537 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/.snap/snap-subdir-test 2019-12-05T06:09:25.569 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/30 file after the snapshot 2019-12-05T06:09:25.570 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/31 file after the snapshot 2019-12-05T06:09:25.570 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/32 file after the snapshot 2019-12-05T06:09:25.574 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/33 file after the snapshot 2019-12-05T06:09:25.577 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/34 file after the snapshot 2019-12-05T06:09:25.580 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/35 file after the snapshot 2019-12-05T06:09:25.583 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/36 file after the snapshot 2019-12-05T06:09:25.586 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/37 file after the snapshot 2019-12-05T06:09:25.589 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/38 file after the snapshot 2019-12-05T06:09:25.593 INFO:tasks.workunit.client.0.smithi016.stdout: create 178/39 file after the snapshot 2019-12-05T06:09:25.595 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/.snap/snap-subdir-test 2019-12-05T06:09:25.618 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/30 file after the snapshot 2019-12-05T06:09:25.621 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/31 file after the snapshot 2019-12-05T06:09:25.623 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/32 file after the snapshot 2019-12-05T06:09:25.626 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/33 file after the snapshot 2019-12-05T06:09:25.629 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/34 file after the snapshot 2019-12-05T06:09:25.632 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/35 file after the snapshot 2019-12-05T06:09:25.634 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/36 file after the snapshot 2019-12-05T06:09:25.637 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/37 file after the snapshot 2019-12-05T06:09:25.639 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/38 file after the snapshot 2019-12-05T06:09:25.642 INFO:tasks.workunit.client.0.smithi016.stdout: create 179/39 file after the snapshot 2019-12-05T06:09:25.656 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/.snap/snap-subdir-test 2019-12-05T06:09:25.667 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/30 file after the snapshot 2019-12-05T06:09:25.670 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/31 file after the snapshot 2019-12-05T06:09:25.672 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/32 file after the snapshot 2019-12-05T06:09:25.674 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/33 file after the snapshot 2019-12-05T06:09:25.677 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/34 file after the snapshot 2019-12-05T06:09:25.679 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/35 file after the snapshot 2019-12-05T06:09:25.682 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/36 file after the snapshot 2019-12-05T06:09:25.685 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/37 file after the snapshot 2019-12-05T06:09:25.687 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/38 file after the snapshot 2019-12-05T06:09:25.690 INFO:tasks.workunit.client.0.smithi016.stdout: create 180/39 file after the snapshot 2019-12-05T06:09:25.692 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/.snap/snap-subdir-test 2019-12-05T06:09:25.715 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/30 file after the snapshot 2019-12-05T06:09:25.717 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/31 file after the snapshot 2019-12-05T06:09:25.720 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/32 file after the snapshot 2019-12-05T06:09:25.722 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/33 file after the snapshot 2019-12-05T06:09:25.724 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/34 file after the snapshot 2019-12-05T06:09:25.726 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/35 file after the snapshot 2019-12-05T06:09:25.729 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/36 file after the snapshot 2019-12-05T06:09:25.732 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/37 file after the snapshot 2019-12-05T06:09:25.734 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/38 file after the snapshot 2019-12-05T06:09:25.737 INFO:tasks.workunit.client.0.smithi016.stdout: create 181/39 file after the snapshot 2019-12-05T06:09:25.739 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/.snap/snap-subdir-test 2019-12-05T06:09:25.760 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/30 file after the snapshot 2019-12-05T06:09:25.762 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/31 file after the snapshot 2019-12-05T06:09:25.765 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/32 file after the snapshot 2019-12-05T06:09:25.767 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/33 file after the snapshot 2019-12-05T06:09:25.770 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/34 file after the snapshot 2019-12-05T06:09:25.772 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/35 file after the snapshot 2019-12-05T06:09:25.777 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/36 file after the snapshot 2019-12-05T06:09:25.779 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/37 file after the snapshot 2019-12-05T06:09:25.781 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/38 file after the snapshot 2019-12-05T06:09:25.784 INFO:tasks.workunit.client.0.smithi016.stdout: create 182/39 file after the snapshot 2019-12-05T06:09:25.786 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/.snap/snap-subdir-test 2019-12-05T06:09:26.524 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/30 file after the snapshot 2019-12-05T06:09:26.767 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/31 file after the snapshot 2019-12-05T06:09:26.772 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/32 file after the snapshot 2019-12-05T06:09:26.779 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/33 file after the snapshot 2019-12-05T06:09:26.784 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/34 file after the snapshot 2019-12-05T06:09:26.791 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/35 file after the snapshot 2019-12-05T06:09:26.797 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/36 file after the snapshot 2019-12-05T06:09:26.802 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/37 file after the snapshot 2019-12-05T06:09:26.806 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/38 file after the snapshot 2019-12-05T06:09:26.811 INFO:tasks.workunit.client.0.smithi016.stdout: create 183/39 file after the snapshot 2019-12-05T06:09:26.815 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/.snap/snap-subdir-test 2019-12-05T06:09:26.877 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/30 file after the snapshot 2019-12-05T06:09:26.880 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/31 file after the snapshot 2019-12-05T06:09:26.884 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/32 file after the snapshot 2019-12-05T06:09:26.887 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/33 file after the snapshot 2019-12-05T06:09:26.892 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/34 file after the snapshot 2019-12-05T06:09:26.896 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/35 file after the snapshot 2019-12-05T06:09:26.904 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/36 file after the snapshot 2019-12-05T06:09:26.904 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/37 file after the snapshot 2019-12-05T06:09:26.907 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/38 file after the snapshot 2019-12-05T06:09:26.911 INFO:tasks.workunit.client.0.smithi016.stdout: create 184/39 file after the snapshot 2019-12-05T06:09:26.915 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/.snap/snap-subdir-test 2019-12-05T06:09:26.943 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/30 file after the snapshot 2019-12-05T06:09:26.957 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/31 file after the snapshot 2019-12-05T06:09:26.957 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/32 file after the snapshot 2019-12-05T06:09:26.957 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/33 file after the snapshot 2019-12-05T06:09:26.958 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/34 file after the snapshot 2019-12-05T06:09:27.007 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/35 file after the snapshot 2019-12-05T06:09:27.010 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/36 file after the snapshot 2019-12-05T06:09:27.013 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/37 file after the snapshot 2019-12-05T06:09:27.017 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/38 file after the snapshot 2019-12-05T06:09:27.020 INFO:tasks.workunit.client.0.smithi016.stdout: create 185/39 file after the snapshot 2019-12-05T06:09:27.023 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/.snap/snap-subdir-test 2019-12-05T06:09:27.072 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/30 file after the snapshot 2019-12-05T06:09:27.075 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/31 file after the snapshot 2019-12-05T06:09:27.079 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/32 file after the snapshot 2019-12-05T06:09:27.081 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/33 file after the snapshot 2019-12-05T06:09:27.085 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/34 file after the snapshot 2019-12-05T06:09:27.087 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/35 file after the snapshot 2019-12-05T06:09:27.090 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/36 file after the snapshot 2019-12-05T06:09:27.092 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/37 file after the snapshot 2019-12-05T06:09:27.096 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/38 file after the snapshot 2019-12-05T06:09:27.099 INFO:tasks.workunit.client.0.smithi016.stdout: create 186/39 file after the snapshot 2019-12-05T06:09:27.101 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/.snap/snap-subdir-test 2019-12-05T06:09:27.794 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/30 file after the snapshot 2019-12-05T06:09:27.799 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/31 file after the snapshot 2019-12-05T06:09:27.805 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/32 file after the snapshot 2019-12-05T06:09:27.810 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/33 file after the snapshot 2019-12-05T06:09:27.815 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/34 file after the snapshot 2019-12-05T06:09:27.820 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/35 file after the snapshot 2019-12-05T06:09:27.825 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/36 file after the snapshot 2019-12-05T06:09:27.831 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/37 file after the snapshot 2019-12-05T06:09:27.842 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/38 file after the snapshot 2019-12-05T06:09:27.842 INFO:tasks.workunit.client.0.smithi016.stdout: create 187/39 file after the snapshot 2019-12-05T06:09:27.843 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/.snap/snap-subdir-test 2019-12-05T06:09:27.881 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/30 file after the snapshot 2019-12-05T06:09:27.883 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/31 file after the snapshot 2019-12-05T06:09:27.886 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/32 file after the snapshot 2019-12-05T06:09:27.890 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/33 file after the snapshot 2019-12-05T06:09:27.894 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/34 file after the snapshot 2019-12-05T06:09:27.898 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/35 file after the snapshot 2019-12-05T06:09:27.901 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/36 file after the snapshot 2019-12-05T06:09:27.905 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/37 file after the snapshot 2019-12-05T06:09:27.908 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/38 file after the snapshot 2019-12-05T06:09:27.912 INFO:tasks.workunit.client.0.smithi016.stdout: create 188/39 file after the snapshot 2019-12-05T06:09:27.918 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/.snap/snap-subdir-test 2019-12-05T06:09:27.943 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/30 file after the snapshot 2019-12-05T06:09:27.947 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/31 file after the snapshot 2019-12-05T06:09:27.949 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/32 file after the snapshot 2019-12-05T06:09:27.952 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/33 file after the snapshot 2019-12-05T06:09:27.955 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/34 file after the snapshot 2019-12-05T06:09:27.971 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/35 file after the snapshot 2019-12-05T06:09:27.972 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/36 file after the snapshot 2019-12-05T06:09:27.975 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/37 file after the snapshot 2019-12-05T06:09:27.981 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/38 file after the snapshot 2019-12-05T06:09:27.981 INFO:tasks.workunit.client.0.smithi016.stdout: create 189/39 file after the snapshot 2019-12-05T06:09:27.984 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/.snap/snap-subdir-test 2019-12-05T06:09:28.229 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:09:28.236 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:09:28.962 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/30 file after the snapshot 2019-12-05T06:09:28.997 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/31 file after the snapshot 2019-12-05T06:09:29.002 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/32 file after the snapshot 2019-12-05T06:09:29.009 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/33 file after the snapshot 2019-12-05T06:09:29.017 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/34 file after the snapshot 2019-12-05T06:09:29.031 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/35 file after the snapshot 2019-12-05T06:09:29.038 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/36 file after the snapshot 2019-12-05T06:09:29.042 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/37 file after the snapshot 2019-12-05T06:09:29.046 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/38 file after the snapshot 2019-12-05T06:09:29.050 INFO:tasks.workunit.client.0.smithi016.stdout: create 190/39 file after the snapshot 2019-12-05T06:09:29.054 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/.snap/snap-subdir-test 2019-12-05T06:09:29.081 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/30 file after the snapshot 2019-12-05T06:09:29.084 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/31 file after the snapshot 2019-12-05T06:09:29.088 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/32 file after the snapshot 2019-12-05T06:09:29.093 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/33 file after the snapshot 2019-12-05T06:09:29.097 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/34 file after the snapshot 2019-12-05T06:09:29.101 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/35 file after the snapshot 2019-12-05T06:09:29.104 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/36 file after the snapshot 2019-12-05T06:09:29.107 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/37 file after the snapshot 2019-12-05T06:09:29.110 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/38 file after the snapshot 2019-12-05T06:09:29.113 INFO:tasks.workunit.client.0.smithi016.stdout: create 191/39 file after the snapshot 2019-12-05T06:09:29.117 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/.snap/snap-subdir-test 2019-12-05T06:09:29.149 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/30 file after the snapshot 2019-12-05T06:09:29.149 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/31 file after the snapshot 2019-12-05T06:09:29.150 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/32 file after the snapshot 2019-12-05T06:09:29.152 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/33 file after the snapshot 2019-12-05T06:09:29.157 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/34 file after the snapshot 2019-12-05T06:09:29.158 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/35 file after the snapshot 2019-12-05T06:09:29.162 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/36 file after the snapshot 2019-12-05T06:09:29.164 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/37 file after the snapshot 2019-12-05T06:09:29.168 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/38 file after the snapshot 2019-12-05T06:09:29.171 INFO:tasks.workunit.client.0.smithi016.stdout: create 192/39 file after the snapshot 2019-12-05T06:09:29.174 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/.snap/snap-subdir-test 2019-12-05T06:09:29.197 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/30 file after the snapshot 2019-12-05T06:09:29.200 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/31 file after the snapshot 2019-12-05T06:09:29.202 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/32 file after the snapshot 2019-12-05T06:09:29.205 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/33 file after the snapshot 2019-12-05T06:09:29.208 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/34 file after the snapshot 2019-12-05T06:09:29.210 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/35 file after the snapshot 2019-12-05T06:09:29.212 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/36 file after the snapshot 2019-12-05T06:09:29.215 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/37 file after the snapshot 2019-12-05T06:09:29.217 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/38 file after the snapshot 2019-12-05T06:09:29.222 INFO:tasks.workunit.client.0.smithi016.stdout: create 193/39 file after the snapshot 2019-12-05T06:09:29.226 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/.snap/snap-subdir-test 2019-12-05T06:09:29.249 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/30 file after the snapshot 2019-12-05T06:09:29.251 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/31 file after the snapshot 2019-12-05T06:09:29.253 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/32 file after the snapshot 2019-12-05T06:09:29.255 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/33 file after the snapshot 2019-12-05T06:09:29.258 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/34 file after the snapshot 2019-12-05T06:09:29.261 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/35 file after the snapshot 2019-12-05T06:09:29.263 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/36 file after the snapshot 2019-12-05T06:09:29.266 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/37 file after the snapshot 2019-12-05T06:09:29.268 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/38 file after the snapshot 2019-12-05T06:09:29.271 INFO:tasks.workunit.client.0.smithi016.stdout: create 194/39 file after the snapshot 2019-12-05T06:09:29.273 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/.snap/snap-subdir-test 2019-12-05T06:09:29.295 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/30 file after the snapshot 2019-12-05T06:09:29.296 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/31 file after the snapshot 2019-12-05T06:09:29.303 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/32 file after the snapshot 2019-12-05T06:09:29.303 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/33 file after the snapshot 2019-12-05T06:09:29.305 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/34 file after the snapshot 2019-12-05T06:09:29.306 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/35 file after the snapshot 2019-12-05T06:09:29.309 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/36 file after the snapshot 2019-12-05T06:09:29.312 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/37 file after the snapshot 2019-12-05T06:09:29.315 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/38 file after the snapshot 2019-12-05T06:09:29.317 INFO:tasks.workunit.client.0.smithi016.stdout: create 195/39 file after the snapshot 2019-12-05T06:09:29.319 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/.snap/snap-subdir-test 2019-12-05T06:09:29.340 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/30 file after the snapshot 2019-12-05T06:09:29.341 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/31 file after the snapshot 2019-12-05T06:09:29.344 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/32 file after the snapshot 2019-12-05T06:09:29.346 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/33 file after the snapshot 2019-12-05T06:09:29.349 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/34 file after the snapshot 2019-12-05T06:09:29.351 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/35 file after the snapshot 2019-12-05T06:09:29.354 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/36 file after the snapshot 2019-12-05T06:09:29.356 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/37 file after the snapshot 2019-12-05T06:09:29.360 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/38 file after the snapshot 2019-12-05T06:09:29.361 INFO:tasks.workunit.client.0.smithi016.stdout: create 196/39 file after the snapshot 2019-12-05T06:09:29.363 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/.snap/snap-subdir-test 2019-12-05T06:09:29.383 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/30 file after the snapshot 2019-12-05T06:09:29.385 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/31 file after the snapshot 2019-12-05T06:09:29.388 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/32 file after the snapshot 2019-12-05T06:09:29.391 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/33 file after the snapshot 2019-12-05T06:09:29.392 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/34 file after the snapshot 2019-12-05T06:09:29.395 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/35 file after the snapshot 2019-12-05T06:09:29.397 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/36 file after the snapshot 2019-12-05T06:09:29.401 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/37 file after the snapshot 2019-12-05T06:09:29.404 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/38 file after the snapshot 2019-12-05T06:09:29.405 INFO:tasks.workunit.client.0.smithi016.stdout: create 197/39 file after the snapshot 2019-12-05T06:09:29.407 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/.snap/snap-subdir-test 2019-12-05T06:09:29.435 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/30 file after the snapshot 2019-12-05T06:09:29.436 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/31 file after the snapshot 2019-12-05T06:09:29.436 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/32 file after the snapshot 2019-12-05T06:09:29.436 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/33 file after the snapshot 2019-12-05T06:09:29.436 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/34 file after the snapshot 2019-12-05T06:09:29.439 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/35 file after the snapshot 2019-12-05T06:09:29.442 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/36 file after the snapshot 2019-12-05T06:09:29.444 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/37 file after the snapshot 2019-12-05T06:09:29.453 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/38 file after the snapshot 2019-12-05T06:09:29.453 INFO:tasks.workunit.client.0.smithi016.stdout: create 198/39 file after the snapshot 2019-12-05T06:09:29.453 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/.snap/snap-subdir-test 2019-12-05T06:09:30.129 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/30 file after the snapshot 2019-12-05T06:09:30.531 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/31 file after the snapshot 2019-12-05T06:09:30.540 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/32 file after the snapshot 2019-12-05T06:09:30.545 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/33 file after the snapshot 2019-12-05T06:09:30.551 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/34 file after the snapshot 2019-12-05T06:09:30.558 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/35 file after the snapshot 2019-12-05T06:09:30.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/36 file after the snapshot 2019-12-05T06:09:30.565 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/37 file after the snapshot 2019-12-05T06:09:30.595 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/38 file after the snapshot 2019-12-05T06:09:30.595 INFO:tasks.workunit.client.0.smithi016.stdout: create 199/39 file after the snapshot 2019-12-05T06:09:30.596 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/.snap/snap-subdir-test 2019-12-05T06:09:31.004 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/30 file after the snapshot 2019-12-05T06:09:31.060 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/31 file after the snapshot 2019-12-05T06:09:31.064 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/32 file after the snapshot 2019-12-05T06:09:31.069 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/33 file after the snapshot 2019-12-05T06:09:31.074 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/34 file after the snapshot 2019-12-05T06:09:31.078 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/35 file after the snapshot 2019-12-05T06:09:31.083 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/36 file after the snapshot 2019-12-05T06:09:31.086 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/37 file after the snapshot 2019-12-05T06:09:31.091 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/38 file after the snapshot 2019-12-05T06:09:31.096 INFO:tasks.workunit.client.0.smithi016.stdout: create 200/39 file after the snapshot 2019-12-05T06:09:31.099 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/.snap/snap-subdir-test 2019-12-05T06:09:31.167 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/30 file after the snapshot 2019-12-05T06:09:31.171 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/31 file after the snapshot 2019-12-05T06:09:31.174 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/32 file after the snapshot 2019-12-05T06:09:31.183 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/33 file after the snapshot 2019-12-05T06:09:31.184 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/34 file after the snapshot 2019-12-05T06:09:31.186 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/35 file after the snapshot 2019-12-05T06:09:31.190 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/36 file after the snapshot 2019-12-05T06:09:31.194 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/37 file after the snapshot 2019-12-05T06:09:31.198 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/38 file after the snapshot 2019-12-05T06:09:31.202 INFO:tasks.workunit.client.0.smithi016.stdout: create 201/39 file after the snapshot 2019-12-05T06:09:31.206 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/.snap/snap-subdir-test 2019-12-05T06:09:31.983 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/30 file after the snapshot 2019-12-05T06:09:31.988 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/31 file after the snapshot 2019-12-05T06:09:31.993 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/32 file after the snapshot 2019-12-05T06:09:31.997 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/33 file after the snapshot 2019-12-05T06:09:32.003 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/34 file after the snapshot 2019-12-05T06:09:32.008 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/35 file after the snapshot 2019-12-05T06:09:32.011 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/36 file after the snapshot 2019-12-05T06:09:32.016 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/37 file after the snapshot 2019-12-05T06:09:32.019 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/38 file after the snapshot 2019-12-05T06:09:32.024 INFO:tasks.workunit.client.0.smithi016.stdout: create 202/39 file after the snapshot 2019-12-05T06:09:32.028 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/.snap/snap-subdir-test 2019-12-05T06:09:32.065 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/30 file after the snapshot 2019-12-05T06:09:32.068 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/31 file after the snapshot 2019-12-05T06:09:32.072 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/32 file after the snapshot 2019-12-05T06:09:32.076 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/33 file after the snapshot 2019-12-05T06:09:32.079 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/34 file after the snapshot 2019-12-05T06:09:32.083 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/35 file after the snapshot 2019-12-05T06:09:32.086 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/36 file after the snapshot 2019-12-05T06:09:32.089 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/37 file after the snapshot 2019-12-05T06:09:32.092 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/38 file after the snapshot 2019-12-05T06:09:32.096 INFO:tasks.workunit.client.0.smithi016.stdout: create 203/39 file after the snapshot 2019-12-05T06:09:32.100 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/.snap/snap-subdir-test 2019-12-05T06:09:32.126 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/30 file after the snapshot 2019-12-05T06:09:32.129 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/31 file after the snapshot 2019-12-05T06:09:32.132 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/32 file after the snapshot 2019-12-05T06:09:32.135 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/33 file after the snapshot 2019-12-05T06:09:32.138 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/34 file after the snapshot 2019-12-05T06:09:32.141 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/35 file after the snapshot 2019-12-05T06:09:32.143 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/36 file after the snapshot 2019-12-05T06:09:32.146 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/37 file after the snapshot 2019-12-05T06:09:32.150 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/38 file after the snapshot 2019-12-05T06:09:32.153 INFO:tasks.workunit.client.0.smithi016.stdout: create 204/39 file after the snapshot 2019-12-05T06:09:32.156 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/.snap/snap-subdir-test 2019-12-05T06:09:32.178 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/30 file after the snapshot 2019-12-05T06:09:32.181 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/31 file after the snapshot 2019-12-05T06:09:32.183 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/32 file after the snapshot 2019-12-05T06:09:32.186 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/33 file after the snapshot 2019-12-05T06:09:32.189 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/34 file after the snapshot 2019-12-05T06:09:32.192 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/35 file after the snapshot 2019-12-05T06:09:32.195 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/36 file after the snapshot 2019-12-05T06:09:32.198 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/37 file after the snapshot 2019-12-05T06:09:32.201 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/38 file after the snapshot 2019-12-05T06:09:32.204 INFO:tasks.workunit.client.0.smithi016.stdout: create 205/39 file after the snapshot 2019-12-05T06:09:32.208 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/.snap/snap-subdir-test 2019-12-05T06:09:32.227 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/30 file after the snapshot 2019-12-05T06:09:32.229 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/31 file after the snapshot 2019-12-05T06:09:32.232 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/32 file after the snapshot 2019-12-05T06:09:32.236 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/33 file after the snapshot 2019-12-05T06:09:32.237 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/34 file after the snapshot 2019-12-05T06:09:32.240 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/35 file after the snapshot 2019-12-05T06:09:32.243 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/36 file after the snapshot 2019-12-05T06:09:32.245 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/37 file after the snapshot 2019-12-05T06:09:32.247 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/38 file after the snapshot 2019-12-05T06:09:32.250 INFO:tasks.workunit.client.0.smithi016.stdout: create 206/39 file after the snapshot 2019-12-05T06:09:32.252 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/.snap/snap-subdir-test 2019-12-05T06:09:32.274 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/30 file after the snapshot 2019-12-05T06:09:32.276 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/31 file after the snapshot 2019-12-05T06:09:32.279 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/32 file after the snapshot 2019-12-05T06:09:32.280 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/33 file after the snapshot 2019-12-05T06:09:32.282 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/34 file after the snapshot 2019-12-05T06:09:32.285 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/35 file after the snapshot 2019-12-05T06:09:32.288 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/36 file after the snapshot 2019-12-05T06:09:32.291 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/37 file after the snapshot 2019-12-05T06:09:32.293 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/38 file after the snapshot 2019-12-05T06:09:32.295 INFO:tasks.workunit.client.0.smithi016.stdout: create 207/39 file after the snapshot 2019-12-05T06:09:32.298 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/.snap/snap-subdir-test 2019-12-05T06:09:32.318 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/30 file after the snapshot 2019-12-05T06:09:32.320 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/31 file after the snapshot 2019-12-05T06:09:32.322 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/32 file after the snapshot 2019-12-05T06:09:32.325 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/33 file after the snapshot 2019-12-05T06:09:32.327 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/34 file after the snapshot 2019-12-05T06:09:32.330 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/35 file after the snapshot 2019-12-05T06:09:32.333 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/36 file after the snapshot 2019-12-05T06:09:32.335 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/37 file after the snapshot 2019-12-05T06:09:32.397 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/38 file after the snapshot 2019-12-05T06:09:32.401 INFO:tasks.workunit.client.0.smithi016.stdout: create 208/39 file after the snapshot 2019-12-05T06:09:32.405 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/.snap/snap-subdir-test 2019-12-05T06:09:32.431 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/30 file after the snapshot 2019-12-05T06:09:32.438 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/31 file after the snapshot 2019-12-05T06:09:32.438 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/32 file after the snapshot 2019-12-05T06:09:32.441 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/33 file after the snapshot 2019-12-05T06:09:32.444 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/34 file after the snapshot 2019-12-05T06:09:32.449 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/35 file after the snapshot 2019-12-05T06:09:32.451 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/36 file after the snapshot 2019-12-05T06:09:32.457 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/37 file after the snapshot 2019-12-05T06:09:32.457 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/38 file after the snapshot 2019-12-05T06:09:32.463 INFO:tasks.workunit.client.0.smithi016.stdout: create 209/39 file after the snapshot 2019-12-05T06:09:32.463 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/.snap/snap-subdir-test 2019-12-05T06:09:32.488 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/30 file after the snapshot 2019-12-05T06:09:32.491 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/31 file after the snapshot 2019-12-05T06:09:32.494 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/32 file after the snapshot 2019-12-05T06:09:32.497 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/33 file after the snapshot 2019-12-05T06:09:32.500 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/34 file after the snapshot 2019-12-05T06:09:32.503 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/35 file after the snapshot 2019-12-05T06:09:32.506 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/36 file after the snapshot 2019-12-05T06:09:32.509 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/37 file after the snapshot 2019-12-05T06:09:32.511 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/38 file after the snapshot 2019-12-05T06:09:32.515 INFO:tasks.workunit.client.0.smithi016.stdout: create 210/39 file after the snapshot 2019-12-05T06:09:32.518 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/.snap/snap-subdir-test 2019-12-05T06:09:32.539 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/30 file after the snapshot 2019-12-05T06:09:32.541 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/31 file after the snapshot 2019-12-05T06:09:32.543 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/32 file after the snapshot 2019-12-05T06:09:32.546 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/33 file after the snapshot 2019-12-05T06:09:32.549 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/34 file after the snapshot 2019-12-05T06:09:32.551 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/35 file after the snapshot 2019-12-05T06:09:32.554 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/36 file after the snapshot 2019-12-05T06:09:32.556 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/37 file after the snapshot 2019-12-05T06:09:32.559 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/38 file after the snapshot 2019-12-05T06:09:32.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 211/39 file after the snapshot 2019-12-05T06:09:32.564 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/.snap/snap-subdir-test 2019-12-05T06:09:32.585 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/30 file after the snapshot 2019-12-05T06:09:32.587 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/31 file after the snapshot 2019-12-05T06:09:32.590 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/32 file after the snapshot 2019-12-05T06:09:32.592 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/33 file after the snapshot 2019-12-05T06:09:32.597 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/34 file after the snapshot 2019-12-05T06:09:32.598 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/35 file after the snapshot 2019-12-05T06:09:32.603 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/36 file after the snapshot 2019-12-05T06:09:32.603 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/37 file after the snapshot 2019-12-05T06:09:32.605 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/38 file after the snapshot 2019-12-05T06:09:32.608 INFO:tasks.workunit.client.0.smithi016.stdout: create 212/39 file after the snapshot 2019-12-05T06:09:32.813 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/.snap/snap-subdir-test 2019-12-05T06:09:32.854 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/30 file after the snapshot 2019-12-05T06:09:32.858 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/31 file after the snapshot 2019-12-05T06:09:32.866 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/32 file after the snapshot 2019-12-05T06:09:32.870 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/33 file after the snapshot 2019-12-05T06:09:32.875 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/34 file after the snapshot 2019-12-05T06:09:32.879 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/35 file after the snapshot 2019-12-05T06:09:32.883 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/36 file after the snapshot 2019-12-05T06:09:32.888 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/37 file after the snapshot 2019-12-05T06:09:32.893 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/38 file after the snapshot 2019-12-05T06:09:32.896 INFO:tasks.workunit.client.0.smithi016.stdout: create 213/39 file after the snapshot 2019-12-05T06:09:32.900 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/.snap/snap-subdir-test 2019-12-05T06:09:32.935 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/30 file after the snapshot 2019-12-05T06:09:32.935 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/31 file after the snapshot 2019-12-05T06:09:32.936 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/32 file after the snapshot 2019-12-05T06:09:32.939 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/33 file after the snapshot 2019-12-05T06:09:32.942 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/34 file after the snapshot 2019-12-05T06:09:32.946 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/35 file after the snapshot 2019-12-05T06:09:32.950 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/36 file after the snapshot 2019-12-05T06:09:32.953 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/37 file after the snapshot 2019-12-05T06:09:32.960 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/38 file after the snapshot 2019-12-05T06:09:32.979 INFO:tasks.workunit.client.0.smithi016.stdout: create 214/39 file after the snapshot 2019-12-05T06:09:32.981 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/.snap/snap-subdir-test 2019-12-05T06:09:33.017 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/30 file after the snapshot 2019-12-05T06:09:33.018 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/31 file after the snapshot 2019-12-05T06:09:33.020 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/32 file after the snapshot 2019-12-05T06:09:33.023 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/33 file after the snapshot 2019-12-05T06:09:33.026 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/34 file after the snapshot 2019-12-05T06:09:33.028 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/35 file after the snapshot 2019-12-05T06:09:33.032 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/36 file after the snapshot 2019-12-05T06:09:33.034 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/37 file after the snapshot 2019-12-05T06:09:33.037 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/38 file after the snapshot 2019-12-05T06:09:33.039 INFO:tasks.workunit.client.0.smithi016.stdout: create 215/39 file after the snapshot 2019-12-05T06:09:33.042 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/.snap/snap-subdir-test 2019-12-05T06:09:33.064 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/30 file after the snapshot 2019-12-05T06:09:33.066 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/31 file after the snapshot 2019-12-05T06:09:33.068 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/32 file after the snapshot 2019-12-05T06:09:33.071 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/33 file after the snapshot 2019-12-05T06:09:33.073 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/34 file after the snapshot 2019-12-05T06:09:33.076 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/35 file after the snapshot 2019-12-05T06:09:33.078 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/36 file after the snapshot 2019-12-05T06:09:33.081 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/37 file after the snapshot 2019-12-05T06:09:33.083 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/38 file after the snapshot 2019-12-05T06:09:33.086 INFO:tasks.workunit.client.0.smithi016.stdout: create 216/39 file after the snapshot 2019-12-05T06:09:33.088 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/.snap/snap-subdir-test 2019-12-05T06:09:33.389 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/30 file after the snapshot 2019-12-05T06:09:33.394 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/31 file after the snapshot 2019-12-05T06:09:33.398 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/32 file after the snapshot 2019-12-05T06:09:33.403 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/33 file after the snapshot 2019-12-05T06:09:33.409 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/34 file after the snapshot 2019-12-05T06:09:33.413 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/35 file after the snapshot 2019-12-05T06:09:33.417 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/36 file after the snapshot 2019-12-05T06:09:33.423 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/37 file after the snapshot 2019-12-05T06:09:33.427 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/38 file after the snapshot 2019-12-05T06:09:33.431 INFO:tasks.workunit.client.0.smithi016.stdout: create 217/39 file after the snapshot 2019-12-05T06:09:33.435 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/.snap/snap-subdir-test 2019-12-05T06:09:33.898 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/30 file after the snapshot 2019-12-05T06:09:33.902 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/31 file after the snapshot 2019-12-05T06:09:33.907 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/32 file after the snapshot 2019-12-05T06:09:33.911 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/33 file after the snapshot 2019-12-05T06:09:33.916 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/34 file after the snapshot 2019-12-05T06:09:33.920 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/35 file after the snapshot 2019-12-05T06:09:33.925 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/36 file after the snapshot 2019-12-05T06:09:33.929 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/37 file after the snapshot 2019-12-05T06:09:33.934 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/38 file after the snapshot 2019-12-05T06:09:33.939 INFO:tasks.workunit.client.0.smithi016.stdout: create 218/39 file after the snapshot 2019-12-05T06:09:33.943 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/.snap/snap-subdir-test 2019-12-05T06:09:33.982 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/30 file after the snapshot 2019-12-05T06:09:33.985 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/31 file after the snapshot 2019-12-05T06:09:33.988 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/32 file after the snapshot 2019-12-05T06:09:33.993 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/33 file after the snapshot 2019-12-05T06:09:33.997 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/34 file after the snapshot 2019-12-05T06:09:34.001 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/35 file after the snapshot 2019-12-05T06:09:34.005 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/36 file after the snapshot 2019-12-05T06:09:34.008 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/37 file after the snapshot 2019-12-05T06:09:34.013 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/38 file after the snapshot 2019-12-05T06:09:34.016 INFO:tasks.workunit.client.0.smithi016.stdout: create 219/39 file after the snapshot 2019-12-05T06:09:34.028 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/.snap/snap-subdir-test 2019-12-05T06:09:34.053 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/30 file after the snapshot 2019-12-05T06:09:34.056 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/31 file after the snapshot 2019-12-05T06:09:34.059 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/32 file after the snapshot 2019-12-05T06:09:34.069 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/33 file after the snapshot 2019-12-05T06:09:34.069 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/34 file after the snapshot 2019-12-05T06:09:34.070 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/35 file after the snapshot 2019-12-05T06:09:34.072 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/36 file after the snapshot 2019-12-05T06:09:34.074 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/37 file after the snapshot 2019-12-05T06:09:34.077 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/38 file after the snapshot 2019-12-05T06:09:34.092 INFO:tasks.workunit.client.0.smithi016.stdout: create 220/39 file after the snapshot 2019-12-05T06:09:34.092 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/.snap/snap-subdir-test 2019-12-05T06:09:34.147 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/30 file after the snapshot 2019-12-05T06:09:34.151 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/31 file after the snapshot 2019-12-05T06:09:34.155 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/32 file after the snapshot 2019-12-05T06:09:34.159 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/33 file after the snapshot 2019-12-05T06:09:34.163 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/34 file after the snapshot 2019-12-05T06:09:34.167 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/35 file after the snapshot 2019-12-05T06:09:34.170 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/36 file after the snapshot 2019-12-05T06:09:34.174 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/37 file after the snapshot 2019-12-05T06:09:34.177 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/38 file after the snapshot 2019-12-05T06:09:34.181 INFO:tasks.workunit.client.0.smithi016.stdout: create 221/39 file after the snapshot 2019-12-05T06:09:34.185 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/.snap/snap-subdir-test 2019-12-05T06:09:34.210 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/30 file after the snapshot 2019-12-05T06:09:34.213 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/31 file after the snapshot 2019-12-05T06:09:34.216 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/32 file after the snapshot 2019-12-05T06:09:34.219 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/33 file after the snapshot 2019-12-05T06:09:34.222 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/34 file after the snapshot 2019-12-05T06:09:34.225 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/35 file after the snapshot 2019-12-05T06:09:34.229 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/36 file after the snapshot 2019-12-05T06:09:34.232 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/37 file after the snapshot 2019-12-05T06:09:34.235 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/38 file after the snapshot 2019-12-05T06:09:34.239 INFO:tasks.workunit.client.0.smithi016.stdout: create 222/39 file after the snapshot 2019-12-05T06:09:34.241 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/.snap/snap-subdir-test 2019-12-05T06:09:34.267 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/30 file after the snapshot 2019-12-05T06:09:34.270 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/31 file after the snapshot 2019-12-05T06:09:34.273 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/32 file after the snapshot 2019-12-05T06:09:34.277 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/33 file after the snapshot 2019-12-05T06:09:34.279 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/34 file after the snapshot 2019-12-05T06:09:34.281 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/35 file after the snapshot 2019-12-05T06:09:34.284 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/36 file after the snapshot 2019-12-05T06:09:34.287 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/37 file after the snapshot 2019-12-05T06:09:34.290 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/38 file after the snapshot 2019-12-05T06:09:34.293 INFO:tasks.workunit.client.0.smithi016.stdout: create 223/39 file after the snapshot 2019-12-05T06:09:34.296 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/.snap/snap-subdir-test 2019-12-05T06:09:35.851 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/30 file after the snapshot 2019-12-05T06:09:35.859 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/31 file after the snapshot 2019-12-05T06:09:35.864 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/32 file after the snapshot 2019-12-05T06:09:35.869 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/33 file after the snapshot 2019-12-05T06:09:35.873 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/34 file after the snapshot 2019-12-05T06:09:35.879 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/35 file after the snapshot 2019-12-05T06:09:35.888 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/36 file after the snapshot 2019-12-05T06:09:35.888 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/37 file after the snapshot 2019-12-05T06:09:35.896 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/38 file after the snapshot 2019-12-05T06:09:35.900 INFO:tasks.workunit.client.0.smithi016.stdout: create 224/39 file after the snapshot 2019-12-05T06:09:35.906 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/.snap/snap-subdir-test 2019-12-05T06:09:35.935 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/30 file after the snapshot 2019-12-05T06:09:35.939 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/31 file after the snapshot 2019-12-05T06:09:35.942 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/32 file after the snapshot 2019-12-05T06:09:35.945 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/33 file after the snapshot 2019-12-05T06:09:35.950 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/34 file after the snapshot 2019-12-05T06:09:35.952 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/35 file after the snapshot 2019-12-05T06:09:35.956 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/36 file after the snapshot 2019-12-05T06:09:35.967 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/37 file after the snapshot 2019-12-05T06:09:35.970 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/38 file after the snapshot 2019-12-05T06:09:35.973 INFO:tasks.workunit.client.0.smithi016.stdout: create 225/39 file after the snapshot 2019-12-05T06:09:35.977 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/.snap/snap-subdir-test 2019-12-05T06:09:36.015 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/30 file after the snapshot 2019-12-05T06:09:36.015 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/31 file after the snapshot 2019-12-05T06:09:36.016 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/32 file after the snapshot 2019-12-05T06:09:36.023 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/33 file after the snapshot 2019-12-05T06:09:36.024 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/34 file after the snapshot 2019-12-05T06:09:36.036 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/35 file after the snapshot 2019-12-05T06:09:36.036 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/36 file after the snapshot 2019-12-05T06:09:36.037 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/37 file after the snapshot 2019-12-05T06:09:36.037 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/38 file after the snapshot 2019-12-05T06:09:36.037 INFO:tasks.workunit.client.0.smithi016.stdout: create 226/39 file after the snapshot 2019-12-05T06:09:36.044 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/.snap/snap-subdir-test 2019-12-05T06:09:36.063 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/30 file after the snapshot 2019-12-05T06:09:36.065 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/31 file after the snapshot 2019-12-05T06:09:36.076 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/32 file after the snapshot 2019-12-05T06:09:36.076 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/33 file after the snapshot 2019-12-05T06:09:36.076 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/34 file after the snapshot 2019-12-05T06:09:36.076 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/35 file after the snapshot 2019-12-05T06:09:36.078 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/36 file after the snapshot 2019-12-05T06:09:36.081 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/37 file after the snapshot 2019-12-05T06:09:36.084 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/38 file after the snapshot 2019-12-05T06:09:36.088 INFO:tasks.workunit.client.0.smithi016.stdout: create 227/39 file after the snapshot 2019-12-05T06:09:36.090 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/.snap/snap-subdir-test 2019-12-05T06:09:36.115 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/30 file after the snapshot 2019-12-05T06:09:36.117 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/31 file after the snapshot 2019-12-05T06:09:36.120 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/32 file after the snapshot 2019-12-05T06:09:36.123 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/33 file after the snapshot 2019-12-05T06:09:36.125 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/34 file after the snapshot 2019-12-05T06:09:36.128 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/35 file after the snapshot 2019-12-05T06:09:36.131 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/36 file after the snapshot 2019-12-05T06:09:36.133 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/37 file after the snapshot 2019-12-05T06:09:36.146 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/38 file after the snapshot 2019-12-05T06:09:36.146 INFO:tasks.workunit.client.0.smithi016.stdout: create 228/39 file after the snapshot 2019-12-05T06:09:36.146 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/.snap/snap-subdir-test 2019-12-05T06:09:36.160 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/30 file after the snapshot 2019-12-05T06:09:36.162 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/31 file after the snapshot 2019-12-05T06:09:36.164 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/32 file after the snapshot 2019-12-05T06:09:36.167 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/33 file after the snapshot 2019-12-05T06:09:36.169 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/34 file after the snapshot 2019-12-05T06:09:36.171 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/35 file after the snapshot 2019-12-05T06:09:36.173 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/36 file after the snapshot 2019-12-05T06:09:36.175 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/37 file after the snapshot 2019-12-05T06:09:36.178 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/38 file after the snapshot 2019-12-05T06:09:36.180 INFO:tasks.workunit.client.0.smithi016.stdout: create 229/39 file after the snapshot 2019-12-05T06:09:36.182 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/.snap/snap-subdir-test 2019-12-05T06:09:36.873 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/30 file after the snapshot 2019-12-05T06:09:36.878 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/31 file after the snapshot 2019-12-05T06:09:36.884 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/32 file after the snapshot 2019-12-05T06:09:36.889 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/33 file after the snapshot 2019-12-05T06:09:36.893 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/34 file after the snapshot 2019-12-05T06:09:36.899 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/35 file after the snapshot 2019-12-05T06:09:36.904 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/36 file after the snapshot 2019-12-05T06:09:36.906 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/37 file after the snapshot 2019-12-05T06:09:36.911 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/38 file after the snapshot 2019-12-05T06:09:36.915 INFO:tasks.workunit.client.0.smithi016.stdout: create 230/39 file after the snapshot 2019-12-05T06:09:36.920 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/.snap/snap-subdir-test 2019-12-05T06:09:36.953 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/30 file after the snapshot 2019-12-05T06:09:36.954 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/31 file after the snapshot 2019-12-05T06:09:36.960 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/32 file after the snapshot 2019-12-05T06:09:36.975 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/33 file after the snapshot 2019-12-05T06:09:36.978 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/34 file after the snapshot 2019-12-05T06:09:36.981 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/35 file after the snapshot 2019-12-05T06:09:36.985 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/36 file after the snapshot 2019-12-05T06:09:36.988 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/37 file after the snapshot 2019-12-05T06:09:36.992 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/38 file after the snapshot 2019-12-05T06:09:36.996 INFO:tasks.workunit.client.0.smithi016.stdout: create 231/39 file after the snapshot 2019-12-05T06:09:37.000 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/.snap/snap-subdir-test 2019-12-05T06:09:37.033 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/30 file after the snapshot 2019-12-05T06:09:37.034 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/31 file after the snapshot 2019-12-05T06:09:37.037 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/32 file after the snapshot 2019-12-05T06:09:37.040 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/33 file after the snapshot 2019-12-05T06:09:37.043 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/34 file after the snapshot 2019-12-05T06:09:37.045 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/35 file after the snapshot 2019-12-05T06:09:37.048 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/36 file after the snapshot 2019-12-05T06:09:37.051 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/37 file after the snapshot 2019-12-05T06:09:37.054 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/38 file after the snapshot 2019-12-05T06:09:37.059 INFO:tasks.workunit.client.0.smithi016.stdout: create 232/39 file after the snapshot 2019-12-05T06:09:37.060 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/.snap/snap-subdir-test 2019-12-05T06:09:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/30 file after the snapshot 2019-12-05T06:09:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/31 file after the snapshot 2019-12-05T06:09:37.090 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/32 file after the snapshot 2019-12-05T06:09:37.091 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/33 file after the snapshot 2019-12-05T06:09:37.097 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/34 file after the snapshot 2019-12-05T06:09:37.097 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/35 file after the snapshot 2019-12-05T06:09:37.105 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/36 file after the snapshot 2019-12-05T06:09:37.106 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/37 file after the snapshot 2019-12-05T06:09:37.106 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/38 file after the snapshot 2019-12-05T06:09:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: create 233/39 file after the snapshot 2019-12-05T06:09:37.111 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/.snap/snap-subdir-test 2019-12-05T06:09:37.130 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/30 file after the snapshot 2019-12-05T06:09:37.133 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/31 file after the snapshot 2019-12-05T06:09:37.136 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/32 file after the snapshot 2019-12-05T06:09:37.141 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/33 file after the snapshot 2019-12-05T06:09:37.142 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/34 file after the snapshot 2019-12-05T06:09:37.144 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/35 file after the snapshot 2019-12-05T06:09:37.146 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/36 file after the snapshot 2019-12-05T06:09:37.151 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/37 file after the snapshot 2019-12-05T06:09:37.151 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/38 file after the snapshot 2019-12-05T06:09:37.153 INFO:tasks.workunit.client.0.smithi016.stdout: create 234/39 file after the snapshot 2019-12-05T06:09:37.156 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/.snap/snap-subdir-test 2019-12-05T06:09:37.180 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/30 file after the snapshot 2019-12-05T06:09:37.182 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/31 file after the snapshot 2019-12-05T06:09:37.184 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/32 file after the snapshot 2019-12-05T06:09:37.186 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/33 file after the snapshot 2019-12-05T06:09:37.189 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/34 file after the snapshot 2019-12-05T06:09:37.192 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/35 file after the snapshot 2019-12-05T06:09:37.194 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/36 file after the snapshot 2019-12-05T06:09:37.196 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/37 file after the snapshot 2019-12-05T06:09:37.198 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/38 file after the snapshot 2019-12-05T06:09:37.201 INFO:tasks.workunit.client.0.smithi016.stdout: create 235/39 file after the snapshot 2019-12-05T06:09:37.203 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/.snap/snap-subdir-test 2019-12-05T06:09:37.223 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/30 file after the snapshot 2019-12-05T06:09:37.225 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/31 file after the snapshot 2019-12-05T06:09:37.228 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/32 file after the snapshot 2019-12-05T06:09:37.230 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/33 file after the snapshot 2019-12-05T06:09:37.232 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/34 file after the snapshot 2019-12-05T06:09:37.235 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/35 file after the snapshot 2019-12-05T06:09:37.238 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/36 file after the snapshot 2019-12-05T06:09:37.241 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/37 file after the snapshot 2019-12-05T06:09:37.244 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/38 file after the snapshot 2019-12-05T06:09:37.247 INFO:tasks.workunit.client.0.smithi016.stdout: create 236/39 file after the snapshot 2019-12-05T06:09:37.249 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/.snap/snap-subdir-test 2019-12-05T06:09:37.269 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/30 file after the snapshot 2019-12-05T06:09:37.272 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/31 file after the snapshot 2019-12-05T06:09:37.274 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/32 file after the snapshot 2019-12-05T06:09:37.277 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/33 file after the snapshot 2019-12-05T06:09:37.279 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/34 file after the snapshot 2019-12-05T06:09:37.282 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/35 file after the snapshot 2019-12-05T06:09:37.284 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/36 file after the snapshot 2019-12-05T06:09:37.287 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/37 file after the snapshot 2019-12-05T06:09:37.289 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/38 file after the snapshot 2019-12-05T06:09:37.291 INFO:tasks.workunit.client.0.smithi016.stdout: create 237/39 file after the snapshot 2019-12-05T06:09:37.294 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/.snap/snap-subdir-test 2019-12-05T06:09:37.332 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/30 file after the snapshot 2019-12-05T06:09:37.334 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/31 file after the snapshot 2019-12-05T06:09:37.336 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/32 file after the snapshot 2019-12-05T06:09:37.340 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/33 file after the snapshot 2019-12-05T06:09:37.342 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/34 file after the snapshot 2019-12-05T06:09:37.345 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/35 file after the snapshot 2019-12-05T06:09:37.348 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/36 file after the snapshot 2019-12-05T06:09:37.350 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/37 file after the snapshot 2019-12-05T06:09:37.352 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/38 file after the snapshot 2019-12-05T06:09:37.355 INFO:tasks.workunit.client.0.smithi016.stdout: create 238/39 file after the snapshot 2019-12-05T06:09:37.357 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/.snap/snap-subdir-test 2019-12-05T06:09:37.377 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/30 file after the snapshot 2019-12-05T06:09:37.379 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/31 file after the snapshot 2019-12-05T06:09:37.381 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/32 file after the snapshot 2019-12-05T06:09:37.384 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/33 file after the snapshot 2019-12-05T06:09:37.392 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/34 file after the snapshot 2019-12-05T06:09:37.393 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/35 file after the snapshot 2019-12-05T06:09:37.393 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/36 file after the snapshot 2019-12-05T06:09:37.394 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/37 file after the snapshot 2019-12-05T06:09:37.396 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/38 file after the snapshot 2019-12-05T06:09:37.398 INFO:tasks.workunit.client.0.smithi016.stdout: create 239/39 file after the snapshot 2019-12-05T06:09:37.401 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/.snap/snap-subdir-test 2019-12-05T06:09:37.421 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/30 file after the snapshot 2019-12-05T06:09:37.423 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/31 file after the snapshot 2019-12-05T06:09:37.425 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/32 file after the snapshot 2019-12-05T06:09:37.427 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/33 file after the snapshot 2019-12-05T06:09:37.429 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/34 file after the snapshot 2019-12-05T06:09:37.432 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/35 file after the snapshot 2019-12-05T06:09:37.434 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/36 file after the snapshot 2019-12-05T06:09:37.436 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/37 file after the snapshot 2019-12-05T06:09:37.438 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/38 file after the snapshot 2019-12-05T06:09:37.441 INFO:tasks.workunit.client.0.smithi016.stdout: create 240/39 file after the snapshot 2019-12-05T06:09:37.443 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/.snap/snap-subdir-test 2019-12-05T06:09:37.462 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/30 file after the snapshot 2019-12-05T06:09:37.464 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/31 file after the snapshot 2019-12-05T06:09:37.466 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/32 file after the snapshot 2019-12-05T06:09:37.469 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/33 file after the snapshot 2019-12-05T06:09:37.471 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/34 file after the snapshot 2019-12-05T06:09:37.474 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/35 file after the snapshot 2019-12-05T06:09:37.476 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/36 file after the snapshot 2019-12-05T06:09:37.479 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/37 file after the snapshot 2019-12-05T06:09:37.481 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/38 file after the snapshot 2019-12-05T06:09:37.483 INFO:tasks.workunit.client.0.smithi016.stdout: create 241/39 file after the snapshot 2019-12-05T06:09:37.486 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/.snap/snap-subdir-test 2019-12-05T06:09:37.507 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/30 file after the snapshot 2019-12-05T06:09:37.509 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/31 file after the snapshot 2019-12-05T06:09:37.512 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/32 file after the snapshot 2019-12-05T06:09:37.514 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/33 file after the snapshot 2019-12-05T06:09:37.517 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/34 file after the snapshot 2019-12-05T06:09:37.520 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/35 file after the snapshot 2019-12-05T06:09:37.523 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/36 file after the snapshot 2019-12-05T06:09:37.525 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/37 file after the snapshot 2019-12-05T06:09:37.527 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/38 file after the snapshot 2019-12-05T06:09:37.529 INFO:tasks.workunit.client.0.smithi016.stdout: create 242/39 file after the snapshot 2019-12-05T06:09:37.532 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/.snap/snap-subdir-test 2019-12-05T06:09:37.551 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/30 file after the snapshot 2019-12-05T06:09:37.553 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/31 file after the snapshot 2019-12-05T06:09:37.555 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/32 file after the snapshot 2019-12-05T06:09:37.557 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/33 file after the snapshot 2019-12-05T06:09:37.560 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/34 file after the snapshot 2019-12-05T06:09:37.562 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/35 file after the snapshot 2019-12-05T06:09:37.565 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/36 file after the snapshot 2019-12-05T06:09:37.567 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/37 file after the snapshot 2019-12-05T06:09:37.570 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/38 file after the snapshot 2019-12-05T06:09:37.572 INFO:tasks.workunit.client.0.smithi016.stdout: create 243/39 file after the snapshot 2019-12-05T06:09:37.575 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/.snap/snap-subdir-test 2019-12-05T06:09:37.593 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/30 file after the snapshot 2019-12-05T06:09:37.596 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/31 file after the snapshot 2019-12-05T06:09:37.598 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/32 file after the snapshot 2019-12-05T06:09:37.601 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/33 file after the snapshot 2019-12-05T06:09:37.603 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/34 file after the snapshot 2019-12-05T06:09:37.605 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/35 file after the snapshot 2019-12-05T06:09:37.608 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/36 file after the snapshot 2019-12-05T06:09:37.610 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/37 file after the snapshot 2019-12-05T06:09:37.612 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/38 file after the snapshot 2019-12-05T06:09:37.614 INFO:tasks.workunit.client.0.smithi016.stdout: create 244/39 file after the snapshot 2019-12-05T06:09:37.617 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/.snap/snap-subdir-test 2019-12-05T06:09:37.635 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/30 file after the snapshot 2019-12-05T06:09:37.637 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/31 file after the snapshot 2019-12-05T06:09:37.639 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/32 file after the snapshot 2019-12-05T06:09:37.642 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/33 file after the snapshot 2019-12-05T06:09:37.644 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/34 file after the snapshot 2019-12-05T06:09:37.647 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/35 file after the snapshot 2019-12-05T06:09:37.649 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/36 file after the snapshot 2019-12-05T06:09:37.651 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/37 file after the snapshot 2019-12-05T06:09:37.654 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/38 file after the snapshot 2019-12-05T06:09:37.656 INFO:tasks.workunit.client.0.smithi016.stdout: create 245/39 file after the snapshot 2019-12-05T06:09:37.658 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/.snap/snap-subdir-test 2019-12-05T06:09:37.677 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/30 file after the snapshot 2019-12-05T06:09:37.679 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/31 file after the snapshot 2019-12-05T06:09:37.681 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/32 file after the snapshot 2019-12-05T06:09:37.683 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/33 file after the snapshot 2019-12-05T06:09:37.686 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/34 file after the snapshot 2019-12-05T06:09:37.688 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/35 file after the snapshot 2019-12-05T06:09:37.691 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/36 file after the snapshot 2019-12-05T06:09:37.693 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/37 file after the snapshot 2019-12-05T06:09:37.695 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/38 file after the snapshot 2019-12-05T06:09:37.698 INFO:tasks.workunit.client.0.smithi016.stdout: create 246/39 file after the snapshot 2019-12-05T06:09:37.700 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/.snap/snap-subdir-test 2019-12-05T06:09:37.719 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/30 file after the snapshot 2019-12-05T06:09:37.721 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/31 file after the snapshot 2019-12-05T06:09:37.724 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/32 file after the snapshot 2019-12-05T06:09:37.727 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/33 file after the snapshot 2019-12-05T06:09:37.728 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/34 file after the snapshot 2019-12-05T06:09:37.732 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/35 file after the snapshot 2019-12-05T06:09:37.735 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/36 file after the snapshot 2019-12-05T06:09:37.736 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/37 file after the snapshot 2019-12-05T06:09:37.739 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/38 file after the snapshot 2019-12-05T06:09:37.741 INFO:tasks.workunit.client.0.smithi016.stdout: create 247/39 file after the snapshot 2019-12-05T06:09:37.743 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/.snap/snap-subdir-test 2019-12-05T06:09:37.761 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/30 file after the snapshot 2019-12-05T06:09:37.763 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/31 file after the snapshot 2019-12-05T06:09:37.766 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/32 file after the snapshot 2019-12-05T06:09:37.768 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/33 file after the snapshot 2019-12-05T06:09:37.770 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/34 file after the snapshot 2019-12-05T06:09:37.772 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/35 file after the snapshot 2019-12-05T06:09:37.775 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/36 file after the snapshot 2019-12-05T06:09:37.778 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/37 file after the snapshot 2019-12-05T06:09:37.780 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/38 file after the snapshot 2019-12-05T06:09:37.782 INFO:tasks.workunit.client.0.smithi016.stdout: create 248/39 file after the snapshot 2019-12-05T06:09:37.785 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/.snap/snap-subdir-test 2019-12-05T06:09:38.121 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/30 file after the snapshot 2019-12-05T06:09:38.126 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/31 file after the snapshot 2019-12-05T06:09:38.132 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/32 file after the snapshot 2019-12-05T06:09:38.138 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/33 file after the snapshot 2019-12-05T06:09:38.143 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/34 file after the snapshot 2019-12-05T06:09:38.146 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/35 file after the snapshot 2019-12-05T06:09:38.151 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/36 file after the snapshot 2019-12-05T06:09:38.155 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/37 file after the snapshot 2019-12-05T06:09:38.159 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/38 file after the snapshot 2019-12-05T06:09:38.163 INFO:tasks.workunit.client.0.smithi016.stdout: create 249/39 file after the snapshot 2019-12-05T06:09:38.167 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/.snap/snap-subdir-test 2019-12-05T06:09:38.247 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/30 file after the snapshot 2019-12-05T06:09:38.251 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/31 file after the snapshot 2019-12-05T06:09:38.261 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/32 file after the snapshot 2019-12-05T06:09:38.262 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/33 file after the snapshot 2019-12-05T06:09:38.276 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/34 file after the snapshot 2019-12-05T06:09:38.277 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/35 file after the snapshot 2019-12-05T06:09:38.277 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/36 file after the snapshot 2019-12-05T06:09:38.277 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/37 file after the snapshot 2019-12-05T06:09:38.283 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/38 file after the snapshot 2019-12-05T06:09:38.283 INFO:tasks.workunit.client.0.smithi016.stdout: create 250/39 file after the snapshot 2019-12-05T06:09:38.286 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/.snap/snap-subdir-test 2019-12-05T06:09:38.329 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/30 file after the snapshot 2019-12-05T06:09:38.329 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/31 file after the snapshot 2019-12-05T06:09:38.329 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/32 file after the snapshot 2019-12-05T06:09:38.330 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/33 file after the snapshot 2019-12-05T06:09:38.330 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/34 file after the snapshot 2019-12-05T06:09:38.330 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/35 file after the snapshot 2019-12-05T06:09:38.330 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/36 file after the snapshot 2019-12-05T06:09:38.333 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/37 file after the snapshot 2019-12-05T06:09:38.336 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/38 file after the snapshot 2019-12-05T06:09:38.339 INFO:tasks.workunit.client.0.smithi016.stdout: create 251/39 file after the snapshot 2019-12-05T06:09:38.342 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/.snap/snap-subdir-test 2019-12-05T06:09:38.364 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/30 file after the snapshot 2019-12-05T06:09:38.367 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/31 file after the snapshot 2019-12-05T06:09:38.369 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/32 file after the snapshot 2019-12-05T06:09:38.372 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/33 file after the snapshot 2019-12-05T06:09:38.376 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/34 file after the snapshot 2019-12-05T06:09:38.378 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/35 file after the snapshot 2019-12-05T06:09:38.381 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/36 file after the snapshot 2019-12-05T06:09:38.384 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/37 file after the snapshot 2019-12-05T06:09:38.386 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/38 file after the snapshot 2019-12-05T06:09:38.390 INFO:tasks.workunit.client.0.smithi016.stdout: create 252/39 file after the snapshot 2019-12-05T06:09:38.393 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/.snap/snap-subdir-test 2019-12-05T06:09:38.413 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/30 file after the snapshot 2019-12-05T06:09:38.415 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/31 file after the snapshot 2019-12-05T06:09:38.417 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/32 file after the snapshot 2019-12-05T06:09:38.420 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/33 file after the snapshot 2019-12-05T06:09:38.423 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/34 file after the snapshot 2019-12-05T06:09:38.426 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/35 file after the snapshot 2019-12-05T06:09:38.428 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/36 file after the snapshot 2019-12-05T06:09:38.431 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/37 file after the snapshot 2019-12-05T06:09:38.434 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/38 file after the snapshot 2019-12-05T06:09:38.442 INFO:tasks.workunit.client.0.smithi016.stdout: create 253/39 file after the snapshot 2019-12-05T06:09:38.442 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/.snap/snap-subdir-test 2019-12-05T06:09:38.458 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/30 file after the snapshot 2019-12-05T06:09:38.460 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/31 file after the snapshot 2019-12-05T06:09:38.462 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/32 file after the snapshot 2019-12-05T06:09:38.466 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/33 file after the snapshot 2019-12-05T06:09:38.468 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/34 file after the snapshot 2019-12-05T06:09:38.471 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/35 file after the snapshot 2019-12-05T06:09:38.474 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/36 file after the snapshot 2019-12-05T06:09:38.477 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/37 file after the snapshot 2019-12-05T06:09:38.480 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/38 file after the snapshot 2019-12-05T06:09:38.482 INFO:tasks.workunit.client.0.smithi016.stdout: create 254/39 file after the snapshot 2019-12-05T06:09:38.485 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/.snap/snap-subdir-test 2019-12-05T06:09:38.503 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/30 file after the snapshot 2019-12-05T06:09:38.505 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/31 file after the snapshot 2019-12-05T06:09:38.508 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/32 file after the snapshot 2019-12-05T06:09:38.510 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/33 file after the snapshot 2019-12-05T06:09:38.529 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/34 file after the snapshot 2019-12-05T06:09:38.530 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/35 file after the snapshot 2019-12-05T06:09:38.530 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/36 file after the snapshot 2019-12-05T06:09:38.530 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/37 file after the snapshot 2019-12-05T06:09:38.530 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/38 file after the snapshot 2019-12-05T06:09:38.530 INFO:tasks.workunit.client.0.smithi016.stdout: create 255/39 file after the snapshot 2019-12-05T06:09:38.530 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/.snap/snap-subdir-test 2019-12-05T06:09:38.545 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/30 file after the snapshot 2019-12-05T06:09:38.550 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/31 file after the snapshot 2019-12-05T06:09:38.550 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/32 file after the snapshot 2019-12-05T06:09:38.551 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/33 file after the snapshot 2019-12-05T06:09:38.553 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/34 file after the snapshot 2019-12-05T06:09:38.556 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/35 file after the snapshot 2019-12-05T06:09:38.558 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/36 file after the snapshot 2019-12-05T06:09:38.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/37 file after the snapshot 2019-12-05T06:09:38.563 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/38 file after the snapshot 2019-12-05T06:09:38.566 INFO:tasks.workunit.client.0.smithi016.stdout: create 256/39 file after the snapshot 2019-12-05T06:09:38.569 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/.snap/snap-subdir-test 2019-12-05T06:09:39.017 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/30 file after the snapshot 2019-12-05T06:09:39.023 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/31 file after the snapshot 2019-12-05T06:09:39.032 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/32 file after the snapshot 2019-12-05T06:09:39.516 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/33 file after the snapshot 2019-12-05T06:09:39.523 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/34 file after the snapshot 2019-12-05T06:09:39.528 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/35 file after the snapshot 2019-12-05T06:09:39.533 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/36 file after the snapshot 2019-12-05T06:09:39.538 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/37 file after the snapshot 2019-12-05T06:09:39.544 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/38 file after the snapshot 2019-12-05T06:09:39.549 INFO:tasks.workunit.client.0.smithi016.stdout: create 257/39 file after the snapshot 2019-12-05T06:09:39.554 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/.snap/snap-subdir-test 2019-12-05T06:09:39.589 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/30 file after the snapshot 2019-12-05T06:09:39.592 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/31 file after the snapshot 2019-12-05T06:09:39.597 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/32 file after the snapshot 2019-12-05T06:09:39.611 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/33 file after the snapshot 2019-12-05T06:09:39.611 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/34 file after the snapshot 2019-12-05T06:09:39.612 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/35 file after the snapshot 2019-12-05T06:09:39.617 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/36 file after the snapshot 2019-12-05T06:09:39.621 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/37 file after the snapshot 2019-12-05T06:09:39.625 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/38 file after the snapshot 2019-12-05T06:09:39.630 INFO:tasks.workunit.client.0.smithi016.stdout: create 258/39 file after the snapshot 2019-12-05T06:09:39.633 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/.snap/snap-subdir-test 2019-12-05T06:09:39.663 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/30 file after the snapshot 2019-12-05T06:09:39.666 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/31 file after the snapshot 2019-12-05T06:09:39.670 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/32 file after the snapshot 2019-12-05T06:09:39.674 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/33 file after the snapshot 2019-12-05T06:09:39.678 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/34 file after the snapshot 2019-12-05T06:09:39.683 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/35 file after the snapshot 2019-12-05T06:09:39.686 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/36 file after the snapshot 2019-12-05T06:09:39.689 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/37 file after the snapshot 2019-12-05T06:09:39.693 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/38 file after the snapshot 2019-12-05T06:09:39.696 INFO:tasks.workunit.client.0.smithi016.stdout: create 259/39 file after the snapshot 2019-12-05T06:09:39.699 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/.snap/snap-subdir-test 2019-12-05T06:09:39.723 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/30 file after the snapshot 2019-12-05T06:09:39.726 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/31 file after the snapshot 2019-12-05T06:09:39.729 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/32 file after the snapshot 2019-12-05T06:09:39.731 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/33 file after the snapshot 2019-12-05T06:09:39.735 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/34 file after the snapshot 2019-12-05T06:09:39.738 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/35 file after the snapshot 2019-12-05T06:09:39.741 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/36 file after the snapshot 2019-12-05T06:09:39.745 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/37 file after the snapshot 2019-12-05T06:09:39.747 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/38 file after the snapshot 2019-12-05T06:09:39.749 INFO:tasks.workunit.client.0.smithi016.stdout: create 260/39 file after the snapshot 2019-12-05T06:09:39.752 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/.snap/snap-subdir-test 2019-12-05T06:09:39.774 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/30 file after the snapshot 2019-12-05T06:09:39.776 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/31 file after the snapshot 2019-12-05T06:09:39.778 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/32 file after the snapshot 2019-12-05T06:09:39.781 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/33 file after the snapshot 2019-12-05T06:09:39.784 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/34 file after the snapshot 2019-12-05T06:09:39.787 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/35 file after the snapshot 2019-12-05T06:09:39.790 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/36 file after the snapshot 2019-12-05T06:09:39.793 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/37 file after the snapshot 2019-12-05T06:09:39.797 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/38 file after the snapshot 2019-12-05T06:09:39.800 INFO:tasks.workunit.client.0.smithi016.stdout: create 261/39 file after the snapshot 2019-12-05T06:09:39.802 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/.snap/snap-subdir-test 2019-12-05T06:09:39.822 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/30 file after the snapshot 2019-12-05T06:09:39.825 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/31 file after the snapshot 2019-12-05T06:09:39.827 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/32 file after the snapshot 2019-12-05T06:09:39.829 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/33 file after the snapshot 2019-12-05T06:09:39.832 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/34 file after the snapshot 2019-12-05T06:09:39.833 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/35 file after the snapshot 2019-12-05T06:09:39.837 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/36 file after the snapshot 2019-12-05T06:09:39.840 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/37 file after the snapshot 2019-12-05T06:09:39.842 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/38 file after the snapshot 2019-12-05T06:09:39.844 INFO:tasks.workunit.client.0.smithi016.stdout: create 262/39 file after the snapshot 2019-12-05T06:09:39.846 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/.snap/snap-subdir-test 2019-12-05T06:09:39.866 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/30 file after the snapshot 2019-12-05T06:09:39.868 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/31 file after the snapshot 2019-12-05T06:09:39.871 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/32 file after the snapshot 2019-12-05T06:09:39.872 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/33 file after the snapshot 2019-12-05T06:09:39.874 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/34 file after the snapshot 2019-12-05T06:09:39.877 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/35 file after the snapshot 2019-12-05T06:09:39.879 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/36 file after the snapshot 2019-12-05T06:09:39.881 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/37 file after the snapshot 2019-12-05T06:09:39.885 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/38 file after the snapshot 2019-12-05T06:09:39.888 INFO:tasks.workunit.client.0.smithi016.stdout: create 263/39 file after the snapshot 2019-12-05T06:09:39.890 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/.snap/snap-subdir-test 2019-12-05T06:09:39.914 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/30 file after the snapshot 2019-12-05T06:09:39.914 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/31 file after the snapshot 2019-12-05T06:09:39.916 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/32 file after the snapshot 2019-12-05T06:09:39.918 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/33 file after the snapshot 2019-12-05T06:09:39.960 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/34 file after the snapshot 2019-12-05T06:09:40.001 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/35 file after the snapshot 2019-12-05T06:09:40.005 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/36 file after the snapshot 2019-12-05T06:09:40.010 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/37 file after the snapshot 2019-12-05T06:09:40.013 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/38 file after the snapshot 2019-12-05T06:09:40.019 INFO:tasks.workunit.client.0.smithi016.stdout: create 264/39 file after the snapshot 2019-12-05T06:09:40.022 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/.snap/snap-subdir-test 2019-12-05T06:09:40.069 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/30 file after the snapshot 2019-12-05T06:09:40.073 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/31 file after the snapshot 2019-12-05T06:09:40.077 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/32 file after the snapshot 2019-12-05T06:09:40.079 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/33 file after the snapshot 2019-12-05T06:09:40.084 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/34 file after the snapshot 2019-12-05T06:09:40.084 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/35 file after the snapshot 2019-12-05T06:09:40.088 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/36 file after the snapshot 2019-12-05T06:09:40.094 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/37 file after the snapshot 2019-12-05T06:09:40.095 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/38 file after the snapshot 2019-12-05T06:09:40.096 INFO:tasks.workunit.client.0.smithi016.stdout: create 265/39 file after the snapshot 2019-12-05T06:09:40.099 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/.snap/snap-subdir-test 2019-12-05T06:09:40.122 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/30 file after the snapshot 2019-12-05T06:09:40.126 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/31 file after the snapshot 2019-12-05T06:09:40.127 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/32 file after the snapshot 2019-12-05T06:09:40.130 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/33 file after the snapshot 2019-12-05T06:09:40.133 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/34 file after the snapshot 2019-12-05T06:09:40.137 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/35 file after the snapshot 2019-12-05T06:09:40.140 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/36 file after the snapshot 2019-12-05T06:09:40.142 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/37 file after the snapshot 2019-12-05T06:09:40.149 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/38 file after the snapshot 2019-12-05T06:09:40.149 INFO:tasks.workunit.client.0.smithi016.stdout: create 266/39 file after the snapshot 2019-12-05T06:09:40.150 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/.snap/snap-subdir-test 2019-12-05T06:09:40.171 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/30 file after the snapshot 2019-12-05T06:09:40.173 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/31 file after the snapshot 2019-12-05T06:09:40.175 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/32 file after the snapshot 2019-12-05T06:09:40.181 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/33 file after the snapshot 2019-12-05T06:09:40.181 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/34 file after the snapshot 2019-12-05T06:09:40.182 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/35 file after the snapshot 2019-12-05T06:09:40.184 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/36 file after the snapshot 2019-12-05T06:09:40.193 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/37 file after the snapshot 2019-12-05T06:09:40.193 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/38 file after the snapshot 2019-12-05T06:09:40.193 INFO:tasks.workunit.client.0.smithi016.stdout: create 267/39 file after the snapshot 2019-12-05T06:09:40.195 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/.snap/snap-subdir-test 2019-12-05T06:09:40.214 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/30 file after the snapshot 2019-12-05T06:09:40.219 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/31 file after the snapshot 2019-12-05T06:09:40.219 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/32 file after the snapshot 2019-12-05T06:09:40.224 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/33 file after the snapshot 2019-12-05T06:09:40.224 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/34 file after the snapshot 2019-12-05T06:09:40.227 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/35 file after the snapshot 2019-12-05T06:09:40.229 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/36 file after the snapshot 2019-12-05T06:09:40.232 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/37 file after the snapshot 2019-12-05T06:09:40.234 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/38 file after the snapshot 2019-12-05T06:09:40.237 INFO:tasks.workunit.client.0.smithi016.stdout: create 268/39 file after the snapshot 2019-12-05T06:09:40.239 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/.snap/snap-subdir-test 2019-12-05T06:09:40.258 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/30 file after the snapshot 2019-12-05T06:09:40.261 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/31 file after the snapshot 2019-12-05T06:09:40.264 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/32 file after the snapshot 2019-12-05T06:09:40.267 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/33 file after the snapshot 2019-12-05T06:09:40.270 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/34 file after the snapshot 2019-12-05T06:09:40.271 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/35 file after the snapshot 2019-12-05T06:09:40.274 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/36 file after the snapshot 2019-12-05T06:09:40.276 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/37 file after the snapshot 2019-12-05T06:09:40.279 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/38 file after the snapshot 2019-12-05T06:09:40.282 INFO:tasks.workunit.client.0.smithi016.stdout: create 269/39 file after the snapshot 2019-12-05T06:09:40.284 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/.snap/snap-subdir-test 2019-12-05T06:09:40.302 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/30 file after the snapshot 2019-12-05T06:09:40.306 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/31 file after the snapshot 2019-12-05T06:09:40.307 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/32 file after the snapshot 2019-12-05T06:09:40.309 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/33 file after the snapshot 2019-12-05T06:09:40.312 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/34 file after the snapshot 2019-12-05T06:09:40.316 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/35 file after the snapshot 2019-12-05T06:09:40.317 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/36 file after the snapshot 2019-12-05T06:09:40.320 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/37 file after the snapshot 2019-12-05T06:09:40.322 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/38 file after the snapshot 2019-12-05T06:09:40.324 INFO:tasks.workunit.client.0.smithi016.stdout: create 270/39 file after the snapshot 2019-12-05T06:09:40.327 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/.snap/snap-subdir-test 2019-12-05T06:09:40.345 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/30 file after the snapshot 2019-12-05T06:09:40.348 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/31 file after the snapshot 2019-12-05T06:09:40.349 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/32 file after the snapshot 2019-12-05T06:09:40.351 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/33 file after the snapshot 2019-12-05T06:09:40.354 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/34 file after the snapshot 2019-12-05T06:09:40.962 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/35 file after the snapshot 2019-12-05T06:09:40.977 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/36 file after the snapshot 2019-12-05T06:09:40.981 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/37 file after the snapshot 2019-12-05T06:09:40.986 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/38 file after the snapshot 2019-12-05T06:09:40.991 INFO:tasks.workunit.client.0.smithi016.stdout: create 271/39 file after the snapshot 2019-12-05T06:09:40.997 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/.snap/snap-subdir-test 2019-12-05T06:09:41.236 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/30 file after the snapshot 2019-12-05T06:09:41.241 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/31 file after the snapshot 2019-12-05T06:09:41.246 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/32 file after the snapshot 2019-12-05T06:09:41.253 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/33 file after the snapshot 2019-12-05T06:09:41.258 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/34 file after the snapshot 2019-12-05T06:09:41.262 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/35 file after the snapshot 2019-12-05T06:09:41.467 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/36 file after the snapshot 2019-12-05T06:09:41.473 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/37 file after the snapshot 2019-12-05T06:09:41.478 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/38 file after the snapshot 2019-12-05T06:09:41.483 INFO:tasks.workunit.client.0.smithi016.stdout: create 272/39 file after the snapshot 2019-12-05T06:09:41.489 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/.snap/snap-subdir-test 2019-12-05T06:09:41.758 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/30 file after the snapshot 2019-12-05T06:09:41.763 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/31 file after the snapshot 2019-12-05T06:09:41.770 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/32 file after the snapshot 2019-12-05T06:09:41.776 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/33 file after the snapshot 2019-12-05T06:09:41.780 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/34 file after the snapshot 2019-12-05T06:09:41.785 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/35 file after the snapshot 2019-12-05T06:09:41.790 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/36 file after the snapshot 2019-12-05T06:09:41.795 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/37 file after the snapshot 2019-12-05T06:09:41.801 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/38 file after the snapshot 2019-12-05T06:09:41.805 INFO:tasks.workunit.client.0.smithi016.stdout: create 273/39 file after the snapshot 2019-12-05T06:09:41.809 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/.snap/snap-subdir-test 2019-12-05T06:09:41.839 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/30 file after the snapshot 2019-12-05T06:09:41.842 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/31 file after the snapshot 2019-12-05T06:09:41.846 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/32 file after the snapshot 2019-12-05T06:09:41.850 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/33 file after the snapshot 2019-12-05T06:09:41.854 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/34 file after the snapshot 2019-12-05T06:09:41.859 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/35 file after the snapshot 2019-12-05T06:09:41.862 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/36 file after the snapshot 2019-12-05T06:09:41.866 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/37 file after the snapshot 2019-12-05T06:09:41.870 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/38 file after the snapshot 2019-12-05T06:09:41.875 INFO:tasks.workunit.client.0.smithi016.stdout: create 274/39 file after the snapshot 2019-12-05T06:09:41.877 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/.snap/snap-subdir-test 2019-12-05T06:09:41.911 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/30 file after the snapshot 2019-12-05T06:09:41.911 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/31 file after the snapshot 2019-12-05T06:09:41.913 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/32 file after the snapshot 2019-12-05T06:09:41.916 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/33 file after the snapshot 2019-12-05T06:09:41.919 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/34 file after the snapshot 2019-12-05T06:09:41.922 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/35 file after the snapshot 2019-12-05T06:09:41.925 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/36 file after the snapshot 2019-12-05T06:09:41.928 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/37 file after the snapshot 2019-12-05T06:09:41.931 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/38 file after the snapshot 2019-12-05T06:09:41.934 INFO:tasks.workunit.client.0.smithi016.stdout: create 275/39 file after the snapshot 2019-12-05T06:09:41.937 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/.snap/snap-subdir-test 2019-12-05T06:09:41.964 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/30 file after the snapshot 2019-12-05T06:09:41.986 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/31 file after the snapshot 2019-12-05T06:09:41.989 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/32 file after the snapshot 2019-12-05T06:09:41.992 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/33 file after the snapshot 2019-12-05T06:09:41.996 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/34 file after the snapshot 2019-12-05T06:09:41.999 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/35 file after the snapshot 2019-12-05T06:09:42.002 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/36 file after the snapshot 2019-12-05T06:09:42.004 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/37 file after the snapshot 2019-12-05T06:09:42.007 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/38 file after the snapshot 2019-12-05T06:09:42.010 INFO:tasks.workunit.client.0.smithi016.stdout: create 276/39 file after the snapshot 2019-12-05T06:09:42.012 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/.snap/snap-subdir-test 2019-12-05T06:09:42.659 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/30 file after the snapshot 2019-12-05T06:09:42.665 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/31 file after the snapshot 2019-12-05T06:09:42.670 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/32 file after the snapshot 2019-12-05T06:09:42.676 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/33 file after the snapshot 2019-12-05T06:09:42.681 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/34 file after the snapshot 2019-12-05T06:09:42.687 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/35 file after the snapshot 2019-12-05T06:09:42.691 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/36 file after the snapshot 2019-12-05T06:09:42.696 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/37 file after the snapshot 2019-12-05T06:09:42.700 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/38 file after the snapshot 2019-12-05T06:09:42.705 INFO:tasks.workunit.client.0.smithi016.stdout: create 277/39 file after the snapshot 2019-12-05T06:09:42.709 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/.snap/snap-subdir-test 2019-12-05T06:09:42.737 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/30 file after the snapshot 2019-12-05T06:09:42.741 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/31 file after the snapshot 2019-12-05T06:09:42.744 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/32 file after the snapshot 2019-12-05T06:09:42.747 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/33 file after the snapshot 2019-12-05T06:09:42.751 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/34 file after the snapshot 2019-12-05T06:09:42.755 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/35 file after the snapshot 2019-12-05T06:09:42.758 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/36 file after the snapshot 2019-12-05T06:09:42.762 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/37 file after the snapshot 2019-12-05T06:09:42.766 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/38 file after the snapshot 2019-12-05T06:09:42.769 INFO:tasks.workunit.client.0.smithi016.stdout: create 278/39 file after the snapshot 2019-12-05T06:09:42.772 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/.snap/snap-subdir-test 2019-12-05T06:09:42.795 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/30 file after the snapshot 2019-12-05T06:09:42.798 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/31 file after the snapshot 2019-12-05T06:09:42.803 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/32 file after the snapshot 2019-12-05T06:09:42.805 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/33 file after the snapshot 2019-12-05T06:09:42.809 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/34 file after the snapshot 2019-12-05T06:09:42.812 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/35 file after the snapshot 2019-12-05T06:09:42.815 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/36 file after the snapshot 2019-12-05T06:09:42.818 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/37 file after the snapshot 2019-12-05T06:09:42.820 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/38 file after the snapshot 2019-12-05T06:09:42.825 INFO:tasks.workunit.client.0.smithi016.stdout: create 279/39 file after the snapshot 2019-12-05T06:09:42.827 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/.snap/snap-subdir-test 2019-12-05T06:09:42.848 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/30 file after the snapshot 2019-12-05T06:09:42.851 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/31 file after the snapshot 2019-12-05T06:09:42.853 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/32 file after the snapshot 2019-12-05T06:09:42.856 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/33 file after the snapshot 2019-12-05T06:09:42.859 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/34 file after the snapshot 2019-12-05T06:09:42.862 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/35 file after the snapshot 2019-12-05T06:09:42.865 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/36 file after the snapshot 2019-12-05T06:09:42.868 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/37 file after the snapshot 2019-12-05T06:09:42.871 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/38 file after the snapshot 2019-12-05T06:09:42.873 INFO:tasks.workunit.client.0.smithi016.stdout: create 280/39 file after the snapshot 2019-12-05T06:09:42.876 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/.snap/snap-subdir-test 2019-12-05T06:09:43.465 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/30 file after the snapshot 2019-12-05T06:09:43.469 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/31 file after the snapshot 2019-12-05T06:09:43.473 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/32 file after the snapshot 2019-12-05T06:09:43.477 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/33 file after the snapshot 2019-12-05T06:09:43.500 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/34 file after the snapshot 2019-12-05T06:09:43.500 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/35 file after the snapshot 2019-12-05T06:09:43.500 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/36 file after the snapshot 2019-12-05T06:09:43.500 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/37 file after the snapshot 2019-12-05T06:09:43.501 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/38 file after the snapshot 2019-12-05T06:09:43.503 INFO:tasks.workunit.client.0.smithi016.stdout: create 281/39 file after the snapshot 2019-12-05T06:09:43.506 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/.snap/snap-subdir-test 2019-12-05T06:09:43.531 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/30 file after the snapshot 2019-12-05T06:09:43.534 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/31 file after the snapshot 2019-12-05T06:09:43.538 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/32 file after the snapshot 2019-12-05T06:09:43.541 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/33 file after the snapshot 2019-12-05T06:09:43.544 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/34 file after the snapshot 2019-12-05T06:09:43.548 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/35 file after the snapshot 2019-12-05T06:09:43.551 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/36 file after the snapshot 2019-12-05T06:09:43.555 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/37 file after the snapshot 2019-12-05T06:09:43.558 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/38 file after the snapshot 2019-12-05T06:09:43.560 INFO:tasks.workunit.client.0.smithi016.stdout: create 282/39 file after the snapshot 2019-12-05T06:09:43.563 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/.snap/snap-subdir-test 2019-12-05T06:09:44.043 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/30 file after the snapshot 2019-12-05T06:09:44.088 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/31 file after the snapshot 2019-12-05T06:09:45.028 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/32 file after the snapshot 2019-12-05T06:09:45.033 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/33 file after the snapshot 2019-12-05T06:09:45.038 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/34 file after the snapshot 2019-12-05T06:09:45.043 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/35 file after the snapshot 2019-12-05T06:09:45.048 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/36 file after the snapshot 2019-12-05T06:09:45.053 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/37 file after the snapshot 2019-12-05T06:09:45.056 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/38 file after the snapshot 2019-12-05T06:09:45.061 INFO:tasks.workunit.client.0.smithi016.stdout: create 283/39 file after the snapshot 2019-12-05T06:09:45.065 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/.snap/snap-subdir-test 2019-12-05T06:09:45.119 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/30 file after the snapshot 2019-12-05T06:09:45.122 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/31 file after the snapshot 2019-12-05T06:09:45.127 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/32 file after the snapshot 2019-12-05T06:09:45.130 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/33 file after the snapshot 2019-12-05T06:09:45.134 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/34 file after the snapshot 2019-12-05T06:09:45.138 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/35 file after the snapshot 2019-12-05T06:09:45.141 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/36 file after the snapshot 2019-12-05T06:09:45.145 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/37 file after the snapshot 2019-12-05T06:09:45.147 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/38 file after the snapshot 2019-12-05T06:09:45.151 INFO:tasks.workunit.client.0.smithi016.stdout: create 284/39 file after the snapshot 2019-12-05T06:09:45.155 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/.snap/snap-subdir-test 2019-12-05T06:09:45.933 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/30 file after the snapshot 2019-12-05T06:09:45.937 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/31 file after the snapshot 2019-12-05T06:09:45.942 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/32 file after the snapshot 2019-12-05T06:09:45.960 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/33 file after the snapshot 2019-12-05T06:09:45.960 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/34 file after the snapshot 2019-12-05T06:09:45.961 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/35 file after the snapshot 2019-12-05T06:09:46.011 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/36 file after the snapshot 2019-12-05T06:09:46.016 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/37 file after the snapshot 2019-12-05T06:09:46.020 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/38 file after the snapshot 2019-12-05T06:09:46.025 INFO:tasks.workunit.client.0.smithi016.stdout: create 285/39 file after the snapshot 2019-12-05T06:09:46.030 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/.snap/snap-subdir-test 2019-12-05T06:09:46.077 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/30 file after the snapshot 2019-12-05T06:09:46.080 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/31 file after the snapshot 2019-12-05T06:09:46.084 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/32 file after the snapshot 2019-12-05T06:09:46.088 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/33 file after the snapshot 2019-12-05T06:09:46.091 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/34 file after the snapshot 2019-12-05T06:09:46.094 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/35 file after the snapshot 2019-12-05T06:09:46.098 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/36 file after the snapshot 2019-12-05T06:09:46.100 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/37 file after the snapshot 2019-12-05T06:09:46.103 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/38 file after the snapshot 2019-12-05T06:09:46.107 INFO:tasks.workunit.client.0.smithi016.stdout: create 286/39 file after the snapshot 2019-12-05T06:09:46.110 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/.snap/snap-subdir-test 2019-12-05T06:09:46.131 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/30 file after the snapshot 2019-12-05T06:09:46.133 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/31 file after the snapshot 2019-12-05T06:09:46.137 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/32 file after the snapshot 2019-12-05T06:09:46.140 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/33 file after the snapshot 2019-12-05T06:09:46.142 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/34 file after the snapshot 2019-12-05T06:09:46.146 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/35 file after the snapshot 2019-12-05T06:09:46.149 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/36 file after the snapshot 2019-12-05T06:09:46.152 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/37 file after the snapshot 2019-12-05T06:09:46.154 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/38 file after the snapshot 2019-12-05T06:09:46.157 INFO:tasks.workunit.client.0.smithi016.stdout: create 287/39 file after the snapshot 2019-12-05T06:09:46.161 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/.snap/snap-subdir-test 2019-12-05T06:09:46.873 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/30 file after the snapshot 2019-12-05T06:09:46.875 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/31 file after the snapshot 2019-12-05T06:09:46.881 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/32 file after the snapshot 2019-12-05T06:09:46.885 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/33 file after the snapshot 2019-12-05T06:09:46.889 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/34 file after the snapshot 2019-12-05T06:09:46.895 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/35 file after the snapshot 2019-12-05T06:09:46.901 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/36 file after the snapshot 2019-12-05T06:09:46.912 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/37 file after the snapshot 2019-12-05T06:09:46.912 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/38 file after the snapshot 2019-12-05T06:09:46.918 INFO:tasks.workunit.client.0.smithi016.stdout: create 288/39 file after the snapshot 2019-12-05T06:09:46.918 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/.snap/snap-subdir-test 2019-12-05T06:09:47.420 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/30 file after the snapshot 2019-12-05T06:09:47.425 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/31 file after the snapshot 2019-12-05T06:09:47.430 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/32 file after the snapshot 2019-12-05T06:09:47.435 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/33 file after the snapshot 2019-12-05T06:09:47.440 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/34 file after the snapshot 2019-12-05T06:09:47.444 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/35 file after the snapshot 2019-12-05T06:09:47.450 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/36 file after the snapshot 2019-12-05T06:09:47.455 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/37 file after the snapshot 2019-12-05T06:09:47.461 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/38 file after the snapshot 2019-12-05T06:09:47.464 INFO:tasks.workunit.client.0.smithi016.stdout: create 289/39 file after the snapshot 2019-12-05T06:09:47.471 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/.snap/snap-subdir-test 2019-12-05T06:09:47.517 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/30 file after the snapshot 2019-12-05T06:09:47.521 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/31 file after the snapshot 2019-12-05T06:09:47.524 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/32 file after the snapshot 2019-12-05T06:09:47.528 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/33 file after the snapshot 2019-12-05T06:09:47.532 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/34 file after the snapshot 2019-12-05T06:09:47.536 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/35 file after the snapshot 2019-12-05T06:09:47.540 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/36 file after the snapshot 2019-12-05T06:09:47.544 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/37 file after the snapshot 2019-12-05T06:09:47.547 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/38 file after the snapshot 2019-12-05T06:09:47.550 INFO:tasks.workunit.client.0.smithi016.stdout: create 290/39 file after the snapshot 2019-12-05T06:09:47.554 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/.snap/snap-subdir-test 2019-12-05T06:09:47.577 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/30 file after the snapshot 2019-12-05T06:09:47.580 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/31 file after the snapshot 2019-12-05T06:09:47.583 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/32 file after the snapshot 2019-12-05T06:09:47.587 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/33 file after the snapshot 2019-12-05T06:09:47.590 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/34 file after the snapshot 2019-12-05T06:09:47.594 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/35 file after the snapshot 2019-12-05T06:09:47.598 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/36 file after the snapshot 2019-12-05T06:09:47.601 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/37 file after the snapshot 2019-12-05T06:09:47.604 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/38 file after the snapshot 2019-12-05T06:09:47.607 INFO:tasks.workunit.client.0.smithi016.stdout: create 291/39 file after the snapshot 2019-12-05T06:09:47.611 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/.snap/snap-subdir-test 2019-12-05T06:09:47.631 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/30 file after the snapshot 2019-12-05T06:09:47.633 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/31 file after the snapshot 2019-12-05T06:09:47.636 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/32 file after the snapshot 2019-12-05T06:09:47.640 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/33 file after the snapshot 2019-12-05T06:09:47.642 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/34 file after the snapshot 2019-12-05T06:09:47.645 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/35 file after the snapshot 2019-12-05T06:09:47.647 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/36 file after the snapshot 2019-12-05T06:09:47.650 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/37 file after the snapshot 2019-12-05T06:09:47.652 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/38 file after the snapshot 2019-12-05T06:09:47.655 INFO:tasks.workunit.client.0.smithi016.stdout: create 292/39 file after the snapshot 2019-12-05T06:09:47.659 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/.snap/snap-subdir-test 2019-12-05T06:09:47.677 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/30 file after the snapshot 2019-12-05T06:09:47.678 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/31 file after the snapshot 2019-12-05T06:09:47.680 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/32 file after the snapshot 2019-12-05T06:09:47.682 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/33 file after the snapshot 2019-12-05T06:09:47.685 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/34 file after the snapshot 2019-12-05T06:09:47.687 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/35 file after the snapshot 2019-12-05T06:09:47.690 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/36 file after the snapshot 2019-12-05T06:09:47.692 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/37 file after the snapshot 2019-12-05T06:09:47.694 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/38 file after the snapshot 2019-12-05T06:09:47.697 INFO:tasks.workunit.client.0.smithi016.stdout: create 293/39 file after the snapshot 2019-12-05T06:09:47.699 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/.snap/snap-subdir-test 2019-12-05T06:09:48.313 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/30 file after the snapshot 2019-12-05T06:09:48.318 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/31 file after the snapshot 2019-12-05T06:09:48.323 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/32 file after the snapshot 2019-12-05T06:09:48.333 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/33 file after the snapshot 2019-12-05T06:09:48.333 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/34 file after the snapshot 2019-12-05T06:09:48.336 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/35 file after the snapshot 2019-12-05T06:09:48.341 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/36 file after the snapshot 2019-12-05T06:09:48.346 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/37 file after the snapshot 2019-12-05T06:09:48.351 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/38 file after the snapshot 2019-12-05T06:09:48.355 INFO:tasks.workunit.client.0.smithi016.stdout: create 294/39 file after the snapshot 2019-12-05T06:09:48.360 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/.snap/snap-subdir-test 2019-12-05T06:09:48.386 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/30 file after the snapshot 2019-12-05T06:09:48.389 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/31 file after the snapshot 2019-12-05T06:09:48.393 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/32 file after the snapshot 2019-12-05T06:09:48.397 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/33 file after the snapshot 2019-12-05T06:09:48.401 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/34 file after the snapshot 2019-12-05T06:09:48.406 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/35 file after the snapshot 2019-12-05T06:09:48.409 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/36 file after the snapshot 2019-12-05T06:09:48.414 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/37 file after the snapshot 2019-12-05T06:09:48.417 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/38 file after the snapshot 2019-12-05T06:09:48.420 INFO:tasks.workunit.client.0.smithi016.stdout: create 295/39 file after the snapshot 2019-12-05T06:09:48.424 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/.snap/snap-subdir-test 2019-12-05T06:09:48.445 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/30 file after the snapshot 2019-12-05T06:09:48.448 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/31 file after the snapshot 2019-12-05T06:09:48.450 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/32 file after the snapshot 2019-12-05T06:09:48.454 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/33 file after the snapshot 2019-12-05T06:09:48.458 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/34 file after the snapshot 2019-12-05T06:09:48.461 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/35 file after the snapshot 2019-12-05T06:09:48.464 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/36 file after the snapshot 2019-12-05T06:09:48.467 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/37 file after the snapshot 2019-12-05T06:09:48.470 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/38 file after the snapshot 2019-12-05T06:09:48.473 INFO:tasks.workunit.client.0.smithi016.stdout: create 296/39 file after the snapshot 2019-12-05T06:09:48.475 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/.snap/snap-subdir-test 2019-12-05T06:09:48.495 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/30 file after the snapshot 2019-12-05T06:09:48.498 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/31 file after the snapshot 2019-12-05T06:09:48.501 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/32 file after the snapshot 2019-12-05T06:09:48.504 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/33 file after the snapshot 2019-12-05T06:09:48.507 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/34 file after the snapshot 2019-12-05T06:09:48.510 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/35 file after the snapshot 2019-12-05T06:09:48.512 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/36 file after the snapshot 2019-12-05T06:09:48.515 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/37 file after the snapshot 2019-12-05T06:09:48.518 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/38 file after the snapshot 2019-12-05T06:09:48.520 INFO:tasks.workunit.client.0.smithi016.stdout: create 297/39 file after the snapshot 2019-12-05T06:09:48.523 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/.snap/snap-subdir-test 2019-12-05T06:09:48.544 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/30 file after the snapshot 2019-12-05T06:09:48.547 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/31 file after the snapshot 2019-12-05T06:09:48.549 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/32 file after the snapshot 2019-12-05T06:09:48.552 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/33 file after the snapshot 2019-12-05T06:09:48.555 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/34 file after the snapshot 2019-12-05T06:09:48.559 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/35 file after the snapshot 2019-12-05T06:09:48.561 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/36 file after the snapshot 2019-12-05T06:09:48.563 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/37 file after the snapshot 2019-12-05T06:09:48.566 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/38 file after the snapshot 2019-12-05T06:09:48.569 INFO:tasks.workunit.client.0.smithi016.stdout: create 298/39 file after the snapshot 2019-12-05T06:09:48.572 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/.snap/snap-subdir-test 2019-12-05T06:09:48.589 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/30 file after the snapshot 2019-12-05T06:09:48.591 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/31 file after the snapshot 2019-12-05T06:09:48.594 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/32 file after the snapshot 2019-12-05T06:09:48.596 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/33 file after the snapshot 2019-12-05T06:09:48.599 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/34 file after the snapshot 2019-12-05T06:09:48.602 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/35 file after the snapshot 2019-12-05T06:09:48.604 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/36 file after the snapshot 2019-12-05T06:09:48.607 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/37 file after the snapshot 2019-12-05T06:09:48.609 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/38 file after the snapshot 2019-12-05T06:09:48.611 INFO:tasks.workunit.client.0.smithi016.stdout: create 299/39 file after the snapshot 2019-12-05T06:09:48.614 INFO:tasks.workunit.client.0.smithi016.stdout:Take second snapshot .snap/test2 2019-12-05T06:09:48.756 INFO:tasks.workunit.client.0.smithi016.stdout:Copy content of .snap/test1 to copyofsnap1 ... 2019-12-05T06:09:52.404 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1’ -> ‘copyofsnap1/test1’ 2019-12-05T06:09:52.405 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166’ -> ‘copyofsnap1/test1/166’ 2019-12-05T06:09:52.405 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/19’ -> ‘copyofsnap1/test1/166/19’ 2019-12-05T06:09:52.405 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/15’ -> ‘copyofsnap1/test1/166/15’ 2019-12-05T06:09:52.405 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/20’ -> ‘copyofsnap1/test1/166/20’ 2019-12-05T06:09:52.405 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/12’ -> ‘copyofsnap1/test1/166/12’ 2019-12-05T06:09:52.405 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/10’ -> ‘copyofsnap1/test1/166/10’ 2019-12-05T06:09:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/11’ -> ‘copyofsnap1/test1/166/11’ 2019-12-05T06:09:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/16’ -> ‘copyofsnap1/test1/166/16’ 2019-12-05T06:09:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/14’ -> ‘copyofsnap1/test1/166/14’ 2019-12-05T06:09:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/18’ -> ‘copyofsnap1/test1/166/18’ 2019-12-05T06:09:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/13’ -> ‘copyofsnap1/test1/166/13’ 2019-12-05T06:09:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/166/17’ -> ‘copyofsnap1/test1/166/17’ 2019-12-05T06:09:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147’ -> ‘copyofsnap1/test1/147’ 2019-12-05T06:09:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/19’ -> ‘copyofsnap1/test1/147/19’ 2019-12-05T06:09:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/15’ -> ‘copyofsnap1/test1/147/15’ 2019-12-05T06:09:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/20’ -> ‘copyofsnap1/test1/147/20’ 2019-12-05T06:09:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/12’ -> ‘copyofsnap1/test1/147/12’ 2019-12-05T06:09:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/10’ -> ‘copyofsnap1/test1/147/10’ 2019-12-05T06:09:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/11’ -> ‘copyofsnap1/test1/147/11’ 2019-12-05T06:09:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/16’ -> ‘copyofsnap1/test1/147/16’ 2019-12-05T06:09:52.408 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/14’ -> ‘copyofsnap1/test1/147/14’ 2019-12-05T06:09:52.408 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/18’ -> ‘copyofsnap1/test1/147/18’ 2019-12-05T06:09:52.408 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/13’ -> ‘copyofsnap1/test1/147/13’ 2019-12-05T06:09:52.408 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/147/17’ -> ‘copyofsnap1/test1/147/17’ 2019-12-05T06:09:52.408 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151’ -> ‘copyofsnap1/test1/151’ 2019-12-05T06:09:52.408 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/19’ -> ‘copyofsnap1/test1/151/19’ 2019-12-05T06:09:52.408 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/15’ -> ‘copyofsnap1/test1/151/15’ 2019-12-05T06:09:52.409 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/20’ -> ‘copyofsnap1/test1/151/20’ 2019-12-05T06:09:52.409 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/12’ -> ‘copyofsnap1/test1/151/12’ 2019-12-05T06:09:52.409 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/10’ -> ‘copyofsnap1/test1/151/10’ 2019-12-05T06:09:52.409 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/11’ -> ‘copyofsnap1/test1/151/11’ 2019-12-05T06:09:52.409 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/16’ -> ‘copyofsnap1/test1/151/16’ 2019-12-05T06:09:52.409 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/14’ -> ‘copyofsnap1/test1/151/14’ 2019-12-05T06:09:52.409 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/18’ -> ‘copyofsnap1/test1/151/18’ 2019-12-05T06:09:52.410 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/13’ -> ‘copyofsnap1/test1/151/13’ 2019-12-05T06:09:52.410 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/151/17’ -> ‘copyofsnap1/test1/151/17’ 2019-12-05T06:09:52.410 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170’ -> ‘copyofsnap1/test1/170’ 2019-12-05T06:09:52.410 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/19’ -> ‘copyofsnap1/test1/170/19’ 2019-12-05T06:09:52.410 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/15’ -> ‘copyofsnap1/test1/170/15’ 2019-12-05T06:09:52.410 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/20’ -> ‘copyofsnap1/test1/170/20’ 2019-12-05T06:09:52.410 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/12’ -> ‘copyofsnap1/test1/170/12’ 2019-12-05T06:09:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/10’ -> ‘copyofsnap1/test1/170/10’ 2019-12-05T06:09:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/11’ -> ‘copyofsnap1/test1/170/11’ 2019-12-05T06:09:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/16’ -> ‘copyofsnap1/test1/170/16’ 2019-12-05T06:09:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/14’ -> ‘copyofsnap1/test1/170/14’ 2019-12-05T06:09:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/18’ -> ‘copyofsnap1/test1/170/18’ 2019-12-05T06:09:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/13’ -> ‘copyofsnap1/test1/170/13’ 2019-12-05T06:09:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/170/17’ -> ‘copyofsnap1/test1/170/17’ 2019-12-05T06:09:52.412 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167’ -> ‘copyofsnap1/test1/167’ 2019-12-05T06:09:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/19’ -> ‘copyofsnap1/test1/167/19’ 2019-12-05T06:09:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/15’ -> ‘copyofsnap1/test1/167/15’ 2019-12-05T06:09:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/20’ -> ‘copyofsnap1/test1/167/20’ 2019-12-05T06:09:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/12’ -> ‘copyofsnap1/test1/167/12’ 2019-12-05T06:09:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/10’ -> ‘copyofsnap1/test1/167/10’ 2019-12-05T06:09:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/11’ -> ‘copyofsnap1/test1/167/11’ 2019-12-05T06:09:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/16’ -> ‘copyofsnap1/test1/167/16’ 2019-12-05T06:09:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/14’ -> ‘copyofsnap1/test1/167/14’ 2019-12-05T06:09:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/18’ -> ‘copyofsnap1/test1/167/18’ 2019-12-05T06:09:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/13’ -> ‘copyofsnap1/test1/167/13’ 2019-12-05T06:09:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/167/17’ -> ‘copyofsnap1/test1/167/17’ 2019-12-05T06:09:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140’ -> ‘copyofsnap1/test1/140’ 2019-12-05T06:09:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/19’ -> ‘copyofsnap1/test1/140/19’ 2019-12-05T06:09:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/15’ -> ‘copyofsnap1/test1/140/15’ 2019-12-05T06:09:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/20’ -> ‘copyofsnap1/test1/140/20’ 2019-12-05T06:09:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/12’ -> ‘copyofsnap1/test1/140/12’ 2019-12-05T06:09:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/10’ -> ‘copyofsnap1/test1/140/10’ 2019-12-05T06:09:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/11’ -> ‘copyofsnap1/test1/140/11’ 2019-12-05T06:09:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/16’ -> ‘copyofsnap1/test1/140/16’ 2019-12-05T06:09:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/14’ -> ‘copyofsnap1/test1/140/14’ 2019-12-05T06:09:55.090 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/18’ -> ‘copyofsnap1/test1/140/18’ 2019-12-05T06:09:55.090 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/13’ -> ‘copyofsnap1/test1/140/13’ 2019-12-05T06:09:55.090 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/140/17’ -> ‘copyofsnap1/test1/140/17’ 2019-12-05T06:09:55.090 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142’ -> ‘copyofsnap1/test1/142’ 2019-12-05T06:09:55.090 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/19’ -> ‘copyofsnap1/test1/142/19’ 2019-12-05T06:09:55.091 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/15’ -> ‘copyofsnap1/test1/142/15’ 2019-12-05T06:09:55.091 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/20’ -> ‘copyofsnap1/test1/142/20’ 2019-12-05T06:09:55.091 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/12’ -> ‘copyofsnap1/test1/142/12’ 2019-12-05T06:09:55.091 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/10’ -> ‘copyofsnap1/test1/142/10’ 2019-12-05T06:09:55.091 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/11’ -> ‘copyofsnap1/test1/142/11’ 2019-12-05T06:09:55.092 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/16’ -> ‘copyofsnap1/test1/142/16’ 2019-12-05T06:09:55.092 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/14’ -> ‘copyofsnap1/test1/142/14’ 2019-12-05T06:09:55.092 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/18’ -> ‘copyofsnap1/test1/142/18’ 2019-12-05T06:09:55.092 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/13’ -> ‘copyofsnap1/test1/142/13’ 2019-12-05T06:09:55.092 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/142/17’ -> ‘copyofsnap1/test1/142/17’ 2019-12-05T06:09:55.092 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150’ -> ‘copyofsnap1/test1/150’ 2019-12-05T06:09:55.093 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/19’ -> ‘copyofsnap1/test1/150/19’ 2019-12-05T06:09:55.093 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/15’ -> ‘copyofsnap1/test1/150/15’ 2019-12-05T06:09:55.093 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/20’ -> ‘copyofsnap1/test1/150/20’ 2019-12-05T06:09:55.093 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/12’ -> ‘copyofsnap1/test1/150/12’ 2019-12-05T06:09:55.093 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/10’ -> ‘copyofsnap1/test1/150/10’ 2019-12-05T06:09:55.093 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/11’ -> ‘copyofsnap1/test1/150/11’ 2019-12-05T06:09:55.094 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/16’ -> ‘copyofsnap1/test1/150/16’ 2019-12-05T06:09:55.094 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/14’ -> ‘copyofsnap1/test1/150/14’ 2019-12-05T06:09:55.094 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/18’ -> ‘copyofsnap1/test1/150/18’ 2019-12-05T06:09:55.094 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/13’ -> ‘copyofsnap1/test1/150/13’ 2019-12-05T06:09:55.094 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/150/17’ -> ‘copyofsnap1/test1/150/17’ 2019-12-05T06:09:55.094 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105’ -> ‘copyofsnap1/test1/105’ 2019-12-05T06:09:55.094 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/19’ -> ‘copyofsnap1/test1/105/19’ 2019-12-05T06:09:55.095 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/15’ -> ‘copyofsnap1/test1/105/15’ 2019-12-05T06:09:55.095 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/20’ -> ‘copyofsnap1/test1/105/20’ 2019-12-05T06:09:55.095 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/12’ -> ‘copyofsnap1/test1/105/12’ 2019-12-05T06:09:55.095 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/10’ -> ‘copyofsnap1/test1/105/10’ 2019-12-05T06:09:55.095 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/11’ -> ‘copyofsnap1/test1/105/11’ 2019-12-05T06:09:55.095 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/16’ -> ‘copyofsnap1/test1/105/16’ 2019-12-05T06:09:55.095 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/14’ -> ‘copyofsnap1/test1/105/14’ 2019-12-05T06:09:55.096 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/18’ -> ‘copyofsnap1/test1/105/18’ 2019-12-05T06:09:55.096 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/13’ -> ‘copyofsnap1/test1/105/13’ 2019-12-05T06:09:55.096 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/105/17’ -> ‘copyofsnap1/test1/105/17’ 2019-12-05T06:09:55.096 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149’ -> ‘copyofsnap1/test1/149’ 2019-12-05T06:09:55.096 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/19’ -> ‘copyofsnap1/test1/149/19’ 2019-12-05T06:09:55.096 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/15’ -> ‘copyofsnap1/test1/149/15’ 2019-12-05T06:09:55.097 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/20’ -> ‘copyofsnap1/test1/149/20’ 2019-12-05T06:09:55.097 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/12’ -> ‘copyofsnap1/test1/149/12’ 2019-12-05T06:09:55.097 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/10’ -> ‘copyofsnap1/test1/149/10’ 2019-12-05T06:09:55.097 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/11’ -> ‘copyofsnap1/test1/149/11’ 2019-12-05T06:09:55.097 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/16’ -> ‘copyofsnap1/test1/149/16’ 2019-12-05T06:09:55.098 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/14’ -> ‘copyofsnap1/test1/149/14’ 2019-12-05T06:09:55.098 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/18’ -> ‘copyofsnap1/test1/149/18’ 2019-12-05T06:09:55.098 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/13’ -> ‘copyofsnap1/test1/149/13’ 2019-12-05T06:09:55.098 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/149/17’ -> ‘copyofsnap1/test1/149/17’ 2019-12-05T06:09:55.098 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109’ -> ‘copyofsnap1/test1/109’ 2019-12-05T06:09:55.098 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/19’ -> ‘copyofsnap1/test1/109/19’ 2019-12-05T06:09:55.098 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/15’ -> ‘copyofsnap1/test1/109/15’ 2019-12-05T06:09:55.099 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/20’ -> ‘copyofsnap1/test1/109/20’ 2019-12-05T06:09:55.099 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/12’ -> ‘copyofsnap1/test1/109/12’ 2019-12-05T06:09:55.099 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/10’ -> ‘copyofsnap1/test1/109/10’ 2019-12-05T06:09:55.099 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/11’ -> ‘copyofsnap1/test1/109/11’ 2019-12-05T06:09:55.099 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/16’ -> ‘copyofsnap1/test1/109/16’ 2019-12-05T06:09:55.099 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/14’ -> ‘copyofsnap1/test1/109/14’ 2019-12-05T06:09:55.100 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/18’ -> ‘copyofsnap1/test1/109/18’ 2019-12-05T06:09:55.100 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/13’ -> ‘copyofsnap1/test1/109/13’ 2019-12-05T06:09:55.100 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/109/17’ -> ‘copyofsnap1/test1/109/17’ 2019-12-05T06:09:55.100 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158’ -> ‘copyofsnap1/test1/158’ 2019-12-05T06:09:55.100 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/19’ -> ‘copyofsnap1/test1/158/19’ 2019-12-05T06:09:55.100 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/15’ -> ‘copyofsnap1/test1/158/15’ 2019-12-05T06:09:55.100 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/20’ -> ‘copyofsnap1/test1/158/20’ 2019-12-05T06:09:55.101 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/12’ -> ‘copyofsnap1/test1/158/12’ 2019-12-05T06:09:55.101 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/10’ -> ‘copyofsnap1/test1/158/10’ 2019-12-05T06:09:55.101 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/11’ -> ‘copyofsnap1/test1/158/11’ 2019-12-05T06:09:57.867 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/16’ -> ‘copyofsnap1/test1/158/16’ 2019-12-05T06:09:57.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/14’ -> ‘copyofsnap1/test1/158/14’ 2019-12-05T06:09:57.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/18’ -> ‘copyofsnap1/test1/158/18’ 2019-12-05T06:09:57.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/13’ -> ‘copyofsnap1/test1/158/13’ 2019-12-05T06:09:57.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/158/17’ -> ‘copyofsnap1/test1/158/17’ 2019-12-05T06:09:57.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106’ -> ‘copyofsnap1/test1/106’ 2019-12-05T06:09:57.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/19’ -> ‘copyofsnap1/test1/106/19’ 2019-12-05T06:09:57.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/15’ -> ‘copyofsnap1/test1/106/15’ 2019-12-05T06:09:57.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/20’ -> ‘copyofsnap1/test1/106/20’ 2019-12-05T06:09:57.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/12’ -> ‘copyofsnap1/test1/106/12’ 2019-12-05T06:09:57.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/10’ -> ‘copyofsnap1/test1/106/10’ 2019-12-05T06:09:57.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/11’ -> ‘copyofsnap1/test1/106/11’ 2019-12-05T06:09:57.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/16’ -> ‘copyofsnap1/test1/106/16’ 2019-12-05T06:09:57.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/14’ -> ‘copyofsnap1/test1/106/14’ 2019-12-05T06:09:57.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/18’ -> ‘copyofsnap1/test1/106/18’ 2019-12-05T06:09:57.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/13’ -> ‘copyofsnap1/test1/106/13’ 2019-12-05T06:09:57.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/106/17’ -> ‘copyofsnap1/test1/106/17’ 2019-12-05T06:09:57.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189’ -> ‘copyofsnap1/test1/189’ 2019-12-05T06:09:57.871 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/19’ -> ‘copyofsnap1/test1/189/19’ 2019-12-05T06:09:57.871 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/15’ -> ‘copyofsnap1/test1/189/15’ 2019-12-05T06:09:57.871 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/20’ -> ‘copyofsnap1/test1/189/20’ 2019-12-05T06:09:57.871 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/12’ -> ‘copyofsnap1/test1/189/12’ 2019-12-05T06:09:57.871 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/10’ -> ‘copyofsnap1/test1/189/10’ 2019-12-05T06:09:57.871 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/11’ -> ‘copyofsnap1/test1/189/11’ 2019-12-05T06:09:57.872 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/16’ -> ‘copyofsnap1/test1/189/16’ 2019-12-05T06:09:57.872 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/14’ -> ‘copyofsnap1/test1/189/14’ 2019-12-05T06:09:57.872 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/18’ -> ‘copyofsnap1/test1/189/18’ 2019-12-05T06:09:57.872 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/13’ -> ‘copyofsnap1/test1/189/13’ 2019-12-05T06:09:57.872 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/189/17’ -> ‘copyofsnap1/test1/189/17’ 2019-12-05T06:09:57.872 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133’ -> ‘copyofsnap1/test1/133’ 2019-12-05T06:09:57.872 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/19’ -> ‘copyofsnap1/test1/133/19’ 2019-12-05T06:09:57.873 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/15’ -> ‘copyofsnap1/test1/133/15’ 2019-12-05T06:09:57.873 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/20’ -> ‘copyofsnap1/test1/133/20’ 2019-12-05T06:09:57.873 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/12’ -> ‘copyofsnap1/test1/133/12’ 2019-12-05T06:09:57.873 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/10’ -> ‘copyofsnap1/test1/133/10’ 2019-12-05T06:09:57.873 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/11’ -> ‘copyofsnap1/test1/133/11’ 2019-12-05T06:09:57.873 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/16’ -> ‘copyofsnap1/test1/133/16’ 2019-12-05T06:09:57.873 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/14’ -> ‘copyofsnap1/test1/133/14’ 2019-12-05T06:09:57.874 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/18’ -> ‘copyofsnap1/test1/133/18’ 2019-12-05T06:09:57.874 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/13’ -> ‘copyofsnap1/test1/133/13’ 2019-12-05T06:09:57.874 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/133/17’ -> ‘copyofsnap1/test1/133/17’ 2019-12-05T06:09:57.874 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173’ -> ‘copyofsnap1/test1/173’ 2019-12-05T06:09:57.874 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/19’ -> ‘copyofsnap1/test1/173/19’ 2019-12-05T06:09:57.874 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/15’ -> ‘copyofsnap1/test1/173/15’ 2019-12-05T06:09:57.875 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/20’ -> ‘copyofsnap1/test1/173/20’ 2019-12-05T06:09:57.875 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/12’ -> ‘copyofsnap1/test1/173/12’ 2019-12-05T06:09:57.875 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/10’ -> ‘copyofsnap1/test1/173/10’ 2019-12-05T06:09:57.875 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/11’ -> ‘copyofsnap1/test1/173/11’ 2019-12-05T06:09:57.875 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/16’ -> ‘copyofsnap1/test1/173/16’ 2019-12-05T06:09:57.875 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/14’ -> ‘copyofsnap1/test1/173/14’ 2019-12-05T06:09:57.876 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/18’ -> ‘copyofsnap1/test1/173/18’ 2019-12-05T06:09:57.876 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/13’ -> ‘copyofsnap1/test1/173/13’ 2019-12-05T06:09:57.876 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/173/17’ -> ‘copyofsnap1/test1/173/17’ 2019-12-05T06:09:57.876 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174’ -> ‘copyofsnap1/test1/174’ 2019-12-05T06:09:57.876 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/19’ -> ‘copyofsnap1/test1/174/19’ 2019-12-05T06:09:57.876 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/15’ -> ‘copyofsnap1/test1/174/15’ 2019-12-05T06:09:57.876 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/20’ -> ‘copyofsnap1/test1/174/20’ 2019-12-05T06:09:57.877 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/12’ -> ‘copyofsnap1/test1/174/12’ 2019-12-05T06:09:57.877 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/10’ -> ‘copyofsnap1/test1/174/10’ 2019-12-05T06:09:57.877 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/11’ -> ‘copyofsnap1/test1/174/11’ 2019-12-05T06:09:57.877 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/16’ -> ‘copyofsnap1/test1/174/16’ 2019-12-05T06:09:57.877 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/14’ -> ‘copyofsnap1/test1/174/14’ 2019-12-05T06:09:57.877 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/18’ -> ‘copyofsnap1/test1/174/18’ 2019-12-05T06:09:57.878 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/13’ -> ‘copyofsnap1/test1/174/13’ 2019-12-05T06:09:57.878 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/174/17’ -> ‘copyofsnap1/test1/174/17’ 2019-12-05T06:09:57.878 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155’ -> ‘copyofsnap1/test1/155’ 2019-12-05T06:09:57.878 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/19’ -> ‘copyofsnap1/test1/155/19’ 2019-12-05T06:09:57.878 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/15’ -> ‘copyofsnap1/test1/155/15’ 2019-12-05T06:09:57.878 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/20’ -> ‘copyofsnap1/test1/155/20’ 2019-12-05T06:09:57.878 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/12’ -> ‘copyofsnap1/test1/155/12’ 2019-12-05T06:09:58.442 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:09:58.446 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:10:01.712 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/10’ -> ‘copyofsnap1/test1/155/10’ 2019-12-05T06:10:01.712 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/11’ -> ‘copyofsnap1/test1/155/11’ 2019-12-05T06:10:01.712 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/16’ -> ‘copyofsnap1/test1/155/16’ 2019-12-05T06:10:01.712 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/14’ -> ‘copyofsnap1/test1/155/14’ 2019-12-05T06:10:01.712 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/18’ -> ‘copyofsnap1/test1/155/18’ 2019-12-05T06:10:01.713 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/13’ -> ‘copyofsnap1/test1/155/13’ 2019-12-05T06:10:01.713 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/155/17’ -> ‘copyofsnap1/test1/155/17’ 2019-12-05T06:10:01.713 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114’ -> ‘copyofsnap1/test1/114’ 2019-12-05T06:10:01.713 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/19’ -> ‘copyofsnap1/test1/114/19’ 2019-12-05T06:10:01.713 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/15’ -> ‘copyofsnap1/test1/114/15’ 2019-12-05T06:10:01.713 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/20’ -> ‘copyofsnap1/test1/114/20’ 2019-12-05T06:10:01.714 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/12’ -> ‘copyofsnap1/test1/114/12’ 2019-12-05T06:10:01.714 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/10’ -> ‘copyofsnap1/test1/114/10’ 2019-12-05T06:10:01.714 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/11’ -> ‘copyofsnap1/test1/114/11’ 2019-12-05T06:10:01.714 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/16’ -> ‘copyofsnap1/test1/114/16’ 2019-12-05T06:10:01.714 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/14’ -> ‘copyofsnap1/test1/114/14’ 2019-12-05T06:10:01.714 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/18’ -> ‘copyofsnap1/test1/114/18’ 2019-12-05T06:10:01.715 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/13’ -> ‘copyofsnap1/test1/114/13’ 2019-12-05T06:10:01.715 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/114/17’ -> ‘copyofsnap1/test1/114/17’ 2019-12-05T06:10:01.716 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159’ -> ‘copyofsnap1/test1/159’ 2019-12-05T06:10:01.716 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/19’ -> ‘copyofsnap1/test1/159/19’ 2019-12-05T06:10:01.716 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/15’ -> ‘copyofsnap1/test1/159/15’ 2019-12-05T06:10:01.716 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/20’ -> ‘copyofsnap1/test1/159/20’ 2019-12-05T06:10:01.717 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/12’ -> ‘copyofsnap1/test1/159/12’ 2019-12-05T06:10:01.717 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/10’ -> ‘copyofsnap1/test1/159/10’ 2019-12-05T06:10:01.717 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/11’ -> ‘copyofsnap1/test1/159/11’ 2019-12-05T06:10:01.717 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/16’ -> ‘copyofsnap1/test1/159/16’ 2019-12-05T06:10:01.717 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/14’ -> ‘copyofsnap1/test1/159/14’ 2019-12-05T06:10:01.717 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/18’ -> ‘copyofsnap1/test1/159/18’ 2019-12-05T06:10:01.718 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/13’ -> ‘copyofsnap1/test1/159/13’ 2019-12-05T06:10:01.718 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/159/17’ -> ‘copyofsnap1/test1/159/17’ 2019-12-05T06:10:01.718 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187’ -> ‘copyofsnap1/test1/187’ 2019-12-05T06:10:01.718 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/19’ -> ‘copyofsnap1/test1/187/19’ 2019-12-05T06:10:01.718 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/15’ -> ‘copyofsnap1/test1/187/15’ 2019-12-05T06:10:01.718 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/20’ -> ‘copyofsnap1/test1/187/20’ 2019-12-05T06:10:01.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/12’ -> ‘copyofsnap1/test1/187/12’ 2019-12-05T06:10:01.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/10’ -> ‘copyofsnap1/test1/187/10’ 2019-12-05T06:10:01.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/11’ -> ‘copyofsnap1/test1/187/11’ 2019-12-05T06:10:01.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/16’ -> ‘copyofsnap1/test1/187/16’ 2019-12-05T06:10:01.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/14’ -> ‘copyofsnap1/test1/187/14’ 2019-12-05T06:10:01.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/18’ -> ‘copyofsnap1/test1/187/18’ 2019-12-05T06:10:01.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/13’ -> ‘copyofsnap1/test1/187/13’ 2019-12-05T06:10:01.720 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/187/17’ -> ‘copyofsnap1/test1/187/17’ 2019-12-05T06:10:01.720 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185’ -> ‘copyofsnap1/test1/185’ 2019-12-05T06:10:01.720 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/19’ -> ‘copyofsnap1/test1/185/19’ 2019-12-05T06:10:01.720 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/15’ -> ‘copyofsnap1/test1/185/15’ 2019-12-05T06:10:01.720 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/20’ -> ‘copyofsnap1/test1/185/20’ 2019-12-05T06:10:01.720 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/12’ -> ‘copyofsnap1/test1/185/12’ 2019-12-05T06:10:01.721 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/10’ -> ‘copyofsnap1/test1/185/10’ 2019-12-05T06:10:01.721 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/11’ -> ‘copyofsnap1/test1/185/11’ 2019-12-05T06:10:01.721 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/16’ -> ‘copyofsnap1/test1/185/16’ 2019-12-05T06:10:01.721 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/14’ -> ‘copyofsnap1/test1/185/14’ 2019-12-05T06:10:01.721 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/18’ -> ‘copyofsnap1/test1/185/18’ 2019-12-05T06:10:01.722 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/13’ -> ‘copyofsnap1/test1/185/13’ 2019-12-05T06:10:01.722 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/185/17’ -> ‘copyofsnap1/test1/185/17’ 2019-12-05T06:10:01.722 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120’ -> ‘copyofsnap1/test1/120’ 2019-12-05T06:10:01.722 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/19’ -> ‘copyofsnap1/test1/120/19’ 2019-12-05T06:10:01.722 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/15’ -> ‘copyofsnap1/test1/120/15’ 2019-12-05T06:10:01.722 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/20’ -> ‘copyofsnap1/test1/120/20’ 2019-12-05T06:10:01.723 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/12’ -> ‘copyofsnap1/test1/120/12’ 2019-12-05T06:10:01.723 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/10’ -> ‘copyofsnap1/test1/120/10’ 2019-12-05T06:10:01.723 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/11’ -> ‘copyofsnap1/test1/120/11’ 2019-12-05T06:10:01.723 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/16’ -> ‘copyofsnap1/test1/120/16’ 2019-12-05T06:10:01.723 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/14’ -> ‘copyofsnap1/test1/120/14’ 2019-12-05T06:10:01.723 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/18’ -> ‘copyofsnap1/test1/120/18’ 2019-12-05T06:10:01.723 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/13’ -> ‘copyofsnap1/test1/120/13’ 2019-12-05T06:10:01.724 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/120/17’ -> ‘copyofsnap1/test1/120/17’ 2019-12-05T06:10:01.724 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110’ -> ‘copyofsnap1/test1/110’ 2019-12-05T06:10:01.724 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/19’ -> ‘copyofsnap1/test1/110/19’ 2019-12-05T06:10:01.724 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/15’ -> ‘copyofsnap1/test1/110/15’ 2019-12-05T06:10:06.418 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/20’ -> ‘copyofsnap1/test1/110/20’ 2019-12-05T06:10:06.418 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/12’ -> ‘copyofsnap1/test1/110/12’ 2019-12-05T06:10:06.418 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/10’ -> ‘copyofsnap1/test1/110/10’ 2019-12-05T06:10:06.418 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/11’ -> ‘copyofsnap1/test1/110/11’ 2019-12-05T06:10:06.419 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/16’ -> ‘copyofsnap1/test1/110/16’ 2019-12-05T06:10:06.419 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/14’ -> ‘copyofsnap1/test1/110/14’ 2019-12-05T06:10:06.419 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/18’ -> ‘copyofsnap1/test1/110/18’ 2019-12-05T06:10:06.419 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/13’ -> ‘copyofsnap1/test1/110/13’ 2019-12-05T06:10:06.419 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/110/17’ -> ‘copyofsnap1/test1/110/17’ 2019-12-05T06:10:06.419 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137’ -> ‘copyofsnap1/test1/137’ 2019-12-05T06:10:06.419 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/19’ -> ‘copyofsnap1/test1/137/19’ 2019-12-05T06:10:06.420 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/15’ -> ‘copyofsnap1/test1/137/15’ 2019-12-05T06:10:06.420 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/20’ -> ‘copyofsnap1/test1/137/20’ 2019-12-05T06:10:06.420 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/12’ -> ‘copyofsnap1/test1/137/12’ 2019-12-05T06:10:06.420 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/10’ -> ‘copyofsnap1/test1/137/10’ 2019-12-05T06:10:06.420 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/11’ -> ‘copyofsnap1/test1/137/11’ 2019-12-05T06:10:06.420 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/16’ -> ‘copyofsnap1/test1/137/16’ 2019-12-05T06:10:06.421 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/14’ -> ‘copyofsnap1/test1/137/14’ 2019-12-05T06:10:06.421 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/18’ -> ‘copyofsnap1/test1/137/18’ 2019-12-05T06:10:06.421 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/13’ -> ‘copyofsnap1/test1/137/13’ 2019-12-05T06:10:06.421 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/137/17’ -> ‘copyofsnap1/test1/137/17’ 2019-12-05T06:10:06.421 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198’ -> ‘copyofsnap1/test1/198’ 2019-12-05T06:10:06.421 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/19’ -> ‘copyofsnap1/test1/198/19’ 2019-12-05T06:10:06.421 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/15’ -> ‘copyofsnap1/test1/198/15’ 2019-12-05T06:10:06.422 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/20’ -> ‘copyofsnap1/test1/198/20’ 2019-12-05T06:10:06.422 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/12’ -> ‘copyofsnap1/test1/198/12’ 2019-12-05T06:10:06.422 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/10’ -> ‘copyofsnap1/test1/198/10’ 2019-12-05T06:10:06.422 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/11’ -> ‘copyofsnap1/test1/198/11’ 2019-12-05T06:10:06.422 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/16’ -> ‘copyofsnap1/test1/198/16’ 2019-12-05T06:10:06.422 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/14’ -> ‘copyofsnap1/test1/198/14’ 2019-12-05T06:10:06.422 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/18’ -> ‘copyofsnap1/test1/198/18’ 2019-12-05T06:10:06.423 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/13’ -> ‘copyofsnap1/test1/198/13’ 2019-12-05T06:10:06.423 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/198/17’ -> ‘copyofsnap1/test1/198/17’ 2019-12-05T06:10:06.423 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136’ -> ‘copyofsnap1/test1/136’ 2019-12-05T06:10:06.423 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/19’ -> ‘copyofsnap1/test1/136/19’ 2019-12-05T06:10:06.423 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/15’ -> ‘copyofsnap1/test1/136/15’ 2019-12-05T06:10:06.423 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/20’ -> ‘copyofsnap1/test1/136/20’ 2019-12-05T06:10:06.423 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/12’ -> ‘copyofsnap1/test1/136/12’ 2019-12-05T06:10:06.424 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/10’ -> ‘copyofsnap1/test1/136/10’ 2019-12-05T06:10:06.424 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/11’ -> ‘copyofsnap1/test1/136/11’ 2019-12-05T06:10:06.424 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/16’ -> ‘copyofsnap1/test1/136/16’ 2019-12-05T06:10:06.424 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/14’ -> ‘copyofsnap1/test1/136/14’ 2019-12-05T06:10:06.424 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/18’ -> ‘copyofsnap1/test1/136/18’ 2019-12-05T06:10:06.424 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/13’ -> ‘copyofsnap1/test1/136/13’ 2019-12-05T06:10:06.425 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/136/17’ -> ‘copyofsnap1/test1/136/17’ 2019-12-05T06:10:06.425 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125’ -> ‘copyofsnap1/test1/125’ 2019-12-05T06:10:06.425 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/19’ -> ‘copyofsnap1/test1/125/19’ 2019-12-05T06:10:06.425 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/15’ -> ‘copyofsnap1/test1/125/15’ 2019-12-05T06:10:06.425 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/20’ -> ‘copyofsnap1/test1/125/20’ 2019-12-05T06:10:06.425 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/12’ -> ‘copyofsnap1/test1/125/12’ 2019-12-05T06:10:06.425 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/10’ -> ‘copyofsnap1/test1/125/10’ 2019-12-05T06:10:06.426 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/11’ -> ‘copyofsnap1/test1/125/11’ 2019-12-05T06:10:06.426 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/16’ -> ‘copyofsnap1/test1/125/16’ 2019-12-05T06:10:06.426 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/14’ -> ‘copyofsnap1/test1/125/14’ 2019-12-05T06:10:06.426 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/18’ -> ‘copyofsnap1/test1/125/18’ 2019-12-05T06:10:06.426 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/13’ -> ‘copyofsnap1/test1/125/13’ 2019-12-05T06:10:06.426 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/125/17’ -> ‘copyofsnap1/test1/125/17’ 2019-12-05T06:10:06.426 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194’ -> ‘copyofsnap1/test1/194’ 2019-12-05T06:10:06.427 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/19’ -> ‘copyofsnap1/test1/194/19’ 2019-12-05T06:10:06.427 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/15’ -> ‘copyofsnap1/test1/194/15’ 2019-12-05T06:10:06.427 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/20’ -> ‘copyofsnap1/test1/194/20’ 2019-12-05T06:10:06.427 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/12’ -> ‘copyofsnap1/test1/194/12’ 2019-12-05T06:10:06.427 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/10’ -> ‘copyofsnap1/test1/194/10’ 2019-12-05T06:10:06.427 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/11’ -> ‘copyofsnap1/test1/194/11’ 2019-12-05T06:10:06.427 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/16’ -> ‘copyofsnap1/test1/194/16’ 2019-12-05T06:10:06.428 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/14’ -> ‘copyofsnap1/test1/194/14’ 2019-12-05T06:10:06.428 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/18’ -> ‘copyofsnap1/test1/194/18’ 2019-12-05T06:10:06.428 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/13’ -> ‘copyofsnap1/test1/194/13’ 2019-12-05T06:10:06.428 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/194/17’ -> ‘copyofsnap1/test1/194/17’ 2019-12-05T06:10:06.429 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119’ -> ‘copyofsnap1/test1/119’ 2019-12-05T06:10:12.299 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/19’ -> ‘copyofsnap1/test1/119/19’ 2019-12-05T06:10:12.299 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/15’ -> ‘copyofsnap1/test1/119/15’ 2019-12-05T06:10:12.299 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/20’ -> ‘copyofsnap1/test1/119/20’ 2019-12-05T06:10:12.299 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/12’ -> ‘copyofsnap1/test1/119/12’ 2019-12-05T06:10:12.299 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/10’ -> ‘copyofsnap1/test1/119/10’ 2019-12-05T06:10:12.300 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/11’ -> ‘copyofsnap1/test1/119/11’ 2019-12-05T06:10:12.300 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/16’ -> ‘copyofsnap1/test1/119/16’ 2019-12-05T06:10:12.300 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/14’ -> ‘copyofsnap1/test1/119/14’ 2019-12-05T06:10:12.300 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/18’ -> ‘copyofsnap1/test1/119/18’ 2019-12-05T06:10:12.300 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/13’ -> ‘copyofsnap1/test1/119/13’ 2019-12-05T06:10:12.300 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/119/17’ -> ‘copyofsnap1/test1/119/17’ 2019-12-05T06:10:12.301 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157’ -> ‘copyofsnap1/test1/157’ 2019-12-05T06:10:12.301 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/19’ -> ‘copyofsnap1/test1/157/19’ 2019-12-05T06:10:12.301 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/15’ -> ‘copyofsnap1/test1/157/15’ 2019-12-05T06:10:12.301 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/20’ -> ‘copyofsnap1/test1/157/20’ 2019-12-05T06:10:12.301 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/12’ -> ‘copyofsnap1/test1/157/12’ 2019-12-05T06:10:12.301 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/10’ -> ‘copyofsnap1/test1/157/10’ 2019-12-05T06:10:12.301 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/11’ -> ‘copyofsnap1/test1/157/11’ 2019-12-05T06:10:12.302 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/16’ -> ‘copyofsnap1/test1/157/16’ 2019-12-05T06:10:12.302 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/14’ -> ‘copyofsnap1/test1/157/14’ 2019-12-05T06:10:12.302 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/18’ -> ‘copyofsnap1/test1/157/18’ 2019-12-05T06:10:12.302 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/13’ -> ‘copyofsnap1/test1/157/13’ 2019-12-05T06:10:12.302 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/157/17’ -> ‘copyofsnap1/test1/157/17’ 2019-12-05T06:10:12.302 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172’ -> ‘copyofsnap1/test1/172’ 2019-12-05T06:10:12.302 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/19’ -> ‘copyofsnap1/test1/172/19’ 2019-12-05T06:10:12.303 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/15’ -> ‘copyofsnap1/test1/172/15’ 2019-12-05T06:10:12.303 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/20’ -> ‘copyofsnap1/test1/172/20’ 2019-12-05T06:10:12.303 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/12’ -> ‘copyofsnap1/test1/172/12’ 2019-12-05T06:10:12.303 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/10’ -> ‘copyofsnap1/test1/172/10’ 2019-12-05T06:10:12.303 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/11’ -> ‘copyofsnap1/test1/172/11’ 2019-12-05T06:10:12.303 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/16’ -> ‘copyofsnap1/test1/172/16’ 2019-12-05T06:10:12.303 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/14’ -> ‘copyofsnap1/test1/172/14’ 2019-12-05T06:10:12.304 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/18’ -> ‘copyofsnap1/test1/172/18’ 2019-12-05T06:10:12.304 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/13’ -> ‘copyofsnap1/test1/172/13’ 2019-12-05T06:10:12.304 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/172/17’ -> ‘copyofsnap1/test1/172/17’ 2019-12-05T06:10:12.304 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186’ -> ‘copyofsnap1/test1/186’ 2019-12-05T06:10:12.304 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/19’ -> ‘copyofsnap1/test1/186/19’ 2019-12-05T06:10:12.304 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/15’ -> ‘copyofsnap1/test1/186/15’ 2019-12-05T06:10:12.305 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/20’ -> ‘copyofsnap1/test1/186/20’ 2019-12-05T06:10:12.305 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/12’ -> ‘copyofsnap1/test1/186/12’ 2019-12-05T06:10:12.305 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/10’ -> ‘copyofsnap1/test1/186/10’ 2019-12-05T06:10:12.305 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/11’ -> ‘copyofsnap1/test1/186/11’ 2019-12-05T06:10:12.305 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/16’ -> ‘copyofsnap1/test1/186/16’ 2019-12-05T06:10:12.305 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/14’ -> ‘copyofsnap1/test1/186/14’ 2019-12-05T06:10:12.305 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/18’ -> ‘copyofsnap1/test1/186/18’ 2019-12-05T06:10:12.306 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/13’ -> ‘copyofsnap1/test1/186/13’ 2019-12-05T06:10:12.306 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/186/17’ -> ‘copyofsnap1/test1/186/17’ 2019-12-05T06:10:12.306 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190’ -> ‘copyofsnap1/test1/190’ 2019-12-05T06:10:12.306 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/19’ -> ‘copyofsnap1/test1/190/19’ 2019-12-05T06:10:12.306 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/15’ -> ‘copyofsnap1/test1/190/15’ 2019-12-05T06:10:12.306 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/20’ -> ‘copyofsnap1/test1/190/20’ 2019-12-05T06:10:12.306 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/12’ -> ‘copyofsnap1/test1/190/12’ 2019-12-05T06:10:12.307 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/10’ -> ‘copyofsnap1/test1/190/10’ 2019-12-05T06:10:12.307 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/11’ -> ‘copyofsnap1/test1/190/11’ 2019-12-05T06:10:12.307 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/16’ -> ‘copyofsnap1/test1/190/16’ 2019-12-05T06:10:12.307 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/14’ -> ‘copyofsnap1/test1/190/14’ 2019-12-05T06:10:12.307 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/18’ -> ‘copyofsnap1/test1/190/18’ 2019-12-05T06:10:12.307 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/13’ -> ‘copyofsnap1/test1/190/13’ 2019-12-05T06:10:12.308 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/190/17’ -> ‘copyofsnap1/test1/190/17’ 2019-12-05T06:10:12.308 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104’ -> ‘copyofsnap1/test1/104’ 2019-12-05T06:10:12.308 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/19’ -> ‘copyofsnap1/test1/104/19’ 2019-12-05T06:10:12.308 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/15’ -> ‘copyofsnap1/test1/104/15’ 2019-12-05T06:10:12.308 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/20’ -> ‘copyofsnap1/test1/104/20’ 2019-12-05T06:10:12.308 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/12’ -> ‘copyofsnap1/test1/104/12’ 2019-12-05T06:10:12.308 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/10’ -> ‘copyofsnap1/test1/104/10’ 2019-12-05T06:10:12.309 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/11’ -> ‘copyofsnap1/test1/104/11’ 2019-12-05T06:10:12.309 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/16’ -> ‘copyofsnap1/test1/104/16’ 2019-12-05T06:10:12.309 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/14’ -> ‘copyofsnap1/test1/104/14’ 2019-12-05T06:10:12.309 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/18’ -> ‘copyofsnap1/test1/104/18’ 2019-12-05T06:10:12.309 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/13’ -> ‘copyofsnap1/test1/104/13’ 2019-12-05T06:10:19.262 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/104/17’ -> ‘copyofsnap1/test1/104/17’ 2019-12-05T06:10:19.262 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164’ -> ‘copyofsnap1/test1/164’ 2019-12-05T06:10:19.262 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/19’ -> ‘copyofsnap1/test1/164/19’ 2019-12-05T06:10:19.263 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/15’ -> ‘copyofsnap1/test1/164/15’ 2019-12-05T06:10:19.263 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/20’ -> ‘copyofsnap1/test1/164/20’ 2019-12-05T06:10:19.263 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/12’ -> ‘copyofsnap1/test1/164/12’ 2019-12-05T06:10:19.263 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/10’ -> ‘copyofsnap1/test1/164/10’ 2019-12-05T06:10:19.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/11’ -> ‘copyofsnap1/test1/164/11’ 2019-12-05T06:10:19.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/16’ -> ‘copyofsnap1/test1/164/16’ 2019-12-05T06:10:19.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/14’ -> ‘copyofsnap1/test1/164/14’ 2019-12-05T06:10:19.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/18’ -> ‘copyofsnap1/test1/164/18’ 2019-12-05T06:10:19.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/13’ -> ‘copyofsnap1/test1/164/13’ 2019-12-05T06:10:19.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/164/17’ -> ‘copyofsnap1/test1/164/17’ 2019-12-05T06:10:19.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112’ -> ‘copyofsnap1/test1/112’ 2019-12-05T06:10:19.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/19’ -> ‘copyofsnap1/test1/112/19’ 2019-12-05T06:10:19.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/15’ -> ‘copyofsnap1/test1/112/15’ 2019-12-05T06:10:19.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/20’ -> ‘copyofsnap1/test1/112/20’ 2019-12-05T06:10:19.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/12’ -> ‘copyofsnap1/test1/112/12’ 2019-12-05T06:10:19.266 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/10’ -> ‘copyofsnap1/test1/112/10’ 2019-12-05T06:10:19.266 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/11’ -> ‘copyofsnap1/test1/112/11’ 2019-12-05T06:10:19.266 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/16’ -> ‘copyofsnap1/test1/112/16’ 2019-12-05T06:10:19.266 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/14’ -> ‘copyofsnap1/test1/112/14’ 2019-12-05T06:10:19.266 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/18’ -> ‘copyofsnap1/test1/112/18’ 2019-12-05T06:10:19.267 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/13’ -> ‘copyofsnap1/test1/112/13’ 2019-12-05T06:10:19.267 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/112/17’ -> ‘copyofsnap1/test1/112/17’ 2019-12-05T06:10:19.267 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100’ -> ‘copyofsnap1/test1/100’ 2019-12-05T06:10:19.268 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/19’ -> ‘copyofsnap1/test1/100/19’ 2019-12-05T06:10:19.268 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/15’ -> ‘copyofsnap1/test1/100/15’ 2019-12-05T06:10:19.269 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/20’ -> ‘copyofsnap1/test1/100/20’ 2019-12-05T06:10:19.269 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/12’ -> ‘copyofsnap1/test1/100/12’ 2019-12-05T06:10:19.269 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/10’ -> ‘copyofsnap1/test1/100/10’ 2019-12-05T06:10:19.269 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/11’ -> ‘copyofsnap1/test1/100/11’ 2019-12-05T06:10:19.269 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/16’ -> ‘copyofsnap1/test1/100/16’ 2019-12-05T06:10:19.269 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/14’ -> ‘copyofsnap1/test1/100/14’ 2019-12-05T06:10:19.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/18’ -> ‘copyofsnap1/test1/100/18’ 2019-12-05T06:10:19.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/13’ -> ‘copyofsnap1/test1/100/13’ 2019-12-05T06:10:19.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/100/17’ -> ‘copyofsnap1/test1/100/17’ 2019-12-05T06:10:19.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134’ -> ‘copyofsnap1/test1/134’ 2019-12-05T06:10:19.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/19’ -> ‘copyofsnap1/test1/134/19’ 2019-12-05T06:10:19.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/15’ -> ‘copyofsnap1/test1/134/15’ 2019-12-05T06:10:19.271 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/20’ -> ‘copyofsnap1/test1/134/20’ 2019-12-05T06:10:19.273 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/12’ -> ‘copyofsnap1/test1/134/12’ 2019-12-05T06:10:19.273 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/10’ -> ‘copyofsnap1/test1/134/10’ 2019-12-05T06:10:19.273 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/11’ -> ‘copyofsnap1/test1/134/11’ 2019-12-05T06:10:19.274 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/16’ -> ‘copyofsnap1/test1/134/16’ 2019-12-05T06:10:19.274 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/14’ -> ‘copyofsnap1/test1/134/14’ 2019-12-05T06:10:19.274 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/18’ -> ‘copyofsnap1/test1/134/18’ 2019-12-05T06:10:19.275 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/13’ -> ‘copyofsnap1/test1/134/13’ 2019-12-05T06:10:19.275 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/134/17’ -> ‘copyofsnap1/test1/134/17’ 2019-12-05T06:10:19.275 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115’ -> ‘copyofsnap1/test1/115’ 2019-12-05T06:10:19.276 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/19’ -> ‘copyofsnap1/test1/115/19’ 2019-12-05T06:10:19.276 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/15’ -> ‘copyofsnap1/test1/115/15’ 2019-12-05T06:10:19.276 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/20’ -> ‘copyofsnap1/test1/115/20’ 2019-12-05T06:10:19.276 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/12’ -> ‘copyofsnap1/test1/115/12’ 2019-12-05T06:10:19.277 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/10’ -> ‘copyofsnap1/test1/115/10’ 2019-12-05T06:10:19.277 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/11’ -> ‘copyofsnap1/test1/115/11’ 2019-12-05T06:10:19.277 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/16’ -> ‘copyofsnap1/test1/115/16’ 2019-12-05T06:10:19.277 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/14’ -> ‘copyofsnap1/test1/115/14’ 2019-12-05T06:10:19.278 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/18’ -> ‘copyofsnap1/test1/115/18’ 2019-12-05T06:10:19.278 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/13’ -> ‘copyofsnap1/test1/115/13’ 2019-12-05T06:10:19.278 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/115/17’ -> ‘copyofsnap1/test1/115/17’ 2019-12-05T06:10:19.279 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116’ -> ‘copyofsnap1/test1/116’ 2019-12-05T06:10:19.279 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/19’ -> ‘copyofsnap1/test1/116/19’ 2019-12-05T06:10:19.279 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/15’ -> ‘copyofsnap1/test1/116/15’ 2019-12-05T06:10:19.279 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/20’ -> ‘copyofsnap1/test1/116/20’ 2019-12-05T06:10:19.280 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/12’ -> ‘copyofsnap1/test1/116/12’ 2019-12-05T06:10:19.280 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/10’ -> ‘copyofsnap1/test1/116/10’ 2019-12-05T06:10:19.280 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/11’ -> ‘copyofsnap1/test1/116/11’ 2019-12-05T06:10:19.281 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/16’ -> ‘copyofsnap1/test1/116/16’ 2019-12-05T06:10:19.281 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/14’ -> ‘copyofsnap1/test1/116/14’ 2019-12-05T06:10:25.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/18’ -> ‘copyofsnap1/test1/116/18’ 2019-12-05T06:10:25.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/13’ -> ‘copyofsnap1/test1/116/13’ 2019-12-05T06:10:25.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/116/17’ -> ‘copyofsnap1/test1/116/17’ 2019-12-05T06:10:25.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195’ -> ‘copyofsnap1/test1/195’ 2019-12-05T06:10:25.264 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/19’ -> ‘copyofsnap1/test1/195/19’ 2019-12-05T06:10:25.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/15’ -> ‘copyofsnap1/test1/195/15’ 2019-12-05T06:10:25.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/20’ -> ‘copyofsnap1/test1/195/20’ 2019-12-05T06:10:25.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/12’ -> ‘copyofsnap1/test1/195/12’ 2019-12-05T06:10:25.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/10’ -> ‘copyofsnap1/test1/195/10’ 2019-12-05T06:10:25.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/11’ -> ‘copyofsnap1/test1/195/11’ 2019-12-05T06:10:25.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/16’ -> ‘copyofsnap1/test1/195/16’ 2019-12-05T06:10:25.265 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/14’ -> ‘copyofsnap1/test1/195/14’ 2019-12-05T06:10:25.266 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/18’ -> ‘copyofsnap1/test1/195/18’ 2019-12-05T06:10:25.266 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/13’ -> ‘copyofsnap1/test1/195/13’ 2019-12-05T06:10:25.267 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/195/17’ -> ‘copyofsnap1/test1/195/17’ 2019-12-05T06:10:25.267 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138’ -> ‘copyofsnap1/test1/138’ 2019-12-05T06:10:25.267 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/19’ -> ‘copyofsnap1/test1/138/19’ 2019-12-05T06:10:25.267 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/15’ -> ‘copyofsnap1/test1/138/15’ 2019-12-05T06:10:25.267 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/20’ -> ‘copyofsnap1/test1/138/20’ 2019-12-05T06:10:25.267 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/12’ -> ‘copyofsnap1/test1/138/12’ 2019-12-05T06:10:25.267 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/10’ -> ‘copyofsnap1/test1/138/10’ 2019-12-05T06:10:25.268 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/11’ -> ‘copyofsnap1/test1/138/11’ 2019-12-05T06:10:25.268 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/16’ -> ‘copyofsnap1/test1/138/16’ 2019-12-05T06:10:25.268 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/14’ -> ‘copyofsnap1/test1/138/14’ 2019-12-05T06:10:25.268 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/18’ -> ‘copyofsnap1/test1/138/18’ 2019-12-05T06:10:25.268 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/13’ -> ‘copyofsnap1/test1/138/13’ 2019-12-05T06:10:25.268 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/138/17’ -> ‘copyofsnap1/test1/138/17’ 2019-12-05T06:10:25.269 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135’ -> ‘copyofsnap1/test1/135’ 2019-12-05T06:10:25.269 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/19’ -> ‘copyofsnap1/test1/135/19’ 2019-12-05T06:10:25.269 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/15’ -> ‘copyofsnap1/test1/135/15’ 2019-12-05T06:10:25.269 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/20’ -> ‘copyofsnap1/test1/135/20’ 2019-12-05T06:10:25.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/12’ -> ‘copyofsnap1/test1/135/12’ 2019-12-05T06:10:25.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/10’ -> ‘copyofsnap1/test1/135/10’ 2019-12-05T06:10:25.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/11’ -> ‘copyofsnap1/test1/135/11’ 2019-12-05T06:10:25.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/16’ -> ‘copyofsnap1/test1/135/16’ 2019-12-05T06:10:25.270 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/14’ -> ‘copyofsnap1/test1/135/14’ 2019-12-05T06:10:25.271 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/18’ -> ‘copyofsnap1/test1/135/18’ 2019-12-05T06:10:25.271 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/13’ -> ‘copyofsnap1/test1/135/13’ 2019-12-05T06:10:25.271 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/135/17’ -> ‘copyofsnap1/test1/135/17’ 2019-12-05T06:10:25.271 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117’ -> ‘copyofsnap1/test1/117’ 2019-12-05T06:10:25.271 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/19’ -> ‘copyofsnap1/test1/117/19’ 2019-12-05T06:10:25.271 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/15’ -> ‘copyofsnap1/test1/117/15’ 2019-12-05T06:10:25.272 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/20’ -> ‘copyofsnap1/test1/117/20’ 2019-12-05T06:10:25.272 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/12’ -> ‘copyofsnap1/test1/117/12’ 2019-12-05T06:10:25.272 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/10’ -> ‘copyofsnap1/test1/117/10’ 2019-12-05T06:10:25.272 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/11’ -> ‘copyofsnap1/test1/117/11’ 2019-12-05T06:10:25.272 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/16’ -> ‘copyofsnap1/test1/117/16’ 2019-12-05T06:10:25.272 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/14’ -> ‘copyofsnap1/test1/117/14’ 2019-12-05T06:10:25.272 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/18’ -> ‘copyofsnap1/test1/117/18’ 2019-12-05T06:10:25.273 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/13’ -> ‘copyofsnap1/test1/117/13’ 2019-12-05T06:10:25.273 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/117/17’ -> ‘copyofsnap1/test1/117/17’ 2019-12-05T06:10:25.273 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143’ -> ‘copyofsnap1/test1/143’ 2019-12-05T06:10:25.273 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/19’ -> ‘copyofsnap1/test1/143/19’ 2019-12-05T06:10:25.273 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/15’ -> ‘copyofsnap1/test1/143/15’ 2019-12-05T06:10:25.273 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/20’ -> ‘copyofsnap1/test1/143/20’ 2019-12-05T06:10:25.273 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/12’ -> ‘copyofsnap1/test1/143/12’ 2019-12-05T06:10:25.274 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/10’ -> ‘copyofsnap1/test1/143/10’ 2019-12-05T06:10:25.274 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/11’ -> ‘copyofsnap1/test1/143/11’ 2019-12-05T06:10:25.274 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/16’ -> ‘copyofsnap1/test1/143/16’ 2019-12-05T06:10:25.274 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/14’ -> ‘copyofsnap1/test1/143/14’ 2019-12-05T06:10:25.274 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/18’ -> ‘copyofsnap1/test1/143/18’ 2019-12-05T06:10:25.274 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/13’ -> ‘copyofsnap1/test1/143/13’ 2019-12-05T06:10:25.275 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/143/17’ -> ‘copyofsnap1/test1/143/17’ 2019-12-05T06:10:25.275 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180’ -> ‘copyofsnap1/test1/180’ 2019-12-05T06:10:25.275 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/19’ -> ‘copyofsnap1/test1/180/19’ 2019-12-05T06:10:25.275 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/15’ -> ‘copyofsnap1/test1/180/15’ 2019-12-05T06:10:25.275 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/20’ -> ‘copyofsnap1/test1/180/20’ 2019-12-05T06:10:25.275 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/12’ -> ‘copyofsnap1/test1/180/12’ 2019-12-05T06:10:25.275 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/10’ -> ‘copyofsnap1/test1/180/10’ 2019-12-05T06:10:25.276 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/11’ -> ‘copyofsnap1/test1/180/11’ 2019-12-05T06:10:28.561 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/16’ -> ‘copyofsnap1/test1/180/16’ 2019-12-05T06:10:28.561 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/14’ -> ‘copyofsnap1/test1/180/14’ 2019-12-05T06:10:28.562 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/18’ -> ‘copyofsnap1/test1/180/18’ 2019-12-05T06:10:28.562 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/13’ -> ‘copyofsnap1/test1/180/13’ 2019-12-05T06:10:28.562 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/180/17’ -> ‘copyofsnap1/test1/180/17’ 2019-12-05T06:10:28.562 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126’ -> ‘copyofsnap1/test1/126’ 2019-12-05T06:10:28.563 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/19’ -> ‘copyofsnap1/test1/126/19’ 2019-12-05T06:10:28.563 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/15’ -> ‘copyofsnap1/test1/126/15’ 2019-12-05T06:10:28.563 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/20’ -> ‘copyofsnap1/test1/126/20’ 2019-12-05T06:10:28.563 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/12’ -> ‘copyofsnap1/test1/126/12’ 2019-12-05T06:10:28.564 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/10’ -> ‘copyofsnap1/test1/126/10’ 2019-12-05T06:10:28.564 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/11’ -> ‘copyofsnap1/test1/126/11’ 2019-12-05T06:10:28.564 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/16’ -> ‘copyofsnap1/test1/126/16’ 2019-12-05T06:10:28.573 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/14’ -> ‘copyofsnap1/test1/126/14’ 2019-12-05T06:10:28.573 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/18’ -> ‘copyofsnap1/test1/126/18’ 2019-12-05T06:10:28.573 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/13’ -> ‘copyofsnap1/test1/126/13’ 2019-12-05T06:10:28.574 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/126/17’ -> ‘copyofsnap1/test1/126/17’ 2019-12-05T06:10:28.574 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161’ -> ‘copyofsnap1/test1/161’ 2019-12-05T06:10:28.574 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/19’ -> ‘copyofsnap1/test1/161/19’ 2019-12-05T06:10:28.574 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/15’ -> ‘copyofsnap1/test1/161/15’ 2019-12-05T06:10:28.574 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/20’ -> ‘copyofsnap1/test1/161/20’ 2019-12-05T06:10:28.575 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/12’ -> ‘copyofsnap1/test1/161/12’ 2019-12-05T06:10:28.577 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/10’ -> ‘copyofsnap1/test1/161/10’ 2019-12-05T06:10:28.577 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/11’ -> ‘copyofsnap1/test1/161/11’ 2019-12-05T06:10:28.577 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/16’ -> ‘copyofsnap1/test1/161/16’ 2019-12-05T06:10:28.577 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/14’ -> ‘copyofsnap1/test1/161/14’ 2019-12-05T06:10:28.578 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/18’ -> ‘copyofsnap1/test1/161/18’ 2019-12-05T06:10:28.578 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/13’ -> ‘copyofsnap1/test1/161/13’ 2019-12-05T06:10:28.578 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/161/17’ -> ‘copyofsnap1/test1/161/17’ 2019-12-05T06:10:28.579 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176’ -> ‘copyofsnap1/test1/176’ 2019-12-05T06:10:28.579 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/19’ -> ‘copyofsnap1/test1/176/19’ 2019-12-05T06:10:28.579 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/15’ -> ‘copyofsnap1/test1/176/15’ 2019-12-05T06:10:28.579 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/20’ -> ‘copyofsnap1/test1/176/20’ 2019-12-05T06:10:28.580 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/12’ -> ‘copyofsnap1/test1/176/12’ 2019-12-05T06:10:28.580 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/10’ -> ‘copyofsnap1/test1/176/10’ 2019-12-05T06:10:28.580 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/11’ -> ‘copyofsnap1/test1/176/11’ 2019-12-05T06:10:28.580 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/16’ -> ‘copyofsnap1/test1/176/16’ 2019-12-05T06:10:28.581 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/14’ -> ‘copyofsnap1/test1/176/14’ 2019-12-05T06:10:28.581 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/18’ -> ‘copyofsnap1/test1/176/18’ 2019-12-05T06:10:28.581 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/13’ -> ‘copyofsnap1/test1/176/13’ 2019-12-05T06:10:28.581 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/176/17’ -> ‘copyofsnap1/test1/176/17’ 2019-12-05T06:10:28.582 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111’ -> ‘copyofsnap1/test1/111’ 2019-12-05T06:10:28.582 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/19’ -> ‘copyofsnap1/test1/111/19’ 2019-12-05T06:10:28.582 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/15’ -> ‘copyofsnap1/test1/111/15’ 2019-12-05T06:10:28.582 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/20’ -> ‘copyofsnap1/test1/111/20’ 2019-12-05T06:10:28.582 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/12’ -> ‘copyofsnap1/test1/111/12’ 2019-12-05T06:10:28.582 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/10’ -> ‘copyofsnap1/test1/111/10’ 2019-12-05T06:10:28.583 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/11’ -> ‘copyofsnap1/test1/111/11’ 2019-12-05T06:10:28.583 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/16’ -> ‘copyofsnap1/test1/111/16’ 2019-12-05T06:10:28.583 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/14’ -> ‘copyofsnap1/test1/111/14’ 2019-12-05T06:10:28.583 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/18’ -> ‘copyofsnap1/test1/111/18’ 2019-12-05T06:10:28.583 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/13’ -> ‘copyofsnap1/test1/111/13’ 2019-12-05T06:10:28.583 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/111/17’ -> ‘copyofsnap1/test1/111/17’ 2019-12-05T06:10:28.583 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191’ -> ‘copyofsnap1/test1/191’ 2019-12-05T06:10:28.585 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/19’ -> ‘copyofsnap1/test1/191/19’ 2019-12-05T06:10:28.585 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/15’ -> ‘copyofsnap1/test1/191/15’ 2019-12-05T06:10:28.585 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/20’ -> ‘copyofsnap1/test1/191/20’ 2019-12-05T06:10:28.585 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/12’ -> ‘copyofsnap1/test1/191/12’ 2019-12-05T06:10:28.585 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/10’ -> ‘copyofsnap1/test1/191/10’ 2019-12-05T06:10:28.586 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/11’ -> ‘copyofsnap1/test1/191/11’ 2019-12-05T06:10:28.586 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/16’ -> ‘copyofsnap1/test1/191/16’ 2019-12-05T06:10:28.586 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/14’ -> ‘copyofsnap1/test1/191/14’ 2019-12-05T06:10:28.587 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/18’ -> ‘copyofsnap1/test1/191/18’ 2019-12-05T06:10:28.588 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/13’ -> ‘copyofsnap1/test1/191/13’ 2019-12-05T06:10:28.588 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/191/17’ -> ‘copyofsnap1/test1/191/17’ 2019-12-05T06:10:28.588 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184’ -> ‘copyofsnap1/test1/184’ 2019-12-05T06:10:28.588 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/19’ -> ‘copyofsnap1/test1/184/19’ 2019-12-05T06:10:28.589 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/15’ -> ‘copyofsnap1/test1/184/15’ 2019-12-05T06:10:28.589 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/20’ -> ‘copyofsnap1/test1/184/20’ 2019-12-05T06:10:28.589 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/12’ -> ‘copyofsnap1/test1/184/12’ 2019-12-05T06:10:28.615 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:10:28.622 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:10:29.714 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/10’ -> ‘copyofsnap1/test1/184/10’ 2019-12-05T06:10:29.714 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/11’ -> ‘copyofsnap1/test1/184/11’ 2019-12-05T06:10:29.715 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/16’ -> ‘copyofsnap1/test1/184/16’ 2019-12-05T06:10:29.715 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/14’ -> ‘copyofsnap1/test1/184/14’ 2019-12-05T06:10:29.716 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/18’ -> ‘copyofsnap1/test1/184/18’ 2019-12-05T06:10:29.716 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/13’ -> ‘copyofsnap1/test1/184/13’ 2019-12-05T06:10:29.716 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/184/17’ -> ‘copyofsnap1/test1/184/17’ 2019-12-05T06:10:29.717 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192’ -> ‘copyofsnap1/test1/192’ 2019-12-05T06:10:29.717 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/19’ -> ‘copyofsnap1/test1/192/19’ 2019-12-05T06:10:29.717 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/15’ -> ‘copyofsnap1/test1/192/15’ 2019-12-05T06:10:29.717 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/20’ -> ‘copyofsnap1/test1/192/20’ 2019-12-05T06:10:29.718 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/12’ -> ‘copyofsnap1/test1/192/12’ 2019-12-05T06:10:29.718 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/10’ -> ‘copyofsnap1/test1/192/10’ 2019-12-05T06:10:29.718 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/11’ -> ‘copyofsnap1/test1/192/11’ 2019-12-05T06:10:29.718 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/16’ -> ‘copyofsnap1/test1/192/16’ 2019-12-05T06:10:29.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/14’ -> ‘copyofsnap1/test1/192/14’ 2019-12-05T06:10:29.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/18’ -> ‘copyofsnap1/test1/192/18’ 2019-12-05T06:10:29.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/13’ -> ‘copyofsnap1/test1/192/13’ 2019-12-05T06:10:29.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/192/17’ -> ‘copyofsnap1/test1/192/17’ 2019-12-05T06:10:29.719 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103’ -> ‘copyofsnap1/test1/103’ 2019-12-05T06:10:29.720 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/19’ -> ‘copyofsnap1/test1/103/19’ 2019-12-05T06:10:29.720 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/15’ -> ‘copyofsnap1/test1/103/15’ 2019-12-05T06:10:29.720 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/20’ -> ‘copyofsnap1/test1/103/20’ 2019-12-05T06:10:29.721 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/12’ -> ‘copyofsnap1/test1/103/12’ 2019-12-05T06:10:29.721 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/10’ -> ‘copyofsnap1/test1/103/10’ 2019-12-05T06:10:29.722 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/11’ -> ‘copyofsnap1/test1/103/11’ 2019-12-05T06:10:29.731 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/16’ -> ‘copyofsnap1/test1/103/16’ 2019-12-05T06:10:29.731 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/14’ -> ‘copyofsnap1/test1/103/14’ 2019-12-05T06:10:29.731 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/18’ -> ‘copyofsnap1/test1/103/18’ 2019-12-05T06:10:29.731 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/13’ -> ‘copyofsnap1/test1/103/13’ 2019-12-05T06:10:29.732 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/103/17’ -> ‘copyofsnap1/test1/103/17’ 2019-12-05T06:10:29.732 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128’ -> ‘copyofsnap1/test1/128’ 2019-12-05T06:10:29.732 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/19’ -> ‘copyofsnap1/test1/128/19’ 2019-12-05T06:10:29.732 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/15’ -> ‘copyofsnap1/test1/128/15’ 2019-12-05T06:10:29.733 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/20’ -> ‘copyofsnap1/test1/128/20’ 2019-12-05T06:10:29.733 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/12’ -> ‘copyofsnap1/test1/128/12’ 2019-12-05T06:10:29.733 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/10’ -> ‘copyofsnap1/test1/128/10’ 2019-12-05T06:10:29.734 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/11’ -> ‘copyofsnap1/test1/128/11’ 2019-12-05T06:10:29.734 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/16’ -> ‘copyofsnap1/test1/128/16’ 2019-12-05T06:10:29.734 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/14’ -> ‘copyofsnap1/test1/128/14’ 2019-12-05T06:10:29.735 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/18’ -> ‘copyofsnap1/test1/128/18’ 2019-12-05T06:10:29.735 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/13’ -> ‘copyofsnap1/test1/128/13’ 2019-12-05T06:10:29.735 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/128/17’ -> ‘copyofsnap1/test1/128/17’ 2019-12-05T06:10:29.735 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144’ -> ‘copyofsnap1/test1/144’ 2019-12-05T06:10:29.736 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/19’ -> ‘copyofsnap1/test1/144/19’ 2019-12-05T06:10:29.736 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/15’ -> ‘copyofsnap1/test1/144/15’ 2019-12-05T06:10:29.736 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/20’ -> ‘copyofsnap1/test1/144/20’ 2019-12-05T06:10:29.736 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/12’ -> ‘copyofsnap1/test1/144/12’ 2019-12-05T06:10:29.737 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/10’ -> ‘copyofsnap1/test1/144/10’ 2019-12-05T06:10:29.740 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/11’ -> ‘copyofsnap1/test1/144/11’ 2019-12-05T06:10:29.741 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/16’ -> ‘copyofsnap1/test1/144/16’ 2019-12-05T06:10:29.741 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/14’ -> ‘copyofsnap1/test1/144/14’ 2019-12-05T06:10:29.741 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/18’ -> ‘copyofsnap1/test1/144/18’ 2019-12-05T06:10:29.741 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/13’ -> ‘copyofsnap1/test1/144/13’ 2019-12-05T06:10:29.742 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/144/17’ -> ‘copyofsnap1/test1/144/17’ 2019-12-05T06:10:29.742 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146’ -> ‘copyofsnap1/test1/146’ 2019-12-05T06:10:29.742 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/19’ -> ‘copyofsnap1/test1/146/19’ 2019-12-05T06:10:29.742 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/15’ -> ‘copyofsnap1/test1/146/15’ 2019-12-05T06:10:29.744 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/20’ -> ‘copyofsnap1/test1/146/20’ 2019-12-05T06:10:29.744 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/12’ -> ‘copyofsnap1/test1/146/12’ 2019-12-05T06:10:29.744 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/10’ -> ‘copyofsnap1/test1/146/10’ 2019-12-05T06:10:29.745 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/11’ -> ‘copyofsnap1/test1/146/11’ 2019-12-05T06:10:29.745 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/16’ -> ‘copyofsnap1/test1/146/16’ 2019-12-05T06:10:29.745 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/14’ -> ‘copyofsnap1/test1/146/14’ 2019-12-05T06:10:29.746 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/18’ -> ‘copyofsnap1/test1/146/18’ 2019-12-05T06:10:29.746 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/13’ -> ‘copyofsnap1/test1/146/13’ 2019-12-05T06:10:29.746 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/146/17’ -> ‘copyofsnap1/test1/146/17’ 2019-12-05T06:10:29.746 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141’ -> ‘copyofsnap1/test1/141’ 2019-12-05T06:10:29.747 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/19’ -> ‘copyofsnap1/test1/141/19’ 2019-12-05T06:10:29.747 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/15’ -> ‘copyofsnap1/test1/141/15’ 2019-12-05T06:10:33.316 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/20’ -> ‘copyofsnap1/test1/141/20’ 2019-12-05T06:10:33.317 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/12’ -> ‘copyofsnap1/test1/141/12’ 2019-12-05T06:10:33.317 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/10’ -> ‘copyofsnap1/test1/141/10’ 2019-12-05T06:10:33.317 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/11’ -> ‘copyofsnap1/test1/141/11’ 2019-12-05T06:10:33.317 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/16’ -> ‘copyofsnap1/test1/141/16’ 2019-12-05T06:10:33.317 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/14’ -> ‘copyofsnap1/test1/141/14’ 2019-12-05T06:10:33.318 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/18’ -> ‘copyofsnap1/test1/141/18’ 2019-12-05T06:10:33.318 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/13’ -> ‘copyofsnap1/test1/141/13’ 2019-12-05T06:10:33.318 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/141/17’ -> ‘copyofsnap1/test1/141/17’ 2019-12-05T06:10:33.318 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182’ -> ‘copyofsnap1/test1/182’ 2019-12-05T06:10:33.318 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/19’ -> ‘copyofsnap1/test1/182/19’ 2019-12-05T06:10:33.319 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/15’ -> ‘copyofsnap1/test1/182/15’ 2019-12-05T06:10:33.319 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/20’ -> ‘copyofsnap1/test1/182/20’ 2019-12-05T06:10:33.319 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/12’ -> ‘copyofsnap1/test1/182/12’ 2019-12-05T06:10:33.320 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/10’ -> ‘copyofsnap1/test1/182/10’ 2019-12-05T06:10:33.320 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/11’ -> ‘copyofsnap1/test1/182/11’ 2019-12-05T06:10:33.320 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/16’ -> ‘copyofsnap1/test1/182/16’ 2019-12-05T06:10:33.320 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/14’ -> ‘copyofsnap1/test1/182/14’ 2019-12-05T06:10:33.320 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/18’ -> ‘copyofsnap1/test1/182/18’ 2019-12-05T06:10:33.320 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/13’ -> ‘copyofsnap1/test1/182/13’ 2019-12-05T06:10:33.321 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/182/17’ -> ‘copyofsnap1/test1/182/17’ 2019-12-05T06:10:33.321 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107’ -> ‘copyofsnap1/test1/107’ 2019-12-05T06:10:33.321 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/19’ -> ‘copyofsnap1/test1/107/19’ 2019-12-05T06:10:33.323 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/15’ -> ‘copyofsnap1/test1/107/15’ 2019-12-05T06:10:33.323 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/20’ -> ‘copyofsnap1/test1/107/20’ 2019-12-05T06:10:33.323 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/12’ -> ‘copyofsnap1/test1/107/12’ 2019-12-05T06:10:33.324 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/10’ -> ‘copyofsnap1/test1/107/10’ 2019-12-05T06:10:33.324 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/11’ -> ‘copyofsnap1/test1/107/11’ 2019-12-05T06:10:33.324 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/16’ -> ‘copyofsnap1/test1/107/16’ 2019-12-05T06:10:33.324 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/14’ -> ‘copyofsnap1/test1/107/14’ 2019-12-05T06:10:33.324 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/18’ -> ‘copyofsnap1/test1/107/18’ 2019-12-05T06:10:33.324 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/13’ -> ‘copyofsnap1/test1/107/13’ 2019-12-05T06:10:33.325 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/107/17’ -> ‘copyofsnap1/test1/107/17’ 2019-12-05T06:10:33.325 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178’ -> ‘copyofsnap1/test1/178’ 2019-12-05T06:10:33.325 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/19’ -> ‘copyofsnap1/test1/178/19’ 2019-12-05T06:10:33.325 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/15’ -> ‘copyofsnap1/test1/178/15’ 2019-12-05T06:10:33.325 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/20’ -> ‘copyofsnap1/test1/178/20’ 2019-12-05T06:10:33.325 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/12’ -> ‘copyofsnap1/test1/178/12’ 2019-12-05T06:10:33.325 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/10’ -> ‘copyofsnap1/test1/178/10’ 2019-12-05T06:10:33.326 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/11’ -> ‘copyofsnap1/test1/178/11’ 2019-12-05T06:10:33.326 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/16’ -> ‘copyofsnap1/test1/178/16’ 2019-12-05T06:10:33.326 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/14’ -> ‘copyofsnap1/test1/178/14’ 2019-12-05T06:10:33.326 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/18’ -> ‘copyofsnap1/test1/178/18’ 2019-12-05T06:10:33.326 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/13’ -> ‘copyofsnap1/test1/178/13’ 2019-12-05T06:10:33.326 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/178/17’ -> ‘copyofsnap1/test1/178/17’ 2019-12-05T06:10:33.326 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156’ -> ‘copyofsnap1/test1/156’ 2019-12-05T06:10:33.327 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/19’ -> ‘copyofsnap1/test1/156/19’ 2019-12-05T06:10:33.327 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/15’ -> ‘copyofsnap1/test1/156/15’ 2019-12-05T06:10:33.327 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/20’ -> ‘copyofsnap1/test1/156/20’ 2019-12-05T06:10:33.327 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/12’ -> ‘copyofsnap1/test1/156/12’ 2019-12-05T06:10:33.327 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/10’ -> ‘copyofsnap1/test1/156/10’ 2019-12-05T06:10:33.327 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/11’ -> ‘copyofsnap1/test1/156/11’ 2019-12-05T06:10:33.328 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/16’ -> ‘copyofsnap1/test1/156/16’ 2019-12-05T06:10:33.328 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/14’ -> ‘copyofsnap1/test1/156/14’ 2019-12-05T06:10:33.328 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/18’ -> ‘copyofsnap1/test1/156/18’ 2019-12-05T06:10:33.328 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/13’ -> ‘copyofsnap1/test1/156/13’ 2019-12-05T06:10:33.328 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/156/17’ -> ‘copyofsnap1/test1/156/17’ 2019-12-05T06:10:33.328 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131’ -> ‘copyofsnap1/test1/131’ 2019-12-05T06:10:33.329 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/19’ -> ‘copyofsnap1/test1/131/19’ 2019-12-05T06:10:33.329 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/15’ -> ‘copyofsnap1/test1/131/15’ 2019-12-05T06:10:33.330 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/20’ -> ‘copyofsnap1/test1/131/20’ 2019-12-05T06:10:33.330 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/12’ -> ‘copyofsnap1/test1/131/12’ 2019-12-05T06:10:33.330 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/10’ -> ‘copyofsnap1/test1/131/10’ 2019-12-05T06:10:33.330 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/11’ -> ‘copyofsnap1/test1/131/11’ 2019-12-05T06:10:33.331 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/16’ -> ‘copyofsnap1/test1/131/16’ 2019-12-05T06:10:33.331 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/14’ -> ‘copyofsnap1/test1/131/14’ 2019-12-05T06:10:33.331 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/18’ -> ‘copyofsnap1/test1/131/18’ 2019-12-05T06:10:33.331 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/13’ -> ‘copyofsnap1/test1/131/13’ 2019-12-05T06:10:33.331 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/131/17’ -> ‘copyofsnap1/test1/131/17’ 2019-12-05T06:10:33.331 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123’ -> ‘copyofsnap1/test1/123’ 2019-12-05T06:10:41.683 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/19’ -> ‘copyofsnap1/test1/123/19’ 2019-12-05T06:10:41.683 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/15’ -> ‘copyofsnap1/test1/123/15’ 2019-12-05T06:10:41.684 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/20’ -> ‘copyofsnap1/test1/123/20’ 2019-12-05T06:10:41.684 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/12’ -> ‘copyofsnap1/test1/123/12’ 2019-12-05T06:10:41.684 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/10’ -> ‘copyofsnap1/test1/123/10’ 2019-12-05T06:10:41.684 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/11’ -> ‘copyofsnap1/test1/123/11’ 2019-12-05T06:10:41.684 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/16’ -> ‘copyofsnap1/test1/123/16’ 2019-12-05T06:10:41.685 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/14’ -> ‘copyofsnap1/test1/123/14’ 2019-12-05T06:10:41.685 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/18’ -> ‘copyofsnap1/test1/123/18’ 2019-12-05T06:10:41.685 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/13’ -> ‘copyofsnap1/test1/123/13’ 2019-12-05T06:10:41.685 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/123/17’ -> ‘copyofsnap1/test1/123/17’ 2019-12-05T06:10:41.685 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160’ -> ‘copyofsnap1/test1/160’ 2019-12-05T06:10:41.685 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/19’ -> ‘copyofsnap1/test1/160/19’ 2019-12-05T06:10:41.685 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/15’ -> ‘copyofsnap1/test1/160/15’ 2019-12-05T06:10:41.686 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/20’ -> ‘copyofsnap1/test1/160/20’ 2019-12-05T06:10:41.686 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/12’ -> ‘copyofsnap1/test1/160/12’ 2019-12-05T06:10:41.686 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/10’ -> ‘copyofsnap1/test1/160/10’ 2019-12-05T06:10:41.686 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/11’ -> ‘copyofsnap1/test1/160/11’ 2019-12-05T06:10:41.686 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/16’ -> ‘copyofsnap1/test1/160/16’ 2019-12-05T06:10:41.686 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/14’ -> ‘copyofsnap1/test1/160/14’ 2019-12-05T06:10:41.686 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/18’ -> ‘copyofsnap1/test1/160/18’ 2019-12-05T06:10:41.687 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/13’ -> ‘copyofsnap1/test1/160/13’ 2019-12-05T06:10:41.687 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/160/17’ -> ‘copyofsnap1/test1/160/17’ 2019-12-05T06:10:41.687 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165’ -> ‘copyofsnap1/test1/165’ 2019-12-05T06:10:41.687 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/19’ -> ‘copyofsnap1/test1/165/19’ 2019-12-05T06:10:41.687 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/15’ -> ‘copyofsnap1/test1/165/15’ 2019-12-05T06:10:41.687 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/20’ -> ‘copyofsnap1/test1/165/20’ 2019-12-05T06:10:41.688 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/12’ -> ‘copyofsnap1/test1/165/12’ 2019-12-05T06:10:41.688 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/10’ -> ‘copyofsnap1/test1/165/10’ 2019-12-05T06:10:41.688 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/11’ -> ‘copyofsnap1/test1/165/11’ 2019-12-05T06:10:41.688 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/16’ -> ‘copyofsnap1/test1/165/16’ 2019-12-05T06:10:41.688 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/14’ -> ‘copyofsnap1/test1/165/14’ 2019-12-05T06:10:41.688 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/18’ -> ‘copyofsnap1/test1/165/18’ 2019-12-05T06:10:41.688 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/13’ -> ‘copyofsnap1/test1/165/13’ 2019-12-05T06:10:41.689 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/165/17’ -> ‘copyofsnap1/test1/165/17’ 2019-12-05T06:10:41.689 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101’ -> ‘copyofsnap1/test1/101’ 2019-12-05T06:10:41.689 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/19’ -> ‘copyofsnap1/test1/101/19’ 2019-12-05T06:10:41.689 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/15’ -> ‘copyofsnap1/test1/101/15’ 2019-12-05T06:10:41.689 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/20’ -> ‘copyofsnap1/test1/101/20’ 2019-12-05T06:10:41.689 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/12’ -> ‘copyofsnap1/test1/101/12’ 2019-12-05T06:10:41.689 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/10’ -> ‘copyofsnap1/test1/101/10’ 2019-12-05T06:10:41.690 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/11’ -> ‘copyofsnap1/test1/101/11’ 2019-12-05T06:10:41.690 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/16’ -> ‘copyofsnap1/test1/101/16’ 2019-12-05T06:10:41.690 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/14’ -> ‘copyofsnap1/test1/101/14’ 2019-12-05T06:10:41.690 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/18’ -> ‘copyofsnap1/test1/101/18’ 2019-12-05T06:10:41.690 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/13’ -> ‘copyofsnap1/test1/101/13’ 2019-12-05T06:10:41.690 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/101/17’ -> ‘copyofsnap1/test1/101/17’ 2019-12-05T06:10:41.690 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148’ -> ‘copyofsnap1/test1/148’ 2019-12-05T06:10:41.691 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/19’ -> ‘copyofsnap1/test1/148/19’ 2019-12-05T06:10:41.691 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/15’ -> ‘copyofsnap1/test1/148/15’ 2019-12-05T06:10:41.691 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/20’ -> ‘copyofsnap1/test1/148/20’ 2019-12-05T06:10:41.691 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/12’ -> ‘copyofsnap1/test1/148/12’ 2019-12-05T06:10:41.691 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/10’ -> ‘copyofsnap1/test1/148/10’ 2019-12-05T06:10:41.691 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/11’ -> ‘copyofsnap1/test1/148/11’ 2019-12-05T06:10:41.691 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/16’ -> ‘copyofsnap1/test1/148/16’ 2019-12-05T06:10:41.692 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/14’ -> ‘copyofsnap1/test1/148/14’ 2019-12-05T06:10:41.692 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/18’ -> ‘copyofsnap1/test1/148/18’ 2019-12-05T06:10:41.692 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/13’ -> ‘copyofsnap1/test1/148/13’ 2019-12-05T06:10:41.692 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/148/17’ -> ‘copyofsnap1/test1/148/17’ 2019-12-05T06:10:41.692 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168’ -> ‘copyofsnap1/test1/168’ 2019-12-05T06:10:41.692 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/19’ -> ‘copyofsnap1/test1/168/19’ 2019-12-05T06:10:41.693 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/15’ -> ‘copyofsnap1/test1/168/15’ 2019-12-05T06:10:41.693 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/20’ -> ‘copyofsnap1/test1/168/20’ 2019-12-05T06:10:41.693 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/12’ -> ‘copyofsnap1/test1/168/12’ 2019-12-05T06:10:41.693 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/10’ -> ‘copyofsnap1/test1/168/10’ 2019-12-05T06:10:41.693 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/11’ -> ‘copyofsnap1/test1/168/11’ 2019-12-05T06:10:41.693 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/16’ -> ‘copyofsnap1/test1/168/16’ 2019-12-05T06:10:41.693 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/14’ -> ‘copyofsnap1/test1/168/14’ 2019-12-05T06:10:41.694 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/18’ -> ‘copyofsnap1/test1/168/18’ 2019-12-05T06:10:41.694 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/13’ -> ‘copyofsnap1/test1/168/13’ 2019-12-05T06:10:48.057 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:48.215 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/168/17’ -> ‘copyofsnap1/test1/168/17’ 2019-12-05T06:10:48.215 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179’ -> ‘copyofsnap1/test1/179’ 2019-12-05T06:10:48.216 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/19’ -> ‘copyofsnap1/test1/179/19’ 2019-12-05T06:10:48.216 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/15’ -> ‘copyofsnap1/test1/179/15’ 2019-12-05T06:10:48.216 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/20’ -> ‘copyofsnap1/test1/179/20’ 2019-12-05T06:10:48.216 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/12’ -> ‘copyofsnap1/test1/179/12’ 2019-12-05T06:10:48.216 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/10’ -> ‘copyofsnap1/test1/179/10’ 2019-12-05T06:10:48.216 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/11’ -> ‘copyofsnap1/test1/179/11’ 2019-12-05T06:10:48.216 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/16’ -> ‘copyofsnap1/test1/179/16’ 2019-12-05T06:10:48.217 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/14’ -> ‘copyofsnap1/test1/179/14’ 2019-12-05T06:10:48.217 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/18’ -> ‘copyofsnap1/test1/179/18’ 2019-12-05T06:10:48.217 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/13’ -> ‘copyofsnap1/test1/179/13’ 2019-12-05T06:10:48.217 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/179/17’ -> ‘copyofsnap1/test1/179/17’ 2019-12-05T06:10:48.217 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132’ -> ‘copyofsnap1/test1/132’ 2019-12-05T06:10:48.217 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/19’ -> ‘copyofsnap1/test1/132/19’ 2019-12-05T06:10:48.217 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/15’ -> ‘copyofsnap1/test1/132/15’ 2019-12-05T06:10:48.218 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/20’ -> ‘copyofsnap1/test1/132/20’ 2019-12-05T06:10:48.218 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/12’ -> ‘copyofsnap1/test1/132/12’ 2019-12-05T06:10:48.218 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/10’ -> ‘copyofsnap1/test1/132/10’ 2019-12-05T06:10:48.218 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/11’ -> ‘copyofsnap1/test1/132/11’ 2019-12-05T06:10:48.218 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/16’ -> ‘copyofsnap1/test1/132/16’ 2019-12-05T06:10:48.218 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/14’ -> ‘copyofsnap1/test1/132/14’ 2019-12-05T06:10:48.219 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/18’ -> ‘copyofsnap1/test1/132/18’ 2019-12-05T06:10:48.219 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/13’ -> ‘copyofsnap1/test1/132/13’ 2019-12-05T06:10:48.219 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/132/17’ -> ‘copyofsnap1/test1/132/17’ 2019-12-05T06:10:48.219 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199’ -> ‘copyofsnap1/test1/199’ 2019-12-05T06:10:48.219 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/19’ -> ‘copyofsnap1/test1/199/19’ 2019-12-05T06:10:48.219 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/15’ -> ‘copyofsnap1/test1/199/15’ 2019-12-05T06:10:48.219 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/20’ -> ‘copyofsnap1/test1/199/20’ 2019-12-05T06:10:48.220 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/12’ -> ‘copyofsnap1/test1/199/12’ 2019-12-05T06:10:48.220 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/10’ -> ‘copyofsnap1/test1/199/10’ 2019-12-05T06:10:48.220 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/11’ -> ‘copyofsnap1/test1/199/11’ 2019-12-05T06:10:48.220 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/16’ -> ‘copyofsnap1/test1/199/16’ 2019-12-05T06:10:48.220 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/14’ -> ‘copyofsnap1/test1/199/14’ 2019-12-05T06:10:48.220 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/18’ -> ‘copyofsnap1/test1/199/18’ 2019-12-05T06:10:48.220 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/13’ -> ‘copyofsnap1/test1/199/13’ 2019-12-05T06:10:48.221 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/199/17’ -> ‘copyofsnap1/test1/199/17’ 2019-12-05T06:10:48.221 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163’ -> ‘copyofsnap1/test1/163’ 2019-12-05T06:10:48.221 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/19’ -> ‘copyofsnap1/test1/163/19’ 2019-12-05T06:10:48.221 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/15’ -> ‘copyofsnap1/test1/163/15’ 2019-12-05T06:10:48.221 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/20’ -> ‘copyofsnap1/test1/163/20’ 2019-12-05T06:10:48.221 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/12’ -> ‘copyofsnap1/test1/163/12’ 2019-12-05T06:10:48.221 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/10’ -> ‘copyofsnap1/test1/163/10’ 2019-12-05T06:10:48.222 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/11’ -> ‘copyofsnap1/test1/163/11’ 2019-12-05T06:10:48.222 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/16’ -> ‘copyofsnap1/test1/163/16’ 2019-12-05T06:10:48.222 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/14’ -> ‘copyofsnap1/test1/163/14’ 2019-12-05T06:10:48.222 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/18’ -> ‘copyofsnap1/test1/163/18’ 2019-12-05T06:10:48.222 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/13’ -> ‘copyofsnap1/test1/163/13’ 2019-12-05T06:10:48.222 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/163/17’ -> ‘copyofsnap1/test1/163/17’ 2019-12-05T06:10:48.223 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153’ -> ‘copyofsnap1/test1/153’ 2019-12-05T06:10:48.223 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/19’ -> ‘copyofsnap1/test1/153/19’ 2019-12-05T06:10:48.223 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/15’ -> ‘copyofsnap1/test1/153/15’ 2019-12-05T06:10:48.223 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/20’ -> ‘copyofsnap1/test1/153/20’ 2019-12-05T06:10:48.223 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/12’ -> ‘copyofsnap1/test1/153/12’ 2019-12-05T06:10:48.224 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/10’ -> ‘copyofsnap1/test1/153/10’ 2019-12-05T06:10:48.224 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/11’ -> ‘copyofsnap1/test1/153/11’ 2019-12-05T06:10:48.224 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/16’ -> ‘copyofsnap1/test1/153/16’ 2019-12-05T06:10:48.224 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/14’ -> ‘copyofsnap1/test1/153/14’ 2019-12-05T06:10:48.224 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/18’ -> ‘copyofsnap1/test1/153/18’ 2019-12-05T06:10:48.224 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/13’ -> ‘copyofsnap1/test1/153/13’ 2019-12-05T06:10:48.225 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/153/17’ -> ‘copyofsnap1/test1/153/17’ 2019-12-05T06:10:48.225 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122’ -> ‘copyofsnap1/test1/122’ 2019-12-05T06:10:48.225 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/19’ -> ‘copyofsnap1/test1/122/19’ 2019-12-05T06:10:48.225 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/15’ -> ‘copyofsnap1/test1/122/15’ 2019-12-05T06:10:48.225 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/20’ -> ‘copyofsnap1/test1/122/20’ 2019-12-05T06:10:48.225 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/12’ -> ‘copyofsnap1/test1/122/12’ 2019-12-05T06:10:48.226 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/10’ -> ‘copyofsnap1/test1/122/10’ 2019-12-05T06:10:48.226 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/11’ -> ‘copyofsnap1/test1/122/11’ 2019-12-05T06:10:48.226 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/16’ -> ‘copyofsnap1/test1/122/16’ 2019-12-05T06:10:48.226 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/14’ -> ‘copyofsnap1/test1/122/14’ 2019-12-05T06:10:48.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.294+0000 7fad0cecd700 1 Processor -- start 2019-12-05T06:10:48.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.294+0000 7fad0cecd700 1 -- start start 2019-12-05T06:10:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.295+0000 7fad0cecd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fad0808c6d0 0x7fad0808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0cecd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad0808f720 0x7fad0808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0cecd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0808fb80 con 0x7fad0808fef0 2019-12-05T06:10:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0cecd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad0808df90 con 0x7fad0808f720 2019-12-05T06:10:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0cecd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fad080a3f70 con 0x7fad0808c6d0 2019-12-05T06:10:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad0808f720 0x7fad0808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:48.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fad0808f720 0x7fad0808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48118/0 (socket says 172.21.15.16:48118) 2019-12-05T06:10:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0759e700 1 -- 172.21.15.16:0/1798908150 learned_addr learned my addr 172.21.15.16:0/1798908150 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:10:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad06d9d700 1 --2- 172.21.15.16:0/1798908150 >> v2:172.21.15.76:3300/0 conn(0x7fad0808c6d0 0x7fad0808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0759e700 1 -- 172.21.15.16:0/1798908150 >> v2:172.21.15.76:3300/0 conn(0x7fad0808c6d0 msgr2=0x7fad0808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0759e700 1 --2- 172.21.15.16:0/1798908150 >> v2:172.21.15.76:3300/0 conn(0x7fad0808c6d0 0x7fad0808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0759e700 1 -- 172.21.15.16:0/1798908150 >> v1:172.21.15.16:6789/0 conn(0x7fad0808fef0 legacy=0x7fad0808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad05d9b700 1 -- 172.21.15.16:0/1798908150 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199396580 0 0) 0x7facfc001100 con 0x7fad0808fef0 2019-12-05T06:10:48.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0759e700 1 -- 172.21.15.16:0/1798908150 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0808d540 con 0x7fad0808f720 2019-12-05T06:10:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0759e700 1 --2- 172.21.15.16:0/1798908150 >> v2:172.21.15.16:3300/0 conn(0x7fad0808f720 0x7fad0808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7facf800f6b0 tx=0x7facf800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0759e700 1 --2- 172.21.15.16:0/1798908150 >> v2:172.21.15.16:3300/0 conn(0x7fad0808f720 0x7fad0808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7facf800f6b0 tx=0x7facf800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.296+0000 7fad0759e700 1 --2- 172.21.15.16:0/1798908150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0808f720 0x7fad0808d000 secure :-1 s=READY pgs=271 cs=0 l=1 rx=0x7facf800f6b0 tx=0x7facf800fa50).ready entity=mon.0 client_cookie=e8bf282698d10f81 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.297+0000 7fad0759e700 1 --2- 172.21.15.16:0/1798908150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0808f720 0x7fad0808d000 secure :-1 s=THROTTLE_DONE pgs=271 cs=0 l=1 rx=0x7facf800f6b0 tx=0x7facf800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.297+0000 7fad05d9b700 1 -- 172.21.15.16:0/1798908150 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facf8000a70 con 0x7fad0808f720 2019-12-05T06:10:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.297+0000 7fad0759e700 1 --2- 172.21.15.16:0/1798908150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0808f720 0x7fad0808d000 secure :-1 s=THROTTLE_DONE pgs=271 cs=0 l=1 rx=0x7facf800f6b0 tx=0x7facf800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.297+0000 7fad0759e700 1 --2- 172.21.15.16:0/1798908150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0808f720 0x7fad0808d000 secure :-1 s=THROTTLE_DONE pgs=271 cs=0 l=1 rx=0x7facf800f6b0 tx=0x7facf800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.297+0000 7fad05d9b700 1 -- 172.21.15.16:0/1798908150 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf80010e0 con 0x7fad0808f720 2019-12-05T06:10:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.297+0000 7fad05d9b700 1 -- 172.21.15.16:0/1798908150 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facf80017d0 con 0x7fad0808f720 2019-12-05T06:10:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.297+0000 7fad0cecd700 1 -- 172.21.15.16:0/1798908150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0808f720 msgr2=0x7fad0808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.297+0000 7fad0cecd700 1 --2- 172.21.15.16:0/1798908150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0808f720 0x7fad0808d000 secure :-1 s=READY pgs=271 cs=0 l=1 rx=0x7facf800f6b0 tx=0x7facf800fa50).stop 2019-12-05T06:10:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.298+0000 7fad0cecd700 1 -- 172.21.15.16:0/1798908150 shutdown_connections 2019-12-05T06:10:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.298+0000 7fad0cecd700 1 --2- 172.21.15.16:0/1798908150 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fad0808f720 0x7fad0808d000 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:48.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.298+0000 7fad0cecd700 1 --2- 172.21.15.16:0/1798908150 >> v2:172.21.15.76:3300/0 conn(0x7fad0808c6d0 0x7fad0808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.298+0000 7fad0cecd700 1 -- 172.21.15.16:0/1798908150 shutdown_connections 2019-12-05T06:10:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.298+0000 7fad0cecd700 1 -- 172.21.15.16:0/1798908150 wait complete. 2019-12-05T06:10:48.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.298+0000 7fad0cecd700 1 -- 172.21.15.16:0/1798908150 >> 172.21.15.16:0/1798908150 conn(0x7fad08061940 msgr2=0x7fad08096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:48.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.303+0000 7fad0cecd700 1 Processor -- start 2019-12-05T06:10:48.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.303+0000 7fad0cecd700 1 -- start start 2019-12-05T06:10:48.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0cecd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:48.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0cecd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad08098da0 con 0x7fad08132d00 2019-12-05T06:10:48.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0cecd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad0809f0a0 con 0x7fad0814d400 2019-12-05T06:10:48.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0cecd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad0809bb50 con 0x7fad0814d7f0 2019-12-05T06:10:48.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fad08132d00 0x7fad081550f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33292/0 (socket says 172.21.15.16:33292) 2019-12-05T06:10:48.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0659c700 1 -- 172.21.15.16:0/3798491668 learned_addr learned my addr 172.21.15.16:0/3798491668 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:10:48.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:48.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2694496652 0 0) 0x7facfc000f40 con 0x7fad08132d00 2019-12-05T06:10:48.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facec002e80 con 0x7fad08132d00 2019-12-05T06:10:48.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3078524270 0 0) 0x7fad00001430 con 0x7fad0814d400 2019-12-05T06:10:48.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facec004170 con 0x7fad0814d400 2019-12-05T06:10:48.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0759e700 1 -- 172.21.15.16:0/3798491668 >> v1:172.21.15.76:6789/0 conn(0x7fad0814d400 legacy=0x7fad08131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:48.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0759e700 1 -- 172.21.15.16:0/3798491668 >> v1:172.21.15.16:6789/0 conn(0x7fad08132d00 legacy=0x7fad081550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:48.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0759e700 1 -- 172.21.15.16:0/3798491668 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad0809b0f0 con 0x7fad0814d7f0 2019-12-05T06:10:48.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.305+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.306+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.306+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=READY pgs=82 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:48.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.306+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.306+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facf8008180 con 0x7fad0814d7f0 2019-12-05T06:10:48.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.306+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.306+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facf8000ba0 con 0x7fad0814d7f0 2019-12-05T06:10:48.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.306+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.306+0000 7fad0cecd700 1 -- 172.21.15.16:0/3798491668 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad0809b0f0 con 0x7fad0814d7f0 2019-12-05T06:10:48.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.306+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facf80014a0 con 0x7fad0814d7f0 2019-12-05T06:10:48.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.306+0000 7fad0cecd700 1 -- 172.21.15.16:0/3798491668 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad0809b0f0 con 0x7fad0814d7f0 2019-12-05T06:10:48.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.307+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.307+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7facf8001200 con 0x7fad0814d7f0 2019-12-05T06:10:48.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.307+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.308+0000 7facf77fe700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7facec023650 0x7facec025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:48.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.308+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7facf8045d70 con 0x7fad0814d7f0 2019-12-05T06:10:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.308+0000 7fad06d9d700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7facec023650 0x7facec025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.309+0000 7fad06d9d700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7facec023650 0x7facec025ae0 crc :-1 s=READY pgs=136 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:48.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.309+0000 7fad0cecd700 1 -- 172.21.15.16:0/3798491668 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad0809b0f0 con 0x7fad0814d7f0 2019-12-05T06:10:48.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.316+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.317+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7facf8000eb0 con 0x7fad0814d7f0 2019-12-05T06:10:48.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.524+0000 7fad0cecd700 1 -- 172.21.15.16:0/3798491668 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad0809b0f0 con 0x7fad0814d7f0 2019-12-05T06:10:48.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.524+0000 7fad0759e700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=THROTTLE_DONE pgs=82 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.525+0000 7facf77fe700 1 -- 172.21.15.16:0/3798491668 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2108 (secure 0 0 0) 0x7facf8001160 con 0x7fad0814d7f0 2019-12-05T06:10:48.524 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2019-12-05T06:10:48.524 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:10:48.525 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:57.018632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":34,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.529+0000 7fad0cecd700 1 -- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7facec023650 msgr2=0x7facec025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.529+0000 7fad0cecd700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7facec023650 0x7facec025ae0 crc :-1 s=READY pgs=136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.529+0000 7fad0cecd700 1 -- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 msgr2=0x7fad08132530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:48.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.529+0000 7fad0cecd700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 secure :-1 s=READY pgs=82 cs=0 l=1 rx=0x7facf8010380 tx=0x7facf8010720).stop 2019-12-05T06:10:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.529+0000 7fad0cecd700 1 -- 172.21.15.16:0/3798491668 shutdown_connections 2019-12-05T06:10:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.529+0000 7fad0cecd700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7facec023650 0x7facec025ae0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.529+0000 7fad0cecd700 1 --2- 172.21.15.16:0/3798491668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fad0814d7f0 0x7fad08132530 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.529+0000 7fad0cecd700 1 -- 172.21.15.16:0/3798491668 shutdown_connections 2019-12-05T06:10:48.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.529+0000 7fad0cecd700 1 -- 172.21.15.16:0/3798491668 wait complete. 2019-12-05T06:10:48.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.529+0000 7fad0cecd700 1 -- 172.21.15.16:0/3798491668 >> 172.21.15.16:0/3798491668 conn(0x7fad080714f0 msgr2=0x7fad0809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:48.537 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:10:48.538 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:10:48.538 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:10:48.555 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:10:48.555 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:48.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.773+0000 7f6a681f0700 1 Processor -- start 2019-12-05T06:10:48.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.773+0000 7f6a681f0700 1 -- start start 2019-12-05T06:10:48.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.775+0000 7f6a681f0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a6008d2c0 0x7f6a6008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.775+0000 7f6a681f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a6008fef0 0x7f6a6008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.775+0000 7f6a681f0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a600958e0 con 0x7f6a6008c6d0 2019-12-05T06:10:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.775+0000 7f6a681f0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a6008df90 con 0x7f6a6008fef0 2019-12-05T06:10:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.775+0000 7f6a681f0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a600a3f70 con 0x7f6a6008d2c0 2019-12-05T06:10:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.775+0000 7f6a66f8e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a6008fef0 0x7f6a6008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.775+0000 7f6a6678d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6a6008c6d0 0x7f6a6008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33306/0 (socket says 172.21.15.16:33306) 2019-12-05T06:10:48.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.775+0000 7f6a6678d700 1 -- 172.21.15.16:0/3032774301 learned_addr learned my addr 172.21.15.16:0/3032774301 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:10:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.775+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/3032774301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a6008d2c0 0x7f6a6008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.776+0000 7f6a66f8e700 1 -- 172.21.15.16:0/3032774301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a6008d2c0 msgr2=0x7f6a6008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.776+0000 7f6a66f8e700 1 --2- 172.21.15.16:0/3032774301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a6008d2c0 0x7f6a6008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.776+0000 7f6a66f8e700 1 -- 172.21.15.16:0/3032774301 >> v1:172.21.15.16:6789/0 conn(0x7f6a6008c6d0 legacy=0x7f6a6008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.776+0000 7f6a66f8e700 1 -- 172.21.15.16:0/3032774301 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6008e620 con 0x7f6a6008fef0 2019-12-05T06:10:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.776+0000 7f6a6578b700 1 -- 172.21.15.16:0/3032774301 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1026193957 0 0) 0x7f6a580014d0 con 0x7f6a6008c6d0 2019-12-05T06:10:48.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.776+0000 7f6a66f8e700 1 --2- 172.21.15.16:0/3032774301 >> v2:172.21.15.16:3300/0 conn(0x7f6a6008fef0 0x7f6a6008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6a5000af50 tx=0x7f6a5000b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.776+0000 7f6a66f8e700 1 --2- 172.21.15.16:0/3032774301 >> v2:172.21.15.16:3300/0 conn(0x7f6a6008fef0 0x7f6a6008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6a5000af50 tx=0x7f6a5000b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.776+0000 7f6a66f8e700 1 --2- 172.21.15.16:0/3032774301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a6008fef0 0x7f6a6008f460 secure :-1 s=READY pgs=274 cs=0 l=1 rx=0x7f6a5000af50 tx=0x7f6a5000b6e0).ready entity=mon.0 client_cookie=92843cd6055d79b1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:48.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.777+0000 7f6a66f8e700 1 --2- 172.21.15.16:0/3032774301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a6008fef0 0x7f6a6008f460 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f6a5000af50 tx=0x7f6a5000b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.777+0000 7f6a6578b700 1 -- 172.21.15.16:0/3032774301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a50009ba0 con 0x7f6a6008fef0 2019-12-05T06:10:48.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.777+0000 7f6a66f8e700 1 --2- 172.21.15.16:0/3032774301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a6008fef0 0x7f6a6008f460 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f6a5000af50 tx=0x7f6a5000b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.777+0000 7f6a66f8e700 1 --2- 172.21.15.16:0/3032774301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a6008fef0 0x7f6a6008f460 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f6a5000af50 tx=0x7f6a5000b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.777+0000 7f6a6578b700 1 -- 172.21.15.16:0/3032774301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a500097c0 con 0x7f6a6008fef0 2019-12-05T06:10:48.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.777+0000 7f6a6578b700 1 -- 172.21.15.16:0/3032774301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a5000a180 con 0x7f6a6008fef0 2019-12-05T06:10:48.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.777+0000 7f6a681f0700 1 -- 172.21.15.16:0/3032774301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a6008fef0 msgr2=0x7f6a6008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:48.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.777+0000 7f6a681f0700 1 --2- 172.21.15.16:0/3032774301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a6008fef0 0x7f6a6008f460 secure :-1 s=READY pgs=274 cs=0 l=1 rx=0x7f6a5000af50 tx=0x7f6a5000b6e0).stop 2019-12-05T06:10:48.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.779+0000 7f6a681f0700 1 -- 172.21.15.16:0/3032774301 shutdown_connections 2019-12-05T06:10:48.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.779+0000 7f6a681f0700 1 --2- 172.21.15.16:0/3032774301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a6008fef0 0x7f6a6008f460 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:48.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.779+0000 7f6a681f0700 1 --2- 172.21.15.16:0/3032774301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a6008d2c0 0x7f6a6008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:48.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.779+0000 7f6a681f0700 1 -- 172.21.15.16:0/3032774301 shutdown_connections 2019-12-05T06:10:48.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.779+0000 7f6a681f0700 1 -- 172.21.15.16:0/3032774301 wait complete. 2019-12-05T06:10:48.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.779+0000 7f6a681f0700 1 -- 172.21.15.16:0/3032774301 >> 172.21.15.16:0/3032774301 conn(0x7f6a60061940 msgr2=0x7f6a60096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.784+0000 7f6a681f0700 1 Processor -- start 2019-12-05T06:10:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.784+0000 7f6a681f0700 1 -- start start 2019-12-05T06:10:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.785+0000 7f6a681f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a600992d0 0x7f6a60098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.785+0000 7f6a681f0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a60089fb0 con 0x7f6a60156cb0 2019-12-05T06:10:48.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.785+0000 7f6a681f0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6006e300 con 0x7f6a6006df10 2019-12-05T06:10:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.785+0000 7f6a681f0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a60097370 con 0x7f6a600992d0 2019-12-05T06:10:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.786+0000 7f6a65f8c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a600992d0 0x7f6a60098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.786+0000 7f6a65f8c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a600992d0 0x7f6a60098b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48148/0 (socket says 172.21.15.16:48148) 2019-12-05T06:10:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.786+0000 7f6a65f8c700 1 -- 172.21.15.16:0/1208355608 learned_addr learned my addr 172.21.15.16:0/1208355608 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:10:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.786+0000 7f6a5effd700 1 -- 172.21.15.16:0/1208355608 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231258044 0 0) 0x7f6a58001480 con 0x7f6a6006df10 2019-12-05T06:10:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.786+0000 7f6a65f8c700 1 -- 172.21.15.16:0/1208355608 >> v1:172.21.15.76:6789/0 conn(0x7f6a6006df10 legacy=0x7f6a60098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:48.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.786+0000 7f6a65f8c700 1 -- 172.21.15.16:0/1208355608 >> v1:172.21.15.16:6789/0 conn(0x7f6a60156cb0 legacy=0x7f6a601590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:48.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.786+0000 7f6a65f8c700 1 -- 172.21.15.16:0/1208355608 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60097a00 con 0x7f6a600992d0 2019-12-05T06:10:48.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/1208355608 >> v2:172.21.15.16:3300/0 conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/1208355608 >> v2:172.21.15.16:3300/0 conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=READY pgs=276 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=THROTTLE_DONE pgs=276 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=THROTTLE_DONE pgs=276 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a5effd700 1 -- 172.21.15.16:0/1208355608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a54000940 con 0x7f6a600992d0 2019-12-05T06:10:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=THROTTLE_DONE pgs=276 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a5effd700 1 -- 172.21.15.16:0/1208355608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a54000e80 con 0x7f6a600992d0 2019-12-05T06:10:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a681f0700 1 -- 172.21.15.16:0/1208355608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a60097a00 con 0x7f6a600992d0 2019-12-05T06:10:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a5effd700 1 -- 172.21.15.16:0/1208355608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a540014d0 con 0x7f6a600992d0 2019-12-05T06:10:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.787+0000 7f6a681f0700 1 -- 172.21.15.16:0/1208355608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a60097a00 con 0x7f6a600992d0 2019-12-05T06:10:48.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.789+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=THROTTLE_DONE pgs=276 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.790+0000 7f6a5effd700 1 -- 172.21.15.16:0/1208355608 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6a54001070 con 0x7f6a600992d0 2019-12-05T06:10:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.790+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=THROTTLE_DONE pgs=276 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.790+0000 7f6a5effd700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a40023630 0x7f6a40025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:48.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.790+0000 7f6a5effd700 1 -- 172.21.15.16:0/1208355608 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f6a54045e60 con 0x7f6a600992d0 2019-12-05T06:10:48.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.790+0000 7f6a6678d700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a40023630 0x7f6a40025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:48.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.791+0000 7f6a6678d700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a40023630 0x7f6a40025ac0 crc :-1 s=READY pgs=137 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:48.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.805+0000 7f6a681f0700 1 -- 172.21.15.16:0/1208355608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a60097a00 con 0x7f6a600992d0 2019-12-05T06:10:48.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.813+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=THROTTLE_DONE pgs=276 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:48.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:48.813+0000 7f6a5effd700 1 -- 172.21.15.16:0/1208355608 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6a54000d80 con 0x7f6a600992d0 2019-12-05T06:10:49.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.066+0000 7f6a681f0700 1 -- 172.21.15.16:0/1208355608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a60097a00 con 0x7f6a600992d0 2019-12-05T06:10:49.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.066+0000 7f6a65f8c700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=THROTTLE_DONE pgs=276 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.066+0000 7f6a5effd700 1 -- 172.21.15.16:0/1208355608 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f6a54000d80 con 0x7f6a600992d0 2019-12-05T06:10:49.065 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2019-12-05T06:10:49.065 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:10:49.065 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:57.018632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":34,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:49.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.069+0000 7f6a681f0700 1 -- 172.21.15.16:0/1208355608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a40023630 msgr2=0x7f6a40025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:49.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.069+0000 7f6a681f0700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a40023630 0x7f6a40025ac0 crc :-1 s=READY pgs=137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:49.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.069+0000 7f6a681f0700 1 -- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 msgr2=0x7f6a60098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:49.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.069+0000 7f6a681f0700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 0x7f6a60098b00 secure :-1 s=READY pgs=276 cs=0 l=1 rx=0x7f6a54010450 tx=0x7f6a540107f0).stop 2019-12-05T06:10:49.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.069+0000 7f6a681f0700 1 -- 172.21.15.16:0/1208355608 shutdown_connections 2019-12-05T06:10:49.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.069+0000 7f6a681f0700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a40023630 0x7f6a40025ac0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:49.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.069+0000 7f6a681f0700 1 --2- 172.21.15.16:0/1208355608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a600992d0 0x7f6a60098b00 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:49.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.070+0000 7f6a681f0700 1 -- 172.21.15.16:0/1208355608 shutdown_connections 2019-12-05T06:10:49.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.070+0000 7f6a681f0700 1 -- 172.21.15.16:0/1208355608 wait complete. 2019-12-05T06:10:49.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:49.070+0000 7f6a681f0700 1 -- 172.21.15.16:0/1208355608 >> 172.21.15.16:0/1208355608 conn(0x7f6a600714f0 msgr2=0x7f6a6009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:49.078 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:10:51.079 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:51.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.366+0000 7ffbd3326700 1 Processor -- start 2019-12-05T06:10:51.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.366+0000 7ffbd3326700 1 -- start start 2019-12-05T06:10:51.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.368+0000 7ffbd3326700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ffbcc08d2c0 0x7ffbcc08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:51.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.368+0000 7ffbd3326700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffbcc08fef0 0x7ffbcc08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:51.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.368+0000 7ffbd3326700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc0958e0 con 0x7ffbcc08c6d0 2019-12-05T06:10:51.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.368+0000 7ffbd3326700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbcc08df90 con 0x7ffbcc08d2c0 2019-12-05T06:10:51.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.368+0000 7ffbd3326700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc0a3f70 con 0x7ffbcc08fef0 2019-12-05T06:10:51.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.368+0000 7ffbd18c3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffbcc08c6d0 0x7ffbcc08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33324/0 (socket says 172.21.15.16:33324) 2019-12-05T06:10:51.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.368+0000 7ffbd18c3700 1 -- 172.21.15.16:0/931318116 learned_addr learned my addr 172.21.15.16:0/931318116 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:10:51.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.368+0000 7ffbd20c4700 1 --2- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffbcc08fef0 0x7ffbcc08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:51.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.368+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/931318116 >> v2:172.21.15.76:3300/0 conn(0x7ffbcc08d2c0 0x7ffbcc08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:51.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.369+0000 7ffbd08c1700 1 -- 172.21.15.16:0/931318116 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852844189 0 0) 0x7ffbbc001100 con 0x7ffbcc08c6d0 2019-12-05T06:10:51.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.369+0000 7ffbd08c1700 1 -- 172.21.15.16:0/931318116 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffbb4002e80 con 0x7ffbcc08c6d0 2019-12-05T06:10:51.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.369+0000 7ffbd10c2700 1 -- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffbcc08fef0 msgr2=0x7ffbcc08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:51.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.369+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffbcc08fef0 0x7ffbcc08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:51.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.369+0000 7ffbd10c2700 1 -- 172.21.15.16:0/931318116 >> v1:172.21.15.16:6789/0 conn(0x7ffbcc08c6d0 legacy=0x7ffbcc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:51.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.369+0000 7ffbd10c2700 1 -- 172.21.15.16:0/931318116 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc08e620 con 0x7ffbcc08d2c0 2019-12-05T06:10:51.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.369+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/931318116 >> v2:172.21.15.76:3300/0 conn(0x7ffbcc08d2c0 0x7ffbcc08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffbc000b2d0 tx=0x7ffbc000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.370+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/931318116 >> v2:172.21.15.76:3300/0 conn(0x7ffbcc08d2c0 0x7ffbcc08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffbc000b2d0 tx=0x7ffbc000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.370+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbcc08d2c0 0x7ffbcc08ef20 secure :-1 s=READY pgs=229 cs=0 l=1 rx=0x7ffbc000b2d0 tx=0x7ffbc000b5d0).ready entity=mon.1 client_cookie=a66660befd1c6c16 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:51.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.370+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbcc08d2c0 0x7ffbcc08ef20 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7ffbc000b2d0 tx=0x7ffbc000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.370+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbcc08d2c0 0x7ffbcc08ef20 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7ffbc000b2d0 tx=0x7ffbc000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.370+0000 7ffbd08c1700 1 -- 172.21.15.16:0/931318116 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbc0007590 con 0x7ffbcc08d2c0 2019-12-05T06:10:51.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.370+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbcc08d2c0 0x7ffbcc08ef20 secure :-1 s=THROTTLE_DONE pgs=229 cs=0 l=1 rx=0x7ffbc000b2d0 tx=0x7ffbc000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.370+0000 7ffbd08c1700 1 -- 172.21.15.16:0/931318116 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbc0009d30 con 0x7ffbcc08d2c0 2019-12-05T06:10:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.370+0000 7ffbd08c1700 1 -- 172.21.15.16:0/931318116 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbc000a400 con 0x7ffbcc08d2c0 2019-12-05T06:10:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.370+0000 7ffbd3326700 1 -- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbcc08d2c0 msgr2=0x7ffbcc08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.370+0000 7ffbd3326700 1 --2- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbcc08d2c0 0x7ffbcc08ef20 secure :-1 s=READY pgs=229 cs=0 l=1 rx=0x7ffbc000b2d0 tx=0x7ffbc000b5d0).stop 2019-12-05T06:10:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.372+0000 7ffbd3326700 1 -- 172.21.15.16:0/931318116 shutdown_connections 2019-12-05T06:10:51.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.372+0000 7ffbd3326700 1 --2- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffbcc08d2c0 0x7ffbcc08ef20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.372+0000 7ffbd3326700 1 --2- 172.21.15.16:0/931318116 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffbcc08fef0 0x7ffbcc08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.373+0000 7ffbd3326700 1 -- 172.21.15.16:0/931318116 shutdown_connections 2019-12-05T06:10:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.373+0000 7ffbd3326700 1 -- 172.21.15.16:0/931318116 wait complete. 2019-12-05T06:10:51.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.373+0000 7ffbd3326700 1 -- 172.21.15.16:0/931318116 >> 172.21.15.16:0/931318116 conn(0x7ffbcc061940 msgr2=0x7ffbcc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:51.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.378+0000 7ffbd3326700 1 Processor -- start 2019-12-05T06:10:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.378+0000 7ffbd3326700 1 -- start start 2019-12-05T06:10:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.379+0000 7ffbd3326700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffbcc1302f0 0x7ffbcc09d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.379+0000 7ffbd3326700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffbcc09e7d0 0x7ffbcc09dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.379+0000 7ffbd3326700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbcc12cb90 con 0x7ffbcc09e280 2019-12-05T06:10:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.379+0000 7ffbd3326700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbcc131ff0 con 0x7ffbcc1302f0 2019-12-05T06:10:51.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.379+0000 7ffbd3326700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc1315c0 con 0x7ffbcc09e7d0 2019-12-05T06:10:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.379+0000 7ffbd18c3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffbcc1302f0 0x7ffbcc09d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.379+0000 7ffbd20c4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffbcc09e280 0x7ffbcc06dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33334/0 (socket says 172.21.15.16:33334) 2019-12-05T06:10:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.380+0000 7ffbd20c4700 1 -- 172.21.15.16:0/1133392095 learned_addr learned my addr 172.21.15.16:0/1133392095 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:10:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.380+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffbcc09e7d0 0x7ffbcc09dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.380+0000 7ffbd18c3700 1 -- 172.21.15.16:0/1133392095 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffbcc09e7d0 msgr2=0x7ffbcc09dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.380+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffbcc09e7d0 0x7ffbcc09dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:51.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.380+0000 7ffbd18c3700 1 -- 172.21.15.16:0/1133392095 >> v1:172.21.15.16:6789/0 conn(0x7ffbcc09e280 legacy=0x7ffbcc06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:51.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.380+0000 7ffbca7fc700 1 -- 172.21.15.16:0/1133392095 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760840013 0 0) 0x7ffbc4002180 con 0x7ffbcc09e280 2019-12-05T06:10:51.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.380+0000 7ffbd18c3700 1 -- 172.21.15.16:0/1133392095 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc132680 con 0x7ffbcc1302f0 2019-12-05T06:10:51.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.380+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> v2:172.21.15.16:3300/0 conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.380+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> v2:172.21.15.16:3300/0 conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.380+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=READY pgs=279 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:51.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.381+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.381+0000 7ffbca7fc700 1 -- 172.21.15.16:0/1133392095 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbbc000d80 con 0x7ffbcc1302f0 2019-12-05T06:10:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.381+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.381+0000 7ffbca7fc700 1 -- 172.21.15.16:0/1133392095 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbbc000c90 con 0x7ffbcc1302f0 2019-12-05T06:10:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.381+0000 7ffbd3326700 1 -- 172.21.15.16:0/1133392095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbcc132680 con 0x7ffbcc1302f0 2019-12-05T06:10:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.381+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.381+0000 7ffbca7fc700 1 -- 172.21.15.16:0/1133392095 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbbc001660 con 0x7ffbcc1302f0 2019-12-05T06:10:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.381+0000 7ffbd3326700 1 -- 172.21.15.16:0/1133392095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbcc132680 con 0x7ffbcc1302f0 2019-12-05T06:10:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.382+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.383+0000 7ffbca7fc700 1 -- 172.21.15.16:0/1133392095 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ffbbc000f20 con 0x7ffbcc1302f0 2019-12-05T06:10:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.383+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.383+0000 7ffbca7fc700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffbac023630 0x7ffbac025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:51.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.383+0000 7ffbca7fc700 1 -- 172.21.15.16:0/1133392095 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7ffbbc045d50 con 0x7ffbcc1302f0 2019-12-05T06:10:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.384+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffbac023630 0x7ffbac025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:51.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.384+0000 7ffbd10c2700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffbac023630 0x7ffbac025ac0 crc :-1 s=READY pgs=138 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:51.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.397+0000 7ffbd3326700 1 -- 172.21.15.16:0/1133392095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbcc132680 con 0x7ffbcc1302f0 2019-12-05T06:10:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.405+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.405+0000 7ffbca7fc700 1 -- 172.21.15.16:0/1133392095 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ffbbc001b70 con 0x7ffbcc1302f0 2019-12-05T06:10:51.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.642+0000 7ffbd3326700 1 -- 172.21.15.16:0/1133392095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffbcc132680 con 0x7ffbcc1302f0 2019-12-05T06:10:51.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.643+0000 7ffbd18c3700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=THROTTLE_DONE pgs=279 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:51.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.644+0000 7ffbca7fc700 1 -- 172.21.15.16:0/1133392095 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2108 (secure 0 0 0) 0x7ffbbc000d60 con 0x7ffbcc1302f0 2019-12-05T06:10:51.643 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2019-12-05T06:10:51.643 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:10:51.643 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:57.018632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":34,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.648+0000 7ffbd3326700 1 -- 172.21.15.16:0/1133392095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffbac023630 msgr2=0x7ffbac025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:51.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.648+0000 7ffbd3326700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffbac023630 0x7ffbac025ac0 crc :-1 s=READY pgs=138 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.648+0000 7ffbd3326700 1 -- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 msgr2=0x7ffbcc09d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.648+0000 7ffbd3326700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 0x7ffbcc09d800 secure :-1 s=READY pgs=279 cs=0 l=1 rx=0x7ffbbc010380 tx=0x7ffbbc0101b0).stop 2019-12-05T06:10:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.648+0000 7ffbd3326700 1 -- 172.21.15.16:0/1133392095 shutdown_connections 2019-12-05T06:10:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.648+0000 7ffbd3326700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffbcc09e7d0 0x7ffbcc09dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.648+0000 7ffbd3326700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffbac023630 0x7ffbac025ac0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.648+0000 7ffbd3326700 1 --2- 172.21.15.16:0/1133392095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffbcc1302f0 0x7ffbcc09d800 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:51.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.648+0000 7ffbd3326700 1 -- 172.21.15.16:0/1133392095 shutdown_connections 2019-12-05T06:10:51.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.649+0000 7ffbd3326700 1 -- 172.21.15.16:0/1133392095 wait complete. 2019-12-05T06:10:51.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:51.649+0000 7ffbd3326700 1 -- 172.21.15.16:0/1133392095 >> 172.21.15.16:0/1133392095 conn(0x7ffbcc0a2850 msgr2=0x7ffbcc06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:51.657 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:10:53.658 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:53.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.953+0000 7fc6de027700 1 Processor -- start 2019-12-05T06:10:53.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.954+0000 7fc6de027700 1 -- start start 2019-12-05T06:10:53.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.955+0000 7fc6de027700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc6d808d2c0 0x7fc6d808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:53.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.955+0000 7fc6de027700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d80958e0 con 0x7fc6d808c6d0 2019-12-05T06:10:53.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.955+0000 7fc6de027700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d80629e0 con 0x7fc6d808fef0 2019-12-05T06:10:53.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.955+0000 7fc6de027700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d808df90 con 0x7fc6d808d2c0 2019-12-05T06:10:53.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.956+0000 7fc6d7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc6d808c6d0 0x7fc6d808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33340/0 (socket says 172.21.15.16:33340) 2019-12-05T06:10:53.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.956+0000 7fc6d7fff700 1 -- 172.21.15.16:0/3801717868 learned_addr learned my addr 172.21.15.16:0/3801717868 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:10:53.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.956+0000 7fc6d77fe700 1 --2- 172.21.15.16:0/3801717868 >> v2:172.21.15.76:3300/0 conn(0x7fc6d808d2c0 0x7fc6d808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:53.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.956+0000 7fc6d6ffd700 1 -- 172.21.15.16:0/3801717868 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3702710126 0 0) 0x7fc6c80014d0 con 0x7fc6d808c6d0 2019-12-05T06:10:53.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.956+0000 7fc6d6ffd700 1 -- 172.21.15.16:0/3801717868 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c4002e80 con 0x7fc6d808c6d0 2019-12-05T06:10:53.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.956+0000 7fc6d6ffd700 1 -- 172.21.15.16:0/3801717868 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299395780 0 0) 0x7fc6d00009b0 con 0x7fc6d808fef0 2019-12-05T06:10:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.957+0000 7fc6d6ffd700 1 -- 172.21.15.16:0/3801717868 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6c4004170 con 0x7fc6d808fef0 2019-12-05T06:10:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.957+0000 7fc6d77fe700 1 -- 172.21.15.16:0/3801717868 >> v1:172.21.15.76:6789/0 conn(0x7fc6d808fef0 legacy=0x7fc6d808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.957+0000 7fc6d77fe700 1 -- 172.21.15.16:0/3801717868 >> v1:172.21.15.16:6789/0 conn(0x7fc6d808c6d0 legacy=0x7fc6d808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.957+0000 7fc6d6ffd700 1 -- 172.21.15.16:0/3801717868 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3470057448 0 0) 0x7fc6c80014b0 con 0x7fc6d808c6d0 2019-12-05T06:10:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.957+0000 7fc6d77fe700 1 -- 172.21.15.16:0/3801717868 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d808e620 con 0x7fc6d808d2c0 2019-12-05T06:10:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.957+0000 7fc6d77fe700 1 --2- 172.21.15.16:0/3801717868 >> v2:172.21.15.76:3300/0 conn(0x7fc6d808d2c0 0x7fc6d808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc6c000af50 tx=0x7fc6c000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.957+0000 7fc6d77fe700 1 --2- 172.21.15.16:0/3801717868 >> v2:172.21.15.76:3300/0 conn(0x7fc6d808d2c0 0x7fc6d808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc6c000af50 tx=0x7fc6c000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.957+0000 7fc6d77fe700 1 --2- 172.21.15.16:0/3801717868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6d808d2c0 0x7fc6d808ef20 secure :-1 s=READY pgs=231 cs=0 l=1 rx=0x7fc6c000af50 tx=0x7fc6c000b760).ready entity=mon.1 client_cookie=9c75d3e70580e1ce server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.958+0000 7fc6d77fe700 1 --2- 172.21.15.16:0/3801717868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6d808d2c0 0x7fc6d808ef20 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fc6c000af50 tx=0x7fc6c000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.958+0000 7fc6d77fe700 1 --2- 172.21.15.16:0/3801717868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6d808d2c0 0x7fc6d808ef20 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fc6c000af50 tx=0x7fc6c000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.958+0000 7fc6d6ffd700 1 -- 172.21.15.16:0/3801717868 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6c0000960 con 0x7fc6d808d2c0 2019-12-05T06:10:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.958+0000 7fc6d77fe700 1 --2- 172.21.15.16:0/3801717868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6d808d2c0 0x7fc6d808ef20 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fc6c000af50 tx=0x7fc6c000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.958+0000 7fc6d6ffd700 1 -- 172.21.15.16:0/3801717868 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c00098e0 con 0x7fc6d808d2c0 2019-12-05T06:10:53.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.958+0000 7fc6d6ffd700 1 -- 172.21.15.16:0/3801717868 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6c0009eb0 con 0x7fc6d808d2c0 2019-12-05T06:10:53.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.958+0000 7fc6de027700 1 -- 172.21.15.16:0/3801717868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6d808d2c0 msgr2=0x7fc6d808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:53.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.958+0000 7fc6de027700 1 --2- 172.21.15.16:0/3801717868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6d808d2c0 0x7fc6d808ef20 secure :-1 s=READY pgs=231 cs=0 l=1 rx=0x7fc6c000af50 tx=0x7fc6c000b760).stop 2019-12-05T06:10:53.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.960+0000 7fc6de027700 1 -- 172.21.15.16:0/3801717868 shutdown_connections 2019-12-05T06:10:53.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.960+0000 7fc6de027700 1 --2- 172.21.15.16:0/3801717868 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc6d808d2c0 0x7fc6d808ef20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:53.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.960+0000 7fc6de027700 1 -- 172.21.15.16:0/3801717868 shutdown_connections 2019-12-05T06:10:53.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.960+0000 7fc6de027700 1 -- 172.21.15.16:0/3801717868 wait complete. 2019-12-05T06:10:53.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.960+0000 7fc6de027700 1 -- 172.21.15.16:0/3801717868 >> 172.21.15.16:0/3801717868 conn(0x7fc6d8061940 msgr2=0x7fc6d8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:53.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.965+0000 7fc6de027700 1 Processor -- start 2019-12-05T06:10:53.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.966+0000 7fc6de027700 1 -- start start 2019-12-05T06:10:53.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.967+0000 7fc6de027700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc6d806dc50 0x7fc6d806e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:53.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.968+0000 7fc6de027700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d81350e0 con 0x7fc6d8134c80 2019-12-05T06:10:53.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.968+0000 7fc6de027700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d8128af0 con 0x7fc6d8135450 2019-12-05T06:10:53.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.968+0000 7fc6de027700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d8132830 con 0x7fc6d806dc50 2019-12-05T06:10:53.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.968+0000 7fc6d7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc6d806dc50 0x7fc6d806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:53.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.968+0000 7fc6d7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc6d806dc50 0x7fc6d806e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48178/0 (socket says 172.21.15.16:48178) 2019-12-05T06:10:53.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.968+0000 7fc6dcdc5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc6d8134c80 0x7fc6d81577d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33350/0 (socket says 172.21.15.16:33350) 2019-12-05T06:10:53.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.968+0000 7fc6d7fff700 1 -- 172.21.15.16:0/2350210832 learned_addr learned my addr 172.21.15.16:0/2350210832 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:10:53.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d7fff700 1 -- 172.21.15.16:0/2350210832 >> v1:172.21.15.76:6789/0 conn(0x7fc6d8135450 legacy=0x7fc6d8134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:53.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d7fff700 1 -- 172.21.15.16:0/2350210832 >> v1:172.21.15.16:6789/0 conn(0x7fc6d8134c80 legacy=0x7fc6d81577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:53.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d7fff700 1 -- 172.21.15.16:0/2350210832 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d8133b80 con 0x7fc6d806dc50 2019-12-05T06:10:53.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d4ff9700 1 -- 172.21.15.16:0/2350210832 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1626202481 0 0) 0x7fc6c000ac00 con 0x7fc6d8135450 2019-12-05T06:10:53.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d7fff700 1 --2- 172.21.15.16:0/2350210832 >> v2:172.21.15.16:3300/0 conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d7fff700 1 --2- 172.21.15.16:0/2350210832 >> v2:172.21.15.16:3300/0 conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d7fff700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=READY pgs=282 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:53.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d7fff700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d4ff9700 1 -- 172.21.15.16:0/2350210832 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6c8001010 con 0x7fc6d806dc50 2019-12-05T06:10:53.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d7fff700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d7fff700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d4ff9700 1 -- 172.21.15.16:0/2350210832 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6c8007950 con 0x7fc6d806dc50 2019-12-05T06:10:53.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6d4ff9700 1 -- 172.21.15.16:0/2350210832 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6c8001010 con 0x7fc6d806dc50 2019-12-05T06:10:53.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.969+0000 7fc6de027700 1 -- 172.21.15.16:0/2350210832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d8133b80 con 0x7fc6d806dc50 2019-12-05T06:10:53.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.970+0000 7fc6de027700 1 -- 172.21.15.16:0/2350210832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d8133b80 con 0x7fc6d806dc50 2019-12-05T06:10:53.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.970+0000 7fc6d7fff700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.971+0000 7fc6d4ff9700 1 -- 172.21.15.16:0/2350210832 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc6c8001480 con 0x7fc6d806dc50 2019-12-05T06:10:53.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.971+0000 7fc6d7fff700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.972+0000 7fc6d4ff9700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6bc023630 0x7fc6bc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:53.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.972+0000 7fc6d4ff9700 1 -- 172.21.15.16:0/2350210832 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fc6c8046160 con 0x7fc6d806dc50 2019-12-05T06:10:53.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.972+0000 7fc6d77fe700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6bc023630 0x7fc6bc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:53.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.973+0000 7fc6d77fe700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6bc023630 0x7fc6bc025ac0 crc :-1 s=READY pgs=139 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:53.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.985+0000 7fc6de027700 1 -- 172.21.15.16:0/2350210832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d8133b80 con 0x7fc6d806dc50 2019-12-05T06:10:53.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.996+0000 7fc6d7fff700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:53.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:53.996+0000 7fc6d4ff9700 1 -- 172.21.15.16:0/2350210832 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc6c8001270 con 0x7fc6d806dc50 2019-12-05T06:10:54.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.233+0000 7fc6de027700 1 -- 172.21.15.16:0/2350210832 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc6d8133b80 con 0x7fc6d806dc50 2019-12-05T06:10:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.233+0000 7fc6d7fff700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.233+0000 7fc6d4ff9700 1 -- 172.21.15.16:0/2350210832 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fc6c8001270 con 0x7fc6d806dc50 2019-12-05T06:10:54.231 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2019-12-05T06:10:54.232 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:10:54.232 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:57.018632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":34,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:54.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.236+0000 7fc6de027700 1 -- 172.21.15.16:0/2350210832 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6bc023630 msgr2=0x7fc6bc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.236+0000 7fc6de027700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6bc023630 0x7fc6bc025ac0 crc :-1 s=READY pgs=139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.236+0000 7fc6de027700 1 -- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 msgr2=0x7fc6d806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.236+0000 7fc6de027700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 0x7fc6d806e040 secure :-1 s=READY pgs=282 cs=0 l=1 rx=0x7fc6c8010760 tx=0x7fc6c8010b00).stop 2019-12-05T06:10:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.236+0000 7fc6de027700 1 -- 172.21.15.16:0/2350210832 shutdown_connections 2019-12-05T06:10:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.236+0000 7fc6de027700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc6bc023630 0x7fc6bc025ac0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.236+0000 7fc6de027700 1 --2- 172.21.15.16:0/2350210832 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc6d806dc50 0x7fc6d806e040 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:54.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.236+0000 7fc6de027700 1 -- 172.21.15.16:0/2350210832 shutdown_connections 2019-12-05T06:10:54.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.236+0000 7fc6de027700 1 -- 172.21.15.16:0/2350210832 wait complete. 2019-12-05T06:10:54.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:54.236+0000 7fc6de027700 1 -- 172.21.15.16:0/2350210832 >> 172.21.15.16:0/2350210832 conn(0x7fc6d80a2850 msgr2=0x7fc6d806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:54.245 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:10:56.245 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.519+0000 7fb8825f4700 1 Processor -- start 2019-12-05T06:10:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.519+0000 7fb8825f4700 1 -- start start 2019-12-05T06:10:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.521+0000 7fb8825f4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 0x7fb87c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:56.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.521+0000 7fb8825f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c0958e0 con 0x7fb87c08d2c0 2019-12-05T06:10:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.521+0000 7fb8825f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c0629e0 con 0x7fb87c08c6d0 2019-12-05T06:10:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.521+0000 7fb8825f4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb87c08df90 con 0x7fb87c08fef0 2019-12-05T06:10:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.521+0000 7fb87bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb87c08d2c0 0x7fb87c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33356/0 (socket says 172.21.15.16:33356) 2019-12-05T06:10:56.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.521+0000 7fb87bfff700 1 -- 172.21.15.16:0/63153936 learned_addr learned my addr 172.21.15.16:0/63153936 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:10:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.521+0000 7fb881392700 1 --2- 172.21.15.16:0/63153936 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 0x7fb87c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.522+0000 7fb87b7fe700 1 -- 172.21.15.16:0/63153936 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411637452 0 0) 0x7fb86c001400 con 0x7fb87c08c6d0 2019-12-05T06:10:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.522+0000 7fb87b7fe700 1 -- 172.21.15.16:0/63153936 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb868002e80 con 0x7fb87c08c6d0 2019-12-05T06:10:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.522+0000 7fb87b7fe700 1 -- 172.21.15.16:0/63153936 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1715034395 0 0) 0x7fb874001150 con 0x7fb87c08d2c0 2019-12-05T06:10:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.522+0000 7fb881392700 1 -- 172.21.15.16:0/63153936 >> v1:172.21.15.76:6789/0 conn(0x7fb87c08c6d0 legacy=0x7fb87c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:56.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.522+0000 7fb881392700 1 -- 172.21.15.16:0/63153936 >> v1:172.21.15.16:6789/0 conn(0x7fb87c08d2c0 legacy=0x7fb87c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.522+0000 7fb881392700 1 -- 172.21.15.16:0/63153936 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c08e620 con 0x7fb87c08fef0 2019-12-05T06:10:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.522+0000 7fb881392700 1 --2- 172.21.15.16:0/63153936 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 0x7fb87c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb86400af80 tx=0x7fb86400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.523+0000 7fb881392700 1 --2- 172.21.15.16:0/63153936 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 0x7fb87c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb86400af80 tx=0x7fb86400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.523+0000 7fb881392700 1 --2- 172.21.15.16:0/63153936 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 0x7fb87c08f720 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7fb86400af80 tx=0x7fb86400b790).ready entity=mon.2 client_cookie=56cdfed780f5506e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.523+0000 7fb881392700 1 --2- 172.21.15.16:0/63153936 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 0x7fb87c08f720 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fb86400af80 tx=0x7fb86400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.523+0000 7fb87b7fe700 1 -- 172.21.15.16:0/63153936 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb864007240 con 0x7fb87c08fef0 2019-12-05T06:10:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.523+0000 7fb881392700 1 --2- 172.21.15.16:0/63153936 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 0x7fb87c08f720 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fb86400af80 tx=0x7fb86400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.523+0000 7fb881392700 1 --2- 172.21.15.16:0/63153936 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 0x7fb87c08f720 secure :-1 s=THROTTLE_DONE pgs=83 cs=0 l=1 rx=0x7fb86400af80 tx=0x7fb86400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.523+0000 7fb87b7fe700 1 -- 172.21.15.16:0/63153936 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb864000bb0 con 0x7fb87c08fef0 2019-12-05T06:10:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.523+0000 7fb87b7fe700 1 -- 172.21.15.16:0/63153936 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb864009e60 con 0x7fb87c08fef0 2019-12-05T06:10:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.523+0000 7fb8825f4700 1 -- 172.21.15.16:0/63153936 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 msgr2=0x7fb87c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.523+0000 7fb8825f4700 1 --2- 172.21.15.16:0/63153936 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 0x7fb87c08f720 secure :-1 s=READY pgs=83 cs=0 l=1 rx=0x7fb86400af80 tx=0x7fb86400b790).stop 2019-12-05T06:10:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.525+0000 7fb8825f4700 1 -- 172.21.15.16:0/63153936 shutdown_connections 2019-12-05T06:10:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.525+0000 7fb8825f4700 1 --2- 172.21.15.16:0/63153936 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb87c08fef0 0x7fb87c08f720 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:56.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.525+0000 7fb8825f4700 1 -- 172.21.15.16:0/63153936 shutdown_connections 2019-12-05T06:10:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.525+0000 7fb8825f4700 1 -- 172.21.15.16:0/63153936 wait complete. 2019-12-05T06:10:56.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.525+0000 7fb8825f4700 1 -- 172.21.15.16:0/63153936 >> 172.21.15.16:0/63153936 conn(0x7fb87c061940 msgr2=0x7fb87c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.531+0000 7fb8825f4700 1 Processor -- start 2019-12-05T06:10:56.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.531+0000 7fb8825f4700 1 -- start start 2019-12-05T06:10:56.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.533+0000 7fb8825f4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb87c1325e0 0x7fb87c12b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.533+0000 7fb8825f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c128b20 con 0x7fb87c12c740 2019-12-05T06:10:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.533+0000 7fb8825f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb87c128660 con 0x7fb87c154cb0 2019-12-05T06:10:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.533+0000 7fb8825f4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb87c12a7e0 con 0x7fb87c1325e0 2019-12-05T06:10:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.533+0000 7fb87bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb87c12c740 0x7fb87c12bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33364/0 (socket says 172.21.15.16:33364) 2019-12-05T06:10:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.533+0000 7fb87bfff700 1 -- 172.21.15.16:0/3340805134 learned_addr learned my addr 172.21.15.16:0/3340805134 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:10:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.533+0000 7fb880b91700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb87c1325e0 0x7fb87c12b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:56.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb8797fa700 1 -- 172.21.15.16:0/3340805134 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2634824452 0 0) 0x7fb864009990 con 0x7fb87c154cb0 2019-12-05T06:10:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb8797fa700 1 -- 172.21.15.16:0/3340805134 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb860002e80 con 0x7fb87c154cb0 2019-12-05T06:10:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb880b91700 1 -- 172.21.15.16:0/3340805134 >> v1:172.21.15.76:6789/0 conn(0x7fb87c154cb0 legacy=0x7fb87c1570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb8797fa700 1 -- 172.21.15.16:0/3340805134 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3293567309 0 0) 0x7fb874001150 con 0x7fb87c12c740 2019-12-05T06:10:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb880b91700 1 -- 172.21.15.16:0/3340805134 >> v1:172.21.15.16:6789/0 conn(0x7fb87c12c740 legacy=0x7fb87c12bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb880b91700 1 -- 172.21.15.16:0/3340805134 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb87c14a820 con 0x7fb87c1325e0 2019-12-05T06:10:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb880b91700 1 --2- 172.21.15.16:0/3340805134 >> v2:172.21.15.16:3300/0 conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb880b91700 1 --2- 172.21.15.16:0/3340805134 >> v2:172.21.15.16:3300/0 conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb880b91700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=READY pgs=285 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:56.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb880b91700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.534+0000 7fb8797fa700 1 -- 172.21.15.16:0/3340805134 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb86c00f370 con 0x7fb87c1325e0 2019-12-05T06:10:56.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.535+0000 7fb8825f4700 1 -- 172.21.15.16:0/3340805134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb87c12ae70 con 0x7fb87c1325e0 2019-12-05T06:10:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.535+0000 7fb880b91700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.535+0000 7fb8797fa700 1 -- 172.21.15.16:0/3340805134 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86c00f0d0 con 0x7fb87c1325e0 2019-12-05T06:10:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.535+0000 7fb880b91700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.535+0000 7fb8825f4700 1 -- 172.21.15.16:0/3340805134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87c14a820 con 0x7fb87c1325e0 2019-12-05T06:10:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.535+0000 7fb8797fa700 1 -- 172.21.15.16:0/3340805134 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb86c00f7c0 con 0x7fb87c1325e0 2019-12-05T06:10:56.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.536+0000 7fb880b91700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.537+0000 7fb8797fa700 1 -- 172.21.15.16:0/3340805134 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb86c00f550 con 0x7fb87c1325e0 2019-12-05T06:10:56.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.537+0000 7fb880b91700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.537+0000 7fb8797fa700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb860023680 0x7fb860025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:56.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.537+0000 7fb8797fa700 1 -- 172.21.15.16:0/3340805134 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7fb86c0461d0 con 0x7fb87c1325e0 2019-12-05T06:10:56.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.537+0000 7fb87bfff700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb860023680 0x7fb860025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:56.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.538+0000 7fb87bfff700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb860023680 0x7fb860025af0 crc :-1 s=READY pgs=140 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:56.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.550+0000 7fb8825f4700 1 -- 172.21.15.16:0/3340805134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb87c12ae70 con 0x7fb87c1325e0 2019-12-05T06:10:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.559+0000 7fb880b91700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.559+0000 7fb8797fa700 1 -- 172.21.15.16:0/3340805134 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb86c02cf80 con 0x7fb87c1325e0 2019-12-05T06:10:56.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.819+0000 7fb8825f4700 1 -- 172.21.15.16:0/3340805134 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb87c14a820 con 0x7fb87c1325e0 2019-12-05T06:10:56.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.820+0000 7fb880b91700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:56.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.820+0000 7fb8797fa700 1 -- 172.21.15.16:0/3340805134 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fb86c02cf80 con 0x7fb87c1325e0 2019-12-05T06:10:56.818 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2019-12-05T06:10:56.819 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:10:56.819 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:57.018632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":34,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.822+0000 7fb8825f4700 1 -- 172.21.15.16:0/3340805134 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb860023680 msgr2=0x7fb860025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.822+0000 7fb8825f4700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb860023680 0x7fb860025af0 crc :-1 s=READY pgs=140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.822+0000 7fb8825f4700 1 -- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 msgr2=0x7fb87c12b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.822+0000 7fb8825f4700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 0x7fb87c12b770 secure :-1 s=READY pgs=285 cs=0 l=1 rx=0x7fb86c010750 tx=0x7fb86c010af0).stop 2019-12-05T06:10:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.823+0000 7fb8825f4700 1 -- 172.21.15.16:0/3340805134 shutdown_connections 2019-12-05T06:10:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.823+0000 7fb8825f4700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb860023680 0x7fb860025af0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:56.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.823+0000 7fb8825f4700 1 --2- 172.21.15.16:0/3340805134 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb87c1325e0 0x7fb87c12b770 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.823+0000 7fb8825f4700 1 -- 172.21.15.16:0/3340805134 shutdown_connections 2019-12-05T06:10:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.823+0000 7fb8825f4700 1 -- 172.21.15.16:0/3340805134 wait complete. 2019-12-05T06:10:56.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:56.823+0000 7fb8825f4700 1 -- 172.21.15.16:0/3340805134 >> 172.21.15.16:0/3340805134 conn(0x7fb87c0a2850 msgr2=0x7fb87c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:56.831 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:10:58.761 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:10:58.766 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:10:58.832 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:10:59.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.043+0000 7f64f974b700 1 Processor -- start 2019-12-05T06:10:59.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.043+0000 7f64f974b700 1 -- start start 2019-12-05T06:10:59.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.044+0000 7f64f974b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f64f408c6d0 0x7f64f408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:59.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.044+0000 7f64f974b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64f408fef0 0x7f64f408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:59.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.044+0000 7f64f974b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f64f408ef20 0x7f64f408f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:10:59.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.044+0000 7f64f974b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f408df90 con 0x7f64f408fef0 2019-12-05T06:10:59.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.044+0000 7f64f974b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f40a3f70 con 0x7f64f408c6d0 2019-12-05T06:10:59.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.044+0000 7f64f974b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64f408d830 con 0x7f64f408ef20 2019-12-05T06:10:59.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.045+0000 7f64f2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64f408fef0 0x7f64f408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:59.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.045+0000 7f64f2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64f408fef0 0x7f64f408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48208/0 (socket says 172.21.15.16:48208) 2019-12-05T06:10:59.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.045+0000 7f64f2ffd700 1 -- 172.21.15.16:0/684834355 learned_addr learned my addr 172.21.15.16:0/684834355 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:10:59.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.045+0000 7f64f3fff700 1 --2- 172.21.15.16:0/684834355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f64f408ef20 0x7f64f408f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:59.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.045+0000 7f64f37fe700 1 --2- 172.21.15.16:0/684834355 >> v2:172.21.15.76:3300/0 conn(0x7f64f408c6d0 0x7f64f408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:59.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.045+0000 7f64f2ffd700 1 -- 172.21.15.16:0/684834355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f64f408ef20 msgr2=0x7f64f408f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:59.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.045+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/684834355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f64f408ef20 0x7f64f408f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:59.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.045+0000 7f64f2ffd700 1 -- 172.21.15.16:0/684834355 >> v2:172.21.15.76:3300/0 conn(0x7f64f408c6d0 msgr2=0x7f64f408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:10:59.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.045+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/684834355 >> v2:172.21.15.76:3300/0 conn(0x7f64f408c6d0 0x7f64f408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:59.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.045+0000 7f64f2ffd700 1 -- 172.21.15.16:0/684834355 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f408d540 con 0x7f64f408fef0 2019-12-05T06:10:59.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/684834355 >> v2:172.21.15.16:3300/0 conn(0x7f64f408fef0 0x7f64f408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f64dc00fac0 tx=0x7f64dc00f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/684834355 >> v2:172.21.15.16:3300/0 conn(0x7f64f408fef0 0x7f64f408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f64dc00fac0 tx=0x7f64dc00f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/684834355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f408fef0 0x7f64f408d000 secure :-1 s=READY pgs=288 cs=0 l=1 rx=0x7f64dc00fac0 tx=0x7f64dc00f970).ready entity=mon.0 client_cookie=401c918926bc74f6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:59.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/684834355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f408fef0 0x7f64f408d000 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7f64dc00fac0 tx=0x7f64dc00f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/684834355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f408fef0 0x7f64f408d000 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7f64dc00fac0 tx=0x7f64dc00f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/684834355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f408fef0 0x7f64f408d000 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7f64dc00fac0 tx=0x7f64dc00f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f27fc700 1 -- 172.21.15.16:0/684834355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64dc000ce0 con 0x7f64f408fef0 2019-12-05T06:10:59.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f27fc700 1 -- 172.21.15.16:0/684834355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc001010 con 0x7f64f408fef0 2019-12-05T06:10:59.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f27fc700 1 -- 172.21.15.16:0/684834355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64dc001720 con 0x7f64f408fef0 2019-12-05T06:10:59.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f974b700 1 -- 172.21.15.16:0/684834355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f408fef0 msgr2=0x7f64f408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:59.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.046+0000 7f64f974b700 1 --2- 172.21.15.16:0/684834355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f408fef0 0x7f64f408d000 secure :-1 s=READY pgs=288 cs=0 l=1 rx=0x7f64dc00fac0 tx=0x7f64dc00f970).stop 2019-12-05T06:10:59.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.048+0000 7f64f974b700 1 -- 172.21.15.16:0/684834355 shutdown_connections 2019-12-05T06:10:59.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.048+0000 7f64f974b700 1 --2- 172.21.15.16:0/684834355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f64f408ef20 0x7f64f408f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:59.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.048+0000 7f64f974b700 1 --2- 172.21.15.16:0/684834355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f408fef0 0x7f64f408d000 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:59.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.048+0000 7f64f974b700 1 --2- 172.21.15.16:0/684834355 >> v2:172.21.15.76:3300/0 conn(0x7f64f408c6d0 0x7f64f408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:10:59.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.048+0000 7f64f974b700 1 -- 172.21.15.16:0/684834355 shutdown_connections 2019-12-05T06:10:59.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.048+0000 7f64f974b700 1 -- 172.21.15.16:0/684834355 wait complete. 2019-12-05T06:10:59.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.048+0000 7f64f974b700 1 -- 172.21.15.16:0/684834355 >> 172.21.15.16:0/684834355 conn(0x7f64f4061940 msgr2=0x7f64f4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:59.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.052+0000 7f64f974b700 1 Processor -- start 2019-12-05T06:10:59.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.053+0000 7f64f974b700 1 -- start start 2019-12-05T06:10:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.054+0000 7f64f974b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64f40992d0 0x7f64f4098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.054+0000 7f64f974b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f4089fb0 con 0x7f64f406df10 2019-12-05T06:10:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.054+0000 7f64f974b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64f406e300 con 0x7f64f4156cb0 2019-12-05T06:10:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.054+0000 7f64f974b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64f4097370 con 0x7f64f40992d0 2019-12-05T06:10:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.054+0000 7f64f2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64f40992d0 0x7f64f4098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:59.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.054+0000 7f64f37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f64f406df10 0x7f64f4098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33384/0 (socket says 172.21.15.16:33384) 2019-12-05T06:10:59.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.054+0000 7f64f2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f64f40992d0 0x7f64f4098b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48214/0 (socket says 172.21.15.16:48214) 2019-12-05T06:10:59.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.054+0000 7f64f37fe700 1 -- 172.21.15.16:0/1416383297 learned_addr learned my addr 172.21.15.16:0/1416383297 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:10:59.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64ebfff700 1 -- 172.21.15.16:0/1416383297 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192757183 0 0) 0x7f64ec001080 con 0x7f64f4156cb0 2019-12-05T06:10:59.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64f2ffd700 1 -- 172.21.15.16:0/1416383297 >> v1:172.21.15.76:6789/0 conn(0x7f64f4156cb0 legacy=0x7f64f41590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:59.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64f2ffd700 1 -- 172.21.15.16:0/1416383297 >> v1:172.21.15.16:6789/0 conn(0x7f64f406df10 legacy=0x7f64f4098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:59.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64f2ffd700 1 -- 172.21.15.16:0/1416383297 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64f4097a00 con 0x7f64f40992d0 2019-12-05T06:10:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/1416383297 >> v2:172.21.15.16:3300/0 conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/1416383297 >> v2:172.21.15.16:3300/0 conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=READY pgs=290 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=THROTTLE_DONE pgs=290 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=THROTTLE_DONE pgs=290 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64ebfff700 1 -- 172.21.15.16:0/1416383297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64dc005710 con 0x7f64f40992d0 2019-12-05T06:10:59.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=THROTTLE_DONE pgs=290 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64ebfff700 1 -- 172.21.15.16:0/1416383297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64dc0080c0 con 0x7f64f40992d0 2019-12-05T06:10:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64ebfff700 1 -- 172.21.15.16:0/1416383297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64dc005cf0 con 0x7f64f40992d0 2019-12-05T06:10:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.055+0000 7f64f974b700 1 -- 172.21.15.16:0/1416383297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64f4097a00 con 0x7f64f40992d0 2019-12-05T06:10:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.056+0000 7f64f974b700 1 -- 172.21.15.16:0/1416383297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64f4097a00 con 0x7f64f40992d0 2019-12-05T06:10:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.056+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=THROTTLE_DONE pgs=290 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.057+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=THROTTLE_DONE pgs=290 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.057+0000 7f64ebfff700 1 -- 172.21.15.16:0/1416383297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f64dc005710 con 0x7f64f40992d0 2019-12-05T06:10:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.057+0000 7f64ebfff700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64d8023630 0x7f64d8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:10:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.057+0000 7f64ebfff700 1 -- 172.21.15.16:0/1416383297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f64dc08a140 con 0x7f64f40992d0 2019-12-05T06:10:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.058+0000 7f64f37fe700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64d8023630 0x7f64d8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:10:59.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.058+0000 7f64f974b700 1 -- 172.21.15.16:0/1416383297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64f4097a00 con 0x7f64f40992d0 2019-12-05T06:10:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.058+0000 7f64f37fe700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64d8023630 0x7f64d8025ac0 crc :-1 s=READY pgs=141 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:10:59.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.062+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=THROTTLE_DONE pgs=290 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.062+0000 7f64ebfff700 1 -- 172.21.15.16:0/1416383297 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f64dc005500 con 0x7f64f40992d0 2019-12-05T06:10:59.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.263+0000 7f64f974b700 1 -- 172.21.15.16:0/1416383297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64f4097a00 con 0x7f64f40992d0 2019-12-05T06:10:59.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.264+0000 7f64f2ffd700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=THROTTLE_DONE pgs=290 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:10:59.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.264+0000 7f64ebfff700 1 -- 172.21.15.16:0/1416383297 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f64dc005710 con 0x7f64f40992d0 2019-12-05T06:10:59.262 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2019-12-05T06:10:59.262 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:10:59.262 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:57.018632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":34,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:10:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.266+0000 7f64f974b700 1 -- 172.21.15.16:0/1416383297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64d8023630 msgr2=0x7f64d8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.266+0000 7f64f974b700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64d8023630 0x7f64d8025ac0 crc :-1 s=READY pgs=141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.266+0000 7f64f974b700 1 -- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 msgr2=0x7f64f4098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:10:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.266+0000 7f64f974b700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 0x7f64f4098b00 secure :-1 s=READY pgs=290 cs=0 l=1 rx=0x7f64dc007f40 tx=0x7f64dc054b00).stop 2019-12-05T06:10:59.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.266+0000 7f64f974b700 1 -- 172.21.15.16:0/1416383297 shutdown_connections 2019-12-05T06:10:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.266+0000 7f64f974b700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64d8023630 0x7f64d8025ac0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.266+0000 7f64f974b700 1 --2- 172.21.15.16:0/1416383297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f64f40992d0 0x7f64f4098b00 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:10:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.267+0000 7f64f974b700 1 -- 172.21.15.16:0/1416383297 shutdown_connections 2019-12-05T06:10:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.267+0000 7f64f974b700 1 -- 172.21.15.16:0/1416383297 wait complete. 2019-12-05T06:10:59.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:10:59.267+0000 7f64f974b700 1 -- 172.21.15.16:0/1416383297 >> 172.21.15.16:0/1416383297 conn(0x7f64f40714f0 msgr2=0x7f64f409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:10:59.275 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:11:01.276 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.565+0000 7f9850961700 1 Processor -- start 2019-12-05T06:11:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.565+0000 7f9850961700 1 -- start start 2019-12-05T06:11:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f9850961700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f984c08c6d0 0x7f984c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f9850961700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c08fef0 0x7f984c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:01.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f9850961700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f984c08ef20 0x7f984c08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f9850961700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984c08df90 con 0x7f984c08c6d0 2019-12-05T06:11:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f9850961700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984c0a3f70 con 0x7f984c08ef20 2019-12-05T06:11:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f9850961700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984c08d830 con 0x7f984c08fef0 2019-12-05T06:11:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f984ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f984c08c6d0 0x7f984c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f984ad9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f984c08c6d0 0x7f984c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48226/0 (socket says 172.21.15.16:48226) 2019-12-05T06:11:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f984ad9d700 1 -- 172.21.15.16:0/1666718027 learned_addr learned my addr 172.21.15.16:0/1666718027 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:11:01.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f984b59e700 1 --2- 172.21.15.16:0/1666718027 >> v2:172.21.15.76:3300/0 conn(0x7f984c08ef20 0x7f984c08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.567+0000 7f984a59c700 1 --2- 172.21.15.16:0/1666718027 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c08fef0 0x7f984c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.568+0000 7f984ad9d700 1 -- 172.21.15.16:0/1666718027 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c08fef0 msgr2=0x7f984c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.568+0000 7f984ad9d700 1 --2- 172.21.15.16:0/1666718027 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c08fef0 0x7f984c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:01.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.568+0000 7f984ad9d700 1 -- 172.21.15.16:0/1666718027 >> v2:172.21.15.76:3300/0 conn(0x7f984c08ef20 msgr2=0x7f984c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:01.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.568+0000 7f984ad9d700 1 --2- 172.21.15.16:0/1666718027 >> v2:172.21.15.76:3300/0 conn(0x7f984c08ef20 0x7f984c08f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.568+0000 7f984ad9d700 1 -- 172.21.15.16:0/1666718027 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c08d540 con 0x7f984c08c6d0 2019-12-05T06:11:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.568+0000 7f984ad9d700 1 --2- 172.21.15.16:0/1666718027 >> v2:172.21.15.16:3300/0 conn(0x7f984c08c6d0 0x7f984c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f983c00f6c0 tx=0x7f983c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.568+0000 7f984ad9d700 1 --2- 172.21.15.16:0/1666718027 >> v2:172.21.15.16:3300/0 conn(0x7f984c08c6d0 0x7f984c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f983c00f6c0 tx=0x7f983c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.568+0000 7f984ad9d700 1 --2- 172.21.15.16:0/1666718027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984c08c6d0 0x7f984c08cac0 secure :-1 s=READY pgs=291 cs=0 l=1 rx=0x7f983c00f6c0 tx=0x7f983c00f570).ready entity=mon.0 client_cookie=f4ca8fafea18661b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.569+0000 7f984ad9d700 1 --2- 172.21.15.16:0/1666718027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984c08c6d0 0x7f984c08cac0 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7f983c00f6c0 tx=0x7f983c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.569+0000 7f9849d9b700 1 -- 172.21.15.16:0/1666718027 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f983c000960 con 0x7f984c08c6d0 2019-12-05T06:11:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.569+0000 7f984ad9d700 1 --2- 172.21.15.16:0/1666718027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984c08c6d0 0x7f984c08cac0 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7f983c00f6c0 tx=0x7f983c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.569+0000 7f984ad9d700 1 --2- 172.21.15.16:0/1666718027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984c08c6d0 0x7f984c08cac0 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7f983c00f6c0 tx=0x7f983c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.569+0000 7f9849d9b700 1 -- 172.21.15.16:0/1666718027 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f983c000c90 con 0x7f984c08c6d0 2019-12-05T06:11:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.569+0000 7f9849d9b700 1 -- 172.21.15.16:0/1666718027 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f983c0013a0 con 0x7f984c08c6d0 2019-12-05T06:11:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.569+0000 7f9850961700 1 -- 172.21.15.16:0/1666718027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984c08c6d0 msgr2=0x7f984c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.569+0000 7f9850961700 1 --2- 172.21.15.16:0/1666718027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984c08c6d0 0x7f984c08cac0 secure :-1 s=READY pgs=291 cs=0 l=1 rx=0x7f983c00f6c0 tx=0x7f983c00f570).stop 2019-12-05T06:11:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.571+0000 7f9850961700 1 -- 172.21.15.16:0/1666718027 shutdown_connections 2019-12-05T06:11:01.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.571+0000 7f9850961700 1 --2- 172.21.15.16:0/1666718027 >> v2:172.21.15.76:3300/0 conn(0x7f984c08ef20 0x7f984c08f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.571+0000 7f9850961700 1 --2- 172.21.15.16:0/1666718027 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c08fef0 0x7f984c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.571+0000 7f9850961700 1 --2- 172.21.15.16:0/1666718027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f984c08c6d0 0x7f984c08cac0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.571+0000 7f9850961700 1 -- 172.21.15.16:0/1666718027 shutdown_connections 2019-12-05T06:11:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.571+0000 7f9850961700 1 -- 172.21.15.16:0/1666718027 wait complete. 2019-12-05T06:11:01.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.571+0000 7f9850961700 1 -- 172.21.15.16:0/1666718027 >> 172.21.15.16:0/1666718027 conn(0x7f984c061940 msgr2=0x7f984c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:01.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.576+0000 7f9850961700 1 Processor -- start 2019-12-05T06:11:01.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.577+0000 7f9850961700 1 -- start start 2019-12-05T06:11:01.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.578+0000 7f9850961700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f984c132d00 0x7f984c132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:01.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.578+0000 7f9850961700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c154b00 0x7f984c156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:01.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.578+0000 7f9850961700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f984c098da0 con 0x7f984c09b850 2019-12-05T06:11:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.578+0000 7f9850961700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f984c09aa60 con 0x7f984c132d00 2019-12-05T06:11:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.578+0000 7f9850961700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f984c09a030 con 0x7f984c154b00 2019-12-05T06:11:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.579+0000 7f984a59c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f984c132d00 0x7f984c132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.579+0000 7f984ad9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f984c09b850 0x7f984c131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:54702/0 (socket says 172.21.15.16:54702) 2019-12-05T06:11:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.579+0000 7f984ad9d700 1 -- 172.21.15.16:0/619322369 learned_addr learned my addr 172.21.15.16:0/619322369 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:11:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.579+0000 7f984b59e700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c154b00 0x7f984c156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:01.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.579+0000 7f984a59c700 1 -- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c154b00 msgr2=0x7f984c156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.579+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c154b00 0x7f984c156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.579+0000 7f98437fe700 1 -- 172.21.15.16:0/619322369 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 446964891 0 0) 0x7f983c007910 con 0x7f984c09b850 2019-12-05T06:11:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.579+0000 7f984a59c700 1 -- 172.21.15.16:0/619322369 >> v1:172.21.15.76:6789/0 conn(0x7f984c09b850 legacy=0x7f984c131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.579+0000 7f984a59c700 1 -- 172.21.15.16:0/619322369 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f984c09b0f0 con 0x7f984c132d00 2019-12-05T06:11:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.579+0000 7f984b59e700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c154b00 0x7f984c156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:11:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> v2:172.21.15.76:3300/0 conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> v2:172.21.15.76:3300/0 conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=READY pgs=237 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:01.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f98437fe700 1 -- 172.21.15.16:0/619322369 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9834000940 con 0x7f984c132d00 2019-12-05T06:11:01.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f98437fe700 1 -- 172.21.15.16:0/619322369 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9834000940 con 0x7f984c132d00 2019-12-05T06:11:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f9850961700 1 -- 172.21.15.16:0/619322369 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f984c09b0f0 con 0x7f984c132d00 2019-12-05T06:11:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f98437fe700 1 -- 172.21.15.16:0/619322369 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9834005810 con 0x7f984c132d00 2019-12-05T06:11:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.580+0000 7f9850961700 1 -- 172.21.15.16:0/619322369 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f984c09b0f0 con 0x7f984c132d00 2019-12-05T06:11:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.581+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.582+0000 7f98437fe700 1 -- 172.21.15.16:0/619322369 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9834005390 con 0x7f984c132d00 2019-12-05T06:11:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.582+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.583+0000 7f98437fe700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9830023630 0x7f9830025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.583+0000 7f98437fe700 1 -- 172.21.15.16:0/619322369 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 6145+0+0 (secure 0 0 0) 0x7f9834045d30 con 0x7f984c132d00 2019-12-05T06:11:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.583+0000 7f984ad9d700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9830023630 0x7f9830025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:01.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.584+0000 7f984ad9d700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9830023630 0x7f9830025ac0 crc :-1 s=READY pgs=142 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:01.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.596+0000 7f9850961700 1 -- 172.21.15.16:0/619322369 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f984c09b0f0 con 0x7f984c132d00 2019-12-05T06:11:01.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.605+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.605+0000 7f98437fe700 1 -- 172.21.15.16:0/619322369 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9834005180 con 0x7f984c132d00 2019-12-05T06:11:01.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.860+0000 7f9850961700 1 -- 172.21.15.16:0/619322369 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f984c09b0f0 con 0x7f984c132d00 2019-12-05T06:11:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.861+0000 7f984a59c700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:01.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.861+0000 7f98437fe700 1 -- 172.21.15.16:0/619322369 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f9834005c40 con 0x7f984c132d00 2019-12-05T06:11:01.862 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:11:01.862 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4839,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":23}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:08:57.018632+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":27,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4229},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4229":{"gid":4229,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":34,"addr":"172.21.15.76:6835/3164208461","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3164208461},{"type":"v1","addr":"172.21.15.76:6835","nonce":3164208461}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:01.862 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 23 2019-12-05T06:11:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.865+0000 7f9850961700 1 -- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9830023630 msgr2=0x7f9830025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.865+0000 7f9850961700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9830023630 0x7f9830025ac0 crc :-1 s=READY pgs=142 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:01.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.865+0000 7f9850961700 1 -- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 msgr2=0x7f984c132530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.865+0000 7f9850961700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 0x7f984c132530 secure :-1 s=READY pgs=237 cs=0 l=1 rx=0x7f98340103b0 tx=0x7f9834010750).stop 2019-12-05T06:11:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.866+0000 7f9850961700 1 -- 172.21.15.16:0/619322369 shutdown_connections 2019-12-05T06:11:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.866+0000 7f9850961700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9830023630 0x7f9830025ac0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.866+0000 7f9850961700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f984c154b00 0x7f984c156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.866+0000 7f9850961700 1 --2- 172.21.15.16:0/619322369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f984c132d00 0x7f984c132530 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.866+0000 7f9850961700 1 -- 172.21.15.16:0/619322369 shutdown_connections 2019-12-05T06:11:01.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.866+0000 7f9850961700 1 -- 172.21.15.16:0/619322369 wait complete. 2019-12-05T06:11:01.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:01.866+0000 7f9850961700 1 -- 172.21.15.16:0/619322369 >> 172.21.15.16:0/619322369 conn(0x7f984c0714f0 msgr2=0x7f984c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:01.874 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:11:03.875 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.078+0000 7fb67ad92700 1 Processor -- start 2019-12-05T06:11:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.079+0000 7fb67ad92700 1 -- start start 2019-12-05T06:11:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb67ad92700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67408c6d0 0x7fb67408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb67ad92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb67408fef0 0x7fb67408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb67ad92700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb67408ef20 0x7fb67408f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb67ad92700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6740a3f70 con 0x7fb67408fef0 2019-12-05T06:11:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb67ad92700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67408d830 con 0x7fb67408ef20 2019-12-05T06:11:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb67ad92700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb67408d540 con 0x7fb67408c6d0 2019-12-05T06:11:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb678b2e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb67408fef0 0x7fb67408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb678b2e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb67408fef0 0x7fb67408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48260/0 (socket says 172.21.15.16:48260) 2019-12-05T06:11:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb678b2e700 1 -- 172.21.15.16:0/894293019 learned_addr learned my addr 172.21.15.16:0/894293019 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:11:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb67932f700 1 --2- 172.21.15.16:0/894293019 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67408c6d0 0x7fb67408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.080+0000 7fb679b30700 1 --2- 172.21.15.16:0/894293019 >> v2:172.21.15.76:3300/0 conn(0x7fb67408ef20 0x7fb67408f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 -- 172.21.15.16:0/894293019 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67408c6d0 msgr2=0x7fb67408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 --2- 172.21.15.16:0/894293019 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67408c6d0 0x7fb67408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 -- 172.21.15.16:0/894293019 >> v2:172.21.15.76:3300/0 conn(0x7fb67408ef20 msgr2=0x7fb67408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:04.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 --2- 172.21.15.16:0/894293019 >> v2:172.21.15.76:3300/0 conn(0x7fb67408ef20 0x7fb67408f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:04.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 -- 172.21.15.16:0/894293019 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67408e620 con 0x7fb67408fef0 2019-12-05T06:11:04.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 --2- 172.21.15.16:0/894293019 >> v2:172.21.15.16:3300/0 conn(0x7fb67408fef0 0x7fb67408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb67000f6c0 tx=0x7fb67000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 --2- 172.21.15.16:0/894293019 >> v2:172.21.15.16:3300/0 conn(0x7fb67408fef0 0x7fb67408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb67000f6c0 tx=0x7fb67000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 --2- 172.21.15.16:0/894293019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb67408fef0 0x7fb67408d000 secure :-1 s=READY pgs=292 cs=0 l=1 rx=0x7fb67000f6c0 tx=0x7fb67000f570).ready entity=mon.0 client_cookie=bab8d4700e258518 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:04.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 --2- 172.21.15.16:0/894293019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb67408fef0 0x7fb67408d000 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7fb67000f6c0 tx=0x7fb67000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 --2- 172.21.15.16:0/894293019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb67408fef0 0x7fb67408d000 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7fb67000f6c0 tx=0x7fb67000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb678b2e700 1 --2- 172.21.15.16:0/894293019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb67408fef0 0x7fb67408d000 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7fb67000f6c0 tx=0x7fb67000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb66bfff700 1 -- 172.21.15.16:0/894293019 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb670000960 con 0x7fb67408fef0 2019-12-05T06:11:04.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb66bfff700 1 -- 172.21.15.16:0/894293019 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb670000c90 con 0x7fb67408fef0 2019-12-05T06:11:04.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb66bfff700 1 -- 172.21.15.16:0/894293019 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6700013a0 con 0x7fb67408fef0 2019-12-05T06:11:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb67ad92700 1 -- 172.21.15.16:0/894293019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb67408fef0 msgr2=0x7fb67408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.081+0000 7fb67ad92700 1 --2- 172.21.15.16:0/894293019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb67408fef0 0x7fb67408d000 secure :-1 s=READY pgs=292 cs=0 l=1 rx=0x7fb67000f6c0 tx=0x7fb67000f570).stop 2019-12-05T06:11:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.082+0000 7fb67ad92700 1 -- 172.21.15.16:0/894293019 shutdown_connections 2019-12-05T06:11:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.082+0000 7fb67ad92700 1 --2- 172.21.15.16:0/894293019 >> v2:172.21.15.76:3300/0 conn(0x7fb67408ef20 0x7fb67408f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.082+0000 7fb67ad92700 1 --2- 172.21.15.16:0/894293019 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb67408fef0 0x7fb67408d000 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.082+0000 7fb67ad92700 1 --2- 172.21.15.16:0/894293019 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67408c6d0 0x7fb67408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.083+0000 7fb67ad92700 1 -- 172.21.15.16:0/894293019 shutdown_connections 2019-12-05T06:11:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.083+0000 7fb67ad92700 1 -- 172.21.15.16:0/894293019 wait complete. 2019-12-05T06:11:04.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.083+0000 7fb67ad92700 1 -- 172.21.15.16:0/894293019 >> 172.21.15.16:0/894293019 conn(0x7fb674061940 msgr2=0x7fb674096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.086+0000 7fb67ad92700 1 Processor -- start 2019-12-05T06:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.087+0000 7fb67ad92700 1 -- start start 2019-12-05T06:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb67ad92700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb674096e90 0x7fb67409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb67ad92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb674084f30 0x7fb67409b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb67ad92700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb674084b90 con 0x7fb674097280 2019-12-05T06:11:04.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb67ad92700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67409a450 con 0x7fb674084f30 2019-12-05T06:11:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb67ad92700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb674099a20 con 0x7fb674096e90 2019-12-05T06:11:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb678b2e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb674084f30 0x7fb67409b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb678b2e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb674084f30 0x7fb67409b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48268/0 (socket says 172.21.15.16:48268) 2019-12-05T06:11:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb678b2e700 1 -- 172.21.15.16:0/434244292 learned_addr learned my addr 172.21.15.16:0/434244292 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:11:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb67932f700 1 --2- 172.21.15.16:0/434244292 >> v2:172.21.15.76:3300/0 conn(0x7fb674096e90 0x7fb67409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:04.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb678b2e700 1 -- 172.21.15.16:0/434244292 >> v2:172.21.15.76:3300/0 conn(0x7fb674096e90 msgr2=0x7fb67409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> v2:172.21.15.76:3300/0 conn(0x7fb674096e90 0x7fb67409b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.088+0000 7fb678b2e700 1 -- 172.21.15.16:0/434244292 >> v1:172.21.15.76:6789/0 conn(0x7fb674097280 legacy=0x7fb6740976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb678b2e700 1 -- 172.21.15.16:0/434244292 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67409aae0 con 0x7fb674084f30 2019-12-05T06:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb67932f700 1 --2- 172.21.15.16:0/434244292 >> v2:172.21.15.76:3300/0 conn(0x7fb674096e90 0x7fb67409b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:11:04.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> v2:172.21.15.16:3300/0 conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> v2:172.21.15.16:3300/0 conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=READY pgs=293 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb669ffb700 1 -- 172.21.15.16:0/434244292 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb670005180 con 0x7fb674084f30 2019-12-05T06:11:04.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb669ffb700 1 -- 172.21.15.16:0/434244292 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb670007d40 con 0x7fb674084f30 2019-12-05T06:11:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb67ad92700 1 -- 172.21.15.16:0/434244292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67409aae0 con 0x7fb674084f30 2019-12-05T06:11:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb669ffb700 1 -- 172.21.15.16:0/434244292 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb670005760 con 0x7fb674084f30 2019-12-05T06:11:04.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.089+0000 7fb67ad92700 1 -- 172.21.15.16:0/434244292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67409aae0 con 0x7fb674084f30 2019-12-05T06:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.090+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.090+0000 7fb669ffb700 1 -- 172.21.15.16:0/434244292 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb670005180 con 0x7fb674084f30 2019-12-05T06:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.090+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.090+0000 7fb669ffb700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb654023630 0x7fb654025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.090+0000 7fb669ffb700 1 -- 172.21.15.16:0/434244292 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7fb67008a0a0 con 0x7fb674084f30 2019-12-05T06:11:04.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.091+0000 7fb67932f700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb654023630 0x7fb654025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.091+0000 7fb67932f700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb654023630 0x7fb654025ae0 crc :-1 s=READY pgs=143 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.094+0000 7fb67ad92700 1 -- 172.21.15.16:0/434244292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67409aae0 con 0x7fb674084f30 2019-12-05T06:11:04.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.098+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.098+0000 7fb669ffb700 1 -- 172.21.15.16:0/434244292 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb670000940 con 0x7fb674084f30 2019-12-05T06:11:04.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.316+0000 7fb67ad92700 1 -- 172.21.15.16:0/434244292 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb67409aae0 con 0x7fb674084f30 2019-12-05T06:11:04.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.317+0000 7fb678b2e700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.317+0000 7fb669ffb700 1 -- 172.21.15.16:0/434244292 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fb670007cd0 con 0x7fb674084f30 2019-12-05T06:11:04.315 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 24 2019-12-05T06:11:04.315 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:11:04.315 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:02.969673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:04.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.319+0000 7fb67ad92700 1 -- 172.21.15.16:0/434244292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb654023630 msgr2=0x7fb654025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.319+0000 7fb67ad92700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb654023630 0x7fb654025ae0 crc :-1 s=READY pgs=143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:04.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.320+0000 7fb67ad92700 1 -- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 msgr2=0x7fb67409b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.320+0000 7fb67ad92700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 0x7fb67409b920 secure :-1 s=READY pgs=293 cs=0 l=1 rx=0x7fb670054470 tx=0x7fb670054810).stop 2019-12-05T06:11:04.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.320+0000 7fb67ad92700 1 -- 172.21.15.16:0/434244292 shutdown_connections 2019-12-05T06:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.320+0000 7fb67ad92700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb654023630 0x7fb654025ae0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.320+0000 7fb67ad92700 1 --2- 172.21.15.16:0/434244292 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb674084f30 0x7fb67409b920 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.320+0000 7fb67ad92700 1 --2- 172.21.15.16:0/434244292 >> v2:172.21.15.76:3300/0 conn(0x7fb674096e90 0x7fb67409b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.320+0000 7fb67ad92700 1 -- 172.21.15.16:0/434244292 shutdown_connections 2019-12-05T06:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.320+0000 7fb67ad92700 1 -- 172.21.15.16:0/434244292 wait complete. 2019-12-05T06:11:04.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.320+0000 7fb67ad92700 1 -- 172.21.15.16:0/434244292 >> 172.21.15.16:0/434244292 conn(0x7fb6740714f0 msgr2=0x7fb67409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:04.327 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:11:04.328 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:11:04.329 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:04.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.509+0000 7f6afae97700 1 Processor -- start 2019-12-05T06:11:04.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.509+0000 7f6afae97700 1 -- start start 2019-12-05T06:11:04.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.511+0000 7f6afae97700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6af408d2c0 0x7f6af408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:04.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.511+0000 7f6afae97700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af40958e0 con 0x7f6af408c6d0 2019-12-05T06:11:04.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.511+0000 7f6afae97700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af40629e0 con 0x7f6af408fef0 2019-12-05T06:11:04.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.511+0000 7f6afae97700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af408df90 con 0x7f6af408d2c0 2019-12-05T06:11:04.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.511+0000 7f6af9434700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6af408c6d0 0x7f6af408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33446/0 (socket says 172.21.15.16:33446) 2019-12-05T06:11:04.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.511+0000 7f6af9434700 1 -- 172.21.15.16:0/4263048785 learned_addr learned my addr 172.21.15.16:0/4263048785 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:11:04.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.511+0000 7f6af8c33700 1 --2- 172.21.15.16:0/4263048785 >> v2:172.21.15.76:3300/0 conn(0x7f6af408d2c0 0x7f6af408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:04.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.511+0000 7f6af3fff700 1 -- 172.21.15.16:0/4263048785 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1830147457 0 0) 0x7f6ae4001150 con 0x7f6af408c6d0 2019-12-05T06:11:04.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.512+0000 7f6af3fff700 1 -- 172.21.15.16:0/4263048785 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6adc002e80 con 0x7f6af408c6d0 2019-12-05T06:11:04.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.512+0000 7f6af3fff700 1 -- 172.21.15.16:0/4263048785 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3873933117 0 0) 0x7f6aec000d30 con 0x7f6af408fef0 2019-12-05T06:11:04.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.512+0000 7f6af8c33700 1 -- 172.21.15.16:0/4263048785 >> v1:172.21.15.76:6789/0 conn(0x7f6af408fef0 legacy=0x7f6af408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.512+0000 7f6af8c33700 1 -- 172.21.15.16:0/4263048785 >> v1:172.21.15.16:6789/0 conn(0x7f6af408c6d0 legacy=0x7f6af408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.512+0000 7f6af8c33700 1 -- 172.21.15.16:0/4263048785 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af408e620 con 0x7f6af408d2c0 2019-12-05T06:11:04.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.512+0000 7f6af8c33700 1 --2- 172.21.15.16:0/4263048785 >> v2:172.21.15.76:3300/0 conn(0x7f6af408d2c0 0x7f6af408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6ae800af50 tx=0x7f6ae800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.512+0000 7f6af8c33700 1 --2- 172.21.15.16:0/4263048785 >> v2:172.21.15.76:3300/0 conn(0x7f6af408d2c0 0x7f6af408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6ae800af50 tx=0x7f6ae800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.513+0000 7f6af8c33700 1 --2- 172.21.15.16:0/4263048785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af408d2c0 0x7f6af408ef20 secure :-1 s=READY pgs=240 cs=0 l=1 rx=0x7f6ae800af50 tx=0x7f6ae800b760).ready entity=mon.1 client_cookie=ab6550b063966185 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.513+0000 7f6af8c33700 1 --2- 172.21.15.16:0/4263048785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af408d2c0 0x7f6af408ef20 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7f6ae800af50 tx=0x7f6ae800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.513+0000 7f6af3fff700 1 -- 172.21.15.16:0/4263048785 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae8007210 con 0x7f6af408d2c0 2019-12-05T06:11:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.513+0000 7f6af8c33700 1 --2- 172.21.15.16:0/4263048785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af408d2c0 0x7f6af408ef20 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7f6ae800af50 tx=0x7f6ae800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.513+0000 7f6af8c33700 1 --2- 172.21.15.16:0/4263048785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af408d2c0 0x7f6af408ef20 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7f6ae800af50 tx=0x7f6ae800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.513+0000 7f6af3fff700 1 -- 172.21.15.16:0/4263048785 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae80099b0 con 0x7f6af408d2c0 2019-12-05T06:11:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.513+0000 7f6af3fff700 1 -- 172.21.15.16:0/4263048785 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae800a080 con 0x7f6af408d2c0 2019-12-05T06:11:04.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.513+0000 7f6afae97700 1 -- 172.21.15.16:0/4263048785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af408d2c0 msgr2=0x7f6af408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.513+0000 7f6afae97700 1 --2- 172.21.15.16:0/4263048785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af408d2c0 0x7f6af408ef20 secure :-1 s=READY pgs=240 cs=0 l=1 rx=0x7f6ae800af50 tx=0x7f6ae800b760).stop 2019-12-05T06:11:04.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.514+0000 7f6afae97700 1 -- 172.21.15.16:0/4263048785 shutdown_connections 2019-12-05T06:11:04.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.514+0000 7f6afae97700 1 --2- 172.21.15.16:0/4263048785 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af408d2c0 0x7f6af408ef20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:04.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.515+0000 7f6afae97700 1 -- 172.21.15.16:0/4263048785 shutdown_connections 2019-12-05T06:11:04.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.515+0000 7f6afae97700 1 -- 172.21.15.16:0/4263048785 wait complete. 2019-12-05T06:11:04.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.515+0000 7f6afae97700 1 -- 172.21.15.16:0/4263048785 >> 172.21.15.16:0/4263048785 conn(0x7f6af4061940 msgr2=0x7f6af4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:04.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.519+0000 7f6afae97700 1 Processor -- start 2019-12-05T06:11:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.519+0000 7f6afae97700 1 -- start start 2019-12-05T06:11:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.520+0000 7f6afae97700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6af414d4a0 0x7f6af414d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.520+0000 7f6afae97700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af409eaf0 con 0x7f6af4128360 2019-12-05T06:11:04.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.520+0000 7f6afae97700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6af40a23c0 con 0x7f6af40a1fa0 2019-12-05T06:11:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.520+0000 7f6afae97700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6af41323f0 con 0x7f6af414d4a0 2019-12-05T06:11:04.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.521+0000 7f6af8c33700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6af4128360 0x7f6af40a17a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33454/0 (socket says 172.21.15.16:33454) 2019-12-05T06:11:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.521+0000 7f6af8c33700 1 -- 172.21.15.16:0/1360183462 learned_addr learned my addr 172.21.15.16:0/1360183462 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:11:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.521+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> v2:172.21.15.76:3300/0 conn(0x7f6af414d4a0 0x7f6af414d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.521+0000 7f6af1ffb700 1 -- 172.21.15.16:0/1360183462 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667596445 0 0) 0x7f6ae800b400 con 0x7f6af4128360 2019-12-05T06:11:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.521+0000 7f6af1ffb700 1 -- 172.21.15.16:0/1360183462 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ad4002e80 con 0x7f6af4128360 2019-12-05T06:11:04.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.521+0000 7f6af1ffb700 1 -- 172.21.15.16:0/1360183462 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815003506 0 0) 0x7f6aec0011a0 con 0x7f6af40a1fa0 2019-12-05T06:11:04.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.521+0000 7f6af9434700 1 -- 172.21.15.16:0/1360183462 >> v1:172.21.15.76:6789/0 conn(0x7f6af40a1fa0 legacy=0x7f6af4157190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.521+0000 7f6af9434700 1 -- 172.21.15.16:0/1360183462 >> v1:172.21.15.16:6789/0 conn(0x7f6af4128360 legacy=0x7f6af40a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.521+0000 7f6af9434700 1 -- 172.21.15.16:0/1360183462 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af40a0ea0 con 0x7f6af414d4a0 2019-12-05T06:11:04.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> v2:172.21.15.76:3300/0 conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> v2:172.21.15.76:3300/0 conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=READY pgs=242 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=THROTTLE_DONE pgs=242 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6af1ffb700 1 -- 172.21.15.16:0/1360183462 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae4000c90 con 0x7f6af414d4a0 2019-12-05T06:11:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=THROTTLE_DONE pgs=242 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6afae97700 1 -- 172.21.15.16:0/1360183462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af40a0ea0 con 0x7f6af414d4a0 2019-12-05T06:11:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6af1ffb700 1 -- 172.21.15.16:0/1360183462 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ae4000c90 con 0x7f6af414d4a0 2019-12-05T06:11:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=THROTTLE_DONE pgs=242 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6af1ffb700 1 -- 172.21.15.16:0/1360183462 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ae4004620 con 0x7f6af414d4a0 2019-12-05T06:11:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.522+0000 7f6afae97700 1 -- 172.21.15.16:0/1360183462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af40a0ea0 con 0x7f6af414d4a0 2019-12-05T06:11:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.523+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=THROTTLE_DONE pgs=242 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.524+0000 7f6af1ffb700 1 -- 172.21.15.16:0/1360183462 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6ae4000c90 con 0x7f6af414d4a0 2019-12-05T06:11:04.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.524+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=THROTTLE_DONE pgs=242 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.524+0000 7f6af1ffb700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ad4023680 0x7f6ad4025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:04.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.524+0000 7f6af1ffb700 1 -- 172.21.15.16:0/1360183462 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7f6ae4045de0 con 0x7f6af414d4a0 2019-12-05T06:11:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.524+0000 7f6af8c33700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ad4023680 0x7f6ad4025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.525+0000 7f6af8c33700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ad4023680 0x7f6ad4025af0 crc :-1 s=READY pgs=144 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.525+0000 7f6afae97700 1 -- 172.21.15.16:0/1360183462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af40a0ea0 con 0x7f6af414d4a0 2019-12-05T06:11:04.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.535+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=THROTTLE_DONE pgs=242 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.535+0000 7f6af1ffb700 1 -- 172.21.15.16:0/1360183462 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6ae4001210 con 0x7f6af414d4a0 2019-12-05T06:11:04.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.789+0000 7f6afae97700 1 -- 172.21.15.16:0/1360183462 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6af40a0ea0 con 0x7f6af414d4a0 2019-12-05T06:11:04.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.790+0000 7f6af9434700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=THROTTLE_DONE pgs=242 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:04.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.790+0000 7f6af1ffb700 1 -- 172.21.15.16:0/1360183462 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f6ae4000c90 con 0x7f6af414d4a0 2019-12-05T06:11:04.789 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:11:04.789 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":24,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:02.969673+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:04.790 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 24 2019-12-05T06:11:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.794+0000 7f6afae97700 1 -- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ad4023680 msgr2=0x7f6ad4025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.794+0000 7f6afae97700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ad4023680 0x7f6ad4025af0 crc :-1 s=READY pgs=144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:04.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.794+0000 7f6afae97700 1 -- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 msgr2=0x7f6af414d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.794+0000 7f6afae97700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 0x7f6af414d890 secure :-1 s=READY pgs=242 cs=0 l=1 rx=0x7f6ae4010370 tx=0x7f6ae4010710).stop 2019-12-05T06:11:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.794+0000 7f6afae97700 1 -- 172.21.15.16:0/1360183462 shutdown_connections 2019-12-05T06:11:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.794+0000 7f6afae97700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ad4023680 0x7f6ad4025af0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.794+0000 7f6afae97700 1 --2- 172.21.15.16:0/1360183462 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6af414d4a0 0x7f6af414d890 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:04.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.795+0000 7f6afae97700 1 -- 172.21.15.16:0/1360183462 shutdown_connections 2019-12-05T06:11:04.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.795+0000 7f6afae97700 1 -- 172.21.15.16:0/1360183462 wait complete. 2019-12-05T06:11:04.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:04.795+0000 7f6afae97700 1 -- 172.21.15.16:0/1360183462 >> 172.21.15.16:0/1360183462 conn(0x7f6af409cf40 msgr2=0x7f6af4098db0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:04.804 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 24, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 24, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4839": { "addr": "172.21.15.16:6833/4138078868", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 4138078868, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 4138078868, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 4839, "incarnation": 24, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 31, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:11:02.969673+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4839 } } } ], "standbys": [] } 2019-12-05T06:11:06.804 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:07.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.039+0000 7f20503f0700 1 Processor -- start 2019-12-05T06:11:07.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.039+0000 7f20503f0700 1 -- start start 2019-12-05T06:11:07.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.040+0000 7f20503f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f204808c6d0 0x7f204808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:07.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.040+0000 7f20503f0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204808fb80 con 0x7f204808f720 2019-12-05T06:11:07.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.040+0000 7f20503f0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20480958e0 con 0x7f204808fef0 2019-12-05T06:11:07.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.040+0000 7f20503f0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204808df90 con 0x7f204808c6d0 2019-12-05T06:11:07.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204e98d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f204808c6d0 0x7f204808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204f18e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f204808f720 0x7f20481533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33470/0 (socket says 172.21.15.16:33470) 2019-12-05T06:11:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204e98d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f204808c6d0 0x7f204808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48298/0 (socket says 172.21.15.16:48298) 2019-12-05T06:11:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204f18e700 1 -- 172.21.15.16:0/1423836592 learned_addr learned my addr 172.21.15.16:0/1423836592 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:11:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204e98d700 1 -- 172.21.15.16:0/1423836592 >> v1:172.21.15.76:6789/0 conn(0x7f204808fef0 legacy=0x7f204808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204e98d700 1 -- 172.21.15.16:0/1423836592 >> v1:172.21.15.16:6789/0 conn(0x7f204808f720 legacy=0x7f20481533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204d98b700 1 -- 172.21.15.16:0/1423836592 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840617451 0 0) 0x7f2038001150 con 0x7f204808f720 2019-12-05T06:11:07.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204e98d700 1 -- 172.21.15.16:0/1423836592 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204808e620 con 0x7f204808c6d0 2019-12-05T06:11:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204e98d700 1 --2- 172.21.15.16:0/1423836592 >> v2:172.21.15.16:3300/0 conn(0x7f204808c6d0 0x7f204808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f204000af50 tx=0x7f204000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204e98d700 1 --2- 172.21.15.16:0/1423836592 >> v2:172.21.15.16:3300/0 conn(0x7f204808c6d0 0x7f204808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f204000af50 tx=0x7f204000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.041+0000 7f204e98d700 1 --2- 172.21.15.16:0/1423836592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204808c6d0 0x7f204808cac0 secure :-1 s=READY pgs=299 cs=0 l=1 rx=0x7f204000af50 tx=0x7f204000b760).ready entity=mon.0 client_cookie=fc4bc4943382a55b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.042+0000 7f204e98d700 1 --2- 172.21.15.16:0/1423836592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204808c6d0 0x7f204808cac0 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7f204000af50 tx=0x7f204000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.042+0000 7f204d98b700 1 -- 172.21.15.16:0/1423836592 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2040007210 con 0x7f204808c6d0 2019-12-05T06:11:07.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.042+0000 7f204e98d700 1 --2- 172.21.15.16:0/1423836592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204808c6d0 0x7f204808cac0 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7f204000af50 tx=0x7f204000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.042+0000 7f204e98d700 1 --2- 172.21.15.16:0/1423836592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204808c6d0 0x7f204808cac0 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7f204000af50 tx=0x7f204000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.042+0000 7f204d98b700 1 -- 172.21.15.16:0/1423836592 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2040000940 con 0x7f204808c6d0 2019-12-05T06:11:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.042+0000 7f204d98b700 1 -- 172.21.15.16:0/1423836592 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2040009ba0 con 0x7f204808c6d0 2019-12-05T06:11:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.042+0000 7f20503f0700 1 -- 172.21.15.16:0/1423836592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204808c6d0 msgr2=0x7f204808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.042+0000 7f20503f0700 1 --2- 172.21.15.16:0/1423836592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204808c6d0 0x7f204808cac0 secure :-1 s=READY pgs=299 cs=0 l=1 rx=0x7f204000af50 tx=0x7f204000b760).stop 2019-12-05T06:11:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.044+0000 7f20503f0700 1 -- 172.21.15.16:0/1423836592 shutdown_connections 2019-12-05T06:11:07.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.044+0000 7f20503f0700 1 --2- 172.21.15.16:0/1423836592 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204808c6d0 0x7f204808cac0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.044+0000 7f20503f0700 1 -- 172.21.15.16:0/1423836592 shutdown_connections 2019-12-05T06:11:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.044+0000 7f20503f0700 1 -- 172.21.15.16:0/1423836592 wait complete. 2019-12-05T06:11:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.044+0000 7f20503f0700 1 -- 172.21.15.16:0/1423836592 >> 172.21.15.16:0/1423836592 conn(0x7f2048061940 msgr2=0x7f2048096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.049+0000 7f20503f0700 1 Processor -- start 2019-12-05T06:11:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.049+0000 7f20503f0700 1 -- start start 2019-12-05T06:11:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.050+0000 7f20503f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f204806df10 0x7f204809d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:07.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.050+0000 7f20503f0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f204809e580 0x7f204809daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.050+0000 7f20503f0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f204809e030 0x7f2048088fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.050+0000 7f20503f0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204809c620 con 0x7f204806df10 2019-12-05T06:11:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.050+0000 7f20503f0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f204808ada0 con 0x7f204809e580 2019-12-05T06:11:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.050+0000 7f20503f0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f204809bec0 con 0x7f204809e030 2019-12-05T06:11:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.050+0000 7f204e98d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f204806df10 0x7f204809d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.050+0000 7f204e98d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f204806df10 0x7f204809d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48304/0 (socket says 172.21.15.16:48304) 2019-12-05T06:11:07.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.050+0000 7f204e98d700 1 -- 172.21.15.16:0/3258072510 learned_addr learned my addr 172.21.15.16:0/3258072510 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:11:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204f18e700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f204809e030 0x7f2048088fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e18c700 1 --2- 172.21.15.16:0/3258072510 >> v2:172.21.15.76:3300/0 conn(0x7f204809e580 0x7f204809daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e98d700 1 -- 172.21.15.16:0/3258072510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f204809e030 msgr2=0x7f2048088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:07.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f204809e030 0x7f2048088fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:07.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e98d700 1 -- 172.21.15.16:0/3258072510 >> v2:172.21.15.76:3300/0 conn(0x7f204809e580 msgr2=0x7f204809daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:07.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> v2:172.21.15.76:3300/0 conn(0x7f204809e580 0x7f204809daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:07.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e98d700 1 -- 172.21.15.16:0/3258072510 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204809ccb0 con 0x7f204806df10 2019-12-05T06:11:07.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e18c700 1 --2- 172.21.15.16:0/3258072510 >> v2:172.21.15.76:3300/0 conn(0x7f204809e580 0x7f204809daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:11:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> v2:172.21.15.16:3300/0 conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> v2:172.21.15.16:3300/0 conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=READY pgs=300 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.051+0000 7f203f7fe700 1 -- 172.21.15.16:0/3258072510 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2040000940 con 0x7f204806df10 2019-12-05T06:11:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.052+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.052+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.052+0000 7f203f7fe700 1 -- 172.21.15.16:0/3258072510 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2040009980 con 0x7f204806df10 2019-12-05T06:11:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.052+0000 7f20503f0700 1 -- 172.21.15.16:0/3258072510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204809ccb0 con 0x7f204806df10 2019-12-05T06:11:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.052+0000 7f203f7fe700 1 -- 172.21.15.16:0/3258072510 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2040009b70 con 0x7f204806df10 2019-12-05T06:11:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.052+0000 7f20503f0700 1 -- 172.21.15.16:0/3258072510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f204809ccb0 con 0x7f204806df10 2019-12-05T06:11:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.052+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.053+0000 7f203f7fe700 1 -- 172.21.15.16:0/3258072510 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2040009580 con 0x7f204806df10 2019-12-05T06:11:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.053+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.054+0000 7f203f7fe700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2028023630 0x7f2028025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.054+0000 7f203f7fe700 1 -- 172.21.15.16:0/3258072510 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7f2040045d70 con 0x7f204806df10 2019-12-05T06:11:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.054+0000 7f204e18c700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2028023630 0x7f2028025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.054+0000 7f20503f0700 1 -- 172.21.15.16:0/3258072510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f204809ccb0 con 0x7f204806df10 2019-12-05T06:11:07.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.054+0000 7f204e18c700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2028023630 0x7f2028025ac0 crc :-1 s=READY pgs=145 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.064+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.064+0000 7f203f7fe700 1 -- 172.21.15.16:0/3258072510 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f204002b020 con 0x7f204806df10 2019-12-05T06:11:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.262+0000 7f20503f0700 1 -- 172.21.15.16:0/3258072510 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f204809ccb0 con 0x7f204806df10 2019-12-05T06:11:07.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.263+0000 7f204e98d700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.263+0000 7f203f7fe700 1 -- 172.21.15.16:0/3258072510 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+1778 (secure 0 0 0) 0x7f2040009580 con 0x7f204806df10 2019-12-05T06:11:07.261 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 27 2019-12-05T06:11:07.262 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:11:07.262 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":27,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:07.008895+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:clientreplay","state_seq":35,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 -- 172.21.15.16:0/3258072510 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2028023630 msgr2=0x7f2028025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2028023630 0x7f2028025ac0 crc :-1 s=READY pgs=145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:07.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 -- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 msgr2=0x7f204809d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 0x7f204809d5b0 secure :-1 s=READY pgs=300 cs=0 l=1 rx=0x7f2040010450 tx=0x7f20400102d0).stop 2019-12-05T06:11:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 -- 172.21.15.16:0/3258072510 shutdown_connections 2019-12-05T06:11:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f204809e030 0x7f2048088fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 --2- 172.21.15.16:0/3258072510 >> v2:172.21.15.76:3300/0 conn(0x7f204809e580 0x7f204809daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2028023630 0x7f2028025ac0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:07.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 --2- 172.21.15.16:0/3258072510 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f204806df10 0x7f204809d5b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:07.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 -- 172.21.15.16:0/3258072510 shutdown_connections 2019-12-05T06:11:07.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 -- 172.21.15.16:0/3258072510 wait complete. 2019-12-05T06:11:07.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:07.266+0000 7f20503f0700 1 -- 172.21.15.16:0/3258072510 >> 172.21.15.16:0/3258072510 conn(0x7f2048098480 msgr2=0x7f2048071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:07.276 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:11:09.276 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:09.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.513+0000 7faf8153f700 1 Processor -- start 2019-12-05T06:11:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.513+0000 7faf8153f700 1 -- start start 2019-12-05T06:11:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.514+0000 7faf8153f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf7c08c6d0 0x7faf7c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.514+0000 7faf8153f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c08fb80 con 0x7faf7c08f720 2019-12-05T06:11:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.514+0000 7faf8153f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c0958e0 con 0x7faf7c08fef0 2019-12-05T06:11:09.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.514+0000 7faf8153f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf7c08df90 con 0x7faf7c08c6d0 2019-12-05T06:11:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.514+0000 7faf7b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf7c08c6d0 0x7faf7c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.514+0000 7faf7bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faf7c08f720 0x7faf7c1533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33496/0 (socket says 172.21.15.16:33496) 2019-12-05T06:11:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.514+0000 7faf7bfff700 1 -- 172.21.15.16:0/4187754420 learned_addr learned my addr 172.21.15.16:0/4187754420 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:11:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7a7fc700 1 -- 172.21.15.16:0/4187754420 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485789314 0 0) 0x7faf64001080 con 0x7faf7c08fef0 2019-12-05T06:11:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7b7fe700 1 -- 172.21.15.16:0/4187754420 >> v1:172.21.15.76:6789/0 conn(0x7faf7c08fef0 legacy=0x7faf7c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7b7fe700 1 -- 172.21.15.16:0/4187754420 >> v1:172.21.15.16:6789/0 conn(0x7faf7c08f720 legacy=0x7faf7c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:09.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7b7fe700 1 -- 172.21.15.16:0/4187754420 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c08e620 con 0x7faf7c08c6d0 2019-12-05T06:11:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/4187754420 >> v2:172.21.15.16:3300/0 conn(0x7faf7c08c6d0 0x7faf7c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faf6c00af50 tx=0x7faf6c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/4187754420 >> v2:172.21.15.16:3300/0 conn(0x7faf7c08c6d0 0x7faf7c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faf6c00af50 tx=0x7faf6c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/4187754420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7c08c6d0 0x7faf7c08cac0 secure :-1 s=READY pgs=305 cs=0 l=1 rx=0x7faf6c00af50 tx=0x7faf6c00b760).ready entity=mon.0 client_cookie=b03e41efcdbfa159 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/4187754420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7c08c6d0 0x7faf7c08cac0 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7faf6c00af50 tx=0x7faf6c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7a7fc700 1 -- 172.21.15.16:0/4187754420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf6c000960 con 0x7faf7c08c6d0 2019-12-05T06:11:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/4187754420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7c08c6d0 0x7faf7c08cac0 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7faf6c00af50 tx=0x7faf6c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/4187754420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7c08c6d0 0x7faf7c08cac0 secure :-1 s=THROTTLE_DONE pgs=305 cs=0 l=1 rx=0x7faf6c00af50 tx=0x7faf6c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7a7fc700 1 -- 172.21.15.16:0/4187754420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c009860 con 0x7faf7c08c6d0 2019-12-05T06:11:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf7a7fc700 1 -- 172.21.15.16:0/4187754420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf6c009f30 con 0x7faf7c08c6d0 2019-12-05T06:11:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf8153f700 1 -- 172.21.15.16:0/4187754420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7c08c6d0 msgr2=0x7faf7c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.515+0000 7faf8153f700 1 --2- 172.21.15.16:0/4187754420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7c08c6d0 0x7faf7c08cac0 secure :-1 s=READY pgs=305 cs=0 l=1 rx=0x7faf6c00af50 tx=0x7faf6c00b760).stop 2019-12-05T06:11:09.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.516+0000 7faf8153f700 1 -- 172.21.15.16:0/4187754420 shutdown_connections 2019-12-05T06:11:09.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.516+0000 7faf8153f700 1 --2- 172.21.15.16:0/4187754420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7c08c6d0 0x7faf7c08cac0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:09.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.517+0000 7faf8153f700 1 -- 172.21.15.16:0/4187754420 shutdown_connections 2019-12-05T06:11:09.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.517+0000 7faf8153f700 1 -- 172.21.15.16:0/4187754420 wait complete. 2019-12-05T06:11:09.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.517+0000 7faf8153f700 1 -- 172.21.15.16:0/4187754420 >> 172.21.15.16:0/4187754420 conn(0x7faf7c061940 msgr2=0x7faf7c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.520+0000 7faf8153f700 1 Processor -- start 2019-12-05T06:11:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.520+0000 7faf8153f700 1 -- start start 2019-12-05T06:11:09.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.521+0000 7faf8153f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.521+0000 7faf8153f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c128690 con 0x7faf7c130be0 2019-12-05T06:11:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.521+0000 7faf8153f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf7c128b40 con 0x7faf7c130410 2019-12-05T06:11:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.521+0000 7faf8153f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf7c09e690 con 0x7faf7c06dc10 2019-12-05T06:11:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf7affd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faf7c130be0 0x7faf7c12fc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33500/0 (socket says 172.21.15.16:33500) 2019-12-05T06:11:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf7affd700 1 -- 172.21.15.16:0/1616909003 learned_addr learned my addr 172.21.15.16:0/1616909003 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:11:09.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:09.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3742491405 0 0) 0x7faf64000a80 con 0x7faf7c130be0 2019-12-05T06:11:09.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf60002e80 con 0x7faf7c130be0 2019-12-05T06:11:09.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2619300940 0 0) 0x7faf740012c0 con 0x7faf7c130410 2019-12-05T06:11:09.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf60004220 con 0x7faf7c130410 2019-12-05T06:11:09.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf7b7fe700 1 -- 172.21.15.16:0/1616909003 >> v1:172.21.15.76:6789/0 conn(0x7faf7c130410 legacy=0x7faf7c1537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:09.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf7b7fe700 1 -- 172.21.15.16:0/1616909003 >> v1:172.21.15.16:6789/0 conn(0x7faf7c130be0 legacy=0x7faf7c12fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:09.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf7b7fe700 1 -- 172.21.15.16:0/1616909003 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7c12f310 con 0x7faf7c06dc10 2019-12-05T06:11:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.522+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=READY pgs=84 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.523+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=THROTTLE_DONE pgs=84 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.523+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf6c000940 con 0x7faf7c06dc10 2019-12-05T06:11:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.523+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=THROTTLE_DONE pgs=84 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.523+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=THROTTLE_DONE pgs=84 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.523+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c000940 con 0x7faf7c06dc10 2019-12-05T06:11:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.523+0000 7faf8153f700 1 -- 172.21.15.16:0/1616909003 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf7c12f310 con 0x7faf7c06dc10 2019-12-05T06:11:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.523+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf6c009b70 con 0x7faf7c06dc10 2019-12-05T06:11:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.523+0000 7faf8153f700 1 -- 172.21.15.16:0/1616909003 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf7c12f310 con 0x7faf7c06dc10 2019-12-05T06:11:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.523+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=THROTTLE_DONE pgs=84 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.524+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faf6c00a2d0 con 0x7faf7c06dc10 2019-12-05T06:11:09.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.524+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=THROTTLE_DONE pgs=84 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.524+0000 7faf73fff700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf60023650 0x7faf60025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.524+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7faf6c045dd0 con 0x7faf7c06dc10 2019-12-05T06:11:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.524+0000 7faf7affd700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf60023650 0x7faf60025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.525+0000 7faf7affd700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf60023650 0x7faf60025ae0 crc :-1 s=READY pgs=146 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:09.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.528+0000 7faf8153f700 1 -- 172.21.15.16:0/1616909003 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf7c12f310 con 0x7faf7c06dc10 2019-12-05T06:11:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.533+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=THROTTLE_DONE pgs=84 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.533+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faf6c019e10 con 0x7faf7c06dc10 2019-12-05T06:11:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.739+0000 7faf8153f700 1 -- 172.21.15.16:0/1616909003 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf7c12f310 con 0x7faf7c06dc10 2019-12-05T06:11:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.740+0000 7faf7b7fe700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=THROTTLE_DONE pgs=84 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.740+0000 7faf73fff700 1 -- 172.21.15.16:0/1616909003 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+1772 (secure 0 0 0) 0x7faf6c019e10 con 0x7faf7c06dc10 2019-12-05T06:11:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 28 2019-12-05T06:11:09.737 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:11:09.738 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:08.017512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":36,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:09.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.742+0000 7faf8153f700 1 -- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf60023650 msgr2=0x7faf60025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.742+0000 7faf8153f700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf60023650 0x7faf60025ae0 crc :-1 s=READY pgs=146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.742+0000 7faf8153f700 1 -- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 msgr2=0x7faf7c06e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.742+0000 7faf8153f700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 secure :-1 s=READY pgs=84 cs=0 l=1 rx=0x7faf6c010390 tx=0x7faf6c010730).stop 2019-12-05T06:11:09.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.742+0000 7faf8153f700 1 -- 172.21.15.16:0/1616909003 shutdown_connections 2019-12-05T06:11:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.742+0000 7faf8153f700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf60023650 0x7faf60025ae0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.742+0000 7faf8153f700 1 --2- 172.21.15.16:0/1616909003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7c06dc10 0x7faf7c06e000 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.742+0000 7faf8153f700 1 -- 172.21.15.16:0/1616909003 shutdown_connections 2019-12-05T06:11:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.742+0000 7faf8153f700 1 -- 172.21.15.16:0/1616909003 wait complete. 2019-12-05T06:11:09.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:09.742+0000 7faf8153f700 1 -- 172.21.15.16:0/1616909003 >> 172.21.15.16:0/1616909003 conn(0x7faf7c098480 msgr2=0x7faf7c071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:09.750 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=4229 2019-12-05T06:11:09.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.b 2019-12-05T06:11:10.751 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:11:10.751 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:11:10.752 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:11:10.757 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:11:10.757 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:10.862 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/18’ -> ‘copyofsnap1/test1/122/18’ 2019-12-05T06:11:10.862 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/13’ -> ‘copyofsnap1/test1/122/13’ 2019-12-05T06:11:10.863 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/122/17’ -> ‘copyofsnap1/test1/122/17’ 2019-12-05T06:11:10.863 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162’ -> ‘copyofsnap1/test1/162’ 2019-12-05T06:11:10.863 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/19’ -> ‘copyofsnap1/test1/162/19’ 2019-12-05T06:11:10.863 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/15’ -> ‘copyofsnap1/test1/162/15’ 2019-12-05T06:11:10.863 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/20’ -> ‘copyofsnap1/test1/162/20’ 2019-12-05T06:11:10.863 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/12’ -> ‘copyofsnap1/test1/162/12’ 2019-12-05T06:11:10.864 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/10’ -> ‘copyofsnap1/test1/162/10’ 2019-12-05T06:11:10.864 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/11’ -> ‘copyofsnap1/test1/162/11’ 2019-12-05T06:11:10.864 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/16’ -> ‘copyofsnap1/test1/162/16’ 2019-12-05T06:11:10.864 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/14’ -> ‘copyofsnap1/test1/162/14’ 2019-12-05T06:11:10.864 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/18’ -> ‘copyofsnap1/test1/162/18’ 2019-12-05T06:11:10.864 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/13’ -> ‘copyofsnap1/test1/162/13’ 2019-12-05T06:11:10.864 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/162/17’ -> ‘copyofsnap1/test1/162/17’ 2019-12-05T06:11:10.865 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197’ -> ‘copyofsnap1/test1/197’ 2019-12-05T06:11:10.865 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/19’ -> ‘copyofsnap1/test1/197/19’ 2019-12-05T06:11:10.865 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/15’ -> ‘copyofsnap1/test1/197/15’ 2019-12-05T06:11:10.865 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/20’ -> ‘copyofsnap1/test1/197/20’ 2019-12-05T06:11:10.865 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/12’ -> ‘copyofsnap1/test1/197/12’ 2019-12-05T06:11:10.865 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/10’ -> ‘copyofsnap1/test1/197/10’ 2019-12-05T06:11:10.865 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/11’ -> ‘copyofsnap1/test1/197/11’ 2019-12-05T06:11:10.866 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/16’ -> ‘copyofsnap1/test1/197/16’ 2019-12-05T06:11:10.866 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/14’ -> ‘copyofsnap1/test1/197/14’ 2019-12-05T06:11:10.866 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/18’ -> ‘copyofsnap1/test1/197/18’ 2019-12-05T06:11:10.866 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/13’ -> ‘copyofsnap1/test1/197/13’ 2019-12-05T06:11:10.866 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/197/17’ -> ‘copyofsnap1/test1/197/17’ 2019-12-05T06:11:10.866 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113’ -> ‘copyofsnap1/test1/113’ 2019-12-05T06:11:10.866 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/19’ -> ‘copyofsnap1/test1/113/19’ 2019-12-05T06:11:10.867 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/15’ -> ‘copyofsnap1/test1/113/15’ 2019-12-05T06:11:10.867 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/20’ -> ‘copyofsnap1/test1/113/20’ 2019-12-05T06:11:10.867 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/12’ -> ‘copyofsnap1/test1/113/12’ 2019-12-05T06:11:10.867 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/10’ -> ‘copyofsnap1/test1/113/10’ 2019-12-05T06:11:10.867 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/11’ -> ‘copyofsnap1/test1/113/11’ 2019-12-05T06:11:10.867 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/16’ -> ‘copyofsnap1/test1/113/16’ 2019-12-05T06:11:10.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/14’ -> ‘copyofsnap1/test1/113/14’ 2019-12-05T06:11:10.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/18’ -> ‘copyofsnap1/test1/113/18’ 2019-12-05T06:11:10.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/13’ -> ‘copyofsnap1/test1/113/13’ 2019-12-05T06:11:10.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/113/17’ -> ‘copyofsnap1/test1/113/17’ 2019-12-05T06:11:10.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152’ -> ‘copyofsnap1/test1/152’ 2019-12-05T06:11:10.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/19’ -> ‘copyofsnap1/test1/152/19’ 2019-12-05T06:11:10.868 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/15’ -> ‘copyofsnap1/test1/152/15’ 2019-12-05T06:11:10.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/20’ -> ‘copyofsnap1/test1/152/20’ 2019-12-05T06:11:10.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/12’ -> ‘copyofsnap1/test1/152/12’ 2019-12-05T06:11:10.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/10’ -> ‘copyofsnap1/test1/152/10’ 2019-12-05T06:11:10.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/11’ -> ‘copyofsnap1/test1/152/11’ 2019-12-05T06:11:10.869 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/16’ -> ‘copyofsnap1/test1/152/16’ 2019-12-05T06:11:10.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/14’ -> ‘copyofsnap1/test1/152/14’ 2019-12-05T06:11:10.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/18’ -> ‘copyofsnap1/test1/152/18’ 2019-12-05T06:11:10.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/13’ -> ‘copyofsnap1/test1/152/13’ 2019-12-05T06:11:10.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/152/17’ -> ‘copyofsnap1/test1/152/17’ 2019-12-05T06:11:10.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121’ -> ‘copyofsnap1/test1/121’ 2019-12-05T06:11:10.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/19’ -> ‘copyofsnap1/test1/121/19’ 2019-12-05T06:11:10.870 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/15’ -> ‘copyofsnap1/test1/121/15’ 2019-12-05T06:11:10.871 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/20’ -> ‘copyofsnap1/test1/121/20’ 2019-12-05T06:11:10.871 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/12’ -> ‘copyofsnap1/test1/121/12’ 2019-12-05T06:11:10.871 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/10’ -> ‘copyofsnap1/test1/121/10’ 2019-12-05T06:11:10.872 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/11’ -> ‘copyofsnap1/test1/121/11’ 2019-12-05T06:11:10.872 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/16’ -> ‘copyofsnap1/test1/121/16’ 2019-12-05T06:11:10.873 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/14’ -> ‘copyofsnap1/test1/121/14’ 2019-12-05T06:11:10.873 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/18’ -> ‘copyofsnap1/test1/121/18’ 2019-12-05T06:11:10.873 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/13’ -> ‘copyofsnap1/test1/121/13’ 2019-12-05T06:11:10.874 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/121/17’ -> ‘copyofsnap1/test1/121/17’ 2019-12-05T06:11:10.874 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118’ -> ‘copyofsnap1/test1/118’ 2019-12-05T06:11:10.874 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/19’ -> ‘copyofsnap1/test1/118/19’ 2019-12-05T06:11:10.874 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/15’ -> ‘copyofsnap1/test1/118/15’ 2019-12-05T06:11:10.875 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/20’ -> ‘copyofsnap1/test1/118/20’ 2019-12-05T06:11:10.875 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/12’ -> ‘copyofsnap1/test1/118/12’ 2019-12-05T06:11:10.875 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/10’ -> ‘copyofsnap1/test1/118/10’ 2019-12-05T06:11:10.875 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/11’ -> ‘copyofsnap1/test1/118/11’ 2019-12-05T06:11:10.943 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:11:10.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.994+0000 7fefff4a0700 1 Processor -- start 2019-12-05T06:11:10.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.994+0000 7fefff4a0700 1 -- start start 2019-12-05T06:11:10.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.995+0000 7fefff4a0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 0x7feff808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:10.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.995+0000 7fefff4a0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7feff808fef0 0x7feff808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:10.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.995+0000 7fefff4a0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff80958e0 con 0x7feff808ef20 2019-12-05T06:11:10.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.995+0000 7fefff4a0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff80a3f70 con 0x7feff808fef0 2019-12-05T06:11:10.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.995+0000 7fefff4a0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff808d830 con 0x7feff808c6d0 2019-12-05T06:11:10.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.995+0000 7feffe23e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7feff808ef20 0x7feff808f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33518/0 (socket says 172.21.15.16:33518) 2019-12-05T06:11:10.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.995+0000 7feffe23e700 1 -- 172.21.15.16:0/767319287 learned_addr learned my addr 172.21.15.16:0/767319287 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:11:10.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.995+0000 7feffd23c700 1 --2- 172.21.15.16:0/767319287 >> v2:172.21.15.76:3300/0 conn(0x7feff808fef0 0x7feff808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:10.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.995+0000 7feffda3d700 1 --2- 172.21.15.16:0/767319287 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 0x7feff808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:10.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffca3b700 1 -- 172.21.15.16:0/767319287 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1797788091 0 0) 0x7fefe8001150 con 0x7feff808ef20 2019-12-05T06:11:10.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffca3b700 1 -- 172.21.15.16:0/767319287 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefe0002e80 con 0x7feff808ef20 2019-12-05T06:11:10.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffda3d700 1 -- 172.21.15.16:0/767319287 >> v2:172.21.15.76:3300/0 conn(0x7feff808fef0 msgr2=0x7feff808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:11:10.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffda3d700 1 --2- 172.21.15.16:0/767319287 >> v2:172.21.15.76:3300/0 conn(0x7feff808fef0 0x7feff808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:10.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffda3d700 1 -- 172.21.15.16:0/767319287 >> v1:172.21.15.16:6789/0 conn(0x7feff808ef20 legacy=0x7feff808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:10.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffca3b700 1 -- 172.21.15.16:0/767319287 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 609871344 0 0) 0x7fefe8001130 con 0x7feff808ef20 2019-12-05T06:11:11.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffda3d700 1 -- 172.21.15.16:0/767319287 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff808d540 con 0x7feff808c6d0 2019-12-05T06:11:11.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffd23c700 1 --2- 172.21.15.16:0/767319287 >> v2:172.21.15.76:3300/0 conn(0x7feff808fef0 0x7feff808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:11:11.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffda3d700 1 --2- 172.21.15.16:0/767319287 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 0x7feff808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7feff000b2d0 tx=0x7feff000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffda3d700 1 --2- 172.21.15.16:0/767319287 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 0x7feff808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7feff000b2d0 tx=0x7feff000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffda3d700 1 --2- 172.21.15.16:0/767319287 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 0x7feff808cac0 secure :-1 s=READY pgs=86 cs=0 l=1 rx=0x7feff000b2d0 tx=0x7feff000bae0).ready entity=mon.2 client_cookie=7e2218e7689b730b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:11.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffda3d700 1 --2- 172.21.15.16:0/767319287 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 0x7feff808cac0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7feff000b2d0 tx=0x7feff000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffca3b700 1 -- 172.21.15.16:0/767319287 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feff0007590 con 0x7feff808c6d0 2019-12-05T06:11:11.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffda3d700 1 --2- 172.21.15.16:0/767319287 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 0x7feff808cac0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7feff000b2d0 tx=0x7feff000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffda3d700 1 --2- 172.21.15.16:0/767319287 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 0x7feff808cac0 secure :-1 s=THROTTLE_DONE pgs=86 cs=0 l=1 rx=0x7feff000b2d0 tx=0x7feff000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffca3b700 1 -- 172.21.15.16:0/767319287 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feff0009d30 con 0x7feff808c6d0 2019-12-05T06:11:11.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.996+0000 7feffca3b700 1 -- 172.21.15.16:0/767319287 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feff0009f20 con 0x7feff808c6d0 2019-12-05T06:11:11.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.997+0000 7fefff4a0700 1 -- 172.21.15.16:0/767319287 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 msgr2=0x7feff808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:11.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.997+0000 7fefff4a0700 1 --2- 172.21.15.16:0/767319287 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 0x7feff808cac0 secure :-1 s=READY pgs=86 cs=0 l=1 rx=0x7feff000b2d0 tx=0x7feff000bae0).stop 2019-12-05T06:11:11.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.997+0000 7fefff4a0700 1 -- 172.21.15.16:0/767319287 shutdown_connections 2019-12-05T06:11:11.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.997+0000 7fefff4a0700 1 --2- 172.21.15.16:0/767319287 >> v2:172.21.15.76:3300/0 conn(0x7feff808fef0 0x7feff808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:11:11.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.997+0000 7fefff4a0700 1 --2- 172.21.15.16:0/767319287 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feff808c6d0 0x7feff808cac0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:11.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.998+0000 7fefff4a0700 1 -- 172.21.15.16:0/767319287 shutdown_connections 2019-12-05T06:11:11.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.998+0000 7fefff4a0700 1 -- 172.21.15.16:0/767319287 wait complete. 2019-12-05T06:11:11.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:10.998+0000 7fefff4a0700 1 -- 172.21.15.16:0/767319287 >> 172.21.15.16:0/767319287 conn(0x7feff8061940 msgr2=0x7feff8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:11.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.001+0000 7fefff4a0700 1 Processor -- start 2019-12-05T06:11:11.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.001+0000 7fefff4a0700 1 -- start start 2019-12-05T06:11:11.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.001+0000 7fefff4a0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feff80992d0 0x7feff8098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:11.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7fefff4a0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff8089fb0 con 0x7feff8156cb0 2019-12-05T06:11:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7fefff4a0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff806e300 con 0x7feff806df10 2019-12-05T06:11:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7fefff4a0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff8097370 con 0x7feff80992d0 2019-12-05T06:11:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feffda3d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7feff806df10 0x7feff8098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:54824/0 (socket says 172.21.15.16:54824) 2019-12-05T06:11:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feffda3d700 1 -- 172.21.15.16:0/2844326735 learned_addr learned my addr 172.21.15.16:0/2844326735 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:11:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> v2:172.21.15.16:3300/0 conn(0x7feff80992d0 0x7feff8098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feff67fc700 1 -- 172.21.15.16:0/2844326735 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1944446007 0 0) 0x7feff000b140 con 0x7feff806df10 2019-12-05T06:11:11.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feff67fc700 1 -- 172.21.15.16:0/2844326735 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd8002e80 con 0x7feff806df10 2019-12-05T06:11:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feffd23c700 1 -- 172.21.15.16:0/2844326735 >> v1:172.21.15.76:6789/0 conn(0x7feff806df10 legacy=0x7feff8098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feffd23c700 1 -- 172.21.15.16:0/2844326735 >> v1:172.21.15.16:6789/0 conn(0x7feff8156cb0 legacy=0x7feff81590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feffd23c700 1 -- 172.21.15.16:0/2844326735 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff8097a00 con 0x7feff80992d0 2019-12-05T06:11:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> v2:172.21.15.16:3300/0 conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> v2:172.21.15.16:3300/0 conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.002+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=READY pgs=311 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:11.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.003+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.003+0000 7feff67fc700 1 -- 172.21.15.16:0/2844326735 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefec000940 con 0x7feff80992d0 2019-12-05T06:11:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.003+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.003+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.003+0000 7feff67fc700 1 -- 172.21.15.16:0/2844326735 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefec009770 con 0x7feff80992d0 2019-12-05T06:11:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.003+0000 7feff67fc700 1 -- 172.21.15.16:0/2844326735 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefec009e00 con 0x7feff80992d0 2019-12-05T06:11:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.003+0000 7fefff4a0700 1 -- 172.21.15.16:0/2844326735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff8097a00 con 0x7feff80992d0 2019-12-05T06:11:11.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.003+0000 7fefff4a0700 1 -- 172.21.15.16:0/2844326735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff8097a00 con 0x7feff80992d0 2019-12-05T06:11:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.003+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.004+0000 7feff67fc700 1 -- 172.21.15.16:0/2844326735 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fefec009960 con 0x7feff80992d0 2019-12-05T06:11:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.004+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.004+0000 7feff67fc700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fefd8023670 0x7fefd8025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.004+0000 7feff67fc700 1 -- 172.21.15.16:0/2844326735 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 6203+0+0 (secure 0 0 0) 0x7fefec045de0 con 0x7feff80992d0 2019-12-05T06:11:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.004+0000 7feffda3d700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fefd8023670 0x7fefd8025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:11.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.005+0000 7feffda3d700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fefd8023670 0x7fefd8025ae0 crc :-1 s=READY pgs=147 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.009+0000 7fefff4a0700 1 -- 172.21.15.16:0/2844326735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff8097a00 con 0x7feff80992d0 2019-12-05T06:11:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.013+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.013+0000 7feff67fc700 1 -- 172.21.15.16:0/2844326735 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fefec032990 con 0x7feff80992d0 2019-12-05T06:11:11.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.225+0000 7fefff4a0700 1 -- 172.21.15.16:0/2844326735 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feff8097a00 con 0x7feff80992d0 2019-12-05T06:11:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.226+0000 7feffd23c700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=THROTTLE_DONE pgs=311 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.226+0000 7feff67fc700 1 -- 172.21.15.16:0/2844326735 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fefec032990 con 0x7feff80992d0 2019-12-05T06:11:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2019-12-05T06:11:11.225 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:11:11.225 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:08.017512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":36,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.229+0000 7fefff4a0700 1 -- 172.21.15.16:0/2844326735 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fefd8023670 msgr2=0x7fefd8025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.229+0000 7fefff4a0700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fefd8023670 0x7fefd8025ae0 crc :-1 s=READY pgs=147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.229+0000 7fefff4a0700 1 -- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 msgr2=0x7feff8098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.229+0000 7fefff4a0700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 0x7feff8098b00 secure :-1 s=READY pgs=311 cs=0 l=1 rx=0x7fefec010390 tx=0x7fefec010730).stop 2019-12-05T06:11:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.229+0000 7fefff4a0700 1 -- 172.21.15.16:0/2844326735 shutdown_connections 2019-12-05T06:11:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.229+0000 7fefff4a0700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fefd8023670 0x7fefd8025ae0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.229+0000 7fefff4a0700 1 --2- 172.21.15.16:0/2844326735 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feff80992d0 0x7feff8098b00 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.229+0000 7fefff4a0700 1 -- 172.21.15.16:0/2844326735 shutdown_connections 2019-12-05T06:11:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.229+0000 7fefff4a0700 1 -- 172.21.15.16:0/2844326735 wait complete. 2019-12-05T06:11:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:11.229+0000 7fefff4a0700 1 -- 172.21.15.16:0/2844326735 >> 172.21.15.16:0/2844326735 conn(0x7feff80714f0 msgr2=0x7feff809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:11.237 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:11:11.238 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2019-12-05T06:11:13.232 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/16’ -> ‘copyofsnap1/test1/118/16’ 2019-12-05T06:11:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/14’ -> ‘copyofsnap1/test1/118/14’ 2019-12-05T06:11:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/18’ -> ‘copyofsnap1/test1/118/18’ 2019-12-05T06:11:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/13’ -> ‘copyofsnap1/test1/118/13’ 2019-12-05T06:11:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/118/17’ -> ‘copyofsnap1/test1/118/17’ 2019-12-05T06:11:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196’ -> ‘copyofsnap1/test1/196’ 2019-12-05T06:11:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/19’ -> ‘copyofsnap1/test1/196/19’ 2019-12-05T06:11:13.234 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/15’ -> ‘copyofsnap1/test1/196/15’ 2019-12-05T06:11:13.234 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/20’ -> ‘copyofsnap1/test1/196/20’ 2019-12-05T06:11:13.234 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/12’ -> ‘copyofsnap1/test1/196/12’ 2019-12-05T06:11:13.234 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/10’ -> ‘copyofsnap1/test1/196/10’ 2019-12-05T06:11:13.234 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/11’ -> ‘copyofsnap1/test1/196/11’ 2019-12-05T06:11:13.234 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/16’ -> ‘copyofsnap1/test1/196/16’ 2019-12-05T06:11:13.235 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/14’ -> ‘copyofsnap1/test1/196/14’ 2019-12-05T06:11:13.235 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/18’ -> ‘copyofsnap1/test1/196/18’ 2019-12-05T06:11:13.235 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/13’ -> ‘copyofsnap1/test1/196/13’ 2019-12-05T06:11:13.235 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/196/17’ -> ‘copyofsnap1/test1/196/17’ 2019-12-05T06:11:13.235 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124’ -> ‘copyofsnap1/test1/124’ 2019-12-05T06:11:13.235 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/19’ -> ‘copyofsnap1/test1/124/19’ 2019-12-05T06:11:13.235 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/15’ -> ‘copyofsnap1/test1/124/15’ 2019-12-05T06:11:13.236 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/20’ -> ‘copyofsnap1/test1/124/20’ 2019-12-05T06:11:13.236 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/12’ -> ‘copyofsnap1/test1/124/12’ 2019-12-05T06:11:13.236 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/10’ -> ‘copyofsnap1/test1/124/10’ 2019-12-05T06:11:13.236 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/11’ -> ‘copyofsnap1/test1/124/11’ 2019-12-05T06:11:13.236 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/16’ -> ‘copyofsnap1/test1/124/16’ 2019-12-05T06:11:13.236 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/14’ -> ‘copyofsnap1/test1/124/14’ 2019-12-05T06:11:13.236 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/18’ -> ‘copyofsnap1/test1/124/18’ 2019-12-05T06:11:13.237 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/13’ -> ‘copyofsnap1/test1/124/13’ 2019-12-05T06:11:13.237 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/124/17’ -> ‘copyofsnap1/test1/124/17’ 2019-12-05T06:11:13.237 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188’ -> ‘copyofsnap1/test1/188’ 2019-12-05T06:11:13.237 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/19’ -> ‘copyofsnap1/test1/188/19’ 2019-12-05T06:11:13.237 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/15’ -> ‘copyofsnap1/test1/188/15’ 2019-12-05T06:11:13.237 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/20’ -> ‘copyofsnap1/test1/188/20’ 2019-12-05T06:11:13.237 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/12’ -> ‘copyofsnap1/test1/188/12’ 2019-12-05T06:11:13.238 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/10’ -> ‘copyofsnap1/test1/188/10’ 2019-12-05T06:11:13.238 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/11’ -> ‘copyofsnap1/test1/188/11’ 2019-12-05T06:11:13.238 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/16’ -> ‘copyofsnap1/test1/188/16’ 2019-12-05T06:11:13.238 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/14’ -> ‘copyofsnap1/test1/188/14’ 2019-12-05T06:11:13.238 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/18’ -> ‘copyofsnap1/test1/188/18’ 2019-12-05T06:11:13.238 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/13’ -> ‘copyofsnap1/test1/188/13’ 2019-12-05T06:11:13.238 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/188/17’ -> ‘copyofsnap1/test1/188/17’ 2019-12-05T06:11:13.239 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145’ -> ‘copyofsnap1/test1/145’ 2019-12-05T06:11:13.239 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/19’ -> ‘copyofsnap1/test1/145/19’ 2019-12-05T06:11:13.239 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/15’ -> ‘copyofsnap1/test1/145/15’ 2019-12-05T06:11:13.239 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/20’ -> ‘copyofsnap1/test1/145/20’ 2019-12-05T06:11:13.239 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/12’ -> ‘copyofsnap1/test1/145/12’ 2019-12-05T06:11:13.239 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/10’ -> ‘copyofsnap1/test1/145/10’ 2019-12-05T06:11:13.239 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/11’ -> ‘copyofsnap1/test1/145/11’ 2019-12-05T06:11:13.240 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/16’ -> ‘copyofsnap1/test1/145/16’ 2019-12-05T06:11:13.240 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/14’ -> ‘copyofsnap1/test1/145/14’ 2019-12-05T06:11:13.240 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/18’ -> ‘copyofsnap1/test1/145/18’ 2019-12-05T06:11:13.240 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/13’ -> ‘copyofsnap1/test1/145/13’ 2019-12-05T06:11:13.241 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/145/17’ -> ‘copyofsnap1/test1/145/17’ 2019-12-05T06:11:13.241 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130’ -> ‘copyofsnap1/test1/130’ 2019-12-05T06:11:13.241 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/19’ -> ‘copyofsnap1/test1/130/19’ 2019-12-05T06:11:13.241 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/15’ -> ‘copyofsnap1/test1/130/15’ 2019-12-05T06:11:13.241 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/20’ -> ‘copyofsnap1/test1/130/20’ 2019-12-05T06:11:13.242 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/12’ -> ‘copyofsnap1/test1/130/12’ 2019-12-05T06:11:13.242 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/10’ -> ‘copyofsnap1/test1/130/10’ 2019-12-05T06:11:13.242 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/11’ -> ‘copyofsnap1/test1/130/11’ 2019-12-05T06:11:13.242 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/16’ -> ‘copyofsnap1/test1/130/16’ 2019-12-05T06:11:13.242 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/14’ -> ‘copyofsnap1/test1/130/14’ 2019-12-05T06:11:13.242 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/18’ -> ‘copyofsnap1/test1/130/18’ 2019-12-05T06:11:13.242 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/13’ -> ‘copyofsnap1/test1/130/13’ 2019-12-05T06:11:13.243 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/130/17’ -> ‘copyofsnap1/test1/130/17’ 2019-12-05T06:11:13.243 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193’ -> ‘copyofsnap1/test1/193’ 2019-12-05T06:11:13.243 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/19’ -> ‘copyofsnap1/test1/193/19’ 2019-12-05T06:11:13.243 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/15’ -> ‘copyofsnap1/test1/193/15’ 2019-12-05T06:11:13.243 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/20’ -> ‘copyofsnap1/test1/193/20’ 2019-12-05T06:11:13.243 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/12’ -> ‘copyofsnap1/test1/193/12’ 2019-12-05T06:11:15.918 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/10’ -> ‘copyofsnap1/test1/193/10’ 2019-12-05T06:11:15.918 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/11’ -> ‘copyofsnap1/test1/193/11’ 2019-12-05T06:11:15.919 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/16’ -> ‘copyofsnap1/test1/193/16’ 2019-12-05T06:11:15.919 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/14’ -> ‘copyofsnap1/test1/193/14’ 2019-12-05T06:11:15.919 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/18’ -> ‘copyofsnap1/test1/193/18’ 2019-12-05T06:11:15.919 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/13’ -> ‘copyofsnap1/test1/193/13’ 2019-12-05T06:11:15.920 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/193/17’ -> ‘copyofsnap1/test1/193/17’ 2019-12-05T06:11:15.920 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175’ -> ‘copyofsnap1/test1/175’ 2019-12-05T06:11:15.920 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/19’ -> ‘copyofsnap1/test1/175/19’ 2019-12-05T06:11:15.920 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/15’ -> ‘copyofsnap1/test1/175/15’ 2019-12-05T06:11:15.921 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/20’ -> ‘copyofsnap1/test1/175/20’ 2019-12-05T06:11:15.921 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/12’ -> ‘copyofsnap1/test1/175/12’ 2019-12-05T06:11:15.921 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/10’ -> ‘copyofsnap1/test1/175/10’ 2019-12-05T06:11:15.921 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/11’ -> ‘copyofsnap1/test1/175/11’ 2019-12-05T06:11:15.922 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/16’ -> ‘copyofsnap1/test1/175/16’ 2019-12-05T06:11:15.922 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/14’ -> ‘copyofsnap1/test1/175/14’ 2019-12-05T06:11:15.923 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/18’ -> ‘copyofsnap1/test1/175/18’ 2019-12-05T06:11:15.923 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/13’ -> ‘copyofsnap1/test1/175/13’ 2019-12-05T06:11:15.923 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/175/17’ -> ‘copyofsnap1/test1/175/17’ 2019-12-05T06:11:15.924 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181’ -> ‘copyofsnap1/test1/181’ 2019-12-05T06:11:15.924 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/19’ -> ‘copyofsnap1/test1/181/19’ 2019-12-05T06:11:15.924 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/15’ -> ‘copyofsnap1/test1/181/15’ 2019-12-05T06:11:15.924 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/20’ -> ‘copyofsnap1/test1/181/20’ 2019-12-05T06:11:15.925 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/12’ -> ‘copyofsnap1/test1/181/12’ 2019-12-05T06:11:15.925 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/10’ -> ‘copyofsnap1/test1/181/10’ 2019-12-05T06:11:15.925 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/11’ -> ‘copyofsnap1/test1/181/11’ 2019-12-05T06:11:15.925 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/16’ -> ‘copyofsnap1/test1/181/16’ 2019-12-05T06:11:15.926 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/14’ -> ‘copyofsnap1/test1/181/14’ 2019-12-05T06:11:15.926 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/18’ -> ‘copyofsnap1/test1/181/18’ 2019-12-05T06:11:15.926 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/13’ -> ‘copyofsnap1/test1/181/13’ 2019-12-05T06:11:15.926 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/181/17’ -> ‘copyofsnap1/test1/181/17’ 2019-12-05T06:11:15.927 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127’ -> ‘copyofsnap1/test1/127’ 2019-12-05T06:11:15.927 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/19’ -> ‘copyofsnap1/test1/127/19’ 2019-12-05T06:11:15.927 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/15’ -> ‘copyofsnap1/test1/127/15’ 2019-12-05T06:11:15.927 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/20’ -> ‘copyofsnap1/test1/127/20’ 2019-12-05T06:11:15.927 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/12’ -> ‘copyofsnap1/test1/127/12’ 2019-12-05T06:11:15.928 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/10’ -> ‘copyofsnap1/test1/127/10’ 2019-12-05T06:11:15.928 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/11’ -> ‘copyofsnap1/test1/127/11’ 2019-12-05T06:11:15.928 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/16’ -> ‘copyofsnap1/test1/127/16’ 2019-12-05T06:11:15.928 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/14’ -> ‘copyofsnap1/test1/127/14’ 2019-12-05T06:11:15.929 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/18’ -> ‘copyofsnap1/test1/127/18’ 2019-12-05T06:11:15.929 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/13’ -> ‘copyofsnap1/test1/127/13’ 2019-12-05T06:11:15.929 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/127/17’ -> ‘copyofsnap1/test1/127/17’ 2019-12-05T06:11:15.929 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108’ -> ‘copyofsnap1/test1/108’ 2019-12-05T06:11:15.930 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/19’ -> ‘copyofsnap1/test1/108/19’ 2019-12-05T06:11:15.930 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/15’ -> ‘copyofsnap1/test1/108/15’ 2019-12-05T06:11:15.930 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/20’ -> ‘copyofsnap1/test1/108/20’ 2019-12-05T06:11:15.930 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/12’ -> ‘copyofsnap1/test1/108/12’ 2019-12-05T06:11:15.930 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/10’ -> ‘copyofsnap1/test1/108/10’ 2019-12-05T06:11:15.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/11’ -> ‘copyofsnap1/test1/108/11’ 2019-12-05T06:11:15.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/16’ -> ‘copyofsnap1/test1/108/16’ 2019-12-05T06:11:15.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/14’ -> ‘copyofsnap1/test1/108/14’ 2019-12-05T06:11:15.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/18’ -> ‘copyofsnap1/test1/108/18’ 2019-12-05T06:11:15.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/13’ -> ‘copyofsnap1/test1/108/13’ 2019-12-05T06:11:15.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/108/17’ -> ‘copyofsnap1/test1/108/17’ 2019-12-05T06:11:15.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183’ -> ‘copyofsnap1/test1/183’ 2019-12-05T06:11:15.932 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/19’ -> ‘copyofsnap1/test1/183/19’ 2019-12-05T06:11:15.932 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/15’ -> ‘copyofsnap1/test1/183/15’ 2019-12-05T06:11:15.932 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/20’ -> ‘copyofsnap1/test1/183/20’ 2019-12-05T06:11:15.932 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/12’ -> ‘copyofsnap1/test1/183/12’ 2019-12-05T06:11:15.932 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/10’ -> ‘copyofsnap1/test1/183/10’ 2019-12-05T06:11:15.932 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/11’ -> ‘copyofsnap1/test1/183/11’ 2019-12-05T06:11:15.933 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/16’ -> ‘copyofsnap1/test1/183/16’ 2019-12-05T06:11:15.933 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/14’ -> ‘copyofsnap1/test1/183/14’ 2019-12-05T06:11:15.933 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/18’ -> ‘copyofsnap1/test1/183/18’ 2019-12-05T06:11:15.934 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/13’ -> ‘copyofsnap1/test1/183/13’ 2019-12-05T06:11:15.934 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/183/17’ -> ‘copyofsnap1/test1/183/17’ 2019-12-05T06:11:15.934 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177’ -> ‘copyofsnap1/test1/177’ 2019-12-05T06:11:15.934 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/19’ -> ‘copyofsnap1/test1/177/19’ 2019-12-05T06:11:15.934 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/15’ -> ‘copyofsnap1/test1/177/15’ 2019-12-05T06:11:20.076 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/20’ -> ‘copyofsnap1/test1/177/20’ 2019-12-05T06:11:20.076 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/12’ -> ‘copyofsnap1/test1/177/12’ 2019-12-05T06:11:20.076 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/10’ -> ‘copyofsnap1/test1/177/10’ 2019-12-05T06:11:20.077 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/11’ -> ‘copyofsnap1/test1/177/11’ 2019-12-05T06:11:20.077 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/16’ -> ‘copyofsnap1/test1/177/16’ 2019-12-05T06:11:20.077 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/14’ -> ‘copyofsnap1/test1/177/14’ 2019-12-05T06:11:20.077 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/18’ -> ‘copyofsnap1/test1/177/18’ 2019-12-05T06:11:20.077 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/13’ -> ‘copyofsnap1/test1/177/13’ 2019-12-05T06:11:20.077 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/177/17’ -> ‘copyofsnap1/test1/177/17’ 2019-12-05T06:11:20.077 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169’ -> ‘copyofsnap1/test1/169’ 2019-12-05T06:11:20.078 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/19’ -> ‘copyofsnap1/test1/169/19’ 2019-12-05T06:11:20.078 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/15’ -> ‘copyofsnap1/test1/169/15’ 2019-12-05T06:11:20.078 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/20’ -> ‘copyofsnap1/test1/169/20’ 2019-12-05T06:11:20.079 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/12’ -> ‘copyofsnap1/test1/169/12’ 2019-12-05T06:11:20.079 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/10’ -> ‘copyofsnap1/test1/169/10’ 2019-12-05T06:11:20.079 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/11’ -> ‘copyofsnap1/test1/169/11’ 2019-12-05T06:11:20.079 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/16’ -> ‘copyofsnap1/test1/169/16’ 2019-12-05T06:11:20.079 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/14’ -> ‘copyofsnap1/test1/169/14’ 2019-12-05T06:11:20.079 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/18’ -> ‘copyofsnap1/test1/169/18’ 2019-12-05T06:11:20.079 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/13’ -> ‘copyofsnap1/test1/169/13’ 2019-12-05T06:11:20.080 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/169/17’ -> ‘copyofsnap1/test1/169/17’ 2019-12-05T06:11:20.080 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154’ -> ‘copyofsnap1/test1/154’ 2019-12-05T06:11:20.080 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/19’ -> ‘copyofsnap1/test1/154/19’ 2019-12-05T06:11:20.080 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/15’ -> ‘copyofsnap1/test1/154/15’ 2019-12-05T06:11:20.080 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/20’ -> ‘copyofsnap1/test1/154/20’ 2019-12-05T06:11:20.080 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/12’ -> ‘copyofsnap1/test1/154/12’ 2019-12-05T06:11:20.081 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/10’ -> ‘copyofsnap1/test1/154/10’ 2019-12-05T06:11:20.081 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/11’ -> ‘copyofsnap1/test1/154/11’ 2019-12-05T06:11:20.081 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/16’ -> ‘copyofsnap1/test1/154/16’ 2019-12-05T06:11:20.081 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/14’ -> ‘copyofsnap1/test1/154/14’ 2019-12-05T06:11:20.081 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/18’ -> ‘copyofsnap1/test1/154/18’ 2019-12-05T06:11:20.081 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/13’ -> ‘copyofsnap1/test1/154/13’ 2019-12-05T06:11:20.082 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/154/17’ -> ‘copyofsnap1/test1/154/17’ 2019-12-05T06:11:20.082 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171’ -> ‘copyofsnap1/test1/171’ 2019-12-05T06:11:20.082 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/19’ -> ‘copyofsnap1/test1/171/19’ 2019-12-05T06:11:20.082 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/15’ -> ‘copyofsnap1/test1/171/15’ 2019-12-05T06:11:20.082 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/20’ -> ‘copyofsnap1/test1/171/20’ 2019-12-05T06:11:20.082 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/12’ -> ‘copyofsnap1/test1/171/12’ 2019-12-05T06:11:20.083 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/10’ -> ‘copyofsnap1/test1/171/10’ 2019-12-05T06:11:20.083 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/11’ -> ‘copyofsnap1/test1/171/11’ 2019-12-05T06:11:20.083 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/16’ -> ‘copyofsnap1/test1/171/16’ 2019-12-05T06:11:20.083 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/14’ -> ‘copyofsnap1/test1/171/14’ 2019-12-05T06:11:20.084 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/18’ -> ‘copyofsnap1/test1/171/18’ 2019-12-05T06:11:20.084 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/13’ -> ‘copyofsnap1/test1/171/13’ 2019-12-05T06:11:20.084 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/171/17’ -> ‘copyofsnap1/test1/171/17’ 2019-12-05T06:11:20.084 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139’ -> ‘copyofsnap1/test1/139’ 2019-12-05T06:11:20.084 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/19’ -> ‘copyofsnap1/test1/139/19’ 2019-12-05T06:11:20.084 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/15’ -> ‘copyofsnap1/test1/139/15’ 2019-12-05T06:11:20.085 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/20’ -> ‘copyofsnap1/test1/139/20’ 2019-12-05T06:11:20.085 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/12’ -> ‘copyofsnap1/test1/139/12’ 2019-12-05T06:11:20.085 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/10’ -> ‘copyofsnap1/test1/139/10’ 2019-12-05T06:11:20.085 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/11’ -> ‘copyofsnap1/test1/139/11’ 2019-12-05T06:11:20.085 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/16’ -> ‘copyofsnap1/test1/139/16’ 2019-12-05T06:11:20.085 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/14’ -> ‘copyofsnap1/test1/139/14’ 2019-12-05T06:11:20.085 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/18’ -> ‘copyofsnap1/test1/139/18’ 2019-12-05T06:11:20.086 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/13’ -> ‘copyofsnap1/test1/139/13’ 2019-12-05T06:11:20.086 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/139/17’ -> ‘copyofsnap1/test1/139/17’ 2019-12-05T06:11:20.086 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102’ -> ‘copyofsnap1/test1/102’ 2019-12-05T06:11:20.086 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/19’ -> ‘copyofsnap1/test1/102/19’ 2019-12-05T06:11:20.086 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/15’ -> ‘copyofsnap1/test1/102/15’ 2019-12-05T06:11:20.086 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/20’ -> ‘copyofsnap1/test1/102/20’ 2019-12-05T06:11:20.087 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/12’ -> ‘copyofsnap1/test1/102/12’ 2019-12-05T06:11:20.087 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/10’ -> ‘copyofsnap1/test1/102/10’ 2019-12-05T06:11:20.087 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/11’ -> ‘copyofsnap1/test1/102/11’ 2019-12-05T06:11:20.087 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/16’ -> ‘copyofsnap1/test1/102/16’ 2019-12-05T06:11:20.087 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/14’ -> ‘copyofsnap1/test1/102/14’ 2019-12-05T06:11:20.088 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/18’ -> ‘copyofsnap1/test1/102/18’ 2019-12-05T06:11:20.088 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/13’ -> ‘copyofsnap1/test1/102/13’ 2019-12-05T06:11:20.088 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/102/17’ -> ‘copyofsnap1/test1/102/17’ 2019-12-05T06:11:20.089 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129’ -> ‘copyofsnap1/test1/129’ 2019-12-05T06:11:20.930 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/19’ -> ‘copyofsnap1/test1/129/19’ 2019-12-05T06:11:20.930 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/15’ -> ‘copyofsnap1/test1/129/15’ 2019-12-05T06:11:20.930 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/20’ -> ‘copyofsnap1/test1/129/20’ 2019-12-05T06:11:20.930 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/12’ -> ‘copyofsnap1/test1/129/12’ 2019-12-05T06:11:20.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/10’ -> ‘copyofsnap1/test1/129/10’ 2019-12-05T06:11:20.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/11’ -> ‘copyofsnap1/test1/129/11’ 2019-12-05T06:11:20.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/16’ -> ‘copyofsnap1/test1/129/16’ 2019-12-05T06:11:20.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/14’ -> ‘copyofsnap1/test1/129/14’ 2019-12-05T06:11:20.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/18’ -> ‘copyofsnap1/test1/129/18’ 2019-12-05T06:11:20.931 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/13’ -> ‘copyofsnap1/test1/129/13’ 2019-12-05T06:11:20.932 INFO:tasks.workunit.client.0.smithi016.stdout:‘.snap/test1/129/17’ -> ‘copyofsnap1/test1/129/17’ 2019-12-05T06:11:20.932 INFO:tasks.workunit.client.0.smithi016.stdout:Take third snapshot .snap/test3 2019-12-05T06:11:20.988 INFO:tasks.workunit.client.0.smithi016.stdout:Delete the snapshots... 2019-12-05T06:11:23.447 INFO:tasks.workunit.client.0.smithi016.stdout:.//.snap/test1 2019-12-05T06:11:23.447 INFO:tasks.workunit.client.0.smithi016.stdout:.//.snap/test2 2019-12-05T06:11:23.447 INFO:tasks.workunit.client.0.smithi016.stdout:.//.snap/test3 2019-12-05T06:11:23.458 INFO:tasks.workunit.client.0.smithi016.stdout:./274/.snap/snap-subdir-test 2019-12-05T06:11:23.471 INFO:tasks.workunit.client.0.smithi016.stdout:./166/.snap/snap-subdir-test 2019-12-05T06:11:23.484 INFO:tasks.workunit.client.0.smithi016.stdout:./256/.snap/snap-subdir-test 2019-12-05T06:11:23.493 INFO:tasks.workunit.client.0.smithi016.stdout:./247/.snap/snap-subdir-test 2019-12-05T06:11:23.503 INFO:tasks.workunit.client.0.smithi016.stdout:./248/.snap/snap-subdir-test 2019-12-05T06:11:24.367 INFO:tasks.workunit.client.0.smithi016.stdout:./147/.snap/snap-subdir-test 2019-12-05T06:11:24.393 INFO:tasks.workunit.client.0.smithi016.stdout:./151/.snap/snap-subdir-test 2019-12-05T06:11:24.416 INFO:tasks.workunit.client.0.smithi016.stdout:./170/.snap/snap-subdir-test 2019-12-05T06:11:24.435 INFO:tasks.workunit.client.0.smithi016.stdout:./167/.snap/snap-subdir-test 2019-12-05T06:11:24.454 INFO:tasks.workunit.client.0.smithi016.stdout:./140/.snap/snap-subdir-test 2019-12-05T06:11:24.471 INFO:tasks.workunit.client.0.smithi016.stdout:./142/.snap/snap-subdir-test 2019-12-05T06:11:24.488 INFO:tasks.workunit.client.0.smithi016.stdout:./150/.snap/snap-subdir-test 2019-12-05T06:11:24.503 INFO:tasks.workunit.client.0.smithi016.stdout:./273/.snap/snap-subdir-test 2019-12-05T06:11:24.517 INFO:tasks.workunit.client.0.smithi016.stdout:./269/.snap/snap-subdir-test 2019-12-05T06:11:24.531 INFO:tasks.workunit.client.0.smithi016.stdout:./105/.snap/snap-subdir-test 2019-12-05T06:11:24.557 INFO:tasks.workunit.client.0.smithi016.stdout:./210/.snap/snap-subdir-test 2019-12-05T06:11:24.565 INFO:tasks.workunit.client.0.smithi016.stdout:./265/.snap/snap-subdir-test 2019-12-05T06:11:24.573 INFO:tasks.workunit.client.0.smithi016.stdout:./218/.snap/snap-subdir-test 2019-12-05T06:11:24.584 INFO:tasks.workunit.client.0.smithi016.stdout:./290/.snap/snap-subdir-test 2019-12-05T06:11:24.597 INFO:tasks.workunit.client.0.smithi016.stdout:./149/.snap/snap-subdir-test 2019-12-05T06:11:24.608 INFO:tasks.workunit.client.0.smithi016.stdout:./109/.snap/snap-subdir-test 2019-12-05T06:11:24.619 INFO:tasks.workunit.client.0.smithi016.stdout:./292/.snap/snap-subdir-test 2019-12-05T06:11:24.629 INFO:tasks.workunit.client.0.smithi016.stdout:./242/.snap/snap-subdir-test 2019-12-05T06:11:24.641 INFO:tasks.workunit.client.0.smithi016.stdout:./158/.snap/snap-subdir-test 2019-12-05T06:11:24.651 INFO:tasks.workunit.client.0.smithi016.stdout:./243/.snap/snap-subdir-test 2019-12-05T06:11:24.663 INFO:tasks.workunit.client.0.smithi016.stdout:./106/.snap/snap-subdir-test 2019-12-05T06:11:24.674 INFO:tasks.workunit.client.0.smithi016.stdout:./250/.snap/snap-subdir-test 2019-12-05T06:11:24.684 INFO:tasks.workunit.client.0.smithi016.stdout:./221/.snap/snap-subdir-test 2019-12-05T06:11:24.695 INFO:tasks.workunit.client.0.smithi016.stdout:./244/.snap/snap-subdir-test 2019-12-05T06:11:24.706 INFO:tasks.workunit.client.0.smithi016.stdout:./258/.snap/snap-subdir-test 2019-12-05T06:11:24.719 INFO:tasks.workunit.client.0.smithi016.stdout:./268/.snap/snap-subdir-test 2019-12-05T06:11:24.919 INFO:tasks.workunit.client.0.smithi016.stdout:./189/.snap/snap-subdir-test 2019-12-05T06:11:24.943 INFO:tasks.workunit.client.0.smithi016.stdout:./285/.snap/snap-subdir-test 2019-12-05T06:11:24.966 INFO:tasks.workunit.client.0.smithi016.stdout:./208/.snap/snap-subdir-test 2019-12-05T06:11:24.988 INFO:tasks.workunit.client.0.smithi016.stdout:./288/.snap/snap-subdir-test 2019-12-05T06:11:25.018 INFO:tasks.workunit.client.0.smithi016.stdout:./209/.snap/snap-subdir-test 2019-12-05T06:11:25.033 INFO:tasks.workunit.client.0.smithi016.stdout:./253/.snap/snap-subdir-test 2019-12-05T06:11:25.048 INFO:tasks.workunit.client.0.smithi016.stdout:./249/.snap/snap-subdir-test 2019-12-05T06:11:25.062 INFO:tasks.workunit.client.0.smithi016.stdout:./133/.snap/snap-subdir-test 2019-12-05T06:11:25.074 INFO:tasks.workunit.client.0.smithi016.stdout:./291/.snap/snap-subdir-test 2019-12-05T06:11:25.085 INFO:tasks.workunit.client.0.smithi016.stdout:./226/.snap/snap-subdir-test 2019-12-05T06:11:25.098 INFO:tasks.workunit.client.0.smithi016.stdout:./173/.snap/snap-subdir-test 2019-12-05T06:11:25.109 INFO:tasks.workunit.client.0.smithi016.stdout:./174/.snap/snap-subdir-test 2019-12-05T06:11:25.120 INFO:tasks.workunit.client.0.smithi016.stdout:./299/.snap/snap-subdir-test 2019-12-05T06:11:25.131 INFO:tasks.workunit.client.0.smithi016.stdout:./262/.snap/snap-subdir-test 2019-12-05T06:11:25.493 INFO:tasks.workunit.client.0.smithi016.stdout:./231/.snap/snap-subdir-test 2019-12-05T06:11:25.515 INFO:tasks.workunit.client.0.smithi016.stdout:./214/.snap/snap-subdir-test 2019-12-05T06:11:25.536 INFO:tasks.workunit.client.0.smithi016.stdout:./281/.snap/snap-subdir-test 2019-12-05T06:11:25.556 INFO:tasks.workunit.client.0.smithi016.stdout:./289/.snap/snap-subdir-test 2019-12-05T06:11:25.572 INFO:tasks.workunit.client.0.smithi016.stdout:./283/.snap/snap-subdir-test 2019-12-05T06:11:25.589 INFO:tasks.workunit.client.0.smithi016.stdout:./232/.snap/snap-subdir-test 2019-12-05T06:11:25.604 INFO:tasks.workunit.client.0.smithi016.stdout:./298/.snap/snap-subdir-test 2019-12-05T06:11:25.620 INFO:tasks.workunit.client.0.smithi016.stdout:./233/.snap/snap-subdir-test 2019-12-05T06:11:25.635 INFO:tasks.workunit.client.0.smithi016.stdout:./155/.snap/snap-subdir-test 2019-12-05T06:11:25.658 INFO:tasks.workunit.client.0.smithi016.stdout:./114/.snap/snap-subdir-test 2019-12-05T06:11:25.664 INFO:tasks.workunit.client.0.smithi016.stdout:./159/.snap/snap-subdir-test 2019-12-05T06:11:25.678 INFO:tasks.workunit.client.0.smithi016.stdout:./187/.snap/snap-subdir-test 2019-12-05T06:11:25.691 INFO:tasks.workunit.client.0.smithi016.stdout:./185/.snap/snap-subdir-test 2019-12-05T06:11:25.704 INFO:tasks.workunit.client.0.smithi016.stdout:./293/.snap/snap-subdir-test 2019-12-05T06:11:25.715 INFO:tasks.workunit.client.0.smithi016.stdout:./228/.snap/snap-subdir-test 2019-12-05T06:11:25.726 INFO:tasks.workunit.client.0.smithi016.stdout:./120/.snap/snap-subdir-test 2019-12-05T06:11:25.737 INFO:tasks.workunit.client.0.smithi016.stdout:./110/.snap/snap-subdir-test 2019-12-05T06:11:25.747 INFO:tasks.workunit.client.0.smithi016.stdout:./267/.snap/snap-subdir-test 2019-12-05T06:11:25.759 INFO:tasks.workunit.client.0.smithi016.stdout:./202/.snap/snap-subdir-test 2019-12-05T06:11:25.768 INFO:tasks.workunit.client.0.smithi016.stdout:./137/.snap/snap-subdir-test 2019-12-05T06:11:25.779 INFO:tasks.workunit.client.0.smithi016.stdout:./246/.snap/snap-subdir-test 2019-12-05T06:11:25.789 INFO:tasks.workunit.client.0.smithi016.stdout:./219/.snap/snap-subdir-test 2019-12-05T06:11:25.974 INFO:tasks.workunit.client.0.smithi016.stdout:./198/.snap/snap-subdir-test 2019-12-05T06:11:25.982 INFO:tasks.workunit.client.0.smithi016.stdout:./136/.snap/snap-subdir-test 2019-12-05T06:11:26.004 INFO:tasks.workunit.client.0.smithi016.stdout:./270/.snap/snap-subdir-test 2019-12-05T06:11:26.017 INFO:tasks.workunit.client.0.smithi016.stdout:./125/.snap/snap-subdir-test 2019-12-05T06:11:26.033 INFO:tasks.workunit.client.0.smithi016.stdout:./194/.snap/snap-subdir-test 2019-12-05T06:11:26.049 INFO:tasks.workunit.client.0.smithi016.stdout:./119/.snap/snap-subdir-test 2019-12-05T06:11:26.059 INFO:tasks.workunit.client.0.smithi016.stdout:./297/.snap/snap-subdir-test 2019-12-05T06:11:26.071 INFO:tasks.workunit.client.0.smithi016.stdout:./230/.snap/snap-subdir-test 2019-12-05T06:11:26.346 INFO:tasks.workunit.client.0.smithi016.stdout:./260/.snap/snap-subdir-test 2019-12-05T06:11:27.934 INFO:tasks.workunit.client.0.smithi016.stdout:./157/.snap/snap-subdir-test 2019-12-05T06:11:27.958 INFO:tasks.workunit.client.0.smithi016.stdout:./172/.snap/snap-subdir-test 2019-12-05T06:11:27.978 INFO:tasks.workunit.client.0.smithi016.stdout:./277/.snap/snap-subdir-test 2019-12-05T06:11:28.918 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:11:28.923 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:11:28.930 INFO:tasks.workunit.client.0.smithi016.stdout:./186/.snap/snap-subdir-test 2019-12-05T06:11:29.706 INFO:tasks.workunit.client.0.smithi016.stdout:./237/.snap/snap-subdir-test 2019-12-05T06:11:29.732 INFO:tasks.workunit.client.0.smithi016.stdout:./190/.snap/snap-subdir-test 2019-12-05T06:11:29.754 INFO:tasks.workunit.client.0.smithi016.stdout:./213/.snap/snap-subdir-test 2019-12-05T06:11:29.775 INFO:tasks.workunit.client.0.smithi016.stdout:./104/.snap/snap-subdir-test 2019-12-05T06:11:29.795 INFO:tasks.workunit.client.0.smithi016.stdout:./164/.snap/snap-subdir-test 2019-12-05T06:11:29.813 INFO:tasks.workunit.client.0.smithi016.stdout:./112/.snap/snap-subdir-test 2019-12-05T06:11:29.829 INFO:tasks.workunit.client.0.smithi016.stdout:./295/.snap/snap-subdir-test 2019-12-05T06:11:29.843 INFO:tasks.workunit.client.0.smithi016.stdout:./272/.snap/snap-subdir-test 2019-12-05T06:11:29.858 INFO:tasks.workunit.client.0.smithi016.stdout:./100/.snap/snap-subdir-test 2019-12-05T06:11:29.873 INFO:tasks.workunit.client.0.smithi016.stdout:./134/.snap/snap-subdir-test 2019-12-05T06:11:30.786 INFO:tasks.workunit.client.0.smithi016.stdout:./200/.snap/snap-subdir-test 2019-12-05T06:11:30.811 INFO:tasks.workunit.client.0.smithi016.stdout:./115/.snap/snap-subdir-test 2019-12-05T06:11:30.830 INFO:tasks.workunit.client.0.smithi016.stdout:./284/.snap/snap-subdir-test 2019-12-05T06:11:30.847 INFO:tasks.workunit.client.0.smithi016.stdout:./280/.snap/snap-subdir-test 2019-12-05T06:11:30.865 INFO:tasks.workunit.client.0.smithi016.stdout:./116/.snap/snap-subdir-test 2019-12-05T06:11:30.882 INFO:tasks.workunit.client.0.smithi016.stdout:./245/.snap/snap-subdir-test 2019-12-05T06:11:30.896 INFO:tasks.workunit.client.0.smithi016.stdout:./195/.snap/snap-subdir-test 2019-12-05T06:11:30.912 INFO:tasks.workunit.client.0.smithi016.stdout:./138/.snap/snap-subdir-test 2019-12-05T06:11:30.926 INFO:tasks.workunit.client.0.smithi016.stdout:./135/.snap/snap-subdir-test 2019-12-05T06:11:30.940 INFO:tasks.workunit.client.0.smithi016.stdout:./117/.snap/snap-subdir-test 2019-12-05T06:11:31.679 INFO:tasks.workunit.client.0.smithi016.stdout:./241/.snap/snap-subdir-test 2019-12-05T06:11:31.703 INFO:tasks.workunit.client.0.smithi016.stdout:./143/.snap/snap-subdir-test 2019-12-05T06:11:31.723 INFO:tasks.workunit.client.0.smithi016.stdout:./180/.snap/snap-subdir-test 2019-12-05T06:11:31.741 INFO:tasks.workunit.client.0.smithi016.stdout:./126/.snap/snap-subdir-test 2019-12-05T06:11:31.759 INFO:tasks.workunit.client.0.smithi016.stdout:./161/.snap/snap-subdir-test 2019-12-05T06:11:31.775 INFO:tasks.workunit.client.0.smithi016.stdout:./204/.snap/snap-subdir-test 2019-12-05T06:11:31.792 INFO:tasks.workunit.client.0.smithi016.stdout:./176/.snap/snap-subdir-test 2019-12-05T06:11:31.806 INFO:tasks.workunit.client.0.smithi016.stdout:./255/.snap/snap-subdir-test 2019-12-05T06:11:31.820 INFO:tasks.workunit.client.0.smithi016.stdout:./215/.snap/snap-subdir-test 2019-12-05T06:11:31.834 INFO:tasks.workunit.client.0.smithi016.stdout:./111/.snap/snap-subdir-test 2019-12-05T06:11:31.847 INFO:tasks.workunit.client.0.smithi016.stdout:./238/.snap/snap-subdir-test 2019-12-05T06:11:31.860 INFO:tasks.workunit.client.0.smithi016.stdout:./191/.snap/snap-subdir-test 2019-12-05T06:11:31.876 INFO:tasks.workunit.client.0.smithi016.stdout:./184/.snap/snap-subdir-test 2019-12-05T06:11:31.886 INFO:tasks.workunit.client.0.smithi016.stdout:./192/.snap/snap-subdir-test 2019-12-05T06:11:31.898 INFO:tasks.workunit.client.0.smithi016.stdout:./103/.snap/snap-subdir-test 2019-12-05T06:11:31.909 INFO:tasks.workunit.client.0.smithi016.stdout:./128/.snap/snap-subdir-test 2019-12-05T06:11:31.920 INFO:tasks.workunit.client.0.smithi016.stdout:./229/.snap/snap-subdir-test 2019-12-05T06:11:31.931 INFO:tasks.workunit.client.0.smithi016.stdout:./144/.snap/snap-subdir-test 2019-12-05T06:11:31.942 INFO:tasks.workunit.client.0.smithi016.stdout:./146/.snap/snap-subdir-test 2019-12-05T06:11:31.953 INFO:tasks.workunit.client.0.smithi016.stdout:./227/.snap/snap-subdir-test 2019-12-05T06:11:31.963 INFO:tasks.workunit.client.0.smithi016.stdout:./278/.snap/snap-subdir-test 2019-12-05T06:11:31.975 INFO:tasks.workunit.client.0.smithi016.stdout:./141/.snap/snap-subdir-test 2019-12-05T06:11:31.985 INFO:tasks.workunit.client.0.smithi016.stdout:./234/.snap/snap-subdir-test 2019-12-05T06:11:31.995 INFO:tasks.workunit.client.0.smithi016.stdout:./225/.snap/snap-subdir-test 2019-12-05T06:11:32.515 INFO:tasks.workunit.client.0.smithi016.stdout:./182/.snap/snap-subdir-test 2019-12-05T06:11:32.540 INFO:tasks.workunit.client.0.smithi016.stdout:./107/.snap/snap-subdir-test 2019-12-05T06:11:32.768 INFO:tasks.workunit.client.0.smithi016.stdout:./178/.snap/snap-subdir-test 2019-12-05T06:11:32.786 INFO:tasks.workunit.client.0.smithi016.stdout:./266/.snap/snap-subdir-test 2019-12-05T06:11:32.806 INFO:tasks.workunit.client.0.smithi016.stdout:./240/.snap/snap-subdir-test 2019-12-05T06:11:32.827 INFO:tasks.workunit.client.0.smithi016.stdout:./156/.snap/snap-subdir-test 2019-12-05T06:11:32.846 INFO:tasks.workunit.client.0.smithi016.stdout:./131/.snap/snap-subdir-test 2019-12-05T06:11:32.864 INFO:tasks.workunit.client.0.smithi016.stdout:./123/.snap/snap-subdir-test 2019-12-05T06:11:32.881 INFO:tasks.workunit.client.0.smithi016.stdout:./160/.snap/snap-subdir-test 2019-12-05T06:11:32.897 INFO:tasks.workunit.client.0.smithi016.stdout:./165/.snap/snap-subdir-test 2019-12-05T06:11:33.454 INFO:tasks.workunit.client.0.smithi016.stdout:./101/.snap/snap-subdir-test 2019-12-05T06:11:33.477 INFO:tasks.workunit.client.0.smithi016.stdout:./148/.snap/snap-subdir-test 2019-12-05T06:11:33.498 INFO:tasks.workunit.client.0.smithi016.stdout:./168/.snap/snap-subdir-test 2019-12-05T06:11:33.518 INFO:tasks.workunit.client.0.smithi016.stdout:./179/.snap/snap-subdir-test 2019-12-05T06:11:34.225 INFO:tasks.workunit.client.0.smithi016.stdout:./257/.snap/snap-subdir-test 2019-12-05T06:11:34.249 INFO:tasks.workunit.client.0.smithi016.stdout:./252/.snap/snap-subdir-test 2019-12-05T06:11:34.271 INFO:tasks.workunit.client.0.smithi016.stdout:./132/.snap/snap-subdir-test 2019-12-05T06:11:34.293 INFO:tasks.workunit.client.0.smithi016.stdout:./236/.snap/snap-subdir-test 2019-12-05T06:11:34.314 INFO:tasks.workunit.client.0.smithi016.stdout:./211/.snap/snap-subdir-test 2019-12-05T06:11:34.329 INFO:tasks.workunit.client.0.smithi016.stdout:./199/.snap/snap-subdir-test 2019-12-05T06:11:34.345 INFO:tasks.workunit.client.0.smithi016.stdout:./205/.snap/snap-subdir-test 2019-12-05T06:11:34.360 INFO:tasks.workunit.client.0.smithi016.stdout:./163/.snap/snap-subdir-test 2019-12-05T06:11:34.373 INFO:tasks.workunit.client.0.smithi016.stdout:./261/.snap/snap-subdir-test 2019-12-05T06:11:34.387 INFO:tasks.workunit.client.0.smithi016.stdout:./201/.snap/snap-subdir-test 2019-12-05T06:11:34.399 INFO:tasks.workunit.client.0.smithi016.stdout:./282/.snap/snap-subdir-test 2019-12-05T06:11:34.411 INFO:tasks.workunit.client.0.smithi016.stdout:./296/.snap/snap-subdir-test 2019-12-05T06:11:34.424 INFO:tasks.workunit.client.0.smithi016.stdout:./217/.snap/snap-subdir-test 2019-12-05T06:11:34.437 INFO:tasks.workunit.client.0.smithi016.stdout:./153/.snap/snap-subdir-test 2019-12-05T06:11:34.449 INFO:tasks.workunit.client.0.smithi016.stdout:./206/.snap/snap-subdir-test 2019-12-05T06:11:34.462 INFO:tasks.workunit.client.0.smithi016.stdout:./122/.snap/snap-subdir-test 2019-12-05T06:11:34.473 INFO:tasks.workunit.client.0.smithi016.stdout:./162/.snap/snap-subdir-test 2019-12-05T06:11:34.485 INFO:tasks.workunit.client.0.smithi016.stdout:./197/.snap/snap-subdir-test 2019-12-05T06:11:34.495 INFO:tasks.workunit.client.0.smithi016.stdout:./216/.snap/snap-subdir-test 2019-12-05T06:11:38.603 INFO:tasks.workunit.client.0.smithi016.stdout:./254/.snap/snap-subdir-test 2019-12-05T06:11:38.616 INFO:tasks.workunit.client.0.smithi016.stdout:./113/.snap/snap-subdir-test 2019-12-05T06:11:38.626 INFO:tasks.workunit.client.0.smithi016.stdout:./152/.snap/snap-subdir-test 2019-12-05T06:11:38.637 INFO:tasks.workunit.client.0.smithi016.stdout:./121/.snap/snap-subdir-test 2019-12-05T06:11:38.648 INFO:tasks.workunit.client.0.smithi016.stdout:./118/.snap/snap-subdir-test 2019-12-05T06:11:38.660 INFO:tasks.workunit.client.0.smithi016.stdout:./196/.snap/snap-subdir-test 2019-12-05T06:11:38.669 INFO:tasks.workunit.client.0.smithi016.stdout:./124/.snap/snap-subdir-test 2019-12-05T06:11:38.679 INFO:tasks.workunit.client.0.smithi016.stdout:./239/.snap/snap-subdir-test 2019-12-05T06:11:38.690 INFO:tasks.workunit.client.0.smithi016.stdout:./188/.snap/snap-subdir-test 2019-12-05T06:11:38.699 INFO:tasks.workunit.client.0.smithi016.stdout:./287/.snap/snap-subdir-test 2019-12-05T06:11:38.709 INFO:tasks.workunit.client.0.smithi016.stdout:./275/.snap/snap-subdir-test 2019-12-05T06:11:38.718 INFO:tasks.workunit.client.0.smithi016.stdout:./259/.snap/snap-subdir-test 2019-12-05T06:11:38.728 INFO:tasks.workunit.client.0.smithi016.stdout:./224/.snap/snap-subdir-test 2019-12-05T06:11:38.738 INFO:tasks.workunit.client.0.smithi016.stdout:./264/.snap/snap-subdir-test 2019-12-05T06:11:38.748 INFO:tasks.workunit.client.0.smithi016.stdout:./145/.snap/snap-subdir-test 2019-12-05T06:11:38.758 INFO:tasks.workunit.client.0.smithi016.stdout:./279/.snap/snap-subdir-test 2019-12-05T06:11:38.768 INFO:tasks.workunit.client.0.smithi016.stdout:./263/.snap/snap-subdir-test 2019-12-05T06:11:38.778 INFO:tasks.workunit.client.0.smithi016.stdout:./130/.snap/snap-subdir-test 2019-12-05T06:11:38.789 INFO:tasks.workunit.client.0.smithi016.stdout:./193/.snap/snap-subdir-test 2019-12-05T06:11:38.798 INFO:tasks.workunit.client.0.smithi016.stdout:./223/.snap/snap-subdir-test 2019-12-05T06:11:38.808 INFO:tasks.workunit.client.0.smithi016.stdout:./207/.snap/snap-subdir-test 2019-12-05T06:11:38.818 INFO:tasks.workunit.client.0.smithi016.stdout:./276/.snap/snap-subdir-test 2019-12-05T06:11:38.827 INFO:tasks.workunit.client.0.smithi016.stdout:./271/.snap/snap-subdir-test 2019-12-05T06:11:38.837 INFO:tasks.workunit.client.0.smithi016.stdout:./251/.snap/snap-subdir-test 2019-12-05T06:11:39.598 INFO:tasks.workunit.client.0.smithi016.stdout:./175/.snap/snap-subdir-test 2019-12-05T06:11:39.620 INFO:tasks.workunit.client.0.smithi016.stdout:./222/.snap/snap-subdir-test 2019-12-05T06:11:39.640 INFO:tasks.workunit.client.0.smithi016.stdout:./220/.snap/snap-subdir-test 2019-12-05T06:11:39.658 INFO:tasks.workunit.client.0.smithi016.stdout:./203/.snap/snap-subdir-test 2019-12-05T06:11:39.677 INFO:tasks.workunit.client.0.smithi016.stdout:./181/.snap/snap-subdir-test 2019-12-05T06:11:39.692 INFO:tasks.workunit.client.0.smithi016.stdout:./286/.snap/snap-subdir-test 2019-12-05T06:11:39.709 INFO:tasks.workunit.client.0.smithi016.stdout:./127/.snap/snap-subdir-test 2019-12-05T06:11:39.725 INFO:tasks.workunit.client.0.smithi016.stdout:./108/.snap/snap-subdir-test 2019-12-05T06:11:39.739 INFO:tasks.workunit.client.0.smithi016.stdout:./183/.snap/snap-subdir-test 2019-12-05T06:11:39.752 INFO:tasks.workunit.client.0.smithi016.stdout:./294/.snap/snap-subdir-test 2019-12-05T06:11:39.765 INFO:tasks.workunit.client.0.smithi016.stdout:./235/.snap/snap-subdir-test 2019-12-05T06:11:39.781 INFO:tasks.workunit.client.0.smithi016.stdout:./177/.snap/snap-subdir-test 2019-12-05T06:11:39.791 INFO:tasks.workunit.client.0.smithi016.stdout:./169/.snap/snap-subdir-test 2019-12-05T06:11:39.802 INFO:tasks.workunit.client.0.smithi016.stdout:./212/.snap/snap-subdir-test 2019-12-05T06:11:39.814 INFO:tasks.workunit.client.0.smithi016.stdout:./154/.snap/snap-subdir-test 2019-12-05T06:11:39.825 INFO:tasks.workunit.client.0.smithi016.stdout:./171/.snap/snap-subdir-test 2019-12-05T06:11:39.836 INFO:tasks.workunit.client.0.smithi016.stdout:./139/.snap/snap-subdir-test 2019-12-05T06:11:39.847 INFO:tasks.workunit.client.0.smithi016.stdout:./102/.snap/snap-subdir-test 2019-12-05T06:11:39.857 INFO:tasks.workunit.client.0.smithi016.stdout:./129/.snap/snap-subdir-test 2019-12-05T06:11:48.518 INFO:tasks.workunit.client.0.smithi016.stdout:Delete all the files and directories ... 2019-12-05T06:11:52.384 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/19’ 2019-12-05T06:11:52.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/15’ 2019-12-05T06:11:52.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/31’ 2019-12-05T06:11:52.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/36’ 2019-12-05T06:11:52.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/20’ 2019-12-05T06:11:52.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/12’ 2019-12-05T06:11:52.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/39’ 2019-12-05T06:11:52.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/10’ 2019-12-05T06:11:52.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/34’ 2019-12-05T06:11:52.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/38’ 2019-12-05T06:11:52.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/11’ 2019-12-05T06:11:52.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/16’ 2019-12-05T06:11:52.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/32’ 2019-12-05T06:11:52.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/30’ 2019-12-05T06:11:52.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/14’ 2019-12-05T06:11:52.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/18’ 2019-12-05T06:11:52.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/13’ 2019-12-05T06:11:52.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/33’ 2019-12-05T06:11:52.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/17’ 2019-12-05T06:11:52.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/35’ 2019-12-05T06:11:52.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./100/37’ 2019-12-05T06:11:52.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./100’ 2019-12-05T06:11:52.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/19’ 2019-12-05T06:11:52.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/15’ 2019-12-05T06:11:52.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/31’ 2019-12-05T06:11:52.389 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/36’ 2019-12-05T06:11:52.389 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/20’ 2019-12-05T06:11:52.389 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/12’ 2019-12-05T06:11:52.389 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/39’ 2019-12-05T06:11:52.389 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/10’ 2019-12-05T06:11:52.389 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/34’ 2019-12-05T06:11:52.390 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/38’ 2019-12-05T06:11:52.390 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/11’ 2019-12-05T06:11:52.390 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/16’ 2019-12-05T06:11:52.390 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/32’ 2019-12-05T06:11:52.390 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/30’ 2019-12-05T06:11:52.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/14’ 2019-12-05T06:11:52.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/18’ 2019-12-05T06:11:52.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/13’ 2019-12-05T06:11:52.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/33’ 2019-12-05T06:11:52.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/17’ 2019-12-05T06:11:52.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/35’ 2019-12-05T06:11:52.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./101/37’ 2019-12-05T06:11:52.392 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./101’ 2019-12-05T06:11:52.392 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/19’ 2019-12-05T06:11:52.392 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/15’ 2019-12-05T06:11:52.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/31’ 2019-12-05T06:11:52.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/36’ 2019-12-05T06:11:52.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/20’ 2019-12-05T06:11:52.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/12’ 2019-12-05T06:11:52.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/39’ 2019-12-05T06:11:52.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/10’ 2019-12-05T06:11:52.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/34’ 2019-12-05T06:11:52.394 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/38’ 2019-12-05T06:11:52.394 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/11’ 2019-12-05T06:11:52.394 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/16’ 2019-12-05T06:11:52.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/32’ 2019-12-05T06:11:52.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/30’ 2019-12-05T06:11:52.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/14’ 2019-12-05T06:11:52.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/18’ 2019-12-05T06:11:52.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/13’ 2019-12-05T06:11:52.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/33’ 2019-12-05T06:11:52.396 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/17’ 2019-12-05T06:11:52.396 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/35’ 2019-12-05T06:11:52.396 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./102/37’ 2019-12-05T06:11:52.396 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./102’ 2019-12-05T06:11:52.396 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/19’ 2019-12-05T06:11:52.396 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/15’ 2019-12-05T06:11:52.396 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/31’ 2019-12-05T06:11:52.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/36’ 2019-12-05T06:11:52.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/20’ 2019-12-05T06:11:52.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/12’ 2019-12-05T06:11:52.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/39’ 2019-12-05T06:11:52.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/10’ 2019-12-05T06:11:52.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/34’ 2019-12-05T06:11:52.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/38’ 2019-12-05T06:11:52.398 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/11’ 2019-12-05T06:11:52.398 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/16’ 2019-12-05T06:11:52.398 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/32’ 2019-12-05T06:11:52.398 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/30’ 2019-12-05T06:11:52.398 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/14’ 2019-12-05T06:11:52.398 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/18’ 2019-12-05T06:11:52.398 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/13’ 2019-12-05T06:11:52.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/33’ 2019-12-05T06:11:52.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/17’ 2019-12-05T06:11:52.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/35’ 2019-12-05T06:11:52.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./103/37’ 2019-12-05T06:11:52.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./103’ 2019-12-05T06:11:52.400 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/19’ 2019-12-05T06:11:52.400 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/15’ 2019-12-05T06:11:52.400 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/31’ 2019-12-05T06:11:52.400 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/36’ 2019-12-05T06:11:52.400 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/20’ 2019-12-05T06:11:52.400 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/12’ 2019-12-05T06:11:52.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/39’ 2019-12-05T06:11:52.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/10’ 2019-12-05T06:11:52.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/34’ 2019-12-05T06:11:52.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/38’ 2019-12-05T06:11:52.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/11’ 2019-12-05T06:11:52.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/16’ 2019-12-05T06:11:52.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/32’ 2019-12-05T06:11:52.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/30’ 2019-12-05T06:11:52.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/14’ 2019-12-05T06:11:52.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/18’ 2019-12-05T06:11:52.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/13’ 2019-12-05T06:11:52.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/33’ 2019-12-05T06:11:52.405 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/17’ 2019-12-05T06:11:52.405 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/35’ 2019-12-05T06:11:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./104/37’ 2019-12-05T06:11:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./104’ 2019-12-05T06:11:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/19’ 2019-12-05T06:11:52.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/15’ 2019-12-05T06:11:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/31’ 2019-12-05T06:11:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/36’ 2019-12-05T06:11:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/20’ 2019-12-05T06:11:52.407 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/12’ 2019-12-05T06:11:52.408 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/39’ 2019-12-05T06:11:52.408 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/10’ 2019-12-05T06:11:52.408 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/34’ 2019-12-05T06:11:52.409 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/38’ 2019-12-05T06:11:52.409 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/11’ 2019-12-05T06:11:52.409 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/16’ 2019-12-05T06:11:52.410 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/32’ 2019-12-05T06:11:52.410 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/30’ 2019-12-05T06:11:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/14’ 2019-12-05T06:11:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/18’ 2019-12-05T06:11:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/13’ 2019-12-05T06:11:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/33’ 2019-12-05T06:11:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/17’ 2019-12-05T06:11:52.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/35’ 2019-12-05T06:11:52.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./105/37’ 2019-12-05T06:11:52.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./105’ 2019-12-05T06:11:52.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/19’ 2019-12-05T06:11:52.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/15’ 2019-12-05T06:11:52.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/31’ 2019-12-05T06:11:52.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/36’ 2019-12-05T06:11:52.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/20’ 2019-12-05T06:11:52.413 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/12’ 2019-12-05T06:11:52.413 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/39’ 2019-12-05T06:11:52.414 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/10’ 2019-12-05T06:11:52.414 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/34’ 2019-12-05T06:11:52.414 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/38’ 2019-12-05T06:11:52.414 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/11’ 2019-12-05T06:11:52.414 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/16’ 2019-12-05T06:11:52.414 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/32’ 2019-12-05T06:11:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/30’ 2019-12-05T06:11:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/14’ 2019-12-05T06:11:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/18’ 2019-12-05T06:11:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/13’ 2019-12-05T06:11:52.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/33’ 2019-12-05T06:11:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/17’ 2019-12-05T06:11:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/35’ 2019-12-05T06:11:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./106/37’ 2019-12-05T06:11:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./106’ 2019-12-05T06:11:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/19’ 2019-12-05T06:11:52.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/15’ 2019-12-05T06:11:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/31’ 2019-12-05T06:11:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/36’ 2019-12-05T06:11:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/20’ 2019-12-05T06:11:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/12’ 2019-12-05T06:11:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/39’ 2019-12-05T06:11:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/10’ 2019-12-05T06:11:52.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/34’ 2019-12-05T06:11:52.418 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/38’ 2019-12-05T06:11:52.418 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/11’ 2019-12-05T06:11:52.418 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/16’ 2019-12-05T06:11:52.419 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/32’ 2019-12-05T06:11:52.419 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/30’ 2019-12-05T06:11:52.419 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/14’ 2019-12-05T06:11:52.419 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/18’ 2019-12-05T06:11:52.419 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/13’ 2019-12-05T06:11:52.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/33’ 2019-12-05T06:11:52.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/17’ 2019-12-05T06:11:52.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/35’ 2019-12-05T06:11:52.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./107/37’ 2019-12-05T06:11:55.314 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./107’ 2019-12-05T06:11:55.315 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/19’ 2019-12-05T06:11:55.315 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/15’ 2019-12-05T06:11:55.315 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/31’ 2019-12-05T06:11:55.316 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/36’ 2019-12-05T06:11:55.316 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/20’ 2019-12-05T06:11:55.316 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/12’ 2019-12-05T06:11:55.316 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/39’ 2019-12-05T06:11:55.319 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/10’ 2019-12-05T06:11:55.319 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/34’ 2019-12-05T06:11:55.319 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/38’ 2019-12-05T06:11:55.320 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/11’ 2019-12-05T06:11:55.320 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/16’ 2019-12-05T06:11:55.321 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/32’ 2019-12-05T06:11:55.321 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/30’ 2019-12-05T06:11:55.321 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/14’ 2019-12-05T06:11:55.321 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/18’ 2019-12-05T06:11:55.322 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/13’ 2019-12-05T06:11:55.322 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/33’ 2019-12-05T06:11:55.322 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/17’ 2019-12-05T06:11:55.322 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/35’ 2019-12-05T06:11:55.323 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./108/37’ 2019-12-05T06:11:55.323 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./108’ 2019-12-05T06:11:55.323 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/19’ 2019-12-05T06:11:55.323 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/15’ 2019-12-05T06:11:55.324 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/31’ 2019-12-05T06:11:55.324 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/36’ 2019-12-05T06:11:55.324 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/20’ 2019-12-05T06:11:55.324 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/12’ 2019-12-05T06:11:55.325 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/39’ 2019-12-05T06:11:55.325 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/10’ 2019-12-05T06:11:55.325 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/34’ 2019-12-05T06:11:55.325 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/38’ 2019-12-05T06:11:55.327 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/11’ 2019-12-05T06:11:55.328 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/16’ 2019-12-05T06:11:55.328 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/32’ 2019-12-05T06:11:55.328 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/30’ 2019-12-05T06:11:55.328 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/14’ 2019-12-05T06:11:55.329 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/18’ 2019-12-05T06:11:55.329 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/13’ 2019-12-05T06:11:55.329 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/33’ 2019-12-05T06:11:55.329 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/17’ 2019-12-05T06:11:55.330 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/35’ 2019-12-05T06:11:55.330 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./109/37’ 2019-12-05T06:11:55.330 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./109’ 2019-12-05T06:11:55.330 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/19’ 2019-12-05T06:11:55.330 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/15’ 2019-12-05T06:11:55.330 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/31’ 2019-12-05T06:11:55.330 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/36’ 2019-12-05T06:11:55.331 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/20’ 2019-12-05T06:11:55.331 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/12’ 2019-12-05T06:11:55.331 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/39’ 2019-12-05T06:11:55.331 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/10’ 2019-12-05T06:11:55.331 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/34’ 2019-12-05T06:11:55.331 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/38’ 2019-12-05T06:11:55.331 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/11’ 2019-12-05T06:11:55.332 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/16’ 2019-12-05T06:11:55.332 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/32’ 2019-12-05T06:11:55.332 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/30’ 2019-12-05T06:11:55.332 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/14’ 2019-12-05T06:11:55.332 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/18’ 2019-12-05T06:11:55.332 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/13’ 2019-12-05T06:11:55.333 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/33’ 2019-12-05T06:11:55.333 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/17’ 2019-12-05T06:11:55.333 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/35’ 2019-12-05T06:11:55.333 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./110/37’ 2019-12-05T06:11:55.334 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./110’ 2019-12-05T06:11:55.334 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/19’ 2019-12-05T06:11:55.334 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/15’ 2019-12-05T06:11:55.334 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/31’ 2019-12-05T06:11:55.334 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/36’ 2019-12-05T06:11:55.334 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/20’ 2019-12-05T06:11:55.334 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/12’ 2019-12-05T06:11:55.335 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/39’ 2019-12-05T06:11:55.335 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/10’ 2019-12-05T06:11:55.335 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/34’ 2019-12-05T06:11:55.335 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/38’ 2019-12-05T06:11:55.335 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/11’ 2019-12-05T06:11:55.335 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/16’ 2019-12-05T06:11:55.335 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/32’ 2019-12-05T06:11:55.336 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/30’ 2019-12-05T06:11:55.336 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/14’ 2019-12-05T06:11:55.336 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/18’ 2019-12-05T06:11:55.336 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/13’ 2019-12-05T06:11:55.336 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/33’ 2019-12-05T06:11:55.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/17’ 2019-12-05T06:11:55.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/35’ 2019-12-05T06:11:55.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./111/37’ 2019-12-05T06:11:55.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./111’ 2019-12-05T06:11:55.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/19’ 2019-12-05T06:11:55.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/15’ 2019-12-05T06:11:55.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/31’ 2019-12-05T06:11:55.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/36’ 2019-12-05T06:11:55.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/20’ 2019-12-05T06:11:55.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/12’ 2019-12-05T06:11:55.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/39’ 2019-12-05T06:11:55.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/10’ 2019-12-05T06:11:55.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/34’ 2019-12-05T06:11:55.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/38’ 2019-12-05T06:11:55.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/11’ 2019-12-05T06:11:55.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/16’ 2019-12-05T06:11:55.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/32’ 2019-12-05T06:11:55.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/30’ 2019-12-05T06:11:55.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/14’ 2019-12-05T06:11:55.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/18’ 2019-12-05T06:11:55.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/13’ 2019-12-05T06:11:55.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/33’ 2019-12-05T06:11:55.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/17’ 2019-12-05T06:11:55.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/35’ 2019-12-05T06:11:55.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./112/37’ 2019-12-05T06:11:55.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./112’ 2019-12-05T06:11:55.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/19’ 2019-12-05T06:11:55.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/15’ 2019-12-05T06:11:55.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/31’ 2019-12-05T06:11:55.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/36’ 2019-12-05T06:11:55.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/20’ 2019-12-05T06:11:55.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/12’ 2019-12-05T06:11:55.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/39’ 2019-12-05T06:11:55.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/10’ 2019-12-05T06:11:55.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/34’ 2019-12-05T06:11:55.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/38’ 2019-12-05T06:11:55.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/11’ 2019-12-05T06:11:55.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/16’ 2019-12-05T06:11:55.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/32’ 2019-12-05T06:11:55.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/30’ 2019-12-05T06:11:55.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/14’ 2019-12-05T06:11:55.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/18’ 2019-12-05T06:11:55.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/13’ 2019-12-05T06:11:55.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/33’ 2019-12-05T06:11:55.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/17’ 2019-12-05T06:11:55.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/35’ 2019-12-05T06:11:55.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./113/37’ 2019-12-05T06:11:55.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./113’ 2019-12-05T06:11:55.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/19’ 2019-12-05T06:11:55.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/15’ 2019-12-05T06:11:55.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/31’ 2019-12-05T06:11:55.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/36’ 2019-12-05T06:11:55.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/20’ 2019-12-05T06:11:55.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/12’ 2019-12-05T06:11:55.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/39’ 2019-12-05T06:11:55.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/10’ 2019-12-05T06:11:55.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/34’ 2019-12-05T06:11:55.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/38’ 2019-12-05T06:11:55.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/11’ 2019-12-05T06:11:55.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/16’ 2019-12-05T06:11:55.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/32’ 2019-12-05T06:11:55.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/30’ 2019-12-05T06:11:55.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/14’ 2019-12-05T06:11:55.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/18’ 2019-12-05T06:11:55.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/13’ 2019-12-05T06:11:55.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/33’ 2019-12-05T06:11:55.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/17’ 2019-12-05T06:11:55.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/35’ 2019-12-05T06:11:55.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./114/37’ 2019-12-05T06:11:55.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./114’ 2019-12-05T06:11:55.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/19’ 2019-12-05T06:11:55.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/15’ 2019-12-05T06:11:55.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/31’ 2019-12-05T06:11:55.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/36’ 2019-12-05T06:11:55.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/20’ 2019-12-05T06:11:55.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/12’ 2019-12-05T06:11:55.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/39’ 2019-12-05T06:11:55.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/10’ 2019-12-05T06:11:55.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/34’ 2019-12-05T06:11:55.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/38’ 2019-12-05T06:11:55.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/11’ 2019-12-05T06:11:55.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/16’ 2019-12-05T06:11:55.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/32’ 2019-12-05T06:11:55.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/30’ 2019-12-05T06:11:55.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/14’ 2019-12-05T06:11:55.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/18’ 2019-12-05T06:11:55.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/13’ 2019-12-05T06:11:55.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/33’ 2019-12-05T06:11:55.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/17’ 2019-12-05T06:11:55.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/35’ 2019-12-05T06:11:55.949 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./115/37’ 2019-12-05T06:11:55.949 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./115’ 2019-12-05T06:11:55.949 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/19’ 2019-12-05T06:11:55.950 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/15’ 2019-12-05T06:11:55.950 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/31’ 2019-12-05T06:11:55.950 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/36’ 2019-12-05T06:11:55.950 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/20’ 2019-12-05T06:11:55.950 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/12’ 2019-12-05T06:11:55.950 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/39’ 2019-12-05T06:11:55.950 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/10’ 2019-12-05T06:11:55.951 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/34’ 2019-12-05T06:11:55.951 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/38’ 2019-12-05T06:11:55.951 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/11’ 2019-12-05T06:11:55.951 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/16’ 2019-12-05T06:11:55.951 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/32’ 2019-12-05T06:11:55.951 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/30’ 2019-12-05T06:11:55.952 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/14’ 2019-12-05T06:11:55.952 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/18’ 2019-12-05T06:11:55.952 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/13’ 2019-12-05T06:11:55.952 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/33’ 2019-12-05T06:11:55.952 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/17’ 2019-12-05T06:11:55.952 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/35’ 2019-12-05T06:11:55.952 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./116/37’ 2019-12-05T06:11:55.953 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./116’ 2019-12-05T06:11:55.953 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/19’ 2019-12-05T06:11:55.953 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/15’ 2019-12-05T06:11:55.953 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/31’ 2019-12-05T06:11:55.953 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/36’ 2019-12-05T06:11:55.953 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/20’ 2019-12-05T06:11:55.953 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/12’ 2019-12-05T06:11:55.954 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/39’ 2019-12-05T06:11:55.954 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/10’ 2019-12-05T06:11:55.954 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/34’ 2019-12-05T06:11:55.954 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/38’ 2019-12-05T06:11:55.954 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/11’ 2019-12-05T06:11:55.954 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/16’ 2019-12-05T06:11:55.954 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/32’ 2019-12-05T06:11:55.955 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/30’ 2019-12-05T06:11:55.955 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/14’ 2019-12-05T06:11:55.955 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/18’ 2019-12-05T06:11:55.955 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/13’ 2019-12-05T06:11:55.955 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/33’ 2019-12-05T06:11:55.955 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/17’ 2019-12-05T06:11:55.955 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/35’ 2019-12-05T06:11:55.956 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./117/37’ 2019-12-05T06:11:55.956 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./117’ 2019-12-05T06:11:55.956 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/19’ 2019-12-05T06:11:55.956 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/15’ 2019-12-05T06:11:55.956 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/31’ 2019-12-05T06:11:55.956 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/36’ 2019-12-05T06:11:55.956 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/20’ 2019-12-05T06:11:55.957 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/12’ 2019-12-05T06:11:55.957 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/39’ 2019-12-05T06:11:55.957 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/10’ 2019-12-05T06:11:55.957 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/34’ 2019-12-05T06:11:55.957 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/38’ 2019-12-05T06:11:55.957 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/11’ 2019-12-05T06:11:55.957 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/16’ 2019-12-05T06:11:55.958 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/32’ 2019-12-05T06:11:55.958 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/30’ 2019-12-05T06:11:55.958 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/14’ 2019-12-05T06:11:55.958 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/18’ 2019-12-05T06:11:55.958 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/13’ 2019-12-05T06:11:55.958 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/33’ 2019-12-05T06:11:55.958 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/17’ 2019-12-05T06:11:55.959 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/35’ 2019-12-05T06:11:55.959 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./118/37’ 2019-12-05T06:11:55.959 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./118’ 2019-12-05T06:11:55.959 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/19’ 2019-12-05T06:11:55.959 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/15’ 2019-12-05T06:11:55.959 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/31’ 2019-12-05T06:11:55.959 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/36’ 2019-12-05T06:11:55.960 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/20’ 2019-12-05T06:11:55.960 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/12’ 2019-12-05T06:11:55.960 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/39’ 2019-12-05T06:11:55.960 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/10’ 2019-12-05T06:11:55.960 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/34’ 2019-12-05T06:11:55.960 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/38’ 2019-12-05T06:11:55.961 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/11’ 2019-12-05T06:11:55.961 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/16’ 2019-12-05T06:11:55.961 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/32’ 2019-12-05T06:11:55.961 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/30’ 2019-12-05T06:11:55.961 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/14’ 2019-12-05T06:11:55.961 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/18’ 2019-12-05T06:11:55.961 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/13’ 2019-12-05T06:11:55.962 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/33’ 2019-12-05T06:11:55.962 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/17’ 2019-12-05T06:11:55.962 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/35’ 2019-12-05T06:11:55.962 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./119/37’ 2019-12-05T06:11:55.962 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./119’ 2019-12-05T06:11:55.962 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/19’ 2019-12-05T06:11:55.962 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/15’ 2019-12-05T06:11:55.963 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/31’ 2019-12-05T06:11:55.963 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/36’ 2019-12-05T06:11:55.963 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/20’ 2019-12-05T06:11:55.963 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/12’ 2019-12-05T06:11:55.963 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/39’ 2019-12-05T06:11:55.963 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/10’ 2019-12-05T06:11:55.964 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/34’ 2019-12-05T06:11:55.964 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/38’ 2019-12-05T06:11:55.964 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/11’ 2019-12-05T06:11:55.964 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/16’ 2019-12-05T06:11:55.964 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/32’ 2019-12-05T06:11:55.964 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/30’ 2019-12-05T06:11:55.965 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/14’ 2019-12-05T06:11:55.965 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/18’ 2019-12-05T06:11:55.965 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/13’ 2019-12-05T06:11:55.965 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/33’ 2019-12-05T06:11:55.965 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/17’ 2019-12-05T06:11:55.965 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/35’ 2019-12-05T06:11:55.965 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./120/37’ 2019-12-05T06:11:55.966 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./120’ 2019-12-05T06:11:55.966 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/19’ 2019-12-05T06:11:55.966 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/15’ 2019-12-05T06:11:55.966 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/31’ 2019-12-05T06:11:55.966 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/36’ 2019-12-05T06:11:55.966 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/20’ 2019-12-05T06:11:55.967 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/12’ 2019-12-05T06:11:55.967 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/39’ 2019-12-05T06:11:55.967 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/10’ 2019-12-05T06:11:55.967 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/34’ 2019-12-05T06:11:55.967 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/38’ 2019-12-05T06:11:55.967 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/11’ 2019-12-05T06:11:55.968 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/16’ 2019-12-05T06:11:55.968 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/32’ 2019-12-05T06:11:55.968 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/30’ 2019-12-05T06:11:55.968 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/14’ 2019-12-05T06:11:55.968 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/18’ 2019-12-05T06:11:55.968 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/13’ 2019-12-05T06:11:55.968 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/33’ 2019-12-05T06:11:55.969 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/17’ 2019-12-05T06:11:55.969 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/35’ 2019-12-05T06:11:55.969 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./121/37’ 2019-12-05T06:11:55.969 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./121’ 2019-12-05T06:11:55.969 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/19’ 2019-12-05T06:11:55.969 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/15’ 2019-12-05T06:11:55.969 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/31’ 2019-12-05T06:11:55.970 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/36’ 2019-12-05T06:11:55.970 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/20’ 2019-12-05T06:11:55.971 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/12’ 2019-12-05T06:11:55.971 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/39’ 2019-12-05T06:11:55.971 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/10’ 2019-12-05T06:11:55.971 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/34’ 2019-12-05T06:11:55.971 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/38’ 2019-12-05T06:11:55.971 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/11’ 2019-12-05T06:11:55.972 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/16’ 2019-12-05T06:11:55.972 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/32’ 2019-12-05T06:11:55.972 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/30’ 2019-12-05T06:11:55.973 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/14’ 2019-12-05T06:11:55.973 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/18’ 2019-12-05T06:11:55.973 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/13’ 2019-12-05T06:11:55.973 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/33’ 2019-12-05T06:11:55.974 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/17’ 2019-12-05T06:11:55.974 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/35’ 2019-12-05T06:11:55.974 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./122/37’ 2019-12-05T06:11:55.974 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./122’ 2019-12-05T06:11:55.975 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/19’ 2019-12-05T06:11:55.975 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/15’ 2019-12-05T06:11:55.975 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/31’ 2019-12-05T06:11:55.975 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/36’ 2019-12-05T06:11:55.975 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/20’ 2019-12-05T06:11:55.975 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/12’ 2019-12-05T06:11:55.975 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/39’ 2019-12-05T06:11:55.976 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/10’ 2019-12-05T06:11:55.976 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/34’ 2019-12-05T06:11:55.976 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/38’ 2019-12-05T06:11:55.976 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/11’ 2019-12-05T06:11:55.977 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/16’ 2019-12-05T06:11:55.978 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/32’ 2019-12-05T06:11:55.978 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/30’ 2019-12-05T06:11:55.978 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/14’ 2019-12-05T06:11:55.978 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/18’ 2019-12-05T06:11:55.979 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/13’ 2019-12-05T06:11:55.979 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/33’ 2019-12-05T06:11:55.979 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/17’ 2019-12-05T06:11:55.980 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/35’ 2019-12-05T06:11:58.007 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./123/37’ 2019-12-05T06:11:58.007 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./123’ 2019-12-05T06:11:58.007 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/19’ 2019-12-05T06:11:58.007 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/15’ 2019-12-05T06:11:58.007 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/31’ 2019-12-05T06:11:58.008 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/36’ 2019-12-05T06:11:58.008 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/20’ 2019-12-05T06:11:58.008 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/12’ 2019-12-05T06:11:58.008 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/39’ 2019-12-05T06:11:58.009 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/10’ 2019-12-05T06:11:58.009 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/34’ 2019-12-05T06:11:58.009 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/38’ 2019-12-05T06:11:58.009 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/11’ 2019-12-05T06:11:58.009 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/16’ 2019-12-05T06:11:58.009 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/32’ 2019-12-05T06:11:58.010 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/30’ 2019-12-05T06:11:58.010 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/14’ 2019-12-05T06:11:58.010 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/18’ 2019-12-05T06:11:58.010 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/13’ 2019-12-05T06:11:58.010 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/33’ 2019-12-05T06:11:58.010 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/17’ 2019-12-05T06:11:58.010 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/35’ 2019-12-05T06:11:58.011 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./124/37’ 2019-12-05T06:11:58.011 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./124’ 2019-12-05T06:11:58.011 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/19’ 2019-12-05T06:11:58.011 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/15’ 2019-12-05T06:11:58.011 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/31’ 2019-12-05T06:11:58.011 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/36’ 2019-12-05T06:11:58.012 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/20’ 2019-12-05T06:11:58.012 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/12’ 2019-12-05T06:11:58.012 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/39’ 2019-12-05T06:11:58.012 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/10’ 2019-12-05T06:11:58.012 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/34’ 2019-12-05T06:11:58.012 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/38’ 2019-12-05T06:11:58.012 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/11’ 2019-12-05T06:11:58.013 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/16’ 2019-12-05T06:11:58.013 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/32’ 2019-12-05T06:11:58.013 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/30’ 2019-12-05T06:11:58.013 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/14’ 2019-12-05T06:11:58.013 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/18’ 2019-12-05T06:11:58.013 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/13’ 2019-12-05T06:11:58.013 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/33’ 2019-12-05T06:11:58.014 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/17’ 2019-12-05T06:11:58.014 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/35’ 2019-12-05T06:11:58.014 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./125/37’ 2019-12-05T06:11:58.014 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./125’ 2019-12-05T06:11:58.014 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/19’ 2019-12-05T06:11:58.014 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/15’ 2019-12-05T06:11:58.014 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/31’ 2019-12-05T06:11:58.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/36’ 2019-12-05T06:11:58.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/20’ 2019-12-05T06:11:58.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/12’ 2019-12-05T06:11:58.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/39’ 2019-12-05T06:11:58.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/10’ 2019-12-05T06:11:58.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/34’ 2019-12-05T06:11:58.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/38’ 2019-12-05T06:11:58.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/11’ 2019-12-05T06:11:58.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/16’ 2019-12-05T06:11:58.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/32’ 2019-12-05T06:11:58.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/30’ 2019-12-05T06:11:58.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/14’ 2019-12-05T06:11:58.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/18’ 2019-12-05T06:11:58.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/13’ 2019-12-05T06:11:58.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/33’ 2019-12-05T06:11:58.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/17’ 2019-12-05T06:11:58.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/35’ 2019-12-05T06:11:58.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./126/37’ 2019-12-05T06:11:58.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./126’ 2019-12-05T06:11:58.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/19’ 2019-12-05T06:11:58.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/15’ 2019-12-05T06:11:58.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/31’ 2019-12-05T06:11:58.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/36’ 2019-12-05T06:11:58.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/20’ 2019-12-05T06:11:58.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/12’ 2019-12-05T06:11:58.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/39’ 2019-12-05T06:11:58.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/10’ 2019-12-05T06:11:58.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/34’ 2019-12-05T06:11:58.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/38’ 2019-12-05T06:11:58.020 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/11’ 2019-12-05T06:11:58.020 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/16’ 2019-12-05T06:11:58.020 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/32’ 2019-12-05T06:11:58.020 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/30’ 2019-12-05T06:11:58.020 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/14’ 2019-12-05T06:11:58.020 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/18’ 2019-12-05T06:11:58.021 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/13’ 2019-12-05T06:11:58.021 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/33’ 2019-12-05T06:11:58.021 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/17’ 2019-12-05T06:11:58.021 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/35’ 2019-12-05T06:11:58.021 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./127/37’ 2019-12-05T06:11:58.022 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./127’ 2019-12-05T06:11:58.022 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/19’ 2019-12-05T06:11:58.022 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/15’ 2019-12-05T06:11:58.022 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/31’ 2019-12-05T06:11:58.022 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/36’ 2019-12-05T06:11:58.022 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/20’ 2019-12-05T06:11:58.022 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/12’ 2019-12-05T06:11:58.023 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/39’ 2019-12-05T06:11:58.023 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/10’ 2019-12-05T06:11:58.023 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/34’ 2019-12-05T06:11:58.023 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/38’ 2019-12-05T06:11:58.023 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/11’ 2019-12-05T06:11:58.023 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/16’ 2019-12-05T06:11:58.023 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/32’ 2019-12-05T06:11:58.024 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/30’ 2019-12-05T06:11:58.024 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/14’ 2019-12-05T06:11:58.024 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/18’ 2019-12-05T06:11:58.024 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/13’ 2019-12-05T06:11:58.024 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/33’ 2019-12-05T06:11:58.024 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/17’ 2019-12-05T06:11:58.024 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/35’ 2019-12-05T06:11:58.025 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./128/37’ 2019-12-05T06:11:58.025 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./128’ 2019-12-05T06:11:58.025 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/19’ 2019-12-05T06:11:58.025 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/15’ 2019-12-05T06:11:58.025 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/31’ 2019-12-05T06:11:58.025 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/36’ 2019-12-05T06:11:58.025 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/20’ 2019-12-05T06:11:58.026 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/12’ 2019-12-05T06:11:58.026 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/39’ 2019-12-05T06:11:58.026 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/10’ 2019-12-05T06:11:58.026 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/34’ 2019-12-05T06:11:58.026 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/38’ 2019-12-05T06:11:58.026 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/11’ 2019-12-05T06:11:58.026 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/16’ 2019-12-05T06:11:58.027 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/32’ 2019-12-05T06:11:58.027 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/30’ 2019-12-05T06:11:58.027 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/14’ 2019-12-05T06:11:58.027 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/18’ 2019-12-05T06:11:58.027 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/13’ 2019-12-05T06:11:58.027 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/33’ 2019-12-05T06:11:58.027 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/17’ 2019-12-05T06:11:58.028 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/35’ 2019-12-05T06:11:58.028 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./129/37’ 2019-12-05T06:11:58.028 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./129’ 2019-12-05T06:11:58.028 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/19’ 2019-12-05T06:11:58.029 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/15’ 2019-12-05T06:11:58.029 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/31’ 2019-12-05T06:11:58.029 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/36’ 2019-12-05T06:11:58.029 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/20’ 2019-12-05T06:11:58.030 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/12’ 2019-12-05T06:11:58.030 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/39’ 2019-12-05T06:11:58.030 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/10’ 2019-12-05T06:11:58.030 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/34’ 2019-12-05T06:11:58.030 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/38’ 2019-12-05T06:11:58.030 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/11’ 2019-12-05T06:11:58.030 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/16’ 2019-12-05T06:11:58.031 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/32’ 2019-12-05T06:11:58.031 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/30’ 2019-12-05T06:11:58.031 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/14’ 2019-12-05T06:11:58.031 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/18’ 2019-12-05T06:11:58.031 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/13’ 2019-12-05T06:11:58.031 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/33’ 2019-12-05T06:11:58.031 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/17’ 2019-12-05T06:11:58.032 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/35’ 2019-12-05T06:11:58.032 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./130/37’ 2019-12-05T06:11:58.032 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./130’ 2019-12-05T06:11:58.032 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/19’ 2019-12-05T06:11:58.032 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/15’ 2019-12-05T06:11:58.032 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/31’ 2019-12-05T06:11:58.032 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/36’ 2019-12-05T06:11:58.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/20’ 2019-12-05T06:11:58.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/12’ 2019-12-05T06:11:58.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/39’ 2019-12-05T06:11:58.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/10’ 2019-12-05T06:11:58.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/34’ 2019-12-05T06:11:58.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/38’ 2019-12-05T06:11:58.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/11’ 2019-12-05T06:11:58.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/16’ 2019-12-05T06:11:58.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/32’ 2019-12-05T06:11:58.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/30’ 2019-12-05T06:11:58.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/14’ 2019-12-05T06:11:58.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/18’ 2019-12-05T06:11:58.035 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/13’ 2019-12-05T06:11:58.036 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/33’ 2019-12-05T06:11:58.036 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/17’ 2019-12-05T06:11:59.239 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:11:59.267 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:11:59.303 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:11:59.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.451+0000 7f3ab8b3e700 1 Processor -- start 2019-12-05T06:11:59.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.451+0000 7f3ab8b3e700 1 -- start start 2019-12-05T06:11:59.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.452+0000 7f3ab8b3e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ab408fef0 0x7f3ab408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:11:59.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.452+0000 7f3ab8b3e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab40958e0 con 0x7f3ab408d2c0 2019-12-05T06:11:59.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.452+0000 7f3ab8b3e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab40629e0 con 0x7f3ab408c6d0 2019-12-05T06:11:59.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.452+0000 7f3ab8b3e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab408df90 con 0x7f3ab408fef0 2019-12-05T06:11:59.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.452+0000 7f3ab259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3ab408d2c0 0x7f3ab408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33920/0 (socket says 172.21.15.16:33920) 2019-12-05T06:11:59.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.452+0000 7f3ab259c700 1 -- 172.21.15.16:0/842053321 learned_addr learned my addr 172.21.15.16:0/842053321 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:11:59.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.452+0000 7f3ab359e700 1 --2- 172.21.15.16:0/842053321 >> v2:172.21.15.16:3300/0 conn(0x7f3ab408fef0 0x7f3ab408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:59.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.453+0000 7f3ab1d9b700 1 -- 172.21.15.16:0/842053321 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4208356906 0 0) 0x7f3a9c000ff0 con 0x7f3ab408d2c0 2019-12-05T06:11:59.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.453+0000 7f3ab1d9b700 1 -- 172.21.15.16:0/842053321 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3aa0002e80 con 0x7f3ab408d2c0 2019-12-05T06:11:59.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.454+0000 7f3ab359e700 1 -- 172.21.15.16:0/842053321 >> v1:172.21.15.76:6789/0 conn(0x7f3ab408c6d0 legacy=0x7f3ab408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.454+0000 7f3ab359e700 1 -- 172.21.15.16:0/842053321 >> v1:172.21.15.16:6789/0 conn(0x7f3ab408d2c0 legacy=0x7f3ab408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.454+0000 7f3ab1d9b700 1 -- 172.21.15.16:0/842053321 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 752133050 0 0) 0x7f3a9c000fd0 con 0x7f3ab408d2c0 2019-12-05T06:11:59.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.454+0000 7f3ab359e700 1 -- 172.21.15.16:0/842053321 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab408e620 con 0x7f3ab408fef0 2019-12-05T06:11:59.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.454+0000 7f3ab359e700 1 --2- 172.21.15.16:0/842053321 >> v2:172.21.15.16:3300/0 conn(0x7f3ab408fef0 0x7f3ab408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3aac00b2d0 tx=0x7f3aac00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.455+0000 7f3ab359e700 1 --2- 172.21.15.16:0/842053321 >> v2:172.21.15.16:3300/0 conn(0x7f3ab408fef0 0x7f3ab408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3aac00b2d0 tx=0x7f3aac00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.455+0000 7f3ab359e700 1 --2- 172.21.15.16:0/842053321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab408fef0 0x7f3ab408f720 secure :-1 s=READY pgs=318 cs=0 l=1 rx=0x7f3aac00b2d0 tx=0x7f3aac00bae0).ready entity=mon.0 client_cookie=6be181b994932306 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:59.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.455+0000 7f3ab359e700 1 --2- 172.21.15.16:0/842053321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab408fef0 0x7f3ab408f720 secure :-1 s=THROTTLE_DONE pgs=318 cs=0 l=1 rx=0x7f3aac00b2d0 tx=0x7f3aac00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.455+0000 7f3ab359e700 1 --2- 172.21.15.16:0/842053321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab408fef0 0x7f3ab408f720 secure :-1 s=THROTTLE_DONE pgs=318 cs=0 l=1 rx=0x7f3aac00b2d0 tx=0x7f3aac00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.455+0000 7f3ab1d9b700 1 -- 172.21.15.16:0/842053321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aac000df0 con 0x7f3ab408fef0 2019-12-05T06:11:59.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.455+0000 7f3ab359e700 1 --2- 172.21.15.16:0/842053321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab408fef0 0x7f3ab408f720 secure :-1 s=THROTTLE_DONE pgs=318 cs=0 l=1 rx=0x7f3aac00b2d0 tx=0x7f3aac00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.455+0000 7f3ab1d9b700 1 -- 172.21.15.16:0/842053321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac009d30 con 0x7f3ab408fef0 2019-12-05T06:11:59.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.455+0000 7f3ab1d9b700 1 -- 172.21.15.16:0/842053321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aac00a400 con 0x7f3ab408fef0 2019-12-05T06:11:59.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.455+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/842053321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab408fef0 msgr2=0x7f3ab408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.455+0000 7f3ab8b3e700 1 --2- 172.21.15.16:0/842053321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab408fef0 0x7f3ab408f720 secure :-1 s=READY pgs=318 cs=0 l=1 rx=0x7f3aac00b2d0 tx=0x7f3aac00bae0).stop 2019-12-05T06:11:59.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.457+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/842053321 shutdown_connections 2019-12-05T06:11:59.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.457+0000 7f3ab8b3e700 1 --2- 172.21.15.16:0/842053321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab408fef0 0x7f3ab408f720 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:59.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.458+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/842053321 shutdown_connections 2019-12-05T06:11:59.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.458+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/842053321 wait complete. 2019-12-05T06:11:59.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.458+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/842053321 >> 172.21.15.16:0/842053321 conn(0x7f3ab4061940 msgr2=0x7f3ab4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:59.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.461+0000 7f3ab8b3e700 1 Processor -- start 2019-12-05T06:11:59.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.461+0000 7f3ab8b3e700 1 -- start start 2019-12-05T06:11:59.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.462+0000 7f3ab8b3e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ab4086520 0x7f3ab41a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:59.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.462+0000 7f3ab8b3e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab412a440 con 0x7f3ab4086cf0 2019-12-05T06:11:59.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.462+0000 7f3ab8b3e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab412c0e0 con 0x7f3ab40826b0 2019-12-05T06:11:59.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.462+0000 7f3ab8b3e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab4084d90 con 0x7f3ab4086520 2019-12-05T06:11:59.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.462+0000 7f3ab359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ab4086520 0x7f3ab41a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:59.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.462+0000 7f3ab359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ab4086520 0x7f3ab41a2c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48760/0 (socket says 172.21.15.16:48760) 2019-12-05T06:11:59.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.462+0000 7f3ab359e700 1 -- 172.21.15.16:0/2928200296 learned_addr learned my addr 172.21.15.16:0/2928200296 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:11:59.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.462+0000 7f3ab2d9d700 1 --1- 172.21.15.16:0/2928200296 >> v1:172.21.15.76:6789/0 conn(0x7f3ab40826b0 0x7f3ab4085d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:55226/0 (socket says 172.21.15.16:55226) 2019-12-05T06:11:59.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.463+0000 7f3ab359e700 1 -- 172.21.15.16:0/2928200296 >> v1:172.21.15.76:6789/0 conn(0x7f3ab40826b0 legacy=0x7f3ab4085d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.463+0000 7f3ab359e700 1 -- 172.21.15.16:0/2928200296 >> v1:172.21.15.16:6789/0 conn(0x7f3ab4086cf0 legacy=0x7f3ab419f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.464+0000 7f3ab359e700 1 -- 172.21.15.16:0/2928200296 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab4085420 con 0x7f3ab4086520 2019-12-05T06:11:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.464+0000 7f3ab359e700 1 --2- 172.21.15.16:0/2928200296 >> v2:172.21.15.16:3300/0 conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.464+0000 7f3ab359e700 1 --2- 172.21.15.16:0/2928200296 >> v2:172.21.15.16:3300/0 conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.464+0000 7f3ab359e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=READY pgs=320 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.464+0000 7f3ab359e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.464+0000 7f3ab359e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.464+0000 7f3ab359e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.464+0000 7f3aab7fe700 1 -- 172.21.15.16:0/2928200296 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aac00b700 con 0x7f3ab4086520 2019-12-05T06:11:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.464+0000 7f3aab7fe700 1 -- 172.21.15.16:0/2928200296 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aac009b10 con 0x7f3ab4086520 2019-12-05T06:11:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.465+0000 7f3aab7fe700 1 -- 172.21.15.16:0/2928200296 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aac00a220 con 0x7f3ab4086520 2019-12-05T06:11:59.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.465+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/2928200296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab4085420 con 0x7f3ab4086520 2019-12-05T06:11:59.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.465+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/2928200296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab4085420 con 0x7f3ab4086520 2019-12-05T06:11:59.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.466+0000 7f3ab359e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.466+0000 7f3aab7fe700 1 -- 172.21.15.16:0/2928200296 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3aac00af80 con 0x7f3ab4086520 2019-12-05T06:11:59.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.467+0000 7f3aab7fe700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a98023630 0x7f3a98025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:11:59.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.467+0000 7f3ab359e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.467+0000 7f3aab7fe700 1 -- 172.21.15.16:0/2928200296 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f3aac00af80 con 0x7f3ab4086520 2019-12-05T06:11:59.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.467+0000 7f3ab2d9d700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a98023630 0x7f3a98025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:11:59.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.468+0000 7f3ab2d9d700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a98023630 0x7f3a98025ae0 crc :-1 s=READY pgs=149 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:11:59.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.481+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/2928200296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab4085420 con 0x7f3ab4086520 2019-12-05T06:11:59.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.487+0000 7f3ab359e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.487+0000 7f3aab7fe700 1 -- 172.21.15.16:0/2928200296 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3aac00af80 con 0x7f3ab4086520 2019-12-05T06:11:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.690+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/2928200296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ab4085420 con 0x7f3ab4086520 2019-12-05T06:11:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.691+0000 7f3ab359e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:11:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.691+0000 7f3aab7fe700 1 -- 172.21.15.16:0/2928200296 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f3aac00b700 con 0x7f3ab4086520 2019-12-05T06:11:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2019-12-05T06:11:59.688 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:11:59.689 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:08.017512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":36,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:11:59.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.694+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/2928200296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a98023630 msgr2=0x7f3a98025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.694+0000 7f3ab8b3e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a98023630 0x7f3a98025ae0 crc :-1 s=READY pgs=149 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:59.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.694+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 msgr2=0x7f3ab41a2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:11:59.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.694+0000 7f3ab8b3e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 0x7f3ab41a2c60 secure :-1 s=READY pgs=320 cs=0 l=1 rx=0x7f3aac0107c0 tx=0x7f3aac010b60).stop 2019-12-05T06:11:59.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.694+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/2928200296 shutdown_connections 2019-12-05T06:11:59.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.694+0000 7f3ab8b3e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a98023630 0x7f3a98025ae0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:59.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.694+0000 7f3ab8b3e700 1 --2- 172.21.15.16:0/2928200296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ab4086520 0x7f3ab41a2c60 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:11:59.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.694+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/2928200296 shutdown_connections 2019-12-05T06:11:59.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.694+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/2928200296 wait complete. 2019-12-05T06:11:59.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:11:59.694+0000 7f3ab8b3e700 1 -- 172.21.15.16:0/2928200296 >> 172.21.15.16:0/2928200296 conn(0x7f3ab406fe20 msgr2=0x7f3ab406f550 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:11:59.701 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:11:59.701 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:11:59.701 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:11:59.710 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:11:59.710 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.016+0000 7f0a472f2700 1 Processor -- start 2019-12-05T06:12:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.016+0000 7f0a472f2700 1 -- start start 2019-12-05T06:12:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.018+0000 7f0a472f2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0a4008d2c0 0x7f0a4008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.018+0000 7f0a472f2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a4008fef0 0x7f0a4008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:00.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.018+0000 7f0a472f2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a400958e0 con 0x7f0a4008c6d0 2019-12-05T06:12:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.018+0000 7f0a472f2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a4008df90 con 0x7f0a4008d2c0 2019-12-05T06:12:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.018+0000 7f0a472f2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a400a3f70 con 0x7f0a4008fef0 2019-12-05T06:12:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.018+0000 7f0a4588f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0a4008c6d0 0x7f0a4008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33942/0 (socket says 172.21.15.16:33942) 2019-12-05T06:12:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.018+0000 7f0a4588f700 1 -- 172.21.15.16:0/1966615341 learned_addr learned my addr 172.21.15.16:0/1966615341 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.018+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1966615341 >> v2:172.21.15.76:3300/0 conn(0x7f0a4008d2c0 0x7f0a4008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.018+0000 7f0a46090700 1 --2- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a4008fef0 0x7f0a4008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:00.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4488d700 1 -- 172.21.15.16:0/1966615341 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3849454506 0 0) 0x7f0a380014d0 con 0x7f0a4008c6d0 2019-12-05T06:12:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4488d700 1 -- 172.21.15.16:0/1966615341 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a28002e80 con 0x7f0a4008c6d0 2019-12-05T06:12:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4508e700 1 -- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a4008fef0 msgr2=0x7f0a4008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a4008fef0 0x7f0a4008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4508e700 1 -- 172.21.15.16:0/1966615341 >> v1:172.21.15.16:6789/0 conn(0x7f0a4008c6d0 legacy=0x7f0a4008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4508e700 1 -- 172.21.15.16:0/1966615341 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a4008e620 con 0x7f0a4008d2c0 2019-12-05T06:12:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1966615341 >> v2:172.21.15.76:3300/0 conn(0x7f0a4008d2c0 0x7f0a4008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0a3400af50 tx=0x7f0a3400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1966615341 >> v2:172.21.15.76:3300/0 conn(0x7f0a4008d2c0 0x7f0a4008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0a3400af50 tx=0x7f0a3400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a4008d2c0 0x7f0a4008ef20 secure :-1 s=READY pgs=253 cs=0 l=1 rx=0x7f0a3400af50 tx=0x7f0a3400b250).ready entity=mon.1 client_cookie=27871b4790042503 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a4008d2c0 0x7f0a4008ef20 secure :-1 s=THROTTLE_DONE pgs=253 cs=0 l=1 rx=0x7f0a3400af50 tx=0x7f0a3400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.019+0000 7f0a4488d700 1 -- 172.21.15.16:0/1966615341 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a34007210 con 0x7f0a4008d2c0 2019-12-05T06:12:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.020+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a4008d2c0 0x7f0a4008ef20 secure :-1 s=THROTTLE_DONE pgs=253 cs=0 l=1 rx=0x7f0a3400af50 tx=0x7f0a3400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.020+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a4008d2c0 0x7f0a4008ef20 secure :-1 s=THROTTLE_DONE pgs=253 cs=0 l=1 rx=0x7f0a3400af50 tx=0x7f0a3400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.020+0000 7f0a4488d700 1 -- 172.21.15.16:0/1966615341 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a340099b0 con 0x7f0a4008d2c0 2019-12-05T06:12:00.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.020+0000 7f0a4488d700 1 -- 172.21.15.16:0/1966615341 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a3400a080 con 0x7f0a4008d2c0 2019-12-05T06:12:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.020+0000 7f0a472f2700 1 -- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a4008d2c0 msgr2=0x7f0a4008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.020+0000 7f0a472f2700 1 --2- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a4008d2c0 0x7f0a4008ef20 secure :-1 s=READY pgs=253 cs=0 l=1 rx=0x7f0a3400af50 tx=0x7f0a3400b250).stop 2019-12-05T06:12:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.021+0000 7f0a472f2700 1 -- 172.21.15.16:0/1966615341 shutdown_connections 2019-12-05T06:12:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.021+0000 7f0a472f2700 1 --2- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0a4008d2c0 0x7f0a4008ef20 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.021+0000 7f0a472f2700 1 --2- 172.21.15.16:0/1966615341 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a4008fef0 0x7f0a4008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.022+0000 7f0a472f2700 1 -- 172.21.15.16:0/1966615341 shutdown_connections 2019-12-05T06:12:00.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.022+0000 7f0a472f2700 1 -- 172.21.15.16:0/1966615341 wait complete. 2019-12-05T06:12:00.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.022+0000 7f0a472f2700 1 -- 172.21.15.16:0/1966615341 >> 172.21.15.16:0/1966615341 conn(0x7f0a40061940 msgr2=0x7f0a40096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:00.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.027+0000 7f0a472f2700 1 Processor -- start 2019-12-05T06:12:00.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.027+0000 7f0a472f2700 1 -- start start 2019-12-05T06:12:00.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.028+0000 7f0a472f2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a401302f0 0x7f0a4009d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:00.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.028+0000 7f0a472f2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0a4009e7d0 0x7f0a4009dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:00.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.028+0000 7f0a472f2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a4012cb90 con 0x7f0a4009e280 2019-12-05T06:12:00.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.028+0000 7f0a472f2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a401315c0 con 0x7f0a4009e7d0 2019-12-05T06:12:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.028+0000 7f0a472f2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a40131890 con 0x7f0a401302f0 2019-12-05T06:12:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.028+0000 7f0a4508e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0a4009e7d0 0x7f0a4009dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.028+0000 7f0a46090700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0a4009e280 0x7f0a4006dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33952/0 (socket says 172.21.15.16:33952) 2019-12-05T06:12:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.028+0000 7f0a46090700 1 -- 172.21.15.16:0/1747937696 learned_addr learned my addr 172.21.15.16:0/1747937696 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.028+0000 7f0a4588f700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a401302f0 0x7f0a4009d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4508e700 1 -- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a401302f0 msgr2=0x7f0a4009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:00.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a401302f0 0x7f0a4009d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4508e700 1 -- 172.21.15.16:0/1747937696 >> v1:172.21.15.16:6789/0 conn(0x7f0a4009e280 legacy=0x7f0a4006dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4508e700 1 -- 172.21.15.16:0/1747937696 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a40132680 con 0x7f0a4009e7d0 2019-12-05T06:12:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4588f700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a401302f0 0x7f0a4009d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a3e7fc700 1 -- 172.21.15.16:0/1747937696 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424717078 0 0) 0x7f0a30001150 con 0x7f0a4009e280 2019-12-05T06:12:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> v2:172.21.15.16:3300/0 conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> v2:172.21.15.16:3300/0 conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=READY pgs=323 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:00.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a3e7fc700 1 -- 172.21.15.16:0/1747937696 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a340009e0 con 0x7f0a4009e7d0 2019-12-05T06:12:00.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.029+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.030+0000 7f0a3e7fc700 1 -- 172.21.15.16:0/1747937696 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a34009790 con 0x7f0a4009e7d0 2019-12-05T06:12:00.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.030+0000 7f0a472f2700 1 -- 172.21.15.16:0/1747937696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a40132680 con 0x7f0a4009e7d0 2019-12-05T06:12:00.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.030+0000 7f0a3e7fc700 1 -- 172.21.15.16:0/1747937696 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a34009e20 con 0x7f0a4009e7d0 2019-12-05T06:12:00.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.030+0000 7f0a472f2700 1 -- 172.21.15.16:0/1747937696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a40132680 con 0x7f0a4009e7d0 2019-12-05T06:12:00.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.030+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.031+0000 7f0a3e7fc700 1 -- 172.21.15.16:0/1747937696 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0a34009980 con 0x7f0a4009e7d0 2019-12-05T06:12:00.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.031+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.031+0000 7f0a3e7fc700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0a20023630 0x7f0a20025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:00.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.031+0000 7f0a3e7fc700 1 -- 172.21.15.16:0/1747937696 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f0a34089ff0 con 0x7f0a4009e7d0 2019-12-05T06:12:00.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.032+0000 7f0a4588f700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0a20023630 0x7f0a20025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:00.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.032+0000 7f0a4588f700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0a20023630 0x7f0a20025ae0 crc :-1 s=READY pgs=150 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:00.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.047+0000 7f0a472f2700 1 -- 172.21.15.16:0/1747937696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a40132680 con 0x7f0a4009e7d0 2019-12-05T06:12:00.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.054+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.054+0000 7f0a3e7fc700 1 -- 172.21.15.16:0/1747937696 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0a34009580 con 0x7f0a4009e7d0 2019-12-05T06:12:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.271+0000 7f0a472f2700 1 -- 172.21.15.16:0/1747937696 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0a40132680 con 0x7f0a4009e7d0 2019-12-05T06:12:00.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.272+0000 7f0a4508e700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=THROTTLE_DONE pgs=323 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.272+0000 7f0a3e7fc700 1 -- 172.21.15.16:0/1747937696 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f0a34009790 con 0x7f0a4009e7d0 2019-12-05T06:12:00.269 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2019-12-05T06:12:00.269 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:00.269 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:08.017512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":36,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:00.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 -- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0a20023630 msgr2=0x7f0a20025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:00.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0a20023630 0x7f0a20025ae0 crc :-1 s=READY pgs=150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:00.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 -- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 msgr2=0x7f0a4009dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 0x7f0a4009dd40 secure :-1 s=READY pgs=323 cs=0 l=1 rx=0x7f0a34054390 tx=0x7f0a34054730).stop 2019-12-05T06:12:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 -- 172.21.15.16:0/1747937696 shutdown_connections 2019-12-05T06:12:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0a20023630 0x7f0a20025ae0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0a4009e7d0 0x7f0a4009dd40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 --2- 172.21.15.16:0/1747937696 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0a401302f0 0x7f0a4009d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:00.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 -- 172.21.15.16:0/1747937696 shutdown_connections 2019-12-05T06:12:00.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 -- 172.21.15.16:0/1747937696 wait complete. 2019-12-05T06:12:00.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:00.274+0000 7f0a472f2700 1 -- 172.21.15.16:0/1747937696 >> 172.21.15.16:0/1747937696 conn(0x7f0a400a2850 msgr2=0x7f0a4006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:00.282 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:02.283 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:02.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.573+0000 7f16082db700 1 Processor -- start 2019-12-05T06:12:02.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.573+0000 7f16082db700 1 -- start start 2019-12-05T06:12:02.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.575+0000 7f16082db700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f160008fef0 0x7f160008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:02.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.575+0000 7f16082db700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16000958e0 con 0x7f160008c6d0 2019-12-05T06:12:02.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.575+0000 7f16082db700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16000629e0 con 0x7f160008d2c0 2019-12-05T06:12:02.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.575+0000 7f16082db700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160008df90 con 0x7f160008fef0 2019-12-05T06:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.575+0000 7f1606878700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f160008c6d0 0x7f160008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33960/0 (socket says 172.21.15.16:33960) 2019-12-05T06:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.575+0000 7f1606878700 1 -- 172.21.15.16:0/3821347392 learned_addr learned my addr 172.21.15.16:0/3821347392 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.576+0000 7f1607079700 1 --2- 172.21.15.16:0/3821347392 >> v2:172.21.15.76:3300/0 conn(0x7f160008fef0 0x7f160008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:02.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.576+0000 7f1605876700 1 -- 172.21.15.16:0/3821347392 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58079412 0 0) 0x7f15f80014d0 con 0x7f160008c6d0 2019-12-05T06:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.576+0000 7f1605876700 1 -- 172.21.15.16:0/3821347392 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e8002e80 con 0x7f160008c6d0 2019-12-05T06:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.576+0000 7f1605876700 1 -- 172.21.15.16:0/3821347392 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 76919470 0 0) 0x7f15f4001080 con 0x7f160008d2c0 2019-12-05T06:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.576+0000 7f1605876700 1 -- 172.21.15.16:0/3821347392 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15e8004170 con 0x7f160008d2c0 2019-12-05T06:12:02.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.576+0000 7f1607079700 1 -- 172.21.15.16:0/3821347392 >> v1:172.21.15.76:6789/0 conn(0x7f160008d2c0 legacy=0x7f160008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.577+0000 7f1607079700 1 -- 172.21.15.16:0/3821347392 >> v1:172.21.15.16:6789/0 conn(0x7f160008c6d0 legacy=0x7f160008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.577+0000 7f1607079700 1 -- 172.21.15.16:0/3821347392 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f160008e620 con 0x7f160008fef0 2019-12-05T06:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.577+0000 7f1607079700 1 --2- 172.21.15.16:0/3821347392 >> v2:172.21.15.76:3300/0 conn(0x7f160008fef0 0x7f160008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f15f000af80 tx=0x7f15f000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.577+0000 7f1607079700 1 --2- 172.21.15.16:0/3821347392 >> v2:172.21.15.76:3300/0 conn(0x7f160008fef0 0x7f160008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f15f000af80 tx=0x7f15f000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.577+0000 7f1607079700 1 --2- 172.21.15.16:0/3821347392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f160008fef0 0x7f160008f720 secure :-1 s=READY pgs=255 cs=0 l=1 rx=0x7f15f000af80 tx=0x7f15f000b790).ready entity=mon.1 client_cookie=a4e63731f8cd40ca server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.577+0000 7f1607079700 1 --2- 172.21.15.16:0/3821347392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f160008fef0 0x7f160008f720 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f15f000af80 tx=0x7f15f000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.577+0000 7f1605876700 1 -- 172.21.15.16:0/3821347392 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15f0007240 con 0x7f160008fef0 2019-12-05T06:12:02.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.577+0000 7f1607079700 1 --2- 172.21.15.16:0/3821347392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f160008fef0 0x7f160008f720 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f15f000af80 tx=0x7f15f000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.577+0000 7f1605876700 1 -- 172.21.15.16:0/3821347392 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f0000bb0 con 0x7f160008fef0 2019-12-05T06:12:02.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.578+0000 7f1607079700 1 --2- 172.21.15.16:0/3821347392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f160008fef0 0x7f160008f720 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f15f000af80 tx=0x7f15f000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.578+0000 7f1605876700 1 -- 172.21.15.16:0/3821347392 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15f0009bd0 con 0x7f160008fef0 2019-12-05T06:12:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.578+0000 7f16082db700 1 -- 172.21.15.16:0/3821347392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f160008fef0 msgr2=0x7f160008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.578+0000 7f16082db700 1 --2- 172.21.15.16:0/3821347392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f160008fef0 0x7f160008f720 secure :-1 s=READY pgs=255 cs=0 l=1 rx=0x7f15f000af80 tx=0x7f15f000b790).stop 2019-12-05T06:12:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.580+0000 7f16082db700 1 -- 172.21.15.16:0/3821347392 shutdown_connections 2019-12-05T06:12:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.580+0000 7f16082db700 1 --2- 172.21.15.16:0/3821347392 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f160008fef0 0x7f160008f720 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.580+0000 7f16082db700 1 -- 172.21.15.16:0/3821347392 shutdown_connections 2019-12-05T06:12:02.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.580+0000 7f16082db700 1 -- 172.21.15.16:0/3821347392 wait complete. 2019-12-05T06:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.580+0000 7f16082db700 1 -- 172.21.15.16:0/3821347392 >> 172.21.15.16:0/3821347392 conn(0x7f1600061940 msgr2=0x7f1600096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.585+0000 7f16082db700 1 Processor -- start 2019-12-05T06:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.586+0000 7f16082db700 1 -- start start 2019-12-05T06:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.587+0000 7f16082db700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1600153a50 0x7f1600153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.587+0000 7f16082db700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f160012a1d0 0x7f1600129200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.587+0000 7f16082db700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f160006e5c0 con 0x7f1600129740 2019-12-05T06:12:02.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.587+0000 7f16082db700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f160006e250 con 0x7f1600153a50 2019-12-05T06:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.587+0000 7f16082db700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f160006eac0 con 0x7f160012a1d0 2019-12-05T06:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.587+0000 7f1606878700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1600153a50 0x7f1600153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.588+0000 7f1606878700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1600153a50 0x7f1600153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48798/0 (socket says 172.21.15.16:48798) 2019-12-05T06:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.588+0000 7f1606878700 1 -- 172.21.15.16:0/2709318596 learned_addr learned my addr 172.21.15.16:0/2709318596 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.588+0000 7f1606077700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f160012a1d0 0x7f1600129200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:02.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.588+0000 7f15ff7fe700 1 -- 172.21.15.16:0/2709318596 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070508661 0 0) 0x7f15f0009990 con 0x7f1600129740 2019-12-05T06:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.588+0000 7f1606878700 1 -- 172.21.15.16:0/2709318596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f160012a1d0 msgr2=0x7f1600129200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.588+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f160012a1d0 0x7f1600129200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.588+0000 7f1606878700 1 -- 172.21.15.16:0/2709318596 >> v1:172.21.15.76:6789/0 conn(0x7f1600129740 legacy=0x7f16000a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.588+0000 7f1606878700 1 -- 172.21.15.16:0/2709318596 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1600128900 con 0x7f1600153a50 2019-12-05T06:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.588+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> v2:172.21.15.16:3300/0 conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.589+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> v2:172.21.15.16:3300/0 conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.589+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=READY pgs=325 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:02.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.589+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.589+0000 7f15ff7fe700 1 -- 172.21.15.16:0/2709318596 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15f8001010 con 0x7f1600153a50 2019-12-05T06:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.589+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.589+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.589+0000 7f15ff7fe700 1 -- 172.21.15.16:0/2709318596 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f15f8001440 con 0x7f1600153a50 2019-12-05T06:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.589+0000 7f16082db700 1 -- 172.21.15.16:0/2709318596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1600128900 con 0x7f1600153a50 2019-12-05T06:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.589+0000 7f15ff7fe700 1 -- 172.21.15.16:0/2709318596 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f15f8001b10 con 0x7f1600153a50 2019-12-05T06:12:02.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.589+0000 7f16082db700 1 -- 172.21.15.16:0/2709318596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1600128900 con 0x7f1600153a50 2019-12-05T06:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.590+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.591+0000 7f15ff7fe700 1 -- 172.21.15.16:0/2709318596 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f15f8001010 con 0x7f1600153a50 2019-12-05T06:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.591+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.592+0000 7f15ff7fe700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f15e0023630 0x7f15e0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.592+0000 7f15ff7fe700 1 -- 172.21.15.16:0/2709318596 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f15f8046130 con 0x7f1600153a50 2019-12-05T06:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.592+0000 7f1606077700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f15e0023630 0x7f15e0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:02.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.593+0000 7f1606077700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f15e0023630 0x7f15e0025ac0 crc :-1 s=READY pgs=151 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.604+0000 7f16082db700 1 -- 172.21.15.16:0/2709318596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1600128900 con 0x7f1600153a50 2019-12-05T06:12:02.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.613+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.613+0000 7f15ff7fe700 1 -- 172.21.15.16:0/2709318596 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f15f8007920 con 0x7f1600153a50 2019-12-05T06:12:02.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.879+0000 7f16082db700 1 -- 172.21.15.16:0/2709318596 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1600128900 con 0x7f1600153a50 2019-12-05T06:12:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.879+0000 7f1606878700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=THROTTLE_DONE pgs=325 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:02.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.879+0000 7f15ff7fe700 1 -- 172.21.15.16:0/2709318596 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f15f8001010 con 0x7f1600153a50 2019-12-05T06:12:02.877 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2019-12-05T06:12:02.877 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:02.877 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:08.017512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":36,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:02.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.882+0000 7f16082db700 1 -- 172.21.15.16:0/2709318596 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f15e0023630 msgr2=0x7f15e0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.882+0000 7f16082db700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f15e0023630 0x7f15e0025ac0 crc :-1 s=READY pgs=151 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.882+0000 7f16082db700 1 -- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 msgr2=0x7f1600153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.882+0000 7f16082db700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 0x7f1600153e40 secure :-1 s=READY pgs=325 cs=0 l=1 rx=0x7f15f8010760 tx=0x7f15f80105e0).stop 2019-12-05T06:12:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.882+0000 7f16082db700 1 -- 172.21.15.16:0/2709318596 shutdown_connections 2019-12-05T06:12:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.882+0000 7f16082db700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f160012a1d0 0x7f1600129200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.882+0000 7f16082db700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f15e0023630 0x7f15e0025ac0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:02.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.882+0000 7f16082db700 1 --2- 172.21.15.16:0/2709318596 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1600153a50 0x7f1600153e40 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:02.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.882+0000 7f16082db700 1 -- 172.21.15.16:0/2709318596 shutdown_connections 2019-12-05T06:12:02.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.883+0000 7f16082db700 1 -- 172.21.15.16:0/2709318596 wait complete. 2019-12-05T06:12:02.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:02.883+0000 7f16082db700 1 -- 172.21.15.16:0/2709318596 >> 172.21.15.16:0/2709318596 conn(0x7f160006d260 msgr2=0x7f160009aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:02.899 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:04.900 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:05.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.171+0000 7f3330e3f700 1 Processor -- start 2019-12-05T06:12:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.171+0000 7f3330e3f700 1 -- start start 2019-12-05T06:12:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.173+0000 7f3330e3f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f332c08c6d0 0x7f332c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.173+0000 7f3330e3f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c08f720 0x7f332c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.173+0000 7f3330e3f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332c08fb70 con 0x7f332c08fef0 2019-12-05T06:12:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.173+0000 7f3330e3f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f332c08df90 con 0x7f332c08c6d0 2019-12-05T06:12:05.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.173+0000 7f3330e3f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c0a3f70 con 0x7f332c08f720 2019-12-05T06:12:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.174+0000 7f332ad9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f332c08c6d0 0x7f332c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.174+0000 7f332ad9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f332c08c6d0 0x7f332c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:34898/0 (socket says 172.21.15.16:34898) 2019-12-05T06:12:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.174+0000 7f332b59e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c08f720 0x7f332c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:05.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.174+0000 7f332ad9d700 1 -- 172.21.15.16:0/2639884030 learned_addr learned my addr 172.21.15.16:0/2639884030 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:12:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.174+0000 7f332ad9d700 1 -- 172.21.15.16:0/2639884030 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c08f720 msgr2=0x7f332c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.174+0000 7f332ad9d700 1 --2- 172.21.15.16:0/2639884030 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c08f720 0x7f332c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.174+0000 7f332ad9d700 1 -- 172.21.15.16:0/2639884030 >> v1:172.21.15.76:6789/0 conn(0x7f332c08fef0 legacy=0x7f332c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:05.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.174+0000 7f332ad9d700 1 -- 172.21.15.16:0/2639884030 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c08d540 con 0x7f332c08c6d0 2019-12-05T06:12:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.174+0000 7f3329d9b700 1 -- 172.21.15.16:0/2639884030 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576481838 0 0) 0x7f3314001080 con 0x7f332c08fef0 2019-12-05T06:12:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f332ad9d700 1 --2- 172.21.15.16:0/2639884030 >> v2:172.21.15.76:3300/0 conn(0x7f332c08c6d0 0x7f332c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f332400fa40 tx=0x7f332400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f332ad9d700 1 --2- 172.21.15.16:0/2639884030 >> v2:172.21.15.76:3300/0 conn(0x7f332c08c6d0 0x7f332c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f332400fa40 tx=0x7f332400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f332ad9d700 1 --2- 172.21.15.16:0/2639884030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f332c08c6d0 0x7f332c08cac0 secure :-1 s=READY pgs=258 cs=0 l=1 rx=0x7f332400fa40 tx=0x7f332400fde0).ready entity=mon.1 client_cookie=fdbacc806ea8b50b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:05.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f332ad9d700 1 --2- 172.21.15.16:0/2639884030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f332c08c6d0 0x7f332c08cac0 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f332400fa40 tx=0x7f332400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f3329d9b700 1 -- 172.21.15.16:0/2639884030 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3324000ce0 con 0x7f332c08c6d0 2019-12-05T06:12:05.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f332ad9d700 1 --2- 172.21.15.16:0/2639884030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f332c08c6d0 0x7f332c08cac0 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f332400fa40 tx=0x7f332400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f332ad9d700 1 --2- 172.21.15.16:0/2639884030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f332c08c6d0 0x7f332c08cac0 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f332400fa40 tx=0x7f332400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f3329d9b700 1 -- 172.21.15.16:0/2639884030 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3324001010 con 0x7f332c08c6d0 2019-12-05T06:12:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f3329d9b700 1 -- 172.21.15.16:0/2639884030 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3324001720 con 0x7f332c08c6d0 2019-12-05T06:12:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f3330e3f700 1 -- 172.21.15.16:0/2639884030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f332c08c6d0 msgr2=0x7f332c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.175+0000 7f3330e3f700 1 --2- 172.21.15.16:0/2639884030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f332c08c6d0 0x7f332c08cac0 secure :-1 s=READY pgs=258 cs=0 l=1 rx=0x7f332400fa40 tx=0x7f332400fde0).stop 2019-12-05T06:12:05.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.176+0000 7f3330e3f700 1 -- 172.21.15.16:0/2639884030 shutdown_connections 2019-12-05T06:12:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.176+0000 7f3330e3f700 1 --2- 172.21.15.16:0/2639884030 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c08f720 0x7f332c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.176+0000 7f3330e3f700 1 --2- 172.21.15.16:0/2639884030 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f332c08c6d0 0x7f332c08cac0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.176+0000 7f3330e3f700 1 -- 172.21.15.16:0/2639884030 shutdown_connections 2019-12-05T06:12:05.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.176+0000 7f3330e3f700 1 -- 172.21.15.16:0/2639884030 wait complete. 2019-12-05T06:12:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.176+0000 7f3330e3f700 1 -- 172.21.15.16:0/2639884030 >> 172.21.15.16:0/2639884030 conn(0x7f332c061940 msgr2=0x7f332c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.182+0000 7f3330e3f700 1 Processor -- start 2019-12-05T06:12:05.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.182+0000 7f3330e3f700 1 -- start start 2019-12-05T06:12:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.183+0000 7f3330e3f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.184+0000 7f3330e3f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332c131bf0 con 0x7f332c1302f0 2019-12-05T06:12:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.184+0000 7f3330e3f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f332c12b280 con 0x7f332c12b640 2019-12-05T06:12:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.184+0000 7f3330e3f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f332c1296e0 con 0x7f332c12ae70 2019-12-05T06:12:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.184+0000 7f332ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f332c1302f0 0x7f332c12a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:33982/0 (socket says 172.21.15.16:33982) 2019-12-05T06:12:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.184+0000 7f332ad9d700 1 -- 172.21.15.16:0/3779776288 learned_addr learned my addr 172.21.15.16:0/3779776288 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.184+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.184+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518937223 0 0) 0x7f3324007950 con 0x7f332c1302f0 2019-12-05T06:12:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.184+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3310002e80 con 0x7f332c1302f0 2019-12-05T06:12:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.184+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2368220787 0 0) 0x7f3314001240 con 0x7f332c12b640 2019-12-05T06:12:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33100030b0 con 0x7f332c12b640 2019-12-05T06:12:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f332b59e700 1 -- 172.21.15.16:0/3779776288 >> v1:172.21.15.76:6789/0 conn(0x7f332c12b640 legacy=0x7f332c153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f332b59e700 1 -- 172.21.15.16:0/3779776288 >> v1:172.21.15.16:6789/0 conn(0x7f332c1302f0 legacy=0x7f332c12a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f332b59e700 1 -- 172.21.15.16:0/3779776288 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f332c129d70 con 0x7f332c12ae70 2019-12-05T06:12:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f331c009990 con 0x7f332c12ae70 2019-12-05T06:12:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f331c0095b0 con 0x7f332c12ae70 2019-12-05T06:12:05.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f3330e3f700 1 -- 172.21.15.16:0/3779776288 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f332c129d70 con 0x7f332c12ae70 2019-12-05T06:12:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.185+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f331c009840 con 0x7f332c12ae70 2019-12-05T06:12:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.186+0000 7f3330e3f700 1 -- 172.21.15.16:0/3779776288 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f332c129d70 con 0x7f332c12ae70 2019-12-05T06:12:05.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.186+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.187+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f331c009a50 con 0x7f332c12ae70 2019-12-05T06:12:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.187+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.187+0000 7f33237fe700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3310023650 0x7f3310025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:05.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.188+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6231+0+0 (secure 0 0 0) 0x7f331c04a180 con 0x7f332c12ae70 2019-12-05T06:12:05.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.188+0000 7f332ad9d700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3310023650 0x7f3310025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:05.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.188+0000 7f332ad9d700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3310023650 0x7f3310025ae0 crc :-1 s=READY pgs=152 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:05.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.203+0000 7f3330e3f700 1 -- 172.21.15.16:0/3779776288 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f332c129d70 con 0x7f332c12ae70 2019-12-05T06:12:05.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.211+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.211+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f331c009840 con 0x7f332c12ae70 2019-12-05T06:12:05.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.482+0000 7f3330e3f700 1 -- 172.21.15.16:0/3779776288 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f332c129d70 con 0x7f332c12ae70 2019-12-05T06:12:05.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.484+0000 7f332b59e700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=THROTTLE_DONE pgs=90 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:05.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.484+0000 7f33237fe700 1 -- 172.21.15.16:0/3779776288 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f331c009a50 con 0x7f332c12ae70 2019-12-05T06:12:05.485 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2019-12-05T06:12:05.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.487+0000 7f3330e3f700 1 -- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3310023650 msgr2=0x7f3310025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:05.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.487+0000 7f3330e3f700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3310023650 0x7f3310025ae0 crc :-1 s=READY pgs=152 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:05.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.488+0000 7f3330e3f700 1 -- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 msgr2=0x7f332c157370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:05.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.488+0000 7f3330e3f700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 secure :-1 s=READY pgs=90 cs=0 l=1 rx=0x7f331c00e3e0 tx=0x7f331c00fee0).stop 2019-12-05T06:12:05.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.488+0000 7f3330e3f700 1 -- 172.21.15.16:0/3779776288 shutdown_connections 2019-12-05T06:12:05.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.488+0000 7f3330e3f700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3310023650 0x7f3310025ae0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:05.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.488+0000 7f3330e3f700 1 --2- 172.21.15.16:0/3779776288 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f332c12ae70 0x7f332c157370 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:05.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.488+0000 7f3330e3f700 1 -- 172.21.15.16:0/3779776288 shutdown_connections 2019-12-05T06:12:05.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.488+0000 7f3330e3f700 1 -- 172.21.15.16:0/3779776288 wait complete. 2019-12-05T06:12:05.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:05.488+0000 7f3330e3f700 1 -- 172.21.15.16:0/3779776288 >> 172.21.15.16:0/3779776288 conn(0x7f332c0a2850 msgr2=0x7f332c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:05.488 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:05.488 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:08.017512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":36,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:05.496 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:07.497 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.785+0000 7fb33a686700 1 Processor -- start 2019-12-05T06:12:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.785+0000 7fb33a686700 1 -- start start 2019-12-05T06:12:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.787+0000 7fb33a686700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb33408d2c0 0x7fb33408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.787+0000 7fb33a686700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 0x7fb33408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:07.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.787+0000 7fb33a686700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3340958e0 con 0x7fb33408c6d0 2019-12-05T06:12:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.787+0000 7fb33a686700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb33408df90 con 0x7fb33408d2c0 2019-12-05T06:12:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.787+0000 7fb33a686700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3340a3f70 con 0x7fb33408fef0 2019-12-05T06:12:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.788+0000 7fb333fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb33408d2c0 0x7fb33408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.788+0000 7fb338c23700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb33408c6d0 0x7fb33408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:55294/0 (socket says 172.21.15.16:55294) 2019-12-05T06:12:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.788+0000 7fb339424700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 0x7fb33408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:07.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.788+0000 7fb338c23700 1 -- 172.21.15.16:0/31581835 learned_addr learned my addr 172.21.15.16:0/31581835 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.788+0000 7fb3337fe700 1 -- 172.21.15.16:0/31581835 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 407232124 0 0) 0x7fb31c000c60 con 0x7fb33408c6d0 2019-12-05T06:12:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.788+0000 7fb3337fe700 1 -- 172.21.15.16:0/31581835 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb320002e80 con 0x7fb33408c6d0 2019-12-05T06:12:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.788+0000 7fb339424700 1 -- 172.21.15.16:0/31581835 >> v2:172.21.15.76:3300/0 conn(0x7fb33408d2c0 msgr2=0x7fb33408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.788+0000 7fb339424700 1 --2- 172.21.15.16:0/31581835 >> v2:172.21.15.76:3300/0 conn(0x7fb33408d2c0 0x7fb33408ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.788+0000 7fb339424700 1 -- 172.21.15.16:0/31581835 >> v1:172.21.15.76:6789/0 conn(0x7fb33408c6d0 legacy=0x7fb33408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.788+0000 7fb339424700 1 -- 172.21.15.16:0/31581835 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb33408e620 con 0x7fb33408fef0 2019-12-05T06:12:07.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.789+0000 7fb333fff700 1 --2- 172.21.15.16:0/31581835 >> v2:172.21.15.76:3300/0 conn(0x7fb33408d2c0 0x7fb33408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.789+0000 7fb339424700 1 --2- 172.21.15.16:0/31581835 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 0x7fb33408f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb32c00af80 tx=0x7fb32c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.789+0000 7fb339424700 1 --2- 172.21.15.16:0/31581835 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 0x7fb33408f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb32c00af80 tx=0x7fb32c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.789+0000 7fb339424700 1 --2- 172.21.15.16:0/31581835 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 0x7fb33408f460 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7fb32c00af80 tx=0x7fb32c00b790).ready entity=mon.2 client_cookie=f86c341cadf5b857 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.789+0000 7fb339424700 1 --2- 172.21.15.16:0/31581835 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 0x7fb33408f460 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7fb32c00af80 tx=0x7fb32c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.789+0000 7fb339424700 1 --2- 172.21.15.16:0/31581835 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 0x7fb33408f460 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7fb32c00af80 tx=0x7fb32c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.789+0000 7fb3337fe700 1 -- 172.21.15.16:0/31581835 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb32c000bc0 con 0x7fb33408fef0 2019-12-05T06:12:07.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.789+0000 7fb339424700 1 --2- 172.21.15.16:0/31581835 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 0x7fb33408f460 secure :-1 s=THROTTLE_DONE pgs=91 cs=0 l=1 rx=0x7fb32c00af80 tx=0x7fb32c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.789+0000 7fb3337fe700 1 -- 172.21.15.16:0/31581835 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb32c0097f0 con 0x7fb33408fef0 2019-12-05T06:12:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.789+0000 7fb3337fe700 1 -- 172.21.15.16:0/31581835 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb32c009f60 con 0x7fb33408fef0 2019-12-05T06:12:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.790+0000 7fb33a686700 1 -- 172.21.15.16:0/31581835 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 msgr2=0x7fb33408f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:07.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.790+0000 7fb33a686700 1 --2- 172.21.15.16:0/31581835 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 0x7fb33408f460 secure :-1 s=READY pgs=91 cs=0 l=1 rx=0x7fb32c00af80 tx=0x7fb32c00b790).stop 2019-12-05T06:12:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.791+0000 7fb33a686700 1 -- 172.21.15.16:0/31581835 shutdown_connections 2019-12-05T06:12:07.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.791+0000 7fb33a686700 1 --2- 172.21.15.16:0/31581835 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb33408fef0 0x7fb33408f460 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:07.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.791+0000 7fb33a686700 1 --2- 172.21.15.16:0/31581835 >> v2:172.21.15.76:3300/0 conn(0x7fb33408d2c0 0x7fb33408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:07.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.791+0000 7fb33a686700 1 -- 172.21.15.16:0/31581835 shutdown_connections 2019-12-05T06:12:07.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.791+0000 7fb33a686700 1 -- 172.21.15.16:0/31581835 wait complete. 2019-12-05T06:12:07.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.791+0000 7fb33a686700 1 -- 172.21.15.16:0/31581835 >> 172.21.15.16:0/31581835 conn(0x7fb334061940 msgr2=0x7fb334096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:07.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.797+0000 7fb33a686700 1 Processor -- start 2019-12-05T06:12:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.797+0000 7fb33a686700 1 -- start start 2019-12-05T06:12:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.799+0000 7fb33a686700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb33406dc50 0x7fb33406e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.799+0000 7fb33a686700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb334135450 0x7fb33406e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:07.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.799+0000 7fb33a686700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3341350e0 con 0x7fb334134480 2019-12-05T06:12:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.799+0000 7fb33a686700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb334132830 con 0x7fb334135450 2019-12-05T06:12:07.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.799+0000 7fb33a686700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3341334f0 con 0x7fb33406dc50 2019-12-05T06:12:07.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.799+0000 7fb333fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb334135450 0x7fb33406e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:07.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.799+0000 7fb333fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb334135450 0x7fb33406e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48832/0 (socket says 172.21.15.16:48832) 2019-12-05T06:12:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.799+0000 7fb333fff700 1 -- 172.21.15.16:0/69172318 learned_addr learned my addr 172.21.15.16:0/69172318 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:12:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.800+0000 7fb338c23700 1 --2- 172.21.15.16:0/69172318 >> v2:172.21.15.76:3300/0 conn(0x7fb33406dc50 0x7fb33406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:07.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.800+0000 7fb333fff700 1 -- 172.21.15.16:0/69172318 >> v2:172.21.15.76:3300/0 conn(0x7fb33406dc50 msgr2=0x7fb33406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.800+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> v2:172.21.15.76:3300/0 conn(0x7fb33406dc50 0x7fb33406e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.800+0000 7fb333fff700 1 -- 172.21.15.16:0/69172318 >> v1:172.21.15.76:6789/0 conn(0x7fb334134480 legacy=0x7fb3341348e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.800+0000 7fb333fff700 1 -- 172.21.15.16:0/69172318 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb334133b80 con 0x7fb334135450 2019-12-05T06:12:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.800+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> v2:172.21.15.16:3300/0 conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.800+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> v2:172.21.15.16:3300/0 conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.800+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=READY pgs=327 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.801+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.801+0000 7fb3317fa700 1 -- 172.21.15.16:0/69172318 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb32400ac40 con 0x7fb334135450 2019-12-05T06:12:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.801+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.801+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.801+0000 7fb3317fa700 1 -- 172.21.15.16:0/69172318 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb324009af0 con 0x7fb334135450 2019-12-05T06:12:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.801+0000 7fb33a686700 1 -- 172.21.15.16:0/69172318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb334133b80 con 0x7fb334135450 2019-12-05T06:12:07.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.801+0000 7fb3317fa700 1 -- 172.21.15.16:0/69172318 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb32400ac40 con 0x7fb334135450 2019-12-05T06:12:07.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.801+0000 7fb33a686700 1 -- 172.21.15.16:0/69172318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb334133b80 con 0x7fb334135450 2019-12-05T06:12:07.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.802+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.803+0000 7fb3317fa700 1 -- 172.21.15.16:0/69172318 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb324009ce0 con 0x7fb334135450 2019-12-05T06:12:07.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.803+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.803+0000 7fb3317fa700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb318023630 0x7fb318025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:07.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.803+0000 7fb3317fa700 1 -- 172.21.15.16:0/69172318 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fb3240461c0 con 0x7fb334135450 2019-12-05T06:12:07.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.803+0000 7fb338c23700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb318023630 0x7fb318025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:07.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.804+0000 7fb338c23700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb318023630 0x7fb318025ac0 crc :-1 s=READY pgs=153 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:07.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.816+0000 7fb33a686700 1 -- 172.21.15.16:0/69172318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb334133b80 con 0x7fb334135450 2019-12-05T06:12:07.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.825+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:07.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:07.825+0000 7fb3317fa700 1 -- 172.21.15.16:0/69172318 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb32400ac40 con 0x7fb334135450 2019-12-05T06:12:08.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.059+0000 7fb33a686700 1 -- 172.21.15.16:0/69172318 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb334133b80 con 0x7fb334135450 2019-12-05T06:12:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.060+0000 7fb333fff700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=THROTTLE_DONE pgs=327 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.060+0000 7fb3317fa700 1 -- 172.21.15.16:0/69172318 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fb32400a590 con 0x7fb334135450 2019-12-05T06:12:08.057 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2019-12-05T06:12:08.057 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:08.057 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:08.017512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":36,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.062+0000 7fb33a686700 1 -- 172.21.15.16:0/69172318 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb318023630 msgr2=0x7fb318025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:08.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.062+0000 7fb33a686700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb318023630 0x7fb318025ac0 crc :-1 s=READY pgs=153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.062+0000 7fb33a686700 1 -- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 msgr2=0x7fb33406e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.062+0000 7fb33a686700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 0x7fb33406e580 secure :-1 s=READY pgs=327 cs=0 l=1 rx=0x7fb324010790 tx=0x7fb324010b30).stop 2019-12-05T06:12:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.063+0000 7fb33a686700 1 -- 172.21.15.16:0/69172318 shutdown_connections 2019-12-05T06:12:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.063+0000 7fb33a686700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb318023630 0x7fb318025ac0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.063+0000 7fb33a686700 1 --2- 172.21.15.16:0/69172318 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb334135450 0x7fb33406e580 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:08.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.063+0000 7fb33a686700 1 --2- 172.21.15.16:0/69172318 >> v2:172.21.15.76:3300/0 conn(0x7fb33406dc50 0x7fb33406e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.063+0000 7fb33a686700 1 -- 172.21.15.16:0/69172318 shutdown_connections 2019-12-05T06:12:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.063+0000 7fb33a686700 1 -- 172.21.15.16:0/69172318 wait complete. 2019-12-05T06:12:08.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:08.063+0000 7fb33a686700 1 -- 172.21.15.16:0/69172318 >> 172.21.15.16:0/69172318 conn(0x7fb3340a2850 msgr2=0x7fb33406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:08.070 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:10.071 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:10.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.369+0000 7faca115c700 1 Processor -- start 2019-12-05T06:12:10.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.369+0000 7faca115c700 1 -- start start 2019-12-05T06:12:10.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.371+0000 7faca115c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac9c08d2c0 0x7fac9c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:10.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.371+0000 7faca115c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac9c08fef0 0x7fac9c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:10.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.371+0000 7faca115c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9c0958e0 con 0x7fac9c08c6d0 2019-12-05T06:12:10.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.371+0000 7faca115c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9c08df90 con 0x7fac9c08fef0 2019-12-05T06:12:10.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.371+0000 7faca115c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac9c0a3f70 con 0x7fac9c08d2c0 2019-12-05T06:12:10.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.371+0000 7faca095b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac9c08fef0 0x7fac9c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:10.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.371+0000 7faca095b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac9c08fef0 0x7fac9c08f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48842/0 (socket says 172.21.15.16:48842) 2019-12-05T06:12:10.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.371+0000 7faca095b700 1 -- 172.21.15.16:0/3493681722 learned_addr learned my addr 172.21.15.16:0/3493681722 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:12:10.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.371+0000 7fac9ad9d700 1 --2- 172.21.15.16:0/3493681722 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac9c08d2c0 0x7fac9c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:10.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.372+0000 7faca095b700 1 -- 172.21.15.16:0/3493681722 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac9c08d2c0 msgr2=0x7fac9c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:10.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.372+0000 7fac9a59c700 1 -- 172.21.15.16:0/3493681722 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616679773 0 0) 0x7fac8c001080 con 0x7fac9c08c6d0 2019-12-05T06:12:10.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.372+0000 7faca095b700 1 --2- 172.21.15.16:0/3493681722 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac9c08d2c0 0x7fac9c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:10.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.372+0000 7faca095b700 1 -- 172.21.15.16:0/3493681722 >> v1:172.21.15.76:6789/0 conn(0x7fac9c08c6d0 legacy=0x7fac9c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:10.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.372+0000 7faca095b700 1 -- 172.21.15.16:0/3493681722 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c08e620 con 0x7fac9c08fef0 2019-12-05T06:12:10.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.372+0000 7faca095b700 1 --2- 172.21.15.16:0/3493681722 >> v2:172.21.15.16:3300/0 conn(0x7fac9c08fef0 0x7fac9c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac9400f730 tx=0x7fac94007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.372+0000 7faca095b700 1 --2- 172.21.15.16:0/3493681722 >> v2:172.21.15.16:3300/0 conn(0x7fac9c08fef0 0x7fac9c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac9400f730 tx=0x7fac94007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.372+0000 7faca095b700 1 --2- 172.21.15.16:0/3493681722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c08fef0 0x7fac9c08f460 secure :-1 s=READY pgs=328 cs=0 l=1 rx=0x7fac9400f730 tx=0x7fac94007d50).ready entity=mon.0 client_cookie=7aae18118ca5afa4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:10.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.372+0000 7faca095b700 1 --2- 172.21.15.16:0/3493681722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c08fef0 0x7fac9c08f460 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7fac9400f730 tx=0x7fac94007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.373+0000 7fac9a59c700 1 -- 172.21.15.16:0/3493681722 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac94000a70 con 0x7fac9c08fef0 2019-12-05T06:12:10.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.373+0000 7faca095b700 1 --2- 172.21.15.16:0/3493681722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c08fef0 0x7fac9c08f460 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7fac9400f730 tx=0x7fac94007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.373+0000 7faca095b700 1 --2- 172.21.15.16:0/3493681722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c08fef0 0x7fac9c08f460 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7fac9400f730 tx=0x7fac94007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.373+0000 7fac9a59c700 1 -- 172.21.15.16:0/3493681722 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac940010e0 con 0x7fac9c08fef0 2019-12-05T06:12:10.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.373+0000 7fac9a59c700 1 -- 172.21.15.16:0/3493681722 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac940017b0 con 0x7fac9c08fef0 2019-12-05T06:12:10.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.373+0000 7faca115c700 1 -- 172.21.15.16:0/3493681722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c08fef0 msgr2=0x7fac9c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:10.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.373+0000 7faca115c700 1 --2- 172.21.15.16:0/3493681722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c08fef0 0x7fac9c08f460 secure :-1 s=READY pgs=328 cs=0 l=1 rx=0x7fac9400f730 tx=0x7fac94007d50).stop 2019-12-05T06:12:10.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.375+0000 7faca115c700 1 -- 172.21.15.16:0/3493681722 shutdown_connections 2019-12-05T06:12:10.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.375+0000 7faca115c700 1 --2- 172.21.15.16:0/3493681722 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c08fef0 0x7fac9c08f460 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:10.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.375+0000 7faca115c700 1 --2- 172.21.15.16:0/3493681722 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac9c08d2c0 0x7fac9c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:10.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.375+0000 7faca115c700 1 -- 172.21.15.16:0/3493681722 shutdown_connections 2019-12-05T06:12:10.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.375+0000 7faca115c700 1 -- 172.21.15.16:0/3493681722 wait complete. 2019-12-05T06:12:10.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.375+0000 7faca115c700 1 -- 172.21.15.16:0/3493681722 >> 172.21.15.16:0/3493681722 conn(0x7fac9c061940 msgr2=0x7fac9c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.380+0000 7faca115c700 1 Processor -- start 2019-12-05T06:12:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.381+0000 7faca115c700 1 -- start start 2019-12-05T06:12:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.382+0000 7faca115c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac9c06df10 0x7fac9c098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.382+0000 7faca115c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fac9c0992d0 0x7fac9c098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:10.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.382+0000 7faca115c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac9c06e300 con 0x7fac9c098d80 2019-12-05T06:12:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.383+0000 7faca115c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9c097370 con 0x7fac9c06df10 2019-12-05T06:12:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.383+0000 7faca115c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9c08ad00 con 0x7fac9c0992d0 2019-12-05T06:12:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.383+0000 7fac9b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac9c06df10 0x7fac9c098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.383+0000 7faca095b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fac9c098d80 0x7fac9c088fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34016/0 (socket says 172.21.15.16:34016) 2019-12-05T06:12:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.383+0000 7faca095b700 1 -- 172.21.15.16:0/1953074651 learned_addr learned my addr 172.21.15.16:0/1953074651 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.383+0000 7fac9ad9d700 1 --2- 172.21.15.16:0/1953074651 >> v2:172.21.15.76:3300/0 conn(0x7fac9c0992d0 0x7fac9c098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:10.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.383+0000 7fac9b59e700 1 -- 172.21.15.16:0/1953074651 >> v2:172.21.15.76:3300/0 conn(0x7fac9c0992d0 msgr2=0x7fac9c098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:10.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> v2:172.21.15.76:3300/0 conn(0x7fac9c0992d0 0x7fac9c098840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:10.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac93fff700 1 -- 172.21.15.16:0/1953074651 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 772791821 0 0) 0x7fac94000940 con 0x7fac9c098d80 2019-12-05T06:12:10.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac9b59e700 1 -- 172.21.15.16:0/1953074651 >> v1:172.21.15.16:6789/0 conn(0x7fac9c098d80 legacy=0x7fac9c088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:10.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac9b59e700 1 -- 172.21.15.16:0/1953074651 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9c097a00 con 0x7fac9c06df10 2019-12-05T06:12:10.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac9ad9d700 1 --2- 172.21.15.16:0/1953074651 >> v2:172.21.15.76:3300/0 conn(0x7fac9c0992d0 0x7fac9c098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:10.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> v2:172.21.15.16:3300/0 conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> v2:172.21.15.16:3300/0 conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=READY pgs=330 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:10.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac93fff700 1 -- 172.21.15.16:0/1953074651 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac8c000b70 con 0x7fac9c06df10 2019-12-05T06:12:10.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac93fff700 1 -- 172.21.15.16:0/1953074651 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8c00eca0 con 0x7fac9c06df10 2019-12-05T06:12:10.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7faca115c700 1 -- 172.21.15.16:0/1953074651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9c097a00 con 0x7fac9c06df10 2019-12-05T06:12:10.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.384+0000 7fac93fff700 1 -- 172.21.15.16:0/1953074651 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac8c000b70 con 0x7fac9c06df10 2019-12-05T06:12:10.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.385+0000 7faca115c700 1 -- 172.21.15.16:0/1953074651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9c097a00 con 0x7fac9c06df10 2019-12-05T06:12:10.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.385+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.386+0000 7fac93fff700 1 -- 172.21.15.16:0/1953074651 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac8c00ee90 con 0x7fac9c06df10 2019-12-05T06:12:10.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.387+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.387+0000 7fac93fff700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac80023630 0x7fac80025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:10.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.387+0000 7fac93fff700 1 -- 172.21.15.16:0/1953074651 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fac8c045d80 con 0x7fac9c06df10 2019-12-05T06:12:10.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.387+0000 7fac9ad9d700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac80023630 0x7fac80025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:10.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.388+0000 7fac9ad9d700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac80023630 0x7fac80025ac0 crc :-1 s=READY pgs=154 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:10.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.400+0000 7faca115c700 1 -- 172.21.15.16:0/1953074651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9c097a00 con 0x7fac9c06df10 2019-12-05T06:12:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.408+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.408+0000 7fac93fff700 1 -- 172.21.15.16:0/1953074651 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac8c00e9d0 con 0x7fac9c06df10 2019-12-05T06:12:10.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.666+0000 7faca115c700 1 -- 172.21.15.16:0/1953074651 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac9c097a00 con 0x7fac9c06df10 2019-12-05T06:12:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.667+0000 7fac9b59e700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.667+0000 7fac93fff700 1 -- 172.21.15.16:0/1953074651 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fac8c00ee90 con 0x7fac9c06df10 2019-12-05T06:12:10.664 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2019-12-05T06:12:10.665 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:10.665 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:08.017512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":36,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 -- 172.21.15.16:0/1953074651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac80023630 msgr2=0x7fac80025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:10.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac80023630 0x7fac80025ac0 crc :-1 s=READY pgs=154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 -- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 msgr2=0x7fac9c098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 0x7fac9c098300 secure :-1 s=READY pgs=330 cs=0 l=1 rx=0x7fac8c0103a0 tx=0x7fac8c010740).stop 2019-12-05T06:12:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 -- 172.21.15.16:0/1953074651 shutdown_connections 2019-12-05T06:12:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 --2- 172.21.15.16:0/1953074651 >> v2:172.21.15.76:3300/0 conn(0x7fac9c0992d0 0x7fac9c098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac80023630 0x7fac80025ac0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:10.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 --2- 172.21.15.16:0/1953074651 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9c06df10 0x7fac9c098300 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:10.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 -- 172.21.15.16:0/1953074651 shutdown_connections 2019-12-05T06:12:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 -- 172.21.15.16:0/1953074651 wait complete. 2019-12-05T06:12:10.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:10.670+0000 7faca115c700 1 -- 172.21.15.16:0/1953074651 >> 172.21.15.16:0/1953074651 conn(0x7fac9c0714f0 msgr2=0x7fac9c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:10.677 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:12.678 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:12.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.951+0000 7fab972ca700 1 Processor -- start 2019-12-05T06:12:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.952+0000 7fab972ca700 1 -- start start 2019-12-05T06:12:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.953+0000 7fab972ca700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 0x7fab9008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.954+0000 7fab972ca700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab900958e0 con 0x7fab9008d2c0 2019-12-05T06:12:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.954+0000 7fab972ca700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab900629e0 con 0x7fab9008c6d0 2019-12-05T06:12:12.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.954+0000 7fab972ca700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab9008df90 con 0x7fab9008fef0 2019-12-05T06:12:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.954+0000 7fab95066700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fab9008d2c0 0x7fab9008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34024/0 (socket says 172.21.15.16:34024) 2019-12-05T06:12:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.954+0000 7fab95066700 1 -- 172.21.15.16:0/3316037161 learned_addr learned my addr 172.21.15.16:0/3316037161 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.954+0000 7fab96068700 1 --2- 172.21.15.16:0/3316037161 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 0x7fab9008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.954+0000 7fab94865700 1 -- 172.21.15.16:0/3316037161 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308991962 0 0) 0x7fab84001150 con 0x7fab9008d2c0 2019-12-05T06:12:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.954+0000 7fab94865700 1 -- 172.21.15.16:0/3316037161 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab78002e80 con 0x7fab9008d2c0 2019-12-05T06:12:12.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.954+0000 7fab94865700 1 -- 172.21.15.16:0/3316037161 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797476691 0 0) 0x7fab88001370 con 0x7fab9008c6d0 2019-12-05T06:12:12.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.955+0000 7fab94865700 1 -- 172.21.15.16:0/3316037161 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab78004170 con 0x7fab9008c6d0 2019-12-05T06:12:12.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.955+0000 7fab96068700 1 -- 172.21.15.16:0/3316037161 >> v1:172.21.15.76:6789/0 conn(0x7fab9008c6d0 legacy=0x7fab9008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:12.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.955+0000 7fab96068700 1 -- 172.21.15.16:0/3316037161 >> v1:172.21.15.16:6789/0 conn(0x7fab9008d2c0 legacy=0x7fab9008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.955+0000 7fab96068700 1 -- 172.21.15.16:0/3316037161 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9008e620 con 0x7fab9008fef0 2019-12-05T06:12:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.955+0000 7fab96068700 1 --2- 172.21.15.16:0/3316037161 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 0x7fab9008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fab8000af80 tx=0x7fab8000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.955+0000 7fab96068700 1 --2- 172.21.15.16:0/3316037161 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 0x7fab9008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fab8000af80 tx=0x7fab8000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.955+0000 7fab96068700 1 --2- 172.21.15.16:0/3316037161 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 0x7fab9008f720 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7fab8000af80 tx=0x7fab8000b790).ready entity=mon.2 client_cookie=47e32e5b086547a4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:12.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.956+0000 7fab96068700 1 --2- 172.21.15.16:0/3316037161 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 0x7fab9008f720 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fab8000af80 tx=0x7fab8000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.956+0000 7fab94865700 1 -- 172.21.15.16:0/3316037161 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab80007240 con 0x7fab9008fef0 2019-12-05T06:12:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.956+0000 7fab96068700 1 --2- 172.21.15.16:0/3316037161 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 0x7fab9008f720 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fab8000af80 tx=0x7fab8000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.956+0000 7fab96068700 1 --2- 172.21.15.16:0/3316037161 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 0x7fab9008f720 secure :-1 s=THROTTLE_DONE pgs=92 cs=0 l=1 rx=0x7fab8000af80 tx=0x7fab8000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.956+0000 7fab94865700 1 -- 172.21.15.16:0/3316037161 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab80000bb0 con 0x7fab9008fef0 2019-12-05T06:12:12.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.956+0000 7fab94865700 1 -- 172.21.15.16:0/3316037161 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab80009e60 con 0x7fab9008fef0 2019-12-05T06:12:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.956+0000 7fab972ca700 1 -- 172.21.15.16:0/3316037161 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 msgr2=0x7fab9008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.956+0000 7fab972ca700 1 --2- 172.21.15.16:0/3316037161 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 0x7fab9008f720 secure :-1 s=READY pgs=92 cs=0 l=1 rx=0x7fab8000af80 tx=0x7fab8000b790).stop 2019-12-05T06:12:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.957+0000 7fab972ca700 1 -- 172.21.15.16:0/3316037161 shutdown_connections 2019-12-05T06:12:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.957+0000 7fab972ca700 1 --2- 172.21.15.16:0/3316037161 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab9008fef0 0x7fab9008f720 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:12.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.958+0000 7fab972ca700 1 -- 172.21.15.16:0/3316037161 shutdown_connections 2019-12-05T06:12:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.958+0000 7fab972ca700 1 -- 172.21.15.16:0/3316037161 wait complete. 2019-12-05T06:12:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.958+0000 7fab972ca700 1 -- 172.21.15.16:0/3316037161 >> 172.21.15.16:0/3316037161 conn(0x7fab90061940 msgr2=0x7fab90096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.963+0000 7fab972ca700 1 Processor -- start 2019-12-05T06:12:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.963+0000 7fab972ca700 1 -- start start 2019-12-05T06:12:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.964+0000 7fab972ca700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fab901325e0 0x7fab9012b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.964+0000 7fab972ca700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:12.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.964+0000 7fab972ca700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab90128b20 con 0x7fab9012c740 2019-12-05T06:12:12.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.964+0000 7fab972ca700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab9012a7e0 con 0x7fab901325e0 2019-12-05T06:12:12.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.964+0000 7fab972ca700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab90129db0 con 0x7fab90154cb0 2019-12-05T06:12:12.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.964+0000 7fab95867700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fab901325e0 0x7fab9012b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:12.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.964+0000 7fab96068700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.964+0000 7fab95867700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fab901325e0 0x7fab9012b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:34950/0 (socket says 172.21.15.16:34950) 2019-12-05T06:12:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.964+0000 7fab95867700 1 -- 172.21.15.16:0/2392357856 learned_addr learned my addr 172.21.15.16:0/2392357856 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:12:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.965+0000 7fab96068700 1 -- 172.21.15.16:0/2392357856 >> v2:172.21.15.76:3300/0 conn(0x7fab901325e0 msgr2=0x7fab9012b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.965+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> v2:172.21.15.76:3300/0 conn(0x7fab901325e0 0x7fab9012b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.965+0000 7fab96068700 1 -- 172.21.15.16:0/2392357856 >> v1:172.21.15.76:6789/0 conn(0x7fab9012c740 legacy=0x7fab9012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.965+0000 7fab96068700 1 -- 172.21.15.16:0/2392357856 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab9014a820 con 0x7fab90154cb0 2019-12-05T06:12:12.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.965+0000 7fab95867700 1 --2- 172.21.15.16:0/2392357856 >> v2:172.21.15.76:3300/0 conn(0x7fab901325e0 0x7fab9012b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:12.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.965+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.965+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.965+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=READY pgs=93 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:12.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.966+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.966+0000 7fab8e7fc700 1 -- 172.21.15.16:0/2392357856 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab8000af50 con 0x7fab90154cb0 2019-12-05T06:12:12.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.966+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.966+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.966+0000 7fab8e7fc700 1 -- 172.21.15.16:0/2392357856 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab800097c0 con 0x7fab90154cb0 2019-12-05T06:12:12.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.966+0000 7fab972ca700 1 -- 172.21.15.16:0/2392357856 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9012ae70 con 0x7fab90154cb0 2019-12-05T06:12:12.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.966+0000 7fab8e7fc700 1 -- 172.21.15.16:0/2392357856 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab80009ed0 con 0x7fab90154cb0 2019-12-05T06:12:12.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.966+0000 7fab972ca700 1 -- 172.21.15.16:0/2392357856 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9014a820 con 0x7fab90154cb0 2019-12-05T06:12:12.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.967+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.968+0000 7fab8e7fc700 1 -- 172.21.15.16:0/2392357856 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fab800099b0 con 0x7fab90154cb0 2019-12-05T06:12:12.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.968+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.968+0000 7fab8e7fc700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fab70023630 0x7fab70025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:12.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.968+0000 7fab8e7fc700 1 -- 172.21.15.16:0/2392357856 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fab80045d20 con 0x7fab90154cb0 2019-12-05T06:12:12.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.968+0000 7fab95867700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fab70023630 0x7fab70025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:12.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.969+0000 7fab95867700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fab70023630 0x7fab70025ac0 crc :-1 s=READY pgs=155 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:12.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.983+0000 7fab972ca700 1 -- 172.21.15.16:0/2392357856 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9012ae70 con 0x7fab90154cb0 2019-12-05T06:12:12.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.992+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:12.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:12.992+0000 7fab8e7fc700 1 -- 172.21.15.16:0/2392357856 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fab8002cfd0 con 0x7fab90154cb0 2019-12-05T06:12:13.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.234+0000 7fab972ca700 1 -- 172.21.15.16:0/2392357856 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab9014a820 con 0x7fab90154cb0 2019-12-05T06:12:13.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.236+0000 7fab96068700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=THROTTLE_DONE pgs=93 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:13.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.236+0000 7fab8e7fc700 1 -- 172.21.15.16:0/2392357856 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fab8002cfd0 con 0x7fab90154cb0 2019-12-05T06:12:13.234 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2019-12-05T06:12:13.234 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:13.235 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:08.017512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":36,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:13.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 -- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fab70023630 msgr2=0x7fab70025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:13.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fab70023630 0x7fab70025ac0 crc :-1 s=READY pgs=155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:13.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 -- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 msgr2=0x7fab901570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:13.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 secure :-1 s=READY pgs=93 cs=0 l=1 rx=0x7fab800103c0 tx=0x7fab80010760).stop 2019-12-05T06:12:13.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 -- 172.21.15.16:0/2392357856 shutdown_connections 2019-12-05T06:12:13.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fab70023630 0x7fab70025ac0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:13.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 --2- 172.21.15.16:0/2392357856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab90154cb0 0x7fab901570b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:13.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 --2- 172.21.15.16:0/2392357856 >> v2:172.21.15.76:3300/0 conn(0x7fab901325e0 0x7fab9012b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:13.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 -- 172.21.15.16:0/2392357856 shutdown_connections 2019-12-05T06:12:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 -- 172.21.15.16:0/2392357856 wait complete. 2019-12-05T06:12:13.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:13.240+0000 7fab972ca700 1 -- 172.21.15.16:0/2392357856 >> 172.21.15.16:0/2392357856 conn(0x7fab900a2850 msgr2=0x7fab9006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:13.248 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:15.248 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.498+0000 7fe0cb8c6700 1 Processor -- start 2019-12-05T06:12:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.499+0000 7fe0cb8c6700 1 -- start start 2019-12-05T06:12:15.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.501+0000 7fe0cb8c6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 0x7fe0c408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.501+0000 7fe0cb8c6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c40958e0 con 0x7fe0c408c6d0 2019-12-05T06:12:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.501+0000 7fe0cb8c6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c40629e0 con 0x7fe0c408fef0 2019-12-05T06:12:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.501+0000 7fe0cb8c6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c408df90 con 0x7fe0c408d2c0 2019-12-05T06:12:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.501+0000 7fe0c9e63700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe0c408c6d0 0x7fe0c408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34036/0 (socket says 172.21.15.16:34036) 2019-12-05T06:12:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.501+0000 7fe0c9e63700 1 -- 172.21.15.16:0/3206298300 learned_addr learned my addr 172.21.15.16:0/3206298300 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:15.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.501+0000 7fe0c9662700 1 --2- 172.21.15.16:0/3206298300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 0x7fe0c408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.502+0000 7fe0c8e61700 1 -- 172.21.15.16:0/3206298300 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506889285 0 0) 0x7fe0c00014d0 con 0x7fe0c408c6d0 2019-12-05T06:12:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.502+0000 7fe0c8e61700 1 -- 172.21.15.16:0/3206298300 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b0002e80 con 0x7fe0c408c6d0 2019-12-05T06:12:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.502+0000 7fe0c8e61700 1 -- 172.21.15.16:0/3206298300 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470136441 0 0) 0x7fe0bc0009b0 con 0x7fe0c408fef0 2019-12-05T06:12:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.502+0000 7fe0c8e61700 1 -- 172.21.15.16:0/3206298300 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b00030b0 con 0x7fe0c408fef0 2019-12-05T06:12:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.502+0000 7fe0c8e61700 1 -- 172.21.15.16:0/3206298300 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2826460824 0 0) 0x7fe0c00014b0 con 0x7fe0c408c6d0 2019-12-05T06:12:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.502+0000 7fe0c9662700 1 -- 172.21.15.16:0/3206298300 >> v1:172.21.15.76:6789/0 conn(0x7fe0c408fef0 legacy=0x7fe0c408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:15.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.502+0000 7fe0c9662700 1 -- 172.21.15.16:0/3206298300 >> v1:172.21.15.16:6789/0 conn(0x7fe0c408c6d0 legacy=0x7fe0c408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.502+0000 7fe0c9662700 1 -- 172.21.15.16:0/3206298300 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c408e620 con 0x7fe0c408d2c0 2019-12-05T06:12:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.502+0000 7fe0c9662700 1 --2- 172.21.15.16:0/3206298300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 0x7fe0c408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe0b400af50 tx=0x7fe0b400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.503+0000 7fe0c9662700 1 --2- 172.21.15.16:0/3206298300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 0x7fe0c408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe0b400af50 tx=0x7fe0b400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.503+0000 7fe0c9662700 1 --2- 172.21.15.16:0/3206298300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 0x7fe0c408ef20 secure :-1 s=READY pgs=94 cs=0 l=1 rx=0x7fe0b400af50 tx=0x7fe0b400b760).ready entity=mon.2 client_cookie=9b79efb456eb12c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.503+0000 7fe0c9662700 1 --2- 172.21.15.16:0/3206298300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 0x7fe0c408ef20 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7fe0b400af50 tx=0x7fe0b400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.503+0000 7fe0c9662700 1 --2- 172.21.15.16:0/3206298300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 0x7fe0c408ef20 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7fe0b400af50 tx=0x7fe0b400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.503+0000 7fe0c8e61700 1 -- 172.21.15.16:0/3206298300 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0b4000960 con 0x7fe0c408d2c0 2019-12-05T06:12:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.503+0000 7fe0c9662700 1 --2- 172.21.15.16:0/3206298300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 0x7fe0c408ef20 secure :-1 s=THROTTLE_DONE pgs=94 cs=0 l=1 rx=0x7fe0b400af50 tx=0x7fe0b400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.503+0000 7fe0c8e61700 1 -- 172.21.15.16:0/3206298300 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0b40098e0 con 0x7fe0c408d2c0 2019-12-05T06:12:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.503+0000 7fe0c8e61700 1 -- 172.21.15.16:0/3206298300 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0b400a050 con 0x7fe0c408d2c0 2019-12-05T06:12:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.503+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3206298300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 msgr2=0x7fe0c408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.503+0000 7fe0cb8c6700 1 --2- 172.21.15.16:0/3206298300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 0x7fe0c408ef20 secure :-1 s=READY pgs=94 cs=0 l=1 rx=0x7fe0b400af50 tx=0x7fe0b400b760).stop 2019-12-05T06:12:15.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.505+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3206298300 shutdown_connections 2019-12-05T06:12:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.505+0000 7fe0cb8c6700 1 --2- 172.21.15.16:0/3206298300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c408d2c0 0x7fe0c408ef20 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.505+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3206298300 shutdown_connections 2019-12-05T06:12:15.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.505+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3206298300 wait complete. 2019-12-05T06:12:15.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.505+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3206298300 >> 172.21.15.16:0/3206298300 conn(0x7fe0c4061940 msgr2=0x7fe0c4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.509+0000 7fe0cb8c6700 1 Processor -- start 2019-12-05T06:12:15.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.509+0000 7fe0cb8c6700 1 -- start start 2019-12-05T06:12:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0cb8c6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c409e7d0 0x7fe0c409e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0cb8c6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe0c4156e70 0x7fe0c4159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0cb8c6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0c412cb90 con 0x7fe0c41302f0 2019-12-05T06:12:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0cb8c6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0c4131ff0 con 0x7fe0c4156e70 2019-12-05T06:12:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0cb8c6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0c41315c0 con 0x7fe0c409e7d0 2019-12-05T06:12:15.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0c9e63700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe0c41302f0 0x7fe0c409d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34042/0 (socket says 172.21.15.16:34042) 2019-12-05T06:12:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0c9e63700 1 -- 172.21.15.16:0/3154297180 learned_addr learned my addr 172.21.15.16:0/3154297180 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> v2:172.21.15.76:3300/0 conn(0x7fe0c4156e70 0x7fe0c4159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0c9662700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c409e7d0 0x7fe0c409e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0ba7fc700 1 -- 172.21.15.16:0/3154297180 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1443933261 0 0) 0x7fe0c00012c0 con 0x7fe0c41302f0 2019-12-05T06:12:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.511+0000 7fe0ba7fc700 1 -- 172.21.15.16:0/3154297180 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0ac002e80 con 0x7fe0c41302f0 2019-12-05T06:12:15.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ca664700 1 -- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c409e7d0 msgr2=0x7fe0c409e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:15.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c409e7d0 0x7fe0c409e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:15.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ca664700 1 -- 172.21.15.16:0/3154297180 >> v1:172.21.15.16:6789/0 conn(0x7fe0c41302f0 legacy=0x7fe0c409d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:15.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ca664700 1 -- 172.21.15.16:0/3154297180 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c4132680 con 0x7fe0c4156e70 2019-12-05T06:12:15.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> v2:172.21.15.76:3300/0 conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> v2:172.21.15.76:3300/0 conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=READY pgs=266 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ba7fc700 1 -- 172.21.15.16:0/3154297180 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0bc000a70 con 0x7fe0c4156e70 2019-12-05T06:12:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ba7fc700 1 -- 172.21.15.16:0/3154297180 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0bc00ee70 con 0x7fe0c4156e70 2019-12-05T06:12:15.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0ba7fc700 1 -- 172.21.15.16:0/3154297180 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0bc00f060 con 0x7fe0c4156e70 2019-12-05T06:12:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.512+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3154297180 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0c4132680 con 0x7fe0c4156e70 2019-12-05T06:12:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.513+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3154297180 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0c4132680 con 0x7fe0c4156e70 2019-12-05T06:12:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.513+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.514+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.514+0000 7fe0ba7fc700 1 -- 172.21.15.16:0/3154297180 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe0bc00f4a0 con 0x7fe0c4156e70 2019-12-05T06:12:15.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.514+0000 7fe0ba7fc700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe0ac023670 0x7fe0ac025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.515+0000 7fe0ba7fc700 1 -- 172.21.15.16:0/3154297180 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 6231+0+0 (secure 0 0 0) 0x7fe0bc045d50 con 0x7fe0c4156e70 2019-12-05T06:12:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.515+0000 7fe0c9e63700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe0ac023670 0x7fe0ac025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.515+0000 7fe0c9e63700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe0ac023670 0x7fe0ac025ae0 crc :-1 s=READY pgs=156 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:15.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.531+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3154297180 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0c4132680 con 0x7fe0c4156e70 2019-12-05T06:12:15.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.540+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.540+0000 7fe0ba7fc700 1 -- 172.21.15.16:0/3154297180 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe0bc00f290 con 0x7fe0c4156e70 2019-12-05T06:12:15.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.786+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3154297180 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0c4132680 con 0x7fe0c4156e70 2019-12-05T06:12:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.787+0000 7fe0ca664700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:15.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.787+0000 7fe0ba7fc700 1 -- 172.21.15.16:0/3154297180 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fe0bc00f4a0 con 0x7fe0c4156e70 2019-12-05T06:12:15.785 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:15.785 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4532,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":29}],"filesystems":[{"mdsmap":{"epoch":28,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:11:08.017512+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":31,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4839},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4839":{"gid":4839,"name":"a","rank":0,"incarnation":24,"state":"up:active","state_seq":36,"addr":"172.21.15.16:6833/4138078868","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4138078868},{"type":"v1","addr":"172.21.15.16:6833","nonce":4138078868}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:15.786 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 29 2019-12-05T06:12:15.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.791+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe0ac023670 msgr2=0x7fe0ac025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.791+0000 7fe0cb8c6700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe0ac023670 0x7fe0ac025ae0 crc :-1 s=READY pgs=156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.791+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 msgr2=0x7fe0c4159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.791+0000 7fe0cb8c6700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 0x7fe0c4159270 secure :-1 s=READY pgs=266 cs=0 l=1 rx=0x7fe0bc0103a0 tx=0x7fe0bc010740).stop 2019-12-05T06:12:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.791+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3154297180 shutdown_connections 2019-12-05T06:12:15.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.791+0000 7fe0cb8c6700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe0ac023670 0x7fe0ac025ae0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.791+0000 7fe0cb8c6700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe0c4156e70 0x7fe0c4159270 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.791+0000 7fe0cb8c6700 1 --2- 172.21.15.16:0/3154297180 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe0c409e7d0 0x7fe0c409e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.792+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3154297180 shutdown_connections 2019-12-05T06:12:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.792+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3154297180 wait complete. 2019-12-05T06:12:15.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:15.792+0000 7fe0cb8c6700 1 -- 172.21.15.16:0/3154297180 >> 172.21.15.16:0/3154297180 conn(0x7fe0c40a2850 msgr2=0x7fe0c406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:15.798 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:12:17.799 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:18.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.073+0000 7fd5487de700 1 Processor -- start 2019-12-05T06:12:18.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.073+0000 7fd5487de700 1 -- start start 2019-12-05T06:12:18.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.074+0000 7fd5487de700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd54008c6d0 0x7fd54008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:18.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.074+0000 7fd5487de700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.074+0000 7fd5487de700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5400958e0 con 0x7fd54008ef20 2019-12-05T06:12:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.074+0000 7fd5487de700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd54008df90 con 0x7fd54008c6d0 2019-12-05T06:12:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.074+0000 7fd5487de700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5400a3f70 con 0x7fd54008fef0 2019-12-05T06:12:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.075+0000 7fd54657a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.075+0000 7fd546d7b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd54008c6d0 0x7fd54008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.075+0000 7fd54757c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd54008ef20 0x7fd54008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:55356/0 (socket says 172.21.15.16:55356) 2019-12-05T06:12:18.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.075+0000 7fd54657a700 1 --2- 172.21.15.16:0/2327833420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:52872/0 (socket says 172.21.15.16:52872) 2019-12-05T06:12:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.075+0000 7fd54757c700 1 -- 172.21.15.16:0/2327833420 learned_addr learned my addr 172.21.15.16:0/2327833420 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.075+0000 7fd54657a700 1 -- 172.21.15.16:0/2327833420 >> v2:172.21.15.76:3300/0 conn(0x7fd54008c6d0 msgr2=0x7fd54008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.075+0000 7fd54657a700 1 --2- 172.21.15.16:0/2327833420 >> v2:172.21.15.76:3300/0 conn(0x7fd54008c6d0 0x7fd54008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.075+0000 7fd54657a700 1 -- 172.21.15.16:0/2327833420 >> v1:172.21.15.76:6789/0 conn(0x7fd54008ef20 legacy=0x7fd54008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.075+0000 7fd545d79700 1 -- 172.21.15.16:0/2327833420 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1982200570 0 0) 0x7fd538001480 con 0x7fd54008ef20 2019-12-05T06:12:18.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd54657a700 1 -- 172.21.15.16:0/2327833420 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd54008d540 con 0x7fd54008fef0 2019-12-05T06:12:18.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd546d7b700 1 --2- 172.21.15.16:0/2327833420 >> v2:172.21.15.76:3300/0 conn(0x7fd54008c6d0 0x7fd54008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:18.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd54657a700 1 --2- 172.21.15.16:0/2327833420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd53000af50 tx=0x7fd53000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd54657a700 1 --2- 172.21.15.16:0/2327833420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd53000af50 tx=0x7fd53000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd54657a700 1 --2- 172.21.15.16:0/2327833420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7fd53000af50 tx=0x7fd53000b760).ready entity=mon.2 client_cookie=a27cf29368420b5a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:18.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd54657a700 1 --2- 172.21.15.16:0/2327833420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7fd53000af50 tx=0x7fd53000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd545d79700 1 -- 172.21.15.16:0/2327833420 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd530007210 con 0x7fd54008fef0 2019-12-05T06:12:18.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd54657a700 1 --2- 172.21.15.16:0/2327833420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7fd53000af50 tx=0x7fd53000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd54657a700 1 --2- 172.21.15.16:0/2327833420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 secure :-1 s=THROTTLE_DONE pgs=95 cs=0 l=1 rx=0x7fd53000af50 tx=0x7fd53000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd545d79700 1 -- 172.21.15.16:0/2327833420 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd530007210 con 0x7fd54008fef0 2019-12-05T06:12:18.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd545d79700 1 -- 172.21.15.16:0/2327833420 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd530009ba0 con 0x7fd54008fef0 2019-12-05T06:12:18.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd5487de700 1 -- 172.21.15.16:0/2327833420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 msgr2=0x7fd54008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.076+0000 7fd5487de700 1 --2- 172.21.15.16:0/2327833420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 secure :-1 s=READY pgs=95 cs=0 l=1 rx=0x7fd53000af50 tx=0x7fd53000b760).stop 2019-12-05T06:12:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.078+0000 7fd5487de700 1 -- 172.21.15.16:0/2327833420 shutdown_connections 2019-12-05T06:12:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.078+0000 7fd5487de700 1 --2- 172.21.15.16:0/2327833420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54008fef0 0x7fd54008d000 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.078+0000 7fd5487de700 1 --2- 172.21.15.16:0/2327833420 >> v2:172.21.15.76:3300/0 conn(0x7fd54008c6d0 0x7fd54008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.078+0000 7fd5487de700 1 -- 172.21.15.16:0/2327833420 shutdown_connections 2019-12-05T06:12:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.078+0000 7fd5487de700 1 -- 172.21.15.16:0/2327833420 wait complete. 2019-12-05T06:12:18.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.078+0000 7fd5487de700 1 -- 172.21.15.16:0/2327833420 >> 172.21.15.16:0/2327833420 conn(0x7fd540061940 msgr2=0x7fd540096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.082+0000 7fd5487de700 1 Processor -- start 2019-12-05T06:12:18.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.082+0000 7fd5487de700 1 -- start start 2019-12-05T06:12:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.083+0000 7fd5487de700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd5401302f0 0x7fd54012a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.083+0000 7fd5487de700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54012b640 0x7fd54012abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.083+0000 7fd5487de700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd540131bf0 con 0x7fd54012b0f0 2019-12-05T06:12:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.083+0000 7fd5487de700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5401296e0 con 0x7fd5401302f0 2019-12-05T06:12:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.083+0000 7fd5487de700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd540128cb0 con 0x7fd54012b640 2019-12-05T06:12:18.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd54657a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54012b640 0x7fd54012abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd54757c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd54012b0f0 0x7fd5401306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34060/0 (socket says 172.21.15.16:34060) 2019-12-05T06:12:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd546d7b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd5401302f0 0x7fd54012a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd54757c700 1 -- 172.21.15.16:0/3821260077 learned_addr learned my addr 172.21.15.16:0/3821260077 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:18.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd5377fe700 1 -- 172.21.15.16:0/3821260077 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4271000781 0 0) 0x7fd538001690 con 0x7fd54012b0f0 2019-12-05T06:12:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd5377fe700 1 -- 172.21.15.16:0/3821260077 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd528002e80 con 0x7fd54012b0f0 2019-12-05T06:12:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd546d7b700 1 -- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54012b640 msgr2=0x7fd54012abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54012b640 0x7fd54012abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd546d7b700 1 -- 172.21.15.16:0/3821260077 >> v1:172.21.15.16:6789/0 conn(0x7fd54012b0f0 legacy=0x7fd5401306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd546d7b700 1 -- 172.21.15.16:0/3821260077 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd540129d70 con 0x7fd5401302f0 2019-12-05T06:12:18.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.084+0000 7fd54657a700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54012b640 0x7fd54012abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:18.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> v2:172.21.15.76:3300/0 conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> v2:172.21.15.76:3300/0 conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=READY pgs=268 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd5377fe700 1 -- 172.21.15.16:0/3821260077 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd53c009bc0 con 0x7fd5401302f0 2019-12-05T06:12:18.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd5377fe700 1 -- 172.21.15.16:0/3821260077 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd53c0097e0 con 0x7fd5401302f0 2019-12-05T06:12:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd5377fe700 1 -- 172.21.15.16:0/3821260077 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd53c009db0 con 0x7fd5401302f0 2019-12-05T06:12:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd5487de700 1 -- 172.21.15.16:0/3821260077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd540129d70 con 0x7fd5401302f0 2019-12-05T06:12:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.085+0000 7fd5487de700 1 -- 172.21.15.16:0/3821260077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd540129d70 con 0x7fd5401302f0 2019-12-05T06:12:18.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.086+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.087+0000 7fd5377fe700 1 -- 172.21.15.16:0/3821260077 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd53c0099d0 con 0x7fd5401302f0 2019-12-05T06:12:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.087+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.087+0000 7fd5377fe700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd528023670 0x7fd528025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.087+0000 7fd5377fe700 1 -- 172.21.15.16:0/3821260077 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fd53c045d60 con 0x7fd5401302f0 2019-12-05T06:12:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.088+0000 7fd54657a700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd528023670 0x7fd528025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.088+0000 7fd54657a700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd528023670 0x7fd528025ae0 crc :-1 s=READY pgs=157 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:18.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.088+0000 7fd5487de700 1 -- 172.21.15.16:0/3821260077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd540129d70 con 0x7fd5401302f0 2019-12-05T06:12:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.097+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.097+0000 7fd5377fe700 1 -- 172.21.15.16:0/3821260077 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd53c0095d0 con 0x7fd5401302f0 2019-12-05T06:12:18.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.316+0000 7fd5487de700 1 -- 172.21.15.16:0/3821260077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd540129d70 con 0x7fd5401302f0 2019-12-05T06:12:18.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.318+0000 7fd546d7b700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.318+0000 7fd5377fe700 1 -- 172.21.15.16:0/3821260077 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+1768 (secure 0 0 0) 0x7fd53c0099a0 con 0x7fd5401302f0 2019-12-05T06:12:18.316 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2019-12-05T06:12:18.316 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:18.316 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:17.986349+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:18.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.323+0000 7fd5487de700 1 -- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd528023670 msgr2=0x7fd528025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.323+0000 7fd5487de700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd528023670 0x7fd528025ae0 crc :-1 s=READY pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.323+0000 7fd5487de700 1 -- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 msgr2=0x7fd54012a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.323+0000 7fd5487de700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 0x7fd54012a670 secure :-1 s=READY pgs=268 cs=0 l=1 rx=0x7fd53c0103a0 tx=0x7fd53c010740).stop 2019-12-05T06:12:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.324+0000 7fd5487de700 1 -- 172.21.15.16:0/3821260077 shutdown_connections 2019-12-05T06:12:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.324+0000 7fd5487de700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd528023670 0x7fd528025ae0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.324+0000 7fd5487de700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd54012b640 0x7fd54012abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.324+0000 7fd5487de700 1 --2- 172.21.15.16:0/3821260077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd5401302f0 0x7fd54012a670 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.324+0000 7fd5487de700 1 -- 172.21.15.16:0/3821260077 shutdown_connections 2019-12-05T06:12:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.324+0000 7fd5487de700 1 -- 172.21.15.16:0/3821260077 wait complete. 2019-12-05T06:12:18.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.324+0000 7fd5487de700 1 -- 172.21.15.16:0/3821260077 >> 172.21.15.16:0/3821260077 conn(0x7fd5400a2850 msgr2=0x7fd54006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:18.337 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:12:18.337 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:12:18.337 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.556+0000 7fdde9376700 1 Processor -- start 2019-12-05T06:12:18.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.556+0000 7fdde9376700 1 -- start start 2019-12-05T06:12:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde9376700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde408c6d0 0x7fdde408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde9376700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde408fef0 0x7fdde408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde9376700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdde408ef20 0x7fdde408f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde9376700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde40a3f70 con 0x7fdde408fef0 2019-12-05T06:12:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde9376700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde408d830 con 0x7fdde408ef20 2019-12-05T06:12:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde9376700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde408d540 con 0x7fdde408c6d0 2019-12-05T06:12:18.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde408fef0 0x7fdde408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde3fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdde408ef20 0x7fdde408f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde2ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde408fef0 0x7fdde408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48896/0 (socket says 172.21.15.16:48896) 2019-12-05T06:12:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde37fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde408c6d0 0x7fdde408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.557+0000 7fdde2ffd700 1 -- 172.21.15.16:0/2352224037 learned_addr learned my addr 172.21.15.16:0/2352224037 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:12:18.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.558+0000 7fdde2ffd700 1 -- 172.21.15.16:0/2352224037 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde408c6d0 msgr2=0x7fdde408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.558+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/2352224037 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde408c6d0 0x7fdde408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.558+0000 7fdde2ffd700 1 -- 172.21.15.16:0/2352224037 >> v2:172.21.15.76:3300/0 conn(0x7fdde408ef20 msgr2=0x7fdde408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.558+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/2352224037 >> v2:172.21.15.76:3300/0 conn(0x7fdde408ef20 0x7fdde408f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.558+0000 7fdde2ffd700 1 -- 172.21.15.16:0/2352224037 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde408e620 con 0x7fdde408fef0 2019-12-05T06:12:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.558+0000 7fdde3fff700 1 --2- 172.21.15.16:0/2352224037 >> v2:172.21.15.76:3300/0 conn(0x7fdde408ef20 0x7fdde408f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.558+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/2352224037 >> v2:172.21.15.16:3300/0 conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fddcc00f740 tx=0x7fddcc00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.558+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/2352224037 >> v2:172.21.15.16:3300/0 conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fddcc00f740 tx=0x7fddcc00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.559+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/2352224037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=READY pgs=335 cs=0 l=1 rx=0x7fddcc00f740 tx=0x7fddcc00f570).ready entity=mon.0 client_cookie=df2f275b24c8a85f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.559+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/2352224037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7fddcc00f740 tx=0x7fddcc00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.559+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/2352224037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7fddcc00f740 tx=0x7fddcc00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.559+0000 7fdde27fc700 1 -- 172.21.15.16:0/2352224037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddcc000960 con 0x7fdde408fef0 2019-12-05T06:12:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.559+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/2352224037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7fddcc00f740 tx=0x7fddcc00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.559+0000 7fdde27fc700 1 -- 172.21.15.16:0/2352224037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc000c90 con 0x7fdde408fef0 2019-12-05T06:12:18.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.559+0000 7fdde27fc700 1 -- 172.21.15.16:0/2352224037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddcc0013a0 con 0x7fdde408fef0 2019-12-05T06:12:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.559+0000 7fdde9376700 1 -- 172.21.15.16:0/2352224037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 msgr2=0x7fdde408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.559+0000 7fdde9376700 1 --2- 172.21.15.16:0/2352224037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 secure :-1 s=READY pgs=335 cs=0 l=1 rx=0x7fddcc00f740 tx=0x7fddcc00f570).stop 2019-12-05T06:12:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.561+0000 7fdde9376700 1 -- 172.21.15.16:0/2352224037 shutdown_connections 2019-12-05T06:12:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.561+0000 7fdde9376700 1 --2- 172.21.15.16:0/2352224037 >> v2:172.21.15.76:3300/0 conn(0x7fdde408ef20 0x7fdde408f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.561+0000 7fdde9376700 1 --2- 172.21.15.16:0/2352224037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdde408fef0 0x7fdde408d000 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.561+0000 7fdde9376700 1 --2- 172.21.15.16:0/2352224037 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde408c6d0 0x7fdde408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:18.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.561+0000 7fdde9376700 1 -- 172.21.15.16:0/2352224037 shutdown_connections 2019-12-05T06:12:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.561+0000 7fdde9376700 1 -- 172.21.15.16:0/2352224037 wait complete. 2019-12-05T06:12:18.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.561+0000 7fdde9376700 1 -- 172.21.15.16:0/2352224037 >> 172.21.15.16:0/2352224037 conn(0x7fdde4061940 msgr2=0x7fdde4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:18.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.565+0000 7fdde9376700 1 Processor -- start 2019-12-05T06:12:18.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.566+0000 7fdde9376700 1 -- start start 2019-12-05T06:12:18.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde9376700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdde40992d0 0x7fdde4098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:18.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde9376700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:18.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde9376700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdde4089fb0 con 0x7fdde406df10 2019-12-05T06:12:18.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde9376700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdde4097370 con 0x7fdde40992d0 2019-12-05T06:12:18.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde9376700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdde408ad00 con 0x7fdde4156cb0 2019-12-05T06:12:18.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdde406df10 0x7fdde4098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34070/0 (socket says 172.21.15.16:34070) 2019-12-05T06:12:18.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde37fe700 1 -- 172.21.15.16:0/1709898514 learned_addr learned my addr 172.21.15.16:0/1709898514 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:18.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde2ffd700 1 --2- 172.21.15.16:0/1709898514 >> v2:172.21.15.16:3300/0 conn(0x7fdde40992d0 0x7fdde4098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde37fe700 1 -- 172.21.15.16:0/1709898514 >> v1:172.21.15.16:6789/0 conn(0x7fdde406df10 legacy=0x7fdde4098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:12:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde37fe700 1 -- 172.21.15.16:0/1709898514 >> v1:172.21.15.16:6789/0 conn(0x7fdde406df10 legacy=0x7fdde4098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:12:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde37fe700 1 --1- 172.21.15.16:0/1709898514 >> v1:172.21.15.16:6789/0 conn(0x7fdde406df10 0x7fdde4098300 :-1 s=OPENED pgs=336 cs=1 l=1).handle_message read tag failed 2019-12-05T06:12:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.567+0000 7fdde37fe700 1 --1- 172.21.15.16:0/1709898514 >> v1:172.21.15.16:6789/0 conn(0x7fdde406df10 0x7fdde4098300 :-1 s=OPENED pgs=336 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:12:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde3fff700 1 -- 172.21.15.16:0/1709898514 >> v2:172.21.15.16:3300/0 conn(0x7fdde40992d0 msgr2=0x7fdde4098b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:18.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> v2:172.21.15.16:3300/0 conn(0x7fdde40992d0 0x7fdde4098b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde3fff700 1 -- 172.21.15.16:0/1709898514 >> v1:172.21.15.16:6789/0 conn(0x7fdde406df10 legacy=0x7fdde4098300 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:12:18.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde3fff700 1 -- 172.21.15.16:0/1709898514 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdde4097a00 con 0x7fdde4156cb0 2019-12-05T06:12:18.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=READY pgs=96 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:18.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdddbfff700 1 -- 172.21.15.16:0/1709898514 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddd400ab90 con 0x7fdde4156cb0 2019-12-05T06:12:18.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdddbfff700 1 -- 172.21.15.16:0/1709898514 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddd400a670 con 0x7fdde4156cb0 2019-12-05T06:12:18.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde9376700 1 -- 172.21.15.16:0/1709898514 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdde4097a00 con 0x7fdde4156cb0 2019-12-05T06:12:18.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdddbfff700 1 -- 172.21.15.16:0/1709898514 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddd4009b80 con 0x7fdde4156cb0 2019-12-05T06:12:18.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.568+0000 7fdde9376700 1 -- 172.21.15.16:0/1709898514 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdde4097a00 con 0x7fdde4156cb0 2019-12-05T06:12:18.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.569+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.570+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.570+0000 7fdddbfff700 1 -- 172.21.15.16:0/1709898514 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fddd4009650 con 0x7fdde4156cb0 2019-12-05T06:12:18.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.570+0000 7fdddbfff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023630 0x7fddc8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:18.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.570+0000 7fdddbfff700 1 -- 172.21.15.16:0/1709898514 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fddd4045c50 con 0x7fdde4156cb0 2019-12-05T06:12:18.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.571+0000 7fdde37fe700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023630 0x7fddc8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.571+0000 7fdde9376700 1 -- 172.21.15.16:0/1709898514 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdde4097a00 con 0x7fdde4156cb0 2019-12-05T06:12:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.571+0000 7fdde37fe700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023630 0x7fddc8025ac0 crc :-1 s=READY pgs=158 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.578+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.578+0000 7fdddbfff700 1 -- 172.21.15.16:0/1709898514 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fddd4000a70 con 0x7fdde4156cb0 2019-12-05T06:12:18.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.825+0000 7fdde9376700 1 -- 172.21.15.16:0/1709898514 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdde4097a00 con 0x7fdde4156cb0 2019-12-05T06:12:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.827+0000 7fdde3fff700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=THROTTLE_DONE pgs=96 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.827+0000 7fdddbfff700 1 -- 172.21.15.16:0/1709898514 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+1768 (secure 0 0 0) 0x7fddd40095b0 con 0x7fdde4156cb0 2019-12-05T06:12:18.823 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 30 2019-12-05T06:12:18.823 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:18.824 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":30,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:17.986349+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:replay","state_seq":1,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.828+0000 7fdde9376700 1 -- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023630 msgr2=0x7fddc8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.828+0000 7fdde9376700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023630 0x7fddc8025ac0 crc :-1 s=READY pgs=158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.828+0000 7fdde9376700 1 -- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 msgr2=0x7fdde41590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.828+0000 7fdde9376700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 secure :-1 s=READY pgs=96 cs=0 l=1 rx=0x7fddd40102d0 tx=0x7fddd4010670).stop 2019-12-05T06:12:18.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.829+0000 7fdde9376700 1 -- 172.21.15.16:0/1709898514 shutdown_connections 2019-12-05T06:12:18.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.829+0000 7fdde9376700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddc8023630 0x7fddc8025ac0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.829+0000 7fdde9376700 1 --2- 172.21.15.16:0/1709898514 >> v2:172.21.15.16:3300/0 conn(0x7fdde40992d0 0x7fdde4098b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.829+0000 7fdde9376700 1 --2- 172.21.15.16:0/1709898514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdde4156cb0 0x7fdde41590b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:18.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.829+0000 7fdde9376700 1 -- 172.21.15.16:0/1709898514 shutdown_connections 2019-12-05T06:12:18.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.829+0000 7fdde9376700 1 -- 172.21.15.16:0/1709898514 wait complete. 2019-12-05T06:12:18.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:18.829+0000 7fdde9376700 1 -- 172.21.15.16:0/1709898514 >> 172.21.15.16:0/1709898514 conn(0x7fdde40714f0 msgr2=0x7fdde409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:18.840 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 30, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 30, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_4532": { "addr": "172.21.15.76:6835/913501478", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 913501478, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 913501478, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 4532, "incarnation": 30, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 41, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:12:17.986349+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 4532 } } } ], "standbys": [] } 2019-12-05T06:12:20.841 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:21.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.093+0000 7f7e76a94700 1 Processor -- start 2019-12-05T06:12:21.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.093+0000 7f7e76a94700 1 -- start start 2019-12-05T06:12:21.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.095+0000 7f7e76a94700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7e7008d2c0 0x7f7e7008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:21.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.095+0000 7f7e76a94700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e700958e0 con 0x7f7e7008c6d0 2019-12-05T06:12:21.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.095+0000 7f7e76a94700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e700629e0 con 0x7f7e7008fef0 2019-12-05T06:12:21.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.095+0000 7f7e76a94700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e7008df90 con 0x7f7e7008d2c0 2019-12-05T06:12:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.095+0000 7f7e75031700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e7008c6d0 0x7f7e7008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34082/0 (socket says 172.21.15.16:34082) 2019-12-05T06:12:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.095+0000 7f7e75031700 1 -- 172.21.15.16:0/3713560809 learned_addr learned my addr 172.21.15.16:0/3713560809 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.095+0000 7f7e74830700 1 --2- 172.21.15.16:0/3713560809 >> v2:172.21.15.76:3300/0 conn(0x7f7e7008d2c0 0x7f7e7008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.095+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3007301508 0 0) 0x7f7e60001150 con 0x7f7e7008c6d0 2019-12-05T06:12:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e58002e80 con 0x7f7e7008c6d0 2019-12-05T06:12:21.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1274257635 0 0) 0x7f7e60001130 con 0x7f7e7008c6d0 2019-12-05T06:12:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 >> v2:172.21.15.76:3300/0 conn(0x7f7e7008d2c0 msgr2=0x7f7e7008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 --2- 172.21.15.16:0/3713560809 >> v2:172.21.15.76:3300/0 conn(0x7f7e7008d2c0 0x7f7e7008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 >> v1:172.21.15.76:6789/0 conn(0x7f7e7008fef0 legacy=0x7f7e7008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e7008e620 con 0x7f7e7008c6d0 2019-12-05T06:12:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f7e60001660 con 0x7f7e7008c6d0 2019-12-05T06:12:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 >> v1:172.21.15.16:6789/0 conn(0x7f7e7008c6d0 legacy=0x7f7e7008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:21.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e58003b20 0x7f7e58005f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e58006700 0x7f7e58008bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 0x7f7e5800b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5800c0b0 con 0x7f7e58009430 2019-12-05T06:12:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580502b0 con 0x7f7e58006700 2019-12-05T06:12:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580504e0 con 0x7f7e58003b20 2019-12-05T06:12:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.096+0000 7f7e75832700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 0x7f7e5800b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:21.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e74830700 1 --2- 172.21.15.16:0/3713560809 >> v2:172.21.15.76:3300/0 conn(0x7f7e7008d2c0 0x7f7e7008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:21.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75031700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e58003b20 0x7f7e58005f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:21.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 -- 172.21.15.16:0/3713560809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e58003b20 msgr2=0x7f7e58005f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:21.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e58003b20 0x7f7e58005f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:21.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e74830700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e58006700 0x7f7e58008bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:21.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 -- 172.21.15.16:0/3713560809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e58006700 msgr2=0x7f7e58008bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:21.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e58006700 0x7f7e58008bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 -- 172.21.15.16:0/3713560809 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e58050790 con 0x7f7e58009430 2019-12-05T06:12:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e74830700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7e58006700 0x7f7e58008bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:12:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 0x7f7e5800b900 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7e6800b2e0 tx=0x7f7e6800ba70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 -- 172.21.15.16:0/3713560809 reap_dead start 2019-12-05T06:12:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 0x7f7e5800b900 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7e6800b2e0 tx=0x7f7e6800ba70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 0x7f7e5800b900 secure :-1 s=READY pgs=339 cs=0 l=1 rx=0x7f7e6800b2e0 tx=0x7f7e6800ba70).ready entity=mon.0 client_cookie=6ac1f8801e3fdcf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:21.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 0x7f7e5800b900 secure :-1 s=THROTTLE_DONE pgs=339 cs=0 l=1 rx=0x7f7e6800b2e0 tx=0x7f7e6800ba70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e68000df0 con 0x7f7e58009430 2019-12-05T06:12:21.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 0x7f7e5800b900 secure :-1 s=THROTTLE_DONE pgs=339 cs=0 l=1 rx=0x7f7e6800b2e0 tx=0x7f7e6800ba70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.097+0000 7f7e75832700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 0x7f7e5800b900 secure :-1 s=THROTTLE_DONE pgs=339 cs=0 l=1 rx=0x7f7e6800b2e0 tx=0x7f7e6800ba70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.098+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e68009940 con 0x7f7e58009430 2019-12-05T06:12:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.098+0000 7f7e6ffff700 1 -- 172.21.15.16:0/3713560809 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e6800a330 con 0x7f7e58009430 2019-12-05T06:12:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.098+0000 7f7e76a94700 1 -- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 msgr2=0x7f7e5800b900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.098+0000 7f7e76a94700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 0x7f7e5800b900 secure :-1 s=READY pgs=339 cs=0 l=1 rx=0x7f7e6800b2e0 tx=0x7f7e6800ba70).stop 2019-12-05T06:12:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.098+0000 7f7e76a94700 1 -- 172.21.15.16:0/3713560809 shutdown_connections 2019-12-05T06:12:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.098+0000 7f7e76a94700 1 --2- 172.21.15.16:0/3713560809 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e58009430 0x7f7e5800b900 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.099+0000 7f7e76a94700 1 -- 172.21.15.16:0/3713560809 shutdown_connections 2019-12-05T06:12:21.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.099+0000 7f7e76a94700 1 -- 172.21.15.16:0/3713560809 wait complete. 2019-12-05T06:12:21.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.099+0000 7f7e76a94700 1 -- 172.21.15.16:0/3713560809 >> 172.21.15.16:0/3713560809 conn(0x7f7e70061940 msgr2=0x7f7e70096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.103+0000 7f7e76a94700 1 Processor -- start 2019-12-05T06:12:21.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.104+0000 7f7e76a94700 1 -- start start 2019-12-05T06:12:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.105+0000 7f7e76a94700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e7012b640 0x7f7e7012ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.105+0000 7f7e76a94700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e70131bf0 con 0x7f7e70153ba0 2019-12-05T06:12:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.105+0000 7f7e76a94700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e701316f0 con 0x7f7e701302f0 2019-12-05T06:12:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.105+0000 7f7e76a94700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e701296e0 con 0x7f7e7012b640 2019-12-05T06:12:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.105+0000 7f7e74830700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e7012b640 0x7f7e7012ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.105+0000 7f7e74830700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e7012b640 0x7f7e7012ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48928/0 (socket says 172.21.15.16:48928) 2019-12-05T06:12:21.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.105+0000 7f7e74830700 1 -- 172.21.15.16:0/137616355 learned_addr learned my addr 172.21.15.16:0/137616355 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:12:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.106+0000 7f7e6dffb700 1 -- 172.21.15.16:0/137616355 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134473655 0 0) 0x7f7e60001900 con 0x7f7e701302f0 2019-12-05T06:12:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.106+0000 7f7e6dffb700 1 -- 172.21.15.16:0/137616355 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e50002e80 con 0x7f7e701302f0 2019-12-05T06:12:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.106+0000 7f7e6dffb700 1 -- 172.21.15.16:0/137616355 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968536310 0 0) 0x7f7e6800b390 con 0x7f7e70153ba0 2019-12-05T06:12:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.106+0000 7f7e74830700 1 -- 172.21.15.16:0/137616355 >> v1:172.21.15.76:6789/0 conn(0x7f7e701302f0 legacy=0x7f7e7012a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:21.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.106+0000 7f7e74830700 1 -- 172.21.15.16:0/137616355 >> v1:172.21.15.16:6789/0 conn(0x7f7e70153ba0 legacy=0x7f7e70155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.106+0000 7f7e74830700 1 -- 172.21.15.16:0/137616355 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e70129d70 con 0x7f7e7012b640 2019-12-05T06:12:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.106+0000 7f7e74830700 1 --2- 172.21.15.16:0/137616355 >> v2:172.21.15.16:3300/0 conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.106+0000 7f7e74830700 1 --2- 172.21.15.16:0/137616355 >> v2:172.21.15.16:3300/0 conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.106+0000 7f7e74830700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=READY pgs=341 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.107+0000 7f7e74830700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=THROTTLE_DONE pgs=341 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.107+0000 7f7e6dffb700 1 -- 172.21.15.16:0/137616355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e64009960 con 0x7f7e7012b640 2019-12-05T06:12:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.107+0000 7f7e74830700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=THROTTLE_DONE pgs=341 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.107+0000 7f7e74830700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=THROTTLE_DONE pgs=341 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.107+0000 7f7e6dffb700 1 -- 172.21.15.16:0/137616355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e64000940 con 0x7f7e7012b640 2019-12-05T06:12:21.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.107+0000 7f7e76a94700 1 -- 172.21.15.16:0/137616355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e70129d70 con 0x7f7e7012b640 2019-12-05T06:12:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.107+0000 7f7e6dffb700 1 -- 172.21.15.16:0/137616355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e64009f40 con 0x7f7e7012b640 2019-12-05T06:12:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.107+0000 7f7e76a94700 1 -- 172.21.15.16:0/137616355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e70129d70 con 0x7f7e7012b640 2019-12-05T06:12:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.107+0000 7f7e74830700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=THROTTLE_DONE pgs=341 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.108+0000 7f7e6dffb700 1 -- 172.21.15.16:0/137616355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7e640096c0 con 0x7f7e7012b640 2019-12-05T06:12:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.108+0000 7f7e74830700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=THROTTLE_DONE pgs=341 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.108+0000 7f7e6dffb700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e50023680 0x7f7e50025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.108+0000 7f7e6dffb700 1 -- 172.21.15.16:0/137616355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f7e640460e0 con 0x7f7e7012b640 2019-12-05T06:12:21.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.108+0000 7f7e75031700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e50023680 0x7f7e50025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:21.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.109+0000 7f7e75031700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e50023680 0x7f7e50025af0 crc :-1 s=READY pgs=159 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:21.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.124+0000 7f7e76a94700 1 -- 172.21.15.16:0/137616355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e70129d70 con 0x7f7e7012b640 2019-12-05T06:12:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.130+0000 7f7e74830700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=THROTTLE_DONE pgs=341 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.130+0000 7f7e6dffb700 1 -- 172.21.15.16:0/137616355 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7e64000940 con 0x7f7e7012b640 2019-12-05T06:12:21.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.361+0000 7f7e76a94700 1 -- 172.21.15.16:0/137616355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e70129d70 con 0x7f7e7012b640 2019-12-05T06:12:21.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.362+0000 7f7e74830700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=THROTTLE_DONE pgs=341 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.362+0000 7f7e6dffb700 1 -- 172.21.15.16:0/137616355 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1769 (secure 0 0 0) 0x7f7e64000940 con 0x7f7e7012b640 2019-12-05T06:12:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 32 2019-12-05T06:12:21.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:21.364 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":32,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":32,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:21.019186+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.366+0000 7f7e76a94700 1 -- 172.21.15.16:0/137616355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e50023680 msgr2=0x7f7e50025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:21.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.366+0000 7f7e76a94700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e50023680 0x7f7e50025af0 crc :-1 s=READY pgs=159 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.366+0000 7f7e76a94700 1 -- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 msgr2=0x7f7e7012ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.366+0000 7f7e76a94700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 0x7f7e7012ae70 secure :-1 s=READY pgs=341 cs=0 l=1 rx=0x7f7e64010420 tx=0x7f7e640107c0).stop 2019-12-05T06:12:21.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.367+0000 7f7e76a94700 1 -- 172.21.15.16:0/137616355 shutdown_connections 2019-12-05T06:12:21.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.367+0000 7f7e76a94700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e50023680 0x7f7e50025af0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:21.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.367+0000 7f7e76a94700 1 --2- 172.21.15.16:0/137616355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e7012b640 0x7f7e7012ae70 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:21.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.367+0000 7f7e76a94700 1 -- 172.21.15.16:0/137616355 shutdown_connections 2019-12-05T06:12:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.367+0000 7f7e76a94700 1 -- 172.21.15.16:0/137616355 wait complete. 2019-12-05T06:12:21.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:21.367+0000 7f7e76a94700 1 -- 172.21.15.16:0/137616355 >> 172.21.15.16:0/137616355 conn(0x7f7e700a2850 msgr2=0x7f7e7006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:21.376 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:12:23.377 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.556+0000 7f4322b6c700 1 Processor -- start 2019-12-05T06:12:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.556+0000 7f4322b6c700 1 -- start start 2019-12-05T06:12:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.557+0000 7f4322b6c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f431c08c6d0 0x7f431c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.557+0000 7f4322b6c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c08fb80 con 0x7f431c08fef0 2019-12-05T06:12:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.557+0000 7f4322b6c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c0958e0 con 0x7f431c08f720 2019-12-05T06:12:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.557+0000 7f4322b6c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431c08df90 con 0x7f431c08c6d0 2019-12-05T06:12:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.557+0000 7f4321109700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f431c08c6d0 0x7f431c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.557+0000 7f4320908700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f431c08fef0 0x7f431c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34112/0 (socket says 172.21.15.16:34112) 2019-12-05T06:12:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.557+0000 7f4320908700 1 -- 172.21.15.16:0/1922795463 learned_addr learned my addr 172.21.15.16:0/1922795463 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4321109700 1 -- 172.21.15.16:0/1922795463 >> v1:172.21.15.76:6789/0 conn(0x7f431c08f720 legacy=0x7f431c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4321109700 1 -- 172.21.15.16:0/1922795463 >> v1:172.21.15.16:6789/0 conn(0x7f431c08fef0 legacy=0x7f431c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4313fff700 1 -- 172.21.15.16:0/1922795463 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1969358147 0 0) 0x7f4318001150 con 0x7f431c08fef0 2019-12-05T06:12:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4321109700 1 -- 172.21.15.16:0/1922795463 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c08e620 con 0x7f431c08c6d0 2019-12-05T06:12:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4321109700 1 --2- 172.21.15.16:0/1922795463 >> v2:172.21.15.16:3300/0 conn(0x7f431c08c6d0 0x7f431c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f430c00af50 tx=0x7f430c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4321109700 1 --2- 172.21.15.16:0/1922795463 >> v2:172.21.15.16:3300/0 conn(0x7f431c08c6d0 0x7f431c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f430c00af50 tx=0x7f430c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4321109700 1 --2- 172.21.15.16:0/1922795463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c08c6d0 0x7f431c08cac0 secure :-1 s=READY pgs=344 cs=0 l=1 rx=0x7f430c00af50 tx=0x7f430c00b760).ready entity=mon.0 client_cookie=65deb8306aa3c042 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4321109700 1 --2- 172.21.15.16:0/1922795463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c08c6d0 0x7f431c08cac0 secure :-1 s=THROTTLE_DONE pgs=344 cs=0 l=1 rx=0x7f430c00af50 tx=0x7f430c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4313fff700 1 -- 172.21.15.16:0/1922795463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c000960 con 0x7f431c08c6d0 2019-12-05T06:12:23.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4321109700 1 --2- 172.21.15.16:0/1922795463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c08c6d0 0x7f431c08cac0 secure :-1 s=THROTTLE_DONE pgs=344 cs=0 l=1 rx=0x7f430c00af50 tx=0x7f430c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4321109700 1 --2- 172.21.15.16:0/1922795463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c08c6d0 0x7f431c08cac0 secure :-1 s=THROTTLE_DONE pgs=344 cs=0 l=1 rx=0x7f430c00af50 tx=0x7f430c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4313fff700 1 -- 172.21.15.16:0/1922795463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c009860 con 0x7f431c08c6d0 2019-12-05T06:12:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4313fff700 1 -- 172.21.15.16:0/1922795463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c000960 con 0x7f431c08c6d0 2019-12-05T06:12:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4322b6c700 1 -- 172.21.15.16:0/1922795463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c08c6d0 msgr2=0x7f431c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.558+0000 7f4322b6c700 1 --2- 172.21.15.16:0/1922795463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c08c6d0 0x7f431c08cac0 secure :-1 s=READY pgs=344 cs=0 l=1 rx=0x7f430c00af50 tx=0x7f430c00b760).stop 2019-12-05T06:12:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.559+0000 7f4322b6c700 1 -- 172.21.15.16:0/1922795463 shutdown_connections 2019-12-05T06:12:23.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.559+0000 7f4322b6c700 1 --2- 172.21.15.16:0/1922795463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c08c6d0 0x7f431c08cac0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.559+0000 7f4322b6c700 1 -- 172.21.15.16:0/1922795463 shutdown_connections 2019-12-05T06:12:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.559+0000 7f4322b6c700 1 -- 172.21.15.16:0/1922795463 wait complete. 2019-12-05T06:12:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.559+0000 7f4322b6c700 1 -- 172.21.15.16:0/1922795463 >> 172.21.15.16:0/1922795463 conn(0x7f431c061940 msgr2=0x7f431c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.561+0000 7f4322b6c700 1 Processor -- start 2019-12-05T06:12:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.561+0000 7f4322b6c700 1 -- start start 2019-12-05T06:12:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.562+0000 7f4322b6c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f431c09b850 0x7f431c131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:23.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.562+0000 7f4322b6c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f431c154b00 0x7f431c156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.562+0000 7f4322b6c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431c098da0 con 0x7f431c132d00 2019-12-05T06:12:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.562+0000 7f4322b6c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431c09aa60 con 0x7f431c154b00 2019-12-05T06:12:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.562+0000 7f4322b6c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431c09a030 con 0x7f431c09b850 2019-12-05T06:12:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f431c154b00 0x7f431c156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f431c154b00 0x7f431c156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:48954/0 (socket says 172.21.15.16:48954) 2019-12-05T06:12:23.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 -- 172.21.15.16:0/1673340452 learned_addr learned my addr 172.21.15.16:0/1673340452 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:12:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f4320908700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f431c132d00 0x7f431c132270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34120/0 (socket says 172.21.15.16:34120) 2019-12-05T06:12:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f4321109700 1 --2- 172.21.15.16:0/1673340452 >> v2:172.21.15.76:3300/0 conn(0x7f431c09b850 0x7f431c131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 -- 172.21.15.16:0/1673340452 >> v2:172.21.15.76:3300/0 conn(0x7f431c09b850 msgr2=0x7f431c131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> v2:172.21.15.76:3300/0 conn(0x7f431c09b850 0x7f431c131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 -- 172.21.15.16:0/1673340452 >> v1:172.21.15.16:6789/0 conn(0x7f431c132d00 legacy=0x7f431c132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 -- 172.21.15.16:0/1673340452 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431c09b0f0 con 0x7f431c154b00 2019-12-05T06:12:23.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f4311ffb700 1 -- 172.21.15.16:0/1673340452 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2626558378 0 0) 0x7f4318001100 con 0x7f431c132d00 2019-12-05T06:12:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> v2:172.21.15.16:3300/0 conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> v2:172.21.15.16:3300/0 conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=READY pgs=346 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f4311ffb700 1 -- 172.21.15.16:0/1673340452 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4314000cc0 con 0x7f431c154b00 2019-12-05T06:12:23.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f4311ffb700 1 -- 172.21.15.16:0/1673340452 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4314000cc0 con 0x7f431c154b00 2019-12-05T06:12:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f4322b6c700 1 -- 172.21.15.16:0/1673340452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431c09b0f0 con 0x7f431c154b00 2019-12-05T06:12:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f4311ffb700 1 -- 172.21.15.16:0/1673340452 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43140017c0 con 0x7f431c154b00 2019-12-05T06:12:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.563+0000 7f4322b6c700 1 -- 172.21.15.16:0/1673340452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431c09b0f0 con 0x7f431c154b00 2019-12-05T06:12:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.564+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.564+0000 7f4311ffb700 1 -- 172.21.15.16:0/1673340452 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f43140013e0 con 0x7f431c154b00 2019-12-05T06:12:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.564+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.564+0000 7f4311ffb700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023630 0x7f42fc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.564+0000 7f4311ffb700 1 -- 172.21.15.16:0/1673340452 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f4314046150 con 0x7f431c154b00 2019-12-05T06:12:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.565+0000 7f4321109700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023630 0x7f42fc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:23.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.565+0000 7f4321109700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023630 0x7f42fc025ac0 crc :-1 s=READY pgs=160 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.572+0000 7f4322b6c700 1 -- 172.21.15.16:0/1673340452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431c09b0f0 con 0x7f431c154b00 2019-12-05T06:12:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.576+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.576+0000 7f4311ffb700 1 -- 172.21.15.16:0/1673340452 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4314007930 con 0x7f431c154b00 2019-12-05T06:12:23.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.788+0000 7f4322b6c700 1 -- 172.21.15.16:0/1673340452 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f431c09b0f0 con 0x7f431c154b00 2019-12-05T06:12:23.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.789+0000 7f432190a700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:23.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.789+0000 7f4311ffb700 1 -- 172.21.15.16:0/1673340452 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+1769 (secure 0 0 0) 0x7f43140013e0 con 0x7f431c154b00 2019-12-05T06:12:23.786 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 34 2019-12-05T06:12:23.786 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:23.786 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:23.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.791+0000 7f4322b6c700 1 -- 172.21.15.16:0/1673340452 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023630 msgr2=0x7f42fc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.791+0000 7f4322b6c700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023630 0x7f42fc025ac0 crc :-1 s=READY pgs=160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.791+0000 7f4322b6c700 1 -- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 msgr2=0x7f431c156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:23.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.791+0000 7f4322b6c700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 0x7f431c156f00 secure :-1 s=READY pgs=346 cs=0 l=1 rx=0x7f4314010740 tx=0x7f4314010ae0).stop 2019-12-05T06:12:23.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.792+0000 7f4322b6c700 1 -- 172.21.15.16:0/1673340452 shutdown_connections 2019-12-05T06:12:23.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.792+0000 7f4322b6c700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023630 0x7f42fc025ac0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.792+0000 7f4322b6c700 1 --2- 172.21.15.16:0/1673340452 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f431c154b00 0x7f431c156f00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.792+0000 7f4322b6c700 1 --2- 172.21.15.16:0/1673340452 >> v2:172.21.15.76:3300/0 conn(0x7f431c09b850 0x7f431c131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:23.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.792+0000 7f4322b6c700 1 -- 172.21.15.16:0/1673340452 shutdown_connections 2019-12-05T06:12:23.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.792+0000 7f4322b6c700 1 -- 172.21.15.16:0/1673340452 wait complete. 2019-12-05T06:12:23.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:23.792+0000 7f4322b6c700 1 -- 172.21.15.16:0/1673340452 >> 172.21.15.16:0/1673340452 conn(0x7f431c0714f0 msgr2=0x7f431c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:23.799 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=4839 2019-12-05T06:12:23.799 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.a 2019-12-05T06:12:24.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/35’ 2019-12-05T06:12:24.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./131/37’ 2019-12-05T06:12:24.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./131’ 2019-12-05T06:12:24.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/19’ 2019-12-05T06:12:24.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/15’ 2019-12-05T06:12:24.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/31’ 2019-12-05T06:12:24.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/36’ 2019-12-05T06:12:24.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/20’ 2019-12-05T06:12:24.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/12’ 2019-12-05T06:12:24.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/39’ 2019-12-05T06:12:24.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/10’ 2019-12-05T06:12:24.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/34’ 2019-12-05T06:12:24.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/38’ 2019-12-05T06:12:24.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/11’ 2019-12-05T06:12:24.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/16’ 2019-12-05T06:12:24.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/32’ 2019-12-05T06:12:24.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/30’ 2019-12-05T06:12:24.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/14’ 2019-12-05T06:12:24.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/18’ 2019-12-05T06:12:24.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/13’ 2019-12-05T06:12:24.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/33’ 2019-12-05T06:12:24.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/17’ 2019-12-05T06:12:24.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/35’ 2019-12-05T06:12:24.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./132/37’ 2019-12-05T06:12:24.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./132’ 2019-12-05T06:12:24.203 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/19’ 2019-12-05T06:12:24.203 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/15’ 2019-12-05T06:12:24.203 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/31’ 2019-12-05T06:12:24.203 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/36’ 2019-12-05T06:12:24.203 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/20’ 2019-12-05T06:12:24.203 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/12’ 2019-12-05T06:12:24.203 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/39’ 2019-12-05T06:12:24.204 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/10’ 2019-12-05T06:12:24.204 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/34’ 2019-12-05T06:12:24.204 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/38’ 2019-12-05T06:12:24.204 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/11’ 2019-12-05T06:12:24.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/16’ 2019-12-05T06:12:24.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/32’ 2019-12-05T06:12:24.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/30’ 2019-12-05T06:12:24.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/14’ 2019-12-05T06:12:24.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/18’ 2019-12-05T06:12:24.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/13’ 2019-12-05T06:12:24.206 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/33’ 2019-12-05T06:12:24.206 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/17’ 2019-12-05T06:12:24.206 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/35’ 2019-12-05T06:12:24.206 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./133/37’ 2019-12-05T06:12:24.206 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./133’ 2019-12-05T06:12:24.206 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/19’ 2019-12-05T06:12:24.207 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/15’ 2019-12-05T06:12:24.207 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/31’ 2019-12-05T06:12:24.207 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/36’ 2019-12-05T06:12:24.207 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/20’ 2019-12-05T06:12:24.208 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/12’ 2019-12-05T06:12:24.208 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/39’ 2019-12-05T06:12:24.208 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/10’ 2019-12-05T06:12:24.208 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/34’ 2019-12-05T06:12:24.208 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/38’ 2019-12-05T06:12:24.209 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/11’ 2019-12-05T06:12:24.209 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/16’ 2019-12-05T06:12:24.209 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/32’ 2019-12-05T06:12:24.209 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/30’ 2019-12-05T06:12:24.209 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/14’ 2019-12-05T06:12:24.210 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/18’ 2019-12-05T06:12:24.210 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/13’ 2019-12-05T06:12:24.210 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/33’ 2019-12-05T06:12:24.210 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/17’ 2019-12-05T06:12:24.210 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/35’ 2019-12-05T06:12:24.210 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./134/37’ 2019-12-05T06:12:24.210 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./134’ 2019-12-05T06:12:24.211 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/19’ 2019-12-05T06:12:24.211 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/15’ 2019-12-05T06:12:24.211 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/31’ 2019-12-05T06:12:24.211 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/36’ 2019-12-05T06:12:24.211 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/20’ 2019-12-05T06:12:24.211 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/12’ 2019-12-05T06:12:24.212 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/39’ 2019-12-05T06:12:24.212 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/10’ 2019-12-05T06:12:24.212 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/34’ 2019-12-05T06:12:24.212 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/38’ 2019-12-05T06:12:24.213 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/11’ 2019-12-05T06:12:24.213 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/16’ 2019-12-05T06:12:24.213 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/32’ 2019-12-05T06:12:24.213 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/30’ 2019-12-05T06:12:24.213 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/14’ 2019-12-05T06:12:24.213 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/18’ 2019-12-05T06:12:24.214 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/13’ 2019-12-05T06:12:24.214 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/33’ 2019-12-05T06:12:24.214 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/17’ 2019-12-05T06:12:24.214 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/35’ 2019-12-05T06:12:24.214 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./135/37’ 2019-12-05T06:12:24.214 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./135’ 2019-12-05T06:12:24.215 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/19’ 2019-12-05T06:12:24.215 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/15’ 2019-12-05T06:12:24.215 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/31’ 2019-12-05T06:12:24.216 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/36’ 2019-12-05T06:12:24.216 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/20’ 2019-12-05T06:12:24.216 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/12’ 2019-12-05T06:12:24.216 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/39’ 2019-12-05T06:12:24.216 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/10’ 2019-12-05T06:12:24.216 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/34’ 2019-12-05T06:12:24.217 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/38’ 2019-12-05T06:12:24.217 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/11’ 2019-12-05T06:12:24.218 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/16’ 2019-12-05T06:12:24.218 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/32’ 2019-12-05T06:12:24.218 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/30’ 2019-12-05T06:12:24.219 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/14’ 2019-12-05T06:12:24.219 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/18’ 2019-12-05T06:12:24.219 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/13’ 2019-12-05T06:12:24.219 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/33’ 2019-12-05T06:12:24.219 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/17’ 2019-12-05T06:12:24.219 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/35’ 2019-12-05T06:12:24.219 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./136/37’ 2019-12-05T06:12:24.220 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./136’ 2019-12-05T06:12:24.220 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/19’ 2019-12-05T06:12:24.220 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/15’ 2019-12-05T06:12:24.220 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/31’ 2019-12-05T06:12:24.220 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/36’ 2019-12-05T06:12:24.220 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/20’ 2019-12-05T06:12:24.220 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/12’ 2019-12-05T06:12:24.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/39’ 2019-12-05T06:12:24.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/10’ 2019-12-05T06:12:24.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/34’ 2019-12-05T06:12:24.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/38’ 2019-12-05T06:12:24.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/11’ 2019-12-05T06:12:24.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/16’ 2019-12-05T06:12:24.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/32’ 2019-12-05T06:12:24.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/30’ 2019-12-05T06:12:24.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/14’ 2019-12-05T06:12:24.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/18’ 2019-12-05T06:12:24.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/13’ 2019-12-05T06:12:24.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/33’ 2019-12-05T06:12:24.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/17’ 2019-12-05T06:12:24.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/35’ 2019-12-05T06:12:24.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./137/37’ 2019-12-05T06:12:24.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./137’ 2019-12-05T06:12:24.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/19’ 2019-12-05T06:12:24.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/15’ 2019-12-05T06:12:24.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/31’ 2019-12-05T06:12:24.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/36’ 2019-12-05T06:12:24.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/20’ 2019-12-05T06:12:24.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/12’ 2019-12-05T06:12:24.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/39’ 2019-12-05T06:12:24.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/10’ 2019-12-05T06:12:24.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/34’ 2019-12-05T06:12:24.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/38’ 2019-12-05T06:12:24.225 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/11’ 2019-12-05T06:12:24.225 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/16’ 2019-12-05T06:12:24.225 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/32’ 2019-12-05T06:12:24.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/30’ 2019-12-05T06:12:24.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/14’ 2019-12-05T06:12:24.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/18’ 2019-12-05T06:12:24.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/13’ 2019-12-05T06:12:24.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/33’ 2019-12-05T06:12:24.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/17’ 2019-12-05T06:12:24.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/35’ 2019-12-05T06:12:24.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./138/37’ 2019-12-05T06:12:24.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./138’ 2019-12-05T06:12:24.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/19’ 2019-12-05T06:12:24.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/15’ 2019-12-05T06:12:24.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/31’ 2019-12-05T06:12:24.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/36’ 2019-12-05T06:12:24.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/20’ 2019-12-05T06:12:24.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/12’ 2019-12-05T06:12:24.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/39’ 2019-12-05T06:12:24.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/10’ 2019-12-05T06:12:24.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/34’ 2019-12-05T06:12:24.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/38’ 2019-12-05T06:12:24.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/11’ 2019-12-05T06:12:24.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/16’ 2019-12-05T06:12:24.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/32’ 2019-12-05T06:12:24.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/30’ 2019-12-05T06:12:24.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/14’ 2019-12-05T06:12:24.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/18’ 2019-12-05T06:12:24.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/13’ 2019-12-05T06:12:24.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/33’ 2019-12-05T06:12:26.714 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/17’ 2019-12-05T06:12:26.714 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/35’ 2019-12-05T06:12:26.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./139/37’ 2019-12-05T06:12:26.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./139’ 2019-12-05T06:12:26.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/19’ 2019-12-05T06:12:26.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/15’ 2019-12-05T06:12:26.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/31’ 2019-12-05T06:12:26.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/36’ 2019-12-05T06:12:26.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/20’ 2019-12-05T06:12:26.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/12’ 2019-12-05T06:12:26.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/39’ 2019-12-05T06:12:26.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/10’ 2019-12-05T06:12:26.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/34’ 2019-12-05T06:12:26.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/38’ 2019-12-05T06:12:26.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/11’ 2019-12-05T06:12:26.717 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/16’ 2019-12-05T06:12:26.717 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/32’ 2019-12-05T06:12:26.717 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/30’ 2019-12-05T06:12:26.717 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/14’ 2019-12-05T06:12:26.718 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/18’ 2019-12-05T06:12:26.718 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/13’ 2019-12-05T06:12:26.718 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/33’ 2019-12-05T06:12:26.718 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/17’ 2019-12-05T06:12:26.718 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/35’ 2019-12-05T06:12:26.718 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./140/37’ 2019-12-05T06:12:26.718 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./140’ 2019-12-05T06:12:26.719 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/19’ 2019-12-05T06:12:26.719 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/15’ 2019-12-05T06:12:26.719 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/31’ 2019-12-05T06:12:26.719 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/36’ 2019-12-05T06:12:26.719 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/20’ 2019-12-05T06:12:26.719 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/12’ 2019-12-05T06:12:26.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/39’ 2019-12-05T06:12:26.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/10’ 2019-12-05T06:12:26.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/34’ 2019-12-05T06:12:26.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/38’ 2019-12-05T06:12:26.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/11’ 2019-12-05T06:12:26.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/16’ 2019-12-05T06:12:26.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/32’ 2019-12-05T06:12:26.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/30’ 2019-12-05T06:12:26.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/14’ 2019-12-05T06:12:26.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/18’ 2019-12-05T06:12:26.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/13’ 2019-12-05T06:12:26.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/33’ 2019-12-05T06:12:26.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/17’ 2019-12-05T06:12:26.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/35’ 2019-12-05T06:12:26.722 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./141/37’ 2019-12-05T06:12:26.722 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./141’ 2019-12-05T06:12:26.722 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/19’ 2019-12-05T06:12:26.722 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/15’ 2019-12-05T06:12:26.722 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/31’ 2019-12-05T06:12:26.722 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/36’ 2019-12-05T06:12:26.723 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/20’ 2019-12-05T06:12:26.723 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/12’ 2019-12-05T06:12:26.723 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/39’ 2019-12-05T06:12:26.723 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/10’ 2019-12-05T06:12:26.723 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/34’ 2019-12-05T06:12:26.723 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/38’ 2019-12-05T06:12:26.724 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/11’ 2019-12-05T06:12:26.724 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/16’ 2019-12-05T06:12:26.724 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/32’ 2019-12-05T06:12:26.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/30’ 2019-12-05T06:12:26.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/14’ 2019-12-05T06:12:26.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/18’ 2019-12-05T06:12:26.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/13’ 2019-12-05T06:12:26.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/33’ 2019-12-05T06:12:26.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/17’ 2019-12-05T06:12:26.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/35’ 2019-12-05T06:12:26.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./142/37’ 2019-12-05T06:12:26.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./142’ 2019-12-05T06:12:26.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/19’ 2019-12-05T06:12:26.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/15’ 2019-12-05T06:12:26.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/31’ 2019-12-05T06:12:26.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/36’ 2019-12-05T06:12:26.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/20’ 2019-12-05T06:12:26.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/12’ 2019-12-05T06:12:26.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/39’ 2019-12-05T06:12:26.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/10’ 2019-12-05T06:12:26.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/34’ 2019-12-05T06:12:26.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/38’ 2019-12-05T06:12:26.734 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/11’ 2019-12-05T06:12:26.734 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/16’ 2019-12-05T06:12:26.734 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/32’ 2019-12-05T06:12:26.734 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/30’ 2019-12-05T06:12:26.734 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/14’ 2019-12-05T06:12:26.734 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/18’ 2019-12-05T06:12:26.735 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/13’ 2019-12-05T06:12:26.735 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/33’ 2019-12-05T06:12:26.735 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/17’ 2019-12-05T06:12:26.735 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/35’ 2019-12-05T06:12:26.736 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./143/37’ 2019-12-05T06:12:26.736 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./143’ 2019-12-05T06:12:26.736 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/19’ 2019-12-05T06:12:26.736 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/15’ 2019-12-05T06:12:26.736 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/31’ 2019-12-05T06:12:26.736 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/36’ 2019-12-05T06:12:26.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/20’ 2019-12-05T06:12:26.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/12’ 2019-12-05T06:12:26.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/39’ 2019-12-05T06:12:26.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/10’ 2019-12-05T06:12:26.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/34’ 2019-12-05T06:12:26.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/38’ 2019-12-05T06:12:26.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/11’ 2019-12-05T06:12:26.738 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/16’ 2019-12-05T06:12:26.738 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/32’ 2019-12-05T06:12:26.738 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/30’ 2019-12-05T06:12:26.738 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/14’ 2019-12-05T06:12:26.738 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/18’ 2019-12-05T06:12:26.738 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/13’ 2019-12-05T06:12:26.739 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/33’ 2019-12-05T06:12:26.739 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/17’ 2019-12-05T06:12:26.739 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/35’ 2019-12-05T06:12:26.739 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./144/37’ 2019-12-05T06:12:26.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./144’ 2019-12-05T06:12:26.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/19’ 2019-12-05T06:12:26.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/15’ 2019-12-05T06:12:26.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/31’ 2019-12-05T06:12:26.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/36’ 2019-12-05T06:12:26.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/20’ 2019-12-05T06:12:26.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/12’ 2019-12-05T06:12:26.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/39’ 2019-12-05T06:12:26.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/10’ 2019-12-05T06:12:26.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/34’ 2019-12-05T06:12:26.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/38’ 2019-12-05T06:12:26.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/11’ 2019-12-05T06:12:26.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/16’ 2019-12-05T06:12:26.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/32’ 2019-12-05T06:12:26.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/30’ 2019-12-05T06:12:26.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/14’ 2019-12-05T06:12:26.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/18’ 2019-12-05T06:12:26.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/13’ 2019-12-05T06:12:26.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/33’ 2019-12-05T06:12:26.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/17’ 2019-12-05T06:12:26.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/35’ 2019-12-05T06:12:26.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./145/37’ 2019-12-05T06:12:26.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./145’ 2019-12-05T06:12:26.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/19’ 2019-12-05T06:12:26.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/15’ 2019-12-05T06:12:26.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/31’ 2019-12-05T06:12:26.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/36’ 2019-12-05T06:12:26.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/20’ 2019-12-05T06:12:26.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/12’ 2019-12-05T06:12:26.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/39’ 2019-12-05T06:12:26.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/10’ 2019-12-05T06:12:26.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/34’ 2019-12-05T06:12:26.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/38’ 2019-12-05T06:12:26.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/11’ 2019-12-05T06:12:26.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/16’ 2019-12-05T06:12:26.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/32’ 2019-12-05T06:12:26.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/30’ 2019-12-05T06:12:26.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/14’ 2019-12-05T06:12:26.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/18’ 2019-12-05T06:12:26.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/13’ 2019-12-05T06:12:26.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/33’ 2019-12-05T06:12:26.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/17’ 2019-12-05T06:12:26.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/35’ 2019-12-05T06:12:26.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./146/37’ 2019-12-05T06:12:26.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./146’ 2019-12-05T06:12:26.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/19’ 2019-12-05T06:12:26.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/15’ 2019-12-05T06:12:26.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/31’ 2019-12-05T06:12:26.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/36’ 2019-12-05T06:12:26.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/20’ 2019-12-05T06:12:26.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/12’ 2019-12-05T06:12:26.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/39’ 2019-12-05T06:12:26.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/10’ 2019-12-05T06:12:26.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/34’ 2019-12-05T06:12:26.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/38’ 2019-12-05T06:12:26.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/11’ 2019-12-05T06:12:26.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/16’ 2019-12-05T06:12:26.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/32’ 2019-12-05T06:12:26.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/30’ 2019-12-05T06:12:26.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/14’ 2019-12-05T06:12:26.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/18’ 2019-12-05T06:12:26.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/13’ 2019-12-05T06:12:26.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/33’ 2019-12-05T06:12:29.447 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:12:29.453 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:12:29.800 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:12:29.801 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:12:29.801 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:12:29.806 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:12:29.807 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:29.998 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:12:30.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.033+0000 7f12012fd700 1 Processor -- start 2019-12-05T06:12:30.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.033+0000 7f12012fd700 1 -- start start 2019-12-05T06:12:30.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.034+0000 7f12012fd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11fc08d2c0 0x7f11fc08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:30.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.034+0000 7f12012fd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f11fc08fef0 0x7f11fc08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:30.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.034+0000 7f12012fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11fc0958e0 con 0x7f11fc08c6d0 2019-12-05T06:12:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.034+0000 7f12012fd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc08df90 con 0x7f11fc08fef0 2019-12-05T06:12:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.034+0000 7f12012fd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc0a3f70 con 0x7f11fc08d2c0 2019-12-05T06:12:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.034+0000 7f11fb7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f11fc08c6d0 0x7f11fc08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34174/0 (socket says 172.21.15.16:34174) 2019-12-05T06:12:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.034+0000 7f11fb7fe700 1 -- 172.21.15.16:0/942430946 learned_addr learned my addr 172.21.15.16:0/942430946 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.034+0000 7f11fbfff700 1 --2- 172.21.15.16:0/942430946 >> v2:172.21.15.76:3300/0 conn(0x7f11fc08fef0 0x7f11fc08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.034+0000 7f11faffd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11fc08d2c0 0x7f11fc08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1947762290 0 0) 0x7f11e4001150 con 0x7f11fc08c6d0 2019-12-05T06:12:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11dc002e80 con 0x7f11fc08c6d0 2019-12-05T06:12:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 308516309 0 0) 0x7f11e4001130 con 0x7f11fc08c6d0 2019-12-05T06:12:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11fc08d2c0 msgr2=0x7f11fc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11fc08d2c0 0x7f11fc08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 >> v2:172.21.15.76:3300/0 conn(0x7f11fc08fef0 msgr2=0x7f11fc08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 --2- 172.21.15.16:0/942430946 >> v2:172.21.15.76:3300/0 conn(0x7f11fc08fef0 0x7f11fc08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc08e620 con 0x7f11fc08c6d0 2019-12-05T06:12:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f11e40016c0 con 0x7f11fc08c6d0 2019-12-05T06:12:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 >> v1:172.21.15.16:6789/0 conn(0x7f11fc08c6d0 legacy=0x7f11fc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11dc047a80 0x7f11dc049ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc04a690 0x7f11dc04cb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11dc04d3c0 0x7f11dc04f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc04fff0 con 0x7f11dc04a690 2019-12-05T06:12:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc0501e0 con 0x7f11dc047a80 2019-12-05T06:12:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11dc050410 con 0x7f11dc04d3c0 2019-12-05T06:12:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fbfff700 1 --2- 172.21.15.16:0/942430946 >> v2:172.21.15.76:3300/0 conn(0x7f11fc08fef0 0x7f11fc08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11faffd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc04a690 0x7f11dc04cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fb7fe700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11dc047a80 0x7f11dc049ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11fbfff700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11dc04d3c0 0x7f11dc04f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11faffd700 1 -- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11dc04d3c0 msgr2=0x7f11dc04f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11faffd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11dc04d3c0 0x7f11dc04f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11faffd700 1 -- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11dc047a80 msgr2=0x7f11dc049ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.035+0000 7f11faffd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11dc047a80 0x7f11dc049ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11faffd700 1 -- 172.21.15.16:0/942430946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc0506c0 con 0x7f11dc04a690 2019-12-05T06:12:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11fbfff700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11dc04d3c0 0x7f11dc04f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:12:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11faffd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc04a690 0x7f11dc04cb60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f11f400f630 tx=0x7f11f4002bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11faffd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc04a690 0x7f11dc04cb60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f11f400f630 tx=0x7f11f4002bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11faffd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc04a690 0x7f11dc04cb60 secure :-1 s=READY pgs=349 cs=0 l=1 rx=0x7f11f400f630 tx=0x7f11f4002bf0).ready entity=mon.0 client_cookie=c709eb6b1c592a95 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11faffd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc04a690 0x7f11dc04cb60 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f11f400f630 tx=0x7f11f4002bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11f4000960 con 0x7f11dc04a690 2019-12-05T06:12:30.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11faffd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc04a690 0x7f11dc04cb60 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f11f400f630 tx=0x7f11f4002bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11faffd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc04a690 0x7f11dc04cb60 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f11f400f630 tx=0x7f11f4002bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11f4000e80 con 0x7f11dc04a690 2019-12-05T06:12:30.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11fa7fc700 1 -- 172.21.15.16:0/942430946 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11f4001510 con 0x7f11dc04a690 2019-12-05T06:12:30.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f12012fd700 1 -- 172.21.15.16:0/942430946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc04a690 msgr2=0x7f11dc04cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:30.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f12012fd700 1 --2- 172.21.15.16:0/942430946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11dc04a690 0x7f11dc04cb60 secure :-1 s=READY pgs=349 cs=0 l=1 rx=0x7f11f400f630 tx=0x7f11f4002bf0).stop 2019-12-05T06:12:30.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.036+0000 7f11fbfff700 1 -- 172.21.15.16:0/942430946 reap_dead start 2019-12-05T06:12:30.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.037+0000 7f12012fd700 1 -- 172.21.15.16:0/942430946 shutdown_connections 2019-12-05T06:12:30.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.037+0000 7f12012fd700 1 -- 172.21.15.16:0/942430946 shutdown_connections 2019-12-05T06:12:30.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.037+0000 7f12012fd700 1 -- 172.21.15.16:0/942430946 wait complete. 2019-12-05T06:12:30.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.037+0000 7f12012fd700 1 -- 172.21.15.16:0/942430946 >> 172.21.15.16:0/942430946 conn(0x7f11fc061940 msgr2=0x7f11fc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:30.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.040+0000 7f12012fd700 1 Processor -- start 2019-12-05T06:12:30.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.041+0000 7f12012fd700 1 -- start start 2019-12-05T06:12:30.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f12012fd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f11fc1302f0 0x7f11fc12a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f12012fd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11fc12b640 0x7f11fc12abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f12012fd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11fc12b0f0 0x7f11fc1306e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f12012fd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc1296e0 con 0x7f11fc12b0f0 2019-12-05T06:12:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f12012fd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11fc128cb0 con 0x7f11fc1302f0 2019-12-05T06:12:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f12012fd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11fc128f80 con 0x7f11fc12b640 2019-12-05T06:12:30.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f11fbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11fc12b0f0 0x7f11fc1306e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:30.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f11fbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11fc12b0f0 0x7f11fc1306e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49022/0 (socket says 172.21.15.16:49022) 2019-12-05T06:12:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f11fbfff700 1 -- 172.21.15.16:0/2287645679 learned_addr learned my addr 172.21.15.16:0/2287645679 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:12:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f11faffd700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11fc12b640 0x7f11fc12abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.042+0000 7f11fb7fe700 1 --2- 172.21.15.16:0/2287645679 >> v2:172.21.15.76:3300/0 conn(0x7f11fc1302f0 0x7f11fc12a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 -- 172.21.15.16:0/2287645679 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11fc12b640 msgr2=0x7f11fc12abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11fc12b640 0x7f11fc12abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 -- 172.21.15.16:0/2287645679 >> v2:172.21.15.76:3300/0 conn(0x7f11fc1302f0 msgr2=0x7f11fc12a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> v2:172.21.15.76:3300/0 conn(0x7f11fc1302f0 0x7f11fc12a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 -- 172.21.15.16:0/2287645679 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11fc129d70 con 0x7f11fc12b0f0 2019-12-05T06:12:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> v2:172.21.15.16:3300/0 conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> v2:172.21.15.16:3300/0 conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=READY pgs=350 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11f3fff700 1 -- 172.21.15.16:0/2287645679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11ec009d10 con 0x7f11fc12b0f0 2019-12-05T06:12:30.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11f3fff700 1 -- 172.21.15.16:0/2287645679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11ec00ac30 con 0x7f11fc12b0f0 2019-12-05T06:12:30.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f12012fd700 1 -- 172.21.15.16:0/2287645679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11fc129d70 con 0x7f11fc12b0f0 2019-12-05T06:12:30.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f11f3fff700 1 -- 172.21.15.16:0/2287645679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11ec00a2f0 con 0x7f11fc12b0f0 2019-12-05T06:12:30.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.043+0000 7f12012fd700 1 -- 172.21.15.16:0/2287645679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11fc129d70 con 0x7f11fc12b0f0 2019-12-05T06:12:30.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.044+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.044+0000 7f11f3fff700 1 -- 172.21.15.16:0/2287645679 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f11ec009a70 con 0x7f11fc12b0f0 2019-12-05T06:12:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.044+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.044+0000 7f11f3fff700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11d4023630 0x7f11d4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.044+0000 7f11f3fff700 1 -- 172.21.15.16:0/2287645679 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f11ec0460b0 con 0x7f11fc12b0f0 2019-12-05T06:12:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.045+0000 7f11fb7fe700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11d4023630 0x7f11d4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.045+0000 7f11fb7fe700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11d4023630 0x7f11d4025ac0 crc :-1 s=READY pgs=161 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.048+0000 7f12012fd700 1 -- 172.21.15.16:0/2287645679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11fc129d70 con 0x7f11fc12b0f0 2019-12-05T06:12:30.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.052+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.052+0000 7f11f3fff700 1 -- 172.21.15.16:0/2287645679 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f11ec000df0 con 0x7f11fc12b0f0 2019-12-05T06:12:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.257+0000 7f12012fd700 1 -- 172.21.15.16:0/2287645679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11fc129d70 con 0x7f11fc12b0f0 2019-12-05T06:12:30.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.258+0000 7f11fbfff700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.258+0000 7f11f3fff700 1 -- 172.21.15.16:0/2287645679 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+1769 (secure 0 0 0) 0x7f11ec0099d0 con 0x7f11fc12b0f0 2019-12-05T06:12:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 34 2019-12-05T06:12:30.255 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:30.255 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.260+0000 7f12012fd700 1 -- 172.21.15.16:0/2287645679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11d4023630 msgr2=0x7f11d4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.260+0000 7f12012fd700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11d4023630 0x7f11d4025ac0 crc :-1 s=READY pgs=161 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.260+0000 7f12012fd700 1 -- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 msgr2=0x7f11fc1306e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.260+0000 7f12012fd700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 0x7f11fc1306e0 secure :-1 s=READY pgs=350 cs=0 l=1 rx=0x7f11ec0107b0 tx=0x7f11ec0106b0).stop 2019-12-05T06:12:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.261+0000 7f12012fd700 1 -- 172.21.15.16:0/2287645679 shutdown_connections 2019-12-05T06:12:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.261+0000 7f12012fd700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11d4023630 0x7f11d4025ac0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.261+0000 7f12012fd700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11fc12b0f0 0x7f11fc1306e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.261+0000 7f12012fd700 1 --2- 172.21.15.16:0/2287645679 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11fc12b640 0x7f11fc12abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.261+0000 7f12012fd700 1 --2- 172.21.15.16:0/2287645679 >> v2:172.21.15.76:3300/0 conn(0x7f11fc1302f0 0x7f11fc12a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.261+0000 7f12012fd700 1 -- 172.21.15.16:0/2287645679 shutdown_connections 2019-12-05T06:12:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.261+0000 7f12012fd700 1 -- 172.21.15.16:0/2287645679 wait complete. 2019-12-05T06:12:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:30.261+0000 7f12012fd700 1 -- 172.21.15.16:0/2287645679 >> 172.21.15.16:0/2287645679 conn(0x7f11fc0a2850 msgr2=0x7f11fc06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:30.275 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:12:31.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/17’ 2019-12-05T06:12:31.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/35’ 2019-12-05T06:12:31.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./147/37’ 2019-12-05T06:12:31.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./147’ 2019-12-05T06:12:31.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/19’ 2019-12-05T06:12:31.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/15’ 2019-12-05T06:12:31.761 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/31’ 2019-12-05T06:12:31.761 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/36’ 2019-12-05T06:12:31.761 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/20’ 2019-12-05T06:12:31.761 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/12’ 2019-12-05T06:12:31.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/39’ 2019-12-05T06:12:31.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/10’ 2019-12-05T06:12:31.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/34’ 2019-12-05T06:12:31.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/38’ 2019-12-05T06:12:31.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/11’ 2019-12-05T06:12:31.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/16’ 2019-12-05T06:12:31.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/32’ 2019-12-05T06:12:31.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/30’ 2019-12-05T06:12:31.764 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/14’ 2019-12-05T06:12:31.764 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/18’ 2019-12-05T06:12:31.764 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/13’ 2019-12-05T06:12:31.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/33’ 2019-12-05T06:12:31.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/17’ 2019-12-05T06:12:31.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/35’ 2019-12-05T06:12:31.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./148/37’ 2019-12-05T06:12:31.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./148’ 2019-12-05T06:12:31.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/19’ 2019-12-05T06:12:31.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/15’ 2019-12-05T06:12:31.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/31’ 2019-12-05T06:12:31.767 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/36’ 2019-12-05T06:12:31.767 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/20’ 2019-12-05T06:12:31.767 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/12’ 2019-12-05T06:12:31.768 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/39’ 2019-12-05T06:12:31.768 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/10’ 2019-12-05T06:12:31.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/34’ 2019-12-05T06:12:31.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/38’ 2019-12-05T06:12:31.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/11’ 2019-12-05T06:12:31.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/16’ 2019-12-05T06:12:31.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/32’ 2019-12-05T06:12:31.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/30’ 2019-12-05T06:12:31.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/14’ 2019-12-05T06:12:31.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/18’ 2019-12-05T06:12:31.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/13’ 2019-12-05T06:12:31.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/33’ 2019-12-05T06:12:31.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/17’ 2019-12-05T06:12:31.771 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/35’ 2019-12-05T06:12:31.771 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./149/37’ 2019-12-05T06:12:31.771 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./149’ 2019-12-05T06:12:31.771 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/19’ 2019-12-05T06:12:31.771 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/15’ 2019-12-05T06:12:31.771 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/31’ 2019-12-05T06:12:31.771 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/36’ 2019-12-05T06:12:31.772 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/20’ 2019-12-05T06:12:31.772 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/12’ 2019-12-05T06:12:31.772 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/39’ 2019-12-05T06:12:31.772 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/10’ 2019-12-05T06:12:31.772 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/34’ 2019-12-05T06:12:31.772 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/38’ 2019-12-05T06:12:31.773 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/11’ 2019-12-05T06:12:31.773 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/16’ 2019-12-05T06:12:31.773 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/32’ 2019-12-05T06:12:31.773 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/30’ 2019-12-05T06:12:31.773 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/14’ 2019-12-05T06:12:31.773 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/18’ 2019-12-05T06:12:31.773 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/13’ 2019-12-05T06:12:31.774 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/33’ 2019-12-05T06:12:31.774 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/17’ 2019-12-05T06:12:31.774 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/35’ 2019-12-05T06:12:31.774 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./150/37’ 2019-12-05T06:12:31.774 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./150’ 2019-12-05T06:12:31.774 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/19’ 2019-12-05T06:12:31.774 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/15’ 2019-12-05T06:12:31.775 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/31’ 2019-12-05T06:12:31.775 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/36’ 2019-12-05T06:12:31.775 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/20’ 2019-12-05T06:12:31.775 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/12’ 2019-12-05T06:12:31.775 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/39’ 2019-12-05T06:12:31.775 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/10’ 2019-12-05T06:12:31.775 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/34’ 2019-12-05T06:12:31.776 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/38’ 2019-12-05T06:12:31.776 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/11’ 2019-12-05T06:12:31.776 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/16’ 2019-12-05T06:12:31.776 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/32’ 2019-12-05T06:12:31.776 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/30’ 2019-12-05T06:12:31.776 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/14’ 2019-12-05T06:12:31.776 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/18’ 2019-12-05T06:12:31.777 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/13’ 2019-12-05T06:12:31.777 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/33’ 2019-12-05T06:12:31.777 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/17’ 2019-12-05T06:12:31.777 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/35’ 2019-12-05T06:12:31.777 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./151/37’ 2019-12-05T06:12:31.777 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./151’ 2019-12-05T06:12:31.778 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/19’ 2019-12-05T06:12:31.778 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/15’ 2019-12-05T06:12:31.778 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/31’ 2019-12-05T06:12:31.778 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/36’ 2019-12-05T06:12:31.778 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/20’ 2019-12-05T06:12:31.778 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/12’ 2019-12-05T06:12:31.779 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/39’ 2019-12-05T06:12:31.780 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/10’ 2019-12-05T06:12:31.780 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/34’ 2019-12-05T06:12:31.780 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/38’ 2019-12-05T06:12:31.780 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/11’ 2019-12-05T06:12:31.780 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/16’ 2019-12-05T06:12:31.780 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/32’ 2019-12-05T06:12:31.781 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/30’ 2019-12-05T06:12:31.781 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/14’ 2019-12-05T06:12:31.781 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/18’ 2019-12-05T06:12:31.781 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/13’ 2019-12-05T06:12:31.781 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/33’ 2019-12-05T06:12:31.781 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/17’ 2019-12-05T06:12:31.781 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/35’ 2019-12-05T06:12:31.782 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./152/37’ 2019-12-05T06:12:31.782 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./152’ 2019-12-05T06:12:31.782 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/19’ 2019-12-05T06:12:31.782 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/15’ 2019-12-05T06:12:31.782 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/31’ 2019-12-05T06:12:31.782 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/36’ 2019-12-05T06:12:31.782 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/20’ 2019-12-05T06:12:31.783 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/12’ 2019-12-05T06:12:31.783 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/39’ 2019-12-05T06:12:31.783 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/10’ 2019-12-05T06:12:31.783 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/34’ 2019-12-05T06:12:31.783 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/38’ 2019-12-05T06:12:31.783 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/11’ 2019-12-05T06:12:31.783 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/16’ 2019-12-05T06:12:31.784 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/32’ 2019-12-05T06:12:31.784 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/30’ 2019-12-05T06:12:31.784 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/14’ 2019-12-05T06:12:31.784 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/18’ 2019-12-05T06:12:31.784 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/13’ 2019-12-05T06:12:31.784 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/33’ 2019-12-05T06:12:31.785 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/17’ 2019-12-05T06:12:31.785 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/35’ 2019-12-05T06:12:31.785 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./153/37’ 2019-12-05T06:12:31.785 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./153’ 2019-12-05T06:12:31.785 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/19’ 2019-12-05T06:12:31.785 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/15’ 2019-12-05T06:12:31.785 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/31’ 2019-12-05T06:12:31.786 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/36’ 2019-12-05T06:12:31.786 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/20’ 2019-12-05T06:12:31.786 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/12’ 2019-12-05T06:12:31.786 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/39’ 2019-12-05T06:12:31.786 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/10’ 2019-12-05T06:12:31.786 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/34’ 2019-12-05T06:12:31.786 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/38’ 2019-12-05T06:12:31.787 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/11’ 2019-12-05T06:12:31.787 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/16’ 2019-12-05T06:12:31.787 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/32’ 2019-12-05T06:12:31.787 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/30’ 2019-12-05T06:12:31.787 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/14’ 2019-12-05T06:12:31.787 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/18’ 2019-12-05T06:12:31.787 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/13’ 2019-12-05T06:12:31.788 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/33’ 2019-12-05T06:12:31.788 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/17’ 2019-12-05T06:12:31.788 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/35’ 2019-12-05T06:12:31.788 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./154/37’ 2019-12-05T06:12:31.788 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./154’ 2019-12-05T06:12:31.788 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/19’ 2019-12-05T06:12:31.789 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/15’ 2019-12-05T06:12:31.789 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/31’ 2019-12-05T06:12:31.790 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/36’ 2019-12-05T06:12:31.790 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/20’ 2019-12-05T06:12:31.790 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/12’ 2019-12-05T06:12:31.790 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/39’ 2019-12-05T06:12:31.790 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/10’ 2019-12-05T06:12:31.791 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/34’ 2019-12-05T06:12:31.791 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/38’ 2019-12-05T06:12:31.791 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/11’ 2019-12-05T06:12:31.791 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/16’ 2019-12-05T06:12:31.791 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/32’ 2019-12-05T06:12:31.791 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/30’ 2019-12-05T06:12:31.792 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/14’ 2019-12-05T06:12:31.792 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/18’ 2019-12-05T06:12:31.792 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/13’ 2019-12-05T06:12:32.275 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:32.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.519+0000 7f36cb7fd700 1 Processor -- start 2019-12-05T06:12:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.519+0000 7f36cb7fd700 1 -- start start 2019-12-05T06:12:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.520+0000 7f36cb7fd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f36c408c6d0 0x7f36c408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.520+0000 7f36cb7fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c408fb80 con 0x7f36c408f720 2019-12-05T06:12:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.520+0000 7f36cb7fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c40958e0 con 0x7f36c408fef0 2019-12-05T06:12:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.520+0000 7f36cb7fd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36c408df90 con 0x7f36c408c6d0 2019-12-05T06:12:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36ca59b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f36c408f720 0x7f36c41533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34226/0 (socket says 172.21.15.16:34226) 2019-12-05T06:12:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36ca59b700 1 -- 172.21.15.16:0/486469201 learned_addr learned my addr 172.21.15.16:0/486469201 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/486469201 >> v2:172.21.15.76:3300/0 conn(0x7f36c408c6d0 0x7f36c408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36c8d98700 1 -- 172.21.15.16:0/486469201 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598604188 0 0) 0x7f36b4001150 con 0x7f36c408f720 2019-12-05T06:12:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36c8d98700 1 -- 172.21.15.16:0/486469201 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ac002e80 con 0x7f36c408f720 2019-12-05T06:12:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36c8d98700 1 -- 172.21.15.16:0/486469201 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 888581739 0 0) 0x7f36b8001080 con 0x7f36c408fef0 2019-12-05T06:12:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36c8d98700 1 -- 172.21.15.16:0/486469201 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36ac004220 con 0x7f36c408fef0 2019-12-05T06:12:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36c9d9a700 1 -- 172.21.15.16:0/486469201 >> v1:172.21.15.76:6789/0 conn(0x7f36c408fef0 legacy=0x7f36c408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36c9d9a700 1 -- 172.21.15.16:0/486469201 >> v1:172.21.15.16:6789/0 conn(0x7f36c408f720 legacy=0x7f36c41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36c9d9a700 1 -- 172.21.15.16:0/486469201 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c408e620 con 0x7f36c408c6d0 2019-12-05T06:12:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.521+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/486469201 >> v2:172.21.15.76:3300/0 conn(0x7f36c408c6d0 0x7f36c408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f36bc00b2d0 tx=0x7f36bc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.522+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/486469201 >> v2:172.21.15.76:3300/0 conn(0x7f36c408c6d0 0x7f36c408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f36bc00b2d0 tx=0x7f36bc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.522+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/486469201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36c408c6d0 0x7f36c408cac0 secure :-1 s=READY pgs=275 cs=0 l=1 rx=0x7f36bc00b2d0 tx=0x7f36bc00bae0).ready entity=mon.1 client_cookie=3a930e32aec6f320 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.522+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/486469201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36c408c6d0 0x7f36c408cac0 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7f36bc00b2d0 tx=0x7f36bc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.522+0000 7f36c8d98700 1 -- 172.21.15.16:0/486469201 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36bc000ce0 con 0x7f36c408c6d0 2019-12-05T06:12:32.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.522+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/486469201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36c408c6d0 0x7f36c408cac0 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7f36bc00b2d0 tx=0x7f36bc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.522+0000 7f36c8d98700 1 -- 172.21.15.16:0/486469201 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36bc009be0 con 0x7f36c408c6d0 2019-12-05T06:12:32.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.522+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/486469201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36c408c6d0 0x7f36c408cac0 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7f36bc00b2d0 tx=0x7f36bc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.522+0000 7f36c8d98700 1 -- 172.21.15.16:0/486469201 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36bc000ce0 con 0x7f36c408c6d0 2019-12-05T06:12:32.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.522+0000 7f36cb7fd700 1 -- 172.21.15.16:0/486469201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36c408c6d0 msgr2=0x7f36c408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:32.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.522+0000 7f36cb7fd700 1 --2- 172.21.15.16:0/486469201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36c408c6d0 0x7f36c408cac0 secure :-1 s=READY pgs=275 cs=0 l=1 rx=0x7f36bc00b2d0 tx=0x7f36bc00bae0).stop 2019-12-05T06:12:32.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.523+0000 7f36cb7fd700 1 -- 172.21.15.16:0/486469201 shutdown_connections 2019-12-05T06:12:32.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.523+0000 7f36cb7fd700 1 --2- 172.21.15.16:0/486469201 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f36c408c6d0 0x7f36c408cac0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:32.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.523+0000 7f36cb7fd700 1 -- 172.21.15.16:0/486469201 shutdown_connections 2019-12-05T06:12:32.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.523+0000 7f36cb7fd700 1 -- 172.21.15.16:0/486469201 wait complete. 2019-12-05T06:12:32.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.523+0000 7f36cb7fd700 1 -- 172.21.15.16:0/486469201 >> 172.21.15.16:0/486469201 conn(0x7f36c4061940 msgr2=0x7f36c4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:32.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.527+0000 7f36cb7fd700 1 Processor -- start 2019-12-05T06:12:32.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.527+0000 7f36cb7fd700 1 -- start start 2019-12-05T06:12:32.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.528+0000 7f36cb7fd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:32.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.528+0000 7f36cb7fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c4131bf0 con 0x7f36c412b640 2019-12-05T06:12:32.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.528+0000 7f36cb7fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36c41316f0 con 0x7f36c4153ba0 2019-12-05T06:12:32.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.528+0000 7f36cb7fd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36c4128cb0 con 0x7f36c41302f0 2019-12-05T06:12:32.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.528+0000 7f36c9599700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f36c412b640 0x7f36c412abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34228/0 (socket says 172.21.15.16:34228) 2019-12-05T06:12:32.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.528+0000 7f36c9599700 1 -- 172.21.15.16:0/3873012878 learned_addr learned my addr 172.21.15.16:0/3873012878 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:32.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:32.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959024393 0 0) 0x7f36b8001240 con 0x7f36c412b640 2019-12-05T06:12:32.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a4002e80 con 0x7f36c412b640 2019-12-05T06:12:32.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580143202 0 0) 0x7f36b40009b0 con 0x7f36c4153ba0 2019-12-05T06:12:32.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a40030b0 con 0x7f36c4153ba0 2019-12-05T06:12:32.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c9d9a700 1 -- 172.21.15.16:0/3873012878 >> v1:172.21.15.76:6789/0 conn(0x7f36c4153ba0 legacy=0x7f36c4155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:32.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c9d9a700 1 -- 172.21.15.16:0/3873012878 >> v1:172.21.15.16:6789/0 conn(0x7f36c412b640 legacy=0x7f36c412abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:32.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c9d9a700 1 -- 172.21.15.16:0/3873012878 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36c4129d70 con 0x7f36c41302f0 2019-12-05T06:12:32.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.529+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=READY pgs=97 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:32.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.530+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.530+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36bc000cc0 con 0x7f36c41302f0 2019-12-05T06:12:32.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.530+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.530+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36bc000cc0 con 0x7f36c41302f0 2019-12-05T06:12:32.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.530+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.530+0000 7f36cb7fd700 1 -- 172.21.15.16:0/3873012878 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36c4129d70 con 0x7f36c41302f0 2019-12-05T06:12:32.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.530+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36bc009ce0 con 0x7f36c41302f0 2019-12-05T06:12:32.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.530+0000 7f36cb7fd700 1 -- 172.21.15.16:0/3873012878 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36c4129d70 con 0x7f36c41302f0 2019-12-05T06:12:32.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.531+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.531+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.531+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f36bc009900 con 0x7f36c41302f0 2019-12-05T06:12:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.531+0000 7f36c27fc700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36a4023650 0x7f36a4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.531+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f36bc0463c0 con 0x7f36c41302f0 2019-12-05T06:12:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.532+0000 7f36c9599700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36a4023650 0x7f36a4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.532+0000 7f36c9599700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36a4023650 0x7f36a4025ae0 crc :-1 s=READY pgs=162 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.533+0000 7f36cb7fd700 1 -- 172.21.15.16:0/3873012878 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36c4129d70 con 0x7f36c41302f0 2019-12-05T06:12:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.542+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.542+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f36bc00b930 con 0x7f36c41302f0 2019-12-05T06:12:32.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.763+0000 7f36cb7fd700 1 -- 172.21.15.16:0/3873012878 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36c4129d70 con 0x7f36c41302f0 2019-12-05T06:12:32.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.764+0000 7f36c9d9a700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=THROTTLE_DONE pgs=97 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:32.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.764+0000 7f36c27fc700 1 -- 172.21.15.16:0/3873012878 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+1769 (secure 0 0 0) 0x7f36bc009900 con 0x7f36c41302f0 2019-12-05T06:12:32.761 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 34 2019-12-05T06:12:32.761 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:32.761 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:32.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.767+0000 7f36cb7fd700 1 -- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36a4023650 msgr2=0x7f36a4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.767+0000 7f36cb7fd700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36a4023650 0x7f36a4025ae0 crc :-1 s=READY pgs=162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:32.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.767+0000 7f36cb7fd700 1 -- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 msgr2=0x7f36c412a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.767+0000 7f36cb7fd700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 secure :-1 s=READY pgs=97 cs=0 l=1 rx=0x7f36bc0106d0 tx=0x7f36bc010a70).stop 2019-12-05T06:12:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.767+0000 7f36cb7fd700 1 -- 172.21.15.16:0/3873012878 shutdown_connections 2019-12-05T06:12:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.767+0000 7f36cb7fd700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36a4023650 0x7f36a4025ae0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.767+0000 7f36cb7fd700 1 --2- 172.21.15.16:0/3873012878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f36c41302f0 0x7f36c412a670 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.767+0000 7f36cb7fd700 1 -- 172.21.15.16:0/3873012878 shutdown_connections 2019-12-05T06:12:32.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.767+0000 7f36cb7fd700 1 -- 172.21.15.16:0/3873012878 wait complete. 2019-12-05T06:12:32.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:32.767+0000 7f36cb7fd700 1 -- 172.21.15.16:0/3873012878 >> 172.21.15.16:0/3873012878 conn(0x7f36c40a2850 msgr2=0x7f36c406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:32.774 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:12:33.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/33’ 2019-12-05T06:12:33.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/17’ 2019-12-05T06:12:33.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/35’ 2019-12-05T06:12:33.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./155/37’ 2019-12-05T06:12:33.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./155’ 2019-12-05T06:12:33.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/19’ 2019-12-05T06:12:33.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/15’ 2019-12-05T06:12:33.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/31’ 2019-12-05T06:12:33.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/36’ 2019-12-05T06:12:33.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/20’ 2019-12-05T06:12:33.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/12’ 2019-12-05T06:12:33.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/39’ 2019-12-05T06:12:33.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/10’ 2019-12-05T06:12:33.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/34’ 2019-12-05T06:12:33.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/38’ 2019-12-05T06:12:33.404 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/11’ 2019-12-05T06:12:33.404 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/16’ 2019-12-05T06:12:33.404 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/32’ 2019-12-05T06:12:33.404 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/30’ 2019-12-05T06:12:33.404 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/14’ 2019-12-05T06:12:33.405 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/18’ 2019-12-05T06:12:33.405 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/13’ 2019-12-05T06:12:33.405 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/33’ 2019-12-05T06:12:33.405 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/17’ 2019-12-05T06:12:33.405 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/35’ 2019-12-05T06:12:33.405 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./156/37’ 2019-12-05T06:12:33.405 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./156’ 2019-12-05T06:12:33.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/19’ 2019-12-05T06:12:33.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/15’ 2019-12-05T06:12:33.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/31’ 2019-12-05T06:12:33.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/36’ 2019-12-05T06:12:33.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/20’ 2019-12-05T06:12:33.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/12’ 2019-12-05T06:12:33.406 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/39’ 2019-12-05T06:12:33.407 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/10’ 2019-12-05T06:12:33.409 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/34’ 2019-12-05T06:12:33.409 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/38’ 2019-12-05T06:12:33.409 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/11’ 2019-12-05T06:12:33.410 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/16’ 2019-12-05T06:12:33.410 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/32’ 2019-12-05T06:12:33.410 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/30’ 2019-12-05T06:12:33.410 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/14’ 2019-12-05T06:12:33.410 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/18’ 2019-12-05T06:12:33.410 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/13’ 2019-12-05T06:12:33.410 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/33’ 2019-12-05T06:12:33.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/17’ 2019-12-05T06:12:33.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/35’ 2019-12-05T06:12:33.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./157/37’ 2019-12-05T06:12:33.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./157’ 2019-12-05T06:12:33.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/19’ 2019-12-05T06:12:33.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/15’ 2019-12-05T06:12:33.411 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/31’ 2019-12-05T06:12:33.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/36’ 2019-12-05T06:12:33.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/20’ 2019-12-05T06:12:33.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/12’ 2019-12-05T06:12:33.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/39’ 2019-12-05T06:12:33.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/10’ 2019-12-05T06:12:33.412 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/34’ 2019-12-05T06:12:33.413 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/38’ 2019-12-05T06:12:33.413 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/11’ 2019-12-05T06:12:33.414 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/16’ 2019-12-05T06:12:33.414 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/32’ 2019-12-05T06:12:33.414 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/30’ 2019-12-05T06:12:33.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/14’ 2019-12-05T06:12:33.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/18’ 2019-12-05T06:12:33.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/13’ 2019-12-05T06:12:33.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/33’ 2019-12-05T06:12:33.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/17’ 2019-12-05T06:12:33.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/35’ 2019-12-05T06:12:33.415 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./158/37’ 2019-12-05T06:12:33.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./158’ 2019-12-05T06:12:33.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/19’ 2019-12-05T06:12:33.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/15’ 2019-12-05T06:12:33.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/31’ 2019-12-05T06:12:33.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/36’ 2019-12-05T06:12:33.416 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/20’ 2019-12-05T06:12:33.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/12’ 2019-12-05T06:12:33.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/39’ 2019-12-05T06:12:33.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/10’ 2019-12-05T06:12:33.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/34’ 2019-12-05T06:12:33.417 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/38’ 2019-12-05T06:12:33.419 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/11’ 2019-12-05T06:12:33.419 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/16’ 2019-12-05T06:12:33.419 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/32’ 2019-12-05T06:12:33.419 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/30’ 2019-12-05T06:12:33.419 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/14’ 2019-12-05T06:12:33.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/18’ 2019-12-05T06:12:33.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/13’ 2019-12-05T06:12:33.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/33’ 2019-12-05T06:12:33.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/17’ 2019-12-05T06:12:33.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/35’ 2019-12-05T06:12:33.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./159/37’ 2019-12-05T06:12:33.420 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./159’ 2019-12-05T06:12:33.421 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/19’ 2019-12-05T06:12:33.421 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/15’ 2019-12-05T06:12:33.421 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/31’ 2019-12-05T06:12:33.421 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/36’ 2019-12-05T06:12:33.421 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/20’ 2019-12-05T06:12:33.421 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/12’ 2019-12-05T06:12:33.421 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/39’ 2019-12-05T06:12:33.422 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/10’ 2019-12-05T06:12:33.422 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/34’ 2019-12-05T06:12:33.422 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/38’ 2019-12-05T06:12:33.422 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/11’ 2019-12-05T06:12:33.422 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/16’ 2019-12-05T06:12:33.422 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/32’ 2019-12-05T06:12:33.423 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/30’ 2019-12-05T06:12:33.423 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/14’ 2019-12-05T06:12:33.423 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/18’ 2019-12-05T06:12:33.424 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/13’ 2019-12-05T06:12:33.424 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/33’ 2019-12-05T06:12:33.424 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/17’ 2019-12-05T06:12:33.424 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/35’ 2019-12-05T06:12:33.424 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./160/37’ 2019-12-05T06:12:33.424 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./160’ 2019-12-05T06:12:33.425 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/19’ 2019-12-05T06:12:33.426 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/15’ 2019-12-05T06:12:33.426 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/31’ 2019-12-05T06:12:33.426 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/36’ 2019-12-05T06:12:33.427 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/20’ 2019-12-05T06:12:33.427 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/12’ 2019-12-05T06:12:33.427 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/39’ 2019-12-05T06:12:33.427 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/10’ 2019-12-05T06:12:33.427 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/34’ 2019-12-05T06:12:33.427 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/38’ 2019-12-05T06:12:33.430 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/11’ 2019-12-05T06:12:33.430 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/16’ 2019-12-05T06:12:33.430 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/32’ 2019-12-05T06:12:33.430 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/30’ 2019-12-05T06:12:33.430 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/14’ 2019-12-05T06:12:33.431 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/18’ 2019-12-05T06:12:33.431 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/13’ 2019-12-05T06:12:33.431 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/33’ 2019-12-05T06:12:33.431 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/17’ 2019-12-05T06:12:33.431 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/35’ 2019-12-05T06:12:33.431 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./161/37’ 2019-12-05T06:12:33.432 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./161’ 2019-12-05T06:12:33.432 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/19’ 2019-12-05T06:12:33.432 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/15’ 2019-12-05T06:12:33.432 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/31’ 2019-12-05T06:12:33.433 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/36’ 2019-12-05T06:12:33.433 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/20’ 2019-12-05T06:12:33.433 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/12’ 2019-12-05T06:12:33.433 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/39’ 2019-12-05T06:12:33.433 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/10’ 2019-12-05T06:12:33.434 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/34’ 2019-12-05T06:12:33.434 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/38’ 2019-12-05T06:12:33.434 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/11’ 2019-12-05T06:12:33.434 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/16’ 2019-12-05T06:12:33.434 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/32’ 2019-12-05T06:12:33.435 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/30’ 2019-12-05T06:12:33.436 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/14’ 2019-12-05T06:12:33.436 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/18’ 2019-12-05T06:12:33.436 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/13’ 2019-12-05T06:12:33.437 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/33’ 2019-12-05T06:12:33.437 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/17’ 2019-12-05T06:12:33.437 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/35’ 2019-12-05T06:12:33.438 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./162/37’ 2019-12-05T06:12:33.439 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./162’ 2019-12-05T06:12:33.439 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/19’ 2019-12-05T06:12:33.439 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/15’ 2019-12-05T06:12:33.439 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/31’ 2019-12-05T06:12:33.440 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/36’ 2019-12-05T06:12:33.440 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/20’ 2019-12-05T06:12:33.440 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/12’ 2019-12-05T06:12:33.440 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/39’ 2019-12-05T06:12:33.441 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/10’ 2019-12-05T06:12:33.441 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/34’ 2019-12-05T06:12:33.441 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/38’ 2019-12-05T06:12:33.441 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/11’ 2019-12-05T06:12:33.442 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/16’ 2019-12-05T06:12:33.442 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/32’ 2019-12-05T06:12:33.442 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/30’ 2019-12-05T06:12:33.442 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/14’ 2019-12-05T06:12:33.443 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/18’ 2019-12-05T06:12:34.775 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:12:34.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.932+0000 7f7833ce6700 1 Processor -- start 2019-12-05T06:12:34.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.932+0000 7f7833ce6700 1 -- start start 2019-12-05T06:12:34.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7833ce6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f782c08c6d0 0x7f782c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:34.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7833ce6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c08fef0 0x7f782c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7833ce6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c0958e0 con 0x7f782c08ef20 2019-12-05T06:12:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7833ce6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782c08df90 con 0x7f782c08c6d0 2019-12-05T06:12:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7833ce6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c0a3f70 con 0x7f782c08fef0 2019-12-05T06:12:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7832a84700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f782c08ef20 0x7f782c08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34260/0 (socket says 172.21.15.16:34260) 2019-12-05T06:12:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7832a84700 1 -- 172.21.15.16:0/2895430346 learned_addr learned my addr 172.21.15.16:0/2895430346 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> v2:172.21.15.76:3300/0 conn(0x7f782c08c6d0 0x7f782c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:34.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7831a82700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c08fef0 0x7f782c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475522295 0 0) 0x7f781c001170 con 0x7f782c08ef20 2019-12-05T06:12:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7818002e80 con 0x7f782c08ef20 2019-12-05T06:12:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 692781455 0 0) 0x7f781c001170 con 0x7f782c08ef20 2019-12-05T06:12:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c08fef0 msgr2=0x7f782c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7831281700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c08fef0 0x7f782c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 >> v2:172.21.15.76:3300/0 conn(0x7f782c08c6d0 msgr2=0x7f782c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:34.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7831281700 1 --2- 172.21.15.16:0/2895430346 >> v2:172.21.15.76:3300/0 conn(0x7f782c08c6d0 0x7f782c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:34.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.934+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c08d540 con 0x7f782c08ef20 2019-12-05T06:12:34.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f781c001700 con 0x7f782c08ef20 2019-12-05T06:12:34.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7831a82700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c08fef0 0x7f782c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:34.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> v2:172.21.15.76:3300/0 conn(0x7f782c08c6d0 0x7f782c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:12:34.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 >> v1:172.21.15.16:6789/0 conn(0x7f782c08ef20 legacy=0x7f782c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:34.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7831281700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180479e0 0x7f7818049e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:34.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7831281700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f781804a5c0 0x7f781804ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:34.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7831281700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f781804d2f0 0x7f781804f790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:12:34.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180479e0 0x7f7818049e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:34.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f781804ff40 con 0x7f78180479e0 2019-12-05T06:12:34.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7818050130 con 0x7f781804a5c0 2019-12-05T06:12:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7818050360 con 0x7f781804d2f0 2019-12-05T06:12:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7831a82700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f781804a5c0 0x7f781804ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832a84700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f781804d2f0 0x7f781804f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832283700 1 -- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f781804d2f0 msgr2=0x7f781804f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f781804d2f0 0x7f781804f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:34.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832283700 1 -- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f781804a5c0 msgr2=0x7f781804ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:12:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f781804a5c0 0x7f781804ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:12:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832283700 1 -- 172.21.15.16:0/2895430346 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7818050610 con 0x7f78180479e0 2019-12-05T06:12:34.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180479e0 0x7f7818049e40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f782400d8b0 tx=0x7f782400dbb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180479e0 0x7f7818049e40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f782400d8b0 tx=0x7f782400dbb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.935+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180479e0 0x7f7818049e40 secure :-1 s=READY pgs=354 cs=0 l=1 rx=0x7f782400d8b0 tx=0x7f782400dbb0).ready entity=mon.0 client_cookie=40038f9c53859f5a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.936+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180479e0 0x7f7818049e40 secure :-1 s=THROTTLE_DONE pgs=354 cs=0 l=1 rx=0x7f782400d8b0 tx=0x7f782400dbb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.936+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7824000ce0 con 0x7f78180479e0 2019-12-05T06:12:34.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.936+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180479e0 0x7f7818049e40 secure :-1 s=THROTTLE_DONE pgs=354 cs=0 l=1 rx=0x7f782400d8b0 tx=0x7f782400dbb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.936+0000 7f7832283700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180479e0 0x7f7818049e40 secure :-1 s=THROTTLE_DONE pgs=354 cs=0 l=1 rx=0x7f782400d8b0 tx=0x7f782400dbb0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.936+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7824009af0 con 0x7f78180479e0 2019-12-05T06:12:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.936+0000 7f7831281700 1 -- 172.21.15.16:0/2895430346 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7824000ce0 con 0x7f78180479e0 2019-12-05T06:12:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.936+0000 7f7833ce6700 1 -- 172.21.15.16:0/2895430346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180479e0 msgr2=0x7f7818049e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.936+0000 7f7833ce6700 1 --2- 172.21.15.16:0/2895430346 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f78180479e0 0x7f7818049e40 secure :-1 s=READY pgs=354 cs=0 l=1 rx=0x7f782400d8b0 tx=0x7f782400dbb0).stop 2019-12-05T06:12:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.936+0000 7f7832a84700 1 -- 172.21.15.16:0/2895430346 reap_dead start 2019-12-05T06:12:34.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.937+0000 7f7833ce6700 1 -- 172.21.15.16:0/2895430346 shutdown_connections 2019-12-05T06:12:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.937+0000 7f7833ce6700 1 -- 172.21.15.16:0/2895430346 shutdown_connections 2019-12-05T06:12:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.937+0000 7f7833ce6700 1 -- 172.21.15.16:0/2895430346 wait complete. 2019-12-05T06:12:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.937+0000 7f7833ce6700 1 -- 172.21.15.16:0/2895430346 >> 172.21.15.16:0/2895430346 conn(0x7f782c061940 msgr2=0x7f782c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.940+0000 7f7833ce6700 1 Processor -- start 2019-12-05T06:12:34.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.940+0000 7f7833ce6700 1 -- start start 2019-12-05T06:12:34.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.940+0000 7f7833ce6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f782c06dc50 0x7f782c06e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.940+0000 7f7833ce6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.940+0000 7f7833ce6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f782c1350d0 con 0x7f782c135450 2019-12-05T06:12:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.940+0000 7f7833ce6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f782c132830 con 0x7f782c06dc50 2019-12-05T06:12:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.940+0000 7f7833ce6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f782c1334f0 con 0x7f782c134c80 2019-12-05T06:12:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7832283700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f782c06dc50 0x7f782c06e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7831a82700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f782c135450 0x7f782c134480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:55572/0 (socket says 172.21.15.16:55572) 2019-12-05T06:12:34.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7831a82700 1 -- 172.21.15.16:0/2086056803 learned_addr learned my addr 172.21.15.16:0/2086056803 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:12:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7832a84700 1 -- 172.21.15.16:0/2086056803 >> v2:172.21.15.76:3300/0 conn(0x7f782c06dc50 msgr2=0x7f782c06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> v2:172.21.15.76:3300/0 conn(0x7f782c06dc50 0x7f782c06e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7832a84700 1 -- 172.21.15.16:0/2086056803 >> v1:172.21.15.76:6789/0 conn(0x7f782c135450 legacy=0x7f782c134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7832a84700 1 -- 172.21.15.16:0/2086056803 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f782c133b80 con 0x7f782c134c80 2019-12-05T06:12:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.941+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.942+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.942+0000 7f782affd700 1 -- 172.21.15.16:0/2086056803 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f781c004240 con 0x7f782c134c80 2019-12-05T06:12:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.942+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.942+0000 7f782affd700 1 -- 172.21.15.16:0/2086056803 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f781c000ef0 con 0x7f782c134c80 2019-12-05T06:12:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.942+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.942+0000 7f7833ce6700 1 -- 172.21.15.16:0/2086056803 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f782c133b80 con 0x7f782c134c80 2019-12-05T06:12:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.942+0000 7f782affd700 1 -- 172.21.15.16:0/2086056803 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f781c0013b0 con 0x7f782c134c80 2019-12-05T06:12:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.942+0000 7f7833ce6700 1 -- 172.21.15.16:0/2086056803 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f782c133b80 con 0x7f782c134c80 2019-12-05T06:12:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.942+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.943+0000 7f782affd700 1 -- 172.21.15.16:0/2086056803 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f781c001310 con 0x7f782c134c80 2019-12-05T06:12:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.943+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.943+0000 7f782affd700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7810023630 0x7f7810025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:12:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.943+0000 7f782affd700 1 -- 172.21.15.16:0/2086056803 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f781c045d00 con 0x7f782c134c80 2019-12-05T06:12:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.943+0000 7f7832283700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7810023630 0x7f7810025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:12:34.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.944+0000 7f7832283700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7810023630 0x7f7810025ac0 crc :-1 s=READY pgs=163 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:12:34.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.948+0000 7f7833ce6700 1 -- 172.21.15.16:0/2086056803 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f782c133b80 con 0x7f782c134c80 2019-12-05T06:12:34.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.956+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:34.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:34.956+0000 7f782affd700 1 -- 172.21.15.16:0/2086056803 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f781c000a60 con 0x7f782c134c80 2019-12-05T06:12:35.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.162+0000 7f7833ce6700 1 -- 172.21.15.16:0/2086056803 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f782c133b80 con 0x7f782c134c80 2019-12-05T06:12:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.162+0000 7f7832a84700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=THROTTLE_DONE pgs=98 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:12:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.163+0000 7f782affd700 1 -- 172.21.15.16:0/2086056803 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f781c001270 con 0x7f782c134c80 2019-12-05T06:12:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2019-12-05T06:12:35.159 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:12:35.160 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5049,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 -- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7810023630 msgr2=0x7f7810025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7810023630 0x7f7810025ac0 crc :-1 s=READY pgs=163 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 -- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 msgr2=0x7f782c06e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:12:35.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 secure :-1 s=READY pgs=98 cs=0 l=1 rx=0x7f781c010390 tx=0x7f781c0101c0).stop 2019-12-05T06:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 -- 172.21.15.16:0/2086056803 shutdown_connections 2019-12-05T06:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7810023630 0x7f7810025ac0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 --2- 172.21.15.16:0/2086056803 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f782c134c80 0x7f782c06e580 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 --2- 172.21.15.16:0/2086056803 >> v2:172.21.15.76:3300/0 conn(0x7f782c06dc50 0x7f782c06e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 -- 172.21.15.16:0/2086056803 shutdown_connections 2019-12-05T06:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 -- 172.21.15.16:0/2086056803 wait complete. 2019-12-05T06:12:35.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:12:35.165+0000 7f7833ce6700 1 -- 172.21.15.16:0/2086056803 >> 172.21.15.16:0/2086056803 conn(0x7f782c0a2850 msgr2=0x7f782c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:12:35.173 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:12:35.173 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 77 secs before thrashing 2019-12-05T06:12:35.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/13’ 2019-12-05T06:12:35.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/33’ 2019-12-05T06:12:35.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/17’ 2019-12-05T06:12:35.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/35’ 2019-12-05T06:12:35.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./163/37’ 2019-12-05T06:12:35.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./163’ 2019-12-05T06:12:35.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/19’ 2019-12-05T06:12:35.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/15’ 2019-12-05T06:12:35.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/31’ 2019-12-05T06:12:35.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/36’ 2019-12-05T06:12:35.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/20’ 2019-12-05T06:12:35.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/12’ 2019-12-05T06:12:35.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/39’ 2019-12-05T06:12:35.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/10’ 2019-12-05T06:12:35.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/34’ 2019-12-05T06:12:35.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/38’ 2019-12-05T06:12:35.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/11’ 2019-12-05T06:12:35.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/16’ 2019-12-05T06:12:35.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/32’ 2019-12-05T06:12:35.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/30’ 2019-12-05T06:12:35.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/14’ 2019-12-05T06:12:35.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/18’ 2019-12-05T06:12:35.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/13’ 2019-12-05T06:12:35.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/33’ 2019-12-05T06:12:35.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/17’ 2019-12-05T06:12:35.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/35’ 2019-12-05T06:12:35.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./164/37’ 2019-12-05T06:12:35.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./164’ 2019-12-05T06:12:35.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/19’ 2019-12-05T06:12:35.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/15’ 2019-12-05T06:12:35.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/31’ 2019-12-05T06:12:35.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/36’ 2019-12-05T06:12:35.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/20’ 2019-12-05T06:12:35.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/12’ 2019-12-05T06:12:35.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/39’ 2019-12-05T06:12:35.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/10’ 2019-12-05T06:12:35.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/34’ 2019-12-05T06:12:35.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/38’ 2019-12-05T06:12:35.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/11’ 2019-12-05T06:12:35.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/16’ 2019-12-05T06:12:35.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/32’ 2019-12-05T06:12:35.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/30’ 2019-12-05T06:12:35.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/14’ 2019-12-05T06:12:35.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/18’ 2019-12-05T06:12:35.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/13’ 2019-12-05T06:12:35.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/33’ 2019-12-05T06:12:35.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/17’ 2019-12-05T06:12:35.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/35’ 2019-12-05T06:12:35.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./165/37’ 2019-12-05T06:12:35.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./165’ 2019-12-05T06:12:35.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/19’ 2019-12-05T06:12:35.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/15’ 2019-12-05T06:12:35.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/31’ 2019-12-05T06:12:35.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/36’ 2019-12-05T06:12:35.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/20’ 2019-12-05T06:12:35.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/12’ 2019-12-05T06:12:35.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/39’ 2019-12-05T06:12:35.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/10’ 2019-12-05T06:12:35.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/34’ 2019-12-05T06:12:35.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/38’ 2019-12-05T06:12:35.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/11’ 2019-12-05T06:12:35.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/16’ 2019-12-05T06:12:35.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/32’ 2019-12-05T06:12:35.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/30’ 2019-12-05T06:12:35.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/14’ 2019-12-05T06:12:35.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/18’ 2019-12-05T06:12:35.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/13’ 2019-12-05T06:12:35.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/33’ 2019-12-05T06:12:35.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/17’ 2019-12-05T06:12:35.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/35’ 2019-12-05T06:12:35.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./166/37’ 2019-12-05T06:12:35.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./166’ 2019-12-05T06:12:35.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/19’ 2019-12-05T06:12:35.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/15’ 2019-12-05T06:12:35.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/31’ 2019-12-05T06:12:35.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/36’ 2019-12-05T06:12:35.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/20’ 2019-12-05T06:12:35.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/12’ 2019-12-05T06:12:35.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/39’ 2019-12-05T06:12:35.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/10’ 2019-12-05T06:12:35.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/34’ 2019-12-05T06:12:35.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/38’ 2019-12-05T06:12:35.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/11’ 2019-12-05T06:12:35.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/16’ 2019-12-05T06:12:35.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/32’ 2019-12-05T06:12:35.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/30’ 2019-12-05T06:12:35.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/14’ 2019-12-05T06:12:35.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/18’ 2019-12-05T06:12:35.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/13’ 2019-12-05T06:12:35.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/33’ 2019-12-05T06:12:35.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/17’ 2019-12-05T06:12:35.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/35’ 2019-12-05T06:12:35.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./167/37’ 2019-12-05T06:12:35.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./167’ 2019-12-05T06:12:35.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/19’ 2019-12-05T06:12:35.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/15’ 2019-12-05T06:12:35.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/31’ 2019-12-05T06:12:35.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/36’ 2019-12-05T06:12:35.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/20’ 2019-12-05T06:12:35.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/12’ 2019-12-05T06:12:35.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/39’ 2019-12-05T06:12:35.518 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/10’ 2019-12-05T06:12:35.518 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/34’ 2019-12-05T06:12:35.518 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/38’ 2019-12-05T06:12:35.518 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/11’ 2019-12-05T06:12:35.518 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/16’ 2019-12-05T06:12:35.518 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/32’ 2019-12-05T06:12:35.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/30’ 2019-12-05T06:12:35.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/14’ 2019-12-05T06:12:35.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/18’ 2019-12-05T06:12:35.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/13’ 2019-12-05T06:12:35.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/33’ 2019-12-05T06:12:35.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/17’ 2019-12-05T06:12:35.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/35’ 2019-12-05T06:12:35.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./168/37’ 2019-12-05T06:12:35.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./168’ 2019-12-05T06:12:35.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/19’ 2019-12-05T06:12:35.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/15’ 2019-12-05T06:12:35.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/31’ 2019-12-05T06:12:35.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/36’ 2019-12-05T06:12:35.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/20’ 2019-12-05T06:12:35.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/12’ 2019-12-05T06:12:35.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/39’ 2019-12-05T06:12:35.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/10’ 2019-12-05T06:12:35.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/34’ 2019-12-05T06:12:35.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/38’ 2019-12-05T06:12:35.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/11’ 2019-12-05T06:12:35.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/16’ 2019-12-05T06:12:35.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/32’ 2019-12-05T06:12:35.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/30’ 2019-12-05T06:12:35.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/14’ 2019-12-05T06:12:35.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/18’ 2019-12-05T06:12:35.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/13’ 2019-12-05T06:12:35.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/33’ 2019-12-05T06:12:35.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/17’ 2019-12-05T06:12:35.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/35’ 2019-12-05T06:12:35.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./169/37’ 2019-12-05T06:12:35.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./169’ 2019-12-05T06:12:35.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/19’ 2019-12-05T06:12:35.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/15’ 2019-12-05T06:12:35.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/31’ 2019-12-05T06:12:35.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/36’ 2019-12-05T06:12:35.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/20’ 2019-12-05T06:12:35.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/12’ 2019-12-05T06:12:35.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/39’ 2019-12-05T06:12:35.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/10’ 2019-12-05T06:12:35.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/34’ 2019-12-05T06:12:35.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/38’ 2019-12-05T06:12:35.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/11’ 2019-12-05T06:12:35.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/16’ 2019-12-05T06:12:35.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/32’ 2019-12-05T06:12:35.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/30’ 2019-12-05T06:12:35.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/14’ 2019-12-05T06:12:35.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/18’ 2019-12-05T06:12:35.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/13’ 2019-12-05T06:12:35.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/33’ 2019-12-05T06:12:35.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/17’ 2019-12-05T06:12:35.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/35’ 2019-12-05T06:12:35.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./170/37’ 2019-12-05T06:12:35.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./170’ 2019-12-05T06:12:35.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/19’ 2019-12-05T06:12:35.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/15’ 2019-12-05T06:12:35.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/31’ 2019-12-05T06:12:35.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/36’ 2019-12-05T06:12:35.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/20’ 2019-12-05T06:12:35.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/12’ 2019-12-05T06:12:35.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/39’ 2019-12-05T06:12:35.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/10’ 2019-12-05T06:12:35.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/34’ 2019-12-05T06:12:35.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/38’ 2019-12-05T06:12:35.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/11’ 2019-12-05T06:12:35.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/16’ 2019-12-05T06:12:35.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/32’ 2019-12-05T06:12:35.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/30’ 2019-12-05T06:12:35.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/14’ 2019-12-05T06:12:35.530 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/18’ 2019-12-05T06:12:40.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/13’ 2019-12-05T06:12:40.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/33’ 2019-12-05T06:12:40.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/17’ 2019-12-05T06:12:40.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/35’ 2019-12-05T06:12:40.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./171/37’ 2019-12-05T06:12:40.175 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./171’ 2019-12-05T06:12:40.175 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/19’ 2019-12-05T06:12:40.175 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/15’ 2019-12-05T06:12:40.175 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/31’ 2019-12-05T06:12:40.175 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/36’ 2019-12-05T06:12:40.175 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/20’ 2019-12-05T06:12:40.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/12’ 2019-12-05T06:12:40.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/39’ 2019-12-05T06:12:40.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/10’ 2019-12-05T06:12:40.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/34’ 2019-12-05T06:12:40.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/38’ 2019-12-05T06:12:40.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/11’ 2019-12-05T06:12:40.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/16’ 2019-12-05T06:12:40.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/32’ 2019-12-05T06:12:40.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/30’ 2019-12-05T06:12:40.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/14’ 2019-12-05T06:12:40.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/18’ 2019-12-05T06:12:40.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/13’ 2019-12-05T06:12:40.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/33’ 2019-12-05T06:12:40.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/17’ 2019-12-05T06:12:40.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/35’ 2019-12-05T06:12:40.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./172/37’ 2019-12-05T06:12:40.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./172’ 2019-12-05T06:12:40.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/19’ 2019-12-05T06:12:40.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/15’ 2019-12-05T06:12:40.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/31’ 2019-12-05T06:12:40.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/36’ 2019-12-05T06:12:40.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/20’ 2019-12-05T06:12:40.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/12’ 2019-12-05T06:12:40.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/39’ 2019-12-05T06:12:40.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/10’ 2019-12-05T06:12:40.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/34’ 2019-12-05T06:12:40.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/38’ 2019-12-05T06:12:40.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/11’ 2019-12-05T06:12:40.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/16’ 2019-12-05T06:12:40.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/32’ 2019-12-05T06:12:40.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/30’ 2019-12-05T06:12:40.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/14’ 2019-12-05T06:12:40.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/18’ 2019-12-05T06:12:40.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/13’ 2019-12-05T06:12:40.181 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/33’ 2019-12-05T06:12:40.181 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/17’ 2019-12-05T06:12:40.181 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/35’ 2019-12-05T06:12:40.181 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./173/37’ 2019-12-05T06:12:40.182 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./173’ 2019-12-05T06:12:40.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/19’ 2019-12-05T06:12:40.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/15’ 2019-12-05T06:12:40.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/31’ 2019-12-05T06:12:40.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/36’ 2019-12-05T06:12:40.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/20’ 2019-12-05T06:12:40.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/12’ 2019-12-05T06:12:40.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/39’ 2019-12-05T06:12:40.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/10’ 2019-12-05T06:12:40.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/34’ 2019-12-05T06:12:40.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/38’ 2019-12-05T06:12:40.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/11’ 2019-12-05T06:12:40.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/16’ 2019-12-05T06:12:40.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/32’ 2019-12-05T06:12:40.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/30’ 2019-12-05T06:12:40.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/14’ 2019-12-05T06:12:40.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/18’ 2019-12-05T06:12:40.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/13’ 2019-12-05T06:12:40.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/33’ 2019-12-05T06:12:40.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/17’ 2019-12-05T06:12:40.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/35’ 2019-12-05T06:12:40.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./174/37’ 2019-12-05T06:12:40.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./174’ 2019-12-05T06:12:40.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/19’ 2019-12-05T06:12:40.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/15’ 2019-12-05T06:12:40.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/31’ 2019-12-05T06:12:40.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/36’ 2019-12-05T06:12:40.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/20’ 2019-12-05T06:12:40.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/12’ 2019-12-05T06:12:40.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/39’ 2019-12-05T06:12:40.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/10’ 2019-12-05T06:12:40.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/34’ 2019-12-05T06:12:40.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/38’ 2019-12-05T06:12:40.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/11’ 2019-12-05T06:12:40.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/16’ 2019-12-05T06:12:40.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/32’ 2019-12-05T06:12:40.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/30’ 2019-12-05T06:12:40.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/14’ 2019-12-05T06:12:40.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/18’ 2019-12-05T06:12:40.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/13’ 2019-12-05T06:12:40.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/33’ 2019-12-05T06:12:40.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/17’ 2019-12-05T06:12:40.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/35’ 2019-12-05T06:12:40.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./175/37’ 2019-12-05T06:12:40.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./175’ 2019-12-05T06:12:40.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/19’ 2019-12-05T06:12:40.190 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/15’ 2019-12-05T06:12:40.190 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/31’ 2019-12-05T06:12:40.190 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/36’ 2019-12-05T06:12:40.190 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/20’ 2019-12-05T06:12:40.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/12’ 2019-12-05T06:12:40.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/39’ 2019-12-05T06:12:40.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/10’ 2019-12-05T06:12:40.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/34’ 2019-12-05T06:12:40.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/38’ 2019-12-05T06:12:40.192 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/11’ 2019-12-05T06:12:40.192 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/16’ 2019-12-05T06:12:40.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/32’ 2019-12-05T06:12:40.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/30’ 2019-12-05T06:12:40.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/14’ 2019-12-05T06:12:40.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/18’ 2019-12-05T06:12:40.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/13’ 2019-12-05T06:12:40.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/33’ 2019-12-05T06:12:40.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/17’ 2019-12-05T06:12:40.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/35’ 2019-12-05T06:12:40.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./176/37’ 2019-12-05T06:12:40.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./176’ 2019-12-05T06:12:40.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/19’ 2019-12-05T06:12:40.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/15’ 2019-12-05T06:12:40.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/31’ 2019-12-05T06:12:40.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/36’ 2019-12-05T06:12:40.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/20’ 2019-12-05T06:12:40.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/12’ 2019-12-05T06:12:40.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/39’ 2019-12-05T06:12:40.196 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/10’ 2019-12-05T06:12:40.196 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/34’ 2019-12-05T06:12:40.196 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/38’ 2019-12-05T06:12:40.196 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/11’ 2019-12-05T06:12:40.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/16’ 2019-12-05T06:12:40.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/32’ 2019-12-05T06:12:40.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/30’ 2019-12-05T06:12:40.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/14’ 2019-12-05T06:12:40.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/18’ 2019-12-05T06:12:40.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/13’ 2019-12-05T06:12:40.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/33’ 2019-12-05T06:12:40.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/17’ 2019-12-05T06:12:40.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/35’ 2019-12-05T06:12:40.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./177/37’ 2019-12-05T06:12:40.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./177’ 2019-12-05T06:12:40.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/19’ 2019-12-05T06:12:40.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/15’ 2019-12-05T06:12:40.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/31’ 2019-12-05T06:12:40.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/36’ 2019-12-05T06:12:40.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/20’ 2019-12-05T06:12:40.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/12’ 2019-12-05T06:12:40.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/39’ 2019-12-05T06:12:40.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/10’ 2019-12-05T06:12:40.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/34’ 2019-12-05T06:12:40.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/38’ 2019-12-05T06:12:40.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/11’ 2019-12-05T06:12:40.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/16’ 2019-12-05T06:12:40.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/32’ 2019-12-05T06:12:40.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/30’ 2019-12-05T06:12:40.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/14’ 2019-12-05T06:12:40.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/18’ 2019-12-05T06:12:40.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/13’ 2019-12-05T06:12:40.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/33’ 2019-12-05T06:12:40.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/17’ 2019-12-05T06:12:40.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/35’ 2019-12-05T06:12:40.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./178/37’ 2019-12-05T06:12:40.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./178’ 2019-12-05T06:12:40.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/19’ 2019-12-05T06:12:40.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/15’ 2019-12-05T06:12:40.202 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/31’ 2019-12-05T06:12:40.203 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/36’ 2019-12-05T06:12:40.204 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/20’ 2019-12-05T06:12:40.204 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/12’ 2019-12-05T06:12:40.204 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/39’ 2019-12-05T06:12:40.204 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/10’ 2019-12-05T06:12:40.204 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/34’ 2019-12-05T06:12:40.204 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/38’ 2019-12-05T06:12:40.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/11’ 2019-12-05T06:12:40.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/16’ 2019-12-05T06:12:40.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/32’ 2019-12-05T06:12:40.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/30’ 2019-12-05T06:12:40.205 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/14’ 2019-12-05T06:12:44.636 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/18’ 2019-12-05T06:12:44.637 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/13’ 2019-12-05T06:12:44.637 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/33’ 2019-12-05T06:12:44.637 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/17’ 2019-12-05T06:12:44.637 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/35’ 2019-12-05T06:12:44.637 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./179/37’ 2019-12-05T06:12:44.637 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./179’ 2019-12-05T06:12:44.637 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/19’ 2019-12-05T06:12:44.638 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/15’ 2019-12-05T06:12:44.638 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/31’ 2019-12-05T06:12:44.638 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/36’ 2019-12-05T06:12:44.638 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/20’ 2019-12-05T06:12:44.638 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/12’ 2019-12-05T06:12:44.638 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/39’ 2019-12-05T06:12:44.638 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/10’ 2019-12-05T06:12:44.639 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/34’ 2019-12-05T06:12:44.639 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/38’ 2019-12-05T06:12:44.639 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/11’ 2019-12-05T06:12:44.639 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/16’ 2019-12-05T06:12:44.639 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/32’ 2019-12-05T06:12:44.639 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/30’ 2019-12-05T06:12:44.639 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/14’ 2019-12-05T06:12:44.640 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/18’ 2019-12-05T06:12:44.640 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/13’ 2019-12-05T06:12:44.640 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/33’ 2019-12-05T06:12:44.640 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/17’ 2019-12-05T06:12:44.640 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/35’ 2019-12-05T06:12:44.640 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./180/37’ 2019-12-05T06:12:44.641 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./180’ 2019-12-05T06:12:44.641 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/19’ 2019-12-05T06:12:44.641 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/15’ 2019-12-05T06:12:44.641 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/31’ 2019-12-05T06:12:44.641 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/36’ 2019-12-05T06:12:44.641 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/20’ 2019-12-05T06:12:44.641 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/12’ 2019-12-05T06:12:44.642 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/39’ 2019-12-05T06:12:44.642 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/10’ 2019-12-05T06:12:44.642 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/34’ 2019-12-05T06:12:44.642 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/38’ 2019-12-05T06:12:44.642 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/11’ 2019-12-05T06:12:44.642 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/16’ 2019-12-05T06:12:44.642 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/32’ 2019-12-05T06:12:44.642 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/30’ 2019-12-05T06:12:44.643 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/14’ 2019-12-05T06:12:44.643 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/18’ 2019-12-05T06:12:44.643 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/13’ 2019-12-05T06:12:44.643 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/33’ 2019-12-05T06:12:44.643 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/17’ 2019-12-05T06:12:44.643 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/35’ 2019-12-05T06:12:44.643 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./181/37’ 2019-12-05T06:12:44.644 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./181’ 2019-12-05T06:12:44.644 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/19’ 2019-12-05T06:12:44.644 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/15’ 2019-12-05T06:12:44.644 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/31’ 2019-12-05T06:12:44.644 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/36’ 2019-12-05T06:12:44.644 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/20’ 2019-12-05T06:12:44.645 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/12’ 2019-12-05T06:12:44.645 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/39’ 2019-12-05T06:12:44.645 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/10’ 2019-12-05T06:12:44.645 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/34’ 2019-12-05T06:12:44.645 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/38’ 2019-12-05T06:12:44.645 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/11’ 2019-12-05T06:12:44.645 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/16’ 2019-12-05T06:12:44.646 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/32’ 2019-12-05T06:12:44.646 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/30’ 2019-12-05T06:12:44.646 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/14’ 2019-12-05T06:12:44.646 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/18’ 2019-12-05T06:12:44.646 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/13’ 2019-12-05T06:12:44.646 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/33’ 2019-12-05T06:12:44.646 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/17’ 2019-12-05T06:12:44.647 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/35’ 2019-12-05T06:12:44.647 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./182/37’ 2019-12-05T06:12:44.647 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./182’ 2019-12-05T06:12:44.647 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/19’ 2019-12-05T06:12:44.647 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/15’ 2019-12-05T06:12:44.647 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/31’ 2019-12-05T06:12:44.647 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/36’ 2019-12-05T06:12:44.648 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/20’ 2019-12-05T06:12:44.648 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/12’ 2019-12-05T06:12:44.648 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/39’ 2019-12-05T06:12:44.648 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/10’ 2019-12-05T06:12:44.652 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/34’ 2019-12-05T06:12:44.652 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/38’ 2019-12-05T06:12:44.652 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/11’ 2019-12-05T06:12:44.652 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/16’ 2019-12-05T06:12:44.653 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/32’ 2019-12-05T06:12:44.653 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/30’ 2019-12-05T06:12:44.653 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/14’ 2019-12-05T06:12:44.653 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/18’ 2019-12-05T06:12:44.653 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/13’ 2019-12-05T06:12:44.653 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/33’ 2019-12-05T06:12:44.653 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/17’ 2019-12-05T06:12:44.654 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/35’ 2019-12-05T06:12:44.654 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./183/37’ 2019-12-05T06:12:44.654 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./183’ 2019-12-05T06:12:44.654 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/19’ 2019-12-05T06:12:44.654 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/15’ 2019-12-05T06:12:44.654 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/31’ 2019-12-05T06:12:44.654 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/36’ 2019-12-05T06:12:44.655 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/20’ 2019-12-05T06:12:44.655 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/12’ 2019-12-05T06:12:44.655 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/39’ 2019-12-05T06:12:44.655 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/10’ 2019-12-05T06:12:44.655 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/34’ 2019-12-05T06:12:44.655 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/38’ 2019-12-05T06:12:44.655 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/11’ 2019-12-05T06:12:44.656 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/16’ 2019-12-05T06:12:44.656 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/32’ 2019-12-05T06:12:44.656 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/30’ 2019-12-05T06:12:44.656 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/14’ 2019-12-05T06:12:44.656 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/18’ 2019-12-05T06:12:44.656 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/13’ 2019-12-05T06:12:44.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/33’ 2019-12-05T06:12:44.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/17’ 2019-12-05T06:12:44.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/35’ 2019-12-05T06:12:44.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./184/37’ 2019-12-05T06:12:44.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./184’ 2019-12-05T06:12:44.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/19’ 2019-12-05T06:12:44.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/15’ 2019-12-05T06:12:44.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/31’ 2019-12-05T06:12:44.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/36’ 2019-12-05T06:12:44.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/20’ 2019-12-05T06:12:44.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/12’ 2019-12-05T06:12:44.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/39’ 2019-12-05T06:12:44.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/10’ 2019-12-05T06:12:44.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/34’ 2019-12-05T06:12:44.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/38’ 2019-12-05T06:12:44.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/11’ 2019-12-05T06:12:44.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/16’ 2019-12-05T06:12:44.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/32’ 2019-12-05T06:12:44.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/30’ 2019-12-05T06:12:44.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/14’ 2019-12-05T06:12:44.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/18’ 2019-12-05T06:12:44.660 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/13’ 2019-12-05T06:12:44.660 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/33’ 2019-12-05T06:12:44.660 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/17’ 2019-12-05T06:12:44.660 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/35’ 2019-12-05T06:12:44.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./185/37’ 2019-12-05T06:12:44.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./185’ 2019-12-05T06:12:44.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/19’ 2019-12-05T06:12:44.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/15’ 2019-12-05T06:12:44.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/31’ 2019-12-05T06:12:44.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/36’ 2019-12-05T06:12:44.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/20’ 2019-12-05T06:12:44.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/12’ 2019-12-05T06:12:44.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/39’ 2019-12-05T06:12:44.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/10’ 2019-12-05T06:12:44.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/34’ 2019-12-05T06:12:44.663 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/38’ 2019-12-05T06:12:44.663 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/11’ 2019-12-05T06:12:44.663 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/16’ 2019-12-05T06:12:44.663 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/32’ 2019-12-05T06:12:44.664 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/30’ 2019-12-05T06:12:44.664 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/14’ 2019-12-05T06:12:44.664 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/18’ 2019-12-05T06:12:44.664 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/13’ 2019-12-05T06:12:44.664 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/33’ 2019-12-05T06:12:44.665 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/17’ 2019-12-05T06:12:44.665 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/35’ 2019-12-05T06:12:44.665 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./186/37’ 2019-12-05T06:12:44.665 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./186’ 2019-12-05T06:12:44.666 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/19’ 2019-12-05T06:12:44.666 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/15’ 2019-12-05T06:12:44.666 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/31’ 2019-12-05T06:12:44.666 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/36’ 2019-12-05T06:12:44.666 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/20’ 2019-12-05T06:12:44.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/12’ 2019-12-05T06:12:44.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/39’ 2019-12-05T06:12:44.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/10’ 2019-12-05T06:12:44.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/34’ 2019-12-05T06:12:44.668 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/38’ 2019-12-05T06:12:44.668 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/11’ 2019-12-05T06:12:44.668 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/16’ 2019-12-05T06:12:44.668 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/32’ 2019-12-05T06:12:44.668 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/30’ 2019-12-05T06:12:46.225 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/14’ 2019-12-05T06:12:46.225 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/18’ 2019-12-05T06:12:46.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/13’ 2019-12-05T06:12:46.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/33’ 2019-12-05T06:12:46.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/17’ 2019-12-05T06:12:46.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/35’ 2019-12-05T06:12:46.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./187/37’ 2019-12-05T06:12:46.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./187’ 2019-12-05T06:12:46.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/19’ 2019-12-05T06:12:46.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/15’ 2019-12-05T06:12:46.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/31’ 2019-12-05T06:12:46.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/36’ 2019-12-05T06:12:46.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/20’ 2019-12-05T06:12:46.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/12’ 2019-12-05T06:12:46.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/39’ 2019-12-05T06:12:46.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/10’ 2019-12-05T06:12:46.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/34’ 2019-12-05T06:12:46.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/38’ 2019-12-05T06:12:46.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/11’ 2019-12-05T06:12:46.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/16’ 2019-12-05T06:12:46.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/32’ 2019-12-05T06:12:46.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/30’ 2019-12-05T06:12:46.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/14’ 2019-12-05T06:12:46.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/18’ 2019-12-05T06:12:46.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/13’ 2019-12-05T06:12:46.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/33’ 2019-12-05T06:12:46.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/17’ 2019-12-05T06:12:46.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/35’ 2019-12-05T06:12:46.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./188/37’ 2019-12-05T06:12:46.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./188’ 2019-12-05T06:12:46.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/19’ 2019-12-05T06:12:46.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/15’ 2019-12-05T06:12:46.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/31’ 2019-12-05T06:12:46.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/36’ 2019-12-05T06:12:46.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/20’ 2019-12-05T06:12:46.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/12’ 2019-12-05T06:12:46.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/39’ 2019-12-05T06:12:46.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/10’ 2019-12-05T06:12:46.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/34’ 2019-12-05T06:12:46.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/38’ 2019-12-05T06:12:46.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/11’ 2019-12-05T06:12:46.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/16’ 2019-12-05T06:12:46.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/32’ 2019-12-05T06:12:46.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/30’ 2019-12-05T06:12:46.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/14’ 2019-12-05T06:12:46.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/18’ 2019-12-05T06:12:46.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/13’ 2019-12-05T06:12:46.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/33’ 2019-12-05T06:12:46.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/17’ 2019-12-05T06:12:46.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/35’ 2019-12-05T06:12:46.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./189/37’ 2019-12-05T06:12:46.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./189’ 2019-12-05T06:12:46.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/19’ 2019-12-05T06:12:46.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/15’ 2019-12-05T06:12:46.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/31’ 2019-12-05T06:12:46.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/36’ 2019-12-05T06:12:46.234 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/20’ 2019-12-05T06:12:46.234 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/12’ 2019-12-05T06:12:46.234 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/39’ 2019-12-05T06:12:46.234 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/10’ 2019-12-05T06:12:46.234 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/34’ 2019-12-05T06:12:46.234 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/38’ 2019-12-05T06:12:46.234 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/11’ 2019-12-05T06:12:46.235 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/16’ 2019-12-05T06:12:46.235 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/32’ 2019-12-05T06:12:46.235 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/30’ 2019-12-05T06:12:46.235 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/14’ 2019-12-05T06:12:46.235 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/18’ 2019-12-05T06:12:46.235 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/13’ 2019-12-05T06:12:46.235 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/33’ 2019-12-05T06:12:46.236 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/17’ 2019-12-05T06:12:46.236 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/35’ 2019-12-05T06:12:46.236 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./190/37’ 2019-12-05T06:12:46.236 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./190’ 2019-12-05T06:12:46.236 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/19’ 2019-12-05T06:12:46.236 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/15’ 2019-12-05T06:12:46.237 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/31’ 2019-12-05T06:12:46.237 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/36’ 2019-12-05T06:12:46.237 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/20’ 2019-12-05T06:12:46.237 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/12’ 2019-12-05T06:12:46.237 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/39’ 2019-12-05T06:12:46.237 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/10’ 2019-12-05T06:12:46.237 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/34’ 2019-12-05T06:12:46.238 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/38’ 2019-12-05T06:12:46.238 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/11’ 2019-12-05T06:12:46.238 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/16’ 2019-12-05T06:12:46.238 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/32’ 2019-12-05T06:12:46.238 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/30’ 2019-12-05T06:12:46.238 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/14’ 2019-12-05T06:12:46.238 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/18’ 2019-12-05T06:12:46.239 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/13’ 2019-12-05T06:12:46.239 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/33’ 2019-12-05T06:12:46.239 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/17’ 2019-12-05T06:12:46.239 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/35’ 2019-12-05T06:12:46.239 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./191/37’ 2019-12-05T06:12:46.239 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./191’ 2019-12-05T06:12:46.239 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/19’ 2019-12-05T06:12:46.240 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/15’ 2019-12-05T06:12:46.240 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/31’ 2019-12-05T06:12:46.240 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/36’ 2019-12-05T06:12:46.240 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/20’ 2019-12-05T06:12:46.240 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/12’ 2019-12-05T06:12:46.241 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/39’ 2019-12-05T06:12:46.241 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/10’ 2019-12-05T06:12:46.241 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/34’ 2019-12-05T06:12:46.241 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/38’ 2019-12-05T06:12:46.241 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/11’ 2019-12-05T06:12:46.241 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/16’ 2019-12-05T06:12:46.241 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/32’ 2019-12-05T06:12:46.242 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/30’ 2019-12-05T06:12:46.242 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/14’ 2019-12-05T06:12:46.242 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/18’ 2019-12-05T06:12:46.242 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/13’ 2019-12-05T06:12:46.242 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/33’ 2019-12-05T06:12:46.242 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/17’ 2019-12-05T06:12:46.242 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/35’ 2019-12-05T06:12:46.243 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./192/37’ 2019-12-05T06:12:46.243 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./192’ 2019-12-05T06:12:46.243 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/19’ 2019-12-05T06:12:46.243 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/15’ 2019-12-05T06:12:46.243 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/31’ 2019-12-05T06:12:46.243 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/36’ 2019-12-05T06:12:46.243 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/20’ 2019-12-05T06:12:46.244 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/12’ 2019-12-05T06:12:46.244 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/39’ 2019-12-05T06:12:46.244 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/10’ 2019-12-05T06:12:46.244 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/34’ 2019-12-05T06:12:46.244 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/38’ 2019-12-05T06:12:46.244 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/11’ 2019-12-05T06:12:46.244 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/16’ 2019-12-05T06:12:46.245 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/32’ 2019-12-05T06:12:46.245 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/30’ 2019-12-05T06:12:46.245 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/14’ 2019-12-05T06:12:46.245 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/18’ 2019-12-05T06:12:46.245 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/13’ 2019-12-05T06:12:46.245 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/33’ 2019-12-05T06:12:46.245 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/17’ 2019-12-05T06:12:46.246 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/35’ 2019-12-05T06:12:46.246 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./193/37’ 2019-12-05T06:12:46.246 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./193’ 2019-12-05T06:12:46.246 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/19’ 2019-12-05T06:12:46.246 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/15’ 2019-12-05T06:12:46.246 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/31’ 2019-12-05T06:12:46.247 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/36’ 2019-12-05T06:12:46.247 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/20’ 2019-12-05T06:12:46.247 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/12’ 2019-12-05T06:12:46.247 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/39’ 2019-12-05T06:12:46.247 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/10’ 2019-12-05T06:12:46.247 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/34’ 2019-12-05T06:12:46.248 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/38’ 2019-12-05T06:12:46.248 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/11’ 2019-12-05T06:12:46.248 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/16’ 2019-12-05T06:12:46.248 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/32’ 2019-12-05T06:12:46.248 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/30’ 2019-12-05T06:12:46.248 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/14’ 2019-12-05T06:12:46.249 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/18’ 2019-12-05T06:12:46.249 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/13’ 2019-12-05T06:12:46.249 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/33’ 2019-12-05T06:12:46.249 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/17’ 2019-12-05T06:12:46.249 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/35’ 2019-12-05T06:12:46.249 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./194/37’ 2019-12-05T06:12:46.250 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./194’ 2019-12-05T06:12:46.250 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/19’ 2019-12-05T06:12:46.250 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/15’ 2019-12-05T06:12:46.250 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/31’ 2019-12-05T06:12:46.250 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/36’ 2019-12-05T06:12:46.250 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/20’ 2019-12-05T06:12:46.250 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/12’ 2019-12-05T06:12:46.251 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/39’ 2019-12-05T06:12:46.251 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/10’ 2019-12-05T06:12:46.251 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/34’ 2019-12-05T06:12:46.251 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/38’ 2019-12-05T06:12:46.251 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/11’ 2019-12-05T06:12:46.251 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/16’ 2019-12-05T06:12:46.251 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/32’ 2019-12-05T06:12:46.252 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/30’ 2019-12-05T06:12:48.800 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/14’ 2019-12-05T06:12:48.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/18’ 2019-12-05T06:12:48.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/13’ 2019-12-05T06:12:48.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/33’ 2019-12-05T06:12:48.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/17’ 2019-12-05T06:12:48.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/35’ 2019-12-05T06:12:48.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./195/37’ 2019-12-05T06:12:48.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./195’ 2019-12-05T06:12:48.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/19’ 2019-12-05T06:12:48.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/15’ 2019-12-05T06:12:48.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/31’ 2019-12-05T06:12:48.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/36’ 2019-12-05T06:12:48.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/20’ 2019-12-05T06:12:48.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/12’ 2019-12-05T06:12:48.803 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/39’ 2019-12-05T06:12:48.803 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/10’ 2019-12-05T06:12:48.803 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/34’ 2019-12-05T06:12:48.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/38’ 2019-12-05T06:12:48.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/11’ 2019-12-05T06:12:48.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/16’ 2019-12-05T06:12:48.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/32’ 2019-12-05T06:12:48.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/30’ 2019-12-05T06:12:48.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/14’ 2019-12-05T06:12:48.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/18’ 2019-12-05T06:12:48.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/13’ 2019-12-05T06:12:48.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/33’ 2019-12-05T06:12:48.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/17’ 2019-12-05T06:12:48.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/35’ 2019-12-05T06:12:48.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./196/37’ 2019-12-05T06:12:48.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./196’ 2019-12-05T06:12:48.806 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/19’ 2019-12-05T06:12:48.806 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/15’ 2019-12-05T06:12:48.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/31’ 2019-12-05T06:12:48.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/36’ 2019-12-05T06:12:48.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/20’ 2019-12-05T06:12:48.808 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/12’ 2019-12-05T06:12:48.808 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/39’ 2019-12-05T06:12:48.808 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/10’ 2019-12-05T06:12:48.808 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/34’ 2019-12-05T06:12:48.808 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/38’ 2019-12-05T06:12:48.808 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/11’ 2019-12-05T06:12:48.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/16’ 2019-12-05T06:12:48.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/32’ 2019-12-05T06:12:48.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/30’ 2019-12-05T06:12:48.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/14’ 2019-12-05T06:12:48.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/18’ 2019-12-05T06:12:48.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/13’ 2019-12-05T06:12:48.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/33’ 2019-12-05T06:12:48.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/17’ 2019-12-05T06:12:48.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/35’ 2019-12-05T06:12:48.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./197/37’ 2019-12-05T06:12:48.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./197’ 2019-12-05T06:12:48.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/19’ 2019-12-05T06:12:48.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/15’ 2019-12-05T06:12:48.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/31’ 2019-12-05T06:12:48.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/36’ 2019-12-05T06:12:48.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/20’ 2019-12-05T06:12:48.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/12’ 2019-12-05T06:12:48.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/39’ 2019-12-05T06:12:48.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/10’ 2019-12-05T06:12:48.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/34’ 2019-12-05T06:12:48.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/38’ 2019-12-05T06:12:48.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/11’ 2019-12-05T06:12:48.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/16’ 2019-12-05T06:12:48.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/32’ 2019-12-05T06:12:48.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/30’ 2019-12-05T06:12:48.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/14’ 2019-12-05T06:12:48.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/18’ 2019-12-05T06:12:48.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/13’ 2019-12-05T06:12:48.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/33’ 2019-12-05T06:12:48.814 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/17’ 2019-12-05T06:12:48.814 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/35’ 2019-12-05T06:12:48.814 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./198/37’ 2019-12-05T06:12:48.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./198’ 2019-12-05T06:12:48.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/19’ 2019-12-05T06:12:48.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/15’ 2019-12-05T06:12:48.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/31’ 2019-12-05T06:12:48.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/36’ 2019-12-05T06:12:48.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/20’ 2019-12-05T06:12:48.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/12’ 2019-12-05T06:12:48.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/39’ 2019-12-05T06:12:48.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/10’ 2019-12-05T06:12:48.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/34’ 2019-12-05T06:12:48.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/38’ 2019-12-05T06:12:48.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/11’ 2019-12-05T06:12:48.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/16’ 2019-12-05T06:12:48.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/32’ 2019-12-05T06:12:48.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/30’ 2019-12-05T06:12:48.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/14’ 2019-12-05T06:12:48.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/18’ 2019-12-05T06:12:48.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/13’ 2019-12-05T06:12:48.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/33’ 2019-12-05T06:12:48.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/17’ 2019-12-05T06:12:48.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/35’ 2019-12-05T06:12:48.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./199/37’ 2019-12-05T06:12:48.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./199’ 2019-12-05T06:12:48.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/21’ 2019-12-05T06:12:48.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/27’ 2019-12-05T06:12:48.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/31’ 2019-12-05T06:12:48.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/36’ 2019-12-05T06:12:48.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/20’ 2019-12-05T06:12:48.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/39’ 2019-12-05T06:12:48.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/34’ 2019-12-05T06:12:48.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/38’ 2019-12-05T06:12:48.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/28’ 2019-12-05T06:12:48.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/24’ 2019-12-05T06:12:48.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/29’ 2019-12-05T06:12:48.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/32’ 2019-12-05T06:12:48.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/22’ 2019-12-05T06:12:48.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/25’ 2019-12-05T06:12:48.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/30’ 2019-12-05T06:12:48.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/33’ 2019-12-05T06:12:48.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/23’ 2019-12-05T06:12:48.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/35’ 2019-12-05T06:12:48.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/26’ 2019-12-05T06:12:48.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./200/37’ 2019-12-05T06:12:48.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./200’ 2019-12-05T06:12:48.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/21’ 2019-12-05T06:12:48.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/27’ 2019-12-05T06:12:48.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/31’ 2019-12-05T06:12:48.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/36’ 2019-12-05T06:12:48.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/20’ 2019-12-05T06:12:48.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/39’ 2019-12-05T06:12:48.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/34’ 2019-12-05T06:12:48.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/38’ 2019-12-05T06:12:48.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/28’ 2019-12-05T06:12:48.824 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/24’ 2019-12-05T06:12:48.824 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/29’ 2019-12-05T06:12:48.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/32’ 2019-12-05T06:12:48.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/22’ 2019-12-05T06:12:48.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/25’ 2019-12-05T06:12:48.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/30’ 2019-12-05T06:12:48.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/33’ 2019-12-05T06:12:48.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/23’ 2019-12-05T06:12:48.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/35’ 2019-12-05T06:12:48.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/26’ 2019-12-05T06:12:48.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./201/37’ 2019-12-05T06:12:48.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./201’ 2019-12-05T06:12:48.827 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/21’ 2019-12-05T06:12:48.827 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/27’ 2019-12-05T06:12:48.829 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/31’ 2019-12-05T06:12:48.829 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/36’ 2019-12-05T06:12:48.829 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/20’ 2019-12-05T06:12:48.829 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/39’ 2019-12-05T06:12:48.830 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/34’ 2019-12-05T06:12:48.830 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/38’ 2019-12-05T06:12:48.830 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/28’ 2019-12-05T06:12:48.831 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/24’ 2019-12-05T06:12:48.831 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/29’ 2019-12-05T06:12:48.831 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/32’ 2019-12-05T06:12:48.832 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/22’ 2019-12-05T06:12:48.832 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/25’ 2019-12-05T06:12:48.832 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/30’ 2019-12-05T06:12:48.832 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/33’ 2019-12-05T06:12:48.832 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/23’ 2019-12-05T06:12:48.833 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/35’ 2019-12-05T06:12:48.833 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/26’ 2019-12-05T06:12:48.833 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./202/37’ 2019-12-05T06:12:48.833 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./202’ 2019-12-05T06:12:48.833 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/21’ 2019-12-05T06:12:48.833 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/27’ 2019-12-05T06:12:48.834 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/31’ 2019-12-05T06:12:48.834 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/36’ 2019-12-05T06:12:48.834 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/20’ 2019-12-05T06:12:48.834 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/39’ 2019-12-05T06:12:48.836 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/34’ 2019-12-05T06:12:48.836 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/38’ 2019-12-05T06:12:48.837 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/28’ 2019-12-05T06:12:48.837 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/24’ 2019-12-05T06:12:48.837 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/29’ 2019-12-05T06:12:48.837 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/32’ 2019-12-05T06:12:48.837 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/22’ 2019-12-05T06:12:48.837 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/25’ 2019-12-05T06:12:48.838 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/30’ 2019-12-05T06:12:48.838 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/33’ 2019-12-05T06:12:51.874 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/23’ 2019-12-05T06:12:51.875 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/35’ 2019-12-05T06:12:51.875 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/26’ 2019-12-05T06:12:51.875 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./203/37’ 2019-12-05T06:12:51.875 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./203’ 2019-12-05T06:12:51.875 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/21’ 2019-12-05T06:12:51.876 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/27’ 2019-12-05T06:12:51.876 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/31’ 2019-12-05T06:12:51.876 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/36’ 2019-12-05T06:12:51.876 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/20’ 2019-12-05T06:12:51.876 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/39’ 2019-12-05T06:12:51.876 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/34’ 2019-12-05T06:12:51.876 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/38’ 2019-12-05T06:12:51.877 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/28’ 2019-12-05T06:12:51.877 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/24’ 2019-12-05T06:12:51.877 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/29’ 2019-12-05T06:12:51.877 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/32’ 2019-12-05T06:12:51.877 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/22’ 2019-12-05T06:12:51.877 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/25’ 2019-12-05T06:12:51.877 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/30’ 2019-12-05T06:12:51.878 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/33’ 2019-12-05T06:12:51.878 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/23’ 2019-12-05T06:12:51.878 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/35’ 2019-12-05T06:12:51.878 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/26’ 2019-12-05T06:12:51.878 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./204/37’ 2019-12-05T06:12:51.878 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./204’ 2019-12-05T06:12:51.878 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/21’ 2019-12-05T06:12:51.879 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/27’ 2019-12-05T06:12:51.879 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/31’ 2019-12-05T06:12:51.879 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/36’ 2019-12-05T06:12:51.879 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/20’ 2019-12-05T06:12:51.879 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/39’ 2019-12-05T06:12:51.879 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/34’ 2019-12-05T06:12:51.879 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/38’ 2019-12-05T06:12:51.880 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/28’ 2019-12-05T06:12:51.880 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/24’ 2019-12-05T06:12:51.880 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/29’ 2019-12-05T06:12:51.880 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/32’ 2019-12-05T06:12:51.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/22’ 2019-12-05T06:12:51.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/25’ 2019-12-05T06:12:51.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/30’ 2019-12-05T06:12:51.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/33’ 2019-12-05T06:12:51.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/23’ 2019-12-05T06:12:51.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/35’ 2019-12-05T06:12:51.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/26’ 2019-12-05T06:12:51.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./205/37’ 2019-12-05T06:12:51.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./205’ 2019-12-05T06:12:51.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/21’ 2019-12-05T06:12:51.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/27’ 2019-12-05T06:12:51.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/31’ 2019-12-05T06:12:51.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/36’ 2019-12-05T06:12:51.883 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/20’ 2019-12-05T06:12:51.883 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/39’ 2019-12-05T06:12:51.883 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/34’ 2019-12-05T06:12:51.883 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/38’ 2019-12-05T06:12:51.883 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/28’ 2019-12-05T06:12:51.883 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/24’ 2019-12-05T06:12:51.883 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/29’ 2019-12-05T06:12:51.884 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/32’ 2019-12-05T06:12:51.884 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/22’ 2019-12-05T06:12:51.884 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/25’ 2019-12-05T06:12:51.884 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/30’ 2019-12-05T06:12:51.884 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/33’ 2019-12-05T06:12:51.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/23’ 2019-12-05T06:12:51.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/35’ 2019-12-05T06:12:51.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/26’ 2019-12-05T06:12:51.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./206/37’ 2019-12-05T06:12:51.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./206’ 2019-12-05T06:12:51.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/21’ 2019-12-05T06:12:51.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/27’ 2019-12-05T06:12:51.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/31’ 2019-12-05T06:12:51.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/36’ 2019-12-05T06:12:51.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/20’ 2019-12-05T06:12:51.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/39’ 2019-12-05T06:12:51.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/34’ 2019-12-05T06:12:51.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/38’ 2019-12-05T06:12:51.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/28’ 2019-12-05T06:12:51.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/24’ 2019-12-05T06:12:51.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/29’ 2019-12-05T06:12:51.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/32’ 2019-12-05T06:12:51.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/22’ 2019-12-05T06:12:51.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/25’ 2019-12-05T06:12:51.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/30’ 2019-12-05T06:12:51.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/33’ 2019-12-05T06:12:51.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/23’ 2019-12-05T06:12:51.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/35’ 2019-12-05T06:12:51.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/26’ 2019-12-05T06:12:51.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./207/37’ 2019-12-05T06:12:51.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./207’ 2019-12-05T06:12:51.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/21’ 2019-12-05T06:12:51.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/27’ 2019-12-05T06:12:51.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/31’ 2019-12-05T06:12:51.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/36’ 2019-12-05T06:12:51.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/20’ 2019-12-05T06:12:51.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/39’ 2019-12-05T06:12:51.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/34’ 2019-12-05T06:12:51.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/38’ 2019-12-05T06:12:51.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/28’ 2019-12-05T06:12:51.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/24’ 2019-12-05T06:12:51.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/29’ 2019-12-05T06:12:51.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/32’ 2019-12-05T06:12:51.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/22’ 2019-12-05T06:12:51.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/25’ 2019-12-05T06:12:51.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/30’ 2019-12-05T06:12:51.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/33’ 2019-12-05T06:12:51.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/23’ 2019-12-05T06:12:51.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/35’ 2019-12-05T06:12:51.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/26’ 2019-12-05T06:12:51.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./208/37’ 2019-12-05T06:12:51.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./208’ 2019-12-05T06:12:51.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/21’ 2019-12-05T06:12:51.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/27’ 2019-12-05T06:12:51.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/31’ 2019-12-05T06:12:51.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/36’ 2019-12-05T06:12:51.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/20’ 2019-12-05T06:12:51.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/39’ 2019-12-05T06:12:51.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/34’ 2019-12-05T06:12:51.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/38’ 2019-12-05T06:12:51.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/28’ 2019-12-05T06:12:51.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/24’ 2019-12-05T06:12:51.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/29’ 2019-12-05T06:12:51.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/32’ 2019-12-05T06:12:51.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/22’ 2019-12-05T06:12:51.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/25’ 2019-12-05T06:12:51.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/30’ 2019-12-05T06:12:51.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/33’ 2019-12-05T06:12:51.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/23’ 2019-12-05T06:12:51.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/35’ 2019-12-05T06:12:51.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/26’ 2019-12-05T06:12:51.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./209/37’ 2019-12-05T06:12:51.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./209’ 2019-12-05T06:12:51.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/21’ 2019-12-05T06:12:51.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/27’ 2019-12-05T06:12:51.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/31’ 2019-12-05T06:12:51.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/36’ 2019-12-05T06:12:51.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/20’ 2019-12-05T06:12:51.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/39’ 2019-12-05T06:12:51.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/34’ 2019-12-05T06:12:51.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/38’ 2019-12-05T06:12:51.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/28’ 2019-12-05T06:12:51.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/24’ 2019-12-05T06:12:51.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/29’ 2019-12-05T06:12:51.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/32’ 2019-12-05T06:12:51.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/22’ 2019-12-05T06:12:51.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/25’ 2019-12-05T06:12:51.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/30’ 2019-12-05T06:12:51.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/33’ 2019-12-05T06:12:51.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/23’ 2019-12-05T06:12:51.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/35’ 2019-12-05T06:12:51.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/26’ 2019-12-05T06:12:51.898 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./210/37’ 2019-12-05T06:12:51.898 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./210’ 2019-12-05T06:12:51.898 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/21’ 2019-12-05T06:12:51.898 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/27’ 2019-12-05T06:12:51.898 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/31’ 2019-12-05T06:12:51.898 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/36’ 2019-12-05T06:12:51.898 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/20’ 2019-12-05T06:12:51.899 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/39’ 2019-12-05T06:12:51.899 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/34’ 2019-12-05T06:12:51.899 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/38’ 2019-12-05T06:12:51.899 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/28’ 2019-12-05T06:12:51.899 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/24’ 2019-12-05T06:12:51.900 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/29’ 2019-12-05T06:12:51.900 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/32’ 2019-12-05T06:12:51.900 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/22’ 2019-12-05T06:12:51.900 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/25’ 2019-12-05T06:12:51.900 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/30’ 2019-12-05T06:12:51.902 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/33’ 2019-12-05T06:12:51.902 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/23’ 2019-12-05T06:12:51.903 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/35’ 2019-12-05T06:12:51.903 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/26’ 2019-12-05T06:12:51.903 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./211/37’ 2019-12-05T06:12:51.903 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./211’ 2019-12-05T06:12:51.903 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/21’ 2019-12-05T06:12:51.904 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/27’ 2019-12-05T06:12:54.493 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/31’ 2019-12-05T06:12:54.493 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/36’ 2019-12-05T06:12:54.494 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/20’ 2019-12-05T06:12:54.494 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/39’ 2019-12-05T06:12:54.495 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/34’ 2019-12-05T06:12:54.495 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/38’ 2019-12-05T06:12:54.495 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/28’ 2019-12-05T06:12:54.495 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/24’ 2019-12-05T06:12:54.495 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/29’ 2019-12-05T06:12:54.496 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/32’ 2019-12-05T06:12:54.496 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/22’ 2019-12-05T06:12:54.496 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/25’ 2019-12-05T06:12:54.496 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/30’ 2019-12-05T06:12:54.496 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/33’ 2019-12-05T06:12:54.496 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/23’ 2019-12-05T06:12:54.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/35’ 2019-12-05T06:12:54.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/26’ 2019-12-05T06:12:54.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./212/37’ 2019-12-05T06:12:54.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./212’ 2019-12-05T06:12:54.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/21’ 2019-12-05T06:12:54.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/27’ 2019-12-05T06:12:54.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/31’ 2019-12-05T06:12:54.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/36’ 2019-12-05T06:12:54.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/20’ 2019-12-05T06:12:54.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/39’ 2019-12-05T06:12:54.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/34’ 2019-12-05T06:12:54.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/38’ 2019-12-05T06:12:54.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/28’ 2019-12-05T06:12:54.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/24’ 2019-12-05T06:12:54.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/29’ 2019-12-05T06:12:54.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/32’ 2019-12-05T06:12:54.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/22’ 2019-12-05T06:12:54.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/25’ 2019-12-05T06:12:54.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/30’ 2019-12-05T06:12:54.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/33’ 2019-12-05T06:12:54.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/23’ 2019-12-05T06:12:54.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/35’ 2019-12-05T06:12:54.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/26’ 2019-12-05T06:12:54.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./213/37’ 2019-12-05T06:12:54.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./213’ 2019-12-05T06:12:54.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/21’ 2019-12-05T06:12:54.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/27’ 2019-12-05T06:12:54.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/31’ 2019-12-05T06:12:54.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/36’ 2019-12-05T06:12:54.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/20’ 2019-12-05T06:12:54.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/39’ 2019-12-05T06:12:54.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/34’ 2019-12-05T06:12:54.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/38’ 2019-12-05T06:12:54.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/28’ 2019-12-05T06:12:54.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/24’ 2019-12-05T06:12:54.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/29’ 2019-12-05T06:12:54.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/32’ 2019-12-05T06:12:54.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/22’ 2019-12-05T06:12:54.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/25’ 2019-12-05T06:12:54.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/30’ 2019-12-05T06:12:54.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/33’ 2019-12-05T06:12:54.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/23’ 2019-12-05T06:12:54.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/35’ 2019-12-05T06:12:54.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/26’ 2019-12-05T06:12:54.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./214/37’ 2019-12-05T06:12:54.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./214’ 2019-12-05T06:12:54.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/21’ 2019-12-05T06:12:54.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/27’ 2019-12-05T06:12:54.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/31’ 2019-12-05T06:12:54.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/36’ 2019-12-05T06:12:54.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/20’ 2019-12-05T06:12:54.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/39’ 2019-12-05T06:12:54.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/34’ 2019-12-05T06:12:54.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/38’ 2019-12-05T06:12:54.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/28’ 2019-12-05T06:12:54.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/24’ 2019-12-05T06:12:54.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/29’ 2019-12-05T06:12:54.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/32’ 2019-12-05T06:12:54.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/22’ 2019-12-05T06:12:54.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/25’ 2019-12-05T06:12:54.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/30’ 2019-12-05T06:12:54.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/33’ 2019-12-05T06:12:54.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/23’ 2019-12-05T06:12:54.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/35’ 2019-12-05T06:12:54.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/26’ 2019-12-05T06:12:54.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./215/37’ 2019-12-05T06:12:54.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./215’ 2019-12-05T06:12:54.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/21’ 2019-12-05T06:12:54.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/27’ 2019-12-05T06:12:54.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/31’ 2019-12-05T06:12:54.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/36’ 2019-12-05T06:12:54.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/20’ 2019-12-05T06:12:54.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/39’ 2019-12-05T06:12:54.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/34’ 2019-12-05T06:12:54.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/38’ 2019-12-05T06:12:54.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/28’ 2019-12-05T06:12:54.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/24’ 2019-12-05T06:12:54.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/29’ 2019-12-05T06:12:54.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/32’ 2019-12-05T06:12:54.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/22’ 2019-12-05T06:12:54.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/25’ 2019-12-05T06:12:54.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/30’ 2019-12-05T06:12:54.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/33’ 2019-12-05T06:12:54.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/23’ 2019-12-05T06:12:54.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/35’ 2019-12-05T06:12:54.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/26’ 2019-12-05T06:12:54.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./216/37’ 2019-12-05T06:12:54.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./216’ 2019-12-05T06:12:54.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/21’ 2019-12-05T06:12:54.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/27’ 2019-12-05T06:12:54.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/31’ 2019-12-05T06:12:54.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/36’ 2019-12-05T06:12:54.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/20’ 2019-12-05T06:12:54.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/39’ 2019-12-05T06:12:54.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/34’ 2019-12-05T06:12:54.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/38’ 2019-12-05T06:12:54.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/28’ 2019-12-05T06:12:54.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/24’ 2019-12-05T06:12:54.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/29’ 2019-12-05T06:12:54.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/32’ 2019-12-05T06:12:54.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/22’ 2019-12-05T06:12:54.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/25’ 2019-12-05T06:12:54.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/30’ 2019-12-05T06:12:54.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/33’ 2019-12-05T06:12:54.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/23’ 2019-12-05T06:12:54.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/35’ 2019-12-05T06:12:54.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/26’ 2019-12-05T06:12:54.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./217/37’ 2019-12-05T06:12:54.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./217’ 2019-12-05T06:12:54.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/21’ 2019-12-05T06:12:54.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/27’ 2019-12-05T06:12:54.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/31’ 2019-12-05T06:12:54.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/36’ 2019-12-05T06:12:54.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/20’ 2019-12-05T06:12:54.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/39’ 2019-12-05T06:12:54.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/34’ 2019-12-05T06:12:54.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/38’ 2019-12-05T06:12:54.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/28’ 2019-12-05T06:12:54.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/24’ 2019-12-05T06:12:54.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/29’ 2019-12-05T06:12:54.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/32’ 2019-12-05T06:12:54.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/22’ 2019-12-05T06:12:54.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/25’ 2019-12-05T06:12:54.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/30’ 2019-12-05T06:12:54.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/33’ 2019-12-05T06:12:54.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/23’ 2019-12-05T06:12:54.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/35’ 2019-12-05T06:12:54.530 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/26’ 2019-12-05T06:12:54.530 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./218/37’ 2019-12-05T06:12:54.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./218’ 2019-12-05T06:12:54.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/21’ 2019-12-05T06:12:54.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/27’ 2019-12-05T06:12:54.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/31’ 2019-12-05T06:12:54.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/36’ 2019-12-05T06:12:54.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/20’ 2019-12-05T06:12:54.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/39’ 2019-12-05T06:12:54.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/34’ 2019-12-05T06:12:54.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/38’ 2019-12-05T06:12:54.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/28’ 2019-12-05T06:12:54.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/24’ 2019-12-05T06:12:54.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/29’ 2019-12-05T06:12:54.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/32’ 2019-12-05T06:12:54.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/22’ 2019-12-05T06:12:54.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/25’ 2019-12-05T06:12:54.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/30’ 2019-12-05T06:12:54.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/33’ 2019-12-05T06:12:54.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/23’ 2019-12-05T06:12:54.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/35’ 2019-12-05T06:12:54.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/26’ 2019-12-05T06:12:54.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./219/37’ 2019-12-05T06:12:54.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./219’ 2019-12-05T06:12:54.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/21’ 2019-12-05T06:12:54.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/27’ 2019-12-05T06:12:54.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/31’ 2019-12-05T06:12:54.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/36’ 2019-12-05T06:12:54.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/20’ 2019-12-05T06:12:54.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/39’ 2019-12-05T06:12:54.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/34’ 2019-12-05T06:12:54.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/38’ 2019-12-05T06:12:54.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/28’ 2019-12-05T06:12:59.603 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:12:59.609 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:12:59.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/24’ 2019-12-05T06:12:59.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/29’ 2019-12-05T06:12:59.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/32’ 2019-12-05T06:12:59.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/22’ 2019-12-05T06:12:59.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/25’ 2019-12-05T06:12:59.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/30’ 2019-12-05T06:12:59.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/33’ 2019-12-05T06:12:59.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/23’ 2019-12-05T06:12:59.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/35’ 2019-12-05T06:12:59.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/26’ 2019-12-05T06:12:59.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./220/37’ 2019-12-05T06:12:59.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./220’ 2019-12-05T06:12:59.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/21’ 2019-12-05T06:12:59.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/27’ 2019-12-05T06:12:59.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/31’ 2019-12-05T06:12:59.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/36’ 2019-12-05T06:12:59.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/20’ 2019-12-05T06:12:59.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/39’ 2019-12-05T06:12:59.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/34’ 2019-12-05T06:12:59.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/38’ 2019-12-05T06:12:59.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/28’ 2019-12-05T06:12:59.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/24’ 2019-12-05T06:12:59.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/29’ 2019-12-05T06:12:59.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/32’ 2019-12-05T06:12:59.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/22’ 2019-12-05T06:12:59.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/25’ 2019-12-05T06:12:59.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/30’ 2019-12-05T06:12:59.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/33’ 2019-12-05T06:12:59.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/23’ 2019-12-05T06:12:59.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/35’ 2019-12-05T06:12:59.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/26’ 2019-12-05T06:12:59.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./221/37’ 2019-12-05T06:12:59.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./221’ 2019-12-05T06:12:59.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/21’ 2019-12-05T06:12:59.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/27’ 2019-12-05T06:12:59.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/31’ 2019-12-05T06:12:59.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/36’ 2019-12-05T06:12:59.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/20’ 2019-12-05T06:12:59.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/39’ 2019-12-05T06:12:59.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/34’ 2019-12-05T06:12:59.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/38’ 2019-12-05T06:12:59.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/28’ 2019-12-05T06:12:59.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/24’ 2019-12-05T06:12:59.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/29’ 2019-12-05T06:12:59.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/32’ 2019-12-05T06:12:59.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/22’ 2019-12-05T06:12:59.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/25’ 2019-12-05T06:12:59.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/30’ 2019-12-05T06:12:59.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/33’ 2019-12-05T06:12:59.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/23’ 2019-12-05T06:12:59.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/35’ 2019-12-05T06:12:59.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/26’ 2019-12-05T06:12:59.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./222/37’ 2019-12-05T06:12:59.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./222’ 2019-12-05T06:12:59.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/21’ 2019-12-05T06:12:59.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/27’ 2019-12-05T06:12:59.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/31’ 2019-12-05T06:12:59.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/36’ 2019-12-05T06:12:59.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/20’ 2019-12-05T06:12:59.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/39’ 2019-12-05T06:12:59.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/34’ 2019-12-05T06:12:59.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/38’ 2019-12-05T06:12:59.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/28’ 2019-12-05T06:12:59.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/24’ 2019-12-05T06:12:59.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/29’ 2019-12-05T06:12:59.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/32’ 2019-12-05T06:12:59.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/22’ 2019-12-05T06:12:59.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/25’ 2019-12-05T06:12:59.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/30’ 2019-12-05T06:12:59.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/33’ 2019-12-05T06:12:59.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/23’ 2019-12-05T06:12:59.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/35’ 2019-12-05T06:12:59.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/26’ 2019-12-05T06:12:59.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./223/37’ 2019-12-05T06:12:59.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./223’ 2019-12-05T06:12:59.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/21’ 2019-12-05T06:12:59.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/27’ 2019-12-05T06:12:59.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/31’ 2019-12-05T06:12:59.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/36’ 2019-12-05T06:12:59.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/20’ 2019-12-05T06:12:59.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/39’ 2019-12-05T06:12:59.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/34’ 2019-12-05T06:12:59.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/38’ 2019-12-05T06:12:59.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/28’ 2019-12-05T06:12:59.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/24’ 2019-12-05T06:12:59.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/29’ 2019-12-05T06:12:59.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/32’ 2019-12-05T06:12:59.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/22’ 2019-12-05T06:12:59.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/25’ 2019-12-05T06:12:59.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/30’ 2019-12-05T06:12:59.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/33’ 2019-12-05T06:12:59.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/23’ 2019-12-05T06:12:59.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/35’ 2019-12-05T06:12:59.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/26’ 2019-12-05T06:12:59.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./224/37’ 2019-12-05T06:12:59.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./224’ 2019-12-05T06:12:59.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/21’ 2019-12-05T06:12:59.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/27’ 2019-12-05T06:12:59.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/31’ 2019-12-05T06:12:59.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/36’ 2019-12-05T06:12:59.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/20’ 2019-12-05T06:12:59.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/39’ 2019-12-05T06:12:59.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/34’ 2019-12-05T06:12:59.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/38’ 2019-12-05T06:12:59.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/28’ 2019-12-05T06:12:59.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/24’ 2019-12-05T06:12:59.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/29’ 2019-12-05T06:12:59.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/32’ 2019-12-05T06:12:59.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/22’ 2019-12-05T06:12:59.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/25’ 2019-12-05T06:12:59.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/30’ 2019-12-05T06:12:59.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/33’ 2019-12-05T06:12:59.761 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/23’ 2019-12-05T06:12:59.761 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/35’ 2019-12-05T06:12:59.761 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/26’ 2019-12-05T06:12:59.761 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./225/37’ 2019-12-05T06:12:59.761 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./225’ 2019-12-05T06:12:59.761 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/21’ 2019-12-05T06:12:59.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/27’ 2019-12-05T06:12:59.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/31’ 2019-12-05T06:12:59.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/36’ 2019-12-05T06:12:59.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/20’ 2019-12-05T06:12:59.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/39’ 2019-12-05T06:12:59.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/34’ 2019-12-05T06:12:59.762 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/38’ 2019-12-05T06:12:59.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/28’ 2019-12-05T06:12:59.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/24’ 2019-12-05T06:12:59.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/29’ 2019-12-05T06:12:59.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/32’ 2019-12-05T06:12:59.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/22’ 2019-12-05T06:12:59.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/25’ 2019-12-05T06:12:59.763 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/30’ 2019-12-05T06:12:59.764 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/33’ 2019-12-05T06:12:59.764 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/23’ 2019-12-05T06:12:59.764 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/35’ 2019-12-05T06:12:59.764 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/26’ 2019-12-05T06:12:59.764 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./226/37’ 2019-12-05T06:12:59.764 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./226’ 2019-12-05T06:12:59.764 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/21’ 2019-12-05T06:12:59.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/27’ 2019-12-05T06:12:59.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/31’ 2019-12-05T06:12:59.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/36’ 2019-12-05T06:12:59.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/20’ 2019-12-05T06:12:59.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/39’ 2019-12-05T06:12:59.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/34’ 2019-12-05T06:12:59.765 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/38’ 2019-12-05T06:12:59.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/28’ 2019-12-05T06:12:59.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/24’ 2019-12-05T06:12:59.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/29’ 2019-12-05T06:12:59.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/32’ 2019-12-05T06:12:59.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/22’ 2019-12-05T06:12:59.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/25’ 2019-12-05T06:12:59.766 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/30’ 2019-12-05T06:12:59.767 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/33’ 2019-12-05T06:12:59.767 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/23’ 2019-12-05T06:12:59.767 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/35’ 2019-12-05T06:12:59.767 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/26’ 2019-12-05T06:12:59.767 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./227/37’ 2019-12-05T06:12:59.767 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./227’ 2019-12-05T06:12:59.768 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/21’ 2019-12-05T06:12:59.768 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/27’ 2019-12-05T06:12:59.768 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/31’ 2019-12-05T06:12:59.768 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/36’ 2019-12-05T06:12:59.768 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/20’ 2019-12-05T06:12:59.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/39’ 2019-12-05T06:12:59.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/34’ 2019-12-05T06:12:59.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/38’ 2019-12-05T06:12:59.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/28’ 2019-12-05T06:12:59.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/24’ 2019-12-05T06:12:59.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/29’ 2019-12-05T06:12:59.769 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/32’ 2019-12-05T06:12:59.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/22’ 2019-12-05T06:12:59.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/25’ 2019-12-05T06:12:59.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/30’ 2019-12-05T06:12:59.770 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/33’ 2019-12-05T06:13:03.853 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/23’ 2019-12-05T06:13:03.854 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/35’ 2019-12-05T06:13:03.854 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/26’ 2019-12-05T06:13:03.854 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./228/37’ 2019-12-05T06:13:03.854 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./228’ 2019-12-05T06:13:03.854 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/21’ 2019-12-05T06:13:03.854 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/27’ 2019-12-05T06:13:03.854 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/31’ 2019-12-05T06:13:03.855 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/36’ 2019-12-05T06:13:03.855 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/20’ 2019-12-05T06:13:03.855 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/39’ 2019-12-05T06:13:03.855 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/34’ 2019-12-05T06:13:03.855 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/38’ 2019-12-05T06:13:03.855 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/28’ 2019-12-05T06:13:03.856 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/24’ 2019-12-05T06:13:03.856 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/29’ 2019-12-05T06:13:03.856 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/32’ 2019-12-05T06:13:03.856 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/22’ 2019-12-05T06:13:03.856 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/25’ 2019-12-05T06:13:03.856 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/30’ 2019-12-05T06:13:03.856 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/33’ 2019-12-05T06:13:03.857 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/23’ 2019-12-05T06:13:03.857 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/35’ 2019-12-05T06:13:03.857 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/26’ 2019-12-05T06:13:03.857 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./229/37’ 2019-12-05T06:13:03.857 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./229’ 2019-12-05T06:13:03.857 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/21’ 2019-12-05T06:13:03.858 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/27’ 2019-12-05T06:13:03.859 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/31’ 2019-12-05T06:13:03.859 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/36’ 2019-12-05T06:13:03.859 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/20’ 2019-12-05T06:13:03.859 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/39’ 2019-12-05T06:13:03.859 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/34’ 2019-12-05T06:13:03.859 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/38’ 2019-12-05T06:13:03.860 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/28’ 2019-12-05T06:13:03.860 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/24’ 2019-12-05T06:13:03.860 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/29’ 2019-12-05T06:13:03.860 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/32’ 2019-12-05T06:13:03.860 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/22’ 2019-12-05T06:13:03.861 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/25’ 2019-12-05T06:13:03.861 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/30’ 2019-12-05T06:13:03.861 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/33’ 2019-12-05T06:13:03.861 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/23’ 2019-12-05T06:13:03.862 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/35’ 2019-12-05T06:13:03.862 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/26’ 2019-12-05T06:13:03.862 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./230/37’ 2019-12-05T06:13:03.862 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./230’ 2019-12-05T06:13:03.863 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/21’ 2019-12-05T06:13:03.863 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/27’ 2019-12-05T06:13:03.869 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/31’ 2019-12-05T06:13:03.869 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/36’ 2019-12-05T06:13:03.869 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/20’ 2019-12-05T06:13:03.869 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/39’ 2019-12-05T06:13:03.870 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/34’ 2019-12-05T06:13:03.870 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/38’ 2019-12-05T06:13:03.870 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/28’ 2019-12-05T06:13:03.870 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/24’ 2019-12-05T06:13:03.870 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/29’ 2019-12-05T06:13:03.870 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/32’ 2019-12-05T06:13:03.870 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/22’ 2019-12-05T06:13:03.871 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/25’ 2019-12-05T06:13:03.871 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/30’ 2019-12-05T06:13:03.871 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/33’ 2019-12-05T06:13:03.871 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/23’ 2019-12-05T06:13:03.871 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/35’ 2019-12-05T06:13:03.871 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/26’ 2019-12-05T06:13:03.871 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./231/37’ 2019-12-05T06:13:03.872 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./231’ 2019-12-05T06:13:03.872 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/21’ 2019-12-05T06:13:03.872 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/27’ 2019-12-05T06:13:03.872 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/31’ 2019-12-05T06:13:03.879 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/36’ 2019-12-05T06:13:03.879 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/20’ 2019-12-05T06:13:03.880 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/39’ 2019-12-05T06:13:03.880 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/34’ 2019-12-05T06:13:03.880 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/38’ 2019-12-05T06:13:03.880 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/28’ 2019-12-05T06:13:03.880 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/24’ 2019-12-05T06:13:03.880 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/29’ 2019-12-05T06:13:03.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/32’ 2019-12-05T06:13:03.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/22’ 2019-12-05T06:13:03.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/25’ 2019-12-05T06:13:03.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/30’ 2019-12-05T06:13:03.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/33’ 2019-12-05T06:13:03.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/23’ 2019-12-05T06:13:03.881 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/35’ 2019-12-05T06:13:03.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/26’ 2019-12-05T06:13:03.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./232/37’ 2019-12-05T06:13:03.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./232’ 2019-12-05T06:13:03.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/21’ 2019-12-05T06:13:03.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/27’ 2019-12-05T06:13:03.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/31’ 2019-12-05T06:13:03.882 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/36’ 2019-12-05T06:13:03.883 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/20’ 2019-12-05T06:13:03.884 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/39’ 2019-12-05T06:13:03.884 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/34’ 2019-12-05T06:13:03.884 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/38’ 2019-12-05T06:13:03.884 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/28’ 2019-12-05T06:13:03.884 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/24’ 2019-12-05T06:13:03.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/29’ 2019-12-05T06:13:03.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/32’ 2019-12-05T06:13:03.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/22’ 2019-12-05T06:13:03.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/25’ 2019-12-05T06:13:03.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/30’ 2019-12-05T06:13:03.885 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/33’ 2019-12-05T06:13:03.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/23’ 2019-12-05T06:13:03.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/35’ 2019-12-05T06:13:03.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/26’ 2019-12-05T06:13:03.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./233/37’ 2019-12-05T06:13:03.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./233’ 2019-12-05T06:13:03.886 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/21’ 2019-12-05T06:13:03.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/27’ 2019-12-05T06:13:03.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/31’ 2019-12-05T06:13:03.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/36’ 2019-12-05T06:13:03.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/20’ 2019-12-05T06:13:03.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/39’ 2019-12-05T06:13:03.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/34’ 2019-12-05T06:13:03.887 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/38’ 2019-12-05T06:13:03.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/28’ 2019-12-05T06:13:03.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/24’ 2019-12-05T06:13:03.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/29’ 2019-12-05T06:13:03.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/32’ 2019-12-05T06:13:03.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/22’ 2019-12-05T06:13:03.888 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/25’ 2019-12-05T06:13:03.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/30’ 2019-12-05T06:13:03.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/33’ 2019-12-05T06:13:03.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/23’ 2019-12-05T06:13:03.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/35’ 2019-12-05T06:13:03.889 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/26’ 2019-12-05T06:13:03.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./234/37’ 2019-12-05T06:13:03.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./234’ 2019-12-05T06:13:03.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/21’ 2019-12-05T06:13:03.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/27’ 2019-12-05T06:13:03.890 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/31’ 2019-12-05T06:13:03.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/36’ 2019-12-05T06:13:03.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/20’ 2019-12-05T06:13:03.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/39’ 2019-12-05T06:13:03.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/34’ 2019-12-05T06:13:03.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/38’ 2019-12-05T06:13:03.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/28’ 2019-12-05T06:13:03.891 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/24’ 2019-12-05T06:13:03.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/29’ 2019-12-05T06:13:03.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/32’ 2019-12-05T06:13:03.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/22’ 2019-12-05T06:13:03.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/25’ 2019-12-05T06:13:03.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/30’ 2019-12-05T06:13:03.892 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/33’ 2019-12-05T06:13:03.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/23’ 2019-12-05T06:13:03.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/35’ 2019-12-05T06:13:03.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/26’ 2019-12-05T06:13:03.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./235/37’ 2019-12-05T06:13:03.893 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./235’ 2019-12-05T06:13:03.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/21’ 2019-12-05T06:13:03.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/27’ 2019-12-05T06:13:03.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/31’ 2019-12-05T06:13:03.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/36’ 2019-12-05T06:13:03.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/20’ 2019-12-05T06:13:03.894 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/39’ 2019-12-05T06:13:03.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/34’ 2019-12-05T06:13:03.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/38’ 2019-12-05T06:13:03.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/28’ 2019-12-05T06:13:03.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/24’ 2019-12-05T06:13:03.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/29’ 2019-12-05T06:13:03.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/32’ 2019-12-05T06:13:03.895 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/22’ 2019-12-05T06:13:03.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/25’ 2019-12-05T06:13:03.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/30’ 2019-12-05T06:13:03.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/33’ 2019-12-05T06:13:03.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/23’ 2019-12-05T06:13:03.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/35’ 2019-12-05T06:13:03.896 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/26’ 2019-12-05T06:13:03.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./236/37’ 2019-12-05T06:13:03.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./236’ 2019-12-05T06:13:03.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/21’ 2019-12-05T06:13:03.897 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/27’ 2019-12-05T06:13:06.705 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/31’ 2019-12-05T06:13:06.705 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/36’ 2019-12-05T06:13:06.705 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/20’ 2019-12-05T06:13:06.705 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/39’ 2019-12-05T06:13:06.706 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/34’ 2019-12-05T06:13:06.706 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/38’ 2019-12-05T06:13:06.706 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/28’ 2019-12-05T06:13:06.706 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/24’ 2019-12-05T06:13:06.706 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/29’ 2019-12-05T06:13:06.706 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/32’ 2019-12-05T06:13:06.706 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/22’ 2019-12-05T06:13:06.707 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/25’ 2019-12-05T06:13:06.707 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/30’ 2019-12-05T06:13:06.707 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/33’ 2019-12-05T06:13:06.707 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/23’ 2019-12-05T06:13:06.707 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/35’ 2019-12-05T06:13:06.707 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/26’ 2019-12-05T06:13:06.707 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./237/37’ 2019-12-05T06:13:06.710 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./237’ 2019-12-05T06:13:06.710 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/21’ 2019-12-05T06:13:06.710 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/27’ 2019-12-05T06:13:06.710 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/31’ 2019-12-05T06:13:06.711 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/36’ 2019-12-05T06:13:06.711 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/20’ 2019-12-05T06:13:06.711 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/39’ 2019-12-05T06:13:06.711 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/34’ 2019-12-05T06:13:06.711 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/38’ 2019-12-05T06:13:06.712 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/28’ 2019-12-05T06:13:06.712 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/24’ 2019-12-05T06:13:06.712 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/29’ 2019-12-05T06:13:06.712 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/32’ 2019-12-05T06:13:06.712 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/22’ 2019-12-05T06:13:06.712 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/25’ 2019-12-05T06:13:06.713 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/30’ 2019-12-05T06:13:06.713 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/33’ 2019-12-05T06:13:06.713 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/23’ 2019-12-05T06:13:06.713 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/35’ 2019-12-05T06:13:06.713 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/26’ 2019-12-05T06:13:06.714 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./238/37’ 2019-12-05T06:13:06.714 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./238’ 2019-12-05T06:13:06.714 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/21’ 2019-12-05T06:13:06.714 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/27’ 2019-12-05T06:13:06.714 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/31’ 2019-12-05T06:13:06.714 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/36’ 2019-12-05T06:13:06.714 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/20’ 2019-12-05T06:13:06.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/39’ 2019-12-05T06:13:06.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/34’ 2019-12-05T06:13:06.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/38’ 2019-12-05T06:13:06.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/28’ 2019-12-05T06:13:06.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/24’ 2019-12-05T06:13:06.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/29’ 2019-12-05T06:13:06.715 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/32’ 2019-12-05T06:13:06.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/22’ 2019-12-05T06:13:06.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/25’ 2019-12-05T06:13:06.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/30’ 2019-12-05T06:13:06.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/33’ 2019-12-05T06:13:06.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/23’ 2019-12-05T06:13:06.716 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/35’ 2019-12-05T06:13:06.717 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/26’ 2019-12-05T06:13:06.717 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./239/37’ 2019-12-05T06:13:06.717 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./239’ 2019-12-05T06:13:06.717 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/21’ 2019-12-05T06:13:06.717 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/27’ 2019-12-05T06:13:06.717 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/31’ 2019-12-05T06:13:06.718 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/36’ 2019-12-05T06:13:06.718 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/20’ 2019-12-05T06:13:06.719 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/39’ 2019-12-05T06:13:06.719 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/34’ 2019-12-05T06:13:06.719 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/38’ 2019-12-05T06:13:06.719 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/28’ 2019-12-05T06:13:06.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/24’ 2019-12-05T06:13:06.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/29’ 2019-12-05T06:13:06.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/32’ 2019-12-05T06:13:06.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/22’ 2019-12-05T06:13:06.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/25’ 2019-12-05T06:13:06.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/30’ 2019-12-05T06:13:06.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/33’ 2019-12-05T06:13:06.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/23’ 2019-12-05T06:13:06.721 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/35’ 2019-12-05T06:13:06.722 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/26’ 2019-12-05T06:13:06.722 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./240/37’ 2019-12-05T06:13:06.722 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./240’ 2019-12-05T06:13:06.722 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/21’ 2019-12-05T06:13:06.723 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/27’ 2019-12-05T06:13:06.723 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/31’ 2019-12-05T06:13:06.723 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/36’ 2019-12-05T06:13:06.724 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/20’ 2019-12-05T06:13:06.724 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/39’ 2019-12-05T06:13:06.724 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/34’ 2019-12-05T06:13:06.724 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/38’ 2019-12-05T06:13:06.725 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/28’ 2019-12-05T06:13:06.725 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/24’ 2019-12-05T06:13:06.725 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/29’ 2019-12-05T06:13:06.725 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/32’ 2019-12-05T06:13:06.726 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/22’ 2019-12-05T06:13:06.726 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/25’ 2019-12-05T06:13:06.726 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/30’ 2019-12-05T06:13:06.726 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/33’ 2019-12-05T06:13:06.727 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/23’ 2019-12-05T06:13:06.727 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/35’ 2019-12-05T06:13:06.727 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/26’ 2019-12-05T06:13:06.727 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./241/37’ 2019-12-05T06:13:06.728 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./241’ 2019-12-05T06:13:06.728 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/21’ 2019-12-05T06:13:06.728 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/27’ 2019-12-05T06:13:06.728 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/31’ 2019-12-05T06:13:06.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/36’ 2019-12-05T06:13:06.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/20’ 2019-12-05T06:13:06.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/39’ 2019-12-05T06:13:06.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/34’ 2019-12-05T06:13:06.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/38’ 2019-12-05T06:13:06.730 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/28’ 2019-12-05T06:13:06.730 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/24’ 2019-12-05T06:13:06.730 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/29’ 2019-12-05T06:13:06.730 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/32’ 2019-12-05T06:13:06.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/22’ 2019-12-05T06:13:06.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/25’ 2019-12-05T06:13:06.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/30’ 2019-12-05T06:13:06.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/33’ 2019-12-05T06:13:06.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/23’ 2019-12-05T06:13:06.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/35’ 2019-12-05T06:13:06.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/26’ 2019-12-05T06:13:06.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./242/37’ 2019-12-05T06:13:06.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./242’ 2019-12-05T06:13:06.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/21’ 2019-12-05T06:13:06.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/27’ 2019-12-05T06:13:06.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/31’ 2019-12-05T06:13:06.733 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/36’ 2019-12-05T06:13:06.734 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/20’ 2019-12-05T06:13:06.734 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/39’ 2019-12-05T06:13:06.734 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/34’ 2019-12-05T06:13:06.734 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/38’ 2019-12-05T06:13:06.735 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/28’ 2019-12-05T06:13:06.735 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/24’ 2019-12-05T06:13:06.735 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/29’ 2019-12-05T06:13:06.735 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/32’ 2019-12-05T06:13:06.735 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/22’ 2019-12-05T06:13:06.736 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/25’ 2019-12-05T06:13:06.736 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/30’ 2019-12-05T06:13:06.736 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/33’ 2019-12-05T06:13:06.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/23’ 2019-12-05T06:13:06.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/35’ 2019-12-05T06:13:06.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/26’ 2019-12-05T06:13:06.737 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./243/37’ 2019-12-05T06:13:06.738 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./243’ 2019-12-05T06:13:06.738 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/21’ 2019-12-05T06:13:06.738 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/27’ 2019-12-05T06:13:06.738 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/31’ 2019-12-05T06:13:06.739 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/36’ 2019-12-05T06:13:06.739 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/20’ 2019-12-05T06:13:06.739 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/39’ 2019-12-05T06:13:06.739 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/34’ 2019-12-05T06:13:06.739 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/38’ 2019-12-05T06:13:06.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/28’ 2019-12-05T06:13:06.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/24’ 2019-12-05T06:13:06.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/29’ 2019-12-05T06:13:06.740 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/32’ 2019-12-05T06:13:06.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/22’ 2019-12-05T06:13:06.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/25’ 2019-12-05T06:13:06.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/30’ 2019-12-05T06:13:06.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/33’ 2019-12-05T06:13:06.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/23’ 2019-12-05T06:13:06.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/35’ 2019-12-05T06:13:06.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/26’ 2019-12-05T06:13:06.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./244/37’ 2019-12-05T06:13:06.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./244’ 2019-12-05T06:13:06.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/21’ 2019-12-05T06:13:06.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/27’ 2019-12-05T06:13:06.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/31’ 2019-12-05T06:13:06.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/36’ 2019-12-05T06:13:06.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/20’ 2019-12-05T06:13:06.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/39’ 2019-12-05T06:13:06.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/34’ 2019-12-05T06:13:06.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/38’ 2019-12-05T06:13:06.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/28’ 2019-12-05T06:13:06.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/24’ 2019-12-05T06:13:09.123 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/29’ 2019-12-05T06:13:09.124 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/32’ 2019-12-05T06:13:09.124 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/22’ 2019-12-05T06:13:09.124 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/25’ 2019-12-05T06:13:09.124 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/30’ 2019-12-05T06:13:09.124 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/33’ 2019-12-05T06:13:09.124 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/23’ 2019-12-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/35’ 2019-12-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/26’ 2019-12-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./245/37’ 2019-12-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./245’ 2019-12-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/21’ 2019-12-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/27’ 2019-12-05T06:13:09.125 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/31’ 2019-12-05T06:13:09.126 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/36’ 2019-12-05T06:13:09.126 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/20’ 2019-12-05T06:13:09.126 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/39’ 2019-12-05T06:13:09.126 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/34’ 2019-12-05T06:13:09.127 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/38’ 2019-12-05T06:13:09.127 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/28’ 2019-12-05T06:13:09.127 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/24’ 2019-12-05T06:13:09.127 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/29’ 2019-12-05T06:13:09.127 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/32’ 2019-12-05T06:13:09.127 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/22’ 2019-12-05T06:13:09.128 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/25’ 2019-12-05T06:13:09.128 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/30’ 2019-12-05T06:13:09.128 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/33’ 2019-12-05T06:13:09.128 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/23’ 2019-12-05T06:13:09.128 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/35’ 2019-12-05T06:13:09.128 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/26’ 2019-12-05T06:13:09.129 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./246/37’ 2019-12-05T06:13:09.129 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./246’ 2019-12-05T06:13:09.129 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/21’ 2019-12-05T06:13:09.129 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/27’ 2019-12-05T06:13:09.129 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/31’ 2019-12-05T06:13:09.129 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/36’ 2019-12-05T06:13:09.129 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/20’ 2019-12-05T06:13:09.130 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/39’ 2019-12-05T06:13:09.130 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/34’ 2019-12-05T06:13:09.130 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/38’ 2019-12-05T06:13:09.130 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/28’ 2019-12-05T06:13:09.131 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/24’ 2019-12-05T06:13:09.131 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/29’ 2019-12-05T06:13:09.131 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/32’ 2019-12-05T06:13:09.131 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/22’ 2019-12-05T06:13:09.132 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/25’ 2019-12-05T06:13:09.132 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/30’ 2019-12-05T06:13:09.132 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/33’ 2019-12-05T06:13:09.132 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/23’ 2019-12-05T06:13:09.132 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/35’ 2019-12-05T06:13:09.132 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/26’ 2019-12-05T06:13:09.132 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./247/37’ 2019-12-05T06:13:09.133 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./247’ 2019-12-05T06:13:09.133 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/21’ 2019-12-05T06:13:09.133 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/27’ 2019-12-05T06:13:09.133 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/31’ 2019-12-05T06:13:09.133 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/36’ 2019-12-05T06:13:09.134 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/20’ 2019-12-05T06:13:09.134 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/39’ 2019-12-05T06:13:09.134 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/34’ 2019-12-05T06:13:09.134 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/38’ 2019-12-05T06:13:09.134 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/28’ 2019-12-05T06:13:09.134 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/24’ 2019-12-05T06:13:09.134 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/29’ 2019-12-05T06:13:09.135 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/32’ 2019-12-05T06:13:09.135 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/22’ 2019-12-05T06:13:09.135 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/25’ 2019-12-05T06:13:09.135 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/30’ 2019-12-05T06:13:09.135 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/33’ 2019-12-05T06:13:09.135 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/23’ 2019-12-05T06:13:09.135 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/35’ 2019-12-05T06:13:09.136 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/26’ 2019-12-05T06:13:09.136 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./248/37’ 2019-12-05T06:13:09.136 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./248’ 2019-12-05T06:13:09.136 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/21’ 2019-12-05T06:13:09.136 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/27’ 2019-12-05T06:13:09.136 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/31’ 2019-12-05T06:13:09.136 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/36’ 2019-12-05T06:13:09.137 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/20’ 2019-12-05T06:13:09.137 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/39’ 2019-12-05T06:13:09.137 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/34’ 2019-12-05T06:13:09.137 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/38’ 2019-12-05T06:13:09.137 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/28’ 2019-12-05T06:13:09.138 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/24’ 2019-12-05T06:13:09.138 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/29’ 2019-12-05T06:13:09.138 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/32’ 2019-12-05T06:13:09.138 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/22’ 2019-12-05T06:13:09.138 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/25’ 2019-12-05T06:13:09.139 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/30’ 2019-12-05T06:13:09.139 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/33’ 2019-12-05T06:13:09.139 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/23’ 2019-12-05T06:13:09.139 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/35’ 2019-12-05T06:13:09.139 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/26’ 2019-12-05T06:13:09.139 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./249/37’ 2019-12-05T06:13:09.139 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./249’ 2019-12-05T06:13:09.140 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/21’ 2019-12-05T06:13:09.140 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/27’ 2019-12-05T06:13:09.140 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/31’ 2019-12-05T06:13:09.140 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/36’ 2019-12-05T06:13:09.140 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/20’ 2019-12-05T06:13:09.141 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/39’ 2019-12-05T06:13:09.141 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/34’ 2019-12-05T06:13:09.141 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/38’ 2019-12-05T06:13:09.142 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/28’ 2019-12-05T06:13:09.142 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/24’ 2019-12-05T06:13:09.142 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/29’ 2019-12-05T06:13:09.142 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/32’ 2019-12-05T06:13:09.142 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/22’ 2019-12-05T06:13:09.143 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/25’ 2019-12-05T06:13:09.143 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/30’ 2019-12-05T06:13:09.143 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/33’ 2019-12-05T06:13:09.143 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/23’ 2019-12-05T06:13:09.143 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/35’ 2019-12-05T06:13:09.143 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/26’ 2019-12-05T06:13:09.144 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./250/37’ 2019-12-05T06:13:09.144 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./250’ 2019-12-05T06:13:09.144 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/21’ 2019-12-05T06:13:09.144 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/27’ 2019-12-05T06:13:09.144 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/31’ 2019-12-05T06:13:09.144 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/36’ 2019-12-05T06:13:09.145 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/20’ 2019-12-05T06:13:09.145 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/39’ 2019-12-05T06:13:09.145 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/34’ 2019-12-05T06:13:09.145 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/38’ 2019-12-05T06:13:09.145 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/28’ 2019-12-05T06:13:09.145 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/24’ 2019-12-05T06:13:09.146 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/29’ 2019-12-05T06:13:09.146 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/32’ 2019-12-05T06:13:09.146 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/22’ 2019-12-05T06:13:09.146 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/25’ 2019-12-05T06:13:09.146 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/30’ 2019-12-05T06:13:09.146 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/33’ 2019-12-05T06:13:09.146 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/23’ 2019-12-05T06:13:09.147 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/35’ 2019-12-05T06:13:09.147 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/26’ 2019-12-05T06:13:09.147 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./251/37’ 2019-12-05T06:13:09.147 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./251’ 2019-12-05T06:13:09.147 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/21’ 2019-12-05T06:13:09.147 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/27’ 2019-12-05T06:13:09.148 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/31’ 2019-12-05T06:13:09.148 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/36’ 2019-12-05T06:13:09.148 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/20’ 2019-12-05T06:13:09.148 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/39’ 2019-12-05T06:13:09.148 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/34’ 2019-12-05T06:13:09.148 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/38’ 2019-12-05T06:13:09.148 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/28’ 2019-12-05T06:13:09.149 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/24’ 2019-12-05T06:13:09.149 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/29’ 2019-12-05T06:13:09.149 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/32’ 2019-12-05T06:13:09.149 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/22’ 2019-12-05T06:13:09.149 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/25’ 2019-12-05T06:13:09.149 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/30’ 2019-12-05T06:13:09.150 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/33’ 2019-12-05T06:13:09.150 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/23’ 2019-12-05T06:13:09.150 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/35’ 2019-12-05T06:13:09.150 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/26’ 2019-12-05T06:13:09.150 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./252/37’ 2019-12-05T06:13:09.150 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./252’ 2019-12-05T06:13:09.151 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/21’ 2019-12-05T06:13:09.151 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/27’ 2019-12-05T06:13:09.151 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/31’ 2019-12-05T06:13:09.151 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/36’ 2019-12-05T06:13:09.151 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/20’ 2019-12-05T06:13:09.151 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/39’ 2019-12-05T06:13:09.152 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/34’ 2019-12-05T06:13:09.152 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/38’ 2019-12-05T06:13:09.152 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/28’ 2019-12-05T06:13:09.152 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/24’ 2019-12-05T06:13:09.152 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/29’ 2019-12-05T06:13:09.153 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/32’ 2019-12-05T06:13:09.153 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/22’ 2019-12-05T06:13:09.153 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/25’ 2019-12-05T06:13:09.153 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/30’ 2019-12-05T06:13:09.153 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/33’ 2019-12-05T06:13:09.154 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/23’ 2019-12-05T06:13:10.794 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/35’ 2019-12-05T06:13:10.794 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/26’ 2019-12-05T06:13:10.794 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./253/37’ 2019-12-05T06:13:10.794 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./253’ 2019-12-05T06:13:10.794 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/21’ 2019-12-05T06:13:10.795 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/27’ 2019-12-05T06:13:10.795 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/31’ 2019-12-05T06:13:10.795 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/36’ 2019-12-05T06:13:10.795 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/20’ 2019-12-05T06:13:10.795 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/39’ 2019-12-05T06:13:10.796 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/34’ 2019-12-05T06:13:10.796 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/38’ 2019-12-05T06:13:10.796 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/28’ 2019-12-05T06:13:10.797 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/24’ 2019-12-05T06:13:10.797 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/29’ 2019-12-05T06:13:10.797 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/32’ 2019-12-05T06:13:10.797 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/22’ 2019-12-05T06:13:10.798 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/25’ 2019-12-05T06:13:10.798 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/30’ 2019-12-05T06:13:10.798 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/33’ 2019-12-05T06:13:10.798 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/23’ 2019-12-05T06:13:10.798 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/35’ 2019-12-05T06:13:10.799 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/26’ 2019-12-05T06:13:10.799 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./254/37’ 2019-12-05T06:13:10.799 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./254’ 2019-12-05T06:13:10.799 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/21’ 2019-12-05T06:13:10.799 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/27’ 2019-12-05T06:13:10.799 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/31’ 2019-12-05T06:13:10.800 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/36’ 2019-12-05T06:13:10.800 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/20’ 2019-12-05T06:13:10.800 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/39’ 2019-12-05T06:13:10.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/34’ 2019-12-05T06:13:10.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/38’ 2019-12-05T06:13:10.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/28’ 2019-12-05T06:13:10.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/24’ 2019-12-05T06:13:10.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/29’ 2019-12-05T06:13:10.801 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/32’ 2019-12-05T06:13:10.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/22’ 2019-12-05T06:13:10.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/25’ 2019-12-05T06:13:10.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/30’ 2019-12-05T06:13:10.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/33’ 2019-12-05T06:13:10.802 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/23’ 2019-12-05T06:13:10.803 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/35’ 2019-12-05T06:13:10.803 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/26’ 2019-12-05T06:13:10.803 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./255/37’ 2019-12-05T06:13:10.803 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./255’ 2019-12-05T06:13:10.803 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/21’ 2019-12-05T06:13:10.803 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/27’ 2019-12-05T06:13:10.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/31’ 2019-12-05T06:13:10.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/36’ 2019-12-05T06:13:10.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/20’ 2019-12-05T06:13:10.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/39’ 2019-12-05T06:13:10.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/34’ 2019-12-05T06:13:10.804 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/38’ 2019-12-05T06:13:10.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/28’ 2019-12-05T06:13:10.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/24’ 2019-12-05T06:13:10.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/29’ 2019-12-05T06:13:10.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/32’ 2019-12-05T06:13:10.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/22’ 2019-12-05T06:13:10.805 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/25’ 2019-12-05T06:13:10.806 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/30’ 2019-12-05T06:13:10.806 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/33’ 2019-12-05T06:13:10.806 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/23’ 2019-12-05T06:13:10.806 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/35’ 2019-12-05T06:13:10.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/26’ 2019-12-05T06:13:10.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./256/37’ 2019-12-05T06:13:10.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./256’ 2019-12-05T06:13:10.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/21’ 2019-12-05T06:13:10.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/27’ 2019-12-05T06:13:10.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/31’ 2019-12-05T06:13:10.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/36’ 2019-12-05T06:13:10.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/20’ 2019-12-05T06:13:10.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/39’ 2019-12-05T06:13:10.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/34’ 2019-12-05T06:13:10.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/38’ 2019-12-05T06:13:10.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/28’ 2019-12-05T06:13:10.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/24’ 2019-12-05T06:13:10.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/29’ 2019-12-05T06:13:10.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/32’ 2019-12-05T06:13:10.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/22’ 2019-12-05T06:13:10.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/25’ 2019-12-05T06:13:10.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/30’ 2019-12-05T06:13:10.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/33’ 2019-12-05T06:13:10.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/23’ 2019-12-05T06:13:10.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/35’ 2019-12-05T06:13:10.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/26’ 2019-12-05T06:13:10.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./257/37’ 2019-12-05T06:13:10.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./257’ 2019-12-05T06:13:10.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/21’ 2019-12-05T06:13:10.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/27’ 2019-12-05T06:13:10.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/31’ 2019-12-05T06:13:10.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/36’ 2019-12-05T06:13:10.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/20’ 2019-12-05T06:13:10.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/39’ 2019-12-05T06:13:10.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/34’ 2019-12-05T06:13:10.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/38’ 2019-12-05T06:13:10.823 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/28’ 2019-12-05T06:13:10.824 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/24’ 2019-12-05T06:13:10.824 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/29’ 2019-12-05T06:13:10.824 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/32’ 2019-12-05T06:13:10.824 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/22’ 2019-12-05T06:13:10.824 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/25’ 2019-12-05T06:13:10.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/30’ 2019-12-05T06:13:10.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/33’ 2019-12-05T06:13:10.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/23’ 2019-12-05T06:13:10.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/35’ 2019-12-05T06:13:10.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/26’ 2019-12-05T06:13:10.825 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./258/37’ 2019-12-05T06:13:10.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./258’ 2019-12-05T06:13:10.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/21’ 2019-12-05T06:13:10.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/27’ 2019-12-05T06:13:10.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/31’ 2019-12-05T06:13:10.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/36’ 2019-12-05T06:13:10.826 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/20’ 2019-12-05T06:13:10.827 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/39’ 2019-12-05T06:13:10.827 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/34’ 2019-12-05T06:13:10.827 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/38’ 2019-12-05T06:13:10.827 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/28’ 2019-12-05T06:13:10.827 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/24’ 2019-12-05T06:13:10.827 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/29’ 2019-12-05T06:13:10.838 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/32’ 2019-12-05T06:13:10.838 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/22’ 2019-12-05T06:13:10.839 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/25’ 2019-12-05T06:13:10.839 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/30’ 2019-12-05T06:13:10.839 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/33’ 2019-12-05T06:13:10.840 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/23’ 2019-12-05T06:13:10.840 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/35’ 2019-12-05T06:13:10.840 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/26’ 2019-12-05T06:13:10.840 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./259/37’ 2019-12-05T06:13:10.840 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./259’ 2019-12-05T06:13:10.840 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/21’ 2019-12-05T06:13:10.841 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/27’ 2019-12-05T06:13:10.841 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/31’ 2019-12-05T06:13:10.841 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/36’ 2019-12-05T06:13:10.841 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/20’ 2019-12-05T06:13:10.841 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/39’ 2019-12-05T06:13:10.841 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/34’ 2019-12-05T06:13:10.841 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/38’ 2019-12-05T06:13:10.842 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/28’ 2019-12-05T06:13:10.842 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/24’ 2019-12-05T06:13:10.842 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/29’ 2019-12-05T06:13:10.842 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/32’ 2019-12-05T06:13:10.842 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/22’ 2019-12-05T06:13:10.842 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/25’ 2019-12-05T06:13:10.842 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/30’ 2019-12-05T06:13:10.843 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/33’ 2019-12-05T06:13:10.843 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/23’ 2019-12-05T06:13:10.843 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/35’ 2019-12-05T06:13:10.843 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/26’ 2019-12-05T06:13:10.843 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./260/37’ 2019-12-05T06:13:10.843 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./260’ 2019-12-05T06:13:10.844 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/21’ 2019-12-05T06:13:10.844 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/27’ 2019-12-05T06:13:10.844 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/31’ 2019-12-05T06:13:10.845 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/36’ 2019-12-05T06:13:10.845 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/20’ 2019-12-05T06:13:10.845 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/39’ 2019-12-05T06:13:10.845 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/34’ 2019-12-05T06:13:10.845 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/38’ 2019-12-05T06:13:10.846 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/28’ 2019-12-05T06:13:10.846 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/24’ 2019-12-05T06:13:10.846 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/29’ 2019-12-05T06:13:10.846 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/32’ 2019-12-05T06:13:10.846 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/22’ 2019-12-05T06:13:10.847 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/25’ 2019-12-05T06:13:10.847 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/30’ 2019-12-05T06:13:10.847 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/33’ 2019-12-05T06:13:10.847 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/23’ 2019-12-05T06:13:10.847 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/35’ 2019-12-05T06:13:10.848 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/26’ 2019-12-05T06:13:10.848 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./261/37’ 2019-12-05T06:13:10.848 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./261’ 2019-12-05T06:13:10.848 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/21’ 2019-12-05T06:13:10.848 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/27’ 2019-12-05T06:13:10.849 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/31’ 2019-12-05T06:13:13.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/36’ 2019-12-05T06:13:13.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/20’ 2019-12-05T06:13:13.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/39’ 2019-12-05T06:13:13.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/34’ 2019-12-05T06:13:13.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/38’ 2019-12-05T06:13:13.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/28’ 2019-12-05T06:13:13.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/24’ 2019-12-05T06:13:13.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/29’ 2019-12-05T06:13:13.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/32’ 2019-12-05T06:13:13.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/22’ 2019-12-05T06:13:13.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/25’ 2019-12-05T06:13:13.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/30’ 2019-12-05T06:13:13.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/33’ 2019-12-05T06:13:13.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/23’ 2019-12-05T06:13:13.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/35’ 2019-12-05T06:13:13.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/26’ 2019-12-05T06:13:13.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./262/37’ 2019-12-05T06:13:13.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./262’ 2019-12-05T06:13:13.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/21’ 2019-12-05T06:13:13.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/27’ 2019-12-05T06:13:13.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/31’ 2019-12-05T06:13:13.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/36’ 2019-12-05T06:13:13.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/20’ 2019-12-05T06:13:13.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/39’ 2019-12-05T06:13:13.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/34’ 2019-12-05T06:13:13.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/38’ 2019-12-05T06:13:13.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/28’ 2019-12-05T06:13:13.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/24’ 2019-12-05T06:13:13.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/29’ 2019-12-05T06:13:13.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/32’ 2019-12-05T06:13:13.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/22’ 2019-12-05T06:13:13.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/25’ 2019-12-05T06:13:13.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/30’ 2019-12-05T06:13:13.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/33’ 2019-12-05T06:13:13.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/23’ 2019-12-05T06:13:13.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/35’ 2019-12-05T06:13:13.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/26’ 2019-12-05T06:13:13.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./263/37’ 2019-12-05T06:13:13.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./263’ 2019-12-05T06:13:13.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/21’ 2019-12-05T06:13:13.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/27’ 2019-12-05T06:13:13.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/31’ 2019-12-05T06:13:13.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/36’ 2019-12-05T06:13:13.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/20’ 2019-12-05T06:13:13.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/39’ 2019-12-05T06:13:13.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/34’ 2019-12-05T06:13:13.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/38’ 2019-12-05T06:13:13.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/28’ 2019-12-05T06:13:13.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/24’ 2019-12-05T06:13:13.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/29’ 2019-12-05T06:13:13.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/32’ 2019-12-05T06:13:13.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/22’ 2019-12-05T06:13:13.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/25’ 2019-12-05T06:13:13.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/30’ 2019-12-05T06:13:13.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/33’ 2019-12-05T06:13:13.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/23’ 2019-12-05T06:13:13.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/35’ 2019-12-05T06:13:13.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/26’ 2019-12-05T06:13:13.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./264/37’ 2019-12-05T06:13:13.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./264’ 2019-12-05T06:13:13.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/21’ 2019-12-05T06:13:13.518 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/27’ 2019-12-05T06:13:13.518 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/31’ 2019-12-05T06:13:13.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/36’ 2019-12-05T06:13:13.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/20’ 2019-12-05T06:13:13.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/39’ 2019-12-05T06:13:13.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/34’ 2019-12-05T06:13:13.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/38’ 2019-12-05T06:13:13.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/28’ 2019-12-05T06:13:13.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/24’ 2019-12-05T06:13:13.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/29’ 2019-12-05T06:13:13.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/32’ 2019-12-05T06:13:13.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/22’ 2019-12-05T06:13:13.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/25’ 2019-12-05T06:13:13.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/30’ 2019-12-05T06:13:13.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/33’ 2019-12-05T06:13:13.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/23’ 2019-12-05T06:13:13.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/35’ 2019-12-05T06:13:13.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/26’ 2019-12-05T06:13:13.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./265/37’ 2019-12-05T06:13:13.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./265’ 2019-12-05T06:13:13.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/21’ 2019-12-05T06:13:13.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/27’ 2019-12-05T06:13:13.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/31’ 2019-12-05T06:13:13.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/36’ 2019-12-05T06:13:13.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/20’ 2019-12-05T06:13:13.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/39’ 2019-12-05T06:13:13.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/34’ 2019-12-05T06:13:13.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/38’ 2019-12-05T06:13:13.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/28’ 2019-12-05T06:13:13.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/24’ 2019-12-05T06:13:13.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/29’ 2019-12-05T06:13:13.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/32’ 2019-12-05T06:13:13.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/22’ 2019-12-05T06:13:13.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/25’ 2019-12-05T06:13:13.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/30’ 2019-12-05T06:13:13.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/33’ 2019-12-05T06:13:13.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/23’ 2019-12-05T06:13:13.528 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/35’ 2019-12-05T06:13:13.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/26’ 2019-12-05T06:13:13.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./266/37’ 2019-12-05T06:13:13.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./266’ 2019-12-05T06:13:13.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/21’ 2019-12-05T06:13:13.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/27’ 2019-12-05T06:13:13.529 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/31’ 2019-12-05T06:13:13.530 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/36’ 2019-12-05T06:13:13.530 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/20’ 2019-12-05T06:13:13.530 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/39’ 2019-12-05T06:13:13.530 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/34’ 2019-12-05T06:13:13.530 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/38’ 2019-12-05T06:13:13.530 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/28’ 2019-12-05T06:13:13.530 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/24’ 2019-12-05T06:13:13.531 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/29’ 2019-12-05T06:13:13.531 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/32’ 2019-12-05T06:13:13.531 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/22’ 2019-12-05T06:13:13.531 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/25’ 2019-12-05T06:13:13.531 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/30’ 2019-12-05T06:13:13.531 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/33’ 2019-12-05T06:13:13.531 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/23’ 2019-12-05T06:13:13.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/35’ 2019-12-05T06:13:13.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/26’ 2019-12-05T06:13:13.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./267/37’ 2019-12-05T06:13:13.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./267’ 2019-12-05T06:13:13.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/21’ 2019-12-05T06:13:13.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/27’ 2019-12-05T06:13:13.532 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/31’ 2019-12-05T06:13:13.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/36’ 2019-12-05T06:13:13.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/20’ 2019-12-05T06:13:13.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/39’ 2019-12-05T06:13:13.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/34’ 2019-12-05T06:13:13.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/38’ 2019-12-05T06:13:13.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/28’ 2019-12-05T06:13:13.533 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/24’ 2019-12-05T06:13:13.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/29’ 2019-12-05T06:13:13.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/32’ 2019-12-05T06:13:13.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/22’ 2019-12-05T06:13:13.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/25’ 2019-12-05T06:13:13.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/30’ 2019-12-05T06:13:13.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/33’ 2019-12-05T06:13:13.534 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/23’ 2019-12-05T06:13:13.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/35’ 2019-12-05T06:13:13.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/26’ 2019-12-05T06:13:13.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./268/37’ 2019-12-05T06:13:13.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./268’ 2019-12-05T06:13:13.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/21’ 2019-12-05T06:13:13.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/27’ 2019-12-05T06:13:13.535 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/31’ 2019-12-05T06:13:13.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/36’ 2019-12-05T06:13:13.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/20’ 2019-12-05T06:13:13.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/39’ 2019-12-05T06:13:13.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/34’ 2019-12-05T06:13:13.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/38’ 2019-12-05T06:13:13.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/28’ 2019-12-05T06:13:13.536 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/24’ 2019-12-05T06:13:13.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/29’ 2019-12-05T06:13:13.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/32’ 2019-12-05T06:13:13.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/22’ 2019-12-05T06:13:13.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/25’ 2019-12-05T06:13:13.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/30’ 2019-12-05T06:13:13.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/33’ 2019-12-05T06:13:13.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/23’ 2019-12-05T06:13:13.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/35’ 2019-12-05T06:13:13.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/26’ 2019-12-05T06:13:13.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./269/37’ 2019-12-05T06:13:13.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./269’ 2019-12-05T06:13:13.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/21’ 2019-12-05T06:13:13.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/27’ 2019-12-05T06:13:13.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/31’ 2019-12-05T06:13:13.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/36’ 2019-12-05T06:13:13.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/20’ 2019-12-05T06:13:13.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/39’ 2019-12-05T06:13:13.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/34’ 2019-12-05T06:13:13.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/38’ 2019-12-05T06:13:13.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/28’ 2019-12-05T06:13:13.540 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/24’ 2019-12-05T06:13:17.173 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/29’ 2019-12-05T06:13:17.173 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/32’ 2019-12-05T06:13:17.173 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/22’ 2019-12-05T06:13:17.173 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/25’ 2019-12-05T06:13:17.173 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/30’ 2019-12-05T06:13:17.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/33’ 2019-12-05T06:13:17.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/23’ 2019-12-05T06:13:17.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/35’ 2019-12-05T06:13:17.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/26’ 2019-12-05T06:13:17.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./270/37’ 2019-12-05T06:13:17.174 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./270’ 2019-12-05T06:13:17.175 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/21’ 2019-12-05T06:13:17.175 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/27’ 2019-12-05T06:13:17.175 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/31’ 2019-12-05T06:13:17.175 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/36’ 2019-12-05T06:13:17.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/20’ 2019-12-05T06:13:17.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/39’ 2019-12-05T06:13:17.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/34’ 2019-12-05T06:13:17.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/38’ 2019-12-05T06:13:17.176 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/28’ 2019-12-05T06:13:17.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/24’ 2019-12-05T06:13:17.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/29’ 2019-12-05T06:13:17.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/32’ 2019-12-05T06:13:17.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/22’ 2019-12-05T06:13:17.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/25’ 2019-12-05T06:13:17.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/30’ 2019-12-05T06:13:17.177 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/33’ 2019-12-05T06:13:17.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/23’ 2019-12-05T06:13:17.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/35’ 2019-12-05T06:13:17.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/26’ 2019-12-05T06:13:17.178 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./271/37’ 2019-12-05T06:13:17.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./271’ 2019-12-05T06:13:17.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/21’ 2019-12-05T06:13:17.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/27’ 2019-12-05T06:13:17.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/31’ 2019-12-05T06:13:17.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/36’ 2019-12-05T06:13:17.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/20’ 2019-12-05T06:13:17.179 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/39’ 2019-12-05T06:13:17.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/34’ 2019-12-05T06:13:17.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/38’ 2019-12-05T06:13:17.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/28’ 2019-12-05T06:13:17.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/24’ 2019-12-05T06:13:17.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/29’ 2019-12-05T06:13:17.180 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/32’ 2019-12-05T06:13:17.181 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/22’ 2019-12-05T06:13:17.181 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/25’ 2019-12-05T06:13:17.181 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/30’ 2019-12-05T06:13:17.181 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/33’ 2019-12-05T06:13:17.181 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/23’ 2019-12-05T06:13:17.181 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/35’ 2019-12-05T06:13:17.182 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/26’ 2019-12-05T06:13:17.182 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./272/37’ 2019-12-05T06:13:17.182 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./272’ 2019-12-05T06:13:17.182 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/21’ 2019-12-05T06:13:17.182 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/27’ 2019-12-05T06:13:17.182 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/31’ 2019-12-05T06:13:17.182 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/36’ 2019-12-05T06:13:17.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/20’ 2019-12-05T06:13:17.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/39’ 2019-12-05T06:13:17.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/34’ 2019-12-05T06:13:17.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/38’ 2019-12-05T06:13:17.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/28’ 2019-12-05T06:13:17.183 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/24’ 2019-12-05T06:13:17.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/29’ 2019-12-05T06:13:17.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/32’ 2019-12-05T06:13:17.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/22’ 2019-12-05T06:13:17.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/25’ 2019-12-05T06:13:17.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/30’ 2019-12-05T06:13:17.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/33’ 2019-12-05T06:13:17.184 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/23’ 2019-12-05T06:13:17.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/35’ 2019-12-05T06:13:17.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/26’ 2019-12-05T06:13:17.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./273/37’ 2019-12-05T06:13:17.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./273’ 2019-12-05T06:13:17.185 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/21’ 2019-12-05T06:13:17.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/27’ 2019-12-05T06:13:17.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/31’ 2019-12-05T06:13:17.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/36’ 2019-12-05T06:13:17.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/20’ 2019-12-05T06:13:17.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/39’ 2019-12-05T06:13:17.186 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/34’ 2019-12-05T06:13:17.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/38’ 2019-12-05T06:13:17.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/28’ 2019-12-05T06:13:17.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/24’ 2019-12-05T06:13:17.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/29’ 2019-12-05T06:13:17.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/32’ 2019-12-05T06:13:17.187 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/22’ 2019-12-05T06:13:17.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/25’ 2019-12-05T06:13:17.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/30’ 2019-12-05T06:13:17.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/33’ 2019-12-05T06:13:17.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/23’ 2019-12-05T06:13:17.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/35’ 2019-12-05T06:13:17.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/26’ 2019-12-05T06:13:17.188 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./274/37’ 2019-12-05T06:13:17.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./274’ 2019-12-05T06:13:17.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/21’ 2019-12-05T06:13:17.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/27’ 2019-12-05T06:13:17.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/31’ 2019-12-05T06:13:17.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/36’ 2019-12-05T06:13:17.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/20’ 2019-12-05T06:13:17.189 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/39’ 2019-12-05T06:13:17.190 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/34’ 2019-12-05T06:13:17.190 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/38’ 2019-12-05T06:13:17.190 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/28’ 2019-12-05T06:13:17.190 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/24’ 2019-12-05T06:13:17.190 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/29’ 2019-12-05T06:13:17.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/32’ 2019-12-05T06:13:17.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/22’ 2019-12-05T06:13:17.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/25’ 2019-12-05T06:13:17.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/30’ 2019-12-05T06:13:17.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/33’ 2019-12-05T06:13:17.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/23’ 2019-12-05T06:13:17.191 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/35’ 2019-12-05T06:13:17.192 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/26’ 2019-12-05T06:13:17.192 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./275/37’ 2019-12-05T06:13:17.192 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./275’ 2019-12-05T06:13:17.192 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/21’ 2019-12-05T06:13:17.192 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/27’ 2019-12-05T06:13:17.192 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/31’ 2019-12-05T06:13:17.192 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/36’ 2019-12-05T06:13:17.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/20’ 2019-12-05T06:13:17.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/39’ 2019-12-05T06:13:17.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/34’ 2019-12-05T06:13:17.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/38’ 2019-12-05T06:13:17.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/28’ 2019-12-05T06:13:17.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/24’ 2019-12-05T06:13:17.193 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/29’ 2019-12-05T06:13:17.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/32’ 2019-12-05T06:13:17.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/22’ 2019-12-05T06:13:17.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/25’ 2019-12-05T06:13:17.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/30’ 2019-12-05T06:13:17.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/33’ 2019-12-05T06:13:17.194 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/23’ 2019-12-05T06:13:17.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/35’ 2019-12-05T06:13:17.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/26’ 2019-12-05T06:13:17.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./276/37’ 2019-12-05T06:13:17.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./276’ 2019-12-05T06:13:17.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/21’ 2019-12-05T06:13:17.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/27’ 2019-12-05T06:13:17.195 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/31’ 2019-12-05T06:13:17.196 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/36’ 2019-12-05T06:13:17.196 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/20’ 2019-12-05T06:13:17.196 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/39’ 2019-12-05T06:13:17.196 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/34’ 2019-12-05T06:13:17.196 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/38’ 2019-12-05T06:13:17.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/28’ 2019-12-05T06:13:17.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/24’ 2019-12-05T06:13:17.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/29’ 2019-12-05T06:13:17.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/32’ 2019-12-05T06:13:17.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/22’ 2019-12-05T06:13:17.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/25’ 2019-12-05T06:13:17.197 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/30’ 2019-12-05T06:13:17.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/33’ 2019-12-05T06:13:17.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/23’ 2019-12-05T06:13:17.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/35’ 2019-12-05T06:13:17.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/26’ 2019-12-05T06:13:17.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./277/37’ 2019-12-05T06:13:17.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./277’ 2019-12-05T06:13:17.198 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/21’ 2019-12-05T06:13:17.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/27’ 2019-12-05T06:13:17.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/31’ 2019-12-05T06:13:17.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/36’ 2019-12-05T06:13:17.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/20’ 2019-12-05T06:13:17.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/39’ 2019-12-05T06:13:17.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/34’ 2019-12-05T06:13:17.199 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/38’ 2019-12-05T06:13:17.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/28’ 2019-12-05T06:13:17.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/24’ 2019-12-05T06:13:17.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/29’ 2019-12-05T06:13:17.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/32’ 2019-12-05T06:13:17.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/22’ 2019-12-05T06:13:17.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/25’ 2019-12-05T06:13:17.200 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/30’ 2019-12-05T06:13:17.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/33’ 2019-12-05T06:13:17.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/23’ 2019-12-05T06:13:17.201 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/35’ 2019-12-05T06:13:20.496 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/26’ 2019-12-05T06:13:20.496 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./278/37’ 2019-12-05T06:13:20.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./278’ 2019-12-05T06:13:20.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/21’ 2019-12-05T06:13:20.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/27’ 2019-12-05T06:13:20.497 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/31’ 2019-12-05T06:13:20.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/36’ 2019-12-05T06:13:20.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/20’ 2019-12-05T06:13:20.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/39’ 2019-12-05T06:13:20.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/34’ 2019-12-05T06:13:20.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/38’ 2019-12-05T06:13:20.498 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/28’ 2019-12-05T06:13:20.499 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/24’ 2019-12-05T06:13:20.499 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/29’ 2019-12-05T06:13:20.499 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/32’ 2019-12-05T06:13:20.499 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/22’ 2019-12-05T06:13:20.499 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/25’ 2019-12-05T06:13:20.499 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/30’ 2019-12-05T06:13:20.499 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/33’ 2019-12-05T06:13:20.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/23’ 2019-12-05T06:13:20.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/35’ 2019-12-05T06:13:20.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/26’ 2019-12-05T06:13:20.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./279/37’ 2019-12-05T06:13:20.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./279’ 2019-12-05T06:13:20.500 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/21’ 2019-12-05T06:13:20.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/27’ 2019-12-05T06:13:20.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/31’ 2019-12-05T06:13:20.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/36’ 2019-12-05T06:13:20.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/20’ 2019-12-05T06:13:20.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/39’ 2019-12-05T06:13:20.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/34’ 2019-12-05T06:13:20.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/38’ 2019-12-05T06:13:20.501 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/28’ 2019-12-05T06:13:20.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/24’ 2019-12-05T06:13:20.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/29’ 2019-12-05T06:13:20.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/32’ 2019-12-05T06:13:20.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/22’ 2019-12-05T06:13:20.502 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/25’ 2019-12-05T06:13:20.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/30’ 2019-12-05T06:13:20.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/33’ 2019-12-05T06:13:20.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/23’ 2019-12-05T06:13:20.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/35’ 2019-12-05T06:13:20.503 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/26’ 2019-12-05T06:13:20.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./280/37’ 2019-12-05T06:13:20.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./280’ 2019-12-05T06:13:20.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/21’ 2019-12-05T06:13:20.504 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/27’ 2019-12-05T06:13:20.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/31’ 2019-12-05T06:13:20.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/36’ 2019-12-05T06:13:20.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/20’ 2019-12-05T06:13:20.505 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/39’ 2019-12-05T06:13:20.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/34’ 2019-12-05T06:13:20.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/38’ 2019-12-05T06:13:20.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/28’ 2019-12-05T06:13:20.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/24’ 2019-12-05T06:13:20.506 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/29’ 2019-12-05T06:13:20.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/32’ 2019-12-05T06:13:20.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/22’ 2019-12-05T06:13:20.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/25’ 2019-12-05T06:13:20.507 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/30’ 2019-12-05T06:13:20.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/33’ 2019-12-05T06:13:20.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/23’ 2019-12-05T06:13:20.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/35’ 2019-12-05T06:13:20.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/26’ 2019-12-05T06:13:20.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./281/37’ 2019-12-05T06:13:20.508 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./281’ 2019-12-05T06:13:20.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/21’ 2019-12-05T06:13:20.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/27’ 2019-12-05T06:13:20.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/31’ 2019-12-05T06:13:20.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/36’ 2019-12-05T06:13:20.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/20’ 2019-12-05T06:13:20.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/39’ 2019-12-05T06:13:20.509 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/34’ 2019-12-05T06:13:20.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/38’ 2019-12-05T06:13:20.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/28’ 2019-12-05T06:13:20.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/24’ 2019-12-05T06:13:20.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/29’ 2019-12-05T06:13:20.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/32’ 2019-12-05T06:13:20.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/22’ 2019-12-05T06:13:20.510 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/25’ 2019-12-05T06:13:20.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/30’ 2019-12-05T06:13:20.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/33’ 2019-12-05T06:13:20.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/23’ 2019-12-05T06:13:20.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/35’ 2019-12-05T06:13:20.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/26’ 2019-12-05T06:13:20.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./282/37’ 2019-12-05T06:13:20.511 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./282’ 2019-12-05T06:13:20.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/21’ 2019-12-05T06:13:20.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/27’ 2019-12-05T06:13:20.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/31’ 2019-12-05T06:13:20.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/36’ 2019-12-05T06:13:20.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/20’ 2019-12-05T06:13:20.512 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/39’ 2019-12-05T06:13:20.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/34’ 2019-12-05T06:13:20.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/38’ 2019-12-05T06:13:20.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/28’ 2019-12-05T06:13:20.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/24’ 2019-12-05T06:13:20.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/29’ 2019-12-05T06:13:20.513 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/32’ 2019-12-05T06:13:20.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/22’ 2019-12-05T06:13:20.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/25’ 2019-12-05T06:13:20.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/30’ 2019-12-05T06:13:20.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/33’ 2019-12-05T06:13:20.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/23’ 2019-12-05T06:13:20.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/35’ 2019-12-05T06:13:20.514 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/26’ 2019-12-05T06:13:20.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./283/37’ 2019-12-05T06:13:20.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./283’ 2019-12-05T06:13:20.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/21’ 2019-12-05T06:13:20.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/27’ 2019-12-05T06:13:20.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/31’ 2019-12-05T06:13:20.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/36’ 2019-12-05T06:13:20.515 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/20’ 2019-12-05T06:13:20.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/39’ 2019-12-05T06:13:20.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/34’ 2019-12-05T06:13:20.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/38’ 2019-12-05T06:13:20.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/28’ 2019-12-05T06:13:20.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/24’ 2019-12-05T06:13:20.516 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/29’ 2019-12-05T06:13:20.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/32’ 2019-12-05T06:13:20.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/22’ 2019-12-05T06:13:20.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/25’ 2019-12-05T06:13:20.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/30’ 2019-12-05T06:13:20.517 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/33’ 2019-12-05T06:13:20.518 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/23’ 2019-12-05T06:13:20.518 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/35’ 2019-12-05T06:13:20.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/26’ 2019-12-05T06:13:20.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./284/37’ 2019-12-05T06:13:20.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./284’ 2019-12-05T06:13:20.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/21’ 2019-12-05T06:13:20.519 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/27’ 2019-12-05T06:13:20.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/31’ 2019-12-05T06:13:20.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/36’ 2019-12-05T06:13:20.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/20’ 2019-12-05T06:13:20.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/39’ 2019-12-05T06:13:20.520 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/34’ 2019-12-05T06:13:20.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/38’ 2019-12-05T06:13:20.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/28’ 2019-12-05T06:13:20.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/24’ 2019-12-05T06:13:20.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/29’ 2019-12-05T06:13:20.521 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/32’ 2019-12-05T06:13:20.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/22’ 2019-12-05T06:13:20.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/25’ 2019-12-05T06:13:20.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/30’ 2019-12-05T06:13:20.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/33’ 2019-12-05T06:13:20.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/23’ 2019-12-05T06:13:20.522 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/35’ 2019-12-05T06:13:20.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/26’ 2019-12-05T06:13:20.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./285/37’ 2019-12-05T06:13:20.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./285’ 2019-12-05T06:13:20.523 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/21’ 2019-12-05T06:13:20.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/27’ 2019-12-05T06:13:20.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/31’ 2019-12-05T06:13:20.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/36’ 2019-12-05T06:13:20.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/20’ 2019-12-05T06:13:20.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/39’ 2019-12-05T06:13:20.524 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/34’ 2019-12-05T06:13:20.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/38’ 2019-12-05T06:13:20.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/28’ 2019-12-05T06:13:20.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/24’ 2019-12-05T06:13:20.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/29’ 2019-12-05T06:13:20.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/32’ 2019-12-05T06:13:20.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/22’ 2019-12-05T06:13:20.525 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/25’ 2019-12-05T06:13:20.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/30’ 2019-12-05T06:13:20.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/33’ 2019-12-05T06:13:20.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/23’ 2019-12-05T06:13:20.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/35’ 2019-12-05T06:13:20.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/26’ 2019-12-05T06:13:20.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./286/37’ 2019-12-05T06:13:20.526 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./286’ 2019-12-05T06:13:20.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/21’ 2019-12-05T06:13:20.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/27’ 2019-12-05T06:13:20.527 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/31’ 2019-12-05T06:13:24.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/36’ 2019-12-05T06:13:24.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/20’ 2019-12-05T06:13:24.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/39’ 2019-12-05T06:13:24.537 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/34’ 2019-12-05T06:13:24.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/38’ 2019-12-05T06:13:24.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/28’ 2019-12-05T06:13:24.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/24’ 2019-12-05T06:13:24.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/29’ 2019-12-05T06:13:24.538 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/32’ 2019-12-05T06:13:24.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/22’ 2019-12-05T06:13:24.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/25’ 2019-12-05T06:13:24.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/30’ 2019-12-05T06:13:24.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/33’ 2019-12-05T06:13:24.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/23’ 2019-12-05T06:13:24.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/35’ 2019-12-05T06:13:24.539 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/26’ 2019-12-05T06:13:24.540 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./287/37’ 2019-12-05T06:13:24.540 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./287’ 2019-12-05T06:13:24.540 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/21’ 2019-12-05T06:13:24.540 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/27’ 2019-12-05T06:13:24.540 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/31’ 2019-12-05T06:13:24.540 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/36’ 2019-12-05T06:13:24.541 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/20’ 2019-12-05T06:13:24.541 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/39’ 2019-12-05T06:13:24.541 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/34’ 2019-12-05T06:13:24.541 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/38’ 2019-12-05T06:13:24.541 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/28’ 2019-12-05T06:13:24.541 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/24’ 2019-12-05T06:13:24.541 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/29’ 2019-12-05T06:13:24.542 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/32’ 2019-12-05T06:13:24.542 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/22’ 2019-12-05T06:13:24.542 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/25’ 2019-12-05T06:13:24.542 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/30’ 2019-12-05T06:13:24.542 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/33’ 2019-12-05T06:13:24.543 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/23’ 2019-12-05T06:13:24.543 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/35’ 2019-12-05T06:13:24.543 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/26’ 2019-12-05T06:13:24.543 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./288/37’ 2019-12-05T06:13:24.543 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./288’ 2019-12-05T06:13:24.543 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/21’ 2019-12-05T06:13:24.544 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/27’ 2019-12-05T06:13:24.544 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/31’ 2019-12-05T06:13:24.544 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/36’ 2019-12-05T06:13:24.544 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/20’ 2019-12-05T06:13:24.544 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/39’ 2019-12-05T06:13:24.545 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/34’ 2019-12-05T06:13:24.545 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/38’ 2019-12-05T06:13:24.545 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/28’ 2019-12-05T06:13:24.545 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/24’ 2019-12-05T06:13:24.545 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/29’ 2019-12-05T06:13:24.545 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/32’ 2019-12-05T06:13:24.546 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/22’ 2019-12-05T06:13:24.546 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/25’ 2019-12-05T06:13:24.546 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/30’ 2019-12-05T06:13:24.546 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/33’ 2019-12-05T06:13:24.546 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/23’ 2019-12-05T06:13:24.546 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/35’ 2019-12-05T06:13:24.547 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/26’ 2019-12-05T06:13:24.547 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./289/37’ 2019-12-05T06:13:24.547 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./289’ 2019-12-05T06:13:24.547 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/21’ 2019-12-05T06:13:24.547 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/27’ 2019-12-05T06:13:24.547 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/31’ 2019-12-05T06:13:24.547 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/36’ 2019-12-05T06:13:24.548 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/20’ 2019-12-05T06:13:24.548 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/39’ 2019-12-05T06:13:24.548 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/34’ 2019-12-05T06:13:24.548 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/38’ 2019-12-05T06:13:24.548 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/28’ 2019-12-05T06:13:24.549 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/24’ 2019-12-05T06:13:24.549 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/29’ 2019-12-05T06:13:24.549 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/32’ 2019-12-05T06:13:24.549 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/22’ 2019-12-05T06:13:24.549 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/25’ 2019-12-05T06:13:24.549 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/30’ 2019-12-05T06:13:24.550 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/33’ 2019-12-05T06:13:24.550 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/23’ 2019-12-05T06:13:24.550 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/35’ 2019-12-05T06:13:24.550 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/26’ 2019-12-05T06:13:24.550 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./290/37’ 2019-12-05T06:13:24.550 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./290’ 2019-12-05T06:13:24.551 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/21’ 2019-12-05T06:13:24.551 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/27’ 2019-12-05T06:13:24.551 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/31’ 2019-12-05T06:13:24.551 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/36’ 2019-12-05T06:13:24.551 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/20’ 2019-12-05T06:13:24.551 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/39’ 2019-12-05T06:13:24.551 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/34’ 2019-12-05T06:13:24.552 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/38’ 2019-12-05T06:13:24.552 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/28’ 2019-12-05T06:13:24.552 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/24’ 2019-12-05T06:13:24.552 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/29’ 2019-12-05T06:13:24.552 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/32’ 2019-12-05T06:13:24.552 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/22’ 2019-12-05T06:13:24.553 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/25’ 2019-12-05T06:13:24.553 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/30’ 2019-12-05T06:13:24.553 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/33’ 2019-12-05T06:13:24.553 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/23’ 2019-12-05T06:13:24.553 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/35’ 2019-12-05T06:13:24.553 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/26’ 2019-12-05T06:13:24.554 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./291/37’ 2019-12-05T06:13:24.554 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./291’ 2019-12-05T06:13:24.554 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/21’ 2019-12-05T06:13:24.554 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/27’ 2019-12-05T06:13:24.555 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/31’ 2019-12-05T06:13:24.555 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/36’ 2019-12-05T06:13:24.555 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/20’ 2019-12-05T06:13:24.556 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/39’ 2019-12-05T06:13:24.556 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/34’ 2019-12-05T06:13:24.556 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/38’ 2019-12-05T06:13:24.556 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/28’ 2019-12-05T06:13:24.556 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/24’ 2019-12-05T06:13:24.556 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/29’ 2019-12-05T06:13:24.557 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/32’ 2019-12-05T06:13:24.557 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/22’ 2019-12-05T06:13:24.557 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/25’ 2019-12-05T06:13:24.557 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/30’ 2019-12-05T06:13:24.557 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/33’ 2019-12-05T06:13:24.557 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/23’ 2019-12-05T06:13:24.558 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/35’ 2019-12-05T06:13:24.558 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/26’ 2019-12-05T06:13:24.558 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./292/37’ 2019-12-05T06:13:24.558 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./292’ 2019-12-05T06:13:24.558 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/21’ 2019-12-05T06:13:24.559 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/27’ 2019-12-05T06:13:24.559 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/31’ 2019-12-05T06:13:24.559 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/36’ 2019-12-05T06:13:24.559 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/20’ 2019-12-05T06:13:24.560 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/39’ 2019-12-05T06:13:24.560 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/34’ 2019-12-05T06:13:24.560 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/38’ 2019-12-05T06:13:24.561 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/28’ 2019-12-05T06:13:24.561 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/24’ 2019-12-05T06:13:24.561 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/29’ 2019-12-05T06:13:24.561 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/32’ 2019-12-05T06:13:24.562 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/22’ 2019-12-05T06:13:24.562 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/25’ 2019-12-05T06:13:24.562 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/30’ 2019-12-05T06:13:24.562 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/33’ 2019-12-05T06:13:24.562 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/23’ 2019-12-05T06:13:24.562 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/35’ 2019-12-05T06:13:24.563 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/26’ 2019-12-05T06:13:24.563 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./293/37’ 2019-12-05T06:13:24.563 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./293’ 2019-12-05T06:13:24.563 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/21’ 2019-12-05T06:13:24.563 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/27’ 2019-12-05T06:13:24.563 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/31’ 2019-12-05T06:13:24.563 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/36’ 2019-12-05T06:13:24.564 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/20’ 2019-12-05T06:13:24.564 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/39’ 2019-12-05T06:13:24.564 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/34’ 2019-12-05T06:13:24.564 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/38’ 2019-12-05T06:13:24.564 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/28’ 2019-12-05T06:13:24.564 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/24’ 2019-12-05T06:13:24.565 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/29’ 2019-12-05T06:13:24.565 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/32’ 2019-12-05T06:13:24.565 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/22’ 2019-12-05T06:13:24.565 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/25’ 2019-12-05T06:13:24.565 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/30’ 2019-12-05T06:13:24.565 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/33’ 2019-12-05T06:13:24.565 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/23’ 2019-12-05T06:13:24.566 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/35’ 2019-12-05T06:13:24.566 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/26’ 2019-12-05T06:13:24.566 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./294/37’ 2019-12-05T06:13:24.566 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./294’ 2019-12-05T06:13:24.566 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/21’ 2019-12-05T06:13:24.566 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/27’ 2019-12-05T06:13:24.566 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/31’ 2019-12-05T06:13:24.567 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/36’ 2019-12-05T06:13:24.567 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/20’ 2019-12-05T06:13:24.567 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/39’ 2019-12-05T06:13:24.567 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/34’ 2019-12-05T06:13:24.567 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/38’ 2019-12-05T06:13:24.567 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/28’ 2019-12-05T06:13:24.568 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/24’ 2019-12-05T06:13:24.568 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/29’ 2019-12-05T06:13:29.094 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/32’ 2019-12-05T06:13:29.095 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/22’ 2019-12-05T06:13:29.095 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/25’ 2019-12-05T06:13:29.095 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/30’ 2019-12-05T06:13:29.095 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/33’ 2019-12-05T06:13:29.095 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/23’ 2019-12-05T06:13:29.095 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/35’ 2019-12-05T06:13:29.096 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/26’ 2019-12-05T06:13:29.096 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./295/37’ 2019-12-05T06:13:29.096 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./295’ 2019-12-05T06:13:29.096 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/21’ 2019-12-05T06:13:29.096 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/27’ 2019-12-05T06:13:29.096 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/31’ 2019-12-05T06:13:29.096 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/36’ 2019-12-05T06:13:29.097 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/20’ 2019-12-05T06:13:29.097 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/39’ 2019-12-05T06:13:29.097 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/34’ 2019-12-05T06:13:29.097 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/38’ 2019-12-05T06:13:29.097 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/28’ 2019-12-05T06:13:29.097 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/24’ 2019-12-05T06:13:29.098 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/29’ 2019-12-05T06:13:29.098 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/32’ 2019-12-05T06:13:29.098 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/22’ 2019-12-05T06:13:29.098 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/25’ 2019-12-05T06:13:29.098 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/30’ 2019-12-05T06:13:29.098 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/33’ 2019-12-05T06:13:29.098 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/23’ 2019-12-05T06:13:29.099 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/35’ 2019-12-05T06:13:29.099 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/26’ 2019-12-05T06:13:29.099 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./296/37’ 2019-12-05T06:13:29.099 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./296’ 2019-12-05T06:13:29.099 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/21’ 2019-12-05T06:13:29.099 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/27’ 2019-12-05T06:13:29.099 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/31’ 2019-12-05T06:13:29.099 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/36’ 2019-12-05T06:13:29.100 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/20’ 2019-12-05T06:13:29.100 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/39’ 2019-12-05T06:13:29.100 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/34’ 2019-12-05T06:13:29.100 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/38’ 2019-12-05T06:13:29.100 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/28’ 2019-12-05T06:13:29.100 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/24’ 2019-12-05T06:13:29.101 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/29’ 2019-12-05T06:13:29.101 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/32’ 2019-12-05T06:13:29.101 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/22’ 2019-12-05T06:13:29.101 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/25’ 2019-12-05T06:13:29.101 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/30’ 2019-12-05T06:13:29.101 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/33’ 2019-12-05T06:13:29.101 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/23’ 2019-12-05T06:13:29.101 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/35’ 2019-12-05T06:13:29.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/26’ 2019-12-05T06:13:29.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./297/37’ 2019-12-05T06:13:29.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./297’ 2019-12-05T06:13:29.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/21’ 2019-12-05T06:13:29.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/27’ 2019-12-05T06:13:29.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/31’ 2019-12-05T06:13:29.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/36’ 2019-12-05T06:13:29.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/20’ 2019-12-05T06:13:29.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/39’ 2019-12-05T06:13:29.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/34’ 2019-12-05T06:13:29.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/38’ 2019-12-05T06:13:29.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/28’ 2019-12-05T06:13:29.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/24’ 2019-12-05T06:13:29.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/29’ 2019-12-05T06:13:29.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/32’ 2019-12-05T06:13:29.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/22’ 2019-12-05T06:13:29.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/25’ 2019-12-05T06:13:29.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/30’ 2019-12-05T06:13:29.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/33’ 2019-12-05T06:13:29.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/23’ 2019-12-05T06:13:29.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/35’ 2019-12-05T06:13:29.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/26’ 2019-12-05T06:13:29.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./298/37’ 2019-12-05T06:13:29.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./298’ 2019-12-05T06:13:29.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/21’ 2019-12-05T06:13:29.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/27’ 2019-12-05T06:13:29.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/31’ 2019-12-05T06:13:29.106 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/36’ 2019-12-05T06:13:29.106 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/20’ 2019-12-05T06:13:29.106 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/39’ 2019-12-05T06:13:29.106 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/34’ 2019-12-05T06:13:29.106 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/38’ 2019-12-05T06:13:29.106 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/28’ 2019-12-05T06:13:29.106 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/24’ 2019-12-05T06:13:29.107 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/29’ 2019-12-05T06:13:29.107 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/32’ 2019-12-05T06:13:29.107 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/22’ 2019-12-05T06:13:29.107 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/25’ 2019-12-05T06:13:29.107 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/30’ 2019-12-05T06:13:29.107 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/33’ 2019-12-05T06:13:29.107 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/23’ 2019-12-05T06:13:29.108 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/35’ 2019-12-05T06:13:29.108 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/26’ 2019-12-05T06:13:29.108 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./299/37’ 2019-12-05T06:13:29.108 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./299’ 2019-12-05T06:13:29.108 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/19’ 2019-12-05T06:13:29.108 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/15’ 2019-12-05T06:13:29.108 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/20’ 2019-12-05T06:13:29.109 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/12’ 2019-12-05T06:13:29.109 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/10’ 2019-12-05T06:13:29.109 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/11’ 2019-12-05T06:13:29.109 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/16’ 2019-12-05T06:13:29.109 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/14’ 2019-12-05T06:13:29.109 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/18’ 2019-12-05T06:13:29.110 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/13’ 2019-12-05T06:13:29.110 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/166/17’ 2019-12-05T06:13:29.110 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/166’ 2019-12-05T06:13:29.110 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/19’ 2019-12-05T06:13:29.110 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/15’ 2019-12-05T06:13:29.110 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/20’ 2019-12-05T06:13:29.110 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/12’ 2019-12-05T06:13:29.111 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/10’ 2019-12-05T06:13:29.111 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/11’ 2019-12-05T06:13:29.111 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/16’ 2019-12-05T06:13:29.111 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/14’ 2019-12-05T06:13:29.111 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/18’ 2019-12-05T06:13:29.111 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/13’ 2019-12-05T06:13:29.111 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/147/17’ 2019-12-05T06:13:29.112 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/147’ 2019-12-05T06:13:29.112 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/19’ 2019-12-05T06:13:29.112 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/15’ 2019-12-05T06:13:29.112 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/20’ 2019-12-05T06:13:29.112 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/12’ 2019-12-05T06:13:29.112 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/10’ 2019-12-05T06:13:29.113 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/11’ 2019-12-05T06:13:29.113 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/16’ 2019-12-05T06:13:29.113 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/14’ 2019-12-05T06:13:29.113 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/18’ 2019-12-05T06:13:29.113 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/13’ 2019-12-05T06:13:29.113 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/151/17’ 2019-12-05T06:13:29.113 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/151’ 2019-12-05T06:13:29.114 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/19’ 2019-12-05T06:13:29.114 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/15’ 2019-12-05T06:13:29.114 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/20’ 2019-12-05T06:13:29.114 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/12’ 2019-12-05T06:13:29.114 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/10’ 2019-12-05T06:13:29.114 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/11’ 2019-12-05T06:13:29.114 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/16’ 2019-12-05T06:13:29.114 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/14’ 2019-12-05T06:13:29.115 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/18’ 2019-12-05T06:13:29.773 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:13:29.778 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:13:31.674 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/13’ 2019-12-05T06:13:31.674 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/170/17’ 2019-12-05T06:13:31.674 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/170’ 2019-12-05T06:13:31.675 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/19’ 2019-12-05T06:13:31.675 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/15’ 2019-12-05T06:13:31.675 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/20’ 2019-12-05T06:13:31.675 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/12’ 2019-12-05T06:13:31.675 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/10’ 2019-12-05T06:13:31.675 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/11’ 2019-12-05T06:13:31.675 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/16’ 2019-12-05T06:13:31.676 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/14’ 2019-12-05T06:13:31.676 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/18’ 2019-12-05T06:13:31.676 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/13’ 2019-12-05T06:13:31.676 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/167/17’ 2019-12-05T06:13:31.676 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/167’ 2019-12-05T06:13:31.684 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/19’ 2019-12-05T06:13:31.684 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/15’ 2019-12-05T06:13:31.684 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/20’ 2019-12-05T06:13:31.684 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/12’ 2019-12-05T06:13:31.684 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/10’ 2019-12-05T06:13:31.685 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/11’ 2019-12-05T06:13:31.685 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/16’ 2019-12-05T06:13:31.685 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/14’ 2019-12-05T06:13:31.685 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/18’ 2019-12-05T06:13:31.685 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/13’ 2019-12-05T06:13:31.685 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/140/17’ 2019-12-05T06:13:31.685 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/140’ 2019-12-05T06:13:31.686 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/19’ 2019-12-05T06:13:31.686 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/15’ 2019-12-05T06:13:31.686 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/20’ 2019-12-05T06:13:31.686 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/12’ 2019-12-05T06:13:31.686 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/10’ 2019-12-05T06:13:31.686 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/11’ 2019-12-05T06:13:31.687 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/16’ 2019-12-05T06:13:31.687 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/14’ 2019-12-05T06:13:31.687 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/18’ 2019-12-05T06:13:31.687 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/13’ 2019-12-05T06:13:31.687 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/142/17’ 2019-12-05T06:13:31.687 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/142’ 2019-12-05T06:13:31.688 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/19’ 2019-12-05T06:13:31.688 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/15’ 2019-12-05T06:13:31.688 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/20’ 2019-12-05T06:13:31.688 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/12’ 2019-12-05T06:13:31.688 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/10’ 2019-12-05T06:13:31.688 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/11’ 2019-12-05T06:13:31.688 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/16’ 2019-12-05T06:13:31.689 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/14’ 2019-12-05T06:13:31.689 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/18’ 2019-12-05T06:13:31.689 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/13’ 2019-12-05T06:13:31.689 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/150/17’ 2019-12-05T06:13:31.689 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/150’ 2019-12-05T06:13:31.689 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/19’ 2019-12-05T06:13:31.690 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/15’ 2019-12-05T06:13:31.690 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/20’ 2019-12-05T06:13:31.690 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/12’ 2019-12-05T06:13:31.690 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/10’ 2019-12-05T06:13:31.690 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/11’ 2019-12-05T06:13:31.690 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/16’ 2019-12-05T06:13:31.690 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/14’ 2019-12-05T06:13:31.691 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/18’ 2019-12-05T06:13:31.691 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/13’ 2019-12-05T06:13:31.691 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/105/17’ 2019-12-05T06:13:31.691 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/105’ 2019-12-05T06:13:31.691 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/19’ 2019-12-05T06:13:31.691 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/15’ 2019-12-05T06:13:31.691 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/20’ 2019-12-05T06:13:31.692 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/12’ 2019-12-05T06:13:31.692 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/10’ 2019-12-05T06:13:31.692 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/11’ 2019-12-05T06:13:31.692 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/16’ 2019-12-05T06:13:31.692 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/14’ 2019-12-05T06:13:31.695 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/18’ 2019-12-05T06:13:31.695 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/13’ 2019-12-05T06:13:31.695 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/149/17’ 2019-12-05T06:13:31.695 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/149’ 2019-12-05T06:13:31.695 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/19’ 2019-12-05T06:13:31.695 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/15’ 2019-12-05T06:13:31.696 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/20’ 2019-12-05T06:13:31.696 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/12’ 2019-12-05T06:13:31.696 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/10’ 2019-12-05T06:13:31.696 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/11’ 2019-12-05T06:13:31.700 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/16’ 2019-12-05T06:13:31.700 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/14’ 2019-12-05T06:13:31.700 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/18’ 2019-12-05T06:13:31.700 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/13’ 2019-12-05T06:13:31.700 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/109/17’ 2019-12-05T06:13:31.701 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/109’ 2019-12-05T06:13:31.701 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/19’ 2019-12-05T06:13:31.701 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/15’ 2019-12-05T06:13:31.701 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/20’ 2019-12-05T06:13:31.701 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/12’ 2019-12-05T06:13:31.701 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/10’ 2019-12-05T06:13:31.701 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/11’ 2019-12-05T06:13:31.702 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/16’ 2019-12-05T06:13:31.702 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/14’ 2019-12-05T06:13:31.702 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/18’ 2019-12-05T06:13:31.702 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/13’ 2019-12-05T06:13:31.702 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/158/17’ 2019-12-05T06:13:33.720 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/158’ 2019-12-05T06:13:33.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/19’ 2019-12-05T06:13:33.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/15’ 2019-12-05T06:13:33.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/20’ 2019-12-05T06:13:33.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/12’ 2019-12-05T06:13:33.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/10’ 2019-12-05T06:13:33.729 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/11’ 2019-12-05T06:13:33.730 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/16’ 2019-12-05T06:13:33.730 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/14’ 2019-12-05T06:13:33.730 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/18’ 2019-12-05T06:13:33.730 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/13’ 2019-12-05T06:13:33.730 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/106/17’ 2019-12-05T06:13:33.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/106’ 2019-12-05T06:13:33.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/19’ 2019-12-05T06:13:33.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/15’ 2019-12-05T06:13:33.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/20’ 2019-12-05T06:13:33.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/12’ 2019-12-05T06:13:33.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/10’ 2019-12-05T06:13:33.731 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/11’ 2019-12-05T06:13:33.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/16’ 2019-12-05T06:13:33.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/14’ 2019-12-05T06:13:33.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/18’ 2019-12-05T06:13:33.732 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/13’ 2019-12-05T06:13:33.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/189/17’ 2019-12-05T06:13:33.741 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/189’ 2019-12-05T06:13:33.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/19’ 2019-12-05T06:13:33.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/15’ 2019-12-05T06:13:33.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/20’ 2019-12-05T06:13:33.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/12’ 2019-12-05T06:13:33.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/10’ 2019-12-05T06:13:33.742 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/11’ 2019-12-05T06:13:33.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/16’ 2019-12-05T06:13:33.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/14’ 2019-12-05T06:13:33.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/18’ 2019-12-05T06:13:33.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/13’ 2019-12-05T06:13:33.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/133/17’ 2019-12-05T06:13:33.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/133’ 2019-12-05T06:13:33.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/19’ 2019-12-05T06:13:33.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/15’ 2019-12-05T06:13:33.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/20’ 2019-12-05T06:13:33.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/12’ 2019-12-05T06:13:33.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/10’ 2019-12-05T06:13:33.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/11’ 2019-12-05T06:13:33.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/16’ 2019-12-05T06:13:33.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/14’ 2019-12-05T06:13:33.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/18’ 2019-12-05T06:13:33.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/13’ 2019-12-05T06:13:33.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/173/17’ 2019-12-05T06:13:33.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/173’ 2019-12-05T06:13:33.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/19’ 2019-12-05T06:13:33.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/15’ 2019-12-05T06:13:33.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/20’ 2019-12-05T06:13:33.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/12’ 2019-12-05T06:13:33.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/10’ 2019-12-05T06:13:33.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/11’ 2019-12-05T06:13:33.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/16’ 2019-12-05T06:13:33.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/14’ 2019-12-05T06:13:33.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/18’ 2019-12-05T06:13:33.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/13’ 2019-12-05T06:13:33.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/174/17’ 2019-12-05T06:13:33.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/174’ 2019-12-05T06:13:33.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/19’ 2019-12-05T06:13:33.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/15’ 2019-12-05T06:13:33.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/20’ 2019-12-05T06:13:33.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/12’ 2019-12-05T06:13:33.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/10’ 2019-12-05T06:13:33.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/11’ 2019-12-05T06:13:33.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/16’ 2019-12-05T06:13:33.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/14’ 2019-12-05T06:13:33.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/18’ 2019-12-05T06:13:33.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/13’ 2019-12-05T06:13:33.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/155/17’ 2019-12-05T06:13:33.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/155’ 2019-12-05T06:13:33.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/19’ 2019-12-05T06:13:33.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/15’ 2019-12-05T06:13:33.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/20’ 2019-12-05T06:13:33.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/12’ 2019-12-05T06:13:33.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/10’ 2019-12-05T06:13:33.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/11’ 2019-12-05T06:13:33.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/16’ 2019-12-05T06:13:33.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/14’ 2019-12-05T06:13:33.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/18’ 2019-12-05T06:13:33.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/13’ 2019-12-05T06:13:33.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/114/17’ 2019-12-05T06:13:33.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/114’ 2019-12-05T06:13:33.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/19’ 2019-12-05T06:13:33.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/15’ 2019-12-05T06:13:33.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/20’ 2019-12-05T06:13:33.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/12’ 2019-12-05T06:13:33.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/10’ 2019-12-05T06:13:33.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/11’ 2019-12-05T06:13:33.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/16’ 2019-12-05T06:13:33.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/14’ 2019-12-05T06:13:33.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/18’ 2019-12-05T06:13:33.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/13’ 2019-12-05T06:13:33.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/159/17’ 2019-12-05T06:13:33.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/159’ 2019-12-05T06:13:33.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/19’ 2019-12-05T06:13:33.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/15’ 2019-12-05T06:13:35.806 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/20’ 2019-12-05T06:13:35.806 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/12’ 2019-12-05T06:13:35.806 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/10’ 2019-12-05T06:13:35.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/11’ 2019-12-05T06:13:35.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/16’ 2019-12-05T06:13:35.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/14’ 2019-12-05T06:13:35.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/18’ 2019-12-05T06:13:35.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/13’ 2019-12-05T06:13:35.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/187/17’ 2019-12-05T06:13:35.807 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/187’ 2019-12-05T06:13:35.808 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/19’ 2019-12-05T06:13:35.808 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/15’ 2019-12-05T06:13:35.808 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/20’ 2019-12-05T06:13:35.808 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/12’ 2019-12-05T06:13:35.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/10’ 2019-12-05T06:13:35.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/11’ 2019-12-05T06:13:35.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/16’ 2019-12-05T06:13:35.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/14’ 2019-12-05T06:13:35.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/18’ 2019-12-05T06:13:35.809 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/13’ 2019-12-05T06:13:35.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/185/17’ 2019-12-05T06:13:35.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/185’ 2019-12-05T06:13:35.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/19’ 2019-12-05T06:13:35.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/15’ 2019-12-05T06:13:35.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/20’ 2019-12-05T06:13:35.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/12’ 2019-12-05T06:13:35.810 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/10’ 2019-12-05T06:13:35.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/11’ 2019-12-05T06:13:35.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/16’ 2019-12-05T06:13:35.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/14’ 2019-12-05T06:13:35.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/18’ 2019-12-05T06:13:35.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/13’ 2019-12-05T06:13:35.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/120/17’ 2019-12-05T06:13:35.811 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/120’ 2019-12-05T06:13:35.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/19’ 2019-12-05T06:13:35.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/15’ 2019-12-05T06:13:35.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/20’ 2019-12-05T06:13:35.812 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/12’ 2019-12-05T06:13:35.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/10’ 2019-12-05T06:13:35.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/11’ 2019-12-05T06:13:35.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/16’ 2019-12-05T06:13:35.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/14’ 2019-12-05T06:13:35.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/18’ 2019-12-05T06:13:35.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/13’ 2019-12-05T06:13:35.813 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/110/17’ 2019-12-05T06:13:35.814 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/110’ 2019-12-05T06:13:35.814 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/19’ 2019-12-05T06:13:35.814 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/15’ 2019-12-05T06:13:35.814 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/20’ 2019-12-05T06:13:35.814 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/12’ 2019-12-05T06:13:35.814 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/10’ 2019-12-05T06:13:35.814 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/11’ 2019-12-05T06:13:35.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/16’ 2019-12-05T06:13:35.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/14’ 2019-12-05T06:13:35.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/18’ 2019-12-05T06:13:35.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/13’ 2019-12-05T06:13:35.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/137/17’ 2019-12-05T06:13:35.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/137’ 2019-12-05T06:13:35.815 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/19’ 2019-12-05T06:13:35.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/15’ 2019-12-05T06:13:35.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/20’ 2019-12-05T06:13:35.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/12’ 2019-12-05T06:13:35.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/10’ 2019-12-05T06:13:35.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/11’ 2019-12-05T06:13:35.816 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/16’ 2019-12-05T06:13:35.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/14’ 2019-12-05T06:13:35.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/18’ 2019-12-05T06:13:35.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/13’ 2019-12-05T06:13:35.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/198/17’ 2019-12-05T06:13:35.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/198’ 2019-12-05T06:13:35.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/19’ 2019-12-05T06:13:35.817 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/15’ 2019-12-05T06:13:35.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/20’ 2019-12-05T06:13:35.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/12’ 2019-12-05T06:13:35.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/10’ 2019-12-05T06:13:35.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/11’ 2019-12-05T06:13:35.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/16’ 2019-12-05T06:13:35.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/14’ 2019-12-05T06:13:35.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/18’ 2019-12-05T06:13:35.818 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/13’ 2019-12-05T06:13:35.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/136/17’ 2019-12-05T06:13:35.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/136’ 2019-12-05T06:13:35.819 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/19’ 2019-12-05T06:13:35.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/15’ 2019-12-05T06:13:35.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/20’ 2019-12-05T06:13:35.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/12’ 2019-12-05T06:13:35.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/10’ 2019-12-05T06:13:35.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/11’ 2019-12-05T06:13:35.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/16’ 2019-12-05T06:13:35.820 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/14’ 2019-12-05T06:13:35.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/18’ 2019-12-05T06:13:35.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/13’ 2019-12-05T06:13:35.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/125/17’ 2019-12-05T06:13:35.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/125’ 2019-12-05T06:13:35.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/19’ 2019-12-05T06:13:35.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/15’ 2019-12-05T06:13:35.821 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/20’ 2019-12-05T06:13:35.822 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/12’ 2019-12-05T06:13:38.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/10’ 2019-12-05T06:13:38.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/11’ 2019-12-05T06:13:38.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/16’ 2019-12-05T06:13:38.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/14’ 2019-12-05T06:13:38.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/18’ 2019-12-05T06:13:38.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/13’ 2019-12-05T06:13:38.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/194/17’ 2019-12-05T06:13:38.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/194’ 2019-12-05T06:13:38.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/19’ 2019-12-05T06:13:38.338 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/15’ 2019-12-05T06:13:38.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/20’ 2019-12-05T06:13:38.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/12’ 2019-12-05T06:13:38.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/10’ 2019-12-05T06:13:38.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/11’ 2019-12-05T06:13:38.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/16’ 2019-12-05T06:13:38.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/14’ 2019-12-05T06:13:38.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/18’ 2019-12-05T06:13:38.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/13’ 2019-12-05T06:13:38.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/119/17’ 2019-12-05T06:13:38.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/119’ 2019-12-05T06:13:38.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/19’ 2019-12-05T06:13:38.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/15’ 2019-12-05T06:13:38.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/20’ 2019-12-05T06:13:38.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/12’ 2019-12-05T06:13:38.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/10’ 2019-12-05T06:13:38.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/11’ 2019-12-05T06:13:38.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/16’ 2019-12-05T06:13:38.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/14’ 2019-12-05T06:13:38.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/18’ 2019-12-05T06:13:38.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/13’ 2019-12-05T06:13:38.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/157/17’ 2019-12-05T06:13:38.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/157’ 2019-12-05T06:13:38.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/19’ 2019-12-05T06:13:38.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/15’ 2019-12-05T06:13:38.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/20’ 2019-12-05T06:13:38.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/12’ 2019-12-05T06:13:38.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/10’ 2019-12-05T06:13:38.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/11’ 2019-12-05T06:13:38.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/16’ 2019-12-05T06:13:38.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/14’ 2019-12-05T06:13:38.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/18’ 2019-12-05T06:13:38.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/13’ 2019-12-05T06:13:38.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/172/17’ 2019-12-05T06:13:38.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/172’ 2019-12-05T06:13:38.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/19’ 2019-12-05T06:13:38.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/15’ 2019-12-05T06:13:38.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/20’ 2019-12-05T06:13:38.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/12’ 2019-12-05T06:13:38.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/10’ 2019-12-05T06:13:38.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/11’ 2019-12-05T06:13:38.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/16’ 2019-12-05T06:13:38.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/14’ 2019-12-05T06:13:38.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/18’ 2019-12-05T06:13:38.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/13’ 2019-12-05T06:13:38.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/186/17’ 2019-12-05T06:13:38.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/186’ 2019-12-05T06:13:38.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/19’ 2019-12-05T06:13:38.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/15’ 2019-12-05T06:13:38.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/20’ 2019-12-05T06:13:38.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/12’ 2019-12-05T06:13:38.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/10’ 2019-12-05T06:13:38.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/11’ 2019-12-05T06:13:38.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/16’ 2019-12-05T06:13:38.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/14’ 2019-12-05T06:13:38.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/18’ 2019-12-05T06:13:38.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/13’ 2019-12-05T06:13:38.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/190/17’ 2019-12-05T06:13:38.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/190’ 2019-12-05T06:13:38.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/19’ 2019-12-05T06:13:38.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/15’ 2019-12-05T06:13:38.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/20’ 2019-12-05T06:13:38.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/12’ 2019-12-05T06:13:38.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/10’ 2019-12-05T06:13:38.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/11’ 2019-12-05T06:13:38.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/16’ 2019-12-05T06:13:38.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/14’ 2019-12-05T06:13:38.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/18’ 2019-12-05T06:13:38.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/13’ 2019-12-05T06:13:38.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/104/17’ 2019-12-05T06:13:38.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/104’ 2019-12-05T06:13:38.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/19’ 2019-12-05T06:13:38.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/15’ 2019-12-05T06:13:38.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/20’ 2019-12-05T06:13:38.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/12’ 2019-12-05T06:13:38.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/10’ 2019-12-05T06:13:38.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/11’ 2019-12-05T06:13:38.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/16’ 2019-12-05T06:13:38.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/14’ 2019-12-05T06:13:38.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/18’ 2019-12-05T06:13:38.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/13’ 2019-12-05T06:13:38.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/164/17’ 2019-12-05T06:13:38.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/164’ 2019-12-05T06:13:38.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/19’ 2019-12-05T06:13:38.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/15’ 2019-12-05T06:13:38.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/20’ 2019-12-05T06:13:38.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/12’ 2019-12-05T06:13:38.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/10’ 2019-12-05T06:13:38.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/11’ 2019-12-05T06:13:40.383 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/16’ 2019-12-05T06:13:40.383 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/14’ 2019-12-05T06:13:40.384 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/18’ 2019-12-05T06:13:40.384 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/13’ 2019-12-05T06:13:40.384 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/112/17’ 2019-12-05T06:13:40.384 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/112’ 2019-12-05T06:13:40.384 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/19’ 2019-12-05T06:13:40.384 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/15’ 2019-12-05T06:13:40.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/20’ 2019-12-05T06:13:40.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/12’ 2019-12-05T06:13:40.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/10’ 2019-12-05T06:13:40.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/11’ 2019-12-05T06:13:40.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/16’ 2019-12-05T06:13:40.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/14’ 2019-12-05T06:13:40.385 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/18’ 2019-12-05T06:13:40.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/13’ 2019-12-05T06:13:40.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/100/17’ 2019-12-05T06:13:40.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/100’ 2019-12-05T06:13:40.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/19’ 2019-12-05T06:13:40.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/15’ 2019-12-05T06:13:40.386 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/20’ 2019-12-05T06:13:40.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/12’ 2019-12-05T06:13:40.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/10’ 2019-12-05T06:13:40.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/11’ 2019-12-05T06:13:40.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/16’ 2019-12-05T06:13:40.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/14’ 2019-12-05T06:13:40.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/18’ 2019-12-05T06:13:40.387 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/13’ 2019-12-05T06:13:40.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/134/17’ 2019-12-05T06:13:40.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/134’ 2019-12-05T06:13:40.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/19’ 2019-12-05T06:13:40.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/15’ 2019-12-05T06:13:40.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/20’ 2019-12-05T06:13:40.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/12’ 2019-12-05T06:13:40.388 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/10’ 2019-12-05T06:13:40.389 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/11’ 2019-12-05T06:13:40.389 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/16’ 2019-12-05T06:13:40.389 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/14’ 2019-12-05T06:13:40.389 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/18’ 2019-12-05T06:13:40.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/13’ 2019-12-05T06:13:40.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/115/17’ 2019-12-05T06:13:40.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/115’ 2019-12-05T06:13:40.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/19’ 2019-12-05T06:13:40.391 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/15’ 2019-12-05T06:13:40.392 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/20’ 2019-12-05T06:13:40.392 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/12’ 2019-12-05T06:13:40.392 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/10’ 2019-12-05T06:13:40.392 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/11’ 2019-12-05T06:13:40.392 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/16’ 2019-12-05T06:13:40.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/14’ 2019-12-05T06:13:40.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/18’ 2019-12-05T06:13:40.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/13’ 2019-12-05T06:13:40.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/116/17’ 2019-12-05T06:13:40.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/116’ 2019-12-05T06:13:40.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/19’ 2019-12-05T06:13:40.393 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/15’ 2019-12-05T06:13:40.394 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/20’ 2019-12-05T06:13:40.394 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/12’ 2019-12-05T06:13:40.394 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/10’ 2019-12-05T06:13:40.394 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/11’ 2019-12-05T06:13:40.394 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/16’ 2019-12-05T06:13:40.394 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/14’ 2019-12-05T06:13:40.394 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/18’ 2019-12-05T06:13:40.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/13’ 2019-12-05T06:13:40.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/195/17’ 2019-12-05T06:13:40.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/195’ 2019-12-05T06:13:40.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/19’ 2019-12-05T06:13:40.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/15’ 2019-12-05T06:13:40.395 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/20’ 2019-12-05T06:13:40.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/12’ 2019-12-05T06:13:40.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/10’ 2019-12-05T06:13:40.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/11’ 2019-12-05T06:13:40.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/16’ 2019-12-05T06:13:40.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/14’ 2019-12-05T06:13:40.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/18’ 2019-12-05T06:13:40.397 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/13’ 2019-12-05T06:13:40.398 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/138/17’ 2019-12-05T06:13:40.398 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/138’ 2019-12-05T06:13:40.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/19’ 2019-12-05T06:13:40.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/15’ 2019-12-05T06:13:40.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/20’ 2019-12-05T06:13:40.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/12’ 2019-12-05T06:13:40.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/10’ 2019-12-05T06:13:40.399 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/11’ 2019-12-05T06:13:40.400 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/16’ 2019-12-05T06:13:40.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/14’ 2019-12-05T06:13:40.401 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/18’ 2019-12-05T06:13:40.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/13’ 2019-12-05T06:13:40.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/135/17’ 2019-12-05T06:13:40.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/135’ 2019-12-05T06:13:40.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/19’ 2019-12-05T06:13:40.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/15’ 2019-12-05T06:13:40.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/20’ 2019-12-05T06:13:40.402 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/12’ 2019-12-05T06:13:40.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/10’ 2019-12-05T06:13:40.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/11’ 2019-12-05T06:13:40.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/16’ 2019-12-05T06:13:40.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/14’ 2019-12-05T06:13:40.403 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/18’ 2019-12-05T06:13:42.336 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/13’ 2019-12-05T06:13:42.336 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/117/17’ 2019-12-05T06:13:42.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/117’ 2019-12-05T06:13:42.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/19’ 2019-12-05T06:13:42.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/15’ 2019-12-05T06:13:42.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/20’ 2019-12-05T06:13:42.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/12’ 2019-12-05T06:13:42.337 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/10’ 2019-12-05T06:13:42.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/11’ 2019-12-05T06:13:42.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/16’ 2019-12-05T06:13:42.339 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/14’ 2019-12-05T06:13:42.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/18’ 2019-12-05T06:13:42.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/13’ 2019-12-05T06:13:42.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/143/17’ 2019-12-05T06:13:42.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/143’ 2019-12-05T06:13:42.340 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/19’ 2019-12-05T06:13:42.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/15’ 2019-12-05T06:13:42.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/20’ 2019-12-05T06:13:42.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/12’ 2019-12-05T06:13:42.341 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/10’ 2019-12-05T06:13:42.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/11’ 2019-12-05T06:13:42.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/16’ 2019-12-05T06:13:42.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/14’ 2019-12-05T06:13:42.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/18’ 2019-12-05T06:13:42.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/13’ 2019-12-05T06:13:42.342 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/180/17’ 2019-12-05T06:13:42.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/180’ 2019-12-05T06:13:42.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/19’ 2019-12-05T06:13:42.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/15’ 2019-12-05T06:13:42.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/20’ 2019-12-05T06:13:42.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/12’ 2019-12-05T06:13:42.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/10’ 2019-12-05T06:13:42.343 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/11’ 2019-12-05T06:13:42.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/16’ 2019-12-05T06:13:42.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/14’ 2019-12-05T06:13:42.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/18’ 2019-12-05T06:13:42.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/13’ 2019-12-05T06:13:42.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/126/17’ 2019-12-05T06:13:42.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/126’ 2019-12-05T06:13:42.344 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/19’ 2019-12-05T06:13:42.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/15’ 2019-12-05T06:13:42.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/20’ 2019-12-05T06:13:42.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/12’ 2019-12-05T06:13:42.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/10’ 2019-12-05T06:13:42.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/11’ 2019-12-05T06:13:42.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/16’ 2019-12-05T06:13:42.345 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/14’ 2019-12-05T06:13:42.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/18’ 2019-12-05T06:13:42.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/13’ 2019-12-05T06:13:42.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/161/17’ 2019-12-05T06:13:42.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/161’ 2019-12-05T06:13:42.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/19’ 2019-12-05T06:13:42.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/15’ 2019-12-05T06:13:42.346 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/20’ 2019-12-05T06:13:42.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/12’ 2019-12-05T06:13:42.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/10’ 2019-12-05T06:13:42.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/11’ 2019-12-05T06:13:42.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/16’ 2019-12-05T06:13:42.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/14’ 2019-12-05T06:13:42.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/18’ 2019-12-05T06:13:42.347 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/13’ 2019-12-05T06:13:42.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/176/17’ 2019-12-05T06:13:42.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/176’ 2019-12-05T06:13:42.348 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/19’ 2019-12-05T06:13:42.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/15’ 2019-12-05T06:13:42.349 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/20’ 2019-12-05T06:13:42.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/12’ 2019-12-05T06:13:42.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/10’ 2019-12-05T06:13:42.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/11’ 2019-12-05T06:13:42.350 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/16’ 2019-12-05T06:13:42.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/14’ 2019-12-05T06:13:42.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/18’ 2019-12-05T06:13:42.351 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/13’ 2019-12-05T06:13:42.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/111/17’ 2019-12-05T06:13:42.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/111’ 2019-12-05T06:13:42.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/19’ 2019-12-05T06:13:42.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/15’ 2019-12-05T06:13:42.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/20’ 2019-12-05T06:13:42.352 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/12’ 2019-12-05T06:13:42.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/10’ 2019-12-05T06:13:42.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/11’ 2019-12-05T06:13:42.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/16’ 2019-12-05T06:13:42.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/14’ 2019-12-05T06:13:42.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/18’ 2019-12-05T06:13:42.353 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/13’ 2019-12-05T06:13:42.360 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/191/17’ 2019-12-05T06:13:42.360 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/191’ 2019-12-05T06:13:42.360 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/19’ 2019-12-05T06:13:42.360 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/15’ 2019-12-05T06:13:42.361 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/20’ 2019-12-05T06:13:42.361 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/12’ 2019-12-05T06:13:42.361 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/10’ 2019-12-05T06:13:42.361 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/11’ 2019-12-05T06:13:42.361 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/16’ 2019-12-05T06:13:42.361 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/14’ 2019-12-05T06:13:42.362 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/18’ 2019-12-05T06:13:42.362 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/13’ 2019-12-05T06:13:42.362 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/184/17’ 2019-12-05T06:13:44.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/184’ 2019-12-05T06:13:44.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/19’ 2019-12-05T06:13:44.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/15’ 2019-12-05T06:13:44.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/20’ 2019-12-05T06:13:44.015 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/12’ 2019-12-05T06:13:44.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/10’ 2019-12-05T06:13:44.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/11’ 2019-12-05T06:13:44.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/16’ 2019-12-05T06:13:44.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/14’ 2019-12-05T06:13:44.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/18’ 2019-12-05T06:13:44.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/13’ 2019-12-05T06:13:44.016 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/192/17’ 2019-12-05T06:13:44.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/192’ 2019-12-05T06:13:44.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/19’ 2019-12-05T06:13:44.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/15’ 2019-12-05T06:13:44.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/20’ 2019-12-05T06:13:44.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/12’ 2019-12-05T06:13:44.017 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/10’ 2019-12-05T06:13:44.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/11’ 2019-12-05T06:13:44.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/16’ 2019-12-05T06:13:44.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/14’ 2019-12-05T06:13:44.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/18’ 2019-12-05T06:13:44.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/13’ 2019-12-05T06:13:44.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/103/17’ 2019-12-05T06:13:44.018 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/103’ 2019-12-05T06:13:44.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/19’ 2019-12-05T06:13:44.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/15’ 2019-12-05T06:13:44.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/20’ 2019-12-05T06:13:44.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/12’ 2019-12-05T06:13:44.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/10’ 2019-12-05T06:13:44.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/11’ 2019-12-05T06:13:44.019 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/16’ 2019-12-05T06:13:44.020 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/14’ 2019-12-05T06:13:44.020 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/18’ 2019-12-05T06:13:44.020 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/13’ 2019-12-05T06:13:44.020 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/128/17’ 2019-12-05T06:13:44.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/128’ 2019-12-05T06:13:44.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/19’ 2019-12-05T06:13:44.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/15’ 2019-12-05T06:13:44.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/20’ 2019-12-05T06:13:44.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/12’ 2019-12-05T06:13:44.033 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/10’ 2019-12-05T06:13:44.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/11’ 2019-12-05T06:13:44.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/16’ 2019-12-05T06:13:44.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/14’ 2019-12-05T06:13:44.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/18’ 2019-12-05T06:13:44.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/13’ 2019-12-05T06:13:44.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/144/17’ 2019-12-05T06:13:44.034 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/144’ 2019-12-05T06:13:44.035 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/19’ 2019-12-05T06:13:44.035 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/15’ 2019-12-05T06:13:44.035 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/20’ 2019-12-05T06:13:44.035 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/12’ 2019-12-05T06:13:44.035 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/10’ 2019-12-05T06:13:44.035 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/11’ 2019-12-05T06:13:44.036 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/16’ 2019-12-05T06:13:44.036 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/14’ 2019-12-05T06:13:44.036 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/18’ 2019-12-05T06:13:44.036 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/13’ 2019-12-05T06:13:44.036 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/146/17’ 2019-12-05T06:13:44.036 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/146’ 2019-12-05T06:13:44.036 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/19’ 2019-12-05T06:13:44.037 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/15’ 2019-12-05T06:13:44.037 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/20’ 2019-12-05T06:13:44.037 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/12’ 2019-12-05T06:13:44.037 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/10’ 2019-12-05T06:13:44.037 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/11’ 2019-12-05T06:13:44.037 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/16’ 2019-12-05T06:13:44.037 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/14’ 2019-12-05T06:13:44.038 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/18’ 2019-12-05T06:13:44.038 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/13’ 2019-12-05T06:13:44.038 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/141/17’ 2019-12-05T06:13:44.038 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/141’ 2019-12-05T06:13:44.038 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/19’ 2019-12-05T06:13:44.038 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/15’ 2019-12-05T06:13:44.039 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/20’ 2019-12-05T06:13:44.049 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/12’ 2019-12-05T06:13:44.049 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/10’ 2019-12-05T06:13:44.049 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/11’ 2019-12-05T06:13:44.049 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/16’ 2019-12-05T06:13:44.050 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/14’ 2019-12-05T06:13:44.050 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/18’ 2019-12-05T06:13:44.050 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/13’ 2019-12-05T06:13:44.050 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/182/17’ 2019-12-05T06:13:44.050 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/182’ 2019-12-05T06:13:44.050 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/19’ 2019-12-05T06:13:44.051 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/15’ 2019-12-05T06:13:44.051 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/20’ 2019-12-05T06:13:44.051 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/12’ 2019-12-05T06:13:44.051 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/10’ 2019-12-05T06:13:44.051 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/11’ 2019-12-05T06:13:44.051 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/16’ 2019-12-05T06:13:44.052 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/14’ 2019-12-05T06:13:44.052 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/18’ 2019-12-05T06:13:44.052 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/13’ 2019-12-05T06:13:44.052 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/107/17’ 2019-12-05T06:13:44.052 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/107’ 2019-12-05T06:13:44.052 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/19’ 2019-12-05T06:13:47.743 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/15’ 2019-12-05T06:13:47.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/20’ 2019-12-05T06:13:47.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/12’ 2019-12-05T06:13:47.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/10’ 2019-12-05T06:13:47.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/11’ 2019-12-05T06:13:47.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/16’ 2019-12-05T06:13:47.744 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/14’ 2019-12-05T06:13:47.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/18’ 2019-12-05T06:13:47.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/13’ 2019-12-05T06:13:47.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/178/17’ 2019-12-05T06:13:47.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/178’ 2019-12-05T06:13:47.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/19’ 2019-12-05T06:13:47.745 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/15’ 2019-12-05T06:13:47.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/20’ 2019-12-05T06:13:47.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/12’ 2019-12-05T06:13:47.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/10’ 2019-12-05T06:13:47.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/11’ 2019-12-05T06:13:47.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/16’ 2019-12-05T06:13:47.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/14’ 2019-12-05T06:13:47.746 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/18’ 2019-12-05T06:13:47.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/13’ 2019-12-05T06:13:47.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/156/17’ 2019-12-05T06:13:47.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/156’ 2019-12-05T06:13:47.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/19’ 2019-12-05T06:13:47.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/15’ 2019-12-05T06:13:47.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/20’ 2019-12-05T06:13:47.747 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/12’ 2019-12-05T06:13:47.748 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/10’ 2019-12-05T06:13:47.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/11’ 2019-12-05T06:13:47.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/16’ 2019-12-05T06:13:47.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/14’ 2019-12-05T06:13:47.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/18’ 2019-12-05T06:13:47.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/13’ 2019-12-05T06:13:47.749 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/131/17’ 2019-12-05T06:13:47.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/131’ 2019-12-05T06:13:47.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/19’ 2019-12-05T06:13:47.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/15’ 2019-12-05T06:13:47.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/20’ 2019-12-05T06:13:47.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/12’ 2019-12-05T06:13:47.750 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/10’ 2019-12-05T06:13:47.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/11’ 2019-12-05T06:13:47.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/16’ 2019-12-05T06:13:47.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/14’ 2019-12-05T06:13:47.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/18’ 2019-12-05T06:13:47.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/13’ 2019-12-05T06:13:47.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/123/17’ 2019-12-05T06:13:47.751 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/123’ 2019-12-05T06:13:47.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/19’ 2019-12-05T06:13:47.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/15’ 2019-12-05T06:13:47.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/20’ 2019-12-05T06:13:47.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/12’ 2019-12-05T06:13:47.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/10’ 2019-12-05T06:13:47.752 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/11’ 2019-12-05T06:13:47.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/16’ 2019-12-05T06:13:47.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/14’ 2019-12-05T06:13:47.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/18’ 2019-12-05T06:13:47.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/13’ 2019-12-05T06:13:47.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/160/17’ 2019-12-05T06:13:47.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/160’ 2019-12-05T06:13:47.753 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/19’ 2019-12-05T06:13:47.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/15’ 2019-12-05T06:13:47.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/20’ 2019-12-05T06:13:47.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/12’ 2019-12-05T06:13:47.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/10’ 2019-12-05T06:13:47.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/11’ 2019-12-05T06:13:47.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/16’ 2019-12-05T06:13:47.754 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/14’ 2019-12-05T06:13:47.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/18’ 2019-12-05T06:13:47.755 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/13’ 2019-12-05T06:13:47.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/165/17’ 2019-12-05T06:13:47.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/165’ 2019-12-05T06:13:47.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/19’ 2019-12-05T06:13:47.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/15’ 2019-12-05T06:13:47.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/20’ 2019-12-05T06:13:47.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/12’ 2019-12-05T06:13:47.756 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/10’ 2019-12-05T06:13:47.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/11’ 2019-12-05T06:13:47.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/16’ 2019-12-05T06:13:47.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/14’ 2019-12-05T06:13:47.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/18’ 2019-12-05T06:13:47.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/13’ 2019-12-05T06:13:47.757 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/101/17’ 2019-12-05T06:13:47.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/101’ 2019-12-05T06:13:47.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/19’ 2019-12-05T06:13:47.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/15’ 2019-12-05T06:13:47.758 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/20’ 2019-12-05T06:13:47.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/12’ 2019-12-05T06:13:47.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/10’ 2019-12-05T06:13:47.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/11’ 2019-12-05T06:13:47.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/16’ 2019-12-05T06:13:47.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/14’ 2019-12-05T06:13:47.759 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/18’ 2019-12-05T06:13:47.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/13’ 2019-12-05T06:13:47.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/148/17’ 2019-12-05T06:13:47.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/148’ 2019-12-05T06:13:47.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/19’ 2019-12-05T06:13:47.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/15’ 2019-12-05T06:13:47.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/20’ 2019-12-05T06:13:47.760 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/12’ 2019-12-05T06:13:48.261 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/10’ 2019-12-05T06:13:48.262 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/11’ 2019-12-05T06:13:48.262 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/16’ 2019-12-05T06:13:48.262 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/14’ 2019-12-05T06:13:48.262 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/18’ 2019-12-05T06:13:48.262 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/13’ 2019-12-05T06:13:48.262 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/168/17’ 2019-12-05T06:13:48.263 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/168’ 2019-12-05T06:13:48.263 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/19’ 2019-12-05T06:13:48.263 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/15’ 2019-12-05T06:13:48.263 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/20’ 2019-12-05T06:13:48.263 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/12’ 2019-12-05T06:13:48.269 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/10’ 2019-12-05T06:13:48.269 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/11’ 2019-12-05T06:13:48.269 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/16’ 2019-12-05T06:13:48.269 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/14’ 2019-12-05T06:13:48.269 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/18’ 2019-12-05T06:13:48.270 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/13’ 2019-12-05T06:13:48.270 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/179/17’ 2019-12-05T06:13:48.270 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/179’ 2019-12-05T06:13:48.270 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/19’ 2019-12-05T06:13:48.270 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/15’ 2019-12-05T06:13:48.270 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/20’ 2019-12-05T06:13:48.271 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/12’ 2019-12-05T06:13:48.271 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/10’ 2019-12-05T06:13:48.271 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/11’ 2019-12-05T06:13:48.271 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/16’ 2019-12-05T06:13:48.271 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/14’ 2019-12-05T06:13:48.271 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/18’ 2019-12-05T06:13:48.272 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/13’ 2019-12-05T06:13:48.272 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/132/17’ 2019-12-05T06:13:48.272 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/132’ 2019-12-05T06:13:48.285 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/19’ 2019-12-05T06:13:48.285 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/15’ 2019-12-05T06:13:48.285 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/20’ 2019-12-05T06:13:48.286 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/12’ 2019-12-05T06:13:48.286 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/10’ 2019-12-05T06:13:48.286 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/11’ 2019-12-05T06:13:48.286 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/16’ 2019-12-05T06:13:48.286 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/14’ 2019-12-05T06:13:48.287 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/18’ 2019-12-05T06:13:48.287 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/13’ 2019-12-05T06:13:48.287 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/199/17’ 2019-12-05T06:13:48.287 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/199’ 2019-12-05T06:13:48.287 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/19’ 2019-12-05T06:13:48.287 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/15’ 2019-12-05T06:13:48.288 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/20’ 2019-12-05T06:13:48.288 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/12’ 2019-12-05T06:13:48.288 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/10’ 2019-12-05T06:13:48.288 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/11’ 2019-12-05T06:13:48.288 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/16’ 2019-12-05T06:13:48.288 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/14’ 2019-12-05T06:13:48.289 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/18’ 2019-12-05T06:13:48.289 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/13’ 2019-12-05T06:13:48.289 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/163/17’ 2019-12-05T06:13:48.289 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/163’ 2019-12-05T06:13:48.289 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/19’ 2019-12-05T06:13:48.295 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/15’ 2019-12-05T06:13:48.296 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/20’ 2019-12-05T06:13:48.296 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/12’ 2019-12-05T06:13:48.296 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/10’ 2019-12-05T06:13:48.296 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/11’ 2019-12-05T06:13:48.296 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/16’ 2019-12-05T06:13:48.296 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/14’ 2019-12-05T06:13:48.297 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/18’ 2019-12-05T06:13:48.297 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/13’ 2019-12-05T06:13:48.297 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/153/17’ 2019-12-05T06:13:48.297 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/153’ 2019-12-05T06:13:48.297 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/19’ 2019-12-05T06:13:48.297 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/15’ 2019-12-05T06:13:48.298 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/20’ 2019-12-05T06:13:48.298 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/12’ 2019-12-05T06:13:48.298 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/10’ 2019-12-05T06:13:48.298 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/11’ 2019-12-05T06:13:48.298 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/16’ 2019-12-05T06:13:48.298 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/14’ 2019-12-05T06:13:48.299 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/18’ 2019-12-05T06:13:48.299 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/13’ 2019-12-05T06:13:48.299 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/122/17’ 2019-12-05T06:13:48.299 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/122’ 2019-12-05T06:13:48.299 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/19’ 2019-12-05T06:13:48.299 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/15’ 2019-12-05T06:13:48.300 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/20’ 2019-12-05T06:13:48.300 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/12’ 2019-12-05T06:13:48.300 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/10’ 2019-12-05T06:13:48.301 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/11’ 2019-12-05T06:13:48.301 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/16’ 2019-12-05T06:13:48.301 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/14’ 2019-12-05T06:13:48.301 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/18’ 2019-12-05T06:13:48.302 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/13’ 2019-12-05T06:13:48.302 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/162/17’ 2019-12-05T06:13:48.302 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/162’ 2019-12-05T06:13:48.302 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/19’ 2019-12-05T06:13:48.302 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/15’ 2019-12-05T06:13:48.302 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/20’ 2019-12-05T06:13:48.303 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/12’ 2019-12-05T06:13:48.303 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/10’ 2019-12-05T06:13:48.303 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/11’ 2019-12-05T06:13:49.075 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/16’ 2019-12-05T06:13:49.075 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/14’ 2019-12-05T06:13:49.075 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/18’ 2019-12-05T06:13:49.075 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/13’ 2019-12-05T06:13:49.075 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/197/17’ 2019-12-05T06:13:49.075 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/197’ 2019-12-05T06:13:49.076 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/19’ 2019-12-05T06:13:49.076 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/15’ 2019-12-05T06:13:49.076 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/20’ 2019-12-05T06:13:49.076 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/12’ 2019-12-05T06:13:49.076 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/10’ 2019-12-05T06:13:49.076 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/11’ 2019-12-05T06:13:49.077 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/16’ 2019-12-05T06:13:49.077 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/14’ 2019-12-05T06:13:49.077 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/18’ 2019-12-05T06:13:49.077 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/13’ 2019-12-05T06:13:49.077 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/113/17’ 2019-12-05T06:13:49.077 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/113’ 2019-12-05T06:13:49.077 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/19’ 2019-12-05T06:13:49.078 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/15’ 2019-12-05T06:13:49.078 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/20’ 2019-12-05T06:13:49.078 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/12’ 2019-12-05T06:13:49.078 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/10’ 2019-12-05T06:13:49.078 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/11’ 2019-12-05T06:13:49.078 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/16’ 2019-12-05T06:13:49.079 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/14’ 2019-12-05T06:13:49.079 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/18’ 2019-12-05T06:13:49.079 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/13’ 2019-12-05T06:13:49.079 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/152/17’ 2019-12-05T06:13:49.079 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/152’ 2019-12-05T06:13:49.079 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/19’ 2019-12-05T06:13:49.079 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/15’ 2019-12-05T06:13:49.080 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/20’ 2019-12-05T06:13:49.080 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/12’ 2019-12-05T06:13:49.080 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/10’ 2019-12-05T06:13:49.080 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/11’ 2019-12-05T06:13:49.080 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/16’ 2019-12-05T06:13:49.080 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/14’ 2019-12-05T06:13:49.080 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/18’ 2019-12-05T06:13:49.081 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/13’ 2019-12-05T06:13:49.081 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/121/17’ 2019-12-05T06:13:49.081 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/121’ 2019-12-05T06:13:49.081 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/19’ 2019-12-05T06:13:49.081 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/15’ 2019-12-05T06:13:49.088 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/20’ 2019-12-05T06:13:49.088 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/12’ 2019-12-05T06:13:49.088 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/10’ 2019-12-05T06:13:49.088 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/11’ 2019-12-05T06:13:49.088 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/16’ 2019-12-05T06:13:49.089 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/14’ 2019-12-05T06:13:49.089 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/18’ 2019-12-05T06:13:49.089 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/13’ 2019-12-05T06:13:49.089 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/118/17’ 2019-12-05T06:13:49.089 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/118’ 2019-12-05T06:13:49.089 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/19’ 2019-12-05T06:13:49.090 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/15’ 2019-12-05T06:13:49.090 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/20’ 2019-12-05T06:13:49.090 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/12’ 2019-12-05T06:13:49.090 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/10’ 2019-12-05T06:13:49.090 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/11’ 2019-12-05T06:13:49.090 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/16’ 2019-12-05T06:13:49.090 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/14’ 2019-12-05T06:13:49.091 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/18’ 2019-12-05T06:13:49.091 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/13’ 2019-12-05T06:13:49.091 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/196/17’ 2019-12-05T06:13:49.091 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/196’ 2019-12-05T06:13:49.091 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/19’ 2019-12-05T06:13:49.091 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/15’ 2019-12-05T06:13:49.092 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/20’ 2019-12-05T06:13:49.092 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/12’ 2019-12-05T06:13:49.092 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/10’ 2019-12-05T06:13:49.092 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/11’ 2019-12-05T06:13:49.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/16’ 2019-12-05T06:13:49.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/14’ 2019-12-05T06:13:49.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/18’ 2019-12-05T06:13:49.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/13’ 2019-12-05T06:13:49.102 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/124/17’ 2019-12-05T06:13:49.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/124’ 2019-12-05T06:13:49.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/19’ 2019-12-05T06:13:49.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/15’ 2019-12-05T06:13:49.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/20’ 2019-12-05T06:13:49.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/12’ 2019-12-05T06:13:49.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/10’ 2019-12-05T06:13:49.103 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/11’ 2019-12-05T06:13:49.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/16’ 2019-12-05T06:13:49.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/14’ 2019-12-05T06:13:49.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/18’ 2019-12-05T06:13:49.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/13’ 2019-12-05T06:13:49.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/188/17’ 2019-12-05T06:13:49.104 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/188’ 2019-12-05T06:13:49.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/19’ 2019-12-05T06:13:49.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/15’ 2019-12-05T06:13:49.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/20’ 2019-12-05T06:13:49.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/12’ 2019-12-05T06:13:49.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/10’ 2019-12-05T06:13:49.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/11’ 2019-12-05T06:13:49.105 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/16’ 2019-12-05T06:13:49.106 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/14’ 2019-12-05T06:13:50.656 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/18’ 2019-12-05T06:13:50.656 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/13’ 2019-12-05T06:13:50.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/145/17’ 2019-12-05T06:13:50.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/145’ 2019-12-05T06:13:50.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/19’ 2019-12-05T06:13:50.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/15’ 2019-12-05T06:13:50.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/20’ 2019-12-05T06:13:50.657 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/12’ 2019-12-05T06:13:50.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/10’ 2019-12-05T06:13:50.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/11’ 2019-12-05T06:13:50.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/16’ 2019-12-05T06:13:50.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/14’ 2019-12-05T06:13:50.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/18’ 2019-12-05T06:13:50.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/13’ 2019-12-05T06:13:50.658 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/130/17’ 2019-12-05T06:13:50.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/130’ 2019-12-05T06:13:50.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/19’ 2019-12-05T06:13:50.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/15’ 2019-12-05T06:13:50.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/20’ 2019-12-05T06:13:50.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/12’ 2019-12-05T06:13:50.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/10’ 2019-12-05T06:13:50.659 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/11’ 2019-12-05T06:13:50.660 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/16’ 2019-12-05T06:13:50.660 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/14’ 2019-12-05T06:13:50.660 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/18’ 2019-12-05T06:13:50.660 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/13’ 2019-12-05T06:13:50.660 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/193/17’ 2019-12-05T06:13:50.660 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/193’ 2019-12-05T06:13:50.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/19’ 2019-12-05T06:13:50.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/15’ 2019-12-05T06:13:50.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/20’ 2019-12-05T06:13:50.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/12’ 2019-12-05T06:13:50.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/10’ 2019-12-05T06:13:50.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/11’ 2019-12-05T06:13:50.661 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/16’ 2019-12-05T06:13:50.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/14’ 2019-12-05T06:13:50.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/18’ 2019-12-05T06:13:50.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/13’ 2019-12-05T06:13:50.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/175/17’ 2019-12-05T06:13:50.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/175’ 2019-12-05T06:13:50.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/19’ 2019-12-05T06:13:50.662 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/15’ 2019-12-05T06:13:50.663 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/20’ 2019-12-05T06:13:50.663 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/12’ 2019-12-05T06:13:50.663 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/10’ 2019-12-05T06:13:50.663 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/11’ 2019-12-05T06:13:50.663 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/16’ 2019-12-05T06:13:50.663 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/14’ 2019-12-05T06:13:50.664 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/18’ 2019-12-05T06:13:50.665 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/13’ 2019-12-05T06:13:50.665 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/181/17’ 2019-12-05T06:13:50.665 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/181’ 2019-12-05T06:13:50.665 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/19’ 2019-12-05T06:13:50.666 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/15’ 2019-12-05T06:13:50.666 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/20’ 2019-12-05T06:13:50.666 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/12’ 2019-12-05T06:13:50.666 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/10’ 2019-12-05T06:13:50.666 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/11’ 2019-12-05T06:13:50.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/16’ 2019-12-05T06:13:50.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/14’ 2019-12-05T06:13:50.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/18’ 2019-12-05T06:13:50.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/13’ 2019-12-05T06:13:50.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/127/17’ 2019-12-05T06:13:50.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/127’ 2019-12-05T06:13:50.667 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/19’ 2019-12-05T06:13:50.668 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/15’ 2019-12-05T06:13:50.668 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/20’ 2019-12-05T06:13:50.668 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/12’ 2019-12-05T06:13:50.668 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/10’ 2019-12-05T06:13:50.668 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/11’ 2019-12-05T06:13:50.669 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/16’ 2019-12-05T06:13:50.669 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/14’ 2019-12-05T06:13:50.669 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/18’ 2019-12-05T06:13:50.669 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/13’ 2019-12-05T06:13:50.669 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/108/17’ 2019-12-05T06:13:50.669 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/108’ 2019-12-05T06:13:50.669 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/19’ 2019-12-05T06:13:50.670 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/15’ 2019-12-05T06:13:50.670 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/20’ 2019-12-05T06:13:50.670 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/12’ 2019-12-05T06:13:50.670 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/10’ 2019-12-05T06:13:50.670 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/11’ 2019-12-05T06:13:50.670 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/16’ 2019-12-05T06:13:50.670 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/14’ 2019-12-05T06:13:50.671 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/18’ 2019-12-05T06:13:50.671 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/13’ 2019-12-05T06:13:50.671 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/183/17’ 2019-12-05T06:13:50.672 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/183’ 2019-12-05T06:13:50.672 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/19’ 2019-12-05T06:13:50.672 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/15’ 2019-12-05T06:13:50.672 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/20’ 2019-12-05T06:13:50.672 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/12’ 2019-12-05T06:13:50.672 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/10’ 2019-12-05T06:13:50.673 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/11’ 2019-12-05T06:13:50.673 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/16’ 2019-12-05T06:13:50.673 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/14’ 2019-12-05T06:13:50.673 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/18’ 2019-12-05T06:13:50.673 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/13’ 2019-12-05T06:13:50.673 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/177/17’ 2019-12-05T06:13:52.174 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:52.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.460+0000 7f9991e37700 1 Processor -- start 2019-12-05T06:13:52.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.460+0000 7f9991e37700 1 -- start start 2019-12-05T06:13:52.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.462+0000 7f9991e37700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 0x7f998c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:52.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.462+0000 7f9991e37700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998c0958e0 con 0x7f998c08d2c0 2019-12-05T06:13:52.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.462+0000 7f9991e37700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998c0629e0 con 0x7f998c08c6d0 2019-12-05T06:13:52.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.462+0000 7f9991e37700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c08df90 con 0x7f998c08fef0 2019-12-05T06:13:52.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.462+0000 7f998b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f998c08d2c0 0x7f998c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34784/0 (socket says 172.21.15.16:34784) 2019-12-05T06:13:52.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.462+0000 7f998b7fe700 1 -- 172.21.15.16:0/1062049992 learned_addr learned my addr 172.21.15.16:0/1062049992 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:13:52.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.462+0000 7f9990bd5700 1 --2- 172.21.15.16:0/1062049992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 0x7f998c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:52.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.463+0000 7f998affd700 1 -- 172.21.15.16:0/1062049992 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1936487021 0 0) 0x7f9974001150 con 0x7f998c08d2c0 2019-12-05T06:13:52.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.463+0000 7f998affd700 1 -- 172.21.15.16:0/1062049992 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9978002e80 con 0x7f998c08d2c0 2019-12-05T06:13:52.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.463+0000 7f9990bd5700 1 -- 172.21.15.16:0/1062049992 >> v1:172.21.15.76:6789/0 conn(0x7f998c08c6d0 legacy=0x7f998c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:52.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.463+0000 7f9990bd5700 1 -- 172.21.15.16:0/1062049992 >> v1:172.21.15.16:6789/0 conn(0x7f998c08d2c0 legacy=0x7f998c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:52.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.463+0000 7f9990bd5700 1 -- 172.21.15.16:0/1062049992 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c08e620 con 0x7f998c08fef0 2019-12-05T06:13:52.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.464+0000 7f9990bd5700 1 --2- 172.21.15.16:0/1062049992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 0x7f998c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f998400af80 tx=0x7f998400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.464+0000 7f9990bd5700 1 --2- 172.21.15.16:0/1062049992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 0x7f998c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f998400af80 tx=0x7f998400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.464+0000 7f9990bd5700 1 --2- 172.21.15.16:0/1062049992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 0x7f998c08f720 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f998400af80 tx=0x7f998400b790).ready entity=mon.2 client_cookie=4a0377b2ab42439d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:52.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.464+0000 7f9990bd5700 1 --2- 172.21.15.16:0/1062049992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 0x7f998c08f720 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f998400af80 tx=0x7f998400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.464+0000 7f998affd700 1 -- 172.21.15.16:0/1062049992 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9984007240 con 0x7f998c08fef0 2019-12-05T06:13:52.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.464+0000 7f9990bd5700 1 --2- 172.21.15.16:0/1062049992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 0x7f998c08f720 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f998400af80 tx=0x7f998400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.464+0000 7f9990bd5700 1 --2- 172.21.15.16:0/1062049992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 0x7f998c08f720 secure :-1 s=THROTTLE_DONE pgs=100 cs=0 l=1 rx=0x7f998400af80 tx=0x7f998400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.464+0000 7f998affd700 1 -- 172.21.15.16:0/1062049992 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9984000bb0 con 0x7f998c08fef0 2019-12-05T06:13:52.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.464+0000 7f998affd700 1 -- 172.21.15.16:0/1062049992 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9984009e60 con 0x7f998c08fef0 2019-12-05T06:13:52.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.465+0000 7f9991e37700 1 -- 172.21.15.16:0/1062049992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 msgr2=0x7f998c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:52.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.465+0000 7f9991e37700 1 --2- 172.21.15.16:0/1062049992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 0x7f998c08f720 secure :-1 s=READY pgs=100 cs=0 l=1 rx=0x7f998400af80 tx=0x7f998400b790).stop 2019-12-05T06:13:52.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.466+0000 7f9991e37700 1 -- 172.21.15.16:0/1062049992 shutdown_connections 2019-12-05T06:13:52.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.466+0000 7f9991e37700 1 --2- 172.21.15.16:0/1062049992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c08fef0 0x7f998c08f720 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:52.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.466+0000 7f9991e37700 1 -- 172.21.15.16:0/1062049992 shutdown_connections 2019-12-05T06:13:52.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.466+0000 7f9991e37700 1 -- 172.21.15.16:0/1062049992 wait complete. 2019-12-05T06:13:52.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.466+0000 7f9991e37700 1 -- 172.21.15.16:0/1062049992 >> 172.21.15.16:0/1062049992 conn(0x7f998c061940 msgr2=0x7f998c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:52.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.472+0000 7f9991e37700 1 Processor -- start 2019-12-05T06:13:52.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.472+0000 7f9991e37700 1 -- start start 2019-12-05T06:13:52.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.474+0000 7f9991e37700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f998c12b4c0 0x7f998c12acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:52.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.474+0000 7f9991e37700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c19f440 0x7f998c1a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:52.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.474+0000 7f9991e37700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f998c06f6d0 con 0x7f998c06d550 2019-12-05T06:13:52.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.474+0000 7f9991e37700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f998c129560 con 0x7f998c12b4c0 2019-12-05T06:13:52.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.474+0000 7f9991e37700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f998c128b30 con 0x7f998c19f440 2019-12-05T06:13:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.474+0000 7f998b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f998c12b4c0 0x7f998c12acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.474+0000 7f998b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f998c12b4c0 0x7f998c12acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49624/0 (socket says 172.21.15.16:49624) 2019-12-05T06:13:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.474+0000 7f998b7fe700 1 -- 172.21.15.16:0/3370525577 learned_addr learned my addr 172.21.15.16:0/3370525577 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:13:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.474+0000 7f9990bd5700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c19f440 0x7f998c1a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.475+0000 7f998b7fe700 1 -- 172.21.15.16:0/3370525577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c19f440 msgr2=0x7f998c1a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.475+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c19f440 0x7f998c1a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.475+0000 7f9988ff9700 1 -- 172.21.15.16:0/3370525577 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361312918 0 0) 0x7f997c001400 con 0x7f998c06d550 2019-12-05T06:13:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.475+0000 7f998b7fe700 1 -- 172.21.15.16:0/3370525577 >> v1:172.21.15.76:6789/0 conn(0x7f998c06d550 legacy=0x7f998c12a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.475+0000 7f998b7fe700 1 -- 172.21.15.16:0/3370525577 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f998c129bf0 con 0x7f998c12b4c0 2019-12-05T06:13:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.475+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> v2:172.21.15.16:3300/0 conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.475+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> v2:172.21.15.16:3300/0 conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.475+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=READY pgs=360 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.475+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.476+0000 7f9988ff9700 1 -- 172.21.15.16:0/3370525577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9974000c90 con 0x7f998c12b4c0 2019-12-05T06:13:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.476+0000 7f9991e37700 1 -- 172.21.15.16:0/3370525577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f998c129bf0 con 0x7f998c12b4c0 2019-12-05T06:13:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.476+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.476+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.476+0000 7f9988ff9700 1 -- 172.21.15.16:0/3370525577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9974000c90 con 0x7f998c12b4c0 2019-12-05T06:13:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.476+0000 7f9988ff9700 1 -- 172.21.15.16:0/3370525577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9974000e80 con 0x7f998c12b4c0 2019-12-05T06:13:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.476+0000 7f9991e37700 1 -- 172.21.15.16:0/3370525577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f998c129bf0 con 0x7f998c12b4c0 2019-12-05T06:13:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.477+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.477+0000 7f9988ff9700 1 -- 172.21.15.16:0/3370525577 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9974000e80 con 0x7f998c12b4c0 2019-12-05T06:13:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.477+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.478+0000 7f9988ff9700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9970023630 0x7f9970025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.478+0000 7f9988ff9700 1 -- 172.21.15.16:0/3370525577 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f9974045da0 con 0x7f998c12b4c0 2019-12-05T06:13:52.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.478+0000 7f998bfff700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9970023630 0x7f9970025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.479+0000 7f998bfff700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9970023630 0x7f9970025ac0 crc :-1 s=READY pgs=165 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:52.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.490+0000 7f9991e37700 1 -- 172.21.15.16:0/3370525577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f998c129bf0 con 0x7f998c12b4c0 2019-12-05T06:13:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.497+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.498+0000 7f9988ff9700 1 -- 172.21.15.16:0/3370525577 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f99740075d0 con 0x7f998c12b4c0 2019-12-05T06:13:52.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.761+0000 7f9991e37700 1 -- 172.21.15.16:0/3370525577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f998c129bf0 con 0x7f998c12b4c0 2019-12-05T06:13:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.762+0000 7f998b7fe700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=THROTTLE_DONE pgs=360 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.762+0000 7f9988ff9700 1 -- 172.21.15.16:0/3370525577 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f9974000e50 con 0x7f998c12b4c0 2019-12-05T06:13:52.757 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2019-12-05T06:13:52.757 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:13:52.757 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5049,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:52.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.766+0000 7f9991e37700 1 -- 172.21.15.16:0/3370525577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9970023630 msgr2=0x7f9970025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:52.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.766+0000 7f9991e37700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9970023630 0x7f9970025ac0 crc :-1 s=READY pgs=165 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:52.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.766+0000 7f9991e37700 1 -- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 msgr2=0x7f998c12acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:52.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.766+0000 7f9991e37700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 0x7f998c12acf0 secure :-1 s=READY pgs=360 cs=0 l=1 rx=0x7f99740103e0 tx=0x7f9974010260).stop 2019-12-05T06:13:52.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.766+0000 7f9991e37700 1 -- 172.21.15.16:0/3370525577 shutdown_connections 2019-12-05T06:13:52.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.766+0000 7f9991e37700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f998c19f440 0x7f998c1a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:52.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.766+0000 7f9991e37700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9970023630 0x7f9970025ac0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:52.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.766+0000 7f9991e37700 1 --2- 172.21.15.16:0/3370525577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f998c12b4c0 0x7f998c12acf0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:52.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.767+0000 7f9991e37700 1 -- 172.21.15.16:0/3370525577 shutdown_connections 2019-12-05T06:13:52.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.767+0000 7f9991e37700 1 -- 172.21.15.16:0/3370525577 wait complete. 2019-12-05T06:13:52.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.767+0000 7f9991e37700 1 -- 172.21.15.16:0/3370525577 >> 172.21.15.16:0/3370525577 conn(0x7f998c0829c0 msgr2=0x7f998c09ad20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:52.778 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:13:52.778 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:13:52.778 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:13:52.789 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:13:52.790 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:52.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.981+0000 7ff08e620700 1 Processor -- start 2019-12-05T06:13:52.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.981+0000 7ff08e620700 1 -- start start 2019-12-05T06:13:52.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff08e620700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff08808c6d0 0x7ff08808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:52.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff08e620700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff08808f720 0x7ff08808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:52.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff08e620700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0880958e0 con 0x7ff08808fef0 2019-12-05T06:13:52.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff08e620700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0880a3f70 con 0x7ff08808f720 2019-12-05T06:13:52.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff08e620700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff08808d830 con 0x7ff08808c6d0 2019-12-05T06:13:52.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff07ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff08808fef0 0x7ff08808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34804/0 (socket says 172.21.15.16:34804) 2019-12-05T06:13:52.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff07ffff700 1 -- 172.21.15.16:0/2230117834 learned_addr learned my addr 172.21.15.16:0/2230117834 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:13:52.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff08d3be700 1 --2- 172.21.15.16:0/2230117834 >> v2:172.21.15.76:3300/0 conn(0x7ff08808f720 0x7ff08808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:52.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff08808c6d0 0x7ff08808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:52.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff07f7fe700 1 -- 172.21.15.16:0/2230117834 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779868831 0 0) 0x7ff084001150 con 0x7ff08808fef0 2019-12-05T06:13:52.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.983+0000 7ff07f7fe700 1 -- 172.21.15.16:0/2230117834 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff070002e80 con 0x7ff08808fef0 2019-12-05T06:13:52.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08d3be700 1 -- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff08808c6d0 msgr2=0x7ff08808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:52.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08d3be700 1 --2- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff08808c6d0 0x7ff08808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:52.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08d3be700 1 -- 172.21.15.16:0/2230117834 >> v1:172.21.15.16:6789/0 conn(0x7ff08808fef0 legacy=0x7ff08808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:52.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08d3be700 1 -- 172.21.15.16:0/2230117834 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff08808fbc0 con 0x7ff08808f720 2019-12-05T06:13:52.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff08808c6d0 0x7ff08808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:13:52.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08d3be700 1 --2- 172.21.15.16:0/2230117834 >> v2:172.21.15.76:3300/0 conn(0x7ff08808f720 0x7ff08808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff08000b300 tx=0x7ff08000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08d3be700 1 --2- 172.21.15.16:0/2230117834 >> v2:172.21.15.76:3300/0 conn(0x7ff08808f720 0x7ff08808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff08000b300 tx=0x7ff08000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08d3be700 1 --2- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08808f720 0x7ff08808d000 secure :-1 s=READY pgs=280 cs=0 l=1 rx=0x7ff08000b300 tx=0x7ff08000bb10).ready entity=mon.1 client_cookie=7d3d97d7b8833526 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08d3be700 1 --2- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08808f720 0x7ff08808d000 secure :-1 s=THROTTLE_DONE pgs=280 cs=0 l=1 rx=0x7ff08000b300 tx=0x7ff08000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff07f7fe700 1 -- 172.21.15.16:0/2230117834 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff080000f40 con 0x7ff08808f720 2019-12-05T06:13:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08d3be700 1 --2- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08808f720 0x7ff08808d000 secure :-1 s=THROTTLE_DONE pgs=280 cs=0 l=1 rx=0x7ff08000b300 tx=0x7ff08000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.984+0000 7ff08d3be700 1 --2- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08808f720 0x7ff08808d000 secure :-1 s=THROTTLE_DONE pgs=280 cs=0 l=1 rx=0x7ff08000b300 tx=0x7ff08000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.985+0000 7ff07f7fe700 1 -- 172.21.15.16:0/2230117834 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff080009b70 con 0x7ff08808f720 2019-12-05T06:13:52.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.985+0000 7ff07f7fe700 1 -- 172.21.15.16:0/2230117834 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff08000a240 con 0x7ff08808f720 2019-12-05T06:13:52.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.985+0000 7ff08e620700 1 -- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08808f720 msgr2=0x7ff08808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:52.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.985+0000 7ff08e620700 1 --2- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08808f720 0x7ff08808d000 secure :-1 s=READY pgs=280 cs=0 l=1 rx=0x7ff08000b300 tx=0x7ff08000bb10).stop 2019-12-05T06:13:52.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.986+0000 7ff08e620700 1 -- 172.21.15.16:0/2230117834 shutdown_connections 2019-12-05T06:13:52.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.986+0000 7ff08e620700 1 --2- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08808f720 0x7ff08808d000 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:52.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.986+0000 7ff08e620700 1 --2- 172.21.15.16:0/2230117834 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff08808c6d0 0x7ff08808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:52.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.986+0000 7ff08e620700 1 -- 172.21.15.16:0/2230117834 shutdown_connections 2019-12-05T06:13:52.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.986+0000 7ff08e620700 1 -- 172.21.15.16:0/2230117834 wait complete. 2019-12-05T06:13:52.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.986+0000 7ff08e620700 1 -- 172.21.15.16:0/2230117834 >> 172.21.15.16:0/2230117834 conn(0x7ff088061940 msgr2=0x7ff088096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:52.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.991+0000 7ff08e620700 1 Processor -- start 2019-12-05T06:13:52.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.991+0000 7ff08e620700 1 -- start start 2019-12-05T06:13:52.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.993+0000 7ff08e620700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff08806dc50 0x7ff08806e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:52.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.993+0000 7ff08e620700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff088134c80 0x7ff08806e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:52.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.993+0000 7ff08e620700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0881350d0 con 0x7ff088135450 2019-12-05T06:13:52.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.993+0000 7ff08e620700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff088132830 con 0x7ff08806dc50 2019-12-05T06:13:52.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.993+0000 7ff08e620700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0881334f0 con 0x7ff088134c80 2019-12-05T06:13:52.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.993+0000 7ff07ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff088135450 0x7ff088134480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34812/0 (socket says 172.21.15.16:34812) 2019-12-05T06:13:52.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.993+0000 7ff07ffff700 1 -- 172.21.15.16:0/2487601665 learned_addr learned my addr 172.21.15.16:0/2487601665 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:13:52.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> v2:172.21.15.76:3300/0 conn(0x7ff08806dc50 0x7ff08806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:52.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff08d3be700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff088134c80 0x7ff08806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:52.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff07d7fa700 1 -- 172.21.15.16:0/2487601665 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241396350 0 0) 0x7ff084000940 con 0x7ff088135450 2019-12-05T06:13:52.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff07d7fa700 1 -- 172.21.15.16:0/2487601665 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff068002e80 con 0x7ff088135450 2019-12-05T06:13:52.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff08cbbd700 1 -- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff088134c80 msgr2=0x7ff08806e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:52.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff088134c80 0x7ff08806e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:52.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff08cbbd700 1 -- 172.21.15.16:0/2487601665 >> v1:172.21.15.16:6789/0 conn(0x7ff088135450 legacy=0x7ff088134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:52.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff08cbbd700 1 -- 172.21.15.16:0/2487601665 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff088133b80 con 0x7ff08806dc50 2019-12-05T06:13:52.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> v2:172.21.15.76:3300/0 conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> v2:172.21.15.76:3300/0 conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.994+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=READY pgs=281 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:52.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.995+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=THROTTLE_DONE pgs=281 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.995+0000 7ff07d7fa700 1 -- 172.21.15.16:0/2487601665 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff078000940 con 0x7ff08806dc50 2019-12-05T06:13:52.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.995+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=THROTTLE_DONE pgs=281 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.995+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=THROTTLE_DONE pgs=281 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.995+0000 7ff07d7fa700 1 -- 172.21.15.16:0/2487601665 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff07800dff0 con 0x7ff08806dc50 2019-12-05T06:13:52.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.995+0000 7ff08e620700 1 -- 172.21.15.16:0/2487601665 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff088133b80 con 0x7ff08806dc50 2019-12-05T06:13:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.995+0000 7ff07d7fa700 1 -- 172.21.15.16:0/2487601665 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff07800e6c0 con 0x7ff08806dc50 2019-12-05T06:13:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.995+0000 7ff08e620700 1 -- 172.21.15.16:0/2487601665 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff088133b80 con 0x7ff08806dc50 2019-12-05T06:13:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.996+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=THROTTLE_DONE pgs=281 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.996+0000 7ff07d7fa700 1 -- 172.21.15.16:0/2487601665 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff07800dc70 con 0x7ff08806dc50 2019-12-05T06:13:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.996+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=THROTTLE_DONE pgs=281 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.996+0000 7ff07d7fa700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff068023670 0x7ff068025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:52.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.997+0000 7ff07d7fa700 1 -- 172.21.15.16:0/2487601665 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7ff078045d30 con 0x7ff08806dc50 2019-12-05T06:13:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.997+0000 7ff07ffff700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff068023670 0x7ff068025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:52.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:52.997+0000 7ff07ffff700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff068023670 0x7ff068025ae0 crc :-1 s=READY pgs=166 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:53.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.011+0000 7ff08e620700 1 -- 172.21.15.16:0/2487601665 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff088133b80 con 0x7ff08806dc50 2019-12-05T06:13:53.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.020+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=THROTTLE_DONE pgs=281 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:53.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.020+0000 7ff07d7fa700 1 -- 172.21.15.16:0/2487601665 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff07800da10 con 0x7ff08806dc50 2019-12-05T06:13:53.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.234+0000 7ff08e620700 1 -- 172.21.15.16:0/2487601665 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff088133b80 con 0x7ff08806dc50 2019-12-05T06:13:53.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.235+0000 7ff08cbbd700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=THROTTLE_DONE pgs=281 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:53.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.236+0000 7ff07d7fa700 1 -- 172.21.15.16:0/2487601665 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7ff07800dc20 con 0x7ff08806dc50 2019-12-05T06:13:53.232 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2019-12-05T06:13:53.232 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:13:53.232 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5049,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 -- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff068023670 msgr2=0x7ff068025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff068023670 0x7ff068025ae0 crc :-1 s=READY pgs=166 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 -- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 msgr2=0x7ff08806e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 0x7ff08806e040 secure :-1 s=READY pgs=281 cs=0 l=1 rx=0x7ff0780103a0 tx=0x7ff0780101d0).stop 2019-12-05T06:13:53.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 -- 172.21.15.16:0/2487601665 shutdown_connections 2019-12-05T06:13:53.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff068023670 0x7ff068025ae0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:53.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff088134c80 0x7ff08806e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:53.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 --2- 172.21.15.16:0/2487601665 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff08806dc50 0x7ff08806e040 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:53.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 -- 172.21.15.16:0/2487601665 shutdown_connections 2019-12-05T06:13:53.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 -- 172.21.15.16:0/2487601665 wait complete. 2019-12-05T06:13:53.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:53.241+0000 7ff08e620700 1 -- 172.21.15.16:0/2487601665 >> 172.21.15.16:0/2487601665 conn(0x7ff0880a2850 msgr2=0x7ff08806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:53.250 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:13:55.252 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:55.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.542+0000 7f4abde62700 1 Processor -- start 2019-12-05T06:13:55.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.542+0000 7f4abde62700 1 -- start start 2019-12-05T06:13:55.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.543+0000 7f4abde62700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4ab808c6d0 0x7f4ab808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:55.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.543+0000 7f4abde62700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ab808fef0 0x7f4ab808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:55.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.543+0000 7f4abde62700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ab808ef20 0x7f4ab808f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:55.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.543+0000 7f4abde62700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab808df90 con 0x7f4ab808ef20 2019-12-05T06:13:55.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.543+0000 7f4abde62700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab80a3f70 con 0x7f4ab808c6d0 2019-12-05T06:13:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.543+0000 7f4abde62700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ab808d830 con 0x7f4ab808fef0 2019-12-05T06:13:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.544+0000 7f4ab7fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4ab808c6d0 0x7f4ab808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.544+0000 7f4abcc00700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ab808ef20 0x7f4ab808f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:55.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.544+0000 7f4ab77fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ab808fef0 0x7f4ab808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.544+0000 7f4ab7fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4ab808c6d0 0x7f4ab808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:35744/0 (socket says 172.21.15.16:35744) 2019-12-05T06:13:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.544+0000 7f4ab7fff700 1 -- 172.21.15.16:0/388813084 learned_addr learned my addr 172.21.15.16:0/388813084 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:13:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.544+0000 7f4abcc00700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ab808ef20 0x7f4ab808f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49658/0 (socket says 172.21.15.16:49658) 2019-12-05T06:13:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.544+0000 7f4ab7fff700 1 -- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ab808fef0 msgr2=0x7f4ab808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:55.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab7fff700 1 --2- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ab808fef0 0x7f4ab808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:55.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab7fff700 1 -- 172.21.15.16:0/388813084 >> v2:172.21.15.16:3300/0 conn(0x7f4ab808ef20 msgr2=0x7f4ab808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:55.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab7fff700 1 --2- 172.21.15.16:0/388813084 >> v2:172.21.15.16:3300/0 conn(0x7f4ab808ef20 0x7f4ab808f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:55.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab7fff700 1 -- 172.21.15.16:0/388813084 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab808d540 con 0x7f4ab808c6d0 2019-12-05T06:13:55.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4abcc00700 1 --2- 172.21.15.16:0/388813084 >> v2:172.21.15.16:3300/0 conn(0x7f4ab808ef20 0x7f4ab808f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:13:55.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ab808fef0 0x7f4ab808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:13:55.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab7fff700 1 --2- 172.21.15.16:0/388813084 >> v2:172.21.15.76:3300/0 conn(0x7f4ab808c6d0 0x7f4ab808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4aa800fac0 tx=0x7f4aa800fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab7fff700 1 --2- 172.21.15.16:0/388813084 >> v2:172.21.15.76:3300/0 conn(0x7f4ab808c6d0 0x7f4ab808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4aa800fac0 tx=0x7f4aa800fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab7fff700 1 --2- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ab808c6d0 0x7f4ab808cac0 secure :-1 s=READY pgs=282 cs=0 l=1 rx=0x7f4aa800fac0 tx=0x7f4aa800fe60).ready entity=mon.1 client_cookie=63ce3a00614c729e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab7fff700 1 --2- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ab808c6d0 0x7f4ab808cac0 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7f4aa800fac0 tx=0x7f4aa800fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab7fff700 1 --2- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ab808c6d0 0x7f4ab808cac0 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7f4aa800fac0 tx=0x7f4aa800fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab6ffd700 1 -- 172.21.15.16:0/388813084 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aa8000cc0 con 0x7f4ab808c6d0 2019-12-05T06:13:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.545+0000 7f4ab7fff700 1 --2- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ab808c6d0 0x7f4ab808cac0 secure :-1 s=THROTTLE_DONE pgs=282 cs=0 l=1 rx=0x7f4aa800fac0 tx=0x7f4aa800fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.546+0000 7f4ab6ffd700 1 -- 172.21.15.16:0/388813084 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa8001200 con 0x7f4ab808c6d0 2019-12-05T06:13:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.546+0000 7f4ab6ffd700 1 -- 172.21.15.16:0/388813084 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aa8001890 con 0x7f4ab808c6d0 2019-12-05T06:13:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.546+0000 7f4abde62700 1 -- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ab808c6d0 msgr2=0x7f4ab808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.546+0000 7f4abde62700 1 --2- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ab808c6d0 0x7f4ab808cac0 secure :-1 s=READY pgs=282 cs=0 l=1 rx=0x7f4aa800fac0 tx=0x7f4aa800fe60).stop 2019-12-05T06:13:55.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.547+0000 7f4abde62700 1 -- 172.21.15.16:0/388813084 shutdown_connections 2019-12-05T06:13:55.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.547+0000 7f4abde62700 1 --2- 172.21.15.16:0/388813084 >> v2:172.21.15.16:3300/0 conn(0x7f4ab808ef20 0x7f4ab808f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:55.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.547+0000 7f4abde62700 1 --2- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ab808fef0 0x7f4ab808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:55.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.547+0000 7f4abde62700 1 --2- 172.21.15.16:0/388813084 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ab808c6d0 0x7f4ab808cac0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:55.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.547+0000 7f4abde62700 1 -- 172.21.15.16:0/388813084 shutdown_connections 2019-12-05T06:13:55.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.547+0000 7f4abde62700 1 -- 172.21.15.16:0/388813084 wait complete. 2019-12-05T06:13:55.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.547+0000 7f4abde62700 1 -- 172.21.15.16:0/388813084 >> 172.21.15.16:0/388813084 conn(0x7f4ab8061940 msgr2=0x7f4ab8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:55.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.551+0000 7f4abde62700 1 Processor -- start 2019-12-05T06:13:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.552+0000 7f4abde62700 1 -- start start 2019-12-05T06:13:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.553+0000 7f4abde62700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ab8132d00 0x7f4ab8132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:55.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.553+0000 7f4abde62700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4ab8154b00 0x7f4ab8156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.553+0000 7f4abde62700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ab8098da0 con 0x7f4ab809b850 2019-12-05T06:13:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.553+0000 7f4abde62700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab809aa60 con 0x7f4ab8132d00 2019-12-05T06:13:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.553+0000 7f4abde62700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ab809a030 con 0x7f4ab8154b00 2019-12-05T06:13:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.553+0000 7f4ab77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ab8132d00 0x7f4ab8132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:55.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.553+0000 7f4ab7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4ab809b850 0x7f4ab8131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34828/0 (socket says 172.21.15.16:34828) 2019-12-05T06:13:55.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.553+0000 7f4ab7fff700 1 -- 172.21.15.16:0/591384115 learned_addr learned my addr 172.21.15.16:0/591384115 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:13:55.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.553+0000 7f4abcc00700 1 --2- 172.21.15.16:0/591384115 >> v2:172.21.15.76:3300/0 conn(0x7f4ab8154b00 0x7f4ab8156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:55.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab77fe700 1 -- 172.21.15.16:0/591384115 >> v2:172.21.15.76:3300/0 conn(0x7f4ab8154b00 msgr2=0x7f4ab8156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:55.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> v2:172.21.15.76:3300/0 conn(0x7f4ab8154b00 0x7f4ab8156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:55.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab77fe700 1 -- 172.21.15.16:0/591384115 >> v1:172.21.15.16:6789/0 conn(0x7f4ab809b850 legacy=0x7f4ab8131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:55.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab4ff9700 1 -- 172.21.15.16:0/591384115 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104672211 0 0) 0x7f4aa8008190 con 0x7f4ab809b850 2019-12-05T06:13:55.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab77fe700 1 -- 172.21.15.16:0/591384115 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ab809b0f0 con 0x7f4ab8132d00 2019-12-05T06:13:55.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> v2:172.21.15.16:3300/0 conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> v2:172.21.15.16:3300/0 conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=READY pgs=364 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:55.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab4ff9700 1 -- 172.21.15.16:0/591384115 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aa0000940 con 0x7f4ab8132d00 2019-12-05T06:13:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab4ff9700 1 -- 172.21.15.16:0/591384115 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4aa000de60 con 0x7f4ab8132d00 2019-12-05T06:13:55.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4ab4ff9700 1 -- 172.21.15.16:0/591384115 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4aa000e4f0 con 0x7f4ab8132d00 2019-12-05T06:13:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.554+0000 7f4abde62700 1 -- 172.21.15.16:0/591384115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ab809b0f0 con 0x7f4ab8132d00 2019-12-05T06:13:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.555+0000 7f4abde62700 1 -- 172.21.15.16:0/591384115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ab809b0f0 con 0x7f4ab8132d00 2019-12-05T06:13:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.556+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.556+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.556+0000 7f4ab4ff9700 1 -- 172.21.15.16:0/591384115 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4aa000d970 con 0x7f4ab8132d00 2019-12-05T06:13:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.557+0000 7f4ab4ff9700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4a9c023630 0x7f4a9c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.557+0000 7f4ab4ff9700 1 -- 172.21.15.16:0/591384115 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f4aa0089d00 con 0x7f4ab8132d00 2019-12-05T06:13:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.557+0000 7f4ab7fff700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4a9c023630 0x7f4a9c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:55.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.558+0000 7f4ab7fff700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4a9c023630 0x7f4a9c025ac0 crc :-1 s=READY pgs=167 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:55.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.572+0000 7f4abde62700 1 -- 172.21.15.16:0/591384115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ab809b0f0 con 0x7f4ab8132d00 2019-12-05T06:13:55.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.580+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.581+0000 7f4ab4ff9700 1 -- 172.21.15.16:0/591384115 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4aa006bf80 con 0x7f4ab8132d00 2019-12-05T06:13:55.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.788+0000 7f4abde62700 1 -- 172.21.15.16:0/591384115 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ab809b0f0 con 0x7f4ab8132d00 2019-12-05T06:13:55.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.788+0000 7f4ab77fe700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=THROTTLE_DONE pgs=364 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:55.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.789+0000 7f4ab4ff9700 1 -- 172.21.15.16:0/591384115 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f4aa006bf80 con 0x7f4ab8132d00 2019-12-05T06:13:55.783 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2019-12-05T06:13:55.783 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:13:55.784 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5049,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.791+0000 7f4abde62700 1 -- 172.21.15.16:0/591384115 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4a9c023630 msgr2=0x7f4a9c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.791+0000 7f4abde62700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4a9c023630 0x7f4a9c025ac0 crc :-1 s=READY pgs=167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:55.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.791+0000 7f4abde62700 1 -- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 msgr2=0x7f4ab8132530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.791+0000 7f4abde62700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 0x7f4ab8132530 secure :-1 s=READY pgs=364 cs=0 l=1 rx=0x7f4aa00543a0 tx=0x7f4aa00541a0).stop 2019-12-05T06:13:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.791+0000 7f4abde62700 1 -- 172.21.15.16:0/591384115 shutdown_connections 2019-12-05T06:13:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.791+0000 7f4abde62700 1 --2- 172.21.15.16:0/591384115 >> v2:172.21.15.76:3300/0 conn(0x7f4ab8154b00 0x7f4ab8156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.791+0000 7f4abde62700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4a9c023630 0x7f4a9c025ac0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:55.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.791+0000 7f4abde62700 1 --2- 172.21.15.16:0/591384115 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ab8132d00 0x7f4ab8132530 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.792+0000 7f4abde62700 1 -- 172.21.15.16:0/591384115 shutdown_connections 2019-12-05T06:13:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.792+0000 7f4abde62700 1 -- 172.21.15.16:0/591384115 wait complete. 2019-12-05T06:13:55.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:55.792+0000 7f4abde62700 1 -- 172.21.15.16:0/591384115 >> 172.21.15.16:0/591384115 conn(0x7f4ab80714f0 msgr2=0x7f4ab809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:55.803 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:13:57.804 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:13:58.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.061+0000 7fddb7fbe700 1 Processor -- start 2019-12-05T06:13:58.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.061+0000 7fddb7fbe700 1 -- start start 2019-12-05T06:13:58.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.063+0000 7fddb7fbe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fddb008c6d0 0x7fddb008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:58.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.063+0000 7fddb7fbe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb008fef0 0x7fddb008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:13:58.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.063+0000 7fddb7fbe700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb00958e0 con 0x7fddb008ef20 2019-12-05T06:13:58.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.063+0000 7fddb7fbe700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb008df90 con 0x7fddb008c6d0 2019-12-05T06:13:58.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.063+0000 7fddb7fbe700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb00a3f70 con 0x7fddb008fef0 2019-12-05T06:13:58.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.063+0000 7fddb6d5c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fddb008ef20 0x7fddb008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34840/0 (socket says 172.21.15.16:34840) 2019-12-05T06:13:58.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.063+0000 7fddb6d5c700 1 -- 172.21.15.16:0/578410863 learned_addr learned my addr 172.21.15.16:0/578410863 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:13:58.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.063+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb008fef0 0x7fddb008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:58.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.063+0000 7fddb655b700 1 --2- 172.21.15.16:0/578410863 >> v2:172.21.15.76:3300/0 conn(0x7fddb008c6d0 0x7fddb008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:58.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.064+0000 7fddb5559700 1 -- 172.21.15.16:0/578410863 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1487410804 0 0) 0x7fdda0001150 con 0x7fddb008ef20 2019-12-05T06:13:58.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.064+0000 7fddb5559700 1 -- 172.21.15.16:0/578410863 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd98002e80 con 0x7fddb008ef20 2019-12-05T06:13:58.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.064+0000 7fddb655b700 1 -- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb008fef0 msgr2=0x7fddb008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:13:58.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.064+0000 7fddb655b700 1 --2- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb008fef0 0x7fddb008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.064+0000 7fddb655b700 1 -- 172.21.15.16:0/578410863 >> v1:172.21.15.16:6789/0 conn(0x7fddb008ef20 legacy=0x7fddb008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.064+0000 7fddb655b700 1 -- 172.21.15.16:0/578410863 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb008d540 con 0x7fddb008c6d0 2019-12-05T06:13:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.064+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb008fef0 0x7fddb008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:13:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb655b700 1 --2- 172.21.15.16:0/578410863 >> v2:172.21.15.76:3300/0 conn(0x7fddb008c6d0 0x7fddb008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdda800b2d0 tx=0x7fdda800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb655b700 1 --2- 172.21.15.16:0/578410863 >> v2:172.21.15.76:3300/0 conn(0x7fddb008c6d0 0x7fddb008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdda800b2d0 tx=0x7fdda800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb655b700 1 --2- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddb008c6d0 0x7fddb008cac0 secure :-1 s=READY pgs=283 cs=0 l=1 rx=0x7fdda800b2d0 tx=0x7fdda800bae0).ready entity=mon.1 client_cookie=a64c068079505a6a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb655b700 1 --2- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddb008c6d0 0x7fddb008cac0 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7fdda800b2d0 tx=0x7fdda800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb655b700 1 --2- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddb008c6d0 0x7fddb008cac0 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7fdda800b2d0 tx=0x7fdda800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb5559700 1 -- 172.21.15.16:0/578410863 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdda8007590 con 0x7fddb008c6d0 2019-12-05T06:13:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb655b700 1 --2- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddb008c6d0 0x7fddb008cac0 secure :-1 s=THROTTLE_DONE pgs=283 cs=0 l=1 rx=0x7fdda800b2d0 tx=0x7fdda800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb5559700 1 -- 172.21.15.16:0/578410863 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda8009d30 con 0x7fddb008c6d0 2019-12-05T06:13:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb5559700 1 -- 172.21.15.16:0/578410863 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdda800a400 con 0x7fddb008c6d0 2019-12-05T06:13:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb7fbe700 1 -- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddb008c6d0 msgr2=0x7fddb008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:58.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.065+0000 7fddb7fbe700 1 --2- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddb008c6d0 0x7fddb008cac0 secure :-1 s=READY pgs=283 cs=0 l=1 rx=0x7fdda800b2d0 tx=0x7fdda800bae0).stop 2019-12-05T06:13:58.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.067+0000 7fddb7fbe700 1 -- 172.21.15.16:0/578410863 shutdown_connections 2019-12-05T06:13:58.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.068+0000 7fddb7fbe700 1 --2- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb008fef0 0x7fddb008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:13:58.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.068+0000 7fddb7fbe700 1 --2- 172.21.15.16:0/578410863 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddb008c6d0 0x7fddb008cac0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:58.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.068+0000 7fddb7fbe700 1 -- 172.21.15.16:0/578410863 shutdown_connections 2019-12-05T06:13:58.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.068+0000 7fddb7fbe700 1 -- 172.21.15.16:0/578410863 wait complete. 2019-12-05T06:13:58.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.068+0000 7fddb7fbe700 1 -- 172.21.15.16:0/578410863 >> 172.21.15.16:0/578410863 conn(0x7fddb0061940 msgr2=0x7fddb0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.073+0000 7fddb7fbe700 1 Processor -- start 2019-12-05T06:13:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.073+0000 7fddb7fbe700 1 -- start start 2019-12-05T06:13:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.074+0000 7fddb7fbe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fddb01302f0 0x7fddb009d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.074+0000 7fddb7fbe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:58.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.074+0000 7fddb7fbe700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddb012cb90 con 0x7fddb009e280 2019-12-05T06:13:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.074+0000 7fddb7fbe700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddb0131ff0 con 0x7fddb01302f0 2019-12-05T06:13:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.074+0000 7fddb7fbe700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddb01315c0 con 0x7fddb009e7d0 2019-12-05T06:13:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddb5d5a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddb655b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fddb01302f0 0x7fddb009d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddb5d5a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:53668/0 (socket says 172.21.15.16:53668) 2019-12-05T06:13:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddb5d5a700 1 -- 172.21.15.16:0/1975143187 learned_addr learned my addr 172.21.15.16:0/1975143187 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:13:58.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddb5d5a700 1 -- 172.21.15.16:0/1975143187 >> v2:172.21.15.76:3300/0 conn(0x7fddb01302f0 msgr2=0x7fddb009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> v2:172.21.15.76:3300/0 conn(0x7fddb01302f0 0x7fddb009d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddb5d5a700 1 -- 172.21.15.16:0/1975143187 >> v1:172.21.15.76:6789/0 conn(0x7fddb009e280 legacy=0x7fddb006dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddaeffd700 1 -- 172.21.15.16:0/1975143187 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780770426 0 0) 0x7fdda00009b0 con 0x7fddb009e280 2019-12-05T06:13:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddb5d5a700 1 -- 172.21.15.16:0/1975143187 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddb0132680 con 0x7fddb009e7d0 2019-12-05T06:13:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddb655b700 1 --2- 172.21.15.16:0/1975143187 >> v2:172.21.15.76:3300/0 conn(0x7fddb01302f0 0x7fddb009d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:13:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.075+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.076+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.076+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.076+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.076+0000 7fddaeffd700 1 -- 172.21.15.16:0/1975143187 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdda4007510 con 0x7fddb009e7d0 2019-12-05T06:13:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.076+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.076+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.076+0000 7fddaeffd700 1 -- 172.21.15.16:0/1975143187 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda400edd0 con 0x7fddb009e7d0 2019-12-05T06:13:58.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.076+0000 7fddb7fbe700 1 -- 172.21.15.16:0/1975143187 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddb0132680 con 0x7fddb009e7d0 2019-12-05T06:13:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.076+0000 7fddaeffd700 1 -- 172.21.15.16:0/1975143187 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdda400f460 con 0x7fddb009e7d0 2019-12-05T06:13:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.076+0000 7fddb7fbe700 1 -- 172.21.15.16:0/1975143187 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddb0132680 con 0x7fddb009e7d0 2019-12-05T06:13:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.077+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.078+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.078+0000 7fddaeffd700 1 -- 172.21.15.16:0/1975143187 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdda400e9b0 con 0x7fddb009e7d0 2019-12-05T06:13:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.078+0000 7fddaeffd700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd90023630 0x7fdd90025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:13:58.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.078+0000 7fddaeffd700 1 -- 172.21.15.16:0/1975143187 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fdda4045df0 con 0x7fddb009e7d0 2019-12-05T06:13:58.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.078+0000 7fddb655b700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd90023630 0x7fdd90025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:13:58.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.079+0000 7fddb655b700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd90023630 0x7fdd90025ac0 crc :-1 s=READY pgs=168 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:13:58.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.093+0000 7fddb7fbe700 1 -- 172.21.15.16:0/1975143187 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddb0132680 con 0x7fddb009e7d0 2019-12-05T06:13:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.101+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.101+0000 7fddaeffd700 1 -- 172.21.15.16:0/1975143187 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdda4000940 con 0x7fddb009e7d0 2019-12-05T06:13:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.346+0000 7fddb7fbe700 1 -- 172.21.15.16:0/1975143187 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fddb0132680 con 0x7fddb009e7d0 2019-12-05T06:13:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.347+0000 7fddb5d5a700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=THROTTLE_DONE pgs=101 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:13:58.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.347+0000 7fddaeffd700 1 -- 172.21.15.16:0/1975143187 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fdda400e9b0 con 0x7fddb009e7d0 2019-12-05T06:13:58.349 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:13:58.349 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5049,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:13:58.350 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2019-12-05T06:13:58.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 -- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd90023630 msgr2=0x7fdd90025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:58.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd90023630 0x7fdd90025ac0 crc :-1 s=READY pgs=168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:58.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 -- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 msgr2=0x7fddb009dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:13:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 secure :-1 s=READY pgs=101 cs=0 l=1 rx=0x7fdda4010430 tx=0x7fdda40107d0).stop 2019-12-05T06:13:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 -- 172.21.15.16:0/1975143187 shutdown_connections 2019-12-05T06:13:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd90023630 0x7fdd90025ac0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 --2- 172.21.15.16:0/1975143187 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddb009e7d0 0x7fddb009dd40 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 --2- 172.21.15.16:0/1975143187 >> v2:172.21.15.76:3300/0 conn(0x7fddb01302f0 0x7fddb009d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:13:58.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 -- 172.21.15.16:0/1975143187 shutdown_connections 2019-12-05T06:13:58.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 -- 172.21.15.16:0/1975143187 wait complete. 2019-12-05T06:13:58.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:13:58.351+0000 7fddb7fbe700 1 -- 172.21.15.16:0/1975143187 >> 172.21.15.16:0/1975143187 conn(0x7fddb00a2850 msgr2=0x7fddb006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:13:58.363 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:13:59.935 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:13:59.940 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:14:00.364 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:00.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.648+0000 7f5b73650700 1 Processor -- start 2019-12-05T06:14:00.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.649+0000 7f5b73650700 1 -- start start 2019-12-05T06:14:00.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.650+0000 7f5b73650700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:00.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.650+0000 7f5b73650700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5b6c08f720 0x7f5b6c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:00.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.650+0000 7f5b73650700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6c08fb80 con 0x7f5b6c08fef0 2019-12-05T06:14:00.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.650+0000 7f5b73650700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c08df90 con 0x7f5b6c08c6d0 2019-12-05T06:14:00.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.650+0000 7f5b73650700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c0a3f70 con 0x7f5b6c08f720 2019-12-05T06:14:00.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.651+0000 7f5b71bed700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:00.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.651+0000 7f5b71bed700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49686/0 (socket says 172.21.15.16:49686) 2019-12-05T06:14:00.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.651+0000 7f5b71bed700 1 -- 172.21.15.16:0/1237180988 learned_addr learned my addr 172.21.15.16:0/1237180988 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:00.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.651+0000 7f5b713ec700 1 --1- 172.21.15.16:0/1237180988 >> v1:172.21.15.16:6789/0 conn(0x7f5b6c08fef0 0x7f5b6c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34858/0 (socket says 172.21.15.16:34858) 2019-12-05T06:14:00.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.651+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1237180988 >> v2:172.21.15.76:3300/0 conn(0x7f5b6c08f720 0x7f5b6c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:00.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.651+0000 7f5b71bed700 1 -- 172.21.15.16:0/1237180988 >> v2:172.21.15.76:3300/0 conn(0x7f5b6c08f720 msgr2=0x7f5b6c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:00.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.651+0000 7f5b71bed700 1 --2- 172.21.15.16:0/1237180988 >> v2:172.21.15.76:3300/0 conn(0x7f5b6c08f720 0x7f5b6c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:00.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.651+0000 7f5b71bed700 1 -- 172.21.15.16:0/1237180988 >> v1:172.21.15.16:6789/0 conn(0x7f5b6c08fef0 legacy=0x7f5b6c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:00.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.651+0000 7f5b71bed700 1 -- 172.21.15.16:0/1237180988 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c08d540 con 0x7f5b6c08c6d0 2019-12-05T06:14:00.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b71bed700 1 --2- 172.21.15.16:0/1237180988 >> v2:172.21.15.16:3300/0 conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5b6800f6c0 tx=0x7f5b6800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b71bed700 1 --2- 172.21.15.16:0/1237180988 >> v2:172.21.15.16:3300/0 conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5b6800f6c0 tx=0x7f5b6800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b71bed700 1 --2- 172.21.15.16:0/1237180988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 secure :-1 s=READY pgs=367 cs=0 l=1 rx=0x7f5b6800f6c0 tx=0x7f5b6800fa60).ready entity=mon.0 client_cookie=c2635cec0948f81a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:00.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b71bed700 1 --2- 172.21.15.16:0/1237180988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7f5b6800f6c0 tx=0x7f5b6800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b71bed700 1 --2- 172.21.15.16:0/1237180988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7f5b6800f6c0 tx=0x7f5b6800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b70beb700 1 -- 172.21.15.16:0/1237180988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b68000960 con 0x7f5b6c08c6d0 2019-12-05T06:14:00.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b71bed700 1 --2- 172.21.15.16:0/1237180988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7f5b6800f6c0 tx=0x7f5b6800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b70beb700 1 -- 172.21.15.16:0/1237180988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b68000c90 con 0x7f5b6c08c6d0 2019-12-05T06:14:00.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b70beb700 1 -- 172.21.15.16:0/1237180988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b680013a0 con 0x7f5b6c08c6d0 2019-12-05T06:14:00.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b73650700 1 -- 172.21.15.16:0/1237180988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c08c6d0 msgr2=0x7f5b6c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:00.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.652+0000 7f5b73650700 1 --2- 172.21.15.16:0/1237180988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 secure :-1 s=READY pgs=367 cs=0 l=1 rx=0x7f5b6800f6c0 tx=0x7f5b6800fa60).stop 2019-12-05T06:14:00.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.654+0000 7f5b73650700 1 -- 172.21.15.16:0/1237180988 shutdown_connections 2019-12-05T06:14:00.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.654+0000 7f5b73650700 1 --2- 172.21.15.16:0/1237180988 >> v2:172.21.15.76:3300/0 conn(0x7f5b6c08f720 0x7f5b6c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:00.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.654+0000 7f5b73650700 1 --2- 172.21.15.16:0/1237180988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c08c6d0 0x7f5b6c08cac0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:00.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.655+0000 7f5b73650700 1 -- 172.21.15.16:0/1237180988 shutdown_connections 2019-12-05T06:14:00.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.655+0000 7f5b73650700 1 -- 172.21.15.16:0/1237180988 wait complete. 2019-12-05T06:14:00.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.655+0000 7f5b73650700 1 -- 172.21.15.16:0/1237180988 >> 172.21.15.16:0/1237180988 conn(0x7f5b6c061940 msgr2=0x7f5b6c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:00.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.660+0000 7f5b73650700 1 Processor -- start 2019-12-05T06:14:00.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.660+0000 7f5b73650700 1 -- start start 2019-12-05T06:14:00.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.661+0000 7f5b73650700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b6c14d540 0x7f5b6c14d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:00.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.661+0000 7f5b73650700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6c088240 0x7f5b6c19b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:00.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.661+0000 7f5b73650700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b6c09dee0 con 0x7f5b6c088a10 2019-12-05T06:14:00.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.661+0000 7f5b73650700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b6c085190 con 0x7f5b6c088240 2019-12-05T06:14:00.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.661+0000 7f5b73650700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c086080 con 0x7f5b6c14d540 2019-12-05T06:14:00.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.661+0000 7f5b723ee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6c088240 0x7f5b6c19b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:00.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.662+0000 7f5b723ee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b6c088240 0x7f5b6c19b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49696/0 (socket says 172.21.15.16:49696) 2019-12-05T06:14:00.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.662+0000 7f5b723ee700 1 -- 172.21.15.16:0/1540239066 learned_addr learned my addr 172.21.15.16:0/1540239066 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:00.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.662+0000 7f5b71bed700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b6c14d540 0x7f5b6c14d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:00.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.662+0000 7f5b723ee700 1 -- 172.21.15.16:0/1540239066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b6c14d540 msgr2=0x7f5b6c14d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:00.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.662+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b6c14d540 0x7f5b6c14d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:00.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.662+0000 7f5b627fc700 1 -- 172.21.15.16:0/1540239066 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000376747 0 0) 0x7f5b5c001100 con 0x7f5b6c088a10 2019-12-05T06:14:00.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.662+0000 7f5b723ee700 1 -- 172.21.15.16:0/1540239066 >> v1:172.21.15.76:6789/0 conn(0x7f5b6c088a10 legacy=0x7f5b6c087a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:00.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.662+0000 7f5b723ee700 1 -- 172.21.15.16:0/1540239066 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c087140 con 0x7f5b6c088240 2019-12-05T06:14:00.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.662+0000 7f5b71bed700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b6c14d540 0x7f5b6c14d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:14:00.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.662+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> v2:172.21.15.16:3300/0 conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.663+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> v2:172.21.15.16:3300/0 conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.663+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=READY pgs=368 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:00.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.663+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.663+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.663+0000 7f5b627fc700 1 -- 172.21.15.16:0/1540239066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b64000e90 con 0x7f5b6c088240 2019-12-05T06:14:00.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.663+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.663+0000 7f5b627fc700 1 -- 172.21.15.16:0/1540239066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6400f2c0 con 0x7f5b6c088240 2019-12-05T06:14:00.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.663+0000 7f5b627fc700 1 -- 172.21.15.16:0/1540239066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b6400f990 con 0x7f5b6c088240 2019-12-05T06:14:00.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.663+0000 7f5b73650700 1 -- 172.21.15.16:0/1540239066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c087140 con 0x7f5b6c088240 2019-12-05T06:14:00.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.664+0000 7f5b73650700 1 -- 172.21.15.16:0/1540239066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c087140 con 0x7f5b6c088240 2019-12-05T06:14:00.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.664+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.665+0000 7f5b627fc700 1 -- 172.21.15.16:0/1540239066 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5b6400f4b0 con 0x7f5b6c088240 2019-12-05T06:14:00.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.665+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.665+0000 7f5b627fc700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5b54023630 0x7f5b54025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:00.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.665+0000 7f5b627fc700 1 -- 172.21.15.16:0/1540239066 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f5b640461b0 con 0x7f5b6c088240 2019-12-05T06:14:00.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.666+0000 7f5b71bed700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5b54023630 0x7f5b54025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:00.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.666+0000 7f5b71bed700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5b54023630 0x7f5b54025ac0 crc :-1 s=READY pgs=169 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:00.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.680+0000 7f5b73650700 1 -- 172.21.15.16:0/1540239066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6c087140 con 0x7f5b6c088240 2019-12-05T06:14:00.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.688+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.688+0000 7f5b627fc700 1 -- 172.21.15.16:0/1540239066 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5b64000f30 con 0x7f5b6c088240 2019-12-05T06:14:00.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.933+0000 7f5b73650700 1 -- 172.21.15.16:0/1540239066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b6c087140 con 0x7f5b6c088240 2019-12-05T06:14:00.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.934+0000 7f5b723ee700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:00.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.934+0000 7f5b627fc700 1 -- 172.21.15.16:0/1540239066 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f5b6400ee20 con 0x7f5b6c088240 2019-12-05T06:14:00.928 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2019-12-05T06:14:00.928 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:14:00.929 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5049,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.936+0000 7f5b73650700 1 -- 172.21.15.16:0/1540239066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5b54023630 msgr2=0x7f5b54025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.936+0000 7f5b73650700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5b54023630 0x7f5b54025ac0 crc :-1 s=READY pgs=169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.936+0000 7f5b73650700 1 -- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 msgr2=0x7f5b6c19b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:00.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.936+0000 7f5b73650700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 0x7f5b6c19b400 secure :-1 s=READY pgs=368 cs=0 l=1 rx=0x7f5b640107d0 tx=0x7f5b64010b70).stop 2019-12-05T06:14:00.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.936+0000 7f5b73650700 1 -- 172.21.15.16:0/1540239066 shutdown_connections 2019-12-05T06:14:00.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.936+0000 7f5b73650700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5b54023630 0x7f5b54025ac0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.936+0000 7f5b73650700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b6c088240 0x7f5b6c19b400 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.936+0000 7f5b73650700 1 --2- 172.21.15.16:0/1540239066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b6c14d540 0x7f5b6c14d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.937+0000 7f5b73650700 1 -- 172.21.15.16:0/1540239066 shutdown_connections 2019-12-05T06:14:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.937+0000 7f5b73650700 1 -- 172.21.15.16:0/1540239066 wait complete. 2019-12-05T06:14:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:00.937+0000 7f5b73650700 1 -- 172.21.15.16:0/1540239066 >> 172.21.15.16:0/1540239066 conn(0x7f5b6c0700c0 msgr2=0x7f5b6c06ef20 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:00.942 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:14:02.943 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.194+0000 7fd366b10700 1 Processor -- start 2019-12-05T06:14:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.195+0000 7fd366b10700 1 -- start start 2019-12-05T06:14:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.196+0000 7fd366b10700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd36008d2c0 0x7fd36008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.196+0000 7fd366b10700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.196+0000 7fd366b10700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3600958e0 con 0x7fd36008c6d0 2019-12-05T06:14:03.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.196+0000 7fd366b10700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd36008df90 con 0x7fd36008d2c0 2019-12-05T06:14:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.196+0000 7fd366b10700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3600a3f70 con 0x7fd36008fef0 2019-12-05T06:14:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.196+0000 7fd3658ae700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.196+0000 7fd3648ac700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd36008d2c0 0x7fd36008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.196+0000 7fd3658ae700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:53692/0 (socket says 172.21.15.16:53692) 2019-12-05T06:14:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.196+0000 7fd3658ae700 1 -- 172.21.15.16:0/2859411022 learned_addr learned my addr 172.21.15.16:0/2859411022 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd3658ae700 1 -- 172.21.15.16:0/2859411022 >> v2:172.21.15.76:3300/0 conn(0x7fd36008d2c0 msgr2=0x7fd36008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:03.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd3658ae700 1 --2- 172.21.15.16:0/2859411022 >> v2:172.21.15.76:3300/0 conn(0x7fd36008d2c0 0x7fd36008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd3658ae700 1 -- 172.21.15.16:0/2859411022 >> v1:172.21.15.76:6789/0 conn(0x7fd36008c6d0 legacy=0x7fd36008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd3658ae700 1 -- 172.21.15.16:0/2859411022 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd36008e620 con 0x7fd36008fef0 2019-12-05T06:14:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd3658ae700 1 --2- 172.21.15.16:0/2859411022 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd35800fae0 tx=0x7fd35800f910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd3658ae700 1 --2- 172.21.15.16:0/2859411022 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd35800fae0 tx=0x7fd35800f910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd3658ae700 1 --2- 172.21.15.16:0/2859411022 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 secure :-1 s=READY pgs=102 cs=0 l=1 rx=0x7fd35800fae0 tx=0x7fd35800f910).ready entity=mon.2 client_cookie=3160d702d407731 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:03.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd3658ae700 1 --2- 172.21.15.16:0/2859411022 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7fd35800fae0 tx=0x7fd35800f910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd35ffff700 1 -- 172.21.15.16:0/2859411022 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd358000df0 con 0x7fd36008fef0 2019-12-05T06:14:03.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd3658ae700 1 --2- 172.21.15.16:0/2859411022 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7fd35800fae0 tx=0x7fd35800f910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd3658ae700 1 --2- 172.21.15.16:0/2859411022 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 secure :-1 s=THROTTLE_DONE pgs=102 cs=0 l=1 rx=0x7fd35800fae0 tx=0x7fd35800f910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.197+0000 7fd35ffff700 1 -- 172.21.15.16:0/2859411022 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3580011f0 con 0x7fd36008fef0 2019-12-05T06:14:03.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.198+0000 7fd35ffff700 1 -- 172.21.15.16:0/2859411022 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd358001880 con 0x7fd36008fef0 2019-12-05T06:14:03.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.198+0000 7fd366b10700 1 -- 172.21.15.16:0/2859411022 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 msgr2=0x7fd36008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:03.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.198+0000 7fd366b10700 1 --2- 172.21.15.16:0/2859411022 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 secure :-1 s=READY pgs=102 cs=0 l=1 rx=0x7fd35800fae0 tx=0x7fd35800f910).stop 2019-12-05T06:14:03.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.199+0000 7fd366b10700 1 -- 172.21.15.16:0/2859411022 shutdown_connections 2019-12-05T06:14:03.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.199+0000 7fd366b10700 1 --2- 172.21.15.16:0/2859411022 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd36008fef0 0x7fd36008f460 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:03.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.199+0000 7fd366b10700 1 --2- 172.21.15.16:0/2859411022 >> v2:172.21.15.76:3300/0 conn(0x7fd36008d2c0 0x7fd36008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:03.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.199+0000 7fd366b10700 1 -- 172.21.15.16:0/2859411022 shutdown_connections 2019-12-05T06:14:03.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.199+0000 7fd366b10700 1 -- 172.21.15.16:0/2859411022 wait complete. 2019-12-05T06:14:03.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.199+0000 7fd366b10700 1 -- 172.21.15.16:0/2859411022 >> 172.21.15.16:0/2859411022 conn(0x7fd360061940 msgr2=0x7fd360096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.203+0000 7fd366b10700 1 Processor -- start 2019-12-05T06:14:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.203+0000 7fd366b10700 1 -- start start 2019-12-05T06:14:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.204+0000 7fd366b10700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd3601302f0 0x7fd36009d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.204+0000 7fd366b10700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd360156e70 0x7fd360159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.204+0000 7fd366b10700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd36012cb90 con 0x7fd36009e7d0 2019-12-05T06:14:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.204+0000 7fd366b10700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3601315c0 con 0x7fd360156e70 2019-12-05T06:14:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.204+0000 7fd366b10700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd360131890 con 0x7fd3601302f0 2019-12-05T06:14:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.204+0000 7fd3658ae700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd360156e70 0x7fd360159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.204+0000 7fd3650ad700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd3601302f0 0x7fd36009d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.204+0000 7fd3658ae700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd360156e70 0x7fd360159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:35798/0 (socket says 172.21.15.16:35798) 2019-12-05T06:14:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.204+0000 7fd3658ae700 1 -- 172.21.15.16:0/3315538564 learned_addr learned my addr 172.21.15.16:0/3315538564 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.204+0000 7fd3648ac700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd36009e7d0 0x7fd36009dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:56178/0 (socket says 172.21.15.16:56178) 2019-12-05T06:14:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.205+0000 7fd3658ae700 1 -- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd3601302f0 msgr2=0x7fd36009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.205+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd3601302f0 0x7fd36009d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.205+0000 7fd3658ae700 1 -- 172.21.15.16:0/3315538564 >> v1:172.21.15.76:6789/0 conn(0x7fd36009e7d0 legacy=0x7fd36009dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.205+0000 7fd3658ae700 1 -- 172.21.15.16:0/3315538564 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd360132680 con 0x7fd360156e70 2019-12-05T06:14:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.205+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> v2:172.21.15.76:3300/0 conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.205+0000 7fd35dffb700 1 -- 172.21.15.16:0/3315538564 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304816138 0 0) 0x7fd354000c90 con 0x7fd36009e7d0 2019-12-05T06:14:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.205+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> v2:172.21.15.76:3300/0 conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.205+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=READY pgs=288 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.206+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.206+0000 7fd35dffb700 1 -- 172.21.15.16:0/3315538564 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd35800fa10 con 0x7fd360156e70 2019-12-05T06:14:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.206+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.206+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.206+0000 7fd35dffb700 1 -- 172.21.15.16:0/3315538564 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3580083a0 con 0x7fd360156e70 2019-12-05T06:14:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.206+0000 7fd35dffb700 1 -- 172.21.15.16:0/3315538564 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd358010090 con 0x7fd360156e70 2019-12-05T06:14:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.206+0000 7fd366b10700 1 -- 172.21.15.16:0/3315538564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd360132680 con 0x7fd360156e70 2019-12-05T06:14:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.206+0000 7fd366b10700 1 -- 172.21.15.16:0/3315538564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd360132680 con 0x7fd360156e70 2019-12-05T06:14:03.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.207+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.207+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.207+0000 7fd35dffb700 1 -- 172.21.15.16:0/3315538564 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd35800fa10 con 0x7fd360156e70 2019-12-05T06:14:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.208+0000 7fd35dffb700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd340023630 0x7fd340025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:03.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.208+0000 7fd35dffb700 1 -- 172.21.15.16:0/3315538564 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7fd358046160 con 0x7fd360156e70 2019-12-05T06:14:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.208+0000 7fd3650ad700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd340023630 0x7fd340025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.208+0000 7fd3650ad700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd340023630 0x7fd340025ac0 crc :-1 s=READY pgs=170 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:03.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.210+0000 7fd366b10700 1 -- 172.21.15.16:0/3315538564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd360132680 con 0x7fd360156e70 2019-12-05T06:14:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.218+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.219+0000 7fd35dffb700 1 -- 172.21.15.16:0/3315538564 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd358007950 con 0x7fd360156e70 2019-12-05T06:14:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.446+0000 7fd366b10700 1 -- 172.21.15.16:0/3315538564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd360132680 con 0x7fd360156e70 2019-12-05T06:14:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.447+0000 7fd3658ae700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.447+0000 7fd35dffb700 1 -- 172.21.15.16:0/3315538564 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fd35800fa10 con 0x7fd360156e70 2019-12-05T06:14:03.445 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2019-12-05T06:14:03.446 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:14:03.446 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5049,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 -- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd340023630 msgr2=0x7fd340025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd340023630 0x7fd340025ac0 crc :-1 s=READY pgs=170 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 -- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 msgr2=0x7fd360159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 0x7fd360159270 secure :-1 s=READY pgs=288 cs=0 l=1 rx=0x7fd3580078f0 tx=0x7fd35800c610).stop 2019-12-05T06:14:03.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 -- 172.21.15.16:0/3315538564 shutdown_connections 2019-12-05T06:14:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd340023630 0x7fd340025ac0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd360156e70 0x7fd360159270 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 --2- 172.21.15.16:0/3315538564 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd3601302f0 0x7fd36009d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:03.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 -- 172.21.15.16:0/3315538564 shutdown_connections 2019-12-05T06:14:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 -- 172.21.15.16:0/3315538564 wait complete. 2019-12-05T06:14:03.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:03.452+0000 7fd366b10700 1 -- 172.21.15.16:0/3315538564 >> 172.21.15.16:0/3315538564 conn(0x7fd3600a2850 msgr2=0x7fd36006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:03.460 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:14:05.461 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.749+0000 7f9bfb0ef700 1 Processor -- start 2019-12-05T06:14:05.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.749+0000 7f9bfb0ef700 1 -- start start 2019-12-05T06:14:05.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.751+0000 7f9bfb0ef700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf408c6d0 0x7f9bf408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:05.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.751+0000 7f9bfb0ef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bf408f720 0x7f9bf408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:05.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.751+0000 7f9bfb0ef700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf40958e0 con 0x7f9bf408fef0 2019-12-05T06:14:05.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.751+0000 7f9bfb0ef700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf40a3f70 con 0x7f9bf408f720 2019-12-05T06:14:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.751+0000 7f9bfb0ef700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf408d830 con 0x7f9bf408c6d0 2019-12-05T06:14:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.751+0000 7f9bf9e8d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bf408f720 0x7f9bf408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.752+0000 7f9bf8e8b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9bf408fef0 0x7f9bf408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:56192/0 (socket says 172.21.15.16:56192) 2019-12-05T06:14:05.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.752+0000 7f9bf8e8b700 1 -- 172.21.15.16:0/2419677492 learned_addr learned my addr 172.21.15.16:0/2419677492 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.752+0000 7f9bf968c700 1 --2- 172.21.15.16:0/2419677492 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf408c6d0 0x7f9bf408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.752+0000 7f9bf3fff700 1 -- 172.21.15.16:0/2419677492 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283571795 0 0) 0x7f9be8001100 con 0x7f9bf408fef0 2019-12-05T06:14:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.752+0000 7f9bf3fff700 1 -- 172.21.15.16:0/2419677492 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bdc002e80 con 0x7f9bf408fef0 2019-12-05T06:14:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.752+0000 7f9bf9e8d700 1 -- 172.21.15.16:0/2419677492 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf408c6d0 msgr2=0x7f9bf408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:05.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.752+0000 7f9bf9e8d700 1 --2- 172.21.15.16:0/2419677492 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf408c6d0 0x7f9bf408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.752+0000 7f9bf9e8d700 1 -- 172.21.15.16:0/2419677492 >> v1:172.21.15.76:6789/0 conn(0x7f9bf408fef0 legacy=0x7f9bf408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.752+0000 7f9bf9e8d700 1 -- 172.21.15.16:0/2419677492 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf408fbc0 con 0x7f9bf408f720 2019-12-05T06:14:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.752+0000 7f9bf968c700 1 --2- 172.21.15.16:0/2419677492 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf408c6d0 0x7f9bf408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:14:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bf9e8d700 1 --2- 172.21.15.16:0/2419677492 >> v2:172.21.15.16:3300/0 conn(0x7f9bf408f720 0x7f9bf408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9be400af50 tx=0x7f9be400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bf9e8d700 1 --2- 172.21.15.16:0/2419677492 >> v2:172.21.15.16:3300/0 conn(0x7f9bf408f720 0x7f9bf408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9be400af50 tx=0x7f9be400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bf9e8d700 1 --2- 172.21.15.16:0/2419677492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf408f720 0x7f9bf408d000 secure :-1 s=READY pgs=369 cs=0 l=1 rx=0x7f9be400af50 tx=0x7f9be400b760).ready entity=mon.0 client_cookie=505cba58ae2a801c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bf9e8d700 1 --2- 172.21.15.16:0/2419677492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf408f720 0x7f9bf408d000 secure :-1 s=THROTTLE_DONE pgs=369 cs=0 l=1 rx=0x7f9be400af50 tx=0x7f9be400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bf9e8d700 1 --2- 172.21.15.16:0/2419677492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf408f720 0x7f9bf408d000 secure :-1 s=THROTTLE_DONE pgs=369 cs=0 l=1 rx=0x7f9be400af50 tx=0x7f9be400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bf3fff700 1 -- 172.21.15.16:0/2419677492 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9be4009ba0 con 0x7f9bf408f720 2019-12-05T06:14:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bf9e8d700 1 --2- 172.21.15.16:0/2419677492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf408f720 0x7f9bf408d000 secure :-1 s=THROTTLE_DONE pgs=369 cs=0 l=1 rx=0x7f9be400af50 tx=0x7f9be400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bf3fff700 1 -- 172.21.15.16:0/2419677492 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be40097c0 con 0x7f9bf408f720 2019-12-05T06:14:05.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bf3fff700 1 -- 172.21.15.16:0/2419677492 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9be4000bc0 con 0x7f9bf408f720 2019-12-05T06:14:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/2419677492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf408f720 msgr2=0x7f9bf408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.753+0000 7f9bfb0ef700 1 --2- 172.21.15.16:0/2419677492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf408f720 0x7f9bf408d000 secure :-1 s=READY pgs=369 cs=0 l=1 rx=0x7f9be400af50 tx=0x7f9be400b760).stop 2019-12-05T06:14:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.755+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/2419677492 shutdown_connections 2019-12-05T06:14:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.755+0000 7f9bfb0ef700 1 --2- 172.21.15.16:0/2419677492 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf408f720 0x7f9bf408d000 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.755+0000 7f9bfb0ef700 1 --2- 172.21.15.16:0/2419677492 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf408c6d0 0x7f9bf408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.755+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/2419677492 shutdown_connections 2019-12-05T06:14:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.755+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/2419677492 wait complete. 2019-12-05T06:14:05.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.755+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/2419677492 >> 172.21.15.16:0/2419677492 conn(0x7f9bf4061940 msgr2=0x7f9bf4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.760+0000 7f9bfb0ef700 1 Processor -- start 2019-12-05T06:14:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.761+0000 7f9bfb0ef700 1 -- start start 2019-12-05T06:14:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.762+0000 7f9bfb0ef700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:05.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.762+0000 7f9bfb0ef700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf4089fb0 con 0x7f9bf406df10 2019-12-05T06:14:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.762+0000 7f9bfb0ef700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf406e300 con 0x7f9bf4156cb0 2019-12-05T06:14:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.762+0000 7f9bfb0ef700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf4097370 con 0x7f9bf40992d0 2019-12-05T06:14:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.762+0000 7f9bf968c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9bf406df10 0x7f9bf4098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34894/0 (socket says 172.21.15.16:34894) 2019-12-05T06:14:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.762+0000 7f9bf968c700 1 -- 172.21.15.16:0/240250880 learned_addr learned my addr 172.21.15.16:0/240250880 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.763+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:05.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.763+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796881965 0 0) 0x7f9bec00e110 con 0x7f9bf406df10 2019-12-05T06:14:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.763+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd4002e80 con 0x7f9bf406df10 2019-12-05T06:14:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.763+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169611443 0 0) 0x7f9be4000940 con 0x7f9bf4156cb0 2019-12-05T06:14:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.763+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9bd40030b0 con 0x7f9bf4156cb0 2019-12-05T06:14:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.763+0000 7f9bf8e8b700 1 -- 172.21.15.16:0/240250880 >> v1:172.21.15.76:6789/0 conn(0x7f9bf4156cb0 legacy=0x7f9bf41590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf8e8b700 1 -- 172.21.15.16:0/240250880 >> v1:172.21.15.16:6789/0 conn(0x7f9bf406df10 legacy=0x7f9bf4098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf8e8b700 1 -- 172.21.15.16:0/240250880 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf4097a00 con 0x7f9bf40992d0 2019-12-05T06:14:05.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9be8000e90 con 0x7f9bf40992d0 2019-12-05T06:14:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9be80041c0 con 0x7f9bf40992d0 2019-12-05T06:14:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/240250880 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf4097a00 con 0x7f9bf40992d0 2019-12-05T06:14:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9be8001220 con 0x7f9bf40992d0 2019-12-05T06:14:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.764+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/240250880 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf4097a00 con 0x7f9bf40992d0 2019-12-05T06:14:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.765+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.766+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9be80043b0 con 0x7f9bf40992d0 2019-12-05T06:14:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.766+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.767+0000 7f9bf1ffb700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd4023650 0x7f9bd4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:05.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.767+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6289+0+0 (secure 0 0 0) 0x7f9be8045da0 con 0x7f9bf40992d0 2019-12-05T06:14:05.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.767+0000 7f9bf968c700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd4023650 0x7f9bd4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:05.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.767+0000 7f9bf968c700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd4023650 0x7f9bd4025ae0 crc :-1 s=READY pgs=171 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:05.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.780+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/240250880 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf4097a00 con 0x7f9bf40992d0 2019-12-05T06:14:05.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.790+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:05.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:05.790+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9be8000c10 con 0x7f9bf40992d0 2019-12-05T06:14:06.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.050+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/240250880 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bf4097a00 con 0x7f9bf40992d0 2019-12-05T06:14:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.051+0000 7f9bf8e8b700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=THROTTLE_DONE pgs=103 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:06.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.051+0000 7f9bf1ffb700 1 -- 172.21.15.16:0/240250880 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f9be8004c60 con 0x7f9bf40992d0 2019-12-05T06:14:06.047 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 35 2019-12-05T06:14:06.048 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:14:06.048 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":35,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5049,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":35}],"filesystems":[{"mdsmap":{"epoch":34,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:12:23.035737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":41,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4532},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4532":{"gid":4532,"name":"b","rank":0,"incarnation":30,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/913501478","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":913501478},{"type":"v1","addr":"172.21.15.76:6835","nonce":913501478}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:06.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.055+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd4023650 msgr2=0x7f9bd4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:06.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.055+0000 7f9bfb0ef700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd4023650 0x7f9bd4025ae0 crc :-1 s=READY pgs=171 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:06.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.055+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 msgr2=0x7f9bf4098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.056+0000 7f9bfb0ef700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 secure :-1 s=READY pgs=103 cs=0 l=1 rx=0x7f9be8010360 tx=0x7f9be8010700).stop 2019-12-05T06:14:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.056+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/240250880 shutdown_connections 2019-12-05T06:14:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.056+0000 7f9bfb0ef700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd4023650 0x7f9bd4025ae0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:06.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.056+0000 7f9bfb0ef700 1 --2- 172.21.15.16:0/240250880 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf40992d0 0x7f9bf4098b00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.056+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/240250880 shutdown_connections 2019-12-05T06:14:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.056+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/240250880 wait complete. 2019-12-05T06:14:06.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:06.056+0000 7f9bfb0ef700 1 -- 172.21.15.16:0/240250880 >> 172.21.15.16:0/240250880 conn(0x7f9bf40714f0 msgr2=0x7f9bf409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:06.061 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:14:08.062 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.249+0000 7f19d7385700 1 Processor -- start 2019-12-05T06:14:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.249+0000 7f19d7385700 1 -- start start 2019-12-05T06:14:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.250+0000 7f19d7385700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f19d008c6d0 0x7f19d008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.250+0000 7f19d7385700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f19d008fef0 0x7f19d008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.250+0000 7f19d7385700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d00958e0 con 0x7f19d008ef20 2019-12-05T06:14:08.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.250+0000 7f19d7385700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d008df90 con 0x7f19d008c6d0 2019-12-05T06:14:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.250+0000 7f19d7385700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19d00a3f70 con 0x7f19d008fef0 2019-12-05T06:14:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.250+0000 7f19d5922700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f19d008c6d0 0x7f19d008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.250+0000 7f19d5922700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f19d008c6d0 0x7f19d008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:35846/0 (socket says 172.21.15.16:35846) 2019-12-05T06:14:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.250+0000 7f19d5121700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f19d008fef0 0x7f19d008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.250+0000 7f19d5922700 1 -- 172.21.15.16:0/4134528284 learned_addr learned my addr 172.21.15.16:0/4134528284 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.251+0000 7f19d5922700 1 -- 172.21.15.16:0/4134528284 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f19d008fef0 msgr2=0x7f19d008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:08.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.251+0000 7f19d5922700 1 --2- 172.21.15.16:0/4134528284 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f19d008fef0 0x7f19d008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.251+0000 7f19d5922700 1 -- 172.21.15.16:0/4134528284 >> v1:172.21.15.76:6789/0 conn(0x7f19d008ef20 legacy=0x7f19d008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.251+0000 7f19d5922700 1 -- 172.21.15.16:0/4134528284 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d008d540 con 0x7f19d008c6d0 2019-12-05T06:14:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.251+0000 7f19d5922700 1 --2- 172.21.15.16:0/4134528284 >> v2:172.21.15.76:3300/0 conn(0x7f19d008c6d0 0x7f19d008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f19c800fac0 tx=0x7f19c800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.251+0000 7f19d5922700 1 --2- 172.21.15.16:0/4134528284 >> v2:172.21.15.76:3300/0 conn(0x7f19d008c6d0 0x7f19d008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f19c800fac0 tx=0x7f19c800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.251+0000 7f19d5922700 1 --2- 172.21.15.16:0/4134528284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19d008c6d0 0x7f19d008cac0 secure :-1 s=READY pgs=292 cs=0 l=1 rx=0x7f19c800fac0 tx=0x7f19c800f8f0).ready entity=mon.1 client_cookie=c5d37eb0e8fa8d93 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:08.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.252+0000 7f19d5922700 1 --2- 172.21.15.16:0/4134528284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19d008c6d0 0x7f19d008cac0 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f19c800fac0 tx=0x7f19c800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.252+0000 7f19d4920700 1 -- 172.21.15.16:0/4134528284 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19c8000ce0 con 0x7f19d008c6d0 2019-12-05T06:14:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.252+0000 7f19d5922700 1 --2- 172.21.15.16:0/4134528284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19d008c6d0 0x7f19d008cac0 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f19c800fac0 tx=0x7f19c800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.252+0000 7f19d5922700 1 --2- 172.21.15.16:0/4134528284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19d008c6d0 0x7f19d008cac0 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7f19c800fac0 tx=0x7f19c800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.252+0000 7f19d4920700 1 -- 172.21.15.16:0/4134528284 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c8007590 con 0x7f19d008c6d0 2019-12-05T06:14:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.252+0000 7f19d4920700 1 -- 172.21.15.16:0/4134528284 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19c8001200 con 0x7f19d008c6d0 2019-12-05T06:14:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.252+0000 7f19d7385700 1 -- 172.21.15.16:0/4134528284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19d008c6d0 msgr2=0x7f19d008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.252+0000 7f19d7385700 1 --2- 172.21.15.16:0/4134528284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19d008c6d0 0x7f19d008cac0 secure :-1 s=READY pgs=292 cs=0 l=1 rx=0x7f19c800fac0 tx=0x7f19c800f8f0).stop 2019-12-05T06:14:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.253+0000 7f19d7385700 1 -- 172.21.15.16:0/4134528284 shutdown_connections 2019-12-05T06:14:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.253+0000 7f19d7385700 1 --2- 172.21.15.16:0/4134528284 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f19d008fef0 0x7f19d008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:08.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.253+0000 7f19d7385700 1 --2- 172.21.15.16:0/4134528284 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f19d008c6d0 0x7f19d008cac0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.253+0000 7f19d7385700 1 -- 172.21.15.16:0/4134528284 shutdown_connections 2019-12-05T06:14:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.253+0000 7f19d7385700 1 -- 172.21.15.16:0/4134528284 wait complete. 2019-12-05T06:14:08.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.253+0000 7f19d7385700 1 -- 172.21.15.16:0/4134528284 >> 172.21.15.16:0/4134528284 conn(0x7f19d0061940 msgr2=0x7f19d0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:08.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.257+0000 7f19d7385700 1 Processor -- start 2019-12-05T06:14:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.257+0000 7f19d7385700 1 -- start start 2019-12-05T06:14:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.258+0000 7f19d7385700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f19d009c110 0x7f19d009e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.258+0000 7f19d7385700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19d009f7c0 0x7f19d009ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.258+0000 7f19d7385700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19d009ca80 con 0x7f19d009f270 2019-12-05T06:14:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.258+0000 7f19d7385700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d012d280 con 0x7f19d009f7c0 2019-12-05T06:14:08.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.258+0000 7f19d7385700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19d012d550 con 0x7f19d009c110 2019-12-05T06:14:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5121700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19d009f7c0 0x7f19d009ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5121700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f19d009f7c0 0x7f19d009ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49764/0 (socket says 172.21.15.16:49764) 2019-12-05T06:14:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5121700 1 -- 172.21.15.16:0/3484648511 learned_addr learned my addr 172.21.15.16:0/3484648511 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5922700 1 --2- 172.21.15.16:0/3484648511 >> v2:172.21.15.76:3300/0 conn(0x7f19d009c110 0x7f19d009e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5121700 1 -- 172.21.15.16:0/3484648511 >> v2:172.21.15.76:3300/0 conn(0x7f19d009c110 msgr2=0x7f19d009e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> v2:172.21.15.76:3300/0 conn(0x7f19d009c110 0x7f19d009e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5121700 1 -- 172.21.15.16:0/3484648511 >> v1:172.21.15.16:6789/0 conn(0x7f19d009f270 legacy=0x7f19d012b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5121700 1 -- 172.21.15.16:0/3484648511 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19d009def0 con 0x7f19d009f7c0 2019-12-05T06:14:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> v2:172.21.15.16:3300/0 conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> v2:172.21.15.16:3300/0 conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.259+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=READY pgs=373 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.260+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.260+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.260+0000 7f19c27fc700 1 -- 172.21.15.16:0/3484648511 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19c4000940 con 0x7f19d009f7c0 2019-12-05T06:14:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.260+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.260+0000 7f19c27fc700 1 -- 172.21.15.16:0/3484648511 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19c4001090 con 0x7f19d009f7c0 2019-12-05T06:14:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.260+0000 7f19d7385700 1 -- 172.21.15.16:0/3484648511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19d009def0 con 0x7f19d009f7c0 2019-12-05T06:14:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.260+0000 7f19c27fc700 1 -- 172.21.15.16:0/3484648511 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19c4001720 con 0x7f19d009f7c0 2019-12-05T06:14:08.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.260+0000 7f19d7385700 1 -- 172.21.15.16:0/3484648511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19d009def0 con 0x7f19d009f7c0 2019-12-05T06:14:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.261+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.261+0000 7f19c27fc700 1 -- 172.21.15.16:0/3484648511 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f19c4000c90 con 0x7f19d009f7c0 2019-12-05T06:14:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.261+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.261+0000 7f19c27fc700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f19b8023630 0x7f19b8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.261+0000 7f19c27fc700 1 -- 172.21.15.16:0/3484648511 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f19c4047f60 con 0x7f19d009f7c0 2019-12-05T06:14:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.262+0000 7f19d5922700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f19b8023630 0x7f19b8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:08.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.262+0000 7f19d5922700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f19b8023630 0x7f19b8025ac0 crc :-1 s=READY pgs=172 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:08.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.268+0000 7f19d7385700 1 -- 172.21.15.16:0/3484648511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19d009def0 con 0x7f19d009f7c0 2019-12-05T06:14:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.274+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.274+0000 7f19c27fc700 1 -- 172.21.15.16:0/3484648511 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f19c4000940 con 0x7f19d009f7c0 2019-12-05T06:14:08.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.468+0000 7f19d7385700 1 -- 172.21.15.16:0/3484648511 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f19d009def0 con 0x7f19d009f7c0 2019-12-05T06:14:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.469+0000 7f19d5121700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=THROTTLE_DONE pgs=373 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.469+0000 7f19c27fc700 1 -- 172.21.15.16:0/3484648511 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f19c4000c90 con 0x7f19d009f7c0 2019-12-05T06:14:08.464 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2019-12-05T06:14:08.464 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:14:08.464 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:08.011695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:08.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 -- 172.21.15.16:0/3484648511 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f19b8023630 msgr2=0x7f19b8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f19b8023630 0x7f19b8025ac0 crc :-1 s=READY pgs=172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 -- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 msgr2=0x7f19d009ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 0x7f19d009ed30 secure :-1 s=READY pgs=373 cs=0 l=1 rx=0x7f19c40102b0 tx=0x7f19c40128f0).stop 2019-12-05T06:14:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 -- 172.21.15.16:0/3484648511 shutdown_connections 2019-12-05T06:14:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f19b8023630 0x7f19b8025ac0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 --2- 172.21.15.16:0/3484648511 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f19d009f7c0 0x7f19d009ed30 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 --2- 172.21.15.16:0/3484648511 >> v2:172.21.15.76:3300/0 conn(0x7f19d009c110 0x7f19d009e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 -- 172.21.15.16:0/3484648511 shutdown_connections 2019-12-05T06:14:08.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 -- 172.21.15.16:0/3484648511 wait complete. 2019-12-05T06:14:08.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.472+0000 7f19d7385700 1 -- 172.21.15.16:0/3484648511 >> 172.21.15.16:0/3484648511 conn(0x7f19d008acd0 msgr2=0x7f19d0099650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:08.477 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:14:08.477 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:14:08.478 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.631+0000 7f1a52907700 1 Processor -- start 2019-12-05T06:14:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.631+0000 7f1a52907700 1 -- start start 2019-12-05T06:14:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a52907700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c08c6d0 0x7f1a4c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a52907700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a4c08f720 0x7f1a4c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a52907700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4c0958e0 con 0x7f1a4c08fef0 2019-12-05T06:14:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a52907700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4c0a3f70 con 0x7f1a4c08f720 2019-12-05T06:14:08.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a52907700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c08d830 con 0x7f1a4c08c6d0 2019-12-05T06:14:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a516a5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a4c08f720 0x7f1a4c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a516a5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a4c08f720 0x7f1a4c08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49774/0 (socket says 172.21.15.16:49774) 2019-12-05T06:14:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a516a5700 1 -- 172.21.15.16:0/1705231861 learned_addr learned my addr 172.21.15.16:0/1705231861 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/1705231861 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c08c6d0 0x7f1a4c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a516a5700 1 -- 172.21.15.16:0/1705231861 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c08c6d0 msgr2=0x7f1a4c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a516a5700 1 --2- 172.21.15.16:0/1705231861 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c08c6d0 0x7f1a4c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a516a5700 1 -- 172.21.15.16:0/1705231861 >> v1:172.21.15.76:6789/0 conn(0x7f1a4c08fef0 legacy=0x7f1a4c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.632+0000 7f1a516a5700 1 -- 172.21.15.16:0/1705231861 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c08fbc0 con 0x7f1a4c08f720 2019-12-05T06:14:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a516a5700 1 --2- 172.21.15.16:0/1705231861 >> v2:172.21.15.16:3300/0 conn(0x7f1a4c08f720 0x7f1a4c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1a4400fa30 tx=0x7f1a4400fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a516a5700 1 --2- 172.21.15.16:0/1705231861 >> v2:172.21.15.16:3300/0 conn(0x7f1a4c08f720 0x7f1a4c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1a4400fa30 tx=0x7f1a4400fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a516a5700 1 --2- 172.21.15.16:0/1705231861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c08f720 0x7f1a4c08d000 secure :-1 s=READY pgs=374 cs=0 l=1 rx=0x7f1a4400fa30 tx=0x7f1a4400fdd0).ready entity=mon.0 client_cookie=c1fef5258b2b370 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a516a5700 1 --2- 172.21.15.16:0/1705231861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c08f720 0x7f1a4c08d000 secure :-1 s=THROTTLE_DONE pgs=374 cs=0 l=1 rx=0x7f1a4400fa30 tx=0x7f1a4400fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a516a5700 1 --2- 172.21.15.16:0/1705231861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c08f720 0x7f1a4c08d000 secure :-1 s=THROTTLE_DONE pgs=374 cs=0 l=1 rx=0x7f1a4400fa30 tx=0x7f1a4400fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a4b7fe700 1 -- 172.21.15.16:0/1705231861 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a44000df0 con 0x7f1a4c08f720 2019-12-05T06:14:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a516a5700 1 --2- 172.21.15.16:0/1705231861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c08f720 0x7f1a4c08d000 secure :-1 s=THROTTLE_DONE pgs=374 cs=0 l=1 rx=0x7f1a4400fa30 tx=0x7f1a4400fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a4b7fe700 1 -- 172.21.15.16:0/1705231861 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a44001460 con 0x7f1a4c08f720 2019-12-05T06:14:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a4b7fe700 1 -- 172.21.15.16:0/1705231861 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a44001b30 con 0x7f1a4c08f720 2019-12-05T06:14:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a52907700 1 -- 172.21.15.16:0/1705231861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c08f720 msgr2=0x7f1a4c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.633+0000 7f1a52907700 1 --2- 172.21.15.16:0/1705231861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c08f720 0x7f1a4c08d000 secure :-1 s=READY pgs=374 cs=0 l=1 rx=0x7f1a4400fa30 tx=0x7f1a4400fdd0).stop 2019-12-05T06:14:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.634+0000 7f1a52907700 1 -- 172.21.15.16:0/1705231861 shutdown_connections 2019-12-05T06:14:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.634+0000 7f1a52907700 1 --2- 172.21.15.16:0/1705231861 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c08f720 0x7f1a4c08d000 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.634+0000 7f1a52907700 1 --2- 172.21.15.16:0/1705231861 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c08c6d0 0x7f1a4c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.634+0000 7f1a52907700 1 -- 172.21.15.16:0/1705231861 shutdown_connections 2019-12-05T06:14:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.634+0000 7f1a52907700 1 -- 172.21.15.16:0/1705231861 wait complete. 2019-12-05T06:14:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.634+0000 7f1a52907700 1 -- 172.21.15.16:0/1705231861 >> 172.21.15.16:0/1705231861 conn(0x7f1a4c061940 msgr2=0x7f1a4c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.636+0000 7f1a52907700 1 Processor -- start 2019-12-05T06:14:08.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.636+0000 7f1a52907700 1 -- start start 2019-12-05T06:14:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.637+0000 7f1a52907700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1a4c096e90 0x7f1a4c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.637+0000 7f1a52907700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c084f30 0x7f1a4c09b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.637+0000 7f1a52907700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a4c084b90 con 0x7f1a4c097280 2019-12-05T06:14:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.637+0000 7f1a52907700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a4c09a450 con 0x7f1a4c096e90 2019-12-05T06:14:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.637+0000 7f1a52907700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4c099a20 con 0x7f1a4c084f30 2019-12-05T06:14:08.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.637+0000 7f1a516a5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1a4c097280 0x7f1a4c0976c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34946/0 (socket says 172.21.15.16:34946) 2019-12-05T06:14:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.637+0000 7f1a516a5700 1 -- 172.21.15.16:0/460654970 learned_addr learned my addr 172.21.15.16:0/460654970 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.637+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> v2:172.21.15.16:3300/0 conn(0x7f1a4c096e90 0x7f1a4c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:08.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a50ea4700 1 -- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c084f30 msgr2=0x7f1a4c09b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a4bfff700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c084f30 0x7f1a4c09b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a497fa700 1 -- 172.21.15.16:0/460654970 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3213571211 0 0) 0x7f1a44000cc0 con 0x7f1a4c097280 2019-12-05T06:14:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c084f30 0x7f1a4c09b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a50ea4700 1 -- 172.21.15.16:0/460654970 >> v1:172.21.15.16:6789/0 conn(0x7f1a4c097280 legacy=0x7f1a4c0976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a50ea4700 1 -- 172.21.15.16:0/460654970 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4c09aae0 con 0x7f1a4c096e90 2019-12-05T06:14:08.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a4bfff700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c084f30 0x7f1a4c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:14:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> v2:172.21.15.16:3300/0 conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> v2:172.21.15.16:3300/0 conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=READY pgs=376 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a497fa700 1 -- 172.21.15.16:0/460654970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a34000940 con 0x7f1a4c096e90 2019-12-05T06:14:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a497fa700 1 -- 172.21.15.16:0/460654970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a34001170 con 0x7f1a4c096e90 2019-12-05T06:14:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a497fa700 1 -- 172.21.15.16:0/460654970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a34001800 con 0x7f1a4c096e90 2019-12-05T06:14:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a52907700 1 -- 172.21.15.16:0/460654970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4c09aae0 con 0x7f1a4c096e90 2019-12-05T06:14:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.638+0000 7f1a52907700 1 -- 172.21.15.16:0/460654970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a4c09aae0 con 0x7f1a4c096e90 2019-12-05T06:14:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.639+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.639+0000 7f1a497fa700 1 -- 172.21.15.16:0/460654970 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1a34000d70 con 0x7f1a4c096e90 2019-12-05T06:14:08.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.639+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.639+0000 7f1a497fa700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1a30023630 0x7f1a30025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.639+0000 7f1a497fa700 1 -- 172.21.15.16:0/460654970 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f1a34047f40 con 0x7f1a4c096e90 2019-12-05T06:14:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.640+0000 7f1a4bfff700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1a30023630 0x7f1a30025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:08.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.640+0000 7f1a4bfff700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1a30023630 0x7f1a30025ac0 crc :-1 s=READY pgs=173 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.647+0000 7f1a52907700 1 -- 172.21.15.16:0/460654970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a4c09aae0 con 0x7f1a4c096e90 2019-12-05T06:14:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.650+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.650+0000 7f1a497fa700 1 -- 172.21.15.16:0/460654970 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1a340458d0 con 0x7f1a4c096e90 2019-12-05T06:14:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.851+0000 7f1a52907700 1 -- 172.21.15.16:0/460654970 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a4c09aae0 con 0x7f1a4c096e90 2019-12-05T06:14:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.852+0000 7f1a50ea4700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.852+0000 7f1a497fa700 1 -- 172.21.15.16:0/460654970 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f1a340458d0 con 0x7f1a4c096e90 2019-12-05T06:14:08.847 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 36 2019-12-05T06:14:08.847 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:14:08.847 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":36,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":36,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:08.011695+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:08.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.855+0000 7f1a52907700 1 -- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1a30023630 msgr2=0x7f1a30025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.855+0000 7f1a52907700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1a30023630 0x7f1a30025ac0 crc :-1 s=READY pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.855+0000 7f1a52907700 1 -- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 msgr2=0x7f1a4c09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.856+0000 7f1a52907700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 0x7f1a4c09b3e0 secure :-1 s=READY pgs=376 cs=0 l=1 rx=0x7f1a34010380 tx=0x7f1a34010720).stop 2019-12-05T06:14:08.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.856+0000 7f1a52907700 1 -- 172.21.15.16:0/460654970 shutdown_connections 2019-12-05T06:14:08.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.856+0000 7f1a52907700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1a4c084f30 0x7f1a4c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.856+0000 7f1a52907700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1a30023630 0x7f1a30025ac0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.856+0000 7f1a52907700 1 --2- 172.21.15.16:0/460654970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1a4c096e90 0x7f1a4c09b3e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:08.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.856+0000 7f1a52907700 1 -- 172.21.15.16:0/460654970 shutdown_connections 2019-12-05T06:14:08.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.856+0000 7f1a52907700 1 -- 172.21.15.16:0/460654970 wait complete. 2019-12-05T06:14:08.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:08.856+0000 7f1a52907700 1 -- 172.21.15.16:0/460654970 >> 172.21.15.16:0/460654970 conn(0x7f1a4c0714f0 msgr2=0x7f1a4c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:08.861 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 36, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 36, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5049": { "addr": "172.21.15.16:6833/4256914848", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 4256914848, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 4256914848, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5049, "incarnation": 36, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 42, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:14:08.011695+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5049 } } } ], "standbys": [] } 2019-12-05T06:14:10.862 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.131+0000 7f8f5bbd9700 1 Processor -- start 2019-12-05T06:14:11.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.131+0000 7f8f5bbd9700 1 -- start start 2019-12-05T06:14:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.133+0000 7f8f5bbd9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8f5408fef0 0x7f8f5408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.133+0000 7f8f5bbd9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f540958e0 con 0x7f8f5408d2c0 2019-12-05T06:14:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.133+0000 7f8f5bbd9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f540629e0 con 0x7f8f5408c6d0 2019-12-05T06:14:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.133+0000 7f8f5bbd9700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f5408df90 con 0x7f8f5408fef0 2019-12-05T06:14:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.133+0000 7f8f59975700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8f5408d2c0 0x7f8f5408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34956/0 (socket says 172.21.15.16:34956) 2019-12-05T06:14:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.133+0000 7f8f59975700 1 -- 172.21.15.16:0/3640360654 learned_addr learned my addr 172.21.15.16:0/3640360654 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:11.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.134+0000 7f8f5a977700 1 --2- 172.21.15.16:0/3640360654 >> v2:172.21.15.76:3300/0 conn(0x7f8f5408fef0 0x7f8f5408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:11.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.134+0000 7f8f59174700 1 -- 172.21.15.16:0/3640360654 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701253637 0 0) 0x7f8f44001150 con 0x7f8f5408d2c0 2019-12-05T06:14:11.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.134+0000 7f8f59174700 1 -- 172.21.15.16:0/3640360654 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f40002e80 con 0x7f8f5408d2c0 2019-12-05T06:14:11.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.134+0000 7f8f59174700 1 -- 172.21.15.16:0/3640360654 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128363252 0 0) 0x7f8f50001040 con 0x7f8f5408c6d0 2019-12-05T06:14:11.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.134+0000 7f8f59174700 1 -- 172.21.15.16:0/3640360654 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f40002e80 con 0x7f8f5408c6d0 2019-12-05T06:14:11.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.134+0000 7f8f5a977700 1 -- 172.21.15.16:0/3640360654 >> v1:172.21.15.76:6789/0 conn(0x7f8f5408c6d0 legacy=0x7f8f5408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:11.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.134+0000 7f8f5a977700 1 -- 172.21.15.16:0/3640360654 >> v1:172.21.15.16:6789/0 conn(0x7f8f5408d2c0 legacy=0x7f8f5408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:11.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.134+0000 7f8f59174700 1 -- 172.21.15.16:0/3640360654 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3560375173 0 0) 0x7f8f44001130 con 0x7f8f5408d2c0 2019-12-05T06:14:11.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.134+0000 7f8f5a977700 1 -- 172.21.15.16:0/3640360654 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f5408e620 con 0x7f8f5408fef0 2019-12-05T06:14:11.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.134+0000 7f8f5a977700 1 --2- 172.21.15.16:0/3640360654 >> v2:172.21.15.76:3300/0 conn(0x7f8f5408fef0 0x7f8f5408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8f4c00b300 tx=0x7f8f4c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.135+0000 7f8f5a977700 1 --2- 172.21.15.16:0/3640360654 >> v2:172.21.15.76:3300/0 conn(0x7f8f5408fef0 0x7f8f5408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8f4c00b300 tx=0x7f8f4c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.135+0000 7f8f5a977700 1 --2- 172.21.15.16:0/3640360654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f5408fef0 0x7f8f5408f720 secure :-1 s=READY pgs=295 cs=0 l=1 rx=0x7f8f4c00b300 tx=0x7f8f4c00bb10).ready entity=mon.1 client_cookie=b8419fed57d18d58 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:11.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.135+0000 7f8f5a977700 1 --2- 172.21.15.16:0/3640360654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f5408fef0 0x7f8f5408f720 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f8f4c00b300 tx=0x7f8f4c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.135+0000 7f8f5a977700 1 --2- 172.21.15.16:0/3640360654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f5408fef0 0x7f8f5408f720 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f8f4c00b300 tx=0x7f8f4c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.135+0000 7f8f59174700 1 -- 172.21.15.16:0/3640360654 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f4c0075c0 con 0x7f8f5408fef0 2019-12-05T06:14:11.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.135+0000 7f8f5a977700 1 --2- 172.21.15.16:0/3640360654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f5408fef0 0x7f8f5408f720 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f8f4c00b300 tx=0x7f8f4c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.135+0000 7f8f59174700 1 -- 172.21.15.16:0/3640360654 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f4c009d60 con 0x7f8f5408fef0 2019-12-05T06:14:11.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.135+0000 7f8f59174700 1 -- 172.21.15.16:0/3640360654 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f4c00a3f0 con 0x7f8f5408fef0 2019-12-05T06:14:11.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.135+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/3640360654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f5408fef0 msgr2=0x7f8f5408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:11.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.135+0000 7f8f5bbd9700 1 --2- 172.21.15.16:0/3640360654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f5408fef0 0x7f8f5408f720 secure :-1 s=READY pgs=295 cs=0 l=1 rx=0x7f8f4c00b300 tx=0x7f8f4c00bb10).stop 2019-12-05T06:14:11.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.137+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/3640360654 shutdown_connections 2019-12-05T06:14:11.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.137+0000 7f8f5bbd9700 1 --2- 172.21.15.16:0/3640360654 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8f5408fef0 0x7f8f5408f720 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:11.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.137+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/3640360654 shutdown_connections 2019-12-05T06:14:11.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.137+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/3640360654 wait complete. 2019-12-05T06:14:11.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.137+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/3640360654 >> 172.21.15.16:0/3640360654 conn(0x7f8f54061940 msgr2=0x7f8f54096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:11.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.141+0000 7f8f5bbd9700 1 Processor -- start 2019-12-05T06:14:11.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.142+0000 7f8f5bbd9700 1 -- start start 2019-12-05T06:14:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.143+0000 7f8f5bbd9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8f541302f0 0x7f8f5412a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.143+0000 7f8f5bbd9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:11.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.143+0000 7f8f5bbd9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f54131bf0 con 0x7f8f5412b0f0 2019-12-05T06:14:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.143+0000 7f8f5bbd9700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f541296e0 con 0x7f8f541302f0 2019-12-05T06:14:11.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.143+0000 7f8f5bbd9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f54128cb0 con 0x7f8f5412b640 2019-12-05T06:14:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.143+0000 7f8f5a176700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8f541302f0 0x7f8f5412a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.143+0000 7f8f59975700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.143+0000 7f8f5a977700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8f5412b0f0 0x7f8f541306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:34964/0 (socket says 172.21.15.16:34964) 2019-12-05T06:14:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.143+0000 7f8f5a977700 1 -- 172.21.15.16:0/2456885527 learned_addr learned my addr 172.21.15.16:0/2456885527 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.144+0000 7f8f4affd700 1 -- 172.21.15.16:0/2456885527 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1108442786 0 0) 0x7f8f4c009930 con 0x7f8f5412b0f0 2019-12-05T06:14:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.144+0000 7f8f4affd700 1 -- 172.21.15.16:0/2456885527 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f3c002e80 con 0x7f8f5412b0f0 2019-12-05T06:14:11.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.144+0000 7f8f59975700 1 -- 172.21.15.16:0/2456885527 >> v2:172.21.15.76:3300/0 conn(0x7f8f541302f0 msgr2=0x7f8f5412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.144+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> v2:172.21.15.76:3300/0 conn(0x7f8f541302f0 0x7f8f5412a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.144+0000 7f8f59975700 1 -- 172.21.15.16:0/2456885527 >> v1:172.21.15.16:6789/0 conn(0x7f8f5412b0f0 legacy=0x7f8f541306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.144+0000 7f8f59975700 1 -- 172.21.15.16:0/2456885527 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54129d70 con 0x7f8f5412b640 2019-12-05T06:14:11.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.144+0000 7f8f5a176700 1 --2- 172.21.15.16:0/2456885527 >> v2:172.21.15.76:3300/0 conn(0x7f8f541302f0 0x7f8f5412a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:14:11.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.144+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.144+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.144+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:11.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.145+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.145+0000 7f8f4affd700 1 -- 172.21.15.16:0/2456885527 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f44000c90 con 0x7f8f5412b640 2019-12-05T06:14:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.145+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.145+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.145+0000 7f8f4affd700 1 -- 172.21.15.16:0/2456885527 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f44001420 con 0x7f8f5412b640 2019-12-05T06:14:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.145+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/2456885527 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f54129d70 con 0x7f8f5412b640 2019-12-05T06:14:11.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.145+0000 7f8f4affd700 1 -- 172.21.15.16:0/2456885527 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f44001b70 con 0x7f8f5412b640 2019-12-05T06:14:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.145+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/2456885527 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f54129d70 con 0x7f8f5412b640 2019-12-05T06:14:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.146+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.147+0000 7f8f4affd700 1 -- 172.21.15.16:0/2456885527 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8f440010b0 con 0x7f8f5412b640 2019-12-05T06:14:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.147+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.147+0000 7f8f4affd700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8f3c023670 0x7f8f3c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.147+0000 7f8f4affd700 1 -- 172.21.15.16:0/2456885527 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f8f44047ef0 con 0x7f8f5412b640 2019-12-05T06:14:11.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.147+0000 7f8f5a176700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8f3c023670 0x7f8f3c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:11.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.148+0000 7f8f5a176700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8f3c023670 0x7f8f3c025ae0 crc :-1 s=READY pgs=174 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:11.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.163+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/2456885527 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f54129d70 con 0x7f8f5412b640 2019-12-05T06:14:11.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.171+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.171+0000 7f8f4affd700 1 -- 172.21.15.16:0/2456885527 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8f44000ea0 con 0x7f8f5412b640 2019-12-05T06:14:11.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.419+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/2456885527 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f54129d70 con 0x7f8f5412b640 2019-12-05T06:14:11.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.420+0000 7f8f59975700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=THROTTLE_DONE pgs=104 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:11.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.420+0000 7f8f4affd700 1 -- 172.21.15.16:0/2456885527 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 38 v38) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f8f440010b0 con 0x7f8f5412b640 2019-12-05T06:14:11.416 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 38 2019-12-05T06:14:11.416 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:14:11.417 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":38,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":38,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:11.118123+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:rejoin","state_seq":28,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:11.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8f3c023670 msgr2=0x7f8f3c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:11.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8f3c023670 0x7f8f3c025ae0 crc :-1 s=READY pgs=174 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 msgr2=0x7f8f5412abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 secure :-1 s=READY pgs=104 cs=0 l=1 rx=0x7f8f44012510 tx=0x7f8f440128b0).stop 2019-12-05T06:14:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/2456885527 shutdown_connections 2019-12-05T06:14:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8f3c023670 0x7f8f3c025ae0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:11.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 --2- 172.21.15.16:0/2456885527 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8f5412b640 0x7f8f5412abb0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 --2- 172.21.15.16:0/2456885527 >> v2:172.21.15.76:3300/0 conn(0x7f8f541302f0 0x7f8f5412a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/2456885527 shutdown_connections 2019-12-05T06:14:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/2456885527 wait complete. 2019-12-05T06:14:11.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:11.425+0000 7f8f5bbd9700 1 -- 172.21.15.16:0/2456885527 >> 172.21.15.16:0/2456885527 conn(0x7f8f540a2850 msgr2=0x7f8f5406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:11.431 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:14:13.220 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/177’ 2019-12-05T06:14:13.220 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/19’ 2019-12-05T06:14:13.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/15’ 2019-12-05T06:14:13.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/20’ 2019-12-05T06:14:13.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/12’ 2019-12-05T06:14:13.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/10’ 2019-12-05T06:14:13.221 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/11’ 2019-12-05T06:14:13.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/16’ 2019-12-05T06:14:13.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/14’ 2019-12-05T06:14:13.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/18’ 2019-12-05T06:14:13.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/13’ 2019-12-05T06:14:13.222 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/169/17’ 2019-12-05T06:14:13.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/169’ 2019-12-05T06:14:13.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/19’ 2019-12-05T06:14:13.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/15’ 2019-12-05T06:14:13.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/20’ 2019-12-05T06:14:13.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/12’ 2019-12-05T06:14:13.223 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/10’ 2019-12-05T06:14:13.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/11’ 2019-12-05T06:14:13.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/16’ 2019-12-05T06:14:13.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/14’ 2019-12-05T06:14:13.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/18’ 2019-12-05T06:14:13.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/13’ 2019-12-05T06:14:13.224 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/154/17’ 2019-12-05T06:14:13.225 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/154’ 2019-12-05T06:14:13.225 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/19’ 2019-12-05T06:14:13.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/15’ 2019-12-05T06:14:13.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/20’ 2019-12-05T06:14:13.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/12’ 2019-12-05T06:14:13.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/10’ 2019-12-05T06:14:13.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/11’ 2019-12-05T06:14:13.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/16’ 2019-12-05T06:14:13.226 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/14’ 2019-12-05T06:14:13.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/18’ 2019-12-05T06:14:13.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/13’ 2019-12-05T06:14:13.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/171/17’ 2019-12-05T06:14:13.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/171’ 2019-12-05T06:14:13.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/19’ 2019-12-05T06:14:13.227 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/15’ 2019-12-05T06:14:13.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/20’ 2019-12-05T06:14:13.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/12’ 2019-12-05T06:14:13.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/10’ 2019-12-05T06:14:13.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/11’ 2019-12-05T06:14:13.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/16’ 2019-12-05T06:14:13.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/14’ 2019-12-05T06:14:13.228 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/18’ 2019-12-05T06:14:13.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/13’ 2019-12-05T06:14:13.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/139/17’ 2019-12-05T06:14:13.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/139’ 2019-12-05T06:14:13.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/19’ 2019-12-05T06:14:13.229 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/15’ 2019-12-05T06:14:13.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/20’ 2019-12-05T06:14:13.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/12’ 2019-12-05T06:14:13.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/10’ 2019-12-05T06:14:13.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/11’ 2019-12-05T06:14:13.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/16’ 2019-12-05T06:14:13.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/14’ 2019-12-05T06:14:13.230 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/18’ 2019-12-05T06:14:13.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/13’ 2019-12-05T06:14:13.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/102/17’ 2019-12-05T06:14:13.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/102’ 2019-12-05T06:14:13.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/19’ 2019-12-05T06:14:13.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/15’ 2019-12-05T06:14:13.231 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/20’ 2019-12-05T06:14:13.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/12’ 2019-12-05T06:14:13.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/10’ 2019-12-05T06:14:13.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/11’ 2019-12-05T06:14:13.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/16’ 2019-12-05T06:14:13.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/14’ 2019-12-05T06:14:13.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/18’ 2019-12-05T06:14:13.232 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/13’ 2019-12-05T06:14:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed ‘./copyofsnap1/test1/129/17’ 2019-12-05T06:14:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1/129’ 2019-12-05T06:14:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1/test1’ 2019-12-05T06:14:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:removed directory: ‘./copyofsnap1’ 2019-12-05T06:14:13.233 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2019-12-05T06:14:13.236 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:14:13.236 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:14:13.366 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-authwb.sh... 2019-12-05T06:14:13.366 INFO:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snaptest-authwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-authwb.sh 2019-12-05T06:14:13.444 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:13.464 INFO:tasks.workunit.client.0.smithi016.stderr:+ set -e 2019-12-05T06:14:13.465 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:14:13.611 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.611+0000 7fddfc54d700 1 Processor -- start 2019-12-05T06:14:13.611 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.611+0000 7fddfc54d700 1 -- start start 2019-12-05T06:14:13.611 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.612+0000 7fddfc54d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fddf41490a0 0x7fddf4149490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:13.611 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.612+0000 7fddfc54d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddf409f1c0 0x7fddf409e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:13.612 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.612+0000 7fddfc54d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf405f670 con 0x7fddf409e730 2019-12-05T06:14:13.612 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.612+0000 7fddfc54d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf40a3f40 con 0x7fddf41490a0 2019-12-05T06:14:13.612 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.612+0000 7fddfc54d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddf409d260 con 0x7fddf409f1c0 2019-12-05T06:14:13.612 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.613+0000 7fddfaaea700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fddf41490a0 0x7fddf4149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:13.612 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.613+0000 7fddfb2eb700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fddf409e730 0x7fddf409ae70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:56292/0 (socket says 172.21.15.16:56292) 2019-12-05T06:14:13.612 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.613+0000 7fddfb2eb700 1 -- 172.21.15.16:0/737234544 learned_addr learned my addr 172.21.15.16:0/737234544 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:13.613 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.613+0000 7fddfa2e9700 1 --2- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddf409f1c0 0x7fddf409e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:13.613 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.613+0000 7fddfaaea700 1 -- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddf409f1c0 msgr2=0x7fddf409e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:13.613 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.613+0000 7fddfaaea700 1 --2- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddf409f1c0 0x7fddf409e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:13.613 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.613+0000 7fddfaaea700 1 -- 172.21.15.16:0/737234544 >> v1:172.21.15.76:6789/0 conn(0x7fddf409e730 legacy=0x7fddf409ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:13.613 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.613+0000 7fddfaaea700 1 -- 172.21.15.16:0/737234544 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf409d8f0 con 0x7fddf41490a0 2019-12-05T06:14:13.614 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.613+0000 7fddfaaea700 1 --2- 172.21.15.16:0/737234544 >> v2:172.21.15.76:3300/0 conn(0x7fddf41490a0 0x7fddf4149490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdde800b2d0 tx=0x7fdde800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.614 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.614+0000 7fddfaaea700 1 --2- 172.21.15.16:0/737234544 >> v2:172.21.15.76:3300/0 conn(0x7fddf41490a0 0x7fddf4149490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdde800b2d0 tx=0x7fdde800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.614 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.614+0000 7fddfaaea700 1 --2- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddf41490a0 0x7fddf4149490 secure :-1 s=READY pgs=297 cs=0 l=1 rx=0x7fdde800b2d0 tx=0x7fdde800b5d0).ready entity=mon.1 client_cookie=56e6481cf5790f1e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:13.614 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.614+0000 7fddfaaea700 1 --2- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddf41490a0 0x7fddf4149490 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7fdde800b2d0 tx=0x7fdde800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.614 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.614+0000 7fddf9ae8700 1 -- 172.21.15.16:0/737234544 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdde8007590 con 0x7fddf41490a0 2019-12-05T06:14:13.614 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.614+0000 7fddfaaea700 1 --2- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddf41490a0 0x7fddf4149490 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7fdde800b2d0 tx=0x7fdde800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.615 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.614+0000 7fddfaaea700 1 --2- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddf41490a0 0x7fddf4149490 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7fdde800b2d0 tx=0x7fdde800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.615 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.614+0000 7fddf9ae8700 1 -- 172.21.15.16:0/737234544 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde8009d30 con 0x7fddf41490a0 2019-12-05T06:14:13.615 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.614+0000 7fddf9ae8700 1 -- 172.21.15.16:0/737234544 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdde800a400 con 0x7fddf41490a0 2019-12-05T06:14:13.615 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.614+0000 7fddfc54d700 1 -- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddf41490a0 msgr2=0x7fddf4149490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:13.615 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.614+0000 7fddfc54d700 1 --2- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddf41490a0 0x7fddf4149490 secure :-1 s=READY pgs=297 cs=0 l=1 rx=0x7fdde800b2d0 tx=0x7fdde800b5d0).stop 2019-12-05T06:14:13.615 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.615+0000 7fddfc54d700 1 -- 172.21.15.16:0/737234544 shutdown_connections 2019-12-05T06:14:13.616 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.615+0000 7fddfc54d700 1 --2- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fddf409f1c0 0x7fddf409e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:13.616 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.615+0000 7fddfc54d700 1 --2- 172.21.15.16:0/737234544 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fddf41490a0 0x7fddf4149490 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:13.616 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.615+0000 7fddfc54d700 1 -- 172.21.15.16:0/737234544 shutdown_connections 2019-12-05T06:14:13.617 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.615+0000 7fddfc54d700 1 -- 172.21.15.16:0/737234544 wait complete. 2019-12-05T06:14:13.617 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.615+0000 7fddfc54d700 1 -- 172.21.15.16:0/737234544 >> 172.21.15.16:0/737234544 conn(0x7fddf4062220 msgr2=0x7fddf40a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:13.618 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.618+0000 7fddfc54d700 1 Processor -- start 2019-12-05T06:14:13.618 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.618+0000 7fddfc54d700 1 -- start start 2019-12-05T06:14:13.618 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.618+0000 7fddfc54d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fddf411cba0 0x7fddf411cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:13.619 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.618+0000 7fddfc54d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf41316a0 con 0x7fddf4131a10 2019-12-05T06:14:13.619 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.618+0000 7fddfc54d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddf4079f70 con 0x7fddf4130a40 2019-12-05T06:14:13.619 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.618+0000 7fddfc54d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddf40799f0 con 0x7fddf411cba0 2019-12-05T06:14:13.622 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.618+0000 7fddfaaea700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fddf411cba0 0x7fddf411cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:13.622 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.618+0000 7fddfaaea700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fddf411cba0 0x7fddf411cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49824/0 (socket says 172.21.15.16:49824) 2019-12-05T06:14:13.622 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.618+0000 7fddfaaea700 1 -- 172.21.15.16:0/1853688875 learned_addr learned my addr 172.21.15.16:0/1853688875 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:13.622 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfaaea700 1 -- 172.21.15.16:0/1853688875 >> v1:172.21.15.76:6789/0 conn(0x7fddf4130a40 legacy=0x7fddf4130ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:13.623 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfaaea700 1 -- 172.21.15.16:0/1853688875 >> v1:172.21.15.16:6789/0 conn(0x7fddf4131a10 legacy=0x7fddf411d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:13.623 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfaaea700 1 -- 172.21.15.16:0/1853688875 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddf4130140 con 0x7fddf411cba0 2019-12-05T06:14:13.623 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddf37fe700 1 -- 172.21.15.16:0/1853688875 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 283606132 0 0) 0x7fddec0010b0 con 0x7fddf4131a10 2019-12-05T06:14:13.623 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> v2:172.21.15.16:3300/0 conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.623 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> v2:172.21.15.16:3300/0 conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.623 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=READY pgs=380 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:13.624 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.624 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddf37fe700 1 -- 172.21.15.16:0/1853688875 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdde8000cc0 con 0x7fddf411cba0 2019-12-05T06:14:13.624 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.624 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.624 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddf37fe700 1 -- 172.21.15.16:0/1853688875 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdde8009d00 con 0x7fddf411cba0 2019-12-05T06:14:13.625 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddf37fe700 1 -- 172.21.15.16:0/1853688875 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdde8000cc0 con 0x7fddf411cba0 2019-12-05T06:14:13.625 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddf4130140 con 0x7fddf411cba0 2019-12-05T06:14:13.625 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.619+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddf4130140 con 0x7fddf411cba0 2019-12-05T06:14:13.625 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.620+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.625 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.621+0000 7fddf37fe700 1 -- 172.21.15.16:0/1853688875 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdde8009ef0 con 0x7fddf411cba0 2019-12-05T06:14:13.625 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.621+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.625 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.621+0000 7fddf37fe700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddd4023630 0x7fddd4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:13.626 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.621+0000 7fddf37fe700 1 -- 172.21.15.16:0/1853688875 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fdde80462b0 con 0x7fddf411cba0 2019-12-05T06:14:13.626 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.621+0000 7fddfa2e9700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddd4023630 0x7fddd4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:13.626 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.621+0000 7fddfa2e9700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddd4023630 0x7fddd4025ac0 crc :-1 s=READY pgs=175 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:13.626 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.627+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddf4130140 con 0x7fddf411cba0 2019-12-05T06:14:13.627 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.633+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.627 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.633+0000 7fddf37fe700 1 -- 172.21.15.16:0/1853688875 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdde8009900 con 0x7fddf411cba0 2019-12-05T06:14:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.645+0000 7f22bd3e1700 1 Processor -- start 2019-12-05T06:14:13.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.645+0000 7f22bd3e1700 1 -- start start 2019-12-05T06:14:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22bd3e1700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b808c6d0 0x7f22b808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22bd3e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b808f720 0x7f22b808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:13.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22bd3e1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b80958e0 con 0x7f22b808fef0 2019-12-05T06:14:13.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22bd3e1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b80a3f70 con 0x7f22b808f720 2019-12-05T06:14:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22bd3e1700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b808d830 con 0x7f22b808c6d0 2019-12-05T06:14:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22b7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b808f720 0x7f22b808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22b7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b808f720 0x7f22b808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49832/0 (socket says 172.21.15.16:49832) 2019-12-05T06:14:13.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22b7fff700 1 -- 172.21.15.16:0/2959418545 learned_addr learned my addr 172.21.15.16:0/2959418545 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:13.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22b77fe700 1 --2- 172.21.15.16:0/2959418545 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b808c6d0 0x7f22b808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:13.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22b7fff700 1 -- 172.21.15.16:0/2959418545 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b808c6d0 msgr2=0x7f22b808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:13.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22b7fff700 1 --2- 172.21.15.16:0/2959418545 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b808c6d0 0x7f22b808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:13.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22b7fff700 1 -- 172.21.15.16:0/2959418545 >> v1:172.21.15.76:6789/0 conn(0x7f22b808fef0 legacy=0x7f22b808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:13.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22b7fff700 1 -- 172.21.15.16:0/2959418545 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b808fbc0 con 0x7f22b808f720 2019-12-05T06:14:13.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22b77fe700 1 --2- 172.21.15.16:0/2959418545 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b808c6d0 0x7f22b808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:14:13.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.646+0000 7f22b7fff700 1 --2- 172.21.15.16:0/2959418545 >> v2:172.21.15.16:3300/0 conn(0x7f22b808f720 0x7f22b808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f22a800f730 tx=0x7f22a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.647+0000 7f22b7fff700 1 --2- 172.21.15.16:0/2959418545 >> v2:172.21.15.16:3300/0 conn(0x7f22b808f720 0x7f22b808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f22a800f730 tx=0x7f22a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.647+0000 7f22b7fff700 1 --2- 172.21.15.16:0/2959418545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b808f720 0x7f22b808d000 secure :-1 s=READY pgs=381 cs=0 l=1 rx=0x7f22a800f730 tx=0x7f22a800fad0).ready entity=mon.0 client_cookie=ff24155fb7622599 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:13.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.647+0000 7f22b7fff700 1 --2- 172.21.15.16:0/2959418545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b808f720 0x7f22b808d000 secure :-1 s=THROTTLE_DONE pgs=381 cs=0 l=1 rx=0x7f22a800f730 tx=0x7f22a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.647+0000 7f22b7fff700 1 --2- 172.21.15.16:0/2959418545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b808f720 0x7f22b808d000 secure :-1 s=THROTTLE_DONE pgs=381 cs=0 l=1 rx=0x7f22a800f730 tx=0x7f22a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.647+0000 7f22b7fff700 1 --2- 172.21.15.16:0/2959418545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b808f720 0x7f22b808d000 secure :-1 s=THROTTLE_DONE pgs=381 cs=0 l=1 rx=0x7f22a800f730 tx=0x7f22a800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.647+0000 7f22b67fc700 1 -- 172.21.15.16:0/2959418545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a8000b90 con 0x7f22b808f720 2019-12-05T06:14:13.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.647+0000 7f22b67fc700 1 -- 172.21.15.16:0/2959418545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a80010e0 con 0x7f22b808f720 2019-12-05T06:14:13.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.647+0000 7f22b67fc700 1 -- 172.21.15.16:0/2959418545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a8001830 con 0x7f22b808f720 2019-12-05T06:14:13.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.647+0000 7f22bd3e1700 1 -- 172.21.15.16:0/2959418545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b808f720 msgr2=0x7f22b808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:13.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.647+0000 7f22bd3e1700 1 --2- 172.21.15.16:0/2959418545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b808f720 0x7f22b808d000 secure :-1 s=READY pgs=381 cs=0 l=1 rx=0x7f22a800f730 tx=0x7f22a800fad0).stop 2019-12-05T06:14:13.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.648+0000 7f22bd3e1700 1 -- 172.21.15.16:0/2959418545 shutdown_connections 2019-12-05T06:14:13.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.648+0000 7f22bd3e1700 1 --2- 172.21.15.16:0/2959418545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b808f720 0x7f22b808d000 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:13.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.648+0000 7f22bd3e1700 1 --2- 172.21.15.16:0/2959418545 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b808c6d0 0x7f22b808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:13.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.648+0000 7f22bd3e1700 1 -- 172.21.15.16:0/2959418545 shutdown_connections 2019-12-05T06:14:13.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.648+0000 7f22bd3e1700 1 -- 172.21.15.16:0/2959418545 wait complete. 2019-12-05T06:14:13.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.648+0000 7f22bd3e1700 1 -- 172.21.15.16:0/2959418545 >> 172.21.15.16:0/2959418545 conn(0x7f22b8061940 msgr2=0x7f22b8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:13.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.651+0000 7f22bd3e1700 1 Processor -- start 2019-12-05T06:14:13.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.651+0000 7f22bd3e1700 1 -- start start 2019-12-05T06:14:13.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22bd3e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b809d3d0 0x7f22b8152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:13.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22bd3e1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b8133b50 con 0x7f22b8133ed0 2019-12-05T06:14:13.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22bd3e1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22b8139810 con 0x7f22b813c4b0 2019-12-05T06:14:13.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22bd3e1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b813b520 con 0x7f22b809d3d0 2019-12-05T06:14:13.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22b77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b809d3d0 0x7f22b8152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:13.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22b7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f22b8133ed0 0x7f22b819b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35010/0 (socket says 172.21.15.16:35010) 2019-12-05T06:14:13.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22b7fff700 1 -- 172.21.15.16:0/1489217197 learned_addr learned my addr 172.21.15.16:0/1489217197 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:13.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22affff700 1 -- 172.21.15.16:0/1489217197 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673338646 0 0) 0x7f22a8007580 con 0x7f22b8133ed0 2019-12-05T06:14:13.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22affff700 1 -- 172.21.15.16:0/1489217197 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2290002e80 con 0x7f22b8133ed0 2019-12-05T06:14:13.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22b77fe700 1 -- 172.21.15.16:0/1489217197 >> v1:172.21.15.76:6789/0 conn(0x7f22b813c4b0 legacy=0x7f22b8133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:13.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22affff700 1 -- 172.21.15.16:0/1489217197 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 844597323 0 0) 0x7f22b0001400 con 0x7f22b813c4b0 2019-12-05T06:14:13.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22b77fe700 1 -- 172.21.15.16:0/1489217197 >> v1:172.21.15.16:6789/0 conn(0x7f22b8133ed0 legacy=0x7f22b819b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:13.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.652+0000 7f22b77fe700 1 -- 172.21.15.16:0/1489217197 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b813bbb0 con 0x7f22b809d3d0 2019-12-05T06:14:13.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22b77fe700 1 --2- 172.21.15.16:0/1489217197 >> v2:172.21.15.16:3300/0 conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22b77fe700 1 --2- 172.21.15.16:0/1489217197 >> v2:172.21.15.16:3300/0 conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22b77fe700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=READY pgs=383 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:13.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22b77fe700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22affff700 1 -- 172.21.15.16:0/1489217197 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a00097a0 con 0x7f22b809d3d0 2019-12-05T06:14:13.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22b77fe700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22b77fe700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22affff700 1 -- 172.21.15.16:0/1489217197 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a0000db0 con 0x7f22b809d3d0 2019-12-05T06:14:13.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22affff700 1 -- 172.21.15.16:0/1489217197 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a00097a0 con 0x7f22b809d3d0 2019-12-05T06:14:13.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22bd3e1700 1 -- 172.21.15.16:0/1489217197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b813bbb0 con 0x7f22b809d3d0 2019-12-05T06:14:13.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.653+0000 7f22bd3e1700 1 -- 172.21.15.16:0/1489217197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b813bbb0 con 0x7f22b809d3d0 2019-12-05T06:14:13.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.654+0000 7f22b77fe700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.654+0000 7f22affff700 1 -- 172.21.15.16:0/1489217197 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f22a0000940 con 0x7f22b809d3d0 2019-12-05T06:14:13.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.654+0000 7f22b77fe700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.655+0000 7f22affff700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2290023680 0x7f2290025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:13.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.655+0000 7f22affff700 1 -- 172.21.15.16:0/1489217197 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f22a0045da0 con 0x7f22b809d3d0 2019-12-05T06:14:13.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.655+0000 7f22b6ffd700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2290023680 0x7f2290025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:13.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.655+0000 7f22b6ffd700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2290023680 0x7f2290025af0 crc :-1 s=READY pgs=176 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.661+0000 7f22bd3e1700 1 -- 172.21.15.16:0/1489217197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b813bbb0 con 0x7f22b809d3d0 2019-12-05T06:14:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.665+0000 7f22b77fe700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.665+0000 7f22affff700 1 -- 172.21.15.16:0/1489217197 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f22a0009750 con 0x7f22b809d3d0 2019-12-05T06:14:13.825 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:13.831+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fddf4130140 con 0x7fddf411cba0 2019-12-05T06:14:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.898+0000 7f22bd3e1700 1 -- 172.21.15.16:0/1489217197 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22b813bbb0 con 0x7f22b809d3d0 2019-12-05T06:14:13.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.899+0000 7f22b77fe700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:13.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.899+0000 7f22affff700 1 -- 172.21.15.16:0/1489217197 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f22a0000940 con 0x7f22b809d3d0 2019-12-05T06:14:13.895 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 40 2019-12-05T06:14:13.895 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:14:13.895 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":40,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:13.135462+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:13.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.902+0000 7f22bd3e1700 1 -- 172.21.15.16:0/1489217197 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2290023680 msgr2=0x7f2290025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:13.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.902+0000 7f22bd3e1700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2290023680 0x7f2290025af0 crc :-1 s=READY pgs=176 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:13.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.902+0000 7f22bd3e1700 1 -- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 msgr2=0x7f22b8152d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.902+0000 7f22bd3e1700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 0x7f22b8152d00 secure :-1 s=READY pgs=383 cs=0 l=1 rx=0x7f22a0010390 tx=0x7f22a0010730).stop 2019-12-05T06:14:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.902+0000 7f22bd3e1700 1 -- 172.21.15.16:0/1489217197 shutdown_connections 2019-12-05T06:14:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.902+0000 7f22bd3e1700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2290023680 0x7f2290025af0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.902+0000 7f22bd3e1700 1 --2- 172.21.15.16:0/1489217197 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f22b809d3d0 0x7f22b8152d00 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.903+0000 7f22bd3e1700 1 -- 172.21.15.16:0/1489217197 shutdown_connections 2019-12-05T06:14:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.903+0000 7f22bd3e1700 1 -- 172.21.15.16:0/1489217197 wait complete. 2019-12-05T06:14:13.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:13.903+0000 7f22bd3e1700 1 -- 172.21.15.16:0/1489217197 >> 172.21.15.16:0/1489217197 conn(0x7f22b80994e0 msgr2=0x7f22b809f270 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:13.907 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=4532 2019-12-05T06:14:13.908 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.b 2019-12-05T06:14:14.140 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:14.146+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:14.140 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:14.146+0000 7fddf37fe700 1 -- 172.21.15.16:0/1853688875 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v41) v1 ==== 167+0+0 (secure 0 0 0) 0x7fdde800a2c0 con 0x7fddf411cba0 2019-12-05T06:14:14.388 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:14.394+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fddf4130140 con 0x7fddf411cba0 2019-12-05T06:14:15.155 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.160+0000 7fddfaaea700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.155 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.160+0000 7fddf37fe700 1 -- 172.21.15.16:0/1853688875 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v42) v1 ==== 167+0+0 (secure 0 0 0) 0x7fdde8009ed0 con 0x7fddf411cba0 2019-12-05T06:14:15.204 INFO:tasks.workunit.client.0.smithi016.stderr:enabled new snapshots 2019-12-05T06:14:15.209 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.214+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddd4023630 msgr2=0x7fddd4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:15.209 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.214+0000 7fddfc54d700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddd4023630 0x7fddd4025ac0 crc :-1 s=READY pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:15.209 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.214+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 msgr2=0x7fddf411cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:15.209 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.214+0000 7fddfc54d700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 secure :-1 s=READY pgs=380 cs=0 l=1 rx=0x7fdde8010790 tx=0x7fdde8010b30).stop 2019-12-05T06:14:15.209 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.214+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 shutdown_connections 2019-12-05T06:14:15.209 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.214+0000 7fddfc54d700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fddd4023630 0x7fddd4025ac0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:15.210 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.214+0000 7fddfc54d700 1 --2- 172.21.15.16:0/1853688875 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fddf411cba0 0x7fddf411cf90 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:15.210 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.215+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 shutdown_connections 2019-12-05T06:14:15.210 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.215+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 wait complete. 2019-12-05T06:14:15.210 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.215+0000 7fddfc54d700 1 -- 172.21.15.16:0/1853688875 >> 172.21.15.16:0/1853688875 conn(0x7fddf407c510 msgr2=0x7fddf411f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:15.221 INFO:tasks.workunit.client.0.smithi016.stderr:+ touch foo 2019-12-05T06:14:15.234 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod +x foo 2019-12-05T06:14:15.240 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir .snap/s 2019-12-05T06:14:15.264 INFO:tasks.workunit.client.0.smithi016.stderr:+ find .snap/s/foo -executable 2019-12-05T06:14:15.264 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep foo 2019-12-05T06:14:15.347 INFO:tasks.workunit.client.0.smithi016.stdout:.snap/s/foo 2019-12-05T06:14:15.348 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir .snap/s 2019-12-05T06:14:15.364 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm foo 2019-12-05T06:14:15.372 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2019-12-05T06:14:15.372 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2019-12-05T06:14:15.373 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:14:15.374 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:14:15.494 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-capwb.sh... 2019-12-05T06:14:15.494 INFO:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snaptest-capwb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-capwb.sh 2019-12-05T06:14:15.586 INFO:tasks.workunit.client.0.smithi016.stderr:+ set -e 2019-12-05T06:14:15.587 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir foo 2019-12-05T06:14:15.591 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:14:15.730 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.729+0000 7fb575bc2700 1 Processor -- start 2019-12-05T06:14:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.729+0000 7fb575bc2700 1 -- start start 2019-12-05T06:14:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.730+0000 7fb575bc2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb57009e9f0 0x7fb570149490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.730+0000 7fb575bc2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57009ee50 con 0x7fb5701490a0 2019-12-05T06:14:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.730+0000 7fb575bc2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb57005f670 con 0x7fb57009f1c0 2019-12-05T06:14:15.731 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.730+0000 7fb575bc2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5700a3f40 con 0x7fb57009e9f0 2019-12-05T06:14:15.732 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.730+0000 7fb56ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb5701490a0 0x7fb57009e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35018/0 (socket says 172.21.15.16:35018) 2019-12-05T06:14:15.732 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.730+0000 7fb56ffff700 1 -- 172.21.15.16:0/2223186598 learned_addr learned my addr 172.21.15.16:0/2223186598 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:15.732 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.730+0000 7fb574960700 1 --2- 172.21.15.16:0/2223186598 >> v2:172.21.15.76:3300/0 conn(0x7fb57009e9f0 0x7fb570149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:15.732 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.731+0000 7fb56effd700 1 -- 172.21.15.16:0/2223186598 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3252227199 0 0) 0x7fb560001150 con 0x7fb5701490a0 2019-12-05T06:14:15.732 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.731+0000 7fb56effd700 1 -- 172.21.15.16:0/2223186598 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb554002e80 con 0x7fb5701490a0 2019-12-05T06:14:15.732 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.731+0000 7fb574960700 1 -- 172.21.15.16:0/2223186598 >> v1:172.21.15.76:6789/0 conn(0x7fb57009f1c0 legacy=0x7fb570150de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.731+0000 7fb56effd700 1 -- 172.21.15.16:0/2223186598 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4114517705 0 0) 0x7fb55c001080 con 0x7fb57009f1c0 2019-12-05T06:14:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.731+0000 7fb574960700 1 -- 172.21.15.16:0/2223186598 >> v1:172.21.15.16:6789/0 conn(0x7fb5701490a0 legacy=0x7fb57009e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.731+0000 7fb574960700 1 -- 172.21.15.16:0/2223186598 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57009d8f0 con 0x7fb57009e9f0 2019-12-05T06:14:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.732+0000 7fb574960700 1 --2- 172.21.15.16:0/2223186598 >> v2:172.21.15.76:3300/0 conn(0x7fb57009e9f0 0x7fb570149490 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb56800b300 tx=0x7fb56800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.732+0000 7fb574960700 1 --2- 172.21.15.16:0/2223186598 >> v2:172.21.15.76:3300/0 conn(0x7fb57009e9f0 0x7fb570149490 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb56800b300 tx=0x7fb56800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.733 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.732+0000 7fb574960700 1 --2- 172.21.15.16:0/2223186598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb57009e9f0 0x7fb570149490 secure :-1 s=READY pgs=302 cs=0 l=1 rx=0x7fb56800b300 tx=0x7fb56800bb10).ready entity=mon.1 client_cookie=480d45b51e8315d3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.732+0000 7fb574960700 1 --2- 172.21.15.16:0/2223186598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb57009e9f0 0x7fb570149490 secure :-1 s=THROTTLE_DONE pgs=302 cs=0 l=1 rx=0x7fb56800b300 tx=0x7fb56800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.732+0000 7fb56effd700 1 -- 172.21.15.16:0/2223186598 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5680075c0 con 0x7fb57009e9f0 2019-12-05T06:14:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.732+0000 7fb574960700 1 --2- 172.21.15.16:0/2223186598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb57009e9f0 0x7fb570149490 secure :-1 s=THROTTLE_DONE pgs=302 cs=0 l=1 rx=0x7fb56800b300 tx=0x7fb56800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.732+0000 7fb56effd700 1 -- 172.21.15.16:0/2223186598 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb568000f30 con 0x7fb57009e9f0 2019-12-05T06:14:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.732+0000 7fb574960700 1 --2- 172.21.15.16:0/2223186598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb57009e9f0 0x7fb570149490 secure :-1 s=THROTTLE_DONE pgs=302 cs=0 l=1 rx=0x7fb56800b300 tx=0x7fb56800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.732+0000 7fb56effd700 1 -- 172.21.15.16:0/2223186598 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb568009f50 con 0x7fb57009e9f0 2019-12-05T06:14:15.734 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.732+0000 7fb575bc2700 1 -- 172.21.15.16:0/2223186598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb57009e9f0 msgr2=0x7fb570149490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:15.735 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.733+0000 7fb575bc2700 1 --2- 172.21.15.16:0/2223186598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb57009e9f0 0x7fb570149490 secure :-1 s=READY pgs=302 cs=0 l=1 rx=0x7fb56800b300 tx=0x7fb56800bb10).stop 2019-12-05T06:14:15.735 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.734+0000 7fb575bc2700 1 -- 172.21.15.16:0/2223186598 shutdown_connections 2019-12-05T06:14:15.735 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.734+0000 7fb575bc2700 1 --2- 172.21.15.16:0/2223186598 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb57009e9f0 0x7fb570149490 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:15.735 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.734+0000 7fb575bc2700 1 -- 172.21.15.16:0/2223186598 shutdown_connections 2019-12-05T06:14:15.735 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.734+0000 7fb575bc2700 1 -- 172.21.15.16:0/2223186598 wait complete. 2019-12-05T06:14:15.735 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.734+0000 7fb575bc2700 1 -- 172.21.15.16:0/2223186598 >> 172.21.15.16:0/2223186598 conn(0x7fb570062220 msgr2=0x7fb5700a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:15.736 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.737+0000 7fb575bc2700 1 Processor -- start 2019-12-05T06:14:15.736 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.738+0000 7fb575bc2700 1 -- start start 2019-12-05T06:14:15.736 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.738+0000 7fb575bc2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb57007a500 0x7fb57007a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:15.737 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.738+0000 7fb575bc2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb570081b60 0x7fb57007ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:15.739 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.738+0000 7fb575bc2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb570123ec0 con 0x7fb570082330 2019-12-05T06:14:15.740 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.738+0000 7fb575bc2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb570123820 con 0x7fb57007a500 2019-12-05T06:14:15.740 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.738+0000 7fb575bc2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5700803d0 con 0x7fb570081b60 2019-12-05T06:14:15.740 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.739+0000 7fb56ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb57007a500 0x7fb57007a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:15.740 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.739+0000 7fb56ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb57007a500 0x7fb57007a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49856/0 (socket says 172.21.15.16:49856) 2019-12-05T06:14:15.740 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.739+0000 7fb56ffff700 1 -- 172.21.15.16:0/221180736 learned_addr learned my addr 172.21.15.16:0/221180736 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:15.740 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.739+0000 7fb574960700 1 --2- 172.21.15.16:0/221180736 >> v2:172.21.15.76:3300/0 conn(0x7fb570081b60 0x7fb57007ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:15.741 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.739+0000 7fb56ffff700 1 -- 172.21.15.16:0/221180736 >> v2:172.21.15.76:3300/0 conn(0x7fb570081b60 msgr2=0x7fb57007ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:15.741 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.739+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> v2:172.21.15.76:3300/0 conn(0x7fb570081b60 0x7fb57007ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:15.741 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.739+0000 7fb56ffff700 1 -- 172.21.15.16:0/221180736 >> v1:172.21.15.76:6789/0 conn(0x7fb570082330 legacy=0x7fb570081360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:15.741 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb56ffff700 1 -- 172.21.15.16:0/221180736 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb57007b370 con 0x7fb57007a500 2019-12-05T06:14:15.741 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb574960700 1 --2- 172.21.15.16:0/221180736 >> v2:172.21.15.76:3300/0 conn(0x7fb570081b60 0x7fb57007ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:14:15.742 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> v2:172.21.15.16:3300/0 conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.742 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> v2:172.21.15.16:3300/0 conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.742 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=READY pgs=385 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:15.742 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.742 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb56cff9700 1 -- 172.21.15.16:0/221180736 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb560000c90 con 0x7fb57007a500 2019-12-05T06:14:15.742 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.743 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb56cff9700 1 -- 172.21.15.16:0/221180736 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5600010c0 con 0x7fb57007a500 2019-12-05T06:14:15.743 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.743 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb57007b370 con 0x7fb57007a500 2019-12-05T06:14:15.743 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb56cff9700 1 -- 172.21.15.16:0/221180736 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb560000e90 con 0x7fb57007a500 2019-12-05T06:14:15.743 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.740+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb57007b370 con 0x7fb57007a500 2019-12-05T06:14:15.743 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.742+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.743 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.742+0000 7fb56cff9700 1 -- 172.21.15.16:0/221180736 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb560000c90 con 0x7fb57007a500 2019-12-05T06:14:15.744 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.742+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.744 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.743+0000 7fb56cff9700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c023630 0x7fb54c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:15.744 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.743+0000 7fb56cff9700 1 -- 172.21.15.16:0/221180736 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fb560045e10 con 0x7fb57007a500 2019-12-05T06:14:15.744 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.743+0000 7fb56f7fe700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c023630 0x7fb54c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:15.744 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.743+0000 7fb56f7fe700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c023630 0x7fb54c025ac0 crc :-1 s=READY pgs=177 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:15.744 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.745+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb570080a60 con 0x7fb57007a500 2019-12-05T06:14:15.747 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.753+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:15.748 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:15.753+0000 7fb56cff9700 1 -- 172.21.15.16:0/221180736 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb5600075d0 con 0x7fb57007a500 2019-12-05T06:14:15.909 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:14:15.909 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:14:15.909 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:14:15.915 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:14:15.915 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:16.016 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:16.021+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb570080a60 con 0x7fb57007a500 2019-12-05T06:14:16.093 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:14:16.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.147+0000 7fb8948bd700 1 Processor -- start 2019-12-05T06:14:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.147+0000 7fb8948bd700 1 -- start start 2019-12-05T06:14:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.148+0000 7fb8948bd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb89008d2c0 0x7fb89008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.148+0000 7fb8948bd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb89008fef0 0x7fb89008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.148+0000 7fb8948bd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8900958e0 con 0x7fb89008c6d0 2019-12-05T06:14:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.148+0000 7fb8948bd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb89008df90 con 0x7fb89008d2c0 2019-12-05T06:14:16.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.148+0000 7fb8948bd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8900a3f70 con 0x7fb89008fef0 2019-12-05T06:14:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb89008d2c0 0x7fb89008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb89008d2c0 0x7fb89008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49860/0 (socket says 172.21.15.16:49860) 2019-12-05T06:14:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 -- 172.21.15.16:0/2624652143 learned_addr learned my addr 172.21.15.16:0/2624652143 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:16.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88ed9d700 1 --1- 172.21.15.16:0/2624652143 >> v1:172.21.15.16:6789/0 conn(0x7fb89008c6d0 0x7fb89008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35028/0 (socket says 172.21.15.16:35028) 2019-12-05T06:14:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88f59e700 1 --2- 172.21.15.16:0/2624652143 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb89008fef0 0x7fb89008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 -- 172.21.15.16:0/2624652143 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb89008fef0 msgr2=0x7fb89008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:16.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 --2- 172.21.15.16:0/2624652143 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb89008fef0 0x7fb89008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 -- 172.21.15.16:0/2624652143 >> v1:172.21.15.16:6789/0 conn(0x7fb89008c6d0 legacy=0x7fb89008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 -- 172.21.15.16:0/2624652143 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89008e620 con 0x7fb89008d2c0 2019-12-05T06:14:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 --2- 172.21.15.16:0/2624652143 >> v2:172.21.15.16:3300/0 conn(0x7fb89008d2c0 0x7fb89008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb87800fac0 tx=0x7fb87800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 --2- 172.21.15.16:0/2624652143 >> v2:172.21.15.16:3300/0 conn(0x7fb89008d2c0 0x7fb89008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb87800fac0 tx=0x7fb87800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 --2- 172.21.15.16:0/2624652143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89008d2c0 0x7fb89008ef20 secure :-1 s=READY pgs=389 cs=0 l=1 rx=0x7fb87800fac0 tx=0x7fb87800f8f0).ready entity=mon.0 client_cookie=57de821c42e79278 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 --2- 172.21.15.16:0/2624652143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89008d2c0 0x7fb89008ef20 secure :-1 s=THROTTLE_DONE pgs=389 cs=0 l=1 rx=0x7fb87800fac0 tx=0x7fb87800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88dd9b700 1 -- 172.21.15.16:0/2624652143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb878000ce0 con 0x7fb89008d2c0 2019-12-05T06:14:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.149+0000 7fb88e59c700 1 --2- 172.21.15.16:0/2624652143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89008d2c0 0x7fb89008ef20 secure :-1 s=THROTTLE_DONE pgs=389 cs=0 l=1 rx=0x7fb87800fac0 tx=0x7fb87800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.150+0000 7fb88e59c700 1 --2- 172.21.15.16:0/2624652143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89008d2c0 0x7fb89008ef20 secure :-1 s=THROTTLE_DONE pgs=389 cs=0 l=1 rx=0x7fb87800fac0 tx=0x7fb87800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.150+0000 7fb88dd9b700 1 -- 172.21.15.16:0/2624652143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb878001010 con 0x7fb89008d2c0 2019-12-05T06:14:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.150+0000 7fb88dd9b700 1 -- 172.21.15.16:0/2624652143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb878001720 con 0x7fb89008d2c0 2019-12-05T06:14:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.150+0000 7fb8948bd700 1 -- 172.21.15.16:0/2624652143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89008d2c0 msgr2=0x7fb89008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.150+0000 7fb8948bd700 1 --2- 172.21.15.16:0/2624652143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89008d2c0 0x7fb89008ef20 secure :-1 s=READY pgs=389 cs=0 l=1 rx=0x7fb87800fac0 tx=0x7fb87800f8f0).stop 2019-12-05T06:14:16.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.152+0000 7fb8948bd700 1 -- 172.21.15.16:0/2624652143 shutdown_connections 2019-12-05T06:14:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.152+0000 7fb8948bd700 1 --2- 172.21.15.16:0/2624652143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89008d2c0 0x7fb89008ef20 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.152+0000 7fb8948bd700 1 --2- 172.21.15.16:0/2624652143 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb89008fef0 0x7fb89008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.152+0000 7fb8948bd700 1 -- 172.21.15.16:0/2624652143 shutdown_connections 2019-12-05T06:14:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.152+0000 7fb8948bd700 1 -- 172.21.15.16:0/2624652143 wait complete. 2019-12-05T06:14:16.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.152+0000 7fb8948bd700 1 -- 172.21.15.16:0/2624652143 >> 172.21.15.16:0/2624652143 conn(0x7fb890061940 msgr2=0x7fb890096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:16.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.155+0000 7fb8948bd700 1 Processor -- start 2019-12-05T06:14:16.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.155+0000 7fb8948bd700 1 -- start start 2019-12-05T06:14:16.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.157+0000 7fb8948bd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb89014ca10 0x7fb8900869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:16.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.157+0000 7fb8948bd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb89009be80 0x7fb89009c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:16.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.157+0000 7fb8948bd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb890153170 con 0x7fb8900879b0 2019-12-05T06:14:16.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.157+0000 7fb8948bd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb890085020 con 0x7fb89009be80 2019-12-05T06:14:16.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.157+0000 7fb8948bd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8900852f0 con 0x7fb89014ca10 2019-12-05T06:14:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.157+0000 7fb88f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb89009be80 0x7fb89009c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.157+0000 7fb88e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb8900879b0 0x7fb890086f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35036/0 (socket says 172.21.15.16:35036) 2019-12-05T06:14:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.157+0000 7fb88f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb89009be80 0x7fb89009c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49868/0 (socket says 172.21.15.16:49868) 2019-12-05T06:14:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.157+0000 7fb88e59c700 1 -- 172.21.15.16:0/202154581 learned_addr learned my addr 172.21.15.16:0/202154581 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.157+0000 7fb88ed9d700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb89014ca10 0x7fb8900869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.158+0000 7fb88f59e700 1 -- 172.21.15.16:0/202154581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb89014ca10 msgr2=0x7fb8900869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.158+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb89014ca10 0x7fb8900869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.158+0000 7fb88f59e700 1 -- 172.21.15.16:0/202154581 >> v1:172.21.15.16:6789/0 conn(0x7fb8900879b0 legacy=0x7fb890086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.158+0000 7fb8877fe700 1 -- 172.21.15.16:0/202154581 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2219304548 0 0) 0x7fb878008100 con 0x7fb8900879b0 2019-12-05T06:14:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.158+0000 7fb88f59e700 1 -- 172.21.15.16:0/202154581 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8900860e0 con 0x7fb89009be80 2019-12-05T06:14:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.158+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> v2:172.21.15.16:3300/0 conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.158+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> v2:172.21.15.16:3300/0 conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.158+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=READY pgs=391 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:16.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.161+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.161+0000 7fb8877fe700 1 -- 172.21.15.16:0/202154581 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb888001250 con 0x7fb89009be80 2019-12-05T06:14:16.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.161+0000 7fb8948bd700 1 -- 172.21.15.16:0/202154581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8900860e0 con 0x7fb89009be80 2019-12-05T06:14:16.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.161+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.161+0000 7fb8948bd700 1 -- 172.21.15.16:0/202154581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb890153170 con 0x7fb89009be80 2019-12-05T06:14:16.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.161+0000 7fb8877fe700 1 -- 172.21.15.16:0/202154581 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb888000e70 con 0x7fb89009be80 2019-12-05T06:14:16.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.161+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.161+0000 7fb8877fe700 1 -- 172.21.15.16:0/202154581 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb888001560 con 0x7fb89009be80 2019-12-05T06:14:16.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.162+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.163+0000 7fb8877fe700 1 -- 172.21.15.16:0/202154581 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb8880012f0 con 0x7fb89009be80 2019-12-05T06:14:16.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.163+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.164+0000 7fb8877fe700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 0x7fb868025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:16.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.164+0000 7fb8877fe700 1 -- 172.21.15.16:0/202154581 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6375+0+0 (secure 0 0 0) 0x7fb888045d40 con 0x7fb89009be80 2019-12-05T06:14:16.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.164+0000 7fb88ed9d700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 0x7fb868025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:16.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.164+0000 7fb88ed9d700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 0x7fb868025ac0 crc :-1 s=READY pgs=178 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:16.163 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:16.166+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.163 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:16.166+0000 7fb56cff9700 1 -- 172.21.15.16:0/221180736 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v43) v1 ==== 167+0+0 (secure 0 0 0) 0x7fb560000c90 con 0x7fb57007a500 2019-12-05T06:14:16.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.179+0000 7fb8948bd700 1 -- 172.21.15.16:0/202154581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8900860e0 con 0x7fb89009be80 2019-12-05T06:14:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.187+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.187+0000 7fb8877fe700 1 -- 172.21.15.16:0/202154581 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb8880010e0 con 0x7fb89009be80 2019-12-05T06:14:16.329 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:16.335+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb570080a60 con 0x7fb57007a500 2019-12-05T06:14:16.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.403+0000 7fb8948bd700 1 -- 172.21.15.16:0/202154581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8900860e0 con 0x7fb89009be80 2019-12-05T06:14:16.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.404+0000 7fb88f59e700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:16.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.404+0000 7fb8877fe700 1 -- 172.21.15.16:0/202154581 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 43 v43) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fb8880012f0 con 0x7fb89009be80 2019-12-05T06:14:16.398 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 43 2019-12-05T06:14:16.398 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:14:16.399 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":43,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":43,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:16.157663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:16.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 -- 172.21.15.16:0/202154581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 msgr2=0x7fb868025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 0x7fb868025ac0 crc :-1 s=READY pgs=178 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 -- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 msgr2=0x7fb89009c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 0x7fb89009c2e0 secure :-1 s=READY pgs=391 cs=0 l=1 rx=0x7fb888010380 tx=0x7fb88800ae30).stop 2019-12-05T06:14:16.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 -- 172.21.15.16:0/202154581 shutdown_connections 2019-12-05T06:14:16.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 0x7fb868025ac0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:16.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb89009be80 0x7fb89009c2e0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:16.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 --2- 172.21.15.16:0/202154581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb89014ca10 0x7fb8900869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:16.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 -- 172.21.15.16:0/202154581 shutdown_connections 2019-12-05T06:14:16.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 -- 172.21.15.16:0/202154581 wait complete. 2019-12-05T06:14:16.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:16.407+0000 7fb8948bd700 1 -- 172.21.15.16:0/202154581 >> 172.21.15.16:0/202154581 conn(0x7fb890098480 msgr2=0x7fb890071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:16.417 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:14:17.169 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.175+0000 7fb56ffff700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:17.169 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.175+0000 7fb56cff9700 1 -- 172.21.15.16:0/221180736 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v44) v1 ==== 167+0+0 (secure 0 0 0) 0x7fb5600075d0 con 0x7fb57007a500 2019-12-05T06:14:17.194 INFO:tasks.workunit.client.0.smithi016.stderr:enabled new snapshots 2019-12-05T06:14:17.198 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.204+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c023630 msgr2=0x7fb54c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:17.199 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.204+0000 7fb575bc2700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c023630 0x7fb54c025ac0 crc :-1 s=READY pgs=177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:17.199 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.204+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 msgr2=0x7fb57007a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:17.199 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.204+0000 7fb575bc2700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 secure :-1 s=READY pgs=385 cs=0 l=1 rx=0x7fb5600103e0 tx=0x7fb560010780).stop 2019-12-05T06:14:17.199 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.204+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 shutdown_connections 2019-12-05T06:14:17.199 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.204+0000 7fb575bc2700 1 --2- 172.21.15.16:0/221180736 >> v2:172.21.15.76:3300/0 conn(0x7fb570081b60 0x7fb57007ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:17.199 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.204+0000 7fb575bc2700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c023630 0x7fb54c025ac0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:17.200 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.204+0000 7fb575bc2700 1 --2- 172.21.15.16:0/221180736 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb57007a500 0x7fb57007a8f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:17.200 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.205+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 shutdown_connections 2019-12-05T06:14:17.200 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.205+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 wait complete. 2019-12-05T06:14:17.201 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:17.205+0000 7fb575bc2700 1 -- 172.21.15.16:0/221180736 >> 172.21.15.16:0/221180736 conn(0x7fb570079ff0 msgr2=0x7fb57007ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:17.217 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo x 2019-12-05T06:14:17.224 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:17.409 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir foo/.snap/ss 2019-12-05T06:14:17.429 INFO:tasks.workunit.client.0.smithi016.stderr:+ ln foo/x foo/xx 2019-12-05T06:14:17.438 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat foo/.snap/ss/x 2019-12-05T06:14:17.453 INFO:tasks.workunit.client.0.smithi016.stdout:x 2019-12-05T06:14:17.453 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir foo/.snap/ss 2019-12-05T06:14:17.466 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo a 2019-12-05T06:14:17.470 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo b 2019-12-05T06:14:17.474 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir foo/.snap/s 2019-12-05T06:14:18.318 INFO:tasks.workunit.client.0.smithi016.stderr:++ cat foo/.snap/s/a 2019-12-05T06:14:18.339 INFO:tasks.workunit.client.0.smithi016.stderr:+ r=a 2019-12-05T06:14:18.339 INFO:tasks.workunit.client.0.smithi016.stderr:+ '[' -z a ']' 2019-12-05T06:14:18.339 INFO:tasks.workunit.client.0.smithi016.stderr:+ ln foo/b foo/b2 2019-12-05T06:14:18.349 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat foo/.snap/s/b 2019-12-05T06:14:18.364 INFO:tasks.workunit.client.0.smithi016.stdout:b 2019-12-05T06:14:18.364 INFO:tasks.workunit.client.0.smithi016.stdout:this used to hang: 2019-12-05T06:14:18.365 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'this used to hang:' 2019-12-05T06:14:18.365 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo more 2019-12-05T06:14:18.369 INFO:tasks.workunit.client.0.smithi016.stdout:oh, it didn't hang! good job. 2019-12-05T06:14:18.370 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'oh, it didn'\''t hang! good job.' 2019-12-05T06:14:18.370 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat foo/b 2019-12-05T06:14:18.373 INFO:tasks.workunit.client.0.smithi016.stdout:b 2019-12-05T06:14:18.373 INFO:tasks.workunit.client.0.smithi016.stdout:more 2019-12-05T06:14:18.373 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir foo/.snap/s 2019-12-05T06:14:18.389 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm -r foo 2019-12-05T06:14:18.418 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:18.424 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2019-12-05T06:14:18.425 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2019-12-05T06:14:18.425 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:14:18.425 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:14:18.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.585+0000 7f7c00596700 1 Processor -- start 2019-12-05T06:14:18.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.585+0000 7f7c00596700 1 -- start start 2019-12-05T06:14:18.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7c00596700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7bf808fef0 0x7f7bf808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:18.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7c00596700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf80958e0 con 0x7f7bf808d2c0 2019-12-05T06:14:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7c00596700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf80629e0 con 0x7f7bf808c6d0 2019-12-05T06:14:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7c00596700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf808df90 con 0x7f7bf808fef0 2019-12-05T06:14:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7bff334700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7bf808fef0 0x7f7bf808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7bfe332700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7bf808d2c0 0x7f7bf808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35048/0 (socket says 172.21.15.16:35048) 2019-12-05T06:14:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7bfe332700 1 -- 172.21.15.16:0/2853509036 learned_addr learned my addr 172.21.15.16:0/2853509036 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7bff334700 1 -- 172.21.15.16:0/2853509036 >> v1:172.21.15.76:6789/0 conn(0x7f7bf808c6d0 legacy=0x7f7bf808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:18.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7bff334700 1 -- 172.21.15.16:0/2853509036 >> v1:172.21.15.16:6789/0 conn(0x7f7bf808d2c0 legacy=0x7f7bf808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7bff334700 1 -- 172.21.15.16:0/2853509036 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bf808e620 con 0x7f7bf808fef0 2019-12-05T06:14:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7bfdb31700 1 -- 172.21.15.16:0/2853509036 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949636244 0 0) 0x7f7bec0014d0 con 0x7f7bf808d2c0 2019-12-05T06:14:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7bff334700 1 --2- 172.21.15.16:0/2853509036 >> v2:172.21.15.16:3300/0 conn(0x7f7bf808fef0 0x7f7bf808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7be800af50 tx=0x7f7be800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7bff334700 1 --2- 172.21.15.16:0/2853509036 >> v2:172.21.15.16:3300/0 conn(0x7f7bf808fef0 0x7f7bf808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7be800af50 tx=0x7f7be800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.586+0000 7f7bff334700 1 --2- 172.21.15.16:0/2853509036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf808fef0 0x7f7bf808f720 secure :-1 s=READY pgs=393 cs=0 l=1 rx=0x7f7be800af50 tx=0x7f7be800b760).ready entity=mon.0 client_cookie=3308ef5f5313c7c1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.587+0000 7f7bff334700 1 --2- 172.21.15.16:0/2853509036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf808fef0 0x7f7bf808f720 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f7be800af50 tx=0x7f7be800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.587+0000 7f7bfdb31700 1 -- 172.21.15.16:0/2853509036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7be8009ba0 con 0x7f7bf808fef0 2019-12-05T06:14:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.587+0000 7f7bff334700 1 --2- 172.21.15.16:0/2853509036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf808fef0 0x7f7bf808f720 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f7be800af50 tx=0x7f7be800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.587+0000 7f7bfdb31700 1 -- 172.21.15.16:0/2853509036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7be8000a70 con 0x7f7bf808fef0 2019-12-05T06:14:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.587+0000 7f7bff334700 1 --2- 172.21.15.16:0/2853509036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf808fef0 0x7f7bf808f720 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f7be800af50 tx=0x7f7be800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.587+0000 7f7bfdb31700 1 -- 172.21.15.16:0/2853509036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7be8009c40 con 0x7f7bf808fef0 2019-12-05T06:14:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.587+0000 7f7c00596700 1 -- 172.21.15.16:0/2853509036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf808fef0 msgr2=0x7f7bf808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.587+0000 7f7c00596700 1 --2- 172.21.15.16:0/2853509036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf808fef0 0x7f7bf808f720 secure :-1 s=READY pgs=393 cs=0 l=1 rx=0x7f7be800af50 tx=0x7f7be800b760).stop 2019-12-05T06:14:18.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.588+0000 7f7c00596700 1 -- 172.21.15.16:0/2853509036 shutdown_connections 2019-12-05T06:14:18.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.588+0000 7f7c00596700 1 --2- 172.21.15.16:0/2853509036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bf808fef0 0x7f7bf808f720 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.588+0000 7f7c00596700 1 -- 172.21.15.16:0/2853509036 shutdown_connections 2019-12-05T06:14:18.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.588+0000 7f7c00596700 1 -- 172.21.15.16:0/2853509036 wait complete. 2019-12-05T06:14:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.588+0000 7f7c00596700 1 -- 172.21.15.16:0/2853509036 >> 172.21.15.16:0/2853509036 conn(0x7f7bf8061940 msgr2=0x7f7bf8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.590+0000 7f7c00596700 1 Processor -- start 2019-12-05T06:14:18.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.590+0000 7f7c00596700 1 -- start start 2019-12-05T06:14:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7c00596700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7bf814c3a0 0x7f7bf809d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7c00596700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7bf806e210 0x7f7bf809daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7c00596700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7bf8154170 con 0x7f7bf809e030 2019-12-05T06:14:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7c00596700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7bf808ada0 con 0x7f7bf806e210 2019-12-05T06:14:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7c00596700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bf809bec0 con 0x7f7bf814c3a0 2019-12-05T06:14:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7bff334700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7bf809e030 0x7f7bf8088fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35056/0 (socket says 172.21.15.16:35056) 2019-12-05T06:14:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7bff334700 1 -- 172.21.15.16:0/2671429751 learned_addr learned my addr 172.21.15.16:0/2671429751 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> v2:172.21.15.76:3300/0 conn(0x7f7bf806e210 0x7f7bf809daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7bfeb33700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7bf814c3a0 0x7f7bf809d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236537365 0 0) 0x7f7be800b3d0 con 0x7f7bf809e030 2019-12-05T06:14:18.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.591+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bd8002e80 con 0x7f7bf809e030 2019-12-05T06:14:18.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 563289764 0 0) 0x7f7be800abe0 con 0x7f7bf809e030 2019-12-05T06:14:18.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7bd80032e0 con 0x7f7bf809e030 2019-12-05T06:14:18.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f7be800afc0 con 0x7f7bf809e030 2019-12-05T06:14:18.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7bf814c3a0 msgr2=0x7f7bf809d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7bf814c3a0 0x7f7bf809d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 >> v2:172.21.15.76:3300/0 conn(0x7f7bf806e210 msgr2=0x7f7bf809daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 --2- 172.21.15.16:0/2671429751 >> v2:172.21.15.76:3300/0 conn(0x7f7bf806e210 0x7f7bf809daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 >> v1:172.21.15.16:6789/0 conn(0x7f7bf809e030 legacy=0x7f7bf8088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7bd80033a0 0x7f7bd80037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7bd8004890 0x7f7bd8004d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd800bb60 con 0x7f7bd8003cf0 2019-12-05T06:14:18.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd800bd50 con 0x7f7bd80033a0 2019-12-05T06:14:18.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7bd800bf80 con 0x7f7bd8004890 2019-12-05T06:14:18.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfeb33700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7bd80033a0 0x7f7bd80037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfe332700 1 -- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7bd8004890 msgr2=0x7f7bd8004d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7bd8004890 0x7f7bd8004d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfe332700 1 -- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7bd80033a0 msgr2=0x7f7bd80037b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7bd80033a0 0x7f7bd80037b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfe332700 1 -- 172.21.15.16:0/2671429751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bd800c230 con 0x7f7bd8003cf0 2019-12-05T06:14:18.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfeb33700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7bd80033a0 0x7f7bd80037b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:14:18.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7c00596700 1 -- 172.21.15.16:0/2671429751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7bf809ccb0 con 0x7f7bd8003cf0 2019-12-05T06:14:18.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7c00596700 1 -- 172.21.15.16:0/2671429751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7bf8154170 con 0x7f7bd8003cf0 2019-12-05T06:14:18.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=READY pgs=395 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:18.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.592+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=THROTTLE_DONE pgs=395 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.593+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7bec001270 con 0x7f7bd8003cf0 2019-12-05T06:14:18.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.593+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=THROTTLE_DONE pgs=395 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.593+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7bec001270 con 0x7f7bd8003cf0 2019-12-05T06:14:18.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.593+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=THROTTLE_DONE pgs=395 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.593+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7bec000cc0 con 0x7f7bd8003cf0 2019-12-05T06:14:18.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.593+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=THROTTLE_DONE pgs=395 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.593+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7bec00c850 con 0x7f7bd8003cf0 2019-12-05T06:14:18.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.593+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=THROTTLE_DONE pgs=395 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.594+0000 7f7bf77fe700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7bd802b9b0 0x7f7bd802dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.594+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6375+0+0 (secure 0 0 0) 0x7f7bec0462d0 con 0x7f7bd8003cf0 2019-12-05T06:14:18.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.594+0000 7f7bfeb33700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7bd802b9b0 0x7f7bd802dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.594+0000 7f7bfeb33700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7bd802b9b0 0x7f7bd802dde0 crc :-1 s=READY pgs=179 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:18.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.600+0000 7f7c00596700 1 -- 172.21.15.16:0/2671429751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7bf809ccb0 con 0x7f7bd8003cf0 2019-12-05T06:14:18.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.605+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=THROTTLE_DONE pgs=395 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.605+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7bec00c640 con 0x7f7bd8003cf0 2019-12-05T06:14:18.625 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-dir-rename.sh... 2019-12-05T06:14:18.625 INFO:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snaptest-dir-rename.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-dir-rename.sh 2019-12-05T06:14:18.697 INFO:tasks.workunit.client.0.smithi016.stderr:+ set -e 2019-12-05T06:14:18.697 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:14:18.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.805+0000 7f7c00596700 1 -- 172.21.15.16:0/2671429751 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7bf809ccb0 con 0x7f7bd8003cf0 2019-12-05T06:14:18.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.805+0000 7f7bfe332700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=THROTTLE_DONE pgs=395 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.805+0000 7f7bf77fe700 1 -- 172.21.15.16:0/2671429751 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 44 v44) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f7bec001250 con 0x7f7bd8003cf0 2019-12-05T06:14:18.800 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 44 2019-12-05T06:14:18.800 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:14:18.800 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":44,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":44,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:17.165457+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:18.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.808+0000 7f7c00596700 1 -- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7bd802b9b0 msgr2=0x7f7bd802dde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.808+0000 7f7c00596700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7bd802b9b0 0x7f7bd802dde0 crc :-1 s=READY pgs=179 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.808+0000 7f7c00596700 1 -- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 msgr2=0x7f7bd8004150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.808+0000 7f7bff334700 1 -- 172.21.15.16:0/2671429751 reap_dead start 2019-12-05T06:14:18.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.808+0000 7f7c00596700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 secure :-1 s=READY pgs=395 cs=0 l=1 rx=0x7f7bec0109b0 tx=0x7f7bec00b6b0).stop 2019-12-05T06:14:18.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.809+0000 7f7c00596700 1 -- 172.21.15.16:0/2671429751 shutdown_connections 2019-12-05T06:14:18.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.809+0000 7f7c00596700 1 --2- 172.21.15.16:0/2671429751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7bd8003cf0 0x7f7bd8004150 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.809+0000 7f7c00596700 1 -- 172.21.15.16:0/2671429751 shutdown_connections 2019-12-05T06:14:18.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.809+0000 7f7c00596700 1 -- 172.21.15.16:0/2671429751 wait complete. 2019-12-05T06:14:18.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:18.809+0000 7f7c00596700 1 -- 172.21.15.16:0/2671429751 >> 172.21.15.16:0/2671429751 conn(0x7f7bf8098480 msgr2=0x7f7bf8071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:18.812 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.812+0000 7ff3314b4700 1 Processor -- start 2019-12-05T06:14:18.813 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.812+0000 7ff3314b4700 1 -- start start 2019-12-05T06:14:18.813 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.813+0000 7ff3314b4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff32c09f1c0 0x7ff32c09e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:18.813 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.813+0000 7ff3314b4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c05f670 con 0x7ff32c1490a0 2019-12-05T06:14:18.813 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.813+0000 7ff3314b4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c1498f0 con 0x7ff32c149490 2019-12-05T06:14:18.813 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.813+0000 7ff3314b4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff32c0a3f40 con 0x7ff32c09f1c0 2019-12-05T06:14:18.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.813+0000 7ff32b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff32c1490a0 0x7ff32c09e1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35066/0 (socket says 172.21.15.16:35066) 2019-12-05T06:14:18.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.813+0000 7ff32b7fe700 1 -- 172.21.15.16:0/4159142626 learned_addr learned my addr 172.21.15.16:0/4159142626 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:18.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.813+0000 7ff32affd700 1 --2- 172.21.15.16:0/4159142626 >> v2:172.21.15.76:3300/0 conn(0x7ff32c09f1c0 0x7ff32c09e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.814+0000 7ff32a7fc700 1 -- 172.21.15.16:0/4159142626 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357340852 0 0) 0x7ff31c0014d0 con 0x7ff32c1490a0 2019-12-05T06:14:18.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.814+0000 7ff32a7fc700 1 -- 172.21.15.16:0/4159142626 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff30c002e80 con 0x7ff32c1490a0 2019-12-05T06:14:18.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.814+0000 7ff32affd700 1 -- 172.21.15.16:0/4159142626 >> v1:172.21.15.76:6789/0 conn(0x7ff32c149490 legacy=0x7ff32c152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.814 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.814+0000 7ff32affd700 1 -- 172.21.15.16:0/4159142626 >> v1:172.21.15.16:6789/0 conn(0x7ff32c1490a0 legacy=0x7ff32c09e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.815 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.814+0000 7ff32affd700 1 -- 172.21.15.16:0/4159142626 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff32c09d8f0 con 0x7ff32c09f1c0 2019-12-05T06:14:18.815 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.814+0000 7ff32affd700 1 --2- 172.21.15.16:0/4159142626 >> v2:172.21.15.76:3300/0 conn(0x7ff32c09f1c0 0x7ff32c09e9f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff32400af50 tx=0x7ff32400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.815 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.814+0000 7ff32affd700 1 --2- 172.21.15.16:0/4159142626 >> v2:172.21.15.76:3300/0 conn(0x7ff32c09f1c0 0x7ff32c09e9f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff32400af50 tx=0x7ff32400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.815 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.814+0000 7ff32affd700 1 --2- 172.21.15.16:0/4159142626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff32c09f1c0 0x7ff32c09e9f0 secure :-1 s=READY pgs=309 cs=0 l=1 rx=0x7ff32400af50 tx=0x7ff32400b760).ready entity=mon.1 client_cookie=7919618afe4217a4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:18.815 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.815+0000 7ff32affd700 1 --2- 172.21.15.16:0/4159142626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff32c09f1c0 0x7ff32c09e9f0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7ff32400af50 tx=0x7ff32400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.815 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.815+0000 7ff32a7fc700 1 -- 172.21.15.16:0/4159142626 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff324000960 con 0x7ff32c09f1c0 2019-12-05T06:14:18.816 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.815+0000 7ff32affd700 1 --2- 172.21.15.16:0/4159142626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff32c09f1c0 0x7ff32c09e9f0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7ff32400af50 tx=0x7ff32400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.816 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.815+0000 7ff32affd700 1 --2- 172.21.15.16:0/4159142626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff32c09f1c0 0x7ff32c09e9f0 secure :-1 s=THROTTLE_DONE pgs=309 cs=0 l=1 rx=0x7ff32400af50 tx=0x7ff32400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.816 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.815+0000 7ff32a7fc700 1 -- 172.21.15.16:0/4159142626 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff324009860 con 0x7ff32c09f1c0 2019-12-05T06:14:18.816 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.815+0000 7ff32a7fc700 1 -- 172.21.15.16:0/4159142626 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff324009f30 con 0x7ff32c09f1c0 2019-12-05T06:14:18.816 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.815+0000 7ff3314b4700 1 -- 172.21.15.16:0/4159142626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff32c09f1c0 msgr2=0x7ff32c09e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.816 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.815+0000 7ff3314b4700 1 --2- 172.21.15.16:0/4159142626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff32c09f1c0 0x7ff32c09e9f0 secure :-1 s=READY pgs=309 cs=0 l=1 rx=0x7ff32400af50 tx=0x7ff32400b760).stop 2019-12-05T06:14:18.816 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.816+0000 7ff3314b4700 1 -- 172.21.15.16:0/4159142626 shutdown_connections 2019-12-05T06:14:18.817 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.816+0000 7ff3314b4700 1 --2- 172.21.15.16:0/4159142626 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff32c09f1c0 0x7ff32c09e9f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.817 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.816+0000 7ff3314b4700 1 -- 172.21.15.16:0/4159142626 shutdown_connections 2019-12-05T06:14:18.817 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.816+0000 7ff3314b4700 1 -- 172.21.15.16:0/4159142626 wait complete. 2019-12-05T06:14:18.817 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.816+0000 7ff3314b4700 1 -- 172.21.15.16:0/4159142626 >> 172.21.15.16:0/4159142626 conn(0x7ff32c062220 msgr2=0x7ff32c0a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:18.823 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.819+0000 7ff3314b4700 1 Processor -- start 2019-12-05T06:14:18.823 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.819+0000 7ff3314b4700 1 -- start start 2019-12-05T06:14:18.824 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.820+0000 7ff3314b4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff32c080c90 0x7ff32c19c630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.824 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.820+0000 7ff3314b4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c0810f0 con 0x7ff32c081460 2019-12-05T06:14:18.824 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.820+0000 7ff3314b4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff32c120d00 con 0x7ff32c11ee50 2019-12-05T06:14:18.824 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.820+0000 7ff3314b4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff32c07f500 con 0x7ff32c080c90 2019-12-05T06:14:18.824 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.820+0000 7ff32affd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff32c081460 0x7ff32c152a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35074/0 (socket says 172.21.15.16:35074) 2019-12-05T06:14:18.824 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.820+0000 7ff32affd700 1 -- 172.21.15.16:0/2043127313 learned_addr learned my addr 172.21.15.16:0/2043127313 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:18.824 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.820+0000 7ff32bfff700 1 --2- 172.21.15.16:0/2043127313 >> v2:172.21.15.76:3300/0 conn(0x7ff32c080c90 0x7ff32c19c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.825 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.820+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2288501924 0 0) 0x7ff32400afb0 con 0x7ff32c081460 2019-12-05T06:14:18.825 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.820+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff304002e80 con 0x7ff32c081460 2019-12-05T06:14:18.825 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2252838023 0 0) 0x7ff324000940 con 0x7ff32c081460 2019-12-05T06:14:18.825 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff3040032e0 con 0x7ff32c081460 2019-12-05T06:14:18.825 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7ff32400b080 con 0x7ff32c081460 2019-12-05T06:14:18.825 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 >> v2:172.21.15.76:3300/0 conn(0x7ff32c080c90 msgr2=0x7ff32c19c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.825 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 --2- 172.21.15.16:0/2043127313 >> v2:172.21.15.76:3300/0 conn(0x7ff32c080c90 0x7ff32c19c630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.826 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 >> v1:172.21.15.76:6789/0 conn(0x7ff32c11ee50 legacy=0x7ff32c080490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.826 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 >> v1:172.21.15.16:6789/0 conn(0x7ff32c081460 legacy=0x7ff32c152a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.826 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.826 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff304008b20 0x7ff30400af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.826 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff30400b460 0x7ff30400d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.826 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30400dfc0 con 0x7ff304005dd0 2019-12-05T06:14:18.826 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff30400e1f0 con 0x7ff30400b460 2019-12-05T06:14:18.827 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff30400e420 con 0x7ff304008b20 2019-12-05T06:14:18.827 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.827 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff32bfff700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff30400b460 0x7ff30400d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.827 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff32b7fe700 1 -- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff304008b20 msgr2=0x7ff30400af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.827 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff304008b20 0x7ff30400af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.827 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff32b7fe700 1 -- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff30400b460 msgr2=0x7ff30400d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:18.827 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff30400b460 0x7ff30400d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff32b7fe700 1 -- 172.21.15.16:0/2043127313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30400e6d0 con 0x7ff304005dd0 2019-12-05T06:14:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff32c07fb90 con 0x7ff304005dd0 2019-12-05T06:14:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.821+0000 7ff32bfff700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff30400b460 0x7ff30400d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:14:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff32c0810f0 con 0x7ff304005dd0 2019-12-05T06:14:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff32bfff700 1 -- 172.21.15.16:0/2043127313 reap_dead start 2019-12-05T06:14:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.828 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.829 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=READY pgs=398 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:18.829 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.829 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff31c0017e0 con 0x7ff304005dd0 2019-12-05T06:14:18.829 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.829 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff31c000ce0 con 0x7ff304005dd0 2019-12-05T06:14:18.829 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.830 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff31c001880 con 0x7ff304005dd0 2019-12-05T06:14:18.830 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.822+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.830 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.823+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff31c001400 con 0x7ff304005dd0 2019-12-05T06:14:18.830 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.823+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.831 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.823+0000 7ff323fff700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff304026400 0x7ff304028830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:18.831 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.823+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6375+0+0 (secure 0 0 0) 0x7ff31c046210 con 0x7ff304005dd0 2019-12-05T06:14:18.831 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.824+0000 7ff32affd700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff304026400 0x7ff304028830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:18.831 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.824+0000 7ff32affd700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff304026400 0x7ff304028830 crc :-1 s=READY pgs=180 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:18.831 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.828+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff308002e60 con 0x7ff304005dd0 2019-12-05T06:14:18.832 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:14:18.832 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.834+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:18.833 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:18.834+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff31c000cc0 con 0x7ff304005dd0 2019-12-05T06:14:19.045 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:19.051+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff308001d60 con 0x7ff304005dd0 2019-12-05T06:14:19.174 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:19.180+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:19.174 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:19.180+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v45) v1 ==== 167+0+0 (secure 0 0 0) 0x7ff31c001370 con 0x7ff304005dd0 2019-12-05T06:14:19.403 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:19.408+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff308002eb0 con 0x7ff304005dd0 2019-12-05T06:14:20.182 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.188+0000 7ff32b7fe700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=THROTTLE_DONE pgs=398 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:20.182 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.188+0000 7ff323fff700 1 -- 172.21.15.16:0/2043127313 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v46) v1 ==== 167+0+0 (secure 0 0 0) 0x7ff31c000cc0 con 0x7ff304005dd0 2019-12-05T06:14:20.198 INFO:tasks.ceph.osd.0.smithi016.stderr:2019-12-05T06:14:20.204+0000 7f23e096d700 -1 osd.0 pg_epoch: 43 pg[3.1s0( v 42'1395 (0'0,42'1395] local-lis/les=21/22 n=2 ec=21/21 lis/c=21/21 les/c/f=22/22/0 sis=21) [0,4,3,5]p0(0) r=0 lpr=21 crt=42'1395 lcod 42'1394 mlcod 42'1394 active+clean+snaptrim trimq=[1a6~2] ps=[1a2~4]] removing snap head 2019-12-05T06:14:20.218 INFO:tasks.workunit.client.0.smithi016.stderr:enabled new snapshots 2019-12-05T06:14:20.221 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.228+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff304026400 msgr2=0x7ff304028830 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.222 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.228+0000 7ff3314b4700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff304026400 0x7ff304028830 crc :-1 s=READY pgs=180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.222 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.228+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 msgr2=0x7ff3040085e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:20.222 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.228+0000 7ff3314b4700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 secure :-1 s=READY pgs=398 cs=0 l=1 rx=0x7ff31c010680 tx=0x7ff31c010a20).stop 2019-12-05T06:14:20.223 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.228+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 shutdown_connections 2019-12-05T06:14:20.223 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.228+0000 7ff3314b4700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff304026400 0x7ff304028830 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.223 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.228+0000 7ff3314b4700 1 --2- 172.21.15.16:0/2043127313 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff304005dd0 0x7ff3040085e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:20.223 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.228+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 shutdown_connections 2019-12-05T06:14:20.224 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.228+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 wait complete. 2019-12-05T06:14:20.224 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:20.228+0000 7ff3314b4700 1 -- 172.21.15.16:0/2043127313 >> 172.21.15.16:0/2043127313 conn(0x7ff32c07a860 msgr2=0x7ff32c08a2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:20.235 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:20.241 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir .snap/bar 2019-12-05T06:14:20.833 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:14:20.867 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/foo 2019-12-05T06:14:20.882 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/foo 2019-12-05T06:14:20.897 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:20.904 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/bar/a 2019-12-05T06:14:20.919 INFO:tasks.workunit.client.0.smithi016.stdout: File: ‘.snap/bar/a’ 2019-12-05T06:14:20.920 INFO:tasks.workunit.client.0.smithi016.stdout: Size: 0 Blocks: 1 IO Block: 4096 directory 2019-12-05T06:14:20.920 INFO:tasks.workunit.client.0.smithi016.stdout:Device: 29h/41d Inode: 59955269551003028 Links: 2 2019-12-05T06:14:20.920 INFO:tasks.workunit.client.0.smithi016.stdout:Access: (0775/drwxrwxr-x) Uid: ( 1000/ ubuntu) Gid: ( 1215/ ubuntu) 2019-12-05T06:14:20.920 INFO:tasks.workunit.client.0.smithi016.stdout:Context: system_u:object_r:fusefs_t:s0 2019-12-05T06:14:20.920 INFO:tasks.workunit.client.0.smithi016.stdout:Access: 2019-12-05 06:14:20.244755486 +0000 2019-12-05T06:14:20.921 INFO:tasks.workunit.client.0.smithi016.stdout:Modify: 2019-12-05 06:14:20.244755486 +0000 2019-12-05T06:14:20.921 INFO:tasks.workunit.client.0.smithi016.stdout:Change: 2019-12-05 06:14:20.244755486 +0000 2019-12-05T06:14:20.921 INFO:tasks.workunit.client.0.smithi016.stdout: Birth: - 2019-12-05T06:14:20.921 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir .snap/bar 2019-12-05T06:14:20.937 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2019-12-05T06:14:20.938 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2019-12-05T06:14:20.938 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:14:20.938 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:14:21.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.071+0000 7fbd45e31700 1 Processor -- start 2019-12-05T06:14:21.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.071+0000 7fbd45e31700 1 -- start start 2019-12-05T06:14:21.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd45e31700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd4008d2c0 0x7fbd4008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:21.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd45e31700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbd4008fef0 0x7fbd4008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:21.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd45e31700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd400958e0 con 0x7fbd4008c6d0 2019-12-05T06:14:21.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd45e31700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd4008df90 con 0x7fbd4008fef0 2019-12-05T06:14:21.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd45e31700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd400a3f70 con 0x7fbd4008d2c0 2019-12-05T06:14:21.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd44bcf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbd4008fef0 0x7fbd4008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:21.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd3f7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd4008d2c0 0x7fbd4008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd3f7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd4008d2c0 0x7fbd4008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:53910/0 (socket says 172.21.15.16:53910) 2019-12-05T06:14:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd3ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbd4008c6d0 0x7fbd4008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:56388/0 (socket says 172.21.15.16:56388) 2019-12-05T06:14:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd44bcf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbd4008fef0 0x7fbd4008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:36012/0 (socket says 172.21.15.16:36012) 2019-12-05T06:14:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.073+0000 7fbd3f7fe700 1 -- 172.21.15.16:0/3125826052 learned_addr learned my addr 172.21.15.16:0/3125826052 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.074+0000 7fbd44bcf700 1 -- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd4008d2c0 msgr2=0x7fbd4008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.074+0000 7fbd44bcf700 1 --2- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd4008d2c0 0x7fbd4008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:21.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.074+0000 7fbd44bcf700 1 -- 172.21.15.16:0/3125826052 >> v1:172.21.15.76:6789/0 conn(0x7fbd4008c6d0 legacy=0x7fbd4008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.074+0000 7fbd3effd700 1 -- 172.21.15.16:0/3125826052 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323464660 0 0) 0x7fbd30001080 con 0x7fbd4008c6d0 2019-12-05T06:14:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.074+0000 7fbd44bcf700 1 -- 172.21.15.16:0/3125826052 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4008e620 con 0x7fbd4008fef0 2019-12-05T06:14:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.074+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd4008d2c0 0x7fbd4008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:14:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.074+0000 7fbd44bcf700 1 --2- 172.21.15.16:0/3125826052 >> v2:172.21.15.76:3300/0 conn(0x7fbd4008fef0 0x7fbd4008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbd3800b300 tx=0x7fbd3800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.074+0000 7fbd44bcf700 1 --2- 172.21.15.16:0/3125826052 >> v2:172.21.15.76:3300/0 conn(0x7fbd4008fef0 0x7fbd4008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbd3800b300 tx=0x7fbd3800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.075+0000 7fbd44bcf700 1 --2- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd4008fef0 0x7fbd4008f460 secure :-1 s=READY pgs=312 cs=0 l=1 rx=0x7fbd3800b300 tx=0x7fbd3800bb10).ready entity=mon.1 client_cookie=25d440d70c896d40 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:21.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.075+0000 7fbd44bcf700 1 --2- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd4008fef0 0x7fbd4008f460 secure :-1 s=THROTTLE_DONE pgs=312 cs=0 l=1 rx=0x7fbd3800b300 tx=0x7fbd3800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.075+0000 7fbd3effd700 1 -- 172.21.15.16:0/3125826052 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd38000f40 con 0x7fbd4008fef0 2019-12-05T06:14:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.075+0000 7fbd44bcf700 1 --2- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd4008fef0 0x7fbd4008f460 secure :-1 s=THROTTLE_DONE pgs=312 cs=0 l=1 rx=0x7fbd3800b300 tx=0x7fbd3800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.075+0000 7fbd3effd700 1 -- 172.21.15.16:0/3125826052 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd38009b70 con 0x7fbd4008fef0 2019-12-05T06:14:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.075+0000 7fbd44bcf700 1 --2- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd4008fef0 0x7fbd4008f460 secure :-1 s=THROTTLE_DONE pgs=312 cs=0 l=1 rx=0x7fbd3800b300 tx=0x7fbd3800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.075+0000 7fbd3effd700 1 -- 172.21.15.16:0/3125826052 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd38000df0 con 0x7fbd4008fef0 2019-12-05T06:14:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.075+0000 7fbd45e31700 1 -- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd4008fef0 msgr2=0x7fbd4008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.075+0000 7fbd45e31700 1 --2- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd4008fef0 0x7fbd4008f460 secure :-1 s=READY pgs=312 cs=0 l=1 rx=0x7fbd3800b300 tx=0x7fbd3800bb10).stop 2019-12-05T06:14:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.076+0000 7fbd45e31700 1 -- 172.21.15.16:0/3125826052 shutdown_connections 2019-12-05T06:14:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.076+0000 7fbd45e31700 1 --2- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbd4008fef0 0x7fbd4008f460 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.076+0000 7fbd45e31700 1 --2- 172.21.15.16:0/3125826052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbd4008d2c0 0x7fbd4008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:21.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.076+0000 7fbd45e31700 1 -- 172.21.15.16:0/3125826052 shutdown_connections 2019-12-05T06:14:21.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.077+0000 7fbd45e31700 1 -- 172.21.15.16:0/3125826052 wait complete. 2019-12-05T06:14:21.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.077+0000 7fbd45e31700 1 -- 172.21.15.16:0/3125826052 >> 172.21.15.16:0/3125826052 conn(0x7fbd40061940 msgr2=0x7fbd40096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.081+0000 7fbd45e31700 1 Processor -- start 2019-12-05T06:14:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.082+0000 7fbd45e31700 1 -- start start 2019-12-05T06:14:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.083+0000 7fbd45e31700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd400a0910 0x7fbd4014d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:21.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.083+0000 7fbd45e31700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbd400a0140 0x7fbd40157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.083+0000 7fbd45e31700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd40070770 con 0x7fbd4014d410 2019-12-05T06:14:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.083+0000 7fbd45e31700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd40070400 con 0x7fbd400a0910 2019-12-05T06:14:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.083+0000 7fbd45e31700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd40070c70 con 0x7fbd400a0140 2019-12-05T06:14:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.083+0000 7fbd3f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd400a0910 0x7fbd4014d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.084+0000 7fbd3f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbd400a0910 0x7fbd4014d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49926/0 (socket says 172.21.15.16:49926) 2019-12-05T06:14:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.084+0000 7fbd3f7fe700 1 -- 172.21.15.16:0/3209603304 learned_addr learned my addr 172.21.15.16:0/3209603304 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:21.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.084+0000 7fbd44bcf700 1 --2- 172.21.15.16:0/3209603304 >> v2:172.21.15.76:3300/0 conn(0x7fbd400a0140 0x7fbd40157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.084+0000 7fbd3f7fe700 1 -- 172.21.15.16:0/3209603304 >> v2:172.21.15.76:3300/0 conn(0x7fbd400a0140 msgr2=0x7fbd40157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.084+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> v2:172.21.15.76:3300/0 conn(0x7fbd400a0140 0x7fbd40157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.084+0000 7fbd3f7fe700 1 -- 172.21.15.16:0/3209603304 >> v1:172.21.15.76:6789/0 conn(0x7fbd4014d410 legacy=0x7fbd4009f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.084+0000 7fbd3f7fe700 1 -- 172.21.15.16:0/3209603304 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd4009f040 con 0x7fbd400a0910 2019-12-05T06:14:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.084+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> v2:172.21.15.16:3300/0 conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.084+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> v2:172.21.15.16:3300/0 conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.084+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=READY pgs=399 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.085+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=THROTTLE_DONE pgs=399 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.085+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=THROTTLE_DONE pgs=399 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.085+0000 7fbd3cff9700 1 -- 172.21.15.16:0/3209603304 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd34000940 con 0x7fbd400a0910 2019-12-05T06:14:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.085+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=THROTTLE_DONE pgs=399 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.085+0000 7fbd3cff9700 1 -- 172.21.15.16:0/3209603304 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd34000e80 con 0x7fbd400a0910 2019-12-05T06:14:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.085+0000 7fbd3cff9700 1 -- 172.21.15.16:0/3209603304 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbd34001550 con 0x7fbd400a0910 2019-12-05T06:14:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.085+0000 7fbd45e31700 1 -- 172.21.15.16:0/3209603304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd4009f040 con 0x7fbd400a0910 2019-12-05T06:14:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.085+0000 7fbd45e31700 1 -- 172.21.15.16:0/3209603304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd4009f040 con 0x7fbd400a0910 2019-12-05T06:14:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.086+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=THROTTLE_DONE pgs=399 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.087+0000 7fbd3cff9700 1 -- 172.21.15.16:0/3209603304 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbd34001070 con 0x7fbd400a0910 2019-12-05T06:14:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.087+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=THROTTLE_DONE pgs=399 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.087+0000 7fbd3cff9700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd20023630 0x7fbd20025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:21.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.087+0000 7fbd3cff9700 1 -- 172.21.15.16:0/3209603304 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fbd34045dc0 con 0x7fbd400a0910 2019-12-05T06:14:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.087+0000 7fbd3ffff700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd20023630 0x7fbd20025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:21.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.088+0000 7fbd3ffff700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd20023630 0x7fbd20025ac0 crc :-1 s=READY pgs=182 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:21.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.102+0000 7fbd45e31700 1 -- 172.21.15.16:0/3209603304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd4009f040 con 0x7fbd400a0910 2019-12-05T06:14:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.110+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=THROTTLE_DONE pgs=399 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.110+0000 7fbd3cff9700 1 -- 172.21.15.16:0/3209603304 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbd34007610 con 0x7fbd400a0910 2019-12-05T06:14:21.130 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-double-null.sh... 2019-12-05T06:14:21.130 INFO:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snaptest-double-null.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-double-null.sh 2019-12-05T06:14:21.208 INFO:tasks.workunit.client.0.smithi016.stderr:+ set -e 2019-12-05T06:14:21.208 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:14:21.325 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.312+0000 7fcfdbef6700 1 Processor -- start 2019-12-05T06:14:21.325 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.312+0000 7fcfdbef6700 1 -- start start 2019-12-05T06:14:21.325 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.313+0000 7fcfdbef6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcfd41490a0 0x7fcfd4149490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.313+0000 7fcfdbef6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcfd409e9f0 0x7fcfd4152df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.313+0000 7fcfdbef6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd409ee50 con 0x7fcfd409f1c0 2019-12-05T06:14:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.313+0000 7fcfdbef6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd40a3f40 con 0x7fcfd409e9f0 2019-12-05T06:14:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.313+0000 7fcfdbef6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd409d260 con 0x7fcfd41490a0 2019-12-05T06:14:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.313+0000 7fcfdac94700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcfd409e9f0 0x7fcfd4152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.314+0000 7fcfdac94700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcfd409e9f0 0x7fcfd4152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49934/0 (socket says 172.21.15.16:49934) 2019-12-05T06:14:21.326 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.314+0000 7fcfdac94700 1 -- 172.21.15.16:0/3922655999 learned_addr learned my addr 172.21.15.16:0/3922655999 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:21.327 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.314+0000 7fcfda493700 1 --2- 172.21.15.16:0/3922655999 >> v2:172.21.15.76:3300/0 conn(0x7fcfd41490a0 0x7fcfd4149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:21.327 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.314+0000 7fcfdac94700 1 -- 172.21.15.16:0/3922655999 >> v2:172.21.15.76:3300/0 conn(0x7fcfd41490a0 msgr2=0x7fcfd4149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:21.327 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.314+0000 7fcfdac94700 1 --2- 172.21.15.16:0/3922655999 >> v2:172.21.15.76:3300/0 conn(0x7fcfd41490a0 0x7fcfd4149490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:21.327 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.314+0000 7fcfdac94700 1 -- 172.21.15.16:0/3922655999 >> v1:172.21.15.76:6789/0 conn(0x7fcfd409f1c0 legacy=0x7fcfd409e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:21.328 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.314+0000 7fcfdac94700 1 -- 172.21.15.16:0/3922655999 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd409d8f0 con 0x7fcfd409e9f0 2019-12-05T06:14:21.328 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.314+0000 7fcfdac94700 1 --2- 172.21.15.16:0/3922655999 >> v2:172.21.15.16:3300/0 conn(0x7fcfd409e9f0 0x7fcfd4152df0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcfc4008550 tx=0x7fcfc400fa80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.314+0000 7fcfdac94700 1 --2- 172.21.15.16:0/3922655999 >> v2:172.21.15.16:3300/0 conn(0x7fcfd409e9f0 0x7fcfd4152df0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcfc4008550 tx=0x7fcfc400fa80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.314+0000 7fcfdac94700 1 --2- 172.21.15.16:0/3922655999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd409e9f0 0x7fcfd4152df0 secure :-1 s=READY pgs=400 cs=0 l=1 rx=0x7fcfc4008550 tx=0x7fcfc400fa80).ready entity=mon.0 client_cookie=34fbb9aaa0c6a46e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.315+0000 7fcfdac94700 1 --2- 172.21.15.16:0/3922655999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd409e9f0 0x7fcfd4152df0 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7fcfc4008550 tx=0x7fcfc400fa80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.315+0000 7fcfdac94700 1 --2- 172.21.15.16:0/3922655999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd409e9f0 0x7fcfd4152df0 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7fcfc4008550 tx=0x7fcfc400fa80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.315+0000 7fcfd9491700 1 -- 172.21.15.16:0/3922655999 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfc4000bc0 con 0x7fcfd409e9f0 2019-12-05T06:14:21.329 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.315+0000 7fcfdac94700 1 --2- 172.21.15.16:0/3922655999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd409e9f0 0x7fcfd4152df0 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7fcfc4008550 tx=0x7fcfc400fa80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.330 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.315+0000 7fcfd9491700 1 -- 172.21.15.16:0/3922655999 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfc40010e0 con 0x7fcfd409e9f0 2019-12-05T06:14:21.330 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.315+0000 7fcfd9491700 1 -- 172.21.15.16:0/3922655999 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfc40017f0 con 0x7fcfd409e9f0 2019-12-05T06:14:21.330 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.315+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3922655999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd409e9f0 msgr2=0x7fcfd4152df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:21.331 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.315+0000 7fcfdbef6700 1 --2- 172.21.15.16:0/3922655999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd409e9f0 0x7fcfd4152df0 secure :-1 s=READY pgs=400 cs=0 l=1 rx=0x7fcfc4008550 tx=0x7fcfc400fa80).stop 2019-12-05T06:14:21.331 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.316+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3922655999 shutdown_connections 2019-12-05T06:14:21.332 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.316+0000 7fcfdbef6700 1 --2- 172.21.15.16:0/3922655999 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd409e9f0 0x7fcfd4152df0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:21.332 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.316+0000 7fcfdbef6700 1 --2- 172.21.15.16:0/3922655999 >> v2:172.21.15.76:3300/0 conn(0x7fcfd41490a0 0x7fcfd4149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:21.332 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.316+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3922655999 shutdown_connections 2019-12-05T06:14:21.333 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.316+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3922655999 wait complete. 2019-12-05T06:14:21.333 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.316+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3922655999 >> 172.21.15.16:0/3922655999 conn(0x7fcfd4062220 msgr2=0x7fcfd40a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:21.333 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.319+0000 7fcfdbef6700 1 Processor -- start 2019-12-05T06:14:21.333 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.319+0000 7fcfdbef6700 1 -- start start 2019-12-05T06:14:21.333 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.319+0000 7fcfdbef6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcfd411ed30 0x7fcfd4123bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:21.333 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.319+0000 7fcfdbef6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcfd40800e0 0x7fcfd407f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:21.334 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfdbef6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcfd4120f40 con 0x7fcfd407fb90 2019-12-05T06:14:21.334 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfdbef6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcfd4122210 con 0x7fcfd40800e0 2019-12-05T06:14:21.334 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfdbef6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcfd41224e0 con 0x7fcfd411ed30 2019-12-05T06:14:21.334 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcfd40800e0 0x7fcfd407f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:21.334 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcfd40800e0 0x7fcfd407f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49940/0 (socket says 172.21.15.16:49940) 2019-12-05T06:14:21.334 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 -- 172.21.15.16:0/3689074062 learned_addr learned my addr 172.21.15.16:0/3689074062 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:21.334 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfda493700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcfd411ed30 0x7fcfd4123bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:21.335 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 -- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcfd411ed30 msgr2=0x7fcfd4123bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:21.335 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcfd411ed30 0x7fcfd4123bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:21.335 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 -- 172.21.15.16:0/3689074062 >> v1:172.21.15.16:6789/0 conn(0x7fcfd407fb90 legacy=0x7fcfd411f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:21.335 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 -- 172.21.15.16:0/3689074062 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfd41232d0 con 0x7fcfd40800e0 2019-12-05T06:14:21.335 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> v2:172.21.15.16:3300/0 conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.335 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfda493700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcfd411ed30 0x7fcfd4123bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:14:21.335 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> v2:172.21.15.16:3300/0 conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.336 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=READY pgs=402 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:21.336 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.336 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfcaffd700 1 -- 172.21.15.16:0/3689074062 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfd00010a0 con 0x7fcfd40800e0 2019-12-05T06:14:21.336 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.336 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.320+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.337 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.321+0000 7fcfcaffd700 1 -- 172.21.15.16:0/3689074062 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcfd0000940 con 0x7fcfd40800e0 2019-12-05T06:14:21.337 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.321+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfd41232d0 con 0x7fcfd40800e0 2019-12-05T06:14:21.337 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.321+0000 7fcfcaffd700 1 -- 172.21.15.16:0/3689074062 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcfd0001680 con 0x7fcfd40800e0 2019-12-05T06:14:21.337 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.321+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfd4120f40 con 0x7fcfd40800e0 2019-12-05T06:14:21.338 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.321+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.339 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.322+0000 7fcfcaffd700 1 -- 172.21.15.16:0/3689074062 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcfd00010c0 con 0x7fcfd40800e0 2019-12-05T06:14:21.339 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.322+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.339 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.323+0000 7fcfcaffd700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcfbc023630 0x7fcfbc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:21.339 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.323+0000 7fcfcaffd700 1 -- 172.21.15.16:0/3689074062 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6431+0+0 (secure 0 0 0) 0x7fcfd0045d90 con 0x7fcfd40800e0 2019-12-05T06:14:21.339 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.323+0000 7fcfda493700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcfbc023630 0x7fcfbc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:21.340 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.323+0000 7fcfda493700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcfbc023630 0x7fcfbc025ac0 crc :-1 s=READY pgs=184 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:21.340 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.328+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfd41232d0 con 0x7fcfd40800e0 2019-12-05T06:14:21.344 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.335+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.345 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.335+0000 7fcfcaffd700 1 -- 172.21.15.16:0/3689074062 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcfd0000eb0 con 0x7fcfd40800e0 2019-12-05T06:14:21.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.338+0000 7fbd45e31700 1 -- 172.21.15.16:0/3209603304 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd4009f040 con 0x7fbd400a0910 2019-12-05T06:14:21.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.339+0000 7fbd3f7fe700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=THROTTLE_DONE pgs=399 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:21.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.339+0000 7fbd3cff9700 1 -- 172.21.15.16:0/3209603304 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fbd34001070 con 0x7fbd400a0910 2019-12-05T06:14:21.356 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 46 2019-12-05T06:14:21.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 -- 172.21.15.16:0/3209603304 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd20023630 msgr2=0x7fbd20025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:21.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd20023630 0x7fbd20025ac0 crc :-1 s=READY pgs=182 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:21.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 -- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 msgr2=0x7fbd4014d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:21.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 0x7fbd4014d800 secure :-1 s=READY pgs=399 cs=0 l=1 rx=0x7fbd34010460 tx=0x7fbd34010290).stop 2019-12-05T06:14:21.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 -- 172.21.15.16:0/3209603304 shutdown_connections 2019-12-05T06:14:21.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 --2- 172.21.15.16:0/3209603304 >> v2:172.21.15.76:3300/0 conn(0x7fbd400a0140 0x7fbd40157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:21.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbd20023630 0x7fbd20025ac0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:21.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 --2- 172.21.15.16:0/3209603304 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbd400a0910 0x7fbd4014d800 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:21.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 -- 172.21.15.16:0/3209603304 shutdown_connections 2019-12-05T06:14:21.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 -- 172.21.15.16:0/3209603304 wait complete. 2019-12-05T06:14:21.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:14:21.342+0000 7fbd45e31700 1 -- 172.21.15.16:0/3209603304 >> 172.21.15.16:0/3209603304 conn(0x7fbd4008acd0 msgr2=0x7fbd40099650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:21.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:14:21.364 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":46,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":46,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:20.183066+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:14:21.365 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:14:21.365 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 76 secs before thrashing 2019-12-05T06:14:21.529 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:21.534+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcfd41232d0 con 0x7fcfd40800e0 2019-12-05T06:14:22.197 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:22.202+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:22.197 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:22.203+0000 7fcfcaffd700 1 -- 172.21.15.16:0/3689074062 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v47) v1 ==== 167+0+0 (secure 0 0 0) 0x7fcfd002adb0 con 0x7fcfd40800e0 2019-12-05T06:14:22.452 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:22.458+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fcfd41232d0 con 0x7fcfd40800e0 2019-12-05T06:14:23.210 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.216+0000 7fcfd9c92700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:23.211 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.216+0000 7fcfcaffd700 1 -- 172.21.15.16:0/3689074062 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v48) v1 ==== 167+0+0 (secure 0 0 0) 0x7fcfd002adb0 con 0x7fcfd40800e0 2019-12-05T06:14:23.223 INFO:tasks.workunit.client.0.smithi016.stderr:enabled new snapshots 2019-12-05T06:14:23.223 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.228+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcfbc023630 msgr2=0x7fcfbc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:23.223 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.228+0000 7fcfdbef6700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcfbc023630 0x7fcfbc025ac0 crc :-1 s=READY pgs=184 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:23.223 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.228+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 msgr2=0x7fcfd407f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:23.223 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.228+0000 7fcfdbef6700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 secure :-1 s=READY pgs=402 cs=0 l=1 rx=0x7fcfd0010410 tx=0x7fcfd00107b0).stop 2019-12-05T06:14:23.224 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.228+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 shutdown_connections 2019-12-05T06:14:23.224 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.228+0000 7fcfdbef6700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcfbc023630 0x7fcfbc025ac0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:23.224 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.228+0000 7fcfdbef6700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcfd40800e0 0x7fcfd407f650 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:23.224 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.228+0000 7fcfdbef6700 1 --2- 172.21.15.16:0/3689074062 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcfd411ed30 0x7fcfd4123bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:23.224 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.229+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 shutdown_connections 2019-12-05T06:14:23.224 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.229+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 wait complete. 2019-12-05T06:14:23.225 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:23.229+0000 7fcfdbef6700 1 -- 172.21.15.16:0/3689074062 >> 172.21.15.16:0/3689074062 conn(0x7fcfd4079ff0 msgr2=0x7fcfd407ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:23.245 INFO:tasks.workunit.client.0.smithi016.stderr:++ seq 1 20 2019-12-05T06:14:23.245 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:23.245 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:23.246 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:23.246 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:23.273 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:23.284 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:23.290 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:23.458 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:23.480 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:23.481 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:23.481 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:23.481 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:23.481 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:23.481 INFO:tasks.workunit.client.0.smithi016.stdout: 21641 ? 00:00:00 ps 2019-12-05T06:14:23.484 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:24.042 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:24.053 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:24.064 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:24.065 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:24.069 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:24.069 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:24.090 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:24.103 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:24.108 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:24.173 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:24.189 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:24.189 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:24.189 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:24.189 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:24.189 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:24.189 INFO:tasks.workunit.client.0.smithi016.stdout: 21652 ? 00:00:00 ps 2019-12-05T06:14:24.193 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:24.218 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:24.225 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:24.236 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:24.237 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:24.244 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:24.245 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:24.259 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:24.270 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:24.275 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:24.370 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:24.388 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:24.388 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:24.388 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:24.389 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:24.389 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:24.389 INFO:tasks.workunit.client.0.smithi016.stdout: 21662 ? 00:00:00 ps 2019-12-05T06:14:24.392 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:24.425 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:24.426 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:24.436 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:24.436 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:24.440 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:24.440 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:24.457 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:24.468 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:24.473 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:24.541 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:24.560 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:24.561 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:24.561 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:24.561 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:24.561 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:24.561 INFO:tasks.workunit.client.0.smithi016.stdout: 21672 ? 00:00:00 ps 2019-12-05T06:14:24.563 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:24.593 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:24.601 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:24.609 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:24.610 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:24.613 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:24.614 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:24.632 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:24.642 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:24.646 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:24.708 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:24.728 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:24.728 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:24.728 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:24.728 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:24.729 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:24.729 INFO:tasks.workunit.client.0.smithi016.stdout: 21682 ? 00:00:00 ps 2019-12-05T06:14:24.732 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:24.761 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:24.768 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:24.779 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:24.780 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:24.784 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:24.784 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:24.802 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:25.012 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:25.020 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:25.092 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:25.114 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:25.114 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:25.114 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:25.115 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:25.115 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:25.115 INFO:tasks.workunit.client.0.smithi016.stdout: 21693 ? 00:00:00 ps 2019-12-05T06:14:25.116 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:25.674 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:25.682 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:25.694 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:25.694 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:25.698 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:25.699 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:25.719 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:25.735 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:25.736 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:25.834 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:25.854 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:25.854 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:25.854 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:25.854 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:25.854 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:25.855 INFO:tasks.workunit.client.0.smithi016.stdout: 21703 ? 00:00:00 ps 2019-12-05T06:14:25.858 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:26.607 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:26.615 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:26.626 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:26.626 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:26.631 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:26.631 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:26.655 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:26.668 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:26.673 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:26.751 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:26.767 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:26.767 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:26.767 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:26.767 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:26.767 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:26.768 INFO:tasks.workunit.client.0.smithi016.stdout: 21715 ? 00:00:00 ps 2019-12-05T06:14:26.770 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:26.798 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:26.806 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:26.817 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:26.818 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:26.818 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:26.818 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:26.838 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:26.848 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:26.852 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:26.900 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:26.913 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:26.914 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:26.914 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:26.914 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:26.914 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:26.914 INFO:tasks.workunit.client.0.smithi016.stdout: 21725 ? 00:00:00 ps 2019-12-05T06:14:26.916 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:26.940 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:26.946 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:26.957 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:26.958 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:26.960 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:26.961 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:26.980 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:26.990 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:26.994 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:27.067 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:27.087 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:27.087 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:27.087 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:27.087 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:27.088 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:27.088 INFO:tasks.workunit.client.0.smithi016.stdout: 21738 ? 00:00:00 ps 2019-12-05T06:14:27.091 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:27.254 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:27.263 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:27.277 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:27.277 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:27.282 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:27.282 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:27.308 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:27.321 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:27.325 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:27.409 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:27.429 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:27.429 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:27.429 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:27.429 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:27.430 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:27.430 INFO:tasks.workunit.client.0.smithi016.stdout: 21749 ? 00:00:00 ps 2019-12-05T06:14:27.433 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:27.464 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:27.471 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:27.483 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:27.483 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:27.488 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:27.488 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:27.507 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:27.520 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:27.524 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:27.613 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:27.635 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:27.635 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:27.635 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:27.635 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:27.635 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:27.635 INFO:tasks.workunit.client.0.smithi016.stdout: 21759 ? 00:00:00 ps 2019-12-05T06:14:27.638 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:27.679 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:27.680 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:27.688 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:27.688 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:27.692 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:27.692 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:27.714 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:27.725 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:27.730 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:27.801 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:27.818 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:27.818 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:27.818 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:27.818 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:27.818 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:27.818 INFO:tasks.workunit.client.0.smithi016.stdout: 21769 ? 00:00:00 ps 2019-12-05T06:14:27.822 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:28.635 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:28.642 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:28.658 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:28.658 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:28.662 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:28.662 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:28.688 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:28.701 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:28.706 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:28.777 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:28.789 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:28.789 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:28.790 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:28.790 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:28.790 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:28.790 INFO:tasks.workunit.client.0.smithi016.stdout: 21780 ? 00:00:00 ps 2019-12-05T06:14:28.793 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:29.477 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:29.485 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:29.501 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:29.501 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:29.506 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:29.507 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:29.530 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:29.546 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:29.549 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:29.661 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:29.681 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:29.681 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:29.681 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:29.681 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:29.682 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:29.682 INFO:tasks.workunit.client.0.smithi016.stdout: 21790 ? 00:00:00 ps 2019-12-05T06:14:29.686 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:30.091 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:14:30.093 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:30.098 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:14:30.102 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:30.116 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:30.116 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:30.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:30.121 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:30.144 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:30.156 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:30.160 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:30.277 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:30.297 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:30.297 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:30.297 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:30.297 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:30.297 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:30.297 INFO:tasks.workunit.client.0.smithi016.stdout: 21829 ? 00:00:00 ps 2019-12-05T06:14:30.301 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:30.341 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:30.347 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:30.358 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:30.359 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:30.363 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:30.363 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:30.384 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:30.396 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:30.401 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:30.504 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:30.524 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:30.525 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:30.525 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:30.533 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:30.533 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:30.533 INFO:tasks.workunit.client.0.smithi016.stdout: 21839 ? 00:00:00 ps 2019-12-05T06:14:30.534 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:30.562 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:30.569 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:30.580 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:30.581 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:30.585 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:30.585 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:30.606 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:30.619 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:30.623 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:30.686 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:30.699 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:30.700 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:30.700 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:30.700 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:30.700 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:30.701 INFO:tasks.workunit.client.0.smithi016.stdout: 21849 ? 00:00:00 ps 2019-12-05T06:14:30.702 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:30.731 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:30.736 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:30.746 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:30.746 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:30.749 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:30.749 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:30.768 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:30.777 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:30.782 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:30.845 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:30.860 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:30.861 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:30.861 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:30.861 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:30.861 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:30.861 INFO:tasks.workunit.client.0.smithi016.stdout: 21859 ? 00:00:00 ps 2019-12-05T06:14:30.864 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:30.895 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:30.899 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:30.909 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 20`' 2019-12-05T06:14:30.909 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a 2019-12-05T06:14:30.915 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/one 2019-12-05T06:14:30.916 INFO:tasks.workunit.client.0.smithi016.stderr:+ cat 2019-12-05T06:14:30.932 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir a/.snap/two 2019-12-05T06:14:30.942 INFO:tasks.workunit.client.0.smithi016.stderr:+ chmod 777 a/foo 2019-12-05T06:14:30.946 INFO:tasks.workunit.client.0.smithi016.stderr:+ sync 2019-12-05T06:14:31.005 INFO:tasks.workunit.client.0.smithi016.stderr:+ ps 2019-12-05T06:14:31.020 INFO:tasks.workunit.client.0.smithi016.stdout: PID TTY TIME CMD 2019-12-05T06:14:31.020 INFO:tasks.workunit.client.0.smithi016.stdout: 3722 ? 00:00:01 sshd 2019-12-05T06:14:31.020 INFO:tasks.workunit.client.0.smithi016.stdout: 21572 ? 00:00:00 bash 2019-12-05T06:14:31.021 INFO:tasks.workunit.client.0.smithi016.stdout: 21598 ? 00:00:00 timeout 2019-12-05T06:14:31.021 INFO:tasks.workunit.client.0.smithi016.stdout: 21600 ? 00:00:00 snaptest-double 2019-12-05T06:14:31.021 INFO:tasks.workunit.client.0.smithi016.stdout: 21869 ? 00:00:00 ps 2019-12-05T06:14:31.024 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a/.snap/one a/.snap/two 2019-12-05T06:14:31.072 INFO:tasks.workunit.client.0.smithi016.stderr:+ rm a/foo 2019-12-05T06:14:31.077 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir a 2019-12-05T06:14:31.088 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2019-12-05T06:14:31.089 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2019-12-05T06:14:31.090 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:14:31.090 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:14:31.337 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-estale.sh... 2019-12-05T06:14:31.337 INFO:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snaptest-estale.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-estale.sh 2019-12-05T06:14:31.445 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:14:31.618 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.606+0000 7fdc5d21b700 1 Processor -- start 2019-12-05T06:14:31.618 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.606+0000 7fdc5d21b700 1 -- start start 2019-12-05T06:14:31.618 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.607+0000 7fdc5d21b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc581490a0 0x7fdc58149490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:31.618 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.607+0000 7fdc5d21b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdc5809f1c0 0x7fdc5809e1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:31.619 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.608+0000 7fdc5d21b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc5809e730 0x7fdc5809ae70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:31.619 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.608+0000 7fdc5d21b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc580a3f40 con 0x7fdc5809e730 2019-12-05T06:14:31.619 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.608+0000 7fdc5d21b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc5809d260 con 0x7fdc5809f1c0 2019-12-05T06:14:31.619 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.608+0000 7fdc5d21b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5809c830 con 0x7fdc581490a0 2019-12-05T06:14:31.619 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.608+0000 7fdc5ca1a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc5809e730 0x7fdc5809ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:31.619 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.608+0000 7fdc5ca1a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc5809e730 0x7fdc5809ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:49976/0 (socket says 172.21.15.16:49976) 2019-12-05T06:14:31.619 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.608+0000 7fdc5ca1a700 1 -- 172.21.15.16:0/679551874 learned_addr learned my addr 172.21.15.16:0/679551874 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:31.620 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.608+0000 7fdc56d9d700 1 --2- 172.21.15.16:0/679551874 >> v2:172.21.15.76:3300/0 conn(0x7fdc5809f1c0 0x7fdc5809e1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:31.620 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.608+0000 7fdc5759e700 1 --2- 172.21.15.16:0/679551874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc581490a0 0x7fdc58149490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:31.620 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.609+0000 7fdc5ca1a700 1 -- 172.21.15.16:0/679551874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc581490a0 msgr2=0x7fdc58149490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:31.620 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.609+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/679551874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc581490a0 0x7fdc58149490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:31.621 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.609+0000 7fdc5ca1a700 1 -- 172.21.15.16:0/679551874 >> v2:172.21.15.76:3300/0 conn(0x7fdc5809f1c0 msgr2=0x7fdc5809e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:31.622 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.609+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/679551874 >> v2:172.21.15.76:3300/0 conn(0x7fdc5809f1c0 0x7fdc5809e1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:31.622 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.609+0000 7fdc5ca1a700 1 -- 172.21.15.16:0/679551874 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5809d8f0 con 0x7fdc5809e730 2019-12-05T06:14:31.634 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.609+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/679551874 >> v2:172.21.15.16:3300/0 conn(0x7fdc5809e730 0x7fdc5809ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdc4800f730 tx=0x7fdc4800f5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.634 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.609+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/679551874 >> v2:172.21.15.16:3300/0 conn(0x7fdc5809e730 0x7fdc5809ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdc4800f730 tx=0x7fdc4800f5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.635 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.609+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/679551874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc5809e730 0x7fdc5809ae70 secure :-1 s=READY pgs=406 cs=0 l=1 rx=0x7fdc4800f730 tx=0x7fdc4800f5b0).ready entity=mon.0 client_cookie=47eb54d4b670934b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:31.635 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.610+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/679551874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc5809e730 0x7fdc5809ae70 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7fdc4800f730 tx=0x7fdc4800f5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.635 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.610+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/679551874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc5809e730 0x7fdc5809ae70 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7fdc4800f730 tx=0x7fdc4800f5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.635 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.610+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/679551874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc5809e730 0x7fdc5809ae70 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7fdc4800f730 tx=0x7fdc4800f5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.635 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.610+0000 7fdc5659c700 1 -- 172.21.15.16:0/679551874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc48000a70 con 0x7fdc5809e730 2019-12-05T06:14:31.635 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.610+0000 7fdc5659c700 1 -- 172.21.15.16:0/679551874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc480010e0 con 0x7fdc5809e730 2019-12-05T06:14:31.636 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.610+0000 7fdc5659c700 1 -- 172.21.15.16:0/679551874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc480017b0 con 0x7fdc5809e730 2019-12-05T06:14:31.636 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.610+0000 7fdc5d21b700 1 -- 172.21.15.16:0/679551874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc5809e730 msgr2=0x7fdc5809ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:31.636 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.610+0000 7fdc5d21b700 1 --2- 172.21.15.16:0/679551874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc5809e730 0x7fdc5809ae70 secure :-1 s=READY pgs=406 cs=0 l=1 rx=0x7fdc4800f730 tx=0x7fdc4800f5b0).stop 2019-12-05T06:14:31.636 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.612+0000 7fdc5d21b700 1 -- 172.21.15.16:0/679551874 shutdown_connections 2019-12-05T06:14:31.636 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.612+0000 7fdc5d21b700 1 --2- 172.21.15.16:0/679551874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc5809e730 0x7fdc5809ae70 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:31.636 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.612+0000 7fdc5d21b700 1 --2- 172.21.15.16:0/679551874 >> v2:172.21.15.76:3300/0 conn(0x7fdc5809f1c0 0x7fdc5809e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:31.637 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.612+0000 7fdc5d21b700 1 --2- 172.21.15.16:0/679551874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc581490a0 0x7fdc58149490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:31.637 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.612+0000 7fdc5d21b700 1 -- 172.21.15.16:0/679551874 shutdown_connections 2019-12-05T06:14:31.637 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.612+0000 7fdc5d21b700 1 -- 172.21.15.16:0/679551874 wait complete. 2019-12-05T06:14:31.637 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.612+0000 7fdc5d21b700 1 -- 172.21.15.16:0/679551874 >> 172.21.15.16:0/679551874 conn(0x7fdc58062220 msgr2=0x7fdc580a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:31.637 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.617+0000 7fdc5d21b700 1 Processor -- start 2019-12-05T06:14:31.637 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.617+0000 7fdc5d21b700 1 -- start start 2019-12-05T06:14:31.637 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.618+0000 7fdc5d21b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc58130480 0x7fdc5814fb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:31.637 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.618+0000 7fdc5d21b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc58120d10 con 0x7fdc5814f720 2019-12-05T06:14:31.638 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.618+0000 7fdc5d21b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc581211c0 con 0x7fdc58130b60 2019-12-05T06:14:31.638 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.618+0000 7fdc5d21b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc581386e0 con 0x7fdc58130480 2019-12-05T06:14:31.638 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.619+0000 7fdc5ca1a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc58130480 0x7fdc5814fb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:31.638 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.619+0000 7fdc5759e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdc5814f720 0x7fdc5812fc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35148/0 (socket says 172.21.15.16:35148) 2019-12-05T06:14:31.638 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.619+0000 7fdc5759e700 1 -- 172.21.15.16:0/2412771137 learned_addr learned my addr 172.21.15.16:0/2412771137 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:14:31.638 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.619+0000 7fdc5ca1a700 1 -- 172.21.15.16:0/2412771137 >> v1:172.21.15.76:6789/0 conn(0x7fdc58130b60 legacy=0x7fdc581573b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:31.638 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.619+0000 7fdc5ca1a700 1 -- 172.21.15.16:0/2412771137 >> v1:172.21.15.16:6789/0 conn(0x7fdc5814f720 legacy=0x7fdc5812fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:31.639 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.619+0000 7fdc5ca1a700 1 -- 172.21.15.16:0/2412771137 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc581376a0 con 0x7fdc58130480 2019-12-05T06:14:31.639 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.619+0000 7fdc4ffff700 1 -- 172.21.15.16:0/2412771137 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 418900659 0 0) 0x7fdc50001430 con 0x7fdc5814f720 2019-12-05T06:14:31.639 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.619+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> v2:172.21.15.16:3300/0 conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.639 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.620+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> v2:172.21.15.16:3300/0 conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.639 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.620+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=READY pgs=408 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:31.639 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.620+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.639 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.620+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.640 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.620+0000 7fdc4ffff700 1 -- 172.21.15.16:0/2412771137 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc480014a0 con 0x7fdc58130480 2019-12-05T06:14:31.640 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.620+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.640 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.620+0000 7fdc4ffff700 1 -- 172.21.15.16:0/2412771137 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc480010c0 con 0x7fdc58130480 2019-12-05T06:14:31.640 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.620+0000 7fdc4ffff700 1 -- 172.21.15.16:0/2412771137 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc48001a80 con 0x7fdc58130480 2019-12-05T06:14:31.640 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.620+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc581376a0 con 0x7fdc58130480 2019-12-05T06:14:31.640 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.620+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc581376a0 con 0x7fdc58130480 2019-12-05T06:14:31.640 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.621+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.641 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.622+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.641 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.622+0000 7fdc4ffff700 1 -- 172.21.15.16:0/2412771137 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdc480012b0 con 0x7fdc58130480 2019-12-05T06:14:31.641 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.622+0000 7fdc4ffff700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdc3c023630 0x7fdc3c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:31.641 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.623+0000 7fdc4ffff700 1 -- 172.21.15.16:0/2412771137 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 6463+0+0 (secure 0 0 0) 0x7fdc48089df0 con 0x7fdc58130480 2019-12-05T06:14:31.641 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.623+0000 7fdc5759e700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdc3c023630 0x7fdc3c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:31.641 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.623+0000 7fdc5759e700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdc3c023630 0x7fdc3c025ac0 crc :-1 s=READY pgs=185 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:31.642 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.638+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc581376a0 con 0x7fdc58130480 2019-12-05T06:14:31.642 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.645+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:31.642 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.645+0000 7fdc4ffff700 1 -- 172.21.15.16:0/2412771137 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdc4806cee0 con 0x7fdc58130480 2019-12-05T06:14:31.907 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:31.913+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdc581376a0 con 0x7fdc58130480 2019-12-05T06:14:32.052 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:32.056+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:32.052 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:32.056+0000 7fdc4ffff700 1 -- 172.21.15.16:0/2412771137 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v49) v1 ==== 167+0+0 (secure 0 0 0) 0x7fdc480012b0 con 0x7fdc58130480 2019-12-05T06:14:32.259 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:32.265+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fdc581376a0 con 0x7fdc58130480 2019-12-05T06:14:33.057 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.063+0000 7fdc5ca1a700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=THROTTLE_DONE pgs=408 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:33.057 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.063+0000 7fdc4ffff700 1 -- 172.21.15.16:0/2412771137 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v50) v1 ==== 167+0+0 (secure 0 0 0) 0x7fdc48000eb0 con 0x7fdc58130480 2019-12-05T06:14:33.074 INFO:tasks.workunit.client.0.smithi016.stderr:enabled new snapshots 2019-12-05T06:14:33.079 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.085+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdc3c023630 msgr2=0x7fdc3c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:33.079 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.085+0000 7fdc5d21b700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdc3c023630 0x7fdc3c025ac0 crc :-1 s=READY pgs=185 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:33.079 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.085+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 msgr2=0x7fdc5814fb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:33.080 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.085+0000 7fdc5d21b700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 secure :-1 s=READY pgs=408 cs=0 l=1 rx=0x7fdc48054380 tx=0x7fdc48054720).stop 2019-12-05T06:14:33.080 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.085+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 shutdown_connections 2019-12-05T06:14:33.080 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.085+0000 7fdc5d21b700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdc3c023630 0x7fdc3c025ac0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:33.081 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.085+0000 7fdc5d21b700 1 --2- 172.21.15.16:0/2412771137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc58130480 0x7fdc5814fb10 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:33.081 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.085+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 shutdown_connections 2019-12-05T06:14:33.081 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.085+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 wait complete. 2019-12-05T06:14:33.081 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:33.085+0000 7fdc5d21b700 1 -- 172.21.15.16:0/2412771137 >> 172.21.15.16:0/2412771137 conn(0x7fdc5807c080 msgr2=0x7fdc5811a900 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:33.093 INFO:tasks.workunit.client.0.smithi016.stderr:+ mkdir .snap/foo 2019-12-05T06:14:33.117 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo 'We want ENOENT, not ESTALE, here.' 2019-12-05T06:14:33.117 INFO:tasks.workunit.client.0.smithi016.stderr:++ seq 1 100 2019-12-05T06:14:33.117 INFO:tasks.workunit.client.0.smithi016.stdout:We want ENOENT, not ESTALE, here. 2019-12-05T06:14:33.118 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/1 2019-12-05T06:14:33.119 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.130 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/1’: No such file or directory 2019-12-05T06:14:33.131 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/2’: No such file or directory 2019-12-05T06:14:33.131 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.131 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/2 2019-12-05T06:14:33.132 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.132 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.132 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/3 2019-12-05T06:14:33.132 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.136 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/3’: No such file or directory 2019-12-05T06:14:33.136 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.137 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/4 2019-12-05T06:14:33.137 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.141 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/4’: No such file or directory 2019-12-05T06:14:33.141 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.142 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/5 2019-12-05T06:14:33.142 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.149 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/5’: No such file or directory 2019-12-05T06:14:33.150 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.150 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/6 2019-12-05T06:14:33.150 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.151 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/6’: No such file or directory 2019-12-05T06:14:33.152 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.152 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/7 2019-12-05T06:14:33.153 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.157 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/7’: No such file or directory 2019-12-05T06:14:33.157 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.158 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/8 2019-12-05T06:14:33.158 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.162 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/8’: No such file or directory 2019-12-05T06:14:33.162 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.162 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/9 2019-12-05T06:14:33.162 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.167 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/9’: No such file or directory 2019-12-05T06:14:33.167 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.167 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/10 2019-12-05T06:14:33.167 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.172 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/10’: No such file or directory 2019-12-05T06:14:33.172 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.172 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/11 2019-12-05T06:14:33.173 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.177 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/11’: No such file or directory 2019-12-05T06:14:33.177 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.177 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/12 2019-12-05T06:14:33.177 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.182 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/12’: No such file or directory 2019-12-05T06:14:33.182 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.182 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/13 2019-12-05T06:14:33.182 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.186 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/13’: No such file or directory 2019-12-05T06:14:33.187 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.187 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/14 2019-12-05T06:14:33.187 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.191 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/14’: No such file or directory 2019-12-05T06:14:33.191 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.191 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/15 2019-12-05T06:14:33.191 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.195 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/15’: No such file or directory 2019-12-05T06:14:33.196 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.196 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/16 2019-12-05T06:14:33.196 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.200 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/16’: No such file or directory 2019-12-05T06:14:33.200 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.201 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/17 2019-12-05T06:14:33.201 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.204 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/17’: No such file or directory 2019-12-05T06:14:33.204 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.204 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/18 2019-12-05T06:14:33.205 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.208 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/18’: No such file or directory 2019-12-05T06:14:33.209 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.209 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/19 2019-12-05T06:14:33.209 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/19’: No such file or directory 2019-12-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.213 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/20 2019-12-05T06:14:33.214 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.217 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/20’: No such file or directory 2019-12-05T06:14:33.217 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.217 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/21 2019-12-05T06:14:33.218 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.221 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/21’: No such file or directory 2019-12-05T06:14:33.222 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.222 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/22 2019-12-05T06:14:33.223 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.225 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/22’: No such file or directory 2019-12-05T06:14:33.226 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.226 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/23 2019-12-05T06:14:33.226 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.230 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/23’: No such file or directory 2019-12-05T06:14:33.230 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.230 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/24 2019-12-05T06:14:33.230 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.234 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/24’: No such file or directory 2019-12-05T06:14:33.234 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.234 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/25 2019-12-05T06:14:33.234 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.238 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/25’: No such file or directory 2019-12-05T06:14:33.238 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.239 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/26 2019-12-05T06:14:33.239 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.242 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/26’: No such file or directory 2019-12-05T06:14:33.242 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.243 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/27 2019-12-05T06:14:33.243 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.246 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/27’: No such file or directory 2019-12-05T06:14:33.246 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.247 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/28 2019-12-05T06:14:33.247 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.250 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/28’: No such file or directory 2019-12-05T06:14:33.250 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.250 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/29 2019-12-05T06:14:33.251 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.254 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/29’: No such file or directory 2019-12-05T06:14:33.255 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.255 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/30 2019-12-05T06:14:33.255 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.260 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/30’: No such file or directory 2019-12-05T06:14:33.260 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.261 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/31 2019-12-05T06:14:33.261 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.264 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/31’: No such file or directory 2019-12-05T06:14:33.264 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.264 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/32 2019-12-05T06:14:33.264 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.267 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/32’: No such file or directory 2019-12-05T06:14:33.268 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.268 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/33 2019-12-05T06:14:33.268 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.271 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/33’: No such file or directory 2019-12-05T06:14:33.271 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.271 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/34 2019-12-05T06:14:33.272 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.275 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/34’: No such file or directory 2019-12-05T06:14:33.275 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.275 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/35 2019-12-05T06:14:33.275 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.278 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/35’: No such file or directory 2019-12-05T06:14:33.279 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.279 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/36 2019-12-05T06:14:33.279 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.281 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/36’: No such file or directory 2019-12-05T06:14:33.282 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.282 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/37 2019-12-05T06:14:33.282 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.285 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/37’: No such file or directory 2019-12-05T06:14:33.285 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.285 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/38 2019-12-05T06:14:33.285 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.288 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/38’: No such file or directory 2019-12-05T06:14:33.289 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.289 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/39 2019-12-05T06:14:33.289 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.911 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/39’: No such file or directory 2019-12-05T06:14:33.911 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.911 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/40 2019-12-05T06:14:33.912 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.925 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/40’: No such file or directory 2019-12-05T06:14:33.926 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.926 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/41 2019-12-05T06:14:33.926 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.927 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/41’: No such file or directory 2019-12-05T06:14:33.927 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.928 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/42 2019-12-05T06:14:33.928 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.934 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/42’: No such file or directory 2019-12-05T06:14:33.934 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.935 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/43 2019-12-05T06:14:33.935 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.941 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/43’: No such file or directory 2019-12-05T06:14:33.941 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.942 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/44 2019-12-05T06:14:33.942 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.948 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/44’: No such file or directory 2019-12-05T06:14:33.948 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.949 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/45 2019-12-05T06:14:33.953 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.954 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/45’: No such file or directory 2019-12-05T06:14:33.955 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.955 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/46 2019-12-05T06:14:33.957 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.960 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/46’: No such file or directory 2019-12-05T06:14:33.961 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.961 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/47 2019-12-05T06:14:33.961 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.968 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/47’: No such file or directory 2019-12-05T06:14:33.968 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.969 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/48 2019-12-05T06:14:33.969 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.981 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/48’: No such file or directory 2019-12-05T06:14:33.981 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/49’: No such file or directory 2019-12-05T06:14:33.982 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.982 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/49 2019-12-05T06:14:33.982 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.982 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.982 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/50 2019-12-05T06:14:33.982 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.983 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/50’: No such file or directory 2019-12-05T06:14:33.983 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.984 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/51 2019-12-05T06:14:33.985 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.990 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/51’: No such file or directory 2019-12-05T06:14:33.991 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.991 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/52 2019-12-05T06:14:33.991 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.993 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/52’: No such file or directory 2019-12-05T06:14:33.993 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.993 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/53 2019-12-05T06:14:33.994 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:33.998 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/53’: No such file or directory 2019-12-05T06:14:33.998 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:33.999 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/54 2019-12-05T06:14:33.999 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.003 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/54’: No such file or directory 2019-12-05T06:14:34.004 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.004 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/55 2019-12-05T06:14:34.004 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.008 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/55’: No such file or directory 2019-12-05T06:14:34.008 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.009 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/56 2019-12-05T06:14:34.009 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.013 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/56’: No such file or directory 2019-12-05T06:14:34.013 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.013 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/57 2019-12-05T06:14:34.014 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.017 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/57’: No such file or directory 2019-12-05T06:14:34.017 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.018 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/58 2019-12-05T06:14:34.018 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.023 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/58’: No such file or directory 2019-12-05T06:14:34.023 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.023 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/59 2019-12-05T06:14:34.023 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.028 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/59’: No such file or directory 2019-12-05T06:14:34.028 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.028 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/60 2019-12-05T06:14:34.029 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.033 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/60’: No such file or directory 2019-12-05T06:14:34.034 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.034 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/61 2019-12-05T06:14:34.034 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.040 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/61’: No such file or directory 2019-12-05T06:14:34.040 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.041 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/62 2019-12-05T06:14:34.041 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.042 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/62’: No such file or directory 2019-12-05T06:14:34.043 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.043 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/63 2019-12-05T06:14:34.043 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.047 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/63’: No such file or directory 2019-12-05T06:14:34.048 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.048 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/64 2019-12-05T06:14:34.048 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.053 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/64’: No such file or directory 2019-12-05T06:14:34.053 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.053 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/65 2019-12-05T06:14:34.053 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.056 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/65’: No such file or directory 2019-12-05T06:14:34.057 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.057 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/66 2019-12-05T06:14:34.057 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.061 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/66’: No such file or directory 2019-12-05T06:14:34.061 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.062 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/67 2019-12-05T06:14:34.062 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.065 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/67’: No such file or directory 2019-12-05T06:14:34.066 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.066 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/68 2019-12-05T06:14:34.066 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.994 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/68’: No such file or directory 2019-12-05T06:14:34.994 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:34.994 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/69 2019-12-05T06:14:34.995 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:34.999 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/69’: No such file or directory 2019-12-05T06:14:35.000 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.000 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/70 2019-12-05T06:14:35.001 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.012 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/70’: No such file or directory 2019-12-05T06:14:35.013 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.013 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/71 2019-12-05T06:14:35.013 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.016 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/71’: No such file or directory 2019-12-05T06:14:35.016 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.017 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/72 2019-12-05T06:14:35.017 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.024 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/72’: No such file or directory 2019-12-05T06:14:35.024 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.024 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/73 2019-12-05T06:14:35.024 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.029 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/73’: No such file or directory 2019-12-05T06:14:35.029 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.030 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/74 2019-12-05T06:14:35.030 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.036 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/74’: No such file or directory 2019-12-05T06:14:35.037 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.037 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/75 2019-12-05T06:14:35.037 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.042 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/75’: No such file or directory 2019-12-05T06:14:35.042 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.043 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/76 2019-12-05T06:14:35.043 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.047 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/76’: No such file or directory 2019-12-05T06:14:35.048 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.048 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/77 2019-12-05T06:14:35.048 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.053 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/77’: No such file or directory 2019-12-05T06:14:35.053 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.054 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/78 2019-12-05T06:14:35.054 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.061 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/78’: No such file or directory 2019-12-05T06:14:35.061 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.061 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/79 2019-12-05T06:14:35.062 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.065 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/79’: No such file or directory 2019-12-05T06:14:35.065 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.065 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/80 2019-12-05T06:14:35.066 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.071 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/80’: No such file or directory 2019-12-05T06:14:35.071 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.071 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/81 2019-12-05T06:14:35.071 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.076 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/81’: No such file or directory 2019-12-05T06:14:35.076 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.077 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/82 2019-12-05T06:14:35.077 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.082 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/82’: No such file or directory 2019-12-05T06:14:35.082 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.082 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/83 2019-12-05T06:14:35.083 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.086 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/83’: No such file or directory 2019-12-05T06:14:35.086 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.086 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/84 2019-12-05T06:14:35.086 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.091 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/84’: No such file or directory 2019-12-05T06:14:35.091 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.092 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/85 2019-12-05T06:14:35.092 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.096 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/85’: No such file or directory 2019-12-05T06:14:35.096 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.097 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/86 2019-12-05T06:14:35.097 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.101 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/86’: No such file or directory 2019-12-05T06:14:35.101 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.110 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/87 2019-12-05T06:14:35.110 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.111 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.111 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/88 2019-12-05T06:14:35.112 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.112 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/87’: No such file or directory 2019-12-05T06:14:35.115 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/88’: No such file or directory 2019-12-05T06:14:35.115 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/89’: No such file or directory 2019-12-05T06:14:35.115 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.115 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/89 2019-12-05T06:14:35.115 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.116 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.116 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/90 2019-12-05T06:14:35.116 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.119 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/90’: No such file or directory 2019-12-05T06:14:35.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/91 2019-12-05T06:14:35.120 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.124 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/91’: No such file or directory 2019-12-05T06:14:35.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/92 2019-12-05T06:14:35.125 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.129 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/92’: No such file or directory 2019-12-05T06:14:35.130 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.130 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/93 2019-12-05T06:14:35.130 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.133 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/93’: No such file or directory 2019-12-05T06:14:35.133 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.134 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/94 2019-12-05T06:14:35.134 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.137 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/94’: No such file or directory 2019-12-05T06:14:35.137 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.138 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/95 2019-12-05T06:14:35.138 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.142 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/95’: No such file or directory 2019-12-05T06:14:35.142 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.142 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/96 2019-12-05T06:14:35.142 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.148 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/96’: No such file or directory 2019-12-05T06:14:35.148 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.148 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/97 2019-12-05T06:14:35.149 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.151 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/97’: No such file or directory 2019-12-05T06:14:35.152 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.153 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/98 2019-12-05T06:14:35.153 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.155 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/98’: No such file or directory 2019-12-05T06:14:35.155 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.155 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/99 2019-12-05T06:14:35.155 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.158 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/99’: No such file or directory 2019-12-05T06:14:35.159 INFO:tasks.workunit.client.0.smithi016.stderr:+ for f in '`seq 1 100`' 2019-12-05T06:14:35.159 INFO:tasks.workunit.client.0.smithi016.stderr:+ stat .snap/foo/100 2019-12-05T06:14:35.159 INFO:tasks.workunit.client.0.smithi016.stderr:+ grep 'No such file' 2019-12-05T06:14:35.162 INFO:tasks.workunit.client.0.smithi016.stdout:stat: cannot stat ‘.snap/foo/100’: No such file or directory 2019-12-05T06:14:35.162 INFO:tasks.workunit.client.0.smithi016.stderr:+ rmdir .snap/foo 2019-12-05T06:14:35.175 INFO:tasks.workunit.client.0.smithi016.stdout:OK 2019-12-05T06:14:35.175 INFO:tasks.workunit.client.0.smithi016.stderr:+ echo OK 2019-12-05T06:14:35.176 INFO:teuthology.orchestra.run:Running command with timeout 3600 2019-12-05T06:14:35.176 INFO:teuthology.orchestra.run.smithi016:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2019-12-05T06:14:35.296 INFO:tasks.workunit:Running workunit fs/snaps/snaptest-git-ceph.sh... 2019-12-05T06:14:35.296 INFO:teuthology.orchestra.run.smithi016:workunit test fs/snaps/snaptest-git-ceph.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c5290805033483ba0cec165c06bde07dae5f19ec TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/snaps/snaptest-git-ceph.sh 2019-12-05T06:14:35.401 INFO:tasks.workunit.client.0.smithi016.stderr:+ set -e 2019-12-05T06:14:35.401 INFO:tasks.workunit.client.0.smithi016.stderr:+ ceph fs set cephfs allow_new_snaps true --yes-i-really-mean-it 2019-12-05T06:14:35.577 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.571+0000 7ff1148dc700 1 Processor -- start 2019-12-05T06:14:35.577 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.572+0000 7ff1148dc700 1 -- start start 2019-12-05T06:14:35.578 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.574+0000 7ff1148dc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:35.578 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.574+0000 7ff1148dc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff110149490 0x7ff110152df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:14:35.578 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.574+0000 7ff1148dc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff110149930 con 0x7ff1101490a0 2019-12-05T06:14:35.579 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.574+0000 7ff1148dc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1100a3f40 con 0x7ff110149490 2019-12-05T06:14:35.579 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.574+0000 7ff1148dc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11009d260 con 0x7ff11009f1c0 2019-12-05T06:14:35.579 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.574+0000 7ff10f59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff110149490 0x7ff110152df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:35.579 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.574+0000 7ff10e59c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:35.580 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.574+0000 7ff10e59c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:54004/0 (socket says 172.21.15.16:54004) 2019-12-05T06:14:35.580 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.574+0000 7ff10f59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff110149490 0x7ff110152df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:36104/0 (socket says 172.21.15.16:36104) 2019-12-05T06:14:35.580 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.574+0000 7ff10e59c700 1 -- 172.21.15.16:0/1032685159 learned_addr learned my addr 172.21.15.16:0/1032685159 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:35.580 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.575+0000 7ff10e59c700 1 -- 172.21.15.16:0/1032685159 >> v2:172.21.15.76:3300/0 conn(0x7ff110149490 msgr2=0x7ff110152df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:14:35.581 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.575+0000 7ff10e59c700 1 --2- 172.21.15.16:0/1032685159 >> v2:172.21.15.76:3300/0 conn(0x7ff110149490 0x7ff110152df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:35.585 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.575+0000 7ff10e59c700 1 -- 172.21.15.16:0/1032685159 >> v1:172.21.15.76:6789/0 conn(0x7ff1101490a0 legacy=0x7ff11009e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:35.585 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.575+0000 7ff10dd9b700 1 -- 172.21.15.16:0/1032685159 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183817004 0 0) 0x7ff108001400 con 0x7ff1101490a0 2019-12-05T06:14:35.585 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.575+0000 7ff10e59c700 1 -- 172.21.15.16:0/1032685159 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11009d8f0 con 0x7ff11009f1c0 2019-12-05T06:14:35.585 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.575+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1032685159 >> v2:172.21.15.76:3300/0 conn(0x7ff110149490 0x7ff110152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:14:35.586 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.575+0000 7ff10e59c700 1 --2- 172.21.15.16:0/1032685159 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff0f800f740 tx=0x7ff0f800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.586 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.576+0000 7ff10e59c700 1 --2- 172.21.15.16:0/1032685159 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff0f800f740 tx=0x7ff0f800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.586 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.576+0000 7ff10e59c700 1 --2- 172.21.15.16:0/1032685159 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7ff0f800f740 tx=0x7ff0f800fae0).ready entity=mon.2 client_cookie=d08c3bcc0b25902 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:35.586 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.576+0000 7ff10e59c700 1 --2- 172.21.15.16:0/1032685159 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7ff0f800f740 tx=0x7ff0f800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.587 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.576+0000 7ff10dd9b700 1 -- 172.21.15.16:0/1032685159 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0f8000960 con 0x7ff11009f1c0 2019-12-05T06:14:35.588 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.576+0000 7ff10e59c700 1 --2- 172.21.15.16:0/1032685159 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7ff0f800f740 tx=0x7ff0f800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.588 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.576+0000 7ff10dd9b700 1 -- 172.21.15.16:0/1032685159 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f8000c90 con 0x7ff11009f1c0 2019-12-05T06:14:35.588 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.576+0000 7ff10e59c700 1 --2- 172.21.15.16:0/1032685159 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 secure :-1 s=THROTTLE_DONE pgs=107 cs=0 l=1 rx=0x7ff0f800f740 tx=0x7ff0f800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.588 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.576+0000 7ff10dd9b700 1 -- 172.21.15.16:0/1032685159 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0f8007210 con 0x7ff11009f1c0 2019-12-05T06:14:35.588 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.576+0000 7ff1148dc700 1 -- 172.21.15.16:0/1032685159 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 msgr2=0x7ff11009e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:35.589 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.577+0000 7ff1148dc700 1 --2- 172.21.15.16:0/1032685159 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 secure :-1 s=READY pgs=107 cs=0 l=1 rx=0x7ff0f800f740 tx=0x7ff0f800fae0).stop 2019-12-05T06:14:35.589 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.578+0000 7ff1148dc700 1 -- 172.21.15.16:0/1032685159 shutdown_connections 2019-12-05T06:14:35.589 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.578+0000 7ff1148dc700 1 --2- 172.21.15.16:0/1032685159 >> v2:172.21.15.76:3300/0 conn(0x7ff110149490 0x7ff110152df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:14:35.589 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.578+0000 7ff1148dc700 1 --2- 172.21.15.16:0/1032685159 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff11009f1c0 0x7ff11009e9f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:35.589 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.579+0000 7ff1148dc700 1 -- 172.21.15.16:0/1032685159 shutdown_connections 2019-12-05T06:14:35.589 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.579+0000 7ff1148dc700 1 -- 172.21.15.16:0/1032685159 wait complete. 2019-12-05T06:14:35.590 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.579+0000 7ff1148dc700 1 -- 172.21.15.16:0/1032685159 >> 172.21.15.16:0/1032685159 conn(0x7ff110062220 msgr2=0x7ff1100a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.584+0000 7ff1148dc700 1 Processor -- start 2019-12-05T06:14:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.584+0000 7ff1148dc700 1 -- start start 2019-12-05T06:14:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.585+0000 7ff1148dc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff1100793f0 0x7ff1100797e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.585+0000 7ff1148dc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff1101311a0 0x7ff110079d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.585+0000 7ff1148dc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1101301d0 0x7ff110130650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:35.594 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.585+0000 7ff1148dc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff110137be0 con 0x7ff1101301d0 2019-12-05T06:14:35.595 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.585+0000 7ff1148dc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff11011e500 con 0x7ff1101311a0 2019-12-05T06:14:35.595 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.585+0000 7ff1148dc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff11011e7d0 con 0x7ff1100793f0 2019-12-05T06:14:35.595 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1101301d0 0x7ff110130650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:35.595 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff1101301d0 0x7ff110130650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50022/0 (socket says 172.21.15.16:50022) 2019-12-05T06:14:35.595 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10e59c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff1101311a0 0x7ff110079d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:35.595 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10f59e700 1 -- 172.21.15.16:0/1757147135 learned_addr learned my addr 172.21.15.16:0/1757147135 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:14:35.595 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10ed9d700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff1100793f0 0x7ff1100797e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:35.596 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10f59e700 1 -- 172.21.15.16:0/1757147135 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff1100793f0 msgr2=0x7ff1100797e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:35.596 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff1100793f0 0x7ff1100797e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:35.596 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10f59e700 1 -- 172.21.15.16:0/1757147135 >> v2:172.21.15.76:3300/0 conn(0x7ff1101311a0 msgr2=0x7ff110079d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:35.596 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> v2:172.21.15.76:3300/0 conn(0x7ff1101311a0 0x7ff110079d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:35.596 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10f59e700 1 -- 172.21.15.16:0/1757147135 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11011f5c0 con 0x7ff1101301d0 2019-12-05T06:14:35.597 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.586+0000 7ff10e59c700 1 --2- 172.21.15.16:0/1757147135 >> v2:172.21.15.76:3300/0 conn(0x7ff1101311a0 0x7ff110079d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:14:35.597 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> v2:172.21.15.16:3300/0 conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.597 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> v2:172.21.15.16:3300/0 conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.597 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=READY pgs=409 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:35.597 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.598 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff1077fe700 1 -- 172.21.15.16:0/1757147135 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff100000a70 con 0x7ff1101301d0 2019-12-05T06:14:35.598 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.599 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff1077fe700 1 -- 172.21.15.16:0/1757147135 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1000095b0 con 0x7ff1101301d0 2019-12-05T06:14:35.599 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.599 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff11011f5c0 con 0x7ff1101301d0 2019-12-05T06:14:35.599 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff1077fe700 1 -- 172.21.15.16:0/1757147135 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff10000aa60 con 0x7ff1101301d0 2019-12-05T06:14:35.599 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.587+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff11011f5c0 con 0x7ff1101301d0 2019-12-05T06:14:35.599 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.588+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.600 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.589+0000 7ff1077fe700 1 -- 172.21.15.16:0/1757147135 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff10000aa60 con 0x7ff1101301d0 2019-12-05T06:14:35.600 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.589+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.600 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.590+0000 7ff1077fe700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff0f4023630 0x7ff0f4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:14:35.600 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.590+0000 7ff1077fe700 1 -- 172.21.15.16:0/1757147135 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 6431+0+0 (secure 0 0 0) 0x7ff100045d80 con 0x7ff1101301d0 2019-12-05T06:14:35.600 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.590+0000 7ff10ed9d700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff0f4023630 0x7ff0f4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:14:35.600 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.590+0000 7ff10ed9d700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff0f4023630 0x7ff0f4025ac0 crc :-1 s=READY pgs=186 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:14:35.601 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.603+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff11011f5c0 con 0x7ff1101301d0 2019-12-05T06:14:35.606 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.612+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:35.606 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.612+0000 7ff1077fe700 1 -- 172.21.15.16:0/1757147135 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff10000a750 con 0x7ff1101301d0 2019-12-05T06:14:35.867 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:35.874+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff11011f5c0 con 0x7ff1101301d0 2019-12-05T06:14:36.066 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:36.072+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:36.067 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:36.072+0000 7ff1077fe700 1 -- 172.21.15.16:0/1757147135 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v51) v1 ==== 167+0+0 (secure 0 0 0) 0x7ff10000aa60 con 0x7ff1101301d0 2019-12-05T06:14:36.263 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:36.269+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true} v 0) v1 -- 0x7ff11011f5c0 con 0x7ff1101301d0 2019-12-05T06:14:37.074 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.080+0000 7ff10f59e700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:14:37.074 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.080+0000 7ff1077fe700 1 -- 172.21.15.16:0/1757147135 <== mon.0 v2:172.21.15.16:3300/0 8 ==== mon_command_ack([{"var": "allow_new_snaps", "prefix": "fs set", "fs_name": "cephfs", "val": "true", "yes_i_really_mean_it": true}]=0 enabled new snapshots v52) v1 ==== 167+0+0 (secure 0 0 0) 0x7ff10000a750 con 0x7ff1101301d0 2019-12-05T06:14:37.078 INFO:tasks.workunit.client.0.smithi016.stderr:enabled new snapshots 2019-12-05T06:14:37.084 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.089+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff0f4023630 msgr2=0x7ff0f4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:37.084 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.089+0000 7ff1148dc700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff0f4023630 0x7ff0f4025ac0 crc :-1 s=READY pgs=186 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.084 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.089+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 msgr2=0x7ff110130650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:14:37.085 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.089+0000 7ff1148dc700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 secure :-1 s=READY pgs=409 cs=0 l=1 rx=0x7ff100010430 tx=0x7ff1000107d0).stop 2019-12-05T06:14:37.085 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.089+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 shutdown_connections 2019-12-05T06:14:37.085 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.089+0000 7ff1148dc700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff0f4023630 0x7ff0f4025ac0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.085 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.089+0000 7ff1148dc700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff1101301d0 0x7ff110130650 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.086 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.089+0000 7ff1148dc700 1 --2- 172.21.15.16:0/1757147135 >> v2:172.21.15.76:3300/0 conn(0x7ff1101311a0 0x7ff110079d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.086 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.089+0000 7ff1148dc700 1 --2- 172.21.15.16:0/1757147135 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff1100793f0 0x7ff1100797e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:14:37.086 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.090+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 shutdown_connections 2019-12-05T06:14:37.086 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.090+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 wait complete. 2019-12-05T06:14:37.086 INFO:tasks.workunit.client.0.smithi016.stderr:2019-12-05T06:14:37.090+0000 7ff1148dc700 1 -- 172.21.15.16:0/1757147135 >> 172.21.15.16:0/1757147135 conn(0x7ff11007c510 msgr2=0x7ff11011f9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:14:37.107 INFO:tasks.workunit.client.0.smithi016.stderr:+ git clone git://git.ceph.com/ceph.git 2019-12-05T06:15:00.310 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:15:00.315 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:15:08.726 INFO:tasks.workunit.client.0.smithi016.stdout:Cloning into 'ceph'... 2019-12-05T06:15:30.465 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:15:30.470 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:15:37.367 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:37.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.659+0000 7f6992dd4700 1 Processor -- start 2019-12-05T06:15:37.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.660+0000 7f6992dd4700 1 -- start start 2019-12-05T06:15:37.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.661+0000 7f6992dd4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f698c08d2c0 0x7f698c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:37.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.661+0000 7f6992dd4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f698c08fef0 0x7f698c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:37.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.662+0000 7f6992dd4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f698c0958e0 con 0x7f698c08c6d0 2019-12-05T06:15:37.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.662+0000 7f6992dd4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f698c08df90 con 0x7f698c08fef0 2019-12-05T06:15:37.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.662+0000 7f6992dd4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f698c0a3f70 con 0x7f698c08d2c0 2019-12-05T06:15:37.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.662+0000 7f6991b72700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f698c08fef0 0x7f698c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:37.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.662+0000 7f6991371700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f698c08c6d0 0x7f698c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35208/0 (socket says 172.21.15.16:35208) 2019-12-05T06:15:37.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.662+0000 7f6991371700 1 -- 172.21.15.16:0/1212462396 learned_addr learned my addr 172.21.15.16:0/1212462396 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:37.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.662+0000 7f6990b70700 1 --2- 172.21.15.16:0/1212462396 >> v2:172.21.15.76:3300/0 conn(0x7f698c08d2c0 0x7f698c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:37.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.663+0000 7f698bfff700 1 -- 172.21.15.16:0/1212462396 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557614418 0 0) 0x7f697c001150 con 0x7f698c08c6d0 2019-12-05T06:15:37.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.663+0000 7f698bfff700 1 -- 172.21.15.16:0/1212462396 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6974002e80 con 0x7f698c08c6d0 2019-12-05T06:15:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.663+0000 7f6990b70700 1 -- 172.21.15.16:0/1212462396 >> v2:172.21.15.16:3300/0 conn(0x7f698c08fef0 msgr2=0x7f698c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:15:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.663+0000 7f6990b70700 1 --2- 172.21.15.16:0/1212462396 >> v2:172.21.15.16:3300/0 conn(0x7f698c08fef0 0x7f698c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.663+0000 7f6990b70700 1 -- 172.21.15.16:0/1212462396 >> v1:172.21.15.16:6789/0 conn(0x7f698c08c6d0 legacy=0x7f698c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.663+0000 7f6990b70700 1 -- 172.21.15.16:0/1212462396 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c08e620 con 0x7f698c08d2c0 2019-12-05T06:15:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.663+0000 7f6991b72700 1 --2- 172.21.15.16:0/1212462396 >> v2:172.21.15.16:3300/0 conn(0x7f698c08fef0 0x7f698c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.663+0000 7f6990b70700 1 --2- 172.21.15.16:0/1212462396 >> v2:172.21.15.76:3300/0 conn(0x7f698c08d2c0 0x7f698c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f698000b2d0 tx=0x7f698000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.663+0000 7f6990b70700 1 --2- 172.21.15.16:0/1212462396 >> v2:172.21.15.76:3300/0 conn(0x7f698c08d2c0 0x7f698c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f698000b2d0 tx=0x7f698000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.663+0000 7f6990b70700 1 --2- 172.21.15.16:0/1212462396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f698c08d2c0 0x7f698c08ef20 secure :-1 s=READY pgs=317 cs=0 l=1 rx=0x7f698000b2d0 tx=0x7f698000bae0).ready entity=mon.1 client_cookie=325a47a2c0c3e25 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.664+0000 7f6990b70700 1 --2- 172.21.15.16:0/1212462396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f698c08d2c0 0x7f698c08ef20 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f698000b2d0 tx=0x7f698000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.664+0000 7f698bfff700 1 -- 172.21.15.16:0/1212462396 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6980007590 con 0x7f698c08d2c0 2019-12-05T06:15:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.664+0000 7f6990b70700 1 --2- 172.21.15.16:0/1212462396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f698c08d2c0 0x7f698c08ef20 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f698000b2d0 tx=0x7f698000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.664+0000 7f6990b70700 1 --2- 172.21.15.16:0/1212462396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f698c08d2c0 0x7f698c08ef20 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f698000b2d0 tx=0x7f698000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.664+0000 7f698bfff700 1 -- 172.21.15.16:0/1212462396 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6980009d30 con 0x7f698c08d2c0 2019-12-05T06:15:37.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.664+0000 7f698bfff700 1 -- 172.21.15.16:0/1212462396 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f698000a400 con 0x7f698c08d2c0 2019-12-05T06:15:37.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.664+0000 7f6992dd4700 1 -- 172.21.15.16:0/1212462396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f698c08d2c0 msgr2=0x7f698c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:37.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.664+0000 7f6992dd4700 1 --2- 172.21.15.16:0/1212462396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f698c08d2c0 0x7f698c08ef20 secure :-1 s=READY pgs=317 cs=0 l=1 rx=0x7f698000b2d0 tx=0x7f698000bae0).stop 2019-12-05T06:15:37.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.666+0000 7f6992dd4700 1 -- 172.21.15.16:0/1212462396 shutdown_connections 2019-12-05T06:15:37.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.666+0000 7f6992dd4700 1 --2- 172.21.15.16:0/1212462396 >> v2:172.21.15.16:3300/0 conn(0x7f698c08fef0 0x7f698c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:37.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.666+0000 7f6992dd4700 1 --2- 172.21.15.16:0/1212462396 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f698c08d2c0 0x7f698c08ef20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:37.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.666+0000 7f6992dd4700 1 -- 172.21.15.16:0/1212462396 shutdown_connections 2019-12-05T06:15:37.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.666+0000 7f6992dd4700 1 -- 172.21.15.16:0/1212462396 wait complete. 2019-12-05T06:15:37.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.666+0000 7f6992dd4700 1 -- 172.21.15.16:0/1212462396 >> 172.21.15.16:0/1212462396 conn(0x7f698c061940 msgr2=0x7f698c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.672+0000 7f6992dd4700 1 Processor -- start 2019-12-05T06:15:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.672+0000 7f6992dd4700 1 -- start start 2019-12-05T06:15:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.673+0000 7f6992dd4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f698c12b640 0x7f698c12ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.674+0000 7f6992dd4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.674+0000 7f6992dd4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f698c131bf0 con 0x7f698c1302f0 2019-12-05T06:15:37.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.674+0000 7f6992dd4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f698c1296e0 con 0x7f698c12b640 2019-12-05T06:15:37.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.674+0000 7f6992dd4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f698c128cb0 con 0x7f698c153b50 2019-12-05T06:15:37.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.674+0000 7f6990b70700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f698c12b640 0x7f698c12ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:37.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.674+0000 7f6991b72700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:37.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.674+0000 7f6991371700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f698c1302f0 0x7f698c12a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:56516/0 (socket says 172.21.15.16:56516) 2019-12-05T06:15:37.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.674+0000 7f6991371700 1 -- 172.21.15.16:0/3914623772 learned_addr learned my addr 172.21.15.16:0/3914623772 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:37.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.675+0000 7f6991b72700 1 -- 172.21.15.16:0/3914623772 >> v2:172.21.15.76:3300/0 conn(0x7f698c12b640 msgr2=0x7f698c12ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:37.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.675+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> v2:172.21.15.76:3300/0 conn(0x7f698c12b640 0x7f698c12ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.675+0000 7f6991b72700 1 -- 172.21.15.16:0/3914623772 >> v1:172.21.15.76:6789/0 conn(0x7f698c1302f0 legacy=0x7f698c12a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.675+0000 7f6989ffb700 1 -- 172.21.15.16:0/3914623772 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620118067 0 0) 0x7f697c000f40 con 0x7f698c1302f0 2019-12-05T06:15:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.675+0000 7f6991b72700 1 -- 172.21.15.16:0/3914623772 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c129d70 con 0x7f698c153b50 2019-12-05T06:15:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.675+0000 7f6990b70700 1 --2- 172.21.15.16:0/3914623772 >> v2:172.21.15.76:3300/0 conn(0x7f698c12b640 0x7f698c12ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.675+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.675+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.675+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=READY pgs=111 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:37.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.676+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.676+0000 7f6989ffb700 1 -- 172.21.15.16:0/3914623772 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f698400edd0 con 0x7f698c153b50 2019-12-05T06:15:37.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.676+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.676+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.676+0000 7f6989ffb700 1 -- 172.21.15.16:0/3914623772 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69840009e0 con 0x7f698c153b50 2019-12-05T06:15:37.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.676+0000 7f6992dd4700 1 -- 172.21.15.16:0/3914623772 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698c129d70 con 0x7f698c153b50 2019-12-05T06:15:37.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.676+0000 7f6989ffb700 1 -- 172.21.15.16:0/3914623772 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f698400f450 con 0x7f698c153b50 2019-12-05T06:15:37.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.676+0000 7f6992dd4700 1 -- 172.21.15.16:0/3914623772 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698c129d70 con 0x7f698c153b50 2019-12-05T06:15:37.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.677+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.678+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.678+0000 7f6989ffb700 1 -- 172.21.15.16:0/3914623772 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f698400eb80 con 0x7f698c153b50 2019-12-05T06:15:37.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.679+0000 7f6989ffb700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f696c023630 0x7f696c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:37.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.679+0000 7f6989ffb700 1 -- 172.21.15.16:0/3914623772 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6419+0+0 (secure 0 0 0) 0x7f6984045d40 con 0x7f698c153b50 2019-12-05T06:15:37.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.679+0000 7f6991371700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f696c023630 0x7f696c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:37.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.680+0000 7f6991371700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f696c023630 0x7f696c025ac0 crc :-1 s=READY pgs=188 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:37.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.691+0000 7f6992dd4700 1 -- 172.21.15.16:0/3914623772 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698c129d70 con 0x7f698c153b50 2019-12-05T06:15:37.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.701+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.701+0000 7f6989ffb700 1 -- 172.21.15.16:0/3914623772 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f698400e860 con 0x7f698c153b50 2019-12-05T06:15:37.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.940+0000 7f6992dd4700 1 -- 172.21.15.16:0/3914623772 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f698c129d70 con 0x7f698c153b50 2019-12-05T06:15:37.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.941+0000 7f6991b72700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=THROTTLE_DONE pgs=111 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:37.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.941+0000 7f6989ffb700 1 -- 172.21.15.16:0/3914623772 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f698400ea70 con 0x7f698c153b50 2019-12-05T06:15:37.937 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:37.937 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:37.070869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:37.938 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2019-12-05T06:15:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.945+0000 7f6992dd4700 1 -- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f696c023630 msgr2=0x7f696c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.945+0000 7f6992dd4700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f696c023630 0x7f696c025ac0 crc :-1 s=READY pgs=188 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:37.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.945+0000 7f6992dd4700 1 -- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 msgr2=0x7f698c155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:37.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.945+0000 7f6992dd4700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 secure :-1 s=READY pgs=111 cs=0 l=1 rx=0x7f69840103d0 tx=0x7f6984010770).stop 2019-12-05T06:15:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.945+0000 7f6992dd4700 1 -- 172.21.15.16:0/3914623772 shutdown_connections 2019-12-05T06:15:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.946+0000 7f6992dd4700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f696c023630 0x7f696c025ac0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.946+0000 7f6992dd4700 1 --2- 172.21.15.16:0/3914623772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f698c153b50 0x7f698c155f50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:37.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.946+0000 7f6992dd4700 1 --2- 172.21.15.16:0/3914623772 >> v2:172.21.15.76:3300/0 conn(0x7f698c12b640 0x7f698c12ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.946+0000 7f6992dd4700 1 -- 172.21.15.16:0/3914623772 shutdown_connections 2019-12-05T06:15:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.946+0000 7f6992dd4700 1 -- 172.21.15.16:0/3914623772 wait complete. 2019-12-05T06:15:37.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:37.946+0000 7f6992dd4700 1 -- 172.21.15.16:0/3914623772 >> 172.21.15.16:0/3914623772 conn(0x7f698c0a2850 msgr2=0x7f698c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:37.950 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:15:37.951 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:15:37.951 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:15:37.965 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:15:37.966 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:38.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.266+0000 7f1732efb700 1 Processor -- start 2019-12-05T06:15:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.266+0000 7f1732efb700 1 -- start start 2019-12-05T06:15:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.267+0000 7f1732efb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f172c08c6d0 0x7f172c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.268+0000 7f1732efb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172c08fb80 con 0x7f172c08fef0 2019-12-05T06:15:38.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.268+0000 7f1732efb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172c0958e0 con 0x7f172c08f720 2019-12-05T06:15:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.268+0000 7f1732efb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f172c08df90 con 0x7f172c08c6d0 2019-12-05T06:15:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.268+0000 7f1731498700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f172c08c6d0 0x7f172c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.268+0000 7f1730c97700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f172c08fef0 0x7f172c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35226/0 (socket says 172.21.15.16:35226) 2019-12-05T06:15:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.268+0000 7f1730c97700 1 -- 172.21.15.16:0/1306622730 learned_addr learned my addr 172.21.15.16:0/1306622730 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:38.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.268+0000 7f1723fff700 1 -- 172.21.15.16:0/1306622730 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959653906 0 0) 0x7f1728001150 con 0x7f172c08fef0 2019-12-05T06:15:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.268+0000 7f1731498700 1 -- 172.21.15.16:0/1306622730 >> v1:172.21.15.76:6789/0 conn(0x7f172c08f720 legacy=0x7f172c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.268+0000 7f1731498700 1 -- 172.21.15.16:0/1306622730 >> v1:172.21.15.16:6789/0 conn(0x7f172c08fef0 legacy=0x7f172c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.268+0000 7f1731498700 1 -- 172.21.15.16:0/1306622730 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172c08e620 con 0x7f172c08c6d0 2019-12-05T06:15:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1731498700 1 --2- 172.21.15.16:0/1306622730 >> v2:172.21.15.16:3300/0 conn(0x7f172c08c6d0 0x7f172c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f172400b2d0 tx=0x7f172400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1731498700 1 --2- 172.21.15.16:0/1306622730 >> v2:172.21.15.16:3300/0 conn(0x7f172c08c6d0 0x7f172c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f172400b2d0 tx=0x7f172400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1731498700 1 --2- 172.21.15.16:0/1306622730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c08c6d0 0x7f172c08cac0 secure :-1 s=READY pgs=412 cs=0 l=1 rx=0x7f172400b2d0 tx=0x7f172400bae0).ready entity=mon.0 client_cookie=7e4a30def876073f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:38.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1731498700 1 --2- 172.21.15.16:0/1306622730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c08c6d0 0x7f172c08cac0 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7f172400b2d0 tx=0x7f172400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1723fff700 1 -- 172.21.15.16:0/1306622730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1724000ce0 con 0x7f172c08c6d0 2019-12-05T06:15:38.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1731498700 1 --2- 172.21.15.16:0/1306622730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c08c6d0 0x7f172c08cac0 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7f172400b2d0 tx=0x7f172400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1731498700 1 --2- 172.21.15.16:0/1306622730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c08c6d0 0x7f172c08cac0 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7f172400b2d0 tx=0x7f172400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1723fff700 1 -- 172.21.15.16:0/1306622730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1724009be0 con 0x7f172c08c6d0 2019-12-05T06:15:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1723fff700 1 -- 172.21.15.16:0/1306622730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f172400a2b0 con 0x7f172c08c6d0 2019-12-05T06:15:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1732efb700 1 -- 172.21.15.16:0/1306622730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c08c6d0 msgr2=0x7f172c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.269+0000 7f1732efb700 1 --2- 172.21.15.16:0/1306622730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c08c6d0 0x7f172c08cac0 secure :-1 s=READY pgs=412 cs=0 l=1 rx=0x7f172400b2d0 tx=0x7f172400bae0).stop 2019-12-05T06:15:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.271+0000 7f1732efb700 1 -- 172.21.15.16:0/1306622730 shutdown_connections 2019-12-05T06:15:38.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.271+0000 7f1732efb700 1 --2- 172.21.15.16:0/1306622730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f172c08c6d0 0x7f172c08cac0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.271+0000 7f1732efb700 1 -- 172.21.15.16:0/1306622730 shutdown_connections 2019-12-05T06:15:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.271+0000 7f1732efb700 1 -- 172.21.15.16:0/1306622730 wait complete. 2019-12-05T06:15:38.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.271+0000 7f1732efb700 1 -- 172.21.15.16:0/1306622730 >> 172.21.15.16:0/1306622730 conn(0x7f172c061940 msgr2=0x7f172c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.275+0000 7f1732efb700 1 Processor -- start 2019-12-05T06:15:38.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.275+0000 7f1732efb700 1 -- start start 2019-12-05T06:15:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1732efb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f172c07a610 0x7f172c079ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1732efb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f172c0896a0 0x7f172c089ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1732efb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f172c099830 con 0x7f172c0892b0 2019-12-05T06:15:38.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1732efb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f172c098f00 con 0x7f172c0896a0 2019-12-05T06:15:38.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1732efb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f172c0a0900 con 0x7f172c07a610 2019-12-05T06:15:38.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1731498700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f172c0892b0 0x7f172c0a1890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35232/0 (socket says 172.21.15.16:35232) 2019-12-05T06:15:38.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1731498700 1 -- 172.21.15.16:0/3712656230 learned_addr learned my addr 172.21.15.16:0/3712656230 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:38.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1731c99700 1 --2- 172.21.15.16:0/3712656230 >> v2:172.21.15.16:3300/0 conn(0x7f172c0896a0 0x7f172c089ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:38.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> v2:172.21.15.76:3300/0 conn(0x7f172c07a610 0x7f172c079ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:38.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1721ffb700 1 -- 172.21.15.16:0/3712656230 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2464487042 0 0) 0x7f172400b750 con 0x7f172c0892b0 2019-12-05T06:15:38.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.277+0000 7f1721ffb700 1 -- 172.21.15.16:0/3712656230 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f170c002e80 con 0x7f172c0892b0 2019-12-05T06:15:38.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1730c97700 1 -- 172.21.15.16:0/3712656230 >> v2:172.21.15.16:3300/0 conn(0x7f172c0896a0 msgr2=0x7f172c089ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:38.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> v2:172.21.15.16:3300/0 conn(0x7f172c0896a0 0x7f172c089ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:38.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1730c97700 1 -- 172.21.15.16:0/3712656230 >> v1:172.21.15.16:6789/0 conn(0x7f172c0892b0 legacy=0x7f172c0a1890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:38.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1730c97700 1 -- 172.21.15.16:0/3712656230 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f172c0a0f90 con 0x7f172c07a610 2019-12-05T06:15:38.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1731c99700 1 --2- 172.21.15.16:0/3712656230 >> v2:172.21.15.16:3300/0 conn(0x7f172c0896a0 0x7f172c089ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:38.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> v2:172.21.15.76:3300/0 conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> v2:172.21.15.76:3300/0 conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=READY pgs=320 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:38.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1721ffb700 1 -- 172.21.15.16:0/3712656230 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1728000ef0 con 0x7f172c07a610 2019-12-05T06:15:38.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1721ffb700 1 -- 172.21.15.16:0/3712656230 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1728004240 con 0x7f172c07a610 2019-12-05T06:15:38.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1732efb700 1 -- 172.21.15.16:0/3712656230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f172c0a0f90 con 0x7f172c07a610 2019-12-05T06:15:38.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1721ffb700 1 -- 172.21.15.16:0/3712656230 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1728001200 con 0x7f172c07a610 2019-12-05T06:15:38.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.278+0000 7f1732efb700 1 -- 172.21.15.16:0/3712656230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f172c0a0f90 con 0x7f172c07a610 2019-12-05T06:15:38.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.279+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.280+0000 7f1721ffb700 1 -- 172.21.15.16:0/3712656230 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1728004430 con 0x7f172c07a610 2019-12-05T06:15:38.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.280+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.280+0000 7f1721ffb700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f170c023670 0x7f170c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:38.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.280+0000 7f1721ffb700 1 -- 172.21.15.16:0/3712656230 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6419+0+0 (secure 0 0 0) 0x7f1728045d20 con 0x7f172c07a610 2019-12-05T06:15:38.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.281+0000 7f1731498700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f170c023670 0x7f170c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:38.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.281+0000 7f1731498700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f170c023670 0x7f170c025ae0 crc :-1 s=READY pgs=189 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:38.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.281+0000 7f1732efb700 1 -- 172.21.15.16:0/3712656230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f172c0a0f90 con 0x7f172c07a610 2019-12-05T06:15:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.290+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.290+0000 7f1721ffb700 1 -- 172.21.15.16:0/3712656230 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1728000c90 con 0x7f172c07a610 2019-12-05T06:15:38.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.502+0000 7f1732efb700 1 -- 172.21.15.16:0/3712656230 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f172c0a0f90 con 0x7f172c07a610 2019-12-05T06:15:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.504+0000 7f1730c97700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:38.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.504+0000 7f1721ffb700 1 -- 172.21.15.16:0/3712656230 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f1728004430 con 0x7f172c07a610 2019-12-05T06:15:38.500 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2019-12-05T06:15:38.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:38.500 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:37.070869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.508+0000 7f1732efb700 1 -- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f170c023670 msgr2=0x7f170c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.508+0000 7f1732efb700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f170c023670 0x7f170c025ae0 crc :-1 s=READY pgs=189 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.508+0000 7f1732efb700 1 -- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 msgr2=0x7f172c079ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.508+0000 7f1732efb700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 0x7f172c079ea0 secure :-1 s=READY pgs=320 cs=0 l=1 rx=0x7f1728010370 tx=0x7f1728010710).stop 2019-12-05T06:15:38.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.508+0000 7f1732efb700 1 -- 172.21.15.16:0/3712656230 shutdown_connections 2019-12-05T06:15:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.508+0000 7f1732efb700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f170c023670 0x7f170c025ae0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.508+0000 7f1732efb700 1 --2- 172.21.15.16:0/3712656230 >> v2:172.21.15.16:3300/0 conn(0x7f172c0896a0 0x7f172c089ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.508+0000 7f1732efb700 1 --2- 172.21.15.16:0/3712656230 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f172c07a610 0x7f172c079ea0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.509+0000 7f1732efb700 1 -- 172.21.15.16:0/3712656230 shutdown_connections 2019-12-05T06:15:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.509+0000 7f1732efb700 1 -- 172.21.15.16:0/3712656230 wait complete. 2019-12-05T06:15:38.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:38.509+0000 7f1732efb700 1 -- 172.21.15.16:0/3712656230 >> 172.21.15.16:0/3712656230 conn(0x7f172c06ecf0 msgr2=0x7f172c0706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:38.514 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:15:40.525 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:40.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.783+0000 7f2554048700 1 Processor -- start 2019-12-05T06:15:40.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.783+0000 7f2554048700 1 -- start start 2019-12-05T06:15:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.784+0000 7f2554048700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f254c08c6d0 0x7f254c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.784+0000 7f2554048700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c08fef0 0x7f254c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.784+0000 7f2554048700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f254c08ef20 0x7f254c08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.784+0000 7f2554048700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c08df90 con 0x7f254c08c6d0 2019-12-05T06:15:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.784+0000 7f2554048700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f254c0a3f70 con 0x7f254c08ef20 2019-12-05T06:15:40.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.784+0000 7f2554048700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c08d830 con 0x7f254c08fef0 2019-12-05T06:15:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.784+0000 7f25525e5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f254c08c6d0 0x7f254c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.785+0000 7f25525e5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f254c08c6d0 0x7f254c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50076/0 (socket says 172.21.15.16:50076) 2019-12-05T06:15:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.785+0000 7f25525e5700 1 -- 172.21.15.16:0/905667335 learned_addr learned my addr 172.21.15.16:0/905667335 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:15:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.785+0000 7f2551de4700 1 --2- 172.21.15.16:0/905667335 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c08fef0 0x7f254c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.785+0000 7f2552de6700 1 --2- 172.21.15.16:0/905667335 >> v2:172.21.15.76:3300/0 conn(0x7f254c08ef20 0x7f254c08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:40.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.785+0000 7f25525e5700 1 -- 172.21.15.16:0/905667335 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c08fef0 msgr2=0x7f254c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:15:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.785+0000 7f25525e5700 1 --2- 172.21.15.16:0/905667335 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c08fef0 0x7f254c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.785+0000 7f25525e5700 1 -- 172.21.15.16:0/905667335 >> v2:172.21.15.76:3300/0 conn(0x7f254c08ef20 msgr2=0x7f254c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:15:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.785+0000 7f25525e5700 1 --2- 172.21.15.16:0/905667335 >> v2:172.21.15.76:3300/0 conn(0x7f254c08ef20 0x7f254c08f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.785+0000 7f25525e5700 1 -- 172.21.15.16:0/905667335 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c08d540 con 0x7f254c08c6d0 2019-12-05T06:15:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.786+0000 7f25525e5700 1 --2- 172.21.15.16:0/905667335 >> v2:172.21.15.16:3300/0 conn(0x7f254c08c6d0 0x7f254c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f254400fa40 tx=0x7f254400f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.786+0000 7f25525e5700 1 --2- 172.21.15.16:0/905667335 >> v2:172.21.15.16:3300/0 conn(0x7f254c08c6d0 0x7f254c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f254400fa40 tx=0x7f254400f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.786+0000 7f25525e5700 1 --2- 172.21.15.16:0/905667335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f254c08c6d0 0x7f254c08cac0 secure :-1 s=READY pgs=414 cs=0 l=1 rx=0x7f254400fa40 tx=0x7f254400f8f0).ready entity=mon.0 client_cookie=f620052e87ec198a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:40.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.786+0000 7f25525e5700 1 --2- 172.21.15.16:0/905667335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f254c08c6d0 0x7f254c08cac0 secure :-1 s=THROTTLE_DONE pgs=414 cs=0 l=1 rx=0x7f254400fa40 tx=0x7f254400f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.786+0000 7f25525e5700 1 --2- 172.21.15.16:0/905667335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f254c08c6d0 0x7f254c08cac0 secure :-1 s=THROTTLE_DONE pgs=414 cs=0 l=1 rx=0x7f254400fa40 tx=0x7f254400f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.786+0000 7f25525e5700 1 --2- 172.21.15.16:0/905667335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f254c08c6d0 0x7f254c08cac0 secure :-1 s=THROTTLE_DONE pgs=414 cs=0 l=1 rx=0x7f254400fa40 tx=0x7f254400f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.786+0000 7f25515e3700 1 -- 172.21.15.16:0/905667335 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2544000ce0 con 0x7f254c08c6d0 2019-12-05T06:15:40.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.787+0000 7f25515e3700 1 -- 172.21.15.16:0/905667335 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2544001010 con 0x7f254c08c6d0 2019-12-05T06:15:40.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.787+0000 7f25515e3700 1 -- 172.21.15.16:0/905667335 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2544001720 con 0x7f254c08c6d0 2019-12-05T06:15:40.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.787+0000 7f2554048700 1 -- 172.21.15.16:0/905667335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f254c08c6d0 msgr2=0x7f254c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:40.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.787+0000 7f2554048700 1 --2- 172.21.15.16:0/905667335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f254c08c6d0 0x7f254c08cac0 secure :-1 s=READY pgs=414 cs=0 l=1 rx=0x7f254400fa40 tx=0x7f254400f8f0).stop 2019-12-05T06:15:40.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.788+0000 7f2554048700 1 -- 172.21.15.16:0/905667335 shutdown_connections 2019-12-05T06:15:40.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.788+0000 7f2554048700 1 --2- 172.21.15.16:0/905667335 >> v2:172.21.15.76:3300/0 conn(0x7f254c08ef20 0x7f254c08f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:40.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.788+0000 7f2554048700 1 --2- 172.21.15.16:0/905667335 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c08fef0 0x7f254c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:40.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.788+0000 7f2554048700 1 --2- 172.21.15.16:0/905667335 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f254c08c6d0 0x7f254c08cac0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:40.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.788+0000 7f2554048700 1 -- 172.21.15.16:0/905667335 shutdown_connections 2019-12-05T06:15:40.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.788+0000 7f2554048700 1 -- 172.21.15.16:0/905667335 wait complete. 2019-12-05T06:15:40.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.788+0000 7f2554048700 1 -- 172.21.15.16:0/905667335 >> 172.21.15.16:0/905667335 conn(0x7f254c061940 msgr2=0x7f254c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:40.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.793+0000 7f2554048700 1 Processor -- start 2019-12-05T06:15:40.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.793+0000 7f2554048700 1 -- start start 2019-12-05T06:15:40.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f2554048700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:40.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f2554048700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254c098da0 con 0x7f254c132d00 2019-12-05T06:15:40.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f2554048700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f254c09f0a0 con 0x7f254c09b850 2019-12-05T06:15:40.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f2554048700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f254c09aa60 con 0x7f254c132530 2019-12-05T06:15:40.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f2551de4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f254c132d00 0x7f254c154b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35252/0 (socket says 172.21.15.16:35252) 2019-12-05T06:15:40.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f2551de4700 1 -- 172.21.15.16:0/1521678506 learned_addr learned my addr 172.21.15.16:0/1521678506 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:40.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:40.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398462545 0 0) 0x7f2540009850 con 0x7f254c132d00 2019-12-05T06:15:40.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f252c002e80 con 0x7f254c132d00 2019-12-05T06:15:40.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123969904 0 0) 0x7f2544007900 con 0x7f254c09b850 2019-12-05T06:15:40.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.795+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f252c004170 con 0x7f254c09b850 2019-12-05T06:15:40.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.796+0000 7f2552de6700 1 -- 172.21.15.16:0/1521678506 >> v1:172.21.15.76:6789/0 conn(0x7f254c09b850 legacy=0x7f254c131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:40.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.796+0000 7f2552de6700 1 -- 172.21.15.16:0/1521678506 >> v1:172.21.15.16:6789/0 conn(0x7f254c132d00 legacy=0x7f254c154b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:40.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.796+0000 7f2552de6700 1 -- 172.21.15.16:0/1521678506 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f254c09b0f0 con 0x7f254c132530 2019-12-05T06:15:40.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.796+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.796+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.796+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:40.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.796+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.796+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f253c009990 con 0x7f254c132530 2019-12-05T06:15:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.796+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.797+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f253c0095b0 con 0x7f254c132530 2019-12-05T06:15:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.797+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.797+0000 7f2554048700 1 -- 172.21.15.16:0/1521678506 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f254c09b0f0 con 0x7f254c132530 2019-12-05T06:15:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.797+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f253c009c20 con 0x7f254c132530 2019-12-05T06:15:40.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.797+0000 7f2554048700 1 -- 172.21.15.16:0/1521678506 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f254c09b0f0 con 0x7f254c132530 2019-12-05T06:15:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.798+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.798+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f253c0095b0 con 0x7f254c132530 2019-12-05T06:15:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.799+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.799+0000 7f254affd700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f252c023650 0x7f252c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.799+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6419+0+0 (secure 0 0 0) 0x7f253c045d70 con 0x7f254c132530 2019-12-05T06:15:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.799+0000 7f25525e5700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f252c023650 0x7f252c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:40.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.800+0000 7f25525e5700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f252c023650 0x7f252c025ae0 crc :-1 s=READY pgs=190 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:40.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.814+0000 7f2554048700 1 -- 172.21.15.16:0/1521678506 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f254c09b0f0 con 0x7f254c132530 2019-12-05T06:15:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.824+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:40.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:40.824+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f253c009a30 con 0x7f254c132530 2019-12-05T06:15:41.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.054+0000 7f2554048700 1 -- 172.21.15.16:0/1521678506 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f254c09b0f0 con 0x7f254c132530 2019-12-05T06:15:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.056+0000 7f2552de6700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=THROTTLE_DONE pgs=112 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:41.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.056+0000 7f254affd700 1 -- 172.21.15.16:0/1521678506 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f253c0095b0 con 0x7f254c132530 2019-12-05T06:15:41.051 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:41.052 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:37.070869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:41.052 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2019-12-05T06:15:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.060+0000 7f2554048700 1 -- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f252c023650 msgr2=0x7f252c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.060+0000 7f2554048700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f252c023650 0x7f252c025ae0 crc :-1 s=READY pgs=190 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.060+0000 7f2554048700 1 -- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 msgr2=0x7f254c158320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:41.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.060+0000 7f2554048700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 secure :-1 s=READY pgs=112 cs=0 l=1 rx=0x7f253c010390 tx=0x7f253c010730).stop 2019-12-05T06:15:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.060+0000 7f2554048700 1 -- 172.21.15.16:0/1521678506 shutdown_connections 2019-12-05T06:15:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.060+0000 7f2554048700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f252c023650 0x7f252c025ae0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.060+0000 7f2554048700 1 --2- 172.21.15.16:0/1521678506 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f254c132530 0x7f254c158320 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.060+0000 7f2554048700 1 -- 172.21.15.16:0/1521678506 shutdown_connections 2019-12-05T06:15:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.060+0000 7f2554048700 1 -- 172.21.15.16:0/1521678506 wait complete. 2019-12-05T06:15:41.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:41.060+0000 7f2554048700 1 -- 172.21.15.16:0/1521678506 >> 172.21.15.16:0/1521678506 conn(0x7f254c0714f0 msgr2=0x7f254c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:41.065 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:15:43.066 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:43.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.320+0000 7fb0af1b3700 1 Processor -- start 2019-12-05T06:15:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.321+0000 7fb0af1b3700 1 -- start start 2019-12-05T06:15:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.322+0000 7fb0af1b3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb0a808c6d0 0x7fb0a808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.322+0000 7fb0af1b3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a808fef0 0x7fb0a808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.322+0000 7fb0af1b3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a80958e0 con 0x7fb0a808ef20 2019-12-05T06:15:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.322+0000 7fb0af1b3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a808df90 con 0x7fb0a808c6d0 2019-12-05T06:15:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.322+0000 7fb0af1b3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a80a3f70 con 0x7fb0a808fef0 2019-12-05T06:15:43.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.323+0000 7fb0adf51700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb0a808ef20 0x7fb0a808f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35262/0 (socket says 172.21.15.16:35262) 2019-12-05T06:15:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.323+0000 7fb0adf51700 1 -- 172.21.15.16:0/49464162 learned_addr learned my addr 172.21.15.16:0/49464162 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.323+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a808fef0 0x7fb0a808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.323+0000 7fb0ad750700 1 --2- 172.21.15.16:0/49464162 >> v2:172.21.15.76:3300/0 conn(0x7fb0a808c6d0 0x7fb0a808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.323+0000 7fb0a7fff700 1 -- 172.21.15.16:0/49464162 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1524295479 0 0) 0x7fb0980014d0 con 0x7fb0a808ef20 2019-12-05T06:15:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.323+0000 7fb0a7fff700 1 -- 172.21.15.16:0/49464162 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb090002e80 con 0x7fb0a808ef20 2019-12-05T06:15:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.324+0000 7fb0ad750700 1 -- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a808fef0 msgr2=0x7fb0a808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:15:43.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.324+0000 7fb0ad750700 1 --2- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a808fef0 0x7fb0a808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.324+0000 7fb0ad750700 1 -- 172.21.15.16:0/49464162 >> v1:172.21.15.16:6789/0 conn(0x7fb0a808ef20 legacy=0x7fb0a808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.324+0000 7fb0ad750700 1 -- 172.21.15.16:0/49464162 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a808d540 con 0x7fb0a808c6d0 2019-12-05T06:15:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.324+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a808fef0 0x7fb0a808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.324+0000 7fb0ad750700 1 --2- 172.21.15.16:0/49464162 >> v2:172.21.15.76:3300/0 conn(0x7fb0a808c6d0 0x7fb0a808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb0a000af50 tx=0x7fb0a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.324+0000 7fb0ad750700 1 --2- 172.21.15.16:0/49464162 >> v2:172.21.15.76:3300/0 conn(0x7fb0a808c6d0 0x7fb0a808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb0a000af50 tx=0x7fb0a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.324+0000 7fb0ad750700 1 --2- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0a808c6d0 0x7fb0a808cac0 secure :-1 s=READY pgs=322 cs=0 l=1 rx=0x7fb0a000af50 tx=0x7fb0a000b760).ready entity=mon.1 client_cookie=2ba8b840750ef43a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:43.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.325+0000 7fb0ad750700 1 --2- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0a808c6d0 0x7fb0a808cac0 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7fb0a000af50 tx=0x7fb0a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.325+0000 7fb0a7fff700 1 -- 172.21.15.16:0/49464162 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0a0007210 con 0x7fb0a808c6d0 2019-12-05T06:15:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.325+0000 7fb0ad750700 1 --2- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0a808c6d0 0x7fb0a808cac0 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7fb0a000af50 tx=0x7fb0a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.325+0000 7fb0ad750700 1 --2- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0a808c6d0 0x7fb0a808cac0 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7fb0a000af50 tx=0x7fb0a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.325+0000 7fb0a7fff700 1 -- 172.21.15.16:0/49464162 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a00099b0 con 0x7fb0a808c6d0 2019-12-05T06:15:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.325+0000 7fb0a7fff700 1 -- 172.21.15.16:0/49464162 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0a000a080 con 0x7fb0a808c6d0 2019-12-05T06:15:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.325+0000 7fb0af1b3700 1 -- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0a808c6d0 msgr2=0x7fb0a808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.325+0000 7fb0af1b3700 1 --2- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0a808c6d0 0x7fb0a808cac0 secure :-1 s=READY pgs=322 cs=0 l=1 rx=0x7fb0a000af50 tx=0x7fb0a000b760).stop 2019-12-05T06:15:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.327+0000 7fb0af1b3700 1 -- 172.21.15.16:0/49464162 shutdown_connections 2019-12-05T06:15:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.327+0000 7fb0af1b3700 1 --2- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a808fef0 0x7fb0a808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.327+0000 7fb0af1b3700 1 --2- 172.21.15.16:0/49464162 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb0a808c6d0 0x7fb0a808cac0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:43.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.327+0000 7fb0af1b3700 1 -- 172.21.15.16:0/49464162 shutdown_connections 2019-12-05T06:15:43.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.327+0000 7fb0af1b3700 1 -- 172.21.15.16:0/49464162 wait complete. 2019-12-05T06:15:43.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.327+0000 7fb0af1b3700 1 -- 172.21.15.16:0/49464162 >> 172.21.15.16:0/49464162 conn(0x7fb0a8061940 msgr2=0x7fb0a8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.332+0000 7fb0af1b3700 1 Processor -- start 2019-12-05T06:15:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.332+0000 7fb0af1b3700 1 -- start start 2019-12-05T06:15:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.334+0000 7fb0af1b3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a814c340 0x7fb0a814c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.334+0000 7fb0af1b3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0a809ecf0 0x7fb0a809dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:43.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.334+0000 7fb0af1b3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0a8135650 con 0x7fb0a809e260 2019-12-05T06:15:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.334+0000 7fb0af1b3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a812cf70 con 0x7fb0a809ecf0 2019-12-05T06:15:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.334+0000 7fb0af1b3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a809c360 con 0x7fb0a814c340 2019-12-05T06:15:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.334+0000 7fb0acf4f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0a809ecf0 0x7fb0a809dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:43.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.335+0000 7fb0acf4f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb0a809ecf0 0x7fb0a809dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50100/0 (socket says 172.21.15.16:50100) 2019-12-05T06:15:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.335+0000 7fb0acf4f700 1 -- 172.21.15.16:0/568373259 learned_addr learned my addr 172.21.15.16:0/568373259 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:15:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.335+0000 7fb0ad750700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a814c340 0x7fb0a814c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.335+0000 7fb0a5ffb700 1 -- 172.21.15.16:0/568373259 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1215498246 0 0) 0x7fb098000d30 con 0x7fb0a809e260 2019-12-05T06:15:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.335+0000 7fb0acf4f700 1 -- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a814c340 msgr2=0x7fb0a814c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:43.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.335+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a814c340 0x7fb0a814c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.335+0000 7fb0acf4f700 1 -- 172.21.15.16:0/568373259 >> v1:172.21.15.76:6789/0 conn(0x7fb0a809e260 legacy=0x7fb0a812ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.335+0000 7fb0acf4f700 1 -- 172.21.15.16:0/568373259 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a809d420 con 0x7fb0a809ecf0 2019-12-05T06:15:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.335+0000 7fb0ad750700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a814c340 0x7fb0a814c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.335+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> v2:172.21.15.16:3300/0 conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.336+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> v2:172.21.15.16:3300/0 conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.336+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=READY pgs=417 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:43.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.336+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=THROTTLE_DONE pgs=417 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.336+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=THROTTLE_DONE pgs=417 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.336+0000 7fb0a5ffb700 1 -- 172.21.15.16:0/568373259 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb09c00a8c0 con 0x7fb0a809ecf0 2019-12-05T06:15:43.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.336+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=THROTTLE_DONE pgs=417 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.336+0000 7fb0a5ffb700 1 -- 172.21.15.16:0/568373259 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb09c009770 con 0x7fb0a809ecf0 2019-12-05T06:15:43.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.336+0000 7fb0a5ffb700 1 -- 172.21.15.16:0/568373259 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb09c009ec0 con 0x7fb0a809ecf0 2019-12-05T06:15:43.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.336+0000 7fb0af1b3700 1 -- 172.21.15.16:0/568373259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a809d420 con 0x7fb0a809ecf0 2019-12-05T06:15:43.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.336+0000 7fb0af1b3700 1 -- 172.21.15.16:0/568373259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a809e9c0 con 0x7fb0a809ecf0 2019-12-05T06:15:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.337+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=THROTTLE_DONE pgs=417 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.339+0000 7fb0a5ffb700 1 -- 172.21.15.16:0/568373259 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb09c009960 con 0x7fb0a809ecf0 2019-12-05T06:15:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.339+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=THROTTLE_DONE pgs=417 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.339+0000 7fb0a5ffb700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb088023630 0x7fb088025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.339+0000 7fb0a5ffb700 1 -- 172.21.15.16:0/568373259 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6419+0+0 (secure 0 0 0) 0x7fb09c045df0 con 0x7fb0a809ecf0 2019-12-05T06:15:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.339+0000 7fb0ad750700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb088023630 0x7fb088025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:43.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.340+0000 7fb0ad750700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb088023630 0x7fb088025ac0 crc :-1 s=READY pgs=191 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:43.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.352+0000 7fb0af1b3700 1 -- 172.21.15.16:0/568373259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a809d420 con 0x7fb0a809ecf0 2019-12-05T06:15:43.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.361+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=THROTTLE_DONE pgs=417 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.361+0000 7fb0a5ffb700 1 -- 172.21.15.16:0/568373259 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb09c01aa20 con 0x7fb0a809ecf0 2019-12-05T06:15:43.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.620+0000 7fb0af1b3700 1 -- 172.21.15.16:0/568373259 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb0a809d420 con 0x7fb0a809ecf0 2019-12-05T06:15:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.621+0000 7fb0acf4f700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=THROTTLE_DONE pgs=417 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.621+0000 7fb0a5ffb700 1 -- 172.21.15.16:0/568373259 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fb09c01aa20 con 0x7fb0a809ecf0 2019-12-05T06:15:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2019-12-05T06:15:43.616 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:43.616 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:37.070869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.623+0000 7fb0af1b3700 1 -- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb088023630 msgr2=0x7fb088025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.623+0000 7fb0af1b3700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb088023630 0x7fb088025ac0 crc :-1 s=READY pgs=191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.623+0000 7fb0af1b3700 1 -- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 msgr2=0x7fb0a809dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.623+0000 7fb0af1b3700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 0x7fb0a809dd20 secure :-1 s=READY pgs=417 cs=0 l=1 rx=0x7fb09c010410 tx=0x7fb09c0107b0).stop 2019-12-05T06:15:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.624+0000 7fb0af1b3700 1 -- 172.21.15.16:0/568373259 shutdown_connections 2019-12-05T06:15:43.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.624+0000 7fb0af1b3700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb088023630 0x7fb088025ac0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.624+0000 7fb0af1b3700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb0a809ecf0 0x7fb0a809dd20 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.624+0000 7fb0af1b3700 1 --2- 172.21.15.16:0/568373259 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb0a814c340 0x7fb0a814c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.624+0000 7fb0af1b3700 1 -- 172.21.15.16:0/568373259 shutdown_connections 2019-12-05T06:15:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.624+0000 7fb0af1b3700 1 -- 172.21.15.16:0/568373259 wait complete. 2019-12-05T06:15:43.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:43.624+0000 7fb0af1b3700 1 -- 172.21.15.16:0/568373259 >> 172.21.15.16:0/568373259 conn(0x7fb0a806ef90 msgr2=0x7fb0a8099690 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:43.628 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:15:45.629 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.906+0000 7f872a0b7700 1 Processor -- start 2019-12-05T06:15:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.906+0000 7f872a0b7700 1 -- start start 2019-12-05T06:15:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.908+0000 7f872a0b7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872408c6d0 0x7f872408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.908+0000 7f872a0b7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f872408f720 0x7f872408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.908+0000 7f872a0b7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87240958e0 con 0x7f872408fef0 2019-12-05T06:15:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.908+0000 7f872a0b7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87240a3f70 con 0x7f872408f720 2019-12-05T06:15:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.908+0000 7f872a0b7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f872408d830 con 0x7f872408c6d0 2019-12-05T06:15:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.909+0000 7f87237fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f872408fef0 0x7f872408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35282/0 (socket says 172.21.15.16:35282) 2019-12-05T06:15:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.909+0000 7f87237fe700 1 -- 172.21.15.16:0/158238294 learned_addr learned my addr 172.21.15.16:0/158238294 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.909+0000 7f8728e55700 1 --2- 172.21.15.16:0/158238294 >> v2:172.21.15.76:3300/0 conn(0x7f872408f720 0x7f872408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.909+0000 7f8723fff700 1 --2- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872408c6d0 0x7f872408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.909+0000 7f8722ffd700 1 -- 172.21.15.16:0/158238294 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142138688 0 0) 0x7f8710001150 con 0x7f872408fef0 2019-12-05T06:15:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.909+0000 7f8722ffd700 1 -- 172.21.15.16:0/158238294 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8708002e80 con 0x7f872408fef0 2019-12-05T06:15:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.910+0000 7f8728e55700 1 -- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872408c6d0 msgr2=0x7f872408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:15:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.910+0000 7f8728e55700 1 --2- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872408c6d0 0x7f872408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.910+0000 7f8728e55700 1 -- 172.21.15.16:0/158238294 >> v1:172.21.15.16:6789/0 conn(0x7f872408fef0 legacy=0x7f872408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.910+0000 7f8728e55700 1 -- 172.21.15.16:0/158238294 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872408fbc0 con 0x7f872408f720 2019-12-05T06:15:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.910+0000 7f8723fff700 1 --2- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872408c6d0 0x7f872408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.910+0000 7f8728e55700 1 --2- 172.21.15.16:0/158238294 >> v2:172.21.15.76:3300/0 conn(0x7f872408f720 0x7f872408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f871c00b300 tx=0x7f871c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.910+0000 7f8728e55700 1 --2- 172.21.15.16:0/158238294 >> v2:172.21.15.76:3300/0 conn(0x7f872408f720 0x7f872408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f871c00b300 tx=0x7f871c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.910+0000 7f8728e55700 1 --2- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f872408f720 0x7f872408d000 secure :-1 s=READY pgs=324 cs=0 l=1 rx=0x7f871c00b300 tx=0x7f871c00bb10).ready entity=mon.1 client_cookie=72a31f099c0a9314 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.910+0000 7f8728e55700 1 --2- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f872408f720 0x7f872408d000 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7f871c00b300 tx=0x7f871c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.910+0000 7f8722ffd700 1 -- 172.21.15.16:0/158238294 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f871c000f40 con 0x7f872408f720 2019-12-05T06:15:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.911+0000 7f8728e55700 1 --2- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f872408f720 0x7f872408d000 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7f871c00b300 tx=0x7f871c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.911+0000 7f8728e55700 1 --2- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f872408f720 0x7f872408d000 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7f871c00b300 tx=0x7f871c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.911+0000 7f8722ffd700 1 -- 172.21.15.16:0/158238294 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c0075c0 con 0x7f872408f720 2019-12-05T06:15:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.911+0000 7f8722ffd700 1 -- 172.21.15.16:0/158238294 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f871c000f70 con 0x7f872408f720 2019-12-05T06:15:45.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.911+0000 7f872a0b7700 1 -- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f872408f720 msgr2=0x7f872408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:45.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.911+0000 7f872a0b7700 1 --2- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f872408f720 0x7f872408d000 secure :-1 s=READY pgs=324 cs=0 l=1 rx=0x7f871c00b300 tx=0x7f871c00bb10).stop 2019-12-05T06:15:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.913+0000 7f872a0b7700 1 -- 172.21.15.16:0/158238294 shutdown_connections 2019-12-05T06:15:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.913+0000 7f872a0b7700 1 --2- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f872408f720 0x7f872408d000 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.913+0000 7f872a0b7700 1 --2- 172.21.15.16:0/158238294 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872408c6d0 0x7f872408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.913+0000 7f872a0b7700 1 -- 172.21.15.16:0/158238294 shutdown_connections 2019-12-05T06:15:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.913+0000 7f872a0b7700 1 -- 172.21.15.16:0/158238294 wait complete. 2019-12-05T06:15:45.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.913+0000 7f872a0b7700 1 -- 172.21.15.16:0/158238294 >> 172.21.15.16:0/158238294 conn(0x7f8724061940 msgr2=0x7f8724096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.918+0000 7f872a0b7700 1 Processor -- start 2019-12-05T06:15:45.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.918+0000 7f872a0b7700 1 -- start start 2019-12-05T06:15:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.920+0000 7f872a0b7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872409e7c0 0x7f872409dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.920+0000 7f872a0b7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8724156d30 0x7f8724159130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.920+0000 7f872a0b7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8724130410 con 0x7f872412ca70 2019-12-05T06:15:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.920+0000 7f872a0b7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8724131c20 con 0x7f8724156d30 2019-12-05T06:15:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.920+0000 7f872a0b7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87241311f0 con 0x7f872409e7c0 2019-12-05T06:15:45.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.921+0000 7f87237fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872409e7c0 0x7f872409dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:45.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.921+0000 7f8728e55700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8724156d30 0x7f8724159130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:45.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.921+0000 7f87237fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872409e7c0 0x7f872409dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:54112/0 (socket says 172.21.15.16:54112) 2019-12-05T06:15:45.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.921+0000 7f8723fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f872412ca70 0x7f872409d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:56590/0 (socket says 172.21.15.16:56590) 2019-12-05T06:15:45.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.921+0000 7f8728e55700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8724156d30 0x7f8724159130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:36214/0 (socket says 172.21.15.16:36214) 2019-12-05T06:15:45.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.921+0000 7f87237fe700 1 -- 172.21.15.16:0/3048400848 learned_addr learned my addr 172.21.15.16:0/3048400848 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:15:45.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.921+0000 7f8728e55700 1 -- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872409e7c0 msgr2=0x7f872409dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:45.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.921+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872409e7c0 0x7f872409dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:45.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.921+0000 7f8720ff9700 1 -- 172.21.15.16:0/3048400848 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655995491 0 0) 0x7f871400db80 con 0x7f872412ca70 2019-12-05T06:15:45.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8728e55700 1 -- 172.21.15.16:0/3048400848 >> v1:172.21.15.76:6789/0 conn(0x7f872412ca70 legacy=0x7f872409d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:45.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8728e55700 1 -- 172.21.15.16:0/3048400848 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87241322b0 con 0x7f8724156d30 2019-12-05T06:15:45.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f87237fe700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872409e7c0 0x7f872409dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:45.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> v2:172.21.15.76:3300/0 conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> v2:172.21.15.76:3300/0 conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=READY pgs=326 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:45.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=THROTTLE_DONE pgs=326 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8720ff9700 1 -- 172.21.15.16:0/3048400848 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f871c00b360 con 0x7f8724156d30 2019-12-05T06:15:45.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=THROTTLE_DONE pgs=326 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=THROTTLE_DONE pgs=326 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8720ff9700 1 -- 172.21.15.16:0/3048400848 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f871c009b20 con 0x7f8724156d30 2019-12-05T06:15:45.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f872a0b7700 1 -- 172.21.15.16:0/3048400848 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87241322b0 con 0x7f8724156d30 2019-12-05T06:15:45.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f8720ff9700 1 -- 172.21.15.16:0/3048400848 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f871c00a1f0 con 0x7f8724156d30 2019-12-05T06:15:45.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.922+0000 7f872a0b7700 1 -- 172.21.15.16:0/3048400848 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87241322b0 con 0x7f8724156d30 2019-12-05T06:15:45.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.923+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=THROTTLE_DONE pgs=326 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.924+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=THROTTLE_DONE pgs=326 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.924+0000 7f8720ff9700 1 -- 172.21.15.16:0/3048400848 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f871c009d10 con 0x7f8724156d30 2019-12-05T06:15:45.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.925+0000 7f8720ff9700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8700023630 0x7f8700025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:45.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.925+0000 7f8720ff9700 1 -- 172.21.15.16:0/3048400848 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6419+0+0 (secure 0 0 0) 0x7f871c0460f0 con 0x7f8724156d30 2019-12-05T06:15:45.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.925+0000 7f8723fff700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8700023630 0x7f8700025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:45.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.926+0000 7f8723fff700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8700023630 0x7f8700025ac0 crc :-1 s=READY pgs=192 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:45.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.938+0000 7f872a0b7700 1 -- 172.21.15.16:0/3048400848 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87241322b0 con 0x7f8724156d30 2019-12-05T06:15:45.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.946+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=THROTTLE_DONE pgs=326 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:45.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:45.946+0000 7f8720ff9700 1 -- 172.21.15.16:0/3048400848 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f871c00af60 con 0x7f8724156d30 2019-12-05T06:15:46.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.206+0000 7f872a0b7700 1 -- 172.21.15.16:0/3048400848 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87241322b0 con 0x7f8724156d30 2019-12-05T06:15:46.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.207+0000 7f8728e55700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=THROTTLE_DONE pgs=326 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:46.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.207+0000 7f8720ff9700 1 -- 172.21.15.16:0/3048400848 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f871c009d10 con 0x7f8724156d30 2019-12-05T06:15:46.203 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2019-12-05T06:15:46.203 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:46.203 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:37.070869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:46.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.211+0000 7f872a0b7700 1 -- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8700023630 msgr2=0x7f8700025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:46.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.211+0000 7f872a0b7700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8700023630 0x7f8700025ac0 crc :-1 s=READY pgs=192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:46.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.211+0000 7f872a0b7700 1 -- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 msgr2=0x7f8724159130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:46.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.211+0000 7f872a0b7700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 0x7f8724159130 secure :-1 s=READY pgs=326 cs=0 l=1 rx=0x7f871c010730 tx=0x7f871c010ad0).stop 2019-12-05T06:15:46.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.211+0000 7f872a0b7700 1 -- 172.21.15.16:0/3048400848 shutdown_connections 2019-12-05T06:15:46.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.211+0000 7f872a0b7700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8700023630 0x7f8700025ac0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.211+0000 7f872a0b7700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8724156d30 0x7f8724159130 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.211+0000 7f872a0b7700 1 --2- 172.21.15.16:0/3048400848 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f872409e7c0 0x7f872409dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.212+0000 7f872a0b7700 1 -- 172.21.15.16:0/3048400848 shutdown_connections 2019-12-05T06:15:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.212+0000 7f872a0b7700 1 -- 172.21.15.16:0/3048400848 wait complete. 2019-12-05T06:15:46.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:46.212+0000 7f872a0b7700 1 -- 172.21.15.16:0/3048400848 >> 172.21.15.16:0/3048400848 conn(0x7f87240a2850 msgr2=0x7f872406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:46.217 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:15:48.218 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.474+0000 7f57b776c700 1 Processor -- start 2019-12-05T06:15:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.474+0000 7f57b776c700 1 -- start start 2019-12-05T06:15:48.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.476+0000 7f57b776c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57b008fef0 0x7f57b008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:48.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.476+0000 7f57b776c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b00958e0 con 0x7f57b008d2c0 2019-12-05T06:15:48.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.476+0000 7f57b776c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b00629e0 con 0x7f57b008c6d0 2019-12-05T06:15:48.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.476+0000 7f57b776c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b008df90 con 0x7f57b008fef0 2019-12-05T06:15:48.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.476+0000 7f57b650a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57b008fef0 0x7f57b008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:48.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.476+0000 7f57b5508700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f57b008d2c0 0x7f57b008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35298/0 (socket says 172.21.15.16:35298) 2019-12-05T06:15:48.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.476+0000 7f57b5508700 1 -- 172.21.15.16:0/3266721831 learned_addr learned my addr 172.21.15.16:0/3266721831 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:48.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.477+0000 7f57b650a700 1 -- 172.21.15.16:0/3266721831 >> v1:172.21.15.76:6789/0 conn(0x7f57b008c6d0 legacy=0x7f57b008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:48.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.477+0000 7f57b650a700 1 -- 172.21.15.16:0/3266721831 >> v1:172.21.15.16:6789/0 conn(0x7f57b008d2c0 legacy=0x7f57b008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:48.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.477+0000 7f57b4d07700 1 -- 172.21.15.16:0/3266721831 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 996124226 0 0) 0x7f57a8001400 con 0x7f57b008c6d0 2019-12-05T06:15:48.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.477+0000 7f57b650a700 1 -- 172.21.15.16:0/3266721831 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b008e620 con 0x7f57b008fef0 2019-12-05T06:15:48.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.477+0000 7f57b650a700 1 --2- 172.21.15.16:0/3266721831 >> v2:172.21.15.16:3300/0 conn(0x7f57b008fef0 0x7f57b008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f57a000af50 tx=0x7f57a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.477+0000 7f57b650a700 1 --2- 172.21.15.16:0/3266721831 >> v2:172.21.15.16:3300/0 conn(0x7f57b008fef0 0x7f57b008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f57a000af50 tx=0x7f57a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.477+0000 7f57b650a700 1 --2- 172.21.15.16:0/3266721831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b008fef0 0x7f57b008f720 secure :-1 s=READY pgs=421 cs=0 l=1 rx=0x7f57a000af50 tx=0x7f57a000b760).ready entity=mon.0 client_cookie=1f54220d9bbf63d6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:48.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.478+0000 7f57b650a700 1 --2- 172.21.15.16:0/3266721831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b008fef0 0x7f57b008f720 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f57a000af50 tx=0x7f57a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.478+0000 7f57b650a700 1 --2- 172.21.15.16:0/3266721831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b008fef0 0x7f57b008f720 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f57a000af50 tx=0x7f57a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.478+0000 7f57b4d07700 1 -- 172.21.15.16:0/3266721831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57a0009ba0 con 0x7f57b008fef0 2019-12-05T06:15:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.478+0000 7f57b650a700 1 --2- 172.21.15.16:0/3266721831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b008fef0 0x7f57b008f720 secure :-1 s=THROTTLE_DONE pgs=421 cs=0 l=1 rx=0x7f57a000af50 tx=0x7f57a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.478+0000 7f57b4d07700 1 -- 172.21.15.16:0/3266721831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a0000a70 con 0x7f57b008fef0 2019-12-05T06:15:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.478+0000 7f57b4d07700 1 -- 172.21.15.16:0/3266721831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57a000a180 con 0x7f57b008fef0 2019-12-05T06:15:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.478+0000 7f57b776c700 1 -- 172.21.15.16:0/3266721831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b008fef0 msgr2=0x7f57b008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:48.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.478+0000 7f57b776c700 1 --2- 172.21.15.16:0/3266721831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b008fef0 0x7f57b008f720 secure :-1 s=READY pgs=421 cs=0 l=1 rx=0x7f57a000af50 tx=0x7f57a000b760).stop 2019-12-05T06:15:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.480+0000 7f57b776c700 1 -- 172.21.15.16:0/3266721831 shutdown_connections 2019-12-05T06:15:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.480+0000 7f57b776c700 1 --2- 172.21.15.16:0/3266721831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57b008fef0 0x7f57b008f720 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.480+0000 7f57b776c700 1 -- 172.21.15.16:0/3266721831 shutdown_connections 2019-12-05T06:15:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.480+0000 7f57b776c700 1 -- 172.21.15.16:0/3266721831 wait complete. 2019-12-05T06:15:48.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.480+0000 7f57b776c700 1 -- 172.21.15.16:0/3266721831 >> 172.21.15.16:0/3266721831 conn(0x7f57b0061940 msgr2=0x7f57b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:48.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.485+0000 7f57b776c700 1 Processor -- start 2019-12-05T06:15:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.485+0000 7f57b776c700 1 -- start start 2019-12-05T06:15:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.486+0000 7f57b776c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.486+0000 7f57b776c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f57b0098b00 0x7f57b01592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.487+0000 7f57b776c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57b0089fb0 con 0x7f57b00992d0 2019-12-05T06:15:48.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.487+0000 7f57b776c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57b008ad00 con 0x7f57b0098b00 2019-12-05T06:15:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.487+0000 7f57b776c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57b0096c10 con 0x7f57b014f5b0 2019-12-05T06:15:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.487+0000 7f57b5508700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f57b00992d0 0x7f57b0098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35304/0 (socket says 172.21.15.16:35304) 2019-12-05T06:15:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.487+0000 7f57b5508700 1 -- 172.21.15.16:0/2936657940 learned_addr learned my addr 172.21.15.16:0/2936657940 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.487+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.487+0000 7f57b650a700 1 --2- 172.21.15.16:0/2936657940 >> v2:172.21.15.76:3300/0 conn(0x7f57b0098b00 0x7f57b01592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.487+0000 7f57ae7fc700 1 -- 172.21.15.16:0/2936657940 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4159810073 0 0) 0x7f57a4001150 con 0x7f57b00992d0 2019-12-05T06:15:48.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.487+0000 7f57ae7fc700 1 -- 172.21.15.16:0/2936657940 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5790002e80 con 0x7f57b00992d0 2019-12-05T06:15:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.488+0000 7f57b5d09700 1 -- 172.21.15.16:0/2936657940 >> v2:172.21.15.76:3300/0 conn(0x7f57b0098b00 msgr2=0x7f57b01592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.488+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> v2:172.21.15.76:3300/0 conn(0x7f57b0098b00 0x7f57b01592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.488+0000 7f57b5d09700 1 -- 172.21.15.16:0/2936657940 >> v1:172.21.15.16:6789/0 conn(0x7f57b00992d0 legacy=0x7f57b0098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.488+0000 7f57ae7fc700 1 -- 172.21.15.16:0/2936657940 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4154933307 0 0) 0x7f57a4001300 con 0x7f57b00992d0 2019-12-05T06:15:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.488+0000 7f57b5d09700 1 -- 172.21.15.16:0/2936657940 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57b0097a00 con 0x7f57b014f5b0 2019-12-05T06:15:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.488+0000 7f57b650a700 1 --2- 172.21.15.16:0/2936657940 >> v2:172.21.15.76:3300/0 conn(0x7f57b0098b00 0x7f57b01592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:48.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.488+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.488+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.488+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:48.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.489+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.489+0000 7f57ae7fc700 1 -- 172.21.15.16:0/2936657940 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57a800f5b0 con 0x7f57b014f5b0 2019-12-05T06:15:48.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.489+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.489+0000 7f57ae7fc700 1 -- 172.21.15.16:0/2936657940 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a8000df0 con 0x7f57b014f5b0 2019-12-05T06:15:48.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.489+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.489+0000 7f57b776c700 1 -- 172.21.15.16:0/2936657940 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57b0097a00 con 0x7f57b014f5b0 2019-12-05T06:15:48.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.489+0000 7f57ae7fc700 1 -- 172.21.15.16:0/2936657940 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57a800f6f0 con 0x7f57b014f5b0 2019-12-05T06:15:48.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.489+0000 7f57b776c700 1 -- 172.21.15.16:0/2936657940 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57b0097a00 con 0x7f57b014f5b0 2019-12-05T06:15:48.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.490+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.491+0000 7f57ae7fc700 1 -- 172.21.15.16:0/2936657940 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f57a800f730 con 0x7f57b014f5b0 2019-12-05T06:15:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.491+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.491+0000 7f57ae7fc700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5790023680 0x7f5790025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.491+0000 7f57ae7fc700 1 -- 172.21.15.16:0/2936657940 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6419+0+0 (secure 0 0 0) 0x7f57a80460d0 con 0x7f57b014f5b0 2019-12-05T06:15:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.491+0000 7f57b5508700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5790023680 0x7f5790025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.492+0000 7f57b5508700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5790023680 0x7f5790025af0 crc :-1 s=READY pgs=193 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:48.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.506+0000 7f57b776c700 1 -- 172.21.15.16:0/2936657940 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57b0097a00 con 0x7f57b014f5b0 2019-12-05T06:15:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.516+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.516+0000 7f57ae7fc700 1 -- 172.21.15.16:0/2936657940 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f57a800f3c0 con 0x7f57b014f5b0 2019-12-05T06:15:48.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.764+0000 7f57b776c700 1 -- 172.21.15.16:0/2936657940 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57b0097a00 con 0x7f57b014f5b0 2019-12-05T06:15:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.765+0000 7f57b5d09700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=THROTTLE_DONE pgs=113 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:48.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.765+0000 7f57ae7fc700 1 -- 172.21.15.16:0/2936657940 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f57a800f690 con 0x7f57b014f5b0 2019-12-05T06:15:48.761 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2019-12-05T06:15:48.761 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:48.761 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:37.070869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:48.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.771+0000 7f57b776c700 1 -- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5790023680 msgr2=0x7f5790025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:48.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.771+0000 7f57b776c700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5790023680 0x7f5790025af0 crc :-1 s=READY pgs=193 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:48.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.771+0000 7f57b776c700 1 -- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 msgr2=0x7f57b014f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:48.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.771+0000 7f57b776c700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 secure :-1 s=READY pgs=113 cs=0 l=1 rx=0x7f57a8010720 tx=0x7f57a8010ac0).stop 2019-12-05T06:15:48.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.771+0000 7f57b776c700 1 -- 172.21.15.16:0/2936657940 shutdown_connections 2019-12-05T06:15:48.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.771+0000 7f57b776c700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5790023680 0x7f5790025af0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:48.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.771+0000 7f57b776c700 1 --2- 172.21.15.16:0/2936657940 >> v2:172.21.15.76:3300/0 conn(0x7f57b0098b00 0x7f57b01592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:48.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.771+0000 7f57b776c700 1 --2- 172.21.15.16:0/2936657940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57b014f5b0 0x7f57b014f9a0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:48.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.771+0000 7f57b776c700 1 -- 172.21.15.16:0/2936657940 shutdown_connections 2019-12-05T06:15:48.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.771+0000 7f57b776c700 1 -- 172.21.15.16:0/2936657940 wait complete. 2019-12-05T06:15:48.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:48.772+0000 7f57b776c700 1 -- 172.21.15.16:0/2936657940 >> 172.21.15.16:0/2936657940 conn(0x7f57b00714f0 msgr2=0x7f57b009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:48.776 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:15:50.777 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.014+0000 7f03ae89e700 1 Processor -- start 2019-12-05T06:15:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.014+0000 7f03ae89e700 1 -- start start 2019-12-05T06:15:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.015+0000 7f03ae89e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 0x7f03a808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.015+0000 7f03ae89e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f03a808fef0 0x7f03a808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.015+0000 7f03ae89e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a80958e0 con 0x7f03a808c6d0 2019-12-05T06:15:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.015+0000 7f03ae89e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a808df90 con 0x7f03a808fef0 2019-12-05T06:15:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.015+0000 7f03ae89e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a80a3f70 con 0x7f03a808d2c0 2019-12-05T06:15:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03ace3b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f03a808c6d0 0x7f03a808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35312/0 (socket says 172.21.15.16:35312) 2019-12-05T06:15:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03ace3b700 1 -- 172.21.15.16:0/875508060 learned_addr learned my addr 172.21.15.16:0/875508060 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03ad63c700 1 --2- 172.21.15.16:0/875508060 >> v2:172.21.15.16:3300/0 conn(0x7f03a808fef0 0x7f03a808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03a7fff700 1 --2- 172.21.15.16:0/875508060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 0x7f03a808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03a77fe700 1 -- 172.21.15.16:0/875508060 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77635099 0 0) 0x7f0390001150 con 0x7f03a808c6d0 2019-12-05T06:15:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03a77fe700 1 -- 172.21.15.16:0/875508060 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0394002e80 con 0x7f03a808c6d0 2019-12-05T06:15:51.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03a7fff700 1 -- 172.21.15.16:0/875508060 >> v2:172.21.15.16:3300/0 conn(0x7f03a808fef0 msgr2=0x7f03a808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:15:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03a7fff700 1 --2- 172.21.15.16:0/875508060 >> v2:172.21.15.16:3300/0 conn(0x7f03a808fef0 0x7f03a808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03a7fff700 1 -- 172.21.15.16:0/875508060 >> v1:172.21.15.16:6789/0 conn(0x7f03a808c6d0 legacy=0x7f03a808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03a7fff700 1 -- 172.21.15.16:0/875508060 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a808e620 con 0x7f03a808d2c0 2019-12-05T06:15:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.016+0000 7f03a7fff700 1 --2- 172.21.15.16:0/875508060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 0x7f03a808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f039800af50 tx=0x7f039800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.017+0000 7f03a7fff700 1 --2- 172.21.15.16:0/875508060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 0x7f03a808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f039800af50 tx=0x7f039800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.017+0000 7f03a7fff700 1 --2- 172.21.15.16:0/875508060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 0x7f03a808ef20 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7f039800af50 tx=0x7f039800b250).ready entity=mon.2 client_cookie=7f21cac88a8b43c1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:51.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.017+0000 7f03a7fff700 1 --2- 172.21.15.16:0/875508060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 0x7f03a808ef20 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f039800af50 tx=0x7f039800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.017+0000 7f03a77fe700 1 -- 172.21.15.16:0/875508060 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0398007210 con 0x7f03a808d2c0 2019-12-05T06:15:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.017+0000 7f03a7fff700 1 --2- 172.21.15.16:0/875508060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 0x7f03a808ef20 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f039800af50 tx=0x7f039800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.017+0000 7f03a7fff700 1 --2- 172.21.15.16:0/875508060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 0x7f03a808ef20 secure :-1 s=THROTTLE_DONE pgs=114 cs=0 l=1 rx=0x7f039800af50 tx=0x7f039800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.017+0000 7f03a77fe700 1 -- 172.21.15.16:0/875508060 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03980099b0 con 0x7f03a808d2c0 2019-12-05T06:15:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.017+0000 7f03a77fe700 1 -- 172.21.15.16:0/875508060 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f039800a080 con 0x7f03a808d2c0 2019-12-05T06:15:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.017+0000 7f03ae89e700 1 -- 172.21.15.16:0/875508060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 msgr2=0x7f03a808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.017+0000 7f03ae89e700 1 --2- 172.21.15.16:0/875508060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 0x7f03a808ef20 secure :-1 s=READY pgs=114 cs=0 l=1 rx=0x7f039800af50 tx=0x7f039800b250).stop 2019-12-05T06:15:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.019+0000 7f03ae89e700 1 -- 172.21.15.16:0/875508060 shutdown_connections 2019-12-05T06:15:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.019+0000 7f03ae89e700 1 --2- 172.21.15.16:0/875508060 >> v2:172.21.15.16:3300/0 conn(0x7f03a808fef0 0x7f03a808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.019+0000 7f03ae89e700 1 --2- 172.21.15.16:0/875508060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a808d2c0 0x7f03a808ef20 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:51.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.019+0000 7f03ae89e700 1 -- 172.21.15.16:0/875508060 shutdown_connections 2019-12-05T06:15:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.019+0000 7f03ae89e700 1 -- 172.21.15.16:0/875508060 wait complete. 2019-12-05T06:15:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.019+0000 7f03ae89e700 1 -- 172.21.15.16:0/875508060 >> 172.21.15.16:0/875508060 conn(0x7f03a8061940 msgr2=0x7f03a8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.022+0000 7f03ae89e700 1 Processor -- start 2019-12-05T06:15:51.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.023+0000 7f03ae89e700 1 -- start start 2019-12-05T06:15:51.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.024+0000 7f03ae89e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:51.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.024+0000 7f03ae89e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f03a812b640 0x7f03a812abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:51.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.024+0000 7f03ae89e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a8131bf0 con 0x7f03a812b0f0 2019-12-05T06:15:51.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.024+0000 7f03ae89e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a8128cb0 con 0x7f03a812b640 2019-12-05T06:15:51.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.024+0000 7f03ae89e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a8128f80 con 0x7f03a81302f0 2019-12-05T06:15:51.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.024+0000 7f03ad63c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f03a812b0f0 0x7f03a81306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35320/0 (socket says 172.21.15.16:35320) 2019-12-05T06:15:51.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.024+0000 7f03ace3b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:51.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.024+0000 7f03ad63c700 1 -- 172.21.15.16:0/391215638 learned_addr learned my addr 172.21.15.16:0/391215638 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:51.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.024+0000 7f03a7fff700 1 --2- 172.21.15.16:0/391215638 >> v2:172.21.15.76:3300/0 conn(0x7f03a812b640 0x7f03a812abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:51.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03a57fa700 1 -- 172.21.15.16:0/391215638 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3585431627 0 0) 0x7f03a0000da0 con 0x7f03a812b0f0 2019-12-05T06:15:51.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03a57fa700 1 -- 172.21.15.16:0/391215638 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f038c002e80 con 0x7f03a812b0f0 2019-12-05T06:15:51.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ace3b700 1 -- 172.21.15.16:0/391215638 >> v2:172.21.15.76:3300/0 conn(0x7f03a812b640 msgr2=0x7f03a812abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:51.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> v2:172.21.15.76:3300/0 conn(0x7f03a812b640 0x7f03a812abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:51.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ace3b700 1 -- 172.21.15.16:0/391215638 >> v1:172.21.15.16:6789/0 conn(0x7f03a812b0f0 legacy=0x7f03a81306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:51.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ace3b700 1 -- 172.21.15.16:0/391215638 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a8129d70 con 0x7f03a81302f0 2019-12-05T06:15:51.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=READY pgs=115 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03a57fa700 1 -- 172.21.15.16:0/391215638 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0390000d80 con 0x7f03a81302f0 2019-12-05T06:15:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03a57fa700 1 -- 172.21.15.16:0/391215638 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0390001380 con 0x7f03a81302f0 2019-12-05T06:15:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ae89e700 1 -- 172.21.15.16:0/391215638 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03a8129d70 con 0x7f03a81302f0 2019-12-05T06:15:51.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03a57fa700 1 -- 172.21.15.16:0/391215638 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0390001a10 con 0x7f03a81302f0 2019-12-05T06:15:51.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.025+0000 7f03ae89e700 1 -- 172.21.15.16:0/391215638 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a8129d70 con 0x7f03a81302f0 2019-12-05T06:15:51.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.026+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.027+0000 7f03a57fa700 1 -- 172.21.15.16:0/391215638 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f03900018a0 con 0x7f03a81302f0 2019-12-05T06:15:51.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.027+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.027+0000 7f03a57fa700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f038c023670 0x7f038c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:51.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.027+0000 7f03a57fa700 1 -- 172.21.15.16:0/391215638 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6419+0+0 (secure 0 0 0) 0x7f0390045da0 con 0x7f03a81302f0 2019-12-05T06:15:51.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.027+0000 7f03a7fff700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f038c023670 0x7f038c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:51.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.028+0000 7f03a7fff700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f038c023670 0x7f038c025ae0 crc :-1 s=READY pgs=194 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:51.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.029+0000 7f03ae89e700 1 -- 172.21.15.16:0/391215638 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a8129d70 con 0x7f03a81302f0 2019-12-05T06:15:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.038+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.039+0000 7f03a57fa700 1 -- 172.21.15.16:0/391215638 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f03900015f0 con 0x7f03a81302f0 2019-12-05T06:15:51.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.244+0000 7f03ae89e700 1 -- 172.21.15.16:0/391215638 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f03a8129d70 con 0x7f03a81302f0 2019-12-05T06:15:51.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.246+0000 7f03ace3b700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=THROTTLE_DONE pgs=115 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:51.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.246+0000 7f03a57fa700 1 -- 172.21.15.16:0/391215638 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f0390001360 con 0x7f03a81302f0 2019-12-05T06:15:51.241 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:51.241 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":52,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5148,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":46}],"filesystems":[{"mdsmap":{"epoch":52,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:14:37.070869+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":42,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5049},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5049":{"gid":5049,"name":"a","rank":0,"incarnation":36,"state":"up:active","state_seq":30,"addr":"172.21.15.16:6833/4256914848","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4256914848},{"type":"v1","addr":"172.21.15.16:6833","nonce":4256914848}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:51.241 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 52 2019-12-05T06:15:51.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 -- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f038c023670 msgr2=0x7f038c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:51.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f038c023670 0x7f038c025ae0 crc :-1 s=READY pgs=194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:51.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 -- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 msgr2=0x7f03a812a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:51.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 secure :-1 s=READY pgs=115 cs=0 l=1 rx=0x7f0390010380 tx=0x7f03900101b0).stop 2019-12-05T06:15:51.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 -- 172.21.15.16:0/391215638 shutdown_connections 2019-12-05T06:15:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f038c023670 0x7f038c025ae0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 --2- 172.21.15.16:0/391215638 >> v2:172.21.15.76:3300/0 conn(0x7f03a812b640 0x7f03a812abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 --2- 172.21.15.16:0/391215638 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f03a81302f0 0x7f03a812a670 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 -- 172.21.15.16:0/391215638 shutdown_connections 2019-12-05T06:15:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 -- 172.21.15.16:0/391215638 wait complete. 2019-12-05T06:15:51.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:51.250+0000 7f03ae89e700 1 -- 172.21.15.16:0/391215638 >> 172.21.15.16:0/391215638 conn(0x7f03a80a2850 msgr2=0x7f03a806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:51.255 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:15:53.257 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.533+0000 7f3f4836b700 1 Processor -- start 2019-12-05T06:15:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.533+0000 7f3f4836b700 1 -- start start 2019-12-05T06:15:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.535+0000 7f3f4836b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3f4008d2c0 0x7f3f4008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.535+0000 7f3f4836b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4008fef0 0x7f3f4008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.535+0000 7f3f4836b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f400958e0 con 0x7f3f4008c6d0 2019-12-05T06:15:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.535+0000 7f3f4836b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4008df90 con 0x7f3f4008d2c0 2019-12-05T06:15:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.535+0000 7f3f4836b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f400a3f70 con 0x7f3f4008fef0 2019-12-05T06:15:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.536+0000 7f3f46908700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3f4008c6d0 0x7f3f4008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35326/0 (socket says 172.21.15.16:35326) 2019-12-05T06:15:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.536+0000 7f3f46908700 1 -- 172.21.15.16:0/4060729890 learned_addr learned my addr 172.21.15.16:0/4060729890 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.536+0000 7f3f46107700 1 --2- 172.21.15.16:0/4060729890 >> v2:172.21.15.16:3300/0 conn(0x7f3f4008d2c0 0x7f3f4008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.536+0000 7f3f47109700 1 --2- 172.21.15.16:0/4060729890 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4008fef0 0x7f3f4008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.536+0000 7f3f45906700 1 -- 172.21.15.16:0/4060729890 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2006902845 0 0) 0x7f3f380014f0 con 0x7f3f4008c6d0 2019-12-05T06:15:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.536+0000 7f3f45906700 1 -- 172.21.15.16:0/4060729890 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f28002e80 con 0x7f3f4008c6d0 2019-12-05T06:15:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f46107700 1 -- 172.21.15.16:0/4060729890 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4008fef0 msgr2=0x7f3f4008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:15:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f46107700 1 --2- 172.21.15.16:0/4060729890 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4008fef0 0x7f3f4008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f46107700 1 -- 172.21.15.16:0/4060729890 >> v1:172.21.15.16:6789/0 conn(0x7f3f4008c6d0 legacy=0x7f3f4008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:53.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f46107700 1 -- 172.21.15.16:0/4060729890 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4008e620 con 0x7f3f4008d2c0 2019-12-05T06:15:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f47109700 1 --2- 172.21.15.16:0/4060729890 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4008fef0 0x7f3f4008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f46107700 1 --2- 172.21.15.16:0/4060729890 >> v2:172.21.15.16:3300/0 conn(0x7f3f4008d2c0 0x7f3f4008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3f3400af50 tx=0x7f3f3400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f46107700 1 --2- 172.21.15.16:0/4060729890 >> v2:172.21.15.16:3300/0 conn(0x7f3f4008d2c0 0x7f3f4008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3f3400af50 tx=0x7f3f3400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f46107700 1 --2- 172.21.15.16:0/4060729890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4008d2c0 0x7f3f4008ef20 secure :-1 s=READY pgs=426 cs=0 l=1 rx=0x7f3f3400af50 tx=0x7f3f3400b760).ready entity=mon.0 client_cookie=19a24b95f597f74e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f46107700 1 --2- 172.21.15.16:0/4060729890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4008d2c0 0x7f3f4008ef20 secure :-1 s=THROTTLE_DONE pgs=426 cs=0 l=1 rx=0x7f3f3400af50 tx=0x7f3f3400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f45906700 1 -- 172.21.15.16:0/4060729890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f34007210 con 0x7f3f4008d2c0 2019-12-05T06:15:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.537+0000 7f3f46107700 1 --2- 172.21.15.16:0/4060729890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4008d2c0 0x7f3f4008ef20 secure :-1 s=THROTTLE_DONE pgs=426 cs=0 l=1 rx=0x7f3f3400af50 tx=0x7f3f3400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.538+0000 7f3f46107700 1 --2- 172.21.15.16:0/4060729890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4008d2c0 0x7f3f4008ef20 secure :-1 s=THROTTLE_DONE pgs=426 cs=0 l=1 rx=0x7f3f3400af50 tx=0x7f3f3400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.538+0000 7f3f45906700 1 -- 172.21.15.16:0/4060729890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f340099b0 con 0x7f3f4008d2c0 2019-12-05T06:15:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.538+0000 7f3f45906700 1 -- 172.21.15.16:0/4060729890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f3400a080 con 0x7f3f4008d2c0 2019-12-05T06:15:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.538+0000 7f3f4836b700 1 -- 172.21.15.16:0/4060729890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4008d2c0 msgr2=0x7f3f4008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.538+0000 7f3f4836b700 1 --2- 172.21.15.16:0/4060729890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4008d2c0 0x7f3f4008ef20 secure :-1 s=READY pgs=426 cs=0 l=1 rx=0x7f3f3400af50 tx=0x7f3f3400b760).stop 2019-12-05T06:15:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.540+0000 7f3f4836b700 1 -- 172.21.15.16:0/4060729890 shutdown_connections 2019-12-05T06:15:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.540+0000 7f3f4836b700 1 --2- 172.21.15.16:0/4060729890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3f4008d2c0 0x7f3f4008ef20 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.540+0000 7f3f4836b700 1 --2- 172.21.15.16:0/4060729890 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4008fef0 0x7f3f4008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.540+0000 7f3f4836b700 1 -- 172.21.15.16:0/4060729890 shutdown_connections 2019-12-05T06:15:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.540+0000 7f3f4836b700 1 -- 172.21.15.16:0/4060729890 wait complete. 2019-12-05T06:15:53.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.540+0000 7f3f4836b700 1 -- 172.21.15.16:0/4060729890 >> 172.21.15.16:0/4060729890 conn(0x7f3f40061940 msgr2=0x7f3f40096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.545+0000 7f3f4836b700 1 Processor -- start 2019-12-05T06:15:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.545+0000 7f3f4836b700 1 -- start start 2019-12-05T06:15:53.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.547+0000 7f3f4836b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.547+0000 7f3f4836b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3f40132d00 0x7f3f40131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.547+0000 7f3f4836b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f40098da0 con 0x7f3f40132270 2019-12-05T06:15:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.547+0000 7f3f4836b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f4009bb50 con 0x7f3f40132d00 2019-12-05T06:15:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.547+0000 7f3f4836b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4009a030 con 0x7f3f4014d400 2019-12-05T06:15:53.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.547+0000 7f3f46908700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:53.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.547+0000 7f3f46908700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:54154/0 (socket says 172.21.15.16:54154) 2019-12-05T06:15:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.547+0000 7f3f46908700 1 -- 172.21.15.16:0/3941940870 learned_addr learned my addr 172.21.15.16:0/3941940870 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:15:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.547+0000 7f3f46107700 1 --2- 172.21.15.16:0/3941940870 >> v2:172.21.15.76:3300/0 conn(0x7f3f40132d00 0x7f3f40131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46908700 1 -- 172.21.15.16:0/3941940870 >> v2:172.21.15.76:3300/0 conn(0x7f3f40132d00 msgr2=0x7f3f40131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> v2:172.21.15.76:3300/0 conn(0x7f3f40132d00 0x7f3f40131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46908700 1 -- 172.21.15.16:0/3941940870 >> v1:172.21.15.76:6789/0 conn(0x7f3f40132270 legacy=0x7f3f4006dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46908700 1 -- 172.21.15.16:0/3941940870 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f4009b0f0 con 0x7f3f4014d400 2019-12-05T06:15:53.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46107700 1 --2- 172.21.15.16:0/3941940870 >> v2:172.21.15.76:3300/0 conn(0x7f3f40132d00 0x7f3f40131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:15:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f3f7fe700 1 -- 172.21.15.16:0/3941940870 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f38001010 con 0x7f3f4014d400 2019-12-05T06:15:53.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f3f7fe700 1 -- 172.21.15.16:0/3941940870 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f38001440 con 0x7f3f4014d400 2019-12-05T06:15:53.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f3f7fe700 1 -- 172.21.15.16:0/3941940870 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f38001a90 con 0x7f3f4014d400 2019-12-05T06:15:53.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f4836b700 1 -- 172.21.15.16:0/3941940870 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4009b0f0 con 0x7f3f4014d400 2019-12-05T06:15:53.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.548+0000 7f3f4836b700 1 -- 172.21.15.16:0/3941940870 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f4009b0f0 con 0x7f3f4014d400 2019-12-05T06:15:53.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.549+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.549+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.549+0000 7f3f3f7fe700 1 -- 172.21.15.16:0/3941940870 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3f38001010 con 0x7f3f4014d400 2019-12-05T06:15:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.550+0000 7f3f3f7fe700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3f20023630 0x7f3f20025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.550+0000 7f3f3f7fe700 1 -- 172.21.15.16:0/3941940870 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f3f38046170 con 0x7f3f4014d400 2019-12-05T06:15:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.550+0000 7f3f46107700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3f20023630 0x7f3f20025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.550+0000 7f3f46107700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3f20023630 0x7f3f20025ac0 crc :-1 s=READY pgs=195 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:53.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.565+0000 7f3f4836b700 1 -- 172.21.15.16:0/3941940870 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4009b0f0 con 0x7f3f4014d400 2019-12-05T06:15:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.571+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.571+0000 7f3f3f7fe700 1 -- 172.21.15.16:0/3941940870 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3f38007950 con 0x7f3f4014d400 2019-12-05T06:15:53.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.822+0000 7f3f4836b700 1 -- 172.21.15.16:0/3941940870 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f4009b0f0 con 0x7f3f4014d400 2019-12-05T06:15:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.823+0000 7f3f46908700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=THROTTLE_DONE pgs=116 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.823+0000 7f3f3f7fe700 1 -- 172.21.15.16:0/3941940870 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f3f38001010 con 0x7f3f4014d400 2019-12-05T06:15:53.817 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2019-12-05T06:15:53.818 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:53.818 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:53.036617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.827+0000 7f3f4836b700 1 -- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3f20023630 msgr2=0x7f3f20025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.827+0000 7f3f4836b700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3f20023630 0x7f3f20025ac0 crc :-1 s=READY pgs=195 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:53.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.827+0000 7f3f4836b700 1 -- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 msgr2=0x7f3f4014d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.827+0000 7f3f4836b700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 secure :-1 s=READY pgs=116 cs=0 l=1 rx=0x7f3f38010760 tx=0x7f3f38010b00).stop 2019-12-05T06:15:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.827+0000 7f3f4836b700 1 -- 172.21.15.16:0/3941940870 shutdown_connections 2019-12-05T06:15:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.827+0000 7f3f4836b700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3f20023630 0x7f3f20025ac0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.827+0000 7f3f4836b700 1 --2- 172.21.15.16:0/3941940870 >> v2:172.21.15.76:3300/0 conn(0x7f3f40132d00 0x7f3f40131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.827+0000 7f3f4836b700 1 --2- 172.21.15.16:0/3941940870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3f4014d400 0x7f3f4014d7f0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.827+0000 7f3f4836b700 1 -- 172.21.15.16:0/3941940870 shutdown_connections 2019-12-05T06:15:53.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.828+0000 7f3f4836b700 1 -- 172.21.15.16:0/3941940870 wait complete. 2019-12-05T06:15:53.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:53.828+0000 7f3f4836b700 1 -- 172.21.15.16:0/3941940870 >> 172.21.15.16:0/3941940870 conn(0x7f3f400714f0 msgr2=0x7f3f4009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:53.831 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:15:53.831 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:15:53.832 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.026+0000 7fdf82935700 1 Processor -- start 2019-12-05T06:15:54.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.026+0000 7fdf82935700 1 -- start start 2019-12-05T06:15:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.027+0000 7fdf82935700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.027+0000 7fdf82935700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c08fef0 0x7fdf7c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf82935700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c0958e0 con 0x7fdf7c08ef20 2019-12-05T06:15:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf82935700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7c08df90 con 0x7fdf7c08c6d0 2019-12-05T06:15:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf82935700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c0a3f70 con 0x7fdf7c08fef0 2019-12-05T06:15:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf80ed2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:54.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf80ed2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50172/0 (socket says 172.21.15.16:50172) 2019-12-05T06:15:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf816d3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdf7c08ef20 0x7fdf7c08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35342/0 (socket says 172.21.15.16:35342) 2019-12-05T06:15:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf80ed2700 1 -- 172.21.15.16:0/2846586453 learned_addr learned my addr 172.21.15.16:0/2846586453 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:15:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/2846586453 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c08fef0 0x7fdf7c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf80ed2700 1 -- 172.21.15.16:0/2846586453 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c08fef0 msgr2=0x7fdf7c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:15:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf80ed2700 1 --2- 172.21.15.16:0/2846586453 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c08fef0 0x7fdf7c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf80ed2700 1 -- 172.21.15.16:0/2846586453 >> v1:172.21.15.16:6789/0 conn(0x7fdf7c08ef20 legacy=0x7fdf7c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:54.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf7b7fe700 1 -- 172.21.15.16:0/2846586453 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2521325876 0 0) 0x7fdf740009b0 con 0x7fdf7c08ef20 2019-12-05T06:15:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.028+0000 7fdf80ed2700 1 -- 172.21.15.16:0/2846586453 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c08d540 con 0x7fdf7c08c6d0 2019-12-05T06:15:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/2846586453 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c08fef0 0x7fdf7c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf80ed2700 1 --2- 172.21.15.16:0/2846586453 >> v2:172.21.15.16:3300/0 conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdf6c00fac0 tx=0x7fdf6c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf80ed2700 1 --2- 172.21.15.16:0/2846586453 >> v2:172.21.15.16:3300/0 conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdf6c00fac0 tx=0x7fdf6c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf80ed2700 1 --2- 172.21.15.16:0/2846586453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 secure :-1 s=READY pgs=428 cs=0 l=1 rx=0x7fdf6c00fac0 tx=0x7fdf6c00fe60).ready entity=mon.0 client_cookie=7914fa113c090201 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf80ed2700 1 --2- 172.21.15.16:0/2846586453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 secure :-1 s=THROTTLE_DONE pgs=428 cs=0 l=1 rx=0x7fdf6c00fac0 tx=0x7fdf6c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf7b7fe700 1 -- 172.21.15.16:0/2846586453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf6c000ce0 con 0x7fdf7c08c6d0 2019-12-05T06:15:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf80ed2700 1 --2- 172.21.15.16:0/2846586453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 secure :-1 s=THROTTLE_DONE pgs=428 cs=0 l=1 rx=0x7fdf6c00fac0 tx=0x7fdf6c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf80ed2700 1 --2- 172.21.15.16:0/2846586453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 secure :-1 s=THROTTLE_DONE pgs=428 cs=0 l=1 rx=0x7fdf6c00fac0 tx=0x7fdf6c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf7b7fe700 1 -- 172.21.15.16:0/2846586453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf6c001010 con 0x7fdf7c08c6d0 2019-12-05T06:15:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf7b7fe700 1 -- 172.21.15.16:0/2846586453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf6c001720 con 0x7fdf7c08c6d0 2019-12-05T06:15:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf82935700 1 -- 172.21.15.16:0/2846586453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c08c6d0 msgr2=0x7fdf7c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.029+0000 7fdf82935700 1 --2- 172.21.15.16:0/2846586453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 secure :-1 s=READY pgs=428 cs=0 l=1 rx=0x7fdf6c00fac0 tx=0x7fdf6c00fe60).stop 2019-12-05T06:15:54.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.031+0000 7fdf82935700 1 -- 172.21.15.16:0/2846586453 shutdown_connections 2019-12-05T06:15:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.031+0000 7fdf82935700 1 --2- 172.21.15.16:0/2846586453 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c08fef0 0x7fdf7c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:15:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.031+0000 7fdf82935700 1 --2- 172.21.15.16:0/2846586453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c08c6d0 0x7fdf7c08cac0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.031+0000 7fdf82935700 1 -- 172.21.15.16:0/2846586453 shutdown_connections 2019-12-05T06:15:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.031+0000 7fdf82935700 1 -- 172.21.15.16:0/2846586453 wait complete. 2019-12-05T06:15:54.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.031+0000 7fdf82935700 1 -- 172.21.15.16:0/2846586453 >> 172.21.15.16:0/2846586453 conn(0x7fdf7c061940 msgr2=0x7fdf7c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:54.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.035+0000 7fdf82935700 1 Processor -- start 2019-12-05T06:15:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.035+0000 7fdf82935700 1 -- start start 2019-12-05T06:15:54.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf82935700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdf7c0992d0 0x7fdf7c098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf82935700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c156cb0 0x7fdf7c1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf82935700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf7c089fb0 con 0x7fdf7c06df10 2019-12-05T06:15:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf82935700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf7c097370 con 0x7fdf7c0992d0 2019-12-05T06:15:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf82935700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf7c08ad00 con 0x7fdf7c156cb0 2019-12-05T06:15:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf7bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdf7c0992d0 0x7fdf7c098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:54.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf80ed2700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdf7c06df10 0x7fdf7c098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35346/0 (socket says 172.21.15.16:35346) 2019-12-05T06:15:54.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf80ed2700 1 -- 172.21.15.16:0/4097007848 learned_addr learned my addr 172.21.15.16:0/4097007848 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:54.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf816d3700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c156cb0 0x7fdf7c1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:54.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf7bfff700 1 -- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c156cb0 msgr2=0x7fdf7c1590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:54.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c156cb0 0x7fdf7c1590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:54.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf7bfff700 1 -- 172.21.15.16:0/4097007848 >> v1:172.21.15.16:6789/0 conn(0x7fdf7c06df10 legacy=0x7fdf7c098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:54.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf7bfff700 1 -- 172.21.15.16:0/4097007848 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf7c097a00 con 0x7fdf7c0992d0 2019-12-05T06:15:54.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf797fa700 1 -- 172.21.15.16:0/4097007848 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509189044 0 0) 0x7fdf6c008190 con 0x7fdf7c06df10 2019-12-05T06:15:54.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.037+0000 7fdf816d3700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c156cb0 0x7fdf7c1590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:15:54.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> v2:172.21.15.16:3300/0 conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> v2:172.21.15.16:3300/0 conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=READY pgs=430 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:54.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf797fa700 1 -- 172.21.15.16:0/4097007848 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf70000940 con 0x7fdf7c0992d0 2019-12-05T06:15:54.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf797fa700 1 -- 172.21.15.16:0/4097007848 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf70000940 con 0x7fdf7c0992d0 2019-12-05T06:15:54.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf82935700 1 -- 172.21.15.16:0/4097007848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf7c097a00 con 0x7fdf7c0992d0 2019-12-05T06:15:54.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf797fa700 1 -- 172.21.15.16:0/4097007848 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf70008900 con 0x7fdf7c0992d0 2019-12-05T06:15:54.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.038+0000 7fdf82935700 1 -- 172.21.15.16:0/4097007848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf7c097a00 con 0x7fdf7c0992d0 2019-12-05T06:15:54.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.039+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.040+0000 7fdf797fa700 1 -- 172.21.15.16:0/4097007848 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdf70008520 con 0x7fdf7c0992d0 2019-12-05T06:15:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.040+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.040+0000 7fdf797fa700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf5c02be40 0x7fdf5c02e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.040+0000 7fdf797fa700 1 -- 172.21.15.16:0/4097007848 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fdf70045d10 con 0x7fdf7c0992d0 2019-12-05T06:15:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.040+0000 7fdf80ed2700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf5c02be40 0x7fdf5c02e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.041+0000 7fdf80ed2700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf5c02be40 0x7fdf5c02e2d0 crc :-1 s=READY pgs=196 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:54.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.041+0000 7fdf82935700 1 -- 172.21.15.16:0/4097007848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf7c097a00 con 0x7fdf7c0992d0 2019-12-05T06:15:54.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.046+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.046+0000 7fdf797fa700 1 -- 172.21.15.16:0/4097007848 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdf70005180 con 0x7fdf7c0992d0 2019-12-05T06:15:54.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.272+0000 7fdf82935700 1 -- 172.21.15.16:0/4097007848 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf7c097a00 con 0x7fdf7c0992d0 2019-12-05T06:15:54.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.272+0000 7fdf7bfff700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=THROTTLE_DONE pgs=430 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:54.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.272+0000 7fdf797fa700 1 -- 172.21.15.16:0/4097007848 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fdf70008520 con 0x7fdf7c0992d0 2019-12-05T06:15:54.266 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 53 2019-12-05T06:15:54.267 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:54.267 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":53,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:53.036617+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 -- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf5c02be40 msgr2=0x7fdf5c02e2d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf5c02be40 0x7fdf5c02e2d0 crc :-1 s=READY pgs=196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:54.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 -- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 msgr2=0x7fdf7c098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 0x7fdf7c098b00 secure :-1 s=READY pgs=430 cs=0 l=1 rx=0x7fdf70010390 tx=0x7fdf70010730).stop 2019-12-05T06:15:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 -- 172.21.15.16:0/4097007848 shutdown_connections 2019-12-05T06:15:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf5c02be40 0x7fdf5c02e2d0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdf7c0992d0 0x7fdf7c098b00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 --2- 172.21.15.16:0/4097007848 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf7c156cb0 0x7fdf7c1590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 -- 172.21.15.16:0/4097007848 shutdown_connections 2019-12-05T06:15:54.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 -- 172.21.15.16:0/4097007848 wait complete. 2019-12-05T06:15:54.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:54.275+0000 7fdf82935700 1 -- 172.21.15.16:0/4097007848 >> 172.21.15.16:0/4097007848 conn(0x7fdf7c0714f0 msgr2=0x7fdf7c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:54.281 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 53, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 53, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5148": { "addr": "172.21.15.76:6835/3345019712", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 3345019712, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 3345019712, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5148, "incarnation": 53, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 54, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:15:53.036617+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5148 } } } ], "standbys": [] } 2019-12-05T06:15:56.282 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:56.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.549+0000 7f363af10700 1 Processor -- start 2019-12-05T06:15:56.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.549+0000 7f363af10700 1 -- start start 2019-12-05T06:15:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.551+0000 7f363af10700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f363408c6d0 0x7f363408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.551+0000 7f363af10700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363408fb80 con 0x7f363408fef0 2019-12-05T06:15:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.551+0000 7f363af10700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36340958e0 con 0x7f363408f720 2019-12-05T06:15:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.551+0000 7f363af10700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f363408df90 con 0x7f363408c6d0 2019-12-05T06:15:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.551+0000 7f36394ad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f363408c6d0 0x7f363408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:56.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.551+0000 7f36394ad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f363408c6d0 0x7f363408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50188/0 (socket says 172.21.15.16:50188) 2019-12-05T06:15:56.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.551+0000 7f36394ad700 1 -- 172.21.15.16:0/1360615102 learned_addr learned my addr 172.21.15.16:0/1360615102 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:15:56.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.551+0000 7f3638cac700 1 --1- 172.21.15.16:0/1360615102 >> v1:172.21.15.16:6789/0 conn(0x7f363408fef0 0x7f363408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35360/0 (socket says 172.21.15.16:35360) 2019-12-05T06:15:56.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.552+0000 7f3633fff700 1 -- 172.21.15.16:0/1360615102 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178167269 0 0) 0x7f36240009b0 con 0x7f363408f720 2019-12-05T06:15:56.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.552+0000 7f36394ad700 1 -- 172.21.15.16:0/1360615102 >> v1:172.21.15.76:6789/0 conn(0x7f363408f720 legacy=0x7f36341533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:56.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.552+0000 7f36394ad700 1 -- 172.21.15.16:0/1360615102 >> v1:172.21.15.16:6789/0 conn(0x7f363408fef0 legacy=0x7f363408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:56.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.552+0000 7f36394ad700 1 -- 172.21.15.16:0/1360615102 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f363408e620 con 0x7f363408c6d0 2019-12-05T06:15:56.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.552+0000 7f36394ad700 1 --2- 172.21.15.16:0/1360615102 >> v2:172.21.15.16:3300/0 conn(0x7f363408c6d0 0x7f363408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f362c00fac0 tx=0x7f362c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.552+0000 7f36394ad700 1 --2- 172.21.15.16:0/1360615102 >> v2:172.21.15.16:3300/0 conn(0x7f363408c6d0 0x7f363408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f362c00fac0 tx=0x7f362c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.552+0000 7f36394ad700 1 --2- 172.21.15.16:0/1360615102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f363408c6d0 0x7f363408cac0 secure :-1 s=READY pgs=432 cs=0 l=1 rx=0x7f362c00fac0 tx=0x7f362c00fe60).ready entity=mon.0 client_cookie=7388d145e8a50efd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.553+0000 7f36394ad700 1 --2- 172.21.15.16:0/1360615102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f363408c6d0 0x7f363408cac0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f362c00fac0 tx=0x7f362c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.553+0000 7f36394ad700 1 --2- 172.21.15.16:0/1360615102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f363408c6d0 0x7f363408cac0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f362c00fac0 tx=0x7f362c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.553+0000 7f3633fff700 1 -- 172.21.15.16:0/1360615102 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f362c000ce0 con 0x7f363408c6d0 2019-12-05T06:15:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.553+0000 7f36394ad700 1 --2- 172.21.15.16:0/1360615102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f363408c6d0 0x7f363408cac0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f362c00fac0 tx=0x7f362c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.553+0000 7f3633fff700 1 -- 172.21.15.16:0/1360615102 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f362c001010 con 0x7f363408c6d0 2019-12-05T06:15:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.553+0000 7f3633fff700 1 -- 172.21.15.16:0/1360615102 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f362c0016e0 con 0x7f363408c6d0 2019-12-05T06:15:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.553+0000 7f363af10700 1 -- 172.21.15.16:0/1360615102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f363408c6d0 msgr2=0x7f363408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.553+0000 7f363af10700 1 --2- 172.21.15.16:0/1360615102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f363408c6d0 0x7f363408cac0 secure :-1 s=READY pgs=432 cs=0 l=1 rx=0x7f362c00fac0 tx=0x7f362c00fe60).stop 2019-12-05T06:15:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.555+0000 7f363af10700 1 -- 172.21.15.16:0/1360615102 shutdown_connections 2019-12-05T06:15:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.555+0000 7f363af10700 1 --2- 172.21.15.16:0/1360615102 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f363408c6d0 0x7f363408cac0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.555+0000 7f363af10700 1 -- 172.21.15.16:0/1360615102 shutdown_connections 2019-12-05T06:15:56.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.555+0000 7f363af10700 1 -- 172.21.15.16:0/1360615102 wait complete. 2019-12-05T06:15:56.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.555+0000 7f363af10700 1 -- 172.21.15.16:0/1360615102 >> 172.21.15.16:0/1360615102 conn(0x7f3634061940 msgr2=0x7f3634096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.560+0000 7f363af10700 1 Processor -- start 2019-12-05T06:15:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.560+0000 7f363af10700 1 -- start start 2019-12-05T06:15:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.562+0000 7f363af10700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f363409ae80 0x7f3634133220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.562+0000 7f363af10700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363409ab10 con 0x7f3634132630 2019-12-05T06:15:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.562+0000 7f363af10700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f363406e7c0 con 0x7f3634099eb0 2019-12-05T06:15:56.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.562+0000 7f363af10700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3634098f20 con 0x7f363409ae80 2019-12-05T06:15:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.562+0000 7f36394ad700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3634132630 0x7f3634132a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35362/0 (socket says 172.21.15.16:35362) 2019-12-05T06:15:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.562+0000 7f36394ad700 1 -- 172.21.15.16:0/3666331865 learned_addr learned my addr 172.21.15.16:0/3666331865 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.562+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> v2:172.21.15.76:3300/0 conn(0x7f363409ae80 0x7f3634133220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:56.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.562+0000 7f3631ffb700 1 -- 172.21.15.16:0/3666331865 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1007172762 0 0) 0x7f362c007b60 con 0x7f3634132630 2019-12-05T06:15:56.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.563+0000 7f3631ffb700 1 -- 172.21.15.16:0/3666331865 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3614002e80 con 0x7f3634132630 2019-12-05T06:15:56.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.563+0000 7f3631ffb700 1 -- 172.21.15.16:0/3666331865 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1192988170 0 0) 0x7f3624001200 con 0x7f3634099eb0 2019-12-05T06:15:56.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.563+0000 7f3638cac700 1 -- 172.21.15.16:0/3666331865 >> v1:172.21.15.76:6789/0 conn(0x7f3634099eb0 legacy=0x7f363409a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:56.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.563+0000 7f3638cac700 1 -- 172.21.15.16:0/3666331865 >> v1:172.21.15.16:6789/0 conn(0x7f3634132630 legacy=0x7f3634132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:56.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.563+0000 7f3638cac700 1 -- 172.21.15.16:0/3666331865 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36340995b0 con 0x7f363409ae80 2019-12-05T06:15:56.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.563+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> v2:172.21.15.76:3300/0 conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.563+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> v2:172.21.15.76:3300/0 conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.563+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=READY pgs=331 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.564+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=THROTTLE_DONE pgs=331 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.564+0000 7f3631ffb700 1 -- 172.21.15.16:0/3666331865 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3628000ea0 con 0x7f363409ae80 2019-12-05T06:15:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.564+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=THROTTLE_DONE pgs=331 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.564+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=THROTTLE_DONE pgs=331 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.564+0000 7f3631ffb700 1 -- 172.21.15.16:0/3666331865 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3628001220 con 0x7f363409ae80 2019-12-05T06:15:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.564+0000 7f3631ffb700 1 -- 172.21.15.16:0/3666331865 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3628004810 con 0x7f363409ae80 2019-12-05T06:15:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.564+0000 7f363af10700 1 -- 172.21.15.16:0/3666331865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36340995b0 con 0x7f363409ae80 2019-12-05T06:15:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.564+0000 7f363af10700 1 -- 172.21.15.16:0/3666331865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36340995b0 con 0x7f363409ae80 2019-12-05T06:15:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.565+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=THROTTLE_DONE pgs=331 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.566+0000 7f3631ffb700 1 -- 172.21.15.16:0/3666331865 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3628004240 con 0x7f363409ae80 2019-12-05T06:15:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.566+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=THROTTLE_DONE pgs=331 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.566+0000 7f3631ffb700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3614023680 0x7f3614025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:56.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.566+0000 7f3631ffb700 1 -- 172.21.15.16:0/3666331865 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f3628045d90 con 0x7f363409ae80 2019-12-05T06:15:56.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.566+0000 7f36394ad700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3614023680 0x7f3614025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:56.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.567+0000 7f36394ad700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3614023680 0x7f3614025af0 crc :-1 s=READY pgs=197 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:56.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.580+0000 7f363af10700 1 -- 172.21.15.16:0/3666331865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36340995b0 con 0x7f363409ae80 2019-12-05T06:15:56.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.590+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=THROTTLE_DONE pgs=331 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.590+0000 7f3631ffb700 1 -- 172.21.15.16:0/3666331865 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f36280344e0 con 0x7f363409ae80 2019-12-05T06:15:56.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.858+0000 7f363af10700 1 -- 172.21.15.16:0/3666331865 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36340995b0 con 0x7f363409ae80 2019-12-05T06:15:56.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.859+0000 7f3638cac700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=THROTTLE_DONE pgs=331 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:56.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.859+0000 7f3631ffb700 1 -- 172.21.15.16:0/3666331865 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 54 v54) v1 ==== 94+0+1775 (secure 0 0 0) 0x7f36280344e0 con 0x7f363409ae80 2019-12-05T06:15:56.855 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 54 2019-12-05T06:15:56.855 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:56.855 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":54,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":54,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:55.992878+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:reconnect","state_seq":26,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:56.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.863+0000 7f363af10700 1 -- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3614023680 msgr2=0x7f3614025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.863+0000 7f363af10700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3614023680 0x7f3614025af0 crc :-1 s=READY pgs=197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.863+0000 7f363af10700 1 -- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 msgr2=0x7f3634133220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:56.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.863+0000 7f363af10700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 0x7f3634133220 secure :-1 s=READY pgs=331 cs=0 l=1 rx=0x7f3628010370 tx=0x7f3628010710).stop 2019-12-05T06:15:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.864+0000 7f363af10700 1 -- 172.21.15.16:0/3666331865 shutdown_connections 2019-12-05T06:15:56.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.864+0000 7f363af10700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3614023680 0x7f3614025af0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:56.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.864+0000 7f363af10700 1 --2- 172.21.15.16:0/3666331865 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f363409ae80 0x7f3634133220 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:56.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.864+0000 7f363af10700 1 -- 172.21.15.16:0/3666331865 shutdown_connections 2019-12-05T06:15:56.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.864+0000 7f363af10700 1 -- 172.21.15.16:0/3666331865 wait complete. 2019-12-05T06:15:56.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:56.864+0000 7f363af10700 1 -- 172.21.15.16:0/3666331865 >> 172.21.15.16:0/3666331865 conn(0x7f36340714f0 msgr2=0x7f363409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:56.868 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:15:58.871 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:15:59.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.123+0000 7f2b5bd17700 1 Processor -- start 2019-12-05T06:15:59.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.123+0000 7f2b5bd17700 1 -- start start 2019-12-05T06:15:59.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.125+0000 7f2b5bd17700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 0x7f2b5408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:15:59.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.125+0000 7f2b5bd17700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b540958e0 con 0x7f2b5408d2c0 2019-12-05T06:15:59.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.125+0000 7f2b5bd17700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b540629e0 con 0x7f2b5408c6d0 2019-12-05T06:15:59.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.125+0000 7f2b5bd17700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5408df90 con 0x7f2b5408fef0 2019-12-05T06:15:59.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.125+0000 7f2b59ab3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2b5408d2c0 0x7f2b5408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35372/0 (socket says 172.21.15.16:35372) 2019-12-05T06:15:59.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.125+0000 7f2b59ab3700 1 -- 172.21.15.16:0/2879543966 learned_addr learned my addr 172.21.15.16:0/2879543966 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:15:59.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.126+0000 7f2b5aab5700 1 --2- 172.21.15.16:0/2879543966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 0x7f2b5408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:59.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.126+0000 7f2b592b2700 1 -- 172.21.15.16:0/2879543966 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489443860 0 0) 0x7f2b44001150 con 0x7f2b5408d2c0 2019-12-05T06:15:59.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.126+0000 7f2b592b2700 1 -- 172.21.15.16:0/2879543966 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b40002e80 con 0x7f2b5408d2c0 2019-12-05T06:15:59.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.126+0000 7f2b592b2700 1 -- 172.21.15.16:0/2879543966 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929778492 0 0) 0x7f2b50001080 con 0x7f2b5408c6d0 2019-12-05T06:15:59.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.126+0000 7f2b592b2700 1 -- 172.21.15.16:0/2879543966 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b400030b0 con 0x7f2b5408c6d0 2019-12-05T06:15:59.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.126+0000 7f2b592b2700 1 -- 172.21.15.16:0/2879543966 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 344551169 0 0) 0x7f2b44001130 con 0x7f2b5408d2c0 2019-12-05T06:15:59.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.126+0000 7f2b5aab5700 1 -- 172.21.15.16:0/2879543966 >> v1:172.21.15.76:6789/0 conn(0x7f2b5408c6d0 legacy=0x7f2b5408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:59.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.126+0000 7f2b5aab5700 1 -- 172.21.15.16:0/2879543966 >> v1:172.21.15.16:6789/0 conn(0x7f2b5408d2c0 legacy=0x7f2b5408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:59.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.126+0000 7f2b5aab5700 1 -- 172.21.15.16:0/2879543966 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5408e620 con 0x7f2b5408fef0 2019-12-05T06:15:59.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.127+0000 7f2b5aab5700 1 --2- 172.21.15.16:0/2879543966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 0x7f2b5408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2b4c00b300 tx=0x7f2b4c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.127+0000 7f2b5aab5700 1 --2- 172.21.15.16:0/2879543966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 0x7f2b5408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2b4c00b300 tx=0x7f2b4c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.127+0000 7f2b5aab5700 1 --2- 172.21.15.16:0/2879543966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 0x7f2b5408f720 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f2b4c00b300 tx=0x7f2b4c00bb10).ready entity=mon.2 client_cookie=9edd8401c6979d14 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:59.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.127+0000 7f2b5aab5700 1 --2- 172.21.15.16:0/2879543966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 0x7f2b5408f720 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f2b4c00b300 tx=0x7f2b4c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.127+0000 7f2b592b2700 1 -- 172.21.15.16:0/2879543966 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b4c0075c0 con 0x7f2b5408fef0 2019-12-05T06:15:59.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.127+0000 7f2b5aab5700 1 --2- 172.21.15.16:0/2879543966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 0x7f2b5408f720 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f2b4c00b300 tx=0x7f2b4c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.127+0000 7f2b5aab5700 1 --2- 172.21.15.16:0/2879543966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 0x7f2b5408f720 secure :-1 s=THROTTLE_DONE pgs=118 cs=0 l=1 rx=0x7f2b4c00b300 tx=0x7f2b4c00bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.127+0000 7f2b592b2700 1 -- 172.21.15.16:0/2879543966 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b4c000f30 con 0x7f2b5408fef0 2019-12-05T06:15:59.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.127+0000 7f2b592b2700 1 -- 172.21.15.16:0/2879543966 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b4c00a1e0 con 0x7f2b5408fef0 2019-12-05T06:15:59.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.128+0000 7f2b5bd17700 1 -- 172.21.15.16:0/2879543966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 msgr2=0x7f2b5408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:59.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.128+0000 7f2b5bd17700 1 --2- 172.21.15.16:0/2879543966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 0x7f2b5408f720 secure :-1 s=READY pgs=118 cs=0 l=1 rx=0x7f2b4c00b300 tx=0x7f2b4c00bb10).stop 2019-12-05T06:15:59.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.129+0000 7f2b5bd17700 1 -- 172.21.15.16:0/2879543966 shutdown_connections 2019-12-05T06:15:59.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.129+0000 7f2b5bd17700 1 --2- 172.21.15.16:0/2879543966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5408fef0 0x7f2b5408f720 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:59.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.129+0000 7f2b5bd17700 1 -- 172.21.15.16:0/2879543966 shutdown_connections 2019-12-05T06:15:59.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.129+0000 7f2b5bd17700 1 -- 172.21.15.16:0/2879543966 wait complete. 2019-12-05T06:15:59.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.129+0000 7f2b5bd17700 1 -- 172.21.15.16:0/2879543966 >> 172.21.15.16:0/2879543966 conn(0x7f2b54061940 msgr2=0x7f2b54096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:59.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.134+0000 7f2b5bd17700 1 Processor -- start 2019-12-05T06:15:59.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.135+0000 7f2b5bd17700 1 -- start start 2019-12-05T06:15:59.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.136+0000 7f2b5bd17700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5406dc50 0x7f2b5406e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:59.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.136+0000 7f2b5bd17700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2b54135450 0x7f2b5406e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:59.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.136+0000 7f2b5bd17700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b54128af0 con 0x7f2b54134480 2019-12-05T06:15:59.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.136+0000 7f2b5bd17700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b54132830 con 0x7f2b54135450 2019-12-05T06:15:59.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.136+0000 7f2b5bd17700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b54129d80 con 0x7f2b5406dc50 2019-12-05T06:15:59.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.136+0000 7f2b59ab3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2b54135450 0x7f2b5406e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.136+0000 7f2b5a2b4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5406dc50 0x7f2b5406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.136+0000 7f2b5a2b4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5406dc50 0x7f2b5406e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:54198/0 (socket says 172.21.15.16:54198) 2019-12-05T06:15:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.136+0000 7f2b59ab3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2b54135450 0x7f2b5406e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:36298/0 (socket says 172.21.15.16:36298) 2019-12-05T06:15:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.136+0000 7f2b5a2b4700 1 -- 172.21.15.16:0/3491205606 learned_addr learned my addr 172.21.15.16:0/3491205606 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:15:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.137+0000 7f2b59ab3700 1 -- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5406dc50 msgr2=0x7f2b5406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:59.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.137+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5406dc50 0x7f2b5406e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.137+0000 7f2b59ab3700 1 -- 172.21.15.16:0/3491205606 >> v1:172.21.15.76:6789/0 conn(0x7f2b54134480 legacy=0x7f2b54134900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.137+0000 7f2b59ab3700 1 -- 172.21.15.16:0/3491205606 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b54133b80 con 0x7f2b54135450 2019-12-05T06:15:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.137+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> v2:172.21.15.76:3300/0 conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.137+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> v2:172.21.15.76:3300/0 conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.137+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=READY pgs=334 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.137+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.137+0000 7f2b4affd700 1 -- 172.21.15.16:0/3491205606 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b44000c90 con 0x7f2b54135450 2019-12-05T06:15:59.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.138+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.138+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.138+0000 7f2b4affd700 1 -- 172.21.15.16:0/3491205606 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b440013e0 con 0x7f2b54135450 2019-12-05T06:15:59.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.138+0000 7f2b5bd17700 1 -- 172.21.15.16:0/3491205606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b54133b80 con 0x7f2b54135450 2019-12-05T06:15:59.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.138+0000 7f2b4affd700 1 -- 172.21.15.16:0/3491205606 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b44001b30 con 0x7f2b54135450 2019-12-05T06:15:59.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.138+0000 7f2b5bd17700 1 -- 172.21.15.16:0/3491205606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b54133b80 con 0x7f2b54135450 2019-12-05T06:15:59.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.138+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.139+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.139+0000 7f2b4affd700 1 -- 172.21.15.16:0/3491205606 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2b440010b0 con 0x7f2b54135450 2019-12-05T06:15:59.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.139+0000 7f2b4affd700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2b3c023630 0x7f2b3c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:15:59.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.139+0000 7f2b4affd700 1 -- 172.21.15.16:0/3491205606 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f2b44045d10 con 0x7f2b54135450 2019-12-05T06:15:59.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.139+0000 7f2b5a2b4700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2b3c023630 0x7f2b3c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:15:59.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.140+0000 7f2b5a2b4700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2b3c023630 0x7f2b3c025ac0 crc :-1 s=READY pgs=198 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:15:59.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.155+0000 7f2b5bd17700 1 -- 172.21.15.16:0/3491205606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b54133b80 con 0x7f2b54135450 2019-12-05T06:15:59.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.164+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.164+0000 7f2b4affd700 1 -- 172.21.15.16:0/3491205606 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2b44000ea0 con 0x7f2b54135450 2019-12-05T06:15:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.411+0000 7f2b5bd17700 1 -- 172.21.15.16:0/3491205606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b54133b80 con 0x7f2b54135450 2019-12-05T06:15:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.412+0000 7f2b59ab3700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:15:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.413+0000 7f2b4affd700 1 -- 172.21.15.16:0/3491205606 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f2b440010b0 con 0x7f2b54135450 2019-12-05T06:15:59.408 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:15:59.409 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":56,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:58.002271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":28,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:15:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 56 2019-12-05T06:15:59.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.417+0000 7f2b5bd17700 1 -- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2b3c023630 msgr2=0x7f2b3c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.417+0000 7f2b5bd17700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2b3c023630 0x7f2b3c025ac0 crc :-1 s=READY pgs=198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.417+0000 7f2b5bd17700 1 -- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 msgr2=0x7f2b5406e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:15:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.417+0000 7f2b5bd17700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 0x7f2b5406e580 secure :-1 s=READY pgs=334 cs=0 l=1 rx=0x7f2b44010380 tx=0x7f2b440101b0).stop 2019-12-05T06:15:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.417+0000 7f2b5bd17700 1 -- 172.21.15.16:0/3491205606 shutdown_connections 2019-12-05T06:15:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.417+0000 7f2b5bd17700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2b3c023630 0x7f2b3c025ac0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.417+0000 7f2b5bd17700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2b54135450 0x7f2b5406e580 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:59.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.417+0000 7f2b5bd17700 1 --2- 172.21.15.16:0/3491205606 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2b5406dc50 0x7f2b5406e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:15:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.418+0000 7f2b5bd17700 1 -- 172.21.15.16:0/3491205606 shutdown_connections 2019-12-05T06:15:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.418+0000 7f2b5bd17700 1 -- 172.21.15.16:0/3491205606 wait complete. 2019-12-05T06:15:59.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:15:59.418+0000 7f2b5bd17700 1 -- 172.21.15.16:0/3491205606 >> 172.21.15.16:0/3491205606 conn(0x7f2b540a2850 msgr2=0x7f2b5406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:15:59.423 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5049 2019-12-05T06:15:59.423 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2019-12-05T06:16:00.635 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:16:00.641 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:16:06.433 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:16:06.433 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:16:06.434 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:16:06.438 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:16:06.439 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:06.618 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:16:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.670+0000 7fde95e31700 1 Processor -- start 2019-12-05T06:16:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.670+0000 7fde95e31700 1 -- start start 2019-12-05T06:16:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.671+0000 7fde95e31700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde9008c6d0 0x7fde9008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.671+0000 7fde95e31700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fde9008f720 0x7fde9008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.671+0000 7fde95e31700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9008fb80 con 0x7fde9008fef0 2019-12-05T06:16:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.671+0000 7fde95e31700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9008df90 con 0x7fde9008c6d0 2019-12-05T06:16:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.671+0000 7fde95e31700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde900a3f70 con 0x7fde9008f720 2019-12-05T06:16:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.671+0000 7fde8ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde9008c6d0 0x7fde9008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.671+0000 7fde8ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde9008c6d0 0x7fde9008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50236/0 (socket says 172.21.15.16:50236) 2019-12-05T06:16:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.671+0000 7fde8ffff700 1 -- 172.21.15.16:0/2159339448 learned_addr learned my addr 172.21.15.16:0/2159339448 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.671+0000 7fde94bcf700 1 --2- 172.21.15.16:0/2159339448 >> v2:172.21.15.76:3300/0 conn(0x7fde9008f720 0x7fde9008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:06.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.671+0000 7fde8ffff700 1 -- 172.21.15.16:0/2159339448 >> v2:172.21.15.76:3300/0 conn(0x7fde9008f720 msgr2=0x7fde9008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8ffff700 1 --2- 172.21.15.16:0/2159339448 >> v2:172.21.15.76:3300/0 conn(0x7fde9008f720 0x7fde9008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8ffff700 1 -- 172.21.15.16:0/2159339448 >> v1:172.21.15.76:6789/0 conn(0x7fde9008fef0 legacy=0x7fde9008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8ffff700 1 -- 172.21.15.16:0/2159339448 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9008d540 con 0x7fde9008c6d0 2019-12-05T06:16:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8ffff700 1 --2- 172.21.15.16:0/2159339448 >> v2:172.21.15.16:3300/0 conn(0x7fde9008c6d0 0x7fde9008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fde8000f6c0 tx=0x7fde8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8ffff700 1 --2- 172.21.15.16:0/2159339448 >> v2:172.21.15.16:3300/0 conn(0x7fde9008c6d0 0x7fde9008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fde8000f6c0 tx=0x7fde8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8ffff700 1 --2- 172.21.15.16:0/2159339448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9008c6d0 0x7fde9008cac0 secure :-1 s=READY pgs=439 cs=0 l=1 rx=0x7fde8000f6c0 tx=0x7fde8000fa60).ready entity=mon.0 client_cookie=217b8740e3ebbfc9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:06.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8ffff700 1 --2- 172.21.15.16:0/2159339448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9008c6d0 0x7fde9008cac0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7fde8000f6c0 tx=0x7fde8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8ffff700 1 --2- 172.21.15.16:0/2159339448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9008c6d0 0x7fde9008cac0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7fde8000f6c0 tx=0x7fde8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8ffff700 1 --2- 172.21.15.16:0/2159339448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9008c6d0 0x7fde9008cac0 secure :-1 s=THROTTLE_DONE pgs=439 cs=0 l=1 rx=0x7fde8000f6c0 tx=0x7fde8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8f7fe700 1 -- 172.21.15.16:0/2159339448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde80000960 con 0x7fde9008c6d0 2019-12-05T06:16:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8f7fe700 1 -- 172.21.15.16:0/2159339448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde80000c90 con 0x7fde9008c6d0 2019-12-05T06:16:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde8f7fe700 1 -- 172.21.15.16:0/2159339448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde800013a0 con 0x7fde9008c6d0 2019-12-05T06:16:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde95e31700 1 -- 172.21.15.16:0/2159339448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9008c6d0 msgr2=0x7fde9008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.672+0000 7fde95e31700 1 --2- 172.21.15.16:0/2159339448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9008c6d0 0x7fde9008cac0 secure :-1 s=READY pgs=439 cs=0 l=1 rx=0x7fde8000f6c0 tx=0x7fde8000fa60).stop 2019-12-05T06:16:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.674+0000 7fde95e31700 1 -- 172.21.15.16:0/2159339448 shutdown_connections 2019-12-05T06:16:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.674+0000 7fde95e31700 1 --2- 172.21.15.16:0/2159339448 >> v2:172.21.15.76:3300/0 conn(0x7fde9008f720 0x7fde9008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.674+0000 7fde95e31700 1 --2- 172.21.15.16:0/2159339448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9008c6d0 0x7fde9008cac0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:06.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.674+0000 7fde95e31700 1 -- 172.21.15.16:0/2159339448 shutdown_connections 2019-12-05T06:16:06.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.674+0000 7fde95e31700 1 -- 172.21.15.16:0/2159339448 wait complete. 2019-12-05T06:16:06.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.674+0000 7fde95e31700 1 -- 172.21.15.16:0/2159339448 >> 172.21.15.16:0/2159339448 conn(0x7fde90061940 msgr2=0x7fde90096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:06.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.677+0000 7fde95e31700 1 Processor -- start 2019-12-05T06:16:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.677+0000 7fde95e31700 1 -- start start 2019-12-05T06:16:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.678+0000 7fde95e31700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde9009bbe0 0x7fde901539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.678+0000 7fde95e31700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde90084f30 0x7fde90155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:06.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.678+0000 7fde95e31700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde90084b90 con 0x7fde90096e90 2019-12-05T06:16:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.678+0000 7fde95e31700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9009a450 con 0x7fde90084f30 2019-12-05T06:16:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.678+0000 7fde95e31700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde90099a20 con 0x7fde9009bbe0 2019-12-05T06:16:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde90084f30 0x7fde90155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde90084f30 0x7fde90155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50246/0 (socket says 172.21.15.16:50246) 2019-12-05T06:16:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 -- 172.21.15.16:0/97169220 learned_addr learned my addr 172.21.15.16:0/97169220 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde87fff700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde9009bbe0 0x7fde901539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:06.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 -- 172.21.15.16:0/97169220 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde9009bbe0 msgr2=0x7fde901539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde9009bbe0 0x7fde901539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 -- 172.21.15.16:0/97169220 >> v1:172.21.15.76:6789/0 conn(0x7fde90096e90 legacy=0x7fde9009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 -- 172.21.15.16:0/97169220 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9009aae0 con 0x7fde90084f30 2019-12-05T06:16:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> v2:172.21.15.16:3300/0 conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> v2:172.21.15.16:3300/0 conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=READY pgs=440 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=THROTTLE_DONE pgs=440 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.679+0000 7fde8d7fa700 1 -- 172.21.15.16:0/97169220 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde8800f410 con 0x7fde90084f30 2019-12-05T06:16:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.680+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=THROTTLE_DONE pgs=440 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.680+0000 7fde8d7fa700 1 -- 172.21.15.16:0/97169220 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde88000df0 con 0x7fde90084f30 2019-12-05T06:16:06.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.680+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=THROTTLE_DONE pgs=440 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.680+0000 7fde95e31700 1 -- 172.21.15.16:0/97169220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9009aae0 con 0x7fde90084f30 2019-12-05T06:16:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.680+0000 7fde8d7fa700 1 -- 172.21.15.16:0/97169220 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde8800f9f0 con 0x7fde90084f30 2019-12-05T06:16:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.680+0000 7fde95e31700 1 -- 172.21.15.16:0/97169220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde9009aae0 con 0x7fde90084f30 2019-12-05T06:16:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.680+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=THROTTLE_DONE pgs=440 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.681+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=THROTTLE_DONE pgs=440 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.681+0000 7fde8d7fa700 1 -- 172.21.15.16:0/97169220 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fde8800f170 con 0x7fde90084f30 2019-12-05T06:16:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.681+0000 7fde8d7fa700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde68023630 0x7fde68025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.682+0000 7fde8d7fa700 1 -- 172.21.15.16:0/97169220 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7fde880461a0 con 0x7fde90084f30 2019-12-05T06:16:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.682+0000 7fde8ffff700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde68023630 0x7fde68025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.682+0000 7fde8ffff700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde68023630 0x7fde68025ac0 crc :-1 s=READY pgs=199 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:06.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.684+0000 7fde95e31700 1 -- 172.21.15.16:0/97169220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde9009aae0 con 0x7fde90084f30 2019-12-05T06:16:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.689+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=THROTTLE_DONE pgs=440 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.689+0000 7fde8d7fa700 1 -- 172.21.15.16:0/97169220 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fde8800ee00 con 0x7fde90084f30 2019-12-05T06:16:06.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.912+0000 7fde95e31700 1 -- 172.21.15.16:0/97169220 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde9009aae0 con 0x7fde90084f30 2019-12-05T06:16:06.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.913+0000 7fde94bcf700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=THROTTLE_DONE pgs=440 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:06.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.913+0000 7fde8d7fa700 1 -- 172.21.15.16:0/97169220 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fde8800f0b0 con 0x7fde90084f30 2019-12-05T06:16:06.907 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 57 2019-12-05T06:16:06.907 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:06.907 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:58.002271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":28,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:06.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 -- 172.21.15.16:0/97169220 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde68023630 msgr2=0x7fde68025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde68023630 0x7fde68025ac0 crc :-1 s=READY pgs=199 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 -- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 msgr2=0x7fde90155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 0x7fde90155f30 secure :-1 s=READY pgs=440 cs=0 l=1 rx=0x7fde880107d0 tx=0x7fde88010600).stop 2019-12-05T06:16:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 -- 172.21.15.16:0/97169220 shutdown_connections 2019-12-05T06:16:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde68023630 0x7fde68025ac0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde90084f30 0x7fde90155f30 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:06.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 --2- 172.21.15.16:0/97169220 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde9009bbe0 0x7fde901539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 -- 172.21.15.16:0/97169220 shutdown_connections 2019-12-05T06:16:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 -- 172.21.15.16:0/97169220 wait complete. 2019-12-05T06:16:06.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:06.915+0000 7fde95e31700 1 -- 172.21.15.16:0/97169220 >> 172.21.15.16:0/97169220 conn(0x7fde900714f0 msgr2=0x7fde9009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:06.919 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:16:06.919 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 16 secs before thrashing 2019-12-05T06:16:22.924 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.176+0000 7f37600ad700 1 Processor -- start 2019-12-05T06:16:23.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.176+0000 7f37600ad700 1 -- start start 2019-12-05T06:16:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.177+0000 7f37600ad700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f375808d2c0 0x7f375808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.177+0000 7f37600ad700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.177+0000 7f37600ad700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37580958e0 con 0x7f375808c6d0 2019-12-05T06:16:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.177+0000 7f37600ad700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f375808df90 con 0x7f375808d2c0 2019-12-05T06:16:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.177+0000 7f37600ad700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37580a3f70 con 0x7f375808fef0 2019-12-05T06:16:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.178+0000 7f375de49700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f375808d2c0 0x7f375808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:23.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.178+0000 7f375ee4b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.178+0000 7f375e64a700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f375808c6d0 0x7f375808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:56728/0 (socket says 172.21.15.16:56728) 2019-12-05T06:16:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.178+0000 7f375de49700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f375808d2c0 0x7f375808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:36352/0 (socket says 172.21.15.16:36352) 2019-12-05T06:16:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.178+0000 7f375ee4b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:54250/0 (socket says 172.21.15.16:54250) 2019-12-05T06:16:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.178+0000 7f375e64a700 1 -- 172.21.15.16:0/628880903 learned_addr learned my addr 172.21.15.16:0/628880903 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:16:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.178+0000 7f375ee4b700 1 -- 172.21.15.16:0/628880903 >> v2:172.21.15.76:3300/0 conn(0x7f375808d2c0 msgr2=0x7f375808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.178+0000 7f375ee4b700 1 --2- 172.21.15.16:0/628880903 >> v2:172.21.15.76:3300/0 conn(0x7f375808d2c0 0x7f375808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:23.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.178+0000 7f375ee4b700 1 -- 172.21.15.16:0/628880903 >> v1:172.21.15.76:6789/0 conn(0x7f375808c6d0 legacy=0x7f375808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.178+0000 7f375ee4b700 1 -- 172.21.15.16:0/628880903 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f375808e620 con 0x7f375808fef0 2019-12-05T06:16:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.179+0000 7f375ee4b700 1 --2- 172.21.15.16:0/628880903 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f374800af80 tx=0x7f374800b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.179+0000 7f375ee4b700 1 --2- 172.21.15.16:0/628880903 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f374800af80 tx=0x7f374800b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.179+0000 7f375ee4b700 1 --2- 172.21.15.16:0/628880903 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 secure :-1 s=READY pgs=120 cs=0 l=1 rx=0x7f374800af80 tx=0x7f374800b280).ready entity=mon.2 client_cookie=f3ab9f87c2428165 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.179+0000 7f375ee4b700 1 --2- 172.21.15.16:0/628880903 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f374800af80 tx=0x7f374800b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.179+0000 7f375ee4b700 1 --2- 172.21.15.16:0/628880903 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f374800af80 tx=0x7f374800b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.179+0000 7f375ee4b700 1 --2- 172.21.15.16:0/628880903 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 secure :-1 s=THROTTLE_DONE pgs=120 cs=0 l=1 rx=0x7f374800af80 tx=0x7f374800b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.179+0000 7f375d648700 1 -- 172.21.15.16:0/628880903 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3748007240 con 0x7f375808fef0 2019-12-05T06:16:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.179+0000 7f375d648700 1 -- 172.21.15.16:0/628880903 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37480099e0 con 0x7f375808fef0 2019-12-05T06:16:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.179+0000 7f375d648700 1 -- 172.21.15.16:0/628880903 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3748009fb0 con 0x7f375808fef0 2019-12-05T06:16:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.180+0000 7f37600ad700 1 -- 172.21.15.16:0/628880903 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 msgr2=0x7f375808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.180+0000 7f37600ad700 1 --2- 172.21.15.16:0/628880903 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 secure :-1 s=READY pgs=120 cs=0 l=1 rx=0x7f374800af80 tx=0x7f374800b280).stop 2019-12-05T06:16:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.181+0000 7f37600ad700 1 -- 172.21.15.16:0/628880903 shutdown_connections 2019-12-05T06:16:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.181+0000 7f37600ad700 1 --2- 172.21.15.16:0/628880903 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f375808fef0 0x7f375808f460 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.181+0000 7f37600ad700 1 --2- 172.21.15.16:0/628880903 >> v2:172.21.15.76:3300/0 conn(0x7f375808d2c0 0x7f375808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.181+0000 7f37600ad700 1 -- 172.21.15.16:0/628880903 shutdown_connections 2019-12-05T06:16:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.181+0000 7f37600ad700 1 -- 172.21.15.16:0/628880903 wait complete. 2019-12-05T06:16:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.181+0000 7f37600ad700 1 -- 172.21.15.16:0/628880903 >> 172.21.15.16:0/628880903 conn(0x7f3758061940 msgr2=0x7f3758096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.185+0000 7f37600ad700 1 Processor -- start 2019-12-05T06:16:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.185+0000 7f37600ad700 1 -- start start 2019-12-05T06:16:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.187+0000 7f37600ad700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f375806e840 0x7f3758134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.187+0000 7f37600ad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3758135450 0x7f37581349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.187+0000 7f37600ad700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3758128af0 con 0x7f375806dc50 2019-12-05T06:16:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.187+0000 7f37600ad700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3758132830 con 0x7f3758135450 2019-12-05T06:16:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.187+0000 7f37600ad700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37581334f0 con 0x7f375806e840 2019-12-05T06:16:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.187+0000 7f375ee4b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3758135450 0x7f37581349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.187+0000 7f375ee4b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3758135450 0x7f37581349c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50268/0 (socket says 172.21.15.16:50268) 2019-12-05T06:16:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.187+0000 7f375e64a700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f375806dc50 0x7f375806e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:56734/0 (socket says 172.21.15.16:56734) 2019-12-05T06:16:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.187+0000 7f375ee4b700 1 -- 172.21.15.16:0/1717957947 learned_addr learned my addr 172.21.15.16:0/1717957947 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.187+0000 7f375de49700 1 --2- 172.21.15.16:0/1717957947 >> v2:172.21.15.76:3300/0 conn(0x7f375806e840 0x7f3758134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:23.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f375ee4b700 1 -- 172.21.15.16:0/1717957947 >> v2:172.21.15.76:3300/0 conn(0x7f375806e840 msgr2=0x7f3758134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> v2:172.21.15.76:3300/0 conn(0x7f375806e840 0x7f3758134480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f375ee4b700 1 -- 172.21.15.16:0/1717957947 >> v1:172.21.15.76:6789/0 conn(0x7f375806dc50 legacy=0x7f375806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f375ee4b700 1 -- 172.21.15.16:0/1717957947 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3758133b80 con 0x7f3758135450 2019-12-05T06:16:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> v2:172.21.15.16:3300/0 conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> v2:172.21.15.16:3300/0 conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=READY pgs=441 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.188+0000 7f3756ffd700 1 -- 172.21.15.16:0/1717957947 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374800afb0 con 0x7f3758135450 2019-12-05T06:16:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.189+0000 7f3756ffd700 1 -- 172.21.15.16:0/1717957947 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37480097a0 con 0x7f3758135450 2019-12-05T06:16:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.189+0000 7f37600ad700 1 -- 172.21.15.16:0/1717957947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3758133b80 con 0x7f3758135450 2019-12-05T06:16:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.189+0000 7f3756ffd700 1 -- 172.21.15.16:0/1717957947 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3748009eb0 con 0x7f3758135450 2019-12-05T06:16:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.189+0000 7f37600ad700 1 -- 172.21.15.16:0/1717957947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3758133b80 con 0x7f3758135450 2019-12-05T06:16:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.190+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.190+0000 7f3756ffd700 1 -- 172.21.15.16:0/1717957947 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3748009990 con 0x7f3758135450 2019-12-05T06:16:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.190+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.191+0000 7f3756ffd700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3738023630 0x7f3738025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.191+0000 7f3756ffd700 1 -- 172.21.15.16:0/1717957947 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f3748045db0 con 0x7f3758135450 2019-12-05T06:16:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.191+0000 7f375e64a700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3738023630 0x7f3738025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.192+0000 7f375e64a700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3738023630 0x7f3738025ac0 crc :-1 s=READY pgs=202 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.192+0000 7f37600ad700 1 -- 172.21.15.16:0/1717957947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3758133b80 con 0x7f3758135450 2019-12-05T06:16:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.200+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.200+0000 7f3756ffd700 1 -- 172.21.15.16:0/1717957947 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3748000a70 con 0x7f3758135450 2019-12-05T06:16:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.465+0000 7f37600ad700 1 -- 172.21.15.16:0/1717957947 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3758133b80 con 0x7f3758135450 2019-12-05T06:16:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.466+0000 7f375ee4b700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=THROTTLE_DONE pgs=441 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.466+0000 7f3756ffd700 1 -- 172.21.15.16:0/1717957947 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f3748009990 con 0x7f3758135450 2019-12-05T06:16:23.462 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:23.462 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:58.002271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":28,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:23.462 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 57 2019-12-05T06:16:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 -- 172.21.15.16:0/1717957947 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3738023630 msgr2=0x7f3738025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:23.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3738023630 0x7f3738025ac0 crc :-1 s=READY pgs=202 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 -- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 msgr2=0x7f37581349c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 0x7f37581349c0 secure :-1 s=READY pgs=441 cs=0 l=1 rx=0x7f3748010410 tx=0x7f37480102c0).stop 2019-12-05T06:16:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 -- 172.21.15.16:0/1717957947 shutdown_connections 2019-12-05T06:16:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3738023630 0x7f3738025ac0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 --2- 172.21.15.16:0/1717957947 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3758135450 0x7f37581349c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 --2- 172.21.15.16:0/1717957947 >> v2:172.21.15.76:3300/0 conn(0x7f375806e840 0x7f3758134480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:23.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 -- 172.21.15.16:0/1717957947 shutdown_connections 2019-12-05T06:16:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 -- 172.21.15.16:0/1717957947 wait complete. 2019-12-05T06:16:23.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.471+0000 7f37600ad700 1 -- 172.21.15.16:0/1717957947 >> 172.21.15.16:0/1717957947 conn(0x7f37580a2850 msgr2=0x7f375806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:23.482 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:16:23.483 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:16:23.483 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:16:23.498 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:16:23.499 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:23.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.752+0000 7f806939c700 1 Processor -- start 2019-12-05T06:16:23.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.752+0000 7f806939c700 1 -- start start 2019-12-05T06:16:23.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.754+0000 7f806939c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f806408c6d0 0x7f806408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:23.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.754+0000 7f806939c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f806408fef0 0x7f806408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:23.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.754+0000 7f806939c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80640958e0 con 0x7f806408ef20 2019-12-05T06:16:23.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.754+0000 7f806939c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f806408df90 con 0x7f806408c6d0 2019-12-05T06:16:23.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.754+0000 7f806939c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80640a3f70 con 0x7f806408fef0 2019-12-05T06:16:23.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.754+0000 7f80637fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f806408c6d0 0x7f806408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:23.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.754+0000 7f80637fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f806408c6d0 0x7f806408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50276/0 (socket says 172.21.15.16:50276) 2019-12-05T06:16:23.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.754+0000 7f80637fe700 1 -- 172.21.15.16:0/3574889204 learned_addr learned my addr 172.21.15.16:0/3574889204 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:23.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.754+0000 7f8062ffd700 1 --2- 172.21.15.16:0/3574889204 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f806408fef0 0x7f806408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:23.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.755+0000 7f80637fe700 1 -- 172.21.15.16:0/3574889204 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f806408fef0 msgr2=0x7f806408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.755+0000 7f80637fe700 1 --2- 172.21.15.16:0/3574889204 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f806408fef0 0x7f806408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.755+0000 7f80637fe700 1 -- 172.21.15.16:0/3574889204 >> v1:172.21.15.76:6789/0 conn(0x7f806408ef20 legacy=0x7f806408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.755+0000 7f80637fe700 1 -- 172.21.15.16:0/3574889204 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806408d540 con 0x7f806408c6d0 2019-12-05T06:16:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.755+0000 7f80637fe700 1 --2- 172.21.15.16:0/3574889204 >> v2:172.21.15.16:3300/0 conn(0x7f806408c6d0 0x7f806408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f805c00fac0 tx=0x7f805c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.755+0000 7f80637fe700 1 --2- 172.21.15.16:0/3574889204 >> v2:172.21.15.16:3300/0 conn(0x7f806408c6d0 0x7f806408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f805c00fac0 tx=0x7f805c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.755+0000 7f80637fe700 1 --2- 172.21.15.16:0/3574889204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806408c6d0 0x7f806408cac0 secure :-1 s=READY pgs=442 cs=0 l=1 rx=0x7f805c00fac0 tx=0x7f805c00f8f0).ready entity=mon.0 client_cookie=99ee5e22791546e9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.756+0000 7f80637fe700 1 --2- 172.21.15.16:0/3574889204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806408c6d0 0x7f806408cac0 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7f805c00fac0 tx=0x7f805c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.756+0000 7f80637fe700 1 --2- 172.21.15.16:0/3574889204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806408c6d0 0x7f806408cac0 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7f805c00fac0 tx=0x7f805c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.756+0000 7f80627fc700 1 -- 172.21.15.16:0/3574889204 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f805c000ce0 con 0x7f806408c6d0 2019-12-05T06:16:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.756+0000 7f80637fe700 1 --2- 172.21.15.16:0/3574889204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806408c6d0 0x7f806408cac0 secure :-1 s=THROTTLE_DONE pgs=442 cs=0 l=1 rx=0x7f805c00fac0 tx=0x7f805c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.756+0000 7f80627fc700 1 -- 172.21.15.16:0/3574889204 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c001010 con 0x7f806408c6d0 2019-12-05T06:16:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.756+0000 7f80627fc700 1 -- 172.21.15.16:0/3574889204 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f805c001720 con 0x7f806408c6d0 2019-12-05T06:16:23.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.756+0000 7f806939c700 1 -- 172.21.15.16:0/3574889204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806408c6d0 msgr2=0x7f806408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.756+0000 7f806939c700 1 --2- 172.21.15.16:0/3574889204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806408c6d0 0x7f806408cac0 secure :-1 s=READY pgs=442 cs=0 l=1 rx=0x7f805c00fac0 tx=0x7f805c00f8f0).stop 2019-12-05T06:16:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.758+0000 7f806939c700 1 -- 172.21.15.16:0/3574889204 shutdown_connections 2019-12-05T06:16:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.758+0000 7f806939c700 1 --2- 172.21.15.16:0/3574889204 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f806408fef0 0x7f806408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.758+0000 7f806939c700 1 --2- 172.21.15.16:0/3574889204 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f806408c6d0 0x7f806408cac0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.758+0000 7f806939c700 1 -- 172.21.15.16:0/3574889204 shutdown_connections 2019-12-05T06:16:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.758+0000 7f806939c700 1 -- 172.21.15.16:0/3574889204 wait complete. 2019-12-05T06:16:23.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.758+0000 7f806939c700 1 -- 172.21.15.16:0/3574889204 >> 172.21.15.16:0/3574889204 conn(0x7f8064061940 msgr2=0x7f8064096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:23.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.763+0000 7f806939c700 1 Processor -- start 2019-12-05T06:16:23.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.764+0000 7f806939c700 1 -- start start 2019-12-05T06:16:23.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.765+0000 7f806939c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8064132630 0x7f8064132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:23.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.765+0000 7f806939c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f806409ae80 0x7f8064132f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:23.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.765+0000 7f806939c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f806409ab10 con 0x7f8064099eb0 2019-12-05T06:16:23.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.765+0000 7f806939c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8064098f20 con 0x7f8064132630 2019-12-05T06:16:23.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.765+0000 7f806939c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8064134390 con 0x7f806409ae80 2019-12-05T06:16:23.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.766+0000 7f80637fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8064132630 0x7f8064132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:23.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.766+0000 7f80637fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8064132630 0x7f8064132a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50282/0 (socket says 172.21.15.16:50282) 2019-12-05T06:16:23.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.766+0000 7f80637fe700 1 -- 172.21.15.16:0/1096781354 learned_addr learned my addr 172.21.15.16:0/1096781354 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:23.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.766+0000 7f8063fff700 1 --1- 172.21.15.16:0/1096781354 >> v1:172.21.15.16:6789/0 conn(0x7f8064099eb0 0x7f806409a310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35454/0 (socket says 172.21.15.16:35454) 2019-12-05T06:16:23.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.766+0000 7f8062ffd700 1 --2- 172.21.15.16:0/1096781354 >> v2:172.21.15.76:3300/0 conn(0x7f806409ae80 0x7f8064132f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:23.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.766+0000 7f80637fe700 1 -- 172.21.15.16:0/1096781354 >> v2:172.21.15.76:3300/0 conn(0x7f806409ae80 msgr2=0x7f8064132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:23.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.766+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> v2:172.21.15.76:3300/0 conn(0x7f806409ae80 0x7f8064132f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:23.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.766+0000 7f80637fe700 1 -- 172.21.15.16:0/1096781354 >> v1:172.21.15.16:6789/0 conn(0x7f8064099eb0 legacy=0x7f806409a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:23.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f80637fe700 1 -- 172.21.15.16:0/1096781354 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80640995b0 con 0x7f8064132630 2019-12-05T06:16:23.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f805bfff700 1 -- 172.21.15.16:0/1096781354 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 730607645 0 0) 0x7f8054000b90 con 0x7f8064099eb0 2019-12-05T06:16:23.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> v2:172.21.15.16:3300/0 conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> v2:172.21.15.16:3300/0 conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=READY pgs=444 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:23.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=THROTTLE_DONE pgs=444 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f805bfff700 1 -- 172.21.15.16:0/1096781354 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f805c008190 con 0x7f8064132630 2019-12-05T06:16:23.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=THROTTLE_DONE pgs=444 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=THROTTLE_DONE pgs=444 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f805bfff700 1 -- 172.21.15.16:0/1096781354 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f805c005710 con 0x7f8064132630 2019-12-05T06:16:23.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f806939c700 1 -- 172.21.15.16:0/1096781354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80640995b0 con 0x7f8064132630 2019-12-05T06:16:23.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f805bfff700 1 -- 172.21.15.16:0/1096781354 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f805c005d60 con 0x7f8064132630 2019-12-05T06:16:23.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.767+0000 7f806939c700 1 -- 172.21.15.16:0/1096781354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80640995b0 con 0x7f8064132630 2019-12-05T06:16:23.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.768+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=THROTTLE_DONE pgs=444 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.769+0000 7f805bfff700 1 -- 172.21.15.16:0/1096781354 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f805c005900 con 0x7f8064132630 2019-12-05T06:16:23.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.769+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=THROTTLE_DONE pgs=444 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.770+0000 7f805bfff700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8048023630 0x7f8048025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:23.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.770+0000 7f805bfff700 1 -- 172.21.15.16:0/1096781354 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f805c046170 con 0x7f8064132630 2019-12-05T06:16:23.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.770+0000 7f8062ffd700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8048023630 0x7f8048025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:23.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.771+0000 7f8062ffd700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8048023630 0x7f8048025ac0 crc :-1 s=READY pgs=203 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:23.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.784+0000 7f806939c700 1 -- 172.21.15.16:0/1096781354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80640995b0 con 0x7f8064132630 2019-12-05T06:16:23.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.791+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=THROTTLE_DONE pgs=444 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:23.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:23.791+0000 7f805bfff700 1 -- 172.21.15.16:0/1096781354 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f805c033e60 con 0x7f8064132630 2019-12-05T06:16:24.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.050+0000 7f806939c700 1 -- 172.21.15.16:0/1096781354 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80640995b0 con 0x7f8064132630 2019-12-05T06:16:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.050+0000 7f80637fe700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=THROTTLE_DONE pgs=444 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.050+0000 7f805bfff700 1 -- 172.21.15.16:0/1096781354 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f805c033e60 con 0x7f8064132630 2019-12-05T06:16:24.044 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 57 2019-12-05T06:16:24.045 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:24.045 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:58.002271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":28,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:24.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.053+0000 7f806939c700 1 -- 172.21.15.16:0/1096781354 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8048023630 msgr2=0x7f8048025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:24.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.053+0000 7f806939c700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8048023630 0x7f8048025ac0 crc :-1 s=READY pgs=203 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:24.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.053+0000 7f806939c700 1 -- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 msgr2=0x7f8064132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:24.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.053+0000 7f806939c700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 0x7f8064132a20 secure :-1 s=READY pgs=444 cs=0 l=1 rx=0x7f805c010760 tx=0x7f805c010b00).stop 2019-12-05T06:16:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.053+0000 7f806939c700 1 -- 172.21.15.16:0/1096781354 shutdown_connections 2019-12-05T06:16:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.053+0000 7f806939c700 1 --2- 172.21.15.16:0/1096781354 >> v2:172.21.15.76:3300/0 conn(0x7f806409ae80 0x7f8064132f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.053+0000 7f806939c700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8048023630 0x7f8048025ac0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.053+0000 7f806939c700 1 --2- 172.21.15.16:0/1096781354 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8064132630 0x7f8064132a20 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:24.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.054+0000 7f806939c700 1 -- 172.21.15.16:0/1096781354 shutdown_connections 2019-12-05T06:16:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.054+0000 7f806939c700 1 -- 172.21.15.16:0/1096781354 wait complete. 2019-12-05T06:16:24.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:24.054+0000 7f806939c700 1 -- 172.21.15.16:0/1096781354 >> 172.21.15.16:0/1096781354 conn(0x7f80640714f0 msgr2=0x7f806409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:24.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:26.058 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.313+0000 7f7a7e881700 1 Processor -- start 2019-12-05T06:16:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.313+0000 7f7a7e881700 1 -- start start 2019-12-05T06:16:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.315+0000 7f7a7e881700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7a7808c6d0 0x7f7a7808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:26.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.315+0000 7f7a7e881700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a7808fb80 con 0x7f7a7808fef0 2019-12-05T06:16:26.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.315+0000 7f7a7e881700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a780958e0 con 0x7f7a7808f720 2019-12-05T06:16:26.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.315+0000 7f7a7e881700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a7808df90 con 0x7f7a7808c6d0 2019-12-05T06:16:26.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.315+0000 7f7a77fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a7808fef0 0x7f7a7808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35462/0 (socket says 172.21.15.16:35462) 2019-12-05T06:16:26.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.315+0000 7f7a77fff700 1 -- 172.21.15.16:0/2862128991 learned_addr learned my addr 172.21.15.16:0/2862128991 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:16:26.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.315+0000 7f7a7ce1e700 1 --2- 172.21.15.16:0/2862128991 >> v2:172.21.15.76:3300/0 conn(0x7f7a7808c6d0 0x7f7a7808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:26.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.316+0000 7f7a777fe700 1 -- 172.21.15.16:0/2862128991 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617477801 0 0) 0x7f7a68001150 con 0x7f7a7808fef0 2019-12-05T06:16:26.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.316+0000 7f7a777fe700 1 -- 172.21.15.16:0/2862128991 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a64002e80 con 0x7f7a7808fef0 2019-12-05T06:16:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.316+0000 7f7a777fe700 1 -- 172.21.15.16:0/2862128991 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528697971 0 0) 0x7f7a60001080 con 0x7f7a7808f720 2019-12-05T06:16:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.316+0000 7f7a777fe700 1 -- 172.21.15.16:0/2862128991 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a640030b0 con 0x7f7a7808f720 2019-12-05T06:16:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.316+0000 7f7a7ce1e700 1 -- 172.21.15.16:0/2862128991 >> v1:172.21.15.76:6789/0 conn(0x7f7a7808f720 legacy=0x7f7a781533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.316+0000 7f7a7ce1e700 1 -- 172.21.15.16:0/2862128991 >> v1:172.21.15.16:6789/0 conn(0x7f7a7808fef0 legacy=0x7f7a7808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.316+0000 7f7a7ce1e700 1 -- 172.21.15.16:0/2862128991 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7808e620 con 0x7f7a7808c6d0 2019-12-05T06:16:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.316+0000 7f7a7ce1e700 1 --2- 172.21.15.16:0/2862128991 >> v2:172.21.15.76:3300/0 conn(0x7f7a7808c6d0 0x7f7a7808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7a7000b2d0 tx=0x7f7a7000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.317+0000 7f7a7ce1e700 1 --2- 172.21.15.16:0/2862128991 >> v2:172.21.15.76:3300/0 conn(0x7f7a7808c6d0 0x7f7a7808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7a7000b2d0 tx=0x7f7a7000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.317+0000 7f7a7ce1e700 1 --2- 172.21.15.16:0/2862128991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a7808c6d0 0x7f7a7808cac0 secure :-1 s=READY pgs=344 cs=0 l=1 rx=0x7f7a7000b2d0 tx=0x7f7a7000bae0).ready entity=mon.1 client_cookie=52a26b64ffe9158e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.317+0000 7f7a7ce1e700 1 --2- 172.21.15.16:0/2862128991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a7808c6d0 0x7f7a7808cac0 secure :-1 s=THROTTLE_DONE pgs=344 cs=0 l=1 rx=0x7f7a7000b2d0 tx=0x7f7a7000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.317+0000 7f7a7ce1e700 1 --2- 172.21.15.16:0/2862128991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a7808c6d0 0x7f7a7808cac0 secure :-1 s=THROTTLE_DONE pgs=344 cs=0 l=1 rx=0x7f7a7000b2d0 tx=0x7f7a7000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.317+0000 7f7a777fe700 1 -- 172.21.15.16:0/2862128991 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a70000ce0 con 0x7f7a7808c6d0 2019-12-05T06:16:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.317+0000 7f7a7ce1e700 1 --2- 172.21.15.16:0/2862128991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a7808c6d0 0x7f7a7808cac0 secure :-1 s=THROTTLE_DONE pgs=344 cs=0 l=1 rx=0x7f7a7000b2d0 tx=0x7f7a7000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.317+0000 7f7a777fe700 1 -- 172.21.15.16:0/2862128991 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a70009c60 con 0x7f7a7808c6d0 2019-12-05T06:16:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.317+0000 7f7a777fe700 1 -- 172.21.15.16:0/2862128991 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a7000a3d0 con 0x7f7a7808c6d0 2019-12-05T06:16:26.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.317+0000 7f7a7e881700 1 -- 172.21.15.16:0/2862128991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a7808c6d0 msgr2=0x7f7a7808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.317+0000 7f7a7e881700 1 --2- 172.21.15.16:0/2862128991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a7808c6d0 0x7f7a7808cac0 secure :-1 s=READY pgs=344 cs=0 l=1 rx=0x7f7a7000b2d0 tx=0x7f7a7000bae0).stop 2019-12-05T06:16:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.319+0000 7f7a7e881700 1 -- 172.21.15.16:0/2862128991 shutdown_connections 2019-12-05T06:16:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.319+0000 7f7a7e881700 1 --2- 172.21.15.16:0/2862128991 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a7808c6d0 0x7f7a7808cac0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.319+0000 7f7a7e881700 1 -- 172.21.15.16:0/2862128991 shutdown_connections 2019-12-05T06:16:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.319+0000 7f7a7e881700 1 -- 172.21.15.16:0/2862128991 wait complete. 2019-12-05T06:16:26.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.319+0000 7f7a7e881700 1 -- 172.21.15.16:0/2862128991 >> 172.21.15.16:0/2862128991 conn(0x7f7a78061940 msgr2=0x7f7a78096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:26.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.324+0000 7f7a7e881700 1 Processor -- start 2019-12-05T06:16:26.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.324+0000 7f7a7e881700 1 -- start start 2019-12-05T06:16:26.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.326+0000 7f7a7e881700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a78128360 0x7f7a78128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:26.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.326+0000 7f7a7e881700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7a78154770 0x7f7a7812bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:26.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.326+0000 7f7a7e881700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a78099720 con 0x7f7a78137ea0 2019-12-05T06:16:26.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.326+0000 7f7a7e881700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a78129c10 con 0x7f7a78154770 2019-12-05T06:16:26.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.326+0000 7f7a7e881700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a78129ee0 con 0x7f7a78128360 2019-12-05T06:16:26.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.326+0000 7f7a7ce1e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a78128360 0x7f7a78128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.327+0000 7f7a77fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7a78137ea0 0x7f7a7812b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:56770/0 (socket says 172.21.15.16:56770) 2019-12-05T06:16:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.327+0000 7f7a77fff700 1 -- 172.21.15.16:0/684220943 learned_addr learned my addr 172.21.15.16:0/684220943 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:16:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.327+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> v2:172.21.15.76:3300/0 conn(0x7f7a78154770 0x7f7a7812bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.327+0000 7f7a757fa700 1 -- 172.21.15.16:0/684220943 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3049788979 0 0) 0x7f7a68000f40 con 0x7f7a78137ea0 2019-12-05T06:16:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.327+0000 7f7a757fa700 1 -- 172.21.15.16:0/684220943 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a5c002e80 con 0x7f7a78137ea0 2019-12-05T06:16:26.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.327+0000 7f7a7d61f700 1 -- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a78128360 msgr2=0x7f7a78128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:26.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.327+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a78128360 0x7f7a78128750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:26.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.327+0000 7f7a7d61f700 1 -- 172.21.15.16:0/684220943 >> v1:172.21.15.76:6789/0 conn(0x7f7a78137ea0 legacy=0x7f7a7812b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:26.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.327+0000 7f7a7d61f700 1 -- 172.21.15.16:0/684220943 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7812acd0 con 0x7f7a78154770 2019-12-05T06:16:26.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a7ce1e700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a78128360 0x7f7a78128750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:16:26.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> v2:172.21.15.76:3300/0 conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> v2:172.21.15.76:3300/0 conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=READY pgs=346 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:26.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a757fa700 1 -- 172.21.15.16:0/684220943 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a6000f1b0 con 0x7f7a78154770 2019-12-05T06:16:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a757fa700 1 -- 172.21.15.16:0/684220943 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a6000ed30 con 0x7f7a78154770 2019-12-05T06:16:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a757fa700 1 -- 172.21.15.16:0/684220943 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a6000f790 con 0x7f7a78154770 2019-12-05T06:16:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a7e881700 1 -- 172.21.15.16:0/684220943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a7812acd0 con 0x7f7a78154770 2019-12-05T06:16:26.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.328+0000 7f7a7e881700 1 -- 172.21.15.16:0/684220943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a7812acd0 con 0x7f7a78154770 2019-12-05T06:16:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.329+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.330+0000 7f7a757fa700 1 -- 172.21.15.16:0/684220943 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7a6000f3e0 con 0x7f7a78154770 2019-12-05T06:16:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.330+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.331+0000 7f7a757fa700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a5c023670 0x7f7a5c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.331+0000 7f7a757fa700 1 -- 172.21.15.16:0/684220943 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f7a60045d30 con 0x7f7a78154770 2019-12-05T06:16:26.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.331+0000 7f7a7ce1e700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a5c023670 0x7f7a5c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:26.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.331+0000 7f7a7ce1e700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a5c023670 0x7f7a5c025ae0 crc :-1 s=READY pgs=204 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:26.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.345+0000 7f7a7e881700 1 -- 172.21.15.16:0/684220943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a7812acd0 con 0x7f7a78154770 2019-12-05T06:16:26.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.354+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.354+0000 7f7a757fa700 1 -- 172.21.15.16:0/684220943 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7a6000f1d0 con 0x7f7a78154770 2019-12-05T06:16:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.617+0000 7f7a7e881700 1 -- 172.21.15.16:0/684220943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a7812acd0 con 0x7f7a78154770 2019-12-05T06:16:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.619+0000 7f7a7d61f700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=THROTTLE_DONE pgs=346 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.619+0000 7f7a757fa700 1 -- 172.21.15.16:0/684220943 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f7a6000f3e0 con 0x7f7a78154770 2019-12-05T06:16:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 57 2019-12-05T06:16:26.614 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:26.614 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:58.002271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":28,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 -- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a5c023670 msgr2=0x7f7a5c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a5c023670 0x7f7a5c025ae0 crc :-1 s=READY pgs=204 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 -- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 msgr2=0x7f7a7812bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 0x7f7a7812bdd0 secure :-1 s=READY pgs=346 cs=0 l=1 rx=0x7f7a600103a0 tx=0x7f7a60010740).stop 2019-12-05T06:16:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 -- 172.21.15.16:0/684220943 shutdown_connections 2019-12-05T06:16:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a5c023670 0x7f7a5c025ae0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:26.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a78154770 0x7f7a7812bdd0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:26.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 --2- 172.21.15.16:0/684220943 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a78128360 0x7f7a78128750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:26.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 -- 172.21.15.16:0/684220943 shutdown_connections 2019-12-05T06:16:26.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 -- 172.21.15.16:0/684220943 wait complete. 2019-12-05T06:16:26.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:26.623+0000 7f7a7e881700 1 -- 172.21.15.16:0/684220943 >> 172.21.15.16:0/684220943 conn(0x7f7a7806d260 msgr2=0x7f7a7809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:26.627 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:28.628 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:28.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.903+0000 7f4c540a3700 1 Processor -- start 2019-12-05T06:16:28.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.903+0000 7f4c540a3700 1 -- start start 2019-12-05T06:16:28.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.904+0000 7f4c540a3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:28.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.904+0000 7f4c540a3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4c4c08fef0 0x7f4c4c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:28.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.904+0000 7f4c540a3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c4c08fb80 con 0x7f4c4c08ef20 2019-12-05T06:16:28.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.904+0000 7f4c540a3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c08df90 con 0x7f4c4c08c6d0 2019-12-05T06:16:28.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.904+0000 7f4c540a3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c0a3f70 con 0x7f4c4c08fef0 2019-12-05T06:16:28.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.905+0000 7f4c52640700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:28.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.905+0000 7f4c52640700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50310/0 (socket says 172.21.15.16:50310) 2019-12-05T06:16:28.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.905+0000 7f4c51e3f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4c4c08fef0 0x7f4c4c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:28.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.905+0000 7f4c52640700 1 -- 172.21.15.16:0/2865211499 learned_addr learned my addr 172.21.15.16:0/2865211499 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:28.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.905+0000 7f4c52640700 1 -- 172.21.15.16:0/2865211499 >> v2:172.21.15.76:3300/0 conn(0x7f4c4c08fef0 msgr2=0x7f4c4c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:28.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.905+0000 7f4c52640700 1 --2- 172.21.15.16:0/2865211499 >> v2:172.21.15.76:3300/0 conn(0x7f4c4c08fef0 0x7f4c4c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:28.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.905+0000 7f4c52640700 1 -- 172.21.15.16:0/2865211499 >> v1:172.21.15.16:6789/0 conn(0x7f4c4c08ef20 legacy=0x7f4c4c08f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:28.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c52640700 1 -- 172.21.15.16:0/2865211499 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c08d540 con 0x7f4c4c08c6d0 2019-12-05T06:16:28.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c52640700 1 --2- 172.21.15.16:0/2865211499 >> v2:172.21.15.16:3300/0 conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4c4400f740 tx=0x7f4c4400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c52640700 1 --2- 172.21.15.16:0/2865211499 >> v2:172.21.15.16:3300/0 conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4c4400f740 tx=0x7f4c4400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c52640700 1 --2- 172.21.15.16:0/2865211499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 secure :-1 s=READY pgs=447 cs=0 l=1 rx=0x7f4c4400f740 tx=0x7f4c4400f570).ready entity=mon.0 client_cookie=7df96f680e1e772c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:28.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c52640700 1 --2- 172.21.15.16:0/2865211499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f4c4400f740 tx=0x7f4c4400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c52640700 1 --2- 172.21.15.16:0/2865211499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f4c4400f740 tx=0x7f4c4400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c5163e700 1 -- 172.21.15.16:0/2865211499 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c44000960 con 0x7f4c4c08c6d0 2019-12-05T06:16:28.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c52640700 1 --2- 172.21.15.16:0/2865211499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f4c4400f740 tx=0x7f4c4400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c5163e700 1 -- 172.21.15.16:0/2865211499 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44000c90 con 0x7f4c4c08c6d0 2019-12-05T06:16:28.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c5163e700 1 -- 172.21.15.16:0/2865211499 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c440013a0 con 0x7f4c4c08c6d0 2019-12-05T06:16:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c540a3700 1 -- 172.21.15.16:0/2865211499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c08c6d0 msgr2=0x7f4c4c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.906+0000 7f4c540a3700 1 --2- 172.21.15.16:0/2865211499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 secure :-1 s=READY pgs=447 cs=0 l=1 rx=0x7f4c4400f740 tx=0x7f4c4400f570).stop 2019-12-05T06:16:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.908+0000 7f4c540a3700 1 -- 172.21.15.16:0/2865211499 shutdown_connections 2019-12-05T06:16:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.908+0000 7f4c540a3700 1 --2- 172.21.15.16:0/2865211499 >> v2:172.21.15.76:3300/0 conn(0x7f4c4c08fef0 0x7f4c4c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.908+0000 7f4c540a3700 1 --2- 172.21.15.16:0/2865211499 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c08c6d0 0x7f4c4c08cac0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.908+0000 7f4c540a3700 1 -- 172.21.15.16:0/2865211499 shutdown_connections 2019-12-05T06:16:28.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.908+0000 7f4c540a3700 1 -- 172.21.15.16:0/2865211499 wait complete. 2019-12-05T06:16:28.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.908+0000 7f4c540a3700 1 -- 172.21.15.16:0/2865211499 >> 172.21.15.16:0/2865211499 conn(0x7f4c4c061940 msgr2=0x7f4c4c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:28.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.912+0000 7f4c540a3700 1 Processor -- start 2019-12-05T06:16:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.912+0000 7f4c540a3700 1 -- start start 2019-12-05T06:16:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.913+0000 7f4c540a3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c4c096e90 0x7f4c4c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.913+0000 7f4c540a3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4c4c084f30 0x7f4c4c09b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.913+0000 7f4c540a3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4c4c097280 0x7f4c4c0976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:28.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.913+0000 7f4c540a3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c09a450 con 0x7f4c4c096e90 2019-12-05T06:16:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.913+0000 7f4c540a3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c4c099a20 con 0x7f4c4c084f30 2019-12-05T06:16:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.913+0000 7f4c540a3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c4c099cf0 con 0x7f4c4c097280 2019-12-05T06:16:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.914+0000 7f4c52640700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c4c096e90 0x7f4c4c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:28.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.914+0000 7f4c52640700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4c4c096e90 0x7f4c4c09b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50320/0 (socket says 172.21.15.16:50320) 2019-12-05T06:16:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.914+0000 7f4c52640700 1 -- 172.21.15.16:0/2002102645 learned_addr learned my addr 172.21.15.16:0/2002102645 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.914+0000 7f4c51e3f700 1 --2- 172.21.15.16:0/2002102645 >> v2:172.21.15.76:3300/0 conn(0x7f4c4c084f30 0x7f4c4c09b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.914+0000 7f4c52e41700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4c4c097280 0x7f4c4c0976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.914+0000 7f4c52640700 1 -- 172.21.15.16:0/2002102645 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4c4c097280 msgr2=0x7f4c4c0976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.914+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4c4c097280 0x7f4c4c0976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.914+0000 7f4c52640700 1 -- 172.21.15.16:0/2002102645 >> v2:172.21.15.76:3300/0 conn(0x7f4c4c084f30 msgr2=0x7f4c4c09b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:28.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.914+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> v2:172.21.15.76:3300/0 conn(0x7f4c4c084f30 0x7f4c4c09b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:28.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.914+0000 7f4c52640700 1 -- 172.21.15.16:0/2002102645 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c4c09aae0 con 0x7f4c4c096e90 2019-12-05T06:16:28.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> v2:172.21.15.16:3300/0 conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> v2:172.21.15.16:3300/0 conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=READY pgs=448 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:28.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c42ffd700 1 -- 172.21.15.16:0/2002102645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c44007e10 con 0x7f4c4c096e90 2019-12-05T06:16:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c42ffd700 1 -- 172.21.15.16:0/2002102645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c44005390 con 0x7f4c4c096e90 2019-12-05T06:16:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c540a3700 1 -- 172.21.15.16:0/2002102645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c4c09aae0 con 0x7f4c4c096e90 2019-12-05T06:16:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c42ffd700 1 -- 172.21.15.16:0/2002102645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c440059e0 con 0x7f4c4c096e90 2019-12-05T06:16:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.915+0000 7f4c540a3700 1 -- 172.21.15.16:0/2002102645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c4c09aae0 con 0x7f4c4c096e90 2019-12-05T06:16:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.916+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.917+0000 7f4c42ffd700 1 -- 172.21.15.16:0/2002102645 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4c44005580 con 0x7f4c4c096e90 2019-12-05T06:16:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.917+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.917+0000 7f4c42ffd700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4c2c023630 0x7f4c2c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.918+0000 7f4c42ffd700 1 -- 172.21.15.16:0/2002102645 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f4c44045da0 con 0x7f4c4c096e90 2019-12-05T06:16:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.918+0000 7f4c51e3f700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4c2c023630 0x7f4c2c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.918+0000 7f4c51e3f700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4c2c023630 0x7f4c2c025ac0 crc :-1 s=READY pgs=205 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:28.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.919+0000 7f4c540a3700 1 -- 172.21.15.16:0/2002102645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c4c09aae0 con 0x7f4c4c096e90 2019-12-05T06:16:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.928+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:28.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:28.928+0000 7f4c42ffd700 1 -- 172.21.15.16:0/2002102645 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4c44005180 con 0x7f4c4c096e90 2019-12-05T06:16:29.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.180+0000 7f4c540a3700 1 -- 172.21.15.16:0/2002102645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4c4c09aae0 con 0x7f4c4c096e90 2019-12-05T06:16:29.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.181+0000 7f4c52640700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=448 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:29.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.181+0000 7f4c42ffd700 1 -- 172.21.15.16:0/2002102645 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f4c44005580 con 0x7f4c4c096e90 2019-12-05T06:16:29.175 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:29.175 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:58.002271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":28,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:29.175 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 57 2019-12-05T06:16:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 -- 172.21.15.16:0/2002102645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4c2c023630 msgr2=0x7f4c2c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4c2c023630 0x7f4c2c025ac0 crc :-1 s=READY pgs=205 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 -- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 msgr2=0x7f4c4c09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:29.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 0x7f4c4c09b3e0 secure :-1 s=READY pgs=448 cs=0 l=1 rx=0x7f4c440103e0 tx=0x7f4c44010260).stop 2019-12-05T06:16:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 -- 172.21.15.16:0/2002102645 shutdown_connections 2019-12-05T06:16:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4c4c097280 0x7f4c4c0976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 --2- 172.21.15.16:0/2002102645 >> v2:172.21.15.76:3300/0 conn(0x7f4c4c084f30 0x7f4c4c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4c2c023630 0x7f4c2c025ac0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:29.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 --2- 172.21.15.16:0/2002102645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4c4c096e90 0x7f4c4c09b3e0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:29.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 -- 172.21.15.16:0/2002102645 shutdown_connections 2019-12-05T06:16:29.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 -- 172.21.15.16:0/2002102645 wait complete. 2019-12-05T06:16:29.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:29.184+0000 7f4c540a3700 1 -- 172.21.15.16:0/2002102645 >> 172.21.15.16:0/2002102645 conn(0x7f4c4c0714f0 msgr2=0x7f4c4c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:29.189 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:30.792 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:16:30.797 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:16:31.190 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:31.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.438+0000 7f0ce2e21700 1 Processor -- start 2019-12-05T06:16:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.438+0000 7f0ce2e21700 1 -- start start 2019-12-05T06:16:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.440+0000 7f0ce2e21700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.440+0000 7f0ce2e21700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc08f720 0x7f0cdc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:31.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.440+0000 7f0ce2e21700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cdc08fb70 con 0x7f0cdc08fef0 2019-12-05T06:16:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.440+0000 7f0ce2e21700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cdc08df90 con 0x7f0cdc08c6d0 2019-12-05T06:16:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.440+0000 7f0ce2e21700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc0a3f70 con 0x7f0cdc08f720 2019-12-05T06:16:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.440+0000 7f0ce13be700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.440+0000 7f0ce13be700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50326/0 (socket says 172.21.15.16:50326) 2019-12-05T06:16:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.440+0000 7f0ce13be700 1 -- 172.21.15.16:0/839018118 learned_addr learned my addr 172.21.15.16:0/839018118 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.441+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/839018118 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc08f720 0x7f0cdc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:31.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.441+0000 7f0ce13be700 1 -- 172.21.15.16:0/839018118 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc08f720 msgr2=0x7f0cdc08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:31.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.441+0000 7f0ce13be700 1 --2- 172.21.15.16:0/839018118 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc08f720 0x7f0cdc08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.441+0000 7f0ce13be700 1 -- 172.21.15.16:0/839018118 >> v1:172.21.15.76:6789/0 conn(0x7f0cdc08fef0 legacy=0x7f0cdc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.441+0000 7f0ce13be700 1 -- 172.21.15.16:0/839018118 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc08d540 con 0x7f0cdc08c6d0 2019-12-05T06:16:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.441+0000 7f0ce13be700 1 --2- 172.21.15.16:0/839018118 >> v2:172.21.15.16:3300/0 conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0cd400fac0 tx=0x7f0cd400f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.441+0000 7f0ce13be700 1 --2- 172.21.15.16:0/839018118 >> v2:172.21.15.16:3300/0 conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0cd400fac0 tx=0x7f0cd400f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.441+0000 7f0ce13be700 1 --2- 172.21.15.16:0/839018118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 secure :-1 s=READY pgs=449 cs=0 l=1 rx=0x7f0cd400fac0 tx=0x7f0cd400f8f0).ready entity=mon.0 client_cookie=7e3727a1353ae2e5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:31.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.442+0000 7f0ce13be700 1 --2- 172.21.15.16:0/839018118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f0cd400fac0 tx=0x7f0cd400f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.442+0000 7f0ce13be700 1 --2- 172.21.15.16:0/839018118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f0cd400fac0 tx=0x7f0cd400f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.442+0000 7f0cdbfff700 1 -- 172.21.15.16:0/839018118 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cd4000ce0 con 0x7f0cdc08c6d0 2019-12-05T06:16:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.442+0000 7f0ce13be700 1 --2- 172.21.15.16:0/839018118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f0cd400fac0 tx=0x7f0cd400f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.442+0000 7f0cdbfff700 1 -- 172.21.15.16:0/839018118 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd4001010 con 0x7f0cdc08c6d0 2019-12-05T06:16:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.442+0000 7f0cdbfff700 1 -- 172.21.15.16:0/839018118 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cd4001720 con 0x7f0cdc08c6d0 2019-12-05T06:16:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.442+0000 7f0ce2e21700 1 -- 172.21.15.16:0/839018118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc08c6d0 msgr2=0x7f0cdc08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.442+0000 7f0ce2e21700 1 --2- 172.21.15.16:0/839018118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 secure :-1 s=READY pgs=449 cs=0 l=1 rx=0x7f0cd400fac0 tx=0x7f0cd400f8f0).stop 2019-12-05T06:16:31.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.443+0000 7f0ce2e21700 1 -- 172.21.15.16:0/839018118 shutdown_connections 2019-12-05T06:16:31.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.443+0000 7f0ce2e21700 1 --2- 172.21.15.16:0/839018118 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc08f720 0x7f0cdc08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:31.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.443+0000 7f0ce2e21700 1 --2- 172.21.15.16:0/839018118 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0cdc08c6d0 0x7f0cdc08cac0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:31.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.444+0000 7f0ce2e21700 1 -- 172.21.15.16:0/839018118 shutdown_connections 2019-12-05T06:16:31.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.444+0000 7f0ce2e21700 1 -- 172.21.15.16:0/839018118 wait complete. 2019-12-05T06:16:31.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.444+0000 7f0ce2e21700 1 -- 172.21.15.16:0/839018118 >> 172.21.15.16:0/839018118 conn(0x7f0cdc061940 msgr2=0x7f0cdc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:31.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.449+0000 7f0ce2e21700 1 Processor -- start 2019-12-05T06:16:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.449+0000 7f0ce2e21700 1 -- start start 2019-12-05T06:16:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.451+0000 7f0ce2e21700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0cdc09ae80 0x7f0cdc133220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.451+0000 7f0ce2e21700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.451+0000 7f0ce2e21700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cdc06e7c0 con 0x7f0cdc132630 2019-12-05T06:16:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.451+0000 7f0ce2e21700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cdc098f20 con 0x7f0cdc09ae80 2019-12-05T06:16:31.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.451+0000 7f0ce2e21700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc134390 con 0x7f0cdc099eb0 2019-12-05T06:16:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.451+0000 7f0ce13be700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0cdc132630 0x7f0cdc132a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35500/0 (socket says 172.21.15.16:35500) 2019-12-05T06:16:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.451+0000 7f0ce13be700 1 -- 172.21.15.16:0/1718815627 learned_addr learned my addr 172.21.15.16:0/1718815627 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:16:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.451+0000 7f0ce0bbd700 1 --2- 172.21.15.16:0/1718815627 >> v2:172.21.15.76:3300/0 conn(0x7f0cdc09ae80 0x7f0cdc133220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.451+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:31.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.452+0000 7f0cd9ffb700 1 -- 172.21.15.16:0/1718815627 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2150076003 0 0) 0x7f0cd4007d20 con 0x7f0cdc132630 2019-12-05T06:16:31.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.452+0000 7f0ce1bbf700 1 -- 172.21.15.16:0/1718815627 >> v2:172.21.15.76:3300/0 conn(0x7f0cdc09ae80 msgr2=0x7f0cdc133220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:31.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.452+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> v2:172.21.15.76:3300/0 conn(0x7f0cdc09ae80 0x7f0cdc133220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.452+0000 7f0ce1bbf700 1 -- 172.21.15.16:0/1718815627 >> v1:172.21.15.16:6789/0 conn(0x7f0cdc132630 legacy=0x7f0cdc132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.452+0000 7f0ce1bbf700 1 -- 172.21.15.16:0/1718815627 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc0995b0 con 0x7f0cdc099eb0 2019-12-05T06:16:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.452+0000 7f0ce0bbd700 1 --2- 172.21.15.16:0/1718815627 >> v2:172.21.15.76:3300/0 conn(0x7f0cdc09ae80 0x7f0cdc133220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:16:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.452+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.453+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.453+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:31.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.453+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.453+0000 7f0cd9ffb700 1 -- 172.21.15.16:0/1718815627 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ccc009990 con 0x7f0cdc099eb0 2019-12-05T06:16:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.453+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.453+0000 7f0cd9ffb700 1 -- 172.21.15.16:0/1718815627 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ccc00eaa0 con 0x7f0cdc099eb0 2019-12-05T06:16:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.453+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.453+0000 7f0ce2e21700 1 -- 172.21.15.16:0/1718815627 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cdc0995b0 con 0x7f0cdc099eb0 2019-12-05T06:16:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.453+0000 7f0cd9ffb700 1 -- 172.21.15.16:0/1718815627 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0ccc00ec90 con 0x7f0cdc099eb0 2019-12-05T06:16:31.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.453+0000 7f0ce2e21700 1 -- 172.21.15.16:0/1718815627 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cdc0995b0 con 0x7f0cdc099eb0 2019-12-05T06:16:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.454+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.455+0000 7f0cd9ffb700 1 -- 172.21.15.16:0/1718815627 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0ccc00f510 con 0x7f0cdc099eb0 2019-12-05T06:16:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.455+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.455+0000 7f0cd9ffb700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0cbc023630 0x7f0cbc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.455+0000 7f0cd9ffb700 1 -- 172.21.15.16:0/1718815627 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f0ccc045d50 con 0x7f0cdc099eb0 2019-12-05T06:16:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.455+0000 7f0ce13be700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0cbc023630 0x7f0cbc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:31.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.456+0000 7f0ce13be700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0cbc023630 0x7f0cbc025ac0 crc :-1 s=READY pgs=206 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:31.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.470+0000 7f0ce2e21700 1 -- 172.21.15.16:0/1718815627 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cdc0995b0 con 0x7f0cdc099eb0 2019-12-05T06:16:31.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.479+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.479+0000 7f0cd9ffb700 1 -- 172.21.15.16:0/1718815627 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0ccc00f300 con 0x7f0cdc099eb0 2019-12-05T06:16:31.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.730+0000 7f0ce2e21700 1 -- 172.21.15.16:0/1718815627 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0cdc0995b0 con 0x7f0cdc099eb0 2019-12-05T06:16:31.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.732+0000 7f0ce1bbf700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=THROTTLE_DONE pgs=121 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.732+0000 7f0cd9ffb700 1 -- 172.21.15.16:0/1718815627 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f0ccc00f510 con 0x7f0cdc099eb0 2019-12-05T06:16:31.727 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 57 2019-12-05T06:16:31.727 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:31.727 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:58.002271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":28,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:31.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.736+0000 7f0ce2e21700 1 -- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0cbc023630 msgr2=0x7f0cbc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:31.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.736+0000 7f0ce2e21700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0cbc023630 0x7f0cbc025ac0 crc :-1 s=READY pgs=206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:31.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.736+0000 7f0ce2e21700 1 -- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 msgr2=0x7f0cdc09a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:31.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.736+0000 7f0ce2e21700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 secure :-1 s=READY pgs=121 cs=0 l=1 rx=0x7f0ccc010390 tx=0x7f0ccc010730).stop 2019-12-05T06:16:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.736+0000 7f0ce2e21700 1 -- 172.21.15.16:0/1718815627 shutdown_connections 2019-12-05T06:16:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.736+0000 7f0ce2e21700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0cbc023630 0x7f0cbc025ac0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.736+0000 7f0ce2e21700 1 --2- 172.21.15.16:0/1718815627 >> v2:172.21.15.76:3300/0 conn(0x7f0cdc09ae80 0x7f0cdc133220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:31.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.736+0000 7f0ce2e21700 1 --2- 172.21.15.16:0/1718815627 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0cdc099eb0 0x7f0cdc09a310 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:31.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.737+0000 7f0ce2e21700 1 -- 172.21.15.16:0/1718815627 shutdown_connections 2019-12-05T06:16:31.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.737+0000 7f0ce2e21700 1 -- 172.21.15.16:0/1718815627 wait complete. 2019-12-05T06:16:31.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:31.737+0000 7f0ce2e21700 1 -- 172.21.15.16:0/1718815627 >> 172.21.15.16:0/1718815627 conn(0x7f0cdc0714f0 msgr2=0x7f0cdc09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:31.740 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:33.742 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:34.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.018+0000 7f8505e18700 1 Processor -- start 2019-12-05T06:16:34.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.018+0000 7f8505e18700 1 -- start start 2019-12-05T06:16:34.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.020+0000 7f8505e18700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f850008c6d0 0x7f850008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:34.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.020+0000 7f8505e18700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850008fb80 con 0x7f850008f720 2019-12-05T06:16:34.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.020+0000 7f8505e18700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85000958e0 con 0x7f850008fef0 2019-12-05T06:16:34.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.020+0000 7f8505e18700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850008df90 con 0x7f850008c6d0 2019-12-05T06:16:34.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.020+0000 7f84fffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f850008c6d0 0x7f850008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:34.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.020+0000 7f8504bb6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f850008f720 0x7f85001533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35518/0 (socket says 172.21.15.16:35518) 2019-12-05T06:16:34.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.020+0000 7f84fffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f850008c6d0 0x7f850008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50346/0 (socket says 172.21.15.16:50346) 2019-12-05T06:16:34.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.020+0000 7f8504bb6700 1 -- 172.21.15.16:0/3216764145 learned_addr learned my addr 172.21.15.16:0/3216764145 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:16:34.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.021+0000 7f84fffff700 1 -- 172.21.15.16:0/3216764145 >> v1:172.21.15.76:6789/0 conn(0x7f850008fef0 legacy=0x7f850008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:34.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.021+0000 7f84fffff700 1 -- 172.21.15.16:0/3216764145 >> v1:172.21.15.16:6789/0 conn(0x7f850008f720 legacy=0x7f85001533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:34.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.021+0000 7f84feffd700 1 -- 172.21.15.16:0/3216764145 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3109891272 0 0) 0x7f84f80014d0 con 0x7f850008f720 2019-12-05T06:16:34.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.021+0000 7f84fffff700 1 -- 172.21.15.16:0/3216764145 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850008e620 con 0x7f850008c6d0 2019-12-05T06:16:34.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.021+0000 7f84fffff700 1 --2- 172.21.15.16:0/3216764145 >> v2:172.21.15.16:3300/0 conn(0x7f850008c6d0 0x7f850008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f84f000af50 tx=0x7f84f000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.022+0000 7f84fffff700 1 --2- 172.21.15.16:0/3216764145 >> v2:172.21.15.16:3300/0 conn(0x7f850008c6d0 0x7f850008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f84f000af50 tx=0x7f84f000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.022+0000 7f84fffff700 1 --2- 172.21.15.16:0/3216764145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f850008c6d0 0x7f850008cac0 secure :-1 s=READY pgs=452 cs=0 l=1 rx=0x7f84f000af50 tx=0x7f84f000b760).ready entity=mon.0 client_cookie=ceea00aa8dfed584 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:34.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.022+0000 7f84fffff700 1 --2- 172.21.15.16:0/3216764145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f850008c6d0 0x7f850008cac0 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7f84f000af50 tx=0x7f84f000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.022+0000 7f84feffd700 1 -- 172.21.15.16:0/3216764145 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84f0000960 con 0x7f850008c6d0 2019-12-05T06:16:34.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.022+0000 7f84fffff700 1 --2- 172.21.15.16:0/3216764145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f850008c6d0 0x7f850008cac0 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7f84f000af50 tx=0x7f84f000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.022+0000 7f84fffff700 1 --2- 172.21.15.16:0/3216764145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f850008c6d0 0x7f850008cac0 secure :-1 s=THROTTLE_DONE pgs=452 cs=0 l=1 rx=0x7f84f000af50 tx=0x7f84f000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.022+0000 7f84feffd700 1 -- 172.21.15.16:0/3216764145 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f0009860 con 0x7f850008c6d0 2019-12-05T06:16:34.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.022+0000 7f84feffd700 1 -- 172.21.15.16:0/3216764145 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84f0009f30 con 0x7f850008c6d0 2019-12-05T06:16:34.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.022+0000 7f8505e18700 1 -- 172.21.15.16:0/3216764145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f850008c6d0 msgr2=0x7f850008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:34.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.022+0000 7f8505e18700 1 --2- 172.21.15.16:0/3216764145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f850008c6d0 0x7f850008cac0 secure :-1 s=READY pgs=452 cs=0 l=1 rx=0x7f84f000af50 tx=0x7f84f000b760).stop 2019-12-05T06:16:34.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.024+0000 7f8505e18700 1 -- 172.21.15.16:0/3216764145 shutdown_connections 2019-12-05T06:16:34.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.024+0000 7f8505e18700 1 --2- 172.21.15.16:0/3216764145 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f850008c6d0 0x7f850008cac0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:34.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.024+0000 7f8505e18700 1 -- 172.21.15.16:0/3216764145 shutdown_connections 2019-12-05T06:16:34.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.024+0000 7f8505e18700 1 -- 172.21.15.16:0/3216764145 wait complete. 2019-12-05T06:16:34.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.024+0000 7f8505e18700 1 -- 172.21.15.16:0/3216764145 >> 172.21.15.16:0/3216764145 conn(0x7f8500061940 msgr2=0x7f8500096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:34.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.029+0000 7f8505e18700 1 Processor -- start 2019-12-05T06:16:34.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.030+0000 7f8505e18700 1 -- start start 2019-12-05T06:16:34.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.032+0000 7f8505e18700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8500132630 0x7f8500132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:34.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.032+0000 7f8505e18700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8500099eb0 0x7f850009a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:34.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.032+0000 7f8505e18700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850006e7c0 con 0x7f850009ae80 2019-12-05T06:16:34.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.032+0000 7f8505e18700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8500098f20 con 0x7f8500132630 2019-12-05T06:16:34.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.032+0000 7f8505e18700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8500134390 con 0x7f8500099eb0 2019-12-05T06:16:34.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.032+0000 7f84fffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8500132630 0x7f8500132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:34.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.032+0000 7f84ff7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f850009ae80 0x7f8500132f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35522/0 (socket says 172.21.15.16:35522) 2019-12-05T06:16:34.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.032+0000 7f84ff7fe700 1 -- 172.21.15.16:0/3499713453 learned_addr learned my addr 172.21.15.16:0/3499713453 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:16:34.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.032+0000 7f84fffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8500132630 0x7f8500132a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50352/0 (socket says 172.21.15.16:50352) 2019-12-05T06:16:34.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.032+0000 7f8504bb6700 1 --2- 172.21.15.16:0/3499713453 >> v2:172.21.15.76:3300/0 conn(0x7f8500099eb0 0x7f850009a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:34.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fffff700 1 -- 172.21.15.16:0/3499713453 >> v2:172.21.15.76:3300/0 conn(0x7f8500099eb0 msgr2=0x7f850009a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:34.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> v2:172.21.15.76:3300/0 conn(0x7f8500099eb0 0x7f850009a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:34.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fffff700 1 -- 172.21.15.16:0/3499713453 >> v1:172.21.15.16:6789/0 conn(0x7f850009ae80 legacy=0x7f8500132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:34.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fffff700 1 -- 172.21.15.16:0/3499713453 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85000995b0 con 0x7f8500132630 2019-12-05T06:16:34.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fcff9700 1 -- 172.21.15.16:0/3499713453 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905314064 0 0) 0x7f84f4000b40 con 0x7f850009ae80 2019-12-05T06:16:34.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> v2:172.21.15.16:3300/0 conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> v2:172.21.15.16:3300/0 conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=READY pgs=454 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:34.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=THROTTLE_DONE pgs=454 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fcff9700 1 -- 172.21.15.16:0/3499713453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84f0009790 con 0x7f8500132630 2019-12-05T06:16:34.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=THROTTLE_DONE pgs=454 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=THROTTLE_DONE pgs=454 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fcff9700 1 -- 172.21.15.16:0/3499713453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84f000b5d0 con 0x7f8500132630 2019-12-05T06:16:34.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f8505e18700 1 -- 172.21.15.16:0/3499713453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85000995b0 con 0x7f8500132630 2019-12-05T06:16:34.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.033+0000 7f84fcff9700 1 -- 172.21.15.16:0/3499713453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84f0009d70 con 0x7f8500132630 2019-12-05T06:16:34.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.034+0000 7f8505e18700 1 -- 172.21.15.16:0/3499713453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85000995b0 con 0x7f8500132630 2019-12-05T06:16:34.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.035+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=THROTTLE_DONE pgs=454 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.035+0000 7f84fcff9700 1 -- 172.21.15.16:0/3499713453 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f84f000a1a0 con 0x7f8500132630 2019-12-05T06:16:34.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.035+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=THROTTLE_DONE pgs=454 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.036+0000 7f84fcff9700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84e0023630 0x7f84e0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:34.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.036+0000 7f84fcff9700 1 -- 172.21.15.16:0/3499713453 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f84f0045d60 con 0x7f8500132630 2019-12-05T06:16:34.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.037+0000 7f84ff7fe700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84e0023630 0x7f84e0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:34.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.037+0000 7f84ff7fe700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84e0023630 0x7f84e0025ac0 crc :-1 s=READY pgs=207 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:34.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.049+0000 7f8505e18700 1 -- 172.21.15.16:0/3499713453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85000995b0 con 0x7f8500132630 2019-12-05T06:16:34.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.057+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=THROTTLE_DONE pgs=454 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.057+0000 7f84fcff9700 1 -- 172.21.15.16:0/3499713453 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f84f0009f90 con 0x7f8500132630 2019-12-05T06:16:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.303+0000 7f8505e18700 1 -- 172.21.15.16:0/3499713453 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85000995b0 con 0x7f8500132630 2019-12-05T06:16:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.304+0000 7f84fffff700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=THROTTLE_DONE pgs=454 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.304+0000 7f84fcff9700 1 -- 172.21.15.16:0/3499713453 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f84f000a1a0 con 0x7f8500132630 2019-12-05T06:16:34.303 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 57 2019-12-05T06:16:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.306+0000 7f8505e18700 1 -- 172.21.15.16:0/3499713453 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84e0023630 msgr2=0x7f84e0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.306+0000 7f8505e18700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84e0023630 0x7f84e0025ac0 crc :-1 s=READY pgs=207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.306+0000 7f8505e18700 1 -- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 msgr2=0x7f8500132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.306+0000 7f8505e18700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 0x7f8500132a20 secure :-1 s=READY pgs=454 cs=0 l=1 rx=0x7f84f00103d0 tx=0x7f84f0010200).stop 2019-12-05T06:16:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.306+0000 7f8505e18700 1 -- 172.21.15.16:0/3499713453 shutdown_connections 2019-12-05T06:16:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.306+0000 7f8505e18700 1 --2- 172.21.15.16:0/3499713453 >> v2:172.21.15.76:3300/0 conn(0x7f8500099eb0 0x7f850009a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:34.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.306+0000 7f8505e18700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84e0023630 0x7f84e0025ac0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.306+0000 7f8505e18700 1 --2- 172.21.15.16:0/3499713453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8500132630 0x7f8500132a20 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.307+0000 7f8505e18700 1 -- 172.21.15.16:0/3499713453 shutdown_connections 2019-12-05T06:16:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.307+0000 7f8505e18700 1 -- 172.21.15.16:0/3499713453 wait complete. 2019-12-05T06:16:34.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:34.307+0000 7f8505e18700 1 -- 172.21.15.16:0/3499713453 >> 172.21.15.16:0/3499713453 conn(0x7f85000714f0 msgr2=0x7f850009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:34.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:34.305 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:58.002271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":28,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:34.311 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:36.311 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:36.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.624+0000 7f04a292d700 1 Processor -- start 2019-12-05T06:16:36.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.624+0000 7f04a292d700 1 -- start start 2019-12-05T06:16:36.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.626+0000 7f04a292d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f049c08fef0 0x7f049c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.626+0000 7f04a292d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049c0958e0 con 0x7f049c08d2c0 2019-12-05T06:16:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.626+0000 7f04a292d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049c0629e0 con 0x7f049c08c6d0 2019-12-05T06:16:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.626+0000 7f04a292d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049c08df90 con 0x7f049c08fef0 2019-12-05T06:16:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.626+0000 7f04a16cb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f049c08fef0 0x7f049c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.627+0000 7f04a16cb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f049c08fef0 0x7f049c08f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50366/0 (socket says 172.21.15.16:50366) 2019-12-05T06:16:36.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.627+0000 7f04a16cb700 1 -- 172.21.15.16:0/48619453 learned_addr learned my addr 172.21.15.16:0/48619453 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:36.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.627+0000 7f04a16cb700 1 -- 172.21.15.16:0/48619453 >> v1:172.21.15.76:6789/0 conn(0x7f049c08c6d0 legacy=0x7f049c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:36.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.627+0000 7f04a16cb700 1 -- 172.21.15.16:0/48619453 >> v1:172.21.15.16:6789/0 conn(0x7f049c08d2c0 legacy=0x7f049c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:36.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.627+0000 7f049b7fe700 1 -- 172.21.15.16:0/48619453 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2717516764 0 0) 0x7f0494001400 con 0x7f049c08c6d0 2019-12-05T06:16:36.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.627+0000 7f04a16cb700 1 -- 172.21.15.16:0/48619453 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c08e620 con 0x7f049c08fef0 2019-12-05T06:16:36.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f04a16cb700 1 --2- 172.21.15.16:0/48619453 >> v2:172.21.15.16:3300/0 conn(0x7f049c08fef0 0x7f049c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f048400f730 tx=0x7f048400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f04a16cb700 1 --2- 172.21.15.16:0/48619453 >> v2:172.21.15.16:3300/0 conn(0x7f049c08fef0 0x7f049c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f048400f730 tx=0x7f048400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f04a16cb700 1 --2- 172.21.15.16:0/48619453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f049c08fef0 0x7f049c08f720 secure :-1 s=READY pgs=456 cs=0 l=1 rx=0x7f048400f730 tx=0x7f048400fad0).ready entity=mon.0 client_cookie=67ac4ebba41a6749 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:36.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f04a16cb700 1 --2- 172.21.15.16:0/48619453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f049c08fef0 0x7f049c08f720 secure :-1 s=THROTTLE_DONE pgs=456 cs=0 l=1 rx=0x7f048400f730 tx=0x7f048400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f049b7fe700 1 -- 172.21.15.16:0/48619453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04840010e0 con 0x7f049c08fef0 2019-12-05T06:16:36.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f04a16cb700 1 --2- 172.21.15.16:0/48619453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f049c08fef0 0x7f049c08f720 secure :-1 s=THROTTLE_DONE pgs=456 cs=0 l=1 rx=0x7f048400f730 tx=0x7f048400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f04a16cb700 1 --2- 172.21.15.16:0/48619453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f049c08fef0 0x7f049c08f720 secure :-1 s=THROTTLE_DONE pgs=456 cs=0 l=1 rx=0x7f048400f730 tx=0x7f048400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f049b7fe700 1 -- 172.21.15.16:0/48619453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0484000a70 con 0x7f049c08fef0 2019-12-05T06:16:36.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f049b7fe700 1 -- 172.21.15.16:0/48619453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04840016c0 con 0x7f049c08fef0 2019-12-05T06:16:36.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f04a292d700 1 -- 172.21.15.16:0/48619453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f049c08fef0 msgr2=0x7f049c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:36.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.628+0000 7f04a292d700 1 --2- 172.21.15.16:0/48619453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f049c08fef0 0x7f049c08f720 secure :-1 s=READY pgs=456 cs=0 l=1 rx=0x7f048400f730 tx=0x7f048400fad0).stop 2019-12-05T06:16:36.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.630+0000 7f04a292d700 1 -- 172.21.15.16:0/48619453 shutdown_connections 2019-12-05T06:16:36.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.630+0000 7f04a292d700 1 --2- 172.21.15.16:0/48619453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f049c08fef0 0x7f049c08f720 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:36.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.630+0000 7f04a292d700 1 -- 172.21.15.16:0/48619453 shutdown_connections 2019-12-05T06:16:36.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.630+0000 7f04a292d700 1 -- 172.21.15.16:0/48619453 wait complete. 2019-12-05T06:16:36.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.630+0000 7f04a292d700 1 -- 172.21.15.16:0/48619453 >> 172.21.15.16:0/48619453 conn(0x7f049c061940 msgr2=0x7f049c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:36.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.636+0000 7f04a292d700 1 Processor -- start 2019-12-05T06:16:36.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.636+0000 7f04a292d700 1 -- start start 2019-12-05T06:16:36.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.639+0000 7f04a292d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f049c14ca10 0x7f049c0869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:36.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.639+0000 7f04a292d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:36.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.639+0000 7f04a292d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f049c09e030 con 0x7f049c0879b0 2019-12-05T06:16:36.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.639+0000 7f04a292d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f049c085a50 con 0x7f049c14ca10 2019-12-05T06:16:36.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.639+0000 7f04a292d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f049c085020 con 0x7f049c09ce90 2019-12-05T06:16:36.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.639+0000 7f04a16cb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:36.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.639+0000 7f04a0eca700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f049c14ca10 0x7f049c0869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:36.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.639+0000 7f04a16cb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:54362/0 (socket says 172.21.15.16:54362) 2019-12-05T06:16:36.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.639+0000 7f04a16cb700 1 -- 172.21.15.16:0/1344629901 learned_addr learned my addr 172.21.15.16:0/1344629901 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:36.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.640+0000 7f04a16cb700 1 -- 172.21.15.16:0/1344629901 >> v2:172.21.15.76:3300/0 conn(0x7f049c14ca10 msgr2=0x7f049c0869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:36.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.640+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> v2:172.21.15.76:3300/0 conn(0x7f049c14ca10 0x7f049c0869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:36.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.640+0000 7f04a16cb700 1 -- 172.21.15.16:0/1344629901 >> v1:172.21.15.76:6789/0 conn(0x7f049c0879b0 legacy=0x7f049c086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:36.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.640+0000 7f04a16cb700 1 -- 172.21.15.16:0/1344629901 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f049c0860e0 con 0x7f049c09ce90 2019-12-05T06:16:36.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.640+0000 7f04a0eca700 1 --2- 172.21.15.16:0/1344629901 >> v2:172.21.15.76:3300/0 conn(0x7f049c14ca10 0x7f049c0869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:16:36.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.640+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.641+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.641+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:36.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.641+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.641+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.641+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.641+0000 7f04997fa700 1 -- 172.21.15.16:0/1344629901 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0484001290 con 0x7f049c09ce90 2019-12-05T06:16:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.641+0000 7f04997fa700 1 -- 172.21.15.16:0/1344629901 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0484000eb0 con 0x7f049c09ce90 2019-12-05T06:16:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.641+0000 7f04a292d700 1 -- 172.21.15.16:0/1344629901 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049c0860e0 con 0x7f049c09ce90 2019-12-05T06:16:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.641+0000 7f04997fa700 1 -- 172.21.15.16:0/1344629901 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0484001870 con 0x7f049c09ce90 2019-12-05T06:16:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.641+0000 7f04a292d700 1 -- 172.21.15.16:0/1344629901 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049c0860e0 con 0x7f049c09ce90 2019-12-05T06:16:36.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.642+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.644+0000 7f04997fa700 1 -- 172.21.15.16:0/1344629901 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f04840010a0 con 0x7f049c09ce90 2019-12-05T06:16:36.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.644+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.644+0000 7f04997fa700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0480023630 0x7f0480025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:36.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.644+0000 7f04997fa700 1 -- 172.21.15.16:0/1344629901 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 6461+0+0 (secure 0 0 0) 0x7f0484045e10 con 0x7f049c09ce90 2019-12-05T06:16:36.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.644+0000 7f04a0eca700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0480023630 0x7f0480025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:36.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.645+0000 7f04a0eca700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0480023630 0x7f0480025ac0 crc :-1 s=READY pgs=208 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:36.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.655+0000 7f04a292d700 1 -- 172.21.15.16:0/1344629901 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049c0860e0 con 0x7f049c09ce90 2019-12-05T06:16:36.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.665+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.665+0000 7f04997fa700 1 -- 172.21.15.16:0/1344629901 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f04840077e0 con 0x7f049c09ce90 2019-12-05T06:16:36.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.921+0000 7f04a292d700 1 -- 172.21.15.16:0/1344629901 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f049c0860e0 con 0x7f049c09ce90 2019-12-05T06:16:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.923+0000 7f04a16cb700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=THROTTLE_DONE pgs=123 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:36.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.923+0000 7f04997fa700 1 -- 172.21.15.16:0/1344629901 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f0484001930 con 0x7f049c09ce90 2019-12-05T06:16:36.917 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:36.918 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":57,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5295,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":57}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:15:58.002271+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":54,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5148},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5148":{"gid":5148,"name":"b","rank":0,"incarnation":53,"state":"up:active","state_seq":28,"addr":"172.21.15.76:6835/3345019712","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3345019712},{"type":"v1","addr":"172.21.15.76:6835","nonce":3345019712}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:36.918 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 57 2019-12-05T06:16:36.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.926+0000 7f04a292d700 1 -- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0480023630 msgr2=0x7f0480025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:36.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.926+0000 7f04a292d700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0480023630 0x7f0480025ac0 crc :-1 s=READY pgs=208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:36.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.926+0000 7f04a292d700 1 -- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 msgr2=0x7f049c19a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:36.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.926+0000 7f04a292d700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 secure :-1 s=READY pgs=123 cs=0 l=1 rx=0x7f04840104e0 tx=0x7f0484010880).stop 2019-12-05T06:16:36.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.927+0000 7f04a292d700 1 -- 172.21.15.16:0/1344629901 shutdown_connections 2019-12-05T06:16:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.927+0000 7f04a292d700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0480023630 0x7f0480025ac0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.927+0000 7f04a292d700 1 --2- 172.21.15.16:0/1344629901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f049c09ce90 0x7f049c19a400 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.927+0000 7f04a292d700 1 --2- 172.21.15.16:0/1344629901 >> v2:172.21.15.76:3300/0 conn(0x7f049c14ca10 0x7f049c0869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.927+0000 7f04a292d700 1 -- 172.21.15.16:0/1344629901 shutdown_connections 2019-12-05T06:16:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.927+0000 7f04a292d700 1 -- 172.21.15.16:0/1344629901 wait complete. 2019-12-05T06:16:36.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:36.927+0000 7f04a292d700 1 -- 172.21.15.16:0/1344629901 >> 172.21.15.16:0/1344629901 conn(0x7f049c098480 msgr2=0x7f049c071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:36.938 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:16:38.939 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:39.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.116+0000 7ff790e3c700 1 Processor -- start 2019-12-05T06:16:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.116+0000 7ff790e3c700 1 -- start start 2019-12-05T06:16:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.117+0000 7ff790e3c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff78c08c6d0 0x7ff78c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.117+0000 7ff790e3c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff78c08fef0 0x7ff78c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.117+0000 7ff790e3c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78c0958e0 con 0x7ff78c08ef20 2019-12-05T06:16:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.117+0000 7ff790e3c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff78c08df90 con 0x7ff78c08c6d0 2019-12-05T06:16:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.117+0000 7ff790e3c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff78c0a3f70 con 0x7ff78c08fef0 2019-12-05T06:16:39.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.117+0000 7ff78b59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff78c08ef20 0x7ff78c08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35564/0 (socket says 172.21.15.16:35564) 2019-12-05T06:16:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.117+0000 7ff78b59e700 1 -- 172.21.15.16:0/984331970 learned_addr learned my addr 172.21.15.16:0/984331970 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:16:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/984331970 >> v2:172.21.15.76:3300/0 conn(0x7ff78c08c6d0 0x7ff78c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff78a59c700 1 --2- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff78c08fef0 0x7ff78c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff789d9b700 1 -- 172.21.15.16:0/984331970 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350265808 0 0) 0x7ff7840014d0 con 0x7ff78c08ef20 2019-12-05T06:16:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff789d9b700 1 -- 172.21.15.16:0/984331970 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff76c002e80 con 0x7ff78c08ef20 2019-12-05T06:16:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff78ad9d700 1 -- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff78c08fef0 msgr2=0x7ff78c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:39.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff78c08fef0 0x7ff78c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff78ad9d700 1 -- 172.21.15.16:0/984331970 >> v1:172.21.15.16:6789/0 conn(0x7ff78c08ef20 legacy=0x7ff78c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff789d9b700 1 -- 172.21.15.16:0/984331970 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3181559189 0 0) 0x7ff7840014b0 con 0x7ff78c08ef20 2019-12-05T06:16:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff78ad9d700 1 -- 172.21.15.16:0/984331970 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c08d540 con 0x7ff78c08c6d0 2019-12-05T06:16:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/984331970 >> v2:172.21.15.76:3300/0 conn(0x7ff78c08c6d0 0x7ff78c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff77c00af50 tx=0x7ff77c00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/984331970 >> v2:172.21.15.76:3300/0 conn(0x7ff78c08c6d0 0x7ff78c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff77c00af50 tx=0x7ff77c00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78c08c6d0 0x7ff78c08cac0 secure :-1 s=READY pgs=351 cs=0 l=1 rx=0x7ff77c00af50 tx=0x7ff77c00b250).ready entity=mon.1 client_cookie=73a705f666569ba1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:39.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.118+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78c08c6d0 0x7ff78c08cac0 secure :-1 s=THROTTLE_DONE pgs=351 cs=0 l=1 rx=0x7ff77c00af50 tx=0x7ff77c00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff789d9b700 1 -- 172.21.15.16:0/984331970 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff77c007210 con 0x7ff78c08c6d0 2019-12-05T06:16:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78c08c6d0 0x7ff78c08cac0 secure :-1 s=THROTTLE_DONE pgs=351 cs=0 l=1 rx=0x7ff77c00af50 tx=0x7ff77c00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78c08c6d0 0x7ff78c08cac0 secure :-1 s=THROTTLE_DONE pgs=351 cs=0 l=1 rx=0x7ff77c00af50 tx=0x7ff77c00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff789d9b700 1 -- 172.21.15.16:0/984331970 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff77c0099b0 con 0x7ff78c08c6d0 2019-12-05T06:16:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff789d9b700 1 -- 172.21.15.16:0/984331970 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff77c009ba0 con 0x7ff78c08c6d0 2019-12-05T06:16:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff790e3c700 1 -- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78c08c6d0 msgr2=0x7ff78c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff790e3c700 1 --2- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78c08c6d0 0x7ff78c08cac0 secure :-1 s=READY pgs=351 cs=0 l=1 rx=0x7ff77c00af50 tx=0x7ff77c00b250).stop 2019-12-05T06:16:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff790e3c700 1 -- 172.21.15.16:0/984331970 shutdown_connections 2019-12-05T06:16:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff790e3c700 1 --2- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff78c08fef0 0x7ff78c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff790e3c700 1 --2- 172.21.15.16:0/984331970 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff78c08c6d0 0x7ff78c08cac0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff790e3c700 1 -- 172.21.15.16:0/984331970 shutdown_connections 2019-12-05T06:16:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff790e3c700 1 -- 172.21.15.16:0/984331970 wait complete. 2019-12-05T06:16:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.119+0000 7ff790e3c700 1 -- 172.21.15.16:0/984331970 >> 172.21.15.16:0/984331970 conn(0x7ff78c061940 msgr2=0x7ff78c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.122+0000 7ff790e3c700 1 Processor -- start 2019-12-05T06:16:39.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.122+0000 7ff790e3c700 1 -- start start 2019-12-05T06:16:39.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff790e3c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff78c1325e0 0x7ff78c12b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:39.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff790e3c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff78c154c60 0x7ff78c157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:39.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff790e3c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff78c128b20 con 0x7ff78c12c740 2019-12-05T06:16:39.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff790e3c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff78c12a7e0 con 0x7ff78c154c60 2019-12-05T06:16:39.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff790e3c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff78c129db0 con 0x7ff78c1325e0 2019-12-05T06:16:39.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff78c154c60 0x7ff78c157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff78c154c60 0x7ff78c157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50404/0 (socket says 172.21.15.16:50404) 2019-12-05T06:16:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 -- 172.21.15.16:0/1680767942 learned_addr learned my addr 172.21.15.16:0/1680767942 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/1680767942 >> v2:172.21.15.76:3300/0 conn(0x7ff78c1325e0 0x7ff78c12b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 -- 172.21.15.16:0/1680767942 >> v2:172.21.15.76:3300/0 conn(0x7ff78c1325e0 msgr2=0x7ff78c12b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> v2:172.21.15.76:3300/0 conn(0x7ff78c1325e0 0x7ff78c12b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 -- 172.21.15.16:0/1680767942 >> v1:172.21.15.76:6789/0 conn(0x7ff78c12c740 legacy=0x7ff78c12bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 -- 172.21.15.16:0/1680767942 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff78c12ae70 con 0x7ff78c154c60 2019-12-05T06:16:39.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> v2:172.21.15.16:3300/0 conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> v2:172.21.15.16:3300/0 conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=READY pgs=458 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:39.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=THROTTLE_DONE pgs=458 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=THROTTLE_DONE pgs=458 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=THROTTLE_DONE pgs=458 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff7837fe700 1 -- 172.21.15.16:0/1680767942 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff784000cc0 con 0x7ff78c154c60 2019-12-05T06:16:39.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff7837fe700 1 -- 172.21.15.16:0/1680767942 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7840045c0 con 0x7ff78c154c60 2019-12-05T06:16:39.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.123+0000 7ff790e3c700 1 -- 172.21.15.16:0/1680767942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff78c12ae70 con 0x7ff78c154c60 2019-12-05T06:16:39.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.124+0000 7ff7837fe700 1 -- 172.21.15.16:0/1680767942 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff784004c90 con 0x7ff78c154c60 2019-12-05T06:16:39.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.124+0000 7ff790e3c700 1 -- 172.21.15.16:0/1680767942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff78c12ae70 con 0x7ff78c154c60 2019-12-05T06:16:39.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.124+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=THROTTLE_DONE pgs=458 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.124+0000 7ff7837fe700 1 -- 172.21.15.16:0/1680767942 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff7840047b0 con 0x7ff78c154c60 2019-12-05T06:16:39.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.124+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=THROTTLE_DONE pgs=458 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.125+0000 7ff7837fe700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff764023630 0x7ff764025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:39.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.125+0000 7ff7837fe700 1 -- 172.21.15.16:0/1680767942 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7ff7840461c0 con 0x7ff78c154c60 2019-12-05T06:16:39.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.125+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff764023630 0x7ff764025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:39.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.125+0000 7ff78ad9d700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff764023630 0x7ff764025ac0 crc :-1 s=READY pgs=209 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:39.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.132+0000 7ff790e3c700 1 -- 172.21.15.16:0/1680767942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff78c12ae70 con 0x7ff78c154c60 2019-12-05T06:16:39.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.135+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=THROTTLE_DONE pgs=458 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.135+0000 7ff7837fe700 1 -- 172.21.15.16:0/1680767942 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff784001660 con 0x7ff78c154c60 2019-12-05T06:16:39.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.333+0000 7ff790e3c700 1 -- 172.21.15.16:0/1680767942 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff78c12ae70 con 0x7ff78c154c60 2019-12-05T06:16:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.334+0000 7ff78b59e700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=THROTTLE_DONE pgs=458 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.334+0000 7ff7837fe700 1 -- 172.21.15.16:0/1680767942 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+1771 (secure 0 0 0) 0x7ff784000f20 con 0x7ff78c154c60 2019-12-05T06:16:39.328 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 58 2019-12-05T06:16:39.328 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:39.328 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:38.047498+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:39.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 -- 172.21.15.16:0/1680767942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff764023630 msgr2=0x7ff764025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff764023630 0x7ff764025ac0 crc :-1 s=READY pgs=209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 -- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 msgr2=0x7ff78c157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 0x7ff78c157060 secure :-1 s=READY pgs=458 cs=0 l=1 rx=0x7ff7840107b0 tx=0x7ff7840105e0).stop 2019-12-05T06:16:39.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 -- 172.21.15.16:0/1680767942 shutdown_connections 2019-12-05T06:16:39.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff764023630 0x7ff764025ac0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 --2- 172.21.15.16:0/1680767942 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff78c154c60 0x7ff78c157060 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 --2- 172.21.15.16:0/1680767942 >> v2:172.21.15.76:3300/0 conn(0x7ff78c1325e0 0x7ff78c12b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 -- 172.21.15.16:0/1680767942 shutdown_connections 2019-12-05T06:16:39.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 -- 172.21.15.16:0/1680767942 wait complete. 2019-12-05T06:16:39.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.337+0000 7ff790e3c700 1 -- 172.21.15.16:0/1680767942 >> 172.21.15.16:0/1680767942 conn(0x7ff78c0a2850 msgr2=0x7ff78c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:39.340 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:16:39.340 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:16:39.341 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:39.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.487+0000 7f7d93f25700 1 Processor -- start 2019-12-05T06:16:39.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.487+0000 7f7d93f25700 1 -- start start 2019-12-05T06:16:39.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d93f25700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d8c08fef0 0x7f7d8c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:39.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d93f25700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8c0958e0 con 0x7f7d8c08c6d0 2019-12-05T06:16:39.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d93f25700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8c0629e0 con 0x7f7d8c08d2c0 2019-12-05T06:16:39.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d93f25700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8c08df90 con 0x7f7d8c08fef0 2019-12-05T06:16:39.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d92cc3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d8c08fef0 0x7f7d8c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:39.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d92cc3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d8c08fef0 0x7f7d8c08f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50412/0 (socket says 172.21.15.16:50412) 2019-12-05T06:16:39.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d92cc3700 1 -- 172.21.15.16:0/3944809303 learned_addr learned my addr 172.21.15.16:0/3944809303 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:39.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d924c2700 1 --1- 172.21.15.16:0/3944809303 >> v1:172.21.15.16:6789/0 conn(0x7f7d8c08c6d0 0x7f7d8c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35576/0 (socket says 172.21.15.16:35576) 2019-12-05T06:16:39.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d92cc3700 1 -- 172.21.15.16:0/3944809303 >> v1:172.21.15.76:6789/0 conn(0x7f7d8c08d2c0 legacy=0x7f7d8c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:39.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d92cc3700 1 -- 172.21.15.16:0/3944809303 >> v1:172.21.15.16:6789/0 conn(0x7f7d8c08c6d0 legacy=0x7f7d8c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d92cc3700 1 -- 172.21.15.16:0/3944809303 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8c08e620 con 0x7f7d8c08fef0 2019-12-05T06:16:39.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.488+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/3944809303 >> v2:172.21.15.16:3300/0 conn(0x7f7d8c08fef0 0x7f7d8c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7d8400fab0 tx=0x7f7d8400fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.489+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/3944809303 >> v2:172.21.15.16:3300/0 conn(0x7f7d8c08fef0 0x7f7d8c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7d8400fab0 tx=0x7f7d8400fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.489+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/3944809303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c08fef0 0x7f7d8c08f720 secure :-1 s=READY pgs=460 cs=0 l=1 rx=0x7f7d8400fab0 tx=0x7f7d8400fe50).ready entity=mon.0 client_cookie=40a1afb1eba5e6b3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.489+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/3944809303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c08fef0 0x7f7d8c08f720 secure :-1 s=THROTTLE_DONE pgs=460 cs=0 l=1 rx=0x7f7d8400fab0 tx=0x7f7d8400fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.489+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/3944809303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c08fef0 0x7f7d8c08f720 secure :-1 s=THROTTLE_DONE pgs=460 cs=0 l=1 rx=0x7f7d8400fab0 tx=0x7f7d8400fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.489+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/3944809303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c08fef0 0x7f7d8c08f720 secure :-1 s=THROTTLE_DONE pgs=460 cs=0 l=1 rx=0x7f7d8400fab0 tx=0x7f7d8400fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.489+0000 7f7d914c0700 1 -- 172.21.15.16:0/3944809303 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d84001460 con 0x7f7d8c08fef0 2019-12-05T06:16:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.489+0000 7f7d914c0700 1 -- 172.21.15.16:0/3944809303 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d84000df0 con 0x7f7d8c08fef0 2019-12-05T06:16:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.489+0000 7f7d914c0700 1 -- 172.21.15.16:0/3944809303 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d84001a40 con 0x7f7d8c08fef0 2019-12-05T06:16:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.489+0000 7f7d93f25700 1 -- 172.21.15.16:0/3944809303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c08fef0 msgr2=0x7f7d8c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.489+0000 7f7d93f25700 1 --2- 172.21.15.16:0/3944809303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c08fef0 0x7f7d8c08f720 secure :-1 s=READY pgs=460 cs=0 l=1 rx=0x7f7d8400fab0 tx=0x7f7d8400fe50).stop 2019-12-05T06:16:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.490+0000 7f7d93f25700 1 -- 172.21.15.16:0/3944809303 shutdown_connections 2019-12-05T06:16:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.490+0000 7f7d93f25700 1 --2- 172.21.15.16:0/3944809303 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c08fef0 0x7f7d8c08f720 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.490+0000 7f7d93f25700 1 -- 172.21.15.16:0/3944809303 shutdown_connections 2019-12-05T06:16:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.490+0000 7f7d93f25700 1 -- 172.21.15.16:0/3944809303 wait complete. 2019-12-05T06:16:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.490+0000 7f7d93f25700 1 -- 172.21.15.16:0/3944809303 >> 172.21.15.16:0/3944809303 conn(0x7f7d8c061940 msgr2=0x7f7d8c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.492+0000 7f7d93f25700 1 Processor -- start 2019-12-05T06:16:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.492+0000 7f7d93f25700 1 -- start start 2019-12-05T06:16:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d93f25700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7d8c0879b0 0x7f7d8c0871e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:39.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d93f25700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d8c09ce90 0x7f7d8c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:39.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d93f25700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d8c09e030 con 0x7f7d8c14ca10 2019-12-05T06:16:39.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d93f25700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8c085a50 con 0x7f7d8c09ce90 2019-12-05T06:16:39.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d93f25700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d8c085020 con 0x7f7d8c0879b0 2019-12-05T06:16:39.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d92cc3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d8c09ce90 0x7f7d8c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:39.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d92cc3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d8c09ce90 0x7f7d8c19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50416/0 (socket says 172.21.15.16:50416) 2019-12-05T06:16:39.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d92cc3700 1 -- 172.21.15.16:0/2366617605 learned_addr learned my addr 172.21.15.16:0/2366617605 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d91cc1700 1 --2- 172.21.15.16:0/2366617605 >> v2:172.21.15.76:3300/0 conn(0x7f7d8c0879b0 0x7f7d8c0871e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d92cc3700 1 -- 172.21.15.16:0/2366617605 >> v2:172.21.15.76:3300/0 conn(0x7f7d8c0879b0 msgr2=0x7f7d8c0871e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> v2:172.21.15.76:3300/0 conn(0x7f7d8c0879b0 0x7f7d8c0871e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d92cc3700 1 -- 172.21.15.16:0/2366617605 >> v1:172.21.15.16:6789/0 conn(0x7f7d8c14ca10 legacy=0x7f7d8c0869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d82ffd700 1 -- 172.21.15.16:0/2366617605 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178554621 0 0) 0x7f7d88001100 con 0x7f7d8c14ca10 2019-12-05T06:16:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d92cc3700 1 -- 172.21.15.16:0/2366617605 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8c0860e0 con 0x7f7d8c09ce90 2019-12-05T06:16:39.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> v2:172.21.15.16:3300/0 conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> v2:172.21.15.16:3300/0 conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.493+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=READY pgs=462 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.494+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=THROTTLE_DONE pgs=462 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.494+0000 7f7d82ffd700 1 -- 172.21.15.16:0/2366617605 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d84001820 con 0x7f7d8c09ce90 2019-12-05T06:16:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.494+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=THROTTLE_DONE pgs=462 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.494+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=THROTTLE_DONE pgs=462 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.494+0000 7f7d82ffd700 1 -- 172.21.15.16:0/2366617605 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d84001440 con 0x7f7d8c09ce90 2019-12-05T06:16:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.494+0000 7f7d82ffd700 1 -- 172.21.15.16:0/2366617605 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d84007ea0 con 0x7f7d8c09ce90 2019-12-05T06:16:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.494+0000 7f7d93f25700 1 -- 172.21.15.16:0/2366617605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8c0860e0 con 0x7f7d8c09ce90 2019-12-05T06:16:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.494+0000 7f7d93f25700 1 -- 172.21.15.16:0/2366617605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8c09e030 con 0x7f7d8c09ce90 2019-12-05T06:16:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.494+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=THROTTLE_DONE pgs=462 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.495+0000 7f7d82ffd700 1 -- 172.21.15.16:0/2366617605 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7d84001630 con 0x7f7d8c09ce90 2019-12-05T06:16:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.495+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=THROTTLE_DONE pgs=462 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.495+0000 7f7d82ffd700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d74023630 0x7f7d74025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.495+0000 7f7d82ffd700 1 -- 172.21.15.16:0/2366617605 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f7d84046160 con 0x7f7d8c09ce90 2019-12-05T06:16:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.496+0000 7f7d924c2700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d74023630 0x7f7d74025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:39.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.496+0000 7f7d924c2700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d74023630 0x7f7d74025ac0 crc :-1 s=READY pgs=210 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:39.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.502+0000 7f7d93f25700 1 -- 172.21.15.16:0/2366617605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8c0860e0 con 0x7f7d8c09ce90 2019-12-05T06:16:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.506+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=THROTTLE_DONE pgs=462 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.506+0000 7f7d82ffd700 1 -- 172.21.15.16:0/2366617605 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7d84001230 con 0x7f7d8c09ce90 2019-12-05T06:16:39.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.713+0000 7f7d93f25700 1 -- 172.21.15.16:0/2366617605 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d8c0860e0 con 0x7f7d8c09ce90 2019-12-05T06:16:39.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.713+0000 7f7d92cc3700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=THROTTLE_DONE pgs=462 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:39.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.713+0000 7f7d82ffd700 1 -- 172.21.15.16:0/2366617605 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f7d84001630 con 0x7f7d8c09ce90 2019-12-05T06:16:39.707 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 58 2019-12-05T06:16:39.707 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:39.708 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":58,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":58,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:38.047498+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 -- 172.21.15.16:0/2366617605 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d74023630 msgr2=0x7f7d74025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d74023630 0x7f7d74025ac0 crc :-1 s=READY pgs=210 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 -- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 msgr2=0x7f7d8c19a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 0x7f7d8c19a400 secure :-1 s=READY pgs=462 cs=0 l=1 rx=0x7f7d84010790 tx=0x7f7d84010b30).stop 2019-12-05T06:16:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 -- 172.21.15.16:0/2366617605 shutdown_connections 2019-12-05T06:16:39.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d74023630 0x7f7d74025ac0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 --2- 172.21.15.16:0/2366617605 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d8c09ce90 0x7f7d8c19a400 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 --2- 172.21.15.16:0/2366617605 >> v2:172.21.15.76:3300/0 conn(0x7f7d8c0879b0 0x7f7d8c0871e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 -- 172.21.15.16:0/2366617605 shutdown_connections 2019-12-05T06:16:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 -- 172.21.15.16:0/2366617605 wait complete. 2019-12-05T06:16:39.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:39.716+0000 7f7d93f25700 1 -- 172.21.15.16:0/2366617605 >> 172.21.15.16:0/2366617605 conn(0x7f7d8c098480 msgr2=0x7f7d8c071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:39.720 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 58, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 58, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5295": { "addr": "172.21.15.16:6833/1580628040", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 1580628040, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 1580628040, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5295, "incarnation": 58, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 55, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:16:38.047498+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5295 } } } ], "standbys": [] } 2019-12-05T06:16:41.725 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.982+0000 7f3688445700 1 Processor -- start 2019-12-05T06:16:41.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.983+0000 7f3688445700 1 -- start start 2019-12-05T06:16:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f3688445700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f368008d2c0 0x7f368008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f3688445700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f368008fef0 0x7f368008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f3688445700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36800958e0 con 0x7f368008c6d0 2019-12-05T06:16:41.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f3688445700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368008df90 con 0x7f368008d2c0 2019-12-05T06:16:41.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f3688445700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36800a3f70 con 0x7f368008fef0 2019-12-05T06:16:41.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f36861e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f368008d2c0 0x7f368008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:41.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f36871e3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f368008fef0 0x7f368008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:41.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f36861e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f368008d2c0 0x7f368008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50428/0 (socket says 172.21.15.16:50428) 2019-12-05T06:16:41.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f36871e3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f368008fef0 0x7f368008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:36516/0 (socket says 172.21.15.16:36516) 2019-12-05T06:16:41.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f36861e1700 1 -- 172.21.15.16:0/2298661638 learned_addr learned my addr 172.21.15.16:0/2298661638 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:41.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.985+0000 7f36869e2700 1 --1- 172.21.15.16:0/2298661638 >> v1:172.21.15.76:6789/0 conn(0x7f368008c6d0 0x7f368008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:56894/0 (socket says 172.21.15.16:56894) 2019-12-05T06:16:41.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.986+0000 7f36861e1700 1 -- 172.21.15.16:0/2298661638 >> v2:172.21.15.76:3300/0 conn(0x7f368008fef0 msgr2=0x7f368008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:41.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.986+0000 7f36861e1700 1 --2- 172.21.15.16:0/2298661638 >> v2:172.21.15.76:3300/0 conn(0x7f368008fef0 0x7f368008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:41.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.986+0000 7f36861e1700 1 -- 172.21.15.16:0/2298661638 >> v1:172.21.15.76:6789/0 conn(0x7f368008c6d0 legacy=0x7f368008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:41.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.986+0000 7f36861e1700 1 -- 172.21.15.16:0/2298661638 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368008e620 con 0x7f368008d2c0 2019-12-05T06:16:41.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.986+0000 7f36861e1700 1 --2- 172.21.15.16:0/2298661638 >> v2:172.21.15.16:3300/0 conn(0x7f368008d2c0 0x7f368008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f367400f740 tx=0x7f367400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:41.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.986+0000 7f36861e1700 1 --2- 172.21.15.16:0/2298661638 >> v2:172.21.15.16:3300/0 conn(0x7f368008d2c0 0x7f368008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f367400f740 tx=0x7f367400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.986+0000 7f36861e1700 1 --2- 172.21.15.16:0/2298661638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f368008d2c0 0x7f368008ef20 secure :-1 s=READY pgs=463 cs=0 l=1 rx=0x7f367400f740 tx=0x7f367400f570).ready entity=mon.0 client_cookie=8863258728df4577 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.987+0000 7f36861e1700 1 --2- 172.21.15.16:0/2298661638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f368008d2c0 0x7f368008ef20 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f367400f740 tx=0x7f367400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.987+0000 7f36861e1700 1 --2- 172.21.15.16:0/2298661638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f368008d2c0 0x7f368008ef20 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f367400f740 tx=0x7f367400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.987+0000 7f36859e0700 1 -- 172.21.15.16:0/2298661638 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3674000960 con 0x7f368008d2c0 2019-12-05T06:16:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.987+0000 7f36861e1700 1 --2- 172.21.15.16:0/2298661638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f368008d2c0 0x7f368008ef20 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f367400f740 tx=0x7f367400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:41.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.987+0000 7f36859e0700 1 -- 172.21.15.16:0/2298661638 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3674000c90 con 0x7f368008d2c0 2019-12-05T06:16:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.987+0000 7f36859e0700 1 -- 172.21.15.16:0/2298661638 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36740013a0 con 0x7f368008d2c0 2019-12-05T06:16:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.987+0000 7f3688445700 1 -- 172.21.15.16:0/2298661638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f368008d2c0 msgr2=0x7f368008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.987+0000 7f3688445700 1 --2- 172.21.15.16:0/2298661638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f368008d2c0 0x7f368008ef20 secure :-1 s=READY pgs=463 cs=0 l=1 rx=0x7f367400f740 tx=0x7f367400f570).stop 2019-12-05T06:16:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.989+0000 7f3688445700 1 -- 172.21.15.16:0/2298661638 shutdown_connections 2019-12-05T06:16:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.989+0000 7f3688445700 1 --2- 172.21.15.16:0/2298661638 >> v2:172.21.15.76:3300/0 conn(0x7f368008fef0 0x7f368008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.989+0000 7f3688445700 1 --2- 172.21.15.16:0/2298661638 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f368008d2c0 0x7f368008ef20 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:41.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.989+0000 7f3688445700 1 -- 172.21.15.16:0/2298661638 shutdown_connections 2019-12-05T06:16:41.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.989+0000 7f3688445700 1 -- 172.21.15.16:0/2298661638 wait complete. 2019-12-05T06:16:41.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.989+0000 7f3688445700 1 -- 172.21.15.16:0/2298661638 >> 172.21.15.16:0/2298661638 conn(0x7f3680061940 msgr2=0x7f3680096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.995+0000 7f3688445700 1 Processor -- start 2019-12-05T06:16:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.995+0000 7f3688445700 1 -- start start 2019-12-05T06:16:41.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.996+0000 7f3688445700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3680096e90 0x7f368009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.996+0000 7f3688445700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3680084f30 0x7f368009b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.996+0000 7f3688445700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3680084b90 con 0x7f3680097280 2019-12-05T06:16:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.996+0000 7f3688445700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f368009a450 con 0x7f3680096e90 2019-12-05T06:16:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.996+0000 7f3688445700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3680099a20 con 0x7f3680084f30 2019-12-05T06:16:41.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.996+0000 7f36869e2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3680096e90 0x7f368009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.997+0000 7f36869e2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3680096e90 0x7f368009b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50430/0 (socket says 172.21.15.16:50430) 2019-12-05T06:16:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.997+0000 7f36869e2700 1 -- 172.21.15.16:0/567521949 learned_addr learned my addr 172.21.15.16:0/567521949 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.997+0000 7f36861e1700 1 --2- 172.21.15.16:0/567521949 >> v2:172.21.15.76:3300/0 conn(0x7f3680084f30 0x7f368009b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:41.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.997+0000 7f36869e2700 1 -- 172.21.15.16:0/567521949 >> v2:172.21.15.76:3300/0 conn(0x7f3680084f30 msgr2=0x7f368009b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.997+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> v2:172.21.15.76:3300/0 conn(0x7f3680084f30 0x7f368009b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.997+0000 7f36869e2700 1 -- 172.21.15.16:0/567521949 >> v1:172.21.15.76:6789/0 conn(0x7f3680097280 legacy=0x7f36800976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.997+0000 7f36869e2700 1 -- 172.21.15.16:0/567521949 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f368009aae0 con 0x7f3680096e90 2019-12-05T06:16:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.997+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> v2:172.21.15.16:3300/0 conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:42.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.997+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> v2:172.21.15.16:3300/0 conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.997+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=READY pgs=464 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.998+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.998+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.998+0000 7f367f7fe700 1 -- 172.21.15.16:0/567521949 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3678001080 con 0x7f3680096e90 2019-12-05T06:16:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.998+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:42.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.998+0000 7f367f7fe700 1 -- 172.21.15.16:0/567521949 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3678004730 con 0x7f3680096e90 2019-12-05T06:16:42.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.998+0000 7f367f7fe700 1 -- 172.21.15.16:0/567521949 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3678004e00 con 0x7f3680096e90 2019-12-05T06:16:42.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.998+0000 7f3688445700 1 -- 172.21.15.16:0/567521949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f368009aae0 con 0x7f3680096e90 2019-12-05T06:16:42.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.998+0000 7f3688445700 1 -- 172.21.15.16:0/567521949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f368009aae0 con 0x7f3680096e90 2019-12-05T06:16:42.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:41.999+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:42.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.000+0000 7f367f7fe700 1 -- 172.21.15.16:0/567521949 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3678000fe0 con 0x7f3680096e90 2019-12-05T06:16:42.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.000+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:42.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.000+0000 7f367f7fe700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3660023630 0x7f3660025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:42.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.000+0000 7f367f7fe700 1 -- 172.21.15.16:0/567521949 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f36780461c0 con 0x7f3680096e90 2019-12-05T06:16:42.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.000+0000 7f36861e1700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3660023630 0x7f3660025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:42.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.001+0000 7f36861e1700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3660023630 0x7f3660025ac0 crc :-1 s=READY pgs=211 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:42.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.015+0000 7f3688445700 1 -- 172.21.15.16:0/567521949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f368009aae0 con 0x7f3680096e90 2019-12-05T06:16:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.022+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.022+0000 7f367f7fe700 1 -- 172.21.15.16:0/567521949 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3678001680 con 0x7f3680096e90 2019-12-05T06:16:42.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.305+0000 7f3688445700 1 -- 172.21.15.16:0/567521949 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f368009aae0 con 0x7f3680096e90 2019-12-05T06:16:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.306+0000 7f36869e2700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=THROTTLE_DONE pgs=464 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:42.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.306+0000 7f367f7fe700 1 -- 172.21.15.16:0/567521949 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 60 v60) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f3678000fe0 con 0x7f3680096e90 2019-12-05T06:16:42.301 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 60 2019-12-05T06:16:42.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:42.302 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":60,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":60,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:42.113962+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:rejoin","state_seq":11,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.311+0000 7f3688445700 1 -- 172.21.15.16:0/567521949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3660023630 msgr2=0x7f3660025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:42.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.311+0000 7f3688445700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3660023630 0x7f3660025ac0 crc :-1 s=READY pgs=211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.311+0000 7f3688445700 1 -- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 msgr2=0x7f368009b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.312+0000 7f3688445700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 0x7f368009b3e0 secure :-1 s=READY pgs=464 cs=0 l=1 rx=0x7f36780107c0 tx=0x7f36780105f0).stop 2019-12-05T06:16:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.312+0000 7f3688445700 1 -- 172.21.15.16:0/567521949 shutdown_connections 2019-12-05T06:16:42.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.312+0000 7f3688445700 1 --2- 172.21.15.16:0/567521949 >> v2:172.21.15.76:3300/0 conn(0x7f3680084f30 0x7f368009b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.312+0000 7f3688445700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3660023630 0x7f3660025ac0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.312+0000 7f3688445700 1 --2- 172.21.15.16:0/567521949 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3680096e90 0x7f368009b3e0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.312+0000 7f3688445700 1 -- 172.21.15.16:0/567521949 shutdown_connections 2019-12-05T06:16:42.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.312+0000 7f3688445700 1 -- 172.21.15.16:0/567521949 wait complete. 2019-12-05T06:16:42.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:42.312+0000 7f3688445700 1 -- 172.21.15.16:0/567521949 >> 172.21.15.16:0/567521949 conn(0x7f36800714f0 msgr2=0x7f368009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:42.317 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:16:44.318 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:44.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.585+0000 7fac1fa11700 1 Processor -- start 2019-12-05T06:16:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.586+0000 7fac1fa11700 1 -- start start 2019-12-05T06:16:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.587+0000 7fac1fa11700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fac1808c6d0 0x7fac1808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.587+0000 7fac1fa11700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac1808fef0 0x7fac1808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:44.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.587+0000 7fac1fa11700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac1808fb80 con 0x7fac1808ef20 2019-12-05T06:16:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.587+0000 7fac1fa11700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1808df90 con 0x7fac1808fef0 2019-12-05T06:16:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.587+0000 7fac1fa11700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac180a3f70 con 0x7fac1808c6d0 2019-12-05T06:16:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.588+0000 7fac1d7ad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac1808fef0 0x7fac1808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.588+0000 7fac1dfae700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fac1808c6d0 0x7fac1808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.588+0000 7fac1d7ad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac1808fef0 0x7fac1808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50446/0 (socket says 172.21.15.16:50446) 2019-12-05T06:16:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.588+0000 7fac1d7ad700 1 -- 172.21.15.16:0/3504584939 learned_addr learned my addr 172.21.15.16:0/3504584939 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:44.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1d7ad700 1 -- 172.21.15.16:0/3504584939 >> v2:172.21.15.76:3300/0 conn(0x7fac1808c6d0 msgr2=0x7fac1808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1d7ad700 1 --2- 172.21.15.16:0/3504584939 >> v2:172.21.15.76:3300/0 conn(0x7fac1808c6d0 0x7fac1808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1d7ad700 1 -- 172.21.15.16:0/3504584939 >> v1:172.21.15.76:6789/0 conn(0x7fac1808ef20 legacy=0x7fac1808f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1d7ad700 1 -- 172.21.15.16:0/3504584939 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1808d540 con 0x7fac1808fef0 2019-12-05T06:16:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1dfae700 1 --2- 172.21.15.16:0/3504584939 >> v2:172.21.15.76:3300/0 conn(0x7fac1808c6d0 0x7fac1808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:16:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1cfac700 1 -- 172.21.15.16:0/3504584939 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41048471 0 0) 0x7fac080009b0 con 0x7fac1808ef20 2019-12-05T06:16:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1d7ad700 1 --2- 172.21.15.16:0/3504584939 >> v2:172.21.15.16:3300/0 conn(0x7fac1808fef0 0x7fac1808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac0c00f740 tx=0x7fac0c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1d7ad700 1 --2- 172.21.15.16:0/3504584939 >> v2:172.21.15.16:3300/0 conn(0x7fac1808fef0 0x7fac1808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac0c00f740 tx=0x7fac0c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1d7ad700 1 --2- 172.21.15.16:0/3504584939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1808fef0 0x7fac1808d000 secure :-1 s=READY pgs=465 cs=0 l=1 rx=0x7fac0c00f740 tx=0x7fac0c00fae0).ready entity=mon.0 client_cookie=1986ac7fcb33b8d6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1d7ad700 1 --2- 172.21.15.16:0/3504584939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1808fef0 0x7fac1808d000 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7fac0c00f740 tx=0x7fac0c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1d7ad700 1 --2- 172.21.15.16:0/3504584939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1808fef0 0x7fac1808d000 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7fac0c00f740 tx=0x7fac0c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1d7ad700 1 --2- 172.21.15.16:0/3504584939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1808fef0 0x7fac1808d000 secure :-1 s=THROTTLE_DONE pgs=465 cs=0 l=1 rx=0x7fac0c00f740 tx=0x7fac0c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.589+0000 7fac1cfac700 1 -- 172.21.15.16:0/3504584939 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac0c000960 con 0x7fac1808fef0 2019-12-05T06:16:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.590+0000 7fac1cfac700 1 -- 172.21.15.16:0/3504584939 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0c000c90 con 0x7fac1808fef0 2019-12-05T06:16:44.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.590+0000 7fac1cfac700 1 -- 172.21.15.16:0/3504584939 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac0c001320 con 0x7fac1808fef0 2019-12-05T06:16:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.590+0000 7fac1fa11700 1 -- 172.21.15.16:0/3504584939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1808fef0 msgr2=0x7fac1808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.590+0000 7fac1fa11700 1 --2- 172.21.15.16:0/3504584939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1808fef0 0x7fac1808d000 secure :-1 s=READY pgs=465 cs=0 l=1 rx=0x7fac0c00f740 tx=0x7fac0c00fae0).stop 2019-12-05T06:16:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.591+0000 7fac1fa11700 1 -- 172.21.15.16:0/3504584939 shutdown_connections 2019-12-05T06:16:44.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.591+0000 7fac1fa11700 1 --2- 172.21.15.16:0/3504584939 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac1808fef0 0x7fac1808d000 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.591+0000 7fac1fa11700 1 --2- 172.21.15.16:0/3504584939 >> v2:172.21.15.76:3300/0 conn(0x7fac1808c6d0 0x7fac1808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.592+0000 7fac1fa11700 1 -- 172.21.15.16:0/3504584939 shutdown_connections 2019-12-05T06:16:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.592+0000 7fac1fa11700 1 -- 172.21.15.16:0/3504584939 wait complete. 2019-12-05T06:16:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.592+0000 7fac1fa11700 1 -- 172.21.15.16:0/3504584939 >> 172.21.15.16:0/3504584939 conn(0x7fac18061940 msgr2=0x7fac18096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.597+0000 7fac1fa11700 1 Processor -- start 2019-12-05T06:16:44.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.597+0000 7fac1fa11700 1 -- start start 2019-12-05T06:16:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1fa11700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fac18096e90 0x7fac1809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1fa11700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac18084f30 0x7fac1809b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1fa11700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac18097280 0x7fac180976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1fa11700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac1809a450 con 0x7fac18097280 2019-12-05T06:16:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1fa11700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac18099a20 con 0x7fac18096e90 2019-12-05T06:16:44.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1fa11700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac18099cf0 con 0x7fac18084f30 2019-12-05T06:16:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1e7af700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac18097280 0x7fac180976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1e7af700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac18097280 0x7fac180976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50450/0 (socket says 172.21.15.16:50450) 2019-12-05T06:16:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1e7af700 1 -- 172.21.15.16:0/1987908451 learned_addr learned my addr 172.21.15.16:0/1987908451 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1dfae700 1 --2- 172.21.15.16:0/1987908451 >> v2:172.21.15.76:3300/0 conn(0x7fac18096e90 0x7fac1809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.599+0000 7fac1d7ad700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac18084f30 0x7fac1809b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:44.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.600+0000 7fac1e7af700 1 -- 172.21.15.16:0/1987908451 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac18084f30 msgr2=0x7fac1809b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:44.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.600+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac18084f30 0x7fac1809b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:44.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.600+0000 7fac1e7af700 1 -- 172.21.15.16:0/1987908451 >> v2:172.21.15.76:3300/0 conn(0x7fac18096e90 msgr2=0x7fac1809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:44.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.600+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> v2:172.21.15.76:3300/0 conn(0x7fac18096e90 0x7fac1809b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:44.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.600+0000 7fac1e7af700 1 -- 172.21.15.16:0/1987908451 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac1809aae0 con 0x7fac18097280 2019-12-05T06:16:44.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.600+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> v2:172.21.15.16:3300/0 conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.600+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> v2:172.21.15.16:3300/0 conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.600+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=READY pgs=466 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:44.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.600+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.601+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.601+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.601+0000 7fac167fc700 1 -- 172.21.15.16:0/1987908451 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac08005770 con 0x7fac18097280 2019-12-05T06:16:44.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.601+0000 7fac167fc700 1 -- 172.21.15.16:0/1987908451 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac08005390 con 0x7fac18097280 2019-12-05T06:16:44.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.601+0000 7fac1fa11700 1 -- 172.21.15.16:0/1987908451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac1809aae0 con 0x7fac18097280 2019-12-05T06:16:44.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.601+0000 7fac167fc700 1 -- 172.21.15.16:0/1987908451 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac08000940 con 0x7fac18097280 2019-12-05T06:16:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.601+0000 7fac1fa11700 1 -- 172.21.15.16:0/1987908451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac1809aae0 con 0x7fac18097280 2019-12-05T06:16:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.602+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.603+0000 7fac167fc700 1 -- 172.21.15.16:0/1987908451 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac08005580 con 0x7fac18097280 2019-12-05T06:16:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.603+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.603+0000 7fac167fc700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fabf8023630 0x7fabf8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.603+0000 7fac167fc700 1 -- 172.21.15.16:0/1987908451 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7fac08045d40 con 0x7fac18097280 2019-12-05T06:16:44.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.604+0000 7fac1dfae700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fabf8023630 0x7fabf8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:44.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.604+0000 7fac1dfae700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fabf8023630 0x7fabf8025ac0 crc :-1 s=READY pgs=212 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:44.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.617+0000 7fac1fa11700 1 -- 172.21.15.16:0/1987908451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac1809aae0 con 0x7fac18097280 2019-12-05T06:16:44.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.624+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.624+0000 7fac167fc700 1 -- 172.21.15.16:0/1987908451 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac080075a0 con 0x7fac18097280 2019-12-05T06:16:44.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.882+0000 7fac1fa11700 1 -- 172.21.15.16:0/1987908451 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac1809aae0 con 0x7fac18097280 2019-12-05T06:16:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.883+0000 7fac1e7af700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=THROTTLE_DONE pgs=466 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.883+0000 7fac167fc700 1 -- 172.21.15.16:0/1987908451 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fac08005390 con 0x7fac18097280 2019-12-05T06:16:44.876 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2019-12-05T06:16:44.877 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:44.877 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:43.123168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":12,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:44.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.885+0000 7fac1fa11700 1 -- 172.21.15.16:0/1987908451 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fabf8023630 msgr2=0x7fabf8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:44.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.885+0000 7fac1fa11700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fabf8023630 0x7fabf8025ac0 crc :-1 s=READY pgs=212 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:44.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.885+0000 7fac1fa11700 1 -- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 msgr2=0x7fac180976c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.885+0000 7fac1fa11700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 0x7fac180976c0 secure :-1 s=READY pgs=466 cs=0 l=1 rx=0x7fac08010410 tx=0x7fac08010310).stop 2019-12-05T06:16:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.886+0000 7fac1fa11700 1 -- 172.21.15.16:0/1987908451 shutdown_connections 2019-12-05T06:16:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.886+0000 7fac1fa11700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fabf8023630 0x7fabf8025ac0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:44.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.886+0000 7fac1fa11700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac18097280 0x7fac180976c0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.886+0000 7fac1fa11700 1 --2- 172.21.15.16:0/1987908451 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac18084f30 0x7fac1809b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.886+0000 7fac1fa11700 1 --2- 172.21.15.16:0/1987908451 >> v2:172.21.15.76:3300/0 conn(0x7fac18096e90 0x7fac1809b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.886+0000 7fac1fa11700 1 -- 172.21.15.16:0/1987908451 shutdown_connections 2019-12-05T06:16:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.886+0000 7fac1fa11700 1 -- 172.21.15.16:0/1987908451 wait complete. 2019-12-05T06:16:44.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:44.886+0000 7fac1fa11700 1 -- 172.21.15.16:0/1987908451 >> 172.21.15.16:0/1987908451 conn(0x7fac180714f0 msgr2=0x7fac1809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:44.890 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5148 2019-12-05T06:16:44.890 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2019-12-05T06:16:44.890 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:16:44.891 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:16:44.891 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:16:44.895 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:16:44.896 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:45.077 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:16:45.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.117+0000 7f4001e6f700 1 Processor -- start 2019-12-05T06:16:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.117+0000 7f4001e6f700 1 -- start start 2019-12-05T06:16:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.118+0000 7f4001e6f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3ffc08c6d0 0x7f3ffc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.118+0000 7f4001e6f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ffc08fef0 0x7f3ffc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.118+0000 7f4001e6f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc0958e0 con 0x7f3ffc08ef20 2019-12-05T06:16:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.118+0000 7f4001e6f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ffc0a3f70 con 0x7f3ffc08fef0 2019-12-05T06:16:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.118+0000 7f4001e6f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ffc08d830 con 0x7f3ffc08c6d0 2019-12-05T06:16:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.118+0000 7f3ff37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ffc08fef0 0x7f3ffc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ff37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3ffc08fef0 0x7f3ffc08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50456/0 (socket says 172.21.15.16:50456) 2019-12-05T06:16:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ff37fe700 1 -- 172.21.15.16:0/1852904293 learned_addr learned my addr 172.21.15.16:0/1852904293 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f4000c0d700 1 --1- 172.21.15.16:0/1852904293 >> v1:172.21.15.16:6789/0 conn(0x7f3ffc08ef20 0x7f3ffc08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35626/0 (socket says 172.21.15.16:35626) 2019-12-05T06:16:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ffbfff700 1 --2- 172.21.15.16:0/1852904293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3ffc08c6d0 0x7f3ffc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ff37fe700 1 -- 172.21.15.16:0/1852904293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3ffc08c6d0 msgr2=0x7f3ffc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1852904293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3ffc08c6d0 0x7f3ffc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ff37fe700 1 -- 172.21.15.16:0/1852904293 >> v1:172.21.15.16:6789/0 conn(0x7f3ffc08ef20 legacy=0x7f3ffc08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ff37fe700 1 -- 172.21.15.16:0/1852904293 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc08d540 con 0x7f3ffc08fef0 2019-12-05T06:16:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1852904293 >> v2:172.21.15.16:3300/0 conn(0x7f3ffc08fef0 0x7f3ffc08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3fe400f740 tx=0x7f3fe400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1852904293 >> v2:172.21.15.16:3300/0 conn(0x7f3ffc08fef0 0x7f3ffc08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3fe400f740 tx=0x7f3fe400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1852904293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc08fef0 0x7f3ffc08d000 secure :-1 s=READY pgs=470 cs=0 l=1 rx=0x7f3fe400f740 tx=0x7f3fe400f570).ready entity=mon.0 client_cookie=1e5dece9d7629a71 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.119+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1852904293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc08fef0 0x7f3ffc08d000 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7f3fe400f740 tx=0x7f3fe400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.120+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1852904293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc08fef0 0x7f3ffc08d000 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7f3fe400f740 tx=0x7f3fe400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.120+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1852904293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc08fef0 0x7f3ffc08d000 secure :-1 s=THROTTLE_DONE pgs=470 cs=0 l=1 rx=0x7f3fe400f740 tx=0x7f3fe400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.120+0000 7f3ffb7fe700 1 -- 172.21.15.16:0/1852904293 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fe4000960 con 0x7f3ffc08fef0 2019-12-05T06:16:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.120+0000 7f3ffb7fe700 1 -- 172.21.15.16:0/1852904293 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe4000c90 con 0x7f3ffc08fef0 2019-12-05T06:16:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.120+0000 7f3ffb7fe700 1 -- 172.21.15.16:0/1852904293 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fe40013a0 con 0x7f3ffc08fef0 2019-12-05T06:16:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.120+0000 7f4001e6f700 1 -- 172.21.15.16:0/1852904293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc08fef0 msgr2=0x7f3ffc08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.120+0000 7f4001e6f700 1 --2- 172.21.15.16:0/1852904293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc08fef0 0x7f3ffc08d000 secure :-1 s=READY pgs=470 cs=0 l=1 rx=0x7f3fe400f740 tx=0x7f3fe400f570).stop 2019-12-05T06:16:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.121+0000 7f4001e6f700 1 -- 172.21.15.16:0/1852904293 shutdown_connections 2019-12-05T06:16:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.121+0000 7f4001e6f700 1 --2- 172.21.15.16:0/1852904293 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3ffc08fef0 0x7f3ffc08d000 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.121+0000 7f4001e6f700 1 --2- 172.21.15.16:0/1852904293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3ffc08c6d0 0x7f3ffc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.121+0000 7f4001e6f700 1 -- 172.21.15.16:0/1852904293 shutdown_connections 2019-12-05T06:16:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.121+0000 7f4001e6f700 1 -- 172.21.15.16:0/1852904293 wait complete. 2019-12-05T06:16:45.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.121+0000 7f4001e6f700 1 -- 172.21.15.16:0/1852904293 >> 172.21.15.16:0/1852904293 conn(0x7f3ffc061940 msgr2=0x7f3ffc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.125+0000 7f4001e6f700 1 Processor -- start 2019-12-05T06:16:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.125+0000 7f4001e6f700 1 -- start start 2019-12-05T06:16:45.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.127+0000 7f4001e6f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.127+0000 7f4001e6f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc084b90 con 0x7f3ffc096e90 2019-12-05T06:16:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.127+0000 7f4001e6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ffc098790 con 0x7f3ffc084f30 2019-12-05T06:16:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.127+0000 7f4001e6f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ffc09a450 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.127+0000 7f3ffbfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3ffc096e90 0x7f3ffc09b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35628/0 (socket says 172.21.15.16:35628) 2019-12-05T06:16:45.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.127+0000 7f3ffbfff700 1 -- 172.21.15.16:0/1658838461 learned_addr learned my addr 172.21.15.16:0/1658838461 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:16:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.127+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> v2:172.21.15.76:3300/0 conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.127+0000 7f3ff97fa700 1 -- 172.21.15.16:0/1658838461 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337478041 0 0) 0x7f3fec001430 con 0x7f3ffc096e90 2019-12-05T06:16:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.127+0000 7f3ff97fa700 1 -- 172.21.15.16:0/1658838461 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3fe0002e80 con 0x7f3ffc096e90 2019-12-05T06:16:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.128+0000 7f3ff97fa700 1 -- 172.21.15.16:0/1658838461 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4113807631 0 0) 0x7f3fec000cc0 con 0x7f3ffc096e90 2019-12-05T06:16:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.128+0000 7f3ff37fe700 1 -- 172.21.15.16:0/1658838461 >> v1:172.21.15.76:6789/0 conn(0x7f3ffc084f30 legacy=0x7f3ffc155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.128+0000 7f3ff37fe700 1 -- 172.21.15.16:0/1658838461 >> v1:172.21.15.16:6789/0 conn(0x7f3ffc096e90 legacy=0x7f3ffc09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.128+0000 7f3ff37fe700 1 -- 172.21.15.16:0/1658838461 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ffc09aae0 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.128+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> v2:172.21.15.76:3300/0 conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.128+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> v2:172.21.15.76:3300/0 conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.128+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=READY pgs=361 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:45.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.129+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.129+0000 7f3ff97fa700 1 -- 172.21.15.16:0/1658838461 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fe4000d20 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.129+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.129+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.129+0000 7f4001e6f700 1 -- 172.21.15.16:0/1658838461 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ffc09aae0 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.129+0000 7f3ff97fa700 1 -- 172.21.15.16:0/1658838461 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3fe4007e10 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.129+0000 7f3ff97fa700 1 -- 172.21.15.16:0/1658838461 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3fe4000dc0 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.129+0000 7f4001e6f700 1 -- 172.21.15.16:0/1658838461 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ffc09aae0 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.130+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.131+0000 7f3ff97fa700 1 -- 172.21.15.16:0/1658838461 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3fe4000940 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.131+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.132+0000 7f3ff97fa700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fe0023680 0x7f3fe0025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.132+0000 7f3ff97fa700 1 -- 172.21.15.16:0/1658838461 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f3fe4045d70 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.132+0000 7f3ffbfff700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fe0023680 0x7f3fe0025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:45.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.132+0000 7f3ffbfff700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fe0023680 0x7f3fe0025af0 crc :-1 s=READY pgs=213 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:45.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.133+0000 7f4001e6f700 1 -- 172.21.15.16:0/1658838461 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ffc09aae0 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.143+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.143+0000 7f3ff97fa700 1 -- 172.21.15.16:0/1658838461 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3fe4007580 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.384+0000 7f4001e6f700 1 -- 172.21.15.16:0/1658838461 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3ffc09aae0 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.385+0000 7f3ff37fe700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=THROTTLE_DONE pgs=361 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.385+0000 7f3ff97fa700 1 -- 172.21.15.16:0/1658838461 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f3fe4000940 con 0x7f3ffc09bbe0 2019-12-05T06:16:45.380 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:45.380 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:43.123168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":12,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2019-12-05T06:16:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.389+0000 7f4001e6f700 1 -- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fe0023680 msgr2=0x7f3fe0025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.389+0000 7f4001e6f700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fe0023680 0x7f3fe0025af0 crc :-1 s=READY pgs=213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.389+0000 7f4001e6f700 1 -- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 msgr2=0x7f3ffc1539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.389+0000 7f4001e6f700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 secure :-1 s=READY pgs=361 cs=0 l=1 rx=0x7f3fe4008320 tx=0x7f3fe4010770).stop 2019-12-05T06:16:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.389+0000 7f4001e6f700 1 -- 172.21.15.16:0/1658838461 shutdown_connections 2019-12-05T06:16:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.389+0000 7f4001e6f700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3fe0023680 0x7f3fe0025af0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.389+0000 7f4001e6f700 1 --2- 172.21.15.16:0/1658838461 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3ffc09bbe0 0x7f3ffc1539e0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.390+0000 7f4001e6f700 1 -- 172.21.15.16:0/1658838461 shutdown_connections 2019-12-05T06:16:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.390+0000 7f4001e6f700 1 -- 172.21.15.16:0/1658838461 wait complete. 2019-12-05T06:16:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:45.390+0000 7f4001e6f700 1 -- 172.21.15.16:0/1658838461 >> 172.21.15.16:0/1658838461 conn(0x7f3ffc0714f0 msgr2=0x7f3ffc09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:45.394 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:16:47.395 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:47.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.681+0000 7f2b170b5700 1 Processor -- start 2019-12-05T06:16:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.682+0000 7f2b170b5700 1 -- start start 2019-12-05T06:16:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.683+0000 7f2b170b5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2b1008d2c0 0x7f2b1008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:47.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.683+0000 7f2b170b5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b1008fef0 0x7f2b1008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b170b5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b100958e0 con 0x7f2b1008c6d0 2019-12-05T06:16:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b170b5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1008df90 con 0x7f2b1008fef0 2019-12-05T06:16:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b170b5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b100a3f70 con 0x7f2b1008d2c0 2019-12-05T06:16:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b15e53700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b1008fef0 0x7f2b1008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b15e53700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b1008fef0 0x7f2b1008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50474/0 (socket says 172.21.15.16:50474) 2019-12-05T06:16:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b15e53700 1 -- 172.21.15.16:0/3993837972 learned_addr learned my addr 172.21.15.16:0/3993837972 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:47.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b14e51700 1 --2- 172.21.15.16:0/3993837972 >> v2:172.21.15.76:3300/0 conn(0x7f2b1008d2c0 0x7f2b1008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b15e53700 1 -- 172.21.15.16:0/3993837972 >> v2:172.21.15.76:3300/0 conn(0x7f2b1008d2c0 msgr2=0x7f2b1008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b15e53700 1 --2- 172.21.15.16:0/3993837972 >> v2:172.21.15.76:3300/0 conn(0x7f2b1008d2c0 0x7f2b1008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b15e53700 1 -- 172.21.15.16:0/3993837972 >> v1:172.21.15.76:6789/0 conn(0x7f2b1008c6d0 legacy=0x7f2b1008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.684+0000 7f2b15e53700 1 -- 172.21.15.16:0/3993837972 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1008e620 con 0x7f2b1008fef0 2019-12-05T06:16:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b15e53700 1 --2- 172.21.15.16:0/3993837972 >> v2:172.21.15.16:3300/0 conn(0x7f2b1008fef0 0x7f2b1008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2b0800fab0 tx=0x7f2b080080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b15e53700 1 --2- 172.21.15.16:0/3993837972 >> v2:172.21.15.16:3300/0 conn(0x7f2b1008fef0 0x7f2b1008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2b0800fab0 tx=0x7f2b080080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b15e53700 1 --2- 172.21.15.16:0/3993837972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1008fef0 0x7f2b1008f460 secure :-1 s=READY pgs=473 cs=0 l=1 rx=0x7f2b0800fab0 tx=0x7f2b080080d0).ready entity=mon.0 client_cookie=b82979707524761d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b15e53700 1 --2- 172.21.15.16:0/3993837972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1008fef0 0x7f2b1008f460 secure :-1 s=THROTTLE_DONE pgs=473 cs=0 l=1 rx=0x7f2b0800fab0 tx=0x7f2b080080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b15e53700 1 --2- 172.21.15.16:0/3993837972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1008fef0 0x7f2b1008f460 secure :-1 s=THROTTLE_DONE pgs=473 cs=0 l=1 rx=0x7f2b0800fab0 tx=0x7f2b080080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b07fff700 1 -- 172.21.15.16:0/3993837972 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b08000df0 con 0x7f2b1008fef0 2019-12-05T06:16:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b15e53700 1 --2- 172.21.15.16:0/3993837972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1008fef0 0x7f2b1008f460 secure :-1 s=THROTTLE_DONE pgs=473 cs=0 l=1 rx=0x7f2b0800fab0 tx=0x7f2b080080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b07fff700 1 -- 172.21.15.16:0/3993837972 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b08001460 con 0x7f2b1008fef0 2019-12-05T06:16:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b07fff700 1 -- 172.21.15.16:0/3993837972 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b08001b30 con 0x7f2b1008fef0 2019-12-05T06:16:47.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b170b5700 1 -- 172.21.15.16:0/3993837972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1008fef0 msgr2=0x7f2b1008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.685+0000 7f2b170b5700 1 --2- 172.21.15.16:0/3993837972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1008fef0 0x7f2b1008f460 secure :-1 s=READY pgs=473 cs=0 l=1 rx=0x7f2b0800fab0 tx=0x7f2b080080d0).stop 2019-12-05T06:16:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.687+0000 7f2b170b5700 1 -- 172.21.15.16:0/3993837972 shutdown_connections 2019-12-05T06:16:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.687+0000 7f2b170b5700 1 --2- 172.21.15.16:0/3993837972 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1008fef0 0x7f2b1008f460 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.687+0000 7f2b170b5700 1 --2- 172.21.15.16:0/3993837972 >> v2:172.21.15.76:3300/0 conn(0x7f2b1008d2c0 0x7f2b1008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.687+0000 7f2b170b5700 1 -- 172.21.15.16:0/3993837972 shutdown_connections 2019-12-05T06:16:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.687+0000 7f2b170b5700 1 -- 172.21.15.16:0/3993837972 wait complete. 2019-12-05T06:16:47.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.687+0000 7f2b170b5700 1 -- 172.21.15.16:0/3993837972 >> 172.21.15.16:0/3993837972 conn(0x7f2b10061940 msgr2=0x7f2b10096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.692+0000 7f2b170b5700 1 Processor -- start 2019-12-05T06:16:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.692+0000 7f2b170b5700 1 -- start start 2019-12-05T06:16:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.694+0000 7f2b170b5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2b10096e90 0x7f2b1009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.694+0000 7f2b170b5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b1009b920 0x7f2b10097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:47.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.694+0000 7f2b170b5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b10084b90 con 0x7f2b10084f30 2019-12-05T06:16:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.694+0000 7f2b170b5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b1009a450 con 0x7f2b1009b920 2019-12-05T06:16:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.694+0000 7f2b170b5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b10099a20 con 0x7f2b10096e90 2019-12-05T06:16:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.694+0000 7f2b15e53700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b1009b920 0x7f2b10097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.694+0000 7f2b15e53700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2b1009b920 0x7f2b10097a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50480/0 (socket says 172.21.15.16:50480) 2019-12-05T06:16:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.694+0000 7f2b15e53700 1 -- 172.21.15.16:0/1251930394 learned_addr learned my addr 172.21.15.16:0/1251930394 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:47.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.694+0000 7f2b15652700 1 --2- 172.21.15.16:0/1251930394 >> v2:172.21.15.76:3300/0 conn(0x7f2b10096e90 0x7f2b1009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b15e53700 1 -- 172.21.15.16:0/1251930394 >> v2:172.21.15.76:3300/0 conn(0x7f2b10096e90 msgr2=0x7f2b1009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> v2:172.21.15.76:3300/0 conn(0x7f2b10096e90 0x7f2b1009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b05ffb700 1 -- 172.21.15.16:0/1251930394 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455485944 0 0) 0x7f2b00000c90 con 0x7f2b10084f30 2019-12-05T06:16:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b15e53700 1 -- 172.21.15.16:0/1251930394 >> v1:172.21.15.76:6789/0 conn(0x7f2b10084f30 legacy=0x7f2b10097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:47.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b15e53700 1 -- 172.21.15.16:0/1251930394 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1009aae0 con 0x7f2b1009b920 2019-12-05T06:16:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> v2:172.21.15.16:3300/0 conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> v2:172.21.15.16:3300/0 conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=READY pgs=474 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b05ffb700 1 -- 172.21.15.16:0/1251930394 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b08001910 con 0x7f2b1009b920 2019-12-05T06:16:47.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b05ffb700 1 -- 172.21.15.16:0/1251930394 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b08001530 con 0x7f2b1009b920 2019-12-05T06:16:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b05ffb700 1 -- 172.21.15.16:0/1251930394 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b08001720 con 0x7f2b1009b920 2019-12-05T06:16:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.695+0000 7f2b170b5700 1 -- 172.21.15.16:0/1251930394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b1009aae0 con 0x7f2b1009b920 2019-12-05T06:16:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.696+0000 7f2b170b5700 1 -- 172.21.15.16:0/1251930394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b1009aae0 con 0x7f2b1009b920 2019-12-05T06:16:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.696+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.697+0000 7f2b05ffb700 1 -- 172.21.15.16:0/1251930394 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2b08001230 con 0x7f2b1009b920 2019-12-05T06:16:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.697+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.698+0000 7f2b05ffb700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2af0023630 0x7f2af0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.698+0000 7f2b05ffb700 1 -- 172.21.15.16:0/1251930394 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f2b08046190 con 0x7f2b1009b920 2019-12-05T06:16:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.698+0000 7f2b15652700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2af0023630 0x7f2af0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:47.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.698+0000 7f2b15652700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2af0023630 0x7f2af0025ac0 crc :-1 s=READY pgs=214 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:47.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.713+0000 7f2b170b5700 1 -- 172.21.15.16:0/1251930394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b1009aae0 con 0x7f2b1009b920 2019-12-05T06:16:47.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.720+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.720+0000 7f2b05ffb700 1 -- 172.21.15.16:0/1251930394 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2b0802b1b0 con 0x7f2b1009b920 2019-12-05T06:16:47.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.973+0000 7f2b170b5700 1 -- 172.21.15.16:0/1251930394 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2b1009aae0 con 0x7f2b1009b920 2019-12-05T06:16:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.974+0000 7f2b15e53700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:47.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.974+0000 7f2b05ffb700 1 -- 172.21.15.16:0/1251930394 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 61 v61) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f2b08001230 con 0x7f2b1009b920 2019-12-05T06:16:47.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:47.968 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":61,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:43.123168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":12,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:47.968 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 61 2019-12-05T06:16:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.976+0000 7f2b170b5700 1 -- 172.21.15.16:0/1251930394 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2af0023630 msgr2=0x7f2af0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.976+0000 7f2b170b5700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2af0023630 0x7f2af0025ac0 crc :-1 s=READY pgs=214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.976+0000 7f2b170b5700 1 -- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 msgr2=0x7f2b10097a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:47.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.976+0000 7f2b170b5700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 0x7f2b10097a80 secure :-1 s=READY pgs=474 cs=0 l=1 rx=0x7f2b08010800 tx=0x7f2b08010630).stop 2019-12-05T06:16:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.977+0000 7f2b170b5700 1 -- 172.21.15.16:0/1251930394 shutdown_connections 2019-12-05T06:16:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.977+0000 7f2b170b5700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2af0023630 0x7f2af0025ac0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.977+0000 7f2b170b5700 1 --2- 172.21.15.16:0/1251930394 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2b1009b920 0x7f2b10097a80 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.977+0000 7f2b170b5700 1 --2- 172.21.15.16:0/1251930394 >> v2:172.21.15.76:3300/0 conn(0x7f2b10096e90 0x7f2b1009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.977+0000 7f2b170b5700 1 -- 172.21.15.16:0/1251930394 shutdown_connections 2019-12-05T06:16:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.977+0000 7f2b170b5700 1 -- 172.21.15.16:0/1251930394 wait complete. 2019-12-05T06:16:47.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:47.977+0000 7f2b170b5700 1 -- 172.21.15.16:0/1251930394 >> 172.21.15.16:0/1251930394 conn(0x7f2b100714f0 msgr2=0x7f2b1009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:47.981 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:16:49.982 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:16:50.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.245+0000 7f8c5777d700 1 Processor -- start 2019-12-05T06:16:50.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.245+0000 7f8c5777d700 1 -- start start 2019-12-05T06:16:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.247+0000 7f8c5777d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c5008d2c0 0x7f8c5008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.247+0000 7f8c5777d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8c5008fef0 0x7f8c5008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:16:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.247+0000 7f8c5777d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c500958e0 con 0x7f8c5008c6d0 2019-12-05T06:16:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.247+0000 7f8c5777d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c5008df90 con 0x7f8c5008d2c0 2019-12-05T06:16:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.247+0000 7f8c5777d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c500a3f70 con 0x7f8c5008fef0 2019-12-05T06:16:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.247+0000 7f8c55519700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c5008d2c0 0x7f8c5008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.247+0000 7f8c55519700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c5008d2c0 0x7f8c5008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50494/0 (socket says 172.21.15.16:50494) 2019-12-05T06:16:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.247+0000 7f8c55519700 1 -- 172.21.15.16:0/2081694858 learned_addr learned my addr 172.21.15.16:0/2081694858 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:16:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.247+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2081694858 >> v2:172.21.15.76:3300/0 conn(0x7f8c5008fef0 0x7f8c5008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.248+0000 7f8c55519700 1 -- 172.21.15.16:0/2081694858 >> v2:172.21.15.76:3300/0 conn(0x7f8c5008fef0 msgr2=0x7f8c5008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:16:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.248+0000 7f8c55519700 1 --2- 172.21.15.16:0/2081694858 >> v2:172.21.15.76:3300/0 conn(0x7f8c5008fef0 0x7f8c5008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.248+0000 7f8c55519700 1 -- 172.21.15.16:0/2081694858 >> v1:172.21.15.76:6789/0 conn(0x7f8c5008c6d0 legacy=0x7f8c5008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.248+0000 7f8c55519700 1 -- 172.21.15.16:0/2081694858 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5008e620 con 0x7f8c5008d2c0 2019-12-05T06:16:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.248+0000 7f8c55519700 1 --2- 172.21.15.16:0/2081694858 >> v2:172.21.15.16:3300/0 conn(0x7f8c5008d2c0 0x7f8c5008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8c3800fa40 tx=0x7f8c3800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.248+0000 7f8c55519700 1 --2- 172.21.15.16:0/2081694858 >> v2:172.21.15.16:3300/0 conn(0x7f8c5008d2c0 0x7f8c5008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8c3800fa40 tx=0x7f8c3800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.248+0000 7f8c55519700 1 --2- 172.21.15.16:0/2081694858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5008d2c0 0x7f8c5008ef20 secure :-1 s=READY pgs=478 cs=0 l=1 rx=0x7f8c3800fa40 tx=0x7f8c3800fde0).ready entity=mon.0 client_cookie=1a8882e40670dec6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:50.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.248+0000 7f8c55519700 1 --2- 172.21.15.16:0/2081694858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5008d2c0 0x7f8c5008ef20 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f8c3800fa40 tx=0x7f8c3800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.249+0000 7f8c55519700 1 --2- 172.21.15.16:0/2081694858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5008d2c0 0x7f8c5008ef20 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f8c3800fa40 tx=0x7f8c3800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.249+0000 7f8c54d18700 1 -- 172.21.15.16:0/2081694858 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c38000ce0 con 0x7f8c5008d2c0 2019-12-05T06:16:50.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.249+0000 7f8c55519700 1 --2- 172.21.15.16:0/2081694858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5008d2c0 0x7f8c5008ef20 secure :-1 s=THROTTLE_DONE pgs=478 cs=0 l=1 rx=0x7f8c3800fa40 tx=0x7f8c3800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.249+0000 7f8c54d18700 1 -- 172.21.15.16:0/2081694858 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c38001010 con 0x7f8c5008d2c0 2019-12-05T06:16:50.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.249+0000 7f8c54d18700 1 -- 172.21.15.16:0/2081694858 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c38001720 con 0x7f8c5008d2c0 2019-12-05T06:16:50.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.249+0000 7f8c5777d700 1 -- 172.21.15.16:0/2081694858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5008d2c0 msgr2=0x7f8c5008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:50.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.249+0000 7f8c5777d700 1 --2- 172.21.15.16:0/2081694858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5008d2c0 0x7f8c5008ef20 secure :-1 s=READY pgs=478 cs=0 l=1 rx=0x7f8c3800fa40 tx=0x7f8c3800fde0).stop 2019-12-05T06:16:50.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.251+0000 7f8c5777d700 1 -- 172.21.15.16:0/2081694858 shutdown_connections 2019-12-05T06:16:50.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.251+0000 7f8c5777d700 1 --2- 172.21.15.16:0/2081694858 >> v2:172.21.15.76:3300/0 conn(0x7f8c5008fef0 0x7f8c5008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:16:50.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.251+0000 7f8c5777d700 1 --2- 172.21.15.16:0/2081694858 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5008d2c0 0x7f8c5008ef20 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:50.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.251+0000 7f8c5777d700 1 -- 172.21.15.16:0/2081694858 shutdown_connections 2019-12-05T06:16:50.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.251+0000 7f8c5777d700 1 -- 172.21.15.16:0/2081694858 wait complete. 2019-12-05T06:16:50.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.251+0000 7f8c5777d700 1 -- 172.21.15.16:0/2081694858 >> 172.21.15.16:0/2081694858 conn(0x7f8c50061940 msgr2=0x7f8c50096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:50.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.255+0000 7f8c5777d700 1 Processor -- start 2019-12-05T06:16:50.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.255+0000 7f8c5777d700 1 -- start start 2019-12-05T06:16:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.256+0000 7f8c5777d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c5014d7f0 0x7f8c50132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.256+0000 7f8c5777d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c50098da0 con 0x7f8c50132d00 2019-12-05T06:16:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.256+0000 7f8c5777d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c5009f0a0 con 0x7f8c5014d400 2019-12-05T06:16:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.256+0000 7f8c5777d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c5009bb50 con 0x7f8c5014d7f0 2019-12-05T06:16:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.256+0000 7f8c5651b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c5014d7f0 0x7f8c50132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.256+0000 7f8c55519700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8c50132d00 0x7f8c501550f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35668/0 (socket says 172.21.15.16:35668) 2019-12-05T06:16:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.256+0000 7f8c5651b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8c5014d7f0 0x7f8c50132530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50498/0 (socket says 172.21.15.16:50498) 2019-12-05T06:16:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.256+0000 7f8c55519700 1 -- 172.21.15.16:0/2800545280 learned_addr learned my addr 172.21.15.16:0/2800545280 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:16:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.257+0000 7f8c5651b700 1 -- 172.21.15.16:0/2800545280 >> v1:172.21.15.76:6789/0 conn(0x7f8c5014d400 legacy=0x7f8c50131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.257+0000 7f8c5651b700 1 -- 172.21.15.16:0/2800545280 >> v1:172.21.15.16:6789/0 conn(0x7f8c50132d00 legacy=0x7f8c501550f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.257+0000 7f8c5651b700 1 -- 172.21.15.16:0/2800545280 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c5009b0f0 con 0x7f8c5014d7f0 2019-12-05T06:16:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.257+0000 7f8c4e7fc700 1 -- 172.21.15.16:0/2800545280 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1219436257 0 0) 0x7f8c380080a0 con 0x7f8c50132d00 2019-12-05T06:16:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.257+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2800545280 >> v2:172.21.15.16:3300/0 conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.257+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2800545280 >> v2:172.21.15.16:3300/0 conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.257+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=READY pgs=480 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.258+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.258+0000 7f8c4e7fc700 1 -- 172.21.15.16:0/2800545280 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c4800ee80 con 0x7f8c5014d7f0 2019-12-05T06:16:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.258+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.258+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.258+0000 7f8c4e7fc700 1 -- 172.21.15.16:0/2800545280 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c4800f080 con 0x7f8c5014d7f0 2019-12-05T06:16:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.258+0000 7f8c5777d700 1 -- 172.21.15.16:0/2800545280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c5009b0f0 con 0x7f8c5014d7f0 2019-12-05T06:16:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.258+0000 7f8c4e7fc700 1 -- 172.21.15.16:0/2800545280 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c4800f750 con 0x7f8c5014d7f0 2019-12-05T06:16:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.258+0000 7f8c5777d700 1 -- 172.21.15.16:0/2800545280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c5009b0f0 con 0x7f8c5014d7f0 2019-12-05T06:16:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.259+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.259+0000 7f8c4e7fc700 1 -- 172.21.15.16:0/2800545280 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8c4800ebe0 con 0x7f8c5014d7f0 2019-12-05T06:16:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.260+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.260+0000 7f8c4e7fc700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c34023630 0x7f8c34025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:16:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.260+0000 7f8c4e7fc700 1 -- 172.21.15.16:0/2800545280 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f8c48045d90 con 0x7f8c5014d7f0 2019-12-05T06:16:50.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.260+0000 7f8c55d1a700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c34023630 0x7f8c34025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:16:50.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.261+0000 7f8c55d1a700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c34023630 0x7f8c34025ac0 crc :-1 s=READY pgs=216 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:16:50.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.276+0000 7f8c5777d700 1 -- 172.21.15.16:0/2800545280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c5009b0f0 con 0x7f8c5014d7f0 2019-12-05T06:16:50.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.284+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.284+0000 7f8c4e7fc700 1 -- 172.21.15.16:0/2800545280 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8c480401e0 con 0x7f8c5014d7f0 2019-12-05T06:16:50.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.528+0000 7f8c5777d700 1 -- 172.21.15.16:0/2800545280 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c5009b0f0 con 0x7f8c5014d7f0 2019-12-05T06:16:50.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.529+0000 7f8c5651b700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=THROTTLE_DONE pgs=480 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:16:50.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.529+0000 7f8c4e7fc700 1 -- 172.21.15.16:0/2800545280 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f8c4800ef80 con 0x7f8c5014d7f0 2019-12-05T06:16:50.524 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 62 2019-12-05T06:16:50.524 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:16:50.524 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5382,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:43.123168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":12,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:16:50.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.535+0000 7f8c5777d700 1 -- 172.21.15.16:0/2800545280 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c34023630 msgr2=0x7f8c34025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.535+0000 7f8c5777d700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c34023630 0x7f8c34025ac0 crc :-1 s=READY pgs=216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.535+0000 7f8c5777d700 1 -- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 msgr2=0x7f8c50132530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:16:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.535+0000 7f8c5777d700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 0x7f8c50132530 secure :-1 s=READY pgs=480 cs=0 l=1 rx=0x7f8c4800ea40 tx=0x7f8c480106f0).stop 2019-12-05T06:16:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.535+0000 7f8c5777d700 1 -- 172.21.15.16:0/2800545280 shutdown_connections 2019-12-05T06:16:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.535+0000 7f8c5777d700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c34023630 0x7f8c34025ac0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.535+0000 7f8c5777d700 1 --2- 172.21.15.16:0/2800545280 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8c5014d7f0 0x7f8c50132530 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:16:50.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.535+0000 7f8c5777d700 1 -- 172.21.15.16:0/2800545280 shutdown_connections 2019-12-05T06:16:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.536+0000 7f8c5777d700 1 -- 172.21.15.16:0/2800545280 wait complete. 2019-12-05T06:16:50.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:16:50.536+0000 7f8c5777d700 1 -- 172.21.15.16:0/2800545280 >> 172.21.15.16:0/2800545280 conn(0x7f8c500714f0 msgr2=0x7f8c5009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:16:50.545 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:16:50.545 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 54 secs before thrashing 2019-12-05T06:17:00.942 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:17:00.950 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:17:31.130 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:17:31.135 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:17:44.546 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.803+0000 7f4fff2fc700 1 Processor -- start 2019-12-05T06:17:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.804+0000 7f4fff2fc700 1 -- start start 2019-12-05T06:17:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.805+0000 7f4fff2fc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4ff808d2c0 0x7f4ff808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.805+0000 7f4fff2fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ff808fef0 0x7f4ff808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.805+0000 7f4fff2fc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff80958e0 con 0x7f4ff808c6d0 2019-12-05T06:17:44.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.805+0000 7f4fff2fc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff808df90 con 0x7f4ff808fef0 2019-12-05T06:17:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.805+0000 7f4fff2fc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff80a3f70 con 0x7f4ff808d2c0 2019-12-05T06:17:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.806+0000 7f4ffe09a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ff808fef0 0x7f4ff808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.806+0000 7f4ffe09a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ff808fef0 0x7f4ff808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50520/0 (socket says 172.21.15.16:50520) 2019-12-05T06:17:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.806+0000 7f4ffe09a700 1 -- 172.21.15.16:0/1237622616 learned_addr learned my addr 172.21.15.16:0/1237622616 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:17:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.806+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1237622616 >> v2:172.21.15.76:3300/0 conn(0x7f4ff808d2c0 0x7f4ff808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.806+0000 7f4ffe09a700 1 -- 172.21.15.16:0/1237622616 >> v2:172.21.15.76:3300/0 conn(0x7f4ff808d2c0 msgr2=0x7f4ff808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:44.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.806+0000 7f4ffe09a700 1 --2- 172.21.15.16:0/1237622616 >> v2:172.21.15.76:3300/0 conn(0x7f4ff808d2c0 0x7f4ff808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.806+0000 7f4ffe09a700 1 -- 172.21.15.16:0/1237622616 >> v1:172.21.15.76:6789/0 conn(0x7f4ff808c6d0 legacy=0x7f4ff808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.806+0000 7f4ffe09a700 1 -- 172.21.15.16:0/1237622616 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff808e620 con 0x7f4ff808fef0 2019-12-05T06:17:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.807+0000 7f4ffe09a700 1 --2- 172.21.15.16:0/1237622616 >> v2:172.21.15.16:3300/0 conn(0x7f4ff808fef0 0x7f4ff808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4fe800f730 tx=0x7f4fe8007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.807+0000 7f4ffe09a700 1 --2- 172.21.15.16:0/1237622616 >> v2:172.21.15.16:3300/0 conn(0x7f4ff808fef0 0x7f4ff808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4fe800f730 tx=0x7f4fe8007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.807+0000 7f4ffe09a700 1 --2- 172.21.15.16:0/1237622616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff808fef0 0x7f4ff808f460 secure :-1 s=READY pgs=482 cs=0 l=1 rx=0x7f4fe800f730 tx=0x7f4fe8007d50).ready entity=mon.0 client_cookie=77e835757c7c57b4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:44.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.807+0000 7f4ffe09a700 1 --2- 172.21.15.16:0/1237622616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff808fef0 0x7f4ff808f460 secure :-1 s=THROTTLE_DONE pgs=482 cs=0 l=1 rx=0x7f4fe800f730 tx=0x7f4fe8007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.807+0000 7f4ffe09a700 1 --2- 172.21.15.16:0/1237622616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff808fef0 0x7f4ff808f460 secure :-1 s=THROTTLE_DONE pgs=482 cs=0 l=1 rx=0x7f4fe800f730 tx=0x7f4fe8007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.807+0000 7f4ffc897700 1 -- 172.21.15.16:0/1237622616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fe8000a70 con 0x7f4ff808fef0 2019-12-05T06:17:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.807+0000 7f4ffe09a700 1 --2- 172.21.15.16:0/1237622616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff808fef0 0x7f4ff808f460 secure :-1 s=THROTTLE_DONE pgs=482 cs=0 l=1 rx=0x7f4fe800f730 tx=0x7f4fe8007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.807+0000 7f4ffc897700 1 -- 172.21.15.16:0/1237622616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fe80010e0 con 0x7f4ff808fef0 2019-12-05T06:17:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.807+0000 7f4ffc897700 1 -- 172.21.15.16:0/1237622616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fe80017b0 con 0x7f4ff808fef0 2019-12-05T06:17:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.808+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1237622616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff808fef0 msgr2=0x7f4ff808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:44.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.808+0000 7f4fff2fc700 1 --2- 172.21.15.16:0/1237622616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff808fef0 0x7f4ff808f460 secure :-1 s=READY pgs=482 cs=0 l=1 rx=0x7f4fe800f730 tx=0x7f4fe8007d50).stop 2019-12-05T06:17:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.809+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1237622616 shutdown_connections 2019-12-05T06:17:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.809+0000 7f4fff2fc700 1 --2- 172.21.15.16:0/1237622616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ff808fef0 0x7f4ff808f460 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.809+0000 7f4fff2fc700 1 --2- 172.21.15.16:0/1237622616 >> v2:172.21.15.76:3300/0 conn(0x7f4ff808d2c0 0x7f4ff808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.810+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1237622616 shutdown_connections 2019-12-05T06:17:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.810+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1237622616 wait complete. 2019-12-05T06:17:44.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.810+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1237622616 >> 172.21.15.16:0/1237622616 conn(0x7f4ff8061940 msgr2=0x7f4ff8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.814+0000 7f4fff2fc700 1 Processor -- start 2019-12-05T06:17:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.814+0000 7f4fff2fc700 1 -- start start 2019-12-05T06:17:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.816+0000 7f4fff2fc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4ff8132d00 0x7f4ff814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.816+0000 7f4fff2fc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff8098da0 con 0x7f4ff8132530 2019-12-05T06:17:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.816+0000 7f4fff2fc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff809f0a0 con 0x7f4ff814d400 2019-12-05T06:17:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.816+0000 7f4fff2fc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ff809bb50 con 0x7f4ff8132d00 2019-12-05T06:17:44.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.816+0000 7f4ffe09a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4ff8132530 0x7f4ff8157100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35694/0 (socket says 172.21.15.16:35694) 2019-12-05T06:17:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.816+0000 7f4ffe09a700 1 -- 172.21.15.16:0/1638389512 learned_addr learned my addr 172.21.15.16:0/1638389512 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:17:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.816+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> v2:172.21.15.76:3300/0 conn(0x7f4ff8132d00 0x7f4ff814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.817+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1984210784 0 0) 0x7f4fe8000940 con 0x7f4ff8132530 2019-12-05T06:17:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.817+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd8002e80 con 0x7f4ff8132530 2019-12-05T06:17:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.817+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1391853642 0 0) 0x7f4ff0001400 con 0x7f4ff814d400 2019-12-05T06:17:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.817+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd8004170 con 0x7f4ff814d400 2019-12-05T06:17:44.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.817+0000 7f4ffd098700 1 -- 172.21.15.16:0/1638389512 >> v1:172.21.15.76:6789/0 conn(0x7f4ff814d400 legacy=0x7f4ff8131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.817+0000 7f4ffd098700 1 -- 172.21.15.16:0/1638389512 >> v1:172.21.15.16:6789/0 conn(0x7f4ff8132530 legacy=0x7f4ff8157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.817+0000 7f4ffd098700 1 -- 172.21.15.16:0/1638389512 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff809b0f0 con 0x7f4ff8132d00 2019-12-05T06:17:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.817+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> v2:172.21.15.76:3300/0 conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.818+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> v2:172.21.15.76:3300/0 conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.818+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=READY pgs=372 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.818+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=THROTTLE_DONE pgs=372 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.818+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fec009640 con 0x7f4ff8132d00 2019-12-05T06:17:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.818+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=THROTTLE_DONE pgs=372 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.818+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fec000940 con 0x7f4ff8132d00 2019-12-05T06:17:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.818+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=THROTTLE_DONE pgs=372 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.818+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1638389512 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff809b0f0 con 0x7f4ff8132d00 2019-12-05T06:17:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.818+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4fec009e50 con 0x7f4ff8132d00 2019-12-05T06:17:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.819+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1638389512 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff809b0f0 con 0x7f4ff8132d00 2019-12-05T06:17:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.819+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=THROTTLE_DONE pgs=372 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.820+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4fec009a70 con 0x7f4ff8132d00 2019-12-05T06:17:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.820+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=THROTTLE_DONE pgs=372 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.821+0000 7f4ff67fc700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fd8023650 0x7f4fd8025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:44.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.821+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f4fec045dc0 con 0x7f4ff8132d00 2019-12-05T06:17:44.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.821+0000 7f4ffd899700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fd8023650 0x7f4fd8025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:44.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.821+0000 7f4ffd899700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fd8023650 0x7f4fd8025ae0 crc :-1 s=READY pgs=219 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:44.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.835+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1638389512 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff809b0f0 con 0x7f4ff8132d00 2019-12-05T06:17:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.844+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=THROTTLE_DONE pgs=372 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:44.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:44.844+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4fec009860 con 0x7f4ff8132d00 2019-12-05T06:17:45.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.109+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1638389512 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ff809b0f0 con 0x7f4ff8132d00 2019-12-05T06:17:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.110+0000 7f4ffd098700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=THROTTLE_DONE pgs=372 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.110+0000 7f4ff67fc700 1 -- 172.21.15.16:0/1638389512 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f4fec009a50 con 0x7f4ff8132d00 2019-12-05T06:17:45.104 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:17:45.104 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5382,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:43.123168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":12,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 62 2019-12-05T06:17:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.114+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fd8023650 msgr2=0x7f4fd8025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.114+0000 7f4fff2fc700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fd8023650 0x7f4fd8025ae0 crc :-1 s=READY pgs=219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.114+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 msgr2=0x7f4ff814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.114+0000 7f4fff2fc700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 0x7f4ff814d7f0 secure :-1 s=READY pgs=372 cs=0 l=1 rx=0x7f4fec010390 tx=0x7f4fec010730).stop 2019-12-05T06:17:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.115+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1638389512 shutdown_connections 2019-12-05T06:17:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.115+0000 7f4fff2fc700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fd8023650 0x7f4fd8025ae0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.115+0000 7f4fff2fc700 1 --2- 172.21.15.16:0/1638389512 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4ff8132d00 0x7f4ff814d7f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.115+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1638389512 shutdown_connections 2019-12-05T06:17:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.115+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1638389512 wait complete. 2019-12-05T06:17:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.115+0000 7f4fff2fc700 1 -- 172.21.15.16:0/1638389512 >> 172.21.15.16:0/1638389512 conn(0x7f4ff80714f0 msgr2=0x7f4ff809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:45.118 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:17:45.118 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:17:45.118 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:17:45.131 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:17:45.131 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.458+0000 7f04d7cba700 1 Processor -- start 2019-12-05T06:17:45.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.458+0000 7f04d7cba700 1 -- start start 2019-12-05T06:17:45.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.460+0000 7f04d7cba700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 0x7f04d008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:45.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.460+0000 7f04d7cba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00958e0 con 0x7f04d008fef0 2019-12-05T06:17:45.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.460+0000 7f04d7cba700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d00629e0 con 0x7f04d008c6d0 2019-12-05T06:17:45.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.460+0000 7f04d7cba700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d008df90 con 0x7f04d008d2c0 2019-12-05T06:17:45.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.460+0000 7f04d6a58700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04d008fef0 0x7f04d008f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35706/0 (socket says 172.21.15.16:35706) 2019-12-05T06:17:45.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.460+0000 7f04d6a58700 1 -- 172.21.15.16:0/598739293 learned_addr learned my addr 172.21.15.16:0/598739293 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:17:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.460+0000 7f04d5a56700 1 --2- 172.21.15.16:0/598739293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 0x7f04d008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.461+0000 7f04d5255700 1 -- 172.21.15.16:0/598739293 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 459578790 0 0) 0x7f04c0001100 con 0x7f04d008fef0 2019-12-05T06:17:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.461+0000 7f04d5255700 1 -- 172.21.15.16:0/598739293 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b8002e80 con 0x7f04d008fef0 2019-12-05T06:17:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.461+0000 7f04d5255700 1 -- 172.21.15.16:0/598739293 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1614846262 0 0) 0x7f04c8001400 con 0x7f04d008c6d0 2019-12-05T06:17:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.461+0000 7f04d5255700 1 -- 172.21.15.16:0/598739293 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04b8004220 con 0x7f04d008c6d0 2019-12-05T06:17:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.461+0000 7f04d5a56700 1 -- 172.21.15.16:0/598739293 >> v1:172.21.15.76:6789/0 conn(0x7f04d008c6d0 legacy=0x7f04d008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:45.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.461+0000 7f04d5a56700 1 -- 172.21.15.16:0/598739293 >> v1:172.21.15.16:6789/0 conn(0x7f04d008fef0 legacy=0x7f04d008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.461+0000 7f04d5a56700 1 -- 172.21.15.16:0/598739293 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d008e620 con 0x7f04d008d2c0 2019-12-05T06:17:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d5a56700 1 --2- 172.21.15.16:0/598739293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 0x7f04d008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f04c400af50 tx=0x7f04c400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d5a56700 1 --2- 172.21.15.16:0/598739293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 0x7f04d008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f04c400af50 tx=0x7f04c400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d5a56700 1 --2- 172.21.15.16:0/598739293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 0x7f04d008ef20 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7f04c400af50 tx=0x7f04c400b760).ready entity=mon.2 client_cookie=f5e5c8e62a3dd47f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d5a56700 1 --2- 172.21.15.16:0/598739293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 0x7f04d008ef20 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f04c400af50 tx=0x7f04c400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d5a56700 1 --2- 172.21.15.16:0/598739293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 0x7f04d008ef20 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f04c400af50 tx=0x7f04c400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d5255700 1 -- 172.21.15.16:0/598739293 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04c4000960 con 0x7f04d008d2c0 2019-12-05T06:17:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d5a56700 1 --2- 172.21.15.16:0/598739293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 0x7f04d008ef20 secure :-1 s=THROTTLE_DONE pgs=125 cs=0 l=1 rx=0x7f04c400af50 tx=0x7f04c400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d5255700 1 -- 172.21.15.16:0/598739293 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c40098e0 con 0x7f04d008d2c0 2019-12-05T06:17:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d5255700 1 -- 172.21.15.16:0/598739293 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04c400a050 con 0x7f04d008d2c0 2019-12-05T06:17:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d7cba700 1 -- 172.21.15.16:0/598739293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 msgr2=0x7f04d008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.462+0000 7f04d7cba700 1 --2- 172.21.15.16:0/598739293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 0x7f04d008ef20 secure :-1 s=READY pgs=125 cs=0 l=1 rx=0x7f04c400af50 tx=0x7f04c400b760).stop 2019-12-05T06:17:45.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.464+0000 7f04d7cba700 1 -- 172.21.15.16:0/598739293 shutdown_connections 2019-12-05T06:17:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.464+0000 7f04d7cba700 1 --2- 172.21.15.16:0/598739293 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d008d2c0 0x7f04d008ef20 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.464+0000 7f04d7cba700 1 -- 172.21.15.16:0/598739293 shutdown_connections 2019-12-05T06:17:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.465+0000 7f04d7cba700 1 -- 172.21.15.16:0/598739293 wait complete. 2019-12-05T06:17:45.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.465+0000 7f04d7cba700 1 -- 172.21.15.16:0/598739293 >> 172.21.15.16:0/598739293 conn(0x7f04d0061940 msgr2=0x7f04d0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.468+0000 7f04d7cba700 1 Processor -- start 2019-12-05T06:17:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.469+0000 7f04d7cba700 1 -- start start 2019-12-05T06:17:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.470+0000 7f04d7cba700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f04d009e7d0 0x7f04d009e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.470+0000 7f04d7cba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04d0156e70 0x7f04d0159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.470+0000 7f04d7cba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d012cb90 con 0x7f04d01302f0 2019-12-05T06:17:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.470+0000 7f04d7cba700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d0131ff0 con 0x7f04d0156e70 2019-12-05T06:17:45.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.470+0000 7f04d7cba700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d01315c0 con 0x7f04d009e7d0 2019-12-05T06:17:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.470+0000 7f04d6257700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04d01302f0 0x7f04d009d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35710/0 (socket says 172.21.15.16:35710) 2019-12-05T06:17:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.470+0000 7f04d6a58700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04d0156e70 0x7f04d0159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.470+0000 7f04d6257700 1 -- 172.21.15.16:0/185613152 learned_addr learned my addr 172.21.15.16:0/185613152 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:17:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.470+0000 7f04d5a56700 1 --2- 172.21.15.16:0/185613152 >> v2:172.21.15.76:3300/0 conn(0x7f04d009e7d0 0x7f04d009e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04ceffd700 1 -- 172.21.15.16:0/185613152 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211842956 0 0) 0x7f04c80015c0 con 0x7f04d01302f0 2019-12-05T06:17:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d6a58700 1 -- 172.21.15.16:0/185613152 >> v2:172.21.15.76:3300/0 conn(0x7f04d009e7d0 msgr2=0x7f04d009e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:45.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> v2:172.21.15.76:3300/0 conn(0x7f04d009e7d0 0x7f04d009e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d6a58700 1 -- 172.21.15.16:0/185613152 >> v1:172.21.15.16:6789/0 conn(0x7f04d01302f0 legacy=0x7f04d009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d6a58700 1 -- 172.21.15.16:0/185613152 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d0132680 con 0x7f04d0156e70 2019-12-05T06:17:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d5a56700 1 --2- 172.21.15.16:0/185613152 >> v2:172.21.15.76:3300/0 conn(0x7f04d009e7d0 0x7f04d009e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:17:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> v2:172.21.15.16:3300/0 conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> v2:172.21.15.16:3300/0 conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=READY pgs=486 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:45.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04ceffd700 1 -- 172.21.15.16:0/185613152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04c0000ef0 con 0x7f04d0156e70 2019-12-05T06:17:45.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04ceffd700 1 -- 172.21.15.16:0/185613152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c0004240 con 0x7f04d0156e70 2019-12-05T06:17:45.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04ceffd700 1 -- 172.21.15.16:0/185613152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04c0000ef0 con 0x7f04d0156e70 2019-12-05T06:17:45.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.471+0000 7f04d7cba700 1 -- 172.21.15.16:0/185613152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d0132680 con 0x7f04d0156e70 2019-12-05T06:17:45.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.472+0000 7f04d7cba700 1 -- 172.21.15.16:0/185613152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d0132680 con 0x7f04d0156e70 2019-12-05T06:17:45.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.473+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.474+0000 7f04ceffd700 1 -- 172.21.15.16:0/185613152 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f04c0004430 con 0x7f04d0156e70 2019-12-05T06:17:45.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.474+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.474+0000 7f04d7cba700 1 -- 172.21.15.16:0/185613152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d0132680 con 0x7f04d0156e70 2019-12-05T06:17:45.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.474+0000 7f04ceffd700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04b0023630 0x7f04b0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:45.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.474+0000 7f04ceffd700 1 -- 172.21.15.16:0/185613152 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f04c0045d50 con 0x7f04d0156e70 2019-12-05T06:17:45.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.474+0000 7f04d6257700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04b0023630 0x7f04b0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:45.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.475+0000 7f04d6257700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04b0023630 0x7f04b0025ac0 crc :-1 s=READY pgs=220 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.482+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.482+0000 7f04ceffd700 1 -- 172.21.15.16:0/185613152 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f04c0000ef0 con 0x7f04d0156e70 2019-12-05T06:17:45.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.739+0000 7f04d7cba700 1 -- 172.21.15.16:0/185613152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04d0132680 con 0x7f04d0156e70 2019-12-05T06:17:45.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.740+0000 7f04d6a58700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=THROTTLE_DONE pgs=486 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:45.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.740+0000 7f04ceffd700 1 -- 172.21.15.16:0/185613152 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f04c0004430 con 0x7f04d0156e70 2019-12-05T06:17:45.733 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 62 2019-12-05T06:17:45.733 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:17:45.733 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5382,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:43.123168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":12,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:45.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 -- 172.21.15.16:0/185613152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04b0023630 msgr2=0x7f04b0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:45.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04b0023630 0x7f04b0025ac0 crc :-1 s=READY pgs=220 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 -- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 msgr2=0x7f04d0159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 0x7f04d0159270 secure :-1 s=READY pgs=486 cs=0 l=1 rx=0x7f04c0010370 tx=0x7f04c0010710).stop 2019-12-05T06:17:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 -- 172.21.15.16:0/185613152 shutdown_connections 2019-12-05T06:17:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04b0023630 0x7f04b0025ac0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 --2- 172.21.15.16:0/185613152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d0156e70 0x7f04d0159270 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:45.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 --2- 172.21.15.16:0/185613152 >> v2:172.21.15.76:3300/0 conn(0x7f04d009e7d0 0x7f04d009e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:45.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 -- 172.21.15.16:0/185613152 shutdown_connections 2019-12-05T06:17:45.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 -- 172.21.15.16:0/185613152 wait complete. 2019-12-05T06:17:45.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:45.743+0000 7f04d7cba700 1 -- 172.21.15.16:0/185613152 >> 172.21.15.16:0/185613152 conn(0x7f04d00a2850 msgr2=0x7f04d006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:45.746 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:17:47.747 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:48.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.010+0000 7f6a977fe700 1 Processor -- start 2019-12-05T06:17:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.011+0000 7f6a977fe700 1 -- start start 2019-12-05T06:17:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.012+0000 7f6a977fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6a9008c6d0 0x7f6a9008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.012+0000 7f6a977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a9008fef0 0x7f6a9008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.012+0000 7f6a977fe700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a9008fb80 con 0x7f6a9008ef20 2019-12-05T06:17:48.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.012+0000 7f6a977fe700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a9008df90 con 0x7f6a9008fef0 2019-12-05T06:17:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.012+0000 7f6a977fe700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a900a3f70 con 0x7f6a9008c6d0 2019-12-05T06:17:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.013+0000 7f6a9559a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a9008fef0 0x7f6a9008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.013+0000 7f6a9559a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a9008fef0 0x7f6a9008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50558/0 (socket says 172.21.15.16:50558) 2019-12-05T06:17:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.013+0000 7f6a9559a700 1 -- 172.21.15.16:0/1195877867 learned_addr learned my addr 172.21.15.16:0/1195877867 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:17:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.013+0000 7f6a9659c700 1 --1- 172.21.15.16:0/1195877867 >> v1:172.21.15.16:6789/0 conn(0x7f6a9008ef20 0x7f6a9008f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35724/0 (socket says 172.21.15.16:35724) 2019-12-05T06:17:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.013+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/1195877867 >> v2:172.21.15.76:3300/0 conn(0x7f6a9008c6d0 0x7f6a9008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.013+0000 7f6a9559a700 1 -- 172.21.15.16:0/1195877867 >> v2:172.21.15.76:3300/0 conn(0x7f6a9008c6d0 msgr2=0x7f6a9008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.013+0000 7f6a9559a700 1 --2- 172.21.15.16:0/1195877867 >> v2:172.21.15.76:3300/0 conn(0x7f6a9008c6d0 0x7f6a9008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a9559a700 1 -- 172.21.15.16:0/1195877867 >> v1:172.21.15.16:6789/0 conn(0x7f6a9008ef20 legacy=0x7f6a9008f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a94d99700 1 -- 172.21.15.16:0/1195877867 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015061410 0 0) 0x7f6a800009b0 con 0x7f6a9008ef20 2019-12-05T06:17:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a9559a700 1 -- 172.21.15.16:0/1195877867 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9008d540 con 0x7f6a9008fef0 2019-12-05T06:17:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/1195877867 >> v2:172.21.15.76:3300/0 conn(0x7f6a9008c6d0 0x7f6a9008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:17:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a9559a700 1 --2- 172.21.15.16:0/1195877867 >> v2:172.21.15.16:3300/0 conn(0x7f6a9008fef0 0x7f6a9008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6a8400f740 tx=0x7f6a8400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a9559a700 1 --2- 172.21.15.16:0/1195877867 >> v2:172.21.15.16:3300/0 conn(0x7f6a9008fef0 0x7f6a9008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6a8400f740 tx=0x7f6a8400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a9559a700 1 --2- 172.21.15.16:0/1195877867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9008fef0 0x7f6a9008d000 secure :-1 s=READY pgs=488 cs=0 l=1 rx=0x7f6a8400f740 tx=0x7f6a8400fae0).ready entity=mon.0 client_cookie=ab4b85992936301 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a9559a700 1 --2- 172.21.15.16:0/1195877867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9008fef0 0x7f6a9008d000 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f6a8400f740 tx=0x7f6a8400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a94d99700 1 -- 172.21.15.16:0/1195877867 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a84000960 con 0x7f6a9008fef0 2019-12-05T06:17:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a9559a700 1 --2- 172.21.15.16:0/1195877867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9008fef0 0x7f6a9008d000 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f6a8400f740 tx=0x7f6a8400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.014+0000 7f6a9559a700 1 --2- 172.21.15.16:0/1195877867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9008fef0 0x7f6a9008d000 secure :-1 s=THROTTLE_DONE pgs=488 cs=0 l=1 rx=0x7f6a8400f740 tx=0x7f6a8400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.015+0000 7f6a94d99700 1 -- 172.21.15.16:0/1195877867 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a84000c90 con 0x7f6a9008fef0 2019-12-05T06:17:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.015+0000 7f6a94d99700 1 -- 172.21.15.16:0/1195877867 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a84007210 con 0x7f6a9008fef0 2019-12-05T06:17:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.015+0000 7f6a977fe700 1 -- 172.21.15.16:0/1195877867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9008fef0 msgr2=0x7f6a9008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.015+0000 7f6a977fe700 1 --2- 172.21.15.16:0/1195877867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9008fef0 0x7f6a9008d000 secure :-1 s=READY pgs=488 cs=0 l=1 rx=0x7f6a8400f740 tx=0x7f6a8400fae0).stop 2019-12-05T06:17:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.016+0000 7f6a977fe700 1 -- 172.21.15.16:0/1195877867 shutdown_connections 2019-12-05T06:17:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.016+0000 7f6a977fe700 1 --2- 172.21.15.16:0/1195877867 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9008fef0 0x7f6a9008d000 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.016+0000 7f6a977fe700 1 --2- 172.21.15.16:0/1195877867 >> v2:172.21.15.76:3300/0 conn(0x7f6a9008c6d0 0x7f6a9008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:48.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.017+0000 7f6a977fe700 1 -- 172.21.15.16:0/1195877867 shutdown_connections 2019-12-05T06:17:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.017+0000 7f6a977fe700 1 -- 172.21.15.16:0/1195877867 wait complete. 2019-12-05T06:17:48.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.017+0000 7f6a977fe700 1 -- 172.21.15.16:0/1195877867 >> 172.21.15.16:0/1195877867 conn(0x7f6a90061940 msgr2=0x7f6a90096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.022+0000 7f6a977fe700 1 Processor -- start 2019-12-05T06:17:48.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.022+0000 7f6a977fe700 1 -- start start 2019-12-05T06:17:48.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.023+0000 7f6a977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a9006df10 0x7f6a90098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:48.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.023+0000 7f6a977fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a90156cb0 0x7f6a901590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:48.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.023+0000 7f6a977fe700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a90089fb0 con 0x7f6a900992d0 2019-12-05T06:17:48.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.023+0000 7f6a977fe700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a90097370 con 0x7f6a9006df10 2019-12-05T06:17:48.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.023+0000 7f6a977fe700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a9008ad00 con 0x7f6a90156cb0 2019-12-05T06:17:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.024+0000 7f6a95d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a9006df10 0x7f6a90098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.024+0000 7f6a95d9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a9006df10 0x7f6a90098300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50562/0 (socket says 172.21.15.16:50562) 2019-12-05T06:17:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.024+0000 7f6a95d9b700 1 -- 172.21.15.16:0/3053656437 learned_addr learned my addr 172.21.15.16:0/3053656437 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:17:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.024+0000 7f6a9659c700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a90156cb0 0x7f6a901590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.024+0000 7f6a95d9b700 1 -- 172.21.15.16:0/3053656437 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a90156cb0 msgr2=0x7f6a901590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.024+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a90156cb0 0x7f6a901590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:48.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.024+0000 7f6a95d9b700 1 -- 172.21.15.16:0/3053656437 >> v1:172.21.15.76:6789/0 conn(0x7f6a900992d0 legacy=0x7f6a90098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.024+0000 7f6a95d9b700 1 -- 172.21.15.16:0/3053656437 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a90097a00 con 0x7f6a9006df10 2019-12-05T06:17:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.024+0000 7f6a8e7fc700 1 -- 172.21.15.16:0/3053656437 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554462780 0 0) 0x7f6a84007900 con 0x7f6a900992d0 2019-12-05T06:17:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.025+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> v2:172.21.15.16:3300/0 conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.025+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> v2:172.21.15.16:3300/0 conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.025+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=READY pgs=489 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:48.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.025+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=THROTTLE_DONE pgs=489 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.025+0000 7f6a8e7fc700 1 -- 172.21.15.16:0/3053656437 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a88000ce0 con 0x7f6a9006df10 2019-12-05T06:17:48.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.025+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=THROTTLE_DONE pgs=489 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.025+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=THROTTLE_DONE pgs=489 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.025+0000 7f6a8e7fc700 1 -- 172.21.15.16:0/3053656437 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a88009af0 con 0x7f6a9006df10 2019-12-05T06:17:48.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.025+0000 7f6a977fe700 1 -- 172.21.15.16:0/3053656437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a90097a00 con 0x7f6a9006df10 2019-12-05T06:17:48.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.025+0000 7f6a8e7fc700 1 -- 172.21.15.16:0/3053656437 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a8800a240 con 0x7f6a9006df10 2019-12-05T06:17:48.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.026+0000 7f6a977fe700 1 -- 172.21.15.16:0/3053656437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a90097a00 con 0x7f6a9006df10 2019-12-05T06:17:48.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.026+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=THROTTLE_DONE pgs=489 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.027+0000 7f6a8e7fc700 1 -- 172.21.15.16:0/3053656437 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6a88009ce0 con 0x7f6a9006df10 2019-12-05T06:17:48.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.027+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=THROTTLE_DONE pgs=489 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.027+0000 7f6a8e7fc700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a70023630 0x7f6a70025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:48.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.028+0000 7f6a8e7fc700 1 -- 172.21.15.16:0/3053656437 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f6a88046160 con 0x7f6a9006df10 2019-12-05T06:17:48.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.028+0000 7f6a9559a700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a70023630 0x7f6a70025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:48.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.028+0000 7f6a9559a700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a70023630 0x7f6a70025ac0 crc :-1 s=READY pgs=221 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:48.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.042+0000 7f6a977fe700 1 -- 172.21.15.16:0/3053656437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a90097a00 con 0x7f6a9006df10 2019-12-05T06:17:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.050+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=THROTTLE_DONE pgs=489 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.050+0000 7f6a8e7fc700 1 -- 172.21.15.16:0/3053656437 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6a8802b3c0 con 0x7f6a9006df10 2019-12-05T06:17:48.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.289+0000 7f6a977fe700 1 -- 172.21.15.16:0/3053656437 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a90097a00 con 0x7f6a9006df10 2019-12-05T06:17:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.290+0000 7f6a95d9b700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=THROTTLE_DONE pgs=489 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:48.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.290+0000 7f6a8e7fc700 1 -- 172.21.15.16:0/3053656437 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f6a8802b5d0 con 0x7f6a9006df10 2019-12-05T06:17:48.283 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 62 2019-12-05T06:17:48.283 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:17:48.283 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5382,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:43.123168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":12,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 -- 172.21.15.16:0/3053656437 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a70023630 msgr2=0x7f6a70025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:48.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a70023630 0x7f6a70025ac0 crc :-1 s=READY pgs=221 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 -- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 msgr2=0x7f6a90098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 0x7f6a90098300 secure :-1 s=READY pgs=489 cs=0 l=1 rx=0x7f6a88010710 tx=0x7f6a88010ab0).stop 2019-12-05T06:17:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 -- 172.21.15.16:0/3053656437 shutdown_connections 2019-12-05T06:17:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a90156cb0 0x7f6a901590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a70023630 0x7f6a70025ac0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:48.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 --2- 172.21.15.16:0/3053656437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a9006df10 0x7f6a90098300 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:48.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 -- 172.21.15.16:0/3053656437 shutdown_connections 2019-12-05T06:17:48.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 -- 172.21.15.16:0/3053656437 wait complete. 2019-12-05T06:17:48.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:48.293+0000 7f6a977fe700 1 -- 172.21.15.16:0/3053656437 >> 172.21.15.16:0/3053656437 conn(0x7f6a900714f0 msgr2=0x7f6a9009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:48.296 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:17:50.297 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:50.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.574+0000 7f11dd53b700 1 Processor -- start 2019-12-05T06:17:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.574+0000 7f11dd53b700 1 -- start start 2019-12-05T06:17:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.576+0000 7f11dd53b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.576+0000 7f11dd53b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f11d808f720 0x7f11d808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.576+0000 7f11dd53b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d80958e0 con 0x7f11d808fef0 2019-12-05T06:17:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.576+0000 7f11dd53b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d80a3f70 con 0x7f11d808f720 2019-12-05T06:17:50.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.576+0000 7f11dd53b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11d808d830 con 0x7f11d808c6d0 2019-12-05T06:17:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.576+0000 7f11d7fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f11d808f720 0x7f11d808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.576+0000 7f11d77fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.576+0000 7f11d7fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f11d808f720 0x7f11d808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:36662/0 (socket says 172.21.15.16:36662) 2019-12-05T06:17:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.576+0000 7f11d77fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:54560/0 (socket says 172.21.15.16:54560) 2019-12-05T06:17:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.576+0000 7f11d7fff700 1 -- 172.21.15.16:0/1150240008 learned_addr learned my addr 172.21.15.16:0/1150240008 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:17:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.577+0000 7f11d77fe700 1 -- 172.21.15.16:0/1150240008 >> v2:172.21.15.76:3300/0 conn(0x7f11d808f720 msgr2=0x7f11d808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:50.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.577+0000 7f11d77fe700 1 --2- 172.21.15.16:0/1150240008 >> v2:172.21.15.76:3300/0 conn(0x7f11d808f720 0x7f11d808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.577+0000 7f11d77fe700 1 -- 172.21.15.16:0/1150240008 >> v1:172.21.15.76:6789/0 conn(0x7f11d808fef0 legacy=0x7f11d808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.577+0000 7f11d67fc700 1 -- 172.21.15.16:0/1150240008 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496231350 0 0) 0x7f11c0001080 con 0x7f11d808fef0 2019-12-05T06:17:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.577+0000 7f11d77fe700 1 -- 172.21.15.16:0/1150240008 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d808fbc0 con 0x7f11d808c6d0 2019-12-05T06:17:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.577+0000 7f11d7fff700 1 --2- 172.21.15.16:0/1150240008 >> v2:172.21.15.76:3300/0 conn(0x7f11d808f720 0x7f11d808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:17:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.577+0000 7f11d77fe700 1 --2- 172.21.15.16:0/1150240008 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f11c800af50 tx=0x7f11c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.578+0000 7f11d77fe700 1 --2- 172.21.15.16:0/1150240008 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f11c800af50 tx=0x7f11c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.578+0000 7f11d77fe700 1 --2- 172.21.15.16:0/1150240008 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 secure :-1 s=READY pgs=126 cs=0 l=1 rx=0x7f11c800af50 tx=0x7f11c800b760).ready entity=mon.2 client_cookie=f84b134f3099144e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:50.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.578+0000 7f11d77fe700 1 --2- 172.21.15.16:0/1150240008 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 secure :-1 s=THROTTLE_DONE pgs=126 cs=0 l=1 rx=0x7f11c800af50 tx=0x7f11c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.578+0000 7f11d77fe700 1 --2- 172.21.15.16:0/1150240008 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 secure :-1 s=THROTTLE_DONE pgs=126 cs=0 l=1 rx=0x7f11c800af50 tx=0x7f11c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.578+0000 7f11d67fc700 1 -- 172.21.15.16:0/1150240008 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11c8007210 con 0x7f11d808c6d0 2019-12-05T06:17:50.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.578+0000 7f11d77fe700 1 --2- 172.21.15.16:0/1150240008 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 secure :-1 s=THROTTLE_DONE pgs=126 cs=0 l=1 rx=0x7f11c800af50 tx=0x7f11c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.578+0000 7f11d67fc700 1 -- 172.21.15.16:0/1150240008 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c80099b0 con 0x7f11d808c6d0 2019-12-05T06:17:50.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.578+0000 7f11d67fc700 1 -- 172.21.15.16:0/1150240008 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11c8009f80 con 0x7f11d808c6d0 2019-12-05T06:17:50.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.578+0000 7f11dd53b700 1 -- 172.21.15.16:0/1150240008 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 msgr2=0x7f11d808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:50.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.578+0000 7f11dd53b700 1 --2- 172.21.15.16:0/1150240008 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 secure :-1 s=READY pgs=126 cs=0 l=1 rx=0x7f11c800af50 tx=0x7f11c800b760).stop 2019-12-05T06:17:50.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.580+0000 7f11dd53b700 1 -- 172.21.15.16:0/1150240008 shutdown_connections 2019-12-05T06:17:50.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.580+0000 7f11dd53b700 1 --2- 172.21.15.16:0/1150240008 >> v2:172.21.15.76:3300/0 conn(0x7f11d808f720 0x7f11d808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:50.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.580+0000 7f11dd53b700 1 --2- 172.21.15.16:0/1150240008 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11d808c6d0 0x7f11d808cac0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:50.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.580+0000 7f11dd53b700 1 -- 172.21.15.16:0/1150240008 shutdown_connections 2019-12-05T06:17:50.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.580+0000 7f11dd53b700 1 -- 172.21.15.16:0/1150240008 wait complete. 2019-12-05T06:17:50.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.580+0000 7f11dd53b700 1 -- 172.21.15.16:0/1150240008 >> 172.21.15.16:0/1150240008 conn(0x7f11d8061940 msgr2=0x7f11d8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:50.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.586+0000 7f11dd53b700 1 Processor -- start 2019-12-05T06:17:50.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.586+0000 7f11dd53b700 1 -- start start 2019-12-05T06:17:50.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.588+0000 7f11dd53b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f11d81302f0 0x7f11d809d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:50.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.588+0000 7f11dd53b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d812cb90 con 0x7f11d809e7d0 2019-12-05T06:17:50.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.588+0000 7f11dd53b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d812d040 con 0x7f11d8156e70 2019-12-05T06:17:50.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.588+0000 7f11dd53b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11d8131ff0 con 0x7f11d81302f0 2019-12-05T06:17:50.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.588+0000 7f11d6ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f11d809e7d0 0x7f11d809dd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35744/0 (socket says 172.21.15.16:35744) 2019-12-05T06:17:50.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.588+0000 7f11d6ffd700 1 -- 172.21.15.16:0/2198139371 learned_addr learned my addr 172.21.15.16:0/2198139371 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:17:50.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.588+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> v2:172.21.15.76:3300/0 conn(0x7f11d81302f0 0x7f11d809d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:50.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.589+0000 7f11cffff700 1 -- 172.21.15.16:0/2198139371 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623213531 0 0) 0x7f11c00012c0 con 0x7f11d809e7d0 2019-12-05T06:17:50.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.589+0000 7f11cffff700 1 -- 172.21.15.16:0/2198139371 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11bc002e80 con 0x7f11d809e7d0 2019-12-05T06:17:50.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.589+0000 7f11cffff700 1 -- 172.21.15.16:0/2198139371 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608158835 0 0) 0x7f11d0007b60 con 0x7f11d8156e70 2019-12-05T06:17:50.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.589+0000 7f11d77fe700 1 -- 172.21.15.16:0/2198139371 >> v1:172.21.15.76:6789/0 conn(0x7f11d8156e70 legacy=0x7f11d8159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:50.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.589+0000 7f11d77fe700 1 -- 172.21.15.16:0/2198139371 >> v1:172.21.15.16:6789/0 conn(0x7f11d809e7d0 legacy=0x7f11d809dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:50.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.589+0000 7f11d77fe700 1 -- 172.21.15.16:0/2198139371 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d8132680 con 0x7f11d81302f0 2019-12-05T06:17:50.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.589+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> v2:172.21.15.76:3300/0 conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.589+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> v2:172.21.15.76:3300/0 conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.589+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=READY pgs=377 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:50.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.590+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.590+0000 7f11cffff700 1 -- 172.21.15.16:0/2198139371 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11c8009580 con 0x7f11d81302f0 2019-12-05T06:17:50.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.590+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.590+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.590+0000 7f11cffff700 1 -- 172.21.15.16:0/2198139371 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11c800b480 con 0x7f11d81302f0 2019-12-05T06:17:50.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.590+0000 7f11cffff700 1 -- 172.21.15.16:0/2198139371 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11c8009d00 con 0x7f11d81302f0 2019-12-05T06:17:50.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.590+0000 7f11dd53b700 1 -- 172.21.15.16:0/2198139371 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11d8132680 con 0x7f11d81302f0 2019-12-05T06:17:50.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.590+0000 7f11dd53b700 1 -- 172.21.15.16:0/2198139371 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11d8132680 con 0x7f11d81302f0 2019-12-05T06:17:50.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.591+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.592+0000 7f11cffff700 1 -- 172.21.15.16:0/2198139371 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f11c8009580 con 0x7f11d81302f0 2019-12-05T06:17:50.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.592+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.592+0000 7f11cffff700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11bc023680 0x7f11bc025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:50.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.592+0000 7f11cffff700 1 -- 172.21.15.16:0/2198139371 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f11c8045dc0 con 0x7f11d81302f0 2019-12-05T06:17:50.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.592+0000 7f11d6ffd700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11bc023680 0x7f11bc025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:50.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.593+0000 7f11d6ffd700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11bc023680 0x7f11bc025af0 crc :-1 s=READY pgs=222 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:50.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.605+0000 7f11dd53b700 1 -- 172.21.15.16:0/2198139371 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11d8132680 con 0x7f11d81302f0 2019-12-05T06:17:50.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.615+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.615+0000 7f11cffff700 1 -- 172.21.15.16:0/2198139371 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f11c800abb0 con 0x7f11d81302f0 2019-12-05T06:17:50.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.864+0000 7f11dd53b700 1 -- 172.21.15.16:0/2198139371 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11d8132680 con 0x7f11d81302f0 2019-12-05T06:17:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.865+0000 7f11d77fe700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:50.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.865+0000 7f11cffff700 1 -- 172.21.15.16:0/2198139371 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f11c8009580 con 0x7f11d81302f0 2019-12-05T06:17:50.859 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 62 2019-12-05T06:17:50.859 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:17:50.859 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5382,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:43.123168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":12,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.871+0000 7f11dd53b700 1 -- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11bc023680 msgr2=0x7f11bc025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.871+0000 7f11dd53b700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11bc023680 0x7f11bc025af0 crc :-1 s=READY pgs=222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.871+0000 7f11dd53b700 1 -- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 msgr2=0x7f11d809d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.871+0000 7f11dd53b700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 0x7f11d809d800 secure :-1 s=READY pgs=377 cs=0 l=1 rx=0x7f11c8010380 tx=0x7f11c8010720).stop 2019-12-05T06:17:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.871+0000 7f11dd53b700 1 -- 172.21.15.16:0/2198139371 shutdown_connections 2019-12-05T06:17:50.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.871+0000 7f11dd53b700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f11bc023680 0x7f11bc025af0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:50.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.871+0000 7f11dd53b700 1 --2- 172.21.15.16:0/2198139371 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f11d81302f0 0x7f11d809d800 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:50.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.871+0000 7f11dd53b700 1 -- 172.21.15.16:0/2198139371 shutdown_connections 2019-12-05T06:17:50.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.871+0000 7f11dd53b700 1 -- 172.21.15.16:0/2198139371 wait complete. 2019-12-05T06:17:50.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:50.871+0000 7f11dd53b700 1 -- 172.21.15.16:0/2198139371 >> 172.21.15.16:0/2198139371 conn(0x7f11d80a2850 msgr2=0x7f11d806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:50.885 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:17:52.885 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:53.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.136+0000 7f002567d700 1 Processor -- start 2019-12-05T06:17:53.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.136+0000 7f002567d700 1 -- start start 2019-12-05T06:17:53.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.137+0000 7f002567d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002008c6d0 0x7f002008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:53.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.137+0000 7f002567d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f002008fef0 0x7f002008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:53.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.137+0000 7f002567d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00200958e0 con 0x7f002008ef20 2019-12-05T06:17:53.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.137+0000 7f002567d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00200a3f70 con 0x7f002008fef0 2019-12-05T06:17:53.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.138+0000 7f002567d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002008d830 con 0x7f002008c6d0 2019-12-05T06:17:53.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.138+0000 7f001ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f002008ef20 0x7f002008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35756/0 (socket says 172.21.15.16:35756) 2019-12-05T06:17:53.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.138+0000 7f001ffff700 1 -- 172.21.15.16:0/3641209788 learned_addr learned my addr 172.21.15.16:0/3641209788 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:17:53.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.138+0000 7f001effd700 1 --2- 172.21.15.16:0/3641209788 >> v2:172.21.15.76:3300/0 conn(0x7f002008fef0 0x7f002008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:53.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.138+0000 7f001f7fe700 1 --2- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002008c6d0 0x7f002008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:53.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.138+0000 7f001e7fc700 1 -- 172.21.15.16:0/3641209788 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493406464 0 0) 0x7f0018001150 con 0x7f002008ef20 2019-12-05T06:17:53.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.139+0000 7f001e7fc700 1 -- 172.21.15.16:0/3641209788 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0008002e80 con 0x7f002008ef20 2019-12-05T06:17:53.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.139+0000 7f001effd700 1 -- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002008c6d0 msgr2=0x7f002008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:53.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.139+0000 7f001effd700 1 --2- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002008c6d0 0x7f002008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:53.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.139+0000 7f001effd700 1 -- 172.21.15.16:0/3641209788 >> v1:172.21.15.16:6789/0 conn(0x7f002008ef20 legacy=0x7f002008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:53.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.139+0000 7f001effd700 1 -- 172.21.15.16:0/3641209788 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002008d540 con 0x7f002008fef0 2019-12-05T06:17:53.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.139+0000 7f001f7fe700 1 --2- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002008c6d0 0x7f002008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:17:53.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.139+0000 7f001effd700 1 --2- 172.21.15.16:0/3641209788 >> v2:172.21.15.76:3300/0 conn(0x7f002008fef0 0x7f002008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f001400af50 tx=0x7f001400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.139+0000 7f001effd700 1 --2- 172.21.15.16:0/3641209788 >> v2:172.21.15.76:3300/0 conn(0x7f002008fef0 0x7f002008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f001400af50 tx=0x7f001400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.139+0000 7f001effd700 1 --2- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f002008fef0 0x7f002008d000 secure :-1 s=READY pgs=378 cs=0 l=1 rx=0x7f001400af50 tx=0x7f001400b760).ready entity=mon.1 client_cookie=a9f3a5094de40b1e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:53.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.140+0000 7f001effd700 1 --2- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f002008fef0 0x7f002008d000 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7f001400af50 tx=0x7f001400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.140+0000 7f001e7fc700 1 -- 172.21.15.16:0/3641209788 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0014007210 con 0x7f002008fef0 2019-12-05T06:17:53.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.140+0000 7f001effd700 1 --2- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f002008fef0 0x7f002008d000 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7f001400af50 tx=0x7f001400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.140+0000 7f001effd700 1 --2- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f002008fef0 0x7f002008d000 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7f001400af50 tx=0x7f001400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.140+0000 7f001e7fc700 1 -- 172.21.15.16:0/3641209788 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00140099b0 con 0x7f002008fef0 2019-12-05T06:17:53.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.140+0000 7f001e7fc700 1 -- 172.21.15.16:0/3641209788 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f001400a080 con 0x7f002008fef0 2019-12-05T06:17:53.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.140+0000 7f002567d700 1 -- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f002008fef0 msgr2=0x7f002008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:53.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.140+0000 7f002567d700 1 --2- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f002008fef0 0x7f002008d000 secure :-1 s=READY pgs=378 cs=0 l=1 rx=0x7f001400af50 tx=0x7f001400b760).stop 2019-12-05T06:17:53.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.141+0000 7f002567d700 1 -- 172.21.15.16:0/3641209788 shutdown_connections 2019-12-05T06:17:53.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.141+0000 7f002567d700 1 --2- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f002008fef0 0x7f002008d000 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:53.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.141+0000 7f002567d700 1 --2- 172.21.15.16:0/3641209788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002008c6d0 0x7f002008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.142+0000 7f002567d700 1 -- 172.21.15.16:0/3641209788 shutdown_connections 2019-12-05T06:17:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.142+0000 7f002567d700 1 -- 172.21.15.16:0/3641209788 wait complete. 2019-12-05T06:17:53.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.142+0000 7f002567d700 1 -- 172.21.15.16:0/3641209788 >> 172.21.15.16:0/3641209788 conn(0x7f0020061940 msgr2=0x7f0020096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:53.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.146+0000 7f002567d700 1 Processor -- start 2019-12-05T06:17:53.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.147+0000 7f002567d700 1 -- start start 2019-12-05T06:17:53.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f002567d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:53.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f002567d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0020153b50 0x7f0020155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:53.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f002567d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0020131bf0 con 0x7f00201302f0 2019-12-05T06:17:53.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f002567d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00201296e0 con 0x7f0020153b50 2019-12-05T06:17:53.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f002567d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0020128cb0 con 0x7f002012b640 2019-12-05T06:17:53.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f001ffff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0020153b50 0x7f0020155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:53.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f001effd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:53.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f001f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f00201302f0 0x7f002012a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57062/0 (socket says 172.21.15.16:57062) 2019-12-05T06:17:53.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f001effd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:54582/0 (socket says 172.21.15.16:54582) 2019-12-05T06:17:53.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f001ffff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0020153b50 0x7f0020155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:36686/0 (socket says 172.21.15.16:36686) 2019-12-05T06:17:53.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.148+0000 7f001f7fe700 1 -- 172.21.15.16:0/2188344565 learned_addr learned my addr 172.21.15.16:0/2188344565 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:17:53.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.149+0000 7f001effd700 1 -- 172.21.15.16:0/2188344565 >> v2:172.21.15.76:3300/0 conn(0x7f0020153b50 msgr2=0x7f0020155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.149+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> v2:172.21.15.76:3300/0 conn(0x7f0020153b50 0x7f0020155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.149+0000 7f001effd700 1 -- 172.21.15.16:0/2188344565 >> v1:172.21.15.76:6789/0 conn(0x7f00201302f0 legacy=0x7f002012a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.149+0000 7f001effd700 1 -- 172.21.15.16:0/2188344565 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020129d70 con 0x7f002012b640 2019-12-05T06:17:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.149+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.149+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.150+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.150+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.150+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.150+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.150+0000 7f0007fff700 1 -- 172.21.15.16:0/2188344565 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0014009960 con 0x7f002012b640 2019-12-05T06:17:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.150+0000 7f0007fff700 1 -- 172.21.15.16:0/2188344565 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0014009580 con 0x7f002012b640 2019-12-05T06:17:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.150+0000 7f002567d700 1 -- 172.21.15.16:0/2188344565 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0020129d70 con 0x7f002012b640 2019-12-05T06:17:53.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.150+0000 7f0007fff700 1 -- 172.21.15.16:0/2188344565 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0014009f40 con 0x7f002012b640 2019-12-05T06:17:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.150+0000 7f002567d700 1 -- 172.21.15.16:0/2188344565 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0020129d70 con 0x7f002012b640 2019-12-05T06:17:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.151+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.152+0000 7f0007fff700 1 -- 172.21.15.16:0/2188344565 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0014009580 con 0x7f002012b640 2019-12-05T06:17:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.152+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.152+0000 7f0007fff700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0000023630 0x7f0000025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:53.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.153+0000 7f0007fff700 1 -- 172.21.15.16:0/2188344565 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7f0014045d00 con 0x7f002012b640 2019-12-05T06:17:53.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.153+0000 7f001f7fe700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0000023630 0x7f0000025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:53.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.153+0000 7f001f7fe700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0000023630 0x7f0000025ac0 crc :-1 s=READY pgs=223 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:53.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.167+0000 7f002567d700 1 -- 172.21.15.16:0/2188344565 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0020129d70 con 0x7f002012b640 2019-12-05T06:17:53.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.176+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.176+0000 7f0007fff700 1 -- 172.21.15.16:0/2188344565 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f001400b3d0 con 0x7f002012b640 2019-12-05T06:17:53.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.415+0000 7f002567d700 1 -- 172.21.15.16:0/2188344565 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0020129d70 con 0x7f002012b640 2019-12-05T06:17:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.416+0000 7f001effd700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=THROTTLE_DONE pgs=127 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:53.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.416+0000 7f0007fff700 1 -- 172.21.15.16:0/2188344565 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f0014009580 con 0x7f002012b640 2019-12-05T06:17:53.410 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:17:53.410 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5382,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:43.123168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":12,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:53.410 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 62 2019-12-05T06:17:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.420+0000 7f002567d700 1 -- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0000023630 msgr2=0x7f0000025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.420+0000 7f002567d700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0000023630 0x7f0000025ac0 crc :-1 s=READY pgs=223 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:53.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.420+0000 7f002567d700 1 -- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 msgr2=0x7f002012ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.420+0000 7f002567d700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 secure :-1 s=READY pgs=127 cs=0 l=1 rx=0x7f0014010370 tx=0x7f00140101f0).stop 2019-12-05T06:17:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.420+0000 7f002567d700 1 -- 172.21.15.16:0/2188344565 shutdown_connections 2019-12-05T06:17:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.420+0000 7f002567d700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0000023630 0x7f0000025ac0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.420+0000 7f002567d700 1 --2- 172.21.15.16:0/2188344565 >> v2:172.21.15.76:3300/0 conn(0x7f0020153b50 0x7f0020155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.420+0000 7f002567d700 1 --2- 172.21.15.16:0/2188344565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f002012b640 0x7f002012ae70 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.421+0000 7f002567d700 1 -- 172.21.15.16:0/2188344565 shutdown_connections 2019-12-05T06:17:53.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.421+0000 7f002567d700 1 -- 172.21.15.16:0/2188344565 wait complete. 2019-12-05T06:17:53.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:53.421+0000 7f002567d700 1 -- 172.21.15.16:0/2188344565 >> 172.21.15.16:0/2188344565 conn(0x7f00200a2850 msgr2=0x7f002006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:53.424 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:17:55.426 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.682+0000 7fdaa3445700 1 Processor -- start 2019-12-05T06:17:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.682+0000 7fdaa3445700 1 -- start start 2019-12-05T06:17:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.684+0000 7fdaa3445700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 0x7fda9c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:55.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.684+0000 7fdaa3445700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9c0958e0 con 0x7fda9c08fef0 2019-12-05T06:17:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.684+0000 7fdaa3445700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9c0629e0 con 0x7fda9c08c6d0 2019-12-05T06:17:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.684+0000 7fdaa3445700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9c08df90 con 0x7fda9c08d2c0 2019-12-05T06:17:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.684+0000 7fdaa21e3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fda9c08fef0 0x7fda9c08f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35770/0 (socket says 172.21.15.16:35770) 2019-12-05T06:17:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.684+0000 7fdaa21e3700 1 -- 172.21.15.16:0/1163407850 learned_addr learned my addr 172.21.15.16:0/1163407850 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:17:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.684+0000 7fdaa11e1700 1 --2- 172.21.15.16:0/1163407850 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 0x7fda9c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.685+0000 7fdaa09e0700 1 -- 172.21.15.16:0/1163407850 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780420947 0 0) 0x7fda8c001150 con 0x7fda9c08fef0 2019-12-05T06:17:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.685+0000 7fdaa09e0700 1 -- 172.21.15.16:0/1163407850 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda84002e80 con 0x7fda9c08fef0 2019-12-05T06:17:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.685+0000 7fdaa09e0700 1 -- 172.21.15.16:0/1163407850 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345791635 0 0) 0x7fda94001400 con 0x7fda9c08c6d0 2019-12-05T06:17:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.685+0000 7fdaa09e0700 1 -- 172.21.15.16:0/1163407850 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda84002ec0 con 0x7fda9c08c6d0 2019-12-05T06:17:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.685+0000 7fdaa11e1700 1 -- 172.21.15.16:0/1163407850 >> v1:172.21.15.76:6789/0 conn(0x7fda9c08c6d0 legacy=0x7fda9c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.685+0000 7fdaa11e1700 1 -- 172.21.15.16:0/1163407850 >> v1:172.21.15.16:6789/0 conn(0x7fda9c08fef0 legacy=0x7fda9c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.685+0000 7fdaa11e1700 1 -- 172.21.15.16:0/1163407850 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c08e620 con 0x7fda9c08d2c0 2019-12-05T06:17:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.685+0000 7fdaa11e1700 1 --2- 172.21.15.16:0/1163407850 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 0x7fda9c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fda9000af50 tx=0x7fda9000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.686+0000 7fdaa11e1700 1 --2- 172.21.15.16:0/1163407850 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 0x7fda9c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fda9000af50 tx=0x7fda9000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.686+0000 7fdaa11e1700 1 --2- 172.21.15.16:0/1163407850 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 0x7fda9c08ef20 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7fda9000af50 tx=0x7fda9000b760).ready entity=mon.2 client_cookie=d6c81266f7437393 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.686+0000 7fdaa11e1700 1 --2- 172.21.15.16:0/1163407850 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 0x7fda9c08ef20 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7fda9000af50 tx=0x7fda9000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.686+0000 7fdaa11e1700 1 --2- 172.21.15.16:0/1163407850 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 0x7fda9c08ef20 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7fda9000af50 tx=0x7fda9000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.686+0000 7fdaa09e0700 1 -- 172.21.15.16:0/1163407850 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda90000960 con 0x7fda9c08d2c0 2019-12-05T06:17:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.686+0000 7fdaa11e1700 1 --2- 172.21.15.16:0/1163407850 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 0x7fda9c08ef20 secure :-1 s=THROTTLE_DONE pgs=128 cs=0 l=1 rx=0x7fda9000af50 tx=0x7fda9000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.686+0000 7fdaa09e0700 1 -- 172.21.15.16:0/1163407850 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda900098e0 con 0x7fda9c08d2c0 2019-12-05T06:17:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.686+0000 7fdaa09e0700 1 -- 172.21.15.16:0/1163407850 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda90009eb0 con 0x7fda9c08d2c0 2019-12-05T06:17:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.686+0000 7fdaa3445700 1 -- 172.21.15.16:0/1163407850 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 msgr2=0x7fda9c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.686+0000 7fdaa3445700 1 --2- 172.21.15.16:0/1163407850 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 0x7fda9c08ef20 secure :-1 s=READY pgs=128 cs=0 l=1 rx=0x7fda9000af50 tx=0x7fda9000b760).stop 2019-12-05T06:17:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.688+0000 7fdaa3445700 1 -- 172.21.15.16:0/1163407850 shutdown_connections 2019-12-05T06:17:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.688+0000 7fdaa3445700 1 --2- 172.21.15.16:0/1163407850 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda9c08d2c0 0x7fda9c08ef20 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.689+0000 7fdaa3445700 1 -- 172.21.15.16:0/1163407850 shutdown_connections 2019-12-05T06:17:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.689+0000 7fdaa3445700 1 -- 172.21.15.16:0/1163407850 wait complete. 2019-12-05T06:17:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.689+0000 7fdaa3445700 1 -- 172.21.15.16:0/1163407850 >> 172.21.15.16:0/1163407850 conn(0x7fda9c061940 msgr2=0x7fda9c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.694+0000 7fdaa3445700 1 Processor -- start 2019-12-05T06:17:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.694+0000 7fdaa3445700 1 -- start start 2019-12-05T06:17:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.696+0000 7fdaa3445700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda9c14d5d0 0x7fda9c14d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.696+0000 7fdaa3445700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9c06e250 con 0x7fda9c09dba0 2019-12-05T06:17:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.696+0000 7fdaa3445700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda9c099460 con 0x7fda9c09e2c0 2019-12-05T06:17:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.696+0000 7fdaa3445700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda9c12b3f0 con 0x7fda9c14d5d0 2019-12-05T06:17:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.696+0000 7fdaa19e2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda9c14d5d0 0x7fda9c14d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:55.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.696+0000 7fdaa19e2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda9c14d5d0 0x7fda9c14d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50606/0 (socket says 172.21.15.16:50606) 2019-12-05T06:17:55.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.696+0000 7fdaa19e2700 1 -- 172.21.15.16:0/116487327 learned_addr learned my addr 172.21.15.16:0/116487327 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:17:55.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fdaa19e2700 1 -- 172.21.15.16:0/116487327 >> v1:172.21.15.76:6789/0 conn(0x7fda9c09e2c0 legacy=0x7fda9c09d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:55.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fdaa19e2700 1 -- 172.21.15.16:0/116487327 >> v1:172.21.15.16:6789/0 conn(0x7fda9c09dba0 legacy=0x7fda9c153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fdaa19e2700 1 -- 172.21.15.16:0/116487327 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda9c12ba80 con 0x7fda9c14d5d0 2019-12-05T06:17:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fda9a7fc700 1 -- 172.21.15.16:0/116487327 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338732486 0 0) 0x7fda90000940 con 0x7fda9c09e2c0 2019-12-05T06:17:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fdaa19e2700 1 --2- 172.21.15.16:0/116487327 >> v2:172.21.15.16:3300/0 conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fdaa19e2700 1 --2- 172.21.15.16:0/116487327 >> v2:172.21.15.16:3300/0 conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fdaa19e2700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=READY pgs=494 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fdaa19e2700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=THROTTLE_DONE pgs=494 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fda9a7fc700 1 -- 172.21.15.16:0/116487327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda94005af0 con 0x7fda9c14d5d0 2019-12-05T06:17:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fdaa19e2700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=THROTTLE_DONE pgs=494 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fdaa19e2700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=THROTTLE_DONE pgs=494 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fda9a7fc700 1 -- 172.21.15.16:0/116487327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda94005710 con 0x7fda9c14d5d0 2019-12-05T06:17:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fdaa3445700 1 -- 172.21.15.16:0/116487327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda9c12ba80 con 0x7fda9c14d5d0 2019-12-05T06:17:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.697+0000 7fda9a7fc700 1 -- 172.21.15.16:0/116487327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda940060d0 con 0x7fda9c14d5d0 2019-12-05T06:17:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.698+0000 7fdaa3445700 1 -- 172.21.15.16:0/116487327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda9c12ba80 con 0x7fda9c14d5d0 2019-12-05T06:17:55.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.698+0000 7fdaa19e2700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=THROTTLE_DONE pgs=494 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.699+0000 7fda9a7fc700 1 -- 172.21.15.16:0/116487327 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fda94005900 con 0x7fda9c14d5d0 2019-12-05T06:17:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.699+0000 7fdaa19e2700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=THROTTLE_DONE pgs=494 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.699+0000 7fda9a7fc700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda7c023630 0x7fda7c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.700+0000 7fda9a7fc700 1 -- 172.21.15.16:0/116487327 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 6547+0+0 (secure 0 0 0) 0x7fda94046430 con 0x7fda9c14d5d0 2019-12-05T06:17:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.700+0000 7fdaa11e1700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda7c023630 0x7fda7c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:55.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.700+0000 7fdaa11e1700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda7c023630 0x7fda7c025ac0 crc :-1 s=READY pgs=224 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:55.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.714+0000 7fdaa3445700 1 -- 172.21.15.16:0/116487327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda9c12ba80 con 0x7fda9c14d5d0 2019-12-05T06:17:55.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.721+0000 7fdaa19e2700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=THROTTLE_DONE pgs=494 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.721+0000 7fda9a7fc700 1 -- 172.21.15.16:0/116487327 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fda94007920 con 0x7fda9c14d5d0 2019-12-05T06:17:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.962+0000 7fdaa3445700 1 -- 172.21.15.16:0/116487327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda9c12ba80 con 0x7fda9c14d5d0 2019-12-05T06:17:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.963+0000 7fdaa19e2700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=THROTTLE_DONE pgs=494 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:55.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.963+0000 7fda9a7fc700 1 -- 172.21.15.16:0/116487327 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fda94005710 con 0x7fda9c14d5d0 2019-12-05T06:17:55.958 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 62 2019-12-05T06:17:55.959 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:17:55.959 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":62,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5382,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":62}],"filesystems":[{"mdsmap":{"epoch":61,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:16:43.123168+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":55,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5295},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5295":{"gid":5295,"name":"a","rank":0,"incarnation":58,"state":"up:active","state_seq":12,"addr":"172.21.15.16:6833/1580628040","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1580628040},{"type":"v1","addr":"172.21.15.16:6833","nonce":1580628040}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:55.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.965+0000 7fdaa3445700 1 -- 172.21.15.16:0/116487327 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda7c023630 msgr2=0x7fda7c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:55.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.966+0000 7fdaa3445700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda7c023630 0x7fda7c025ac0 crc :-1 s=READY pgs=224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:55.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.966+0000 7fdaa3445700 1 -- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 msgr2=0x7fda9c14d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:55.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.966+0000 7fdaa3445700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 0x7fda9c14d9c0 secure :-1 s=READY pgs=494 cs=0 l=1 rx=0x7fda94010790 tx=0x7fda94010b30).stop 2019-12-05T06:17:55.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.966+0000 7fdaa3445700 1 -- 172.21.15.16:0/116487327 shutdown_connections 2019-12-05T06:17:55.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.966+0000 7fdaa3445700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda7c023630 0x7fda7c025ac0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:55.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.966+0000 7fdaa3445700 1 --2- 172.21.15.16:0/116487327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda9c14d5d0 0x7fda9c14d9c0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:55.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.966+0000 7fdaa3445700 1 -- 172.21.15.16:0/116487327 shutdown_connections 2019-12-05T06:17:55.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.966+0000 7fdaa3445700 1 -- 172.21.15.16:0/116487327 wait complete. 2019-12-05T06:17:55.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:55.966+0000 7fdaa3445700 1 -- 172.21.15.16:0/116487327 >> 172.21.15.16:0/116487327 conn(0x7fda9c06d260 msgr2=0x7fda9c09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:55.969 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:17:57.970 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.239+0000 7f0744219700 1 Processor -- start 2019-12-05T06:17:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.239+0000 7f0744219700 1 -- start start 2019-12-05T06:17:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.241+0000 7f0744219700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f073c08d2c0 0x7f073c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.241+0000 7f0744219700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f073c08fef0 0x7f073c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.241+0000 7f0744219700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c0958e0 con 0x7f073c08c6d0 2019-12-05T06:17:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.241+0000 7f0744219700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073c08df90 con 0x7f073c08d2c0 2019-12-05T06:17:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.241+0000 7f0744219700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f073c0a3f70 con 0x7f073c08fef0 2019-12-05T06:17:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.241+0000 7f0741fb5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f073c08d2c0 0x7f073c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.241+0000 7f07427b6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f073c08c6d0 0x7f073c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35784/0 (socket says 172.21.15.16:35784) 2019-12-05T06:17:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.242+0000 7f07427b6700 1 -- 172.21.15.16:0/934030186 learned_addr learned my addr 172.21.15.16:0/934030186 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:17:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.242+0000 7f0742fb7700 1 --2- 172.21.15.16:0/934030186 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f073c08fef0 0x7f073c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.242+0000 7f0741fb5700 1 -- 172.21.15.16:0/934030186 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f073c08fef0 msgr2=0x7f073c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:17:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.242+0000 7f0741fb5700 1 --2- 172.21.15.16:0/934030186 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f073c08fef0 0x7f073c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.242+0000 7f0741fb5700 1 -- 172.21.15.16:0/934030186 >> v1:172.21.15.16:6789/0 conn(0x7f073c08c6d0 legacy=0x7f073c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.242+0000 7f07417b4700 1 -- 172.21.15.16:0/934030186 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581264216 0 0) 0x7f0738001150 con 0x7f073c08c6d0 2019-12-05T06:17:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.242+0000 7f0741fb5700 1 -- 172.21.15.16:0/934030186 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c08e620 con 0x7f073c08d2c0 2019-12-05T06:17:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f0741fb5700 1 --2- 172.21.15.16:0/934030186 >> v2:172.21.15.16:3300/0 conn(0x7f073c08d2c0 0x7f073c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f072c00af50 tx=0x7f072c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f0741fb5700 1 --2- 172.21.15.16:0/934030186 >> v2:172.21.15.16:3300/0 conn(0x7f073c08d2c0 0x7f073c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f072c00af50 tx=0x7f072c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f0741fb5700 1 --2- 172.21.15.16:0/934030186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c08d2c0 0x7f073c08ef20 secure :-1 s=READY pgs=496 cs=0 l=1 rx=0x7f072c00af50 tx=0x7f072c00b6e0).ready entity=mon.0 client_cookie=c6ec6321771228b8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f0741fb5700 1 --2- 172.21.15.16:0/934030186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c08d2c0 0x7f073c08ef20 secure :-1 s=THROTTLE_DONE pgs=496 cs=0 l=1 rx=0x7f072c00af50 tx=0x7f072c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f07417b4700 1 -- 172.21.15.16:0/934030186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f072c007210 con 0x7f073c08d2c0 2019-12-05T06:17:58.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f0741fb5700 1 --2- 172.21.15.16:0/934030186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c08d2c0 0x7f073c08ef20 secure :-1 s=THROTTLE_DONE pgs=496 cs=0 l=1 rx=0x7f072c00af50 tx=0x7f072c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f0741fb5700 1 --2- 172.21.15.16:0/934030186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c08d2c0 0x7f073c08ef20 secure :-1 s=THROTTLE_DONE pgs=496 cs=0 l=1 rx=0x7f072c00af50 tx=0x7f072c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f07417b4700 1 -- 172.21.15.16:0/934030186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f072c0099b0 con 0x7f073c08d2c0 2019-12-05T06:17:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f07417b4700 1 -- 172.21.15.16:0/934030186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f072c00a080 con 0x7f073c08d2c0 2019-12-05T06:17:58.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f0744219700 1 -- 172.21.15.16:0/934030186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c08d2c0 msgr2=0x7f073c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.243+0000 7f0744219700 1 --2- 172.21.15.16:0/934030186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c08d2c0 0x7f073c08ef20 secure :-1 s=READY pgs=496 cs=0 l=1 rx=0x7f072c00af50 tx=0x7f072c00b6e0).stop 2019-12-05T06:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.245+0000 7f0744219700 1 -- 172.21.15.16:0/934030186 shutdown_connections 2019-12-05T06:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.245+0000 7f0744219700 1 --2- 172.21.15.16:0/934030186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c08d2c0 0x7f073c08ef20 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.245+0000 7f0744219700 1 --2- 172.21.15.16:0/934030186 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f073c08fef0 0x7f073c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.246+0000 7f0744219700 1 -- 172.21.15.16:0/934030186 shutdown_connections 2019-12-05T06:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.246+0000 7f0744219700 1 -- 172.21.15.16:0/934030186 wait complete. 2019-12-05T06:17:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.246+0000 7f0744219700 1 -- 172.21.15.16:0/934030186 >> 172.21.15.16:0/934030186 conn(0x7f073c061940 msgr2=0x7f073c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.251+0000 7f0744219700 1 Processor -- start 2019-12-05T06:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.251+0000 7f0744219700 1 -- start start 2019-12-05T06:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.253+0000 7f0744219700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f073c084f30 0x7f073c157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.253+0000 7f0744219700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c084b90 con 0x7f073c09bbe0 2019-12-05T06:17:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.253+0000 7f0744219700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f073c098790 con 0x7f073c096e90 2019-12-05T06:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.253+0000 7f0744219700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f073c09a450 con 0x7f073c084f30 2019-12-05T06:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.253+0000 7f0742fb7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f073c084f30 0x7f073c157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.253+0000 7f0742fb7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f073c084f30 0x7f073c157200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50626/0 (socket says 172.21.15.16:50626) 2019-12-05T06:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.253+0000 7f0742fb7700 1 -- 172.21.15.16:0/4282844679 learned_addr learned my addr 172.21.15.16:0/4282844679 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.253+0000 7f0741fb5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f073c09bbe0 0x7f073c1539e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35792/0 (socket says 172.21.15.16:35792) 2019-12-05T06:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.253+0000 7f07427b6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f073c096e90 0x7f073c09b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57092/0 (socket says 172.21.15.16:57092) 2019-12-05T06:17:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.253+0000 7f0732ffd700 1 -- 172.21.15.16:0/4282844679 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1387256857 0 0) 0x7f0738001100 con 0x7f073c096e90 2019-12-05T06:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0732ffd700 1 -- 172.21.15.16:0/4282844679 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0724002e80 con 0x7f073c096e90 2019-12-05T06:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0742fb7700 1 -- 172.21.15.16:0/4282844679 >> v1:172.21.15.76:6789/0 conn(0x7f073c096e90 legacy=0x7f073c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0742fb7700 1 -- 172.21.15.16:0/4282844679 >> v1:172.21.15.16:6789/0 conn(0x7f073c09bbe0 legacy=0x7f073c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0732ffd700 1 -- 172.21.15.16:0/4282844679 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409413163 0 0) 0x7f072c00ac00 con 0x7f073c09bbe0 2019-12-05T06:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0742fb7700 1 -- 172.21.15.16:0/4282844679 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073c09aae0 con 0x7f073c084f30 2019-12-05T06:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0742fb7700 1 --2- 172.21.15.16:0/4282844679 >> v2:172.21.15.16:3300/0 conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0742fb7700 1 --2- 172.21.15.16:0/4282844679 >> v2:172.21.15.16:3300/0 conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0742fb7700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=READY pgs=498 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0742fb7700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0732ffd700 1 -- 172.21.15.16:0/4282844679 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f073400f410 con 0x7f073c084f30 2019-12-05T06:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0742fb7700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.254+0000 7f0742fb7700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.255+0000 7f0732ffd700 1 -- 172.21.15.16:0/4282844679 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0734000df0 con 0x7f073c084f30 2019-12-05T06:17:58.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.255+0000 7f0744219700 1 -- 172.21.15.16:0/4282844679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f073c09aae0 con 0x7f073c084f30 2019-12-05T06:17:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.255+0000 7f0732ffd700 1 -- 172.21.15.16:0/4282844679 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f073400f9f0 con 0x7f073c084f30 2019-12-05T06:17:58.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.255+0000 7f0744219700 1 -- 172.21.15.16:0/4282844679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f073c09aae0 con 0x7f073c084f30 2019-12-05T06:17:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.256+0000 7f0742fb7700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.257+0000 7f0732ffd700 1 -- 172.21.15.16:0/4282844679 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f073400f0d0 con 0x7f073c084f30 2019-12-05T06:17:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.257+0000 7f0742fb7700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.257+0000 7f0732ffd700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0724023680 0x7f0724025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.257+0000 7f0732ffd700 1 -- 172.21.15.16:0/4282844679 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f07340461c0 con 0x7f073c084f30 2019-12-05T06:17:58.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.257+0000 7f07427b6700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0724023680 0x7f0724025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:58.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.258+0000 7f07427b6700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0724023680 0x7f0724025af0 crc :-1 s=READY pgs=225 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:58.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.271+0000 7f0744219700 1 -- 172.21.15.16:0/4282844679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f073c09aae0 con 0x7f073c084f30 2019-12-05T06:17:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.278+0000 7f0742fb7700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.278+0000 7f0732ffd700 1 -- 172.21.15.16:0/4282844679 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0734036f80 con 0x7f073c084f30 2019-12-05T06:17:58.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.532+0000 7f0744219700 1 -- 172.21.15.16:0/4282844679 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f073c09aae0 con 0x7f073c084f30 2019-12-05T06:17:58.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.533+0000 7f0742fb7700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=THROTTLE_DONE pgs=498 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.533+0000 7f0732ffd700 1 -- 172.21.15.16:0/4282844679 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f0734036f80 con 0x7f073c084f30 2019-12-05T06:17:58.525 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 63 2019-12-05T06:17:58.526 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:17:58.526 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:17:58.067270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:58.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.535+0000 7f0744219700 1 -- 172.21.15.16:0/4282844679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0724023680 msgr2=0x7f0724025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.535+0000 7f0744219700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0724023680 0x7f0724025af0 crc :-1 s=READY pgs=225 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.535+0000 7f0744219700 1 -- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 msgr2=0x7f073c157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.535+0000 7f0744219700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 0x7f073c157200 secure :-1 s=READY pgs=498 cs=0 l=1 rx=0x7f07340107d0 tx=0x7f0734010b70).stop 2019-12-05T06:17:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.535+0000 7f0744219700 1 -- 172.21.15.16:0/4282844679 shutdown_connections 2019-12-05T06:17:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.536+0000 7f0744219700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0724023680 0x7f0724025af0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:58.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.536+0000 7f0744219700 1 --2- 172.21.15.16:0/4282844679 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f073c084f30 0x7f073c157200 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.536+0000 7f0744219700 1 -- 172.21.15.16:0/4282844679 shutdown_connections 2019-12-05T06:17:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.536+0000 7f0744219700 1 -- 172.21.15.16:0/4282844679 wait complete. 2019-12-05T06:17:58.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.536+0000 7f0744219700 1 -- 172.21.15.16:0/4282844679 >> 172.21.15.16:0/4282844679 conn(0x7f073c0714f0 msgr2=0x7f073c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:58.538 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:17:58.538 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:17:58.538 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:17:58.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.786+0000 7fed9dd1c700 1 Processor -- start 2019-12-05T06:17:58.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.786+0000 7fed9dd1c700 1 -- start start 2019-12-05T06:17:58.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.788+0000 7fed9dd1c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 0x7fed9808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:17:58.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.788+0000 7fed9dd1c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed980958e0 con 0x7fed9808c6d0 2019-12-05T06:17:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.788+0000 7fed9dd1c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed980629e0 con 0x7fed9808d2c0 2019-12-05T06:17:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.788+0000 7fed9dd1c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed9808df90 con 0x7fed9808fef0 2019-12-05T06:17:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.788+0000 7fed97fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fed9808c6d0 0x7fed9808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35798/0 (socket says 172.21.15.16:35798) 2019-12-05T06:17:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.788+0000 7fed97fff700 1 -- 172.21.15.16:0/4170900875 learned_addr learned my addr 172.21.15.16:0/4170900875 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:17:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.788+0000 7fed9caba700 1 --2- 172.21.15.16:0/4170900875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 0x7fed9808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.788+0000 7fed96ffd700 1 -- 172.21.15.16:0/4170900875 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4260025524 0 0) 0x7fed88001150 con 0x7fed9808c6d0 2019-12-05T06:17:58.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.789+0000 7fed96ffd700 1 -- 172.21.15.16:0/4170900875 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed80002e80 con 0x7fed9808c6d0 2019-12-05T06:17:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.789+0000 7fed96ffd700 1 -- 172.21.15.16:0/4170900875 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664027709 0 0) 0x7fed8c001080 con 0x7fed9808d2c0 2019-12-05T06:17:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.789+0000 7fed96ffd700 1 -- 172.21.15.16:0/4170900875 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed80004170 con 0x7fed9808d2c0 2019-12-05T06:17:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.789+0000 7fed9caba700 1 -- 172.21.15.16:0/4170900875 >> v1:172.21.15.76:6789/0 conn(0x7fed9808d2c0 legacy=0x7fed9808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.789+0000 7fed9caba700 1 -- 172.21.15.16:0/4170900875 >> v1:172.21.15.16:6789/0 conn(0x7fed9808c6d0 legacy=0x7fed9808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.789+0000 7fed9caba700 1 -- 172.21.15.16:0/4170900875 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9808e620 con 0x7fed9808fef0 2019-12-05T06:17:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.789+0000 7fed9caba700 1 --2- 172.21.15.16:0/4170900875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 0x7fed9808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fed9000b300 tx=0x7fed9000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.789+0000 7fed9caba700 1 --2- 172.21.15.16:0/4170900875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 0x7fed9808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fed9000b300 tx=0x7fed9000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.789+0000 7fed9caba700 1 --2- 172.21.15.16:0/4170900875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 0x7fed9808f720 secure :-1 s=READY pgs=129 cs=0 l=1 rx=0x7fed9000b300 tx=0x7fed9000bb10).ready entity=mon.2 client_cookie=c6d5723ab32a50da server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.790+0000 7fed9caba700 1 --2- 172.21.15.16:0/4170900875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 0x7fed9808f720 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7fed9000b300 tx=0x7fed9000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.790+0000 7fed96ffd700 1 -- 172.21.15.16:0/4170900875 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed900075c0 con 0x7fed9808fef0 2019-12-05T06:17:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.790+0000 7fed9caba700 1 --2- 172.21.15.16:0/4170900875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 0x7fed9808f720 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7fed9000b300 tx=0x7fed9000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.790+0000 7fed9caba700 1 --2- 172.21.15.16:0/4170900875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 0x7fed9808f720 secure :-1 s=THROTTLE_DONE pgs=129 cs=0 l=1 rx=0x7fed9000b300 tx=0x7fed9000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.790+0000 7fed96ffd700 1 -- 172.21.15.16:0/4170900875 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed90000f30 con 0x7fed9808fef0 2019-12-05T06:17:58.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.790+0000 7fed96ffd700 1 -- 172.21.15.16:0/4170900875 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed9000a1e0 con 0x7fed9808fef0 2019-12-05T06:17:58.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.790+0000 7fed9dd1c700 1 -- 172.21.15.16:0/4170900875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 msgr2=0x7fed9808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.790+0000 7fed9dd1c700 1 --2- 172.21.15.16:0/4170900875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 0x7fed9808f720 secure :-1 s=READY pgs=129 cs=0 l=1 rx=0x7fed9000b300 tx=0x7fed9000bb10).stop 2019-12-05T06:17:58.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.792+0000 7fed9dd1c700 1 -- 172.21.15.16:0/4170900875 shutdown_connections 2019-12-05T06:17:58.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.792+0000 7fed9dd1c700 1 --2- 172.21.15.16:0/4170900875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9808fef0 0x7fed9808f720 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:58.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.792+0000 7fed9dd1c700 1 -- 172.21.15.16:0/4170900875 shutdown_connections 2019-12-05T06:17:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.792+0000 7fed9dd1c700 1 -- 172.21.15.16:0/4170900875 wait complete. 2019-12-05T06:17:58.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.792+0000 7fed9dd1c700 1 -- 172.21.15.16:0/4170900875 >> 172.21.15.16:0/4170900875 conn(0x7fed98061940 msgr2=0x7fed98096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:58.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.796+0000 7fed9dd1c700 1 Processor -- start 2019-12-05T06:17:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.796+0000 7fed9dd1c700 1 -- start start 2019-12-05T06:17:58.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.797+0000 7fed9dd1c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed98129ae0 0x7fed98129310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:58.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.797+0000 7fed9dd1c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fed9815b170 0x7fed9815d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed9dd1c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed98132930 con 0x7fed980829b0 2019-12-05T06:17:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed9dd1c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed9812fff0 con 0x7fed98129ae0 2019-12-05T06:17:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed9dd1c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed98131040 con 0x7fed9815b170 2019-12-05T06:17:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed98129ae0 0x7fed98129310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed98129ae0 0x7fed98129310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50636/0 (socket says 172.21.15.16:50636) 2019-12-05T06:17:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed9caba700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fed9815b170 0x7fed9815d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:58.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed977fe700 1 -- 172.21.15.16:0/2499329893 learned_addr learned my addr 172.21.15.16:0/2499329893 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:17:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed977fe700 1 -- 172.21.15.16:0/2499329893 >> v2:172.21.15.76:3300/0 conn(0x7fed9815b170 msgr2=0x7fed9815d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> v2:172.21.15.76:3300/0 conn(0x7fed9815b170 0x7fed9815d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed977fe700 1 -- 172.21.15.16:0/2499329893 >> v1:172.21.15.76:6789/0 conn(0x7fed980829b0 legacy=0x7fed98128b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:58.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed977fe700 1 -- 172.21.15.16:0/2499329893 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed98131f00 con 0x7fed98129ae0 2019-12-05T06:17:58.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed9caba700 1 --2- 172.21.15.16:0/2499329893 >> v2:172.21.15.76:3300/0 conn(0x7fed9815b170 0x7fed9815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:17:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.798+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> v2:172.21.15.16:3300/0 conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.799+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> v2:172.21.15.16:3300/0 conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.799+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=READY pgs=500 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.799+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.799+0000 7fed94ff9700 1 -- 172.21.15.16:0/2499329893 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed8c005770 con 0x7fed98129ae0 2019-12-05T06:17:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.799+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.799+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.799+0000 7fed94ff9700 1 -- 172.21.15.16:0/2499329893 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed8c005390 con 0x7fed98129ae0 2019-12-05T06:17:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.799+0000 7fed94ff9700 1 -- 172.21.15.16:0/2499329893 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed8c005d50 con 0x7fed98129ae0 2019-12-05T06:17:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.799+0000 7fed9dd1c700 1 -- 172.21.15.16:0/2499329893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed98131f00 con 0x7fed98129ae0 2019-12-05T06:17:58.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.799+0000 7fed9dd1c700 1 -- 172.21.15.16:0/2499329893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed98131f00 con 0x7fed98129ae0 2019-12-05T06:17:58.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.800+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.801+0000 7fed94ff9700 1 -- 172.21.15.16:0/2499329893 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fed8c005580 con 0x7fed98129ae0 2019-12-05T06:17:58.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.801+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.801+0000 7fed94ff9700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed78023630 0x7fed78025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:17:58.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.801+0000 7fed94ff9700 1 -- 172.21.15.16:0/2499329893 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7fed8c045d90 con 0x7fed98129ae0 2019-12-05T06:17:58.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.801+0000 7fed97fff700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed78023630 0x7fed78025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:17:58.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.802+0000 7fed97fff700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed78023630 0x7fed78025ac0 crc :-1 s=READY pgs=226 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:17:58.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.817+0000 7fed9dd1c700 1 -- 172.21.15.16:0/2499329893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed98132930 con 0x7fed98129ae0 2019-12-05T06:17:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.825+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:58.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:58.825+0000 7fed94ff9700 1 -- 172.21.15.16:0/2499329893 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fed8c0075a0 con 0x7fed98129ae0 2019-12-05T06:17:59.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.076+0000 7fed9dd1c700 1 -- 172.21.15.16:0/2499329893 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed98132930 con 0x7fed98129ae0 2019-12-05T06:17:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.077+0000 7fed977fe700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=THROTTLE_DONE pgs=500 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:17:59.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.077+0000 7fed94ff9700 1 -- 172.21.15.16:0/2499329893 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 63 v63) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fed8c03f4c0 con 0x7fed98129ae0 2019-12-05T06:17:59.069 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:17:59.070 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":63,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":63,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:17:58.067270+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:17:59.070 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 63 2019-12-05T06:17:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.079+0000 7fed9dd1c700 1 -- 172.21.15.16:0/2499329893 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed78023630 msgr2=0x7fed78025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.079+0000 7fed9dd1c700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed78023630 0x7fed78025ac0 crc :-1 s=READY pgs=226 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.080+0000 7fed9dd1c700 1 -- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 msgr2=0x7fed98129310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:17:59.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.080+0000 7fed9dd1c700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 0x7fed98129310 secure :-1 s=READY pgs=500 cs=0 l=1 rx=0x7fed8c010410 tx=0x7fed8c0107b0).stop 2019-12-05T06:17:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.080+0000 7fed9dd1c700 1 -- 172.21.15.16:0/2499329893 shutdown_connections 2019-12-05T06:17:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.080+0000 7fed9dd1c700 1 --2- 172.21.15.16:0/2499329893 >> v2:172.21.15.76:3300/0 conn(0x7fed9815b170 0x7fed9815d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.080+0000 7fed9dd1c700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed78023630 0x7fed78025ac0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.080+0000 7fed9dd1c700 1 --2- 172.21.15.16:0/2499329893 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed98129ae0 0x7fed98129310 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:17:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.080+0000 7fed9dd1c700 1 -- 172.21.15.16:0/2499329893 shutdown_connections 2019-12-05T06:17:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.080+0000 7fed9dd1c700 1 -- 172.21.15.16:0/2499329893 wait complete. 2019-12-05T06:17:59.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:17:59.080+0000 7fed9dd1c700 1 -- 172.21.15.16:0/2499329893 >> 172.21.15.16:0/2499329893 conn(0x7fed980a2850 msgr2=0x7fed9806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:17:59.090 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 63, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 63, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5382": { "addr": "172.21.15.76:6835/1423600579", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 1423600579, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 1423600579, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5382, "incarnation": 63, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 56, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:17:58.067270+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5382 } } } ], "standbys": [] } 2019-12-05T06:18:01.091 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:18:01.272 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:18:01.276 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.343+0000 7fc9c63a9700 1 Processor -- start 2019-12-05T06:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.343+0000 7fc9c63a9700 1 -- start start 2019-12-05T06:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.344+0000 7fc9c63a9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc9c008c6d0 0x7fc9c008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:01.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c63a9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c008f720 0x7fc9c008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c63a9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c008fb70 con 0x7fc9c008fef0 2019-12-05T06:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c63a9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c008df90 con 0x7fc9c008c6d0 2019-12-05T06:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c63a9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c00a3f70 con 0x7fc9c008f720 2019-12-05T06:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c4946700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc9c008c6d0 0x7fc9c008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9b7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc9c008fef0 0x7fc9c008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35816/0 (socket says 172.21.15.16:35816) 2019-12-05T06:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c4946700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc9c008c6d0 0x7fc9c008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50646/0 (socket says 172.21.15.16:50646) 2019-12-05T06:18:01.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9b7fff700 1 -- 172.21.15.16:0/2148027075 learned_addr learned my addr 172.21.15.16:0/2148027075 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:18:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c5147700 1 --2- 172.21.15.16:0/2148027075 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c008f720 0x7fc9c008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c4946700 1 -- 172.21.15.16:0/2148027075 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c008f720 msgr2=0x7fc9c008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:18:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c4946700 1 --2- 172.21.15.16:0/2148027075 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c008f720 0x7fc9c008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:18:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c4946700 1 -- 172.21.15.16:0/2148027075 >> v1:172.21.15.16:6789/0 conn(0x7fc9c008fef0 legacy=0x7fc9c008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9b77fe700 1 -- 172.21.15.16:0/2148027075 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2782574243 0 0) 0x7fc9b0001100 con 0x7fc9c008fef0 2019-12-05T06:18:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.345+0000 7fc9c4946700 1 -- 172.21.15.16:0/2148027075 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c008d540 con 0x7fc9c008c6d0 2019-12-05T06:18:01.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9c4946700 1 --2- 172.21.15.16:0/2148027075 >> v2:172.21.15.16:3300/0 conn(0x7fc9c008c6d0 0x7fc9c008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc9bc00af50 tx=0x7fc9bc00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9c4946700 1 --2- 172.21.15.16:0/2148027075 >> v2:172.21.15.16:3300/0 conn(0x7fc9c008c6d0 0x7fc9c008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc9bc00af50 tx=0x7fc9bc00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9c4946700 1 --2- 172.21.15.16:0/2148027075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9c008c6d0 0x7fc9c008cac0 secure :-1 s=READY pgs=502 cs=0 l=1 rx=0x7fc9bc00af50 tx=0x7fc9bc00b6e0).ready entity=mon.0 client_cookie=771d854289bb4f26 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:01.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9c4946700 1 --2- 172.21.15.16:0/2148027075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9c008c6d0 0x7fc9c008cac0 secure :-1 s=THROTTLE_DONE pgs=502 cs=0 l=1 rx=0x7fc9bc00af50 tx=0x7fc9bc00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9b77fe700 1 -- 172.21.15.16:0/2148027075 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9bc007210 con 0x7fc9c008c6d0 2019-12-05T06:18:01.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9c4946700 1 --2- 172.21.15.16:0/2148027075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9c008c6d0 0x7fc9c008cac0 secure :-1 s=THROTTLE_DONE pgs=502 cs=0 l=1 rx=0x7fc9bc00af50 tx=0x7fc9bc00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9b77fe700 1 -- 172.21.15.16:0/2148027075 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9bc000960 con 0x7fc9c008c6d0 2019-12-05T06:18:01.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9c4946700 1 --2- 172.21.15.16:0/2148027075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9c008c6d0 0x7fc9c008cac0 secure :-1 s=THROTTLE_DONE pgs=502 cs=0 l=1 rx=0x7fc9bc00af50 tx=0x7fc9bc00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9b77fe700 1 -- 172.21.15.16:0/2148027075 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9bc009ba0 con 0x7fc9c008c6d0 2019-12-05T06:18:01.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9c63a9700 1 -- 172.21.15.16:0/2148027075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9c008c6d0 msgr2=0x7fc9c008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:01.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.346+0000 7fc9c63a9700 1 --2- 172.21.15.16:0/2148027075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9c008c6d0 0x7fc9c008cac0 secure :-1 s=READY pgs=502 cs=0 l=1 rx=0x7fc9bc00af50 tx=0x7fc9bc00b6e0).stop 2019-12-05T06:18:01.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.347+0000 7fc9c63a9700 1 -- 172.21.15.16:0/2148027075 shutdown_connections 2019-12-05T06:18:01.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.347+0000 7fc9c63a9700 1 --2- 172.21.15.16:0/2148027075 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c008f720 0x7fc9c008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:18:01.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.347+0000 7fc9c63a9700 1 --2- 172.21.15.16:0/2148027075 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc9c008c6d0 0x7fc9c008cac0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:01.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.348+0000 7fc9c63a9700 1 -- 172.21.15.16:0/2148027075 shutdown_connections 2019-12-05T06:18:01.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.348+0000 7fc9c63a9700 1 -- 172.21.15.16:0/2148027075 wait complete. 2019-12-05T06:18:01.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.348+0000 7fc9c63a9700 1 -- 172.21.15.16:0/2148027075 >> 172.21.15.16:0/2148027075 conn(0x7fc9c0061940 msgr2=0x7fc9c0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:01.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.352+0000 7fc9c63a9700 1 Processor -- start 2019-12-05T06:18:01.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.352+0000 7fc9c63a9700 1 -- start start 2019-12-05T06:18:01.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.353+0000 7fc9c63a9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc9c00a09a0 0x7fc9c00845d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:01.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.353+0000 7fc9c63a9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:01.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.353+0000 7fc9c63a9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c0087c40 con 0x7fc9c00855a0 2019-12-05T06:18:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.353+0000 7fc9c63a9700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c0089900 con 0x7fc9c00a09a0 2019-12-05T06:18:01.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.353+0000 7fc9c63a9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c0088ed0 con 0x7fc9c0151ac0 2019-12-05T06:18:01.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9b7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc9c00855a0 0x7fc9c0084b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35822/0 (socket says 172.21.15.16:35822) 2019-12-05T06:18:01.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9b7fff700 1 -- 172.21.15.16:0/3087413619 learned_addr learned my addr 172.21.15.16:0/3087413619 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:18:01.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9c4946700 1 --2- 172.21.15.16:0/3087413619 >> v2:172.21.15.76:3300/0 conn(0x7fc9c00a09a0 0x7fc9c00845d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:01.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:01.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9b57fa700 1 -- 172.21.15.16:0/3087413619 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146060894 0 0) 0x7fc9b0001170 con 0x7fc9c00855a0 2019-12-05T06:18:01.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9b57fa700 1 -- 172.21.15.16:0/3087413619 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9a0002e80 con 0x7fc9c00855a0 2019-12-05T06:18:01.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9c5147700 1 -- 172.21.15.16:0/3087413619 >> v2:172.21.15.76:3300/0 conn(0x7fc9c00a09a0 msgr2=0x7fc9c00845d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> v2:172.21.15.76:3300/0 conn(0x7fc9c00a09a0 0x7fc9c00845d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9c5147700 1 -- 172.21.15.16:0/3087413619 >> v1:172.21.15.16:6789/0 conn(0x7fc9c00855a0 legacy=0x7fc9c0084b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9c5147700 1 -- 172.21.15.16:0/3087413619 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c0089f90 con 0x7fc9c0151ac0 2019-12-05T06:18:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9c4946700 1 --2- 172.21.15.16:0/3087413619 >> v2:172.21.15.76:3300/0 conn(0x7fc9c00a09a0 0x7fc9c00845d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:18:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.354+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.355+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.355+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.355+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.355+0000 7fc9b57fa700 1 -- 172.21.15.16:0/3087413619 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9b8009d10 con 0x7fc9c0151ac0 2019-12-05T06:18:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.355+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.355+0000 7fc9c63a9700 1 -- 172.21.15.16:0/3087413619 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c0089f90 con 0x7fc9c0151ac0 2019-12-05T06:18:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.355+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.355+0000 7fc9b57fa700 1 -- 172.21.15.16:0/3087413619 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9b8009930 con 0x7fc9c0151ac0 2019-12-05T06:18:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.355+0000 7fc9b57fa700 1 -- 172.21.15.16:0/3087413619 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9b8009b20 con 0x7fc9c0151ac0 2019-12-05T06:18:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.355+0000 7fc9c63a9700 1 -- 172.21.15.16:0/3087413619 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c0089f90 con 0x7fc9c0151ac0 2019-12-05T06:18:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.356+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.357+0000 7fc9b57fa700 1 -- 172.21.15.16:0/3087413619 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc9b8009b20 con 0x7fc9c0151ac0 2019-12-05T06:18:01.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.357+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.357+0000 7fc9b57fa700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc9a0023670 0x7fc9a0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.357+0000 7fc9b57fa700 1 -- 172.21.15.16:0/3087413619 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7fc9b80482a0 con 0x7fc9c0151ac0 2019-12-05T06:18:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.357+0000 7fc9c4946700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc9a0023670 0x7fc9a0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.357+0000 7fc9c63a9700 1 -- 172.21.15.16:0/3087413619 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c0089f90 con 0x7fc9c0151ac0 2019-12-05T06:18:01.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.358+0000 7fc9c4946700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc9a0023670 0x7fc9a0025ae0 crc :-1 s=READY pgs=227 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:01.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.365+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.365+0000 7fc9b57fa700 1 -- 172.21.15.16:0/3087413619 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc9b80329b0 con 0x7fc9c0151ac0 2019-12-05T06:18:01.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.564+0000 7fc9c63a9700 1 -- 172.21.15.16:0/3087413619 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9c0089f90 con 0x7fc9c0151ac0 2019-12-05T06:18:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.566+0000 7fc9c5147700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=THROTTLE_DONE pgs=130 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:01.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.566+0000 7fc9b57fa700 1 -- 172.21.15.16:0/3087413619 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 65 v65) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fc9b80329b0 con 0x7fc9c0151ac0 2019-12-05T06:18:01.559 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 65 2019-12-05T06:18:01.559 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:18:01.559 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":65,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":65,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:01.174830+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:18:01.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.569+0000 7fc9c63a9700 1 -- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc9a0023670 msgr2=0x7fc9a0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:01.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.569+0000 7fc9c63a9700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc9a0023670 0x7fc9a0025ae0 crc :-1 s=READY pgs=227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:01.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.569+0000 7fc9c63a9700 1 -- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 msgr2=0x7fc9c0151eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:01.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.569+0000 7fc9c63a9700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 secure :-1 s=READY pgs=130 cs=0 l=1 rx=0x7fc9b8012930 tx=0x7fc9b8012cd0).stop 2019-12-05T06:18:01.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.570+0000 7fc9c63a9700 1 -- 172.21.15.16:0/3087413619 shutdown_connections 2019-12-05T06:18:01.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.570+0000 7fc9c63a9700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc9a0023670 0x7fc9a0025ae0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:01.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.570+0000 7fc9c63a9700 1 --2- 172.21.15.16:0/3087413619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc9c0151ac0 0x7fc9c0151eb0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:01.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.570+0000 7fc9c63a9700 1 --2- 172.21.15.16:0/3087413619 >> v2:172.21.15.76:3300/0 conn(0x7fc9c00a09a0 0x7fc9c00845d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:01.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.570+0000 7fc9c63a9700 1 -- 172.21.15.16:0/3087413619 shutdown_connections 2019-12-05T06:18:01.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.570+0000 7fc9c63a9700 1 -- 172.21.15.16:0/3087413619 wait complete. 2019-12-05T06:18:01.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:01.570+0000 7fc9c63a9700 1 -- 172.21.15.16:0/3087413619 >> 172.21.15.16:0/3087413619 conn(0x7fc9c00710b0 msgr2=0x7fc9c012d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:01.573 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:18:03.574 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:18:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.853+0000 7f24a8c6f700 1 Processor -- start 2019-12-05T06:18:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.853+0000 7f24a8c6f700 1 -- start start 2019-12-05T06:18:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.855+0000 7f24a8c6f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24a408c6d0 0x7f24a408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.855+0000 7f24a8c6f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f24a408f720 0x7f24a408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.855+0000 7f24a8c6f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a408fb80 con 0x7f24a408fef0 2019-12-05T06:18:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.855+0000 7f24a8c6f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a408df90 con 0x7f24a408c6d0 2019-12-05T06:18:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.855+0000 7f24a8c6f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a40a3f70 con 0x7f24a408f720 2019-12-05T06:18:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.856+0000 7f24a2d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24a408c6d0 0x7f24a408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.856+0000 7f24a359e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f24a408f720 0x7f24a408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.856+0000 7f24a2d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24a408c6d0 0x7f24a408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50660/0 (socket says 172.21.15.16:50660) 2019-12-05T06:18:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.856+0000 7f24a259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f24a408fef0 0x7f24a408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35832/0 (socket says 172.21.15.16:35832) 2019-12-05T06:18:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.856+0000 7f24a2d9d700 1 -- 172.21.15.16:0/318439448 learned_addr learned my addr 172.21.15.16:0/318439448 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:18:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.856+0000 7f24a2d9d700 1 -- 172.21.15.16:0/318439448 >> v2:172.21.15.76:3300/0 conn(0x7f24a408f720 msgr2=0x7f24a408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:18:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.856+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/318439448 >> v2:172.21.15.76:3300/0 conn(0x7f24a408f720 0x7f24a408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:18:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.856+0000 7f24a2d9d700 1 -- 172.21.15.16:0/318439448 >> v1:172.21.15.16:6789/0 conn(0x7f24a408fef0 legacy=0x7f24a408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.857+0000 7f24a2d9d700 1 -- 172.21.15.16:0/318439448 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a408d540 con 0x7f24a408c6d0 2019-12-05T06:18:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.857+0000 7f24a1d9b700 1 -- 172.21.15.16:0/318439448 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997452240 0 0) 0x7f248c001100 con 0x7f24a408fef0 2019-12-05T06:18:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.857+0000 7f24a359e700 1 --2- 172.21.15.16:0/318439448 >> v2:172.21.15.76:3300/0 conn(0x7f24a408f720 0x7f24a408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:18:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.857+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/318439448 >> v2:172.21.15.16:3300/0 conn(0x7f24a408c6d0 0x7f24a408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f249400fac0 tx=0x7f249400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.857+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/318439448 >> v2:172.21.15.16:3300/0 conn(0x7f24a408c6d0 0x7f24a408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f249400fac0 tx=0x7f249400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.857+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/318439448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24a408c6d0 0x7f24a408cac0 secure :-1 s=READY pgs=505 cs=0 l=1 rx=0x7f249400fac0 tx=0x7f249400fe60).ready entity=mon.0 client_cookie=34b12b89647eab98 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.857+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/318439448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24a408c6d0 0x7f24a408cac0 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7f249400fac0 tx=0x7f249400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.857+0000 7f24a1d9b700 1 -- 172.21.15.16:0/318439448 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2494000ce0 con 0x7f24a408c6d0 2019-12-05T06:18:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.857+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/318439448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24a408c6d0 0x7f24a408cac0 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7f249400fac0 tx=0x7f249400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.857+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/318439448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24a408c6d0 0x7f24a408cac0 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7f249400fac0 tx=0x7f249400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.858+0000 7f24a1d9b700 1 -- 172.21.15.16:0/318439448 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2494001010 con 0x7f24a408c6d0 2019-12-05T06:18:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.858+0000 7f24a1d9b700 1 -- 172.21.15.16:0/318439448 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2494001720 con 0x7f24a408c6d0 2019-12-05T06:18:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.858+0000 7f24a8c6f700 1 -- 172.21.15.16:0/318439448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24a408c6d0 msgr2=0x7f24a408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.858+0000 7f24a8c6f700 1 --2- 172.21.15.16:0/318439448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24a408c6d0 0x7f24a408cac0 secure :-1 s=READY pgs=505 cs=0 l=1 rx=0x7f249400fac0 tx=0x7f249400fe60).stop 2019-12-05T06:18:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.860+0000 7f24a8c6f700 1 -- 172.21.15.16:0/318439448 shutdown_connections 2019-12-05T06:18:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.860+0000 7f24a8c6f700 1 --2- 172.21.15.16:0/318439448 >> v2:172.21.15.76:3300/0 conn(0x7f24a408f720 0x7f24a408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:18:03.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.860+0000 7f24a8c6f700 1 --2- 172.21.15.16:0/318439448 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24a408c6d0 0x7f24a408cac0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:03.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.860+0000 7f24a8c6f700 1 -- 172.21.15.16:0/318439448 shutdown_connections 2019-12-05T06:18:03.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.860+0000 7f24a8c6f700 1 -- 172.21.15.16:0/318439448 wait complete. 2019-12-05T06:18:03.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.860+0000 7f24a8c6f700 1 -- 172.21.15.16:0/318439448 >> 172.21.15.16:0/318439448 conn(0x7f24a4061940 msgr2=0x7f24a4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:03.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.865+0000 7f24a8c6f700 1 Processor -- start 2019-12-05T06:18:03.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.865+0000 7f24a8c6f700 1 -- start start 2019-12-05T06:18:03.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.867+0000 7f24a8c6f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f24a4132630 0x7f24a4132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:03.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.867+0000 7f24a8c6f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a409ab10 con 0x7f24a409ae80 2019-12-05T06:18:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.867+0000 7f24a8c6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24a406e7c0 con 0x7f24a4099eb0 2019-12-05T06:18:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.867+0000 7f24a8c6f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24a4098f20 con 0x7f24a4132630 2019-12-05T06:18:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.867+0000 7f24a259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f24a409ae80 0x7f24a4132f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35836/0 (socket says 172.21.15.16:35836) 2019-12-05T06:18:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.867+0000 7f24a259c700 1 -- 172.21.15.16:0/321253486 learned_addr learned my addr 172.21.15.16:0/321253486 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:18:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.867+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> v2:172.21.15.76:3300/0 conn(0x7f24a4132630 0x7f24a4132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:03.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.867+0000 7f249b7fe700 1 -- 172.21.15.16:0/321253486 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534659385 0 0) 0x7f248c001150 con 0x7f24a409ae80 2019-12-05T06:18:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.868+0000 7f249b7fe700 1 -- 172.21.15.16:0/321253486 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2488002e80 con 0x7f24a409ae80 2019-12-05T06:18:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.868+0000 7f24a2d9d700 1 -- 172.21.15.16:0/321253486 >> v1:172.21.15.76:6789/0 conn(0x7f24a4099eb0 legacy=0x7f24a409a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.868+0000 7f24a2d9d700 1 -- 172.21.15.16:0/321253486 >> v1:172.21.15.16:6789/0 conn(0x7f24a409ae80 legacy=0x7f24a4132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.868+0000 7f249b7fe700 1 -- 172.21.15.16:0/321253486 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320895649 0 0) 0x7f249c0097c0 con 0x7f24a4099eb0 2019-12-05T06:18:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.868+0000 7f24a2d9d700 1 -- 172.21.15.16:0/321253486 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a40995b0 con 0x7f24a4132630 2019-12-05T06:18:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.868+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> v2:172.21.15.76:3300/0 conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.868+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> v2:172.21.15.76:3300/0 conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.868+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=READY pgs=386 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.869+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.869+0000 7f249b7fe700 1 -- 172.21.15.16:0/321253486 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2494008190 con 0x7f24a4132630 2019-12-05T06:18:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.869+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.869+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.869+0000 7f249b7fe700 1 -- 172.21.15.16:0/321253486 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2494005710 con 0x7f24a4132630 2019-12-05T06:18:03.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.869+0000 7f249b7fe700 1 -- 172.21.15.16:0/321253486 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2494005e80 con 0x7f24a4132630 2019-12-05T06:18:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.869+0000 7f24a8c6f700 1 -- 172.21.15.16:0/321253486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a40995b0 con 0x7f24a4132630 2019-12-05T06:18:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.869+0000 7f24a8c6f700 1 -- 172.21.15.16:0/321253486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a40995b0 con 0x7f24a4132630 2019-12-05T06:18:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.870+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.871+0000 7f249b7fe700 1 -- 172.21.15.16:0/321253486 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2494008210 con 0x7f24a4132630 2019-12-05T06:18:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.871+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.871+0000 7f249b7fe700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2488023680 0x7f2488025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:03.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.871+0000 7f249b7fe700 1 -- 172.21.15.16:0/321253486 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f249408a160 con 0x7f24a4132630 2019-12-05T06:18:03.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.872+0000 7f24a259c700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2488023680 0x7f2488025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:03.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.872+0000 7f24a259c700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2488023680 0x7f2488025af0 crc :-1 s=READY pgs=228 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:03.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.885+0000 7f24a8c6f700 1 -- 172.21.15.16:0/321253486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a40995b0 con 0x7f24a4132630 2019-12-05T06:18:03.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.895+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:03.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:03.895+0000 7f249b7fe700 1 -- 172.21.15.16:0/321253486 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2494005500 con 0x7f24a4132630 2019-12-05T06:18:04.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.138+0000 7f24a8c6f700 1 -- 172.21.15.16:0/321253486 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24a40995b0 con 0x7f24a4132630 2019-12-05T06:18:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.139+0000 7f24a2d9d700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:04.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.139+0000 7f249b7fe700 1 -- 172.21.15.16:0/321253486 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f2494008170 con 0x7f24a4132630 2019-12-05T06:18:04.133 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 66 2019-12-05T06:18:04.133 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:18:04.134 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:18:04.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.144+0000 7f24a8c6f700 1 -- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2488023680 msgr2=0x7f2488025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.144+0000 7f24a8c6f700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2488023680 0x7f2488025af0 crc :-1 s=READY pgs=228 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.144+0000 7f24a8c6f700 1 -- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 msgr2=0x7f24a4132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.144+0000 7f24a8c6f700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 0x7f24a4132a20 secure :-1 s=READY pgs=386 cs=0 l=1 rx=0x7f2494054740 tx=0x7f2494054ae0).stop 2019-12-05T06:18:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.144+0000 7f24a8c6f700 1 -- 172.21.15.16:0/321253486 shutdown_connections 2019-12-05T06:18:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.144+0000 7f24a8c6f700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2488023680 0x7f2488025af0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.144+0000 7f24a8c6f700 1 --2- 172.21.15.16:0/321253486 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f24a4132630 0x7f24a4132a20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:04.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.144+0000 7f24a8c6f700 1 -- 172.21.15.16:0/321253486 shutdown_connections 2019-12-05T06:18:04.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.145+0000 7f24a8c6f700 1 -- 172.21.15.16:0/321253486 wait complete. 2019-12-05T06:18:04.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:04.145+0000 7f24a8c6f700 1 -- 172.21.15.16:0/321253486 >> 172.21.15.16:0/321253486 conn(0x7f24a40714f0 msgr2=0x7f24a409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:04.147 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5295 2019-12-05T06:18:04.147 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2019-12-05T06:18:12.148 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:18:12.148 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:18:12.149 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:18:12.156 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:18:12.156 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:18:12.324 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:18:12.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.389+0000 7f82be9b7700 1 Processor -- start 2019-12-05T06:18:12.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.389+0000 7f82be9b7700 1 -- start start 2019-12-05T06:18:12.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.390+0000 7f82be9b7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82b808d2c0 0x7f82b808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.390+0000 7f82be9b7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b80958e0 con 0x7f82b808fef0 2019-12-05T06:18:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.390+0000 7f82be9b7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b80629e0 con 0x7f82b808c6d0 2019-12-05T06:18:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.390+0000 7f82be9b7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b808df90 con 0x7f82b808d2c0 2019-12-05T06:18:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.390+0000 7f82b7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82b808d2c0 0x7f82b808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82b808d2c0 0x7f82b808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50696/0 (socket says 172.21.15.16:50696) 2019-12-05T06:18:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 -- 172.21.15.16:0/2732314643 learned_addr learned my addr 172.21.15.16:0/2732314643 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:18:12.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 -- 172.21.15.16:0/2732314643 >> v1:172.21.15.76:6789/0 conn(0x7f82b808c6d0 legacy=0x7f82b808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 -- 172.21.15.16:0/2732314643 >> v1:172.21.15.16:6789/0 conn(0x7f82b808fef0 legacy=0x7f82b808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 -- 172.21.15.16:0/2732314643 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b808e620 con 0x7f82b808d2c0 2019-12-05T06:18:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 --2- 172.21.15.16:0/2732314643 >> v2:172.21.15.16:3300/0 conn(0x7f82b808d2c0 0x7f82b808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f82b000fac0 tx=0x7f82b000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 --2- 172.21.15.16:0/2732314643 >> v2:172.21.15.16:3300/0 conn(0x7f82b808d2c0 0x7f82b808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f82b000fac0 tx=0x7f82b000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 --2- 172.21.15.16:0/2732314643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b808d2c0 0x7f82b808ef20 secure :-1 s=READY pgs=511 cs=0 l=1 rx=0x7f82b000fac0 tx=0x7f82b000fe60).ready entity=mon.0 client_cookie=c334ea848150f086 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 --2- 172.21.15.16:0/2732314643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b808d2c0 0x7f82b808ef20 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7f82b000fac0 tx=0x7f82b000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b77fe700 1 -- 172.21.15.16:0/2732314643 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82b0000ce0 con 0x7f82b808d2c0 2019-12-05T06:18:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 --2- 172.21.15.16:0/2732314643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b808d2c0 0x7f82b808ef20 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7f82b000fac0 tx=0x7f82b000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b7fff700 1 --2- 172.21.15.16:0/2732314643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b808d2c0 0x7f82b808ef20 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7f82b000fac0 tx=0x7f82b000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b77fe700 1 -- 172.21.15.16:0/2732314643 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b0001010 con 0x7f82b808d2c0 2019-12-05T06:18:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.391+0000 7f82b77fe700 1 -- 172.21.15.16:0/2732314643 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82b00016e0 con 0x7f82b808d2c0 2019-12-05T06:18:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.392+0000 7f82be9b7700 1 -- 172.21.15.16:0/2732314643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b808d2c0 msgr2=0x7f82b808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.392+0000 7f82be9b7700 1 --2- 172.21.15.16:0/2732314643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b808d2c0 0x7f82b808ef20 secure :-1 s=READY pgs=511 cs=0 l=1 rx=0x7f82b000fac0 tx=0x7f82b000fe60).stop 2019-12-05T06:18:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.392+0000 7f82be9b7700 1 -- 172.21.15.16:0/2732314643 shutdown_connections 2019-12-05T06:18:12.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.392+0000 7f82be9b7700 1 --2- 172.21.15.16:0/2732314643 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b808d2c0 0x7f82b808ef20 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:12.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.393+0000 7f82be9b7700 1 -- 172.21.15.16:0/2732314643 shutdown_connections 2019-12-05T06:18:12.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.393+0000 7f82be9b7700 1 -- 172.21.15.16:0/2732314643 wait complete. 2019-12-05T06:18:12.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.393+0000 7f82be9b7700 1 -- 172.21.15.16:0/2732314643 >> 172.21.15.16:0/2732314643 conn(0x7f82b8061940 msgr2=0x7f82b8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:12.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.397+0000 7f82be9b7700 1 Processor -- start 2019-12-05T06:18:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.397+0000 7f82be9b7700 1 -- start start 2019-12-05T06:18:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.399+0000 7f82be9b7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82b814ca10 0x7f82b80869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.399+0000 7f82be9b7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82b80879b0 0x7f82b8086f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.399+0000 7f82be9b7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f82b809e030 con 0x7f82b8087460 2019-12-05T06:18:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.399+0000 7f82be9b7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82b8085020 con 0x7f82b80879b0 2019-12-05T06:18:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.399+0000 7f82be9b7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82b80852f0 con 0x7f82b814ca10 2019-12-05T06:18:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.399+0000 7f82b7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82b80879b0 0x7f82b8086f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.399+0000 7f82b7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f82b80879b0 0x7f82b8086f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50700/0 (socket says 172.21.15.16:50700) 2019-12-05T06:18:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.399+0000 7f82b7fff700 1 -- 172.21.15.16:0/1359756729 learned_addr learned my addr 172.21.15.16:0/1359756729 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:18:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.399+0000 7f82bcf54700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82b814ca10 0x7f82b80869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b7fff700 1 -- 172.21.15.16:0/1359756729 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82b814ca10 msgr2=0x7f82b80869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82b814ca10 0x7f82b80869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:12.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b7fff700 1 -- 172.21.15.16:0/1359756729 >> v1:172.21.15.76:6789/0 conn(0x7f82b8087460 legacy=0x7f82b809be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b7fff700 1 -- 172.21.15.16:0/1359756729 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82b80860e0 con 0x7f82b80879b0 2019-12-05T06:18:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> v2:172.21.15.16:3300/0 conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> v2:172.21.15.16:3300/0 conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=READY pgs=512 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=THROTTLE_DONE pgs=512 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=THROTTLE_DONE pgs=512 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b57fa700 1 -- 172.21.15.16:0/1359756729 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82b0000cc0 con 0x7f82b80879b0 2019-12-05T06:18:12.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=THROTTLE_DONE pgs=512 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b57fa700 1 -- 172.21.15.16:0/1359756729 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f82b00012d0 con 0x7f82b80879b0 2019-12-05T06:18:12.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82be9b7700 1 -- 172.21.15.16:0/1359756729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b80860e0 con 0x7f82b80879b0 2019-12-05T06:18:12.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82b57fa700 1 -- 172.21.15.16:0/1359756729 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82b00019e0 con 0x7f82b80879b0 2019-12-05T06:18:12.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.400+0000 7f82be9b7700 1 -- 172.21.15.16:0/1359756729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b80860e0 con 0x7f82b80879b0 2019-12-05T06:18:12.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.401+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=THROTTLE_DONE pgs=512 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.402+0000 7f82b57fa700 1 -- 172.21.15.16:0/1359756729 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f82b0000d60 con 0x7f82b80879b0 2019-12-05T06:18:12.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.402+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=THROTTLE_DONE pgs=512 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.402+0000 7f82b57fa700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f829c023630 0x7f829c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:12.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.402+0000 7f82b57fa700 1 -- 172.21.15.16:0/1359756729 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f82b0046410 con 0x7f82b80879b0 2019-12-05T06:18:12.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.402+0000 7f82bcf54700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f829c023630 0x7f829c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:12.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.403+0000 7f82bcf54700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f829c023630 0x7f829c025ae0 crc :-1 s=READY pgs=229 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:12.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.403+0000 7f82be9b7700 1 -- 172.21.15.16:0/1359756729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b80860e0 con 0x7f82b80879b0 2019-12-05T06:18:12.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.407+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=THROTTLE_DONE pgs=512 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.407+0000 7f82b57fa700 1 -- 172.21.15.16:0/1359756729 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f82b0007ac0 con 0x7f82b80879b0 2019-12-05T06:18:12.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.630+0000 7f82be9b7700 1 -- 172.21.15.16:0/1359756729 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f82b80860e0 con 0x7f82b80879b0 2019-12-05T06:18:12.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.631+0000 7f82b7fff700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=THROTTLE_DONE pgs=512 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:12.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.631+0000 7f82b57fa700 1 -- 172.21.15.16:0/1359756729 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f82b0000cc0 con 0x7f82b80879b0 2019-12-05T06:18:12.623 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 66 2019-12-05T06:18:12.623 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:18:12.623 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:18:12.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.633+0000 7f82be9b7700 1 -- 172.21.15.16:0/1359756729 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f829c023630 msgr2=0x7f829c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:12.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.633+0000 7f82be9b7700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f829c023630 0x7f829c025ae0 crc :-1 s=READY pgs=229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:12.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.633+0000 7f82be9b7700 1 -- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 msgr2=0x7f82b8086f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:12.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.633+0000 7f82be9b7700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 0x7f82b8086f20 secure :-1 s=READY pgs=512 cs=0 l=1 rx=0x7f82b0010800 tx=0x7f82b0010600).stop 2019-12-05T06:18:12.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.634+0000 7f82be9b7700 1 -- 172.21.15.16:0/1359756729 shutdown_connections 2019-12-05T06:18:12.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.634+0000 7f82be9b7700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f829c023630 0x7f829c025ae0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:12.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.634+0000 7f82be9b7700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f82b80879b0 0x7f82b8086f20 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:12.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.634+0000 7f82be9b7700 1 --2- 172.21.15.16:0/1359756729 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f82b814ca10 0x7f82b80869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:12.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.634+0000 7f82be9b7700 1 -- 172.21.15.16:0/1359756729 shutdown_connections 2019-12-05T06:18:12.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.634+0000 7f82be9b7700 1 -- 172.21.15.16:0/1359756729 wait complete. 2019-12-05T06:18:12.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:12.634+0000 7f82be9b7700 1 -- 172.21.15.16:0/1359756729 >> 172.21.15.16:0/1359756729 conn(0x7f82b8098480 msgr2=0x7f82b8071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:12.639 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:18:14.641 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:18:14.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.927+0000 7f7e1d85d700 1 Processor -- start 2019-12-05T06:18:14.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.927+0000 7f7e1d85d700 1 -- start start 2019-12-05T06:18:14.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.929+0000 7f7e1d85d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7e1808c6d0 0x7f7e1808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:14.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.929+0000 7f7e1d85d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e1808f720 0x7f7e1808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:14.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.929+0000 7f7e1d85d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e1808fb80 con 0x7f7e1808fef0 2019-12-05T06:18:14.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.929+0000 7f7e1d85d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1808df90 con 0x7f7e1808f720 2019-12-05T06:18:14.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.929+0000 7f7e1d85d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e180a3f70 con 0x7f7e1808c6d0 2019-12-05T06:18:14.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.929+0000 7f7e17fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e1808f720 0x7f7e1808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:14.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.929+0000 7f7e16ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e1808fef0 0x7f7e1808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35880/0 (socket says 172.21.15.16:35880) 2019-12-05T06:18:14.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.929+0000 7f7e17fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e1808f720 0x7f7e1808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50710/0 (socket says 172.21.15.16:50710) 2019-12-05T06:18:14.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.929+0000 7f7e16ffd700 1 -- 172.21.15.16:0/4243500186 learned_addr learned my addr 172.21.15.16:0/4243500186 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:18:14.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.929+0000 7f7e177fe700 1 --2- 172.21.15.16:0/4243500186 >> v2:172.21.15.76:3300/0 conn(0x7f7e1808c6d0 0x7f7e1808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:14.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.930+0000 7f7e17fff700 1 -- 172.21.15.16:0/4243500186 >> v2:172.21.15.76:3300/0 conn(0x7f7e1808c6d0 msgr2=0x7f7e1808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:18:14.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.930+0000 7f7e17fff700 1 --2- 172.21.15.16:0/4243500186 >> v2:172.21.15.76:3300/0 conn(0x7f7e1808c6d0 0x7f7e1808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:18:14.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.930+0000 7f7e17fff700 1 -- 172.21.15.16:0/4243500186 >> v1:172.21.15.16:6789/0 conn(0x7f7e1808fef0 legacy=0x7f7e1808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:14.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.930+0000 7f7e167fc700 1 -- 172.21.15.16:0/4243500186 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191686150 0 0) 0x7f7e00001150 con 0x7f7e1808fef0 2019-12-05T06:18:14.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.930+0000 7f7e17fff700 1 -- 172.21.15.16:0/4243500186 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1808d540 con 0x7f7e1808f720 2019-12-05T06:18:14.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.930+0000 7f7e17fff700 1 --2- 172.21.15.16:0/4243500186 >> v2:172.21.15.16:3300/0 conn(0x7f7e1808f720 0x7f7e1808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7e0800af50 tx=0x7f7e0800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.930+0000 7f7e17fff700 1 --2- 172.21.15.16:0/4243500186 >> v2:172.21.15.16:3300/0 conn(0x7f7e1808f720 0x7f7e1808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7e0800af50 tx=0x7f7e0800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.930+0000 7f7e17fff700 1 --2- 172.21.15.16:0/4243500186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1808f720 0x7f7e1808d000 secure :-1 s=READY pgs=514 cs=0 l=1 rx=0x7f7e0800af50 tx=0x7f7e0800b250).ready entity=mon.0 client_cookie=7844f3e30cbd6c3a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:14.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.931+0000 7f7e17fff700 1 --2- 172.21.15.16:0/4243500186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1808f720 0x7f7e1808d000 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f7e0800af50 tx=0x7f7e0800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.931+0000 7f7e167fc700 1 -- 172.21.15.16:0/4243500186 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e08009ba0 con 0x7f7e1808f720 2019-12-05T06:18:14.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.931+0000 7f7e17fff700 1 --2- 172.21.15.16:0/4243500186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1808f720 0x7f7e1808d000 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f7e0800af50 tx=0x7f7e0800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.931+0000 7f7e17fff700 1 --2- 172.21.15.16:0/4243500186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1808f720 0x7f7e1808d000 secure :-1 s=THROTTLE_DONE pgs=514 cs=0 l=1 rx=0x7f7e0800af50 tx=0x7f7e0800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.931+0000 7f7e167fc700 1 -- 172.21.15.16:0/4243500186 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e080097c0 con 0x7f7e1808f720 2019-12-05T06:18:14.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.931+0000 7f7e167fc700 1 -- 172.21.15.16:0/4243500186 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e0800a180 con 0x7f7e1808f720 2019-12-05T06:18:14.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.931+0000 7f7e1d85d700 1 -- 172.21.15.16:0/4243500186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1808f720 msgr2=0x7f7e1808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:14.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.931+0000 7f7e1d85d700 1 --2- 172.21.15.16:0/4243500186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1808f720 0x7f7e1808d000 secure :-1 s=READY pgs=514 cs=0 l=1 rx=0x7f7e0800af50 tx=0x7f7e0800b250).stop 2019-12-05T06:18:14.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.933+0000 7f7e1d85d700 1 -- 172.21.15.16:0/4243500186 shutdown_connections 2019-12-05T06:18:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.933+0000 7f7e1d85d700 1 --2- 172.21.15.16:0/4243500186 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e1808f720 0x7f7e1808d000 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.933+0000 7f7e1d85d700 1 --2- 172.21.15.16:0/4243500186 >> v2:172.21.15.76:3300/0 conn(0x7f7e1808c6d0 0x7f7e1808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:18:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.933+0000 7f7e1d85d700 1 -- 172.21.15.16:0/4243500186 shutdown_connections 2019-12-05T06:18:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.933+0000 7f7e1d85d700 1 -- 172.21.15.16:0/4243500186 wait complete. 2019-12-05T06:18:14.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.933+0000 7f7e1d85d700 1 -- 172.21.15.16:0/4243500186 >> 172.21.15.16:0/4243500186 conn(0x7f7e18061940 msgr2=0x7f7e18096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.938+0000 7f7e1d85d700 1 Processor -- start 2019-12-05T06:18:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.939+0000 7f7e1d85d700 1 -- start start 2019-12-05T06:18:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.940+0000 7f7e1d85d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.940+0000 7f7e1d85d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7e18132d00 0x7f7e18131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.940+0000 7f7e1d85d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e18132270 0x7f7e1806dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:14.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.940+0000 7f7e1d85d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1809bb50 con 0x7f7e18132270 2019-12-05T06:18:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.940+0000 7f7e1d85d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e1809a030 con 0x7f7e18132d00 2019-12-05T06:18:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.941+0000 7f7e1d85d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e1809a300 con 0x7f7e1814d400 2019-12-05T06:18:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.941+0000 7f7e17fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e18132270 0x7f7e1806dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.941+0000 7f7e17fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e18132270 0x7f7e1806dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50718/0 (socket says 172.21.15.16:50718) 2019-12-05T06:18:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.941+0000 7f7e17fff700 1 -- 172.21.15.16:0/2598346904 learned_addr learned my addr 172.21.15.16:0/2598346904 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:18:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.941+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:14.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.941+0000 7f7e16ffd700 1 --2- 172.21.15.16:0/2598346904 >> v2:172.21.15.76:3300/0 conn(0x7f7e18132d00 0x7f7e18131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 -- 172.21.15.16:0/2598346904 >> v2:172.21.15.76:3300/0 conn(0x7f7e18132d00 msgr2=0x7f7e18131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> v2:172.21.15.76:3300/0 conn(0x7f7e18132d00 0x7f7e18131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 -- 172.21.15.16:0/2598346904 >> v2:172.21.15.16:3300/0 conn(0x7f7e18132270 msgr2=0x7f7e1806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> v2:172.21.15.16:3300/0 conn(0x7f7e18132270 0x7f7e1806dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 -- 172.21.15.16:0/2598346904 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e1809b0f0 con 0x7f7e1814d400 2019-12-05T06:18:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e17fff700 1 --2- 172.21.15.16:0/2598346904 >> v2:172.21.15.16:3300/0 conn(0x7f7e18132270 0x7f7e1806dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:18:14.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=READY pgs=132 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:14.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e0ffff700 1 -- 172.21.15.16:0/2598346904 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e10000cc0 con 0x7f7e1814d400 2019-12-05T06:18:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.942+0000 7f7e0ffff700 1 -- 172.21.15.16:0/2598346904 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e10009f50 con 0x7f7e1814d400 2019-12-05T06:18:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.943+0000 7f7e0ffff700 1 -- 172.21.15.16:0/2598346904 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e1000a560 con 0x7f7e1814d400 2019-12-05T06:18:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.943+0000 7f7e1d85d700 1 -- 172.21.15.16:0/2598346904 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e1809b0f0 con 0x7f7e1814d400 2019-12-05T06:18:14.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.943+0000 7f7e1d85d700 1 -- 172.21.15.16:0/2598346904 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e1809b0f0 con 0x7f7e1814d400 2019-12-05T06:18:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.943+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.944+0000 7f7e0ffff700 1 -- 172.21.15.16:0/2598346904 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7e10009970 con 0x7f7e1814d400 2019-12-05T06:18:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.944+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.945+0000 7f7e0ffff700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dfc023630 0x7f7dfc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.945+0000 7f7e0ffff700 1 -- 172.21.15.16:0/2598346904 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f7e10046090 con 0x7f7e1814d400 2019-12-05T06:18:14.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.945+0000 7f7e16ffd700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dfc023630 0x7f7dfc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:14.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.946+0000 7f7e16ffd700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dfc023630 0x7f7dfc025ac0 crc :-1 s=READY pgs=230 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:14.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.958+0000 7f7e1d85d700 1 -- 172.21.15.16:0/2598346904 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e1809b0f0 con 0x7f7e1814d400 2019-12-05T06:18:14.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.968+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:14.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:14.968+0000 7f7e0ffff700 1 -- 172.21.15.16:0/2598346904 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7e1002b300 con 0x7f7e1814d400 2019-12-05T06:18:15.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.219+0000 7f7e1d85d700 1 -- 172.21.15.16:0/2598346904 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e1809b0f0 con 0x7f7e1814d400 2019-12-05T06:18:15.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.221+0000 7f7e177fe700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=THROTTLE_DONE pgs=132 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.221+0000 7f7e0ffff700 1 -- 172.21.15.16:0/2598346904 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 66 v66) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f7e10009970 con 0x7f7e1814d400 2019-12-05T06:18:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 66 2019-12-05T06:18:15.214 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:18:15.214 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":66,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:18:15.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 -- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dfc023630 msgr2=0x7f7dfc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dfc023630 0x7f7dfc025ac0 crc :-1 s=READY pgs=230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 -- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 msgr2=0x7f7e1814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 secure :-1 s=READY pgs=132 cs=0 l=1 rx=0x7f7e10010710 tx=0x7f7e10010ab0).stop 2019-12-05T06:18:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 -- 172.21.15.16:0/2598346904 shutdown_connections 2019-12-05T06:18:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dfc023630 0x7f7dfc025ac0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 --2- 172.21.15.16:0/2598346904 >> v2:172.21.15.16:3300/0 conn(0x7f7e18132270 0x7f7e1806dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:15.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 --2- 172.21.15.16:0/2598346904 >> v2:172.21.15.76:3300/0 conn(0x7f7e18132d00 0x7f7e18131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 --2- 172.21.15.16:0/2598346904 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e1814d400 0x7f7e1814d7f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 -- 172.21.15.16:0/2598346904 shutdown_connections 2019-12-05T06:18:15.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.225+0000 7f7e1d85d700 1 -- 172.21.15.16:0/2598346904 wait complete. 2019-12-05T06:18:15.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:15.226+0000 7f7e1d85d700 1 -- 172.21.15.16:0/2598346904 >> 172.21.15.16:0/2598346904 conn(0x7f7e180714f0 msgr2=0x7f7e1809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:15.233 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:18:17.235 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:18:17.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.509+0000 7f2e02890700 1 Processor -- start 2019-12-05T06:18:17.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.509+0000 7f2e02890700 1 -- start start 2019-12-05T06:18:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.511+0000 7f2e02890700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2dfc08c6d0 0x7f2dfc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.511+0000 7f2e02890700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dfc08fef0 0x7f2dfc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:18:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.511+0000 7f2e02890700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dfc0958e0 con 0x7f2dfc08ef20 2019-12-05T06:18:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.511+0000 7f2e02890700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dfc0a3f70 con 0x7f2dfc08fef0 2019-12-05T06:18:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.511+0000 7f2e02890700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2dfc08d830 con 0x7f2dfc08c6d0 2019-12-05T06:18:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.511+0000 7f2dfbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dfc08fef0 0x7f2dfc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:17.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.511+0000 7f2dfbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dfc08fef0 0x7f2dfc08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50732/0 (socket says 172.21.15.16:50732) 2019-12-05T06:18:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.511+0000 7f2dfbfff700 1 -- 172.21.15.16:0/4227738472 learned_addr learned my addr 172.21.15.16:0/4227738472 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:18:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.511+0000 7f2e0162e700 1 --1- 172.21.15.16:0/4227738472 >> v1:172.21.15.16:6789/0 conn(0x7f2dfc08ef20 0x7f2dfc08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35904/0 (socket says 172.21.15.16:35904) 2019-12-05T06:18:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.511+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/4227738472 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2dfc08c6d0 0x7f2dfc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfbfff700 1 -- 172.21.15.16:0/4227738472 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2dfc08c6d0 msgr2=0x7f2dfc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:18:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfbfff700 1 --2- 172.21.15.16:0/4227738472 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2dfc08c6d0 0x7f2dfc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:18:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfbfff700 1 -- 172.21.15.16:0/4227738472 >> v1:172.21.15.16:6789/0 conn(0x7f2dfc08ef20 legacy=0x7f2dfc08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:17.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfb7fe700 1 -- 172.21.15.16:0/4227738472 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426867690 0 0) 0x7f2df4000d30 con 0x7f2dfc08ef20 2019-12-05T06:18:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfbfff700 1 -- 172.21.15.16:0/4227738472 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc08d540 con 0x7f2dfc08fef0 2019-12-05T06:18:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/4227738472 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2dfc08c6d0 0x7f2dfc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:18:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfbfff700 1 --2- 172.21.15.16:0/4227738472 >> v2:172.21.15.16:3300/0 conn(0x7f2dfc08fef0 0x7f2dfc08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2dec00f740 tx=0x7f2dec00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfbfff700 1 --2- 172.21.15.16:0/4227738472 >> v2:172.21.15.16:3300/0 conn(0x7f2dfc08fef0 0x7f2dfc08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2dec00f740 tx=0x7f2dec00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfbfff700 1 --2- 172.21.15.16:0/4227738472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc08fef0 0x7f2dfc08d000 secure :-1 s=READY pgs=519 cs=0 l=1 rx=0x7f2dec00f740 tx=0x7f2dec00fae0).ready entity=mon.0 client_cookie=7f4ec69d1c69ccdc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfbfff700 1 --2- 172.21.15.16:0/4227738472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc08fef0 0x7f2dfc08d000 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f2dec00f740 tx=0x7f2dec00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfb7fe700 1 -- 172.21.15.16:0/4227738472 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dec000960 con 0x7f2dfc08fef0 2019-12-05T06:18:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfbfff700 1 --2- 172.21.15.16:0/4227738472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc08fef0 0x7f2dfc08d000 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f2dec00f740 tx=0x7f2dec00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.512+0000 7f2dfbfff700 1 --2- 172.21.15.16:0/4227738472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc08fef0 0x7f2dfc08d000 secure :-1 s=THROTTLE_DONE pgs=519 cs=0 l=1 rx=0x7f2dec00f740 tx=0x7f2dec00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.513+0000 7f2dfb7fe700 1 -- 172.21.15.16:0/4227738472 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2dec000c90 con 0x7f2dfc08fef0 2019-12-05T06:18:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.513+0000 7f2dfb7fe700 1 -- 172.21.15.16:0/4227738472 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2dec0013a0 con 0x7f2dfc08fef0 2019-12-05T06:18:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.513+0000 7f2e02890700 1 -- 172.21.15.16:0/4227738472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc08fef0 msgr2=0x7f2dfc08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.513+0000 7f2e02890700 1 --2- 172.21.15.16:0/4227738472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc08fef0 0x7f2dfc08d000 secure :-1 s=READY pgs=519 cs=0 l=1 rx=0x7f2dec00f740 tx=0x7f2dec00fae0).stop 2019-12-05T06:18:17.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.514+0000 7f2e02890700 1 -- 172.21.15.16:0/4227738472 shutdown_connections 2019-12-05T06:18:17.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.514+0000 7f2e02890700 1 --2- 172.21.15.16:0/4227738472 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc08fef0 0x7f2dfc08d000 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:17.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.514+0000 7f2e02890700 1 --2- 172.21.15.16:0/4227738472 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2dfc08c6d0 0x7f2dfc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:18:17.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.514+0000 7f2e02890700 1 -- 172.21.15.16:0/4227738472 shutdown_connections 2019-12-05T06:18:17.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.514+0000 7f2e02890700 1 -- 172.21.15.16:0/4227738472 wait complete. 2019-12-05T06:18:17.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.514+0000 7f2e02890700 1 -- 172.21.15.16:0/4227738472 >> 172.21.15.16:0/4227738472 conn(0x7f2dfc061940 msgr2=0x7f2dfc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.519+0000 7f2e02890700 1 Processor -- start 2019-12-05T06:18:17.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.519+0000 7f2e02890700 1 -- start start 2019-12-05T06:18:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.521+0000 7f2e02890700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dfc096e90 0x7f2dfc09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.521+0000 7f2e02890700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dfc084b90 con 0x7f2dfc09b920 2019-12-05T06:18:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.521+0000 7f2e02890700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dfc098790 con 0x7f2dfc084f30 2019-12-05T06:18:17.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.521+0000 7f2e02890700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2dfc09a450 con 0x7f2dfc096e90 2019-12-05T06:18:17.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.521+0000 7f2e00e2d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2dfc096e90 0x7f2dfc09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:17.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.521+0000 7f2e0162e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2dfc09b920 0x7f2dfc097a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35910/0 (socket says 172.21.15.16:35910) 2019-12-05T06:18:17.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.521+0000 7f2e0162e700 1 -- 172.21.15.16:0/578695686 learned_addr learned my addr 172.21.15.16:0/578695686 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:18:17.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.521+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223088091 0 0) 0x7f2df4001300 con 0x7f2dfc09b920 2019-12-05T06:18:17.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de0002e80 con 0x7f2dfc09b920 2019-12-05T06:18:17.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396456532 0 0) 0x7f2dec007900 con 0x7f2dfc084f30 2019-12-05T06:18:17.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2de00030b0 con 0x7f2dfc084f30 2019-12-05T06:18:17.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2e00e2d700 1 -- 172.21.15.16:0/578695686 >> v1:172.21.15.76:6789/0 conn(0x7f2dfc084f30 legacy=0x7f2dfc097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:17.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2e00e2d700 1 -- 172.21.15.16:0/578695686 >> v1:172.21.15.16:6789/0 conn(0x7f2dfc09b920 legacy=0x7f2dfc097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:17.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2e00e2d700 1 -- 172.21.15.16:0/578695686 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dfc09aae0 con 0x7f2dfc096e90 2019-12-05T06:18:17.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/578695686 >> v2:172.21.15.16:3300/0 conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/578695686 >> v2:172.21.15.16:3300/0 conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=READY pgs=521 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.522+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2de4000940 con 0x7f2dfc096e90 2019-12-05T06:18:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.523+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.523+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2de4009770 con 0x7f2dfc096e90 2019-12-05T06:18:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.523+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.523+0000 7f2e02890700 1 -- 172.21.15.16:0/578695686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dfc09aae0 con 0x7f2dfc096e90 2019-12-05T06:18:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.523+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2de4009960 con 0x7f2dfc096e90 2019-12-05T06:18:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.523+0000 7f2e02890700 1 -- 172.21.15.16:0/578695686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dfc09aae0 con 0x7f2dfc096e90 2019-12-05T06:18:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.524+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.525+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2de4009960 con 0x7f2dfc096e90 2019-12-05T06:18:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.525+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.525+0000 7f2df97fa700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2de0023650 0x7f2de0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:18:17.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.525+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f2de4045de0 con 0x7f2dfc096e90 2019-12-05T06:18:17.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.525+0000 7f2dfbfff700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2de0023650 0x7f2de0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:18:17.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.526+0000 7f2dfbfff700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2de0023650 0x7f2de0025ae0 crc :-1 s=READY pgs=232 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:18:17.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.540+0000 7f2e02890700 1 -- 172.21.15.16:0/578695686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dfc09aae0 con 0x7f2dfc096e90 2019-12-05T06:18:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.545+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.545+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2de4005180 con 0x7f2dfc096e90 2019-12-05T06:18:17.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.791+0000 7f2e02890700 1 -- 172.21.15.16:0/578695686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2dfc09aae0 con 0x7f2dfc096e90 2019-12-05T06:18:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.792+0000 7f2e00e2d700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=THROTTLE_DONE pgs=521 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:18:17.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.792+0000 7f2df97fa700 1 -- 172.21.15.16:0/578695686 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f2de400a210 con 0x7f2dfc096e90 2019-12-05T06:18:17.786 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 67 2019-12-05T06:18:17.786 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:18:17.786 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:18:17.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.796+0000 7f2e02890700 1 -- 172.21.15.16:0/578695686 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2de0023650 msgr2=0x7f2de0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.796+0000 7f2e02890700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2de0023650 0x7f2de0025ae0 crc :-1 s=READY pgs=232 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:17.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.796+0000 7f2e02890700 1 -- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 msgr2=0x7f2dfc09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:18:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.796+0000 7f2e02890700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 0x7f2dfc09b3e0 secure :-1 s=READY pgs=521 cs=0 l=1 rx=0x7f2de4010390 tx=0x7f2de4010730).stop 2019-12-05T06:18:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.797+0000 7f2e02890700 1 -- 172.21.15.16:0/578695686 shutdown_connections 2019-12-05T06:18:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.797+0000 7f2e02890700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2de0023650 0x7f2de0025ae0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:17.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.797+0000 7f2e02890700 1 --2- 172.21.15.16:0/578695686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2dfc096e90 0x7f2dfc09b3e0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:18:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.797+0000 7f2e02890700 1 -- 172.21.15.16:0/578695686 shutdown_connections 2019-12-05T06:18:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.797+0000 7f2e02890700 1 -- 172.21.15.16:0/578695686 wait complete. 2019-12-05T06:18:17.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:18:17.797+0000 7f2e02890700 1 -- 172.21.15.16:0/578695686 >> 172.21.15.16:0/578695686 conn(0x7f2dfc0714f0 msgr2=0x7f2dfc09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:18:17.806 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:18:17.806 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 59 secs before thrashing 2019-12-05T06:18:31.401 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:18:31.406 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:19:01.570 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:19:01.575 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:19:16.807 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:17.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.076+0000 7f47b695b700 1 Processor -- start 2019-12-05T06:19:17.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.076+0000 7f47b695b700 1 -- start start 2019-12-05T06:19:17.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.077+0000 7f47b695b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:17.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.077+0000 7f47b695b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f47b008fef0 0x7f47b008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.078+0000 7f47b695b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47b00958e0 con 0x7f47b008c6d0 2019-12-05T06:19:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.078+0000 7f47b695b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b008df90 con 0x7f47b008fef0 2019-12-05T06:19:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.078+0000 7f47b695b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b00a3f70 con 0x7f47b008d2c0 2019-12-05T06:19:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.078+0000 7f47affff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.078+0000 7f47b56f9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f47b008fef0 0x7f47b008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.078+0000 7f47b56f9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f47b008fef0 0x7f47b008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:36862/0 (socket says 172.21.15.16:36862) 2019-12-05T06:19:17.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.078+0000 7f47affff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:54760/0 (socket says 172.21.15.16:54760) 2019-12-05T06:19:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.078+0000 7f47b56f9700 1 -- 172.21.15.16:0/1992921742 learned_addr learned my addr 172.21.15.16:0/1992921742 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47affff700 1 -- 172.21.15.16:0/1992921742 >> v2:172.21.15.76:3300/0 conn(0x7f47b008fef0 msgr2=0x7f47b008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47affff700 1 --2- 172.21.15.16:0/1992921742 >> v2:172.21.15.76:3300/0 conn(0x7f47b008fef0 0x7f47b008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47affff700 1 -- 172.21.15.16:0/1992921742 >> v1:172.21.15.76:6789/0 conn(0x7f47b008c6d0 legacy=0x7f47b008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47affff700 1 -- 172.21.15.16:0/1992921742 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b008e620 con 0x7f47b008d2c0 2019-12-05T06:19:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47affff700 1 --2- 172.21.15.16:0/1992921742 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f47a000af50 tx=0x7f47a000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47affff700 1 --2- 172.21.15.16:0/1992921742 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f47a000af50 tx=0x7f47a000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47affff700 1 --2- 172.21.15.16:0/1992921742 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 secure :-1 s=READY pgs=134 cs=0 l=1 rx=0x7f47a000af50 tx=0x7f47a000b250).ready entity=mon.2 client_cookie=e4e6cd9ac5f13b17 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47affff700 1 --2- 172.21.15.16:0/1992921742 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f47a000af50 tx=0x7f47a000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47affff700 1 --2- 172.21.15.16:0/1992921742 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f47a000af50 tx=0x7f47a000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47affff700 1 --2- 172.21.15.16:0/1992921742 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 secure :-1 s=THROTTLE_DONE pgs=134 cs=0 l=1 rx=0x7f47a000af50 tx=0x7f47a000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.079+0000 7f47af7fe700 1 -- 172.21.15.16:0/1992921742 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47a0007210 con 0x7f47b008d2c0 2019-12-05T06:19:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.080+0000 7f47af7fe700 1 -- 172.21.15.16:0/1992921742 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a00099b0 con 0x7f47b008d2c0 2019-12-05T06:19:17.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.080+0000 7f47af7fe700 1 -- 172.21.15.16:0/1992921742 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47a000a080 con 0x7f47b008d2c0 2019-12-05T06:19:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.080+0000 7f47b695b700 1 -- 172.21.15.16:0/1992921742 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 msgr2=0x7f47b008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.080+0000 7f47b695b700 1 --2- 172.21.15.16:0/1992921742 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 secure :-1 s=READY pgs=134 cs=0 l=1 rx=0x7f47a000af50 tx=0x7f47a000b250).stop 2019-12-05T06:19:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.081+0000 7f47b695b700 1 -- 172.21.15.16:0/1992921742 shutdown_connections 2019-12-05T06:19:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.081+0000 7f47b695b700 1 --2- 172.21.15.16:0/1992921742 >> v2:172.21.15.76:3300/0 conn(0x7f47b008fef0 0x7f47b008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.081+0000 7f47b695b700 1 --2- 172.21.15.16:0/1992921742 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b008d2c0 0x7f47b008ef20 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.081+0000 7f47b695b700 1 -- 172.21.15.16:0/1992921742 shutdown_connections 2019-12-05T06:19:17.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.081+0000 7f47b695b700 1 -- 172.21.15.16:0/1992921742 wait complete. 2019-12-05T06:19:17.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.081+0000 7f47b695b700 1 -- 172.21.15.16:0/1992921742 >> 172.21.15.16:0/1992921742 conn(0x7f47b0061940 msgr2=0x7f47b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:17.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.086+0000 7f47b695b700 1 Processor -- start 2019-12-05T06:19:17.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.086+0000 7f47b695b700 1 -- start start 2019-12-05T06:19:17.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.088+0000 7f47b695b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f47b014d5d0 0x7f47b014d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:17.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.088+0000 7f47b695b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47b009e2c0 0x7f47b009d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:17.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.088+0000 7f47b695b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b009d8e0 0x7f47b0098180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:17.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.088+0000 7f47b695b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b012b3f0 con 0x7f47b009e2c0 2019-12-05T06:19:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.088+0000 7f47b695b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47b012a9c0 con 0x7f47b014d5d0 2019-12-05T06:19:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.088+0000 7f47b695b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47b012ac90 con 0x7f47b009d8e0 2019-12-05T06:19:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.088+0000 7f47affff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47b009e2c0 0x7f47b009d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:17.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.088+0000 7f47affff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f47b009e2c0 0x7f47b009d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50774/0 (socket says 172.21.15.16:50774) 2019-12-05T06:19:17.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.088+0000 7f47affff700 1 -- 172.21.15.16:0/3043722345 learned_addr learned my addr 172.21.15.16:0/3043722345 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:17.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.089+0000 7f47b56f9700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b009d8e0 0x7f47b0098180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:17.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.089+0000 7f47b4ef8700 1 --2- 172.21.15.16:0/3043722345 >> v2:172.21.15.76:3300/0 conn(0x7f47b014d5d0 0x7f47b014d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:17.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.089+0000 7f47affff700 1 -- 172.21.15.16:0/3043722345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b009d8e0 msgr2=0x7f47b0098180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.089+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b009d8e0 0x7f47b0098180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.089+0000 7f47affff700 1 -- 172.21.15.16:0/3043722345 >> v2:172.21.15.76:3300/0 conn(0x7f47b014d5d0 msgr2=0x7f47b014d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.089+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> v2:172.21.15.76:3300/0 conn(0x7f47b014d5d0 0x7f47b014d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.089+0000 7f47affff700 1 -- 172.21.15.16:0/3043722345 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b012ba80 con 0x7f47b009e2c0 2019-12-05T06:19:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.089+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> v2:172.21.15.16:3300/0 conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.089+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> v2:172.21.15.16:3300/0 conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.089+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=READY pgs=522 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.090+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.090+0000 7f47ad7fa700 1 -- 172.21.15.16:0/3043722345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47a0000940 con 0x7f47b009e2c0 2019-12-05T06:19:17.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.090+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.090+0000 7f47b695b700 1 -- 172.21.15.16:0/3043722345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47b012ba80 con 0x7f47b009e2c0 2019-12-05T06:19:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.090+0000 7f47ad7fa700 1 -- 172.21.15.16:0/3043722345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47a0000940 con 0x7f47b009e2c0 2019-12-05T06:19:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.090+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.090+0000 7f47ad7fa700 1 -- 172.21.15.16:0/3043722345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47a000abb0 con 0x7f47b009e2c0 2019-12-05T06:19:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.090+0000 7f47b695b700 1 -- 172.21.15.16:0/3043722345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47b006e250 con 0x7f47b009e2c0 2019-12-05T06:19:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.091+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.092+0000 7f47ad7fa700 1 -- 172.21.15.16:0/3043722345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f47a0009580 con 0x7f47b009e2c0 2019-12-05T06:19:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.092+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.092+0000 7f47ad7fa700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4794023630 0x7f4794025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.092+0000 7f47ad7fa700 1 -- 172.21.15.16:0/3043722345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f47a0045d90 con 0x7f47b009e2c0 2019-12-05T06:19:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.092+0000 7f47b4ef8700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4794023630 0x7f4794025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:17.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.093+0000 7f47b4ef8700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4794023630 0x7f4794025ac0 crc :-1 s=READY pgs=234 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:17.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.107+0000 7f47b695b700 1 -- 172.21.15.16:0/3043722345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47b012ba80 con 0x7f47b009e2c0 2019-12-05T06:19:17.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.114+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.114+0000 7f47ad7fa700 1 -- 172.21.15.16:0/3043722345 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f47a000d5e0 con 0x7f47b009e2c0 2019-12-05T06:19:17.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.340+0000 7f47b695b700 1 -- 172.21.15.16:0/3043722345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f47b012ba80 con 0x7f47b009e2c0 2019-12-05T06:19:17.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.341+0000 7f47affff700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=THROTTLE_DONE pgs=522 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.341+0000 7f47ad7fa700 1 -- 172.21.15.16:0/3043722345 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f47a0009580 con 0x7f47b009e2c0 2019-12-05T06:19:17.332 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:17.333 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:17.333 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 67 2019-12-05T06:19:17.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 -- 172.21.15.16:0/3043722345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4794023630 msgr2=0x7f4794025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4794023630 0x7f4794025ac0 crc :-1 s=READY pgs=234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 -- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 msgr2=0x7f47b009d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 0x7f47b009d3a0 secure :-1 s=READY pgs=522 cs=0 l=1 rx=0x7f47a0010460 tx=0x7f47a0010310).stop 2019-12-05T06:19:17.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 -- 172.21.15.16:0/3043722345 shutdown_connections 2019-12-05T06:19:17.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47b009d8e0 0x7f47b0098180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4794023630 0x7f4794025ac0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 --2- 172.21.15.16:0/3043722345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f47b009e2c0 0x7f47b009d3a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 --2- 172.21.15.16:0/3043722345 >> v2:172.21.15.76:3300/0 conn(0x7f47b014d5d0 0x7f47b014d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 -- 172.21.15.16:0/3043722345 shutdown_connections 2019-12-05T06:19:17.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 -- 172.21.15.16:0/3043722345 wait complete. 2019-12-05T06:19:17.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.344+0000 7f47b695b700 1 -- 172.21.15.16:0/3043722345 >> 172.21.15.16:0/3043722345 conn(0x7f47b006d260 msgr2=0x7f47b009aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:17.353 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:19:17.353 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:19:17.353 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:19:17.369 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:19:17.369 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.638+0000 7f2ae078a700 1 Processor -- start 2019-12-05T06:19:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.638+0000 7f2ae078a700 1 -- start start 2019-12-05T06:19:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.640+0000 7f2ae078a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2ad808fef0 0x7f2ad808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.640+0000 7f2ae078a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80958e0 con 0x7f2ad808c6d0 2019-12-05T06:19:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.640+0000 7f2ae078a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad80629e0 con 0x7f2ad808d2c0 2019-12-05T06:19:17.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.640+0000 7f2ae078a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ad808df90 con 0x7f2ad808fef0 2019-12-05T06:19:17.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.641+0000 7f2aded27700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ad808c6d0 0x7f2ad808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35954/0 (socket says 172.21.15.16:35954) 2019-12-05T06:19:17.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.641+0000 7f2aded27700 1 -- 172.21.15.16:0/4095986912 learned_addr learned my addr 172.21.15.16:0/4095986912 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:17.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.641+0000 7f2adf528700 1 --2- 172.21.15.16:0/4095986912 >> v2:172.21.15.76:3300/0 conn(0x7f2ad808fef0 0x7f2ad808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:17.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.641+0000 7f2addd25700 1 -- 172.21.15.16:0/4095986912 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 141069837 0 0) 0x7f2ac8001150 con 0x7f2ad808c6d0 2019-12-05T06:19:17.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.642+0000 7f2addd25700 1 -- 172.21.15.16:0/4095986912 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac0002e80 con 0x7f2ad808c6d0 2019-12-05T06:19:17.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.642+0000 7f2adf528700 1 -- 172.21.15.16:0/4095986912 >> v1:172.21.15.76:6789/0 conn(0x7f2ad808d2c0 legacy=0x7f2ad808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.642+0000 7f2addd25700 1 -- 172.21.15.16:0/4095986912 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3622153859 0 0) 0x7f2acc001400 con 0x7f2ad808d2c0 2019-12-05T06:19:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.642+0000 7f2adf528700 1 -- 172.21.15.16:0/4095986912 >> v1:172.21.15.16:6789/0 conn(0x7f2ad808c6d0 legacy=0x7f2ad808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.642+0000 7f2adf528700 1 -- 172.21.15.16:0/4095986912 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad808e620 con 0x7f2ad808fef0 2019-12-05T06:19:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.642+0000 7f2adf528700 1 --2- 172.21.15.16:0/4095986912 >> v2:172.21.15.76:3300/0 conn(0x7f2ad808fef0 0x7f2ad808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2ad000af80 tx=0x7f2ad000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.642+0000 7f2adf528700 1 --2- 172.21.15.16:0/4095986912 >> v2:172.21.15.76:3300/0 conn(0x7f2ad808fef0 0x7f2ad808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2ad000af80 tx=0x7f2ad000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.642+0000 7f2adf528700 1 --2- 172.21.15.16:0/4095986912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad808fef0 0x7f2ad808f720 secure :-1 s=READY pgs=394 cs=0 l=1 rx=0x7f2ad000af80 tx=0x7f2ad000b790).ready entity=mon.1 client_cookie=1bae02a88ba95c40 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:17.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.643+0000 7f2adf528700 1 --2- 172.21.15.16:0/4095986912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad808fef0 0x7f2ad808f720 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f2ad000af80 tx=0x7f2ad000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.643+0000 7f2addd25700 1 -- 172.21.15.16:0/4095986912 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ad0007240 con 0x7f2ad808fef0 2019-12-05T06:19:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.643+0000 7f2adf528700 1 --2- 172.21.15.16:0/4095986912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad808fef0 0x7f2ad808f720 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f2ad000af80 tx=0x7f2ad000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.643+0000 7f2adf528700 1 --2- 172.21.15.16:0/4095986912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad808fef0 0x7f2ad808f720 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f2ad000af80 tx=0x7f2ad000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.643+0000 7f2addd25700 1 -- 172.21.15.16:0/4095986912 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ad0000bb0 con 0x7f2ad808fef0 2019-12-05T06:19:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.643+0000 7f2addd25700 1 -- 172.21.15.16:0/4095986912 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ad0009e60 con 0x7f2ad808fef0 2019-12-05T06:19:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.643+0000 7f2ae078a700 1 -- 172.21.15.16:0/4095986912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad808fef0 msgr2=0x7f2ad808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.643+0000 7f2ae078a700 1 --2- 172.21.15.16:0/4095986912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad808fef0 0x7f2ad808f720 secure :-1 s=READY pgs=394 cs=0 l=1 rx=0x7f2ad000af80 tx=0x7f2ad000b790).stop 2019-12-05T06:19:17.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.645+0000 7f2ae078a700 1 -- 172.21.15.16:0/4095986912 shutdown_connections 2019-12-05T06:19:17.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.645+0000 7f2ae078a700 1 --2- 172.21.15.16:0/4095986912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2ad808fef0 0x7f2ad808f720 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.645+0000 7f2ae078a700 1 -- 172.21.15.16:0/4095986912 shutdown_connections 2019-12-05T06:19:17.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.645+0000 7f2ae078a700 1 -- 172.21.15.16:0/4095986912 wait complete. 2019-12-05T06:19:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.645+0000 7f2ae078a700 1 -- 172.21.15.16:0/4095986912 >> 172.21.15.16:0/4095986912 conn(0x7f2ad8061940 msgr2=0x7f2ad8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.650+0000 7f2ae078a700 1 Processor -- start 2019-12-05T06:19:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.650+0000 7f2ae078a700 1 -- start start 2019-12-05T06:19:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.651+0000 7f2ae078a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ad812b640 0x7f2ad812ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.651+0000 7f2ae078a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2ad8153ba0 0x7f2ad8155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.651+0000 7f2ae078a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ad8131bf0 con 0x7f2ad81302f0 2019-12-05T06:19:17.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.651+0000 7f2ae078a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ad81296e0 con 0x7f2ad812b640 2019-12-05T06:19:17.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.651+0000 7f2ae078a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ad8128cb0 con 0x7f2ad8153ba0 2019-12-05T06:19:17.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.652+0000 7f2ade526700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ad812b640 0x7f2ad812ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:17.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.652+0000 7f2aded27700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ad81302f0 0x7f2ad812a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35960/0 (socket says 172.21.15.16:35960) 2019-12-05T06:19:17.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.652+0000 7f2aded27700 1 -- 172.21.15.16:0/2226383641 learned_addr learned my addr 172.21.15.16:0/2226383641 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:17.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.652+0000 7f2adf528700 1 --2- 172.21.15.16:0/2226383641 >> v2:172.21.15.76:3300/0 conn(0x7f2ad8153ba0 0x7f2ad8155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:17.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.652+0000 7f2ade526700 1 -- 172.21.15.16:0/2226383641 >> v2:172.21.15.76:3300/0 conn(0x7f2ad8153ba0 msgr2=0x7f2ad8155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.652+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> v2:172.21.15.76:3300/0 conn(0x7f2ad8153ba0 0x7f2ad8155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.652+0000 7f2ade526700 1 -- 172.21.15.16:0/2226383641 >> v1:172.21.15.16:6789/0 conn(0x7f2ad81302f0 legacy=0x7f2ad812a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.652+0000 7f2ade526700 1 -- 172.21.15.16:0/2226383641 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ad8129d70 con 0x7f2ad812b640 2019-12-05T06:19:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.652+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2226383641 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795157833 0 0) 0x7f2ac8000f40 con 0x7f2ad81302f0 2019-12-05T06:19:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> v2:172.21.15.16:3300/0 conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> v2:172.21.15.16:3300/0 conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=READY pgs=525 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=THROTTLE_DONE pgs=525 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2226383641 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2acc001220 con 0x7f2ad812b640 2019-12-05T06:19:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=THROTTLE_DONE pgs=525 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2226383641 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2acc005c60 con 0x7f2ad812b640 2019-12-05T06:19:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=THROTTLE_DONE pgs=525 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2226383641 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2acc001220 con 0x7f2ad812b640 2019-12-05T06:19:17.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ae078a700 1 -- 172.21.15.16:0/2226383641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ad8129d70 con 0x7f2ad812b640 2019-12-05T06:19:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.653+0000 7f2ae078a700 1 -- 172.21.15.16:0/2226383641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ad8129d70 con 0x7f2ad812b640 2019-12-05T06:19:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.654+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=THROTTLE_DONE pgs=525 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.655+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=THROTTLE_DONE pgs=525 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.655+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2226383641 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2acc005e50 con 0x7f2ad812b640 2019-12-05T06:19:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.655+0000 7f2ad77fe700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ab8023630 0x7f2ab8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.656+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2226383641 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f2acc048280 con 0x7f2ad812b640 2019-12-05T06:19:17.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.656+0000 7f2aded27700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ab8023630 0x7f2ab8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:17.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.656+0000 7f2aded27700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ab8023630 0x7f2ab8025ac0 crc :-1 s=READY pgs=235 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:17.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.670+0000 7f2ae078a700 1 -- 172.21.15.16:0/2226383641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ad8129d70 con 0x7f2ad812b640 2019-12-05T06:19:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.677+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=THROTTLE_DONE pgs=525 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.677+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2226383641 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2acc005860 con 0x7f2ad812b640 2019-12-05T06:19:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.938+0000 7f2ae078a700 1 -- 172.21.15.16:0/2226383641 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ad8129d70 con 0x7f2ad812b640 2019-12-05T06:19:17.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.939+0000 7f2ade526700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=THROTTLE_DONE pgs=525 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:17.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.939+0000 7f2ad77fe700 1 -- 172.21.15.16:0/2226383641 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f2acc005e50 con 0x7f2ad812b640 2019-12-05T06:19:17.932 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:17.932 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:17.933 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 67 2019-12-05T06:19:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.944+0000 7f2ae078a700 1 -- 172.21.15.16:0/2226383641 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ab8023630 msgr2=0x7f2ab8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.944+0000 7f2ae078a700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ab8023630 0x7f2ab8025ac0 crc :-1 s=READY pgs=235 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.944+0000 7f2ae078a700 1 -- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 msgr2=0x7f2ad812ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.944+0000 7f2ae078a700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 0x7f2ad812ae70 secure :-1 s=READY pgs=525 cs=0 l=1 rx=0x7f2acc012890 tx=0x7f2acc012c30).stop 2019-12-05T06:19:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.944+0000 7f2ae078a700 1 -- 172.21.15.16:0/2226383641 shutdown_connections 2019-12-05T06:19:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.944+0000 7f2ae078a700 1 --2- 172.21.15.16:0/2226383641 >> v2:172.21.15.76:3300/0 conn(0x7f2ad8153ba0 0x7f2ad8155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.944+0000 7f2ae078a700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ab8023630 0x7f2ab8025ac0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.944+0000 7f2ae078a700 1 --2- 172.21.15.16:0/2226383641 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ad812b640 0x7f2ad812ae70 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:17.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.945+0000 7f2ae078a700 1 -- 172.21.15.16:0/2226383641 shutdown_connections 2019-12-05T06:19:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.945+0000 7f2ae078a700 1 -- 172.21.15.16:0/2226383641 wait complete. 2019-12-05T06:19:17.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:17.945+0000 7f2ae078a700 1 -- 172.21.15.16:0/2226383641 >> 172.21.15.16:0/2226383641 conn(0x7f2ad80a2850 msgr2=0x7f2ad806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:17.946 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:19.947 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.223+0000 7f3176e2a700 1 Processor -- start 2019-12-05T06:19:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.223+0000 7f3176e2a700 1 -- start start 2019-12-05T06:19:20.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.225+0000 7f3176e2a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 0x7f317008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:20.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.225+0000 7f3176e2a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31700958e0 con 0x7f317008c6d0 2019-12-05T06:19:20.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.225+0000 7f3176e2a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31700629e0 con 0x7f317008d2c0 2019-12-05T06:19:20.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.225+0000 7f3176e2a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f317008df90 con 0x7f317008fef0 2019-12-05T06:19:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.225+0000 7f31753c7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f317008c6d0 0x7f317008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35972/0 (socket says 172.21.15.16:35972) 2019-12-05T06:19:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.225+0000 7f31753c7700 1 -- 172.21.15.16:0/2095192199 learned_addr learned my addr 172.21.15.16:0/2095192199 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.226+0000 7f3175bc8700 1 --2- 172.21.15.16:0/2095192199 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 0x7f317008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:20.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.226+0000 7f316ffff700 1 -- 172.21.15.16:0/2095192199 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1142027674 0 0) 0x7f3160001100 con 0x7f317008c6d0 2019-12-05T06:19:20.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.226+0000 7f316ffff700 1 -- 172.21.15.16:0/2095192199 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3158002e80 con 0x7f317008c6d0 2019-12-05T06:19:20.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.226+0000 7f316ffff700 1 -- 172.21.15.16:0/2095192199 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586783826 0 0) 0x7f3164001400 con 0x7f317008d2c0 2019-12-05T06:19:20.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.226+0000 7f316ffff700 1 -- 172.21.15.16:0/2095192199 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3158004220 con 0x7f317008d2c0 2019-12-05T06:19:20.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.227+0000 7f3175bc8700 1 -- 172.21.15.16:0/2095192199 >> v1:172.21.15.76:6789/0 conn(0x7f317008d2c0 legacy=0x7f317008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.227+0000 7f3175bc8700 1 -- 172.21.15.16:0/2095192199 >> v1:172.21.15.16:6789/0 conn(0x7f317008c6d0 legacy=0x7f317008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.227+0000 7f316ffff700 1 -- 172.21.15.16:0/2095192199 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 372684383 0 0) 0x7f31600010e0 con 0x7f317008c6d0 2019-12-05T06:19:20.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.227+0000 7f3175bc8700 1 -- 172.21.15.16:0/2095192199 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317008e620 con 0x7f317008fef0 2019-12-05T06:19:20.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.227+0000 7f3175bc8700 1 --2- 172.21.15.16:0/2095192199 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 0x7f317008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f316800af80 tx=0x7f316800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.227+0000 7f3175bc8700 1 --2- 172.21.15.16:0/2095192199 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 0x7f317008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f316800af80 tx=0x7f316800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.227+0000 7f3175bc8700 1 --2- 172.21.15.16:0/2095192199 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 0x7f317008f720 secure :-1 s=READY pgs=135 cs=0 l=1 rx=0x7f316800af80 tx=0x7f316800b790).ready entity=mon.2 client_cookie=24e0608e7a457dd1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.227+0000 7f3175bc8700 1 --2- 172.21.15.16:0/2095192199 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 0x7f317008f720 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7f316800af80 tx=0x7f316800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.228+0000 7f316ffff700 1 -- 172.21.15.16:0/2095192199 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3168007240 con 0x7f317008fef0 2019-12-05T06:19:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.228+0000 7f3175bc8700 1 --2- 172.21.15.16:0/2095192199 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 0x7f317008f720 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7f316800af80 tx=0x7f316800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.228+0000 7f3175bc8700 1 --2- 172.21.15.16:0/2095192199 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 0x7f317008f720 secure :-1 s=THROTTLE_DONE pgs=135 cs=0 l=1 rx=0x7f316800af80 tx=0x7f316800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.228+0000 7f316ffff700 1 -- 172.21.15.16:0/2095192199 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3168000bb0 con 0x7f317008fef0 2019-12-05T06:19:20.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.228+0000 7f316ffff700 1 -- 172.21.15.16:0/2095192199 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3168009e60 con 0x7f317008fef0 2019-12-05T06:19:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.228+0000 7f3176e2a700 1 -- 172.21.15.16:0/2095192199 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 msgr2=0x7f317008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.228+0000 7f3176e2a700 1 --2- 172.21.15.16:0/2095192199 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 0x7f317008f720 secure :-1 s=READY pgs=135 cs=0 l=1 rx=0x7f316800af80 tx=0x7f316800b790).stop 2019-12-05T06:19:20.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.230+0000 7f3176e2a700 1 -- 172.21.15.16:0/2095192199 shutdown_connections 2019-12-05T06:19:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.230+0000 7f3176e2a700 1 --2- 172.21.15.16:0/2095192199 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f317008fef0 0x7f317008f720 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.230+0000 7f3176e2a700 1 -- 172.21.15.16:0/2095192199 shutdown_connections 2019-12-05T06:19:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.230+0000 7f3176e2a700 1 -- 172.21.15.16:0/2095192199 wait complete. 2019-12-05T06:19:20.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.230+0000 7f3176e2a700 1 -- 172.21.15.16:0/2095192199 >> 172.21.15.16:0/2095192199 conn(0x7f3170061940 msgr2=0x7f3170096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:20.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.235+0000 7f3176e2a700 1 Processor -- start 2019-12-05T06:19:20.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.235+0000 7f3176e2a700 1 -- start start 2019-12-05T06:19:20.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.237+0000 7f3176e2a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31701325e0 0x7f317012b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:20.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.237+0000 7f3176e2a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f317012c740 0x7f317012bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:20.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.237+0000 7f3176e2a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3170128b20 con 0x7f317012c1f0 2019-12-05T06:19:20.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.237+0000 7f3176e2a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f317012a7e0 con 0x7f31701325e0 2019-12-05T06:19:20.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.237+0000 7f3176e2a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3170129db0 con 0x7f317012c740 2019-12-05T06:19:20.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.237+0000 7f31753c7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31701325e0 0x7f317012b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.237+0000 7f31753c7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31701325e0 0x7f317012b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50808/0 (socket says 172.21.15.16:50808) 2019-12-05T06:19:20.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.237+0000 7f31753c7700 1 -- 172.21.15.16:0/1540281229 learned_addr learned my addr 172.21.15.16:0/1540281229 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.237+0000 7f3174bc6700 1 --2- 172.21.15.16:0/1540281229 >> v2:172.21.15.76:3300/0 conn(0x7f317012c740 0x7f317012bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.238+0000 7f31753c7700 1 -- 172.21.15.16:0/1540281229 >> v2:172.21.15.76:3300/0 conn(0x7f317012c740 msgr2=0x7f317012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.238+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> v2:172.21.15.76:3300/0 conn(0x7f317012c740 0x7f317012bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.238+0000 7f31753c7700 1 -- 172.21.15.16:0/1540281229 >> v1:172.21.15.16:6789/0 conn(0x7f317012c1f0 legacy=0x7f317006dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.238+0000 7f31753c7700 1 -- 172.21.15.16:0/1540281229 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f317014a820 con 0x7f31701325e0 2019-12-05T06:19:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.238+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> v2:172.21.15.16:3300/0 conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.238+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> v2:172.21.15.16:3300/0 conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.238+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=READY pgs=528 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:20.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.239+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=THROTTLE_DONE pgs=528 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.239+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=THROTTLE_DONE pgs=528 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.239+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=THROTTLE_DONE pgs=528 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.239+0000 7f316dffb700 1 -- 172.21.15.16:0/1540281229 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3160000c90 con 0x7f31701325e0 2019-12-05T06:19:20.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.239+0000 7f316dffb700 1 -- 172.21.15.16:0/1540281229 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3160004430 con 0x7f31701325e0 2019-12-05T06:19:20.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.239+0000 7f3176e2a700 1 -- 172.21.15.16:0/1540281229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f317012ae70 con 0x7f31701325e0 2019-12-05T06:19:20.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.239+0000 7f316dffb700 1 -- 172.21.15.16:0/1540281229 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3160004ac0 con 0x7f31701325e0 2019-12-05T06:19:20.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.239+0000 7f3176e2a700 1 -- 172.21.15.16:0/1540281229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f317014a820 con 0x7f31701325e0 2019-12-05T06:19:20.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.240+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=THROTTLE_DONE pgs=528 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.241+0000 7f316dffb700 1 -- 172.21.15.16:0/1540281229 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3160004620 con 0x7f31701325e0 2019-12-05T06:19:20.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.241+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=THROTTLE_DONE pgs=528 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.241+0000 7f316dffb700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3150023630 0x7f3150025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.241+0000 7f316dffb700 1 -- 172.21.15.16:0/1540281229 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f3160045e20 con 0x7f31701325e0 2019-12-05T06:19:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.242+0000 7f3174bc6700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3150023630 0x7f3150025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:20.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.242+0000 7f3174bc6700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3150023630 0x7f3150025ac0 crc :-1 s=READY pgs=236 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.255+0000 7f3176e2a700 1 -- 172.21.15.16:0/1540281229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f317012ae70 con 0x7f31701325e0 2019-12-05T06:19:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.262+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=THROTTLE_DONE pgs=528 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.262+0000 7f316dffb700 1 -- 172.21.15.16:0/1540281229 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3160031ec0 con 0x7f31701325e0 2019-12-05T06:19:20.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.523+0000 7f3176e2a700 1 -- 172.21.15.16:0/1540281229 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f317014a820 con 0x7f31701325e0 2019-12-05T06:19:20.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.524+0000 7f31753c7700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=THROTTLE_DONE pgs=528 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.524+0000 7f316dffb700 1 -- 172.21.15.16:0/1540281229 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f3160031ec0 con 0x7f31701325e0 2019-12-05T06:19:20.515 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 67 2019-12-05T06:19:20.515 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:20.515 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:20.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.526+0000 7f3176e2a700 1 -- 172.21.15.16:0/1540281229 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3150023630 msgr2=0x7f3150025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.526+0000 7f3176e2a700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3150023630 0x7f3150025ac0 crc :-1 s=READY pgs=236 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.526+0000 7f3176e2a700 1 -- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 msgr2=0x7f317012b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:20.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.526+0000 7f3176e2a700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 0x7f317012b770 secure :-1 s=READY pgs=528 cs=0 l=1 rx=0x7f3160010430 tx=0x7f3160010260).stop 2019-12-05T06:19:20.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.526+0000 7f3176e2a700 1 -- 172.21.15.16:0/1540281229 shutdown_connections 2019-12-05T06:19:20.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.526+0000 7f3176e2a700 1 --2- 172.21.15.16:0/1540281229 >> v2:172.21.15.76:3300/0 conn(0x7f317012c740 0x7f317012bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.526+0000 7f3176e2a700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3150023630 0x7f3150025ac0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.526+0000 7f3176e2a700 1 --2- 172.21.15.16:0/1540281229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31701325e0 0x7f317012b770 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.527+0000 7f3176e2a700 1 -- 172.21.15.16:0/1540281229 shutdown_connections 2019-12-05T06:19:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.527+0000 7f3176e2a700 1 -- 172.21.15.16:0/1540281229 wait complete. 2019-12-05T06:19:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:20.527+0000 7f3176e2a700 1 -- 172.21.15.16:0/1540281229 >> 172.21.15.16:0/1540281229 conn(0x7f31700a2850 msgr2=0x7f317006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:20.534 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:22.536 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:22.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.809+0000 7f88d4ac0700 1 Processor -- start 2019-12-05T06:19:22.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.809+0000 7f88d4ac0700 1 -- start start 2019-12-05T06:19:22.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.811+0000 7f88d4ac0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 0x7f88d008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:22.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.811+0000 7f88d4ac0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00958e0 con 0x7f88d008fef0 2019-12-05T06:19:22.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.812+0000 7f88d4ac0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d00629e0 con 0x7f88d008f720 2019-12-05T06:19:22.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.812+0000 7f88d4ac0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d00a3f70 con 0x7f88d008c6d0 2019-12-05T06:19:22.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.812+0000 7f88ce59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f88d008fef0 0x7f88d008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35990/0 (socket says 172.21.15.16:35990) 2019-12-05T06:19:22.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.812+0000 7f88ce59c700 1 -- 172.21.15.16:0/3271204661 learned_addr learned my addr 172.21.15.16:0/3271204661 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:22.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.812+0000 7f88ced9d700 1 --2- 172.21.15.16:0/3271204661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 0x7f88d008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:22.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.812+0000 7f88cdd9b700 1 -- 172.21.15.16:0/3271204661 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545102649 0 0) 0x7f88b8001150 con 0x7f88d008fef0 2019-12-05T06:19:22.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.812+0000 7f88cdd9b700 1 -- 172.21.15.16:0/3271204661 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88bc002e80 con 0x7f88d008fef0 2019-12-05T06:19:22.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88cdd9b700 1 -- 172.21.15.16:0/3271204661 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222204849 0 0) 0x7f88c0000d30 con 0x7f88d008f720 2019-12-05T06:19:22.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88cdd9b700 1 -- 172.21.15.16:0/3271204661 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88bc0030b0 con 0x7f88d008f720 2019-12-05T06:19:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88ced9d700 1 -- 172.21.15.16:0/3271204661 >> v1:172.21.15.76:6789/0 conn(0x7f88d008f720 legacy=0x7f88d01533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88ced9d700 1 -- 172.21.15.16:0/3271204661 >> v1:172.21.15.16:6789/0 conn(0x7f88d008fef0 legacy=0x7f88d008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88ced9d700 1 -- 172.21.15.16:0/3271204661 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d008fbc0 con 0x7f88d008c6d0 2019-12-05T06:19:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88ced9d700 1 --2- 172.21.15.16:0/3271204661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 0x7f88d008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f88c800af50 tx=0x7f88c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88ced9d700 1 --2- 172.21.15.16:0/3271204661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 0x7f88d008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f88c800af50 tx=0x7f88c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88ced9d700 1 --2- 172.21.15.16:0/3271204661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 0x7f88d008cac0 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7f88c800af50 tx=0x7f88c800b760).ready entity=mon.2 client_cookie=6cdc6a89c23ce027 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:22.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88ced9d700 1 --2- 172.21.15.16:0/3271204661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 0x7f88d008cac0 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f88c800af50 tx=0x7f88c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88cdd9b700 1 -- 172.21.15.16:0/3271204661 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c8000960 con 0x7f88d008c6d0 2019-12-05T06:19:22.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.813+0000 7f88ced9d700 1 --2- 172.21.15.16:0/3271204661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 0x7f88d008cac0 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f88c800af50 tx=0x7f88c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.814+0000 7f88ced9d700 1 --2- 172.21.15.16:0/3271204661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 0x7f88d008cac0 secure :-1 s=THROTTLE_DONE pgs=136 cs=0 l=1 rx=0x7f88c800af50 tx=0x7f88c800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.814+0000 7f88cdd9b700 1 -- 172.21.15.16:0/3271204661 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c8009860 con 0x7f88d008c6d0 2019-12-05T06:19:22.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.814+0000 7f88cdd9b700 1 -- 172.21.15.16:0/3271204661 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c8009f30 con 0x7f88d008c6d0 2019-12-05T06:19:22.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.814+0000 7f88d4ac0700 1 -- 172.21.15.16:0/3271204661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 msgr2=0x7f88d008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.814+0000 7f88d4ac0700 1 --2- 172.21.15.16:0/3271204661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 0x7f88d008cac0 secure :-1 s=READY pgs=136 cs=0 l=1 rx=0x7f88c800af50 tx=0x7f88c800b760).stop 2019-12-05T06:19:22.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.816+0000 7f88d4ac0700 1 -- 172.21.15.16:0/3271204661 shutdown_connections 2019-12-05T06:19:22.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.816+0000 7f88d4ac0700 1 --2- 172.21.15.16:0/3271204661 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d008c6d0 0x7f88d008cac0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:22.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.816+0000 7f88d4ac0700 1 -- 172.21.15.16:0/3271204661 shutdown_connections 2019-12-05T06:19:22.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.816+0000 7f88d4ac0700 1 -- 172.21.15.16:0/3271204661 wait complete. 2019-12-05T06:19:22.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.816+0000 7f88d4ac0700 1 -- 172.21.15.16:0/3271204661 >> 172.21.15.16:0/3271204661 conn(0x7f88d0061940 msgr2=0x7f88d0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:22.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.821+0000 7f88d4ac0700 1 Processor -- start 2019-12-05T06:19:22.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.821+0000 7f88d4ac0700 1 -- start start 2019-12-05T06:19:22.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.822+0000 7f88d4ac0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:22.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.822+0000 7f88d4ac0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d008ab00 con 0x7f88d007b730 2019-12-05T06:19:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.822+0000 7f88d4ac0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88d0088210 con 0x7f88d007bf00 2019-12-05T06:19:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.822+0000 7f88d4ac0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88d0089f00 con 0x7f88d014d410 2019-12-05T06:19:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.823+0000 7f88cf59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f88d007b730 0x7f88d0153150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:35998/0 (socket says 172.21.15.16:35998) 2019-12-05T06:19:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.823+0000 7f88cf59e700 1 -- 172.21.15.16:0/1184470785 learned_addr learned my addr 172.21.15.16:0/1184470785 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.823+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.823+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335209805 0 0) 0x7f88c0001430 con 0x7f88d007b730 2019-12-05T06:19:22.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.823+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b4002e80 con 0x7f88d007b730 2019-12-05T06:19:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.823+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56947688 0 0) 0x7f88b8000f40 con 0x7f88d007bf00 2019-12-05T06:19:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.823+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88b4004170 con 0x7f88d007bf00 2019-12-05T06:19:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88ced9d700 1 -- 172.21.15.16:0/1184470785 >> v1:172.21.15.76:6789/0 conn(0x7f88d007bf00 legacy=0x7f88d007af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88ced9d700 1 -- 172.21.15.16:0/1184470785 >> v1:172.21.15.16:6789/0 conn(0x7f88d007b730 legacy=0x7f88d0153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88ced9d700 1 -- 172.21.15.16:0/1184470785 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88d007bbd0 con 0x7f88d014d410 2019-12-05T06:19:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=READY pgs=137 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c8000940 con 0x7f88d014d410 2019-12-05T06:19:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88c8009770 con 0x7f88d014d410 2019-12-05T06:19:22.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88d4ac0700 1 -- 172.21.15.16:0/1184470785 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88d007bbd0 con 0x7f88d014d410 2019-12-05T06:19:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88c8009ec0 con 0x7f88d014d410 2019-12-05T06:19:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.824+0000 7f88d4ac0700 1 -- 172.21.15.16:0/1184470785 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88d007bbd0 con 0x7f88d014d410 2019-12-05T06:19:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.825+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.826+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.826+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f88c8009960 con 0x7f88d014d410 2019-12-05T06:19:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.826+0000 7f88c77fe700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f88b4023650 0x7f88b4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:22.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.826+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f88c8045dc0 con 0x7f88d014d410 2019-12-05T06:19:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.827+0000 7f88ce59c700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f88b4023650 0x7f88b4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:22.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.827+0000 7f88ce59c700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f88b4023650 0x7f88b4025ae0 crc :-1 s=READY pgs=238 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:22.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.840+0000 7f88d4ac0700 1 -- 172.21.15.16:0/1184470785 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88d007a630 con 0x7f88d014d410 2019-12-05T06:19:22.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.849+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:22.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:22.849+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f88c800b540 con 0x7f88d014d410 2019-12-05T06:19:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.089+0000 7f88d4ac0700 1 -- 172.21.15.16:0/1184470785 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88d007a630 con 0x7f88d014d410 2019-12-05T06:19:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.093+0000 7f88ced9d700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=THROTTLE_DONE pgs=137 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.093+0000 7f88c77fe700 1 -- 172.21.15.16:0/1184470785 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f88c8009960 con 0x7f88d014d410 2019-12-05T06:19:23.087 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:23.087 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:23.088 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 67 2019-12-05T06:19:23.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.099+0000 7f88d4ac0700 1 -- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f88b4023650 msgr2=0x7f88b4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.099+0000 7f88d4ac0700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f88b4023650 0x7f88b4025ae0 crc :-1 s=READY pgs=238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.099+0000 7f88d4ac0700 1 -- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 msgr2=0x7f88d014d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.099+0000 7f88d4ac0700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 secure :-1 s=READY pgs=137 cs=0 l=1 rx=0x7f88c8010380 tx=0x7f88c8010720).stop 2019-12-05T06:19:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.099+0000 7f88d4ac0700 1 -- 172.21.15.16:0/1184470785 shutdown_connections 2019-12-05T06:19:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.099+0000 7f88d4ac0700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f88b4023650 0x7f88b4025ae0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.099+0000 7f88d4ac0700 1 --2- 172.21.15.16:0/1184470785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88d014d410 0x7f88d014d800 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:23.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.100+0000 7f88d4ac0700 1 -- 172.21.15.16:0/1184470785 shutdown_connections 2019-12-05T06:19:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.100+0000 7f88d4ac0700 1 -- 172.21.15.16:0/1184470785 wait complete. 2019-12-05T06:19:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:23.100+0000 7f88d4ac0700 1 -- 172.21.15.16:0/1184470785 >> 172.21.15.16:0/1184470785 conn(0x7f88d006dfd0 msgr2=0x7f88d0070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:23.102 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:25.103 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:25.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.358+0000 7fa6d045e700 1 Processor -- start 2019-12-05T06:19:25.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.358+0000 7fa6d045e700 1 -- start start 2019-12-05T06:19:25.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6d045e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c808c6d0 0x7fa6c808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:25.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6d045e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6c808fef0 0x7fa6c808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:25.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6d045e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa6c808ef20 0x7fa6c808f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:25.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6d045e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c80a3f70 con 0x7fa6c808fef0 2019-12-05T06:19:25.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6d045e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c808d830 con 0x7fa6c808ef20 2019-12-05T06:19:25.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6d045e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c808d540 con 0x7fa6c808c6d0 2019-12-05T06:19:25.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6ce1fa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6c808fef0 0x7fa6c808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:25.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6ce1fa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6c808fef0 0x7fa6c808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50838/0 (socket says 172.21.15.16:50838) 2019-12-05T06:19:25.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6ce1fa700 1 -- 172.21.15.16:0/1769571772 learned_addr learned my addr 172.21.15.16:0/1769571772 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:25.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6ce9fb700 1 --2- 172.21.15.16:0/1769571772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c808c6d0 0x7fa6c808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:25.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.360+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/1769571772 >> v2:172.21.15.76:3300/0 conn(0x7fa6c808ef20 0x7fa6c808f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:25.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.361+0000 7fa6ce1fa700 1 -- 172.21.15.16:0/1769571772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c808c6d0 msgr2=0x7fa6c808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:25.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.361+0000 7fa6ce1fa700 1 --2- 172.21.15.16:0/1769571772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c808c6d0 0x7fa6c808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:25.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.361+0000 7fa6ce1fa700 1 -- 172.21.15.16:0/1769571772 >> v2:172.21.15.76:3300/0 conn(0x7fa6c808ef20 msgr2=0x7fa6c808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:25.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.361+0000 7fa6ce1fa700 1 --2- 172.21.15.16:0/1769571772 >> v2:172.21.15.76:3300/0 conn(0x7fa6c808ef20 0x7fa6c808f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:25.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.361+0000 7fa6ce1fa700 1 -- 172.21.15.16:0/1769571772 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c808e620 con 0x7fa6c808fef0 2019-12-05T06:19:25.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.361+0000 7fa6ce9fb700 1 --2- 172.21.15.16:0/1769571772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c808c6d0 0x7fa6c808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:25.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.361+0000 7fa6ce1fa700 1 --2- 172.21.15.16:0/1769571772 >> v2:172.21.15.16:3300/0 conn(0x7fa6c808fef0 0x7fa6c808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa6bc00f740 tx=0x7fa6bc00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.361+0000 7fa6ce1fa700 1 --2- 172.21.15.16:0/1769571772 >> v2:172.21.15.16:3300/0 conn(0x7fa6c808fef0 0x7fa6c808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa6bc00f740 tx=0x7fa6bc00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.361+0000 7fa6ce1fa700 1 --2- 172.21.15.16:0/1769571772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c808fef0 0x7fa6c808d000 secure :-1 s=READY pgs=531 cs=0 l=1 rx=0x7fa6bc00f740 tx=0x7fa6bc00f5f0).ready entity=mon.0 client_cookie=7a41dea30bf989c4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:25.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.362+0000 7fa6ce1fa700 1 --2- 172.21.15.16:0/1769571772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c808fef0 0x7fa6c808d000 secure :-1 s=THROTTLE_DONE pgs=531 cs=0 l=1 rx=0x7fa6bc00f740 tx=0x7fa6bc00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.362+0000 7fa6ce1fa700 1 --2- 172.21.15.16:0/1769571772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c808fef0 0x7fa6c808d000 secure :-1 s=THROTTLE_DONE pgs=531 cs=0 l=1 rx=0x7fa6bc00f740 tx=0x7fa6bc00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.362+0000 7fa6ce1fa700 1 --2- 172.21.15.16:0/1769571772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c808fef0 0x7fa6c808d000 secure :-1 s=THROTTLE_DONE pgs=531 cs=0 l=1 rx=0x7fa6bc00f740 tx=0x7fa6bc00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.362+0000 7fa6cd9f9700 1 -- 172.21.15.16:0/1769571772 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6bc000960 con 0x7fa6c808fef0 2019-12-05T06:19:25.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.362+0000 7fa6cd9f9700 1 -- 172.21.15.16:0/1769571772 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6bc000c90 con 0x7fa6c808fef0 2019-12-05T06:19:25.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.362+0000 7fa6cd9f9700 1 -- 172.21.15.16:0/1769571772 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6bc0013a0 con 0x7fa6c808fef0 2019-12-05T06:19:25.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.362+0000 7fa6d045e700 1 -- 172.21.15.16:0/1769571772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c808fef0 msgr2=0x7fa6c808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:25.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.362+0000 7fa6d045e700 1 --2- 172.21.15.16:0/1769571772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c808fef0 0x7fa6c808d000 secure :-1 s=READY pgs=531 cs=0 l=1 rx=0x7fa6bc00f740 tx=0x7fa6bc00f5f0).stop 2019-12-05T06:19:25.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.364+0000 7fa6d045e700 1 -- 172.21.15.16:0/1769571772 shutdown_connections 2019-12-05T06:19:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.364+0000 7fa6d045e700 1 --2- 172.21.15.16:0/1769571772 >> v2:172.21.15.76:3300/0 conn(0x7fa6c808ef20 0x7fa6c808f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.364+0000 7fa6d045e700 1 --2- 172.21.15.16:0/1769571772 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c808fef0 0x7fa6c808d000 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:25.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.364+0000 7fa6d045e700 1 --2- 172.21.15.16:0/1769571772 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c808c6d0 0x7fa6c808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:25.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.364+0000 7fa6d045e700 1 -- 172.21.15.16:0/1769571772 shutdown_connections 2019-12-05T06:19:25.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.364+0000 7fa6d045e700 1 -- 172.21.15.16:0/1769571772 wait complete. 2019-12-05T06:19:25.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.364+0000 7fa6d045e700 1 -- 172.21.15.16:0/1769571772 >> 172.21.15.16:0/1769571772 conn(0x7fa6c8061940 msgr2=0x7fa6c8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:25.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.369+0000 7fa6d045e700 1 Processor -- start 2019-12-05T06:19:25.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.370+0000 7fa6d045e700 1 -- start start 2019-12-05T06:19:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6d045e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c8096e90 0x7fa6c809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6d045e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa6c8084f30 0x7fa6c809b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6d045e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6c8097280 0x7fa6c8097700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6d045e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c8099a20 con 0x7fa6c8097280 2019-12-05T06:19:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6d045e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6c8099cf0 con 0x7fa6c8084f30 2019-12-05T06:19:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6d045e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6c809aae0 con 0x7fa6c8096e90 2019-12-05T06:19:25.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6cf1fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6c8097280 0x7fa6c8097700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:25.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6cf1fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6c8097280 0x7fa6c8097700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50846/0 (socket says 172.21.15.16:50846) 2019-12-05T06:19:25.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6cf1fc700 1 -- 172.21.15.16:0/3729307862 learned_addr learned my addr 172.21.15.16:0/3729307862 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:25.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6ce9fb700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c8096e90 0x7fa6c809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:25.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.371+0000 7fa6ce1fa700 1 --2- 172.21.15.16:0/3729307862 >> v2:172.21.15.76:3300/0 conn(0x7fa6c8084f30 0x7fa6c809b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:25.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.372+0000 7fa6cf1fc700 1 -- 172.21.15.16:0/3729307862 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c8096e90 msgr2=0x7fa6c809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:25.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.372+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c8096e90 0x7fa6c809b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:25.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.372+0000 7fa6cf1fc700 1 -- 172.21.15.16:0/3729307862 >> v2:172.21.15.76:3300/0 conn(0x7fa6c8084f30 msgr2=0x7fa6c809b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:25.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.372+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> v2:172.21.15.76:3300/0 conn(0x7fa6c8084f30 0x7fa6c809b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:25.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.372+0000 7fa6cf1fc700 1 -- 172.21.15.16:0/3729307862 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6c8084b90 con 0x7fa6c8097280 2019-12-05T06:19:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.372+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> v2:172.21.15.16:3300/0 conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.372+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> v2:172.21.15.16:3300/0 conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.372+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=READY pgs=532 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.372+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.373+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.373+0000 7fa6c77fe700 1 -- 172.21.15.16:0/3729307862 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6c0000b10 con 0x7fa6c8097280 2019-12-05T06:19:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.373+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.373+0000 7fa6c77fe700 1 -- 172.21.15.16:0/3729307862 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6c000ef40 con 0x7fa6c8097280 2019-12-05T06:19:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.373+0000 7fa6d045e700 1 -- 172.21.15.16:0/3729307862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6c8084b90 con 0x7fa6c8097280 2019-12-05T06:19:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.373+0000 7fa6c77fe700 1 -- 172.21.15.16:0/3729307862 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6c000f5d0 con 0x7fa6c8097280 2019-12-05T06:19:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.373+0000 7fa6d045e700 1 -- 172.21.15.16:0/3729307862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6c8084b90 con 0x7fa6c8097280 2019-12-05T06:19:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.374+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.375+0000 7fa6c77fe700 1 -- 172.21.15.16:0/3729307862 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa6c000eb20 con 0x7fa6c8097280 2019-12-05T06:19:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.375+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.375+0000 7fa6c77fe700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6a8023630 0x7fa6a8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.375+0000 7fa6c77fe700 1 -- 172.21.15.16:0/3729307862 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7fa6c0045d40 con 0x7fa6c8097280 2019-12-05T06:19:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.375+0000 7fa6ce9fb700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6a8023630 0x7fa6a8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.376+0000 7fa6ce9fb700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6a8023630 0x7fa6a8025ac0 crc :-1 s=READY pgs=239 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:25.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.389+0000 7fa6d045e700 1 -- 172.21.15.16:0/3729307862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6c80958e0 con 0x7fa6c8097280 2019-12-05T06:19:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.396+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.397+0000 7fa6c77fe700 1 -- 172.21.15.16:0/3729307862 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa6c0000b10 con 0x7fa6c8097280 2019-12-05T06:19:25.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.644+0000 7fa6d045e700 1 -- 172.21.15.16:0/3729307862 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa6c80958e0 con 0x7fa6c8097280 2019-12-05T06:19:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.645+0000 7fa6cf1fc700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.645+0000 7fa6c77fe700 1 -- 172.21.15.16:0/3729307862 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fa6c000ea80 con 0x7fa6c8097280 2019-12-05T06:19:25.636 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 67 2019-12-05T06:19:25.637 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:25.637 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.647+0000 7fa6d045e700 1 -- 172.21.15.16:0/3729307862 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6a8023630 msgr2=0x7fa6a8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.647+0000 7fa6d045e700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6a8023630 0x7fa6a8025ac0 crc :-1 s=READY pgs=239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.647+0000 7fa6d045e700 1 -- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 msgr2=0x7fa6c8097700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.647+0000 7fa6d045e700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 0x7fa6c8097700 secure :-1 s=READY pgs=532 cs=0 l=1 rx=0x7fa6c0010450 tx=0x7fa6c0010300).stop 2019-12-05T06:19:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.648+0000 7fa6d045e700 1 -- 172.21.15.16:0/3729307862 shutdown_connections 2019-12-05T06:19:25.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.648+0000 7fa6d045e700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa6a8023630 0x7fa6a8025ac0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.648+0000 7fa6d045e700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6c8097280 0x7fa6c8097700 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.648+0000 7fa6d045e700 1 --2- 172.21.15.16:0/3729307862 >> v2:172.21.15.76:3300/0 conn(0x7fa6c8084f30 0x7fa6c809b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.648+0000 7fa6d045e700 1 --2- 172.21.15.16:0/3729307862 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6c8096e90 0x7fa6c809b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.648+0000 7fa6d045e700 1 -- 172.21.15.16:0/3729307862 shutdown_connections 2019-12-05T06:19:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.648+0000 7fa6d045e700 1 -- 172.21.15.16:0/3729307862 wait complete. 2019-12-05T06:19:25.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:25.648+0000 7fa6d045e700 1 -- 172.21.15.16:0/3729307862 >> 172.21.15.16:0/3729307862 conn(0x7fa6c80714f0 msgr2=0x7fa6c809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:25.655 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:27.657 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:27.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.901+0000 7f27c22aa700 1 Processor -- start 2019-12-05T06:19:27.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.901+0000 7f27c22aa700 1 -- start start 2019-12-05T06:19:27.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.902+0000 7f27c22aa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27bc08d2c0 0x7f27bc08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:27.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.902+0000 7f27c22aa700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f27bc08fef0 0x7f27bc08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:27.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.902+0000 7f27c22aa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc0958e0 con 0x7f27bc08c6d0 2019-12-05T06:19:27.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.902+0000 7f27c22aa700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc08df90 con 0x7f27bc08d2c0 2019-12-05T06:19:27.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.902+0000 7f27c22aa700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f27bc0a3f70 con 0x7f27bc08fef0 2019-12-05T06:19:27.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.903+0000 7f27bbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27bc08d2c0 0x7f27bc08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:27.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.903+0000 7f27bbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27bc08d2c0 0x7f27bc08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50858/0 (socket says 172.21.15.16:50858) 2019-12-05T06:19:27.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.903+0000 7f27bbfff700 1 -- 172.21.15.16:0/437891616 learned_addr learned my addr 172.21.15.16:0/437891616 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:27.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.903+0000 7f27c1048700 1 --2- 172.21.15.16:0/437891616 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f27bc08fef0 0x7f27bc08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:27.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.903+0000 7f27bbfff700 1 -- 172.21.15.16:0/437891616 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f27bc08fef0 msgr2=0x7f27bc08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.903+0000 7f27bbfff700 1 --2- 172.21.15.16:0/437891616 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f27bc08fef0 0x7f27bc08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.903+0000 7f27bbfff700 1 -- 172.21.15.16:0/437891616 >> v1:172.21.15.76:6789/0 conn(0x7f27bc08c6d0 legacy=0x7f27bc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.903+0000 7f27bbfff700 1 -- 172.21.15.16:0/437891616 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc08e620 con 0x7f27bc08d2c0 2019-12-05T06:19:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.903+0000 7f27bb7fe700 1 -- 172.21.15.16:0/437891616 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241451949 0 0) 0x7f27ac001080 con 0x7f27bc08c6d0 2019-12-05T06:19:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.903+0000 7f27bbfff700 1 --2- 172.21.15.16:0/437891616 >> v2:172.21.15.16:3300/0 conn(0x7f27bc08d2c0 0x7f27bc08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f27b000f6c0 tx=0x7f27b000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.904+0000 7f27bbfff700 1 --2- 172.21.15.16:0/437891616 >> v2:172.21.15.16:3300/0 conn(0x7f27bc08d2c0 0x7f27bc08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f27b000f6c0 tx=0x7f27b000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.904+0000 7f27bbfff700 1 --2- 172.21.15.16:0/437891616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc08d2c0 0x7f27bc08ef20 secure :-1 s=READY pgs=533 cs=0 l=1 rx=0x7f27b000f6c0 tx=0x7f27b000fa60).ready entity=mon.0 client_cookie=4e30be4f8b7b0848 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.904+0000 7f27bbfff700 1 --2- 172.21.15.16:0/437891616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc08d2c0 0x7f27bc08ef20 secure :-1 s=THROTTLE_DONE pgs=533 cs=0 l=1 rx=0x7f27b000f6c0 tx=0x7f27b000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.904+0000 7f27bb7fe700 1 -- 172.21.15.16:0/437891616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27b0000960 con 0x7f27bc08d2c0 2019-12-05T06:19:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.904+0000 7f27bbfff700 1 --2- 172.21.15.16:0/437891616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc08d2c0 0x7f27bc08ef20 secure :-1 s=THROTTLE_DONE pgs=533 cs=0 l=1 rx=0x7f27b000f6c0 tx=0x7f27b000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.904+0000 7f27bb7fe700 1 -- 172.21.15.16:0/437891616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b0000c90 con 0x7f27bc08d2c0 2019-12-05T06:19:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.904+0000 7f27bbfff700 1 --2- 172.21.15.16:0/437891616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc08d2c0 0x7f27bc08ef20 secure :-1 s=THROTTLE_DONE pgs=533 cs=0 l=1 rx=0x7f27b000f6c0 tx=0x7f27b000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.904+0000 7f27bb7fe700 1 -- 172.21.15.16:0/437891616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27b0007210 con 0x7f27bc08d2c0 2019-12-05T06:19:27.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.904+0000 7f27c22aa700 1 -- 172.21.15.16:0/437891616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc08d2c0 msgr2=0x7f27bc08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.904+0000 7f27c22aa700 1 --2- 172.21.15.16:0/437891616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc08d2c0 0x7f27bc08ef20 secure :-1 s=READY pgs=533 cs=0 l=1 rx=0x7f27b000f6c0 tx=0x7f27b000fa60).stop 2019-12-05T06:19:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.906+0000 7f27c22aa700 1 -- 172.21.15.16:0/437891616 shutdown_connections 2019-12-05T06:19:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.906+0000 7f27c22aa700 1 --2- 172.21.15.16:0/437891616 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f27bc08fef0 0x7f27bc08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.906+0000 7f27c22aa700 1 --2- 172.21.15.16:0/437891616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc08d2c0 0x7f27bc08ef20 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.906+0000 7f27c22aa700 1 -- 172.21.15.16:0/437891616 shutdown_connections 2019-12-05T06:19:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.906+0000 7f27c22aa700 1 -- 172.21.15.16:0/437891616 wait complete. 2019-12-05T06:19:27.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.906+0000 7f27c22aa700 1 -- 172.21.15.16:0/437891616 >> 172.21.15.16:0/437891616 conn(0x7f27bc061940 msgr2=0x7f27bc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.911+0000 7f27c22aa700 1 Processor -- start 2019-12-05T06:19:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.911+0000 7f27c22aa700 1 -- start start 2019-12-05T06:19:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.912+0000 7f27c22aa700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f27bc0879b0 0x7f27bc0871e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.912+0000 7f27c22aa700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27bc09be80 0x7f27bc09c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.912+0000 7f27c22aa700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27bc153170 con 0x7f27bc14ca10 2019-12-05T06:19:27.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.912+0000 7f27c22aa700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc085a50 con 0x7f27bc09be80 2019-12-05T06:19:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.912+0000 7f27c22aa700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f27bc085020 con 0x7f27bc0879b0 2019-12-05T06:19:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27c1048700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27bc09be80 0x7f27bc09c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27c1048700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f27bc09be80 0x7f27bc09c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50866/0 (socket says 172.21.15.16:50866) 2019-12-05T06:19:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27c1048700 1 -- 172.21.15.16:0/3171266336 learned_addr learned my addr 172.21.15.16:0/3171266336 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27bbfff700 1 --2- 172.21.15.16:0/3171266336 >> v2:172.21.15.76:3300/0 conn(0x7f27bc0879b0 0x7f27bc0871e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27c1048700 1 -- 172.21.15.16:0/3171266336 >> v2:172.21.15.76:3300/0 conn(0x7f27bc0879b0 msgr2=0x7f27bc0871e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:27.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> v2:172.21.15.76:3300/0 conn(0x7f27bc0879b0 0x7f27bc0871e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27c1048700 1 -- 172.21.15.16:0/3171266336 >> v1:172.21.15.16:6789/0 conn(0x7f27bc14ca10 legacy=0x7f27bc0869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27c1048700 1 -- 172.21.15.16:0/3171266336 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27bc0860e0 con 0x7f27bc09be80 2019-12-05T06:19:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> v2:172.21.15.16:3300/0 conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> v2:172.21.15.16:3300/0 conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.913+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=READY pgs=535 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.914+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.914+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.914+0000 7f27b97fa700 1 -- 172.21.15.16:0/3171266336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27b400f410 con 0x7f27bc09be80 2019-12-05T06:19:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.914+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.914+0000 7f27b97fa700 1 -- 172.21.15.16:0/3171266336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27b4000df0 con 0x7f27bc09be80 2019-12-05T06:19:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.914+0000 7f27c22aa700 1 -- 172.21.15.16:0/3171266336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27bc0860e0 con 0x7f27bc09be80 2019-12-05T06:19:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.914+0000 7f27b97fa700 1 -- 172.21.15.16:0/3171266336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27b400f9f0 con 0x7f27bc09be80 2019-12-05T06:19:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.914+0000 7f27c22aa700 1 -- 172.21.15.16:0/3171266336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27bc0860e0 con 0x7f27bc09be80 2019-12-05T06:19:27.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.915+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.915+0000 7f27b97fa700 1 -- 172.21.15.16:0/3171266336 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f27b400f170 con 0x7f27bc09be80 2019-12-05T06:19:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.915+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.916+0000 7f27b97fa700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f279c023630 0x7f279c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.916+0000 7f27b97fa700 1 -- 172.21.15.16:0/3171266336 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f27b4048350 con 0x7f27bc09be80 2019-12-05T06:19:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.916+0000 7f27c0847700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f279c023630 0x7f279c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:27.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.917+0000 7f27c0847700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f279c023630 0x7f279c025ac0 crc :-1 s=READY pgs=240 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:27.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.932+0000 7f27c22aa700 1 -- 172.21.15.16:0/3171266336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27bc0860e0 con 0x7f27bc09be80 2019-12-05T06:19:27.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.939+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:27.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:27.939+0000 7f27b97fa700 1 -- 172.21.15.16:0/3171266336 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f27b400ee20 con 0x7f27bc09be80 2019-12-05T06:19:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.176+0000 7f27c22aa700 1 -- 172.21.15.16:0/3171266336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27bc0860e0 con 0x7f27bc09be80 2019-12-05T06:19:28.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.177+0000 7f27c1048700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=THROTTLE_DONE pgs=535 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:28.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.177+0000 7f27b97fa700 1 -- 172.21.15.16:0/3171266336 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f27b400f0b0 con 0x7f27bc09be80 2019-12-05T06:19:28.168 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:28.168 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:28.169 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 67 2019-12-05T06:19:28.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.179+0000 7f27c22aa700 1 -- 172.21.15.16:0/3171266336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f279c023630 msgr2=0x7f279c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:28.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.179+0000 7f27c22aa700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f279c023630 0x7f279c025ac0 crc :-1 s=READY pgs=240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:28.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.179+0000 7f27c22aa700 1 -- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 msgr2=0x7f27bc09c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:28.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.179+0000 7f27c22aa700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 0x7f27bc09c2e0 secure :-1 s=READY pgs=535 cs=0 l=1 rx=0x7f27b40128c0 tx=0x7f27b4012c60).stop 2019-12-05T06:19:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.180+0000 7f27c22aa700 1 -- 172.21.15.16:0/3171266336 shutdown_connections 2019-12-05T06:19:28.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.180+0000 7f27c22aa700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f279c023630 0x7f279c025ac0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.180+0000 7f27c22aa700 1 --2- 172.21.15.16:0/3171266336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f27bc09be80 0x7f27bc09c2e0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.180+0000 7f27c22aa700 1 --2- 172.21.15.16:0/3171266336 >> v2:172.21.15.76:3300/0 conn(0x7f27bc0879b0 0x7f27bc0871e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.180+0000 7f27c22aa700 1 -- 172.21.15.16:0/3171266336 shutdown_connections 2019-12-05T06:19:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.180+0000 7f27c22aa700 1 -- 172.21.15.16:0/3171266336 wait complete. 2019-12-05T06:19:28.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:28.180+0000 7f27c22aa700 1 -- 172.21.15.16:0/3171266336 >> 172.21.15.16:0/3171266336 conn(0x7f27bc098480 msgr2=0x7f27bc071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:28.185 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:30.190 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.480+0000 7f3624ce7700 1 Processor -- start 2019-12-05T06:19:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.480+0000 7f3624ce7700 1 -- start start 2019-12-05T06:19:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.482+0000 7f3624ce7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f362008c6d0 0x7f362008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:30.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.482+0000 7f3624ce7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f362008fef0 0x7f362008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.482+0000 7f3624ce7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36200958e0 con 0x7f362008ef20 2019-12-05T06:19:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.482+0000 7f3624ce7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36200a3f70 con 0x7f362008fef0 2019-12-05T06:19:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.482+0000 7f3624ce7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f362008d830 con 0x7f362008c6d0 2019-12-05T06:19:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.482+0000 7f361f59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f362008ef20 0x7f362008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36042/0 (socket says 172.21.15.16:36042) 2019-12-05T06:19:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.482+0000 7f361f59e700 1 -- 172.21.15.16:0/2858088082 learned_addr learned my addr 172.21.15.16:0/2858088082 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.483+0000 7f361e59c700 1 --2- 172.21.15.16:0/2858088082 >> v2:172.21.15.76:3300/0 conn(0x7f362008fef0 0x7f362008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.483+0000 7f361ed9d700 1 --2- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f362008c6d0 0x7f362008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.483+0000 7f361dd9b700 1 -- 172.21.15.16:0/2858088082 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329726273 0 0) 0x7f3610001150 con 0x7f362008ef20 2019-12-05T06:19:30.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.483+0000 7f361dd9b700 1 -- 172.21.15.16:0/2858088082 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f360c002e80 con 0x7f362008ef20 2019-12-05T06:19:30.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.483+0000 7f361e59c700 1 -- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f362008c6d0 msgr2=0x7f362008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:30.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.483+0000 7f361e59c700 1 --2- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f362008c6d0 0x7f362008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:30.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.483+0000 7f361e59c700 1 -- 172.21.15.16:0/2858088082 >> v1:172.21.15.16:6789/0 conn(0x7f362008ef20 legacy=0x7f362008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.484+0000 7f361e59c700 1 -- 172.21.15.16:0/2858088082 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f362008d540 con 0x7f362008fef0 2019-12-05T06:19:30.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.484+0000 7f361ed9d700 1 --2- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f362008c6d0 0x7f362008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.484+0000 7f361e59c700 1 --2- 172.21.15.16:0/2858088082 >> v2:172.21.15.76:3300/0 conn(0x7f362008fef0 0x7f362008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f360800b2d0 tx=0x7f360800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.484+0000 7f361e59c700 1 --2- 172.21.15.16:0/2858088082 >> v2:172.21.15.76:3300/0 conn(0x7f362008fef0 0x7f362008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f360800b2d0 tx=0x7f360800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.484+0000 7f361e59c700 1 --2- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362008fef0 0x7f362008d000 secure :-1 s=READY pgs=400 cs=0 l=1 rx=0x7f360800b2d0 tx=0x7f360800bae0).ready entity=mon.1 client_cookie=f419bddd3b9d070d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.484+0000 7f361e59c700 1 --2- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362008fef0 0x7f362008d000 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f360800b2d0 tx=0x7f360800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.484+0000 7f361e59c700 1 --2- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362008fef0 0x7f362008d000 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f360800b2d0 tx=0x7f360800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.484+0000 7f361dd9b700 1 -- 172.21.15.16:0/2858088082 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3608007590 con 0x7f362008fef0 2019-12-05T06:19:30.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.484+0000 7f361e59c700 1 --2- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362008fef0 0x7f362008d000 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f360800b2d0 tx=0x7f360800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.485+0000 7f361dd9b700 1 -- 172.21.15.16:0/2858088082 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608009d30 con 0x7f362008fef0 2019-12-05T06:19:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.485+0000 7f361dd9b700 1 -- 172.21.15.16:0/2858088082 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f360800a400 con 0x7f362008fef0 2019-12-05T06:19:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.485+0000 7f3624ce7700 1 -- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362008fef0 msgr2=0x7f362008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.485+0000 7f3624ce7700 1 --2- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362008fef0 0x7f362008d000 secure :-1 s=READY pgs=400 cs=0 l=1 rx=0x7f360800b2d0 tx=0x7f360800bae0).stop 2019-12-05T06:19:30.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.487+0000 7f3624ce7700 1 -- 172.21.15.16:0/2858088082 shutdown_connections 2019-12-05T06:19:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.487+0000 7f3624ce7700 1 --2- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f362008fef0 0x7f362008d000 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.487+0000 7f3624ce7700 1 --2- 172.21.15.16:0/2858088082 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f362008c6d0 0x7f362008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.487+0000 7f3624ce7700 1 -- 172.21.15.16:0/2858088082 shutdown_connections 2019-12-05T06:19:30.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.487+0000 7f3624ce7700 1 -- 172.21.15.16:0/2858088082 wait complete. 2019-12-05T06:19:30.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.487+0000 7f3624ce7700 1 -- 172.21.15.16:0/2858088082 >> 172.21.15.16:0/2858088082 conn(0x7f3620061940 msgr2=0x7f3620096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:30.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.492+0000 7f3624ce7700 1 Processor -- start 2019-12-05T06:19:30.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.492+0000 7f3624ce7700 1 -- start start 2019-12-05T06:19:30.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.494+0000 7f3624ce7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f362012b640 0x7f362012ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:30.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.494+0000 7f3624ce7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3620153b50 0x7f3620155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:30.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.494+0000 7f3624ce7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3620131bf0 con 0x7f36201302f0 2019-12-05T06:19:30.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.494+0000 7f3624ce7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36201296e0 con 0x7f3620153b50 2019-12-05T06:19:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.494+0000 7f3624ce7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3620128cb0 con 0x7f362012b640 2019-12-05T06:19:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.494+0000 7f361f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3620153b50 0x7f3620155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.494+0000 7f361f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3620153b50 0x7f3620155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50882/0 (socket says 172.21.15.16:50882) 2019-12-05T06:19:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.494+0000 7f361f59e700 1 -- 172.21.15.16:0/1133361587 learned_addr learned my addr 172.21.15.16:0/1133361587 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.494+0000 7f361e59c700 1 --2- 172.21.15.16:0/1133361587 >> v2:172.21.15.76:3300/0 conn(0x7f362012b640 0x7f362012ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f361f59e700 1 -- 172.21.15.16:0/1133361587 >> v2:172.21.15.76:3300/0 conn(0x7f362012b640 msgr2=0x7f362012ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> v2:172.21.15.76:3300/0 conn(0x7f362012b640 0x7f362012ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f361f59e700 1 -- 172.21.15.16:0/1133361587 >> v1:172.21.15.76:6789/0 conn(0x7f36201302f0 legacy=0x7f362012a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f36177fe700 1 -- 172.21.15.16:0/1133361587 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222628187 0 0) 0x7f361800db80 con 0x7f36201302f0 2019-12-05T06:19:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f361f59e700 1 -- 172.21.15.16:0/1133361587 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3620129d70 con 0x7f3620153b50 2019-12-05T06:19:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> v2:172.21.15.16:3300/0 conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> v2:172.21.15.16:3300/0 conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=READY pgs=537 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:30.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=THROTTLE_DONE pgs=537 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f36177fe700 1 -- 172.21.15.16:0/1133361587 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3610000ef0 con 0x7f3620153b50 2019-12-05T06:19:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=THROTTLE_DONE pgs=537 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=THROTTLE_DONE pgs=537 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f3624ce7700 1 -- 172.21.15.16:0/1133361587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3620129d70 con 0x7f3620153b50 2019-12-05T06:19:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f36177fe700 1 -- 172.21.15.16:0/1133361587 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3610004430 con 0x7f3620153b50 2019-12-05T06:19:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.495+0000 7f36177fe700 1 -- 172.21.15.16:0/1133361587 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3610004620 con 0x7f3620153b50 2019-12-05T06:19:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.496+0000 7f3624ce7700 1 -- 172.21.15.16:0/1133361587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3620129d70 con 0x7f3620153b50 2019-12-05T06:19:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.497+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=THROTTLE_DONE pgs=537 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.498+0000 7f36177fe700 1 -- 172.21.15.16:0/1133361587 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3610004620 con 0x7f3620153b50 2019-12-05T06:19:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.498+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=THROTTLE_DONE pgs=537 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.498+0000 7f36177fe700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3604023630 0x7f3604025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.498+0000 7f36177fe700 1 -- 172.21.15.16:0/1133361587 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7f3610045e30 con 0x7f3620153b50 2019-12-05T06:19:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.499+0000 7f361ed9d700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3604023630 0x7f3604025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.499+0000 7f361ed9d700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3604023630 0x7f3604025ac0 crc :-1 s=READY pgs=241 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:30.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.512+0000 7f3624ce7700 1 -- 172.21.15.16:0/1133361587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3620129d70 con 0x7f3620153b50 2019-12-05T06:19:30.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.520+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=THROTTLE_DONE pgs=537 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.520+0000 7f36177fe700 1 -- 172.21.15.16:0/1133361587 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f36100012e0 con 0x7f3620153b50 2019-12-05T06:19:30.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.768+0000 7f3624ce7700 1 -- 172.21.15.16:0/1133361587 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3620129d70 con 0x7f3620153b50 2019-12-05T06:19:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.768+0000 7f361f59e700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=THROTTLE_DONE pgs=537 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.768+0000 7f36177fe700 1 -- 172.21.15.16:0/1133361587 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f3610004620 con 0x7f3620153b50 2019-12-05T06:19:30.759 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 67 2019-12-05T06:19:30.759 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:30.760 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":67,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5490,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":67}],"filesystems":[{"mdsmap":{"epoch":66,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:18:02.181344+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":56,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5382},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5382":{"gid":5382,"name":"b","rank":0,"incarnation":63,"state":"up:active","state_seq":22,"addr":"172.21.15.76:6835/1423600579","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1423600579},{"type":"v1","addr":"172.21.15.76:6835","nonce":1423600579}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.771+0000 7f3624ce7700 1 -- 172.21.15.16:0/1133361587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3604023630 msgr2=0x7f3604025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.771+0000 7f3624ce7700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3604023630 0x7f3604025ac0 crc :-1 s=READY pgs=241 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.771+0000 7f3624ce7700 1 -- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 msgr2=0x7f3620155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.771+0000 7f3624ce7700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 0x7f3620155f50 secure :-1 s=READY pgs=537 cs=0 l=1 rx=0x7f3610010430 tx=0x7f3610010260).stop 2019-12-05T06:19:30.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.771+0000 7f3624ce7700 1 -- 172.21.15.16:0/1133361587 shutdown_connections 2019-12-05T06:19:30.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.771+0000 7f3624ce7700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3604023630 0x7f3604025ac0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.771+0000 7f3624ce7700 1 --2- 172.21.15.16:0/1133361587 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3620153b50 0x7f3620155f50 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.771+0000 7f3624ce7700 1 --2- 172.21.15.16:0/1133361587 >> v2:172.21.15.76:3300/0 conn(0x7f362012b640 0x7f362012ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.772+0000 7f3624ce7700 1 -- 172.21.15.16:0/1133361587 shutdown_connections 2019-12-05T06:19:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.772+0000 7f3624ce7700 1 -- 172.21.15.16:0/1133361587 wait complete. 2019-12-05T06:19:30.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:30.772+0000 7f3624ce7700 1 -- 172.21.15.16:0/1133361587 >> 172.21.15.16:0/1133361587 conn(0x7f36200a2850 msgr2=0x7f362006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:30.773 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:19:31.712 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:19:31.725 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:19:32.774 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.056+0000 7fec04071700 1 Processor -- start 2019-12-05T06:19:33.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.056+0000 7fec04071700 1 -- start start 2019-12-05T06:19:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.057+0000 7fec04071700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7febfc08c6d0 0x7febfc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec04071700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc08fef0 0x7febfc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec04071700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7febfc08ef20 0x7febfc08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec04071700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febfc08df90 con 0x7febfc08c6d0 2019-12-05T06:19:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec04071700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febfc0a3f70 con 0x7febfc08ef20 2019-12-05T06:19:33.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec04071700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febfc08d830 con 0x7febfc08fef0 2019-12-05T06:19:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec0260e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7febfc08c6d0 0x7febfc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec0260e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7febfc08c6d0 0x7febfc08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50890/0 (socket says 172.21.15.16:50890) 2019-12-05T06:19:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec0260e700 1 -- 172.21.15.16:0/15758961 learned_addr learned my addr 172.21.15.16:0/15758961 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec01e0d700 1 --2- 172.21.15.16:0/15758961 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc08fef0 0x7febfc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec02e0f700 1 --2- 172.21.15.16:0/15758961 >> v2:172.21.15.76:3300/0 conn(0x7febfc08ef20 0x7febfc08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec0260e700 1 -- 172.21.15.16:0/15758961 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc08fef0 msgr2=0x7febfc08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:33.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.058+0000 7fec0260e700 1 --2- 172.21.15.16:0/15758961 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc08fef0 0x7febfc08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0260e700 1 -- 172.21.15.16:0/15758961 >> v2:172.21.15.76:3300/0 conn(0x7febfc08ef20 msgr2=0x7febfc08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0260e700 1 --2- 172.21.15.16:0/15758961 >> v2:172.21.15.76:3300/0 conn(0x7febfc08ef20 0x7febfc08f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0260e700 1 -- 172.21.15.16:0/15758961 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc08d540 con 0x7febfc08c6d0 2019-12-05T06:19:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec01e0d700 1 --2- 172.21.15.16:0/15758961 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc08fef0 0x7febfc08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:19:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0260e700 1 --2- 172.21.15.16:0/15758961 >> v2:172.21.15.16:3300/0 conn(0x7febfc08c6d0 0x7febfc08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7febf400f740 tx=0x7febf400f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0260e700 1 --2- 172.21.15.16:0/15758961 >> v2:172.21.15.16:3300/0 conn(0x7febfc08c6d0 0x7febfc08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7febf400f740 tx=0x7febf400f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0260e700 1 --2- 172.21.15.16:0/15758961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febfc08c6d0 0x7febfc08cac0 secure :-1 s=READY pgs=538 cs=0 l=1 rx=0x7febf400f740 tx=0x7febf400f5f0).ready entity=mon.0 client_cookie=443dc329d4b9f317 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0260e700 1 --2- 172.21.15.16:0/15758961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febfc08c6d0 0x7febfc08cac0 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7febf400f740 tx=0x7febf400f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0260e700 1 --2- 172.21.15.16:0/15758961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febfc08c6d0 0x7febfc08cac0 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7febf400f740 tx=0x7febf400f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0160c700 1 -- 172.21.15.16:0/15758961 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febf4000960 con 0x7febfc08c6d0 2019-12-05T06:19:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0260e700 1 --2- 172.21.15.16:0/15758961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febfc08c6d0 0x7febfc08cac0 secure :-1 s=THROTTLE_DONE pgs=538 cs=0 l=1 rx=0x7febf400f740 tx=0x7febf400f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0160c700 1 -- 172.21.15.16:0/15758961 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febf4000c90 con 0x7febfc08c6d0 2019-12-05T06:19:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.059+0000 7fec0160c700 1 -- 172.21.15.16:0/15758961 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febf40013c0 con 0x7febfc08c6d0 2019-12-05T06:19:33.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.060+0000 7fec04071700 1 -- 172.21.15.16:0/15758961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febfc08c6d0 msgr2=0x7febfc08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.060+0000 7fec04071700 1 --2- 172.21.15.16:0/15758961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febfc08c6d0 0x7febfc08cac0 secure :-1 s=READY pgs=538 cs=0 l=1 rx=0x7febf400f740 tx=0x7febf400f5f0).stop 2019-12-05T06:19:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.061+0000 7fec04071700 1 -- 172.21.15.16:0/15758961 shutdown_connections 2019-12-05T06:19:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.061+0000 7fec04071700 1 --2- 172.21.15.16:0/15758961 >> v2:172.21.15.76:3300/0 conn(0x7febfc08ef20 0x7febfc08f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:33.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.061+0000 7fec04071700 1 --2- 172.21.15.16:0/15758961 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc08fef0 0x7febfc08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:33.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.061+0000 7fec04071700 1 --2- 172.21.15.16:0/15758961 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febfc08c6d0 0x7febfc08cac0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.061+0000 7fec04071700 1 -- 172.21.15.16:0/15758961 shutdown_connections 2019-12-05T06:19:33.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.061+0000 7fec04071700 1 -- 172.21.15.16:0/15758961 wait complete. 2019-12-05T06:19:33.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.061+0000 7fec04071700 1 -- 172.21.15.16:0/15758961 >> 172.21.15.16:0/15758961 conn(0x7febfc061940 msgr2=0x7febfc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:33.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.066+0000 7fec04071700 1 Processor -- start 2019-12-05T06:19:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.066+0000 7fec04071700 1 -- start start 2019-12-05T06:19:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.067+0000 7fec04071700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7febfc09bbe0 0x7febfc1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.067+0000 7fec04071700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.067+0000 7fec04071700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febfc084b90 con 0x7febfc096e90 2019-12-05T06:19:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.067+0000 7fec04071700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febfc09a450 con 0x7febfc09bbe0 2019-12-05T06:19:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.067+0000 7fec04071700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febfc099a20 con 0x7febfc084f30 2019-12-05T06:19:33.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.068+0000 7fec01e0d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7febfc09bbe0 0x7febfc1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.068+0000 7fec02e0f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.068+0000 7fec01e0d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7febfc09bbe0 0x7febfc1539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:36988/0 (socket says 172.21.15.16:36988) 2019-12-05T06:19:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.068+0000 7fec0260e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7febfc096e90 0x7febfc09b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57366/0 (socket says 172.21.15.16:57366) 2019-12-05T06:19:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.068+0000 7fec01e0d700 1 -- 172.21.15.16:0/2256837049 learned_addr learned my addr 172.21.15.16:0/2256837049 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:33.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.068+0000 7fec02e0f700 1 -- 172.21.15.16:0/2256837049 >> v2:172.21.15.76:3300/0 conn(0x7febfc09bbe0 msgr2=0x7febfc1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.068+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> v2:172.21.15.76:3300/0 conn(0x7febfc09bbe0 0x7febfc1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.068+0000 7fec02e0f700 1 -- 172.21.15.16:0/2256837049 >> v1:172.21.15.76:6789/0 conn(0x7febfc096e90 legacy=0x7febfc09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7fec02e0f700 1 -- 172.21.15.16:0/2256837049 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febfc09aae0 con 0x7febfc084f30 2019-12-05T06:19:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7fec01e0d700 1 --2- 172.21.15.16:0/2256837049 >> v2:172.21.15.76:3300/0 conn(0x7febfc09bbe0 0x7febfc1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.068+0000 7febf2ffd700 1 -- 172.21.15.16:0/2256837049 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 138740575 0 0) 0x7febf4007a60 con 0x7febfc096e90 2019-12-05T06:19:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=READY pgs=138 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7febf2ffd700 1 -- 172.21.15.16:0/2256837049 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febec009d10 con 0x7febfc084f30 2019-12-05T06:19:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7fec04071700 1 -- 172.21.15.16:0/2256837049 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febfc09aae0 con 0x7febfc084f30 2019-12-05T06:19:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7febf2ffd700 1 -- 172.21.15.16:0/2256837049 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febec009930 con 0x7febfc084f30 2019-12-05T06:19:33.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7febf2ffd700 1 -- 172.21.15.16:0/2256837049 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febec009b20 con 0x7febfc084f30 2019-12-05T06:19:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.069+0000 7fec04071700 1 -- 172.21.15.16:0/2256837049 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febfc09aae0 con 0x7febfc084f30 2019-12-05T06:19:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.070+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.071+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.071+0000 7febf2ffd700 1 -- 172.21.15.16:0/2256837049 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7febec009b20 con 0x7febfc084f30 2019-12-05T06:19:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.071+0000 7febf2ffd700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7febdc023630 0x7febdc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:33.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.072+0000 7febf2ffd700 1 -- 172.21.15.16:0/2256837049 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 6633+0+0 (secure 0 0 0) 0x7febec0460c0 con 0x7febfc084f30 2019-12-05T06:19:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.072+0000 7fec0260e700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7febdc023630 0x7febdc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:33.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.072+0000 7fec0260e700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7febdc023630 0x7febdc025ac0 crc :-1 s=READY pgs=242 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.087+0000 7fec04071700 1 -- 172.21.15.16:0/2256837049 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febfc09aae0 con 0x7febfc084f30 2019-12-05T06:19:33.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.096+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.096+0000 7febf2ffd700 1 -- 172.21.15.16:0/2256837049 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7febec00a190 con 0x7febfc084f30 2019-12-05T06:19:33.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.347+0000 7fec04071700 1 -- 172.21.15.16:0/2256837049 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febfc09aae0 con 0x7febfc084f30 2019-12-05T06:19:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.348+0000 7fec02e0f700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=THROTTLE_DONE pgs=138 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.348+0000 7febf2ffd700 1 -- 172.21.15.16:0/2256837049 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+1771 (secure 0 0 0) 0x7febec009b20 con 0x7febfc084f30 2019-12-05T06:19:33.339 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 68 2019-12-05T06:19:33.339 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:33.340 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:33.089334+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:33.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 -- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7febdc023630 msgr2=0x7febdc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7febdc023630 0x7febdc025ac0 crc :-1 s=READY pgs=242 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 -- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 msgr2=0x7febfc155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 secure :-1 s=READY pgs=138 cs=0 l=1 rx=0x7febec010710 tx=0x7febec010ab0).stop 2019-12-05T06:19:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 -- 172.21.15.16:0/2256837049 shutdown_connections 2019-12-05T06:19:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7febdc023630 0x7febdc025ac0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 --2- 172.21.15.16:0/2256837049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febfc084f30 0x7febfc155f30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 --2- 172.21.15.16:0/2256837049 >> v2:172.21.15.76:3300/0 conn(0x7febfc09bbe0 0x7febfc1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 -- 172.21.15.16:0/2256837049 shutdown_connections 2019-12-05T06:19:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 -- 172.21.15.16:0/2256837049 wait complete. 2019-12-05T06:19:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.351+0000 7fec04071700 1 -- 172.21.15.16:0/2256837049 >> 172.21.15.16:0/2256837049 conn(0x7febfc0714f0 msgr2=0x7febfc09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:33.355 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:19:33.355 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:19:33.355 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.495+0000 7fe6b9798700 1 Processor -- start 2019-12-05T06:19:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.495+0000 7fe6b9798700 1 -- start start 2019-12-05T06:19:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b9798700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6b408fef0 0x7fe6b408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:33.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b9798700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b40958e0 con 0x7fe6b408c6d0 2019-12-05T06:19:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b9798700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b40629e0 con 0x7fe6b408d2c0 2019-12-05T06:19:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b9798700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b408df90 con 0x7fe6b408fef0 2019-12-05T06:19:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6b408fef0 0x7fe6b408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b37fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe6b408c6d0 0x7fe6b408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36086/0 (socket says 172.21.15.16:36086) 2019-12-05T06:19:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b37fe700 1 -- 172.21.15.16:0/2065641642 learned_addr learned my addr 172.21.15.16:0/2065641642 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b3fff700 1 -- 172.21.15.16:0/2065641642 >> v1:172.21.15.76:6789/0 conn(0x7fe6b408d2c0 legacy=0x7fe6b408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:33.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b3fff700 1 -- 172.21.15.16:0/2065641642 >> v1:172.21.15.16:6789/0 conn(0x7fe6b408c6d0 legacy=0x7fe6b408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b27fc700 1 -- 172.21.15.16:0/2065641642 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3051403678 0 0) 0x7fe6ac001150 con 0x7fe6b408c6d0 2019-12-05T06:19:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b3fff700 1 -- 172.21.15.16:0/2065641642 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b408e620 con 0x7fe6b408fef0 2019-12-05T06:19:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b3fff700 1 --2- 172.21.15.16:0/2065641642 >> v2:172.21.15.16:3300/0 conn(0x7fe6b408fef0 0x7fe6b408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe6a400b2d0 tx=0x7fe6a400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.496+0000 7fe6b3fff700 1 --2- 172.21.15.16:0/2065641642 >> v2:172.21.15.16:3300/0 conn(0x7fe6b408fef0 0x7fe6b408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe6a400b2d0 tx=0x7fe6a400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.497+0000 7fe6b3fff700 1 --2- 172.21.15.16:0/2065641642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b408fef0 0x7fe6b408f720 secure :-1 s=READY pgs=540 cs=0 l=1 rx=0x7fe6a400b2d0 tx=0x7fe6a400bae0).ready entity=mon.0 client_cookie=dc0d41326435224e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.497+0000 7fe6b3fff700 1 --2- 172.21.15.16:0/2065641642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b408fef0 0x7fe6b408f720 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7fe6a400b2d0 tx=0x7fe6a400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.497+0000 7fe6b3fff700 1 --2- 172.21.15.16:0/2065641642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b408fef0 0x7fe6b408f720 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7fe6a400b2d0 tx=0x7fe6a400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.497+0000 7fe6b27fc700 1 -- 172.21.15.16:0/2065641642 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a4009f20 con 0x7fe6b408fef0 2019-12-05T06:19:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.497+0000 7fe6b3fff700 1 --2- 172.21.15.16:0/2065641642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b408fef0 0x7fe6b408f720 secure :-1 s=THROTTLE_DONE pgs=540 cs=0 l=1 rx=0x7fe6a400b2d0 tx=0x7fe6a400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.497+0000 7fe6b27fc700 1 -- 172.21.15.16:0/2065641642 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6a4000df0 con 0x7fe6b408fef0 2019-12-05T06:19:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.497+0000 7fe6b27fc700 1 -- 172.21.15.16:0/2065641642 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6a400a500 con 0x7fe6b408fef0 2019-12-05T06:19:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.497+0000 7fe6b9798700 1 -- 172.21.15.16:0/2065641642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b408fef0 msgr2=0x7fe6b408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.497+0000 7fe6b9798700 1 --2- 172.21.15.16:0/2065641642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b408fef0 0x7fe6b408f720 secure :-1 s=READY pgs=540 cs=0 l=1 rx=0x7fe6a400b2d0 tx=0x7fe6a400bae0).stop 2019-12-05T06:19:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.498+0000 7fe6b9798700 1 -- 172.21.15.16:0/2065641642 shutdown_connections 2019-12-05T06:19:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.498+0000 7fe6b9798700 1 --2- 172.21.15.16:0/2065641642 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b408fef0 0x7fe6b408f720 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.498+0000 7fe6b9798700 1 -- 172.21.15.16:0/2065641642 shutdown_connections 2019-12-05T06:19:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.498+0000 7fe6b9798700 1 -- 172.21.15.16:0/2065641642 wait complete. 2019-12-05T06:19:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.498+0000 7fe6b9798700 1 -- 172.21.15.16:0/2065641642 >> 172.21.15.16:0/2065641642 conn(0x7fe6b4061940 msgr2=0x7fe6b4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.500+0000 7fe6b9798700 1 Processor -- start 2019-12-05T06:19:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.500+0000 7fe6b9798700 1 -- start start 2019-12-05T06:19:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.501+0000 7fe6b9798700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6b414ca10 0x7fe6b40869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.501+0000 7fe6b9798700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe6b409ce90 0x7fe6b419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.501+0000 7fe6b9798700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6b409e030 con 0x7fe6b40879b0 2019-12-05T06:19:33.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.501+0000 7fe6b9798700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6b4085a50 con 0x7fe6b414ca10 2019-12-05T06:19:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.501+0000 7fe6b9798700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6b4085020 con 0x7fe6b409ce90 2019-12-05T06:19:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.501+0000 7fe6b37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6b414ca10 0x7fe6b40869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.501+0000 7fe6b37fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe6b414ca10 0x7fe6b40869e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50924/0 (socket says 172.21.15.16:50924) 2019-12-05T06:19:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.501+0000 7fe6b37fe700 1 -- 172.21.15.16:0/2477444995 learned_addr learned my addr 172.21.15.16:0/2477444995 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b3fff700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe6b409ce90 0x7fe6b419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b37fe700 1 -- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe6b409ce90 msgr2=0x7fe6b419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe6b409ce90 0x7fe6b419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b37fe700 1 -- 172.21.15.16:0/2477444995 >> v1:172.21.15.16:6789/0 conn(0x7fe6b40879b0 legacy=0x7fe6b4086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6abfff700 1 -- 172.21.15.16:0/2477444995 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3455196235 0 0) 0x7fe69c0010c0 con 0x7fe6b40879b0 2019-12-05T06:19:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b37fe700 1 -- 172.21.15.16:0/2477444995 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b40860e0 con 0x7fe6b414ca10 2019-12-05T06:19:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b3fff700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe6b409ce90 0x7fe6b419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:19:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> v2:172.21.15.16:3300/0 conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> v2:172.21.15.16:3300/0 conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=READY pgs=542 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6abfff700 1 -- 172.21.15.16:0/2477444995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6ac004430 con 0x7fe6b414ca10 2019-12-05T06:19:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6abfff700 1 -- 172.21.15.16:0/2477444995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6ac000ed0 con 0x7fe6b414ca10 2019-12-05T06:19:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6abfff700 1 -- 172.21.15.16:0/2477444995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6ac004a10 con 0x7fe6b414ca10 2019-12-05T06:19:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b9798700 1 -- 172.21.15.16:0/2477444995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6b40860e0 con 0x7fe6b414ca10 2019-12-05T06:19:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.502+0000 7fe6b9798700 1 -- 172.21.15.16:0/2477444995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6b40860e0 con 0x7fe6b414ca10 2019-12-05T06:19:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.503+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.504+0000 7fe6abfff700 1 -- 172.21.15.16:0/2477444995 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe6ac004240 con 0x7fe6b414ca10 2019-12-05T06:19:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.504+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.504+0000 7fe6abfff700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe698023630 0x7fe698025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.504+0000 7fe6abfff700 1 -- 172.21.15.16:0/2477444995 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fe6ac04e5d0 con 0x7fe6b414ca10 2019-12-05T06:19:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.504+0000 7fe6b2ffd700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe698023630 0x7fe698025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.504+0000 7fe6b2ffd700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe698023630 0x7fe698025ac0 crc :-1 s=READY pgs=243 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.510+0000 7fe6b9798700 1 -- 172.21.15.16:0/2477444995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6b40860e0 con 0x7fe6b414ca10 2019-12-05T06:19:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.514+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.514+0000 7fe6abfff700 1 -- 172.21.15.16:0/2477444995 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe6ac032fd0 con 0x7fe6b414ca10 2019-12-05T06:19:33.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.711+0000 7fe6b9798700 1 -- 172.21.15.16:0/2477444995 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe6b40860e0 con 0x7fe6b414ca10 2019-12-05T06:19:33.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.712+0000 7fe6b37fe700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=THROTTLE_DONE pgs=542 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:33.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.712+0000 7fe6abfff700 1 -- 172.21.15.16:0/2477444995 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fe6ac032fd0 con 0x7fe6b414ca10 2019-12-05T06:19:33.703 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 68 2019-12-05T06:19:33.703 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:33.703 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":68,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:33.089334+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.714+0000 7fe6b9798700 1 -- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe698023630 msgr2=0x7fe698025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.714+0000 7fe6b9798700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe698023630 0x7fe698025ac0 crc :-1 s=READY pgs=243 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.714+0000 7fe6b9798700 1 -- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 msgr2=0x7fe6b40869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:33.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.714+0000 7fe6b9798700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 0x7fe6b40869e0 secure :-1 s=READY pgs=542 cs=0 l=1 rx=0x7fe6ac001ca0 tx=0x7fe6ac009750).stop 2019-12-05T06:19:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.715+0000 7fe6b9798700 1 -- 172.21.15.16:0/2477444995 shutdown_connections 2019-12-05T06:19:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.715+0000 7fe6b9798700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe6b409ce90 0x7fe6b419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.715+0000 7fe6b9798700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe698023630 0x7fe698025ac0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.715+0000 7fe6b9798700 1 --2- 172.21.15.16:0/2477444995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe6b414ca10 0x7fe6b40869e0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.715+0000 7fe6b9798700 1 -- 172.21.15.16:0/2477444995 shutdown_connections 2019-12-05T06:19:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.715+0000 7fe6b9798700 1 -- 172.21.15.16:0/2477444995 wait complete. 2019-12-05T06:19:33.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:33.715+0000 7fe6b9798700 1 -- 172.21.15.16:0/2477444995 >> 172.21.15.16:0/2477444995 conn(0x7fe6b4098480 msgr2=0x7fe6b4071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:33.716 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 68, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 68, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5490": { "addr": "172.21.15.16:6833/4122423358", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 4122423358, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 4122423358, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5490, "incarnation": 68, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 57, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:19:33.089334+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5490 } } } ], "standbys": [] } 2019-12-05T06:19:35.718 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.965+0000 7efcce071700 1 Processor -- start 2019-12-05T06:19:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.965+0000 7efcce071700 1 -- start start 2019-12-05T06:19:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.966+0000 7efcce071700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efcc808d2c0 0x7efcc808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.967+0000 7efcce071700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efcc808fef0 0x7efcc808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.967+0000 7efcce071700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc80958e0 con 0x7efcc808c6d0 2019-12-05T06:19:35.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.967+0000 7efcce071700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc808df90 con 0x7efcc808fef0 2019-12-05T06:19:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.967+0000 7efcce071700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80a3f70 con 0x7efcc808d2c0 2019-12-05T06:19:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.967+0000 7efccce0f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efcc808fef0 0x7efcc808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.967+0000 7efccce0f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efcc808fef0 0x7efcc808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50934/0 (socket says 172.21.15.16:50934) 2019-12-05T06:19:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.967+0000 7efccce0f700 1 -- 172.21.15.16:0/3251339229 learned_addr learned my addr 172.21.15.16:0/3251339229 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.967+0000 7efcc77fe700 1 --2- 172.21.15.16:0/3251339229 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efcc808d2c0 0x7efcc808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efccce0f700 1 -- 172.21.15.16:0/3251339229 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efcc808d2c0 msgr2=0x7efcc808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:35.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efccce0f700 1 --2- 172.21.15.16:0/3251339229 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efcc808d2c0 0x7efcc808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efcc6ffd700 1 -- 172.21.15.16:0/3251339229 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1540753621 0 0) 0x7efcb8001400 con 0x7efcc808c6d0 2019-12-05T06:19:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efccce0f700 1 -- 172.21.15.16:0/3251339229 >> v1:172.21.15.76:6789/0 conn(0x7efcc808c6d0 legacy=0x7efcc808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efccce0f700 1 -- 172.21.15.16:0/3251339229 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc808e620 con 0x7efcc808fef0 2019-12-05T06:19:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efccce0f700 1 --2- 172.21.15.16:0/3251339229 >> v2:172.21.15.16:3300/0 conn(0x7efcc808fef0 0x7efcc808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efcc000f6b0 tx=0x7efcc000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efccce0f700 1 --2- 172.21.15.16:0/3251339229 >> v2:172.21.15.16:3300/0 conn(0x7efcc808fef0 0x7efcc808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efcc000f6b0 tx=0x7efcc000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efccce0f700 1 --2- 172.21.15.16:0/3251339229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc808fef0 0x7efcc808f460 secure :-1 s=READY pgs=543 cs=0 l=1 rx=0x7efcc000f6b0 tx=0x7efcc000fa50).ready entity=mon.0 client_cookie=267ab8966bef09c6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:35.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efccce0f700 1 --2- 172.21.15.16:0/3251339229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc808fef0 0x7efcc808f460 secure :-1 s=THROTTLE_DONE pgs=543 cs=0 l=1 rx=0x7efcc000f6b0 tx=0x7efcc000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efccce0f700 1 --2- 172.21.15.16:0/3251339229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc808fef0 0x7efcc808f460 secure :-1 s=THROTTLE_DONE pgs=543 cs=0 l=1 rx=0x7efcc000f6b0 tx=0x7efcc000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efcc6ffd700 1 -- 172.21.15.16:0/3251339229 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcc0000a70 con 0x7efcc808fef0 2019-12-05T06:19:35.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.968+0000 7efccce0f700 1 --2- 172.21.15.16:0/3251339229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc808fef0 0x7efcc808f460 secure :-1 s=THROTTLE_DONE pgs=543 cs=0 l=1 rx=0x7efcc000f6b0 tx=0x7efcc000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.969+0000 7efcc6ffd700 1 -- 172.21.15.16:0/3251339229 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcc00010e0 con 0x7efcc808fef0 2019-12-05T06:19:35.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.969+0000 7efcc6ffd700 1 -- 172.21.15.16:0/3251339229 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcc00017b0 con 0x7efcc808fef0 2019-12-05T06:19:35.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.969+0000 7efcce071700 1 -- 172.21.15.16:0/3251339229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc808fef0 msgr2=0x7efcc808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:35.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.969+0000 7efcce071700 1 --2- 172.21.15.16:0/3251339229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc808fef0 0x7efcc808f460 secure :-1 s=READY pgs=543 cs=0 l=1 rx=0x7efcc000f6b0 tx=0x7efcc000fa50).stop 2019-12-05T06:19:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.970+0000 7efcce071700 1 -- 172.21.15.16:0/3251339229 shutdown_connections 2019-12-05T06:19:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.970+0000 7efcce071700 1 --2- 172.21.15.16:0/3251339229 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efcc808fef0 0x7efcc808f460 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.970+0000 7efcce071700 1 --2- 172.21.15.16:0/3251339229 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efcc808d2c0 0x7efcc808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.970+0000 7efcce071700 1 -- 172.21.15.16:0/3251339229 shutdown_connections 2019-12-05T06:19:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.970+0000 7efcce071700 1 -- 172.21.15.16:0/3251339229 wait complete. 2019-12-05T06:19:35.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.970+0000 7efcce071700 1 -- 172.21.15.16:0/3251339229 >> 172.21.15.16:0/3251339229 conn(0x7efcc8061940 msgr2=0x7efcc8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:35.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.976+0000 7efcce071700 1 Processor -- start 2019-12-05T06:19:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.976+0000 7efcce071700 1 -- start start 2019-12-05T06:19:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.978+0000 7efcce071700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7efcc8132d00 0x7efcc814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.978+0000 7efcce071700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc8098da0 con 0x7efcc814d400 2019-12-05T06:19:35.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.978+0000 7efcce071700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcc809f0a0 con 0x7efcc8132530 2019-12-05T06:19:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.978+0000 7efcce071700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcc809bb50 con 0x7efcc8132d00 2019-12-05T06:19:35.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.978+0000 7efcc7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efcc814d400 0x7efcc8131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36104/0 (socket says 172.21.15.16:36104) 2019-12-05T06:19:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.978+0000 7efcc7fff700 1 -- 172.21.15.16:0/2491647946 learned_addr learned my addr 172.21.15.16:0/2491647946 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.978+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> v2:172.21.15.76:3300/0 conn(0x7efcc8132d00 0x7efcc814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.979+0000 7efcc4ff9700 1 -- 172.21.15.16:0/2491647946 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338886665 0 0) 0x7efcb8001640 con 0x7efcc814d400 2019-12-05T06:19:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.979+0000 7efcc4ff9700 1 -- 172.21.15.16:0/2491647946 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcac002e80 con 0x7efcc814d400 2019-12-05T06:19:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.979+0000 7efcc77fe700 1 -- 172.21.15.16:0/2491647946 >> v1:172.21.15.76:6789/0 conn(0x7efcc8132530 legacy=0x7efcc8157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.979+0000 7efcc77fe700 1 -- 172.21.15.16:0/2491647946 >> v1:172.21.15.16:6789/0 conn(0x7efcc814d400 legacy=0x7efcc8131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:35.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.979+0000 7efcc77fe700 1 -- 172.21.15.16:0/2491647946 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcc809b0f0 con 0x7efcc8132d00 2019-12-05T06:19:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.979+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> v2:172.21.15.76:3300/0 conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.979+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> v2:172.21.15.76:3300/0 conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.979+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=READY pgs=406 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.980+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.980+0000 7efcc4ff9700 1 -- 172.21.15.16:0/2491647946 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcb0000940 con 0x7efcc8132d00 2019-12-05T06:19:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.980+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.980+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.980+0000 7efcc4ff9700 1 -- 172.21.15.16:0/2491647946 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcb0009c60 con 0x7efcc8132d00 2019-12-05T06:19:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.980+0000 7efcce071700 1 -- 172.21.15.16:0/2491647946 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc809b0f0 con 0x7efcc8132d00 2019-12-05T06:19:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.980+0000 7efcc4ff9700 1 -- 172.21.15.16:0/2491647946 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcb000a2f0 con 0x7efcc8132d00 2019-12-05T06:19:35.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.980+0000 7efcce071700 1 -- 172.21.15.16:0/2491647946 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc809b0f0 con 0x7efcc8132d00 2019-12-05T06:19:35.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.981+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.982+0000 7efcc4ff9700 1 -- 172.21.15.16:0/2491647946 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efcb00098e0 con 0x7efcc8132d00 2019-12-05T06:19:35.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.982+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.982+0000 7efcc4ff9700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcac023670 0x7efcac025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:35.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.982+0000 7efcc4ff9700 1 -- 172.21.15.16:0/2491647946 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7efcb0045e00 con 0x7efcc8132d00 2019-12-05T06:19:35.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.982+0000 7efcc7fff700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcac023670 0x7efcac025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:35.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.983+0000 7efcc7fff700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcac023670 0x7efcac025ae0 crc :-1 s=READY pgs=244 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:35.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:35.996+0000 7efcce071700 1 -- 172.21.15.16:0/2491647946 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc809b0f0 con 0x7efcc8132d00 2019-12-05T06:19:35.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.006+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:35.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.006+0000 7efcc4ff9700 1 -- 172.21.15.16:0/2491647946 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efcb0012dc0 con 0x7efcc8132d00 2019-12-05T06:19:36.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.245+0000 7efcce071700 1 -- 172.21.15.16:0/2491647946 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efcc809b0f0 con 0x7efcc8132d00 2019-12-05T06:19:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.247+0000 7efcc77fe700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=THROTTLE_DONE pgs=406 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:36.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.247+0000 7efcc4ff9700 1 -- 172.21.15.16:0/2491647946 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+1775 (secure 0 0 0) 0x7efcb0009680 con 0x7efcc8132d00 2019-12-05T06:19:36.239 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 69 2019-12-05T06:19:36.247 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:36.247 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":69,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":69,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:36.049468+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:reconnect","state_seq":22,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:36.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.250+0000 7efcce071700 1 -- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcac023670 msgr2=0x7efcac025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.250+0000 7efcce071700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcac023670 0x7efcac025ae0 crc :-1 s=READY pgs=244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.251+0000 7efcce071700 1 -- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 msgr2=0x7efcc814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.251+0000 7efcce071700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 0x7efcc814d7f0 secure :-1 s=READY pgs=406 cs=0 l=1 rx=0x7efcb0010390 tx=0x7efcb0010730).stop 2019-12-05T06:19:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.251+0000 7efcce071700 1 -- 172.21.15.16:0/2491647946 shutdown_connections 2019-12-05T06:19:36.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.251+0000 7efcce071700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efcac023670 0x7efcac025ae0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:36.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.251+0000 7efcce071700 1 --2- 172.21.15.16:0/2491647946 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efcc8132d00 0x7efcc814d7f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:36.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.251+0000 7efcce071700 1 -- 172.21.15.16:0/2491647946 shutdown_connections 2019-12-05T06:19:36.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.251+0000 7efcce071700 1 -- 172.21.15.16:0/2491647946 wait complete. 2019-12-05T06:19:36.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:36.251+0000 7efcce071700 1 -- 172.21.15.16:0/2491647946 >> 172.21.15.16:0/2491647946 conn(0x7efcc80714f0 msgr2=0x7efcc809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:36.253 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:19:38.253 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:38.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.557+0000 7fb7e4a83700 1 Processor -- start 2019-12-05T06:19:38.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.557+0000 7fb7e4a83700 1 -- start start 2019-12-05T06:19:38.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.559+0000 7fb7e4a83700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e008d2c0 0x7fb7e008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:38.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.559+0000 7fb7e4a83700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7e008fef0 0x7fb7e008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.559+0000 7fb7e4a83700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e00958e0 con 0x7fb7e008c6d0 2019-12-05T06:19:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.559+0000 7fb7e4a83700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7e008df90 con 0x7fb7e008fef0 2019-12-05T06:19:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.559+0000 7fb7e4a83700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e00a3f70 con 0x7fb7e008d2c0 2019-12-05T06:19:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.559+0000 7fb7df59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7e008fef0 0x7fb7e008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.560+0000 7fb7df59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7e008fef0 0x7fb7e008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50948/0 (socket says 172.21.15.16:50948) 2019-12-05T06:19:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.560+0000 7fb7ded9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb7e008c6d0 0x7fb7e008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36114/0 (socket says 172.21.15.16:36114) 2019-12-05T06:19:38.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.560+0000 7fb7df59e700 1 -- 172.21.15.16:0/492804941 learned_addr learned my addr 172.21.15.16:0/492804941 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.560+0000 7fb7de59c700 1 --2- 172.21.15.16:0/492804941 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e008d2c0 0x7fb7e008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.560+0000 7fb7df59e700 1 -- 172.21.15.16:0/492804941 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e008d2c0 msgr2=0x7fb7e008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.560+0000 7fb7df59e700 1 --2- 172.21.15.16:0/492804941 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e008d2c0 0x7fb7e008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.560+0000 7fb7df59e700 1 -- 172.21.15.16:0/492804941 >> v1:172.21.15.16:6789/0 conn(0x7fb7e008c6d0 legacy=0x7fb7e008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.560+0000 7fb7ddd9b700 1 -- 172.21.15.16:0/492804941 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2863848752 0 0) 0x7fb7d0001480 con 0x7fb7e008c6d0 2019-12-05T06:19:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.560+0000 7fb7df59e700 1 -- 172.21.15.16:0/492804941 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e008e620 con 0x7fb7e008fef0 2019-12-05T06:19:38.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.561+0000 7fb7df59e700 1 --2- 172.21.15.16:0/492804941 >> v2:172.21.15.16:3300/0 conn(0x7fb7e008fef0 0x7fb7e008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb7d800f6b0 tx=0x7fb7d800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.561+0000 7fb7df59e700 1 --2- 172.21.15.16:0/492804941 >> v2:172.21.15.16:3300/0 conn(0x7fb7e008fef0 0x7fb7e008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb7d800f6b0 tx=0x7fb7d800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.561+0000 7fb7df59e700 1 --2- 172.21.15.16:0/492804941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7e008fef0 0x7fb7e008f460 secure :-1 s=READY pgs=546 cs=0 l=1 rx=0x7fb7d800f6b0 tx=0x7fb7d800fa50).ready entity=mon.0 client_cookie=c010455a58346f83 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.561+0000 7fb7df59e700 1 --2- 172.21.15.16:0/492804941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7e008fef0 0x7fb7e008f460 secure :-1 s=THROTTLE_DONE pgs=546 cs=0 l=1 rx=0x7fb7d800f6b0 tx=0x7fb7d800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.561+0000 7fb7ddd9b700 1 -- 172.21.15.16:0/492804941 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7d8000a70 con 0x7fb7e008fef0 2019-12-05T06:19:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.561+0000 7fb7df59e700 1 --2- 172.21.15.16:0/492804941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7e008fef0 0x7fb7e008f460 secure :-1 s=THROTTLE_DONE pgs=546 cs=0 l=1 rx=0x7fb7d800f6b0 tx=0x7fb7d800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.561+0000 7fb7df59e700 1 --2- 172.21.15.16:0/492804941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7e008fef0 0x7fb7e008f460 secure :-1 s=THROTTLE_DONE pgs=546 cs=0 l=1 rx=0x7fb7d800f6b0 tx=0x7fb7d800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.561+0000 7fb7ddd9b700 1 -- 172.21.15.16:0/492804941 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d80010e0 con 0x7fb7e008fef0 2019-12-05T06:19:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.561+0000 7fb7ddd9b700 1 -- 172.21.15.16:0/492804941 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7d80017d0 con 0x7fb7e008fef0 2019-12-05T06:19:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.562+0000 7fb7e4a83700 1 -- 172.21.15.16:0/492804941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7e008fef0 msgr2=0x7fb7e008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.562+0000 7fb7e4a83700 1 --2- 172.21.15.16:0/492804941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7e008fef0 0x7fb7e008f460 secure :-1 s=READY pgs=546 cs=0 l=1 rx=0x7fb7d800f6b0 tx=0x7fb7d800fa50).stop 2019-12-05T06:19:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.564+0000 7fb7e4a83700 1 -- 172.21.15.16:0/492804941 shutdown_connections 2019-12-05T06:19:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.564+0000 7fb7e4a83700 1 --2- 172.21.15.16:0/492804941 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7e008fef0 0x7fb7e008f460 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.564+0000 7fb7e4a83700 1 --2- 172.21.15.16:0/492804941 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e008d2c0 0x7fb7e008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:38.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.564+0000 7fb7e4a83700 1 -- 172.21.15.16:0/492804941 shutdown_connections 2019-12-05T06:19:38.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.564+0000 7fb7e4a83700 1 -- 172.21.15.16:0/492804941 wait complete. 2019-12-05T06:19:38.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.564+0000 7fb7e4a83700 1 -- 172.21.15.16:0/492804941 >> 172.21.15.16:0/492804941 conn(0x7fb7e0061940 msgr2=0x7fb7e0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:38.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.569+0000 7fb7e4a83700 1 Processor -- start 2019-12-05T06:19:38.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.570+0000 7fb7e4a83700 1 -- start start 2019-12-05T06:19:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.571+0000 7fb7e4a83700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.571+0000 7fb7e4a83700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e0084b90 con 0x7fb7e0096e90 2019-12-05T06:19:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.571+0000 7fb7e4a83700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7e0098790 con 0x7fb7e009bbe0 2019-12-05T06:19:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.571+0000 7fb7e4a83700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7e009a450 con 0x7fb7e0084f30 2019-12-05T06:19:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.572+0000 7fb7ded9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb7e0096e90 0x7fb7e009b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36120/0 (socket says 172.21.15.16:36120) 2019-12-05T06:19:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.572+0000 7fb7ded9d700 1 -- 172.21.15.16:0/860582931 learned_addr learned my addr 172.21.15.16:0/860582931 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:38.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.572+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:38.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.572+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454245564 0 0) 0x7fb7d00014f0 con 0x7fb7e0096e90 2019-12-05T06:19:38.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.572+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c4002e80 con 0x7fb7e0096e90 2019-12-05T06:19:38.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.572+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590756694 0 0) 0x7fb7c8000c90 con 0x7fb7e009bbe0 2019-12-05T06:19:38.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.572+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb7c4004170 con 0x7fb7e009bbe0 2019-12-05T06:19:38.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.572+0000 7fb7df59e700 1 -- 172.21.15.16:0/860582931 >> v1:172.21.15.76:6789/0 conn(0x7fb7e009bbe0 legacy=0x7fb7e01539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:38.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.572+0000 7fb7df59e700 1 -- 172.21.15.16:0/860582931 >> v1:172.21.15.16:6789/0 conn(0x7fb7e0096e90 legacy=0x7fb7e009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.572+0000 7fb7df59e700 1 -- 172.21.15.16:0/860582931 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e009aae0 con 0x7fb7e0084f30 2019-12-05T06:19:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:38.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7d8001290 con 0x7fb7e0084f30 2019-12-05T06:19:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7d8000eb0 con 0x7fb7e0084f30 2019-12-05T06:19:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7e4a83700 1 -- 172.21.15.16:0/860582931 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7e009aae0 con 0x7fb7e0084f30 2019-12-05T06:19:38.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7d8001910 con 0x7fb7e0084f30 2019-12-05T06:19:38.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.573+0000 7fb7e4a83700 1 -- 172.21.15.16:0/860582931 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7e009aae0 con 0x7fb7e0084f30 2019-12-05T06:19:38.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.574+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.575+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb7d80010a0 con 0x7fb7e0084f30 2019-12-05T06:19:38.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.575+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.576+0000 7fb7d77fe700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7c4023650 0x7fb7c4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:38.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.576+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fb7d8045d90 con 0x7fb7e0084f30 2019-12-05T06:19:38.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.576+0000 7fb7ded9d700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7c4023650 0x7fb7c4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:38.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.576+0000 7fb7ded9d700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7c4023650 0x7fb7c4025ae0 crc :-1 s=READY pgs=245 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:38.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.589+0000 7fb7e4a83700 1 -- 172.21.15.16:0/860582931 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7e009aae0 con 0x7fb7e0084f30 2019-12-05T06:19:38.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.598+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.598+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb7d8007610 con 0x7fb7e0084f30 2019-12-05T06:19:38.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.863+0000 7fb7e4a83700 1 -- 172.21.15.16:0/860582931 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7e009aae0 con 0x7fb7e0084f30 2019-12-05T06:19:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.864+0000 7fb7df59e700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=THROTTLE_DONE pgs=139 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.864+0000 7fb7d77fe700 1 -- 172.21.15.16:0/860582931 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fb7d8001070 con 0x7fb7e0084f30 2019-12-05T06:19:38.855 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 71 2019-12-05T06:19:38.856 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:38.856 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:38.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.867+0000 7fb7e4a83700 1 -- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7c4023650 msgr2=0x7fb7c4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:38.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.867+0000 7fb7e4a83700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7c4023650 0x7fb7c4025ae0 crc :-1 s=READY pgs=245 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:38.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.867+0000 7fb7e4a83700 1 -- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 msgr2=0x7fb7e0157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:38.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.867+0000 7fb7e4a83700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 secure :-1 s=READY pgs=139 cs=0 l=1 rx=0x7fb7d8010380 tx=0x7fb7d8010720).stop 2019-12-05T06:19:38.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.867+0000 7fb7e4a83700 1 -- 172.21.15.16:0/860582931 shutdown_connections 2019-12-05T06:19:38.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.867+0000 7fb7e4a83700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7c4023650 0x7fb7c4025ae0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:38.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.867+0000 7fb7e4a83700 1 --2- 172.21.15.16:0/860582931 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb7e0084f30 0x7fb7e0157200 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:38.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.867+0000 7fb7e4a83700 1 -- 172.21.15.16:0/860582931 shutdown_connections 2019-12-05T06:19:38.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.867+0000 7fb7e4a83700 1 -- 172.21.15.16:0/860582931 wait complete. 2019-12-05T06:19:38.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:38.867+0000 7fb7e4a83700 1 -- 172.21.15.16:0/860582931 >> 172.21.15.16:0/860582931 conn(0x7fb7e00714f0 msgr2=0x7fb7e009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:38.868 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5382 2019-12-05T06:19:38.869 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2019-12-05T06:19:43.869 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:19:43.869 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:19:43.870 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:19:43.874 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:19:43.875 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:44.059 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:19:44.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.136+0000 7f8270a16700 1 Processor -- start 2019-12-05T06:19:44.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.136+0000 7f8270a16700 1 -- start start 2019-12-05T06:19:44.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.138+0000 7f8270a16700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f826c08c6d0 0x7f826c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:44.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.138+0000 7f8270a16700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f826c08fef0 0x7f826c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:44.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.138+0000 7f8270a16700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c0958e0 con 0x7f826c08ef20 2019-12-05T06:19:44.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.138+0000 7f8270a16700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c0a3f70 con 0x7f826c08fef0 2019-12-05T06:19:44.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.138+0000 7f8270a16700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f826c08d830 con 0x7f826c08c6d0 2019-12-05T06:19:44.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.138+0000 7f826a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f826c08fef0 0x7f826c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:44.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.138+0000 7f826a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f826c08fef0 0x7f826c08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50962/0 (socket says 172.21.15.16:50962) 2019-12-05T06:19:44.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.138+0000 7f826b59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f826c08ef20 0x7f826c08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57432/0 (socket says 172.21.15.16:57432) 2019-12-05T06:19:44.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.138+0000 7f826ad9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f826c08c6d0 0x7f826c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:44.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.138+0000 7f826a59c700 1 -- 172.21.15.16:0/1047514342 learned_addr learned my addr 172.21.15.16:0/1047514342 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:44.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.139+0000 7f8269d9b700 1 -- 172.21.15.16:0/1047514342 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 677719031 0 0) 0x7f825c0009b0 con 0x7f826c08ef20 2019-12-05T06:19:44.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.139+0000 7f826a59c700 1 -- 172.21.15.16:0/1047514342 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f826c08c6d0 msgr2=0x7f826c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:44.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.139+0000 7f826a59c700 1 --2- 172.21.15.16:0/1047514342 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f826c08c6d0 0x7f826c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:44.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.139+0000 7f826a59c700 1 -- 172.21.15.16:0/1047514342 >> v1:172.21.15.76:6789/0 conn(0x7f826c08ef20 legacy=0x7f826c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.139+0000 7f826a59c700 1 -- 172.21.15.16:0/1047514342 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c08d540 con 0x7f826c08fef0 2019-12-05T06:19:44.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.139+0000 7f826ad9d700 1 --2- 172.21.15.16:0/1047514342 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f826c08c6d0 0x7f826c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:44.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.139+0000 7f826a59c700 1 --2- 172.21.15.16:0/1047514342 >> v2:172.21.15.16:3300/0 conn(0x7f826c08fef0 0x7f826c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f825400f740 tx=0x7f825400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.140+0000 7f826a59c700 1 --2- 172.21.15.16:0/1047514342 >> v2:172.21.15.16:3300/0 conn(0x7f826c08fef0 0x7f826c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f825400f740 tx=0x7f825400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.140+0000 7f826a59c700 1 --2- 172.21.15.16:0/1047514342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c08fef0 0x7f826c08d000 secure :-1 s=READY pgs=552 cs=0 l=1 rx=0x7f825400f740 tx=0x7f825400fae0).ready entity=mon.0 client_cookie=b6920ab981b816e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:44.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.140+0000 7f826a59c700 1 --2- 172.21.15.16:0/1047514342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c08fef0 0x7f826c08d000 secure :-1 s=THROTTLE_DONE pgs=552 cs=0 l=1 rx=0x7f825400f740 tx=0x7f825400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.140+0000 7f8269d9b700 1 -- 172.21.15.16:0/1047514342 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8254000960 con 0x7f826c08fef0 2019-12-05T06:19:44.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.140+0000 7f826a59c700 1 --2- 172.21.15.16:0/1047514342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c08fef0 0x7f826c08d000 secure :-1 s=THROTTLE_DONE pgs=552 cs=0 l=1 rx=0x7f825400f740 tx=0x7f825400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.140+0000 7f826a59c700 1 --2- 172.21.15.16:0/1047514342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c08fef0 0x7f826c08d000 secure :-1 s=THROTTLE_DONE pgs=552 cs=0 l=1 rx=0x7f825400f740 tx=0x7f825400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.140+0000 7f8269d9b700 1 -- 172.21.15.16:0/1047514342 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8254000c90 con 0x7f826c08fef0 2019-12-05T06:19:44.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.140+0000 7f8269d9b700 1 -- 172.21.15.16:0/1047514342 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f82540013a0 con 0x7f826c08fef0 2019-12-05T06:19:44.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.140+0000 7f8270a16700 1 -- 172.21.15.16:0/1047514342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c08fef0 msgr2=0x7f826c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.140+0000 7f8270a16700 1 --2- 172.21.15.16:0/1047514342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c08fef0 0x7f826c08d000 secure :-1 s=READY pgs=552 cs=0 l=1 rx=0x7f825400f740 tx=0x7f825400fae0).stop 2019-12-05T06:19:44.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.141+0000 7f8270a16700 1 -- 172.21.15.16:0/1047514342 shutdown_connections 2019-12-05T06:19:44.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.141+0000 7f8270a16700 1 --2- 172.21.15.16:0/1047514342 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c08fef0 0x7f826c08d000 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.141+0000 7f8270a16700 1 --2- 172.21.15.16:0/1047514342 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f826c08c6d0 0x7f826c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:44.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.141+0000 7f8270a16700 1 -- 172.21.15.16:0/1047514342 shutdown_connections 2019-12-05T06:19:44.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.141+0000 7f8270a16700 1 -- 172.21.15.16:0/1047514342 wait complete. 2019-12-05T06:19:44.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.141+0000 7f8270a16700 1 -- 172.21.15.16:0/1047514342 >> 172.21.15.16:0/1047514342 conn(0x7f826c061940 msgr2=0x7f826c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:44.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.146+0000 7f8270a16700 1 Processor -- start 2019-12-05T06:19:44.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.146+0000 7f8270a16700 1 -- start start 2019-12-05T06:19:44.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.147+0000 7f8270a16700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f826c096e90 0x7f826c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:44.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.147+0000 7f8270a16700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f826c09b920 0x7f826c097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:44.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.147+0000 7f8270a16700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f826c084b90 con 0x7f826c084f30 2019-12-05T06:19:44.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.147+0000 7f8270a16700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c09a450 con 0x7f826c09b920 2019-12-05T06:19:44.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.147+0000 7f8270a16700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f826c099a20 con 0x7f826c096e90 2019-12-05T06:19:44.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.147+0000 7f826b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f826c09b920 0x7f826c097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:44.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.147+0000 7f826a59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f826c084f30 0x7f826c097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36134/0 (socket says 172.21.15.16:36134) 2019-12-05T06:19:44.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.147+0000 7f826a59c700 1 -- 172.21.15.16:0/2341573931 learned_addr learned my addr 172.21.15.16:0/2341573931 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:44.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.147+0000 7f826ad9d700 1 --2- 172.21.15.16:0/2341573931 >> v2:172.21.15.76:3300/0 conn(0x7f826c096e90 0x7f826c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:44.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.148+0000 7f826b59e700 1 -- 172.21.15.16:0/2341573931 >> v2:172.21.15.76:3300/0 conn(0x7f826c096e90 msgr2=0x7f826c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.148+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> v2:172.21.15.76:3300/0 conn(0x7f826c096e90 0x7f826c09b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.148+0000 7f826b59e700 1 -- 172.21.15.16:0/2341573931 >> v1:172.21.15.16:6789/0 conn(0x7f826c084f30 legacy=0x7f826c097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.148+0000 7f82637fe700 1 -- 172.21.15.16:0/2341573931 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2240796727 0 0) 0x7f8254007da0 con 0x7f826c084f30 2019-12-05T06:19:44.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.148+0000 7f826b59e700 1 -- 172.21.15.16:0/2341573931 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f826c09aae0 con 0x7f826c09b920 2019-12-05T06:19:44.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.148+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> v2:172.21.15.16:3300/0 conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.148+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> v2:172.21.15.16:3300/0 conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.148+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=READY pgs=554 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:44.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.149+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=THROTTLE_DONE pgs=554 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.149+0000 7f82637fe700 1 -- 172.21.15.16:0/2341573931 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f825c000940 con 0x7f826c09b920 2019-12-05T06:19:44.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.149+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=THROTTLE_DONE pgs=554 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.149+0000 7f82637fe700 1 -- 172.21.15.16:0/2341573931 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c0041c0 con 0x7f826c09b920 2019-12-05T06:19:44.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.149+0000 7f8270a16700 1 -- 172.21.15.16:0/2341573931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f826c09aae0 con 0x7f826c09b920 2019-12-05T06:19:44.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.149+0000 7f8270a16700 1 -- 172.21.15.16:0/2341573931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826c09aae0 con 0x7f826c09b920 2019-12-05T06:19:44.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.149+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=THROTTLE_DONE pgs=554 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.149+0000 7f82637fe700 1 -- 172.21.15.16:0/2341573931 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f825c000e70 con 0x7f826c09b920 2019-12-05T06:19:44.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.150+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=THROTTLE_DONE pgs=554 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.151+0000 7f82637fe700 1 -- 172.21.15.16:0/2341573931 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f825c0045c0 con 0x7f826c09b920 2019-12-05T06:19:44.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.151+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=THROTTLE_DONE pgs=554 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.151+0000 7f82637fe700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8250023630 0x7f8250025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:44.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.151+0000 7f82637fe700 1 -- 172.21.15.16:0/2341573931 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f825c045d40 con 0x7f826c09b920 2019-12-05T06:19:44.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.151+0000 7f826ad9d700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8250023630 0x7f8250025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:44.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.152+0000 7f826ad9d700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8250023630 0x7f8250025ac0 crc :-1 s=READY pgs=246 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:44.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.152+0000 7f8270a16700 1 -- 172.21.15.16:0/2341573931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f826c09aae0 con 0x7f826c09b920 2019-12-05T06:19:44.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.162+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=THROTTLE_DONE pgs=554 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.162+0000 7f82637fe700 1 -- 172.21.15.16:0/2341573931 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f825c004380 con 0x7f826c09b920 2019-12-05T06:19:44.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.421+0000 7f8270a16700 1 -- 172.21.15.16:0/2341573931 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f826c09aae0 con 0x7f826c09b920 2019-12-05T06:19:44.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.421+0000 7f826b59e700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=THROTTLE_DONE pgs=554 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:44.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.422+0000 7f82637fe700 1 -- 172.21.15.16:0/2341573931 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f825c004590 con 0x7f826c09b920 2019-12-05T06:19:44.412 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 71 2019-12-05T06:19:44.413 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:44.413 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:44.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.424+0000 7f8270a16700 1 -- 172.21.15.16:0/2341573931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8250023630 msgr2=0x7f8250025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.424+0000 7f8270a16700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8250023630 0x7f8250025ac0 crc :-1 s=READY pgs=246 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.424+0000 7f8270a16700 1 -- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 msgr2=0x7f826c097a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.424+0000 7f8270a16700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 0x7f826c097a80 secure :-1 s=READY pgs=554 cs=0 l=1 rx=0x7f825c010370 tx=0x7f825c0101a0).stop 2019-12-05T06:19:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.424+0000 7f8270a16700 1 -- 172.21.15.16:0/2341573931 shutdown_connections 2019-12-05T06:19:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.424+0000 7f8270a16700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8250023630 0x7f8250025ac0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.424+0000 7f8270a16700 1 --2- 172.21.15.16:0/2341573931 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f826c09b920 0x7f826c097a80 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.424+0000 7f8270a16700 1 --2- 172.21.15.16:0/2341573931 >> v2:172.21.15.76:3300/0 conn(0x7f826c096e90 0x7f826c09b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.425+0000 7f8270a16700 1 -- 172.21.15.16:0/2341573931 shutdown_connections 2019-12-05T06:19:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.425+0000 7f8270a16700 1 -- 172.21.15.16:0/2341573931 wait complete. 2019-12-05T06:19:44.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:44.425+0000 7f8270a16700 1 -- 172.21.15.16:0/2341573931 >> 172.21.15.16:0/2341573931 conn(0x7f826c0714f0 msgr2=0x7f826c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:44.425 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:19:46.425 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.725+0000 7f103245a700 1 Processor -- start 2019-12-05T06:19:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.726+0000 7f103245a700 1 -- start start 2019-12-05T06:19:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.728+0000 7f103245a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f102c08fef0 0x7f102c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.728+0000 7f103245a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c0958e0 con 0x7f102c08d2c0 2019-12-05T06:19:46.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.728+0000 7f103245a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c0629e0 con 0x7f102c08c6d0 2019-12-05T06:19:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.728+0000 7f103245a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f102c08df90 con 0x7f102c08fef0 2019-12-05T06:19:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.728+0000 7f102bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f102c08d2c0 0x7f102c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36144/0 (socket says 172.21.15.16:36144) 2019-12-05T06:19:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.728+0000 7f102bfff700 1 -- 172.21.15.16:0/392996432 learned_addr learned my addr 172.21.15.16:0/392996432 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.728+0000 7f10311f8700 1 --2- 172.21.15.16:0/392996432 >> v2:172.21.15.76:3300/0 conn(0x7f102c08fef0 0x7f102c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.729+0000 7f102b7fe700 1 -- 172.21.15.16:0/392996432 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3320762365 0 0) 0x7f101c001150 con 0x7f102c08d2c0 2019-12-05T06:19:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.729+0000 7f102b7fe700 1 -- 172.21.15.16:0/392996432 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1018002e80 con 0x7f102c08d2c0 2019-12-05T06:19:46.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.729+0000 7f102b7fe700 1 -- 172.21.15.16:0/392996432 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480988251 0 0) 0x7f1024001400 con 0x7f102c08c6d0 2019-12-05T06:19:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.729+0000 7f102b7fe700 1 -- 172.21.15.16:0/392996432 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1018004170 con 0x7f102c08c6d0 2019-12-05T06:19:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.729+0000 7f10311f8700 1 -- 172.21.15.16:0/392996432 >> v1:172.21.15.76:6789/0 conn(0x7f102c08c6d0 legacy=0x7f102c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.729+0000 7f10311f8700 1 -- 172.21.15.16:0/392996432 >> v1:172.21.15.16:6789/0 conn(0x7f102c08d2c0 legacy=0x7f102c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.729+0000 7f10311f8700 1 -- 172.21.15.16:0/392996432 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c08e620 con 0x7f102c08fef0 2019-12-05T06:19:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.729+0000 7f10311f8700 1 --2- 172.21.15.16:0/392996432 >> v2:172.21.15.76:3300/0 conn(0x7f102c08fef0 0x7f102c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f101400af80 tx=0x7f101400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.729+0000 7f10311f8700 1 --2- 172.21.15.16:0/392996432 >> v2:172.21.15.76:3300/0 conn(0x7f102c08fef0 0x7f102c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f101400af80 tx=0x7f101400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.729+0000 7f10311f8700 1 --2- 172.21.15.16:0/392996432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f102c08fef0 0x7f102c08f720 secure :-1 s=READY pgs=413 cs=0 l=1 rx=0x7f101400af80 tx=0x7f101400b790).ready entity=mon.1 client_cookie=30975b24ff64fcd6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.730+0000 7f10311f8700 1 --2- 172.21.15.16:0/392996432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f102c08fef0 0x7f102c08f720 secure :-1 s=THROTTLE_DONE pgs=413 cs=0 l=1 rx=0x7f101400af80 tx=0x7f101400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.730+0000 7f10311f8700 1 --2- 172.21.15.16:0/392996432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f102c08fef0 0x7f102c08f720 secure :-1 s=THROTTLE_DONE pgs=413 cs=0 l=1 rx=0x7f101400af80 tx=0x7f101400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.730+0000 7f102b7fe700 1 -- 172.21.15.16:0/392996432 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1014007240 con 0x7f102c08fef0 2019-12-05T06:19:46.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.730+0000 7f10311f8700 1 --2- 172.21.15.16:0/392996432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f102c08fef0 0x7f102c08f720 secure :-1 s=THROTTLE_DONE pgs=413 cs=0 l=1 rx=0x7f101400af80 tx=0x7f101400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.730+0000 7f102b7fe700 1 -- 172.21.15.16:0/392996432 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1014000c30 con 0x7f102c08fef0 2019-12-05T06:19:46.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.730+0000 7f102b7fe700 1 -- 172.21.15.16:0/392996432 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1014009f10 con 0x7f102c08fef0 2019-12-05T06:19:46.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.730+0000 7f103245a700 1 -- 172.21.15.16:0/392996432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f102c08fef0 msgr2=0x7f102c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.730+0000 7f103245a700 1 --2- 172.21.15.16:0/392996432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f102c08fef0 0x7f102c08f720 secure :-1 s=READY pgs=413 cs=0 l=1 rx=0x7f101400af80 tx=0x7f101400b790).stop 2019-12-05T06:19:46.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.732+0000 7f103245a700 1 -- 172.21.15.16:0/392996432 shutdown_connections 2019-12-05T06:19:46.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.732+0000 7f103245a700 1 --2- 172.21.15.16:0/392996432 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f102c08fef0 0x7f102c08f720 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:46.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.732+0000 7f103245a700 1 -- 172.21.15.16:0/392996432 shutdown_connections 2019-12-05T06:19:46.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.732+0000 7f103245a700 1 -- 172.21.15.16:0/392996432 wait complete. 2019-12-05T06:19:46.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.732+0000 7f103245a700 1 -- 172.21.15.16:0/392996432 >> 172.21.15.16:0/392996432 conn(0x7f102c061940 msgr2=0x7f102c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.738+0000 7f103245a700 1 Processor -- start 2019-12-05T06:19:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.738+0000 7f103245a700 1 -- start start 2019-12-05T06:19:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.740+0000 7f103245a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:46.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.740+0000 7f103245a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c128b20 con 0x7f102c1325e0 2019-12-05T06:19:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.740+0000 7f103245a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f102c128660 con 0x7f102c12c740 2019-12-05T06:19:46.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.740+0000 7f103245a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f102c12a7e0 con 0x7f102c12bf70 2019-12-05T06:19:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.740+0000 7f10309f7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f102c1325e0 0x7f102c12b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36148/0 (socket says 172.21.15.16:36148) 2019-12-05T06:19:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.740+0000 7f10309f7700 1 -- 172.21.15.16:0/239761886 learned_addr learned my addr 172.21.15.16:0/239761886 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.740+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.741+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2594603118 0 0) 0x7f10240015c0 con 0x7f102c1325e0 2019-12-05T06:19:46.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.741+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1010002e80 con 0x7f102c1325e0 2019-12-05T06:19:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.741+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225048759 0 0) 0x7f101c000f40 con 0x7f102c12c740 2019-12-05T06:19:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.741+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1010004220 con 0x7f102c12c740 2019-12-05T06:19:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.741+0000 7f10311f8700 1 -- 172.21.15.16:0/239761886 >> v1:172.21.15.76:6789/0 conn(0x7f102c12c740 legacy=0x7f102c154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.741+0000 7f10311f8700 1 -- 172.21.15.16:0/239761886 >> v1:172.21.15.16:6789/0 conn(0x7f102c1325e0 legacy=0x7f102c12b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.741+0000 7f10311f8700 1 -- 172.21.15.16:0/239761886 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f102c14a820 con 0x7f102c12bf70 2019-12-05T06:19:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.741+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.742+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.742+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:46.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.742+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.742+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10140099b0 con 0x7f102c12bf70 2019-12-05T06:19:46.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.742+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.742+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.742+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1014009bb0 con 0x7f102c12bf70 2019-12-05T06:19:46.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.742+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f101400a280 con 0x7f102c12bf70 2019-12-05T06:19:46.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.742+0000 7f103245a700 1 -- 172.21.15.16:0/239761886 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c12ae70 con 0x7f102c12bf70 2019-12-05T06:19:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.742+0000 7f103245a700 1 -- 172.21.15.16:0/239761886 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102c14a820 con 0x7f102c12bf70 2019-12-05T06:19:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.743+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.744+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1014009da0 con 0x7f102c12bf70 2019-12-05T06:19:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.744+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.744+0000 7f10297fa700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1010023650 0x7f1010025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.745+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f1014045e40 con 0x7f102c12bf70 2019-12-05T06:19:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.745+0000 7f10309f7700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1010023650 0x7f1010025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:46.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.745+0000 7f10309f7700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1010023650 0x7f1010025ae0 crc :-1 s=READY pgs=247 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:46.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.757+0000 7f103245a700 1 -- 172.21.15.16:0/239761886 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102c12ae70 con 0x7f102c12bf70 2019-12-05T06:19:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.767+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:46.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:46.767+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1014034f00 con 0x7f102c12bf70 2019-12-05T06:19:47.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.032+0000 7f103245a700 1 -- 172.21.15.16:0/239761886 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f102c14a820 con 0x7f102c12bf70 2019-12-05T06:19:47.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.034+0000 7f10311f8700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=THROTTLE_DONE pgs=140 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:47.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.034+0000 7f10297fa700 1 -- 172.21.15.16:0/239761886 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 71 v71) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f1014034f00 con 0x7f102c12bf70 2019-12-05T06:19:47.026 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:47.026 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":71,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:47.026 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 71 2019-12-05T06:19:47.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.038+0000 7f103245a700 1 -- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1010023650 msgr2=0x7f1010025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:47.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.038+0000 7f103245a700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1010023650 0x7f1010025ae0 crc :-1 s=READY pgs=247 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:47.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.038+0000 7f103245a700 1 -- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 msgr2=0x7f102c1584d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:47.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.038+0000 7f103245a700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 secure :-1 s=READY pgs=140 cs=0 l=1 rx=0x7f10140103c0 tx=0x7f1014010760).stop 2019-12-05T06:19:47.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.038+0000 7f103245a700 1 -- 172.21.15.16:0/239761886 shutdown_connections 2019-12-05T06:19:47.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.038+0000 7f103245a700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1010023650 0x7f1010025ae0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:47.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.038+0000 7f103245a700 1 --2- 172.21.15.16:0/239761886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f102c12bf70 0x7f102c1584d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:47.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.038+0000 7f103245a700 1 -- 172.21.15.16:0/239761886 shutdown_connections 2019-12-05T06:19:47.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.038+0000 7f103245a700 1 -- 172.21.15.16:0/239761886 wait complete. 2019-12-05T06:19:47.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:47.038+0000 7f103245a700 1 -- 172.21.15.16:0/239761886 >> 172.21.15.16:0/239761886 conn(0x7f102c0a2850 msgr2=0x7f102c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:47.039 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:19:49.040 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:19:49.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.297+0000 7f79b607f700 1 Processor -- start 2019-12-05T06:19:49.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.297+0000 7f79b607f700 1 -- start start 2019-12-05T06:19:49.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79b607f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79b008c6d0 0x7f79b008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:49.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79b607f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f79b008fef0 0x7f79b008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:49.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79b607f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b008ef20 0x7f79b008f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:19:49.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79b607f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b008df90 con 0x7f79b008c6d0 2019-12-05T06:19:49.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79b607f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b00a3f70 con 0x7f79b008fef0 2019-12-05T06:19:49.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79b607f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b008d830 con 0x7f79b008ef20 2019-12-05T06:19:49.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79affff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79b008c6d0 0x7f79b008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:49.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79affff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79b008c6d0 0x7f79b008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:50998/0 (socket says 172.21.15.16:50998) 2019-12-05T06:19:49.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79affff700 1 -- 172.21.15.16:0/3461825857 learned_addr learned my addr 172.21.15.16:0/3461825857 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:19:49.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3461825857 >> v2:172.21.15.76:3300/0 conn(0x7f79b008fef0 0x7f79b008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:49.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.299+0000 7f79b4e1d700 1 --2- 172.21.15.16:0/3461825857 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b008ef20 0x7f79b008f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:49.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.300+0000 7f79affff700 1 -- 172.21.15.16:0/3461825857 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b008ef20 msgr2=0x7f79b008f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:49.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.300+0000 7f79affff700 1 --2- 172.21.15.16:0/3461825857 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b008ef20 0x7f79b008f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:49.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.300+0000 7f79affff700 1 -- 172.21.15.16:0/3461825857 >> v2:172.21.15.76:3300/0 conn(0x7f79b008fef0 msgr2=0x7f79b008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:19:49.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.300+0000 7f79affff700 1 --2- 172.21.15.16:0/3461825857 >> v2:172.21.15.76:3300/0 conn(0x7f79b008fef0 0x7f79b008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:49.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.300+0000 7f79affff700 1 -- 172.21.15.16:0/3461825857 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b008d540 con 0x7f79b008c6d0 2019-12-05T06:19:49.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.300+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3461825857 >> v2:172.21.15.76:3300/0 conn(0x7f79b008fef0 0x7f79b008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:49.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.300+0000 7f79affff700 1 --2- 172.21.15.16:0/3461825857 >> v2:172.21.15.16:3300/0 conn(0x7f79b008c6d0 0x7f79b008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f79a000fac0 tx=0x7f79a000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.301+0000 7f79affff700 1 --2- 172.21.15.16:0/3461825857 >> v2:172.21.15.16:3300/0 conn(0x7f79b008c6d0 0x7f79b008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f79a000fac0 tx=0x7f79a000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.301+0000 7f79affff700 1 --2- 172.21.15.16:0/3461825857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b008c6d0 0x7f79b008cac0 secure :-1 s=READY pgs=557 cs=0 l=1 rx=0x7f79a000fac0 tx=0x7f79a000f8f0).ready entity=mon.0 client_cookie=5e8375df1752d8a7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:49.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.301+0000 7f79affff700 1 --2- 172.21.15.16:0/3461825857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b008c6d0 0x7f79b008cac0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f79a000fac0 tx=0x7f79a000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.301+0000 7f79affff700 1 --2- 172.21.15.16:0/3461825857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b008c6d0 0x7f79b008cac0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f79a000fac0 tx=0x7f79a000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.301+0000 7f79affff700 1 --2- 172.21.15.16:0/3461825857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b008c6d0 0x7f79b008cac0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7f79a000fac0 tx=0x7f79a000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.301+0000 7f79aeffd700 1 -- 172.21.15.16:0/3461825857 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79a0000ce0 con 0x7f79b008c6d0 2019-12-05T06:19:49.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.301+0000 7f79aeffd700 1 -- 172.21.15.16:0/3461825857 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a0001010 con 0x7f79b008c6d0 2019-12-05T06:19:49.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.301+0000 7f79aeffd700 1 -- 172.21.15.16:0/3461825857 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79a0001720 con 0x7f79b008c6d0 2019-12-05T06:19:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.301+0000 7f79b607f700 1 -- 172.21.15.16:0/3461825857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b008c6d0 msgr2=0x7f79b008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.301+0000 7f79b607f700 1 --2- 172.21.15.16:0/3461825857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b008c6d0 0x7f79b008cac0 secure :-1 s=READY pgs=557 cs=0 l=1 rx=0x7f79a000fac0 tx=0x7f79a000f8f0).stop 2019-12-05T06:19:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.303+0000 7f79b607f700 1 -- 172.21.15.16:0/3461825857 shutdown_connections 2019-12-05T06:19:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.303+0000 7f79b607f700 1 --2- 172.21.15.16:0/3461825857 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b008ef20 0x7f79b008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.303+0000 7f79b607f700 1 --2- 172.21.15.16:0/3461825857 >> v2:172.21.15.76:3300/0 conn(0x7f79b008fef0 0x7f79b008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:19:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.303+0000 7f79b607f700 1 --2- 172.21.15.16:0/3461825857 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b008c6d0 0x7f79b008cac0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:49.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.303+0000 7f79b607f700 1 -- 172.21.15.16:0/3461825857 shutdown_connections 2019-12-05T06:19:49.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.303+0000 7f79b607f700 1 -- 172.21.15.16:0/3461825857 wait complete. 2019-12-05T06:19:49.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.303+0000 7f79b607f700 1 -- 172.21.15.16:0/3461825857 >> 172.21.15.16:0/3461825857 conn(0x7f79b0061940 msgr2=0x7f79b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:49.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.308+0000 7f79b607f700 1 Processor -- start 2019-12-05T06:19:49.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.309+0000 7f79b607f700 1 -- start start 2019-12-05T06:19:49.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.310+0000 7f79b607f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b0096e90 0x7f79b009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:49.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.310+0000 7f79b607f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f79b0084f30 0x7f79b009b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:49.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.310+0000 7f79b607f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b0084b90 con 0x7f79b0097280 2019-12-05T06:19:49.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.310+0000 7f79b607f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b0099a20 con 0x7f79b0084f30 2019-12-05T06:19:49.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.310+0000 7f79b607f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b0099cf0 con 0x7f79b0096e90 2019-12-05T06:19:49.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.311+0000 7f79b4e1d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f79b0097280 0x7f79b0097700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36176/0 (socket says 172.21.15.16:36176) 2019-12-05T06:19:49.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.311+0000 7f79af7fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f79b0084f30 0x7f79b009b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:49.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.311+0000 7f79b4e1d700 1 -- 172.21.15.16:0/3184949132 learned_addr learned my addr 172.21.15.16:0/3184949132 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:19:49.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.311+0000 7f79affff700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b0096e90 0x7f79b009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:49.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.311+0000 7f79acff9700 1 -- 172.21.15.16:0/3184949132 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353829869 0 0) 0x7f79a80095b0 con 0x7f79b0097280 2019-12-05T06:19:49.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.311+0000 7f79acff9700 1 -- 172.21.15.16:0/3184949132 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7990002e80 con 0x7f79b0097280 2019-12-05T06:19:49.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.311+0000 7f79af7fe700 1 -- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b0096e90 msgr2=0x7f79b009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:49.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.311+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b0096e90 0x7f79b009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:49.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.311+0000 7f79af7fe700 1 -- 172.21.15.16:0/3184949132 >> v1:172.21.15.16:6789/0 conn(0x7f79b0097280 legacy=0x7f79b0097700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79af7fe700 1 -- 172.21.15.16:0/3184949132 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b009aae0 con 0x7f79b0084f30 2019-12-05T06:19:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79affff700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b0096e90 0x7f79b009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:19:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> v2:172.21.15.76:3300/0 conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> v2:172.21.15.76:3300/0 conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=READY pgs=415 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79acff9700 1 -- 172.21.15.16:0/3184949132 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79a4000940 con 0x7f79b0084f30 2019-12-05T06:19:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79acff9700 1 -- 172.21.15.16:0/3184949132 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a400dff0 con 0x7f79b0084f30 2019-12-05T06:19:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79acff9700 1 -- 172.21.15.16:0/3184949132 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79a400e680 con 0x7f79b0084f30 2019-12-05T06:19:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79b607f700 1 -- 172.21.15.16:0/3184949132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b009aae0 con 0x7f79b0084f30 2019-12-05T06:19:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.312+0000 7f79b607f700 1 -- 172.21.15.16:0/3184949132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b009aae0 con 0x7f79b0084f30 2019-12-05T06:19:49.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.313+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.314+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.314+0000 7f79acff9700 1 -- 172.21.15.16:0/3184949132 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f79a400db00 con 0x7f79b0084f30 2019-12-05T06:19:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.315+0000 7f79acff9700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7990023670 0x7f7990025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:19:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.315+0000 7f79acff9700 1 -- 172.21.15.16:0/3184949132 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f79a404a0f0 con 0x7f79b0084f30 2019-12-05T06:19:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.315+0000 7f79affff700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7990023670 0x7f7990025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:19:49.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.315+0000 7f79affff700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7990023670 0x7f7990025ae0 crc :-1 s=READY pgs=249 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:19:49.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.329+0000 7f79b607f700 1 -- 172.21.15.16:0/3184949132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b009aae0 con 0x7f79b0084f30 2019-12-05T06:19:49.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.338+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.338+0000 7f79acff9700 1 -- 172.21.15.16:0/3184949132 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f79a4000940 con 0x7f79b0084f30 2019-12-05T06:19:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.603+0000 7f79b607f700 1 -- 172.21.15.16:0/3184949132 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79b009aae0 con 0x7f79b0084f30 2019-12-05T06:19:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.604+0000 7f79af7fe700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=THROTTLE_DONE pgs=415 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:19:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.605+0000 7f79acff9700 1 -- 172.21.15.16:0/3184949132 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f79a400da10 con 0x7f79b0084f30 2019-12-05T06:19:49.597 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:19:49.597 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5604,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:19:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2019-12-05T06:19:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 -- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7990023670 msgr2=0x7f7990025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7990023670 0x7f7990025ae0 crc :-1 s=READY pgs=249 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 -- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 msgr2=0x7f79b009b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:19:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 0x7f79b009b920 secure :-1 s=READY pgs=415 cs=0 l=1 rx=0x7f79a40147b0 tx=0x7f79a4014b50).stop 2019-12-05T06:19:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 -- 172.21.15.16:0/3184949132 shutdown_connections 2019-12-05T06:19:49.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7990023670 0x7f7990025ae0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f79b0084f30 0x7f79b009b920 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 --2- 172.21.15.16:0/3184949132 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b0096e90 0x7f79b009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:19:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 -- 172.21.15.16:0/3184949132 shutdown_connections 2019-12-05T06:19:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 -- 172.21.15.16:0/3184949132 wait complete. 2019-12-05T06:19:49.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:19:49.609+0000 7f79b607f700 1 -- 172.21.15.16:0/3184949132 >> 172.21.15.16:0/3184949132 conn(0x7f79b00714f0 msgr2=0x7f79b009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:19:49.611 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:19:49.611 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2019-12-05T06:20:01.868 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:20:01.875 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:20:12.617 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.917+0000 7f800275e700 1 Processor -- start 2019-12-05T06:20:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.917+0000 7f800275e700 1 -- start start 2019-12-05T06:20:12.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.919+0000 7f800275e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 0x7f7ffc08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.919+0000 7f800275e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ffc0958e0 con 0x7f7ffc08c6d0 2019-12-05T06:20:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.919+0000 7f800275e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ffc0629e0 con 0x7f7ffc08d2c0 2019-12-05T06:20:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.919+0000 7f800275e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ffc08df90 con 0x7f7ffc08fef0 2019-12-05T06:20:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.919+0000 7f8000cfb700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7ffc08c6d0 0x7f7ffc08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36192/0 (socket says 172.21.15.16:36192) 2019-12-05T06:20:12.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.919+0000 7f8000cfb700 1 -- 172.21.15.16:0/1259612473 learned_addr learned my addr 172.21.15.16:0/1259612473 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.919+0000 7f80014fc700 1 --2- 172.21.15.16:0/1259612473 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 0x7f7ffc08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.920+0000 7f7ffb7fe700 1 -- 172.21.15.16:0/1259612473 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2733411094 0 0) 0x7f7fe4001150 con 0x7f7ffc08c6d0 2019-12-05T06:20:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.920+0000 7f7ffb7fe700 1 -- 172.21.15.16:0/1259612473 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe8002e80 con 0x7f7ffc08c6d0 2019-12-05T06:20:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.920+0000 7f7ffb7fe700 1 -- 172.21.15.16:0/1259612473 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617665894 0 0) 0x7f7fec001080 con 0x7f7ffc08d2c0 2019-12-05T06:20:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.920+0000 7f7ffb7fe700 1 -- 172.21.15.16:0/1259612473 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe8002e80 con 0x7f7ffc08d2c0 2019-12-05T06:20:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.920+0000 7f80014fc700 1 -- 172.21.15.16:0/1259612473 >> v1:172.21.15.76:6789/0 conn(0x7f7ffc08d2c0 legacy=0x7f7ffc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:12.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.920+0000 7f80014fc700 1 -- 172.21.15.16:0/1259612473 >> v1:172.21.15.16:6789/0 conn(0x7f7ffc08c6d0 legacy=0x7f7ffc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:12.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.920+0000 7f80014fc700 1 -- 172.21.15.16:0/1259612473 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc08e620 con 0x7f7ffc08fef0 2019-12-05T06:20:12.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.920+0000 7f80014fc700 1 --2- 172.21.15.16:0/1259612473 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 0x7f7ffc08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7ff400b300 tx=0x7f7ff400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.921+0000 7f80014fc700 1 --2- 172.21.15.16:0/1259612473 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 0x7f7ffc08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7ff400b300 tx=0x7f7ff400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.921+0000 7f80014fc700 1 --2- 172.21.15.16:0/1259612473 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 0x7f7ffc08f720 secure :-1 s=READY pgs=141 cs=0 l=1 rx=0x7f7ff400b300 tx=0x7f7ff400bb10).ready entity=mon.2 client_cookie=bf51fc6341dcb0b3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.921+0000 7f80014fc700 1 --2- 172.21.15.16:0/1259612473 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 0x7f7ffc08f720 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f7ff400b300 tx=0x7f7ff400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.921+0000 7f7ffb7fe700 1 -- 172.21.15.16:0/1259612473 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ff40075c0 con 0x7f7ffc08fef0 2019-12-05T06:20:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.921+0000 7f80014fc700 1 --2- 172.21.15.16:0/1259612473 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 0x7f7ffc08f720 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f7ff400b300 tx=0x7f7ff400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.921+0000 7f7ffb7fe700 1 -- 172.21.15.16:0/1259612473 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ff4000f30 con 0x7f7ffc08fef0 2019-12-05T06:20:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.921+0000 7f80014fc700 1 --2- 172.21.15.16:0/1259612473 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 0x7f7ffc08f720 secure :-1 s=THROTTLE_DONE pgs=141 cs=0 l=1 rx=0x7f7ff400b300 tx=0x7f7ff400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.921+0000 7f7ffb7fe700 1 -- 172.21.15.16:0/1259612473 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ff4009f50 con 0x7f7ffc08fef0 2019-12-05T06:20:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.921+0000 7f800275e700 1 -- 172.21.15.16:0/1259612473 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 msgr2=0x7f7ffc08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:12.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.921+0000 7f800275e700 1 --2- 172.21.15.16:0/1259612473 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 0x7f7ffc08f720 secure :-1 s=READY pgs=141 cs=0 l=1 rx=0x7f7ff400b300 tx=0x7f7ff400bb10).stop 2019-12-05T06:20:12.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.923+0000 7f800275e700 1 -- 172.21.15.16:0/1259612473 shutdown_connections 2019-12-05T06:20:12.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.923+0000 7f800275e700 1 --2- 172.21.15.16:0/1259612473 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ffc08fef0 0x7f7ffc08f720 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:12.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.923+0000 7f800275e700 1 -- 172.21.15.16:0/1259612473 shutdown_connections 2019-12-05T06:20:12.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.923+0000 7f800275e700 1 -- 172.21.15.16:0/1259612473 wait complete. 2019-12-05T06:20:12.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.923+0000 7f800275e700 1 -- 172.21.15.16:0/1259612473 >> 172.21.15.16:0/1259612473 conn(0x7f7ffc061940 msgr2=0x7f7ffc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.929+0000 7f800275e700 1 Processor -- start 2019-12-05T06:20:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.929+0000 7f800275e700 1 -- start start 2019-12-05T06:20:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.931+0000 7f800275e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ffc153a50 0x7f7ffc153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.931+0000 7f800275e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ffc06e5c0 con 0x7f7ffc12a1d0 2019-12-05T06:20:12.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.931+0000 7f800275e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ffc098500 con 0x7f7ffc129a00 2019-12-05T06:20:12.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.931+0000 7f800275e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ffc06e250 con 0x7f7ffc153a50 2019-12-05T06:20:12.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.931+0000 7f8000cfb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ffc153a50 0x7f7ffc153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:12.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.932+0000 7f8000cfb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ffc153a50 0x7f7ffc153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51028/0 (socket says 172.21.15.16:51028) 2019-12-05T06:20:12.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.932+0000 7f8000cfb700 1 -- 172.21.15.16:0/4227140440 learned_addr learned my addr 172.21.15.16:0/4227140440 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:12.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.932+0000 7f7ff97fa700 1 -- 172.21.15.16:0/4227140440 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2264430648 0 0) 0x7f7ff4009d10 con 0x7f7ffc129a00 2019-12-05T06:20:12.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.932+0000 7f7ff97fa700 1 -- 172.21.15.16:0/4227140440 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fe0002e80 con 0x7f7ffc129a00 2019-12-05T06:20:12.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.932+0000 7f8000cfb700 1 -- 172.21.15.16:0/4227140440 >> v1:172.21.15.76:6789/0 conn(0x7f7ffc129a00 legacy=0x7f7ffc152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:12.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.932+0000 7f8000cfb700 1 -- 172.21.15.16:0/4227140440 >> v1:172.21.15.16:6789/0 conn(0x7f7ffc12a1d0 legacy=0x7f7ffc129200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:12.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.932+0000 7f8000cfb700 1 -- 172.21.15.16:0/4227140440 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ffc128900 con 0x7f7ffc153a50 2019-12-05T06:20:12.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.932+0000 7f8000cfb700 1 --2- 172.21.15.16:0/4227140440 >> v2:172.21.15.16:3300/0 conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.933+0000 7f8000cfb700 1 --2- 172.21.15.16:0/4227140440 >> v2:172.21.15.16:3300/0 conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.933+0000 7f8000cfb700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=READY pgs=562 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:12.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.933+0000 7f8000cfb700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.933+0000 7f7ff97fa700 1 -- 172.21.15.16:0/4227140440 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fe4000c90 con 0x7f7ffc153a50 2019-12-05T06:20:12.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.933+0000 7f8000cfb700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.933+0000 7f8000cfb700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.933+0000 7f7ff97fa700 1 -- 172.21.15.16:0/4227140440 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe40010c0 con 0x7f7ffc153a50 2019-12-05T06:20:12.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.933+0000 7f800275e700 1 -- 172.21.15.16:0/4227140440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ffc128900 con 0x7f7ffc153a50 2019-12-05T06:20:12.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.933+0000 7f7ff97fa700 1 -- 172.21.15.16:0/4227140440 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fe40012b0 con 0x7f7ffc153a50 2019-12-05T06:20:12.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.933+0000 7f800275e700 1 -- 172.21.15.16:0/4227140440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ffc128900 con 0x7f7ffc153a50 2019-12-05T06:20:12.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.934+0000 7f8000cfb700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.935+0000 7f7ff97fa700 1 -- 172.21.15.16:0/4227140440 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7fe4000d30 con 0x7f7ffc153a50 2019-12-05T06:20:12.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.935+0000 7f8000cfb700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.935+0000 7f7ff97fa700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7fe0023670 0x7f7fe0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:12.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.935+0000 7f7ff97fa700 1 -- 172.21.15.16:0/4227140440 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f7fe4045e00 con 0x7f7ffc153a50 2019-12-05T06:20:12.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.936+0000 7f7ffbfff700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7fe0023670 0x7f7fe0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:12.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.936+0000 7f7ffbfff700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7fe0023670 0x7f7fe0025ae0 crc :-1 s=READY pgs=250 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:12.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.948+0000 7f800275e700 1 -- 172.21.15.16:0/4227140440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ffc128900 con 0x7f7ffc153a50 2019-12-05T06:20:12.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.955+0000 7f8000cfb700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:12.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:12.955+0000 7f7ff97fa700 1 -- 172.21.15.16:0/4227140440 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7fe40075d0 con 0x7f7ffc153a50 2019-12-05T06:20:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.193+0000 7f800275e700 1 -- 172.21.15.16:0/4227140440 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ffc128900 con 0x7f7ffc153a50 2019-12-05T06:20:13.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.193+0000 7f8000cfb700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=THROTTLE_DONE pgs=562 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.193+0000 7f7ff97fa700 1 -- 172.21.15.16:0/4227140440 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f7fe4000c90 con 0x7f7ffc153a50 2019-12-05T06:20:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2019-12-05T06:20:13.184 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:13.184 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5604,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.196+0000 7f800275e700 1 -- 172.21.15.16:0/4227140440 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7fe0023670 msgr2=0x7f7fe0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.196+0000 7f800275e700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7fe0023670 0x7f7fe0025ae0 crc :-1 s=READY pgs=250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.196+0000 7f800275e700 1 -- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 msgr2=0x7f7ffc153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.196+0000 7f800275e700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 0x7f7ffc153e40 secure :-1 s=READY pgs=562 cs=0 l=1 rx=0x7f7fe40103e0 tx=0x7f7fe4010780).stop 2019-12-05T06:20:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.196+0000 7f800275e700 1 -- 172.21.15.16:0/4227140440 shutdown_connections 2019-12-05T06:20:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.196+0000 7f800275e700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7fe0023670 0x7f7fe0025ae0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.196+0000 7f800275e700 1 --2- 172.21.15.16:0/4227140440 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ffc153a50 0x7f7ffc153e40 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.196+0000 7f800275e700 1 -- 172.21.15.16:0/4227140440 shutdown_connections 2019-12-05T06:20:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.197+0000 7f800275e700 1 -- 172.21.15.16:0/4227140440 wait complete. 2019-12-05T06:20:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.197+0000 7f800275e700 1 -- 172.21.15.16:0/4227140440 >> 172.21.15.16:0/4227140440 conn(0x7f7ffc06d260 msgr2=0x7f7ffc09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:13.197 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:20:13.198 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:20:13.198 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:20:13.211 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:20:13.211 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:13.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.551+0000 7f57c643b700 1 Processor -- start 2019-12-05T06:20:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.551+0000 7f57c643b700 1 -- start start 2019-12-05T06:20:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.552+0000 7f57c643b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f57c008c6d0 0x7f57c008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.552+0000 7f57c643b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c008fef0 0x7f57c008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.553+0000 7f57c643b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c00958e0 con 0x7f57c008ef20 2019-12-05T06:20:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.553+0000 7f57c643b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c008df90 con 0x7f57c008c6d0 2019-12-05T06:20:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.553+0000 7f57c643b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c00a3f70 con 0x7f57c008fef0 2019-12-05T06:20:13.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.553+0000 7f57bffff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c008fef0 0x7f57c008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.553+0000 7f57c49d8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f57c008c6d0 0x7f57c008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.553+0000 7f57c49d8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f57c008c6d0 0x7f57c008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:37130/0 (socket says 172.21.15.16:37130) 2019-12-05T06:20:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.553+0000 7f57bffff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c008fef0 0x7f57c008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:55030/0 (socket says 172.21.15.16:55030) 2019-12-05T06:20:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.553+0000 7f57c49d8700 1 -- 172.21.15.16:0/422805514 learned_addr learned my addr 172.21.15.16:0/422805514 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.554+0000 7f57c49d8700 1 -- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c008fef0 msgr2=0x7f57c008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.554+0000 7f57c49d8700 1 --2- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c008fef0 0x7f57c008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:13.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.554+0000 7f57c49d8700 1 -- 172.21.15.16:0/422805514 >> v1:172.21.15.76:6789/0 conn(0x7f57c008ef20 legacy=0x7f57c008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.554+0000 7f57bf7fe700 1 -- 172.21.15.16:0/422805514 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151375134 0 0) 0x7f57a80009b0 con 0x7f57c008ef20 2019-12-05T06:20:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.554+0000 7f57c49d8700 1 -- 172.21.15.16:0/422805514 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c008d540 con 0x7f57c008c6d0 2019-12-05T06:20:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.554+0000 7f57bffff700 1 --2- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c008fef0 0x7f57c008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.554+0000 7f57c49d8700 1 --2- 172.21.15.16:0/422805514 >> v2:172.21.15.76:3300/0 conn(0x7f57c008c6d0 0x7f57c008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f57b800f740 tx=0x7f57b800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.554+0000 7f57c49d8700 1 --2- 172.21.15.16:0/422805514 >> v2:172.21.15.76:3300/0 conn(0x7f57c008c6d0 0x7f57c008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f57b800f740 tx=0x7f57b800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.554+0000 7f57c49d8700 1 --2- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c008c6d0 0x7f57c008cac0 secure :-1 s=READY pgs=419 cs=0 l=1 rx=0x7f57b800f740 tx=0x7f57b800fae0).ready entity=mon.1 client_cookie=b83204aed6e0424 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:13.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.555+0000 7f57c49d8700 1 --2- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c008c6d0 0x7f57c008cac0 secure :-1 s=THROTTLE_DONE pgs=419 cs=0 l=1 rx=0x7f57b800f740 tx=0x7f57b800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.555+0000 7f57bf7fe700 1 -- 172.21.15.16:0/422805514 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57b8000960 con 0x7f57c008c6d0 2019-12-05T06:20:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.555+0000 7f57c49d8700 1 --2- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c008c6d0 0x7f57c008cac0 secure :-1 s=THROTTLE_DONE pgs=419 cs=0 l=1 rx=0x7f57b800f740 tx=0x7f57b800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.555+0000 7f57c49d8700 1 --2- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c008c6d0 0x7f57c008cac0 secure :-1 s=THROTTLE_DONE pgs=419 cs=0 l=1 rx=0x7f57b800f740 tx=0x7f57b800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.555+0000 7f57bf7fe700 1 -- 172.21.15.16:0/422805514 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57b8000c90 con 0x7f57c008c6d0 2019-12-05T06:20:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.555+0000 7f57bf7fe700 1 -- 172.21.15.16:0/422805514 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57b80013a0 con 0x7f57c008c6d0 2019-12-05T06:20:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.555+0000 7f57c643b700 1 -- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c008c6d0 msgr2=0x7f57c008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:13.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.555+0000 7f57c643b700 1 --2- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c008c6d0 0x7f57c008cac0 secure :-1 s=READY pgs=419 cs=0 l=1 rx=0x7f57b800f740 tx=0x7f57b800fae0).stop 2019-12-05T06:20:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.557+0000 7f57c643b700 1 -- 172.21.15.16:0/422805514 shutdown_connections 2019-12-05T06:20:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.557+0000 7f57c643b700 1 --2- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c008fef0 0x7f57c008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.557+0000 7f57c643b700 1 --2- 172.21.15.16:0/422805514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f57c008c6d0 0x7f57c008cac0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.557+0000 7f57c643b700 1 -- 172.21.15.16:0/422805514 shutdown_connections 2019-12-05T06:20:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.557+0000 7f57c643b700 1 -- 172.21.15.16:0/422805514 wait complete. 2019-12-05T06:20:13.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.557+0000 7f57c643b700 1 -- 172.21.15.16:0/422805514 >> 172.21.15.16:0/422805514 conn(0x7f57c0061940 msgr2=0x7f57c0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:13.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.563+0000 7f57c643b700 1 Processor -- start 2019-12-05T06:20:13.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.563+0000 7f57c643b700 1 -- start start 2019-12-05T06:20:13.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.565+0000 7f57c643b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c01325e0 0x7f57c012b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:13.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.565+0000 7f57c643b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f57c0154c60 0x7f57c0157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:13.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.565+0000 7f57c643b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57c0128b20 con 0x7f57c012c740 2019-12-05T06:20:13.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.565+0000 7f57c643b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57c0129db0 con 0x7f57c0154c60 2019-12-05T06:20:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.565+0000 7f57c643b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57c012a080 con 0x7f57c01325e0 2019-12-05T06:20:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.565+0000 7f57bffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f57c012c740 0x7f57c012bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36214/0 (socket says 172.21.15.16:36214) 2019-12-05T06:20:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.565+0000 7f57bffff700 1 -- 172.21.15.16:0/1833478648 learned_addr learned my addr 172.21.15.16:0/1833478648 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.565+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> v2:172.21.15.16:3300/0 conn(0x7f57c0154c60 0x7f57c0157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.565+0000 7f57c49d8700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c01325e0 0x7f57c012b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.566+0000 7f57c51d9700 1 -- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c01325e0 msgr2=0x7f57c012b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:13.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.566+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c01325e0 0x7f57c012b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.566+0000 7f57c51d9700 1 -- 172.21.15.16:0/1833478648 >> v1:172.21.15.16:6789/0 conn(0x7f57c012c740 legacy=0x7f57c012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.566+0000 7f57c51d9700 1 -- 172.21.15.16:0/1833478648 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57c012ae70 con 0x7f57c0154c60 2019-12-05T06:20:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.566+0000 7f57bd7fa700 1 -- 172.21.15.16:0/1833478648 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438509867 0 0) 0x7f57b000e110 con 0x7f57c012c740 2019-12-05T06:20:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.566+0000 7f57c49d8700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c01325e0 0x7f57c012b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.566+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> v2:172.21.15.16:3300/0 conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.566+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> v2:172.21.15.16:3300/0 conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.566+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=READY pgs=564 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.567+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.567+0000 7f57bd7fa700 1 -- 172.21.15.16:0/1833478648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57a8000940 con 0x7f57c0154c60 2019-12-05T06:20:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.567+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.567+0000 7f57bd7fa700 1 -- 172.21.15.16:0/1833478648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57a800efe0 con 0x7f57c0154c60 2019-12-05T06:20:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.567+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.567+0000 7f57c643b700 1 -- 172.21.15.16:0/1833478648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57c012ae70 con 0x7f57c0154c60 2019-12-05T06:20:13.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.567+0000 7f57bd7fa700 1 -- 172.21.15.16:0/1833478648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57a8000940 con 0x7f57c0154c60 2019-12-05T06:20:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.567+0000 7f57c643b700 1 -- 172.21.15.16:0/1833478648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57c012ae70 con 0x7f57c0154c60 2019-12-05T06:20:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.568+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.569+0000 7f57bd7fa700 1 -- 172.21.15.16:0/1833478648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f57a800ebb0 con 0x7f57c0154c60 2019-12-05T06:20:13.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.569+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.569+0000 7f57bd7fa700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f57a4023630 0x7f57a4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.569+0000 7f57bd7fa700 1 -- 172.21.15.16:0/1833478648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f57a8045f60 con 0x7f57c0154c60 2019-12-05T06:20:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.570+0000 7f57c49d8700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f57a4023630 0x7f57a4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:13.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.570+0000 7f57c49d8700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f57a4023630 0x7f57a4025ac0 crc :-1 s=READY pgs=251 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:13.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.582+0000 7f57c643b700 1 -- 172.21.15.16:0/1833478648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57c012ae70 con 0x7f57c0154c60 2019-12-05T06:20:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.588+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.588+0000 7f57bd7fa700 1 -- 172.21.15.16:0/1833478648 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f57a8000940 con 0x7f57c0154c60 2019-12-05T06:20:13.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.849+0000 7f57c643b700 1 -- 172.21.15.16:0/1833478648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f57c012ae70 con 0x7f57c0154c60 2019-12-05T06:20:13.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.850+0000 7f57c51d9700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=THROTTLE_DONE pgs=564 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.850+0000 7f57bd7fa700 1 -- 172.21.15.16:0/1833478648 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f57a800eb10 con 0x7f57c0154c60 2019-12-05T06:20:13.840 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2019-12-05T06:20:13.840 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:13.841 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5604,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:13.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.852+0000 7f57c643b700 1 -- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f57a4023630 msgr2=0x7f57a4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:13.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.852+0000 7f57c643b700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f57a4023630 0x7f57a4025ac0 crc :-1 s=READY pgs=251 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:13.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.852+0000 7f57c643b700 1 -- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 msgr2=0x7f57c0157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:13.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.852+0000 7f57c643b700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 0x7f57c0157060 secure :-1 s=READY pgs=564 cs=0 l=1 rx=0x7f57a80103c0 tx=0x7f57a8010760).stop 2019-12-05T06:20:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.852+0000 7f57c643b700 1 -- 172.21.15.16:0/1833478648 shutdown_connections 2019-12-05T06:20:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.852+0000 7f57c643b700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f57a4023630 0x7f57a4025ac0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.852+0000 7f57c643b700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f57c0154c60 0x7f57c0157060 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.853+0000 7f57c643b700 1 --2- 172.21.15.16:0/1833478648 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f57c01325e0 0x7f57c012b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.853+0000 7f57c643b700 1 -- 172.21.15.16:0/1833478648 shutdown_connections 2019-12-05T06:20:13.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.853+0000 7f57c643b700 1 -- 172.21.15.16:0/1833478648 wait complete. 2019-12-05T06:20:13.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:13.853+0000 7f57c643b700 1 -- 172.21.15.16:0/1833478648 >> 172.21.15.16:0/1833478648 conn(0x7f57c00a2850 msgr2=0x7f57c006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:13.854 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:20:15.855 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:16.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.155+0000 7fab16ae3700 1 Processor -- start 2019-12-05T06:20:16.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.155+0000 7fab16ae3700 1 -- start start 2019-12-05T06:20:16.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.157+0000 7fab16ae3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fab1008c6d0 0x7fab1008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:16.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.157+0000 7fab16ae3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab1008fef0 0x7fab1008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:16.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.157+0000 7fab16ae3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab1008fb80 con 0x7fab1008ef20 2019-12-05T06:20:16.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.157+0000 7fab16ae3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab1008df90 con 0x7fab1008fef0 2019-12-05T06:20:16.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.157+0000 7fab16ae3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab100a3f70 con 0x7fab1008c6d0 2019-12-05T06:20:16.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.157+0000 7fab1487f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab1008fef0 0x7fab1008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:16.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.158+0000 7fab1487f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab1008fef0 0x7fab1008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51060/0 (socket says 172.21.15.16:51060) 2019-12-05T06:20:16.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.158+0000 7fab1487f700 1 -- 172.21.15.16:0/715477167 learned_addr learned my addr 172.21.15.16:0/715477167 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:16.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.158+0000 7fab15881700 1 --1- 172.21.15.16:0/715477167 >> v1:172.21.15.16:6789/0 conn(0x7fab1008ef20 0x7fab1008f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36226/0 (socket says 172.21.15.16:36226) 2019-12-05T06:20:16.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.158+0000 7fab15080700 1 --2- 172.21.15.16:0/715477167 >> v2:172.21.15.76:3300/0 conn(0x7fab1008c6d0 0x7fab1008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:16.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.158+0000 7fab1487f700 1 -- 172.21.15.16:0/715477167 >> v2:172.21.15.76:3300/0 conn(0x7fab1008c6d0 msgr2=0x7fab1008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:16.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.158+0000 7fab1487f700 1 --2- 172.21.15.16:0/715477167 >> v2:172.21.15.76:3300/0 conn(0x7fab1008c6d0 0x7fab1008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:16.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.158+0000 7fab1487f700 1 -- 172.21.15.16:0/715477167 >> v1:172.21.15.16:6789/0 conn(0x7fab1008ef20 legacy=0x7fab1008f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:16.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.158+0000 7fab0ffff700 1 -- 172.21.15.16:0/715477167 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2278111602 0 0) 0x7fab080009b0 con 0x7fab1008ef20 2019-12-05T06:20:16.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.158+0000 7fab1487f700 1 -- 172.21.15.16:0/715477167 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1008d540 con 0x7fab1008fef0 2019-12-05T06:20:16.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab1487f700 1 --2- 172.21.15.16:0/715477167 >> v2:172.21.15.16:3300/0 conn(0x7fab1008fef0 0x7fab1008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faaf800f6c0 tx=0x7faaf800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab1487f700 1 --2- 172.21.15.16:0/715477167 >> v2:172.21.15.16:3300/0 conn(0x7fab1008fef0 0x7fab1008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faaf800f6c0 tx=0x7faaf800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab1487f700 1 --2- 172.21.15.16:0/715477167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1008fef0 0x7fab1008d000 secure :-1 s=READY pgs=566 cs=0 l=1 rx=0x7faaf800f6c0 tx=0x7faaf800fa60).ready entity=mon.0 client_cookie=4c6f1c01737431ef server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab1487f700 1 --2- 172.21.15.16:0/715477167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1008fef0 0x7fab1008d000 secure :-1 s=THROTTLE_DONE pgs=566 cs=0 l=1 rx=0x7faaf800f6c0 tx=0x7faaf800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab0ffff700 1 -- 172.21.15.16:0/715477167 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaf8000960 con 0x7fab1008fef0 2019-12-05T06:20:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab1487f700 1 --2- 172.21.15.16:0/715477167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1008fef0 0x7fab1008d000 secure :-1 s=THROTTLE_DONE pgs=566 cs=0 l=1 rx=0x7faaf800f6c0 tx=0x7faaf800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab1487f700 1 --2- 172.21.15.16:0/715477167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1008fef0 0x7fab1008d000 secure :-1 s=THROTTLE_DONE pgs=566 cs=0 l=1 rx=0x7faaf800f6c0 tx=0x7faaf800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab0ffff700 1 -- 172.21.15.16:0/715477167 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf8000c90 con 0x7fab1008fef0 2019-12-05T06:20:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab0ffff700 1 -- 172.21.15.16:0/715477167 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaf80013a0 con 0x7fab1008fef0 2019-12-05T06:20:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab16ae3700 1 -- 172.21.15.16:0/715477167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1008fef0 msgr2=0x7fab1008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.159+0000 7fab16ae3700 1 --2- 172.21.15.16:0/715477167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1008fef0 0x7fab1008d000 secure :-1 s=READY pgs=566 cs=0 l=1 rx=0x7faaf800f6c0 tx=0x7faaf800fa60).stop 2019-12-05T06:20:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.161+0000 7fab16ae3700 1 -- 172.21.15.16:0/715477167 shutdown_connections 2019-12-05T06:20:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.161+0000 7fab16ae3700 1 --2- 172.21.15.16:0/715477167 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab1008fef0 0x7fab1008d000 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:16.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.161+0000 7fab16ae3700 1 --2- 172.21.15.16:0/715477167 >> v2:172.21.15.76:3300/0 conn(0x7fab1008c6d0 0x7fab1008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:16.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.162+0000 7fab16ae3700 1 -- 172.21.15.16:0/715477167 shutdown_connections 2019-12-05T06:20:16.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.162+0000 7fab16ae3700 1 -- 172.21.15.16:0/715477167 wait complete. 2019-12-05T06:20:16.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.162+0000 7fab16ae3700 1 -- 172.21.15.16:0/715477167 >> 172.21.15.16:0/715477167 conn(0x7fab10061940 msgr2=0x7fab10096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:16.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.168+0000 7fab16ae3700 1 Processor -- start 2019-12-05T06:20:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.168+0000 7fab16ae3700 1 -- start start 2019-12-05T06:20:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.170+0000 7fab16ae3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab10096e90 0x7fab1009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.170+0000 7fab16ae3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab10084f30 0x7fab1009b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:16.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.170+0000 7fab16ae3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab10084b90 con 0x7fab10097280 2019-12-05T06:20:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.170+0000 7fab16ae3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab1009a450 con 0x7fab10096e90 2019-12-05T06:20:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.170+0000 7fab16ae3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab10099a20 con 0x7fab10084f30 2019-12-05T06:20:16.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.170+0000 7fab15080700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab10096e90 0x7fab1009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.170+0000 7fab15881700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fab10097280 0x7fab100976c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36234/0 (socket says 172.21.15.16:36234) 2019-12-05T06:20:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.171+0000 7fab15881700 1 -- 172.21.15.16:0/3684545258 learned_addr learned my addr 172.21.15.16:0/3684545258 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.171+0000 7fab1487f700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab10084f30 0x7fab1009b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.171+0000 7fab15080700 1 -- 172.21.15.16:0/3684545258 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab10084f30 msgr2=0x7fab1009b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.171+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab10084f30 0x7fab1009b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.171+0000 7fab15080700 1 -- 172.21.15.16:0/3684545258 >> v1:172.21.15.16:6789/0 conn(0x7fab10097280 legacy=0x7fab100976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.171+0000 7fab0dffb700 1 -- 172.21.15.16:0/3684545258 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 343112998 0 0) 0x7fab08001310 con 0x7fab10097280 2019-12-05T06:20:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.171+0000 7fab15080700 1 -- 172.21.15.16:0/3684545258 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1009aae0 con 0x7fab10096e90 2019-12-05T06:20:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.171+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> v2:172.21.15.16:3300/0 conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.172+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> v2:172.21.15.16:3300/0 conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.172+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=READY pgs=568 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.172+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.172+0000 7fab0dffb700 1 -- 172.21.15.16:0/3684545258 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab00000cc0 con 0x7fab10096e90 2019-12-05T06:20:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.172+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.172+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.172+0000 7fab0dffb700 1 -- 172.21.15.16:0/3684545258 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab00001010 con 0x7fab10096e90 2019-12-05T06:20:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.172+0000 7fab16ae3700 1 -- 172.21.15.16:0/3684545258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1009aae0 con 0x7fab10096e90 2019-12-05T06:20:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.172+0000 7fab0dffb700 1 -- 172.21.15.16:0/3684545258 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab000016e0 con 0x7fab10096e90 2019-12-05T06:20:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.172+0000 7fab16ae3700 1 -- 172.21.15.16:0/3684545258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1009aae0 con 0x7fab10096e90 2019-12-05T06:20:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.173+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.174+0000 7fab0dffb700 1 -- 172.21.15.16:0/3684545258 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fab00001200 con 0x7fab10096e90 2019-12-05T06:20:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.174+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.175+0000 7fab0dffb700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaf4023630 0x7faaf4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.175+0000 7fab0dffb700 1 -- 172.21.15.16:0/3684545258 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fab00046050 con 0x7fab10096e90 2019-12-05T06:20:16.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.175+0000 7fab1487f700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaf4023630 0x7faaf4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:16.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.175+0000 7fab1487f700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaf4023630 0x7faaf4025ac0 crc :-1 s=READY pgs=252 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:16.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.186+0000 7fab16ae3700 1 -- 172.21.15.16:0/3684545258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab1009aae0 con 0x7fab10096e90 2019-12-05T06:20:16.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.194+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.194+0000 7fab0dffb700 1 -- 172.21.15.16:0/3684545258 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fab000095d0 con 0x7fab10096e90 2019-12-05T06:20:16.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.459+0000 7fab16ae3700 1 -- 172.21.15.16:0/3684545258 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab1009aae0 con 0x7fab10096e90 2019-12-05T06:20:16.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.460+0000 7fab15080700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=THROTTLE_DONE pgs=568 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:16.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.460+0000 7fab0dffb700 1 -- 172.21.15.16:0/3684545258 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fab00001200 con 0x7fab10096e90 2019-12-05T06:20:16.450 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2019-12-05T06:20:16.450 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:16.451 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5604,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:16.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.462+0000 7fab16ae3700 1 -- 172.21.15.16:0/3684545258 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaf4023630 msgr2=0x7faaf4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:16.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.462+0000 7fab16ae3700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaf4023630 0x7faaf4025ac0 crc :-1 s=READY pgs=252 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:16.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.462+0000 7fab16ae3700 1 -- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 msgr2=0x7fab1009b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.462+0000 7fab16ae3700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 0x7fab1009b3e0 secure :-1 s=READY pgs=568 cs=0 l=1 rx=0x7fab000106e0 tx=0x7fab00010540).stop 2019-12-05T06:20:16.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.463+0000 7fab16ae3700 1 -- 172.21.15.16:0/3684545258 shutdown_connections 2019-12-05T06:20:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.463+0000 7fab16ae3700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab10084f30 0x7fab1009b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.463+0000 7fab16ae3700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaf4023630 0x7faaf4025ac0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.463+0000 7fab16ae3700 1 --2- 172.21.15.16:0/3684545258 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab10096e90 0x7fab1009b3e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.463+0000 7fab16ae3700 1 -- 172.21.15.16:0/3684545258 shutdown_connections 2019-12-05T06:20:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.463+0000 7fab16ae3700 1 -- 172.21.15.16:0/3684545258 wait complete. 2019-12-05T06:20:16.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:16.463+0000 7fab16ae3700 1 -- 172.21.15.16:0/3684545258 >> 172.21.15.16:0/3684545258 conn(0x7fab100714f0 msgr2=0x7fab1009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:16.465 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:20:18.466 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.745+0000 7f2452397700 1 Processor -- start 2019-12-05T06:20:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.745+0000 7f2452397700 1 -- start start 2019-12-05T06:20:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.747+0000 7f2452397700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 0x7f244c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:18.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.747+0000 7f2452397700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244c0958e0 con 0x7f244c08fef0 2019-12-05T06:20:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.747+0000 7f2452397700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f244c0629e0 con 0x7f244c08f720 2019-12-05T06:20:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.747+0000 7f2452397700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c0a3f70 con 0x7f244c08c6d0 2019-12-05T06:20:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.748+0000 7f244bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f244c08fef0 0x7f244c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36242/0 (socket says 172.21.15.16:36242) 2019-12-05T06:20:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.748+0000 7f244bfff700 1 -- 172.21.15.16:0/1479655725 learned_addr learned my addr 172.21.15.16:0/1479655725 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.748+0000 7f2450934700 1 --2- 172.21.15.16:0/1479655725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 0x7f244c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:18.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.748+0000 7f244b7fe700 1 -- 172.21.15.16:0/1479655725 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946633883 0 0) 0x7f243c001150 con 0x7f244c08fef0 2019-12-05T06:20:18.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.748+0000 7f244b7fe700 1 -- 172.21.15.16:0/1479655725 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2438002e80 con 0x7f244c08fef0 2019-12-05T06:20:18.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.748+0000 7f244b7fe700 1 -- 172.21.15.16:0/1479655725 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294258869 0 0) 0x7f24340009b0 con 0x7f244c08f720 2019-12-05T06:20:18.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.748+0000 7f244b7fe700 1 -- 172.21.15.16:0/1479655725 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2438004170 con 0x7f244c08f720 2019-12-05T06:20:18.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f2450934700 1 -- 172.21.15.16:0/1479655725 >> v1:172.21.15.76:6789/0 conn(0x7f244c08f720 legacy=0x7f244c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:18.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f2450934700 1 -- 172.21.15.16:0/1479655725 >> v1:172.21.15.16:6789/0 conn(0x7f244c08fef0 legacy=0x7f244c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:18.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f2450934700 1 -- 172.21.15.16:0/1479655725 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c08fbc0 con 0x7f244c08c6d0 2019-12-05T06:20:18.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f2450934700 1 --2- 172.21.15.16:0/1479655725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 0x7f244c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f244400b2d0 tx=0x7f244400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f2450934700 1 --2- 172.21.15.16:0/1479655725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 0x7f244c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f244400b2d0 tx=0x7f244400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f2450934700 1 --2- 172.21.15.16:0/1479655725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 0x7f244c08cac0 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7f244400b2d0 tx=0x7f244400bae0).ready entity=mon.2 client_cookie=10b35f529da11821 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:18.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f2450934700 1 --2- 172.21.15.16:0/1479655725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 0x7f244c08cac0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f244400b2d0 tx=0x7f244400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f244b7fe700 1 -- 172.21.15.16:0/1479655725 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2444000ce0 con 0x7f244c08c6d0 2019-12-05T06:20:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f2450934700 1 --2- 172.21.15.16:0/1479655725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 0x7f244c08cac0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f244400b2d0 tx=0x7f244400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f2450934700 1 --2- 172.21.15.16:0/1479655725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 0x7f244c08cac0 secure :-1 s=THROTTLE_DONE pgs=142 cs=0 l=1 rx=0x7f244400b2d0 tx=0x7f244400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f244b7fe700 1 -- 172.21.15.16:0/1479655725 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2444009be0 con 0x7f244c08c6d0 2019-12-05T06:20:18.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.749+0000 7f244b7fe700 1 -- 172.21.15.16:0/1479655725 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f244400a2b0 con 0x7f244c08c6d0 2019-12-05T06:20:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.750+0000 7f2452397700 1 -- 172.21.15.16:0/1479655725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 msgr2=0x7f244c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.750+0000 7f2452397700 1 --2- 172.21.15.16:0/1479655725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 0x7f244c08cac0 secure :-1 s=READY pgs=142 cs=0 l=1 rx=0x7f244400b2d0 tx=0x7f244400bae0).stop 2019-12-05T06:20:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.751+0000 7f2452397700 1 -- 172.21.15.16:0/1479655725 shutdown_connections 2019-12-05T06:20:18.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.751+0000 7f2452397700 1 --2- 172.21.15.16:0/1479655725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c08c6d0 0x7f244c08cac0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.751+0000 7f2452397700 1 -- 172.21.15.16:0/1479655725 shutdown_connections 2019-12-05T06:20:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.751+0000 7f2452397700 1 -- 172.21.15.16:0/1479655725 wait complete. 2019-12-05T06:20:18.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.751+0000 7f2452397700 1 -- 172.21.15.16:0/1479655725 >> 172.21.15.16:0/1479655725 conn(0x7f244c061940 msgr2=0x7f244c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.757+0000 7f2452397700 1 Processor -- start 2019-12-05T06:20:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.757+0000 7f2452397700 1 -- start start 2019-12-05T06:20:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.759+0000 7f2452397700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f244c14d5d0 0x7f244c14d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:18.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.759+0000 7f2452397700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f244c09e2c0 0x7f244c09d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:18.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.759+0000 7f2452397700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c09d8e0 0x7f244c098180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:18.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.759+0000 7f2452397700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f244c12b3f0 con 0x7f244c09e2c0 2019-12-05T06:20:18.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.759+0000 7f2452397700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f244c12a9c0 con 0x7f244c14d5d0 2019-12-05T06:20:18.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.759+0000 7f2452397700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f244c12ac90 con 0x7f244c09d8e0 2019-12-05T06:20:18.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.759+0000 7f244bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f244c09e2c0 0x7f244c09d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:18.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.759+0000 7f244bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f244c09e2c0 0x7f244c09d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51080/0 (socket says 172.21.15.16:51080) 2019-12-05T06:20:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.759+0000 7f244bfff700 1 -- 172.21.15.16:0/532309608 learned_addr learned my addr 172.21.15.16:0/532309608 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.759+0000 7f2450934700 1 --2- 172.21.15.16:0/532309608 >> v2:172.21.15.76:3300/0 conn(0x7f244c14d5d0 0x7f244c14d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.760+0000 7f2451135700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c09d8e0 0x7f244c098180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.760+0000 7f244bfff700 1 -- 172.21.15.16:0/532309608 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c09d8e0 msgr2=0x7f244c098180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.760+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c09d8e0 0x7f244c098180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.760+0000 7f244bfff700 1 -- 172.21.15.16:0/532309608 >> v2:172.21.15.76:3300/0 conn(0x7f244c14d5d0 msgr2=0x7f244c14d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.760+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> v2:172.21.15.76:3300/0 conn(0x7f244c14d5d0 0x7f244c14d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.760+0000 7f244bfff700 1 -- 172.21.15.16:0/532309608 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c12ba80 con 0x7f244c09e2c0 2019-12-05T06:20:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.760+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> v2:172.21.15.16:3300/0 conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.760+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> v2:172.21.15.16:3300/0 conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.760+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=READY pgs=570 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.761+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.761+0000 7f24497fa700 1 -- 172.21.15.16:0/532309608 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f243c000c90 con 0x7f244c09e2c0 2019-12-05T06:20:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.761+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.761+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.761+0000 7f24497fa700 1 -- 172.21.15.16:0/532309608 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f243c0010c0 con 0x7f244c09e2c0 2019-12-05T06:20:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.761+0000 7f2452397700 1 -- 172.21.15.16:0/532309608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c12ba80 con 0x7f244c09e2c0 2019-12-05T06:20:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.761+0000 7f24497fa700 1 -- 172.21.15.16:0/532309608 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f243c001710 con 0x7f244c09e2c0 2019-12-05T06:20:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.761+0000 7f2452397700 1 -- 172.21.15.16:0/532309608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c12ba80 con 0x7f244c09e2c0 2019-12-05T06:20:18.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.762+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.763+0000 7f24497fa700 1 -- 172.21.15.16:0/532309608 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f243c000c90 con 0x7f244c09e2c0 2019-12-05T06:20:18.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.763+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.763+0000 7f24497fa700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2430023630 0x7f2430025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:18.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.763+0000 7f24497fa700 1 -- 172.21.15.16:0/532309608 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f243c045cd0 con 0x7f244c09e2c0 2019-12-05T06:20:18.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.763+0000 7f2450934700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2430023630 0x7f2430025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:18.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.764+0000 7f2450934700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2430023630 0x7f2430025ac0 crc :-1 s=READY pgs=253 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.776+0000 7f2452397700 1 -- 172.21.15.16:0/532309608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c12ba80 con 0x7f244c09e2c0 2019-12-05T06:20:18.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.783+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:18.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:18.783+0000 7f24497fa700 1 -- 172.21.15.16:0/532309608 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f243c013120 con 0x7f244c09e2c0 2019-12-05T06:20:19.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.029+0000 7f2452397700 1 -- 172.21.15.16:0/532309608 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f244c12ba80 con 0x7f244c09e2c0 2019-12-05T06:20:19.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.029+0000 7f244bfff700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=THROTTLE_DONE pgs=570 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:19.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.030+0000 7f24497fa700 1 -- 172.21.15.16:0/532309608 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f243c013120 con 0x7f244c09e2c0 2019-12-05T06:20:19.021 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2019-12-05T06:20:19.021 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:19.021 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5604,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:19.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.032+0000 7f2452397700 1 -- 172.21.15.16:0/532309608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2430023630 msgr2=0x7f2430025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:19.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.032+0000 7f2452397700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2430023630 0x7f2430025ac0 crc :-1 s=READY pgs=253 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:19.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.032+0000 7f2452397700 1 -- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 msgr2=0x7f244c09d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:19.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.032+0000 7f2452397700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 0x7f244c09d3a0 secure :-1 s=READY pgs=570 cs=0 l=1 rx=0x7f243c0103e0 tx=0x7f243c010260).stop 2019-12-05T06:20:19.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.032+0000 7f2452397700 1 -- 172.21.15.16:0/532309608 shutdown_connections 2019-12-05T06:20:19.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.032+0000 7f2452397700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f244c09d8e0 0x7f244c098180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:19.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.032+0000 7f2452397700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2430023630 0x7f2430025ac0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:19.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.032+0000 7f2452397700 1 --2- 172.21.15.16:0/532309608 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f244c09e2c0 0x7f244c09d3a0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.032+0000 7f2452397700 1 --2- 172.21.15.16:0/532309608 >> v2:172.21.15.76:3300/0 conn(0x7f244c14d5d0 0x7f244c14d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.032+0000 7f2452397700 1 -- 172.21.15.16:0/532309608 shutdown_connections 2019-12-05T06:20:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.033+0000 7f2452397700 1 -- 172.21.15.16:0/532309608 wait complete. 2019-12-05T06:20:19.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:19.033+0000 7f2452397700 1 -- 172.21.15.16:0/532309608 >> 172.21.15.16:0/532309608 conn(0x7f244c06d260 msgr2=0x7f244c09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:19.033 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:20:21.034 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:21.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.320+0000 7f515999f700 1 Processor -- start 2019-12-05T06:20:21.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.320+0000 7f515999f700 1 -- start start 2019-12-05T06:20:21.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.321+0000 7f515999f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f515408c6d0 0x7f515408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:21.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.321+0000 7f515999f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f515408fef0 0x7f515408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:21.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.322+0000 7f515999f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515408fb80 con 0x7f515408ef20 2019-12-05T06:20:21.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.322+0000 7f515999f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515408df90 con 0x7f515408fef0 2019-12-05T06:20:21.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.322+0000 7f515999f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51540a3f70 con 0x7f515408c6d0 2019-12-05T06:20:21.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.322+0000 7f5152ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f515408fef0 0x7f515408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:21.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.322+0000 7f5152ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f515408fef0 0x7f515408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51090/0 (socket says 172.21.15.16:51090) 2019-12-05T06:20:21.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.322+0000 7f5152ffd700 1 -- 172.21.15.16:0/3348096441 learned_addr learned my addr 172.21.15.16:0/3348096441 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:21.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.322+0000 7f5153fff700 1 --1- 172.21.15.16:0/3348096441 >> v1:172.21.15.16:6789/0 conn(0x7f515408ef20 0x7f515408f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36260/0 (socket says 172.21.15.16:36260) 2019-12-05T06:20:21.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.322+0000 7f51537fe700 1 --2- 172.21.15.16:0/3348096441 >> v2:172.21.15.76:3300/0 conn(0x7f515408c6d0 0x7f515408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:21.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f5152ffd700 1 -- 172.21.15.16:0/3348096441 >> v2:172.21.15.76:3300/0 conn(0x7f515408c6d0 msgr2=0x7f515408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:21.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f5152ffd700 1 --2- 172.21.15.16:0/3348096441 >> v2:172.21.15.76:3300/0 conn(0x7f515408c6d0 0x7f515408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:21.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f5152ffd700 1 -- 172.21.15.16:0/3348096441 >> v1:172.21.15.16:6789/0 conn(0x7f515408ef20 legacy=0x7f515408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:21.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f5152ffd700 1 -- 172.21.15.16:0/3348096441 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f515408d540 con 0x7f515408fef0 2019-12-05T06:20:21.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f5152ffd700 1 --2- 172.21.15.16:0/3348096441 >> v2:172.21.15.16:3300/0 conn(0x7f515408fef0 0x7f515408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f514c00f740 tx=0x7f514c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f5152ffd700 1 --2- 172.21.15.16:0/3348096441 >> v2:172.21.15.16:3300/0 conn(0x7f515408fef0 0x7f515408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f514c00f740 tx=0x7f514c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f5152ffd700 1 --2- 172.21.15.16:0/3348096441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f515408fef0 0x7f515408d000 secure :-1 s=READY pgs=572 cs=0 l=1 rx=0x7f514c00f740 tx=0x7f514c00f570).ready entity=mon.0 client_cookie=97a301bb3f03d9c9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:21.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f5152ffd700 1 --2- 172.21.15.16:0/3348096441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f515408fef0 0x7f515408d000 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7f514c00f740 tx=0x7f514c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f5152ffd700 1 --2- 172.21.15.16:0/3348096441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f515408fef0 0x7f515408d000 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7f514c00f740 tx=0x7f514c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f5152ffd700 1 --2- 172.21.15.16:0/3348096441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f515408fef0 0x7f515408d000 secure :-1 s=THROTTLE_DONE pgs=572 cs=0 l=1 rx=0x7f514c00f740 tx=0x7f514c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f51527fc700 1 -- 172.21.15.16:0/3348096441 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f514c000960 con 0x7f515408fef0 2019-12-05T06:20:21.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f51527fc700 1 -- 172.21.15.16:0/3348096441 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f514c000c90 con 0x7f515408fef0 2019-12-05T06:20:21.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.323+0000 7f51527fc700 1 -- 172.21.15.16:0/3348096441 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f514c0013a0 con 0x7f515408fef0 2019-12-05T06:20:21.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.324+0000 7f515999f700 1 -- 172.21.15.16:0/3348096441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f515408fef0 msgr2=0x7f515408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.324+0000 7f515999f700 1 --2- 172.21.15.16:0/3348096441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f515408fef0 0x7f515408d000 secure :-1 s=READY pgs=572 cs=0 l=1 rx=0x7f514c00f740 tx=0x7f514c00f570).stop 2019-12-05T06:20:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.325+0000 7f515999f700 1 -- 172.21.15.16:0/3348096441 shutdown_connections 2019-12-05T06:20:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.325+0000 7f515999f700 1 --2- 172.21.15.16:0/3348096441 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f515408fef0 0x7f515408d000 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.325+0000 7f515999f700 1 --2- 172.21.15.16:0/3348096441 >> v2:172.21.15.76:3300/0 conn(0x7f515408c6d0 0x7f515408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.325+0000 7f515999f700 1 -- 172.21.15.16:0/3348096441 shutdown_connections 2019-12-05T06:20:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.325+0000 7f515999f700 1 -- 172.21.15.16:0/3348096441 wait complete. 2019-12-05T06:20:21.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.325+0000 7f515999f700 1 -- 172.21.15.16:0/3348096441 >> 172.21.15.16:0/3348096441 conn(0x7f5154061940 msgr2=0x7f5154096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:21.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.331+0000 7f515999f700 1 Processor -- start 2019-12-05T06:20:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.331+0000 7f515999f700 1 -- start start 2019-12-05T06:20:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.333+0000 7f515999f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51541529e0 0x7f5154152dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.333+0000 7f515999f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5154089700 0x7f5154089bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.333+0000 7f515999f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f515409be80 con 0x7f515408a100 2019-12-05T06:20:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.333+0000 7f515999f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515409cff0 con 0x7f51541529e0 2019-12-05T06:20:21.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.333+0000 7f515999f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f515409c5c0 con 0x7f5154089700 2019-12-05T06:20:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.333+0000 7f51537fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51541529e0 0x7f5154152dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.333+0000 7f51537fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51541529e0 0x7f5154152dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51094/0 (socket says 172.21.15.16:51094) 2019-12-05T06:20:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.333+0000 7f51537fe700 1 -- 172.21.15.16:0/2044756416 learned_addr learned my addr 172.21.15.16:0/2044756416 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:21.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.333+0000 7f5152ffd700 1 --2- 172.21.15.16:0/2044756416 >> v2:172.21.15.76:3300/0 conn(0x7f5154089700 0x7f5154089bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.334+0000 7f51537fe700 1 -- 172.21.15.16:0/2044756416 >> v2:172.21.15.76:3300/0 conn(0x7f5154089700 msgr2=0x7f5154089bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.334+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> v2:172.21.15.76:3300/0 conn(0x7f5154089700 0x7f5154089bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.334+0000 7f51537fe700 1 -- 172.21.15.16:0/2044756416 >> v1:172.21.15.76:6789/0 conn(0x7f515408a100 legacy=0x7f515409d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.334+0000 7f51537fe700 1 -- 172.21.15.16:0/2044756416 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5154079ea0 con 0x7f51541529e0 2019-12-05T06:20:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.334+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> v2:172.21.15.16:3300/0 conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.334+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> v2:172.21.15.16:3300/0 conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.334+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=READY pgs=573 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.334+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.334+0000 7f514bfff700 1 -- 172.21.15.16:0/2044756416 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5144000cc0 con 0x7f51541529e0 2019-12-05T06:20:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.334+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.335+0000 7f514bfff700 1 -- 172.21.15.16:0/2044756416 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5144001200 con 0x7f51541529e0 2019-12-05T06:20:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.335+0000 7f515999f700 1 -- 172.21.15.16:0/2044756416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5154079ea0 con 0x7f51541529e0 2019-12-05T06:20:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.335+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.335+0000 7f514bfff700 1 -- 172.21.15.16:0/2044756416 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5144000cc0 con 0x7f51541529e0 2019-12-05T06:20:21.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.335+0000 7f515999f700 1 -- 172.21.15.16:0/2044756416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5154079ea0 con 0x7f51541529e0 2019-12-05T06:20:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.336+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.337+0000 7f514bfff700 1 -- 172.21.15.16:0/2044756416 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f51440013f0 con 0x7f51541529e0 2019-12-05T06:20:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.337+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.337+0000 7f514bfff700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f512c023630 0x7f512c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.337+0000 7f514bfff700 1 -- 172.21.15.16:0/2044756416 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f5144046190 con 0x7f51541529e0 2019-12-05T06:20:21.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.337+0000 7f5152ffd700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f512c023630 0x7f512c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:21.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.338+0000 7f5152ffd700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f512c023630 0x7f512c025ac0 crc :-1 s=READY pgs=254 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:21.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.351+0000 7f515999f700 1 -- 172.21.15.16:0/2044756416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f515409e680 con 0x7f51541529e0 2019-12-05T06:20:21.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.360+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.360+0000 7f514bfff700 1 -- 172.21.15.16:0/2044756416 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f51440095d0 con 0x7f51541529e0 2019-12-05T06:20:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.608+0000 7f515999f700 1 -- 172.21.15.16:0/2044756416 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f515409e680 con 0x7f51541529e0 2019-12-05T06:20:21.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.608+0000 7f51537fe700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=THROTTLE_DONE pgs=573 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:21.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.608+0000 7f514bfff700 1 -- 172.21.15.16:0/2044756416 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f51440013f0 con 0x7f51541529e0 2019-12-05T06:20:21.599 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2019-12-05T06:20:21.599 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:21.599 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5604,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.611+0000 7f515999f700 1 -- 172.21.15.16:0/2044756416 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f512c023630 msgr2=0x7f512c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.611+0000 7f515999f700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f512c023630 0x7f512c025ac0 crc :-1 s=READY pgs=254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.611+0000 7f515999f700 1 -- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 msgr2=0x7f5154152dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.611+0000 7f515999f700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 0x7f5154152dd0 secure :-1 s=READY pgs=573 cs=0 l=1 rx=0x7f51440107d0 tx=0x7f5144010600).stop 2019-12-05T06:20:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.611+0000 7f515999f700 1 -- 172.21.15.16:0/2044756416 shutdown_connections 2019-12-05T06:20:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.611+0000 7f515999f700 1 --2- 172.21.15.16:0/2044756416 >> v2:172.21.15.76:3300/0 conn(0x7f5154089700 0x7f5154089bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.611+0000 7f515999f700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f512c023630 0x7f512c025ac0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.611+0000 7f515999f700 1 --2- 172.21.15.16:0/2044756416 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51541529e0 0x7f5154152dd0 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.611+0000 7f515999f700 1 -- 172.21.15.16:0/2044756416 shutdown_connections 2019-12-05T06:20:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.611+0000 7f515999f700 1 -- 172.21.15.16:0/2044756416 wait complete. 2019-12-05T06:20:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:21.612+0000 7f515999f700 1 -- 172.21.15.16:0/2044756416 >> 172.21.15.16:0/2044756416 conn(0x7f5154098480 msgr2=0x7f5154071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:21.616 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:20:23.617 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.878+0000 7f679c744700 1 Processor -- start 2019-12-05T06:20:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.878+0000 7f679c744700 1 -- start start 2019-12-05T06:20:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.881+0000 7f679c744700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f679408fef0 0x7f679408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.881+0000 7f679c744700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67940958e0 con 0x7f679408d2c0 2019-12-05T06:20:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.881+0000 7f679c744700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67940629e0 con 0x7f679408c6d0 2019-12-05T06:20:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.881+0000 7f679c744700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679408df90 con 0x7f679408fef0 2019-12-05T06:20:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.881+0000 7f679a4e0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f679408d2c0 0x7f679408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36274/0 (socket says 172.21.15.16:36274) 2019-12-05T06:20:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.881+0000 7f679b4e2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f679408fef0 0x7f679408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.881+0000 7f679a4e0700 1 -- 172.21.15.16:0/3489830965 learned_addr learned my addr 172.21.15.16:0/3489830965 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.882+0000 7f6799cdf700 1 -- 172.21.15.16:0/3489830965 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81721124 0 0) 0x7f6784001150 con 0x7f679408d2c0 2019-12-05T06:20:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.882+0000 7f6799cdf700 1 -- 172.21.15.16:0/3489830965 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6780002e80 con 0x7f679408d2c0 2019-12-05T06:20:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.882+0000 7f6799cdf700 1 -- 172.21.15.16:0/3489830965 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3453810498 0 0) 0x7f6790000fb0 con 0x7f679408c6d0 2019-12-05T06:20:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.882+0000 7f679b4e2700 1 -- 172.21.15.16:0/3489830965 >> v1:172.21.15.76:6789/0 conn(0x7f679408c6d0 legacy=0x7f679408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.882+0000 7f679b4e2700 1 -- 172.21.15.16:0/3489830965 >> v1:172.21.15.16:6789/0 conn(0x7f679408d2c0 legacy=0x7f679408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.882+0000 7f679b4e2700 1 -- 172.21.15.16:0/3489830965 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679408e620 con 0x7f679408fef0 2019-12-05T06:20:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.882+0000 7f679b4e2700 1 --2- 172.21.15.16:0/3489830965 >> v2:172.21.15.16:3300/0 conn(0x7f679408fef0 0x7f679408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f678c00b2d0 tx=0x7f678c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.883+0000 7f679b4e2700 1 --2- 172.21.15.16:0/3489830965 >> v2:172.21.15.16:3300/0 conn(0x7f679408fef0 0x7f679408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f678c00b2d0 tx=0x7f678c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.883+0000 7f679b4e2700 1 --2- 172.21.15.16:0/3489830965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679408fef0 0x7f679408f720 secure :-1 s=READY pgs=575 cs=0 l=1 rx=0x7f678c00b2d0 tx=0x7f678c00bae0).ready entity=mon.0 client_cookie=a2184a71c9204cc4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.883+0000 7f679b4e2700 1 --2- 172.21.15.16:0/3489830965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679408fef0 0x7f679408f720 secure :-1 s=THROTTLE_DONE pgs=575 cs=0 l=1 rx=0x7f678c00b2d0 tx=0x7f678c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.883+0000 7f6799cdf700 1 -- 172.21.15.16:0/3489830965 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f678c009f20 con 0x7f679408fef0 2019-12-05T06:20:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.883+0000 7f679b4e2700 1 --2- 172.21.15.16:0/3489830965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679408fef0 0x7f679408f720 secure :-1 s=THROTTLE_DONE pgs=575 cs=0 l=1 rx=0x7f678c00b2d0 tx=0x7f678c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.883+0000 7f679b4e2700 1 --2- 172.21.15.16:0/3489830965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679408fef0 0x7f679408f720 secure :-1 s=THROTTLE_DONE pgs=575 cs=0 l=1 rx=0x7f678c00b2d0 tx=0x7f678c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.883+0000 7f6799cdf700 1 -- 172.21.15.16:0/3489830965 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f678c000df0 con 0x7f679408fef0 2019-12-05T06:20:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.883+0000 7f6799cdf700 1 -- 172.21.15.16:0/3489830965 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f678c00a500 con 0x7f679408fef0 2019-12-05T06:20:23.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.883+0000 7f679c744700 1 -- 172.21.15.16:0/3489830965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679408fef0 msgr2=0x7f679408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:23.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.883+0000 7f679c744700 1 --2- 172.21.15.16:0/3489830965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679408fef0 0x7f679408f720 secure :-1 s=READY pgs=575 cs=0 l=1 rx=0x7f678c00b2d0 tx=0x7f678c00bae0).stop 2019-12-05T06:20:23.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.885+0000 7f679c744700 1 -- 172.21.15.16:0/3489830965 shutdown_connections 2019-12-05T06:20:23.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.885+0000 7f679c744700 1 --2- 172.21.15.16:0/3489830965 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679408fef0 0x7f679408f720 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:23.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.885+0000 7f679c744700 1 -- 172.21.15.16:0/3489830965 shutdown_connections 2019-12-05T06:20:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.885+0000 7f679c744700 1 -- 172.21.15.16:0/3489830965 wait complete. 2019-12-05T06:20:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.885+0000 7f679c744700 1 -- 172.21.15.16:0/3489830965 >> 172.21.15.16:0/3489830965 conn(0x7f6794061940 msgr2=0x7f6794096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:23.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.891+0000 7f679c744700 1 Processor -- start 2019-12-05T06:20:23.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.892+0000 7f679c744700 1 -- start start 2019-12-05T06:20:23.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.893+0000 7f679c744700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f679409ae80 0x7f6794133220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:23.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.893+0000 7f679c744700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6794099eb0 0x7f679409a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:23.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.893+0000 7f679c744700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679406e7c0 con 0x7f6794132630 2019-12-05T06:20:23.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.893+0000 7f679c744700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6794098f20 con 0x7f679409ae80 2019-12-05T06:20:23.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.893+0000 7f679c744700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6794134390 con 0x7f6794099eb0 2019-12-05T06:20:23.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.894+0000 7f679ace1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6794132630 0x7f6794132a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36278/0 (socket says 172.21.15.16:36278) 2019-12-05T06:20:23.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.894+0000 7f679ace1700 1 -- 172.21.15.16:0/962529580 learned_addr learned my addr 172.21.15.16:0/962529580 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:23.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.894+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> v2:172.21.15.76:3300/0 conn(0x7f679409ae80 0x7f6794133220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:23.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.894+0000 7f679b4e2700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6794099eb0 0x7f679409a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:23.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.894+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 75968749 0 0) 0x7f6790001100 con 0x7f6794132630 2019-12-05T06:20:23.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.894+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f677c002e80 con 0x7f6794132630 2019-12-05T06:20:23.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2675007418 0 0) 0x7f67900011f0 con 0x7f6794132630 2019-12-05T06:20:23.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f677c0032e0 con 0x7f6794132630 2019-12-05T06:20:23.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f67900017c0 con 0x7f6794132630 2019-12-05T06:20:23.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679a4e0700 1 -- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6794099eb0 msgr2=0x7f679409a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:23.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6794099eb0 0x7f679409a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:23.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679a4e0700 1 -- 172.21.15.16:0/962529580 >> v1:172.21.15.16:6789/0 conn(0x7f6794132630 legacy=0x7f6794132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:23.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679a4e0700 1 -- 172.21.15.16:0/962529580 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67940995b0 con 0x7f679409ae80 2019-12-05T06:20:23.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> v2:172.21.15.76:3300/0 conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> v2:172.21.15.76:3300/0 conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=READY pgs=423 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679b4e2700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6794099eb0 0x7f679409a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=THROTTLE_DONE pgs=423 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6784000c90 con 0x7f679409ae80 2019-12-05T06:20:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=THROTTLE_DONE pgs=423 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6784001380 con 0x7f679409ae80 2019-12-05T06:20:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=THROTTLE_DONE pgs=423 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6784001570 con 0x7f679409ae80 2019-12-05T06:20:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.895+0000 7f679c744700 1 -- 172.21.15.16:0/962529580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67940995b0 con 0x7f679409ae80 2019-12-05T06:20:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.896+0000 7f679c744700 1 -- 172.21.15.16:0/962529580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67940995b0 con 0x7f679409ae80 2019-12-05T06:20:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.896+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=THROTTLE_DONE pgs=423 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.897+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=THROTTLE_DONE pgs=423 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.897+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6784000c90 con 0x7f679409ae80 2019-12-05T06:20:23.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.897+0000 7f678b7fe700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f677c023670 0x7f677c025b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:23.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.897+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f6784045f40 con 0x7f679409ae80 2019-12-05T06:20:23.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.898+0000 7f679ace1700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f677c023670 0x7f677c025b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:23.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.898+0000 7f679ace1700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f677c023670 0x7f677c025b00 crc :-1 s=READY pgs=255 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:23.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.910+0000 7f679c744700 1 -- 172.21.15.16:0/962529580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67940995b0 con 0x7f679409ae80 2019-12-05T06:20:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.920+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=THROTTLE_DONE pgs=423 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:23.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:23.920+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6784000ea0 con 0x7f679409ae80 2019-12-05T06:20:24.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.191+0000 7f679c744700 1 -- 172.21.15.16:0/962529580 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67940995b0 con 0x7f679409ae80 2019-12-05T06:20:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.193+0000 7f679a4e0700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=THROTTLE_DONE pgs=423 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:24.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.193+0000 7f678b7fe700 1 -- 172.21.15.16:0/962529580 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f6784000c90 con 0x7f679409ae80 2019-12-05T06:20:24.185 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:24.185 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5604,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:24.186 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2019-12-05T06:20:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 -- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f677c023670 msgr2=0x7f677c025b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f677c023670 0x7f677c025b00 crc :-1 s=READY pgs=255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 -- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 msgr2=0x7f6794133220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 0x7f6794133220 secure :-1 s=READY pgs=423 cs=0 l=1 rx=0x7f6784010380 tx=0x7f6784010720).stop 2019-12-05T06:20:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 -- 172.21.15.16:0/962529580 shutdown_connections 2019-12-05T06:20:24.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f677c023670 0x7f677c025b00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f679409ae80 0x7f6794133220 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 --2- 172.21.15.16:0/962529580 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6794099eb0 0x7f679409a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 -- 172.21.15.16:0/962529580 shutdown_connections 2019-12-05T06:20:24.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 -- 172.21.15.16:0/962529580 wait complete. 2019-12-05T06:20:24.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:24.197+0000 7f679c744700 1 -- 172.21.15.16:0/962529580 >> 172.21.15.16:0/962529580 conn(0x7f67940714f0 msgr2=0x7f679409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:24.198 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:20:26.200 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:26.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.492+0000 7f553d4c4700 1 Processor -- start 2019-12-05T06:20:26.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.492+0000 7f553d4c4700 1 -- start start 2019-12-05T06:20:26.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.494+0000 7f553d4c4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f553808c6d0 0x7f553808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:26.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.494+0000 7f553d4c4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f553808fef0 0x7f553808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:26.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.494+0000 7f553d4c4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f553808ef20 0x7f553808f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:26.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.494+0000 7f553d4c4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553808df90 con 0x7f553808fef0 2019-12-05T06:20:26.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.494+0000 7f553d4c4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55380a3f70 con 0x7f553808c6d0 2019-12-05T06:20:26.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.494+0000 7f553d4c4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f553808d830 con 0x7f553808ef20 2019-12-05T06:20:26.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f5536ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f553808fef0 0x7f553808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:26.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f5536ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f553808fef0 0x7f553808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51122/0 (socket says 172.21.15.16:51122) 2019-12-05T06:20:26.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f55377fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f553808c6d0 0x7f553808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:26.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f5536ffd700 1 -- 172.21.15.16:0/1476725543 learned_addr learned my addr 172.21.15.16:0/1476725543 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:26.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f5537fff700 1 --2- 172.21.15.16:0/1476725543 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f553808ef20 0x7f553808f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:26.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f5536ffd700 1 -- 172.21.15.16:0/1476725543 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f553808ef20 msgr2=0x7f553808f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:26.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f5536ffd700 1 --2- 172.21.15.16:0/1476725543 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f553808ef20 0x7f553808f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:26.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f5536ffd700 1 -- 172.21.15.16:0/1476725543 >> v2:172.21.15.76:3300/0 conn(0x7f553808c6d0 msgr2=0x7f553808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:26.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f5536ffd700 1 --2- 172.21.15.16:0/1476725543 >> v2:172.21.15.76:3300/0 conn(0x7f553808c6d0 0x7f553808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:26.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f5536ffd700 1 -- 172.21.15.16:0/1476725543 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f553808d540 con 0x7f553808fef0 2019-12-05T06:20:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.495+0000 7f55377fe700 1 --2- 172.21.15.16:0/1476725543 >> v2:172.21.15.76:3300/0 conn(0x7f553808c6d0 0x7f553808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.496+0000 7f5536ffd700 1 --2- 172.21.15.16:0/1476725543 >> v2:172.21.15.16:3300/0 conn(0x7f553808fef0 0x7f553808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f552000f6c0 tx=0x7f552000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.496+0000 7f5536ffd700 1 --2- 172.21.15.16:0/1476725543 >> v2:172.21.15.16:3300/0 conn(0x7f553808fef0 0x7f553808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f552000f6c0 tx=0x7f552000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.496+0000 7f5536ffd700 1 --2- 172.21.15.16:0/1476725543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553808fef0 0x7f553808d000 secure :-1 s=READY pgs=577 cs=0 l=1 rx=0x7f552000f6c0 tx=0x7f552000fa60).ready entity=mon.0 client_cookie=483500c1feb6cebe server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.496+0000 7f5536ffd700 1 --2- 172.21.15.16:0/1476725543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553808fef0 0x7f553808d000 secure :-1 s=THROTTLE_DONE pgs=577 cs=0 l=1 rx=0x7f552000f6c0 tx=0x7f552000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.496+0000 7f5536ffd700 1 --2- 172.21.15.16:0/1476725543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553808fef0 0x7f553808d000 secure :-1 s=THROTTLE_DONE pgs=577 cs=0 l=1 rx=0x7f552000f6c0 tx=0x7f552000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.496+0000 7f5536ffd700 1 --2- 172.21.15.16:0/1476725543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553808fef0 0x7f553808d000 secure :-1 s=THROTTLE_DONE pgs=577 cs=0 l=1 rx=0x7f552000f6c0 tx=0x7f552000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.496+0000 7f55367fc700 1 -- 172.21.15.16:0/1476725543 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5520000960 con 0x7f553808fef0 2019-12-05T06:20:26.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.497+0000 7f55367fc700 1 -- 172.21.15.16:0/1476725543 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5520000c90 con 0x7f553808fef0 2019-12-05T06:20:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.497+0000 7f55367fc700 1 -- 172.21.15.16:0/1476725543 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5520001440 con 0x7f553808fef0 2019-12-05T06:20:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.497+0000 7f553d4c4700 1 -- 172.21.15.16:0/1476725543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553808fef0 msgr2=0x7f553808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.497+0000 7f553d4c4700 1 --2- 172.21.15.16:0/1476725543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553808fef0 0x7f553808d000 secure :-1 s=READY pgs=577 cs=0 l=1 rx=0x7f552000f6c0 tx=0x7f552000fa60).stop 2019-12-05T06:20:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.498+0000 7f553d4c4700 1 -- 172.21.15.16:0/1476725543 shutdown_connections 2019-12-05T06:20:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.498+0000 7f553d4c4700 1 --2- 172.21.15.16:0/1476725543 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f553808ef20 0x7f553808f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:26.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.498+0000 7f553d4c4700 1 --2- 172.21.15.16:0/1476725543 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553808fef0 0x7f553808d000 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.498+0000 7f553d4c4700 1 --2- 172.21.15.16:0/1476725543 >> v2:172.21.15.76:3300/0 conn(0x7f553808c6d0 0x7f553808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.498+0000 7f553d4c4700 1 -- 172.21.15.16:0/1476725543 shutdown_connections 2019-12-05T06:20:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.498+0000 7f553d4c4700 1 -- 172.21.15.16:0/1476725543 wait complete. 2019-12-05T06:20:26.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.498+0000 7f553d4c4700 1 -- 172.21.15.16:0/1476725543 >> 172.21.15.16:0/1476725543 conn(0x7f5538061940 msgr2=0x7f5538096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.504+0000 7f553d4c4700 1 Processor -- start 2019-12-05T06:20:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.504+0000 7f553d4c4700 1 -- start start 2019-12-05T06:20:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.506+0000 7f553d4c4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f553806df10 0x7f5538098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.506+0000 7f553d4c4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f55380992d0 0x7f5538098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:26.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.506+0000 7f553d4c4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f553806e300 con 0x7f5538098d80 2019-12-05T06:20:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.506+0000 7f553d4c4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5538097370 con 0x7f553806df10 2019-12-05T06:20:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.506+0000 7f553d4c4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f553808ad00 con 0x7f55380992d0 2019-12-05T06:20:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.506+0000 7f55377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f553806df10 0x7f5538098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.506+0000 7f55377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f553806df10 0x7f5538098300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51126/0 (socket says 172.21.15.16:51126) 2019-12-05T06:20:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.506+0000 7f55377fe700 1 -- 172.21.15.16:0/383365482 learned_addr learned my addr 172.21.15.16:0/383365482 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:26.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.506+0000 7f5536ffd700 1 --2- 172.21.15.16:0/383365482 >> v2:172.21.15.76:3300/0 conn(0x7f55380992d0 0x7f5538098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.507+0000 7f55377fe700 1 -- 172.21.15.16:0/383365482 >> v2:172.21.15.76:3300/0 conn(0x7f55380992d0 msgr2=0x7f5538098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.507+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> v2:172.21.15.76:3300/0 conn(0x7f55380992d0 0x7f5538098840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.507+0000 7f55377fe700 1 -- 172.21.15.16:0/383365482 >> v1:172.21.15.76:6789/0 conn(0x7f5538098d80 legacy=0x7f5538088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.507+0000 7f55377fe700 1 -- 172.21.15.16:0/383365482 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5538097a00 con 0x7f553806df10 2019-12-05T06:20:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.507+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> v2:172.21.15.16:3300/0 conn(0x7f553806df10 0x7f5538098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.507+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> v2:172.21.15.16:3300/0 conn(0x7f553806df10 0x7f5538098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.507+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 0x7f5538098300 secure :-1 s=READY pgs=578 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.508+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 0x7f5538098300 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.508+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 0x7f5538098300 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.508+0000 7f552ffff700 1 -- 172.21.15.16:0/383365482 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5530000cc0 con 0x7f553806df10 2019-12-05T06:20:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.508+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 0x7f5538098300 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.508+0000 7f552ffff700 1 -- 172.21.15.16:0/383365482 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5530009900 con 0x7f553806df10 2019-12-05T06:20:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.508+0000 7f553d4c4700 1 -- 172.21.15.16:0/383365482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5538097a00 con 0x7f553806df10 2019-12-05T06:20:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.508+0000 7f552ffff700 1 -- 172.21.15.16:0/383365482 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5530009f50 con 0x7f553806df10 2019-12-05T06:20:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.508+0000 7f553d4c4700 1 -- 172.21.15.16:0/383365482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5538097a00 con 0x7f553806df10 2019-12-05T06:20:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.509+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 0x7f5538098300 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.510+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 0x7f5538098300 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.510+0000 7f552ffff700 1 -- 172.21.15.16:0/383365482 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5530009af0 con 0x7f553806df10 2019-12-05T06:20:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.510+0000 7f552ffff700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f551c023630 0x7f551c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.511+0000 7f552ffff700 1 -- 172.21.15.16:0/383365482 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f55300461b0 con 0x7f553806df10 2019-12-05T06:20:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.511+0000 7f5536ffd700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f551c023630 0x7f551c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:26.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.511+0000 7f5536ffd700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f551c023630 0x7f551c025ac0 crc :-1 s=READY pgs=256 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:26.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.524+0000 7f553d4c4700 1 -- 172.21.15.16:0/383365482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5538097a00 con 0x7f553806df10 2019-12-05T06:20:26.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.533+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 0x7f5538098300 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.533+0000 7f552ffff700 1 -- 172.21.15.16:0/383365482 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f553000a910 con 0x7f553806df10 2019-12-05T06:20:26.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.804+0000 7f553d4c4700 1 -- 172.21.15.16:0/383365482 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5538097a00 con 0x7f553806df10 2019-12-05T06:20:26.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.805+0000 7f55377fe700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 0x7f5538098300 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:26.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.805+0000 7f552ffff700 1 -- 172.21.15.16:0/383365482 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f5530009af0 con 0x7f553806df10 2019-12-05T06:20:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2019-12-05T06:20:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.808+0000 7f553d4c4700 1 -- 172.21.15.16:0/383365482 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f551c023630 msgr2=0x7f551c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.808+0000 7f553d4c4700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f551c023630 0x7f551c025ac0 crc :-1 s=READY pgs=256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.808+0000 7f553d4c4700 1 -- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 msgr2=0x7f5538098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.808+0000 7f553d4c4700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 0x7f5538098300 secure :-1 s=READY pgs=578 cs=0 l=1 rx=0x7f5530010740 tx=0x7f5530010ae0).stop 2019-12-05T06:20:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.808+0000 7f553d4c4700 1 -- 172.21.15.16:0/383365482 shutdown_connections 2019-12-05T06:20:26.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.808+0000 7f553d4c4700 1 --2- 172.21.15.16:0/383365482 >> v2:172.21.15.76:3300/0 conn(0x7f55380992d0 0x7f5538098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.808+0000 7f553d4c4700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f551c023630 0x7f551c025ac0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.808+0000 7f553d4c4700 1 --2- 172.21.15.16:0/383365482 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f553806df10 0x7f5538098300 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.808+0000 7f553d4c4700 1 -- 172.21.15.16:0/383365482 shutdown_connections 2019-12-05T06:20:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.808+0000 7f553d4c4700 1 -- 172.21.15.16:0/383365482 wait complete. 2019-12-05T06:20:26.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:26.809+0000 7f553d4c4700 1 -- 172.21.15.16:0/383365482 >> 172.21.15.16:0/383365482 conn(0x7f55380714f0 msgr2=0x7f553809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:26.805 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:26.805 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5604,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:26.813 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:20:28.815 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:29.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.078+0000 7fb58942d700 1 Processor -- start 2019-12-05T06:20:29.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.078+0000 7fb58942d700 1 -- start start 2019-12-05T06:20:29.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.079+0000 7fb58942d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb58408c6d0 0x7fb58408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:29.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.079+0000 7fb58942d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb58408f720 0x7fb58408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:29.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.079+0000 7fb58942d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb58408fb80 con 0x7fb58408fef0 2019-12-05T06:20:29.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.079+0000 7fb58942d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb58408df90 con 0x7fb58408f720 2019-12-05T06:20:29.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.080+0000 7fb58942d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5840a3f70 con 0x7fb58408c6d0 2019-12-05T06:20:29.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.080+0000 7fb583fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb58408f720 0x7fb58408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:29.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.080+0000 7fb583fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb58408f720 0x7fb58408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51138/0 (socket says 172.21.15.16:51138) 2019-12-05T06:20:29.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.080+0000 7fb583fff700 1 -- 172.21.15.16:0/232768514 learned_addr learned my addr 172.21.15.16:0/232768514 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.080+0000 7fb5837fe700 1 --2- 172.21.15.16:0/232768514 >> v2:172.21.15.76:3300/0 conn(0x7fb58408c6d0 0x7fb58408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.080+0000 7fb583fff700 1 -- 172.21.15.16:0/232768514 >> v2:172.21.15.76:3300/0 conn(0x7fb58408c6d0 msgr2=0x7fb58408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.080+0000 7fb583fff700 1 --2- 172.21.15.16:0/232768514 >> v2:172.21.15.76:3300/0 conn(0x7fb58408c6d0 0x7fb58408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.080+0000 7fb583fff700 1 -- 172.21.15.16:0/232768514 >> v1:172.21.15.76:6789/0 conn(0x7fb58408fef0 legacy=0x7fb58408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb583fff700 1 -- 172.21.15.16:0/232768514 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58408d540 con 0x7fb58408f720 2019-12-05T06:20:29.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.080+0000 7fb5827fc700 1 -- 172.21.15.16:0/232768514 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025759588 0 0) 0x7fb574001080 con 0x7fb58408fef0 2019-12-05T06:20:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb583fff700 1 --2- 172.21.15.16:0/232768514 >> v2:172.21.15.16:3300/0 conn(0x7fb58408f720 0x7fb58408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb56c00f6b0 tx=0x7fb56c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb583fff700 1 --2- 172.21.15.16:0/232768514 >> v2:172.21.15.16:3300/0 conn(0x7fb58408f720 0x7fb58408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb56c00f6b0 tx=0x7fb56c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb583fff700 1 --2- 172.21.15.16:0/232768514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb58408f720 0x7fb58408d000 secure :-1 s=READY pgs=579 cs=0 l=1 rx=0x7fb56c00f6b0 tx=0x7fb56c00fa50).ready entity=mon.0 client_cookie=41e28104f37fe2fa server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb583fff700 1 --2- 172.21.15.16:0/232768514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb58408f720 0x7fb58408d000 secure :-1 s=THROTTLE_DONE pgs=579 cs=0 l=1 rx=0x7fb56c00f6b0 tx=0x7fb56c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb583fff700 1 --2- 172.21.15.16:0/232768514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb58408f720 0x7fb58408d000 secure :-1 s=THROTTLE_DONE pgs=579 cs=0 l=1 rx=0x7fb56c00f6b0 tx=0x7fb56c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb5827fc700 1 -- 172.21.15.16:0/232768514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb56c000a70 con 0x7fb58408f720 2019-12-05T06:20:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb583fff700 1 --2- 172.21.15.16:0/232768514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb58408f720 0x7fb58408d000 secure :-1 s=THROTTLE_DONE pgs=579 cs=0 l=1 rx=0x7fb56c00f6b0 tx=0x7fb56c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb5827fc700 1 -- 172.21.15.16:0/232768514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56c0010e0 con 0x7fb58408f720 2019-12-05T06:20:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb5827fc700 1 -- 172.21.15.16:0/232768514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb56c0017b0 con 0x7fb58408f720 2019-12-05T06:20:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb58942d700 1 -- 172.21.15.16:0/232768514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb58408f720 msgr2=0x7fb58408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.081+0000 7fb58942d700 1 --2- 172.21.15.16:0/232768514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb58408f720 0x7fb58408d000 secure :-1 s=READY pgs=579 cs=0 l=1 rx=0x7fb56c00f6b0 tx=0x7fb56c00fa50).stop 2019-12-05T06:20:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.083+0000 7fb58942d700 1 -- 172.21.15.16:0/232768514 shutdown_connections 2019-12-05T06:20:29.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.083+0000 7fb58942d700 1 --2- 172.21.15.16:0/232768514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb58408f720 0x7fb58408d000 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:29.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.083+0000 7fb58942d700 1 --2- 172.21.15.16:0/232768514 >> v2:172.21.15.76:3300/0 conn(0x7fb58408c6d0 0x7fb58408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:29.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.083+0000 7fb58942d700 1 -- 172.21.15.16:0/232768514 shutdown_connections 2019-12-05T06:20:29.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.083+0000 7fb58942d700 1 -- 172.21.15.16:0/232768514 wait complete. 2019-12-05T06:20:29.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.083+0000 7fb58942d700 1 -- 172.21.15.16:0/232768514 >> 172.21.15.16:0/232768514 conn(0x7fb584061940 msgr2=0x7fb584096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.089+0000 7fb58942d700 1 Processor -- start 2019-12-05T06:20:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.090+0000 7fb58942d700 1 -- start start 2019-12-05T06:20:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.091+0000 7fb58942d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb58406df10 0x7fb584098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:29.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.091+0000 7fb58942d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5840992d0 0x7fb584098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.092+0000 7fb58942d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb584098d80 0x7fb584088fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:29.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.092+0000 7fb58942d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb58408ad00 con 0x7fb5840992d0 2019-12-05T06:20:29.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.092+0000 7fb58942d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb584096c10 con 0x7fb584098d80 2019-12-05T06:20:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.092+0000 7fb58942d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb584097a00 con 0x7fb58406df10 2019-12-05T06:20:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.092+0000 7fb582ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5840992d0 0x7fb584098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.092+0000 7fb582ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5840992d0 0x7fb584098840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51142/0 (socket says 172.21.15.16:51142) 2019-12-05T06:20:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.092+0000 7fb582ffd700 1 -- 172.21.15.16:0/2614306231 learned_addr learned my addr 172.21.15.16:0/2614306231 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.092+0000 7fb5837fe700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb58406df10 0x7fb584098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.092+0000 7fb583fff700 1 --2- 172.21.15.16:0/2614306231 >> v2:172.21.15.76:3300/0 conn(0x7fb584098d80 0x7fb584088fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:29.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 -- 172.21.15.16:0/2614306231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb58406df10 msgr2=0x7fb584098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb58406df10 0x7fb584098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 -- 172.21.15.16:0/2614306231 >> v2:172.21.15.76:3300/0 conn(0x7fb584098d80 msgr2=0x7fb584088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> v2:172.21.15.76:3300/0 conn(0x7fb584098d80 0x7fb584088fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 -- 172.21.15.16:0/2614306231 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58406e300 con 0x7fb5840992d0 2019-12-05T06:20:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> v2:172.21.15.16:3300/0 conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> v2:172.21.15.16:3300/0 conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=READY pgs=580 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb57bfff700 1 -- 172.21.15.16:0/2614306231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb574000b50 con 0x7fb5840992d0 2019-12-05T06:20:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb57bfff700 1 -- 172.21.15.16:0/2614306231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb574009a30 con 0x7fb5840992d0 2019-12-05T06:20:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb57bfff700 1 -- 172.21.15.16:0/2614306231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb57400a100 con 0x7fb5840992d0 2019-12-05T06:20:29.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.093+0000 7fb58942d700 1 -- 172.21.15.16:0/2614306231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb58406e300 con 0x7fb5840992d0 2019-12-05T06:20:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.094+0000 7fb58942d700 1 -- 172.21.15.16:0/2614306231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb58406e300 con 0x7fb5840992d0 2019-12-05T06:20:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.094+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.095+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.095+0000 7fb57bfff700 1 -- 172.21.15.16:0/2614306231 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb574009630 con 0x7fb5840992d0 2019-12-05T06:20:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.096+0000 7fb57bfff700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb55c023630 0x7fb55c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.096+0000 7fb57bfff700 1 -- 172.21.15.16:0/2614306231 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7fb574045d20 con 0x7fb5840992d0 2019-12-05T06:20:29.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.096+0000 7fb5837fe700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb55c023630 0x7fb55c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:29.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.097+0000 7fb5837fe700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb55c023630 0x7fb55c025ac0 crc :-1 s=READY pgs=257 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:29.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.109+0000 7fb58942d700 1 -- 172.21.15.16:0/2614306231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb58408fb80 con 0x7fb5840992d0 2019-12-05T06:20:29.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.117+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.117+0000 7fb57bfff700 1 -- 172.21.15.16:0/2614306231 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb574000b50 con 0x7fb5840992d0 2019-12-05T06:20:29.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.358+0000 7fb58942d700 1 -- 172.21.15.16:0/2614306231 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb58408fb80 con 0x7fb5840992d0 2019-12-05T06:20:29.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.359+0000 7fb582ffd700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=THROTTLE_DONE pgs=580 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:29.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.359+0000 7fb57bfff700 1 -- 172.21.15.16:0/2614306231 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fb574009630 con 0x7fb5840992d0 2019-12-05T06:20:29.349 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2019-12-05T06:20:29.349 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:29.350 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5604,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:29.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 -- 172.21.15.16:0/2614306231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb55c023630 msgr2=0x7fb55c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb55c023630 0x7fb55c025ac0 crc :-1 s=READY pgs=257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 -- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 msgr2=0x7fb584098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 0x7fb584098840 secure :-1 s=READY pgs=580 cs=0 l=1 rx=0x7fb574010450 tx=0x7fb5740102d0).stop 2019-12-05T06:20:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 -- 172.21.15.16:0/2614306231 shutdown_connections 2019-12-05T06:20:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 --2- 172.21.15.16:0/2614306231 >> v2:172.21.15.76:3300/0 conn(0x7fb584098d80 0x7fb584088fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb55c023630 0x7fb55c025ac0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5840992d0 0x7fb584098840 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 --2- 172.21.15.16:0/2614306231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb58406df10 0x7fb584098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 -- 172.21.15.16:0/2614306231 shutdown_connections 2019-12-05T06:20:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 -- 172.21.15.16:0/2614306231 wait complete. 2019-12-05T06:20:29.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:29.362+0000 7fb58942d700 1 -- 172.21.15.16:0/2614306231 >> 172.21.15.16:0/2614306231 conn(0x7fb5840714f0 msgr2=0x7fb58409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:29.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:20:31.380 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:31.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.662+0000 7f7decd17700 1 Processor -- start 2019-12-05T06:20:31.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.662+0000 7f7decd17700 1 -- start start 2019-12-05T06:20:31.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.664+0000 7f7decd17700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7de808c6d0 0x7f7de808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:31.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.664+0000 7f7decd17700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 0x7f7de808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:31.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.664+0000 7f7decd17700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7de808fb70 con 0x7f7de808fef0 2019-12-05T06:20:31.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.664+0000 7f7decd17700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de808df90 con 0x7f7de808c6d0 2019-12-05T06:20:31.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.664+0000 7f7decd17700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de80a3f70 con 0x7f7de808f720 2019-12-05T06:20:31.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.665+0000 7f7de6d9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7de808c6d0 0x7f7de808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:31.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.665+0000 7f7de759e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 0x7f7de808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:31.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.665+0000 7f7de659c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7de808fef0 0x7f7de808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57622/0 (socket says 172.21.15.16:57622) 2019-12-05T06:20:31.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.665+0000 7f7de659c700 1 -- 172.21.15.16:0/3071390823 learned_addr learned my addr 172.21.15.16:0/3071390823 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:31.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.665+0000 7f7de5d9b700 1 -- 172.21.15.16:0/3071390823 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106060684 0 0) 0x7f7dd0000c60 con 0x7f7de808fef0 2019-12-05T06:20:31.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.666+0000 7f7de5d9b700 1 -- 172.21.15.16:0/3071390823 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7dc8002e80 con 0x7f7de808fef0 2019-12-05T06:20:31.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.666+0000 7f7de759e700 1 -- 172.21.15.16:0/3071390823 >> v2:172.21.15.76:3300/0 conn(0x7f7de808c6d0 msgr2=0x7f7de808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:31.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.666+0000 7f7de759e700 1 --2- 172.21.15.16:0/3071390823 >> v2:172.21.15.76:3300/0 conn(0x7f7de808c6d0 0x7f7de808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:31.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.666+0000 7f7de759e700 1 -- 172.21.15.16:0/3071390823 >> v1:172.21.15.76:6789/0 conn(0x7f7de808fef0 legacy=0x7f7de808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:31.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.666+0000 7f7de759e700 1 -- 172.21.15.16:0/3071390823 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de808d540 con 0x7f7de808f720 2019-12-05T06:20:31.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.666+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3071390823 >> v2:172.21.15.76:3300/0 conn(0x7f7de808c6d0 0x7f7de808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:31.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.666+0000 7f7de759e700 1 --2- 172.21.15.16:0/3071390823 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 0x7f7de808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7dd800af80 tx=0x7f7dd800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.666+0000 7f7de759e700 1 --2- 172.21.15.16:0/3071390823 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 0x7f7de808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7dd800af80 tx=0x7f7dd800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.666+0000 7f7de759e700 1 --2- 172.21.15.16:0/3071390823 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 0x7f7de808d000 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7f7dd800af80 tx=0x7f7dd800b790).ready entity=mon.2 client_cookie=7efe21cac0c27030 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:31.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.667+0000 7f7de759e700 1 --2- 172.21.15.16:0/3071390823 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 0x7f7de808d000 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7f7dd800af80 tx=0x7f7dd800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.667+0000 7f7de759e700 1 --2- 172.21.15.16:0/3071390823 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 0x7f7de808d000 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7f7dd800af80 tx=0x7f7dd800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.667+0000 7f7de5d9b700 1 -- 172.21.15.16:0/3071390823 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dd8000bc0 con 0x7f7de808f720 2019-12-05T06:20:31.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.667+0000 7f7de759e700 1 --2- 172.21.15.16:0/3071390823 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 0x7f7de808d000 secure :-1 s=THROTTLE_DONE pgs=143 cs=0 l=1 rx=0x7f7dd800af80 tx=0x7f7dd800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.667+0000 7f7de5d9b700 1 -- 172.21.15.16:0/3071390823 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dd80097f0 con 0x7f7de808f720 2019-12-05T06:20:31.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.667+0000 7f7de5d9b700 1 -- 172.21.15.16:0/3071390823 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dd8009f60 con 0x7f7de808f720 2019-12-05T06:20:31.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.667+0000 7f7decd17700 1 -- 172.21.15.16:0/3071390823 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 msgr2=0x7f7de808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:31.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.667+0000 7f7decd17700 1 --2- 172.21.15.16:0/3071390823 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 0x7f7de808d000 secure :-1 s=READY pgs=143 cs=0 l=1 rx=0x7f7dd800af80 tx=0x7f7dd800b790).stop 2019-12-05T06:20:31.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.669+0000 7f7decd17700 1 -- 172.21.15.16:0/3071390823 shutdown_connections 2019-12-05T06:20:31.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.669+0000 7f7decd17700 1 --2- 172.21.15.16:0/3071390823 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de808f720 0x7f7de808d000 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:31.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.669+0000 7f7decd17700 1 --2- 172.21.15.16:0/3071390823 >> v2:172.21.15.76:3300/0 conn(0x7f7de808c6d0 0x7f7de808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:31.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.669+0000 7f7decd17700 1 -- 172.21.15.16:0/3071390823 shutdown_connections 2019-12-05T06:20:31.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.670+0000 7f7decd17700 1 -- 172.21.15.16:0/3071390823 wait complete. 2019-12-05T06:20:31.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.670+0000 7f7decd17700 1 -- 172.21.15.16:0/3071390823 >> 172.21.15.16:0/3071390823 conn(0x7f7de8061940 msgr2=0x7f7de8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:31.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.675+0000 7f7decd17700 1 Processor -- start 2019-12-05T06:20:31.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.675+0000 7f7decd17700 1 -- start start 2019-12-05T06:20:31.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.676+0000 7f7decd17700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:31.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.676+0000 7f7decd17700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7de8088190 0x7f7de8084600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:31.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.676+0000 7f7decd17700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7de80976e0 0x7f7de8084b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.676+0000 7f7decd17700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de80867d0 con 0x7f7de8088190 2019-12-05T06:20:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.676+0000 7f7decd17700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7de8086aa0 con 0x7f7de80976e0 2019-12-05T06:20:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.676+0000 7f7decd17700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7de80a2600 con 0x7f7de8084210 2019-12-05T06:20:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de6d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de6d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:55144/0 (socket says 172.21.15.16:55144) 2019-12-05T06:20:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de759e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7de80976e0 0x7f7de8084b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:31.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de6d9d700 1 -- 172.21.15.16:0/3846766538 learned_addr learned my addr 172.21.15.16:0/3846766538 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de659c700 1 --2- 172.21.15.16:0/3846766538 >> v2:172.21.15.16:3300/0 conn(0x7f7de8088190 0x7f7de8084600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de6d9d700 1 -- 172.21.15.16:0/3846766538 >> v2:172.21.15.76:3300/0 conn(0x7f7de80976e0 msgr2=0x7f7de8084b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> v2:172.21.15.76:3300/0 conn(0x7f7de80976e0 0x7f7de8084b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de6d9d700 1 -- 172.21.15.16:0/3846766538 >> v2:172.21.15.16:3300/0 conn(0x7f7de8088190 msgr2=0x7f7de8084600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> v2:172.21.15.16:3300/0 conn(0x7f7de8088190 0x7f7de8084600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de6d9d700 1 -- 172.21.15.16:0/3846766538 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7de8087890 con 0x7f7de8084210 2019-12-05T06:20:31.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.677+0000 7f7de759e700 1 --2- 172.21.15.16:0/3846766538 >> v2:172.21.15.76:3300/0 conn(0x7f7de80976e0 0x7f7de8084b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7ddf7fe700 1 -- 172.21.15.16:0/3846766538 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7de0007890 con 0x7f7de8084210 2019-12-05T06:20:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7ddf7fe700 1 -- 172.21.15.16:0/3846766538 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7de000f150 con 0x7f7de8084210 2019-12-05T06:20:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7ddf7fe700 1 -- 172.21.15.16:0/3846766538 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7de000f7e0 con 0x7f7de8084210 2019-12-05T06:20:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7decd17700 1 -- 172.21.15.16:0/3846766538 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7de8087890 con 0x7f7de8084210 2019-12-05T06:20:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.678+0000 7f7decd17700 1 -- 172.21.15.16:0/3846766538 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7de8087890 con 0x7f7de8084210 2019-12-05T06:20:31.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.679+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.680+0000 7f7ddf7fe700 1 -- 172.21.15.16:0/3846766538 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7de000ed30 con 0x7f7de8084210 2019-12-05T06:20:31.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.680+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.680+0000 7f7ddf7fe700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dc0023630 0x7f7dc0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:31.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.680+0000 7f7ddf7fe700 1 -- 172.21.15.16:0/3846766538 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 6719+0+0 (secure 0 0 0) 0x7f7de0046060 con 0x7f7de8084210 2019-12-05T06:20:31.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.681+0000 7f7de659c700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dc0023630 0x7f7dc0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:31.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.681+0000 7f7de659c700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dc0023630 0x7f7dc0025ac0 crc :-1 s=READY pgs=258 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:31.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.694+0000 7f7decd17700 1 -- 172.21.15.16:0/3846766538 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7de8087890 con 0x7f7de8084210 2019-12-05T06:20:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.704+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.704+0000 7f7ddf7fe700 1 -- 172.21.15.16:0/3846766538 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7de0007890 con 0x7f7de8084210 2019-12-05T06:20:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.942+0000 7f7decd17700 1 -- 172.21.15.16:0/3846766538 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7de8087890 con 0x7f7de8084210 2019-12-05T06:20:31.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.943+0000 7f7de6d9d700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=THROTTLE_DONE pgs=144 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:31.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.943+0000 7f7ddf7fe700 1 -- 172.21.15.16:0/3846766538 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f7de0000d40 con 0x7f7de8084210 2019-12-05T06:20:31.935 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 72 2019-12-05T06:20:31.935 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:31.935 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":72,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5604,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":72}],"filesystems":[{"mdsmap":{"epoch":71,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:19:38.059402+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":57,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5490},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5490":{"gid":5490,"name":"a","rank":0,"incarnation":68,"state":"up:active","state_seq":24,"addr":"172.21.15.16:6833/4122423358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4122423358},{"type":"v1","addr":"172.21.15.16:6833","nonce":4122423358}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:31.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 -- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dc0023630 msgr2=0x7f7dc0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:31.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dc0023630 0x7f7dc0025ac0 crc :-1 s=READY pgs=258 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:31.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 -- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 msgr2=0x7f7de8096b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:31.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 secure :-1 s=READY pgs=144 cs=0 l=1 rx=0x7f7de00107b0 tx=0x7f7de00106b0).stop 2019-12-05T06:20:31.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 -- 172.21.15.16:0/3846766538 shutdown_connections 2019-12-05T06:20:31.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 --2- 172.21.15.16:0/3846766538 >> v2:172.21.15.76:3300/0 conn(0x7f7de80976e0 0x7f7de8084b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:31.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7dc0023630 0x7f7dc0025ac0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:31.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 --2- 172.21.15.16:0/3846766538 >> v2:172.21.15.16:3300/0 conn(0x7f7de8088190 0x7f7de8084600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:31.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 --2- 172.21.15.16:0/3846766538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7de8084210 0x7f7de8096b90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:31.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 -- 172.21.15.16:0/3846766538 shutdown_connections 2019-12-05T06:20:31.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 -- 172.21.15.16:0/3846766538 wait complete. 2019-12-05T06:20:31.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:31.948+0000 7f7decd17700 1 -- 172.21.15.16:0/3846766538 >> 172.21.15.16:0/3846766538 conn(0x7f7de81331c0 msgr2=0x7f7de8088620 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:31.949 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:20:32.084 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:20:32.088 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:20:33.951 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:34.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.183+0000 7f3711bba700 1 Processor -- start 2019-12-05T06:20:34.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.183+0000 7f3711bba700 1 -- start start 2019-12-05T06:20:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.184+0000 7f3711bba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f370c08d2c0 0x7f370c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f3711bba700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c08fef0 0x7f370c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f3711bba700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f370c0958e0 con 0x7f370c08c6d0 2019-12-05T06:20:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f3711bba700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370c08df90 con 0x7f370c08d2c0 2019-12-05T06:20:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f3711bba700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c0a3f70 con 0x7f370c08fef0 2019-12-05T06:20:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f370b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f370c08d2c0 0x7f370c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:34.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f370b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f370c08d2c0 0x7f370c08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51166/0 (socket says 172.21.15.16:51166) 2019-12-05T06:20:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f370b7fe700 1 -- 172.21.15.16:0/1932840561 learned_addr learned my addr 172.21.15.16:0/1932840561 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f370bfff700 1 --1- 172.21.15.16:0/1932840561 >> v1:172.21.15.76:6789/0 conn(0x7f370c08c6d0 0x7f370c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57636/0 (socket says 172.21.15.16:57636) 2019-12-05T06:20:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f3710958700 1 --2- 172.21.15.16:0/1932840561 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c08fef0 0x7f370c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f370affd700 1 -- 172.21.15.16:0/1932840561 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3034270137 0 0) 0x7f36fc001080 con 0x7f370c08c6d0 2019-12-05T06:20:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f370b7fe700 1 -- 172.21.15.16:0/1932840561 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c08fef0 msgr2=0x7f370c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:34.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f370b7fe700 1 --2- 172.21.15.16:0/1932840561 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c08fef0 0x7f370c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f370b7fe700 1 -- 172.21.15.16:0/1932840561 >> v1:172.21.15.76:6789/0 conn(0x7f370c08c6d0 legacy=0x7f370c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f370b7fe700 1 -- 172.21.15.16:0/1932840561 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c08e620 con 0x7f370c08d2c0 2019-12-05T06:20:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.185+0000 7f370b7fe700 1 --2- 172.21.15.16:0/1932840561 >> v2:172.21.15.16:3300/0 conn(0x7f370c08d2c0 0x7f370c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f36f400f740 tx=0x7f36f400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.186+0000 7f370b7fe700 1 --2- 172.21.15.16:0/1932840561 >> v2:172.21.15.16:3300/0 conn(0x7f370c08d2c0 0x7f370c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f36f400f740 tx=0x7f36f400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.186+0000 7f370b7fe700 1 --2- 172.21.15.16:0/1932840561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370c08d2c0 0x7f370c08ef20 secure :-1 s=READY pgs=581 cs=0 l=1 rx=0x7f36f400f740 tx=0x7f36f400f570).ready entity=mon.0 client_cookie=b7207ce10f4e0512 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.186+0000 7f370b7fe700 1 --2- 172.21.15.16:0/1932840561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370c08d2c0 0x7f370c08ef20 secure :-1 s=THROTTLE_DONE pgs=581 cs=0 l=1 rx=0x7f36f400f740 tx=0x7f36f400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.186+0000 7f370b7fe700 1 --2- 172.21.15.16:0/1932840561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370c08d2c0 0x7f370c08ef20 secure :-1 s=THROTTLE_DONE pgs=581 cs=0 l=1 rx=0x7f36f400f740 tx=0x7f36f400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.186+0000 7f370affd700 1 -- 172.21.15.16:0/1932840561 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36f4000960 con 0x7f370c08d2c0 2019-12-05T06:20:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.186+0000 7f370b7fe700 1 --2- 172.21.15.16:0/1932840561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370c08d2c0 0x7f370c08ef20 secure :-1 s=THROTTLE_DONE pgs=581 cs=0 l=1 rx=0x7f36f400f740 tx=0x7f36f400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.186+0000 7f370affd700 1 -- 172.21.15.16:0/1932840561 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36f4000c90 con 0x7f370c08d2c0 2019-12-05T06:20:34.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.186+0000 7f370affd700 1 -- 172.21.15.16:0/1932840561 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36f40013e0 con 0x7f370c08d2c0 2019-12-05T06:20:34.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.186+0000 7f3711bba700 1 -- 172.21.15.16:0/1932840561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370c08d2c0 msgr2=0x7f370c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.186+0000 7f3711bba700 1 --2- 172.21.15.16:0/1932840561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370c08d2c0 0x7f370c08ef20 secure :-1 s=READY pgs=581 cs=0 l=1 rx=0x7f36f400f740 tx=0x7f36f400f570).stop 2019-12-05T06:20:34.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.187+0000 7f3711bba700 1 -- 172.21.15.16:0/1932840561 shutdown_connections 2019-12-05T06:20:34.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.187+0000 7f3711bba700 1 --2- 172.21.15.16:0/1932840561 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c08fef0 0x7f370c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:34.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.187+0000 7f3711bba700 1 --2- 172.21.15.16:0/1932840561 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f370c08d2c0 0x7f370c08ef20 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.187+0000 7f3711bba700 1 -- 172.21.15.16:0/1932840561 shutdown_connections 2019-12-05T06:20:34.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.187+0000 7f3711bba700 1 -- 172.21.15.16:0/1932840561 wait complete. 2019-12-05T06:20:34.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.187+0000 7f3711bba700 1 -- 172.21.15.16:0/1932840561 >> 172.21.15.16:0/1932840561 conn(0x7f370c061940 msgr2=0x7f370c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:34.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.192+0000 7f3711bba700 1 Processor -- start 2019-12-05T06:20:34.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.192+0000 7f3711bba700 1 -- start start 2019-12-05T06:20:34.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f3711bba700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:34.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f3711bba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f370c09c770 0x7f370c1571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:34.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f3711bba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f370c087d70 con 0x7f370c09cf40 2019-12-05T06:20:34.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f3711bba700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f370c138b80 con 0x7f370c09c770 2019-12-05T06:20:34.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f3711bba700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f370c089000 con 0x7f370c14d450 2019-12-05T06:20:34.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f3710958700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f370c09c770 0x7f370c1571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:34.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f370b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f370c09cf40 0x7f370c09bf70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36342/0 (socket says 172.21.15.16:36342) 2019-12-05T06:20:34.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f370bfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:34.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f370b7fe700 1 -- 172.21.15.16:0/2722062793 learned_addr learned my addr 172.21.15.16:0/2722062793 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:34.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f370bfff700 1 -- 172.21.15.16:0/2722062793 >> v2:172.21.15.16:3300/0 conn(0x7f370c09c770 msgr2=0x7f370c1571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> v2:172.21.15.16:3300/0 conn(0x7f370c09c770 0x7f370c1571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f370bfff700 1 -- 172.21.15.16:0/2722062793 >> v1:172.21.15.16:6789/0 conn(0x7f370c09cf40 legacy=0x7f370c09bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f370bfff700 1 -- 172.21.15.16:0/2722062793 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f370c09b670 con 0x7f370c14d450 2019-12-05T06:20:34.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.194+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.195+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.195+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=READY pgs=145 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:34.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.195+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.195+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.195+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.195+0000 7f3708ff9700 1 -- 172.21.15.16:0/2722062793 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36fc000c10 con 0x7f370c14d450 2019-12-05T06:20:34.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.195+0000 7f3708ff9700 1 -- 172.21.15.16:0/2722062793 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36fc0041c0 con 0x7f370c14d450 2019-12-05T06:20:34.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.195+0000 7f3711bba700 1 -- 172.21.15.16:0/2722062793 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f370c09b670 con 0x7f370c14d450 2019-12-05T06:20:34.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.195+0000 7f3708ff9700 1 -- 172.21.15.16:0/2722062793 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36fc004890 con 0x7f370c14d450 2019-12-05T06:20:34.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.195+0000 7f3711bba700 1 -- 172.21.15.16:0/2722062793 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f370c09b670 con 0x7f370c14d450 2019-12-05T06:20:34.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.196+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.196+0000 7f3708ff9700 1 -- 172.21.15.16:0/2722062793 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f36fc0043b0 con 0x7f370c14d450 2019-12-05T06:20:34.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.196+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.196+0000 7f3708ff9700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36f0023630 0x7f36f0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:34.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.197+0000 7f3708ff9700 1 -- 172.21.15.16:0/2722062793 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f36fc045d30 con 0x7f370c14d450 2019-12-05T06:20:34.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.197+0000 7f370b7fe700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36f0023630 0x7f36f0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.197+0000 7f370b7fe700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36f0023630 0x7f36f0025ac0 crc :-1 s=READY pgs=259 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.198+0000 7f3711bba700 1 -- 172.21.15.16:0/2722062793 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f370c09b670 con 0x7f370c14d450 2019-12-05T06:20:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.202+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.202+0000 7f3708ff9700 1 -- 172.21.15.16:0/2722062793 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f36fc001220 con 0x7f370c14d450 2019-12-05T06:20:34.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.415+0000 7f3711bba700 1 -- 172.21.15.16:0/2722062793 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f370c09b670 con 0x7f370c14d450 2019-12-05T06:20:34.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.415+0000 7f370bfff700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=THROTTLE_DONE pgs=145 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.416+0000 7f3708ff9700 1 -- 172.21.15.16:0/2722062793 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f36fc0043b0 con 0x7f370c14d450 2019-12-05T06:20:34.407 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2019-12-05T06:20:34.408 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:34.408 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:33.104325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:34.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 -- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36f0023630 msgr2=0x7f36f0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36f0023630 0x7f36f0025ac0 crc :-1 s=READY pgs=259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 -- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 msgr2=0x7f370c14d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 secure :-1 s=READY pgs=145 cs=0 l=1 rx=0x7f36fc010370 tx=0x7f36fc0101a0).stop 2019-12-05T06:20:34.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 -- 172.21.15.16:0/2722062793 shutdown_connections 2019-12-05T06:20:34.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f36f0023630 0x7f36f0025ac0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 --2- 172.21.15.16:0/2722062793 >> v2:172.21.15.16:3300/0 conn(0x7f370c09c770 0x7f370c1571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 --2- 172.21.15.16:0/2722062793 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f370c14d450 0x7f370c14d840 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 -- 172.21.15.16:0/2722062793 shutdown_connections 2019-12-05T06:20:34.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 -- 172.21.15.16:0/2722062793 wait complete. 2019-12-05T06:20:34.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.422+0000 7f3711bba700 1 -- 172.21.15.16:0/2722062793 >> 172.21.15.16:0/2722062793 conn(0x7f370c07aeb0 msgr2=0x7f370c0a2040 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:34.428 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:20:34.429 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:20:34.429 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.691+0000 7f4627290700 1 Processor -- start 2019-12-05T06:20:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.692+0000 7f4627290700 1 -- start start 2019-12-05T06:20:34.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.693+0000 7f4627290700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f462008c6d0 0x7f462008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.693+0000 7f4627290700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f462008fef0 0x7f462008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.693+0000 7f4627290700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46200958e0 con 0x7f462008ef20 2019-12-05T06:20:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.693+0000 7f4627290700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462008df90 con 0x7f462008c6d0 2019-12-05T06:20:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.693+0000 7f4627290700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46200a3f70 con 0x7f462008fef0 2019-12-05T06:20:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.694+0000 7f462582d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f462008c6d0 0x7f462008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.694+0000 7f462582d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f462008c6d0 0x7f462008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51180/0 (socket says 172.21.15.16:51180) 2019-12-05T06:20:34.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.694+0000 7f462582d700 1 -- 172.21.15.16:0/3312609878 learned_addr learned my addr 172.21.15.16:0/3312609878 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.694+0000 7f462602e700 1 --1- 172.21.15.16:0/3312609878 >> v1:172.21.15.76:6789/0 conn(0x7f462008ef20 0x7f462008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57652/0 (socket says 172.21.15.16:57652) 2019-12-05T06:20:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.694+0000 7f462502c700 1 --2- 172.21.15.16:0/3312609878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f462008fef0 0x7f462008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.694+0000 7f462482b700 1 -- 172.21.15.16:0/3312609878 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1025093882 0 0) 0x7f46180009b0 con 0x7f462008ef20 2019-12-05T06:20:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.694+0000 7f462582d700 1 -- 172.21.15.16:0/3312609878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f462008fef0 msgr2=0x7f462008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.694+0000 7f462582d700 1 --2- 172.21.15.16:0/3312609878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f462008fef0 0x7f462008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:34.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462582d700 1 -- 172.21.15.16:0/3312609878 >> v1:172.21.15.76:6789/0 conn(0x7f462008ef20 legacy=0x7f462008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462582d700 1 -- 172.21.15.16:0/3312609878 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462008d540 con 0x7f462008c6d0 2019-12-05T06:20:34.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462502c700 1 --2- 172.21.15.16:0/3312609878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f462008fef0 0x7f462008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462582d700 1 --2- 172.21.15.16:0/3312609878 >> v2:172.21.15.16:3300/0 conn(0x7f462008c6d0 0x7f462008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f461c00fac0 tx=0x7f461c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462582d700 1 --2- 172.21.15.16:0/3312609878 >> v2:172.21.15.16:3300/0 conn(0x7f462008c6d0 0x7f462008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f461c00fac0 tx=0x7f461c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462582d700 1 --2- 172.21.15.16:0/3312609878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f462008c6d0 0x7f462008cac0 secure :-1 s=READY pgs=583 cs=0 l=1 rx=0x7f461c00fac0 tx=0x7f461c00fe60).ready entity=mon.0 client_cookie=17a528df59a88903 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462582d700 1 --2- 172.21.15.16:0/3312609878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f462008c6d0 0x7f462008cac0 secure :-1 s=THROTTLE_DONE pgs=583 cs=0 l=1 rx=0x7f461c00fac0 tx=0x7f461c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462482b700 1 -- 172.21.15.16:0/3312609878 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f461c000ce0 con 0x7f462008c6d0 2019-12-05T06:20:34.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462582d700 1 --2- 172.21.15.16:0/3312609878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f462008c6d0 0x7f462008cac0 secure :-1 s=THROTTLE_DONE pgs=583 cs=0 l=1 rx=0x7f461c00fac0 tx=0x7f461c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462582d700 1 --2- 172.21.15.16:0/3312609878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f462008c6d0 0x7f462008cac0 secure :-1 s=THROTTLE_DONE pgs=583 cs=0 l=1 rx=0x7f461c00fac0 tx=0x7f461c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462482b700 1 -- 172.21.15.16:0/3312609878 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461c001010 con 0x7f462008c6d0 2019-12-05T06:20:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f462482b700 1 -- 172.21.15.16:0/3312609878 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f461c001720 con 0x7f462008c6d0 2019-12-05T06:20:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f4627290700 1 -- 172.21.15.16:0/3312609878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f462008c6d0 msgr2=0x7f462008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.695+0000 7f4627290700 1 --2- 172.21.15.16:0/3312609878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f462008c6d0 0x7f462008cac0 secure :-1 s=READY pgs=583 cs=0 l=1 rx=0x7f461c00fac0 tx=0x7f461c00fe60).stop 2019-12-05T06:20:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.696+0000 7f4627290700 1 -- 172.21.15.16:0/3312609878 shutdown_connections 2019-12-05T06:20:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.696+0000 7f4627290700 1 --2- 172.21.15.16:0/3312609878 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f462008fef0 0x7f462008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:34.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.696+0000 7f4627290700 1 --2- 172.21.15.16:0/3312609878 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f462008c6d0 0x7f462008cac0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.697+0000 7f4627290700 1 -- 172.21.15.16:0/3312609878 shutdown_connections 2019-12-05T06:20:34.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.697+0000 7f4627290700 1 -- 172.21.15.16:0/3312609878 wait complete. 2019-12-05T06:20:34.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.697+0000 7f4627290700 1 -- 172.21.15.16:0/3312609878 >> 172.21.15.16:0/3312609878 conn(0x7f4620061940 msgr2=0x7f4620096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:34.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.702+0000 7f4627290700 1 Processor -- start 2019-12-05T06:20:34.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.703+0000 7f4627290700 1 -- start start 2019-12-05T06:20:34.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.704+0000 7f4627290700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4620096e90 0x7f462009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.704+0000 7f4627290700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.704+0000 7f4627290700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4620084b90 con 0x7f4620097280 2019-12-05T06:20:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.704+0000 7f4627290700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f462009a450 con 0x7f4620096e90 2019-12-05T06:20:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.704+0000 7f4627290700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4620099a20 con 0x7f4620084f30 2019-12-05T06:20:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.704+0000 7f462602e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4620097280 0x7f46200976c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36356/0 (socket says 172.21.15.16:36356) 2019-12-05T06:20:34.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.704+0000 7f462602e700 1 -- 172.21.15.16:0/2144585725 learned_addr learned my addr 172.21.15.16:0/2144585725 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.705+0000 7f462582d700 1 --2- 172.21.15.16:0/2144585725 >> v2:172.21.15.76:3300/0 conn(0x7f4620096e90 0x7f462009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.705+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.705+0000 7f46167fc700 1 -- 172.21.15.16:0/2144585725 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498083026 0 0) 0x7f4618001290 con 0x7f4620097280 2019-12-05T06:20:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.705+0000 7f46167fc700 1 -- 172.21.15.16:0/2144585725 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4608002e80 con 0x7f4620097280 2019-12-05T06:20:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.705+0000 7f462502c700 1 -- 172.21.15.16:0/2144585725 >> v2:172.21.15.76:3300/0 conn(0x7f4620096e90 msgr2=0x7f462009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.705+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> v2:172.21.15.76:3300/0 conn(0x7f4620096e90 0x7f462009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.705+0000 7f462502c700 1 -- 172.21.15.16:0/2144585725 >> v1:172.21.15.16:6789/0 conn(0x7f4620097280 legacy=0x7f46200976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.705+0000 7f462502c700 1 -- 172.21.15.16:0/2144585725 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f462009aae0 con 0x7f4620084f30 2019-12-05T06:20:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.705+0000 7f462582d700 1 --2- 172.21.15.16:0/2144585725 >> v2:172.21.15.76:3300/0 conn(0x7f4620096e90 0x7f462009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.705+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.706+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.706+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=READY pgs=146 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:34.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.706+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.706+0000 7f46167fc700 1 -- 172.21.15.16:0/2144585725 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4610000940 con 0x7f4620084f30 2019-12-05T06:20:34.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.706+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.706+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.706+0000 7f46167fc700 1 -- 172.21.15.16:0/2144585725 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f461000dff0 con 0x7f4620084f30 2019-12-05T06:20:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.706+0000 7f4627290700 1 -- 172.21.15.16:0/2144585725 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f462009aae0 con 0x7f4620084f30 2019-12-05T06:20:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.706+0000 7f46167fc700 1 -- 172.21.15.16:0/2144585725 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f461000e680 con 0x7f4620084f30 2019-12-05T06:20:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.706+0000 7f4627290700 1 -- 172.21.15.16:0/2144585725 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f462009aae0 con 0x7f4620084f30 2019-12-05T06:20:34.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.707+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.708+0000 7f46167fc700 1 -- 172.21.15.16:0/2144585725 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f461000db00 con 0x7f4620084f30 2019-12-05T06:20:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.708+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.708+0000 7f46167fc700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4608023670 0x7f4608025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.708+0000 7f46167fc700 1 -- 172.21.15.16:0/2144585725 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f4610045d70 con 0x7f4620084f30 2019-12-05T06:20:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.708+0000 7f462582d700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4608023670 0x7f4608025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:34.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.709+0000 7f462582d700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4608023670 0x7f4608025ae0 crc :-1 s=READY pgs=260 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:34.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.723+0000 7f4627290700 1 -- 172.21.15.16:0/2144585725 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f462009aae0 con 0x7f4620084f30 2019-12-05T06:20:34.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.730+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.730+0000 7f46167fc700 1 -- 172.21.15.16:0/2144585725 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4610000940 con 0x7f4620084f30 2019-12-05T06:20:34.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.966+0000 7f4627290700 1 -- 172.21.15.16:0/2144585725 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f462009aae0 con 0x7f4620084f30 2019-12-05T06:20:34.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.967+0000 7f462502c700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=THROTTLE_DONE pgs=146 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:34.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.967+0000 7f46167fc700 1 -- 172.21.15.16:0/2144585725 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f461000da10 con 0x7f4620084f30 2019-12-05T06:20:34.960 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 73 2019-12-05T06:20:34.960 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:34.961 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":73,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":73,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:33.104325+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:34.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.973+0000 7f4627290700 1 -- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4608023670 msgr2=0x7f4608025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.973+0000 7f4627290700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4608023670 0x7f4608025ae0 crc :-1 s=READY pgs=260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.973+0000 7f4627290700 1 -- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 msgr2=0x7f462009b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.973+0000 7f4627290700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 secure :-1 s=READY pgs=146 cs=0 l=1 rx=0x7f46100103a0 tx=0x7f4610010740).stop 2019-12-05T06:20:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.974+0000 7f4627290700 1 -- 172.21.15.16:0/2144585725 shutdown_connections 2019-12-05T06:20:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.974+0000 7f4627290700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4608023670 0x7f4608025ae0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.974+0000 7f4627290700 1 --2- 172.21.15.16:0/2144585725 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4620084f30 0x7f462009b920 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.974+0000 7f4627290700 1 --2- 172.21.15.16:0/2144585725 >> v2:172.21.15.76:3300/0 conn(0x7f4620096e90 0x7f462009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.974+0000 7f4627290700 1 -- 172.21.15.16:0/2144585725 shutdown_connections 2019-12-05T06:20:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.974+0000 7f4627290700 1 -- 172.21.15.16:0/2144585725 wait complete. 2019-12-05T06:20:34.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:34.974+0000 7f4627290700 1 -- 172.21.15.16:0/2144585725 >> 172.21.15.16:0/2144585725 conn(0x7f46200714f0 msgr2=0x7f462009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:34.974 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 73, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 73, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5604": { "addr": "172.21.15.76:6835/4248142088", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 4248142088, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 4248142088, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5604, "incarnation": 73, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 58, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:20:33.104325+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5604 } } } ], "standbys": [] } 2019-12-05T06:20:36.975 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:37.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.237+0000 7f12c3ac3700 1 Processor -- start 2019-12-05T06:20:37.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.237+0000 7f12c3ac3700 1 -- start start 2019-12-05T06:20:37.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.238+0000 7f12c3ac3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12bc08c6d0 0x7f12bc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:37.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.238+0000 7f12c3ac3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc08fb80 con 0x7f12bc08f720 2019-12-05T06:20:37.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.238+0000 7f12c3ac3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc0958e0 con 0x7f12bc08fef0 2019-12-05T06:20:37.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.238+0000 7f12c3ac3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12bc08df90 con 0x7f12bc08c6d0 2019-12-05T06:20:37.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.238+0000 7f12c2060700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12bc08c6d0 0x7f12bc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:37.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f12bc08c6d0 0x7f12bc08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51198/0 (socket says 172.21.15.16:51198) 2019-12-05T06:20:37.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 -- 172.21.15.16:0/176606635 learned_addr learned my addr 172.21.15.16:0/176606635 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:37.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2861700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f12bc08f720 0x7f12bc1533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36366/0 (socket says 172.21.15.16:36366) 2019-12-05T06:20:37.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 -- 172.21.15.16:0/176606635 >> v1:172.21.15.76:6789/0 conn(0x7f12bc08fef0 legacy=0x7f12bc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:37.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 -- 172.21.15.16:0/176606635 >> v1:172.21.15.16:6789/0 conn(0x7f12bc08f720 legacy=0x7f12bc1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:37.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 -- 172.21.15.16:0/176606635 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc08e620 con 0x7f12bc08c6d0 2019-12-05T06:20:37.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c105e700 1 -- 172.21.15.16:0/176606635 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297424089 0 0) 0x7f12ac0009b0 con 0x7f12bc08f720 2019-12-05T06:20:37.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 --2- 172.21.15.16:0/176606635 >> v2:172.21.15.16:3300/0 conn(0x7f12bc08c6d0 0x7f12bc08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f12b400fac0 tx=0x7f12b400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 --2- 172.21.15.16:0/176606635 >> v2:172.21.15.16:3300/0 conn(0x7f12bc08c6d0 0x7f12bc08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f12b400fac0 tx=0x7f12b400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 --2- 172.21.15.16:0/176606635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12bc08c6d0 0x7f12bc08cac0 secure :-1 s=READY pgs=588 cs=0 l=1 rx=0x7f12b400fac0 tx=0x7f12b400fe60).ready entity=mon.0 client_cookie=8f35daf774a0507c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:37.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 --2- 172.21.15.16:0/176606635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12bc08c6d0 0x7f12bc08cac0 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7f12b400fac0 tx=0x7f12b400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c105e700 1 -- 172.21.15.16:0/176606635 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12b4000ce0 con 0x7f12bc08c6d0 2019-12-05T06:20:37.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 --2- 172.21.15.16:0/176606635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12bc08c6d0 0x7f12bc08cac0 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7f12b400fac0 tx=0x7f12b400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c2060700 1 --2- 172.21.15.16:0/176606635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12bc08c6d0 0x7f12bc08cac0 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7f12b400fac0 tx=0x7f12b400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.239+0000 7f12c105e700 1 -- 172.21.15.16:0/176606635 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b4001010 con 0x7f12bc08c6d0 2019-12-05T06:20:37.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.240+0000 7f12c105e700 1 -- 172.21.15.16:0/176606635 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12b40016e0 con 0x7f12bc08c6d0 2019-12-05T06:20:37.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.240+0000 7f12c3ac3700 1 -- 172.21.15.16:0/176606635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12bc08c6d0 msgr2=0x7f12bc08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:37.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.240+0000 7f12c3ac3700 1 --2- 172.21.15.16:0/176606635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12bc08c6d0 0x7f12bc08cac0 secure :-1 s=READY pgs=588 cs=0 l=1 rx=0x7f12b400fac0 tx=0x7f12b400fe60).stop 2019-12-05T06:20:37.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.241+0000 7f12c3ac3700 1 -- 172.21.15.16:0/176606635 shutdown_connections 2019-12-05T06:20:37.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.241+0000 7f12c3ac3700 1 --2- 172.21.15.16:0/176606635 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f12bc08c6d0 0x7f12bc08cac0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:37.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.241+0000 7f12c3ac3700 1 -- 172.21.15.16:0/176606635 shutdown_connections 2019-12-05T06:20:37.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.241+0000 7f12c3ac3700 1 -- 172.21.15.16:0/176606635 wait complete. 2019-12-05T06:20:37.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.241+0000 7f12c3ac3700 1 -- 172.21.15.16:0/176606635 >> 172.21.15.16:0/176606635 conn(0x7f12bc061940 msgr2=0x7f12bc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:37.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.244+0000 7f12c3ac3700 1 Processor -- start 2019-12-05T06:20:37.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.244+0000 7f12c3ac3700 1 -- start start 2019-12-05T06:20:37.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.245+0000 7f12c3ac3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f12bc14f5b0 0x7f12bc14f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:37.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.245+0000 7f12c3ac3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc089fb0 con 0x7f12bc098b00 2019-12-05T06:20:37.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.245+0000 7f12c3ac3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12bc098f40 con 0x7f12bc0992d0 2019-12-05T06:20:37.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.245+0000 7f12c3ac3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12bc097370 con 0x7f12bc14f5b0 2019-12-05T06:20:37.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.245+0000 7f12c2861700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f12bc098b00 0x7f12bc1592b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36374/0 (socket says 172.21.15.16:36374) 2019-12-05T06:20:37.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.245+0000 7f12c2861700 1 -- 172.21.15.16:0/1733714456 learned_addr learned my addr 172.21.15.16:0/1733714456 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:37.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.245+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> v2:172.21.15.76:3300/0 conn(0x7f12bc14f5b0 0x7f12bc14f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:37.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.245+0000 7f12b2ffd700 1 -- 172.21.15.16:0/1733714456 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208954972 0 0) 0x7f12ac001310 con 0x7f12bc098b00 2019-12-05T06:20:37.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.245+0000 7f12b2ffd700 1 -- 172.21.15.16:0/1733714456 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f129c002e80 con 0x7f12bc098b00 2019-12-05T06:20:37.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c2060700 1 -- 172.21.15.16:0/1733714456 >> v1:172.21.15.76:6789/0 conn(0x7f12bc0992d0 legacy=0x7f12bc098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:37.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c2060700 1 -- 172.21.15.16:0/1733714456 >> v1:172.21.15.16:6789/0 conn(0x7f12bc098b00 legacy=0x7f12bc1592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:37.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c2060700 1 -- 172.21.15.16:0/1733714456 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12bc097a00 con 0x7f12bc14f5b0 2019-12-05T06:20:37.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> v2:172.21.15.76:3300/0 conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> v2:172.21.15.76:3300/0 conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=READY pgs=432 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:37.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12b2ffd700 1 -- 172.21.15.16:0/1733714456 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12b4000cc0 con 0x7f12bc14f5b0 2019-12-05T06:20:37.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12b2ffd700 1 -- 172.21.15.16:0/1733714456 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12b4000cc0 con 0x7f12bc14f5b0 2019-12-05T06:20:37.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c3ac3700 1 -- 172.21.15.16:0/1733714456 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12bc097a00 con 0x7f12bc14f5b0 2019-12-05T06:20:37.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12b2ffd700 1 -- 172.21.15.16:0/1733714456 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12b40014c0 con 0x7f12bc14f5b0 2019-12-05T06:20:37.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.246+0000 7f12c3ac3700 1 -- 172.21.15.16:0/1733714456 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12bc097a00 con 0x7f12bc14f5b0 2019-12-05T06:20:37.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.247+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.247+0000 7f12b2ffd700 1 -- 172.21.15.16:0/1733714456 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f12b40010e0 con 0x7f12bc14f5b0 2019-12-05T06:20:37.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.247+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.247+0000 7f12b2ffd700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f129c023670 0x7f129c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:37.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.247+0000 7f12b2ffd700 1 -- 172.21.15.16:0/1733714456 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f12b40463a0 con 0x7f12bc14f5b0 2019-12-05T06:20:37.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.248+0000 7f12c185f700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f129c023670 0x7f129c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:37.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.248+0000 7f12c185f700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f129c023670 0x7f129c025ae0 crc :-1 s=READY pgs=261 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:37.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.253+0000 7f12c3ac3700 1 -- 172.21.15.16:0/1733714456 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12bc097a00 con 0x7f12bc14f5b0 2019-12-05T06:20:37.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.263+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.263+0000 7f12b2ffd700 1 -- 172.21.15.16:0/1733714456 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f12b4000ed0 con 0x7f12bc14f5b0 2019-12-05T06:20:37.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.493+0000 7f12c3ac3700 1 -- 172.21.15.16:0/1733714456 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12bc097a00 con 0x7f12bc14f5b0 2019-12-05T06:20:37.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.495+0000 7f12c2060700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=THROTTLE_DONE pgs=432 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:37.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.495+0000 7f12b2ffd700 1 -- 172.21.15.16:0/1733714456 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 75 v75) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f12b4000ed0 con 0x7f12bc14f5b0 2019-12-05T06:20:37.485 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 75 2019-12-05T06:20:37.486 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:37.486 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":75,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":75,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:37.135582+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:rejoin","state_seq":15,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:37.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.500+0000 7f12c3ac3700 1 -- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f129c023670 msgr2=0x7f129c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.500+0000 7f12c3ac3700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f129c023670 0x7f129c025ae0 crc :-1 s=READY pgs=261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.500+0000 7f12c3ac3700 1 -- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 msgr2=0x7f12bc14f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.500+0000 7f12c3ac3700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 0x7f12bc14f9a0 secure :-1 s=READY pgs=432 cs=0 l=1 rx=0x7f12b4010710 tx=0x7f12b4010ab0).stop 2019-12-05T06:20:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.500+0000 7f12c3ac3700 1 -- 172.21.15.16:0/1733714456 shutdown_connections 2019-12-05T06:20:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.500+0000 7f12c3ac3700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f129c023670 0x7f129c025ae0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:37.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.500+0000 7f12c3ac3700 1 --2- 172.21.15.16:0/1733714456 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f12bc14f5b0 0x7f12bc14f9a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.500+0000 7f12c3ac3700 1 -- 172.21.15.16:0/1733714456 shutdown_connections 2019-12-05T06:20:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.500+0000 7f12c3ac3700 1 -- 172.21.15.16:0/1733714456 wait complete. 2019-12-05T06:20:37.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:37.500+0000 7f12c3ac3700 1 -- 172.21.15.16:0/1733714456 >> 172.21.15.16:0/1733714456 conn(0x7f12bc0714f0 msgr2=0x7f12bc09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:37.504 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:20:39.505 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:39.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.789+0000 7f6d63178700 1 Processor -- start 2019-12-05T06:20:39.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.789+0000 7f6d63178700 1 -- start start 2019-12-05T06:20:39.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.791+0000 7f6d63178700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:39.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.791+0000 7f6d63178700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5c0958e0 con 0x7f6d5c08c6d0 2019-12-05T06:20:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.791+0000 7f6d63178700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5c0629e0 con 0x7f6d5c08fef0 2019-12-05T06:20:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.791+0000 7f6d63178700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d5c08df90 con 0x7f6d5c08d2c0 2019-12-05T06:20:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.791+0000 7f6d60f14700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.791+0000 7f6d60f14700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51212/0 (socket says 172.21.15.16:51212) 2019-12-05T06:20:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.791+0000 7f6d60f14700 1 -- 172.21.15.16:0/1308680916 learned_addr learned my addr 172.21.15.16:0/1308680916 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.792+0000 7f6d60f14700 1 -- 172.21.15.16:0/1308680916 >> v1:172.21.15.76:6789/0 conn(0x7f6d5c08fef0 legacy=0x7f6d5c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:39.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.792+0000 7f6d60f14700 1 -- 172.21.15.16:0/1308680916 >> v1:172.21.15.16:6789/0 conn(0x7f6d5c08c6d0 legacy=0x7f6d5c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.792+0000 7f6d60f14700 1 -- 172.21.15.16:0/1308680916 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c08e620 con 0x7f6d5c08d2c0 2019-12-05T06:20:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.792+0000 7f6d53fff700 1 -- 172.21.15.16:0/1308680916 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423417498 0 0) 0x7f6d54000d30 con 0x7f6d5c08fef0 2019-12-05T06:20:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.792+0000 7f6d60f14700 1 --2- 172.21.15.16:0/1308680916 >> v2:172.21.15.16:3300/0 conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6d4c00f740 tx=0x7f6d4c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.792+0000 7f6d60f14700 1 --2- 172.21.15.16:0/1308680916 >> v2:172.21.15.16:3300/0 conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6d4c00f740 tx=0x7f6d4c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.792+0000 7f6d60f14700 1 --2- 172.21.15.16:0/1308680916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 secure :-1 s=READY pgs=591 cs=0 l=1 rx=0x7f6d4c00f740 tx=0x7f6d4c00fae0).ready entity=mon.0 client_cookie=ee06cc54488a1438 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:39.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.793+0000 7f6d60f14700 1 --2- 172.21.15.16:0/1308680916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 secure :-1 s=THROTTLE_DONE pgs=591 cs=0 l=1 rx=0x7f6d4c00f740 tx=0x7f6d4c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.793+0000 7f6d53fff700 1 -- 172.21.15.16:0/1308680916 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d4c000960 con 0x7f6d5c08d2c0 2019-12-05T06:20:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.793+0000 7f6d60f14700 1 --2- 172.21.15.16:0/1308680916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 secure :-1 s=THROTTLE_DONE pgs=591 cs=0 l=1 rx=0x7f6d4c00f740 tx=0x7f6d4c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.793+0000 7f6d60f14700 1 --2- 172.21.15.16:0/1308680916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 secure :-1 s=THROTTLE_DONE pgs=591 cs=0 l=1 rx=0x7f6d4c00f740 tx=0x7f6d4c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.793+0000 7f6d53fff700 1 -- 172.21.15.16:0/1308680916 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d4c000c90 con 0x7f6d5c08d2c0 2019-12-05T06:20:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.793+0000 7f6d53fff700 1 -- 172.21.15.16:0/1308680916 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d4c001360 con 0x7f6d5c08d2c0 2019-12-05T06:20:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.793+0000 7f6d63178700 1 -- 172.21.15.16:0/1308680916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c08d2c0 msgr2=0x7f6d5c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.793+0000 7f6d63178700 1 --2- 172.21.15.16:0/1308680916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 secure :-1 s=READY pgs=591 cs=0 l=1 rx=0x7f6d4c00f740 tx=0x7f6d4c00fae0).stop 2019-12-05T06:20:39.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.796+0000 7f6d63178700 1 -- 172.21.15.16:0/1308680916 shutdown_connections 2019-12-05T06:20:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.796+0000 7f6d63178700 1 --2- 172.21.15.16:0/1308680916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c08d2c0 0x7f6d5c08ef20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.796+0000 7f6d63178700 1 -- 172.21.15.16:0/1308680916 shutdown_connections 2019-12-05T06:20:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.796+0000 7f6d63178700 1 -- 172.21.15.16:0/1308680916 wait complete. 2019-12-05T06:20:39.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.796+0000 7f6d63178700 1 -- 172.21.15.16:0/1308680916 >> 172.21.15.16:0/1308680916 conn(0x7f6d5c061940 msgr2=0x7f6d5c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.800+0000 7f6d63178700 1 Processor -- start 2019-12-05T06:20:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.801+0000 7f6d63178700 1 -- start start 2019-12-05T06:20:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.802+0000 7f6d63178700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:39.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.802+0000 7f6d63178700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d5c0992d0 0x7f6d5c098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.802+0000 7f6d63178700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d5c06e7c0 con 0x7f6d5c098840 2019-12-05T06:20:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.802+0000 7f6d63178700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d5c097370 con 0x7f6d5c14f5b0 2019-12-05T06:20:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.802+0000 7f6d63178700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d5c08ad00 con 0x7f6d5c0992d0 2019-12-05T06:20:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.803+0000 7f6d61715700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.803+0000 7f6d61f16700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6d5c098840 0x7f6d5c088fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36388/0 (socket says 172.21.15.16:36388) 2019-12-05T06:20:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.803+0000 7f6d61715700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51216/0 (socket says 172.21.15.16:51216) 2019-12-05T06:20:39.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.803+0000 7f6d61f16700 1 -- 172.21.15.16:0/1349946865 learned_addr learned my addr 172.21.15.16:0/1349946865 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.803+0000 7f6d60f14700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d5c0992d0 0x7f6d5c098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.803+0000 7f6d61715700 1 -- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d5c0992d0 msgr2=0x7f6d5c098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.803+0000 7f6d51ffb700 1 -- 172.21.15.16:0/1349946865 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1813515337 0 0) 0x7f6d540015c0 con 0x7f6d5c098840 2019-12-05T06:20:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.803+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d5c0992d0 0x7f6d5c098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:39.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.803+0000 7f6d61715700 1 -- 172.21.15.16:0/1349946865 >> v1:172.21.15.16:6789/0 conn(0x7f6d5c098840 legacy=0x7f6d5c088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:39.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.803+0000 7f6d61715700 1 -- 172.21.15.16:0/1349946865 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c097a00 con 0x7f6d5c14f5b0 2019-12-05T06:20:39.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d60f14700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d5c0992d0 0x7f6d5c098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:39.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> v2:172.21.15.16:3300/0 conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> v2:172.21.15.16:3300/0 conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=READY pgs=593 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:39.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d51ffb700 1 -- 172.21.15.16:0/1349946865 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d5800eff0 con 0x7f6d5c14f5b0 2019-12-05T06:20:39.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d51ffb700 1 -- 172.21.15.16:0/1349946865 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5800ec10 con 0x7f6d5c14f5b0 2019-12-05T06:20:39.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d63178700 1 -- 172.21.15.16:0/1349946865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d5c097a00 con 0x7f6d5c14f5b0 2019-12-05T06:20:39.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d51ffb700 1 -- 172.21.15.16:0/1349946865 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d5800f1e0 con 0x7f6d5c14f5b0 2019-12-05T06:20:39.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.804+0000 7f6d63178700 1 -- 172.21.15.16:0/1349946865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d5c097a00 con 0x7f6d5c14f5b0 2019-12-05T06:20:39.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.805+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.806+0000 7f6d51ffb700 1 -- 172.21.15.16:0/1349946865 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6d5800ee00 con 0x7f6d5c14f5b0 2019-12-05T06:20:39.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.807+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.807+0000 7f6d51ffb700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d3c023630 0x7f6d3c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:39.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.807+0000 7f6d51ffb700 1 -- 172.21.15.16:0/1349946865 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f6d58045d60 con 0x7f6d5c14f5b0 2019-12-05T06:20:39.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.807+0000 7f6d60f14700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d3c023630 0x7f6d3c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:39.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.808+0000 7f6d60f14700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d3c023630 0x7f6d3c025ac0 crc :-1 s=READY pgs=262 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:39.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.820+0000 7f6d63178700 1 -- 172.21.15.16:0/1349946865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d5c097a00 con 0x7f6d5c14f5b0 2019-12-05T06:20:39.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.827+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:39.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:39.827+0000 7f6d51ffb700 1 -- 172.21.15.16:0/1349946865 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6d5800ea00 con 0x7f6d5c14f5b0 2019-12-05T06:20:40.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.044+0000 7f6d63178700 1 -- 172.21.15.16:0/1349946865 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d5c097a00 con 0x7f6d5c14f5b0 2019-12-05T06:20:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.044+0000 7f6d61715700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=THROTTLE_DONE pgs=593 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:40.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.044+0000 7f6d51ffb700 1 -- 172.21.15.16:0/1349946865 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f6d5800ee00 con 0x7f6d5c14f5b0 2019-12-05T06:20:40.036 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 76 2019-12-05T06:20:40.036 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:40.036 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:40.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.049+0000 7f6d63178700 1 -- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d3c023630 msgr2=0x7f6d3c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:40.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.049+0000 7f6d63178700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d3c023630 0x7f6d3c025ac0 crc :-1 s=READY pgs=262 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:40.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.049+0000 7f6d63178700 1 -- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 msgr2=0x7f6d5c14f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:40.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.049+0000 7f6d63178700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 secure :-1 s=READY pgs=593 cs=0 l=1 rx=0x7f6d580103b0 tx=0x7f6d58010750).stop 2019-12-05T06:20:40.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.049+0000 7f6d63178700 1 -- 172.21.15.16:0/1349946865 shutdown_connections 2019-12-05T06:20:40.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.049+0000 7f6d63178700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6d5c0992d0 0x7f6d5c098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:40.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.049+0000 7f6d63178700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6d3c023630 0x7f6d3c025ac0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:40.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.049+0000 7f6d63178700 1 --2- 172.21.15.16:0/1349946865 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6d5c14f5b0 0x7f6d5c14f9a0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:40.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.049+0000 7f6d63178700 1 -- 172.21.15.16:0/1349946865 shutdown_connections 2019-12-05T06:20:40.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.050+0000 7f6d63178700 1 -- 172.21.15.16:0/1349946865 wait complete. 2019-12-05T06:20:40.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:40.050+0000 7f6d63178700 1 -- 172.21.15.16:0/1349946865 >> 172.21.15.16:0/1349946865 conn(0x7f6d5c0714f0 msgr2=0x7f6d5c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:40.049 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5490 2019-12-05T06:20:40.049 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2019-12-05T06:20:47.049 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:20:47.050 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:20:47.050 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:20:47.055 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:20:47.056 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:47.238 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:20:47.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.323+0000 7f99b4999700 1 Processor -- start 2019-12-05T06:20:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.323+0000 7f99b4999700 1 -- start start 2019-12-05T06:20:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.326+0000 7f99b4999700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99b008c6d0 0x7f99b008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.326+0000 7f99b4999700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b008fef0 0x7f99b008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.326+0000 7f99b4999700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99b00958e0 con 0x7f99b008ef20 2019-12-05T06:20:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.326+0000 7f99b4999700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b008df90 con 0x7f99b008c6d0 2019-12-05T06:20:47.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.326+0000 7f99b4999700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b00a3f70 con 0x7f99b008fef0 2019-12-05T06:20:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.326+0000 7f99aed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99b008c6d0 0x7f99b008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.326+0000 7f99aed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99b008c6d0 0x7f99b008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51236/0 (socket says 172.21.15.16:51236) 2019-12-05T06:20:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.326+0000 7f99af59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f99b008ef20 0x7f99b008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57710/0 (socket says 172.21.15.16:57710) 2019-12-05T06:20:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.326+0000 7f99aed9d700 1 -- 172.21.15.16:0/4159595593 learned_addr learned my addr 172.21.15.16:0/4159595593 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.326+0000 7f99ae59c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b008fef0 0x7f99b008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:47.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.327+0000 7f99add9b700 1 -- 172.21.15.16:0/4159595593 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210993602 0 0) 0x7f99a8000d30 con 0x7f99b008ef20 2019-12-05T06:20:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.327+0000 7f99aed9d700 1 -- 172.21.15.16:0/4159595593 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b008fef0 msgr2=0x7f99b008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.327+0000 7f99aed9d700 1 --2- 172.21.15.16:0/4159595593 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b008fef0 0x7f99b008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:47.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.327+0000 7f99aed9d700 1 -- 172.21.15.16:0/4159595593 >> v1:172.21.15.76:6789/0 conn(0x7f99b008ef20 legacy=0x7f99b008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.327+0000 7f99aed9d700 1 -- 172.21.15.16:0/4159595593 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b008d540 con 0x7f99b008c6d0 2019-12-05T06:20:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.327+0000 7f99ae59c700 1 --2- 172.21.15.16:0/4159595593 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b008fef0 0x7f99b008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.327+0000 7f99aed9d700 1 --2- 172.21.15.16:0/4159595593 >> v2:172.21.15.16:3300/0 conn(0x7f99b008c6d0 0x7f99b008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f99a000f740 tx=0x7f99a000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.327+0000 7f99aed9d700 1 --2- 172.21.15.16:0/4159595593 >> v2:172.21.15.16:3300/0 conn(0x7f99b008c6d0 0x7f99b008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f99a000f740 tx=0x7f99a000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.327+0000 7f99aed9d700 1 --2- 172.21.15.16:0/4159595593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b008c6d0 0x7f99b008cac0 secure :-1 s=READY pgs=596 cs=0 l=1 rx=0x7f99a000f740 tx=0x7f99a000fae0).ready entity=mon.0 client_cookie=41dc69b253cded9b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:47.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.328+0000 7f99aed9d700 1 --2- 172.21.15.16:0/4159595593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b008c6d0 0x7f99b008cac0 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7f99a000f740 tx=0x7f99a000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.328+0000 7f99aed9d700 1 --2- 172.21.15.16:0/4159595593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b008c6d0 0x7f99b008cac0 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7f99a000f740 tx=0x7f99a000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.328+0000 7f99add9b700 1 -- 172.21.15.16:0/4159595593 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99a0000960 con 0x7f99b008c6d0 2019-12-05T06:20:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.328+0000 7f99aed9d700 1 --2- 172.21.15.16:0/4159595593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b008c6d0 0x7f99b008cac0 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7f99a000f740 tx=0x7f99a000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.328+0000 7f99add9b700 1 -- 172.21.15.16:0/4159595593 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a0000c90 con 0x7f99b008c6d0 2019-12-05T06:20:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.328+0000 7f99add9b700 1 -- 172.21.15.16:0/4159595593 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99a00013a0 con 0x7f99b008c6d0 2019-12-05T06:20:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.328+0000 7f99b4999700 1 -- 172.21.15.16:0/4159595593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b008c6d0 msgr2=0x7f99b008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:47.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.328+0000 7f99b4999700 1 --2- 172.21.15.16:0/4159595593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b008c6d0 0x7f99b008cac0 secure :-1 s=READY pgs=596 cs=0 l=1 rx=0x7f99a000f740 tx=0x7f99a000fae0).stop 2019-12-05T06:20:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.330+0000 7f99b4999700 1 -- 172.21.15.16:0/4159595593 shutdown_connections 2019-12-05T06:20:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.330+0000 7f99b4999700 1 --2- 172.21.15.16:0/4159595593 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b008fef0 0x7f99b008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.330+0000 7f99b4999700 1 --2- 172.21.15.16:0/4159595593 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f99b008c6d0 0x7f99b008cac0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.330+0000 7f99b4999700 1 -- 172.21.15.16:0/4159595593 shutdown_connections 2019-12-05T06:20:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.330+0000 7f99b4999700 1 -- 172.21.15.16:0/4159595593 wait complete. 2019-12-05T06:20:47.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.330+0000 7f99b4999700 1 -- 172.21.15.16:0/4159595593 >> 172.21.15.16:0/4159595593 conn(0x7f99b0061940 msgr2=0x7f99b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.335+0000 7f99b4999700 1 Processor -- start 2019-12-05T06:20:47.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.335+0000 7f99b4999700 1 -- start start 2019-12-05T06:20:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99b4999700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99b4999700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99b006e210 0x7f99b009b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99b4999700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f99b0128510 0x7f99b009b680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99b4999700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b0099780 con 0x7f99b006e210 2019-12-05T06:20:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99b4999700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99b0099a50 con 0x7f99b0128510 2019-12-05T06:20:47.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99b4999700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99b009a840 con 0x7f99b014d420 2019-12-05T06:20:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99ae59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99b006e210 0x7f99b009b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99ae59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f99b006e210 0x7f99b009b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51244/0 (socket says 172.21.15.16:51244) 2019-12-05T06:20:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99ae59c700 1 -- 172.21.15.16:0/2587637750 learned_addr learned my addr 172.21.15.16:0/2587637750 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:20:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.337+0000 7f99af59e700 1 --2- 172.21.15.16:0/2587637750 >> v2:172.21.15.76:3300/0 conn(0x7f99b0128510 0x7f99b009b680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:47.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.338+0000 7f99aed9d700 1 -- 172.21.15.16:0/2587637750 >> v2:172.21.15.76:3300/0 conn(0x7f99b0128510 msgr2=0x7f99b009b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.338+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> v2:172.21.15.76:3300/0 conn(0x7f99b0128510 0x7f99b009b680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.338+0000 7f99aed9d700 1 -- 172.21.15.16:0/2587637750 >> v2:172.21.15.16:3300/0 conn(0x7f99b006e210 msgr2=0x7f99b009b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:47.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.338+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> v2:172.21.15.16:3300/0 conn(0x7f99b006e210 0x7f99b009b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:47.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.338+0000 7f99aed9d700 1 -- 172.21.15.16:0/2587637750 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b006e8d0 con 0x7f99b014d420 2019-12-05T06:20:47.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.338+0000 7f99ae59c700 1 --2- 172.21.15.16:0/2587637750 >> v2:172.21.15.16:3300/0 conn(0x7f99b006e210 0x7f99b009b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:47.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.338+0000 7f99af59e700 1 --2- 172.21.15.16:0/2587637750 >> v2:172.21.15.76:3300/0 conn(0x7f99b0128510 0x7f99b009b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:47.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.338+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.338+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.338+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:47.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.339+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.339+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.339+0000 7f999f7fe700 1 -- 172.21.15.16:0/2587637750 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99a0005560 con 0x7f99b014d420 2019-12-05T06:20:47.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.339+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.339+0000 7f999f7fe700 1 -- 172.21.15.16:0/2587637750 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99a0005180 con 0x7f99b014d420 2019-12-05T06:20:47.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.339+0000 7f99b4999700 1 -- 172.21.15.16:0/2587637750 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b006e8d0 con 0x7f99b014d420 2019-12-05T06:20:47.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.339+0000 7f999f7fe700 1 -- 172.21.15.16:0/2587637750 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f99a0005b40 con 0x7f99b014d420 2019-12-05T06:20:47.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.339+0000 7f99b4999700 1 -- 172.21.15.16:0/2587637750 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b006e8d0 con 0x7f99b014d420 2019-12-05T06:20:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.340+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.341+0000 7f999f7fe700 1 -- 172.21.15.16:0/2587637750 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f99a0005370 con 0x7f99b014d420 2019-12-05T06:20:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.341+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.341+0000 7f999f7fe700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9994023630 0x7f9994025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:47.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.341+0000 7f999f7fe700 1 -- 172.21.15.16:0/2587637750 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f99a0045c50 con 0x7f99b014d420 2019-12-05T06:20:47.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.342+0000 7f99ae59c700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9994023630 0x7f9994025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:47.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.342+0000 7f99ae59c700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9994023630 0x7f9994025ac0 crc :-1 s=READY pgs=263 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:47.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.355+0000 7f99b4999700 1 -- 172.21.15.16:0/2587637750 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b009a840 con 0x7f99b014d420 2019-12-05T06:20:47.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.365+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.365+0000 7f999f7fe700 1 -- 172.21.15.16:0/2587637750 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f99a0007310 con 0x7f99b014d420 2019-12-05T06:20:47.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.633+0000 7f99b4999700 1 -- 172.21.15.16:0/2587637750 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99b009a840 con 0x7f99b014d420 2019-12-05T06:20:47.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.635+0000 7f99aed9d700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=THROTTLE_DONE pgs=149 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:47.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.635+0000 7f999f7fe700 1 -- 172.21.15.16:0/2587637750 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f99a0005370 con 0x7f99b014d420 2019-12-05T06:20:47.625 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 76 2019-12-05T06:20:47.625 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:47.625 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:47.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 -- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9994023630 msgr2=0x7f9994025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:47.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9994023630 0x7f9994025ac0 crc :-1 s=READY pgs=263 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:47.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 -- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 msgr2=0x7f99b014d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:47.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 secure :-1 s=READY pgs=149 cs=0 l=1 rx=0x7f99a00103a0 tx=0x7f99a0010740).stop 2019-12-05T06:20:47.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 -- 172.21.15.16:0/2587637750 shutdown_connections 2019-12-05T06:20:47.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 --2- 172.21.15.16:0/2587637750 >> v2:172.21.15.76:3300/0 conn(0x7f99b0128510 0x7f99b009b680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:47.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9994023630 0x7f9994025ac0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:47.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 --2- 172.21.15.16:0/2587637750 >> v2:172.21.15.16:3300/0 conn(0x7f99b006e210 0x7f99b009b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:47.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 --2- 172.21.15.16:0/2587637750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f99b014d420 0x7f99b014d810 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:47.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 -- 172.21.15.16:0/2587637750 shutdown_connections 2019-12-05T06:20:47.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 -- 172.21.15.16:0/2587637750 wait complete. 2019-12-05T06:20:47.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:47.639+0000 7f99b4999700 1 -- 172.21.15.16:0/2587637750 >> 172.21.15.16:0/2587637750 conn(0x7f99b0082220 msgr2=0x7f99b006d380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:47.638 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:20:49.640 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:49.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.916+0000 7f7a52348700 1 Processor -- start 2019-12-05T06:20:49.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.916+0000 7f7a52348700 1 -- start start 2019-12-05T06:20:49.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.918+0000 7f7a52348700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a4c08fef0 0x7f7a4c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:49.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.918+0000 7f7a52348700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c0958e0 con 0x7f7a4c08d2c0 2019-12-05T06:20:49.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.918+0000 7f7a52348700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c0629e0 con 0x7f7a4c08c6d0 2019-12-05T06:20:49.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.918+0000 7f7a52348700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4c08df90 con 0x7f7a4c08fef0 2019-12-05T06:20:49.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.918+0000 7f7a510e6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a4c08fef0 0x7f7a4c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:49.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.918+0000 7f7a4bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a4c08d2c0 0x7f7a4c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36420/0 (socket says 172.21.15.16:36420) 2019-12-05T06:20:49.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.918+0000 7f7a4bfff700 1 -- 172.21.15.16:0/907295294 learned_addr learned my addr 172.21.15.16:0/907295294 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:49.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.919+0000 7f7a4b7fe700 1 -- 172.21.15.16:0/907295294 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3024658099 0 0) 0x7f7a3c001150 con 0x7f7a4c08d2c0 2019-12-05T06:20:49.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.919+0000 7f7a4b7fe700 1 -- 172.21.15.16:0/907295294 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a38002e80 con 0x7f7a4c08d2c0 2019-12-05T06:20:49.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.919+0000 7f7a4b7fe700 1 -- 172.21.15.16:0/907295294 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177428081 0 0) 0x7f7a44001400 con 0x7f7a4c08c6d0 2019-12-05T06:20:49.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.919+0000 7f7a510e6700 1 -- 172.21.15.16:0/907295294 >> v1:172.21.15.76:6789/0 conn(0x7f7a4c08c6d0 legacy=0x7f7a4c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:49.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.919+0000 7f7a510e6700 1 -- 172.21.15.16:0/907295294 >> v1:172.21.15.16:6789/0 conn(0x7f7a4c08d2c0 legacy=0x7f7a4c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:49.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.919+0000 7f7a510e6700 1 -- 172.21.15.16:0/907295294 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c08e620 con 0x7f7a4c08fef0 2019-12-05T06:20:49.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.919+0000 7f7a510e6700 1 --2- 172.21.15.16:0/907295294 >> v2:172.21.15.16:3300/0 conn(0x7f7a4c08fef0 0x7f7a4c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7a3400af50 tx=0x7f7a3400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.919+0000 7f7a510e6700 1 --2- 172.21.15.16:0/907295294 >> v2:172.21.15.16:3300/0 conn(0x7f7a4c08fef0 0x7f7a4c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7a3400af50 tx=0x7f7a3400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.919+0000 7f7a510e6700 1 --2- 172.21.15.16:0/907295294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c08fef0 0x7f7a4c08f720 secure :-1 s=READY pgs=598 cs=0 l=1 rx=0x7f7a3400af50 tx=0x7f7a3400b760).ready entity=mon.0 client_cookie=264edc05ed635654 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.920+0000 7f7a510e6700 1 --2- 172.21.15.16:0/907295294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c08fef0 0x7f7a4c08f720 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f7a3400af50 tx=0x7f7a3400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.920+0000 7f7a510e6700 1 --2- 172.21.15.16:0/907295294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c08fef0 0x7f7a4c08f720 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f7a3400af50 tx=0x7f7a3400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.920+0000 7f7a4b7fe700 1 -- 172.21.15.16:0/907295294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a34009ba0 con 0x7f7a4c08fef0 2019-12-05T06:20:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.920+0000 7f7a510e6700 1 --2- 172.21.15.16:0/907295294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c08fef0 0x7f7a4c08f720 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f7a3400af50 tx=0x7f7a3400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.920+0000 7f7a4b7fe700 1 -- 172.21.15.16:0/907295294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a34000a70 con 0x7f7a4c08fef0 2019-12-05T06:20:49.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.920+0000 7f7a4b7fe700 1 -- 172.21.15.16:0/907295294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a3400a180 con 0x7f7a4c08fef0 2019-12-05T06:20:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.920+0000 7f7a52348700 1 -- 172.21.15.16:0/907295294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c08fef0 msgr2=0x7f7a4c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.920+0000 7f7a52348700 1 --2- 172.21.15.16:0/907295294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c08fef0 0x7f7a4c08f720 secure :-1 s=READY pgs=598 cs=0 l=1 rx=0x7f7a3400af50 tx=0x7f7a3400b760).stop 2019-12-05T06:20:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.923+0000 7f7a52348700 1 -- 172.21.15.16:0/907295294 shutdown_connections 2019-12-05T06:20:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.923+0000 7f7a52348700 1 --2- 172.21.15.16:0/907295294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a4c08fef0 0x7f7a4c08f720 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.923+0000 7f7a52348700 1 -- 172.21.15.16:0/907295294 shutdown_connections 2019-12-05T06:20:49.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.923+0000 7f7a52348700 1 -- 172.21.15.16:0/907295294 wait complete. 2019-12-05T06:20:49.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.923+0000 7f7a52348700 1 -- 172.21.15.16:0/907295294 >> 172.21.15.16:0/907295294 conn(0x7f7a4c061940 msgr2=0x7f7a4c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:49.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.927+0000 7f7a52348700 1 Processor -- start 2019-12-05T06:20:49.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.927+0000 7f7a52348700 1 -- start start 2019-12-05T06:20:49.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.929+0000 7f7a52348700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:49.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.929+0000 7f7a52348700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7a4c084f30 0x7f7a4c09b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:49.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.929+0000 7f7a52348700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a4c084b90 con 0x7f7a4c097280 2019-12-05T06:20:49.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.929+0000 7f7a52348700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a4c099a20 con 0x7f7a4c084f30 2019-12-05T06:20:49.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.929+0000 7f7a52348700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c099cf0 con 0x7f7a4c096e90 2019-12-05T06:20:49.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.929+0000 7f7a510e6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a4c097280 0x7f7a4c097700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36428/0 (socket says 172.21.15.16:36428) 2019-12-05T06:20:49.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.929+0000 7f7a510e6700 1 -- 172.21.15.16:0/843647918 learned_addr learned my addr 172.21.15.16:0/843647918 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:49.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.929+0000 7f7a4bfff700 1 --2- 172.21.15.16:0/843647918 >> v2:172.21.15.76:3300/0 conn(0x7f7a4c084f30 0x7f7a4c09b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:49.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.929+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:49.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.930+0000 7f7a497fa700 1 -- 172.21.15.16:0/843647918 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1875216312 0 0) 0x7f7a34000940 con 0x7f7a4c097280 2019-12-05T06:20:49.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.930+0000 7f7a497fa700 1 -- 172.21.15.16:0/843647918 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a30002e80 con 0x7f7a4c097280 2019-12-05T06:20:49.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.930+0000 7f7a508e5700 1 -- 172.21.15.16:0/843647918 >> v2:172.21.15.76:3300/0 conn(0x7f7a4c084f30 msgr2=0x7f7a4c09b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:49.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.930+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> v2:172.21.15.76:3300/0 conn(0x7f7a4c084f30 0x7f7a4c09b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:49.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.930+0000 7f7a508e5700 1 -- 172.21.15.16:0/843647918 >> v1:172.21.15.16:6789/0 conn(0x7f7a4c097280 legacy=0x7f7a4c097700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:49.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.930+0000 7f7a497fa700 1 -- 172.21.15.16:0/843647918 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1275571114 0 0) 0x7f7a34000b60 con 0x7f7a4c097280 2019-12-05T06:20:49.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.930+0000 7f7a508e5700 1 -- 172.21.15.16:0/843647918 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c09aae0 con 0x7f7a4c096e90 2019-12-05T06:20:49.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.930+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.931+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.931+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:49.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.931+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.931+0000 7f7a497fa700 1 -- 172.21.15.16:0/843647918 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a44000df0 con 0x7f7a4c096e90 2019-12-05T06:20:49.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.931+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.931+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.931+0000 7f7a497fa700 1 -- 172.21.15.16:0/843647918 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a44009f70 con 0x7f7a4c096e90 2019-12-05T06:20:49.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.931+0000 7f7a52348700 1 -- 172.21.15.16:0/843647918 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4c09aae0 con 0x7f7a4c096e90 2019-12-05T06:20:49.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.931+0000 7f7a497fa700 1 -- 172.21.15.16:0/843647918 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a4400a680 con 0x7f7a4c096e90 2019-12-05T06:20:49.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.931+0000 7f7a52348700 1 -- 172.21.15.16:0/843647918 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4c09aae0 con 0x7f7a4c096e90 2019-12-05T06:20:49.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.932+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.933+0000 7f7a497fa700 1 -- 172.21.15.16:0/843647918 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7a44009ae0 con 0x7f7a4c096e90 2019-12-05T06:20:49.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.933+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.933+0000 7f7a497fa700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a30023680 0x7f7a30025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:49.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.933+0000 7f7a497fa700 1 -- 172.21.15.16:0/843647918 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f7a440460e0 con 0x7f7a4c096e90 2019-12-05T06:20:49.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.934+0000 7f7a4bfff700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a30023680 0x7f7a30025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:49.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.934+0000 7f7a4bfff700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a30023680 0x7f7a30025af0 crc :-1 s=READY pgs=264 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:49.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.948+0000 7f7a52348700 1 -- 172.21.15.16:0/843647918 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4c09aae0 con 0x7f7a4c096e90 2019-12-05T06:20:49.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.958+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:49.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:49.958+0000 7f7a497fa700 1 -- 172.21.15.16:0/843647918 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7a44000dd0 con 0x7f7a4c096e90 2019-12-05T06:20:50.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.191+0000 7f7a52348700 1 -- 172.21.15.16:0/843647918 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a4c09aae0 con 0x7f7a4c096e90 2019-12-05T06:20:50.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.193+0000 7f7a508e5700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=THROTTLE_DONE pgs=150 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:50.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.193+0000 7f7a497fa700 1 -- 172.21.15.16:0/843647918 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 76 v76) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f7a44009960 con 0x7f7a4c096e90 2019-12-05T06:20:50.189 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:50.190 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":76,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 76 2019-12-05T06:20:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.196+0000 7f7a52348700 1 -- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a30023680 msgr2=0x7f7a30025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.196+0000 7f7a52348700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a30023680 0x7f7a30025af0 crc :-1 s=READY pgs=264 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.196+0000 7f7a52348700 1 -- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 msgr2=0x7f7a4c09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:50.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.196+0000 7f7a52348700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 secure :-1 s=READY pgs=150 cs=0 l=1 rx=0x7f7a44010720 tx=0x7f7a4400b660).stop 2019-12-05T06:20:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.197+0000 7f7a52348700 1 -- 172.21.15.16:0/843647918 shutdown_connections 2019-12-05T06:20:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.197+0000 7f7a52348700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a30023680 0x7f7a30025af0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.197+0000 7f7a52348700 1 --2- 172.21.15.16:0/843647918 >> v2:172.21.15.76:3300/0 conn(0x7f7a4c084f30 0x7f7a4c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.197+0000 7f7a52348700 1 --2- 172.21.15.16:0/843647918 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a4c096e90 0x7f7a4c09b3e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.197+0000 7f7a52348700 1 -- 172.21.15.16:0/843647918 shutdown_connections 2019-12-05T06:20:50.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.197+0000 7f7a52348700 1 -- 172.21.15.16:0/843647918 wait complete. 2019-12-05T06:20:50.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:50.197+0000 7f7a52348700 1 -- 172.21.15.16:0/843647918 >> 172.21.15.16:0/843647918 conn(0x7f7a4c0714f0 msgr2=0x7f7a4c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:50.197 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:20:52.198 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:20:52.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.476+0000 7fec4bf7f700 1 Processor -- start 2019-12-05T06:20:52.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.476+0000 7fec4bf7f700 1 -- start start 2019-12-05T06:20:52.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.479+0000 7fec4bf7f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec4408d2c0 0x7fec4408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:52.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.479+0000 7fec4bf7f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4408fef0 0x7fec4408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:20:52.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.479+0000 7fec4bf7f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec440958e0 con 0x7fec4408c6d0 2019-12-05T06:20:52.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.479+0000 7fec4bf7f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4408df90 con 0x7fec4408d2c0 2019-12-05T06:20:52.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.479+0000 7fec4bf7f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec440a3f70 con 0x7fec4408fef0 2019-12-05T06:20:52.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.479+0000 7fec49d1b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec4408d2c0 0x7fec4408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:52.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.479+0000 7fec4a51c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fec4408c6d0 0x7fec4408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36434/0 (socket says 172.21.15.16:36434) 2019-12-05T06:20:52.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.479+0000 7fec4a51c700 1 -- 172.21.15.16:0/1102400698 learned_addr learned my addr 172.21.15.16:0/1102400698 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:52.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.479+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/1102400698 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4408fef0 0x7fec4408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:52.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.480+0000 7fec4951a700 1 -- 172.21.15.16:0/1102400698 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2875718621 0 0) 0x7fec3c0014d0 con 0x7fec4408c6d0 2019-12-05T06:20:52.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.480+0000 7fec4951a700 1 -- 172.21.15.16:0/1102400698 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec2c002e80 con 0x7fec4408c6d0 2019-12-05T06:20:52.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.480+0000 7fec49d1b700 1 -- 172.21.15.16:0/1102400698 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4408fef0 msgr2=0x7fec4408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:20:52.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.480+0000 7fec49d1b700 1 --2- 172.21.15.16:0/1102400698 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4408fef0 0x7fec4408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:52.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.480+0000 7fec49d1b700 1 -- 172.21.15.16:0/1102400698 >> v1:172.21.15.16:6789/0 conn(0x7fec4408c6d0 legacy=0x7fec4408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.480+0000 7fec49d1b700 1 -- 172.21.15.16:0/1102400698 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4408e620 con 0x7fec4408d2c0 2019-12-05T06:20:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.480+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/1102400698 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4408fef0 0x7fec4408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:20:52.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.480+0000 7fec49d1b700 1 --2- 172.21.15.16:0/1102400698 >> v2:172.21.15.16:3300/0 conn(0x7fec4408d2c0 0x7fec4408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fec3800af50 tx=0x7fec3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.480+0000 7fec49d1b700 1 --2- 172.21.15.16:0/1102400698 >> v2:172.21.15.16:3300/0 conn(0x7fec4408d2c0 0x7fec4408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fec3800af50 tx=0x7fec3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.480+0000 7fec49d1b700 1 --2- 172.21.15.16:0/1102400698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec4408d2c0 0x7fec4408ef20 secure :-1 s=READY pgs=601 cs=0 l=1 rx=0x7fec3800af50 tx=0x7fec3800b760).ready entity=mon.0 client_cookie=12fcc70dd6ee092d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.481+0000 7fec49d1b700 1 --2- 172.21.15.16:0/1102400698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec4408d2c0 0x7fec4408ef20 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7fec3800af50 tx=0x7fec3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.481+0000 7fec49d1b700 1 --2- 172.21.15.16:0/1102400698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec4408d2c0 0x7fec4408ef20 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7fec3800af50 tx=0x7fec3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.481+0000 7fec4951a700 1 -- 172.21.15.16:0/1102400698 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec38007210 con 0x7fec4408d2c0 2019-12-05T06:20:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.481+0000 7fec49d1b700 1 --2- 172.21.15.16:0/1102400698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec4408d2c0 0x7fec4408ef20 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7fec3800af50 tx=0x7fec3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.481+0000 7fec4951a700 1 -- 172.21.15.16:0/1102400698 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec380099b0 con 0x7fec4408d2c0 2019-12-05T06:20:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.481+0000 7fec4951a700 1 -- 172.21.15.16:0/1102400698 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec3800a080 con 0x7fec4408d2c0 2019-12-05T06:20:52.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.481+0000 7fec4bf7f700 1 -- 172.21.15.16:0/1102400698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec4408d2c0 msgr2=0x7fec4408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.481+0000 7fec4bf7f700 1 --2- 172.21.15.16:0/1102400698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec4408d2c0 0x7fec4408ef20 secure :-1 s=READY pgs=601 cs=0 l=1 rx=0x7fec3800af50 tx=0x7fec3800b760).stop 2019-12-05T06:20:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.483+0000 7fec4bf7f700 1 -- 172.21.15.16:0/1102400698 shutdown_connections 2019-12-05T06:20:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.483+0000 7fec4bf7f700 1 --2- 172.21.15.16:0/1102400698 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec4408d2c0 0x7fec4408ef20 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.483+0000 7fec4bf7f700 1 --2- 172.21.15.16:0/1102400698 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4408fef0 0x7fec4408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:20:52.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.483+0000 7fec4bf7f700 1 -- 172.21.15.16:0/1102400698 shutdown_connections 2019-12-05T06:20:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.483+0000 7fec4bf7f700 1 -- 172.21.15.16:0/1102400698 wait complete. 2019-12-05T06:20:52.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.483+0000 7fec4bf7f700 1 -- 172.21.15.16:0/1102400698 >> 172.21.15.16:0/1102400698 conn(0x7fec44061940 msgr2=0x7fec44096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.488+0000 7fec4bf7f700 1 Processor -- start 2019-12-05T06:20:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.489+0000 7fec4bf7f700 1 -- start start 2019-12-05T06:20:52.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.490+0000 7fec4bf7f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4414d400 0x7fec4414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:52.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.490+0000 7fec4bf7f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec44132530 0x7fec44157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:52.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.490+0000 7fec4bf7f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec44098da0 con 0x7fec44132d00 2019-12-05T06:20:52.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.490+0000 7fec4bf7f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec4409bb50 con 0x7fec44132530 2019-12-05T06:20:52.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.490+0000 7fec4bf7f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec4409a030 con 0x7fec4414d400 2019-12-05T06:20:52.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.490+0000 7fec4ad1d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fec44132530 0x7fec44157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:52.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.490+0000 7fec49d1b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fec44132d00 0x7fec44131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36442/0 (socket says 172.21.15.16:36442) 2019-12-05T06:20:52.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.490+0000 7fec49d1b700 1 -- 172.21.15.16:0/191359871 learned_addr learned my addr 172.21.15.16:0/191359871 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:20:52.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.490+0000 7fec4a51c700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4414d400 0x7fec4414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:52.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.491+0000 7fec4ad1d700 1 -- 172.21.15.16:0/191359871 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4414d400 msgr2=0x7fec4414d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:52.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.491+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4414d400 0x7fec4414d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:52.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.491+0000 7fec4ad1d700 1 -- 172.21.15.16:0/191359871 >> v1:172.21.15.16:6789/0 conn(0x7fec44132d00 legacy=0x7fec44131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:52.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.491+0000 7fec42ffd700 1 -- 172.21.15.16:0/191359871 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 959696170 0 0) 0x7fec3800ae90 con 0x7fec44132d00 2019-12-05T06:20:52.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.491+0000 7fec4ad1d700 1 -- 172.21.15.16:0/191359871 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec4409b0f0 con 0x7fec44132530 2019-12-05T06:20:52.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.491+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> v2:172.21.15.16:3300/0 conn(0x7fec44132530 0x7fec44157100 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.491+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> v2:172.21.15.16:3300/0 conn(0x7fec44132530 0x7fec44157100 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.491+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 0x7fec44157100 secure :-1 s=READY pgs=603 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.492+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 0x7fec44157100 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.492+0000 7fec42ffd700 1 -- 172.21.15.16:0/191359871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec3400ab90 con 0x7fec44132530 2019-12-05T06:20:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.492+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 0x7fec44157100 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.492+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 0x7fec44157100 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.492+0000 7fec42ffd700 1 -- 172.21.15.16:0/191359871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec340097a0 con 0x7fec44132530 2019-12-05T06:20:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.492+0000 7fec42ffd700 1 -- 172.21.15.16:0/191359871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec34009e70 con 0x7fec44132530 2019-12-05T06:20:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.492+0000 7fec4bf7f700 1 -- 172.21.15.16:0/191359871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec4409b0f0 con 0x7fec44132530 2019-12-05T06:20:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.492+0000 7fec4bf7f700 1 -- 172.21.15.16:0/191359871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec4409b0f0 con 0x7fec44132530 2019-12-05T06:20:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.493+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 0x7fec44157100 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.494+0000 7fec42ffd700 1 -- 172.21.15.16:0/191359871 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fec34009990 con 0x7fec44132530 2019-12-05T06:20:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.494+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 0x7fec44157100 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.494+0000 7fec42ffd700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fec24023630 0x7fec24025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:20:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.494+0000 7fec42ffd700 1 -- 172.21.15.16:0/191359871 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7fec34045d40 con 0x7fec44132530 2019-12-05T06:20:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.494+0000 7fec4a51c700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fec24023630 0x7fec24025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:20:52.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.495+0000 7fec4a51c700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fec24023630 0x7fec24025ac0 crc :-1 s=READY pgs=266 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:20:52.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.508+0000 7fec4bf7f700 1 -- 172.21.15.16:0/191359871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec4409b0f0 con 0x7fec44132530 2019-12-05T06:20:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.515+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 0x7fec44157100 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.516+0000 7fec42ffd700 1 -- 172.21.15.16:0/191359871 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fec34000a70 con 0x7fec44132530 2019-12-05T06:20:52.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.755+0000 7fec4bf7f700 1 -- 172.21.15.16:0/191359871 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec4409b0f0 con 0x7fec44132530 2019-12-05T06:20:52.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.756+0000 7fec4ad1d700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 0x7fec44157100 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:20:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.756+0000 7fec42ffd700 1 -- 172.21.15.16:0/191359871 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fec3400a200 con 0x7fec44132530 2019-12-05T06:20:52.746 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 77 2019-12-05T06:20:52.746 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:20:52.746 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:20:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.758+0000 7fec4bf7f700 1 -- 172.21.15.16:0/191359871 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fec24023630 msgr2=0x7fec24025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.758+0000 7fec4bf7f700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fec24023630 0x7fec24025ac0 crc :-1 s=READY pgs=266 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.758+0000 7fec4bf7f700 1 -- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 msgr2=0x7fec44157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:20:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.758+0000 7fec4bf7f700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 0x7fec44157100 secure :-1 s=READY pgs=603 cs=0 l=1 rx=0x7fec34010300 tx=0x7fec340106a0).stop 2019-12-05T06:20:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.759+0000 7fec4bf7f700 1 -- 172.21.15.16:0/191359871 shutdown_connections 2019-12-05T06:20:52.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.759+0000 7fec4bf7f700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fec24023630 0x7fec24025ac0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.759+0000 7fec4bf7f700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fec44132530 0x7fec44157100 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.759+0000 7fec4bf7f700 1 --2- 172.21.15.16:0/191359871 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fec4414d400 0x7fec4414d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:20:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.759+0000 7fec4bf7f700 1 -- 172.21.15.16:0/191359871 shutdown_connections 2019-12-05T06:20:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.759+0000 7fec4bf7f700 1 -- 172.21.15.16:0/191359871 wait complete. 2019-12-05T06:20:52.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:20:52.759+0000 7fec4bf7f700 1 -- 172.21.15.16:0/191359871 >> 172.21.15.16:0/191359871 conn(0x7fec440714f0 msgr2=0x7fec4409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:20:52.759 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:20:52.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 33 secs before thrashing 2019-12-05T06:21:02.252 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:21:02.257 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:21:25.760 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:26.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.042+0000 7f6c2d53a700 1 Processor -- start 2019-12-05T06:21:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.042+0000 7f6c2d53a700 1 -- start start 2019-12-05T06:21:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.044+0000 7f6c2d53a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 0x7f6c2808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.044+0000 7f6c2d53a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6c2808f720 0x7f6c2808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.044+0000 7f6c2d53a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c280958e0 con 0x7f6c2808fef0 2019-12-05T06:21:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.044+0000 7f6c2d53a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c280a3f70 con 0x7f6c2808f720 2019-12-05T06:21:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.044+0000 7f6c2d53a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2808d830 con 0x7f6c2808c6d0 2019-12-05T06:21:26.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.045+0000 7f6c26ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6c2808fef0 0x7f6c2808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36466/0 (socket says 172.21.15.16:36466) 2019-12-05T06:21:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.045+0000 7f6c26ffd700 1 -- 172.21.15.16:0/4049572843 learned_addr learned my addr 172.21.15.16:0/4049572843 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.045+0000 7f6c27fff700 1 --2- 172.21.15.16:0/4049572843 >> v2:172.21.15.76:3300/0 conn(0x7f6c2808f720 0x7f6c2808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.045+0000 7f6c277fe700 1 --2- 172.21.15.16:0/4049572843 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 0x7f6c2808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.045+0000 7f6c267fc700 1 -- 172.21.15.16:0/4049572843 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716418665 0 0) 0x7f6c10001150 con 0x7f6c2808fef0 2019-12-05T06:21:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.045+0000 7f6c267fc700 1 -- 172.21.15.16:0/4049572843 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c14002e80 con 0x7f6c2808fef0 2019-12-05T06:21:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.045+0000 7f6c277fe700 1 -- 172.21.15.16:0/4049572843 >> v2:172.21.15.76:3300/0 conn(0x7f6c2808f720 msgr2=0x7f6c2808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:26.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.045+0000 7f6c277fe700 1 --2- 172.21.15.16:0/4049572843 >> v2:172.21.15.76:3300/0 conn(0x7f6c2808f720 0x7f6c2808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.045+0000 7f6c277fe700 1 -- 172.21.15.16:0/4049572843 >> v1:172.21.15.16:6789/0 conn(0x7f6c2808fef0 legacy=0x7f6c2808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.046+0000 7f6c277fe700 1 -- 172.21.15.16:0/4049572843 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c2808fbc0 con 0x7f6c2808c6d0 2019-12-05T06:21:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.046+0000 7f6c27fff700 1 --2- 172.21.15.16:0/4049572843 >> v2:172.21.15.76:3300/0 conn(0x7f6c2808f720 0x7f6c2808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:21:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.046+0000 7f6c277fe700 1 --2- 172.21.15.16:0/4049572843 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 0x7f6c2808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6c2000b2d0 tx=0x7f6c2000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.046+0000 7f6c277fe700 1 --2- 172.21.15.16:0/4049572843 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 0x7f6c2808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6c2000b2d0 tx=0x7f6c2000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.046+0000 7f6c277fe700 1 --2- 172.21.15.16:0/4049572843 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 0x7f6c2808cac0 secure :-1 s=READY pgs=151 cs=0 l=1 rx=0x7f6c2000b2d0 tx=0x7f6c2000bae0).ready entity=mon.2 client_cookie=e8907465255514e2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:26.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.046+0000 7f6c277fe700 1 --2- 172.21.15.16:0/4049572843 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 0x7f6c2808cac0 secure :-1 s=THROTTLE_DONE pgs=151 cs=0 l=1 rx=0x7f6c2000b2d0 tx=0x7f6c2000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.046+0000 7f6c267fc700 1 -- 172.21.15.16:0/4049572843 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c20007590 con 0x7f6c2808c6d0 2019-12-05T06:21:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.046+0000 7f6c277fe700 1 --2- 172.21.15.16:0/4049572843 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 0x7f6c2808cac0 secure :-1 s=THROTTLE_DONE pgs=151 cs=0 l=1 rx=0x7f6c2000b2d0 tx=0x7f6c2000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.046+0000 7f6c277fe700 1 --2- 172.21.15.16:0/4049572843 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 0x7f6c2808cac0 secure :-1 s=THROTTLE_DONE pgs=151 cs=0 l=1 rx=0x7f6c2000b2d0 tx=0x7f6c2000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.046+0000 7f6c267fc700 1 -- 172.21.15.16:0/4049572843 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c20009d30 con 0x7f6c2808c6d0 2019-12-05T06:21:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.047+0000 7f6c267fc700 1 -- 172.21.15.16:0/4049572843 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c2000a400 con 0x7f6c2808c6d0 2019-12-05T06:21:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.047+0000 7f6c2d53a700 1 -- 172.21.15.16:0/4049572843 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 msgr2=0x7f6c2808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.047+0000 7f6c2d53a700 1 --2- 172.21.15.16:0/4049572843 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 0x7f6c2808cac0 secure :-1 s=READY pgs=151 cs=0 l=1 rx=0x7f6c2000b2d0 tx=0x7f6c2000bae0).stop 2019-12-05T06:21:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.048+0000 7f6c2d53a700 1 -- 172.21.15.16:0/4049572843 shutdown_connections 2019-12-05T06:21:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.048+0000 7f6c2d53a700 1 --2- 172.21.15.16:0/4049572843 >> v2:172.21.15.76:3300/0 conn(0x7f6c2808f720 0x7f6c2808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.048+0000 7f6c2d53a700 1 --2- 172.21.15.16:0/4049572843 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2808c6d0 0x7f6c2808cac0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.048+0000 7f6c2d53a700 1 -- 172.21.15.16:0/4049572843 shutdown_connections 2019-12-05T06:21:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.048+0000 7f6c2d53a700 1 -- 172.21.15.16:0/4049572843 wait complete. 2019-12-05T06:21:26.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.048+0000 7f6c2d53a700 1 -- 172.21.15.16:0/4049572843 >> 172.21.15.16:0/4049572843 conn(0x7f6c28061940 msgr2=0x7f6c28096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.053+0000 7f6c2d53a700 1 Processor -- start 2019-12-05T06:21:26.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.054+0000 7f6c2d53a700 1 -- start start 2019-12-05T06:21:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.055+0000 7f6c2d53a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2812ca70 0x7f6c2809d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.055+0000 7f6c2d53a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6c2809e7c0 0x7f6c2809dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.055+0000 7f6c2d53a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c2809e270 0x7f6c2806dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:26.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.055+0000 7f6c2d53a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c281311f0 con 0x7f6c2809e270 2019-12-05T06:21:26.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.055+0000 7f6c2d53a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c281314c0 con 0x7f6c2809e7c0 2019-12-05T06:21:26.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.055+0000 7f6c2d53a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2812fff0 con 0x7f6c2812ca70 2019-12-05T06:21:26.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c27fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c2809e270 0x7f6c2806dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c27fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6c2809e270 0x7f6c2806dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51306/0 (socket says 172.21.15.16:51306) 2019-12-05T06:21:26.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c27fff700 1 -- 172.21.15.16:0/1594354414 learned_addr learned my addr 172.21.15.16:0/1594354414 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:26.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c26ffd700 1 --2- 172.21.15.16:0/1594354414 >> v2:172.21.15.76:3300/0 conn(0x7f6c2809e7c0 0x7f6c2809dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c277fe700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2812ca70 0x7f6c2809d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c27fff700 1 -- 172.21.15.16:0/1594354414 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2812ca70 msgr2=0x7f6c2809d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2812ca70 0x7f6c2809d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c27fff700 1 -- 172.21.15.16:0/1594354414 >> v2:172.21.15.76:3300/0 conn(0x7f6c2809e7c0 msgr2=0x7f6c2809dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> v2:172.21.15.76:3300/0 conn(0x7f6c2809e7c0 0x7f6c2809dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c27fff700 1 -- 172.21.15.16:0/1594354414 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c281322b0 con 0x7f6c2809e270 2019-12-05T06:21:26.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.056+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> v2:172.21.15.16:3300/0 conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.057+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> v2:172.21.15.16:3300/0 conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.057+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=READY pgs=607 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.057+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=THROTTLE_DONE pgs=607 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.057+0000 7f6c1ffff700 1 -- 172.21.15.16:0/1594354414 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c1800aa60 con 0x7f6c2809e270 2019-12-05T06:21:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.057+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=THROTTLE_DONE pgs=607 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.057+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=THROTTLE_DONE pgs=607 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.057+0000 7f6c1ffff700 1 -- 172.21.15.16:0/1594354414 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c180095b0 con 0x7f6c2809e270 2019-12-05T06:21:26.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.057+0000 7f6c2d53a700 1 -- 172.21.15.16:0/1594354414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c281322b0 con 0x7f6c2809e270 2019-12-05T06:21:26.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.057+0000 7f6c1ffff700 1 -- 172.21.15.16:0/1594354414 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c1800a750 con 0x7f6c2809e270 2019-12-05T06:21:26.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.057+0000 7f6c2d53a700 1 -- 172.21.15.16:0/1594354414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c281322b0 con 0x7f6c2809e270 2019-12-05T06:21:26.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.058+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=THROTTLE_DONE pgs=607 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.059+0000 7f6c1ffff700 1 -- 172.21.15.16:0/1594354414 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6c180097a0 con 0x7f6c2809e270 2019-12-05T06:21:26.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.059+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=THROTTLE_DONE pgs=607 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.060+0000 7f6c1ffff700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c0c023630 0x7f6c0c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:26.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.060+0000 7f6c1ffff700 1 -- 172.21.15.16:0/1594354414 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f6c18045d10 con 0x7f6c2809e270 2019-12-05T06:21:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.060+0000 7f6c277fe700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c0c023630 0x7f6c0c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.060+0000 7f6c277fe700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c0c023630 0x7f6c0c025ac0 crc :-1 s=READY pgs=270 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.074+0000 7f6c2d53a700 1 -- 172.21.15.16:0/1594354414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c281322b0 con 0x7f6c2809e270 2019-12-05T06:21:26.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.081+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=THROTTLE_DONE pgs=607 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.081+0000 7f6c1ffff700 1 -- 172.21.15.16:0/1594354414 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6c18000a70 con 0x7f6c2809e270 2019-12-05T06:21:26.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.328+0000 7f6c2d53a700 1 -- 172.21.15.16:0/1594354414 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6c281322b0 con 0x7f6c2809e270 2019-12-05T06:21:26.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.329+0000 7f6c27fff700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=THROTTLE_DONE pgs=607 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.329+0000 7f6c1ffff700 1 -- 172.21.15.16:0/1594354414 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f6c1800a050 con 0x7f6c2809e270 2019-12-05T06:21:26.319 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 77 2019-12-05T06:21:26.319 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:26.320 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 -- 172.21.15.16:0/1594354414 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c0c023630 msgr2=0x7f6c0c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c0c023630 0x7f6c0c025ac0 crc :-1 s=READY pgs=270 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 -- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 msgr2=0x7f6c2806dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 0x7f6c2806dc50 secure :-1 s=READY pgs=607 cs=0 l=1 rx=0x7f6c180103b0 tx=0x7f6c180102b0).stop 2019-12-05T06:21:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 -- 172.21.15.16:0/1594354414 shutdown_connections 2019-12-05T06:21:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6c0c023630 0x7f6c0c025ac0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6c2809e270 0x7f6c2806dc50 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 --2- 172.21.15.16:0/1594354414 >> v2:172.21.15.76:3300/0 conn(0x7f6c2809e7c0 0x7f6c2809dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 --2- 172.21.15.16:0/1594354414 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6c2812ca70 0x7f6c2809d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 -- 172.21.15.16:0/1594354414 shutdown_connections 2019-12-05T06:21:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 -- 172.21.15.16:0/1594354414 wait complete. 2019-12-05T06:21:26.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.334+0000 7f6c2d53a700 1 -- 172.21.15.16:0/1594354414 >> 172.21.15.16:0/1594354414 conn(0x7f6c280a2850 msgr2=0x7f6c2806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:26.332 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:21:26.332 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:21:26.332 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:21:26.359 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:21:26.359 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.603+0000 7f74275b0700 1 Processor -- start 2019-12-05T06:21:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.603+0000 7f74275b0700 1 -- start start 2019-12-05T06:21:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.605+0000 7f74275b0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f742008c6d0 0x7f742008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.605+0000 7f74275b0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f742008f720 0x7f742008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:26.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.605+0000 7f74275b0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74200958e0 con 0x7f742008fef0 2019-12-05T06:21:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.605+0000 7f74275b0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74200a3f70 con 0x7f742008f720 2019-12-05T06:21:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.605+0000 7f74275b0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742008d830 con 0x7f742008c6d0 2019-12-05T06:21:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.605+0000 7f742634e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f742008f720 0x7f742008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.605+0000 7f742634e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f742008f720 0x7f742008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51316/0 (socket says 172.21.15.16:51316) 2019-12-05T06:21:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.605+0000 7f742634e700 1 -- 172.21.15.16:0/1838244437 learned_addr learned my addr 172.21.15.16:0/1838244437 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.605+0000 7f7425b4d700 1 --2- 172.21.15.16:0/1838244437 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f742008c6d0 0x7f742008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.606+0000 7f7424b4b700 1 -- 172.21.15.16:0/1838244437 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2154751230 0 0) 0x7f7414001030 con 0x7f742008fef0 2019-12-05T06:21:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.606+0000 7f742634e700 1 -- 172.21.15.16:0/1838244437 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f742008c6d0 msgr2=0x7f742008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.606+0000 7f742634e700 1 --2- 172.21.15.16:0/1838244437 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f742008c6d0 0x7f742008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.606+0000 7f742634e700 1 -- 172.21.15.16:0/1838244437 >> v1:172.21.15.76:6789/0 conn(0x7f742008fef0 legacy=0x7f742008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.606+0000 7f742634e700 1 -- 172.21.15.16:0/1838244437 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f742008fbc0 con 0x7f742008f720 2019-12-05T06:21:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.606+0000 7f742634e700 1 --2- 172.21.15.16:0/1838244437 >> v2:172.21.15.16:3300/0 conn(0x7f742008f720 0x7f742008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f741000f730 tx=0x7f7410007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.606+0000 7f742634e700 1 --2- 172.21.15.16:0/1838244437 >> v2:172.21.15.16:3300/0 conn(0x7f742008f720 0x7f742008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f741000f730 tx=0x7f7410007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.606+0000 7f742634e700 1 --2- 172.21.15.16:0/1838244437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f742008f720 0x7f742008d000 secure :-1 s=READY pgs=608 cs=0 l=1 rx=0x7f741000f730 tx=0x7f7410007d50).ready entity=mon.0 client_cookie=11046402aa728354 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.606+0000 7f742634e700 1 --2- 172.21.15.16:0/1838244437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f742008f720 0x7f742008d000 secure :-1 s=THROTTLE_DONE pgs=608 cs=0 l=1 rx=0x7f741000f730 tx=0x7f7410007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.607+0000 7f7424b4b700 1 -- 172.21.15.16:0/1838244437 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7410000a70 con 0x7f742008f720 2019-12-05T06:21:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.607+0000 7f742634e700 1 --2- 172.21.15.16:0/1838244437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f742008f720 0x7f742008d000 secure :-1 s=THROTTLE_DONE pgs=608 cs=0 l=1 rx=0x7f741000f730 tx=0x7f7410007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.607+0000 7f742634e700 1 --2- 172.21.15.16:0/1838244437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f742008f720 0x7f742008d000 secure :-1 s=THROTTLE_DONE pgs=608 cs=0 l=1 rx=0x7f741000f730 tx=0x7f7410007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.607+0000 7f7424b4b700 1 -- 172.21.15.16:0/1838244437 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74100010e0 con 0x7f742008f720 2019-12-05T06:21:26.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.607+0000 7f7424b4b700 1 -- 172.21.15.16:0/1838244437 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74100012d0 con 0x7f742008f720 2019-12-05T06:21:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.607+0000 7f74275b0700 1 -- 172.21.15.16:0/1838244437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f742008f720 msgr2=0x7f742008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.607+0000 7f74275b0700 1 --2- 172.21.15.16:0/1838244437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f742008f720 0x7f742008d000 secure :-1 s=READY pgs=608 cs=0 l=1 rx=0x7f741000f730 tx=0x7f7410007d50).stop 2019-12-05T06:21:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.608+0000 7f74275b0700 1 -- 172.21.15.16:0/1838244437 shutdown_connections 2019-12-05T06:21:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.608+0000 7f74275b0700 1 --2- 172.21.15.16:0/1838244437 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f742008f720 0x7f742008d000 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.608+0000 7f74275b0700 1 --2- 172.21.15.16:0/1838244437 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f742008c6d0 0x7f742008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.609+0000 7f74275b0700 1 -- 172.21.15.16:0/1838244437 shutdown_connections 2019-12-05T06:21:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.609+0000 7f74275b0700 1 -- 172.21.15.16:0/1838244437 wait complete. 2019-12-05T06:21:26.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.609+0000 7f74275b0700 1 -- 172.21.15.16:0/1838244437 >> 172.21.15.16:0/1838244437 conn(0x7f7420061940 msgr2=0x7f7420096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.613+0000 7f74275b0700 1 Processor -- start 2019-12-05T06:21:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.613+0000 7f74275b0700 1 -- start start 2019-12-05T06:21:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.615+0000 7f74275b0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7420130210 0x7f7420081220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.615+0000 7f74275b0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7420088c30 con 0x7f742014da50 2019-12-05T06:21:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.615+0000 7f74275b0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7420088730 con 0x7f74201309e0 2019-12-05T06:21:26.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.615+0000 7f74275b0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f742008a8f0 con 0x7f7420130210 2019-12-05T06:21:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.615+0000 7f7425b4d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f742014da50 0x7f742012fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36488/0 (socket says 172.21.15.16:36488) 2019-12-05T06:21:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.615+0000 7f7425b4d700 1 -- 172.21.15.16:0/3824569420 learned_addr learned my addr 172.21.15.16:0/3824569420 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.615+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7420130210 0x7f7420081220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.615+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 412363133 0 0) 0x7f7418001430 con 0x7f742014da50 2019-12-05T06:21:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7400002e80 con 0x7f742014da50 2019-12-05T06:21:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132302627 0 0) 0x7f7414001070 con 0x7f74201309e0 2019-12-05T06:21:26.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74000030b0 con 0x7f74201309e0 2019-12-05T06:21:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3206667013 0 0) 0x7f7418001630 con 0x7f742014da50 2019-12-05T06:21:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74000033c0 con 0x7f742014da50 2019-12-05T06:21:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f7418001c00 con 0x7f742014da50 2019-12-05T06:21:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7420130210 msgr2=0x7f7420081220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7420130210 0x7f7420081220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 >> v1:172.21.15.76:6789/0 conn(0x7f74201309e0 legacy=0x7f74201551c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 >> v1:172.21.15.16:6789/0 conn(0x7f742014da50 legacy=0x7f742012fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74000039d0 0x7f7400003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f74000057b0 0x7f7400005c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7400004c30 con 0x7f7400006160 2019-12-05T06:21:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7400004e60 con 0x7f74000039d0 2019-12-05T06:21:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7400005090 con 0x7f74000057b0 2019-12-05T06:21:26.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.616+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7420130210 0x7f7420081220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:21:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.617+0000 7f7425b4d700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74000039d0 0x7f7400003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.617+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.617+0000 7f742534c700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f74000057b0 0x7f7400005c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.617+0000 7f742634e700 1 -- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f74000057b0 msgr2=0x7f7400005c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.617+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f74000057b0 0x7f7400005c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.617+0000 7f742634e700 1 -- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74000039d0 msgr2=0x7f7400003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.617+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f74000039d0 0x7f7400003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.617+0000 7f742634e700 1 -- 172.21.15.16:0/3824569420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7400005340 con 0x7f7400006160 2019-12-05T06:21:26.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.617+0000 7f742534c700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f74000057b0 0x7f7400005c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:21:26.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.617+0000 7f74275b0700 1 -- 172.21.15.16:0/3824569420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f742012f110 con 0x7f7400006160 2019-12-05T06:21:26.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.618+0000 7f74275b0700 1 -- 172.21.15.16:0/3824569420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7420088c30 con 0x7f7400006160 2019-12-05T06:21:26.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.618+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.618+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.618+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=READY pgs=610 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:26.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.618+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=THROTTLE_DONE pgs=610 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.618+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74100078d0 con 0x7f7400006160 2019-12-05T06:21:26.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.618+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=THROTTLE_DONE pgs=610 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.618+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74100078d0 con 0x7f7400006160 2019-12-05T06:21:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.618+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=THROTTLE_DONE pgs=610 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.618+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74100014a0 con 0x7f7400006160 2019-12-05T06:21:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.619+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=THROTTLE_DONE pgs=610 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.620+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f74100010c0 con 0x7f7400006160 2019-12-05T06:21:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.620+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=THROTTLE_DONE pgs=610 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.620+0000 7f741e7fc700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f740002bad0 0x7f740002df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:26.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.620+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f7410089cc0 con 0x7f7400006160 2019-12-05T06:21:26.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.621+0000 7f7425b4d700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f740002bad0 0x7f740002df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:26.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.621+0000 7f7425b4d700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f740002bad0 0x7f740002df00 crc :-1 s=READY pgs=271 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:26.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.634+0000 7f74275b0700 1 -- 172.21.15.16:0/3824569420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73fc002e60 con 0x7f7400006160 2019-12-05T06:21:26.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.640+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=THROTTLE_DONE pgs=610 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.640+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7410000eb0 con 0x7f7400006160 2019-12-05T06:21:26.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.899+0000 7f74275b0700 1 -- 172.21.15.16:0/3824569420 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f73fc001d10 con 0x7f7400006160 2019-12-05T06:21:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.900+0000 7f742634e700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=THROTTLE_DONE pgs=610 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.900+0000 7f741e7fc700 1 -- 172.21.15.16:0/3824569420 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f74100010c0 con 0x7f7400006160 2019-12-05T06:21:26.889 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 77 2019-12-05T06:21:26.890 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:26.890 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.903+0000 7f74275b0700 1 -- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f740002bad0 msgr2=0x7f740002df00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.903+0000 7f74275b0700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f740002bad0 0x7f740002df00 crc :-1 s=READY pgs=271 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:26.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.903+0000 7f74275b0700 1 -- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 msgr2=0x7f74000046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.903+0000 7f74275b0700 1 --2- 172.21.15.16:0/3824569420 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7400006160 0x7f74000046f0 secure :-1 s=READY pgs=610 cs=0 l=1 rx=0x7f7410054410 tx=0x7f7410054290).stop 2019-12-05T06:21:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.903+0000 7f742634e700 1 -- 172.21.15.16:0/3824569420 reap_dead start 2019-12-05T06:21:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.903+0000 7f742634e700 1 -- 172.21.15.16:0/3824569420 reap_dead start 2019-12-05T06:21:26.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.903+0000 7f74275b0700 1 -- 172.21.15.16:0/3824569420 shutdown_connections 2019-12-05T06:21:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.903+0000 7f74275b0700 1 -- 172.21.15.16:0/3824569420 shutdown_connections 2019-12-05T06:21:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.903+0000 7f74275b0700 1 -- 172.21.15.16:0/3824569420 wait complete. 2019-12-05T06:21:26.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:26.903+0000 7f74275b0700 1 -- 172.21.15.16:0/3824569420 >> 172.21.15.16:0/3824569420 conn(0x7f742009de30 msgr2=0x7f7420070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:26.903 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:21:28.904 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:29.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.191+0000 7f14b6968700 1 Processor -- start 2019-12-05T06:21:29.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.191+0000 7f14b6968700 1 -- start start 2019-12-05T06:21:29.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.192+0000 7f14b6968700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14b008d2c0 0x7f14b008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:29.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.192+0000 7f14b6968700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b00958e0 con 0x7f14b008fef0 2019-12-05T06:21:29.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.192+0000 7f14b6968700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b00629e0 con 0x7f14b008c6d0 2019-12-05T06:21:29.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.192+0000 7f14b6968700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b008df90 con 0x7f14b008d2c0 2019-12-05T06:21:29.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.193+0000 7f14affff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14b008d2c0 0x7f14b008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:29.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.193+0000 7f14b5706700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f14b008fef0 0x7f14b008f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36512/0 (socket says 172.21.15.16:36512) 2019-12-05T06:21:29.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.193+0000 7f14b5706700 1 -- 172.21.15.16:0/3884192917 learned_addr learned my addr 172.21.15.16:0/3884192917 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:29.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.193+0000 7f14b4f05700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f14b008c6d0 0x7f14b008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57810/0 (socket says 172.21.15.16:57810) 2019-12-05T06:21:29.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14af7fe700 1 -- 172.21.15.16:0/3884192917 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855717851 0 0) 0x7f14a8001400 con 0x7f14b008c6d0 2019-12-05T06:21:29.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14affff700 1 -- 172.21.15.16:0/3884192917 >> v1:172.21.15.76:6789/0 conn(0x7f14b008c6d0 legacy=0x7f14b008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14affff700 1 -- 172.21.15.16:0/3884192917 >> v1:172.21.15.16:6789/0 conn(0x7f14b008fef0 legacy=0x7f14b008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14affff700 1 -- 172.21.15.16:0/3884192917 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b008e620 con 0x7f14b008d2c0 2019-12-05T06:21:29.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14affff700 1 --2- 172.21.15.16:0/3884192917 >> v2:172.21.15.16:3300/0 conn(0x7f14b008d2c0 0x7f14b008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f14a000af50 tx=0x7f14a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14affff700 1 --2- 172.21.15.16:0/3884192917 >> v2:172.21.15.16:3300/0 conn(0x7f14b008d2c0 0x7f14b008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f14a000af50 tx=0x7f14a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14affff700 1 --2- 172.21.15.16:0/3884192917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b008d2c0 0x7f14b008ef20 secure :-1 s=READY pgs=612 cs=0 l=1 rx=0x7f14a000af50 tx=0x7f14a000b760).ready entity=mon.0 client_cookie=f60a85e90194bedd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:29.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14affff700 1 --2- 172.21.15.16:0/3884192917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b008d2c0 0x7f14b008ef20 secure :-1 s=THROTTLE_DONE pgs=612 cs=0 l=1 rx=0x7f14a000af50 tx=0x7f14a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14affff700 1 --2- 172.21.15.16:0/3884192917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b008d2c0 0x7f14b008ef20 secure :-1 s=THROTTLE_DONE pgs=612 cs=0 l=1 rx=0x7f14a000af50 tx=0x7f14a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14af7fe700 1 -- 172.21.15.16:0/3884192917 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14a0007210 con 0x7f14b008d2c0 2019-12-05T06:21:29.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14affff700 1 --2- 172.21.15.16:0/3884192917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b008d2c0 0x7f14b008ef20 secure :-1 s=THROTTLE_DONE pgs=612 cs=0 l=1 rx=0x7f14a000af50 tx=0x7f14a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.194+0000 7f14af7fe700 1 -- 172.21.15.16:0/3884192917 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14a00099b0 con 0x7f14b008d2c0 2019-12-05T06:21:29.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.195+0000 7f14af7fe700 1 -- 172.21.15.16:0/3884192917 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14a000a080 con 0x7f14b008d2c0 2019-12-05T06:21:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.195+0000 7f14b6968700 1 -- 172.21.15.16:0/3884192917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b008d2c0 msgr2=0x7f14b008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.195+0000 7f14b6968700 1 --2- 172.21.15.16:0/3884192917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b008d2c0 0x7f14b008ef20 secure :-1 s=READY pgs=612 cs=0 l=1 rx=0x7f14a000af50 tx=0x7f14a000b760).stop 2019-12-05T06:21:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.196+0000 7f14b6968700 1 -- 172.21.15.16:0/3884192917 shutdown_connections 2019-12-05T06:21:29.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.196+0000 7f14b6968700 1 --2- 172.21.15.16:0/3884192917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b008d2c0 0x7f14b008ef20 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.197+0000 7f14b6968700 1 -- 172.21.15.16:0/3884192917 shutdown_connections 2019-12-05T06:21:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.197+0000 7f14b6968700 1 -- 172.21.15.16:0/3884192917 wait complete. 2019-12-05T06:21:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.197+0000 7f14b6968700 1 -- 172.21.15.16:0/3884192917 >> 172.21.15.16:0/3884192917 conn(0x7f14b0061940 msgr2=0x7f14b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:29.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.202+0000 7f14b6968700 1 Processor -- start 2019-12-05T06:21:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.202+0000 7f14b6968700 1 -- start start 2019-12-05T06:21:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.204+0000 7f14b6968700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f14b0088d00 0x7f14b0088530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.204+0000 7f14b6968700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14b0156e40 0x7f14b0159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.204+0000 7f14b6968700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14b00a0a30 con 0x7f14b00a1800 2019-12-05T06:21:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.204+0000 7f14b6968700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14b006ff50 con 0x7f14b0156e40 2019-12-05T06:21:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.204+0000 7f14b6968700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14b006f520 con 0x7f14b0088d00 2019-12-05T06:21:29.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.204+0000 7f14b5706700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14b0156e40 0x7f14b0159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.204+0000 7f14b5706700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f14b0156e40 0x7f14b0159240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51350/0 (socket says 172.21.15.16:51350) 2019-12-05T06:21:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.204+0000 7f14affff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f14b0088d00 0x7f14b0088530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.204+0000 7f14b5706700 1 -- 172.21.15.16:0/2685718194 learned_addr learned my addr 172.21.15.16:0/2685718194 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.205+0000 7f14ad7fa700 1 -- 172.21.15.16:0/2685718194 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205162548 0 0) 0x7f14a8001400 con 0x7f14b00a1800 2019-12-05T06:21:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.205+0000 7f14ad7fa700 1 -- 172.21.15.16:0/2685718194 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1494002e80 con 0x7f14b00a1800 2019-12-05T06:21:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.205+0000 7f14b5706700 1 -- 172.21.15.16:0/2685718194 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f14b0088d00 msgr2=0x7f14b0088530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.205+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f14b0088d00 0x7f14b0088530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.205+0000 7f14b5706700 1 -- 172.21.15.16:0/2685718194 >> v1:172.21.15.76:6789/0 conn(0x7f14b00a1800 legacy=0x7f14b0087d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.205+0000 7f14b5706700 1 -- 172.21.15.16:0/2685718194 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b00705e0 con 0x7f14b0156e40 2019-12-05T06:21:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.205+0000 7f14affff700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f14b0088d00 0x7f14b0088530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:21:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.205+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> v2:172.21.15.16:3300/0 conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.205+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> v2:172.21.15.16:3300/0 conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.205+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=READY pgs=613 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:29.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.206+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.206+0000 7f14ad7fa700 1 -- 172.21.15.16:0/2685718194 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1498000bc0 con 0x7f14b0156e40 2019-12-05T06:21:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.206+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.206+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.206+0000 7f14ad7fa700 1 -- 172.21.15.16:0/2685718194 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14980010e0 con 0x7f14b0156e40 2019-12-05T06:21:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.206+0000 7f14b6968700 1 -- 172.21.15.16:0/2685718194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b00705e0 con 0x7f14b0156e40 2019-12-05T06:21:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.206+0000 7f14ad7fa700 1 -- 172.21.15.16:0/2685718194 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1498001770 con 0x7f14b0156e40 2019-12-05T06:21:29.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.206+0000 7f14b6968700 1 -- 172.21.15.16:0/2685718194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b00705e0 con 0x7f14b0156e40 2019-12-05T06:21:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.207+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.208+0000 7f14ad7fa700 1 -- 172.21.15.16:0/2685718194 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f14980012d0 con 0x7f14b0156e40 2019-12-05T06:21:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.208+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.208+0000 7f14ad7fa700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1494023670 0x7f1494025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:29.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.208+0000 7f14ad7fa700 1 -- 172.21.15.16:0/2685718194 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f1498045dc0 con 0x7f14b0156e40 2019-12-05T06:21:29.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.209+0000 7f14b4f05700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1494023670 0x7f1494025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:29.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.209+0000 7f14b4f05700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1494023670 0x7f1494025ae0 crc :-1 s=READY pgs=273 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:29.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.222+0000 7f14b6968700 1 -- 172.21.15.16:0/2685718194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b00705e0 con 0x7f14b0156e40 2019-12-05T06:21:29.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.229+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.229+0000 7f14ad7fa700 1 -- 172.21.15.16:0/2685718194 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f149802ab40 con 0x7f14b0156e40 2019-12-05T06:21:29.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.485+0000 7f14b6968700 1 -- 172.21.15.16:0/2685718194 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f14b00705e0 con 0x7f14b0156e40 2019-12-05T06:21:29.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.486+0000 7f14b5706700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=THROTTLE_DONE pgs=613 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:29.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.486+0000 7f14ad7fa700 1 -- 172.21.15.16:0/2685718194 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f149802ab40 con 0x7f14b0156e40 2019-12-05T06:21:29.474 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 77 2019-12-05T06:21:29.475 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:29.475 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:29.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 -- 172.21.15.16:0/2685718194 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1494023670 msgr2=0x7f1494025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1494023670 0x7f1494025ae0 crc :-1 s=READY pgs=273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 -- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 msgr2=0x7f14b0159240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:29.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 0x7f14b0159240 secure :-1 s=READY pgs=613 cs=0 l=1 rx=0x7f14980103e0 tx=0x7f1498010780).stop 2019-12-05T06:21:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 -- 172.21.15.16:0/2685718194 shutdown_connections 2019-12-05T06:21:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1494023670 0x7f1494025ae0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f14b0156e40 0x7f14b0159240 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:29.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 --2- 172.21.15.16:0/2685718194 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f14b0088d00 0x7f14b0088530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:29.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 -- 172.21.15.16:0/2685718194 shutdown_connections 2019-12-05T06:21:29.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 -- 172.21.15.16:0/2685718194 wait complete. 2019-12-05T06:21:29.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:29.489+0000 7f14b6968700 1 -- 172.21.15.16:0/2685718194 >> 172.21.15.16:0/2685718194 conn(0x7f14b007aeb0 msgr2=0x7f14b00a2040 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:29.487 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:21:31.489 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:31.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.791+0000 7fd7f71c5700 1 Processor -- start 2019-12-05T06:21:31.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.792+0000 7fd7f71c5700 1 -- start start 2019-12-05T06:21:31.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.794+0000 7fd7f71c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7f008c6d0 0x7fd7f008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.794+0000 7fd7f71c5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd7f008fef0 0x7fd7f008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.794+0000 7fd7f71c5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f008fb80 con 0x7fd7f008ef20 2019-12-05T06:21:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.794+0000 7fd7f71c5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f008df90 con 0x7fd7f008c6d0 2019-12-05T06:21:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.794+0000 7fd7f71c5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f00a3f70 con 0x7fd7f008fef0 2019-12-05T06:21:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.794+0000 7fd7f5762700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7f008c6d0 0x7fd7f008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.794+0000 7fd7f5762700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7f008c6d0 0x7fd7f008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51360/0 (socket says 172.21.15.16:51360) 2019-12-05T06:21:31.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.794+0000 7fd7f5762700 1 -- 172.21.15.16:0/1292028457 learned_addr learned my addr 172.21.15.16:0/1292028457 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:31.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.794+0000 7fd7f4f61700 1 --2- 172.21.15.16:0/1292028457 >> v2:172.21.15.76:3300/0 conn(0x7fd7f008fef0 0x7fd7f008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:31.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7f5762700 1 -- 172.21.15.16:0/1292028457 >> v2:172.21.15.76:3300/0 conn(0x7fd7f008fef0 msgr2=0x7fd7f008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:31.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7f5762700 1 --2- 172.21.15.16:0/1292028457 >> v2:172.21.15.76:3300/0 conn(0x7fd7f008fef0 0x7fd7f008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:31.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7f5762700 1 -- 172.21.15.16:0/1292028457 >> v1:172.21.15.76:6789/0 conn(0x7fd7f008ef20 legacy=0x7fd7f008f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:31.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7f5762700 1 -- 172.21.15.16:0/1292028457 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f008d540 con 0x7fd7f008c6d0 2019-12-05T06:21:31.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7f5762700 1 --2- 172.21.15.16:0/1292028457 >> v2:172.21.15.16:3300/0 conn(0x7fd7f008c6d0 0x7fd7f008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd7e800fac0 tx=0x7fd7e800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7f5762700 1 --2- 172.21.15.16:0/1292028457 >> v2:172.21.15.16:3300/0 conn(0x7fd7f008c6d0 0x7fd7f008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd7e800fac0 tx=0x7fd7e800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7f5762700 1 --2- 172.21.15.16:0/1292028457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f008c6d0 0x7fd7f008cac0 secure :-1 s=READY pgs=614 cs=0 l=1 rx=0x7fd7e800fac0 tx=0x7fd7e800f8f0).ready entity=mon.0 client_cookie=682dbc2784824d06 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7f5762700 1 --2- 172.21.15.16:0/1292028457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f008c6d0 0x7fd7f008cac0 secure :-1 s=THROTTLE_DONE pgs=614 cs=0 l=1 rx=0x7fd7e800fac0 tx=0x7fd7e800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7effff700 1 -- 172.21.15.16:0/1292028457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e8000ce0 con 0x7fd7f008c6d0 2019-12-05T06:21:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7f5762700 1 --2- 172.21.15.16:0/1292028457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f008c6d0 0x7fd7f008cac0 secure :-1 s=THROTTLE_DONE pgs=614 cs=0 l=1 rx=0x7fd7e800fac0 tx=0x7fd7e800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7f5762700 1 --2- 172.21.15.16:0/1292028457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f008c6d0 0x7fd7f008cac0 secure :-1 s=THROTTLE_DONE pgs=614 cs=0 l=1 rx=0x7fd7e800fac0 tx=0x7fd7e800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7effff700 1 -- 172.21.15.16:0/1292028457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e8001010 con 0x7fd7f008c6d0 2019-12-05T06:21:31.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.795+0000 7fd7effff700 1 -- 172.21.15.16:0/1292028457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e8001300 con 0x7fd7f008c6d0 2019-12-05T06:21:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.796+0000 7fd7f71c5700 1 -- 172.21.15.16:0/1292028457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f008c6d0 msgr2=0x7fd7f008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.796+0000 7fd7f71c5700 1 --2- 172.21.15.16:0/1292028457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f008c6d0 0x7fd7f008cac0 secure :-1 s=READY pgs=614 cs=0 l=1 rx=0x7fd7e800fac0 tx=0x7fd7e800f8f0).stop 2019-12-05T06:21:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.797+0000 7fd7f71c5700 1 -- 172.21.15.16:0/1292028457 shutdown_connections 2019-12-05T06:21:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.797+0000 7fd7f71c5700 1 --2- 172.21.15.16:0/1292028457 >> v2:172.21.15.76:3300/0 conn(0x7fd7f008fef0 0x7fd7f008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.797+0000 7fd7f71c5700 1 --2- 172.21.15.16:0/1292028457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f008c6d0 0x7fd7f008cac0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.797+0000 7fd7f71c5700 1 -- 172.21.15.16:0/1292028457 shutdown_connections 2019-12-05T06:21:31.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.797+0000 7fd7f71c5700 1 -- 172.21.15.16:0/1292028457 wait complete. 2019-12-05T06:21:31.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.797+0000 7fd7f71c5700 1 -- 172.21.15.16:0/1292028457 >> 172.21.15.16:0/1292028457 conn(0x7fd7f0061940 msgr2=0x7fd7f0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:31.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.802+0000 7fd7f71c5700 1 Processor -- start 2019-12-05T06:21:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.802+0000 7fd7f71c5700 1 -- start start 2019-12-05T06:21:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f71c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7f014d400 0x7fd7f014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f71c5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd7f0132d00 0x7fd7f0131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f71c5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7f0098da0 con 0x7fd7f0132270 2019-12-05T06:21:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f71c5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7f009bb50 con 0x7fd7f014d400 2019-12-05T06:21:31.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f71c5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f009aa60 con 0x7fd7f0132d00 2019-12-05T06:21:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f5762700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7f014d400 0x7fd7f014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f5762700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd7f014d400 0x7fd7f014d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51366/0 (socket says 172.21.15.16:51366) 2019-12-05T06:21:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f5762700 1 -- 172.21.15.16:0/3873352829 learned_addr learned my addr 172.21.15.16:0/3873352829 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f4f61700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd7f0132d00 0x7fd7f0131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f5762700 1 -- 172.21.15.16:0/3873352829 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd7f0132d00 msgr2=0x7fd7f0131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd7f0132d00 0x7fd7f0131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:31.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f5762700 1 -- 172.21.15.16:0/3873352829 >> v1:172.21.15.76:6789/0 conn(0x7fd7f0132270 legacy=0x7fd7f006dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.804+0000 7fd7f5762700 1 -- 172.21.15.16:0/3873352829 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f009b0f0 con 0x7fd7f014d400 2019-12-05T06:21:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.805+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> v2:172.21.15.16:3300/0 conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.805+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> v2:172.21.15.16:3300/0 conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.805+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=READY pgs=615 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:31.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.805+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.805+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.805+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.805+0000 7fd7edffb700 1 -- 172.21.15.16:0/3873352829 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e8008190 con 0x7fd7f014d400 2019-12-05T06:21:31.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.805+0000 7fd7edffb700 1 -- 172.21.15.16:0/3873352829 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7e8005710 con 0x7fd7f014d400 2019-12-05T06:21:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.805+0000 7fd7f71c5700 1 -- 172.21.15.16:0/3873352829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f009b0f0 con 0x7fd7f014d400 2019-12-05T06:21:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.805+0000 7fd7edffb700 1 -- 172.21.15.16:0/3873352829 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7e8005da0 con 0x7fd7f014d400 2019-12-05T06:21:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.806+0000 7fd7f71c5700 1 -- 172.21.15.16:0/3873352829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f009b0f0 con 0x7fd7f014d400 2019-12-05T06:21:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.806+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.807+0000 7fd7edffb700 1 -- 172.21.15.16:0/3873352829 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd7e8005900 con 0x7fd7f014d400 2019-12-05T06:21:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.807+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.807+0000 7fd7edffb700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd7d0023630 0x7fd7d0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.807+0000 7fd7edffb700 1 -- 172.21.15.16:0/3873352829 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7fd7e8046420 con 0x7fd7f014d400 2019-12-05T06:21:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.807+0000 7fd7f4f61700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd7d0023630 0x7fd7d0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:31.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.808+0000 7fd7f4f61700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd7d0023630 0x7fd7d0025ac0 crc :-1 s=READY pgs=274 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:31.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.823+0000 7fd7f71c5700 1 -- 172.21.15.16:0/3873352829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f009b0f0 con 0x7fd7f014d400 2019-12-05T06:21:31.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.830+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:31.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:31.830+0000 7fd7edffb700 1 -- 172.21.15.16:0/3873352829 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd7e8005500 con 0x7fd7f014d400 2019-12-05T06:21:32.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.068+0000 7fd7f71c5700 1 -- 172.21.15.16:0/3873352829 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7f009b0f0 con 0x7fd7f014d400 2019-12-05T06:21:32.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.068+0000 7fd7f5762700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=THROTTLE_DONE pgs=615 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:32.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.068+0000 7fd7edffb700 1 -- 172.21.15.16:0/3873352829 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fd7e8005710 con 0x7fd7f014d400 2019-12-05T06:21:32.057 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 77 2019-12-05T06:21:32.057 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:32.057 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:32.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.071+0000 7fd7f71c5700 1 -- 172.21.15.16:0/3873352829 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd7d0023630 msgr2=0x7fd7d0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:32.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.071+0000 7fd7f71c5700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd7d0023630 0x7fd7d0025ac0 crc :-1 s=READY pgs=274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:32.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.071+0000 7fd7f71c5700 1 -- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 msgr2=0x7fd7f014d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:32.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.071+0000 7fd7f71c5700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 0x7fd7f014d7f0 secure :-1 s=READY pgs=615 cs=0 l=1 rx=0x7fd7e80107e0 tx=0x7fd7e80105e0).stop 2019-12-05T06:21:32.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.071+0000 7fd7f71c5700 1 -- 172.21.15.16:0/3873352829 shutdown_connections 2019-12-05T06:21:32.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.071+0000 7fd7f71c5700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd7f0132d00 0x7fd7f0131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:32.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.071+0000 7fd7f71c5700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd7d0023630 0x7fd7d0025ac0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:32.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.071+0000 7fd7f71c5700 1 --2- 172.21.15.16:0/3873352829 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd7f014d400 0x7fd7f014d7f0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:32.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.072+0000 7fd7f71c5700 1 -- 172.21.15.16:0/3873352829 shutdown_connections 2019-12-05T06:21:32.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.072+0000 7fd7f71c5700 1 -- 172.21.15.16:0/3873352829 wait complete. 2019-12-05T06:21:32.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:32.072+0000 7fd7f71c5700 1 -- 172.21.15.16:0/3873352829 >> 172.21.15.16:0/3873352829 conn(0x7fd7f00714f0 msgr2=0x7fd7f009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:32.069 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:21:32.395 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:21:32.400 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:21:34.070 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.319+0000 7f1d93319700 1 Processor -- start 2019-12-05T06:21:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.319+0000 7f1d93319700 1 -- start start 2019-12-05T06:21:34.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.320+0000 7f1d93319700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1d8c08c6d0 0x7f1d8c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:34.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.320+0000 7f1d93319700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 0x7f1d8c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:34.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.320+0000 7f1d93319700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8c0958e0 con 0x7f1d8c08ef20 2019-12-05T06:21:34.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.320+0000 7f1d93319700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d8c08df90 con 0x7f1d8c08c6d0 2019-12-05T06:21:34.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.320+0000 7f1d93319700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c0a3f70 con 0x7f1d8c08fef0 2019-12-05T06:21:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.321+0000 7f1d920b7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1d8c08ef20 0x7f1d8c08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36550/0 (socket says 172.21.15.16:36550) 2019-12-05T06:21:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.321+0000 7f1d920b7700 1 -- 172.21.15.16:0/2947447424 learned_addr learned my addr 172.21.15.16:0/2947447424 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.321+0000 7f1d918b6700 1 --2- 172.21.15.16:0/2947447424 >> v2:172.21.15.76:3300/0 conn(0x7f1d8c08c6d0 0x7f1d8c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.321+0000 7f1d910b5700 1 --2- 172.21.15.16:0/2947447424 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 0x7f1d8c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.321+0000 7f1d908b4700 1 -- 172.21.15.16:0/2947447424 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 504368267 0 0) 0x7f1d84001150 con 0x7f1d8c08ef20 2019-12-05T06:21:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.321+0000 7f1d908b4700 1 -- 172.21.15.16:0/2947447424 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d78002e80 con 0x7f1d8c08ef20 2019-12-05T06:21:34.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d910b5700 1 -- 172.21.15.16:0/2947447424 >> v2:172.21.15.76:3300/0 conn(0x7f1d8c08c6d0 msgr2=0x7f1d8c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d910b5700 1 --2- 172.21.15.16:0/2947447424 >> v2:172.21.15.76:3300/0 conn(0x7f1d8c08c6d0 0x7f1d8c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d910b5700 1 -- 172.21.15.16:0/2947447424 >> v1:172.21.15.16:6789/0 conn(0x7f1d8c08ef20 legacy=0x7f1d8c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d910b5700 1 -- 172.21.15.16:0/2947447424 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c08d540 con 0x7f1d8c08fef0 2019-12-05T06:21:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d918b6700 1 --2- 172.21.15.16:0/2947447424 >> v2:172.21.15.76:3300/0 conn(0x7f1d8c08c6d0 0x7f1d8c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:21:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d910b5700 1 --2- 172.21.15.16:0/2947447424 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 0x7f1d8c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1d7c00af50 tx=0x7f1d7c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d910b5700 1 --2- 172.21.15.16:0/2947447424 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 0x7f1d8c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1d7c00af50 tx=0x7f1d7c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d910b5700 1 --2- 172.21.15.16:0/2947447424 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 0x7f1d8c08d000 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7f1d7c00af50 tx=0x7f1d7c00b760).ready entity=mon.2 client_cookie=c1eea42d2d01d36c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d910b5700 1 --2- 172.21.15.16:0/2947447424 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 0x7f1d8c08d000 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f1d7c00af50 tx=0x7f1d7c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d910b5700 1 --2- 172.21.15.16:0/2947447424 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 0x7f1d8c08d000 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f1d7c00af50 tx=0x7f1d7c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d908b4700 1 -- 172.21.15.16:0/2947447424 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d7c007210 con 0x7f1d8c08fef0 2019-12-05T06:21:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d910b5700 1 --2- 172.21.15.16:0/2947447424 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 0x7f1d8c08d000 secure :-1 s=THROTTLE_DONE pgs=152 cs=0 l=1 rx=0x7f1d7c00af50 tx=0x7f1d7c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d908b4700 1 -- 172.21.15.16:0/2947447424 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d7c0099b0 con 0x7f1d8c08fef0 2019-12-05T06:21:34.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.322+0000 7f1d908b4700 1 -- 172.21.15.16:0/2947447424 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d7c00a080 con 0x7f1d8c08fef0 2019-12-05T06:21:34.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.323+0000 7f1d93319700 1 -- 172.21.15.16:0/2947447424 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 msgr2=0x7f1d8c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.323+0000 7f1d93319700 1 --2- 172.21.15.16:0/2947447424 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 0x7f1d8c08d000 secure :-1 s=READY pgs=152 cs=0 l=1 rx=0x7f1d7c00af50 tx=0x7f1d7c00b760).stop 2019-12-05T06:21:34.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.324+0000 7f1d93319700 1 -- 172.21.15.16:0/2947447424 shutdown_connections 2019-12-05T06:21:34.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.324+0000 7f1d93319700 1 --2- 172.21.15.16:0/2947447424 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c08fef0 0x7f1d8c08d000 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.324+0000 7f1d93319700 1 --2- 172.21.15.16:0/2947447424 >> v2:172.21.15.76:3300/0 conn(0x7f1d8c08c6d0 0x7f1d8c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:34.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.325+0000 7f1d93319700 1 -- 172.21.15.16:0/2947447424 shutdown_connections 2019-12-05T06:21:34.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.325+0000 7f1d93319700 1 -- 172.21.15.16:0/2947447424 wait complete. 2019-12-05T06:21:34.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.325+0000 7f1d93319700 1 -- 172.21.15.16:0/2947447424 >> 172.21.15.16:0/2947447424 conn(0x7f1d8c061940 msgr2=0x7f1d8c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:34.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.328+0000 7f1d93319700 1 Processor -- start 2019-12-05T06:21:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.328+0000 7f1d93319700 1 -- start start 2019-12-05T06:21:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.329+0000 7f1d93319700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.329+0000 7f1d93319700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1d8c12b640 0x7f1d8c12abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.329+0000 7f1d93319700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d8c131bf0 con 0x7f1d8c12b0f0 2019-12-05T06:21:34.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.329+0000 7f1d93319700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d8c128cb0 con 0x7f1d8c12b640 2019-12-05T06:21:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.329+0000 7f1d93319700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d8c128f80 con 0x7f1d8c1302f0 2019-12-05T06:21:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.330+0000 7f1d920b7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1d8c12b0f0 0x7f1d8c1306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36556/0 (socket says 172.21.15.16:36556) 2019-12-05T06:21:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.330+0000 7f1d920b7700 1 -- 172.21.15.16:0/3806895665 learned_addr learned my addr 172.21.15.16:0/3806895665 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:34.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.330+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.330+0000 7f1d910b5700 1 --2- 172.21.15.16:0/3806895665 >> v2:172.21.15.76:3300/0 conn(0x7f1d8c12b640 0x7f1d8c12abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.330+0000 7f1d827fc700 1 -- 172.21.15.16:0/3806895665 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318467987 0 0) 0x7f1d84000a40 con 0x7f1d8c12b0f0 2019-12-05T06:21:34.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.330+0000 7f1d827fc700 1 -- 172.21.15.16:0/3806895665 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d74002e80 con 0x7f1d8c12b0f0 2019-12-05T06:21:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.330+0000 7f1d827fc700 1 -- 172.21.15.16:0/3806895665 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1846804439 0 0) 0x7f1d84001300 con 0x7f1d8c12b0f0 2019-12-05T06:21:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d918b6700 1 -- 172.21.15.16:0/3806895665 >> v2:172.21.15.76:3300/0 conn(0x7f1d8c12b640 msgr2=0x7f1d8c12abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> v2:172.21.15.76:3300/0 conn(0x7f1d8c12b640 0x7f1d8c12abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d918b6700 1 -- 172.21.15.16:0/3806895665 >> v1:172.21.15.16:6789/0 conn(0x7f1d8c12b0f0 legacy=0x7f1d8c1306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d918b6700 1 -- 172.21.15.16:0/3806895665 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d8c129d70 con 0x7f1d8c1302f0 2019-12-05T06:21:34.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d910b5700 1 --2- 172.21.15.16:0/3806895665 >> v2:172.21.15.76:3300/0 conn(0x7f1d8c12b640 0x7f1d8c12abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:21:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=READY pgs=153 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:34.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d827fc700 1 -- 172.21.15.16:0/3806895665 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d88000cc0 con 0x7f1d8c1302f0 2019-12-05T06:21:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d827fc700 1 -- 172.21.15.16:0/3806895665 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d8800e370 con 0x7f1d8c1302f0 2019-12-05T06:21:34.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d93319700 1 -- 172.21.15.16:0/3806895665 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d8c129d70 con 0x7f1d8c1302f0 2019-12-05T06:21:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.331+0000 7f1d827fc700 1 -- 172.21.15.16:0/3806895665 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d8800e560 con 0x7f1d8c1302f0 2019-12-05T06:21:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.332+0000 7f1d93319700 1 -- 172.21.15.16:0/3806895665 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d8c129d70 con 0x7f1d8c1302f0 2019-12-05T06:21:34.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.332+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.333+0000 7f1d827fc700 1 -- 172.21.15.16:0/3806895665 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1d8800dde0 con 0x7f1d8c1302f0 2019-12-05T06:21:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.333+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.334+0000 7f1d827fc700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023680 0x7f1d74025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:34.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.334+0000 7f1d827fc700 1 -- 172.21.15.16:0/3806895665 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f1d88046100 con 0x7f1d8c1302f0 2019-12-05T06:21:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.334+0000 7f1d910b5700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023680 0x7f1d74025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.334+0000 7f1d910b5700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023680 0x7f1d74025af0 crc :-1 s=READY pgs=275 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:34.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.335+0000 7f1d93319700 1 -- 172.21.15.16:0/3806895665 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d8c129d70 con 0x7f1d8c1302f0 2019-12-05T06:21:34.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.345+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.345+0000 7f1d827fc700 1 -- 172.21.15.16:0/3806895665 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1d8800ea70 con 0x7f1d8c1302f0 2019-12-05T06:21:34.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.583+0000 7f1d93319700 1 -- 172.21.15.16:0/3806895665 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d8c129d70 con 0x7f1d8c1302f0 2019-12-05T06:21:34.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.584+0000 7f1d918b6700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=THROTTLE_DONE pgs=153 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:34.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.584+0000 7f1d827fc700 1 -- 172.21.15.16:0/3806895665 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f1d8800dd60 con 0x7f1d8c1302f0 2019-12-05T06:21:34.574 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:34.574 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:34.575 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 77 2019-12-05T06:21:34.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 -- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023680 msgr2=0x7f1d74025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023680 0x7f1d74025af0 crc :-1 s=READY pgs=275 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 -- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 msgr2=0x7f1d8c12a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:34.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 secure :-1 s=READY pgs=153 cs=0 l=1 rx=0x7f1d88010720 tx=0x7f1d88010ac0).stop 2019-12-05T06:21:34.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 -- 172.21.15.16:0/3806895665 shutdown_connections 2019-12-05T06:21:34.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023680 0x7f1d74025af0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 --2- 172.21.15.16:0/3806895665 >> v2:172.21.15.76:3300/0 conn(0x7f1d8c12b640 0x7f1d8c12abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 --2- 172.21.15.16:0/3806895665 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d8c1302f0 0x7f1d8c12a670 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:34.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 -- 172.21.15.16:0/3806895665 shutdown_connections 2019-12-05T06:21:34.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 -- 172.21.15.16:0/3806895665 wait complete. 2019-12-05T06:21:34.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:34.589+0000 7f1d93319700 1 -- 172.21.15.16:0/3806895665 >> 172.21.15.16:0/3806895665 conn(0x7f1d8c0a2850 msgr2=0x7f1d8c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:34.594 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:21:36.600 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:36.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.882+0000 7f63dc4ba700 1 Processor -- start 2019-12-05T06:21:36.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.882+0000 7f63dc4ba700 1 -- start start 2019-12-05T06:21:36.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.884+0000 7f63dc4ba700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f63d408c6d0 0x7f63d408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:36.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.884+0000 7f63dc4ba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d408fb80 con 0x7f63d408f720 2019-12-05T06:21:36.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.884+0000 7f63dc4ba700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d40958e0 con 0x7f63d408fef0 2019-12-05T06:21:36.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.884+0000 7f63dc4ba700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d408df90 con 0x7f63d408c6d0 2019-12-05T06:21:36.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.885+0000 7f63db258700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f63d408f720 0x7f63d41533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36566/0 (socket says 172.21.15.16:36566) 2019-12-05T06:21:36.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.885+0000 7f63db258700 1 -- 172.21.15.16:0/540855012 learned_addr learned my addr 172.21.15.16:0/540855012 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:36.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.885+0000 7f63daa57700 1 --2- 172.21.15.16:0/540855012 >> v2:172.21.15.76:3300/0 conn(0x7f63d408c6d0 0x7f63d408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:36.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.885+0000 7f63d9a55700 1 -- 172.21.15.16:0/540855012 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846693944 0 0) 0x7f63c4001150 con 0x7f63d408f720 2019-12-05T06:21:36.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.885+0000 7f63d9a55700 1 -- 172.21.15.16:0/540855012 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63bc002e80 con 0x7f63d408f720 2019-12-05T06:21:36.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.885+0000 7f63d9a55700 1 -- 172.21.15.16:0/540855012 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648373937 0 0) 0x7f63c8001080 con 0x7f63d408fef0 2019-12-05T06:21:36.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.885+0000 7f63d9a55700 1 -- 172.21.15.16:0/540855012 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63bc004170 con 0x7f63d408fef0 2019-12-05T06:21:36.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.886+0000 7f63daa57700 1 -- 172.21.15.16:0/540855012 >> v1:172.21.15.76:6789/0 conn(0x7f63d408fef0 legacy=0x7f63d408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:36.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.886+0000 7f63daa57700 1 -- 172.21.15.16:0/540855012 >> v1:172.21.15.16:6789/0 conn(0x7f63d408f720 legacy=0x7f63d41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:36.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.886+0000 7f63daa57700 1 -- 172.21.15.16:0/540855012 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d408e620 con 0x7f63d408c6d0 2019-12-05T06:21:36.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.886+0000 7f63daa57700 1 --2- 172.21.15.16:0/540855012 >> v2:172.21.15.76:3300/0 conn(0x7f63d408c6d0 0x7f63d408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f63cc00b2d0 tx=0x7f63cc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.886+0000 7f63daa57700 1 --2- 172.21.15.16:0/540855012 >> v2:172.21.15.76:3300/0 conn(0x7f63d408c6d0 0x7f63d408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f63cc00b2d0 tx=0x7f63cc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.886+0000 7f63daa57700 1 --2- 172.21.15.16:0/540855012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d408c6d0 0x7f63d408cac0 secure :-1 s=READY pgs=447 cs=0 l=1 rx=0x7f63cc00b2d0 tx=0x7f63cc00bae0).ready entity=mon.1 client_cookie=23cc1325c6ca615b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:36.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.887+0000 7f63daa57700 1 --2- 172.21.15.16:0/540855012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d408c6d0 0x7f63d408cac0 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f63cc00b2d0 tx=0x7f63cc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.887+0000 7f63daa57700 1 --2- 172.21.15.16:0/540855012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d408c6d0 0x7f63d408cac0 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f63cc00b2d0 tx=0x7f63cc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.887+0000 7f63d9a55700 1 -- 172.21.15.16:0/540855012 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63cc000ce0 con 0x7f63d408c6d0 2019-12-05T06:21:36.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.887+0000 7f63daa57700 1 --2- 172.21.15.16:0/540855012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d408c6d0 0x7f63d408cac0 secure :-1 s=THROTTLE_DONE pgs=447 cs=0 l=1 rx=0x7f63cc00b2d0 tx=0x7f63cc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.887+0000 7f63d9a55700 1 -- 172.21.15.16:0/540855012 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc009be0 con 0x7f63d408c6d0 2019-12-05T06:21:36.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.887+0000 7f63d9a55700 1 -- 172.21.15.16:0/540855012 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63cc00a2b0 con 0x7f63d408c6d0 2019-12-05T06:21:36.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.887+0000 7f63dc4ba700 1 -- 172.21.15.16:0/540855012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d408c6d0 msgr2=0x7f63d408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:36.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.887+0000 7f63dc4ba700 1 --2- 172.21.15.16:0/540855012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d408c6d0 0x7f63d408cac0 secure :-1 s=READY pgs=447 cs=0 l=1 rx=0x7f63cc00b2d0 tx=0x7f63cc00bae0).stop 2019-12-05T06:21:36.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.889+0000 7f63dc4ba700 1 -- 172.21.15.16:0/540855012 shutdown_connections 2019-12-05T06:21:36.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.889+0000 7f63dc4ba700 1 --2- 172.21.15.16:0/540855012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d408c6d0 0x7f63d408cac0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:36.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.889+0000 7f63dc4ba700 1 -- 172.21.15.16:0/540855012 shutdown_connections 2019-12-05T06:21:36.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.889+0000 7f63dc4ba700 1 -- 172.21.15.16:0/540855012 wait complete. 2019-12-05T06:21:36.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.889+0000 7f63dc4ba700 1 -- 172.21.15.16:0/540855012 >> 172.21.15.16:0/540855012 conn(0x7f63d4061940 msgr2=0x7f63d4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:36.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.894+0000 7f63dc4ba700 1 Processor -- start 2019-12-05T06:21:36.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.894+0000 7f63dc4ba700 1 -- start start 2019-12-05T06:21:36.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.896+0000 7f63dc4ba700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f63d4128360 0x7f63d4128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.896+0000 7f63dc4ba700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d4099720 con 0x7f63d4137ea0 2019-12-05T06:21:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.896+0000 7f63dc4ba700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d40986d0 con 0x7f63d4154770 2019-12-05T06:21:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.896+0000 7f63dc4ba700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63d412a640 con 0x7f63d4128360 2019-12-05T06:21:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.897+0000 7f63da256700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f63d4137ea0 0x7f63d412b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36572/0 (socket says 172.21.15.16:36572) 2019-12-05T06:21:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.897+0000 7f63da256700 1 -- 172.21.15.16:0/577627424 learned_addr learned my addr 172.21.15.16:0/577627424 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.897+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> v2:172.21.15.76:3300/0 conn(0x7f63d4128360 0x7f63d4128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:36.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.897+0000 7f63d37fe700 1 -- 172.21.15.16:0/577627424 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 724728524 0 0) 0x7f63c8001240 con 0x7f63d4137ea0 2019-12-05T06:21:36.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.897+0000 7f63d37fe700 1 -- 172.21.15.16:0/577627424 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b4002e80 con 0x7f63d4137ea0 2019-12-05T06:21:36.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.897+0000 7f63daa57700 1 -- 172.21.15.16:0/577627424 >> v1:172.21.15.76:6789/0 conn(0x7f63d4154770 legacy=0x7f63d419ac00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:36.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.897+0000 7f63daa57700 1 -- 172.21.15.16:0/577627424 >> v1:172.21.15.16:6789/0 conn(0x7f63d4137ea0 legacy=0x7f63d412b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:36.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.897+0000 7f63d37fe700 1 -- 172.21.15.16:0/577627424 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1308072962 0 0) 0x7f63c40009b0 con 0x7f63d4154770 2019-12-05T06:21:36.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.897+0000 7f63daa57700 1 -- 172.21.15.16:0/577627424 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d412acd0 con 0x7f63d4128360 2019-12-05T06:21:36.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> v2:172.21.15.76:3300/0 conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> v2:172.21.15.76:3300/0 conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=READY pgs=449 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:36.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63d37fe700 1 -- 172.21.15.16:0/577627424 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63cc000cc0 con 0x7f63d4128360 2019-12-05T06:21:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63d37fe700 1 -- 172.21.15.16:0/577627424 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63cc009af0 con 0x7f63d4128360 2019-12-05T06:21:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63d37fe700 1 -- 172.21.15.16:0/577627424 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63cc009ce0 con 0x7f63d4128360 2019-12-05T06:21:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63dc4ba700 1 -- 172.21.15.16:0/577627424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d412acd0 con 0x7f63d4128360 2019-12-05T06:21:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.898+0000 7f63dc4ba700 1 -- 172.21.15.16:0/577627424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d412acd0 con 0x7f63d4128360 2019-12-05T06:21:36.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.899+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.900+0000 7f63d37fe700 1 -- 172.21.15.16:0/577627424 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f63cc009ce0 con 0x7f63d4128360 2019-12-05T06:21:36.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.900+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.900+0000 7f63d37fe700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f63b4023680 0x7f63b4025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:36.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.901+0000 7f63d37fe700 1 -- 172.21.15.16:0/577627424 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f63cc048390 con 0x7f63d4128360 2019-12-05T06:21:36.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.901+0000 7f63da256700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f63b4023680 0x7f63b4025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:36.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.901+0000 7f63da256700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f63b4023680 0x7f63b4025af0 crc :-1 s=READY pgs=276 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:36.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.914+0000 7f63dc4ba700 1 -- 172.21.15.16:0/577627424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d412acd0 con 0x7f63d4128360 2019-12-05T06:21:36.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.923+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:36.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:36.923+0000 7f63d37fe700 1 -- 172.21.15.16:0/577627424 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f63cc007e90 con 0x7f63d4128360 2019-12-05T06:21:37.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.182+0000 7f63dc4ba700 1 -- 172.21.15.16:0/577627424 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63d412acd0 con 0x7f63d4128360 2019-12-05T06:21:37.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.183+0000 7f63daa57700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=THROTTLE_DONE pgs=449 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:37.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.183+0000 7f63d37fe700 1 -- 172.21.15.16:0/577627424 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f63cc007e90 con 0x7f63d4128360 2019-12-05T06:21:37.173 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 77 2019-12-05T06:21:37.173 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:37.173 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:37.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.188+0000 7f63dc4ba700 1 -- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f63b4023680 msgr2=0x7f63b4025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:37.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.188+0000 7f63dc4ba700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f63b4023680 0x7f63b4025af0 crc :-1 s=READY pgs=276 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.188+0000 7f63dc4ba700 1 -- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 msgr2=0x7f63d4128750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:37.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.188+0000 7f63dc4ba700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 0x7f63d4128750 secure :-1 s=READY pgs=449 cs=0 l=1 rx=0x7f63cc012890 tx=0x7f63cc012c30).stop 2019-12-05T06:21:37.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.188+0000 7f63dc4ba700 1 -- 172.21.15.16:0/577627424 shutdown_connections 2019-12-05T06:21:37.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.188+0000 7f63dc4ba700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f63b4023680 0x7f63b4025af0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.188+0000 7f63dc4ba700 1 --2- 172.21.15.16:0/577627424 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f63d4128360 0x7f63d4128750 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:37.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.188+0000 7f63dc4ba700 1 -- 172.21.15.16:0/577627424 shutdown_connections 2019-12-05T06:21:37.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.188+0000 7f63dc4ba700 1 -- 172.21.15.16:0/577627424 wait complete. 2019-12-05T06:21:37.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:37.188+0000 7f63dc4ba700 1 -- 172.21.15.16:0/577627424 >> 172.21.15.16:0/577627424 conn(0x7f63d406d260 msgr2=0x7f63d409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:37.187 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:21:39.188 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:39.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.478+0000 7faea34a5700 1 Processor -- start 2019-12-05T06:21:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.479+0000 7faea34a5700 1 -- start start 2019-12-05T06:21:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.480+0000 7faea34a5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fae9c08c6d0 0x7fae9c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.480+0000 7faea34a5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c08fef0 0x7fae9c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.480+0000 7faea34a5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae9c0958e0 con 0x7fae9c08ef20 2019-12-05T06:21:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.480+0000 7faea34a5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c08df90 con 0x7fae9c08c6d0 2019-12-05T06:21:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.480+0000 7faea34a5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c0a3f70 con 0x7fae9c08fef0 2019-12-05T06:21:39.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.481+0000 7faea1241700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c08fef0 0x7fae9c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.481+0000 7faea1a42700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fae9c08c6d0 0x7fae9c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.481+0000 7faea2243700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fae9c08ef20 0x7fae9c08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57888/0 (socket says 172.21.15.16:57888) 2019-12-05T06:21:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.481+0000 7faea2243700 1 -- 172.21.15.16:0/2422197807 learned_addr learned my addr 172.21.15.16:0/2422197807 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.482+0000 7faea1a42700 1 -- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c08fef0 msgr2=0x7fae9c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.482+0000 7faea1a42700 1 --2- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c08fef0 0x7fae9c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.482+0000 7faea1a42700 1 -- 172.21.15.16:0/2422197807 >> v1:172.21.15.76:6789/0 conn(0x7fae9c08ef20 legacy=0x7fae9c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.482+0000 7faea0a40700 1 -- 172.21.15.16:0/2422197807 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3436832181 0 0) 0x7fae94001100 con 0x7fae9c08ef20 2019-12-05T06:21:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.482+0000 7faea1a42700 1 -- 172.21.15.16:0/2422197807 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c08d540 con 0x7fae9c08c6d0 2019-12-05T06:21:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.482+0000 7faea1241700 1 --2- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c08fef0 0x7fae9c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:21:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.482+0000 7faea1a42700 1 --2- 172.21.15.16:0/2422197807 >> v2:172.21.15.76:3300/0 conn(0x7fae9c08c6d0 0x7fae9c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fae8c00af50 tx=0x7fae8c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.482+0000 7faea1a42700 1 --2- 172.21.15.16:0/2422197807 >> v2:172.21.15.76:3300/0 conn(0x7fae9c08c6d0 0x7fae9c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fae8c00af50 tx=0x7fae8c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.482+0000 7faea1a42700 1 --2- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae9c08c6d0 0x7fae9c08cac0 secure :-1 s=READY pgs=451 cs=0 l=1 rx=0x7fae8c00af50 tx=0x7fae8c00b760).ready entity=mon.1 client_cookie=779129c15855d953 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.483+0000 7faea1a42700 1 --2- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae9c08c6d0 0x7fae9c08cac0 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7fae8c00af50 tx=0x7fae8c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.483+0000 7faea1a42700 1 --2- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae9c08c6d0 0x7fae9c08cac0 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7fae8c00af50 tx=0x7fae8c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.483+0000 7faea0a40700 1 -- 172.21.15.16:0/2422197807 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae8c007210 con 0x7fae9c08c6d0 2019-12-05T06:21:39.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.483+0000 7faea1a42700 1 --2- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae9c08c6d0 0x7fae9c08cac0 secure :-1 s=THROTTLE_DONE pgs=451 cs=0 l=1 rx=0x7fae8c00af50 tx=0x7fae8c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.483+0000 7faea0a40700 1 -- 172.21.15.16:0/2422197807 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae8c0099b0 con 0x7fae9c08c6d0 2019-12-05T06:21:39.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.483+0000 7faea0a40700 1 -- 172.21.15.16:0/2422197807 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae8c00a080 con 0x7fae9c08c6d0 2019-12-05T06:21:39.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.483+0000 7faea34a5700 1 -- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae9c08c6d0 msgr2=0x7fae9c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.483+0000 7faea34a5700 1 --2- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae9c08c6d0 0x7fae9c08cac0 secure :-1 s=READY pgs=451 cs=0 l=1 rx=0x7fae8c00af50 tx=0x7fae8c00b760).stop 2019-12-05T06:21:39.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.485+0000 7faea34a5700 1 -- 172.21.15.16:0/2422197807 shutdown_connections 2019-12-05T06:21:39.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.485+0000 7faea34a5700 1 --2- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c08fef0 0x7fae9c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:39.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.485+0000 7faea34a5700 1 --2- 172.21.15.16:0/2422197807 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fae9c08c6d0 0x7fae9c08cac0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.485+0000 7faea34a5700 1 -- 172.21.15.16:0/2422197807 shutdown_connections 2019-12-05T06:21:39.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.485+0000 7faea34a5700 1 -- 172.21.15.16:0/2422197807 wait complete. 2019-12-05T06:21:39.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.485+0000 7faea34a5700 1 -- 172.21.15.16:0/2422197807 >> 172.21.15.16:0/2422197807 conn(0x7fae9c061940 msgr2=0x7fae9c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:39.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.491+0000 7faea34a5700 1 Processor -- start 2019-12-05T06:21:39.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.491+0000 7faea34a5700 1 -- start start 2019-12-05T06:21:39.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea34a5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fae9c1325e0 0x7fae9c12b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:39.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea34a5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae9c12c740 0x7fae9c12bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:39.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea34a5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c12c1f0 0x7fae9c06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:39.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea34a5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c12a7e0 con 0x7fae9c12c740 2019-12-05T06:21:39.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea34a5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae9c129db0 con 0x7fae9c1325e0 2019-12-05T06:21:39.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea34a5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae9c12a080 con 0x7fae9c12c1f0 2019-12-05T06:21:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea1241700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae9c12c740 0x7fae9c12bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea1241700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fae9c12c740 0x7fae9c12bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51422/0 (socket says 172.21.15.16:51422) 2019-12-05T06:21:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea1241700 1 -- 172.21.15.16:0/668026427 learned_addr learned my addr 172.21.15.16:0/668026427 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea1a42700 1 --2- 172.21.15.16:0/668026427 >> v2:172.21.15.76:3300/0 conn(0x7fae9c1325e0 0x7fae9c12b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.493+0000 7faea2243700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c12c1f0 0x7fae9c06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7faea1241700 1 -- 172.21.15.16:0/668026427 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c12c1f0 msgr2=0x7fae9c06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c12c1f0 0x7fae9c06dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7faea1241700 1 -- 172.21.15.16:0/668026427 >> v2:172.21.15.76:3300/0 conn(0x7fae9c1325e0 msgr2=0x7fae9c12b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> v2:172.21.15.76:3300/0 conn(0x7fae9c1325e0 0x7fae9c12b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7faea1241700 1 -- 172.21.15.16:0/668026427 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c12ae70 con 0x7fae9c12c740 2019-12-05T06:21:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> v2:172.21.15.16:3300/0 conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> v2:172.21.15.16:3300/0 conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=READY pgs=620 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:39.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7fae9a7fc700 1 -- 172.21.15.16:0/668026427 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae9000ac40 con 0x7fae9c12c740 2019-12-05T06:21:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.494+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.495+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.495+0000 7fae9a7fc700 1 -- 172.21.15.16:0/668026427 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9000dec0 con 0x7fae9c12c740 2019-12-05T06:21:39.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.495+0000 7faea34a5700 1 -- 172.21.15.16:0/668026427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae9c12ae70 con 0x7fae9c12c740 2019-12-05T06:21:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.495+0000 7fae9a7fc700 1 -- 172.21.15.16:0/668026427 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae9000e510 con 0x7fae9c12c740 2019-12-05T06:21:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.495+0000 7faea34a5700 1 -- 172.21.15.16:0/668026427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae9c12ae70 con 0x7fae9c12c740 2019-12-05T06:21:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.496+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.497+0000 7fae9a7fc700 1 -- 172.21.15.16:0/668026427 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fae90009900 con 0x7fae9c12c740 2019-12-05T06:21:39.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.497+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.497+0000 7fae9a7fc700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fae7c023630 0x7fae7c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.497+0000 7fae9a7fc700 1 -- 172.21.15.16:0/668026427 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7fae900460b0 con 0x7fae9c12c740 2019-12-05T06:21:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.497+0000 7faea1a42700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fae7c023630 0x7fae7c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:39.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.498+0000 7faea1a42700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fae7c023630 0x7fae7c025ac0 crc :-1 s=READY pgs=277 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.510+0000 7faea34a5700 1 -- 172.21.15.16:0/668026427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae9c12ae70 con 0x7fae9c12c740 2019-12-05T06:21:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.520+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.520+0000 7fae9a7fc700 1 -- 172.21.15.16:0/668026427 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fae9000a910 con 0x7fae9c12c740 2019-12-05T06:21:39.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.774+0000 7faea34a5700 1 -- 172.21.15.16:0/668026427 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fae9c12ae70 con 0x7fae9c12c740 2019-12-05T06:21:39.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.775+0000 7faea1241700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=620 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:39.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.775+0000 7fae9a7fc700 1 -- 172.21.15.16:0/668026427 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fae90009900 con 0x7fae9c12c740 2019-12-05T06:21:39.763 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 77 2019-12-05T06:21:39.763 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:39.764 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:39.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 -- 172.21.15.16:0/668026427 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fae7c023630 msgr2=0x7fae7c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fae7c023630 0x7fae7c025ac0 crc :-1 s=READY pgs=277 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 -- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 msgr2=0x7fae9c12bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:39.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 0x7fae9c12bcb0 secure :-1 s=READY pgs=620 cs=0 l=1 rx=0x7fae90010790 tx=0x7fae90010690).stop 2019-12-05T06:21:39.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 -- 172.21.15.16:0/668026427 shutdown_connections 2019-12-05T06:21:39.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fae9c12c1f0 0x7fae9c06dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fae7c023630 0x7fae7c025ac0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 --2- 172.21.15.16:0/668026427 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fae9c12c740 0x7fae9c12bcb0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 --2- 172.21.15.16:0/668026427 >> v2:172.21.15.76:3300/0 conn(0x7fae9c1325e0 0x7fae9c12b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:39.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 -- 172.21.15.16:0/668026427 shutdown_connections 2019-12-05T06:21:39.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 -- 172.21.15.16:0/668026427 wait complete. 2019-12-05T06:21:39.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:39.778+0000 7faea34a5700 1 -- 172.21.15.16:0/668026427 >> 172.21.15.16:0/668026427 conn(0x7fae9c0a2850 msgr2=0x7fae9c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:39.777 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:21:41.777 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.056+0000 7f6a2b8d6700 1 Processor -- start 2019-12-05T06:21:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.057+0000 7f6a2b8d6700 1 -- start start 2019-12-05T06:21:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.058+0000 7f6a2b8d6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a2408d2c0 0x7f6a2408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.058+0000 7f6a2b8d6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6a2408fef0 0x7f6a2408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.059+0000 7f6a2b8d6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a240958e0 con 0x7f6a2408c6d0 2019-12-05T06:21:42.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.059+0000 7f6a2b8d6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a2408df90 con 0x7f6a2408d2c0 2019-12-05T06:21:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.059+0000 7f6a2b8d6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a240a3f70 con 0x7f6a2408fef0 2019-12-05T06:21:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.059+0000 7f6a29672700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a2408d2c0 0x7f6a2408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.059+0000 7f6a29672700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a2408d2c0 0x7f6a2408ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51434/0 (socket says 172.21.15.16:51434) 2019-12-05T06:21:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.059+0000 7f6a29672700 1 -- 172.21.15.16:0/13170782 learned_addr learned my addr 172.21.15.16:0/13170782 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.059+0000 7f6a2a674700 1 --2- 172.21.15.16:0/13170782 >> v2:172.21.15.76:3300/0 conn(0x7f6a2408fef0 0x7f6a2408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a29672700 1 -- 172.21.15.16:0/13170782 >> v2:172.21.15.76:3300/0 conn(0x7f6a2408fef0 msgr2=0x7f6a2408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:42.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a29672700 1 --2- 172.21.15.16:0/13170782 >> v2:172.21.15.76:3300/0 conn(0x7f6a2408fef0 0x7f6a2408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a29672700 1 -- 172.21.15.16:0/13170782 >> v1:172.21.15.76:6789/0 conn(0x7f6a2408c6d0 legacy=0x7f6a2408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a29672700 1 -- 172.21.15.16:0/13170782 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2408e620 con 0x7f6a2408d2c0 2019-12-05T06:21:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a28e71700 1 -- 172.21.15.16:0/13170782 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91428046 0 0) 0x7f6a14001080 con 0x7f6a2408c6d0 2019-12-05T06:21:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a2a674700 1 --2- 172.21.15.16:0/13170782 >> v2:172.21.15.76:3300/0 conn(0x7f6a2408fef0 0x7f6a2408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:21:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a29672700 1 --2- 172.21.15.16:0/13170782 >> v2:172.21.15.16:3300/0 conn(0x7f6a2408d2c0 0x7f6a2408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6a1c00f740 tx=0x7f6a1c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a29672700 1 --2- 172.21.15.16:0/13170782 >> v2:172.21.15.16:3300/0 conn(0x7f6a2408d2c0 0x7f6a2408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6a1c00f740 tx=0x7f6a1c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a29672700 1 --2- 172.21.15.16:0/13170782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a2408d2c0 0x7f6a2408ef20 secure :-1 s=READY pgs=621 cs=0 l=1 rx=0x7f6a1c00f740 tx=0x7f6a1c00fae0).ready entity=mon.0 client_cookie=f59733dedd43e983 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a29672700 1 --2- 172.21.15.16:0/13170782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a2408d2c0 0x7f6a2408ef20 secure :-1 s=THROTTLE_DONE pgs=621 cs=0 l=1 rx=0x7f6a1c00f740 tx=0x7f6a1c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a28e71700 1 -- 172.21.15.16:0/13170782 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a1c000960 con 0x7f6a2408d2c0 2019-12-05T06:21:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a29672700 1 --2- 172.21.15.16:0/13170782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a2408d2c0 0x7f6a2408ef20 secure :-1 s=THROTTLE_DONE pgs=621 cs=0 l=1 rx=0x7f6a1c00f740 tx=0x7f6a1c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.060+0000 7f6a29672700 1 --2- 172.21.15.16:0/13170782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a2408d2c0 0x7f6a2408ef20 secure :-1 s=THROTTLE_DONE pgs=621 cs=0 l=1 rx=0x7f6a1c00f740 tx=0x7f6a1c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.061+0000 7f6a28e71700 1 -- 172.21.15.16:0/13170782 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a1c000c90 con 0x7f6a2408d2c0 2019-12-05T06:21:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.061+0000 7f6a28e71700 1 -- 172.21.15.16:0/13170782 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a1c000960 con 0x7f6a2408d2c0 2019-12-05T06:21:42.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.061+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/13170782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a2408d2c0 msgr2=0x7f6a2408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.061+0000 7f6a2b8d6700 1 --2- 172.21.15.16:0/13170782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a2408d2c0 0x7f6a2408ef20 secure :-1 s=READY pgs=621 cs=0 l=1 rx=0x7f6a1c00f740 tx=0x7f6a1c00fae0).stop 2019-12-05T06:21:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.062+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/13170782 shutdown_connections 2019-12-05T06:21:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.062+0000 7f6a2b8d6700 1 --2- 172.21.15.16:0/13170782 >> v2:172.21.15.76:3300/0 conn(0x7f6a2408fef0 0x7f6a2408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.062+0000 7f6a2b8d6700 1 --2- 172.21.15.16:0/13170782 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a2408d2c0 0x7f6a2408ef20 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.063+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/13170782 shutdown_connections 2019-12-05T06:21:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.063+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/13170782 wait complete. 2019-12-05T06:21:42.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.063+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/13170782 >> 172.21.15.16:0/13170782 conn(0x7f6a24061940 msgr2=0x7f6a24096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:42.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.068+0000 7f6a2b8d6700 1 Processor -- start 2019-12-05T06:21:42.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.068+0000 7f6a2b8d6700 1 -- start start 2019-12-05T06:21:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a2b8d6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a24096e90 0x7f6a2409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a2b8d6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a24084f30 0x7f6a2409b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a2b8d6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a24084b90 con 0x7f6a24097280 2019-12-05T06:21:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a2b8d6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a2409a450 con 0x7f6a24096e90 2019-12-05T06:21:42.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a2b8d6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a24099a20 con 0x7f6a24084f30 2019-12-05T06:21:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a29e73700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a24096e90 0x7f6a2409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a2a674700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6a24097280 0x7f6a240976c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36610/0 (socket says 172.21.15.16:36610) 2019-12-05T06:21:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a2a674700 1 -- 172.21.15.16:0/1745959077 learned_addr learned my addr 172.21.15.16:0/1745959077 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a29672700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a24084f30 0x7f6a2409b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a29e73700 1 -- 172.21.15.16:0/1745959077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a24084f30 msgr2=0x7f6a2409b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:42.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a24084f30 0x7f6a2409b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.070+0000 7f6a29e73700 1 -- 172.21.15.16:0/1745959077 >> v1:172.21.15.16:6789/0 conn(0x7f6a24097280 legacy=0x7f6a240976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a29e73700 1 -- 172.21.15.16:0/1745959077 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a2409aae0 con 0x7f6a24096e90 2019-12-05T06:21:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a227fc700 1 -- 172.21.15.16:0/1745959077 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445435486 0 0) 0x7f6a0c009c10 con 0x7f6a24097280 2019-12-05T06:21:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> v2:172.21.15.16:3300/0 conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> v2:172.21.15.16:3300/0 conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=READY pgs=623 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=THROTTLE_DONE pgs=623 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a227fc700 1 -- 172.21.15.16:0/1745959077 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a14000ea0 con 0x7f6a24096e90 2019-12-05T06:21:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/1745959077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a2409aae0 con 0x7f6a24096e90 2019-12-05T06:21:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=THROTTLE_DONE pgs=623 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=THROTTLE_DONE pgs=623 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a227fc700 1 -- 172.21.15.16:0/1745959077 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a14000a50 con 0x7f6a24096e90 2019-12-05T06:21:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.071+0000 7f6a227fc700 1 -- 172.21.15.16:0/1745959077 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a14005ad0 con 0x7f6a24096e90 2019-12-05T06:21:42.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.072+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/1745959077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a2409aae0 con 0x7f6a24096e90 2019-12-05T06:21:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.073+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=THROTTLE_DONE pgs=623 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.074+0000 7f6a227fc700 1 -- 172.21.15.16:0/1745959077 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6a140056f0 con 0x7f6a24096e90 2019-12-05T06:21:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.074+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=THROTTLE_DONE pgs=623 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.074+0000 7f6a227fc700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a08023630 0x7f6a08025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.075+0000 7f6a227fc700 1 -- 172.21.15.16:0/1745959077 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 6805+0+0 (secure 0 0 0) 0x7f6a14045d10 con 0x7f6a24096e90 2019-12-05T06:21:42.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.075+0000 7f6a29672700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a08023630 0x7f6a08025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:42.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.075+0000 7f6a29672700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a08023630 0x7f6a08025ac0 crc :-1 s=READY pgs=279 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:42.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.088+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/1745959077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2409aae0 con 0x7f6a24096e90 2019-12-05T06:21:42.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.095+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=THROTTLE_DONE pgs=623 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.095+0000 7f6a227fc700 1 -- 172.21.15.16:0/1745959077 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6a14013460 con 0x7f6a24096e90 2019-12-05T06:21:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.344+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/1745959077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a2409aae0 con 0x7f6a24096e90 2019-12-05T06:21:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.345+0000 7f6a29e73700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=THROTTLE_DONE pgs=623 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.345+0000 7f6a227fc700 1 -- 172.21.15.16:0/1745959077 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f6a14013460 con 0x7f6a24096e90 2019-12-05T06:21:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 77 2019-12-05T06:21:42.335 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:42.336 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":77,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5776,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":77}],"filesystems":[{"mdsmap":{"epoch":76,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:20:38.143649+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":58,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5604},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5604":{"gid":5604,"name":"b","rank":0,"incarnation":73,"state":"up:active","state_seq":16,"addr":"172.21.15.76:6835/4248142088","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":4248142088},{"type":"v1","addr":"172.21.15.76:6835","nonce":4248142088}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.350+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/1745959077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a08023630 msgr2=0x7f6a08025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.350+0000 7f6a2b8d6700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a08023630 0x7f6a08025ac0 crc :-1 s=READY pgs=279 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.350+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 msgr2=0x7f6a2409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.350+0000 7f6a2b8d6700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 0x7f6a2409b3e0 secure :-1 s=READY pgs=623 cs=0 l=1 rx=0x7f6a14010380 tx=0x7f6a140101b0).stop 2019-12-05T06:21:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.350+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/1745959077 shutdown_connections 2019-12-05T06:21:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.350+0000 7f6a2b8d6700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a24084f30 0x7f6a2409b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.350+0000 7f6a2b8d6700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a08023630 0x7f6a08025ac0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.350+0000 7f6a2b8d6700 1 --2- 172.21.15.16:0/1745959077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a24096e90 0x7f6a2409b3e0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:42.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.350+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/1745959077 shutdown_connections 2019-12-05T06:21:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.351+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/1745959077 wait complete. 2019-12-05T06:21:42.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:42.351+0000 7f6a2b8d6700 1 -- 172.21.15.16:0/1745959077 >> 172.21.15.16:0/1745959077 conn(0x7f6a240714f0 msgr2=0x7f6a2409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:42.349 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:21:44.350 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:44.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.541+0000 7f888eaf5700 1 Processor -- start 2019-12-05T06:21:44.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.541+0000 7f888eaf5700 1 -- start start 2019-12-05T06:21:44.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.542+0000 7f888eaf5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f888808d2c0 0x7f888808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:44.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.542+0000 7f888eaf5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f888808fef0 0x7f888808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:44.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.542+0000 7f888eaf5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88880958e0 con 0x7f888808c6d0 2019-12-05T06:21:44.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.542+0000 7f888eaf5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888808df90 con 0x7f888808d2c0 2019-12-05T06:21:44.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.542+0000 7f888eaf5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88880a3f70 con 0x7f888808fef0 2019-12-05T06:21:44.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.542+0000 7f888c891700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f888808d2c0 0x7f888808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:44.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.542+0000 7f888c891700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f888808d2c0 0x7f888808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51462/0 (socket says 172.21.15.16:51462) 2019-12-05T06:21:44.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.542+0000 7f888c891700 1 -- 172.21.15.16:0/1299007653 learned_addr learned my addr 172.21.15.16:0/1299007653 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:44.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.542+0000 7f888d893700 1 --2- 172.21.15.16:0/1299007653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f888808fef0 0x7f888808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:44.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888c891700 1 -- 172.21.15.16:0/1299007653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f888808fef0 msgr2=0x7f888808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:44.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888c891700 1 --2- 172.21.15.16:0/1299007653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f888808fef0 0x7f888808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:44.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888c891700 1 -- 172.21.15.16:0/1299007653 >> v1:172.21.15.76:6789/0 conn(0x7f888808c6d0 legacy=0x7f888808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:44.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888c891700 1 -- 172.21.15.16:0/1299007653 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888808e620 con 0x7f888808d2c0 2019-12-05T06:21:44.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888c891700 1 --2- 172.21.15.16:0/1299007653 >> v2:172.21.15.16:3300/0 conn(0x7f888808d2c0 0x7f888808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f888400f740 tx=0x7f888400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888c891700 1 --2- 172.21.15.16:0/1299007653 >> v2:172.21.15.16:3300/0 conn(0x7f888808d2c0 0x7f888808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f888400f740 tx=0x7f888400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888c891700 1 --2- 172.21.15.16:0/1299007653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888808d2c0 0x7f888808ef20 secure :-1 s=READY pgs=624 cs=0 l=1 rx=0x7f888400f740 tx=0x7f888400f570).ready entity=mon.0 client_cookie=6a8882f026785ee5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:44.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888c891700 1 --2- 172.21.15.16:0/1299007653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888808d2c0 0x7f888808ef20 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f888400f740 tx=0x7f888400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888c891700 1 --2- 172.21.15.16:0/1299007653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888808d2c0 0x7f888808ef20 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f888400f740 tx=0x7f888400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f887bfff700 1 -- 172.21.15.16:0/1299007653 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8884000960 con 0x7f888808d2c0 2019-12-05T06:21:44.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888c891700 1 --2- 172.21.15.16:0/1299007653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888808d2c0 0x7f888808ef20 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f888400f740 tx=0x7f888400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f887bfff700 1 -- 172.21.15.16:0/1299007653 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8884000c90 con 0x7f888808d2c0 2019-12-05T06:21:44.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f887bfff700 1 -- 172.21.15.16:0/1299007653 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88840013a0 con 0x7f888808d2c0 2019-12-05T06:21:44.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888eaf5700 1 -- 172.21.15.16:0/1299007653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888808d2c0 msgr2=0x7f888808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:44.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.543+0000 7f888eaf5700 1 --2- 172.21.15.16:0/1299007653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888808d2c0 0x7f888808ef20 secure :-1 s=READY pgs=624 cs=0 l=1 rx=0x7f888400f740 tx=0x7f888400f570).stop 2019-12-05T06:21:44.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.544+0000 7f888eaf5700 1 -- 172.21.15.16:0/1299007653 shutdown_connections 2019-12-05T06:21:44.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.544+0000 7f888eaf5700 1 --2- 172.21.15.16:0/1299007653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f888808fef0 0x7f888808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:44.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.544+0000 7f888eaf5700 1 --2- 172.21.15.16:0/1299007653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f888808d2c0 0x7f888808ef20 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.544+0000 7f888eaf5700 1 -- 172.21.15.16:0/1299007653 shutdown_connections 2019-12-05T06:21:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.544+0000 7f888eaf5700 1 -- 172.21.15.16:0/1299007653 wait complete. 2019-12-05T06:21:44.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.544+0000 7f888eaf5700 1 -- 172.21.15.16:0/1299007653 >> 172.21.15.16:0/1299007653 conn(0x7f8888061940 msgr2=0x7f8888096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.638+0000 7f888eaf5700 1 Processor -- start 2019-12-05T06:21:44.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.638+0000 7f888eaf5700 1 -- start start 2019-12-05T06:21:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888eaf5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8888132d00 0x7f888814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888eaf5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8888132530 0x7f8888157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888eaf5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8888132970 con 0x7f888814d400 2019-12-05T06:21:44.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888eaf5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f888809bb50 con 0x7f8888132530 2019-12-05T06:21:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888eaf5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f888809aa60 con 0x7f8888132d00 2019-12-05T06:21:44.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888d893700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8888132530 0x7f8888157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888c891700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8888132d00 0x7f888814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888d893700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8888132530 0x7f8888157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51470/0 (socket says 172.21.15.16:51470) 2019-12-05T06:21:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888d893700 1 -- 172.21.15.16:0/3581290360 learned_addr learned my addr 172.21.15.16:0/3581290360 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:44.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888d092700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f888814d400 0x7f8888131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57936/0 (socket says 172.21.15.16:57936) 2019-12-05T06:21:44.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888d893700 1 -- 172.21.15.16:0/3581290360 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8888132d00 msgr2=0x7f888814d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:44.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8888132d00 0x7f888814d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:44.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.640+0000 7f888d893700 1 -- 172.21.15.16:0/3581290360 >> v1:172.21.15.76:6789/0 conn(0x7f888814d400 legacy=0x7f8888131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:44.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f888d893700 1 -- 172.21.15.16:0/3581290360 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888809b0f0 con 0x7f8888132530 2019-12-05T06:21:44.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> v2:172.21.15.16:3300/0 conn(0x7f8888132530 0x7f8888157100 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> v2:172.21.15.16:3300/0 conn(0x7f8888132530 0x7f8888157100 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 0x7f8888157100 secure :-1 s=READY pgs=625 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:44.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 0x7f8888157100 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 0x7f8888157100 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 0x7f8888157100 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f8879ffb700 1 -- 172.21.15.16:0/3581290360 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f888000aa30 con 0x7f8888132530 2019-12-05T06:21:44.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f8879ffb700 1 -- 172.21.15.16:0/3581290360 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88800013e0 con 0x7f8888132530 2019-12-05T06:21:44.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f8879ffb700 1 -- 172.21.15.16:0/3581290360 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88800019b0 con 0x7f8888132530 2019-12-05T06:21:44.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f888eaf5700 1 -- 172.21.15.16:0/3581290360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f888809b0f0 con 0x7f8888132530 2019-12-05T06:21:44.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.641+0000 7f888eaf5700 1 -- 172.21.15.16:0/3581290360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8888132970 con 0x7f8888132530 2019-12-05T06:21:44.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.642+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 0x7f8888157100 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.643+0000 7f8879ffb700 1 -- 172.21.15.16:0/3581290360 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8880000df0 con 0x7f8888132530 2019-12-05T06:21:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.643+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 0x7f8888157100 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.643+0000 7f8879ffb700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8868023630 0x7f8868025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.643+0000 7f8879ffb700 1 -- 172.21.15.16:0/3581290360 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f8880046170 con 0x7f8888132530 2019-12-05T06:21:44.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.643+0000 7f888d092700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8868023630 0x7f8868025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.644+0000 7f888d092700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8868023630 0x7f8868025ac0 crc :-1 s=READY pgs=280 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:44.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.656+0000 7f888eaf5700 1 -- 172.21.15.16:0/3581290360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f888809b0f0 con 0x7f8888132530 2019-12-05T06:21:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.662+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 0x7f8888157100 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.662+0000 7f8879ffb700 1 -- 172.21.15.16:0/3581290360 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f888002b190 con 0x7f8888132530 2019-12-05T06:21:44.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.905+0000 7f888eaf5700 1 -- 172.21.15.16:0/3581290360 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f888809b0f0 con 0x7f8888132530 2019-12-05T06:21:44.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.905+0000 7f888d893700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 0x7f8888157100 secure :-1 s=THROTTLE_DONE pgs=625 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:44.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.905+0000 7f8879ffb700 1 -- 172.21.15.16:0/3581290360 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f8880001d30 con 0x7f8888132530 2019-12-05T06:21:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2019-12-05T06:21:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.908+0000 7f888eaf5700 1 -- 172.21.15.16:0/3581290360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8868023630 msgr2=0x7f8868025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.908+0000 7f888eaf5700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8868023630 0x7f8868025ac0 crc :-1 s=READY pgs=280 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:44.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.908+0000 7f888eaf5700 1 -- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 msgr2=0x7f8888157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.908+0000 7f888eaf5700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 0x7f8888157100 secure :-1 s=READY pgs=625 cs=0 l=1 rx=0x7f88800107e0 tx=0x7f88800105e0).stop 2019-12-05T06:21:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.909+0000 7f888eaf5700 1 -- 172.21.15.16:0/3581290360 shutdown_connections 2019-12-05T06:21:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.909+0000 7f888eaf5700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8868023630 0x7f8868025ac0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.909+0000 7f888eaf5700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8888132530 0x7f8888157100 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.909+0000 7f888eaf5700 1 --2- 172.21.15.16:0/3581290360 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8888132d00 0x7f888814d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:44.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.909+0000 7f888eaf5700 1 -- 172.21.15.16:0/3581290360 shutdown_connections 2019-12-05T06:21:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.909+0000 7f888eaf5700 1 -- 172.21.15.16:0/3581290360 wait complete. 2019-12-05T06:21:44.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:44.909+0000 7f888eaf5700 1 -- 172.21.15.16:0/3581290360 >> 172.21.15.16:0/3581290360 conn(0x7f88880714f0 msgr2=0x7f888809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:44.899 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:44.899 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:43.122616+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:44.907 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:21:44.907 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:21:44.907 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:45.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.067+0000 7f08acdb4700 1 Processor -- start 2019-12-05T06:21:45.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.068+0000 7f08acdb4700 1 -- start start 2019-12-05T06:21:45.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.068+0000 7f08acdb4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08a808c6d0 0x7f08a808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:45.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.068+0000 7f08acdb4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f08a808fef0 0x7f08a808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:45.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08acdb4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a808fb80 con 0x7f08a808ef20 2019-12-05T06:21:45.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08acdb4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a808df90 con 0x7f08a808c6d0 2019-12-05T06:21:45.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08acdb4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a80a3f70 con 0x7f08a808fef0 2019-12-05T06:21:45.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08a808c6d0 0x7f08a808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:45.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08a808c6d0 0x7f08a808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51474/0 (socket says 172.21.15.16:51474) 2019-12-05T06:21:45.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 -- 172.21.15.16:0/3777292554 learned_addr learned my addr 172.21.15.16:0/3777292554 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a659c700 1 --2- 172.21.15.16:0/3777292554 >> v2:172.21.15.76:3300/0 conn(0x7f08a808fef0 0x7f08a808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 -- 172.21.15.16:0/3777292554 >> v2:172.21.15.76:3300/0 conn(0x7f08a808fef0 msgr2=0x7f08a808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/3777292554 >> v2:172.21.15.76:3300/0 conn(0x7f08a808fef0 0x7f08a808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 -- 172.21.15.16:0/3777292554 >> v1:172.21.15.76:6789/0 conn(0x7f08a808ef20 legacy=0x7f08a808f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 -- 172.21.15.16:0/3777292554 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a808d540 con 0x7f08a808c6d0 2019-12-05T06:21:45.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/3777292554 >> v2:172.21.15.16:3300/0 conn(0x7f08a808c6d0 0x7f08a808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f08a000fac0 tx=0x7f08a000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/3777292554 >> v2:172.21.15.16:3300/0 conn(0x7f08a808c6d0 0x7f08a808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f08a000fac0 tx=0x7f08a000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/3777292554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a808c6d0 0x7f08a808cac0 secure :-1 s=READY pgs=626 cs=0 l=1 rx=0x7f08a000fac0 tx=0x7f08a000f8f0).ready entity=mon.0 client_cookie=d3ac8325f6fdc499 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/3777292554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a808c6d0 0x7f08a808cac0 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7f08a000fac0 tx=0x7f08a000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a5d9b700 1 -- 172.21.15.16:0/3777292554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a0000ce0 con 0x7f08a808c6d0 2019-12-05T06:21:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/3777292554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a808c6d0 0x7f08a808cac0 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7f08a000fac0 tx=0x7f08a000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/3777292554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a808c6d0 0x7f08a808cac0 secure :-1 s=THROTTLE_DONE pgs=626 cs=0 l=1 rx=0x7f08a000fac0 tx=0x7f08a000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a5d9b700 1 -- 172.21.15.16:0/3777292554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a0001010 con 0x7f08a808c6d0 2019-12-05T06:21:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.069+0000 7f08a5d9b700 1 -- 172.21.15.16:0/3777292554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a0001720 con 0x7f08a808c6d0 2019-12-05T06:21:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.070+0000 7f08acdb4700 1 -- 172.21.15.16:0/3777292554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a808c6d0 msgr2=0x7f08a808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:45.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.070+0000 7f08acdb4700 1 --2- 172.21.15.16:0/3777292554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a808c6d0 0x7f08a808cac0 secure :-1 s=READY pgs=626 cs=0 l=1 rx=0x7f08a000fac0 tx=0x7f08a000f8f0).stop 2019-12-05T06:21:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.070+0000 7f08acdb4700 1 -- 172.21.15.16:0/3777292554 shutdown_connections 2019-12-05T06:21:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.070+0000 7f08acdb4700 1 --2- 172.21.15.16:0/3777292554 >> v2:172.21.15.76:3300/0 conn(0x7f08a808fef0 0x7f08a808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:45.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.070+0000 7f08acdb4700 1 --2- 172.21.15.16:0/3777292554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a808c6d0 0x7f08a808cac0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.070+0000 7f08acdb4700 1 -- 172.21.15.16:0/3777292554 shutdown_connections 2019-12-05T06:21:45.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.070+0000 7f08acdb4700 1 -- 172.21.15.16:0/3777292554 wait complete. 2019-12-05T06:21:45.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.070+0000 7f08acdb4700 1 -- 172.21.15.16:0/3777292554 >> 172.21.15.16:0/3777292554 conn(0x7f08a8061940 msgr2=0x7f08a8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:45.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.073+0000 7f08acdb4700 1 Processor -- start 2019-12-05T06:21:45.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.073+0000 7f08acdb4700 1 -- start start 2019-12-05T06:21:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08acdb4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08a809e440 0x7f08a8152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08acdb4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08a8088380 0x7f08a819b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:45.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08acdb4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08a8084ed0 con 0x7f08a8088b50 2019-12-05T06:21:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08acdb4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08a8086bf0 con 0x7f08a809e440 2019-12-05T06:21:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08acdb4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08a80861c0 con 0x7f08a8088380 2019-12-05T06:21:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08a809e440 0x7f08a8152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08a809e440 0x7f08a8152c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51484/0 (socket says 172.21.15.16:51484) 2019-12-05T06:21:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a6d9d700 1 -- 172.21.15.16:0/2052973143 learned_addr learned my addr 172.21.15.16:0/2052973143 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:45.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a759e700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08a8088380 0x7f08a819b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a6d9d700 1 -- 172.21.15.16:0/2052973143 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08a8088380 msgr2=0x7f08a819b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08a8088380 0x7f08a819b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a6d9d700 1 -- 172.21.15.16:0/2052973143 >> v1:172.21.15.76:6789/0 conn(0x7f08a8088b50 legacy=0x7f08a8087b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a6d9d700 1 -- 172.21.15.16:0/2052973143 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a8087280 con 0x7f08a809e440 2019-12-05T06:21:45.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> v2:172.21.15.16:3300/0 conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> v2:172.21.15.16:3300/0 conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.074+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=READY pgs=627 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.075+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.075+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.075+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.075+0000 7f089f7fe700 1 -- 172.21.15.16:0/2052973143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a00010c0 con 0x7f08a809e440 2019-12-05T06:21:45.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.075+0000 7f089f7fe700 1 -- 172.21.15.16:0/2052973143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a00080f0 con 0x7f08a809e440 2019-12-05T06:21:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.075+0000 7f08acdb4700 1 -- 172.21.15.16:0/2052973143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08a8087280 con 0x7f08a809e440 2019-12-05T06:21:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.075+0000 7f089f7fe700 1 -- 172.21.15.16:0/2052973143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a00016a0 con 0x7f08a809e440 2019-12-05T06:21:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.075+0000 7f08acdb4700 1 -- 172.21.15.16:0/2052973143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08a8084ed0 con 0x7f08a809e440 2019-12-05T06:21:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.075+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.076+0000 7f089f7fe700 1 -- 172.21.15.16:0/2052973143 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f08a00010e0 con 0x7f08a809e440 2019-12-05T06:21:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.076+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.076+0000 7f089f7fe700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f088c023630 0x7f088c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.076+0000 7f089f7fe700 1 -- 172.21.15.16:0/2052973143 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f08a0046180 con 0x7f08a809e440 2019-12-05T06:21:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.076+0000 7f08a659c700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f088c023630 0x7f088c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.077+0000 7f08a659c700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f088c023630 0x7f088c025ac0 crc :-1 s=READY pgs=281 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:45.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.083+0000 7f08acdb4700 1 -- 172.21.15.16:0/2052973143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08a8087280 con 0x7f08a809e440 2019-12-05T06:21:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.089+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.089+0000 7f089f7fe700 1 -- 172.21.15.16:0/2052973143 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f08a0000ed0 con 0x7f08a809e440 2019-12-05T06:21:45.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.272+0000 7f08acdb4700 1 -- 172.21.15.16:0/2052973143 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08a8087280 con 0x7f08a809e440 2019-12-05T06:21:45.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.272+0000 7f08a6d9d700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=THROTTLE_DONE pgs=627 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:45.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.272+0000 7f089f7fe700 1 -- 172.21.15.16:0/2052973143 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f08a00010e0 con 0x7f08a809e440 2019-12-05T06:21:45.261 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 78 2019-12-05T06:21:45.261 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:45.261 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":78,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":78,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:43.122616+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:45.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 -- 172.21.15.16:0/2052973143 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f088c023630 msgr2=0x7f088c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:45.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f088c023630 0x7f088c025ac0 crc :-1 s=READY pgs=281 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 -- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 msgr2=0x7f08a8152c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 0x7f08a8152c20 secure :-1 s=READY pgs=627 cs=0 l=1 rx=0x7f08a00107c0 tx=0x7f08a00105f0).stop 2019-12-05T06:21:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 -- 172.21.15.16:0/2052973143 shutdown_connections 2019-12-05T06:21:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08a8088380 0x7f08a819b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:45.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f088c023630 0x7f088c025ac0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:45.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 --2- 172.21.15.16:0/2052973143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08a809e440 0x7f08a8152c20 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:45.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 -- 172.21.15.16:0/2052973143 shutdown_connections 2019-12-05T06:21:45.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 -- 172.21.15.16:0/2052973143 wait complete. 2019-12-05T06:21:45.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:45.275+0000 7f08acdb4700 1 -- 172.21.15.16:0/2052973143 >> 172.21.15.16:0/2052973143 conn(0x7f08a8098480 msgr2=0x7f08a8071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:45.274 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 78, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 78, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5776": { "addr": "172.21.15.16:6833/3114985010", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 3114985010, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 3114985010, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5776, "incarnation": 78, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 59, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:21:43.122616+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5776 } } } ], "standbys": [] } 2019-12-05T06:21:47.275 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:47.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.565+0000 7fb95bc03700 1 Processor -- start 2019-12-05T06:21:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.565+0000 7fb95bc03700 1 -- start start 2019-12-05T06:21:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.567+0000 7fb95bc03700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.567+0000 7fb95bc03700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb95408f720 0x7fb95408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.567+0000 7fb95bc03700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9540958e0 con 0x7fb95408fef0 2019-12-05T06:21:47.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.567+0000 7fb95bc03700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9540a3f70 con 0x7fb95408f720 2019-12-05T06:21:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.567+0000 7fb95bc03700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb95408d830 con 0x7fb95408c6d0 2019-12-05T06:21:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.567+0000 7fb95a1a0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.567+0000 7fb95a9a1700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb95408f720 0x7fb95408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.568+0000 7fb95a1a0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:55480/0 (socket says 172.21.15.16:55480) 2019-12-05T06:21:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.568+0000 7fb95a9a1700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb95408f720 0x7fb95408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:37584/0 (socket says 172.21.15.16:37584) 2019-12-05T06:21:47.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.568+0000 7fb95a1a0700 1 -- 172.21.15.16:0/1112367770 learned_addr learned my addr 172.21.15.16:0/1112367770 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.568+0000 7fb95999f700 1 --1- 172.21.15.16:0/1112367770 >> v1:172.21.15.76:6789/0 conn(0x7fb95408fef0 0x7fb95408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:57962/0 (socket says 172.21.15.16:57962) 2019-12-05T06:21:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.568+0000 7fb95a1a0700 1 -- 172.21.15.16:0/1112367770 >> v2:172.21.15.76:3300/0 conn(0x7fb95408f720 msgr2=0x7fb95408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.568+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1112367770 >> v2:172.21.15.76:3300/0 conn(0x7fb95408f720 0x7fb95408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.568+0000 7fb95a1a0700 1 -- 172.21.15.16:0/1112367770 >> v1:172.21.15.76:6789/0 conn(0x7fb95408fef0 legacy=0x7fb95408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.568+0000 7fb95a1a0700 1 -- 172.21.15.16:0/1112367770 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95408fbc0 con 0x7fb95408c6d0 2019-12-05T06:21:47.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.568+0000 7fb95a9a1700 1 --2- 172.21.15.16:0/1112367770 >> v2:172.21.15.76:3300/0 conn(0x7fb95408f720 0x7fb95408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:21:47.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.568+0000 7fb95919e700 1 -- 172.21.15.16:0/1112367770 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3512999933 0 0) 0x7fb948001080 con 0x7fb95408fef0 2019-12-05T06:21:47.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.569+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1112367770 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb94c00fac0 tx=0x7fb94c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.569+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1112367770 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb94c00fac0 tx=0x7fb94c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.569+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1112367770 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 secure :-1 s=READY pgs=155 cs=0 l=1 rx=0x7fb94c00fac0 tx=0x7fb94c00fe60).ready entity=mon.2 client_cookie=4833de73e9d4fa31 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:47.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.569+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1112367770 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7fb94c00fac0 tx=0x7fb94c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.569+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1112367770 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7fb94c00fac0 tx=0x7fb94c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.569+0000 7fb95919e700 1 -- 172.21.15.16:0/1112367770 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb94c000ce0 con 0x7fb95408c6d0 2019-12-05T06:21:47.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.569+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1112367770 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 secure :-1 s=THROTTLE_DONE pgs=155 cs=0 l=1 rx=0x7fb94c00fac0 tx=0x7fb94c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.569+0000 7fb95919e700 1 -- 172.21.15.16:0/1112367770 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c001010 con 0x7fb95408c6d0 2019-12-05T06:21:47.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.569+0000 7fb95919e700 1 -- 172.21.15.16:0/1112367770 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb94c001760 con 0x7fb95408c6d0 2019-12-05T06:21:47.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.569+0000 7fb95bc03700 1 -- 172.21.15.16:0/1112367770 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 msgr2=0x7fb95408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:47.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.570+0000 7fb95bc03700 1 --2- 172.21.15.16:0/1112367770 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 secure :-1 s=READY pgs=155 cs=0 l=1 rx=0x7fb94c00fac0 tx=0x7fb94c00fe60).stop 2019-12-05T06:21:47.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.571+0000 7fb95bc03700 1 -- 172.21.15.16:0/1112367770 shutdown_connections 2019-12-05T06:21:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.571+0000 7fb95bc03700 1 --2- 172.21.15.16:0/1112367770 >> v2:172.21.15.76:3300/0 conn(0x7fb95408f720 0x7fb95408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.571+0000 7fb95bc03700 1 --2- 172.21.15.16:0/1112367770 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb95408c6d0 0x7fb95408cac0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.571+0000 7fb95bc03700 1 -- 172.21.15.16:0/1112367770 shutdown_connections 2019-12-05T06:21:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.571+0000 7fb95bc03700 1 -- 172.21.15.16:0/1112367770 wait complete. 2019-12-05T06:21:47.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.571+0000 7fb95bc03700 1 -- 172.21.15.16:0/1112367770 >> 172.21.15.16:0/1112367770 conn(0x7fb954061940 msgr2=0x7fb954096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:47.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.576+0000 7fb95bc03700 1 Processor -- start 2019-12-05T06:21:47.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.577+0000 7fb95bc03700 1 -- start start 2019-12-05T06:21:47.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.578+0000 7fb95bc03700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:47.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.578+0000 7fb95bc03700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb954128b20 con 0x7fb954154c60 2019-12-05T06:21:47.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.578+0000 7fb95bc03700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb954128660 con 0x7fb95412c740 2019-12-05T06:21:47.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.578+0000 7fb95bc03700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb954129db0 con 0x7fb9541325e0 2019-12-05T06:21:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.578+0000 7fb95a9a1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb954154c60 0x7fb954157060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36668/0 (socket says 172.21.15.16:36668) 2019-12-05T06:21:47.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.578+0000 7fb95a9a1700 1 -- 172.21.15.16:0/1442563778 learned_addr learned my addr 172.21.15.16:0/1442563778 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.578+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.579+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317755093 0 0) 0x7fb944009890 con 0x7fb954154c60 2019-12-05T06:21:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.579+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb934002e80 con 0x7fb954154c60 2019-12-05T06:21:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.579+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 104859138 0 0) 0x7fb948001080 con 0x7fb95412c740 2019-12-05T06:21:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.579+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb934004220 con 0x7fb95412c740 2019-12-05T06:21:47.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.579+0000 7fb95a1a0700 1 -- 172.21.15.16:0/1442563778 >> v1:172.21.15.76:6789/0 conn(0x7fb95412c740 legacy=0x7fb95412bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.579+0000 7fb95a1a0700 1 -- 172.21.15.16:0/1442563778 >> v1:172.21.15.16:6789/0 conn(0x7fb954154c60 legacy=0x7fb954157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.579+0000 7fb95a1a0700 1 -- 172.21.15.16:0/1442563778 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95412ae70 con 0x7fb9541325e0 2019-12-05T06:21:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.579+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.580+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.580+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=READY pgs=156 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:47.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.580+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.580+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.580+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb94c008190 con 0x7fb9541325e0 2019-12-05T06:21:47.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.580+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.580+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb94c005710 con 0x7fb9541325e0 2019-12-05T06:21:47.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.580+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb94c005f00 con 0x7fb9541325e0 2019-12-05T06:21:47.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.580+0000 7fb95bc03700 1 -- 172.21.15.16:0/1442563778 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb95412ae70 con 0x7fb9541325e0 2019-12-05T06:21:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.580+0000 7fb95bc03700 1 -- 172.21.15.16:0/1442563778 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb95412ae70 con 0x7fb9541325e0 2019-12-05T06:21:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.581+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.582+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb94c008230 con 0x7fb9541325e0 2019-12-05T06:21:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.582+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.583+0000 7fb952ffd700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb934023650 0x7fb934025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.583+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fb94c08a150 con 0x7fb9541325e0 2019-12-05T06:21:47.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.583+0000 7fb95999f700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb934023650 0x7fb934025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:47.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.584+0000 7fb95999f700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb934023650 0x7fb934025ae0 crc :-1 s=READY pgs=282 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:47.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.596+0000 7fb95bc03700 1 -- 172.21.15.16:0/1442563778 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb95412ae70 con 0x7fb9541325e0 2019-12-05T06:21:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.605+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.605+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb94c05c480 con 0x7fb9541325e0 2019-12-05T06:21:47.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.849+0000 7fb95bc03700 1 -- 172.21.15.16:0/1442563778 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb95412ae70 con 0x7fb9541325e0 2019-12-05T06:21:47.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.850+0000 7fb95a1a0700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=THROTTLE_DONE pgs=156 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:47.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.850+0000 7fb952ffd700 1 -- 172.21.15.16:0/1442563778 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fb94c05c480 con 0x7fb9541325e0 2019-12-05T06:21:47.840 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 80 2019-12-05T06:21:47.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:47.841 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":80,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":80,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:46.966798+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:rejoin","state_seq":17,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.856+0000 7fb95bc03700 1 -- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb934023650 msgr2=0x7fb934025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.856+0000 7fb95bc03700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb934023650 0x7fb934025ae0 crc :-1 s=READY pgs=282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:47.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.856+0000 7fb95bc03700 1 -- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 msgr2=0x7fb95412b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:47.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.856+0000 7fb95bc03700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 secure :-1 s=READY pgs=156 cs=0 l=1 rx=0x7fb94c054740 tx=0x7fb94c054ae0).stop 2019-12-05T06:21:47.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.856+0000 7fb95bc03700 1 -- 172.21.15.16:0/1442563778 shutdown_connections 2019-12-05T06:21:47.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.856+0000 7fb95bc03700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb934023650 0x7fb934025ae0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:47.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.856+0000 7fb95bc03700 1 --2- 172.21.15.16:0/1442563778 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9541325e0 0x7fb95412b770 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:47.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.856+0000 7fb95bc03700 1 -- 172.21.15.16:0/1442563778 shutdown_connections 2019-12-05T06:21:47.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.856+0000 7fb95bc03700 1 -- 172.21.15.16:0/1442563778 wait complete. 2019-12-05T06:21:47.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:47.856+0000 7fb95bc03700 1 -- 172.21.15.16:0/1442563778 >> 172.21.15.16:0/1442563778 conn(0x7fb9540a2850 msgr2=0x7fb95406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:47.860 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:21:49.861 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:50.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.101+0000 7fb2df574700 1 Processor -- start 2019-12-05T06:21:50.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.101+0000 7fb2df574700 1 -- start start 2019-12-05T06:21:50.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.103+0000 7fb2df574700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb2d808c6d0 0x7fb2d808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:50.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.103+0000 7fb2df574700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d808f720 0x7fb2d808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:50.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.103+0000 7fb2df574700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d808fb70 con 0x7fb2d808fef0 2019-12-05T06:21:50.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.103+0000 7fb2df574700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb2d808df90 con 0x7fb2d808c6d0 2019-12-05T06:21:50.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.103+0000 7fb2df574700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d80a3f70 con 0x7fb2d808f720 2019-12-05T06:21:50.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.103+0000 7fb2ddb11700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb2d808c6d0 0x7fb2d808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.103+0000 7fb2ddb11700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb2d808c6d0 0x7fb2d808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51512/0 (socket says 172.21.15.16:51512) 2019-12-05T06:21:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.103+0000 7fb2ddb11700 1 -- 172.21.15.16:0/3381520037 learned_addr learned my addr 172.21.15.16:0/3381520037 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.103+0000 7fb2dd310700 1 --1- 172.21.15.16:0/3381520037 >> v1:172.21.15.16:6789/0 conn(0x7fb2d808fef0 0x7fb2d808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36682/0 (socket says 172.21.15.16:36682) 2019-12-05T06:21:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.103+0000 7fb2de312700 1 --2- 172.21.15.16:0/3381520037 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d808f720 0x7fb2d808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2ddb11700 1 -- 172.21.15.16:0/3381520037 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d808f720 msgr2=0x7fb2d808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:50.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2ddb11700 1 --2- 172.21.15.16:0/3381520037 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d808f720 0x7fb2d808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:50.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2ddb11700 1 -- 172.21.15.16:0/3381520037 >> v1:172.21.15.16:6789/0 conn(0x7fb2d808fef0 legacy=0x7fb2d808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:50.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2dcb0f700 1 -- 172.21.15.16:0/3381520037 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4008308874 0 0) 0x7fb2c8001100 con 0x7fb2d808fef0 2019-12-05T06:21:50.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2ddb11700 1 -- 172.21.15.16:0/3381520037 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d808d540 con 0x7fb2d808c6d0 2019-12-05T06:21:50.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2ddb11700 1 --2- 172.21.15.16:0/3381520037 >> v2:172.21.15.16:3300/0 conn(0x7fb2d808c6d0 0x7fb2d808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb2d400f6c0 tx=0x7fb2d400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2ddb11700 1 --2- 172.21.15.16:0/3381520037 >> v2:172.21.15.16:3300/0 conn(0x7fb2d808c6d0 0x7fb2d808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb2d400f6c0 tx=0x7fb2d400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2ddb11700 1 --2- 172.21.15.16:0/3381520037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2d808c6d0 0x7fb2d808cac0 secure :-1 s=READY pgs=630 cs=0 l=1 rx=0x7fb2d400f6c0 tx=0x7fb2d400fa60).ready entity=mon.0 client_cookie=a5ef80c38621fa69 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:50.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2ddb11700 1 --2- 172.21.15.16:0/3381520037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2d808c6d0 0x7fb2d808cac0 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7fb2d400f6c0 tx=0x7fb2d400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2dcb0f700 1 -- 172.21.15.16:0/3381520037 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d4000960 con 0x7fb2d808c6d0 2019-12-05T06:21:50.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2ddb11700 1 --2- 172.21.15.16:0/3381520037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2d808c6d0 0x7fb2d808cac0 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7fb2d400f6c0 tx=0x7fb2d400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.104+0000 7fb2ddb11700 1 --2- 172.21.15.16:0/3381520037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2d808c6d0 0x7fb2d808cac0 secure :-1 s=THROTTLE_DONE pgs=630 cs=0 l=1 rx=0x7fb2d400f6c0 tx=0x7fb2d400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.105+0000 7fb2dcb0f700 1 -- 172.21.15.16:0/3381520037 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2d4000c90 con 0x7fb2d808c6d0 2019-12-05T06:21:50.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.105+0000 7fb2dcb0f700 1 -- 172.21.15.16:0/3381520037 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2d40013a0 con 0x7fb2d808c6d0 2019-12-05T06:21:50.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.105+0000 7fb2df574700 1 -- 172.21.15.16:0/3381520037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2d808c6d0 msgr2=0x7fb2d808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:50.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.105+0000 7fb2df574700 1 --2- 172.21.15.16:0/3381520037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2d808c6d0 0x7fb2d808cac0 secure :-1 s=READY pgs=630 cs=0 l=1 rx=0x7fb2d400f6c0 tx=0x7fb2d400fa60).stop 2019-12-05T06:21:50.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.106+0000 7fb2df574700 1 -- 172.21.15.16:0/3381520037 shutdown_connections 2019-12-05T06:21:50.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.106+0000 7fb2df574700 1 --2- 172.21.15.16:0/3381520037 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d808f720 0x7fb2d808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:50.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.106+0000 7fb2df574700 1 --2- 172.21.15.16:0/3381520037 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb2d808c6d0 0x7fb2d808cac0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:50.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.107+0000 7fb2df574700 1 -- 172.21.15.16:0/3381520037 shutdown_connections 2019-12-05T06:21:50.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.107+0000 7fb2df574700 1 -- 172.21.15.16:0/3381520037 wait complete. 2019-12-05T06:21:50.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.107+0000 7fb2df574700 1 -- 172.21.15.16:0/3381520037 >> 172.21.15.16:0/3381520037 conn(0x7fb2d8061940 msgr2=0x7fb2d8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:50.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.111+0000 7fb2df574700 1 Processor -- start 2019-12-05T06:21:50.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.111+0000 7fb2df574700 1 -- start start 2019-12-05T06:21:50.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.112+0000 7fb2df574700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:50.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.113+0000 7fb2df574700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d8132970 con 0x7fb2d8132530 2019-12-05T06:21:50.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.113+0000 7fb2df574700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb2d8098da0 con 0x7fb2d814d400 2019-12-05T06:21:50.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.113+0000 7fb2df574700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d809bb50 con 0x7fb2d8132d00 2019-12-05T06:21:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.113+0000 7fb2de312700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb2d8132530 0x7fb2d8157100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36690/0 (socket says 172.21.15.16:36690) 2019-12-05T06:21:50.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.113+0000 7fb2de312700 1 -- 172.21.15.16:0/3720012589 learned_addr learned my addr 172.21.15.16:0/3720012589 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.113+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.113+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3278173725 0 0) 0x7fb2d00014d0 con 0x7fb2d8132530 2019-12-05T06:21:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.113+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c0002e80 con 0x7fb2d8132530 2019-12-05T06:21:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.113+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566465180 0 0) 0x7fb2d4008020 con 0x7fb2d814d400 2019-12-05T06:21:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.113+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2c00030b0 con 0x7fb2d814d400 2019-12-05T06:21:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2dd310700 1 -- 172.21.15.16:0/3720012589 >> v1:172.21.15.76:6789/0 conn(0x7fb2d814d400 legacy=0x7fb2d8131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:50.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2dd310700 1 -- 172.21.15.16:0/3720012589 >> v1:172.21.15.16:6789/0 conn(0x7fb2d8132530 legacy=0x7fb2d8157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:50.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2dd310700 1 -- 172.21.15.16:0/3720012589 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d809b0f0 con 0x7fb2d8132d00 2019-12-05T06:21:50.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:50.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c8004240 con 0x7fb2d8132d00 2019-12-05T06:21:50.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2df574700 1 -- 172.21.15.16:0/3720012589 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d809b0f0 con 0x7fb2d8132d00 2019-12-05T06:21:50.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2c8000c90 con 0x7fb2d8132d00 2019-12-05T06:21:50.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.114+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2c8004620 con 0x7fb2d8132d00 2019-12-05T06:21:50.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.115+0000 7fb2df574700 1 -- 172.21.15.16:0/3720012589 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d809b0f0 con 0x7fb2d8132d00 2019-12-05T06:21:50.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.115+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.116+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.116+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb2c8004240 con 0x7fb2d8132d00 2019-12-05T06:21:50.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.116+0000 7fb2ce7fc700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2c0023650 0x7fb2c0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:50.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.116+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fb2c8045e00 con 0x7fb2d8132d00 2019-12-05T06:21:50.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.117+0000 7fb2ddb11700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2c0023650 0x7fb2c0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:50.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.117+0000 7fb2ddb11700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2c0023650 0x7fb2c0025ae0 crc :-1 s=READY pgs=283 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:50.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.132+0000 7fb2df574700 1 -- 172.21.15.16:0/3720012589 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d809b0f0 con 0x7fb2d8132d00 2019-12-05T06:21:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.141+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.141+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb2c8000980 con 0x7fb2d8132d00 2019-12-05T06:21:50.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.380+0000 7fb2df574700 1 -- 172.21.15.16:0/3720012589 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb2d809b0f0 con 0x7fb2d8132d00 2019-12-05T06:21:50.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.381+0000 7fb2dd310700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=THROTTLE_DONE pgs=157 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:50.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.381+0000 7fb2ce7fc700 1 -- 172.21.15.16:0/3720012589 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fb2c8000980 con 0x7fb2d8132d00 2019-12-05T06:21:50.369 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 81 2019-12-05T06:21:50.369 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:50.376 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:47.975651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":18,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:50.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.383+0000 7fb2df574700 1 -- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2c0023650 msgr2=0x7fb2c0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:50.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.383+0000 7fb2df574700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2c0023650 0x7fb2c0025ae0 crc :-1 s=READY pgs=283 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:50.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.383+0000 7fb2df574700 1 -- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 msgr2=0x7fb2d814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:50.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.383+0000 7fb2df574700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 secure :-1 s=READY pgs=157 cs=0 l=1 rx=0x7fb2c8010370 tx=0x7fb2c8010710).stop 2019-12-05T06:21:50.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.383+0000 7fb2df574700 1 -- 172.21.15.16:0/3720012589 shutdown_connections 2019-12-05T06:21:50.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.383+0000 7fb2df574700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2c0023650 0x7fb2c0025ae0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:50.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.383+0000 7fb2df574700 1 --2- 172.21.15.16:0/3720012589 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb2d8132d00 0x7fb2d814d7f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:50.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.383+0000 7fb2df574700 1 -- 172.21.15.16:0/3720012589 shutdown_connections 2019-12-05T06:21:50.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.383+0000 7fb2df574700 1 -- 172.21.15.16:0/3720012589 wait complete. 2019-12-05T06:21:50.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:50.383+0000 7fb2df574700 1 -- 172.21.15.16:0/3720012589 >> 172.21.15.16:0/3720012589 conn(0x7fb2d80714f0 msgr2=0x7fb2d809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:50.388 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5604 2019-12-05T06:21:50.388 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2019-12-05T06:21:55.390 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:21:55.390 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:21:55.390 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:21:55.403 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:21:55.403 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:55.572 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:21:55.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.675+0000 7f0b4c8dc700 1 Processor -- start 2019-12-05T06:21:55.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.675+0000 7f0b4c8dc700 1 -- start start 2019-12-05T06:21:55.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.677+0000 7f0b4c8dc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 0x7f0b4808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:55.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.677+0000 7f0b4c8dc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b480958e0 con 0x7f0b4808d2c0 2019-12-05T06:21:55.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.677+0000 7f0b4c8dc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b480629e0 con 0x7f0b4808c6d0 2019-12-05T06:21:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.677+0000 7f0b4c8dc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4808df90 con 0x7f0b4808fef0 2019-12-05T06:21:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.677+0000 7f0b4659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0b4808d2c0 0x7f0b4808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36730/0 (socket says 172.21.15.16:36730) 2019-12-05T06:21:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.677+0000 7f0b4659c700 1 -- 172.21.15.16:0/134132837 learned_addr learned my addr 172.21.15.16:0/134132837 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.677+0000 7f0b4759e700 1 --2- 172.21.15.16:0/134132837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 0x7f0b4808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.678+0000 7f0b45d9b700 1 -- 172.21.15.16:0/134132837 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780006128 0 0) 0x7f0b300014d0 con 0x7f0b4808d2c0 2019-12-05T06:21:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.678+0000 7f0b45d9b700 1 -- 172.21.15.16:0/134132837 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b28002e80 con 0x7f0b4808d2c0 2019-12-05T06:21:55.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.678+0000 7f0b45d9b700 1 -- 172.21.15.16:0/134132837 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533844794 0 0) 0x7f0b38001080 con 0x7f0b4808c6d0 2019-12-05T06:21:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.678+0000 7f0b45d9b700 1 -- 172.21.15.16:0/134132837 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b28004170 con 0x7f0b4808c6d0 2019-12-05T06:21:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.678+0000 7f0b4759e700 1 -- 172.21.15.16:0/134132837 >> v1:172.21.15.76:6789/0 conn(0x7f0b4808c6d0 legacy=0x7f0b4808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.678+0000 7f0b4759e700 1 -- 172.21.15.16:0/134132837 >> v1:172.21.15.16:6789/0 conn(0x7f0b4808d2c0 legacy=0x7f0b4808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.678+0000 7f0b4759e700 1 -- 172.21.15.16:0/134132837 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b4808e620 con 0x7f0b4808fef0 2019-12-05T06:21:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.678+0000 7f0b4759e700 1 --2- 172.21.15.16:0/134132837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 0x7f0b4808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0b4000af80 tx=0x7f0b4000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.678+0000 7f0b4759e700 1 --2- 172.21.15.16:0/134132837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 0x7f0b4808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0b4000af80 tx=0x7f0b4000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.679+0000 7f0b4759e700 1 --2- 172.21.15.16:0/134132837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 0x7f0b4808f720 secure :-1 s=READY pgs=158 cs=0 l=1 rx=0x7f0b4000af80 tx=0x7f0b4000b790).ready entity=mon.2 client_cookie=8807e2a35b434331 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.679+0000 7f0b4759e700 1 --2- 172.21.15.16:0/134132837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 0x7f0b4808f720 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f0b4000af80 tx=0x7f0b4000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.679+0000 7f0b4759e700 1 --2- 172.21.15.16:0/134132837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 0x7f0b4808f720 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f0b4000af80 tx=0x7f0b4000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.679+0000 7f0b45d9b700 1 -- 172.21.15.16:0/134132837 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b40007240 con 0x7f0b4808fef0 2019-12-05T06:21:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.679+0000 7f0b4759e700 1 --2- 172.21.15.16:0/134132837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 0x7f0b4808f720 secure :-1 s=THROTTLE_DONE pgs=158 cs=0 l=1 rx=0x7f0b4000af80 tx=0x7f0b4000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.679+0000 7f0b45d9b700 1 -- 172.21.15.16:0/134132837 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b40000bb0 con 0x7f0b4808fef0 2019-12-05T06:21:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.679+0000 7f0b45d9b700 1 -- 172.21.15.16:0/134132837 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b40009e60 con 0x7f0b4808fef0 2019-12-05T06:21:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.679+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/134132837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 msgr2=0x7f0b4808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.679+0000 7f0b4c8dc700 1 --2- 172.21.15.16:0/134132837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 0x7f0b4808f720 secure :-1 s=READY pgs=158 cs=0 l=1 rx=0x7f0b4000af80 tx=0x7f0b4000b790).stop 2019-12-05T06:21:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.681+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/134132837 shutdown_connections 2019-12-05T06:21:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.681+0000 7f0b4c8dc700 1 --2- 172.21.15.16:0/134132837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b4808fef0 0x7f0b4808f720 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.681+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/134132837 shutdown_connections 2019-12-05T06:21:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.681+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/134132837 wait complete. 2019-12-05T06:21:55.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.681+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/134132837 >> 172.21.15.16:0/134132837 conn(0x7f0b48061940 msgr2=0x7f0b48096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:55.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.684+0000 7f0b4c8dc700 1 Processor -- start 2019-12-05T06:21:55.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.684+0000 7f0b4c8dc700 1 -- start start 2019-12-05T06:21:55.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.685+0000 7f0b4c8dc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0b48153a50 0x7f0b48153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:55.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.685+0000 7f0b4c8dc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b48129a00 0x7f0b48152770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:55.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.685+0000 7f0b4c8dc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b4806e5c0 con 0x7f0b4812a1d0 2019-12-05T06:21:55.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.685+0000 7f0b4c8dc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b4806e250 con 0x7f0b48153a50 2019-12-05T06:21:55.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.685+0000 7f0b4c8dc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b4806eac0 con 0x7f0b48129a00 2019-12-05T06:21:55.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b4659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0b4812a1d0 0x7f0b48129200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:36740/0 (socket says 172.21.15.16:36740) 2019-12-05T06:21:55.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b4659c700 1 -- 172.21.15.16:0/3640950477 learned_addr learned my addr 172.21.15.16:0/3640950477 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:21:55.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b4759e700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b48129a00 0x7f0b48152770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:55.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> v2:172.21.15.76:3300/0 conn(0x7f0b48153a50 0x7f0b48153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b3f7fe700 1 -- 172.21.15.16:0/3640950477 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294713252 0 0) 0x7f0b300014d0 con 0x7f0b4812a1d0 2019-12-05T06:21:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b3f7fe700 1 -- 172.21.15.16:0/3640950477 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b20002e80 con 0x7f0b4812a1d0 2019-12-05T06:21:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b46d9d700 1 -- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b48129a00 msgr2=0x7f0b48152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:55.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b48129a00 0x7f0b48152770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b46d9d700 1 -- 172.21.15.16:0/3640950477 >> v1:172.21.15.16:6789/0 conn(0x7f0b4812a1d0 legacy=0x7f0b48129200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b46d9d700 1 -- 172.21.15.16:0/3640950477 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b48128900 con 0x7f0b48153a50 2019-12-05T06:21:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.686+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> v2:172.21.15.76:3300/0 conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.687+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> v2:172.21.15.76:3300/0 conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.687+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=READY pgs=463 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.687+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.687+0000 7f0b3f7fe700 1 -- 172.21.15.16:0/3640950477 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b38009960 con 0x7f0b48153a50 2019-12-05T06:21:55.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.687+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.687+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.687+0000 7f0b3f7fe700 1 -- 172.21.15.16:0/3640950477 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b38000aa0 con 0x7f0b48153a50 2019-12-05T06:21:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.687+0000 7f0b3f7fe700 1 -- 172.21.15.16:0/3640950477 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b38009f40 con 0x7f0b48153a50 2019-12-05T06:21:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.687+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/3640950477 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b48128900 con 0x7f0b48153a50 2019-12-05T06:21:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.687+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/3640950477 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b48128900 con 0x7f0b48153a50 2019-12-05T06:21:55.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.688+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.688+0000 7f0b3f7fe700 1 -- 172.21.15.16:0/3640950477 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0b38009ad0 con 0x7f0b48153a50 2019-12-05T06:21:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.688+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.689+0000 7f0b3f7fe700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b20023670 0x7f0b20025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.689+0000 7f0b3f7fe700 1 -- 172.21.15.16:0/3640950477 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f0b38045fb0 con 0x7f0b48153a50 2019-12-05T06:21:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.689+0000 7f0b4659c700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b20023670 0x7f0b20025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.690+0000 7f0b4659c700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b20023670 0x7f0b20025ae0 crc :-1 s=READY pgs=284 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:55.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.690+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/3640950477 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b48128900 con 0x7f0b48153a50 2019-12-05T06:21:55.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.699+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.699+0000 7f0b3f7fe700 1 -- 172.21.15.16:0/3640950477 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0b380098c0 con 0x7f0b48153a50 2019-12-05T06:21:55.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.947+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/3640950477 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b48128900 con 0x7f0b48153a50 2019-12-05T06:21:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.948+0000 7f0b46d9d700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=THROTTLE_DONE pgs=463 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:55.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.948+0000 7f0b3f7fe700 1 -- 172.21.15.16:0/3640950477 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 81 v81) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f0b38009580 con 0x7f0b48153a50 2019-12-05T06:21:55.937 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 81 2019-12-05T06:21:55.937 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:55.937 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":81,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:47.975651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":18,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:55.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.952+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b20023670 msgr2=0x7f0b20025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:55.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.952+0000 7f0b4c8dc700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b20023670 0x7f0b20025ae0 crc :-1 s=READY pgs=284 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:55.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.952+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 msgr2=0x7f0b48153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:55.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.952+0000 7f0b4c8dc700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 0x7f0b48153e40 secure :-1 s=READY pgs=463 cs=0 l=1 rx=0x7f0b380103a0 tx=0x7f0b3800b2e0).stop 2019-12-05T06:21:55.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.952+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/3640950477 shutdown_connections 2019-12-05T06:21:55.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.952+0000 7f0b4c8dc700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b20023670 0x7f0b20025ae0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:55.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.952+0000 7f0b4c8dc700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b48129a00 0x7f0b48152770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:55.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.952+0000 7f0b4c8dc700 1 --2- 172.21.15.16:0/3640950477 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b48153a50 0x7f0b48153e40 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:55.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.952+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/3640950477 shutdown_connections 2019-12-05T06:21:55.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.953+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/3640950477 wait complete. 2019-12-05T06:21:55.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:55.953+0000 7f0b4c8dc700 1 -- 172.21.15.16:0/3640950477 >> 172.21.15.16:0/3640950477 conn(0x7f0b4806d260 msgr2=0x7f0b4809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:55.953 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:21:57.954 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:21:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.117+0000 7faf11d7b700 1 Processor -- start 2019-12-05T06:21:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.117+0000 7faf11d7b700 1 -- start start 2019-12-05T06:21:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.118+0000 7faf11d7b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf0c08c6d0 0x7faf0c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:58.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.118+0000 7faf11d7b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c08fef0 0x7faf0c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:21:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.118+0000 7faf11d7b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0c0958e0 con 0x7faf0c08ef20 2019-12-05T06:21:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.118+0000 7faf11d7b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf0c08df90 con 0x7faf0c08c6d0 2019-12-05T06:21:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.118+0000 7faf11d7b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c0a3f70 con 0x7faf0c08fef0 2019-12-05T06:21:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.118+0000 7faf0bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf0c08c6d0 0x7faf0c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.118+0000 7faf0bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf0c08c6d0 0x7faf0c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51594/0 (socket says 172.21.15.16:51594) 2019-12-05T06:21:58.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.118+0000 7faf0bfff700 1 -- 172.21.15.16:0/2237916751 learned_addr learned my addr 172.21.15.16:0/2237916751 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.118+0000 7faf037fe700 1 --2- 172.21.15.16:0/2237916751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c08fef0 0x7faf0c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0bfff700 1 -- 172.21.15.16:0/2237916751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c08fef0 msgr2=0x7faf0c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:21:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0bfff700 1 --2- 172.21.15.16:0/2237916751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c08fef0 0x7faf0c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0bfff700 1 -- 172.21.15.16:0/2237916751 >> v1:172.21.15.76:6789/0 conn(0x7faf0c08ef20 legacy=0x7faf0c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0bfff700 1 -- 172.21.15.16:0/2237916751 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c08d540 con 0x7faf0c08c6d0 2019-12-05T06:21:58.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0bfff700 1 --2- 172.21.15.16:0/2237916751 >> v2:172.21.15.16:3300/0 conn(0x7faf0c08c6d0 0x7faf0c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faefc00f6c0 tx=0x7faefc00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0bfff700 1 --2- 172.21.15.16:0/2237916751 >> v2:172.21.15.16:3300/0 conn(0x7faf0c08c6d0 0x7faf0c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faefc00f6c0 tx=0x7faefc00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0bfff700 1 --2- 172.21.15.16:0/2237916751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c08c6d0 0x7faf0c08cac0 secure :-1 s=READY pgs=636 cs=0 l=1 rx=0x7faefc00f6c0 tx=0x7faefc00fa60).ready entity=mon.0 client_cookie=326e7437196673ed server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:58.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0bfff700 1 --2- 172.21.15.16:0/2237916751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c08c6d0 0x7faf0c08cac0 secure :-1 s=THROTTLE_DONE pgs=636 cs=0 l=1 rx=0x7faefc00f6c0 tx=0x7faefc00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0bfff700 1 --2- 172.21.15.16:0/2237916751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c08c6d0 0x7faf0c08cac0 secure :-1 s=THROTTLE_DONE pgs=636 cs=0 l=1 rx=0x7faefc00f6c0 tx=0x7faefc00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0b7fe700 1 -- 172.21.15.16:0/2237916751 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faefc000960 con 0x7faf0c08c6d0 2019-12-05T06:21:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0bfff700 1 --2- 172.21.15.16:0/2237916751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c08c6d0 0x7faf0c08cac0 secure :-1 s=THROTTLE_DONE pgs=636 cs=0 l=1 rx=0x7faefc00f6c0 tx=0x7faefc00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0b7fe700 1 -- 172.21.15.16:0/2237916751 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faefc000c90 con 0x7faf0c08c6d0 2019-12-05T06:21:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf0b7fe700 1 -- 172.21.15.16:0/2237916751 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faefc0013a0 con 0x7faf0c08c6d0 2019-12-05T06:21:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf11d7b700 1 -- 172.21.15.16:0/2237916751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c08c6d0 msgr2=0x7faf0c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.119+0000 7faf11d7b700 1 --2- 172.21.15.16:0/2237916751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c08c6d0 0x7faf0c08cac0 secure :-1 s=READY pgs=636 cs=0 l=1 rx=0x7faefc00f6c0 tx=0x7faefc00fa60).stop 2019-12-05T06:21:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.120+0000 7faf11d7b700 1 -- 172.21.15.16:0/2237916751 shutdown_connections 2019-12-05T06:21:58.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.120+0000 7faf11d7b700 1 --2- 172.21.15.16:0/2237916751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c08fef0 0x7faf0c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:21:58.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.120+0000 7faf11d7b700 1 --2- 172.21.15.16:0/2237916751 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf0c08c6d0 0x7faf0c08cac0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:58.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.120+0000 7faf11d7b700 1 -- 172.21.15.16:0/2237916751 shutdown_connections 2019-12-05T06:21:58.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.120+0000 7faf11d7b700 1 -- 172.21.15.16:0/2237916751 wait complete. 2019-12-05T06:21:58.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.120+0000 7faf11d7b700 1 -- 172.21.15.16:0/2237916751 >> 172.21.15.16:0/2237916751 conn(0x7faf0c061940 msgr2=0x7faf0c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:58.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.122+0000 7faf11d7b700 1 Processor -- start 2019-12-05T06:21:58.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.122+0000 7faf11d7b700 1 -- start start 2019-12-05T06:21:58.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.123+0000 7faf11d7b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:58.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.123+0000 7faf11d7b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7faf0c084f30 0x7faf0c09b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:58.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.123+0000 7faf11d7b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0c084b90 con 0x7faf0c097280 2019-12-05T06:21:58.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.123+0000 7faf11d7b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf0c099a20 con 0x7faf0c084f30 2019-12-05T06:21:58.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.123+0000 7faf11d7b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf0c099cf0 con 0x7faf0c096e90 2019-12-05T06:21:58.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.124+0000 7faf0bfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:58.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.124+0000 7faf037fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7faf0c084f30 0x7faf0c09b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:58.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.124+0000 7faf0bfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:55590/0 (socket says 172.21.15.16:55590) 2019-12-05T06:21:58.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.124+0000 7faf0bfff700 1 -- 172.21.15.16:0/849930433 learned_addr learned my addr 172.21.15.16:0/849930433 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:21:58.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.124+0000 7faf0bfff700 1 -- 172.21.15.16:0/849930433 >> v2:172.21.15.76:3300/0 conn(0x7faf0c084f30 msgr2=0x7faf0c09b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:58.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.124+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> v2:172.21.15.76:3300/0 conn(0x7faf0c084f30 0x7faf0c09b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:58.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.124+0000 7faf0bfff700 1 -- 172.21.15.16:0/849930433 >> v1:172.21.15.76:6789/0 conn(0x7faf0c097280 legacy=0x7faf0c097700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:58.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.124+0000 7faf0bfff700 1 -- 172.21.15.16:0/849930433 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf0c09aae0 con 0x7faf0c096e90 2019-12-05T06:21:58.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.124+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.124+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.125+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=READY pgs=159 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:58.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.125+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.125+0000 7faf097fa700 1 -- 172.21.15.16:0/849930433 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faefc005370 con 0x7faf0c096e90 2019-12-05T06:21:58.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.125+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.125+0000 7faf097fa700 1 -- 172.21.15.16:0/849930433 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faefc007a90 con 0x7faf0c096e90 2019-12-05T06:21:58.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.125+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.125+0000 7faf097fa700 1 -- 172.21.15.16:0/849930433 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faefc005560 con 0x7faf0c096e90 2019-12-05T06:21:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.125+0000 7faf11d7b700 1 -- 172.21.15.16:0/849930433 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf0c09aae0 con 0x7faf0c096e90 2019-12-05T06:21:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.125+0000 7faf11d7b700 1 -- 172.21.15.16:0/849930433 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf0c09aae0 con 0x7faf0c096e90 2019-12-05T06:21:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.126+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.126+0000 7faf097fa700 1 -- 172.21.15.16:0/849930433 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faefc005180 con 0x7faf0c096e90 2019-12-05T06:21:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.126+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.126+0000 7faf097fa700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faef0023630 0x7faef0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:21:58.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.127+0000 7faf097fa700 1 -- 172.21.15.16:0/849930433 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7faefc045dd0 con 0x7faf0c096e90 2019-12-05T06:21:58.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.127+0000 7faf037fe700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faef0023630 0x7faef0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:21:58.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.127+0000 7faf037fe700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faef0023630 0x7faef0025ac0 crc :-1 s=READY pgs=286 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:21:58.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.132+0000 7faf11d7b700 1 -- 172.21.15.16:0/849930433 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf0c09aae0 con 0x7faf0c096e90 2019-12-05T06:21:58.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.141+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.141+0000 7faf097fa700 1 -- 172.21.15.16:0/849930433 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faefc0072c0 con 0x7faf0c096e90 2019-12-05T06:21:58.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.349+0000 7faf11d7b700 1 -- 172.21.15.16:0/849930433 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf0c09aae0 con 0x7faf0c096e90 2019-12-05T06:21:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.350+0000 7faf0bfff700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=159 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:21:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.350+0000 7faf097fa700 1 -- 172.21.15.16:0/849930433 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2105 (secure 0 0 0) 0x7faefc005180 con 0x7faf0c096e90 2019-12-05T06:21:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 82 2019-12-05T06:21:58.338 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:21:58.338 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5878,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:47.975651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":18,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:21:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.352+0000 7faf11d7b700 1 -- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faef0023630 msgr2=0x7faef0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.352+0000 7faf11d7b700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faef0023630 0x7faef0025ac0 crc :-1 s=READY pgs=286 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.352+0000 7faf11d7b700 1 -- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 msgr2=0x7faf0c09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:21:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.352+0000 7faf11d7b700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 secure :-1 s=READY pgs=159 cs=0 l=1 rx=0x7faefc010440 tx=0x7faefc0102f0).stop 2019-12-05T06:21:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.352+0000 7faf11d7b700 1 -- 172.21.15.16:0/849930433 shutdown_connections 2019-12-05T06:21:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.352+0000 7faf11d7b700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faef0023630 0x7faef0025ac0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.352+0000 7faf11d7b700 1 --2- 172.21.15.16:0/849930433 >> v2:172.21.15.76:3300/0 conn(0x7faf0c084f30 0x7faf0c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.352+0000 7faf11d7b700 1 --2- 172.21.15.16:0/849930433 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf0c096e90 0x7faf0c09b3e0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:21:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.353+0000 7faf11d7b700 1 -- 172.21.15.16:0/849930433 shutdown_connections 2019-12-05T06:21:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.353+0000 7faf11d7b700 1 -- 172.21.15.16:0/849930433 wait complete. 2019-12-05T06:21:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:21:58.353+0000 7faf11d7b700 1 -- 172.21.15.16:0/849930433 >> 172.21.15.16:0/849930433 conn(0x7faf0c0714f0 msgr2=0x7faf0c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:21:58.351 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:21:58.351 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 71 secs before thrashing 2019-12-05T06:22:02.562 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:22:02.569 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:22:32.758 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:22:32.763 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:23:02.905 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:23:02.909 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:23:09.352 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:09.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.577+0000 7fafafef4700 1 Processor -- start 2019-12-05T06:23:09.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.577+0000 7fafafef4700 1 -- start start 2019-12-05T06:23:09.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafafef4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 0x7fafa808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafafef4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80958e0 con 0x7fafa808c6d0 2019-12-05T06:23:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafafef4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa80629e0 con 0x7fafa808fef0 2019-12-05T06:23:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafafef4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafa808df90 con 0x7fafa808d2c0 2019-12-05T06:23:09.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafae491700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fafa808c6d0 0x7fafa808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37068/0 (socket says 172.21.15.16:37068) 2019-12-05T06:23:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafae491700 1 -- 172.21.15.16:0/2607606822 learned_addr learned my addr 172.21.15.16:0/2607606822 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafadc90700 1 --2- 172.21.15.16:0/2607606822 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 0x7fafa808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafad48f700 1 -- 172.21.15.16:0/2607606822 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 300442670 0 0) 0x7fafa4001150 con 0x7fafa808c6d0 2019-12-05T06:23:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafad48f700 1 -- 172.21.15.16:0/2607606822 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf94002e80 con 0x7fafa808c6d0 2019-12-05T06:23:09.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafad48f700 1 -- 172.21.15.16:0/2607606822 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3664999229 0 0) 0x7fafa0001400 con 0x7fafa808fef0 2019-12-05T06:23:09.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafad48f700 1 -- 172.21.15.16:0/2607606822 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf940030b0 con 0x7fafa808fef0 2019-12-05T06:23:09.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafad48f700 1 -- 172.21.15.16:0/2607606822 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 329188695 0 0) 0x7fafa4001130 con 0x7fafa808c6d0 2019-12-05T06:23:09.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.579+0000 7fafadc90700 1 -- 172.21.15.16:0/2607606822 >> v1:172.21.15.76:6789/0 conn(0x7fafa808fef0 legacy=0x7fafa808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafadc90700 1 -- 172.21.15.16:0/2607606822 >> v1:172.21.15.16:6789/0 conn(0x7fafa808c6d0 legacy=0x7fafa808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafadc90700 1 -- 172.21.15.16:0/2607606822 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa808e620 con 0x7fafa808d2c0 2019-12-05T06:23:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafadc90700 1 --2- 172.21.15.16:0/2607606822 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 0x7fafa808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faf9800af50 tx=0x7faf9800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafadc90700 1 --2- 172.21.15.16:0/2607606822 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 0x7fafa808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faf9800af50 tx=0x7faf9800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafadc90700 1 --2- 172.21.15.16:0/2607606822 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 0x7fafa808ef20 secure :-1 s=READY pgs=162 cs=0 l=1 rx=0x7faf9800af50 tx=0x7faf9800b760).ready entity=mon.2 client_cookie=954b311c21dfeac6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafadc90700 1 --2- 172.21.15.16:0/2607606822 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 0x7fafa808ef20 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7faf9800af50 tx=0x7faf9800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafad48f700 1 -- 172.21.15.16:0/2607606822 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf98000960 con 0x7fafa808d2c0 2019-12-05T06:23:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafadc90700 1 --2- 172.21.15.16:0/2607606822 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 0x7fafa808ef20 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7faf9800af50 tx=0x7faf9800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafadc90700 1 --2- 172.21.15.16:0/2607606822 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 0x7fafa808ef20 secure :-1 s=THROTTLE_DONE pgs=162 cs=0 l=1 rx=0x7faf9800af50 tx=0x7faf9800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafad48f700 1 -- 172.21.15.16:0/2607606822 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf98000960 con 0x7fafa808d2c0 2019-12-05T06:23:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafad48f700 1 -- 172.21.15.16:0/2607606822 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf98009ce0 con 0x7fafa808d2c0 2019-12-05T06:23:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafafef4700 1 -- 172.21.15.16:0/2607606822 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 msgr2=0x7fafa808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.580+0000 7fafafef4700 1 --2- 172.21.15.16:0/2607606822 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 0x7fafa808ef20 secure :-1 s=READY pgs=162 cs=0 l=1 rx=0x7faf9800af50 tx=0x7faf9800b760).stop 2019-12-05T06:23:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.581+0000 7fafafef4700 1 -- 172.21.15.16:0/2607606822 shutdown_connections 2019-12-05T06:23:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.581+0000 7fafafef4700 1 --2- 172.21.15.16:0/2607606822 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fafa808d2c0 0x7fafa808ef20 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.581+0000 7fafafef4700 1 -- 172.21.15.16:0/2607606822 shutdown_connections 2019-12-05T06:23:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.581+0000 7fafafef4700 1 -- 172.21.15.16:0/2607606822 wait complete. 2019-12-05T06:23:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.581+0000 7fafafef4700 1 -- 172.21.15.16:0/2607606822 >> 172.21.15.16:0/2607606822 conn(0x7fafa8061940 msgr2=0x7fafa8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.584+0000 7fafafef4700 1 Processor -- start 2019-12-05T06:23:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.584+0000 7fafafef4700 1 -- start start 2019-12-05T06:23:09.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafafef4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fafa809e7d0 0x7fafa809e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafafef4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fafa8156e70 0x7fafa8159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafafef4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa812cb90 con 0x7fafa81302f0 2019-12-05T06:23:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafafef4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa8131ff0 con 0x7fafa809e7d0 2019-12-05T06:23:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafafef4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa81315c0 con 0x7fafa8156e70 2019-12-05T06:23:09.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafadc90700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fafa809e7d0 0x7fafa809e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:09.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafadc90700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fafa809e7d0 0x7fafa809e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51910/0 (socket says 172.21.15.16:51910) 2019-12-05T06:23:09.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafadc90700 1 -- 172.21.15.16:0/2910537773 learned_addr learned my addr 172.21.15.16:0/2910537773 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:09.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafaec92700 1 --2- 172.21.15.16:0/2910537773 >> v2:172.21.15.76:3300/0 conn(0x7fafa8156e70 0x7fafa8159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:09.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafadc90700 1 -- 172.21.15.16:0/2910537773 >> v2:172.21.15.76:3300/0 conn(0x7fafa8156e70 msgr2=0x7fafa8159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> v2:172.21.15.76:3300/0 conn(0x7fafa8156e70 0x7fafa8159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafadc90700 1 -- 172.21.15.16:0/2910537773 >> v1:172.21.15.76:6789/0 conn(0x7fafa81302f0 legacy=0x7fafa809d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafadc90700 1 -- 172.21.15.16:0/2910537773 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa8132680 con 0x7fafa809e7d0 2019-12-05T06:23:09.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> v2:172.21.15.16:3300/0 conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> v2:172.21.15.16:3300/0 conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.585+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=READY pgs=642 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.586+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=THROTTLE_DONE pgs=642 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.586+0000 7faf9effd700 1 -- 172.21.15.16:0/2910537773 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf98000940 con 0x7fafa809e7d0 2019-12-05T06:23:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.586+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=THROTTLE_DONE pgs=642 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.586+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=THROTTLE_DONE pgs=642 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.586+0000 7faf9effd700 1 -- 172.21.15.16:0/2910537773 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf98009980 con 0x7fafa809e7d0 2019-12-05T06:23:09.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.586+0000 7faf9effd700 1 -- 172.21.15.16:0/2910537773 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf9800a050 con 0x7fafa809e7d0 2019-12-05T06:23:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.586+0000 7fafafef4700 1 -- 172.21.15.16:0/2910537773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa8132680 con 0x7fafa809e7d0 2019-12-05T06:23:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.586+0000 7fafafef4700 1 -- 172.21.15.16:0/2910537773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa8132680 con 0x7fafa809e7d0 2019-12-05T06:23:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.586+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=THROTTLE_DONE pgs=642 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.587+0000 7faf9effd700 1 -- 172.21.15.16:0/2910537773 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faf98009b70 con 0x7fafa809e7d0 2019-12-05T06:23:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.587+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=THROTTLE_DONE pgs=642 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.587+0000 7faf9effd700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf90023630 0x7faf90025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.587+0000 7faf9effd700 1 -- 172.21.15.16:0/2910537773 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7faf98045e00 con 0x7fafa809e7d0 2019-12-05T06:23:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.587+0000 7fafae491700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf90023630 0x7faf90025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:09.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.588+0000 7fafae491700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf90023630 0x7faf90025ac0 crc :-1 s=READY pgs=289 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:09.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.593+0000 7fafafef4700 1 -- 172.21.15.16:0/2910537773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa8132680 con 0x7fafa809e7d0 2019-12-05T06:23:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.597+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=THROTTLE_DONE pgs=642 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.597+0000 7faf9effd700 1 -- 172.21.15.16:0/2910537773 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faf98009580 con 0x7fafa809e7d0 2019-12-05T06:23:09.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.796+0000 7fafafef4700 1 -- 172.21.15.16:0/2910537773 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa8132680 con 0x7fafa809e7d0 2019-12-05T06:23:09.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.797+0000 7fafadc90700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=THROTTLE_DONE pgs=642 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:09.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.797+0000 7faf9effd700 1 -- 172.21.15.16:0/2910537773 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2105 (secure 0 0 0) 0x7faf98009b70 con 0x7fafa809e7d0 2019-12-05T06:23:09.785 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 82 2019-12-05T06:23:09.785 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:09.785 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5878,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:47.975651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":18,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:09.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 -- 172.21.15.16:0/2910537773 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf90023630 msgr2=0x7faf90025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:09.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf90023630 0x7faf90025ac0 crc :-1 s=READY pgs=289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:09.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 -- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 msgr2=0x7fafa809e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:09.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 0x7fafa809e000 secure :-1 s=READY pgs=642 cs=0 l=1 rx=0x7faf98010410 tx=0x7faf980102c0).stop 2019-12-05T06:23:09.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 -- 172.21.15.16:0/2910537773 shutdown_connections 2019-12-05T06:23:09.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 --2- 172.21.15.16:0/2910537773 >> v2:172.21.15.76:3300/0 conn(0x7fafa8156e70 0x7fafa8159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf90023630 0x7faf90025ac0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 --2- 172.21.15.16:0/2910537773 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa809e7d0 0x7fafa809e000 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 -- 172.21.15.16:0/2910537773 shutdown_connections 2019-12-05T06:23:09.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 -- 172.21.15.16:0/2910537773 wait complete. 2019-12-05T06:23:09.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:09.800+0000 7fafafef4700 1 -- 172.21.15.16:0/2910537773 >> 172.21.15.16:0/2910537773 conn(0x7fafa80a2850 msgr2=0x7fafa806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:09.797 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:23:09.797 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:23:09.798 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:23:09.805 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:23:09.805 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.112+0000 7f6653447700 1 Processor -- start 2019-12-05T06:23:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.113+0000 7f6653447700 1 -- start start 2019-12-05T06:23:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.114+0000 7f6653447700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f664c08d2c0 0x7f664c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:10.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.114+0000 7f6653447700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f664c08fef0 0x7f664c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.114+0000 7f6653447700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c0958e0 con 0x7f664c08c6d0 2019-12-05T06:23:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.114+0000 7f6653447700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664c08df90 con 0x7f664c08fef0 2019-12-05T06:23:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.114+0000 7f6653447700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664c0a3f70 con 0x7f664c08d2c0 2019-12-05T06:23:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.114+0000 7f66519e4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f664c08c6d0 0x7f664c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37086/0 (socket says 172.21.15.16:37086) 2019-12-05T06:23:10.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.114+0000 7f66519e4700 1 -- 172.21.15.16:0/3866246129 learned_addr learned my addr 172.21.15.16:0/3866246129 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.114+0000 7f66521e5700 1 --2- 172.21.15.16:0/3866246129 >> v2:172.21.15.16:3300/0 conn(0x7f664c08fef0 0x7f664c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.115+0000 7f66511e3700 1 --2- 172.21.15.16:0/3866246129 >> v2:172.21.15.76:3300/0 conn(0x7f664c08d2c0 0x7f664c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.115+0000 7f66509e2700 1 -- 172.21.15.16:0/3866246129 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229947017 0 0) 0x7f66440014f0 con 0x7f664c08c6d0 2019-12-05T06:23:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.115+0000 7f66509e2700 1 -- 172.21.15.16:0/3866246129 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6634002e80 con 0x7f664c08c6d0 2019-12-05T06:23:10.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.115+0000 7f66521e5700 1 -- 172.21.15.16:0/3866246129 >> v2:172.21.15.76:3300/0 conn(0x7f664c08d2c0 msgr2=0x7f664c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.115+0000 7f66521e5700 1 --2- 172.21.15.16:0/3866246129 >> v2:172.21.15.76:3300/0 conn(0x7f664c08d2c0 0x7f664c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.115+0000 7f66521e5700 1 -- 172.21.15.16:0/3866246129 >> v1:172.21.15.16:6789/0 conn(0x7f664c08c6d0 legacy=0x7f664c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.115+0000 7f66521e5700 1 -- 172.21.15.16:0/3866246129 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c08e620 con 0x7f664c08fef0 2019-12-05T06:23:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.115+0000 7f66511e3700 1 --2- 172.21.15.16:0/3866246129 >> v2:172.21.15.76:3300/0 conn(0x7f664c08d2c0 0x7f664c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:10.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.115+0000 7f66521e5700 1 --2- 172.21.15.16:0/3866246129 >> v2:172.21.15.16:3300/0 conn(0x7f664c08fef0 0x7f664c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f663c00b0d0 tx=0x7f663c00b880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.116+0000 7f66521e5700 1 --2- 172.21.15.16:0/3866246129 >> v2:172.21.15.16:3300/0 conn(0x7f664c08fef0 0x7f664c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f663c00b0d0 tx=0x7f663c00b880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.116+0000 7f66521e5700 1 --2- 172.21.15.16:0/3866246129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c08fef0 0x7f664c08f460 secure :-1 s=READY pgs=644 cs=0 l=1 rx=0x7f663c00b0d0 tx=0x7f663c00b880).ready entity=mon.0 client_cookie=ab7dc8f170326768 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.116+0000 7f66521e5700 1 --2- 172.21.15.16:0/3866246129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c08fef0 0x7f664c08f460 secure :-1 s=THROTTLE_DONE pgs=644 cs=0 l=1 rx=0x7f663c00b0d0 tx=0x7f663c00b880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.116+0000 7f66521e5700 1 --2- 172.21.15.16:0/3866246129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c08fef0 0x7f664c08f460 secure :-1 s=THROTTLE_DONE pgs=644 cs=0 l=1 rx=0x7f663c00b0d0 tx=0x7f663c00b880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.116+0000 7f66509e2700 1 -- 172.21.15.16:0/3866246129 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f663c0010e0 con 0x7f664c08fef0 2019-12-05T06:23:10.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.116+0000 7f66521e5700 1 --2- 172.21.15.16:0/3866246129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c08fef0 0x7f664c08f460 secure :-1 s=THROTTLE_DONE pgs=644 cs=0 l=1 rx=0x7f663c00b0d0 tx=0x7f663c00b880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.116+0000 7f66509e2700 1 -- 172.21.15.16:0/3866246129 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c000bc0 con 0x7f664c08fef0 2019-12-05T06:23:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.116+0000 7f66509e2700 1 -- 172.21.15.16:0/3866246129 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f663c001760 con 0x7f664c08fef0 2019-12-05T06:23:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.116+0000 7f6653447700 1 -- 172.21.15.16:0/3866246129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c08fef0 msgr2=0x7f664c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.116+0000 7f6653447700 1 --2- 172.21.15.16:0/3866246129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c08fef0 0x7f664c08f460 secure :-1 s=READY pgs=644 cs=0 l=1 rx=0x7f663c00b0d0 tx=0x7f663c00b880).stop 2019-12-05T06:23:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.117+0000 7f6653447700 1 -- 172.21.15.16:0/3866246129 shutdown_connections 2019-12-05T06:23:10.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.117+0000 7f6653447700 1 --2- 172.21.15.16:0/3866246129 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c08fef0 0x7f664c08f460 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.117+0000 7f6653447700 1 --2- 172.21.15.16:0/3866246129 >> v2:172.21.15.76:3300/0 conn(0x7f664c08d2c0 0x7f664c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.118+0000 7f6653447700 1 -- 172.21.15.16:0/3866246129 shutdown_connections 2019-12-05T06:23:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.118+0000 7f6653447700 1 -- 172.21.15.16:0/3866246129 wait complete. 2019-12-05T06:23:10.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.118+0000 7f6653447700 1 -- 172.21.15.16:0/3866246129 >> 172.21.15.16:0/3866246129 conn(0x7f664c061940 msgr2=0x7f664c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.123+0000 7f6653447700 1 Processor -- start 2019-12-05T06:23:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.123+0000 7f6653447700 1 -- start start 2019-12-05T06:23:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.125+0000 7f6653447700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f664c09bbe0 0x7f664c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.125+0000 7f6653447700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f664c084f30 0x7f664c155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:10.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.125+0000 7f6653447700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f664c084b90 con 0x7f664c096e90 2019-12-05T06:23:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.125+0000 7f6653447700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f664c09a450 con 0x7f664c084f30 2019-12-05T06:23:10.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.125+0000 7f6653447700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f664c099a20 con 0x7f664c09bbe0 2019-12-05T06:23:10.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.125+0000 7f66519e4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f664c096e90 0x7f664c09b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37092/0 (socket says 172.21.15.16:37092) 2019-12-05T06:23:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.125+0000 7f66519e4700 1 -- 172.21.15.16:0/1549471797 learned_addr learned my addr 172.21.15.16:0/1549471797 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.125+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> v2:172.21.15.16:3300/0 conn(0x7f664c084f30 0x7f664c155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.125+0000 7f66511e3700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f664c09bbe0 0x7f664c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f66521e5700 1 -- 172.21.15.16:0/1549471797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f664c09bbe0 msgr2=0x7f664c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:10.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f664c09bbe0 0x7f664c1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f66521e5700 1 -- 172.21.15.16:0/1549471797 >> v1:172.21.15.16:6789/0 conn(0x7f664c096e90 legacy=0x7f664c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f664a7fc700 1 -- 172.21.15.16:0/1549471797 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 791131645 0 0) 0x7f6644000cc0 con 0x7f664c096e90 2019-12-05T06:23:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f66521e5700 1 -- 172.21.15.16:0/1549471797 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c09aae0 con 0x7f664c084f30 2019-12-05T06:23:10.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> v2:172.21.15.16:3300/0 conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> v2:172.21.15.16:3300/0 conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=READY pgs=646 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f664a7fc700 1 -- 172.21.15.16:0/1549471797 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f663c000bc0 con 0x7f664c084f30 2019-12-05T06:23:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f664a7fc700 1 -- 172.21.15.16:0/1549471797 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f663c000bc0 con 0x7f664c084f30 2019-12-05T06:23:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f6653447700 1 -- 172.21.15.16:0/1549471797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f664c09aae0 con 0x7f664c084f30 2019-12-05T06:23:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.126+0000 7f664a7fc700 1 -- 172.21.15.16:0/1549471797 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f663c0012d0 con 0x7f664c084f30 2019-12-05T06:23:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.127+0000 7f6653447700 1 -- 172.21.15.16:0/1549471797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f664c09aae0 con 0x7f664c084f30 2019-12-05T06:23:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.127+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.129+0000 7f664a7fc700 1 -- 172.21.15.16:0/1549471797 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f663c000ef0 con 0x7f664c084f30 2019-12-05T06:23:10.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.129+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.129+0000 7f664a7fc700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f662c023630 0x7f662c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.129+0000 7f664a7fc700 1 -- 172.21.15.16:0/1549471797 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f663c045d70 con 0x7f664c084f30 2019-12-05T06:23:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.130+0000 7f66519e4700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f662c023630 0x7f662c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:10.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.130+0000 7f66519e4700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f662c023630 0x7f662c025ac0 crc :-1 s=READY pgs=290 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:10.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.143+0000 7f6653447700 1 -- 172.21.15.16:0/1549471797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f664c09aae0 con 0x7f664c084f30 2019-12-05T06:23:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.151+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.151+0000 7f664a7fc700 1 -- 172.21.15.16:0/1549471797 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f663c00d5b0 con 0x7f664c084f30 2019-12-05T06:23:10.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.398+0000 7f6653447700 1 -- 172.21.15.16:0/1549471797 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f664c09aae0 con 0x7f664c084f30 2019-12-05T06:23:10.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.399+0000 7f66521e5700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=THROTTLE_DONE pgs=646 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:10.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.399+0000 7f664a7fc700 1 -- 172.21.15.16:0/1549471797 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f663c043f50 con 0x7f664c084f30 2019-12-05T06:23:10.386 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 82 2019-12-05T06:23:10.386 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:10.386 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5878,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:47.975651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":18,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:10.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 -- 172.21.15.16:0/1549471797 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f662c023630 msgr2=0x7f662c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:10.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f662c023630 0x7f662c025ac0 crc :-1 s=READY pgs=290 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:10.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 -- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 msgr2=0x7f664c155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:10.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 0x7f664c155f30 secure :-1 s=READY pgs=646 cs=0 l=1 rx=0x7f663c0102f0 tx=0x7f663c010690).stop 2019-12-05T06:23:10.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 -- 172.21.15.16:0/1549471797 shutdown_connections 2019-12-05T06:23:10.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f662c023630 0x7f662c025ac0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:10.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f664c084f30 0x7f664c155f30 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:10.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 --2- 172.21.15.16:0/1549471797 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f664c09bbe0 0x7f664c1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:10.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 -- 172.21.15.16:0/1549471797 shutdown_connections 2019-12-05T06:23:10.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 -- 172.21.15.16:0/1549471797 wait complete. 2019-12-05T06:23:10.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:10.402+0000 7f6653447700 1 -- 172.21.15.16:0/1549471797 >> 172.21.15.16:0/1549471797 conn(0x7f664c0714f0 msgr2=0x7f664c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:10.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:23:12.401 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:12.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.678+0000 7fa78bd96700 1 Processor -- start 2019-12-05T06:23:12.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.678+0000 7fa78bd96700 1 -- start start 2019-12-05T06:23:12.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.680+0000 7fa78bd96700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa78408c6d0 0x7fa78408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:12.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.680+0000 7fa78bd96700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa78408fef0 0x7fa78408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:12.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.680+0000 7fa78bd96700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa78408fb80 con 0x7fa78408ef20 2019-12-05T06:23:12.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.680+0000 7fa78bd96700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78408df90 con 0x7fa78408c6d0 2019-12-05T06:23:12.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.680+0000 7fa78bd96700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7840a3f70 con 0x7fa78408fef0 2019-12-05T06:23:12.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.680+0000 7fa78a333700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa78408c6d0 0x7fa78408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:12.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.680+0000 7fa78a333700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa78408c6d0 0x7fa78408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51936/0 (socket says 172.21.15.16:51936) 2019-12-05T06:23:12.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.680+0000 7fa78ab34700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa78408ef20 0x7fa78408f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37106/0 (socket says 172.21.15.16:37106) 2019-12-05T06:23:12.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.680+0000 7fa789b32700 1 --2- 172.21.15.16:0/3790237898 >> v2:172.21.15.76:3300/0 conn(0x7fa78408fef0 0x7fa78408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:12.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.680+0000 7fa78a333700 1 -- 172.21.15.16:0/3790237898 learned_addr learned my addr 172.21.15.16:0/3790237898 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:12.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789b32700 1 -- 172.21.15.16:0/3790237898 >> v2:172.21.15.16:3300/0 conn(0x7fa78408c6d0 msgr2=0x7fa78408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:12.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789b32700 1 --2- 172.21.15.16:0/3790237898 >> v2:172.21.15.16:3300/0 conn(0x7fa78408c6d0 0x7fa78408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:12.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789b32700 1 -- 172.21.15.16:0/3790237898 >> v1:172.21.15.16:6789/0 conn(0x7fa78408ef20 legacy=0x7fa78408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:12.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789b32700 1 -- 172.21.15.16:0/3790237898 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa78408d540 con 0x7fa78408fef0 2019-12-05T06:23:12.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789b32700 1 --2- 172.21.15.16:0/3790237898 >> v2:172.21.15.76:3300/0 conn(0x7fa78408fef0 0x7fa78408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa77800af50 tx=0x7fa77800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789b32700 1 --2- 172.21.15.16:0/3790237898 >> v2:172.21.15.76:3300/0 conn(0x7fa78408fef0 0x7fa78408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa77800af50 tx=0x7fa77800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789b32700 1 --2- 172.21.15.16:0/3790237898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa78408fef0 0x7fa78408d000 secure :-1 s=READY pgs=474 cs=0 l=1 rx=0x7fa77800af50 tx=0x7fa77800b250).ready entity=mon.1 client_cookie=7405e05c8b83ef7e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:12.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789b32700 1 --2- 172.21.15.16:0/3790237898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa78408fef0 0x7fa78408d000 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7fa77800af50 tx=0x7fa77800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789b32700 1 --2- 172.21.15.16:0/3790237898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa78408fef0 0x7fa78408d000 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7fa77800af50 tx=0x7fa77800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789b32700 1 --2- 172.21.15.16:0/3790237898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa78408fef0 0x7fa78408d000 secure :-1 s=THROTTLE_DONE pgs=474 cs=0 l=1 rx=0x7fa77800af50 tx=0x7fa77800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.681+0000 7fa789331700 1 -- 172.21.15.16:0/3790237898 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa778007210 con 0x7fa78408fef0 2019-12-05T06:23:12.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.682+0000 7fa789331700 1 -- 172.21.15.16:0/3790237898 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7780099b0 con 0x7fa78408fef0 2019-12-05T06:23:12.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.682+0000 7fa789331700 1 -- 172.21.15.16:0/3790237898 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa77800a080 con 0x7fa78408fef0 2019-12-05T06:23:12.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.682+0000 7fa78bd96700 1 -- 172.21.15.16:0/3790237898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa78408fef0 msgr2=0x7fa78408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:12.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.682+0000 7fa78bd96700 1 --2- 172.21.15.16:0/3790237898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa78408fef0 0x7fa78408d000 secure :-1 s=READY pgs=474 cs=0 l=1 rx=0x7fa77800af50 tx=0x7fa77800b250).stop 2019-12-05T06:23:12.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.683+0000 7fa78bd96700 1 -- 172.21.15.16:0/3790237898 shutdown_connections 2019-12-05T06:23:12.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.683+0000 7fa78bd96700 1 --2- 172.21.15.16:0/3790237898 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa78408fef0 0x7fa78408d000 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:12.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.683+0000 7fa78bd96700 1 --2- 172.21.15.16:0/3790237898 >> v2:172.21.15.16:3300/0 conn(0x7fa78408c6d0 0x7fa78408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:12.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.684+0000 7fa78bd96700 1 -- 172.21.15.16:0/3790237898 shutdown_connections 2019-12-05T06:23:12.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.684+0000 7fa78bd96700 1 -- 172.21.15.16:0/3790237898 wait complete. 2019-12-05T06:23:12.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.684+0000 7fa78bd96700 1 -- 172.21.15.16:0/3790237898 >> 172.21.15.16:0/3790237898 conn(0x7fa784061940 msgr2=0x7fa784096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.688+0000 7fa78bd96700 1 Processor -- start 2019-12-05T06:23:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.688+0000 7fa78bd96700 1 -- start start 2019-12-05T06:23:12.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.689+0000 7fa78bd96700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa784128f50 0x7fa784152760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.689+0000 7fa78bd96700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa784137ea0 0x7fa784152ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.689+0000 7fa78bd96700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa784099720 con 0x7fa784128360 2019-12-05T06:23:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.689+0000 7fa78bd96700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa78412a640 con 0x7fa784128f50 2019-12-05T06:23:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.689+0000 7fa78bd96700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa784129c10 con 0x7fa784137ea0 2019-12-05T06:23:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.689+0000 7fa78a333700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa784128360 0x7fa784128750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37110/0 (socket says 172.21.15.16:37110) 2019-12-05T06:23:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.689+0000 7fa78a333700 1 -- 172.21.15.16:0/1675113818 learned_addr learned my addr 172.21.15.16:0/1675113818 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:12.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.689+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> v2:172.21.15.16:3300/0 conn(0x7fa784128f50 0x7fa784152760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:12.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.689+0000 7fa78ab34700 1 --2- 172.21.15.16:0/1675113818 >> v2:172.21.15.76:3300/0 conn(0x7fa784137ea0 0x7fa784152ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:12.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.690+0000 7fa789b32700 1 -- 172.21.15.16:0/1675113818 >> v2:172.21.15.76:3300/0 conn(0x7fa784137ea0 msgr2=0x7fa784152ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:12.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.690+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> v2:172.21.15.76:3300/0 conn(0x7fa784137ea0 0x7fa784152ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:12.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.690+0000 7fa789b32700 1 -- 172.21.15.16:0/1675113818 >> v1:172.21.15.16:6789/0 conn(0x7fa784128360 legacy=0x7fa784128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:12.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.690+0000 7fa789b32700 1 -- 172.21.15.16:0/1675113818 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7841531e0 con 0x7fa784128f50 2019-12-05T06:23:12.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.690+0000 7fa782ffd700 1 -- 172.21.15.16:0/1675113818 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1036596788 0 0) 0x7fa77c001430 con 0x7fa784128360 2019-12-05T06:23:12.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.690+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> v2:172.21.15.16:3300/0 conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.690+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> v2:172.21.15.16:3300/0 conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.690+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=READY pgs=649 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:12.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.690+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.691+0000 7fa782ffd700 1 -- 172.21.15.16:0/1675113818 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa778000940 con 0x7fa784128f50 2019-12-05T06:23:12.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.691+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.691+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.691+0000 7fa782ffd700 1 -- 172.21.15.16:0/1675113818 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa77800a9c0 con 0x7fa784128f50 2019-12-05T06:23:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.691+0000 7fa78bd96700 1 -- 172.21.15.16:0/1675113818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7841531e0 con 0x7fa784128f50 2019-12-05T06:23:12.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.691+0000 7fa782ffd700 1 -- 172.21.15.16:0/1675113818 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa778009e00 con 0x7fa784128f50 2019-12-05T06:23:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.691+0000 7fa78bd96700 1 -- 172.21.15.16:0/1675113818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7841531e0 con 0x7fa784128f50 2019-12-05T06:23:12.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.692+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.692+0000 7fa782ffd700 1 -- 172.21.15.16:0/1675113818 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa778009790 con 0x7fa784128f50 2019-12-05T06:23:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.693+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.693+0000 7fa782ffd700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa764023630 0x7fa764025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.693+0000 7fa782ffd700 1 -- 172.21.15.16:0/1675113818 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fa778045d50 con 0x7fa784128f50 2019-12-05T06:23:12.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.693+0000 7fa78a333700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa764023630 0x7fa764025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:12.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.694+0000 7fa78a333700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa764023630 0x7fa764025ac0 crc :-1 s=READY pgs=291 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:12.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.694+0000 7fa78bd96700 1 -- 172.21.15.16:0/1675113818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa78412acd0 con 0x7fa784128f50 2019-12-05T06:23:12.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.700+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.700+0000 7fa782ffd700 1 -- 172.21.15.16:0/1675113818 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa778009580 con 0x7fa784128f50 2019-12-05T06:23:12.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.935+0000 7fa78bd96700 1 -- 172.21.15.16:0/1675113818 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa78412acd0 con 0x7fa784128f50 2019-12-05T06:23:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.936+0000 7fa789b32700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.936+0000 7fa782ffd700 1 -- 172.21.15.16:0/1675113818 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fa778009790 con 0x7fa784128f50 2019-12-05T06:23:12.923 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 82 2019-12-05T06:23:12.923 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:12.923 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5878,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:47.975651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":18,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.938+0000 7fa78bd96700 1 -- 172.21.15.16:0/1675113818 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa764023630 msgr2=0x7fa764025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.938+0000 7fa78bd96700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa764023630 0x7fa764025ac0 crc :-1 s=READY pgs=291 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.938+0000 7fa78bd96700 1 -- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 msgr2=0x7fa784152760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:12.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.938+0000 7fa78bd96700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 0x7fa784152760 secure :-1 s=READY pgs=649 cs=0 l=1 rx=0x7fa778010340 tx=0x7fa7780106e0).stop 2019-12-05T06:23:12.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.939+0000 7fa78bd96700 1 -- 172.21.15.16:0/1675113818 shutdown_connections 2019-12-05T06:23:12.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.939+0000 7fa78bd96700 1 --2- 172.21.15.16:0/1675113818 >> v2:172.21.15.76:3300/0 conn(0x7fa784137ea0 0x7fa784152ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:12.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.939+0000 7fa78bd96700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa764023630 0x7fa764025ac0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:12.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.939+0000 7fa78bd96700 1 --2- 172.21.15.16:0/1675113818 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa784128f50 0x7fa784152760 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:12.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.939+0000 7fa78bd96700 1 -- 172.21.15.16:0/1675113818 shutdown_connections 2019-12-05T06:23:12.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.939+0000 7fa78bd96700 1 -- 172.21.15.16:0/1675113818 wait complete. 2019-12-05T06:23:12.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:12.939+0000 7fa78bd96700 1 -- 172.21.15.16:0/1675113818 >> 172.21.15.16:0/1675113818 conn(0x7fa78406d260 msgr2=0x7fa78409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:12.937 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:23:14.938 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.204+0000 7f722e23f700 1 Processor -- start 2019-12-05T06:23:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.204+0000 7f722e23f700 1 -- start start 2019-12-05T06:23:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.205+0000 7f722e23f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f722808d2c0 0x7f722808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:15.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.205+0000 7f722e23f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 0x7f722808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.205+0000 7f722e23f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72280958e0 con 0x7f722808c6d0 2019-12-05T06:23:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.205+0000 7f722e23f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722808df90 con 0x7f722808d2c0 2019-12-05T06:23:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.205+0000 7f722e23f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72280a3f70 con 0x7f722808fef0 2019-12-05T06:23:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.205+0000 7f7227fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f722808c6d0 0x7f722808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37120/0 (socket says 172.21.15.16:37120) 2019-12-05T06:23:15.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.205+0000 7f7227fff700 1 -- 172.21.15.16:0/3400031221 learned_addr learned my addr 172.21.15.16:0/3400031221 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:15.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.205+0000 7f722cfdd700 1 --2- 172.21.15.16:0/3400031221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 0x7f722808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:15.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.205+0000 7f72277fe700 1 --2- 172.21.15.16:0/3400031221 >> v2:172.21.15.76:3300/0 conn(0x7f722808d2c0 0x7f722808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:15.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.206+0000 7f7226ffd700 1 -- 172.21.15.16:0/3400031221 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3450083526 0 0) 0x7f7218001150 con 0x7f722808c6d0 2019-12-05T06:23:15.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.206+0000 7f7226ffd700 1 -- 172.21.15.16:0/3400031221 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7210002e80 con 0x7f722808c6d0 2019-12-05T06:23:15.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.206+0000 7f722cfdd700 1 -- 172.21.15.16:0/3400031221 >> v2:172.21.15.76:3300/0 conn(0x7f722808d2c0 msgr2=0x7f722808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.206+0000 7f722cfdd700 1 --2- 172.21.15.16:0/3400031221 >> v2:172.21.15.76:3300/0 conn(0x7f722808d2c0 0x7f722808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.206+0000 7f722cfdd700 1 -- 172.21.15.16:0/3400031221 >> v1:172.21.15.16:6789/0 conn(0x7f722808c6d0 legacy=0x7f722808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.206+0000 7f722cfdd700 1 -- 172.21.15.16:0/3400031221 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722808e620 con 0x7f722808fef0 2019-12-05T06:23:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.206+0000 7f72277fe700 1 --2- 172.21.15.16:0/3400031221 >> v2:172.21.15.76:3300/0 conn(0x7f722808d2c0 0x7f722808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.206+0000 7f722cfdd700 1 --2- 172.21.15.16:0/3400031221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 0x7f722808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f722000af80 tx=0x7f722000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.206+0000 7f722cfdd700 1 --2- 172.21.15.16:0/3400031221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 0x7f722808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f722000af80 tx=0x7f722000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.206+0000 7f722cfdd700 1 --2- 172.21.15.16:0/3400031221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 0x7f722808f460 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7f722000af80 tx=0x7f722000b790).ready entity=mon.2 client_cookie=dc69a74cc72a35d6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.207+0000 7f722cfdd700 1 --2- 172.21.15.16:0/3400031221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 0x7f722808f460 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f722000af80 tx=0x7f722000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.207+0000 7f7226ffd700 1 -- 172.21.15.16:0/3400031221 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7220000bc0 con 0x7f722808fef0 2019-12-05T06:23:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.207+0000 7f722cfdd700 1 --2- 172.21.15.16:0/3400031221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 0x7f722808f460 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f722000af80 tx=0x7f722000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.207+0000 7f722cfdd700 1 --2- 172.21.15.16:0/3400031221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 0x7f722808f460 secure :-1 s=THROTTLE_DONE pgs=163 cs=0 l=1 rx=0x7f722000af80 tx=0x7f722000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.207+0000 7f7226ffd700 1 -- 172.21.15.16:0/3400031221 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72200097f0 con 0x7f722808fef0 2019-12-05T06:23:15.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.207+0000 7f7226ffd700 1 -- 172.21.15.16:0/3400031221 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7220009ec0 con 0x7f722808fef0 2019-12-05T06:23:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.207+0000 7f722e23f700 1 -- 172.21.15.16:0/3400031221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 msgr2=0x7f722808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.207+0000 7f722e23f700 1 --2- 172.21.15.16:0/3400031221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 0x7f722808f460 secure :-1 s=READY pgs=163 cs=0 l=1 rx=0x7f722000af80 tx=0x7f722000b790).stop 2019-12-05T06:23:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.209+0000 7f722e23f700 1 -- 172.21.15.16:0/3400031221 shutdown_connections 2019-12-05T06:23:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.209+0000 7f722e23f700 1 --2- 172.21.15.16:0/3400031221 >> v2:172.21.15.76:3300/0 conn(0x7f722808d2c0 0x7f722808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.209+0000 7f722e23f700 1 --2- 172.21.15.16:0/3400031221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722808fef0 0x7f722808f460 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:15.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.209+0000 7f722e23f700 1 -- 172.21.15.16:0/3400031221 shutdown_connections 2019-12-05T06:23:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.209+0000 7f722e23f700 1 -- 172.21.15.16:0/3400031221 wait complete. 2019-12-05T06:23:15.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.209+0000 7f722e23f700 1 -- 172.21.15.16:0/3400031221 >> 172.21.15.16:0/3400031221 conn(0x7f7228061940 msgr2=0x7f7228096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:15.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.213+0000 7f722e23f700 1 Processor -- start 2019-12-05T06:23:15.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.213+0000 7f722e23f700 1 -- start start 2019-12-05T06:23:15.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.214+0000 7f722e23f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72280829b0 0x7f722809ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:15.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.214+0000 7f722e23f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722815b0e0 0x7f722815d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:15.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.214+0000 7f722e23f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7228132730 con 0x7f722809fc30 2019-12-05T06:23:15.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.214+0000 7f722e23f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f722809dcd0 con 0x7f72280829b0 2019-12-05T06:23:15.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.214+0000 7f722e23f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f722809d570 con 0x7f722815b0e0 2019-12-05T06:23:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.215+0000 7f7227fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72280829b0 0x7f722809ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.215+0000 7f7227fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f72280829b0 0x7f722809ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51960/0 (socket says 172.21.15.16:51960) 2019-12-05T06:23:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.215+0000 7f7227fff700 1 -- 172.21.15.16:0/1895792975 learned_addr learned my addr 172.21.15.16:0/1895792975 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:15.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.215+0000 7f722cfdd700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722815b0e0 0x7f722815d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:15.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.215+0000 7f7224ff9700 1 -- 172.21.15.16:0/1895792975 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654398100 0 0) 0x7f721c00df00 con 0x7f722809fc30 2019-12-05T06:23:15.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.215+0000 7f7224ff9700 1 -- 172.21.15.16:0/1895792975 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7208002e80 con 0x7f722809fc30 2019-12-05T06:23:15.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.215+0000 7f7227fff700 1 -- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722815b0e0 msgr2=0x7f722815d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:15.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722815b0e0 0x7f722815d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7227fff700 1 -- 172.21.15.16:0/1895792975 >> v1:172.21.15.76:6789/0 conn(0x7f722809fc30 legacy=0x7f722809f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7227fff700 1 -- 172.21.15.16:0/1895792975 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f722809e360 con 0x7f72280829b0 2019-12-05T06:23:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f722cfdd700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722815b0e0 0x7f722815d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> v2:172.21.15.16:3300/0 conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> v2:172.21.15.16:3300/0 conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=READY pgs=651 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:15.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7224ff9700 1 -- 172.21.15.16:0/1895792975 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7218000c90 con 0x7f72280829b0 2019-12-05T06:23:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7224ff9700 1 -- 172.21.15.16:0/1895792975 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72180010c0 con 0x7f72280829b0 2019-12-05T06:23:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f722e23f700 1 -- 172.21.15.16:0/1895792975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f722809e360 con 0x7f72280829b0 2019-12-05T06:23:15.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f7224ff9700 1 -- 172.21.15.16:0/1895792975 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72180016d0 con 0x7f72280829b0 2019-12-05T06:23:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.216+0000 7f722e23f700 1 -- 172.21.15.16:0/1895792975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f722809e360 con 0x7f72280829b0 2019-12-05T06:23:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.217+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.218+0000 7f7224ff9700 1 -- 172.21.15.16:0/1895792975 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7218000c90 con 0x7f72280829b0 2019-12-05T06:23:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.218+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.218+0000 7f722e23f700 1 -- 172.21.15.16:0/1895792975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f722809e360 con 0x7f72280829b0 2019-12-05T06:23:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.219+0000 7f7224ff9700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7208023670 0x7f7208025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:15.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.219+0000 7f7224ff9700 1 -- 172.21.15.16:0/1895792975 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f7218045df0 con 0x7f72280829b0 2019-12-05T06:23:15.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.219+0000 7f72277fe700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7208023670 0x7f7208025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:15.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.220+0000 7f72277fe700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7208023670 0x7f7208025ae0 crc :-1 s=READY pgs=292 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:15.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.223+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.223+0000 7f7224ff9700 1 -- 172.21.15.16:0/1895792975 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f72180075d0 con 0x7f72280829b0 2019-12-05T06:23:15.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.421+0000 7f722e23f700 1 -- 172.21.15.16:0/1895792975 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f722809e360 con 0x7f72280829b0 2019-12-05T06:23:15.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.421+0000 7f7227fff700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:15.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.421+0000 7f7224ff9700 1 -- 172.21.15.16:0/1895792975 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f7218000c90 con 0x7f72280829b0 2019-12-05T06:23:15.409 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 82 2019-12-05T06:23:15.409 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:15.409 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5878,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:47.975651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":18,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:15.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 -- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7208023670 msgr2=0x7f7208025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:15.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7208023670 0x7f7208025ae0 crc :-1 s=READY pgs=292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:15.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 -- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 msgr2=0x7f722809ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:15.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 0x7f722809ec60 secure :-1 s=READY pgs=651 cs=0 l=1 rx=0x7f72180103e0 tx=0x7f7218010780).stop 2019-12-05T06:23:15.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 -- 172.21.15.16:0/1895792975 shutdown_connections 2019-12-05T06:23:15.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f722815b0e0 0x7f722815d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:15.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7208023670 0x7f7208025ae0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:15.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 --2- 172.21.15.16:0/1895792975 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f72280829b0 0x7f722809ec60 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:15.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 -- 172.21.15.16:0/1895792975 shutdown_connections 2019-12-05T06:23:15.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 -- 172.21.15.16:0/1895792975 wait complete. 2019-12-05T06:23:15.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:15.424+0000 7f722e23f700 1 -- 172.21.15.16:0/1895792975 >> 172.21.15.16:0/1895792975 conn(0x7f72280719f0 msgr2=0x7f722812e690 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:15.423 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:23:17.424 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:17.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.713+0000 7f16f329e700 1 Processor -- start 2019-12-05T06:23:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.714+0000 7f16f329e700 1 -- start start 2019-12-05T06:23:17.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.716+0000 7f16f329e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec08d2c0 0x7f16ec08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.716+0000 7f16f329e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f16ec08fef0 0x7f16ec08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.716+0000 7f16f329e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec0958e0 con 0x7f16ec08c6d0 2019-12-05T06:23:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.716+0000 7f16f329e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ec08df90 con 0x7f16ec08fef0 2019-12-05T06:23:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.716+0000 7f16f329e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec0a3f70 con 0x7f16ec08d2c0 2019-12-05T06:23:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.716+0000 7f16f203c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f16ec08fef0 0x7f16ec08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:17.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.716+0000 7f16f103a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec08d2c0 0x7f16ec08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.716+0000 7f16f203c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f16ec08fef0 0x7f16ec08f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:38062/0 (socket says 172.21.15.16:38062) 2019-12-05T06:23:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.716+0000 7f16f103a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec08d2c0 0x7f16ec08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:55960/0 (socket says 172.21.15.16:55960) 2019-12-05T06:23:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.716+0000 7f16f203c700 1 -- 172.21.15.16:0/1108387050 learned_addr learned my addr 172.21.15.16:0/1108387050 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.717+0000 7f16f203c700 1 -- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec08d2c0 msgr2=0x7f16ec08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:17.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.717+0000 7f16f203c700 1 --2- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec08d2c0 0x7f16ec08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.717+0000 7f16f203c700 1 -- 172.21.15.16:0/1108387050 >> v1:172.21.15.76:6789/0 conn(0x7f16ec08c6d0 legacy=0x7f16ec08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.717+0000 7f16f203c700 1 -- 172.21.15.16:0/1108387050 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec08e620 con 0x7f16ec08fef0 2019-12-05T06:23:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.717+0000 7f16f103a700 1 --2- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec08d2c0 0x7f16ec08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.717+0000 7f16f203c700 1 --2- 172.21.15.16:0/1108387050 >> v2:172.21.15.76:3300/0 conn(0x7f16ec08fef0 0x7f16ec08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f16e000f760 tx=0x7f16e000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.717+0000 7f16f203c700 1 --2- 172.21.15.16:0/1108387050 >> v2:172.21.15.76:3300/0 conn(0x7f16ec08fef0 0x7f16ec08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f16e000f760 tx=0x7f16e000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.717+0000 7f16f203c700 1 --2- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16ec08fef0 0x7f16ec08f460 secure :-1 s=READY pgs=477 cs=0 l=1 rx=0x7f16e000f760 tx=0x7f16e000fb00).ready entity=mon.1 client_cookie=bd97d939821006c9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:17.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.718+0000 7f16f203c700 1 --2- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16ec08fef0 0x7f16ec08f460 secure :-1 s=THROTTLE_DONE pgs=477 cs=0 l=1 rx=0x7f16e000f760 tx=0x7f16e000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.718+0000 7f16f203c700 1 --2- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16ec08fef0 0x7f16ec08f460 secure :-1 s=THROTTLE_DONE pgs=477 cs=0 l=1 rx=0x7f16e000f760 tx=0x7f16e000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.718+0000 7f16f203c700 1 --2- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16ec08fef0 0x7f16ec08f460 secure :-1 s=THROTTLE_DONE pgs=477 cs=0 l=1 rx=0x7f16e000f760 tx=0x7f16e000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.718+0000 7f16f0839700 1 -- 172.21.15.16:0/1108387050 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16e0000a70 con 0x7f16ec08fef0 2019-12-05T06:23:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.718+0000 7f16f0839700 1 -- 172.21.15.16:0/1108387050 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e0000e70 con 0x7f16ec08fef0 2019-12-05T06:23:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.718+0000 7f16f0839700 1 -- 172.21.15.16:0/1108387050 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16e0001500 con 0x7f16ec08fef0 2019-12-05T06:23:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.718+0000 7f16f329e700 1 -- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16ec08fef0 msgr2=0x7f16ec08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.718+0000 7f16f329e700 1 --2- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16ec08fef0 0x7f16ec08f460 secure :-1 s=READY pgs=477 cs=0 l=1 rx=0x7f16e000f760 tx=0x7f16e000fb00).stop 2019-12-05T06:23:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.720+0000 7f16f329e700 1 -- 172.21.15.16:0/1108387050 shutdown_connections 2019-12-05T06:23:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.720+0000 7f16f329e700 1 --2- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f16ec08fef0 0x7f16ec08f460 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:17.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.720+0000 7f16f329e700 1 --2- 172.21.15.16:0/1108387050 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec08d2c0 0x7f16ec08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.720+0000 7f16f329e700 1 -- 172.21.15.16:0/1108387050 shutdown_connections 2019-12-05T06:23:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.720+0000 7f16f329e700 1 -- 172.21.15.16:0/1108387050 wait complete. 2019-12-05T06:23:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.720+0000 7f16f329e700 1 -- 172.21.15.16:0/1108387050 >> 172.21.15.16:0/1108387050 conn(0x7f16ec061940 msgr2=0x7f16ec096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.726+0000 7f16f329e700 1 Processor -- start 2019-12-05T06:23:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.726+0000 7f16f329e700 1 -- start start 2019-12-05T06:23:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.728+0000 7f16f329e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec1302f0 0x7f16ec12a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:17.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.728+0000 7f16f329e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 0x7f16ec155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.728+0000 7f16f329e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ec131bf0 con 0x7f16ec12b640 2019-12-05T06:23:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.728+0000 7f16f329e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16ec128cb0 con 0x7f16ec153b50 2019-12-05T06:23:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.728+0000 7f16f329e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16ec128f80 con 0x7f16ec1302f0 2019-12-05T06:23:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.729+0000 7f16f103a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f16ec12b640 0x7f16ec12abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37142/0 (socket says 172.21.15.16:37142) 2019-12-05T06:23:17.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.729+0000 7f16f103a700 1 -- 172.21.15.16:0/3217833468 learned_addr learned my addr 172.21.15.16:0/3217833468 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.729+0000 7f16f203c700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 0x7f16ec155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.729+0000 7f16f183b700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec1302f0 0x7f16ec12a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.729+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3785108622 0 0) 0x7f16e400e110 con 0x7f16ec12b640 2019-12-05T06:23:17.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.729+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16cc002e80 con 0x7f16ec12b640 2019-12-05T06:23:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16f203c700 1 -- 172.21.15.16:0/3217833468 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec1302f0 msgr2=0x7f16ec12a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16f203c700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec1302f0 0x7f16ec12a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16f203c700 1 -- 172.21.15.16:0/3217833468 >> v1:172.21.15.16:6789/0 conn(0x7f16ec12b640 legacy=0x7f16ec12abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2606238040 0 0) 0x7f16e4001420 con 0x7f16ec12b640 2019-12-05T06:23:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16f203c700 1 -- 172.21.15.16:0/3217833468 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ec129d70 con 0x7f16ec153b50 2019-12-05T06:23:17.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16f183b700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f16ec1302f0 0x7f16ec12a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16f203c700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 0x7f16ec155f50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f16e000c170 tx=0x7f16e000c260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16f203c700 1 -- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 msgr2=0x7f16ec155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2019-12-05T06:23:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16f203c700 1 -- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 msgr2=0x7f16ec155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:23:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16f203c700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 0x7f16ec155f50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f16e000c170 tx=0x7f16e000c260).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:23:17.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.730+0000 7f16f203c700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 0x7f16ec155f50 secure :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0x7f16e000c170 tx=0x7f16e000c260)._fault waiting 0.200000 2019-12-05T06:23:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.931+0000 7f16f203c700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 0x7f16ec155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.931+0000 7f16f203c700 0 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 0x7f16ec155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T06:23:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.931+0000 7f16f203c700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 0x7f16ec155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.931+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 msgr2=0x7f16ec155f50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:23:17.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.931+0000 7f16ea7fc700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.76:3300/0 conn(0x7f16ec153b50 0x7f16ec155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.931+0000 7f16ea7fc700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.16:3300/0 conn(0x7f16cc006d50 0x7f16cc009150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.931+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16cc0098a0 con 0x7f16cc0030b0 2019-12-05T06:23:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.931+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16cc00abb0 con 0x7f16cc006d50 2019-12-05T06:23:17.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.931+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.16:3300/0 conn(0x7f16cc006d50 0x7f16cc009150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:17.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.932+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308359747 0 0) 0x7f16dc00a5e0 con 0x7f16cc0030b0 2019-12-05T06:23:17.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.932+0000 7f16f103a700 1 -- 172.21.15.16:0/3217833468 >> v1:172.21.15.76:6789/0 conn(0x7f16cc0030b0 legacy=0x7f16cc005540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:17.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.932+0000 7f16f103a700 1 -- 172.21.15.16:0/3217833468 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16cc00ae60 con 0x7f16cc006d50 2019-12-05T06:23:17.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.932+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.16:3300/0 conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.933+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> v2:172.21.15.16:3300/0 conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.933+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=READY pgs=653 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:17.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.933+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.933+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16e4001840 con 0x7f16cc006d50 2019-12-05T06:23:17.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.933+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.933+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.933+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e4002010 con 0x7f16cc006d50 2019-12-05T06:23:17.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.933+0000 7f16f329e700 1 -- 172.21.15.16:0/3217833468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ec129d70 con 0x7f16cc006d50 2019-12-05T06:23:17.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.933+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16e4002760 con 0x7f16cc006d50 2019-12-05T06:23:17.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.934+0000 7f16f329e700 1 -- 172.21.15.16:0/3217833468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ec129d70 con 0x7f16cc006d50 2019-12-05T06:23:17.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.934+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.935+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f16e4001840 con 0x7f16cc006d50 2019-12-05T06:23:17.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.935+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.936+0000 7f16ea7fc700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f16cc029fb0 0x7f16cc02c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:17.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.936+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f16e40460d0 con 0x7f16cc006d50 2019-12-05T06:23:17.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.936+0000 7f16f183b700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f16cc029fb0 0x7f16cc02c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:17.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.937+0000 7f16f183b700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f16cc029fb0 0x7f16cc02c440 crc :-1 s=READY pgs=293 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.979+0000 7f16f329e700 1 -- 172.21.15.16:0/3217833468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ec129d70 con 0x7f16cc006d50 2019-12-05T06:23:17.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.988+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:17.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:17.988+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f16e4001c10 con 0x7f16cc006d50 2019-12-05T06:23:18.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.273+0000 7f16f329e700 1 -- 172.21.15.16:0/3217833468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f16ec129d70 con 0x7f16cc006d50 2019-12-05T06:23:18.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.273+0000 7f16f103a700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.273+0000 7f16ea7fc700 1 -- 172.21.15.16:0/3217833468 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f16e4001840 con 0x7f16cc006d50 2019-12-05T06:23:18.261 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 82 2019-12-05T06:23:18.261 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:18.261 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5878,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:47.975651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":18,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.276+0000 7f16f329e700 1 -- 172.21.15.16:0/3217833468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f16cc029fb0 msgr2=0x7f16cc02c440 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.276+0000 7f16f329e700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f16cc029fb0 0x7f16cc02c440 crc :-1 s=READY pgs=293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.276+0000 7f16f329e700 1 -- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 msgr2=0x7f16cc009150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.276+0000 7f16f329e700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 0x7f16cc009150 secure :-1 s=READY pgs=653 cs=0 l=1 rx=0x7f16e4010750 tx=0x7f16e4010af0).stop 2019-12-05T06:23:18.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.276+0000 7f16f203c700 1 -- 172.21.15.16:0/3217833468 reap_dead start 2019-12-05T06:23:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.276+0000 7f16f203c700 1 -- 172.21.15.16:0/3217833468 reap_dead start 2019-12-05T06:23:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.277+0000 7f16f329e700 1 -- 172.21.15.16:0/3217833468 shutdown_connections 2019-12-05T06:23:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.277+0000 7f16f329e700 1 --2- 172.21.15.16:0/3217833468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f16cc006d50 0x7f16cc009150 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.277+0000 7f16f329e700 1 -- 172.21.15.16:0/3217833468 shutdown_connections 2019-12-05T06:23:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.277+0000 7f16f329e700 1 -- 172.21.15.16:0/3217833468 wait complete. 2019-12-05T06:23:18.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:18.277+0000 7f16f329e700 1 -- 172.21.15.16:0/3217833468 >> 172.21.15.16:0/3217833468 conn(0x7f16ec0a2850 msgr2=0x7f16ec06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:18.275 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:23:20.276 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.524+0000 7f143a22c700 1 Processor -- start 2019-12-05T06:23:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.524+0000 7f143a22c700 1 -- start start 2019-12-05T06:23:20.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.525+0000 7f143a22c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f143408c6d0 0x7f143408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:20.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.526+0000 7f143a22c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f143408f720 0x7f143408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:20.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.526+0000 7f143a22c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14340958e0 con 0x7f143408fef0 2019-12-05T06:23:20.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.526+0000 7f143a22c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14340a3f70 con 0x7f143408f720 2019-12-05T06:23:20.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.526+0000 7f143a22c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143408d830 con 0x7f143408c6d0 2019-12-05T06:23:20.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.526+0000 7f1438fca700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f143408f720 0x7f143408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:20.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.526+0000 7f1438fca700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f143408f720 0x7f143408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51996/0 (socket says 172.21.15.16:51996) 2019-12-05T06:23:20.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.526+0000 7f1438fca700 1 -- 172.21.15.16:0/1092047024 learned_addr learned my addr 172.21.15.16:0/1092047024 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:20.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.526+0000 7f1433fff700 1 --2- 172.21.15.16:0/1092047024 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f143408c6d0 0x7f143408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:20.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1432ffd700 1 -- 172.21.15.16:0/1092047024 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 397539315 0 0) 0x7f1428001400 con 0x7f143408fef0 2019-12-05T06:23:20.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1438fca700 1 -- 172.21.15.16:0/1092047024 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f143408c6d0 msgr2=0x7f143408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1438fca700 1 --2- 172.21.15.16:0/1092047024 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f143408c6d0 0x7f143408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1438fca700 1 -- 172.21.15.16:0/1092047024 >> v1:172.21.15.76:6789/0 conn(0x7f143408fef0 legacy=0x7f143408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1438fca700 1 -- 172.21.15.16:0/1092047024 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143408fbc0 con 0x7f143408f720 2019-12-05T06:23:20.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1438fca700 1 --2- 172.21.15.16:0/1092047024 >> v2:172.21.15.16:3300/0 conn(0x7f143408f720 0x7f143408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f142c00f730 tx=0x7f142c007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1438fca700 1 --2- 172.21.15.16:0/1092047024 >> v2:172.21.15.16:3300/0 conn(0x7f143408f720 0x7f143408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f142c00f730 tx=0x7f142c007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1438fca700 1 --2- 172.21.15.16:0/1092047024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143408f720 0x7f143408d000 secure :-1 s=READY pgs=654 cs=0 l=1 rx=0x7f142c00f730 tx=0x7f142c007d50).ready entity=mon.0 client_cookie=f02dcc145caf43ed server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:20.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1438fca700 1 --2- 172.21.15.16:0/1092047024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143408f720 0x7f143408d000 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7f142c00f730 tx=0x7f142c007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1432ffd700 1 -- 172.21.15.16:0/1092047024 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f142c000a70 con 0x7f143408f720 2019-12-05T06:23:20.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1438fca700 1 --2- 172.21.15.16:0/1092047024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143408f720 0x7f143408d000 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7f142c00f730 tx=0x7f142c007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1438fca700 1 --2- 172.21.15.16:0/1092047024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143408f720 0x7f143408d000 secure :-1 s=THROTTLE_DONE pgs=654 cs=0 l=1 rx=0x7f142c00f730 tx=0x7f142c007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.527+0000 7f1432ffd700 1 -- 172.21.15.16:0/1092047024 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f142c0010e0 con 0x7f143408f720 2019-12-05T06:23:20.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.528+0000 7f1432ffd700 1 -- 172.21.15.16:0/1092047024 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f142c0017b0 con 0x7f143408f720 2019-12-05T06:23:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.528+0000 7f143a22c700 1 -- 172.21.15.16:0/1092047024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143408f720 msgr2=0x7f143408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.528+0000 7f143a22c700 1 --2- 172.21.15.16:0/1092047024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143408f720 0x7f143408d000 secure :-1 s=READY pgs=654 cs=0 l=1 rx=0x7f142c00f730 tx=0x7f142c007d50).stop 2019-12-05T06:23:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.529+0000 7f143a22c700 1 -- 172.21.15.16:0/1092047024 shutdown_connections 2019-12-05T06:23:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.529+0000 7f143a22c700 1 --2- 172.21.15.16:0/1092047024 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143408f720 0x7f143408d000 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.529+0000 7f143a22c700 1 --2- 172.21.15.16:0/1092047024 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f143408c6d0 0x7f143408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.529+0000 7f143a22c700 1 -- 172.21.15.16:0/1092047024 shutdown_connections 2019-12-05T06:23:20.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.529+0000 7f143a22c700 1 -- 172.21.15.16:0/1092047024 wait complete. 2019-12-05T06:23:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.529+0000 7f143a22c700 1 -- 172.21.15.16:0/1092047024 >> 172.21.15.16:0/1092047024 conn(0x7f1434061940 msgr2=0x7f1434096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:20.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.535+0000 7f143a22c700 1 Processor -- start 2019-12-05T06:23:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.535+0000 7f143a22c700 1 -- start start 2019-12-05T06:23:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.537+0000 7f143a22c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f143414d400 0x7f143414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:20.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.537+0000 7f143a22c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1434132530 0x7f1434157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.537+0000 7f143a22c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1434132980 con 0x7f1434132d00 2019-12-05T06:23:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.537+0000 7f143a22c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f143409bb50 con 0x7f143414d400 2019-12-05T06:23:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.537+0000 7f143a22c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f143409aa60 con 0x7f1434132530 2019-12-05T06:23:20.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.537+0000 7f1433fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f143414d400 0x7f143414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.537+0000 7f1433fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f143414d400 0x7f143414d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:51998/0 (socket says 172.21.15.16:51998) 2019-12-05T06:23:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.537+0000 7f1433fff700 1 -- 172.21.15.16:0/1955147581 learned_addr learned my addr 172.21.15.16:0/1955147581 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.537+0000 7f1438fca700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1434132530 0x7f1434157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.538+0000 7f1430ff9700 1 -- 172.21.15.16:0/1955147581 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1067202586 0 0) 0x7f14280015c0 con 0x7f1434132d00 2019-12-05T06:23:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.538+0000 7f1433fff700 1 -- 172.21.15.16:0/1955147581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1434132530 msgr2=0x7f1434157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:20.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.538+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1434132530 0x7f1434157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.538+0000 7f1433fff700 1 -- 172.21.15.16:0/1955147581 >> v1:172.21.15.76:6789/0 conn(0x7f1434132d00 legacy=0x7f1434131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.538+0000 7f1433fff700 1 -- 172.21.15.16:0/1955147581 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f143409b0f0 con 0x7f143414d400 2019-12-05T06:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.538+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> v2:172.21.15.16:3300/0 conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.538+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> v2:172.21.15.16:3300/0 conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.538+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=READY pgs=655 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.539+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=THROTTLE_DONE pgs=655 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.539+0000 7f1430ff9700 1 -- 172.21.15.16:0/1955147581 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1424000940 con 0x7f143414d400 2019-12-05T06:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.539+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=THROTTLE_DONE pgs=655 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.539+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=THROTTLE_DONE pgs=655 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.539+0000 7f1430ff9700 1 -- 172.21.15.16:0/1955147581 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1424000e80 con 0x7f143414d400 2019-12-05T06:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.539+0000 7f1430ff9700 1 -- 172.21.15.16:0/1955147581 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14240014d0 con 0x7f143414d400 2019-12-05T06:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.539+0000 7f143a22c700 1 -- 172.21.15.16:0/1955147581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f143409b0f0 con 0x7f143414d400 2019-12-05T06:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.539+0000 7f143a22c700 1 -- 172.21.15.16:0/1955147581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f143409b0f0 con 0x7f143414d400 2019-12-05T06:23:20.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.540+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=THROTTLE_DONE pgs=655 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.541+0000 7f1430ff9700 1 -- 172.21.15.16:0/1955147581 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1424001070 con 0x7f143414d400 2019-12-05T06:23:20.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.541+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=THROTTLE_DONE pgs=655 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.541+0000 7f1430ff9700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1414023630 0x7f1414025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.541+0000 7f1430ff9700 1 -- 172.21.15.16:0/1955147581 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7f1424045de0 con 0x7f143414d400 2019-12-05T06:23:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.541+0000 7f14337fe700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1414023630 0x7f1414025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:20.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.542+0000 7f14337fe700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1414023630 0x7f1414025ac0 crc :-1 s=READY pgs=294 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:20.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.555+0000 7f143a22c700 1 -- 172.21.15.16:0/1955147581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f143409b0f0 con 0x7f143414d400 2019-12-05T06:23:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.563+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=THROTTLE_DONE pgs=655 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.563+0000 7f1430ff9700 1 -- 172.21.15.16:0/1955147581 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1424009250 con 0x7f143414d400 2019-12-05T06:23:20.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.790+0000 7f143a22c700 1 -- 172.21.15.16:0/1955147581 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f143409b0f0 con 0x7f143414d400 2019-12-05T06:23:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.790+0000 7f1433fff700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=THROTTLE_DONE pgs=655 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.790+0000 7f1430ff9700 1 -- 172.21.15.16:0/1955147581 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 82 v82) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f1424001070 con 0x7f143414d400 2019-12-05T06:23:20.777 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 82 2019-12-05T06:23:20.777 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:20.778 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":82,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5878,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":82}],"filesystems":[{"mdsmap":{"epoch":81,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:21:47.975651+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":59,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5776},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5776":{"gid":5776,"name":"a","rank":0,"incarnation":78,"state":"up:active","state_seq":18,"addr":"172.21.15.16:6833/3114985010","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3114985010},{"type":"v1","addr":"172.21.15.16:6833","nonce":3114985010}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:20.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.793+0000 7f143a22c700 1 -- 172.21.15.16:0/1955147581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1414023630 msgr2=0x7f1414025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.793+0000 7f143a22c700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1414023630 0x7f1414025ac0 crc :-1 s=READY pgs=294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.793+0000 7f143a22c700 1 -- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 msgr2=0x7f143414d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.793+0000 7f143a22c700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 0x7f143414d7f0 secure :-1 s=READY pgs=655 cs=0 l=1 rx=0x7f1424010450 tx=0x7f1424010280).stop 2019-12-05T06:23:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.793+0000 7f143a22c700 1 -- 172.21.15.16:0/1955147581 shutdown_connections 2019-12-05T06:23:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.793+0000 7f143a22c700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1434132530 0x7f1434157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:20.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.793+0000 7f143a22c700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1414023630 0x7f1414025ac0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:20.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.793+0000 7f143a22c700 1 --2- 172.21.15.16:0/1955147581 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f143414d400 0x7f143414d7f0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:20.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.794+0000 7f143a22c700 1 -- 172.21.15.16:0/1955147581 shutdown_connections 2019-12-05T06:23:20.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.794+0000 7f143a22c700 1 -- 172.21.15.16:0/1955147581 wait complete. 2019-12-05T06:23:20.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:20.794+0000 7f143a22c700 1 -- 172.21.15.16:0/1955147581 >> 172.21.15.16:0/1955147581 conn(0x7f14340714f0 msgr2=0x7f143409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:20.791 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:23:22.793 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:23.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.079+0000 7fb509c63700 1 Processor -- start 2019-12-05T06:23:23.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.079+0000 7fb509c63700 1 -- start start 2019-12-05T06:23:23.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.081+0000 7fb509c63700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb50408c6d0 0x7fb50408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:23.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.081+0000 7fb509c63700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb50408f720 0x7fb50408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:23.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.081+0000 7fb509c63700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb50408fb70 con 0x7fb50408fef0 2019-12-05T06:23:23.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.081+0000 7fb509c63700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb50408df90 con 0x7fb50408c6d0 2019-12-05T06:23:23.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.081+0000 7fb509c63700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5040a3f70 con 0x7fb50408f720 2019-12-05T06:23:23.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.081+0000 7fb503fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb50408c6d0 0x7fb50408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:23.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.081+0000 7fb503fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb50408c6d0 0x7fb50408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52012/0 (socket says 172.21.15.16:52012) 2019-12-05T06:23:23.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.081+0000 7fb503fff700 1 -- 172.21.15.16:0/3735494050 learned_addr learned my addr 172.21.15.16:0/3735494050 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:23.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb508a01700 1 --2- 172.21.15.16:0/3735494050 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb50408f720 0x7fb50408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:23.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb503fff700 1 -- 172.21.15.16:0/3735494050 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb50408f720 msgr2=0x7fb50408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:23.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb503fff700 1 --2- 172.21.15.16:0/3735494050 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb50408f720 0x7fb50408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:23.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb503fff700 1 -- 172.21.15.16:0/3735494050 >> v1:172.21.15.76:6789/0 conn(0x7fb50408fef0 legacy=0x7fb50408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb503fff700 1 -- 172.21.15.16:0/3735494050 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50408d540 con 0x7fb50408c6d0 2019-12-05T06:23:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb503fff700 1 --2- 172.21.15.16:0/3735494050 >> v2:172.21.15.16:3300/0 conn(0x7fb50408c6d0 0x7fb50408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb4ec00f740 tx=0x7fb4ec00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb503fff700 1 --2- 172.21.15.16:0/3735494050 >> v2:172.21.15.16:3300/0 conn(0x7fb50408c6d0 0x7fb50408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb4ec00f740 tx=0x7fb4ec00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb503fff700 1 --2- 172.21.15.16:0/3735494050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb50408c6d0 0x7fb50408cac0 secure :-1 s=READY pgs=656 cs=0 l=1 rx=0x7fb4ec00f740 tx=0x7fb4ec00f570).ready entity=mon.0 client_cookie=354e2e3fea44d87a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb503fff700 1 --2- 172.21.15.16:0/3735494050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb50408c6d0 0x7fb50408cac0 secure :-1 s=THROTTLE_DONE pgs=656 cs=0 l=1 rx=0x7fb4ec00f740 tx=0x7fb4ec00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb503fff700 1 --2- 172.21.15.16:0/3735494050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb50408c6d0 0x7fb50408cac0 secure :-1 s=THROTTLE_DONE pgs=656 cs=0 l=1 rx=0x7fb4ec00f740 tx=0x7fb4ec00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.083+0000 7fb503fff700 1 --2- 172.21.15.16:0/3735494050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb50408c6d0 0x7fb50408cac0 secure :-1 s=THROTTLE_DONE pgs=656 cs=0 l=1 rx=0x7fb4ec00f740 tx=0x7fb4ec00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.082+0000 7fb5037fe700 1 -- 172.21.15.16:0/3735494050 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4ec000960 con 0x7fb50408c6d0 2019-12-05T06:23:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.083+0000 7fb5037fe700 1 -- 172.21.15.16:0/3735494050 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4ec000c90 con 0x7fb50408c6d0 2019-12-05T06:23:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.083+0000 7fb5037fe700 1 -- 172.21.15.16:0/3735494050 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4ec0013a0 con 0x7fb50408c6d0 2019-12-05T06:23:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.083+0000 7fb509c63700 1 -- 172.21.15.16:0/3735494050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb50408c6d0 msgr2=0x7fb50408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.083+0000 7fb509c63700 1 --2- 172.21.15.16:0/3735494050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb50408c6d0 0x7fb50408cac0 secure :-1 s=READY pgs=656 cs=0 l=1 rx=0x7fb4ec00f740 tx=0x7fb4ec00f570).stop 2019-12-05T06:23:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.084+0000 7fb509c63700 1 -- 172.21.15.16:0/3735494050 shutdown_connections 2019-12-05T06:23:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.084+0000 7fb509c63700 1 --2- 172.21.15.16:0/3735494050 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb50408f720 0x7fb50408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.084+0000 7fb509c63700 1 --2- 172.21.15.16:0/3735494050 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb50408c6d0 0x7fb50408cac0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.084+0000 7fb509c63700 1 -- 172.21.15.16:0/3735494050 shutdown_connections 2019-12-05T06:23:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.084+0000 7fb509c63700 1 -- 172.21.15.16:0/3735494050 wait complete. 2019-12-05T06:23:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.084+0000 7fb509c63700 1 -- 172.21.15.16:0/3735494050 >> 172.21.15.16:0/3735494050 conn(0x7fb504061940 msgr2=0x7fb504096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:23.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.089+0000 7fb509c63700 1 Processor -- start 2019-12-05T06:23:23.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.089+0000 7fb509c63700 1 -- start start 2019-12-05T06:23:23.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.090+0000 7fb509c63700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb504128660 0x7fb50409eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:23.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.090+0000 7fb509c63700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb504156cc0 0x7fb5041590c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:23.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.090+0000 7fb509c63700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb50409c1f0 con 0x7fb50409fe70 2019-12-05T06:23:23.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.090+0000 7fb509c63700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb50409d480 con 0x7fb504156cc0 2019-12-05T06:23:23.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.090+0000 7fb509c63700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb50409d750 con 0x7fb504128660 2019-12-05T06:23:23.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.090+0000 7fb508a01700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb504156cc0 0x7fb5041590c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:23.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.090+0000 7fb508a01700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb504156cc0 0x7fb5041590c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52018/0 (socket says 172.21.15.16:52018) 2019-12-05T06:23:23.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.090+0000 7fb4fbfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb50409fe70 0x7fb50409f3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37184/0 (socket says 172.21.15.16:37184) 2019-12-05T06:23:23.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.091+0000 7fb508a01700 1 -- 172.21.15.16:0/253820556 learned_addr learned my addr 172.21.15.16:0/253820556 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.091+0000 7fb503fff700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb504128660 0x7fb50409eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.091+0000 7fb508a01700 1 -- 172.21.15.16:0/253820556 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb504128660 msgr2=0x7fb50409eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.091+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb504128660 0x7fb50409eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.091+0000 7fb508a01700 1 -- 172.21.15.16:0/253820556 >> v1:172.21.15.16:6789/0 conn(0x7fb50409fe70 legacy=0x7fb50409f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.091+0000 7fb508a01700 1 -- 172.21.15.16:0/253820556 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb50409e540 con 0x7fb504156cc0 2019-12-05T06:23:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.091+0000 7fb5017fa700 1 -- 172.21.15.16:0/253820556 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399110396 0 0) 0x7fb4f40010c0 con 0x7fb50409fe70 2019-12-05T06:23:23.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.091+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> v2:172.21.15.16:3300/0 conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.091+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> v2:172.21.15.16:3300/0 conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.091+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=READY pgs=658 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.092+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.092+0000 7fb5017fa700 1 -- 172.21.15.16:0/253820556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4fc00aa30 con 0x7fb504156cc0 2019-12-05T06:23:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.092+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.092+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.092+0000 7fb5017fa700 1 -- 172.21.15.16:0/253820556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4fc0013e0 con 0x7fb504156cc0 2019-12-05T06:23:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.092+0000 7fb5017fa700 1 -- 172.21.15.16:0/253820556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4fc0019b0 con 0x7fb504156cc0 2019-12-05T06:23:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.092+0000 7fb509c63700 1 -- 172.21.15.16:0/253820556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb50409e540 con 0x7fb504156cc0 2019-12-05T06:23:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.092+0000 7fb509c63700 1 -- 172.21.15.16:0/253820556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb50409e540 con 0x7fb504156cc0 2019-12-05T06:23:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.093+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.094+0000 7fb5017fa700 1 -- 172.21.15.16:0/253820556 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb4fc00aa30 con 0x7fb504156cc0 2019-12-05T06:23:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.094+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.094+0000 7fb5017fa700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb4e8023630 0x7fb4e8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:23.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.094+0000 7fb5017fa700 1 -- 172.21.15.16:0/253820556 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 6891+0+0 (secure 0 0 0) 0x7fb4fc0461a0 con 0x7fb504156cc0 2019-12-05T06:23:23.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.094+0000 7fb503fff700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb4e8023630 0x7fb4e8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:23.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.095+0000 7fb503fff700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb4e8023630 0x7fb4e8025ac0 crc :-1 s=READY pgs=295 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.111+0000 7fb509c63700 1 -- 172.21.15.16:0/253820556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb50409e540 con 0x7fb504156cc0 2019-12-05T06:23:23.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.119+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.119+0000 7fb5017fa700 1 -- 172.21.15.16:0/253820556 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb4fc000ee0 con 0x7fb504156cc0 2019-12-05T06:23:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.367+0000 7fb509c63700 1 -- 172.21.15.16:0/253820556 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb50409e540 con 0x7fb504156cc0 2019-12-05T06:23:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.368+0000 7fb508a01700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=THROTTLE_DONE pgs=658 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.368+0000 7fb5017fa700 1 -- 172.21.15.16:0/253820556 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+1768 (secure 0 0 0) 0x7fb4fc00aa30 con 0x7fb504156cc0 2019-12-05T06:23:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 83 2019-12-05T06:23:23.355 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:23.355 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:23.146643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.370+0000 7fb509c63700 1 -- 172.21.15.16:0/253820556 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb4e8023630 msgr2=0x7fb4e8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.370+0000 7fb509c63700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb4e8023630 0x7fb4e8025ac0 crc :-1 s=READY pgs=295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.370+0000 7fb509c63700 1 -- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 msgr2=0x7fb5041590c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.371+0000 7fb509c63700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 0x7fb5041590c0 secure :-1 s=READY pgs=658 cs=0 l=1 rx=0x7fb4fc0107e0 tx=0x7fb4fc0105e0).stop 2019-12-05T06:23:23.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.371+0000 7fb509c63700 1 -- 172.21.15.16:0/253820556 shutdown_connections 2019-12-05T06:23:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.371+0000 7fb509c63700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb4e8023630 0x7fb4e8025ac0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.371+0000 7fb509c63700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb504156cc0 0x7fb5041590c0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.371+0000 7fb509c63700 1 --2- 172.21.15.16:0/253820556 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb504128660 0x7fb50409eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.371+0000 7fb509c63700 1 -- 172.21.15.16:0/253820556 shutdown_connections 2019-12-05T06:23:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.371+0000 7fb509c63700 1 -- 172.21.15.16:0/253820556 wait complete. 2019-12-05T06:23:23.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.371+0000 7fb509c63700 1 -- 172.21.15.16:0/253820556 >> 172.21.15.16:0/253820556 conn(0x7fb50408a5e0 msgr2=0x7fb50406ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:23.373 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:23:23.373 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:23:23.373 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:23.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.563+0000 7f96c8967700 1 Processor -- start 2019-12-05T06:23:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.563+0000 7f96c8967700 1 -- start start 2019-12-05T06:23:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.565+0000 7f96c8967700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 0x7f96c408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.565+0000 7f96c8967700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c40958e0 con 0x7f96c408c6d0 2019-12-05T06:23:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.565+0000 7f96c8967700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c40629e0 con 0x7f96c408d2c0 2019-12-05T06:23:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.565+0000 7f96c8967700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c408df90 con 0x7f96c408fef0 2019-12-05T06:23:23.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.565+0000 7f96c2d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f96c408c6d0 0x7f96c408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37196/0 (socket says 172.21.15.16:37196) 2019-12-05T06:23:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.565+0000 7f96c2d9d700 1 -- 172.21.15.16:0/3455833029 learned_addr learned my addr 172.21.15.16:0/3455833029 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.565+0000 7f96c359e700 1 --2- 172.21.15.16:0/3455833029 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 0x7f96c408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.565+0000 7f96c1d9b700 1 -- 172.21.15.16:0/3455833029 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631219695 0 0) 0x7f96bc001480 con 0x7f96c408c6d0 2019-12-05T06:23:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.565+0000 7f96c1d9b700 1 -- 172.21.15.16:0/3455833029 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b0002e80 con 0x7f96c408c6d0 2019-12-05T06:23:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.565+0000 7f96c1d9b700 1 -- 172.21.15.16:0/3455833029 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569994316 0 0) 0x7f96ac001080 con 0x7f96c408d2c0 2019-12-05T06:23:23.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.566+0000 7f96c1d9b700 1 -- 172.21.15.16:0/3455833029 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96b0002e80 con 0x7f96c408d2c0 2019-12-05T06:23:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.566+0000 7f96c1d9b700 1 -- 172.21.15.16:0/3455833029 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3096540244 0 0) 0x7f96bc001460 con 0x7f96c408c6d0 2019-12-05T06:23:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.566+0000 7f96c359e700 1 -- 172.21.15.16:0/3455833029 >> v1:172.21.15.76:6789/0 conn(0x7f96c408d2c0 legacy=0x7f96c408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.566+0000 7f96c359e700 1 -- 172.21.15.16:0/3455833029 >> v1:172.21.15.16:6789/0 conn(0x7f96c408c6d0 legacy=0x7f96c408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.566+0000 7f96c359e700 1 -- 172.21.15.16:0/3455833029 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c408e620 con 0x7f96c408fef0 2019-12-05T06:23:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.566+0000 7f96c359e700 1 --2- 172.21.15.16:0/3455833029 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 0x7f96c408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f96b400af80 tx=0x7f96b400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.566+0000 7f96c359e700 1 --2- 172.21.15.16:0/3455833029 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 0x7f96c408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f96b400af80 tx=0x7f96b400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.566+0000 7f96c359e700 1 --2- 172.21.15.16:0/3455833029 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 0x7f96c408f720 secure :-1 s=READY pgs=164 cs=0 l=1 rx=0x7f96b400af80 tx=0x7f96b400b790).ready entity=mon.2 client_cookie=50e4115e16d178be server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.567+0000 7f96c359e700 1 --2- 172.21.15.16:0/3455833029 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 0x7f96c408f720 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7f96b400af80 tx=0x7f96b400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.567+0000 7f96c1d9b700 1 -- 172.21.15.16:0/3455833029 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96b4007240 con 0x7f96c408fef0 2019-12-05T06:23:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.567+0000 7f96c359e700 1 --2- 172.21.15.16:0/3455833029 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 0x7f96c408f720 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7f96b400af80 tx=0x7f96b400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.567+0000 7f96c359e700 1 --2- 172.21.15.16:0/3455833029 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 0x7f96c408f720 secure :-1 s=THROTTLE_DONE pgs=164 cs=0 l=1 rx=0x7f96b400af80 tx=0x7f96b400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.567+0000 7f96c1d9b700 1 -- 172.21.15.16:0/3455833029 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96b4000bb0 con 0x7f96c408fef0 2019-12-05T06:23:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.567+0000 7f96c1d9b700 1 -- 172.21.15.16:0/3455833029 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96b4009e60 con 0x7f96c408fef0 2019-12-05T06:23:23.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.567+0000 7f96c8967700 1 -- 172.21.15.16:0/3455833029 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 msgr2=0x7f96c408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.567+0000 7f96c8967700 1 --2- 172.21.15.16:0/3455833029 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 0x7f96c408f720 secure :-1 s=READY pgs=164 cs=0 l=1 rx=0x7f96b400af80 tx=0x7f96b400b790).stop 2019-12-05T06:23:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.569+0000 7f96c8967700 1 -- 172.21.15.16:0/3455833029 shutdown_connections 2019-12-05T06:23:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.569+0000 7f96c8967700 1 --2- 172.21.15.16:0/3455833029 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c408fef0 0x7f96c408f720 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.569+0000 7f96c8967700 1 -- 172.21.15.16:0/3455833029 shutdown_connections 2019-12-05T06:23:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.569+0000 7f96c8967700 1 -- 172.21.15.16:0/3455833029 wait complete. 2019-12-05T06:23:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.569+0000 7f96c8967700 1 -- 172.21.15.16:0/3455833029 >> 172.21.15.16:0/3455833029 conn(0x7f96c4061940 msgr2=0x7f96c4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:23.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.572+0000 7f96c8967700 1 Processor -- start 2019-12-05T06:23:23.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.572+0000 7f96c8967700 1 -- start start 2019-12-05T06:23:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.573+0000 7f96c8967700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.573+0000 7f96c8967700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c4131bf0 con 0x7f96c41302f0 2019-12-05T06:23:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.573+0000 7f96c8967700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96c41316f0 con 0x7f96c4153ba0 2019-12-05T06:23:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.573+0000 7f96c8967700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96c41296e0 con 0x7f96c412b640 2019-12-05T06:23:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96c2d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f96c41302f0 0x7f96c412a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37202/0 (socket says 172.21.15.16:37202) 2019-12-05T06:23:23.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96c2d9d700 1 -- 172.21.15.16:0/1757869905 learned_addr learned my addr 172.21.15.16:0/1757869905 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1433476768 0 0) 0x7f96bc0010e0 con 0x7f96c41302f0 2019-12-05T06:23:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a8002e80 con 0x7f96c41302f0 2019-12-05T06:23:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 947712516 0 0) 0x7f96b4009990 con 0x7f96c4153ba0 2019-12-05T06:23:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f96a80030b0 con 0x7f96c4153ba0 2019-12-05T06:23:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2651434598 0 0) 0x7f96bc001710 con 0x7f96c41302f0 2019-12-05T06:23:23.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96c259c700 1 -- 172.21.15.16:0/1757869905 >> v1:172.21.15.76:6789/0 conn(0x7f96c4153ba0 legacy=0x7f96c4155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96c259c700 1 -- 172.21.15.16:0/1757869905 >> v1:172.21.15.16:6789/0 conn(0x7f96c41302f0 legacy=0x7f96c412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96c259c700 1 -- 172.21.15.16:0/1757869905 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96c4129d70 con 0x7f96c412b640 2019-12-05T06:23:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.574+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.575+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.575+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=READY pgs=165 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.575+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.575+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96ac009b70 con 0x7f96c412b640 2019-12-05T06:23:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.575+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.575+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.575+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ac009830 con 0x7f96c412b640 2019-12-05T06:23:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.575+0000 7f96c8967700 1 -- 172.21.15.16:0/1757869905 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96c4129d70 con 0x7f96c412b640 2019-12-05T06:23:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.575+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96ac009a20 con 0x7f96c412b640 2019-12-05T06:23:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.575+0000 7f96c8967700 1 -- 172.21.15.16:0/1757869905 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96c4129d70 con 0x7f96c412b640 2019-12-05T06:23:23.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.576+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.576+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f96ac009ac0 con 0x7f96c412b640 2019-12-05T06:23:23.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.576+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.577+0000 7f96bb7fe700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f96a8023650 0x7f96a8025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:23.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.577+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f96ac045db0 con 0x7f96c412b640 2019-12-05T06:23:23.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.577+0000 7f96c2d9d700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f96a8023650 0x7f96a8025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:23.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.577+0000 7f96c2d9d700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f96a8023650 0x7f96a8025ae0 crc :-1 s=READY pgs=296 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:23.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.579+0000 7f96c8967700 1 -- 172.21.15.16:0/1757869905 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96c4129d70 con 0x7f96c412b640 2019-12-05T06:23:23.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.586+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.586+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f96ac009580 con 0x7f96c412b640 2019-12-05T06:23:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.846+0000 7f96c8967700 1 -- 172.21.15.16:0/1757869905 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f96c4129d70 con 0x7f96c412b640 2019-12-05T06:23:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.847+0000 7f96c259c700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=THROTTLE_DONE pgs=165 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:23.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.847+0000 7f96bb7fe700 1 -- 172.21.15.16:0/1757869905 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+1768 (secure 0 0 0) 0x7f96ac009a20 con 0x7f96c412b640 2019-12-05T06:23:23.835 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 83 2019-12-05T06:23:23.836 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:23.836 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":83,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:23.146643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:replay","state_seq":1,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:23.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.852+0000 7f96c8967700 1 -- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f96a8023650 msgr2=0x7f96a8025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.852+0000 7f96c8967700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f96a8023650 0x7f96a8025ae0 crc :-1 s=READY pgs=296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:23.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.852+0000 7f96c8967700 1 -- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 msgr2=0x7f96c412ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:23.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.852+0000 7f96c8967700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 secure :-1 s=READY pgs=165 cs=0 l=1 rx=0x7f96ac0103a0 tx=0x7f96ac010740).stop 2019-12-05T06:23:23.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.853+0000 7f96c8967700 1 -- 172.21.15.16:0/1757869905 shutdown_connections 2019-12-05T06:23:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.853+0000 7f96c8967700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f96a8023650 0x7f96a8025ae0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.853+0000 7f96c8967700 1 --2- 172.21.15.16:0/1757869905 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f96c412b640 0x7f96c412ae70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.853+0000 7f96c8967700 1 -- 172.21.15.16:0/1757869905 shutdown_connections 2019-12-05T06:23:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.853+0000 7f96c8967700 1 -- 172.21.15.16:0/1757869905 wait complete. 2019-12-05T06:23:23.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:23.853+0000 7f96c8967700 1 -- 172.21.15.16:0/1757869905 >> 172.21.15.16:0/1757869905 conn(0x7f96c40a2850 msgr2=0x7f96c406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:23.850 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 83, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 83, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5878": { "addr": "172.21.15.76:6835/945852410", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 945852410, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 945852410, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5878, "incarnation": 83, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 60, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:23:23.146643+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5878 } } } ], "standbys": [] } 2019-12-05T06:23:25.851 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:26.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.147+0000 7fed10b58700 1 Processor -- start 2019-12-05T06:23:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.147+0000 7fed10b58700 1 -- start start 2019-12-05T06:23:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.149+0000 7fed10b58700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed0c08d2c0 0x7fed0c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.149+0000 7fed10b58700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c08fef0 0x7fed0c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.149+0000 7fed10b58700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c0958e0 con 0x7fed0c08c6d0 2019-12-05T06:23:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.149+0000 7fed10b58700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c08df90 con 0x7fed0c08d2c0 2019-12-05T06:23:26.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.149+0000 7fed10b58700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c0a3f70 con 0x7fed0c08fef0 2019-12-05T06:23:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.149+0000 7fed0a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed0c08d2c0 0x7fed0c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.150+0000 7fed0a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fed0c08d2c0 0x7fed0c08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52048/0 (socket says 172.21.15.16:52048) 2019-12-05T06:23:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.150+0000 7fed0a59c700 1 -- 172.21.15.16:0/4046774502 learned_addr learned my addr 172.21.15.16:0/4046774502 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.150+0000 7fed0ad9d700 1 --1- 172.21.15.16:0/4046774502 >> v1:172.21.15.16:6789/0 conn(0x7fed0c08c6d0 0x7fed0c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37214/0 (socket says 172.21.15.16:37214) 2019-12-05T06:23:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.150+0000 7fed0b59e700 1 --2- 172.21.15.16:0/4046774502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c08fef0 0x7fed0c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:26.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.150+0000 7fed0a59c700 1 -- 172.21.15.16:0/4046774502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c08fef0 msgr2=0x7fed0c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.150+0000 7fed0a59c700 1 --2- 172.21.15.16:0/4046774502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c08fef0 0x7fed0c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.150+0000 7fed0a59c700 1 -- 172.21.15.16:0/4046774502 >> v1:172.21.15.16:6789/0 conn(0x7fed0c08c6d0 legacy=0x7fed0c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.150+0000 7fed0a59c700 1 -- 172.21.15.16:0/4046774502 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c08e620 con 0x7fed0c08d2c0 2019-12-05T06:23:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.150+0000 7fed09d9b700 1 -- 172.21.15.16:0/4046774502 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2659456666 0 0) 0x7fed04001100 con 0x7fed0c08c6d0 2019-12-05T06:23:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed0a59c700 1 --2- 172.21.15.16:0/4046774502 >> v2:172.21.15.16:3300/0 conn(0x7fed0c08d2c0 0x7fed0c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fecf400fa40 tx=0x7fecf400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed0a59c700 1 --2- 172.21.15.16:0/4046774502 >> v2:172.21.15.16:3300/0 conn(0x7fed0c08d2c0 0x7fed0c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fecf400fa40 tx=0x7fecf400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed0a59c700 1 --2- 172.21.15.16:0/4046774502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c08d2c0 0x7fed0c08ef20 secure :-1 s=READY pgs=663 cs=0 l=1 rx=0x7fecf400fa40 tx=0x7fecf400fde0).ready entity=mon.0 client_cookie=3cc389fda0239f0d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed0a59c700 1 --2- 172.21.15.16:0/4046774502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c08d2c0 0x7fed0c08ef20 secure :-1 s=THROTTLE_DONE pgs=663 cs=0 l=1 rx=0x7fecf400fa40 tx=0x7fecf400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed09d9b700 1 -- 172.21.15.16:0/4046774502 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecf4000ce0 con 0x7fed0c08d2c0 2019-12-05T06:23:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed0a59c700 1 --2- 172.21.15.16:0/4046774502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c08d2c0 0x7fed0c08ef20 secure :-1 s=THROTTLE_DONE pgs=663 cs=0 l=1 rx=0x7fecf400fa40 tx=0x7fecf400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed0a59c700 1 --2- 172.21.15.16:0/4046774502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c08d2c0 0x7fed0c08ef20 secure :-1 s=THROTTLE_DONE pgs=663 cs=0 l=1 rx=0x7fecf400fa40 tx=0x7fecf400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed09d9b700 1 -- 172.21.15.16:0/4046774502 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fecf4001010 con 0x7fed0c08d2c0 2019-12-05T06:23:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed09d9b700 1 -- 172.21.15.16:0/4046774502 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fecf4001720 con 0x7fed0c08d2c0 2019-12-05T06:23:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed10b58700 1 -- 172.21.15.16:0/4046774502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c08d2c0 msgr2=0x7fed0c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:26.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.151+0000 7fed10b58700 1 --2- 172.21.15.16:0/4046774502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c08d2c0 0x7fed0c08ef20 secure :-1 s=READY pgs=663 cs=0 l=1 rx=0x7fecf400fa40 tx=0x7fecf400fde0).stop 2019-12-05T06:23:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.153+0000 7fed10b58700 1 -- 172.21.15.16:0/4046774502 shutdown_connections 2019-12-05T06:23:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.153+0000 7fed10b58700 1 --2- 172.21.15.16:0/4046774502 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed0c08d2c0 0x7fed0c08ef20 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.153+0000 7fed10b58700 1 --2- 172.21.15.16:0/4046774502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c08fef0 0x7fed0c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.153+0000 7fed10b58700 1 -- 172.21.15.16:0/4046774502 shutdown_connections 2019-12-05T06:23:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.153+0000 7fed10b58700 1 -- 172.21.15.16:0/4046774502 wait complete. 2019-12-05T06:23:26.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.153+0000 7fed10b58700 1 -- 172.21.15.16:0/4046774502 >> 172.21.15.16:0/4046774502 conn(0x7fed0c061940 msgr2=0x7fed0c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:26.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.158+0000 7fed10b58700 1 Processor -- start 2019-12-05T06:23:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.158+0000 7fed10b58700 1 -- start start 2019-12-05T06:23:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.160+0000 7fed10b58700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.160+0000 7fed10b58700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fed0c09b920 0x7fed0c097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.160+0000 7fed10b58700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed0c084b90 con 0x7fed0c084f30 2019-12-05T06:23:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.160+0000 7fed10b58700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed0c099a20 con 0x7fed0c09b920 2019-12-05T06:23:26.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.160+0000 7fed10b58700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed0c099cf0 con 0x7fed0c096e90 2019-12-05T06:23:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.160+0000 7fed0b59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fed0c09b920 0x7fed0c097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.160+0000 7fed0ad9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.160+0000 7fed0b59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fed0c09b920 0x7fed0c097a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:38146/0 (socket says 172.21.15.16:38146) 2019-12-05T06:23:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.160+0000 7fed0b59e700 1 -- 172.21.15.16:0/884621200 learned_addr learned my addr 172.21.15.16:0/884621200 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.161+0000 7fed0ad9d700 1 -- 172.21.15.16:0/884621200 >> v2:172.21.15.76:3300/0 conn(0x7fed0c09b920 msgr2=0x7fed0c097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.161+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> v2:172.21.15.76:3300/0 conn(0x7fed0c09b920 0x7fed0c097a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:26.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.161+0000 7fed0ad9d700 1 -- 172.21.15.16:0/884621200 >> v1:172.21.15.76:6789/0 conn(0x7fed0c084f30 legacy=0x7fed0c097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.161+0000 7fed037fe700 1 -- 172.21.15.16:0/884621200 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3214023954 0 0) 0x7fecf4007c80 con 0x7fed0c084f30 2019-12-05T06:23:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.161+0000 7fed0ad9d700 1 -- 172.21.15.16:0/884621200 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed0c09aae0 con 0x7fed0c096e90 2019-12-05T06:23:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.161+0000 7fed0b59e700 1 --2- 172.21.15.16:0/884621200 >> v2:172.21.15.76:3300/0 conn(0x7fed0c09b920 0x7fed0c097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.161+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.161+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.161+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=READY pgs=166 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:26.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.162+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.162+0000 7fed037fe700 1 -- 172.21.15.16:0/884621200 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed04000ea0 con 0x7fed0c096e90 2019-12-05T06:23:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.162+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.162+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.162+0000 7fed037fe700 1 -- 172.21.15.16:0/884621200 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed04001220 con 0x7fed0c096e90 2019-12-05T06:23:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.162+0000 7fed037fe700 1 -- 172.21.15.16:0/884621200 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed04000ea0 con 0x7fed0c096e90 2019-12-05T06:23:26.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.162+0000 7fed10b58700 1 -- 172.21.15.16:0/884621200 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed0c09aae0 con 0x7fed0c096e90 2019-12-05T06:23:26.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.162+0000 7fed10b58700 1 -- 172.21.15.16:0/884621200 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed0c09aae0 con 0x7fed0c096e90 2019-12-05T06:23:26.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.163+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.164+0000 7fed037fe700 1 -- 172.21.15.16:0/884621200 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fed04004240 con 0x7fed0c096e90 2019-12-05T06:23:26.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.164+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.164+0000 7fed037fe700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fece4023630 0x7fece4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:26.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.164+0000 7fed037fe700 1 -- 172.21.15.16:0/884621200 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fed04047f10 con 0x7fed0c096e90 2019-12-05T06:23:26.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.165+0000 7fed0a59c700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fece4023630 0x7fece4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:26.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.165+0000 7fed0a59c700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fece4023630 0x7fece4025ac0 crc :-1 s=READY pgs=297 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:26.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.179+0000 7fed10b58700 1 -- 172.21.15.16:0/884621200 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed0c09aae0 con 0x7fed0c096e90 2019-12-05T06:23:26.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.187+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.187+0000 7fed037fe700 1 -- 172.21.15.16:0/884621200 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fed04000c90 con 0x7fed0c096e90 2019-12-05T06:23:26.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.446+0000 7fed10b58700 1 -- 172.21.15.16:0/884621200 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed0c09aae0 con 0x7fed0c096e90 2019-12-05T06:23:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.447+0000 7fed0ad9d700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=THROTTLE_DONE pgs=166 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:26.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.447+0000 7fed037fe700 1 -- 172.21.15.16:0/884621200 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fed04004240 con 0x7fed0c096e90 2019-12-05T06:23:26.436 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 84 2019-12-05T06:23:26.437 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:26.437 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":84,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":84,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:26.125877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:reconnect","state_seq":24,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:26.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 -- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fece4023630 msgr2=0x7fece4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fece4023630 0x7fece4025ac0 crc :-1 s=READY pgs=297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 -- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 msgr2=0x7fed0c09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 secure :-1 s=READY pgs=166 cs=0 l=1 rx=0x7fed040124e0 tx=0x7fed04012880).stop 2019-12-05T06:23:26.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 -- 172.21.15.16:0/884621200 shutdown_connections 2019-12-05T06:23:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fece4023630 0x7fece4025ac0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 --2- 172.21.15.16:0/884621200 >> v2:172.21.15.76:3300/0 conn(0x7fed0c09b920 0x7fed0c097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 --2- 172.21.15.16:0/884621200 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed0c096e90 0x7fed0c09b3e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:26.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 -- 172.21.15.16:0/884621200 shutdown_connections 2019-12-05T06:23:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 -- 172.21.15.16:0/884621200 wait complete. 2019-12-05T06:23:26.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:26.454+0000 7fed10b58700 1 -- 172.21.15.16:0/884621200 >> 172.21.15.16:0/884621200 conn(0x7fed0c0714f0 msgr2=0x7fed0c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:26.457 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:23:28.457 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.662+0000 7fcc0fdc7700 1 Processor -- start 2019-12-05T06:23:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.662+0000 7fcc0fdc7700 1 -- start start 2019-12-05T06:23:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.664+0000 7fcc0fdc7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcc0808c6d0 0x7fcc0808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.664+0000 7fcc0fdc7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc0808fb80 con 0x7fcc0808fef0 2019-12-05T06:23:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.664+0000 7fcc0fdc7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc080958e0 con 0x7fcc0808f720 2019-12-05T06:23:28.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.664+0000 7fcc0fdc7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc0808df90 con 0x7fcc0808c6d0 2019-12-05T06:23:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.664+0000 7fcc0db63700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc0808fef0 0x7fcc0808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37236/0 (socket says 172.21.15.16:37236) 2019-12-05T06:23:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.665+0000 7fcc0db63700 1 -- 172.21.15.16:0/19210720 learned_addr learned my addr 172.21.15.16:0/19210720 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.665+0000 7fcc0e364700 1 --2- 172.21.15.16:0/19210720 >> v2:172.21.15.76:3300/0 conn(0x7fcc0808c6d0 0x7fcc0808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.665+0000 7fcc0d362700 1 -- 172.21.15.16:0/19210720 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2543239497 0 0) 0x7fcbfc001070 con 0x7fcc0808fef0 2019-12-05T06:23:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.665+0000 7fcc0d362700 1 -- 172.21.15.16:0/19210720 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf0002e80 con 0x7fcc0808fef0 2019-12-05T06:23:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.665+0000 7fcc0d362700 1 -- 172.21.15.16:0/19210720 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2988238894 0 0) 0x7fcbf80009b0 con 0x7fcc0808f720 2019-12-05T06:23:28.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.665+0000 7fcc0d362700 1 -- 172.21.15.16:0/19210720 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcbf0004170 con 0x7fcc0808f720 2019-12-05T06:23:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.665+0000 7fcc0e364700 1 -- 172.21.15.16:0/19210720 >> v1:172.21.15.76:6789/0 conn(0x7fcc0808f720 legacy=0x7fcc081533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.665+0000 7fcc0e364700 1 -- 172.21.15.16:0/19210720 >> v1:172.21.15.16:6789/0 conn(0x7fcc0808fef0 legacy=0x7fcc0808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.665+0000 7fcc0e364700 1 -- 172.21.15.16:0/19210720 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0808e620 con 0x7fcc0808c6d0 2019-12-05T06:23:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.665+0000 7fcc0e364700 1 --2- 172.21.15.16:0/19210720 >> v2:172.21.15.76:3300/0 conn(0x7fcc0808c6d0 0x7fcc0808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc0000b2d0 tx=0x7fcc0000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.666+0000 7fcc0e364700 1 --2- 172.21.15.16:0/19210720 >> v2:172.21.15.76:3300/0 conn(0x7fcc0808c6d0 0x7fcc0808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc0000b2d0 tx=0x7fcc0000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.666+0000 7fcc0e364700 1 --2- 172.21.15.16:0/19210720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0808c6d0 0x7fcc0808cac0 secure :-1 s=READY pgs=487 cs=0 l=1 rx=0x7fcc0000b2d0 tx=0x7fcc0000bae0).ready entity=mon.1 client_cookie=a9de248ec6ee381f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:28.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.666+0000 7fcc0e364700 1 --2- 172.21.15.16:0/19210720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0808c6d0 0x7fcc0808cac0 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7fcc0000b2d0 tx=0x7fcc0000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.666+0000 7fcc0d362700 1 -- 172.21.15.16:0/19210720 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc00000ce0 con 0x7fcc0808c6d0 2019-12-05T06:23:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.666+0000 7fcc0e364700 1 --2- 172.21.15.16:0/19210720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0808c6d0 0x7fcc0808cac0 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7fcc0000b2d0 tx=0x7fcc0000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.666+0000 7fcc0e364700 1 --2- 172.21.15.16:0/19210720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0808c6d0 0x7fcc0808cac0 secure :-1 s=THROTTLE_DONE pgs=487 cs=0 l=1 rx=0x7fcc0000b2d0 tx=0x7fcc0000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.666+0000 7fcc0d362700 1 -- 172.21.15.16:0/19210720 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc00009be0 con 0x7fcc0808c6d0 2019-12-05T06:23:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.666+0000 7fcc0d362700 1 -- 172.21.15.16:0/19210720 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc0000a2b0 con 0x7fcc0808c6d0 2019-12-05T06:23:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.666+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/19210720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0808c6d0 msgr2=0x7fcc0808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.666+0000 7fcc0fdc7700 1 --2- 172.21.15.16:0/19210720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0808c6d0 0x7fcc0808cac0 secure :-1 s=READY pgs=487 cs=0 l=1 rx=0x7fcc0000b2d0 tx=0x7fcc0000bae0).stop 2019-12-05T06:23:28.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.668+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/19210720 shutdown_connections 2019-12-05T06:23:28.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.668+0000 7fcc0fdc7700 1 --2- 172.21.15.16:0/19210720 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcc0808c6d0 0x7fcc0808cac0 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.668+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/19210720 shutdown_connections 2019-12-05T06:23:28.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.668+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/19210720 wait complete. 2019-12-05T06:23:28.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.668+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/19210720 >> 172.21.15.16:0/19210720 conn(0x7fcc08061940 msgr2=0x7fcc08096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.672+0000 7fcc0fdc7700 1 Processor -- start 2019-12-05T06:23:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.672+0000 7fcc0fdc7700 1 -- start start 2019-12-05T06:23:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.673+0000 7fcc0fdc7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc0814d5d0 0x7fcc0814d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.674+0000 7fcc0fdc7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc0809e2c0 0x7fcc0809d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.674+0000 7fcc0fdc7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcc0809d8e0 0x7fcc08098180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:28.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.674+0000 7fcc0fdc7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc0812a9c0 con 0x7fcc0809e2c0 2019-12-05T06:23:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.674+0000 7fcc0fdc7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc0812ac90 con 0x7fcc0809d8e0 2019-12-05T06:23:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.674+0000 7fcc0fdc7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0806eb60 con 0x7fcc0814d5d0 2019-12-05T06:23:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.674+0000 7fcc0db63700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc0809e2c0 0x7fcc0809d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.674+0000 7fcc0db63700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc0809e2c0 0x7fcc0809d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52072/0 (socket says 172.21.15.16:52072) 2019-12-05T06:23:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.674+0000 7fcc0e364700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc0814d5d0 0x7fcc0814d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.674+0000 7fcc0db63700 1 -- 172.21.15.16:0/1592502224 learned_addr learned my addr 172.21.15.16:0/1592502224 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.674+0000 7fcc0eb65700 1 --2- 172.21.15.16:0/1592502224 >> v2:172.21.15.76:3300/0 conn(0x7fcc0809d8e0 0x7fcc08098180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 -- 172.21.15.16:0/1592502224 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc0814d5d0 msgr2=0x7fcc0814d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc0814d5d0 0x7fcc0814d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 -- 172.21.15.16:0/1592502224 >> v2:172.21.15.76:3300/0 conn(0x7fcc0809d8e0 msgr2=0x7fcc08098180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> v2:172.21.15.76:3300/0 conn(0x7fcc0809d8e0 0x7fcc08098180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 -- 172.21.15.16:0/1592502224 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0812ba80 con 0x7fcc0809e2c0 2019-12-05T06:23:28.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0e364700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc0814d5d0 0x7fcc0814d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:23:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> v2:172.21.15.16:3300/0 conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> v2:172.21.15.16:3300/0 conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=READY pgs=665 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=THROTTLE_DONE pgs=665 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=THROTTLE_DONE pgs=665 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=THROTTLE_DONE pgs=665 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc06ffd700 1 -- 172.21.15.16:0/1592502224 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbfc000960 con 0x7fcc0809e2c0 2019-12-05T06:23:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc06ffd700 1 -- 172.21.15.16:0/1592502224 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbfc003fe0 con 0x7fcc0809e2c0 2019-12-05T06:23:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc06ffd700 1 -- 172.21.15.16:0/1592502224 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbfc004630 con 0x7fcc0809e2c0 2019-12-05T06:23:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/1592502224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0812ba80 con 0x7fcc0809e2c0 2019-12-05T06:23:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.675+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/1592502224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0812ba80 con 0x7fcc0809e2c0 2019-12-05T06:23:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.676+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=THROTTLE_DONE pgs=665 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.677+0000 7fcc06ffd700 1 -- 172.21.15.16:0/1592502224 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcbfc0041d0 con 0x7fcc0809e2c0 2019-12-05T06:23:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.677+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=THROTTLE_DONE pgs=665 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.678+0000 7fcc06ffd700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcbe8023630 0x7fcbe8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.678+0000 7fcc06ffd700 1 -- 172.21.15.16:0/1592502224 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fcbfc045d00 con 0x7fcc0809e2c0 2019-12-05T06:23:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.678+0000 7fcc0e364700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcbe8023630 0x7fcbe8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:28.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.679+0000 7fcc0e364700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcbe8023630 0x7fcbe8025ac0 crc :-1 s=READY pgs=298 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:28.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.694+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/1592502224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0812ba80 con 0x7fcc0809e2c0 2019-12-05T06:23:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.701+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=THROTTLE_DONE pgs=665 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.701+0000 7fcc06ffd700 1 -- 172.21.15.16:0/1592502224 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcbfc015b20 con 0x7fcc0809e2c0 2019-12-05T06:23:28.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.942+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/1592502224 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc0812ba80 con 0x7fcc0809e2c0 2019-12-05T06:23:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.943+0000 7fcc0db63700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=THROTTLE_DONE pgs=665 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:28.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.943+0000 7fcc06ffd700 1 -- 172.21.15.16:0/1592502224 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+1769 (secure 0 0 0) 0x7fcbfc015b20 con 0x7fcc0809e2c0 2019-12-05T06:23:28.932 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 86 2019-12-05T06:23:28.932 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:28.932 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.949+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/1592502224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcbe8023630 msgr2=0x7fcbe8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.949+0000 7fcc0fdc7700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcbe8023630 0x7fcbe8025ac0 crc :-1 s=READY pgs=298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.949+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 msgr2=0x7fcc0809d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:28.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.949+0000 7fcc0fdc7700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 0x7fcc0809d3a0 secure :-1 s=READY pgs=665 cs=0 l=1 rx=0x7fcbfc010430 tx=0x7fcbfc0102b0).stop 2019-12-05T06:23:28.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.949+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/1592502224 shutdown_connections 2019-12-05T06:23:28.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.949+0000 7fcc0fdc7700 1 --2- 172.21.15.16:0/1592502224 >> v2:172.21.15.76:3300/0 conn(0x7fcc0809d8e0 0x7fcc08098180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.949+0000 7fcc0fdc7700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcbe8023630 0x7fcbe8025ac0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.949+0000 7fcc0fdc7700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc0809e2c0 0x7fcc0809d3a0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.949+0000 7fcc0fdc7700 1 --2- 172.21.15.16:0/1592502224 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc0814d5d0 0x7fcc0814d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:28.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.949+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/1592502224 shutdown_connections 2019-12-05T06:23:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.950+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/1592502224 wait complete. 2019-12-05T06:23:28.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:28.950+0000 7fcc0fdc7700 1 -- 172.21.15.16:0/1592502224 >> 172.21.15.16:0/1592502224 conn(0x7fcc0806d260 msgr2=0x7fcc0809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:28.952 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5776 2019-12-05T06:23:28.952 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2019-12-05T06:23:33.044 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:23:33.049 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:23:35.953 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:23:35.953 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:23:35.955 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:23:35.959 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:23:35.960 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:36.072 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:23:36.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.156+0000 7f4cd6959700 1 Processor -- start 2019-12-05T06:23:36.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.156+0000 7f4cd6959700 1 -- start start 2019-12-05T06:23:36.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.157+0000 7f4cd6959700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cd008d2c0 0x7f4cd008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:36.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.157+0000 7f4cd6959700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4cd008fef0 0x7f4cd008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:36.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.157+0000 7f4cd6959700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd00958e0 con 0x7f4cd008c6d0 2019-12-05T06:23:36.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.157+0000 7f4cd6959700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd008df90 con 0x7f4cd008d2c0 2019-12-05T06:23:36.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.157+0000 7f4cd6959700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd00a3f70 con 0x7f4cd008fef0 2019-12-05T06:23:36.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.157+0000 7f4ccffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cd008d2c0 0x7f4cd008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:36.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.157+0000 7f4ccffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cd008d2c0 0x7f4cd008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52124/0 (socket says 172.21.15.16:52124) 2019-12-05T06:23:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.157+0000 7f4ccffff700 1 -- 172.21.15.16:0/3107056365 learned_addr learned my addr 172.21.15.16:0/3107056365 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.157+0000 7f4cd56f7700 1 --2- 172.21.15.16:0/3107056365 >> v2:172.21.15.76:3300/0 conn(0x7f4cd008fef0 0x7f4cd008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccffff700 1 -- 172.21.15.16:0/3107056365 >> v2:172.21.15.76:3300/0 conn(0x7f4cd008fef0 msgr2=0x7f4cd008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccffff700 1 --2- 172.21.15.16:0/3107056365 >> v2:172.21.15.76:3300/0 conn(0x7f4cd008fef0 0x7f4cd008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccffff700 1 -- 172.21.15.16:0/3107056365 >> v1:172.21.15.76:6789/0 conn(0x7f4cd008c6d0 legacy=0x7f4cd008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:36.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccffff700 1 -- 172.21.15.16:0/3107056365 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd008e620 con 0x7f4cd008d2c0 2019-12-05T06:23:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccffff700 1 --2- 172.21.15.16:0/3107056365 >> v2:172.21.15.16:3300/0 conn(0x7f4cd008d2c0 0x7f4cd008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4cc000f740 tx=0x7f4cc000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccffff700 1 --2- 172.21.15.16:0/3107056365 >> v2:172.21.15.16:3300/0 conn(0x7f4cd008d2c0 0x7f4cd008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4cc000f740 tx=0x7f4cc000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccffff700 1 --2- 172.21.15.16:0/3107056365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd008d2c0 0x7f4cd008ef20 secure :-1 s=READY pgs=670 cs=0 l=1 rx=0x7f4cc000f740 tx=0x7f4cc000f570).ready entity=mon.0 client_cookie=d10c27dc25ec8083 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccffff700 1 --2- 172.21.15.16:0/3107056365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd008d2c0 0x7f4cd008ef20 secure :-1 s=THROTTLE_DONE pgs=670 cs=0 l=1 rx=0x7f4cc000f740 tx=0x7f4cc000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccffff700 1 --2- 172.21.15.16:0/3107056365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd008d2c0 0x7f4cd008ef20 secure :-1 s=THROTTLE_DONE pgs=670 cs=0 l=1 rx=0x7f4cc000f740 tx=0x7f4cc000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccffff700 1 --2- 172.21.15.16:0/3107056365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd008d2c0 0x7f4cd008ef20 secure :-1 s=THROTTLE_DONE pgs=670 cs=0 l=1 rx=0x7f4cc000f740 tx=0x7f4cc000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccf7fe700 1 -- 172.21.15.16:0/3107056365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cc0000960 con 0x7f4cd008d2c0 2019-12-05T06:23:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccf7fe700 1 -- 172.21.15.16:0/3107056365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc0000c90 con 0x7f4cd008d2c0 2019-12-05T06:23:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4ccf7fe700 1 -- 172.21.15.16:0/3107056365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cc00013e0 con 0x7f4cd008d2c0 2019-12-05T06:23:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4cd6959700 1 -- 172.21.15.16:0/3107056365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd008d2c0 msgr2=0x7f4cd008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.158+0000 7f4cd6959700 1 --2- 172.21.15.16:0/3107056365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd008d2c0 0x7f4cd008ef20 secure :-1 s=READY pgs=670 cs=0 l=1 rx=0x7f4cc000f740 tx=0x7f4cc000f570).stop 2019-12-05T06:23:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.159+0000 7f4cd6959700 1 -- 172.21.15.16:0/3107056365 shutdown_connections 2019-12-05T06:23:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.159+0000 7f4cd6959700 1 --2- 172.21.15.16:0/3107056365 >> v2:172.21.15.76:3300/0 conn(0x7f4cd008fef0 0x7f4cd008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:36.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.159+0000 7f4cd6959700 1 --2- 172.21.15.16:0/3107056365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd008d2c0 0x7f4cd008ef20 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:36.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.159+0000 7f4cd6959700 1 -- 172.21.15.16:0/3107056365 shutdown_connections 2019-12-05T06:23:36.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.159+0000 7f4cd6959700 1 -- 172.21.15.16:0/3107056365 wait complete. 2019-12-05T06:23:36.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.159+0000 7f4cd6959700 1 -- 172.21.15.16:0/3107056365 >> 172.21.15.16:0/3107056365 conn(0x7f4cd0061940 msgr2=0x7f4cd0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.162+0000 7f4cd6959700 1 Processor -- start 2019-12-05T06:23:36.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.162+0000 7f4cd6959700 1 -- start start 2019-12-05T06:23:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd6959700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cd014c3a0 0x7f4cd009d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd6959700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4cd009e580 0x7f4cd019d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd6959700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd0153160 con 0x7f4cd006e210 2019-12-05T06:23:36.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd6959700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd009c620 con 0x7f4cd014c3a0 2019-12-05T06:23:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd6959700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd008ada0 con 0x7f4cd009e580 2019-12-05T06:23:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd4ef6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cd014c3a0 0x7f4cd009d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd4ef6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4cd014c3a0 0x7f4cd009d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52130/0 (socket says 172.21.15.16:52130) 2019-12-05T06:23:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd4ef6700 1 -- 172.21.15.16:0/1570954021 learned_addr learned my addr 172.21.15.16:0/1570954021 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:36.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd56f7700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4cd009e580 0x7f4cd019d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd4ef6700 1 -- 172.21.15.16:0/1570954021 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4cd009e580 msgr2=0x7f4cd019d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4cd009e580 0x7f4cd019d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.163+0000 7f4cd4ef6700 1 -- 172.21.15.16:0/1570954021 >> v1:172.21.15.76:6789/0 conn(0x7f4cd006e210 legacy=0x7f4cd009daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4cd4ef6700 1 -- 172.21.15.16:0/1570954021 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd009ccb0 con 0x7f4cd014c3a0 2019-12-05T06:23:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4ccd7fa700 1 -- 172.21.15.16:0/1570954021 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791546412 0 0) 0x7f4cc0007900 con 0x7f4cd006e210 2019-12-05T06:23:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> v2:172.21.15.16:3300/0 conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> v2:172.21.15.16:3300/0 conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=READY pgs=671 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4ccd7fa700 1 -- 172.21.15.16:0/1570954021 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cc8000ce0 con 0x7f4cd014c3a0 2019-12-05T06:23:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4ccd7fa700 1 -- 172.21.15.16:0/1570954021 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc8001420 con 0x7f4cd014c3a0 2019-12-05T06:23:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4cd6959700 1 -- 172.21.15.16:0/1570954021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd009ccb0 con 0x7f4cd014c3a0 2019-12-05T06:23:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4ccd7fa700 1 -- 172.21.15.16:0/1570954021 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cc8001b70 con 0x7f4cd014c3a0 2019-12-05T06:23:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.164+0000 7f4cd6959700 1 -- 172.21.15.16:0/1570954021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd009ccb0 con 0x7f4cd014c3a0 2019-12-05T06:23:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.165+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.165+0000 7f4ccd7fa700 1 -- 172.21.15.16:0/1570954021 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4cc8001610 con 0x7f4cd014c3a0 2019-12-05T06:23:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.165+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.165+0000 7f4ccd7fa700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4cb4023630 0x7f4cb4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.165+0000 7f4ccd7fa700 1 -- 172.21.15.16:0/1570954021 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f4cc8046150 con 0x7f4cd014c3a0 2019-12-05T06:23:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.165+0000 7f4ccffff700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4cb4023630 0x7f4cb4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:36.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.166+0000 7f4ccffff700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4cb4023630 0x7f4cb4025ac0 crc :-1 s=READY pgs=299 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.171+0000 7f4cd6959700 1 -- 172.21.15.16:0/1570954021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd009ccb0 con 0x7f4cd014c3a0 2019-12-05T06:23:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.175+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.175+0000 7f4ccd7fa700 1 -- 172.21.15.16:0/1570954021 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4cc8007930 con 0x7f4cd014c3a0 2019-12-05T06:23:36.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.374+0000 7f4cd6959700 1 -- 172.21.15.16:0/1570954021 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4cd009ccb0 con 0x7f4cd014c3a0 2019-12-05T06:23:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.375+0000 7f4cd4ef6700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=THROTTLE_DONE pgs=671 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.375+0000 7f4ccd7fa700 1 -- 172.21.15.16:0/1570954021 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+1769 (secure 0 0 0) 0x7f4cc8001610 con 0x7f4cd014c3a0 2019-12-05T06:23:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 86 2019-12-05T06:23:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.377+0000 7f4cd6959700 1 -- 172.21.15.16:0/1570954021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4cb4023630 msgr2=0x7f4cb4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.377+0000 7f4cd6959700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4cb4023630 0x7f4cb4025ac0 crc :-1 s=READY pgs=299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.377+0000 7f4cd6959700 1 -- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 msgr2=0x7f4cd009d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:36.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.377+0000 7f4cd6959700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 0x7f4cd009d5b0 secure :-1 s=READY pgs=671 cs=0 l=1 rx=0x7f4cc8010710 tx=0x7f4cc8010ab0).stop 2019-12-05T06:23:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.378+0000 7f4cd6959700 1 -- 172.21.15.16:0/1570954021 shutdown_connections 2019-12-05T06:23:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.378+0000 7f4cd6959700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4cd009e580 0x7f4cd019d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.378+0000 7f4cd6959700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4cb4023630 0x7f4cb4025ac0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.378+0000 7f4cd6959700 1 --2- 172.21.15.16:0/1570954021 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4cd014c3a0 0x7f4cd009d5b0 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.378+0000 7f4cd6959700 1 -- 172.21.15.16:0/1570954021 shutdown_connections 2019-12-05T06:23:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.378+0000 7f4cd6959700 1 -- 172.21.15.16:0/1570954021 wait complete. 2019-12-05T06:23:36.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:36.378+0000 7f4cd6959700 1 -- 172.21.15.16:0/1570954021 >> 172.21.15.16:0/1570954021 conn(0x7f4cd0098480 msgr2=0x7f4cd0071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:36.367 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:36.367 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:36.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:23:38.376 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.581+0000 7fc0fb6f1700 1 Processor -- start 2019-12-05T06:23:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.581+0000 7fc0fb6f1700 1 -- start start 2019-12-05T06:23:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.582+0000 7fc0fb6f1700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc0f408d2c0 0x7fc0f408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.582+0000 7fc0fb6f1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc0f408fef0 0x7fc0f408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.582+0000 7fc0fb6f1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f40958e0 con 0x7fc0f408c6d0 2019-12-05T06:23:38.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.582+0000 7fc0fb6f1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f408df90 con 0x7fc0f408fef0 2019-12-05T06:23:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.582+0000 7fc0fb6f1700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f40a3f70 con 0x7fc0f408d2c0 2019-12-05T06:23:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.582+0000 7fc0fa48f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc0f408fef0 0x7fc0f408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.582+0000 7fc0f9c8e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc0f408c6d0 0x7fc0f408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:58622/0 (socket says 172.21.15.16:58622) 2019-12-05T06:23:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.582+0000 7fc0fa48f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc0f408fef0 0x7fc0f408f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52156/0 (socket says 172.21.15.16:52156) 2019-12-05T06:23:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.582+0000 7fc0f9c8e700 1 -- 172.21.15.16:0/3009344431 learned_addr learned my addr 172.21.15.16:0/3009344431 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.582+0000 7fc0f948d700 1 --2- 172.21.15.16:0/3009344431 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc0f408d2c0 0x7fc0f408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0fa48f700 1 -- 172.21.15.16:0/3009344431 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc0f408d2c0 msgr2=0x7fc0f408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:38.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0fa48f700 1 --2- 172.21.15.16:0/3009344431 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc0f408d2c0 0x7fc0f408ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:38.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0fa48f700 1 -- 172.21.15.16:0/3009344431 >> v1:172.21.15.76:6789/0 conn(0x7fc0f408c6d0 legacy=0x7fc0f408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0fa48f700 1 -- 172.21.15.16:0/3009344431 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f408e620 con 0x7fc0f408fef0 2019-12-05T06:23:38.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0fa48f700 1 --2- 172.21.15.16:0/3009344431 >> v2:172.21.15.16:3300/0 conn(0x7fc0f408fef0 0x7fc0f408f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc0e400b2d0 tx=0x7fc0e400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0fa48f700 1 --2- 172.21.15.16:0/3009344431 >> v2:172.21.15.16:3300/0 conn(0x7fc0f408fef0 0x7fc0f408f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc0e400b2d0 tx=0x7fc0e400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0fa48f700 1 --2- 172.21.15.16:0/3009344431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f408fef0 0x7fc0f408f460 secure :-1 s=READY pgs=672 cs=0 l=1 rx=0x7fc0e400b2d0 tx=0x7fc0e400ba60).ready entity=mon.0 client_cookie=8649f76cab006027 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:38.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0fa48f700 1 --2- 172.21.15.16:0/3009344431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f408fef0 0x7fc0f408f460 secure :-1 s=THROTTLE_DONE pgs=672 cs=0 l=1 rx=0x7fc0e400b2d0 tx=0x7fc0e400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0fa48f700 1 --2- 172.21.15.16:0/3009344431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f408fef0 0x7fc0f408f460 secure :-1 s=THROTTLE_DONE pgs=672 cs=0 l=1 rx=0x7fc0e400b2d0 tx=0x7fc0e400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0fa48f700 1 --2- 172.21.15.16:0/3009344431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f408fef0 0x7fc0f408f460 secure :-1 s=THROTTLE_DONE pgs=672 cs=0 l=1 rx=0x7fc0e400b2d0 tx=0x7fc0e400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0f8c8c700 1 -- 172.21.15.16:0/3009344431 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0e4009f20 con 0x7fc0f408fef0 2019-12-05T06:23:38.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0f8c8c700 1 -- 172.21.15.16:0/3009344431 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0e4009b40 con 0x7fc0f408fef0 2019-12-05T06:23:38.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.583+0000 7fc0f8c8c700 1 -- 172.21.15.16:0/3009344431 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0e400a500 con 0x7fc0f408fef0 2019-12-05T06:23:38.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.584+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/3009344431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f408fef0 msgr2=0x7fc0f408f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.584+0000 7fc0fb6f1700 1 --2- 172.21.15.16:0/3009344431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f408fef0 0x7fc0f408f460 secure :-1 s=READY pgs=672 cs=0 l=1 rx=0x7fc0e400b2d0 tx=0x7fc0e400ba60).stop 2019-12-05T06:23:38.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.584+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/3009344431 shutdown_connections 2019-12-05T06:23:38.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.584+0000 7fc0fb6f1700 1 --2- 172.21.15.16:0/3009344431 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f408fef0 0x7fc0f408f460 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.584+0000 7fc0fb6f1700 1 --2- 172.21.15.16:0/3009344431 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc0f408d2c0 0x7fc0f408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:23:38.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.584+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/3009344431 shutdown_connections 2019-12-05T06:23:38.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.584+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/3009344431 wait complete. 2019-12-05T06:23:38.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.584+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/3009344431 >> 172.21.15.16:0/3009344431 conn(0x7fc0f4061940 msgr2=0x7fc0f4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:38.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.587+0000 7fc0fb6f1700 1 Processor -- start 2019-12-05T06:23:38.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.587+0000 7fc0fb6f1700 1 -- start start 2019-12-05T06:23:38.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.588+0000 7fc0fb6f1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc0f412f2b0 0x7fc0f414d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:38.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.588+0000 7fc0fb6f1700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc0f412eae0 0x7fc0f419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:38.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.588+0000 7fc0fb6f1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0f412ef40 con 0x7fc0f414d4a0 2019-12-05T06:23:38.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.588+0000 7fc0fb6f1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0f4098f80 con 0x7fc0f412f2b0 2019-12-05T06:23:38.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.588+0000 7fc0fb6f1700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0f4098550 con 0x7fc0f412eae0 2019-12-05T06:23:38.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.588+0000 7fc0f9c8e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc0f414d4a0 0x7fc0f412e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37326/0 (socket says 172.21.15.16:37326) 2019-12-05T06:23:38.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.588+0000 7fc0f9c8e700 1 -- 172.21.15.16:0/2249497951 learned_addr learned my addr 172.21.15.16:0/2249497951 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:38.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> v2:172.21.15.16:3300/0 conn(0x7fc0f412f2b0 0x7fc0f414d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0fa48f700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc0f412eae0 0x7fc0f419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0ea7fc700 1 -- 172.21.15.16:0/2249497951 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830358041 0 0) 0x7fc0ec001080 con 0x7fc0f414d4a0 2019-12-05T06:23:38.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0ea7fc700 1 -- 172.21.15.16:0/2249497951 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0d4002e80 con 0x7fc0f414d4a0 2019-12-05T06:23:38.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0f948d700 1 -- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc0f412eae0 msgr2=0x7fc0f419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc0f412eae0 0x7fc0f419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0f948d700 1 -- 172.21.15.16:0/2249497951 >> v1:172.21.15.16:6789/0 conn(0x7fc0f414d4a0 legacy=0x7fc0f412e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0f948d700 1 -- 172.21.15.16:0/2249497951 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f412d9e0 con 0x7fc0f412f2b0 2019-12-05T06:23:38.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> v2:172.21.15.16:3300/0 conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> v2:172.21.15.16:3300/0 conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=READY pgs=674 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:38.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0ea7fc700 1 -- 172.21.15.16:0/2249497951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0f0000960 con 0x7fc0f412f2b0 2019-12-05T06:23:38.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.589+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.590+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.590+0000 7fc0ea7fc700 1 -- 172.21.15.16:0/2249497951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f0005390 con 0x7fc0f412f2b0 2019-12-05T06:23:38.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.590+0000 7fc0ea7fc700 1 -- 172.21.15.16:0/2249497951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0f0005aa0 con 0x7fc0f412f2b0 2019-12-05T06:23:38.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.590+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/2249497951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0f412d9e0 con 0x7fc0f412f2b0 2019-12-05T06:23:38.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.590+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/2249497951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0f412d9e0 con 0x7fc0f412f2b0 2019-12-05T06:23:38.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.591+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.591+0000 7fc0ea7fc700 1 -- 172.21.15.16:0/2249497951 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc0f0005390 con 0x7fc0f412f2b0 2019-12-05T06:23:38.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.591+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.592+0000 7fc0ea7fc700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 0x7fc0d4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:38.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.592+0000 7fc0ea7fc700 1 -- 172.21.15.16:0/2249497951 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fc0f0045d00 con 0x7fc0f412f2b0 2019-12-05T06:23:38.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.592+0000 7fc0f9c8e700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 0x7fc0d4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.592+0000 7fc0f9c8e700 1 -- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 msgr2=0x7fc0d4025ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:23:38.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.592+0000 7fc0f9c8e700 1 -- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 msgr2=0x7fc0d4025ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:23:38.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.592+0000 7fc0f9c8e700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 0x7fc0d4025ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:23:38.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.592+0000 7fc0f9c8e700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 0x7fc0d4025ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:23:38.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.597+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/2249497951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0f412d9e0 con 0x7fc0f412f2b0 2019-12-05T06:23:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.600+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.600+0000 7fc0ea7fc700 1 -- 172.21.15.16:0/2249497951 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc0f0005180 con 0x7fc0f412f2b0 2019-12-05T06:23:38.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.793+0000 7fc0f9c8e700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 0x7fc0d4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:38.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.793+0000 7fc0f9c8e700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 0x7fc0d4025ae0 crc :-1 s=READY pgs=300 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:38.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.813+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/2249497951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc0f412d9e0 con 0x7fc0f412f2b0 2019-12-05T06:23:38.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.814+0000 7fc0f948d700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:38.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.814+0000 7fc0ea7fc700 1 -- 172.21.15.16:0/2249497951 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 86 v86) v1 ==== 94+0+1769 (secure 0 0 0) 0x7fc0f0005390 con 0x7fc0f412f2b0 2019-12-05T06:23:38.800 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 86 2019-12-05T06:23:38.801 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:38.801 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":86,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:38.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 msgr2=0x7fc0d4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 0x7fc0d4025ae0 crc :-1 s=READY pgs=300 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 msgr2=0x7fc0f414d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:38.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 0x7fc0f414d890 secure :-1 s=READY pgs=674 cs=0 l=1 rx=0x7fc0f0010370 tx=0x7fc0f00101a0).stop 2019-12-05T06:23:38.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/2249497951 shutdown_connections 2019-12-05T06:23:38.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc0d4023670 0x7fc0d4025ae0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc0f412f2b0 0x7fc0f414d890 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 --2- 172.21.15.16:0/2249497951 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc0f412eae0 0x7fc0f419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:38.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/2249497951 shutdown_connections 2019-12-05T06:23:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/2249497951 wait complete. 2019-12-05T06:23:38.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:38.817+0000 7fc0fb6f1700 1 -- 172.21.15.16:0/2249497951 >> 172.21.15.16:0/2249497951 conn(0x7fc0f412d0e0 msgr2=0x7fc0f4135a40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:38.814 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:23:40.815 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:23:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.977+0000 7f94cd989700 1 Processor -- start 2019-12-05T06:23:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.977+0000 7f94cd989700 1 -- start start 2019-12-05T06:23:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.978+0000 7f94cd989700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94c808c6d0 0x7f94c808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:23:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.978+0000 7f94cd989700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c808fb80 con 0x7f94c808fef0 2019-12-05T06:23:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.978+0000 7f94cd989700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c80958e0 con 0x7f94c808f720 2019-12-05T06:23:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.978+0000 7f94cd989700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c808df90 con 0x7f94c808c6d0 2019-12-05T06:23:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.978+0000 7f94c77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94c808c6d0 0x7f94c808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94c808c6d0 0x7f94c808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52182/0 (socket says 172.21.15.16:52182) 2019-12-05T06:23:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 -- 172.21.15.16:0/1206365139 learned_addr learned my addr 172.21.15.16:0/1206365139 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:23:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c6ffd700 1 --1- 172.21.15.16:0/1206365139 >> v1:172.21.15.16:6789/0 conn(0x7f94c808fef0 0x7f94c808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37352/0 (socket says 172.21.15.16:37352) 2019-12-05T06:23:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 -- 172.21.15.16:0/1206365139 >> v1:172.21.15.76:6789/0 conn(0x7f94c808f720 legacy=0x7f94c81533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:23:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 -- 172.21.15.16:0/1206365139 >> v1:172.21.15.16:6789/0 conn(0x7f94c808fef0 legacy=0x7f94c808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 -- 172.21.15.16:0/1206365139 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c808e620 con 0x7f94c808c6d0 2019-12-05T06:23:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 --2- 172.21.15.16:0/1206365139 >> v2:172.21.15.16:3300/0 conn(0x7f94c808c6d0 0x7f94c808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f94c000fac0 tx=0x7f94c000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 --2- 172.21.15.16:0/1206365139 >> v2:172.21.15.16:3300/0 conn(0x7f94c808c6d0 0x7f94c808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f94c000fac0 tx=0x7f94c000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 --2- 172.21.15.16:0/1206365139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c808c6d0 0x7f94c808cac0 secure :-1 s=READY pgs=678 cs=0 l=1 rx=0x7f94c000fac0 tx=0x7f94c000fe60).ready entity=mon.0 client_cookie=5a049ba860fd9b24 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 --2- 172.21.15.16:0/1206365139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c808c6d0 0x7f94c808cac0 secure :-1 s=THROTTLE_DONE pgs=678 cs=0 l=1 rx=0x7f94c000fac0 tx=0x7f94c000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 --2- 172.21.15.16:0/1206365139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c808c6d0 0x7f94c808cac0 secure :-1 s=THROTTLE_DONE pgs=678 cs=0 l=1 rx=0x7f94c000fac0 tx=0x7f94c000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c67fc700 1 -- 172.21.15.16:0/1206365139 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94c0000ce0 con 0x7f94c808c6d0 2019-12-05T06:23:40.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.979+0000 7f94c77fe700 1 --2- 172.21.15.16:0/1206365139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c808c6d0 0x7f94c808cac0 secure :-1 s=THROTTLE_DONE pgs=678 cs=0 l=1 rx=0x7f94c000fac0 tx=0x7f94c000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.980+0000 7f94c67fc700 1 -- 172.21.15.16:0/1206365139 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94c0001010 con 0x7f94c808c6d0 2019-12-05T06:23:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.980+0000 7f94c67fc700 1 -- 172.21.15.16:0/1206365139 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94c00016e0 con 0x7f94c808c6d0 2019-12-05T06:23:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.980+0000 7f94cd989700 1 -- 172.21.15.16:0/1206365139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c808c6d0 msgr2=0x7f94c808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.980+0000 7f94cd989700 1 --2- 172.21.15.16:0/1206365139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c808c6d0 0x7f94c808cac0 secure :-1 s=READY pgs=678 cs=0 l=1 rx=0x7f94c000fac0 tx=0x7f94c000fe60).stop 2019-12-05T06:23:40.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.981+0000 7f94cd989700 1 -- 172.21.15.16:0/1206365139 shutdown_connections 2019-12-05T06:23:40.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.981+0000 7f94cd989700 1 --2- 172.21.15.16:0/1206365139 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94c808c6d0 0x7f94c808cac0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.981+0000 7f94cd989700 1 -- 172.21.15.16:0/1206365139 shutdown_connections 2019-12-05T06:23:40.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.981+0000 7f94cd989700 1 -- 172.21.15.16:0/1206365139 wait complete. 2019-12-05T06:23:40.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.981+0000 7f94cd989700 1 -- 172.21.15.16:0/1206365139 >> 172.21.15.16:0/1206365139 conn(0x7f94c8061940 msgr2=0x7f94c8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:40.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.983+0000 7f94cd989700 1 Processor -- start 2019-12-05T06:23:40.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.984+0000 7f94cd989700 1 -- start start 2019-12-05T06:23:40.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94cd989700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94c814d890 0x7f94c812eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94cd989700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c812c070 con 0x7f94c814d4a0 2019-12-05T06:23:40.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94cd989700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c8154950 con 0x7f94c812f2b0 2019-12-05T06:23:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94cd989700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c8098f80 con 0x7f94c814d890 2019-12-05T06:23:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94c77fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f94c814d4a0 0x7f94c812e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37356/0 (socket says 172.21.15.16:37356) 2019-12-05T06:23:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94c77fe700 1 -- 172.21.15.16:0/1537774077 learned_addr learned my addr 172.21.15.16:0/1537774077 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:23:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94c814d890 0x7f94c812eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652235877 0 0) 0x7f94c0007c40 con 0x7f94c814d4a0 2019-12-05T06:23:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a0002e80 con 0x7f94c814d4a0 2019-12-05T06:23:40.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 18177826 0 0) 0x7f94b8001070 con 0x7f94c812f2b0 2019-12-05T06:23:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.985+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94a00030b0 con 0x7f94c812f2b0 2019-12-05T06:23:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4050775191 0 0) 0x7f94c0008560 con 0x7f94c814d4a0 2019-12-05T06:23:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94a00033c0 con 0x7f94c814d4a0 2019-12-05T06:23:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f94c0007950 con 0x7f94c814d4a0 2019-12-05T06:23:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94c814d890 msgr2=0x7f94c812eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94c814d890 0x7f94c812eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 >> v1:172.21.15.76:6789/0 conn(0x7f94c812f2b0 legacy=0x7f94c8153140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 >> v1:172.21.15.16:6789/0 conn(0x7f94c814d4a0 legacy=0x7f94c812e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94a0005d40 0x7f94a0006130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94a00046f0 0x7f94a0004b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0003940 con 0x7f94a00050d0 2019-12-05T06:23:40.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0003b70 con 0x7f94a00046f0 2019-12-05T06:23:40.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94a0003da0 con 0x7f94a0005d40 2019-12-05T06:23:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94c77fe700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94a0005d40 0x7f94a0006130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.986+0000 7f94c6ffd700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94a00046f0 0x7f94a0004b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94c7fff700 1 -- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94a0005d40 msgr2=0x7f94a0006130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94a0005d40 0x7f94a0006130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94c7fff700 1 -- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94a00046f0 msgr2=0x7f94a0004b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:40.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f94a00046f0 0x7f94a0004b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94c7fff700 1 -- 172.21.15.16:0/1537774077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94a0004050 con 0x7f94a00050d0 2019-12-05T06:23:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94cd989700 1 -- 172.21.15.16:0/1537774077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c812d9e0 con 0x7f94a00050d0 2019-12-05T06:23:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94cd989700 1 -- 172.21.15.16:0/1537774077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c812c070 con 0x7f94a00050d0 2019-12-05T06:23:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=READY pgs=680 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:40.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94b0000960 con 0x7f94a00050d0 2019-12-05T06:23:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.987+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.988+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b0000940 con 0x7f94a00050d0 2019-12-05T06:23:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.988+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.988+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94b0009b90 con 0x7f94a00050d0 2019-12-05T06:23:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.988+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.989+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f94b00097b0 con 0x7f94a00050d0 2019-12-05T06:23:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.989+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.989+0000 7f94bffff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94a002bac0 0x7f94a002def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:23:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.989+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f94b0045c60 con 0x7f94a00050d0 2019-12-05T06:23:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.990+0000 7f94c77fe700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94a002bac0 0x7f94a002def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:23:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.990+0000 7f94c77fe700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94a002bac0 0x7f94a002def0 crc :-1 s=READY pgs=302 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:23:40.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:40.993+0000 7f94cd989700 1 -- 172.21.15.16:0/1537774077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94a4002e60 con 0x7f94a00050d0 2019-12-05T06:23:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.002+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:40.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.002+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f94b00095a0 con 0x7f94a00050d0 2019-12-05T06:23:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.220+0000 7f94cd989700 1 -- 172.21.15.16:0/1537774077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94a4001d10 con 0x7f94a00050d0 2019-12-05T06:23:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.221+0000 7f94c7fff700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=THROTTLE_DONE pgs=680 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:23:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.221+0000 7f94bffff700 1 -- 172.21.15.16:0/1537774077 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f94b00097b0 con 0x7f94a00050d0 2019-12-05T06:23:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2019-12-05T06:23:41.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.223+0000 7f94cd989700 1 -- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94a002bac0 msgr2=0x7f94a002def0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.223+0000 7f94cd989700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94a002bac0 0x7f94a002def0 crc :-1 s=READY pgs=302 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:23:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.223+0000 7f94cd989700 1 -- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 msgr2=0x7f94a00032e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:23:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.223+0000 7f94cd989700 1 --2- 172.21.15.16:0/1537774077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94a00050d0 0x7f94a00032e0 secure :-1 s=READY pgs=680 cs=0 l=1 rx=0x7f94b0010310 tx=0x7f94b00101c0).stop 2019-12-05T06:23:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.223+0000 7f94c7fff700 1 -- 172.21.15.16:0/1537774077 reap_dead start 2019-12-05T06:23:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.223+0000 7f94c7fff700 1 -- 172.21.15.16:0/1537774077 reap_dead start 2019-12-05T06:23:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.223+0000 7f94cd989700 1 -- 172.21.15.16:0/1537774077 shutdown_connections 2019-12-05T06:23:41.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.224+0000 7f94cd989700 1 -- 172.21.15.16:0/1537774077 shutdown_connections 2019-12-05T06:23:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.224+0000 7f94cd989700 1 -- 172.21.15.16:0/1537774077 wait complete. 2019-12-05T06:23:41.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:23:41.224+0000 7f94cd989700 1 -- 172.21.15.16:0/1537774077 >> 172.21.15.16:0/1537774077 conn(0x7f94c812d0e0 msgr2=0x7f94c8135a40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:23:41.216 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:23:41.216 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:23:41.225 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:23:41.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 43 secs before thrashing 2019-12-05T06:24:03.198 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:24:03.203 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:24:24.228 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:24.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.394+0000 7fc33d44c700 1 Processor -- start 2019-12-05T06:24:24.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.394+0000 7fc33d44c700 1 -- start start 2019-12-05T06:24:24.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc33d44c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc33808d2c0 0x7fc33808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:24.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc33d44c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc33808fef0 0x7fc33808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:24.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc33d44c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3380958e0 con 0x7fc33808c6d0 2019-12-05T06:24:24.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc33d44c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33808df90 con 0x7fc33808fef0 2019-12-05T06:24:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc33d44c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3380a3f70 con 0x7fc33808d2c0 2019-12-05T06:24:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc337fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc33808fef0 0x7fc33808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc337fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc33808fef0 0x7fc33808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52408/0 (socket says 172.21.15.16:52408) 2019-12-05T06:24:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc337fff700 1 -- 172.21.15.16:0/4262887052 learned_addr learned my addr 172.21.15.16:0/4262887052 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc336ffd700 1 --2- 172.21.15.16:0/4262887052 >> v2:172.21.15.76:3300/0 conn(0x7fc33808d2c0 0x7fc33808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc337fff700 1 -- 172.21.15.16:0/4262887052 >> v2:172.21.15.76:3300/0 conn(0x7fc33808d2c0 msgr2=0x7fc33808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc337fff700 1 --2- 172.21.15.16:0/4262887052 >> v2:172.21.15.76:3300/0 conn(0x7fc33808d2c0 0x7fc33808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.395+0000 7fc337fff700 1 -- 172.21.15.16:0/4262887052 >> v1:172.21.15.76:6789/0 conn(0x7fc33808c6d0 legacy=0x7fc33808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc337fff700 1 -- 172.21.15.16:0/4262887052 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33808e620 con 0x7fc33808fef0 2019-12-05T06:24:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc337fff700 1 --2- 172.21.15.16:0/4262887052 >> v2:172.21.15.16:3300/0 conn(0x7fc33808fef0 0x7fc33808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc32800f730 tx=0x7fc328007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc337fff700 1 --2- 172.21.15.16:0/4262887052 >> v2:172.21.15.16:3300/0 conn(0x7fc33808fef0 0x7fc33808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc32800f730 tx=0x7fc328007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc337fff700 1 --2- 172.21.15.16:0/4262887052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc33808fef0 0x7fc33808f460 secure :-1 s=READY pgs=685 cs=0 l=1 rx=0x7fc32800f730 tx=0x7fc328007d50).ready entity=mon.0 client_cookie=477de0c01466876e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:24.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc337fff700 1 --2- 172.21.15.16:0/4262887052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc33808fef0 0x7fc33808f460 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fc32800f730 tx=0x7fc328007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc337fff700 1 --2- 172.21.15.16:0/4262887052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc33808fef0 0x7fc33808f460 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fc32800f730 tx=0x7fc328007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc337fff700 1 --2- 172.21.15.16:0/4262887052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc33808fef0 0x7fc33808f460 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fc32800f730 tx=0x7fc328007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc3367fc700 1 -- 172.21.15.16:0/4262887052 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc328000a70 con 0x7fc33808fef0 2019-12-05T06:24:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc3367fc700 1 -- 172.21.15.16:0/4262887052 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3280010e0 con 0x7fc33808fef0 2019-12-05T06:24:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc3367fc700 1 -- 172.21.15.16:0/4262887052 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3280017b0 con 0x7fc33808fef0 2019-12-05T06:24:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc33d44c700 1 -- 172.21.15.16:0/4262887052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc33808fef0 msgr2=0x7fc33808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.396+0000 7fc33d44c700 1 --2- 172.21.15.16:0/4262887052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc33808fef0 0x7fc33808f460 secure :-1 s=READY pgs=685 cs=0 l=1 rx=0x7fc32800f730 tx=0x7fc328007d50).stop 2019-12-05T06:24:24.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.397+0000 7fc33d44c700 1 -- 172.21.15.16:0/4262887052 shutdown_connections 2019-12-05T06:24:24.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.397+0000 7fc33d44c700 1 --2- 172.21.15.16:0/4262887052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc33808fef0 0x7fc33808f460 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.397+0000 7fc33d44c700 1 --2- 172.21.15.16:0/4262887052 >> v2:172.21.15.76:3300/0 conn(0x7fc33808d2c0 0x7fc33808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:24.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.397+0000 7fc33d44c700 1 -- 172.21.15.16:0/4262887052 shutdown_connections 2019-12-05T06:24:24.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.397+0000 7fc33d44c700 1 -- 172.21.15.16:0/4262887052 wait complete. 2019-12-05T06:24:24.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.397+0000 7fc33d44c700 1 -- 172.21.15.16:0/4262887052 >> 172.21.15.16:0/4262887052 conn(0x7fc338061940 msgr2=0x7fc338096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:24.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.400+0000 7fc33d44c700 1 Processor -- start 2019-12-05T06:24:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.400+0000 7fc33d44c700 1 -- start start 2019-12-05T06:24:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc33d44c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc3380a09a0 0x7fc3380845d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc33d44c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc338151ac0 0x7fc338151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc33d44c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc338087c40 con 0x7fc3380855a0 2019-12-05T06:24:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc33d44c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc338088ed0 con 0x7fc338151ac0 2019-12-05T06:24:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc33d44c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3380891a0 con 0x7fc3380a09a0 2019-12-05T06:24:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc338151ac0 0x7fc338151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc338151ac0 0x7fc338151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52416/0 (socket says 172.21.15.16:52416) 2019-12-05T06:24:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 -- 172.21.15.16:0/783349422 learned_addr learned my addr 172.21.15.16:0/783349422 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc3377fe700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc3380a09a0 0x7fc3380845d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 -- 172.21.15.16:0/783349422 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc3380a09a0 msgr2=0x7fc3380845d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc3380a09a0 0x7fc3380845d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 -- 172.21.15.16:0/783349422 >> v1:172.21.15.76:6789/0 conn(0x7fc3380855a0 legacy=0x7fc338084b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 -- 172.21.15.16:0/783349422 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc338089f90 con 0x7fc338151ac0 2019-12-05T06:24:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> v2:172.21.15.16:3300/0 conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> v2:172.21.15.16:3300/0 conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=READY pgs=686 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc32ffff700 1 -- 172.21.15.16:0/783349422 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc328001590 con 0x7fc338151ac0 2019-12-05T06:24:24.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.401+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.402+0000 7fc32ffff700 1 -- 172.21.15.16:0/783349422 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3280011b0 con 0x7fc338151ac0 2019-12-05T06:24:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.402+0000 7fc33d44c700 1 -- 172.21.15.16:0/783349422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc338089f90 con 0x7fc338151ac0 2019-12-05T06:24:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.402+0000 7fc32ffff700 1 -- 172.21.15.16:0/783349422 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc328001b70 con 0x7fc338151ac0 2019-12-05T06:24:24.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.402+0000 7fc33d44c700 1 -- 172.21.15.16:0/783349422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc338089f90 con 0x7fc338151ac0 2019-12-05T06:24:24.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.402+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.403+0000 7fc32ffff700 1 -- 172.21.15.16:0/783349422 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc3280013a0 con 0x7fc338151ac0 2019-12-05T06:24:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.403+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.403+0000 7fc32ffff700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc31c023630 0x7fc31c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.403+0000 7fc32ffff700 1 -- 172.21.15.16:0/783349422 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fc328047e80 con 0x7fc338151ac0 2019-12-05T06:24:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.403+0000 7fc3377fe700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc31c023630 0x7fc31c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.403+0000 7fc3377fe700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc31c023630 0x7fc31c025ac0 crc :-1 s=READY pgs=303 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.408+0000 7fc33d44c700 1 -- 172.21.15.16:0/783349422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc338089f90 con 0x7fc338151ac0 2019-12-05T06:24:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.412+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.412+0000 7fc32ffff700 1 -- 172.21.15.16:0/783349422 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc32803bfb0 con 0x7fc338151ac0 2019-12-05T06:24:24.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.607+0000 7fc33d44c700 1 -- 172.21.15.16:0/783349422 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc338089f90 con 0x7fc338151ac0 2019-12-05T06:24:24.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.608+0000 7fc337fff700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=THROTTLE_DONE pgs=686 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.608+0000 7fc32ffff700 1 -- 172.21.15.16:0/783349422 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fc32803bfb0 con 0x7fc338151ac0 2019-12-05T06:24:24.594 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2019-12-05T06:24:24.594 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:24.594 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:24.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 -- 172.21.15.16:0/783349422 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc31c023630 msgr2=0x7fc31c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc31c023630 0x7fc31c025ac0 crc :-1 s=READY pgs=303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 -- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 msgr2=0x7fc338151f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 0x7fc338151f40 secure :-1 s=READY pgs=686 cs=0 l=1 rx=0x7fc3280102e0 tx=0x7fc328012920).stop 2019-12-05T06:24:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 -- 172.21.15.16:0/783349422 shutdown_connections 2019-12-05T06:24:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc31c023630 0x7fc31c025ac0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc338151ac0 0x7fc338151f40 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 --2- 172.21.15.16:0/783349422 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc3380a09a0 0x7fc3380845d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 -- 172.21.15.16:0/783349422 shutdown_connections 2019-12-05T06:24:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 -- 172.21.15.16:0/783349422 wait complete. 2019-12-05T06:24:24.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.611+0000 7fc33d44c700 1 -- 172.21.15.16:0/783349422 >> 172.21.15.16:0/783349422 conn(0x7fc3380710b0 msgr2=0x7fc33812d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:24.607 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:24:24.608 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:24:24.608 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:24:24.624 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:24:24.624 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:24.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.781+0000 7f5ac7dc4700 1 Processor -- start 2019-12-05T06:24:24.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.781+0000 7f5ac7dc4700 1 -- start start 2019-12-05T06:24:24.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.782+0000 7f5ac7dc4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ac008c6d0 0x7f5ac008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:24.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.782+0000 7f5ac7dc4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5ac008fef0 0x7f5ac008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:24.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.782+0000 7f5ac7dc4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac00958e0 con 0x7f5ac008ef20 2019-12-05T06:24:24.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.782+0000 7f5ac7dc4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ac008df90 con 0x7f5ac008c6d0 2019-12-05T06:24:24.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.782+0000 7f5ac7dc4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac00a3f70 con 0x7f5ac008fef0 2019-12-05T06:24:24.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.782+0000 7f5ac6361700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5ac008c6d0 0x7f5ac008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6b62700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f5ac008ef20 0x7f5ac008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:58894/0 (socket says 172.21.15.16:58894) 2019-12-05T06:24:24.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6b62700 1 -- 172.21.15.16:0/1136686987 learned_addr learned my addr 172.21.15.16:0/1136686987 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:24.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac5b60700 1 --2- 172.21.15.16:0/1136686987 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5ac008fef0 0x7f5ac008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6361700 1 -- 172.21.15.16:0/1136686987 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5ac008fef0 msgr2=0x7f5ac008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:24.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1136686987 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5ac008fef0 0x7f5ac008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:24.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6361700 1 -- 172.21.15.16:0/1136686987 >> v1:172.21.15.76:6789/0 conn(0x7f5ac008ef20 legacy=0x7f5ac008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6361700 1 -- 172.21.15.16:0/1136686987 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ac008d540 con 0x7f5ac008c6d0 2019-12-05T06:24:24.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1136686987 >> v2:172.21.15.16:3300/0 conn(0x7f5ac008c6d0 0x7f5ac008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5abc00af50 tx=0x7f5abc00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1136686987 >> v2:172.21.15.16:3300/0 conn(0x7f5ac008c6d0 0x7f5ac008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5abc00af50 tx=0x7f5abc00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1136686987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac008c6d0 0x7f5ac008cac0 secure :-1 s=READY pgs=687 cs=0 l=1 rx=0x7f5abc00af50 tx=0x7f5abc00b250).ready entity=mon.0 client_cookie=64fad6028c06d40c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:24.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1136686987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac008c6d0 0x7f5ac008cac0 secure :-1 s=THROTTLE_DONE pgs=687 cs=0 l=1 rx=0x7f5abc00af50 tx=0x7f5abc00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac535f700 1 -- 172.21.15.16:0/1136686987 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5abc007210 con 0x7f5ac008c6d0 2019-12-05T06:24:24.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1136686987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac008c6d0 0x7f5ac008cac0 secure :-1 s=THROTTLE_DONE pgs=687 cs=0 l=1 rx=0x7f5abc00af50 tx=0x7f5abc00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1136686987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac008c6d0 0x7f5ac008cac0 secure :-1 s=THROTTLE_DONE pgs=687 cs=0 l=1 rx=0x7f5abc00af50 tx=0x7f5abc00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac535f700 1 -- 172.21.15.16:0/1136686987 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc0099b0 con 0x7f5ac008c6d0 2019-12-05T06:24:24.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac535f700 1 -- 172.21.15.16:0/1136686987 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5abc00a080 con 0x7f5ac008c6d0 2019-12-05T06:24:24.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1136686987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac008c6d0 msgr2=0x7f5ac008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.783+0000 7f5ac7dc4700 1 --2- 172.21.15.16:0/1136686987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac008c6d0 0x7f5ac008cac0 secure :-1 s=READY pgs=687 cs=0 l=1 rx=0x7f5abc00af50 tx=0x7f5abc00b250).stop 2019-12-05T06:24:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.784+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1136686987 shutdown_connections 2019-12-05T06:24:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.784+0000 7f5ac7dc4700 1 --2- 172.21.15.16:0/1136686987 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5ac008fef0 0x7f5ac008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.784+0000 7f5ac7dc4700 1 --2- 172.21.15.16:0/1136686987 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5ac008c6d0 0x7f5ac008cac0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.784+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1136686987 shutdown_connections 2019-12-05T06:24:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.784+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1136686987 wait complete. 2019-12-05T06:24:24.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.784+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1136686987 >> 172.21.15.16:0/1136686987 conn(0x7f5ac0061940 msgr2=0x7f5ac0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.787+0000 7f5ac7dc4700 1 Processor -- start 2019-12-05T06:24:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.787+0000 7f5ac7dc4700 1 -- start start 2019-12-05T06:24:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ac7dc4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5ac0096e90 0x7f5ac009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ac7dc4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac0084b90 con 0x7f5ac0084f30 2019-12-05T06:24:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ac7dc4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ac0098790 con 0x7f5ac009b920 2019-12-05T06:24:24.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ac7dc4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac0099a20 con 0x7f5ac0096e90 2019-12-05T06:24:24.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ac5b60700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5ac0084f30 0x7f5ac0097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37598/0 (socket says 172.21.15.16:37598) 2019-12-05T06:24:24.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ac5b60700 1 -- 172.21.15.16:0/1971995013 learned_addr learned my addr 172.21.15.16:0/1971995013 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:24.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5ac0096e90 0x7f5ac009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554603693 0 0) 0x7f5ab00010b0 con 0x7f5ac0084f30 2019-12-05T06:24:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa8002e80 con 0x7f5ac0084f30 2019-12-05T06:24:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4214924914 0 0) 0x7f5ab8001270 con 0x7f5ac009b920 2019-12-05T06:24:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aa80030b0 con 0x7f5ac009b920 2019-12-05T06:24:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1456568528 0 0) 0x7f5ab00012b0 con 0x7f5ac0084f30 2019-12-05T06:24:24.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5aa8004680 con 0x7f5ac0084f30 2019-12-05T06:24:24.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f5ab0001880 con 0x7f5ac0084f30 2019-12-05T06:24:24.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5ac0096e90 msgr2=0x7f5ac009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5ac0096e90 0x7f5ac009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 >> v1:172.21.15.76:6789/0 conn(0x7f5ac009b920 legacy=0x7f5ac0097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 >> v1:172.21.15.16:6789/0 conn(0x7f5ac0084f30 legacy=0x7f5ac0097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aa8004200 0x7f5aa80086e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5aa8008f40 0x7f5aa800b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa800bb90 con 0x7f5aa8003820 2019-12-05T06:24:24.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa800bdc0 con 0x7f5aa8004200 2019-12-05T06:24:24.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.788+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa800e1c0 con 0x7f5aa8008f40 2019-12-05T06:24:24.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6b62700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5aa8008f40 0x7f5aa800b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac5b60700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aa8004200 0x7f5aa80086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 -- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5aa8008f40 msgr2=0x7f5aa800b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5aa8008f40 0x7f5aa800b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 -- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aa8004200 msgr2=0x7f5aa80086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aa8004200 0x7f5aa80086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 -- 172.21.15.16:0/1971995013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa800e470 con 0x7f5aa8003820 2019-12-05T06:24:24.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1971995013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac009aae0 con 0x7f5aa8003820 2019-12-05T06:24:24.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac5b60700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5aa8004200 0x7f5aa80086e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:24:24.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1971995013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac0084b90 con 0x7f5aa8003820 2019-12-05T06:24:24.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=READY pgs=689 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:24.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=THROTTLE_DONE pgs=689 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=THROTTLE_DONE pgs=689 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=THROTTLE_DONE pgs=689 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5abc000940 con 0x7f5aa8003820 2019-12-05T06:24:24.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5abc009a70 con 0x7f5aa8003820 2019-12-05T06:24:24.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.789+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5abc00a100 con 0x7f5aa8003820 2019-12-05T06:24:24.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.790+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=THROTTLE_DONE pgs=689 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.790+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5abc009c60 con 0x7f5aa8003820 2019-12-05T06:24:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.790+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=THROTTLE_DONE pgs=689 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.791+0000 7f5ab6ffd700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aa802fe90 0x7f5aa80322c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.791+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f5abc045cb0 con 0x7f5aa8003820 2019-12-05T06:24:24.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.791+0000 7f5ac5b60700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aa802fe90 0x7f5aa80322c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:24.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.791+0000 7f5ac5b60700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aa802fe90 0x7f5aa80322c0 crc :-1 s=READY pgs=304 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:24.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.796+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1971995013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac009aae0 con 0x7f5aa8003820 2019-12-05T06:24:24.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.800+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=THROTTLE_DONE pgs=689 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.800+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5abc009670 con 0x7f5aa8003820 2019-12-05T06:24:24.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.996+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1971995013 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ac009aae0 con 0x7f5aa8003820 2019-12-05T06:24:24.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.997+0000 7f5ac6361700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=THROTTLE_DONE pgs=689 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:24.997+0000 7f5ab6ffd700 1 -- 172.21.15.16:0/1971995013 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f5abc009c60 con 0x7f5aa8003820 2019-12-05T06:24:24.983 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2019-12-05T06:24:24.983 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:24.983 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:25.000+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aa802fe90 msgr2=0x7f5aa80322c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:25.000+0000 7f5ac7dc4700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aa802fe90 0x7f5aa80322c0 crc :-1 s=READY pgs=304 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:24.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:25.000+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 msgr2=0x7f5aa8003c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:25.000+0000 7f5ac7dc4700 1 --2- 172.21.15.16:0/1971995013 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5aa8003820 0x7f5aa8003c30 secure :-1 s=READY pgs=689 cs=0 l=1 rx=0x7f5abc0102d0 tx=0x7f5abc010670).stop 2019-12-05T06:24:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:25.000+0000 7f5ac6b62700 1 -- 172.21.15.16:0/1971995013 reap_dead start 2019-12-05T06:24:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:25.000+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1971995013 shutdown_connections 2019-12-05T06:24:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:25.000+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1971995013 shutdown_connections 2019-12-05T06:24:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:25.000+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1971995013 wait complete. 2019-12-05T06:24:24.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:25.000+0000 7f5ac7dc4700 1 -- 172.21.15.16:0/1971995013 >> 172.21.15.16:0/1971995013 conn(0x7f5ac00714f0 msgr2=0x7f5ac009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:24.996 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:24:26.997 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:27.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.196+0000 7f76c1ad4700 1 Processor -- start 2019-12-05T06:24:27.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.196+0000 7f76c1ad4700 1 -- start start 2019-12-05T06:24:27.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.197+0000 7f76c1ad4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc08c6d0 0x7f76bc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:27.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.197+0000 7f76c1ad4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f76bc08fef0 0x7f76bc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:27.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.197+0000 7f76c1ad4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76bc0958e0 con 0x7f76bc08ef20 2019-12-05T06:24:27.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.197+0000 7f76c1ad4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76bc0a3f70 con 0x7f76bc08fef0 2019-12-05T06:24:27.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.197+0000 7f76c1ad4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc08d830 con 0x7f76bc08c6d0 2019-12-05T06:24:27.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.197+0000 7f76c0872700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f76bc08ef20 0x7f76bc08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37620/0 (socket says 172.21.15.16:37620) 2019-12-05T06:24:27.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.197+0000 7f76c0872700 1 -- 172.21.15.16:0/607419668 learned_addr learned my addr 172.21.15.16:0/607419668 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:27.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.197+0000 7f76bbfff700 1 --2- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc08c6d0 0x7f76bc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:27.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.197+0000 7f76bb7fe700 1 --2- 172.21.15.16:0/607419668 >> v2:172.21.15.76:3300/0 conn(0x7f76bc08fef0 0x7f76bc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:27.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.197+0000 7f76baffd700 1 -- 172.21.15.16:0/607419668 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669242744 0 0) 0x7f76b40014d0 con 0x7f76bc08ef20 2019-12-05T06:24:27.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76baffd700 1 -- 172.21.15.16:0/607419668 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f76a4002e80 con 0x7f76bc08ef20 2019-12-05T06:24:27.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bb7fe700 1 -- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc08c6d0 msgr2=0x7f76bc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:27.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76baffd700 1 -- 172.21.15.16:0/607419668 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2734665815 0 0) 0x7f76b40014b0 con 0x7f76bc08ef20 2019-12-05T06:24:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bb7fe700 1 --2- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc08c6d0 0x7f76bc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bb7fe700 1 -- 172.21.15.16:0/607419668 >> v1:172.21.15.16:6789/0 conn(0x7f76bc08ef20 legacy=0x7f76bc08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bb7fe700 1 -- 172.21.15.16:0/607419668 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc08d540 con 0x7f76bc08fef0 2019-12-05T06:24:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bbfff700 1 --2- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc08c6d0 0x7f76bc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bb7fe700 1 --2- 172.21.15.16:0/607419668 >> v2:172.21.15.76:3300/0 conn(0x7f76bc08fef0 0x7f76bc08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f76b000af50 tx=0x7f76b000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bb7fe700 1 --2- 172.21.15.16:0/607419668 >> v2:172.21.15.76:3300/0 conn(0x7f76bc08fef0 0x7f76bc08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f76b000af50 tx=0x7f76b000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bb7fe700 1 --2- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76bc08fef0 0x7f76bc08d000 secure :-1 s=READY pgs=505 cs=0 l=1 rx=0x7f76b000af50 tx=0x7f76b000b760).ready entity=mon.1 client_cookie=a751ea07c872357c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bb7fe700 1 --2- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76bc08fef0 0x7f76bc08d000 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7f76b000af50 tx=0x7f76b000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76baffd700 1 -- 172.21.15.16:0/607419668 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76b0007210 con 0x7f76bc08fef0 2019-12-05T06:24:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bb7fe700 1 --2- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76bc08fef0 0x7f76bc08d000 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7f76b000af50 tx=0x7f76b000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76bb7fe700 1 --2- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76bc08fef0 0x7f76bc08d000 secure :-1 s=THROTTLE_DONE pgs=505 cs=0 l=1 rx=0x7f76b000af50 tx=0x7f76b000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76baffd700 1 -- 172.21.15.16:0/607419668 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b00099b0 con 0x7f76bc08fef0 2019-12-05T06:24:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76baffd700 1 -- 172.21.15.16:0/607419668 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76b000a080 con 0x7f76bc08fef0 2019-12-05T06:24:27.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.198+0000 7f76c1ad4700 1 -- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76bc08fef0 msgr2=0x7f76bc08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.199+0000 7f76c1ad4700 1 --2- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76bc08fef0 0x7f76bc08d000 secure :-1 s=READY pgs=505 cs=0 l=1 rx=0x7f76b000af50 tx=0x7f76b000b760).stop 2019-12-05T06:24:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.199+0000 7f76c1ad4700 1 -- 172.21.15.16:0/607419668 shutdown_connections 2019-12-05T06:24:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.199+0000 7f76c1ad4700 1 --2- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f76bc08fef0 0x7f76bc08d000 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.199+0000 7f76c1ad4700 1 --2- 172.21.15.16:0/607419668 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc08c6d0 0x7f76bc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.199+0000 7f76c1ad4700 1 -- 172.21.15.16:0/607419668 shutdown_connections 2019-12-05T06:24:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.199+0000 7f76c1ad4700 1 -- 172.21.15.16:0/607419668 wait complete. 2019-12-05T06:24:27.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.199+0000 7f76c1ad4700 1 -- 172.21.15.16:0/607419668 >> 172.21.15.16:0/607419668 conn(0x7f76bc061940 msgr2=0x7f76bc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:27.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.202+0000 7f76c1ad4700 1 Processor -- start 2019-12-05T06:24:27.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.202+0000 7f76c1ad4700 1 -- start start 2019-12-05T06:24:27.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.203+0000 7f76c1ad4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f76bc09bbe0 0x7f76bc1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:27.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.203+0000 7f76c1ad4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:27.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.203+0000 7f76c1ad4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76bc084b90 con 0x7f76bc096e90 2019-12-05T06:24:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.203+0000 7f76c1ad4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76bc09a450 con 0x7f76bc09bbe0 2019-12-05T06:24:27.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.203+0000 7f76c1ad4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76bc099a20 con 0x7f76bc084f30 2019-12-05T06:24:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76c0872700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:27.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76bb7fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f76bc09bbe0 0x7f76bc1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76c0872700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:56448/0 (socket says 172.21.15.16:56448) 2019-12-05T06:24:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76c0872700 1 -- 172.21.15.16:0/2013072166 learned_addr learned my addr 172.21.15.16:0/2013072166 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76c0872700 1 -- 172.21.15.16:0/2013072166 >> v2:172.21.15.76:3300/0 conn(0x7f76bc09bbe0 msgr2=0x7f76bc1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> v2:172.21.15.76:3300/0 conn(0x7f76bc09bbe0 0x7f76bc1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76c0872700 1 -- 172.21.15.16:0/2013072166 >> v1:172.21.15.76:6789/0 conn(0x7f76bc096e90 legacy=0x7f76bc09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76c0872700 1 -- 172.21.15.16:0/2013072166 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76bc09aae0 con 0x7f76bc084f30 2019-12-05T06:24:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76b8ff9700 1 -- 172.21.15.16:0/2013072166 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552866816 0 0) 0x7f76ac00db80 con 0x7f76bc096e90 2019-12-05T06:24:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76bb7fe700 1 --2- 172.21.15.16:0/2013072166 >> v2:172.21.15.76:3300/0 conn(0x7f76bc09bbe0 0x7f76bc1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.204+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.205+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.205+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=READY pgs=169 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.205+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.205+0000 7f76b8ff9700 1 -- 172.21.15.16:0/2013072166 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76b4001270 con 0x7f76bc084f30 2019-12-05T06:24:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.205+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.205+0000 7f76c1ad4700 1 -- 172.21.15.16:0/2013072166 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76bc09aae0 con 0x7f76bc084f30 2019-12-05T06:24:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.205+0000 7f76b8ff9700 1 -- 172.21.15.16:0/2013072166 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b4001270 con 0x7f76bc084f30 2019-12-05T06:24:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.205+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.205+0000 7f76b8ff9700 1 -- 172.21.15.16:0/2013072166 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76b40049a0 con 0x7f76bc084f30 2019-12-05T06:24:27.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.205+0000 7f76c1ad4700 1 -- 172.21.15.16:0/2013072166 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76bc09aae0 con 0x7f76bc084f30 2019-12-05T06:24:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.206+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.206+0000 7f76b8ff9700 1 -- 172.21.15.16:0/2013072166 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f76b40045c0 con 0x7f76bc084f30 2019-12-05T06:24:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.206+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.207+0000 7f76b8ff9700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f769c023630 0x7f769c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:27.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.207+0000 7f76b8ff9700 1 -- 172.21.15.16:0/2013072166 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f76b4046180 con 0x7f76bc084f30 2019-12-05T06:24:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.207+0000 7f76bbfff700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f769c023630 0x7f769c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:27.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.207+0000 7f76bbfff700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f769c023630 0x7f769c025ac0 crc :-1 s=READY pgs=305 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:27.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.211+0000 7f76c1ad4700 1 -- 172.21.15.16:0/2013072166 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76bc09aae0 con 0x7f76bc084f30 2019-12-05T06:24:27.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.221+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.221+0000 7f76b8ff9700 1 -- 172.21.15.16:0/2013072166 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f76b4000cc0 con 0x7f76bc084f30 2019-12-05T06:24:27.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.419+0000 7f76c1ad4700 1 -- 172.21.15.16:0/2013072166 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76bc09aae0 con 0x7f76bc084f30 2019-12-05T06:24:27.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.420+0000 7f76c0872700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=THROTTLE_DONE pgs=169 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:27.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.420+0000 7f76b8ff9700 1 -- 172.21.15.16:0/2013072166 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f76b40045c0 con 0x7f76bc084f30 2019-12-05T06:24:27.406 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2019-12-05T06:24:27.406 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:27.408 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.422+0000 7f76c1ad4700 1 -- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f769c023630 msgr2=0x7f769c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.422+0000 7f76c1ad4700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f769c023630 0x7f769c025ac0 crc :-1 s=READY pgs=305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.422+0000 7f76c1ad4700 1 -- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 msgr2=0x7f76bc155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:27.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.422+0000 7f76c1ad4700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 secure :-1 s=READY pgs=169 cs=0 l=1 rx=0x7f76b40107b0 tx=0x7f76b4010b50).stop 2019-12-05T06:24:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.423+0000 7f76c1ad4700 1 -- 172.21.15.16:0/2013072166 shutdown_connections 2019-12-05T06:24:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.423+0000 7f76c1ad4700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f769c023630 0x7f769c025ac0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.423+0000 7f76c1ad4700 1 --2- 172.21.15.16:0/2013072166 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f76bc084f30 0x7f76bc155f30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.423+0000 7f76c1ad4700 1 --2- 172.21.15.16:0/2013072166 >> v2:172.21.15.76:3300/0 conn(0x7f76bc09bbe0 0x7f76bc1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.423+0000 7f76c1ad4700 1 -- 172.21.15.16:0/2013072166 shutdown_connections 2019-12-05T06:24:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.423+0000 7f76c1ad4700 1 -- 172.21.15.16:0/2013072166 wait complete. 2019-12-05T06:24:27.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:27.423+0000 7f76c1ad4700 1 -- 172.21.15.16:0/2013072166 >> 172.21.15.16:0/2013072166 conn(0x7f76bc0714f0 msgr2=0x7f76bc09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:27.419 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:24:29.422 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.582+0000 7f9e7aed6700 1 Processor -- start 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.582+0000 7f9e7aed6700 1 -- start start 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.583+0000 7f9e7aed6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e7408fef0 0x7f9e7408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.583+0000 7f9e7aed6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740958e0 con 0x7f9e7408c6d0 2019-12-05T06:24:29.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.583+0000 7f9e7aed6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e740629e0 con 0x7f9e7408d2c0 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.583+0000 7f9e7aed6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e7408df90 con 0x7f9e7408fef0 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.583+0000 7f9e79c74700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e7408fef0 0x7f9e7408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.583+0000 7f9e79c74700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e7408fef0 0x7f9e7408f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52468/0 (socket says 172.21.15.16:52468) 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.583+0000 7f9e79c74700 1 -- 172.21.15.16:0/3355062899 learned_addr learned my addr 172.21.15.16:0/3355062899 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e79c74700 1 -- 172.21.15.16:0/3355062899 >> v2:172.21.15.16:3300/0 conn(0x7f9e7408fef0 msgr2=0x7f9e7408f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e79c74700 1 -- 172.21.15.16:0/3355062899 >> v2:172.21.15.16:3300/0 conn(0x7f9e7408fef0 msgr2=0x7f9e7408f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:24:29.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> v2:172.21.15.16:3300/0 conn(0x7f9e7408fef0 0x7f9e7408f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> v2:172.21.15.16:3300/0 conn(0x7f9e7408fef0 0x7f9e7408f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3447006113 0 0) 0x7f9e70001100 con 0x7f9e7408c6d0 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e60002e80 con 0x7f9e7408c6d0 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3677050375 0 0) 0x7f9e70001090 con 0x7f9e7408c6d0 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 >> v2:172.21.15.16:3300/0 conn(0x7f9e7408fef0 msgr2=0x7f9e7408f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 --2- 172.21.15.16:0/3355062899 >> v2:172.21.15.16:3300/0 conn(0x7f9e7408fef0 0x7f9e7408f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:29.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 >> v1:172.21.15.76:6789/0 conn(0x7f9e7408d2c0 legacy=0x7f9e7408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7408e620 con 0x7f9e7408c6d0 2019-12-05T06:24:29.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f9e700014a0 con 0x7f9e7408c6d0 2019-12-05T06:24:29.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 >> v1:172.21.15.16:6789/0 conn(0x7f9e7408c6d0 legacy=0x7f9e7408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e60003a70 0x7f9e60005ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:29.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e60006680 0x7f9e60008b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:29.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 0x7f9e6004f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:29.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e6004fff0 con 0x7f9e6004d3a0 2019-12-05T06:24:29.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60050220 con 0x7f9e60003a70 2019-12-05T06:24:29.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e60050450 con 0x7f9e60006680 2019-12-05T06:24:29.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 0x7f9e6004f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e79473700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e60003a70 0x7f9e60005ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.584+0000 7f9e78c72700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e60006680 0x7f9e60008b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 -- 172.21.15.16:0/3355062899 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e60006680 msgr2=0x7f9e60008b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:29.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e60006680 0x7f9e60008b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:29.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 -- 172.21.15.16:0/3355062899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e60003a70 msgr2=0x7f9e60005ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:29.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e60003a70 0x7f9e60005ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 -- 172.21.15.16:0/3355062899 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e60050700 con 0x7f9e6004d3a0 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79473700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e60003a70 0x7f9e60005ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 0x7f9e6004f870 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9e6c00fcd0 tx=0x7f9e6c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 -- 172.21.15.16:0/3355062899 reap_dead start 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 0x7f9e6004f870 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9e6c00fcd0 tx=0x7f9e6c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 0x7f9e6004f870 secure :-1 s=READY pgs=692 cs=0 l=1 rx=0x7f9e6c00fcd0 tx=0x7f9e6c010070).ready entity=mon.0 client_cookie=abb1f0afaf9c1e89 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 0x7f9e6004f870 secure :-1 s=THROTTLE_DONE pgs=692 cs=0 l=1 rx=0x7f9e6c00fcd0 tx=0x7f9e6c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 0x7f9e6004f870 secure :-1 s=THROTTLE_DONE pgs=692 cs=0 l=1 rx=0x7f9e6c00fcd0 tx=0x7f9e6c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e6c000ce0 con 0x7f9e6004d3a0 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e79c74700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 0x7f9e6004f870 secure :-1 s=THROTTLE_DONE pgs=692 cs=0 l=1 rx=0x7f9e6c00fcd0 tx=0x7f9e6c010070).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c007db0 con 0x7f9e6004d3a0 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e6bfff700 1 -- 172.21.15.16:0/3355062899 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e6c001650 con 0x7f9e6004d3a0 2019-12-05T06:24:29.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e7aed6700 1 -- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 msgr2=0x7f9e6004f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.585+0000 7f9e7aed6700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 0x7f9e6004f870 secure :-1 s=READY pgs=692 cs=0 l=1 rx=0x7f9e6c00fcd0 tx=0x7f9e6c010070).stop 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.586+0000 7f9e7aed6700 1 -- 172.21.15.16:0/3355062899 shutdown_connections 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.586+0000 7f9e7aed6700 1 --2- 172.21.15.16:0/3355062899 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e6004d3a0 0x7f9e6004f870 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.586+0000 7f9e7aed6700 1 -- 172.21.15.16:0/3355062899 shutdown_connections 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.586+0000 7f9e7aed6700 1 -- 172.21.15.16:0/3355062899 wait complete. 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.586+0000 7f9e7aed6700 1 -- 172.21.15.16:0/3355062899 >> 172.21.15.16:0/3355062899 conn(0x7f9e74061940 msgr2=0x7f9e74096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:29.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.589+0000 7f9e7aed6700 1 Processor -- start 2019-12-05T06:24:29.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.589+0000 7f9e7aed6700 1 -- start start 2019-12-05T06:24:29.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.590+0000 7f9e7aed6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9e7406e840 0x7f9e74134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:29.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.590+0000 7f9e7aed6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e74128af0 con 0x7f9e74135450 2019-12-05T06:24:29.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.590+0000 7f9e7aed6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e74128600 con 0x7f9e7406dc50 2019-12-05T06:24:29.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.590+0000 7f9e7aed6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e74132830 con 0x7f9e7406e840 2019-12-05T06:24:29.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.590+0000 7f9e79c74700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9e74135450 0x7f9e741349c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37648/0 (socket says 172.21.15.16:37648) 2019-12-05T06:24:29.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.590+0000 7f9e79c74700 1 -- 172.21.15.16:0/965116509 learned_addr learned my addr 172.21.15.16:0/965116509 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:29.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.590+0000 7f9e78c72700 1 --2- 172.21.15.16:0/965116509 >> v2:172.21.15.76:3300/0 conn(0x7f9e7406e840 0x7f9e74134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.590+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2115949017 0 0) 0x7f9e6c007900 con 0x7f9e74135450 2019-12-05T06:24:29.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.590+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e54002e80 con 0x7f9e74135450 2019-12-05T06:24:29.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.590+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3578449905 0 0) 0x7f9e6c007b20 con 0x7f9e74135450 2019-12-05T06:24:29.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e540032e0 con 0x7f9e74135450 2019-12-05T06:24:29.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f9e6c0080f0 con 0x7f9e74135450 2019-12-05T06:24:29.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 >> v2:172.21.15.76:3300/0 conn(0x7f9e7406e840 msgr2=0x7f9e74134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 --2- 172.21.15.16:0/965116509 >> v2:172.21.15.76:3300/0 conn(0x7f9e7406e840 0x7f9e74134480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 >> v1:172.21.15.76:6789/0 conn(0x7f9e7406dc50 legacy=0x7f9e7406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 >> v1:172.21.15.16:6789/0 conn(0x7f9e74135450 legacy=0x7f9e741349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e54003750 0x7f9e54003b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:29.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e540040c0 0x7f9e54004560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:29.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:29.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e540052f0 con 0x7f9e54004bc0 2019-12-05T06:24:29.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5400bec0 con 0x7f9e54003750 2019-12-05T06:24:29.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5400c0f0 con 0x7f9e540040c0 2019-12-05T06:24:29.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e78c72700 1 --2- 172.21.15.16:0/965116509 >> v2:172.21.15.76:3300/0 conn(0x7f9e7406e840 0x7f9e74134480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:29.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79473700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e54003750 0x7f9e54003b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e78c72700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e540040c0 0x7f9e54004560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79c74700 1 -- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e540040c0 msgr2=0x7f9e54004560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e540040c0 0x7f9e54004560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79c74700 1 -- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e54003750 msgr2=0x7f9e54003b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9e54003750 0x7f9e54003b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79c74700 1 -- 172.21.15.16:0/965116509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5400c3a0 con 0x7f9e54004bc0 2019-12-05T06:24:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e7aed6700 1 -- 172.21.15.16:0/965116509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e74133b80 con 0x7f9e54004bc0 2019-12-05T06:24:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e7aed6700 1 -- 172.21.15.16:0/965116509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e74128af0 con 0x7f9e54004bc0 2019-12-05T06:24:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79c74700 1 -- 172.21.15.16:0/965116509 reap_dead start 2019-12-05T06:24:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.591+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=READY pgs=694 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.592+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=THROTTLE_DONE pgs=694 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.592+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e6c007be0 con 0x7f9e54004bc0 2019-12-05T06:24:29.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.592+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=THROTTLE_DONE pgs=694 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.592+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e6c007be0 con 0x7f9e54004bc0 2019-12-05T06:24:29.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.592+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=THROTTLE_DONE pgs=694 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.592+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e6c001650 con 0x7f9e54004bc0 2019-12-05T06:24:29.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.592+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=THROTTLE_DONE pgs=694 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.593+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9e6c001270 con 0x7f9e54004bc0 2019-12-05T06:24:29.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.593+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=THROTTLE_DONE pgs=694 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.593+0000 7f9e69ffb700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e54026350 0x7f9e540287c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:29.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.593+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f9e6c0460e0 con 0x7f9e54004bc0 2019-12-05T06:24:29.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.593+0000 7f9e79473700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e54026350 0x7f9e540287c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:29.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.594+0000 7f9e79473700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e54026350 0x7f9e540287c0 crc :-1 s=READY pgs=306 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:29.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.597+0000 7f9e7aed6700 1 -- 172.21.15.16:0/965116509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e4c002e60 con 0x7f9e54004bc0 2019-12-05T06:24:29.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.601+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=THROTTLE_DONE pgs=694 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.601+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9e6c007900 con 0x7f9e54004bc0 2019-12-05T06:24:29.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.798+0000 7f9e7aed6700 1 -- 172.21.15.16:0/965116509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e4c001d10 con 0x7f9e54004bc0 2019-12-05T06:24:29.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.799+0000 7f9e79c74700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=THROTTLE_DONE pgs=694 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:29.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.799+0000 7f9e69ffb700 1 -- 172.21.15.16:0/965116509 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f9e6c001270 con 0x7f9e54004bc0 2019-12-05T06:24:29.785 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2019-12-05T06:24:29.785 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:29.785 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:29.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.802+0000 7f9e7aed6700 1 -- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e54026350 msgr2=0x7f9e540287c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.802+0000 7f9e7aed6700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e54026350 0x7f9e540287c0 crc :-1 s=READY pgs=306 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.802+0000 7f9e7aed6700 1 -- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 msgr2=0x7f9e5400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.802+0000 7f9e7aed6700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 secure :-1 s=READY pgs=694 cs=0 l=1 rx=0x7f9e6c0107f0 tx=0x7f9e6c0106a0).stop 2019-12-05T06:24:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.802+0000 7f9e7aed6700 1 -- 172.21.15.16:0/965116509 shutdown_connections 2019-12-05T06:24:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.802+0000 7f9e7aed6700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e54026350 0x7f9e540287c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.802+0000 7f9e7aed6700 1 --2- 172.21.15.16:0/965116509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e54004bc0 0x7f9e5400b980 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.802+0000 7f9e7aed6700 1 -- 172.21.15.16:0/965116509 shutdown_connections 2019-12-05T06:24:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.802+0000 7f9e7aed6700 1 -- 172.21.15.16:0/965116509 wait complete. 2019-12-05T06:24:29.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:29.802+0000 7f9e7aed6700 1 -- 172.21.15.16:0/965116509 >> 172.21.15.16:0/965116509 conn(0x7f9e740a2850 msgr2=0x7f9e7406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:29.798 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:24:31.798 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:31.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.978+0000 7fdc150f4700 1 Processor -- start 2019-12-05T06:24:31.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.978+0000 7fdc150f4700 1 -- start start 2019-12-05T06:24:31.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.979+0000 7fdc150f4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc1008c6d0 0x7fdc1008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:31.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.979+0000 7fdc150f4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc1008fef0 0x7fdc1008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:31.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.979+0000 7fdc150f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc100958e0 con 0x7fdc1008ef20 2019-12-05T06:24:31.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.979+0000 7fdc150f4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc1008df90 con 0x7fdc1008c6d0 2019-12-05T06:24:31.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.979+0000 7fdc150f4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc100a3f70 con 0x7fdc1008fef0 2019-12-05T06:24:31.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc1008c6d0 0x7fdc1008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdc1008c6d0 0x7fdc1008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52492/0 (socket says 172.21.15.16:52492) 2019-12-05T06:24:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 -- 172.21.15.16:0/1600124365 learned_addr learned my addr 172.21.15.16:0/1600124365 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0ed9d700 1 --2- 172.21.15.16:0/1600124365 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc1008fef0 0x7fdc1008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 -- 172.21.15.16:0/1600124365 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc1008fef0 msgr2=0x7fdc1008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1600124365 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc1008fef0 0x7fdc1008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 -- 172.21.15.16:0/1600124365 >> v1:172.21.15.76:6789/0 conn(0x7fdc1008ef20 legacy=0x7fdc1008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 -- 172.21.15.16:0/1600124365 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc1008d540 con 0x7fdc1008c6d0 2019-12-05T06:24:31.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1600124365 >> v2:172.21.15.16:3300/0 conn(0x7fdc1008c6d0 0x7fdc1008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdc0000f740 tx=0x7fdc0000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1600124365 >> v2:172.21.15.16:3300/0 conn(0x7fdc1008c6d0 0x7fdc1008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdc0000f740 tx=0x7fdc0000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1600124365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1008c6d0 0x7fdc1008cac0 secure :-1 s=READY pgs=695 cs=0 l=1 rx=0x7fdc0000f740 tx=0x7fdc0000f570).ready entity=mon.0 client_cookie=bd41213f08dc391 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1600124365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1008c6d0 0x7fdc1008cac0 secure :-1 s=THROTTLE_DONE pgs=695 cs=0 l=1 rx=0x7fdc0000f740 tx=0x7fdc0000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1600124365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1008c6d0 0x7fdc1008cac0 secure :-1 s=THROTTLE_DONE pgs=695 cs=0 l=1 rx=0x7fdc0000f740 tx=0x7fdc0000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1600124365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1008c6d0 0x7fdc1008cac0 secure :-1 s=THROTTLE_DONE pgs=695 cs=0 l=1 rx=0x7fdc0000f740 tx=0x7fdc0000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0e59c700 1 -- 172.21.15.16:0/1600124365 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc00000960 con 0x7fdc1008c6d0 2019-12-05T06:24:31.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0e59c700 1 -- 172.21.15.16:0/1600124365 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc00000c90 con 0x7fdc1008c6d0 2019-12-05T06:24:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc0e59c700 1 -- 172.21.15.16:0/1600124365 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc000013a0 con 0x7fdc1008c6d0 2019-12-05T06:24:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc150f4700 1 -- 172.21.15.16:0/1600124365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1008c6d0 msgr2=0x7fdc1008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.980+0000 7fdc150f4700 1 --2- 172.21.15.16:0/1600124365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1008c6d0 0x7fdc1008cac0 secure :-1 s=READY pgs=695 cs=0 l=1 rx=0x7fdc0000f740 tx=0x7fdc0000f570).stop 2019-12-05T06:24:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.981+0000 7fdc150f4700 1 -- 172.21.15.16:0/1600124365 shutdown_connections 2019-12-05T06:24:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.981+0000 7fdc150f4700 1 --2- 172.21.15.16:0/1600124365 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc1008fef0 0x7fdc1008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.981+0000 7fdc150f4700 1 --2- 172.21.15.16:0/1600124365 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdc1008c6d0 0x7fdc1008cac0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:31.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.981+0000 7fdc150f4700 1 -- 172.21.15.16:0/1600124365 shutdown_connections 2019-12-05T06:24:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.981+0000 7fdc150f4700 1 -- 172.21.15.16:0/1600124365 wait complete. 2019-12-05T06:24:31.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.981+0000 7fdc150f4700 1 -- 172.21.15.16:0/1600124365 >> 172.21.15.16:0/1600124365 conn(0x7fdc10061940 msgr2=0x7fdc10096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.984+0000 7fdc150f4700 1 Processor -- start 2019-12-05T06:24:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.985+0000 7fdc150f4700 1 -- start start 2019-12-05T06:24:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdc150f4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc1014ca10 0x7fdc100869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdc150f4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdc100879b0 0x7fdc10086f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:31.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdc150f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc1009e030 con 0x7fdc10087460 2019-12-05T06:24:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdc150f4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc10085020 con 0x7fdc100879b0 2019-12-05T06:24:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdc150f4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc100852f0 con 0x7fdc1014ca10 2019-12-05T06:24:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdc148f3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdc10087460 0x7fdc10152160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37670/0 (socket says 172.21.15.16:37670) 2019-12-05T06:24:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdc148f3700 1 -- 172.21.15.16:0/1695381312 learned_addr learned my addr 172.21.15.16:0/1695381312 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc1014ca10 0x7fdc100869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdc0ed9d700 1 --2- 172.21.15.16:0/1695381312 >> v2:172.21.15.76:3300/0 conn(0x7fdc100879b0 0x7fdc10086f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2026814028 0 0) 0x7fdc08000f10 con 0x7fdc10087460 2019-12-05T06:24:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.986+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdbf0002e80 con 0x7fdc10087460 2019-12-05T06:24:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 795156948 0 0) 0x7fdc08001630 con 0x7fdc10087460 2019-12-05T06:24:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdbf00032e0 con 0x7fdc10087460 2019-12-05T06:24:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fdc08001be0 con 0x7fdc10087460 2019-12-05T06:24:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc1014ca10 msgr2=0x7fdc100869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdc1014ca10 0x7fdc100869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:31.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 >> v2:172.21.15.76:3300/0 conn(0x7fdc100879b0 msgr2=0x7fdc10086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 --2- 172.21.15.16:0/1695381312 >> v2:172.21.15.76:3300/0 conn(0x7fdc100879b0 0x7fdc10086f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 >> v1:172.21.15.16:6789/0 conn(0x7fdc10087460 legacy=0x7fdc10152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdbf0003d60 0x7fdbf0004190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf00047e0 0x7fdbf0004ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf000bb60 con 0x7fdbf00033d0 2019-12-05T06:24:31.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf000bd20 con 0x7fdbf00047e0 2019-12-05T06:24:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdbf000bf50 con 0x7fdbf0003d60 2019-12-05T06:24:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdc148f3700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf00047e0 0x7fdbf0004ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdc0ed9d700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdbf0003d60 0x7fdbf0004190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdc0f59e700 1 -- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdbf0003d60 msgr2=0x7fdbf0004190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdbf0003d60 0x7fdbf0004190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:31.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdc0f59e700 1 -- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf00047e0 msgr2=0x7fdbf0004ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf00047e0 0x7fdbf0004ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdc0f59e700 1 -- 172.21.15.16:0/1695381312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf000c200 con 0x7fdbf00033d0 2019-12-05T06:24:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.987+0000 7fdc150f4700 1 -- 172.21.15.16:0/1695381312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc100860e0 con 0x7fdbf00033d0 2019-12-05T06:24:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdc148f3700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdbf00047e0 0x7fdbf0004ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:24:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdc0ed9d700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdbf0003d60 0x7fdbf0004190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:24:31.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdc150f4700 1 -- 172.21.15.16:0/1695381312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc1009e030 con 0x7fdbf00033d0 2019-12-05T06:24:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=READY pgs=697 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc00000d40 con 0x7fdbf00033d0 2019-12-05T06:24:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc000078b0 con 0x7fdbf00033d0 2019-12-05T06:24:31.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc00000f30 con 0x7fdbf00033d0 2019-12-05T06:24:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.988+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.989+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdc00000b50 con 0x7fdbf00033d0 2019-12-05T06:24:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.989+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.989+0000 7fdbf7fff700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdbf002b9c0 0x7fdbf002ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.989+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fdc00045c60 con 0x7fdbf00033d0 2019-12-05T06:24:31.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.989+0000 7fdc0ed9d700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdbf002b9c0 0x7fdbf002ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:31.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.990+0000 7fdc0ed9d700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdbf002b9c0 0x7fdbf002ddf0 crc :-1 s=READY pgs=307 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:31.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.994+0000 7fdc150f4700 1 -- 172.21.15.16:0/1695381312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbe8002e60 con 0x7fdbf00033d0 2019-12-05T06:24:31.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.997+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:31.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:31.997+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdc00000940 con 0x7fdbf00033d0 2019-12-05T06:24:32.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.195+0000 7fdc150f4700 1 -- 172.21.15.16:0/1695381312 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdbe8001d10 con 0x7fdbf00033d0 2019-12-05T06:24:32.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.195+0000 7fdc0f59e700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:32.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.195+0000 7fdbf7fff700 1 -- 172.21.15.16:0/1695381312 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fdc00000b50 con 0x7fdbf00033d0 2019-12-05T06:24:32.181 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2019-12-05T06:24:32.181 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:32.181 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:32.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.198+0000 7fdc150f4700 1 -- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdbf002b9c0 msgr2=0x7fdbf002ddf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:32.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.198+0000 7fdc150f4700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdbf002b9c0 0x7fdbf002ddf0 crc :-1 s=READY pgs=307 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:32.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.198+0000 7fdc150f4700 1 -- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 msgr2=0x7fdbf0003820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:32.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.198+0000 7fdc150f4700 1 --2- 172.21.15.16:0/1695381312 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdbf00033d0 0x7fdbf0003820 secure :-1 s=READY pgs=697 cs=0 l=1 rx=0x7fdc000103b0 tx=0x7fdc00010750).stop 2019-12-05T06:24:32.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.198+0000 7fdc148f3700 1 -- 172.21.15.16:0/1695381312 reap_dead start 2019-12-05T06:24:32.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.198+0000 7fdc148f3700 1 -- 172.21.15.16:0/1695381312 reap_dead start 2019-12-05T06:24:32.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.199+0000 7fdc150f4700 1 -- 172.21.15.16:0/1695381312 shutdown_connections 2019-12-05T06:24:32.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.199+0000 7fdc150f4700 1 -- 172.21.15.16:0/1695381312 shutdown_connections 2019-12-05T06:24:32.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.199+0000 7fdc150f4700 1 -- 172.21.15.16:0/1695381312 wait complete. 2019-12-05T06:24:32.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:32.199+0000 7fdc150f4700 1 -- 172.21.15.16:0/1695381312 >> 172.21.15.16:0/1695381312 conn(0x7fdc10098480 msgr2=0x7fdc10071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:32.194 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:24:33.334 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:24:33.339 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:24:34.195 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.375+0000 7fe021dc2700 1 Processor -- start 2019-12-05T06:24:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.375+0000 7fe021dc2700 1 -- start start 2019-12-05T06:24:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.375+0000 7fe021dc2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c08c6d0 0x7fe01c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:34.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.375+0000 7fe021dc2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe01c08fef0 0x7fe01c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.376+0000 7fe021dc2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01c0958e0 con 0x7fe01c08ef20 2019-12-05T06:24:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.376+0000 7fe021dc2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01c0a3f70 con 0x7fe01c08fef0 2019-12-05T06:24:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.376+0000 7fe021dc2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c08d830 con 0x7fe01c08c6d0 2019-12-05T06:24:34.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.376+0000 7fe013fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe01c08fef0 0x7fe01c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.376+0000 7fe01bfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c08c6d0 0x7fe01c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.376+0000 7fe013fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe01c08fef0 0x7fe01c08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:38606/0 (socket says 172.21.15.16:38606) 2019-12-05T06:24:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.376+0000 7fe020b60700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fe01c08ef20 0x7fe01c08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:58990/0 (socket says 172.21.15.16:58990) 2019-12-05T06:24:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.376+0000 7fe013fff700 1 -- 172.21.15.16:0/1586638447 learned_addr learned my addr 172.21.15.16:0/1586638447 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe013fff700 1 -- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c08c6d0 msgr2=0x7fe01c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:34.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe013fff700 1 --2- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c08c6d0 0x7fe01c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:34.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe013fff700 1 -- 172.21.15.16:0/1586638447 >> v1:172.21.15.76:6789/0 conn(0x7fe01c08ef20 legacy=0x7fe01c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe013fff700 1 -- 172.21.15.16:0/1586638447 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c08d540 con 0x7fe01c08fef0 2019-12-05T06:24:34.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe01bfff700 1 --2- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c08c6d0 0x7fe01c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:34.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe013fff700 1 --2- 172.21.15.16:0/1586638447 >> v2:172.21.15.76:3300/0 conn(0x7fe01c08fef0 0x7fe01c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe00400f740 tx=0x7fe00400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe013fff700 1 --2- 172.21.15.16:0/1586638447 >> v2:172.21.15.76:3300/0 conn(0x7fe01c08fef0 0x7fe01c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe00400f740 tx=0x7fe00400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe013fff700 1 --2- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe01c08fef0 0x7fe01c08d000 secure :-1 s=READY pgs=511 cs=0 l=1 rx=0x7fe00400f740 tx=0x7fe00400fae0).ready entity=mon.1 client_cookie=fd96b94aee66c6ef server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:34.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe013fff700 1 --2- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe01c08fef0 0x7fe01c08d000 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7fe00400f740 tx=0x7fe00400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe013fff700 1 --2- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe01c08fef0 0x7fe01c08d000 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7fe00400f740 tx=0x7fe00400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe013fff700 1 --2- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe01c08fef0 0x7fe01c08d000 secure :-1 s=THROTTLE_DONE pgs=511 cs=0 l=1 rx=0x7fe00400f740 tx=0x7fe00400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe01b7fe700 1 -- 172.21.15.16:0/1586638447 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe004000960 con 0x7fe01c08fef0 2019-12-05T06:24:34.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe01b7fe700 1 -- 172.21.15.16:0/1586638447 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe004000c90 con 0x7fe01c08fef0 2019-12-05T06:24:34.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.377+0000 7fe01b7fe700 1 -- 172.21.15.16:0/1586638447 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0040012a0 con 0x7fe01c08fef0 2019-12-05T06:24:34.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.378+0000 7fe021dc2700 1 -- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe01c08fef0 msgr2=0x7fe01c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.378+0000 7fe021dc2700 1 --2- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe01c08fef0 0x7fe01c08d000 secure :-1 s=READY pgs=511 cs=0 l=1 rx=0x7fe00400f740 tx=0x7fe00400fae0).stop 2019-12-05T06:24:34.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.378+0000 7fe021dc2700 1 -- 172.21.15.16:0/1586638447 shutdown_connections 2019-12-05T06:24:34.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.378+0000 7fe021dc2700 1 --2- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe01c08fef0 0x7fe01c08d000 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.378+0000 7fe021dc2700 1 --2- 172.21.15.16:0/1586638447 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c08c6d0 0x7fe01c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:34.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.378+0000 7fe021dc2700 1 -- 172.21.15.16:0/1586638447 shutdown_connections 2019-12-05T06:24:34.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.378+0000 7fe021dc2700 1 -- 172.21.15.16:0/1586638447 wait complete. 2019-12-05T06:24:34.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.378+0000 7fe021dc2700 1 -- 172.21.15.16:0/1586638447 >> 172.21.15.16:0/1586638447 conn(0x7fe01c061940 msgr2=0x7fe01c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:34.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.381+0000 7fe021dc2700 1 Processor -- start 2019-12-05T06:24:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.381+0000 7fe021dc2700 1 -- start start 2019-12-05T06:24:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.382+0000 7fe021dc2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.382+0000 7fe021dc2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe01c06dc50 0x7fe01c06e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.382+0000 7fe021dc2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe01c131bf0 con 0x7fe01c12b640 2019-12-05T06:24:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.382+0000 7fe021dc2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe01c128cb0 con 0x7fe01c06dc50 2019-12-05T06:24:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.382+0000 7fe021dc2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe01c128f80 con 0x7fe01c1302f0 2019-12-05T06:24:34.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.382+0000 7fe01bfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.382+0000 7fe020b60700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe01c06dc50 0x7fe01c06e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.382+0000 7fe01bfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:56512/0 (socket says 172.21.15.16:56512) 2019-12-05T06:24:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.382+0000 7fe01bfff700 1 -- 172.21.15.16:0/2998765345 learned_addr learned my addr 172.21.15.16:0/2998765345 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe01bfff700 1 -- 172.21.15.16:0/2998765345 >> v2:172.21.15.76:3300/0 conn(0x7fe01c06dc50 msgr2=0x7fe01c06e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> v2:172.21.15.76:3300/0 conn(0x7fe01c06dc50 0x7fe01c06e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe01bfff700 1 -- 172.21.15.16:0/2998765345 >> v1:172.21.15.76:6789/0 conn(0x7fe01c12b640 legacy=0x7fe01c12abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe01bfff700 1 -- 172.21.15.16:0/2998765345 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe01c129d70 con 0x7fe01c1302f0 2019-12-05T06:24:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=READY pgs=170 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe0197fa700 1 -- 172.21.15.16:0/2998765345 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe00c007510 con 0x7fe01c1302f0 2019-12-05T06:24:34.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe0197fa700 1 -- 172.21.15.16:0/2998765345 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe00c007510 con 0x7fe01c1302f0 2019-12-05T06:24:34.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe021dc2700 1 -- 172.21.15.16:0/2998765345 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe01c129d70 con 0x7fe01c1302f0 2019-12-05T06:24:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.383+0000 7fe0197fa700 1 -- 172.21.15.16:0/2998765345 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe00c00efc0 con 0x7fe01c1302f0 2019-12-05T06:24:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.384+0000 7fe021dc2700 1 -- 172.21.15.16:0/2998765345 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe01c129d70 con 0x7fe01c1302f0 2019-12-05T06:24:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.384+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.385+0000 7fe0197fa700 1 -- 172.21.15.16:0/2998765345 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe00c00e9b0 con 0x7fe01c1302f0 2019-12-05T06:24:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.385+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.385+0000 7fe0197fa700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdff4023630 0x7fdff4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.385+0000 7fe0197fa700 1 -- 172.21.15.16:0/2998765345 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7fe00c045d90 con 0x7fe01c1302f0 2019-12-05T06:24:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.385+0000 7fe013fff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdff4023630 0x7fdff4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.386+0000 7fe013fff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdff4023630 0x7fdff4025ac0 crc :-1 s=READY pgs=308 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:34.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.390+0000 7fe021dc2700 1 -- 172.21.15.16:0/2998765345 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe01c129d70 con 0x7fe01c1302f0 2019-12-05T06:24:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.399+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.399+0000 7fe0197fa700 1 -- 172.21.15.16:0/2998765345 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe00c02adb0 con 0x7fe01c1302f0 2019-12-05T06:24:34.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.603+0000 7fe021dc2700 1 -- 172.21.15.16:0/2998765345 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe01c129d70 con 0x7fe01c1302f0 2019-12-05T06:24:34.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.604+0000 7fe01bfff700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=THROTTLE_DONE pgs=170 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:34.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.604+0000 7fe0197fa700 1 -- 172.21.15.16:0/2998765345 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2105 (secure 0 0 0) 0x7fe00c00e9b0 con 0x7fe01c1302f0 2019-12-05T06:24:34.600 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2019-12-05T06:24:34.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 -- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdff4023630 msgr2=0x7fdff4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdff4023630 0x7fdff4025ac0 crc :-1 s=READY pgs=308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 -- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 msgr2=0x7fe01c12a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:34.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 secure :-1 s=READY pgs=170 cs=0 l=1 rx=0x7fe00c010430 tx=0x7fe00c0102b0).stop 2019-12-05T06:24:34.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 -- 172.21.15.16:0/2998765345 shutdown_connections 2019-12-05T06:24:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdff4023630 0x7fdff4025ac0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 --2- 172.21.15.16:0/2998765345 >> v2:172.21.15.76:3300/0 conn(0x7fe01c06dc50 0x7fe01c06e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 --2- 172.21.15.16:0/2998765345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe01c1302f0 0x7fe01c12a670 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 -- 172.21.15.16:0/2998765345 shutdown_connections 2019-12-05T06:24:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 -- 172.21.15.16:0/2998765345 wait complete. 2019-12-05T06:24:34.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:34.608+0000 7fe021dc2700 1 -- 172.21.15.16:0/2998765345 >> 172.21.15.16:0/2998765345 conn(0x7fe01c0a2850 msgr2=0x7fe01c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:34.603 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:34.603 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:34.604 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:24:36.605 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.834+0000 7f7e98b18700 1 Processor -- start 2019-12-05T06:24:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.834+0000 7f7e98b18700 1 -- start start 2019-12-05T06:24:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e98b18700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7e9408c6d0 0x7f7e9408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:36.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e98b18700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e9408fef0 0x7f7e9408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e98b18700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e9408ef20 0x7f7e9408f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e98b18700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9408df90 con 0x7f7e9408fef0 2019-12-05T06:24:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e98b18700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e940a3f70 con 0x7f7e9408c6d0 2019-12-05T06:24:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e98b18700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9408d830 con 0x7f7e9408ef20 2019-12-05T06:24:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e9259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e9408fef0 0x7f7e9408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e9259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e9408fef0 0x7f7e9408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52530/0 (socket says 172.21.15.16:52530) 2019-12-05T06:24:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e9259c700 1 -- 172.21.15.16:0/233239951 learned_addr learned my addr 172.21.15.16:0/233239951 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e92d9d700 1 --2- 172.21.15.16:0/233239951 >> v2:172.21.15.76:3300/0 conn(0x7f7e9408c6d0 0x7f7e9408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e9359e700 1 --2- 172.21.15.16:0/233239951 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e9408ef20 0x7f7e9408f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e9259c700 1 -- 172.21.15.16:0/233239951 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e9408ef20 msgr2=0x7f7e9408f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e9259c700 1 --2- 172.21.15.16:0/233239951 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e9408ef20 0x7f7e9408f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:36.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e9259c700 1 -- 172.21.15.16:0/233239951 >> v2:172.21.15.76:3300/0 conn(0x7f7e9408c6d0 msgr2=0x7f7e9408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.835+0000 7f7e9259c700 1 --2- 172.21.15.16:0/233239951 >> v2:172.21.15.76:3300/0 conn(0x7f7e9408c6d0 0x7f7e9408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e9259c700 1 -- 172.21.15.16:0/233239951 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9408d540 con 0x7f7e9408fef0 2019-12-05T06:24:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e9259c700 1 --2- 172.21.15.16:0/233239951 >> v2:172.21.15.16:3300/0 conn(0x7f7e9408fef0 0x7f7e9408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7e7c00f6c0 tx=0x7f7e7c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e9259c700 1 --2- 172.21.15.16:0/233239951 >> v2:172.21.15.16:3300/0 conn(0x7f7e9408fef0 0x7f7e9408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7e7c00f6c0 tx=0x7f7e7c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e9259c700 1 --2- 172.21.15.16:0/233239951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e9408fef0 0x7f7e9408d000 secure :-1 s=READY pgs=698 cs=0 l=1 rx=0x7f7e7c00f6c0 tx=0x7f7e7c00f570).ready entity=mon.0 client_cookie=5b5cf2a5790a38a7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e9259c700 1 --2- 172.21.15.16:0/233239951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e9408fef0 0x7f7e9408d000 secure :-1 s=THROTTLE_DONE pgs=698 cs=0 l=1 rx=0x7f7e7c00f6c0 tx=0x7f7e7c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e9259c700 1 --2- 172.21.15.16:0/233239951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e9408fef0 0x7f7e9408d000 secure :-1 s=THROTTLE_DONE pgs=698 cs=0 l=1 rx=0x7f7e7c00f6c0 tx=0x7f7e7c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e91d9b700 1 -- 172.21.15.16:0/233239951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e7c000960 con 0x7f7e9408fef0 2019-12-05T06:24:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e9259c700 1 --2- 172.21.15.16:0/233239951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e9408fef0 0x7f7e9408d000 secure :-1 s=THROTTLE_DONE pgs=698 cs=0 l=1 rx=0x7f7e7c00f6c0 tx=0x7f7e7c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e91d9b700 1 -- 172.21.15.16:0/233239951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e7c000c90 con 0x7f7e9408fef0 2019-12-05T06:24:36.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e91d9b700 1 -- 172.21.15.16:0/233239951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e7c0013a0 con 0x7f7e9408fef0 2019-12-05T06:24:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e98b18700 1 -- 172.21.15.16:0/233239951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e9408fef0 msgr2=0x7f7e9408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.836+0000 7f7e98b18700 1 --2- 172.21.15.16:0/233239951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e9408fef0 0x7f7e9408d000 secure :-1 s=READY pgs=698 cs=0 l=1 rx=0x7f7e7c00f6c0 tx=0x7f7e7c00f570).stop 2019-12-05T06:24:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.838+0000 7f7e98b18700 1 -- 172.21.15.16:0/233239951 shutdown_connections 2019-12-05T06:24:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.838+0000 7f7e98b18700 1 --2- 172.21.15.16:0/233239951 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e9408ef20 0x7f7e9408f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.838+0000 7f7e98b18700 1 --2- 172.21.15.16:0/233239951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e9408fef0 0x7f7e9408d000 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.838+0000 7f7e98b18700 1 --2- 172.21.15.16:0/233239951 >> v2:172.21.15.76:3300/0 conn(0x7f7e9408c6d0 0x7f7e9408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:36.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.838+0000 7f7e98b18700 1 -- 172.21.15.16:0/233239951 shutdown_connections 2019-12-05T06:24:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.838+0000 7f7e98b18700 1 -- 172.21.15.16:0/233239951 wait complete. 2019-12-05T06:24:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.838+0000 7f7e98b18700 1 -- 172.21.15.16:0/233239951 >> 172.21.15.16:0/233239951 conn(0x7f7e94061940 msgr2=0x7f7e94096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.840+0000 7f7e98b18700 1 Processor -- start 2019-12-05T06:24:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.841+0000 7f7e98b18700 1 -- start start 2019-12-05T06:24:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.841+0000 7f7e98b18700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7e941329e0 0x7f7e94151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:36.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.841+0000 7f7e98b18700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.841+0000 7f7e98b18700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e9409c0e0 con 0x7f7e94150c60 2019-12-05T06:24:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.841+0000 7f7e98b18700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e9409dde0 con 0x7f7e941329e0 2019-12-05T06:24:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.841+0000 7f7e98b18700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e9409d3b0 con 0x7f7e941323e0 2019-12-05T06:24:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.841+0000 7f7e92d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e94150c60 0x7f7e9409ed70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37704/0 (socket says 172.21.15.16:37704) 2019-12-05T06:24:36.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.841+0000 7f7e92d9d700 1 -- 172.21.15.16:0/1220851855 learned_addr learned my addr 172.21.15.16:0/1220851855 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e9259c700 1 --2- 172.21.15.16:0/1220851855 >> v2:172.21.15.76:3300/0 conn(0x7f7e941329e0 0x7f7e94151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/1220851855 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346475076 0 0) 0x7f7e8c001430 con 0x7f7e94150c60 2019-12-05T06:24:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/1220851855 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e78002e80 con 0x7f7e94150c60 2019-12-05T06:24:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e9359e700 1 -- 172.21.15.16:0/1220851855 >> v2:172.21.15.76:3300/0 conn(0x7f7e941329e0 msgr2=0x7f7e94151050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> v2:172.21.15.76:3300/0 conn(0x7f7e941329e0 0x7f7e94151050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:36.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e9359e700 1 -- 172.21.15.16:0/1220851855 >> v1:172.21.15.16:6789/0 conn(0x7f7e94150c60 legacy=0x7f7e9409ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/1220851855 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1030380411 0 0) 0x7f7e8c001630 con 0x7f7e94150c60 2019-12-05T06:24:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e9359e700 1 -- 172.21.15.16:0/1220851855 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e9409e470 con 0x7f7e941323e0 2019-12-05T06:24:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e9259c700 1 --2- 172.21.15.16:0/1220851855 >> v2:172.21.15.76:3300/0 conn(0x7f7e941329e0 0x7f7e94151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:24:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.842+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.843+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=READY pgs=171 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.843+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=THROTTLE_DONE pgs=171 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.843+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/1220851855 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e84000a70 con 0x7f7e941323e0 2019-12-05T06:24:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.843+0000 7f7e98b18700 1 -- 172.21.15.16:0/1220851855 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e9409e470 con 0x7f7e941323e0 2019-12-05T06:24:36.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.843+0000 7f7e98b18700 1 -- 172.21.15.16:0/1220851855 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e9409e470 con 0x7f7e941323e0 2019-12-05T06:24:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.843+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=THROTTLE_DONE pgs=171 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.843+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/1220851855 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e84000a70 con 0x7f7e941323e0 2019-12-05T06:24:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.843+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=THROTTLE_DONE pgs=171 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.843+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/1220851855 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e84009ba0 con 0x7f7e941323e0 2019-12-05T06:24:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.844+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=THROTTLE_DONE pgs=171 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.844+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/1220851855 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7e840097c0 con 0x7f7e941323e0 2019-12-05T06:24:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.844+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=THROTTLE_DONE pgs=171 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.844+0000 7f7e8b7fe700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e78023680 0x7f7e78025b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.844+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/1220851855 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f7e84045f30 con 0x7f7e941323e0 2019-12-05T06:24:36.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.845+0000 7f7e92d9d700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e78023680 0x7f7e78025b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.845+0000 7f7e92d9d700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e78023680 0x7f7e78025b10 crc :-1 s=READY pgs=309 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.849+0000 7f7e98b18700 1 -- 172.21.15.16:0/1220851855 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e9409e470 con 0x7f7e941323e0 2019-12-05T06:24:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.858+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=THROTTLE_DONE pgs=171 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:36.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:36.858+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/1220851855 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7e840095b0 con 0x7f7e941323e0 2019-12-05T06:24:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.059+0000 7f7e98b18700 1 -- 172.21.15.16:0/1220851855 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e9409e470 con 0x7f7e941323e0 2019-12-05T06:24:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.060+0000 7f7e9359e700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=THROTTLE_DONE pgs=171 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.060+0000 7f7e8b7fe700 1 -- 172.21.15.16:0/1220851855 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f7e84000a70 con 0x7f7e941323e0 2019-12-05T06:24:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2019-12-05T06:24:37.048 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:37.048 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.066+0000 7f7e98b18700 1 -- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e78023680 msgr2=0x7f7e78025b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.066+0000 7f7e98b18700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e78023680 0x7f7e78025b10 crc :-1 s=READY pgs=309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.066+0000 7f7e98b18700 1 -- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 msgr2=0x7f7e9409b060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.066+0000 7f7e98b18700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 secure :-1 s=READY pgs=171 cs=0 l=1 rx=0x7f7e84010390 tx=0x7f7e84010730).stop 2019-12-05T06:24:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.066+0000 7f7e98b18700 1 -- 172.21.15.16:0/1220851855 shutdown_connections 2019-12-05T06:24:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.066+0000 7f7e98b18700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e78023680 0x7f7e78025b10 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.066+0000 7f7e98b18700 1 --2- 172.21.15.16:0/1220851855 >> v2:172.21.15.76:3300/0 conn(0x7f7e941329e0 0x7f7e94151050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.066+0000 7f7e98b18700 1 --2- 172.21.15.16:0/1220851855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e941323e0 0x7f7e9409b060 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.067+0000 7f7e98b18700 1 -- 172.21.15.16:0/1220851855 shutdown_connections 2019-12-05T06:24:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.067+0000 7f7e98b18700 1 -- 172.21.15.16:0/1220851855 wait complete. 2019-12-05T06:24:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:37.067+0000 7f7e98b18700 1 -- 172.21.15.16:0/1220851855 >> 172.21.15.16:0/1220851855 conn(0x7f7e9409f840 msgr2=0x7f7e940706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:37.062 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:24:39.063 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.296+0000 7f46168f0700 1 Processor -- start 2019-12-05T06:24:39.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.296+0000 7f46168f0700 1 -- start start 2019-12-05T06:24:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.297+0000 7f46168f0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f461008c6d0 0x7f461008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.297+0000 7f46168f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f461008f720 0x7f461008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.297+0000 7f46168f0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46100958e0 con 0x7f461008fef0 2019-12-05T06:24:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.297+0000 7f46168f0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46100a3f70 con 0x7f461008f720 2019-12-05T06:24:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.297+0000 7f46168f0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f461008d830 con 0x7f461008c6d0 2019-12-05T06:24:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f461568e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f461008f720 0x7f461008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f461568e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f461008f720 0x7f461008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52550/0 (socket says 172.21.15.16:52550) 2019-12-05T06:24:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f461568e700 1 -- 172.21.15.16:0/1088490522 learned_addr learned my addr 172.21.15.16:0/1088490522 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f4614e8d700 1 --2- 172.21.15.16:0/1088490522 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f461008c6d0 0x7f461008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f461568e700 1 -- 172.21.15.16:0/1088490522 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f461008c6d0 msgr2=0x7f461008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f461568e700 1 --2- 172.21.15.16:0/1088490522 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f461008c6d0 0x7f461008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f461568e700 1 -- 172.21.15.16:0/1088490522 >> v1:172.21.15.16:6789/0 conn(0x7f461008fef0 legacy=0x7f461008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f460f7fe700 1 -- 172.21.15.16:0/1088490522 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073423412 0 0) 0x7f4604001100 con 0x7f461008fef0 2019-12-05T06:24:39.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f461568e700 1 -- 172.21.15.16:0/1088490522 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f461008fbc0 con 0x7f461008f720 2019-12-05T06:24:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f461568e700 1 --2- 172.21.15.16:0/1088490522 >> v2:172.21.15.16:3300/0 conn(0x7f461008f720 0x7f461008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f460800fab0 tx=0x7f46080080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f461568e700 1 --2- 172.21.15.16:0/1088490522 >> v2:172.21.15.16:3300/0 conn(0x7f461008f720 0x7f461008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f460800fab0 tx=0x7f46080080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.298+0000 7f461568e700 1 --2- 172.21.15.16:0/1088490522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f461008f720 0x7f461008d000 secure :-1 s=READY pgs=701 cs=0 l=1 rx=0x7f460800fab0 tx=0x7f46080080d0).ready entity=mon.0 client_cookie=1b8ee76c97bc0a6e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.299+0000 7f461568e700 1 --2- 172.21.15.16:0/1088490522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f461008f720 0x7f461008d000 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7f460800fab0 tx=0x7f46080080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.299+0000 7f460f7fe700 1 -- 172.21.15.16:0/1088490522 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4608000df0 con 0x7f461008f720 2019-12-05T06:24:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.299+0000 7f461568e700 1 --2- 172.21.15.16:0/1088490522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f461008f720 0x7f461008d000 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7f460800fab0 tx=0x7f46080080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.299+0000 7f460f7fe700 1 -- 172.21.15.16:0/1088490522 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4608000ce0 con 0x7f461008f720 2019-12-05T06:24:39.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.299+0000 7f461568e700 1 --2- 172.21.15.16:0/1088490522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f461008f720 0x7f461008d000 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7f460800fab0 tx=0x7f46080080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.299+0000 7f460f7fe700 1 -- 172.21.15.16:0/1088490522 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4608000ed0 con 0x7f461008f720 2019-12-05T06:24:39.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.299+0000 7f46168f0700 1 -- 172.21.15.16:0/1088490522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f461008f720 msgr2=0x7f461008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.299+0000 7f46168f0700 1 --2- 172.21.15.16:0/1088490522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f461008f720 0x7f461008d000 secure :-1 s=READY pgs=701 cs=0 l=1 rx=0x7f460800fab0 tx=0x7f46080080d0).stop 2019-12-05T06:24:39.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.300+0000 7f46168f0700 1 -- 172.21.15.16:0/1088490522 shutdown_connections 2019-12-05T06:24:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.300+0000 7f46168f0700 1 --2- 172.21.15.16:0/1088490522 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f461008f720 0x7f461008d000 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.300+0000 7f46168f0700 1 --2- 172.21.15.16:0/1088490522 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f461008c6d0 0x7f461008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.300+0000 7f46168f0700 1 -- 172.21.15.16:0/1088490522 shutdown_connections 2019-12-05T06:24:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.300+0000 7f46168f0700 1 -- 172.21.15.16:0/1088490522 wait complete. 2019-12-05T06:24:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.300+0000 7f46168f0700 1 -- 172.21.15.16:0/1088490522 >> 172.21.15.16:0/1088490522 conn(0x7f4610061940 msgr2=0x7f4610096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.303+0000 7f46168f0700 1 Processor -- start 2019-12-05T06:24:39.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.303+0000 7f46168f0700 1 -- start start 2019-12-05T06:24:39.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.304+0000 7f46168f0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f461014ca10 0x7f46100869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:39.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.304+0000 7f46168f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46100879b0 0x7f4610086f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:39.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.304+0000 7f46168f0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4610087460 0x7f4610152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:39.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.304+0000 7f46168f0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4610085a50 con 0x7f46100879b0 2019-12-05T06:24:39.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.304+0000 7f46168f0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4610085020 con 0x7f461014ca10 2019-12-05T06:24:39.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.304+0000 7f46168f0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46100852f0 con 0x7f4610087460 2019-12-05T06:24:39.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.304+0000 7f460ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46100879b0 0x7f4610086f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.305+0000 7f460ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46100879b0 0x7f4610086f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52554/0 (socket says 172.21.15.16:52554) 2019-12-05T06:24:39.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.305+0000 7f460ffff700 1 -- 172.21.15.16:0/628810386 learned_addr learned my addr 172.21.15.16:0/628810386 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:39.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.305+0000 7f461568e700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4610087460 0x7f4610152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.305+0000 7f4614e8d700 1 --2- 172.21.15.16:0/628810386 >> v2:172.21.15.76:3300/0 conn(0x7f461014ca10 0x7f46100869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.305+0000 7f460ffff700 1 -- 172.21.15.16:0/628810386 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4610087460 msgr2=0x7f4610152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.305+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4610087460 0x7f4610152160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.305+0000 7f460ffff700 1 -- 172.21.15.16:0/628810386 >> v2:172.21.15.76:3300/0 conn(0x7f461014ca10 msgr2=0x7f46100869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.305+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> v2:172.21.15.76:3300/0 conn(0x7f461014ca10 0x7f46100869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.305+0000 7f460ffff700 1 -- 172.21.15.16:0/628810386 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46100860e0 con 0x7f46100879b0 2019-12-05T06:24:39.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.305+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> v2:172.21.15.16:3300/0 conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.306+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> v2:172.21.15.16:3300/0 conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.306+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=READY pgs=702 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.306+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.306+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.306+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.306+0000 7f460d7fa700 1 -- 172.21.15.16:0/628810386 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4604000cc0 con 0x7f46100879b0 2019-12-05T06:24:39.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.306+0000 7f460d7fa700 1 -- 172.21.15.16:0/628810386 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f460400ee80 con 0x7f46100879b0 2019-12-05T06:24:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.306+0000 7f46168f0700 1 -- 172.21.15.16:0/628810386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46100860e0 con 0x7f46100879b0 2019-12-05T06:24:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.306+0000 7f460d7fa700 1 -- 172.21.15.16:0/628810386 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f460400f550 con 0x7f46100879b0 2019-12-05T06:24:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.306+0000 7f46168f0700 1 -- 172.21.15.16:0/628810386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46100860e0 con 0x7f46100879b0 2019-12-05T06:24:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.307+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.307+0000 7f460d7fa700 1 -- 172.21.15.16:0/628810386 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f460400f070 con 0x7f46100879b0 2019-12-05T06:24:39.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.307+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.308+0000 7f460d7fa700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f45f0023630 0x7f45f0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:39.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.308+0000 7f460d7fa700 1 -- 172.21.15.16:0/628810386 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f4604045d70 con 0x7f46100879b0 2019-12-05T06:24:39.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.308+0000 7f4614e8d700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f45f0023630 0x7f45f0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:39.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.309+0000 7f4614e8d700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f45f0023630 0x7f45f0025ac0 crc :-1 s=READY pgs=310 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:39.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.310+0000 7f46168f0700 1 -- 172.21.15.16:0/628810386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46100860e0 con 0x7f46100879b0 2019-12-05T06:24:39.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.316+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.316+0000 7f460d7fa700 1 -- 172.21.15.16:0/628810386 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f460400ea80 con 0x7f46100879b0 2019-12-05T06:24:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.514+0000 7f46168f0700 1 -- 172.21.15.16:0/628810386 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46100860e0 con 0x7f46100879b0 2019-12-05T06:24:39.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.514+0000 7f460ffff700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=THROTTLE_DONE pgs=702 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.515+0000 7f460d7fa700 1 -- 172.21.15.16:0/628810386 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f460400f050 con 0x7f46100879b0 2019-12-05T06:24:39.500 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2019-12-05T06:24:39.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:39.503 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 -- 172.21.15.16:0/628810386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f45f0023630 msgr2=0x7f45f0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f45f0023630 0x7f45f0025ac0 crc :-1 s=READY pgs=310 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 -- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 msgr2=0x7f4610086f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 0x7f4610086f20 secure :-1 s=READY pgs=702 cs=0 l=1 rx=0x7f4604010450 tx=0x7f46040102d0).stop 2019-12-05T06:24:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 -- 172.21.15.16:0/628810386 shutdown_connections 2019-12-05T06:24:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4610087460 0x7f4610152160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f45f0023630 0x7f45f0025ac0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 --2- 172.21.15.16:0/628810386 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46100879b0 0x7f4610086f20 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 --2- 172.21.15.16:0/628810386 >> v2:172.21.15.76:3300/0 conn(0x7f461014ca10 0x7f46100869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:39.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 -- 172.21.15.16:0/628810386 shutdown_connections 2019-12-05T06:24:39.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 -- 172.21.15.16:0/628810386 wait complete. 2019-12-05T06:24:39.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:39.517+0000 7f46168f0700 1 -- 172.21.15.16:0/628810386 >> 172.21.15.16:0/628810386 conn(0x7f4610098480 msgr2=0x7f4610071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:39.514 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:24:41.515 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:41.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.724+0000 7f46f3472700 1 Processor -- start 2019-12-05T06:24:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.725+0000 7f46f3472700 1 -- start start 2019-12-05T06:24:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.725+0000 7f46f3472700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f46ec08c6d0 0x7f46ec08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.725+0000 7f46f3472700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46ec08fef0 0x7f46ec08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.725+0000 7f46f3472700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46ec08ef20 0x7f46ec08f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.725+0000 7f46f3472700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec08df90 con 0x7f46ec08fef0 2019-12-05T06:24:41.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.725+0000 7f46f3472700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec0a3f70 con 0x7f46ec08c6d0 2019-12-05T06:24:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.725+0000 7f46f3472700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46ec08d830 con 0x7f46ec08ef20 2019-12-05T06:24:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46ec08fef0 0x7f46ec08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46ec08fef0 0x7f46ec08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52564/0 (socket says 172.21.15.16:52564) 2019-12-05T06:24:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 -- 172.21.15.16:0/4293901253 learned_addr learned my addr 172.21.15.16:0/4293901253 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f2210700 1 --2- 172.21.15.16:0/4293901253 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46ec08ef20 0x7f46ec08f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/4293901253 >> v2:172.21.15.76:3300/0 conn(0x7f46ec08c6d0 0x7f46ec08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 -- 172.21.15.16:0/4293901253 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46ec08ef20 msgr2=0x7f46ec08f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 --2- 172.21.15.16:0/4293901253 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46ec08ef20 0x7f46ec08f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 -- 172.21.15.16:0/4293901253 >> v2:172.21.15.76:3300/0 conn(0x7f46ec08c6d0 msgr2=0x7f46ec08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 --2- 172.21.15.16:0/4293901253 >> v2:172.21.15.76:3300/0 conn(0x7f46ec08c6d0 0x7f46ec08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 -- 172.21.15.16:0/4293901253 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec08d540 con 0x7f46ec08fef0 2019-12-05T06:24:41.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 --2- 172.21.15.16:0/4293901253 >> v2:172.21.15.16:3300/0 conn(0x7f46ec08fef0 0x7f46ec08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f46e800f6c0 tx=0x7f46e800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 --2- 172.21.15.16:0/4293901253 >> v2:172.21.15.16:3300/0 conn(0x7f46ec08fef0 0x7f46ec08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f46e800f6c0 tx=0x7f46e800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 --2- 172.21.15.16:0/4293901253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec08fef0 0x7f46ec08d000 secure :-1 s=READY pgs=703 cs=0 l=1 rx=0x7f46e800f6c0 tx=0x7f46e800f570).ready entity=mon.0 client_cookie=2c0cd886ce841adc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:41.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.726+0000 7f46f120e700 1 --2- 172.21.15.16:0/4293901253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec08fef0 0x7f46ec08d000 secure :-1 s=THROTTLE_DONE pgs=703 cs=0 l=1 rx=0x7f46e800f6c0 tx=0x7f46e800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.727+0000 7f46f120e700 1 --2- 172.21.15.16:0/4293901253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec08fef0 0x7f46ec08d000 secure :-1 s=THROTTLE_DONE pgs=703 cs=0 l=1 rx=0x7f46e800f6c0 tx=0x7f46e800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.727+0000 7f46f120e700 1 --2- 172.21.15.16:0/4293901253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec08fef0 0x7f46ec08d000 secure :-1 s=THROTTLE_DONE pgs=703 cs=0 l=1 rx=0x7f46e800f6c0 tx=0x7f46e800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.727+0000 7f46f0a0d700 1 -- 172.21.15.16:0/4293901253 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46e8000960 con 0x7f46ec08fef0 2019-12-05T06:24:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.727+0000 7f46f0a0d700 1 -- 172.21.15.16:0/4293901253 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e8000c90 con 0x7f46ec08fef0 2019-12-05T06:24:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.727+0000 7f46f0a0d700 1 -- 172.21.15.16:0/4293901253 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46e80013a0 con 0x7f46ec08fef0 2019-12-05T06:24:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.727+0000 7f46f3472700 1 -- 172.21.15.16:0/4293901253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec08fef0 msgr2=0x7f46ec08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.727+0000 7f46f3472700 1 --2- 172.21.15.16:0/4293901253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec08fef0 0x7f46ec08d000 secure :-1 s=READY pgs=703 cs=0 l=1 rx=0x7f46e800f6c0 tx=0x7f46e800f570).stop 2019-12-05T06:24:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.728+0000 7f46f3472700 1 -- 172.21.15.16:0/4293901253 shutdown_connections 2019-12-05T06:24:41.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.728+0000 7f46f3472700 1 --2- 172.21.15.16:0/4293901253 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46ec08ef20 0x7f46ec08f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.728+0000 7f46f3472700 1 --2- 172.21.15.16:0/4293901253 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec08fef0 0x7f46ec08d000 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.728+0000 7f46f3472700 1 --2- 172.21.15.16:0/4293901253 >> v2:172.21.15.76:3300/0 conn(0x7f46ec08c6d0 0x7f46ec08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.728+0000 7f46f3472700 1 -- 172.21.15.16:0/4293901253 shutdown_connections 2019-12-05T06:24:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.728+0000 7f46f3472700 1 -- 172.21.15.16:0/4293901253 wait complete. 2019-12-05T06:24:41.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.728+0000 7f46f3472700 1 -- 172.21.15.16:0/4293901253 >> 172.21.15.16:0/4293901253 conn(0x7f46ec061940 msgr2=0x7f46ec096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.730+0000 7f46f3472700 1 Processor -- start 2019-12-05T06:24:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.730+0000 7f46f3472700 1 -- start start 2019-12-05T06:24:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.731+0000 7f46f3472700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46ec096e90 0x7f46ec09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:41.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.731+0000 7f46f3472700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f46ec09b920 0x7f46ec097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:41.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.731+0000 7f46f3472700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46ec084b90 con 0x7f46ec084f30 2019-12-05T06:24:41.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.731+0000 7f46f3472700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec09a450 con 0x7f46ec096e90 2019-12-05T06:24:41.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.731+0000 7f46f3472700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46ec099a20 con 0x7f46ec09b920 2019-12-05T06:24:41.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.731+0000 7f46f1a0f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46ec096e90 0x7f46ec09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.731+0000 7f46f1a0f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46ec096e90 0x7f46ec09b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52566/0 (socket says 172.21.15.16:52566) 2019-12-05T06:24:41.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.731+0000 7f46f1a0f700 1 -- 172.21.15.16:0/3081775068 learned_addr learned my addr 172.21.15.16:0/3081775068 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f2210700 1 --2- 172.21.15.16:0/3081775068 >> v2:172.21.15.76:3300/0 conn(0x7f46ec09b920 0x7f46ec097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f1a0f700 1 -- 172.21.15.16:0/3081775068 >> v2:172.21.15.76:3300/0 conn(0x7f46ec09b920 msgr2=0x7f46ec097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> v2:172.21.15.76:3300/0 conn(0x7f46ec09b920 0x7f46ec097a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f1a0f700 1 -- 172.21.15.16:0/3081775068 >> v1:172.21.15.76:6789/0 conn(0x7f46ec084f30 legacy=0x7f46ec097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f1a0f700 1 -- 172.21.15.16:0/3081775068 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46ec09aae0 con 0x7f46ec096e90 2019-12-05T06:24:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f2210700 1 --2- 172.21.15.16:0/3081775068 >> v2:172.21.15.76:3300/0 conn(0x7f46ec09b920 0x7f46ec097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:24:41.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> v2:172.21.15.16:3300/0 conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> v2:172.21.15.16:3300/0 conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=READY pgs=704 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46de7fc700 1 -- 172.21.15.16:0/3081775068 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46e0000940 con 0x7f46ec096e90 2019-12-05T06:24:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46de7fc700 1 -- 172.21.15.16:0/3081775068 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46e0000e80 con 0x7f46ec096e90 2019-12-05T06:24:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46de7fc700 1 -- 172.21.15.16:0/3081775068 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f46e00014d0 con 0x7f46ec096e90 2019-12-05T06:24:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.732+0000 7f46f3472700 1 -- 172.21.15.16:0/3081775068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46ec09aae0 con 0x7f46ec096e90 2019-12-05T06:24:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.733+0000 7f46f3472700 1 -- 172.21.15.16:0/3081775068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46ec09aae0 con 0x7f46ec096e90 2019-12-05T06:24:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.733+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.734+0000 7f46de7fc700 1 -- 172.21.15.16:0/3081775068 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f46e0001070 con 0x7f46ec096e90 2019-12-05T06:24:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.734+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.734+0000 7f46de7fc700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f46d4023630 0x7f46d4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:41.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.734+0000 7f46de7fc700 1 -- 172.21.15.16:0/3081775068 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6977+0+0 (secure 0 0 0) 0x7f46e0045e30 con 0x7f46ec096e90 2019-12-05T06:24:41.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.734+0000 7f46f120e700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f46d4023630 0x7f46d4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:41.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.735+0000 7f46f120e700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f46d4023630 0x7f46d4025ac0 crc :-1 s=READY pgs=311 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:41.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.740+0000 7f46f3472700 1 -- 172.21.15.16:0/3081775068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46ec09aae0 con 0x7f46ec096e90 2019-12-05T06:24:41.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.744+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.744+0000 7f46de7fc700 1 -- 172.21.15.16:0/3081775068 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f46e00092e0 con 0x7f46ec096e90 2019-12-05T06:24:41.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.953+0000 7f46f3472700 1 -- 172.21.15.16:0/3081775068 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46ec09aae0 con 0x7f46ec096e90 2019-12-05T06:24:41.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.953+0000 7f46f1a0f700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=THROTTLE_DONE pgs=704 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:41.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.954+0000 7f46de7fc700 1 -- 172.21.15.16:0/3081775068 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 87 v87) v1 ==== 94+0+2105 (secure 0 0 0) 0x7f46e0001920 con 0x7f46ec096e90 2019-12-05T06:24:41.939 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 87 2019-12-05T06:24:41.939 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:41.939 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":87,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5940,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":87}],"filesystems":[{"mdsmap":{"epoch":86,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:23:28.135379+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":60,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5878},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5878":{"gid":5878,"name":"b","rank":0,"incarnation":83,"state":"up:active","state_seq":26,"addr":"172.21.15.76:6835/945852410","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945852410},{"type":"v1","addr":"172.21.15.76:6835","nonce":945852410}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 -- 172.21.15.16:0/3081775068 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f46d4023630 msgr2=0x7f46d4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f46d4023630 0x7f46d4025ac0 crc :-1 s=READY pgs=311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 -- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 msgr2=0x7f46ec09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 0x7f46ec09b3e0 secure :-1 s=READY pgs=704 cs=0 l=1 rx=0x7f46e0010450 tx=0x7f46e00107f0).stop 2019-12-05T06:24:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 -- 172.21.15.16:0/3081775068 shutdown_connections 2019-12-05T06:24:41.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 --2- 172.21.15.16:0/3081775068 >> v2:172.21.15.76:3300/0 conn(0x7f46ec09b920 0x7f46ec097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f46d4023630 0x7f46d4025ac0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 --2- 172.21.15.16:0/3081775068 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46ec096e90 0x7f46ec09b3e0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 -- 172.21.15.16:0/3081775068 shutdown_connections 2019-12-05T06:24:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 -- 172.21.15.16:0/3081775068 wait complete. 2019-12-05T06:24:41.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:41.957+0000 7f46f3472700 1 -- 172.21.15.16:0/3081775068 >> 172.21.15.16:0/3081775068 conn(0x7f46ec0714f0 msgr2=0x7f46ec09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:41.952 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:24:43.953 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:44.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.117+0000 7fb7b4abb700 1 Processor -- start 2019-12-05T06:24:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.117+0000 7fb7b4abb700 1 -- start start 2019-12-05T06:24:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.118+0000 7fb7b4abb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7b008d2c0 0x7fb7b008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.118+0000 7fb7b4abb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b00958e0 con 0x7fb7b008fef0 2019-12-05T06:24:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.118+0000 7fb7b4abb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b00629e0 con 0x7fb7b008c6d0 2019-12-05T06:24:44.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.118+0000 7fb7b4abb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7b008df90 con 0x7fb7b008d2c0 2019-12-05T06:24:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.118+0000 7fb7ae59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7b008d2c0 0x7fb7b008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.118+0000 7fb7ae59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7b008d2c0 0x7fb7b008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52590/0 (socket says 172.21.15.16:52590) 2019-12-05T06:24:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.118+0000 7fb7ae59c700 1 -- 172.21.15.16:0/3645722892 learned_addr learned my addr 172.21.15.16:0/3645722892 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7ae59c700 1 -- 172.21.15.16:0/3645722892 >> v1:172.21.15.76:6789/0 conn(0x7fb7b008c6d0 legacy=0x7fb7b008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:44.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7ae59c700 1 -- 172.21.15.16:0/3645722892 >> v1:172.21.15.16:6789/0 conn(0x7fb7b008fef0 legacy=0x7fb7b008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7ae59c700 1 -- 172.21.15.16:0/3645722892 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b008e620 con 0x7fb7b008d2c0 2019-12-05T06:24:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7ae59c700 1 --2- 172.21.15.16:0/3645722892 >> v2:172.21.15.16:3300/0 conn(0x7fb7b008d2c0 0x7fb7b008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb79800f740 tx=0x7fb79800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7ae59c700 1 --2- 172.21.15.16:0/3645722892 >> v2:172.21.15.16:3300/0 conn(0x7fb7b008d2c0 0x7fb7b008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb79800f740 tx=0x7fb79800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7ae59c700 1 --2- 172.21.15.16:0/3645722892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b008d2c0 0x7fb7b008ef20 secure :-1 s=READY pgs=706 cs=0 l=1 rx=0x7fb79800f740 tx=0x7fb79800fae0).ready entity=mon.0 client_cookie=dd884336c3cc06a0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7ae59c700 1 --2- 172.21.15.16:0/3645722892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b008d2c0 0x7fb7b008ef20 secure :-1 s=THROTTLE_DONE pgs=706 cs=0 l=1 rx=0x7fb79800f740 tx=0x7fb79800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7ae59c700 1 --2- 172.21.15.16:0/3645722892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b008d2c0 0x7fb7b008ef20 secure :-1 s=THROTTLE_DONE pgs=706 cs=0 l=1 rx=0x7fb79800f740 tx=0x7fb79800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7ae59c700 1 --2- 172.21.15.16:0/3645722892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b008d2c0 0x7fb7b008ef20 secure :-1 s=THROTTLE_DONE pgs=706 cs=0 l=1 rx=0x7fb79800f740 tx=0x7fb79800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7add9b700 1 -- 172.21.15.16:0/3645722892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb798000960 con 0x7fb7b008d2c0 2019-12-05T06:24:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7add9b700 1 -- 172.21.15.16:0/3645722892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb798000c90 con 0x7fb7b008d2c0 2019-12-05T06:24:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7add9b700 1 -- 172.21.15.16:0/3645722892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb798001360 con 0x7fb7b008d2c0 2019-12-05T06:24:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7b4abb700 1 -- 172.21.15.16:0/3645722892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b008d2c0 msgr2=0x7fb7b008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.119+0000 7fb7b4abb700 1 --2- 172.21.15.16:0/3645722892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b008d2c0 0x7fb7b008ef20 secure :-1 s=READY pgs=706 cs=0 l=1 rx=0x7fb79800f740 tx=0x7fb79800fae0).stop 2019-12-05T06:24:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.120+0000 7fb7b4abb700 1 -- 172.21.15.16:0/3645722892 shutdown_connections 2019-12-05T06:24:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.120+0000 7fb7b4abb700 1 --2- 172.21.15.16:0/3645722892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b008d2c0 0x7fb7b008ef20 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.120+0000 7fb7b4abb700 1 -- 172.21.15.16:0/3645722892 shutdown_connections 2019-12-05T06:24:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.120+0000 7fb7b4abb700 1 -- 172.21.15.16:0/3645722892 wait complete. 2019-12-05T06:24:44.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.120+0000 7fb7b4abb700 1 -- 172.21.15.16:0/3645722892 >> 172.21.15.16:0/3645722892 conn(0x7fb7b0061940 msgr2=0x7fb7b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.122+0000 7fb7b4abb700 1 Processor -- start 2019-12-05T06:24:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.122+0000 7fb7b4abb700 1 -- start start 2019-12-05T06:24:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.123+0000 7fb7b4abb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7b014ca10 0x7fb7b00869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.123+0000 7fb7b4abb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb7b009ce90 0x7fb7b019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.123+0000 7fb7b4abb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7b009e030 con 0x7fb7b00879b0 2019-12-05T06:24:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.123+0000 7fb7b4abb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7b0085a50 con 0x7fb7b014ca10 2019-12-05T06:24:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.123+0000 7fb7b4abb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7b0085020 con 0x7fb7b009ce90 2019-12-05T06:24:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.123+0000 7fb7aed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7b014ca10 0x7fb7b00869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.123+0000 7fb7aed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb7b014ca10 0x7fb7b00869e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52596/0 (socket says 172.21.15.16:52596) 2019-12-05T06:24:44.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.123+0000 7fb7aed9d700 1 -- 172.21.15.16:0/2382393660 learned_addr learned my addr 172.21.15.16:0/2382393660 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:44.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.123+0000 7fb7af59e700 1 --2- 172.21.15.16:0/2382393660 >> v2:172.21.15.76:3300/0 conn(0x7fb7b009ce90 0x7fb7b019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7aed9d700 1 -- 172.21.15.16:0/2382393660 >> v2:172.21.15.76:3300/0 conn(0x7fb7b009ce90 msgr2=0x7fb7b019a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> v2:172.21.15.76:3300/0 conn(0x7fb7b009ce90 0x7fb7b019a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7aed9d700 1 -- 172.21.15.16:0/2382393660 >> v1:172.21.15.76:6789/0 conn(0x7fb7b00879b0 legacy=0x7fb7b0086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7aed9d700 1 -- 172.21.15.16:0/2382393660 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7b00860e0 con 0x7fb7b014ca10 2019-12-05T06:24:44.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7af59e700 1 --2- 172.21.15.16:0/2382393660 >> v2:172.21.15.76:3300/0 conn(0x7fb7b009ce90 0x7fb7b019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:24:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> v2:172.21.15.16:3300/0 conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> v2:172.21.15.16:3300/0 conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=READY pgs=707 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7a77fe700 1 -- 172.21.15.16:0/2382393660 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7a0000c10 con 0x7fb7b014ca10 2019-12-05T06:24:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7a77fe700 1 -- 172.21.15.16:0/2382393660 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7a00010a0 con 0x7fb7b014ca10 2019-12-05T06:24:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7b4abb700 1 -- 172.21.15.16:0/2382393660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7b00860e0 con 0x7fb7b014ca10 2019-12-05T06:24:44.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7a77fe700 1 -- 172.21.15.16:0/2382393660 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7a0001730 con 0x7fb7b014ca10 2019-12-05T06:24:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.124+0000 7fb7b4abb700 1 -- 172.21.15.16:0/2382393660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7b00860e0 con 0x7fb7b014ca10 2019-12-05T06:24:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.125+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.125+0000 7fb7a77fe700 1 -- 172.21.15.16:0/2382393660 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb7a0001290 con 0x7fb7b014ca10 2019-12-05T06:24:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.125+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.125+0000 7fb7a77fe700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb794023630 0x7fb794025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.125+0000 7fb7a77fe700 1 -- 172.21.15.16:0/2382393660 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fb7a0047ed0 con 0x7fb7b014ca10 2019-12-05T06:24:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.126+0000 7fb7ae59c700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb794023630 0x7fb794025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.126+0000 7fb7ae59c700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb794023630 0x7fb794025ac0 crc :-1 s=READY pgs=312 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.132+0000 7fb7b4abb700 1 -- 172.21.15.16:0/2382393660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7b00860e0 con 0x7fb7b014ca10 2019-12-05T06:24:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.138+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.138+0000 7fb7a77fe700 1 -- 172.21.15.16:0/2382393660 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb7a00075b0 con 0x7fb7b014ca10 2019-12-05T06:24:44.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.334+0000 7fb7b4abb700 1 -- 172.21.15.16:0/2382393660 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7b00860e0 con 0x7fb7b014ca10 2019-12-05T06:24:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.334+0000 7fb7aed9d700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.334+0000 7fb7a77fe700 1 -- 172.21.15.16:0/2382393660 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+1771 (secure 0 0 0) 0x7fb7a0001260 con 0x7fb7b014ca10 2019-12-05T06:24:44.320 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 88 2019-12-05T06:24:44.320 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:44.320 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:43.164713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 -- 172.21.15.16:0/2382393660 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb794023630 msgr2=0x7fb794025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb794023630 0x7fb794025ac0 crc :-1 s=READY pgs=312 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 -- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 msgr2=0x7fb7b00869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 0x7fb7b00869e0 secure :-1 s=READY pgs=707 cs=0 l=1 rx=0x7fb7a0012580 tx=0x7fb7a0012920).stop 2019-12-05T06:24:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 -- 172.21.15.16:0/2382393660 shutdown_connections 2019-12-05T06:24:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 --2- 172.21.15.16:0/2382393660 >> v2:172.21.15.76:3300/0 conn(0x7fb7b009ce90 0x7fb7b019a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb794023630 0x7fb794025ac0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 --2- 172.21.15.16:0/2382393660 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb7b014ca10 0x7fb7b00869e0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 -- 172.21.15.16:0/2382393660 shutdown_connections 2019-12-05T06:24:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 -- 172.21.15.16:0/2382393660 wait complete. 2019-12-05T06:24:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.339+0000 7fb7b4abb700 1 -- 172.21.15.16:0/2382393660 >> 172.21.15.16:0/2382393660 conn(0x7fb7b0098480 msgr2=0x7fb7b0071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:44.334 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:24:44.334 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:24:44.334 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:44.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.483+0000 7f3e3dfe0700 1 Processor -- start 2019-12-05T06:24:44.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.483+0000 7f3e3dfe0700 1 -- start start 2019-12-05T06:24:44.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.484+0000 7f3e3dfe0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e3808fef0 0x7f3e3808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:44.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.484+0000 7f3e3dfe0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380958e0 con 0x7f3e3808d2c0 2019-12-05T06:24:44.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.484+0000 7f3e3dfe0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e380629e0 con 0x7f3e3808c6d0 2019-12-05T06:24:44.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.484+0000 7f3e3dfe0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3808df90 con 0x7f3e3808fef0 2019-12-05T06:24:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.484+0000 7f3e2ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e3808d2c0 0x7f3e3808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37772/0 (socket says 172.21.15.16:37772) 2019-12-05T06:24:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.484+0000 7f3e2ffff700 1 -- 172.21.15.16:0/1185739327 learned_addr learned my addr 172.21.15.16:0/1185739327 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.484+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e3808fef0 0x7f3e3808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.484+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2270137388 0 0) 0x7f3e20001150 con 0x7f3e3808d2c0 2019-12-05T06:24:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.484+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e24002e80 con 0x7f3e3808d2c0 2019-12-05T06:24:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1953857369 0 0) 0x7f3e20001130 con 0x7f3e3808d2c0 2019-12-05T06:24:44.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e3808fef0 msgr2=0x7f3e3808f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:44.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e3808fef0 0x7f3e3808f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:44.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 >> v1:172.21.15.76:6789/0 conn(0x7f3e3808c6d0 legacy=0x7f3e3808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e3808e620 con 0x7f3e3808d2c0 2019-12-05T06:24:44.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f3e20001660 con 0x7f3e3808d2c0 2019-12-05T06:24:44.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 >> v1:172.21.15.16:6789/0 conn(0x7f3e3808d2c0 legacy=0x7f3e3808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e24003ab0 0x7f3e24005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:44.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e24006690 0x7f3e24008af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:44.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e240092f0 0x7f3e2400b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:44.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2400bfd0 con 0x7f3e240092f0 2019-12-05T06:24:44.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e240501d0 con 0x7f3e24003ab0 2019-12-05T06:24:44.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e24050400 con 0x7f3e24006690 2019-12-05T06:24:44.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e240092f0 0x7f3e2400b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e37fff700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e24003ab0 0x7f3e24005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e2ffff700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e24006690 0x7f3e24008af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 -- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e24006690 msgr2=0x7f3e24008af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e24006690 0x7f3e24008af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 -- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e24003ab0 msgr2=0x7f3e24005f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e24003ab0 0x7f3e24005f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 -- 172.21.15.16:0/1185739327 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e240506b0 con 0x7f3e240092f0 2019-12-05T06:24:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e240092f0 0x7f3e2400b7c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3e30002d20 tx=0x7f3e30002c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e240092f0 0x7f3e2400b7c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3e30002d20 tx=0x7f3e30002c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e240092f0 0x7f3e2400b7c0 secure :-1 s=READY pgs=709 cs=0 l=1 rx=0x7f3e30002d20 tx=0x7f3e30002c20).ready entity=mon.0 client_cookie=eef2767b24bab63 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:44.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e240092f0 0x7f3e2400b7c0 secure :-1 s=THROTTLE_DONE pgs=709 cs=0 l=1 rx=0x7f3e30002d20 tx=0x7f3e30002c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e30000df0 con 0x7f3e240092f0 2019-12-05T06:24:44.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.485+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e240092f0 0x7f3e2400b7c0 secure :-1 s=THROTTLE_DONE pgs=709 cs=0 l=1 rx=0x7f3e30002d20 tx=0x7f3e30002c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.486+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e240092f0 0x7f3e2400b7c0 secure :-1 s=THROTTLE_DONE pgs=709 cs=0 l=1 rx=0x7f3e30002d20 tx=0x7f3e30002c20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.486+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e3000f010 con 0x7f3e240092f0 2019-12-05T06:24:44.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.486+0000 7f3e377fe700 1 -- 172.21.15.16:0/1185739327 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e3000f6e0 con 0x7f3e240092f0 2019-12-05T06:24:44.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.486+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/1185739327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e240092f0 msgr2=0x7f3e2400b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.486+0000 7f3e3dfe0700 1 --2- 172.21.15.16:0/1185739327 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e240092f0 0x7f3e2400b7c0 secure :-1 s=READY pgs=709 cs=0 l=1 rx=0x7f3e30002d20 tx=0x7f3e30002c20).stop 2019-12-05T06:24:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.486+0000 7f3e3cd7e700 1 -- 172.21.15.16:0/1185739327 reap_dead start 2019-12-05T06:24:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.486+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/1185739327 shutdown_connections 2019-12-05T06:24:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.486+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/1185739327 shutdown_connections 2019-12-05T06:24:44.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.486+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/1185739327 wait complete. 2019-12-05T06:24:44.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.486+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/1185739327 >> 172.21.15.16:0/1185739327 conn(0x7f3e38061940 msgr2=0x7f3e38096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:44.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.489+0000 7f3e3dfe0700 1 Processor -- start 2019-12-05T06:24:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.489+0000 7f3e3dfe0700 1 -- start start 2019-12-05T06:24:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.490+0000 7f3e3dfe0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e3812bf70 0x7f3e381584d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.490+0000 7f3e3dfe0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e38128b20 con 0x7f3e381325e0 2019-12-05T06:24:44.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.490+0000 7f3e3dfe0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e38128660 con 0x7f3e3812c740 2019-12-05T06:24:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.490+0000 7f3e3dfe0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e3812a7e0 con 0x7f3e3812bf70 2019-12-05T06:24:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.490+0000 7f3e37fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3e381325e0 0x7f3e3812b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37784/0 (socket says 172.21.15.16:37784) 2019-12-05T06:24:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.490+0000 7f3e37fff700 1 -- 172.21.15.16:0/846726345 learned_addr learned my addr 172.21.15.16:0/846726345 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:44.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.490+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e3812bf70 0x7f3e381584d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076828576 0 0) 0x7f3e280010c0 con 0x7f3e381325e0 2019-12-05T06:24:44.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3e1c002e80 con 0x7f3e381325e0 2019-12-05T06:24:44.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1370811065 0 0) 0x7f3e280012c0 con 0x7f3e381325e0 2019-12-05T06:24:44.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3e1c0032e0 con 0x7f3e381325e0 2019-12-05T06:24:44.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f3e28001870 con 0x7f3e381325e0 2019-12-05T06:24:44.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e3812bf70 msgr2=0x7f3e381584d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e3812bf70 0x7f3e381584d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 >> v1:172.21.15.76:6789/0 conn(0x7f3e3812c740 legacy=0x7f3e38154cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 >> v1:172.21.15.16:6789/0 conn(0x7f3e381325e0 legacy=0x7f3e3812b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e1c003ca0 0x7f3e1c004100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e1c004910 0x7f3e1c004da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c00bb90 con 0x7f3e1c003330 2019-12-05T06:24:44.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c00bdc0 con 0x7f3e1c003ca0 2019-12-05T06:24:44.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e1c00bff0 con 0x7f3e1c004910 2019-12-05T06:24:44.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e3cd7e700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e1c004910 0x7f3e1c004da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.491+0000 7f3e2ffff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e1c003ca0 0x7f3e1c004100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 -- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e1c004910 msgr2=0x7f3e1c004da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3e1c004910 0x7f3e1c004da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 -- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e1c003ca0 msgr2=0x7f3e1c004100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e1c003ca0 0x7f3e1c004100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 -- 172.21.15.16:0/846726345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c0106b0 con 0x7f3e1c003330 2019-12-05T06:24:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e2ffff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3e1c003ca0 0x7f3e1c004100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:24:44.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/846726345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e3812ae70 con 0x7f3e1c003330 2019-12-05T06:24:44.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/846726345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e3814a820 con 0x7f3e1c003330 2019-12-05T06:24:44.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=READY pgs=711 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:44.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=THROTTLE_DONE pgs=711 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e28000ca0 con 0x7f3e1c003330 2019-12-05T06:24:44.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=THROTTLE_DONE pgs=711 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=THROTTLE_DONE pgs=711 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e28000f00 con 0x7f3e1c003330 2019-12-05T06:24:44.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e28000ca0 con 0x7f3e1c003330 2019-12-05T06:24:44.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.492+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=THROTTLE_DONE pgs=711 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.493+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3e280010f0 con 0x7f3e1c003330 2019-12-05T06:24:44.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.493+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=THROTTLE_DONE pgs=711 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.493+0000 7f3e357fa700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e1c02fee0 0x7f3e1c032310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:44.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.493+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f3e28045d70 con 0x7f3e1c003330 2019-12-05T06:24:44.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.493+0000 7f3e2ffff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e1c02fee0 0x7f3e1c032310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:44.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.494+0000 7f3e2ffff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e1c02fee0 0x7f3e1c032310 crc :-1 s=READY pgs=313 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:44.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.498+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/846726345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e10002e60 con 0x7f3e1c003330 2019-12-05T06:24:44.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.502+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=THROTTLE_DONE pgs=711 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.502+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3e28044fb0 con 0x7f3e1c003330 2019-12-05T06:24:44.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.695+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/846726345 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e10001d10 con 0x7f3e1c003330 2019-12-05T06:24:44.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.695+0000 7f3e37fff700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=THROTTLE_DONE pgs=711 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:44.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.696+0000 7f3e357fa700 1 -- 172.21.15.16:0/846726345 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 88 v88) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f3e28044fb0 con 0x7f3e1c003330 2019-12-05T06:24:44.690 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 88 2019-12-05T06:24:44.691 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:44.692 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":88,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":88,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:43.164713+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.698+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e1c02fee0 msgr2=0x7f3e1c032310 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.698+0000 7f3e3dfe0700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3e1c02fee0 0x7f3e1c032310 crc :-1 s=READY pgs=313 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:44.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.698+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 msgr2=0x7f3e1c003760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.698+0000 7f3e3dfe0700 1 --2- 172.21.15.16:0/846726345 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3e1c003330 0x7f3e1c003760 secure :-1 s=READY pgs=711 cs=0 l=1 rx=0x7f3e28010440 tx=0x7f3e28010270).stop 2019-12-05T06:24:44.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.698+0000 7f3e3cd7e700 1 -- 172.21.15.16:0/846726345 reap_dead start 2019-12-05T06:24:44.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.698+0000 7f3e3cd7e700 1 -- 172.21.15.16:0/846726345 reap_dead start 2019-12-05T06:24:44.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.698+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/846726345 shutdown_connections 2019-12-05T06:24:44.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.699+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/846726345 shutdown_connections 2019-12-05T06:24:44.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.699+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/846726345 wait complete. 2019-12-05T06:24:44.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:44.699+0000 7f3e3dfe0700 1 -- 172.21.15.16:0/846726345 >> 172.21.15.16:0/846726345 conn(0x7f3e380a2850 msgr2=0x7f3e3806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:44.698 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 88, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 88, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5940": { "addr": "172.21.15.16:6833/3836369875", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 3836369875, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 3836369875, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5940, "incarnation": 88, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 61, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:24:43.164713+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5940 } } } ], "standbys": [] } 2019-12-05T06:24:46.699 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.989+0000 7fe589e1a700 1 Processor -- start 2019-12-05T06:24:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.989+0000 7fe589e1a700 1 -- start start 2019-12-05T06:24:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.991+0000 7fe589e1a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe58408c6d0 0x7fe58408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.991+0000 7fe589e1a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe58408fef0 0x7fe58408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.991+0000 7fe589e1a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5840958e0 con 0x7fe58408ef20 2019-12-05T06:24:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.991+0000 7fe589e1a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58408df90 con 0x7fe58408c6d0 2019-12-05T06:24:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.991+0000 7fe589e1a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5840a3f70 con 0x7fe58408fef0 2019-12-05T06:24:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.991+0000 7fe583fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe58408c6d0 0x7fe58408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.991+0000 7fe583fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe58408c6d0 0x7fe58408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52632/0 (socket says 172.21.15.16:52632) 2019-12-05T06:24:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.991+0000 7fe583fff700 1 -- 172.21.15.16:0/3754577481 learned_addr learned my addr 172.21.15.16:0/3754577481 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.992+0000 7fe5837fe700 1 --2- 172.21.15.16:0/3754577481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe58408fef0 0x7fe58408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.992+0000 7fe583fff700 1 -- 172.21.15.16:0/3754577481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe58408fef0 msgr2=0x7fe58408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.992+0000 7fe583fff700 1 --2- 172.21.15.16:0/3754577481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe58408fef0 0x7fe58408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:46.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.992+0000 7fe583fff700 1 -- 172.21.15.16:0/3754577481 >> v1:172.21.15.76:6789/0 conn(0x7fe58408ef20 legacy=0x7fe58408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.992+0000 7fe582ffd700 1 -- 172.21.15.16:0/3754577481 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 509466641 0 0) 0x7fe57c001080 con 0x7fe58408ef20 2019-12-05T06:24:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.992+0000 7fe583fff700 1 -- 172.21.15.16:0/3754577481 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe58408d540 con 0x7fe58408c6d0 2019-12-05T06:24:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.992+0000 7fe583fff700 1 --2- 172.21.15.16:0/3754577481 >> v2:172.21.15.16:3300/0 conn(0x7fe58408c6d0 0x7fe58408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe57400fa40 tx=0x7fe57400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.993+0000 7fe583fff700 1 --2- 172.21.15.16:0/3754577481 >> v2:172.21.15.16:3300/0 conn(0x7fe58408c6d0 0x7fe58408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe57400fa40 tx=0x7fe57400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.993+0000 7fe583fff700 1 --2- 172.21.15.16:0/3754577481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58408c6d0 0x7fe58408cac0 secure :-1 s=READY pgs=712 cs=0 l=1 rx=0x7fe57400fa40 tx=0x7fe57400fde0).ready entity=mon.0 client_cookie=d48fd4d60b807262 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.993+0000 7fe583fff700 1 --2- 172.21.15.16:0/3754577481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58408c6d0 0x7fe58408cac0 secure :-1 s=THROTTLE_DONE pgs=712 cs=0 l=1 rx=0x7fe57400fa40 tx=0x7fe57400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.993+0000 7fe582ffd700 1 -- 172.21.15.16:0/3754577481 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe574000ce0 con 0x7fe58408c6d0 2019-12-05T06:24:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.993+0000 7fe583fff700 1 --2- 172.21.15.16:0/3754577481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58408c6d0 0x7fe58408cac0 secure :-1 s=THROTTLE_DONE pgs=712 cs=0 l=1 rx=0x7fe57400fa40 tx=0x7fe57400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.993+0000 7fe583fff700 1 --2- 172.21.15.16:0/3754577481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58408c6d0 0x7fe58408cac0 secure :-1 s=THROTTLE_DONE pgs=712 cs=0 l=1 rx=0x7fe57400fa40 tx=0x7fe57400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.993+0000 7fe582ffd700 1 -- 172.21.15.16:0/3754577481 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe574001010 con 0x7fe58408c6d0 2019-12-05T06:24:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.993+0000 7fe582ffd700 1 -- 172.21.15.16:0/3754577481 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe574001720 con 0x7fe58408c6d0 2019-12-05T06:24:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.993+0000 7fe589e1a700 1 -- 172.21.15.16:0/3754577481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58408c6d0 msgr2=0x7fe58408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.993+0000 7fe589e1a700 1 --2- 172.21.15.16:0/3754577481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58408c6d0 0x7fe58408cac0 secure :-1 s=READY pgs=712 cs=0 l=1 rx=0x7fe57400fa40 tx=0x7fe57400fde0).stop 2019-12-05T06:24:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.995+0000 7fe589e1a700 1 -- 172.21.15.16:0/3754577481 shutdown_connections 2019-12-05T06:24:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.995+0000 7fe589e1a700 1 --2- 172.21.15.16:0/3754577481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe58408fef0 0x7fe58408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.995+0000 7fe589e1a700 1 --2- 172.21.15.16:0/3754577481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58408c6d0 0x7fe58408cac0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.995+0000 7fe589e1a700 1 -- 172.21.15.16:0/3754577481 shutdown_connections 2019-12-05T06:24:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.995+0000 7fe589e1a700 1 -- 172.21.15.16:0/3754577481 wait complete. 2019-12-05T06:24:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:46.995+0000 7fe589e1a700 1 -- 172.21.15.16:0/3754577481 >> 172.21.15.16:0/3754577481 conn(0x7fe584061940 msgr2=0x7fe584096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.001+0000 7fe589e1a700 1 Processor -- start 2019-12-05T06:24:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.001+0000 7fe589e1a700 1 -- start start 2019-12-05T06:24:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.003+0000 7fe589e1a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe58406df10 0x7fe584098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.003+0000 7fe589e1a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe5840992d0 0x7fe584098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.003+0000 7fe589e1a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe58406e300 con 0x7fe584098d80 2019-12-05T06:24:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.003+0000 7fe589e1a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe584097370 con 0x7fe58406df10 2019-12-05T06:24:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.003+0000 7fe589e1a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe58408ad00 con 0x7fe5840992d0 2019-12-05T06:24:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.003+0000 7fe583fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe58406df10 0x7fe584098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.003+0000 7fe583fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe58406df10 0x7fe584098300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52638/0 (socket says 172.21.15.16:52638) 2019-12-05T06:24:46.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.003+0000 7fe588bb8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe584098d80 0x7fe584088fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37810/0 (socket says 172.21.15.16:37810) 2019-12-05T06:24:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.003+0000 7fe583fff700 1 -- 172.21.15.16:0/2413140897 learned_addr learned my addr 172.21.15.16:0/2413140897 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.003+0000 7fe5837fe700 1 --2- 172.21.15.16:0/2413140897 >> v2:172.21.15.76:3300/0 conn(0x7fe5840992d0 0x7fe584098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe583fff700 1 -- 172.21.15.16:0/2413140897 >> v2:172.21.15.76:3300/0 conn(0x7fe5840992d0 msgr2=0x7fe584098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> v2:172.21.15.76:3300/0 conn(0x7fe5840992d0 0x7fe584098840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe583fff700 1 -- 172.21.15.16:0/2413140897 >> v1:172.21.15.16:6789/0 conn(0x7fe584098d80 legacy=0x7fe584088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe580ff9700 1 -- 172.21.15.16:0/2413140897 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683057809 0 0) 0x7fe57c000ba0 con 0x7fe584098d80 2019-12-05T06:24:46.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe583fff700 1 -- 172.21.15.16:0/2413140897 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe584097a00 con 0x7fe58406df10 2019-12-05T06:24:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> v2:172.21.15.16:3300/0 conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> v2:172.21.15.16:3300/0 conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=READY pgs=714 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:46.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe580ff9700 1 -- 172.21.15.16:0/2413140897 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5740056f0 con 0x7fe58406df10 2019-12-05T06:24:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe580ff9700 1 -- 172.21.15.16:0/2413140897 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe574007e10 con 0x7fe58406df10 2019-12-05T06:24:46.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe589e1a700 1 -- 172.21.15.16:0/2413140897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe584097a00 con 0x7fe58406df10 2019-12-05T06:24:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.004+0000 7fe580ff9700 1 -- 172.21.15.16:0/2413140897 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5740058e0 con 0x7fe58406df10 2019-12-05T06:24:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.005+0000 7fe589e1a700 1 -- 172.21.15.16:0/2413140897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe584097a00 con 0x7fe58406df10 2019-12-05T06:24:46.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.005+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.007+0000 7fe580ff9700 1 -- 172.21.15.16:0/2413140897 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe574005500 con 0x7fe58406df10 2019-12-05T06:24:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.007+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.007+0000 7fe580ff9700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe568023630 0x7fe568025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.007+0000 7fe580ff9700 1 -- 172.21.15.16:0/2413140897 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fe574046160 con 0x7fe58406df10 2019-12-05T06:24:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.007+0000 7fe5837fe700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe568023630 0x7fe568025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:46.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.008+0000 7fe5837fe700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe568023630 0x7fe568025ac0 crc :-1 s=READY pgs=314 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:47.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.021+0000 7fe589e1a700 1 -- 172.21.15.16:0/2413140897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe584097a00 con 0x7fe58406df10 2019-12-05T06:24:47.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.028+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:47.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.028+0000 7fe580ff9700 1 -- 172.21.15.16:0/2413140897 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe57402ef80 con 0x7fe58406df10 2019-12-05T06:24:47.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.281+0000 7fe589e1a700 1 -- 172.21.15.16:0/2413140897 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe584097a00 con 0x7fe58406df10 2019-12-05T06:24:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.282+0000 7fe583fff700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=THROTTLE_DONE pgs=714 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:47.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.282+0000 7fe580ff9700 1 -- 172.21.15.16:0/2413140897 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 90 v90) v1 ==== 94+0+1772 (secure 0 0 0) 0x7fe57402ef80 con 0x7fe58406df10 2019-12-05T06:24:47.268 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 90 2019-12-05T06:24:47.268 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:47.268 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":90,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":90,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:47.199774+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:47.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 -- 172.21.15.16:0/2413140897 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe568023630 msgr2=0x7fe568025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:47.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe568023630 0x7fe568025ac0 crc :-1 s=READY pgs=314 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:47.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 -- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 msgr2=0x7fe584098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:47.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 0x7fe584098300 secure :-1 s=READY pgs=714 cs=0 l=1 rx=0x7fe5740107c0 tx=0x7fe574010670).stop 2019-12-05T06:24:47.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 -- 172.21.15.16:0/2413140897 shutdown_connections 2019-12-05T06:24:47.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 --2- 172.21.15.16:0/2413140897 >> v2:172.21.15.76:3300/0 conn(0x7fe5840992d0 0x7fe584098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:47.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe568023630 0x7fe568025ac0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:47.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 --2- 172.21.15.16:0/2413140897 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe58406df10 0x7fe584098300 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:47.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 -- 172.21.15.16:0/2413140897 shutdown_connections 2019-12-05T06:24:47.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 -- 172.21.15.16:0/2413140897 wait complete. 2019-12-05T06:24:47.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:47.285+0000 7fe589e1a700 1 -- 172.21.15.16:0/2413140897 >> 172.21.15.16:0/2413140897 conn(0x7fe5840714f0 msgr2=0x7fe58409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:47.280 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:24:49.282 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.537+0000 7f98fafc5700 1 Processor -- start 2019-12-05T06:24:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.537+0000 7f98fafc5700 1 -- start start 2019-12-05T06:24:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.538+0000 7f98fafc5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f98f408fef0 0x7f98f408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:49.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.538+0000 7f98fafc5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f40958e0 con 0x7f98f408d2c0 2019-12-05T06:24:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.538+0000 7f98fafc5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f40629e0 con 0x7f98f408c6d0 2019-12-05T06:24:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.538+0000 7f98fafc5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f408df90 con 0x7f98f408fef0 2019-12-05T06:24:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.538+0000 7f98f8d61700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f98f408d2c0 0x7f98f408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37820/0 (socket says 172.21.15.16:37820) 2019-12-05T06:24:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.538+0000 7f98f8d61700 1 -- 172.21.15.16:0/301229812 learned_addr learned my addr 172.21.15.16:0/301229812 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.538+0000 7f98f9d63700 1 --2- 172.21.15.16:0/301229812 >> v2:172.21.15.76:3300/0 conn(0x7f98f408fef0 0x7f98f408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093724560 0 0) 0x7f98e4001150 con 0x7f98f408d2c0 2019-12-05T06:24:49.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98e0002e80 con 0x7f98f408d2c0 2019-12-05T06:24:49.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1558578990 0 0) 0x7f98e4001130 con 0x7f98f408d2c0 2019-12-05T06:24:49.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 >> v2:172.21.15.76:3300/0 conn(0x7f98f408fef0 msgr2=0x7f98f408f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:49.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 --2- 172.21.15.16:0/301229812 >> v2:172.21.15.76:3300/0 conn(0x7f98f408fef0 0x7f98f408f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:49.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 >> v1:172.21.15.76:6789/0 conn(0x7f98f408c6d0 legacy=0x7f98f408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:49.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f408e620 con 0x7f98f408d2c0 2019-12-05T06:24:49.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f98e40016c0 con 0x7f98f408d2c0 2019-12-05T06:24:49.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 >> v1:172.21.15.16:6789/0 conn(0x7f98f408d2c0 legacy=0x7f98f408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:49.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e0003ab0 0x7f98e0005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:49.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 0x7f98e0008b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:49.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98e0009380 0x7f98e000b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:49.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e000c000 con 0x7f98e0006690 2019-12-05T06:24:49.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98e0050200 con 0x7f98e0003ab0 2019-12-05T06:24:49.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98e0050430 con 0x7f98e0009380 2019-12-05T06:24:49.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98f8d61700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 0x7f98e0008b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:49.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98f9562700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e0003ab0 0x7f98e0005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:49.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.539+0000 7f98f9d63700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98e0009380 0x7f98e000b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:49.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 -- 172.21.15.16:0/301229812 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98e0009380 msgr2=0x7f98e000b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:49.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98e0009380 0x7f98e000b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:49.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 -- 172.21.15.16:0/301229812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e0003ab0 msgr2=0x7f98e0005f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:49.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98e0003ab0 0x7f98e0005f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:49.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 -- 172.21.15.16:0/301229812 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98e00506e0 con 0x7f98e0006690 2019-12-05T06:24:49.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f9d63700 1 -- 172.21.15.16:0/301229812 reap_dead start 2019-12-05T06:24:49.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 0x7f98e0008b20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f98e400f910 tx=0x7f98e400f810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 0x7f98e0008b20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f98e400f910 tx=0x7f98e400f810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 0x7f98e0008b20 secure :-1 s=READY pgs=716 cs=0 l=1 rx=0x7f98e400f910 tx=0x7f98e400f810).ready entity=mon.0 client_cookie=e54c4630cffad8db server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:49.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 0x7f98e0008b20 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f98e400f910 tx=0x7f98e400f810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 0x7f98e0008b20 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f98e400f910 tx=0x7f98e400f810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98e4001130 con 0x7f98e0006690 2019-12-05T06:24:49.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98f8d61700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 0x7f98e0008b20 secure :-1 s=THROTTLE_DONE pgs=716 cs=0 l=1 rx=0x7f98e400f910 tx=0x7f98e400f810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e4004240 con 0x7f98e0006690 2019-12-05T06:24:49.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98ebfff700 1 -- 172.21.15.16:0/301229812 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98e4004990 con 0x7f98e0006690 2019-12-05T06:24:49.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98fafc5700 1 -- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 msgr2=0x7f98e0008b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.540+0000 7f98fafc5700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 0x7f98e0008b20 secure :-1 s=READY pgs=716 cs=0 l=1 rx=0x7f98e400f910 tx=0x7f98e400f810).stop 2019-12-05T06:24:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.541+0000 7f98fafc5700 1 -- 172.21.15.16:0/301229812 shutdown_connections 2019-12-05T06:24:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.541+0000 7f98fafc5700 1 --2- 172.21.15.16:0/301229812 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98e0006690 0x7f98e0008b20 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.541+0000 7f98fafc5700 1 -- 172.21.15.16:0/301229812 shutdown_connections 2019-12-05T06:24:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.541+0000 7f98fafc5700 1 -- 172.21.15.16:0/301229812 wait complete. 2019-12-05T06:24:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.541+0000 7f98fafc5700 1 -- 172.21.15.16:0/301229812 >> 172.21.15.16:0/301229812 conn(0x7f98f4061940 msgr2=0x7f98f4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.544+0000 7f98fafc5700 1 Processor -- start 2019-12-05T06:24:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.544+0000 7f98fafc5700 1 -- start start 2019-12-05T06:24:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.545+0000 7f98fafc5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f98f412b640 0x7f98f412ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.545+0000 7f98fafc5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f98f4153ba0 0x7f98f4155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.545+0000 7f98fafc5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98f4131bf0 con 0x7f98f41302f0 2019-12-05T06:24:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.545+0000 7f98fafc5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f41296e0 con 0x7f98f412b640 2019-12-05T06:24:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.545+0000 7f98fafc5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98f4128cb0 con 0x7f98f4153ba0 2019-12-05T06:24:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f9562700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f98f41302f0 0x7f98f412a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37832/0 (socket says 172.21.15.16:37832) 2019-12-05T06:24:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f9562700 1 -- 172.21.15.16:0/849123549 learned_addr learned my addr 172.21.15.16:0/849123549 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> v2:172.21.15.16:3300/0 conn(0x7f98f412b640 0x7f98f412ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f9d63700 1 --2- 172.21.15.16:0/849123549 >> v2:172.21.15.76:3300/0 conn(0x7f98f4153ba0 0x7f98f4155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98e9ffb700 1 -- 172.21.15.16:0/849123549 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 395231232 0 0) 0x7f98f0001270 con 0x7f98f41302f0 2019-12-05T06:24:49.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98e9ffb700 1 -- 172.21.15.16:0/849123549 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98d4002e80 con 0x7f98f41302f0 2019-12-05T06:24:49.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f8d61700 1 -- 172.21.15.16:0/849123549 >> v2:172.21.15.76:3300/0 conn(0x7f98f4153ba0 msgr2=0x7f98f4155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:49.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> v2:172.21.15.76:3300/0 conn(0x7f98f4153ba0 0x7f98f4155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:49.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f8d61700 1 -- 172.21.15.16:0/849123549 >> v1:172.21.15.16:6789/0 conn(0x7f98f41302f0 legacy=0x7f98f412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:49.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f8d61700 1 -- 172.21.15.16:0/849123549 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f4129d70 con 0x7f98f412b640 2019-12-05T06:24:49.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> v2:172.21.15.16:3300/0 conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> v2:172.21.15.16:3300/0 conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=READY pgs=718 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.546+0000 7f98e9ffb700 1 -- 172.21.15.16:0/849123549 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98e4004240 con 0x7f98f412b640 2019-12-05T06:24:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.547+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.547+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.547+0000 7f98e9ffb700 1 -- 172.21.15.16:0/849123549 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98e4001270 con 0x7f98f412b640 2019-12-05T06:24:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.547+0000 7f98fafc5700 1 -- 172.21.15.16:0/849123549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98f4129d70 con 0x7f98f412b640 2019-12-05T06:24:49.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.547+0000 7f98fafc5700 1 -- 172.21.15.16:0/849123549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98f4129d70 con 0x7f98f412b640 2019-12-05T06:24:49.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.547+0000 7f98e9ffb700 1 -- 172.21.15.16:0/849123549 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98e4004430 con 0x7f98f412b640 2019-12-05T06:24:49.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.547+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.548+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.548+0000 7f98e9ffb700 1 -- 172.21.15.16:0/849123549 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f98e4004240 con 0x7f98f412b640 2019-12-05T06:24:49.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.548+0000 7f98e9ffb700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98d4023670 0x7f98d4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:49.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.548+0000 7f98e9ffb700 1 -- 172.21.15.16:0/849123549 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f98e4045f60 con 0x7f98f412b640 2019-12-05T06:24:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.548+0000 7f98f9562700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98d4023670 0x7f98d4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.549+0000 7f98f9562700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98d4023670 0x7f98d4025ae0 crc :-1 s=READY pgs=315 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.552+0000 7f98fafc5700 1 -- 172.21.15.16:0/849123549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc002e60 con 0x7f98f412b640 2019-12-05T06:24:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.557+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.557+0000 7f98e9ffb700 1 -- 172.21.15.16:0/849123549 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f98e4000ee0 con 0x7f98f412b640 2019-12-05T06:24:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.762+0000 7f98fafc5700 1 -- 172.21.15.16:0/849123549 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98cc001d10 con 0x7f98f412b640 2019-12-05T06:24:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.763+0000 7f98f8d61700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:49.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.763+0000 7f98e9ffb700 1 -- 172.21.15.16:0/849123549 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f98e4004240 con 0x7f98f412b640 2019-12-05T06:24:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 91 2019-12-05T06:24:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 -- 172.21.15.16:0/849123549 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98d4023670 msgr2=0x7f98d4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98d4023670 0x7f98d4025ae0 crc :-1 s=READY pgs=315 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 -- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 msgr2=0x7f98f412ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 0x7f98f412ae70 secure :-1 s=READY pgs=718 cs=0 l=1 rx=0x7f98e4010370 tx=0x7f98e4010710).stop 2019-12-05T06:24:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 -- 172.21.15.16:0/849123549 shutdown_connections 2019-12-05T06:24:49.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 --2- 172.21.15.16:0/849123549 >> v2:172.21.15.76:3300/0 conn(0x7f98f4153ba0 0x7f98f4155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98d4023670 0x7f98d4025ae0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 --2- 172.21.15.16:0/849123549 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98f412b640 0x7f98f412ae70 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 -- 172.21.15.16:0/849123549 shutdown_connections 2019-12-05T06:24:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 -- 172.21.15.16:0/849123549 wait complete. 2019-12-05T06:24:49.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:49.766+0000 7f98fafc5700 1 -- 172.21.15.16:0/849123549 >> 172.21.15.16:0/849123549 conn(0x7f98f40a2850 msgr2=0x7f98f406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:49.766 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:49.766 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:49.768 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5878 2019-12-05T06:24:49.768 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.b 2019-12-05T06:24:50.769 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:24:50.770 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:24:50.770 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:24:50.775 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:24:50.775 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.941+0000 7f9bb8292700 1 Processor -- start 2019-12-05T06:24:50.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.941+0000 7f9bb8292700 1 -- start start 2019-12-05T06:24:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb8292700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bb008d2c0 0x7f9bb008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb8292700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb00958e0 con 0x7f9bb008c6d0 2019-12-05T06:24:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb8292700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bb00629e0 con 0x7f9bb008fef0 2019-12-05T06:24:50.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb8292700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb008df90 con 0x7f9bb008d2c0 2019-12-05T06:24:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb682f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9bb008c6d0 0x7f9bb008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37854/0 (socket says 172.21.15.16:37854) 2019-12-05T06:24:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb682f700 1 -- 172.21.15.16:0/2671604136 learned_addr learned my addr 172.21.15.16:0/2671604136 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb602e700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bb008d2c0 0x7f9bb008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1617300527 0 0) 0x7f9ba0001150 con 0x7f9bb008c6d0 2019-12-05T06:24:50.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b9c002e80 con 0x7f9bb008c6d0 2019-12-05T06:24:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2397725978 0 0) 0x7f9ba0001130 con 0x7f9bb008c6d0 2019-12-05T06:24:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bb008d2c0 msgr2=0x7f9bb008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb582d700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bb008d2c0 0x7f9bb008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 >> v1:172.21.15.76:6789/0 conn(0x7f9bb008fef0 legacy=0x7f9bb008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.942+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb008e620 con 0x7f9bb008c6d0 2019-12-05T06:24:50.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f9ba0001660 con 0x7f9bb008c6d0 2019-12-05T06:24:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 >> v1:172.21.15.16:6789/0 conn(0x7f9bb008c6d0 legacy=0x7f9bb008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b9c003ab0 0x7f9b9c005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9b9c006690 0x7f9b9c008b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b9c009380 0x7f9b9c00b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c00bfd0 con 0x7f9b9c003ab0 2019-12-05T06:24:50.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b9c003ab0 0x7f9b9c005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c0501d0 con 0x7f9b9c006690 2019-12-05T06:24:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c050400 con 0x7f9b9c009380 2019-12-05T06:24:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb602e700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9b9c006690 0x7f9b9c008b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb7030700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b9c009380 0x7f9b9c00b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 -- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b9c009380 msgr2=0x7f9b9c00b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b9c009380 0x7f9b9c00b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:50.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 -- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9b9c006690 msgr2=0x7f9b9c008b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:50.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9b9c006690 0x7f9b9c008b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:50.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 -- 172.21.15.16:0/2671604136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b9c0506b0 con 0x7f9b9c003ab0 2019-12-05T06:24:50.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b9c003ab0 0x7f9b9c005f10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9ba000f900 tx=0x7f9ba000f830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b9c003ab0 0x7f9b9c005f10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9ba000f900 tx=0x7f9ba000f830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b9c003ab0 0x7f9b9c005f10 secure :-1 s=READY pgs=720 cs=0 l=1 rx=0x7f9ba000f900 tx=0x7f9ba000f830).ready entity=mon.0 client_cookie=9b0ed7dd6ac3291f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:50.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b9c003ab0 0x7f9b9c005f10 secure :-1 s=THROTTLE_DONE pgs=720 cs=0 l=1 rx=0x7f9ba000f900 tx=0x7f9ba000f830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ba0004640 con 0x7f9b9c003ab0 2019-12-05T06:24:50.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b9c003ab0 0x7f9b9c005f10 secure :-1 s=THROTTLE_DONE pgs=720 cs=0 l=1 rx=0x7f9ba000f900 tx=0x7f9ba000f830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ba0001130 con 0x7f9b9c003ab0 2019-12-05T06:24:50.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb682f700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b9c003ab0 0x7f9b9c005f10 secure :-1 s=THROTTLE_DONE pgs=720 cs=0 l=1 rx=0x7f9ba000f900 tx=0x7f9ba000f830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb582d700 1 -- 172.21.15.16:0/2671604136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ba0004640 con 0x7f9b9c003ab0 2019-12-05T06:24:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb8292700 1 -- 172.21.15.16:0/2671604136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b9c003ab0 msgr2=0x7f9b9c005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb8292700 1 --2- 172.21.15.16:0/2671604136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9b9c003ab0 0x7f9b9c005f10 secure :-1 s=READY pgs=720 cs=0 l=1 rx=0x7f9ba000f900 tx=0x7f9ba000f830).stop 2019-12-05T06:24:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.943+0000 7f9bb7030700 1 -- 172.21.15.16:0/2671604136 reap_dead start 2019-12-05T06:24:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.945+0000 7f9bb8292700 1 -- 172.21.15.16:0/2671604136 shutdown_connections 2019-12-05T06:24:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.945+0000 7f9bb8292700 1 -- 172.21.15.16:0/2671604136 shutdown_connections 2019-12-05T06:24:50.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.945+0000 7f9bb8292700 1 -- 172.21.15.16:0/2671604136 wait complete. 2019-12-05T06:24:50.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.945+0000 7f9bb8292700 1 -- 172.21.15.16:0/2671604136 >> 172.21.15.16:0/2671604136 conn(0x7f9bb0061940 msgr2=0x7f9bb0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:50.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.947+0000 7f9bb8292700 1 Processor -- start 2019-12-05T06:24:50.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.947+0000 7f9bb8292700 1 -- start start 2019-12-05T06:24:50.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb8292700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9bb009d3a0 0x7f9bb009d790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:50.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb8292700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bb0084480 0x7f9bb009dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:50.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb8292700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bb00a1730 0x7f9bb00a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:50.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb8292700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb01326b0 con 0x7f9bb0084480 2019-12-05T06:24:50.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb8292700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bb00a07a0 con 0x7f9bb009d3a0 2019-12-05T06:24:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb8292700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bb009fd70 con 0x7f9bb00a1730 2019-12-05T06:24:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb602e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bb0084480 0x7f9bb009dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb602e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bb0084480 0x7f9bb009dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52700/0 (socket says 172.21.15.16:52700) 2019-12-05T06:24:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb602e700 1 -- 172.21.15.16:0/3110958046 learned_addr learned my addr 172.21.15.16:0/3110958046 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb682f700 1 --2- 172.21.15.16:0/3110958046 >> v2:172.21.15.76:3300/0 conn(0x7f9bb009d3a0 0x7f9bb009d790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:50.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb7030700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bb00a1730 0x7f9bb00a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:50.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb602e700 1 -- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bb00a1730 msgr2=0x7f9bb00a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:50.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bb00a1730 0x7f9bb00a1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:50.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb602e700 1 -- 172.21.15.16:0/3110958046 >> v2:172.21.15.76:3300/0 conn(0x7f9bb009d3a0 msgr2=0x7f9bb009d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:50.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> v2:172.21.15.76:3300/0 conn(0x7f9bb009d3a0 0x7f9bb009d790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:50.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.948+0000 7f9bb602e700 1 -- 172.21.15.16:0/3110958046 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bb00a0e30 con 0x7f9bb0084480 2019-12-05T06:24:50.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb7030700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bb00a1730 0x7f9bb00a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:24:50.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb682f700 1 --2- 172.21.15.16:0/3110958046 >> v2:172.21.15.76:3300/0 conn(0x7f9bb009d3a0 0x7f9bb009d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:24:50.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> v2:172.21.15.16:3300/0 conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> v2:172.21.15.16:3300/0 conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=READY pgs=721 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:50.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9ba77fe700 1 -- 172.21.15.16:0/3110958046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bac000ce0 con 0x7f9bb0084480 2019-12-05T06:24:50.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9ba77fe700 1 -- 172.21.15.16:0/3110958046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bac001200 con 0x7f9bb0084480 2019-12-05T06:24:50.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb8292700 1 -- 172.21.15.16:0/3110958046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bb00a0e30 con 0x7f9bb0084480 2019-12-05T06:24:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9ba77fe700 1 -- 172.21.15.16:0/3110958046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bac001910 con 0x7f9bb0084480 2019-12-05T06:24:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb8292700 1 -- 172.21.15.16:0/3110958046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bb00a0e30 con 0x7f9bb0084480 2019-12-05T06:24:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.949+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.950+0000 7f9ba77fe700 1 -- 172.21.15.16:0/3110958046 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9bac0013f0 con 0x7f9bb0084480 2019-12-05T06:24:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.950+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.950+0000 7f9ba77fe700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b98023630 0x7f9b98025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.950+0000 7f9ba77fe700 1 -- 172.21.15.16:0/3110958046 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f9bac046140 con 0x7f9bb0084480 2019-12-05T06:24:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.950+0000 7f9bb682f700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b98023630 0x7f9b98025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.951+0000 7f9bb682f700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b98023630 0x7f9b98025ac0 crc :-1 s=READY pgs=316 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.956+0000 7f9bb8292700 1 -- 172.21.15.16:0/3110958046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bb00a23c0 con 0x7f9bb0084480 2019-12-05T06:24:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.960+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:50.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:50.960+0000 7f9ba77fe700 1 -- 172.21.15.16:0/3110958046 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9bac0095d0 con 0x7f9bb0084480 2019-12-05T06:24:50.962 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:24:51.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.169+0000 7f9bb8292700 1 -- 172.21.15.16:0/3110958046 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bb00a0e30 con 0x7f9bb0084480 2019-12-05T06:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.170+0000 7f9bb602e700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=THROTTLE_DONE pgs=721 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:51.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.170+0000 7f9ba77fe700 1 -- 172.21.15.16:0/3110958046 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f9bac0013f0 con 0x7f9bb0084480 2019-12-05T06:24:51.156 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 91 2019-12-05T06:24:51.156 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:51.156 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 -- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b98023630 msgr2=0x7f9b98025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:51.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b98023630 0x7f9b98025ac0 crc :-1 s=READY pgs=316 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 -- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 msgr2=0x7f9bb009dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 0x7f9bb009dcd0 secure :-1 s=READY pgs=721 cs=0 l=1 rx=0x7f9bac0107d0 tx=0x7f9bac010b70).stop 2019-12-05T06:24:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 -- 172.21.15.16:0/3110958046 shutdown_connections 2019-12-05T06:24:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bb00a1730 0x7f9bb00a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b98023630 0x7f9b98025ac0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 --2- 172.21.15.16:0/3110958046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bb0084480 0x7f9bb009dcd0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:51.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 --2- 172.21.15.16:0/3110958046 >> v2:172.21.15.76:3300/0 conn(0x7f9bb009d3a0 0x7f9bb009d790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:51.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 -- 172.21.15.16:0/3110958046 shutdown_connections 2019-12-05T06:24:51.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 -- 172.21.15.16:0/3110958046 wait complete. 2019-12-05T06:24:51.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:51.173+0000 7f9bb8292700 1 -- 172.21.15.16:0/3110958046 >> 172.21.15.16:0/3110958046 conn(0x7f9bb0071480 msgr2=0x7f9bb0070710 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:51.169 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:24:53.170 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.334+0000 7f972d111700 1 Processor -- start 2019-12-05T06:24:53.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.334+0000 7f972d111700 1 -- start start 2019-12-05T06:24:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.335+0000 7f972d111700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f972808d2c0 0x7f972808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.335+0000 7f972d111700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f972808fef0 0x7f972808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.335+0000 7f972d111700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97280958e0 con 0x7f972808c6d0 2019-12-05T06:24:53.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.335+0000 7f972d111700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f972808df90 con 0x7f972808d2c0 2019-12-05T06:24:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.335+0000 7f972d111700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97280a3f70 con 0x7f972808fef0 2019-12-05T06:24:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.335+0000 7f972c910700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f972808fef0 0x7f972808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.335+0000 7f9726d9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f972808d2c0 0x7f972808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.335+0000 7f972c910700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f972808fef0 0x7f972808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:56724/0 (socket says 172.21.15.16:56724) 2019-12-05T06:24:53.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.335+0000 7f972c910700 1 -- 172.21.15.16:0/2161342414 learned_addr learned my addr 172.21.15.16:0/2161342414 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.335+0000 7f972759e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f972808c6d0 0x7f972808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:59202/0 (socket says 172.21.15.16:59202) 2019-12-05T06:24:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f9726d9d700 1 -- 172.21.15.16:0/2161342414 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f972808fef0 msgr2=0x7f972808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:24:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f9726d9d700 1 --2- 172.21.15.16:0/2161342414 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f972808fef0 0x7f972808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:53.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f9726d9d700 1 -- 172.21.15.16:0/2161342414 >> v1:172.21.15.76:6789/0 conn(0x7f972808c6d0 legacy=0x7f972808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f972659c700 1 -- 172.21.15.16:0/2161342414 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1122077137 0 0) 0x7f9720001400 con 0x7f972808c6d0 2019-12-05T06:24:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f9726d9d700 1 -- 172.21.15.16:0/2161342414 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972808e620 con 0x7f972808d2c0 2019-12-05T06:24:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f9726d9d700 1 --2- 172.21.15.16:0/2161342414 >> v2:172.21.15.76:3300/0 conn(0x7f972808d2c0 0x7f972808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f971c00af50 tx=0x7f971c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f9726d9d700 1 --2- 172.21.15.16:0/2161342414 >> v2:172.21.15.76:3300/0 conn(0x7f972808d2c0 0x7f972808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f971c00af50 tx=0x7f971c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f9726d9d700 1 --2- 172.21.15.16:0/2161342414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f972808d2c0 0x7f972808ef20 secure :-1 s=READY pgs=523 cs=0 l=1 rx=0x7f971c00af50 tx=0x7f971c00b6e0).ready entity=mon.1 client_cookie=cf7752a287ca6fc5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f9726d9d700 1 --2- 172.21.15.16:0/2161342414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f972808d2c0 0x7f972808ef20 secure :-1 s=THROTTLE_DONE pgs=523 cs=0 l=1 rx=0x7f971c00af50 tx=0x7f971c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f972659c700 1 -- 172.21.15.16:0/2161342414 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f971c007210 con 0x7f972808d2c0 2019-12-05T06:24:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f9726d9d700 1 --2- 172.21.15.16:0/2161342414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f972808d2c0 0x7f972808ef20 secure :-1 s=THROTTLE_DONE pgs=523 cs=0 l=1 rx=0x7f971c00af50 tx=0x7f971c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f972659c700 1 -- 172.21.15.16:0/2161342414 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971c000940 con 0x7f972808d2c0 2019-12-05T06:24:53.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f9726d9d700 1 --2- 172.21.15.16:0/2161342414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f972808d2c0 0x7f972808ef20 secure :-1 s=THROTTLE_DONE pgs=523 cs=0 l=1 rx=0x7f971c00af50 tx=0x7f971c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f972659c700 1 -- 172.21.15.16:0/2161342414 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f971c009ba0 con 0x7f972808d2c0 2019-12-05T06:24:53.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f972d111700 1 -- 172.21.15.16:0/2161342414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f972808d2c0 msgr2=0x7f972808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:53.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.336+0000 7f972d111700 1 --2- 172.21.15.16:0/2161342414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f972808d2c0 0x7f972808ef20 secure :-1 s=READY pgs=523 cs=0 l=1 rx=0x7f971c00af50 tx=0x7f971c00b6e0).stop 2019-12-05T06:24:53.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.337+0000 7f972d111700 1 -- 172.21.15.16:0/2161342414 shutdown_connections 2019-12-05T06:24:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.337+0000 7f972d111700 1 --2- 172.21.15.16:0/2161342414 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f972808fef0 0x7f972808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:24:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.337+0000 7f972d111700 1 --2- 172.21.15.16:0/2161342414 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f972808d2c0 0x7f972808ef20 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.337+0000 7f972d111700 1 -- 172.21.15.16:0/2161342414 shutdown_connections 2019-12-05T06:24:53.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.337+0000 7f972d111700 1 -- 172.21.15.16:0/2161342414 wait complete. 2019-12-05T06:24:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.337+0000 7f972d111700 1 -- 172.21.15.16:0/2161342414 >> 172.21.15.16:0/2161342414 conn(0x7f9728061940 msgr2=0x7f9728096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:53.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.339+0000 7f972d111700 1 Processor -- start 2019-12-05T06:24:53.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.340+0000 7f972d111700 1 -- start start 2019-12-05T06:24:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.340+0000 7f972d111700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9728128360 0x7f9728128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:53.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.340+0000 7f972d111700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.340+0000 7f972d111700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9728099720 con 0x7f9728152760 2019-12-05T06:24:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.340+0000 7f972d111700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f972812a640 con 0x7f9728128360 2019-12-05T06:24:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.340+0000 7f972d111700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9728129c10 con 0x7f9728137ea0 2019-12-05T06:24:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.341+0000 7f9726d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:53.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.341+0000 7f972759e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9728128360 0x7f9728128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.341+0000 7f9726d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:56730/0 (socket says 172.21.15.16:56730) 2019-12-05T06:24:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.341+0000 7f9726d9d700 1 -- 172.21.15.16:0/3945882937 learned_addr learned my addr 172.21.15.16:0/3945882937 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.341+0000 7f9726d9d700 1 -- 172.21.15.16:0/3945882937 >> v2:172.21.15.76:3300/0 conn(0x7f9728128360 msgr2=0x7f9728128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:53.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.341+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> v2:172.21.15.76:3300/0 conn(0x7f9728128360 0x7f9728128750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.341+0000 7f9726d9d700 1 -- 172.21.15.16:0/3945882937 >> v1:172.21.15.76:6789/0 conn(0x7f9728152760 legacy=0x7f9728152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.341+0000 7f9726d9d700 1 -- 172.21.15.16:0/3945882937 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f972812acd0 con 0x7f9728137ea0 2019-12-05T06:24:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=READY pgs=174 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f970ffff700 1 -- 172.21.15.16:0/3945882937 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f971c00a9f0 con 0x7f9728137ea0 2019-12-05T06:24:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f972d111700 1 -- 172.21.15.16:0/3945882937 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f972812acd0 con 0x7f9728137ea0 2019-12-05T06:24:53.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f972d111700 1 -- 172.21.15.16:0/3945882937 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f972812acd0 con 0x7f9728137ea0 2019-12-05T06:24:53.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f970ffff700 1 -- 172.21.15.16:0/3945882937 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f971c000940 con 0x7f9728137ea0 2019-12-05T06:24:53.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.342+0000 7f970ffff700 1 -- 172.21.15.16:0/3945882937 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f971c009e00 con 0x7f9728137ea0 2019-12-05T06:24:53.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.343+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.343+0000 7f970ffff700 1 -- 172.21.15.16:0/3945882937 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f971c009790 con 0x7f9728137ea0 2019-12-05T06:24:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.343+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.343+0000 7f970ffff700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9708023630 0x7f9708025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:53.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.343+0000 7f970ffff700 1 -- 172.21.15.16:0/3945882937 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f971c045de0 con 0x7f9728137ea0 2019-12-05T06:24:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.344+0000 7f972759e700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9708023630 0x7f9708025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.344+0000 7f972759e700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9708023630 0x7f9708025ac0 crc :-1 s=READY pgs=317 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:53.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.349+0000 7f972d111700 1 -- 172.21.15.16:0/3945882937 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f972812acd0 con 0x7f9728137ea0 2019-12-05T06:24:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.358+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.358+0000 7f970ffff700 1 -- 172.21.15.16:0/3945882937 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f971c009580 con 0x7f9728137ea0 2019-12-05T06:24:53.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.553+0000 7f972d111700 1 -- 172.21.15.16:0/3945882937 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f972812acd0 con 0x7f9728137ea0 2019-12-05T06:24:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.554+0000 7f9726d9d700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=THROTTLE_DONE pgs=174 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:53.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.554+0000 7f970ffff700 1 -- 172.21.15.16:0/3945882937 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 91 v91) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f971c009790 con 0x7f9728137ea0 2019-12-05T06:24:53.541 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 91 2019-12-05T06:24:53.541 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:53.541 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":91,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:53.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.559+0000 7f972d111700 1 -- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9708023630 msgr2=0x7f9708025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.559+0000 7f972d111700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9708023630 0x7f9708025ac0 crc :-1 s=READY pgs=317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.559+0000 7f972d111700 1 -- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 msgr2=0x7f9728128c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.559+0000 7f972d111700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 secure :-1 s=READY pgs=174 cs=0 l=1 rx=0x7f971c010420 tx=0x7f971c0102a0).stop 2019-12-05T06:24:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.560+0000 7f972d111700 1 -- 172.21.15.16:0/3945882937 shutdown_connections 2019-12-05T06:24:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.560+0000 7f972d111700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9708023630 0x7f9708025ac0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.560+0000 7f972d111700 1 --2- 172.21.15.16:0/3945882937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9728137ea0 0x7f9728128c90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.560+0000 7f972d111700 1 --2- 172.21.15.16:0/3945882937 >> v2:172.21.15.76:3300/0 conn(0x7f9728128360 0x7f9728128750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.560+0000 7f972d111700 1 -- 172.21.15.16:0/3945882937 shutdown_connections 2019-12-05T06:24:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.560+0000 7f972d111700 1 -- 172.21.15.16:0/3945882937 wait complete. 2019-12-05T06:24:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:53.560+0000 7f972d111700 1 -- 172.21.15.16:0/3945882937 >> 172.21.15.16:0/3945882937 conn(0x7f972806d260 msgr2=0x7f972809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:53.554 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:24:55.555 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:24:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.803+0000 7ffb36c81700 1 Processor -- start 2019-12-05T06:24:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.803+0000 7ffb36c81700 1 -- start start 2019-12-05T06:24:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.805+0000 7ffb36c81700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 0x7ffb3008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:24:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.805+0000 7ffb36c81700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300958e0 con 0x7ffb3008c6d0 2019-12-05T06:24:55.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.805+0000 7ffb36c81700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb300629e0 con 0x7ffb3008d2c0 2019-12-05T06:24:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.805+0000 7ffb36c81700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb3008df90 con 0x7ffb3008fef0 2019-12-05T06:24:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.805+0000 7ffb3521e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffb3008c6d0 0x7ffb3008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:37932/0 (socket says 172.21.15.16:37932) 2019-12-05T06:24:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.805+0000 7ffb3521e700 1 -- 172.21.15.16:0/3955031681 learned_addr learned my addr 172.21.15.16:0/3955031681 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:24:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.805+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/3955031681 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 0x7ffb3008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.806+0000 7ffb2ffff700 1 -- 172.21.15.16:0/3955031681 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2286226216 0 0) 0x7ffb20001150 con 0x7ffb3008c6d0 2019-12-05T06:24:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.806+0000 7ffb2ffff700 1 -- 172.21.15.16:0/3955031681 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18002e80 con 0x7ffb3008c6d0 2019-12-05T06:24:55.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.806+0000 7ffb2ffff700 1 -- 172.21.15.16:0/3955031681 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1740482979 0 0) 0x7ffb24001400 con 0x7ffb3008d2c0 2019-12-05T06:24:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.806+0000 7ffb2ffff700 1 -- 172.21.15.16:0/3955031681 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffb18004170 con 0x7ffb3008d2c0 2019-12-05T06:24:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.806+0000 7ffb35a1f700 1 -- 172.21.15.16:0/3955031681 >> v1:172.21.15.76:6789/0 conn(0x7ffb3008d2c0 legacy=0x7ffb3008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.806+0000 7ffb35a1f700 1 -- 172.21.15.16:0/3955031681 >> v1:172.21.15.16:6789/0 conn(0x7ffb3008c6d0 legacy=0x7ffb3008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.806+0000 7ffb35a1f700 1 -- 172.21.15.16:0/3955031681 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3008e620 con 0x7ffb3008fef0 2019-12-05T06:24:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.806+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/3955031681 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 0x7ffb3008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffb2800af80 tx=0x7ffb2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.806+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/3955031681 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 0x7ffb3008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffb2800af80 tx=0x7ffb2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.806+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/3955031681 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 0x7ffb3008f720 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7ffb2800af80 tx=0x7ffb2800b790).ready entity=mon.2 client_cookie=358d3b50402e1100 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.807+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/3955031681 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 0x7ffb3008f720 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7ffb2800af80 tx=0x7ffb2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.807+0000 7ffb2ffff700 1 -- 172.21.15.16:0/3955031681 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb28007240 con 0x7ffb3008fef0 2019-12-05T06:24:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.807+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/3955031681 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 0x7ffb3008f720 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7ffb2800af80 tx=0x7ffb2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.807+0000 7ffb2ffff700 1 -- 172.21.15.16:0/3955031681 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb28000bb0 con 0x7ffb3008fef0 2019-12-05T06:24:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.807+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/3955031681 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 0x7ffb3008f720 secure :-1 s=THROTTLE_DONE pgs=175 cs=0 l=1 rx=0x7ffb2800af80 tx=0x7ffb2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.807+0000 7ffb2ffff700 1 -- 172.21.15.16:0/3955031681 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb28009bd0 con 0x7ffb3008fef0 2019-12-05T06:24:55.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.807+0000 7ffb36c81700 1 -- 172.21.15.16:0/3955031681 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 msgr2=0x7ffb3008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.807+0000 7ffb36c81700 1 --2- 172.21.15.16:0/3955031681 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 0x7ffb3008f720 secure :-1 s=READY pgs=175 cs=0 l=1 rx=0x7ffb2800af80 tx=0x7ffb2800b790).stop 2019-12-05T06:24:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.809+0000 7ffb36c81700 1 -- 172.21.15.16:0/3955031681 shutdown_connections 2019-12-05T06:24:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.809+0000 7ffb36c81700 1 --2- 172.21.15.16:0/3955031681 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb3008fef0 0x7ffb3008f720 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.809+0000 7ffb36c81700 1 -- 172.21.15.16:0/3955031681 shutdown_connections 2019-12-05T06:24:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.809+0000 7ffb36c81700 1 -- 172.21.15.16:0/3955031681 wait complete. 2019-12-05T06:24:55.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.809+0000 7ffb36c81700 1 -- 172.21.15.16:0/3955031681 >> 172.21.15.16:0/3955031681 conn(0x7ffb30061940 msgr2=0x7ffb30096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.814+0000 7ffb36c81700 1 Processor -- start 2019-12-05T06:24:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.814+0000 7ffb36c81700 1 -- start start 2019-12-05T06:24:55.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.816+0000 7ffb36c81700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ffb30128360 0x7ffb30128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:55.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.816+0000 7ffb36c81700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb30137ea0 0x7ffb30128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:55.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.816+0000 7ffb36c81700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb3012b5d0 0x7ffb3012ba30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.816+0000 7ffb36c81700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb3012a640 con 0x7ffb3012b5d0 2019-12-05T06:24:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.816+0000 7ffb36c81700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb30129c10 con 0x7ffb30128360 2019-12-05T06:24:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.816+0000 7ffb36c81700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb30129ee0 con 0x7ffb30137ea0 2019-12-05T06:24:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.816+0000 7ffb35a1f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb3012b5d0 0x7ffb3012ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.816+0000 7ffb35a1f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffb3012b5d0 0x7ffb3012ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:52772/0 (socket says 172.21.15.16:52772) 2019-12-05T06:24:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.816+0000 7ffb35a1f700 1 -- 172.21.15.16:0/2892129853 learned_addr learned my addr 172.21.15.16:0/2892129853 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:24:55.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.816+0000 7ffb3521e700 1 --2- 172.21.15.16:0/2892129853 >> v2:172.21.15.76:3300/0 conn(0x7ffb30128360 0x7ffb30128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.817+0000 7ffb34a1d700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb30137ea0 0x7ffb30128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.817+0000 7ffb35a1f700 1 -- 172.21.15.16:0/2892129853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb30137ea0 msgr2=0x7ffb30128c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.817+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb30137ea0 0x7ffb30128c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.817+0000 7ffb35a1f700 1 -- 172.21.15.16:0/2892129853 >> v2:172.21.15.76:3300/0 conn(0x7ffb30128360 msgr2=0x7ffb30128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.817+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> v2:172.21.15.76:3300/0 conn(0x7ffb30128360 0x7ffb30128750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.817+0000 7ffb35a1f700 1 -- 172.21.15.16:0/2892129853 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb3012acd0 con 0x7ffb3012b5d0 2019-12-05T06:24:55.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.817+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> v2:172.21.15.16:3300/0 conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.817+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> v2:172.21.15.16:3300/0 conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.817+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=READY pgs=726 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.818+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.818+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.818+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.818+0000 7ffb2dffb700 1 -- 172.21.15.16:0/2892129853 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb2800afc0 con 0x7ffb3012b5d0 2019-12-05T06:24:55.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.818+0000 7ffb2dffb700 1 -- 172.21.15.16:0/2892129853 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb280099b0 con 0x7ffb3012b5d0 2019-12-05T06:24:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.818+0000 7ffb2dffb700 1 -- 172.21.15.16:0/2892129853 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb2800a040 con 0x7ffb3012b5d0 2019-12-05T06:24:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.818+0000 7ffb36c81700 1 -- 172.21.15.16:0/2892129853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb3012acd0 con 0x7ffb3012b5d0 2019-12-05T06:24:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.818+0000 7ffb36c81700 1 -- 172.21.15.16:0/2892129853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb3012acd0 con 0x7ffb3012b5d0 2019-12-05T06:24:55.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.819+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.820+0000 7ffb2dffb700 1 -- 172.21.15.16:0/2892129853 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ffb280095b0 con 0x7ffb3012b5d0 2019-12-05T06:24:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.820+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.821+0000 7ffb2dffb700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb10023630 0x7ffb10025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:24:55.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.821+0000 7ffb2dffb700 1 -- 172.21.15.16:0/2892129853 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7ffb28045d60 con 0x7ffb3012b5d0 2019-12-05T06:24:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.821+0000 7ffb3521e700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb10023630 0x7ffb10025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:24:55.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.822+0000 7ffb3521e700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb10023630 0x7ffb10025ac0 crc :-1 s=READY pgs=319 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:24:55.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.834+0000 7ffb36c81700 1 -- 172.21.15.16:0/2892129853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb3012acd0 con 0x7ffb3012b5d0 2019-12-05T06:24:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.841+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:55.841+0000 7ffb2dffb700 1 -- 172.21.15.16:0/2892129853 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ffb28028fe0 con 0x7ffb3012b5d0 2019-12-05T06:24:56.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.096+0000 7ffb36c81700 1 -- 172.21.15.16:0/2892129853 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffb3012acd0 con 0x7ffb3012b5d0 2019-12-05T06:24:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.097+0000 7ffb35a1f700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=THROTTLE_DONE pgs=726 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:24:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.097+0000 7ffb2dffb700 1 -- 172.21.15.16:0/2892129853 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2108 (secure 0 0 0) 0x7ffb28028fe0 con 0x7ffb3012b5d0 2019-12-05T06:24:56.082 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 92 2019-12-05T06:24:56.082 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:24:56.082 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:24:56.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.100+0000 7ffb36c81700 1 -- 172.21.15.16:0/2892129853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb10023630 msgr2=0x7ffb10025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:56.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.100+0000 7ffb36c81700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb10023630 0x7ffb10025ac0 crc :-1 s=READY pgs=319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:56.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.100+0000 7ffb36c81700 1 -- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 msgr2=0x7ffb3012ba30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:24:56.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.100+0000 7ffb36c81700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 0x7ffb3012ba30 secure :-1 s=READY pgs=726 cs=0 l=1 rx=0x7ffb28010440 tx=0x7ffb28010340).stop 2019-12-05T06:24:56.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.100+0000 7ffb36c81700 1 -- 172.21.15.16:0/2892129853 shutdown_connections 2019-12-05T06:24:56.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.100+0000 7ffb36c81700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffb10023630 0x7ffb10025ac0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:56.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.100+0000 7ffb36c81700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffb3012b5d0 0x7ffb3012ba30 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:56.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.100+0000 7ffb36c81700 1 --2- 172.21.15.16:0/2892129853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffb30137ea0 0x7ffb30128c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:56.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.100+0000 7ffb36c81700 1 --2- 172.21.15.16:0/2892129853 >> v2:172.21.15.76:3300/0 conn(0x7ffb30128360 0x7ffb30128750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:24:56.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.101+0000 7ffb36c81700 1 -- 172.21.15.16:0/2892129853 shutdown_connections 2019-12-05T06:24:56.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.101+0000 7ffb36c81700 1 -- 172.21.15.16:0/2892129853 wait complete. 2019-12-05T06:24:56.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:24:56.101+0000 7ffb36c81700 1 -- 172.21.15.16:0/2892129853 >> 172.21.15.16:0/2892129853 conn(0x7ffb3006d260 msgr2=0x7ffb3009aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:24:56.103 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:24:56.103 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 70 secs before thrashing 2019-12-05T06:25:03.512 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:25:03.517 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:25:33.679 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:25:33.684 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:26:03.924 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:26:03.931 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:26:06.104 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:06.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.347+0000 7f11b6964700 1 Processor -- start 2019-12-05T06:26:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.347+0000 7f11b6964700 1 -- start start 2019-12-05T06:26:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11b6964700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f11b008c6d0 0x7f11b008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11b6964700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11b008fef0 0x7f11b008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11b6964700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b008ef20 0x7f11b008f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11b6964700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b008df90 con 0x7f11b008fef0 2019-12-05T06:26:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11b6964700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b00a3f70 con 0x7f11b008c6d0 2019-12-05T06:26:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11b6964700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b008d830 con 0x7f11b008ef20 2019-12-05T06:26:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11affff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11b008fef0 0x7f11b008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11affff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f11b008fef0 0x7f11b008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53090/0 (socket says 172.21.15.16:53090) 2019-12-05T06:26:06.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11affff700 1 -- 172.21.15.16:0/2856108997 learned_addr learned my addr 172.21.15.16:0/2856108997 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:06.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2856108997 >> v2:172.21.15.76:3300/0 conn(0x7f11b008c6d0 0x7f11b008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:06.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11b5702700 1 --2- 172.21.15.16:0/2856108997 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b008ef20 0x7f11b008f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:06.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11affff700 1 -- 172.21.15.16:0/2856108997 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b008ef20 msgr2=0x7f11b008f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:06.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11affff700 1 --2- 172.21.15.16:0/2856108997 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b008ef20 0x7f11b008f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:06.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11affff700 1 -- 172.21.15.16:0/2856108997 >> v2:172.21.15.76:3300/0 conn(0x7f11b008c6d0 msgr2=0x7f11b008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:06.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11affff700 1 --2- 172.21.15.16:0/2856108997 >> v2:172.21.15.76:3300/0 conn(0x7f11b008c6d0 0x7f11b008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:06.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11affff700 1 -- 172.21.15.16:0/2856108997 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b008d540 con 0x7f11b008fef0 2019-12-05T06:26:06.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11b5702700 1 --2- 172.21.15.16:0/2856108997 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b008ef20 0x7f11b008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:26:06.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2856108997 >> v2:172.21.15.76:3300/0 conn(0x7f11b008c6d0 0x7f11b008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:26:06.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.348+0000 7f11affff700 1 --2- 172.21.15.16:0/2856108997 >> v2:172.21.15.16:3300/0 conn(0x7f11b008fef0 0x7f11b008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f11a800f740 tx=0x7f11a800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.349+0000 7f11affff700 1 --2- 172.21.15.16:0/2856108997 >> v2:172.21.15.16:3300/0 conn(0x7f11b008fef0 0x7f11b008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f11a800f740 tx=0x7f11a800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.349+0000 7f11affff700 1 --2- 172.21.15.16:0/2856108997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11b008fef0 0x7f11b008d000 secure :-1 s=READY pgs=735 cs=0 l=1 rx=0x7f11a800f740 tx=0x7f11a800fae0).ready entity=mon.0 client_cookie=8f4f115562bc21c2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:06.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.349+0000 7f11affff700 1 --2- 172.21.15.16:0/2856108997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11b008fef0 0x7f11b008d000 secure :-1 s=THROTTLE_DONE pgs=735 cs=0 l=1 rx=0x7f11a800f740 tx=0x7f11a800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.349+0000 7f11affff700 1 --2- 172.21.15.16:0/2856108997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11b008fef0 0x7f11b008d000 secure :-1 s=THROTTLE_DONE pgs=735 cs=0 l=1 rx=0x7f11a800f740 tx=0x7f11a800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.349+0000 7f11af7fe700 1 -- 172.21.15.16:0/2856108997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11a8000960 con 0x7f11b008fef0 2019-12-05T06:26:06.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.349+0000 7f11affff700 1 --2- 172.21.15.16:0/2856108997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11b008fef0 0x7f11b008d000 secure :-1 s=THROTTLE_DONE pgs=735 cs=0 l=1 rx=0x7f11a800f740 tx=0x7f11a800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.349+0000 7f11af7fe700 1 -- 172.21.15.16:0/2856108997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f11a8000e80 con 0x7f11b008fef0 2019-12-05T06:26:06.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.349+0000 7f11af7fe700 1 -- 172.21.15.16:0/2856108997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11a80015d0 con 0x7f11b008fef0 2019-12-05T06:26:06.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.349+0000 7f11b6964700 1 -- 172.21.15.16:0/2856108997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11b008fef0 msgr2=0x7f11b008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:06.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.349+0000 7f11b6964700 1 --2- 172.21.15.16:0/2856108997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11b008fef0 0x7f11b008d000 secure :-1 s=READY pgs=735 cs=0 l=1 rx=0x7f11a800f740 tx=0x7f11a800fae0).stop 2019-12-05T06:26:06.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.351+0000 7f11b6964700 1 -- 172.21.15.16:0/2856108997 shutdown_connections 2019-12-05T06:26:06.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.351+0000 7f11b6964700 1 --2- 172.21.15.16:0/2856108997 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b008ef20 0x7f11b008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:06.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.351+0000 7f11b6964700 1 --2- 172.21.15.16:0/2856108997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f11b008fef0 0x7f11b008d000 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:06.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.351+0000 7f11b6964700 1 --2- 172.21.15.16:0/2856108997 >> v2:172.21.15.76:3300/0 conn(0x7f11b008c6d0 0x7f11b008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:06.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.351+0000 7f11b6964700 1 -- 172.21.15.16:0/2856108997 shutdown_connections 2019-12-05T06:26:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.351+0000 7f11b6964700 1 -- 172.21.15.16:0/2856108997 wait complete. 2019-12-05T06:26:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.351+0000 7f11b6964700 1 -- 172.21.15.16:0/2856108997 >> 172.21.15.16:0/2856108997 conn(0x7f11b0061940 msgr2=0x7f11b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:06.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.353+0000 7f11b6964700 1 Processor -- start 2019-12-05T06:26:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.354+0000 7f11b6964700 1 -- start start 2019-12-05T06:26:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.355+0000 7f11b6964700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.355+0000 7f11b6964700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f11b00873d0 0x7f11b0086400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.355+0000 7f11b6964700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11b00a1010 con 0x7f11b0086940 2019-12-05T06:26:06.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.355+0000 7f11b6964700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11b0085190 con 0x7f11b00873d0 2019-12-05T06:26:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.355+0000 7f11b6964700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f11b009e9b0 con 0x7f11b014f640 2019-12-05T06:26:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.355+0000 7f11b4f01700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.355+0000 7f11affff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f11b00873d0 0x7f11b0086400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.355+0000 7f11b4f01700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:57086/0 (socket says 172.21.15.16:57086) 2019-12-05T06:26:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.355+0000 7f11b4f01700 1 -- 172.21.15.16:0/2842932653 learned_addr learned my addr 172.21.15.16:0/2842932653 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:06.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b4f01700 1 -- 172.21.15.16:0/2842932653 >> v2:172.21.15.76:3300/0 conn(0x7f11b00873d0 msgr2=0x7f11b0086400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> v2:172.21.15.76:3300/0 conn(0x7f11b00873d0 0x7f11b0086400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b4f01700 1 -- 172.21.15.16:0/2842932653 >> v1:172.21.15.76:6789/0 conn(0x7f11b0086940 legacy=0x7f11b009ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b4f01700 1 -- 172.21.15.16:0/2842932653 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11b0085b00 con 0x7f11b014f640 2019-12-05T06:26:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11affff700 1 --2- 172.21.15.16:0/2842932653 >> v2:172.21.15.76:3300/0 conn(0x7f11b00873d0 0x7f11b0086400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:26:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=READY pgs=184 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:06.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11ad7fa700 1 -- 172.21.15.16:0/2842932653 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11980097a0 con 0x7f11b014f640 2019-12-05T06:26:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11ad7fa700 1 -- 172.21.15.16:0/2842932653 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1198000ba0 con 0x7f11b014f640 2019-12-05T06:26:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.356+0000 7f11b6964700 1 -- 172.21.15.16:0/2842932653 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11b0085b00 con 0x7f11b014f640 2019-12-05T06:26:06.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.357+0000 7f11ad7fa700 1 -- 172.21.15.16:0/2842932653 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f11980097a0 con 0x7f11b014f640 2019-12-05T06:26:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.357+0000 7f11b6964700 1 -- 172.21.15.16:0/2842932653 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11b0085b00 con 0x7f11b014f640 2019-12-05T06:26:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.357+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.358+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.358+0000 7f11ad7fa700 1 -- 172.21.15.16:0/2842932653 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1198000940 con 0x7f11b014f640 2019-12-05T06:26:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.358+0000 7f11ad7fa700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1194023630 0x7f1194025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.358+0000 7f11ad7fa700 1 -- 172.21.15.16:0/2842932653 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f1198045df0 con 0x7f11b014f640 2019-12-05T06:26:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.359+0000 7f11affff700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1194023630 0x7f1194025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:06.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.359+0000 7f11affff700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1194023630 0x7f1194025ac0 crc :-1 s=READY pgs=322 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:06.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.362+0000 7f11b6964700 1 -- 172.21.15.16:0/2842932653 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11b0085b00 con 0x7f11b014f640 2019-12-05T06:26:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.371+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.371+0000 7f11ad7fa700 1 -- 172.21.15.16:0/2842932653 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1198045b40 con 0x7f11b014f640 2019-12-05T06:26:06.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.571+0000 7f11b6964700 1 -- 172.21.15.16:0/2842932653 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11b0085b00 con 0x7f11b014f640 2019-12-05T06:26:06.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.572+0000 7f11b4f01700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=THROTTLE_DONE pgs=184 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.572+0000 7f11ad7fa700 1 -- 172.21.15.16:0/2842932653 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f1198000940 con 0x7f11b014f640 2019-12-05T06:26:06.559 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 92 2019-12-05T06:26:06.561 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:06.561 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:06.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.577+0000 7f11b6964700 1 -- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1194023630 msgr2=0x7f1194025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:06.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.577+0000 7f11b6964700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1194023630 0x7f1194025ac0 crc :-1 s=READY pgs=322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:06.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.577+0000 7f11b6964700 1 -- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 msgr2=0x7f11b014fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:06.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.577+0000 7f11b6964700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 secure :-1 s=READY pgs=184 cs=0 l=1 rx=0x7f1198010410 tx=0x7f11980107b0).stop 2019-12-05T06:26:06.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.577+0000 7f11b6964700 1 -- 172.21.15.16:0/2842932653 shutdown_connections 2019-12-05T06:26:06.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.577+0000 7f11b6964700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1194023630 0x7f1194025ac0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:06.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.577+0000 7f11b6964700 1 --2- 172.21.15.16:0/2842932653 >> v2:172.21.15.76:3300/0 conn(0x7f11b00873d0 0x7f11b0086400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:06.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.577+0000 7f11b6964700 1 --2- 172.21.15.16:0/2842932653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f11b014f640 0x7f11b014fa30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:06.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.577+0000 7f11b6964700 1 -- 172.21.15.16:0/2842932653 shutdown_connections 2019-12-05T06:26:06.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.578+0000 7f11b6964700 1 -- 172.21.15.16:0/2842932653 wait complete. 2019-12-05T06:26:06.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.578+0000 7f11b6964700 1 -- 172.21.15.16:0/2842932653 >> 172.21.15.16:0/2842932653 conn(0x7f11b009f840 msgr2=0x7f11b00706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:06.578 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:26:06.578 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:26:06.580 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:26:06.592 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:26:06.593 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.949+0000 7f6a43edb700 1 Processor -- start 2019-12-05T06:26:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.949+0000 7f6a43edb700 1 -- start start 2019-12-05T06:26:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.951+0000 7f6a43edb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a3c08c6d0 0x7f6a3c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.951+0000 7f6a43edb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a3c08fb80 con 0x7f6a3c08f720 2019-12-05T06:26:06.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.951+0000 7f6a43edb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a3c0958e0 con 0x7f6a3c08fef0 2019-12-05T06:26:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.951+0000 7f6a43edb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a3c08df90 con 0x7f6a3c08c6d0 2019-12-05T06:26:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.951+0000 7f6a42478700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a3c08c6d0 0x7f6a3c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.951+0000 7f6a42c79700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6a3c08f720 0x7f6a3c1533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38280/0 (socket says 172.21.15.16:38280) 2019-12-05T06:26:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.951+0000 7f6a42c79700 1 -- 172.21.15.16:0/3443181729 learned_addr learned my addr 172.21.15.16:0/3443181729 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.952+0000 7f6a41476700 1 -- 172.21.15.16:0/3443181729 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 662891551 0 0) 0x7f6a30001080 con 0x7f6a3c08fef0 2019-12-05T06:26:06.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.952+0000 7f6a41476700 1 -- 172.21.15.16:0/3443181729 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a28002e80 con 0x7f6a3c08fef0 2019-12-05T06:26:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.952+0000 7f6a42478700 1 -- 172.21.15.16:0/3443181729 >> v1:172.21.15.76:6789/0 conn(0x7f6a3c08fef0 legacy=0x7f6a3c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.952+0000 7f6a41476700 1 -- 172.21.15.16:0/3443181729 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1591082040 0 0) 0x7f6a340014d0 con 0x7f6a3c08f720 2019-12-05T06:26:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.952+0000 7f6a42478700 1 -- 172.21.15.16:0/3443181729 >> v1:172.21.15.16:6789/0 conn(0x7f6a3c08f720 legacy=0x7f6a3c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.952+0000 7f6a42478700 1 -- 172.21.15.16:0/3443181729 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c08e620 con 0x7f6a3c08c6d0 2019-12-05T06:26:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.952+0000 7f6a42478700 1 --2- 172.21.15.16:0/3443181729 >> v2:172.21.15.16:3300/0 conn(0x7f6a3c08c6d0 0x7f6a3c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6a3800af50 tx=0x7f6a3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.953+0000 7f6a42478700 1 --2- 172.21.15.16:0/3443181729 >> v2:172.21.15.16:3300/0 conn(0x7f6a3c08c6d0 0x7f6a3c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6a3800af50 tx=0x7f6a3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.953+0000 7f6a42478700 1 --2- 172.21.15.16:0/3443181729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c08c6d0 0x7f6a3c08cac0 secure :-1 s=READY pgs=737 cs=0 l=1 rx=0x7f6a3800af50 tx=0x7f6a3800b760).ready entity=mon.0 client_cookie=152301b085fc25da server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.953+0000 7f6a42478700 1 --2- 172.21.15.16:0/3443181729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c08c6d0 0x7f6a3c08cac0 secure :-1 s=THROTTLE_DONE pgs=737 cs=0 l=1 rx=0x7f6a3800af50 tx=0x7f6a3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.953+0000 7f6a42478700 1 --2- 172.21.15.16:0/3443181729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c08c6d0 0x7f6a3c08cac0 secure :-1 s=THROTTLE_DONE pgs=737 cs=0 l=1 rx=0x7f6a3800af50 tx=0x7f6a3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.953+0000 7f6a41476700 1 -- 172.21.15.16:0/3443181729 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a38000960 con 0x7f6a3c08c6d0 2019-12-05T06:26:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.953+0000 7f6a42478700 1 --2- 172.21.15.16:0/3443181729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c08c6d0 0x7f6a3c08cac0 secure :-1 s=THROTTLE_DONE pgs=737 cs=0 l=1 rx=0x7f6a3800af50 tx=0x7f6a3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.953+0000 7f6a41476700 1 -- 172.21.15.16:0/3443181729 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a38009860 con 0x7f6a3c08c6d0 2019-12-05T06:26:06.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.953+0000 7f6a41476700 1 -- 172.21.15.16:0/3443181729 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a38009f30 con 0x7f6a3c08c6d0 2019-12-05T06:26:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.953+0000 7f6a43edb700 1 -- 172.21.15.16:0/3443181729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c08c6d0 msgr2=0x7f6a3c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.953+0000 7f6a43edb700 1 --2- 172.21.15.16:0/3443181729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c08c6d0 0x7f6a3c08cac0 secure :-1 s=READY pgs=737 cs=0 l=1 rx=0x7f6a3800af50 tx=0x7f6a3800b760).stop 2019-12-05T06:26:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.955+0000 7f6a43edb700 1 -- 172.21.15.16:0/3443181729 shutdown_connections 2019-12-05T06:26:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.955+0000 7f6a43edb700 1 --2- 172.21.15.16:0/3443181729 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c08c6d0 0x7f6a3c08cac0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.955+0000 7f6a43edb700 1 -- 172.21.15.16:0/3443181729 shutdown_connections 2019-12-05T06:26:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.955+0000 7f6a43edb700 1 -- 172.21.15.16:0/3443181729 wait complete. 2019-12-05T06:26:06.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.955+0000 7f6a43edb700 1 -- 172.21.15.16:0/3443181729 >> 172.21.15.16:0/3443181729 conn(0x7f6a3c061940 msgr2=0x7f6a3c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:06.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.961+0000 7f6a43edb700 1 Processor -- start 2019-12-05T06:26:06.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.961+0000 7f6a43edb700 1 -- start start 2019-12-05T06:26:06.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.963+0000 7f6a43edb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:06.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.963+0000 7f6a43edb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a3c098b00 0x7f6a3c1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:06.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.963+0000 7f6a43edb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a3c089fb0 con 0x7f6a3c14f5b0 2019-12-05T06:26:06.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.963+0000 7f6a43edb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a3c097370 con 0x7f6a3c0992d0 2019-12-05T06:26:06.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.963+0000 7f6a43edb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3c08ad00 con 0x7f6a3c098b00 2019-12-05T06:26:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.963+0000 7f6a41c77700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.963+0000 7f6a42478700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f6a3c14f5b0 0x7f6a3c098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:59584/0 (socket says 172.21.15.16:59584) 2019-12-05T06:26:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.963+0000 7f6a42478700 1 -- 172.21.15.16:0/966970268 learned_addr learned my addr 172.21.15.16:0/966970268 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.963+0000 7f6a42c79700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a3c098b00 0x7f6a3c1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.964+0000 7f6a2effd700 1 -- 172.21.15.16:0/966970268 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855544626 0 0) 0x7f6a3800ac20 con 0x7f6a3c14f5b0 2019-12-05T06:26:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.964+0000 7f6a2effd700 1 -- 172.21.15.16:0/966970268 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a24002e80 con 0x7f6a3c14f5b0 2019-12-05T06:26:06.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.964+0000 7f6a41c77700 1 -- 172.21.15.16:0/966970268 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a3c098b00 msgr2=0x7f6a3c1592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.964+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a3c098b00 0x7f6a3c1592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.964+0000 7f6a41c77700 1 -- 172.21.15.16:0/966970268 >> v1:172.21.15.76:6789/0 conn(0x7f6a3c14f5b0 legacy=0x7f6a3c098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.964+0000 7f6a41c77700 1 -- 172.21.15.16:0/966970268 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3c097a00 con 0x7f6a3c0992d0 2019-12-05T06:26:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.964+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> v2:172.21.15.16:3300/0 conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.964+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> v2:172.21.15.16:3300/0 conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.964+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=READY pgs=738 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:06.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.965+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=THROTTLE_DONE pgs=738 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.965+0000 7f6a2effd700 1 -- 172.21.15.16:0/966970268 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a300009f0 con 0x7f6a3c0992d0 2019-12-05T06:26:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.965+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=THROTTLE_DONE pgs=738 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.965+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=THROTTLE_DONE pgs=738 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.965+0000 7f6a2effd700 1 -- 172.21.15.16:0/966970268 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3000efb0 con 0x7f6a3c0992d0 2019-12-05T06:26:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.965+0000 7f6a2effd700 1 -- 172.21.15.16:0/966970268 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a3000f700 con 0x7f6a3c0992d0 2019-12-05T06:26:06.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.965+0000 7f6a43edb700 1 -- 172.21.15.16:0/966970268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3c097a00 con 0x7f6a3c0992d0 2019-12-05T06:26:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.965+0000 7f6a43edb700 1 -- 172.21.15.16:0/966970268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3c097a00 con 0x7f6a3c0992d0 2019-12-05T06:26:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.966+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=THROTTLE_DONE pgs=738 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.967+0000 7f6a2effd700 1 -- 172.21.15.16:0/966970268 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6a3000ec80 con 0x7f6a3c0992d0 2019-12-05T06:26:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.967+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=THROTTLE_DONE pgs=738 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.967+0000 7f6a2effd700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a24023670 0x7f6a24025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:06.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.967+0000 7f6a2effd700 1 -- 172.21.15.16:0/966970268 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f6a30045d60 con 0x7f6a3c0992d0 2019-12-05T06:26:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.967+0000 7f6a42478700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a24023670 0x7f6a24025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:06.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.968+0000 7f6a42478700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a24023670 0x7f6a24025ae0 crc :-1 s=READY pgs=323 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:06.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.980+0000 7f6a43edb700 1 -- 172.21.15.16:0/966970268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3c097a00 con 0x7f6a3c0992d0 2019-12-05T06:26:06.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.988+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=THROTTLE_DONE pgs=738 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:06.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:06.988+0000 7f6a2effd700 1 -- 172.21.15.16:0/966970268 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6a3000ea30 con 0x7f6a3c0992d0 2019-12-05T06:26:07.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.260+0000 7f6a43edb700 1 -- 172.21.15.16:0/966970268 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a3c097a00 con 0x7f6a3c0992d0 2019-12-05T06:26:07.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.261+0000 7f6a41c77700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=THROTTLE_DONE pgs=738 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:07.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.261+0000 7f6a2effd700 1 -- 172.21.15.16:0/966970268 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f6a300009f0 con 0x7f6a3c0992d0 2019-12-05T06:26:07.246 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 92 2019-12-05T06:26:07.246 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:07.247 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 -- 172.21.15.16:0/966970268 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a24023670 msgr2=0x7f6a24025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a24023670 0x7f6a24025ae0 crc :-1 s=READY pgs=323 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 -- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 msgr2=0x7f6a3c14f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:07.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 secure :-1 s=READY pgs=738 cs=0 l=1 rx=0x7f6a300103a0 tx=0x7f6a3000b2e0).stop 2019-12-05T06:26:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 -- 172.21.15.16:0/966970268 shutdown_connections 2019-12-05T06:26:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6a3c098b00 0x7f6a3c1592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a24023670 0x7f6a24025ae0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 --2- 172.21.15.16:0/966970268 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3c0992d0 0x7f6a3c14f9a0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 -- 172.21.15.16:0/966970268 shutdown_connections 2019-12-05T06:26:07.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 -- 172.21.15.16:0/966970268 wait complete. 2019-12-05T06:26:07.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:07.264+0000 7f6a43edb700 1 -- 172.21.15.16:0/966970268 >> 172.21.15.16:0/966970268 conn(0x7f6a3c0714f0 msgr2=0x7f6a3c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:07.260 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:26:09.261 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:09.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.563+0000 7f1cd5e8b700 1 Processor -- start 2019-12-05T06:26:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.563+0000 7f1cd5e8b700 1 -- start start 2019-12-05T06:26:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.565+0000 7f1cd5e8b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1cd008c6d0 0x7f1cd008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.565+0000 7f1cd5e8b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1cd008fef0 0x7f1cd008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.565+0000 7f1cd5e8b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1cd008ef20 0x7f1cd008f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.565+0000 7f1cd5e8b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd008df90 con 0x7f1cd008c6d0 2019-12-05T06:26:09.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.565+0000 7f1cd5e8b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd00a3f70 con 0x7f1cd008fef0 2019-12-05T06:26:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.565+0000 7f1cd5e8b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd008d830 con 0x7f1cd008ef20 2019-12-05T06:26:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.565+0000 7f1ccffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1cd008c6d0 0x7f1cd008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.565+0000 7f1ccffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1cd008c6d0 0x7f1cd008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53128/0 (socket says 172.21.15.16:53128) 2019-12-05T06:26:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.566+0000 7f1ccffff700 1 -- 172.21.15.16:0/606839977 learned_addr learned my addr 172.21.15.16:0/606839977 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.566+0000 7f1ccf7fe700 1 --2- 172.21.15.16:0/606839977 >> v2:172.21.15.76:3300/0 conn(0x7f1cd008fef0 0x7f1cd008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.566+0000 7f1cd4c29700 1 --2- 172.21.15.16:0/606839977 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1cd008ef20 0x7f1cd008f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.566+0000 7f1ccffff700 1 -- 172.21.15.16:0/606839977 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1cd008ef20 msgr2=0x7f1cd008f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.566+0000 7f1ccffff700 1 --2- 172.21.15.16:0/606839977 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1cd008ef20 0x7f1cd008f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.566+0000 7f1ccffff700 1 -- 172.21.15.16:0/606839977 >> v2:172.21.15.76:3300/0 conn(0x7f1cd008fef0 msgr2=0x7f1cd008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.566+0000 7f1ccffff700 1 --2- 172.21.15.16:0/606839977 >> v2:172.21.15.76:3300/0 conn(0x7f1cd008fef0 0x7f1cd008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.566+0000 7f1ccffff700 1 -- 172.21.15.16:0/606839977 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd008d540 con 0x7f1cd008c6d0 2019-12-05T06:26:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.566+0000 7f1ccf7fe700 1 --2- 172.21.15.16:0/606839977 >> v2:172.21.15.76:3300/0 conn(0x7f1cd008fef0 0x7f1cd008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:26:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.567+0000 7f1ccffff700 1 --2- 172.21.15.16:0/606839977 >> v2:172.21.15.16:3300/0 conn(0x7f1cd008c6d0 0x7f1cd008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1cc000fac0 tx=0x7f1cc000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.567+0000 7f1ccffff700 1 --2- 172.21.15.16:0/606839977 >> v2:172.21.15.16:3300/0 conn(0x7f1cd008c6d0 0x7f1cd008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1cc000fac0 tx=0x7f1cc000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.567+0000 7f1ccffff700 1 --2- 172.21.15.16:0/606839977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cd008c6d0 0x7f1cd008cac0 secure :-1 s=READY pgs=739 cs=0 l=1 rx=0x7f1cc000fac0 tx=0x7f1cc000f8f0).ready entity=mon.0 client_cookie=45a52a5326680307 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.567+0000 7f1ccffff700 1 --2- 172.21.15.16:0/606839977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cd008c6d0 0x7f1cd008cac0 secure :-1 s=THROTTLE_DONE pgs=739 cs=0 l=1 rx=0x7f1cc000fac0 tx=0x7f1cc000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.567+0000 7f1ccffff700 1 --2- 172.21.15.16:0/606839977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cd008c6d0 0x7f1cd008cac0 secure :-1 s=THROTTLE_DONE pgs=739 cs=0 l=1 rx=0x7f1cc000fac0 tx=0x7f1cc000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.567+0000 7f1ccffff700 1 --2- 172.21.15.16:0/606839977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cd008c6d0 0x7f1cd008cac0 secure :-1 s=THROTTLE_DONE pgs=739 cs=0 l=1 rx=0x7f1cc000fac0 tx=0x7f1cc000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.567+0000 7f1cceffd700 1 -- 172.21.15.16:0/606839977 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cc0000ce0 con 0x7f1cd008c6d0 2019-12-05T06:26:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.568+0000 7f1cceffd700 1 -- 172.21.15.16:0/606839977 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc0001010 con 0x7f1cd008c6d0 2019-12-05T06:26:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.568+0000 7f1cceffd700 1 -- 172.21.15.16:0/606839977 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cc0001720 con 0x7f1cd008c6d0 2019-12-05T06:26:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.568+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/606839977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cd008c6d0 msgr2=0x7f1cd008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.568+0000 7f1cd5e8b700 1 --2- 172.21.15.16:0/606839977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cd008c6d0 0x7f1cd008cac0 secure :-1 s=READY pgs=739 cs=0 l=1 rx=0x7f1cc000fac0 tx=0x7f1cc000f8f0).stop 2019-12-05T06:26:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.569+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/606839977 shutdown_connections 2019-12-05T06:26:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.569+0000 7f1cd5e8b700 1 --2- 172.21.15.16:0/606839977 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1cd008ef20 0x7f1cd008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.570+0000 7f1cd5e8b700 1 --2- 172.21.15.16:0/606839977 >> v2:172.21.15.76:3300/0 conn(0x7f1cd008fef0 0x7f1cd008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.570+0000 7f1cd5e8b700 1 --2- 172.21.15.16:0/606839977 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1cd008c6d0 0x7f1cd008cac0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.570+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/606839977 shutdown_connections 2019-12-05T06:26:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.570+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/606839977 wait complete. 2019-12-05T06:26:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.570+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/606839977 >> 172.21.15.16:0/606839977 conn(0x7f1cd0061940 msgr2=0x7f1cd0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.575+0000 7f1cd5e8b700 1 Processor -- start 2019-12-05T06:26:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.576+0000 7f1cd5e8b700 1 -- start start 2019-12-05T06:26:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.578+0000 7f1cd5e8b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1cd009b850 0x7f1cd0131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.578+0000 7f1cd5e8b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd0098da0 con 0x7f1cd0132d00 2019-12-05T06:26:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.578+0000 7f1cd5e8b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd009f0a0 con 0x7f1cd0154b00 2019-12-05T06:26:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.578+0000 7f1cd5e8b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1cd009aa60 con 0x7f1cd009b850 2019-12-05T06:26:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.578+0000 7f1ccffff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1cd009b850 0x7f1cd0131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.578+0000 7f1ccf7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1cd0132d00 0x7f1cd0132270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38304/0 (socket says 172.21.15.16:38304) 2019-12-05T06:26:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.578+0000 7f1ccf7fe700 1 -- 172.21.15.16:0/3835101564 learned_addr learned my addr 172.21.15.16:0/3835101564 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1cccff9700 1 -- 172.21.15.16:0/3835101564 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4259560623 0 0) 0x7f1cb8000fb0 con 0x7f1cd0132d00 2019-12-05T06:26:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1ccffff700 1 -- 172.21.15.16:0/3835101564 >> v1:172.21.15.76:6789/0 conn(0x7f1cd0154b00 legacy=0x7f1cd0156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1ccffff700 1 -- 172.21.15.16:0/3835101564 >> v1:172.21.15.16:6789/0 conn(0x7f1cd0132d00 legacy=0x7f1cd0132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:09.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1ccffff700 1 -- 172.21.15.16:0/3835101564 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd009b0f0 con 0x7f1cd009b850 2019-12-05T06:26:09.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1ccffff700 1 --2- 172.21.15.16:0/3835101564 >> v2:172.21.15.76:3300/0 conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1ccffff700 1 --2- 172.21.15.16:0/3835101564 >> v2:172.21.15.76:3300/0 conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1ccffff700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=READY pgs=532 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:09.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1ccffff700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1cccff9700 1 -- 172.21.15.16:0/3835101564 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cc0005500 con 0x7f1cd009b850 2019-12-05T06:26:09.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1ccffff700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.579+0000 7f1ccffff700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.580+0000 7f1cccff9700 1 -- 172.21.15.16:0/3835101564 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cc0008190 con 0x7f1cd009b850 2019-12-05T06:26:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.580+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/3835101564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd009b0f0 con 0x7f1cd009b850 2019-12-05T06:26:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.580+0000 7f1cccff9700 1 -- 172.21.15.16:0/3835101564 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1cc0005ae0 con 0x7f1cd009b850 2019-12-05T06:26:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.580+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/3835101564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd009b0f0 con 0x7f1cd009b850 2019-12-05T06:26:09.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.581+0000 7f1ccffff700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.582+0000 7f1cccff9700 1 -- 172.21.15.16:0/3835101564 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1cc0005500 con 0x7f1cd009b850 2019-12-05T06:26:09.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.582+0000 7f1ccffff700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.582+0000 7f1cccff9700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1cb4023630 0x7f1cb4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.582+0000 7f1cccff9700 1 -- 172.21.15.16:0/3835101564 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f1cc008a150 con 0x7f1cd009b850 2019-12-05T06:26:09.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.582+0000 7f1ccf7fe700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1cb4023630 0x7f1cb4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.583+0000 7f1ccf7fe700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1cb4023630 0x7f1cb4025ac0 crc :-1 s=READY pgs=324 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:09.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.594+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/3835101564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd009b0f0 con 0x7f1cd009b850 2019-12-05T06:26:09.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.604+0000 7f1ccffff700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.604+0000 7f1cccff9700 1 -- 172.21.15.16:0/3835101564 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1cc0007d60 con 0x7f1cd009b850 2019-12-05T06:26:09.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.842+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/3835101564 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1cd009b0f0 con 0x7f1cd009b850 2019-12-05T06:26:09.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.844+0000 7f1ccffff700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=THROTTLE_DONE pgs=532 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:09.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.844+0000 7f1cccff9700 1 -- 172.21.15.16:0/3835101564 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f1cc0005500 con 0x7f1cd009b850 2019-12-05T06:26:09.830 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 92 2019-12-05T06:26:09.830 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:09.830 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.848+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1cb4023630 msgr2=0x7f1cb4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.848+0000 7f1cd5e8b700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1cb4023630 0x7f1cb4025ac0 crc :-1 s=READY pgs=324 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.848+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 msgr2=0x7f1cd0131d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.848+0000 7f1cd5e8b700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 0x7f1cd0131d30 secure :-1 s=READY pgs=532 cs=0 l=1 rx=0x7f1cc0054740 tx=0x7f1cc0054ae0).stop 2019-12-05T06:26:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.848+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/3835101564 shutdown_connections 2019-12-05T06:26:09.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.848+0000 7f1cd5e8b700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1cb4023630 0x7f1cb4025ac0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.848+0000 7f1cd5e8b700 1 --2- 172.21.15.16:0/3835101564 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1cd009b850 0x7f1cd0131d30 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.848+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/3835101564 shutdown_connections 2019-12-05T06:26:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.848+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/3835101564 wait complete. 2019-12-05T06:26:09.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:09.849+0000 7f1cd5e8b700 1 -- 172.21.15.16:0/3835101564 >> 172.21.15.16:0/3835101564 conn(0x7f1cd00714f0 msgr2=0x7f1cd009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:09.844 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:26:11.845 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.140+0000 7f652300a700 1 Processor -- start 2019-12-05T06:26:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.140+0000 7f652300a700 1 -- start start 2019-12-05T06:26:12.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.142+0000 7f652300a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f651c08fef0 0x7f651c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.142+0000 7f652300a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0958e0 con 0x7f651c08c6d0 2019-12-05T06:26:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.142+0000 7f652300a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0629e0 con 0x7f651c08d2c0 2019-12-05T06:26:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.142+0000 7f652300a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651c08df90 con 0x7f651c08fef0 2019-12-05T06:26:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.142+0000 7f65215a7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f651c08c6d0 0x7f651c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38312/0 (socket says 172.21.15.16:38312) 2019-12-05T06:26:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.142+0000 7f65215a7700 1 -- 172.21.15.16:0/3735271790 learned_addr learned my addr 172.21.15.16:0/3735271790 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:12.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.142+0000 7f6521da8700 1 --2- 172.21.15.16:0/3735271790 >> v2:172.21.15.76:3300/0 conn(0x7f651c08fef0 0x7f651c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.142+0000 7f651bfff700 1 -- 172.21.15.16:0/3735271790 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1498493542 0 0) 0x7f65140014d0 con 0x7f651c08c6d0 2019-12-05T06:26:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.143+0000 7f651bfff700 1 -- 172.21.15.16:0/3735271790 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6504002e80 con 0x7f651c08c6d0 2019-12-05T06:26:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.143+0000 7f651bfff700 1 -- 172.21.15.16:0/3735271790 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580547999 0 0) 0x7f6510001080 con 0x7f651c08d2c0 2019-12-05T06:26:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.143+0000 7f651bfff700 1 -- 172.21.15.16:0/3735271790 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6504004170 con 0x7f651c08d2c0 2019-12-05T06:26:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.143+0000 7f6521da8700 1 -- 172.21.15.16:0/3735271790 >> v1:172.21.15.76:6789/0 conn(0x7f651c08d2c0 legacy=0x7f651c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.143+0000 7f6521da8700 1 -- 172.21.15.16:0/3735271790 >> v1:172.21.15.16:6789/0 conn(0x7f651c08c6d0 legacy=0x7f651c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:12.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.143+0000 7f651bfff700 1 -- 172.21.15.16:0/3735271790 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1666536008 0 0) 0x7f65140014b0 con 0x7f651c08c6d0 2019-12-05T06:26:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.143+0000 7f6521da8700 1 -- 172.21.15.16:0/3735271790 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c08e620 con 0x7f651c08fef0 2019-12-05T06:26:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.143+0000 7f6521da8700 1 --2- 172.21.15.16:0/3735271790 >> v2:172.21.15.76:3300/0 conn(0x7f651c08fef0 0x7f651c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f650c00af80 tx=0x7f650c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.144+0000 7f6521da8700 1 --2- 172.21.15.16:0/3735271790 >> v2:172.21.15.76:3300/0 conn(0x7f651c08fef0 0x7f651c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f650c00af80 tx=0x7f650c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.144+0000 7f6521da8700 1 --2- 172.21.15.16:0/3735271790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f651c08fef0 0x7f651c08f720 secure :-1 s=READY pgs=534 cs=0 l=1 rx=0x7f650c00af80 tx=0x7f650c00b790).ready entity=mon.1 client_cookie=938f552e00078c33 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.144+0000 7f6521da8700 1 --2- 172.21.15.16:0/3735271790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f651c08fef0 0x7f651c08f720 secure :-1 s=THROTTLE_DONE pgs=534 cs=0 l=1 rx=0x7f650c00af80 tx=0x7f650c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.144+0000 7f6521da8700 1 --2- 172.21.15.16:0/3735271790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f651c08fef0 0x7f651c08f720 secure :-1 s=THROTTLE_DONE pgs=534 cs=0 l=1 rx=0x7f650c00af80 tx=0x7f650c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.144+0000 7f651bfff700 1 -- 172.21.15.16:0/3735271790 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f650c007240 con 0x7f651c08fef0 2019-12-05T06:26:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.144+0000 7f6521da8700 1 --2- 172.21.15.16:0/3735271790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f651c08fef0 0x7f651c08f720 secure :-1 s=THROTTLE_DONE pgs=534 cs=0 l=1 rx=0x7f650c00af80 tx=0x7f650c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.144+0000 7f651bfff700 1 -- 172.21.15.16:0/3735271790 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c000c30 con 0x7f651c08fef0 2019-12-05T06:26:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.144+0000 7f651bfff700 1 -- 172.21.15.16:0/3735271790 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f650c009f10 con 0x7f651c08fef0 2019-12-05T06:26:12.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.144+0000 7f652300a700 1 -- 172.21.15.16:0/3735271790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f651c08fef0 msgr2=0x7f651c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.144+0000 7f652300a700 1 --2- 172.21.15.16:0/3735271790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f651c08fef0 0x7f651c08f720 secure :-1 s=READY pgs=534 cs=0 l=1 rx=0x7f650c00af80 tx=0x7f650c00b790).stop 2019-12-05T06:26:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.146+0000 7f652300a700 1 -- 172.21.15.16:0/3735271790 shutdown_connections 2019-12-05T06:26:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.146+0000 7f652300a700 1 --2- 172.21.15.16:0/3735271790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f651c08fef0 0x7f651c08f720 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.146+0000 7f652300a700 1 -- 172.21.15.16:0/3735271790 shutdown_connections 2019-12-05T06:26:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.147+0000 7f652300a700 1 -- 172.21.15.16:0/3735271790 wait complete. 2019-12-05T06:26:12.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.147+0000 7f652300a700 1 -- 172.21.15.16:0/3735271790 >> 172.21.15.16:0/3735271790 conn(0x7f651c061940 msgr2=0x7f651c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:12.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.151+0000 7f652300a700 1 Processor -- start 2019-12-05T06:26:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.152+0000 7f652300a700 1 -- start start 2019-12-05T06:26:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.153+0000 7f652300a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f651c1302f0 0x7f651c09d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.153+0000 7f652300a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f651c09e7d0 0x7f651c09dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:12.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.153+0000 7f652300a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f651c09e280 0x7f651c06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:12.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.153+0000 7f652300a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651c1315c0 con 0x7f651c09e280 2019-12-05T06:26:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.153+0000 7f652300a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f651c131890 con 0x7f651c09e7d0 2019-12-05T06:26:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.153+0000 7f652300a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651c12c770 con 0x7f651c1302f0 2019-12-05T06:26:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f651c09e280 0x7f651c06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f651c09e280 0x7f651c06dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53152/0 (socket says 172.21.15.16:53152) 2019-12-05T06:26:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 -- 172.21.15.16:0/74922830 learned_addr learned my addr 172.21.15.16:0/74922830 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:12.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6520da6700 1 --2- 172.21.15.16:0/74922830 >> v2:172.21.15.76:3300/0 conn(0x7f651c09e7d0 0x7f651c09dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f65215a7700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f651c1302f0 0x7f651c09d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 -- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f651c1302f0 msgr2=0x7f651c09d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f651c1302f0 0x7f651c09d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 -- 172.21.15.16:0/74922830 >> v2:172.21.15.76:3300/0 conn(0x7f651c09e7d0 msgr2=0x7f651c09dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> v2:172.21.15.76:3300/0 conn(0x7f651c09e7d0 0x7f651c09dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:12.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 -- 172.21.15.16:0/74922830 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c132680 con 0x7f651c09e280 2019-12-05T06:26:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f65215a7700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f651c1302f0 0x7f651c09d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:26:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> v2:172.21.15.16:3300/0 conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> v2:172.21.15.16:3300/0 conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.154+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=READY pgs=742 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.155+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.155+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.155+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.155+0000 7f6519ffb700 1 -- 172.21.15.16:0/74922830 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f650c0097c0 con 0x7f651c09e280 2019-12-05T06:26:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.155+0000 7f6519ffb700 1 -- 172.21.15.16:0/74922830 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c00af00 con 0x7f651c09e280 2019-12-05T06:26:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.155+0000 7f652300a700 1 -- 172.21.15.16:0/74922830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651c132680 con 0x7f651c09e280 2019-12-05T06:26:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.155+0000 7f6519ffb700 1 -- 172.21.15.16:0/74922830 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f650c009e40 con 0x7f651c09e280 2019-12-05T06:26:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.155+0000 7f652300a700 1 -- 172.21.15.16:0/74922830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651c132680 con 0x7f651c09e280 2019-12-05T06:26:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.156+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.157+0000 7f6519ffb700 1 -- 172.21.15.16:0/74922830 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f650c009860 con 0x7f651c09e280 2019-12-05T06:26:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.157+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.157+0000 7f6519ffb700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64fc023630 0x7f64fc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.158+0000 7f6519ffb700 1 -- 172.21.15.16:0/74922830 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f650c045d10 con 0x7f651c09e280 2019-12-05T06:26:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.158+0000 7f65215a7700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64fc023630 0x7f64fc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:12.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.158+0000 7f65215a7700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64fc023630 0x7f64fc025ac0 crc :-1 s=READY pgs=325 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:12.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.171+0000 7f652300a700 1 -- 172.21.15.16:0/74922830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651c132680 con 0x7f651c09e280 2019-12-05T06:26:12.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.180+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.180+0000 7f6519ffb700 1 -- 172.21.15.16:0/74922830 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f650c0095b0 con 0x7f651c09e280 2019-12-05T06:26:12.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.404+0000 7f652300a700 1 -- 172.21.15.16:0/74922830 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f651c132680 con 0x7f651c09e280 2019-12-05T06:26:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.405+0000 7f6521da8700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=THROTTLE_DONE pgs=742 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.405+0000 7f6519ffb700 1 -- 172.21.15.16:0/74922830 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f650c0095b0 con 0x7f651c09e280 2019-12-05T06:26:12.390 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 92 2019-12-05T06:26:12.391 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:12.391 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:12.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.407+0000 7f652300a700 1 -- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64fc023630 msgr2=0x7f64fc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:12.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.407+0000 7f652300a700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64fc023630 0x7f64fc025ac0 crc :-1 s=READY pgs=325 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:12.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.407+0000 7f652300a700 1 -- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 msgr2=0x7f651c06dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:12.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.407+0000 7f652300a700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 0x7f651c06dc50 secure :-1 s=READY pgs=742 cs=0 l=1 rx=0x7f650c010440 tx=0x7f650c010340).stop 2019-12-05T06:26:12.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.408+0000 7f652300a700 1 -- 172.21.15.16:0/74922830 shutdown_connections 2019-12-05T06:26:12.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.408+0000 7f652300a700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f64fc023630 0x7f64fc025ac0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:12.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.408+0000 7f652300a700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f651c09e280 0x7f651c06dc50 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.408+0000 7f652300a700 1 --2- 172.21.15.16:0/74922830 >> v2:172.21.15.76:3300/0 conn(0x7f651c09e7d0 0x7f651c09dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:12.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.408+0000 7f652300a700 1 --2- 172.21.15.16:0/74922830 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f651c1302f0 0x7f651c09d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:12.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.408+0000 7f652300a700 1 -- 172.21.15.16:0/74922830 shutdown_connections 2019-12-05T06:26:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.408+0000 7f652300a700 1 -- 172.21.15.16:0/74922830 wait complete. 2019-12-05T06:26:12.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:12.408+0000 7f652300a700 1 -- 172.21.15.16:0/74922830 >> 172.21.15.16:0/74922830 conn(0x7f651c0a2850 msgr2=0x7f651c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:12.404 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:26:14.405 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:14.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.707+0000 7fede36bf700 1 Processor -- start 2019-12-05T06:26:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.707+0000 7fede36bf700 1 -- start start 2019-12-05T06:26:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.709+0000 7fede36bf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7feddc08c6d0 0x7feddc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.709+0000 7fede36bf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc08fb80 con 0x7feddc08fef0 2019-12-05T06:26:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.709+0000 7fede36bf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc0958e0 con 0x7feddc08f720 2019-12-05T06:26:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.709+0000 7fede36bf700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feddc08df90 con 0x7feddc08c6d0 2019-12-05T06:26:14.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.710+0000 7fede145b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7feddc08fef0 0x7feddc08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38332/0 (socket says 172.21.15.16:38332) 2019-12-05T06:26:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.710+0000 7fede145b700 1 -- 172.21.15.16:0/2462877231 learned_addr learned my addr 172.21.15.16:0/2462877231 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.710+0000 7fede1c5c700 1 --2- 172.21.15.16:0/2462877231 >> v2:172.21.15.76:3300/0 conn(0x7feddc08c6d0 0x7feddc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.710+0000 7fede0c5a700 1 -- 172.21.15.16:0/2462877231 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702343123 0 0) 0x7fedd0001150 con 0x7feddc08fef0 2019-12-05T06:26:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.710+0000 7fede0c5a700 1 -- 172.21.15.16:0/2462877231 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedc4002e80 con 0x7feddc08fef0 2019-12-05T06:26:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede1c5c700 1 -- 172.21.15.16:0/2462877231 >> v1:172.21.15.76:6789/0 conn(0x7feddc08f720 legacy=0x7feddc1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede1c5c700 1 -- 172.21.15.16:0/2462877231 >> v1:172.21.15.16:6789/0 conn(0x7feddc08fef0 legacy=0x7feddc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:14.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede1c5c700 1 -- 172.21.15.16:0/2462877231 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc08e620 con 0x7feddc08c6d0 2019-12-05T06:26:14.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede1c5c700 1 --2- 172.21.15.16:0/2462877231 >> v2:172.21.15.76:3300/0 conn(0x7feddc08c6d0 0x7feddc08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fedd400b2d0 tx=0x7fedd400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede1c5c700 1 --2- 172.21.15.16:0/2462877231 >> v2:172.21.15.76:3300/0 conn(0x7feddc08c6d0 0x7feddc08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fedd400b2d0 tx=0x7fedd400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede1c5c700 1 --2- 172.21.15.16:0/2462877231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feddc08c6d0 0x7feddc08cac0 secure :-1 s=READY pgs=536 cs=0 l=1 rx=0x7fedd400b2d0 tx=0x7fedd400bae0).ready entity=mon.1 client_cookie=8689e988f660f226 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:14.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede1c5c700 1 --2- 172.21.15.16:0/2462877231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feddc08c6d0 0x7feddc08cac0 secure :-1 s=THROTTLE_DONE pgs=536 cs=0 l=1 rx=0x7fedd400b2d0 tx=0x7fedd400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede0c5a700 1 -- 172.21.15.16:0/2462877231 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedd4000ce0 con 0x7feddc08c6d0 2019-12-05T06:26:14.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede1c5c700 1 --2- 172.21.15.16:0/2462877231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feddc08c6d0 0x7feddc08cac0 secure :-1 s=THROTTLE_DONE pgs=536 cs=0 l=1 rx=0x7fedd400b2d0 tx=0x7fedd400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede1c5c700 1 --2- 172.21.15.16:0/2462877231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feddc08c6d0 0x7feddc08cac0 secure :-1 s=THROTTLE_DONE pgs=536 cs=0 l=1 rx=0x7fedd400b2d0 tx=0x7fedd400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede0c5a700 1 -- 172.21.15.16:0/2462877231 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedd4009be0 con 0x7feddc08c6d0 2019-12-05T06:26:14.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.711+0000 7fede0c5a700 1 -- 172.21.15.16:0/2462877231 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedd400a2b0 con 0x7feddc08c6d0 2019-12-05T06:26:14.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.712+0000 7fede36bf700 1 -- 172.21.15.16:0/2462877231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feddc08c6d0 msgr2=0x7feddc08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:14.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.712+0000 7fede36bf700 1 --2- 172.21.15.16:0/2462877231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feddc08c6d0 0x7feddc08cac0 secure :-1 s=READY pgs=536 cs=0 l=1 rx=0x7fedd400b2d0 tx=0x7fedd400bae0).stop 2019-12-05T06:26:14.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.713+0000 7fede36bf700 1 -- 172.21.15.16:0/2462877231 shutdown_connections 2019-12-05T06:26:14.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.713+0000 7fede36bf700 1 --2- 172.21.15.16:0/2462877231 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feddc08c6d0 0x7feddc08cac0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:14.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.713+0000 7fede36bf700 1 -- 172.21.15.16:0/2462877231 shutdown_connections 2019-12-05T06:26:14.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.713+0000 7fede36bf700 1 -- 172.21.15.16:0/2462877231 wait complete. 2019-12-05T06:26:14.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.713+0000 7fede36bf700 1 -- 172.21.15.16:0/2462877231 >> 172.21.15.16:0/2462877231 conn(0x7feddc061940 msgr2=0x7feddc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:14.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.718+0000 7fede36bf700 1 Processor -- start 2019-12-05T06:26:14.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.718+0000 7fede36bf700 1 -- start start 2019-12-05T06:26:14.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.720+0000 7fede36bf700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feddc128360 0x7feddc128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:14.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.720+0000 7fede36bf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feddc154770 0x7feddc12bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:14.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.720+0000 7fede36bf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feddc099720 con 0x7feddc137ea0 2019-12-05T06:26:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.720+0000 7fede36bf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feddc129c10 con 0x7feddc154770 2019-12-05T06:26:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.720+0000 7fede36bf700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feddc129ee0 con 0x7feddc128360 2019-12-05T06:26:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.720+0000 7fede245d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feddc154770 0x7feddc12bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.720+0000 7fede1c5c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feddc128360 0x7feddc128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.720+0000 7fede245d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7feddc154770 0x7feddc12bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53170/0 (socket says 172.21.15.16:53170) 2019-12-05T06:26:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.720+0000 7fede145b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7feddc137ea0 0x7feddc12b5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:59636/0 (socket says 172.21.15.16:59636) 2019-12-05T06:26:14.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.720+0000 7fede245d700 1 -- 172.21.15.16:0/740683721 learned_addr learned my addr 172.21.15.16:0/740683721 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.721+0000 7fede245d700 1 -- 172.21.15.16:0/740683721 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feddc128360 msgr2=0x7feddc128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.721+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feddc128360 0x7feddc128750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.721+0000 7fedda7fc700 1 -- 172.21.15.16:0/740683721 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365164126 0 0) 0x7fedd0000f40 con 0x7feddc137ea0 2019-12-05T06:26:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.721+0000 7fede245d700 1 -- 172.21.15.16:0/740683721 >> v1:172.21.15.76:6789/0 conn(0x7feddc137ea0 legacy=0x7feddc12b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.721+0000 7fede245d700 1 -- 172.21.15.16:0/740683721 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feddc12acd0 con 0x7feddc154770 2019-12-05T06:26:14.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.721+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> v2:172.21.15.16:3300/0 conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.721+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> v2:172.21.15.16:3300/0 conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.721+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=READY pgs=744 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.722+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=THROTTLE_DONE pgs=744 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.722+0000 7fedda7fc700 1 -- 172.21.15.16:0/740683721 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedcc000960 con 0x7feddc154770 2019-12-05T06:26:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.722+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=THROTTLE_DONE pgs=744 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.722+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=THROTTLE_DONE pgs=744 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.722+0000 7fedda7fc700 1 -- 172.21.15.16:0/740683721 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedcc001090 con 0x7feddc154770 2019-12-05T06:26:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.722+0000 7fede36bf700 1 -- 172.21.15.16:0/740683721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feddc12acd0 con 0x7feddc154770 2019-12-05T06:26:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.722+0000 7fedda7fc700 1 -- 172.21.15.16:0/740683721 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedcc000960 con 0x7feddc154770 2019-12-05T06:26:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.722+0000 7fede36bf700 1 -- 172.21.15.16:0/740683721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feddc12acd0 con 0x7feddc154770 2019-12-05T06:26:14.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.723+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=THROTTLE_DONE pgs=744 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.724+0000 7fedda7fc700 1 -- 172.21.15.16:0/740683721 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fedcc001280 con 0x7feddc154770 2019-12-05T06:26:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.724+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=THROTTLE_DONE pgs=744 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.724+0000 7fedda7fc700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedbc023630 0x7fedbc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.725+0000 7fedda7fc700 1 -- 172.21.15.16:0/740683721 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fedcc045de0 con 0x7feddc154770 2019-12-05T06:26:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.725+0000 7fede1c5c700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedbc023630 0x7fedbc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:14.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.725+0000 7fede1c5c700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedbc023630 0x7fedbc025ac0 crc :-1 s=READY pgs=326 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:14.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.739+0000 7fede36bf700 1 -- 172.21.15.16:0/740683721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feddc12acd0 con 0x7feddc154770 2019-12-05T06:26:14.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.746+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=THROTTLE_DONE pgs=744 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.747+0000 7fedda7fc700 1 -- 172.21.15.16:0/740683721 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fedcc007590 con 0x7feddc154770 2019-12-05T06:26:14.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.988+0000 7fede36bf700 1 -- 172.21.15.16:0/740683721 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feddc12acd0 con 0x7feddc154770 2019-12-05T06:26:14.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.989+0000 7fede245d700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=THROTTLE_DONE pgs=744 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:14.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.989+0000 7fedda7fc700 1 -- 172.21.15.16:0/740683721 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fedcc001b30 con 0x7feddc154770 2019-12-05T06:26:14.976 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 92 2019-12-05T06:26:14.976 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:14.976 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:14.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.992+0000 7fede36bf700 1 -- 172.21.15.16:0/740683721 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedbc023630 msgr2=0x7fedbc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:14.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.992+0000 7fede36bf700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedbc023630 0x7fedbc025ac0 crc :-1 s=READY pgs=326 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:14.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.992+0000 7fede36bf700 1 -- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 msgr2=0x7feddc12bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:14.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.992+0000 7fede36bf700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 0x7feddc12bdd0 secure :-1 s=READY pgs=744 cs=0 l=1 rx=0x7fedcc010420 tx=0x7fedcc010280).stop 2019-12-05T06:26:14.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.992+0000 7fede36bf700 1 -- 172.21.15.16:0/740683721 shutdown_connections 2019-12-05T06:26:14.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.992+0000 7fede36bf700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedbc023630 0x7fedbc025ac0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:14.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.992+0000 7fede36bf700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feddc154770 0x7feddc12bdd0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:14.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.992+0000 7fede36bf700 1 --2- 172.21.15.16:0/740683721 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feddc128360 0x7feddc128750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:14.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.993+0000 7fede36bf700 1 -- 172.21.15.16:0/740683721 shutdown_connections 2019-12-05T06:26:14.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.993+0000 7fede36bf700 1 -- 172.21.15.16:0/740683721 wait complete. 2019-12-05T06:26:14.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:14.993+0000 7fede36bf700 1 -- 172.21.15.16:0/740683721 >> 172.21.15.16:0/740683721 conn(0x7feddc06d260 msgr2=0x7feddc09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:14.988 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:26:16.989 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:17.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.275+0000 7f0f405eb700 1 Processor -- start 2019-12-05T06:26:17.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.275+0000 7f0f405eb700 1 -- start start 2019-12-05T06:26:17.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.277+0000 7f0f405eb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0f3808fef0 0x7f0f3808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:17.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.278+0000 7f0f405eb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f380958e0 con 0x7f0f3808d2c0 2019-12-05T06:26:17.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.278+0000 7f0f405eb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f380629e0 con 0x7f0f3808c6d0 2019-12-05T06:26:17.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.278+0000 7f0f405eb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f3808df90 con 0x7f0f3808fef0 2019-12-05T06:26:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.278+0000 7f0f3e387700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f3808d2c0 0x7f0f3808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38346/0 (socket says 172.21.15.16:38346) 2019-12-05T06:26:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.278+0000 7f0f3e387700 1 -- 172.21.15.16:0/800119613 learned_addr learned my addr 172.21.15.16:0/800119613 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.278+0000 7f0f3f389700 1 --2- 172.21.15.16:0/800119613 >> v2:172.21.15.76:3300/0 conn(0x7f0f3808fef0 0x7f0f3808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.279+0000 7f0f3db86700 1 -- 172.21.15.16:0/800119613 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2285110200 0 0) 0x7f0f2c001150 con 0x7f0f3808d2c0 2019-12-05T06:26:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.279+0000 7f0f3db86700 1 -- 172.21.15.16:0/800119613 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f20002e80 con 0x7f0f3808d2c0 2019-12-05T06:26:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.279+0000 7f0f3db86700 1 -- 172.21.15.16:0/800119613 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062386027 0 0) 0x7f0f30001400 con 0x7f0f3808c6d0 2019-12-05T06:26:17.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.279+0000 7f0f3f389700 1 -- 172.21.15.16:0/800119613 >> v1:172.21.15.76:6789/0 conn(0x7f0f3808c6d0 legacy=0x7f0f3808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.279+0000 7f0f3f389700 1 -- 172.21.15.16:0/800119613 >> v1:172.21.15.16:6789/0 conn(0x7f0f3808d2c0 legacy=0x7f0f3808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.279+0000 7f0f3f389700 1 -- 172.21.15.16:0/800119613 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f3808e620 con 0x7f0f3808fef0 2019-12-05T06:26:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.279+0000 7f0f3f389700 1 --2- 172.21.15.16:0/800119613 >> v2:172.21.15.76:3300/0 conn(0x7f0f3808fef0 0x7f0f3808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0f2800af80 tx=0x7f0f2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.279+0000 7f0f3f389700 1 --2- 172.21.15.16:0/800119613 >> v2:172.21.15.76:3300/0 conn(0x7f0f3808fef0 0x7f0f3808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0f2800af80 tx=0x7f0f2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.279+0000 7f0f3f389700 1 --2- 172.21.15.16:0/800119613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f3808fef0 0x7f0f3808f720 secure :-1 s=READY pgs=539 cs=0 l=1 rx=0x7f0f2800af80 tx=0x7f0f2800b790).ready entity=mon.1 client_cookie=5e440a4a90f0c1f9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.280+0000 7f0f3f389700 1 --2- 172.21.15.16:0/800119613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f3808fef0 0x7f0f3808f720 secure :-1 s=THROTTLE_DONE pgs=539 cs=0 l=1 rx=0x7f0f2800af80 tx=0x7f0f2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.280+0000 7f0f3db86700 1 -- 172.21.15.16:0/800119613 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f28007240 con 0x7f0f3808fef0 2019-12-05T06:26:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.280+0000 7f0f3f389700 1 --2- 172.21.15.16:0/800119613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f3808fef0 0x7f0f3808f720 secure :-1 s=THROTTLE_DONE pgs=539 cs=0 l=1 rx=0x7f0f2800af80 tx=0x7f0f2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.280+0000 7f0f3f389700 1 --2- 172.21.15.16:0/800119613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f3808fef0 0x7f0f3808f720 secure :-1 s=THROTTLE_DONE pgs=539 cs=0 l=1 rx=0x7f0f2800af80 tx=0x7f0f2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.280+0000 7f0f3db86700 1 -- 172.21.15.16:0/800119613 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f28000bb0 con 0x7f0f3808fef0 2019-12-05T06:26:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.280+0000 7f0f3db86700 1 -- 172.21.15.16:0/800119613 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f28009e60 con 0x7f0f3808fef0 2019-12-05T06:26:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.280+0000 7f0f405eb700 1 -- 172.21.15.16:0/800119613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f3808fef0 msgr2=0x7f0f3808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.280+0000 7f0f405eb700 1 --2- 172.21.15.16:0/800119613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f3808fef0 0x7f0f3808f720 secure :-1 s=READY pgs=539 cs=0 l=1 rx=0x7f0f2800af80 tx=0x7f0f2800b790).stop 2019-12-05T06:26:17.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.282+0000 7f0f405eb700 1 -- 172.21.15.16:0/800119613 shutdown_connections 2019-12-05T06:26:17.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.282+0000 7f0f405eb700 1 --2- 172.21.15.16:0/800119613 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0f3808fef0 0x7f0f3808f720 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:17.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.282+0000 7f0f405eb700 1 -- 172.21.15.16:0/800119613 shutdown_connections 2019-12-05T06:26:17.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.282+0000 7f0f405eb700 1 -- 172.21.15.16:0/800119613 wait complete. 2019-12-05T06:26:17.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.282+0000 7f0f405eb700 1 -- 172.21.15.16:0/800119613 >> 172.21.15.16:0/800119613 conn(0x7f0f38061940 msgr2=0x7f0f38096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.288+0000 7f0f405eb700 1 Processor -- start 2019-12-05T06:26:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.288+0000 7f0f405eb700 1 -- start start 2019-12-05T06:26:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.290+0000 7f0f405eb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.290+0000 7f0f405eb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f38131bf0 con 0x7f0f38153ba0 2019-12-05T06:26:17.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.290+0000 7f0f405eb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f381316f0 con 0x7f0f381302f0 2019-12-05T06:26:17.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.290+0000 7f0f405eb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f381296e0 con 0x7f0f3812b640 2019-12-05T06:26:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.290+0000 7f0f3f389700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f38153ba0 0x7f0f38155fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38352/0 (socket says 172.21.15.16:38352) 2019-12-05T06:26:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.290+0000 7f0f3f389700 1 -- 172.21.15.16:0/1614412682 learned_addr learned my addr 172.21.15.16:0/1614412682 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:17.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.290+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:17.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.291+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203285633 0 0) 0x7f0f2800ae60 con 0x7f0f38153ba0 2019-12-05T06:26:17.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.291+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f18002e80 con 0x7f0f38153ba0 2019-12-05T06:26:17.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.291+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877730307 0 0) 0x7f0f300013b0 con 0x7f0f381302f0 2019-12-05T06:26:17.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.291+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f18004220 con 0x7f0f381302f0 2019-12-05T06:26:17.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.291+0000 7f0f3e387700 1 -- 172.21.15.16:0/1614412682 >> v1:172.21.15.76:6789/0 conn(0x7f0f381302f0 legacy=0x7f0f3812a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:17.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.291+0000 7f0f3e387700 1 -- 172.21.15.16:0/1614412682 >> v1:172.21.15.16:6789/0 conn(0x7f0f38153ba0 legacy=0x7f0f38155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:17.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.291+0000 7f0f3e387700 1 -- 172.21.15.16:0/1614412682 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f38129d70 con 0x7f0f3812b640 2019-12-05T06:26:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.291+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.291+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.291+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=READY pgs=185 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.292+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.292+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f2c00f130 con 0x7f0f3812b640 2019-12-05T06:26:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.292+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.292+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.292+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f2c00ed50 con 0x7f0f3812b640 2019-12-05T06:26:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.292+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f2c00f7b0 con 0x7f0f3812b640 2019-12-05T06:26:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.292+0000 7f0f405eb700 1 -- 172.21.15.16:0/1614412682 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f38129d70 con 0x7f0f3812b640 2019-12-05T06:26:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.292+0000 7f0f405eb700 1 -- 172.21.15.16:0/1614412682 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f38129d70 con 0x7f0f3812b640 2019-12-05T06:26:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.293+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.294+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0f2c00ef40 con 0x7f0f3812b640 2019-12-05T06:26:17.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.294+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.294+0000 7f0f377fe700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f18023650 0x7f0f18025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:17.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.294+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f0f2c045d90 con 0x7f0f3812b640 2019-12-05T06:26:17.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.295+0000 7f0f3eb88700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f18023650 0x7f0f18025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:17.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.295+0000 7f0f3eb88700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f18023650 0x7f0f18025ae0 crc :-1 s=READY pgs=327 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:17.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.307+0000 7f0f405eb700 1 -- 172.21.15.16:0/1614412682 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f38129d70 con 0x7f0f3812b640 2019-12-05T06:26:17.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.317+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.317+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0f2c00ea50 con 0x7f0f3812b640 2019-12-05T06:26:17.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.580+0000 7f0f405eb700 1 -- 172.21.15.16:0/1614412682 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f38129d70 con 0x7f0f3812b640 2019-12-05T06:26:17.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.581+0000 7f0f3e387700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=THROTTLE_DONE pgs=185 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:17.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.581+0000 7f0f377fe700 1 -- 172.21.15.16:0/1614412682 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f0f2c00ef40 con 0x7f0f3812b640 2019-12-05T06:26:17.568 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:17.568 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:17.568 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 92 2019-12-05T06:26:17.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.585+0000 7f0f405eb700 1 -- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f18023650 msgr2=0x7f0f18025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:17.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.585+0000 7f0f405eb700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f18023650 0x7f0f18025ae0 crc :-1 s=READY pgs=327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:17.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.585+0000 7f0f405eb700 1 -- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 msgr2=0x7f0f3812ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:17.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.585+0000 7f0f405eb700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 secure :-1 s=READY pgs=185 cs=0 l=1 rx=0x7f0f2c010380 tx=0x7f0f2c010720).stop 2019-12-05T06:26:17.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.585+0000 7f0f405eb700 1 -- 172.21.15.16:0/1614412682 shutdown_connections 2019-12-05T06:26:17.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.585+0000 7f0f405eb700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f18023650 0x7f0f18025ae0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:17.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.585+0000 7f0f405eb700 1 --2- 172.21.15.16:0/1614412682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f3812b640 0x7f0f3812ae70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:17.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.586+0000 7f0f405eb700 1 -- 172.21.15.16:0/1614412682 shutdown_connections 2019-12-05T06:26:17.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.586+0000 7f0f405eb700 1 -- 172.21.15.16:0/1614412682 wait complete. 2019-12-05T06:26:17.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:17.586+0000 7f0f405eb700 1 -- 172.21.15.16:0/1614412682 >> 172.21.15.16:0/1614412682 conn(0x7f0f380a2850 msgr2=0x7f0f3806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:17.581 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:26:19.583 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:19.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.858+0000 7fd337727700 1 Processor -- start 2019-12-05T06:26:19.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.859+0000 7fd337727700 1 -- start start 2019-12-05T06:26:19.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.860+0000 7fd337727700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd33008c6d0 0x7fd33008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:19.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.860+0000 7fd337727700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd33008f720 0x7fd33008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:19.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.860+0000 7fd337727700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33008fb70 con 0x7fd33008fef0 2019-12-05T06:26:19.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.860+0000 7fd337727700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd33008df90 con 0x7fd33008c6d0 2019-12-05T06:26:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.860+0000 7fd337727700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3300a3f70 con 0x7fd33008f720 2019-12-05T06:26:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.861+0000 7fd3354c3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd33008fef0 0x7fd33008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38364/0 (socket says 172.21.15.16:38364) 2019-12-05T06:26:19.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.861+0000 7fd3354c3700 1 -- 172.21.15.16:0/1631403295 learned_addr learned my addr 172.21.15.16:0/1631403295 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.861+0000 7fd335cc4700 1 --2- 172.21.15.16:0/1631403295 >> v2:172.21.15.76:3300/0 conn(0x7fd33008c6d0 0x7fd33008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.861+0000 7fd3364c5700 1 --2- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd33008f720 0x7fd33008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.861+0000 7fd334cc2700 1 -- 172.21.15.16:0/1631403295 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2925011900 0 0) 0x7fd3240014d0 con 0x7fd33008fef0 2019-12-05T06:26:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.861+0000 7fd335cc4700 1 -- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd33008f720 msgr2=0x7fd33008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.861+0000 7fd335cc4700 1 --2- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd33008f720 0x7fd33008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:19.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.861+0000 7fd335cc4700 1 -- 172.21.15.16:0/1631403295 >> v1:172.21.15.16:6789/0 conn(0x7fd33008fef0 legacy=0x7fd33008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.861+0000 7fd335cc4700 1 -- 172.21.15.16:0/1631403295 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd33008d540 con 0x7fd33008c6d0 2019-12-05T06:26:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd335cc4700 1 --2- 172.21.15.16:0/1631403295 >> v2:172.21.15.76:3300/0 conn(0x7fd33008c6d0 0x7fd33008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd32000af50 tx=0x7fd32000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd335cc4700 1 --2- 172.21.15.16:0/1631403295 >> v2:172.21.15.76:3300/0 conn(0x7fd33008c6d0 0x7fd33008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd32000af50 tx=0x7fd32000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd335cc4700 1 --2- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd33008c6d0 0x7fd33008cac0 secure :-1 s=READY pgs=541 cs=0 l=1 rx=0x7fd32000af50 tx=0x7fd32000b250).ready entity=mon.1 client_cookie=a24aa6319ca21278 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd335cc4700 1 --2- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd33008c6d0 0x7fd33008cac0 secure :-1 s=THROTTLE_DONE pgs=541 cs=0 l=1 rx=0x7fd32000af50 tx=0x7fd32000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd334cc2700 1 -- 172.21.15.16:0/1631403295 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd320007210 con 0x7fd33008c6d0 2019-12-05T06:26:19.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd335cc4700 1 --2- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd33008c6d0 0x7fd33008cac0 secure :-1 s=THROTTLE_DONE pgs=541 cs=0 l=1 rx=0x7fd32000af50 tx=0x7fd32000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd335cc4700 1 --2- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd33008c6d0 0x7fd33008cac0 secure :-1 s=THROTTLE_DONE pgs=541 cs=0 l=1 rx=0x7fd32000af50 tx=0x7fd32000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd334cc2700 1 -- 172.21.15.16:0/1631403295 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3200099b0 con 0x7fd33008c6d0 2019-12-05T06:26:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd334cc2700 1 -- 172.21.15.16:0/1631403295 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd32000a080 con 0x7fd33008c6d0 2019-12-05T06:26:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd337727700 1 -- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd33008c6d0 msgr2=0x7fd33008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.862+0000 7fd337727700 1 --2- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd33008c6d0 0x7fd33008cac0 secure :-1 s=READY pgs=541 cs=0 l=1 rx=0x7fd32000af50 tx=0x7fd32000b250).stop 2019-12-05T06:26:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.863+0000 7fd337727700 1 -- 172.21.15.16:0/1631403295 shutdown_connections 2019-12-05T06:26:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.863+0000 7fd337727700 1 --2- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd33008f720 0x7fd33008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:19.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.863+0000 7fd337727700 1 --2- 172.21.15.16:0/1631403295 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd33008c6d0 0x7fd33008cac0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.864+0000 7fd337727700 1 -- 172.21.15.16:0/1631403295 shutdown_connections 2019-12-05T06:26:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.864+0000 7fd337727700 1 -- 172.21.15.16:0/1631403295 wait complete. 2019-12-05T06:26:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.864+0000 7fd337727700 1 -- 172.21.15.16:0/1631403295 >> 172.21.15.16:0/1631403295 conn(0x7fd330061940 msgr2=0x7fd330096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.868+0000 7fd337727700 1 Processor -- start 2019-12-05T06:26:19.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.869+0000 7fd337727700 1 -- start start 2019-12-05T06:26:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.870+0000 7fd337727700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd3301302f0 0x7fd33009d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.870+0000 7fd337727700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd33009e7d0 0x7fd33009dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.870+0000 7fd337727700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd33012cb90 con 0x7fd33009e280 2019-12-05T06:26:19.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.870+0000 7fd337727700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd330131ff0 con 0x7fd33009e7d0 2019-12-05T06:26:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.870+0000 7fd337727700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3301315c0 con 0x7fd3301302f0 2019-12-05T06:26:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.870+0000 7fd3354c3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd33009e7d0 0x7fd33009dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.870+0000 7fd3364c5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd33009e280 0x7fd33006dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38366/0 (socket says 172.21.15.16:38366) 2019-12-05T06:26:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.870+0000 7fd3354c3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd33009e7d0 0x7fd33009dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53200/0 (socket says 172.21.15.16:53200) 2019-12-05T06:26:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.870+0000 7fd3364c5700 1 -- 172.21.15.16:0/830716072 learned_addr learned my addr 172.21.15.16:0/830716072 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.870+0000 7fd335cc4700 1 --2- 172.21.15.16:0/830716072 >> v2:172.21.15.76:3300/0 conn(0x7fd3301302f0 0x7fd33009d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:19.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd3354c3700 1 -- 172.21.15.16:0/830716072 >> v2:172.21.15.76:3300/0 conn(0x7fd3301302f0 msgr2=0x7fd33009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> v2:172.21.15.76:3300/0 conn(0x7fd3301302f0 0x7fd33009d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd3354c3700 1 -- 172.21.15.16:0/830716072 >> v1:172.21.15.16:6789/0 conn(0x7fd33009e280 legacy=0x7fd33006dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd32e7fc700 1 -- 172.21.15.16:0/830716072 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1687016130 0 0) 0x7fd328001150 con 0x7fd33009e280 2019-12-05T06:26:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd3354c3700 1 -- 172.21.15.16:0/830716072 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd330132680 con 0x7fd33009e7d0 2019-12-05T06:26:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> v2:172.21.15.16:3300/0 conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> v2:172.21.15.16:3300/0 conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=READY pgs=749 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:19.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd32e7fc700 1 -- 172.21.15.16:0/830716072 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd324001270 con 0x7fd33009e7d0 2019-12-05T06:26:19.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd32e7fc700 1 -- 172.21.15.16:0/830716072 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3240045c0 con 0x7fd33009e7d0 2019-12-05T06:26:19.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd337727700 1 -- 172.21.15.16:0/830716072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd330132680 con 0x7fd33009e7d0 2019-12-05T06:26:19.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.871+0000 7fd32e7fc700 1 -- 172.21.15.16:0/830716072 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd324001580 con 0x7fd33009e7d0 2019-12-05T06:26:19.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.872+0000 7fd337727700 1 -- 172.21.15.16:0/830716072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd330132680 con 0x7fd33009e7d0 2019-12-05T06:26:19.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.872+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.874+0000 7fd32e7fc700 1 -- 172.21.15.16:0/830716072 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd3240047b0 con 0x7fd33009e7d0 2019-12-05T06:26:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.874+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.874+0000 7fd32e7fc700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd310023630 0x7fd310025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.874+0000 7fd32e7fc700 1 -- 172.21.15.16:0/830716072 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7fd324045f00 con 0x7fd33009e7d0 2019-12-05T06:26:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.874+0000 7fd335cc4700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd310023630 0x7fd310025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:19.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.875+0000 7fd335cc4700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd310023630 0x7fd310025ac0 crc :-1 s=READY pgs=328 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:19.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.890+0000 7fd337727700 1 -- 172.21.15.16:0/830716072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd330132680 con 0x7fd33009e7d0 2019-12-05T06:26:19.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.896+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:19.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:19.896+0000 7fd32e7fc700 1 -- 172.21.15.16:0/830716072 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd32403ff20 con 0x7fd33009e7d0 2019-12-05T06:26:20.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.149+0000 7fd337727700 1 -- 172.21.15.16:0/830716072 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd330132680 con 0x7fd33009e7d0 2019-12-05T06:26:20.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.150+0000 7fd3354c3700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=THROTTLE_DONE pgs=749 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.150+0000 7fd32e7fc700 1 -- 172.21.15.16:0/830716072 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fd32403ff20 con 0x7fd33009e7d0 2019-12-05T06:26:20.136 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 92 2019-12-05T06:26:20.136 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:20.136 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.152+0000 7fd337727700 1 -- 172.21.15.16:0/830716072 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd310023630 msgr2=0x7fd310025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.152+0000 7fd337727700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd310023630 0x7fd310025ac0 crc :-1 s=READY pgs=328 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:20.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.152+0000 7fd337727700 1 -- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 msgr2=0x7fd33009dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:20.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.152+0000 7fd337727700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 0x7fd33009dd40 secure :-1 s=READY pgs=749 cs=0 l=1 rx=0x7fd3240106f0 tx=0x7fd324010520).stop 2019-12-05T06:26:20.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.153+0000 7fd337727700 1 -- 172.21.15.16:0/830716072 shutdown_connections 2019-12-05T06:26:20.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.153+0000 7fd337727700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd310023630 0x7fd310025ac0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:20.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.153+0000 7fd337727700 1 --2- 172.21.15.16:0/830716072 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd33009e7d0 0x7fd33009dd40 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:20.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.153+0000 7fd337727700 1 --2- 172.21.15.16:0/830716072 >> v2:172.21.15.76:3300/0 conn(0x7fd3301302f0 0x7fd33009d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:20.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.153+0000 7fd337727700 1 -- 172.21.15.16:0/830716072 shutdown_connections 2019-12-05T06:26:20.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.153+0000 7fd337727700 1 -- 172.21.15.16:0/830716072 wait complete. 2019-12-05T06:26:20.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:20.153+0000 7fd337727700 1 -- 172.21.15.16:0/830716072 >> 172.21.15.16:0/830716072 conn(0x7fd3300a2850 msgr2=0x7fd33006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:20.148 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:26:22.149 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:22.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.444+0000 7f1ebb944700 1 Processor -- start 2019-12-05T06:26:22.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.445+0000 7f1ebb944700 1 -- start start 2019-12-05T06:26:22.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.446+0000 7f1ebb944700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 0x7f1eb408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:22.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.446+0000 7f1ebb944700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb40958e0 con 0x7f1eb408fef0 2019-12-05T06:26:22.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.446+0000 7f1ebb944700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1eb40629e0 con 0x7f1eb408f720 2019-12-05T06:26:22.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.446+0000 7f1ebb944700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40a3f70 con 0x7f1eb408c6d0 2019-12-05T06:26:22.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.447+0000 7f1eb96e0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1eb408fef0 0x7f1eb408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38386/0 (socket says 172.21.15.16:38386) 2019-12-05T06:26:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.447+0000 7f1eb96e0700 1 -- 172.21.15.16:0/1831768602 learned_addr learned my addr 172.21.15.16:0/1831768602 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.447+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/1831768602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 0x7f1eb408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.447+0000 7f1eb8edf700 1 -- 172.21.15.16:0/1831768602 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058688778 0 0) 0x7f1ea8001170 con 0x7f1eb408fef0 2019-12-05T06:26:22.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.448+0000 7f1eb8edf700 1 -- 172.21.15.16:0/1831768602 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e9c002e80 con 0x7f1eb408fef0 2019-12-05T06:26:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.448+0000 7f1eb8edf700 1 -- 172.21.15.16:0/1831768602 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2567756046 0 0) 0x7f1ea40009b0 con 0x7f1eb408f720 2019-12-05T06:26:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.448+0000 7f1eb8edf700 1 -- 172.21.15.16:0/1831768602 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e9c004170 con 0x7f1eb408f720 2019-12-05T06:26:22.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.448+0000 7f1eb9ee1700 1 -- 172.21.15.16:0/1831768602 >> v1:172.21.15.76:6789/0 conn(0x7f1eb408f720 legacy=0x7f1eb41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.448+0000 7f1eb9ee1700 1 -- 172.21.15.16:0/1831768602 >> v1:172.21.15.16:6789/0 conn(0x7f1eb408fef0 legacy=0x7f1eb408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.448+0000 7f1eb9ee1700 1 -- 172.21.15.16:0/1831768602 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb408fbc0 con 0x7f1eb408c6d0 2019-12-05T06:26:22.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.448+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/1831768602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 0x7f1eb408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1eac00b2d0 tx=0x7f1eac00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.448+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/1831768602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 0x7f1eb408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1eac00b2d0 tx=0x7f1eac00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.448+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/1831768602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 0x7f1eb408cac0 secure :-1 s=READY pgs=186 cs=0 l=1 rx=0x7f1eac00b2d0 tx=0x7f1eac00bae0).ready entity=mon.2 client_cookie=cc57787bd3f326b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.449+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/1831768602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 0x7f1eb408cac0 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7f1eac00b2d0 tx=0x7f1eac00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.449+0000 7f1eb8edf700 1 -- 172.21.15.16:0/1831768602 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1eac000ce0 con 0x7f1eb408c6d0 2019-12-05T06:26:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.449+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/1831768602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 0x7f1eb408cac0 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7f1eac00b2d0 tx=0x7f1eac00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.449+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/1831768602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 0x7f1eb408cac0 secure :-1 s=THROTTLE_DONE pgs=186 cs=0 l=1 rx=0x7f1eac00b2d0 tx=0x7f1eac00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.449+0000 7f1eb8edf700 1 -- 172.21.15.16:0/1831768602 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac009be0 con 0x7f1eb408c6d0 2019-12-05T06:26:22.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.449+0000 7f1eb8edf700 1 -- 172.21.15.16:0/1831768602 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1eac000ce0 con 0x7f1eb408c6d0 2019-12-05T06:26:22.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.449+0000 7f1ebb944700 1 -- 172.21.15.16:0/1831768602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 msgr2=0x7f1eb408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:22.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.449+0000 7f1ebb944700 1 --2- 172.21.15.16:0/1831768602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 0x7f1eb408cac0 secure :-1 s=READY pgs=186 cs=0 l=1 rx=0x7f1eac00b2d0 tx=0x7f1eac00bae0).stop 2019-12-05T06:26:22.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.451+0000 7f1ebb944700 1 -- 172.21.15.16:0/1831768602 shutdown_connections 2019-12-05T06:26:22.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.451+0000 7f1ebb944700 1 --2- 172.21.15.16:0/1831768602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb408c6d0 0x7f1eb408cac0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:22.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.451+0000 7f1ebb944700 1 -- 172.21.15.16:0/1831768602 shutdown_connections 2019-12-05T06:26:22.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.451+0000 7f1ebb944700 1 -- 172.21.15.16:0/1831768602 wait complete. 2019-12-05T06:26:22.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.451+0000 7f1ebb944700 1 -- 172.21.15.16:0/1831768602 >> 172.21.15.16:0/1831768602 conn(0x7f1eb4061940 msgr2=0x7f1eb4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:22.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.456+0000 7f1ebb944700 1 Processor -- start 2019-12-05T06:26:22.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.456+0000 7f1ebb944700 1 -- start start 2019-12-05T06:26:22.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1ebb944700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1eb4128360 0x7f1eb4128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:22.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1ebb944700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb4137ea0 0x7f1eb4128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:22.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1ebb944700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1eb4154770 0x7f1eb412b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:22.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1ebb944700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb412a640 con 0x7f1eb4128360 2019-12-05T06:26:22.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1ebb944700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb4129c10 con 0x7f1eb4154770 2019-12-05T06:26:22.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1ebb944700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb4129ee0 con 0x7f1eb4137ea0 2019-12-05T06:26:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1eb9ee1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1eb4128360 0x7f1eb4128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1eb9ee1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1eb4128360 0x7f1eb4128750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53220/0 (socket says 172.21.15.16:53220) 2019-12-05T06:26:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1eb9ee1700 1 -- 172.21.15.16:0/836009044 learned_addr learned my addr 172.21.15.16:0/836009044 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:22.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1eba6e2700 1 --2- 172.21.15.16:0/836009044 >> v2:172.21.15.76:3300/0 conn(0x7f1eb4154770 0x7f1eb412b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.458+0000 7f1eb96e0700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb4137ea0 0x7f1eb4128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 -- 172.21.15.16:0/836009044 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb4137ea0 msgr2=0x7f1eb4128c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb4137ea0 0x7f1eb4128c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:22.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 -- 172.21.15.16:0/836009044 >> v2:172.21.15.76:3300/0 conn(0x7f1eb4154770 msgr2=0x7f1eb412b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:22.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> v2:172.21.15.76:3300/0 conn(0x7f1eb4154770 0x7f1eb412b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:22.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 -- 172.21.15.16:0/836009044 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb412acd0 con 0x7f1eb4128360 2019-12-05T06:26:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> v2:172.21.15.16:3300/0 conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> v2:172.21.15.16:3300/0 conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=READY pgs=752 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:22.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb27fc700 1 -- 172.21.15.16:0/836009044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1eac000cc0 con 0x7f1eb4128360 2019-12-05T06:26:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb27fc700 1 -- 172.21.15.16:0/836009044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac009d00 con 0x7f1eb4128360 2019-12-05T06:26:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1eb27fc700 1 -- 172.21.15.16:0/836009044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1eac00a3d0 con 0x7f1eb4128360 2019-12-05T06:26:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.459+0000 7f1ebb944700 1 -- 172.21.15.16:0/836009044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eb412acd0 con 0x7f1eb4128360 2019-12-05T06:26:22.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.460+0000 7f1ebb944700 1 -- 172.21.15.16:0/836009044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb412acd0 con 0x7f1eb4128360 2019-12-05T06:26:22.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.461+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.462+0000 7f1eb27fc700 1 -- 172.21.15.16:0/836009044 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1eac009ef0 con 0x7f1eb4128360 2019-12-05T06:26:22.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.462+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.462+0000 7f1eb27fc700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e94023630 0x7f1e94025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:22.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.462+0000 7f1eb27fc700 1 -- 172.21.15.16:0/836009044 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 7063+0+0 (secure 0 0 0) 0x7f1eac0460a0 con 0x7f1eb4128360 2019-12-05T06:26:22.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.463+0000 7f1eb96e0700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e94023630 0x7f1e94025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:22.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.463+0000 7f1eb96e0700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e94023630 0x7f1e94025ac0 crc :-1 s=READY pgs=329 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:22.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.476+0000 7f1ebb944700 1 -- 172.21.15.16:0/836009044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb412acd0 con 0x7f1eb4128360 2019-12-05T06:26:22.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.484+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.485+0000 7f1eb27fc700 1 -- 172.21.15.16:0/836009044 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1eac009900 con 0x7f1eb4128360 2019-12-05T06:26:22.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.738+0000 7f1ebb944700 1 -- 172.21.15.16:0/836009044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1eb412acd0 con 0x7f1eb4128360 2019-12-05T06:26:22.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.739+0000 7f1eb9ee1700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=THROTTLE_DONE pgs=752 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:22.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.739+0000 7f1eb27fc700 1 -- 172.21.15.16:0/836009044 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 92 v92) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f1eac009ec0 con 0x7f1eb4128360 2019-12-05T06:26:22.726 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 92 2019-12-05T06:26:22.727 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:22.727 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":92,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5135,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":92}],"filesystems":[{"mdsmap":{"epoch":91,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:24:48.206250+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":61,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5940":{"gid":5940,"name":"a","rank":0,"incarnation":88,"state":"up:active","state_seq":21,"addr":"172.21.15.16:6833/3836369875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3836369875},{"type":"v1","addr":"172.21.15.16:6833","nonce":3836369875}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.744+0000 7f1ebb944700 1 -- 172.21.15.16:0/836009044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e94023630 msgr2=0x7f1e94025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.744+0000 7f1ebb944700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e94023630 0x7f1e94025ac0 crc :-1 s=READY pgs=329 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.744+0000 7f1ebb944700 1 -- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 msgr2=0x7f1eb4128750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:22.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.744+0000 7f1ebb944700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 0x7f1eb4128750 secure :-1 s=READY pgs=752 cs=0 l=1 rx=0x7f1eac0107d0 tx=0x7f1eac0106d0).stop 2019-12-05T06:26:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.744+0000 7f1ebb944700 1 -- 172.21.15.16:0/836009044 shutdown_connections 2019-12-05T06:26:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.744+0000 7f1ebb944700 1 --2- 172.21.15.16:0/836009044 >> v2:172.21.15.76:3300/0 conn(0x7f1eb4154770 0x7f1eb412b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.744+0000 7f1ebb944700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1eb4137ea0 0x7f1eb4128c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:22.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.744+0000 7f1ebb944700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e94023630 0x7f1e94025ac0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.744+0000 7f1ebb944700 1 --2- 172.21.15.16:0/836009044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1eb4128360 0x7f1eb4128750 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.745+0000 7f1ebb944700 1 -- 172.21.15.16:0/836009044 shutdown_connections 2019-12-05T06:26:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.745+0000 7f1ebb944700 1 -- 172.21.15.16:0/836009044 wait complete. 2019-12-05T06:26:22.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:22.745+0000 7f1ebb944700 1 -- 172.21.15.16:0/836009044 >> 172.21.15.16:0/836009044 conn(0x7f1eb406d260 msgr2=0x7f1eb409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:22.741 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:26:24.742 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:25.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.024+0000 7f4d50933700 1 Processor -- start 2019-12-05T06:26:25.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.024+0000 7f4d50933700 1 -- start start 2019-12-05T06:26:25.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.026+0000 7f4d50933700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:25.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.026+0000 7f4d50933700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d4c08f720 0x7f4d4c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:25.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.026+0000 7f4d50933700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c0958e0 con 0x7f4d4c08fef0 2019-12-05T06:26:25.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.026+0000 7f4d50933700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4c0a3f70 con 0x7f4d4c08f720 2019-12-05T06:26:25.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.026+0000 7f4d50933700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4c08d830 con 0x7f4d4c08c6d0 2019-12-05T06:26:25.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.026+0000 7f4d4b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d4c08f720 0x7f4d4c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.026+0000 7f4d4ad9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d4b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4d4c08f720 0x7f4d4c08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53230/0 (socket says 172.21.15.16:53230) 2019-12-05T06:26:25.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d4ad9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:57218/0 (socket says 172.21.15.16:57218) 2019-12-05T06:26:25.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d4a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f4d4c08fef0 0x7f4d4c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:59702/0 (socket says 172.21.15.16:59702) 2019-12-05T06:26:25.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d4b59e700 1 -- 172.21.15.16:0/701145251 learned_addr learned my addr 172.21.15.16:0/701145251 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d49d9b700 1 -- 172.21.15.16:0/701145251 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528986726 0 0) 0x7f4d40001070 con 0x7f4d4c08fef0 2019-12-05T06:26:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d4ad9d700 1 -- 172.21.15.16:0/701145251 >> v2:172.21.15.16:3300/0 conn(0x7f4d4c08f720 msgr2=0x7f4d4c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/701145251 >> v2:172.21.15.16:3300/0 conn(0x7f4d4c08f720 0x7f4d4c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d4ad9d700 1 -- 172.21.15.16:0/701145251 >> v1:172.21.15.76:6789/0 conn(0x7f4d4c08fef0 legacy=0x7f4d4c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d4ad9d700 1 -- 172.21.15.16:0/701145251 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c08fbc0 con 0x7f4d4c08c6d0 2019-12-05T06:26:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d4b59e700 1 --2- 172.21.15.16:0/701145251 >> v2:172.21.15.16:3300/0 conn(0x7f4d4c08f720 0x7f4d4c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:26:25.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.027+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/701145251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4d3c00b2d0 tx=0x7f4d3c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.028+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/701145251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4d3c00b2d0 tx=0x7f4d3c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.028+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/701145251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 secure :-1 s=READY pgs=187 cs=0 l=1 rx=0x7f4d3c00b2d0 tx=0x7f4d3c00bae0).ready entity=mon.2 client_cookie=ee2c1d5b6bca1564 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.028+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/701145251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7f4d3c00b2d0 tx=0x7f4d3c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.028+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/701145251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7f4d3c00b2d0 tx=0x7f4d3c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.028+0000 7f4d49d9b700 1 -- 172.21.15.16:0/701145251 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d3c007590 con 0x7f4d4c08c6d0 2019-12-05T06:26:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.028+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/701145251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 secure :-1 s=THROTTLE_DONE pgs=187 cs=0 l=1 rx=0x7f4d3c00b2d0 tx=0x7f4d3c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.028+0000 7f4d49d9b700 1 -- 172.21.15.16:0/701145251 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c009d30 con 0x7f4d4c08c6d0 2019-12-05T06:26:25.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.028+0000 7f4d49d9b700 1 -- 172.21.15.16:0/701145251 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d3c00a300 con 0x7f4d4c08c6d0 2019-12-05T06:26:25.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.028+0000 7f4d50933700 1 -- 172.21.15.16:0/701145251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 msgr2=0x7f4d4c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.028+0000 7f4d50933700 1 --2- 172.21.15.16:0/701145251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 secure :-1 s=READY pgs=187 cs=0 l=1 rx=0x7f4d3c00b2d0 tx=0x7f4d3c00bae0).stop 2019-12-05T06:26:25.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.030+0000 7f4d50933700 1 -- 172.21.15.16:0/701145251 shutdown_connections 2019-12-05T06:26:25.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.030+0000 7f4d50933700 1 --2- 172.21.15.16:0/701145251 >> v2:172.21.15.16:3300/0 conn(0x7f4d4c08f720 0x7f4d4c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:25.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.030+0000 7f4d50933700 1 --2- 172.21.15.16:0/701145251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4d4c08c6d0 0x7f4d4c08cac0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.030+0000 7f4d50933700 1 -- 172.21.15.16:0/701145251 shutdown_connections 2019-12-05T06:26:25.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.030+0000 7f4d50933700 1 -- 172.21.15.16:0/701145251 wait complete. 2019-12-05T06:26:25.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.030+0000 7f4d50933700 1 -- 172.21.15.16:0/701145251 >> 172.21.15.16:0/701145251 conn(0x7f4d4c061940 msgr2=0x7f4d4c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.035+0000 7f4d50933700 1 Processor -- start 2019-12-05T06:26:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.035+0000 7f4d50933700 1 -- start start 2019-12-05T06:26:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.036+0000 7f4d50933700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4d4c1325e0 0x7f4d4c12b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:25.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.036+0000 7f4d50933700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c128b20 con 0x7f4d4c12c740 2019-12-05T06:26:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.036+0000 7f4d50933700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d4c128660 con 0x7f4d4c154c60 2019-12-05T06:26:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.036+0000 7f4d50933700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d4c12a7e0 con 0x7f4d4c1325e0 2019-12-05T06:26:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.036+0000 7f4d4a59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4d4c12c740 0x7f4d4c12bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38406/0 (socket says 172.21.15.16:38406) 2019-12-05T06:26:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.036+0000 7f4d4ad9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4d4c1325e0 0x7f4d4c12b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.036+0000 7f4d4a59c700 1 -- 172.21.15.16:0/1862222111 learned_addr learned my addr 172.21.15.16:0/1862222111 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.036+0000 7f4d3b7fe700 1 -- 172.21.15.16:0/1862222111 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1787306753 0 0) 0x7f4d40001180 con 0x7f4d4c12c740 2019-12-05T06:26:25.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.037+0000 7f4d3b7fe700 1 -- 172.21.15.16:0/1862222111 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d30002e80 con 0x7f4d4c12c740 2019-12-05T06:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.037+0000 7f4d3b7fe700 1 -- 172.21.15.16:0/1862222111 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 502747689 0 0) 0x7f4d44007580 con 0x7f4d4c154c60 2019-12-05T06:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.037+0000 7f4d4ad9d700 1 -- 172.21.15.16:0/1862222111 >> v1:172.21.15.76:6789/0 conn(0x7f4d4c154c60 legacy=0x7f4d4c157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.037+0000 7f4d4ad9d700 1 -- 172.21.15.16:0/1862222111 >> v1:172.21.15.16:6789/0 conn(0x7f4d4c12c740 legacy=0x7f4d4c12bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.037+0000 7f4d4ad9d700 1 -- 172.21.15.16:0/1862222111 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4c12ae70 con 0x7f4d4c1325e0 2019-12-05T06:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.037+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/1862222111 >> v2:172.21.15.76:3300/0 conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.037+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/1862222111 >> v2:172.21.15.76:3300/0 conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.037+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=READY pgs=548 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.037+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.038+0000 7f4d3b7fe700 1 -- 172.21.15.16:0/1862222111 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d3c009900 con 0x7f4d4c1325e0 2019-12-05T06:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.038+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.038+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.038+0000 7f4d3b7fe700 1 -- 172.21.15.16:0/1862222111 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d3c00b800 con 0x7f4d4c1325e0 2019-12-05T06:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.038+0000 7f4d3b7fe700 1 -- 172.21.15.16:0/1862222111 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d3c00b020 con 0x7f4d4c1325e0 2019-12-05T06:26:25.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.038+0000 7f4d50933700 1 -- 172.21.15.16:0/1862222111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4c12ae70 con 0x7f4d4c1325e0 2019-12-05T06:26:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.038+0000 7f4d50933700 1 -- 172.21.15.16:0/1862222111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d4c12ae70 con 0x7f4d4c1325e0 2019-12-05T06:26:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.039+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.040+0000 7f4d3b7fe700 1 -- 172.21.15.16:0/1862222111 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4d3c009900 con 0x7f4d4c1325e0 2019-12-05T06:26:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.040+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.040+0000 7f4d3b7fe700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d30023680 0x7f4d30025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:25.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.040+0000 7f4d3b7fe700 1 -- 172.21.15.16:0/1862222111 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f4d3c08a140 con 0x7f4d4c1325e0 2019-12-05T06:26:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.040+0000 7f4d4a59c700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d30023680 0x7f4d30025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.041+0000 7f4d4a59c700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d30023680 0x7f4d30025af0 crc :-1 s=READY pgs=330 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:25.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.056+0000 7f4d50933700 1 -- 172.21.15.16:0/1862222111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d4c12ae70 con 0x7f4d4c1325e0 2019-12-05T06:26:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.064+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.064+0000 7f4d3b7fe700 1 -- 172.21.15.16:0/1862222111 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4d3c00b020 con 0x7f4d4c1325e0 2019-12-05T06:26:25.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.325+0000 7f4d50933700 1 -- 172.21.15.16:0/1862222111 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4d4c12ae70 con 0x7f4d4c1325e0 2019-12-05T06:26:25.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.327+0000 7f4d4ad9d700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=THROTTLE_DONE pgs=548 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.327+0000 7f4d3b7fe700 1 -- 172.21.15.16:0/1862222111 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f4d3c009900 con 0x7f4d4c1325e0 2019-12-05T06:26:25.313 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:25.314 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:23.186071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:25.314 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 93 2019-12-05T06:26:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.331+0000 7f4d50933700 1 -- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d30023680 msgr2=0x7f4d30025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.331+0000 7f4d50933700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d30023680 0x7f4d30025af0 crc :-1 s=READY pgs=330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.331+0000 7f4d50933700 1 -- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 msgr2=0x7f4d4c12b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.331+0000 7f4d50933700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 0x7f4d4c12b770 secure :-1 s=READY pgs=548 cs=0 l=1 rx=0x7f4d3c054730 tx=0x7f4d3c054ad0).stop 2019-12-05T06:26:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.331+0000 7f4d50933700 1 -- 172.21.15.16:0/1862222111 shutdown_connections 2019-12-05T06:26:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.331+0000 7f4d50933700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4d30023680 0x7f4d30025af0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.331+0000 7f4d50933700 1 --2- 172.21.15.16:0/1862222111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4d4c1325e0 0x7f4d4c12b770 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.331+0000 7f4d50933700 1 -- 172.21.15.16:0/1862222111 shutdown_connections 2019-12-05T06:26:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.331+0000 7f4d50933700 1 -- 172.21.15.16:0/1862222111 wait complete. 2019-12-05T06:26:25.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.331+0000 7f4d50933700 1 -- 172.21.15.16:0/1862222111 >> 172.21.15.16:0/1862222111 conn(0x7f4d4c0a2850 msgr2=0x7f4d4c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:25.338 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:26:25.338 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:26:25.338 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:25.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.607+0000 7f564e784700 1 Processor -- start 2019-12-05T06:26:25.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.607+0000 7f564e784700 1 -- start start 2019-12-05T06:26:25.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.608+0000 7f564e784700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f564808d2c0 0x7f564808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:25.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.608+0000 7f564e784700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f564808fef0 0x7f564808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:25.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.608+0000 7f564e784700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56480958e0 con 0x7f564808c6d0 2019-12-05T06:26:25.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.608+0000 7f564e784700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564808df90 con 0x7f564808fef0 2019-12-05T06:26:25.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.608+0000 7f564e784700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f56480a3f70 con 0x7f564808d2c0 2019-12-05T06:26:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.608+0000 7f564d522700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f564808fef0 0x7f564808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.608+0000 7f564d522700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f564808fef0 0x7f564808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53246/0 (socket says 172.21.15.16:53246) 2019-12-05T06:26:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.608+0000 7f564d522700 1 -- 172.21.15.16:0/1751154879 learned_addr learned my addr 172.21.15.16:0/1751154879 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.608+0000 7f5647fff700 1 --2- 172.21.15.16:0/1751154879 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f564808d2c0 0x7f564808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.609+0000 7f564d522700 1 -- 172.21.15.16:0/1751154879 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f564808d2c0 msgr2=0x7f564808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.609+0000 7f564d522700 1 --2- 172.21.15.16:0/1751154879 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f564808d2c0 0x7f564808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:25.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.609+0000 7f564d522700 1 -- 172.21.15.16:0/1751154879 >> v1:172.21.15.16:6789/0 conn(0x7f564808c6d0 legacy=0x7f564808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.609+0000 7f56477fe700 1 -- 172.21.15.16:0/1751154879 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2110046245 0 0) 0x7f5640001480 con 0x7f564808c6d0 2019-12-05T06:26:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.609+0000 7f564d522700 1 -- 172.21.15.16:0/1751154879 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564808e620 con 0x7f564808fef0 2019-12-05T06:26:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.609+0000 7f5647fff700 1 --2- 172.21.15.16:0/1751154879 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f564808d2c0 0x7f564808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:26:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.609+0000 7f564d522700 1 --2- 172.21.15.16:0/1751154879 >> v2:172.21.15.16:3300/0 conn(0x7f564808fef0 0x7f564808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f563000f730 tx=0x7f563000fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.610+0000 7f564d522700 1 --2- 172.21.15.16:0/1751154879 >> v2:172.21.15.16:3300/0 conn(0x7f564808fef0 0x7f564808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f563000f730 tx=0x7f563000fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.610+0000 7f564d522700 1 --2- 172.21.15.16:0/1751154879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f564808fef0 0x7f564808f460 secure :-1 s=READY pgs=755 cs=0 l=1 rx=0x7f563000f730 tx=0x7f563000fad0).ready entity=mon.0 client_cookie=fefe3e91cf86a3e6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:25.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.610+0000 7f564d522700 1 --2- 172.21.15.16:0/1751154879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f564808fef0 0x7f564808f460 secure :-1 s=THROTTLE_DONE pgs=755 cs=0 l=1 rx=0x7f563000f730 tx=0x7f563000fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.610+0000 7f56477fe700 1 -- 172.21.15.16:0/1751154879 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5630000b90 con 0x7f564808fef0 2019-12-05T06:26:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.610+0000 7f564d522700 1 --2- 172.21.15.16:0/1751154879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f564808fef0 0x7f564808f460 secure :-1 s=THROTTLE_DONE pgs=755 cs=0 l=1 rx=0x7f563000f730 tx=0x7f563000fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.610+0000 7f564d522700 1 --2- 172.21.15.16:0/1751154879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f564808fef0 0x7f564808f460 secure :-1 s=THROTTLE_DONE pgs=755 cs=0 l=1 rx=0x7f563000f730 tx=0x7f563000fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.610+0000 7f56477fe700 1 -- 172.21.15.16:0/1751154879 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56300010e0 con 0x7f564808fef0 2019-12-05T06:26:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.610+0000 7f56477fe700 1 -- 172.21.15.16:0/1751154879 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5630000b90 con 0x7f564808fef0 2019-12-05T06:26:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.610+0000 7f564e784700 1 -- 172.21.15.16:0/1751154879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f564808fef0 msgr2=0x7f564808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.610+0000 7f564e784700 1 --2- 172.21.15.16:0/1751154879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f564808fef0 0x7f564808f460 secure :-1 s=READY pgs=755 cs=0 l=1 rx=0x7f563000f730 tx=0x7f563000fad0).stop 2019-12-05T06:26:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.612+0000 7f564e784700 1 -- 172.21.15.16:0/1751154879 shutdown_connections 2019-12-05T06:26:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.612+0000 7f564e784700 1 --2- 172.21.15.16:0/1751154879 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f564808fef0 0x7f564808f460 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.612+0000 7f564e784700 1 --2- 172.21.15.16:0/1751154879 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f564808d2c0 0x7f564808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:25.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.612+0000 7f564e784700 1 -- 172.21.15.16:0/1751154879 shutdown_connections 2019-12-05T06:26:25.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.612+0000 7f564e784700 1 -- 172.21.15.16:0/1751154879 wait complete. 2019-12-05T06:26:25.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.612+0000 7f564e784700 1 -- 172.21.15.16:0/1751154879 >> 172.21.15.16:0/1751154879 conn(0x7f5648061940 msgr2=0x7f5648096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.616+0000 7f564e784700 1 Processor -- start 2019-12-05T06:26:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.616+0000 7f564e784700 1 -- start start 2019-12-05T06:26:25.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.618+0000 7f564e784700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5648096e90 0x7f564809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:25.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.618+0000 7f564e784700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5648084f30 0x7f564809b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:25.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.618+0000 7f564e784700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5648084b90 con 0x7f5648097280 2019-12-05T06:26:25.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.618+0000 7f564e784700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f564809a450 con 0x7f5648084f30 2019-12-05T06:26:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.618+0000 7f564e784700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5648099a20 con 0x7f5648096e90 2019-12-05T06:26:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.618+0000 7f5647fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5648084f30 0x7f564809b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.618+0000 7f5647fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5648084f30 0x7f564809b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53250/0 (socket says 172.21.15.16:53250) 2019-12-05T06:26:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.618+0000 7f5647fff700 1 -- 172.21.15.16:0/1218417266 learned_addr learned my addr 172.21.15.16:0/1218417266 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:25.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.618+0000 7f564cd21700 1 --2- 172.21.15.16:0/1218417266 >> v2:172.21.15.76:3300/0 conn(0x7f5648096e90 0x7f564809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f5647fff700 1 -- 172.21.15.16:0/1218417266 >> v2:172.21.15.76:3300/0 conn(0x7f5648096e90 msgr2=0x7f564809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> v2:172.21.15.76:3300/0 conn(0x7f5648096e90 0x7f564809b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f56457fa700 1 -- 172.21.15.16:0/1218417266 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3708727180 0 0) 0x7f5630000b50 con 0x7f5648097280 2019-12-05T06:26:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f5647fff700 1 -- 172.21.15.16:0/1218417266 >> v1:172.21.15.76:6789/0 conn(0x7f5648097280 legacy=0x7f56480976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f5647fff700 1 -- 172.21.15.16:0/1218417266 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564809aae0 con 0x7f5648084f30 2019-12-05T06:26:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f564cd21700 1 --2- 172.21.15.16:0/1218417266 >> v2:172.21.15.76:3300/0 conn(0x7f5648096e90 0x7f564809b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:26:25.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> v2:172.21.15.16:3300/0 conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> v2:172.21.15.16:3300/0 conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=READY pgs=756 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f56457fa700 1 -- 172.21.15.16:0/1218417266 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5638000940 con 0x7f5648084f30 2019-12-05T06:26:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f56457fa700 1 -- 172.21.15.16:0/1218417266 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563800db40 con 0x7f5648084f30 2019-12-05T06:26:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f56457fa700 1 -- 172.21.15.16:0/1218417266 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5638000940 con 0x7f5648084f30 2019-12-05T06:26:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f564e784700 1 -- 172.21.15.16:0/1218417266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564809aae0 con 0x7f5648084f30 2019-12-05T06:26:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.619+0000 7f564e784700 1 -- 172.21.15.16:0/1218417266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564809aae0 con 0x7f5648084f30 2019-12-05T06:26:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.620+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.621+0000 7f56457fa700 1 -- 172.21.15.16:0/1218417266 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f563800dd30 con 0x7f5648084f30 2019-12-05T06:26:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.622+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.622+0000 7f56457fa700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f562c023630 0x7f562c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.622+0000 7f56457fa700 1 -- 172.21.15.16:0/1218417266 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f5638047e80 con 0x7f5648084f30 2019-12-05T06:26:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.622+0000 7f564cd21700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f562c023630 0x7f562c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.623+0000 7f564cd21700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f562c023630 0x7f562c025ac0 crc :-1 s=READY pgs=331 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:25.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.623+0000 7f564e784700 1 -- 172.21.15.16:0/1218417266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564809aae0 con 0x7f5648084f30 2019-12-05T06:26:25.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.632+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.632+0000 7f56457fa700 1 -- 172.21.15.16:0/1218417266 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f563800a5b0 con 0x7f5648084f30 2019-12-05T06:26:25.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.897+0000 7f564e784700 1 -- 172.21.15.16:0/1218417266 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f564809aae0 con 0x7f5648084f30 2019-12-05T06:26:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.897+0000 7f5647fff700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.897+0000 7f56457fa700 1 -- 172.21.15.16:0/1218417266 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 93 v93) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f563800dd30 con 0x7f5648084f30 2019-12-05T06:26:25.883 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 93 2019-12-05T06:26:25.883 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:25.883 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":93,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":93,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:23.186071+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:25.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.900+0000 7f564e784700 1 -- 172.21.15.16:0/1218417266 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f562c023630 msgr2=0x7f562c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.900+0000 7f564e784700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f562c023630 0x7f562c025ac0 crc :-1 s=READY pgs=331 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.900+0000 7f564e784700 1 -- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 msgr2=0x7f564809b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.900+0000 7f564e784700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 0x7f564809b920 secure :-1 s=READY pgs=756 cs=0 l=1 rx=0x7f56380125a0 tx=0x7f5638012940).stop 2019-12-05T06:26:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.900+0000 7f564e784700 1 -- 172.21.15.16:0/1218417266 shutdown_connections 2019-12-05T06:26:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.901+0000 7f564e784700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f562c023630 0x7f562c025ac0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.901+0000 7f564e784700 1 --2- 172.21.15.16:0/1218417266 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5648084f30 0x7f564809b920 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.901+0000 7f564e784700 1 --2- 172.21.15.16:0/1218417266 >> v2:172.21.15.76:3300/0 conn(0x7f5648096e90 0x7f564809b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.901+0000 7f564e784700 1 -- 172.21.15.16:0/1218417266 shutdown_connections 2019-12-05T06:26:25.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.901+0000 7f564e784700 1 -- 172.21.15.16:0/1218417266 wait complete. 2019-12-05T06:26:25.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:25.901+0000 7f564e784700 1 -- 172.21.15.16:0/1218417266 >> 172.21.15.16:0/1218417266 conn(0x7f56480714f0 msgr2=0x7f564809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:25.900 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 93, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 93, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5135": { "addr": "172.21.15.76:6835/2135855524", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 2135855524, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 2135855524, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5135, "incarnation": 93, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 62, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:26:23.186071+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5135 } } } ], "standbys": [] } 2019-12-05T06:26:27.901 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.184+0000 7f3aabd30700 1 Processor -- start 2019-12-05T06:26:28.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.184+0000 7f3aabd30700 1 -- start start 2019-12-05T06:26:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.186+0000 7f3aabd30700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3aa408c6d0 0x7f3aa408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.186+0000 7f3aabd30700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aa408fef0 0x7f3aa408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.186+0000 7f3aabd30700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa408ef20 0x7f3aa408f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.186+0000 7f3aabd30700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa408df90 con 0x7f3aa408fef0 2019-12-05T06:26:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.186+0000 7f3aabd30700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3aa40a3f70 con 0x7f3aa408c6d0 2019-12-05T06:26:28.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.186+0000 7f3aabd30700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa408d830 con 0x7f3aa408ef20 2019-12-05T06:26:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.187+0000 7f3aa9acc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aa408fef0 0x7f3aa408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.187+0000 7f3aa9acc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3aa408fef0 0x7f3aa408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53260/0 (socket says 172.21.15.16:53260) 2019-12-05T06:26:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.187+0000 7f3aa9acc700 1 -- 172.21.15.16:0/1525464378 learned_addr learned my addr 172.21.15.16:0/1525464378 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.187+0000 7f3aaaace700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa408ef20 0x7f3aa408f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.187+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/1525464378 >> v2:172.21.15.76:3300/0 conn(0x7f3aa408c6d0 0x7f3aa408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.187+0000 7f3aa9acc700 1 -- 172.21.15.16:0/1525464378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa408ef20 msgr2=0x7f3aa408f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:28.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.187+0000 7f3aa9acc700 1 --2- 172.21.15.16:0/1525464378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa408ef20 0x7f3aa408f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.187+0000 7f3aa9acc700 1 -- 172.21.15.16:0/1525464378 >> v2:172.21.15.76:3300/0 conn(0x7f3aa408c6d0 msgr2=0x7f3aa408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.187+0000 7f3aa9acc700 1 --2- 172.21.15.16:0/1525464378 >> v2:172.21.15.76:3300/0 conn(0x7f3aa408c6d0 0x7f3aa408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.187+0000 7f3aa9acc700 1 -- 172.21.15.16:0/1525464378 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa408d540 con 0x7f3aa408fef0 2019-12-05T06:26:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.188+0000 7f3aa9acc700 1 --2- 172.21.15.16:0/1525464378 >> v2:172.21.15.16:3300/0 conn(0x7f3aa408fef0 0x7f3aa408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3a9800f6c0 tx=0x7f3a9800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.188+0000 7f3aa9acc700 1 --2- 172.21.15.16:0/1525464378 >> v2:172.21.15.16:3300/0 conn(0x7f3aa408fef0 0x7f3aa408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3a9800f6c0 tx=0x7f3a9800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.188+0000 7f3aa9acc700 1 --2- 172.21.15.16:0/1525464378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa408fef0 0x7f3aa408d000 secure :-1 s=READY pgs=757 cs=0 l=1 rx=0x7f3a9800f6c0 tx=0x7f3a9800f570).ready entity=mon.0 client_cookie=832ada4ed81b9847 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:28.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.190+0000 7f3aa9acc700 1 --2- 172.21.15.16:0/1525464378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa408fef0 0x7f3aa408d000 secure :-1 s=THROTTLE_DONE pgs=757 cs=0 l=1 rx=0x7f3a9800f6c0 tx=0x7f3a9800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.190+0000 7f3aa9acc700 1 --2- 172.21.15.16:0/1525464378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa408fef0 0x7f3aa408d000 secure :-1 s=THROTTLE_DONE pgs=757 cs=0 l=1 rx=0x7f3a9800f6c0 tx=0x7f3a9800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.190+0000 7f3aa92cb700 1 -- 172.21.15.16:0/1525464378 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a98000960 con 0x7f3aa408fef0 2019-12-05T06:26:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.190+0000 7f3aa9acc700 1 --2- 172.21.15.16:0/1525464378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa408fef0 0x7f3aa408d000 secure :-1 s=THROTTLE_DONE pgs=757 cs=0 l=1 rx=0x7f3a9800f6c0 tx=0x7f3a9800f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.191+0000 7f3aa92cb700 1 -- 172.21.15.16:0/1525464378 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a98000c90 con 0x7f3aa408fef0 2019-12-05T06:26:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.191+0000 7f3aa92cb700 1 -- 172.21.15.16:0/1525464378 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a980013a0 con 0x7f3aa408fef0 2019-12-05T06:26:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.191+0000 7f3aabd30700 1 -- 172.21.15.16:0/1525464378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa408fef0 msgr2=0x7f3aa408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:28.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.191+0000 7f3aabd30700 1 --2- 172.21.15.16:0/1525464378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa408fef0 0x7f3aa408d000 secure :-1 s=READY pgs=757 cs=0 l=1 rx=0x7f3a9800f6c0 tx=0x7f3a9800f570).stop 2019-12-05T06:26:28.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.193+0000 7f3aabd30700 1 -- 172.21.15.16:0/1525464378 shutdown_connections 2019-12-05T06:26:28.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.193+0000 7f3aabd30700 1 --2- 172.21.15.16:0/1525464378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa408ef20 0x7f3aa408f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:28.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.193+0000 7f3aabd30700 1 --2- 172.21.15.16:0/1525464378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3aa408fef0 0x7f3aa408d000 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:28.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.193+0000 7f3aabd30700 1 --2- 172.21.15.16:0/1525464378 >> v2:172.21.15.76:3300/0 conn(0x7f3aa408c6d0 0x7f3aa408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:28.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.193+0000 7f3aabd30700 1 -- 172.21.15.16:0/1525464378 shutdown_connections 2019-12-05T06:26:28.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.193+0000 7f3aabd30700 1 -- 172.21.15.16:0/1525464378 wait complete. 2019-12-05T06:26:28.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.193+0000 7f3aabd30700 1 -- 172.21.15.16:0/1525464378 >> 172.21.15.16:0/1525464378 conn(0x7f3aa4061940 msgr2=0x7f3aa4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:28.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.196+0000 7f3aabd30700 1 Processor -- start 2019-12-05T06:26:28.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.196+0000 7f3aabd30700 1 -- start start 2019-12-05T06:26:28.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.197+0000 7f3aabd30700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:28.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.197+0000 7f3aabd30700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa4098da0 con 0x7f3aa4154b00 2019-12-05T06:26:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.197+0000 7f3aabd30700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3aa409f0a0 con 0x7f3aa4132d00 2019-12-05T06:26:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.197+0000 7f3aabd30700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3aa409a030 con 0x7f3aa409b850 2019-12-05T06:26:28.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.198+0000 7f3aaaace700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3aa4154b00 0x7f3aa4156f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38436/0 (socket says 172.21.15.16:38436) 2019-12-05T06:26:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.198+0000 7f3aaaace700 1 -- 172.21.15.16:0/3369230036 learned_addr learned my addr 172.21.15.16:0/3369230036 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.198+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.199+0000 7f3aaa2cd700 1 -- 172.21.15.16:0/3369230036 >> v1:172.21.15.76:6789/0 conn(0x7f3aa4132d00 legacy=0x7f3aa4132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.199+0000 7f3aaa2cd700 1 -- 172.21.15.16:0/3369230036 >> v1:172.21.15.16:6789/0 conn(0x7f3aa4154b00 legacy=0x7f3aa4156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.199+0000 7f3aaa2cd700 1 -- 172.21.15.16:0/3369230036 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa409b0f0 con 0x7f3aa409b850 2019-12-05T06:26:28.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.199+0000 7f3aa2ffd700 1 -- 172.21.15.16:0/3369230036 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2784286390 0 0) 0x7f3a9c009930 con 0x7f3aa4154b00 2019-12-05T06:26:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.199+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.199+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.199+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=READY pgs=188 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.202+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.202+0000 7f3aa2ffd700 1 -- 172.21.15.16:0/3369230036 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a94009640 con 0x7f3aa409b850 2019-12-05T06:26:28.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.202+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.202+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.202+0000 7f3aa2ffd700 1 -- 172.21.15.16:0/3369230036 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a94000940 con 0x7f3aa409b850 2019-12-05T06:26:28.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.202+0000 7f3aa2ffd700 1 -- 172.21.15.16:0/3369230036 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a9400a0e0 con 0x7f3aa409b850 2019-12-05T06:26:28.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.202+0000 7f3aabd30700 1 -- 172.21.15.16:0/3369230036 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aa409b0f0 con 0x7f3aa409b850 2019-12-05T06:26:28.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.202+0000 7f3aabd30700 1 -- 172.21.15.16:0/3369230036 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aa409b0f0 con 0x7f3aa409b850 2019-12-05T06:26:28.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.203+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.205+0000 7f3aa2ffd700 1 -- 172.21.15.16:0/3369230036 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3a94009990 con 0x7f3aa409b850 2019-12-05T06:26:28.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.205+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.205+0000 7f3aa2ffd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a84023630 0x7f3a84025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:28.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.205+0000 7f3aa2ffd700 1 -- 172.21.15.16:0/3369230036 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f3a94045da0 con 0x7f3aa409b850 2019-12-05T06:26:28.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.205+0000 7f3aa9acc700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a84023630 0x7f3a84025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:28.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.206+0000 7f3aa9acc700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a84023630 0x7f3a84025ac0 crc :-1 s=READY pgs=332 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:28.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.215+0000 7f3aabd30700 1 -- 172.21.15.16:0/3369230036 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aa409b0f0 con 0x7f3aa409b850 2019-12-05T06:26:28.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.229+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.229+0000 7f3aa2ffd700 1 -- 172.21.15.16:0/3369230036 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3a940095f0 con 0x7f3aa409b850 2019-12-05T06:26:28.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.493+0000 7f3aabd30700 1 -- 172.21.15.16:0/3369230036 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3aa409b0f0 con 0x7f3aa409b850 2019-12-05T06:26:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.495+0000 7f3aaa2cd700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=THROTTLE_DONE pgs=188 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.495+0000 7f3aa2ffd700 1 -- 172.21.15.16:0/3369230036 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f3a940098b0 con 0x7f3aa409b850 2019-12-05T06:26:28.479 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 96 2019-12-05T06:26:28.480 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:28.480 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:28.191316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:active","state_seq":27,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.496+0000 7f3aabd30700 1 -- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a84023630 msgr2=0x7f3a84025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.496+0000 7f3aabd30700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a84023630 0x7f3a84025ac0 crc :-1 s=READY pgs=332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:28.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.497+0000 7f3aabd30700 1 -- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 msgr2=0x7f3aa4131d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.497+0000 7f3aabd30700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 secure :-1 s=READY pgs=188 cs=0 l=1 rx=0x7f3a94010390 tx=0x7f3a94010730).stop 2019-12-05T06:26:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.497+0000 7f3aabd30700 1 -- 172.21.15.16:0/3369230036 shutdown_connections 2019-12-05T06:26:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.497+0000 7f3aabd30700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a84023630 0x7f3a84025ac0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.497+0000 7f3aabd30700 1 --2- 172.21.15.16:0/3369230036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3aa409b850 0x7f3aa4131d30 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.497+0000 7f3aabd30700 1 -- 172.21.15.16:0/3369230036 shutdown_connections 2019-12-05T06:26:28.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.497+0000 7f3aabd30700 1 -- 172.21.15.16:0/3369230036 wait complete. 2019-12-05T06:26:28.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:28.497+0000 7f3aabd30700 1 -- 172.21.15.16:0/3369230036 >> 172.21.15.16:0/3369230036 conn(0x7f3aa40714f0 msgr2=0x7f3aa409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:28.492 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5940 2019-12-05T06:26:28.492 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.a 2019-12-05T06:26:29.493 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:26:29.494 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:26:29.494 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:26:29.498 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:26:29.499 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:29.614 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:26:29.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.674+0000 7f9bff611700 1 Processor -- start 2019-12-05T06:26:29.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.675+0000 7f9bff611700 1 -- start start 2019-12-05T06:26:29.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.675+0000 7f9bff611700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf808c6d0 0x7f9bf808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.675+0000 7f9bff611700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9bf808f720 0x7f9bf808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.675+0000 7f9bff611700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf80958e0 con 0x7f9bf808fef0 2019-12-05T06:26:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.675+0000 7f9bff611700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf80a3f70 con 0x7f9bf808f720 2019-12-05T06:26:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.675+0000 7f9bff611700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9bf808d830 con 0x7f9bf808c6d0 2019-12-05T06:26:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfd3ad700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9bf808fef0 0x7f9bf808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38464/0 (socket says 172.21.15.16:38464) 2019-12-05T06:26:29.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfd3ad700 1 -- 172.21.15.16:0/2743491658 learned_addr learned my addr 172.21.15.16:0/2743491658 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfe3af700 1 --2- 172.21.15.16:0/2743491658 >> v2:172.21.15.76:3300/0 conn(0x7f9bf808f720 0x7f9bf808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfdbae700 1 --2- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf808c6d0 0x7f9bf808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfcbac700 1 -- 172.21.15.16:0/2743491658 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 488252073 0 0) 0x7f9bec001150 con 0x7f9bf808fef0 2019-12-05T06:26:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfcbac700 1 -- 172.21.15.16:0/2743491658 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9be0002e80 con 0x7f9bf808fef0 2019-12-05T06:26:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfe3af700 1 -- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf808c6d0 msgr2=0x7f9bf808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfe3af700 1 --2- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf808c6d0 0x7f9bf808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:29.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfe3af700 1 -- 172.21.15.16:0/2743491658 >> v1:172.21.15.16:6789/0 conn(0x7f9bf808fef0 legacy=0x7f9bf808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfe3af700 1 -- 172.21.15.16:0/2743491658 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf808fbc0 con 0x7f9bf808f720 2019-12-05T06:26:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfe3af700 1 --2- 172.21.15.16:0/2743491658 >> v2:172.21.15.76:3300/0 conn(0x7f9bf808f720 0x7f9bf808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9bf000af80 tx=0x7f9bf000b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfe3af700 1 --2- 172.21.15.16:0/2743491658 >> v2:172.21.15.76:3300/0 conn(0x7f9bf808f720 0x7f9bf808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9bf000af80 tx=0x7f9bf000b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfe3af700 1 --2- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bf808f720 0x7f9bf808d000 secure :-1 s=READY pgs=553 cs=0 l=1 rx=0x7f9bf000af80 tx=0x7f9bf000b710).ready entity=mon.1 client_cookie=ff9b3fc354855156 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfe3af700 1 --2- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bf808f720 0x7f9bf808d000 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f9bf000af80 tx=0x7f9bf000b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.676+0000 7f9bfcbac700 1 -- 172.21.15.16:0/2743491658 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bf0007240 con 0x7f9bf808f720 2019-12-05T06:26:29.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bfe3af700 1 --2- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bf808f720 0x7f9bf808d000 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f9bf000af80 tx=0x7f9bf000b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bfe3af700 1 --2- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bf808f720 0x7f9bf808d000 secure :-1 s=THROTTLE_DONE pgs=553 cs=0 l=1 rx=0x7f9bf000af80 tx=0x7f9bf000b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bfcbac700 1 -- 172.21.15.16:0/2743491658 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bf0000b10 con 0x7f9bf808f720 2019-12-05T06:26:29.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bfcbac700 1 -- 172.21.15.16:0/2743491658 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bf0009bd0 con 0x7f9bf808f720 2019-12-05T06:26:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bff611700 1 -- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bf808f720 msgr2=0x7f9bf808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bff611700 1 --2- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bf808f720 0x7f9bf808d000 secure :-1 s=READY pgs=553 cs=0 l=1 rx=0x7f9bf000af80 tx=0x7f9bf000b710).stop 2019-12-05T06:26:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bff611700 1 -- 172.21.15.16:0/2743491658 shutdown_connections 2019-12-05T06:26:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bff611700 1 --2- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9bf808f720 0x7f9bf808d000 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bff611700 1 --2- 172.21.15.16:0/2743491658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9bf808c6d0 0x7f9bf808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bff611700 1 -- 172.21.15.16:0/2743491658 shutdown_connections 2019-12-05T06:26:29.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bff611700 1 -- 172.21.15.16:0/2743491658 wait complete. 2019-12-05T06:26:29.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.677+0000 7f9bff611700 1 -- 172.21.15.16:0/2743491658 >> 172.21.15.16:0/2743491658 conn(0x7f9bf8061940 msgr2=0x7f9bf8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.680+0000 7f9bff611700 1 Processor -- start 2019-12-05T06:26:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.680+0000 7f9bff611700 1 -- start start 2019-12-05T06:26:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.680+0000 7f9bff611700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bf812c740 0x7f9bf812bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.680+0000 7f9bff611700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf8128b20 con 0x7f9bf8154c60 2019-12-05T06:26:29.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bff611700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9bf8128660 con 0x7f9bf81325e0 2019-12-05T06:26:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bff611700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9bf812a7e0 con 0x7f9bf812c740 2019-12-05T06:26:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bf812c740 0x7f9bf812bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9bf812c740 0x7f9bf812bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53304/0 (socket says 172.21.15.16:53304) 2019-12-05T06:26:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfe3af700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9bf8154c60 0x7f9bf8157060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38470/0 (socket says 172.21.15.16:38470) 2019-12-05T06:26:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 -- 172.21.15.16:0/3589014831 learned_addr learned my addr 172.21.15.16:0/3589014831 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:26:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 -- 172.21.15.16:0/3589014831 >> v1:172.21.15.76:6789/0 conn(0x7f9bf81325e0 legacy=0x7f9bf812b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:29.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 -- 172.21.15.16:0/3589014831 >> v1:172.21.15.16:6789/0 conn(0x7f9bf8154c60 legacy=0x7f9bf8157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 -- 172.21.15.16:0/3589014831 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bf812ae70 con 0x7f9bf812c740 2019-12-05T06:26:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bf67fc700 1 -- 172.21.15.16:0/3589014831 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164495673 0 0) 0x7f9bf000ac70 con 0x7f9bf8154c60 2019-12-05T06:26:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 --2- 172.21.15.16:0/3589014831 >> v2:172.21.15.16:3300/0 conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 --2- 172.21.15.16:0/3589014831 >> v2:172.21.15.16:3300/0 conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=READY pgs=764 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=THROTTLE_DONE pgs=764 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bf67fc700 1 -- 172.21.15.16:0/3589014831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bec000c90 con 0x7f9bf812c740 2019-12-05T06:26:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=THROTTLE_DONE pgs=764 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bf67fc700 1 -- 172.21.15.16:0/3589014831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bec0010c0 con 0x7f9bf812c740 2019-12-05T06:26:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bfd3ad700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=THROTTLE_DONE pgs=764 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bff611700 1 -- 172.21.15.16:0/3589014831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bf812ae70 con 0x7f9bf812c740 2019-12-05T06:26:29.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.681+0000 7f9bf67fc700 1 -- 172.21.15.16:0/3589014831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bec000e90 con 0x7f9bf812c740 2019-12-05T06:26:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.682+0000 7f9bff611700 1 -- 172.21.15.16:0/3589014831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bf812ae70 con 0x7f9bf812c740 2019-12-05T06:26:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.682+0000 7f9bfd3ad700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=THROTTLE_DONE pgs=764 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.682+0000 7f9bf67fc700 1 -- 172.21.15.16:0/3589014831 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9bec000c90 con 0x7f9bf812c740 2019-12-05T06:26:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.682+0000 7f9bfd3ad700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=THROTTLE_DONE pgs=764 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.682+0000 7f9bf67fc700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd8023630 0x7f9bd8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.682+0000 7f9bf67fc700 1 -- 172.21.15.16:0/3589014831 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f9bec045e00 con 0x7f9bf812c740 2019-12-05T06:26:29.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.683+0000 7f9bfdbae700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd8023630 0x7f9bd8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:29.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.683+0000 7f9bfdbae700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd8023630 0x7f9bd8025ac0 crc :-1 s=READY pgs=333 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:29.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.690+0000 7f9bff611700 1 -- 172.21.15.16:0/3589014831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bf812ae70 con 0x7f9bf812c740 2019-12-05T06:26:29.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.694+0000 7f9bfd3ad700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=THROTTLE_DONE pgs=764 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.694+0000 7f9bf67fc700 1 -- 172.21.15.16:0/3589014831 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9bec0075d0 con 0x7f9bf812c740 2019-12-05T06:26:29.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.887+0000 7f9bff611700 1 -- 172.21.15.16:0/3589014831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9bf812ae70 con 0x7f9bf812c740 2019-12-05T06:26:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.888+0000 7f9bfd3ad700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=THROTTLE_DONE pgs=764 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:29.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.888+0000 7f9bf67fc700 1 -- 172.21.15.16:0/3589014831 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 96 v96) v1 ==== 94+0+1772 (secure 0 0 0) 0x7f9bec000c90 con 0x7f9bf812c740 2019-12-05T06:26:29.876 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 96 2019-12-05T06:26:29.876 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:29.876 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":96,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:28.191316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:active","state_seq":27,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.892+0000 7f9bff611700 1 -- 172.21.15.16:0/3589014831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd8023630 msgr2=0x7f9bd8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.892+0000 7f9bff611700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd8023630 0x7f9bd8025ac0 crc :-1 s=READY pgs=333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.892+0000 7f9bff611700 1 -- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 msgr2=0x7f9bf812bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:29.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.892+0000 7f9bff611700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 0x7f9bf812bf70 secure :-1 s=READY pgs=764 cs=0 l=1 rx=0x7f9bec0103e0 tx=0x7f9bec010780).stop 2019-12-05T06:26:29.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.893+0000 7f9bff611700 1 -- 172.21.15.16:0/3589014831 shutdown_connections 2019-12-05T06:26:29.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.893+0000 7f9bff611700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9bd8023630 0x7f9bd8025ac0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:29.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.893+0000 7f9bff611700 1 --2- 172.21.15.16:0/3589014831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9bf812c740 0x7f9bf812bf70 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.893+0000 7f9bff611700 1 -- 172.21.15.16:0/3589014831 shutdown_connections 2019-12-05T06:26:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.893+0000 7f9bff611700 1 -- 172.21.15.16:0/3589014831 wait complete. 2019-12-05T06:26:29.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:29.893+0000 7f9bff611700 1 -- 172.21.15.16:0/3589014831 >> 172.21.15.16:0/3589014831 conn(0x7f9bf80a2850 msgr2=0x7f9bf806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:29.894 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:26:31.895 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:26:32.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.175+0000 7fddad3a6700 1 Processor -- start 2019-12-05T06:26:32.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.175+0000 7fddad3a6700 1 -- start start 2019-12-05T06:26:32.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.176+0000 7fddad3a6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdda808d2c0 0x7fdda808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:32.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.176+0000 7fddad3a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdda808fef0 0x7fdda808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:26:32.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.176+0000 7fddad3a6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda80958e0 con 0x7fdda808c6d0 2019-12-05T06:26:32.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.176+0000 7fddad3a6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda808df90 con 0x7fdda808fef0 2019-12-05T06:26:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.176+0000 7fddad3a6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda80a3f70 con 0x7fdda808d2c0 2019-12-05T06:26:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.177+0000 7fdda7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdda808fef0 0x7fdda808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.177+0000 7fdda77fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fdda808c6d0 0x7fdda808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:59796/0 (socket says 172.21.15.16:59796) 2019-12-05T06:26:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.177+0000 7fdda7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdda808fef0 0x7fdda808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53328/0 (socket says 172.21.15.16:53328) 2019-12-05T06:26:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.177+0000 7fdda77fe700 1 -- 172.21.15.16:0/38250230 learned_addr learned my addr 172.21.15.16:0/38250230 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.177+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/38250230 >> v2:172.21.15.76:3300/0 conn(0x7fdda808d2c0 0x7fdda808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda7fff700 1 -- 172.21.15.16:0/38250230 >> v2:172.21.15.76:3300/0 conn(0x7fdda808d2c0 msgr2=0x7fdda808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:26:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda7fff700 1 --2- 172.21.15.16:0/38250230 >> v2:172.21.15.76:3300/0 conn(0x7fdda808d2c0 0x7fdda808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda7fff700 1 -- 172.21.15.16:0/38250230 >> v1:172.21.15.76:6789/0 conn(0x7fdda808c6d0 legacy=0x7fdda808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda67fc700 1 -- 172.21.15.16:0/38250230 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854166892 0 0) 0x7fdd98001480 con 0x7fdda808c6d0 2019-12-05T06:26:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda7fff700 1 -- 172.21.15.16:0/38250230 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda808e620 con 0x7fdda808fef0 2019-12-05T06:26:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda7fff700 1 --2- 172.21.15.16:0/38250230 >> v2:172.21.15.16:3300/0 conn(0x7fdda808fef0 0x7fdda808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdda000af50 tx=0x7fdda000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda7fff700 1 --2- 172.21.15.16:0/38250230 >> v2:172.21.15.16:3300/0 conn(0x7fdda808fef0 0x7fdda808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdda000af50 tx=0x7fdda000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda7fff700 1 --2- 172.21.15.16:0/38250230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdda808fef0 0x7fdda808f460 secure :-1 s=READY pgs=765 cs=0 l=1 rx=0x7fdda000af50 tx=0x7fdda000b250).ready entity=mon.0 client_cookie=8e548cba68767af3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda7fff700 1 --2- 172.21.15.16:0/38250230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdda808fef0 0x7fdda808f460 secure :-1 s=THROTTLE_DONE pgs=765 cs=0 l=1 rx=0x7fdda000af50 tx=0x7fdda000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda67fc700 1 -- 172.21.15.16:0/38250230 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdda0009ba0 con 0x7fdda808fef0 2019-12-05T06:26:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda7fff700 1 --2- 172.21.15.16:0/38250230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdda808fef0 0x7fdda808f460 secure :-1 s=THROTTLE_DONE pgs=765 cs=0 l=1 rx=0x7fdda000af50 tx=0x7fdda000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.178+0000 7fdda7fff700 1 --2- 172.21.15.16:0/38250230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdda808fef0 0x7fdda808f460 secure :-1 s=THROTTLE_DONE pgs=765 cs=0 l=1 rx=0x7fdda000af50 tx=0x7fdda000b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.179+0000 7fdda67fc700 1 -- 172.21.15.16:0/38250230 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdda00097c0 con 0x7fdda808fef0 2019-12-05T06:26:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.179+0000 7fdda67fc700 1 -- 172.21.15.16:0/38250230 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdda000a180 con 0x7fdda808fef0 2019-12-05T06:26:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.179+0000 7fddad3a6700 1 -- 172.21.15.16:0/38250230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdda808fef0 msgr2=0x7fdda808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.179+0000 7fddad3a6700 1 --2- 172.21.15.16:0/38250230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdda808fef0 0x7fdda808f460 secure :-1 s=READY pgs=765 cs=0 l=1 rx=0x7fdda000af50 tx=0x7fdda000b250).stop 2019-12-05T06:26:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.180+0000 7fddad3a6700 1 -- 172.21.15.16:0/38250230 shutdown_connections 2019-12-05T06:26:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.180+0000 7fddad3a6700 1 --2- 172.21.15.16:0/38250230 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdda808fef0 0x7fdda808f460 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.180+0000 7fddad3a6700 1 --2- 172.21.15.16:0/38250230 >> v2:172.21.15.76:3300/0 conn(0x7fdda808d2c0 0x7fdda808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:26:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.180+0000 7fddad3a6700 1 -- 172.21.15.16:0/38250230 shutdown_connections 2019-12-05T06:26:32.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.180+0000 7fddad3a6700 1 -- 172.21.15.16:0/38250230 wait complete. 2019-12-05T06:26:32.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.180+0000 7fddad3a6700 1 -- 172.21.15.16:0/38250230 >> 172.21.15.16:0/38250230 conn(0x7fdda8061940 msgr2=0x7fdda8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:32.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.186+0000 7fddad3a6700 1 Processor -- start 2019-12-05T06:26:32.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.186+0000 7fddad3a6700 1 -- start start 2019-12-05T06:26:32.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.188+0000 7fddad3a6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdda8132d00 0x7fdda814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:32.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.188+0000 7fddad3a6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda8098da0 con 0x7fdda8132530 2019-12-05T06:26:32.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.188+0000 7fddad3a6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdda809f0a0 con 0x7fdda814d400 2019-12-05T06:26:32.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.188+0000 7fddad3a6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdda809bb50 con 0x7fdda8132d00 2019-12-05T06:26:32.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.188+0000 7fdda7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdda8132530 0x7fdda8157100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38504/0 (socket says 172.21.15.16:38504) 2019-12-05T06:26:32.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.188+0000 7fdda7fff700 1 -- 172.21.15.16:0/3452249195 learned_addr learned my addr 172.21.15.16:0/3452249195 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:26:32.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.188+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> v2:172.21.15.76:3300/0 conn(0x7fdda8132d00 0x7fdda814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:32.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.189+0000 7fdd9ffff700 1 -- 172.21.15.16:0/3452249195 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177580083 0 0) 0x7fdda0000940 con 0x7fdda8132530 2019-12-05T06:26:32.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.189+0000 7fdd9ffff700 1 -- 172.21.15.16:0/3452249195 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd8c002e80 con 0x7fdda8132530 2019-12-05T06:26:32.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.189+0000 7fdd9ffff700 1 -- 172.21.15.16:0/3452249195 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1148730853 0 0) 0x7fdd98001270 con 0x7fdda814d400 2019-12-05T06:26:32.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.189+0000 7fdda6ffd700 1 -- 172.21.15.16:0/3452249195 >> v1:172.21.15.76:6789/0 conn(0x7fdda814d400 legacy=0x7fdda8131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.189+0000 7fdda6ffd700 1 -- 172.21.15.16:0/3452249195 >> v1:172.21.15.16:6789/0 conn(0x7fdda8132530 legacy=0x7fdda8157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.189+0000 7fdda6ffd700 1 -- 172.21.15.16:0/3452249195 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdda809b0f0 con 0x7fdda8132d00 2019-12-05T06:26:32.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.189+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> v2:172.21.15.76:3300/0 conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.189+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> v2:172.21.15.76:3300/0 conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.189+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=READY pgs=557 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:32.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.190+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.190+0000 7fdd9ffff700 1 -- 172.21.15.16:0/3452249195 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd90000940 con 0x7fdda8132d00 2019-12-05T06:26:32.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.190+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.190+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.190+0000 7fdd9ffff700 1 -- 172.21.15.16:0/3452249195 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd90009c60 con 0x7fdda8132d00 2019-12-05T06:26:32.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.190+0000 7fdd9ffff700 1 -- 172.21.15.16:0/3452249195 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd90000940 con 0x7fdda8132d00 2019-12-05T06:26:32.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.190+0000 7fddad3a6700 1 -- 172.21.15.16:0/3452249195 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdda809b0f0 con 0x7fdda8132d00 2019-12-05T06:26:32.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.190+0000 7fddad3a6700 1 -- 172.21.15.16:0/3452249195 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdda809b0f0 con 0x7fdda8132d00 2019-12-05T06:26:32.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.191+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.192+0000 7fdd9ffff700 1 -- 172.21.15.16:0/3452249195 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdd90009810 con 0x7fdda8132d00 2019-12-05T06:26:32.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.192+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.192+0000 7fdd9ffff700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd8c023680 0x7fdd8c025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:26:32.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.192+0000 7fdd9ffff700 1 -- 172.21.15.16:0/3452249195 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7fdd90045e00 con 0x7fdda8132d00 2019-12-05T06:26:32.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.192+0000 7fdda77fe700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd8c023680 0x7fdd8c025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:26:32.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.193+0000 7fdda77fe700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd8c023680 0x7fdd8c025af0 crc :-1 s=READY pgs=335 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:26:32.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.206+0000 7fddad3a6700 1 -- 172.21.15.16:0/3452249195 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdda809b0f0 con 0x7fdda8132d00 2019-12-05T06:26:32.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.216+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.216+0000 7fdd9ffff700 1 -- 172.21.15.16:0/3452249195 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdd90000940 con 0x7fdda8132d00 2019-12-05T06:26:32.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.471+0000 7fddad3a6700 1 -- 172.21.15.16:0/3452249195 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdda809b0f0 con 0x7fdda8132d00 2019-12-05T06:26:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.472+0000 7fdda6ffd700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=THROTTLE_DONE pgs=557 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:26:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.472+0000 7fdd9ffff700 1 -- 172.21.15.16:0/3452249195 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fdd90009770 con 0x7fdda8132d00 2019-12-05T06:26:32.456 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 97 2019-12-05T06:26:32.457 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:26:32.457 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:28.191316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:active","state_seq":27,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:26:32.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.474+0000 7fddad3a6700 1 -- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd8c023680 msgr2=0x7fdd8c025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.474+0000 7fddad3a6700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd8c023680 0x7fdd8c025af0 crc :-1 s=READY pgs=335 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.474+0000 7fddad3a6700 1 -- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 msgr2=0x7fdda814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:26:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.474+0000 7fddad3a6700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 0x7fdda814d7f0 secure :-1 s=READY pgs=557 cs=0 l=1 rx=0x7fdd90010390 tx=0x7fdd90010730).stop 2019-12-05T06:26:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.475+0000 7fddad3a6700 1 -- 172.21.15.16:0/3452249195 shutdown_connections 2019-12-05T06:26:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.475+0000 7fddad3a6700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdd8c023680 0x7fdd8c025af0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:32.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.475+0000 7fddad3a6700 1 --2- 172.21.15.16:0/3452249195 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdda8132d00 0x7fdda814d7f0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:26:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.475+0000 7fddad3a6700 1 -- 172.21.15.16:0/3452249195 shutdown_connections 2019-12-05T06:26:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.475+0000 7fddad3a6700 1 -- 172.21.15.16:0/3452249195 wait complete. 2019-12-05T06:26:32.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:26:32.475+0000 7fddad3a6700 1 -- 172.21.15.16:0/3452249195 >> 172.21.15.16:0/3452249195 conn(0x7fdda80714f0 msgr2=0x7fdda809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:26:32.470 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:26:32.470 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 35 secs before thrashing 2019-12-05T06:26:34.089 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:26:34.095 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:27:04.253 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:27:04.261 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:27:07.472 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.736+0000 7fac580ac700 1 Processor -- start 2019-12-05T06:27:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.737+0000 7fac580ac700 1 -- start start 2019-12-05T06:27:07.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.738+0000 7fac580ac700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac5008d2c0 0x7fac5008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.738+0000 7fac580ac700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fac5008fef0 0x7fac5008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.738+0000 7fac580ac700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac500958e0 con 0x7fac5008c6d0 2019-12-05T06:27:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.739+0000 7fac580ac700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5008df90 con 0x7fac5008d2c0 2019-12-05T06:27:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.739+0000 7fac580ac700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac500a3f70 con 0x7fac5008fef0 2019-12-05T06:27:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.739+0000 7fac55e48700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac5008d2c0 0x7fac5008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.739+0000 7fac55e48700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac5008d2c0 0x7fac5008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53504/0 (socket says 172.21.15.16:53504) 2019-12-05T06:27:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.739+0000 7fac55e48700 1 -- 172.21.15.16:0/1301573241 learned_addr learned my addr 172.21.15.16:0/1301573241 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:07.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.739+0000 7fac56e4a700 1 --2- 172.21.15.16:0/1301573241 >> v2:172.21.15.76:3300/0 conn(0x7fac5008fef0 0x7fac5008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.739+0000 7fac55e48700 1 -- 172.21.15.16:0/1301573241 >> v2:172.21.15.76:3300/0 conn(0x7fac5008fef0 msgr2=0x7fac5008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.739+0000 7fac55e48700 1 --2- 172.21.15.16:0/1301573241 >> v2:172.21.15.76:3300/0 conn(0x7fac5008fef0 0x7fac5008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.739+0000 7fac55e48700 1 -- 172.21.15.16:0/1301573241 >> v1:172.21.15.76:6789/0 conn(0x7fac5008c6d0 legacy=0x7fac5008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:07.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.739+0000 7fac55e48700 1 -- 172.21.15.16:0/1301573241 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5008e620 con 0x7fac5008d2c0 2019-12-05T06:27:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.740+0000 7fac55e48700 1 --2- 172.21.15.16:0/1301573241 >> v2:172.21.15.16:3300/0 conn(0x7fac5008d2c0 0x7fac5008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac4c00f6c0 tx=0x7fac4c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.740+0000 7fac55e48700 1 --2- 172.21.15.16:0/1301573241 >> v2:172.21.15.16:3300/0 conn(0x7fac5008d2c0 0x7fac5008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac4c00f6c0 tx=0x7fac4c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.740+0000 7fac55e48700 1 --2- 172.21.15.16:0/1301573241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5008d2c0 0x7fac5008ef20 secure :-1 s=READY pgs=768 cs=0 l=1 rx=0x7fac4c00f6c0 tx=0x7fac4c00fa60).ready entity=mon.0 client_cookie=5526ca7b76d51824 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:07.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.740+0000 7fac55e48700 1 --2- 172.21.15.16:0/1301573241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5008d2c0 0x7fac5008ef20 secure :-1 s=THROTTLE_DONE pgs=768 cs=0 l=1 rx=0x7fac4c00f6c0 tx=0x7fac4c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.740+0000 7fac55647700 1 -- 172.21.15.16:0/1301573241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac4c000960 con 0x7fac5008d2c0 2019-12-05T06:27:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.740+0000 7fac55e48700 1 --2- 172.21.15.16:0/1301573241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5008d2c0 0x7fac5008ef20 secure :-1 s=THROTTLE_DONE pgs=768 cs=0 l=1 rx=0x7fac4c00f6c0 tx=0x7fac4c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.740+0000 7fac55e48700 1 --2- 172.21.15.16:0/1301573241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5008d2c0 0x7fac5008ef20 secure :-1 s=THROTTLE_DONE pgs=768 cs=0 l=1 rx=0x7fac4c00f6c0 tx=0x7fac4c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.740+0000 7fac55647700 1 -- 172.21.15.16:0/1301573241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c000c90 con 0x7fac5008d2c0 2019-12-05T06:27:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.740+0000 7fac55647700 1 -- 172.21.15.16:0/1301573241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac4c0013a0 con 0x7fac5008d2c0 2019-12-05T06:27:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.741+0000 7fac580ac700 1 -- 172.21.15.16:0/1301573241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5008d2c0 msgr2=0x7fac5008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.741+0000 7fac580ac700 1 --2- 172.21.15.16:0/1301573241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5008d2c0 0x7fac5008ef20 secure :-1 s=READY pgs=768 cs=0 l=1 rx=0x7fac4c00f6c0 tx=0x7fac4c00fa60).stop 2019-12-05T06:27:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.743+0000 7fac580ac700 1 -- 172.21.15.16:0/1301573241 shutdown_connections 2019-12-05T06:27:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.743+0000 7fac580ac700 1 --2- 172.21.15.16:0/1301573241 >> v2:172.21.15.76:3300/0 conn(0x7fac5008fef0 0x7fac5008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.743+0000 7fac580ac700 1 --2- 172.21.15.16:0/1301573241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac5008d2c0 0x7fac5008ef20 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.743+0000 7fac580ac700 1 -- 172.21.15.16:0/1301573241 shutdown_connections 2019-12-05T06:27:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.743+0000 7fac580ac700 1 -- 172.21.15.16:0/1301573241 wait complete. 2019-12-05T06:27:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.743+0000 7fac580ac700 1 -- 172.21.15.16:0/1301573241 >> 172.21.15.16:0/1301573241 conn(0x7fac50061940 msgr2=0x7fac50096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.747+0000 7fac580ac700 1 Processor -- start 2019-12-05T06:27:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.747+0000 7fac580ac700 1 -- start start 2019-12-05T06:27:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac580ac700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac501284a0 0x7fac5014d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:07.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac580ac700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fac5009b910 0x7fac50157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac580ac700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac5009bd50 con 0x7fac5014d410 2019-12-05T06:27:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac580ac700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5006e210 con 0x7fac501284a0 2019-12-05T06:27:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac580ac700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac5009a180 con 0x7fac5009b910 2019-12-05T06:27:07.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac55e48700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac501284a0 0x7fac5014d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:07.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac55e48700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac501284a0 0x7fac5014d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53510/0 (socket says 172.21.15.16:53510) 2019-12-05T06:27:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac55e48700 1 -- 172.21.15.16:0/1224463854 learned_addr learned my addr 172.21.15.16:0/1224463854 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac56649700 1 --1- 172.21.15.16:0/1224463854 >> v1:172.21.15.16:6789/0 conn(0x7fac5014d410 0x7fac5009b110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38676/0 (socket says 172.21.15.16:38676) 2019-12-05T06:27:07.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac56e4a700 1 --2- 172.21.15.16:0/1224463854 >> v2:172.21.15.76:3300/0 conn(0x7fac5009b910 0x7fac50157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac55e48700 1 -- 172.21.15.16:0/1224463854 >> v2:172.21.15.76:3300/0 conn(0x7fac5009b910 msgr2=0x7fac50157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> v2:172.21.15.76:3300/0 conn(0x7fac5009b910 0x7fac50157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.749+0000 7fac55e48700 1 -- 172.21.15.16:0/1224463854 >> v1:172.21.15.16:6789/0 conn(0x7fac5014d410 legacy=0x7fac5009b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:07.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac55e48700 1 -- 172.21.15.16:0/1224463854 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac5009a810 con 0x7fac501284a0 2019-12-05T06:27:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> v2:172.21.15.16:3300/0 conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> v2:172.21.15.16:3300/0 conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=READY pgs=770 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:07.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=THROTTLE_DONE pgs=770 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac42ffd700 1 -- 172.21.15.16:0/1224463854 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac4c000940 con 0x7fac501284a0 2019-12-05T06:27:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac580ac700 1 -- 172.21.15.16:0/1224463854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac5009a810 con 0x7fac501284a0 2019-12-05T06:27:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=THROTTLE_DONE pgs=770 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac42ffd700 1 -- 172.21.15.16:0/1224463854 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac4c000940 con 0x7fac501284a0 2019-12-05T06:27:07.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=THROTTLE_DONE pgs=770 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac580ac700 1 -- 172.21.15.16:0/1224463854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac5009bd50 con 0x7fac501284a0 2019-12-05T06:27:07.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.750+0000 7fac42ffd700 1 -- 172.21.15.16:0/1224463854 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac4c001140 con 0x7fac501284a0 2019-12-05T06:27:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.751+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=THROTTLE_DONE pgs=770 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.753+0000 7fac42ffd700 1 -- 172.21.15.16:0/1224463854 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac4c000b30 con 0x7fac501284a0 2019-12-05T06:27:07.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.753+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=THROTTLE_DONE pgs=770 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.753+0000 7fac42ffd700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac38023630 0x7fac38025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.753+0000 7fac42ffd700 1 -- 172.21.15.16:0/1224463854 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7fac4c045de0 con 0x7fac501284a0 2019-12-05T06:27:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.753+0000 7fac56649700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac38023630 0x7fac38025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:07.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.754+0000 7fac56649700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac38023630 0x7fac38025ac0 crc :-1 s=READY pgs=337 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.769+0000 7fac580ac700 1 -- 172.21.15.16:0/1224463854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac5009a810 con 0x7fac501284a0 2019-12-05T06:27:07.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.777+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=THROTTLE_DONE pgs=770 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:07.777+0000 7fac42ffd700 1 -- 172.21.15.16:0/1224463854 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac4c02b040 con 0x7fac501284a0 2019-12-05T06:27:07.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.000+0000 7fac580ac700 1 -- 172.21.15.16:0/1224463854 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac5009a810 con 0x7fac501284a0 2019-12-05T06:27:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.000+0000 7fac55e48700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=THROTTLE_DONE pgs=770 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.000+0000 7fac42ffd700 1 -- 172.21.15.16:0/1224463854 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fac4c001c70 con 0x7fac501284a0 2019-12-05T06:27:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 97 2019-12-05T06:27:07.988 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:07.988 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:28.191316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:active","state_seq":27,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:07.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 -- 172.21.15.16:0/1224463854 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac38023630 msgr2=0x7fac38025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac38023630 0x7fac38025ac0 crc :-1 s=READY pgs=337 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 -- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 msgr2=0x7fac5014d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 0x7fac5014d800 secure :-1 s=READY pgs=770 cs=0 l=1 rx=0x7fac4c0103c0 tx=0x7fac4c010760).stop 2019-12-05T06:27:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 -- 172.21.15.16:0/1224463854 shutdown_connections 2019-12-05T06:27:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 --2- 172.21.15.16:0/1224463854 >> v2:172.21.15.76:3300/0 conn(0x7fac5009b910 0x7fac50157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac38023630 0x7fac38025ac0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:07.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 --2- 172.21.15.16:0/1224463854 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac501284a0 0x7fac5014d800 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 -- 172.21.15.16:0/1224463854 shutdown_connections 2019-12-05T06:27:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 -- 172.21.15.16:0/1224463854 wait complete. 2019-12-05T06:27:07.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.003+0000 7fac580ac700 1 -- 172.21.15.16:0/1224463854 >> 172.21.15.16:0/1224463854 conn(0x7fac50082220 msgr2=0x7fac5006d380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:08.000 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:27:08.001 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:27:08.001 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:27:08.011 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:27:08.011 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.190+0000 7f3c05c6f700 1 Processor -- start 2019-12-05T06:27:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.190+0000 7f3c05c6f700 1 -- start start 2019-12-05T06:27:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.191+0000 7f3c05c6f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 0x7f3c0008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:08.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.191+0000 7f3c05c6f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c000958e0 con 0x7f3c0008f720 2019-12-05T06:27:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.191+0000 7f3c05c6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c000629e0 con 0x7f3c0008fef0 2019-12-05T06:27:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.191+0000 7f3c05c6f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c000a3f70 con 0x7f3c0008c6d0 2019-12-05T06:27:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.192+0000 7f3c04a0d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3c0008f720 0x7f3c001533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38692/0 (socket says 172.21.15.16:38692) 2019-12-05T06:27:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.192+0000 7f3c04a0d700 1 -- 172.21.15.16:0/2266687805 learned_addr learned my addr 172.21.15.16:0/2266687805 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.192+0000 7f3bfffff700 1 --2- 172.21.15.16:0/2266687805 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 0x7f3c0008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.192+0000 7f3bfeffd700 1 -- 172.21.15.16:0/2266687805 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21682240 0 0) 0x7f3bf8001150 con 0x7f3c0008f720 2019-12-05T06:27:08.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.192+0000 7f3bfeffd700 1 -- 172.21.15.16:0/2266687805 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bec002e80 con 0x7f3c0008f720 2019-12-05T06:27:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.192+0000 7f3bfeffd700 1 -- 172.21.15.16:0/2266687805 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3825701608 0 0) 0x7f3be8001080 con 0x7f3c0008fef0 2019-12-05T06:27:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.192+0000 7f3bfeffd700 1 -- 172.21.15.16:0/2266687805 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bec0030b0 con 0x7f3c0008fef0 2019-12-05T06:27:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfffff700 1 -- 172.21.15.16:0/2266687805 >> v1:172.21.15.76:6789/0 conn(0x7f3c0008fef0 legacy=0x7f3c0008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfffff700 1 -- 172.21.15.16:0/2266687805 >> v1:172.21.15.16:6789/0 conn(0x7f3c0008f720 legacy=0x7f3c001533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfffff700 1 -- 172.21.15.16:0/2266687805 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0008fbc0 con 0x7f3c0008c6d0 2019-12-05T06:27:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfffff700 1 --2- 172.21.15.16:0/2266687805 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 0x7f3c0008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3bf000b2d0 tx=0x7f3bf000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfffff700 1 --2- 172.21.15.16:0/2266687805 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 0x7f3c0008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3bf000b2d0 tx=0x7f3bf000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfffff700 1 --2- 172.21.15.16:0/2266687805 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 0x7f3c0008cac0 secure :-1 s=READY pgs=192 cs=0 l=1 rx=0x7f3bf000b2d0 tx=0x7f3bf000bae0).ready entity=mon.2 client_cookie=d10b97aadbf9bf59 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:08.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfffff700 1 --2- 172.21.15.16:0/2266687805 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 0x7f3c0008cac0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3bf000b2d0 tx=0x7f3bf000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfeffd700 1 -- 172.21.15.16:0/2266687805 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bf0000ce0 con 0x7f3c0008c6d0 2019-12-05T06:27:08.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfffff700 1 --2- 172.21.15.16:0/2266687805 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 0x7f3c0008cac0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3bf000b2d0 tx=0x7f3bf000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfffff700 1 --2- 172.21.15.16:0/2266687805 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 0x7f3c0008cac0 secure :-1 s=THROTTLE_DONE pgs=192 cs=0 l=1 rx=0x7f3bf000b2d0 tx=0x7f3bf000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfeffd700 1 -- 172.21.15.16:0/2266687805 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf0000ce0 con 0x7f3c0008c6d0 2019-12-05T06:27:08.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3bfeffd700 1 -- 172.21.15.16:0/2266687805 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bf000a060 con 0x7f3c0008c6d0 2019-12-05T06:27:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3c05c6f700 1 -- 172.21.15.16:0/2266687805 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 msgr2=0x7f3c0008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.193+0000 7f3c05c6f700 1 --2- 172.21.15.16:0/2266687805 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 0x7f3c0008cac0 secure :-1 s=READY pgs=192 cs=0 l=1 rx=0x7f3bf000b2d0 tx=0x7f3bf000bae0).stop 2019-12-05T06:27:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.194+0000 7f3c05c6f700 1 -- 172.21.15.16:0/2266687805 shutdown_connections 2019-12-05T06:27:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.194+0000 7f3c05c6f700 1 --2- 172.21.15.16:0/2266687805 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0008c6d0 0x7f3c0008cac0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.194+0000 7f3c05c6f700 1 -- 172.21.15.16:0/2266687805 shutdown_connections 2019-12-05T06:27:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.194+0000 7f3c05c6f700 1 -- 172.21.15.16:0/2266687805 wait complete. 2019-12-05T06:27:08.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.194+0000 7f3c05c6f700 1 -- 172.21.15.16:0/2266687805 >> 172.21.15.16:0/2266687805 conn(0x7f3c00061940 msgr2=0x7f3c00096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:08.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.199+0000 7f3c05c6f700 1 Processor -- start 2019-12-05T06:27:08.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.199+0000 7f3c05c6f700 1 -- start start 2019-12-05T06:27:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.201+0000 7f3c05c6f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3c0014d680 0x7f3c0014da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.201+0000 7f3c05c6f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.201+0000 7f3c05c6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c0012c220 con 0x7f3c0009dc10 2019-12-05T06:27:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.201+0000 7f3c05c6f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c0012ad50 con 0x7f3c0014d680 2019-12-05T06:27:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.201+0000 7f3c05c6f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c00099bb0 con 0x7f3c0009d3a0 2019-12-05T06:27:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.201+0000 7f3bfffff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3c0014d680 0x7f3c0014da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:08.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.201+0000 7f3c04a0d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.201+0000 7f3bfffff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3c0014d680 0x7f3c0014da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:39616/0 (socket says 172.21.15.16:39616) 2019-12-05T06:27:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.201+0000 7f3bff7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3c0009dc10 0x7f3c00098180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:59998/0 (socket says 172.21.15.16:59998) 2019-12-05T06:27:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.201+0000 7f3bfffff700 1 -- 172.21.15.16:0/3653657820 learned_addr learned my addr 172.21.15.16:0/3653657820 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.202+0000 7f3c04a0d700 1 -- 172.21.15.16:0/3653657820 >> v2:172.21.15.76:3300/0 conn(0x7f3c0014d680 msgr2=0x7f3c0014da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.202+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> v2:172.21.15.76:3300/0 conn(0x7f3c0014d680 0x7f3c0014da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.202+0000 7f3c04a0d700 1 -- 172.21.15.16:0/3653657820 >> v1:172.21.15.76:6789/0 conn(0x7f3c0009dc10 legacy=0x7f3c00098180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:08.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.202+0000 7f3c04a0d700 1 -- 172.21.15.16:0/3653657820 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0012b3e0 con 0x7f3c0009d3a0 2019-12-05T06:27:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.202+0000 7f3bfffff700 1 --2- 172.21.15.16:0/3653657820 >> v2:172.21.15.76:3300/0 conn(0x7f3c0014d680 0x7f3c0014da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:27:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.202+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.202+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.202+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=READY pgs=193 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.203+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.203+0000 7f3bfcff9700 1 -- 172.21.15.16:0/3653657820 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bf8001200 con 0x7f3c0009d3a0 2019-12-05T06:27:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.203+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.203+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.203+0000 7f3bfcff9700 1 -- 172.21.15.16:0/3653657820 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bf8004240 con 0x7f3c0009d3a0 2019-12-05T06:27:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.203+0000 7f3c05c6f700 1 -- 172.21.15.16:0/3653657820 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c0012b3e0 con 0x7f3c0009d3a0 2019-12-05T06:27:08.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.203+0000 7f3bfcff9700 1 -- 172.21.15.16:0/3653657820 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bf80048d0 con 0x7f3c0009d3a0 2019-12-05T06:27:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.203+0000 7f3c05c6f700 1 -- 172.21.15.16:0/3653657820 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c0012b3e0 con 0x7f3c0009d3a0 2019-12-05T06:27:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.204+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.205+0000 7f3bfcff9700 1 -- 172.21.15.16:0/3653657820 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3bf8004240 con 0x7f3c0009d3a0 2019-12-05T06:27:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.205+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.205+0000 7f3bfcff9700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3be4023630 0x7f3be4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:08.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.205+0000 7f3bfcff9700 1 -- 172.21.15.16:0/3653657820 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f3bf8045d30 con 0x7f3c0009d3a0 2019-12-05T06:27:08.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.205+0000 7f3bfffff700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3be4023630 0x7f3be4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:08.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.206+0000 7f3bfffff700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3be4023630 0x7f3be4025ac0 crc :-1 s=READY pgs=338 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:08.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.221+0000 7f3c05c6f700 1 -- 172.21.15.16:0/3653657820 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c0012b3e0 con 0x7f3c0009d3a0 2019-12-05T06:27:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.230+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.230+0000 7f3bfcff9700 1 -- 172.21.15.16:0/3653657820 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3bf8000ae0 con 0x7f3c0009d3a0 2019-12-05T06:27:08.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.498+0000 7f3c05c6f700 1 -- 172.21.15.16:0/3653657820 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c0012b3e0 con 0x7f3c0009d3a0 2019-12-05T06:27:08.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.499+0000 7f3c04a0d700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=THROTTLE_DONE pgs=193 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:08.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.499+0000 7f3bfcff9700 1 -- 172.21.15.16:0/3653657820 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f3bf8004240 con 0x7f3c0009d3a0 2019-12-05T06:27:08.487 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 97 2019-12-05T06:27:08.488 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:08.488 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:28.191316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:active","state_seq":27,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:08.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.503+0000 7f3c05c6f700 1 -- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3be4023630 msgr2=0x7f3be4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:08.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.503+0000 7f3c05c6f700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3be4023630 0x7f3be4025ac0 crc :-1 s=READY pgs=338 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:08.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.503+0000 7f3c05c6f700 1 -- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 msgr2=0x7f3c0012bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:08.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.503+0000 7f3c05c6f700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 secure :-1 s=READY pgs=193 cs=0 l=1 rx=0x7f3bf8010370 tx=0x7f3bf8010710).stop 2019-12-05T06:27:08.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.503+0000 7f3c05c6f700 1 -- 172.21.15.16:0/3653657820 shutdown_connections 2019-12-05T06:27:08.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.503+0000 7f3c05c6f700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3be4023630 0x7f3be4025ac0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:08.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.503+0000 7f3c05c6f700 1 --2- 172.21.15.16:0/3653657820 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c0009d3a0 0x7f3c0012bce0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:08.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.503+0000 7f3c05c6f700 1 --2- 172.21.15.16:0/3653657820 >> v2:172.21.15.76:3300/0 conn(0x7f3c0014d680 0x7f3c0014da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:08.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.504+0000 7f3c05c6f700 1 -- 172.21.15.16:0/3653657820 shutdown_connections 2019-12-05T06:27:08.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.504+0000 7f3c05c6f700 1 -- 172.21.15.16:0/3653657820 wait complete. 2019-12-05T06:27:08.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:08.504+0000 7f3c05c6f700 1 -- 172.21.15.16:0/3653657820 >> 172.21.15.16:0/3653657820 conn(0x7f3c0006d260 msgr2=0x7f3c0009aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:08.500 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:27:10.502 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:10.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.763+0000 7fa56b37a700 1 Processor -- start 2019-12-05T06:27:10.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.764+0000 7fa56b37a700 1 -- start start 2019-12-05T06:27:10.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.766+0000 7fa56b37a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa56408c6d0 0x7fa56408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:10.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.766+0000 7fa56b37a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa56408f720 0x7fa56408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:10.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.766+0000 7fa56b37a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa56408fb70 con 0x7fa56408fef0 2019-12-05T06:27:10.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.766+0000 7fa56b37a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa56408df90 con 0x7fa56408c6d0 2019-12-05T06:27:10.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.766+0000 7fa56b37a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5640a3f70 con 0x7fa56408f720 2019-12-05T06:27:10.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.766+0000 7fa569917700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa56408c6d0 0x7fa56408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:10.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.766+0000 7fa569917700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa56408c6d0 0x7fa56408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53540/0 (socket says 172.21.15.16:53540) 2019-12-05T06:27:10.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.766+0000 7fa569917700 1 -- 172.21.15.16:0/3414717731 learned_addr learned my addr 172.21.15.16:0/3414717731 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:10.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.766+0000 7fa56a118700 1 --2- 172.21.15.16:0/3414717731 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa56408f720 0x7fa56408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:10.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.767+0000 7fa569917700 1 -- 172.21.15.16:0/3414717731 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa56408f720 msgr2=0x7fa56408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:10.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.767+0000 7fa569917700 1 --2- 172.21.15.16:0/3414717731 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa56408f720 0x7fa56408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:10.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.767+0000 7fa569917700 1 -- 172.21.15.16:0/3414717731 >> v1:172.21.15.76:6789/0 conn(0x7fa56408fef0 legacy=0x7fa56408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:10.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.767+0000 7fa569917700 1 -- 172.21.15.16:0/3414717731 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa56408d540 con 0x7fa56408c6d0 2019-12-05T06:27:10.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.767+0000 7fa569917700 1 --2- 172.21.15.16:0/3414717731 >> v2:172.21.15.16:3300/0 conn(0x7fa56408c6d0 0x7fa56408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa55c00fa40 tx=0x7fa55c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.767+0000 7fa569917700 1 --2- 172.21.15.16:0/3414717731 >> v2:172.21.15.16:3300/0 conn(0x7fa56408c6d0 0x7fa56408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa55c00fa40 tx=0x7fa55c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.767+0000 7fa569917700 1 --2- 172.21.15.16:0/3414717731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa56408c6d0 0x7fa56408cac0 secure :-1 s=READY pgs=772 cs=0 l=1 rx=0x7fa55c00fa40 tx=0x7fa55c00fde0).ready entity=mon.0 client_cookie=f9343e08c4dc5b5c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:10.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.768+0000 7fa569917700 1 --2- 172.21.15.16:0/3414717731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa56408c6d0 0x7fa56408cac0 secure :-1 s=THROTTLE_DONE pgs=772 cs=0 l=1 rx=0x7fa55c00fa40 tx=0x7fa55c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.768+0000 7fa569917700 1 --2- 172.21.15.16:0/3414717731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa56408c6d0 0x7fa56408cac0 secure :-1 s=THROTTLE_DONE pgs=772 cs=0 l=1 rx=0x7fa55c00fa40 tx=0x7fa55c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.768+0000 7fa569917700 1 --2- 172.21.15.16:0/3414717731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa56408c6d0 0x7fa56408cac0 secure :-1 s=THROTTLE_DONE pgs=772 cs=0 l=1 rx=0x7fa55c00fa40 tx=0x7fa55c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.768+0000 7fa568915700 1 -- 172.21.15.16:0/3414717731 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa55c000ce0 con 0x7fa56408c6d0 2019-12-05T06:27:10.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.768+0000 7fa568915700 1 -- 172.21.15.16:0/3414717731 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55c001010 con 0x7fa56408c6d0 2019-12-05T06:27:10.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.768+0000 7fa568915700 1 -- 172.21.15.16:0/3414717731 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa55c001740 con 0x7fa56408c6d0 2019-12-05T06:27:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.768+0000 7fa56b37a700 1 -- 172.21.15.16:0/3414717731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa56408c6d0 msgr2=0x7fa56408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.768+0000 7fa56b37a700 1 --2- 172.21.15.16:0/3414717731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa56408c6d0 0x7fa56408cac0 secure :-1 s=READY pgs=772 cs=0 l=1 rx=0x7fa55c00fa40 tx=0x7fa55c00fde0).stop 2019-12-05T06:27:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.770+0000 7fa56b37a700 1 -- 172.21.15.16:0/3414717731 shutdown_connections 2019-12-05T06:27:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.770+0000 7fa56b37a700 1 --2- 172.21.15.16:0/3414717731 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa56408f720 0x7fa56408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.770+0000 7fa56b37a700 1 --2- 172.21.15.16:0/3414717731 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa56408c6d0 0x7fa56408cac0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:10.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.770+0000 7fa56b37a700 1 -- 172.21.15.16:0/3414717731 shutdown_connections 2019-12-05T06:27:10.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.770+0000 7fa56b37a700 1 -- 172.21.15.16:0/3414717731 wait complete. 2019-12-05T06:27:10.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.770+0000 7fa56b37a700 1 -- 172.21.15.16:0/3414717731 >> 172.21.15.16:0/3414717731 conn(0x7fa564061940 msgr2=0x7fa564096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.775+0000 7fa56b37a700 1 Processor -- start 2019-12-05T06:27:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.775+0000 7fa56b37a700 1 -- start start 2019-12-05T06:27:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa56b37a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa564096e90 0x7fa56409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa56b37a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa564084f30 0x7fa56409b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa56b37a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa564097280 0x7fa564097700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:10.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa56b37a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa564099a20 con 0x7fa564097280 2019-12-05T06:27:10.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa56b37a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa564099cf0 con 0x7fa564084f30 2019-12-05T06:27:10.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa56b37a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa56409aae0 con 0x7fa564096e90 2019-12-05T06:27:10.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa56a118700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa564097280 0x7fa564097700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:10.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa56a118700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa564097280 0x7fa564097700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53548/0 (socket says 172.21.15.16:53548) 2019-12-05T06:27:10.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa569917700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa564096e90 0x7fa56409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:10.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa56a118700 1 -- 172.21.15.16:0/103316612 learned_addr learned my addr 172.21.15.16:0/103316612 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.777+0000 7fa569116700 1 --2- 172.21.15.16:0/103316612 >> v2:172.21.15.76:3300/0 conn(0x7fa564084f30 0x7fa56409b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.778+0000 7fa56a118700 1 -- 172.21.15.16:0/103316612 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa564096e90 msgr2=0x7fa56409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.778+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa564096e90 0x7fa56409b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.778+0000 7fa56a118700 1 -- 172.21.15.16:0/103316612 >> v2:172.21.15.76:3300/0 conn(0x7fa564084f30 msgr2=0x7fa56409b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.778+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> v2:172.21.15.76:3300/0 conn(0x7fa564084f30 0x7fa56409b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:10.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.778+0000 7fa56a118700 1 -- 172.21.15.16:0/103316612 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa564084b90 con 0x7fa564097280 2019-12-05T06:27:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.778+0000 7fa569917700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa564096e90 0x7fa56409b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:27:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.778+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> v2:172.21.15.16:3300/0 conn(0x7fa564097280 0x7fa564097700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.778+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> v2:172.21.15.16:3300/0 conn(0x7fa564097280 0x7fa564097700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.778+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 0x7fa564097700 secure :-1 s=READY pgs=773 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.779+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 0x7fa564097700 secure :-1 s=THROTTLE_DONE pgs=773 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.779+0000 7fa5627fc700 1 -- 172.21.15.16:0/103316612 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa554000b10 con 0x7fa564097280 2019-12-05T06:27:10.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.779+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 0x7fa564097700 secure :-1 s=THROTTLE_DONE pgs=773 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.779+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 0x7fa564097700 secure :-1 s=THROTTLE_DONE pgs=773 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.779+0000 7fa5627fc700 1 -- 172.21.15.16:0/103316612 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa55400ef40 con 0x7fa564097280 2019-12-05T06:27:10.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.779+0000 7fa56b37a700 1 -- 172.21.15.16:0/103316612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa564084b90 con 0x7fa564097280 2019-12-05T06:27:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.779+0000 7fa5627fc700 1 -- 172.21.15.16:0/103316612 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa55400f5d0 con 0x7fa564097280 2019-12-05T06:27:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.779+0000 7fa56b37a700 1 -- 172.21.15.16:0/103316612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa564084b90 con 0x7fa564097280 2019-12-05T06:27:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.780+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 0x7fa564097700 secure :-1 s=THROTTLE_DONE pgs=773 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.781+0000 7fa5627fc700 1 -- 172.21.15.16:0/103316612 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa55400eb20 con 0x7fa564097280 2019-12-05T06:27:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.781+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 0x7fa564097700 secure :-1 s=THROTTLE_DONE pgs=773 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.781+0000 7fa5627fc700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa544023630 0x7fa544025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:10.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.781+0000 7fa5627fc700 1 -- 172.21.15.16:0/103316612 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7fa554045d50 con 0x7fa564097280 2019-12-05T06:27:10.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.781+0000 7fa569917700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa544023630 0x7fa544025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:10.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.782+0000 7fa569917700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa544023630 0x7fa544025ac0 crc :-1 s=READY pgs=339 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:10.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.795+0000 7fa56b37a700 1 -- 172.21.15.16:0/103316612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5640958e0 con 0x7fa564097280 2019-12-05T06:27:10.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.804+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 0x7fa564097700 secure :-1 s=THROTTLE_DONE pgs=773 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:10.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:10.804+0000 7fa5627fc700 1 -- 172.21.15.16:0/103316612 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa554000ad0 con 0x7fa564097280 2019-12-05T06:27:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.057+0000 7fa56b37a700 1 -- 172.21.15.16:0/103316612 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5640958e0 con 0x7fa564097280 2019-12-05T06:27:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.057+0000 7fa56a118700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 0x7fa564097700 secure :-1 s=THROTTLE_DONE pgs=773 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.057+0000 7fa5627fc700 1 -- 172.21.15.16:0/103316612 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fa55400ea80 con 0x7fa564097280 2019-12-05T06:27:11.045 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:11.045 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:28.191316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:active","state_seq":27,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 97 2019-12-05T06:27:11.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 -- 172.21.15.16:0/103316612 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa544023630 msgr2=0x7fa544025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:11.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa544023630 0x7fa544025ac0 crc :-1 s=READY pgs=339 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:11.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 -- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 msgr2=0x7fa564097700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:11.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 0x7fa564097700 secure :-1 s=READY pgs=773 cs=0 l=1 rx=0x7fa554010450 tx=0x7fa554010300).stop 2019-12-05T06:27:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 -- 172.21.15.16:0/103316612 shutdown_connections 2019-12-05T06:27:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa544023630 0x7fa544025ac0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa564097280 0x7fa564097700 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 --2- 172.21.15.16:0/103316612 >> v2:172.21.15.76:3300/0 conn(0x7fa564084f30 0x7fa56409b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 --2- 172.21.15.16:0/103316612 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa564096e90 0x7fa56409b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 -- 172.21.15.16:0/103316612 shutdown_connections 2019-12-05T06:27:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 -- 172.21.15.16:0/103316612 wait complete. 2019-12-05T06:27:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:11.060+0000 7fa56b37a700 1 -- 172.21.15.16:0/103316612 >> 172.21.15.16:0/103316612 conn(0x7fa5640714f0 msgr2=0x7fa56409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:11.058 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:27:13.059 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:13.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.330+0000 7f92065ea700 1 Processor -- start 2019-12-05T06:27:13.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.330+0000 7f92065ea700 1 -- start start 2019-12-05T06:27:13.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.331+0000 7f92065ea700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f920008c6d0 0x7f920008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:13.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.331+0000 7f92065ea700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f920008fef0 0x7f920008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:13.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.331+0000 7f92065ea700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92000958e0 con 0x7f920008ef20 2019-12-05T06:27:13.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.331+0000 7f92065ea700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92000a3f70 con 0x7f920008fef0 2019-12-05T06:27:13.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.331+0000 7f92065ea700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920008d830 con 0x7f920008c6d0 2019-12-05T06:27:13.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.332+0000 7f91fffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f920008fef0 0x7f920008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:13.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.332+0000 7f9205388700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f920008ef20 0x7f920008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38726/0 (socket says 172.21.15.16:38726) 2019-12-05T06:27:13.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.332+0000 7f9205388700 1 -- 172.21.15.16:0/1361324944 learned_addr learned my addr 172.21.15.16:0/1361324944 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:13.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.332+0000 7f9204b87700 1 --2- 172.21.15.16:0/1361324944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f920008c6d0 0x7f920008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:13.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.332+0000 7f91fffff700 1 -- 172.21.15.16:0/1361324944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f920008c6d0 msgr2=0x7f920008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:13.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.332+0000 7f91fffff700 1 --2- 172.21.15.16:0/1361324944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f920008c6d0 0x7f920008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:13.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.332+0000 7f91fffff700 1 -- 172.21.15.16:0/1361324944 >> v1:172.21.15.16:6789/0 conn(0x7f920008ef20 legacy=0x7f920008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:13.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.332+0000 7f91fffff700 1 -- 172.21.15.16:0/1361324944 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920008d540 con 0x7f920008fef0 2019-12-05T06:27:13.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.332+0000 7f91ff7fe700 1 -- 172.21.15.16:0/1361324944 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3463830169 0 0) 0x7f91f80014d0 con 0x7f920008ef20 2019-12-05T06:27:13.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.333+0000 7f9204b87700 1 --2- 172.21.15.16:0/1361324944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f920008c6d0 0x7f920008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:27:13.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.333+0000 7f91fffff700 1 --2- 172.21.15.16:0/1361324944 >> v2:172.21.15.16:3300/0 conn(0x7f920008fef0 0x7f920008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f91f000af50 tx=0x7f91f000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.333+0000 7f91fffff700 1 --2- 172.21.15.16:0/1361324944 >> v2:172.21.15.16:3300/0 conn(0x7f920008fef0 0x7f920008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f91f000af50 tx=0x7f91f000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.333+0000 7f91fffff700 1 --2- 172.21.15.16:0/1361324944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920008fef0 0x7f920008d000 secure :-1 s=READY pgs=775 cs=0 l=1 rx=0x7f91f000af50 tx=0x7f91f000b760).ready entity=mon.0 client_cookie=4731c2e30b1f583 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:13.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.333+0000 7f91fffff700 1 --2- 172.21.15.16:0/1361324944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920008fef0 0x7f920008d000 secure :-1 s=THROTTLE_DONE pgs=775 cs=0 l=1 rx=0x7f91f000af50 tx=0x7f91f000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.333+0000 7f91ff7fe700 1 -- 172.21.15.16:0/1361324944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91f0007210 con 0x7f920008fef0 2019-12-05T06:27:13.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.333+0000 7f91fffff700 1 --2- 172.21.15.16:0/1361324944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920008fef0 0x7f920008d000 secure :-1 s=THROTTLE_DONE pgs=775 cs=0 l=1 rx=0x7f91f000af50 tx=0x7f91f000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.333+0000 7f91fffff700 1 --2- 172.21.15.16:0/1361324944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920008fef0 0x7f920008d000 secure :-1 s=THROTTLE_DONE pgs=775 cs=0 l=1 rx=0x7f91f000af50 tx=0x7f91f000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.333+0000 7f91ff7fe700 1 -- 172.21.15.16:0/1361324944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f00099b0 con 0x7f920008fef0 2019-12-05T06:27:13.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.334+0000 7f91ff7fe700 1 -- 172.21.15.16:0/1361324944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91f000a080 con 0x7f920008fef0 2019-12-05T06:27:13.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.334+0000 7f92065ea700 1 -- 172.21.15.16:0/1361324944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920008fef0 msgr2=0x7f920008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:13.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.334+0000 7f92065ea700 1 --2- 172.21.15.16:0/1361324944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920008fef0 0x7f920008d000 secure :-1 s=READY pgs=775 cs=0 l=1 rx=0x7f91f000af50 tx=0x7f91f000b760).stop 2019-12-05T06:27:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.335+0000 7f92065ea700 1 -- 172.21.15.16:0/1361324944 shutdown_connections 2019-12-05T06:27:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.335+0000 7f92065ea700 1 --2- 172.21.15.16:0/1361324944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f920008fef0 0x7f920008d000 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.335+0000 7f92065ea700 1 --2- 172.21.15.16:0/1361324944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f920008c6d0 0x7f920008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.335+0000 7f92065ea700 1 -- 172.21.15.16:0/1361324944 shutdown_connections 2019-12-05T06:27:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.335+0000 7f92065ea700 1 -- 172.21.15.16:0/1361324944 wait complete. 2019-12-05T06:27:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.335+0000 7f92065ea700 1 -- 172.21.15.16:0/1361324944 >> 172.21.15.16:0/1361324944 conn(0x7f9200061940 msgr2=0x7f9200096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.340+0000 7f92065ea700 1 Processor -- start 2019-12-05T06:27:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.340+0000 7f92065ea700 1 -- start start 2019-12-05T06:27:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.342+0000 7f92065ea700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f920006df10 0x7f9200098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.342+0000 7f92065ea700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.342+0000 7f92065ea700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920006e300 con 0x7f9200098d80 2019-12-05T06:27:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.342+0000 7f92065ea700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9200097370 con 0x7f920006df10 2019-12-05T06:27:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.342+0000 7f92065ea700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920008ad00 con 0x7f92000992d0 2019-12-05T06:27:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.342+0000 7f9205388700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9200098d80 0x7f9200088fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38732/0 (socket says 172.21.15.16:38732) 2019-12-05T06:27:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.342+0000 7f9205388700 1 -- 172.21.15.16:0/788561028 learned_addr learned my addr 172.21.15.16:0/788561028 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.342+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.342+0000 7f9204b87700 1 --2- 172.21.15.16:0/788561028 >> v2:172.21.15.76:3300/0 conn(0x7f920006df10 0x7f9200098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.343+0000 7f91fd7fa700 1 -- 172.21.15.16:0/788561028 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134440867 0 0) 0x7f91f8000d30 con 0x7f9200098d80 2019-12-05T06:27:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.343+0000 7f91fd7fa700 1 -- 172.21.15.16:0/788561028 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e4002e80 con 0x7f9200098d80 2019-12-05T06:27:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.343+0000 7f91fffff700 1 -- 172.21.15.16:0/788561028 >> v2:172.21.15.76:3300/0 conn(0x7f920006df10 msgr2=0x7f9200098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.343+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> v2:172.21.15.76:3300/0 conn(0x7f920006df10 0x7f9200098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.343+0000 7f91fffff700 1 -- 172.21.15.16:0/788561028 >> v1:172.21.15.16:6789/0 conn(0x7f9200098d80 legacy=0x7f9200088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.343+0000 7f91fffff700 1 -- 172.21.15.16:0/788561028 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9200097a00 con 0x7f92000992d0 2019-12-05T06:27:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.343+0000 7f9204b87700 1 --2- 172.21.15.16:0/788561028 >> v2:172.21.15.76:3300/0 conn(0x7f920006df10 0x7f9200098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:27:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.343+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.344+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.344+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.344+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.344+0000 7f91fd7fa700 1 -- 172.21.15.16:0/788561028 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91f000aca0 con 0x7f92000992d0 2019-12-05T06:27:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.344+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.344+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.344+0000 7f91fd7fa700 1 -- 172.21.15.16:0/788561028 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f000aca0 con 0x7f92000992d0 2019-12-05T06:27:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.344+0000 7f91fd7fa700 1 -- 172.21.15.16:0/788561028 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91f0009960 con 0x7f92000992d0 2019-12-05T06:27:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.344+0000 7f92065ea700 1 -- 172.21.15.16:0/788561028 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9200097a00 con 0x7f92000992d0 2019-12-05T06:27:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.344+0000 7f92065ea700 1 -- 172.21.15.16:0/788561028 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9200097a00 con 0x7f92000992d0 2019-12-05T06:27:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.345+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.346+0000 7f91fd7fa700 1 -- 172.21.15.16:0/788561028 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f91f0009580 con 0x7f92000992d0 2019-12-05T06:27:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.346+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.346+0000 7f91fd7fa700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91e4023670 0x7f91e4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.346+0000 7f91fd7fa700 1 -- 172.21.15.16:0/788561028 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f91f0089d30 con 0x7f92000992d0 2019-12-05T06:27:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.346+0000 7f9204b87700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91e4023670 0x7f91e4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:13.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.347+0000 7f9204b87700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91e4023670 0x7f91e4025ae0 crc :-1 s=READY pgs=340 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.360+0000 7f92065ea700 1 -- 172.21.15.16:0/788561028 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9200097a00 con 0x7f92000992d0 2019-12-05T06:27:13.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.370+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.370+0000 7f91fd7fa700 1 -- 172.21.15.16:0/788561028 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f91f0078f60 con 0x7f92000992d0 2019-12-05T06:27:13.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.611+0000 7f92065ea700 1 -- 172.21.15.16:0/788561028 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9200097a00 con 0x7f92000992d0 2019-12-05T06:27:13.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.612+0000 7f91fffff700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=THROTTLE_DONE pgs=194 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:13.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.612+0000 7f91fd7fa700 1 -- 172.21.15.16:0/788561028 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f91f0078f60 con 0x7f92000992d0 2019-12-05T06:27:13.601 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 97 2019-12-05T06:27:13.601 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:13.602 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:28.191316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:active","state_seq":27,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:13.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.616+0000 7f92065ea700 1 -- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91e4023670 msgr2=0x7f91e4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:13.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.616+0000 7f92065ea700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91e4023670 0x7f91e4025ae0 crc :-1 s=READY pgs=340 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:13.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.616+0000 7f92065ea700 1 -- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 msgr2=0x7f9200098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:13.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.616+0000 7f92065ea700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 secure :-1 s=READY pgs=194 cs=0 l=1 rx=0x7f91f00543b0 tx=0x7f91f0054750).stop 2019-12-05T06:27:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.617+0000 7f92065ea700 1 -- 172.21.15.16:0/788561028 shutdown_connections 2019-12-05T06:27:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.617+0000 7f92065ea700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f91e4023670 0x7f91e4025ae0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.617+0000 7f92065ea700 1 --2- 172.21.15.16:0/788561028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f92000992d0 0x7f9200098840 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.617+0000 7f92065ea700 1 --2- 172.21.15.16:0/788561028 >> v2:172.21.15.76:3300/0 conn(0x7f920006df10 0x7f9200098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.617+0000 7f92065ea700 1 -- 172.21.15.16:0/788561028 shutdown_connections 2019-12-05T06:27:13.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.617+0000 7f92065ea700 1 -- 172.21.15.16:0/788561028 wait complete. 2019-12-05T06:27:13.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:13.617+0000 7f92065ea700 1 -- 172.21.15.16:0/788561028 >> 172.21.15.16:0/788561028 conn(0x7f92000714f0 msgr2=0x7f920009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:13.614 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:27:15.615 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.908+0000 7fda4afc8700 1 Processor -- start 2019-12-05T06:27:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.909+0000 7fda4afc8700 1 -- start start 2019-12-05T06:27:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.911+0000 7fda4afc8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fda4408d2c0 0x7fda4408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.911+0000 7fda4afc8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda4408fef0 0x7fda4408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.911+0000 7fda4afc8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda440958e0 con 0x7fda4408c6d0 2019-12-05T06:27:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.911+0000 7fda4afc8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda4408df90 con 0x7fda4408d2c0 2019-12-05T06:27:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.911+0000 7fda4afc8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda440a3f70 con 0x7fda4408fef0 2019-12-05T06:27:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.911+0000 7fda49565700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fda4408c6d0 0x7fda4408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38738/0 (socket says 172.21.15.16:38738) 2019-12-05T06:27:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.911+0000 7fda49565700 1 -- 172.21.15.16:0/1074612816 learned_addr learned my addr 172.21.15.16:0/1074612816 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.911+0000 7fda49d66700 1 --2- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda4408fef0 0x7fda4408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.911+0000 7fda48d64700 1 --2- 172.21.15.16:0/1074612816 >> v2:172.21.15.76:3300/0 conn(0x7fda4408d2c0 0x7fda4408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.912+0000 7fda43fff700 1 -- 172.21.15.16:0/1074612816 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1372962031 0 0) 0x7fda3c001480 con 0x7fda4408c6d0 2019-12-05T06:27:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.912+0000 7fda43fff700 1 -- 172.21.15.16:0/1074612816 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda2c002e80 con 0x7fda4408c6d0 2019-12-05T06:27:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.912+0000 7fda48d64700 1 -- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda4408fef0 msgr2=0x7fda4408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.912+0000 7fda48d64700 1 --2- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda4408fef0 0x7fda4408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:15.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.912+0000 7fda48d64700 1 -- 172.21.15.16:0/1074612816 >> v1:172.21.15.16:6789/0 conn(0x7fda4408c6d0 legacy=0x7fda4408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:15.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.912+0000 7fda48d64700 1 -- 172.21.15.16:0/1074612816 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda4408e620 con 0x7fda4408d2c0 2019-12-05T06:27:15.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.912+0000 7fda49d66700 1 --2- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda4408fef0 0x7fda4408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:27:15.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.912+0000 7fda48d64700 1 --2- 172.21.15.16:0/1074612816 >> v2:172.21.15.76:3300/0 conn(0x7fda4408d2c0 0x7fda4408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fda3800af50 tx=0x7fda3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.912+0000 7fda48d64700 1 --2- 172.21.15.16:0/1074612816 >> v2:172.21.15.76:3300/0 conn(0x7fda4408d2c0 0x7fda4408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fda3800af50 tx=0x7fda3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.912+0000 7fda48d64700 1 --2- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda4408d2c0 0x7fda4408ef20 secure :-1 s=READY pgs=563 cs=0 l=1 rx=0x7fda3800af50 tx=0x7fda3800b760).ready entity=mon.1 client_cookie=9b3a6688c15c295f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.913+0000 7fda48d64700 1 --2- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda4408d2c0 0x7fda4408ef20 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fda3800af50 tx=0x7fda3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.913+0000 7fda43fff700 1 -- 172.21.15.16:0/1074612816 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda38007210 con 0x7fda4408d2c0 2019-12-05T06:27:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.913+0000 7fda48d64700 1 --2- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda4408d2c0 0x7fda4408ef20 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fda3800af50 tx=0x7fda3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.913+0000 7fda48d64700 1 --2- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda4408d2c0 0x7fda4408ef20 secure :-1 s=THROTTLE_DONE pgs=563 cs=0 l=1 rx=0x7fda3800af50 tx=0x7fda3800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.913+0000 7fda43fff700 1 -- 172.21.15.16:0/1074612816 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda380099b0 con 0x7fda4408d2c0 2019-12-05T06:27:15.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.913+0000 7fda43fff700 1 -- 172.21.15.16:0/1074612816 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda38009ba0 con 0x7fda4408d2c0 2019-12-05T06:27:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.913+0000 7fda4afc8700 1 -- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda4408d2c0 msgr2=0x7fda4408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.913+0000 7fda4afc8700 1 --2- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda4408d2c0 0x7fda4408ef20 secure :-1 s=READY pgs=563 cs=0 l=1 rx=0x7fda3800af50 tx=0x7fda3800b760).stop 2019-12-05T06:27:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.915+0000 7fda4afc8700 1 -- 172.21.15.16:0/1074612816 shutdown_connections 2019-12-05T06:27:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.915+0000 7fda4afc8700 1 --2- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda4408d2c0 0x7fda4408ef20 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.915+0000 7fda4afc8700 1 --2- 172.21.15.16:0/1074612816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda4408fef0 0x7fda4408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:15.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.915+0000 7fda4afc8700 1 -- 172.21.15.16:0/1074612816 shutdown_connections 2019-12-05T06:27:15.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.915+0000 7fda4afc8700 1 -- 172.21.15.16:0/1074612816 wait complete. 2019-12-05T06:27:15.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.915+0000 7fda4afc8700 1 -- 172.21.15.16:0/1074612816 >> 172.21.15.16:0/1074612816 conn(0x7fda44061940 msgr2=0x7fda44096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:15.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.921+0000 7fda4afc8700 1 Processor -- start 2019-12-05T06:27:15.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.921+0000 7fda4afc8700 1 -- start start 2019-12-05T06:27:15.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.923+0000 7fda4afc8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda4412b640 0x7fda4412ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:15.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.923+0000 7fda4afc8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda44131bf0 con 0x7fda441302f0 2019-12-05T06:27:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.923+0000 7fda4afc8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda441316f0 con 0x7fda44153b50 2019-12-05T06:27:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.923+0000 7fda4afc8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda441296e0 con 0x7fda4412b640 2019-12-05T06:27:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.923+0000 7fda48d64700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda4412b640 0x7fda4412ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:15.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.923+0000 7fda48d64700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda4412b640 0x7fda4412ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53578/0 (socket says 172.21.15.16:53578) 2019-12-05T06:27:15.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.923+0000 7fda48d64700 1 -- 172.21.15.16:0/734563178 learned_addr learned my addr 172.21.15.16:0/734563178 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:15.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.924+0000 7fda48d64700 1 -- 172.21.15.16:0/734563178 >> v1:172.21.15.76:6789/0 conn(0x7fda44153b50 legacy=0x7fda44155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:15.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.924+0000 7fda48d64700 1 -- 172.21.15.16:0/734563178 >> v1:172.21.15.16:6789/0 conn(0x7fda441302f0 legacy=0x7fda4412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:15.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.924+0000 7fda41ffb700 1 -- 172.21.15.16:0/734563178 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2532286399 0 0) 0x7fda340097c0 con 0x7fda44153b50 2019-12-05T06:27:15.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.924+0000 7fda48d64700 1 -- 172.21.15.16:0/734563178 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda44129d70 con 0x7fda4412b640 2019-12-05T06:27:15.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.924+0000 7fda48d64700 1 --2- 172.21.15.16:0/734563178 >> v2:172.21.15.16:3300/0 conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.924+0000 7fda48d64700 1 --2- 172.21.15.16:0/734563178 >> v2:172.21.15.16:3300/0 conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.924+0000 7fda48d64700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=READY pgs=779 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:15.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.925+0000 7fda48d64700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=THROTTLE_DONE pgs=779 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.925+0000 7fda41ffb700 1 -- 172.21.15.16:0/734563178 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda3800abb0 con 0x7fda4412b640 2019-12-05T06:27:15.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.925+0000 7fda48d64700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=THROTTLE_DONE pgs=779 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.925+0000 7fda48d64700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=THROTTLE_DONE pgs=779 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.925+0000 7fda41ffb700 1 -- 172.21.15.16:0/734563178 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda38009980 con 0x7fda4412b640 2019-12-05T06:27:15.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.925+0000 7fda4afc8700 1 -- 172.21.15.16:0/734563178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda44129d70 con 0x7fda4412b640 2019-12-05T06:27:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.925+0000 7fda41ffb700 1 -- 172.21.15.16:0/734563178 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda3800abb0 con 0x7fda4412b640 2019-12-05T06:27:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.925+0000 7fda4afc8700 1 -- 172.21.15.16:0/734563178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda44129d70 con 0x7fda4412b640 2019-12-05T06:27:15.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.926+0000 7fda48d64700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=THROTTLE_DONE pgs=779 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.927+0000 7fda41ffb700 1 -- 172.21.15.16:0/734563178 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fda3800abb0 con 0x7fda4412b640 2019-12-05T06:27:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.927+0000 7fda48d64700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=THROTTLE_DONE pgs=779 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.927+0000 7fda41ffb700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda24023630 0x7fda24025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:15.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.927+0000 7fda41ffb700 1 -- 172.21.15.16:0/734563178 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7fda38089e80 con 0x7fda4412b640 2019-12-05T06:27:15.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.927+0000 7fda49565700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda24023630 0x7fda24025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:15.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.928+0000 7fda49565700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda24023630 0x7fda24025ac0 crc :-1 s=READY pgs=341 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:15.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.940+0000 7fda4afc8700 1 -- 172.21.15.16:0/734563178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda44129d70 con 0x7fda4412b640 2019-12-05T06:27:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.948+0000 7fda48d64700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=THROTTLE_DONE pgs=779 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:15.948+0000 7fda41ffb700 1 -- 172.21.15.16:0/734563178 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fda3800a2e0 con 0x7fda4412b640 2019-12-05T06:27:16.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.204+0000 7fda4afc8700 1 -- 172.21.15.16:0/734563178 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda44129d70 con 0x7fda4412b640 2019-12-05T06:27:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.204+0000 7fda48d64700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=THROTTLE_DONE pgs=779 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:16.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.204+0000 7fda41ffb700 1 -- 172.21.15.16:0/734563178 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2108 (secure 0 0 0) 0x7fda3800a2e0 con 0x7fda4412b640 2019-12-05T06:27:16.193 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:16.193 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:28.191316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:active","state_seq":27,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:16.193 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 97 2019-12-05T06:27:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.207+0000 7fda4afc8700 1 -- 172.21.15.16:0/734563178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda24023630 msgr2=0x7fda24025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.207+0000 7fda4afc8700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda24023630 0x7fda24025ac0 crc :-1 s=READY pgs=341 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:16.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.207+0000 7fda4afc8700 1 -- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 msgr2=0x7fda4412ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.207+0000 7fda4afc8700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 0x7fda4412ae70 secure :-1 s=READY pgs=779 cs=0 l=1 rx=0x7fda38054470 tx=0x7fda38054810).stop 2019-12-05T06:27:16.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.207+0000 7fda4afc8700 1 -- 172.21.15.16:0/734563178 shutdown_connections 2019-12-05T06:27:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.207+0000 7fda4afc8700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda24023630 0x7fda24025ac0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.207+0000 7fda4afc8700 1 --2- 172.21.15.16:0/734563178 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fda4412b640 0x7fda4412ae70 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.208+0000 7fda4afc8700 1 -- 172.21.15.16:0/734563178 shutdown_connections 2019-12-05T06:27:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.208+0000 7fda4afc8700 1 -- 172.21.15.16:0/734563178 wait complete. 2019-12-05T06:27:16.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:16.208+0000 7fda4afc8700 1 -- 172.21.15.16:0/734563178 >> 172.21.15.16:0/734563178 conn(0x7fda440a2850 msgr2=0x7fda4406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:16.205 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:27:18.206 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.484+0000 7f71eb31a700 1 Processor -- start 2019-12-05T06:27:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.485+0000 7f71eb31a700 1 -- start start 2019-12-05T06:27:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.487+0000 7f71eb31a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 0x7f71e408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.487+0000 7f71eb31a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e40958e0 con 0x7f71e408c6d0 2019-12-05T06:27:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.487+0000 7f71eb31a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e40629e0 con 0x7f71e408d2c0 2019-12-05T06:27:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.487+0000 7f71eb31a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71e408df90 con 0x7f71e408fef0 2019-12-05T06:27:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.487+0000 7f71e98b7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f71e408c6d0 0x7f71e408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38752/0 (socket says 172.21.15.16:38752) 2019-12-05T06:27:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.487+0000 7f71e98b7700 1 -- 172.21.15.16:0/3744690565 learned_addr learned my addr 172.21.15.16:0/3744690565 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:18.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.487+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/3744690565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 0x7f71e408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:18.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.488+0000 7f71e88b5700 1 -- 172.21.15.16:0/3744690565 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775961508 0 0) 0x7f71dc0014d0 con 0x7f71e408c6d0 2019-12-05T06:27:18.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.488+0000 7f71e88b5700 1 -- 172.21.15.16:0/3744690565 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71cc002e80 con 0x7f71e408c6d0 2019-12-05T06:27:18.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.488+0000 7f71e88b5700 1 -- 172.21.15.16:0/3744690565 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391191462 0 0) 0x7f71d8001080 con 0x7f71e408d2c0 2019-12-05T06:27:18.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.488+0000 7f71e88b5700 1 -- 172.21.15.16:0/3744690565 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71cc004170 con 0x7f71e408d2c0 2019-12-05T06:27:18.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.488+0000 7f71ea0b8700 1 -- 172.21.15.16:0/3744690565 >> v1:172.21.15.76:6789/0 conn(0x7f71e408d2c0 legacy=0x7f71e408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:18.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.488+0000 7f71ea0b8700 1 -- 172.21.15.16:0/3744690565 >> v1:172.21.15.16:6789/0 conn(0x7f71e408c6d0 legacy=0x7f71e408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:18.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.488+0000 7f71ea0b8700 1 -- 172.21.15.16:0/3744690565 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e408e620 con 0x7f71e408fef0 2019-12-05T06:27:18.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.489+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/3744690565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 0x7f71e408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f71d400af80 tx=0x7f71d400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.489+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/3744690565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 0x7f71e408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f71d400af80 tx=0x7f71d400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.489+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/3744690565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 0x7f71e408f720 secure :-1 s=READY pgs=195 cs=0 l=1 rx=0x7f71d400af80 tx=0x7f71d400b790).ready entity=mon.2 client_cookie=8cf7f15b202fb172 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:18.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.489+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/3744690565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 0x7f71e408f720 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f71d400af80 tx=0x7f71d400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.489+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/3744690565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 0x7f71e408f720 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f71d400af80 tx=0x7f71d400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.489+0000 7f71e88b5700 1 -- 172.21.15.16:0/3744690565 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71d4007240 con 0x7f71e408fef0 2019-12-05T06:27:18.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.489+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/3744690565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 0x7f71e408f720 secure :-1 s=THROTTLE_DONE pgs=195 cs=0 l=1 rx=0x7f71d400af80 tx=0x7f71d400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.489+0000 7f71e88b5700 1 -- 172.21.15.16:0/3744690565 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d4000c30 con 0x7f71e408fef0 2019-12-05T06:27:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.489+0000 7f71e88b5700 1 -- 172.21.15.16:0/3744690565 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71d4009f10 con 0x7f71e408fef0 2019-12-05T06:27:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.490+0000 7f71eb31a700 1 -- 172.21.15.16:0/3744690565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 msgr2=0x7f71e408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.490+0000 7f71eb31a700 1 --2- 172.21.15.16:0/3744690565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 0x7f71e408f720 secure :-1 s=READY pgs=195 cs=0 l=1 rx=0x7f71d400af80 tx=0x7f71d400b790).stop 2019-12-05T06:27:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.491+0000 7f71eb31a700 1 -- 172.21.15.16:0/3744690565 shutdown_connections 2019-12-05T06:27:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.491+0000 7f71eb31a700 1 --2- 172.21.15.16:0/3744690565 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f71e408fef0 0x7f71e408f720 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.491+0000 7f71eb31a700 1 -- 172.21.15.16:0/3744690565 shutdown_connections 2019-12-05T06:27:18.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.491+0000 7f71eb31a700 1 -- 172.21.15.16:0/3744690565 wait complete. 2019-12-05T06:27:18.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.492+0000 7f71eb31a700 1 -- 172.21.15.16:0/3744690565 >> 172.21.15.16:0/3744690565 conn(0x7f71e4061940 msgr2=0x7f71e4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:18.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.497+0000 7f71eb31a700 1 Processor -- start 2019-12-05T06:27:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.497+0000 7f71eb31a700 1 -- start start 2019-12-05T06:27:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.498+0000 7f71eb31a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f71e4128360 0x7f71e4128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.498+0000 7f71eb31a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71e412bdd0 0x7f71e4128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.498+0000 7f71eb31a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71e4099720 con 0x7f71e4137ea0 2019-12-05T06:27:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.498+0000 7f71eb31a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e412a640 con 0x7f71e412bdd0 2019-12-05T06:27:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.498+0000 7f71eb31a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71e4129c10 con 0x7f71e4128360 2019-12-05T06:27:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.499+0000 7f71ea0b8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71e412bdd0 0x7f71e4128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.499+0000 7f71ea0b8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f71e412bdd0 0x7f71e4128c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53594/0 (socket says 172.21.15.16:53594) 2019-12-05T06:27:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.499+0000 7f71ea0b8700 1 -- 172.21.15.16:0/4265011 learned_addr learned my addr 172.21.15.16:0/4265011 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:18.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.499+0000 7f71e98b7700 1 --2- 172.21.15.16:0/4265011 >> v2:172.21.15.76:3300/0 conn(0x7f71e4128360 0x7f71e4128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:18.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.499+0000 7f71ea0b8700 1 -- 172.21.15.16:0/4265011 >> v2:172.21.15.76:3300/0 conn(0x7f71e4128360 msgr2=0x7f71e4128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:18.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.499+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> v2:172.21.15.76:3300/0 conn(0x7f71e4128360 0x7f71e4128750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:18.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.499+0000 7f71ea0b8700 1 -- 172.21.15.16:0/4265011 >> v1:172.21.15.76:6789/0 conn(0x7f71e4137ea0 legacy=0x7f71e412b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:18.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.499+0000 7f71ea0b8700 1 -- 172.21.15.16:0/4265011 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71e412acd0 con 0x7f71e412bdd0 2019-12-05T06:27:18.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.500+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> v2:172.21.15.16:3300/0 conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.500+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> v2:172.21.15.16:3300/0 conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.500+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=READY pgs=781 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.500+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.500+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.500+0000 7f71e27fc700 1 -- 172.21.15.16:0/4265011 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71d400afc0 con 0x7f71e412bdd0 2019-12-05T06:27:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.500+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.500+0000 7f71e27fc700 1 -- 172.21.15.16:0/4265011 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71d40099b0 con 0x7f71e412bdd0 2019-12-05T06:27:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.500+0000 7f71e27fc700 1 -- 172.21.15.16:0/4265011 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71d400a0c0 con 0x7f71e412bdd0 2019-12-05T06:27:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.500+0000 7f71eb31a700 1 -- 172.21.15.16:0/4265011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e412acd0 con 0x7f71e412bdd0 2019-12-05T06:27:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.501+0000 7f71eb31a700 1 -- 172.21.15.16:0/4265011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e412acd0 con 0x7f71e412bdd0 2019-12-05T06:27:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.501+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.502+0000 7f71e27fc700 1 -- 172.21.15.16:0/4265011 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f71d400ae60 con 0x7f71e412bdd0 2019-12-05T06:27:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.502+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.502+0000 7f71e27fc700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c4023630 0x7f71c4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.503+0000 7f71e27fc700 1 -- 172.21.15.16:0/4265011 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f71d4045e20 con 0x7f71e412bdd0 2019-12-05T06:27:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.503+0000 7f71e98b7700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c4023630 0x7f71c4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:18.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.503+0000 7f71e98b7700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c4023630 0x7f71c4025ac0 crc :-1 s=READY pgs=342 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:18.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.516+0000 7f71eb31a700 1 -- 172.21.15.16:0/4265011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e412acd0 con 0x7f71e412bdd0 2019-12-05T06:27:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.525+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.525+0000 7f71e27fc700 1 -- 172.21.15.16:0/4265011 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f71d40095b0 con 0x7f71e412bdd0 2019-12-05T06:27:18.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.779+0000 7f71eb31a700 1 -- 172.21.15.16:0/4265011 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71e412acd0 con 0x7f71e412bdd0 2019-12-05T06:27:18.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.780+0000 7f71ea0b8700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=THROTTLE_DONE pgs=781 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.780+0000 7f71e27fc700 1 -- 172.21.15.16:0/4265011 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f71d400ae60 con 0x7f71e412bdd0 2019-12-05T06:27:18.767 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 97 2019-12-05T06:27:18.767 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:18.767 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:28.191316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:active","state_seq":27,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:18.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 -- 172.21.15.16:0/4265011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c4023630 msgr2=0x7f71c4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:18.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c4023630 0x7f71c4025ac0 crc :-1 s=READY pgs=342 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:18.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 -- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 msgr2=0x7f71e4128c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:18.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 0x7f71e4128c90 secure :-1 s=READY pgs=781 cs=0 l=1 rx=0x7f71d4010440 tx=0x7f71d40102c0).stop 2019-12-05T06:27:18.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 -- 172.21.15.16:0/4265011 shutdown_connections 2019-12-05T06:27:18.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f71c4023630 0x7f71c4025ac0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:18.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 --2- 172.21.15.16:0/4265011 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f71e412bdd0 0x7f71e4128c90 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:18.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 --2- 172.21.15.16:0/4265011 >> v2:172.21.15.76:3300/0 conn(0x7f71e4128360 0x7f71e4128750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:18.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 -- 172.21.15.16:0/4265011 shutdown_connections 2019-12-05T06:27:18.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 -- 172.21.15.16:0/4265011 wait complete. 2019-12-05T06:27:18.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:18.783+0000 7f71eb31a700 1 -- 172.21.15.16:0/4265011 >> 172.21.15.16:0/4265011 conn(0x7f71e406d260 msgr2=0x7f71e409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:18.779 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:27:20.780 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:21.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.033+0000 7f9de180c700 1 Processor -- start 2019-12-05T06:27:21.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.033+0000 7f9de180c700 1 -- start start 2019-12-05T06:27:21.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.035+0000 7f9de180c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc08c6d0 0x7f9ddc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:21.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.035+0000 7f9de180c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9ddc08f720 0x7f9ddc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:21.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.035+0000 7f9de180c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc0958e0 con 0x7f9ddc08fef0 2019-12-05T06:27:21.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.035+0000 7f9de180c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ddc0a3f70 con 0x7f9ddc08f720 2019-12-05T06:27:21.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.035+0000 7f9de180c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc08d830 con 0x7f9ddc08c6d0 2019-12-05T06:27:21.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.035+0000 7f9ddaffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9ddc08fef0 0x7f9ddc08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38768/0 (socket says 172.21.15.16:38768) 2019-12-05T06:27:21.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.035+0000 7f9ddaffd700 1 -- 172.21.15.16:0/3848385001 learned_addr learned my addr 172.21.15.16:0/3848385001 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.035+0000 7f9ddb7fe700 1 --2- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc08c6d0 0x7f9ddc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.035+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/3848385001 >> v2:172.21.15.76:3300/0 conn(0x7f9ddc08f720 0x7f9ddc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.036+0000 7f9dda7fc700 1 -- 172.21.15.16:0/3848385001 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319456564 0 0) 0x7f9dc40014d0 con 0x7f9ddc08fef0 2019-12-05T06:27:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.036+0000 7f9dda7fc700 1 -- 172.21.15.16:0/3848385001 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dc8002e80 con 0x7f9ddc08fef0 2019-12-05T06:27:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.036+0000 7f9ddbfff700 1 -- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc08c6d0 msgr2=0x7f9ddc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.036+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc08c6d0 0x7f9ddc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:21.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.036+0000 7f9ddbfff700 1 -- 172.21.15.16:0/3848385001 >> v1:172.21.15.16:6789/0 conn(0x7f9ddc08fef0 legacy=0x7f9ddc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.036+0000 7f9ddbfff700 1 -- 172.21.15.16:0/3848385001 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc08fbc0 con 0x7f9ddc08f720 2019-12-05T06:27:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.036+0000 7f9ddb7fe700 1 --2- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc08c6d0 0x7f9ddc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:27:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.036+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/3848385001 >> v2:172.21.15.76:3300/0 conn(0x7f9ddc08f720 0x7f9ddc08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9dd400af80 tx=0x7f9dd400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.037+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/3848385001 >> v2:172.21.15.76:3300/0 conn(0x7f9ddc08f720 0x7f9ddc08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9dd400af80 tx=0x7f9dd400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.037+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ddc08f720 0x7f9ddc08d000 secure :-1 s=READY pgs=567 cs=0 l=1 rx=0x7f9dd400af80 tx=0x7f9dd400b790).ready entity=mon.1 client_cookie=8cacb9a00cece417 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:21.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.037+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ddc08f720 0x7f9ddc08d000 secure :-1 s=THROTTLE_DONE pgs=567 cs=0 l=1 rx=0x7f9dd400af80 tx=0x7f9dd400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.037+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ddc08f720 0x7f9ddc08d000 secure :-1 s=THROTTLE_DONE pgs=567 cs=0 l=1 rx=0x7f9dd400af80 tx=0x7f9dd400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.037+0000 7f9dda7fc700 1 -- 172.21.15.16:0/3848385001 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dd4000bc0 con 0x7f9ddc08f720 2019-12-05T06:27:21.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.037+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ddc08f720 0x7f9ddc08d000 secure :-1 s=THROTTLE_DONE pgs=567 cs=0 l=1 rx=0x7f9dd400af80 tx=0x7f9dd400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.037+0000 7f9dda7fc700 1 -- 172.21.15.16:0/3848385001 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd40097f0 con 0x7f9ddc08f720 2019-12-05T06:27:21.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.037+0000 7f9dda7fc700 1 -- 172.21.15.16:0/3848385001 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dd4009ec0 con 0x7f9ddc08f720 2019-12-05T06:27:21.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.037+0000 7f9de180c700 1 -- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ddc08f720 msgr2=0x7f9ddc08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:21.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.037+0000 7f9de180c700 1 --2- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ddc08f720 0x7f9ddc08d000 secure :-1 s=READY pgs=567 cs=0 l=1 rx=0x7f9dd400af80 tx=0x7f9dd400b790).stop 2019-12-05T06:27:21.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.039+0000 7f9de180c700 1 -- 172.21.15.16:0/3848385001 shutdown_connections 2019-12-05T06:27:21.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.039+0000 7f9de180c700 1 --2- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9ddc08f720 0x7f9ddc08d000 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:21.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.039+0000 7f9de180c700 1 --2- 172.21.15.16:0/3848385001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc08c6d0 0x7f9ddc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:21.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.039+0000 7f9de180c700 1 -- 172.21.15.16:0/3848385001 shutdown_connections 2019-12-05T06:27:21.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.039+0000 7f9de180c700 1 -- 172.21.15.16:0/3848385001 wait complete. 2019-12-05T06:27:21.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.039+0000 7f9de180c700 1 -- 172.21.15.16:0/3848385001 >> 172.21.15.16:0/3848385001 conn(0x7f9ddc061940 msgr2=0x7f9ddc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:21.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.044+0000 7f9de180c700 1 Processor -- start 2019-12-05T06:27:21.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.044+0000 7f9de180c700 1 -- start start 2019-12-05T06:27:21.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.046+0000 7f9de180c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9ddc1302f0 0x7f9ddc09d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:21.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.046+0000 7f9de180c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:21.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.046+0000 7f9de180c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ddc12cb90 con 0x7f9ddc09e7d0 2019-12-05T06:27:21.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.046+0000 7f9de180c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ddc131ff0 con 0x7f9ddc1302f0 2019-12-05T06:27:21.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.046+0000 7f9de180c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ddc1315c0 con 0x7f9ddc156e70 2019-12-05T06:27:21.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.046+0000 7f9ddb7fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9ddc1302f0 0x7f9ddc09d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:21.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.046+0000 7f9ddbfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:21.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.047+0000 7f9ddb7fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9ddc1302f0 0x7f9ddc09d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:39696/0 (socket says 172.21.15.16:39696) 2019-12-05T06:27:21.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.047+0000 7f9ddb7fe700 1 -- 172.21.15.16:0/2617911817 learned_addr learned my addr 172.21.15.16:0/2617911817 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:21.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.047+0000 7f9ddbfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:57600/0 (socket says 172.21.15.16:57600) 2019-12-05T06:27:21.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.047+0000 7f9ddbfff700 1 -- 172.21.15.16:0/2617911817 >> v2:172.21.15.76:3300/0 conn(0x7f9ddc1302f0 msgr2=0x7f9ddc09d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:21.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.047+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> v2:172.21.15.76:3300/0 conn(0x7f9ddc1302f0 0x7f9ddc09d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:21.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.047+0000 7f9ddbfff700 1 -- 172.21.15.16:0/2617911817 >> v1:172.21.15.76:6789/0 conn(0x7f9ddc09e7d0 legacy=0x7f9ddc09dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:21.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.047+0000 7f9ddbfff700 1 -- 172.21.15.16:0/2617911817 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ddc132680 con 0x7f9ddc156e70 2019-12-05T06:27:21.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.047+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.047+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.047+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=READY pgs=196 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:21.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.048+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.048+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.048+0000 7f9dd3fff700 1 -- 172.21.15.16:0/2617911817 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dd4009990 con 0x7f9ddc156e70 2019-12-05T06:27:21.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.048+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.048+0000 7f9dd3fff700 1 -- 172.21.15.16:0/2617911817 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dd40095b0 con 0x7f9ddc156e70 2019-12-05T06:27:21.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.048+0000 7f9dd3fff700 1 -- 172.21.15.16:0/2617911817 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dd4009f70 con 0x7f9ddc156e70 2019-12-05T06:27:21.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.048+0000 7f9de180c700 1 -- 172.21.15.16:0/2617911817 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ddc132680 con 0x7f9ddc156e70 2019-12-05T06:27:21.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.048+0000 7f9de180c700 1 -- 172.21.15.16:0/2617911817 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ddc132680 con 0x7f9ddc156e70 2019-12-05T06:27:21.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.049+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.050+0000 7f9dd3fff700 1 -- 172.21.15.16:0/2617911817 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9dd4009840 con 0x7f9ddc156e70 2019-12-05T06:27:21.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.050+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.050+0000 7f9dd3fff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9dc0023630 0x7f9dc0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:21.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.050+0000 7f9dd3fff700 1 -- 172.21.15.16:0/2617911817 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 7149+0+0 (secure 0 0 0) 0x7f9dd4045ce0 con 0x7f9ddc156e70 2019-12-05T06:27:21.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.050+0000 7f9ddb7fe700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9dc0023630 0x7f9dc0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:21.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.051+0000 7f9ddb7fe700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9dc0023630 0x7f9dc0025ac0 crc :-1 s=READY pgs=343 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:21.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.064+0000 7f9de180c700 1 -- 172.21.15.16:0/2617911817 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ddc132680 con 0x7f9ddc156e70 2019-12-05T06:27:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.074+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.074+0000 7f9dd3fff700 1 -- 172.21.15.16:0/2617911817 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9dd4002c30 con 0x7f9ddc156e70 2019-12-05T06:27:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.300+0000 7f9de180c700 1 -- 172.21.15.16:0/2617911817 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ddc132680 con 0x7f9ddc156e70 2019-12-05T06:27:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.301+0000 7f9ddbfff700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=THROTTLE_DONE pgs=196 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:21.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.301+0000 7f9dd3fff700 1 -- 172.21.15.16:0/2617911817 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 97 v97) v1 ==== 94+0+2108 (secure 0 0 0) 0x7f9dd40096a0 con 0x7f9ddc156e70 2019-12-05T06:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 97 2019-12-05T06:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 -- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9dc0023630 msgr2=0x7f9dc0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9dc0023630 0x7f9dc0025ac0 crc :-1 s=READY pgs=343 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 -- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 msgr2=0x7f9ddc159270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 secure :-1 s=READY pgs=196 cs=0 l=1 rx=0x7f9dd4010380 tx=0x7f9dd4010200).stop 2019-12-05T06:27:21.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 -- 172.21.15.16:0/2617911817 shutdown_connections 2019-12-05T06:27:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9dc0023630 0x7f9dc0025ac0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 --2- 172.21.15.16:0/2617911817 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ddc156e70 0x7f9ddc159270 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 --2- 172.21.15.16:0/2617911817 >> v2:172.21.15.76:3300/0 conn(0x7f9ddc1302f0 0x7f9ddc09d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 -- 172.21.15.16:0/2617911817 shutdown_connections 2019-12-05T06:27:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 -- 172.21.15.16:0/2617911817 wait complete. 2019-12-05T06:27:21.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:21.307+0000 7f9de180c700 1 -- 172.21.15.16:0/2617911817 >> 172.21.15.16:0/2617911817 conn(0x7f9ddc0a2850 msgr2=0x7f9ddc06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:21.299 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:21.300 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":97,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6165,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":97}],"filesystems":[{"mdsmap":{"epoch":96,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:26:28.191316+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5135},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5135":{"gid":5135,"name":"b","rank":0,"incarnation":93,"state":"up:active","state_seq":27,"addr":"172.21.15.76:6835/2135855524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2135855524},{"type":"v1","addr":"172.21.15.76:6835","nonce":2135855524}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:21.310 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:27:23.312 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:23.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.497+0000 7f7c7b108700 1 Processor -- start 2019-12-05T06:27:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.498+0000 7f7c7b108700 1 -- start start 2019-12-05T06:27:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.498+0000 7f7c7b108700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c7408c6d0 0x7f7c7408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:23.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.498+0000 7f7c7b108700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c7408f720 0x7f7c7408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.498+0000 7f7c7b108700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c7408fb70 con 0x7f7c7408fef0 2019-12-05T06:27:23.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.498+0000 7f7c7b108700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7408df90 con 0x7f7c7408c6d0 2019-12-05T06:27:23.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.498+0000 7f7c7b108700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c740a3f70 con 0x7f7c7408f720 2019-12-05T06:27:23.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c7408c6d0 0x7f7c7408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:23.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c7408c6d0 0x7f7c7408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53628/0 (socket says 172.21.15.16:53628) 2019-12-05T06:27:23.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 -- 172.21.15.16:0/2434177577 learned_addr learned my addr 172.21.15.16:0/2434177577 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:23.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/2434177577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c7408f720 0x7f7c7408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:23.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 -- 172.21.15.16:0/2434177577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c7408f720 msgr2=0x7f7c7408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:23.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 --2- 172.21.15.16:0/2434177577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c7408f720 0x7f7c7408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:23.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 -- 172.21.15.16:0/2434177577 >> v1:172.21.15.76:6789/0 conn(0x7f7c7408fef0 legacy=0x7f7c7408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:23.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 -- 172.21.15.16:0/2434177577 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7408d540 con 0x7f7c7408c6d0 2019-12-05T06:27:23.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 --2- 172.21.15.16:0/2434177577 >> v2:172.21.15.16:3300/0 conn(0x7f7c7408c6d0 0x7f7c7408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7c6c00fac0 tx=0x7f7c6c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 --2- 172.21.15.16:0/2434177577 >> v2:172.21.15.16:3300/0 conn(0x7f7c7408c6d0 0x7f7c7408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7c6c00fac0 tx=0x7f7c6c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 --2- 172.21.15.16:0/2434177577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c7408c6d0 0x7f7c7408cac0 secure :-1 s=READY pgs=783 cs=0 l=1 rx=0x7f7c6c00fac0 tx=0x7f7c6c00f8f0).ready entity=mon.0 client_cookie=a8d67f1b910e8ad3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.499+0000 7f7c796a5700 1 --2- 172.21.15.16:0/2434177577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c7408c6d0 0x7f7c7408cac0 secure :-1 s=THROTTLE_DONE pgs=783 cs=0 l=1 rx=0x7f7c6c00fac0 tx=0x7f7c6c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.500+0000 7f7c796a5700 1 --2- 172.21.15.16:0/2434177577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c7408c6d0 0x7f7c7408cac0 secure :-1 s=THROTTLE_DONE pgs=783 cs=0 l=1 rx=0x7f7c6c00fac0 tx=0x7f7c6c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.500+0000 7f7c796a5700 1 --2- 172.21.15.16:0/2434177577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c7408c6d0 0x7f7c7408cac0 secure :-1 s=THROTTLE_DONE pgs=783 cs=0 l=1 rx=0x7f7c6c00fac0 tx=0x7f7c6c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.500+0000 7f7c67fff700 1 -- 172.21.15.16:0/2434177577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c6c000ce0 con 0x7f7c7408c6d0 2019-12-05T06:27:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.500+0000 7f7c67fff700 1 -- 172.21.15.16:0/2434177577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c6c001010 con 0x7f7c7408c6d0 2019-12-05T06:27:23.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.500+0000 7f7c67fff700 1 -- 172.21.15.16:0/2434177577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c6c001720 con 0x7f7c7408c6d0 2019-12-05T06:27:23.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.500+0000 7f7c7b108700 1 -- 172.21.15.16:0/2434177577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c7408c6d0 msgr2=0x7f7c7408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:23.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.500+0000 7f7c7b108700 1 --2- 172.21.15.16:0/2434177577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c7408c6d0 0x7f7c7408cac0 secure :-1 s=READY pgs=783 cs=0 l=1 rx=0x7f7c6c00fac0 tx=0x7f7c6c00f8f0).stop 2019-12-05T06:27:23.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.500+0000 7f7c7b108700 1 -- 172.21.15.16:0/2434177577 shutdown_connections 2019-12-05T06:27:23.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.500+0000 7f7c7b108700 1 --2- 172.21.15.16:0/2434177577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c7408f720 0x7f7c7408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:23.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.500+0000 7f7c7b108700 1 --2- 172.21.15.16:0/2434177577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c7408c6d0 0x7f7c7408cac0 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:23.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.501+0000 7f7c7b108700 1 -- 172.21.15.16:0/2434177577 shutdown_connections 2019-12-05T06:27:23.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.501+0000 7f7c7b108700 1 -- 172.21.15.16:0/2434177577 wait complete. 2019-12-05T06:27:23.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.501+0000 7f7c7b108700 1 -- 172.21.15.16:0/2434177577 >> 172.21.15.16:0/2434177577 conn(0x7f7c74061940 msgr2=0x7f7c74096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:23.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.503+0000 7f7c7b108700 1 Processor -- start 2019-12-05T06:27:23.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.503+0000 7f7c7b108700 1 -- start start 2019-12-05T06:27:23.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.504+0000 7f7c7b108700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c7409bbe0 0x7f7c741539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:23.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.504+0000 7f7c7b108700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c74084f30 0x7f7c74155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:23.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.504+0000 7f7c7b108700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c74084b90 con 0x7f7c74096e90 2019-12-05T06:27:23.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.504+0000 7f7c7b108700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c7409a450 con 0x7f7c74084f30 2019-12-05T06:27:23.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.504+0000 7f7c7b108700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c74099a20 con 0x7f7c7409bbe0 2019-12-05T06:27:23.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.504+0000 7f7c79ea6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c74084f30 0x7f7c74155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:23.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.504+0000 7f7c79ea6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c74084f30 0x7f7c74155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53638/0 (socket says 172.21.15.16:53638) 2019-12-05T06:27:23.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.504+0000 7f7c79ea6700 1 -- 172.21.15.16:0/1751011719 learned_addr learned my addr 172.21.15.16:0/1751011719 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:23.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.504+0000 7f7c78ea4700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c7409bbe0 0x7f7c741539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:23.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c79ea6700 1 -- 172.21.15.16:0/1751011719 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c7409bbe0 msgr2=0x7f7c741539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:23.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c7409bbe0 0x7f7c741539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:23.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c79ea6700 1 -- 172.21.15.16:0/1751011719 >> v1:172.21.15.76:6789/0 conn(0x7f7c74096e90 legacy=0x7f7c7409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:23.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c79ea6700 1 -- 172.21.15.16:0/1751011719 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c7409aae0 con 0x7f7c74084f30 2019-12-05T06:27:23.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> v2:172.21.15.16:3300/0 conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> v2:172.21.15.16:3300/0 conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=READY pgs=784 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:23.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=THROTTLE_DONE pgs=784 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c65ffb700 1 -- 172.21.15.16:0/1751011719 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c7000f090 con 0x7f7c74084f30 2019-12-05T06:27:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c7b108700 1 -- 172.21.15.16:0/1751011719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c7409aae0 con 0x7f7c74084f30 2019-12-05T06:27:23.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=THROTTLE_DONE pgs=784 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=THROTTLE_DONE pgs=784 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c65ffb700 1 -- 172.21.15.16:0/1751011719 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c7000ecb0 con 0x7f7c74084f30 2019-12-05T06:27:23.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c65ffb700 1 -- 172.21.15.16:0/1751011719 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c7000eea0 con 0x7f7c74084f30 2019-12-05T06:27:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.505+0000 7f7c7b108700 1 -- 172.21.15.16:0/1751011719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c7409aae0 con 0x7f7c74084f30 2019-12-05T06:27:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.506+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=THROTTLE_DONE pgs=784 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.506+0000 7f7c65ffb700 1 -- 172.21.15.16:0/1751011719 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7c7000eea0 con 0x7f7c74084f30 2019-12-05T06:27:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.506+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=THROTTLE_DONE pgs=784 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.506+0000 7f7c65ffb700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c54023630 0x7f7c54025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:23.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.506+0000 7f7c65ffb700 1 -- 172.21.15.16:0/1751011719 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f7c70048030 con 0x7f7c74084f30 2019-12-05T06:27:23.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.507+0000 7f7c796a5700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c54023630 0x7f7c54025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:23.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.507+0000 7f7c796a5700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c54023630 0x7f7c54025ac0 crc :-1 s=READY pgs=344 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:23.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.513+0000 7f7c7b108700 1 -- 172.21.15.16:0/1751011719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c7409aae0 con 0x7f7c74084f30 2019-12-05T06:27:23.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.517+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=THROTTLE_DONE pgs=784 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.517+0000 7f7c65ffb700 1 -- 172.21.15.16:0/1751011719 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7c7000ea80 con 0x7f7c74084f30 2019-12-05T06:27:23.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.715+0000 7f7c7b108700 1 -- 172.21.15.16:0/1751011719 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c7409aae0 con 0x7f7c74084f30 2019-12-05T06:27:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.716+0000 7f7c79ea6700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=THROTTLE_DONE pgs=784 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.716+0000 7f7c65ffb700 1 -- 172.21.15.16:0/1751011719 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f7c7000ec90 con 0x7f7c74084f30 2019-12-05T06:27:23.705 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 98 2019-12-05T06:27:23.705 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:23.705 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:23.200306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.718+0000 7f7c7b108700 1 -- 172.21.15.16:0/1751011719 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c54023630 msgr2=0x7f7c54025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.718+0000 7f7c7b108700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c54023630 0x7f7c54025ac0 crc :-1 s=READY pgs=344 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:23.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.718+0000 7f7c7b108700 1 -- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 msgr2=0x7f7c74155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:23.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.718+0000 7f7c7b108700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 0x7f7c74155f30 secure :-1 s=READY pgs=784 cs=0 l=1 rx=0x7f7c700102b0 tx=0x7f7c700128f0).stop 2019-12-05T06:27:23.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.719+0000 7f7c7b108700 1 -- 172.21.15.16:0/1751011719 shutdown_connections 2019-12-05T06:27:23.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.719+0000 7f7c7b108700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c54023630 0x7f7c54025ac0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:23.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.719+0000 7f7c7b108700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c74084f30 0x7f7c74155f30 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:23.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.719+0000 7f7c7b108700 1 --2- 172.21.15.16:0/1751011719 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c7409bbe0 0x7f7c741539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.719+0000 7f7c7b108700 1 -- 172.21.15.16:0/1751011719 shutdown_connections 2019-12-05T06:27:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.719+0000 7f7c7b108700 1 -- 172.21.15.16:0/1751011719 wait complete. 2019-12-05T06:27:23.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.719+0000 7f7c7b108700 1 -- 172.21.15.16:0/1751011719 >> 172.21.15.16:0/1751011719 conn(0x7f7c740714f0 msgr2=0x7f7c7409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:23.715 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:27:23.715 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:27:23.716 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.883+0000 7f24ff166700 1 Processor -- start 2019-12-05T06:27:23.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.884+0000 7f24ff166700 1 -- start start 2019-12-05T06:27:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.884+0000 7f24ff166700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.884+0000 7f24ff166700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f24f808f720 0x7f24f808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.884+0000 7f24ff166700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f80958e0 con 0x7f24f808fef0 2019-12-05T06:27:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.884+0000 7f24ff166700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f80a3f70 con 0x7f24f808f720 2019-12-05T06:27:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.884+0000 7f24ff166700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24f808d830 con 0x7f24f808c6d0 2019-12-05T06:27:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.885+0000 7f24fd703700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:23.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.885+0000 7f24fd703700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:57632/0 (socket says 172.21.15.16:57632) 2019-12-05T06:27:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.885+0000 7f24fd703700 1 -- 172.21.15.16:0/2469075524 learned_addr learned my addr 172.21.15.16:0/2469075524 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.885+0000 7f24fdf04700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f24f808f720 0x7f24f808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.885+0000 7f24fd703700 1 -- 172.21.15.16:0/2469075524 >> v2:172.21.15.76:3300/0 conn(0x7f24f808f720 msgr2=0x7f24f808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.885+0000 7f24fd703700 1 --2- 172.21.15.16:0/2469075524 >> v2:172.21.15.76:3300/0 conn(0x7f24f808f720 0x7f24f808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.885+0000 7f24fd703700 1 -- 172.21.15.16:0/2469075524 >> v1:172.21.15.76:6789/0 conn(0x7f24f808fef0 legacy=0x7f24f808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.885+0000 7f24fd703700 1 -- 172.21.15.16:0/2469075524 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f808fbc0 con 0x7f24f808c6d0 2019-12-05T06:27:23.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.885+0000 7f24fd703700 1 --2- 172.21.15.16:0/2469075524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f24f000fac0 tx=0x7f24f000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.886+0000 7f24fd703700 1 --2- 172.21.15.16:0/2469075524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f24f000fac0 tx=0x7f24f000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.886+0000 7f24fd703700 1 --2- 172.21.15.16:0/2469075524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 secure :-1 s=READY pgs=197 cs=0 l=1 rx=0x7f24f000fac0 tx=0x7f24f000f8f0).ready entity=mon.2 client_cookie=ab64f2ddb3784986 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:23.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.886+0000 7f24fd703700 1 --2- 172.21.15.16:0/2469075524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f24f000fac0 tx=0x7f24f000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.886+0000 7f24effff700 1 -- 172.21.15.16:0/2469075524 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24f0000ce0 con 0x7f24f808c6d0 2019-12-05T06:27:23.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.886+0000 7f24fd703700 1 --2- 172.21.15.16:0/2469075524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f24f000fac0 tx=0x7f24f000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.886+0000 7f24fd703700 1 --2- 172.21.15.16:0/2469075524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 secure :-1 s=THROTTLE_DONE pgs=197 cs=0 l=1 rx=0x7f24f000fac0 tx=0x7f24f000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.886+0000 7f24effff700 1 -- 172.21.15.16:0/2469075524 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f0001010 con 0x7f24f808c6d0 2019-12-05T06:27:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.886+0000 7f24effff700 1 -- 172.21.15.16:0/2469075524 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24f0001720 con 0x7f24f808c6d0 2019-12-05T06:27:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.886+0000 7f24ff166700 1 -- 172.21.15.16:0/2469075524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 msgr2=0x7f24f808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.886+0000 7f24ff166700 1 --2- 172.21.15.16:0/2469075524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 secure :-1 s=READY pgs=197 cs=0 l=1 rx=0x7f24f000fac0 tx=0x7f24f000f8f0).stop 2019-12-05T06:27:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.887+0000 7f24ff166700 1 -- 172.21.15.16:0/2469075524 shutdown_connections 2019-12-05T06:27:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.887+0000 7f24ff166700 1 --2- 172.21.15.16:0/2469075524 >> v2:172.21.15.76:3300/0 conn(0x7f24f808f720 0x7f24f808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:23.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.887+0000 7f24ff166700 1 --2- 172.21.15.16:0/2469075524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f24f808c6d0 0x7f24f808cac0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.887+0000 7f24ff166700 1 -- 172.21.15.16:0/2469075524 shutdown_connections 2019-12-05T06:27:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.887+0000 7f24ff166700 1 -- 172.21.15.16:0/2469075524 wait complete. 2019-12-05T06:27:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.887+0000 7f24ff166700 1 -- 172.21.15.16:0/2469075524 >> 172.21.15.16:0/2469075524 conn(0x7f24f8061940 msgr2=0x7f24f8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.890+0000 7f24ff166700 1 Processor -- start 2019-12-05T06:27:23.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.890+0000 7f24ff166700 1 -- start start 2019-12-05T06:27:23.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.890+0000 7f24ff166700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24f81302f0 0x7f24f809d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:23.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24ff166700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f24f8156e70 0x7f24f8159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:23.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24ff166700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24f812cb90 con 0x7f24f809e7d0 2019-12-05T06:27:23.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24ff166700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f8131ff0 con 0x7f24f81302f0 2019-12-05T06:27:23.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24ff166700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24f81315c0 con 0x7f24f8156e70 2019-12-05T06:27:23.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24f81302f0 0x7f24f809d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f24f81302f0 0x7f24f809d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53646/0 (socket says 172.21.15.16:53646) 2019-12-05T06:27:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 -- 172.21.15.16:0/2871340252 learned_addr learned my addr 172.21.15.16:0/2871340252 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fdf04700 1 --2- 172.21.15.16:0/2871340252 >> v2:172.21.15.76:3300/0 conn(0x7f24f8156e70 0x7f24f8159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 -- 172.21.15.16:0/2871340252 >> v2:172.21.15.76:3300/0 conn(0x7f24f8156e70 msgr2=0x7f24f8159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> v2:172.21.15.76:3300/0 conn(0x7f24f8156e70 0x7f24f8159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 -- 172.21.15.16:0/2871340252 >> v1:172.21.15.76:6789/0 conn(0x7f24f809e7d0 legacy=0x7f24f809dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:23.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 -- 172.21.15.16:0/2871340252 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24f8132680 con 0x7f24f81302f0 2019-12-05T06:27:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fdf04700 1 --2- 172.21.15.16:0/2871340252 >> v2:172.21.15.76:3300/0 conn(0x7f24f8156e70 0x7f24f8159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:27:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> v2:172.21.15.16:3300/0 conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> v2:172.21.15.16:3300/0 conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=READY pgs=785 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24edffb700 1 -- 172.21.15.16:0/2871340252 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24f0008050 con 0x7f24f81302f0 2019-12-05T06:27:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24edffb700 1 -- 172.21.15.16:0/2871340252 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24f00010c0 con 0x7f24f81302f0 2019-12-05T06:27:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24edffb700 1 -- 172.21.15.16:0/2871340252 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24f00016d0 con 0x7f24f81302f0 2019-12-05T06:27:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24ff166700 1 -- 172.21.15.16:0/2871340252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24f8132680 con 0x7f24f81302f0 2019-12-05T06:27:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.891+0000 7f24ff166700 1 -- 172.21.15.16:0/2871340252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24f8132680 con 0x7f24f81302f0 2019-12-05T06:27:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.892+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.892+0000 7f24edffb700 1 -- 172.21.15.16:0/2871340252 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f24f00012b0 con 0x7f24f81302f0 2019-12-05T06:27:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.892+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.892+0000 7f24edffb700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f24d8023630 0x7f24d8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.893+0000 7f24edffb700 1 -- 172.21.15.16:0/2871340252 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f24f00461d0 con 0x7f24f81302f0 2019-12-05T06:27:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.893+0000 7f24fcf02700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f24d8023630 0x7f24d8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:23.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.893+0000 7f24fcf02700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f24d8023630 0x7f24d8025ac0 crc :-1 s=READY pgs=345 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:23.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.899+0000 7f24ff166700 1 -- 172.21.15.16:0/2871340252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24f8132680 con 0x7f24f81302f0 2019-12-05T06:27:23.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.905+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:23.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:23.905+0000 7f24edffb700 1 -- 172.21.15.16:0/2871340252 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f24f0000cc0 con 0x7f24f81302f0 2019-12-05T06:27:24.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.102+0000 7f24ff166700 1 -- 172.21.15.16:0/2871340252 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24f8132680 con 0x7f24f81302f0 2019-12-05T06:27:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.103+0000 7f24fd703700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=THROTTLE_DONE pgs=785 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:24.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.103+0000 7f24edffb700 1 -- 172.21.15.16:0/2871340252 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 98 v98) v1 ==== 94+0+1771 (secure 0 0 0) 0x7f24f00012b0 con 0x7f24f81302f0 2019-12-05T06:27:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 98 2019-12-05T06:27:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.105+0000 7f24ff166700 1 -- 172.21.15.16:0/2871340252 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f24d8023630 msgr2=0x7f24d8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.105+0000 7f24ff166700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f24d8023630 0x7f24d8025ac0 crc :-1 s=READY pgs=345 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.105+0000 7f24ff166700 1 -- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 msgr2=0x7f24f809d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.105+0000 7f24ff166700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 0x7f24f809d800 secure :-1 s=READY pgs=785 cs=0 l=1 rx=0x7f24f00107c0 tx=0x7f24f0010b60).stop 2019-12-05T06:27:24.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.106+0000 7f24ff166700 1 -- 172.21.15.16:0/2871340252 shutdown_connections 2019-12-05T06:27:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.106+0000 7f24ff166700 1 --2- 172.21.15.16:0/2871340252 >> v2:172.21.15.76:3300/0 conn(0x7f24f8156e70 0x7f24f8159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:24.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.106+0000 7f24ff166700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f24d8023630 0x7f24d8025ac0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:24.095 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:24.095 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":98,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":98,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:23.200306+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.106+0000 7f24ff166700 1 --2- 172.21.15.16:0/2871340252 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f24f81302f0 0x7f24f809d800 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.106+0000 7f24ff166700 1 -- 172.21.15.16:0/2871340252 shutdown_connections 2019-12-05T06:27:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.106+0000 7f24ff166700 1 -- 172.21.15.16:0/2871340252 wait complete. 2019-12-05T06:27:24.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:24.106+0000 7f24ff166700 1 -- 172.21.15.16:0/2871340252 >> 172.21.15.16:0/2871340252 conn(0x7f24f80a2850 msgr2=0x7f24f806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:24.103 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 98, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 98, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6165": { "addr": "172.21.15.16:6833/2514381299", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 2514381299, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 2514381299, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6165, "incarnation": 98, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 63, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:27:23.200306+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6165 } } } ], "standbys": [] } 2019-12-05T06:27:26.105 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:26.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.375+0000 7f5b0e13f700 1 Processor -- start 2019-12-05T06:27:26.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.375+0000 7f5b0e13f700 1 -- start start 2019-12-05T06:27:26.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.377+0000 7f5b0e13f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b0808c6d0 0x7f5b0808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:26.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.377+0000 7f5b0e13f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5b0808fef0 0x7f5b0808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:26.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.377+0000 7f5b0e13f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0808fb80 con 0x7f5b0808ef20 2019-12-05T06:27:26.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.377+0000 7f5b0e13f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b0808df90 con 0x7f5b0808c6d0 2019-12-05T06:27:26.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.377+0000 7f5b0e13f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b080a3f70 con 0x7f5b0808fef0 2019-12-05T06:27:26.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.377+0000 7f5b07fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b0808c6d0 0x7f5b0808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:26.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.377+0000 7f5b07fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5b0808c6d0 0x7f5b0808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53656/0 (socket says 172.21.15.16:53656) 2019-12-05T06:27:26.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.377+0000 7f5b07fff700 1 -- 172.21.15.16:0/2599954894 learned_addr learned my addr 172.21.15.16:0/2599954894 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:26.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.377+0000 7f5b077fe700 1 --2- 172.21.15.16:0/2599954894 >> v2:172.21.15.76:3300/0 conn(0x7f5b0808fef0 0x7f5b0808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.378+0000 7f5b06ffd700 1 -- 172.21.15.16:0/2599954894 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63881795 0 0) 0x7f5b000009b0 con 0x7f5b0808ef20 2019-12-05T06:27:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.378+0000 7f5b07fff700 1 -- 172.21.15.16:0/2599954894 >> v2:172.21.15.76:3300/0 conn(0x7f5b0808fef0 msgr2=0x7f5b0808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.378+0000 7f5b07fff700 1 --2- 172.21.15.16:0/2599954894 >> v2:172.21.15.76:3300/0 conn(0x7f5b0808fef0 0x7f5b0808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.378+0000 7f5b07fff700 1 -- 172.21.15.16:0/2599954894 >> v1:172.21.15.76:6789/0 conn(0x7f5b0808ef20 legacy=0x7f5b0808f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:26.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.378+0000 7f5b07fff700 1 -- 172.21.15.16:0/2599954894 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0808d540 con 0x7f5b0808c6d0 2019-12-05T06:27:26.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.378+0000 7f5b07fff700 1 --2- 172.21.15.16:0/2599954894 >> v2:172.21.15.16:3300/0 conn(0x7f5b0808c6d0 0x7f5b0808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5af800fac0 tx=0x7f5af800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.378+0000 7f5b07fff700 1 --2- 172.21.15.16:0/2599954894 >> v2:172.21.15.16:3300/0 conn(0x7f5b0808c6d0 0x7f5b0808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5af800fac0 tx=0x7f5af800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.378+0000 7f5b07fff700 1 --2- 172.21.15.16:0/2599954894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0808c6d0 0x7f5b0808cac0 secure :-1 s=READY pgs=786 cs=0 l=1 rx=0x7f5af800fac0 tx=0x7f5af800f8f0).ready entity=mon.0 client_cookie=729517d36a4bdb6a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.379+0000 7f5b07fff700 1 --2- 172.21.15.16:0/2599954894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0808c6d0 0x7f5b0808cac0 secure :-1 s=THROTTLE_DONE pgs=786 cs=0 l=1 rx=0x7f5af800fac0 tx=0x7f5af800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.379+0000 7f5b07fff700 1 --2- 172.21.15.16:0/2599954894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0808c6d0 0x7f5b0808cac0 secure :-1 s=THROTTLE_DONE pgs=786 cs=0 l=1 rx=0x7f5af800fac0 tx=0x7f5af800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.379+0000 7f5b06ffd700 1 -- 172.21.15.16:0/2599954894 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af8000ce0 con 0x7f5b0808c6d0 2019-12-05T06:27:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.379+0000 7f5b07fff700 1 --2- 172.21.15.16:0/2599954894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0808c6d0 0x7f5b0808cac0 secure :-1 s=THROTTLE_DONE pgs=786 cs=0 l=1 rx=0x7f5af800fac0 tx=0x7f5af800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.379+0000 7f5b06ffd700 1 -- 172.21.15.16:0/2599954894 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af8001010 con 0x7f5b0808c6d0 2019-12-05T06:27:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.379+0000 7f5b06ffd700 1 -- 172.21.15.16:0/2599954894 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af8001720 con 0x7f5b0808c6d0 2019-12-05T06:27:26.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.379+0000 7f5b0e13f700 1 -- 172.21.15.16:0/2599954894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0808c6d0 msgr2=0x7f5b0808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.379+0000 7f5b0e13f700 1 --2- 172.21.15.16:0/2599954894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0808c6d0 0x7f5b0808cac0 secure :-1 s=READY pgs=786 cs=0 l=1 rx=0x7f5af800fac0 tx=0x7f5af800f8f0).stop 2019-12-05T06:27:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.381+0000 7f5b0e13f700 1 -- 172.21.15.16:0/2599954894 shutdown_connections 2019-12-05T06:27:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.381+0000 7f5b0e13f700 1 --2- 172.21.15.16:0/2599954894 >> v2:172.21.15.76:3300/0 conn(0x7f5b0808fef0 0x7f5b0808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.381+0000 7f5b0e13f700 1 --2- 172.21.15.16:0/2599954894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5b0808c6d0 0x7f5b0808cac0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.381+0000 7f5b0e13f700 1 -- 172.21.15.16:0/2599954894 shutdown_connections 2019-12-05T06:27:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.381+0000 7f5b0e13f700 1 -- 172.21.15.16:0/2599954894 wait complete. 2019-12-05T06:27:26.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.381+0000 7f5b0e13f700 1 -- 172.21.15.16:0/2599954894 >> 172.21.15.16:0/2599954894 conn(0x7f5b08061940 msgr2=0x7f5b08096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:26.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.386+0000 7f5b0e13f700 1 Processor -- start 2019-12-05T06:27:26.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.386+0000 7f5b0e13f700 1 -- start start 2019-12-05T06:27:26.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.388+0000 7f5b0e13f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:26.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.388+0000 7f5b0e13f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b0809ef60 con 0x7f5b0815b110 2019-12-05T06:27:26.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.388+0000 7f5b0e13f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b08096e80 con 0x7f5b080829b0 2019-12-05T06:27:26.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.388+0000 7f5b0e13f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b0809f410 con 0x7f5b0809aa50 2019-12-05T06:27:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.388+0000 7f5b0cedd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5b0815b110 0x7f5b0815d510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38834/0 (socket says 172.21.15.16:38834) 2019-12-05T06:27:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.388+0000 7f5b0cedd700 1 -- 172.21.15.16:0/132639157 learned_addr learned my addr 172.21.15.16:0/132639157 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.388+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.389+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702515257 0 0) 0x7f5b00001240 con 0x7f5b0815b110 2019-12-05T06:27:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.389+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aec002e80 con 0x7f5b0815b110 2019-12-05T06:27:26.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.389+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088784307 0 0) 0x7f5af8007900 con 0x7f5b080829b0 2019-12-05T06:27:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.389+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5aec004170 con 0x7f5b080829b0 2019-12-05T06:27:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.389+0000 7f5b077fe700 1 -- 172.21.15.16:0/132639157 >> v1:172.21.15.76:6789/0 conn(0x7f5b080829b0 legacy=0x7f5b08099a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.389+0000 7f5b077fe700 1 -- 172.21.15.16:0/132639157 >> v1:172.21.15.16:6789/0 conn(0x7f5b0815b110 legacy=0x7f5b0815d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.389+0000 7f5b077fe700 1 -- 172.21.15.16:0/132639157 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b08099180 con 0x7f5b0809aa50 2019-12-05T06:27:26.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.389+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.390+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.390+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=READY pgs=198 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:26.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.390+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.390+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af0009640 con 0x7f5b0809aa50 2019-12-05T06:27:26.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.390+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.390+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.390+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af0000940 con 0x7f5b0809aa50 2019-12-05T06:27:26.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.390+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af000a0e0 con 0x7f5b0809aa50 2019-12-05T06:27:26.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.390+0000 7f5b0e13f700 1 -- 172.21.15.16:0/132639157 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b08099180 con 0x7f5b0809aa50 2019-12-05T06:27:26.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.390+0000 7f5b0e13f700 1 -- 172.21.15.16:0/132639157 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b08099180 con 0x7f5b0809aa50 2019-12-05T06:27:26.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.391+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.392+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5af0009a70 con 0x7f5b0809aa50 2019-12-05T06:27:26.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.392+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.392+0000 7f5b04ff9700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aec023680 0x7f5aec025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:26.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.392+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f5af0045de0 con 0x7f5b0809aa50 2019-12-05T06:27:26.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.392+0000 7f5b07fff700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aec023680 0x7f5aec025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:26.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.393+0000 7f5b07fff700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aec023680 0x7f5aec025af0 crc :-1 s=READY pgs=346 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:26.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.406+0000 7f5b0e13f700 1 -- 172.21.15.16:0/132639157 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b08099180 con 0x7f5b0809aa50 2019-12-05T06:27:26.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.415+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.415+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5af0009840 con 0x7f5b0809aa50 2019-12-05T06:27:26.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.678+0000 7f5b0e13f700 1 -- 172.21.15.16:0/132639157 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b08099180 con 0x7f5b0809aa50 2019-12-05T06:27:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.679+0000 7f5b077fe700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=THROTTLE_DONE pgs=198 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:26.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.679+0000 7f5b04ff9700 1 -- 172.21.15.16:0/132639157 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 99 v99) v1 ==== 94+0+1775 (secure 0 0 0) 0x7f5af0009a50 con 0x7f5b0809aa50 2019-12-05T06:27:26.669 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:26.670 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":99,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":99,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:26.194054+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:26.670 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 99 2019-12-05T06:27:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.684+0000 7f5b0e13f700 1 -- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aec023680 msgr2=0x7f5aec025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:26.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.684+0000 7f5b0e13f700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aec023680 0x7f5aec025af0 crc :-1 s=READY pgs=346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.684+0000 7f5b0e13f700 1 -- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 msgr2=0x7f5b0809a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.684+0000 7f5b0e13f700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 secure :-1 s=READY pgs=198 cs=0 l=1 rx=0x7f5af0010390 tx=0x7f5af0010730).stop 2019-12-05T06:27:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.684+0000 7f5b0e13f700 1 -- 172.21.15.16:0/132639157 shutdown_connections 2019-12-05T06:27:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.684+0000 7f5b0e13f700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5aec023680 0x7f5aec025af0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:26.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.684+0000 7f5b0e13f700 1 --2- 172.21.15.16:0/132639157 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5b0809aa50 0x7f5b0809a280 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.684+0000 7f5b0e13f700 1 -- 172.21.15.16:0/132639157 shutdown_connections 2019-12-05T06:27:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.684+0000 7f5b0e13f700 1 -- 172.21.15.16:0/132639157 wait complete. 2019-12-05T06:27:26.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:26.684+0000 7f5b0e13f700 1 -- 172.21.15.16:0/132639157 >> 172.21.15.16:0/132639157 conn(0x7f5b080714f0 msgr2=0x7f5b0809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:26.682 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:27:28.688 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.977+0000 7fdf716b4700 1 Processor -- start 2019-12-05T06:27:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.977+0000 7fdf716b4700 1 -- start start 2019-12-05T06:27:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.979+0000 7fdf716b4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdf6c08d2c0 0x7fdf6c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.979+0000 7fdf716b4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c0958e0 con 0x7fdf6c08c6d0 2019-12-05T06:27:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.979+0000 7fdf716b4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c0629e0 con 0x7fdf6c08fef0 2019-12-05T06:27:28.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.979+0000 7fdf716b4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6c08df90 con 0x7fdf6c08d2c0 2019-12-05T06:27:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.979+0000 7fdf6b7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdf6c08c6d0 0x7fdf6c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38844/0 (socket says 172.21.15.16:38844) 2019-12-05T06:27:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.979+0000 7fdf6b7fe700 1 -- 172.21.15.16:0/240597076 learned_addr learned my addr 172.21.15.16:0/240597076 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.979+0000 7fdf6affd700 1 --2- 172.21.15.16:0/240597076 >> v2:172.21.15.76:3300/0 conn(0x7fdf6c08d2c0 0x7fdf6c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.980+0000 7fdf6a7fc700 1 -- 172.21.15.16:0/240597076 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817916047 0 0) 0x7fdf5c001150 con 0x7fdf6c08c6d0 2019-12-05T06:27:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.980+0000 7fdf6a7fc700 1 -- 172.21.15.16:0/240597076 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4c002e80 con 0x7fdf6c08c6d0 2019-12-05T06:27:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.980+0000 7fdf6affd700 1 -- 172.21.15.16:0/240597076 >> v1:172.21.15.76:6789/0 conn(0x7fdf6c08fef0 legacy=0x7fdf6c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:28.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.980+0000 7fdf6a7fc700 1 -- 172.21.15.16:0/240597076 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3130847589 0 0) 0x7fdf5c001130 con 0x7fdf6c08c6d0 2019-12-05T06:27:28.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.980+0000 7fdf6affd700 1 -- 172.21.15.16:0/240597076 >> v1:172.21.15.16:6789/0 conn(0x7fdf6c08c6d0 legacy=0x7fdf6c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:28.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.980+0000 7fdf6affd700 1 -- 172.21.15.16:0/240597076 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c08e620 con 0x7fdf6c08d2c0 2019-12-05T06:27:28.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf6affd700 1 --2- 172.21.15.16:0/240597076 >> v2:172.21.15.76:3300/0 conn(0x7fdf6c08d2c0 0x7fdf6c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdf5400b2d0 tx=0x7fdf5400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf6affd700 1 --2- 172.21.15.16:0/240597076 >> v2:172.21.15.76:3300/0 conn(0x7fdf6c08d2c0 0x7fdf6c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdf5400b2d0 tx=0x7fdf5400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf6affd700 1 --2- 172.21.15.16:0/240597076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c08d2c0 0x7fdf6c08ef20 secure :-1 s=READY pgs=576 cs=0 l=1 rx=0x7fdf5400b2d0 tx=0x7fdf5400bae0).ready entity=mon.1 client_cookie=cd8181992cc42483 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:28.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf6affd700 1 --2- 172.21.15.16:0/240597076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c08d2c0 0x7fdf6c08ef20 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7fdf5400b2d0 tx=0x7fdf5400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf6affd700 1 --2- 172.21.15.16:0/240597076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c08d2c0 0x7fdf6c08ef20 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7fdf5400b2d0 tx=0x7fdf5400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf6a7fc700 1 -- 172.21.15.16:0/240597076 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf54000ce0 con 0x7fdf6c08d2c0 2019-12-05T06:27:28.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf6affd700 1 --2- 172.21.15.16:0/240597076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c08d2c0 0x7fdf6c08ef20 secure :-1 s=THROTTLE_DONE pgs=576 cs=0 l=1 rx=0x7fdf5400b2d0 tx=0x7fdf5400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf6a7fc700 1 -- 172.21.15.16:0/240597076 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf54009c60 con 0x7fdf6c08d2c0 2019-12-05T06:27:28.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf6a7fc700 1 -- 172.21.15.16:0/240597076 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf5400a3d0 con 0x7fdf6c08d2c0 2019-12-05T06:27:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf716b4700 1 -- 172.21.15.16:0/240597076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c08d2c0 msgr2=0x7fdf6c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.981+0000 7fdf716b4700 1 --2- 172.21.15.16:0/240597076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c08d2c0 0x7fdf6c08ef20 secure :-1 s=READY pgs=576 cs=0 l=1 rx=0x7fdf5400b2d0 tx=0x7fdf5400bae0).stop 2019-12-05T06:27:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.982+0000 7fdf716b4700 1 -- 172.21.15.16:0/240597076 shutdown_connections 2019-12-05T06:27:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.982+0000 7fdf716b4700 1 --2- 172.21.15.16:0/240597076 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c08d2c0 0x7fdf6c08ef20 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.983+0000 7fdf716b4700 1 -- 172.21.15.16:0/240597076 shutdown_connections 2019-12-05T06:27:28.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.983+0000 7fdf716b4700 1 -- 172.21.15.16:0/240597076 wait complete. 2019-12-05T06:27:28.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.983+0000 7fdf716b4700 1 -- 172.21.15.16:0/240597076 >> 172.21.15.16:0/240597076 conn(0x7fdf6c061940 msgr2=0x7fdf6c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:28.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.987+0000 7fdf716b4700 1 Processor -- start 2019-12-05T06:27:28.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.988+0000 7fdf716b4700 1 -- start start 2019-12-05T06:27:28.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.989+0000 7fdf716b4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdf6c09e7d0 0x7fdf6c09e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:28.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.989+0000 7fdf716b4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c12cb90 con 0x7fdf6c156e70 2019-12-05T06:27:28.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.989+0000 7fdf716b4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6c12d040 con 0x7fdf6c1302f0 2019-12-05T06:27:28.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.989+0000 7fdf716b4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf6c131ff0 con 0x7fdf6c09e7d0 2019-12-05T06:27:28.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.990+0000 7fdf6bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdf6c156e70 0x7fdf6c159270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38852/0 (socket says 172.21.15.16:38852) 2019-12-05T06:27:28.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.990+0000 7fdf6bfff700 1 -- 172.21.15.16:0/3476673489 learned_addr learned my addr 172.21.15.16:0/3476673489 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:28.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.990+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> v2:172.21.15.76:3300/0 conn(0x7fdf6c09e7d0 0x7fdf6c09e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:28.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.990+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1679975778 0 0) 0x7fdf64001240 con 0x7fdf6c156e70 2019-12-05T06:27:28.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.990+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf44002e80 con 0x7fdf6c156e70 2019-12-05T06:27:28.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.990+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439508928 0 0) 0x7fdf5c001100 con 0x7fdf6c1302f0 2019-12-05T06:27:28.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.990+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf44004220 con 0x7fdf6c1302f0 2019-12-05T06:27:28.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.990+0000 7fdf6affd700 1 -- 172.21.15.16:0/3476673489 >> v1:172.21.15.76:6789/0 conn(0x7fdf6c1302f0 legacy=0x7fdf6c09d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:28.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.990+0000 7fdf6affd700 1 -- 172.21.15.16:0/3476673489 >> v1:172.21.15.16:6789/0 conn(0x7fdf6c156e70 legacy=0x7fdf6c159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:28.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.990+0000 7fdf6affd700 1 -- 172.21.15.16:0/3476673489 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf6c132680 con 0x7fdf6c09e7d0 2019-12-05T06:27:28.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> v2:172.21.15.76:3300/0 conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> v2:172.21.15.76:3300/0 conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=READY pgs=578 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:28.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf54000cc0 con 0x7fdf6c09e7d0 2019-12-05T06:27:28.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf54009d00 con 0x7fdf6c09e7d0 2019-12-05T06:27:28.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf5400a3d0 con 0x7fdf6c09e7d0 2019-12-05T06:27:28.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf716b4700 1 -- 172.21.15.16:0/3476673489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf6c132680 con 0x7fdf6c09e7d0 2019-12-05T06:27:28.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.991+0000 7fdf716b4700 1 -- 172.21.15.16:0/3476673489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6c132680 con 0x7fdf6c09e7d0 2019-12-05T06:27:28.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.992+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.993+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdf54009900 con 0x7fdf6c09e7d0 2019-12-05T06:27:28.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.993+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:28.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.994+0000 7fdf63fff700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf44023650 0x7fdf44025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:28.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.994+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fdf54046160 con 0x7fdf6c09e7d0 2019-12-05T06:27:29.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.994+0000 7fdf6b7fe700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf44023650 0x7fdf44025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:29.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:28.994+0000 7fdf6b7fe700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf44023650 0x7fdf44025ae0 crc :-1 s=READY pgs=347 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:29.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.008+0000 7fdf716b4700 1 -- 172.21.15.16:0/3476673489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6c132680 con 0x7fdf6c09e7d0 2019-12-05T06:27:29.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.018+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:29.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.018+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdf5400a4d0 con 0x7fdf6c09e7d0 2019-12-05T06:27:29.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.269+0000 7fdf716b4700 1 -- 172.21.15.16:0/3476673489 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf6c132680 con 0x7fdf6c09e7d0 2019-12-05T06:27:29.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.269+0000 7fdf6affd700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=THROTTLE_DONE pgs=578 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.269+0000 7fdf63fff700 1 -- 172.21.15.16:0/3476673489 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fdf54009900 con 0x7fdf6c09e7d0 2019-12-05T06:27:29.258 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 101 2019-12-05T06:27:29.258 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:29.258 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:29.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.272+0000 7fdf716b4700 1 -- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf44023650 msgr2=0x7fdf44025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:29.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.272+0000 7fdf716b4700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf44023650 0x7fdf44025ae0 crc :-1 s=READY pgs=347 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:29.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.272+0000 7fdf716b4700 1 -- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 msgr2=0x7fdf6c09e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:29.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.272+0000 7fdf716b4700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 0x7fdf6c09e000 secure :-1 s=READY pgs=578 cs=0 l=1 rx=0x7fdf540106f0 tx=0x7fdf54010a90).stop 2019-12-05T06:27:29.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.272+0000 7fdf716b4700 1 -- 172.21.15.16:0/3476673489 shutdown_connections 2019-12-05T06:27:29.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.272+0000 7fdf716b4700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf44023650 0x7fdf44025ae0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:29.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.272+0000 7fdf716b4700 1 --2- 172.21.15.16:0/3476673489 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf6c09e7d0 0x7fdf6c09e000 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:29.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.273+0000 7fdf716b4700 1 -- 172.21.15.16:0/3476673489 shutdown_connections 2019-12-05T06:27:29.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.273+0000 7fdf716b4700 1 -- 172.21.15.16:0/3476673489 wait complete. 2019-12-05T06:27:29.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:29.273+0000 7fdf716b4700 1 -- 172.21.15.16:0/3476673489 >> 172.21.15.16:0/3476673489 conn(0x7fdf6c0a2850 msgr2=0x7fdf6c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:29.271 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5135 2019-12-05T06:27:29.272 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.b 2019-12-05T06:27:34.417 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:27:34.527 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:27:38.272 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:27:38.273 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:27:38.273 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:27:38.278 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:27:38.278 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:38.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.436+0000 7f49d9f64700 1 Processor -- start 2019-12-05T06:27:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.436+0000 7f49d9f64700 1 -- start start 2019-12-05T06:27:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49d9f64700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49d408c6d0 0x7f49d408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49d9f64700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49d408fef0 0x7f49d408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49d9f64700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d40958e0 con 0x7f49d408ef20 2019-12-05T06:27:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49d9f64700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49d40a3f70 con 0x7f49d408fef0 2019-12-05T06:27:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49d9f64700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d408d830 con 0x7f49d408c6d0 2019-12-05T06:27:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49cbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49d408fef0 0x7f49d408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:38.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49cbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f49d408fef0 0x7f49d408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53762/0 (socket says 172.21.15.16:53762) 2019-12-05T06:27:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49cbfff700 1 -- 172.21.15.16:0/2632278176 learned_addr learned my addr 172.21.15.16:0/2632278176 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49d3fff700 1 --2- 172.21.15.16:0/2632278176 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49d408c6d0 0x7f49d408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:38.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49cbfff700 1 -- 172.21.15.16:0/2632278176 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49d408c6d0 msgr2=0x7f49d408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49cbfff700 1 --2- 172.21.15.16:0/2632278176 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49d408c6d0 0x7f49d408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49cbfff700 1 -- 172.21.15.16:0/2632278176 >> v1:172.21.15.76:6789/0 conn(0x7f49d408ef20 legacy=0x7f49d408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49cbfff700 1 -- 172.21.15.16:0/2632278176 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d408d540 con 0x7f49d408fef0 2019-12-05T06:27:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49d3fff700 1 --2- 172.21.15.16:0/2632278176 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49d408c6d0 0x7f49d408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:27:38.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49cbfff700 1 --2- 172.21.15.16:0/2632278176 >> v2:172.21.15.16:3300/0 conn(0x7f49d408fef0 0x7f49d408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f49bc00f740 tx=0x7f49bc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49cbfff700 1 --2- 172.21.15.16:0/2632278176 >> v2:172.21.15.16:3300/0 conn(0x7f49d408fef0 0x7f49d408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f49bc00f740 tx=0x7f49bc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.437+0000 7f49cbfff700 1 --2- 172.21.15.16:0/2632278176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49d408fef0 0x7f49d408d000 secure :-1 s=READY pgs=795 cs=0 l=1 rx=0x7f49bc00f740 tx=0x7f49bc00fae0).ready entity=mon.0 client_cookie=3ebc323e701d4537 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:38.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.438+0000 7f49cbfff700 1 --2- 172.21.15.16:0/2632278176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49d408fef0 0x7f49d408d000 secure :-1 s=THROTTLE_DONE pgs=795 cs=0 l=1 rx=0x7f49bc00f740 tx=0x7f49bc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.438+0000 7f49d37fe700 1 -- 172.21.15.16:0/2632278176 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49bc000960 con 0x7f49d408fef0 2019-12-05T06:27:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.438+0000 7f49cbfff700 1 --2- 172.21.15.16:0/2632278176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49d408fef0 0x7f49d408d000 secure :-1 s=THROTTLE_DONE pgs=795 cs=0 l=1 rx=0x7f49bc00f740 tx=0x7f49bc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.438+0000 7f49cbfff700 1 --2- 172.21.15.16:0/2632278176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49d408fef0 0x7f49d408d000 secure :-1 s=THROTTLE_DONE pgs=795 cs=0 l=1 rx=0x7f49bc00f740 tx=0x7f49bc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.438+0000 7f49d37fe700 1 -- 172.21.15.16:0/2632278176 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49bc000c90 con 0x7f49d408fef0 2019-12-05T06:27:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.438+0000 7f49d37fe700 1 -- 172.21.15.16:0/2632278176 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49bc0013a0 con 0x7f49d408fef0 2019-12-05T06:27:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.438+0000 7f49d9f64700 1 -- 172.21.15.16:0/2632278176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49d408fef0 msgr2=0x7f49d408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:38.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.438+0000 7f49d9f64700 1 --2- 172.21.15.16:0/2632278176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49d408fef0 0x7f49d408d000 secure :-1 s=READY pgs=795 cs=0 l=1 rx=0x7f49bc00f740 tx=0x7f49bc00fae0).stop 2019-12-05T06:27:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.439+0000 7f49d9f64700 1 -- 172.21.15.16:0/2632278176 shutdown_connections 2019-12-05T06:27:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.439+0000 7f49d9f64700 1 --2- 172.21.15.16:0/2632278176 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49d408fef0 0x7f49d408d000 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:38.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.439+0000 7f49d9f64700 1 --2- 172.21.15.16:0/2632278176 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49d408c6d0 0x7f49d408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:38.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.439+0000 7f49d9f64700 1 -- 172.21.15.16:0/2632278176 shutdown_connections 2019-12-05T06:27:38.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.439+0000 7f49d9f64700 1 -- 172.21.15.16:0/2632278176 wait complete. 2019-12-05T06:27:38.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.439+0000 7f49d9f64700 1 -- 172.21.15.16:0/2632278176 >> 172.21.15.16:0/2632278176 conn(0x7f49d4061940 msgr2=0x7f49d4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:38.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.441+0000 7f49d9f64700 1 Processor -- start 2019-12-05T06:27:38.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.441+0000 7f49d9f64700 1 -- start start 2019-12-05T06:27:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.442+0000 7f49d9f64700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49d409a6d0 0x7f49d4157340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.442+0000 7f49d9f64700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d409ab20 con 0x7f49d407bc30 2019-12-05T06:27:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.442+0000 7f49d9f64700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49d407da20 con 0x7f49d409aea0 2019-12-05T06:27:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.442+0000 7f49d9f64700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49d4098f40 con 0x7f49d409a6d0 2019-12-05T06:27:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.442+0000 7f49d3fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f49d407bc30 0x7f49d4099ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38940/0 (socket says 172.21.15.16:38940) 2019-12-05T06:27:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.442+0000 7f49d3fff700 1 -- 172.21.15.16:0/4234356369 learned_addr learned my addr 172.21.15.16:0/4234356369 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.442+0000 7f49d8d02700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49d409a6d0 0x7f49d4157340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.442+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1163248242 0 0) 0x7f49c4009da0 con 0x7f49d407bc30 2019-12-05T06:27:38.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.442+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49b8002e80 con 0x7f49d407bc30 2019-12-05T06:27:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1241567669 0 0) 0x7f49c4009b20 con 0x7f49d407bc30 2019-12-05T06:27:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49b80032e0 con 0x7f49d407bc30 2019-12-05T06:27:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f49c4000fb0 con 0x7f49d407bc30 2019-12-05T06:27:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49d409a6d0 msgr2=0x7f49d4157340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49d409a6d0 0x7f49d4157340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 >> v1:172.21.15.76:6789/0 conn(0x7f49d409aea0 legacy=0x7f49d4153b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:38.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 >> v1:172.21.15.16:6789/0 conn(0x7f49d407bc30 legacy=0x7f49d4099ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:38.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:38.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b8003ca0 0x7f49b8004140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:38.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49b8004950 0x7f49b8004de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:38.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b800bbf0 con 0x7f49b8003330 2019-12-05T06:27:38.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49b800bdf0 con 0x7f49b8003ca0 2019-12-05T06:27:38.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49b800c020 con 0x7f49b8004950 2019-12-05T06:27:38.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:38.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49cbfff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b8003ca0 0x7f49b8004140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:38.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d3fff700 1 -- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49b8004950 msgr2=0x7f49b8004de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:38.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f49b8004950 0x7f49b8004de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:38.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d3fff700 1 -- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b8003ca0 msgr2=0x7f49b8004140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:38.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f49b8003ca0 0x7f49b8004140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:38.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d3fff700 1 -- 172.21.15.16:0/4234356369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49b800c2d0 con 0x7f49b8003330 2019-12-05T06:27:38.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d9f64700 1 -- 172.21.15.16:0/4234356369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d40995d0 con 0x7f49b8003330 2019-12-05T06:27:38.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d9f64700 1 -- 172.21.15.16:0/4234356369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d407da20 con 0x7f49b8003330 2019-12-05T06:27:38.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.443+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.444+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=READY pgs=797 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:38.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.444+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=THROTTLE_DONE pgs=797 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.444+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=THROTTLE_DONE pgs=797 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.444+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=THROTTLE_DONE pgs=797 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.444+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49c4009ab0 con 0x7f49b8003330 2019-12-05T06:27:38.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.444+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49c40017e0 con 0x7f49b8003330 2019-12-05T06:27:38.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.444+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49c4001e70 con 0x7f49b8003330 2019-12-05T06:27:38.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.445+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=THROTTLE_DONE pgs=797 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.445+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f49c40015f0 con 0x7f49b8003330 2019-12-05T06:27:38.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.445+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=THROTTLE_DONE pgs=797 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.446+0000 7f49d17fa700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f49b802baa0 0x7f49b802ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.446+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f49c4047f10 con 0x7f49b8003330 2019-12-05T06:27:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.446+0000 7f49cbfff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f49b802baa0 0x7f49b802ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.446+0000 7f49cbfff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f49b802baa0 0x7f49b802ded0 crc :-1 s=READY pgs=348 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.451+0000 7f49d9f64700 1 -- 172.21.15.16:0/4234356369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49ac002e60 con 0x7f49b8003330 2019-12-05T06:27:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.454+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=THROTTLE_DONE pgs=797 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.454+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f49c40013e0 con 0x7f49b8003330 2019-12-05T06:27:38.469 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:27:38.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.650+0000 7f49d9f64700 1 -- 172.21.15.16:0/4234356369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49ac001d10 con 0x7f49b8003330 2019-12-05T06:27:38.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.650+0000 7f49d3fff700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=THROTTLE_DONE pgs=797 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:38.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.650+0000 7f49d17fa700 1 -- 172.21.15.16:0/4234356369 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f49c40015f0 con 0x7f49b8003330 2019-12-05T06:27:38.639 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 101 2019-12-05T06:27:38.639 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:38.639 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:38.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.654+0000 7f49d9f64700 1 -- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f49b802baa0 msgr2=0x7f49b802ded0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:38.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.654+0000 7f49d9f64700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f49b802baa0 0x7f49b802ded0 crc :-1 s=READY pgs=348 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.654+0000 7f49d9f64700 1 -- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 msgr2=0x7f49b8003760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.654+0000 7f49d9f64700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 secure :-1 s=READY pgs=797 cs=0 l=1 rx=0x7f49c4000a50 tx=0x7f49c4012810).stop 2019-12-05T06:27:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.654+0000 7f49d8d02700 1 -- 172.21.15.16:0/4234356369 reap_dead start 2019-12-05T06:27:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.654+0000 7f49d9f64700 1 -- 172.21.15.16:0/4234356369 shutdown_connections 2019-12-05T06:27:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.654+0000 7f49d9f64700 1 --2- 172.21.15.16:0/4234356369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f49b8003330 0x7f49b8003760 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:38.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.654+0000 7f49d9f64700 1 -- 172.21.15.16:0/4234356369 shutdown_connections 2019-12-05T06:27:38.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.654+0000 7f49d9f64700 1 -- 172.21.15.16:0/4234356369 wait complete. 2019-12-05T06:27:38.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:38.654+0000 7f49d9f64700 1 -- 172.21.15.16:0/4234356369 >> 172.21.15.16:0/4234356369 conn(0x7f49d4082220 msgr2=0x7f49d406d380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:38.656 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:27:40.656 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.940+0000 7f69b352c700 1 Processor -- start 2019-12-05T06:27:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.941+0000 7f69b352c700 1 -- start start 2019-12-05T06:27:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.942+0000 7f69b352c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69ac08c6d0 0x7f69ac08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.942+0000 7f69b352c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f69ac08f720 0x7f69ac08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.942+0000 7f69b352c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ac08fb80 con 0x7f69ac08fef0 2019-12-05T06:27:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.942+0000 7f69b352c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ac08df90 con 0x7f69ac08c6d0 2019-12-05T06:27:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.942+0000 7f69b352c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ac0a3f70 con 0x7f69ac08f720 2019-12-05T06:27:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.943+0000 7f69b1ac9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69ac08c6d0 0x7f69ac08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.943+0000 7f69b12c8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f69ac08fef0 0x7f69ac08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38964/0 (socket says 172.21.15.16:38964) 2019-12-05T06:27:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.943+0000 7f69b12c8700 1 -- 172.21.15.16:0/3727400514 learned_addr learned my addr 172.21.15.16:0/3727400514 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.943+0000 7f69b22ca700 1 --2- 172.21.15.16:0/3727400514 >> v2:172.21.15.76:3300/0 conn(0x7f69ac08f720 0x7f69ac08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.943+0000 7f69b0ac7700 1 -- 172.21.15.16:0/3727400514 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1406021500 0 0) 0x7f69a0001070 con 0x7f69ac08fef0 2019-12-05T06:27:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.943+0000 7f69b0ac7700 1 -- 172.21.15.16:0/3727400514 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6994002e80 con 0x7f69ac08fef0 2019-12-05T06:27:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.943+0000 7f69b1ac9700 1 -- 172.21.15.16:0/3727400514 >> v2:172.21.15.76:3300/0 conn(0x7f69ac08f720 msgr2=0x7f69ac08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.943+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/3727400514 >> v2:172.21.15.76:3300/0 conn(0x7f69ac08f720 0x7f69ac08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.943+0000 7f69b1ac9700 1 -- 172.21.15.16:0/3727400514 >> v1:172.21.15.16:6789/0 conn(0x7f69ac08fef0 legacy=0x7f69ac08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.944+0000 7f69b1ac9700 1 -- 172.21.15.16:0/3727400514 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac08d540 con 0x7f69ac08c6d0 2019-12-05T06:27:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.944+0000 7f69b22ca700 1 --2- 172.21.15.16:0/3727400514 >> v2:172.21.15.76:3300/0 conn(0x7f69ac08f720 0x7f69ac08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:27:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.944+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/3727400514 >> v2:172.21.15.16:3300/0 conn(0x7f69ac08c6d0 0x7f69ac08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f699c00af50 tx=0x7f699c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.944+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/3727400514 >> v2:172.21.15.16:3300/0 conn(0x7f69ac08c6d0 0x7f69ac08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f699c00af50 tx=0x7f699c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.944+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/3727400514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac08c6d0 0x7f69ac08cac0 secure :-1 s=READY pgs=801 cs=0 l=1 rx=0x7f699c00af50 tx=0x7f699c00b760).ready entity=mon.0 client_cookie=e260f1688b91fc0d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.944+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/3727400514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac08c6d0 0x7f69ac08cac0 secure :-1 s=THROTTLE_DONE pgs=801 cs=0 l=1 rx=0x7f699c00af50 tx=0x7f699c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.944+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/3727400514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac08c6d0 0x7f69ac08cac0 secure :-1 s=THROTTLE_DONE pgs=801 cs=0 l=1 rx=0x7f699c00af50 tx=0x7f699c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.944+0000 7f69b0ac7700 1 -- 172.21.15.16:0/3727400514 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f699c007210 con 0x7f69ac08c6d0 2019-12-05T06:27:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.944+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/3727400514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac08c6d0 0x7f69ac08cac0 secure :-1 s=THROTTLE_DONE pgs=801 cs=0 l=1 rx=0x7f699c00af50 tx=0x7f699c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.945+0000 7f69b0ac7700 1 -- 172.21.15.16:0/3727400514 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c0099b0 con 0x7f69ac08c6d0 2019-12-05T06:27:40.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.945+0000 7f69b0ac7700 1 -- 172.21.15.16:0/3727400514 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f699c00a080 con 0x7f69ac08c6d0 2019-12-05T06:27:40.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.945+0000 7f69b352c700 1 -- 172.21.15.16:0/3727400514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac08c6d0 msgr2=0x7f69ac08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:40.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.945+0000 7f69b352c700 1 --2- 172.21.15.16:0/3727400514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac08c6d0 0x7f69ac08cac0 secure :-1 s=READY pgs=801 cs=0 l=1 rx=0x7f699c00af50 tx=0x7f699c00b760).stop 2019-12-05T06:27:40.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.946+0000 7f69b352c700 1 -- 172.21.15.16:0/3727400514 shutdown_connections 2019-12-05T06:27:40.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.946+0000 7f69b352c700 1 --2- 172.21.15.16:0/3727400514 >> v2:172.21.15.76:3300/0 conn(0x7f69ac08f720 0x7f69ac08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.946+0000 7f69b352c700 1 --2- 172.21.15.16:0/3727400514 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac08c6d0 0x7f69ac08cac0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.947+0000 7f69b352c700 1 -- 172.21.15.16:0/3727400514 shutdown_connections 2019-12-05T06:27:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.947+0000 7f69b352c700 1 -- 172.21.15.16:0/3727400514 wait complete. 2019-12-05T06:27:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.947+0000 7f69b352c700 1 -- 172.21.15.16:0/3727400514 >> 172.21.15.16:0/3727400514 conn(0x7f69ac061940 msgr2=0x7f69ac096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:40.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.952+0000 7f69b352c700 1 Processor -- start 2019-12-05T06:27:40.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.952+0000 7f69b352c700 1 -- start start 2019-12-05T06:27:40.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.953+0000 7f69b352c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69ac132630 0x7f69ac132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:40.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.953+0000 7f69b352c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f69ac099eb0 0x7f69ac09a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:40.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.953+0000 7f69b352c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ac06e7c0 con 0x7f69ac09ae80 2019-12-05T06:27:40.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.953+0000 7f69b352c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ac098f20 con 0x7f69ac132630 2019-12-05T06:27:40.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.953+0000 7f69b352c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ac134390 con 0x7f69ac099eb0 2019-12-05T06:27:40.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.953+0000 7f69b1ac9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69ac132630 0x7f69ac132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:40.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.954+0000 7f69b1ac9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f69ac132630 0x7f69ac132a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53802/0 (socket says 172.21.15.16:53802) 2019-12-05T06:27:40.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.954+0000 7f69b1ac9700 1 -- 172.21.15.16:0/2188809654 learned_addr learned my addr 172.21.15.16:0/2188809654 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:40.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.954+0000 7f69b22ca700 1 --2- 172.21.15.16:0/2188809654 >> v2:172.21.15.76:3300/0 conn(0x7f69ac099eb0 0x7f69ac09a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:40.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.954+0000 7f69b1ac9700 1 -- 172.21.15.16:0/2188809654 >> v2:172.21.15.76:3300/0 conn(0x7f69ac099eb0 msgr2=0x7f69ac09a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:40.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.954+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> v2:172.21.15.76:3300/0 conn(0x7f69ac099eb0 0x7f69ac09a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:40.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.954+0000 7f69b1ac9700 1 -- 172.21.15.16:0/2188809654 >> v1:172.21.15.76:6789/0 conn(0x7f69ac09ae80 legacy=0x7f69ac132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:40.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.954+0000 7f69b1ac9700 1 -- 172.21.15.16:0/2188809654 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac0995b0 con 0x7f69ac132630 2019-12-05T06:27:40.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.954+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> v2:172.21.15.16:3300/0 conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.954+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> v2:172.21.15.16:3300/0 conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.954+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=READY pgs=802 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:40.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.955+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.955+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.955+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.955+0000 7f69aa7fc700 1 -- 172.21.15.16:0/2188809654 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f699c00aca0 con 0x7f69ac132630 2019-12-05T06:27:40.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.955+0000 7f69aa7fc700 1 -- 172.21.15.16:0/2188809654 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c009980 con 0x7f69ac132630 2019-12-05T06:27:40.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.955+0000 7f69b352c700 1 -- 172.21.15.16:0/2188809654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac0995b0 con 0x7f69ac132630 2019-12-05T06:27:40.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.955+0000 7f69aa7fc700 1 -- 172.21.15.16:0/2188809654 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f699c00a0d0 con 0x7f69ac132630 2019-12-05T06:27:40.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.955+0000 7f69b352c700 1 -- 172.21.15.16:0/2188809654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac0995b0 con 0x7f69ac132630 2019-12-05T06:27:40.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.956+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.957+0000 7f69aa7fc700 1 -- 172.21.15.16:0/2188809654 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f699c009b70 con 0x7f69ac132630 2019-12-05T06:27:40.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.957+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.957+0000 7f69aa7fc700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f698c023630 0x7f698c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:40.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.957+0000 7f69aa7fc700 1 -- 172.21.15.16:0/2188809654 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f699c089ef0 con 0x7f69ac132630 2019-12-05T06:27:40.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.957+0000 7f69b12c8700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f698c023630 0x7f698c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:40.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.958+0000 7f69b12c8700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f698c023630 0x7f698c025ac0 crc :-1 s=READY pgs=349 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:40.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.972+0000 7f69b352c700 1 -- 172.21.15.16:0/2188809654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac0995b0 con 0x7f69ac132630 2019-12-05T06:27:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.981+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:40.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:40.982+0000 7f69aa7fc700 1 -- 172.21.15.16:0/2188809654 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f699c009580 con 0x7f69ac132630 2019-12-05T06:27:41.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.250+0000 7f69b352c700 1 -- 172.21.15.16:0/2188809654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f69ac0995b0 con 0x7f69ac132630 2019-12-05T06:27:41.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.251+0000 7f69b1ac9700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=THROTTLE_DONE pgs=802 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:41.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.251+0000 7f69aa7fc700 1 -- 172.21.15.16:0/2188809654 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 101 v101) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f699c009880 con 0x7f69ac132630 2019-12-05T06:27:41.240 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 101 2019-12-05T06:27:41.241 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:41.241 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":101,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:41.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 -- 172.21.15.16:0/2188809654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f698c023630 msgr2=0x7f698c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:41.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f698c023630 0x7f698c025ac0 crc :-1 s=READY pgs=349 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:41.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 -- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 msgr2=0x7f69ac132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:41.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 0x7f69ac132a20 secure :-1 s=READY pgs=802 cs=0 l=1 rx=0x7f699c054490 tx=0x7f699c00b230).stop 2019-12-05T06:27:41.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 -- 172.21.15.16:0/2188809654 shutdown_connections 2019-12-05T06:27:41.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 --2- 172.21.15.16:0/2188809654 >> v2:172.21.15.76:3300/0 conn(0x7f69ac099eb0 0x7f69ac09a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:41.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f698c023630 0x7f698c025ac0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:41.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 --2- 172.21.15.16:0/2188809654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f69ac132630 0x7f69ac132a20 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:41.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 -- 172.21.15.16:0/2188809654 shutdown_connections 2019-12-05T06:27:41.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 -- 172.21.15.16:0/2188809654 wait complete. 2019-12-05T06:27:41.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:41.254+0000 7f69b352c700 1 -- 172.21.15.16:0/2188809654 >> 172.21.15.16:0/2188809654 conn(0x7f69ac0714f0 msgr2=0x7f69ac09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:41.252 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:27:43.253 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:43.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.408+0000 7f13f163e700 1 Processor -- start 2019-12-05T06:27:43.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.408+0000 7f13f163e700 1 -- start start 2019-12-05T06:27:43.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.409+0000 7f13f163e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 0x7f13ec08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:43.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.409+0000 7f13f163e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f13ec08fef0 0x7f13ec08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:43.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.409+0000 7f13f163e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec0958e0 con 0x7f13ec08ef20 2019-12-05T06:27:43.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.409+0000 7f13f163e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ec0a3f70 con 0x7f13ec08fef0 2019-12-05T06:27:43.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.409+0000 7f13f163e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec08d830 con 0x7f13ec08c6d0 2019-12-05T06:27:43.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.409+0000 7f13ebfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f13ec08ef20 0x7f13ec08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38990/0 (socket says 172.21.15.16:38990) 2019-12-05T06:27:43.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.409+0000 7f13ebfff700 1 -- 172.21.15.16:0/3773658602 learned_addr learned my addr 172.21.15.16:0/3773658602 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:43.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.409+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3773658602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 0x7f13ec08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:43.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.409+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3773658602 >> v2:172.21.15.76:3300/0 conn(0x7f13ec08fef0 0x7f13ec08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:43.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3773658602 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216236758 0 0) 0x7f13e4001150 con 0x7f13ec08ef20 2019-12-05T06:27:43.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3773658602 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13d4002e80 con 0x7f13ec08ef20 2019-12-05T06:27:43.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3773658602 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3964891600 0 0) 0x7f13e4001130 con 0x7f13ec08ef20 2019-12-05T06:27:43.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13eb7fe700 1 -- 172.21.15.16:0/3773658602 >> v2:172.21.15.76:3300/0 conn(0x7f13ec08fef0 msgr2=0x7f13ec08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:43.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3773658602 >> v2:172.21.15.76:3300/0 conn(0x7f13ec08fef0 0x7f13ec08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:43.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13eb7fe700 1 -- 172.21.15.16:0/3773658602 >> v1:172.21.15.16:6789/0 conn(0x7f13ec08ef20 legacy=0x7f13ec08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:43.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13eb7fe700 1 -- 172.21.15.16:0/3773658602 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ec08d540 con 0x7f13ec08c6d0 2019-12-05T06:27:43.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3773658602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 0x7f13ec08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f13dc00af50 tx=0x7f13dc00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3773658602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 0x7f13ec08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f13dc00af50 tx=0x7f13dc00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3773658602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 0x7f13ec08cac0 secure :-1 s=READY pgs=199 cs=0 l=1 rx=0x7f13dc00af50 tx=0x7f13dc00b250).ready entity=mon.2 client_cookie=3fe2ffc420804122 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:43.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3773658602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 0x7f13ec08cac0 secure :-1 s=THROTTLE_DONE pgs=199 cs=0 l=1 rx=0x7f13dc00af50 tx=0x7f13dc00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.410+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3773658602 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13dc007210 con 0x7f13ec08c6d0 2019-12-05T06:27:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3773658602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 0x7f13ec08cac0 secure :-1 s=THROTTLE_DONE pgs=199 cs=0 l=1 rx=0x7f13dc00af50 tx=0x7f13dc00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3773658602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 0x7f13ec08cac0 secure :-1 s=THROTTLE_DONE pgs=199 cs=0 l=1 rx=0x7f13dc00af50 tx=0x7f13dc00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3773658602 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13dc0099b0 con 0x7f13ec08c6d0 2019-12-05T06:27:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13ea7fc700 1 -- 172.21.15.16:0/3773658602 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13dc000960 con 0x7f13ec08c6d0 2019-12-05T06:27:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13f163e700 1 -- 172.21.15.16:0/3773658602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 msgr2=0x7f13ec08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13f163e700 1 --2- 172.21.15.16:0/3773658602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 0x7f13ec08cac0 secure :-1 s=READY pgs=199 cs=0 l=1 rx=0x7f13dc00af50 tx=0x7f13dc00b250).stop 2019-12-05T06:27:43.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13f163e700 1 -- 172.21.15.16:0/3773658602 shutdown_connections 2019-12-05T06:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13f163e700 1 --2- 172.21.15.16:0/3773658602 >> v2:172.21.15.76:3300/0 conn(0x7f13ec08fef0 0x7f13ec08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13f163e700 1 --2- 172.21.15.16:0/3773658602 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec08c6d0 0x7f13ec08cac0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13f163e700 1 -- 172.21.15.16:0/3773658602 shutdown_connections 2019-12-05T06:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13f163e700 1 -- 172.21.15.16:0/3773658602 wait complete. 2019-12-05T06:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.411+0000 7f13f163e700 1 -- 172.21.15.16:0/3773658602 >> 172.21.15.16:0/3773658602 conn(0x7f13ec061940 msgr2=0x7f13ec096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:43.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.414+0000 7f13f163e700 1 Processor -- start 2019-12-05T06:27:43.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.415+0000 7f13f163e700 1 -- start start 2019-12-05T06:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.415+0000 7f13f163e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f13ec1302f0 0x7f13ec12a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.415+0000 7f13f163e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec12b640 0x7f13ec12abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.415+0000 7f13f163e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13ec131bf0 con 0x7f13ec12b0f0 2019-12-05T06:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.415+0000 7f13f163e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13ec1296e0 con 0x7f13ec1302f0 2019-12-05T06:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.415+0000 7f13f163e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13ec128cb0 con 0x7f13ec12b640 2019-12-05T06:27:43.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.415+0000 7f13ebfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f13ec12b0f0 0x7f13ec1306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:38994/0 (socket says 172.21.15.16:38994) 2019-12-05T06:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.415+0000 7f13ebfff700 1 -- 172.21.15.16:0/3760254174 learned_addr learned my addr 172.21.15.16:0/3760254174 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec12b640 0x7f13ec12abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3760254174 >> v2:172.21.15.76:3300/0 conn(0x7f13ec1302f0 0x7f13ec12a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323078175 0 0) 0x7f13e4000a40 con 0x7f13ec12b0f0 2019-12-05T06:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13cc002e80 con 0x7f13ec12b0f0 2019-12-05T06:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2082964763 0 0) 0x7f13e4000ef0 con 0x7f13ec12b0f0 2019-12-05T06:27:43.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13cc0032e0 con 0x7f13ec12b0f0 2019-12-05T06:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f13e4001790 con 0x7f13ec12b0f0 2019-12-05T06:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec12b640 msgr2=0x7f13ec12abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13ec12b640 0x7f13ec12abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 >> v2:172.21.15.76:3300/0 conn(0x7f13ec1302f0 msgr2=0x7f13ec12a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:43.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 --2- 172.21.15.16:0/3760254174 >> v2:172.21.15.76:3300/0 conn(0x7f13ec1302f0 0x7f13ec12a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:43.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 >> v1:172.21.15.16:6789/0 conn(0x7f13ec12b0f0 legacy=0x7f13ec1306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:43.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13cc0033a0 0x7f13cc0037f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:43.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:43.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13cc0048c0 0x7f13cc004d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:43.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc00bb60 con 0x7f13cc003d30 2019-12-05T06:27:43.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc00bd60 con 0x7f13cc0033a0 2019-12-05T06:27:43.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc00bf90 con 0x7f13cc0048c0 2019-12-05T06:27:43.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:43.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.416+0000 7f13ebfff700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13cc0048c0 0x7f13cc004d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 -- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13cc0048c0 msgr2=0x7f13cc004d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f13cc0048c0 0x7f13cc004d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 -- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13cc0033a0 msgr2=0x7f13cc0037f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f13cc0033a0 0x7f13cc0037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:43.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 -- 172.21.15.16:0/3760254174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc00c240 con 0x7f13cc003d30 2019-12-05T06:27:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13f163e700 1 -- 172.21.15.16:0/3760254174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ec129d70 con 0x7f13cc003d30 2019-12-05T06:27:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13f163e700 1 -- 172.21.15.16:0/3760254174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ec131bf0 con 0x7f13cc003d30 2019-12-05T06:27:43.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=READY pgs=805 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:43.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13e00013f0 con 0x7f13cc003d30 2019-12-05T06:27:43.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13e0001010 con 0x7f13cc003d30 2019-12-05T06:27:43.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13e00015e0 con 0x7f13cc003d30 2019-12-05T06:27:43.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.417+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.418+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f13e0001200 con 0x7f13cc003d30 2019-12-05T06:27:43.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.418+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.418+0000 7f13d3fff700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f13cc02b9e0 0x7f13cc02de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:43.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.418+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f13e0045fa0 con 0x7f13cc003d30 2019-12-05T06:27:43.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.418+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f13cc02b9e0 0x7f13cc02de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:43.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.419+0000 7f13eb7fe700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f13cc02b9e0 0x7f13cc02de10 crc :-1 s=READY pgs=351 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:43.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.423+0000 7f13f163e700 1 -- 172.21.15.16:0/3760254174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c4002e60 con 0x7f13cc003d30 2019-12-05T06:27:43.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.427+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.427+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f13e0032370 con 0x7f13cc003d30 2019-12-05T06:27:43.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.627+0000 7f13f163e700 1 -- 172.21.15.16:0/3760254174 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13c4001d10 con 0x7f13cc003d30 2019-12-05T06:27:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.628+0000 7f13eaffd700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=THROTTLE_DONE pgs=805 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.628+0000 7f13d3fff700 1 -- 172.21.15.16:0/3760254174 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f13e0032370 con 0x7f13cc003d30 2019-12-05T06:27:43.617 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2019-12-05T06:27:43.617 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:43.617 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.631+0000 7f13f163e700 1 -- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f13cc02b9e0 msgr2=0x7f13cc02de10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.631+0000 7f13f163e700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f13cc02b9e0 0x7f13cc02de10 crc :-1 s=READY pgs=351 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.631+0000 7f13f163e700 1 -- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 msgr2=0x7f13cc004180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.631+0000 7f13f163e700 1 --2- 172.21.15.16:0/3760254174 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f13cc003d30 0x7f13cc004180 secure :-1 s=READY pgs=805 cs=0 l=1 rx=0x7f13e00106e0 tx=0x7f13e00105c0).stop 2019-12-05T06:27:43.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.631+0000 7f13ebfff700 1 -- 172.21.15.16:0/3760254174 reap_dead start 2019-12-05T06:27:43.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.631+0000 7f13f163e700 1 -- 172.21.15.16:0/3760254174 shutdown_connections 2019-12-05T06:27:43.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.631+0000 7f13f163e700 1 -- 172.21.15.16:0/3760254174 shutdown_connections 2019-12-05T06:27:43.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.631+0000 7f13f163e700 1 -- 172.21.15.16:0/3760254174 wait complete. 2019-12-05T06:27:43.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:43.631+0000 7f13f163e700 1 -- 172.21.15.16:0/3760254174 >> 172.21.15.16:0/3760254174 conn(0x7f13ec0a2850 msgr2=0x7f13ec06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:43.643 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:27:43.644 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 13 secs before thrashing 2019-12-05T06:27:56.645 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:56.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.912+0000 7f7a31e68700 1 Processor -- start 2019-12-05T06:27:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.912+0000 7f7a31e68700 1 -- start start 2019-12-05T06:27:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a31e68700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a31e68700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7a2c08fef0 0x7f7a2c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:56.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a31e68700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c08fb80 con 0x7f7a2c08ef20 2019-12-05T06:27:56.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a31e68700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c08df90 con 0x7f7a2c08c6d0 2019-12-05T06:27:56.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a31e68700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c0a3f70 con 0x7f7a2c08fef0 2019-12-05T06:27:56.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a2bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:56.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a2bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53892/0 (socket says 172.21.15.16:53892) 2019-12-05T06:27:56.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a30c06700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a2c08ef20 0x7f7a2c08f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39060/0 (socket says 172.21.15.16:39060) 2019-12-05T06:27:56.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a2bfff700 1 -- 172.21.15.16:0/4252739117 learned_addr learned my addr 172.21.15.16:0/4252739117 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:56.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a23fff700 1 --2- 172.21.15.16:0/4252739117 >> v2:172.21.15.76:3300/0 conn(0x7f7a2c08fef0 0x7f7a2c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:56.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a2bfff700 1 -- 172.21.15.16:0/4252739117 >> v2:172.21.15.76:3300/0 conn(0x7f7a2c08fef0 msgr2=0x7f7a2c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:56.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/4252739117 >> v2:172.21.15.76:3300/0 conn(0x7f7a2c08fef0 0x7f7a2c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:56.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a2bfff700 1 -- 172.21.15.16:0/4252739117 >> v1:172.21.15.16:6789/0 conn(0x7f7a2c08ef20 legacy=0x7f7a2c08f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:56.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.914+0000 7f7a2b7fe700 1 -- 172.21.15.16:0/4252739117 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635223666 0 0) 0x7f7a24000d30 con 0x7f7a2c08ef20 2019-12-05T06:27:56.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a2bfff700 1 -- 172.21.15.16:0/4252739117 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c08d540 con 0x7f7a2c08c6d0 2019-12-05T06:27:56.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/4252739117 >> v2:172.21.15.16:3300/0 conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7a1c00f6c0 tx=0x7f7a1c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/4252739117 >> v2:172.21.15.16:3300/0 conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7a1c00f6c0 tx=0x7f7a1c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/4252739117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 secure :-1 s=READY pgs=807 cs=0 l=1 rx=0x7f7a1c00f6c0 tx=0x7f7a1c00fa60).ready entity=mon.0 client_cookie=36f58940883669d1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:56.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/4252739117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f7a1c00f6c0 tx=0x7f7a1c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a2b7fe700 1 -- 172.21.15.16:0/4252739117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a1c000960 con 0x7f7a2c08c6d0 2019-12-05T06:27:56.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/4252739117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f7a1c00f6c0 tx=0x7f7a1c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/4252739117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 secure :-1 s=THROTTLE_DONE pgs=807 cs=0 l=1 rx=0x7f7a1c00f6c0 tx=0x7f7a1c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a2b7fe700 1 -- 172.21.15.16:0/4252739117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c000c90 con 0x7f7a2c08c6d0 2019-12-05T06:27:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a2b7fe700 1 -- 172.21.15.16:0/4252739117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a1c0013a0 con 0x7f7a2c08c6d0 2019-12-05T06:27:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a31e68700 1 -- 172.21.15.16:0/4252739117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c08c6d0 msgr2=0x7f7a2c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:56.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.915+0000 7f7a31e68700 1 --2- 172.21.15.16:0/4252739117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 secure :-1 s=READY pgs=807 cs=0 l=1 rx=0x7f7a1c00f6c0 tx=0x7f7a1c00fa60).stop 2019-12-05T06:27:56.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.916+0000 7f7a31e68700 1 -- 172.21.15.16:0/4252739117 shutdown_connections 2019-12-05T06:27:56.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.916+0000 7f7a31e68700 1 --2- 172.21.15.16:0/4252739117 >> v2:172.21.15.76:3300/0 conn(0x7f7a2c08fef0 0x7f7a2c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:56.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.916+0000 7f7a31e68700 1 --2- 172.21.15.16:0/4252739117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c08c6d0 0x7f7a2c08cac0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:56.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.916+0000 7f7a31e68700 1 -- 172.21.15.16:0/4252739117 shutdown_connections 2019-12-05T06:27:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.916+0000 7f7a31e68700 1 -- 172.21.15.16:0/4252739117 wait complete. 2019-12-05T06:27:56.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.916+0000 7f7a31e68700 1 -- 172.21.15.16:0/4252739117 >> 172.21.15.16:0/4252739117 conn(0x7f7a2c061940 msgr2=0x7f7a2c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:56.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.919+0000 7f7a31e68700 1 Processor -- start 2019-12-05T06:27:56.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.920+0000 7f7a31e68700 1 -- start start 2019-12-05T06:27:56.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.920+0000 7f7a31e68700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a2c096e90 0x7f7a2c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.920+0000 7f7a31e68700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7a2c084f30 0x7f7a2c09b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.920+0000 7f7a31e68700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a2c084b90 con 0x7f7a2c097280 2019-12-05T06:27:56.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.920+0000 7f7a31e68700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c09a450 con 0x7f7a2c096e90 2019-12-05T06:27:56.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.920+0000 7f7a31e68700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a2c099a20 con 0x7f7a2c084f30 2019-12-05T06:27:56.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.920+0000 7f7a2bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a2c096e90 0x7f7a2c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:56.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7a2c096e90 0x7f7a2c09b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53898/0 (socket says 172.21.15.16:53898) 2019-12-05T06:27:56.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 -- 172.21.15.16:0/3055802357 learned_addr learned my addr 172.21.15.16:0/3055802357 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:56.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a23fff700 1 --2- 172.21.15.16:0/3055802357 >> v2:172.21.15.76:3300/0 conn(0x7f7a2c084f30 0x7f7a2c09b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:56.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 -- 172.21.15.16:0/3055802357 >> v2:172.21.15.76:3300/0 conn(0x7f7a2c084f30 msgr2=0x7f7a2c09b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:56.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> v2:172.21.15.76:3300/0 conn(0x7f7a2c084f30 0x7f7a2c09b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:56.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 -- 172.21.15.16:0/3055802357 >> v1:172.21.15.76:6789/0 conn(0x7f7a2c097280 legacy=0x7f7a2c0976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:56.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 -- 172.21.15.16:0/3055802357 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2c09aae0 con 0x7f7a2c096e90 2019-12-05T06:27:56.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> v2:172.21.15.16:3300/0 conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> v2:172.21.15.16:3300/0 conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=READY pgs=808 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:56.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a297fa700 1 -- 172.21.15.16:0/3055802357 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a1c005770 con 0x7f7a2c096e90 2019-12-05T06:27:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a297fa700 1 -- 172.21.15.16:0/3055802357 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a1c005390 con 0x7f7a2c096e90 2019-12-05T06:27:56.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a31e68700 1 -- 172.21.15.16:0/3055802357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a2c09aae0 con 0x7f7a2c096e90 2019-12-05T06:27:56.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a297fa700 1 -- 172.21.15.16:0/3055802357 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a1c005e50 con 0x7f7a2c096e90 2019-12-05T06:27:56.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.921+0000 7f7a31e68700 1 -- 172.21.15.16:0/3055802357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a2c09aae0 con 0x7f7a2c096e90 2019-12-05T06:27:56.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.922+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.923+0000 7f7a297fa700 1 -- 172.21.15.16:0/3055802357 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7a1c005580 con 0x7f7a2c096e90 2019-12-05T06:27:56.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.923+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.923+0000 7f7a297fa700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a10023630 0x7f7a10025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:56.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.923+0000 7f7a297fa700 1 -- 172.21.15.16:0/3055802357 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f7a1c047fe0 con 0x7f7a2c096e90 2019-12-05T06:27:56.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.923+0000 7f7a23fff700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a10023630 0x7f7a10025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:56.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.924+0000 7f7a23fff700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a10023630 0x7f7a10025ac0 crc :-1 s=READY pgs=352 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:56.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.927+0000 7f7a31e68700 1 -- 172.21.15.16:0/3055802357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a2c09aae0 con 0x7f7a2c096e90 2019-12-05T06:27:56.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.932+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:56.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:56.932+0000 7f7a297fa700 1 -- 172.21.15.16:0/3055802357 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7a1c00b160 con 0x7f7a2c096e90 2019-12-05T06:27:57.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.161+0000 7f7a31e68700 1 -- 172.21.15.16:0/3055802357 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a2c09aae0 con 0x7f7a2c096e90 2019-12-05T06:27:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.161+0000 7f7a2bfff700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.161+0000 7f7a297fa700 1 -- 172.21.15.16:0/3055802357 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f7a1c00b160 con 0x7f7a2c096e90 2019-12-05T06:27:57.152 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2019-12-05T06:27:57.152 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:57.152 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:57.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 -- 172.21.15.16:0/3055802357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a10023630 msgr2=0x7f7a10025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:57.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a10023630 0x7f7a10025ac0 crc :-1 s=READY pgs=352 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:57.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 -- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 msgr2=0x7f7a2c09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:57.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 0x7f7a2c09b3e0 secure :-1 s=READY pgs=808 cs=0 l=1 rx=0x7f7a1c0125f0 tx=0x7f7a1c0124a0).stop 2019-12-05T06:27:57.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 -- 172.21.15.16:0/3055802357 shutdown_connections 2019-12-05T06:27:57.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 --2- 172.21.15.16:0/3055802357 >> v2:172.21.15.76:3300/0 conn(0x7f7a2c084f30 0x7f7a2c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:57.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a10023630 0x7f7a10025ac0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:57.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 --2- 172.21.15.16:0/3055802357 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7a2c096e90 0x7f7a2c09b3e0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:57.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 -- 172.21.15.16:0/3055802357 shutdown_connections 2019-12-05T06:27:57.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 -- 172.21.15.16:0/3055802357 wait complete. 2019-12-05T06:27:57.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.164+0000 7f7a31e68700 1 -- 172.21.15.16:0/3055802357 >> 172.21.15.16:0/3055802357 conn(0x7f7a2c0714f0 msgr2=0x7f7a2c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:57.164 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:27:57.165 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:27:57.165 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:27:57.172 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:27:57.172 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.459+0000 7f9f1c13b700 1 Processor -- start 2019-12-05T06:27:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.459+0000 7f9f1c13b700 1 -- start start 2019-12-05T06:27:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.461+0000 7f9f1c13b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f1408c6d0 0x7f9f1408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.461+0000 7f9f1c13b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9f1408fef0 0x7f9f1408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f1c13b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f1408fb80 con 0x7f9f1408ef20 2019-12-05T06:27:57.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f1c13b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1408df90 con 0x7f9f1408c6d0 2019-12-05T06:27:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f1c13b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f140a3f70 con 0x7f9f1408fef0 2019-12-05T06:27:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f1a6d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f1408c6d0 0x7f9f1408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f1a6d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f1408c6d0 0x7f9f1408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53914/0 (socket says 172.21.15.16:53914) 2019-12-05T06:27:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f1a6d8700 1 -- 172.21.15.16:0/2458743556 learned_addr learned my addr 172.21.15.16:0/2458743556 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f19ed7700 1 --2- 172.21.15.16:0/2458743556 >> v2:172.21.15.76:3300/0 conn(0x7f9f1408fef0 0x7f9f1408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f1a6d8700 1 -- 172.21.15.16:0/2458743556 >> v2:172.21.15.76:3300/0 conn(0x7f9f1408fef0 msgr2=0x7f9f1408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:57.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/2458743556 >> v2:172.21.15.76:3300/0 conn(0x7f9f1408fef0 0x7f9f1408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f1a6d8700 1 -- 172.21.15.16:0/2458743556 >> v1:172.21.15.76:6789/0 conn(0x7f9f1408ef20 legacy=0x7f9f1408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.462+0000 7f9f1a6d8700 1 -- 172.21.15.16:0/2458743556 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1408d540 con 0x7f9f1408c6d0 2019-12-05T06:27:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/2458743556 >> v2:172.21.15.16:3300/0 conn(0x7f9f1408c6d0 0x7f9f1408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9f0c00fac0 tx=0x7f9f0c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/2458743556 >> v2:172.21.15.16:3300/0 conn(0x7f9f1408c6d0 0x7f9f1408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9f0c00fac0 tx=0x7f9f0c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/2458743556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1408c6d0 0x7f9f1408cac0 secure :-1 s=READY pgs=809 cs=0 l=1 rx=0x7f9f0c00fac0 tx=0x7f9f0c00f8f0).ready entity=mon.0 client_cookie=c7cc0c9aab721197 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/2458743556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1408c6d0 0x7f9f1408cac0 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f9f0c00fac0 tx=0x7f9f0c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/2458743556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1408c6d0 0x7f9f1408cac0 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f9f0c00fac0 tx=0x7f9f0c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f196d6700 1 -- 172.21.15.16:0/2458743556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f0c000ce0 con 0x7f9f1408c6d0 2019-12-05T06:27:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/2458743556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1408c6d0 0x7f9f1408cac0 secure :-1 s=THROTTLE_DONE pgs=809 cs=0 l=1 rx=0x7f9f0c00fac0 tx=0x7f9f0c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f196d6700 1 -- 172.21.15.16:0/2458743556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c001010 con 0x7f9f1408c6d0 2019-12-05T06:27:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f196d6700 1 -- 172.21.15.16:0/2458743556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f0c001720 con 0x7f9f1408c6d0 2019-12-05T06:27:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f1c13b700 1 -- 172.21.15.16:0/2458743556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1408c6d0 msgr2=0x7f9f1408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.463+0000 7f9f1c13b700 1 --2- 172.21.15.16:0/2458743556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1408c6d0 0x7f9f1408cac0 secure :-1 s=READY pgs=809 cs=0 l=1 rx=0x7f9f0c00fac0 tx=0x7f9f0c00f8f0).stop 2019-12-05T06:27:57.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.465+0000 7f9f1c13b700 1 -- 172.21.15.16:0/2458743556 shutdown_connections 2019-12-05T06:27:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.465+0000 7f9f1c13b700 1 --2- 172.21.15.16:0/2458743556 >> v2:172.21.15.76:3300/0 conn(0x7f9f1408fef0 0x7f9f1408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.465+0000 7f9f1c13b700 1 --2- 172.21.15.16:0/2458743556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1408c6d0 0x7f9f1408cac0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.465+0000 7f9f1c13b700 1 -- 172.21.15.16:0/2458743556 shutdown_connections 2019-12-05T06:27:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.465+0000 7f9f1c13b700 1 -- 172.21.15.16:0/2458743556 wait complete. 2019-12-05T06:27:57.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.465+0000 7f9f1c13b700 1 -- 172.21.15.16:0/2458743556 >> 172.21.15.16:0/2458743556 conn(0x7f9f14061940 msgr2=0x7f9f14096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:57.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.470+0000 7f9f1c13b700 1 Processor -- start 2019-12-05T06:27:57.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.470+0000 7f9f1c13b700 1 -- start start 2019-12-05T06:27:57.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.471+0000 7f9f1c13b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f1406df10 0x7f9f14098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:57.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.471+0000 7f9f1c13b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9f14156cb0 0x7f9f141590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:57.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.471+0000 7f9f1c13b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f14089fb0 con 0x7f9f140992d0 2019-12-05T06:27:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.471+0000 7f9f1c13b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f14097370 con 0x7f9f1406df10 2019-12-05T06:27:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.471+0000 7f9f1c13b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f1408ad00 con 0x7f9f14156cb0 2019-12-05T06:27:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.471+0000 7f9f1a6d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f1406df10 0x7f9f14098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.471+0000 7f9f1a6d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f1406df10 0x7f9f14098300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53920/0 (socket says 172.21.15.16:53920) 2019-12-05T06:27:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.471+0000 7f9f1a6d8700 1 -- 172.21.15.16:0/1201572421 learned_addr learned my addr 172.21.15.16:0/1201572421 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.471+0000 7f9f1aed9700 1 --2- 172.21.15.16:0/1201572421 >> v2:172.21.15.76:3300/0 conn(0x7f9f14156cb0 0x7f9f141590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:57.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1a6d8700 1 -- 172.21.15.16:0/1201572421 >> v2:172.21.15.76:3300/0 conn(0x7f9f14156cb0 msgr2=0x7f9f141590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:57.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> v2:172.21.15.76:3300/0 conn(0x7f9f14156cb0 0x7f9f141590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:57.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1a6d8700 1 -- 172.21.15.16:0/1201572421 >> v1:172.21.15.76:6789/0 conn(0x7f9f140992d0 legacy=0x7f9f14098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:57.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1a6d8700 1 -- 172.21.15.16:0/1201572421 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f14097a00 con 0x7f9f1406df10 2019-12-05T06:27:57.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> v2:172.21.15.16:3300/0 conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> v2:172.21.15.16:3300/0 conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=READY pgs=810 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:57.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f12ffd700 1 -- 172.21.15.16:0/1201572421 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f0c008190 con 0x7f9f1406df10 2019-12-05T06:27:57.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f12ffd700 1 -- 172.21.15.16:0/1201572421 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f0c005710 con 0x7f9f1406df10 2019-12-05T06:27:57.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f12ffd700 1 -- 172.21.15.16:0/1201572421 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f0c005da0 con 0x7f9f1406df10 2019-12-05T06:27:57.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.472+0000 7f9f1c13b700 1 -- 172.21.15.16:0/1201572421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f14097a00 con 0x7f9f1406df10 2019-12-05T06:27:57.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.473+0000 7f9f1c13b700 1 -- 172.21.15.16:0/1201572421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f14097a00 con 0x7f9f1406df10 2019-12-05T06:27:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.473+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.474+0000 7f9f12ffd700 1 -- 172.21.15.16:0/1201572421 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9f0c005900 con 0x7f9f1406df10 2019-12-05T06:27:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.474+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.475+0000 7f9f12ffd700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ef4023630 0x7f9ef4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:27:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.475+0000 7f9f12ffd700 1 -- 172.21.15.16:0/1201572421 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f9f0c0461f0 con 0x7f9f1406df10 2019-12-05T06:27:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.475+0000 7f9f19ed7700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ef4023630 0x7f9ef4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:57.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.475+0000 7f9f19ed7700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ef4023630 0x7f9ef4025ac0 crc :-1 s=READY pgs=353 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:57.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.491+0000 7f9f1c13b700 1 -- 172.21.15.16:0/1201572421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f14097a00 con 0x7f9f1406df10 2019-12-05T06:27:57.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.497+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.497+0000 7f9f12ffd700 1 -- 172.21.15.16:0/1201572421 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9f0c005500 con 0x7f9f1406df10 2019-12-05T06:27:57.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.727+0000 7f9f1c13b700 1 -- 172.21.15.16:0/1201572421 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f14097a00 con 0x7f9f1406df10 2019-12-05T06:27:57.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.728+0000 7f9f1a6d8700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=THROTTLE_DONE pgs=810 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:57.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.728+0000 7f9f12ffd700 1 -- 172.21.15.16:0/1201572421 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f9f0c005900 con 0x7f9f1406df10 2019-12-05T06:27:57.718 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2019-12-05T06:27:57.719 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:27:57.719 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:27:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 -- 172.21.15.16:0/1201572421 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ef4023630 msgr2=0x7f9ef4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ef4023630 0x7f9ef4025ac0 crc :-1 s=READY pgs=353 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 -- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 msgr2=0x7f9f14098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 0x7f9f14098300 secure :-1 s=READY pgs=810 cs=0 l=1 rx=0x7f9f0c0107e0 tx=0x7f9f0c0105e0).stop 2019-12-05T06:27:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 -- 172.21.15.16:0/1201572421 shutdown_connections 2019-12-05T06:27:57.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 --2- 172.21.15.16:0/1201572421 >> v2:172.21.15.76:3300/0 conn(0x7f9f14156cb0 0x7f9f141590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ef4023630 0x7f9ef4025ac0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 --2- 172.21.15.16:0/1201572421 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f1406df10 0x7f9f14098300 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 -- 172.21.15.16:0/1201572421 shutdown_connections 2019-12-05T06:27:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 -- 172.21.15.16:0/1201572421 wait complete. 2019-12-05T06:27:57.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:57.731+0000 7f9f1c13b700 1 -- 172.21.15.16:0/1201572421 >> 172.21.15.16:0/1201572421 conn(0x7f9f140714f0 msgr2=0x7f9f1409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:27:57.737 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:27:59.738 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:27:59.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.995+0000 7fbbc90d0700 1 Processor -- start 2019-12-05T06:27:59.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.995+0000 7fbbc90d0700 1 -- start start 2019-12-05T06:27:59.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.996+0000 7fbbc90d0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbbc408c6d0 0x7fbbc408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:59.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.996+0000 7fbbc90d0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbbc408fef0 0x7fbbc408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:27:59.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc90d0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc408fb80 con 0x7fbbc408ef20 2019-12-05T06:27:59.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc90d0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc408df90 con 0x7fbbc408c6d0 2019-12-05T06:27:59.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc90d0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc40a3f70 con 0x7fbbc408fef0 2019-12-05T06:27:59.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbbc408c6d0 0x7fbbc408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:59.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbbc408c6d0 0x7fbbc408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53932/0 (socket says 172.21.15.16:53932) 2019-12-05T06:27:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc359e700 1 -- 172.21.15.16:0/975393260 learned_addr learned my addr 172.21.15.16:0/975393260 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:27:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc88cf700 1 --1- 172.21.15.16:0/975393260 >> v1:172.21.15.16:6789/0 conn(0x7fbbc408ef20 0x7fbbc408f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39102/0 (socket says 172.21.15.16:39102) 2019-12-05T06:27:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc2d9d700 1 --2- 172.21.15.16:0/975393260 >> v2:172.21.15.76:3300/0 conn(0x7fbbc408fef0 0x7fbbc408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:27:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc359e700 1 -- 172.21.15.16:0/975393260 >> v2:172.21.15.76:3300/0 conn(0x7fbbc408fef0 msgr2=0x7fbbc408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:27:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc359e700 1 --2- 172.21.15.16:0/975393260 >> v2:172.21.15.76:3300/0 conn(0x7fbbc408fef0 0x7fbbc408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:59.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc359e700 1 -- 172.21.15.16:0/975393260 >> v1:172.21.15.16:6789/0 conn(0x7fbbc408ef20 legacy=0x7fbbc408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc359e700 1 -- 172.21.15.16:0/975393260 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc408d540 con 0x7fbbc408c6d0 2019-12-05T06:27:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.997+0000 7fbbc259c700 1 -- 172.21.15.16:0/975393260 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424232147 0 0) 0x7fbbb40009b0 con 0x7fbbc408ef20 2019-12-05T06:27:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc2d9d700 1 --2- 172.21.15.16:0/975393260 >> v2:172.21.15.76:3300/0 conn(0x7fbbc408fef0 0x7fbbc408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:27:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc359e700 1 --2- 172.21.15.16:0/975393260 >> v2:172.21.15.16:3300/0 conn(0x7fbbc408c6d0 0x7fbbc408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbbbc00fac0 tx=0x7fbbbc00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc359e700 1 --2- 172.21.15.16:0/975393260 >> v2:172.21.15.16:3300/0 conn(0x7fbbc408c6d0 0x7fbbc408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbbbc00fac0 tx=0x7fbbbc00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc359e700 1 --2- 172.21.15.16:0/975393260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc408c6d0 0x7fbbc408cac0 secure :-1 s=READY pgs=812 cs=0 l=1 rx=0x7fbbbc00fac0 tx=0x7fbbbc00fe60).ready entity=mon.0 client_cookie=22365563b1673fa7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:27:59.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc359e700 1 --2- 172.21.15.16:0/975393260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc408c6d0 0x7fbbc408cac0 secure :-1 s=THROTTLE_DONE pgs=812 cs=0 l=1 rx=0x7fbbbc00fac0 tx=0x7fbbbc00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:59.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc259c700 1 -- 172.21.15.16:0/975393260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbbc000ce0 con 0x7fbbc408c6d0 2019-12-05T06:27:59.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc359e700 1 --2- 172.21.15.16:0/975393260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc408c6d0 0x7fbbc408cac0 secure :-1 s=THROTTLE_DONE pgs=812 cs=0 l=1 rx=0x7fbbbc00fac0 tx=0x7fbbbc00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:59.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc359e700 1 --2- 172.21.15.16:0/975393260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc408c6d0 0x7fbbc408cac0 secure :-1 s=THROTTLE_DONE pgs=812 cs=0 l=1 rx=0x7fbbbc00fac0 tx=0x7fbbbc00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:27:59.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc259c700 1 -- 172.21.15.16:0/975393260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbbc001010 con 0x7fbbc408c6d0 2019-12-05T06:27:59.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc259c700 1 -- 172.21.15.16:0/975393260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbbc001720 con 0x7fbbc408c6d0 2019-12-05T06:27:59.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc90d0700 1 -- 172.21.15.16:0/975393260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc408c6d0 msgr2=0x7fbbc408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:27:59.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:27:59.998+0000 7fbbc90d0700 1 --2- 172.21.15.16:0/975393260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc408c6d0 0x7fbbc408cac0 secure :-1 s=READY pgs=812 cs=0 l=1 rx=0x7fbbbc00fac0 tx=0x7fbbbc00fe60).stop 2019-12-05T06:27:59.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.000+0000 7fbbc90d0700 1 -- 172.21.15.16:0/975393260 shutdown_connections 2019-12-05T06:27:59.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.000+0000 7fbbc90d0700 1 --2- 172.21.15.16:0/975393260 >> v2:172.21.15.76:3300/0 conn(0x7fbbc408fef0 0x7fbbc408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:27:59.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.000+0000 7fbbc90d0700 1 --2- 172.21.15.16:0/975393260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbc408c6d0 0x7fbbc408cac0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:27:59.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.000+0000 7fbbc90d0700 1 -- 172.21.15.16:0/975393260 shutdown_connections 2019-12-05T06:27:59.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.000+0000 7fbbc90d0700 1 -- 172.21.15.16:0/975393260 wait complete. 2019-12-05T06:27:59.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.000+0000 7fbbc90d0700 1 -- 172.21.15.16:0/975393260 >> 172.21.15.16:0/975393260 conn(0x7fbbc4061940 msgr2=0x7fbbc4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.004+0000 7fbbc90d0700 1 Processor -- start 2019-12-05T06:28:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.004+0000 7fbbc90d0700 1 -- start start 2019-12-05T06:28:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.006+0000 7fbbc90d0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbbc414d400 0x7fbbc414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.006+0000 7fbbc90d0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc4098da0 con 0x7fbbc4132d00 2019-12-05T06:28:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.006+0000 7fbbc90d0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbc409f0a0 con 0x7fbbc4132530 2019-12-05T06:28:00.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.006+0000 7fbbc90d0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbc409bb50 con 0x7fbbc414d400 2019-12-05T06:28:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.006+0000 7fbbc2d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbbc4132d00 0x7fbbc4131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39108/0 (socket says 172.21.15.16:39108) 2019-12-05T06:28:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.006+0000 7fbbc2d9d700 1 -- 172.21.15.16:0/2100138448 learned_addr learned my addr 172.21.15.16:0/2100138448 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.006+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> v2:172.21.15.76:3300/0 conn(0x7fbbc414d400 0x7fbbc414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.006+0000 7fbbbbfff700 1 -- 172.21.15.16:0/2100138448 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228621133 0 0) 0x7fbbac009c90 con 0x7fbbc4132d00 2019-12-05T06:28:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.007+0000 7fbbbbfff700 1 -- 172.21.15.16:0/2100138448 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbba8002e80 con 0x7fbbc4132d00 2019-12-05T06:28:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.007+0000 7fbbc359e700 1 -- 172.21.15.16:0/2100138448 >> v1:172.21.15.76:6789/0 conn(0x7fbbc4132530 legacy=0x7fbbc4157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:00.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.007+0000 7fbbbbfff700 1 -- 172.21.15.16:0/2100138448 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1883583655 0 0) 0x7fbbac000d50 con 0x7fbbc4132d00 2019-12-05T06:28:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.007+0000 7fbbc359e700 1 -- 172.21.15.16:0/2100138448 >> v1:172.21.15.16:6789/0 conn(0x7fbbc4132d00 legacy=0x7fbbc4131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.007+0000 7fbbc359e700 1 -- 172.21.15.16:0/2100138448 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc409b0f0 con 0x7fbbc414d400 2019-12-05T06:28:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.007+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> v2:172.21.15.76:3300/0 conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.007+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> v2:172.21.15.76:3300/0 conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.007+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=READY pgs=588 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:00.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.008+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.008+0000 7fbbbbfff700 1 -- 172.21.15.16:0/2100138448 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbbc0058e0 con 0x7fbbc414d400 2019-12-05T06:28:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.008+0000 7fbbc90d0700 1 -- 172.21.15.16:0/2100138448 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc409b0f0 con 0x7fbbc414d400 2019-12-05T06:28:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.008+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.008+0000 7fbbbbfff700 1 -- 172.21.15.16:0/2100138448 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbbc005500 con 0x7fbbc414d400 2019-12-05T06:28:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.008+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.008+0000 7fbbbbfff700 1 -- 172.21.15.16:0/2100138448 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbbc005c50 con 0x7fbbc414d400 2019-12-05T06:28:00.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.008+0000 7fbbc90d0700 1 -- 172.21.15.16:0/2100138448 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc409b0f0 con 0x7fbbc414d400 2019-12-05T06:28:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.009+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.010+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.010+0000 7fbbbbfff700 1 -- 172.21.15.16:0/2100138448 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbbbc0056f0 con 0x7fbbc414d400 2019-12-05T06:28:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.010+0000 7fbbbbfff700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbba8023680 0x7fbba8025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.010+0000 7fbbbbfff700 1 -- 172.21.15.16:0/2100138448 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7fbbbc046160 con 0x7fbbc414d400 2019-12-05T06:28:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.010+0000 7fbbc2d9d700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbba8023680 0x7fbba8025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:00.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.011+0000 7fbbc90d0700 1 -- 172.21.15.16:0/2100138448 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbc409b0f0 con 0x7fbbc414d400 2019-12-05T06:28:00.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.011+0000 7fbbc2d9d700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbba8023680 0x7fbba8025af0 crc :-1 s=READY pgs=354 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.020+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:00.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.020+0000 7fbbbbfff700 1 -- 172.21.15.16:0/2100138448 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbbbc007690 con 0x7fbbc414d400 2019-12-05T06:28:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.273+0000 7fbbc90d0700 1 -- 172.21.15.16:0/2100138448 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbc409b0f0 con 0x7fbbc414d400 2019-12-05T06:28:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.275+0000 7fbbc359e700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=THROTTLE_DONE pgs=588 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.275+0000 7fbbbbfff700 1 -- 172.21.15.16:0/2100138448 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2111 (secure 0 0 0) 0x7fbbbc0056f0 con 0x7fbbc414d400 2019-12-05T06:28:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2019-12-05T06:28:00.267 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:00.267 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.280+0000 7fbbc90d0700 1 -- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbba8023680 msgr2=0x7fbba8025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.280+0000 7fbbc90d0700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbba8023680 0x7fbba8025af0 crc :-1 s=READY pgs=354 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.280+0000 7fbbc90d0700 1 -- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 msgr2=0x7fbbc414d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:00.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.280+0000 7fbbc90d0700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 0x7fbbc414d7f0 secure :-1 s=READY pgs=588 cs=0 l=1 rx=0x7fbbbc010720 tx=0x7fbbbc010ac0).stop 2019-12-05T06:28:00.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.280+0000 7fbbc90d0700 1 -- 172.21.15.16:0/2100138448 shutdown_connections 2019-12-05T06:28:00.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.280+0000 7fbbc90d0700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbba8023680 0x7fbba8025af0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:00.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.280+0000 7fbbc90d0700 1 --2- 172.21.15.16:0/2100138448 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbc414d400 0x7fbbc414d7f0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:00.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.281+0000 7fbbc90d0700 1 -- 172.21.15.16:0/2100138448 shutdown_connections 2019-12-05T06:28:00.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.281+0000 7fbbc90d0700 1 -- 172.21.15.16:0/2100138448 wait complete. 2019-12-05T06:28:00.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:00.281+0000 7fbbc90d0700 1 -- 172.21.15.16:0/2100138448 >> 172.21.15.16:0/2100138448 conn(0x7fbbc40714f0 msgr2=0x7fbbc409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:00.286 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:02.287 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:02.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.556+0000 7f0f58ffd700 1 Processor -- start 2019-12-05T06:28:02.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.556+0000 7f0f58ffd700 1 -- start start 2019-12-05T06:28:02.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.558+0000 7f0f58ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f5408c6d0 0x7f0f5408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:02.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.558+0000 7f0f58ffd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0f5408f720 0x7f0f5408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:02.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.558+0000 7f0f58ffd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f5408fb80 con 0x7f0f5408fef0 2019-12-05T06:28:02.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.558+0000 7f0f58ffd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5408df90 con 0x7f0f5408c6d0 2019-12-05T06:28:02.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.558+0000 7f0f58ffd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f540a3f70 con 0x7f0f5408f720 2019-12-05T06:28:02.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.558+0000 7f0f52d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f5408c6d0 0x7f0f5408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:02.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.558+0000 7f0f5259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0f5408fef0 0x7f0f5408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39118/0 (socket says 172.21.15.16:39118) 2019-12-05T06:28:02.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.558+0000 7f0f5259c700 1 -- 172.21.15.16:0/1578457309 learned_addr learned my addr 172.21.15.16:0/1578457309 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:02.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.559+0000 7f0f5359e700 1 --2- 172.21.15.16:0/1578457309 >> v2:172.21.15.76:3300/0 conn(0x7f0f5408f720 0x7f0f5408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:02.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.559+0000 7f0f51d9b700 1 -- 172.21.15.16:0/1578457309 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2757883048 0 0) 0x7f0f44001150 con 0x7f0f5408fef0 2019-12-05T06:28:02.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.559+0000 7f0f51d9b700 1 -- 172.21.15.16:0/1578457309 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f34002e80 con 0x7f0f5408fef0 2019-12-05T06:28:02.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.559+0000 7f0f52d9d700 1 -- 172.21.15.16:0/1578457309 >> v2:172.21.15.76:3300/0 conn(0x7f0f5408f720 msgr2=0x7f0f5408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.559+0000 7f0f52d9d700 1 --2- 172.21.15.16:0/1578457309 >> v2:172.21.15.76:3300/0 conn(0x7f0f5408f720 0x7f0f5408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.559+0000 7f0f52d9d700 1 -- 172.21.15.16:0/1578457309 >> v1:172.21.15.16:6789/0 conn(0x7f0f5408fef0 legacy=0x7f0f5408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.559+0000 7f0f52d9d700 1 -- 172.21.15.16:0/1578457309 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5408d540 con 0x7f0f5408c6d0 2019-12-05T06:28:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.559+0000 7f0f52d9d700 1 --2- 172.21.15.16:0/1578457309 >> v2:172.21.15.16:3300/0 conn(0x7f0f5408c6d0 0x7f0f5408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0f3c00b2d0 tx=0x7f0f3c00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.560+0000 7f0f52d9d700 1 --2- 172.21.15.16:0/1578457309 >> v2:172.21.15.16:3300/0 conn(0x7f0f5408c6d0 0x7f0f5408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0f3c00b2d0 tx=0x7f0f3c00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.560+0000 7f0f52d9d700 1 --2- 172.21.15.16:0/1578457309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f5408c6d0 0x7f0f5408cac0 secure :-1 s=READY pgs=815 cs=0 l=1 rx=0x7f0f3c00b2d0 tx=0x7f0f3c00b5d0).ready entity=mon.0 client_cookie=884a0e1c4aa2bf6f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.560+0000 7f0f52d9d700 1 --2- 172.21.15.16:0/1578457309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f5408c6d0 0x7f0f5408cac0 secure :-1 s=THROTTLE_DONE pgs=815 cs=0 l=1 rx=0x7f0f3c00b2d0 tx=0x7f0f3c00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.560+0000 7f0f51d9b700 1 -- 172.21.15.16:0/1578457309 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f3c007590 con 0x7f0f5408c6d0 2019-12-05T06:28:02.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.560+0000 7f0f52d9d700 1 --2- 172.21.15.16:0/1578457309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f5408c6d0 0x7f0f5408cac0 secure :-1 s=THROTTLE_DONE pgs=815 cs=0 l=1 rx=0x7f0f3c00b2d0 tx=0x7f0f3c00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.560+0000 7f0f52d9d700 1 --2- 172.21.15.16:0/1578457309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f5408c6d0 0x7f0f5408cac0 secure :-1 s=THROTTLE_DONE pgs=815 cs=0 l=1 rx=0x7f0f3c00b2d0 tx=0x7f0f3c00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.560+0000 7f0f51d9b700 1 -- 172.21.15.16:0/1578457309 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f3c009d30 con 0x7f0f5408c6d0 2019-12-05T06:28:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.560+0000 7f0f51d9b700 1 -- 172.21.15.16:0/1578457309 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f3c00a400 con 0x7f0f5408c6d0 2019-12-05T06:28:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.560+0000 7f0f58ffd700 1 -- 172.21.15.16:0/1578457309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f5408c6d0 msgr2=0x7f0f5408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:02.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.560+0000 7f0f58ffd700 1 --2- 172.21.15.16:0/1578457309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f5408c6d0 0x7f0f5408cac0 secure :-1 s=READY pgs=815 cs=0 l=1 rx=0x7f0f3c00b2d0 tx=0x7f0f3c00b5d0).stop 2019-12-05T06:28:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.562+0000 7f0f58ffd700 1 -- 172.21.15.16:0/1578457309 shutdown_connections 2019-12-05T06:28:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.562+0000 7f0f58ffd700 1 --2- 172.21.15.16:0/1578457309 >> v2:172.21.15.76:3300/0 conn(0x7f0f5408f720 0x7f0f5408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.562+0000 7f0f58ffd700 1 --2- 172.21.15.16:0/1578457309 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f5408c6d0 0x7f0f5408cac0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.562+0000 7f0f58ffd700 1 -- 172.21.15.16:0/1578457309 shutdown_connections 2019-12-05T06:28:02.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.562+0000 7f0f58ffd700 1 -- 172.21.15.16:0/1578457309 wait complete. 2019-12-05T06:28:02.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.562+0000 7f0f58ffd700 1 -- 172.21.15.16:0/1578457309 >> 172.21.15.16:0/1578457309 conn(0x7f0f54061940 msgr2=0x7f0f54096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.567+0000 7f0f58ffd700 1 Processor -- start 2019-12-05T06:28:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.567+0000 7f0f58ffd700 1 -- start start 2019-12-05T06:28:02.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.568+0000 7f0f58ffd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0f5414d400 0x7f0f5414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.568+0000 7f0f58ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f54132d00 0x7f0f54131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.568+0000 7f0f58ffd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f54098da0 con 0x7f0f54132270 2019-12-05T06:28:02.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.568+0000 7f0f58ffd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5409bb50 con 0x7f0f54132d00 2019-12-05T06:28:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.568+0000 7f0f58ffd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f5409aa60 con 0x7f0f5414d400 2019-12-05T06:28:02.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.568+0000 7f0f5259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f54132d00 0x7f0f54131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.568+0000 7f0f5259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f54132d00 0x7f0f54131d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:53956/0 (socket says 172.21.15.16:53956) 2019-12-05T06:28:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.568+0000 7f0f5259c700 1 -- 172.21.15.16:0/486172219 learned_addr learned my addr 172.21.15.16:0/486172219 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:28:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.568+0000 7f0f52d9d700 1 --2- 172.21.15.16:0/486172219 >> v2:172.21.15.76:3300/0 conn(0x7f0f5414d400 0x7f0f5414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.569+0000 7f0f5259c700 1 -- 172.21.15.16:0/486172219 >> v2:172.21.15.76:3300/0 conn(0x7f0f5414d400 msgr2=0x7f0f5414d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.569+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> v2:172.21.15.76:3300/0 conn(0x7f0f5414d400 0x7f0f5414d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:02.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.569+0000 7f0f5259c700 1 -- 172.21.15.16:0/486172219 >> v1:172.21.15.76:6789/0 conn(0x7f0f54132270 legacy=0x7f0f5406dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:02.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.569+0000 7f0f5259c700 1 -- 172.21.15.16:0/486172219 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f5409b0f0 con 0x7f0f54132d00 2019-12-05T06:28:02.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.569+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> v2:172.21.15.16:3300/0 conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.569+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> v2:172.21.15.16:3300/0 conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.569+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=READY pgs=816 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:02.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.569+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.570+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.570+0000 7f0f4b7fe700 1 -- 172.21.15.16:0/486172219 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f44000c90 con 0x7f0f54132d00 2019-12-05T06:28:02.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.570+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.570+0000 7f0f4b7fe700 1 -- 172.21.15.16:0/486172219 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f440010c0 con 0x7f0f54132d00 2019-12-05T06:28:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.570+0000 7f0f4b7fe700 1 -- 172.21.15.16:0/486172219 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f44001750 con 0x7f0f54132d00 2019-12-05T06:28:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.570+0000 7f0f58ffd700 1 -- 172.21.15.16:0/486172219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5409b0f0 con 0x7f0f54132d00 2019-12-05T06:28:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.570+0000 7f0f58ffd700 1 -- 172.21.15.16:0/486172219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5409b0f0 con 0x7f0f54132d00 2019-12-05T06:28:02.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.571+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.571+0000 7f0f4b7fe700 1 -- 172.21.15.16:0/486172219 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0f44000c90 con 0x7f0f54132d00 2019-12-05T06:28:02.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.571+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.571+0000 7f0f4b7fe700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f2c023630 0x7f0f2c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:02.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.572+0000 7f0f4b7fe700 1 -- 172.21.15.16:0/486172219 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f0f44046050 con 0x7f0f54132d00 2019-12-05T06:28:02.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.572+0000 7f0f52d9d700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f2c023630 0x7f0f2c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:02.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.572+0000 7f0f52d9d700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f2c023630 0x7f0f2c025ac0 crc :-1 s=READY pgs=355 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:02.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.588+0000 7f0f58ffd700 1 -- 172.21.15.16:0/486172219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5409b0f0 con 0x7f0f54132d00 2019-12-05T06:28:02.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.598+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.598+0000 7f0f4b7fe700 1 -- 172.21.15.16:0/486172219 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0f440075d0 con 0x7f0f54132d00 2019-12-05T06:28:02.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.848+0000 7f0f58ffd700 1 -- 172.21.15.16:0/486172219 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f5409b0f0 con 0x7f0f54132d00 2019-12-05T06:28:02.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.849+0000 7f0f5259c700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:02.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.849+0000 7f0f4b7fe700 1 -- 172.21.15.16:0/486172219 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f0f44001970 con 0x7f0f54132d00 2019-12-05T06:28:02.839 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2019-12-05T06:28:02.839 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:02.839 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.851+0000 7f0f58ffd700 1 -- 172.21.15.16:0/486172219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f2c023630 msgr2=0x7f0f2c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.852+0000 7f0f58ffd700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f2c023630 0x7f0f2c025ac0 crc :-1 s=READY pgs=355 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.852+0000 7f0f58ffd700 1 -- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 msgr2=0x7f0f54131d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.852+0000 7f0f58ffd700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 0x7f0f54131d30 secure :-1 s=READY pgs=816 cs=0 l=1 rx=0x7f0f440103e0 tx=0x7f0f44010260).stop 2019-12-05T06:28:02.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.852+0000 7f0f58ffd700 1 -- 172.21.15.16:0/486172219 shutdown_connections 2019-12-05T06:28:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.852+0000 7f0f58ffd700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f2c023630 0x7f0f2c025ac0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.852+0000 7f0f58ffd700 1 --2- 172.21.15.16:0/486172219 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f54132d00 0x7f0f54131d30 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:02.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.852+0000 7f0f58ffd700 1 --2- 172.21.15.16:0/486172219 >> v2:172.21.15.76:3300/0 conn(0x7f0f5414d400 0x7f0f5414d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.852+0000 7f0f58ffd700 1 -- 172.21.15.16:0/486172219 shutdown_connections 2019-12-05T06:28:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.852+0000 7f0f58ffd700 1 -- 172.21.15.16:0/486172219 wait complete. 2019-12-05T06:28:02.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:02.852+0000 7f0f58ffd700 1 -- 172.21.15.16:0/486172219 >> 172.21.15.16:0/486172219 conn(0x7f0f540714f0 msgr2=0x7f0f5409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:02.853 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:04.665 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:28:04.670 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:28:04.854 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.132+0000 7f8c83a18700 1 Processor -- start 2019-12-05T06:28:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.132+0000 7f8c83a18700 1 -- start start 2019-12-05T06:28:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.134+0000 7f8c83a18700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 0x7f8c7c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.134+0000 7f8c83a18700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c0958e0 con 0x7f8c7c08d2c0 2019-12-05T06:28:05.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.134+0000 7f8c83a18700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c0629e0 con 0x7f8c7c08c6d0 2019-12-05T06:28:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.134+0000 7f8c83a18700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8c7c08df90 con 0x7f8c7c08fef0 2019-12-05T06:28:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.134+0000 7f8c817b4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8c7c08d2c0 0x7f8c7c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39138/0 (socket says 172.21.15.16:39138) 2019-12-05T06:28:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.134+0000 7f8c817b4700 1 -- 172.21.15.16:0/2623570826 learned_addr learned my addr 172.21.15.16:0/2623570826 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.134+0000 7f8c827b6700 1 --2- 172.21.15.16:0/2623570826 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 0x7f8c7c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.135+0000 7f8c80fb3700 1 -- 172.21.15.16:0/2623570826 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1981496382 0 0) 0x7f8c70001150 con 0x7f8c7c08d2c0 2019-12-05T06:28:05.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.135+0000 7f8c80fb3700 1 -- 172.21.15.16:0/2623570826 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c64002e80 con 0x7f8c7c08d2c0 2019-12-05T06:28:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.135+0000 7f8c80fb3700 1 -- 172.21.15.16:0/2623570826 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964967017 0 0) 0x7f8c74001400 con 0x7f8c7c08c6d0 2019-12-05T06:28:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.135+0000 7f8c80fb3700 1 -- 172.21.15.16:0/2623570826 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c640030b0 con 0x7f8c7c08c6d0 2019-12-05T06:28:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.135+0000 7f8c827b6700 1 -- 172.21.15.16:0/2623570826 >> v1:172.21.15.76:6789/0 conn(0x7f8c7c08c6d0 legacy=0x7f8c7c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.135+0000 7f8c827b6700 1 -- 172.21.15.16:0/2623570826 >> v1:172.21.15.16:6789/0 conn(0x7f8c7c08d2c0 legacy=0x7f8c7c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.135+0000 7f8c827b6700 1 -- 172.21.15.16:0/2623570826 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c08e620 con 0x7f8c7c08fef0 2019-12-05T06:28:05.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.135+0000 7f8c827b6700 1 --2- 172.21.15.16:0/2623570826 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 0x7f8c7c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8c6c00af80 tx=0x7f8c6c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.136+0000 7f8c827b6700 1 --2- 172.21.15.16:0/2623570826 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 0x7f8c7c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8c6c00af80 tx=0x7f8c6c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.136+0000 7f8c827b6700 1 --2- 172.21.15.16:0/2623570826 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 0x7f8c7c08f720 secure :-1 s=READY pgs=200 cs=0 l=1 rx=0x7f8c6c00af80 tx=0x7f8c6c00b790).ready entity=mon.2 client_cookie=2b5ab3a3f225892d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:05.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.136+0000 7f8c827b6700 1 --2- 172.21.15.16:0/2623570826 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 0x7f8c7c08f720 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f8c6c00af80 tx=0x7f8c6c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.136+0000 7f8c827b6700 1 --2- 172.21.15.16:0/2623570826 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 0x7f8c7c08f720 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f8c6c00af80 tx=0x7f8c6c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.136+0000 7f8c80fb3700 1 -- 172.21.15.16:0/2623570826 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c6c007240 con 0x7f8c7c08fef0 2019-12-05T06:28:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.136+0000 7f8c827b6700 1 --2- 172.21.15.16:0/2623570826 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 0x7f8c7c08f720 secure :-1 s=THROTTLE_DONE pgs=200 cs=0 l=1 rx=0x7f8c6c00af80 tx=0x7f8c6c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.136+0000 7f8c80fb3700 1 -- 172.21.15.16:0/2623570826 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c6c000bb0 con 0x7f8c7c08fef0 2019-12-05T06:28:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.136+0000 7f8c80fb3700 1 -- 172.21.15.16:0/2623570826 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c6c009e60 con 0x7f8c7c08fef0 2019-12-05T06:28:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.136+0000 7f8c83a18700 1 -- 172.21.15.16:0/2623570826 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 msgr2=0x7f8c7c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.136+0000 7f8c83a18700 1 --2- 172.21.15.16:0/2623570826 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 0x7f8c7c08f720 secure :-1 s=READY pgs=200 cs=0 l=1 rx=0x7f8c6c00af80 tx=0x7f8c6c00b790).stop 2019-12-05T06:28:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.138+0000 7f8c83a18700 1 -- 172.21.15.16:0/2623570826 shutdown_connections 2019-12-05T06:28:05.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.138+0000 7f8c83a18700 1 --2- 172.21.15.16:0/2623570826 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8c7c08fef0 0x7f8c7c08f720 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.139+0000 7f8c83a18700 1 -- 172.21.15.16:0/2623570826 shutdown_connections 2019-12-05T06:28:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.139+0000 7f8c83a18700 1 -- 172.21.15.16:0/2623570826 wait complete. 2019-12-05T06:28:05.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.139+0000 7f8c83a18700 1 -- 172.21.15.16:0/2623570826 >> 172.21.15.16:0/2623570826 conn(0x7f8c7c061940 msgr2=0x7f8c7c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:05.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.143+0000 7f8c83a18700 1 Processor -- start 2019-12-05T06:28:05.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.143+0000 7f8c83a18700 1 -- start start 2019-12-05T06:28:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.144+0000 7f8c83a18700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.144+0000 7f8c83a18700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c14de80 con 0x7f8c7c14d5d0 2019-12-05T06:28:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.144+0000 7f8c83a18700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8c7c06e250 con 0x7f8c7c14d9c0 2019-12-05T06:28:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.144+0000 7f8c83a18700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8c7c12b3f0 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.145+0000 7f8c81fb5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8c7c14d5d0 0x7f8c7c09d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39142/0 (socket says 172.21.15.16:39142) 2019-12-05T06:28:05.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.145+0000 7f8c81fb5700 1 -- 172.21.15.16:0/2082116219 learned_addr learned my addr 172.21.15.16:0/2082116219 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.145+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> v2:172.21.15.76:3300/0 conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.145+0000 7f8c7a7fc700 1 -- 172.21.15.16:0/2082116219 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2544358132 0 0) 0x7f8c740015c0 con 0x7f8c7c14d5d0 2019-12-05T06:28:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.145+0000 7f8c7a7fc700 1 -- 172.21.15.16:0/2082116219 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8c5c002e80 con 0x7f8c7c14d5d0 2019-12-05T06:28:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c817b4700 1 -- 172.21.15.16:0/2082116219 >> v1:172.21.15.76:6789/0 conn(0x7f8c7c14d9c0 legacy=0x7f8c7c153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c817b4700 1 -- 172.21.15.16:0/2082116219 >> v1:172.21.15.16:6789/0 conn(0x7f8c7c14d5d0 legacy=0x7f8c7c09d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c7a7fc700 1 -- 172.21.15.16:0/2082116219 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1178040427 0 0) 0x7f8c6c000b40 con 0x7f8c7c14d9c0 2019-12-05T06:28:05.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c817b4700 1 -- 172.21.15.16:0/2082116219 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c7c12ba80 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> v2:172.21.15.76:3300/0 conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> v2:172.21.15.76:3300/0 conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=READY pgs=592 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c7a7fc700 1 -- 172.21.15.16:0/2082116219 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c7000f130 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c7a7fc700 1 -- 172.21.15.16:0/2082116219 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c7000ed50 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c7a7fc700 1 -- 172.21.15.16:0/2082116219 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c7000f7b0 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.146+0000 7f8c83a18700 1 -- 172.21.15.16:0/2082116219 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8c7c12ba80 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.147+0000 7f8c83a18700 1 -- 172.21.15.16:0/2082116219 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8c7c12ba80 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.148+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.149+0000 7f8c7a7fc700 1 -- 172.21.15.16:0/2082116219 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8c7000ef40 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.149+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.149+0000 7f8c7a7fc700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c5c023680 0x7f8c5c025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:05.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.149+0000 7f8c7a7fc700 1 -- 172.21.15.16:0/2082116219 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f8c70045d90 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.149+0000 7f8c81fb5700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c5c023680 0x7f8c5c025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:05.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.150+0000 7f8c81fb5700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c5c023680 0x7f8c5c025af0 crc :-1 s=READY pgs=356 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:05.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.164+0000 7f8c83a18700 1 -- 172.21.15.16:0/2082116219 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8c7c12ba80 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.174+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.174+0000 7f8c7a7fc700 1 -- 172.21.15.16:0/2082116219 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8c7000ea50 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.440+0000 7f8c83a18700 1 -- 172.21.15.16:0/2082116219 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8c7c12ba80 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.441+0000 7f8c817b4700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=THROTTLE_DONE pgs=592 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:05.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.441+0000 7f8c7a7fc700 1 -- 172.21.15.16:0/2082116219 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f8c7000ef40 con 0x7f8c7c09e2c0 2019-12-05T06:28:05.431 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2019-12-05T06:28:05.431 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:05.432 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.447+0000 7f8c83a18700 1 -- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c5c023680 msgr2=0x7f8c5c025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.447+0000 7f8c83a18700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c5c023680 0x7f8c5c025af0 crc :-1 s=READY pgs=356 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.447+0000 7f8c83a18700 1 -- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 msgr2=0x7f8c7c09dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.447+0000 7f8c83a18700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 secure :-1 s=READY pgs=592 cs=0 l=1 rx=0x7f8c70010380 tx=0x7f8c70010720).stop 2019-12-05T06:28:05.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.447+0000 7f8c83a18700 1 -- 172.21.15.16:0/2082116219 shutdown_connections 2019-12-05T06:28:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.447+0000 7f8c83a18700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8c5c023680 0x7f8c5c025af0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.447+0000 7f8c83a18700 1 --2- 172.21.15.16:0/2082116219 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8c7c09e2c0 0x7f8c7c09dba0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.447+0000 7f8c83a18700 1 -- 172.21.15.16:0/2082116219 shutdown_connections 2019-12-05T06:28:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.447+0000 7f8c83a18700 1 -- 172.21.15.16:0/2082116219 wait complete. 2019-12-05T06:28:05.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:05.447+0000 7f8c83a18700 1 -- 172.21.15.16:0/2082116219 >> 172.21.15.16:0/2082116219 conn(0x7f8c7c06d260 msgr2=0x7f8c7c09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:05.446 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:07.447 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:07.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.724+0000 7ff657bdb700 1 Processor -- start 2019-12-05T06:28:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.724+0000 7ff657bdb700 1 -- start start 2019-12-05T06:28:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.725+0000 7ff657bdb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff65008c6d0 0x7ff65008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.725+0000 7ff657bdb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff65008fef0 0x7ff65008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:07.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.725+0000 7ff657bdb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6500958e0 con 0x7ff65008ef20 2019-12-05T06:28:07.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.725+0000 7ff657bdb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6500a3f70 con 0x7ff65008fef0 2019-12-05T06:28:07.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.725+0000 7ff657bdb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff65008d830 con 0x7ff65008c6d0 2019-12-05T06:28:07.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.726+0000 7ff656178700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff65008c6d0 0x7ff65008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:07.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.726+0000 7ff655977700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff65008fef0 0x7ff65008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:07.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.726+0000 7ff656979700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff65008ef20 0x7ff65008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:60454/0 (socket says 172.21.15.16:60454) 2019-12-05T06:28:07.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.726+0000 7ff656979700 1 -- 172.21.15.16:0/3887877183 learned_addr learned my addr 172.21.15.16:0/3887877183 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:07.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.726+0000 7ff655977700 1 -- 172.21.15.16:0/3887877183 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff65008c6d0 msgr2=0x7ff65008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:07.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.726+0000 7ff655977700 1 --2- 172.21.15.16:0/3887877183 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff65008c6d0 0x7ff65008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:07.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.726+0000 7ff655977700 1 -- 172.21.15.16:0/3887877183 >> v1:172.21.15.76:6789/0 conn(0x7ff65008ef20 legacy=0x7ff65008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:07.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff655977700 1 -- 172.21.15.16:0/3887877183 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff65008d540 con 0x7ff65008fef0 2019-12-05T06:28:07.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.726+0000 7ff655176700 1 -- 172.21.15.16:0/3887877183 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500305117 0 0) 0x7ff640001100 con 0x7ff65008ef20 2019-12-05T06:28:07.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff655977700 1 --2- 172.21.15.16:0/3887877183 >> v2:172.21.15.76:3300/0 conn(0x7ff65008fef0 0x7ff65008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff64c00af50 tx=0x7ff64c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff655977700 1 --2- 172.21.15.16:0/3887877183 >> v2:172.21.15.76:3300/0 conn(0x7ff65008fef0 0x7ff65008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff64c00af50 tx=0x7ff64c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff655977700 1 --2- 172.21.15.16:0/3887877183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65008fef0 0x7ff65008d000 secure :-1 s=READY pgs=594 cs=0 l=1 rx=0x7ff64c00af50 tx=0x7ff64c00b6e0).ready entity=mon.1 client_cookie=6a742cb23875b6e4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:07.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff655977700 1 --2- 172.21.15.16:0/3887877183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65008fef0 0x7ff65008d000 secure :-1 s=THROTTLE_DONE pgs=594 cs=0 l=1 rx=0x7ff64c00af50 tx=0x7ff64c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff655176700 1 -- 172.21.15.16:0/3887877183 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff64c007210 con 0x7ff65008fef0 2019-12-05T06:28:07.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff655977700 1 --2- 172.21.15.16:0/3887877183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65008fef0 0x7ff65008d000 secure :-1 s=THROTTLE_DONE pgs=594 cs=0 l=1 rx=0x7ff64c00af50 tx=0x7ff64c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff655977700 1 --2- 172.21.15.16:0/3887877183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65008fef0 0x7ff65008d000 secure :-1 s=THROTTLE_DONE pgs=594 cs=0 l=1 rx=0x7ff64c00af50 tx=0x7ff64c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff655176700 1 -- 172.21.15.16:0/3887877183 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64c0099b0 con 0x7ff65008fef0 2019-12-05T06:28:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff655176700 1 -- 172.21.15.16:0/3887877183 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff64c00a080 con 0x7ff65008fef0 2019-12-05T06:28:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff657bdb700 1 -- 172.21.15.16:0/3887877183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65008fef0 msgr2=0x7ff65008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.727+0000 7ff657bdb700 1 --2- 172.21.15.16:0/3887877183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65008fef0 0x7ff65008d000 secure :-1 s=READY pgs=594 cs=0 l=1 rx=0x7ff64c00af50 tx=0x7ff64c00b6e0).stop 2019-12-05T06:28:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.729+0000 7ff657bdb700 1 -- 172.21.15.16:0/3887877183 shutdown_connections 2019-12-05T06:28:07.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.729+0000 7ff657bdb700 1 --2- 172.21.15.16:0/3887877183 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65008fef0 0x7ff65008d000 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:07.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.729+0000 7ff657bdb700 1 --2- 172.21.15.16:0/3887877183 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff65008c6d0 0x7ff65008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:07.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.729+0000 7ff657bdb700 1 -- 172.21.15.16:0/3887877183 shutdown_connections 2019-12-05T06:28:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.729+0000 7ff657bdb700 1 -- 172.21.15.16:0/3887877183 wait complete. 2019-12-05T06:28:07.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.729+0000 7ff657bdb700 1 -- 172.21.15.16:0/3887877183 >> 172.21.15.16:0/3887877183 conn(0x7ff650061940 msgr2=0x7ff650096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.733+0000 7ff657bdb700 1 Processor -- start 2019-12-05T06:28:07.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.733+0000 7ff657bdb700 1 -- start start 2019-12-05T06:28:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.734+0000 7ff657bdb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff65009e7d0 0x7ff65009e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.734+0000 7ff657bdb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65012cb90 con 0x7ff650156e70 2019-12-05T06:28:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.734+0000 7ff657bdb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff65012d040 con 0x7ff6501302f0 2019-12-05T06:28:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.734+0000 7ff657bdb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff650131ff0 con 0x7ff65009e7d0 2019-12-05T06:28:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.734+0000 7ff656979700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff650156e70 0x7ff650159270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39158/0 (socket says 172.21.15.16:39158) 2019-12-05T06:28:07.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.734+0000 7ff656979700 1 -- 172.21.15.16:0/437602668 learned_addr learned my addr 172.21.15.16:0/437602668 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.734+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> v2:172.21.15.76:3300/0 conn(0x7ff65009e7d0 0x7ff65009e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.734+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 669386714 0 0) 0x7ff640001310 con 0x7ff650156e70 2019-12-05T06:28:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff638002e80 con 0x7ff650156e70 2019-12-05T06:28:07.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1368136986 0 0) 0x7ff6480012b0 con 0x7ff6501302f0 2019-12-05T06:28:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6380030b0 con 0x7ff6501302f0 2019-12-05T06:28:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 237514241 0 0) 0x7ff640001510 con 0x7ff650156e70 2019-12-05T06:28:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff655977700 1 -- 172.21.15.16:0/437602668 >> v1:172.21.15.76:6789/0 conn(0x7ff6501302f0 legacy=0x7ff65009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff655977700 1 -- 172.21.15.16:0/437602668 >> v1:172.21.15.16:6789/0 conn(0x7ff650156e70 legacy=0x7ff650159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff655977700 1 -- 172.21.15.16:0/437602668 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff650132680 con 0x7ff65009e7d0 2019-12-05T06:28:07.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> v2:172.21.15.76:3300/0 conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> v2:172.21.15.76:3300/0 conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=READY pgs=596 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.735+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.736+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff64c00a860 con 0x7ff65009e7d0 2019-12-05T06:28:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.736+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.736+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.736+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff64c0009e0 con 0x7ff65009e7d0 2019-12-05T06:28:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.736+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff64c009d60 con 0x7ff65009e7d0 2019-12-05T06:28:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.736+0000 7ff657bdb700 1 -- 172.21.15.16:0/437602668 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff650132680 con 0x7ff65009e7d0 2019-12-05T06:28:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.736+0000 7ff657bdb700 1 -- 172.21.15.16:0/437602668 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff650132680 con 0x7ff65009e7d0 2019-12-05T06:28:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.737+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.737+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff64c009790 con 0x7ff65009e7d0 2019-12-05T06:28:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.737+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.738+0000 7ff646ffd700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff638023650 0x7ff638025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.738+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7ff64c045db0 con 0x7ff65009e7d0 2019-12-05T06:28:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.738+0000 7ff656178700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff638023650 0x7ff638025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.738+0000 7ff656178700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff638023650 0x7ff638025ae0 crc :-1 s=READY pgs=357 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.739+0000 7ff657bdb700 1 -- 172.21.15.16:0/437602668 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff650132680 con 0x7ff65009e7d0 2019-12-05T06:28:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.748+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.748+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff64c009580 con 0x7ff65009e7d0 2019-12-05T06:28:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.971+0000 7ff657bdb700 1 -- 172.21.15.16:0/437602668 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff650132680 con 0x7ff65009e7d0 2019-12-05T06:28:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.972+0000 7ff655977700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=THROTTLE_DONE pgs=596 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.972+0000 7ff646ffd700 1 -- 172.21.15.16:0/437602668 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2111 (secure 0 0 0) 0x7ff64c009790 con 0x7ff65009e7d0 2019-12-05T06:28:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2019-12-05T06:28:07.963 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:07.964 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:07.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.977+0000 7ff657bdb700 1 -- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff638023650 msgr2=0x7ff638025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:07.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.977+0000 7ff657bdb700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff638023650 0x7ff638025ae0 crc :-1 s=READY pgs=357 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:07.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.977+0000 7ff657bdb700 1 -- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 msgr2=0x7ff65009e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:07.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.977+0000 7ff657bdb700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 0x7ff65009e000 secure :-1 s=READY pgs=596 cs=0 l=1 rx=0x7ff64c0103a0 tx=0x7ff64c010740).stop 2019-12-05T06:28:07.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.977+0000 7ff657bdb700 1 -- 172.21.15.16:0/437602668 shutdown_connections 2019-12-05T06:28:07.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.977+0000 7ff657bdb700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff638023650 0x7ff638025ae0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:07.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.977+0000 7ff657bdb700 1 --2- 172.21.15.16:0/437602668 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff65009e7d0 0x7ff65009e000 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:07.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.977+0000 7ff657bdb700 1 -- 172.21.15.16:0/437602668 shutdown_connections 2019-12-05T06:28:07.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.977+0000 7ff657bdb700 1 -- 172.21.15.16:0/437602668 wait complete. 2019-12-05T06:28:07.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:07.977+0000 7ff657bdb700 1 -- 172.21.15.16:0/437602668 >> 172.21.15.16:0/437602668 conn(0x7ff6500a2850 msgr2=0x7ff65006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:07.984 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:09.985 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:10.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.253+0000 7f1e995f8700 1 Processor -- start 2019-12-05T06:28:10.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.253+0000 7f1e995f8700 1 -- start start 2019-12-05T06:28:10.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.254+0000 7f1e995f8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1e9408c6d0 0x7f1e9408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:10.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.255+0000 7f1e995f8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e9408f720 0x7f1e9408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:10.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.255+0000 7f1e995f8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e9408fb80 con 0x7f1e9408fef0 2019-12-05T06:28:10.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.255+0000 7f1e995f8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9408df90 con 0x7f1e9408f720 2019-12-05T06:28:10.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.255+0000 7f1e995f8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e940a3f70 con 0x7f1e9408c6d0 2019-12-05T06:28:10.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.255+0000 7f1e93fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e9408f720 0x7f1e9408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.255+0000 7f1e93fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1e9408f720 0x7f1e9408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54002/0 (socket says 172.21.15.16:54002) 2019-12-05T06:28:10.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.255+0000 7f1e93fff700 1 -- 172.21.15.16:0/361211775 learned_addr learned my addr 172.21.15.16:0/361211775 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:28:10.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.255+0000 7f1e937fe700 1 --2- 172.21.15.16:0/361211775 >> v2:172.21.15.76:3300/0 conn(0x7f1e9408c6d0 0x7f1e9408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e93fff700 1 -- 172.21.15.16:0/361211775 >> v2:172.21.15.76:3300/0 conn(0x7f1e9408c6d0 msgr2=0x7f1e9408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e93fff700 1 --2- 172.21.15.16:0/361211775 >> v2:172.21.15.76:3300/0 conn(0x7f1e9408c6d0 0x7f1e9408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e93fff700 1 -- 172.21.15.16:0/361211775 >> v1:172.21.15.76:6789/0 conn(0x7f1e9408fef0 legacy=0x7f1e9408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e93fff700 1 -- 172.21.15.16:0/361211775 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e9408d540 con 0x7f1e9408f720 2019-12-05T06:28:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e93fff700 1 --2- 172.21.15.16:0/361211775 >> v2:172.21.15.16:3300/0 conn(0x7f1e9408f720 0x7f1e9408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1e8400f730 tx=0x7f1e84007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e93fff700 1 --2- 172.21.15.16:0/361211775 >> v2:172.21.15.16:3300/0 conn(0x7f1e9408f720 0x7f1e9408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1e8400f730 tx=0x7f1e84007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e93fff700 1 --2- 172.21.15.16:0/361211775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9408f720 0x7f1e9408d000 secure :-1 s=READY pgs=822 cs=0 l=1 rx=0x7f1e8400f730 tx=0x7f1e84007d50).ready entity=mon.0 client_cookie=ae33371a90033d09 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:10.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e93fff700 1 --2- 172.21.15.16:0/361211775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9408f720 0x7f1e9408d000 secure :-1 s=THROTTLE_DONE pgs=822 cs=0 l=1 rx=0x7f1e8400f730 tx=0x7f1e84007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e93fff700 1 --2- 172.21.15.16:0/361211775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9408f720 0x7f1e9408d000 secure :-1 s=THROTTLE_DONE pgs=822 cs=0 l=1 rx=0x7f1e8400f730 tx=0x7f1e84007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e93fff700 1 --2- 172.21.15.16:0/361211775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9408f720 0x7f1e9408d000 secure :-1 s=THROTTLE_DONE pgs=822 cs=0 l=1 rx=0x7f1e8400f730 tx=0x7f1e84007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e927fc700 1 -- 172.21.15.16:0/361211775 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e84000a70 con 0x7f1e9408f720 2019-12-05T06:28:10.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e927fc700 1 -- 172.21.15.16:0/361211775 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e840010e0 con 0x7f1e9408f720 2019-12-05T06:28:10.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.256+0000 7f1e927fc700 1 -- 172.21.15.16:0/361211775 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e840017b0 con 0x7f1e9408f720 2019-12-05T06:28:10.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.257+0000 7f1e995f8700 1 -- 172.21.15.16:0/361211775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9408f720 msgr2=0x7f1e9408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.257+0000 7f1e995f8700 1 --2- 172.21.15.16:0/361211775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9408f720 0x7f1e9408d000 secure :-1 s=READY pgs=822 cs=0 l=1 rx=0x7f1e8400f730 tx=0x7f1e84007d50).stop 2019-12-05T06:28:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.258+0000 7f1e995f8700 1 -- 172.21.15.16:0/361211775 shutdown_connections 2019-12-05T06:28:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.258+0000 7f1e995f8700 1 --2- 172.21.15.16:0/361211775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1e9408f720 0x7f1e9408d000 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.258+0000 7f1e995f8700 1 --2- 172.21.15.16:0/361211775 >> v2:172.21.15.76:3300/0 conn(0x7f1e9408c6d0 0x7f1e9408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.258+0000 7f1e995f8700 1 -- 172.21.15.16:0/361211775 shutdown_connections 2019-12-05T06:28:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.258+0000 7f1e995f8700 1 -- 172.21.15.16:0/361211775 wait complete. 2019-12-05T06:28:10.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.258+0000 7f1e995f8700 1 -- 172.21.15.16:0/361211775 >> 172.21.15.16:0/361211775 conn(0x7f1e94061940 msgr2=0x7f1e94096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.263+0000 7f1e995f8700 1 Processor -- start 2019-12-05T06:28:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.263+0000 7f1e995f8700 1 -- start start 2019-12-05T06:28:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.265+0000 7f1e995f8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1e9406df10 0x7f1e94098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.265+0000 7f1e995f8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1e94156cb0 0x7f1e941590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.265+0000 7f1e995f8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e94089fb0 con 0x7f1e940992d0 2019-12-05T06:28:10.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.265+0000 7f1e995f8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e9408ad00 con 0x7f1e94156cb0 2019-12-05T06:28:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.265+0000 7f1e995f8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e94096c10 con 0x7f1e9406df10 2019-12-05T06:28:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.265+0000 7f1e92ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1e940992d0 0x7f1e94098840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39176/0 (socket says 172.21.15.16:39176) 2019-12-05T06:28:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.265+0000 7f1e92ffd700 1 -- 172.21.15.16:0/2401540004 learned_addr learned my addr 172.21.15.16:0/2401540004 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.265+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> v2:172.21.15.76:3300/0 conn(0x7f1e94156cb0 0x7f1e941590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.265+0000 7f1e937fe700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1e9406df10 0x7f1e94098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e8bfff700 1 -- 172.21.15.16:0/2401540004 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680084523 0 0) 0x7f1e7c0009b0 con 0x7f1e940992d0 2019-12-05T06:28:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e8bfff700 1 -- 172.21.15.16:0/2401540004 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1e78002e80 con 0x7f1e940992d0 2019-12-05T06:28:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e93fff700 1 -- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1e9406df10 msgr2=0x7f1e94098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1e9406df10 0x7f1e94098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e93fff700 1 -- 172.21.15.16:0/2401540004 >> v1:172.21.15.16:6789/0 conn(0x7f1e940992d0 legacy=0x7f1e94098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e93fff700 1 -- 172.21.15.16:0/2401540004 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e94097a00 con 0x7f1e94156cb0 2019-12-05T06:28:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> v2:172.21.15.76:3300/0 conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> v2:172.21.15.76:3300/0 conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=READY pgs=598 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:10.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.266+0000 7f1e8bfff700 1 -- 172.21.15.16:0/2401540004 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e840014a0 con 0x7f1e94156cb0 2019-12-05T06:28:10.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.267+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.267+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.267+0000 7f1e8bfff700 1 -- 172.21.15.16:0/2401540004 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e840010c0 con 0x7f1e94156cb0 2019-12-05T06:28:10.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.267+0000 7f1e995f8700 1 -- 172.21.15.16:0/2401540004 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e94097a00 con 0x7f1e94156cb0 2019-12-05T06:28:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.267+0000 7f1e8bfff700 1 -- 172.21.15.16:0/2401540004 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e84001a80 con 0x7f1e94156cb0 2019-12-05T06:28:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.267+0000 7f1e995f8700 1 -- 172.21.15.16:0/2401540004 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e94097a00 con 0x7f1e94156cb0 2019-12-05T06:28:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.268+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.268+0000 7f1e8bfff700 1 -- 172.21.15.16:0/2401540004 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1e840012b0 con 0x7f1e94156cb0 2019-12-05T06:28:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.268+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.269+0000 7f1e8bfff700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e78023670 0x7f1e78025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.269+0000 7f1e8bfff700 1 -- 172.21.15.16:0/2401540004 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7f1e84045ce0 con 0x7f1e94156cb0 2019-12-05T06:28:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.269+0000 7f1e937fe700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e78023670 0x7f1e78025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.269+0000 7f1e937fe700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e78023670 0x7f1e78025ae0 crc :-1 s=READY pgs=358 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:10.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.284+0000 7f1e995f8700 1 -- 172.21.15.16:0/2401540004 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e94097a00 con 0x7f1e94156cb0 2019-12-05T06:28:10.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.293+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.293+0000 7f1e8bfff700 1 -- 172.21.15.16:0/2401540004 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1e84000eb0 con 0x7f1e94156cb0 2019-12-05T06:28:10.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.528+0000 7f1e995f8700 1 -- 172.21.15.16:0/2401540004 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1e94097a00 con 0x7f1e94156cb0 2019-12-05T06:28:10.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.529+0000 7f1e93fff700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=THROTTLE_DONE pgs=598 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:10.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.529+0000 7f1e8bfff700 1 -- 172.21.15.16:0/2401540004 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2111 (secure 0 0 0) 0x7f1e840011b0 con 0x7f1e94156cb0 2019-12-05T06:28:10.521 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2019-12-05T06:28:10.521 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:10.521 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 -- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e78023670 msgr2=0x7f1e78025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e78023670 0x7f1e78025ae0 crc :-1 s=READY pgs=358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 -- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 msgr2=0x7f1e941590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 0x7f1e941590b0 secure :-1 s=READY pgs=598 cs=0 l=1 rx=0x7f1e84010350 tx=0x7f1e840106f0).stop 2019-12-05T06:28:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 -- 172.21.15.16:0/2401540004 shutdown_connections 2019-12-05T06:28:10.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1e78023670 0x7f1e78025ae0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1e94156cb0 0x7f1e941590b0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 --2- 172.21.15.16:0/2401540004 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1e9406df10 0x7f1e94098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 -- 172.21.15.16:0/2401540004 shutdown_connections 2019-12-05T06:28:10.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 -- 172.21.15.16:0/2401540004 wait complete. 2019-12-05T06:28:10.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:10.534+0000 7f1e995f8700 1 -- 172.21.15.16:0/2401540004 >> 172.21.15.16:0/2401540004 conn(0x7f1e940714f0 msgr2=0x7f1e9409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:10.535 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:12.536 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:12.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.819+0000 7ff0463f5700 1 Processor -- start 2019-12-05T06:28:12.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.819+0000 7ff0463f5700 1 -- start start 2019-12-05T06:28:12.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.821+0000 7ff0463f5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 0x7ff04008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:12.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.821+0000 7ff0463f5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff04008f720 0x7ff04008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:12.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.821+0000 7ff0463f5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0400958e0 con 0x7ff04008fef0 2019-12-05T06:28:12.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.821+0000 7ff0463f5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0400a3f70 con 0x7ff04008f720 2019-12-05T06:28:12.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.821+0000 7ff0463f5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff04008d830 con 0x7ff04008c6d0 2019-12-05T06:28:12.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.821+0000 7ff03ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff04008fef0 0x7ff04008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39184/0 (socket says 172.21.15.16:39184) 2019-12-05T06:28:12.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.821+0000 7ff03ffff700 1 -- 172.21.15.16:0/2734985502 learned_addr learned my addr 172.21.15.16:0/2734985502 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:12.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.821+0000 7ff044992700 1 --2- 172.21.15.16:0/2734985502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 0x7ff04008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:12.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.821+0000 7ff045193700 1 --2- 172.21.15.16:0/2734985502 >> v2:172.21.15.76:3300/0 conn(0x7ff04008f720 0x7ff04008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:12.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.822+0000 7ff044992700 1 -- 172.21.15.16:0/2734985502 >> v2:172.21.15.76:3300/0 conn(0x7ff04008f720 msgr2=0x7ff04008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.822+0000 7ff044992700 1 --2- 172.21.15.16:0/2734985502 >> v2:172.21.15.76:3300/0 conn(0x7ff04008f720 0x7ff04008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.822+0000 7ff044992700 1 -- 172.21.15.16:0/2734985502 >> v1:172.21.15.16:6789/0 conn(0x7ff04008fef0 legacy=0x7ff04008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.822+0000 7ff03f7fe700 1 -- 172.21.15.16:0/2734985502 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3064845690 0 0) 0x7ff034001150 con 0x7ff04008fef0 2019-12-05T06:28:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.822+0000 7ff044992700 1 -- 172.21.15.16:0/2734985502 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff04008fbc0 con 0x7ff04008c6d0 2019-12-05T06:28:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.822+0000 7ff044992700 1 --2- 172.21.15.16:0/2734985502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 0x7ff04008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff03800b2d0 tx=0x7ff03800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.823+0000 7ff044992700 1 --2- 172.21.15.16:0/2734985502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 0x7ff04008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff03800b2d0 tx=0x7ff03800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.823+0000 7ff044992700 1 --2- 172.21.15.16:0/2734985502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 0x7ff04008cac0 secure :-1 s=READY pgs=201 cs=0 l=1 rx=0x7ff03800b2d0 tx=0x7ff03800ba60).ready entity=mon.2 client_cookie=f9718cdb35eadbf4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.823+0000 7ff044992700 1 --2- 172.21.15.16:0/2734985502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 0x7ff04008cac0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7ff03800b2d0 tx=0x7ff03800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.823+0000 7ff03f7fe700 1 -- 172.21.15.16:0/2734985502 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff038007590 con 0x7ff04008c6d0 2019-12-05T06:28:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.823+0000 7ff044992700 1 --2- 172.21.15.16:0/2734985502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 0x7ff04008cac0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7ff03800b2d0 tx=0x7ff03800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.823+0000 7ff044992700 1 --2- 172.21.15.16:0/2734985502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 0x7ff04008cac0 secure :-1 s=THROTTLE_DONE pgs=201 cs=0 l=1 rx=0x7ff03800b2d0 tx=0x7ff03800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.823+0000 7ff03f7fe700 1 -- 172.21.15.16:0/2734985502 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff038009d30 con 0x7ff04008c6d0 2019-12-05T06:28:12.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.823+0000 7ff03f7fe700 1 -- 172.21.15.16:0/2734985502 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff03800a400 con 0x7ff04008c6d0 2019-12-05T06:28:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.823+0000 7ff0463f5700 1 -- 172.21.15.16:0/2734985502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 msgr2=0x7ff04008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.823+0000 7ff0463f5700 1 --2- 172.21.15.16:0/2734985502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 0x7ff04008cac0 secure :-1 s=READY pgs=201 cs=0 l=1 rx=0x7ff03800b2d0 tx=0x7ff03800ba60).stop 2019-12-05T06:28:12.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.825+0000 7ff0463f5700 1 -- 172.21.15.16:0/2734985502 shutdown_connections 2019-12-05T06:28:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.825+0000 7ff0463f5700 1 --2- 172.21.15.16:0/2734985502 >> v2:172.21.15.76:3300/0 conn(0x7ff04008f720 0x7ff04008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.825+0000 7ff0463f5700 1 --2- 172.21.15.16:0/2734985502 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff04008c6d0 0x7ff04008cac0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.825+0000 7ff0463f5700 1 -- 172.21.15.16:0/2734985502 shutdown_connections 2019-12-05T06:28:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.825+0000 7ff0463f5700 1 -- 172.21.15.16:0/2734985502 wait complete. 2019-12-05T06:28:12.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.825+0000 7ff0463f5700 1 -- 172.21.15.16:0/2734985502 >> 172.21.15.16:0/2734985502 conn(0x7ff040061940 msgr2=0x7ff040096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:12.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.830+0000 7ff0463f5700 1 Processor -- start 2019-12-05T06:28:12.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.830+0000 7ff0463f5700 1 -- start start 2019-12-05T06:28:12.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.832+0000 7ff0463f5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff04009e000 0x7ff04015a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:12.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.832+0000 7ff0463f5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04009e440 con 0x7ff04009e7d0 2019-12-05T06:28:12.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.832+0000 7ff0463f5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff04012cb90 con 0x7ff0401302f0 2019-12-05T06:28:12.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.832+0000 7ff0463f5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff040131ff0 con 0x7ff04009e000 2019-12-05T06:28:12.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.832+0000 7ff045193700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff04009e000 0x7ff04015a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:12.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.832+0000 7ff045193700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff04009e000 0x7ff04015a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54024/0 (socket says 172.21.15.16:54024) 2019-12-05T06:28:12.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.832+0000 7ff045193700 1 -- 172.21.15.16:0/270145183 learned_addr learned my addr 172.21.15.16:0/270145183 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:28:12.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.833+0000 7ff03d7fa700 1 -- 172.21.15.16:0/270145183 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1020498714 0 0) 0x7ff03800af30 con 0x7ff0401302f0 2019-12-05T06:28:12.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.833+0000 7ff045193700 1 -- 172.21.15.16:0/270145183 >> v1:172.21.15.76:6789/0 conn(0x7ff0401302f0 legacy=0x7ff04009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:12.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.833+0000 7ff045193700 1 -- 172.21.15.16:0/270145183 >> v1:172.21.15.16:6789/0 conn(0x7ff04009e7d0 legacy=0x7ff040156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:12.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.833+0000 7ff045193700 1 -- 172.21.15.16:0/270145183 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff040132680 con 0x7ff04009e000 2019-12-05T06:28:12.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.833+0000 7ff045193700 1 --2- 172.21.15.16:0/270145183 >> v2:172.21.15.16:3300/0 conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.833+0000 7ff045193700 1 --2- 172.21.15.16:0/270145183 >> v2:172.21.15.16:3300/0 conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.833+0000 7ff045193700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=READY pgs=826 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:12.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.833+0000 7ff045193700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=THROTTLE_DONE pgs=826 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.833+0000 7ff03d7fa700 1 -- 172.21.15.16:0/270145183 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff03000f090 con 0x7ff04009e000 2019-12-05T06:28:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.834+0000 7ff045193700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=THROTTLE_DONE pgs=826 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.834+0000 7ff03d7fa700 1 -- 172.21.15.16:0/270145183 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff030000a70 con 0x7ff04009e000 2019-12-05T06:28:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.834+0000 7ff045193700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=THROTTLE_DONE pgs=826 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.834+0000 7ff0463f5700 1 -- 172.21.15.16:0/270145183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff040132680 con 0x7ff04009e000 2019-12-05T06:28:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.834+0000 7ff03d7fa700 1 -- 172.21.15.16:0/270145183 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff03000f1d0 con 0x7ff04009e000 2019-12-05T06:28:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.834+0000 7ff0463f5700 1 -- 172.21.15.16:0/270145183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff040132680 con 0x7ff04009e000 2019-12-05T06:28:12.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.835+0000 7ff045193700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=THROTTLE_DONE pgs=826 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.836+0000 7ff03d7fa700 1 -- 172.21.15.16:0/270145183 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff03000edf0 con 0x7ff04009e000 2019-12-05T06:28:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.836+0000 7ff045193700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=THROTTLE_DONE pgs=826 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.836+0000 7ff03d7fa700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff020023630 0x7ff020025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.836+0000 7ff03d7fa700 1 -- 172.21.15.16:0/270145183 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 7235+0+0 (secure 0 0 0) 0x7ff030045e90 con 0x7ff04009e000 2019-12-05T06:28:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.836+0000 7ff044992700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff020023630 0x7ff020025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.837+0000 7ff044992700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff020023630 0x7ff020025ac0 crc :-1 s=READY pgs=360 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:12.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.850+0000 7ff0463f5700 1 -- 172.21.15.16:0/270145183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff040132680 con 0x7ff04009e000 2019-12-05T06:28:12.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.857+0000 7ff045193700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=THROTTLE_DONE pgs=826 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:12.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:12.857+0000 7ff03d7fa700 1 -- 172.21.15.16:0/270145183 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff03000eaa0 con 0x7ff04009e000 2019-12-05T06:28:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.110+0000 7ff0463f5700 1 -- 172.21.15.16:0/270145183 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff040132680 con 0x7ff04009e000 2019-12-05T06:28:13.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.110+0000 7ff045193700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=THROTTLE_DONE pgs=826 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.110+0000 7ff03d7fa700 1 -- 172.21.15.16:0/270145183 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 102 v102) v1 ==== 95+0+2111 (secure 0 0 0) 0x7ff03000ed50 con 0x7ff04009e000 2019-12-05T06:28:13.101 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 102 2019-12-05T06:28:13.101 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:13.101 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":102,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6264,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":102}],"filesystems":[{"mdsmap":{"epoch":101,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:27:28.203340+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":63,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6165},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6165":{"gid":6165,"name":"a","rank":0,"incarnation":98,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/2514381299","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2514381299},{"type":"v1","addr":"172.21.15.16:6833","nonce":2514381299}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.113+0000 7ff0463f5700 1 -- 172.21.15.16:0/270145183 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff020023630 msgr2=0x7ff020025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.113+0000 7ff0463f5700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff020023630 0x7ff020025ac0 crc :-1 s=READY pgs=360 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.113+0000 7ff0463f5700 1 -- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 msgr2=0x7ff04015a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:13.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.113+0000 7ff0463f5700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 0x7ff04015a690 secure :-1 s=READY pgs=826 cs=0 l=1 rx=0x7ff030010450 tx=0x7ff0300107f0).stop 2019-12-05T06:28:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.113+0000 7ff0463f5700 1 -- 172.21.15.16:0/270145183 shutdown_connections 2019-12-05T06:28:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.113+0000 7ff0463f5700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff020023630 0x7ff020025ac0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.113+0000 7ff0463f5700 1 --2- 172.21.15.16:0/270145183 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff04009e000 0x7ff04015a690 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.114+0000 7ff0463f5700 1 -- 172.21.15.16:0/270145183 shutdown_connections 2019-12-05T06:28:13.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.114+0000 7ff0463f5700 1 -- 172.21.15.16:0/270145183 wait complete. 2019-12-05T06:28:13.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:13.114+0000 7ff0463f5700 1 -- 172.21.15.16:0/270145183 >> 172.21.15.16:0/270145183 conn(0x7ff0400a2850 msgr2=0x7ff04006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:13.114 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:28:15.114 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:15.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.385+0000 7f8835942700 1 Processor -- start 2019-12-05T06:28:15.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.385+0000 7f8835942700 1 -- start start 2019-12-05T06:28:15.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.386+0000 7f8835942700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f883008c6d0 0x7f883008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:15.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.387+0000 7f8835942700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f883008fb80 con 0x7f883008fef0 2019-12-05T06:28:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.387+0000 7f8835942700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88300958e0 con 0x7f883008f720 2019-12-05T06:28:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.387+0000 7f8835942700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f883008df90 con 0x7f883008c6d0 2019-12-05T06:28:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.387+0000 7f882effd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f883008fef0 0x7f883008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39196/0 (socket says 172.21.15.16:39196) 2019-12-05T06:28:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.387+0000 7f882effd700 1 -- 172.21.15.16:0/2968160779 learned_addr learned my addr 172.21.15.16:0/2968160779 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.387+0000 7f882f7fe700 1 --2- 172.21.15.16:0/2968160779 >> v2:172.21.15.76:3300/0 conn(0x7f883008c6d0 0x7f883008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.387+0000 7f882e7fc700 1 -- 172.21.15.16:0/2968160779 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852113939 0 0) 0x7f8828000d30 con 0x7f883008f720 2019-12-05T06:28:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.387+0000 7f882e7fc700 1 -- 172.21.15.16:0/2968160779 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f881c002e80 con 0x7f883008f720 2019-12-05T06:28:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.387+0000 7f882e7fc700 1 -- 172.21.15.16:0/2968160779 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 705327846 0 0) 0x7f8818001150 con 0x7f883008fef0 2019-12-05T06:28:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882f7fe700 1 -- 172.21.15.16:0/2968160779 >> v1:172.21.15.76:6789/0 conn(0x7f883008f720 legacy=0x7f88301533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882f7fe700 1 -- 172.21.15.16:0/2968160779 >> v1:172.21.15.16:6789/0 conn(0x7f883008fef0 legacy=0x7f883008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882f7fe700 1 -- 172.21.15.16:0/2968160779 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883008e620 con 0x7f883008c6d0 2019-12-05T06:28:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882f7fe700 1 --2- 172.21.15.16:0/2968160779 >> v2:172.21.15.76:3300/0 conn(0x7f883008c6d0 0x7f883008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f882000af50 tx=0x7f882000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882f7fe700 1 --2- 172.21.15.16:0/2968160779 >> v2:172.21.15.76:3300/0 conn(0x7f883008c6d0 0x7f883008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f882000af50 tx=0x7f882000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882f7fe700 1 --2- 172.21.15.16:0/2968160779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f883008c6d0 0x7f883008cac0 secure :-1 s=READY pgs=601 cs=0 l=1 rx=0x7f882000af50 tx=0x7f882000b760).ready entity=mon.1 client_cookie=de33db35442d3261 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882f7fe700 1 --2- 172.21.15.16:0/2968160779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f883008c6d0 0x7f883008cac0 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7f882000af50 tx=0x7f882000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882f7fe700 1 --2- 172.21.15.16:0/2968160779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f883008c6d0 0x7f883008cac0 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7f882000af50 tx=0x7f882000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882e7fc700 1 -- 172.21.15.16:0/2968160779 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8820000960 con 0x7f883008c6d0 2019-12-05T06:28:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882f7fe700 1 --2- 172.21.15.16:0/2968160779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f883008c6d0 0x7f883008cac0 secure :-1 s=THROTTLE_DONE pgs=601 cs=0 l=1 rx=0x7f882000af50 tx=0x7f882000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882e7fc700 1 -- 172.21.15.16:0/2968160779 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88200098e0 con 0x7f883008c6d0 2019-12-05T06:28:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.388+0000 7f882e7fc700 1 -- 172.21.15.16:0/2968160779 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8820009eb0 con 0x7f883008c6d0 2019-12-05T06:28:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.389+0000 7f8835942700 1 -- 172.21.15.16:0/2968160779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f883008c6d0 msgr2=0x7f883008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.389+0000 7f8835942700 1 --2- 172.21.15.16:0/2968160779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f883008c6d0 0x7f883008cac0 secure :-1 s=READY pgs=601 cs=0 l=1 rx=0x7f882000af50 tx=0x7f882000b760).stop 2019-12-05T06:28:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.390+0000 7f8835942700 1 -- 172.21.15.16:0/2968160779 shutdown_connections 2019-12-05T06:28:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.390+0000 7f8835942700 1 --2- 172.21.15.16:0/2968160779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f883008c6d0 0x7f883008cac0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.391+0000 7f8835942700 1 -- 172.21.15.16:0/2968160779 shutdown_connections 2019-12-05T06:28:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.391+0000 7f8835942700 1 -- 172.21.15.16:0/2968160779 wait complete. 2019-12-05T06:28:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.391+0000 7f8835942700 1 -- 172.21.15.16:0/2968160779 >> 172.21.15.16:0/2968160779 conn(0x7f8830061940 msgr2=0x7f8830096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:15.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.396+0000 7f8835942700 1 Processor -- start 2019-12-05T06:28:15.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.396+0000 7f8835942700 1 -- start start 2019-12-05T06:28:15.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.397+0000 7f8835942700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:15.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.397+0000 7f8835942700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f883012c740 0x7f883012bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:15.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.397+0000 7f8835942700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8830128b20 con 0x7f883012c1f0 2019-12-05T06:28:15.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.397+0000 7f8835942700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8830129db0 con 0x7f883012c740 2019-12-05T06:28:15.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.397+0000 7f8835942700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f883012a080 con 0x7f88301325e0 2019-12-05T06:28:15.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.398+0000 7f882effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f883012c740 0x7f883012bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.398+0000 7f882f7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.398+0000 7f882effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f883012c740 0x7f883012bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54032/0 (socket says 172.21.15.16:54032) 2019-12-05T06:28:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.398+0000 7f882effd700 1 -- 172.21.15.16:0/974353198 learned_addr learned my addr 172.21.15.16:0/974353198 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:28:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.398+0000 7f882f7fe700 1 -- 172.21.15.16:0/974353198 >> v2:172.21.15.16:3300/0 conn(0x7f883012c740 msgr2=0x7f883012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.398+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> v2:172.21.15.16:3300/0 conn(0x7f883012c740 0x7f883012bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.398+0000 7f882f7fe700 1 -- 172.21.15.16:0/974353198 >> v1:172.21.15.16:6789/0 conn(0x7f883012c1f0 legacy=0x7f883006dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f882f7fe700 1 -- 172.21.15.16:0/974353198 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f883014a820 con 0x7f88301325e0 2019-12-05T06:28:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=READY pgs=204 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f8827fff700 1 -- 172.21.15.16:0/974353198 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8820000940 con 0x7f88301325e0 2019-12-05T06:28:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f8827fff700 1 -- 172.21.15.16:0/974353198 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8820009770 con 0x7f88301325e0 2019-12-05T06:28:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f8827fff700 1 -- 172.21.15.16:0/974353198 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8820009e00 con 0x7f88301325e0 2019-12-05T06:28:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.399+0000 7f8835942700 1 -- 172.21.15.16:0/974353198 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f883012ae70 con 0x7f88301325e0 2019-12-05T06:28:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.400+0000 7f8835942700 1 -- 172.21.15.16:0/974353198 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f883014a820 con 0x7f88301325e0 2019-12-05T06:28:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.401+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.402+0000 7f8827fff700 1 -- 172.21.15.16:0/974353198 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8820009960 con 0x7f88301325e0 2019-12-05T06:28:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.402+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.402+0000 7f8827fff700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8814023630 0x7f8814025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.402+0000 7f8827fff700 1 -- 172.21.15.16:0/974353198 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f8820045d40 con 0x7f88301325e0 2019-12-05T06:28:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.403+0000 7f882effd700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8814023630 0x7f8814025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:15.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.403+0000 7f882effd700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8814023630 0x7f8814025ac0 crc :-1 s=READY pgs=361 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:15.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.416+0000 7f8835942700 1 -- 172.21.15.16:0/974353198 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f883012ae70 con 0x7f88301325e0 2019-12-05T06:28:15.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.427+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.427+0000 7f8827fff700 1 -- 172.21.15.16:0/974353198 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f882000b5b0 con 0x7f88301325e0 2019-12-05T06:28:15.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.683+0000 7f8835942700 1 -- 172.21.15.16:0/974353198 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f883014a820 con 0x7f88301325e0 2019-12-05T06:28:15.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.684+0000 7f882f7fe700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=THROTTLE_DONE pgs=204 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.684+0000 7f8827fff700 1 -- 172.21.15.16:0/974353198 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 103 v103) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f8820009960 con 0x7f88301325e0 2019-12-05T06:28:15.681 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 103 2019-12-05T06:28:15.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.689+0000 7f8835942700 1 -- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8814023630 msgr2=0x7f8814025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.689+0000 7f8835942700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8814023630 0x7f8814025ac0 crc :-1 s=READY pgs=361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.689+0000 7f8835942700 1 -- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 msgr2=0x7f883012b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.689+0000 7f8835942700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 secure :-1 s=READY pgs=204 cs=0 l=1 rx=0x7f8820010350 tx=0x7f8820010200).stop 2019-12-05T06:28:15.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.690+0000 7f8835942700 1 -- 172.21.15.16:0/974353198 shutdown_connections 2019-12-05T06:28:15.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.690+0000 7f8835942700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8814023630 0x7f8814025ac0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.690+0000 7f8835942700 1 --2- 172.21.15.16:0/974353198 >> v2:172.21.15.16:3300/0 conn(0x7f883012c740 0x7f883012bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.690+0000 7f8835942700 1 --2- 172.21.15.16:0/974353198 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f88301325e0 0x7f883012b770 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.690+0000 7f8835942700 1 -- 172.21.15.16:0/974353198 shutdown_connections 2019-12-05T06:28:15.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.690+0000 7f8835942700 1 -- 172.21.15.16:0/974353198 wait complete. 2019-12-05T06:28:15.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.690+0000 7f8835942700 1 -- 172.21.15.16:0/974353198 >> 172.21.15.16:0/974353198 conn(0x7f88300a2850 msgr2=0x7f883006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:15.683 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:15.684 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":103,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":103,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:13.211909+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:15.698 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:28:15.698 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:28:15.698 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:15.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.933+0000 7fd6f6a89700 1 Processor -- start 2019-12-05T06:28:15.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.934+0000 7fd6f6a89700 1 -- start start 2019-12-05T06:28:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.935+0000 7fd6f6a89700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd6f008d2c0 0x7fd6f008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.935+0000 7fd6f6a89700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f00958e0 con 0x7fd6f008c6d0 2019-12-05T06:28:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.935+0000 7fd6f6a89700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f00629e0 con 0x7fd6f008fef0 2019-12-05T06:28:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.935+0000 7fd6f6a89700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f008df90 con 0x7fd6f008d2c0 2019-12-05T06:28:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.936+0000 7fd6f5026700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd6f008c6d0 0x7fd6f008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39210/0 (socket says 172.21.15.16:39210) 2019-12-05T06:28:15.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.936+0000 7fd6f5026700 1 -- 172.21.15.16:0/3593316547 learned_addr learned my addr 172.21.15.16:0/3593316547 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.936+0000 7fd6f4825700 1 --2- 172.21.15.16:0/3593316547 >> v2:172.21.15.76:3300/0 conn(0x7fd6f008d2c0 0x7fd6f008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.936+0000 7fd6effff700 1 -- 172.21.15.16:0/3593316547 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 103329935 0 0) 0x7fd6e80014d0 con 0x7fd6f008c6d0 2019-12-05T06:28:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.936+0000 7fd6effff700 1 -- 172.21.15.16:0/3593316547 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d8002e80 con 0x7fd6f008c6d0 2019-12-05T06:28:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.937+0000 7fd6effff700 1 -- 172.21.15.16:0/3593316547 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1734153746 0 0) 0x7fd6e00009b0 con 0x7fd6f008fef0 2019-12-05T06:28:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.937+0000 7fd6effff700 1 -- 172.21.15.16:0/3593316547 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6d80030b0 con 0x7fd6f008fef0 2019-12-05T06:28:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.937+0000 7fd6f4825700 1 -- 172.21.15.16:0/3593316547 >> v1:172.21.15.76:6789/0 conn(0x7fd6f008fef0 legacy=0x7fd6f008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.937+0000 7fd6f4825700 1 -- 172.21.15.16:0/3593316547 >> v1:172.21.15.16:6789/0 conn(0x7fd6f008c6d0 legacy=0x7fd6f008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.937+0000 7fd6f4825700 1 -- 172.21.15.16:0/3593316547 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f008e620 con 0x7fd6f008d2c0 2019-12-05T06:28:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.937+0000 7fd6f4825700 1 --2- 172.21.15.16:0/3593316547 >> v2:172.21.15.76:3300/0 conn(0x7fd6f008d2c0 0x7fd6f008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd6e400af50 tx=0x7fd6e400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.937+0000 7fd6f4825700 1 --2- 172.21.15.16:0/3593316547 >> v2:172.21.15.76:3300/0 conn(0x7fd6f008d2c0 0x7fd6f008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd6e400af50 tx=0x7fd6e400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.937+0000 7fd6f4825700 1 --2- 172.21.15.16:0/3593316547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f008d2c0 0x7fd6f008ef20 secure :-1 s=READY pgs=603 cs=0 l=1 rx=0x7fd6e400af50 tx=0x7fd6e400b760).ready entity=mon.1 client_cookie=f0366d29499b9308 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.938+0000 7fd6f4825700 1 --2- 172.21.15.16:0/3593316547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f008d2c0 0x7fd6f008ef20 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fd6e400af50 tx=0x7fd6e400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.938+0000 7fd6f4825700 1 --2- 172.21.15.16:0/3593316547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f008d2c0 0x7fd6f008ef20 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fd6e400af50 tx=0x7fd6e400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.938+0000 7fd6effff700 1 -- 172.21.15.16:0/3593316547 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6e4000960 con 0x7fd6f008d2c0 2019-12-05T06:28:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.938+0000 7fd6f4825700 1 --2- 172.21.15.16:0/3593316547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f008d2c0 0x7fd6f008ef20 secure :-1 s=THROTTLE_DONE pgs=603 cs=0 l=1 rx=0x7fd6e400af50 tx=0x7fd6e400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.938+0000 7fd6effff700 1 -- 172.21.15.16:0/3593316547 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e4009860 con 0x7fd6f008d2c0 2019-12-05T06:28:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.938+0000 7fd6effff700 1 -- 172.21.15.16:0/3593316547 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6e4009f30 con 0x7fd6f008d2c0 2019-12-05T06:28:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.938+0000 7fd6f6a89700 1 -- 172.21.15.16:0/3593316547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f008d2c0 msgr2=0x7fd6f008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.938+0000 7fd6f6a89700 1 --2- 172.21.15.16:0/3593316547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f008d2c0 0x7fd6f008ef20 secure :-1 s=READY pgs=603 cs=0 l=1 rx=0x7fd6e400af50 tx=0x7fd6e400b760).stop 2019-12-05T06:28:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.940+0000 7fd6f6a89700 1 -- 172.21.15.16:0/3593316547 shutdown_connections 2019-12-05T06:28:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.940+0000 7fd6f6a89700 1 --2- 172.21.15.16:0/3593316547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f008d2c0 0x7fd6f008ef20 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.940+0000 7fd6f6a89700 1 -- 172.21.15.16:0/3593316547 shutdown_connections 2019-12-05T06:28:15.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.940+0000 7fd6f6a89700 1 -- 172.21.15.16:0/3593316547 wait complete. 2019-12-05T06:28:15.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.940+0000 7fd6f6a89700 1 -- 172.21.15.16:0/3593316547 >> 172.21.15.16:0/3593316547 conn(0x7fd6f0061940 msgr2=0x7fd6f0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:15.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.945+0000 7fd6f6a89700 1 Processor -- start 2019-12-05T06:28:15.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.945+0000 7fd6f6a89700 1 -- start start 2019-12-05T06:28:15.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.947+0000 7fd6f6a89700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6f014d5d0 0x7fd6f014d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:15.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.947+0000 7fd6f6a89700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd6f013a2b0 0x7fd6f01392e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:15.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.947+0000 7fd6f6a89700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6f012f550 con 0x7fd6f0139820 2019-12-05T06:28:15.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.947+0000 7fd6f6a89700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6f012af20 con 0x7fd6f013a2b0 2019-12-05T06:28:15.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.947+0000 7fd6f6a89700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f012b1f0 con 0x7fd6f014d5d0 2019-12-05T06:28:15.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.947+0000 7fd6f5026700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6f014d5d0 0x7fd6f014d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:15.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.947+0000 7fd6f4825700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd6f013a2b0 0x7fd6f01392e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:15.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.947+0000 7fd6f5827700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd6f0139820 0x7fd6f0128360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:60522/0 (socket says 172.21.15.16:60522) 2019-12-05T06:28:15.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.947+0000 7fd6f5026700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6f014d5d0 0x7fd6f014d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:58038/0 (socket says 172.21.15.16:58038) 2019-12-05T06:28:15.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.947+0000 7fd6f5827700 1 -- 172.21.15.16:0/1575111857 learned_addr learned my addr 172.21.15.16:0/1575111857 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:15.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.948+0000 7fd6f4825700 1 -- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6f014d5d0 msgr2=0x7fd6f014d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.948+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6f014d5d0 0x7fd6f014d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:15.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.948+0000 7fd6f4825700 1 -- 172.21.15.16:0/1575111857 >> v1:172.21.15.76:6789/0 conn(0x7fd6f0139820 legacy=0x7fd6f0128360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:15.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.948+0000 7fd6edffb700 1 -- 172.21.15.16:0/1575111857 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 222175638 0 0) 0x7fd6e00010b0 con 0x7fd6f0139820 2019-12-05T06:28:15.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.948+0000 7fd6f4825700 1 -- 172.21.15.16:0/1575111857 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f012bfe0 con 0x7fd6f013a2b0 2019-12-05T06:28:15.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.948+0000 7fd6f5026700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6f014d5d0 0x7fd6f014d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:15.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.948+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> v2:172.21.15.76:3300/0 conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.948+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> v2:172.21.15.76:3300/0 conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.948+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=READY pgs=605 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.949+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=THROTTLE_DONE pgs=605 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.949+0000 7fd6edffb700 1 -- 172.21.15.16:0/1575111857 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6e4000940 con 0x7fd6f013a2b0 2019-12-05T06:28:15.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.949+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=THROTTLE_DONE pgs=605 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.949+0000 7fd6edffb700 1 -- 172.21.15.16:0/1575111857 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e4009980 con 0x7fd6f013a2b0 2019-12-05T06:28:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.949+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=THROTTLE_DONE pgs=605 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.949+0000 7fd6f6a89700 1 -- 172.21.15.16:0/1575111857 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f012bfe0 con 0x7fd6f013a2b0 2019-12-05T06:28:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.949+0000 7fd6edffb700 1 -- 172.21.15.16:0/1575111857 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6e4009b70 con 0x7fd6f013a2b0 2019-12-05T06:28:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.949+0000 7fd6f6a89700 1 -- 172.21.15.16:0/1575111857 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f012bfe0 con 0x7fd6f013a2b0 2019-12-05T06:28:15.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.950+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=THROTTLE_DONE pgs=605 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.951+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=THROTTLE_DONE pgs=605 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.951+0000 7fd6edffb700 1 -- 172.21.15.16:0/1575111857 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd6e4009580 con 0x7fd6f013a2b0 2019-12-05T06:28:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.951+0000 7fd6edffb700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6d0023630 0x7fd6d0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.951+0000 7fd6edffb700 1 -- 172.21.15.16:0/1575111857 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7fd6e4045d40 con 0x7fd6f013a2b0 2019-12-05T06:28:15.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.952+0000 7fd6f5026700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6d0023630 0x7fd6d0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.952+0000 7fd6f5026700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6d0023630 0x7fd6d0025ac0 crc :-1 s=READY pgs=362 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:15.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.965+0000 7fd6f6a89700 1 -- 172.21.15.16:0/1575111857 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f012bfe0 con 0x7fd6f013a2b0 2019-12-05T06:28:15.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.974+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=THROTTLE_DONE pgs=605 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:15.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:15.975+0000 7fd6edffb700 1 -- 172.21.15.16:0/1575111857 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd6e4000940 con 0x7fd6f013a2b0 2019-12-05T06:28:16.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.250+0000 7fd6f6a89700 1 -- 172.21.15.16:0/1575111857 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6f012bfe0 con 0x7fd6f013a2b0 2019-12-05T06:28:16.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.251+0000 7fd6f4825700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=THROTTLE_DONE pgs=605 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:16.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.251+0000 7fd6edffb700 1 -- 172.21.15.16:0/1575111857 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 104 v104) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fd6e400d850 con 0x7fd6f013a2b0 2019-12-05T06:28:16.243 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 104 2019-12-05T06:28:16.244 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:16.244 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":104,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":104,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:16.168893+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:reconnect","state_seq":11,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:16.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.256+0000 7fd6f6a89700 1 -- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6d0023630 msgr2=0x7fd6d0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:16.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.256+0000 7fd6f6a89700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6d0023630 0x7fd6d0025ac0 crc :-1 s=READY pgs=362 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:16.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.256+0000 7fd6f6a89700 1 -- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 msgr2=0x7fd6f01392e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:16.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.256+0000 7fd6f6a89700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 0x7fd6f01392e0 secure :-1 s=READY pgs=605 cs=0 l=1 rx=0x7fd6e40103c0 tx=0x7fd6e4010760).stop 2019-12-05T06:28:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.257+0000 7fd6f6a89700 1 -- 172.21.15.16:0/1575111857 shutdown_connections 2019-12-05T06:28:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.257+0000 7fd6f6a89700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6d0023630 0x7fd6d0025ac0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.257+0000 7fd6f6a89700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd6f013a2b0 0x7fd6f01392e0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.257+0000 7fd6f6a89700 1 --2- 172.21.15.16:0/1575111857 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6f014d5d0 0x7fd6f014d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.257+0000 7fd6f6a89700 1 -- 172.21.15.16:0/1575111857 shutdown_connections 2019-12-05T06:28:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.257+0000 7fd6f6a89700 1 -- 172.21.15.16:0/1575111857 wait complete. 2019-12-05T06:28:16.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:16.257+0000 7fd6f6a89700 1 -- 172.21.15.16:0/1575111857 >> 172.21.15.16:0/1575111857 conn(0x7fd6f0070a20 msgr2=0x7fd6f0071480 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:16.257 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 104, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 104, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6264": { "addr": "172.21.15.76:6835/3155972571", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 3155972571, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 3155972571, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6264, "incarnation": 103, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 11 } }, "last_failure": 0, "last_failure_osd_epoch": 64, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:28:16.168893+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6264 } } } ], "standbys": [] } 2019-12-05T06:28:18.258 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:18.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.479+0000 7f6ed3ffc700 1 Processor -- start 2019-12-05T06:28:18.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.479+0000 7f6ed3ffc700 1 -- start start 2019-12-05T06:28:18.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.480+0000 7f6ed3ffc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc08c6d0 0x7f6ecc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:18.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.480+0000 7f6ed3ffc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ecc08fef0 0x7f6ecc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.480+0000 7f6ed3ffc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6ecc08ef20 0x7f6ecc08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.480+0000 7f6ed3ffc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ecc0a3f70 con 0x7f6ecc08fef0 2019-12-05T06:28:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.480+0000 7f6ed3ffc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ecc08d830 con 0x7f6ecc08ef20 2019-12-05T06:28:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.480+0000 7f6ed3ffc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc08d540 con 0x7f6ecc08c6d0 2019-12-05T06:28:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.480+0000 7f6ed1d98700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ecc08fef0 0x7f6ecc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.480+0000 7f6ed1d98700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ecc08fef0 0x7f6ecc08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54062/0 (socket says 172.21.15.16:54062) 2019-12-05T06:28:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed1d98700 1 -- 172.21.15.16:0/288336194 learned_addr learned my addr 172.21.15.16:0/288336194 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:28:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed2599700 1 --2- 172.21.15.16:0/288336194 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc08c6d0 0x7f6ecc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/288336194 >> v2:172.21.15.76:3300/0 conn(0x7f6ecc08ef20 0x7f6ecc08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed1d98700 1 -- 172.21.15.16:0/288336194 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc08c6d0 msgr2=0x7f6ecc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed1d98700 1 --2- 172.21.15.16:0/288336194 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc08c6d0 0x7f6ecc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed1d98700 1 -- 172.21.15.16:0/288336194 >> v2:172.21.15.76:3300/0 conn(0x7f6ecc08ef20 msgr2=0x7f6ecc08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed1d98700 1 --2- 172.21.15.16:0/288336194 >> v2:172.21.15.76:3300/0 conn(0x7f6ecc08ef20 0x7f6ecc08f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed1d98700 1 -- 172.21.15.16:0/288336194 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc08e620 con 0x7f6ecc08fef0 2019-12-05T06:28:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/288336194 >> v2:172.21.15.76:3300/0 conn(0x7f6ecc08ef20 0x7f6ecc08f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:28:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed1d98700 1 --2- 172.21.15.16:0/288336194 >> v2:172.21.15.16:3300/0 conn(0x7f6ecc08fef0 0x7f6ecc08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6ec000f6c0 tx=0x7f6ec000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed1d98700 1 --2- 172.21.15.16:0/288336194 >> v2:172.21.15.16:3300/0 conn(0x7f6ecc08fef0 0x7f6ecc08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6ec000f6c0 tx=0x7f6ec000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.481+0000 7f6ed1d98700 1 --2- 172.21.15.16:0/288336194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc08fef0 0x7f6ecc08d000 secure :-1 s=READY pgs=832 cs=0 l=1 rx=0x7f6ec000f6c0 tx=0x7f6ec000fa60).ready entity=mon.0 client_cookie=dbd152c827e1f96d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.482+0000 7f6ed1d98700 1 --2- 172.21.15.16:0/288336194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc08fef0 0x7f6ecc08d000 secure :-1 s=THROTTLE_DONE pgs=832 cs=0 l=1 rx=0x7f6ec000f6c0 tx=0x7f6ec000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.482+0000 7f6ed1d98700 1 --2- 172.21.15.16:0/288336194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc08fef0 0x7f6ecc08d000 secure :-1 s=THROTTLE_DONE pgs=832 cs=0 l=1 rx=0x7f6ec000f6c0 tx=0x7f6ec000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.482+0000 7f6ed1597700 1 -- 172.21.15.16:0/288336194 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ec0000960 con 0x7f6ecc08fef0 2019-12-05T06:28:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.482+0000 7f6ed1d98700 1 --2- 172.21.15.16:0/288336194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc08fef0 0x7f6ecc08d000 secure :-1 s=THROTTLE_DONE pgs=832 cs=0 l=1 rx=0x7f6ec000f6c0 tx=0x7f6ec000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.482+0000 7f6ed1597700 1 -- 172.21.15.16:0/288336194 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec0000c90 con 0x7f6ecc08fef0 2019-12-05T06:28:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.482+0000 7f6ed1597700 1 -- 172.21.15.16:0/288336194 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ec00013a0 con 0x7f6ecc08fef0 2019-12-05T06:28:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.482+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/288336194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc08fef0 msgr2=0x7f6ecc08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.482+0000 7f6ed3ffc700 1 --2- 172.21.15.16:0/288336194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc08fef0 0x7f6ecc08d000 secure :-1 s=READY pgs=832 cs=0 l=1 rx=0x7f6ec000f6c0 tx=0x7f6ec000fa60).stop 2019-12-05T06:28:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.483+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/288336194 shutdown_connections 2019-12-05T06:28:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.483+0000 7f6ed3ffc700 1 --2- 172.21.15.16:0/288336194 >> v2:172.21.15.76:3300/0 conn(0x7f6ecc08ef20 0x7f6ecc08f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.483+0000 7f6ed3ffc700 1 --2- 172.21.15.16:0/288336194 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc08fef0 0x7f6ecc08d000 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.483+0000 7f6ed3ffc700 1 --2- 172.21.15.16:0/288336194 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc08c6d0 0x7f6ecc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.483+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/288336194 shutdown_connections 2019-12-05T06:28:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.483+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/288336194 wait complete. 2019-12-05T06:28:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.483+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/288336194 >> 172.21.15.16:0/288336194 conn(0x7f6ecc061940 msgr2=0x7f6ecc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.487+0000 7f6ed3ffc700 1 Processor -- start 2019-12-05T06:28:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.487+0000 7f6ed3ffc700 1 -- start start 2019-12-05T06:28:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed3ffc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc0992d0 0x7f6ecc098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed3ffc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ecc156cb0 0x7f6ecc1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:18.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed3ffc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ecc089fb0 con 0x7f6ecc06df10 2019-12-05T06:28:18.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed3ffc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ecc097370 con 0x7f6ecc156cb0 2019-12-05T06:28:18.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed3ffc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc08ad00 con 0x7f6ecc0992d0 2019-12-05T06:28:18.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed2d9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ecc156cb0 0x7f6ecc1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:18.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed2d9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6ecc156cb0 0x7f6ecc1590b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54068/0 (socket says 172.21.15.16:54068) 2019-12-05T06:28:18.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed2d9a700 1 -- 172.21.15.16:0/931534985 learned_addr learned my addr 172.21.15.16:0/931534985 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:28:18.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed1d98700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc0992d0 0x7f6ecc098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:18.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed2d9a700 1 -- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc0992d0 msgr2=0x7f6ecc098b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc0992d0 0x7f6ecc098b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed2d9a700 1 -- 172.21.15.16:0/931534985 >> v1:172.21.15.76:6789/0 conn(0x7f6ecc06df10 legacy=0x7f6ecc098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed2d9a700 1 -- 172.21.15.16:0/931534985 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ecc097a00 con 0x7f6ecc156cb0 2019-12-05T06:28:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed1d98700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc0992d0 0x7f6ecc098b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:28:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.489+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> v2:172.21.15.16:3300/0 conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.490+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> v2:172.21.15.16:3300/0 conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.490+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=READY pgs=833 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.490+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=THROTTLE_DONE pgs=833 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.490+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=THROTTLE_DONE pgs=833 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.490+0000 7f6ecaffd700 1 -- 172.21.15.16:0/931534985 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ebc008480 con 0x7f6ecc156cb0 2019-12-05T06:28:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.490+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=THROTTLE_DONE pgs=833 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.490+0000 7f6ecaffd700 1 -- 172.21.15.16:0/931534985 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ebc009a80 con 0x7f6ecc156cb0 2019-12-05T06:28:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.490+0000 7f6ecaffd700 1 -- 172.21.15.16:0/931534985 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ebc00a110 con 0x7f6ecc156cb0 2019-12-05T06:28:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.490+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/931534985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ecc097a00 con 0x7f6ecc156cb0 2019-12-05T06:28:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.490+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/931534985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ecc097a00 con 0x7f6ecc156cb0 2019-12-05T06:28:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.491+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=THROTTLE_DONE pgs=833 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.492+0000 7f6ecaffd700 1 -- 172.21.15.16:0/931534985 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6ebc000a70 con 0x7f6ecc156cb0 2019-12-05T06:28:18.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.492+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=THROTTLE_DONE pgs=833 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.492+0000 7f6ecaffd700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6eac023630 0x7f6eac025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.492+0000 7f6ecaffd700 1 -- 172.21.15.16:0/931534985 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f6ebc045d90 con 0x7f6ecc156cb0 2019-12-05T06:28:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.493+0000 7f6ed2599700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6eac023630 0x7f6eac025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.493+0000 7f6ed2599700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6eac023630 0x7f6eac025ac0 crc :-1 s=READY pgs=363 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:18.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.495+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/931534985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ecc097a00 con 0x7f6ecc156cb0 2019-12-05T06:28:18.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.502+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=THROTTLE_DONE pgs=833 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.502+0000 7f6ecaffd700 1 -- 172.21.15.16:0/931534985 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6ebc0095e0 con 0x7f6ecc156cb0 2019-12-05T06:28:18.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.744+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/931534985 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ecc097a00 con 0x7f6ecc156cb0 2019-12-05T06:28:18.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.744+0000 7f6ed2d9a700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=THROTTLE_DONE pgs=833 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:18.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.744+0000 7f6ecaffd700 1 -- 172.21.15.16:0/931534985 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f6ebc000a70 con 0x7f6ecc156cb0 2019-12-05T06:28:18.735 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 106 2019-12-05T06:28:18.736 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:18.736 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:18.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.747+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6eac023630 msgr2=0x7f6eac025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:18.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.747+0000 7f6ed3ffc700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6eac023630 0x7f6eac025ac0 crc :-1 s=READY pgs=363 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:18.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.747+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 msgr2=0x7f6ecc1590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:18.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.747+0000 7f6ed3ffc700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 0x7f6ecc1590b0 secure :-1 s=READY pgs=833 cs=0 l=1 rx=0x7f6ebc010480 tx=0x7f6ebc010820).stop 2019-12-05T06:28:18.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.747+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/931534985 shutdown_connections 2019-12-05T06:28:18.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.747+0000 7f6ed3ffc700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6eac023630 0x7f6eac025ac0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:18.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.747+0000 7f6ed3ffc700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc156cb0 0x7f6ecc1590b0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:18.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.747+0000 7f6ed3ffc700 1 --2- 172.21.15.16:0/931534985 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc0992d0 0x7f6ecc098b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:18.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.748+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/931534985 shutdown_connections 2019-12-05T06:28:18.740 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.748+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/931534985 wait complete. 2019-12-05T06:28:18.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:18.748+0000 7f6ed3ffc700 1 -- 172.21.15.16:0/931534985 >> 172.21.15.16:0/931534985 conn(0x7f6ecc0714f0 msgr2=0x7f6ecc09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:18.750 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6165 2019-12-05T06:28:18.750 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2019-12-05T06:28:26.751 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:28:26.751 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:28:26.751 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:28:26.756 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:28:26.756 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:26.898 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:28:26.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.937+0000 7f610e792700 1 Processor -- start 2019-12-05T06:28:26.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.937+0000 7f610e792700 1 -- start start 2019-12-05T06:28:26.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.938+0000 7f610e792700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f610808d2c0 0x7f610808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:26.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.938+0000 7f610e792700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:26.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.938+0000 7f610e792700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61080958e0 con 0x7f610808c6d0 2019-12-05T06:28:26.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.938+0000 7f610e792700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610808df90 con 0x7f610808d2c0 2019-12-05T06:28:26.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.938+0000 7f610e792700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61080a3f70 con 0x7f610808fef0 2019-12-05T06:28:26.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.938+0000 7f610d530700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f6107fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f610808d2c0 0x7f610808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f610d530700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:58112/0 (socket says 172.21.15.16:58112) 2019-12-05T06:28:26.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f610d530700 1 -- 172.21.15.16:0/1974850854 learned_addr learned my addr 172.21.15.16:0/1974850854 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:28:26.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f610d530700 1 -- 172.21.15.16:0/1974850854 >> v2:172.21.15.76:3300/0 conn(0x7f610808d2c0 msgr2=0x7f610808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:26.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f610d530700 1 --2- 172.21.15.16:0/1974850854 >> v2:172.21.15.76:3300/0 conn(0x7f610808d2c0 0x7f610808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:26.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f610d530700 1 -- 172.21.15.16:0/1974850854 >> v1:172.21.15.76:6789/0 conn(0x7f610808c6d0 legacy=0x7f610808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f610d530700 1 -- 172.21.15.16:0/1974850854 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610808e620 con 0x7f610808fef0 2019-12-05T06:28:26.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f6107fff700 1 --2- 172.21.15.16:0/1974850854 >> v2:172.21.15.76:3300/0 conn(0x7f610808d2c0 0x7f610808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:28:26.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f610d530700 1 --2- 172.21.15.16:0/1974850854 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f60f000f760 tx=0x7f60f000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f610d530700 1 --2- 172.21.15.16:0/1974850854 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f60f000f760 tx=0x7f60f000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.939+0000 7f610d530700 1 --2- 172.21.15.16:0/1974850854 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 secure :-1 s=READY pgs=205 cs=0 l=1 rx=0x7f60f000f760 tx=0x7f60f000fb00).ready entity=mon.2 client_cookie=4b978029e061de47 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.940+0000 7f610d530700 1 --2- 172.21.15.16:0/1974850854 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 secure :-1 s=THROTTLE_DONE pgs=205 cs=0 l=1 rx=0x7f60f000f760 tx=0x7f60f000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.940+0000 7f61077fe700 1 -- 172.21.15.16:0/1974850854 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60f0000a70 con 0x7f610808fef0 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.940+0000 7f610d530700 1 --2- 172.21.15.16:0/1974850854 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 secure :-1 s=THROTTLE_DONE pgs=205 cs=0 l=1 rx=0x7f60f000f760 tx=0x7f60f000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.940+0000 7f610d530700 1 --2- 172.21.15.16:0/1974850854 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 secure :-1 s=THROTTLE_DONE pgs=205 cs=0 l=1 rx=0x7f60f000f760 tx=0x7f60f000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.940+0000 7f61077fe700 1 -- 172.21.15.16:0/1974850854 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f0000e70 con 0x7f610808fef0 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.940+0000 7f61077fe700 1 -- 172.21.15.16:0/1974850854 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60f0001500 con 0x7f610808fef0 2019-12-05T06:28:26.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.940+0000 7f610e792700 1 -- 172.21.15.16:0/1974850854 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 msgr2=0x7f610808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.940+0000 7f610e792700 1 --2- 172.21.15.16:0/1974850854 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 secure :-1 s=READY pgs=205 cs=0 l=1 rx=0x7f60f000f760 tx=0x7f60f000fb00).stop 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.941+0000 7f610e792700 1 -- 172.21.15.16:0/1974850854 shutdown_connections 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.941+0000 7f610e792700 1 --2- 172.21.15.16:0/1974850854 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f610808fef0 0x7f610808f460 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.941+0000 7f610e792700 1 --2- 172.21.15.16:0/1974850854 >> v2:172.21.15.76:3300/0 conn(0x7f610808d2c0 0x7f610808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.941+0000 7f610e792700 1 -- 172.21.15.16:0/1974850854 shutdown_connections 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.941+0000 7f610e792700 1 -- 172.21.15.16:0/1974850854 wait complete. 2019-12-05T06:28:26.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.941+0000 7f610e792700 1 -- 172.21.15.16:0/1974850854 >> 172.21.15.16:0/1974850854 conn(0x7f6108061940 msgr2=0x7f6108096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:26.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.944+0000 7f610e792700 1 Processor -- start 2019-12-05T06:28:26.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.944+0000 7f610e792700 1 -- start start 2019-12-05T06:28:26.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610e792700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f610809e2c0 0x7f610809dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:26.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610e792700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f610814d9c0 0x7f610819b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:26.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610e792700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f610814de80 con 0x7f610814d5d0 2019-12-05T06:28:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610e792700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610812b3f0 con 0x7f610814d9c0 2019-12-05T06:28:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610e792700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f610812a9c0 con 0x7f610809e2c0 2019-12-05T06:28:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610cd2f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f610814d5d0 0x7f610809d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39294/0 (socket says 172.21.15.16:39294) 2019-12-05T06:28:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610cd2f700 1 -- 172.21.15.16:0/2722479498 learned_addr learned my addr 172.21.15.16:0/2722479498 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> v2:172.21.15.16:3300/0 conn(0x7f610814d9c0 0x7f610819b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f6107fff700 1 --2- 172.21.15.16:0/2722479498 >> v2:172.21.15.76:3300/0 conn(0x7f610809e2c0 0x7f610809dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610d530700 1 -- 172.21.15.16:0/2722479498 >> v2:172.21.15.76:3300/0 conn(0x7f610809e2c0 msgr2=0x7f610809dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> v2:172.21.15.76:3300/0 conn(0x7f610809e2c0 0x7f610809dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610d530700 1 -- 172.21.15.16:0/2722479498 >> v1:172.21.15.16:6789/0 conn(0x7f610814d5d0 legacy=0x7f610809d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610d530700 1 -- 172.21.15.16:0/2722479498 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f610812ba80 con 0x7f610814d9c0 2019-12-05T06:28:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f61057fa700 1 -- 172.21.15.16:0/2722479498 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2662422656 0 0) 0x7f6100001150 con 0x7f610814d5d0 2019-12-05T06:28:26.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.945+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> v2:172.21.15.16:3300/0 conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> v2:172.21.15.16:3300/0 conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=READY pgs=838 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:26.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f61057fa700 1 -- 172.21.15.16:0/2722479498 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60f00072f0 con 0x7f610814d9c0 2019-12-05T06:28:26.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f61057fa700 1 -- 172.21.15.16:0/2722479498 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60f000f670 con 0x7f610814d9c0 2019-12-05T06:28:26.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f61057fa700 1 -- 172.21.15.16:0/2722479498 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60f000fd40 con 0x7f610814d9c0 2019-12-05T06:28:26.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f610e792700 1 -- 172.21.15.16:0/2722479498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f610812ba80 con 0x7f610814d9c0 2019-12-05T06:28:26.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f610e792700 1 -- 172.21.15.16:0/2722479498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f610812ba80 con 0x7f610814d9c0 2019-12-05T06:28:26.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.946+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.947+0000 7f61057fa700 1 -- 172.21.15.16:0/2722479498 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f60f000f860 con 0x7f610814d9c0 2019-12-05T06:28:26.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.947+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.947+0000 7f61057fa700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60ec023630 0x7f60ec025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:26.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.947+0000 7f61057fa700 1 -- 172.21.15.16:0/2722479498 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f60f0045d30 con 0x7f610814d9c0 2019-12-05T06:28:26.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.947+0000 7f610cd2f700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60ec023630 0x7f60ec025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:26.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.948+0000 7f610cd2f700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60ec023630 0x7f60ec025ac0 crc :-1 s=READY pgs=364 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:26.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.953+0000 7f610e792700 1 -- 172.21.15.16:0/2722479498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f610812ba80 con 0x7f610814d9c0 2019-12-05T06:28:26.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.957+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:26.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:26.957+0000 7f61057fa700 1 -- 172.21.15.16:0/2722479498 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f60f00075d0 con 0x7f610814d9c0 2019-12-05T06:28:27.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.152+0000 7f610e792700 1 -- 172.21.15.16:0/2722479498 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f610812ba80 con 0x7f610814d9c0 2019-12-05T06:28:27.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.153+0000 7f610d530700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:27.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.153+0000 7f61057fa700 1 -- 172.21.15.16:0/2722479498 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f60f000f860 con 0x7f610814d9c0 2019-12-05T06:28:27.144 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 106 2019-12-05T06:28:27.144 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:27.145 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:27.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.155+0000 7f610e792700 1 -- 172.21.15.16:0/2722479498 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60ec023630 msgr2=0x7f60ec025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:27.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.156+0000 7f610e792700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60ec023630 0x7f60ec025ac0 crc :-1 s=READY pgs=364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:27.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.156+0000 7f610e792700 1 -- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 msgr2=0x7f610819b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:27.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.156+0000 7f610e792700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 0x7f610819b400 secure :-1 s=READY pgs=838 cs=0 l=1 rx=0x7f60f000c190 tx=0x7f60f000c010).stop 2019-12-05T06:28:27.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.156+0000 7f610e792700 1 -- 172.21.15.16:0/2722479498 shutdown_connections 2019-12-05T06:28:27.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.156+0000 7f610e792700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60ec023630 0x7f60ec025ac0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.156+0000 7f610e792700 1 --2- 172.21.15.16:0/2722479498 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f610814d9c0 0x7f610819b400 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.156+0000 7f610e792700 1 --2- 172.21.15.16:0/2722479498 >> v2:172.21.15.76:3300/0 conn(0x7f610809e2c0 0x7f610809dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.156+0000 7f610e792700 1 -- 172.21.15.16:0/2722479498 shutdown_connections 2019-12-05T06:28:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.156+0000 7f610e792700 1 -- 172.21.15.16:0/2722479498 wait complete. 2019-12-05T06:28:27.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:27.156+0000 7f610e792700 1 -- 172.21.15.16:0/2722479498 >> 172.21.15.16:0/2722479498 conn(0x7f610806d260 msgr2=0x7f610809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:27.161 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:28:29.163 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.449+0000 7f2ca395a700 1 Processor -- start 2019-12-05T06:28:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.449+0000 7f2ca395a700 1 -- start start 2019-12-05T06:28:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.451+0000 7f2ca395a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:29.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.451+0000 7f2ca395a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c08f720 0x7f2c9c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.451+0000 7f2ca395a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9c08fb70 con 0x7f2c9c08fef0 2019-12-05T06:28:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.451+0000 7f2ca395a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9c08df90 con 0x7f2c9c08c6d0 2019-12-05T06:28:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.451+0000 7f2ca395a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c0a3f70 con 0x7f2c9c08f720 2019-12-05T06:28:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.451+0000 7f2ca1ef7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.451+0000 7f2ca1ef7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54134/0 (socket says 172.21.15.16:54134) 2019-12-05T06:28:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.451+0000 7f2ca1ef7700 1 -- 172.21.15.16:0/1828341887 learned_addr learned my addr 172.21.15.16:0/1828341887 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:28:29.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.452+0000 7f2ca26f8700 1 --2- 172.21.15.16:0/1828341887 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c08f720 0x7f2c9c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.452+0000 7f2ca1ef7700 1 -- 172.21.15.16:0/1828341887 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c08f720 msgr2=0x7f2c9c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:28:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.452+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/1828341887 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c08f720 0x7f2c9c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.452+0000 7f2ca1ef7700 1 -- 172.21.15.16:0/1828341887 >> v1:172.21.15.16:6789/0 conn(0x7f2c9c08fef0 legacy=0x7f2c9c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.452+0000 7f2ca1ef7700 1 -- 172.21.15.16:0/1828341887 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c08d540 con 0x7f2c9c08c6d0 2019-12-05T06:28:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.452+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/1828341887 >> v2:172.21.15.16:3300/0 conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2c8c00f740 tx=0x7f2c8c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.452+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/1828341887 >> v2:172.21.15.16:3300/0 conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2c8c00f740 tx=0x7f2c8c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.453+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/1828341887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 secure :-1 s=READY pgs=840 cs=0 l=1 rx=0x7f2c8c00f740 tx=0x7f2c8c00f570).ready entity=mon.0 client_cookie=163139bb9ac5fb1d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.453+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/1828341887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 secure :-1 s=THROTTLE_DONE pgs=840 cs=0 l=1 rx=0x7f2c8c00f740 tx=0x7f2c8c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.453+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/1828341887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 secure :-1 s=THROTTLE_DONE pgs=840 cs=0 l=1 rx=0x7f2c8c00f740 tx=0x7f2c8c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.453+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/1828341887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 secure :-1 s=THROTTLE_DONE pgs=840 cs=0 l=1 rx=0x7f2c8c00f740 tx=0x7f2c8c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.453+0000 7f2ca0ef5700 1 -- 172.21.15.16:0/1828341887 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c8c000960 con 0x7f2c9c08c6d0 2019-12-05T06:28:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.453+0000 7f2ca0ef5700 1 -- 172.21.15.16:0/1828341887 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c000c90 con 0x7f2c9c08c6d0 2019-12-05T06:28:29.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.453+0000 7f2ca0ef5700 1 -- 172.21.15.16:0/1828341887 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c8c0013e0 con 0x7f2c9c08c6d0 2019-12-05T06:28:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.453+0000 7f2ca395a700 1 -- 172.21.15.16:0/1828341887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c08c6d0 msgr2=0x7f2c9c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.453+0000 7f2ca395a700 1 --2- 172.21.15.16:0/1828341887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 secure :-1 s=READY pgs=840 cs=0 l=1 rx=0x7f2c8c00f740 tx=0x7f2c8c00f570).stop 2019-12-05T06:28:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.455+0000 7f2ca395a700 1 -- 172.21.15.16:0/1828341887 shutdown_connections 2019-12-05T06:28:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.455+0000 7f2ca395a700 1 --2- 172.21.15.16:0/1828341887 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c08f720 0x7f2c9c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:28:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.455+0000 7f2ca395a700 1 --2- 172.21.15.16:0/1828341887 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9c08c6d0 0x7f2c9c08cac0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.455+0000 7f2ca395a700 1 -- 172.21.15.16:0/1828341887 shutdown_connections 2019-12-05T06:28:29.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.455+0000 7f2ca395a700 1 -- 172.21.15.16:0/1828341887 wait complete. 2019-12-05T06:28:29.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.455+0000 7f2ca395a700 1 -- 172.21.15.16:0/1828341887 >> 172.21.15.16:0/1828341887 conn(0x7f2c9c061940 msgr2=0x7f2c9c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.460+0000 7f2ca395a700 1 Processor -- start 2019-12-05T06:28:29.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.460+0000 7f2ca395a700 1 -- start start 2019-12-05T06:28:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.462+0000 7f2ca395a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.462+0000 7f2ca395a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9c06e7c0 con 0x7f2c9c09ae80 2019-12-05T06:28:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.462+0000 7f2ca395a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9c133b30 con 0x7f2c9c099eb0 2019-12-05T06:28:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.462+0000 7f2ca395a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c9c134390 con 0x7f2c9c132630 2019-12-05T06:28:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.462+0000 7f2ca16f6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2c9c09ae80 0x7f2c9c132f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39310/0 (socket says 172.21.15.16:39310) 2019-12-05T06:28:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.462+0000 7f2ca16f6700 1 -- 172.21.15.16:0/2261851780 learned_addr learned my addr 172.21.15.16:0/2261851780 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:29.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.462+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.463+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668461669 0 0) 0x7f2c90001480 con 0x7f2c9c09ae80 2019-12-05T06:28:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.463+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c002e80 con 0x7f2c9c09ae80 2019-12-05T06:28:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.463+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2202512294 0 0) 0x7f2c94001080 con 0x7f2c9c099eb0 2019-12-05T06:28:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.463+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c7c0041b0 con 0x7f2c9c099eb0 2019-12-05T06:28:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.463+0000 7f2ca1ef7700 1 -- 172.21.15.16:0/2261851780 >> v1:172.21.15.76:6789/0 conn(0x7f2c9c099eb0 legacy=0x7f2c9c09a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.463+0000 7f2ca1ef7700 1 -- 172.21.15.16:0/2261851780 >> v1:172.21.15.16:6789/0 conn(0x7f2c9c09ae80 legacy=0x7f2c9c132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.463+0000 7f2ca1ef7700 1 -- 172.21.15.16:0/2261851780 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9c0995b0 con 0x7f2c9c132630 2019-12-05T06:28:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.463+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.463+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.463+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=READY pgs=206 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.464+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=THROTTLE_DONE pgs=206 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.464+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c8c005560 con 0x7f2c9c132630 2019-12-05T06:28:29.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.464+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=THROTTLE_DONE pgs=206 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.464+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=THROTTLE_DONE pgs=206 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.464+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c8c005180 con 0x7f2c9c132630 2019-12-05T06:28:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.464+0000 7f2ca395a700 1 -- 172.21.15.16:0/2261851780 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c9c0995b0 con 0x7f2c9c132630 2019-12-05T06:28:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.464+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c8c005b40 con 0x7f2c9c132630 2019-12-05T06:28:29.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.464+0000 7f2ca395a700 1 -- 172.21.15.16:0/2261851780 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c9c0995b0 con 0x7f2c9c132630 2019-12-05T06:28:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.465+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=THROTTLE_DONE pgs=206 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.466+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2c8c005370 con 0x7f2c9c132630 2019-12-05T06:28:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.466+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=THROTTLE_DONE pgs=206 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.466+0000 7f2c9a7fc700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023650 0x7f2c7c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.466+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f2c8c045db0 con 0x7f2c9c132630 2019-12-05T06:28:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.466+0000 7f2ca16f6700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023650 0x7f2c7c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:29.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.467+0000 7f2ca16f6700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023650 0x7f2c7c025ae0 crc :-1 s=READY pgs=365 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:29.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.480+0000 7f2ca395a700 1 -- 172.21.15.16:0/2261851780 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c9c0995b0 con 0x7f2c9c132630 2019-12-05T06:28:29.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.489+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=THROTTLE_DONE pgs=206 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.489+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2c8c007310 con 0x7f2c9c132630 2019-12-05T06:28:29.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.754+0000 7f2ca395a700 1 -- 172.21.15.16:0/2261851780 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c9c0995b0 con 0x7f2c9c132630 2019-12-05T06:28:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.755+0000 7f2ca1ef7700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=THROTTLE_DONE pgs=206 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.755+0000 7f2c9a7fc700 1 -- 172.21.15.16:0/2261851780 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 106 v106) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f2c8c005370 con 0x7f2c9c132630 2019-12-05T06:28:29.746 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 106 2019-12-05T06:28:29.746 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:29.746 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":106,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.758+0000 7f2ca395a700 1 -- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023650 msgr2=0x7f2c7c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.758+0000 7f2ca395a700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023650 0x7f2c7c025ae0 crc :-1 s=READY pgs=365 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.758+0000 7f2ca395a700 1 -- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 msgr2=0x7f2c9c132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.758+0000 7f2ca395a700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 secure :-1 s=READY pgs=206 cs=0 l=1 rx=0x7f2c8c0103a0 tx=0x7f2c8c010740).stop 2019-12-05T06:28:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.758+0000 7f2ca395a700 1 -- 172.21.15.16:0/2261851780 shutdown_connections 2019-12-05T06:28:29.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.758+0000 7f2ca395a700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023650 0x7f2c7c025ae0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.758+0000 7f2ca395a700 1 --2- 172.21.15.16:0/2261851780 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9c132630 0x7f2c9c132a20 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.758+0000 7f2ca395a700 1 -- 172.21.15.16:0/2261851780 shutdown_connections 2019-12-05T06:28:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.758+0000 7f2ca395a700 1 -- 172.21.15.16:0/2261851780 wait complete. 2019-12-05T06:28:29.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:29.758+0000 7f2ca395a700 1 -- 172.21.15.16:0/2261851780 >> 172.21.15.16:0/2261851780 conn(0x7f2c9c0714f0 msgr2=0x7f2c9c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:29.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:28:31.761 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:28:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.996+0000 7f36183fd700 1 Processor -- start 2019-12-05T06:28:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.996+0000 7f36183fd700 1 -- start start 2019-12-05T06:28:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.997+0000 7f36183fd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f361008c6d0 0x7f361008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:28:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.997+0000 7f36183fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361008fb80 con 0x7f361008fef0 2019-12-05T06:28:31.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.997+0000 7f36183fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36100958e0 con 0x7f361008f720 2019-12-05T06:28:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.997+0000 7f36183fd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f361008df90 con 0x7f361008c6d0 2019-12-05T06:28:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.997+0000 7f361699a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f361008c6d0 0x7f361008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.997+0000 7f361699a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f361008c6d0 0x7f361008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54160/0 (socket says 172.21.15.16:54160) 2019-12-05T06:28:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.997+0000 7f361699a700 1 -- 172.21.15.16:0/2229152154 learned_addr learned my addr 172.21.15.16:0/2229152154 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:28:31.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f361699a700 1 -- 172.21.15.16:0/2229152154 >> v1:172.21.15.76:6789/0 conn(0x7f361008f720 legacy=0x7f36101533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f361699a700 1 -- 172.21.15.16:0/2229152154 >> v1:172.21.15.16:6789/0 conn(0x7f361008fef0 legacy=0x7f361008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f361699a700 1 -- 172.21.15.16:0/2229152154 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f361008e620 con 0x7f361008c6d0 2019-12-05T06:28:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f361699a700 1 --2- 172.21.15.16:0/2229152154 >> v2:172.21.15.16:3300/0 conn(0x7f361008c6d0 0x7f361008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f360800f740 tx=0x7f360800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f361699a700 1 --2- 172.21.15.16:0/2229152154 >> v2:172.21.15.16:3300/0 conn(0x7f361008c6d0 0x7f361008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f360800f740 tx=0x7f360800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f361699a700 1 --2- 172.21.15.16:0/2229152154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f361008c6d0 0x7f361008cac0 secure :-1 s=READY pgs=843 cs=0 l=1 rx=0x7f360800f740 tx=0x7f360800fae0).ready entity=mon.0 client_cookie=b0ea3380fc10a8f3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:32.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f361699a700 1 --2- 172.21.15.16:0/2229152154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f361008c6d0 0x7f361008cac0 secure :-1 s=THROTTLE_DONE pgs=843 cs=0 l=1 rx=0x7f360800f740 tx=0x7f360800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f361699a700 1 --2- 172.21.15.16:0/2229152154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f361008c6d0 0x7f361008cac0 secure :-1 s=THROTTLE_DONE pgs=843 cs=0 l=1 rx=0x7f360800f740 tx=0x7f360800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f3615998700 1 -- 172.21.15.16:0/2229152154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3608000960 con 0x7f361008c6d0 2019-12-05T06:28:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f361699a700 1 --2- 172.21.15.16:0/2229152154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f361008c6d0 0x7f361008cac0 secure :-1 s=THROTTLE_DONE pgs=843 cs=0 l=1 rx=0x7f360800f740 tx=0x7f360800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f3615998700 1 -- 172.21.15.16:0/2229152154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3608000c90 con 0x7f361008c6d0 2019-12-05T06:28:32.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f3615998700 1 -- 172.21.15.16:0/2229152154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3608001360 con 0x7f361008c6d0 2019-12-05T06:28:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f36183fd700 1 -- 172.21.15.16:0/2229152154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f361008c6d0 msgr2=0x7f361008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.998+0000 7f36183fd700 1 --2- 172.21.15.16:0/2229152154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f361008c6d0 0x7f361008cac0 secure :-1 s=READY pgs=843 cs=0 l=1 rx=0x7f360800f740 tx=0x7f360800fae0).stop 2019-12-05T06:28:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.999+0000 7f36183fd700 1 -- 172.21.15.16:0/2229152154 shutdown_connections 2019-12-05T06:28:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.999+0000 7f36183fd700 1 --2- 172.21.15.16:0/2229152154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f361008c6d0 0x7f361008cac0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.999+0000 7f36183fd700 1 -- 172.21.15.16:0/2229152154 shutdown_connections 2019-12-05T06:28:32.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.999+0000 7f36183fd700 1 -- 172.21.15.16:0/2229152154 wait complete. 2019-12-05T06:28:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:31.999+0000 7f36183fd700 1 -- 172.21.15.16:0/2229152154 >> 172.21.15.16:0/2229152154 conn(0x7f3610061940 msgr2=0x7f3610096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.002+0000 7f36183fd700 1 Processor -- start 2019-12-05T06:28:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.002+0000 7f36183fd700 1 -- start start 2019-12-05T06:28:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.003+0000 7f36183fd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f36100871e0 0x7f361009ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.003+0000 7f36183fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3610087640 con 0x7f361014ca10 2019-12-05T06:28:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.003+0000 7f36183fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f361009e030 con 0x7f36100879b0 2019-12-05T06:28:32.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.003+0000 7f36183fd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3610085a50 con 0x7f36100871e0 2019-12-05T06:28:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.003+0000 7f361699a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f361014ca10 0x7f36100869e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39334/0 (socket says 172.21.15.16:39334) 2019-12-05T06:28:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.003+0000 7f361699a700 1 -- 172.21.15.16:0/2600264468 learned_addr learned my addr 172.21.15.16:0/2600264468 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:28:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.003+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> v2:172.21.15.16:3300/0 conn(0x7f36100871e0 0x7f361009ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f361719b700 1 -- 172.21.15.16:0/2600264468 >> v1:172.21.15.76:6789/0 conn(0x7f36100879b0 legacy=0x7f3610154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f361719b700 1 -- 172.21.15.16:0/2600264468 >> v1:172.21.15.16:6789/0 conn(0x7f361014ca10 legacy=0x7f36100869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f361719b700 1 -- 172.21.15.16:0/2600264468 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36100860e0 con 0x7f36100871e0 2019-12-05T06:28:32.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f360f7fe700 1 -- 172.21.15.16:0/2600264468 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556685999 0 0) 0x7f3604000fa0 con 0x7f36100879b0 2019-12-05T06:28:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> v2:172.21.15.16:3300/0 conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> v2:172.21.15.16:3300/0 conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=READY pgs=845 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=THROTTLE_DONE pgs=845 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=THROTTLE_DONE pgs=845 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f360f7fe700 1 -- 172.21.15.16:0/2600264468 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3600000ce0 con 0x7f36100871e0 2019-12-05T06:28:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=THROTTLE_DONE pgs=845 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f360f7fe700 1 -- 172.21.15.16:0/2600264468 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3600009d10 con 0x7f36100871e0 2019-12-05T06:28:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f36183fd700 1 -- 172.21.15.16:0/2600264468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36100860e0 con 0x7f36100871e0 2019-12-05T06:28:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.004+0000 7f360f7fe700 1 -- 172.21.15.16:0/2600264468 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f360000a460 con 0x7f36100871e0 2019-12-05T06:28:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.005+0000 7f36183fd700 1 -- 172.21.15.16:0/2600264468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36100860e0 con 0x7f36100871e0 2019-12-05T06:28:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.005+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=THROTTLE_DONE pgs=845 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.006+0000 7f360f7fe700 1 -- 172.21.15.16:0/2600264468 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3600009f00 con 0x7f36100871e0 2019-12-05T06:28:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.006+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=THROTTLE_DONE pgs=845 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.006+0000 7f360f7fe700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f35f0023630 0x7f35f0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:28:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.006+0000 7f360f7fe700 1 -- 172.21.15.16:0/2600264468 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f3600046190 con 0x7f36100871e0 2019-12-05T06:28:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.006+0000 7f361699a700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f35f0023630 0x7f35f0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:28:32.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.007+0000 7f361699a700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f35f0023630 0x7f35f0025ac0 crc :-1 s=READY pgs=367 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:28:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.012+0000 7f36183fd700 1 -- 172.21.15.16:0/2600264468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36100860e0 con 0x7f36100871e0 2019-12-05T06:28:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.016+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=THROTTLE_DONE pgs=845 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.016+0000 7f360f7fe700 1 -- 172.21.15.16:0/2600264468 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3600009910 con 0x7f36100871e0 2019-12-05T06:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.225+0000 7f36183fd700 1 -- 172.21.15.16:0/2600264468 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36100860e0 con 0x7f36100871e0 2019-12-05T06:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.225+0000 7f361719b700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=THROTTLE_DONE pgs=845 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.226+0000 7f360f7fe700 1 -- 172.21.15.16:0/2600264468 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3600009f00 con 0x7f36100871e0 2019-12-05T06:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 107 2019-12-05T06:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.228+0000 7f36183fd700 1 -- 172.21.15.16:0/2600264468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f35f0023630 msgr2=0x7f35f0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:32.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.228+0000 7f36183fd700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f35f0023630 0x7f35f0025ac0 crc :-1 s=READY pgs=367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:32.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.228+0000 7f36183fd700 1 -- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 msgr2=0x7f361009ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:28:32.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.228+0000 7f36183fd700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 0x7f361009ce90 secure :-1 s=READY pgs=845 cs=0 l=1 rx=0x7f3600010730 tx=0x7f3600010ad0).stop 2019-12-05T06:28:32.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.228+0000 7f36183fd700 1 -- 172.21.15.16:0/2600264468 shutdown_connections 2019-12-05T06:28:32.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.228+0000 7f36183fd700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f35f0023630 0x7f35f0025ac0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:32.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.228+0000 7f36183fd700 1 --2- 172.21.15.16:0/2600264468 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f36100871e0 0x7f361009ce90 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:28:32.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.228+0000 7f36183fd700 1 -- 172.21.15.16:0/2600264468 shutdown_connections 2019-12-05T06:28:32.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.228+0000 7f36183fd700 1 -- 172.21.15.16:0/2600264468 wait complete. 2019-12-05T06:28:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:28:32.228+0000 7f36183fd700 1 -- 172.21.15.16:0/2600264468 >> 172.21.15.16:0/2600264468 conn(0x7f3610098480 msgr2=0x7f3610071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:28:32.224 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:28:32.224 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6363,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:28:32.234 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:28:32.235 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 32 secs before thrashing 2019-12-05T06:28:34.930 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:28:34.934 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:29:04.236 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.392+0000 7f7cf6b5b700 1 Processor -- start 2019-12-05T06:29:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.392+0000 7f7cf6b5b700 1 -- start start 2019-12-05T06:29:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.393+0000 7f7cf6b5b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7cf008fef0 0x7f7cf008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.393+0000 7f7cf6b5b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf00958e0 con 0x7f7cf008c6d0 2019-12-05T06:29:04.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.393+0000 7f7cf6b5b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf00629e0 con 0x7f7cf008d2c0 2019-12-05T06:29:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.393+0000 7f7cf6b5b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf008df90 con 0x7f7cf008fef0 2019-12-05T06:29:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.393+0000 7f7cf50f8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7cf008c6d0 0x7f7cf008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39466/0 (socket says 172.21.15.16:39466) 2019-12-05T06:29:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.393+0000 7f7cf50f8700 1 -- 172.21.15.16:0/3954159790 learned_addr learned my addr 172.21.15.16:0/3954159790 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7cf58f9700 1 --2- 172.21.15.16:0/3954159790 >> v2:172.21.15.76:3300/0 conn(0x7f7cf008fef0 0x7f7cf008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518190876 0 0) 0x7f7ce0001150 con 0x7f7cf008c6d0 2019-12-05T06:29:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cd8002e80 con 0x7f7cf008c6d0 2019-12-05T06:29:04.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1698196948 0 0) 0x7f7ce0001130 con 0x7f7cf008c6d0 2019-12-05T06:29:04.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 >> v2:172.21.15.76:3300/0 conn(0x7f7cf008fef0 msgr2=0x7f7cf008f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:04.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 --2- 172.21.15.16:0/3954159790 >> v2:172.21.15.76:3300/0 conn(0x7f7cf008fef0 0x7f7cf008f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:04.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 >> v1:172.21.15.76:6789/0 conn(0x7f7cf008d2c0 legacy=0x7f7cf008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf008e620 con 0x7f7cf008c6d0 2019-12-05T06:29:04.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f7ce0001660 con 0x7f7cf008c6d0 2019-12-05T06:29:04.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 >> v1:172.21.15.16:6789/0 conn(0x7f7cf008c6d0 legacy=0x7f7cf008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 0x7f7cd8005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7cd8006690 0x7f7cd8008b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7cd80093a0 0x7f7cd800b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd800c020 con 0x7f7cd8003a80 2019-12-05T06:29:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 0x7f7cd8005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd8050220 con 0x7f7cd80093a0 2019-12-05T06:29:04.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cd8050450 con 0x7f7cd8006690 2019-12-05T06:29:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7cf48f7700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7cd8006690 0x7f7cd8008b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7cf58f9700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7cd80093a0 0x7f7cd800b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7cf50f8700 1 -- 172.21.15.16:0/3954159790 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7cd8006690 msgr2=0x7f7cd8008b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7cd8006690 0x7f7cd8008b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7cf50f8700 1 -- 172.21.15.16:0/3954159790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7cd80093a0 msgr2=0x7f7cd800b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:04.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7cd80093a0 0x7f7cd800b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7cf50f8700 1 -- 172.21.15.16:0/3954159790 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cd8050700 con 0x7f7cd8003a80 2019-12-05T06:29:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.394+0000 7f7cf58f9700 1 -- 172.21.15.16:0/3954159790 reap_dead start 2019-12-05T06:29:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 0x7f7cd8005ee0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7ce000f890 tx=0x7f7ce0001dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 0x7f7cd8005ee0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7ce000f890 tx=0x7f7ce0001dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 0x7f7cd8005ee0 secure :-1 s=READY pgs=850 cs=0 l=1 rx=0x7f7ce000f890 tx=0x7f7ce0001dd0).ready entity=mon.0 client_cookie=def7b544b5602a25 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 0x7f7cd8005ee0 secure :-1 s=THROTTLE_DONE pgs=850 cs=0 l=1 rx=0x7f7ce000f890 tx=0x7f7ce0001dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 0x7f7cd8005ee0 secure :-1 s=THROTTLE_DONE pgs=850 cs=0 l=1 rx=0x7f7ce000f890 tx=0x7f7ce0001dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ce0001130 con 0x7f7cd8003a80 2019-12-05T06:29:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 0x7f7cd8005ee0 secure :-1 s=THROTTLE_DONE pgs=850 cs=0 l=1 rx=0x7f7ce000f890 tx=0x7f7ce0001dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce0004240 con 0x7f7cd8003a80 2019-12-05T06:29:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7ceffff700 1 -- 172.21.15.16:0/3954159790 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ce0004910 con 0x7f7cd8003a80 2019-12-05T06:29:04.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 msgr2=0x7f7cd8005ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7cf6b5b700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 0x7f7cd8005ee0 secure :-1 s=READY pgs=850 cs=0 l=1 rx=0x7f7ce000f890 tx=0x7f7ce0001dd0).stop 2019-12-05T06:29:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/3954159790 shutdown_connections 2019-12-05T06:29:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.395+0000 7f7cf6b5b700 1 --2- 172.21.15.16:0/3954159790 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cd8003a80 0x7f7cd8005ee0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.396+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/3954159790 shutdown_connections 2019-12-05T06:29:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.396+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/3954159790 wait complete. 2019-12-05T06:29:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.396+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/3954159790 >> 172.21.15.16:0/3954159790 conn(0x7f7cf0061940 msgr2=0x7f7cf0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:04.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.398+0000 7f7cf6b5b700 1 Processor -- start 2019-12-05T06:29:04.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.398+0000 7f7cf6b5b700 1 -- start start 2019-12-05T06:29:04.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf6b5b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7cf01325e0 0x7f7cf012b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:04.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf6b5b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7cf0154cb0 0x7f7cf01570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:04.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf6b5b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cf0128b20 con 0x7f7cf012c740 2019-12-05T06:29:04.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf6b5b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf012a7e0 con 0x7f7cf01325e0 2019-12-05T06:29:04.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf6b5b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cf0129db0 con 0x7f7cf0154cb0 2019-12-05T06:29:04.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf50f8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7cf01325e0 0x7f7cf012b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf48f7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7cf012c740 0x7f7cf012bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39482/0 (socket says 172.21.15.16:39482) 2019-12-05T06:29:04.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf48f7700 1 -- 172.21.15.16:0/2241779170 learned_addr learned my addr 172.21.15.16:0/2241779170 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:04.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf58f9700 1 --2- 172.21.15.16:0/2241779170 >> v2:172.21.15.76:3300/0 conn(0x7f7cf0154cb0 0x7f7cf01570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf50f8700 1 -- 172.21.15.16:0/2241779170 >> v2:172.21.15.76:3300/0 conn(0x7f7cf0154cb0 msgr2=0x7f7cf01570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> v2:172.21.15.76:3300/0 conn(0x7f7cf0154cb0 0x7f7cf01570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:04.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cf50f8700 1 -- 172.21.15.16:0/2241779170 >> v1:172.21.15.16:6789/0 conn(0x7f7cf012c740 legacy=0x7f7cf012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cf50f8700 1 -- 172.21.15.16:0/2241779170 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf014a820 con 0x7f7cf01325e0 2019-12-05T06:29:04.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.399+0000 7f7cedffb700 1 -- 172.21.15.16:0/2241779170 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780955962 0 0) 0x7f7ce40010c0 con 0x7f7cf012c740 2019-12-05T06:29:04.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> v2:172.21.15.16:3300/0 conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> v2:172.21.15.16:3300/0 conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=READY pgs=852 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:04.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=THROTTLE_DONE pgs=852 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=THROTTLE_DONE pgs=852 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cedffb700 1 -- 172.21.15.16:0/2241779170 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ce0001660 con 0x7f7cf01325e0 2019-12-05T06:29:04.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=THROTTLE_DONE pgs=852 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cedffb700 1 -- 172.21.15.16:0/2241779170 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ce0004240 con 0x7f7cf01325e0 2019-12-05T06:29:04.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/2241779170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf012ae70 con 0x7f7cf01325e0 2019-12-05T06:29:04.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cedffb700 1 -- 172.21.15.16:0/2241779170 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ce0004910 con 0x7f7cf01325e0 2019-12-05T06:29:04.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/2241779170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cf014a820 con 0x7f7cf01325e0 2019-12-05T06:29:04.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.400+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=THROTTLE_DONE pgs=852 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.401+0000 7f7cedffb700 1 -- 172.21.15.16:0/2241779170 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7ce0004430 con 0x7f7cf01325e0 2019-12-05T06:29:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.401+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=THROTTLE_DONE pgs=852 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.401+0000 7f7cedffb700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cd0023630 0x7f7cd0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.401+0000 7f7cedffb700 1 -- 172.21.15.16:0/2241779170 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f7ce0045cd0 con 0x7f7cf01325e0 2019-12-05T06:29:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.401+0000 7f7cf48f7700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cd0023630 0x7f7cd0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.402+0000 7f7cf48f7700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cd0023630 0x7f7cd0025ac0 crc :-1 s=READY pgs=369 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:04.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.408+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/2241779170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cf012ae70 con 0x7f7cf01325e0 2019-12-05T06:29:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.412+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=THROTTLE_DONE pgs=852 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.412+0000 7f7cedffb700 1 -- 172.21.15.16:0/2241779170 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7ce001a1d0 con 0x7f7cf01325e0 2019-12-05T06:29:04.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.606+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/2241779170 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cf014a820 con 0x7f7cf01325e0 2019-12-05T06:29:04.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.607+0000 7f7cf50f8700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=THROTTLE_DONE pgs=852 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.607+0000 7f7cedffb700 1 -- 172.21.15.16:0/2241779170 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7ce001a1d0 con 0x7f7cf01325e0 2019-12-05T06:29:04.600 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 107 2019-12-05T06:29:04.600 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:04.600 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6363,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:04.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.609+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/2241779170 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cd0023630 msgr2=0x7f7cd0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.609+0000 7f7cf6b5b700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cd0023630 0x7f7cd0025ac0 crc :-1 s=READY pgs=369 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:04.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.609+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 msgr2=0x7f7cf012b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.609+0000 7f7cf6b5b700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 0x7f7cf012b770 secure :-1 s=READY pgs=852 cs=0 l=1 rx=0x7f7ce0010380 tx=0x7f7ce0010180).stop 2019-12-05T06:29:04.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.610+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/2241779170 shutdown_connections 2019-12-05T06:29:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.610+0000 7f7cf6b5b700 1 --2- 172.21.15.16:0/2241779170 >> v2:172.21.15.76:3300/0 conn(0x7f7cf0154cb0 0x7f7cf01570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.610+0000 7f7cf6b5b700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cd0023630 0x7f7cd0025ac0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.610+0000 7f7cf6b5b700 1 --2- 172.21.15.16:0/2241779170 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cf01325e0 0x7f7cf012b770 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.610+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/2241779170 shutdown_connections 2019-12-05T06:29:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.610+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/2241779170 wait complete. 2019-12-05T06:29:04.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.610+0000 7f7cf6b5b700 1 -- 172.21.15.16:0/2241779170 >> 172.21.15.16:0/2241779170 conn(0x7f7cf00a2850 msgr2=0x7f7cf006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:04.613 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:29:04.613 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:29:04.613 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:29:04.628 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:29:04.629 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:04.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.807+0000 7f2cc039f700 1 Processor -- start 2019-12-05T06:29:04.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.807+0000 7f2cc039f700 1 -- start start 2019-12-05T06:29:04.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.809+0000 7f2cc039f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2cb808c6d0 0x7f2cb808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:04.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.809+0000 7f2cc039f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2cb808fef0 0x7f2cb808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.809+0000 7f2cc039f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb80958e0 con 0x7f2cb808ef20 2019-12-05T06:29:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.809+0000 7f2cc039f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb808df90 con 0x7f2cb808c6d0 2019-12-05T06:29:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.809+0000 7f2cc039f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cb80a3f70 con 0x7f2cb808fef0 2019-12-05T06:29:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.809+0000 7f2cbe93c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2cb808c6d0 0x7f2cb808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.810+0000 7f2cbf13d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2cb808ef20 0x7f2cb808f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39492/0 (socket says 172.21.15.16:39492) 2019-12-05T06:29:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.810+0000 7f2cbf13d700 1 -- 172.21.15.16:0/2439447278 learned_addr learned my addr 172.21.15.16:0/2439447278 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:04.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.810+0000 7f2cbe13b700 1 --2- 172.21.15.16:0/2439447278 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2cb808fef0 0x7f2cb808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.810+0000 7f2cbe93c700 1 -- 172.21.15.16:0/2439447278 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2cb808fef0 msgr2=0x7f2cb808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.810+0000 7f2cbe93c700 1 --2- 172.21.15.16:0/2439447278 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2cb808fef0 0x7f2cb808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.810+0000 7f2cbe93c700 1 -- 172.21.15.16:0/2439447278 >> v1:172.21.15.16:6789/0 conn(0x7f2cb808ef20 legacy=0x7f2cb808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.810+0000 7f2cbd93a700 1 -- 172.21.15.16:0/2439447278 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316628279 0 0) 0x7f2cb00014d0 con 0x7f2cb808ef20 2019-12-05T06:29:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.810+0000 7f2cbe93c700 1 -- 172.21.15.16:0/2439447278 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb808d540 con 0x7f2cb808c6d0 2019-12-05T06:29:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.810+0000 7f2cbe13b700 1 --2- 172.21.15.16:0/2439447278 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2cb808fef0 0x7f2cb808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:29:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.810+0000 7f2cbe93c700 1 --2- 172.21.15.16:0/2439447278 >> v2:172.21.15.16:3300/0 conn(0x7f2cb808c6d0 0x7f2cb808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2cb400af50 tx=0x7f2cb400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.811+0000 7f2cbe93c700 1 --2- 172.21.15.16:0/2439447278 >> v2:172.21.15.16:3300/0 conn(0x7f2cb808c6d0 0x7f2cb808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2cb400af50 tx=0x7f2cb400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.811+0000 7f2cbe93c700 1 --2- 172.21.15.16:0/2439447278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb808c6d0 0x7f2cb808cac0 secure :-1 s=READY pgs=854 cs=0 l=1 rx=0x7f2cb400af50 tx=0x7f2cb400b760).ready entity=mon.0 client_cookie=4d28c4db2a864e0c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.811+0000 7f2cbe93c700 1 --2- 172.21.15.16:0/2439447278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb808c6d0 0x7f2cb808cac0 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f2cb400af50 tx=0x7f2cb400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.811+0000 7f2cbe93c700 1 --2- 172.21.15.16:0/2439447278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb808c6d0 0x7f2cb808cac0 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f2cb400af50 tx=0x7f2cb400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.811+0000 7f2cbd93a700 1 -- 172.21.15.16:0/2439447278 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cb4007210 con 0x7f2cb808c6d0 2019-12-05T06:29:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.811+0000 7f2cbe93c700 1 --2- 172.21.15.16:0/2439447278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb808c6d0 0x7f2cb808cac0 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7f2cb400af50 tx=0x7f2cb400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.811+0000 7f2cbd93a700 1 -- 172.21.15.16:0/2439447278 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb40099b0 con 0x7f2cb808c6d0 2019-12-05T06:29:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.811+0000 7f2cbd93a700 1 -- 172.21.15.16:0/2439447278 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cb4009fc0 con 0x7f2cb808c6d0 2019-12-05T06:29:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.811+0000 7f2cc039f700 1 -- 172.21.15.16:0/2439447278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb808c6d0 msgr2=0x7f2cb808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.811+0000 7f2cc039f700 1 --2- 172.21.15.16:0/2439447278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb808c6d0 0x7f2cb808cac0 secure :-1 s=READY pgs=854 cs=0 l=1 rx=0x7f2cb400af50 tx=0x7f2cb400b760).stop 2019-12-05T06:29:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.812+0000 7f2cc039f700 1 -- 172.21.15.16:0/2439447278 shutdown_connections 2019-12-05T06:29:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.812+0000 7f2cc039f700 1 --2- 172.21.15.16:0/2439447278 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2cb808fef0 0x7f2cb808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:04.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.812+0000 7f2cc039f700 1 --2- 172.21.15.16:0/2439447278 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb808c6d0 0x7f2cb808cac0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.812+0000 7f2cc039f700 1 -- 172.21.15.16:0/2439447278 shutdown_connections 2019-12-05T06:29:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.812+0000 7f2cc039f700 1 -- 172.21.15.16:0/2439447278 wait complete. 2019-12-05T06:29:04.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.812+0000 7f2cc039f700 1 -- 172.21.15.16:0/2439447278 >> 172.21.15.16:0/2439447278 conn(0x7f2cb8061940 msgr2=0x7f2cb8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:04.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.817+0000 7f2cc039f700 1 Processor -- start 2019-12-05T06:29:04.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.817+0000 7f2cc039f700 1 -- start start 2019-12-05T06:29:04.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.818+0000 7f2cc039f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2cb8133220 0x7f2cb809a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.818+0000 7f2cc039f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb806e7c0 con 0x7f2cb8132630 2019-12-05T06:29:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.818+0000 7f2cc039f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cb8133b30 con 0x7f2cb809ae80 2019-12-05T06:29:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.818+0000 7f2cc039f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cb8098f20 con 0x7f2cb8133220 2019-12-05T06:29:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.818+0000 7f2cbf13d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2cb8133220 0x7f2cb809a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.818+0000 7f2cbe93c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2cb8132630 0x7f2cb8132a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39494/0 (socket says 172.21.15.16:39494) 2019-12-05T06:29:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.818+0000 7f2cbe93c700 1 -- 172.21.15.16:0/2489638463 learned_addr learned my addr 172.21.15.16:0/2489638463 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cbf13d700 1 -- 172.21.15.16:0/2489638463 >> v1:172.21.15.76:6789/0 conn(0x7f2cb809ae80 legacy=0x7f2cb8099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cbf13d700 1 -- 172.21.15.16:0/2489638463 >> v1:172.21.15.16:6789/0 conn(0x7f2cb8132630 legacy=0x7f2cb8132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cbf13d700 1 -- 172.21.15.16:0/2489638463 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb80995b0 con 0x7f2cb8133220 2019-12-05T06:29:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cab7fe700 1 -- 172.21.15.16:0/2489638463 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091910733 0 0) 0x7f2cb400abb0 con 0x7f2cb8132630 2019-12-05T06:29:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cbf13d700 1 --2- 172.21.15.16:0/2489638463 >> v2:172.21.15.16:3300/0 conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cbf13d700 1 --2- 172.21.15.16:0/2489638463 >> v2:172.21.15.16:3300/0 conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cbf13d700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=READY pgs=856 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cbf13d700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cab7fe700 1 -- 172.21.15.16:0/2489638463 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cb00018f0 con 0x7f2cb8133220 2019-12-05T06:29:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cbf13d700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cc039f700 1 -- 172.21.15.16:0/2489638463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cb8133680 con 0x7f2cb8133220 2019-12-05T06:29:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cab7fe700 1 -- 172.21.15.16:0/2489638463 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb0000d60 con 0x7f2cb8133220 2019-12-05T06:29:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cbf13d700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cab7fe700 1 -- 172.21.15.16:0/2489638463 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cb0001990 con 0x7f2cb8133220 2019-12-05T06:29:04.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.819+0000 7f2cc039f700 1 -- 172.21.15.16:0/2489638463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cb80995b0 con 0x7f2cb8133220 2019-12-05T06:29:04.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.820+0000 7f2cbf13d700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.821+0000 7f2cab7fe700 1 -- 172.21.15.16:0/2489638463 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2cb0001510 con 0x7f2cb8133220 2019-12-05T06:29:04.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.821+0000 7f2cbf13d700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.821+0000 7f2cab7fe700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ca0023630 0x7f2ca0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:04.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.821+0000 7f2cab7fe700 1 -- 172.21.15.16:0/2489638463 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f2cb0046380 con 0x7f2cb8133220 2019-12-05T06:29:04.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.821+0000 7f2cbe93c700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ca0023630 0x7f2ca0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:04.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.822+0000 7f2cbe93c700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ca0023630 0x7f2ca0025ac0 crc :-1 s=READY pgs=370 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:04.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.822+0000 7f2cc039f700 1 -- 172.21.15.16:0/2489638463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cb8133680 con 0x7f2cb8133220 2019-12-05T06:29:04.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.831+0000 7f2cbf13d700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:04.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:04.831+0000 7f2cab7fe700 1 -- 172.21.15.16:0/2489638463 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2cb0000d60 con 0x7f2cb8133220 2019-12-05T06:29:05.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.048+0000 7f2cc039f700 1 -- 172.21.15.16:0/2489638463 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2cb80995b0 con 0x7f2cb8133220 2019-12-05T06:29:05.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.049+0000 7f2cbf13d700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=THROTTLE_DONE pgs=856 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:05.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.049+0000 7f2cab7fe700 1 -- 172.21.15.16:0/2489638463 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2cb0001270 con 0x7f2cb8133220 2019-12-05T06:29:05.043 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:05.043 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6363,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:05.043 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 107 2019-12-05T06:29:05.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.053+0000 7f2cc039f700 1 -- 172.21.15.16:0/2489638463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ca0023630 msgr2=0x7f2ca0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:05.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.053+0000 7f2cc039f700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ca0023630 0x7f2ca0025ac0 crc :-1 s=READY pgs=370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:05.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.053+0000 7f2cc039f700 1 -- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 msgr2=0x7f2cb809a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:05.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.053+0000 7f2cc039f700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 0x7f2cb809a6b0 secure :-1 s=READY pgs=856 cs=0 l=1 rx=0x7f2cb0010720 tx=0x7f2cb0010ac0).stop 2019-12-05T06:29:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.053+0000 7f2cc039f700 1 -- 172.21.15.16:0/2489638463 shutdown_connections 2019-12-05T06:29:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.053+0000 7f2cc039f700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ca0023630 0x7f2ca0025ac0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.053+0000 7f2cc039f700 1 --2- 172.21.15.16:0/2489638463 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2cb8133220 0x7f2cb809a6b0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:05.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.054+0000 7f2cc039f700 1 -- 172.21.15.16:0/2489638463 shutdown_connections 2019-12-05T06:29:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.054+0000 7f2cc039f700 1 -- 172.21.15.16:0/2489638463 wait complete. 2019-12-05T06:29:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:05.054+0000 7f2cc039f700 1 -- 172.21.15.16:0/2489638463 >> 172.21.15.16:0/2489638463 conn(0x7f2cb80714f0 msgr2=0x7f2cb809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:05.059 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:29:05.061 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:29:05.065 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:29:07.065 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.359+0000 7fa0831a2700 1 Processor -- start 2019-12-05T06:29:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.360+0000 7fa0831a2700 1 -- start start 2019-12-05T06:29:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.362+0000 7fa0831a2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa07c08c6d0 0x7fa07c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:07.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.362+0000 7fa0831a2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c08fb80 con 0x7fa07c08fef0 2019-12-05T06:29:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.362+0000 7fa0831a2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c0958e0 con 0x7fa07c08f720 2019-12-05T06:29:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.362+0000 7fa0831a2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa07c08df90 con 0x7fa07c08c6d0 2019-12-05T06:29:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.362+0000 7fa08173f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa07c08c6d0 0x7fa07c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.362+0000 7fa080f3e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa07c08fef0 0x7fa07c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39510/0 (socket says 172.21.15.16:39510) 2019-12-05T06:29:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.362+0000 7fa080f3e700 1 -- 172.21.15.16:0/2234298625 learned_addr learned my addr 172.21.15.16:0/2234298625 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:07.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.362+0000 7fa081f40700 1 --1- 172.21.15.16:0/2234298625 >> v1:172.21.15.76:6789/0 conn(0x7fa07c08f720 0x7fa07c1533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:60812/0 (socket says 172.21.15.16:60812) 2019-12-05T06:29:07.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.363+0000 7fa07bfff700 1 -- 172.21.15.16:0/2234298625 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004174900 0 0) 0x7fa0740009b0 con 0x7fa07c08f720 2019-12-05T06:29:07.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.363+0000 7fa08173f700 1 -- 172.21.15.16:0/2234298625 >> v1:172.21.15.76:6789/0 conn(0x7fa07c08f720 legacy=0x7fa07c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:07.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.363+0000 7fa08173f700 1 -- 172.21.15.16:0/2234298625 >> v1:172.21.15.16:6789/0 conn(0x7fa07c08fef0 legacy=0x7fa07c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:07.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.363+0000 7fa08173f700 1 -- 172.21.15.16:0/2234298625 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c08e620 con 0x7fa07c08c6d0 2019-12-05T06:29:07.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.363+0000 7fa08173f700 1 --2- 172.21.15.16:0/2234298625 >> v2:172.21.15.16:3300/0 conn(0x7fa07c08c6d0 0x7fa07c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa06c00b2d0 tx=0x7fa06c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.363+0000 7fa08173f700 1 --2- 172.21.15.16:0/2234298625 >> v2:172.21.15.16:3300/0 conn(0x7fa07c08c6d0 0x7fa07c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa06c00b2d0 tx=0x7fa06c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.363+0000 7fa08173f700 1 --2- 172.21.15.16:0/2234298625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa07c08c6d0 0x7fa07c08cac0 secure :-1 s=READY pgs=858 cs=0 l=1 rx=0x7fa06c00b2d0 tx=0x7fa06c00bae0).ready entity=mon.0 client_cookie=6009d6db4bcbd70f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:07.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.364+0000 7fa08173f700 1 --2- 172.21.15.16:0/2234298625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa07c08c6d0 0x7fa07c08cac0 secure :-1 s=THROTTLE_DONE pgs=858 cs=0 l=1 rx=0x7fa06c00b2d0 tx=0x7fa06c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.364+0000 7fa07bfff700 1 -- 172.21.15.16:0/2234298625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa06c000ce0 con 0x7fa07c08c6d0 2019-12-05T06:29:07.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.364+0000 7fa08173f700 1 --2- 172.21.15.16:0/2234298625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa07c08c6d0 0x7fa07c08cac0 secure :-1 s=THROTTLE_DONE pgs=858 cs=0 l=1 rx=0x7fa06c00b2d0 tx=0x7fa06c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.364+0000 7fa08173f700 1 --2- 172.21.15.16:0/2234298625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa07c08c6d0 0x7fa07c08cac0 secure :-1 s=THROTTLE_DONE pgs=858 cs=0 l=1 rx=0x7fa06c00b2d0 tx=0x7fa06c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.364+0000 7fa07bfff700 1 -- 172.21.15.16:0/2234298625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa06c009be0 con 0x7fa07c08c6d0 2019-12-05T06:29:07.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.364+0000 7fa07bfff700 1 -- 172.21.15.16:0/2234298625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa06c00a2b0 con 0x7fa07c08c6d0 2019-12-05T06:29:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.364+0000 7fa0831a2700 1 -- 172.21.15.16:0/2234298625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa07c08c6d0 msgr2=0x7fa07c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.364+0000 7fa0831a2700 1 --2- 172.21.15.16:0/2234298625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa07c08c6d0 0x7fa07c08cac0 secure :-1 s=READY pgs=858 cs=0 l=1 rx=0x7fa06c00b2d0 tx=0x7fa06c00bae0).stop 2019-12-05T06:29:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.366+0000 7fa0831a2700 1 -- 172.21.15.16:0/2234298625 shutdown_connections 2019-12-05T06:29:07.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.366+0000 7fa0831a2700 1 --2- 172.21.15.16:0/2234298625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa07c08c6d0 0x7fa07c08cac0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.366+0000 7fa0831a2700 1 -- 172.21.15.16:0/2234298625 shutdown_connections 2019-12-05T06:29:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.366+0000 7fa0831a2700 1 -- 172.21.15.16:0/2234298625 wait complete. 2019-12-05T06:29:07.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.366+0000 7fa0831a2700 1 -- 172.21.15.16:0/2234298625 >> 172.21.15.16:0/2234298625 conn(0x7fa07c061940 msgr2=0x7fa07c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:07.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.372+0000 7fa0831a2700 1 Processor -- start 2019-12-05T06:29:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.372+0000 7fa0831a2700 1 -- start start 2019-12-05T06:29:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.374+0000 7fa0831a2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa07c0992d0 0x7fa07c14f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.374+0000 7fa0831a2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa07c098b00 0x7fa07c1592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:07.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.374+0000 7fa0831a2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa07c098f60 con 0x7fa07c14f5b0 2019-12-05T06:29:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.374+0000 7fa0831a2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa07c097370 con 0x7fa07c098b00 2019-12-05T06:29:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.374+0000 7fa0831a2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa07c08ad00 con 0x7fa07c0992d0 2019-12-05T06:29:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.374+0000 7fa08173f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa07c14f5b0 0x7fa07c098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39514/0 (socket says 172.21.15.16:39514) 2019-12-05T06:29:07.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.374+0000 7fa08173f700 1 -- 172.21.15.16:0/2482982357 learned_addr learned my addr 172.21.15.16:0/2482982357 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.374+0000 7fa080f3e700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa07c0992d0 0x7fa07c14f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.374+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> v2:172.21.15.76:3300/0 conn(0x7fa07c098b00 0x7fa07c1592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.375+0000 7fa079ffb700 1 -- 172.21.15.16:0/2482982357 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167279425 0 0) 0x7fa06c00b750 con 0x7fa07c14f5b0 2019-12-05T06:29:07.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.375+0000 7fa079ffb700 1 -- 172.21.15.16:0/2482982357 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa05c002e80 con 0x7fa07c14f5b0 2019-12-05T06:29:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.375+0000 7fa081f40700 1 -- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa07c0992d0 msgr2=0x7fa07c14f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.375+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa07c0992d0 0x7fa07c14f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.375+0000 7fa081f40700 1 -- 172.21.15.16:0/2482982357 >> v1:172.21.15.16:6789/0 conn(0x7fa07c14f5b0 legacy=0x7fa07c098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:07.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.375+0000 7fa081f40700 1 -- 172.21.15.16:0/2482982357 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa07c097a00 con 0x7fa07c098b00 2019-12-05T06:29:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.375+0000 7fa080f3e700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa07c0992d0 0x7fa07c14f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:29:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.375+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> v2:172.21.15.76:3300/0 conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.375+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> v2:172.21.15.76:3300/0 conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.375+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=READY pgs=616 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:07.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.376+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=THROTTLE_DONE pgs=616 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.376+0000 7fa079ffb700 1 -- 172.21.15.16:0/2482982357 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa074000940 con 0x7fa07c098b00 2019-12-05T06:29:07.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.376+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=THROTTLE_DONE pgs=616 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.376+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=THROTTLE_DONE pgs=616 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.376+0000 7fa079ffb700 1 -- 172.21.15.16:0/2482982357 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa074001120 con 0x7fa07c098b00 2019-12-05T06:29:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.376+0000 7fa0831a2700 1 -- 172.21.15.16:0/2482982357 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa07c097a00 con 0x7fa07c098b00 2019-12-05T06:29:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.376+0000 7fa079ffb700 1 -- 172.21.15.16:0/2482982357 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa074001870 con 0x7fa07c098b00 2019-12-05T06:29:07.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.376+0000 7fa0831a2700 1 -- 172.21.15.16:0/2482982357 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa07c097a00 con 0x7fa07c098b00 2019-12-05T06:29:07.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.377+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=THROTTLE_DONE pgs=616 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.378+0000 7fa079ffb700 1 -- 172.21.15.16:0/2482982357 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa074001310 con 0x7fa07c098b00 2019-12-05T06:29:07.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.378+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=THROTTLE_DONE pgs=616 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.378+0000 7fa079ffb700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa05c023670 0x7fa05c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.379+0000 7fa079ffb700 1 -- 172.21.15.16:0/2482982357 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7fa074045d50 con 0x7fa07c098b00 2019-12-05T06:29:07.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.379+0000 7fa08173f700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa05c023670 0x7fa05c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:07.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.379+0000 7fa08173f700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa05c023670 0x7fa05c025ae0 crc :-1 s=READY pgs=371 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:07.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.391+0000 7fa0831a2700 1 -- 172.21.15.16:0/2482982357 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa07c097a00 con 0x7fa07c098b00 2019-12-05T06:29:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.401+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=THROTTLE_DONE pgs=616 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.401+0000 7fa079ffb700 1 -- 172.21.15.16:0/2482982357 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa074000e70 con 0x7fa07c098b00 2019-12-05T06:29:07.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.654+0000 7fa0831a2700 1 -- 172.21.15.16:0/2482982357 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa07c097a00 con 0x7fa07c098b00 2019-12-05T06:29:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.655+0000 7fa081f40700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=THROTTLE_DONE pgs=616 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:07.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.655+0000 7fa079ffb700 1 -- 172.21.15.16:0/2482982357 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa074001bc0 con 0x7fa07c098b00 2019-12-05T06:29:07.649 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 107 2019-12-05T06:29:07.649 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:07.649 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6363,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:07.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 -- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa05c023670 msgr2=0x7fa05c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa05c023670 0x7fa05c025ae0 crc :-1 s=READY pgs=371 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 -- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 msgr2=0x7fa07c1592b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 0x7fa07c1592b0 secure :-1 s=READY pgs=616 cs=0 l=1 rx=0x7fa0740103a0 tx=0x7fa074010740).stop 2019-12-05T06:29:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 -- 172.21.15.16:0/2482982357 shutdown_connections 2019-12-05T06:29:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa05c023670 0x7fa05c025ae0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa07c098b00 0x7fa07c1592b0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:07.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 --2- 172.21.15.16:0/2482982357 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa07c0992d0 0x7fa07c14f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 -- 172.21.15.16:0/2482982357 shutdown_connections 2019-12-05T06:29:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 -- 172.21.15.16:0/2482982357 wait complete. 2019-12-05T06:29:07.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:07.659+0000 7fa0831a2700 1 -- 172.21.15.16:0/2482982357 >> 172.21.15.16:0/2482982357 conn(0x7fa07c0714f0 msgr2=0x7fa07c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:07.663 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:29:09.663 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:09.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.910+0000 7faa8fd92700 1 Processor -- start 2019-12-05T06:29:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.910+0000 7faa8fd92700 1 -- start start 2019-12-05T06:29:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.912+0000 7faa8fd92700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7faa8808c6d0 0x7faa8808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.912+0000 7faa8fd92700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faa8808fef0 0x7faa8808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.912+0000 7faa8fd92700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa880958e0 con 0x7faa8808ef20 2019-12-05T06:29:09.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.912+0000 7faa8fd92700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8808df90 con 0x7faa8808c6d0 2019-12-05T06:29:09.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.912+0000 7faa8fd92700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa880a3f70 con 0x7faa8808fef0 2019-12-05T06:29:09.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.912+0000 7faa8db2e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faa8808fef0 0x7faa8808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:09.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.912+0000 7faa8e32f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7faa8808c6d0 0x7faa8808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:09.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.913+0000 7faa8eb30700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7faa8808ef20 0x7faa8808f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:60828/0 (socket says 172.21.15.16:60828) 2019-12-05T06:29:09.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.913+0000 7faa8eb30700 1 -- 172.21.15.16:0/2084586581 learned_addr learned my addr 172.21.15.16:0/2084586581 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:09.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.913+0000 7faa8e32f700 1 --2- 172.21.15.16:0/2084586581 >> v2:172.21.15.76:3300/0 conn(0x7faa8808c6d0 0x7faa8808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:40446/0 (socket says 172.21.15.16:40446) 2019-12-05T06:29:09.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.913+0000 7faa8db2e700 1 --2- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faa8808fef0 0x7faa8808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:58350/0 (socket says 172.21.15.16:58350) 2019-12-05T06:29:09.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.913+0000 7faa8e32f700 1 -- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faa8808fef0 msgr2=0x7faa8808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:09.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.913+0000 7faa8e32f700 1 --2- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faa8808fef0 0x7faa8808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:09.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.913+0000 7faa8e32f700 1 -- 172.21.15.16:0/2084586581 >> v1:172.21.15.76:6789/0 conn(0x7faa8808ef20 legacy=0x7faa8808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:09.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.913+0000 7faa8d32d700 1 -- 172.21.15.16:0/2084586581 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2057917219 0 0) 0x7faa78001100 con 0x7faa8808ef20 2019-12-05T06:29:09.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.913+0000 7faa8e32f700 1 -- 172.21.15.16:0/2084586581 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8808d540 con 0x7faa8808c6d0 2019-12-05T06:29:09.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.913+0000 7faa8db2e700 1 --2- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faa8808fef0 0x7faa8808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:29:09.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.914+0000 7faa8e32f700 1 --2- 172.21.15.16:0/2084586581 >> v2:172.21.15.76:3300/0 conn(0x7faa8808c6d0 0x7faa8808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faa8000b2d0 tx=0x7faa8000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.914+0000 7faa8e32f700 1 --2- 172.21.15.16:0/2084586581 >> v2:172.21.15.76:3300/0 conn(0x7faa8808c6d0 0x7faa8808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faa8000b2d0 tx=0x7faa8000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.914+0000 7faa8e32f700 1 --2- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa8808c6d0 0x7faa8808cac0 secure :-1 s=READY pgs=618 cs=0 l=1 rx=0x7faa8000b2d0 tx=0x7faa8000bae0).ready entity=mon.1 client_cookie=456c941ca12eb38c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:09.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.914+0000 7faa8e32f700 1 --2- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa8808c6d0 0x7faa8808cac0 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7faa8000b2d0 tx=0x7faa8000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.914+0000 7faa8e32f700 1 --2- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa8808c6d0 0x7faa8808cac0 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7faa8000b2d0 tx=0x7faa8000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.914+0000 7faa8d32d700 1 -- 172.21.15.16:0/2084586581 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa80007590 con 0x7faa8808c6d0 2019-12-05T06:29:09.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.914+0000 7faa8e32f700 1 --2- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa8808c6d0 0x7faa8808cac0 secure :-1 s=THROTTLE_DONE pgs=618 cs=0 l=1 rx=0x7faa8000b2d0 tx=0x7faa8000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.914+0000 7faa8d32d700 1 -- 172.21.15.16:0/2084586581 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa80009d30 con 0x7faa8808c6d0 2019-12-05T06:29:09.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.914+0000 7faa8d32d700 1 -- 172.21.15.16:0/2084586581 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa8000a300 con 0x7faa8808c6d0 2019-12-05T06:29:09.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.915+0000 7faa8fd92700 1 -- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa8808c6d0 msgr2=0x7faa8808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:09.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.915+0000 7faa8fd92700 1 --2- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa8808c6d0 0x7faa8808cac0 secure :-1 s=READY pgs=618 cs=0 l=1 rx=0x7faa8000b2d0 tx=0x7faa8000bae0).stop 2019-12-05T06:29:09.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.917+0000 7faa8fd92700 1 -- 172.21.15.16:0/2084586581 shutdown_connections 2019-12-05T06:29:09.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.917+0000 7faa8fd92700 1 --2- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faa8808fef0 0x7faa8808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:09.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.917+0000 7faa8fd92700 1 --2- 172.21.15.16:0/2084586581 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7faa8808c6d0 0x7faa8808cac0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:09.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.917+0000 7faa8fd92700 1 -- 172.21.15.16:0/2084586581 shutdown_connections 2019-12-05T06:29:09.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.917+0000 7faa8fd92700 1 -- 172.21.15.16:0/2084586581 wait complete. 2019-12-05T06:29:09.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.917+0000 7faa8fd92700 1 -- 172.21.15.16:0/2084586581 >> 172.21.15.16:0/2084586581 conn(0x7faa88061940 msgr2=0x7faa88096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:09.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.923+0000 7faa8fd92700 1 Processor -- start 2019-12-05T06:29:09.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.923+0000 7faa8fd92700 1 -- start start 2019-12-05T06:29:09.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.925+0000 7faa8fd92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faa88096e90 0x7faa8809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:09.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.925+0000 7faa8fd92700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7faa88084f30 0x7faa8809b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:09.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.925+0000 7faa8fd92700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa88084b90 con 0x7faa88097280 2019-12-05T06:29:09.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.925+0000 7faa8fd92700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa8809a450 con 0x7faa88096e90 2019-12-05T06:29:09.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.925+0000 7faa8fd92700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa88099a20 con 0x7faa88084f30 2019-12-05T06:29:09.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.925+0000 7faa8e32f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faa88096e90 0x7faa8809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:09.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.925+0000 7faa8e32f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faa88096e90 0x7faa8809b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54362/0 (socket says 172.21.15.16:54362) 2019-12-05T06:29:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.925+0000 7faa8e32f700 1 -- 172.21.15.16:0/3963971136 learned_addr learned my addr 172.21.15.16:0/3963971136 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:29:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.925+0000 7faa8db2e700 1 --2- 172.21.15.16:0/3963971136 >> v2:172.21.15.76:3300/0 conn(0x7faa88084f30 0x7faa8809b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.926+0000 7faa8e32f700 1 -- 172.21.15.16:0/3963971136 >> v2:172.21.15.76:3300/0 conn(0x7faa88084f30 msgr2=0x7faa8809b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.926+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> v2:172.21.15.76:3300/0 conn(0x7faa88084f30 0x7faa8809b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.926+0000 7faa8e32f700 1 -- 172.21.15.16:0/3963971136 >> v1:172.21.15.76:6789/0 conn(0x7faa88097280 legacy=0x7faa880976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:09.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.926+0000 7faa8e32f700 1 -- 172.21.15.16:0/3963971136 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8809aae0 con 0x7faa88096e90 2019-12-05T06:29:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.926+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> v2:172.21.15.16:3300/0 conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.926+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> v2:172.21.15.16:3300/0 conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.926+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=READY pgs=860 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.927+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=THROTTLE_DONE pgs=860 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.927+0000 7faa86ffd700 1 -- 172.21.15.16:0/3963971136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa8000af30 con 0x7faa88096e90 2019-12-05T06:29:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.927+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=THROTTLE_DONE pgs=860 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.927+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=THROTTLE_DONE pgs=860 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.927+0000 7faa86ffd700 1 -- 172.21.15.16:0/3963971136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa80009d00 con 0x7faa88096e90 2019-12-05T06:29:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.927+0000 7faa86ffd700 1 -- 172.21.15.16:0/3963971136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa8000a3d0 con 0x7faa88096e90 2019-12-05T06:29:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.927+0000 7faa8fd92700 1 -- 172.21.15.16:0/3963971136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8809aae0 con 0x7faa88096e90 2019-12-05T06:29:09.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.927+0000 7faa8fd92700 1 -- 172.21.15.16:0/3963971136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8809aae0 con 0x7faa88096e90 2019-12-05T06:29:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.928+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=THROTTLE_DONE pgs=860 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.929+0000 7faa86ffd700 1 -- 172.21.15.16:0/3963971136 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faa80009ef0 con 0x7faa88096e90 2019-12-05T06:29:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.929+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=THROTTLE_DONE pgs=860 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.929+0000 7faa86ffd700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa68023630 0x7faa68025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:09.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.929+0000 7faa86ffd700 1 -- 172.21.15.16:0/3963971136 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7faa80046130 con 0x7faa88096e90 2019-12-05T06:29:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.930+0000 7faa8db2e700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa68023630 0x7faa68025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:09.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.930+0000 7faa8db2e700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa68023630 0x7faa68025ac0 crc :-1 s=READY pgs=372 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:09.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.942+0000 7faa8fd92700 1 -- 172.21.15.16:0/3963971136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa8809aae0 con 0x7faa88096e90 2019-12-05T06:29:09.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.950+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=THROTTLE_DONE pgs=860 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:09.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:09.951+0000 7faa86ffd700 1 -- 172.21.15.16:0/3963971136 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faa80009900 con 0x7faa88096e90 2019-12-05T06:29:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.216+0000 7faa8fd92700 1 -- 172.21.15.16:0/3963971136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faa8809aae0 con 0x7faa88096e90 2019-12-05T06:29:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.217+0000 7faa8e32f700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=THROTTLE_DONE pgs=860 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.217+0000 7faa86ffd700 1 -- 172.21.15.16:0/3963971136 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2112 (secure 0 0 0) 0x7faa8000b540 con 0x7faa88096e90 2019-12-05T06:29:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 107 2019-12-05T06:29:10.210 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:10.210 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6363,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.219+0000 7faa8fd92700 1 -- 172.21.15.16:0/3963971136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa68023630 msgr2=0x7faa68025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.219+0000 7faa8fd92700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa68023630 0x7faa68025ac0 crc :-1 s=READY pgs=372 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.219+0000 7faa8fd92700 1 -- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 msgr2=0x7faa8809b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.219+0000 7faa8fd92700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 0x7faa8809b3e0 secure :-1 s=READY pgs=860 cs=0 l=1 rx=0x7faa800107a0 tx=0x7faa80010620).stop 2019-12-05T06:29:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.220+0000 7faa8fd92700 1 -- 172.21.15.16:0/3963971136 shutdown_connections 2019-12-05T06:29:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.220+0000 7faa8fd92700 1 --2- 172.21.15.16:0/3963971136 >> v2:172.21.15.76:3300/0 conn(0x7faa88084f30 0x7faa8809b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.220+0000 7faa8fd92700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa68023630 0x7faa68025ac0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.220+0000 7faa8fd92700 1 --2- 172.21.15.16:0/3963971136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faa88096e90 0x7faa8809b3e0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.220+0000 7faa8fd92700 1 -- 172.21.15.16:0/3963971136 shutdown_connections 2019-12-05T06:29:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.220+0000 7faa8fd92700 1 -- 172.21.15.16:0/3963971136 wait complete. 2019-12-05T06:29:10.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:10.220+0000 7faa8fd92700 1 -- 172.21.15.16:0/3963971136 >> 172.21.15.16:0/3963971136 conn(0x7faa880714f0 msgr2=0x7faa8809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:10.223 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:29:12.224 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:12.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.500+0000 7f3682f12700 1 Processor -- start 2019-12-05T06:29:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.500+0000 7f3682f12700 1 -- start start 2019-12-05T06:29:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.502+0000 7f3682f12700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f367c08c6d0 0x7f367c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.502+0000 7f3682f12700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f367c08fef0 0x7f367c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.502+0000 7f3682f12700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367c08fb80 con 0x7f367c08ef20 2019-12-05T06:29:12.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.502+0000 7f3682f12700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367c08df90 con 0x7f367c08fef0 2019-12-05T06:29:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.502+0000 7f3682f12700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367c0a3f70 con 0x7f367c08c6d0 2019-12-05T06:29:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.502+0000 7f3680cae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f367c08fef0 0x7f367c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.502+0000 7f3680cae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f367c08fef0 0x7f367c08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54376/0 (socket says 172.21.15.16:54376) 2019-12-05T06:29:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.502+0000 7f3680cae700 1 -- 172.21.15.16:0/4018400671 learned_addr learned my addr 172.21.15.16:0/4018400671 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:29:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.502+0000 7f36814af700 1 --2- 172.21.15.16:0/4018400671 >> v2:172.21.15.76:3300/0 conn(0x7f367c08c6d0 0x7f367c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:12.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.503+0000 7f3680cae700 1 -- 172.21.15.16:0/4018400671 >> v2:172.21.15.76:3300/0 conn(0x7f367c08c6d0 msgr2=0x7f367c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.503+0000 7f3680cae700 1 --2- 172.21.15.16:0/4018400671 >> v2:172.21.15.76:3300/0 conn(0x7f367c08c6d0 0x7f367c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.503+0000 7f3680cae700 1 -- 172.21.15.16:0/4018400671 >> v1:172.21.15.76:6789/0 conn(0x7f367c08ef20 legacy=0x7f367c08f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.503+0000 7f3680cae700 1 -- 172.21.15.16:0/4018400671 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c08d540 con 0x7f367c08fef0 2019-12-05T06:29:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.503+0000 7f3680cae700 1 --2- 172.21.15.16:0/4018400671 >> v2:172.21.15.16:3300/0 conn(0x7f367c08fef0 0x7f367c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f367000f740 tx=0x7f367000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.503+0000 7f3680cae700 1 --2- 172.21.15.16:0/4018400671 >> v2:172.21.15.16:3300/0 conn(0x7f367c08fef0 0x7f367c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f367000f740 tx=0x7f367000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.503+0000 7f3680cae700 1 --2- 172.21.15.16:0/4018400671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c08fef0 0x7f367c08d000 secure :-1 s=READY pgs=861 cs=0 l=1 rx=0x7f367000f740 tx=0x7f367000f570).ready entity=mon.0 client_cookie=737e0ed8640ad2cb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.504+0000 7f3680cae700 1 --2- 172.21.15.16:0/4018400671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c08fef0 0x7f367c08d000 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7f367000f740 tx=0x7f367000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.504+0000 7f3680cae700 1 --2- 172.21.15.16:0/4018400671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c08fef0 0x7f367c08d000 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7f367000f740 tx=0x7f367000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.504+0000 7f367bfff700 1 -- 172.21.15.16:0/4018400671 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3670000960 con 0x7f367c08fef0 2019-12-05T06:29:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.504+0000 7f3680cae700 1 --2- 172.21.15.16:0/4018400671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c08fef0 0x7f367c08d000 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7f367000f740 tx=0x7f367000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.504+0000 7f367bfff700 1 -- 172.21.15.16:0/4018400671 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3670000c90 con 0x7f367c08fef0 2019-12-05T06:29:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.504+0000 7f367bfff700 1 -- 172.21.15.16:0/4018400671 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36700013a0 con 0x7f367c08fef0 2019-12-05T06:29:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.504+0000 7f3682f12700 1 -- 172.21.15.16:0/4018400671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c08fef0 msgr2=0x7f367c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.504+0000 7f3682f12700 1 --2- 172.21.15.16:0/4018400671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c08fef0 0x7f367c08d000 secure :-1 s=READY pgs=861 cs=0 l=1 rx=0x7f367000f740 tx=0x7f367000f570).stop 2019-12-05T06:29:12.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.506+0000 7f3682f12700 1 -- 172.21.15.16:0/4018400671 shutdown_connections 2019-12-05T06:29:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.506+0000 7f3682f12700 1 --2- 172.21.15.16:0/4018400671 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c08fef0 0x7f367c08d000 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.506+0000 7f3682f12700 1 --2- 172.21.15.16:0/4018400671 >> v2:172.21.15.76:3300/0 conn(0x7f367c08c6d0 0x7f367c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.506+0000 7f3682f12700 1 -- 172.21.15.16:0/4018400671 shutdown_connections 2019-12-05T06:29:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.506+0000 7f3682f12700 1 -- 172.21.15.16:0/4018400671 wait complete. 2019-12-05T06:29:12.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.506+0000 7f3682f12700 1 -- 172.21.15.16:0/4018400671 >> 172.21.15.16:0/4018400671 conn(0x7f367c061940 msgr2=0x7f367c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.511+0000 7f3682f12700 1 Processor -- start 2019-12-05T06:29:12.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.511+0000 7f3682f12700 1 -- start start 2019-12-05T06:29:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.513+0000 7f3682f12700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f367c096e90 0x7f367c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.513+0000 7f3682f12700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f367c09b920 0x7f367c097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.513+0000 7f3682f12700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f367c084b90 con 0x7f367c084f30 2019-12-05T06:29:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.513+0000 7f3682f12700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f367c09a450 con 0x7f367c096e90 2019-12-05T06:29:12.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.513+0000 7f3682f12700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f367c099a20 con 0x7f367c09b920 2019-12-05T06:29:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.513+0000 7f36814af700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f367c096e90 0x7f367c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.513+0000 7f3680cae700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f367c084f30 0x7f367c097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39548/0 (socket says 172.21.15.16:39548) 2019-12-05T06:29:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.513+0000 7f3680cae700 1 -- 172.21.15.16:0/3071879713 learned_addr learned my addr 172.21.15.16:0/3071879713 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.513+0000 7f3681cb0700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f367c09b920 0x7f367c097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:12.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.514+0000 7f36814af700 1 -- 172.21.15.16:0/3071879713 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f367c09b920 msgr2=0x7f367c097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.514+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f367c09b920 0x7f367c097a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:12.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.514+0000 7f36814af700 1 -- 172.21.15.16:0/3071879713 >> v1:172.21.15.16:6789/0 conn(0x7f367c084f30 legacy=0x7f367c097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.514+0000 7f36814af700 1 -- 172.21.15.16:0/3071879713 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f367c09aae0 con 0x7f367c096e90 2019-12-05T06:29:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.514+0000 7f3679ffb700 1 -- 172.21.15.16:0/3071879713 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 747062141 0 0) 0x7f3670008070 con 0x7f367c084f30 2019-12-05T06:29:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.514+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> v2:172.21.15.16:3300/0 conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.514+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> v2:172.21.15.16:3300/0 conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.514+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=READY pgs=863 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.515+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.515+0000 7f3679ffb700 1 -- 172.21.15.16:0/3071879713 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3674000cc0 con 0x7f367c096e90 2019-12-05T06:29:12.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.515+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.515+0000 7f3682f12700 1 -- 172.21.15.16:0/3071879713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f367c09aae0 con 0x7f367c096e90 2019-12-05T06:29:12.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.515+0000 7f3679ffb700 1 -- 172.21.15.16:0/3071879713 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3674000cc0 con 0x7f367c096e90 2019-12-05T06:29:12.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.515+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.515+0000 7f3679ffb700 1 -- 172.21.15.16:0/3071879713 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3674001490 con 0x7f367c096e90 2019-12-05T06:29:12.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.515+0000 7f3682f12700 1 -- 172.21.15.16:0/3071879713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f367c09aae0 con 0x7f367c096e90 2019-12-05T06:29:12.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.516+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.517+0000 7f3679ffb700 1 -- 172.21.15.16:0/3071879713 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3674001010 con 0x7f367c096e90 2019-12-05T06:29:12.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.517+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.517+0000 7f3679ffb700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f365c023630 0x7f365c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:12.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.517+0000 7f3679ffb700 1 -- 172.21.15.16:0/3071879713 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f3674046080 con 0x7f367c096e90 2019-12-05T06:29:12.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.517+0000 7f3680cae700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f365c023630 0x7f365c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:12.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.518+0000 7f3680cae700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f365c023630 0x7f365c025ac0 crc :-1 s=READY pgs=373 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:12.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.532+0000 7f3682f12700 1 -- 172.21.15.16:0/3071879713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f367c09aae0 con 0x7f367c096e90 2019-12-05T06:29:12.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.540+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.540+0000 7f3679ffb700 1 -- 172.21.15.16:0/3071879713 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f367403fff0 con 0x7f367c096e90 2019-12-05T06:29:12.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.799+0000 7f3682f12700 1 -- 172.21.15.16:0/3071879713 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f367c09aae0 con 0x7f367c096e90 2019-12-05T06:29:12.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.799+0000 7f36814af700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=THROTTLE_DONE pgs=863 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:12.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.799+0000 7f3679ffb700 1 -- 172.21.15.16:0/3071879713 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f367403fff0 con 0x7f367c096e90 2019-12-05T06:29:12.793 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 107 2019-12-05T06:29:12.793 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:12.794 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6363,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:12.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.802+0000 7f3682f12700 1 -- 172.21.15.16:0/3071879713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f365c023630 msgr2=0x7f365c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.802+0000 7f3682f12700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f365c023630 0x7f365c025ac0 crc :-1 s=READY pgs=373 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.802+0000 7f3682f12700 1 -- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 msgr2=0x7f367c09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:12.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.802+0000 7f3682f12700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 0x7f367c09b3e0 secure :-1 s=READY pgs=863 cs=0 l=1 rx=0x7f3674010660 tx=0x7f3674010a00).stop 2019-12-05T06:29:12.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.803+0000 7f3682f12700 1 -- 172.21.15.16:0/3071879713 shutdown_connections 2019-12-05T06:29:12.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.803+0000 7f3682f12700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f367c09b920 0x7f367c097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:12.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.803+0000 7f3682f12700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f365c023630 0x7f365c025ac0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:12.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.803+0000 7f3682f12700 1 --2- 172.21.15.16:0/3071879713 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f367c096e90 0x7f367c09b3e0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:12.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.803+0000 7f3682f12700 1 -- 172.21.15.16:0/3071879713 shutdown_connections 2019-12-05T06:29:12.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.803+0000 7f3682f12700 1 -- 172.21.15.16:0/3071879713 wait complete. 2019-12-05T06:29:12.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:12.803+0000 7f3682f12700 1 -- 172.21.15.16:0/3071879713 >> 172.21.15.16:0/3071879713 conn(0x7f367c0714f0 msgr2=0x7f367c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:12.811 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:29:14.812 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:15.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.085+0000 7ff02b8e0700 1 Processor -- start 2019-12-05T06:29:15.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.086+0000 7ff02b8e0700 1 -- start start 2019-12-05T06:29:15.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.087+0000 7ff02b8e0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff02408c6d0 0x7ff02408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:15.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.087+0000 7ff02b8e0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff02408fef0 0x7ff02408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:15.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.087+0000 7ff02b8e0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff02408ef20 0x7ff02408f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:15.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.087+0000 7ff02b8e0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff02408df90 con 0x7ff02408ef20 2019-12-05T06:29:15.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.087+0000 7ff02b8e0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0240a3f70 con 0x7ff02408c6d0 2019-12-05T06:29:15.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.087+0000 7ff02b8e0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff02408d830 con 0x7ff02408fef0 2019-12-05T06:29:15.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.088+0000 7ff02a67e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff02408ef20 0x7ff02408f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:15.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.088+0000 7ff02a67e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff02408ef20 0x7ff02408f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54388/0 (socket says 172.21.15.16:54388) 2019-12-05T06:29:15.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.088+0000 7ff02967c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff02408fef0 0x7ff02408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:15.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.088+0000 7ff02a67e700 1 -- 172.21.15.16:0/3615814226 learned_addr learned my addr 172.21.15.16:0/3615814226 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:29:15.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.088+0000 7ff029e7d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff02408c6d0 0x7ff02408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:15.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.089+0000 7ff02a67e700 1 -- 172.21.15.16:0/3615814226 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff02408fef0 msgr2=0x7ff02408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:15.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.089+0000 7ff02a67e700 1 --2- 172.21.15.16:0/3615814226 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff02408fef0 0x7ff02408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:15.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.089+0000 7ff02a67e700 1 -- 172.21.15.16:0/3615814226 >> v2:172.21.15.76:3300/0 conn(0x7ff02408c6d0 msgr2=0x7ff02408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:15.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.089+0000 7ff02a67e700 1 --2- 172.21.15.16:0/3615814226 >> v2:172.21.15.76:3300/0 conn(0x7ff02408c6d0 0x7ff02408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:15.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.089+0000 7ff02a67e700 1 -- 172.21.15.16:0/3615814226 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff02408d540 con 0x7ff02408ef20 2019-12-05T06:29:15.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.089+0000 7ff02967c700 1 --2- 172.21.15.16:0/3615814226 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff02408fef0 0x7ff02408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:29:15.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.089+0000 7ff029e7d700 1 --2- 172.21.15.16:0/3615814226 >> v2:172.21.15.76:3300/0 conn(0x7ff02408c6d0 0x7ff02408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:29:15.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.089+0000 7ff02a67e700 1 --2- 172.21.15.16:0/3615814226 >> v2:172.21.15.16:3300/0 conn(0x7ff02408ef20 0x7ff02408f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff01400f730 tx=0x7ff01400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.089+0000 7ff02a67e700 1 --2- 172.21.15.16:0/3615814226 >> v2:172.21.15.16:3300/0 conn(0x7ff02408ef20 0x7ff02408f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff01400f730 tx=0x7ff01400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.089+0000 7ff02a67e700 1 --2- 172.21.15.16:0/3615814226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02408ef20 0x7ff02408f3c0 secure :-1 s=READY pgs=864 cs=0 l=1 rx=0x7ff01400f730 tx=0x7ff01400fad0).ready entity=mon.0 client_cookie=bc403ef1cacad63e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:15.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.090+0000 7ff02a67e700 1 --2- 172.21.15.16:0/3615814226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02408ef20 0x7ff02408f3c0 secure :-1 s=THROTTLE_DONE pgs=864 cs=0 l=1 rx=0x7ff01400f730 tx=0x7ff01400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.090+0000 7ff02a67e700 1 --2- 172.21.15.16:0/3615814226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02408ef20 0x7ff02408f3c0 secure :-1 s=THROTTLE_DONE pgs=864 cs=0 l=1 rx=0x7ff01400f730 tx=0x7ff01400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.090+0000 7ff028e7b700 1 -- 172.21.15.16:0/3615814226 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0140010e0 con 0x7ff02408ef20 2019-12-05T06:29:15.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.090+0000 7ff02a67e700 1 --2- 172.21.15.16:0/3615814226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02408ef20 0x7ff02408f3c0 secure :-1 s=THROTTLE_DONE pgs=864 cs=0 l=1 rx=0x7ff01400f730 tx=0x7ff01400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.090+0000 7ff028e7b700 1 -- 172.21.15.16:0/3615814226 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff014000b90 con 0x7ff02408ef20 2019-12-05T06:29:15.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.090+0000 7ff028e7b700 1 -- 172.21.15.16:0/3615814226 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0140016c0 con 0x7ff02408ef20 2019-12-05T06:29:15.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.090+0000 7ff02b8e0700 1 -- 172.21.15.16:0/3615814226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02408ef20 msgr2=0x7ff02408f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:15.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.090+0000 7ff02b8e0700 1 --2- 172.21.15.16:0/3615814226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02408ef20 0x7ff02408f3c0 secure :-1 s=READY pgs=864 cs=0 l=1 rx=0x7ff01400f730 tx=0x7ff01400fad0).stop 2019-12-05T06:29:15.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.092+0000 7ff02b8e0700 1 -- 172.21.15.16:0/3615814226 shutdown_connections 2019-12-05T06:29:15.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.092+0000 7ff02b8e0700 1 --2- 172.21.15.16:0/3615814226 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff02408ef20 0x7ff02408f3c0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:15.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.092+0000 7ff02b8e0700 1 --2- 172.21.15.16:0/3615814226 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff02408fef0 0x7ff02408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:15.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.092+0000 7ff02b8e0700 1 --2- 172.21.15.16:0/3615814226 >> v2:172.21.15.76:3300/0 conn(0x7ff02408c6d0 0x7ff02408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:15.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.092+0000 7ff02b8e0700 1 -- 172.21.15.16:0/3615814226 shutdown_connections 2019-12-05T06:29:15.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.092+0000 7ff02b8e0700 1 -- 172.21.15.16:0/3615814226 wait complete. 2019-12-05T06:29:15.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.092+0000 7ff02b8e0700 1 -- 172.21.15.16:0/3615814226 >> 172.21.15.16:0/3615814226 conn(0x7ff024061940 msgr2=0x7ff024096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.097+0000 7ff02b8e0700 1 Processor -- start 2019-12-05T06:29:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.097+0000 7ff02b8e0700 1 -- start start 2019-12-05T06:29:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.099+0000 7ff02b8e0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff024132630 0x7ff024132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.099+0000 7ff02b8e0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02409ab10 con 0x7ff02409ae80 2019-12-05T06:29:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.099+0000 7ff02b8e0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff02406e7c0 con 0x7ff024099eb0 2019-12-05T06:29:15.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.099+0000 7ff02b8e0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff024098f20 con 0x7ff024132630 2019-12-05T06:29:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.099+0000 7ff02967c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff02409ae80 0x7ff024132f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39562/0 (socket says 172.21.15.16:39562) 2019-12-05T06:29:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.099+0000 7ff02967c700 1 -- 172.21.15.16:0/2077258191 learned_addr learned my addr 172.21.15.16:0/2077258191 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.099+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> v2:172.21.15.76:3300/0 conn(0x7ff024132630 0x7ff024132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.100+0000 7ff0227fc700 1 -- 172.21.15.16:0/2077258191 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899793353 0 0) 0x7ff01800dd90 con 0x7ff02409ae80 2019-12-05T06:29:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.100+0000 7ff0227fc700 1 -- 172.21.15.16:0/2077258191 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff004002e80 con 0x7ff02409ae80 2019-12-05T06:29:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.100+0000 7ff029e7d700 1 -- 172.21.15.16:0/2077258191 >> v1:172.21.15.76:6789/0 conn(0x7ff024099eb0 legacy=0x7ff02409a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:15.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.100+0000 7ff029e7d700 1 -- 172.21.15.16:0/2077258191 >> v1:172.21.15.16:6789/0 conn(0x7ff02409ae80 legacy=0x7ff024132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:15.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.100+0000 7ff0227fc700 1 -- 172.21.15.16:0/2077258191 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334500715 0 0) 0x7ff014007e90 con 0x7ff024099eb0 2019-12-05T06:29:15.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.100+0000 7ff029e7d700 1 -- 172.21.15.16:0/2077258191 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0240995b0 con 0x7ff024132630 2019-12-05T06:29:15.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.100+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> v2:172.21.15.76:3300/0 conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.100+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> v2:172.21.15.76:3300/0 conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.100+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=READY pgs=622 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:15.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.101+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=THROTTLE_DONE pgs=622 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.101+0000 7ff0227fc700 1 -- 172.21.15.16:0/2077258191 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff01c000cc0 con 0x7ff024132630 2019-12-05T06:29:15.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.101+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=THROTTLE_DONE pgs=622 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.101+0000 7ff0227fc700 1 -- 172.21.15.16:0/2077258191 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01c00e370 con 0x7ff024132630 2019-12-05T06:29:15.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.101+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=THROTTLE_DONE pgs=622 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.101+0000 7ff02b8e0700 1 -- 172.21.15.16:0/2077258191 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0240995b0 con 0x7ff024132630 2019-12-05T06:29:15.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.101+0000 7ff0227fc700 1 -- 172.21.15.16:0/2077258191 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff01c00e560 con 0x7ff024132630 2019-12-05T06:29:15.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.101+0000 7ff02b8e0700 1 -- 172.21.15.16:0/2077258191 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0240995b0 con 0x7ff024132630 2019-12-05T06:29:15.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.102+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=THROTTLE_DONE pgs=622 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.103+0000 7ff0227fc700 1 -- 172.21.15.16:0/2077258191 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff01c00e560 con 0x7ff024132630 2019-12-05T06:29:15.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.103+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=THROTTLE_DONE pgs=622 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.103+0000 7ff0227fc700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff004023680 0x7ff004025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:15.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.103+0000 7ff0227fc700 1 -- 172.21.15.16:0/2077258191 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7ff01c046190 con 0x7ff024132630 2019-12-05T06:29:15.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.104+0000 7ff02967c700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff004023680 0x7ff004025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:15.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.104+0000 7ff02967c700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff004023680 0x7ff004025af0 crc :-1 s=READY pgs=374 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:15.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.117+0000 7ff02b8e0700 1 -- 172.21.15.16:0/2077258191 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0240995b0 con 0x7ff024132630 2019-12-05T06:29:15.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.126+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=THROTTLE_DONE pgs=622 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.127+0000 7ff0227fc700 1 -- 172.21.15.16:0/2077258191 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff01c00ea70 con 0x7ff024132630 2019-12-05T06:29:15.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.380+0000 7ff02b8e0700 1 -- 172.21.15.16:0/2077258191 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0240995b0 con 0x7ff024132630 2019-12-05T06:29:15.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.381+0000 7ff029e7d700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=THROTTLE_DONE pgs=622 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:15.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.381+0000 7ff0227fc700 1 -- 172.21.15.16:0/2077258191 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2112 (secure 0 0 0) 0x7ff01c00e560 con 0x7ff024132630 2019-12-05T06:29:15.374 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:15.375 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6363,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:15.375 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 107 2019-12-05T06:29:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.385+0000 7ff02b8e0700 1 -- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff004023680 msgr2=0x7ff004025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.385+0000 7ff02b8e0700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff004023680 0x7ff004025af0 crc :-1 s=READY pgs=374 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.385+0000 7ff02b8e0700 1 -- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 msgr2=0x7ff024132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.385+0000 7ff02b8e0700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 0x7ff024132a20 secure :-1 s=READY pgs=622 cs=0 l=1 rx=0x7ff01c010720 tx=0x7ff01c010ac0).stop 2019-12-05T06:29:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.385+0000 7ff02b8e0700 1 -- 172.21.15.16:0/2077258191 shutdown_connections 2019-12-05T06:29:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.385+0000 7ff02b8e0700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff004023680 0x7ff004025af0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.385+0000 7ff02b8e0700 1 --2- 172.21.15.16:0/2077258191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff024132630 0x7ff024132a20 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.385+0000 7ff02b8e0700 1 -- 172.21.15.16:0/2077258191 shutdown_connections 2019-12-05T06:29:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.385+0000 7ff02b8e0700 1 -- 172.21.15.16:0/2077258191 wait complete. 2019-12-05T06:29:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:15.386+0000 7ff02b8e0700 1 -- 172.21.15.16:0/2077258191 >> 172.21.15.16:0/2077258191 conn(0x7ff0240714f0 msgr2=0x7ff02409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:15.388 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:29:17.389 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.676+0000 7f0d43503700 1 Processor -- start 2019-12-05T06:29:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.677+0000 7f0d43503700 1 -- start start 2019-12-05T06:29:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.678+0000 7f0d43503700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0d3c08fef0 0x7f0d3c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.678+0000 7f0d43503700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c0958e0 con 0x7f0d3c08c6d0 2019-12-05T06:29:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.679+0000 7f0d43503700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c0629e0 con 0x7f0d3c08d2c0 2019-12-05T06:29:17.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.679+0000 7f0d43503700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3c08df90 con 0x7f0d3c08fef0 2019-12-05T06:29:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.679+0000 7f0d41aa0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0d3c08c6d0 0x7f0d3c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39570/0 (socket says 172.21.15.16:39570) 2019-12-05T06:29:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.679+0000 7f0d41aa0700 1 -- 172.21.15.16:0/3930163726 learned_addr learned my addr 172.21.15.16:0/3930163726 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.679+0000 7f0d422a1700 1 --2- 172.21.15.16:0/3930163726 >> v2:172.21.15.76:3300/0 conn(0x7f0d3c08fef0 0x7f0d3c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.679+0000 7f0d40a9e700 1 -- 172.21.15.16:0/3930163726 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2977108843 0 0) 0x7f0d34001150 con 0x7f0d3c08c6d0 2019-12-05T06:29:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.680+0000 7f0d40a9e700 1 -- 172.21.15.16:0/3930163726 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d24002e80 con 0x7f0d3c08c6d0 2019-12-05T06:29:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.680+0000 7f0d422a1700 1 -- 172.21.15.16:0/3930163726 >> v1:172.21.15.76:6789/0 conn(0x7f0d3c08d2c0 legacy=0x7f0d3c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:17.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.680+0000 7f0d422a1700 1 -- 172.21.15.16:0/3930163726 >> v1:172.21.15.16:6789/0 conn(0x7f0d3c08c6d0 legacy=0x7f0d3c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.680+0000 7f0d40a9e700 1 -- 172.21.15.16:0/3930163726 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761914944 0 0) 0x7f0d30001400 con 0x7f0d3c08d2c0 2019-12-05T06:29:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.680+0000 7f0d422a1700 1 -- 172.21.15.16:0/3930163726 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c08e620 con 0x7f0d3c08fef0 2019-12-05T06:29:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.680+0000 7f0d422a1700 1 --2- 172.21.15.16:0/3930163726 >> v2:172.21.15.76:3300/0 conn(0x7f0d3c08fef0 0x7f0d3c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0d2c00af80 tx=0x7f0d2c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.680+0000 7f0d422a1700 1 --2- 172.21.15.16:0/3930163726 >> v2:172.21.15.76:3300/0 conn(0x7f0d3c08fef0 0x7f0d3c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0d2c00af80 tx=0x7f0d2c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.680+0000 7f0d422a1700 1 --2- 172.21.15.16:0/3930163726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d3c08fef0 0x7f0d3c08f720 secure :-1 s=READY pgs=624 cs=0 l=1 rx=0x7f0d2c00af80 tx=0x7f0d2c00b790).ready entity=mon.1 client_cookie=cc5c6f9b5fd83790 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:17.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.681+0000 7f0d422a1700 1 --2- 172.21.15.16:0/3930163726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d3c08fef0 0x7f0d3c08f720 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f0d2c00af80 tx=0x7f0d2c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.681+0000 7f0d40a9e700 1 -- 172.21.15.16:0/3930163726 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d2c007240 con 0x7f0d3c08fef0 2019-12-05T06:29:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.681+0000 7f0d422a1700 1 --2- 172.21.15.16:0/3930163726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d3c08fef0 0x7f0d3c08f720 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f0d2c00af80 tx=0x7f0d2c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.681+0000 7f0d422a1700 1 --2- 172.21.15.16:0/3930163726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d3c08fef0 0x7f0d3c08f720 secure :-1 s=THROTTLE_DONE pgs=624 cs=0 l=1 rx=0x7f0d2c00af80 tx=0x7f0d2c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.681+0000 7f0d40a9e700 1 -- 172.21.15.16:0/3930163726 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2c000bb0 con 0x7f0d3c08fef0 2019-12-05T06:29:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.681+0000 7f0d40a9e700 1 -- 172.21.15.16:0/3930163726 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d2c009e60 con 0x7f0d3c08fef0 2019-12-05T06:29:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.681+0000 7f0d43503700 1 -- 172.21.15.16:0/3930163726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d3c08fef0 msgr2=0x7f0d3c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:17.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.681+0000 7f0d43503700 1 --2- 172.21.15.16:0/3930163726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d3c08fef0 0x7f0d3c08f720 secure :-1 s=READY pgs=624 cs=0 l=1 rx=0x7f0d2c00af80 tx=0x7f0d2c00b790).stop 2019-12-05T06:29:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.682+0000 7f0d43503700 1 -- 172.21.15.16:0/3930163726 shutdown_connections 2019-12-05T06:29:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.682+0000 7f0d43503700 1 --2- 172.21.15.16:0/3930163726 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d3c08fef0 0x7f0d3c08f720 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.683+0000 7f0d43503700 1 -- 172.21.15.16:0/3930163726 shutdown_connections 2019-12-05T06:29:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.683+0000 7f0d43503700 1 -- 172.21.15.16:0/3930163726 wait complete. 2019-12-05T06:29:17.682 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.683+0000 7f0d43503700 1 -- 172.21.15.16:0/3930163726 >> 172.21.15.16:0/3930163726 conn(0x7f0d3c061940 msgr2=0x7f0d3c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:17.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.688+0000 7f0d43503700 1 Processor -- start 2019-12-05T06:29:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.688+0000 7f0d43503700 1 -- start start 2019-12-05T06:29:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.690+0000 7f0d43503700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0d3c1302f0 0x7f0d3c12a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.690+0000 7f0d43503700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0d3c12b640 0x7f0d3c12abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.690+0000 7f0d43503700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d3c131bf0 con 0x7f0d3c12b0f0 2019-12-05T06:29:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.690+0000 7f0d43503700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3c1296e0 con 0x7f0d3c1302f0 2019-12-05T06:29:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.690+0000 7f0d43503700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d3c128cb0 con 0x7f0d3c12b640 2019-12-05T06:29:17.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.690+0000 7f0d41aa0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0d3c1302f0 0x7f0d3c12a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.690+0000 7f0d41aa0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0d3c1302f0 0x7f0d3c12a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54408/0 (socket says 172.21.15.16:54408) 2019-12-05T06:29:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.690+0000 7f0d41aa0700 1 -- 172.21.15.16:0/325996069 learned_addr learned my addr 172.21.15.16:0/325996069 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:29:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.690+0000 7f0d4129f700 1 --2- 172.21.15.16:0/325996069 >> v2:172.21.15.76:3300/0 conn(0x7f0d3c12b640 0x7f0d3c12abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.691+0000 7f0d41aa0700 1 -- 172.21.15.16:0/325996069 >> v2:172.21.15.76:3300/0 conn(0x7f0d3c12b640 msgr2=0x7f0d3c12abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.691+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> v2:172.21.15.76:3300/0 conn(0x7f0d3c12b640 0x7f0d3c12abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.691+0000 7f0d41aa0700 1 -- 172.21.15.16:0/325996069 >> v1:172.21.15.16:6789/0 conn(0x7f0d3c12b0f0 legacy=0x7f0d3c1306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:17.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.691+0000 7f0d41aa0700 1 -- 172.21.15.16:0/325996069 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d3c129d70 con 0x7f0d3c1302f0 2019-12-05T06:29:17.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.691+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> v2:172.21.15.16:3300/0 conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.691+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> v2:172.21.15.16:3300/0 conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.691+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=READY pgs=868 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:17.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.691+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.691+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.691+0000 7f0d3a7fc700 1 -- 172.21.15.16:0/325996069 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d34000c90 con 0x7f0d3c1302f0 2019-12-05T06:29:17.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.692+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.692+0000 7f0d3a7fc700 1 -- 172.21.15.16:0/325996069 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d340010c0 con 0x7f0d3c1302f0 2019-12-05T06:29:17.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.692+0000 7f0d3a7fc700 1 -- 172.21.15.16:0/325996069 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d34001710 con 0x7f0d3c1302f0 2019-12-05T06:29:17.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.692+0000 7f0d43503700 1 -- 172.21.15.16:0/325996069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d3c129d70 con 0x7f0d3c1302f0 2019-12-05T06:29:17.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.692+0000 7f0d43503700 1 -- 172.21.15.16:0/325996069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d3c129d70 con 0x7f0d3c1302f0 2019-12-05T06:29:17.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.693+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.693+0000 7f0d3a7fc700 1 -- 172.21.15.16:0/325996069 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0d34000c90 con 0x7f0d3c1302f0 2019-12-05T06:29:17.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.694+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.694+0000 7f0d3a7fc700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d1c023630 0x7f0d1c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:17.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.694+0000 7f0d3a7fc700 1 -- 172.21.15.16:0/325996069 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 7321+0+0 (secure 0 0 0) 0x7f0d34047fd0 con 0x7f0d3c1302f0 2019-12-05T06:29:17.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.694+0000 7f0d4129f700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d1c023630 0x7f0d1c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:17.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.695+0000 7f0d4129f700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d1c023630 0x7f0d1c025ac0 crc :-1 s=READY pgs=375 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:17.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.708+0000 7f0d43503700 1 -- 172.21.15.16:0/325996069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d3c129d70 con 0x7f0d3c1302f0 2019-12-05T06:29:17.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.715+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.715+0000 7f0d3a7fc700 1 -- 172.21.15.16:0/325996069 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0d3402a0c0 con 0x7f0d3c1302f0 2019-12-05T06:29:17.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.968+0000 7f0d43503700 1 -- 172.21.15.16:0/325996069 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d3c129d70 con 0x7f0d3c1302f0 2019-12-05T06:29:17.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.968+0000 7f0d41aa0700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=THROTTLE_DONE pgs=868 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:17.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.968+0000 7f0d3a7fc700 1 -- 172.21.15.16:0/325996069 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 107 v107) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0d3402a0c0 con 0x7f0d3c1302f0 2019-12-05T06:29:17.960 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 107 2019-12-05T06:29:17.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:17.961 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":107,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6363,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":107}],"filesystems":[{"mdsmap":{"epoch":106,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:28:18.186529+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":64,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6264},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6264":{"gid":6264,"name":"b","rank":0,"incarnation":103,"state":"up:active","state_seq":13,"addr":"172.21.15.76:6835/3155972571","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3155972571},{"type":"v1","addr":"172.21.15.76:6835","nonce":3155972571}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:17.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.971+0000 7f0d43503700 1 -- 172.21.15.16:0/325996069 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d1c023630 msgr2=0x7f0d1c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:17.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.971+0000 7f0d43503700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d1c023630 0x7f0d1c025ac0 crc :-1 s=READY pgs=375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:17.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.971+0000 7f0d43503700 1 -- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 msgr2=0x7f0d3c12a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:17.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.971+0000 7f0d43503700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 0x7f0d3c12a670 secure :-1 s=READY pgs=868 cs=0 l=1 rx=0x7f0d34010260 tx=0x7f0d340128d0).stop 2019-12-05T06:29:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.971+0000 7f0d43503700 1 -- 172.21.15.16:0/325996069 shutdown_connections 2019-12-05T06:29:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.971+0000 7f0d43503700 1 --2- 172.21.15.16:0/325996069 >> v2:172.21.15.76:3300/0 conn(0x7f0d3c12b640 0x7f0d3c12abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.971+0000 7f0d43503700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d1c023630 0x7f0d1c025ac0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.971+0000 7f0d43503700 1 --2- 172.21.15.16:0/325996069 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0d3c1302f0 0x7f0d3c12a670 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.971+0000 7f0d43503700 1 -- 172.21.15.16:0/325996069 shutdown_connections 2019-12-05T06:29:17.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.971+0000 7f0d43503700 1 -- 172.21.15.16:0/325996069 wait complete. 2019-12-05T06:29:17.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:17.972+0000 7f0d43503700 1 -- 172.21.15.16:0/325996069 >> 172.21.15.16:0/325996069 conn(0x7f0d3c0a2850 msgr2=0x7f0d3c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:17.973 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:29:19.975 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.156+0000 7f8ce8544700 1 Processor -- start 2019-12-05T06:29:20.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.156+0000 7f8ce8544700 1 -- start start 2019-12-05T06:29:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.157+0000 7f8ce8544700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ce008c6d0 0x7f8ce008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.157+0000 7f8ce8544700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8ce008fef0 0x7f8ce008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.157+0000 7f8ce8544700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce008fb80 con 0x7f8ce008ef20 2019-12-05T06:29:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.157+0000 7f8ce8544700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce008df90 con 0x7f8ce008c6d0 2019-12-05T06:29:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.157+0000 7f8ce8544700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce00a3f70 con 0x7f8ce008fef0 2019-12-05T06:29:20.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.157+0000 7f8ce6ae1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ce008c6d0 0x7f8ce008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.157+0000 7f8ce6ae1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ce008c6d0 0x7f8ce008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54428/0 (socket says 172.21.15.16:54428) 2019-12-05T06:29:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 -- 172.21.15.16:0/504488895 learned_addr learned my addr 172.21.15.16:0/504488895 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:29:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce62e0700 1 --2- 172.21.15.16:0/504488895 >> v2:172.21.15.76:3300/0 conn(0x7f8ce008fef0 0x7f8ce008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 -- 172.21.15.16:0/504488895 >> v2:172.21.15.76:3300/0 conn(0x7f8ce008fef0 msgr2=0x7f8ce008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 --2- 172.21.15.16:0/504488895 >> v2:172.21.15.76:3300/0 conn(0x7f8ce008fef0 0x7f8ce008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:20.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 -- 172.21.15.16:0/504488895 >> v1:172.21.15.76:6789/0 conn(0x7f8ce008ef20 legacy=0x7f8ce008f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 -- 172.21.15.16:0/504488895 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce008d540 con 0x7f8ce008c6d0 2019-12-05T06:29:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 --2- 172.21.15.16:0/504488895 >> v2:172.21.15.16:3300/0 conn(0x7f8ce008c6d0 0x7f8ce008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8cdc00f6c0 tx=0x7f8cdc00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 --2- 172.21.15.16:0/504488895 >> v2:172.21.15.16:3300/0 conn(0x7f8ce008c6d0 0x7f8ce008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8cdc00f6c0 tx=0x7f8cdc00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 --2- 172.21.15.16:0/504488895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce008c6d0 0x7f8ce008cac0 secure :-1 s=READY pgs=869 cs=0 l=1 rx=0x7f8cdc00f6c0 tx=0x7f8cdc00fa60).ready entity=mon.0 client_cookie=2f50151c27f421ec server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 --2- 172.21.15.16:0/504488895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce008c6d0 0x7f8ce008cac0 secure :-1 s=THROTTLE_DONE pgs=869 cs=0 l=1 rx=0x7f8cdc00f6c0 tx=0x7f8cdc00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 --2- 172.21.15.16:0/504488895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce008c6d0 0x7f8ce008cac0 secure :-1 s=THROTTLE_DONE pgs=869 cs=0 l=1 rx=0x7f8cdc00f6c0 tx=0x7f8cdc00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce6ae1700 1 --2- 172.21.15.16:0/504488895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce008c6d0 0x7f8ce008cac0 secure :-1 s=THROTTLE_DONE pgs=869 cs=0 l=1 rx=0x7f8cdc00f6c0 tx=0x7f8cdc00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce5adf700 1 -- 172.21.15.16:0/504488895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cdc000960 con 0x7f8ce008c6d0 2019-12-05T06:29:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce5adf700 1 -- 172.21.15.16:0/504488895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cdc000c90 con 0x7f8ce008c6d0 2019-12-05T06:29:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce5adf700 1 -- 172.21.15.16:0/504488895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cdc0013a0 con 0x7f8ce008c6d0 2019-12-05T06:29:20.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce8544700 1 -- 172.21.15.16:0/504488895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce008c6d0 msgr2=0x7f8ce008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.158+0000 7f8ce8544700 1 --2- 172.21.15.16:0/504488895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce008c6d0 0x7f8ce008cac0 secure :-1 s=READY pgs=869 cs=0 l=1 rx=0x7f8cdc00f6c0 tx=0x7f8cdc00fa60).stop 2019-12-05T06:29:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.159+0000 7f8ce8544700 1 -- 172.21.15.16:0/504488895 shutdown_connections 2019-12-05T06:29:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.159+0000 7f8ce8544700 1 --2- 172.21.15.16:0/504488895 >> v2:172.21.15.76:3300/0 conn(0x7f8ce008fef0 0x7f8ce008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:20.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.159+0000 7f8ce8544700 1 --2- 172.21.15.16:0/504488895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce008c6d0 0x7f8ce008cac0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.159+0000 7f8ce8544700 1 -- 172.21.15.16:0/504488895 shutdown_connections 2019-12-05T06:29:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.159+0000 7f8ce8544700 1 -- 172.21.15.16:0/504488895 wait complete. 2019-12-05T06:29:20.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.159+0000 7f8ce8544700 1 -- 172.21.15.16:0/504488895 >> 172.21.15.16:0/504488895 conn(0x7f8ce0061940 msgr2=0x7f8ce0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:20.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.161+0000 7f8ce8544700 1 Processor -- start 2019-12-05T06:29:20.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.161+0000 7f8ce8544700 1 -- start start 2019-12-05T06:29:20.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.162+0000 7f8ce8544700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8ce009bbe0 0x7f8ce01539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:20.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.162+0000 7f8ce8544700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:20.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.162+0000 7f8ce8544700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ce0084b90 con 0x7f8ce0096e90 2019-12-05T06:29:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.162+0000 7f8ce8544700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ce009a450 con 0x7f8ce009bbe0 2019-12-05T06:29:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.162+0000 7f8ce8544700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce0099a20 con 0x7f8ce0084f30 2019-12-05T06:29:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.162+0000 7f8ce62e0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8ce009bbe0 0x7f8ce01539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.162+0000 7f8ce72e2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.162+0000 7f8ce62e0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8ce009bbe0 0x7f8ce01539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:40526/0 (socket says 172.21.15.16:40526) 2019-12-05T06:29:20.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.162+0000 7f8ce6ae1700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8ce0096e90 0x7f8ce009b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:60904/0 (socket says 172.21.15.16:60904) 2019-12-05T06:29:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.162+0000 7f8ce62e0700 1 -- 172.21.15.16:0/1634351103 learned_addr learned my addr 172.21.15.16:0/1634351103 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:29:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.163+0000 7f8ce72e2700 1 -- 172.21.15.16:0/1634351103 >> v2:172.21.15.76:3300/0 conn(0x7f8ce009bbe0 msgr2=0x7f8ce01539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.163+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> v2:172.21.15.76:3300/0 conn(0x7f8ce009bbe0 0x7f8ce01539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.163+0000 7f8ce72e2700 1 -- 172.21.15.16:0/1634351103 >> v1:172.21.15.76:6789/0 conn(0x7f8ce0096e90 legacy=0x7f8ce009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:20.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.163+0000 7f8ce72e2700 1 -- 172.21.15.16:0/1634351103 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce009aae0 con 0x7f8ce0084f30 2019-12-05T06:29:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.163+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.163+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.163+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=READY pgs=207 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.164+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=THROTTLE_DONE pgs=207 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.164+0000 7f8cd77fe700 1 -- 172.21.15.16:0/1634351103 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cd8009f40 con 0x7f8ce0084f30 2019-12-05T06:29:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.164+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=THROTTLE_DONE pgs=207 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.164+0000 7f8ce8544700 1 -- 172.21.15.16:0/1634351103 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce009aae0 con 0x7f8ce0084f30 2019-12-05T06:29:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.164+0000 7f8cd77fe700 1 -- 172.21.15.16:0/1634351103 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd8009b60 con 0x7f8ce0084f30 2019-12-05T06:29:20.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.164+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=THROTTLE_DONE pgs=207 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.164+0000 7f8cd77fe700 1 -- 172.21.15.16:0/1634351103 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cd8009d50 con 0x7f8ce0084f30 2019-12-05T06:29:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.164+0000 7f8ce8544700 1 -- 172.21.15.16:0/1634351103 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce009aae0 con 0x7f8ce0084f30 2019-12-05T06:29:20.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.165+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=THROTTLE_DONE pgs=207 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.165+0000 7f8cd77fe700 1 -- 172.21.15.16:0/1634351103 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8cd8009c00 con 0x7f8ce0084f30 2019-12-05T06:29:20.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.165+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=THROTTLE_DONE pgs=207 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.165+0000 7f8cd77fe700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8cc8023630 0x7f8cc8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:20.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.165+0000 7f8cd77fe700 1 -- 172.21.15.16:0/1634351103 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f8cd8045ef0 con 0x7f8ce0084f30 2019-12-05T06:29:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.165+0000 7f8ce6ae1700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8cc8023630 0x7f8cc8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.166+0000 7f8ce6ae1700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8cc8023630 0x7f8cc8025ac0 crc :-1 s=READY pgs=376 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:20.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.171+0000 7f8ce8544700 1 -- 172.21.15.16:0/1634351103 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce009aae0 con 0x7f8ce0084f30 2019-12-05T06:29:20.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.180+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=THROTTLE_DONE pgs=207 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.180+0000 7f8cd77fe700 1 -- 172.21.15.16:0/1634351103 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8cd8009950 con 0x7f8ce0084f30 2019-12-05T06:29:20.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.376+0000 7f8ce8544700 1 -- 172.21.15.16:0/1634351103 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ce009aae0 con 0x7f8ce0084f30 2019-12-05T06:29:20.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.377+0000 7f8ce72e2700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=THROTTLE_DONE pgs=207 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.377+0000 7f8cd77fe700 1 -- 172.21.15.16:0/1634351103 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 108 v108) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f8cd8009b60 con 0x7f8ce0084f30 2019-12-05T06:29:20.370 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 108 2019-12-05T06:29:20.371 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:20.371 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":108,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":108,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:18.226785+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:20.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 -- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8cc8023630 msgr2=0x7f8cc8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:20.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8cc8023630 0x7f8cc8025ac0 crc :-1 s=READY pgs=376 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 -- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 msgr2=0x7f8ce0155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:20.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 secure :-1 s=READY pgs=207 cs=0 l=1 rx=0x7f8cd8010720 tx=0x7f8cd8010550).stop 2019-12-05T06:29:20.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 -- 172.21.15.16:0/1634351103 shutdown_connections 2019-12-05T06:29:20.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8cc8023630 0x7f8cc8025ac0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 --2- 172.21.15.16:0/1634351103 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce0084f30 0x7f8ce0155f30 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 --2- 172.21.15.16:0/1634351103 >> v2:172.21.15.76:3300/0 conn(0x7f8ce009bbe0 0x7f8ce01539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 -- 172.21.15.16:0/1634351103 shutdown_connections 2019-12-05T06:29:20.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 -- 172.21.15.16:0/1634351103 wait complete. 2019-12-05T06:29:20.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.382+0000 7f8ce8544700 1 -- 172.21.15.16:0/1634351103 >> 172.21.15.16:0/1634351103 conn(0x7f8ce00714f0 msgr2=0x7f8ce009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:20.385 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:29:20.385 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:29:20.385 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.547+0000 7f205aa4d700 1 Processor -- start 2019-12-05T06:29:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.547+0000 7f205aa4d700 1 -- start start 2019-12-05T06:29:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.547+0000 7f205aa4d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f205408c6d0 0x7f205408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.547+0000 7f205aa4d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f205408fef0 0x7f205408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.547+0000 7f205aa4d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f205408ef20 0x7f205408f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.547+0000 7f205aa4d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205408df90 con 0x7f205408c6d0 2019-12-05T06:29:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.547+0000 7f205aa4d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20540a3f70 con 0x7f205408fef0 2019-12-05T06:29:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.547+0000 7f205aa4d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205408d830 con 0x7f205408ef20 2019-12-05T06:29:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f2058fea700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f205408c6d0 0x7f205408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f2058fea700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f205408c6d0 0x7f205408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54444/0 (socket says 172.21.15.16:54444) 2019-12-05T06:29:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f2058fea700 1 -- 172.21.15.16:0/1063454970 learned_addr learned my addr 172.21.15.16:0/1063454970 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:29:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f2053fff700 1 --2- 172.21.15.16:0/1063454970 >> v2:172.21.15.76:3300/0 conn(0x7f205408fef0 0x7f205408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f20597eb700 1 --2- 172.21.15.16:0/1063454970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f205408ef20 0x7f205408f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f2058fea700 1 -- 172.21.15.16:0/1063454970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f205408ef20 msgr2=0x7f205408f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f2058fea700 1 --2- 172.21.15.16:0/1063454970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f205408ef20 0x7f205408f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f2058fea700 1 -- 172.21.15.16:0/1063454970 >> v2:172.21.15.76:3300/0 conn(0x7f205408fef0 msgr2=0x7f205408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f2058fea700 1 --2- 172.21.15.16:0/1063454970 >> v2:172.21.15.76:3300/0 conn(0x7f205408fef0 0x7f205408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f2058fea700 1 -- 172.21.15.16:0/1063454970 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205408d540 con 0x7f205408c6d0 2019-12-05T06:29:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.548+0000 7f2058fea700 1 --2- 172.21.15.16:0/1063454970 >> v2:172.21.15.16:3300/0 conn(0x7f205408c6d0 0x7f205408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f204c00fac0 tx=0x7f204c00f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.549+0000 7f2058fea700 1 --2- 172.21.15.16:0/1063454970 >> v2:172.21.15.16:3300/0 conn(0x7f205408c6d0 0x7f205408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f204c00fac0 tx=0x7f204c00f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.549+0000 7f2058fea700 1 --2- 172.21.15.16:0/1063454970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205408c6d0 0x7f205408cac0 secure :-1 s=READY pgs=870 cs=0 l=1 rx=0x7f204c00fac0 tx=0x7f204c00f970).ready entity=mon.0 client_cookie=77b52171ca7ac3fd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.549+0000 7f2058fea700 1 --2- 172.21.15.16:0/1063454970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205408c6d0 0x7f205408cac0 secure :-1 s=THROTTLE_DONE pgs=870 cs=0 l=1 rx=0x7f204c00fac0 tx=0x7f204c00f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.549+0000 7f2058fea700 1 --2- 172.21.15.16:0/1063454970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205408c6d0 0x7f205408cac0 secure :-1 s=THROTTLE_DONE pgs=870 cs=0 l=1 rx=0x7f204c00fac0 tx=0x7f204c00f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.549+0000 7f20537fe700 1 -- 172.21.15.16:0/1063454970 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f204c000ce0 con 0x7f205408c6d0 2019-12-05T06:29:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.549+0000 7f2058fea700 1 --2- 172.21.15.16:0/1063454970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205408c6d0 0x7f205408cac0 secure :-1 s=THROTTLE_DONE pgs=870 cs=0 l=1 rx=0x7f204c00fac0 tx=0x7f204c00f970).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.549+0000 7f20537fe700 1 -- 172.21.15.16:0/1063454970 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f204c001010 con 0x7f205408c6d0 2019-12-05T06:29:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.549+0000 7f20537fe700 1 -- 172.21.15.16:0/1063454970 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f204c001720 con 0x7f205408c6d0 2019-12-05T06:29:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.549+0000 7f205aa4d700 1 -- 172.21.15.16:0/1063454970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205408c6d0 msgr2=0x7f205408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.549+0000 7f205aa4d700 1 --2- 172.21.15.16:0/1063454970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205408c6d0 0x7f205408cac0 secure :-1 s=READY pgs=870 cs=0 l=1 rx=0x7f204c00fac0 tx=0x7f204c00f970).stop 2019-12-05T06:29:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.550+0000 7f205aa4d700 1 -- 172.21.15.16:0/1063454970 shutdown_connections 2019-12-05T06:29:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.550+0000 7f205aa4d700 1 --2- 172.21.15.16:0/1063454970 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f205408ef20 0x7f205408f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.550+0000 7f205aa4d700 1 --2- 172.21.15.16:0/1063454970 >> v2:172.21.15.76:3300/0 conn(0x7f205408fef0 0x7f205408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.550+0000 7f205aa4d700 1 --2- 172.21.15.16:0/1063454970 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f205408c6d0 0x7f205408cac0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.550+0000 7f205aa4d700 1 -- 172.21.15.16:0/1063454970 shutdown_connections 2019-12-05T06:29:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.550+0000 7f205aa4d700 1 -- 172.21.15.16:0/1063454970 wait complete. 2019-12-05T06:29:20.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.550+0000 7f205aa4d700 1 -- 172.21.15.16:0/1063454970 >> 172.21.15.16:0/1063454970 conn(0x7f2054061940 msgr2=0x7f2054096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:20.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.553+0000 7f205aa4d700 1 Processor -- start 2019-12-05T06:29:20.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.553+0000 7f205aa4d700 1 -- start start 2019-12-05T06:29:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f205aa4d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2054132d00 0x7f2054132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f205aa4d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2054154b00 0x7f2054156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:20.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f205aa4d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2054098da0 con 0x7f205409b850 2019-12-05T06:29:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f205aa4d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f205409aa60 con 0x7f2054132d00 2019-12-05T06:29:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f205aa4d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f205409a030 con 0x7f2054154b00 2019-12-05T06:29:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f2053fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2054132d00 0x7f2054132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f2053fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2054132d00 0x7f2054132530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54450/0 (socket says 172.21.15.16:54450) 2019-12-05T06:29:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f2053fff700 1 -- 172.21.15.16:0/1187917552 learned_addr learned my addr 172.21.15.16:0/1187917552 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:29:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f20597eb700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2054154b00 0x7f2054156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f2053fff700 1 -- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2054154b00 msgr2=0x7f2054156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.554+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2054154b00 0x7f2054156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f2053fff700 1 -- 172.21.15.16:0/1187917552 >> v1:172.21.15.76:6789/0 conn(0x7f205409b850 legacy=0x7f2054131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f2053fff700 1 -- 172.21.15.16:0/1187917552 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f205409b0f0 con 0x7f2054132d00 2019-12-05T06:29:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f20597eb700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2054154b00 0x7f2054156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:29:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> v2:172.21.15.16:3300/0 conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> v2:172.21.15.16:3300/0 conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=READY pgs=871 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:20.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=THROTTLE_DONE pgs=871 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=THROTTLE_DONE pgs=871 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=THROTTLE_DONE pgs=871 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f20517fa700 1 -- 172.21.15.16:0/1187917552 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2044000940 con 0x7f2054132d00 2019-12-05T06:29:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f20517fa700 1 -- 172.21.15.16:0/1187917552 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2044000e80 con 0x7f2054132d00 2019-12-05T06:29:20.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f205aa4d700 1 -- 172.21.15.16:0/1187917552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f205409b0f0 con 0x7f2054132d00 2019-12-05T06:29:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f20517fa700 1 -- 172.21.15.16:0/1187917552 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20440014d0 con 0x7f2054132d00 2019-12-05T06:29:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.555+0000 7f205aa4d700 1 -- 172.21.15.16:0/1187917552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f205409b0f0 con 0x7f2054132d00 2019-12-05T06:29:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.556+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=THROTTLE_DONE pgs=871 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.556+0000 7f20517fa700 1 -- 172.21.15.16:0/1187917552 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2044001070 con 0x7f2054132d00 2019-12-05T06:29:20.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.556+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=THROTTLE_DONE pgs=871 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.557+0000 7f20517fa700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2038023630 0x7f2038025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.557+0000 7f20517fa700 1 -- 172.21.15.16:0/1187917552 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f2044045ea0 con 0x7f2054132d00 2019-12-05T06:29:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.557+0000 7f2058fea700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2038023630 0x7f2038025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.557+0000 7f2058fea700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2038023630 0x7f2038025ac0 crc :-1 s=READY pgs=377 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:20.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.563+0000 7f205aa4d700 1 -- 172.21.15.16:0/1187917552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f205409b0f0 con 0x7f2054132d00 2019-12-05T06:29:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.567+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=THROTTLE_DONE pgs=871 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.567+0000 7f20517fa700 1 -- 172.21.15.16:0/1187917552 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f20440092b0 con 0x7f2054132d00 2019-12-05T06:29:20.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.808+0000 7f205aa4d700 1 -- 172.21.15.16:0/1187917552 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f205409b0f0 con 0x7f2054132d00 2019-12-05T06:29:20.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.809+0000 7f2053fff700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=THROTTLE_DONE pgs=871 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:20.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.809+0000 7f20517fa700 1 -- 172.21.15.16:0/1187917552 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 109 v109) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f2044000e80 con 0x7f2054132d00 2019-12-05T06:29:20.807 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 109 2019-12-05T06:29:20.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.813+0000 7f205aa4d700 1 -- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2038023630 msgr2=0x7f2038025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:20.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.813+0000 7f205aa4d700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2038023630 0x7f2038025ac0 crc :-1 s=READY pgs=377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.813+0000 7f205aa4d700 1 -- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 msgr2=0x7f2054132530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:20.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.813+0000 7f205aa4d700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 0x7f2054132530 secure :-1 s=READY pgs=871 cs=0 l=1 rx=0x7f2044010450 tx=0x7f20440107f0).stop 2019-12-05T06:29:20.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.814+0000 7f205aa4d700 1 -- 172.21.15.16:0/1187917552 shutdown_connections 2019-12-05T06:29:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.814+0000 7f205aa4d700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2054154b00 0x7f2054156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.814+0000 7f205aa4d700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2038023630 0x7f2038025ac0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.814+0000 7f205aa4d700 1 --2- 172.21.15.16:0/1187917552 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2054132d00 0x7f2054132530 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:20.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.814+0000 7f205aa4d700 1 -- 172.21.15.16:0/1187917552 shutdown_connections 2019-12-05T06:29:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.814+0000 7f205aa4d700 1 -- 172.21.15.16:0/1187917552 wait complete. 2019-12-05T06:29:20.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:20.814+0000 7f205aa4d700 1 -- 172.21.15.16:0/1187917552 >> 172.21.15.16:0/1187917552 conn(0x7f20540714f0 msgr2=0x7f205409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:20.809 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:20.809 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":109,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":109,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:20.435459+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:reconnect","state_seq":15,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:20.820 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 109, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 109, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6363": { "addr": "172.21.15.16:6833/1994840831", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 1994840831, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 1994840831, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6363, "incarnation": 108, "name": "a", "rank": 0, "state": "up:reconnect", "state_seq": 15 } }, "last_failure": 0, "last_failure_osd_epoch": 65, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:29:20.435459+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6363 } } } ], "standbys": [] } 2019-12-05T06:29:22.821 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:23.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.063+0000 7f9c74b92700 1 Processor -- start 2019-12-05T06:29:23.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.063+0000 7f9c74b92700 1 -- start start 2019-12-05T06:29:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.065+0000 7f9c74b92700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9c7008c6d0 0x7f9c7008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.065+0000 7f9c74b92700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 0x7f9c7008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.065+0000 7f9c74b92700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c700958e0 con 0x7f9c7008ef20 2019-12-05T06:29:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.065+0000 7f9c74b92700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c7008df90 con 0x7f9c7008c6d0 2019-12-05T06:29:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.065+0000 7f9c74b92700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c700a3f70 con 0x7f9c7008fef0 2019-12-05T06:29:23.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.066+0000 7f9c6f59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9c7008ef20 0x7f9c7008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39632/0 (socket says 172.21.15.16:39632) 2019-12-05T06:29:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.066+0000 7f9c6f59e700 1 -- 172.21.15.16:0/3567826359 learned_addr learned my addr 172.21.15.16:0/3567826359 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.066+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/3567826359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 0x7f9c7008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.066+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/3567826359 >> v2:172.21.15.76:3300/0 conn(0x7f9c7008c6d0 0x7f9c7008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.066+0000 7f9c6dd9b700 1 -- 172.21.15.16:0/3567826359 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1107243175 0 0) 0x7f9c60001150 con 0x7f9c7008ef20 2019-12-05T06:29:23.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.067+0000 7f9c6dd9b700 1 -- 172.21.15.16:0/3567826359 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c5c002e80 con 0x7f9c7008ef20 2019-12-05T06:29:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.067+0000 7f9c6e59c700 1 -- 172.21.15.16:0/3567826359 >> v2:172.21.15.76:3300/0 conn(0x7f9c7008c6d0 msgr2=0x7f9c7008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.067+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/3567826359 >> v2:172.21.15.76:3300/0 conn(0x7f9c7008c6d0 0x7f9c7008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.067+0000 7f9c6e59c700 1 -- 172.21.15.16:0/3567826359 >> v1:172.21.15.16:6789/0 conn(0x7f9c7008ef20 legacy=0x7f9c7008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.067+0000 7f9c6e59c700 1 -- 172.21.15.16:0/3567826359 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7008d540 con 0x7f9c7008fef0 2019-12-05T06:29:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.067+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/3567826359 >> v2:172.21.15.76:3300/0 conn(0x7f9c7008c6d0 0x7f9c7008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:29:23.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.067+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/3567826359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 0x7f9c7008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9c5800af50 tx=0x7f9c5800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.067+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/3567826359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 0x7f9c7008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9c5800af50 tx=0x7f9c5800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.067+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/3567826359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 0x7f9c7008d000 secure :-1 s=READY pgs=208 cs=0 l=1 rx=0x7f9c5800af50 tx=0x7f9c5800b760).ready entity=mon.2 client_cookie=6109808be988369a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.068+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/3567826359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 0x7f9c7008d000 secure :-1 s=THROTTLE_DONE pgs=208 cs=0 l=1 rx=0x7f9c5800af50 tx=0x7f9c5800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.068+0000 7f9c6dd9b700 1 -- 172.21.15.16:0/3567826359 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c58007210 con 0x7f9c7008fef0 2019-12-05T06:29:23.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.068+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/3567826359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 0x7f9c7008d000 secure :-1 s=THROTTLE_DONE pgs=208 cs=0 l=1 rx=0x7f9c5800af50 tx=0x7f9c5800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.068+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/3567826359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 0x7f9c7008d000 secure :-1 s=THROTTLE_DONE pgs=208 cs=0 l=1 rx=0x7f9c5800af50 tx=0x7f9c5800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.068+0000 7f9c6dd9b700 1 -- 172.21.15.16:0/3567826359 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c580099b0 con 0x7f9c7008fef0 2019-12-05T06:29:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.068+0000 7f9c6dd9b700 1 -- 172.21.15.16:0/3567826359 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c58007210 con 0x7f9c7008fef0 2019-12-05T06:29:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.068+0000 7f9c74b92700 1 -- 172.21.15.16:0/3567826359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 msgr2=0x7f9c7008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.068+0000 7f9c74b92700 1 --2- 172.21.15.16:0/3567826359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 0x7f9c7008d000 secure :-1 s=READY pgs=208 cs=0 l=1 rx=0x7f9c5800af50 tx=0x7f9c5800b760).stop 2019-12-05T06:29:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.070+0000 7f9c74b92700 1 -- 172.21.15.16:0/3567826359 shutdown_connections 2019-12-05T06:29:23.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.070+0000 7f9c74b92700 1 --2- 172.21.15.16:0/3567826359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9c7008fef0 0x7f9c7008d000 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.070+0000 7f9c74b92700 1 --2- 172.21.15.16:0/3567826359 >> v2:172.21.15.76:3300/0 conn(0x7f9c7008c6d0 0x7f9c7008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.070+0000 7f9c74b92700 1 -- 172.21.15.16:0/3567826359 shutdown_connections 2019-12-05T06:29:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.070+0000 7f9c74b92700 1 -- 172.21.15.16:0/3567826359 wait complete. 2019-12-05T06:29:23.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.070+0000 7f9c74b92700 1 -- 172.21.15.16:0/3567826359 >> 172.21.15.16:0/3567826359 conn(0x7f9c70061940 msgr2=0x7f9c70096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:23.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.076+0000 7f9c74b92700 1 Processor -- start 2019-12-05T06:29:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.076+0000 7f9c74b92700 1 -- start start 2019-12-05T06:29:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.077+0000 7f9c74b92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c7009c110 0x7f9c7009e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.078+0000 7f9c74b92700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7009ca80 con 0x7f9c70153b00 2019-12-05T06:29:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.078+0000 7f9c74b92700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c7009cf80 con 0x7f9c7009f7c0 2019-12-05T06:29:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.078+0000 7f9c74b92700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c7012d280 con 0x7f9c7009c110 2019-12-05T06:29:23.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.078+0000 7f9c6ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c7009c110 0x7f9c7009e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.078+0000 7f9c6ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9c7009c110 0x7f9c7009e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54466/0 (socket says 172.21.15.16:54466) 2019-12-05T06:29:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.078+0000 7f9c6ed9d700 1 -- 172.21.15.16:0/4283146171 learned_addr learned my addr 172.21.15.16:0/4283146171 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:29:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.078+0000 7f9c677fe700 1 -- 172.21.15.16:0/4283146171 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340194455 0 0) 0x7f9c58009580 con 0x7f9c7009f7c0 2019-12-05T06:29:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.078+0000 7f9c677fe700 1 -- 172.21.15.16:0/4283146171 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c54002e80 con 0x7f9c7009f7c0 2019-12-05T06:29:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c6ed9d700 1 -- 172.21.15.16:0/4283146171 >> v1:172.21.15.76:6789/0 conn(0x7f9c7009f7c0 legacy=0x7f9c7009ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c6ed9d700 1 -- 172.21.15.16:0/4283146171 >> v1:172.21.15.16:6789/0 conn(0x7f9c70153b00 legacy=0x7f9c70155f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:23.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c6ed9d700 1 -- 172.21.15.16:0/4283146171 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7009def0 con 0x7f9c7009c110 2019-12-05T06:29:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/4283146171 >> v2:172.21.15.16:3300/0 conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/4283146171 >> v2:172.21.15.16:3300/0 conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=READY pgs=875 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=THROTTLE_DONE pgs=875 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c677fe700 1 -- 172.21.15.16:0/4283146171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c68000cc0 con 0x7f9c7009c110 2019-12-05T06:29:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=THROTTLE_DONE pgs=875 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=THROTTLE_DONE pgs=875 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c677fe700 1 -- 172.21.15.16:0/4283146171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c68009900 con 0x7f9c7009c110 2019-12-05T06:29:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c74b92700 1 -- 172.21.15.16:0/4283146171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7009def0 con 0x7f9c7009c110 2019-12-05T06:29:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c677fe700 1 -- 172.21.15.16:0/4283146171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c68009f90 con 0x7f9c7009c110 2019-12-05T06:29:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.079+0000 7f9c74b92700 1 -- 172.21.15.16:0/4283146171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7009def0 con 0x7f9c7009c110 2019-12-05T06:29:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.081+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=THROTTLE_DONE pgs=875 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.081+0000 7f9c677fe700 1 -- 172.21.15.16:0/4283146171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9c6800dcd0 con 0x7f9c7009c110 2019-12-05T06:29:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.081+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=THROTTLE_DONE pgs=875 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.082+0000 7f9c677fe700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c54023670 0x7f9c54025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:23.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.082+0000 7f9c677fe700 1 -- 172.21.15.16:0/4283146171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f9c6804e9e0 con 0x7f9c7009c110 2019-12-05T06:29:23.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.082+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c54023670 0x7f9c54025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:23.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.082+0000 7f9c6e59c700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c54023670 0x7f9c54025ae0 crc :-1 s=READY pgs=378 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:23.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.095+0000 7f9c74b92700 1 -- 172.21.15.16:0/4283146171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c7009def0 con 0x7f9c7009c110 2019-12-05T06:29:23.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.102+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=THROTTLE_DONE pgs=875 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.102+0000 7f9c677fe700 1 -- 172.21.15.16:0/4283146171 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9c6800ac40 con 0x7f9c7009c110 2019-12-05T06:29:23.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.356+0000 7f9c74b92700 1 -- 172.21.15.16:0/4283146171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c7009def0 con 0x7f9c7009c110 2019-12-05T06:29:23.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.356+0000 7f9c6ed9d700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=THROTTLE_DONE pgs=875 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:23.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.356+0000 7f9c677fe700 1 -- 172.21.15.16:0/4283146171 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f9c6800dcd0 con 0x7f9c7009c110 2019-12-05T06:29:23.350 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 111 2019-12-05T06:29:23.350 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:23.350 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:22.445859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.360+0000 7f9c74b92700 1 -- 172.21.15.16:0/4283146171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c54023670 msgr2=0x7f9c54025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.360+0000 7f9c74b92700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c54023670 0x7f9c54025ae0 crc :-1 s=READY pgs=378 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.360+0000 7f9c74b92700 1 -- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 msgr2=0x7f9c7009e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:23.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.360+0000 7f9c74b92700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 0x7f9c7009e7f0 secure :-1 s=READY pgs=875 cs=0 l=1 rx=0x7f9c68007630 tx=0x7f9c6800f2c0).stop 2019-12-05T06:29:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.360+0000 7f9c74b92700 1 -- 172.21.15.16:0/4283146171 shutdown_connections 2019-12-05T06:29:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.360+0000 7f9c74b92700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c54023670 0x7f9c54025ae0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.360+0000 7f9c74b92700 1 --2- 172.21.15.16:0/4283146171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9c7009c110 0x7f9c7009e7f0 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.360+0000 7f9c74b92700 1 -- 172.21.15.16:0/4283146171 shutdown_connections 2019-12-05T06:29:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.360+0000 7f9c74b92700 1 -- 172.21.15.16:0/4283146171 wait complete. 2019-12-05T06:29:23.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:23.360+0000 7f9c74b92700 1 -- 172.21.15.16:0/4283146171 >> 172.21.15.16:0/4283146171 conn(0x7f9c7008acd0 msgr2=0x7f9c70099650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:23.363 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6264 2019-12-05T06:29:23.363 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.b 2019-12-05T06:29:28.364 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:29:28.365 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:29:28.365 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:29:28.371 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:29:28.372 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:28.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.528+0000 7ff724561700 1 Processor -- start 2019-12-05T06:29:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.528+0000 7ff724561700 1 -- start start 2019-12-05T06:29:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.529+0000 7ff724561700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff71c08c6d0 0x7ff71c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:28.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.529+0000 7ff724561700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c0958e0 con 0x7ff71c08fef0 2019-12-05T06:29:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.529+0000 7ff724561700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c0629e0 con 0x7ff71c08f720 2019-12-05T06:29:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.529+0000 7ff724561700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff71c0a3f70 con 0x7ff71c08c6d0 2019-12-05T06:29:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.529+0000 7ff7222fd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff71c08fef0 0x7ff71c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39660/0 (socket says 172.21.15.16:39660) 2019-12-05T06:29:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.529+0000 7ff7222fd700 1 -- 172.21.15.16:0/2341804316 learned_addr learned my addr 172.21.15.16:0/2341804316 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:28.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.529+0000 7ff722afe700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff71c08c6d0 0x7ff71c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2498785413 0 0) 0x7ff70c001150 con 0x7ff71c08fef0 2019-12-05T06:29:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff708002e80 con 0x7ff71c08fef0 2019-12-05T06:29:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 16845172 0 0) 0x7ff70c001130 con 0x7ff71c08fef0 2019-12-05T06:29:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff71c08c6d0 msgr2=0x7ff71c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff71c08c6d0 0x7ff71c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 >> v1:172.21.15.76:6789/0 conn(0x7ff71c08f720 legacy=0x7ff71c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:28.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c08fbc0 con 0x7ff71c08fef0 2019-12-05T06:29:28.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7ff70c001660 con 0x7ff71c08fef0 2019-12-05T06:29:28.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 >> v1:172.21.15.16:6789/0 conn(0x7ff71c08fef0 legacy=0x7ff71c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:28.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff708003ab0 0x7ff708005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:28.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff708006690 0x7ff708008b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:28.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff7080093a0 0x7ff70800b840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:28.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff70800bff0 con 0x7ff708003ab0 2019-12-05T06:29:28.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7080501f0 con 0x7ff708006690 2019-12-05T06:29:28.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff722afe700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff708003ab0 0x7ff708005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:28.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff708050420 con 0x7ff7080093a0 2019-12-05T06:29:28.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff7222fd700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff708006690 0x7ff708008b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:28.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff7232ff700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff7080093a0 0x7ff70800b840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:28.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff722afe700 1 -- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff7080093a0 msgr2=0x7ff70800b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:28.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff722afe700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff7080093a0 0x7ff70800b840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:28.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff722afe700 1 -- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff708006690 msgr2=0x7ff708008b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:28.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff722afe700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff708006690 0x7ff708008b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:28.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.530+0000 7ff722afe700 1 -- 172.21.15.16:0/2341804316 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7080506d0 con 0x7ff708003ab0 2019-12-05T06:29:28.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff7232ff700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff7080093a0 0x7ff70800b840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:29:28.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff722afe700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff708003ab0 0x7ff708005f10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff71800f630 tx=0x7ff718002bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff722afe700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff708003ab0 0x7ff708005f10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff71800f630 tx=0x7ff718002bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff722afe700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff708003ab0 0x7ff708005f10 secure :-1 s=READY pgs=877 cs=0 l=1 rx=0x7ff71800f630 tx=0x7ff718002bf0).ready entity=mon.0 client_cookie=c649c99c9382b89f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:28.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff722afe700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff708003ab0 0x7ff708005f10 secure :-1 s=THROTTLE_DONE pgs=877 cs=0 l=1 rx=0x7ff71800f630 tx=0x7ff718002bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff718000960 con 0x7ff708003ab0 2019-12-05T06:29:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff722afe700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff708003ab0 0x7ff708005f10 secure :-1 s=THROTTLE_DONE pgs=877 cs=0 l=1 rx=0x7ff71800f630 tx=0x7ff718002bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff722afe700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff708003ab0 0x7ff708005f10 secure :-1 s=THROTTLE_DONE pgs=877 cs=0 l=1 rx=0x7ff71800f630 tx=0x7ff718002bf0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff718000e80 con 0x7ff708003ab0 2019-12-05T06:29:28.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff721afc700 1 -- 172.21.15.16:0/2341804316 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff718000960 con 0x7ff708003ab0 2019-12-05T06:29:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff724561700 1 -- 172.21.15.16:0/2341804316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff708003ab0 msgr2=0x7ff708005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:28.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff724561700 1 --2- 172.21.15.16:0/2341804316 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff708003ab0 0x7ff708005f10 secure :-1 s=READY pgs=877 cs=0 l=1 rx=0x7ff71800f630 tx=0x7ff718002bf0).stop 2019-12-05T06:29:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.531+0000 7ff7232ff700 1 -- 172.21.15.16:0/2341804316 reap_dead start 2019-12-05T06:29:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.532+0000 7ff724561700 1 -- 172.21.15.16:0/2341804316 shutdown_connections 2019-12-05T06:29:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.532+0000 7ff724561700 1 -- 172.21.15.16:0/2341804316 shutdown_connections 2019-12-05T06:29:28.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.532+0000 7ff724561700 1 -- 172.21.15.16:0/2341804316 wait complete. 2019-12-05T06:29:28.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.532+0000 7ff724561700 1 -- 172.21.15.16:0/2341804316 >> 172.21.15.16:0/2341804316 conn(0x7ff71c061940 msgr2=0x7ff71c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:28.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.534+0000 7ff724561700 1 Processor -- start 2019-12-05T06:29:28.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.534+0000 7ff724561700 1 -- start start 2019-12-05T06:29:28.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.535+0000 7ff724561700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff71c09e000 0x7ff71c15a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:28.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.535+0000 7ff724561700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c09e440 con 0x7ff71c09e7d0 2019-12-05T06:29:28.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.535+0000 7ff724561700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff71c12cb90 con 0x7ff71c1302f0 2019-12-05T06:29:28.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.535+0000 7ff724561700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff71c131ff0 con 0x7ff71c09e000 2019-12-05T06:29:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.535+0000 7ff7222fd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff71c09e7d0 0x7ff71c156e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39672/0 (socket says 172.21.15.16:39672) 2019-12-05T06:29:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.535+0000 7ff7222fd700 1 -- 172.21.15.16:0/376384044 learned_addr learned my addr 172.21.15.16:0/376384044 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.535+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> v2:172.21.15.16:3300/0 conn(0x7ff71c09e000 0x7ff71c15a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:28.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.535+0000 7ff7137fe700 1 -- 172.21.15.16:0/376384044 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1543462454 0 0) 0x7ff70c001180 con 0x7ff71c09e7d0 2019-12-05T06:29:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7137fe700 1 -- 172.21.15.16:0/376384044 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff704002e80 con 0x7ff71c09e7d0 2019-12-05T06:29:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7232ff700 1 -- 172.21.15.16:0/376384044 >> v1:172.21.15.76:6789/0 conn(0x7ff71c1302f0 legacy=0x7ff71c09d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7232ff700 1 -- 172.21.15.16:0/376384044 >> v1:172.21.15.16:6789/0 conn(0x7ff71c09e7d0 legacy=0x7ff71c156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:28.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7232ff700 1 -- 172.21.15.16:0/376384044 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c132680 con 0x7ff71c09e000 2019-12-05T06:29:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> v2:172.21.15.16:3300/0 conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> v2:172.21.15.16:3300/0 conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=READY pgs=879 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=THROTTLE_DONE pgs=879 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=THROTTLE_DONE pgs=879 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7137fe700 1 -- 172.21.15.16:0/376384044 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff71400c280 con 0x7ff71c09e000 2019-12-05T06:29:28.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=THROTTLE_DONE pgs=879 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7137fe700 1 -- 172.21.15.16:0/376384044 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff714000e70 con 0x7ff71c09e000 2019-12-05T06:29:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff7137fe700 1 -- 172.21.15.16:0/376384044 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff71400c860 con 0x7ff71c09e000 2019-12-05T06:29:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff724561700 1 -- 172.21.15.16:0/376384044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff71c132680 con 0x7ff71c09e000 2019-12-05T06:29:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.536+0000 7ff724561700 1 -- 172.21.15.16:0/376384044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff71c132680 con 0x7ff71c09e000 2019-12-05T06:29:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.537+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=THROTTLE_DONE pgs=879 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.537+0000 7ff7137fe700 1 -- 172.21.15.16:0/376384044 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff71400bfe0 con 0x7ff71c09e000 2019-12-05T06:29:28.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.537+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=THROTTLE_DONE pgs=879 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.537+0000 7ff7137fe700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff704023670 0x7ff704025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.537+0000 7ff7137fe700 1 -- 172.21.15.16:0/376384044 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7ff714046140 con 0x7ff71c09e000 2019-12-05T06:29:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.538+0000 7ff722afe700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff704023670 0x7ff704025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.538+0000 7ff722afe700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff704023670 0x7ff704025ae0 crc :-1 s=READY pgs=379 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.543+0000 7ff724561700 1 -- 172.21.15.16:0/376384044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff71c132680 con 0x7ff71c09e000 2019-12-05T06:29:28.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.547+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=THROTTLE_DONE pgs=879 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.547+0000 7ff7137fe700 1 -- 172.21.15.16:0/376384044 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff71400bc90 con 0x7ff71c09e000 2019-12-05T06:29:28.573 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:29:28.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.744+0000 7ff724561700 1 -- 172.21.15.16:0/376384044 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff71c132680 con 0x7ff71c09e000 2019-12-05T06:29:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.745+0000 7ff7232ff700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=THROTTLE_DONE pgs=879 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.745+0000 7ff7137fe700 1 -- 172.21.15.16:0/376384044 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 111 v111) v1 ==== 95+0+1775 (secure 0 0 0) 0x7ff71400bf40 con 0x7ff71c09e000 2019-12-05T06:29:28.738 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 111 2019-12-05T06:29:28.738 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:28.738 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":111,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:22.445859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.748+0000 7ff724561700 1 -- 172.21.15.16:0/376384044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff704023670 msgr2=0x7ff704025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.748+0000 7ff724561700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff704023670 0x7ff704025ae0 crc :-1 s=READY pgs=379 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.748+0000 7ff724561700 1 -- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 msgr2=0x7ff71c15a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.748+0000 7ff724561700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 0x7ff71c15a690 secure :-1 s=READY pgs=879 cs=0 l=1 rx=0x7ff714010700 tx=0x7ff714010aa0).stop 2019-12-05T06:29:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.748+0000 7ff724561700 1 -- 172.21.15.16:0/376384044 shutdown_connections 2019-12-05T06:29:28.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.748+0000 7ff724561700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff704023670 0x7ff704025ae0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.748+0000 7ff724561700 1 --2- 172.21.15.16:0/376384044 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff71c09e000 0x7ff71c15a690 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.748+0000 7ff724561700 1 -- 172.21.15.16:0/376384044 shutdown_connections 2019-12-05T06:29:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.748+0000 7ff724561700 1 -- 172.21.15.16:0/376384044 wait complete. 2019-12-05T06:29:28.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:28.748+0000 7ff724561700 1 -- 172.21.15.16:0/376384044 >> 172.21.15.16:0/376384044 conn(0x7ff71c0a2850 msgr2=0x7ff71c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:28.751 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:29:30.751 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:29:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.021+0000 7f0da09b0700 1 Processor -- start 2019-12-05T06:29:31.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.021+0000 7f0da09b0700 1 -- start start 2019-12-05T06:29:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.023+0000 7f0da09b0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c08c6d0 0x7f0d9c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.023+0000 7f0da09b0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0d9c08f720 0x7f0d9c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:29:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.023+0000 7f0da09b0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9c0958e0 con 0x7f0d9c08fef0 2019-12-05T06:29:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.023+0000 7f0da09b0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d9c0a3f70 con 0x7f0d9c08f720 2019-12-05T06:29:31.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.023+0000 7f0da09b0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c08d830 con 0x7f0d9c08c6d0 2019-12-05T06:29:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.023+0000 7f0d9ad9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c08c6d0 0x7f0d9c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.024+0000 7f0d9b59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0d9c08f720 0x7f0d9c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.024+0000 7f0d9a59c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f0d9c08fef0 0x7f0d9c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:60994/0 (socket says 172.21.15.16:60994) 2019-12-05T06:29:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.024+0000 7f0d9a59c700 1 -- 172.21.15.16:0/1587691102 learned_addr learned my addr 172.21.15.16:0/1587691102 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:29:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.024+0000 7f0d9b59e700 1 -- 172.21.15.16:0/1587691102 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c08c6d0 msgr2=0x7f0d9c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:29:31.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.024+0000 7f0d9b59e700 1 --2- 172.21.15.16:0/1587691102 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c08c6d0 0x7f0d9c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.024+0000 7f0d9b59e700 1 -- 172.21.15.16:0/1587691102 >> v1:172.21.15.76:6789/0 conn(0x7f0d9c08fef0 legacy=0x7f0d9c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.024+0000 7f0d9b59e700 1 -- 172.21.15.16:0/1587691102 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c08fbc0 con 0x7f0d9c08f720 2019-12-05T06:29:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.024+0000 7f0d99d9b700 1 -- 172.21.15.16:0/1587691102 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593508302 0 0) 0x7f0d84001100 con 0x7f0d9c08fef0 2019-12-05T06:29:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0d9b59e700 1 --2- 172.21.15.16:0/1587691102 >> v2:172.21.15.76:3300/0 conn(0x7f0d9c08f720 0x7f0d9c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0d8c00af80 tx=0x7f0d8c00b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0d9b59e700 1 --2- 172.21.15.16:0/1587691102 >> v2:172.21.15.76:3300/0 conn(0x7f0d9c08f720 0x7f0d9c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0d8c00af80 tx=0x7f0d8c00b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0d9b59e700 1 --2- 172.21.15.16:0/1587691102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c08f720 0x7f0d9c08d000 secure :-1 s=READY pgs=637 cs=0 l=1 rx=0x7f0d8c00af80 tx=0x7f0d8c00b710).ready entity=mon.1 client_cookie=a30287cf6a3ce33b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0d9b59e700 1 --2- 172.21.15.16:0/1587691102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c08f720 0x7f0d9c08d000 secure :-1 s=THROTTLE_DONE pgs=637 cs=0 l=1 rx=0x7f0d8c00af80 tx=0x7f0d8c00b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0d99d9b700 1 -- 172.21.15.16:0/1587691102 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d8c007240 con 0x7f0d9c08f720 2019-12-05T06:29:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0d9b59e700 1 --2- 172.21.15.16:0/1587691102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c08f720 0x7f0d9c08d000 secure :-1 s=THROTTLE_DONE pgs=637 cs=0 l=1 rx=0x7f0d8c00af80 tx=0x7f0d8c00b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0d9b59e700 1 --2- 172.21.15.16:0/1587691102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c08f720 0x7f0d9c08d000 secure :-1 s=THROTTLE_DONE pgs=637 cs=0 l=1 rx=0x7f0d8c00af80 tx=0x7f0d8c00b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0d99d9b700 1 -- 172.21.15.16:0/1587691102 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d8c000b10 con 0x7f0d9c08f720 2019-12-05T06:29:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0d99d9b700 1 -- 172.21.15.16:0/1587691102 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d8c009ea0 con 0x7f0d9c08f720 2019-12-05T06:29:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0da09b0700 1 -- 172.21.15.16:0/1587691102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c08f720 msgr2=0x7f0d9c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.025+0000 7f0da09b0700 1 --2- 172.21.15.16:0/1587691102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c08f720 0x7f0d9c08d000 secure :-1 s=READY pgs=637 cs=0 l=1 rx=0x7f0d8c00af80 tx=0x7f0d8c00b710).stop 2019-12-05T06:29:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.028+0000 7f0da09b0700 1 -- 172.21.15.16:0/1587691102 shutdown_connections 2019-12-05T06:29:31.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.028+0000 7f0da09b0700 1 --2- 172.21.15.16:0/1587691102 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0d9c08f720 0x7f0d9c08d000 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:31.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.028+0000 7f0da09b0700 1 --2- 172.21.15.16:0/1587691102 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c08c6d0 0x7f0d9c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:29:31.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.028+0000 7f0da09b0700 1 -- 172.21.15.16:0/1587691102 shutdown_connections 2019-12-05T06:29:31.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.028+0000 7f0da09b0700 1 -- 172.21.15.16:0/1587691102 wait complete. 2019-12-05T06:29:31.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.028+0000 7f0da09b0700 1 -- 172.21.15.16:0/1587691102 >> 172.21.15.16:0/1587691102 conn(0x7f0d9c061940 msgr2=0x7f0d9c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:31.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.033+0000 7f0da09b0700 1 Processor -- start 2019-12-05T06:29:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.033+0000 7f0da09b0700 1 -- start start 2019-12-05T06:29:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.034+0000 7f0da09b0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0d9c1325e0 0x7f0d9c12b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.034+0000 7f0da09b0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.034+0000 7f0da09b0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d9c128b20 con 0x7f0d9c12c1f0 2019-12-05T06:29:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.034+0000 7f0da09b0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d9c12a7e0 con 0x7f0d9c1325e0 2019-12-05T06:29:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.034+0000 7f0da09b0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d9c129db0 con 0x7f0d9c12c740 2019-12-05T06:29:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.035+0000 7f0d9a59c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.035+0000 7f0d9ad9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0d9c1325e0 0x7f0d9c12b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.035+0000 7f0d9ad9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0d9c1325e0 0x7f0d9c12b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:40616/0 (socket says 172.21.15.16:40616) 2019-12-05T06:29:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.035+0000 7f0d9ad9d700 1 -- 172.21.15.16:0/872053168 learned_addr learned my addr 172.21.15.16:0/872053168 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:29:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d9a59c700 1 -- 172.21.15.16:0/872053168 >> v2:172.21.15.76:3300/0 conn(0x7f0d9c1325e0 msgr2=0x7f0d9c12b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> v2:172.21.15.76:3300/0 conn(0x7f0d9c1325e0 0x7f0d9c12b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d9a59c700 1 -- 172.21.15.16:0/872053168 >> v1:172.21.15.76:6789/0 conn(0x7f0d9c12c1f0 legacy=0x7f0d9c06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d9a59c700 1 -- 172.21.15.16:0/872053168 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c12ae70 con 0x7f0d9c12c740 2019-12-05T06:29:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d9ad9d700 1 --2- 172.21.15.16:0/872053168 >> v2:172.21.15.76:3300/0 conn(0x7f0d9c1325e0 0x7f0d9c12b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:29:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=READY pgs=209 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.036+0000 7f0d937fe700 1 -- 172.21.15.16:0/872053168 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d84001100 con 0x7f0d9c12c740 2019-12-05T06:29:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.037+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.037+0000 7f0d937fe700 1 -- 172.21.15.16:0/872053168 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d840043b0 con 0x7f0d9c12c740 2019-12-05T06:29:31.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.037+0000 7f0da09b0700 1 -- 172.21.15.16:0/872053168 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d9c12ae70 con 0x7f0d9c12c740 2019-12-05T06:29:31.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.037+0000 7f0d937fe700 1 -- 172.21.15.16:0/872053168 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d84004a40 con 0x7f0d9c12c740 2019-12-05T06:29:31.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.037+0000 7f0da09b0700 1 -- 172.21.15.16:0/872053168 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d9c12ae70 con 0x7f0d9c12c740 2019-12-05T06:29:31.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.038+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.039+0000 7f0d937fe700 1 -- 172.21.15.16:0/872053168 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0d84001000 con 0x7f0d9c12c740 2019-12-05T06:29:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.039+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.039+0000 7f0d937fe700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d80023630 0x7f0d80025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:29:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.039+0000 7f0d937fe700 1 -- 172.21.15.16:0/872053168 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f0d84045d00 con 0x7f0d9c12c740 2019-12-05T06:29:31.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.039+0000 7f0d9ad9d700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d80023630 0x7f0d80025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:29:31.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.040+0000 7f0d9ad9d700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d80023630 0x7f0d80025ac0 crc :-1 s=READY pgs=381 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:29:31.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.053+0000 7f0da09b0700 1 -- 172.21.15.16:0/872053168 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d9c12ae70 con 0x7f0d9c12c740 2019-12-05T06:29:31.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.063+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.063+0000 7f0d937fe700 1 -- 172.21.15.16:0/872053168 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0d84000df0 con 0x7f0d9c12c740 2019-12-05T06:29:31.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.334+0000 7f0da09b0700 1 -- 172.21.15.16:0/872053168 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d9c12ae70 con 0x7f0d9c12c740 2019-12-05T06:29:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.335+0000 7f0d9a59c700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=THROTTLE_DONE pgs=209 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:29:31.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.335+0000 7f0d937fe700 1 -- 172.21.15.16:0/872053168 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f0d84001000 con 0x7f0d9c12c740 2019-12-05T06:29:31.330 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:29:31.330 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:22.445859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:29:31.330 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2019-12-05T06:29:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.339+0000 7f0da09b0700 1 -- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d80023630 msgr2=0x7f0d80025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.339+0000 7f0da09b0700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d80023630 0x7f0d80025ac0 crc :-1 s=READY pgs=381 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:31.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.339+0000 7f0da09b0700 1 -- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 msgr2=0x7f0d9c12bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:29:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.339+0000 7f0da09b0700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 secure :-1 s=READY pgs=209 cs=0 l=1 rx=0x7f0d84010390 tx=0x7f0d84010730).stop 2019-12-05T06:29:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.339+0000 7f0da09b0700 1 -- 172.21.15.16:0/872053168 shutdown_connections 2019-12-05T06:29:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.339+0000 7f0da09b0700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0d80023630 0x7f0d80025ac0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.339+0000 7f0da09b0700 1 --2- 172.21.15.16:0/872053168 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0d9c12c740 0x7f0d9c12bcb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.339+0000 7f0da09b0700 1 --2- 172.21.15.16:0/872053168 >> v2:172.21.15.76:3300/0 conn(0x7f0d9c1325e0 0x7f0d9c12b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:29:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.340+0000 7f0da09b0700 1 -- 172.21.15.16:0/872053168 shutdown_connections 2019-12-05T06:29:31.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.340+0000 7f0da09b0700 1 -- 172.21.15.16:0/872053168 wait complete. 2019-12-05T06:29:31.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:29:31.340+0000 7f0da09b0700 1 -- 172.21.15.16:0/872053168 >> 172.21.15.16:0/872053168 conn(0x7f0d9c0a2850 msgr2=0x7f0d9c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:29:31.345 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:29:31.345 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 52 secs before thrashing 2019-12-05T06:29:35.274 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:29:35.280 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:30:05.428 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:30:05.438 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:30:23.346 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:23.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.593+0000 7f9e2d69d700 1 Processor -- start 2019-12-05T06:30:23.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.593+0000 7f9e2d69d700 1 -- start start 2019-12-05T06:30:23.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.595+0000 7f9e2d69d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e2808c6d0 0x7f9e2808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:23.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.595+0000 7f9e2d69d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e2808f720 0x7f9e2808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:23.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.595+0000 7f9e2d69d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e280958e0 con 0x7f9e2808fef0 2019-12-05T06:30:23.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.595+0000 7f9e2d69d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e280a3f70 con 0x7f9e2808f720 2019-12-05T06:30:23.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.595+0000 7f9e2d69d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e2808d830 con 0x7f9e2808c6d0 2019-12-05T06:30:23.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.595+0000 7f9e27fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e2808f720 0x7f9e2808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:23.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.595+0000 7f9e27fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e2808f720 0x7f9e2808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54808/0 (socket says 172.21.15.16:54808) 2019-12-05T06:30:23.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.595+0000 7f9e26ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9e2808fef0 0x7f9e2808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39972/0 (socket says 172.21.15.16:39972) 2019-12-05T06:30:23.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.595+0000 7f9e27fff700 1 -- 172.21.15.16:0/85536674 learned_addr learned my addr 172.21.15.16:0/85536674 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:23.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.595+0000 7f9e277fe700 1 --2- 172.21.15.16:0/85536674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e2808c6d0 0x7f9e2808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:23.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e27fff700 1 -- 172.21.15.16:0/85536674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e2808c6d0 msgr2=0x7f9e2808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:23.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e27fff700 1 --2- 172.21.15.16:0/85536674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e2808c6d0 0x7f9e2808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:23.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e27fff700 1 -- 172.21.15.16:0/85536674 >> v1:172.21.15.16:6789/0 conn(0x7f9e2808fef0 legacy=0x7f9e2808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:23.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e267fc700 1 -- 172.21.15.16:0/85536674 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919945624 0 0) 0x7f9e10001480 con 0x7f9e2808fef0 2019-12-05T06:30:23.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e27fff700 1 -- 172.21.15.16:0/85536674 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2808fbc0 con 0x7f9e2808f720 2019-12-05T06:30:23.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e27fff700 1 --2- 172.21.15.16:0/85536674 >> v2:172.21.15.16:3300/0 conn(0x7f9e2808f720 0x7f9e2808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9e1800f6b0 tx=0x7f9e1800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e27fff700 1 --2- 172.21.15.16:0/85536674 >> v2:172.21.15.16:3300/0 conn(0x7f9e2808f720 0x7f9e2808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9e1800f6b0 tx=0x7f9e1800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e27fff700 1 --2- 172.21.15.16:0/85536674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2808f720 0x7f9e2808d000 secure :-1 s=READY pgs=882 cs=0 l=1 rx=0x7f9e1800f6b0 tx=0x7f9e1800fa50).ready entity=mon.0 client_cookie=f337be5954889dfd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:23.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e27fff700 1 --2- 172.21.15.16:0/85536674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2808f720 0x7f9e2808d000 secure :-1 s=THROTTLE_DONE pgs=882 cs=0 l=1 rx=0x7f9e1800f6b0 tx=0x7f9e1800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e267fc700 1 -- 172.21.15.16:0/85536674 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e18000a70 con 0x7f9e2808f720 2019-12-05T06:30:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e27fff700 1 --2- 172.21.15.16:0/85536674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2808f720 0x7f9e2808d000 secure :-1 s=THROTTLE_DONE pgs=882 cs=0 l=1 rx=0x7f9e1800f6b0 tx=0x7f9e1800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e27fff700 1 --2- 172.21.15.16:0/85536674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2808f720 0x7f9e2808d000 secure :-1 s=THROTTLE_DONE pgs=882 cs=0 l=1 rx=0x7f9e1800f6b0 tx=0x7f9e1800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e267fc700 1 -- 172.21.15.16:0/85536674 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e180010e0 con 0x7f9e2808f720 2019-12-05T06:30:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e267fc700 1 -- 172.21.15.16:0/85536674 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e18000a70 con 0x7f9e2808f720 2019-12-05T06:30:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e2d69d700 1 -- 172.21.15.16:0/85536674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2808f720 msgr2=0x7f9e2808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:23.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.596+0000 7f9e2d69d700 1 --2- 172.21.15.16:0/85536674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2808f720 0x7f9e2808d000 secure :-1 s=READY pgs=882 cs=0 l=1 rx=0x7f9e1800f6b0 tx=0x7f9e1800fa50).stop 2019-12-05T06:30:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.598+0000 7f9e2d69d700 1 -- 172.21.15.16:0/85536674 shutdown_connections 2019-12-05T06:30:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.598+0000 7f9e2d69d700 1 --2- 172.21.15.16:0/85536674 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e2808f720 0x7f9e2808d000 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.598+0000 7f9e2d69d700 1 --2- 172.21.15.16:0/85536674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9e2808c6d0 0x7f9e2808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.598+0000 7f9e2d69d700 1 -- 172.21.15.16:0/85536674 shutdown_connections 2019-12-05T06:30:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.598+0000 7f9e2d69d700 1 -- 172.21.15.16:0/85536674 wait complete. 2019-12-05T06:30:23.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.598+0000 7f9e2d69d700 1 -- 172.21.15.16:0/85536674 >> 172.21.15.16:0/85536674 conn(0x7f9e28061940 msgr2=0x7f9e28096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:23.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.601+0000 7f9e2d69d700 1 Processor -- start 2019-12-05T06:30:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.602+0000 7f9e2d69d700 1 -- start start 2019-12-05T06:30:23.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.602+0000 7f9e2d69d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9e28096e90 0x7f9e2809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e2d69d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9e2809b920 0x7f9e28097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e2d69d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e28084b90 con 0x7f9e28084f30 2019-12-05T06:30:23.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e2d69d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e2809a450 con 0x7f9e28096e90 2019-12-05T06:30:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e2d69d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e28099a20 con 0x7f9e2809b920 2019-12-05T06:30:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e26ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9e28084f30 0x7f9e28097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:39980/0 (socket says 172.21.15.16:39980) 2019-12-05T06:30:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e26ffd700 1 -- 172.21.15.16:0/632202723 learned_addr learned my addr 172.21.15.16:0/632202723 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:30:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> v2:172.21.15.16:3300/0 conn(0x7f9e28096e90 0x7f9e2809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e27fff700 1 --2- 172.21.15.16:0/632202723 >> v2:172.21.15.76:3300/0 conn(0x7f9e2809b920 0x7f9e28097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e1ffff700 1 -- 172.21.15.16:0/632202723 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2404867705 0 0) 0x7f9e100014d0 con 0x7f9e28084f30 2019-12-05T06:30:23.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e1ffff700 1 -- 172.21.15.16:0/632202723 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0c002e80 con 0x7f9e28084f30 2019-12-05T06:30:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e277fe700 1 -- 172.21.15.16:0/632202723 >> v2:172.21.15.76:3300/0 conn(0x7f9e2809b920 msgr2=0x7f9e28097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> v2:172.21.15.76:3300/0 conn(0x7f9e2809b920 0x7f9e28097a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e277fe700 1 -- 172.21.15.16:0/632202723 >> v1:172.21.15.16:6789/0 conn(0x7f9e28084f30 legacy=0x7f9e28097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.603+0000 7f9e277fe700 1 -- 172.21.15.16:0/632202723 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e2809aae0 con 0x7f9e28096e90 2019-12-05T06:30:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> v2:172.21.15.16:3300/0 conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> v2:172.21.15.16:3300/0 conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=READY pgs=884 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e1ffff700 1 -- 172.21.15.16:0/632202723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e20005770 con 0x7f9e28096e90 2019-12-05T06:30:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e1ffff700 1 -- 172.21.15.16:0/632202723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e20005390 con 0x7f9e28096e90 2019-12-05T06:30:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e1ffff700 1 -- 172.21.15.16:0/632202723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e20005d50 con 0x7f9e28096e90 2019-12-05T06:30:23.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e2d69d700 1 -- 172.21.15.16:0/632202723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e2809aae0 con 0x7f9e28096e90 2019-12-05T06:30:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.604+0000 7f9e2d69d700 1 -- 172.21.15.16:0/632202723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e2809aae0 con 0x7f9e28096e90 2019-12-05T06:30:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.605+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.605+0000 7f9e1ffff700 1 -- 172.21.15.16:0/632202723 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9e20005580 con 0x7f9e28096e90 2019-12-05T06:30:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.605+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.606+0000 7f9e1ffff700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e0c023670 0x7f9e0c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.606+0000 7f9e1ffff700 1 -- 172.21.15.16:0/632202723 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f9e20045d30 con 0x7f9e28096e90 2019-12-05T06:30:23.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.606+0000 7f9e26ffd700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e0c023670 0x7f9e0c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:23.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.606+0000 7f9e26ffd700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e0c023670 0x7f9e0c025ae0 crc :-1 s=READY pgs=383 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:23.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.608+0000 7f9e2d69d700 1 -- 172.21.15.16:0/632202723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e2809aae0 con 0x7f9e28096e90 2019-12-05T06:30:23.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.613+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.614+0000 7f9e1ffff700 1 -- 172.21.15.16:0/632202723 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9e20011630 con 0x7f9e28096e90 2019-12-05T06:30:23.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.833+0000 7f9e2d69d700 1 -- 172.21.15.16:0/632202723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e2809aae0 con 0x7f9e28096e90 2019-12-05T06:30:23.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.833+0000 7f9e277fe700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=THROTTLE_DONE pgs=884 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:23.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.833+0000 7f9e1ffff700 1 -- 172.21.15.16:0/632202723 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f9e20011630 con 0x7f9e28096e90 2019-12-05T06:30:23.827 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2019-12-05T06:30:23.828 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:23.828 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:22.445859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:23.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.836+0000 7f9e2d69d700 1 -- 172.21.15.16:0/632202723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e0c023670 msgr2=0x7f9e0c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:23.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.836+0000 7f9e2d69d700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e0c023670 0x7f9e0c025ae0 crc :-1 s=READY pgs=383 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:23.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.836+0000 7f9e2d69d700 1 -- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 msgr2=0x7f9e2809b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:23.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.836+0000 7f9e2d69d700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 0x7f9e2809b3e0 secure :-1 s=READY pgs=884 cs=0 l=1 rx=0x7f9e20010390 tx=0x7f9e20010190).stop 2019-12-05T06:30:23.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.836+0000 7f9e2d69d700 1 -- 172.21.15.16:0/632202723 shutdown_connections 2019-12-05T06:30:23.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.836+0000 7f9e2d69d700 1 --2- 172.21.15.16:0/632202723 >> v2:172.21.15.76:3300/0 conn(0x7f9e2809b920 0x7f9e28097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.836+0000 7f9e2d69d700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9e0c023670 0x7f9e0c025ae0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.836+0000 7f9e2d69d700 1 --2- 172.21.15.16:0/632202723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9e28096e90 0x7f9e2809b3e0 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.836+0000 7f9e2d69d700 1 -- 172.21.15.16:0/632202723 shutdown_connections 2019-12-05T06:30:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.836+0000 7f9e2d69d700 1 -- 172.21.15.16:0/632202723 wait complete. 2019-12-05T06:30:23.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:23.837+0000 7f9e2d69d700 1 -- 172.21.15.16:0/632202723 >> 172.21.15.16:0/632202723 conn(0x7f9e280714f0 msgr2=0x7f9e2809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:23.841 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:30:23.841 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:30:23.841 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:30:23.848 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:30:23.849 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:24.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.158+0000 7fc4852f3700 1 Processor -- start 2019-12-05T06:30:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.158+0000 7fc4852f3700 1 -- start start 2019-12-05T06:30:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.160+0000 7fc4852f3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc48008d2c0 0x7fc48008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.160+0000 7fc4852f3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc48008fef0 0x7fc48008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.160+0000 7fc4852f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4800958e0 con 0x7fc48008c6d0 2019-12-05T06:30:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.160+0000 7fc4852f3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc48008df90 con 0x7fc48008fef0 2019-12-05T06:30:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.160+0000 7fc4852f3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4800a3f70 con 0x7fc48008d2c0 2019-12-05T06:30:24.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.160+0000 7fc47ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc48008fef0 0x7fc48008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.161+0000 7fc47ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc48008fef0 0x7fc48008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54832/0 (socket says 172.21.15.16:54832) 2019-12-05T06:30:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.161+0000 7fc47effd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc48008d2c0 0x7fc48008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.161+0000 7fc47ffff700 1 -- 172.21.15.16:0/2926652428 learned_addr learned my addr 172.21.15.16:0/2926652428 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.161+0000 7fc47effd700 1 -- 172.21.15.16:0/2926652428 >> v2:172.21.15.16:3300/0 conn(0x7fc48008fef0 msgr2=0x7fc48008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.161+0000 7fc47effd700 1 --2- 172.21.15.16:0/2926652428 >> v2:172.21.15.16:3300/0 conn(0x7fc48008fef0 0x7fc48008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.161+0000 7fc47effd700 1 -- 172.21.15.16:0/2926652428 >> v1:172.21.15.76:6789/0 conn(0x7fc48008c6d0 legacy=0x7fc48008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:24.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.161+0000 7fc47effd700 1 -- 172.21.15.16:0/2926652428 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc48008e620 con 0x7fc48008d2c0 2019-12-05T06:30:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.161+0000 7fc47e7fc700 1 -- 172.21.15.16:0/2926652428 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3952624872 0 0) 0x7fc478001400 con 0x7fc48008c6d0 2019-12-05T06:30:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.161+0000 7fc47ffff700 1 --2- 172.21.15.16:0/2926652428 >> v2:172.21.15.16:3300/0 conn(0x7fc48008fef0 0x7fc48008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.161+0000 7fc47effd700 1 --2- 172.21.15.16:0/2926652428 >> v2:172.21.15.76:3300/0 conn(0x7fc48008d2c0 0x7fc48008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc46800af50 tx=0x7fc46800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.162+0000 7fc47effd700 1 --2- 172.21.15.16:0/2926652428 >> v2:172.21.15.76:3300/0 conn(0x7fc48008d2c0 0x7fc48008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc46800af50 tx=0x7fc46800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.162+0000 7fc47effd700 1 --2- 172.21.15.16:0/2926652428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc48008d2c0 0x7fc48008ef20 secure :-1 s=READY pgs=641 cs=0 l=1 rx=0x7fc46800af50 tx=0x7fc46800b760).ready entity=mon.1 client_cookie=19d1c26dcd20d9ef server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.162+0000 7fc47effd700 1 --2- 172.21.15.16:0/2926652428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc48008d2c0 0x7fc48008ef20 secure :-1 s=THROTTLE_DONE pgs=641 cs=0 l=1 rx=0x7fc46800af50 tx=0x7fc46800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.162+0000 7fc47e7fc700 1 -- 172.21.15.16:0/2926652428 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc468007210 con 0x7fc48008d2c0 2019-12-05T06:30:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.162+0000 7fc47effd700 1 --2- 172.21.15.16:0/2926652428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc48008d2c0 0x7fc48008ef20 secure :-1 s=THROTTLE_DONE pgs=641 cs=0 l=1 rx=0x7fc46800af50 tx=0x7fc46800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.162+0000 7fc47effd700 1 --2- 172.21.15.16:0/2926652428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc48008d2c0 0x7fc48008ef20 secure :-1 s=THROTTLE_DONE pgs=641 cs=0 l=1 rx=0x7fc46800af50 tx=0x7fc46800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.162+0000 7fc47e7fc700 1 -- 172.21.15.16:0/2926652428 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4680099b0 con 0x7fc48008d2c0 2019-12-05T06:30:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.162+0000 7fc47e7fc700 1 -- 172.21.15.16:0/2926652428 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc46800a080 con 0x7fc48008d2c0 2019-12-05T06:30:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.162+0000 7fc4852f3700 1 -- 172.21.15.16:0/2926652428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc48008d2c0 msgr2=0x7fc48008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.162+0000 7fc4852f3700 1 --2- 172.21.15.16:0/2926652428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc48008d2c0 0x7fc48008ef20 secure :-1 s=READY pgs=641 cs=0 l=1 rx=0x7fc46800af50 tx=0x7fc46800b760).stop 2019-12-05T06:30:24.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.164+0000 7fc4852f3700 1 -- 172.21.15.16:0/2926652428 shutdown_connections 2019-12-05T06:30:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.164+0000 7fc4852f3700 1 --2- 172.21.15.16:0/2926652428 >> v2:172.21.15.16:3300/0 conn(0x7fc48008fef0 0x7fc48008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.164+0000 7fc4852f3700 1 --2- 172.21.15.16:0/2926652428 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc48008d2c0 0x7fc48008ef20 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.164+0000 7fc4852f3700 1 -- 172.21.15.16:0/2926652428 shutdown_connections 2019-12-05T06:30:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.164+0000 7fc4852f3700 1 -- 172.21.15.16:0/2926652428 wait complete. 2019-12-05T06:30:24.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.164+0000 7fc4852f3700 1 -- 172.21.15.16:0/2926652428 >> 172.21.15.16:0/2926652428 conn(0x7fc480061940 msgr2=0x7fc480096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:24.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.169+0000 7fc4852f3700 1 Processor -- start 2019-12-05T06:30:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.170+0000 7fc4852f3700 1 -- start start 2019-12-05T06:30:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.172+0000 7fc4852f3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4801302f0 0x7fc48009d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.172+0000 7fc4852f3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48012cb90 con 0x7fc480156e70 2019-12-05T06:30:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.172+0000 7fc4852f3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc48012d040 con 0x7fc48009e7d0 2019-12-05T06:30:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.172+0000 7fc4852f3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc480131ff0 con 0x7fc4801302f0 2019-12-05T06:30:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.172+0000 7fc47f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4801302f0 0x7fc48009d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:24.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.172+0000 7fc47f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc4801302f0 0x7fc48009d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54834/0 (socket says 172.21.15.16:54834) 2019-12-05T06:30:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.172+0000 7fc47f7fe700 1 -- 172.21.15.16:0/3317862932 learned_addr learned my addr 172.21.15.16:0/3317862932 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc47f7fe700 1 -- 172.21.15.16:0/3317862932 >> v1:172.21.15.76:6789/0 conn(0x7fc48009e7d0 legacy=0x7fc48009dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc477fff700 1 -- 172.21.15.16:0/3317862932 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5468999 0 0) 0x7fc468000eb0 con 0x7fc48009e7d0 2019-12-05T06:30:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc47f7fe700 1 -- 172.21.15.16:0/3317862932 >> v1:172.21.15.16:6789/0 conn(0x7fc480156e70 legacy=0x7fc480159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:24.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc47f7fe700 1 -- 172.21.15.16:0/3317862932 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc480132680 con 0x7fc4801302f0 2019-12-05T06:30:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc47f7fe700 1 --2- 172.21.15.16:0/3317862932 >> v2:172.21.15.16:3300/0 conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc47f7fe700 1 --2- 172.21.15.16:0/3317862932 >> v2:172.21.15.16:3300/0 conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc47f7fe700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=READY pgs=886 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc47f7fe700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=THROTTLE_DONE pgs=886 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc477fff700 1 -- 172.21.15.16:0/3317862932 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc478000ed0 con 0x7fc4801302f0 2019-12-05T06:30:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc47f7fe700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=THROTTLE_DONE pgs=886 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc47f7fe700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=THROTTLE_DONE pgs=886 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc477fff700 1 -- 172.21.15.16:0/3317862932 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc478009db0 con 0x7fc4801302f0 2019-12-05T06:30:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc4852f3700 1 -- 172.21.15.16:0/3317862932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc480132680 con 0x7fc4801302f0 2019-12-05T06:30:24.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.173+0000 7fc477fff700 1 -- 172.21.15.16:0/3317862932 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc47800a480 con 0x7fc4801302f0 2019-12-05T06:30:24.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.174+0000 7fc4852f3700 1 -- 172.21.15.16:0/3317862932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc480132680 con 0x7fc4801302f0 2019-12-05T06:30:24.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.175+0000 7fc47f7fe700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=THROTTLE_DONE pgs=886 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.176+0000 7fc477fff700 1 -- 172.21.15.16:0/3317862932 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc4780099b0 con 0x7fc4801302f0 2019-12-05T06:30:24.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.176+0000 7fc47f7fe700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=THROTTLE_DONE pgs=886 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.176+0000 7fc477fff700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc464023630 0x7fc464025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:24.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.176+0000 7fc477fff700 1 -- 172.21.15.16:0/3317862932 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7fc4780462a0 con 0x7fc4801302f0 2019-12-05T06:30:24.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.176+0000 7fc47effd700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc464023630 0x7fc464025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:24.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.177+0000 7fc47effd700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc464023630 0x7fc464025ac0 crc :-1 s=READY pgs=384 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:24.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.190+0000 7fc4852f3700 1 -- 172.21.15.16:0/3317862932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc480132680 con 0x7fc4801302f0 2019-12-05T06:30:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.197+0000 7fc47f7fe700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=THROTTLE_DONE pgs=886 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.197+0000 7fc477fff700 1 -- 172.21.15.16:0/3317862932 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc47802b2c0 con 0x7fc4801302f0 2019-12-05T06:30:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.459+0000 7fc4852f3700 1 -- 172.21.15.16:0/3317862932 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc480132680 con 0x7fc4801302f0 2019-12-05T06:30:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.460+0000 7fc47f7fe700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=THROTTLE_DONE pgs=886 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.460+0000 7fc477fff700 1 -- 172.21.15.16:0/3317862932 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fc47800a830 con 0x7fc4801302f0 2019-12-05T06:30:24.454 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2019-12-05T06:30:24.455 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:24.455 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:22.445859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.462+0000 7fc4852f3700 1 -- 172.21.15.16:0/3317862932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc464023630 msgr2=0x7fc464025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:24.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.462+0000 7fc4852f3700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc464023630 0x7fc464025ac0 crc :-1 s=READY pgs=384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.462+0000 7fc4852f3700 1 -- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 msgr2=0x7fc48009d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.462+0000 7fc4852f3700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 0x7fc48009d800 secure :-1 s=READY pgs=886 cs=0 l=1 rx=0x7fc4780107d0 tx=0x7fc478010b70).stop 2019-12-05T06:30:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.463+0000 7fc4852f3700 1 -- 172.21.15.16:0/3317862932 shutdown_connections 2019-12-05T06:30:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.463+0000 7fc4852f3700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc464023630 0x7fc464025ac0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.463+0000 7fc4852f3700 1 --2- 172.21.15.16:0/3317862932 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc4801302f0 0x7fc48009d800 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:24.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.463+0000 7fc4852f3700 1 -- 172.21.15.16:0/3317862932 shutdown_connections 2019-12-05T06:30:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.463+0000 7fc4852f3700 1 -- 172.21.15.16:0/3317862932 wait complete. 2019-12-05T06:30:24.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:24.463+0000 7fc4852f3700 1 -- 172.21.15.16:0/3317862932 >> 172.21.15.16:0/3317862932 conn(0x7fc4800a2850 msgr2=0x7fc48006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:24.468 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:30:26.469 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.767+0000 7fa2bad75700 1 Processor -- start 2019-12-05T06:30:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.768+0000 7fa2bad75700 1 -- start start 2019-12-05T06:30:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.770+0000 7fa2bad75700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2b408d2c0 0x7fa2b408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.770+0000 7fa2bad75700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa2b408fef0 0x7fa2b408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.770+0000 7fa2bad75700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b40958e0 con 0x7fa2b408c6d0 2019-12-05T06:30:26.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.770+0000 7fa2bad75700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b408df90 con 0x7fa2b408fef0 2019-12-05T06:30:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.770+0000 7fa2bad75700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2b40a3f70 con 0x7fa2b408d2c0 2019-12-05T06:30:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.770+0000 7fa2b9312700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa2b408c6d0 0x7fa2b408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40016/0 (socket says 172.21.15.16:40016) 2019-12-05T06:30:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.770+0000 7fa2b9312700 1 -- 172.21.15.16:0/1196043859 learned_addr learned my addr 172.21.15.16:0/1196043859 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:30:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.770+0000 7fa2b9b13700 1 --2- 172.21.15.16:0/1196043859 >> v2:172.21.15.16:3300/0 conn(0x7fa2b408fef0 0x7fa2b408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:26.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.770+0000 7fa2b8b11700 1 --2- 172.21.15.16:0/1196043859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2b408d2c0 0x7fa2b408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.771+0000 7fa2b3fff700 1 -- 172.21.15.16:0/1196043859 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316358581 0 0) 0x7fa2ac0014d0 con 0x7fa2b408c6d0 2019-12-05T06:30:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.771+0000 7fa2b3fff700 1 -- 172.21.15.16:0/1196043859 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29c002e80 con 0x7fa2b408c6d0 2019-12-05T06:30:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.771+0000 7fa2b9b13700 1 -- 172.21.15.16:0/1196043859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2b408d2c0 msgr2=0x7fa2b408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.771+0000 7fa2b9b13700 1 --2- 172.21.15.16:0/1196043859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2b408d2c0 0x7fa2b408ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:26.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.771+0000 7fa2b9b13700 1 -- 172.21.15.16:0/1196043859 >> v1:172.21.15.16:6789/0 conn(0x7fa2b408c6d0 legacy=0x7fa2b408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.771+0000 7fa2b9b13700 1 -- 172.21.15.16:0/1196043859 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b408e620 con 0x7fa2b408fef0 2019-12-05T06:30:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.771+0000 7fa2b8b11700 1 --2- 172.21.15.16:0/1196043859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2b408d2c0 0x7fa2b408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.771+0000 7fa2b9b13700 1 --2- 172.21.15.16:0/1196043859 >> v2:172.21.15.16:3300/0 conn(0x7fa2b408fef0 0x7fa2b408f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa2a400af50 tx=0x7fa2a400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.772+0000 7fa2b9b13700 1 --2- 172.21.15.16:0/1196043859 >> v2:172.21.15.16:3300/0 conn(0x7fa2b408fef0 0x7fa2b408f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa2a400af50 tx=0x7fa2a400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.772+0000 7fa2b9b13700 1 --2- 172.21.15.16:0/1196043859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b408fef0 0x7fa2b408f460 secure :-1 s=READY pgs=889 cs=0 l=1 rx=0x7fa2a400af50 tx=0x7fa2a400b760).ready entity=mon.0 client_cookie=12e6d91217cf4a3e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:26.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.772+0000 7fa2b9b13700 1 --2- 172.21.15.16:0/1196043859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b408fef0 0x7fa2b408f460 secure :-1 s=THROTTLE_DONE pgs=889 cs=0 l=1 rx=0x7fa2a400af50 tx=0x7fa2a400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.772+0000 7fa2b3fff700 1 -- 172.21.15.16:0/1196043859 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2a4009ba0 con 0x7fa2b408fef0 2019-12-05T06:30:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.772+0000 7fa2b9b13700 1 --2- 172.21.15.16:0/1196043859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b408fef0 0x7fa2b408f460 secure :-1 s=THROTTLE_DONE pgs=889 cs=0 l=1 rx=0x7fa2a400af50 tx=0x7fa2a400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.772+0000 7fa2b9b13700 1 --2- 172.21.15.16:0/1196043859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b408fef0 0x7fa2b408f460 secure :-1 s=THROTTLE_DONE pgs=889 cs=0 l=1 rx=0x7fa2a400af50 tx=0x7fa2a400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.772+0000 7fa2b3fff700 1 -- 172.21.15.16:0/1196043859 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2a40097c0 con 0x7fa2b408fef0 2019-12-05T06:30:26.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.772+0000 7fa2b3fff700 1 -- 172.21.15.16:0/1196043859 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2a4000bc0 con 0x7fa2b408fef0 2019-12-05T06:30:26.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.772+0000 7fa2bad75700 1 -- 172.21.15.16:0/1196043859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b408fef0 msgr2=0x7fa2b408f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:26.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.772+0000 7fa2bad75700 1 --2- 172.21.15.16:0/1196043859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b408fef0 0x7fa2b408f460 secure :-1 s=READY pgs=889 cs=0 l=1 rx=0x7fa2a400af50 tx=0x7fa2a400b760).stop 2019-12-05T06:30:26.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.774+0000 7fa2bad75700 1 -- 172.21.15.16:0/1196043859 shutdown_connections 2019-12-05T06:30:26.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.774+0000 7fa2bad75700 1 --2- 172.21.15.16:0/1196043859 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b408fef0 0x7fa2b408f460 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:26.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.774+0000 7fa2bad75700 1 --2- 172.21.15.16:0/1196043859 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa2b408d2c0 0x7fa2b408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:26.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.774+0000 7fa2bad75700 1 -- 172.21.15.16:0/1196043859 shutdown_connections 2019-12-05T06:30:26.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.774+0000 7fa2bad75700 1 -- 172.21.15.16:0/1196043859 wait complete. 2019-12-05T06:30:26.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.774+0000 7fa2bad75700 1 -- 172.21.15.16:0/1196043859 >> 172.21.15.16:0/1196043859 conn(0x7fa2b4061940 msgr2=0x7fa2b4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.780+0000 7fa2bad75700 1 Processor -- start 2019-12-05T06:30:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.780+0000 7fa2bad75700 1 -- start start 2019-12-05T06:30:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.782+0000 7fa2bad75700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa2b414d400 0x7fa2b414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:26.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.782+0000 7fa2bad75700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b4098da0 con 0x7fa2b4132d00 2019-12-05T06:30:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.782+0000 7fa2bad75700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2b409f0a0 con 0x7fa2b4132530 2019-12-05T06:30:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.782+0000 7fa2bad75700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2b409bb50 con 0x7fa2b414d400 2019-12-05T06:30:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.782+0000 7fa2b9312700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa2b414d400 0x7fa2b414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.783+0000 7fa2b9312700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa2b414d400 0x7fa2b414d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54854/0 (socket says 172.21.15.16:54854) 2019-12-05T06:30:26.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.783+0000 7fa2b9312700 1 -- 172.21.15.16:0/2469281157 learned_addr learned my addr 172.21.15.16:0/2469281157 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.783+0000 7fa2b9312700 1 -- 172.21.15.16:0/2469281157 >> v1:172.21.15.76:6789/0 conn(0x7fa2b4132530 legacy=0x7fa2b4157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.783+0000 7fa2b9312700 1 -- 172.21.15.16:0/2469281157 >> v1:172.21.15.16:6789/0 conn(0x7fa2b4132d00 legacy=0x7fa2b4131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.783+0000 7fa2b9312700 1 -- 172.21.15.16:0/2469281157 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2b409b0f0 con 0x7fa2b414d400 2019-12-05T06:30:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.783+0000 7fa2b1ffb700 1 -- 172.21.15.16:0/2469281157 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453045220 0 0) 0x7fa2a400b3d0 con 0x7fa2b4132530 2019-12-05T06:30:26.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.783+0000 7fa2b9312700 1 --2- 172.21.15.16:0/2469281157 >> v2:172.21.15.16:3300/0 conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.784+0000 7fa2b9312700 1 --2- 172.21.15.16:0/2469281157 >> v2:172.21.15.16:3300/0 conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.784+0000 7fa2b9312700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=READY pgs=891 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:26.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.784+0000 7fa2b9312700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=THROTTLE_DONE pgs=891 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.784+0000 7fa2b1ffb700 1 -- 172.21.15.16:0/2469281157 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2ac001010 con 0x7fa2b414d400 2019-12-05T06:30:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.784+0000 7fa2b9312700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=THROTTLE_DONE pgs=891 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.784+0000 7fa2b9312700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=THROTTLE_DONE pgs=891 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.784+0000 7fa2b1ffb700 1 -- 172.21.15.16:0/2469281157 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ac001440 con 0x7fa2b414d400 2019-12-05T06:30:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.784+0000 7fa2bad75700 1 -- 172.21.15.16:0/2469281157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2b409b0f0 con 0x7fa2b414d400 2019-12-05T06:30:26.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.784+0000 7fa2b1ffb700 1 -- 172.21.15.16:0/2469281157 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2ac001b10 con 0x7fa2b414d400 2019-12-05T06:30:26.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.784+0000 7fa2bad75700 1 -- 172.21.15.16:0/2469281157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2b409b0f0 con 0x7fa2b414d400 2019-12-05T06:30:26.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.785+0000 7fa2b9312700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=THROTTLE_DONE pgs=891 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.786+0000 7fa2b1ffb700 1 -- 172.21.15.16:0/2469281157 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa2ac001010 con 0x7fa2b414d400 2019-12-05T06:30:26.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.786+0000 7fa2b9312700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=THROTTLE_DONE pgs=891 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.786+0000 7fa2b1ffb700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa294023630 0x7fa294025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:26.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.786+0000 7fa2b1ffb700 1 -- 172.21.15.16:0/2469281157 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7fa2ac046420 con 0x7fa2b414d400 2019-12-05T06:30:26.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.786+0000 7fa2b8b11700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa294023630 0x7fa294025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:26.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.787+0000 7fa2b8b11700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa294023630 0x7fa294025ac0 crc :-1 s=READY pgs=385 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:26.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.799+0000 7fa2bad75700 1 -- 172.21.15.16:0/2469281157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2b409b0f0 con 0x7fa2b414d400 2019-12-05T06:30:26.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.808+0000 7fa2b9312700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=THROTTLE_DONE pgs=891 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:26.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:26.808+0000 7fa2b1ffb700 1 -- 172.21.15.16:0/2469281157 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa2ac007920 con 0x7fa2b414d400 2019-12-05T06:30:27.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.078+0000 7fa2bad75700 1 -- 172.21.15.16:0/2469281157 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2b409b0f0 con 0x7fa2b414d400 2019-12-05T06:30:27.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.079+0000 7fa2b9312700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=THROTTLE_DONE pgs=891 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.079+0000 7fa2b1ffb700 1 -- 172.21.15.16:0/2469281157 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fa2ac001010 con 0x7fa2b414d400 2019-12-05T06:30:27.073 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2019-12-05T06:30:27.073 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:27.073 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:22.445859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.081+0000 7fa2bad75700 1 -- 172.21.15.16:0/2469281157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa294023630 msgr2=0x7fa294025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.082+0000 7fa2bad75700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa294023630 0x7fa294025ac0 crc :-1 s=READY pgs=385 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.082+0000 7fa2bad75700 1 -- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 msgr2=0x7fa2b414d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:27.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.082+0000 7fa2bad75700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 0x7fa2b414d7f0 secure :-1 s=READY pgs=891 cs=0 l=1 rx=0x7fa2ac010760 tx=0x7fa2ac010b00).stop 2019-12-05T06:30:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.082+0000 7fa2bad75700 1 -- 172.21.15.16:0/2469281157 shutdown_connections 2019-12-05T06:30:27.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.082+0000 7fa2bad75700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa294023630 0x7fa294025ac0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.082+0000 7fa2bad75700 1 --2- 172.21.15.16:0/2469281157 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa2b414d400 0x7fa2b414d7f0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.082+0000 7fa2bad75700 1 -- 172.21.15.16:0/2469281157 shutdown_connections 2019-12-05T06:30:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.082+0000 7fa2bad75700 1 -- 172.21.15.16:0/2469281157 wait complete. 2019-12-05T06:30:27.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:27.082+0000 7fa2bad75700 1 -- 172.21.15.16:0/2469281157 >> 172.21.15.16:0/2469281157 conn(0x7fa2b40714f0 msgr2=0x7fa2b409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:27.087 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:30:29.089 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:29.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.340+0000 7fcc896fc700 1 Processor -- start 2019-12-05T06:30:29.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.341+0000 7fcc896fc700 1 -- start start 2019-12-05T06:30:29.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.341+0000 7fcc896fc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcc8408c6d0 0x7fcc8408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:29.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.341+0000 7fcc896fc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8408fef0 0x7fcc8408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:29.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.341+0000 7fcc896fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc8408ef20 0x7fcc8408f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:29.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.341+0000 7fcc896fc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8408df90 con 0x7fcc8408ef20 2019-12-05T06:30:29.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.341+0000 7fcc896fc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc840a3f70 con 0x7fcc8408c6d0 2019-12-05T06:30:29.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.341+0000 7fcc896fc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc8408d830 con 0x7fcc8408fef0 2019-12-05T06:30:29.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc8408ef20 0x7fcc8408f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc8408ef20 0x7fcc8408f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54864/0 (socket says 172.21.15.16:54864) 2019-12-05T06:30:29.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 -- 172.21.15.16:0/1908276678 learned_addr learned my addr 172.21.15.16:0/1908276678 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:29.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc837fe700 1 --2- 172.21.15.16:0/1908276678 >> v2:172.21.15.76:3300/0 conn(0x7fcc8408c6d0 0x7fcc8408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1908276678 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8408fef0 0x7fcc8408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 -- 172.21.15.16:0/1908276678 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8408fef0 msgr2=0x7fcc8408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:29.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 --2- 172.21.15.16:0/1908276678 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8408fef0 0x7fcc8408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:29.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 -- 172.21.15.16:0/1908276678 >> v2:172.21.15.76:3300/0 conn(0x7fcc8408c6d0 msgr2=0x7fcc8408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:29.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 --2- 172.21.15.16:0/1908276678 >> v2:172.21.15.76:3300/0 conn(0x7fcc8408c6d0 0x7fcc8408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:29.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 -- 172.21.15.16:0/1908276678 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8408d540 con 0x7fcc8408ef20 2019-12-05T06:30:29.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc837fe700 1 --2- 172.21.15.16:0/1908276678 >> v2:172.21.15.76:3300/0 conn(0x7fcc8408c6d0 0x7fcc8408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:30:29.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1908276678 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8408fef0 0x7fcc8408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:30:29.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 --2- 172.21.15.16:0/1908276678 >> v2:172.21.15.16:3300/0 conn(0x7fcc8408ef20 0x7fcc8408f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc7c00fab0 tx=0x7fcc7c00fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 --2- 172.21.15.16:0/1908276678 >> v2:172.21.15.16:3300/0 conn(0x7fcc8408ef20 0x7fcc8408f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc7c00fab0 tx=0x7fcc7c00fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.342+0000 7fcc83fff700 1 --2- 172.21.15.16:0/1908276678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8408ef20 0x7fcc8408f3c0 secure :-1 s=READY pgs=892 cs=0 l=1 rx=0x7fcc7c00fab0 tx=0x7fcc7c00fe50).ready entity=mon.0 client_cookie=6b41b696a16f8144 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:29.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.343+0000 7fcc83fff700 1 --2- 172.21.15.16:0/1908276678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8408ef20 0x7fcc8408f3c0 secure :-1 s=THROTTLE_DONE pgs=892 cs=0 l=1 rx=0x7fcc7c00fab0 tx=0x7fcc7c00fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.343+0000 7fcc83fff700 1 --2- 172.21.15.16:0/1908276678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8408ef20 0x7fcc8408f3c0 secure :-1 s=THROTTLE_DONE pgs=892 cs=0 l=1 rx=0x7fcc7c00fab0 tx=0x7fcc7c00fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.343+0000 7fcc83fff700 1 --2- 172.21.15.16:0/1908276678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8408ef20 0x7fcc8408f3c0 secure :-1 s=THROTTLE_DONE pgs=892 cs=0 l=1 rx=0x7fcc7c00fab0 tx=0x7fcc7c00fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.343+0000 7fcc827fc700 1 -- 172.21.15.16:0/1908276678 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc7c001460 con 0x7fcc8408ef20 2019-12-05T06:30:29.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.343+0000 7fcc827fc700 1 -- 172.21.15.16:0/1908276678 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc7c000f10 con 0x7fcc8408ef20 2019-12-05T06:30:29.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.343+0000 7fcc827fc700 1 -- 172.21.15.16:0/1908276678 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc7c001b40 con 0x7fcc8408ef20 2019-12-05T06:30:29.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.343+0000 7fcc896fc700 1 -- 172.21.15.16:0/1908276678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8408ef20 msgr2=0x7fcc8408f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:29.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.343+0000 7fcc896fc700 1 --2- 172.21.15.16:0/1908276678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8408ef20 0x7fcc8408f3c0 secure :-1 s=READY pgs=892 cs=0 l=1 rx=0x7fcc7c00fab0 tx=0x7fcc7c00fe50).stop 2019-12-05T06:30:29.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.345+0000 7fcc896fc700 1 -- 172.21.15.16:0/1908276678 shutdown_connections 2019-12-05T06:30:29.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.345+0000 7fcc896fc700 1 --2- 172.21.15.16:0/1908276678 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc8408ef20 0x7fcc8408f3c0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.345+0000 7fcc896fc700 1 --2- 172.21.15.16:0/1908276678 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8408fef0 0x7fcc8408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:29.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.345+0000 7fcc896fc700 1 --2- 172.21.15.16:0/1908276678 >> v2:172.21.15.76:3300/0 conn(0x7fcc8408c6d0 0x7fcc8408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:29.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.345+0000 7fcc896fc700 1 -- 172.21.15.16:0/1908276678 shutdown_connections 2019-12-05T06:30:29.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.345+0000 7fcc896fc700 1 -- 172.21.15.16:0/1908276678 wait complete. 2019-12-05T06:30:29.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.345+0000 7fcc896fc700 1 -- 172.21.15.16:0/1908276678 >> 172.21.15.16:0/1908276678 conn(0x7fcc84061940 msgr2=0x7fcc84096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:29.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.348+0000 7fcc896fc700 1 Processor -- start 2019-12-05T06:30:29.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.348+0000 7fcc896fc700 1 -- start start 2019-12-05T06:30:29.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc896fc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcc84088c20 0x7fcc84089010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc896fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc84130570 0x7fcc84089550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc896fc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8412f5a0 0x7fcc8412fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc896fc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8408a920 con 0x7fcc84130570 2019-12-05T06:30:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc896fc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc8406ed20 con 0x7fcc84088c20 2019-12-05T06:30:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc896fc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc840708b0 con 0x7fcc8412f5a0 2019-12-05T06:30:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc82ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc84130570 0x7fcc84089550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc82ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc84130570 0x7fcc84089550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54868/0 (socket says 172.21.15.16:54868) 2019-12-05T06:30:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc82ffd700 1 -- 172.21.15.16:0/1157457544 learned_addr learned my addr 172.21.15.16:0/1157457544 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc83fff700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8412f5a0 0x7fcc8412fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc837fe700 1 --2- 172.21.15.16:0/1157457544 >> v2:172.21.15.76:3300/0 conn(0x7fcc84088c20 0x7fcc84089010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.349+0000 7fcc82ffd700 1 -- 172.21.15.16:0/1157457544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8412f5a0 msgr2=0x7fcc8412fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8412f5a0 0x7fcc8412fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc82ffd700 1 -- 172.21.15.16:0/1157457544 >> v2:172.21.15.76:3300/0 conn(0x7fcc84088c20 msgr2=0x7fcc84089010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:29.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> v2:172.21.15.76:3300/0 conn(0x7fcc84088c20 0x7fcc84089010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc82ffd700 1 -- 172.21.15.16:0/1157457544 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc84089a90 con 0x7fcc84130570 2019-12-05T06:30:29.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> v2:172.21.15.16:3300/0 conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> v2:172.21.15.16:3300/0 conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=READY pgs=893 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:29.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc7bfff700 1 -- 172.21.15.16:0/1157457544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc74000bc0 con 0x7fcc84130570 2019-12-05T06:30:29.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc7bfff700 1 -- 172.21.15.16:0/1157457544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc74009550 con 0x7fcc84130570 2019-12-05T06:30:29.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc7bfff700 1 -- 172.21.15.16:0/1157457544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc74001240 con 0x7fcc84130570 2019-12-05T06:30:29.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc896fc700 1 -- 172.21.15.16:0/1157457544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc84089a90 con 0x7fcc84130570 2019-12-05T06:30:29.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.350+0000 7fcc896fc700 1 -- 172.21.15.16:0/1157457544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc84089a90 con 0x7fcc84130570 2019-12-05T06:30:29.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.351+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.352+0000 7fcc7bfff700 1 -- 172.21.15.16:0/1157457544 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcc74000940 con 0x7fcc84130570 2019-12-05T06:30:29.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.352+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.352+0000 7fcc7bfff700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc68023630 0x7fcc68025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:29.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.353+0000 7fcc7bfff700 1 -- 172.21.15.16:0/1157457544 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7fcc74045ce0 con 0x7fcc84130570 2019-12-05T06:30:29.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.353+0000 7fcc837fe700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc68023630 0x7fcc68025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:29.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.353+0000 7fcc837fe700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc68023630 0x7fcc68025ac0 crc :-1 s=READY pgs=386 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:29.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.356+0000 7fcc896fc700 1 -- 172.21.15.16:0/1157457544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc840958e0 con 0x7fcc84130570 2019-12-05T06:30:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.364+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.365+0000 7fcc7bfff700 1 -- 172.21.15.16:0/1157457544 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcc740379f0 con 0x7fcc84130570 2019-12-05T06:30:29.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.605+0000 7fcc896fc700 1 -- 172.21.15.16:0/1157457544 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc840958e0 con 0x7fcc84130570 2019-12-05T06:30:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.605+0000 7fcc82ffd700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=THROTTLE_DONE pgs=893 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.605+0000 7fcc7bfff700 1 -- 172.21.15.16:0/1157457544 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fcc740379f0 con 0x7fcc84130570 2019-12-05T06:30:29.600 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2019-12-05T06:30:29.601 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:29.601 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:22.445859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.608+0000 7fcc896fc700 1 -- 172.21.15.16:0/1157457544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc68023630 msgr2=0x7fcc68025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.608+0000 7fcc896fc700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc68023630 0x7fcc68025ac0 crc :-1 s=READY pgs=386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.608+0000 7fcc896fc700 1 -- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 msgr2=0x7fcc84089550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:29.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.608+0000 7fcc896fc700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 0x7fcc84089550 secure :-1 s=READY pgs=893 cs=0 l=1 rx=0x7fcc74010430 tx=0x7fcc740102b0).stop 2019-12-05T06:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.608+0000 7fcc896fc700 1 -- 172.21.15.16:0/1157457544 shutdown_connections 2019-12-05T06:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.608+0000 7fcc896fc700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc8412f5a0 0x7fcc8412fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.608+0000 7fcc896fc700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc68023630 0x7fcc68025ac0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.608+0000 7fcc896fc700 1 --2- 172.21.15.16:0/1157457544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc84130570 0x7fcc84089550 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.608+0000 7fcc896fc700 1 --2- 172.21.15.16:0/1157457544 >> v2:172.21.15.76:3300/0 conn(0x7fcc84088c20 0x7fcc84089010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:29.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.609+0000 7fcc896fc700 1 -- 172.21.15.16:0/1157457544 shutdown_connections 2019-12-05T06:30:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.609+0000 7fcc896fc700 1 -- 172.21.15.16:0/1157457544 wait complete. 2019-12-05T06:30:29.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:29.609+0000 7fcc896fc700 1 -- 172.21.15.16:0/1157457544 >> 172.21.15.16:0/1157457544 conn(0x7fcc8409de30 msgr2=0x7fcc84070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:29.618 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:30:31.619 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.865+0000 7fac9cec8700 1 Processor -- start 2019-12-05T06:30:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.865+0000 7fac9cec8700 1 -- start start 2019-12-05T06:30:31.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.867+0000 7fac9cec8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac9808d2c0 0x7fac9808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.867+0000 7fac9cec8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac9808fef0 0x7fac9808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.867+0000 7fac9cec8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac980958e0 con 0x7fac9808c6d0 2019-12-05T06:30:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.867+0000 7fac9cec8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9808df90 con 0x7fac9808d2c0 2019-12-05T06:30:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.867+0000 7fac9cec8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac980a3f70 con 0x7fac9808fef0 2019-12-05T06:30:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.867+0000 7fac9659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac9808d2c0 0x7fac9808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.868+0000 7fac9659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fac9808d2c0 0x7fac9808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54880/0 (socket says 172.21.15.16:54880) 2019-12-05T06:30:31.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.868+0000 7fac9659c700 1 -- 172.21.15.16:0/3228108758 learned_addr learned my addr 172.21.15.16:0/3228108758 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.868+0000 7fac9759e700 1 --2- 172.21.15.16:0/3228108758 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac9808fef0 0x7fac9808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.868+0000 7fac9659c700 1 -- 172.21.15.16:0/3228108758 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac9808fef0 msgr2=0x7fac9808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.868+0000 7fac9659c700 1 --2- 172.21.15.16:0/3228108758 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac9808fef0 0x7fac9808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.868+0000 7fac9659c700 1 -- 172.21.15.16:0/3228108758 >> v1:172.21.15.76:6789/0 conn(0x7fac9808c6d0 legacy=0x7fac9808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.868+0000 7fac9659c700 1 -- 172.21.15.16:0/3228108758 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9808e620 con 0x7fac9808d2c0 2019-12-05T06:30:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.868+0000 7fac95d9b700 1 -- 172.21.15.16:0/3228108758 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692525348 0 0) 0x7fac90001400 con 0x7fac9808c6d0 2019-12-05T06:30:31.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.868+0000 7fac9659c700 1 --2- 172.21.15.16:0/3228108758 >> v2:172.21.15.16:3300/0 conn(0x7fac9808d2c0 0x7fac9808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fac8000f6c0 tx=0x7fac8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.869+0000 7fac9659c700 1 --2- 172.21.15.16:0/3228108758 >> v2:172.21.15.16:3300/0 conn(0x7fac9808d2c0 0x7fac9808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fac8000f6c0 tx=0x7fac8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.869+0000 7fac9659c700 1 --2- 172.21.15.16:0/3228108758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9808d2c0 0x7fac9808ef20 secure :-1 s=READY pgs=894 cs=0 l=1 rx=0x7fac8000f6c0 tx=0x7fac8000fa60).ready entity=mon.0 client_cookie=d9623f4b1ccd5dcc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.869+0000 7fac9659c700 1 --2- 172.21.15.16:0/3228108758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9808d2c0 0x7fac9808ef20 secure :-1 s=THROTTLE_DONE pgs=894 cs=0 l=1 rx=0x7fac8000f6c0 tx=0x7fac8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.869+0000 7fac95d9b700 1 -- 172.21.15.16:0/3228108758 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac80000960 con 0x7fac9808d2c0 2019-12-05T06:30:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.869+0000 7fac9659c700 1 --2- 172.21.15.16:0/3228108758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9808d2c0 0x7fac9808ef20 secure :-1 s=THROTTLE_DONE pgs=894 cs=0 l=1 rx=0x7fac8000f6c0 tx=0x7fac8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.869+0000 7fac9659c700 1 --2- 172.21.15.16:0/3228108758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9808d2c0 0x7fac9808ef20 secure :-1 s=THROTTLE_DONE pgs=894 cs=0 l=1 rx=0x7fac8000f6c0 tx=0x7fac8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.869+0000 7fac95d9b700 1 -- 172.21.15.16:0/3228108758 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac80000c90 con 0x7fac9808d2c0 2019-12-05T06:30:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.869+0000 7fac95d9b700 1 -- 172.21.15.16:0/3228108758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac800013a0 con 0x7fac9808d2c0 2019-12-05T06:30:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.869+0000 7fac9cec8700 1 -- 172.21.15.16:0/3228108758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9808d2c0 msgr2=0x7fac9808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.869+0000 7fac9cec8700 1 --2- 172.21.15.16:0/3228108758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9808d2c0 0x7fac9808ef20 secure :-1 s=READY pgs=894 cs=0 l=1 rx=0x7fac8000f6c0 tx=0x7fac8000fa60).stop 2019-12-05T06:30:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.871+0000 7fac9cec8700 1 -- 172.21.15.16:0/3228108758 shutdown_connections 2019-12-05T06:30:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.871+0000 7fac9cec8700 1 --2- 172.21.15.16:0/3228108758 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac9808fef0 0x7fac9808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.871+0000 7fac9cec8700 1 --2- 172.21.15.16:0/3228108758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fac9808d2c0 0x7fac9808ef20 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:31.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.871+0000 7fac9cec8700 1 -- 172.21.15.16:0/3228108758 shutdown_connections 2019-12-05T06:30:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.871+0000 7fac9cec8700 1 -- 172.21.15.16:0/3228108758 wait complete. 2019-12-05T06:30:31.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.871+0000 7fac9cec8700 1 -- 172.21.15.16:0/3228108758 >> 172.21.15.16:0/3228108758 conn(0x7fac98061940 msgr2=0x7fac98096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.876+0000 7fac9cec8700 1 Processor -- start 2019-12-05T06:30:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.876+0000 7fac9cec8700 1 -- start start 2019-12-05T06:30:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.877+0000 7fac9cec8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fac9809bbe0 0x7fac981539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.877+0000 7fac9cec8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:31.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.877+0000 7fac9cec8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac98084b90 con 0x7fac98096e90 2019-12-05T06:30:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.877+0000 7fac9cec8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac9809a450 con 0x7fac9809bbe0 2019-12-05T06:30:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.877+0000 7fac9cec8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac98099a20 con 0x7fac98084f30 2019-12-05T06:30:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.878+0000 7fac96d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fac98096e90 0x7fac9809b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40050/0 (socket says 172.21.15.16:40050) 2019-12-05T06:30:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.878+0000 7fac96d9d700 1 -- 172.21.15.16:0/218903060 learned_addr learned my addr 172.21.15.16:0/218903060 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:30:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.878+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.878+0000 7fac9659c700 1 --2- 172.21.15.16:0/218903060 >> v2:172.21.15.76:3300/0 conn(0x7fac9809bbe0 0x7fac981539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:31.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.878+0000 7fac8f7fe700 1 -- 172.21.15.16:0/218903060 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1044555692 0 0) 0x7fac90001640 con 0x7fac98096e90 2019-12-05T06:30:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.878+0000 7fac8f7fe700 1 -- 172.21.15.16:0/218903060 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac7c002e80 con 0x7fac98096e90 2019-12-05T06:30:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.878+0000 7fac9759e700 1 -- 172.21.15.16:0/218903060 >> v2:172.21.15.76:3300/0 conn(0x7fac9809bbe0 msgr2=0x7fac981539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> v2:172.21.15.76:3300/0 conn(0x7fac9809bbe0 0x7fac981539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9759e700 1 -- 172.21.15.16:0/218903060 >> v1:172.21.15.16:6789/0 conn(0x7fac98096e90 legacy=0x7fac9809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9759e700 1 -- 172.21.15.16:0/218903060 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac9809aae0 con 0x7fac98084f30 2019-12-05T06:30:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=READY pgs=210 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac8f7fe700 1 -- 172.21.15.16:0/218903060 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac88009990 con 0x7fac98084f30 2019-12-05T06:30:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac8f7fe700 1 -- 172.21.15.16:0/218903060 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac8800eaa0 con 0x7fac98084f30 2019-12-05T06:30:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9cec8700 1 -- 172.21.15.16:0/218903060 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac9809aae0 con 0x7fac98084f30 2019-12-05T06:30:31.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac8f7fe700 1 -- 172.21.15.16:0/218903060 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac8800ec90 con 0x7fac98084f30 2019-12-05T06:30:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.879+0000 7fac9cec8700 1 -- 172.21.15.16:0/218903060 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac9809aae0 con 0x7fac98084f30 2019-12-05T06:30:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.880+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.882+0000 7fac8f7fe700 1 -- 172.21.15.16:0/218903060 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac8800f630 con 0x7fac98084f30 2019-12-05T06:30:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.882+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.882+0000 7fac8f7fe700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac7c023670 0x7fac7c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:31.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.882+0000 7fac8f7fe700 1 -- 172.21.15.16:0/218903060 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7fac88045d10 con 0x7fac98084f30 2019-12-05T06:30:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.882+0000 7fac96d9d700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac7c023670 0x7fac7c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:31.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.883+0000 7fac96d9d700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac7c023670 0x7fac7c025ae0 crc :-1 s=READY pgs=387 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:31.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.897+0000 7fac9cec8700 1 -- 172.21.15.16:0/218903060 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac9809aae0 con 0x7fac98084f30 2019-12-05T06:30:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.907+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:31.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:31.907+0000 7fac8f7fe700 1 -- 172.21.15.16:0/218903060 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac8800f380 con 0x7fac98084f30 2019-12-05T06:30:32.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.151+0000 7fac9cec8700 1 -- 172.21.15.16:0/218903060 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fac9809aae0 con 0x7fac98084f30 2019-12-05T06:30:32.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.152+0000 7fac9759e700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=THROTTLE_DONE pgs=210 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:32.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.152+0000 7fac8f7fe700 1 -- 172.21.15.16:0/218903060 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fac8800f590 con 0x7fac98084f30 2019-12-05T06:30:32.149 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2019-12-05T06:30:32.149 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:32.149 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:22.445859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:32.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 -- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac7c023670 msgr2=0x7fac7c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:32.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac7c023670 0x7fac7c025ae0 crc :-1 s=READY pgs=387 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:32.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 -- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 msgr2=0x7fac98155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:32.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 secure :-1 s=READY pgs=210 cs=0 l=1 rx=0x7fac88010390 tx=0x7fac880101c0).stop 2019-12-05T06:30:32.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 -- 172.21.15.16:0/218903060 shutdown_connections 2019-12-05T06:30:32.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac7c023670 0x7fac7c025ae0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:32.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 --2- 172.21.15.16:0/218903060 >> v2:172.21.15.76:3300/0 conn(0x7fac9809bbe0 0x7fac981539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:32.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 --2- 172.21.15.16:0/218903060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fac98084f30 0x7fac98155f30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:32.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 -- 172.21.15.16:0/218903060 shutdown_connections 2019-12-05T06:30:32.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 -- 172.21.15.16:0/218903060 wait complete. 2019-12-05T06:30:32.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:32.157+0000 7fac9cec8700 1 -- 172.21.15.16:0/218903060 >> 172.21.15.16:0/218903060 conn(0x7fac980714f0 msgr2=0x7fac9809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:32.162 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:30:34.164 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.454+0000 7f2a3565e700 1 Processor -- start 2019-12-05T06:30:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.455+0000 7f2a3565e700 1 -- start start 2019-12-05T06:30:34.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.456+0000 7f2a3565e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a3008d2c0 0x7f2a3008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.456+0000 7f2a3565e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a3008fef0 0x7f2a3008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.456+0000 7f2a3565e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a300958e0 con 0x7f2a3008c6d0 2019-12-05T06:30:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.456+0000 7f2a3565e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a3008df90 con 0x7f2a3008fef0 2019-12-05T06:30:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.456+0000 7f2a3565e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a300a3f70 con 0x7f2a3008d2c0 2019-12-05T06:30:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.457+0000 7f2a2ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a3008fef0 0x7f2a3008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.457+0000 7f2a2ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a3008fef0 0x7f2a3008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54894/0 (socket says 172.21.15.16:54894) 2019-12-05T06:30:34.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.457+0000 7f2a2ffff700 1 -- 172.21.15.16:0/2746497901 learned_addr learned my addr 172.21.15.16:0/2746497901 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.457+0000 7f2a2f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2a3008c6d0 0x7f2a3008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40058/0 (socket says 172.21.15.16:40058) 2019-12-05T06:30:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.457+0000 7f2a2effd700 1 --2- 172.21.15.16:0/2746497901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a3008d2c0 0x7f2a3008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.457+0000 7f2a2ffff700 1 -- 172.21.15.16:0/2746497901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a3008d2c0 msgr2=0x7f2a3008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.457+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/2746497901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a3008d2c0 0x7f2a3008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.458+0000 7f2a2ffff700 1 -- 172.21.15.16:0/2746497901 >> v1:172.21.15.16:6789/0 conn(0x7f2a3008c6d0 legacy=0x7f2a3008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:34.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.458+0000 7f2a2ffff700 1 -- 172.21.15.16:0/2746497901 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3008e620 con 0x7f2a3008fef0 2019-12-05T06:30:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.458+0000 7f2a2e7fc700 1 -- 172.21.15.16:0/2746497901 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915700620 0 0) 0x7f2a28001480 con 0x7f2a3008c6d0 2019-12-05T06:30:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.458+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/2746497901 >> v2:172.21.15.16:3300/0 conn(0x7f2a3008fef0 0x7f2a3008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2a2000f6b0 tx=0x7f2a2000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.458+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/2746497901 >> v2:172.21.15.16:3300/0 conn(0x7f2a3008fef0 0x7f2a3008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2a2000f6b0 tx=0x7f2a2000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.458+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/2746497901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a3008fef0 0x7f2a3008f460 secure :-1 s=READY pgs=897 cs=0 l=1 rx=0x7f2a2000f6b0 tx=0x7f2a2000fa50).ready entity=mon.0 client_cookie=53b15019ed3fd1f6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.458+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/2746497901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a3008fef0 0x7f2a3008f460 secure :-1 s=THROTTLE_DONE pgs=897 cs=0 l=1 rx=0x7f2a2000f6b0 tx=0x7f2a2000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.458+0000 7f2a2e7fc700 1 -- 172.21.15.16:0/2746497901 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a20000a70 con 0x7f2a3008fef0 2019-12-05T06:30:34.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.458+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/2746497901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a3008fef0 0x7f2a3008f460 secure :-1 s=THROTTLE_DONE pgs=897 cs=0 l=1 rx=0x7f2a2000f6b0 tx=0x7f2a2000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.458+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/2746497901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a3008fef0 0x7f2a3008f460 secure :-1 s=THROTTLE_DONE pgs=897 cs=0 l=1 rx=0x7f2a2000f6b0 tx=0x7f2a2000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.459+0000 7f2a2e7fc700 1 -- 172.21.15.16:0/2746497901 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a200010e0 con 0x7f2a3008fef0 2019-12-05T06:30:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.459+0000 7f2a2e7fc700 1 -- 172.21.15.16:0/2746497901 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a200017b0 con 0x7f2a3008fef0 2019-12-05T06:30:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.459+0000 7f2a3565e700 1 -- 172.21.15.16:0/2746497901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a3008fef0 msgr2=0x7f2a3008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.459+0000 7f2a3565e700 1 --2- 172.21.15.16:0/2746497901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a3008fef0 0x7f2a3008f460 secure :-1 s=READY pgs=897 cs=0 l=1 rx=0x7f2a2000f6b0 tx=0x7f2a2000fa50).stop 2019-12-05T06:30:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.460+0000 7f2a3565e700 1 -- 172.21.15.16:0/2746497901 shutdown_connections 2019-12-05T06:30:34.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.460+0000 7f2a3565e700 1 --2- 172.21.15.16:0/2746497901 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a3008fef0 0x7f2a3008f460 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.460+0000 7f2a3565e700 1 --2- 172.21.15.16:0/2746497901 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a3008d2c0 0x7f2a3008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.461+0000 7f2a3565e700 1 -- 172.21.15.16:0/2746497901 shutdown_connections 2019-12-05T06:30:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.461+0000 7f2a3565e700 1 -- 172.21.15.16:0/2746497901 wait complete. 2019-12-05T06:30:34.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.461+0000 7f2a3565e700 1 -- 172.21.15.16:0/2746497901 >> 172.21.15.16:0/2746497901 conn(0x7f2a30061940 msgr2=0x7f2a30096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:34.466 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.466+0000 7f2a3565e700 1 Processor -- start 2019-12-05T06:30:34.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.467+0000 7f2a3565e700 1 -- start start 2019-12-05T06:30:34.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.469+0000 7f2a3565e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2a3009bbe0 0x7f2a301539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:34.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.469+0000 7f2a3565e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a30084f30 0x7f2a30155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:34.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.469+0000 7f2a3565e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a30084b90 con 0x7f2a30096e90 2019-12-05T06:30:34.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.469+0000 7f2a3565e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a3009a450 con 0x7f2a30084f30 2019-12-05T06:30:34.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.469+0000 7f2a3565e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a30099a20 con 0x7f2a3009bbe0 2019-12-05T06:30:34.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.469+0000 7f2a2ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a30084f30 0x7f2a30155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:34.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.469+0000 7f2a2ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a30084f30 0x7f2a30155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54898/0 (socket says 172.21.15.16:54898) 2019-12-05T06:30:34.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.469+0000 7f2a2ffff700 1 -- 172.21.15.16:0/1893816070 learned_addr learned my addr 172.21.15.16:0/1893816070 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:34.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.469+0000 7f2a2effd700 1 --2- 172.21.15.16:0/1893816070 >> v2:172.21.15.76:3300/0 conn(0x7f2a3009bbe0 0x7f2a301539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:34.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2ffff700 1 -- 172.21.15.16:0/1893816070 >> v2:172.21.15.76:3300/0 conn(0x7f2a3009bbe0 msgr2=0x7f2a301539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:34.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> v2:172.21.15.76:3300/0 conn(0x7f2a3009bbe0 0x7f2a301539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:34.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a27fff700 1 -- 172.21.15.16:0/1893816070 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 297605833 0 0) 0x7f2a28001480 con 0x7f2a30096e90 2019-12-05T06:30:34.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2ffff700 1 -- 172.21.15.16:0/1893816070 >> v1:172.21.15.76:6789/0 conn(0x7f2a30096e90 legacy=0x7f2a3009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:34.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2ffff700 1 -- 172.21.15.16:0/1893816070 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3009aae0 con 0x7f2a30084f30 2019-12-05T06:30:34.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2effd700 1 --2- 172.21.15.16:0/1893816070 >> v2:172.21.15.76:3300/0 conn(0x7f2a3009bbe0 0x7f2a301539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:34.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> v2:172.21.15.16:3300/0 conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> v2:172.21.15.16:3300/0 conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=READY pgs=898 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:34.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a27fff700 1 -- 172.21.15.16:0/1893816070 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a200014a0 con 0x7f2a30084f30 2019-12-05T06:30:34.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.470+0000 7f2a27fff700 1 -- 172.21.15.16:0/1893816070 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a200010c0 con 0x7f2a30084f30 2019-12-05T06:30:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.471+0000 7f2a27fff700 1 -- 172.21.15.16:0/1893816070 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a20001a80 con 0x7f2a30084f30 2019-12-05T06:30:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.471+0000 7f2a3565e700 1 -- 172.21.15.16:0/1893816070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3009aae0 con 0x7f2a30084f30 2019-12-05T06:30:34.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.471+0000 7f2a3565e700 1 -- 172.21.15.16:0/1893816070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3009aae0 con 0x7f2a30084f30 2019-12-05T06:30:34.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.472+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.473+0000 7f2a27fff700 1 -- 172.21.15.16:0/1893816070 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2a200012b0 con 0x7f2a30084f30 2019-12-05T06:30:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.473+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.473+0000 7f2a27fff700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a14023630 0x7f2a14025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.473+0000 7f2a27fff700 1 -- 172.21.15.16:0/1893816070 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7f2a20089df0 con 0x7f2a30084f30 2019-12-05T06:30:34.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.474+0000 7f2a2f7fe700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a14023630 0x7f2a14025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:34.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.474+0000 7f2a2f7fe700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a14023630 0x7f2a14025ac0 crc :-1 s=READY pgs=388 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:34.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.486+0000 7f2a3565e700 1 -- 172.21.15.16:0/1893816070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3009aae0 con 0x7f2a30084f30 2019-12-05T06:30:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.494+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.494+0000 7f2a27fff700 1 -- 172.21.15.16:0/1893816070 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2a20000eb0 con 0x7f2a30084f30 2019-12-05T06:30:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.748+0000 7f2a3565e700 1 -- 172.21.15.16:0/1893816070 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a3009aae0 con 0x7f2a30084f30 2019-12-05T06:30:34.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.749+0000 7f2a2ffff700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:34.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.749+0000 7f2a27fff700 1 -- 172.21.15.16:0/1893816070 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f2a200012b0 con 0x7f2a30084f30 2019-12-05T06:30:34.746 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2019-12-05T06:30:34.746 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:34.746 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:22.445859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:34.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.753+0000 7f2a3565e700 1 -- 172.21.15.16:0/1893816070 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a14023630 msgr2=0x7f2a14025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:34.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.753+0000 7f2a3565e700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a14023630 0x7f2a14025ac0 crc :-1 s=READY pgs=388 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:34.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.753+0000 7f2a3565e700 1 -- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 msgr2=0x7f2a30155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:34.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.753+0000 7f2a3565e700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 0x7f2a30155f30 secure :-1 s=READY pgs=898 cs=0 l=1 rx=0x7f2a200544b0 tx=0x7f2a20054850).stop 2019-12-05T06:30:34.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.753+0000 7f2a3565e700 1 -- 172.21.15.16:0/1893816070 shutdown_connections 2019-12-05T06:30:34.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.754+0000 7f2a3565e700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a14023630 0x7f2a14025ac0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:34.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.754+0000 7f2a3565e700 1 --2- 172.21.15.16:0/1893816070 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a30084f30 0x7f2a30155f30 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:34.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.754+0000 7f2a3565e700 1 --2- 172.21.15.16:0/1893816070 >> v2:172.21.15.76:3300/0 conn(0x7f2a3009bbe0 0x7f2a301539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:34.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.754+0000 7f2a3565e700 1 -- 172.21.15.16:0/1893816070 shutdown_connections 2019-12-05T06:30:34.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.754+0000 7f2a3565e700 1 -- 172.21.15.16:0/1893816070 wait complete. 2019-12-05T06:30:34.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:34.754+0000 7f2a3565e700 1 -- 172.21.15.16:0/1893816070 >> 172.21.15.16:0/1893816070 conn(0x7f2a300714f0 msgr2=0x7f2a3009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:34.759 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:30:35.614 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:30:35.619 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:30:36.760 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.042+0000 7fc20e83a700 1 Processor -- start 2019-12-05T06:30:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.042+0000 7fc20e83a700 1 -- start start 2019-12-05T06:30:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.044+0000 7fc20e83a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 0x7fc20808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.044+0000 7fc20e83a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2080958e0 con 0x7fc20808d2c0 2019-12-05T06:30:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.044+0000 7fc20e83a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2080629e0 con 0x7fc20808c6d0 2019-12-05T06:30:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.044+0000 7fc20e83a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc20808df90 con 0x7fc20808fef0 2019-12-05T06:30:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.044+0000 7fc207fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc20808d2c0 0x7fc20808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40074/0 (socket says 172.21.15.16:40074) 2019-12-05T06:30:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.044+0000 7fc207fff700 1 -- 172.21.15.16:0/1803298508 learned_addr learned my addr 172.21.15.16:0/1803298508 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:30:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.045+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/1803298508 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 0x7fc20808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.045+0000 7fc2077fe700 1 -- 172.21.15.16:0/1803298508 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4095232483 0 0) 0x7fc1f8001150 con 0x7fc20808d2c0 2019-12-05T06:30:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.045+0000 7fc2077fe700 1 -- 172.21.15.16:0/1803298508 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f4002e80 con 0x7fc20808d2c0 2019-12-05T06:30:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.045+0000 7fc2077fe700 1 -- 172.21.15.16:0/1803298508 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2888641042 0 0) 0x7fc1f0001080 con 0x7fc20808c6d0 2019-12-05T06:30:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.045+0000 7fc2077fe700 1 -- 172.21.15.16:0/1803298508 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc1f4004170 con 0x7fc20808c6d0 2019-12-05T06:30:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.045+0000 7fc20d5d8700 1 -- 172.21.15.16:0/1803298508 >> v1:172.21.15.76:6789/0 conn(0x7fc20808c6d0 legacy=0x7fc20808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.045+0000 7fc20d5d8700 1 -- 172.21.15.16:0/1803298508 >> v1:172.21.15.16:6789/0 conn(0x7fc20808d2c0 legacy=0x7fc20808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.045+0000 7fc20d5d8700 1 -- 172.21.15.16:0/1803298508 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20808e620 con 0x7fc20808fef0 2019-12-05T06:30:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/1803298508 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 0x7fc20808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc20000b300 tx=0x7fc20000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/1803298508 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 0x7fc20808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc20000b300 tx=0x7fc20000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/1803298508 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 0x7fc20808f720 secure :-1 s=READY pgs=211 cs=0 l=1 rx=0x7fc20000b300 tx=0x7fc20000bb10).ready entity=mon.2 client_cookie=eb3719db302c4db6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/1803298508 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 0x7fc20808f720 secure :-1 s=THROTTLE_DONE pgs=211 cs=0 l=1 rx=0x7fc20000b300 tx=0x7fc20000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc2077fe700 1 -- 172.21.15.16:0/1803298508 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2000075c0 con 0x7fc20808fef0 2019-12-05T06:30:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/1803298508 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 0x7fc20808f720 secure :-1 s=THROTTLE_DONE pgs=211 cs=0 l=1 rx=0x7fc20000b300 tx=0x7fc20000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/1803298508 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 0x7fc20808f720 secure :-1 s=THROTTLE_DONE pgs=211 cs=0 l=1 rx=0x7fc20000b300 tx=0x7fc20000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc2077fe700 1 -- 172.21.15.16:0/1803298508 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc200000f30 con 0x7fc20808fef0 2019-12-05T06:30:37.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc2077fe700 1 -- 172.21.15.16:0/1803298508 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc20000a1e0 con 0x7fc20808fef0 2019-12-05T06:30:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc20e83a700 1 -- 172.21.15.16:0/1803298508 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 msgr2=0x7fc20808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.046+0000 7fc20e83a700 1 --2- 172.21.15.16:0/1803298508 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 0x7fc20808f720 secure :-1 s=READY pgs=211 cs=0 l=1 rx=0x7fc20000b300 tx=0x7fc20000bb10).stop 2019-12-05T06:30:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.048+0000 7fc20e83a700 1 -- 172.21.15.16:0/1803298508 shutdown_connections 2019-12-05T06:30:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.048+0000 7fc20e83a700 1 --2- 172.21.15.16:0/1803298508 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc20808fef0 0x7fc20808f720 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.048+0000 7fc20e83a700 1 -- 172.21.15.16:0/1803298508 shutdown_connections 2019-12-05T06:30:37.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.048+0000 7fc20e83a700 1 -- 172.21.15.16:0/1803298508 wait complete. 2019-12-05T06:30:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.048+0000 7fc20e83a700 1 -- 172.21.15.16:0/1803298508 >> 172.21.15.16:0/1803298508 conn(0x7fc208061940 msgr2=0x7fc208096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.053+0000 7fc20e83a700 1 Processor -- start 2019-12-05T06:30:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.053+0000 7fc20e83a700 1 -- start start 2019-12-05T06:30:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.055+0000 7fc20e83a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc20814d680 0x7fc20814da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.055+0000 7fc20e83a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc20809d3a0 0x7fc20812bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.055+0000 7fc20e83a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc20812c220 con 0x7fc20809dc10 2019-12-05T06:30:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.055+0000 7fc20e83a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc20812ad50 con 0x7fc20809d3a0 2019-12-05T06:30:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.055+0000 7fc20e83a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc208099bb0 con 0x7fc20814d680 2019-12-05T06:30:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.055+0000 7fc20d5d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc20809d3a0 0x7fc20812bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.055+0000 7fc20d5d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc20809d3a0 0x7fc20812bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54916/0 (socket says 172.21.15.16:54916) 2019-12-05T06:30:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.055+0000 7fc20d5d8700 1 -- 172.21.15.16:0/2166755529 learned_addr learned my addr 172.21.15.16:0/2166755529 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.055+0000 7fc20cdd7700 1 --2- 172.21.15.16:0/2166755529 >> v2:172.21.15.76:3300/0 conn(0x7fc20814d680 0x7fc20814da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc20d5d8700 1 -- 172.21.15.16:0/2166755529 >> v2:172.21.15.76:3300/0 conn(0x7fc20814d680 msgr2=0x7fc20814da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> v2:172.21.15.76:3300/0 conn(0x7fc20814d680 0x7fc20814da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc20d5d8700 1 -- 172.21.15.16:0/2166755529 >> v1:172.21.15.76:6789/0 conn(0x7fc20809dc10 legacy=0x7fc208098180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc20d5d8700 1 -- 172.21.15.16:0/2166755529 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc20812b3e0 con 0x7fc20809d3a0 2019-12-05T06:30:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> v2:172.21.15.16:3300/0 conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> v2:172.21.15.16:3300/0 conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=READY pgs=900 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.056+0000 7fc2057fa700 1 -- 172.21.15.16:0/2166755529 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc20000b360 con 0x7fc20809d3a0 2019-12-05T06:30:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.057+0000 7fc2057fa700 1 -- 172.21.15.16:0/2166755529 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc200009d30 con 0x7fc20809d3a0 2019-12-05T06:30:37.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.057+0000 7fc20e83a700 1 -- 172.21.15.16:0/2166755529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc20812b3e0 con 0x7fc20809d3a0 2019-12-05T06:30:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.057+0000 7fc2057fa700 1 -- 172.21.15.16:0/2166755529 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc20000a400 con 0x7fc20809d3a0 2019-12-05T06:30:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.057+0000 7fc20e83a700 1 -- 172.21.15.16:0/2166755529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc20812b3e0 con 0x7fc20809d3a0 2019-12-05T06:30:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.058+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.058+0000 7fc2057fa700 1 -- 172.21.15.16:0/2166755529 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc200009930 con 0x7fc20809d3a0 2019-12-05T06:30:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.058+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.059+0000 7fc2057fa700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc1ec023630 0x7fc1ec025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.059+0000 7fc2057fa700 1 -- 172.21.15.16:0/2166755529 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(65..65 src has 1..65) v4 ==== 7407+0+0 (secure 0 0 0) 0x7fc200046170 con 0x7fc20809d3a0 2019-12-05T06:30:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.059+0000 7fc20cdd7700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc1ec023630 0x7fc1ec025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:37.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.059+0000 7fc20cdd7700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc1ec023630 0x7fc1ec025ac0 crc :-1 s=READY pgs=389 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:37.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.074+0000 7fc20e83a700 1 -- 172.21.15.16:0/2166755529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc20812b3e0 con 0x7fc20809d3a0 2019-12-05T06:30:37.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.082+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.082+0000 7fc2057fa700 1 -- 172.21.15.16:0/2166755529 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc20000b310 con 0x7fc20809d3a0 2019-12-05T06:30:37.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.335+0000 7fc20e83a700 1 -- 172.21.15.16:0/2166755529 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc20812b3e0 con 0x7fc20809d3a0 2019-12-05T06:30:37.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.336+0000 7fc20d5d8700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:37.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.336+0000 7fc2057fa700 1 -- 172.21.15.16:0/2166755529 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 112 v112) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fc200009930 con 0x7fc20809d3a0 2019-12-05T06:30:37.331 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 112 2019-12-05T06:30:37.331 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:37.332 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":112,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6544,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":112}],"filesystems":[{"mdsmap":{"epoch":111,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:29:22.445859+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":65,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6363},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6363":{"gid":6363,"name":"a","rank":0,"incarnation":108,"state":"up:active","state_seq":17,"addr":"172.21.15.16:6833/1994840831","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1994840831},{"type":"v1","addr":"172.21.15.16:6833","nonce":1994840831}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.338+0000 7fc20e83a700 1 -- 172.21.15.16:0/2166755529 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc1ec023630 msgr2=0x7fc1ec025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.338+0000 7fc20e83a700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc1ec023630 0x7fc1ec025ac0 crc :-1 s=READY pgs=389 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.338+0000 7fc20e83a700 1 -- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 msgr2=0x7fc20812bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.338+0000 7fc20e83a700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 0x7fc20812bce0 secure :-1 s=READY pgs=900 cs=0 l=1 rx=0x7fc2000107c0 tx=0x7fc200010640).stop 2019-12-05T06:30:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.339+0000 7fc20e83a700 1 -- 172.21.15.16:0/2166755529 shutdown_connections 2019-12-05T06:30:37.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.339+0000 7fc20e83a700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc1ec023630 0x7fc1ec025ac0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.339+0000 7fc20e83a700 1 --2- 172.21.15.16:0/2166755529 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc20809d3a0 0x7fc20812bce0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.339+0000 7fc20e83a700 1 --2- 172.21.15.16:0/2166755529 >> v2:172.21.15.76:3300/0 conn(0x7fc20814d680 0x7fc20814da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.339+0000 7fc20e83a700 1 -- 172.21.15.16:0/2166755529 shutdown_connections 2019-12-05T06:30:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.339+0000 7fc20e83a700 1 -- 172.21.15.16:0/2166755529 wait complete. 2019-12-05T06:30:37.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:37.339+0000 7fc20e83a700 1 -- 172.21.15.16:0/2166755529 >> 172.21.15.16:0/2166755529 conn(0x7fc20806d260 msgr2=0x7fc20809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:37.349 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:30:39.350 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:39.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.609+0000 7fd3560f8700 1 Processor -- start 2019-12-05T06:30:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.609+0000 7fd3560f8700 1 -- start start 2019-12-05T06:30:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.610+0000 7fd3560f8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd35008d2c0 0x7fd35008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.610+0000 7fd3560f8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.610+0000 7fd3560f8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3500958e0 con 0x7fd35008c6d0 2019-12-05T06:30:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.610+0000 7fd3560f8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35008df90 con 0x7fd35008d2c0 2019-12-05T06:30:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.610+0000 7fd3560f8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3500a3f70 con 0x7fd35008fef0 2019-12-05T06:30:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd354e96700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd34ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd35008c6d0 0x7fd35008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:33158/0 (socket says 172.21.15.16:33158) 2019-12-05T06:30:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd354e96700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:58912/0 (socket says 172.21.15.16:58912) 2019-12-05T06:30:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd34ffff700 1 -- 172.21.15.16:0/194734347 learned_addr learned my addr 172.21.15.16:0/194734347 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:30:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd3477fe700 1 --2- 172.21.15.16:0/194734347 >> v2:172.21.15.76:3300/0 conn(0x7fd35008d2c0 0x7fd35008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd354e96700 1 -- 172.21.15.16:0/194734347 >> v2:172.21.15.76:3300/0 conn(0x7fd35008d2c0 msgr2=0x7fd35008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd354e96700 1 --2- 172.21.15.16:0/194734347 >> v2:172.21.15.76:3300/0 conn(0x7fd35008d2c0 0x7fd35008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd354e96700 1 -- 172.21.15.16:0/194734347 >> v1:172.21.15.76:6789/0 conn(0x7fd35008c6d0 legacy=0x7fd35008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd34f7fe700 1 -- 172.21.15.16:0/194734347 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973987862 0 0) 0x7fd340001100 con 0x7fd35008c6d0 2019-12-05T06:30:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd354e96700 1 -- 172.21.15.16:0/194734347 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35008e620 con 0x7fd35008fef0 2019-12-05T06:30:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd3477fe700 1 --2- 172.21.15.16:0/194734347 >> v2:172.21.15.76:3300/0 conn(0x7fd35008d2c0 0x7fd35008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.611+0000 7fd354e96700 1 --2- 172.21.15.16:0/194734347 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd34800b300 tx=0x7fd34800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.612+0000 7fd354e96700 1 --2- 172.21.15.16:0/194734347 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd34800b300 tx=0x7fd34800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.612+0000 7fd354e96700 1 --2- 172.21.15.16:0/194734347 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 secure :-1 s=READY pgs=212 cs=0 l=1 rx=0x7fd34800b300 tx=0x7fd34800bb10).ready entity=mon.2 client_cookie=6acd04c848b76f9b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.612+0000 7fd354e96700 1 --2- 172.21.15.16:0/194734347 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fd34800b300 tx=0x7fd34800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.612+0000 7fd34f7fe700 1 -- 172.21.15.16:0/194734347 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd348000f40 con 0x7fd35008fef0 2019-12-05T06:30:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.612+0000 7fd354e96700 1 --2- 172.21.15.16:0/194734347 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fd34800b300 tx=0x7fd34800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.612+0000 7fd354e96700 1 --2- 172.21.15.16:0/194734347 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 secure :-1 s=THROTTLE_DONE pgs=212 cs=0 l=1 rx=0x7fd34800b300 tx=0x7fd34800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.612+0000 7fd34f7fe700 1 -- 172.21.15.16:0/194734347 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348009b70 con 0x7fd35008fef0 2019-12-05T06:30:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.612+0000 7fd34f7fe700 1 -- 172.21.15.16:0/194734347 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd34800a240 con 0x7fd35008fef0 2019-12-05T06:30:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.612+0000 7fd3560f8700 1 -- 172.21.15.16:0/194734347 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 msgr2=0x7fd35008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.612+0000 7fd3560f8700 1 --2- 172.21.15.16:0/194734347 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 secure :-1 s=READY pgs=212 cs=0 l=1 rx=0x7fd34800b300 tx=0x7fd34800bb10).stop 2019-12-05T06:30:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.613+0000 7fd3560f8700 1 -- 172.21.15.16:0/194734347 shutdown_connections 2019-12-05T06:30:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.613+0000 7fd3560f8700 1 --2- 172.21.15.16:0/194734347 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35008fef0 0x7fd35008f460 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:39.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.613+0000 7fd3560f8700 1 --2- 172.21.15.16:0/194734347 >> v2:172.21.15.76:3300/0 conn(0x7fd35008d2c0 0x7fd35008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:39.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.613+0000 7fd3560f8700 1 -- 172.21.15.16:0/194734347 shutdown_connections 2019-12-05T06:30:39.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.614+0000 7fd3560f8700 1 -- 172.21.15.16:0/194734347 wait complete. 2019-12-05T06:30:39.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.614+0000 7fd3560f8700 1 -- 172.21.15.16:0/194734347 >> 172.21.15.16:0/194734347 conn(0x7fd350061940 msgr2=0x7fd350096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:39.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.619+0000 7fd3560f8700 1 Processor -- start 2019-12-05T06:30:39.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.619+0000 7fd3560f8700 1 -- start start 2019-12-05T06:30:39.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.621+0000 7fd3560f8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd35009c110 0x7fd35009e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:39.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.621+0000 7fd3560f8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:39.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.621+0000 7fd3560f8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd35009ca80 con 0x7fd35009f270 2019-12-05T06:30:39.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.621+0000 7fd3560f8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd35012d280 con 0x7fd35009c110 2019-12-05T06:30:39.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.621+0000 7fd3560f8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd35012d550 con 0x7fd35009f7c0 2019-12-05T06:30:39.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.621+0000 7fd3477fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:39.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.621+0000 7fd3477fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:58918/0 (socket says 172.21.15.16:58918) 2019-12-05T06:30:39.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.621+0000 7fd3477fe700 1 -- 172.21.15.16:0/1176145677 learned_addr learned my addr 172.21.15.16:0/1176145677 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:39.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.621+0000 7fd34ffff700 1 --2- 172.21.15.16:0/1176145677 >> v2:172.21.15.76:3300/0 conn(0x7fd35009c110 0x7fd35009e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 -- 172.21.15.16:0/1176145677 >> v2:172.21.15.76:3300/0 conn(0x7fd35009c110 msgr2=0x7fd35009e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 --2- 172.21.15.16:0/1176145677 >> v2:172.21.15.76:3300/0 conn(0x7fd35009c110 0x7fd35009e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 -- 172.21.15.16:0/1176145677 >> v1:172.21.15.16:6789/0 conn(0x7fd35009f270 legacy=0x7fd35012b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 -- 172.21.15.16:0/1176145677 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd35009def0 con 0x7fd35009f7c0 2019-12-05T06:30:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd338012680 tx=0x7fd338012480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd338012680 tx=0x7fd338012480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 secure :-1 s=READY pgs=213 cs=0 l=1 rx=0x7fd338012680 tx=0x7fd338012480).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 secure :-1 s=THROTTLE_DONE pgs=213 cs=0 l=1 rx=0x7fd338012680 tx=0x7fd338012480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd338008700 con 0x7fd35009f7c0 2019-12-05T06:30:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 secure :-1 s=THROTTLE_DONE pgs=213 cs=0 l=1 rx=0x7fd338012680 tx=0x7fd338012480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 secure :-1 s=THROTTLE_DONE pgs=213 cs=0 l=1 rx=0x7fd338012680 tx=0x7fd338012480).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd338007630 con 0x7fd35009f7c0 2019-12-05T06:30:39.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd338008ce0 con 0x7fd35009f7c0 2019-12-05T06:30:39.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3560f8700 1 -- 172.21.15.16:0/1176145677 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd35009def0 con 0x7fd35009f7c0 2019-12-05T06:30:39.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3560f8700 1 -- 172.21.15.16:0/1176145677 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd35009def0 con 0x7fd35009f7c0 2019-12-05T06:30:39.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 -- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 msgr2=0x7fd35009ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:30:39.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 -- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 msgr2=0x7fd35009ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:30:39.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 secure :-1 s=READY pgs=213 cs=0 l=1 rx=0x7fd338012680 tx=0x7fd338012480).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:30:39.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd3477fe700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 secure :-1 s=READY pgs=213 cs=0 l=1 rx=0x7fd338012680 tx=0x7fd338012480).stop 2019-12-05T06:30:39.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 msgr2=0x7fd35009ed30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:30:39.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd35009f7c0 0x7fd35009ed30 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:39.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd334003410 0x7fd334005870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:39.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd334006020 0x7fd3340084f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:39.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:39.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33400b9d0 con 0x7fd334003410 2019-12-05T06:30:39.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd33400bc00 con 0x7fd334006020 2019-12-05T06:30:39.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.622+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd33400be30 con 0x7fd334008d50 2019-12-05T06:30:39.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.623+0000 7fd34ffff700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd334003410 0x7fd334005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:39.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.623+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:39.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.623+0000 7fd3477fe700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd334006020 0x7fd3340084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:39.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.623+0000 7fd354e96700 1 -- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd334006020 msgr2=0x7fd3340084f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:39.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.623+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd334006020 0x7fd3340084f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:39.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.623+0000 7fd354e96700 1 -- 172.21.15.16:0/1176145677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd334003410 msgr2=0x7fd334005870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:39.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.623+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd334003410 0x7fd334005870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.624+0000 7fd354e96700 1 -- 172.21.15.16:0/1176145677 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fd33400c160 con 0x7fd334008d50 2019-12-05T06:30:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.624+0000 7fd34ffff700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd334003410 0x7fd334005870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.624+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd348010640 tx=0x7fd3480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.624+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd348010640 tx=0x7fd3480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.624+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 secure :-1 s=READY pgs=214 cs=0 l=1 rx=0x7fd348010640 tx=0x7fd3480109e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.624+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7fd348010640 tx=0x7fd3480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.624+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7fd348010640 tx=0x7fd3480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.624+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd348009f20 con 0x7fd334008d50 2019-12-05T06:30:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.624+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd348009b40 con 0x7fd334008d50 2019-12-05T06:30:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.625+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7fd348010640 tx=0x7fd3480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.626+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd348009d30 con 0x7fd334008d50 2019-12-05T06:30:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.626+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7fd348010640 tx=0x7fd3480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.626+0000 7fd34d7fa700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd33402b9c0 0x7fd33402ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:39.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.626+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 <== mon.2 v2:172.21.15.76:3301/0 4 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fd348046020 con 0x7fd334008d50 2019-12-05T06:30:39.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.626+0000 7fd34ffff700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd33402b9c0 0x7fd33402ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:39.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.627+0000 7fd34ffff700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd33402b9c0 0x7fd33402ddf0 crc :-1 s=READY pgs=390 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:39.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.640+0000 7fd3560f8700 1 -- 172.21.15.16:0/1176145677 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd35009def0 con 0x7fd334008d50 2019-12-05T06:30:39.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.648+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7fd348010640 tx=0x7fd3480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.648+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 <== mon.2 v2:172.21.15.76:3301/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd348009930 con 0x7fd334008d50 2019-12-05T06:30:39.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.902+0000 7fd3560f8700 1 -- 172.21.15.16:0/1176145677 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd35009def0 con 0x7fd334008d50 2019-12-05T06:30:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.903+0000 7fd354e96700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 secure :-1 s=THROTTLE_DONE pgs=214 cs=0 l=1 rx=0x7fd348010640 tx=0x7fd3480109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.903+0000 7fd34d7fa700 1 -- 172.21.15.16:0/1176145677 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fd348009d00 con 0x7fd334008d50 2019-12-05T06:30:39.904 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 113 2019-12-05T06:30:39.904 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:39.905 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:38.244643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:replay","state_seq":1,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:39.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.906+0000 7fd3560f8700 1 -- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd33402b9c0 msgr2=0x7fd33402ddf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:39.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.906+0000 7fd3560f8700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd33402b9c0 0x7fd33402ddf0 crc :-1 s=READY pgs=390 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:39.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.906+0000 7fd3560f8700 1 -- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 msgr2=0x7fd33400b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:39.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.906+0000 7fd3560f8700 1 --2- 172.21.15.16:0/1176145677 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd334008d50 0x7fd33400b220 secure :-1 s=READY pgs=214 cs=0 l=1 rx=0x7fd348010640 tx=0x7fd3480109e0).stop 2019-12-05T06:30:39.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.906+0000 7fd354e96700 1 -- 172.21.15.16:0/1176145677 reap_dead start 2019-12-05T06:30:39.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.906+0000 7fd354e96700 1 -- 172.21.15.16:0/1176145677 reap_dead start 2019-12-05T06:30:39.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.906+0000 7fd3560f8700 1 -- 172.21.15.16:0/1176145677 shutdown_connections 2019-12-05T06:30:39.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.906+0000 7fd3560f8700 1 -- 172.21.15.16:0/1176145677 shutdown_connections 2019-12-05T06:30:39.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.906+0000 7fd3560f8700 1 -- 172.21.15.16:0/1176145677 wait complete. 2019-12-05T06:30:39.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:39.906+0000 7fd3560f8700 1 -- 172.21.15.16:0/1176145677 >> 172.21.15.16:0/1176145677 conn(0x7fd35008acd0 msgr2=0x7fd350099650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:39.912 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:30:39.912 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:30:39.912 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:40.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.097+0000 7f6b9ab9b700 1 Processor -- start 2019-12-05T06:30:40.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.097+0000 7f6b9ab9b700 1 -- start start 2019-12-05T06:30:40.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b9ab9b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b9408c6d0 0x7f6b9408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:40.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b9ab9b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6b9408fef0 0x7f6b9408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:40.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b9ab9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b9408ef20 0x7f6b9408f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:40.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b9ab9b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b940a3f70 con 0x7f6b9408ef20 2019-12-05T06:30:40.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b9ab9b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9408d830 con 0x7f6b9408fef0 2019-12-05T06:30:40.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b9ab9b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9408d540 con 0x7f6b9408c6d0 2019-12-05T06:30:40.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b99939700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b9408ef20 0x7f6b9408f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:40.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b98937700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6b9408fef0 0x7f6b9408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:40.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b99138700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b9408c6d0 0x7f6b9408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:40.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b99939700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b9408ef20 0x7f6b9408f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54944/0 (socket says 172.21.15.16:54944) 2019-12-05T06:30:40.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.099+0000 7f6b99939700 1 -- 172.21.15.16:0/2914585811 learned_addr learned my addr 172.21.15.16:0/2914585811 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:40.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 -- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b9408c6d0 msgr2=0x7f6b9408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:40.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 --2- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b9408c6d0 0x7f6b9408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:40.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 -- 172.21.15.16:0/2914585811 >> v2:172.21.15.16:3300/0 conn(0x7f6b9408ef20 msgr2=0x7f6b9408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:40.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 --2- 172.21.15.16:0/2914585811 >> v2:172.21.15.16:3300/0 conn(0x7f6b9408ef20 0x7f6b9408f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:40.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 -- 172.21.15.16:0/2914585811 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9408e620 con 0x7f6b9408fef0 2019-12-05T06:30:40.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b99138700 1 --2- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b9408c6d0 0x7f6b9408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:40.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b99939700 1 --2- 172.21.15.16:0/2914585811 >> v2:172.21.15.16:3300/0 conn(0x7f6b9408ef20 0x7f6b9408f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:30:40.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 --2- 172.21.15.16:0/2914585811 >> v2:172.21.15.76:3300/0 conn(0x7f6b9408fef0 0x7f6b9408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6b8c00b2d0 tx=0x7f6b8c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 --2- 172.21.15.16:0/2914585811 >> v2:172.21.15.76:3300/0 conn(0x7f6b9408fef0 0x7f6b9408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6b8c00b2d0 tx=0x7f6b8c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 --2- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9408fef0 0x7f6b9408d000 secure :-1 s=READY pgs=649 cs=0 l=1 rx=0x7f6b8c00b2d0 tx=0x7f6b8c00bae0).ready entity=mon.1 client_cookie=5de82f05f52f10a8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:40.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 --2- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9408fef0 0x7f6b9408d000 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7f6b8c00b2d0 tx=0x7f6b8c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 --2- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9408fef0 0x7f6b9408d000 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7f6b8c00b2d0 tx=0x7f6b8c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b98937700 1 --2- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9408fef0 0x7f6b9408d000 secure :-1 s=THROTTLE_DONE pgs=649 cs=0 l=1 rx=0x7f6b8c00b2d0 tx=0x7f6b8c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b93fff700 1 -- 172.21.15.16:0/2914585811 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b8c000ce0 con 0x7f6b9408fef0 2019-12-05T06:30:40.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.100+0000 7f6b93fff700 1 -- 172.21.15.16:0/2914585811 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c009b40 con 0x7f6b9408fef0 2019-12-05T06:30:40.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.101+0000 7f6b93fff700 1 -- 172.21.15.16:0/2914585811 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b8c00a210 con 0x7f6b9408fef0 2019-12-05T06:30:40.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.101+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9408fef0 msgr2=0x7f6b9408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:40.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.101+0000 7f6b9ab9b700 1 --2- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9408fef0 0x7f6b9408d000 secure :-1 s=READY pgs=649 cs=0 l=1 rx=0x7f6b8c00b2d0 tx=0x7f6b8c00bae0).stop 2019-12-05T06:30:40.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.102+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/2914585811 shutdown_connections 2019-12-05T06:30:40.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.102+0000 7f6b9ab9b700 1 --2- 172.21.15.16:0/2914585811 >> v2:172.21.15.16:3300/0 conn(0x7f6b9408ef20 0x7f6b9408f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:40.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.102+0000 7f6b9ab9b700 1 --2- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b9408fef0 0x7f6b9408d000 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:40.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.102+0000 7f6b9ab9b700 1 --2- 172.21.15.16:0/2914585811 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b9408c6d0 0x7f6b9408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:40.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.102+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/2914585811 shutdown_connections 2019-12-05T06:30:40.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.102+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/2914585811 wait complete. 2019-12-05T06:30:40.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.102+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/2914585811 >> 172.21.15.16:0/2914585811 conn(0x7f6b94061940 msgr2=0x7f6b94096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:40.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.106+0000 7f6b9ab9b700 1 Processor -- start 2019-12-05T06:30:40.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.106+0000 7f6b9ab9b700 1 -- start start 2019-12-05T06:30:40.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b9ab9b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6b94130750 0x7f6b9412ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:40.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b9ab9b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b940889b0 con 0x7f6b94154af0 2019-12-05T06:30:40.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b9ab9b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b94081ca0 con 0x7f6b94080290 2019-12-05T06:30:40.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b9ab9b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b9408a670 con 0x7f6b94130750 2019-12-05T06:30:40.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b99939700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6b94154af0 0x7f6b94156ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40118/0 (socket says 172.21.15.16:40118) 2019-12-05T06:30:40.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b99939700 1 -- 172.21.15.16:0/589519864 learned_addr learned my addr 172.21.15.16:0/589519864 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:30:40.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> v2:172.21.15.76:3300/0 conn(0x7f6b94130750 0x7f6b9412ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:40.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319171837 0 0) 0x7f6b7c007790 con 0x7f6b94154af0 2019-12-05T06:30:40.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b78002e80 con 0x7f6b94154af0 2019-12-05T06:30:40.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3145092613 0 0) 0x7f6b8400d970 con 0x7f6b94080290 2019-12-05T06:30:40.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.108+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b78004170 con 0x7f6b94080290 2019-12-05T06:30:40.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b98937700 1 -- 172.21.15.16:0/589519864 >> v1:172.21.15.76:6789/0 conn(0x7f6b94080290 legacy=0x7f6b9412f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:40.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b98937700 1 -- 172.21.15.16:0/589519864 >> v1:172.21.15.16:6789/0 conn(0x7f6b94154af0 legacy=0x7f6b94156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:40.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b98937700 1 -- 172.21.15.16:0/589519864 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b9412ee80 con 0x7f6b94130750 2019-12-05T06:30:40.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> v2:172.21.15.76:3300/0 conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> v2:172.21.15.76:3300/0 conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=READY pgs=651 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:40.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b8c00b000 con 0x7f6b94130750 2019-12-05T06:30:40.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b8c009af0 con 0x7f6b94130750 2019-12-05T06:30:40.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b8c00a180 con 0x7f6b94130750 2019-12-05T06:30:40.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/589519864 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9412ee80 con 0x7f6b94130750 2019-12-05T06:30:40.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.109+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/589519864 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9412ee80 con 0x7f6b94130750 2019-12-05T06:30:40.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.110+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.111+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6b8c009ce0 con 0x7f6b94130750 2019-12-05T06:30:40.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.111+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.111+0000 7f6b91ffb700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b78023650 0x7f6b78025b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.111+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f6b8c08a3a0 con 0x7f6b94130750 2019-12-05T06:30:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.111+0000 7f6b99138700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b78023650 0x7f6b78025b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.111+0000 7f6b99138700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b78023650 0x7f6b78025b00 crc :-1 s=READY pgs=391 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:40.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.112+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/589519864 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9412ee80 con 0x7f6b94130750 2019-12-05T06:30:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.118+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.118+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6b8c00b230 con 0x7f6b94130750 2019-12-05T06:30:40.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.321+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/589519864 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b9412ee80 con 0x7f6b94130750 2019-12-05T06:30:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.322+0000 7f6b98937700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=THROTTLE_DONE pgs=651 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:40.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.322+0000 7f6b91ffb700 1 -- 172.21.15.16:0/589519864 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 113 v113) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f6b8c009af0 con 0x7f6b94130750 2019-12-05T06:30:40.318 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 113 2019-12-05T06:30:40.319 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:40.319 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":113,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":113,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:38.244643+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:replay","state_seq":1,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:40.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.326+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b78023650 msgr2=0x7f6b78025b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:40.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.326+0000 7f6b9ab9b700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b78023650 0x7f6b78025b00 crc :-1 s=READY pgs=391 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:40.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.326+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 msgr2=0x7f6b9412ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:40.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.326+0000 7f6b9ab9b700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 0x7f6b9412ff80 secure :-1 s=READY pgs=651 cs=0 l=1 rx=0x7f6b8c054710 tx=0x7f6b8c054ab0).stop 2019-12-05T06:30:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.326+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/589519864 shutdown_connections 2019-12-05T06:30:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.326+0000 7f6b9ab9b700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b78023650 0x7f6b78025b00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:40.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.326+0000 7f6b9ab9b700 1 --2- 172.21.15.16:0/589519864 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6b94130750 0x7f6b9412ff80 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:40.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.327+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/589519864 shutdown_connections 2019-12-05T06:30:40.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.327+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/589519864 wait complete. 2019-12-05T06:30:40.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:40.327+0000 7f6b9ab9b700 1 -- 172.21.15.16:0/589519864 >> 172.21.15.16:0/589519864 conn(0x7f6b9409de30 msgr2=0x7f6b94070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:40.333 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 113, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 113, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6544": { "addr": "172.21.15.76:6835/945658794", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 945658794, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 945658794, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6544, "incarnation": 113, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 66, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:30:38.244643+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6544 } } } ], "standbys": [] } 2019-12-05T06:30:42.334 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:42.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.610+0000 7f6734c23700 1 Processor -- start 2019-12-05T06:30:42.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.610+0000 7f6734c23700 1 -- start start 2019-12-05T06:30:42.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.612+0000 7f6734c23700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f673008fef0 0x7f673008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:42.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.612+0000 7f6734c23700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67300958e0 con 0x7f673008d2c0 2019-12-05T06:30:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.612+0000 7f6734c23700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67300629e0 con 0x7f673008c6d0 2019-12-05T06:30:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.612+0000 7f6734c23700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f673008df90 con 0x7f673008fef0 2019-12-05T06:30:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.612+0000 7f672e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f673008d2c0 0x7f673008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40126/0 (socket says 172.21.15.16:40126) 2019-12-05T06:30:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.612+0000 7f672e59c700 1 -- 172.21.15.16:0/3396532115 learned_addr learned my addr 172.21.15.16:0/3396532115 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:30:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.612+0000 7f672f59e700 1 --2- 172.21.15.16:0/3396532115 >> v2:172.21.15.76:3300/0 conn(0x7f673008fef0 0x7f673008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:42.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.613+0000 7f672dd9b700 1 -- 172.21.15.16:0/3396532115 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163093269 0 0) 0x7f6718001150 con 0x7f673008d2c0 2019-12-05T06:30:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.613+0000 7f672dd9b700 1 -- 172.21.15.16:0/3396532115 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f671c002e80 con 0x7f673008d2c0 2019-12-05T06:30:42.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.613+0000 7f672dd9b700 1 -- 172.21.15.16:0/3396532115 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 28352560 0 0) 0x7f6728001400 con 0x7f673008c6d0 2019-12-05T06:30:42.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.613+0000 7f672f59e700 1 -- 172.21.15.16:0/3396532115 >> v1:172.21.15.76:6789/0 conn(0x7f673008c6d0 legacy=0x7f673008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.613+0000 7f672f59e700 1 -- 172.21.15.16:0/3396532115 >> v1:172.21.15.16:6789/0 conn(0x7f673008d2c0 legacy=0x7f673008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.613+0000 7f672f59e700 1 -- 172.21.15.16:0/3396532115 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f673008e620 con 0x7f673008fef0 2019-12-05T06:30:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.614+0000 7f672f59e700 1 --2- 172.21.15.16:0/3396532115 >> v2:172.21.15.76:3300/0 conn(0x7f673008fef0 0x7f673008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f672000af80 tx=0x7f672000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.614+0000 7f672f59e700 1 --2- 172.21.15.16:0/3396532115 >> v2:172.21.15.76:3300/0 conn(0x7f673008fef0 0x7f673008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f672000af80 tx=0x7f672000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.614+0000 7f672f59e700 1 --2- 172.21.15.16:0/3396532115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f673008fef0 0x7f673008f720 secure :-1 s=READY pgs=653 cs=0 l=1 rx=0x7f672000af80 tx=0x7f672000b790).ready entity=mon.1 client_cookie=d48d98fae63a1412 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:42.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.614+0000 7f672f59e700 1 --2- 172.21.15.16:0/3396532115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f673008fef0 0x7f673008f720 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f672000af80 tx=0x7f672000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.614+0000 7f672f59e700 1 --2- 172.21.15.16:0/3396532115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f673008fef0 0x7f673008f720 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f672000af80 tx=0x7f672000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.614+0000 7f672dd9b700 1 -- 172.21.15.16:0/3396532115 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6720007240 con 0x7f673008fef0 2019-12-05T06:30:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.614+0000 7f672f59e700 1 --2- 172.21.15.16:0/3396532115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f673008fef0 0x7f673008f720 secure :-1 s=THROTTLE_DONE pgs=653 cs=0 l=1 rx=0x7f672000af80 tx=0x7f672000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.614+0000 7f672dd9b700 1 -- 172.21.15.16:0/3396532115 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6720000c30 con 0x7f673008fef0 2019-12-05T06:30:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.614+0000 7f672dd9b700 1 -- 172.21.15.16:0/3396532115 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6720009e60 con 0x7f673008fef0 2019-12-05T06:30:42.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.615+0000 7f6734c23700 1 -- 172.21.15.16:0/3396532115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f673008fef0 msgr2=0x7f673008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.615+0000 7f6734c23700 1 --2- 172.21.15.16:0/3396532115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f673008fef0 0x7f673008f720 secure :-1 s=READY pgs=653 cs=0 l=1 rx=0x7f672000af80 tx=0x7f672000b790).stop 2019-12-05T06:30:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.617+0000 7f6734c23700 1 -- 172.21.15.16:0/3396532115 shutdown_connections 2019-12-05T06:30:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.617+0000 7f6734c23700 1 --2- 172.21.15.16:0/3396532115 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f673008fef0 0x7f673008f720 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.617+0000 7f6734c23700 1 -- 172.21.15.16:0/3396532115 shutdown_connections 2019-12-05T06:30:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.617+0000 7f6734c23700 1 -- 172.21.15.16:0/3396532115 wait complete. 2019-12-05T06:30:42.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.617+0000 7f6734c23700 1 -- 172.21.15.16:0/3396532115 >> 172.21.15.16:0/3396532115 conn(0x7f6730061940 msgr2=0x7f6730096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.622+0000 7f6734c23700 1 Processor -- start 2019-12-05T06:30:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.623+0000 7f6734c23700 1 -- start start 2019-12-05T06:30:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.624+0000 7f6734c23700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 0x7f673006e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.624+0000 7f6734c23700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6730128af0 con 0x7f6730135450 2019-12-05T06:30:42.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.624+0000 7f6734c23700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6730128600 con 0x7f6730134c80 2019-12-05T06:30:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.624+0000 7f6734c23700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6730132830 con 0x7f673006dc50 2019-12-05T06:30:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.624+0000 7f672e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6730135450 0x7f6730134480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40132/0 (socket says 172.21.15.16:40132) 2019-12-05T06:30:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.624+0000 7f672e59c700 1 -- 172.21.15.16:0/919390052 learned_addr learned my addr 172.21.15.16:0/919390052 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:30:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.625+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 0x7f673006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.625+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 60856638 0 0) 0x7f6718001150 con 0x7f6730135450 2019-12-05T06:30:42.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.625+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6714002e80 con 0x7f6730135450 2019-12-05T06:30:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.625+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2539617762 0 0) 0x7f6720009990 con 0x7f6730134c80 2019-12-05T06:30:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.625+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6714004170 con 0x7f6730134c80 2019-12-05T06:30:42.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.625+0000 7f672ed9d700 1 -- 172.21.15.16:0/919390052 >> v1:172.21.15.76:6789/0 conn(0x7f6730134c80 legacy=0x7f67301577d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.625+0000 7f672ed9d700 1 -- 172.21.15.16:0/919390052 >> v1:172.21.15.16:6789/0 conn(0x7f6730135450 legacy=0x7f6730134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.625+0000 7f672ed9d700 1 -- 172.21.15.16:0/919390052 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6730133b80 con 0x7f673006dc50 2019-12-05T06:30:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.626+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 0x7f673006e040 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6728010720 tx=0x7f6728010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.626+0000 7f672ed9d700 1 -- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 msgr2=0x7f673006e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:30:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.626+0000 7f672ed9d700 1 -- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 msgr2=0x7f673006e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:30:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.626+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 0x7f673006e040 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6728010720 tx=0x7f6728010ac0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:30:42.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.626+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 0x7f673006e040 secure :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0x7f6728010720 tx=0x7f6728010ac0)._fault waiting 0.200000 2019-12-05T06:30:42.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.827+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 0x7f673006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.827+0000 7f672ed9d700 0 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 0x7f673006e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T06:30:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.827+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 0x7f673006e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.827+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 msgr2=0x7f673006e040 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:30:42.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.827+0000 7f67277fe700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f673006dc50 0x7f673006e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.827+0000 7f67277fe700 1 --2- 172.21.15.16:0/919390052 >> v2:172.21.15.16:3300/0 conn(0x7f6714005500 0x7f6714004e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.827+0000 7f67277fe700 1 --2- 172.21.15.16:0/919390052 >> v2:172.21.15.76:3300/0 conn(0x7f6714005960 0x7f6714005e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.827+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6714008660 con 0x7f6714005500 2019-12-05T06:30:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.827+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6714008820 con 0x7f6714005960 2019-12-05T06:30:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.828+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> v2:172.21.15.16:3300/0 conn(0x7f6714005500 0x7f6714004e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:42.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.828+0000 7f672e59c700 1 --2- 172.21.15.16:0/919390052 >> v2:172.21.15.76:3300/0 conn(0x7f6714005960 0x7f6714005e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.828+0000 7f672ed9d700 1 -- 172.21.15.16:0/919390052 >> v2:172.21.15.76:3300/0 conn(0x7f6714005960 msgr2=0x7f6714005e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.828+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> v2:172.21.15.76:3300/0 conn(0x7f6714005960 0x7f6714005e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.828+0000 7f672ed9d700 1 -- 172.21.15.16:0/919390052 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6714008ad0 con 0x7f6714005500 2019-12-05T06:30:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> v2:172.21.15.16:3300/0 conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> v2:172.21.15.16:3300/0 conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=READY pgs=905 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=THROTTLE_DONE pgs=905 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6728000e80 con 0x7f6714005500 2019-12-05T06:30:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=THROTTLE_DONE pgs=905 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=THROTTLE_DONE pgs=905 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6728009d00 con 0x7f6714005500 2019-12-05T06:30:42.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f6734c23700 1 -- 172.21.15.16:0/919390052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6730133b80 con 0x7f6714005500 2019-12-05T06:30:42.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6728000e80 con 0x7f6714005500 2019-12-05T06:30:42.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.829+0000 7f6734c23700 1 -- 172.21.15.16:0/919390052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6730133b80 con 0x7f6714005500 2019-12-05T06:30:42.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.831+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=THROTTLE_DONE pgs=905 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.832+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6728009900 con 0x7f6714005500 2019-12-05T06:30:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.832+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=THROTTLE_DONE pgs=905 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.832+0000 7f67277fe700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6714028cf0 0x7f671402b120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.832+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f6728047810 con 0x7f6714005500 2019-12-05T06:30:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.833+0000 7f672e59c700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6714028cf0 0x7f671402b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:42.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.833+0000 7f672e59c700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6714028cf0 0x7f671402b120 crc :-1 s=READY pgs=392 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:42.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.874+0000 7f6734c23700 1 -- 172.21.15.16:0/919390052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6730133b80 con 0x7f6714005500 2019-12-05T06:30:42.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.885+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=THROTTLE_DONE pgs=905 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:42.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:42.885+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6728000e80 con 0x7f6714005500 2019-12-05T06:30:43.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.150+0000 7f6734c23700 1 -- 172.21.15.16:0/919390052 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6730133b80 con 0x7f6714005500 2019-12-05T06:30:43.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.151+0000 7f672ed9d700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=THROTTLE_DONE pgs=905 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:43.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.151+0000 7f67277fe700 1 -- 172.21.15.16:0/919390052 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 115 v115) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f6728009900 con 0x7f6714005500 2019-12-05T06:30:43.146 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 115 2019-12-05T06:30:43.146 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:43.146 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":115,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":115,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:42.266399+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:rejoin","state_seq":20,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:43.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.154+0000 7f6734c23700 1 -- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6714028cf0 msgr2=0x7f671402b120 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:43.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.154+0000 7f6734c23700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6714028cf0 0x7f671402b120 crc :-1 s=READY pgs=392 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:43.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.154+0000 7f6734c23700 1 -- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 msgr2=0x7f6714004e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:43.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.154+0000 7f672f59e700 1 -- 172.21.15.16:0/919390052 reap_dead start 2019-12-05T06:30:43.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.154+0000 7f6734c23700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 0x7f6714004e40 secure :-1 s=READY pgs=905 cs=0 l=1 rx=0x7f6728011dd0 tx=0x7f6728012170).stop 2019-12-05T06:30:43.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.154+0000 7f6734c23700 1 -- 172.21.15.16:0/919390052 shutdown_connections 2019-12-05T06:30:43.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.154+0000 7f6734c23700 1 --2- 172.21.15.16:0/919390052 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6714005500 0x7f6714004e40 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:43.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.154+0000 7f6734c23700 1 -- 172.21.15.16:0/919390052 shutdown_connections 2019-12-05T06:30:43.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.154+0000 7f6734c23700 1 -- 172.21.15.16:0/919390052 wait complete. 2019-12-05T06:30:43.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:43.154+0000 7f6734c23700 1 -- 172.21.15.16:0/919390052 >> 172.21.15.16:0/919390052 conn(0x7f67300a2850 msgr2=0x7f673006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:43.159 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:30:45.160 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:45.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.375+0000 7fc3e12a4700 1 Processor -- start 2019-12-05T06:30:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.375+0000 7fc3e12a4700 1 -- start start 2019-12-05T06:30:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3e12a4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc3dc08c6d0 0x7fc3dc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:45.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3e12a4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc3dc08fef0 0x7fc3dc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3e12a4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:45.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3e12a4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc08df90 con 0x7fc3dc08ef20 2019-12-05T06:30:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3e12a4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc0a3f70 con 0x7fc3dc08c6d0 2019-12-05T06:30:45.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3e12a4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3dc08d830 con 0x7fc3dc08fef0 2019-12-05T06:30:45.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3dbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:45.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3dbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54986/0 (socket says 172.21.15.16:54986) 2019-12-05T06:30:45.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3dbfff700 1 -- 172.21.15.16:0/363678012 learned_addr learned my addr 172.21.15.16:0/363678012 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3db7fe700 1 --2- 172.21.15.16:0/363678012 >> v2:172.21.15.76:3300/0 conn(0x7fc3dc08c6d0 0x7fc3dc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.376+0000 7fc3daffd700 1 --2- 172.21.15.16:0/363678012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc3dc08fef0 0x7fc3dc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 -- 172.21.15.16:0/363678012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc3dc08fef0 msgr2=0x7fc3dc08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:45.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 --2- 172.21.15.16:0/363678012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc3dc08fef0 0x7fc3dc08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 -- 172.21.15.16:0/363678012 >> v2:172.21.15.76:3300/0 conn(0x7fc3dc08c6d0 msgr2=0x7fc3dc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 --2- 172.21.15.16:0/363678012 >> v2:172.21.15.76:3300/0 conn(0x7fc3dc08c6d0 0x7fc3dc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:45.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 -- 172.21.15.16:0/363678012 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc08d540 con 0x7fc3dc08ef20 2019-12-05T06:30:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 --2- 172.21.15.16:0/363678012 >> v2:172.21.15.16:3300/0 conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc3d400fa30 tx=0x7fc3d40080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 --2- 172.21.15.16:0/363678012 >> v2:172.21.15.16:3300/0 conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc3d400fa30 tx=0x7fc3d40080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 --2- 172.21.15.16:0/363678012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 secure :-1 s=READY pgs=906 cs=0 l=1 rx=0x7fc3d400fa30 tx=0x7fc3d40080d0).ready entity=mon.0 client_cookie=44349bfe23095dfb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 --2- 172.21.15.16:0/363678012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 secure :-1 s=THROTTLE_DONE pgs=906 cs=0 l=1 rx=0x7fc3d400fa30 tx=0x7fc3d40080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 --2- 172.21.15.16:0/363678012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 secure :-1 s=THROTTLE_DONE pgs=906 cs=0 l=1 rx=0x7fc3d400fa30 tx=0x7fc3d40080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3dbfff700 1 --2- 172.21.15.16:0/363678012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 secure :-1 s=THROTTLE_DONE pgs=906 cs=0 l=1 rx=0x7fc3d400fa30 tx=0x7fc3d40080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3da7fc700 1 -- 172.21.15.16:0/363678012 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3d4000df0 con 0x7fc3dc08ef20 2019-12-05T06:30:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3da7fc700 1 -- 172.21.15.16:0/363678012 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d4001460 con 0x7fc3dc08ef20 2019-12-05T06:30:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3da7fc700 1 -- 172.21.15.16:0/363678012 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3d4001bb0 con 0x7fc3dc08ef20 2019-12-05T06:30:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3e12a4700 1 -- 172.21.15.16:0/363678012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc08ef20 msgr2=0x7fc3dc08f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.377+0000 7fc3e12a4700 1 --2- 172.21.15.16:0/363678012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 secure :-1 s=READY pgs=906 cs=0 l=1 rx=0x7fc3d400fa30 tx=0x7fc3d40080d0).stop 2019-12-05T06:30:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.378+0000 7fc3e12a4700 1 -- 172.21.15.16:0/363678012 shutdown_connections 2019-12-05T06:30:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.378+0000 7fc3e12a4700 1 --2- 172.21.15.16:0/363678012 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc08ef20 0x7fc3dc08f3c0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.378+0000 7fc3e12a4700 1 --2- 172.21.15.16:0/363678012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc3dc08fef0 0x7fc3dc08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.378+0000 7fc3e12a4700 1 --2- 172.21.15.16:0/363678012 >> v2:172.21.15.76:3300/0 conn(0x7fc3dc08c6d0 0x7fc3dc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.378+0000 7fc3e12a4700 1 -- 172.21.15.16:0/363678012 shutdown_connections 2019-12-05T06:30:45.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.378+0000 7fc3e12a4700 1 -- 172.21.15.16:0/363678012 wait complete. 2019-12-05T06:30:45.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.378+0000 7fc3e12a4700 1 -- 172.21.15.16:0/363678012 >> 172.21.15.16:0/363678012 conn(0x7fc3dc061940 msgr2=0x7fc3dc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:45.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.382+0000 7fc3e12a4700 1 Processor -- start 2019-12-05T06:30:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.382+0000 7fc3e12a4700 1 -- start start 2019-12-05T06:30:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.383+0000 7fc3e12a4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3dc132d00 0x7fc3dc132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:45.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.383+0000 7fc3e12a4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc3dc154b00 0x7fc3dc156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.383+0000 7fc3e12a4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3dc098da0 con 0x7fc3dc09b850 2019-12-05T06:30:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.383+0000 7fc3e12a4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc09aa60 con 0x7fc3dc132d00 2019-12-05T06:30:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.383+0000 7fc3e12a4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3dc09a030 con 0x7fc3dc154b00 2019-12-05T06:30:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3dc132d00 0x7fc3dc132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc3dc132d00 0x7fc3dc132530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:54990/0 (socket says 172.21.15.16:54990) 2019-12-05T06:30:45.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 -- 172.21.15.16:0/208206154 learned_addr learned my addr 172.21.15.16:0/208206154 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:45.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3dbfff700 1 --2- 172.21.15.16:0/208206154 >> v2:172.21.15.76:3300/0 conn(0x7fc3dc154b00 0x7fc3dc156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:45.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 -- 172.21.15.16:0/208206154 >> v2:172.21.15.76:3300/0 conn(0x7fc3dc154b00 msgr2=0x7fc3dc156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:45.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> v2:172.21.15.76:3300/0 conn(0x7fc3dc154b00 0x7fc3dc156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:45.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 -- 172.21.15.16:0/208206154 >> v1:172.21.15.76:6789/0 conn(0x7fc3dc09b850 legacy=0x7fc3dc131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:45.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 -- 172.21.15.16:0/208206154 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3dc09b0f0 con 0x7fc3dc132d00 2019-12-05T06:30:45.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> v2:172.21.15.16:3300/0 conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> v2:172.21.15.16:3300/0 conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=READY pgs=907 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:45.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=THROTTLE_DONE pgs=907 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=THROTTLE_DONE pgs=907 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=THROTTLE_DONE pgs=907 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.384+0000 7fc3c3fff700 1 -- 172.21.15.16:0/208206154 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3d0000960 con 0x7fc3dc132d00 2019-12-05T06:30:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.385+0000 7fc3c3fff700 1 -- 172.21.15.16:0/208206154 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3d0005410 con 0x7fc3dc132d00 2019-12-05T06:30:45.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.385+0000 7fc3e12a4700 1 -- 172.21.15.16:0/208206154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3dc09b0f0 con 0x7fc3dc132d00 2019-12-05T06:30:45.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.385+0000 7fc3c3fff700 1 -- 172.21.15.16:0/208206154 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3d0005b20 con 0x7fc3dc132d00 2019-12-05T06:30:45.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.385+0000 7fc3e12a4700 1 -- 172.21.15.16:0/208206154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3dc09b0f0 con 0x7fc3dc132d00 2019-12-05T06:30:45.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.385+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=THROTTLE_DONE pgs=907 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.386+0000 7fc3c3fff700 1 -- 172.21.15.16:0/208206154 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc3d0005600 con 0x7fc3dc132d00 2019-12-05T06:30:45.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.386+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=THROTTLE_DONE pgs=907 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.386+0000 7fc3c3fff700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc3bc023630 0x7fc3bc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.386+0000 7fc3c3fff700 1 -- 172.21.15.16:0/208206154 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fc3d0045df0 con 0x7fc3dc132d00 2019-12-05T06:30:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.386+0000 7fc3db7fe700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc3bc023630 0x7fc3bc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:45.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.387+0000 7fc3db7fe700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc3bc023630 0x7fc3bc025ac0 crc :-1 s=READY pgs=393 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:45.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.390+0000 7fc3e12a4700 1 -- 172.21.15.16:0/208206154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3dc09b0f0 con 0x7fc3dc132d00 2019-12-05T06:30:45.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.394+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=THROTTLE_DONE pgs=907 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.394+0000 7fc3c3fff700 1 -- 172.21.15.16:0/208206154 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc3d0005180 con 0x7fc3dc132d00 2019-12-05T06:30:45.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.599+0000 7fc3e12a4700 1 -- 172.21.15.16:0/208206154 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc3dc09b0f0 con 0x7fc3dc132d00 2019-12-05T06:30:45.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.599+0000 7fc3daffd700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=THROTTLE_DONE pgs=907 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:45.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.599+0000 7fc3c3fff700 1 -- 172.21.15.16:0/208206154 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fc3d00055d0 con 0x7fc3dc132d00 2019-12-05T06:30:45.595 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 116 2019-12-05T06:30:45.595 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:45.596 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:43.275354+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":21,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:45.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.602+0000 7fc3e12a4700 1 -- 172.21.15.16:0/208206154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc3bc023630 msgr2=0x7fc3bc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.602+0000 7fc3e12a4700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc3bc023630 0x7fc3bc025ac0 crc :-1 s=READY pgs=393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.602+0000 7fc3e12a4700 1 -- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 msgr2=0x7fc3dc132530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.602+0000 7fc3e12a4700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 0x7fc3dc132530 secure :-1 s=READY pgs=907 cs=0 l=1 rx=0x7fc3d0010450 tx=0x7fc3d0010280).stop 2019-12-05T06:30:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.602+0000 7fc3e12a4700 1 -- 172.21.15.16:0/208206154 shutdown_connections 2019-12-05T06:30:45.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.602+0000 7fc3e12a4700 1 --2- 172.21.15.16:0/208206154 >> v2:172.21.15.76:3300/0 conn(0x7fc3dc154b00 0x7fc3dc156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.602+0000 7fc3e12a4700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc3bc023630 0x7fc3bc025ac0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.602+0000 7fc3e12a4700 1 --2- 172.21.15.16:0/208206154 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc3dc132d00 0x7fc3dc132530 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.603+0000 7fc3e12a4700 1 -- 172.21.15.16:0/208206154 shutdown_connections 2019-12-05T06:30:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.603+0000 7fc3e12a4700 1 -- 172.21.15.16:0/208206154 wait complete. 2019-12-05T06:30:45.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:45.603+0000 7fc3e12a4700 1 -- 172.21.15.16:0/208206154 >> 172.21.15.16:0/208206154 conn(0x7fc3dc0714f0 msgr2=0x7fc3dc09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:45.607 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6363 2019-12-05T06:30:45.607 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.a 2019-12-05T06:30:47.608 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:30:47.609 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:30:47.609 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:30:47.614 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:30:47.615 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:47.781 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:30:47.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.803+0000 7f7c4d126700 1 Processor -- start 2019-12-05T06:30:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.803+0000 7f7c4d126700 1 -- start start 2019-12-05T06:30:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.803+0000 7f7c4d126700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c4808c6d0 0x7f7c4808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.803+0000 7f7c4d126700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c4808fef0 0x7f7c4808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:47.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.803+0000 7f7c4d126700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7c4808ef20 0x7f7c4808f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.803+0000 7f7c4d126700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c4808df90 con 0x7f7c4808c6d0 2019-12-05T06:30:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.803+0000 7f7c4d126700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c480a3f70 con 0x7f7c4808ef20 2019-12-05T06:30:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.803+0000 7f7c4d126700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c4808d830 con 0x7f7c4808fef0 2019-12-05T06:30:47.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.803+0000 7f7c4759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c4808c6d0 0x7f7c4808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c4808c6d0 0x7f7c4808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55016/0 (socket says 172.21.15.16:55016) 2019-12-05T06:30:47.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 -- 172.21.15.16:0/1812811432 learned_addr learned my addr 172.21.15.16:0/1812811432 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:47.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c46d9d700 1 --2- 172.21.15.16:0/1812811432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c4808fef0 0x7f7c4808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:47.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4c925700 1 --2- 172.21.15.16:0/1812811432 >> v2:172.21.15.76:3300/0 conn(0x7f7c4808ef20 0x7f7c4808f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:47.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 -- 172.21.15.16:0/1812811432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c4808fef0 msgr2=0x7f7c4808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:47.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 --2- 172.21.15.16:0/1812811432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c4808fef0 0x7f7c4808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:47.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 -- 172.21.15.16:0/1812811432 >> v2:172.21.15.76:3300/0 conn(0x7f7c4808ef20 msgr2=0x7f7c4808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:47.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 --2- 172.21.15.16:0/1812811432 >> v2:172.21.15.76:3300/0 conn(0x7f7c4808ef20 0x7f7c4808f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:47.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 -- 172.21.15.16:0/1812811432 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c4808d540 con 0x7f7c4808c6d0 2019-12-05T06:30:47.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c46d9d700 1 --2- 172.21.15.16:0/1812811432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c4808fef0 0x7f7c4808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:30:47.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 --2- 172.21.15.16:0/1812811432 >> v2:172.21.15.16:3300/0 conn(0x7f7c4808c6d0 0x7f7c4808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7c3c00f740 tx=0x7f7c3c00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 --2- 172.21.15.16:0/1812811432 >> v2:172.21.15.16:3300/0 conn(0x7f7c4808c6d0 0x7f7c4808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7c3c00f740 tx=0x7f7c3c00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 --2- 172.21.15.16:0/1812811432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4808c6d0 0x7f7c4808cac0 secure :-1 s=READY pgs=910 cs=0 l=1 rx=0x7f7c3c00f740 tx=0x7f7c3c00f5f0).ready entity=mon.0 client_cookie=4c5561d8c1c166fa server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:47.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 --2- 172.21.15.16:0/1812811432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4808c6d0 0x7f7c4808cac0 secure :-1 s=THROTTLE_DONE pgs=910 cs=0 l=1 rx=0x7f7c3c00f740 tx=0x7f7c3c00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 --2- 172.21.15.16:0/1812811432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4808c6d0 0x7f7c4808cac0 secure :-1 s=THROTTLE_DONE pgs=910 cs=0 l=1 rx=0x7f7c3c00f740 tx=0x7f7c3c00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4659c700 1 -- 172.21.15.16:0/1812811432 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c3c000960 con 0x7f7c4808c6d0 2019-12-05T06:30:47.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.804+0000 7f7c4759e700 1 --2- 172.21.15.16:0/1812811432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4808c6d0 0x7f7c4808cac0 secure :-1 s=THROTTLE_DONE pgs=910 cs=0 l=1 rx=0x7f7c3c00f740 tx=0x7f7c3c00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.805+0000 7f7c4659c700 1 -- 172.21.15.16:0/1812811432 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c3c000c90 con 0x7f7c4808c6d0 2019-12-05T06:30:47.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.805+0000 7f7c4659c700 1 -- 172.21.15.16:0/1812811432 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c3c0013a0 con 0x7f7c4808c6d0 2019-12-05T06:30:47.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.805+0000 7f7c4d126700 1 -- 172.21.15.16:0/1812811432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4808c6d0 msgr2=0x7f7c4808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:47.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.805+0000 7f7c4d126700 1 --2- 172.21.15.16:0/1812811432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4808c6d0 0x7f7c4808cac0 secure :-1 s=READY pgs=910 cs=0 l=1 rx=0x7f7c3c00f740 tx=0x7f7c3c00f5f0).stop 2019-12-05T06:30:47.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.806+0000 7f7c4d126700 1 -- 172.21.15.16:0/1812811432 shutdown_connections 2019-12-05T06:30:47.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.806+0000 7f7c4d126700 1 --2- 172.21.15.16:0/1812811432 >> v2:172.21.15.76:3300/0 conn(0x7f7c4808ef20 0x7f7c4808f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:47.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.806+0000 7f7c4d126700 1 --2- 172.21.15.16:0/1812811432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c4808fef0 0x7f7c4808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:47.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.806+0000 7f7c4d126700 1 --2- 172.21.15.16:0/1812811432 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c4808c6d0 0x7f7c4808cac0 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:47.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.806+0000 7f7c4d126700 1 -- 172.21.15.16:0/1812811432 shutdown_connections 2019-12-05T06:30:47.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.806+0000 7f7c4d126700 1 -- 172.21.15.16:0/1812811432 wait complete. 2019-12-05T06:30:47.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.806+0000 7f7c4d126700 1 -- 172.21.15.16:0/1812811432 >> 172.21.15.16:0/1812811432 conn(0x7f7c48061940 msgr2=0x7f7c48096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.808+0000 7f7c4d126700 1 Processor -- start 2019-12-05T06:30:47.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.808+0000 7f7c4d126700 1 -- start start 2019-12-05T06:30:47.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.809+0000 7f7c4d126700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7c48084f30 0x7f7c48157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.809+0000 7f7c4d126700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c48084b90 con 0x7f7c48096e90 2019-12-05T06:30:47.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.809+0000 7f7c4d126700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c48098790 con 0x7f7c4809bbe0 2019-12-05T06:30:47.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.809+0000 7f7c4d126700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c4809a450 con 0x7f7c48084f30 2019-12-05T06:30:47.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.809+0000 7f7c4759e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7c48096e90 0x7f7c4809b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40190/0 (socket says 172.21.15.16:40190) 2019-12-05T06:30:47.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.809+0000 7f7c4759e700 1 -- 172.21.15.16:0/2680416271 learned_addr learned my addr 172.21.15.16:0/2680416271 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:30:47.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> v2:172.21.15.76:3300/0 conn(0x7f7c48084f30 0x7f7c48157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:47.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387518810 0 0) 0x7f7c3c007580 con 0x7f7c48096e90 2019-12-05T06:30:47.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c28002e80 con 0x7f7c48096e90 2019-12-05T06:30:47.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1701442602 0 0) 0x7f7c3c000940 con 0x7f7c48096e90 2019-12-05T06:30:47.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c280032e0 con 0x7f7c48096e90 2019-12-05T06:30:47.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f7c3c007fe0 con 0x7f7c48096e90 2019-12-05T06:30:47.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 >> v2:172.21.15.76:3300/0 conn(0x7f7c48084f30 msgr2=0x7f7c48157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:47.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 --2- 172.21.15.16:0/2680416271 >> v2:172.21.15.76:3300/0 conn(0x7f7c48084f30 0x7f7c48157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:47.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 >> v1:172.21.15.76:6789/0 conn(0x7f7c4809bbe0 legacy=0x7f7c481539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:47.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 >> v1:172.21.15.16:6789/0 conn(0x7f7c48096e90 legacy=0x7f7c4809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:47.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c280037c0 0x7f7c28003bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:47.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c28004110 0x7f7c28004590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:47.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:47.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c28005320 con 0x7f7c28004bf0 2019-12-05T06:30:47.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2800be00 con 0x7f7c28004110 2019-12-05T06:30:47.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2800c030 con 0x7f7c280037c0 2019-12-05T06:30:47.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:47.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c4759e700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c280037c0 0x7f7c28003bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:47.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.810+0000 7f7c46d9d700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c28004110 0x7f7c28004590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:47.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 -- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c280037c0 msgr2=0x7f7c28003bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:47.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c280037c0 0x7f7c28003bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:47.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 -- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c28004110 msgr2=0x7f7c28004590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c28004110 0x7f7c28004590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 -- 172.21.15.16:0/2680416271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2800c2e0 con 0x7f7c28004bf0 2019-12-05T06:30:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c46d9d700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c28004110 0x7f7c28004590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:30:47.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4d126700 1 -- 172.21.15.16:0/2680416271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c4809aae0 con 0x7f7c28004bf0 2019-12-05T06:30:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4d126700 1 -- 172.21.15.16:0/2680416271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c48084b90 con 0x7f7c28004bf0 2019-12-05T06:30:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 -- 172.21.15.16:0/2680416271 reap_dead start 2019-12-05T06:30:47.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=READY pgs=912 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:47.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=THROTTLE_DONE pgs=912 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=THROTTLE_DONE pgs=912 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c40000df0 con 0x7f7c28004bf0 2019-12-05T06:30:47.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=THROTTLE_DONE pgs=912 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c4000ac30 con 0x7f7c28004bf0 2019-12-05T06:30:47.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.811+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c4000f470 con 0x7f7c28004bf0 2019-12-05T06:30:47.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.812+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=THROTTLE_DONE pgs=912 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.812+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7c4000edb0 con 0x7f7c28004bf0 2019-12-05T06:30:47.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.812+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=THROTTLE_DONE pgs=912 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.812+0000 7f7c2ffff700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c28026360 0x7f7c280287b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:47.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.812+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f7c40045f90 con 0x7f7c28004bf0 2019-12-05T06:30:47.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.812+0000 7f7c4759e700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c28026360 0x7f7c280287b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:47.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.813+0000 7f7c4759e700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c28026360 0x7f7c280287b0 crc :-1 s=READY pgs=394 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:47.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.818+0000 7f7c4d126700 1 -- 172.21.15.16:0/2680416271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c20002e60 con 0x7f7c28004bf0 2019-12-05T06:30:47.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.822+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=THROTTLE_DONE pgs=912 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:47.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:47.822+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7c4000aa20 con 0x7f7c28004bf0 2019-12-05T06:30:48.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.016+0000 7f7c4d126700 1 -- 172.21.15.16:0/2680416271 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c20001d10 con 0x7f7c28004bf0 2019-12-05T06:30:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.017+0000 7f7c4c925700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=THROTTLE_DONE pgs=912 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.017+0000 7f7c2ffff700 1 -- 172.21.15.16:0/2680416271 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 116 v116) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f7c4000edb0 con 0x7f7c28004bf0 2019-12-05T06:30:48.011 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 116 2019-12-05T06:30:48.011 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:48.012 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":116,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:43.275354+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":21,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.019+0000 7f7c4d126700 1 -- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c28026360 msgr2=0x7f7c280287b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.019+0000 7f7c4d126700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c28026360 0x7f7c280287b0 crc :-1 s=READY pgs=394 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:48.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.019+0000 7f7c4d126700 1 -- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 msgr2=0x7f7c2800b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.019+0000 7f7c4d126700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 secure :-1 s=READY pgs=912 cs=0 l=1 rx=0x7f7c400109b0 tx=0x7f7c40010830).stop 2019-12-05T06:30:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.020+0000 7f7c4d126700 1 -- 172.21.15.16:0/2680416271 shutdown_connections 2019-12-05T06:30:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.020+0000 7f7c4d126700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c28026360 0x7f7c280287b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.020+0000 7f7c4d126700 1 --2- 172.21.15.16:0/2680416271 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c28004bf0 0x7f7c2800b890 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.020+0000 7f7c4d126700 1 -- 172.21.15.16:0/2680416271 shutdown_connections 2019-12-05T06:30:48.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.020+0000 7f7c4d126700 1 -- 172.21.15.16:0/2680416271 wait complete. 2019-12-05T06:30:48.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:48.020+0000 7f7c4d126700 1 -- 172.21.15.16:0/2680416271 >> 172.21.15.16:0/2680416271 conn(0x7f7c480714f0 msgr2=0x7f7c4809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:48.033 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:30:50.034 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:30:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.246+0000 7f4648213700 1 Processor -- start 2019-12-05T06:30:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.246+0000 7f4648213700 1 -- start start 2019-12-05T06:30:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.247+0000 7f4648213700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f464008c6d0 0x7f464008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:50.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.247+0000 7f4648213700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f464008fef0 0x7f464008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:30:50.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.247+0000 7f4648213700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46400958e0 con 0x7f464008ef20 2019-12-05T06:30:50.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.247+0000 7f4648213700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46400a3f70 con 0x7f464008fef0 2019-12-05T06:30:50.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.247+0000 7f4648213700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f464008d830 con 0x7f464008c6d0 2019-12-05T06:30:50.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.247+0000 7f4645faf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f464008fef0 0x7f464008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:50.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.247+0000 7f46467b0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f464008c6d0 0x7f464008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:50.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.247+0000 7f4645faf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f464008fef0 0x7f464008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:41128/0 (socket says 172.21.15.16:41128) 2019-12-05T06:30:50.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.247+0000 7f4645faf700 1 -- 172.21.15.16:0/1981452018 learned_addr learned my addr 172.21.15.16:0/1981452018 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:50.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.247+0000 7f4646fb1700 1 --1- 172.21.15.16:0/1981452018 >> v1:172.21.15.76:6789/0 conn(0x7f464008ef20 0x7f464008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:33280/0 (socket says 172.21.15.16:33280) 2019-12-05T06:30:50.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.248+0000 7f4645faf700 1 -- 172.21.15.16:0/1981452018 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f464008c6d0 msgr2=0x7f464008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:30:50.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.248+0000 7f4645faf700 1 --2- 172.21.15.16:0/1981452018 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f464008c6d0 0x7f464008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:50.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.248+0000 7f4645faf700 1 -- 172.21.15.16:0/1981452018 >> v1:172.21.15.76:6789/0 conn(0x7f464008ef20 legacy=0x7f464008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:50.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.248+0000 7f46457ae700 1 -- 172.21.15.16:0/1981452018 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1707936883 0 0) 0x7f4638000d30 con 0x7f464008ef20 2019-12-05T06:30:50.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.248+0000 7f4645faf700 1 -- 172.21.15.16:0/1981452018 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464008d540 con 0x7f464008fef0 2019-12-05T06:30:50.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.248+0000 7f4645faf700 1 --2- 172.21.15.16:0/1981452018 >> v2:172.21.15.76:3300/0 conn(0x7f464008fef0 0x7f464008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f463400f6c0 tx=0x7f463400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.248+0000 7f4645faf700 1 --2- 172.21.15.16:0/1981452018 >> v2:172.21.15.76:3300/0 conn(0x7f464008fef0 0x7f464008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f463400f6c0 tx=0x7f463400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.248+0000 7f4645faf700 1 --2- 172.21.15.16:0/1981452018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464008fef0 0x7f464008d000 secure :-1 s=READY pgs=659 cs=0 l=1 rx=0x7f463400f6c0 tx=0x7f463400fa60).ready entity=mon.1 client_cookie=f0831c031d3f6ab5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.249+0000 7f4645faf700 1 --2- 172.21.15.16:0/1981452018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464008fef0 0x7f464008d000 secure :-1 s=THROTTLE_DONE pgs=659 cs=0 l=1 rx=0x7f463400f6c0 tx=0x7f463400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.249+0000 7f46457ae700 1 -- 172.21.15.16:0/1981452018 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4634000960 con 0x7f464008fef0 2019-12-05T06:30:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.249+0000 7f4645faf700 1 --2- 172.21.15.16:0/1981452018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464008fef0 0x7f464008d000 secure :-1 s=THROTTLE_DONE pgs=659 cs=0 l=1 rx=0x7f463400f6c0 tx=0x7f463400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.249+0000 7f4645faf700 1 --2- 172.21.15.16:0/1981452018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464008fef0 0x7f464008d000 secure :-1 s=THROTTLE_DONE pgs=659 cs=0 l=1 rx=0x7f463400f6c0 tx=0x7f463400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.249+0000 7f46457ae700 1 -- 172.21.15.16:0/1981452018 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4634007210 con 0x7f464008fef0 2019-12-05T06:30:50.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.249+0000 7f46457ae700 1 -- 172.21.15.16:0/1981452018 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4634000e80 con 0x7f464008fef0 2019-12-05T06:30:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.249+0000 7f4648213700 1 -- 172.21.15.16:0/1981452018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464008fef0 msgr2=0x7f464008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.249+0000 7f4648213700 1 --2- 172.21.15.16:0/1981452018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464008fef0 0x7f464008d000 secure :-1 s=READY pgs=659 cs=0 l=1 rx=0x7f463400f6c0 tx=0x7f463400fa60).stop 2019-12-05T06:30:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.250+0000 7f4648213700 1 -- 172.21.15.16:0/1981452018 shutdown_connections 2019-12-05T06:30:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.250+0000 7f4648213700 1 --2- 172.21.15.16:0/1981452018 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f464008fef0 0x7f464008d000 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.250+0000 7f4648213700 1 --2- 172.21.15.16:0/1981452018 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f464008c6d0 0x7f464008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:30:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.250+0000 7f4648213700 1 -- 172.21.15.16:0/1981452018 shutdown_connections 2019-12-05T06:30:50.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.250+0000 7f4648213700 1 -- 172.21.15.16:0/1981452018 wait complete. 2019-12-05T06:30:50.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.250+0000 7f4648213700 1 -- 172.21.15.16:0/1981452018 >> 172.21.15.16:0/1981452018 conn(0x7f4640061940 msgr2=0x7f4640096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:50.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.253+0000 7f4648213700 1 Processor -- start 2019-12-05T06:30:50.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.253+0000 7f4648213700 1 -- start start 2019-12-05T06:30:50.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.254+0000 7f4648213700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4640128360 0x7f4640128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:50.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.254+0000 7f4648213700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4640137ea0 0x7f4640128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:50.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.254+0000 7f4648213700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4640099720 con 0x7f4640152760 2019-12-05T06:30:50.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.254+0000 7f4648213700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f464012a640 con 0x7f4640128360 2019-12-05T06:30:50.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.254+0000 7f4648213700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4640129c10 con 0x7f4640137ea0 2019-12-05T06:30:50.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.254+0000 7f4645faf700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4640137ea0 0x7f4640128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:50.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.254+0000 7f46467b0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4640128360 0x7f4640128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:50.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.254+0000 7f4645faf700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4640137ea0 0x7f4640128c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:59036/0 (socket says 172.21.15.16:59036) 2019-12-05T06:30:50.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.254+0000 7f4645faf700 1 -- 172.21.15.16:0/1069499203 learned_addr learned my addr 172.21.15.16:0/1069499203 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:30:50.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f46467b0700 1 -- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4640137ea0 msgr2=0x7f4640128c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:50.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4640137ea0 0x7f4640128c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f46467b0700 1 -- 172.21.15.16:0/1069499203 >> v1:172.21.15.76:6789/0 conn(0x7f4640152760 legacy=0x7f4640152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f46467b0700 1 -- 172.21.15.16:0/1069499203 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f464012acd0 con 0x7f4640128360 2019-12-05T06:30:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> v2:172.21.15.76:3300/0 conn(0x7f4640128360 0x7f4640128750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> v2:172.21.15.76:3300/0 conn(0x7f4640128360 0x7f4640128750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 0x7f4640128750 secure :-1 s=READY pgs=661 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 0x7f4640128750 secure :-1 s=THROTTLE_DONE pgs=661 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f463effd700 1 -- 172.21.15.16:0/1069499203 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4630009640 con 0x7f4640128360 2019-12-05T06:30:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 0x7f4640128750 secure :-1 s=THROTTLE_DONE pgs=661 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 0x7f4640128750 secure :-1 s=THROTTLE_DONE pgs=661 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f463effd700 1 -- 172.21.15.16:0/1069499203 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4630000940 con 0x7f4640128360 2019-12-05T06:30:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f463effd700 1 -- 172.21.15.16:0/1069499203 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f463000a0e0 con 0x7f4640128360 2019-12-05T06:30:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f4648213700 1 -- 172.21.15.16:0/1069499203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f464012acd0 con 0x7f4640128360 2019-12-05T06:30:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.255+0000 7f4648213700 1 -- 172.21.15.16:0/1069499203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f464012acd0 con 0x7f4640128360 2019-12-05T06:30:50.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.256+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 0x7f4640128750 secure :-1 s=THROTTLE_DONE pgs=661 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.256+0000 7f463effd700 1 -- 172.21.15.16:0/1069499203 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4630009a70 con 0x7f4640128360 2019-12-05T06:30:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.256+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 0x7f4640128750 secure :-1 s=THROTTLE_DONE pgs=661 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.257+0000 7f463effd700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4620023630 0x7f4620025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:30:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.257+0000 7f463effd700 1 -- 172.21.15.16:0/1069499203 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f4630045d80 con 0x7f4640128360 2019-12-05T06:30:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.257+0000 7f4645faf700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4620023630 0x7f4620025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:30:50.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.257+0000 7f4645faf700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4620023630 0x7f4620025ac0 crc :-1 s=READY pgs=396 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:30:50.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.262+0000 7f4648213700 1 -- 172.21.15.16:0/1069499203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f464012acd0 con 0x7f4640128360 2019-12-05T06:30:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.270+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 0x7f4640128750 secure :-1 s=THROTTLE_DONE pgs=661 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.270+0000 7f463effd700 1 -- 172.21.15.16:0/1069499203 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4630009840 con 0x7f4640128360 2019-12-05T06:30:50.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.473+0000 7f4648213700 1 -- 172.21.15.16:0/1069499203 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f464012acd0 con 0x7f4640128360 2019-12-05T06:30:50.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.474+0000 7f46467b0700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 0x7f4640128750 secure :-1 s=THROTTLE_DONE pgs=661 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:30:50.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.474+0000 7f463effd700 1 -- 172.21.15.16:0/1069499203 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f4630009a50 con 0x7f4640128360 2019-12-05T06:30:50.469 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 117 2019-12-05T06:30:50.469 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:30:50.469 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6561,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:43.275354+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":21,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:30:50.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 -- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4620023630 msgr2=0x7f4620025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4620023630 0x7f4620025ac0 crc :-1 s=READY pgs=396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 -- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 msgr2=0x7f4640128750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:30:50.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 0x7f4640128750 secure :-1 s=READY pgs=661 cs=0 l=1 rx=0x7f4630010390 tx=0x7f463000b360).stop 2019-12-05T06:30:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 -- 172.21.15.16:0/1069499203 shutdown_connections 2019-12-05T06:30:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4620023630 0x7f4620025ac0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4640137ea0 0x7f4640128c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:50.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 --2- 172.21.15.16:0/1069499203 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4640128360 0x7f4640128750 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:30:50.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 -- 172.21.15.16:0/1069499203 shutdown_connections 2019-12-05T06:30:50.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 -- 172.21.15.16:0/1069499203 wait complete. 2019-12-05T06:30:50.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:30:50.477+0000 7f4648213700 1 -- 172.21.15.16:0/1069499203 >> 172.21.15.16:0/1069499203 conn(0x7f464006d260 msgr2=0x7f464009aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:30:50.482 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:30:50.482 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 53 secs before thrashing 2019-12-05T06:31:05.775 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:31:05.779 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:31:35.873 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:31:35.878 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:31:43.484 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:43.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.634+0000 7f982adbe700 1 Processor -- start 2019-12-05T06:31:43.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.635+0000 7f982adbe700 1 -- start start 2019-12-05T06:31:43.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.635+0000 7f982adbe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f982408d2c0 0x7f982408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:43.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.635+0000 7f982adbe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f982408fef0 0x7f982408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:43.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.635+0000 7f982adbe700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98240958e0 con 0x7f982408c6d0 2019-12-05T06:31:43.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.635+0000 7f982adbe700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f982408df90 con 0x7f982408d2c0 2019-12-05T06:31:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.635+0000 7f982adbe700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98240a3f70 con 0x7f982408fef0 2019-12-05T06:31:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.635+0000 7f982935b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f982408c6d0 0x7f982408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40454/0 (socket says 172.21.15.16:40454) 2019-12-05T06:31:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f982935b700 1 -- 172.21.15.16:0/1996799816 learned_addr learned my addr 172.21.15.16:0/1996799816 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9829b5c700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f982408fef0 0x7f982408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9828b5a700 1 --2- 172.21.15.16:0/1996799816 >> v2:172.21.15.76:3300/0 conn(0x7f982408d2c0 0x7f982408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3062397837 0 0) 0x7f9814001150 con 0x7f982408c6d0 2019-12-05T06:31:43.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f980c002e80 con 0x7f982408c6d0 2019-12-05T06:31:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3677811652 0 0) 0x7f9814001130 con 0x7f982408c6d0 2019-12-05T06:31:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f982408fef0 msgr2=0x7f982408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f982408fef0 0x7f982408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 >> v2:172.21.15.76:3300/0 conn(0x7f982408d2c0 msgr2=0x7f982408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 --2- 172.21.15.16:0/1996799816 >> v2:172.21.15.76:3300/0 conn(0x7f982408d2c0 0x7f982408ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:43.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982408e620 con 0x7f982408c6d0 2019-12-05T06:31:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f9814001660 con 0x7f982408c6d0 2019-12-05T06:31:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 >> v1:172.21.15.16:6789/0 conn(0x7f982408c6d0 legacy=0x7f982408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f980c047a80 0x7f980c049ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f980c04a690 0x7f980c04cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f980c04d3a0 0x7f980c04f840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c04fff0 con 0x7f980c04a690 2019-12-05T06:31:43.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980c0501e0 con 0x7f980c047a80 2019-12-05T06:31:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.636+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f980c050410 con 0x7f980c04d3a0 2019-12-05T06:31:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f980c04a690 0x7f980c04cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f982935b700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f980c047a80 0x7f980c049ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:43.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9829b5c700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f980c04d3a0 0x7f980c04f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 -- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f980c04d3a0 msgr2=0x7f980c04f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f980c04d3a0 0x7f980c04f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 -- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f980c047a80 msgr2=0x7f980c049ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f980c047a80 0x7f980c049ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 -- 172.21.15.16:0/1996799816 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f980c0506c0 con 0x7f980c04a690 2019-12-05T06:31:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9829b5c700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f980c04d3a0 0x7f980c04f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:31:43.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f980c04a690 0x7f980c04cb40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f981800b6e0 tx=0x7f981800b5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f980c04a690 0x7f980c04cb40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f981800b6e0 tx=0x7f981800b5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f980c04a690 0x7f980c04cb40 secure :-1 s=READY pgs=919 cs=0 l=1 rx=0x7f981800b6e0 tx=0x7f981800b5b0).ready entity=mon.0 client_cookie=b9349fcef6ffea00 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f980c04a690 0x7f980c04cb40 secure :-1 s=THROTTLE_DONE pgs=919 cs=0 l=1 rx=0x7f981800b6e0 tx=0x7f981800b5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f980c04a690 0x7f980c04cb40 secure :-1 s=THROTTLE_DONE pgs=919 cs=0 l=1 rx=0x7f981800b6e0 tx=0x7f981800b5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9818000960 con 0x7f980c04a690 2019-12-05T06:31:43.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9828b5a700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f980c04a690 0x7f980c04cb40 secure :-1 s=THROTTLE_DONE pgs=919 cs=0 l=1 rx=0x7f981800b6e0 tx=0x7f981800b5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9818009980 con 0x7f980c04a690 2019-12-05T06:31:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9823fff700 1 -- 172.21.15.16:0/1996799816 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f981800a050 con 0x7f980c04a690 2019-12-05T06:31:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f982adbe700 1 -- 172.21.15.16:0/1996799816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f980c04a690 msgr2=0x7f980c04cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f982adbe700 1 --2- 172.21.15.16:0/1996799816 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f980c04a690 0x7f980c04cb40 secure :-1 s=READY pgs=919 cs=0 l=1 rx=0x7f981800b6e0 tx=0x7f981800b5b0).stop 2019-12-05T06:31:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.637+0000 7f9829b5c700 1 -- 172.21.15.16:0/1996799816 reap_dead start 2019-12-05T06:31:43.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.638+0000 7f982adbe700 1 -- 172.21.15.16:0/1996799816 shutdown_connections 2019-12-05T06:31:43.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.638+0000 7f982adbe700 1 -- 172.21.15.16:0/1996799816 shutdown_connections 2019-12-05T06:31:43.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.638+0000 7f982adbe700 1 -- 172.21.15.16:0/1996799816 wait complete. 2019-12-05T06:31:43.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.638+0000 7f982adbe700 1 -- 172.21.15.16:0/1996799816 >> 172.21.15.16:0/1996799816 conn(0x7f9824061940 msgr2=0x7f9824096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:43.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.641+0000 7f982adbe700 1 Processor -- start 2019-12-05T06:31:43.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.641+0000 7f982adbe700 1 -- start start 2019-12-05T06:31:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f982adbe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f982412b640 0x7f982412ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f982adbe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9824153b50 0x7f9824155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f982adbe700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9824131bf0 con 0x7f98241302f0 2019-12-05T06:31:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f982adbe700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98241296e0 con 0x7f9824153b50 2019-12-05T06:31:43.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f982adbe700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9824128cb0 con 0x7f982412b640 2019-12-05T06:31:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9824153b50 0x7f9824155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9824153b50 0x7f9824155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55302/0 (socket says 172.21.15.16:55302) 2019-12-05T06:31:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 -- 172.21.15.16:0/2278151800 learned_addr learned my addr 172.21.15.16:0/2278151800 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:31:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9828b5a700 1 --2- 172.21.15.16:0/2278151800 >> v2:172.21.15.76:3300/0 conn(0x7f982412b640 0x7f982412ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 -- 172.21.15.16:0/2278151800 >> v2:172.21.15.76:3300/0 conn(0x7f982412b640 msgr2=0x7f982412ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:43.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> v2:172.21.15.76:3300/0 conn(0x7f982412b640 0x7f982412ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 -- 172.21.15.16:0/2278151800 >> v1:172.21.15.76:6789/0 conn(0x7f98241302f0 legacy=0x7f982412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 -- 172.21.15.16:0/2278151800 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9824129d70 con 0x7f9824153b50 2019-12-05T06:31:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> v2:172.21.15.16:3300/0 conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> v2:172.21.15.16:3300/0 conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=READY pgs=920 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9821ffb700 1 -- 172.21.15.16:0/2278151800 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f981c009930 con 0x7f9824153b50 2019-12-05T06:31:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.642+0000 7f9821ffb700 1 -- 172.21.15.16:0/2278151800 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f981c002390 con 0x7f9824153b50 2019-12-05T06:31:43.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.643+0000 7f9821ffb700 1 -- 172.21.15.16:0/2278151800 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f981c002a60 con 0x7f9824153b50 2019-12-05T06:31:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.643+0000 7f982adbe700 1 -- 172.21.15.16:0/2278151800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9824129d70 con 0x7f9824153b50 2019-12-05T06:31:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.643+0000 7f982adbe700 1 -- 172.21.15.16:0/2278151800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9824129d70 con 0x7f9824153b50 2019-12-05T06:31:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.643+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.644+0000 7f9821ffb700 1 -- 172.21.15.16:0/2278151800 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f981c001f90 con 0x7f9824153b50 2019-12-05T06:31:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.644+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.644+0000 7f9821ffb700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9804023630 0x7f9804025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.644+0000 7f9821ffb700 1 -- 172.21.15.16:0/2278151800 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f981c0483d0 con 0x7f9824153b50 2019-12-05T06:31:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.644+0000 7f982935b700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9804023630 0x7f9804025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.645+0000 7f982935b700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9804023630 0x7f9804025ac0 crc :-1 s=READY pgs=398 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:43.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.650+0000 7f982adbe700 1 -- 172.21.15.16:0/2278151800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9824129d70 con 0x7f9824153b50 2019-12-05T06:31:43.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.654+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.654+0000 7f9821ffb700 1 -- 172.21.15.16:0/2278151800 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f981c002c60 con 0x7f9824153b50 2019-12-05T06:31:43.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.855+0000 7f982adbe700 1 -- 172.21.15.16:0/2278151800 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9824129d70 con 0x7f9824153b50 2019-12-05T06:31:43.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.856+0000 7f9829b5c700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:43.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.856+0000 7f9821ffb700 1 -- 172.21.15.16:0/2278151800 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f981c001f90 con 0x7f9824153b50 2019-12-05T06:31:43.852 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 117 2019-12-05T06:31:43.852 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:31:43.852 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6561,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:43.275354+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":21,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 -- 172.21.15.16:0/2278151800 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9804023630 msgr2=0x7f9804025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:43.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9804023630 0x7f9804025ac0 crc :-1 s=READY pgs=398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:43.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 -- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 msgr2=0x7f9824155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:43.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 0x7f9824155f50 secure :-1 s=READY pgs=920 cs=0 l=1 rx=0x7f981c012a10 tx=0x7f981c012810).stop 2019-12-05T06:31:43.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 -- 172.21.15.16:0/2278151800 shutdown_connections 2019-12-05T06:31:43.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9804023630 0x7f9804025ac0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:43.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 --2- 172.21.15.16:0/2278151800 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9824153b50 0x7f9824155f50 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:43.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 --2- 172.21.15.16:0/2278151800 >> v2:172.21.15.76:3300/0 conn(0x7f982412b640 0x7f982412ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:43.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 -- 172.21.15.16:0/2278151800 shutdown_connections 2019-12-05T06:31:43.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 -- 172.21.15.16:0/2278151800 wait complete. 2019-12-05T06:31:43.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:43.859+0000 7f982adbe700 1 -- 172.21.15.16:0/2278151800 >> 172.21.15.16:0/2278151800 conn(0x7f98240a2850 msgr2=0x7f982406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:43.864 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:31:43.865 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:31:43.865 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:31:43.874 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:31:43.874 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:44.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.022+0000 7fd0b4d70700 1 Processor -- start 2019-12-05T06:31:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.022+0000 7fd0b4d70700 1 -- start start 2019-12-05T06:31:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.024+0000 7fd0b4d70700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 0x7fd0b008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.024+0000 7fd0b4d70700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd0b008f720 0x7fd0b008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.024+0000 7fd0b4d70700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b00958e0 con 0x7fd0b008fef0 2019-12-05T06:31:44.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.024+0000 7fd0b4d70700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b00a3f70 con 0x7fd0b008f720 2019-12-05T06:31:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.024+0000 7fd0b4d70700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0b008d830 con 0x7fd0b008c6d0 2019-12-05T06:31:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.024+0000 7fd0ae59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd0b008fef0 0x7fd0b008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40482/0 (socket says 172.21.15.16:40482) 2019-12-05T06:31:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.024+0000 7fd0ae59c700 1 -- 172.21.15.16:0/905639454 learned_addr learned my addr 172.21.15.16:0/905639454 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.024+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/905639454 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 0x7fd0b008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.024+0000 7fd0af59e700 1 --2- 172.21.15.16:0/905639454 >> v2:172.21.15.76:3300/0 conn(0x7fd0b008f720 0x7fd0b008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:44.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.024+0000 7fd0add9b700 1 -- 172.21.15.16:0/905639454 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3404354679 0 0) 0x7fd0a4001100 con 0x7fd0b008fef0 2019-12-05T06:31:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.025+0000 7fd0add9b700 1 -- 172.21.15.16:0/905639454 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd098002e80 con 0x7fd0b008fef0 2019-12-05T06:31:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.025+0000 7fd0aed9d700 1 -- 172.21.15.16:0/905639454 >> v2:172.21.15.76:3300/0 conn(0x7fd0b008f720 msgr2=0x7fd0b008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.025+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/905639454 >> v2:172.21.15.76:3300/0 conn(0x7fd0b008f720 0x7fd0b008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.025+0000 7fd0aed9d700 1 -- 172.21.15.16:0/905639454 >> v1:172.21.15.16:6789/0 conn(0x7fd0b008fef0 legacy=0x7fd0b008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.025+0000 7fd0aed9d700 1 -- 172.21.15.16:0/905639454 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b008fbc0 con 0x7fd0b008c6d0 2019-12-05T06:31:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.025+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/905639454 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 0x7fd0b008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd0a000b2d0 tx=0x7fd0a000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.025+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/905639454 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 0x7fd0b008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd0a000b2d0 tx=0x7fd0a000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.025+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/905639454 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 0x7fd0b008cac0 secure :-1 s=READY pgs=216 cs=0 l=1 rx=0x7fd0a000b2d0 tx=0x7fd0a000b5d0).ready entity=mon.2 client_cookie=8480ca2a34b04199 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.025+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/905639454 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 0x7fd0b008cac0 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7fd0a000b2d0 tx=0x7fd0a000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.025+0000 7fd0add9b700 1 -- 172.21.15.16:0/905639454 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0a0007590 con 0x7fd0b008c6d0 2019-12-05T06:31:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.026+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/905639454 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 0x7fd0b008cac0 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7fd0a000b2d0 tx=0x7fd0a000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.026+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/905639454 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 0x7fd0b008cac0 secure :-1 s=THROTTLE_DONE pgs=216 cs=0 l=1 rx=0x7fd0a000b2d0 tx=0x7fd0a000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.026+0000 7fd0add9b700 1 -- 172.21.15.16:0/905639454 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a0009d30 con 0x7fd0b008c6d0 2019-12-05T06:31:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.026+0000 7fd0add9b700 1 -- 172.21.15.16:0/905639454 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0a000a400 con 0x7fd0b008c6d0 2019-12-05T06:31:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.026+0000 7fd0b4d70700 1 -- 172.21.15.16:0/905639454 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 msgr2=0x7fd0b008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.026+0000 7fd0b4d70700 1 --2- 172.21.15.16:0/905639454 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 0x7fd0b008cac0 secure :-1 s=READY pgs=216 cs=0 l=1 rx=0x7fd0a000b2d0 tx=0x7fd0a000b5d0).stop 2019-12-05T06:31:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.027+0000 7fd0b4d70700 1 -- 172.21.15.16:0/905639454 shutdown_connections 2019-12-05T06:31:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.027+0000 7fd0b4d70700 1 --2- 172.21.15.16:0/905639454 >> v2:172.21.15.76:3300/0 conn(0x7fd0b008f720 0x7fd0b008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:44.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.027+0000 7fd0b4d70700 1 --2- 172.21.15.16:0/905639454 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd0b008c6d0 0x7fd0b008cac0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.027+0000 7fd0b4d70700 1 -- 172.21.15.16:0/905639454 shutdown_connections 2019-12-05T06:31:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.027+0000 7fd0b4d70700 1 -- 172.21.15.16:0/905639454 wait complete. 2019-12-05T06:31:44.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.027+0000 7fd0b4d70700 1 -- 172.21.15.16:0/905639454 >> 172.21.15.16:0/905639454 conn(0x7fd0b0061940 msgr2=0x7fd0b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:44.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.032+0000 7fd0b4d70700 1 Processor -- start 2019-12-05T06:31:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.032+0000 7fd0b4d70700 1 -- start start 2019-12-05T06:31:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0b4d70700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd0b006dc50 0x7fd0b006e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0b4d70700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0b0134c80 0x7fd0b006e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0b4d70700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0b01350e0 con 0x7fd0b0135450 2019-12-05T06:31:44.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0b4d70700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b0132830 con 0x7fd0b0134c80 2019-12-05T06:31:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0b4d70700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0b01334f0 con 0x7fd0b006dc50 2019-12-05T06:31:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0af59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0b0134c80 0x7fd0b006e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0af59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd0b0134c80 0x7fd0b006e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55318/0 (socket says 172.21.15.16:55318) 2019-12-05T06:31:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0af59e700 1 -- 172.21.15.16:0/355168047 learned_addr learned my addr 172.21.15.16:0/355168047 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:31:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/355168047 >> v2:172.21.15.76:3300/0 conn(0x7fd0b006dc50 0x7fd0b006e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:44.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0af59e700 1 -- 172.21.15.16:0/355168047 >> v2:172.21.15.76:3300/0 conn(0x7fd0b006dc50 msgr2=0x7fd0b006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> v2:172.21.15.76:3300/0 conn(0x7fd0b006dc50 0x7fd0b006e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0af59e700 1 -- 172.21.15.16:0/355168047 >> v1:172.21.15.76:6789/0 conn(0x7fd0b0135450 legacy=0x7fd0b0134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0af59e700 1 -- 172.21.15.16:0/355168047 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0b0133b80 con 0x7fd0b0134c80 2019-12-05T06:31:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> v2:172.21.15.16:3300/0 conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> v2:172.21.15.16:3300/0 conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.034+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=READY pgs=922 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.035+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.035+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.035+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.035+0000 7fd09f7fe700 1 -- 172.21.15.16:0/355168047 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0a800f090 con 0x7fd0b0134c80 2019-12-05T06:31:44.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.035+0000 7fd09f7fe700 1 -- 172.21.15.16:0/355168047 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0a8000a70 con 0x7fd0b0134c80 2019-12-05T06:31:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.035+0000 7fd09f7fe700 1 -- 172.21.15.16:0/355168047 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0a800f670 con 0x7fd0b0134c80 2019-12-05T06:31:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.035+0000 7fd0b4d70700 1 -- 172.21.15.16:0/355168047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0b0133b80 con 0x7fd0b0134c80 2019-12-05T06:31:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.035+0000 7fd0b4d70700 1 -- 172.21.15.16:0/355168047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0b0133b80 con 0x7fd0b0134c80 2019-12-05T06:31:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.035+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.036+0000 7fd09f7fe700 1 -- 172.21.15.16:0/355168047 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd0a800ed50 con 0x7fd0b0134c80 2019-12-05T06:31:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.036+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.036+0000 7fd09f7fe700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd094023630 0x7fd094025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.036+0000 7fd09f7fe700 1 -- 172.21.15.16:0/355168047 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fd0a8045e10 con 0x7fd0b0134c80 2019-12-05T06:31:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.036+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd094023630 0x7fd094025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:44.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.037+0000 7fd0aed9d700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd094023630 0x7fd094025ac0 crc :-1 s=READY pgs=399 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:44.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.053+0000 7fd0b4d70700 1 -- 172.21.15.16:0/355168047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0b0133b80 con 0x7fd0b0134c80 2019-12-05T06:31:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.059+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.059+0000 7fd09f7fe700 1 -- 172.21.15.16:0/355168047 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd0a800eaa0 con 0x7fd0b0134c80 2019-12-05T06:31:44.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.299+0000 7fd0b4d70700 1 -- 172.21.15.16:0/355168047 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd0b0133b80 con 0x7fd0b0134c80 2019-12-05T06:31:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.300+0000 7fd0af59e700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=THROTTLE_DONE pgs=922 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.300+0000 7fd09f7fe700 1 -- 172.21.15.16:0/355168047 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fd0a800ec90 con 0x7fd0b0134c80 2019-12-05T06:31:44.296 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 117 2019-12-05T06:31:44.297 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:31:44.297 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6561,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:43.275354+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":21,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:44.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 -- 172.21.15.16:0/355168047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd094023630 msgr2=0x7fd094025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:44.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd094023630 0x7fd094025ac0 crc :-1 s=READY pgs=399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:44.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 -- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 msgr2=0x7fd0b006e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:44.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 0x7fd0b006e580 secure :-1 s=READY pgs=922 cs=0 l=1 rx=0x7fd0a8010450 tx=0x7fd0a8010280).stop 2019-12-05T06:31:44.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 -- 172.21.15.16:0/355168047 shutdown_connections 2019-12-05T06:31:44.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd094023630 0x7fd094025ac0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 --2- 172.21.15.16:0/355168047 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd0b0134c80 0x7fd0b006e580 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 --2- 172.21.15.16:0/355168047 >> v2:172.21.15.76:3300/0 conn(0x7fd0b006dc50 0x7fd0b006e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 -- 172.21.15.16:0/355168047 shutdown_connections 2019-12-05T06:31:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 -- 172.21.15.16:0/355168047 wait complete. 2019-12-05T06:31:44.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:44.303+0000 7fd0b4d70700 1 -- 172.21.15.16:0/355168047 >> 172.21.15.16:0/355168047 conn(0x7fd0b00a2850 msgr2=0x7fd0b006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:44.315 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:31:46.317 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.563+0000 7f084df6d700 1 Processor -- start 2019-12-05T06:31:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.564+0000 7f084df6d700 1 -- start start 2019-12-05T06:31:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f084df6d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f084808c6d0 0x7f084808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:46.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f084df6d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f084808fb80 con 0x7f084808f720 2019-12-05T06:31:46.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f084df6d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08480958e0 con 0x7f084808fef0 2019-12-05T06:31:46.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f084df6d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f084808df90 con 0x7f084808c6d0 2019-12-05T06:31:46.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f0847fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f084808c6d0 0x7f084808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:46.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f084cd0b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f084808f720 0x7f08481533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40506/0 (socket says 172.21.15.16:40506) 2019-12-05T06:31:46.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f0847fff700 1 -- >> v2:172.21.15.16:3300/0 conn(0x7f084808c6d0 msgr2=0x7f084808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T06:31:46.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f0847fff700 1 -- >> v2:172.21.15.16:3300/0 conn(0x7f084808c6d0 msgr2=0x7f084808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:31:46.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f0847fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f084808c6d0 0x7f084808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:31:46.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f084cd0b700 1 -- 172.21.15.16:0/3911364358 learned_addr learned my addr 172.21.15.16:0/3911364358 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:46.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.565+0000 7f0847fff700 1 --2- 172.21.15.16:0/3911364358 >> v2:172.21.15.16:3300/0 conn(0x7f084808c6d0 0x7f084808cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:31:46.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1385220734 0 0) 0x7f0840001310 con 0x7f084808f720 2019-12-05T06:31:46.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0830002e80 con 0x7f084808f720 2019-12-05T06:31:46.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233913421 0 0) 0x7f083c001080 con 0x7f084808fef0 2019-12-05T06:31:46.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f08300030b0 con 0x7f084808fef0 2019-12-05T06:31:46.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 386303837 0 0) 0x7f08400012f0 con 0x7f084808f720 2019-12-05T06:31:46.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 >> v2:172.21.15.16:3300/0 conn(0x7f084808c6d0 msgr2=0x7f084808cac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:31:46.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 --2- 172.21.15.16:0/3911364358 >> v2:172.21.15.16:3300/0 conn(0x7f084808c6d0 0x7f084808cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:46.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 >> v1:172.21.15.76:6789/0 conn(0x7f084808fef0 legacy=0x7f084808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:46.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f084808e620 con 0x7f084808f720 2019-12-05T06:31:46.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f0840001880 con 0x7f084808f720 2019-12-05T06:31:46.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 >> v1:172.21.15.16:6789/0 conn(0x7f084808f720 legacy=0x7f08481533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:46.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0830003b80 0x7f0830005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:46.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08300066d0 0x7f0830008b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:46.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 0x7f083000b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:46.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083004fff0 con 0x7f08300093b0 2019-12-05T06:31:46.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0830050220 con 0x7f08300066d0 2019-12-05T06:31:46.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0830050450 con 0x7f0830003b80 2019-12-05T06:31:46.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.566+0000 7f084cd0b700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 0x7f083000b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:46.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f0847fff700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0830003b80 0x7f0830005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:46.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f08477fe700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08300066d0 0x7f0830008b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:46.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f084cd0b700 1 -- 172.21.15.16:0/3911364358 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0830003b80 msgr2=0x7f0830005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:46.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f084cd0b700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0830003b80 0x7f0830005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:46.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f084cd0b700 1 -- 172.21.15.16:0/3911364358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08300066d0 msgr2=0x7f0830008b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:46.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f084cd0b700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f08300066d0 0x7f0830008b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:46.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f084cd0b700 1 -- 172.21.15.16:0/3911364358 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0830050700 con 0x7f08300093b0 2019-12-05T06:31:46.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f084cd0b700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 0x7f083000b860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f084000f890 tx=0x7f0840001dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f084cd0b700 1 -- 172.21.15.16:0/3911364358 reap_dead start 2019-12-05T06:31:46.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f084cd0b700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 0x7f083000b860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f084000f890 tx=0x7f0840001dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.567+0000 7f084cd0b700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 0x7f083000b860 secure :-1 s=READY pgs=925 cs=0 l=1 rx=0x7f084000f890 tx=0x7f0840001dc0).ready entity=mon.0 client_cookie=848216485f039c46 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:46.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.568+0000 7f084cd0b700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 0x7f083000b860 secure :-1 s=THROTTLE_DONE pgs=925 cs=0 l=1 rx=0x7f084000f890 tx=0x7f0840001dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.568+0000 7f084cd0b700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 0x7f083000b860 secure :-1 s=THROTTLE_DONE pgs=925 cs=0 l=1 rx=0x7f084000f890 tx=0x7f0840001dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.568+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0840004240 con 0x7f08300093b0 2019-12-05T06:31:46.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.568+0000 7f084cd0b700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 0x7f083000b860 secure :-1 s=THROTTLE_DONE pgs=925 cs=0 l=1 rx=0x7f084000f890 tx=0x7f0840001dc0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.568+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08400012f0 con 0x7f08300093b0 2019-12-05T06:31:46.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.568+0000 7f0846ffd700 1 -- 172.21.15.16:0/3911364358 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0840004820 con 0x7f08300093b0 2019-12-05T06:31:46.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.568+0000 7f084df6d700 1 -- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 msgr2=0x7f083000b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:46.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.568+0000 7f084df6d700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 0x7f083000b860 secure :-1 s=READY pgs=925 cs=0 l=1 rx=0x7f084000f890 tx=0x7f0840001dc0).stop 2019-12-05T06:31:46.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.569+0000 7f084df6d700 1 -- 172.21.15.16:0/3911364358 shutdown_connections 2019-12-05T06:31:46.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.569+0000 7f084df6d700 1 --2- 172.21.15.16:0/3911364358 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08300093b0 0x7f083000b860 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:46.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.569+0000 7f084df6d700 1 -- 172.21.15.16:0/3911364358 shutdown_connections 2019-12-05T06:31:46.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.569+0000 7f084df6d700 1 -- 172.21.15.16:0/3911364358 wait complete. 2019-12-05T06:31:46.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.569+0000 7f084df6d700 1 -- 172.21.15.16:0/3911364358 >> 172.21.15.16:0/3911364358 conn(0x7f0848061940 msgr2=0x7f0848096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:46.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.574+0000 7f084df6d700 1 Processor -- start 2019-12-05T06:31:46.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.574+0000 7f084df6d700 1 -- start start 2019-12-05T06:31:46.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.575+0000 7f084df6d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:46.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.575+0000 7f084df6d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f084812b640 0x7f084812abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:46.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.575+0000 7f084df6d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0848131bf0 con 0x7f084812b0f0 2019-12-05T06:31:46.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.575+0000 7f084df6d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0848128cb0 con 0x7f084812b640 2019-12-05T06:31:46.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.575+0000 7f084df6d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0848128f80 con 0x7f08481302f0 2019-12-05T06:31:46.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.576+0000 7f084cd0b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f084812b0f0 0x7f08481306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40516/0 (socket says 172.21.15.16:40516) 2019-12-05T06:31:46.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.576+0000 7f084cd0b700 1 -- 172.21.15.16:0/1706631423 learned_addr learned my addr 172.21.15.16:0/1706631423 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:46.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.576+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:46.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.576+0000 7f08477fe700 1 --2- 172.21.15.16:0/1706631423 >> v2:172.21.15.76:3300/0 conn(0x7f084812b640 0x7f084812abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:46.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.576+0000 7f0844ff9700 1 -- 172.21.15.16:0/1706631423 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574590832 0 0) 0x7f0840000af0 con 0x7f084812b0f0 2019-12-05T06:31:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.576+0000 7f0844ff9700 1 -- 172.21.15.16:0/1706631423 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0828002e80 con 0x7f084812b0f0 2019-12-05T06:31:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.576+0000 7f0847fff700 1 -- 172.21.15.16:0/1706631423 >> v2:172.21.15.76:3300/0 conn(0x7f084812b640 msgr2=0x7f084812abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.576+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> v2:172.21.15.76:3300/0 conn(0x7f084812b640 0x7f084812abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.576+0000 7f0847fff700 1 -- 172.21.15.16:0/1706631423 >> v1:172.21.15.16:6789/0 conn(0x7f084812b0f0 legacy=0x7f08481306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0844ff9700 1 -- 172.21.15.16:0/1706631423 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1426215800 0 0) 0x7f0840001c20 con 0x7f084812b0f0 2019-12-05T06:31:46.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0847fff700 1 -- 172.21.15.16:0/1706631423 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0848129d70 con 0x7f08481302f0 2019-12-05T06:31:46.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=READY pgs=217 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:46.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=THROTTLE_DONE pgs=217 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0844ff9700 1 -- 172.21.15.16:0/1706631423 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0838000ed0 con 0x7f08481302f0 2019-12-05T06:31:46.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=THROTTLE_DONE pgs=217 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=THROTTLE_DONE pgs=217 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0844ff9700 1 -- 172.21.15.16:0/1706631423 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08380011f0 con 0x7f08481302f0 2019-12-05T06:31:46.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f084df6d700 1 -- 172.21.15.16:0/1706631423 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0848129d70 con 0x7f08481302f0 2019-12-05T06:31:46.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f0844ff9700 1 -- 172.21.15.16:0/1706631423 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08380018c0 con 0x7f08481302f0 2019-12-05T06:31:46.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.577+0000 7f084df6d700 1 -- 172.21.15.16:0/1706631423 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0848129d70 con 0x7f08481302f0 2019-12-05T06:31:46.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.578+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=THROTTLE_DONE pgs=217 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.578+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=THROTTLE_DONE pgs=217 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.578+0000 7f0844ff9700 1 -- 172.21.15.16:0/1706631423 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0838000d60 con 0x7f08481302f0 2019-12-05T06:31:46.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.579+0000 7f0844ff9700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0828023680 0x7f0828025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:46.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.579+0000 7f0844ff9700 1 -- 172.21.15.16:0/1706631423 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f08380460d0 con 0x7f08481302f0 2019-12-05T06:31:46.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.579+0000 7f08477fe700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0828023680 0x7f0828025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:46.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.580+0000 7f08477fe700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0828023680 0x7f0828025af0 crc :-1 s=READY pgs=400 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:46.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.595+0000 7f084df6d700 1 -- 172.21.15.16:0/1706631423 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0848129d70 con 0x7f08481302f0 2019-12-05T06:31:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.604+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=THROTTLE_DONE pgs=217 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.604+0000 7f0844ff9700 1 -- 172.21.15.16:0/1706631423 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f083802eef0 con 0x7f08481302f0 2019-12-05T06:31:46.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.844+0000 7f084df6d700 1 -- 172.21.15.16:0/1706631423 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0848129d70 con 0x7f08481302f0 2019-12-05T06:31:46.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.846+0000 7f0847fff700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=THROTTLE_DONE pgs=217 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:46.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.846+0000 7f0844ff9700 1 -- 172.21.15.16:0/1706631423 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f083802e600 con 0x7f08481302f0 2019-12-05T06:31:46.843 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 117 2019-12-05T06:31:46.843 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:31:46.843 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6561,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:43.275354+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":21,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:46.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 -- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0828023680 msgr2=0x7f0828025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0828023680 0x7f0828025af0 crc :-1 s=READY pgs=400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 -- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 msgr2=0x7f084812a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 secure :-1 s=READY pgs=217 cs=0 l=1 rx=0x7f0838010690 tx=0x7f0838010a30).stop 2019-12-05T06:31:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 -- 172.21.15.16:0/1706631423 shutdown_connections 2019-12-05T06:31:46.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0828023680 0x7f0828025af0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 --2- 172.21.15.16:0/1706631423 >> v2:172.21.15.76:3300/0 conn(0x7f084812b640 0x7f084812abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 --2- 172.21.15.16:0/1706631423 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08481302f0 0x7f084812a670 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 -- 172.21.15.16:0/1706631423 shutdown_connections 2019-12-05T06:31:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 -- 172.21.15.16:0/1706631423 wait complete. 2019-12-05T06:31:46.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:46.850+0000 7f084df6d700 1 -- 172.21.15.16:0/1706631423 >> 172.21.15.16:0/1706631423 conn(0x7f08480a2850 msgr2=0x7f084806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:46.856 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:31:48.857 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:49.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.112+0000 7f9cb4093700 1 Processor -- start 2019-12-05T06:31:49.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.112+0000 7f9cb4093700 1 -- start start 2019-12-05T06:31:49.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.114+0000 7f9cb4093700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9cac08d2c0 0x7f9cac08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:49.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.114+0000 7f9cb4093700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac0958e0 con 0x7f9cac08c6d0 2019-12-05T06:31:49.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.114+0000 7f9cb4093700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac0629e0 con 0x7f9cac08fef0 2019-12-05T06:31:49.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.114+0000 7f9cb4093700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cac08df90 con 0x7f9cac08d2c0 2019-12-05T06:31:49.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.114+0000 7f9cb1e2f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9cac08d2c0 0x7f9cac08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.114+0000 7f9cb2630700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9cac08c6d0 0x7f9cac08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40524/0 (socket says 172.21.15.16:40524) 2019-12-05T06:31:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.114+0000 7f9cb1e2f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9cac08d2c0 0x7f9cac08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55360/0 (socket says 172.21.15.16:55360) 2019-12-05T06:31:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.114+0000 7f9cb2630700 1 -- 172.21.15.16:0/1885319009 learned_addr learned my addr 172.21.15.16:0/1885319009 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.115+0000 7f9cb1e2f700 1 -- 172.21.15.16:0/1885319009 >> v1:172.21.15.76:6789/0 conn(0x7f9cac08fef0 legacy=0x7f9cac08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.115+0000 7f9cb1e2f700 1 -- 172.21.15.16:0/1885319009 >> v1:172.21.15.16:6789/0 conn(0x7f9cac08c6d0 legacy=0x7f9cac08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.115+0000 7f9cb162e700 1 -- 172.21.15.16:0/1885319009 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4206492544 0 0) 0x7f9ca40014d0 con 0x7f9cac08c6d0 2019-12-05T06:31:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.115+0000 7f9cb1e2f700 1 -- 172.21.15.16:0/1885319009 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac08e620 con 0x7f9cac08d2c0 2019-12-05T06:31:49.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.115+0000 7f9cb1e2f700 1 --2- 172.21.15.16:0/1885319009 >> v2:172.21.15.16:3300/0 conn(0x7f9cac08d2c0 0x7f9cac08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9ca000af50 tx=0x7f9ca000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.115+0000 7f9cb1e2f700 1 --2- 172.21.15.16:0/1885319009 >> v2:172.21.15.16:3300/0 conn(0x7f9cac08d2c0 0x7f9cac08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9ca000af50 tx=0x7f9ca000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.115+0000 7f9cb1e2f700 1 --2- 172.21.15.16:0/1885319009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac08d2c0 0x7f9cac08ef20 secure :-1 s=READY pgs=928 cs=0 l=1 rx=0x7f9ca000af50 tx=0x7f9ca000b760).ready entity=mon.0 client_cookie=f0ed37f288370cdd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.115+0000 7f9cb1e2f700 1 --2- 172.21.15.16:0/1885319009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac08d2c0 0x7f9cac08ef20 secure :-1 s=THROTTLE_DONE pgs=928 cs=0 l=1 rx=0x7f9ca000af50 tx=0x7f9ca000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.116+0000 7f9cb1e2f700 1 --2- 172.21.15.16:0/1885319009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac08d2c0 0x7f9cac08ef20 secure :-1 s=THROTTLE_DONE pgs=928 cs=0 l=1 rx=0x7f9ca000af50 tx=0x7f9ca000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.116+0000 7f9cb162e700 1 -- 172.21.15.16:0/1885319009 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ca0000960 con 0x7f9cac08d2c0 2019-12-05T06:31:49.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.116+0000 7f9cb1e2f700 1 --2- 172.21.15.16:0/1885319009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac08d2c0 0x7f9cac08ef20 secure :-1 s=THROTTLE_DONE pgs=928 cs=0 l=1 rx=0x7f9ca000af50 tx=0x7f9ca000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.116+0000 7f9cb162e700 1 -- 172.21.15.16:0/1885319009 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca00098e0 con 0x7f9cac08d2c0 2019-12-05T06:31:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.116+0000 7f9cb162e700 1 -- 172.21.15.16:0/1885319009 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ca000a050 con 0x7f9cac08d2c0 2019-12-05T06:31:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.116+0000 7f9cb4093700 1 -- 172.21.15.16:0/1885319009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac08d2c0 msgr2=0x7f9cac08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.116+0000 7f9cb4093700 1 --2- 172.21.15.16:0/1885319009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac08d2c0 0x7f9cac08ef20 secure :-1 s=READY pgs=928 cs=0 l=1 rx=0x7f9ca000af50 tx=0x7f9ca000b760).stop 2019-12-05T06:31:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.118+0000 7f9cb4093700 1 -- 172.21.15.16:0/1885319009 shutdown_connections 2019-12-05T06:31:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.118+0000 7f9cb4093700 1 --2- 172.21.15.16:0/1885319009 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9cac08d2c0 0x7f9cac08ef20 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:49.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.118+0000 7f9cb4093700 1 -- 172.21.15.16:0/1885319009 shutdown_connections 2019-12-05T06:31:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.118+0000 7f9cb4093700 1 -- 172.21.15.16:0/1885319009 wait complete. 2019-12-05T06:31:49.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.118+0000 7f9cb4093700 1 -- 172.21.15.16:0/1885319009 >> 172.21.15.16:0/1885319009 conn(0x7f9cac061940 msgr2=0x7f9cac096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:49.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.123+0000 7f9cb4093700 1 Processor -- start 2019-12-05T06:31:49.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.123+0000 7f9cb4093700 1 -- start start 2019-12-05T06:31:49.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.124+0000 7f9cb4093700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9cac096e90 0x7f9cac09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:49.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.124+0000 7f9cb4093700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9cac084f30 0x7f9cac09b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:49.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.124+0000 7f9cb4093700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cac084b90 con 0x7f9cac097280 2019-12-05T06:31:49.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.124+0000 7f9cb4093700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cac09a450 con 0x7f9cac096e90 2019-12-05T06:31:49.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.124+0000 7f9cb4093700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac099a20 con 0x7f9cac084f30 2019-12-05T06:31:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.125+0000 7f9cb1e2f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9cac084f30 0x7f9cac09b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.125+0000 7f9cb1e2f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9cac084f30 0x7f9cac09b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:59354/0 (socket says 172.21.15.16:59354) 2019-12-05T06:31:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.125+0000 7f9cb2630700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9cac096e90 0x7f9cac09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.125+0000 7f9cb1e2f700 1 -- 172.21.15.16:0/1303173129 learned_addr learned my addr 172.21.15.16:0/1303173129 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:31:49.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9cb2630700 1 -- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9cac084f30 msgr2=0x7f9cac09b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9cac084f30 0x7f9cac09b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9cb2630700 1 -- 172.21.15.16:0/1303173129 >> v1:172.21.15.76:6789/0 conn(0x7f9cac097280 legacy=0x7f9cac0976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9caaffd700 1 -- 172.21.15.16:0/1303173129 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3888362060 0 0) 0x7f9c9c001080 con 0x7f9cac097280 2019-12-05T06:31:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9cb2630700 1 -- 172.21.15.16:0/1303173129 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac09aae0 con 0x7f9cac096e90 2019-12-05T06:31:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9cb1e2f700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9cac084f30 0x7f9cac09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:31:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> v2:172.21.15.76:3300/0 conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> v2:172.21.15.76:3300/0 conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=READY pgs=667 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.126+0000 7f9caaffd700 1 -- 172.21.15.16:0/1303173129 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ca4001250 con 0x7f9cac096e90 2019-12-05T06:31:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.127+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.127+0000 7f9caaffd700 1 -- 172.21.15.16:0/1303173129 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca40045c0 con 0x7f9cac096e90 2019-12-05T06:31:49.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.127+0000 7f9cb4093700 1 -- 172.21.15.16:0/1303173129 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cac09aae0 con 0x7f9cac096e90 2019-12-05T06:31:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.127+0000 7f9caaffd700 1 -- 172.21.15.16:0/1303173129 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ca4004c90 con 0x7f9cac096e90 2019-12-05T06:31:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.127+0000 7f9cb4093700 1 -- 172.21.15.16:0/1303173129 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cac09aae0 con 0x7f9cac096e90 2019-12-05T06:31:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.128+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.128+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.128+0000 7f9caaffd700 1 -- 172.21.15.16:0/1303173129 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9ca40047b0 con 0x7f9cac096e90 2019-12-05T06:31:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.129+0000 7f9caaffd700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c8c023630 0x7f9c8c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.129+0000 7f9caaffd700 1 -- 172.21.15.16:0/1303173129 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7f9ca40460f0 con 0x7f9cac096e90 2019-12-05T06:31:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.129+0000 7f9cb1e2f700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c8c023630 0x7f9c8c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:49.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.130+0000 7f9cb1e2f700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c8c023630 0x7f9c8c025ac0 crc :-1 s=READY pgs=401 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:49.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.143+0000 7f9cb4093700 1 -- 172.21.15.16:0/1303173129 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cac09aae0 con 0x7f9cac096e90 2019-12-05T06:31:49.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.153+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.153+0000 7f9caaffd700 1 -- 172.21.15.16:0/1303173129 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9ca4001010 con 0x7f9cac096e90 2019-12-05T06:31:49.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.394+0000 7f9cb4093700 1 -- 172.21.15.16:0/1303173129 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cac09aae0 con 0x7f9cac096e90 2019-12-05T06:31:49.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.395+0000 7f9cb2630700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=THROTTLE_DONE pgs=667 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.395+0000 7f9caaffd700 1 -- 172.21.15.16:0/1303173129 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f9ca40047b0 con 0x7f9cac096e90 2019-12-05T06:31:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 117 2019-12-05T06:31:49.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 -- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c8c023630 msgr2=0x7f9c8c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c8c023630 0x7f9c8c025ac0 crc :-1 s=READY pgs=401 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 -- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 msgr2=0x7f9cac09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 0x7f9cac09b3e0 secure :-1 s=READY pgs=667 cs=0 l=1 rx=0x7f9ca40106f0 tx=0x7f9ca4010a90).stop 2019-12-05T06:31:49.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 -- 172.21.15.16:0/1303173129 shutdown_connections 2019-12-05T06:31:49.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9c8c023630 0x7f9c8c025ac0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:49.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9cac084f30 0x7f9cac09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:49.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 --2- 172.21.15.16:0/1303173129 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9cac096e90 0x7f9cac09b3e0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:49.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 -- 172.21.15.16:0/1303173129 shutdown_connections 2019-12-05T06:31:49.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 -- 172.21.15.16:0/1303173129 wait complete. 2019-12-05T06:31:49.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:49.400+0000 7f9cb4093700 1 -- 172.21.15.16:0/1303173129 >> 172.21.15.16:0/1303173129 conn(0x7f9cac0714f0 msgr2=0x7f9cac09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:49.462 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:31:49.463 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6561,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:43.275354+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":21,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:49.463 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:31:51.464 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.727+0000 7fcd3f87c700 1 Processor -- start 2019-12-05T06:31:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.727+0000 7fcd3f87c700 1 -- start start 2019-12-05T06:31:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.729+0000 7fcd3f87c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 0x7fcd3808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.729+0000 7fcd3f87c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcd3808fef0 0x7fcd3808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.729+0000 7fcd3f87c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd380958e0 con 0x7fcd3808c6d0 2019-12-05T06:31:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.729+0000 7fcd3f87c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3808df90 con 0x7fcd3808fef0 2019-12-05T06:31:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.729+0000 7fcd3f87c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd380a3f70 con 0x7fcd3808d2c0 2019-12-05T06:31:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.729+0000 7fcd3de19700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcd3808c6d0 0x7fcd3808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:33612/0 (socket says 172.21.15.16:33612) 2019-12-05T06:31:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.729+0000 7fcd3de19700 1 -- 172.21.15.16:0/3555426048 learned_addr learned my addr 172.21.15.16:0/3555426048 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.729+0000 7fcd3d618700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 0x7fcd3808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.729+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/3555426048 >> v2:172.21.15.76:3300/0 conn(0x7fcd3808fef0 0x7fcd3808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.730+0000 7fcd3ce17700 1 -- 172.21.15.16:0/3555426048 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062534116 0 0) 0x7fcd28001080 con 0x7fcd3808c6d0 2019-12-05T06:31:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.730+0000 7fcd3ce17700 1 -- 172.21.15.16:0/3555426048 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd20002e80 con 0x7fcd3808c6d0 2019-12-05T06:31:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.730+0000 7fcd3d618700 1 -- 172.21.15.16:0/3555426048 >> v2:172.21.15.76:3300/0 conn(0x7fcd3808fef0 msgr2=0x7fcd3808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.730+0000 7fcd3d618700 1 --2- 172.21.15.16:0/3555426048 >> v2:172.21.15.76:3300/0 conn(0x7fcd3808fef0 0x7fcd3808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.730+0000 7fcd3d618700 1 -- 172.21.15.16:0/3555426048 >> v1:172.21.15.76:6789/0 conn(0x7fcd3808c6d0 legacy=0x7fcd3808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.730+0000 7fcd3d618700 1 -- 172.21.15.16:0/3555426048 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3808e620 con 0x7fcd3808d2c0 2019-12-05T06:31:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.730+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/3555426048 >> v2:172.21.15.76:3300/0 conn(0x7fcd3808fef0 0x7fcd3808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:31:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.730+0000 7fcd3d618700 1 --2- 172.21.15.16:0/3555426048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 0x7fcd3808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcd2c00af50 tx=0x7fcd2c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.730+0000 7fcd3d618700 1 --2- 172.21.15.16:0/3555426048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 0x7fcd3808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcd2c00af50 tx=0x7fcd2c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.730+0000 7fcd3d618700 1 --2- 172.21.15.16:0/3555426048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 0x7fcd3808ef20 secure :-1 s=READY pgs=218 cs=0 l=1 rx=0x7fcd2c00af50 tx=0x7fcd2c00b760).ready entity=mon.2 client_cookie=44736f395b499e49 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.731+0000 7fcd3d618700 1 --2- 172.21.15.16:0/3555426048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 0x7fcd3808ef20 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7fcd2c00af50 tx=0x7fcd2c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.731+0000 7fcd3ce17700 1 -- 172.21.15.16:0/3555426048 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd2c007210 con 0x7fcd3808d2c0 2019-12-05T06:31:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.731+0000 7fcd3d618700 1 --2- 172.21.15.16:0/3555426048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 0x7fcd3808ef20 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7fcd2c00af50 tx=0x7fcd2c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.731+0000 7fcd3d618700 1 --2- 172.21.15.16:0/3555426048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 0x7fcd3808ef20 secure :-1 s=THROTTLE_DONE pgs=218 cs=0 l=1 rx=0x7fcd2c00af50 tx=0x7fcd2c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.731+0000 7fcd3ce17700 1 -- 172.21.15.16:0/3555426048 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2c0099b0 con 0x7fcd3808d2c0 2019-12-05T06:31:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.731+0000 7fcd3ce17700 1 -- 172.21.15.16:0/3555426048 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd2c00a080 con 0x7fcd3808d2c0 2019-12-05T06:31:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.731+0000 7fcd3f87c700 1 -- 172.21.15.16:0/3555426048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 msgr2=0x7fcd3808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.731+0000 7fcd3f87c700 1 --2- 172.21.15.16:0/3555426048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 0x7fcd3808ef20 secure :-1 s=READY pgs=218 cs=0 l=1 rx=0x7fcd2c00af50 tx=0x7fcd2c00b760).stop 2019-12-05T06:31:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.732+0000 7fcd3f87c700 1 -- 172.21.15.16:0/3555426048 shutdown_connections 2019-12-05T06:31:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.732+0000 7fcd3f87c700 1 --2- 172.21.15.16:0/3555426048 >> v2:172.21.15.76:3300/0 conn(0x7fcd3808fef0 0x7fcd3808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.732+0000 7fcd3f87c700 1 --2- 172.21.15.16:0/3555426048 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3808d2c0 0x7fcd3808ef20 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:51.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.733+0000 7fcd3f87c700 1 -- 172.21.15.16:0/3555426048 shutdown_connections 2019-12-05T06:31:51.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.733+0000 7fcd3f87c700 1 -- 172.21.15.16:0/3555426048 wait complete. 2019-12-05T06:31:51.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.733+0000 7fcd3f87c700 1 -- 172.21.15.16:0/3555426048 >> 172.21.15.16:0/3555426048 conn(0x7fcd38061940 msgr2=0x7fcd38096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:51.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.738+0000 7fcd3f87c700 1 Processor -- start 2019-12-05T06:31:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.738+0000 7fcd3f87c700 1 -- start start 2019-12-05T06:31:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.740+0000 7fcd3f87c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3806e840 0x7fcd38134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.740+0000 7fcd3f87c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcd38135450 0x7fcd381349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.740+0000 7fcd3f87c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd38128af0 con 0x7fcd3806dc50 2019-12-05T06:31:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.740+0000 7fcd3f87c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd38132830 con 0x7fcd38135450 2019-12-05T06:31:51.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.740+0000 7fcd3f87c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd381334f0 con 0x7fcd3806e840 2019-12-05T06:31:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.740+0000 7fcd3de19700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcd3806dc50 0x7fcd3806e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40548/0 (socket says 172.21.15.16:40548) 2019-12-05T06:31:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.740+0000 7fcd3de19700 1 -- 172.21.15.16:0/2637099458 learned_addr learned my addr 172.21.15.16:0/2637099458 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:51.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.740+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> v2:172.21.15.76:3300/0 conn(0x7fcd38135450 0x7fcd381349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.740+0000 7fcd3d618700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3806e840 0x7fcd38134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd367fc700 1 -- 172.21.15.16:0/2637099458 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1603987443 0 0) 0x7fcd28001290 con 0x7fcd3806dc50 2019-12-05T06:31:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3e61a700 1 -- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3806e840 msgr2=0x7fcd38134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:51.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3806e840 0x7fcd38134480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3e61a700 1 -- 172.21.15.16:0/2637099458 >> v1:172.21.15.16:6789/0 conn(0x7fcd3806dc50 legacy=0x7fcd3806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3e61a700 1 -- 172.21.15.16:0/2637099458 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd38133b80 con 0x7fcd38135450 2019-12-05T06:31:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> v2:172.21.15.76:3300/0 conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> v2:172.21.15.76:3300/0 conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=READY pgs=669 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:51.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=THROTTLE_DONE pgs=669 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd367fc700 1 -- 172.21.15.16:0/2637099458 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd3000a9f0 con 0x7fcd38135450 2019-12-05T06:31:51.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=THROTTLE_DONE pgs=669 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=THROTTLE_DONE pgs=669 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd367fc700 1 -- 172.21.15.16:0/2637099458 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd30009930 con 0x7fcd38135450 2019-12-05T06:31:51.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd3f87c700 1 -- 172.21.15.16:0/2637099458 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd38133b80 con 0x7fcd38135450 2019-12-05T06:31:51.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.741+0000 7fcd367fc700 1 -- 172.21.15.16:0/2637099458 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd3000a000 con 0x7fcd38135450 2019-12-05T06:31:51.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.742+0000 7fcd3f87c700 1 -- 172.21.15.16:0/2637099458 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd38133b80 con 0x7fcd38135450 2019-12-05T06:31:51.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.742+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=THROTTLE_DONE pgs=669 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.743+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=THROTTLE_DONE pgs=669 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.743+0000 7fcd367fc700 1 -- 172.21.15.16:0/2637099458 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcd30009b20 con 0x7fcd38135450 2019-12-05T06:31:51.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.743+0000 7fcd367fc700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd18023630 0x7fcd18025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:51.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.743+0000 7fcd367fc700 1 -- 172.21.15.16:0/2637099458 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fcd30046300 con 0x7fcd38135450 2019-12-05T06:31:51.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.743+0000 7fcd3de19700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd18023630 0x7fcd18025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:51.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.744+0000 7fcd3de19700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd18023630 0x7fcd18025ae0 crc :-1 s=READY pgs=402 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:51.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.757+0000 7fcd3f87c700 1 -- 172.21.15.16:0/2637099458 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd38133b80 con 0x7fcd38135450 2019-12-05T06:31:51.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.767+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=THROTTLE_DONE pgs=669 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.767+0000 7fcd367fc700 1 -- 172.21.15.16:0/2637099458 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcd30000df0 con 0x7fcd38135450 2019-12-05T06:31:51.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.972+0000 7fcd3f87c700 1 -- 172.21.15.16:0/2637099458 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd38133b80 con 0x7fcd38135450 2019-12-05T06:31:51.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.973+0000 7fcd3e61a700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=THROTTLE_DONE pgs=669 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:51.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.973+0000 7fcd367fc700 1 -- 172.21.15.16:0/2637099458 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fcd30009930 con 0x7fcd38135450 2019-12-05T06:31:51.971 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 117 2019-12-05T06:31:51.972 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:31:51.972 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6561,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:43.275354+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":21,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:51.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.977+0000 7fcd3f87c700 1 -- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd18023630 msgr2=0x7fcd18025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:51.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.977+0000 7fcd3f87c700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd18023630 0x7fcd18025ae0 crc :-1 s=READY pgs=402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:51.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.977+0000 7fcd3f87c700 1 -- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 msgr2=0x7fcd381349c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:51.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.977+0000 7fcd3f87c700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 0x7fcd381349c0 secure :-1 s=READY pgs=669 cs=0 l=1 rx=0x7fcd30010720 tx=0x7fcd30010550).stop 2019-12-05T06:31:51.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.978+0000 7fcd3f87c700 1 -- 172.21.15.16:0/2637099458 shutdown_connections 2019-12-05T06:31:51.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.978+0000 7fcd3f87c700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd18023630 0x7fcd18025ae0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:51.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.978+0000 7fcd3f87c700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd38135450 0x7fcd381349c0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.978+0000 7fcd3f87c700 1 --2- 172.21.15.16:0/2637099458 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd3806e840 0x7fcd38134480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.978+0000 7fcd3f87c700 1 -- 172.21.15.16:0/2637099458 shutdown_connections 2019-12-05T06:31:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.978+0000 7fcd3f87c700 1 -- 172.21.15.16:0/2637099458 wait complete. 2019-12-05T06:31:51.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:51.978+0000 7fcd3f87c700 1 -- 172.21.15.16:0/2637099458 >> 172.21.15.16:0/2637099458 conn(0x7fcd380a2850 msgr2=0x7fcd3806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:51.987 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:31:53.988 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:54.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.280+0000 7faaa4a66700 1 Processor -- start 2019-12-05T06:31:54.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.280+0000 7faaa4a66700 1 -- start start 2019-12-05T06:31:54.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.282+0000 7faaa4a66700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faaa008d2c0 0x7faaa008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:54.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.282+0000 7faaa4a66700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faaa008fef0 0x7faaa008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:54.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.282+0000 7faaa4a66700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa00958e0 con 0x7faaa008c6d0 2019-12-05T06:31:54.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.282+0000 7faaa4a66700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa008df90 con 0x7faaa008fef0 2019-12-05T06:31:54.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.282+0000 7faaa4a66700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa00a3f70 con 0x7faaa008d2c0 2019-12-05T06:31:54.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.283+0000 7faa9f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faaa008fef0 0x7faaa008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:54.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.283+0000 7faa9f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faaa008fef0 0x7faaa008f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55390/0 (socket says 172.21.15.16:55390) 2019-12-05T06:31:54.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.283+0000 7faa9f59e700 1 -- 172.21.15.16:0/3793851544 learned_addr learned my addr 172.21.15.16:0/3793851544 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:31:54.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.283+0000 7faa9e59c700 1 --2- 172.21.15.16:0/3793851544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faaa008d2c0 0x7faaa008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:54.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.283+0000 7faa9f59e700 1 -- 172.21.15.16:0/3793851544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faaa008d2c0 msgr2=0x7faaa008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:54.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.283+0000 7faa9f59e700 1 --2- 172.21.15.16:0/3793851544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faaa008d2c0 0x7faaa008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:54.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.283+0000 7faa9f59e700 1 -- 172.21.15.16:0/3793851544 >> v1:172.21.15.76:6789/0 conn(0x7faaa008c6d0 legacy=0x7faaa008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:54.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.283+0000 7faa9f59e700 1 -- 172.21.15.16:0/3793851544 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa008e620 con 0x7faaa008fef0 2019-12-05T06:31:54.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.283+0000 7faa9dd9b700 1 -- 172.21.15.16:0/3793851544 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3586201921 0 0) 0x7faa98001400 con 0x7faaa008c6d0 2019-12-05T06:31:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faa9f59e700 1 --2- 172.21.15.16:0/3793851544 >> v2:172.21.15.16:3300/0 conn(0x7faaa008fef0 0x7faaa008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faa9000f730 tx=0x7faa90007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faa9f59e700 1 --2- 172.21.15.16:0/3793851544 >> v2:172.21.15.16:3300/0 conn(0x7faaa008fef0 0x7faaa008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faa9000f730 tx=0x7faa90007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faa9f59e700 1 --2- 172.21.15.16:0/3793851544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa008fef0 0x7faaa008f460 secure :-1 s=READY pgs=930 cs=0 l=1 rx=0x7faa9000f730 tx=0x7faa90007d50).ready entity=mon.0 client_cookie=b6f9bd032fec3664 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:54.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faa9f59e700 1 --2- 172.21.15.16:0/3793851544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa008fef0 0x7faaa008f460 secure :-1 s=THROTTLE_DONE pgs=930 cs=0 l=1 rx=0x7faa9000f730 tx=0x7faa90007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faa9dd9b700 1 -- 172.21.15.16:0/3793851544 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa90000a70 con 0x7faaa008fef0 2019-12-05T06:31:54.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faa9f59e700 1 --2- 172.21.15.16:0/3793851544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa008fef0 0x7faaa008f460 secure :-1 s=THROTTLE_DONE pgs=930 cs=0 l=1 rx=0x7faa9000f730 tx=0x7faa90007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faa9f59e700 1 --2- 172.21.15.16:0/3793851544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa008fef0 0x7faaa008f460 secure :-1 s=THROTTLE_DONE pgs=930 cs=0 l=1 rx=0x7faa9000f730 tx=0x7faa90007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faa9dd9b700 1 -- 172.21.15.16:0/3793851544 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa900010e0 con 0x7faaa008fef0 2019-12-05T06:31:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faa9dd9b700 1 -- 172.21.15.16:0/3793851544 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa900012d0 con 0x7faaa008fef0 2019-12-05T06:31:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faaa4a66700 1 -- 172.21.15.16:0/3793851544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa008fef0 msgr2=0x7faaa008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.284+0000 7faaa4a66700 1 --2- 172.21.15.16:0/3793851544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa008fef0 0x7faaa008f460 secure :-1 s=READY pgs=930 cs=0 l=1 rx=0x7faa9000f730 tx=0x7faa90007d50).stop 2019-12-05T06:31:54.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.286+0000 7faaa4a66700 1 -- 172.21.15.16:0/3793851544 shutdown_connections 2019-12-05T06:31:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.286+0000 7faaa4a66700 1 --2- 172.21.15.16:0/3793851544 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa008fef0 0x7faaa008f460 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.286+0000 7faaa4a66700 1 --2- 172.21.15.16:0/3793851544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faaa008d2c0 0x7faaa008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.286+0000 7faaa4a66700 1 -- 172.21.15.16:0/3793851544 shutdown_connections 2019-12-05T06:31:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.286+0000 7faaa4a66700 1 -- 172.21.15.16:0/3793851544 wait complete. 2019-12-05T06:31:54.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.286+0000 7faaa4a66700 1 -- 172.21.15.16:0/3793851544 >> 172.21.15.16:0/3793851544 conn(0x7faaa0061940 msgr2=0x7faaa0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.292+0000 7faaa4a66700 1 Processor -- start 2019-12-05T06:31:54.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.292+0000 7faaa4a66700 1 -- start start 2019-12-05T06:31:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.294+0000 7faaa4a66700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faaa014d400 0x7faaa014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.294+0000 7faaa4a66700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faaa0132d00 0x7faaa0131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.294+0000 7faaa4a66700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faaa0098da0 con 0x7faaa0132270 2019-12-05T06:31:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.294+0000 7faaa4a66700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faaa009bb50 con 0x7faaa0132d00 2019-12-05T06:31:54.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.294+0000 7faaa4a66700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faaa009a030 con 0x7faaa014d400 2019-12-05T06:31:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.294+0000 7faa9e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faaa0132d00 0x7faaa0131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.294+0000 7faa9e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faaa0132d00 0x7faaa0131d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55398/0 (socket says 172.21.15.16:55398) 2019-12-05T06:31:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.294+0000 7faa9f59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faaa0132270 0x7faaa006dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40564/0 (socket says 172.21.15.16:40564) 2019-12-05T06:31:54.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.294+0000 7faa9e59c700 1 -- 172.21.15.16:0/2666794577 learned_addr learned my addr 172.21.15.16:0/2666794577 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:31:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.294+0000 7faa9ed9d700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faaa014d400 0x7faaa014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa9e59c700 1 -- 172.21.15.16:0/2666794577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faaa014d400 msgr2=0x7faaa014d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:54.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faaa014d400 0x7faaa014d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa9e59c700 1 -- 172.21.15.16:0/2666794577 >> v1:172.21.15.16:6789/0 conn(0x7faaa0132270 legacy=0x7faaa006dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa9e59c700 1 -- 172.21.15.16:0/2666794577 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaa009b0f0 con 0x7faaa0132d00 2019-12-05T06:31:54.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> v2:172.21.15.16:3300/0 conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> v2:172.21.15.16:3300/0 conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=READY pgs=932 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa977fe700 1 -- 172.21.15.16:0/2666794577 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa88000940 con 0x7faaa0132d00 2019-12-05T06:31:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faa977fe700 1 -- 172.21.15.16:0/2666794577 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa88000e80 con 0x7faaa0132d00 2019-12-05T06:31:54.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.295+0000 7faaa4a66700 1 -- 172.21.15.16:0/2666794577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaa009b0f0 con 0x7faaa0132d00 2019-12-05T06:31:54.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.296+0000 7faa977fe700 1 -- 172.21.15.16:0/2666794577 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa880014d0 con 0x7faaa0132d00 2019-12-05T06:31:54.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.296+0000 7faaa4a66700 1 -- 172.21.15.16:0/2666794577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaa009b0f0 con 0x7faaa0132d00 2019-12-05T06:31:54.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.297+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.298+0000 7faa977fe700 1 -- 172.21.15.16:0/2666794577 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faa88001070 con 0x7faaa0132d00 2019-12-05T06:31:54.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.298+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.298+0000 7faa977fe700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa84023630 0x7faa84025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:54.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.298+0000 7faa977fe700 1 -- 172.21.15.16:0/2666794577 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7faa88045de0 con 0x7faaa0132d00 2019-12-05T06:31:54.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.298+0000 7faa9ed9d700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa84023630 0x7faa84025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.299+0000 7faa9ed9d700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa84023630 0x7faa84025ac0 crc :-1 s=READY pgs=403 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:54.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.312+0000 7faaa4a66700 1 -- 172.21.15.16:0/2666794577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaa009b0f0 con 0x7faaa0132d00 2019-12-05T06:31:54.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.321+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.321+0000 7faa977fe700 1 -- 172.21.15.16:0/2666794577 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faa88009250 con 0x7faaa0132d00 2019-12-05T06:31:54.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.582+0000 7faaa4a66700 1 -- 172.21.15.16:0/2666794577 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faaa009b0f0 con 0x7faaa0132d00 2019-12-05T06:31:54.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.583+0000 7faa9e59c700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=THROTTLE_DONE pgs=932 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:54.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.583+0000 7faa977fe700 1 -- 172.21.15.16:0/2666794577 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2109 (secure 0 0 0) 0x7faa88001050 con 0x7faaa0132d00 2019-12-05T06:31:54.579 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 117 2019-12-05T06:31:54.581 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:31:54.581 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6561,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:43.275354+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":21,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.585+0000 7faaa4a66700 1 -- 172.21.15.16:0/2666794577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa84023630 msgr2=0x7faa84025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.585+0000 7faaa4a66700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa84023630 0x7faa84025ac0 crc :-1 s=READY pgs=403 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:54.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.585+0000 7faaa4a66700 1 -- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 msgr2=0x7faaa0131d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.585+0000 7faaa4a66700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 0x7faaa0131d30 secure :-1 s=READY pgs=932 cs=0 l=1 rx=0x7faa88010450 tx=0x7faa88010280).stop 2019-12-05T06:31:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.585+0000 7faaa4a66700 1 -- 172.21.15.16:0/2666794577 shutdown_connections 2019-12-05T06:31:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.585+0000 7faaa4a66700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faa84023630 0x7faa84025ac0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.585+0000 7faaa4a66700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faaa0132d00 0x7faaa0131d30 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.586+0000 7faaa4a66700 1 --2- 172.21.15.16:0/2666794577 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faaa014d400 0x7faaa014d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:54.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.586+0000 7faaa4a66700 1 -- 172.21.15.16:0/2666794577 shutdown_connections 2019-12-05T06:31:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.586+0000 7faaa4a66700 1 -- 172.21.15.16:0/2666794577 wait complete. 2019-12-05T06:31:54.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:54.586+0000 7faaa4a66700 1 -- 172.21.15.16:0/2666794577 >> 172.21.15.16:0/2666794577 conn(0x7faaa00714f0 msgr2=0x7faaa009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:54.593 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:31:56.607 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:56.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.873+0000 7fe613bae700 1 Processor -- start 2019-12-05T06:31:56.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.873+0000 7fe613bae700 1 -- start start 2019-12-05T06:31:56.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.875+0000 7fe613bae700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c08c6d0 0x7fe60c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:56.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.875+0000 7fe613bae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe60c08f720 0x7fe60c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:56.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.875+0000 7fe613bae700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c0958e0 con 0x7fe60c08fef0 2019-12-05T06:31:56.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.875+0000 7fe613bae700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe60c0a3f70 con 0x7fe60c08f720 2019-12-05T06:31:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.875+0000 7fe613bae700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c08d830 con 0x7fe60c08c6d0 2019-12-05T06:31:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.876+0000 7fe61294c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe60c08f720 0x7fe60c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.876+0000 7fe61194a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe60c08fef0 0x7fe60c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40576/0 (socket says 172.21.15.16:40576) 2019-12-05T06:31:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.876+0000 7fe61194a700 1 -- 172.21.15.16:0/1503182825 learned_addr learned my addr 172.21.15.16:0/1503182825 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.876+0000 7fe61214b700 1 --2- 172.21.15.16:0/1503182825 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c08c6d0 0x7fe60c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:56.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.876+0000 7fe61294c700 1 -- 172.21.15.16:0/1503182825 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c08c6d0 msgr2=0x7fe60c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:56.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.876+0000 7fe61294c700 1 --2- 172.21.15.16:0/1503182825 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c08c6d0 0x7fe60c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:56.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.876+0000 7fe61294c700 1 -- 172.21.15.16:0/1503182825 >> v1:172.21.15.16:6789/0 conn(0x7fe60c08fef0 legacy=0x7fe60c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:56.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.876+0000 7fe61294c700 1 -- 172.21.15.16:0/1503182825 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c08fbc0 con 0x7fe60c08f720 2019-12-05T06:31:56.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.876+0000 7fe611149700 1 -- 172.21.15.16:0/1503182825 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1189324976 0 0) 0x7fe600001150 con 0x7fe60c08fef0 2019-12-05T06:31:56.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe61294c700 1 --2- 172.21.15.16:0/1503182825 >> v2:172.21.15.16:3300/0 conn(0x7fe60c08f720 0x7fe60c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe5fc00af50 tx=0x7fe5fc00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe61294c700 1 --2- 172.21.15.16:0/1503182825 >> v2:172.21.15.16:3300/0 conn(0x7fe60c08f720 0x7fe60c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe5fc00af50 tx=0x7fe5fc00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe61294c700 1 --2- 172.21.15.16:0/1503182825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c08f720 0x7fe60c08d000 secure :-1 s=READY pgs=934 cs=0 l=1 rx=0x7fe5fc00af50 tx=0x7fe5fc00b6e0).ready entity=mon.0 client_cookie=8ef0e6e217cac17e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:56.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe61294c700 1 --2- 172.21.15.16:0/1503182825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c08f720 0x7fe60c08d000 secure :-1 s=THROTTLE_DONE pgs=934 cs=0 l=1 rx=0x7fe5fc00af50 tx=0x7fe5fc00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe611149700 1 -- 172.21.15.16:0/1503182825 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5fc009ba0 con 0x7fe60c08f720 2019-12-05T06:31:56.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe61294c700 1 --2- 172.21.15.16:0/1503182825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c08f720 0x7fe60c08d000 secure :-1 s=THROTTLE_DONE pgs=934 cs=0 l=1 rx=0x7fe5fc00af50 tx=0x7fe5fc00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe61294c700 1 --2- 172.21.15.16:0/1503182825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c08f720 0x7fe60c08d000 secure :-1 s=THROTTLE_DONE pgs=934 cs=0 l=1 rx=0x7fe5fc00af50 tx=0x7fe5fc00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe611149700 1 -- 172.21.15.16:0/1503182825 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5fc0097c0 con 0x7fe60c08f720 2019-12-05T06:31:56.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe611149700 1 -- 172.21.15.16:0/1503182825 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5fc00a180 con 0x7fe60c08f720 2019-12-05T06:31:56.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe613bae700 1 -- 172.21.15.16:0/1503182825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c08f720 msgr2=0x7fe60c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:56.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.877+0000 7fe613bae700 1 --2- 172.21.15.16:0/1503182825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c08f720 0x7fe60c08d000 secure :-1 s=READY pgs=934 cs=0 l=1 rx=0x7fe5fc00af50 tx=0x7fe5fc00b6e0).stop 2019-12-05T06:31:56.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.878+0000 7fe613bae700 1 -- 172.21.15.16:0/1503182825 shutdown_connections 2019-12-05T06:31:56.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.878+0000 7fe613bae700 1 --2- 172.21.15.16:0/1503182825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe60c08f720 0x7fe60c08d000 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:56.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.879+0000 7fe613bae700 1 --2- 172.21.15.16:0/1503182825 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c08c6d0 0x7fe60c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:56.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.879+0000 7fe613bae700 1 -- 172.21.15.16:0/1503182825 shutdown_connections 2019-12-05T06:31:56.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.879+0000 7fe613bae700 1 -- 172.21.15.16:0/1503182825 wait complete. 2019-12-05T06:31:56.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.879+0000 7fe613bae700 1 -- 172.21.15.16:0/1503182825 >> 172.21.15.16:0/1503182825 conn(0x7fe60c061940 msgr2=0x7fe60c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.884+0000 7fe613bae700 1 Processor -- start 2019-12-05T06:31:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.884+0000 7fe613bae700 1 -- start start 2019-12-05T06:31:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.885+0000 7fe613bae700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:56.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.885+0000 7fe613bae700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c098da0 con 0x7fe60c132d00 2019-12-05T06:31:56.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.885+0000 7fe613bae700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe60c09f0a0 con 0x7fe60c132530 2019-12-05T06:31:56.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.885+0000 7fe613bae700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe60c09bb50 con 0x7fe60c14d400 2019-12-05T06:31:56.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.886+0000 7fe61194a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe60c132d00 0x7fe60c131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40580/0 (socket says 172.21.15.16:40580) 2019-12-05T06:31:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.886+0000 7fe61194a700 1 -- 172.21.15.16:0/2627855853 learned_addr learned my addr 172.21.15.16:0/2627855853 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.886+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.886+0000 7fe60affd700 1 -- 172.21.15.16:0/2627855853 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821922448 0 0) 0x7fe600001150 con 0x7fe60c132d00 2019-12-05T06:31:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.886+0000 7fe60affd700 1 -- 172.21.15.16:0/2627855853 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5ec002e80 con 0x7fe60c132d00 2019-12-05T06:31:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.886+0000 7fe61214b700 1 -- 172.21.15.16:0/2627855853 >> v1:172.21.15.76:6789/0 conn(0x7fe60c132530 legacy=0x7fe60c157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.886+0000 7fe61214b700 1 -- 172.21.15.16:0/2627855853 >> v1:172.21.15.16:6789/0 conn(0x7fe60c132d00 legacy=0x7fe60c131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:56.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.886+0000 7fe60affd700 1 -- 172.21.15.16:0/2627855853 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1315686753 0 0) 0x7fe5fc0009d0 con 0x7fe60c132530 2019-12-05T06:31:56.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.886+0000 7fe61214b700 1 -- 172.21.15.16:0/2627855853 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe60c09b0f0 con 0x7fe60c14d400 2019-12-05T06:31:56.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=READY pgs=219 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:56.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe60affd700 1 -- 172.21.15.16:0/2627855853 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6040099c0 con 0x7fe60c14d400 2019-12-05T06:31:56.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe60affd700 1 -- 172.21.15.16:0/2627855853 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe604000cc0 con 0x7fe60c14d400 2019-12-05T06:31:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe60affd700 1 -- 172.21.15.16:0/2627855853 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe60400a1d0 con 0x7fe60c14d400 2019-12-05T06:31:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe613bae700 1 -- 172.21.15.16:0/2627855853 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe60c09b0f0 con 0x7fe60c14d400 2019-12-05T06:31:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.887+0000 7fe613bae700 1 -- 172.21.15.16:0/2627855853 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60c09b0f0 con 0x7fe60c14d400 2019-12-05T06:31:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.888+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.889+0000 7fe60affd700 1 -- 172.21.15.16:0/2627855853 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe604009df0 con 0x7fe60c14d400 2019-12-05T06:31:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.889+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.889+0000 7fe60affd700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe5ec023680 0x7fe5ec025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.889+0000 7fe60affd700 1 -- 172.21.15.16:0/2627855853 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 7493+0+0 (secure 0 0 0) 0x7fe604046180 con 0x7fe60c14d400 2019-12-05T06:31:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.889+0000 7fe61194a700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe5ec023680 0x7fe5ec025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:56.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.890+0000 7fe61194a700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe5ec023680 0x7fe5ec025af0 crc :-1 s=READY pgs=404 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:56.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.905+0000 7fe613bae700 1 -- 172.21.15.16:0/2627855853 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe60c09b0f0 con 0x7fe60c14d400 2019-12-05T06:31:56.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.914+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:56.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:56.914+0000 7fe60affd700 1 -- 172.21.15.16:0/2627855853 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe604009bc0 con 0x7fe60c14d400 2019-12-05T06:31:57.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.161+0000 7fe613bae700 1 -- 172.21.15.16:0/2627855853 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe60c09b0f0 con 0x7fe60c14d400 2019-12-05T06:31:57.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.163+0000 7fe61214b700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=THROTTLE_DONE pgs=219 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:57.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.163+0000 7fe60affd700 1 -- 172.21.15.16:0/2627855853 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 117 v117) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fe604009dd0 con 0x7fe60c14d400 2019-12-05T06:31:57.160 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 117 2019-12-05T06:31:57.161 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:31:57.161 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":117,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6561,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":117}],"filesystems":[{"mdsmap":{"epoch":116,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:30:43.275354+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":66,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6544},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6544":{"gid":6544,"name":"b","rank":0,"incarnation":113,"state":"up:active","state_seq":21,"addr":"172.21.15.76:6835/945658794","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":945658794},{"type":"v1","addr":"172.21.15.76:6835","nonce":945658794}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.167+0000 7fe613bae700 1 -- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe5ec023680 msgr2=0x7fe5ec025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.167+0000 7fe613bae700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe5ec023680 0x7fe5ec025af0 crc :-1 s=READY pgs=404 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.167+0000 7fe613bae700 1 -- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 msgr2=0x7fe60c14d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.167+0000 7fe613bae700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 secure :-1 s=READY pgs=219 cs=0 l=1 rx=0x7fe604010710 tx=0x7fe604010ab0).stop 2019-12-05T06:31:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.167+0000 7fe613bae700 1 -- 172.21.15.16:0/2627855853 shutdown_connections 2019-12-05T06:31:57.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.167+0000 7fe613bae700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe5ec023680 0x7fe5ec025af0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.167+0000 7fe613bae700 1 --2- 172.21.15.16:0/2627855853 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe60c14d400 0x7fe60c14d7f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.167+0000 7fe613bae700 1 -- 172.21.15.16:0/2627855853 shutdown_connections 2019-12-05T06:31:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.168+0000 7fe613bae700 1 -- 172.21.15.16:0/2627855853 wait complete. 2019-12-05T06:31:57.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:57.168+0000 7fe613bae700 1 -- 172.21.15.16:0/2627855853 >> 172.21.15.16:0/2627855853 conn(0x7fe60c0714f0 msgr2=0x7fe60c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:57.191 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:31:59.193 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.393+0000 7fb56c8f4700 1 Processor -- start 2019-12-05T06:31:59.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.393+0000 7fb56c8f4700 1 -- start start 2019-12-05T06:31:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.394+0000 7fb56c8f4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb56808c6d0 0x7fb56808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.394+0000 7fb56c8f4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb56808f720 0x7fb56808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:59.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.394+0000 7fb56c8f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56808fb70 con 0x7fb56808fef0 2019-12-05T06:31:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.394+0000 7fb56c8f4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb56808df90 con 0x7fb56808c6d0 2019-12-05T06:31:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.394+0000 7fb56c8f4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5680a3f70 con 0x7fb56808f720 2019-12-05T06:31:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb56659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb56808fef0 0x7fb56808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40612/0 (socket says 172.21.15.16:40612) 2019-12-05T06:31:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb56659c700 1 -- 172.21.15.16:0/4282488074 learned_addr learned my addr 172.21.15.16:0/4282488074 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:59.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb56759e700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb56808f720 0x7fb56808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb566d9d700 1 --2- 172.21.15.16:0/4282488074 >> v2:172.21.15.76:3300/0 conn(0x7fb56808c6d0 0x7fb56808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1530118994 0 0) 0x7fb55c001150 con 0x7fb56808fef0 2019-12-05T06:31:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb550002e80 con 0x7fb56808fef0 2019-12-05T06:31:59.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4068768003 0 0) 0x7fb55c001130 con 0x7fb56808fef0 2019-12-05T06:31:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb56808f720 msgr2=0x7fb56808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb56808f720 0x7fb56808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:59.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 >> v2:172.21.15.76:3300/0 conn(0x7fb56808c6d0 msgr2=0x7fb56808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 --2- 172.21.15.16:0/4282488074 >> v2:172.21.15.76:3300/0 conn(0x7fb56808c6d0 0x7fb56808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56808d540 con 0x7fb56808fef0 2019-12-05T06:31:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fb55c0016c0 con 0x7fb56808fef0 2019-12-05T06:31:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 >> v1:172.21.15.16:6789/0 conn(0x7fb56808fef0 legacy=0x7fb56808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5500479e0 0x7fb550049e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb55004a5f0 0x7fb55004caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb55004d300 0x7fb55004f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55004ff50 con 0x7fb5500479e0 2019-12-05T06:31:59.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb550050140 con 0x7fb55004a5f0 2019-12-05T06:31:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.395+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb550050370 con 0x7fb55004d300 2019-12-05T06:31:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5500479e0 0x7fb550049e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb56659c700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb55004a5f0 0x7fb55004caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb56759e700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb55004d300 0x7fb55004f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 -- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb55004d300 msgr2=0x7fb55004f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb55004d300 0x7fb55004f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 -- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb55004a5f0 msgr2=0x7fb55004caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:31:59.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb55004a5f0 0x7fb55004caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:31:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 -- 172.21.15.16:0/4282488074 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb550050620 con 0x7fb5500479e0 2019-12-05T06:31:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5500479e0 0x7fb550049e40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb55800b660 tx=0x7fb55800b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5500479e0 0x7fb550049e40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb55800b660 tx=0x7fb55800b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5500479e0 0x7fb550049e40 secure :-1 s=READY pgs=937 cs=0 l=1 rx=0x7fb55800b660 tx=0x7fb55800b530).ready entity=mon.0 client_cookie=743b8eb2133e14ec server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5500479e0 0x7fb550049e40 secure :-1 s=THROTTLE_DONE pgs=937 cs=0 l=1 rx=0x7fb55800b660 tx=0x7fb55800b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb558000960 con 0x7fb5500479e0 2019-12-05T06:31:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5500479e0 0x7fb550049e40 secure :-1 s=THROTTLE_DONE pgs=937 cs=0 l=1 rx=0x7fb55800b660 tx=0x7fb55800b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb558009980 con 0x7fb5500479e0 2019-12-05T06:31:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb566d9d700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5500479e0 0x7fb550049e40 secure :-1 s=THROTTLE_DONE pgs=937 cs=0 l=1 rx=0x7fb55800b660 tx=0x7fb55800b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb565d9b700 1 -- 172.21.15.16:0/4282488074 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb558000960 con 0x7fb5500479e0 2019-12-05T06:31:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb56c8f4700 1 -- 172.21.15.16:0/4282488074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5500479e0 msgr2=0x7fb550049e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb56c8f4700 1 --2- 172.21.15.16:0/4282488074 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb5500479e0 0x7fb550049e40 secure :-1 s=READY pgs=937 cs=0 l=1 rx=0x7fb55800b660 tx=0x7fb55800b530).stop 2019-12-05T06:31:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.396+0000 7fb56759e700 1 -- 172.21.15.16:0/4282488074 reap_dead start 2019-12-05T06:31:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.397+0000 7fb56c8f4700 1 -- 172.21.15.16:0/4282488074 shutdown_connections 2019-12-05T06:31:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.397+0000 7fb56c8f4700 1 -- 172.21.15.16:0/4282488074 shutdown_connections 2019-12-05T06:31:59.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.397+0000 7fb56c8f4700 1 -- 172.21.15.16:0/4282488074 wait complete. 2019-12-05T06:31:59.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.397+0000 7fb56c8f4700 1 -- 172.21.15.16:0/4282488074 >> 172.21.15.16:0/4282488074 conn(0x7fb568061940 msgr2=0x7fb568096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.401+0000 7fb56c8f4700 1 Processor -- start 2019-12-05T06:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.401+0000 7fb56c8f4700 1 -- start start 2019-12-05T06:31:59.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb56c8f4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb56809ddf0 0x7fb56809a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb56c8f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56809b340 con 0x7fb568099df0 2019-12-05T06:31:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb56c8f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5681333d0 con 0x7fb56809c410 2019-12-05T06:31:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb56c8f4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb568132440 con 0x7fb56809ddf0 2019-12-05T06:31:59.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb566d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb568099df0 0x7fb56809d5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40624/0 (socket says 172.21.15.16:40624) 2019-12-05T06:31:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb566d9d700 1 -- 172.21.15.16:0/1596246105 learned_addr learned my addr 172.21.15.16:0/1596246105 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb56659c700 1 --2- 172.21.15.16:0/1596246105 >> v2:172.21.15.76:3300/0 conn(0x7fb56809ddf0 0x7fb56809a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1539005795 0 0) 0x7fb55800b6d0 con 0x7fb568099df0 2019-12-05T06:31:59.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb54c002e80 con 0x7fb568099df0 2019-12-05T06:31:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1402626250 0 0) 0x7fb5580009e0 con 0x7fb568099df0 2019-12-05T06:31:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb54c0032e0 con 0x7fb568099df0 2019-12-05T06:31:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fb558009770 con 0x7fb568099df0 2019-12-05T06:31:59.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 >> v2:172.21.15.76:3300/0 conn(0x7fb56809ddf0 msgr2=0x7fb56809a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 --2- 172.21.15.16:0/1596246105 >> v2:172.21.15.76:3300/0 conn(0x7fb56809ddf0 0x7fb56809a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 >> v1:172.21.15.76:6789/0 conn(0x7fb56809c410 legacy=0x7fb568155e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 >> v1:172.21.15.16:6789/0 conn(0x7fb568099df0 legacy=0x7fb56809d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb54c003330 0x7fb54c003760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb54c003ca0 0x7fb54c004120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c00bbf0 con 0x7fb54c004930 2019-12-05T06:31:59.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c00bde0 con 0x7fb54c003330 2019-12-05T06:31:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.402+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb54c00c010 con 0x7fb54c003ca0 2019-12-05T06:31:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56659c700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb54c003ca0 0x7fb54c004120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb566d9d700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb54c003330 0x7fb54c003760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56759e700 1 -- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb54c003ca0 msgr2=0x7fb54c004120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb54c003ca0 0x7fb54c004120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56759e700 1 -- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb54c003330 msgr2=0x7fb54c003760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb54c003330 0x7fb54c003760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56759e700 1 -- 172.21.15.16:0/1596246105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb54c00c2c0 con 0x7fb54c004930 2019-12-05T06:31:59.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb566d9d700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb54c003330 0x7fb54c003760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:31:59.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56c8f4700 1 -- 172.21.15.16:0/1596246105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb568132ad0 con 0x7fb54c004930 2019-12-05T06:31:59.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56c8f4700 1 -- 172.21.15.16:0/1596246105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56809b340 con 0x7fb54c004930 2019-12-05T06:31:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56759e700 1 -- 172.21.15.16:0/1596246105 reap_dead start 2019-12-05T06:31:59.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.403+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=READY pgs=939 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:59.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.404+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=THROTTLE_DONE pgs=939 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.404+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=THROTTLE_DONE pgs=939 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.404+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb560002460 con 0x7fb54c004930 2019-12-05T06:31:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.404+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=THROTTLE_DONE pgs=939 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.404+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb56000f030 con 0x7fb54c004930 2019-12-05T06:31:59.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.404+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb56000f740 con 0x7fb54c004930 2019-12-05T06:31:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.404+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=THROTTLE_DONE pgs=939 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.405+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb560000df0 con 0x7fb54c004930 2019-12-05T06:31:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.405+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=THROTTLE_DONE pgs=939 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.405+0000 7fb5577fe700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c026390 0x7fb54c0287c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.405+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fb560046230 con 0x7fb54c004930 2019-12-05T06:31:59.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.405+0000 7fb566d9d700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c026390 0x7fb54c0287c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.406+0000 7fb566d9d700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c026390 0x7fb54c0287c0 crc :-1 s=READY pgs=405 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.408+0000 7fb56c8f4700 1 -- 172.21.15.16:0/1596246105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb540002e60 con 0x7fb54c004930 2019-12-05T06:31:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.413+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=THROTTLE_DONE pgs=939 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.413+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb56002b250 con 0x7fb54c004930 2019-12-05T06:31:59.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.603+0000 7fb56c8f4700 1 -- 172.21.15.16:0/1596246105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb540001d10 con 0x7fb54c004930 2019-12-05T06:31:59.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.603+0000 7fb56759e700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=THROTTLE_DONE pgs=939 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.604+0000 7fb5577fe700 1 -- 172.21.15.16:0/1596246105 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fb560000df0 con 0x7fb54c004930 2019-12-05T06:31:59.601 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 118 2019-12-05T06:31:59.601 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:31:59.601 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:31:58.267130+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.606+0000 7fb56c8f4700 1 -- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c026390 msgr2=0x7fb54c0287c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.606+0000 7fb56c8f4700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c026390 0x7fb54c0287c0 crc :-1 s=READY pgs=405 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.606+0000 7fb56c8f4700 1 -- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 msgr2=0x7fb54c004dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.606+0000 7fb56c8f4700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 secure :-1 s=READY pgs=939 cs=0 l=1 rx=0x7fb560010740 tx=0x7fb5600105c0).stop 2019-12-05T06:31:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.606+0000 7fb56c8f4700 1 -- 172.21.15.16:0/1596246105 shutdown_connections 2019-12-05T06:31:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.606+0000 7fb56c8f4700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb54c026390 0x7fb54c0287c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.606+0000 7fb56c8f4700 1 --2- 172.21.15.16:0/1596246105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb54c004930 0x7fb54c004dd0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.606+0000 7fb56c8f4700 1 -- 172.21.15.16:0/1596246105 shutdown_connections 2019-12-05T06:31:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.607+0000 7fb56c8f4700 1 -- 172.21.15.16:0/1596246105 wait complete. 2019-12-05T06:31:59.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.607+0000 7fb56c8f4700 1 -- 172.21.15.16:0/1596246105 >> 172.21.15.16:0/1596246105 conn(0x7fb56808ad10 msgr2=0x7fb56806fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:59.614 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:31:59.614 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:31:59.615 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:31:59.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.742+0000 7fcd358b5700 1 Processor -- start 2019-12-05T06:31:59.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.742+0000 7fcd358b5700 1 -- start start 2019-12-05T06:31:59.750 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.743+0000 7fcd358b5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcd3008c6d0 0x7fcd3008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:31:59.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.743+0000 7fcd358b5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd3008fb80 con 0x7fcd3008fef0 2019-12-05T06:31:59.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.743+0000 7fcd358b5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd300958e0 con 0x7fcd3008f720 2019-12-05T06:31:59.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.743+0000 7fcd2effd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcd3008fef0 0x7fcd3008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40640/0 (socket says 172.21.15.16:40640) 2019-12-05T06:31:59.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.743+0000 7fcd358b5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd3008df90 con 0x7fcd3008c6d0 2019-12-05T06:31:59.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.743+0000 7fcd2effd700 1 -- 172.21.15.16:0/2145499444 learned_addr learned my addr 172.21.15.16:0/2145499444 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:31:59.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.743+0000 7fcd2f7fe700 1 --2- 172.21.15.16:0/2145499444 >> v2:172.21.15.76:3300/0 conn(0x7fcd3008c6d0 0x7fcd3008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.743+0000 7fcd2e7fc700 1 -- 172.21.15.16:0/2145499444 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223416879 0 0) 0x7fcd18001150 con 0x7fcd3008fef0 2019-12-05T06:31:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.743+0000 7fcd2e7fc700 1 -- 172.21.15.16:0/2145499444 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd1c002e80 con 0x7fcd3008fef0 2019-12-05T06:31:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2e7fc700 1 -- 172.21.15.16:0/2145499444 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1071949597 0 0) 0x7fcd18001130 con 0x7fcd3008fef0 2019-12-05T06:31:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2f7fe700 1 -- 172.21.15.16:0/2145499444 >> v1:172.21.15.76:6789/0 conn(0x7fcd3008f720 legacy=0x7fcd301533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2f7fe700 1 -- 172.21.15.16:0/2145499444 >> v1:172.21.15.16:6789/0 conn(0x7fcd3008fef0 legacy=0x7fcd3008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2f7fe700 1 -- 172.21.15.16:0/2145499444 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd3008e620 con 0x7fcd3008c6d0 2019-12-05T06:31:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2f7fe700 1 --2- 172.21.15.16:0/2145499444 >> v2:172.21.15.76:3300/0 conn(0x7fcd3008c6d0 0x7fcd3008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcd2800af50 tx=0x7fcd2800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2f7fe700 1 --2- 172.21.15.16:0/2145499444 >> v2:172.21.15.76:3300/0 conn(0x7fcd3008c6d0 0x7fcd3008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcd2800af50 tx=0x7fcd2800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2f7fe700 1 --2- 172.21.15.16:0/2145499444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3008c6d0 0x7fcd3008cac0 secure :-1 s=READY pgs=674 cs=0 l=1 rx=0x7fcd2800af50 tx=0x7fcd2800b760).ready entity=mon.1 client_cookie=81b3943262c0bd09 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2f7fe700 1 --2- 172.21.15.16:0/2145499444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3008c6d0 0x7fcd3008cac0 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7fcd2800af50 tx=0x7fcd2800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2e7fc700 1 -- 172.21.15.16:0/2145499444 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd28000960 con 0x7fcd3008c6d0 2019-12-05T06:31:59.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2f7fe700 1 --2- 172.21.15.16:0/2145499444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3008c6d0 0x7fcd3008cac0 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7fcd2800af50 tx=0x7fcd2800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2e7fc700 1 -- 172.21.15.16:0/2145499444 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd28000940 con 0x7fcd3008c6d0 2019-12-05T06:31:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2f7fe700 1 --2- 172.21.15.16:0/2145499444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3008c6d0 0x7fcd3008cac0 secure :-1 s=THROTTLE_DONE pgs=674 cs=0 l=1 rx=0x7fcd2800af50 tx=0x7fcd2800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd2e7fc700 1 -- 172.21.15.16:0/2145499444 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd28009ba0 con 0x7fcd3008c6d0 2019-12-05T06:31:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd358b5700 1 -- 172.21.15.16:0/2145499444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3008c6d0 msgr2=0x7fcd3008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.744+0000 7fcd358b5700 1 --2- 172.21.15.16:0/2145499444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3008c6d0 0x7fcd3008cac0 secure :-1 s=READY pgs=674 cs=0 l=1 rx=0x7fcd2800af50 tx=0x7fcd2800b760).stop 2019-12-05T06:31:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.745+0000 7fcd358b5700 1 -- 172.21.15.16:0/2145499444 shutdown_connections 2019-12-05T06:31:59.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.745+0000 7fcd358b5700 1 --2- 172.21.15.16:0/2145499444 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fcd3008c6d0 0x7fcd3008cac0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.745+0000 7fcd358b5700 1 -- 172.21.15.16:0/2145499444 shutdown_connections 2019-12-05T06:31:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.745+0000 7fcd358b5700 1 -- 172.21.15.16:0/2145499444 wait complete. 2019-12-05T06:31:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.745+0000 7fcd358b5700 1 -- 172.21.15.16:0/2145499444 >> 172.21.15.16:0/2145499444 conn(0x7fcd30061940 msgr2=0x7fcd30096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.747+0000 7fcd358b5700 1 Processor -- start 2019-12-05T06:31:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.748+0000 7fcd358b5700 1 -- start start 2019-12-05T06:31:59.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.748+0000 7fcd358b5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd301302f0 0x7fcd3012a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.748+0000 7fcd358b5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd3012b640 0x7fcd3012abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.748+0000 7fcd358b5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd30131bf0 con 0x7fcd3012b0f0 2019-12-05T06:31:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.748+0000 7fcd358b5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd30128cb0 con 0x7fcd3012b640 2019-12-05T06:31:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.748+0000 7fcd358b5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd30128f80 con 0x7fcd301302f0 2019-12-05T06:31:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.748+0000 7fcd2effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd3012b640 0x7fcd3012abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.748+0000 7fcd2effd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcd3012b640 0x7fcd3012abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55480/0 (socket says 172.21.15.16:55480) 2019-12-05T06:31:59.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.748+0000 7fcd2effd700 1 -- 172.21.15.16:0/2256840710 learned_addr learned my addr 172.21.15.16:0/2256840710 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:31:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.748+0000 7fcd2f7fe700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd301302f0 0x7fcd3012a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd2effd700 1 -- 172.21.15.16:0/2256840710 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd301302f0 msgr2=0x7fcd3012a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd301302f0 0x7fcd3012a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd2effd700 1 -- 172.21.15.16:0/2256840710 >> v1:172.21.15.16:6789/0 conn(0x7fcd3012b0f0 legacy=0x7fcd301306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.762 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd2effd700 1 -- 172.21.15.16:0/2256840710 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd30129d70 con 0x7fcd3012b640 2019-12-05T06:31:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> v2:172.21.15.16:3300/0 conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> v2:172.21.15.16:3300/0 conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=READY pgs=942 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=THROTTLE_DONE pgs=942 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=THROTTLE_DONE pgs=942 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd27fff700 1 -- 172.21.15.16:0/2256840710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd18000c90 con 0x7fcd3012b640 2019-12-05T06:31:59.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=THROTTLE_DONE pgs=942 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd27fff700 1 -- 172.21.15.16:0/2256840710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd180010c0 con 0x7fcd3012b640 2019-12-05T06:31:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd358b5700 1 -- 172.21.15.16:0/2256840710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd30129d70 con 0x7fcd3012b640 2019-12-05T06:31:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd27fff700 1 -- 172.21.15.16:0/2256840710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd18001790 con 0x7fcd3012b640 2019-12-05T06:31:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.749+0000 7fcd358b5700 1 -- 172.21.15.16:0/2256840710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd30129d70 con 0x7fcd3012b640 2019-12-05T06:31:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.750+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=THROTTLE_DONE pgs=942 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.750+0000 7fcd27fff700 1 -- 172.21.15.16:0/2256840710 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcd18000c90 con 0x7fcd3012b640 2019-12-05T06:31:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.750+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=THROTTLE_DONE pgs=942 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.750+0000 7fcd27fff700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd14023630 0x7fcd14025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:31:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.750+0000 7fcd27fff700 1 -- 172.21.15.16:0/2256840710 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fcd18047fb0 con 0x7fcd3012b640 2019-12-05T06:31:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.750+0000 7fcd2f7fe700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd14023630 0x7fcd14025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:31:59.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.751+0000 7fcd2f7fe700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd14023630 0x7fcd14025ac0 crc :-1 s=READY pgs=406 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:31:59.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.757+0000 7fcd358b5700 1 -- 172.21.15.16:0/2256840710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd30129d70 con 0x7fcd3012b640 2019-12-05T06:31:59.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.761+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=THROTTLE_DONE pgs=942 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.761+0000 7fcd27fff700 1 -- 172.21.15.16:0/2256840710 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcd180075b0 con 0x7fcd3012b640 2019-12-05T06:31:59.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.954+0000 7fcd358b5700 1 -- 172.21.15.16:0/2256840710 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcd30129d70 con 0x7fcd3012b640 2019-12-05T06:31:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.954+0000 7fcd2effd700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=THROTTLE_DONE pgs=942 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:31:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.954+0000 7fcd27fff700 1 -- 172.21.15.16:0/2256840710 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 118 v118) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fcd18000c90 con 0x7fcd3012b640 2019-12-05T06:31:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 118 2019-12-05T06:31:59.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 -- 172.21.15.16:0/2256840710 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd14023630 msgr2=0x7fcd14025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd14023630 0x7fcd14025ac0 crc :-1 s=READY pgs=406 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 -- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 msgr2=0x7fcd3012abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:31:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 0x7fcd3012abb0 secure :-1 s=READY pgs=942 cs=0 l=1 rx=0x7fcd18010270 tx=0x7fcd180128e0).stop 2019-12-05T06:31:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 -- 172.21.15.16:0/2256840710 shutdown_connections 2019-12-05T06:31:59.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcd14023630 0x7fcd14025ac0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.961 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:31:59.961 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":118,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":118,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:31:58.267130+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:replay","state_seq":1,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:31:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcd3012b640 0x7fcd3012abb0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 --2- 172.21.15.16:0/2256840710 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcd301302f0 0x7fcd3012a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:31:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 -- 172.21.15.16:0/2256840710 shutdown_connections 2019-12-05T06:31:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 -- 172.21.15.16:0/2256840710 wait complete. 2019-12-05T06:31:59.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:31:59.957+0000 7fcd358b5700 1 -- 172.21.15.16:0/2256840710 >> 172.21.15.16:0/2256840710 conn(0x7fcd300a2850 msgr2=0x7fcd3006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:31:59.966 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 118, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 118, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6561": { "addr": "172.21.15.16:6833/1695355780", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 1695355780, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 1695355780, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6561, "incarnation": 118, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 67, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:31:58.267130+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6561 } } } ], "standbys": [] } 2019-12-05T06:32:01.967 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:32:02.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.238+0000 7effb94ce700 1 Processor -- start 2019-12-05T06:32:02.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.238+0000 7effb94ce700 1 -- start start 2019-12-05T06:32:02.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.240+0000 7effb94ce700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7effb408c6d0 0x7effb408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:32:02.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.240+0000 7effb94ce700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:32:02.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.240+0000 7effb94ce700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb40958e0 con 0x7effb408ef20 2019-12-05T06:32:02.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.240+0000 7effb94ce700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb408df90 con 0x7effb408c6d0 2019-12-05T06:32:02.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.240+0000 7effb94ce700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb40a3f70 con 0x7effb408fef0 2019-12-05T06:32:02.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.240+0000 7effb2ffd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:02.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.240+0000 7effb37fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7effb408c6d0 0x7effb408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:02.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.241+0000 7effb37fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7effb408c6d0 0x7effb408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:41578/0 (socket says 172.21.15.16:41578) 2019-12-05T06:32:02.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.241+0000 7effb3fff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7effb408ef20 0x7effb408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:33728/0 (socket says 172.21.15.16:33728) 2019-12-05T06:32:02.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.241+0000 7effb37fe700 1 -- 172.21.15.16:0/4105144498 learned_addr learned my addr 172.21.15.16:0/4105144498 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:32:02.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.241+0000 7effb2ffd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:59480/0 (socket says 172.21.15.16:59480) 2019-12-05T06:32:02.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.241+0000 7effb2ffd700 1 -- 172.21.15.16:0/4105144498 >> v2:172.21.15.76:3300/0 conn(0x7effb408c6d0 msgr2=0x7effb408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:32:02.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.241+0000 7effb2ffd700 1 --2- 172.21.15.16:0/4105144498 >> v2:172.21.15.76:3300/0 conn(0x7effb408c6d0 0x7effb408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:32:02.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.241+0000 7effb2ffd700 1 -- 172.21.15.16:0/4105144498 >> v1:172.21.15.76:6789/0 conn(0x7effb408ef20 legacy=0x7effb408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:02.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.241+0000 7effb27fc700 1 -- 172.21.15.16:0/4105144498 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361359655 0 0) 0x7effa40009b0 con 0x7effb408ef20 2019-12-05T06:32:02.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.241+0000 7effb2ffd700 1 -- 172.21.15.16:0/4105144498 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb408d540 con 0x7effb408fef0 2019-12-05T06:32:02.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.241+0000 7effb37fe700 1 --2- 172.21.15.16:0/4105144498 >> v2:172.21.15.76:3300/0 conn(0x7effb408c6d0 0x7effb408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:32:02.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.242+0000 7effb2ffd700 1 --2- 172.21.15.16:0/4105144498 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7eff9c00af50 tx=0x7eff9c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.242+0000 7effb2ffd700 1 --2- 172.21.15.16:0/4105144498 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7eff9c00af50 tx=0x7eff9c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.242+0000 7effb2ffd700 1 --2- 172.21.15.16:0/4105144498 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 secure :-1 s=READY pgs=220 cs=0 l=1 rx=0x7eff9c00af50 tx=0x7eff9c00b760).ready entity=mon.2 client_cookie=4ac5fd046db5c419 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:02.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.242+0000 7effb2ffd700 1 --2- 172.21.15.16:0/4105144498 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 secure :-1 s=THROTTLE_DONE pgs=220 cs=0 l=1 rx=0x7eff9c00af50 tx=0x7eff9c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.242+0000 7effb2ffd700 1 --2- 172.21.15.16:0/4105144498 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 secure :-1 s=THROTTLE_DONE pgs=220 cs=0 l=1 rx=0x7eff9c00af50 tx=0x7eff9c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.242+0000 7effb2ffd700 1 --2- 172.21.15.16:0/4105144498 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 secure :-1 s=THROTTLE_DONE pgs=220 cs=0 l=1 rx=0x7eff9c00af50 tx=0x7eff9c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.242+0000 7effb27fc700 1 -- 172.21.15.16:0/4105144498 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff9c007210 con 0x7effb408fef0 2019-12-05T06:32:02.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.242+0000 7effb27fc700 1 -- 172.21.15.16:0/4105144498 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c0099b0 con 0x7effb408fef0 2019-12-05T06:32:02.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.243+0000 7effb27fc700 1 -- 172.21.15.16:0/4105144498 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff9c009f80 con 0x7effb408fef0 2019-12-05T06:32:02.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.243+0000 7effb94ce700 1 -- 172.21.15.16:0/4105144498 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 msgr2=0x7effb408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:02.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.243+0000 7effb94ce700 1 --2- 172.21.15.16:0/4105144498 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 secure :-1 s=READY pgs=220 cs=0 l=1 rx=0x7eff9c00af50 tx=0x7eff9c00b760).stop 2019-12-05T06:32:02.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.244+0000 7effb94ce700 1 -- 172.21.15.16:0/4105144498 shutdown_connections 2019-12-05T06:32:02.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.244+0000 7effb94ce700 1 --2- 172.21.15.16:0/4105144498 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb408fef0 0x7effb408d000 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:02.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.244+0000 7effb94ce700 1 --2- 172.21.15.16:0/4105144498 >> v2:172.21.15.76:3300/0 conn(0x7effb408c6d0 0x7effb408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:32:02.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.244+0000 7effb94ce700 1 -- 172.21.15.16:0/4105144498 shutdown_connections 2019-12-05T06:32:02.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.244+0000 7effb94ce700 1 -- 172.21.15.16:0/4105144498 wait complete. 2019-12-05T06:32:02.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.244+0000 7effb94ce700 1 -- 172.21.15.16:0/4105144498 >> 172.21.15.16:0/4105144498 conn(0x7effb4061940 msgr2=0x7effb4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:32:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.249+0000 7effb94ce700 1 Processor -- start 2019-12-05T06:32:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.249+0000 7effb94ce700 1 -- start start 2019-12-05T06:32:02.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.251+0000 7effb94ce700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7effb4096e90 0x7effb409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:02.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.251+0000 7effb94ce700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb409b920 0x7effb4097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:02.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.251+0000 7effb94ce700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effb4084b90 con 0x7effb4084f30 2019-12-05T06:32:02.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.251+0000 7effb94ce700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effb409a450 con 0x7effb4096e90 2019-12-05T06:32:02.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.251+0000 7effb94ce700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effb4099a20 con 0x7effb409b920 2019-12-05T06:32:02.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.251+0000 7effb37fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7effb4096e90 0x7effb409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:02.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.251+0000 7effb2ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7effb4084f30 0x7effb4097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40664/0 (socket says 172.21.15.16:40664) 2019-12-05T06:32:02.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.251+0000 7effb2ffd700 1 -- 172.21.15.16:0/3820847331 learned_addr learned my addr 172.21.15.16:0/3820847331 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:32:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.251+0000 7effb3fff700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb409b920 0x7effb4097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effabfff700 1 -- 172.21.15.16:0/3820847331 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773625223 0 0) 0x7eff9c00acc0 con 0x7effb4084f30 2019-12-05T06:32:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effabfff700 1 -- 172.21.15.16:0/3820847331 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff98002e80 con 0x7effb4084f30 2019-12-05T06:32:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb37fe700 1 -- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb409b920 msgr2=0x7effb4097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb409b920 0x7effb4097a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb37fe700 1 -- 172.21.15.16:0/3820847331 >> v1:172.21.15.16:6789/0 conn(0x7effb4084f30 legacy=0x7effb4097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:02.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb37fe700 1 -- 172.21.15.16:0/3820847331 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effb409aae0 con 0x7effb4096e90 2019-12-05T06:32:02.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb3fff700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb409b920 0x7effb4097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:32:02.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> v2:172.21.15.76:3300/0 conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> v2:172.21.15.76:3300/0 conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=READY pgs=676 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:02.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effabfff700 1 -- 172.21.15.16:0/3820847331 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effac001410 con 0x7effb4096e90 2019-12-05T06:32:02.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effabfff700 1 -- 172.21.15.16:0/3820847331 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effac000cc0 con 0x7effb4096e90 2019-12-05T06:32:02.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb94ce700 1 -- 172.21.15.16:0/3820847331 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effb409aae0 con 0x7effb4096e90 2019-12-05T06:32:02.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.252+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.253+0000 7effabfff700 1 -- 172.21.15.16:0/3820847331 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effac001600 con 0x7effb4096e90 2019-12-05T06:32:02.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.253+0000 7effb94ce700 1 -- 172.21.15.16:0/3820847331 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effb409aae0 con 0x7effb4096e90 2019-12-05T06:32:02.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.253+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.254+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.254+0000 7effabfff700 1 -- 172.21.15.16:0/3820847331 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7effac001220 con 0x7effb4096e90 2019-12-05T06:32:02.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.254+0000 7effabfff700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7eff98023670 0x7eff98025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:02.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.255+0000 7effabfff700 1 -- 172.21.15.16:0/3820847331 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7effac046330 con 0x7effb4096e90 2019-12-05T06:32:02.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.255+0000 7effb2ffd700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7eff98023670 0x7eff98025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:02.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.255+0000 7effb2ffd700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7eff98023670 0x7eff98025ae0 crc :-1 s=READY pgs=407 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:02.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.270+0000 7effb94ce700 1 -- 172.21.15.16:0/3820847331 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effb409aae0 con 0x7effb4096e90 2019-12-05T06:32:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.279+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.279+0000 7effabfff700 1 -- 172.21.15.16:0/3820847331 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7effac001010 con 0x7effb4096e90 2019-12-05T06:32:02.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.529+0000 7effb94ce700 1 -- 172.21.15.16:0/3820847331 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effb409aae0 con 0x7effb4096e90 2019-12-05T06:32:02.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.531+0000 7effb37fe700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=THROTTLE_DONE pgs=676 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:02.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.531+0000 7effabfff700 1 -- 172.21.15.16:0/3820847331 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 120 v120) v1 ==== 95+0+1775 (secure 0 0 0) 0x7effac001220 con 0x7effb4096e90 2019-12-05T06:32:02.529 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 120 2019-12-05T06:32:02.529 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:32:02.529 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":120,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":120,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:02.334694+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:rejoin","state_seq":21,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:32:02.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.536+0000 7effb94ce700 1 -- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7eff98023670 msgr2=0x7eff98025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.536+0000 7effb94ce700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7eff98023670 0x7eff98025ae0 crc :-1 s=READY pgs=407 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.536+0000 7effb94ce700 1 -- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 msgr2=0x7effb409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:02.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.536+0000 7effb94ce700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 0x7effb409b3e0 secure :-1 s=READY pgs=676 cs=0 l=1 rx=0x7effac010700 tx=0x7effac010aa0).stop 2019-12-05T06:32:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.536+0000 7effb94ce700 1 -- 172.21.15.16:0/3820847331 shutdown_connections 2019-12-05T06:32:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.536+0000 7effb94ce700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7eff98023670 0x7eff98025ae0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.536+0000 7effb94ce700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7effb409b920 0x7effb4097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:02.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.536+0000 7effb94ce700 1 --2- 172.21.15.16:0/3820847331 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7effb4096e90 0x7effb409b3e0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.537+0000 7effb94ce700 1 -- 172.21.15.16:0/3820847331 shutdown_connections 2019-12-05T06:32:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.537+0000 7effb94ce700 1 -- 172.21.15.16:0/3820847331 wait complete. 2019-12-05T06:32:02.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:02.537+0000 7effb94ce700 1 -- 172.21.15.16:0/3820847331 >> 172.21.15.16:0/3820847331 conn(0x7effb40714f0 msgr2=0x7effb409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:32:02.549 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:32:04.550 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:32:04.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.772+0000 7f559ad6f700 1 Processor -- start 2019-12-05T06:32:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.772+0000 7f559ad6f700 1 -- start start 2019-12-05T06:32:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.773+0000 7f559ad6f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f559408c6d0 0x7f559408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:32:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.773+0000 7f559ad6f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f559408f720 0x7f559408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:32:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.773+0000 7f559ad6f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f559408fb80 con 0x7f559408fef0 2019-12-05T06:32:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.773+0000 7f559ad6f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f559408df90 con 0x7f559408c6d0 2019-12-05T06:32:04.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.773+0000 7f559ad6f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55940a3f70 con 0x7f559408f720 2019-12-05T06:32:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.773+0000 7f559930c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f559408c6d0 0x7f559408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.773+0000 7f5598b0b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f559408fef0 0x7f559408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40678/0 (socket says 172.21.15.16:40678) 2019-12-05T06:32:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.773+0000 7f5598b0b700 1 -- 172.21.15.16:0/1750942453 learned_addr learned my addr 172.21.15.16:0/1750942453 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:32:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f5599b0d700 1 --2- 172.21.15.16:0/1750942453 >> v2:172.21.15.76:3300/0 conn(0x7f559408f720 0x7f559408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559930c700 1 -- 172.21.15.16:0/1750942453 >> v2:172.21.15.76:3300/0 conn(0x7f559408f720 msgr2=0x7f559408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:32:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559930c700 1 --2- 172.21.15.16:0/1750942453 >> v2:172.21.15.76:3300/0 conn(0x7f559408f720 0x7f559408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:32:04.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559930c700 1 -- 172.21.15.16:0/1750942453 >> v1:172.21.15.16:6789/0 conn(0x7f559408fef0 legacy=0x7f559408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f5593fff700 1 -- 172.21.15.16:0/1750942453 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210467105 0 0) 0x7f5584001150 con 0x7f559408fef0 2019-12-05T06:32:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559930c700 1 -- 172.21.15.16:0/1750942453 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f559408d540 con 0x7f559408c6d0 2019-12-05T06:32:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559930c700 1 --2- 172.21.15.16:0/1750942453 >> v2:172.21.15.16:3300/0 conn(0x7f559408c6d0 0x7f559408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f557c00af50 tx=0x7f557c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559930c700 1 --2- 172.21.15.16:0/1750942453 >> v2:172.21.15.16:3300/0 conn(0x7f559408c6d0 0x7f559408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f557c00af50 tx=0x7f557c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559930c700 1 --2- 172.21.15.16:0/1750942453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f559408c6d0 0x7f559408cac0 secure :-1 s=READY pgs=945 cs=0 l=1 rx=0x7f557c00af50 tx=0x7f557c00b6e0).ready entity=mon.0 client_cookie=38ec6ad5ef03218c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:04.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559930c700 1 --2- 172.21.15.16:0/1750942453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f559408c6d0 0x7f559408cac0 secure :-1 s=THROTTLE_DONE pgs=945 cs=0 l=1 rx=0x7f557c00af50 tx=0x7f557c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f5593fff700 1 -- 172.21.15.16:0/1750942453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f557c007210 con 0x7f559408c6d0 2019-12-05T06:32:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559930c700 1 --2- 172.21.15.16:0/1750942453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f559408c6d0 0x7f559408cac0 secure :-1 s=THROTTLE_DONE pgs=945 cs=0 l=1 rx=0x7f557c00af50 tx=0x7f557c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559930c700 1 --2- 172.21.15.16:0/1750942453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f559408c6d0 0x7f559408cac0 secure :-1 s=THROTTLE_DONE pgs=945 cs=0 l=1 rx=0x7f557c00af50 tx=0x7f557c00b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f5593fff700 1 -- 172.21.15.16:0/1750942453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f557c0099b0 con 0x7f559408c6d0 2019-12-05T06:32:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f5593fff700 1 -- 172.21.15.16:0/1750942453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f557c00a080 con 0x7f559408c6d0 2019-12-05T06:32:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559ad6f700 1 -- 172.21.15.16:0/1750942453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f559408c6d0 msgr2=0x7f559408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.774+0000 7f559ad6f700 1 --2- 172.21.15.16:0/1750942453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f559408c6d0 0x7f559408cac0 secure :-1 s=READY pgs=945 cs=0 l=1 rx=0x7f557c00af50 tx=0x7f557c00b6e0).stop 2019-12-05T06:32:04.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.775+0000 7f559ad6f700 1 -- 172.21.15.16:0/1750942453 shutdown_connections 2019-12-05T06:32:04.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.775+0000 7f559ad6f700 1 --2- 172.21.15.16:0/1750942453 >> v2:172.21.15.76:3300/0 conn(0x7f559408f720 0x7f559408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:32:04.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.775+0000 7f559ad6f700 1 --2- 172.21.15.16:0/1750942453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f559408c6d0 0x7f559408cac0 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:04.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.775+0000 7f559ad6f700 1 -- 172.21.15.16:0/1750942453 shutdown_connections 2019-12-05T06:32:04.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.775+0000 7f559ad6f700 1 -- 172.21.15.16:0/1750942453 wait complete. 2019-12-05T06:32:04.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.775+0000 7f559ad6f700 1 -- 172.21.15.16:0/1750942453 >> 172.21.15.16:0/1750942453 conn(0x7f5594061940 msgr2=0x7f5594096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:32:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.778+0000 7f559ad6f700 1 Processor -- start 2019-12-05T06:32:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.778+0000 7f559ad6f700 1 -- start start 2019-12-05T06:32:04.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f559ad6f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55940992d0 0x7f5594098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f559ad6f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5594156cb0 0x7f55941590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f559ad6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5594089fb0 con 0x7f559406df10 2019-12-05T06:32:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f559ad6f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5594097370 con 0x7f55940992d0 2019-12-05T06:32:04.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f559ad6f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f559408ad00 con 0x7f5594156cb0 2019-12-05T06:32:04.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55940992d0 0x7f5594098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:04.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f55940992d0 0x7f5594098b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55516/0 (socket says 172.21.15.16:55516) 2019-12-05T06:32:04.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 -- 172.21.15.16:0/1644011077 learned_addr learned my addr 172.21.15.16:0/1644011077 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:32:04.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5599b0d700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5594156cb0 0x7f55941590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:04.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 -- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5594156cb0 msgr2=0x7f55941590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:04.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5594156cb0 0x7f55941590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:04.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 -- 172.21.15.16:0/1644011077 >> v1:172.21.15.76:6789/0 conn(0x7f559406df10 legacy=0x7f5594098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:04.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 -- 172.21.15.16:0/1644011077 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5594097a00 con 0x7f55940992d0 2019-12-05T06:32:04.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5599b0d700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5594156cb0 0x7f55941590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:32:04.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> v2:172.21.15.16:3300/0 conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> v2:172.21.15.16:3300/0 conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=READY pgs=946 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=THROTTLE_DONE pgs=946 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=THROTTLE_DONE pgs=946 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=THROTTLE_DONE pgs=946 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5591ffb700 1 -- 172.21.15.16:0/1644011077 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5584000ed0 con 0x7f55940992d0 2019-12-05T06:32:04.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5591ffb700 1 -- 172.21.15.16:0/1644011077 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5584004430 con 0x7f55940992d0 2019-12-05T06:32:04.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f5591ffb700 1 -- 172.21.15.16:0/1644011077 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5584004a80 con 0x7f55940992d0 2019-12-05T06:32:04.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.779+0000 7f559ad6f700 1 -- 172.21.15.16:0/1644011077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5594097a00 con 0x7f55940992d0 2019-12-05T06:32:04.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.780+0000 7f559ad6f700 1 -- 172.21.15.16:0/1644011077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5594097a00 con 0x7f55940992d0 2019-12-05T06:32:04.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.780+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=THROTTLE_DONE pgs=946 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.780+0000 7f5591ffb700 1 -- 172.21.15.16:0/1644011077 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5584004620 con 0x7f55940992d0 2019-12-05T06:32:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.780+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=THROTTLE_DONE pgs=946 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.781+0000 7f5591ffb700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5578023630 0x7f5578025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.781+0000 7f5591ffb700 1 -- 172.21.15.16:0/1644011077 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f5584045e80 con 0x7f55940992d0 2019-12-05T06:32:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.781+0000 7f559930c700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5578023630 0x7f5578025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.781+0000 7f559930c700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5578023630 0x7f5578025ac0 crc :-1 s=READY pgs=408 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.788+0000 7f559ad6f700 1 -- 172.21.15.16:0/1644011077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5594097a00 con 0x7f55940992d0 2019-12-05T06:32:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.791+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=THROTTLE_DONE pgs=946 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.791+0000 7f5591ffb700 1 -- 172.21.15.16:0/1644011077 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f55840012b0 con 0x7f55940992d0 2019-12-05T06:32:04.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.990+0000 7f559ad6f700 1 -- 172.21.15.16:0/1644011077 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5594097a00 con 0x7f55940992d0 2019-12-05T06:32:04.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.991+0000 7f5598b0b700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=THROTTLE_DONE pgs=946 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:04.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.991+0000 7f5591ffb700 1 -- 172.21.15.16:0/1644011077 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f5584004600 con 0x7f55940992d0 2019-12-05T06:32:04.988 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 121 2019-12-05T06:32:04.988 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:32:04.988 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:32:04.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.993+0000 7f559ad6f700 1 -- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5578023630 msgr2=0x7f5578025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:04.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.993+0000 7f559ad6f700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5578023630 0x7f5578025ac0 crc :-1 s=READY pgs=408 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:04.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.993+0000 7f559ad6f700 1 -- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 msgr2=0x7f5594098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:04.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.993+0000 7f559ad6f700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 0x7f5594098b00 secure :-1 s=READY pgs=946 cs=0 l=1 rx=0x7f5584010430 tx=0x7f55840107d0).stop 2019-12-05T06:32:04.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.994+0000 7f559ad6f700 1 -- 172.21.15.16:0/1644011077 shutdown_connections 2019-12-05T06:32:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.994+0000 7f559ad6f700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5594156cb0 0x7f55941590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.994+0000 7f559ad6f700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5578023630 0x7f5578025ac0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.994+0000 7f559ad6f700 1 --2- 172.21.15.16:0/1644011077 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f55940992d0 0x7f5594098b00 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.994+0000 7f559ad6f700 1 -- 172.21.15.16:0/1644011077 shutdown_connections 2019-12-05T06:32:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.994+0000 7f559ad6f700 1 -- 172.21.15.16:0/1644011077 wait complete. 2019-12-05T06:32:04.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:04.994+0000 7f559ad6f700 1 -- 172.21.15.16:0/1644011077 >> 172.21.15.16:0/1644011077 conn(0x7f55940714f0 msgr2=0x7f559409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:32:05.001 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6544 2019-12-05T06:32:05.001 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.b 2019-12-05T06:32:06.046 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:32:06.050 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:32:09.002 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:32:09.002 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:32:09.002 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:32:09.007 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:32:09.008 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:32:09.200 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:32:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.271+0000 7fa4433bd700 1 Processor -- start 2019-12-05T06:32:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.271+0000 7fa4433bd700 1 -- start start 2019-12-05T06:32:09.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa4433bd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c08d2c0 0x7fa43c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:32:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa4433bd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa43c08fef0 0x7fa43c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:32:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa4433bd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa43c0958e0 con 0x7fa43c08c6d0 2019-12-05T06:32:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa4433bd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa43c08df90 con 0x7fa43c08fef0 2019-12-05T06:32:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa4433bd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c0a3f70 con 0x7fa43c08d2c0 2019-12-05T06:32:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa44215b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa43c08fef0 0x7fa43c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:09.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa44215b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa43c08fef0 0x7fa43c08f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55556/0 (socket says 172.21.15.16:55556) 2019-12-05T06:32:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa44215b700 1 -- 172.21.15.16:0/2461201079 learned_addr learned my addr 172.21.15.16:0/2461201079 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:32:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa441159700 1 --2- 172.21.15.16:0/2461201079 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c08d2c0 0x7fa43c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa44215b700 1 -- 172.21.15.16:0/2461201079 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c08d2c0 msgr2=0x7fa43c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:32:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa44215b700 1 --2- 172.21.15.16:0/2461201079 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c08d2c0 0x7fa43c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:32:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa44215b700 1 -- 172.21.15.16:0/2461201079 >> v1:172.21.15.76:6789/0 conn(0x7fa43c08c6d0 legacy=0x7fa43c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa44215b700 1 -- 172.21.15.16:0/2461201079 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43c08e620 con 0x7fa43c08fef0 2019-12-05T06:32:09.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.273+0000 7fa44215b700 1 --2- 172.21.15.16:0/2461201079 >> v2:172.21.15.16:3300/0 conn(0x7fa43c08fef0 0x7fa43c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa42c00f730 tx=0x7fa42c007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.274+0000 7fa44215b700 1 --2- 172.21.15.16:0/2461201079 >> v2:172.21.15.16:3300/0 conn(0x7fa43c08fef0 0x7fa43c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa42c00f730 tx=0x7fa42c007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.274+0000 7fa44215b700 1 --2- 172.21.15.16:0/2461201079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c08fef0 0x7fa43c08f460 secure :-1 s=READY pgs=948 cs=0 l=1 rx=0x7fa42c00f730 tx=0x7fa42c007d50).ready entity=mon.0 client_cookie=d3009bface01f07b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.274+0000 7fa44215b700 1 --2- 172.21.15.16:0/2461201079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c08fef0 0x7fa43c08f460 secure :-1 s=THROTTLE_DONE pgs=948 cs=0 l=1 rx=0x7fa42c00f730 tx=0x7fa42c007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.274+0000 7fa44215b700 1 --2- 172.21.15.16:0/2461201079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c08fef0 0x7fa43c08f460 secure :-1 s=THROTTLE_DONE pgs=948 cs=0 l=1 rx=0x7fa42c00f730 tx=0x7fa42c007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.274+0000 7fa440958700 1 -- 172.21.15.16:0/2461201079 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa42c000a70 con 0x7fa43c08fef0 2019-12-05T06:32:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.274+0000 7fa44215b700 1 --2- 172.21.15.16:0/2461201079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c08fef0 0x7fa43c08f460 secure :-1 s=THROTTLE_DONE pgs=948 cs=0 l=1 rx=0x7fa42c00f730 tx=0x7fa42c007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.274+0000 7fa440958700 1 -- 172.21.15.16:0/2461201079 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42c0010e0 con 0x7fa43c08fef0 2019-12-05T06:32:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.274+0000 7fa440958700 1 -- 172.21.15.16:0/2461201079 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa42c0017b0 con 0x7fa43c08fef0 2019-12-05T06:32:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.274+0000 7fa4433bd700 1 -- 172.21.15.16:0/2461201079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c08fef0 msgr2=0x7fa43c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.274+0000 7fa4433bd700 1 --2- 172.21.15.16:0/2461201079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c08fef0 0x7fa43c08f460 secure :-1 s=READY pgs=948 cs=0 l=1 rx=0x7fa42c00f730 tx=0x7fa42c007d50).stop 2019-12-05T06:32:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.275+0000 7fa4433bd700 1 -- 172.21.15.16:0/2461201079 shutdown_connections 2019-12-05T06:32:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.275+0000 7fa4433bd700 1 --2- 172.21.15.16:0/2461201079 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c08fef0 0x7fa43c08f460 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.275+0000 7fa4433bd700 1 --2- 172.21.15.16:0/2461201079 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c08d2c0 0x7fa43c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:32:09.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.275+0000 7fa4433bd700 1 -- 172.21.15.16:0/2461201079 shutdown_connections 2019-12-05T06:32:09.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.275+0000 7fa4433bd700 1 -- 172.21.15.16:0/2461201079 wait complete. 2019-12-05T06:32:09.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.275+0000 7fa4433bd700 1 -- 172.21.15.16:0/2461201079 >> 172.21.15.16:0/2461201079 conn(0x7fa43c061940 msgr2=0x7fa43c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:32:09.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.279+0000 7fa4433bd700 1 Processor -- start 2019-12-05T06:32:09.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.279+0000 7fa4433bd700 1 -- start start 2019-12-05T06:32:09.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.280+0000 7fa4433bd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c06dc10 0x7fa43c06e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:09.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.280+0000 7fa4433bd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa43c130410 0x7fa43c06e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:09.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.280+0000 7fa4433bd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa43c128690 con 0x7fa43c130be0 2019-12-05T06:32:09.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.280+0000 7fa4433bd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa43c09e690 con 0x7fa43c130410 2019-12-05T06:32:09.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.280+0000 7fa4433bd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa43c12e250 con 0x7fa43c06dc10 2019-12-05T06:32:09.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.280+0000 7fa44215b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa43c130410 0x7fa43c06e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:09.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.280+0000 7fa44215b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa43c130410 0x7fa43c06e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55560/0 (socket says 172.21.15.16:55560) 2019-12-05T06:32:09.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.280+0000 7fa44215b700 1 -- 172.21.15.16:0/38101654 learned_addr learned my addr 172.21.15.16:0/38101654 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:32:09.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.280+0000 7fa44195a700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c06dc10 0x7fa43c06e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:09.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.280+0000 7fa44215b700 1 -- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c06dc10 msgr2=0x7fa43c06e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:09.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c06dc10 0x7fa43c06e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:09.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa44215b700 1 -- 172.21.15.16:0/38101654 >> v1:172.21.15.76:6789/0 conn(0x7fa43c130be0 legacy=0x7fa43c12fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:09.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa44215b700 1 -- 172.21.15.16:0/38101654 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa43c12f310 con 0x7fa43c130410 2019-12-05T06:32:09.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa44195a700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c06dc10 0x7fa43c06e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:32:09.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> v2:172.21.15.16:3300/0 conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> v2:172.21.15.16:3300/0 conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=READY pgs=949 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa43a7fc700 1 -- 172.21.15.16:0/38101654 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa42c0014a0 con 0x7fa43c130410 2019-12-05T06:32:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa43a7fc700 1 -- 172.21.15.16:0/38101654 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa42c0010c0 con 0x7fa43c130410 2019-12-05T06:32:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa4433bd700 1 -- 172.21.15.16:0/38101654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa43c12f310 con 0x7fa43c130410 2019-12-05T06:32:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.281+0000 7fa43a7fc700 1 -- 172.21.15.16:0/38101654 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa42c001a80 con 0x7fa43c130410 2019-12-05T06:32:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.282+0000 7fa4433bd700 1 -- 172.21.15.16:0/38101654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa43c12f310 con 0x7fa43c130410 2019-12-05T06:32:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.282+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.283+0000 7fa43a7fc700 1 -- 172.21.15.16:0/38101654 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa42c0012b0 con 0x7fa43c130410 2019-12-05T06:32:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.283+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.283+0000 7fa43a7fc700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa41c023630 0x7fa41c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.283+0000 7fa43a7fc700 1 -- 172.21.15.16:0/38101654 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fa42c04a1d0 con 0x7fa43c130410 2019-12-05T06:32:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.283+0000 7fa44195a700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa41c023630 0x7fa41c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.284+0000 7fa44195a700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa41c023630 0x7fa41c025ac0 crc :-1 s=READY pgs=409 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:09.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.286+0000 7fa4433bd700 1 -- 172.21.15.16:0/38101654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa43c12f310 con 0x7fa43c130410 2019-12-05T06:32:09.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.291+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.291+0000 7fa43a7fc700 1 -- 172.21.15.16:0/38101654 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa42c007810 con 0x7fa43c130410 2019-12-05T06:32:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.509+0000 7fa4433bd700 1 -- 172.21.15.16:0/38101654 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa43c12f310 con 0x7fa43c130410 2019-12-05T06:32:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.509+0000 7fa44215b700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=THROTTLE_DONE pgs=949 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.509+0000 7fa43a7fc700 1 -- 172.21.15.16:0/38101654 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fa42c0012b0 con 0x7fa43c130410 2019-12-05T06:32:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 121 2019-12-05T06:32:09.506 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:32:09.506 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:32:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 -- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa41c023630 msgr2=0x7fa41c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa41c023630 0x7fa41c025ac0 crc :-1 s=READY pgs=409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 -- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 msgr2=0x7fa43c06e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 0x7fa43c06e540 secure :-1 s=READY pgs=949 cs=0 l=1 rx=0x7fa42c014890 tx=0x7fa42c014c30).stop 2019-12-05T06:32:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 -- 172.21.15.16:0/38101654 shutdown_connections 2019-12-05T06:32:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa41c023630 0x7fa41c025ac0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa43c130410 0x7fa43c06e540 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 --2- 172.21.15.16:0/38101654 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa43c06dc10 0x7fa43c06e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 -- 172.21.15.16:0/38101654 shutdown_connections 2019-12-05T06:32:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 -- 172.21.15.16:0/38101654 wait complete. 2019-12-05T06:32:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:09.512+0000 7fa4433bd700 1 -- 172.21.15.16:0/38101654 >> 172.21.15.16:0/38101654 conn(0x7fa43c098480 msgr2=0x7fa43c071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:32:09.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:32:11.520 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:32:11.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.789+0000 7fbc494ad700 1 Processor -- start 2019-12-05T06:32:11.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.789+0000 7fbc494ad700 1 -- start start 2019-12-05T06:32:11.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.791+0000 7fbc494ad700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc4408c6d0 0x7fbc4408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:32:11.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.791+0000 7fbc494ad700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4408fb80 con 0x7fbc4408f720 2019-12-05T06:32:11.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.791+0000 7fbc494ad700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc440958e0 con 0x7fbc4408fef0 2019-12-05T06:32:11.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.791+0000 7fbc494ad700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc4408df90 con 0x7fbc4408c6d0 2019-12-05T06:32:11.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.791+0000 7fbc437fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc4408c6d0 0x7fbc4408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:11.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.791+0000 7fbc437fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbc4408c6d0 0x7fbc4408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55584/0 (socket says 172.21.15.16:55584) 2019-12-05T06:32:11.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.791+0000 7fbc437fe700 1 -- 172.21.15.16:0/2959175189 learned_addr learned my addr 172.21.15.16:0/2959175189 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:32:11.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.791+0000 7fbc43fff700 1 --1- 172.21.15.16:0/2959175189 >> v1:172.21.15.16:6789/0 conn(0x7fbc4408f720 0x7fbc441533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40754/0 (socket says 172.21.15.16:40754) 2019-12-05T06:32:11.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.792+0000 7fbc427fc700 1 -- 172.21.15.16:0/2959175189 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1666750851 0 0) 0x7fbc2c001080 con 0x7fbc4408fef0 2019-12-05T06:32:11.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.792+0000 7fbc427fc700 1 -- 172.21.15.16:0/2959175189 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc30002e80 con 0x7fbc4408fef0 2019-12-05T06:32:11.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.792+0000 7fbc437fe700 1 -- 172.21.15.16:0/2959175189 >> v1:172.21.15.76:6789/0 conn(0x7fbc4408fef0 legacy=0x7fbc4408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:11.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.792+0000 7fbc427fc700 1 -- 172.21.15.16:0/2959175189 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1237943395 0 0) 0x7fbc340009b0 con 0x7fbc4408f720 2019-12-05T06:32:11.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.792+0000 7fbc437fe700 1 -- 172.21.15.16:0/2959175189 >> v1:172.21.15.16:6789/0 conn(0x7fbc4408f720 legacy=0x7fbc441533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:11.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.792+0000 7fbc437fe700 1 -- 172.21.15.16:0/2959175189 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4408e620 con 0x7fbc4408c6d0 2019-12-05T06:32:11.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.792+0000 7fbc437fe700 1 --2- 172.21.15.16:0/2959175189 >> v2:172.21.15.16:3300/0 conn(0x7fbc4408c6d0 0x7fbc4408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbc3c00fac0 tx=0x7fbc3c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.793+0000 7fbc437fe700 1 --2- 172.21.15.16:0/2959175189 >> v2:172.21.15.16:3300/0 conn(0x7fbc4408c6d0 0x7fbc4408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbc3c00fac0 tx=0x7fbc3c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.793+0000 7fbc437fe700 1 --2- 172.21.15.16:0/2959175189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4408c6d0 0x7fbc4408cac0 secure :-1 s=READY pgs=951 cs=0 l=1 rx=0x7fbc3c00fac0 tx=0x7fbc3c00fe60).ready entity=mon.0 client_cookie=bf0f96a4349a48a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:11.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.793+0000 7fbc437fe700 1 --2- 172.21.15.16:0/2959175189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4408c6d0 0x7fbc4408cac0 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7fbc3c00fac0 tx=0x7fbc3c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.793+0000 7fbc437fe700 1 --2- 172.21.15.16:0/2959175189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4408c6d0 0x7fbc4408cac0 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7fbc3c00fac0 tx=0x7fbc3c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.793+0000 7fbc427fc700 1 -- 172.21.15.16:0/2959175189 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc3c000ce0 con 0x7fbc4408c6d0 2019-12-05T06:32:11.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.793+0000 7fbc437fe700 1 --2- 172.21.15.16:0/2959175189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4408c6d0 0x7fbc4408cac0 secure :-1 s=THROTTLE_DONE pgs=951 cs=0 l=1 rx=0x7fbc3c00fac0 tx=0x7fbc3c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.793+0000 7fbc427fc700 1 -- 172.21.15.16:0/2959175189 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc3c001010 con 0x7fbc4408c6d0 2019-12-05T06:32:11.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.793+0000 7fbc427fc700 1 -- 172.21.15.16:0/2959175189 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc3c0016e0 con 0x7fbc4408c6d0 2019-12-05T06:32:11.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.793+0000 7fbc494ad700 1 -- 172.21.15.16:0/2959175189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4408c6d0 msgr2=0x7fbc4408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:11.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.793+0000 7fbc494ad700 1 --2- 172.21.15.16:0/2959175189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4408c6d0 0x7fbc4408cac0 secure :-1 s=READY pgs=951 cs=0 l=1 rx=0x7fbc3c00fac0 tx=0x7fbc3c00fe60).stop 2019-12-05T06:32:11.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.795+0000 7fbc494ad700 1 -- 172.21.15.16:0/2959175189 shutdown_connections 2019-12-05T06:32:11.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.795+0000 7fbc494ad700 1 --2- 172.21.15.16:0/2959175189 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbc4408c6d0 0x7fbc4408cac0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:11.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.795+0000 7fbc494ad700 1 -- 172.21.15.16:0/2959175189 shutdown_connections 2019-12-05T06:32:11.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.795+0000 7fbc494ad700 1 -- 172.21.15.16:0/2959175189 wait complete. 2019-12-05T06:32:11.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.795+0000 7fbc494ad700 1 -- 172.21.15.16:0/2959175189 >> 172.21.15.16:0/2959175189 conn(0x7fbc44061940 msgr2=0x7fbc44096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:32:11.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.800+0000 7fbc494ad700 1 Processor -- start 2019-12-05T06:32:11.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.800+0000 7fbc494ad700 1 -- start start 2019-12-05T06:32:11.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.802+0000 7fbc494ad700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbc4414f9a0 0x7fbc44098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:11.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.802+0000 7fbc494ad700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc44089fb0 con 0x7fbc4414f5b0 2019-12-05T06:32:11.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.802+0000 7fbc494ad700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4406e7c0 con 0x7fbc440992d0 2019-12-05T06:32:11.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.802+0000 7fbc494ad700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbc44097370 con 0x7fbc4414f9a0 2019-12-05T06:32:11.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.802+0000 7fbc437fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbc4414f5b0 0x7fbc44098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40758/0 (socket says 172.21.15.16:40758) 2019-12-05T06:32:11.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.802+0000 7fbc437fe700 1 -- 172.21.15.16:0/1298546224 learned_addr learned my addr 172.21.15.16:0/1298546224 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:32:11.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.802+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> v2:172.21.15.76:3300/0 conn(0x7fbc4414f9a0 0x7fbc44098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:11.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.803+0000 7fbc3bfff700 1 -- 172.21.15.16:0/1298546224 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2476310006 0 0) 0x7fbc3c007c40 con 0x7fbc4414f5b0 2019-12-05T06:32:11.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.803+0000 7fbc3bfff700 1 -- 172.21.15.16:0/1298546224 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc28002e80 con 0x7fbc4414f5b0 2019-12-05T06:32:11.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.803+0000 7fbc3bfff700 1 -- 172.21.15.16:0/1298546224 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4192161463 0 0) 0x7fbc3c008560 con 0x7fbc4414f5b0 2019-12-05T06:32:11.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.803+0000 7fbc43fff700 1 -- 172.21.15.16:0/1298546224 >> v1:172.21.15.76:6789/0 conn(0x7fbc440992d0 legacy=0x7fbc441572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:11.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.803+0000 7fbc43fff700 1 -- 172.21.15.16:0/1298546224 >> v1:172.21.15.16:6789/0 conn(0x7fbc4414f5b0 legacy=0x7fbc44098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:11.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.803+0000 7fbc43fff700 1 -- 172.21.15.16:0/1298546224 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc44097a00 con 0x7fbc4414f9a0 2019-12-05T06:32:11.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.803+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> v2:172.21.15.76:3300/0 conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.804+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> v2:172.21.15.76:3300/0 conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.804+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=READY pgs=685 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:11.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.804+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.804+0000 7fbc3bfff700 1 -- 172.21.15.16:0/1298546224 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc34001570 con 0x7fbc4414f9a0 2019-12-05T06:32:11.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.804+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.804+0000 7fbc3bfff700 1 -- 172.21.15.16:0/1298546224 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc34001190 con 0x7fbc4414f9a0 2019-12-05T06:32:11.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.804+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.804+0000 7fbc494ad700 1 -- 172.21.15.16:0/1298546224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc44097a00 con 0x7fbc4414f9a0 2019-12-05T06:32:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.804+0000 7fbc3bfff700 1 -- 172.21.15.16:0/1298546224 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc34001570 con 0x7fbc4414f9a0 2019-12-05T06:32:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.804+0000 7fbc494ad700 1 -- 172.21.15.16:0/1298546224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc44097a00 con 0x7fbc4414f9a0 2019-12-05T06:32:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.805+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.806+0000 7fbc3bfff700 1 -- 172.21.15.16:0/1298546224 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbc34001380 con 0x7fbc4414f9a0 2019-12-05T06:32:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.806+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.807+0000 7fbc3bfff700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbc28023680 0x7fbc28025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:11.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.807+0000 7fbc3bfff700 1 -- 172.21.15.16:0/1298546224 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fbc34045df0 con 0x7fbc4414f9a0 2019-12-05T06:32:11.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.807+0000 7fbc437fe700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbc28023680 0x7fbc28025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:11.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.807+0000 7fbc437fe700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbc28023680 0x7fbc28025af0 crc :-1 s=READY pgs=410 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:11.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.821+0000 7fbc494ad700 1 -- 172.21.15.16:0/1298546224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc44097a00 con 0x7fbc4414f9a0 2019-12-05T06:32:11.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.829+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:11.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:11.829+0000 7fbc3bfff700 1 -- 172.21.15.16:0/1298546224 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbc34028510 con 0x7fbc4414f9a0 2019-12-05T06:32:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.093+0000 7fbc494ad700 1 -- 172.21.15.16:0/1298546224 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc44097a00 con 0x7fbc4414f9a0 2019-12-05T06:32:12.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.094+0000 7fbc43fff700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=THROTTLE_DONE pgs=685 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:12.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.094+0000 7fbc3bfff700 1 -- 172.21.15.16:0/1298546224 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 121 v121) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fbc34028510 con 0x7fbc4414f9a0 2019-12-05T06:32:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 121 2019-12-05T06:32:12.092 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:32:12.092 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":121,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:32:12.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.100+0000 7fbc494ad700 1 -- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbc28023680 msgr2=0x7fbc28025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.100+0000 7fbc494ad700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbc28023680 0x7fbc28025af0 crc :-1 s=READY pgs=410 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.100+0000 7fbc494ad700 1 -- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 msgr2=0x7fbc44098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.100+0000 7fbc494ad700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 0x7fbc44098b00 secure :-1 s=READY pgs=685 cs=0 l=1 rx=0x7fbc340103a0 tx=0x7fbc34010740).stop 2019-12-05T06:32:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.100+0000 7fbc494ad700 1 -- 172.21.15.16:0/1298546224 shutdown_connections 2019-12-05T06:32:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.100+0000 7fbc494ad700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbc28023680 0x7fbc28025af0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:12.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.100+0000 7fbc494ad700 1 --2- 172.21.15.16:0/1298546224 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbc4414f9a0 0x7fbc44098b00 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.100+0000 7fbc494ad700 1 -- 172.21.15.16:0/1298546224 shutdown_connections 2019-12-05T06:32:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.101+0000 7fbc494ad700 1 -- 172.21.15.16:0/1298546224 wait complete. 2019-12-05T06:32:12.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:12.101+0000 7fbc494ad700 1 -- 172.21.15.16:0/1298546224 >> 172.21.15.16:0/1298546224 conn(0x7fbc440714f0 msgr2=0x7fbc4409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:32:12.113 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:32:14.113 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:32:14.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.270+0000 7fa616c82700 1 Processor -- start 2019-12-05T06:32:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.270+0000 7fa616c82700 1 -- start start 2019-12-05T06:32:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa616c82700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa61008c6d0 0x7fa61008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:32:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa616c82700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa61008f720 0x7fa61008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:32:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa616c82700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa61008fb80 con 0x7fa61008fef0 2019-12-05T06:32:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa616c82700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61008df90 con 0x7fa61008c6d0 2019-12-05T06:32:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa616c82700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6100a3f70 con 0x7fa61008f720 2019-12-05T06:32:14.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa61521f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa61008c6d0 0x7fa61008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa614a1e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa61008fef0 0x7fa61008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40778/0 (socket says 172.21.15.16:40778) 2019-12-05T06:32:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa614a1e700 1 -- 172.21.15.16:0/1366231803 learned_addr learned my addr 172.21.15.16:0/1366231803 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:32:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa615a20700 1 --2- 172.21.15.16:0/1366231803 >> v2:172.21.15.76:3300/0 conn(0x7fa61008f720 0x7fa61008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa61521f700 1 -- 172.21.15.16:0/1366231803 >> v2:172.21.15.76:3300/0 conn(0x7fa61008f720 msgr2=0x7fa61008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:32:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa61521f700 1 --2- 172.21.15.16:0/1366231803 >> v2:172.21.15.76:3300/0 conn(0x7fa61008f720 0x7fa61008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:32:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa61521f700 1 -- 172.21.15.16:0/1366231803 >> v1:172.21.15.16:6789/0 conn(0x7fa61008fef0 legacy=0x7fa61008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:14.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa60ffff700 1 -- 172.21.15.16:0/1366231803 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 409768362 0 0) 0x7fa604001150 con 0x7fa61008fef0 2019-12-05T06:32:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa61521f700 1 -- 172.21.15.16:0/1366231803 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61008d540 con 0x7fa61008c6d0 2019-12-05T06:32:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa61521f700 1 --2- 172.21.15.16:0/1366231803 >> v2:172.21.15.16:3300/0 conn(0x7fa61008c6d0 0x7fa61008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa60800b2d0 tx=0x7fa60800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.271+0000 7fa61521f700 1 --2- 172.21.15.16:0/1366231803 >> v2:172.21.15.16:3300/0 conn(0x7fa61008c6d0 0x7fa61008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa60800b2d0 tx=0x7fa60800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.272+0000 7fa61521f700 1 --2- 172.21.15.16:0/1366231803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa61008c6d0 0x7fa61008cac0 secure :-1 s=READY pgs=954 cs=0 l=1 rx=0x7fa60800b2d0 tx=0x7fa60800ba60).ready entity=mon.0 client_cookie=f5be5dccf30ad277 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.272+0000 7fa61521f700 1 --2- 172.21.15.16:0/1366231803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa61008c6d0 0x7fa61008cac0 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7fa60800b2d0 tx=0x7fa60800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.272+0000 7fa60ffff700 1 -- 172.21.15.16:0/1366231803 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa608007590 con 0x7fa61008c6d0 2019-12-05T06:32:14.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.272+0000 7fa61521f700 1 --2- 172.21.15.16:0/1366231803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa61008c6d0 0x7fa61008cac0 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7fa60800b2d0 tx=0x7fa60800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.272+0000 7fa61521f700 1 --2- 172.21.15.16:0/1366231803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa61008c6d0 0x7fa61008cac0 secure :-1 s=THROTTLE_DONE pgs=954 cs=0 l=1 rx=0x7fa60800b2d0 tx=0x7fa60800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.272+0000 7fa60ffff700 1 -- 172.21.15.16:0/1366231803 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa608009d30 con 0x7fa61008c6d0 2019-12-05T06:32:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.272+0000 7fa60ffff700 1 -- 172.21.15.16:0/1366231803 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa60800a400 con 0x7fa61008c6d0 2019-12-05T06:32:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.272+0000 7fa616c82700 1 -- 172.21.15.16:0/1366231803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa61008c6d0 msgr2=0x7fa61008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.272+0000 7fa616c82700 1 --2- 172.21.15.16:0/1366231803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa61008c6d0 0x7fa61008cac0 secure :-1 s=READY pgs=954 cs=0 l=1 rx=0x7fa60800b2d0 tx=0x7fa60800ba60).stop 2019-12-05T06:32:14.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.273+0000 7fa616c82700 1 -- 172.21.15.16:0/1366231803 shutdown_connections 2019-12-05T06:32:14.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.273+0000 7fa616c82700 1 --2- 172.21.15.16:0/1366231803 >> v2:172.21.15.76:3300/0 conn(0x7fa61008f720 0x7fa61008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:32:14.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.273+0000 7fa616c82700 1 --2- 172.21.15.16:0/1366231803 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa61008c6d0 0x7fa61008cac0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:14.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.273+0000 7fa616c82700 1 -- 172.21.15.16:0/1366231803 shutdown_connections 2019-12-05T06:32:14.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.273+0000 7fa616c82700 1 -- 172.21.15.16:0/1366231803 wait complete. 2019-12-05T06:32:14.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.273+0000 7fa616c82700 1 -- 172.21.15.16:0/1366231803 >> 172.21.15.16:0/1366231803 conn(0x7fa610061940 msgr2=0x7fa610096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:32:14.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.275+0000 7fa616c82700 1 Processor -- start 2019-12-05T06:32:14.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.275+0000 7fa616c82700 1 -- start start 2019-12-05T06:32:14.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa616c82700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa610096e90 0x7fa61009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:14.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa616c82700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa610084b90 con 0x7fa610084f30 2019-12-05T06:32:14.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa616c82700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa610098790 con 0x7fa61009b920 2019-12-05T06:32:14.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa616c82700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa61009a450 con 0x7fa610096e90 2019-12-05T06:32:14.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa610096e90 0x7fa61009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:14.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa610096e90 0x7fa61009b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55616/0 (socket says 172.21.15.16:55616) 2019-12-05T06:32:14.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 -- 172.21.15.16:0/3075411761 learned_addr learned my addr 172.21.15.16:0/3075411761 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:32:14.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa614a1e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa610084f30 0x7fa610097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:40786/0 (socket says 172.21.15.16:40786) 2019-12-05T06:32:14.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 -- 172.21.15.16:0/3075411761 >> v1:172.21.15.76:6789/0 conn(0x7fa61009b920 legacy=0x7fa610097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:14.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 -- 172.21.15.16:0/3075411761 >> v1:172.21.15.16:6789/0 conn(0x7fa610084f30 legacy=0x7fa610097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 -- 172.21.15.16:0/3075411761 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa61009aae0 con 0x7fa610096e90 2019-12-05T06:32:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 --2- 172.21.15.16:0/3075411761 >> v2:172.21.15.16:3300/0 conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 --2- 172.21.15.16:0/3075411761 >> v2:172.21.15.16:3300/0 conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=READY pgs=956 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:14.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=THROTTLE_DONE pgs=956 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=THROTTLE_DONE pgs=956 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa61521f700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=THROTTLE_DONE pgs=956 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa60dffb700 1 -- 172.21.15.16:0/3075411761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa608000cc0 con 0x7fa610096e90 2019-12-05T06:32:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa60dffb700 1 -- 172.21.15.16:0/3075411761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa608009d00 con 0x7fa610096e90 2019-12-05T06:32:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa60dffb700 1 -- 172.21.15.16:0/3075411761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa60800a3d0 con 0x7fa610096e90 2019-12-05T06:32:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.277+0000 7fa616c82700 1 -- 172.21.15.16:0/3075411761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa61009aae0 con 0x7fa610096e90 2019-12-05T06:32:14.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.278+0000 7fa616c82700 1 -- 172.21.15.16:0/3075411761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa61009aae0 con 0x7fa610096e90 2019-12-05T06:32:14.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.278+0000 7fa61521f700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=THROTTLE_DONE pgs=956 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.279+0000 7fa60dffb700 1 -- 172.21.15.16:0/3075411761 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa608009ef0 con 0x7fa610096e90 2019-12-05T06:32:14.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.279+0000 7fa61521f700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=THROTTLE_DONE pgs=956 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.279+0000 7fa60dffb700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa5f0023630 0x7fa5f0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:32:14.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.279+0000 7fa60dffb700 1 -- 172.21.15.16:0/3075411761 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fa608046480 con 0x7fa610096e90 2019-12-05T06:32:14.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.279+0000 7fa614a1e700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa5f0023630 0x7fa5f0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:32:14.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.279+0000 7fa614a1e700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa5f0023630 0x7fa5f0025ac0 crc :-1 s=READY pgs=412 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:32:14.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.285+0000 7fa616c82700 1 -- 172.21.15.16:0/3075411761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa61009aae0 con 0x7fa610096e90 2019-12-05T06:32:14.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.289+0000 7fa61521f700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=THROTTLE_DONE pgs=956 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.289+0000 7fa60dffb700 1 -- 172.21.15.16:0/3075411761 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa608009900 con 0x7fa610096e90 2019-12-05T06:32:14.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.506+0000 7fa616c82700 1 -- 172.21.15.16:0/3075411761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa61009aae0 con 0x7fa610096e90 2019-12-05T06:32:14.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.507+0000 7fa61521f700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=THROTTLE_DONE pgs=956 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:32:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.507+0000 7fa60dffb700 1 -- 172.21.15.16:0/3075411761 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa608009d00 con 0x7fa610096e90 2019-12-05T06:32:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 122 2019-12-05T06:32:14.519 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:32:14.520 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":122}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:32:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.509+0000 7fa616c82700 1 -- 172.21.15.16:0/3075411761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa5f0023630 msgr2=0x7fa5f0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:14.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.509+0000 7fa616c82700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa5f0023630 0x7fa5f0025ac0 crc :-1 s=READY pgs=412 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.509+0000 7fa616c82700 1 -- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 msgr2=0x7fa61009b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:32:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.509+0000 7fa616c82700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 0x7fa61009b3e0 secure :-1 s=READY pgs=956 cs=0 l=1 rx=0x7fa6080107c0 tx=0x7fa608010b60).stop 2019-12-05T06:32:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.510+0000 7fa616c82700 1 -- 172.21.15.16:0/3075411761 shutdown_connections 2019-12-05T06:32:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.510+0000 7fa616c82700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa5f0023630 0x7fa5f0025ac0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:14.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.510+0000 7fa616c82700 1 --2- 172.21.15.16:0/3075411761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa610096e90 0x7fa61009b3e0 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:32:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.510+0000 7fa616c82700 1 -- 172.21.15.16:0/3075411761 shutdown_connections 2019-12-05T06:32:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.510+0000 7fa616c82700 1 -- 172.21.15.16:0/3075411761 wait complete. 2019-12-05T06:32:14.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:32:14.510+0000 7fa616c82700 1 -- 172.21.15.16:0/3075411761 >> 172.21.15.16:0/3075411761 conn(0x7fa6100714f0 msgr2=0x7fa61009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:32:14.527 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:32:14.527 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 57 secs before thrashing 2019-12-05T06:32:36.194 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:32:36.204 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:33:06.358 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:33:06.366 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:33:11.528 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:11.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.688+0000 7fb67fb77700 1 Processor -- start 2019-12-05T06:33:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.688+0000 7fb67fb77700 1 -- start start 2019-12-05T06:33:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.690+0000 7fb67fb77700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb67808c6d0 0x7fb67808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.690+0000 7fb67fb77700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb67808fb80 con 0x7fb67808fef0 2019-12-05T06:33:11.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.690+0000 7fb67fb77700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6780958e0 con 0x7fb67808f720 2019-12-05T06:33:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.690+0000 7fb67fb77700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67808df90 con 0x7fb67808c6d0 2019-12-05T06:33:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.690+0000 7fb67d913700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb67808fef0 0x7fb67808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41054/0 (socket says 172.21.15.16:41054) 2019-12-05T06:33:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.690+0000 7fb67d913700 1 -- 172.21.15.16:0/3142946686 learned_addr learned my addr 172.21.15.16:0/3142946686 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.690+0000 7fb67e114700 1 --2- 172.21.15.16:0/3142946686 >> v2:172.21.15.76:3300/0 conn(0x7fb67808c6d0 0x7fb67808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3913337 0 0) 0x7fb664001150 con 0x7fb67808fef0 2019-12-05T06:33:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb668002e80 con 0x7fb67808fef0 2019-12-05T06:33:11.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047203270 0 0) 0x7fb6700009b0 con 0x7fb67808f720 2019-12-05T06:33:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6680030b0 con 0x7fb67808f720 2019-12-05T06:33:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 296621313 0 0) 0x7fb664001130 con 0x7fb67808fef0 2019-12-05T06:33:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 >> v2:172.21.15.76:3300/0 conn(0x7fb67808c6d0 msgr2=0x7fb67808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 --2- 172.21.15.16:0/3142946686 >> v2:172.21.15.76:3300/0 conn(0x7fb67808c6d0 0x7fb67808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 >> v1:172.21.15.76:6789/0 conn(0x7fb67808f720 legacy=0x7fb6781533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67808e620 con 0x7fb67808fef0 2019-12-05T06:33:11.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fb6640016c0 con 0x7fb67808fef0 2019-12-05T06:33:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 >> v1:172.21.15.16:6789/0 conn(0x7fb67808fef0 legacy=0x7fb67808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb668003b80 0x7fb668005fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb6680066d0 0x7fb668008b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:11.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 0x7fb66804f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66804fff0 con 0x7fb66804d3a0 2019-12-05T06:33:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb668050220 con 0x7fb668003b80 2019-12-05T06:33:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb668050450 con 0x7fb6680066d0 2019-12-05T06:33:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67e915700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 0x7fb66804f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67e114700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb668003b80 0x7fb668005fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.691+0000 7fb67d913700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb6680066d0 0x7fb668008b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:11.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 -- 172.21.15.16:0/3142946686 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb6680066d0 msgr2=0x7fb668008b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb6680066d0 0x7fb668008b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 -- 172.21.15.16:0/3142946686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb668003b80 msgr2=0x7fb668005fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb668003b80 0x7fb668005fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 -- 172.21.15.16:0/3142946686 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb668050700 con 0x7fb66804d3a0 2019-12-05T06:33:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 0x7fb66804f870 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb67000af60 tx=0x7fb67000b260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 -- 172.21.15.16:0/3142946686 reap_dead start 2019-12-05T06:33:11.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 0x7fb66804f870 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb67000af60 tx=0x7fb67000b260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 0x7fb66804f870 secure :-1 s=READY pgs=961 cs=0 l=1 rx=0x7fb67000af60 tx=0x7fb67000b260).ready entity=mon.0 client_cookie=5b6389e4da576f89 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 0x7fb66804f870 secure :-1 s=THROTTLE_DONE pgs=961 cs=0 l=1 rx=0x7fb67000af60 tx=0x7fb67000b260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb67000a5f0 con 0x7fb66804d3a0 2019-12-05T06:33:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.692+0000 7fb67e915700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 0x7fb66804f870 secure :-1 s=THROTTLE_DONE pgs=961 cs=0 l=1 rx=0x7fb67000af60 tx=0x7fb67000b260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.693+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb670000940 con 0x7fb66804d3a0 2019-12-05T06:33:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.693+0000 7fb67e915700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 0x7fb66804f870 secure :-1 s=THROTTLE_DONE pgs=961 cs=0 l=1 rx=0x7fb67000af60 tx=0x7fb67000b260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.693+0000 7fb67d112700 1 -- 172.21.15.16:0/3142946686 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb67000a690 con 0x7fb66804d3a0 2019-12-05T06:33:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.693+0000 7fb67fb77700 1 -- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 msgr2=0x7fb66804f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.693+0000 7fb67fb77700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 0x7fb66804f870 secure :-1 s=READY pgs=961 cs=0 l=1 rx=0x7fb67000af60 tx=0x7fb67000b260).stop 2019-12-05T06:33:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.694+0000 7fb67fb77700 1 -- 172.21.15.16:0/3142946686 shutdown_connections 2019-12-05T06:33:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.694+0000 7fb67fb77700 1 --2- 172.21.15.16:0/3142946686 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb66804d3a0 0x7fb66804f870 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.694+0000 7fb67fb77700 1 -- 172.21.15.16:0/3142946686 shutdown_connections 2019-12-05T06:33:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.694+0000 7fb67fb77700 1 -- 172.21.15.16:0/3142946686 wait complete. 2019-12-05T06:33:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.694+0000 7fb67fb77700 1 -- 172.21.15.16:0/3142946686 >> 172.21.15.16:0/3142946686 conn(0x7fb678061940 msgr2=0x7fb678096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.698+0000 7fb67fb77700 1 Processor -- start 2019-12-05T06:33:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.698+0000 7fb67fb77700 1 -- start start 2019-12-05T06:33:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.699+0000 7fb67fb77700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb6781325e0 0x7fb67812b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.699+0000 7fb67fb77700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67812c740 0x7fb67812bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.699+0000 7fb67fb77700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb678128b20 con 0x7fb67812c1f0 2019-12-05T06:33:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.699+0000 7fb67fb77700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb67812a7e0 con 0x7fb6781325e0 2019-12-05T06:33:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.699+0000 7fb67fb77700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb678129db0 con 0x7fb67812c740 2019-12-05T06:33:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.699+0000 7fb67e114700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb6781325e0 0x7fb67812b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.699+0000 7fb67e114700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb6781325e0 0x7fb67812b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55898/0 (socket says 172.21.15.16:55898) 2019-12-05T06:33:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.699+0000 7fb67e114700 1 -- 172.21.15.16:0/220940682 learned_addr learned my addr 172.21.15.16:0/220940682 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.699+0000 7fb67d913700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67812c740 0x7fb67812bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67e114700 1 -- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67812c740 msgr2=0x7fb67812bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67812c740 0x7fb67812bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67e114700 1 -- 172.21.15.16:0/220940682 >> v1:172.21.15.76:6789/0 conn(0x7fb67812c1f0 legacy=0x7fb67806dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67e114700 1 -- 172.21.15.16:0/220940682 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb67814a820 con 0x7fb6781325e0 2019-12-05T06:33:11.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67d913700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67812c740 0x7fb67812bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:33:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> v2:172.21.15.16:3300/0 conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> v2:172.21.15.16:3300/0 conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=READY pgs=962 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=THROTTLE_DONE pgs=962 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=THROTTLE_DONE pgs=962 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb676ffd700 1 -- 172.21.15.16:0/220940682 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb66c000cc0 con 0x7fb6781325e0 2019-12-05T06:33:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=THROTTLE_DONE pgs=962 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb676ffd700 1 -- 172.21.15.16:0/220940682 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb66c005900 con 0x7fb6781325e0 2019-12-05T06:33:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb676ffd700 1 -- 172.21.15.16:0/220940682 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb66c005f50 con 0x7fb6781325e0 2019-12-05T06:33:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67fb77700 1 -- 172.21.15.16:0/220940682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67812ae70 con 0x7fb6781325e0 2019-12-05T06:33:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.700+0000 7fb67fb77700 1 -- 172.21.15.16:0/220940682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67814a820 con 0x7fb6781325e0 2019-12-05T06:33:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.701+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=THROTTLE_DONE pgs=962 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.702+0000 7fb676ffd700 1 -- 172.21.15.16:0/220940682 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb66c005500 con 0x7fb6781325e0 2019-12-05T06:33:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.702+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=THROTTLE_DONE pgs=962 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.702+0000 7fb676ffd700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb660023630 0x7fb660025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.702+0000 7fb676ffd700 1 -- 172.21.15.16:0/220940682 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fb66c0461a0 con 0x7fb6781325e0 2019-12-05T06:33:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.702+0000 7fb67d913700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb660023630 0x7fb660025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.702+0000 7fb67d913700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb660023630 0x7fb660025ac0 crc :-1 s=READY pgs=415 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.704+0000 7fb67fb77700 1 -- 172.21.15.16:0/220940682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67812ae70 con 0x7fb6781325e0 2019-12-05T06:33:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.708+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=THROTTLE_DONE pgs=962 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.708+0000 7fb676ffd700 1 -- 172.21.15.16:0/220940682 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb66c006420 con 0x7fb6781325e0 2019-12-05T06:33:11.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.925+0000 7fb67fb77700 1 -- 172.21.15.16:0/220940682 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb67814a820 con 0x7fb6781325e0 2019-12-05T06:33:11.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.926+0000 7fb67e114700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=THROTTLE_DONE pgs=962 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:11.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.926+0000 7fb676ffd700 1 -- 172.21.15.16:0/220940682 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb66c005500 con 0x7fb6781325e0 2019-12-05T06:33:11.925 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:11.926 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":122}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 122 2019-12-05T06:33:11.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.930+0000 7fb67fb77700 1 -- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb660023630 msgr2=0x7fb660025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:11.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.930+0000 7fb67fb77700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb660023630 0x7fb660025ac0 crc :-1 s=READY pgs=415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:11.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.930+0000 7fb67fb77700 1 -- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 msgr2=0x7fb67812b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:11.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.930+0000 7fb67fb77700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 0x7fb67812b770 secure :-1 s=READY pgs=962 cs=0 l=1 rx=0x7fb66c0107d0 tx=0x7fb66c010b70).stop 2019-12-05T06:33:11.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.931+0000 7fb67fb77700 1 -- 172.21.15.16:0/220940682 shutdown_connections 2019-12-05T06:33:11.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.931+0000 7fb67fb77700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb67812c740 0x7fb67812bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:11.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.931+0000 7fb67fb77700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb660023630 0x7fb660025ac0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:11.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.931+0000 7fb67fb77700 1 --2- 172.21.15.16:0/220940682 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb6781325e0 0x7fb67812b770 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:11.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.931+0000 7fb67fb77700 1 -- 172.21.15.16:0/220940682 shutdown_connections 2019-12-05T06:33:11.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.931+0000 7fb67fb77700 1 -- 172.21.15.16:0/220940682 wait complete. 2019-12-05T06:33:11.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:11.931+0000 7fb67fb77700 1 -- 172.21.15.16:0/220940682 >> 172.21.15.16:0/220940682 conn(0x7fb6780a2850 msgr2=0x7fb67806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:11.939 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:33:11.940 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:33:11.940 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:33:11.952 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:33:11.953 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:12.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.255+0000 7f357863e700 1 Processor -- start 2019-12-05T06:33:12.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.255+0000 7f357863e700 1 -- start start 2019-12-05T06:33:12.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.258+0000 7f357863e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f357008c6d0 0x7f357008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:12.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.258+0000 7f357863e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f357008fef0 0x7f357008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:12.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.258+0000 7f357863e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35700958e0 con 0x7f357008ef20 2019-12-05T06:33:12.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.258+0000 7f357863e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35700a3f70 con 0x7f357008fef0 2019-12-05T06:33:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.258+0000 7f357863e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f357008d830 con 0x7f357008c6d0 2019-12-05T06:33:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.258+0000 7f35763da700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f357008fef0 0x7f357008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.258+0000 7f35763da700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f357008fef0 0x7f357008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55914/0 (socket says 172.21.15.16:55914) 2019-12-05T06:33:12.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.258+0000 7f35773dc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f357008ef20 0x7f357008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41080/0 (socket says 172.21.15.16:41080) 2019-12-05T06:33:12.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.258+0000 7f35763da700 1 -- 172.21.15.16:0/2081098010 learned_addr learned my addr 172.21.15.16:0/2081098010 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.258+0000 7f3576bdb700 1 --2- 172.21.15.16:0/2081098010 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f357008c6d0 0x7f357008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.259+0000 7f35763da700 1 -- 172.21.15.16:0/2081098010 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f357008c6d0 msgr2=0x7f357008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.259+0000 7f35763da700 1 --2- 172.21.15.16:0/2081098010 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f357008c6d0 0x7f357008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.259+0000 7f35763da700 1 -- 172.21.15.16:0/2081098010 >> v1:172.21.15.16:6789/0 conn(0x7f357008ef20 legacy=0x7f357008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:12.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.259+0000 7f3575bd9700 1 -- 172.21.15.16:0/2081098010 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882617010 0 0) 0x7f35680009b0 con 0x7f357008ef20 2019-12-05T06:33:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.259+0000 7f35763da700 1 -- 172.21.15.16:0/2081098010 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357008d540 con 0x7f357008fef0 2019-12-05T06:33:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.259+0000 7f35763da700 1 --2- 172.21.15.16:0/2081098010 >> v2:172.21.15.16:3300/0 conn(0x7f357008fef0 0x7f357008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f356400f740 tx=0x7f356400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.259+0000 7f35763da700 1 --2- 172.21.15.16:0/2081098010 >> v2:172.21.15.16:3300/0 conn(0x7f357008fef0 0x7f357008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f356400f740 tx=0x7f356400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.259+0000 7f35763da700 1 --2- 172.21.15.16:0/2081098010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f357008fef0 0x7f357008d000 secure :-1 s=READY pgs=964 cs=0 l=1 rx=0x7f356400f740 tx=0x7f356400f570).ready entity=mon.0 client_cookie=72986a419e6ff939 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.260+0000 7f35763da700 1 --2- 172.21.15.16:0/2081098010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f357008fef0 0x7f357008d000 secure :-1 s=THROTTLE_DONE pgs=964 cs=0 l=1 rx=0x7f356400f740 tx=0x7f356400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.260+0000 7f3575bd9700 1 -- 172.21.15.16:0/2081098010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3564000960 con 0x7f357008fef0 2019-12-05T06:33:12.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.260+0000 7f35763da700 1 --2- 172.21.15.16:0/2081098010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f357008fef0 0x7f357008d000 secure :-1 s=THROTTLE_DONE pgs=964 cs=0 l=1 rx=0x7f356400f740 tx=0x7f356400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.260+0000 7f35763da700 1 --2- 172.21.15.16:0/2081098010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f357008fef0 0x7f357008d000 secure :-1 s=THROTTLE_DONE pgs=964 cs=0 l=1 rx=0x7f356400f740 tx=0x7f356400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.260+0000 7f3575bd9700 1 -- 172.21.15.16:0/2081098010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3564000c90 con 0x7f357008fef0 2019-12-05T06:33:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.260+0000 7f3575bd9700 1 -- 172.21.15.16:0/2081098010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35640013a0 con 0x7f357008fef0 2019-12-05T06:33:12.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.260+0000 7f357863e700 1 -- 172.21.15.16:0/2081098010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f357008fef0 msgr2=0x7f357008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.260+0000 7f357863e700 1 --2- 172.21.15.16:0/2081098010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f357008fef0 0x7f357008d000 secure :-1 s=READY pgs=964 cs=0 l=1 rx=0x7f356400f740 tx=0x7f356400f570).stop 2019-12-05T06:33:12.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.262+0000 7f357863e700 1 -- 172.21.15.16:0/2081098010 shutdown_connections 2019-12-05T06:33:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.262+0000 7f357863e700 1 --2- 172.21.15.16:0/2081098010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f357008fef0 0x7f357008d000 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.262+0000 7f357863e700 1 --2- 172.21.15.16:0/2081098010 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f357008c6d0 0x7f357008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.262+0000 7f357863e700 1 -- 172.21.15.16:0/2081098010 shutdown_connections 2019-12-05T06:33:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.262+0000 7f357863e700 1 -- 172.21.15.16:0/2081098010 wait complete. 2019-12-05T06:33:12.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.262+0000 7f357863e700 1 -- 172.21.15.16:0/2081098010 >> 172.21.15.16:0/2081098010 conn(0x7f3570061940 msgr2=0x7f3570096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:12.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.268+0000 7f357863e700 1 Processor -- start 2019-12-05T06:33:12.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.268+0000 7f357863e700 1 -- start start 2019-12-05T06:33:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.269+0000 7f357863e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f357014d400 0x7f357014d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.269+0000 7f357863e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3570098da0 con 0x7f3570132530 2019-12-05T06:33:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.270+0000 7f357863e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f357009f0a0 con 0x7f3570132d00 2019-12-05T06:33:12.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.270+0000 7f357863e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f357009bb50 con 0x7f357014d400 2019-12-05T06:33:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.270+0000 7f35773dc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3570132530 0x7f3570157100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41088/0 (socket says 172.21.15.16:41088) 2019-12-05T06:33:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.270+0000 7f35773dc700 1 -- 172.21.15.16:0/1786406928 learned_addr learned my addr 172.21.15.16:0/1786406928 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.270+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> v2:172.21.15.76:3300/0 conn(0x7f357014d400 0x7f357014d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:12.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.270+0000 7f356f7fe700 1 -- 172.21.15.16:0/1786406928 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1820716382 0 0) 0x7f3568001310 con 0x7f3570132530 2019-12-05T06:33:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.270+0000 7f356f7fe700 1 -- 172.21.15.16:0/1786406928 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3550002e80 con 0x7f3570132530 2019-12-05T06:33:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f3576bdb700 1 -- 172.21.15.16:0/1786406928 >> v1:172.21.15.76:6789/0 conn(0x7f3570132d00 legacy=0x7f3570131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f356f7fe700 1 -- 172.21.15.16:0/1786406928 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891022753 0 0) 0x7f3564008140 con 0x7f3570132d00 2019-12-05T06:33:12.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f3576bdb700 1 -- 172.21.15.16:0/1786406928 >> v1:172.21.15.16:6789/0 conn(0x7f3570132530 legacy=0x7f3570157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f3576bdb700 1 -- 172.21.15.16:0/1786406928 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f357009b0f0 con 0x7f357014d400 2019-12-05T06:33:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> v2:172.21.15.76:3300/0 conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> v2:172.21.15.76:3300/0 conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=READY pgs=690 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:12.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=THROTTLE_DONE pgs=690 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f356f7fe700 1 -- 172.21.15.16:0/1786406928 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3560000cc0 con 0x7f357014d400 2019-12-05T06:33:12.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=THROTTLE_DONE pgs=690 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=THROTTLE_DONE pgs=690 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f356f7fe700 1 -- 172.21.15.16:0/1786406928 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3560005900 con 0x7f357014d400 2019-12-05T06:33:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f356f7fe700 1 -- 172.21.15.16:0/1786406928 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3560005fd0 con 0x7f357014d400 2019-12-05T06:33:12.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.271+0000 7f357863e700 1 -- 172.21.15.16:0/1786406928 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f357009b0f0 con 0x7f357014d400 2019-12-05T06:33:12.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.272+0000 7f357863e700 1 -- 172.21.15.16:0/1786406928 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f357009b0f0 con 0x7f357014d400 2019-12-05T06:33:12.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.273+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=THROTTLE_DONE pgs=690 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.273+0000 7f356f7fe700 1 -- 172.21.15.16:0/1786406928 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f35600055a0 con 0x7f357014d400 2019-12-05T06:33:12.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.273+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=THROTTLE_DONE pgs=690 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.274+0000 7f356f7fe700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3550023680 0x7f3550025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:12.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.274+0000 7f356f7fe700 1 -- 172.21.15.16:0/1786406928 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f3560046180 con 0x7f357014d400 2019-12-05T06:33:12.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.274+0000 7f35763da700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3550023680 0x7f3550025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:12.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.275+0000 7f35763da700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3550023680 0x7f3550025af0 crc :-1 s=READY pgs=416 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:12.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.287+0000 7f357863e700 1 -- 172.21.15.16:0/1786406928 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f357009b0f0 con 0x7f357014d400 2019-12-05T06:33:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.294+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=THROTTLE_DONE pgs=690 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.294+0000 7f356f7fe700 1 -- 172.21.15.16:0/1786406928 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3560000cc0 con 0x7f357014d400 2019-12-05T06:33:12.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.547+0000 7f357863e700 1 -- 172.21.15.16:0/1786406928 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f357009b0f0 con 0x7f357014d400 2019-12-05T06:33:12.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.549+0000 7f3576bdb700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=THROTTLE_DONE pgs=690 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:12.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.549+0000 7f356f7fe700 1 -- 172.21.15.16:0/1786406928 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3560005500 con 0x7f357014d400 2019-12-05T06:33:12.548 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 122 2019-12-05T06:33:12.548 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:12.548 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":122}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.554+0000 7f357863e700 1 -- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3550023680 msgr2=0x7f3550025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.554+0000 7f357863e700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3550023680 0x7f3550025af0 crc :-1 s=READY pgs=416 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:12.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.554+0000 7f357863e700 1 -- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 msgr2=0x7f357014d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.554+0000 7f357863e700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 0x7f357014d7f0 secure :-1 s=READY pgs=690 cs=0 l=1 rx=0x7f3560010730 tx=0x7f3560010ad0).stop 2019-12-05T06:33:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.554+0000 7f357863e700 1 -- 172.21.15.16:0/1786406928 shutdown_connections 2019-12-05T06:33:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.554+0000 7f357863e700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3550023680 0x7f3550025af0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.554+0000 7f357863e700 1 --2- 172.21.15.16:0/1786406928 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f357014d400 0x7f357014d7f0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.555+0000 7f357863e700 1 -- 172.21.15.16:0/1786406928 shutdown_connections 2019-12-05T06:33:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.555+0000 7f357863e700 1 -- 172.21.15.16:0/1786406928 wait complete. 2019-12-05T06:33:12.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:12.555+0000 7f357863e700 1 -- 172.21.15.16:0/1786406928 >> 172.21.15.16:0/1786406928 conn(0x7f35700714f0 msgr2=0x7f357009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:12.569 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:33:14.572 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:14.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.850+0000 7f693fb7a700 1 Processor -- start 2019-12-05T06:33:14.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.851+0000 7f693fb7a700 1 -- start start 2019-12-05T06:33:14.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693fb7a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f693808c6d0 0x7f693808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:14.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693fb7a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f693808fef0 0x7f693808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:14.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693fb7a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f693808ef20 0x7f693808f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:14.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693fb7a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69380a3f70 con 0x7f693808ef20 2019-12-05T06:33:14.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693fb7a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f693808d830 con 0x7f693808fef0 2019-12-05T06:33:14.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693fb7a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f693808d540 con 0x7f693808c6d0 2019-12-05T06:33:14.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693e918700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f693808ef20 0x7f693808f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:14.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693e918700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f693808ef20 0x7f693808f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55930/0 (socket says 172.21.15.16:55930) 2019-12-05T06:33:14.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693e918700 1 -- 172.21.15.16:0/2264130301 learned_addr learned my addr 172.21.15.16:0/2264130301 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693d916700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f693808fef0 0x7f693808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.853+0000 7f693e117700 1 --2- 172.21.15.16:0/2264130301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f693808c6d0 0x7f693808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.854+0000 7f693e918700 1 -- 172.21.15.16:0/2264130301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f693808c6d0 msgr2=0x7f693808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:14.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.854+0000 7f693e918700 1 --2- 172.21.15.16:0/2264130301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f693808c6d0 0x7f693808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.854+0000 7f693e918700 1 -- 172.21.15.16:0/2264130301 >> v2:172.21.15.76:3300/0 conn(0x7f693808fef0 msgr2=0x7f693808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.854+0000 7f693e918700 1 --2- 172.21.15.16:0/2264130301 >> v2:172.21.15.76:3300/0 conn(0x7f693808fef0 0x7f693808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.854+0000 7f693e918700 1 -- 172.21.15.16:0/2264130301 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693808e620 con 0x7f693808ef20 2019-12-05T06:33:14.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.854+0000 7f693d916700 1 --2- 172.21.15.16:0/2264130301 >> v2:172.21.15.76:3300/0 conn(0x7f693808fef0 0x7f693808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.854+0000 7f693e918700 1 --2- 172.21.15.16:0/2264130301 >> v2:172.21.15.16:3300/0 conn(0x7f693808ef20 0x7f693808f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f692800f730 tx=0x7f6928007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.854+0000 7f693e918700 1 --2- 172.21.15.16:0/2264130301 >> v2:172.21.15.16:3300/0 conn(0x7f693808ef20 0x7f693808f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f692800f730 tx=0x7f6928007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.854+0000 7f693e918700 1 --2- 172.21.15.16:0/2264130301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693808ef20 0x7f693808f3c0 secure :-1 s=READY pgs=966 cs=0 l=1 rx=0x7f692800f730 tx=0x7f6928007d50).ready entity=mon.0 client_cookie=a958bf365d82e427 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.855+0000 7f693e918700 1 --2- 172.21.15.16:0/2264130301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693808ef20 0x7f693808f3c0 secure :-1 s=THROTTLE_DONE pgs=966 cs=0 l=1 rx=0x7f692800f730 tx=0x7f6928007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.855+0000 7f693e918700 1 --2- 172.21.15.16:0/2264130301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693808ef20 0x7f693808f3c0 secure :-1 s=THROTTLE_DONE pgs=966 cs=0 l=1 rx=0x7f692800f730 tx=0x7f6928007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.855+0000 7f693d115700 1 -- 172.21.15.16:0/2264130301 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6928000b90 con 0x7f693808ef20 2019-12-05T06:33:14.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.855+0000 7f693e918700 1 --2- 172.21.15.16:0/2264130301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693808ef20 0x7f693808f3c0 secure :-1 s=THROTTLE_DONE pgs=966 cs=0 l=1 rx=0x7f692800f730 tx=0x7f6928007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.855+0000 7f693d115700 1 -- 172.21.15.16:0/2264130301 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69280010e0 con 0x7f693808ef20 2019-12-05T06:33:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.855+0000 7f693d115700 1 -- 172.21.15.16:0/2264130301 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f69280017b0 con 0x7f693808ef20 2019-12-05T06:33:14.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.855+0000 7f693fb7a700 1 -- 172.21.15.16:0/2264130301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693808ef20 msgr2=0x7f693808f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.855+0000 7f693fb7a700 1 --2- 172.21.15.16:0/2264130301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693808ef20 0x7f693808f3c0 secure :-1 s=READY pgs=966 cs=0 l=1 rx=0x7f692800f730 tx=0x7f6928007d50).stop 2019-12-05T06:33:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.857+0000 7f693fb7a700 1 -- 172.21.15.16:0/2264130301 shutdown_connections 2019-12-05T06:33:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.857+0000 7f693fb7a700 1 --2- 172.21.15.16:0/2264130301 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f693808ef20 0x7f693808f3c0 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.857+0000 7f693fb7a700 1 --2- 172.21.15.16:0/2264130301 >> v2:172.21.15.76:3300/0 conn(0x7f693808fef0 0x7f693808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:14.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.857+0000 7f693fb7a700 1 --2- 172.21.15.16:0/2264130301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f693808c6d0 0x7f693808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.857+0000 7f693fb7a700 1 -- 172.21.15.16:0/2264130301 shutdown_connections 2019-12-05T06:33:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.857+0000 7f693fb7a700 1 -- 172.21.15.16:0/2264130301 wait complete. 2019-12-05T06:33:14.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.857+0000 7f693fb7a700 1 -- 172.21.15.16:0/2264130301 >> 172.21.15.16:0/2264130301 conn(0x7f6938061940 msgr2=0x7f6938096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.863+0000 7f693fb7a700 1 Processor -- start 2019-12-05T06:33:14.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.863+0000 7f693fb7a700 1 -- start start 2019-12-05T06:33:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.865+0000 7f693fb7a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6938084f30 0x7f6938157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.865+0000 7f693fb7a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6938084b90 con 0x7f6938096e90 2019-12-05T06:33:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.865+0000 7f693fb7a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6938098790 con 0x7f693809bbe0 2019-12-05T06:33:14.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.865+0000 7f693fb7a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f693809a450 con 0x7f6938084f30 2019-12-05T06:33:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.865+0000 7f693e117700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6938096e90 0x7f693809b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41102/0 (socket says 172.21.15.16:41102) 2019-12-05T06:33:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.865+0000 7f693e918700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6938084f30 0x7f6938157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.865+0000 7f693e117700 1 -- 172.21.15.16:0/2292292597 learned_addr learned my addr 172.21.15.16:0/2292292597 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:14.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.866+0000 7f6936ffd700 1 -- 172.21.15.16:0/2292292597 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073916068 0 0) 0x7f692c0010b0 con 0x7f693809bbe0 2019-12-05T06:33:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.866+0000 7f6936ffd700 1 -- 172.21.15.16:0/2292292597 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6918002e80 con 0x7f693809bbe0 2019-12-05T06:33:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.866+0000 7f693e918700 1 -- 172.21.15.16:0/2292292597 >> v1:172.21.15.76:6789/0 conn(0x7f693809bbe0 legacy=0x7f69381539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.866+0000 7f693e918700 1 -- 172.21.15.16:0/2292292597 >> v1:172.21.15.16:6789/0 conn(0x7f6938096e90 legacy=0x7f693809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:14.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.866+0000 7f6936ffd700 1 -- 172.21.15.16:0/2292292597 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292403049 0 0) 0x7f6930001430 con 0x7f6938096e90 2019-12-05T06:33:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.866+0000 7f693e918700 1 -- 172.21.15.16:0/2292292597 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f693809aae0 con 0x7f6938084f30 2019-12-05T06:33:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.866+0000 7f693e918700 1 --2- 172.21.15.16:0/2292292597 >> v2:172.21.15.16:3300/0 conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.866+0000 7f693e918700 1 --2- 172.21.15.16:0/2292292597 >> v2:172.21.15.16:3300/0 conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.866+0000 7f693e918700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=READY pgs=968 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.867+0000 7f693e918700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.867+0000 7f6936ffd700 1 -- 172.21.15.16:0/2292292597 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6928008180 con 0x7f6938084f30 2019-12-05T06:33:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.867+0000 7f693e918700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.867+0000 7f693e918700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.880 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.867+0000 7f6936ffd700 1 -- 172.21.15.16:0/2292292597 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69280010a0 con 0x7f6938084f30 2019-12-05T06:33:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.867+0000 7f6936ffd700 1 -- 172.21.15.16:0/2292292597 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6928008180 con 0x7f6938084f30 2019-12-05T06:33:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.867+0000 7f693fb7a700 1 -- 172.21.15.16:0/2292292597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693809aae0 con 0x7f6938084f30 2019-12-05T06:33:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.867+0000 7f693fb7a700 1 -- 172.21.15.16:0/2292292597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693809aae0 con 0x7f6938084f30 2019-12-05T06:33:14.881 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.868+0000 7f693e918700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.869+0000 7f6936ffd700 1 -- 172.21.15.16:0/2292292597 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6928001290 con 0x7f6938084f30 2019-12-05T06:33:14.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.869+0000 7f693e918700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.869+0000 7f6936ffd700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6918023680 0x7f6918025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:14.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.869+0000 7f6936ffd700 1 -- 172.21.15.16:0/2292292597 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f6928089d70 con 0x7f6938084f30 2019-12-05T06:33:14.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.869+0000 7f693e117700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6918023680 0x7f6918025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:14.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.870+0000 7f693e117700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6918023680 0x7f6918025af0 crc :-1 s=READY pgs=417 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:14.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.882+0000 7f693fb7a700 1 -- 172.21.15.16:0/2292292597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693809aae0 con 0x7f6938084f30 2019-12-05T06:33:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.890+0000 7f693e918700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:14.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:14.891+0000 7f6936ffd700 1 -- 172.21.15.16:0/2292292597 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f69280076a0 con 0x7f6938084f30 2019-12-05T06:33:15.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.153+0000 7f693fb7a700 1 -- 172.21.15.16:0/2292292597 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f693809aae0 con 0x7f6938084f30 2019-12-05T06:33:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.154+0000 7f693e918700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=THROTTLE_DONE pgs=968 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.154+0000 7f6936ffd700 1 -- 172.21.15.16:0/2292292597 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f6928001260 con 0x7f6938084f30 2019-12-05T06:33:15.151 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 122 2019-12-05T06:33:15.152 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:15.152 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":122}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:15.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.156+0000 7f693fb7a700 1 -- 172.21.15.16:0/2292292597 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6918023680 msgr2=0x7f6918025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:15.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.156+0000 7f693fb7a700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6918023680 0x7f6918025af0 crc :-1 s=READY pgs=417 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:15.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.156+0000 7f693fb7a700 1 -- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 msgr2=0x7f6938157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:15.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.156+0000 7f693fb7a700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 0x7f6938157200 secure :-1 s=READY pgs=968 cs=0 l=1 rx=0x7f69280543b0 tx=0x7f6928054750).stop 2019-12-05T06:33:15.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.157+0000 7f693fb7a700 1 -- 172.21.15.16:0/2292292597 shutdown_connections 2019-12-05T06:33:15.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.157+0000 7f693fb7a700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6918023680 0x7f6918025af0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:15.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.157+0000 7f693fb7a700 1 --2- 172.21.15.16:0/2292292597 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6938084f30 0x7f6938157200 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:15.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.157+0000 7f693fb7a700 1 -- 172.21.15.16:0/2292292597 shutdown_connections 2019-12-05T06:33:15.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.157+0000 7f693fb7a700 1 -- 172.21.15.16:0/2292292597 wait complete. 2019-12-05T06:33:15.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:15.157+0000 7f693fb7a700 1 -- 172.21.15.16:0/2292292597 >> 172.21.15.16:0/2292292597 conn(0x7f69380714f0 msgr2=0x7f693809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:15.168 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:33:17.169 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:17.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.420+0000 7fa474857700 1 Processor -- start 2019-12-05T06:33:17.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.420+0000 7fa474857700 1 -- start start 2019-12-05T06:33:17.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.422+0000 7fa474857700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 0x7fa47008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.422+0000 7fa474857700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4700958e0 con 0x7fa47008f720 2019-12-05T06:33:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.422+0000 7fa474857700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4700629e0 con 0x7fa47008fef0 2019-12-05T06:33:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.422+0000 7fa474857700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4700a3f70 con 0x7fa47008c6d0 2019-12-05T06:33:17.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.422+0000 7fa46f59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa47008f720 0x7fa4701533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41114/0 (socket says 172.21.15.16:41114) 2019-12-05T06:33:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.422+0000 7fa46f59e700 1 -- 172.21.15.16:0/2590970381 learned_addr learned my addr 172.21.15.16:0/2590970381 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.422+0000 7fa46e59c700 1 --1- 172.21.15.16:0/2590970381 >> v1:172.21.15.76:6789/0 conn(0x7fa47008fef0 0x7fa47008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:34186/0 (socket says 172.21.15.16:34186) 2019-12-05T06:33:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.422+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/2590970381 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 0x7fa47008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:17.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.423+0000 7fa46dd9b700 1 -- 172.21.15.16:0/2590970381 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1558236029 0 0) 0x7fa458001080 con 0x7fa47008fef0 2019-12-05T06:33:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.423+0000 7fa46dd9b700 1 -- 172.21.15.16:0/2590970381 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa45c002e80 con 0x7fa47008fef0 2019-12-05T06:33:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.423+0000 7fa46dd9b700 1 -- 172.21.15.16:0/2590970381 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26636695 0 0) 0x7fa4680014d0 con 0x7fa47008f720 2019-12-05T06:33:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.423+0000 7fa46ed9d700 1 -- 172.21.15.16:0/2590970381 >> v1:172.21.15.76:6789/0 conn(0x7fa47008fef0 legacy=0x7fa47008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.423+0000 7fa46ed9d700 1 -- 172.21.15.16:0/2590970381 >> v1:172.21.15.16:6789/0 conn(0x7fa47008f720 legacy=0x7fa4701533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.423+0000 7fa46ed9d700 1 -- 172.21.15.16:0/2590970381 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47008fbc0 con 0x7fa47008c6d0 2019-12-05T06:33:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.423+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/2590970381 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 0x7fa47008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa46000af50 tx=0x7fa46000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.423+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/2590970381 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 0x7fa47008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa46000af50 tx=0x7fa46000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.423+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/2590970381 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 0x7fa47008cac0 secure :-1 s=READY pgs=230 cs=0 l=1 rx=0x7fa46000af50 tx=0x7fa46000b760).ready entity=mon.2 client_cookie=a7dd3de7bca414cb server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.423+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/2590970381 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 0x7fa47008cac0 secure :-1 s=THROTTLE_DONE pgs=230 cs=0 l=1 rx=0x7fa46000af50 tx=0x7fa46000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.424+0000 7fa46dd9b700 1 -- 172.21.15.16:0/2590970381 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa460000960 con 0x7fa47008c6d0 2019-12-05T06:33:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.424+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/2590970381 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 0x7fa47008cac0 secure :-1 s=THROTTLE_DONE pgs=230 cs=0 l=1 rx=0x7fa46000af50 tx=0x7fa46000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.424+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/2590970381 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 0x7fa47008cac0 secure :-1 s=THROTTLE_DONE pgs=230 cs=0 l=1 rx=0x7fa46000af50 tx=0x7fa46000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.424+0000 7fa46dd9b700 1 -- 172.21.15.16:0/2590970381 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa460009860 con 0x7fa47008c6d0 2019-12-05T06:33:17.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.424+0000 7fa46dd9b700 1 -- 172.21.15.16:0/2590970381 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa460009f30 con 0x7fa47008c6d0 2019-12-05T06:33:17.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.424+0000 7fa474857700 1 -- 172.21.15.16:0/2590970381 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 msgr2=0x7fa47008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.424+0000 7fa474857700 1 --2- 172.21.15.16:0/2590970381 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 0x7fa47008cac0 secure :-1 s=READY pgs=230 cs=0 l=1 rx=0x7fa46000af50 tx=0x7fa46000b760).stop 2019-12-05T06:33:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.425+0000 7fa474857700 1 -- 172.21.15.16:0/2590970381 shutdown_connections 2019-12-05T06:33:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.425+0000 7fa474857700 1 --2- 172.21.15.16:0/2590970381 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47008c6d0 0x7fa47008cac0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.425+0000 7fa474857700 1 -- 172.21.15.16:0/2590970381 shutdown_connections 2019-12-05T06:33:17.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.425+0000 7fa474857700 1 -- 172.21.15.16:0/2590970381 wait complete. 2019-12-05T06:33:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.425+0000 7fa474857700 1 -- 172.21.15.16:0/2590970381 >> 172.21.15.16:0/2590970381 conn(0x7fa470061940 msgr2=0x7fa470096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.429+0000 7fa474857700 1 Processor -- start 2019-12-05T06:33:17.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.429+0000 7fa474857700 1 -- start start 2019-12-05T06:33:17.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.430+0000 7fa474857700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa4701325e0 0x7fa47012b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa474857700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47012c740 0x7fa47012bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa474857700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa470128b20 con 0x7fa47012c1f0 2019-12-05T06:33:17.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa474857700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa47012a7e0 con 0x7fa4701325e0 2019-12-05T06:33:17.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa474857700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa470129db0 con 0x7fa47012c740 2019-12-05T06:33:17.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa46ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa4701325e0 0x7fa47012b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:17.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa46f59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa47012c1f0 0x7fa47006dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41122/0 (socket says 172.21.15.16:41122) 2019-12-05T06:33:17.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa46f59e700 1 -- 172.21.15.16:0/524105723 learned_addr learned my addr 172.21.15.16:0/524105723 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:17.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa46e59c700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47012c740 0x7fa47012bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:17.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa46ed9d700 1 -- 172.21.15.16:0/524105723 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47012c740 msgr2=0x7fa47012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47012c740 0x7fa47012bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa46ed9d700 1 -- 172.21.15.16:0/524105723 >> v1:172.21.15.16:6789/0 conn(0x7fa47012c1f0 legacy=0x7fa47006dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa46ed9d700 1 -- 172.21.15.16:0/524105723 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa47014a820 con 0x7fa4701325e0 2019-12-05T06:33:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa4677fe700 1 -- 172.21.15.16:0/524105723 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2182973436 0 0) 0x7fa4680012c0 con 0x7fa47012c1f0 2019-12-05T06:33:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> v2:172.21.15.16:3300/0 conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.431+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> v2:172.21.15.16:3300/0 conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.432+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=READY pgs=971 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.432+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=THROTTLE_DONE pgs=971 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.432+0000 7fa4677fe700 1 -- 172.21.15.16:0/524105723 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa460009790 con 0x7fa4701325e0 2019-12-05T06:33:17.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.432+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=THROTTLE_DONE pgs=971 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.432+0000 7fa4677fe700 1 -- 172.21.15.16:0/524105723 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46000b5d0 con 0x7fa4701325e0 2019-12-05T06:33:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.432+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=THROTTLE_DONE pgs=971 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.432+0000 7fa474857700 1 -- 172.21.15.16:0/524105723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa47012ae70 con 0x7fa4701325e0 2019-12-05T06:33:17.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.432+0000 7fa4677fe700 1 -- 172.21.15.16:0/524105723 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa460000940 con 0x7fa4701325e0 2019-12-05T06:33:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.432+0000 7fa474857700 1 -- 172.21.15.16:0/524105723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa47014a820 con 0x7fa4701325e0 2019-12-05T06:33:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.433+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=THROTTLE_DONE pgs=971 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.433+0000 7fa4677fe700 1 -- 172.21.15.16:0/524105723 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa460009580 con 0x7fa4701325e0 2019-12-05T06:33:17.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.433+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=THROTTLE_DONE pgs=971 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.433+0000 7fa4677fe700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa454023630 0x7fa454025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.433+0000 7fa4677fe700 1 -- 172.21.15.16:0/524105723 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fa460045d00 con 0x7fa4701325e0 2019-12-05T06:33:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.434+0000 7fa46e59c700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa454023630 0x7fa454025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:17.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.434+0000 7fa46e59c700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa454023630 0x7fa454025ac0 crc :-1 s=READY pgs=418 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:17.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.436+0000 7fa474857700 1 -- 172.21.15.16:0/524105723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa47012ae70 con 0x7fa4701325e0 2019-12-05T06:33:17.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.444+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=THROTTLE_DONE pgs=971 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.444+0000 7fa4677fe700 1 -- 172.21.15.16:0/524105723 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa460000940 con 0x7fa4701325e0 2019-12-05T06:33:17.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.700+0000 7fa474857700 1 -- 172.21.15.16:0/524105723 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa47014a820 con 0x7fa4701325e0 2019-12-05T06:33:17.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.701+0000 7fa46ed9d700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=THROTTLE_DONE pgs=971 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.701+0000 7fa4677fe700 1 -- 172.21.15.16:0/524105723 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa460009580 con 0x7fa4701325e0 2019-12-05T06:33:17.698 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 122 2019-12-05T06:33:17.698 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:17.698 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":122}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:17.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.703+0000 7fa474857700 1 -- 172.21.15.16:0/524105723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa454023630 msgr2=0x7fa454025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.703+0000 7fa474857700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa454023630 0x7fa454025ac0 crc :-1 s=READY pgs=418 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.703+0000 7fa474857700 1 -- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 msgr2=0x7fa47012b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.703+0000 7fa474857700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 0x7fa47012b770 secure :-1 s=READY pgs=971 cs=0 l=1 rx=0x7fa4600103d0 tx=0x7fa460010200).stop 2019-12-05T06:33:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.703+0000 7fa474857700 1 -- 172.21.15.16:0/524105723 shutdown_connections 2019-12-05T06:33:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.703+0000 7fa474857700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa47012c740 0x7fa47012bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.703+0000 7fa474857700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa454023630 0x7fa454025ac0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.703+0000 7fa474857700 1 --2- 172.21.15.16:0/524105723 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa4701325e0 0x7fa47012b770 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.704+0000 7fa474857700 1 -- 172.21.15.16:0/524105723 shutdown_connections 2019-12-05T06:33:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.704+0000 7fa474857700 1 -- 172.21.15.16:0/524105723 wait complete. 2019-12-05T06:33:17.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:17.704+0000 7fa474857700 1 -- 172.21.15.16:0/524105723 >> 172.21.15.16:0/524105723 conn(0x7fa4700a2850 msgr2=0x7fa47006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:17.711 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:33:19.712 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:19.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.978+0000 7fb49f308700 1 Processor -- start 2019-12-05T06:33:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.979+0000 7fb49f308700 1 -- start start 2019-12-05T06:33:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.980+0000 7fb49f308700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49808c6d0 0x7fb49808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.980+0000 7fb49f308700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb49808f720 0x7fb49808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:19.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.980+0000 7fb49f308700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4980958e0 con 0x7fb49808fef0 2019-12-05T06:33:19.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.980+0000 7fb49f308700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4980a3f70 con 0x7fb49808f720 2019-12-05T06:33:19.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.980+0000 7fb49f308700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb49808d830 con 0x7fb49808c6d0 2019-12-05T06:33:19.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.980+0000 7fb49e0a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb49808f720 0x7fb49808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:19.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.980+0000 7fb49e0a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb49808f720 0x7fb49808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55962/0 (socket says 172.21.15.16:55962) 2019-12-05T06:33:19.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.980+0000 7fb49e0a6700 1 -- 172.21.15.16:0/806237764 learned_addr learned my addr 172.21.15.16:0/806237764 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.980+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/806237764 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49808c6d0 0x7fb49808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.981+0000 7fb49e0a6700 1 -- 172.21.15.16:0/806237764 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49808c6d0 msgr2=0x7fb49808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.981+0000 7fb49e0a6700 1 --2- 172.21.15.16:0/806237764 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49808c6d0 0x7fb49808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:19.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.981+0000 7fb49e0a6700 1 -- 172.21.15.16:0/806237764 >> v1:172.21.15.16:6789/0 conn(0x7fb49808fef0 legacy=0x7fb49808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.981+0000 7fb49e0a6700 1 -- 172.21.15.16:0/806237764 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49808fbc0 con 0x7fb49808f720 2019-12-05T06:33:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.981+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/806237764 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49808c6d0 0x7fb49808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.981+0000 7fb49e0a6700 1 --2- 172.21.15.16:0/806237764 >> v2:172.21.15.16:3300/0 conn(0x7fb49808f720 0x7fb49808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb48800f730 tx=0x7fb48800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.982+0000 7fb49e0a6700 1 --2- 172.21.15.16:0/806237764 >> v2:172.21.15.16:3300/0 conn(0x7fb49808f720 0x7fb49808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb48800f730 tx=0x7fb48800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.982+0000 7fb49e0a6700 1 --2- 172.21.15.16:0/806237764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb49808f720 0x7fb49808d000 secure :-1 s=READY pgs=973 cs=0 l=1 rx=0x7fb48800f730 tx=0x7fb48800fad0).ready entity=mon.0 client_cookie=1223d32513674bf4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.982+0000 7fb49e0a6700 1 --2- 172.21.15.16:0/806237764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb49808f720 0x7fb49808d000 secure :-1 s=THROTTLE_DONE pgs=973 cs=0 l=1 rx=0x7fb48800f730 tx=0x7fb48800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.982+0000 7fb49e0a6700 1 --2- 172.21.15.16:0/806237764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb49808f720 0x7fb49808d000 secure :-1 s=THROTTLE_DONE pgs=973 cs=0 l=1 rx=0x7fb48800f730 tx=0x7fb48800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.982+0000 7fb49e0a6700 1 --2- 172.21.15.16:0/806237764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb49808f720 0x7fb49808d000 secure :-1 s=THROTTLE_DONE pgs=973 cs=0 l=1 rx=0x7fb48800f730 tx=0x7fb48800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.982+0000 7fb49c8a3700 1 -- 172.21.15.16:0/806237764 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb488000b90 con 0x7fb49808f720 2019-12-05T06:33:19.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.982+0000 7fb49c8a3700 1 -- 172.21.15.16:0/806237764 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4880010e0 con 0x7fb49808f720 2019-12-05T06:33:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.982+0000 7fb49c8a3700 1 -- 172.21.15.16:0/806237764 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb488001790 con 0x7fb49808f720 2019-12-05T06:33:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.982+0000 7fb49f308700 1 -- 172.21.15.16:0/806237764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb49808f720 msgr2=0x7fb49808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.982+0000 7fb49f308700 1 --2- 172.21.15.16:0/806237764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb49808f720 0x7fb49808d000 secure :-1 s=READY pgs=973 cs=0 l=1 rx=0x7fb48800f730 tx=0x7fb48800fad0).stop 2019-12-05T06:33:19.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.984+0000 7fb49f308700 1 -- 172.21.15.16:0/806237764 shutdown_connections 2019-12-05T06:33:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.984+0000 7fb49f308700 1 --2- 172.21.15.16:0/806237764 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb49808f720 0x7fb49808d000 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.984+0000 7fb49f308700 1 --2- 172.21.15.16:0/806237764 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49808c6d0 0x7fb49808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.984+0000 7fb49f308700 1 -- 172.21.15.16:0/806237764 shutdown_connections 2019-12-05T06:33:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.984+0000 7fb49f308700 1 -- 172.21.15.16:0/806237764 wait complete. 2019-12-05T06:33:19.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.984+0000 7fb49f308700 1 -- 172.21.15.16:0/806237764 >> 172.21.15.16:0/806237764 conn(0x7fb498061940 msgr2=0x7fb498096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.988+0000 7fb49f308700 1 Processor -- start 2019-12-05T06:33:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.988+0000 7fb49f308700 1 -- start start 2019-12-05T06:33:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.989+0000 7fb49f308700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.989+0000 7fb49f308700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4980879b0 0x7fb498086f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:19.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.989+0000 7fb49f308700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb49809e030 con 0x7fb498087460 2019-12-05T06:33:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.990+0000 7fb49f308700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb498085020 con 0x7fb4980879b0 2019-12-05T06:33:19.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.990+0000 7fb49f308700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4980852f0 con 0x7fb49814ca10 2019-12-05T06:33:19.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.990+0000 7fb49d0a4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4980879b0 0x7fb498086f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:19.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.990+0000 7fb49d8a5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.990+0000 7fb49d0a4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4980879b0 0x7fb498086f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55968/0 (socket says 172.21.15.16:55968) 2019-12-05T06:33:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.990+0000 7fb49e0a6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb498087460 0x7fb498152160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:34206/0 (socket says 172.21.15.16:34206) 2019-12-05T06:33:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.990+0000 7fb49d0a4700 1 -- 172.21.15.16:0/3783015559 learned_addr learned my addr 172.21.15.16:0/3783015559 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.990+0000 7fb4967fc700 1 -- 172.21.15.16:0/3783015559 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943043402 0 0) 0x7fb488000940 con 0x7fb498087460 2019-12-05T06:33:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.990+0000 7fb4967fc700 1 -- 172.21.15.16:0/3783015559 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb478002e80 con 0x7fb498087460 2019-12-05T06:33:19.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d8a5700 1 -- 172.21.15.16:0/3783015559 >> v2:172.21.15.16:3300/0 conn(0x7fb4980879b0 msgr2=0x7fb498086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> v2:172.21.15.16:3300/0 conn(0x7fb4980879b0 0x7fb498086f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d8a5700 1 -- 172.21.15.16:0/3783015559 >> v1:172.21.15.76:6789/0 conn(0x7fb498087460 legacy=0x7fb498152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d8a5700 1 -- 172.21.15.16:0/3783015559 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4980860e0 con 0x7fb49814ca10 2019-12-05T06:33:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d0a4700 1 --2- 172.21.15.16:0/3783015559 >> v2:172.21.15.16:3300/0 conn(0x7fb4980879b0 0x7fb498086f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:19.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=READY pgs=231 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb4967fc700 1 -- 172.21.15.16:0/3783015559 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb490000cc0 con 0x7fb49814ca10 2019-12-05T06:33:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49f308700 1 -- 172.21.15.16:0/3783015559 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4980860e0 con 0x7fb49814ca10 2019-12-05T06:33:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb4967fc700 1 -- 172.21.15.16:0/3783015559 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb490000cc0 con 0x7fb49814ca10 2019-12-05T06:33:20.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.991+0000 7fb4967fc700 1 -- 172.21.15.16:0/3783015559 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb49000e560 con 0x7fb49814ca10 2019-12-05T06:33:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.992+0000 7fb49f308700 1 -- 172.21.15.16:0/3783015559 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4980860e0 con 0x7fb49814ca10 2019-12-05T06:33:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.992+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.993+0000 7fb4967fc700 1 -- 172.21.15.16:0/3783015559 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb49000de80 con 0x7fb49814ca10 2019-12-05T06:33:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.993+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.994+0000 7fb4967fc700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb478023670 0x7fb478025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:20.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.994+0000 7fb4967fc700 1 -- 172.21.15.16:0/3783015559 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7fb490046100 con 0x7fb49814ca10 2019-12-05T06:33:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.994+0000 7fb49d0a4700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb478023670 0x7fb478025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:20.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:19.994+0000 7fb49d0a4700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb478023670 0x7fb478025ae0 crc :-1 s=READY pgs=419 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:20.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.011+0000 7fb49f308700 1 -- 172.21.15.16:0/3783015559 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4980860e0 con 0x7fb49814ca10 2019-12-05T06:33:20.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.017+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:20.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.017+0000 7fb4967fc700 1 -- 172.21.15.16:0/3783015559 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb49000ea70 con 0x7fb49814ca10 2019-12-05T06:33:20.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.279+0000 7fb49f308700 1 -- 172.21.15.16:0/3783015559 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4980860e0 con 0x7fb49814ca10 2019-12-05T06:33:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.280+0000 7fb49d8a5700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=THROTTLE_DONE pgs=231 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:20.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.280+0000 7fb4967fc700 1 -- 172.21.15.16:0/3783015559 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb49000dd90 con 0x7fb49814ca10 2019-12-05T06:33:20.278 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 122 2019-12-05T06:33:20.278 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:20.279 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":122}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.284+0000 7fb49f308700 1 -- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb478023670 msgr2=0x7fb478025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.284+0000 7fb49f308700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb478023670 0x7fb478025ae0 crc :-1 s=READY pgs=419 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:20.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.284+0000 7fb49f308700 1 -- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 msgr2=0x7fb4980869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.284+0000 7fb49f308700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 secure :-1 s=READY pgs=231 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).stop 2019-12-05T06:33:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.285+0000 7fb49f308700 1 -- 172.21.15.16:0/3783015559 shutdown_connections 2019-12-05T06:33:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.285+0000 7fb49f308700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb478023670 0x7fb478025ae0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.285+0000 7fb49f308700 1 --2- 172.21.15.16:0/3783015559 >> v2:172.21.15.16:3300/0 conn(0x7fb4980879b0 0x7fb498086f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.285+0000 7fb49f308700 1 --2- 172.21.15.16:0/3783015559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb49814ca10 0x7fb4980869e0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:20.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.285+0000 7fb49f308700 1 -- 172.21.15.16:0/3783015559 shutdown_connections 2019-12-05T06:33:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.285+0000 7fb49f308700 1 -- 172.21.15.16:0/3783015559 wait complete. 2019-12-05T06:33:20.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:20.285+0000 7fb49f308700 1 -- 172.21.15.16:0/3783015559 >> 172.21.15.16:0/3783015559 conn(0x7fb498098480 msgr2=0x7fb498071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:20.293 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:33:22.293 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.573+0000 7f53db53f700 1 Processor -- start 2019-12-05T06:33:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.573+0000 7f53db53f700 1 -- start start 2019-12-05T06:33:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.575+0000 7f53db53f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f53d408d2c0 0x7f53d408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.575+0000 7f53db53f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d40958e0 con 0x7f53d408c6d0 2019-12-05T06:33:22.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.575+0000 7f53db53f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d40629e0 con 0x7f53d408fef0 2019-12-05T06:33:22.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.575+0000 7f53db53f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d408df90 con 0x7f53d408d2c0 2019-12-05T06:33:22.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.575+0000 7f53d9adc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f53d408c6d0 0x7f53d408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41144/0 (socket says 172.21.15.16:41144) 2019-12-05T06:33:22.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.575+0000 7f53d9adc700 1 -- 172.21.15.16:0/234976081 learned_addr learned my addr 172.21.15.16:0/234976081 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:22.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.575+0000 7f53d92db700 1 --2- 172.21.15.16:0/234976081 >> v2:172.21.15.76:3300/0 conn(0x7f53d408d2c0 0x7f53d408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:22.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d8ada700 1 -- 172.21.15.16:0/234976081 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93809097 0 0) 0x7f53d0001150 con 0x7f53d408c6d0 2019-12-05T06:33:22.579 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d8ada700 1 -- 172.21.15.16:0/234976081 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53c0002e80 con 0x7f53d408c6d0 2019-12-05T06:33:22.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d92db700 1 -- 172.21.15.16:0/234976081 >> v1:172.21.15.76:6789/0 conn(0x7f53d408fef0 legacy=0x7f53d408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d92db700 1 -- 172.21.15.16:0/234976081 >> v1:172.21.15.16:6789/0 conn(0x7f53d408c6d0 legacy=0x7f53d408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d8ada700 1 -- 172.21.15.16:0/234976081 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2525542001 0 0) 0x7f53cc000d30 con 0x7f53d408fef0 2019-12-05T06:33:22.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d92db700 1 -- 172.21.15.16:0/234976081 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d408e620 con 0x7f53d408d2c0 2019-12-05T06:33:22.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d92db700 1 --2- 172.21.15.16:0/234976081 >> v2:172.21.15.76:3300/0 conn(0x7f53d408d2c0 0x7f53d408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f53c400af50 tx=0x7f53c400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.580 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d92db700 1 --2- 172.21.15.16:0/234976081 >> v2:172.21.15.76:3300/0 conn(0x7f53d408d2c0 0x7f53d408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f53c400af50 tx=0x7f53c400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d92db700 1 --2- 172.21.15.16:0/234976081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d408d2c0 0x7f53d408ef20 secure :-1 s=READY pgs=695 cs=0 l=1 rx=0x7f53c400af50 tx=0x7f53c400b760).ready entity=mon.1 client_cookie=e2184eefcb6f19c8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d92db700 1 --2- 172.21.15.16:0/234976081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d408d2c0 0x7f53d408ef20 secure :-1 s=THROTTLE_DONE pgs=695 cs=0 l=1 rx=0x7f53c400af50 tx=0x7f53c400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d8ada700 1 -- 172.21.15.16:0/234976081 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53c4000960 con 0x7f53d408d2c0 2019-12-05T06:33:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d92db700 1 --2- 172.21.15.16:0/234976081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d408d2c0 0x7f53d408ef20 secure :-1 s=THROTTLE_DONE pgs=695 cs=0 l=1 rx=0x7f53c400af50 tx=0x7f53c400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.576+0000 7f53d92db700 1 --2- 172.21.15.16:0/234976081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d408d2c0 0x7f53d408ef20 secure :-1 s=THROTTLE_DONE pgs=695 cs=0 l=1 rx=0x7f53c400af50 tx=0x7f53c400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.577+0000 7f53d8ada700 1 -- 172.21.15.16:0/234976081 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c4009860 con 0x7f53d408d2c0 2019-12-05T06:33:22.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.577+0000 7f53d8ada700 1 -- 172.21.15.16:0/234976081 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53c4009f30 con 0x7f53d408d2c0 2019-12-05T06:33:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.577+0000 7f53db53f700 1 -- 172.21.15.16:0/234976081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d408d2c0 msgr2=0x7f53d408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.577+0000 7f53db53f700 1 --2- 172.21.15.16:0/234976081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d408d2c0 0x7f53d408ef20 secure :-1 s=READY pgs=695 cs=0 l=1 rx=0x7f53c400af50 tx=0x7f53c400b760).stop 2019-12-05T06:33:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.578+0000 7f53db53f700 1 -- 172.21.15.16:0/234976081 shutdown_connections 2019-12-05T06:33:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.578+0000 7f53db53f700 1 --2- 172.21.15.16:0/234976081 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d408d2c0 0x7f53d408ef20 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.578+0000 7f53db53f700 1 -- 172.21.15.16:0/234976081 shutdown_connections 2019-12-05T06:33:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.578+0000 7f53db53f700 1 -- 172.21.15.16:0/234976081 wait complete. 2019-12-05T06:33:22.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.578+0000 7f53db53f700 1 -- 172.21.15.16:0/234976081 >> 172.21.15.16:0/234976081 conn(0x7f53d4061940 msgr2=0x7f53d4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.583+0000 7f53db53f700 1 Processor -- start 2019-12-05T06:33:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.583+0000 7f53db53f700 1 -- start start 2019-12-05T06:33:22.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.584+0000 7f53db53f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f53d406dc50 0x7f53d406e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:22.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.584+0000 7f53db53f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f53d4135450 0x7f53d406e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:22.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.584+0000 7f53db53f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53d4128af0 con 0x7f53d4134480 2019-12-05T06:33:22.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.584+0000 7f53db53f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53d4132830 con 0x7f53d4135450 2019-12-05T06:33:22.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.584+0000 7f53db53f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53d4129d80 con 0x7f53d406dc50 2019-12-05T06:33:22.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53d92db700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f53d4135450 0x7f53d406e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:22.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53da2dd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f53d4134480 0x7f53d4134900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41154/0 (socket says 172.21.15.16:41154) 2019-12-05T06:33:22.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53da2dd700 1 -- 172.21.15.16:0/3466263267 learned_addr learned my addr 172.21.15.16:0/3466263267 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:22.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53d9adc700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f53d406dc50 0x7f53d406e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:22.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53d92db700 1 -- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f53d406dc50 msgr2=0x7f53d406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f53d406dc50 0x7f53d406e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53d92db700 1 -- 172.21.15.16:0/3466263267 >> v1:172.21.15.16:6789/0 conn(0x7f53d4134480 legacy=0x7f53d4134900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53d92db700 1 -- 172.21.15.16:0/3466263267 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53d4133b80 con 0x7f53d4135450 2019-12-05T06:33:22.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53ca7fc700 1 -- 172.21.15.16:0/3466263267 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397533960 0 0) 0x7f53cc0015c0 con 0x7f53d4134480 2019-12-05T06:33:22.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> v2:172.21.15.76:3300/0 conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> v2:172.21.15.76:3300/0 conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.585+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=READY pgs=696 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:22.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.586+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.586+0000 7f53ca7fc700 1 -- 172.21.15.16:0/3466263267 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53c4000940 con 0x7f53d4135450 2019-12-05T06:33:22.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.586+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.586+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.586+0000 7f53ca7fc700 1 -- 172.21.15.16:0/3466263267 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53c4009980 con 0x7f53d4135450 2019-12-05T06:33:22.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.586+0000 7f53db53f700 1 -- 172.21.15.16:0/3466263267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53d4133b80 con 0x7f53d4135450 2019-12-05T06:33:22.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.586+0000 7f53ca7fc700 1 -- 172.21.15.16:0/3466263267 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53c400a050 con 0x7f53d4135450 2019-12-05T06:33:22.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.586+0000 7f53db53f700 1 -- 172.21.15.16:0/3466263267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53d4133b80 con 0x7f53d4135450 2019-12-05T06:33:22.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.586+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.587+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.587+0000 7f53ca7fc700 1 -- 172.21.15.16:0/3466263267 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f53c4009580 con 0x7f53d4135450 2019-12-05T06:33:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.587+0000 7f53ca7fc700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f53bc023630 0x7f53bc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.587+0000 7f53ca7fc700 1 -- 172.21.15.16:0/3466263267 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f53c4045f10 con 0x7f53d4135450 2019-12-05T06:33:22.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.588+0000 7f53d9adc700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f53bc023630 0x7f53bc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:22.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.588+0000 7f53d9adc700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f53bc023630 0x7f53bc025ac0 crc :-1 s=READY pgs=421 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:22.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.604+0000 7f53db53f700 1 -- 172.21.15.16:0/3466263267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53d4133b80 con 0x7f53d4135450 2019-12-05T06:33:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.610+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.610+0000 7f53ca7fc700 1 -- 172.21.15.16:0/3466263267 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f53c4000940 con 0x7f53d4135450 2019-12-05T06:33:22.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.813+0000 7f53db53f700 1 -- 172.21.15.16:0/3466263267 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53d4133b80 con 0x7f53d4135450 2019-12-05T06:33:22.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.814+0000 7f53d92db700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=THROTTLE_DONE pgs=696 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:22.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.814+0000 7f53ca7fc700 1 -- 172.21.15.16:0/3466263267 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f53c400a230 con 0x7f53d4135450 2019-12-05T06:33:22.813 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 122 2019-12-05T06:33:22.813 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:22.813 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":122}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:22.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.819+0000 7f53db53f700 1 -- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f53bc023630 msgr2=0x7f53bc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.819+0000 7f53db53f700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f53bc023630 0x7f53bc025ac0 crc :-1 s=READY pgs=421 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.819+0000 7f53db53f700 1 -- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 msgr2=0x7f53d406e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:22.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.819+0000 7f53db53f700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 0x7f53d406e580 secure :-1 s=READY pgs=696 cs=0 l=1 rx=0x7f53c40103c0 tx=0x7f53c40101f0).stop 2019-12-05T06:33:22.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.819+0000 7f53db53f700 1 -- 172.21.15.16:0/3466263267 shutdown_connections 2019-12-05T06:33:22.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.819+0000 7f53db53f700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f53bc023630 0x7f53bc025ac0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.819+0000 7f53db53f700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53d4135450 0x7f53d406e580 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.819+0000 7f53db53f700 1 --2- 172.21.15.16:0/3466263267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f53d406dc50 0x7f53d406e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:22.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.819+0000 7f53db53f700 1 -- 172.21.15.16:0/3466263267 shutdown_connections 2019-12-05T06:33:22.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.820+0000 7f53db53f700 1 -- 172.21.15.16:0/3466263267 wait complete. 2019-12-05T06:33:22.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:22.820+0000 7f53db53f700 1 -- 172.21.15.16:0/3466263267 >> 172.21.15.16:0/3466263267 conn(0x7f53d40a2850 msgr2=0x7f53d406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:22.833 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:33:24.834 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:25.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.088+0000 7f7069edc700 1 Processor -- start 2019-12-05T06:33:25.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.088+0000 7f7069edc700 1 -- start start 2019-12-05T06:33:25.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.090+0000 7f7069edc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f706408c6d0 0x7f706408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:25.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.090+0000 7f7069edc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f706408fef0 0x7f706408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:25.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.090+0000 7f7069edc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706408fb80 con 0x7f706408ef20 2019-12-05T06:33:25.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.090+0000 7f7069edc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f706408df90 con 0x7f706408c6d0 2019-12-05T06:33:25.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.090+0000 7f7069edc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70640a3f70 con 0x7f706408fef0 2019-12-05T06:33:25.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.090+0000 7f7063fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f706408c6d0 0x7f706408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:25.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.090+0000 7f7068c7a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f706408ef20 0x7f706408f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41168/0 (socket says 172.21.15.16:41168) 2019-12-05T06:33:25.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.090+0000 7f7063fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f706408c6d0 0x7f706408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:55998/0 (socket says 172.21.15.16:55998) 2019-12-05T06:33:25.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.090+0000 7f7068c7a700 1 -- 172.21.15.16:0/3735936652 learned_addr learned my addr 172.21.15.16:0/3735936652 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:25.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.090+0000 7f70637fe700 1 --2- 172.21.15.16:0/3735936652 >> v2:172.21.15.76:3300/0 conn(0x7f706408fef0 0x7f706408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:25.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.091+0000 7f7063fff700 1 -- 172.21.15.16:0/3735936652 >> v2:172.21.15.76:3300/0 conn(0x7f706408fef0 msgr2=0x7f706408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:25.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.091+0000 7f7063fff700 1 --2- 172.21.15.16:0/3735936652 >> v2:172.21.15.76:3300/0 conn(0x7f706408fef0 0x7f706408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:25.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.091+0000 7f7063fff700 1 -- 172.21.15.16:0/3735936652 >> v1:172.21.15.16:6789/0 conn(0x7f706408ef20 legacy=0x7f706408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:25.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.091+0000 7f7063fff700 1 -- 172.21.15.16:0/3735936652 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f706408d540 con 0x7f706408c6d0 2019-12-05T06:33:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.091+0000 7f7062ffd700 1 -- 172.21.15.16:0/3735936652 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327343264 0 0) 0x7f705c0014d0 con 0x7f706408ef20 2019-12-05T06:33:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.091+0000 7f7063fff700 1 --2- 172.21.15.16:0/3735936652 >> v2:172.21.15.16:3300/0 conn(0x7f706408c6d0 0x7f706408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f705400af50 tx=0x7f705400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.091+0000 7f7063fff700 1 --2- 172.21.15.16:0/3735936652 >> v2:172.21.15.16:3300/0 conn(0x7f706408c6d0 0x7f706408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f705400af50 tx=0x7f705400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.091+0000 7f7063fff700 1 --2- 172.21.15.16:0/3735936652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706408c6d0 0x7f706408cac0 secure :-1 s=READY pgs=977 cs=0 l=1 rx=0x7f705400af50 tx=0x7f705400b250).ready entity=mon.0 client_cookie=f80679678202def7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.092+0000 7f7063fff700 1 --2- 172.21.15.16:0/3735936652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706408c6d0 0x7f706408cac0 secure :-1 s=THROTTLE_DONE pgs=977 cs=0 l=1 rx=0x7f705400af50 tx=0x7f705400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.092+0000 7f7062ffd700 1 -- 172.21.15.16:0/3735936652 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7054007210 con 0x7f706408c6d0 2019-12-05T06:33:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.092+0000 7f7063fff700 1 --2- 172.21.15.16:0/3735936652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706408c6d0 0x7f706408cac0 secure :-1 s=THROTTLE_DONE pgs=977 cs=0 l=1 rx=0x7f705400af50 tx=0x7f705400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.092+0000 7f7063fff700 1 --2- 172.21.15.16:0/3735936652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706408c6d0 0x7f706408cac0 secure :-1 s=THROTTLE_DONE pgs=977 cs=0 l=1 rx=0x7f705400af50 tx=0x7f705400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.092+0000 7f7062ffd700 1 -- 172.21.15.16:0/3735936652 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7054007210 con 0x7f706408c6d0 2019-12-05T06:33:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.092+0000 7f7062ffd700 1 -- 172.21.15.16:0/3735936652 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7054009ba0 con 0x7f706408c6d0 2019-12-05T06:33:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.092+0000 7f7069edc700 1 -- 172.21.15.16:0/3735936652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706408c6d0 msgr2=0x7f706408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:25.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.092+0000 7f7069edc700 1 --2- 172.21.15.16:0/3735936652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706408c6d0 0x7f706408cac0 secure :-1 s=READY pgs=977 cs=0 l=1 rx=0x7f705400af50 tx=0x7f705400b250).stop 2019-12-05T06:33:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.094+0000 7f7069edc700 1 -- 172.21.15.16:0/3735936652 shutdown_connections 2019-12-05T06:33:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.094+0000 7f7069edc700 1 --2- 172.21.15.16:0/3735936652 >> v2:172.21.15.76:3300/0 conn(0x7f706408fef0 0x7f706408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.094+0000 7f7069edc700 1 --2- 172.21.15.16:0/3735936652 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f706408c6d0 0x7f706408cac0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.094+0000 7f7069edc700 1 -- 172.21.15.16:0/3735936652 shutdown_connections 2019-12-05T06:33:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.094+0000 7f7069edc700 1 -- 172.21.15.16:0/3735936652 wait complete. 2019-12-05T06:33:25.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.094+0000 7f7069edc700 1 -- 172.21.15.16:0/3735936652 >> 172.21.15.16:0/3735936652 conn(0x7f7064061940 msgr2=0x7f7064096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.099+0000 7f7069edc700 1 Processor -- start 2019-12-05T06:33:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.099+0000 7f7069edc700 1 -- start start 2019-12-05T06:33:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.101+0000 7f7069edc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f706414ca10 0x7f70640869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:25.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.101+0000 7f7069edc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f70640879b0 0x7f7064086f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.101+0000 7f7069edc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f706409e030 con 0x7f7064087460 2019-12-05T06:33:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.101+0000 7f7069edc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7064085a50 con 0x7f706414ca10 2019-12-05T06:33:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.101+0000 7f7069edc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7064085020 con 0x7f70640879b0 2019-12-05T06:33:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.101+0000 7f7068c7a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7064087460 0x7f7064152160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41174/0 (socket says 172.21.15.16:41174) 2019-12-05T06:33:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.101+0000 7f7068c7a700 1 -- 172.21.15.16:0/1218546359 learned_addr learned my addr 172.21.15.16:0/1218546359 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:25.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.101+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> v2:172.21.15.76:3300/0 conn(0x7f706414ca10 0x7f70640869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.101+0000 7f70637fe700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f70640879b0 0x7f7064086f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7060ff9700 1 -- 172.21.15.16:0/1218546359 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1245166911 0 0) 0x7f705c0014f0 con 0x7f7064087460 2019-12-05T06:33:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7063fff700 1 -- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f70640879b0 msgr2=0x7f7064086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f70640879b0 0x7f7064086f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7063fff700 1 -- 172.21.15.16:0/1218546359 >> v1:172.21.15.16:6789/0 conn(0x7f7064087460 legacy=0x7f7064152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7063fff700 1 -- 172.21.15.16:0/1218546359 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70640860e0 con 0x7f706414ca10 2019-12-05T06:33:25.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> v2:172.21.15.76:3300/0 conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> v2:172.21.15.76:3300/0 conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=READY pgs=697 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:25.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7060ff9700 1 -- 172.21.15.16:0/1218546359 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7054000940 con 0x7f706414ca10 2019-12-05T06:33:25.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7060ff9700 1 -- 172.21.15.16:0/1218546359 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7054009980 con 0x7f706414ca10 2019-12-05T06:33:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7060ff9700 1 -- 172.21.15.16:0/1218546359 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f705400a050 con 0x7f706414ca10 2019-12-05T06:33:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.102+0000 7f7069edc700 1 -- 172.21.15.16:0/1218546359 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70640860e0 con 0x7f706414ca10 2019-12-05T06:33:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.103+0000 7f7069edc700 1 -- 172.21.15.16:0/1218546359 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70640860e0 con 0x7f706414ca10 2019-12-05T06:33:25.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.103+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.104+0000 7f7060ff9700 1 -- 172.21.15.16:0/1218546359 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7054009b70 con 0x7f706414ca10 2019-12-05T06:33:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.104+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.105+0000 7f7060ff9700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7044023630 0x7f7044025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.105+0000 7f7060ff9700 1 -- 172.21.15.16:0/1218546359 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f7054045cd0 con 0x7f706414ca10 2019-12-05T06:33:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.105+0000 7f70637fe700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7044023630 0x7f7044025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:25.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.106+0000 7f70637fe700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7044023630 0x7f7044025ac0 crc :-1 s=READY pgs=422 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:25.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.119+0000 7f7069edc700 1 -- 172.21.15.16:0/1218546359 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70640860e0 con 0x7f706414ca10 2019-12-05T06:33:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.123+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.123+0000 7f7060ff9700 1 -- 172.21.15.16:0/1218546359 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7054016de0 con 0x7f706414ca10 2019-12-05T06:33:25.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.358+0000 7f7069edc700 1 -- 172.21.15.16:0/1218546359 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70640860e0 con 0x7f706414ca10 2019-12-05T06:33:25.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.359+0000 7f7063fff700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=THROTTLE_DONE pgs=697 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:25.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.359+0000 7f7060ff9700 1 -- 172.21.15.16:0/1218546359 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7054009b70 con 0x7f706414ca10 2019-12-05T06:33:25.356 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 122 2019-12-05T06:33:25.356 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:25.356 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":122}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:25.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.361+0000 7f7069edc700 1 -- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7044023630 msgr2=0x7f7044025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:25.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.361+0000 7f7069edc700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7044023630 0x7f7044025ac0 crc :-1 s=READY pgs=422 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:25.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.361+0000 7f7069edc700 1 -- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 msgr2=0x7f70640869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:25.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.361+0000 7f7069edc700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 0x7f70640869e0 secure :-1 s=READY pgs=697 cs=0 l=1 rx=0x7f70540103a0 tx=0x7f70540101d0).stop 2019-12-05T06:33:25.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.361+0000 7f7069edc700 1 -- 172.21.15.16:0/1218546359 shutdown_connections 2019-12-05T06:33:25.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.361+0000 7f7069edc700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7044023630 0x7f7044025ac0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:25.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.361+0000 7f7069edc700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f70640879b0 0x7f7064086f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:25.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.361+0000 7f7069edc700 1 --2- 172.21.15.16:0/1218546359 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f706414ca10 0x7f70640869e0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:25.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.362+0000 7f7069edc700 1 -- 172.21.15.16:0/1218546359 shutdown_connections 2019-12-05T06:33:25.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.362+0000 7f7069edc700 1 -- 172.21.15.16:0/1218546359 wait complete. 2019-12-05T06:33:25.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:25.362+0000 7f7069edc700 1 -- 172.21.15.16:0/1218546359 >> 172.21.15.16:0/1218546359 conn(0x7f7064098480 msgr2=0x7f7064071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:25.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:33:27.375 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.668+0000 7f4b0709c700 1 Processor -- start 2019-12-05T06:33:27.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.669+0000 7f4b0709c700 1 -- start start 2019-12-05T06:33:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.670+0000 7f4b0709c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b0008c6d0 0x7f4b0008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.671+0000 7f4b0709c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b0008fef0 0x7f4b0008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.671+0000 7f4b0709c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b000958e0 con 0x7f4b0008ef20 2019-12-05T06:33:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.671+0000 7f4b0709c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b0008df90 con 0x7f4b0008c6d0 2019-12-05T06:33:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.671+0000 7f4b0709c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b000a3f70 con 0x7f4b0008fef0 2019-12-05T06:33:27.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.671+0000 7f4b05639700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b0008c6d0 0x7f4b0008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.671+0000 7f4b05639700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b0008c6d0 0x7f4b0008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56012/0 (socket says 172.21.15.16:56012) 2019-12-05T06:33:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.671+0000 7f4b05639700 1 -- 172.21.15.16:0/3687169871 learned_addr learned my addr 172.21.15.16:0/3687169871 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.671+0000 7f4b04e38700 1 --2- 172.21.15.16:0/3687169871 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b0008fef0 0x7f4b0008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.671+0000 7f4afffff700 1 -- 172.21.15.16:0/3687169871 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1273276213 0 0) 0x7f4af8000d30 con 0x7f4b0008ef20 2019-12-05T06:33:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b05639700 1 -- 172.21.15.16:0/3687169871 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b0008fef0 msgr2=0x7f4b0008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:27.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b05639700 1 --2- 172.21.15.16:0/3687169871 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b0008fef0 0x7f4b0008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b05639700 1 -- 172.21.15.16:0/3687169871 >> v1:172.21.15.76:6789/0 conn(0x7f4b0008ef20 legacy=0x7f4b0008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b05639700 1 -- 172.21.15.16:0/3687169871 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b0008d540 con 0x7f4b0008c6d0 2019-12-05T06:33:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b04e38700 1 --2- 172.21.15.16:0/3687169871 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b0008fef0 0x7f4b0008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b05639700 1 --2- 172.21.15.16:0/3687169871 >> v2:172.21.15.16:3300/0 conn(0x7f4b0008c6d0 0x7f4b0008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4af000f740 tx=0x7f4af000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b05639700 1 --2- 172.21.15.16:0/3687169871 >> v2:172.21.15.16:3300/0 conn(0x7f4b0008c6d0 0x7f4b0008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4af000f740 tx=0x7f4af000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b05639700 1 --2- 172.21.15.16:0/3687169871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0008c6d0 0x7f4b0008cac0 secure :-1 s=READY pgs=979 cs=0 l=1 rx=0x7f4af000f740 tx=0x7f4af000fae0).ready entity=mon.0 client_cookie=92fe24608a1647b7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:27.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b05639700 1 --2- 172.21.15.16:0/3687169871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0008c6d0 0x7f4b0008cac0 secure :-1 s=THROTTLE_DONE pgs=979 cs=0 l=1 rx=0x7f4af000f740 tx=0x7f4af000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4afffff700 1 -- 172.21.15.16:0/3687169871 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4af0000960 con 0x7f4b0008c6d0 2019-12-05T06:33:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b05639700 1 --2- 172.21.15.16:0/3687169871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0008c6d0 0x7f4b0008cac0 secure :-1 s=THROTTLE_DONE pgs=979 cs=0 l=1 rx=0x7f4af000f740 tx=0x7f4af000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.672+0000 7f4b05639700 1 --2- 172.21.15.16:0/3687169871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0008c6d0 0x7f4b0008cac0 secure :-1 s=THROTTLE_DONE pgs=979 cs=0 l=1 rx=0x7f4af000f740 tx=0x7f4af000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.673+0000 7f4afffff700 1 -- 172.21.15.16:0/3687169871 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af0000c90 con 0x7f4b0008c6d0 2019-12-05T06:33:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.673+0000 7f4afffff700 1 -- 172.21.15.16:0/3687169871 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4af00013a0 con 0x7f4b0008c6d0 2019-12-05T06:33:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.673+0000 7f4b0709c700 1 -- 172.21.15.16:0/3687169871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0008c6d0 msgr2=0x7f4b0008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.673+0000 7f4b0709c700 1 --2- 172.21.15.16:0/3687169871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0008c6d0 0x7f4b0008cac0 secure :-1 s=READY pgs=979 cs=0 l=1 rx=0x7f4af000f740 tx=0x7f4af000fae0).stop 2019-12-05T06:33:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.674+0000 7f4b0709c700 1 -- 172.21.15.16:0/3687169871 shutdown_connections 2019-12-05T06:33:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.675+0000 7f4b0709c700 1 --2- 172.21.15.16:0/3687169871 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b0008fef0 0x7f4b0008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.675+0000 7f4b0709c700 1 --2- 172.21.15.16:0/3687169871 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0008c6d0 0x7f4b0008cac0 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.675+0000 7f4b0709c700 1 -- 172.21.15.16:0/3687169871 shutdown_connections 2019-12-05T06:33:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.675+0000 7f4b0709c700 1 -- 172.21.15.16:0/3687169871 wait complete. 2019-12-05T06:33:27.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.675+0000 7f4b0709c700 1 -- 172.21.15.16:0/3687169871 >> 172.21.15.16:0/3687169871 conn(0x7f4b00061940 msgr2=0x7f4b00096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.680+0000 7f4b0709c700 1 Processor -- start 2019-12-05T06:33:27.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.680+0000 7f4b0709c700 1 -- start start 2019-12-05T06:33:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.682+0000 7f4b0709c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b0006df10 0x7f4b00098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.682+0000 7f4b0709c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b00156cb0 0x7f4b001590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.682+0000 7f4b0709c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b00089fb0 con 0x7f4b000992d0 2019-12-05T06:33:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.682+0000 7f4b0709c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b00097370 con 0x7f4b0006df10 2019-12-05T06:33:27.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.682+0000 7f4b0709c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0008ad00 con 0x7f4b00156cb0 2019-12-05T06:33:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.683+0000 7f4b05639700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4b0006df10 0x7f4b00098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.683+0000 7f4b04e38700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4b000992d0 0x7f4b00098840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41188/0 (socket says 172.21.15.16:41188) 2019-12-05T06:33:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.683+0000 7f4b04e38700 1 -- 172.21.15.16:0/2357847146 learned_addr learned my addr 172.21.15.16:0/2357847146 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.683+0000 7f4b05e3a700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b00156cb0 0x7f4b001590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.683+0000 7f4b05639700 1 -- 172.21.15.16:0/2357847146 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b00156cb0 msgr2=0x7f4b001590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.683+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b00156cb0 0x7f4b001590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.683+0000 7f4b05639700 1 -- 172.21.15.16:0/2357847146 >> v1:172.21.15.16:6789/0 conn(0x7f4b000992d0 legacy=0x7f4b00098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.683+0000 7f4b05639700 1 -- 172.21.15.16:0/2357847146 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b00097a00 con 0x7f4b0006df10 2019-12-05T06:33:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.683+0000 7f4afdffb700 1 -- 172.21.15.16:0/2357847146 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 934776053 0 0) 0x7f4af400dd90 con 0x7f4b000992d0 2019-12-05T06:33:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.683+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> v2:172.21.15.16:3300/0 conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.684+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> v2:172.21.15.16:3300/0 conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.684+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=READY pgs=981 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.684+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.684+0000 7f4afdffb700 1 -- 172.21.15.16:0/2357847146 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4af0005370 con 0x7f4b0006df10 2019-12-05T06:33:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.684+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.684+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.684+0000 7f4afdffb700 1 -- 172.21.15.16:0/2357847146 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4af0007d60 con 0x7f4b0006df10 2019-12-05T06:33:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.684+0000 7f4b0709c700 1 -- 172.21.15.16:0/2357847146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b00097a00 con 0x7f4b0006df10 2019-12-05T06:33:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.684+0000 7f4afdffb700 1 -- 172.21.15.16:0/2357847146 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4af00059f0 con 0x7f4b0006df10 2019-12-05T06:33:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.684+0000 7f4b0709c700 1 -- 172.21.15.16:0/2357847146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b00097a00 con 0x7f4b0006df10 2019-12-05T06:33:27.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.685+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.686+0000 7f4afdffb700 1 -- 172.21.15.16:0/2357847146 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4af0005180 con 0x7f4b0006df10 2019-12-05T06:33:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.686+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.686+0000 7f4afdffb700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ae0023630 0x7f4ae0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.687+0000 7f4afdffb700 1 -- 172.21.15.16:0/2357847146 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 7579+0+0 (secure 0 0 0) 0x7f4af0045ce0 con 0x7f4b0006df10 2019-12-05T06:33:27.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.687+0000 7f4b04e38700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ae0023630 0x7f4ae0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:27.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.687+0000 7f4b04e38700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ae0023630 0x7f4ae0025ac0 crc :-1 s=READY pgs=423 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:27.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.700+0000 7f4b0709c700 1 -- 172.21.15.16:0/2357847146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b00097a00 con 0x7f4b0006df10 2019-12-05T06:33:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.709+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.709+0000 7f4afdffb700 1 -- 172.21.15.16:0/2357847146 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4af0030ee0 con 0x7f4b0006df10 2019-12-05T06:33:27.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.975+0000 7f4b0709c700 1 -- 172.21.15.16:0/2357847146 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b00097a00 con 0x7f4b0006df10 2019-12-05T06:33:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.976+0000 7f4b05639700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=THROTTLE_DONE pgs=981 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.976+0000 7f4afdffb700 1 -- 172.21.15.16:0/2357847146 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 122 v122) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4af0030ee0 con 0x7f4b0006df10 2019-12-05T06:33:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 122 2019-12-05T06:33:27.973 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:27.974 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":122,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5426,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":122}],"filesystems":[{"mdsmap":{"epoch":121,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:32:03.345117+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":67,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6561},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6561":{"gid":6561,"name":"a","rank":0,"incarnation":118,"state":"up:active","state_seq":22,"addr":"172.21.15.16:6833/1695355780","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1695355780},{"type":"v1","addr":"172.21.15.16:6833","nonce":1695355780}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:27.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 -- 172.21.15.16:0/2357847146 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ae0023630 msgr2=0x7f4ae0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ae0023630 0x7f4ae0025ac0 crc :-1 s=READY pgs=423 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 -- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 msgr2=0x7f4b00098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 0x7f4b00098300 secure :-1 s=READY pgs=981 cs=0 l=1 rx=0x7f4af0010350 tx=0x7f4af00106f0).stop 2019-12-05T06:33:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 -- 172.21.15.16:0/2357847146 shutdown_connections 2019-12-05T06:33:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4b00156cb0 0x7f4b001590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ae0023630 0x7f4ae0025ac0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 --2- 172.21.15.16:0/2357847146 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4b0006df10 0x7f4b00098300 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 -- 172.21.15.16:0/2357847146 shutdown_connections 2019-12-05T06:33:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 -- 172.21.15.16:0/2357847146 wait complete. 2019-12-05T06:33:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:27.978+0000 7f4b0709c700 1 -- 172.21.15.16:0/2357847146 >> 172.21.15.16:0/2357847146 conn(0x7f4b000714f0 msgr2=0x7f4b0009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:27.985 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:33:29.992 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.248+0000 7ff0b4fd6700 1 Processor -- start 2019-12-05T06:33:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.248+0000 7ff0b4fd6700 1 -- start start 2019-12-05T06:33:30.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.250+0000 7ff0b4fd6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff0b008c6d0 0x7ff0b008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.250+0000 7ff0b4fd6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff0b008fef0 0x7ff0b008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.250+0000 7ff0b4fd6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0b008ef20 0x7ff0b008f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.250+0000 7ff0b4fd6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b00a3f70 con 0x7ff0b008ef20 2019-12-05T06:33:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.250+0000 7ff0b4fd6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b008d830 con 0x7ff0b008fef0 2019-12-05T06:33:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.250+0000 7ff0b4fd6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b008d540 con 0x7ff0b008c6d0 2019-12-05T06:33:30.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0af59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0b008ef20 0x7ff0b008f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0ae59c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff0b008fef0 0x7ff0b008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0aed9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff0b008c6d0 0x7ff0b008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0ae59c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff0b008fef0 0x7ff0b008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:42118/0 (socket says 172.21.15.16:42118) 2019-12-05T06:33:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0af59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0b008ef20 0x7ff0b008f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56030/0 (socket says 172.21.15.16:56030) 2019-12-05T06:33:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0ae59c700 1 -- 172.21.15.16:0/3702721144 learned_addr learned my addr 172.21.15.16:0/3702721144 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:30.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0ae59c700 1 -- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff0b008c6d0 msgr2=0x7ff0b008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0ae59c700 1 --2- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff0b008c6d0 0x7ff0b008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0ae59c700 1 -- 172.21.15.16:0/3702721144 >> v2:172.21.15.16:3300/0 conn(0x7ff0b008ef20 msgr2=0x7ff0b008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0ae59c700 1 --2- 172.21.15.16:0/3702721144 >> v2:172.21.15.16:3300/0 conn(0x7ff0b008ef20 0x7ff0b008f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0ae59c700 1 -- 172.21.15.16:0/3702721144 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b008e620 con 0x7ff0b008fef0 2019-12-05T06:33:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0aed9d700 1 --2- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff0b008c6d0 0x7ff0b008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:30.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.251+0000 7ff0ae59c700 1 --2- 172.21.15.16:0/3702721144 >> v2:172.21.15.76:3300/0 conn(0x7ff0b008fef0 0x7ff0b008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff09800f740 tx=0x7ff09800f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.252+0000 7ff0ae59c700 1 --2- 172.21.15.16:0/3702721144 >> v2:172.21.15.76:3300/0 conn(0x7ff0b008fef0 0x7ff0b008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff09800f740 tx=0x7ff09800f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.252+0000 7ff0ae59c700 1 --2- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff0b008fef0 0x7ff0b008d000 secure :-1 s=READY pgs=699 cs=0 l=1 rx=0x7ff09800f740 tx=0x7ff09800f5f0).ready entity=mon.1 client_cookie=c2a347b150e5a44d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.252+0000 7ff0ae59c700 1 --2- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff0b008fef0 0x7ff0b008d000 secure :-1 s=THROTTLE_DONE pgs=699 cs=0 l=1 rx=0x7ff09800f740 tx=0x7ff09800f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.252+0000 7ff0ae59c700 1 --2- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff0b008fef0 0x7ff0b008d000 secure :-1 s=THROTTLE_DONE pgs=699 cs=0 l=1 rx=0x7ff09800f740 tx=0x7ff09800f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.252+0000 7ff0ae59c700 1 --2- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff0b008fef0 0x7ff0b008d000 secure :-1 s=THROTTLE_DONE pgs=699 cs=0 l=1 rx=0x7ff09800f740 tx=0x7ff09800f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.252+0000 7ff0add9b700 1 -- 172.21.15.16:0/3702721144 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff098000960 con 0x7ff0b008fef0 2019-12-05T06:33:30.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.252+0000 7ff0add9b700 1 -- 172.21.15.16:0/3702721144 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff098000c90 con 0x7ff0b008fef0 2019-12-05T06:33:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.252+0000 7ff0add9b700 1 -- 172.21.15.16:0/3702721144 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff098001340 con 0x7ff0b008fef0 2019-12-05T06:33:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.252+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff0b008fef0 msgr2=0x7ff0b008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.252+0000 7ff0b4fd6700 1 --2- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff0b008fef0 0x7ff0b008d000 secure :-1 s=READY pgs=699 cs=0 l=1 rx=0x7ff09800f740 tx=0x7ff09800f5f0).stop 2019-12-05T06:33:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.254+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/3702721144 shutdown_connections 2019-12-05T06:33:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.254+0000 7ff0b4fd6700 1 --2- 172.21.15.16:0/3702721144 >> v2:172.21.15.16:3300/0 conn(0x7ff0b008ef20 0x7ff0b008f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:30.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.254+0000 7ff0b4fd6700 1 --2- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff0b008fef0 0x7ff0b008d000 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.254+0000 7ff0b4fd6700 1 --2- 172.21.15.16:0/3702721144 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff0b008c6d0 0x7ff0b008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.254+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/3702721144 shutdown_connections 2019-12-05T06:33:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.254+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/3702721144 wait complete. 2019-12-05T06:33:30.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.254+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/3702721144 >> 172.21.15.16:0/3702721144 conn(0x7ff0b0061940 msgr2=0x7ff0b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.259+0000 7ff0b4fd6700 1 Processor -- start 2019-12-05T06:33:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.259+0000 7ff0b4fd6700 1 -- start start 2019-12-05T06:33:30.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.261+0000 7ff0b4fd6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0b0098b00 0x7ff0b015a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.261+0000 7ff0b4fd6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b0089fb0 con 0x7ff0b00992d0 2019-12-05T06:33:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.261+0000 7ff0b4fd6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0b006e300 con 0x7ff0b006df10 2019-12-05T06:33:30.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.261+0000 7ff0b4fd6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0b0097370 con 0x7ff0b0098b00 2019-12-05T06:33:30.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.261+0000 7ff0af59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0b0098b00 0x7ff0b015a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.261+0000 7ff0ae59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff0b00992d0 0x7ff0b0156cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41202/0 (socket says 172.21.15.16:41202) 2019-12-05T06:33:30.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.261+0000 7ff0aed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff0b006df10 0x7ff0b0098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:34270/0 (socket says 172.21.15.16:34270) 2019-12-05T06:33:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.261+0000 7ff0af59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff0b0098b00 0x7ff0b015a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56034/0 (socket says 172.21.15.16:56034) 2019-12-05T06:33:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.261+0000 7ff0ae59c700 1 -- 172.21.15.16:0/2025545568 learned_addr learned my addr 172.21.15.16:0/2025545568 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.261+0000 7ff0a77fe700 1 -- 172.21.15.16:0/2025545568 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723226166 0 0) 0x7ff0a000d970 con 0x7ff0b006df10 2019-12-05T06:33:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0a77fe700 1 -- 172.21.15.16:0/2025545568 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff094002e80 con 0x7ff0b006df10 2019-12-05T06:33:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0af59e700 1 -- 172.21.15.16:0/2025545568 >> v1:172.21.15.76:6789/0 conn(0x7ff0b006df10 legacy=0x7ff0b0098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:30.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0af59e700 1 -- 172.21.15.16:0/2025545568 >> v1:172.21.15.16:6789/0 conn(0x7ff0b00992d0 legacy=0x7ff0b0156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0a77fe700 1 -- 172.21.15.16:0/2025545568 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1870119454 0 0) 0x7ff098000940 con 0x7ff0b00992d0 2019-12-05T06:33:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0af59e700 1 -- 172.21.15.16:0/2025545568 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0b0097a00 con 0x7ff0b0098b00 2019-12-05T06:33:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0af59e700 1 --2- 172.21.15.16:0/2025545568 >> v2:172.21.15.16:3300/0 conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0af59e700 1 --2- 172.21.15.16:0/2025545568 >> v2:172.21.15.16:3300/0 conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0af59e700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=READY pgs=983 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0af59e700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=THROTTLE_DONE pgs=983 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0a77fe700 1 -- 172.21.15.16:0/2025545568 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0a800ab40 con 0x7ff0b0098b00 2019-12-05T06:33:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0af59e700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=THROTTLE_DONE pgs=983 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0af59e700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=THROTTLE_DONE pgs=983 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0a77fe700 1 -- 172.21.15.16:0/2025545568 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0a8009af0 con 0x7ff0b0098b00 2019-12-05T06:33:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/2025545568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b0097a00 con 0x7ff0b0098b00 2019-12-05T06:33:30.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0a77fe700 1 -- 172.21.15.16:0/2025545568 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0a800ab40 con 0x7ff0b0098b00 2019-12-05T06:33:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.262+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/2025545568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b0097a00 con 0x7ff0b0098b00 2019-12-05T06:33:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.263+0000 7ff0af59e700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=THROTTLE_DONE pgs=983 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.264+0000 7ff0a77fe700 1 -- 172.21.15.16:0/2025545568 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff0a8009900 con 0x7ff0b0098b00 2019-12-05T06:33:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.264+0000 7ff0af59e700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=THROTTLE_DONE pgs=983 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.264+0000 7ff0a77fe700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff094023680 0x7ff094025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:30.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.264+0000 7ff0a77fe700 1 -- 172.21.15.16:0/2025545568 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7ff0a80460c0 con 0x7ff0b0098b00 2019-12-05T06:33:30.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.265+0000 7ff0aed9d700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff094023680 0x7ff094025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:30.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.265+0000 7ff0aed9d700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff094023680 0x7ff094025af0 crc :-1 s=READY pgs=424 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:30.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.280+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/2025545568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b0097a00 con 0x7ff0b0098b00 2019-12-05T06:33:30.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.286+0000 7ff0af59e700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=THROTTLE_DONE pgs=983 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.287+0000 7ff0a77fe700 1 -- 172.21.15.16:0/2025545568 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff0a802b0e0 con 0x7ff0b0098b00 2019-12-05T06:33:30.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.495+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/2025545568 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0b0097a00 con 0x7ff0b0098b00 2019-12-05T06:33:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.496+0000 7ff0af59e700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=THROTTLE_DONE pgs=983 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.496+0000 7ff0a77fe700 1 -- 172.21.15.16:0/2025545568 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+1774 (secure 0 0 0) 0x7ff0a8009900 con 0x7ff0b0098b00 2019-12-05T06:33:30.496 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 123 2019-12-05T06:33:30.496 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:30.496 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:28.291047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:30.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.503+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/2025545568 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff094023680 msgr2=0x7ff094025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:30.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.503+0000 7ff0b4fd6700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff094023680 0x7ff094025af0 crc :-1 s=READY pgs=424 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:30.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.503+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 msgr2=0x7ff0b015a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:30.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.503+0000 7ff0b4fd6700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 0x7ff0b015a4d0 secure :-1 s=READY pgs=983 cs=0 l=1 rx=0x7ff0a80106f0 tx=0x7ff0a8010a90).stop 2019-12-05T06:33:30.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.503+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/2025545568 shutdown_connections 2019-12-05T06:33:30.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.503+0000 7ff0b4fd6700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff094023680 0x7ff094025af0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:30.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.503+0000 7ff0b4fd6700 1 --2- 172.21.15.16:0/2025545568 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff0b0098b00 0x7ff0b015a4d0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:30.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.503+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/2025545568 shutdown_connections 2019-12-05T06:33:30.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.503+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/2025545568 wait complete. 2019-12-05T06:33:30.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.503+0000 7ff0b4fd6700 1 -- 172.21.15.16:0/2025545568 >> 172.21.15.16:0/2025545568 conn(0x7ff0b00714f0 msgr2=0x7ff0b009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:30.518 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:33:30.518 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:33:30.518 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.749+0000 7fa9a7b32700 1 Processor -- start 2019-12-05T06:33:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.749+0000 7fa9a7b32700 1 -- start start 2019-12-05T06:33:30.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.751+0000 7fa9a7b32700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa9a008c6d0 0x7fa9a008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.751+0000 7fa9a7b32700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a008f720 0x7fa9a008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.751+0000 7fa9a7b32700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a008fb70 con 0x7fa9a008fef0 2019-12-05T06:33:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.751+0000 7fa9a7b32700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a008df90 con 0x7fa9a008c6d0 2019-12-05T06:33:30.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.751+0000 7fa9a7b32700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a00a3f70 con 0x7fa9a008f720 2019-12-05T06:33:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.752+0000 7fa9a60cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa9a008c6d0 0x7fa9a008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.752+0000 7fa9a60cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa9a008c6d0 0x7fa9a008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56040/0 (socket says 172.21.15.16:56040) 2019-12-05T06:33:30.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.752+0000 7fa9a60cf700 1 -- 172.21.15.16:0/1386399465 learned_addr learned my addr 172.21.15.16:0/1386399465 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.752+0000 7fa9a58ce700 1 --1- 172.21.15.16:0/1386399465 >> v1:172.21.15.16:6789/0 conn(0x7fa9a008fef0 0x7fa9a008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41212/0 (socket says 172.21.15.16:41212) 2019-12-05T06:33:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.752+0000 7fa9a68d0700 1 --2- 172.21.15.16:0/1386399465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a008f720 0x7fa9a008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:30.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.752+0000 7fa9a60cf700 1 -- 172.21.15.16:0/1386399465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a008f720 msgr2=0x7fa9a008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:30.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.752+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/1386399465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a008f720 0x7fa9a008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:30.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.752+0000 7fa9a60cf700 1 -- 172.21.15.16:0/1386399465 >> v1:172.21.15.16:6789/0 conn(0x7fa9a008fef0 legacy=0x7fa9a008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:30.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.752+0000 7fa9a60cf700 1 -- 172.21.15.16:0/1386399465 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a008d540 con 0x7fa9a008c6d0 2019-12-05T06:33:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.752+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/1386399465 >> v2:172.21.15.16:3300/0 conn(0x7fa9a008c6d0 0x7fa9a008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa99800fac0 tx=0x7fa99800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.753+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/1386399465 >> v2:172.21.15.16:3300/0 conn(0x7fa9a008c6d0 0x7fa9a008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa99800fac0 tx=0x7fa99800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.753+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/1386399465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9a008c6d0 0x7fa9a008cac0 secure :-1 s=READY pgs=985 cs=0 l=1 rx=0x7fa99800fac0 tx=0x7fa99800f8f0).ready entity=mon.0 client_cookie=8fa5cb245ce2c4f1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.753+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/1386399465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9a008c6d0 0x7fa9a008cac0 secure :-1 s=THROTTLE_DONE pgs=985 cs=0 l=1 rx=0x7fa99800fac0 tx=0x7fa99800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.753+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/1386399465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9a008c6d0 0x7fa9a008cac0 secure :-1 s=THROTTLE_DONE pgs=985 cs=0 l=1 rx=0x7fa99800fac0 tx=0x7fa99800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.753+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/1386399465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9a008c6d0 0x7fa9a008cac0 secure :-1 s=THROTTLE_DONE pgs=985 cs=0 l=1 rx=0x7fa99800fac0 tx=0x7fa99800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.753+0000 7fa9a50cd700 1 -- 172.21.15.16:0/1386399465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa998000ce0 con 0x7fa9a008c6d0 2019-12-05T06:33:30.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.753+0000 7fa9a50cd700 1 -- 172.21.15.16:0/1386399465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa998001010 con 0x7fa9a008c6d0 2019-12-05T06:33:30.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.753+0000 7fa9a50cd700 1 -- 172.21.15.16:0/1386399465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9980016a0 con 0x7fa9a008c6d0 2019-12-05T06:33:30.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.753+0000 7fa9a7b32700 1 -- 172.21.15.16:0/1386399465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9a008c6d0 msgr2=0x7fa9a008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:30.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.753+0000 7fa9a7b32700 1 --2- 172.21.15.16:0/1386399465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9a008c6d0 0x7fa9a008cac0 secure :-1 s=READY pgs=985 cs=0 l=1 rx=0x7fa99800fac0 tx=0x7fa99800f8f0).stop 2019-12-05T06:33:30.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.755+0000 7fa9a7b32700 1 -- 172.21.15.16:0/1386399465 shutdown_connections 2019-12-05T06:33:30.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.755+0000 7fa9a7b32700 1 --2- 172.21.15.16:0/1386399465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a008f720 0x7fa9a008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:30.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.755+0000 7fa9a7b32700 1 --2- 172.21.15.16:0/1386399465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9a008c6d0 0x7fa9a008cac0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:30.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.755+0000 7fa9a7b32700 1 -- 172.21.15.16:0/1386399465 shutdown_connections 2019-12-05T06:33:30.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.755+0000 7fa9a7b32700 1 -- 172.21.15.16:0/1386399465 wait complete. 2019-12-05T06:33:30.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.755+0000 7fa9a7b32700 1 -- 172.21.15.16:0/1386399465 >> 172.21.15.16:0/1386399465 conn(0x7fa9a0061940 msgr2=0x7fa9a0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:30.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.760+0000 7fa9a7b32700 1 Processor -- start 2019-12-05T06:33:30.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.760+0000 7fa9a7b32700 1 -- start start 2019-12-05T06:33:30.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa9a7b32700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa9a014da80 0x7fa9a012b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:30.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa9a7b32700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a012ba30 0x7fa9a009d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:30.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa9a7b32700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9a01288a0 con 0x7fa9a00851d0 2019-12-05T06:33:30.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa9a7b32700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9a012a560 con 0x7fa9a014da80 2019-12-05T06:33:30.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa9a7b32700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9a0129b30 con 0x7fa9a012ba30 2019-12-05T06:33:30.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa9a60cf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa9a014da80 0x7fa9a012b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:30.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa9a68d0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa9a00851d0 0x7fa9a009d970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41218/0 (socket says 172.21.15.16:41218) 2019-12-05T06:33:30.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa9a58ce700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a012ba30 0x7fa9a009d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:30.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa9a68d0700 1 -- 172.21.15.16:0/2476283666 learned_addr learned my addr 172.21.15.16:0/2476283666 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:30.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa9a60cf700 1 -- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a012ba30 msgr2=0x7fa9a009d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:30.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a012ba30 0x7fa9a009d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:30.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.762+0000 7fa99effd700 1 -- 172.21.15.16:0/2476283666 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3613393588 0 0) 0x7fa990001150 con 0x7fa9a00851d0 2019-12-05T06:33:30.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a60cf700 1 -- 172.21.15.16:0/2476283666 >> v1:172.21.15.16:6789/0 conn(0x7fa9a00851d0 legacy=0x7fa9a009d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:30.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a60cf700 1 -- 172.21.15.16:0/2476283666 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9a012abf0 con 0x7fa9a014da80 2019-12-05T06:33:30.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a58ce700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a012ba30 0x7fa9a009d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:30.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> v2:172.21.15.76:3300/0 conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> v2:172.21.15.76:3300/0 conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=READY pgs=701 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:30.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa99effd700 1 -- 172.21.15.16:0/2476283666 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9980058e0 con 0x7fa9a014da80 2019-12-05T06:33:30.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa99effd700 1 -- 172.21.15.16:0/2476283666 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa998005500 con 0x7fa9a014da80 2019-12-05T06:33:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa99effd700 1 -- 172.21.15.16:0/2476283666 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa998005ec0 con 0x7fa9a014da80 2019-12-05T06:33:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a7b32700 1 -- 172.21.15.16:0/2476283666 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9a012abf0 con 0x7fa9a014da80 2019-12-05T06:33:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.763+0000 7fa9a7b32700 1 -- 172.21.15.16:0/2476283666 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9a012abf0 con 0x7fa9a014da80 2019-12-05T06:33:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.764+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.765+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.765+0000 7fa99effd700 1 -- 172.21.15.16:0/2476283666 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa9980056f0 con 0x7fa9a014da80 2019-12-05T06:33:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.765+0000 7fa99effd700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa980023630 0x7fa980025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.765+0000 7fa99effd700 1 -- 172.21.15.16:0/2476283666 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7fa9980462b0 con 0x7fa9a014da80 2019-12-05T06:33:30.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.765+0000 7fa9a58ce700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa980023630 0x7fa980025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:30.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.766+0000 7fa9a58ce700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa980023630 0x7fa980025ae0 crc :-1 s=READY pgs=425 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.781+0000 7fa9a7b32700 1 -- 172.21.15.16:0/2476283666 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9a012abf0 con 0x7fa9a014da80 2019-12-05T06:33:30.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.785+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:30.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:30.785+0000 7fa99effd700 1 -- 172.21.15.16:0/2476283666 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa998007690 con 0x7fa9a014da80 2019-12-05T06:33:31.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.041+0000 7fa9a7b32700 1 -- 172.21.15.16:0/2476283666 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9a0084e90 con 0x7fa9a014da80 2019-12-05T06:33:31.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.041+0000 7fa9a60cf700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=THROTTLE_DONE pgs=701 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.042+0000 7fa99effd700 1 -- 172.21.15.16:0/2476283666 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 123 v123) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fa998005500 con 0x7fa9a014da80 2019-12-05T06:33:31.040 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 123 2019-12-05T06:33:31.040 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:31.040 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":123,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":123,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:28.291047+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:31.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.044+0000 7fa9a7b32700 1 -- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa980023630 msgr2=0x7fa980025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:31.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.044+0000 7fa9a7b32700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa980023630 0x7fa980025ae0 crc :-1 s=READY pgs=425 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.044+0000 7fa9a7b32700 1 -- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 msgr2=0x7fa9a012b4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.044+0000 7fa9a7b32700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 0x7fa9a012b4f0 secure :-1 s=READY pgs=701 cs=0 l=1 rx=0x7fa998010720 tx=0x7fa998010ac0).stop 2019-12-05T06:33:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.044+0000 7fa9a7b32700 1 -- 172.21.15.16:0/2476283666 shutdown_connections 2019-12-05T06:33:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.045+0000 7fa9a7b32700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa980023630 0x7fa980025ae0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.045+0000 7fa9a7b32700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9a012ba30 0x7fa9a009d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.045+0000 7fa9a7b32700 1 --2- 172.21.15.16:0/2476283666 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa9a014da80 0x7fa9a012b4f0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:31.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.045+0000 7fa9a7b32700 1 -- 172.21.15.16:0/2476283666 shutdown_connections 2019-12-05T06:33:31.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.045+0000 7fa9a7b32700 1 -- 172.21.15.16:0/2476283666 wait complete. 2019-12-05T06:33:31.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:31.045+0000 7fa9a7b32700 1 -- 172.21.15.16:0/2476283666 >> 172.21.15.16:0/2476283666 conn(0x7fa9a0099490 msgr2=0x7fa9a006ef90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:31.052 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 123, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 123, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5426": { "addr": "172.21.15.76:6835/3530466447", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 3530466447, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 3530466447, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5426, "incarnation": 123, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 68, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:33:28.291047+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5426 } } } ], "standbys": [] } 2019-12-05T06:33:33.053 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.315+0000 7f79ba8e9700 1 Processor -- start 2019-12-05T06:33:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.315+0000 7f79ba8e9700 1 -- start start 2019-12-05T06:33:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.317+0000 7f79ba8e9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 0x7f79b408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.317+0000 7f79ba8e9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b40958e0 con 0x7f79b408fef0 2019-12-05T06:33:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.317+0000 7f79ba8e9700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b40629e0 con 0x7f79b408f720 2019-12-05T06:33:33.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.317+0000 7f79ba8e9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b40a3f70 con 0x7f79b408c6d0 2019-12-05T06:33:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.317+0000 7f79b9687700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f79b408f720 0x7f79b41533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:34294/0 (socket says 172.21.15.16:34294) 2019-12-05T06:33:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.317+0000 7f79b9687700 1 -- 172.21.15.16:0/1430170178 learned_addr learned my addr 172.21.15.16:0/1430170178 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.317+0000 7f79b3fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f79b408fef0 0x7f79b408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41226/0 (socket says 172.21.15.16:41226) 2019-12-05T06:33:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.317+0000 7f79b8e86700 1 --2- 172.21.15.16:0/1430170178 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 0x7f79b408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.318+0000 7f79b37fe700 1 -- 172.21.15.16:0/1430170178 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2979262105 0 0) 0x7f79ac001480 con 0x7f79b408f720 2019-12-05T06:33:33.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.318+0000 7f79b37fe700 1 -- 172.21.15.16:0/1430170178 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f799c002e80 con 0x7f79b408f720 2019-12-05T06:33:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.318+0000 7f79b37fe700 1 -- 172.21.15.16:0/1430170178 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1774567436 0 0) 0x7f79a8001100 con 0x7f79b408fef0 2019-12-05T06:33:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.318+0000 7f79b37fe700 1 -- 172.21.15.16:0/1430170178 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f799c002f30 con 0x7f79b408fef0 2019-12-05T06:33:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.318+0000 7f79b8e86700 1 -- 172.21.15.16:0/1430170178 >> v1:172.21.15.76:6789/0 conn(0x7f79b408f720 legacy=0x7f79b41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.318+0000 7f79b8e86700 1 -- 172.21.15.16:0/1430170178 >> v1:172.21.15.16:6789/0 conn(0x7f79b408fef0 legacy=0x7f79b408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.318+0000 7f79b8e86700 1 -- 172.21.15.16:0/1430170178 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b408fbc0 con 0x7f79b408c6d0 2019-12-05T06:33:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.318+0000 7f79b8e86700 1 --2- 172.21.15.16:0/1430170178 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 0x7f79b408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f79a400af50 tx=0x7f79a400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.318+0000 7f79b8e86700 1 --2- 172.21.15.16:0/1430170178 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 0x7f79b408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f79a400af50 tx=0x7f79a400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.319+0000 7f79b8e86700 1 --2- 172.21.15.16:0/1430170178 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 0x7f79b408cac0 secure :-1 s=READY pgs=235 cs=0 l=1 rx=0x7f79a400af50 tx=0x7f79a400b760).ready entity=mon.2 client_cookie=58ab73205d7a4a72 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.319+0000 7f79b8e86700 1 --2- 172.21.15.16:0/1430170178 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 0x7f79b408cac0 secure :-1 s=THROTTLE_DONE pgs=235 cs=0 l=1 rx=0x7f79a400af50 tx=0x7f79a400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.319+0000 7f79b37fe700 1 -- 172.21.15.16:0/1430170178 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79a4000960 con 0x7f79b408c6d0 2019-12-05T06:33:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.319+0000 7f79b8e86700 1 --2- 172.21.15.16:0/1430170178 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 0x7f79b408cac0 secure :-1 s=THROTTLE_DONE pgs=235 cs=0 l=1 rx=0x7f79a400af50 tx=0x7f79a400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.319+0000 7f79b8e86700 1 --2- 172.21.15.16:0/1430170178 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 0x7f79b408cac0 secure :-1 s=THROTTLE_DONE pgs=235 cs=0 l=1 rx=0x7f79a400af50 tx=0x7f79a400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.319+0000 7f79b37fe700 1 -- 172.21.15.16:0/1430170178 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79a4009860 con 0x7f79b408c6d0 2019-12-05T06:33:33.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.319+0000 7f79b37fe700 1 -- 172.21.15.16:0/1430170178 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79a400a050 con 0x7f79b408c6d0 2019-12-05T06:33:33.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.319+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1430170178 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 msgr2=0x7f79b408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:33.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.319+0000 7f79ba8e9700 1 --2- 172.21.15.16:0/1430170178 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 0x7f79b408cac0 secure :-1 s=READY pgs=235 cs=0 l=1 rx=0x7f79a400af50 tx=0x7f79a400b760).stop 2019-12-05T06:33:33.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.320+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1430170178 shutdown_connections 2019-12-05T06:33:33.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.320+0000 7f79ba8e9700 1 --2- 172.21.15.16:0/1430170178 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b408c6d0 0x7f79b408cac0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.320+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1430170178 shutdown_connections 2019-12-05T06:33:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.320+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1430170178 wait complete. 2019-12-05T06:33:33.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.320+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1430170178 >> 172.21.15.16:0/1430170178 conn(0x7f79b4061940 msgr2=0x7f79b4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.325+0000 7f79ba8e9700 1 Processor -- start 2019-12-05T06:33:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.325+0000 7f79ba8e9700 1 -- start start 2019-12-05T06:33:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.327+0000 7f79ba8e9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b412b640 0x7f79b412ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:33.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.327+0000 7f79ba8e9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79b4153ba0 0x7f79b4155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:33.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.327+0000 7f79ba8e9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79b4131bf0 con 0x7f79b41302f0 2019-12-05T06:33:33.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.327+0000 7f79ba8e9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79b41296e0 con 0x7f79b4153ba0 2019-12-05T06:33:33.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.327+0000 7f79ba8e9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4128cb0 con 0x7f79b412b640 2019-12-05T06:33:33.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.327+0000 7f79b9687700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79b4153ba0 0x7f79b4155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:33.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.327+0000 7f79b9687700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f79b4153ba0 0x7f79b4155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56066/0 (socket says 172.21.15.16:56066) 2019-12-05T06:33:33.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.327+0000 7f79b3fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b412b640 0x7f79b412ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:33.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.327+0000 7f79b9687700 1 -- 172.21.15.16:0/1348453368 learned_addr learned my addr 172.21.15.16:0/1348453368 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b9687700 1 -- 172.21.15.16:0/1348453368 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b412b640 msgr2=0x7f79b412ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b412b640 0x7f79b412ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b9687700 1 -- 172.21.15.16:0/1348453368 >> v1:172.21.15.16:6789/0 conn(0x7f79b41302f0 legacy=0x7f79b412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b9687700 1 -- 172.21.15.16:0/1348453368 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b4129d70 con 0x7f79b4153ba0 2019-12-05T06:33:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> v2:172.21.15.16:3300/0 conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b3fff700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b412b640 0x7f79b412ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:33.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> v2:172.21.15.16:3300/0 conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=READY pgs=989 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=THROTTLE_DONE pgs=989 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=THROTTLE_DONE pgs=989 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=THROTTLE_DONE pgs=989 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b17fa700 1 -- 172.21.15.16:0/1348453368 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79ac000d30 con 0x7f79b4153ba0 2019-12-05T06:33:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b17fa700 1 -- 172.21.15.16:0/1348453368 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79ac004540 con 0x7f79b4153ba0 2019-12-05T06:33:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.328+0000 7f79b17fa700 1 -- 172.21.15.16:0/1348453368 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79ac004b50 con 0x7f79b4153ba0 2019-12-05T06:33:33.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.329+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1348453368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b4129d70 con 0x7f79b4153ba0 2019-12-05T06:33:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.329+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1348453368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b4129d70 con 0x7f79b4153ba0 2019-12-05T06:33:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.329+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=THROTTLE_DONE pgs=989 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.330+0000 7f79b17fa700 1 -- 172.21.15.16:0/1348453368 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f79ac004540 con 0x7f79b4153ba0 2019-12-05T06:33:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.331+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=THROTTLE_DONE pgs=989 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.331+0000 7f79b17fa700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7994023630 0x7f7994025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.331+0000 7f79b17fa700 1 -- 172.21.15.16:0/1348453368 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f79ac046190 con 0x7f79b4153ba0 2019-12-05T06:33:33.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.331+0000 7f79b8e86700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7994023630 0x7f7994025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:33.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.331+0000 7f79b8e86700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7994023630 0x7f7994025ac0 crc :-1 s=READY pgs=426 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:33.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.345+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1348453368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b4129d70 con 0x7f79b4153ba0 2019-12-05T06:33:33.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.351+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=THROTTLE_DONE pgs=989 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.351+0000 7f79b17fa700 1 -- 172.21.15.16:0/1348453368 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f79ac001690 con 0x7f79b4153ba0 2019-12-05T06:33:33.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.597+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1348453368 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79b4129d70 con 0x7f79b4153ba0 2019-12-05T06:33:33.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.597+0000 7f79b9687700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=THROTTLE_DONE pgs=989 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:33.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.597+0000 7f79b17fa700 1 -- 172.21.15.16:0/1348453368 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f79ac004540 con 0x7f79b4153ba0 2019-12-05T06:33:33.595 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 126 2019-12-05T06:33:33.596 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:33.596 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:33.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.600+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1348453368 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7994023630 msgr2=0x7f7994025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:33.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.600+0000 7f79ba8e9700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7994023630 0x7f7994025ac0 crc :-1 s=READY pgs=426 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:33.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.600+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 msgr2=0x7f79b4155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.600+0000 7f79ba8e9700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 0x7f79b4155fa0 secure :-1 s=READY pgs=989 cs=0 l=1 rx=0x7f79ac0107d0 tx=0x7f79ac010b70).stop 2019-12-05T06:33:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.600+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1348453368 shutdown_connections 2019-12-05T06:33:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.600+0000 7f79ba8e9700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7994023630 0x7f7994025ac0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.600+0000 7f79ba8e9700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f79b4153ba0 0x7f79b4155fa0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.600+0000 7f79ba8e9700 1 --2- 172.21.15.16:0/1348453368 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f79b412b640 0x7f79b412ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.601+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1348453368 shutdown_connections 2019-12-05T06:33:33.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.601+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1348453368 wait complete. 2019-12-05T06:33:33.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:33.601+0000 7f79ba8e9700 1 -- 172.21.15.16:0/1348453368 >> 172.21.15.16:0/1348453368 conn(0x7f79b40a2850 msgr2=0x7f79b406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:33.610 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=6561 2019-12-05T06:33:33.611 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2019-12-05T06:33:36.509 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:33:36.514 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:33:41.611 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:33:41.612 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:33:41.612 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:33:41.623 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:33:41.623 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:41.806 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:33:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.848+0000 7f67824e1700 1 Processor -- start 2019-12-05T06:33:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.848+0000 7f67824e1700 1 -- start start 2019-12-05T06:33:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.849+0000 7f67824e1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f677c08c6d0 0x7f677c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.849+0000 7f67824e1700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f677c08f720 0x7f677c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:41.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.849+0000 7f67824e1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677c08fb80 con 0x7f677c08fef0 2019-12-05T06:33:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.849+0000 7f67824e1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677c08df90 con 0x7f677c08c6d0 2019-12-05T06:33:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.849+0000 7f67824e1700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677c0a3f70 con 0x7f677c08f720 2019-12-05T06:33:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.849+0000 7f6780a7e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f677c08c6d0 0x7f677c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.849+0000 7f6780a7e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f677c08c6d0 0x7f677c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56118/0 (socket says 172.21.15.16:56118) 2019-12-05T06:33:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.849+0000 7f6780a7e700 1 -- 172.21.15.16:0/4105846578 learned_addr learned my addr 172.21.15.16:0/4105846578 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.849+0000 7f678127f700 1 --2- 172.21.15.16:0/4105846578 >> v2:172.21.15.76:3300/0 conn(0x7f677c08f720 0x7f677c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:41.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f6780a7e700 1 -- 172.21.15.16:0/4105846578 >> v2:172.21.15.76:3300/0 conn(0x7f677c08f720 msgr2=0x7f677c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:41.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f6780a7e700 1 --2- 172.21.15.16:0/4105846578 >> v2:172.21.15.76:3300/0 conn(0x7f677c08f720 0x7f677c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:41.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f6780a7e700 1 -- 172.21.15.16:0/4105846578 >> v1:172.21.15.76:6789/0 conn(0x7f677c08fef0 legacy=0x7f677c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:41.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f6780a7e700 1 -- 172.21.15.16:0/4105846578 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c08d540 con 0x7f677c08c6d0 2019-12-05T06:33:41.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f678127f700 1 --2- 172.21.15.16:0/4105846578 >> v2:172.21.15.76:3300/0 conn(0x7f677c08f720 0x7f677c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:33:41.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f6780a7e700 1 --2- 172.21.15.16:0/4105846578 >> v2:172.21.15.16:3300/0 conn(0x7f677c08c6d0 0x7f677c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f676400f740 tx=0x7f676400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f6780a7e700 1 --2- 172.21.15.16:0/4105846578 >> v2:172.21.15.16:3300/0 conn(0x7f677c08c6d0 0x7f677c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f676400f740 tx=0x7f676400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f6780a7e700 1 --2- 172.21.15.16:0/4105846578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f677c08c6d0 0x7f677c08cac0 secure :-1 s=READY pgs=991 cs=0 l=1 rx=0x7f676400f740 tx=0x7f676400fae0).ready entity=mon.0 client_cookie=146bf8cea787edca server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:41.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f6780a7e700 1 --2- 172.21.15.16:0/4105846578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f677c08c6d0 0x7f677c08cac0 secure :-1 s=THROTTLE_DONE pgs=991 cs=0 l=1 rx=0x7f676400f740 tx=0x7f676400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f6780a7e700 1 --2- 172.21.15.16:0/4105846578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f677c08c6d0 0x7f677c08cac0 secure :-1 s=THROTTLE_DONE pgs=991 cs=0 l=1 rx=0x7f676400f740 tx=0x7f676400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f6780a7e700 1 --2- 172.21.15.16:0/4105846578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f677c08c6d0 0x7f677c08cac0 secure :-1 s=THROTTLE_DONE pgs=991 cs=0 l=1 rx=0x7f676400f740 tx=0x7f676400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f677b7fe700 1 -- 172.21.15.16:0/4105846578 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6764000960 con 0x7f677c08c6d0 2019-12-05T06:33:41.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f677b7fe700 1 -- 172.21.15.16:0/4105846578 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6764000c90 con 0x7f677c08c6d0 2019-12-05T06:33:41.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f677b7fe700 1 -- 172.21.15.16:0/4105846578 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67640013a0 con 0x7f677c08c6d0 2019-12-05T06:33:41.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f67824e1700 1 -- 172.21.15.16:0/4105846578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f677c08c6d0 msgr2=0x7f677c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:41.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.850+0000 7f67824e1700 1 --2- 172.21.15.16:0/4105846578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f677c08c6d0 0x7f677c08cac0 secure :-1 s=READY pgs=991 cs=0 l=1 rx=0x7f676400f740 tx=0x7f676400fae0).stop 2019-12-05T06:33:41.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.852+0000 7f67824e1700 1 -- 172.21.15.16:0/4105846578 shutdown_connections 2019-12-05T06:33:41.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.852+0000 7f67824e1700 1 --2- 172.21.15.16:0/4105846578 >> v2:172.21.15.76:3300/0 conn(0x7f677c08f720 0x7f677c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:41.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.852+0000 7f67824e1700 1 --2- 172.21.15.16:0/4105846578 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f677c08c6d0 0x7f677c08cac0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:41.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.852+0000 7f67824e1700 1 -- 172.21.15.16:0/4105846578 shutdown_connections 2019-12-05T06:33:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.852+0000 7f67824e1700 1 -- 172.21.15.16:0/4105846578 wait complete. 2019-12-05T06:33:41.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.852+0000 7f67824e1700 1 -- 172.21.15.16:0/4105846578 >> 172.21.15.16:0/4105846578 conn(0x7f677c061940 msgr2=0x7f677c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:41.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.855+0000 7f67824e1700 1 Processor -- start 2019-12-05T06:33:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.855+0000 7f67824e1700 1 -- start start 2019-12-05T06:33:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.856+0000 7f67824e1700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f677c06e800 0x7f677c12fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.856+0000 7f67824e1700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.856+0000 7f67824e1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f677c128690 con 0x7f677c06dc10 2019-12-05T06:33:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.856+0000 7f67824e1700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f677c09e690 con 0x7f677c06e800 2019-12-05T06:33:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.856+0000 7f67824e1700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f677c12ec80 con 0x7f677c130be0 2019-12-05T06:33:41.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.857+0000 7f6780a7e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f677c06dc10 0x7f677c06e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41292/0 (socket says 172.21.15.16:41292) 2019-12-05T06:33:41.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.857+0000 7f6780a7e700 1 -- 172.21.15.16:0/3072283175 learned_addr learned my addr 172.21.15.16:0/3072283175 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:41.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.857+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:41.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.857+0000 7f677bfff700 1 --2- 172.21.15.16:0/3072283175 >> v2:172.21.15.76:3300/0 conn(0x7f677c06e800 0x7f677c12fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:41.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.857+0000 7f67797fa700 1 -- 172.21.15.16:0/3072283175 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550657027 0 0) 0x7f67640075d0 con 0x7f677c06dc10 2019-12-05T06:33:41.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.857+0000 7f67797fa700 1 -- 172.21.15.16:0/3072283175 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6760002e80 con 0x7f677c06dc10 2019-12-05T06:33:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.857+0000 7f678127f700 1 -- 172.21.15.16:0/3072283175 >> v2:172.21.15.76:3300/0 conn(0x7f677c06e800 msgr2=0x7f677c12fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.857+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> v2:172.21.15.76:3300/0 conn(0x7f677c06e800 0x7f677c12fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.857+0000 7f678127f700 1 -- 172.21.15.16:0/3072283175 >> v1:172.21.15.16:6789/0 conn(0x7f677c06dc10 legacy=0x7f677c06e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f67797fa700 1 -- 172.21.15.16:0/3072283175 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2587039544 0 0) 0x7f6764007da0 con 0x7f677c06dc10 2019-12-05T06:33:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f678127f700 1 -- 172.21.15.16:0/3072283175 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f677c12f310 con 0x7f677c130be0 2019-12-05T06:33:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=READY pgs=237 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f67797fa700 1 -- 172.21.15.16:0/3072283175 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6774002ea0 con 0x7f677c130be0 2019-12-05T06:33:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f67797fa700 1 -- 172.21.15.16:0/3072283175 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6774009d60 con 0x7f677c130be0 2019-12-05T06:33:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f67824e1700 1 -- 172.21.15.16:0/3072283175 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f677c12f310 con 0x7f677c130be0 2019-12-05T06:33:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f67797fa700 1 -- 172.21.15.16:0/3072283175 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f677400a4d0 con 0x7f677c130be0 2019-12-05T06:33:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.858+0000 7f67824e1700 1 -- 172.21.15.16:0/3072283175 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f677c12f310 con 0x7f677c130be0 2019-12-05T06:33:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.859+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.860+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.860+0000 7f67797fa700 1 -- 172.21.15.16:0/3072283175 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6774005550 con 0x7f677c130be0 2019-12-05T06:33:41.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.860+0000 7f67797fa700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6760023680 0x7f6760025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:41.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.860+0000 7f67797fa700 1 -- 172.21.15.16:0/3072283175 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f6774046120 con 0x7f677c130be0 2019-12-05T06:33:41.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.860+0000 7f6780a7e700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6760023680 0x7f6760025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:41.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.861+0000 7f6780a7e700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6760023680 0x7f6760025af0 crc :-1 s=READY pgs=427 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:41.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.863+0000 7f67824e1700 1 -- 172.21.15.16:0/3072283175 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f677c12f310 con 0x7f677c130be0 2019-12-05T06:33:41.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.872+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:41.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:41.872+0000 7f67797fa700 1 -- 172.21.15.16:0/3072283175 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6774009960 con 0x7f677c130be0 2019-12-05T06:33:42.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.129+0000 7f67824e1700 1 -- 172.21.15.16:0/3072283175 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f677c12f310 con 0x7f677c130be0 2019-12-05T06:33:42.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.129+0000 7f678127f700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=THROTTLE_DONE pgs=237 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:42.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.129+0000 7f67797fa700 1 -- 172.21.15.16:0/3072283175 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f6774005550 con 0x7f677c130be0 2019-12-05T06:33:42.127 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 126 2019-12-05T06:33:42.128 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:42.128 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:42.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 -- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6760023680 msgr2=0x7f6760025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:42.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6760023680 0x7f6760025af0 crc :-1 s=READY pgs=427 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:42.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 -- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 msgr2=0x7f677c130150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:42.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 secure :-1 s=READY pgs=237 cs=0 l=1 rx=0x7f67740106a0 tx=0x7f6774010a40).stop 2019-12-05T06:33:42.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 -- 172.21.15.16:0/3072283175 shutdown_connections 2019-12-05T06:33:42.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6760023680 0x7f6760025af0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:42.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 --2- 172.21.15.16:0/3072283175 >> v2:172.21.15.76:3300/0 conn(0x7f677c06e800 0x7f677c12fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:42.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 --2- 172.21.15.16:0/3072283175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f677c130be0 0x7f677c130150 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:42.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 -- 172.21.15.16:0/3072283175 shutdown_connections 2019-12-05T06:33:42.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 -- 172.21.15.16:0/3072283175 wait complete. 2019-12-05T06:33:42.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:42.132+0000 7f67824e1700 1 -- 172.21.15.16:0/3072283175 >> 172.21.15.16:0/3072283175 conn(0x7f677c098480 msgr2=0x7f677c071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:42.140 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:33:44.141 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.322+0000 7efe1fece700 1 Processor -- start 2019-12-05T06:33:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.322+0000 7efe1fece700 1 -- start start 2019-12-05T06:33:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1fece700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7efe1808d2c0 0x7efe1808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1fece700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe1808fef0 0x7efe1808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1fece700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe180958e0 con 0x7efe1808c6d0 2019-12-05T06:33:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1fece700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe1808df90 con 0x7efe1808d2c0 2019-12-05T06:33:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1fece700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe180a3f70 con 0x7efe1808fef0 2019-12-05T06:33:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1dc6a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7efe1808d2c0 0x7efe1808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1e46b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7efe1808c6d0 0x7efe1808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:34386/0 (socket says 172.21.15.16:34386) 2019-12-05T06:33:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1e46b700 1 -- 172.21.15.16:0/3077247875 learned_addr learned my addr 172.21.15.16:0/3077247875 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:33:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1ec6c700 1 --2- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe1808fef0 0x7efe1808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1dc6a700 1 -- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe1808fef0 msgr2=0x7efe1808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:33:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1dc6a700 1 --2- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe1808fef0 0x7efe1808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1dc6a700 1 -- 172.21.15.16:0/3077247875 >> v1:172.21.15.76:6789/0 conn(0x7efe1808c6d0 legacy=0x7efe1808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.323+0000 7efe1d469700 1 -- 172.21.15.16:0/3077247875 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477548595 0 0) 0x7efe14001480 con 0x7efe1808c6d0 2019-12-05T06:33:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1dc6a700 1 -- 172.21.15.16:0/3077247875 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe1808e620 con 0x7efe1808d2c0 2019-12-05T06:33:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1ec6c700 1 --2- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe1808fef0 0x7efe1808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:44.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1dc6a700 1 --2- 172.21.15.16:0/3077247875 >> v2:172.21.15.76:3300/0 conn(0x7efe1808d2c0 0x7efe1808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efe0800af50 tx=0x7efe0800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1dc6a700 1 --2- 172.21.15.16:0/3077247875 >> v2:172.21.15.76:3300/0 conn(0x7efe1808d2c0 0x7efe1808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efe0800af50 tx=0x7efe0800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1dc6a700 1 --2- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe1808d2c0 0x7efe1808ef20 secure :-1 s=READY pgs=707 cs=0 l=1 rx=0x7efe0800af50 tx=0x7efe0800b760).ready entity=mon.1 client_cookie=d1c633788fa4f1ff server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:44.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1dc6a700 1 --2- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe1808d2c0 0x7efe1808ef20 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7efe0800af50 tx=0x7efe0800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1d469700 1 -- 172.21.15.16:0/3077247875 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe08007210 con 0x7efe1808d2c0 2019-12-05T06:33:44.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1dc6a700 1 --2- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe1808d2c0 0x7efe1808ef20 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7efe0800af50 tx=0x7efe0800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1dc6a700 1 --2- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe1808d2c0 0x7efe1808ef20 secure :-1 s=THROTTLE_DONE pgs=707 cs=0 l=1 rx=0x7efe0800af50 tx=0x7efe0800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1d469700 1 -- 172.21.15.16:0/3077247875 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe080099b0 con 0x7efe1808d2c0 2019-12-05T06:33:44.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1d469700 1 -- 172.21.15.16:0/3077247875 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe0800a080 con 0x7efe1808d2c0 2019-12-05T06:33:44.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1fece700 1 -- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe1808d2c0 msgr2=0x7efe1808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.324+0000 7efe1fece700 1 --2- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe1808d2c0 0x7efe1808ef20 secure :-1 s=READY pgs=707 cs=0 l=1 rx=0x7efe0800af50 tx=0x7efe0800b760).stop 2019-12-05T06:33:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.325+0000 7efe1fece700 1 -- 172.21.15.16:0/3077247875 shutdown_connections 2019-12-05T06:33:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.325+0000 7efe1fece700 1 --2- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe1808fef0 0x7efe1808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:33:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.325+0000 7efe1fece700 1 --2- 172.21.15.16:0/3077247875 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe1808d2c0 0x7efe1808ef20 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.325+0000 7efe1fece700 1 -- 172.21.15.16:0/3077247875 shutdown_connections 2019-12-05T06:33:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.325+0000 7efe1fece700 1 -- 172.21.15.16:0/3077247875 wait complete. 2019-12-05T06:33:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.325+0000 7efe1fece700 1 -- 172.21.15.16:0/3077247875 >> 172.21.15.16:0/3077247875 conn(0x7efe18061940 msgr2=0x7efe18096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.328+0000 7efe1fece700 1 Processor -- start 2019-12-05T06:33:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.328+0000 7efe1fece700 1 -- start start 2019-12-05T06:33:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1fece700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7efe181302f0 0x7efe1812a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1fece700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe18153b50 0x7efe18155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1fece700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe18131bf0 con 0x7efe1812b640 2019-12-05T06:33:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1fece700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe181296e0 con 0x7efe181302f0 2019-12-05T06:33:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1fece700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe18128cb0 con 0x7efe18153b50 2019-12-05T06:33:44.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1ec6c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe18153b50 0x7efe18155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1ec6c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe18153b50 0x7efe18155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:60148/0 (socket says 172.21.15.16:60148) 2019-12-05T06:33:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1e46b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7efe181302f0 0x7efe1812a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1ec6c700 1 -- 172.21.15.16:0/39758003 learned_addr learned my addr 172.21.15.16:0/39758003 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1dc6a700 1 -- 172.21.15.16:0/39758003 >> v1:172.21.15.76:6789/0 conn(0x7efe1812b640 legacy=0x7efe1812abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:33:44.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1dc6a700 1 -- 172.21.15.16:0/39758003 >> v1:172.21.15.76:6789/0 conn(0x7efe1812b640 legacy=0x7efe1812abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:33:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1dc6a700 1 --1- 172.21.15.16:0/39758003 >> v1:172.21.15.76:6789/0 conn(0x7efe1812b640 0x7efe1812abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify read banner and identify addresses failed 2019-12-05T06:33:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1dc6a700 1 --1- 172.21.15.16:0/39758003 >> v1:172.21.15.76:6789/0 conn(0x7efe1812b640 0x7efe1812abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).fault on lossy channel, failing 2019-12-05T06:33:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1e46b700 1 -- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe18153b50 msgr2=0x7efe18155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:44.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe18153b50 0x7efe18155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1e46b700 1 -- 172.21.15.16:0/39758003 >> v1:172.21.15.76:6789/0 conn(0x7efe1812b640 legacy=0x7efe1812abb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:33:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1e46b700 1 -- 172.21.15.16:0/39758003 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe18129d70 con 0x7efe181302f0 2019-12-05T06:33:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1ec6c700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe18153b50 0x7efe18155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:33:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> v2:172.21.15.76:3300/0 conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> v2:172.21.15.76:3300/0 conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.329+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=READY pgs=708 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.330+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.330+0000 7efe0effd700 1 -- 172.21.15.16:0/39758003 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe14004730 con 0x7efe181302f0 2019-12-05T06:33:44.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.330+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.330+0000 7efe1fece700 1 -- 172.21.15.16:0/39758003 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe18129d70 con 0x7efe181302f0 2019-12-05T06:33:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.330+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.330+0000 7efe0effd700 1 -- 172.21.15.16:0/39758003 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe14001480 con 0x7efe181302f0 2019-12-05T06:33:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.330+0000 7efe0effd700 1 -- 172.21.15.16:0/39758003 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe14004920 con 0x7efe181302f0 2019-12-05T06:33:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.330+0000 7efe1fece700 1 -- 172.21.15.16:0/39758003 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe18129d70 con 0x7efe181302f0 2019-12-05T06:33:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.330+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.331+0000 7efe0effd700 1 -- 172.21.15.16:0/39758003 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efe14001480 con 0x7efe181302f0 2019-12-05T06:33:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.331+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.331+0000 7efe0effd700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efe00023630 0x7efe00025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.331+0000 7efe0effd700 1 -- 172.21.15.16:0/39758003 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7efe14045fa0 con 0x7efe181302f0 2019-12-05T06:33:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.331+0000 7efe1dc6a700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efe00023630 0x7efe00025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.331+0000 7efe1dc6a700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efe00023630 0x7efe00025ac0 crc :-1 s=READY pgs=429 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:44.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.337+0000 7efe1fece700 1 -- 172.21.15.16:0/39758003 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe18129d70 con 0x7efe181302f0 2019-12-05T06:33:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.341+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.341+0000 7efe0effd700 1 -- 172.21.15.16:0/39758003 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efe14001250 con 0x7efe181302f0 2019-12-05T06:33:44.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.543+0000 7efe1fece700 1 -- 172.21.15.16:0/39758003 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efe18129d70 con 0x7efe181302f0 2019-12-05T06:33:44.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.543+0000 7efe1e46b700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=THROTTLE_DONE pgs=708 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:44.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.543+0000 7efe0effd700 1 -- 172.21.15.16:0/39758003 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 126 v126) v1 ==== 95+0+1775 (secure 0 0 0) 0x7efe14001460 con 0x7efe181302f0 2019-12-05T06:33:44.542 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 126 2019-12-05T06:33:44.542 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:44.542 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":126,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:44.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 -- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efe00023630 msgr2=0x7efe00025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:44.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efe00023630 0x7efe00025ac0 crc :-1 s=READY pgs=429 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:44.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 -- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 msgr2=0x7efe1812a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:44.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 0x7efe1812a670 secure :-1 s=READY pgs=708 cs=0 l=1 rx=0x7efe14010640 tx=0x7efe140109e0).stop 2019-12-05T06:33:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 -- 172.21.15.16:0/39758003 shutdown_connections 2019-12-05T06:33:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efe00023630 0x7efe00025ac0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efe18153b50 0x7efe18155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 --2- 172.21.15.16:0/39758003 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7efe181302f0 0x7efe1812a670 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 -- 172.21.15.16:0/39758003 shutdown_connections 2019-12-05T06:33:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 -- 172.21.15.16:0/39758003 wait complete. 2019-12-05T06:33:44.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:44.546+0000 7efe1fece700 1 -- 172.21.15.16:0/39758003 >> 172.21.15.16:0/39758003 conn(0x7efe180a2850 msgr2=0x7efe1806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:44.554 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:33:46.556 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:33:46.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.772+0000 7f45a1487700 1 Processor -- start 2019-12-05T06:33:46.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.772+0000 7f45a1487700 1 -- start start 2019-12-05T06:33:46.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.773+0000 7f45a1487700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f459c08d2c0 0x7f459c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:33:46.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.773+0000 7f45a1487700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c0958e0 con 0x7f459c08c6d0 2019-12-05T06:33:46.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.773+0000 7f45a1487700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c0629e0 con 0x7f459c08fef0 2019-12-05T06:33:46.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.773+0000 7f45a1487700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c08df90 con 0x7f459c08d2c0 2019-12-05T06:33:46.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.773+0000 7f459affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f459c08d2c0 0x7f459c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:46.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.773+0000 7f459affd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f459c08d2c0 0x7f459c08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56182/0 (socket says 172.21.15.16:56182) 2019-12-05T06:33:46.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.773+0000 7f459affd700 1 -- 172.21.15.16:0/330890996 learned_addr learned my addr 172.21.15.16:0/330890996 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:46.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459affd700 1 -- 172.21.15.16:0/330890996 >> v1:172.21.15.76:6789/0 conn(0x7f459c08fef0 legacy=0x7f459c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:46.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459affd700 1 -- 172.21.15.16:0/330890996 >> v1:172.21.15.16:6789/0 conn(0x7f459c08c6d0 legacy=0x7f459c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:46.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459a7fc700 1 -- 172.21.15.16:0/330890996 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833047850 0 0) 0x7f458c0009b0 con 0x7f459c08fef0 2019-12-05T06:33:46.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459affd700 1 -- 172.21.15.16:0/330890996 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c08e620 con 0x7f459c08d2c0 2019-12-05T06:33:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459affd700 1 --2- 172.21.15.16:0/330890996 >> v2:172.21.15.16:3300/0 conn(0x7f459c08d2c0 0x7f459c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f458400fac0 tx=0x7f458400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459affd700 1 --2- 172.21.15.16:0/330890996 >> v2:172.21.15.16:3300/0 conn(0x7f459c08d2c0 0x7f459c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f458400fac0 tx=0x7f458400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459affd700 1 --2- 172.21.15.16:0/330890996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c08d2c0 0x7f459c08ef20 secure :-1 s=READY pgs=994 cs=0 l=1 rx=0x7f458400fac0 tx=0x7f458400fe60).ready entity=mon.0 client_cookie=92f7111be92ea863 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459affd700 1 --2- 172.21.15.16:0/330890996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c08d2c0 0x7f459c08ef20 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7f458400fac0 tx=0x7f458400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459affd700 1 --2- 172.21.15.16:0/330890996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c08d2c0 0x7f459c08ef20 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7f458400fac0 tx=0x7f458400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459a7fc700 1 -- 172.21.15.16:0/330890996 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4584000ce0 con 0x7f459c08d2c0 2019-12-05T06:33:46.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459affd700 1 --2- 172.21.15.16:0/330890996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c08d2c0 0x7f459c08ef20 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7f458400fac0 tx=0x7f458400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459a7fc700 1 -- 172.21.15.16:0/330890996 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4584001010 con 0x7f459c08d2c0 2019-12-05T06:33:46.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f459a7fc700 1 -- 172.21.15.16:0/330890996 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45840016e0 con 0x7f459c08d2c0 2019-12-05T06:33:46.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f45a1487700 1 -- 172.21.15.16:0/330890996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c08d2c0 msgr2=0x7f459c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:46.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.774+0000 7f45a1487700 1 --2- 172.21.15.16:0/330890996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c08d2c0 0x7f459c08ef20 secure :-1 s=READY pgs=994 cs=0 l=1 rx=0x7f458400fac0 tx=0x7f458400fe60).stop 2019-12-05T06:33:46.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.775+0000 7f45a1487700 1 -- 172.21.15.16:0/330890996 shutdown_connections 2019-12-05T06:33:46.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.775+0000 7f45a1487700 1 --2- 172.21.15.16:0/330890996 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c08d2c0 0x7f459c08ef20 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:46.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.775+0000 7f45a1487700 1 -- 172.21.15.16:0/330890996 shutdown_connections 2019-12-05T06:33:46.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.775+0000 7f45a1487700 1 -- 172.21.15.16:0/330890996 wait complete. 2019-12-05T06:33:46.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.775+0000 7f45a1487700 1 -- 172.21.15.16:0/330890996 >> 172.21.15.16:0/330890996 conn(0x7f459c061940 msgr2=0x7f459c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.777+0000 7f45a1487700 1 Processor -- start 2019-12-05T06:33:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.778+0000 7f45a1487700 1 -- start start 2019-12-05T06:33:46.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.778+0000 7f45a1487700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f459c14ca10 0x7f459c0869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.778+0000 7f45a1487700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f459c09ce90 0x7f459c19a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.778+0000 7f45a1487700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f459c09e030 con 0x7f459c0879b0 2019-12-05T06:33:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.778+0000 7f45a1487700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f459c085020 con 0x7f459c09ce90 2019-12-05T06:33:46.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.778+0000 7f45a1487700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f459c0852f0 con 0x7f459c14ca10 2019-12-05T06:33:46.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f459c09ce90 0x7f459c19a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:46.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f459c09ce90 0x7f459c19a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56188/0 (socket says 172.21.15.16:56188) 2019-12-05T06:33:46.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 -- 172.21.15.16:0/2208850152 learned_addr learned my addr 172.21.15.16:0/2208850152 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:33:46.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459b7fe700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f459c14ca10 0x7f459c0869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 -- 172.21.15.16:0/2208850152 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f459c14ca10 msgr2=0x7f459c0869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f459c14ca10 0x7f459c0869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 -- 172.21.15.16:0/2208850152 >> v1:172.21.15.76:6789/0 conn(0x7f459c0879b0 legacy=0x7f459c086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 -- 172.21.15.16:0/2208850152 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f459c0860e0 con 0x7f459c09ce90 2019-12-05T06:33:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> v2:172.21.15.16:3300/0 conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> v2:172.21.15.16:3300/0 conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=READY pgs=995 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=THROTTLE_DONE pgs=995 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=THROTTLE_DONE pgs=995 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f4593fff700 1 -- 172.21.15.16:0/2208850152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f458c005770 con 0x7f459c09ce90 2019-12-05T06:33:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=THROTTLE_DONE pgs=995 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f4593fff700 1 -- 172.21.15.16:0/2208850152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f458c005390 con 0x7f459c09ce90 2019-12-05T06:33:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f45a1487700 1 -- 172.21.15.16:0/2208850152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f459c0860e0 con 0x7f459c09ce90 2019-12-05T06:33:46.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.779+0000 7f4593fff700 1 -- 172.21.15.16:0/2208850152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f458c005d50 con 0x7f459c09ce90 2019-12-05T06:33:46.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.780+0000 7f45a1487700 1 -- 172.21.15.16:0/2208850152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f459c0860e0 con 0x7f459c09ce90 2019-12-05T06:33:46.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.780+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=THROTTLE_DONE pgs=995 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.781+0000 7f4593fff700 1 -- 172.21.15.16:0/2208850152 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f458c005580 con 0x7f459c09ce90 2019-12-05T06:33:46.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.781+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=THROTTLE_DONE pgs=995 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.781+0000 7f4593fff700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4580023630 0x7f4580025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:33:46.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.781+0000 7f4593fff700 1 -- 172.21.15.16:0/2208850152 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f458c045dc0 con 0x7f459c09ce90 2019-12-05T06:33:46.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.781+0000 7f459b7fe700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4580023630 0x7f4580025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:33:46.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.782+0000 7f459b7fe700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4580023630 0x7f4580025ac0 crc :-1 s=READY pgs=431 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:33:46.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.787+0000 7f45a1487700 1 -- 172.21.15.16:0/2208850152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f459c0860e0 con 0x7f459c09ce90 2019-12-05T06:33:46.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.791+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=THROTTLE_DONE pgs=995 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.791+0000 7f4593fff700 1 -- 172.21.15.16:0/2208850152 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f458c0075a0 con 0x7f459c09ce90 2019-12-05T06:33:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.985+0000 7f45a1487700 1 -- 172.21.15.16:0/2208850152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f459c0860e0 con 0x7f459c09ce90 2019-12-05T06:33:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.986+0000 7f459bfff700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=THROTTLE_DONE pgs=995 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:33:46.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.986+0000 7f4593fff700 1 -- 172.21.15.16:0/2208850152 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f458c005580 con 0x7f459c09ce90 2019-12-05T06:33:46.984 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 127 2019-12-05T06:33:46.984 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:33:46.984 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:33:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.988+0000 7f45a1487700 1 -- 172.21.15.16:0/2208850152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4580023630 msgr2=0x7f4580025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.988+0000 7f45a1487700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4580023630 0x7f4580025ac0 crc :-1 s=READY pgs=431 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.988+0000 7f45a1487700 1 -- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 msgr2=0x7f459c19a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:33:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.988+0000 7f45a1487700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 0x7f459c19a400 secure :-1 s=READY pgs=995 cs=0 l=1 rx=0x7f458c010410 tx=0x7f458c010290).stop 2019-12-05T06:33:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.988+0000 7f45a1487700 1 -- 172.21.15.16:0/2208850152 shutdown_connections 2019-12-05T06:33:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.988+0000 7f45a1487700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4580023630 0x7f4580025ac0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.988+0000 7f45a1487700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f459c09ce90 0x7f459c19a400 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.988+0000 7f45a1487700 1 --2- 172.21.15.16:0/2208850152 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f459c14ca10 0x7f459c0869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:33:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.989+0000 7f45a1487700 1 -- 172.21.15.16:0/2208850152 shutdown_connections 2019-12-05T06:33:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.989+0000 7f45a1487700 1 -- 172.21.15.16:0/2208850152 wait complete. 2019-12-05T06:33:46.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:33:46.989+0000 7f45a1487700 1 -- 172.21.15.16:0/2208850152 >> 172.21.15.16:0/2208850152 conn(0x7f459c098480 msgr2=0x7f459c071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:33:46.997 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:33:46.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 118 secs before thrashing 2019-12-05T06:34:06.933 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:34:06.937 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:34:37.121 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:34:37.126 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:35:07.287 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:35:07.293 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:35:37.512 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:35:37.525 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:35:44.998 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:45.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.206+0000 7fdfcc132700 1 Processor -- start 2019-12-05T06:35:45.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.206+0000 7fdfcc132700 1 -- start start 2019-12-05T06:35:45.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcc132700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdfc408c6d0 0x7fdfc408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:45.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcc132700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfc408f720 0x7fdfc408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcc132700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc40958e0 con 0x7fdfc408fef0 2019-12-05T06:35:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcc132700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc40a3f70 con 0x7fdfc408f720 2019-12-05T06:35:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcc132700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdfc408d830 con 0x7fdfc408c6d0 2019-12-05T06:35:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcaed0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfc408f720 0x7fdfc408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcaed0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfc408f720 0x7fdfc408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56684/0 (socket says 172.21.15.16:56684) 2019-12-05T06:35:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcaed0700 1 -- 172.21.15.16:0/2044355937 learned_addr learned my addr 172.21.15.16:0/2044355937 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:35:45.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfca6cf700 1 --2- 172.21.15.16:0/2044355937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdfc408c6d0 0x7fdfc408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcaed0700 1 -- 172.21.15.16:0/2044355937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdfc408c6d0 msgr2=0x7fdfc408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcaed0700 1 --2- 172.21.15.16:0/2044355937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdfc408c6d0 0x7fdfc408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.207+0000 7fdfcaed0700 1 -- 172.21.15.16:0/2044355937 >> v1:172.21.15.76:6789/0 conn(0x7fdfc408fef0 legacy=0x7fdfc408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfcaed0700 1 -- 172.21.15.16:0/2044355937 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc408fbc0 con 0x7fdfc408f720 2019-12-05T06:35:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfcaed0700 1 --2- 172.21.15.16:0/2044355937 >> v2:172.21.15.16:3300/0 conn(0x7fdfc408f720 0x7fdfc408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdfbc00fa30 tx=0x7fdfbc00fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfcaed0700 1 --2- 172.21.15.16:0/2044355937 >> v2:172.21.15.16:3300/0 conn(0x7fdfc408f720 0x7fdfc408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdfbc00fa30 tx=0x7fdfbc00fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfcaed0700 1 --2- 172.21.15.16:0/2044355937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc408f720 0x7fdfc408d000 secure :-1 s=READY pgs=1007 cs=0 l=1 rx=0x7fdfbc00fa30 tx=0x7fdfbc00fdd0).ready entity=mon.0 client_cookie=29be17ad57110d1f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:45.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfcaed0700 1 --2- 172.21.15.16:0/2044355937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc408f720 0x7fdfc408d000 secure :-1 s=THROTTLE_DONE pgs=1007 cs=0 l=1 rx=0x7fdfbc00fa30 tx=0x7fdfbc00fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfcaed0700 1 --2- 172.21.15.16:0/2044355937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc408f720 0x7fdfc408d000 secure :-1 s=THROTTLE_DONE pgs=1007 cs=0 l=1 rx=0x7fdfbc00fa30 tx=0x7fdfbc00fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfcaed0700 1 --2- 172.21.15.16:0/2044355937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc408f720 0x7fdfc408d000 secure :-1 s=THROTTLE_DONE pgs=1007 cs=0 l=1 rx=0x7fdfbc00fa30 tx=0x7fdfbc00fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfc96cd700 1 -- 172.21.15.16:0/2044355937 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfbc000df0 con 0x7fdfc408f720 2019-12-05T06:35:45.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfc96cd700 1 -- 172.21.15.16:0/2044355937 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfbc001460 con 0x7fdfc408f720 2019-12-05T06:35:45.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfc96cd700 1 -- 172.21.15.16:0/2044355937 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfbc001b30 con 0x7fdfc408f720 2019-12-05T06:35:45.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfcc132700 1 -- 172.21.15.16:0/2044355937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc408f720 msgr2=0x7fdfc408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.208+0000 7fdfcc132700 1 --2- 172.21.15.16:0/2044355937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc408f720 0x7fdfc408d000 secure :-1 s=READY pgs=1007 cs=0 l=1 rx=0x7fdfbc00fa30 tx=0x7fdfbc00fdd0).stop 2019-12-05T06:35:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.209+0000 7fdfcc132700 1 -- 172.21.15.16:0/2044355937 shutdown_connections 2019-12-05T06:35:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.209+0000 7fdfcc132700 1 --2- 172.21.15.16:0/2044355937 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc408f720 0x7fdfc408d000 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.209+0000 7fdfcc132700 1 --2- 172.21.15.16:0/2044355937 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdfc408c6d0 0x7fdfc408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.209+0000 7fdfcc132700 1 -- 172.21.15.16:0/2044355937 shutdown_connections 2019-12-05T06:35:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.209+0000 7fdfcc132700 1 -- 172.21.15.16:0/2044355937 wait complete. 2019-12-05T06:35:45.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.209+0000 7fdfcc132700 1 -- 172.21.15.16:0/2044355937 >> 172.21.15.16:0/2044355937 conn(0x7fdfc4061940 msgr2=0x7fdfc4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:45.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.212+0000 7fdfcc132700 1 Processor -- start 2019-12-05T06:35:45.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.212+0000 7fdfcc132700 1 -- start start 2019-12-05T06:35:45.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfcc132700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfc4088b50 0x7fdfc4088380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:45.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfcc132700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdfc4152c20 0x7fdfc419b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:45.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfcc132700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc4084ed0 con 0x7fdfc409e440 2019-12-05T06:35:45.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfcc132700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc4086bf0 con 0x7fdfc4088b50 2019-12-05T06:35:45.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfcc132700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdfc40861c0 con 0x7fdfc4152c20 2019-12-05T06:35:45.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfc4088b50 0x7fdfc4088380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdfc4088b50 0x7fdfc4088380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56690/0 (socket says 172.21.15.16:56690) 2019-12-05T06:35:45.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfca6cf700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdfc409e440 0x7fdfc4087b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41856/0 (socket says 172.21.15.16:41856) 2019-12-05T06:35:45.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 -- 172.21.15.16:0/1668857199 learned_addr learned my addr 172.21.15.16:0/1668857199 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:35:45.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfcaed0700 1 --2- 172.21.15.16:0/1668857199 >> v2:172.21.15.76:3300/0 conn(0x7fdfc4152c20 0x7fdfc419b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 -- 172.21.15.16:0/1668857199 >> v2:172.21.15.76:3300/0 conn(0x7fdfc4152c20 msgr2=0x7fdfc419b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> v2:172.21.15.76:3300/0 conn(0x7fdfc4152c20 0x7fdfc419b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 -- 172.21.15.16:0/1668857199 >> v1:172.21.15.16:6789/0 conn(0x7fdfc409e440 legacy=0x7fdfc4087b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 -- 172.21.15.16:0/1668857199 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc4087280 con 0x7fdfc4088b50 2019-12-05T06:35:45.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc2ffd700 1 -- 172.21.15.16:0/1668857199 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623038449 0 0) 0x7fdfb4006680 con 0x7fdfc409e440 2019-12-05T06:35:45.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> v2:172.21.15.16:3300/0 conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> v2:172.21.15.16:3300/0 conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=READY pgs=1009 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:45.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=THROTTLE_DONE pgs=1009 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc2ffd700 1 -- 172.21.15.16:0/1668857199 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfb80057c0 con 0x7fdfc4088b50 2019-12-05T06:35:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=THROTTLE_DONE pgs=1009 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=THROTTLE_DONE pgs=1009 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfc2ffd700 1 -- 172.21.15.16:0/1668857199 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdfb80053e0 con 0x7fdfc4088b50 2019-12-05T06:35:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.213+0000 7fdfcc132700 1 -- 172.21.15.16:0/1668857199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc4087280 con 0x7fdfc4088b50 2019-12-05T06:35:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.214+0000 7fdfc2ffd700 1 -- 172.21.15.16:0/1668857199 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdfb8005e40 con 0x7fdfc4088b50 2019-12-05T06:35:45.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.214+0000 7fdfcc132700 1 -- 172.21.15.16:0/1668857199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc4087280 con 0x7fdfc4088b50 2019-12-05T06:35:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.214+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=THROTTLE_DONE pgs=1009 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.215+0000 7fdfc2ffd700 1 -- 172.21.15.16:0/1668857199 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdfb8005670 con 0x7fdfc4088b50 2019-12-05T06:35:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.215+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=THROTTLE_DONE pgs=1009 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.215+0000 7fdfc2ffd700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfa4023630 0x7fdfa4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.215+0000 7fdfc2ffd700 1 -- 172.21.15.16:0/1668857199 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7fdfb8045e00 con 0x7fdfc4088b50 2019-12-05T06:35:45.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.215+0000 7fdfca6cf700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfa4023630 0x7fdfa4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.215+0000 7fdfca6cf700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfa4023630 0x7fdfa4025ac0 crc :-1 s=READY pgs=435 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:45.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.221+0000 7fdfcc132700 1 -- 172.21.15.16:0/1668857199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc4087280 con 0x7fdfc4088b50 2019-12-05T06:35:45.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.225+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=THROTTLE_DONE pgs=1009 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.225+0000 7fdfc2ffd700 1 -- 172.21.15.16:0/1668857199 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdfb8007600 con 0x7fdfc4088b50 2019-12-05T06:35:45.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.427+0000 7fdfcc132700 1 -- 172.21.15.16:0/1668857199 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdfc4087280 con 0x7fdfc4088b50 2019-12-05T06:35:45.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.427+0000 7fdfc9ece700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=THROTTLE_DONE pgs=1009 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.428+0000 7fdfc2ffd700 1 -- 172.21.15.16:0/1668857199 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fdfb80055d0 con 0x7fdfc4088b50 2019-12-05T06:35:45.426 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 127 2019-12-05T06:35:45.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:35:45.427 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:45.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.430+0000 7fdfcc132700 1 -- 172.21.15.16:0/1668857199 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfa4023630 msgr2=0x7fdfa4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.430+0000 7fdfcc132700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfa4023630 0x7fdfa4025ac0 crc :-1 s=READY pgs=435 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.430+0000 7fdfcc132700 1 -- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 msgr2=0x7fdfc4088380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.430+0000 7fdfcc132700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 0x7fdfc4088380 secure :-1 s=READY pgs=1009 cs=0 l=1 rx=0x7fdfb8010390 tx=0x7fdfb8010730).stop 2019-12-05T06:35:45.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.431+0000 7fdfcc132700 1 -- 172.21.15.16:0/1668857199 shutdown_connections 2019-12-05T06:35:45.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.431+0000 7fdfcc132700 1 --2- 172.21.15.16:0/1668857199 >> v2:172.21.15.76:3300/0 conn(0x7fdfc4152c20 0x7fdfc419b400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.431+0000 7fdfcc132700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdfa4023630 0x7fdfa4025ac0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.431+0000 7fdfcc132700 1 --2- 172.21.15.16:0/1668857199 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdfc4088b50 0x7fdfc4088380 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.431+0000 7fdfcc132700 1 -- 172.21.15.16:0/1668857199 shutdown_connections 2019-12-05T06:35:45.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.431+0000 7fdfcc132700 1 -- 172.21.15.16:0/1668857199 wait complete. 2019-12-05T06:35:45.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.431+0000 7fdfcc132700 1 -- 172.21.15.16:0/1668857199 >> 172.21.15.16:0/1668857199 conn(0x7fdfc4098480 msgr2=0x7fdfc4071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:45.440 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:35:45.441 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:35:45.441 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:35:45.454 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:35:45.454 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:45.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.611+0000 7f6ccbe23700 1 Processor -- start 2019-12-05T06:35:45.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.611+0000 7f6ccbe23700 1 -- start start 2019-12-05T06:35:45.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.612+0000 7f6ccbe23700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 0x7f6cc408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:45.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.612+0000 7f6ccbe23700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6cc408fef0 0x7f6cc408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:45.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.612+0000 7f6ccbe23700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc40958e0 con 0x7f6cc408ef20 2019-12-05T06:35:45.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.612+0000 7f6ccbe23700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc40a3f70 con 0x7f6cc408fef0 2019-12-05T06:35:45.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.612+0000 7f6ccbe23700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc408d830 con 0x7f6cc408c6d0 2019-12-05T06:35:45.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6ccabc1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6cc408ef20 0x7f6cc408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41874/0 (socket says 172.21.15.16:41874) 2019-12-05T06:35:45.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6ccabc1700 1 -- 172.21.15.16:0/774318874 learned_addr learned my addr 172.21.15.16:0/774318874 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:35:45.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6cca3c0700 1 --2- 172.21.15.16:0/774318874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 0x7f6cc408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6cc9bbf700 1 --2- 172.21.15.16:0/774318874 >> v2:172.21.15.76:3300/0 conn(0x7f6cc408fef0 0x7f6cc408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6cc93be700 1 -- 172.21.15.16:0/774318874 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2610981749 0 0) 0x7f6cb4001150 con 0x7f6cc408ef20 2019-12-05T06:35:45.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6cc93be700 1 -- 172.21.15.16:0/774318874 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cac002e80 con 0x7f6cc408ef20 2019-12-05T06:35:45.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6cca3c0700 1 -- 172.21.15.16:0/774318874 >> v2:172.21.15.76:3300/0 conn(0x7f6cc408fef0 msgr2=0x7f6cc408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:45.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6cca3c0700 1 --2- 172.21.15.16:0/774318874 >> v2:172.21.15.76:3300/0 conn(0x7f6cc408fef0 0x7f6cc408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:45.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6cca3c0700 1 -- 172.21.15.16:0/774318874 >> v1:172.21.15.16:6789/0 conn(0x7f6cc408ef20 legacy=0x7f6cc408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6cca3c0700 1 -- 172.21.15.16:0/774318874 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc408d540 con 0x7f6cc408c6d0 2019-12-05T06:35:45.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.613+0000 7f6cca3c0700 1 --2- 172.21.15.16:0/774318874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 0x7f6cc408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6cbc00b2d0 tx=0x7f6cbc00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.614+0000 7f6cca3c0700 1 --2- 172.21.15.16:0/774318874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 0x7f6cc408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6cbc00b2d0 tx=0x7f6cbc00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.614+0000 7f6cca3c0700 1 --2- 172.21.15.16:0/774318874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 0x7f6cc408cac0 secure :-1 s=READY pgs=238 cs=0 l=1 rx=0x7f6cbc00b2d0 tx=0x7f6cbc00b5d0).ready entity=mon.2 client_cookie=a70f583aa49828f6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:45.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.614+0000 7f6cca3c0700 1 --2- 172.21.15.16:0/774318874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 0x7f6cc408cac0 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7f6cbc00b2d0 tx=0x7f6cbc00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.614+0000 7f6cc93be700 1 -- 172.21.15.16:0/774318874 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cbc007590 con 0x7f6cc408c6d0 2019-12-05T06:35:45.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.614+0000 7f6cca3c0700 1 --2- 172.21.15.16:0/774318874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 0x7f6cc408cac0 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7f6cbc00b2d0 tx=0x7f6cbc00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.614+0000 7f6cca3c0700 1 --2- 172.21.15.16:0/774318874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 0x7f6cc408cac0 secure :-1 s=THROTTLE_DONE pgs=238 cs=0 l=1 rx=0x7f6cbc00b2d0 tx=0x7f6cbc00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.614+0000 7f6cc93be700 1 -- 172.21.15.16:0/774318874 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc007590 con 0x7f6cc408c6d0 2019-12-05T06:35:45.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.614+0000 7f6cc93be700 1 -- 172.21.15.16:0/774318874 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cbc009f20 con 0x7f6cc408c6d0 2019-12-05T06:35:45.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.614+0000 7f6ccbe23700 1 -- 172.21.15.16:0/774318874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 msgr2=0x7f6cc408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.614+0000 7f6ccbe23700 1 --2- 172.21.15.16:0/774318874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 0x7f6cc408cac0 secure :-1 s=READY pgs=238 cs=0 l=1 rx=0x7f6cbc00b2d0 tx=0x7f6cbc00b5d0).stop 2019-12-05T06:35:45.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.616+0000 7f6ccbe23700 1 -- 172.21.15.16:0/774318874 shutdown_connections 2019-12-05T06:35:45.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.616+0000 7f6ccbe23700 1 --2- 172.21.15.16:0/774318874 >> v2:172.21.15.76:3300/0 conn(0x7f6cc408fef0 0x7f6cc408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:45.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.616+0000 7f6ccbe23700 1 --2- 172.21.15.16:0/774318874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6cc408c6d0 0x7f6cc408cac0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.616+0000 7f6ccbe23700 1 -- 172.21.15.16:0/774318874 shutdown_connections 2019-12-05T06:35:45.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.616+0000 7f6ccbe23700 1 -- 172.21.15.16:0/774318874 wait complete. 2019-12-05T06:35:45.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.616+0000 7f6ccbe23700 1 -- 172.21.15.16:0/774318874 >> 172.21.15.16:0/774318874 conn(0x7f6cc4061940 msgr2=0x7f6cc4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:45.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.619+0000 7f6ccbe23700 1 Processor -- start 2019-12-05T06:35:45.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.619+0000 7f6ccbe23700 1 -- start start 2019-12-05T06:35:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.620+0000 7f6ccbe23700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6cc409d8b0 0x7f6cc409d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.620+0000 7f6ccbe23700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6cc4154b00 0x7f6cc4156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.620+0000 7f6ccbe23700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc413ba10 con 0x7f6cc40a0920 2019-12-05T06:35:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.620+0000 7f6ccbe23700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc413d6d0 con 0x7f6cc4154b00 2019-12-05T06:35:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.620+0000 7f6ccbe23700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cc413cca0 con 0x7f6cc409d8b0 2019-12-05T06:35:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.620+0000 7f6ccabc1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6cc4154b00 0x7f6cc4156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.620+0000 7f6ccabc1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6cc4154b00 0x7f6cc4156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56714/0 (socket says 172.21.15.16:56714) 2019-12-05T06:35:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.620+0000 7f6ccabc1700 1 -- 172.21.15.16:0/3204987766 learned_addr learned my addr 172.21.15.16:0/3204987766 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:35:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.620+0000 7f6cc9bbf700 1 --2- 172.21.15.16:0/3204987766 >> v2:172.21.15.76:3300/0 conn(0x7f6cc409d8b0 0x7f6cc409d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccabc1700 1 -- 172.21.15.16:0/3204987766 >> v2:172.21.15.76:3300/0 conn(0x7f6cc409d8b0 msgr2=0x7f6cc409d0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> v2:172.21.15.76:3300/0 conn(0x7f6cc409d8b0 0x7f6cc409d0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccabc1700 1 -- 172.21.15.16:0/3204987766 >> v1:172.21.15.76:6789/0 conn(0x7f6cc40a0920 legacy=0x7f6cc409c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccabc1700 1 -- 172.21.15.16:0/3204987766 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc413b510 con 0x7f6cc4154b00 2019-12-05T06:35:45.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> v2:172.21.15.16:3300/0 conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> v2:172.21.15.16:3300/0 conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=READY pgs=1011 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=THROTTLE_DONE pgs=1011 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=THROTTLE_DONE pgs=1011 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6cc2ffd700 1 -- 172.21.15.16:0/3204987766 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cb4000ef0 con 0x7f6cc4154b00 2019-12-05T06:35:45.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=THROTTLE_DONE pgs=1011 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6cc2ffd700 1 -- 172.21.15.16:0/3204987766 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cb4004240 con 0x7f6cc4154b00 2019-12-05T06:35:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6cc2ffd700 1 -- 172.21.15.16:0/3204987766 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cb4004910 con 0x7f6cc4154b00 2019-12-05T06:35:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccbe23700 1 -- 172.21.15.16:0/3204987766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc413b510 con 0x7f6cc4154b00 2019-12-05T06:35:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.621+0000 7f6ccbe23700 1 -- 172.21.15.16:0/3204987766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc413ba10 con 0x7f6cc4154b00 2019-12-05T06:35:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.622+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=THROTTLE_DONE pgs=1011 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.623+0000 7f6cc2ffd700 1 -- 172.21.15.16:0/3204987766 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6cb4004430 con 0x7f6cc4154b00 2019-12-05T06:35:45.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.623+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=THROTTLE_DONE pgs=1011 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.623+0000 7f6cc2ffd700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ca4023630 0x7f6ca4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.624+0000 7f6cc2ffd700 1 -- 172.21.15.16:0/3204987766 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f6cb4045e80 con 0x7f6cc4154b00 2019-12-05T06:35:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.624+0000 7f6cca3c0700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ca4023630 0x7f6ca4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.624+0000 7f6cca3c0700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ca4023630 0x7f6ca4025ac0 crc :-1 s=READY pgs=436 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:45.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.627+0000 7f6ccbe23700 1 -- 172.21.15.16:0/3204987766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc406cd00 con 0x7f6cc4154b00 2019-12-05T06:35:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.631+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=THROTTLE_DONE pgs=1011 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.631+0000 7f6cc2ffd700 1 -- 172.21.15.16:0/3204987766 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6cb4000ae0 con 0x7f6cc4154b00 2019-12-05T06:35:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.827+0000 7f6ccbe23700 1 -- 172.21.15.16:0/3204987766 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6cc406cd00 con 0x7f6cc4154b00 2019-12-05T06:35:45.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.827+0000 7f6ccabc1700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=THROTTLE_DONE pgs=1011 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.827+0000 7f6cc2ffd700 1 -- 172.21.15.16:0/3204987766 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f6cb40043f0 con 0x7f6cc4154b00 2019-12-05T06:35:45.827 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 127 2019-12-05T06:35:45.827 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:35:45.827 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:45.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.830+0000 7f6ccbe23700 1 -- 172.21.15.16:0/3204987766 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ca4023630 msgr2=0x7f6ca4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.830+0000 7f6ccbe23700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ca4023630 0x7f6ca4025ac0 crc :-1 s=READY pgs=436 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.830+0000 7f6ccbe23700 1 -- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 msgr2=0x7f6cc4156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.830+0000 7f6ccbe23700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 0x7f6cc4156f00 secure :-1 s=READY pgs=1011 cs=0 l=1 rx=0x7f6cb4010430 tx=0x7f6cb4010260).stop 2019-12-05T06:35:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.831+0000 7f6ccbe23700 1 -- 172.21.15.16:0/3204987766 shutdown_connections 2019-12-05T06:35:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.831+0000 7f6ccbe23700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ca4023630 0x7f6ca4025ac0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.831+0000 7f6ccbe23700 1 --2- 172.21.15.16:0/3204987766 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6cc4154b00 0x7f6cc4156f00 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.831+0000 7f6ccbe23700 1 --2- 172.21.15.16:0/3204987766 >> v2:172.21.15.76:3300/0 conn(0x7f6cc409d8b0 0x7f6cc409d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:45.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.831+0000 7f6ccbe23700 1 -- 172.21.15.16:0/3204987766 shutdown_connections 2019-12-05T06:35:45.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.831+0000 7f6ccbe23700 1 -- 172.21.15.16:0/3204987766 wait complete. 2019-12-05T06:35:45.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:45.831+0000 7f6ccbe23700 1 -- 172.21.15.16:0/3204987766 >> 172.21.15.16:0/3204987766 conn(0x7f6cc4130a60 msgr2=0x7f6cc4086470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:45.839 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:35:47.840 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:48.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.107+0000 7f90a73fd700 1 Processor -- start 2019-12-05T06:35:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.107+0000 7f90a73fd700 1 -- start start 2019-12-05T06:35:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.109+0000 7f90a73fd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a008c6d0 0x7f90a008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.109+0000 7f90a73fd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f90a008fef0 0x7f90a008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.109+0000 7f90a73fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a00958e0 con 0x7f90a008ef20 2019-12-05T06:35:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.109+0000 7f90a73fd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a00a3f70 con 0x7f90a008fef0 2019-12-05T06:35:48.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.109+0000 7f90a73fd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a008d830 con 0x7f90a008c6d0 2019-12-05T06:35:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.109+0000 7f90a5199700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f90a008fef0 0x7f90a008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.109+0000 7f90a599a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a008c6d0 0x7f90a008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.110+0000 7f90a619b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f90a008ef20 0x7f90a008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:34970/0 (socket says 172.21.15.16:34970) 2019-12-05T06:35:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.110+0000 7f90a5199700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f90a008fef0 0x7f90a008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:42820/0 (socket says 172.21.15.16:42820) 2019-12-05T06:35:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.110+0000 7f90a599a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a008c6d0 0x7f90a008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:60718/0 (socket says 172.21.15.16:60718) 2019-12-05T06:35:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.110+0000 7f90a619b700 1 -- 172.21.15.16:0/1294319314 learned_addr learned my addr 172.21.15.16:0/1294319314 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:35:48.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.110+0000 7f90a5199700 1 -- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a008c6d0 msgr2=0x7f90a008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.110+0000 7f90a5199700 1 --2- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a008c6d0 0x7f90a008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.110+0000 7f90a5199700 1 -- 172.21.15.16:0/1294319314 >> v1:172.21.15.76:6789/0 conn(0x7f90a008ef20 legacy=0x7f90a008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.111+0000 7f90a5199700 1 -- 172.21.15.16:0/1294319314 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a008d540 con 0x7f90a008fef0 2019-12-05T06:35:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.110+0000 7f90a4998700 1 -- 172.21.15.16:0/1294319314 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382717668 0 0) 0x7f9090001100 con 0x7f90a008ef20 2019-12-05T06:35:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.111+0000 7f90a599a700 1 --2- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a008c6d0 0x7f90a008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:35:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.111+0000 7f90a5199700 1 --2- 172.21.15.16:0/1294319314 >> v2:172.21.15.76:3300/0 conn(0x7f90a008fef0 0x7f90a008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f909400af50 tx=0x7f909400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.111+0000 7f90a5199700 1 --2- 172.21.15.16:0/1294319314 >> v2:172.21.15.76:3300/0 conn(0x7f90a008fef0 0x7f90a008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f909400af50 tx=0x7f909400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.111+0000 7f90a5199700 1 --2- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90a008fef0 0x7f90a008d000 secure :-1 s=READY pgs=718 cs=0 l=1 rx=0x7f909400af50 tx=0x7f909400b760).ready entity=mon.1 client_cookie=a88d324ff9923535 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.111+0000 7f90a5199700 1 --2- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90a008fef0 0x7f90a008d000 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f909400af50 tx=0x7f909400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.111+0000 7f90a5199700 1 --2- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90a008fef0 0x7f90a008d000 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f909400af50 tx=0x7f909400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.111+0000 7f90a4998700 1 -- 172.21.15.16:0/1294319314 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9094007210 con 0x7f90a008fef0 2019-12-05T06:35:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.111+0000 7f90a5199700 1 --2- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90a008fef0 0x7f90a008d000 secure :-1 s=THROTTLE_DONE pgs=718 cs=0 l=1 rx=0x7f909400af50 tx=0x7f909400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.112+0000 7f90a4998700 1 -- 172.21.15.16:0/1294319314 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90940099b0 con 0x7f90a008fef0 2019-12-05T06:35:48.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.112+0000 7f90a4998700 1 -- 172.21.15.16:0/1294319314 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f909400a080 con 0x7f90a008fef0 2019-12-05T06:35:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.112+0000 7f90a73fd700 1 -- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90a008fef0 msgr2=0x7f90a008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.112+0000 7f90a73fd700 1 --2- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90a008fef0 0x7f90a008d000 secure :-1 s=READY pgs=718 cs=0 l=1 rx=0x7f909400af50 tx=0x7f909400b760).stop 2019-12-05T06:35:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.113+0000 7f90a73fd700 1 -- 172.21.15.16:0/1294319314 shutdown_connections 2019-12-05T06:35:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.113+0000 7f90a73fd700 1 --2- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f90a008fef0 0x7f90a008d000 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.113+0000 7f90a73fd700 1 --2- 172.21.15.16:0/1294319314 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a008c6d0 0x7f90a008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.114+0000 7f90a73fd700 1 -- 172.21.15.16:0/1294319314 shutdown_connections 2019-12-05T06:35:48.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.114+0000 7f90a73fd700 1 -- 172.21.15.16:0/1294319314 wait complete. 2019-12-05T06:35:48.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.114+0000 7f90a73fd700 1 -- 172.21.15.16:0/1294319314 >> 172.21.15.16:0/1294319314 conn(0x7f90a0061940 msgr2=0x7f90a0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:48.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.119+0000 7f90a73fd700 1 Processor -- start 2019-12-05T06:35:48.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.119+0000 7f90a73fd700 1 -- start start 2019-12-05T06:35:48.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.121+0000 7f90a73fd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:48.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.121+0000 7f90a73fd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90a0153b50 0x7f90a0155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:48.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.121+0000 7f90a73fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90a0131bf0 con 0x7f90a01302f0 2019-12-05T06:35:48.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.121+0000 7f90a73fd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90a01296e0 con 0x7f90a0153b50 2019-12-05T06:35:48.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.121+0000 7f90a73fd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90a0128cb0 con 0x7f90a012b640 2019-12-05T06:35:48.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.121+0000 7f90a619b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f90a0153b50 0x7f90a0155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:48.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.121+0000 7f90a599a700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f90a01302f0 0x7f90a012a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:34972/0 (socket says 172.21.15.16:34972) 2019-12-05T06:35:48.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.121+0000 7f90a599a700 1 -- 172.21.15.16:0/3650185052 learned_addr learned my addr 172.21.15.16:0/3650185052 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:35:48.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.121+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.122+0000 7f909e7fc700 1 -- 172.21.15.16:0/3650185052 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 797013776 0 0) 0x7f909800df00 con 0x7f90a01302f0 2019-12-05T06:35:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.122+0000 7f909e7fc700 1 -- 172.21.15.16:0/3650185052 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9080002e80 con 0x7f90a01302f0 2019-12-05T06:35:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.122+0000 7f90a5199700 1 -- 172.21.15.16:0/3650185052 >> v2:172.21.15.16:3300/0 conn(0x7f90a0153b50 msgr2=0x7f90a0155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.122+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> v2:172.21.15.16:3300/0 conn(0x7f90a0153b50 0x7f90a0155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.122+0000 7f90a5199700 1 -- 172.21.15.16:0/3650185052 >> v1:172.21.15.76:6789/0 conn(0x7f90a01302f0 legacy=0x7f90a012a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:48.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.122+0000 7f90a5199700 1 -- 172.21.15.16:0/3650185052 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a0129d70 con 0x7f90a012b640 2019-12-05T06:35:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.122+0000 7f90a619b700 1 --2- 172.21.15.16:0/3650185052 >> v2:172.21.15.16:3300/0 conn(0x7f90a0153b50 0x7f90a0155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:35:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.122+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.122+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.122+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=READY pgs=239 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.123+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.123+0000 7f909e7fc700 1 -- 172.21.15.16:0/3650185052 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f909400aca0 con 0x7f90a012b640 2019-12-05T06:35:48.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.123+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.123+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.123+0000 7f909e7fc700 1 -- 172.21.15.16:0/3650185052 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9094009770 con 0x7f90a012b640 2019-12-05T06:35:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.123+0000 7f909e7fc700 1 -- 172.21.15.16:0/3650185052 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9094009ec0 con 0x7f90a012b640 2019-12-05T06:35:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.123+0000 7f90a73fd700 1 -- 172.21.15.16:0/3650185052 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a0129d70 con 0x7f90a012b640 2019-12-05T06:35:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.123+0000 7f90a73fd700 1 -- 172.21.15.16:0/3650185052 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a0129d70 con 0x7f90a012b640 2019-12-05T06:35:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.124+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.125+0000 7f909e7fc700 1 -- 172.21.15.16:0/3650185052 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9094009960 con 0x7f90a012b640 2019-12-05T06:35:48.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.125+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.125+0000 7f909e7fc700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9080023670 0x7f9080025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:48.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.125+0000 7f909e7fc700 1 -- 172.21.15.16:0/3650185052 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f9094089dc0 con 0x7f90a012b640 2019-12-05T06:35:48.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.125+0000 7f90a599a700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9080023670 0x7f9080025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:48.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.126+0000 7f90a599a700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9080023670 0x7f9080025ae0 crc :-1 s=READY pgs=437 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:48.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.138+0000 7f90a73fd700 1 -- 172.21.15.16:0/3650185052 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a0129d70 con 0x7f90a012b640 2019-12-05T06:35:48.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.147+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.147+0000 7f909e7fc700 1 -- 172.21.15.16:0/3650185052 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9094000940 con 0x7f90a012b640 2019-12-05T06:35:48.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.412+0000 7f90a73fd700 1 -- 172.21.15.16:0/3650185052 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90a0129d70 con 0x7f90a012b640 2019-12-05T06:35:48.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.413+0000 7f90a5199700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=THROTTLE_DONE pgs=239 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:48.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.413+0000 7f909e7fc700 1 -- 172.21.15.16:0/3650185052 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9094009940 con 0x7f90a012b640 2019-12-05T06:35:48.414 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 127 2019-12-05T06:35:48.414 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:35:48.414 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.418+0000 7f90a73fd700 1 -- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9080023670 msgr2=0x7f9080025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.418+0000 7f90a73fd700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9080023670 0x7f9080025ae0 crc :-1 s=READY pgs=437 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.418+0000 7f90a73fd700 1 -- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 msgr2=0x7f90a012ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:48.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.418+0000 7f90a73fd700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 secure :-1 s=READY pgs=239 cs=0 l=1 rx=0x7f90940543b0 tx=0x7f9094054750).stop 2019-12-05T06:35:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.418+0000 7f90a73fd700 1 -- 172.21.15.16:0/3650185052 shutdown_connections 2019-12-05T06:35:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.418+0000 7f90a73fd700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9080023670 0x7f9080025ae0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.418+0000 7f90a73fd700 1 --2- 172.21.15.16:0/3650185052 >> v2:172.21.15.16:3300/0 conn(0x7f90a0153b50 0x7f90a0155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.418+0000 7f90a73fd700 1 --2- 172.21.15.16:0/3650185052 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f90a012b640 0x7f90a012ae70 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.419+0000 7f90a73fd700 1 -- 172.21.15.16:0/3650185052 shutdown_connections 2019-12-05T06:35:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.419+0000 7f90a73fd700 1 -- 172.21.15.16:0/3650185052 wait complete. 2019-12-05T06:35:48.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:48.419+0000 7f90a73fd700 1 -- 172.21.15.16:0/3650185052 >> 172.21.15.16:0/3650185052 conn(0x7f90a00a2850 msgr2=0x7f90a006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:48.428 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:35:50.430 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:50.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.732+0000 7fe73c72d700 1 Processor -- start 2019-12-05T06:35:50.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.732+0000 7fe73c72d700 1 -- start start 2019-12-05T06:35:50.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73c72d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe73408c6d0 0x7fe73408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:50.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73c72d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe73408fef0 0x7fe73408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:50.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73c72d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe73408ef20 0x7fe73408f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:50.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73c72d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73408df90 con 0x7fe73408c6d0 2019-12-05T06:35:50.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73c72d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe7340a3f70 con 0x7fe73408fef0 2019-12-05T06:35:50.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73c72d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe73408d830 con 0x7fe73408ef20 2019-12-05T06:35:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73acca700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe73408c6d0 0x7fe73408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73acca700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe73408c6d0 0x7fe73408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56744/0 (socket says 172.21.15.16:56744) 2019-12-05T06:35:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73acca700 1 -- 172.21.15.16:0/1407543785 learned_addr learned my addr 172.21.15.16:0/1407543785 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:35:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1407543785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe73408ef20 0x7fe73408f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.735+0000 7fe73a4c9700 1 --2- 172.21.15.16:0/1407543785 >> v2:172.21.15.76:3300/0 conn(0x7fe73408fef0 0x7fe73408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.736+0000 7fe73acca700 1 -- 172.21.15.16:0/1407543785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe73408ef20 msgr2=0x7fe73408f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:50.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.736+0000 7fe73acca700 1 --2- 172.21.15.16:0/1407543785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe73408ef20 0x7fe73408f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.736+0000 7fe73acca700 1 -- 172.21.15.16:0/1407543785 >> v2:172.21.15.76:3300/0 conn(0x7fe73408fef0 msgr2=0x7fe73408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.736+0000 7fe73acca700 1 --2- 172.21.15.16:0/1407543785 >> v2:172.21.15.76:3300/0 conn(0x7fe73408fef0 0x7fe73408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.736+0000 7fe73acca700 1 -- 172.21.15.16:0/1407543785 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73408d540 con 0x7fe73408c6d0 2019-12-05T06:35:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.736+0000 7fe73acca700 1 --2- 172.21.15.16:0/1407543785 >> v2:172.21.15.16:3300/0 conn(0x7fe73408c6d0 0x7fe73408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe72c00fa40 tx=0x7fe72c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.736+0000 7fe73acca700 1 --2- 172.21.15.16:0/1407543785 >> v2:172.21.15.16:3300/0 conn(0x7fe73408c6d0 0x7fe73408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe72c00fa40 tx=0x7fe72c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.736+0000 7fe73acca700 1 --2- 172.21.15.16:0/1407543785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe73408c6d0 0x7fe73408cac0 secure :-1 s=READY pgs=1012 cs=0 l=1 rx=0x7fe72c00fa40 tx=0x7fe72c00f8f0).ready entity=mon.0 client_cookie=c4a8cc14b93cd5a3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:50.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.737+0000 7fe73acca700 1 --2- 172.21.15.16:0/1407543785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe73408c6d0 0x7fe73408cac0 secure :-1 s=THROTTLE_DONE pgs=1012 cs=0 l=1 rx=0x7fe72c00fa40 tx=0x7fe72c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.737+0000 7fe73acca700 1 --2- 172.21.15.16:0/1407543785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe73408c6d0 0x7fe73408cac0 secure :-1 s=THROTTLE_DONE pgs=1012 cs=0 l=1 rx=0x7fe72c00fa40 tx=0x7fe72c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.737+0000 7fe73acca700 1 --2- 172.21.15.16:0/1407543785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe73408c6d0 0x7fe73408cac0 secure :-1 s=THROTTLE_DONE pgs=1012 cs=0 l=1 rx=0x7fe72c00fa40 tx=0x7fe72c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.737+0000 7fe739cc8700 1 -- 172.21.15.16:0/1407543785 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe72c000ce0 con 0x7fe73408c6d0 2019-12-05T06:35:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.737+0000 7fe739cc8700 1 -- 172.21.15.16:0/1407543785 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72c001010 con 0x7fe73408c6d0 2019-12-05T06:35:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.737+0000 7fe739cc8700 1 -- 172.21.15.16:0/1407543785 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe72c001720 con 0x7fe73408c6d0 2019-12-05T06:35:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.737+0000 7fe73c72d700 1 -- 172.21.15.16:0/1407543785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe73408c6d0 msgr2=0x7fe73408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:50.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.737+0000 7fe73c72d700 1 --2- 172.21.15.16:0/1407543785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe73408c6d0 0x7fe73408cac0 secure :-1 s=READY pgs=1012 cs=0 l=1 rx=0x7fe72c00fa40 tx=0x7fe72c00f8f0).stop 2019-12-05T06:35:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.739+0000 7fe73c72d700 1 -- 172.21.15.16:0/1407543785 shutdown_connections 2019-12-05T06:35:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.739+0000 7fe73c72d700 1 --2- 172.21.15.16:0/1407543785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe73408ef20 0x7fe73408f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.739+0000 7fe73c72d700 1 --2- 172.21.15.16:0/1407543785 >> v2:172.21.15.76:3300/0 conn(0x7fe73408fef0 0x7fe73408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.739+0000 7fe73c72d700 1 --2- 172.21.15.16:0/1407543785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe73408c6d0 0x7fe73408cac0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.739+0000 7fe73c72d700 1 -- 172.21.15.16:0/1407543785 shutdown_connections 2019-12-05T06:35:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.739+0000 7fe73c72d700 1 -- 172.21.15.16:0/1407543785 wait complete. 2019-12-05T06:35:50.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.739+0000 7fe73c72d700 1 -- 172.21.15.16:0/1407543785 >> 172.21.15.16:0/1407543785 conn(0x7fe734061940 msgr2=0x7fe734096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:50.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.745+0000 7fe73c72d700 1 Processor -- start 2019-12-05T06:35:50.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.745+0000 7fe73c72d700 1 -- start start 2019-12-05T06:35:50.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.747+0000 7fe73c72d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe73409bbe0 0x7fe7341539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.747+0000 7fe73c72d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe734084f30 0x7fe734155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.747+0000 7fe73c72d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe734084b90 con 0x7fe734096e90 2019-12-05T06:35:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.747+0000 7fe73c72d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe73409a450 con 0x7fe734084f30 2019-12-05T06:35:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.747+0000 7fe73c72d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe734099a20 con 0x7fe73409bbe0 2019-12-05T06:35:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.747+0000 7fe73b4cb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe734084f30 0x7fe734155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.747+0000 7fe73b4cb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe734084f30 0x7fe734155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56752/0 (socket says 172.21.15.16:56752) 2019-12-05T06:35:50.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.747+0000 7fe73b4cb700 1 -- 172.21.15.16:0/1791992951 learned_addr learned my addr 172.21.15.16:0/1791992951 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:35:50.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.747+0000 7fe73a4c9700 1 --2- 172.21.15.16:0/1791992951 >> v2:172.21.15.76:3300/0 conn(0x7fe73409bbe0 0x7fe7341539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:50.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.748+0000 7fe73b4cb700 1 -- 172.21.15.16:0/1791992951 >> v2:172.21.15.76:3300/0 conn(0x7fe73409bbe0 msgr2=0x7fe7341539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:50.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.748+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> v2:172.21.15.76:3300/0 conn(0x7fe73409bbe0 0x7fe7341539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:50.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.748+0000 7fe73b4cb700 1 -- 172.21.15.16:0/1791992951 >> v1:172.21.15.76:6789/0 conn(0x7fe734096e90 legacy=0x7fe73409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:50.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.748+0000 7fe73b4cb700 1 -- 172.21.15.16:0/1791992951 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe73409aae0 con 0x7fe734084f30 2019-12-05T06:35:50.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.748+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> v2:172.21.15.16:3300/0 conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.748+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> v2:172.21.15.16:3300/0 conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.748+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=READY pgs=1013 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:50.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.748+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=THROTTLE_DONE pgs=1013 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.749+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=THROTTLE_DONE pgs=1013 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.749+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=THROTTLE_DONE pgs=1013 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.749+0000 7fe7337fe700 1 -- 172.21.15.16:0/1791992951 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe724000a70 con 0x7fe734084f30 2019-12-05T06:35:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.749+0000 7fe7337fe700 1 -- 172.21.15.16:0/1791992951 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe72400ecb0 con 0x7fe734084f30 2019-12-05T06:35:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.749+0000 7fe73c72d700 1 -- 172.21.15.16:0/1791992951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe73409aae0 con 0x7fe734084f30 2019-12-05T06:35:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.749+0000 7fe7337fe700 1 -- 172.21.15.16:0/1791992951 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe72400f380 con 0x7fe734084f30 2019-12-05T06:35:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.749+0000 7fe73c72d700 1 -- 172.21.15.16:0/1791992951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe73409aae0 con 0x7fe734084f30 2019-12-05T06:35:50.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.750+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=THROTTLE_DONE pgs=1013 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.751+0000 7fe7337fe700 1 -- 172.21.15.16:0/1791992951 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe72400eea0 con 0x7fe734084f30 2019-12-05T06:35:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.751+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=THROTTLE_DONE pgs=1013 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.751+0000 7fe7337fe700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe714023630 0x7fe714025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.751+0000 7fe7337fe700 1 -- 172.21.15.16:0/1791992951 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7fe724045dd0 con 0x7fe734084f30 2019-12-05T06:35:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.752+0000 7fe73acca700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe714023630 0x7fe714025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:50.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.752+0000 7fe73acca700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe714023630 0x7fe714025ac0 crc :-1 s=READY pgs=438 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:50.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.764+0000 7fe73c72d700 1 -- 172.21.15.16:0/1791992951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe73409aae0 con 0x7fe734084f30 2019-12-05T06:35:50.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.774+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=THROTTLE_DONE pgs=1013 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:50.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:50.774+0000 7fe7337fe700 1 -- 172.21.15.16:0/1791992951 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe724006530 con 0x7fe734084f30 2019-12-05T06:35:51.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.012+0000 7fe73c72d700 1 -- 172.21.15.16:0/1791992951 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe73409aae0 con 0x7fe734084f30 2019-12-05T06:35:51.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.013+0000 7fe73b4cb700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=THROTTLE_DONE pgs=1013 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:51.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.013+0000 7fe7337fe700 1 -- 172.21.15.16:0/1791992951 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe72400f750 con 0x7fe734084f30 2019-12-05T06:35:51.013 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 127 2019-12-05T06:35:51.013 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:35:51.014 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:51.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 -- 172.21.15.16:0/1791992951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe714023630 msgr2=0x7fe714025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:51.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe714023630 0x7fe714025ac0 crc :-1 s=READY pgs=438 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 -- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 msgr2=0x7fe734155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 0x7fe734155f30 secure :-1 s=READY pgs=1013 cs=0 l=1 rx=0x7fe724010440 tx=0x7fe724006f90).stop 2019-12-05T06:35:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 -- 172.21.15.16:0/1791992951 shutdown_connections 2019-12-05T06:35:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe714023630 0x7fe714025ac0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 --2- 172.21.15.16:0/1791992951 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe734084f30 0x7fe734155f30 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:51.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 --2- 172.21.15.16:0/1791992951 >> v2:172.21.15.76:3300/0 conn(0x7fe73409bbe0 0x7fe7341539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 -- 172.21.15.16:0/1791992951 shutdown_connections 2019-12-05T06:35:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 -- 172.21.15.16:0/1791992951 wait complete. 2019-12-05T06:35:51.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:51.016+0000 7fe73c72d700 1 -- 172.21.15.16:0/1791992951 >> 172.21.15.16:0/1791992951 conn(0x7fe7340714f0 msgr2=0x7fe73409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:51.026 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:35:53.027 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.281+0000 7f9a1546e700 1 Processor -- start 2019-12-05T06:35:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.281+0000 7f9a1546e700 1 -- start start 2019-12-05T06:35:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.283+0000 7f9a1546e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9a1008c6d0 0x7f9a1008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.283+0000 7f9a1546e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a1008f720 0x7f9a1008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:53.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.283+0000 7f9a1546e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a1008fb80 con 0x7f9a1008fef0 2019-12-05T06:35:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.283+0000 7f9a1546e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a1008df90 con 0x7f9a1008f720 2019-12-05T06:35:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.283+0000 7f9a1546e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a100a3f70 con 0x7f9a1008c6d0 2019-12-05T06:35:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.283+0000 7f9a0ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a1008f720 0x7f9a1008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.284+0000 7f9a0ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a1008f720 0x7f9a1008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56762/0 (socket says 172.21.15.16:56762) 2019-12-05T06:35:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.284+0000 7f9a0ffff700 1 -- 172.21.15.16:0/4222755179 learned_addr learned my addr 172.21.15.16:0/4222755179 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:35:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.284+0000 7f9a0f7fe700 1 --2- 172.21.15.16:0/4222755179 >> v2:172.21.15.76:3300/0 conn(0x7f9a1008c6d0 0x7f9a1008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:53.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.284+0000 7f9a0ffff700 1 -- 172.21.15.16:0/4222755179 >> v2:172.21.15.76:3300/0 conn(0x7f9a1008c6d0 msgr2=0x7f9a1008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.284+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/4222755179 >> v2:172.21.15.76:3300/0 conn(0x7f9a1008c6d0 0x7f9a1008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.284+0000 7f9a0ffff700 1 -- 172.21.15.16:0/4222755179 >> v1:172.21.15.76:6789/0 conn(0x7f9a1008fef0 legacy=0x7f9a1008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.284+0000 7f9a0ffff700 1 -- 172.21.15.16:0/4222755179 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a1008d540 con 0x7f9a1008f720 2019-12-05T06:35:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.284+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/4222755179 >> v2:172.21.15.16:3300/0 conn(0x7f9a1008f720 0x7f9a1008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9a0000f6b0 tx=0x7f9a0000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.284+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/4222755179 >> v2:172.21.15.16:3300/0 conn(0x7f9a1008f720 0x7f9a1008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9a0000f6b0 tx=0x7f9a0000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.284+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/4222755179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a1008f720 0x7f9a1008d000 secure :-1 s=READY pgs=1014 cs=0 l=1 rx=0x7f9a0000f6b0 tx=0x7f9a0000fa50).ready entity=mon.0 client_cookie=85b7b57928f9b4ae server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:53.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.285+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/4222755179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a1008f720 0x7f9a1008d000 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f9a0000f6b0 tx=0x7f9a0000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.285+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/4222755179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a1008f720 0x7f9a1008d000 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f9a0000f6b0 tx=0x7f9a0000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.285+0000 7f9a0e7fc700 1 -- 172.21.15.16:0/4222755179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a00000a70 con 0x7f9a1008f720 2019-12-05T06:35:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.285+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/4222755179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a1008f720 0x7f9a1008d000 secure :-1 s=THROTTLE_DONE pgs=1014 cs=0 l=1 rx=0x7f9a0000f6b0 tx=0x7f9a0000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.285+0000 7f9a0e7fc700 1 -- 172.21.15.16:0/4222755179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a000010e0 con 0x7f9a1008f720 2019-12-05T06:35:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.285+0000 7f9a0e7fc700 1 -- 172.21.15.16:0/4222755179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a000017b0 con 0x7f9a1008f720 2019-12-05T06:35:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.285+0000 7f9a1546e700 1 -- 172.21.15.16:0/4222755179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a1008f720 msgr2=0x7f9a1008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:53.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.285+0000 7f9a1546e700 1 --2- 172.21.15.16:0/4222755179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a1008f720 0x7f9a1008d000 secure :-1 s=READY pgs=1014 cs=0 l=1 rx=0x7f9a0000f6b0 tx=0x7f9a0000fa50).stop 2019-12-05T06:35:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.286+0000 7f9a1546e700 1 -- 172.21.15.16:0/4222755179 shutdown_connections 2019-12-05T06:35:53.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.286+0000 7f9a1546e700 1 --2- 172.21.15.16:0/4222755179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a1008f720 0x7f9a1008d000 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:53.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.286+0000 7f9a1546e700 1 --2- 172.21.15.16:0/4222755179 >> v2:172.21.15.76:3300/0 conn(0x7f9a1008c6d0 0x7f9a1008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.286+0000 7f9a1546e700 1 -- 172.21.15.16:0/4222755179 shutdown_connections 2019-12-05T06:35:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.286+0000 7f9a1546e700 1 -- 172.21.15.16:0/4222755179 wait complete. 2019-12-05T06:35:53.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.286+0000 7f9a1546e700 1 -- 172.21.15.16:0/4222755179 >> 172.21.15.16:0/4222755179 conn(0x7f9a10061940 msgr2=0x7f9a10096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.291+0000 7f9a1546e700 1 Processor -- start 2019-12-05T06:35:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.292+0000 7f9a1546e700 1 -- start start 2019-12-05T06:35:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.293+0000 7f9a1546e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a100871e0 0x7f9a1009c680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.293+0000 7f9a1546e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a10087640 con 0x7f9a100879b0 2019-12-05T06:35:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.293+0000 7f9a1546e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9a10153170 con 0x7f9a1014ca10 2019-12-05T06:35:53.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.293+0000 7f9a1546e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9a10085a50 con 0x7f9a100871e0 2019-12-05T06:35:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.293+0000 7f9a0ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9a100871e0 0x7f9a1009c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.293+0000 7f9a0effd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9a100879b0 0x7f9a1009be80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41936/0 (socket says 172.21.15.16:41936) 2019-12-05T06:35:53.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.293+0000 7f9a0effd700 1 -- 172.21.15.16:0/1195330042 learned_addr learned my addr 172.21.15.16:0/1195330042 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:35:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.294+0000 7f9a07fff700 1 -- 172.21.15.16:0/1195330042 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522541948 0 0) 0x7f99f80014d0 con 0x7f9a100879b0 2019-12-05T06:35:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.294+0000 7f9a0ffff700 1 -- 172.21.15.16:0/1195330042 >> v1:172.21.15.76:6789/0 conn(0x7f9a1014ca10 legacy=0x7f9a100869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.294+0000 7f9a0ffff700 1 -- 172.21.15.16:0/1195330042 >> v1:172.21.15.16:6789/0 conn(0x7f9a100879b0 legacy=0x7f9a1009be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.294+0000 7f9a0ffff700 1 -- 172.21.15.16:0/1195330042 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a100860e0 con 0x7f9a100871e0 2019-12-05T06:35:53.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.294+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/1195330042 >> v2:172.21.15.16:3300/0 conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.294+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/1195330042 >> v2:172.21.15.16:3300/0 conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.294+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=READY pgs=1016 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.295+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=THROTTLE_DONE pgs=1016 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.295+0000 7f9a07fff700 1 -- 172.21.15.16:0/1195330042 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a00008180 con 0x7f9a100871e0 2019-12-05T06:35:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.295+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=THROTTLE_DONE pgs=1016 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.295+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=THROTTLE_DONE pgs=1016 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.295+0000 7f9a07fff700 1 -- 172.21.15.16:0/1195330042 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a00007c90 con 0x7f9a100871e0 2019-12-05T06:35:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.295+0000 7f9a07fff700 1 -- 172.21.15.16:0/1195330042 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9a00001730 con 0x7f9a100871e0 2019-12-05T06:35:53.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.295+0000 7f9a1546e700 1 -- 172.21.15.16:0/1195330042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a100860e0 con 0x7f9a100871e0 2019-12-05T06:35:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.295+0000 7f9a1546e700 1 -- 172.21.15.16:0/1195330042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a100860e0 con 0x7f9a100871e0 2019-12-05T06:35:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.296+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=THROTTLE_DONE pgs=1016 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.297+0000 7f9a07fff700 1 -- 172.21.15.16:0/1195330042 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9a00007b50 con 0x7f9a100871e0 2019-12-05T06:35:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.297+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=THROTTLE_DONE pgs=1016 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.297+0000 7f9a07fff700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f99f4023630 0x7f99f4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.297+0000 7f9a07fff700 1 -- 172.21.15.16:0/1195330042 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7f9a00045de0 con 0x7f9a100871e0 2019-12-05T06:35:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.297+0000 7f9a0f7fe700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f99f4023630 0x7f99f4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:53.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.298+0000 7f9a0f7fe700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f99f4023630 0x7f99f4025ac0 crc :-1 s=READY pgs=439 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:53.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.313+0000 7f9a1546e700 1 -- 172.21.15.16:0/1195330042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a100860e0 con 0x7f9a100871e0 2019-12-05T06:35:53.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.320+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=THROTTLE_DONE pgs=1016 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.321+0000 7f9a07fff700 1 -- 172.21.15.16:0/1195330042 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9a0002ae00 con 0x7f9a100871e0 2019-12-05T06:35:53.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.546+0000 7f9a1546e700 1 -- 172.21.15.16:0/1195330042 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9a100860e0 con 0x7f9a100871e0 2019-12-05T06:35:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.546+0000 7f9a0ffff700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=THROTTLE_DONE pgs=1016 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.546+0000 7f9a07fff700 1 -- 172.21.15.16:0/1195330042 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9a00007b50 con 0x7f9a100871e0 2019-12-05T06:35:53.546 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 127 2019-12-05T06:35:53.547 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:35:53.547 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.549+0000 7f9a1546e700 1 -- 172.21.15.16:0/1195330042 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f99f4023630 msgr2=0x7f99f4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.549+0000 7f9a1546e700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f99f4023630 0x7f99f4025ac0 crc :-1 s=READY pgs=439 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:53.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.549+0000 7f9a1546e700 1 -- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 msgr2=0x7f9a1009c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.549+0000 7f9a1546e700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 0x7f9a1009c680 secure :-1 s=READY pgs=1016 cs=0 l=1 rx=0x7f9a00010370 tx=0x7f9a00010710).stop 2019-12-05T06:35:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.549+0000 7f9a1546e700 1 -- 172.21.15.16:0/1195330042 shutdown_connections 2019-12-05T06:35:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.549+0000 7f9a1546e700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f99f4023630 0x7f99f4025ac0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.549+0000 7f9a1546e700 1 --2- 172.21.15.16:0/1195330042 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9a100871e0 0x7f9a1009c680 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.550+0000 7f9a1546e700 1 -- 172.21.15.16:0/1195330042 shutdown_connections 2019-12-05T06:35:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.550+0000 7f9a1546e700 1 -- 172.21.15.16:0/1195330042 wait complete. 2019-12-05T06:35:53.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:53.550+0000 7f9a1546e700 1 -- 172.21.15.16:0/1195330042 >> 172.21.15.16:0/1195330042 conn(0x7f9a10098480 msgr2=0x7f9a10071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:53.566 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:35:55.567 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.818+0000 7fd45cca7700 1 Processor -- start 2019-12-05T06:35:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.818+0000 7fd45cca7700 1 -- start start 2019-12-05T06:35:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.819+0000 7fd45cca7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd45808d2c0 0x7fd45808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:55.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.819+0000 7fd45cca7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd45808fef0 0x7fd45808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.819+0000 7fd45cca7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4580958e0 con 0x7fd45808c6d0 2019-12-05T06:35:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.819+0000 7fd45cca7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45808df90 con 0x7fd45808fef0 2019-12-05T06:35:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.819+0000 7fd45cca7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4580a3f70 con 0x7fd45808d2c0 2019-12-05T06:35:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.820+0000 7fd45759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd45808fef0 0x7fd45808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.820+0000 7fd45759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd45808fef0 0x7fd45808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56776/0 (socket says 172.21.15.16:56776) 2019-12-05T06:35:55.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.820+0000 7fd45759e700 1 -- 172.21.15.16:0/1923552231 learned_addr learned my addr 172.21.15.16:0/1923552231 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:35:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.820+0000 7fd456d9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd45808c6d0 0x7fd45808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:35012/0 (socket says 172.21.15.16:35012) 2019-12-05T06:35:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.820+0000 7fd45659c700 1 --2- 172.21.15.16:0/1923552231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd45808d2c0 0x7fd45808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd455d9b700 1 -- 172.21.15.16:0/1923552231 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789651559 0 0) 0x7fd450001400 con 0x7fd45808c6d0 2019-12-05T06:35:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd45759e700 1 -- 172.21.15.16:0/1923552231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd45808d2c0 msgr2=0x7fd45808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:35:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd45759e700 1 --2- 172.21.15.16:0/1923552231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd45808d2c0 0x7fd45808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:55.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd45759e700 1 -- 172.21.15.16:0/1923552231 >> v1:172.21.15.76:6789/0 conn(0x7fd45808c6d0 legacy=0x7fd45808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd45759e700 1 -- 172.21.15.16:0/1923552231 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45808e620 con 0x7fd45808fef0 2019-12-05T06:35:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd45659c700 1 --2- 172.21.15.16:0/1923552231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd45808d2c0 0x7fd45808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:35:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd45759e700 1 --2- 172.21.15.16:0/1923552231 >> v2:172.21.15.16:3300/0 conn(0x7fd45808fef0 0x7fd45808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd44800f730 tx=0x7fd44800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd45759e700 1 --2- 172.21.15.16:0/1923552231 >> v2:172.21.15.16:3300/0 conn(0x7fd45808fef0 0x7fd45808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd44800f730 tx=0x7fd44800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd45759e700 1 --2- 172.21.15.16:0/1923552231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45808fef0 0x7fd45808f460 secure :-1 s=READY pgs=1017 cs=0 l=1 rx=0x7fd44800f730 tx=0x7fd44800fad0).ready entity=mon.0 client_cookie=57907832bd463829 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd45759e700 1 --2- 172.21.15.16:0/1923552231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45808fef0 0x7fd45808f460 secure :-1 s=THROTTLE_DONE pgs=1017 cs=0 l=1 rx=0x7fd44800f730 tx=0x7fd44800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.821+0000 7fd455d9b700 1 -- 172.21.15.16:0/1923552231 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd448000b90 con 0x7fd45808fef0 2019-12-05T06:35:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.822+0000 7fd45759e700 1 --2- 172.21.15.16:0/1923552231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45808fef0 0x7fd45808f460 secure :-1 s=THROTTLE_DONE pgs=1017 cs=0 l=1 rx=0x7fd44800f730 tx=0x7fd44800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.822+0000 7fd45759e700 1 --2- 172.21.15.16:0/1923552231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45808fef0 0x7fd45808f460 secure :-1 s=THROTTLE_DONE pgs=1017 cs=0 l=1 rx=0x7fd44800f730 tx=0x7fd44800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.822+0000 7fd455d9b700 1 -- 172.21.15.16:0/1923552231 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4480010e0 con 0x7fd45808fef0 2019-12-05T06:35:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.822+0000 7fd455d9b700 1 -- 172.21.15.16:0/1923552231 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd448000b90 con 0x7fd45808fef0 2019-12-05T06:35:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.822+0000 7fd45cca7700 1 -- 172.21.15.16:0/1923552231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45808fef0 msgr2=0x7fd45808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.822+0000 7fd45cca7700 1 --2- 172.21.15.16:0/1923552231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45808fef0 0x7fd45808f460 secure :-1 s=READY pgs=1017 cs=0 l=1 rx=0x7fd44800f730 tx=0x7fd44800fad0).stop 2019-12-05T06:35:55.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.824+0000 7fd45cca7700 1 -- 172.21.15.16:0/1923552231 shutdown_connections 2019-12-05T06:35:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.824+0000 7fd45cca7700 1 --2- 172.21.15.16:0/1923552231 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45808fef0 0x7fd45808f460 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.824+0000 7fd45cca7700 1 --2- 172.21.15.16:0/1923552231 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd45808d2c0 0x7fd45808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:35:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.824+0000 7fd45cca7700 1 -- 172.21.15.16:0/1923552231 shutdown_connections 2019-12-05T06:35:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.824+0000 7fd45cca7700 1 -- 172.21.15.16:0/1923552231 wait complete. 2019-12-05T06:35:55.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.824+0000 7fd45cca7700 1 -- 172.21.15.16:0/1923552231 >> 172.21.15.16:0/1923552231 conn(0x7fd458061940 msgr2=0x7fd458096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.829+0000 7fd45cca7700 1 Processor -- start 2019-12-05T06:35:55.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.829+0000 7fd45cca7700 1 -- start start 2019-12-05T06:35:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.831+0000 7fd45cca7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd45814d400 0x7fd45814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.831+0000 7fd45cca7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd458132d00 0x7fd458131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.831+0000 7fd45cca7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd458098da0 con 0x7fd458132270 2019-12-05T06:35:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.831+0000 7fd45cca7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45809bb50 con 0x7fd45814d400 2019-12-05T06:35:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.831+0000 7fd45cca7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd45809aa60 con 0x7fd458132d00 2019-12-05T06:35:55.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.831+0000 7fd456d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd45814d400 0x7fd45814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:55.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd456d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd45814d400 0x7fd45814d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56780/0 (socket says 172.21.15.16:56780) 2019-12-05T06:35:55.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd45759e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd458132270 0x7fd45806dc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41952/0 (socket says 172.21.15.16:41952) 2019-12-05T06:35:55.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd456d9d700 1 -- 172.21.15.16:0/3763506628 learned_addr learned my addr 172.21.15.16:0/3763506628 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:35:55.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd45659c700 1 --2- 172.21.15.16:0/3763506628 >> v2:172.21.15.76:3300/0 conn(0x7fd458132d00 0x7fd458131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:55.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd456d9d700 1 -- 172.21.15.16:0/3763506628 >> v2:172.21.15.76:3300/0 conn(0x7fd458132d00 msgr2=0x7fd458131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:55.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> v2:172.21.15.76:3300/0 conn(0x7fd458132d00 0x7fd458131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:55.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd456d9d700 1 -- 172.21.15.16:0/3763506628 >> v1:172.21.15.16:6789/0 conn(0x7fd458132270 legacy=0x7fd45806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:55.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd456d9d700 1 -- 172.21.15.16:0/3763506628 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd45809b0f0 con 0x7fd45814d400 2019-12-05T06:35:55.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd44f7fe700 1 -- 172.21.15.16:0/3763506628 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465819022 0 0) 0x7fd448000b90 con 0x7fd458132270 2019-12-05T06:35:55.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> v2:172.21.15.16:3300/0 conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> v2:172.21.15.16:3300/0 conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.832+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=READY pgs=1019 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:55.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.833+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.833+0000 7fd44f7fe700 1 -- 172.21.15.16:0/3763506628 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd450004540 con 0x7fd45814d400 2019-12-05T06:35:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.833+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.833+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.833+0000 7fd44f7fe700 1 -- 172.21.15.16:0/3763506628 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd450000fe0 con 0x7fd45814d400 2019-12-05T06:35:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.833+0000 7fd45cca7700 1 -- 172.21.15.16:0/3763506628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45809b0f0 con 0x7fd45814d400 2019-12-05T06:35:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.833+0000 7fd44f7fe700 1 -- 172.21.15.16:0/3763506628 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd450004b20 con 0x7fd45814d400 2019-12-05T06:35:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.833+0000 7fd45cca7700 1 -- 172.21.15.16:0/3763506628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd45809b0f0 con 0x7fd45814d400 2019-12-05T06:35:55.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.834+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.835+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.835+0000 7fd44f7fe700 1 -- 172.21.15.16:0/3763506628 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd450000fe0 con 0x7fd45814d400 2019-12-05T06:35:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.836+0000 7fd44f7fe700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd43c023630 0x7fd43c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.836+0000 7fd44f7fe700 1 -- 172.21.15.16:0/3763506628 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7fd450046170 con 0x7fd45814d400 2019-12-05T06:35:55.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.836+0000 7fd45659c700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd43c023630 0x7fd43c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:55.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.837+0000 7fd45659c700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd43c023630 0x7fd43c025ac0 crc :-1 s=READY pgs=440 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:55.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.850+0000 7fd45cca7700 1 -- 172.21.15.16:0/3763506628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45809b0f0 con 0x7fd45814d400 2019-12-05T06:35:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.858+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:55.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:55.858+0000 7fd44f7fe700 1 -- 172.21.15.16:0/3763506628 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd450001680 con 0x7fd45814d400 2019-12-05T06:35:56.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.121+0000 7fd45cca7700 1 -- 172.21.15.16:0/3763506628 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd45809b0f0 con 0x7fd45814d400 2019-12-05T06:35:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.122+0000 7fd456d9d700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=THROTTLE_DONE pgs=1019 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:56.122 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.122+0000 7fd44f7fe700 1 -- 172.21.15.16:0/3763506628 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fd450000fe0 con 0x7fd45814d400 2019-12-05T06:35:56.124 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 127 2019-12-05T06:35:56.125 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:35:56.125 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:56.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.128+0000 7fd45cca7700 1 -- 172.21.15.16:0/3763506628 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd43c023630 msgr2=0x7fd43c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.128+0000 7fd45cca7700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd43c023630 0x7fd43c025ac0 crc :-1 s=READY pgs=440 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.128+0000 7fd45cca7700 1 -- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 msgr2=0x7fd45814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.128+0000 7fd45cca7700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 0x7fd45814d7f0 secure :-1 s=READY pgs=1019 cs=0 l=1 rx=0x7fd4500107c0 tx=0x7fd4500105f0).stop 2019-12-05T06:35:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.129+0000 7fd45cca7700 1 -- 172.21.15.16:0/3763506628 shutdown_connections 2019-12-05T06:35:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.129+0000 7fd45cca7700 1 --2- 172.21.15.16:0/3763506628 >> v2:172.21.15.76:3300/0 conn(0x7fd458132d00 0x7fd458131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:56.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.129+0000 7fd45cca7700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd43c023630 0x7fd43c025ac0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:56.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.129+0000 7fd45cca7700 1 --2- 172.21.15.16:0/3763506628 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd45814d400 0x7fd45814d7f0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:56.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.129+0000 7fd45cca7700 1 -- 172.21.15.16:0/3763506628 shutdown_connections 2019-12-05T06:35:56.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.129+0000 7fd45cca7700 1 -- 172.21.15.16:0/3763506628 wait complete. 2019-12-05T06:35:56.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:56.129+0000 7fd45cca7700 1 -- 172.21.15.16:0/3763506628 >> 172.21.15.16:0/3763506628 conn(0x7fd4580714f0 msgr2=0x7fd45809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:56.144 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:35:58.146 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:35:58.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.393+0000 7facaf8db700 1 Processor -- start 2019-12-05T06:35:58.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.393+0000 7facaf8db700 1 -- start start 2019-12-05T06:35:58.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.394+0000 7facaf8db700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 0x7faca808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:35:58.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.395+0000 7facaf8db700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca80958e0 con 0x7faca808f720 2019-12-05T06:35:58.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.395+0000 7facaf8db700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca80629e0 con 0x7faca808fef0 2019-12-05T06:35:58.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.395+0000 7facaf8db700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faca80a3f70 con 0x7faca808c6d0 2019-12-05T06:35:58.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.395+0000 7facae679700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faca808f720 0x7faca81533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41964/0 (socket says 172.21.15.16:41964) 2019-12-05T06:35:58.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.395+0000 7facae679700 1 -- 172.21.15.16:0/1553602323 learned_addr learned my addr 172.21.15.16:0/1553602323 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:35:58.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.395+0000 7facade78700 1 --2- 172.21.15.16:0/1553602323 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 0x7faca808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:58.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.395+0000 7facace76700 1 -- 172.21.15.16:0/1553602323 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4202953155 0 0) 0x7fac98001150 con 0x7faca808f720 2019-12-05T06:35:58.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facace76700 1 -- 172.21.15.16:0/1553602323 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac90002e80 con 0x7faca808f720 2019-12-05T06:35:58.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facace76700 1 -- 172.21.15.16:0/1553602323 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2162570281 0 0) 0x7fac9c001080 con 0x7faca808fef0 2019-12-05T06:35:58.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facade78700 1 -- 172.21.15.16:0/1553602323 >> v1:172.21.15.76:6789/0 conn(0x7faca808fef0 legacy=0x7faca808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:58.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facade78700 1 -- 172.21.15.16:0/1553602323 >> v1:172.21.15.16:6789/0 conn(0x7faca808f720 legacy=0x7faca81533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:58.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facade78700 1 -- 172.21.15.16:0/1553602323 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca808fbc0 con 0x7faca808c6d0 2019-12-05T06:35:58.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facade78700 1 --2- 172.21.15.16:0/1553602323 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 0x7faca808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faca000b2d0 tx=0x7faca000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facade78700 1 --2- 172.21.15.16:0/1553602323 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 0x7faca808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faca000b2d0 tx=0x7faca000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facade78700 1 --2- 172.21.15.16:0/1553602323 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 0x7faca808cac0 secure :-1 s=READY pgs=240 cs=0 l=1 rx=0x7faca000b2d0 tx=0x7faca000bae0).ready entity=mon.2 client_cookie=c89c13bbdb3a04f6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:58.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facade78700 1 --2- 172.21.15.16:0/1553602323 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 0x7faca808cac0 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7faca000b2d0 tx=0x7faca000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facace76700 1 -- 172.21.15.16:0/1553602323 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faca0000ce0 con 0x7faca808c6d0 2019-12-05T06:35:58.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.396+0000 7facade78700 1 --2- 172.21.15.16:0/1553602323 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 0x7faca808cac0 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7faca000b2d0 tx=0x7faca000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.397+0000 7facade78700 1 --2- 172.21.15.16:0/1553602323 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 0x7faca808cac0 secure :-1 s=THROTTLE_DONE pgs=240 cs=0 l=1 rx=0x7faca000b2d0 tx=0x7faca000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.397+0000 7facace76700 1 -- 172.21.15.16:0/1553602323 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faca0009be0 con 0x7faca808c6d0 2019-12-05T06:35:58.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.397+0000 7facace76700 1 -- 172.21.15.16:0/1553602323 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faca000a2b0 con 0x7faca808c6d0 2019-12-05T06:35:58.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.397+0000 7facaf8db700 1 -- 172.21.15.16:0/1553602323 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 msgr2=0x7faca808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:58.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.397+0000 7facaf8db700 1 --2- 172.21.15.16:0/1553602323 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 0x7faca808cac0 secure :-1 s=READY pgs=240 cs=0 l=1 rx=0x7faca000b2d0 tx=0x7faca000bae0).stop 2019-12-05T06:35:58.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.398+0000 7facaf8db700 1 -- 172.21.15.16:0/1553602323 shutdown_connections 2019-12-05T06:35:58.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.398+0000 7facaf8db700 1 --2- 172.21.15.16:0/1553602323 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faca808c6d0 0x7faca808cac0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:58.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.398+0000 7facaf8db700 1 -- 172.21.15.16:0/1553602323 shutdown_connections 2019-12-05T06:35:58.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.399+0000 7facaf8db700 1 -- 172.21.15.16:0/1553602323 wait complete. 2019-12-05T06:35:58.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.399+0000 7facaf8db700 1 -- 172.21.15.16:0/1553602323 >> 172.21.15.16:0/1553602323 conn(0x7faca8061940 msgr2=0x7faca8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:58.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.403+0000 7facaf8db700 1 Processor -- start 2019-12-05T06:35:58.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.403+0000 7facaf8db700 1 -- start start 2019-12-05T06:35:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.405+0000 7facaf8db700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7faca81302f0 0x7faca812a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.405+0000 7facaf8db700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faca812b640 0x7faca812abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.405+0000 7facaf8db700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faca8131bf0 con 0x7faca812b0f0 2019-12-05T06:35:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.405+0000 7facaf8db700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca81296e0 con 0x7faca812b640 2019-12-05T06:35:58.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.405+0000 7facaf8db700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faca8128cb0 con 0x7faca81302f0 2019-12-05T06:35:58.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.405+0000 7facad677700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faca812b640 0x7faca812abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:58.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.405+0000 7facad677700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faca812b640 0x7faca812abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56802/0 (socket says 172.21.15.16:56802) 2019-12-05T06:35:58.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.405+0000 7facae679700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faca812b0f0 0x7faca81306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41970/0 (socket says 172.21.15.16:41970) 2019-12-05T06:35:58.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.405+0000 7facad677700 1 -- 172.21.15.16:0/2424662310 learned_addr learned my addr 172.21.15.16:0/2424662310 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:35:58.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.405+0000 7facade78700 1 --2- 172.21.15.16:0/2424662310 >> v2:172.21.15.76:3300/0 conn(0x7faca81302f0 0x7faca812a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:58.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facad677700 1 -- 172.21.15.16:0/2424662310 >> v2:172.21.15.76:3300/0 conn(0x7faca81302f0 msgr2=0x7faca812a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:58.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> v2:172.21.15.76:3300/0 conn(0x7faca81302f0 0x7faca812a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:58.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facad677700 1 -- 172.21.15.16:0/2424662310 >> v1:172.21.15.16:6789/0 conn(0x7faca812b0f0 legacy=0x7faca81306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:58.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facad677700 1 -- 172.21.15.16:0/2424662310 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faca8129d70 con 0x7faca812b640 2019-12-05T06:35:58.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7faca67fc700 1 -- 172.21.15.16:0/2424662310 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 429467556 0 0) 0x7fac98000f40 con 0x7faca812b0f0 2019-12-05T06:35:58.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> v2:172.21.15.16:3300/0 conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> v2:172.21.15.16:3300/0 conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=READY pgs=1022 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:58.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=THROTTLE_DONE pgs=1022 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7faca67fc700 1 -- 172.21.15.16:0/2424662310 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac9c000b50 con 0x7faca812b640 2019-12-05T06:35:58.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=THROTTLE_DONE pgs=1022 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=THROTTLE_DONE pgs=1022 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7faca67fc700 1 -- 172.21.15.16:0/2424662310 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac9c009a30 con 0x7faca812b640 2019-12-05T06:35:58.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7facaf8db700 1 -- 172.21.15.16:0/2424662310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faca8129d70 con 0x7faca812b640 2019-12-05T06:35:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.406+0000 7faca67fc700 1 -- 172.21.15.16:0/2424662310 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac9c00a100 con 0x7faca812b640 2019-12-05T06:35:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.407+0000 7facaf8db700 1 -- 172.21.15.16:0/2424662310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faca8129d70 con 0x7faca812b640 2019-12-05T06:35:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.407+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=THROTTLE_DONE pgs=1022 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.408+0000 7faca67fc700 1 -- 172.21.15.16:0/2424662310 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fac9c009630 con 0x7faca812b640 2019-12-05T06:35:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.408+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=THROTTLE_DONE pgs=1022 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.409+0000 7faca67fc700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac88023630 0x7fac88025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:35:58.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.409+0000 7faca67fc700 1 -- 172.21.15.16:0/2424662310 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7fac9c045e20 con 0x7faca812b640 2019-12-05T06:35:58.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.409+0000 7facade78700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac88023630 0x7fac88025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:35:58.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.410+0000 7facade78700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac88023630 0x7fac88025ac0 crc :-1 s=READY pgs=441 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:35:58.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.425+0000 7facaf8db700 1 -- 172.21.15.16:0/2424662310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faca8129d70 con 0x7faca812b640 2019-12-05T06:35:58.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.432+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=THROTTLE_DONE pgs=1022 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.433+0000 7faca67fc700 1 -- 172.21.15.16:0/2424662310 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fac9c031ec0 con 0x7faca812b640 2019-12-05T06:35:58.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.686+0000 7facaf8db700 1 -- 172.21.15.16:0/2424662310 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faca8129d70 con 0x7faca812b640 2019-12-05T06:35:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.686+0000 7facad677700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=THROTTLE_DONE pgs=1022 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:35:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.686+0000 7faca67fc700 1 -- 172.21.15.16:0/2424662310 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fac9c031ec0 con 0x7faca812b640 2019-12-05T06:35:58.686 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 127 2019-12-05T06:35:58.686 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:35:58.687 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:35:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.689+0000 7facaf8db700 1 -- 172.21.15.16:0/2424662310 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac88023630 msgr2=0x7fac88025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.689+0000 7facaf8db700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac88023630 0x7fac88025ac0 crc :-1 s=READY pgs=441 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.689+0000 7facaf8db700 1 -- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 msgr2=0x7faca812abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:35:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.689+0000 7facaf8db700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 0x7faca812abb0 secure :-1 s=READY pgs=1022 cs=0 l=1 rx=0x7fac9c010450 tx=0x7fac9c00b390).stop 2019-12-05T06:35:58.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.689+0000 7facaf8db700 1 -- 172.21.15.16:0/2424662310 shutdown_connections 2019-12-05T06:35:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.689+0000 7facaf8db700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fac88023630 0x7fac88025ac0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.689+0000 7facaf8db700 1 --2- 172.21.15.16:0/2424662310 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faca812b640 0x7faca812abb0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.689+0000 7facaf8db700 1 --2- 172.21.15.16:0/2424662310 >> v2:172.21.15.76:3300/0 conn(0x7faca81302f0 0x7faca812a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:35:58.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.690+0000 7facaf8db700 1 -- 172.21.15.16:0/2424662310 shutdown_connections 2019-12-05T06:35:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.690+0000 7facaf8db700 1 -- 172.21.15.16:0/2424662310 wait complete. 2019-12-05T06:35:58.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:35:58.690+0000 7facaf8db700 1 -- 172.21.15.16:0/2424662310 >> 172.21.15.16:0/2424662310 conn(0x7faca80a2850 msgr2=0x7faca806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:35:58.703 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:36:00.704 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:00.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.964+0000 7fbbaae67700 1 Processor -- start 2019-12-05T06:36:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.965+0000 7fbbaae67700 1 -- start start 2019-12-05T06:36:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.967+0000 7fbbaae67700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbba408c6d0 0x7fbba408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.967+0000 7fbbaae67700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbba408fef0 0x7fbba408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.967+0000 7fbbaae67700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba40958e0 con 0x7fbba408ef20 2019-12-05T06:36:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.967+0000 7fbbaae67700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba408df90 con 0x7fbba408c6d0 2019-12-05T06:36:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.967+0000 7fbbaae67700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbba40a3f70 con 0x7fbba408fef0 2019-12-05T06:36:00.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.967+0000 7fbba9c05700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbba408ef20 0x7fbba408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:35048/0 (socket says 172.21.15.16:35048) 2019-12-05T06:36:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.967+0000 7fbba9c05700 1 -- 172.21.15.16:0/153081842 learned_addr learned my addr 172.21.15.16:0/153081842 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.967+0000 7fbba9404700 1 --2- 172.21.15.16:0/153081842 >> v2:172.21.15.16:3300/0 conn(0x7fbba408c6d0 0x7fbba408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.967+0000 7fbba8c03700 1 --2- 172.21.15.16:0/153081842 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbba408fef0 0x7fbba408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.968+0000 7fbba3fff700 1 -- 172.21.15.16:0/153081842 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4179942440 0 0) 0x7fbb940009b0 con 0x7fbba408ef20 2019-12-05T06:36:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.968+0000 7fbba3fff700 1 -- 172.21.15.16:0/153081842 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb8c002e80 con 0x7fbba408ef20 2019-12-05T06:36:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.968+0000 7fbba9404700 1 -- 172.21.15.16:0/153081842 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbba408fef0 msgr2=0x7fbba408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:00.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.968+0000 7fbba9404700 1 --2- 172.21.15.16:0/153081842 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbba408fef0 0x7fbba408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:00.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.968+0000 7fbba9404700 1 -- 172.21.15.16:0/153081842 >> v1:172.21.15.76:6789/0 conn(0x7fbba408ef20 legacy=0x7fbba408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:00.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.968+0000 7fbba9404700 1 -- 172.21.15.16:0/153081842 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba408d540 con 0x7fbba408c6d0 2019-12-05T06:36:00.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.968+0000 7fbba8c03700 1 --2- 172.21.15.16:0/153081842 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbba408fef0 0x7fbba408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:36:00.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.968+0000 7fbba9404700 1 --2- 172.21.15.16:0/153081842 >> v2:172.21.15.16:3300/0 conn(0x7fbba408c6d0 0x7fbba408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbb9c00b2d0 tx=0x7fbb9c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.969+0000 7fbba9404700 1 --2- 172.21.15.16:0/153081842 >> v2:172.21.15.16:3300/0 conn(0x7fbba408c6d0 0x7fbba408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbb9c00b2d0 tx=0x7fbb9c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.969+0000 7fbba9404700 1 --2- 172.21.15.16:0/153081842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba408c6d0 0x7fbba408cac0 secure :-1 s=READY pgs=1023 cs=0 l=1 rx=0x7fbb9c00b2d0 tx=0x7fbb9c00bae0).ready entity=mon.0 client_cookie=2d3aa41a14ed6f35 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:00.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.969+0000 7fbba9404700 1 --2- 172.21.15.16:0/153081842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba408c6d0 0x7fbba408cac0 secure :-1 s=THROTTLE_DONE pgs=1023 cs=0 l=1 rx=0x7fbb9c00b2d0 tx=0x7fbb9c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.969+0000 7fbba3fff700 1 -- 172.21.15.16:0/153081842 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb9c007590 con 0x7fbba408c6d0 2019-12-05T06:36:00.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.969+0000 7fbba9404700 1 --2- 172.21.15.16:0/153081842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba408c6d0 0x7fbba408cac0 secure :-1 s=THROTTLE_DONE pgs=1023 cs=0 l=1 rx=0x7fbb9c00b2d0 tx=0x7fbb9c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.969+0000 7fbba9404700 1 --2- 172.21.15.16:0/153081842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba408c6d0 0x7fbba408cac0 secure :-1 s=THROTTLE_DONE pgs=1023 cs=0 l=1 rx=0x7fbb9c00b2d0 tx=0x7fbb9c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.969+0000 7fbba3fff700 1 -- 172.21.15.16:0/153081842 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb9c009d30 con 0x7fbba408c6d0 2019-12-05T06:36:00.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.969+0000 7fbba3fff700 1 -- 172.21.15.16:0/153081842 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb9c00a400 con 0x7fbba408c6d0 2019-12-05T06:36:00.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.969+0000 7fbbaae67700 1 -- 172.21.15.16:0/153081842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba408c6d0 msgr2=0x7fbba408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:00.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.969+0000 7fbbaae67700 1 --2- 172.21.15.16:0/153081842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba408c6d0 0x7fbba408cac0 secure :-1 s=READY pgs=1023 cs=0 l=1 rx=0x7fbb9c00b2d0 tx=0x7fbb9c00bae0).stop 2019-12-05T06:36:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.971+0000 7fbbaae67700 1 -- 172.21.15.16:0/153081842 shutdown_connections 2019-12-05T06:36:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.971+0000 7fbbaae67700 1 --2- 172.21.15.16:0/153081842 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbba408fef0 0x7fbba408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.971+0000 7fbbaae67700 1 --2- 172.21.15.16:0/153081842 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbba408c6d0 0x7fbba408cac0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.972+0000 7fbbaae67700 1 -- 172.21.15.16:0/153081842 shutdown_connections 2019-12-05T06:36:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.972+0000 7fbbaae67700 1 -- 172.21.15.16:0/153081842 wait complete. 2019-12-05T06:36:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.972+0000 7fbbaae67700 1 -- 172.21.15.16:0/153081842 >> 172.21.15.16:0/153081842 conn(0x7fbba4061940 msgr2=0x7fbba4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:00.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.977+0000 7fbbaae67700 1 Processor -- start 2019-12-05T06:36:00.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.977+0000 7fbbaae67700 1 -- start start 2019-12-05T06:36:00.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.979+0000 7fbbaae67700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbba4133220 0x7fbba409a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.979+0000 7fbbaae67700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba406e7c0 con 0x7fbba409ae80 2019-12-05T06:36:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.979+0000 7fbbaae67700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbba4133b30 con 0x7fbba4132630 2019-12-05T06:36:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.979+0000 7fbbaae67700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbba4098f20 con 0x7fbba4133220 2019-12-05T06:36:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.980+0000 7fbba8c03700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbba409ae80 0x7fbba4099eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:41984/0 (socket says 172.21.15.16:41984) 2019-12-05T06:36:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.980+0000 7fbba8c03700 1 -- 172.21.15.16:0/3592667383 learned_addr learned my addr 172.21.15.16:0/3592667383 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.980+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> v2:172.21.15.76:3300/0 conn(0x7fbba4133220 0x7fbba409a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:00.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.980+0000 7fbba1ffb700 1 -- 172.21.15.16:0/3592667383 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3031484407 0 0) 0x7fbb9800dd90 con 0x7fbba409ae80 2019-12-05T06:36:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.980+0000 7fbba1ffb700 1 -- 172.21.15.16:0/3592667383 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb84002e80 con 0x7fbba409ae80 2019-12-05T06:36:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.980+0000 7fbba9c05700 1 -- 172.21.15.16:0/3592667383 >> v1:172.21.15.76:6789/0 conn(0x7fbba4132630 legacy=0x7fbba4132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.980+0000 7fbba9c05700 1 -- 172.21.15.16:0/3592667383 >> v1:172.21.15.16:6789/0 conn(0x7fbba409ae80 legacy=0x7fbba4099eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.981+0000 7fbba9c05700 1 -- 172.21.15.16:0/3592667383 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba40995b0 con 0x7fbba4133220 2019-12-05T06:36:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.981+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> v2:172.21.15.76:3300/0 conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.981+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> v2:172.21.15.76:3300/0 conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.981+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=READY pgs=727 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.981+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.981+0000 7fbba1ffb700 1 -- 172.21.15.16:0/3592667383 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb94000ef0 con 0x7fbba4133220 2019-12-05T06:36:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.982+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.982+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.982+0000 7fbba1ffb700 1 -- 172.21.15.16:0/3592667383 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb940041c0 con 0x7fbba4133220 2019-12-05T06:36:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.982+0000 7fbba1ffb700 1 -- 172.21.15.16:0/3592667383 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb94000ef0 con 0x7fbba4133220 2019-12-05T06:36:00.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.982+0000 7fbbaae67700 1 -- 172.21.15.16:0/3592667383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba4133680 con 0x7fbba4133220 2019-12-05T06:36:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.982+0000 7fbbaae67700 1 -- 172.21.15.16:0/3592667383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba40995b0 con 0x7fbba4133220 2019-12-05T06:36:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.983+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.983+0000 7fbba1ffb700 1 -- 172.21.15.16:0/3592667383 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbb940043b0 con 0x7fbba4133220 2019-12-05T06:36:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.983+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.984+0000 7fbba1ffb700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb84023670 0x7fbb84025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.984+0000 7fbba1ffb700 1 -- 172.21.15.16:0/3592667383 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 7665+0+0 (secure 0 0 0) 0x7fbb94045d90 con 0x7fbba4133220 2019-12-05T06:36:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.984+0000 7fbba9404700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb84023670 0x7fbb84025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:00.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.985+0000 7fbba9404700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb84023670 0x7fbb84025ae0 crc :-1 s=READY pgs=442 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:00.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:00.997+0000 7fbbaae67700 1 -- 172.21.15.16:0/3592667383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba4133680 con 0x7fbba4133220 2019-12-05T06:36:01.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.007+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:01.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.007+0000 7fbba1ffb700 1 -- 172.21.15.16:0/3592667383 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbb940332a0 con 0x7fbba4133220 2019-12-05T06:36:01.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.274+0000 7fbbaae67700 1 -- 172.21.15.16:0/3592667383 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbba40995b0 con 0x7fbba4133220 2019-12-05T06:36:01.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.276+0000 7fbba9c05700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=THROTTLE_DONE pgs=727 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:01.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.276+0000 7fbba1ffb700 1 -- 172.21.15.16:0/3592667383 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 127 v127) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fbb940332a0 con 0x7fbba4133220 2019-12-05T06:36:01.277 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 127 2019-12-05T06:36:01.277 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:01.277 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":127,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5483,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":127}],"filesystems":[{"mdsmap":{"epoch":126,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:33:33.565206+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":68,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5426},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5426":{"gid":5426,"name":"b","rank":0,"incarnation":123,"state":"up:active","state_seq":24,"addr":"172.21.15.76:6835/3530466447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3530466447},{"type":"v1","addr":"172.21.15.76:6835","nonce":3530466447}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:01.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.280+0000 7fbbaae67700 1 -- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb84023670 msgr2=0x7fbb84025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:01.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.280+0000 7fbbaae67700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb84023670 0x7fbb84025ae0 crc :-1 s=READY pgs=442 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:01.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.280+0000 7fbbaae67700 1 -- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 msgr2=0x7fbba409a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:01.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.280+0000 7fbbaae67700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 0x7fbba409a6b0 secure :-1 s=READY pgs=727 cs=0 l=1 rx=0x7fbb94010390 tx=0x7fbb94010730).stop 2019-12-05T06:36:01.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.280+0000 7fbbaae67700 1 -- 172.21.15.16:0/3592667383 shutdown_connections 2019-12-05T06:36:01.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.281+0000 7fbbaae67700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb84023670 0x7fbb84025ae0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:01.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.281+0000 7fbbaae67700 1 --2- 172.21.15.16:0/3592667383 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbba4133220 0x7fbba409a6b0 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:01.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.281+0000 7fbbaae67700 1 -- 172.21.15.16:0/3592667383 shutdown_connections 2019-12-05T06:36:01.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.281+0000 7fbbaae67700 1 -- 172.21.15.16:0/3592667383 wait complete. 2019-12-05T06:36:01.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:01.281+0000 7fbbaae67700 1 -- 172.21.15.16:0/3592667383 >> 172.21.15.16:0/3592667383 conn(0x7fbba40714f0 msgr2=0x7fbba409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:01.300 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:36:03.301 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:03.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.527+0000 7f668067c700 1 Processor -- start 2019-12-05T06:36:03.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.527+0000 7f668067c700 1 -- start start 2019-12-05T06:36:03.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.528+0000 7f668067c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f667808c6d0 0x7f667808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:03.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.528+0000 7f668067c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f667808fef0 0x7f667808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.528+0000 7f668067c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667808fb80 con 0x7f667808ef20 2019-12-05T06:36:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.528+0000 7f668067c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f667808df90 con 0x7f667808fef0 2019-12-05T06:36:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.528+0000 7f668067c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66780a3f70 con 0x7f667808c6d0 2019-12-05T06:36:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f667808fef0 0x7f667808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667ec19700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f667808c6d0 0x7f667808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:03.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f667808fef0 0x7f667808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56838/0 (socket says 172.21.15.16:56838) 2019-12-05T06:36:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667f41a700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f667808ef20 0x7f667808f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:35076/0 (socket says 172.21.15.16:35076) 2019-12-05T06:36:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 -- 172.21.15.16:0/2293160978 learned_addr learned my addr 172.21.15.16:0/2293160978 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 -- 172.21.15.16:0/2293160978 >> v2:172.21.15.76:3300/0 conn(0x7f667808c6d0 msgr2=0x7f667808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 --2- 172.21.15.16:0/2293160978 >> v2:172.21.15.76:3300/0 conn(0x7f667808c6d0 0x7f667808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 -- 172.21.15.16:0/2293160978 >> v1:172.21.15.76:6789/0 conn(0x7f667808ef20 legacy=0x7f667808f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 -- 172.21.15.16:0/2293160978 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f667808d540 con 0x7f667808fef0 2019-12-05T06:36:03.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 --2- 172.21.15.16:0/2293160978 >> v2:172.21.15.16:3300/0 conn(0x7f667808fef0 0x7f667808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f666c00fa40 tx=0x7f666c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 --2- 172.21.15.16:0/2293160978 >> v2:172.21.15.16:3300/0 conn(0x7f667808fef0 0x7f667808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f666c00fa40 tx=0x7f666c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 --2- 172.21.15.16:0/2293160978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667808fef0 0x7f667808d000 secure :-1 s=READY pgs=1025 cs=0 l=1 rx=0x7f666c00fa40 tx=0x7f666c00fde0).ready entity=mon.0 client_cookie=19b9de618c8688b9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 --2- 172.21.15.16:0/2293160978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667808fef0 0x7f667808d000 secure :-1 s=THROTTLE_DONE pgs=1025 cs=0 l=1 rx=0x7f666c00fa40 tx=0x7f666c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 --2- 172.21.15.16:0/2293160978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667808fef0 0x7f667808d000 secure :-1 s=THROTTLE_DONE pgs=1025 cs=0 l=1 rx=0x7f666c00fa40 tx=0x7f666c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667dc17700 1 -- 172.21.15.16:0/2293160978 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666c000ce0 con 0x7f667808fef0 2019-12-05T06:36:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.529+0000 7f667e418700 1 --2- 172.21.15.16:0/2293160978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667808fef0 0x7f667808d000 secure :-1 s=THROTTLE_DONE pgs=1025 cs=0 l=1 rx=0x7f666c00fa40 tx=0x7f666c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.530+0000 7f667dc17700 1 -- 172.21.15.16:0/2293160978 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c001010 con 0x7f667808fef0 2019-12-05T06:36:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.530+0000 7f667dc17700 1 -- 172.21.15.16:0/2293160978 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666c001720 con 0x7f667808fef0 2019-12-05T06:36:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.530+0000 7f668067c700 1 -- 172.21.15.16:0/2293160978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667808fef0 msgr2=0x7f667808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.530+0000 7f668067c700 1 --2- 172.21.15.16:0/2293160978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667808fef0 0x7f667808d000 secure :-1 s=READY pgs=1025 cs=0 l=1 rx=0x7f666c00fa40 tx=0x7f666c00fde0).stop 2019-12-05T06:36:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.531+0000 7f668067c700 1 -- 172.21.15.16:0/2293160978 shutdown_connections 2019-12-05T06:36:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.531+0000 7f668067c700 1 --2- 172.21.15.16:0/2293160978 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f667808fef0 0x7f667808d000 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.531+0000 7f668067c700 1 --2- 172.21.15.16:0/2293160978 >> v2:172.21.15.76:3300/0 conn(0x7f667808c6d0 0x7f667808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:03.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.531+0000 7f668067c700 1 -- 172.21.15.16:0/2293160978 shutdown_connections 2019-12-05T06:36:03.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.531+0000 7f668067c700 1 -- 172.21.15.16:0/2293160978 wait complete. 2019-12-05T06:36:03.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.531+0000 7f668067c700 1 -- 172.21.15.16:0/2293160978 >> 172.21.15.16:0/2293160978 conn(0x7f6678061940 msgr2=0x7f6678096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.533+0000 7f668067c700 1 Processor -- start 2019-12-05T06:36:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.533+0000 7f668067c700 1 -- start start 2019-12-05T06:36:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f668067c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66780879b0 0x7f66780871e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f668067c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6678153170 con 0x7f667809be80 2019-12-05T06:36:03.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f668067c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f667809e030 con 0x7f667814ca10 2019-12-05T06:36:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f668067c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6678085a50 con 0x7f66780879b0 2019-12-05T06:36:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f667e418700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f66780879b0 0x7f66780871e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f667f41a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f667809be80 0x7f667809c2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42012/0 (socket says 172.21.15.16:42012) 2019-12-05T06:36:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f667f41a700 1 -- 172.21.15.16:0/676691336 learned_addr learned my addr 172.21.15.16:0/676691336 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f667e418700 1 -- 172.21.15.16:0/676691336 >> v1:172.21.15.76:6789/0 conn(0x7f667814ca10 legacy=0x7f66780869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f667e418700 1 -- 172.21.15.16:0/676691336 >> v1:172.21.15.16:6789/0 conn(0x7f667809be80 legacy=0x7f667809c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:03.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f667e418700 1 -- 172.21.15.16:0/676691336 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66780860e0 con 0x7f66780879b0 2019-12-05T06:36:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f66777fe700 1 -- 172.21.15.16:0/676691336 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866940322 0 0) 0x7f6668001290 con 0x7f667809be80 2019-12-05T06:36:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.534+0000 7f667e418700 1 --2- 172.21.15.16:0/676691336 >> v2:172.21.15.16:3300/0 conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.535+0000 7f667e418700 1 --2- 172.21.15.16:0/676691336 >> v2:172.21.15.16:3300/0 conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.535+0000 7f667e418700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=READY pgs=1027 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.535+0000 7f667e418700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=THROTTLE_DONE pgs=1027 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.535+0000 7f66777fe700 1 -- 172.21.15.16:0/676691336 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666c0012d0 con 0x7f66780879b0 2019-12-05T06:36:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.535+0000 7f667e418700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=THROTTLE_DONE pgs=1027 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.535+0000 7f667e418700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=THROTTLE_DONE pgs=1027 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.535+0000 7f66777fe700 1 -- 172.21.15.16:0/676691336 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f666c000cc0 con 0x7f66780879b0 2019-12-05T06:36:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.535+0000 7f66777fe700 1 -- 172.21.15.16:0/676691336 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f666c0018b0 con 0x7f66780879b0 2019-12-05T06:36:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.535+0000 7f668067c700 1 -- 172.21.15.16:0/676691336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66780860e0 con 0x7f66780879b0 2019-12-05T06:36:03.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.535+0000 7f668067c700 1 -- 172.21.15.16:0/676691336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66780860e0 con 0x7f66780879b0 2019-12-05T06:36:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.536+0000 7f667e418700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=THROTTLE_DONE pgs=1027 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.536+0000 7f66777fe700 1 -- 172.21.15.16:0/676691336 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f666c0010e0 con 0x7f66780879b0 2019-12-05T06:36:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.536+0000 7f667e418700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=THROTTLE_DONE pgs=1027 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.536+0000 7f66777fe700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6658023630 0x7f6658025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.536+0000 7f66777fe700 1 -- 172.21.15.16:0/676691336 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f666c0460f0 con 0x7f66780879b0 2019-12-05T06:36:03.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.537+0000 7f667ec19700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6658023630 0x7f6658025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.537+0000 7f667ec19700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6658023630 0x7f6658025ac0 crc :-1 s=READY pgs=443 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:03.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.543+0000 7f668067c700 1 -- 172.21.15.16:0/676691336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66780860e0 con 0x7f66780879b0 2019-12-05T06:36:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.546+0000 7f667e418700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=THROTTLE_DONE pgs=1027 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.546+0000 7f66777fe700 1 -- 172.21.15.16:0/676691336 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f666c01a160 con 0x7f66780879b0 2019-12-05T06:36:03.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.752+0000 7f668067c700 1 -- 172.21.15.16:0/676691336 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f66780860e0 con 0x7f66780879b0 2019-12-05T06:36:03.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.752+0000 7f667e418700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=THROTTLE_DONE pgs=1027 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.752+0000 7f66777fe700 1 -- 172.21.15.16:0/676691336 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f666c01a160 con 0x7f66780879b0 2019-12-05T06:36:03.752 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 128 2019-12-05T06:36:03.752 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:03.753 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:03.325194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:03.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.755+0000 7f668067c700 1 -- 172.21.15.16:0/676691336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6658023630 msgr2=0x7f6658025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:03.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.755+0000 7f668067c700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6658023630 0x7f6658025ac0 crc :-1 s=READY pgs=443 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:03.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.755+0000 7f668067c700 1 -- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 msgr2=0x7f66780871e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:03.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.755+0000 7f668067c700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 0x7f66780871e0 secure :-1 s=READY pgs=1027 cs=0 l=1 rx=0x7f666c010710 tx=0x7f666c010ab0).stop 2019-12-05T06:36:03.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.755+0000 7f668067c700 1 -- 172.21.15.16:0/676691336 shutdown_connections 2019-12-05T06:36:03.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.755+0000 7f668067c700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6658023630 0x7f6658025ac0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:03.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.755+0000 7f668067c700 1 --2- 172.21.15.16:0/676691336 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f66780879b0 0x7f66780871e0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:03.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.756+0000 7f668067c700 1 -- 172.21.15.16:0/676691336 shutdown_connections 2019-12-05T06:36:03.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.756+0000 7f668067c700 1 -- 172.21.15.16:0/676691336 wait complete. 2019-12-05T06:36:03.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.756+0000 7f668067c700 1 -- 172.21.15.16:0/676691336 >> 172.21.15.16:0/676691336 conn(0x7f6678098480 msgr2=0x7f6678071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:03.765 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:36:03.766 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:36:03.766 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:03.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.947+0000 7f3b3a6a2700 1 Processor -- start 2019-12-05T06:36:03.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.947+0000 7f3b3a6a2700 1 -- start start 2019-12-05T06:36:03.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.948+0000 7f3b3a6a2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b3408c6d0 0x7f3b3408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:03.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.948+0000 7f3b3a6a2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3b3408f720 0x7f3b3408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:03.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.948+0000 7f3b3a6a2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b3408fb80 con 0x7f3b3408fef0 2019-12-05T06:36:03.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.948+0000 7f3b3a6a2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3408df90 con 0x7f3b3408c6d0 2019-12-05T06:36:03.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.948+0000 7f3b3a6a2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b340a3f70 con 0x7f3b3408f720 2019-12-05T06:36:03.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.948+0000 7f3b38c3f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b3408c6d0 0x7f3b3408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:03.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.948+0000 7f3b38c3f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b3408c6d0 0x7f3b3408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56850/0 (socket says 172.21.15.16:56850) 2019-12-05T06:36:03.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.948+0000 7f3b38c3f700 1 -- 172.21.15.16:0/509332835 learned_addr learned my addr 172.21.15.16:0/509332835 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:03.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b39440700 1 --2- 172.21.15.16:0/509332835 >> v2:172.21.15.76:3300/0 conn(0x7f3b3408f720 0x7f3b3408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:03.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b38c3f700 1 -- 172.21.15.16:0/509332835 >> v2:172.21.15.76:3300/0 conn(0x7f3b3408f720 msgr2=0x7f3b3408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:03.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/509332835 >> v2:172.21.15.76:3300/0 conn(0x7f3b3408f720 0x7f3b3408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b38c3f700 1 -- 172.21.15.16:0/509332835 >> v1:172.21.15.16:6789/0 conn(0x7f3b3408fef0 legacy=0x7f3b3408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b38c3f700 1 -- 172.21.15.16:0/509332835 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3408d540 con 0x7f3b3408c6d0 2019-12-05T06:36:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b337fe700 1 -- 172.21.15.16:0/509332835 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2229825793 0 0) 0x7f3b1c001100 con 0x7f3b3408fef0 2019-12-05T06:36:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/509332835 >> v2:172.21.15.16:3300/0 conn(0x7f3b3408c6d0 0x7f3b3408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3b2c00fac0 tx=0x7f3b2c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/509332835 >> v2:172.21.15.16:3300/0 conn(0x7f3b3408c6d0 0x7f3b3408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3b2c00fac0 tx=0x7f3b2c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/509332835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3408c6d0 0x7f3b3408cac0 secure :-1 s=READY pgs=1029 cs=0 l=1 rx=0x7f3b2c00fac0 tx=0x7f3b2c00f8f0).ready entity=mon.0 client_cookie=c3cbdae1e9f2c80 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:03.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/509332835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3408c6d0 0x7f3b3408cac0 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7f3b2c00fac0 tx=0x7f3b2c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b337fe700 1 -- 172.21.15.16:0/509332835 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b2c000ce0 con 0x7f3b3408c6d0 2019-12-05T06:36:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/509332835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3408c6d0 0x7f3b3408cac0 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7f3b2c00fac0 tx=0x7f3b2c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/509332835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3408c6d0 0x7f3b3408cac0 secure :-1 s=THROTTLE_DONE pgs=1029 cs=0 l=1 rx=0x7f3b2c00fac0 tx=0x7f3b2c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b337fe700 1 -- 172.21.15.16:0/509332835 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b2c000ce0 con 0x7f3b3408c6d0 2019-12-05T06:36:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b337fe700 1 -- 172.21.15.16:0/509332835 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b2c0013f0 con 0x7f3b3408c6d0 2019-12-05T06:36:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/509332835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3408c6d0 msgr2=0x7f3b3408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:03.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.949+0000 7f3b3a6a2700 1 --2- 172.21.15.16:0/509332835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3408c6d0 0x7f3b3408cac0 secure :-1 s=READY pgs=1029 cs=0 l=1 rx=0x7f3b2c00fac0 tx=0x7f3b2c00f8f0).stop 2019-12-05T06:36:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.950+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/509332835 shutdown_connections 2019-12-05T06:36:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.950+0000 7f3b3a6a2700 1 --2- 172.21.15.16:0/509332835 >> v2:172.21.15.76:3300/0 conn(0x7f3b3408f720 0x7f3b3408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.950+0000 7f3b3a6a2700 1 --2- 172.21.15.16:0/509332835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b3408c6d0 0x7f3b3408cac0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.950+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/509332835 shutdown_connections 2019-12-05T06:36:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.950+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/509332835 wait complete. 2019-12-05T06:36:03.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.950+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/509332835 >> 172.21.15.16:0/509332835 conn(0x7f3b34061940 msgr2=0x7f3b34096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.954+0000 7f3b3a6a2700 1 Processor -- start 2019-12-05T06:36:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.954+0000 7f3b3a6a2700 1 -- start start 2019-12-05T06:36:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b3a6a2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3b3406dc10 0x7f3b3406e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b3a6a2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b34130be0 0x7f3b3406e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:03.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b3a6a2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b34130870 con 0x7f3b3412fc10 2019-12-05T06:36:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b3a6a2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3409e690 con 0x7f3b34130be0 2019-12-05T06:36:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b3a6a2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b3412ec80 con 0x7f3b3406dc10 2019-12-05T06:36:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b33fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b34130be0 0x7f3b3406e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b33fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b34130be0 0x7f3b3406e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56858/0 (socket says 172.21.15.16:56858) 2019-12-05T06:36:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b39440700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3b3412fc10 0x7f3b34130070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42028/0 (socket says 172.21.15.16:42028) 2019-12-05T06:36:03.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b33fff700 1 -- 172.21.15.16:0/1300188485 learned_addr learned my addr 172.21.15.16:0/1300188485 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/1300188485 >> v2:172.21.15.76:3300/0 conn(0x7f3b3406dc10 0x7f3b3406e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.955+0000 7f3b33fff700 1 -- 172.21.15.16:0/1300188485 >> v2:172.21.15.76:3300/0 conn(0x7f3b3406dc10 msgr2=0x7f3b3406e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> v2:172.21.15.76:3300/0 conn(0x7f3b3406dc10 0x7f3b3406e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b33fff700 1 -- 172.21.15.16:0/1300188485 >> v1:172.21.15.16:6789/0 conn(0x7f3b3412fc10 legacy=0x7f3b34130070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b33fff700 1 -- 172.21.15.16:0/1300188485 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3412f310 con 0x7f3b34130be0 2019-12-05T06:36:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b317fa700 1 -- 172.21.15.16:0/1300188485 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414050794 0 0) 0x7f3b2400a8a0 con 0x7f3b3412fc10 2019-12-05T06:36:03.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/1300188485 >> v2:172.21.15.76:3300/0 conn(0x7f3b3406dc10 0x7f3b3406e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:36:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> v2:172.21.15.16:3300/0 conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> v2:172.21.15.16:3300/0 conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=READY pgs=1031 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=THROTTLE_DONE pgs=1031 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b317fa700 1 -- 172.21.15.16:0/1300188485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b1c000cc0 con 0x7f3b34130be0 2019-12-05T06:36:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=THROTTLE_DONE pgs=1031 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b317fa700 1 -- 172.21.15.16:0/1300188485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b1c00ee80 con 0x7f3b34130be0 2019-12-05T06:36:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=THROTTLE_DONE pgs=1031 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b317fa700 1 -- 172.21.15.16:0/1300188485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b1c00f070 con 0x7f3b34130be0 2019-12-05T06:36:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/1300188485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b3412f310 con 0x7f3b34130be0 2019-12-05T06:36:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.956+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/1300188485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b3412f310 con 0x7f3b34130be0 2019-12-05T06:36:03.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.957+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=THROTTLE_DONE pgs=1031 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.957+0000 7f3b317fa700 1 -- 172.21.15.16:0/1300188485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3b1c00f280 con 0x7f3b34130be0 2019-12-05T06:36:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.957+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=THROTTLE_DONE pgs=1031 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.957+0000 7f3b317fa700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3b18023630 0x7f3b18025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.957+0000 7f3b317fa700 1 -- 172.21.15.16:0/1300188485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f3b1c045e00 con 0x7f3b34130be0 2019-12-05T06:36:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.958+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3b18023630 0x7f3b18025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:03.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.958+0000 7f3b38c3f700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3b18023630 0x7f3b18025ac0 crc :-1 s=READY pgs=444 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.963+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/1300188485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b3412f310 con 0x7f3b34130be0 2019-12-05T06:36:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.968+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=THROTTLE_DONE pgs=1031 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:03.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:03.968+0000 7f3b317fa700 1 -- 172.21.15.16:0/1300188485 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3b1c00f070 con 0x7f3b34130be0 2019-12-05T06:36:04.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.169+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/1300188485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b3412f310 con 0x7f3b34130be0 2019-12-05T06:36:04.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.170+0000 7f3b33fff700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=THROTTLE_DONE pgs=1031 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:04.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.170+0000 7f3b317fa700 1 -- 172.21.15.16:0/1300188485 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 128 v128) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f3b1c00f280 con 0x7f3b34130be0 2019-12-05T06:36:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 128 2019-12-05T06:36:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/1300188485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3b18023630 msgr2=0x7f3b18025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3b18023630 0x7f3b18025ac0 crc :-1 s=READY pgs=444 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 msgr2=0x7f3b3406e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 0x7f3b3406e540 secure :-1 s=READY pgs=1031 cs=0 l=1 rx=0x7f3b1c010450 tx=0x7f3b1c0107f0).stop 2019-12-05T06:36:04.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/1300188485 shutdown_connections 2019-12-05T06:36:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3b18023630 0x7f3b18025ac0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 --2- 172.21.15.16:0/1300188485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b34130be0 0x7f3b3406e540 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 --2- 172.21.15.16:0/1300188485 >> v2:172.21.15.76:3300/0 conn(0x7f3b3406dc10 0x7f3b3406e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/1300188485 shutdown_connections 2019-12-05T06:36:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/1300188485 wait complete. 2019-12-05T06:36:04.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:04.173+0000 7f3b3a6a2700 1 -- 172.21.15.16:0/1300188485 >> 172.21.15.16:0/1300188485 conn(0x7f3b34098480 msgr2=0x7f3b34071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:04.182 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:04.182 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":128,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":128,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:03.325194+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:04.190 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 128, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 128, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5483": { "addr": "172.21.15.16:6833/3678674944", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 3678674944, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 3678674944, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5483, "incarnation": 128, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 69, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:36:03.325194+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5483 } } } ], "standbys": [] } 2019-12-05T06:36:06.191 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.433+0000 7fb822df5700 1 Processor -- start 2019-12-05T06:36:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.433+0000 7fb822df5700 1 -- start start 2019-12-05T06:36:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.434+0000 7fb822df5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb81c08c6d0 0x7fb81c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.434+0000 7fb822df5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb81c08fef0 0x7fb81c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:06.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.434+0000 7fb822df5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c0958e0 con 0x7fb81c08ef20 2019-12-05T06:36:06.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.434+0000 7fb822df5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb81c08df90 con 0x7fb81c08c6d0 2019-12-05T06:36:06.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.434+0000 7fb822df5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c0a3f70 con 0x7fb81c08fef0 2019-12-05T06:36:06.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.434+0000 7fb821392700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb81c08c6d0 0x7fb81c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:06.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.435+0000 7fb821392700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb81c08c6d0 0x7fb81c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56866/0 (socket says 172.21.15.16:56866) 2019-12-05T06:36:06.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.435+0000 7fb821392700 1 -- 172.21.15.16:0/3448855590 learned_addr learned my addr 172.21.15.16:0/3448855590 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:06.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.435+0000 7fb820b91700 1 --2- 172.21.15.16:0/3448855590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb81c08fef0 0x7fb81c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.435+0000 7fb821392700 1 -- 172.21.15.16:0/3448855590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb81c08fef0 msgr2=0x7fb81c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.435+0000 7fb821392700 1 --2- 172.21.15.16:0/3448855590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb81c08fef0 0x7fb81c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.435+0000 7fb821392700 1 -- 172.21.15.16:0/3448855590 >> v1:172.21.15.76:6789/0 conn(0x7fb81c08ef20 legacy=0x7fb81c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.435+0000 7fb821392700 1 -- 172.21.15.16:0/3448855590 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c08d540 con 0x7fb81c08c6d0 2019-12-05T06:36:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.435+0000 7fb821392700 1 --2- 172.21.15.16:0/3448855590 >> v2:172.21.15.16:3300/0 conn(0x7fb81c08c6d0 0x7fb81c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb80c00f740 tx=0x7fb80c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.435+0000 7fb821392700 1 --2- 172.21.15.16:0/3448855590 >> v2:172.21.15.16:3300/0 conn(0x7fb81c08c6d0 0x7fb81c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb80c00f740 tx=0x7fb80c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.435+0000 7fb821392700 1 --2- 172.21.15.16:0/3448855590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c08c6d0 0x7fb81c08cac0 secure :-1 s=READY pgs=1032 cs=0 l=1 rx=0x7fb80c00f740 tx=0x7fb80c00f570).ready entity=mon.0 client_cookie=99ea7567f7dc917c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.436+0000 7fb821392700 1 --2- 172.21.15.16:0/3448855590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c08c6d0 0x7fb81c08cac0 secure :-1 s=THROTTLE_DONE pgs=1032 cs=0 l=1 rx=0x7fb80c00f740 tx=0x7fb80c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.436+0000 7fb821392700 1 --2- 172.21.15.16:0/3448855590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c08c6d0 0x7fb81c08cac0 secure :-1 s=THROTTLE_DONE pgs=1032 cs=0 l=1 rx=0x7fb80c00f740 tx=0x7fb80c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.436+0000 7fb813fff700 1 -- 172.21.15.16:0/3448855590 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb80c000960 con 0x7fb81c08c6d0 2019-12-05T06:36:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.436+0000 7fb821392700 1 --2- 172.21.15.16:0/3448855590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c08c6d0 0x7fb81c08cac0 secure :-1 s=THROTTLE_DONE pgs=1032 cs=0 l=1 rx=0x7fb80c00f740 tx=0x7fb80c00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.436+0000 7fb813fff700 1 -- 172.21.15.16:0/3448855590 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb80c000c90 con 0x7fb81c08c6d0 2019-12-05T06:36:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.436+0000 7fb813fff700 1 -- 172.21.15.16:0/3448855590 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb80c0013a0 con 0x7fb81c08c6d0 2019-12-05T06:36:06.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.436+0000 7fb822df5700 1 -- 172.21.15.16:0/3448855590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c08c6d0 msgr2=0x7fb81c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.436+0000 7fb822df5700 1 --2- 172.21.15.16:0/3448855590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c08c6d0 0x7fb81c08cac0 secure :-1 s=READY pgs=1032 cs=0 l=1 rx=0x7fb80c00f740 tx=0x7fb80c00f570).stop 2019-12-05T06:36:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.437+0000 7fb822df5700 1 -- 172.21.15.16:0/3448855590 shutdown_connections 2019-12-05T06:36:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.437+0000 7fb822df5700 1 --2- 172.21.15.16:0/3448855590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb81c08fef0 0x7fb81c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.437+0000 7fb822df5700 1 --2- 172.21.15.16:0/3448855590 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c08c6d0 0x7fb81c08cac0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.438+0000 7fb822df5700 1 -- 172.21.15.16:0/3448855590 shutdown_connections 2019-12-05T06:36:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.438+0000 7fb822df5700 1 -- 172.21.15.16:0/3448855590 wait complete. 2019-12-05T06:36:06.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.438+0000 7fb822df5700 1 -- 172.21.15.16:0/3448855590 >> 172.21.15.16:0/3448855590 conn(0x7fb81c061940 msgr2=0x7fb81c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.442+0000 7fb822df5700 1 Processor -- start 2019-12-05T06:36:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.442+0000 7fb822df5700 1 -- start start 2019-12-05T06:36:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.444+0000 7fb822df5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb81c132630 0x7fb81c132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:06.449 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.444+0000 7fb822df5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb81c09ae80 0x7fb81c132f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.444+0000 7fb822df5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb81c06e7c0 con 0x7fb81c099eb0 2019-12-05T06:36:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.444+0000 7fb822df5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb81c134390 con 0x7fb81c09ae80 2019-12-05T06:36:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.444+0000 7fb822df5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb81c134660 con 0x7fb81c132630 2019-12-05T06:36:06.450 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb820b91700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb81c09ae80 0x7fb81c132f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb820b91700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb81c09ae80 0x7fb81c132f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56874/0 (socket says 172.21.15.16:56874) 2019-12-05T06:36:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb820b91700 1 -- 172.21.15.16:0/686153759 learned_addr learned my addr 172.21.15.16:0/686153759 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb821b93700 1 --1- 172.21.15.16:0/686153759 >> v1:172.21.15.16:6789/0 conn(0x7fb81c099eb0 0x7fb81c09a330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42044/0 (socket says 172.21.15.16:42044) 2019-12-05T06:36:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb821392700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb81c132630 0x7fb81c132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb820b91700 1 -- 172.21.15.16:0/686153759 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb81c132630 msgr2=0x7fb81c132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb81c132630 0x7fb81c132a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:06.451 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb820b91700 1 -- 172.21.15.16:0/686153759 >> v1:172.21.15.16:6789/0 conn(0x7fb81c099eb0 legacy=0x7fb81c09a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb811ffb700 1 -- 172.21.15.16:0/686153759 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940048744 0 0) 0x7fb8140009b0 con 0x7fb81c099eb0 2019-12-05T06:36:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb820b91700 1 -- 172.21.15.16:0/686153759 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb81c0995b0 con 0x7fb81c09ae80 2019-12-05T06:36:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.445+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> v2:172.21.15.16:3300/0 conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.446+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> v2:172.21.15.16:3300/0 conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.446+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=READY pgs=1034 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.446+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=THROTTLE_DONE pgs=1034 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.446+0000 7fb811ffb700 1 -- 172.21.15.16:0/686153759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb818000cc0 con 0x7fb81c09ae80 2019-12-05T06:36:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.446+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=THROTTLE_DONE pgs=1034 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.446+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=THROTTLE_DONE pgs=1034 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.446+0000 7fb811ffb700 1 -- 172.21.15.16:0/686153759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb818001200 con 0x7fb81c09ae80 2019-12-05T06:36:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.446+0000 7fb811ffb700 1 -- 172.21.15.16:0/686153759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb818001850 con 0x7fb81c09ae80 2019-12-05T06:36:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.446+0000 7fb822df5700 1 -- 172.21.15.16:0/686153759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb81c0995b0 con 0x7fb81c09ae80 2019-12-05T06:36:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.446+0000 7fb822df5700 1 -- 172.21.15.16:0/686153759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb81c0995b0 con 0x7fb81c09ae80 2019-12-05T06:36:06.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.447+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=THROTTLE_DONE pgs=1034 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.447+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=THROTTLE_DONE pgs=1034 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.447+0000 7fb811ffb700 1 -- 172.21.15.16:0/686153759 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb8180013f0 con 0x7fb81c09ae80 2019-12-05T06:36:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.448+0000 7fb811ffb700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7fc023630 0x7fb7fc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.448+0000 7fb811ffb700 1 -- 172.21.15.16:0/686153759 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fb81804a510 con 0x7fb81c09ae80 2019-12-05T06:36:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.448+0000 7fb821392700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7fc023630 0x7fb7fc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.448+0000 7fb821392700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7fc023630 0x7fb7fc025ac0 crc :-1 s=READY pgs=445 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:06.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.449+0000 7fb822df5700 1 -- 172.21.15.16:0/686153759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb81c0995b0 con 0x7fb81c09ae80 2019-12-05T06:36:06.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.457+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=THROTTLE_DONE pgs=1034 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.457+0000 7fb811ffb700 1 -- 172.21.15.16:0/686153759 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb81802f0a0 con 0x7fb81c09ae80 2019-12-05T06:36:06.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.729+0000 7fb822df5700 1 -- 172.21.15.16:0/686153759 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb81c0995b0 con 0x7fb81c09ae80 2019-12-05T06:36:06.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.730+0000 7fb820b91700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=THROTTLE_DONE pgs=1034 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:06.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.730+0000 7fb811ffb700 1 -- 172.21.15.16:0/686153759 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 129 v129) v1 ==== 95+0+1778 (secure 0 0 0) 0x7fb818009660 con 0x7fb81c09ae80 2019-12-05T06:36:06.730 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 129 2019-12-05T06:36:06.730 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:06.731 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":129,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":129,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:06.321287+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:reconnect","state_seq":37,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:06.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.732+0000 7fb822df5700 1 -- 172.21.15.16:0/686153759 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7fc023630 msgr2=0x7fb7fc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:06.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.732+0000 7fb822df5700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7fc023630 0x7fb7fc025ac0 crc :-1 s=READY pgs=445 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:06.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.732+0000 7fb822df5700 1 -- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 msgr2=0x7fb81c132f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:06.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.732+0000 7fb822df5700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 0x7fb81c132f60 secure :-1 s=READY pgs=1034 cs=0 l=1 rx=0x7fb818014b60 tx=0x7fb818014f00).stop 2019-12-05T06:36:06.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.732+0000 7fb822df5700 1 -- 172.21.15.16:0/686153759 shutdown_connections 2019-12-05T06:36:06.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.732+0000 7fb822df5700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb7fc023630 0x7fb7fc025ac0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:06.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.732+0000 7fb822df5700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb81c09ae80 0x7fb81c132f60 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:06.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.732+0000 7fb822df5700 1 --2- 172.21.15.16:0/686153759 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb81c132630 0x7fb81c132a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:06.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.733+0000 7fb822df5700 1 -- 172.21.15.16:0/686153759 shutdown_connections 2019-12-05T06:36:06.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.733+0000 7fb822df5700 1 -- 172.21.15.16:0/686153759 wait complete. 2019-12-05T06:36:06.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:06.733+0000 7fb822df5700 1 -- 172.21.15.16:0/686153759 >> 172.21.15.16:0/686153759 conn(0x7fb81c0714f0 msgr2=0x7fb81c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:06.743 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:36:07.706 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:36:07.717 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:36:08.746 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:09.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.034+0000 7fc9062eb700 1 Processor -- start 2019-12-05T06:36:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.034+0000 7fc9062eb700 1 -- start start 2019-12-05T06:36:09.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.036+0000 7fc9062eb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc90008d2c0 0x7fc90008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:09.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.036+0000 7fc9062eb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000958e0 con 0x7fc90008fef0 2019-12-05T06:36:09.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.036+0000 7fc9062eb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000629e0 con 0x7fc90008c6d0 2019-12-05T06:36:09.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.036+0000 7fc9062eb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc90008df90 con 0x7fc90008d2c0 2019-12-05T06:36:09.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.036+0000 7fc8fffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc90008d2c0 0x7fc90008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.036+0000 7fc905089700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc90008fef0 0x7fc90008f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42056/0 (socket says 172.21.15.16:42056) 2019-12-05T06:36:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.037+0000 7fc905089700 1 -- 172.21.15.16:0/65485835 learned_addr learned my addr 172.21.15.16:0/65485835 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.037+0000 7fc8ff7fe700 1 -- 172.21.15.16:0/65485835 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022401001 0 0) 0x7fc8f8001400 con 0x7fc90008c6d0 2019-12-05T06:36:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.037+0000 7fc8fffff700 1 -- 172.21.15.16:0/65485835 >> v1:172.21.15.76:6789/0 conn(0x7fc90008c6d0 legacy=0x7fc90008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.037+0000 7fc8fffff700 1 -- 172.21.15.16:0/65485835 >> v1:172.21.15.16:6789/0 conn(0x7fc90008fef0 legacy=0x7fc90008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.037+0000 7fc8fffff700 1 -- 172.21.15.16:0/65485835 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90008e620 con 0x7fc90008d2c0 2019-12-05T06:36:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc8fffff700 1 --2- 172.21.15.16:0/65485835 >> v2:172.21.15.16:3300/0 conn(0x7fc90008d2c0 0x7fc90008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc8e800af50 tx=0x7fc8e800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc8fffff700 1 --2- 172.21.15.16:0/65485835 >> v2:172.21.15.16:3300/0 conn(0x7fc90008d2c0 0x7fc90008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc8e800af50 tx=0x7fc8e800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc8fffff700 1 --2- 172.21.15.16:0/65485835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90008d2c0 0x7fc90008ef20 secure :-1 s=READY pgs=1036 cs=0 l=1 rx=0x7fc8e800af50 tx=0x7fc8e800b760).ready entity=mon.0 client_cookie=7ae5295c4a6a108b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc8fffff700 1 --2- 172.21.15.16:0/65485835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90008d2c0 0x7fc90008ef20 secure :-1 s=THROTTLE_DONE pgs=1036 cs=0 l=1 rx=0x7fc8e800af50 tx=0x7fc8e800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc8ff7fe700 1 -- 172.21.15.16:0/65485835 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8e8000960 con 0x7fc90008d2c0 2019-12-05T06:36:09.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc8fffff700 1 --2- 172.21.15.16:0/65485835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90008d2c0 0x7fc90008ef20 secure :-1 s=THROTTLE_DONE pgs=1036 cs=0 l=1 rx=0x7fc8e800af50 tx=0x7fc8e800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc8fffff700 1 --2- 172.21.15.16:0/65485835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90008d2c0 0x7fc90008ef20 secure :-1 s=THROTTLE_DONE pgs=1036 cs=0 l=1 rx=0x7fc8e800af50 tx=0x7fc8e800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc8ff7fe700 1 -- 172.21.15.16:0/65485835 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8e8009860 con 0x7fc90008d2c0 2019-12-05T06:36:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc8ff7fe700 1 -- 172.21.15.16:0/65485835 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8e8009f30 con 0x7fc90008d2c0 2019-12-05T06:36:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc9062eb700 1 -- 172.21.15.16:0/65485835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90008d2c0 msgr2=0x7fc90008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.038+0000 7fc9062eb700 1 --2- 172.21.15.16:0/65485835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90008d2c0 0x7fc90008ef20 secure :-1 s=READY pgs=1036 cs=0 l=1 rx=0x7fc8e800af50 tx=0x7fc8e800b760).stop 2019-12-05T06:36:09.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.040+0000 7fc9062eb700 1 -- 172.21.15.16:0/65485835 shutdown_connections 2019-12-05T06:36:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.040+0000 7fc9062eb700 1 --2- 172.21.15.16:0/65485835 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc90008d2c0 0x7fc90008ef20 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.040+0000 7fc9062eb700 1 -- 172.21.15.16:0/65485835 shutdown_connections 2019-12-05T06:36:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.040+0000 7fc9062eb700 1 -- 172.21.15.16:0/65485835 wait complete. 2019-12-05T06:36:09.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.040+0000 7fc9062eb700 1 -- 172.21.15.16:0/65485835 >> 172.21.15.16:0/65485835 conn(0x7fc900061940 msgr2=0x7fc900096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:09.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.045+0000 7fc9062eb700 1 Processor -- start 2019-12-05T06:36:09.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.046+0000 7fc9062eb700 1 -- start start 2019-12-05T06:36:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc9062eb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc900132630 0x7fc900132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc9062eb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc90009ae80 0x7fc900132f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc9062eb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc900099eb0 0x7fc90009a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc9062eb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc900098f20 con 0x7fc900099eb0 2019-12-05T06:36:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc9062eb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc900134390 con 0x7fc900132630 2019-12-05T06:36:09.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc9062eb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc900134660 con 0x7fc90009ae80 2019-12-05T06:36:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc905089700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc900099eb0 0x7fc90009a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:09.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc8fffff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc90009ae80 0x7fc900132f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc904888700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc900132630 0x7fc900132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc905089700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc900099eb0 0x7fc90009a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56894/0 (socket says 172.21.15.16:56894) 2019-12-05T06:36:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc8fffff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc90009ae80 0x7fc900132f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:60880/0 (socket says 172.21.15.16:60880) 2019-12-05T06:36:09.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc904888700 1 --2- 172.21.15.16:0/4124787495 >> v2:172.21.15.76:3300/0 conn(0x7fc900132630 0x7fc900132a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:42982/0 (socket says 172.21.15.16:42982) 2019-12-05T06:36:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.048+0000 7fc905089700 1 -- 172.21.15.16:0/4124787495 learned_addr learned my addr 172.21.15.16:0/4124787495 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 -- 172.21.15.16:0/4124787495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc90009ae80 msgr2=0x7fc900132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc90009ae80 0x7fc900132f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 -- 172.21.15.16:0/4124787495 >> v2:172.21.15.76:3300/0 conn(0x7fc900132630 msgr2=0x7fc900132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> v2:172.21.15.76:3300/0 conn(0x7fc900132630 0x7fc900132a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 -- 172.21.15.16:0/4124787495 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9000995b0 con 0x7fc900099eb0 2019-12-05T06:36:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> v2:172.21.15.16:3300/0 conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> v2:172.21.15.16:3300/0 conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=READY pgs=1037 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=THROTTLE_DONE pgs=1037 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=THROTTLE_DONE pgs=1037 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc8fd7fa700 1 -- 172.21.15.16:0/4124787495 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f0000bc0 con 0x7fc900099eb0 2019-12-05T06:36:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=THROTTLE_DONE pgs=1037 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.049+0000 7fc8fd7fa700 1 -- 172.21.15.16:0/4124787495 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f00010e0 con 0x7fc900099eb0 2019-12-05T06:36:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.050+0000 7fc9062eb700 1 -- 172.21.15.16:0/4124787495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9000995b0 con 0x7fc900099eb0 2019-12-05T06:36:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.050+0000 7fc8fd7fa700 1 -- 172.21.15.16:0/4124787495 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8f00017b0 con 0x7fc900099eb0 2019-12-05T06:36:09.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.050+0000 7fc9062eb700 1 -- 172.21.15.16:0/4124787495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9000995b0 con 0x7fc900099eb0 2019-12-05T06:36:09.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.050+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=THROTTLE_DONE pgs=1037 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.051+0000 7fc8fd7fa700 1 -- 172.21.15.16:0/4124787495 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc8f00012d0 con 0x7fc900099eb0 2019-12-05T06:36:09.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.051+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=THROTTLE_DONE pgs=1037 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.052+0000 7fc8fd7fa700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 0x7fc8e4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:09.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.052+0000 7fc8fd7fa700 1 -- 172.21.15.16:0/4124787495 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fc8f004a120 con 0x7fc900099eb0 2019-12-05T06:36:09.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.052+0000 7fc904888700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 0x7fc8e4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:09.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.053+0000 7fc904888700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 0x7fc8e4025ac0 crc :-1 s=READY pgs=446 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:09.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.065+0000 7fc9062eb700 1 -- 172.21.15.16:0/4124787495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9000995b0 con 0x7fc900099eb0 2019-12-05T06:36:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.075+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=THROTTLE_DONE pgs=1037 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.075+0000 7fc8fd7fa700 1 -- 172.21.15.16:0/4124787495 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc8f003cf30 con 0x7fc900099eb0 2019-12-05T06:36:09.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.327+0000 7fc9062eb700 1 -- 172.21.15.16:0/4124787495 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc9000995b0 con 0x7fc900099eb0 2019-12-05T06:36:09.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.327+0000 7fc905089700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=THROTTLE_DONE pgs=1037 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.327+0000 7fc8fd7fa700 1 -- 172.21.15.16:0/4124787495 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fc8f003cf30 con 0x7fc900099eb0 2019-12-05T06:36:09.328 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 131 2019-12-05T06:36:09.328 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:09.329 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:09.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 -- 172.21.15.16:0/4124787495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 msgr2=0x7fc8e4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 0x7fc8e4025ac0 crc :-1 s=READY pgs=446 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 -- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 msgr2=0x7fc90009a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 0x7fc90009a370 secure :-1 s=READY pgs=1037 cs=0 l=1 rx=0x7fc8f00147f0 tx=0x7fc8f0014670).stop 2019-12-05T06:36:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 -- 172.21.15.16:0/4124787495 shutdown_connections 2019-12-05T06:36:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc8e4023630 0x7fc8e4025ac0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc900099eb0 0x7fc90009a370 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 --2- 172.21.15.16:0/4124787495 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc90009ae80 0x7fc900132f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 --2- 172.21.15.16:0/4124787495 >> v2:172.21.15.76:3300/0 conn(0x7fc900132630 0x7fc900132a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 -- 172.21.15.16:0/4124787495 shutdown_connections 2019-12-05T06:36:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 -- 172.21.15.16:0/4124787495 wait complete. 2019-12-05T06:36:09.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.330+0000 7fc9062eb700 1 -- 172.21.15.16:0/4124787495 >> 172.21.15.16:0/4124787495 conn(0x7fc9000714f0 msgr2=0x7fc90009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:09.342 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5426 2019-12-05T06:36:09.343 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 0 secs before reviving mds.b 2019-12-05T06:36:09.343 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:36:09.344 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:36:09.344 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:36:09.350 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:36:09.350 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:09.526 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:36:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.548+0000 7fef13efd700 1 Processor -- start 2019-12-05T06:36:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.548+0000 7fef13efd700 1 -- start start 2019-12-05T06:36:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.549+0000 7fef13efd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef0c08c6d0 0x7fef0c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.549+0000 7fef13efd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fef0c08f720 0x7fef0c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:09.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.549+0000 7fef13efd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c08fb70 con 0x7fef0c08fef0 2019-12-05T06:36:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.549+0000 7fef13efd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0c08df90 con 0x7fef0c08c6d0 2019-12-05T06:36:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.549+0000 7fef13efd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef0c0a3f70 con 0x7fef0c08f720 2019-12-05T06:36:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.549+0000 7fef1249a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef0c08c6d0 0x7fef0c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.549+0000 7fef1249a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef0c08c6d0 0x7fef0c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:56902/0 (socket says 172.21.15.16:56902) 2019-12-05T06:36:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.549+0000 7fef1249a700 1 -- 172.21.15.16:0/1675689149 learned_addr learned my addr 172.21.15.16:0/1675689149 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.549+0000 7fef12c9b700 1 --2- 172.21.15.16:0/1675689149 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fef0c08f720 0x7fef0c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:09.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef1249a700 1 -- 172.21.15.16:0/1675689149 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fef0c08f720 msgr2=0x7fef0c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef1249a700 1 --2- 172.21.15.16:0/1675689149 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fef0c08f720 0x7fef0c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef1249a700 1 -- 172.21.15.16:0/1675689149 >> v1:172.21.15.76:6789/0 conn(0x7fef0c08fef0 legacy=0x7fef0c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef1249a700 1 -- 172.21.15.16:0/1675689149 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c08d540 con 0x7fef0c08c6d0 2019-12-05T06:36:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef1249a700 1 --2- 172.21.15.16:0/1675689149 >> v2:172.21.15.16:3300/0 conn(0x7fef0c08c6d0 0x7fef0c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fef0800fac0 tx=0x7fef0800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef1249a700 1 --2- 172.21.15.16:0/1675689149 >> v2:172.21.15.16:3300/0 conn(0x7fef0c08c6d0 0x7fef0c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fef0800fac0 tx=0x7fef0800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef1249a700 1 --2- 172.21.15.16:0/1675689149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c08c6d0 0x7fef0c08cac0 secure :-1 s=READY pgs=1039 cs=0 l=1 rx=0x7fef0800fac0 tx=0x7fef0800f8f0).ready entity=mon.0 client_cookie=adbc2f058478277d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef1249a700 1 --2- 172.21.15.16:0/1675689149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c08c6d0 0x7fef0c08cac0 secure :-1 s=THROTTLE_DONE pgs=1039 cs=0 l=1 rx=0x7fef0800fac0 tx=0x7fef0800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef1249a700 1 --2- 172.21.15.16:0/1675689149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c08c6d0 0x7fef0c08cac0 secure :-1 s=THROTTLE_DONE pgs=1039 cs=0 l=1 rx=0x7fef0800fac0 tx=0x7fef0800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef11498700 1 -- 172.21.15.16:0/1675689149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef08000ce0 con 0x7fef0c08c6d0 2019-12-05T06:36:09.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef1249a700 1 --2- 172.21.15.16:0/1675689149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c08c6d0 0x7fef0c08cac0 secure :-1 s=THROTTLE_DONE pgs=1039 cs=0 l=1 rx=0x7fef0800fac0 tx=0x7fef0800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef11498700 1 -- 172.21.15.16:0/1675689149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef08001010 con 0x7fef0c08c6d0 2019-12-05T06:36:09.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef11498700 1 -- 172.21.15.16:0/1675689149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef08001720 con 0x7fef0c08c6d0 2019-12-05T06:36:09.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef13efd700 1 -- 172.21.15.16:0/1675689149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c08c6d0 msgr2=0x7fef0c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.550+0000 7fef13efd700 1 --2- 172.21.15.16:0/1675689149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c08c6d0 0x7fef0c08cac0 secure :-1 s=READY pgs=1039 cs=0 l=1 rx=0x7fef0800fac0 tx=0x7fef0800f8f0).stop 2019-12-05T06:36:09.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.551+0000 7fef13efd700 1 -- 172.21.15.16:0/1675689149 shutdown_connections 2019-12-05T06:36:09.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.551+0000 7fef13efd700 1 --2- 172.21.15.16:0/1675689149 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fef0c08f720 0x7fef0c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:09.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.551+0000 7fef13efd700 1 --2- 172.21.15.16:0/1675689149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c08c6d0 0x7fef0c08cac0 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.551+0000 7fef13efd700 1 -- 172.21.15.16:0/1675689149 shutdown_connections 2019-12-05T06:36:09.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.551+0000 7fef13efd700 1 -- 172.21.15.16:0/1675689149 wait complete. 2019-12-05T06:36:09.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.551+0000 7fef13efd700 1 -- 172.21.15.16:0/1675689149 >> 172.21.15.16:0/1675689149 conn(0x7fef0c061940 msgr2=0x7fef0c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:09.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.554+0000 7fef13efd700 1 Processor -- start 2019-12-05T06:36:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.554+0000 7fef13efd700 1 -- start start 2019-12-05T06:36:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef13efd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef0c09bbe0 0x7fef0c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef13efd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c084b90 con 0x7fef0c084f30 2019-12-05T06:36:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef13efd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef0c098790 con 0x7fef0c096e90 2019-12-05T06:36:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef13efd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef0c09a450 con 0x7fef0c09bbe0 2019-12-05T06:36:09.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef11c99700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fef0c09bbe0 0x7fef0c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef12c9b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fef0c084f30 0x7fef0c155f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42076/0 (socket says 172.21.15.16:42076) 2019-12-05T06:36:09.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef12c9b700 1 -- 172.21.15.16:0/2232585825 learned_addr learned my addr 172.21.15.16:0/2232585825 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef11c99700 1 -- 172.21.15.16:0/2232585825 >> v1:172.21.15.76:6789/0 conn(0x7fef0c096e90 legacy=0x7fef0c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef11c99700 1 -- 172.21.15.16:0/2232585825 >> v1:172.21.15.16:6789/0 conn(0x7fef0c084f30 legacy=0x7fef0c155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef11c99700 1 -- 172.21.15.16:0/2232585825 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0c09aae0 con 0x7fef0c09bbe0 2019-12-05T06:36:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef02ffd700 1 -- 172.21.15.16:0/2232585825 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3216392212 0 0) 0x7fef04001150 con 0x7fef0c084f30 2019-12-05T06:36:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef11c99700 1 --2- 172.21.15.16:0/2232585825 >> v2:172.21.15.16:3300/0 conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef11c99700 1 --2- 172.21.15.16:0/2232585825 >> v2:172.21.15.16:3300/0 conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef11c99700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=READY pgs=1041 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:09.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef11c99700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef02ffd700 1 -- 172.21.15.16:0/2232585825 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feefc009790 con 0x7fef0c09bbe0 2019-12-05T06:36:09.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef11c99700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef13efd700 1 -- 172.21.15.16:0/2232585825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef0c09aae0 con 0x7fef0c09bbe0 2019-12-05T06:36:09.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef02ffd700 1 -- 172.21.15.16:0/2232585825 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feefc00a6b0 con 0x7fef0c09bbe0 2019-12-05T06:36:09.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.555+0000 7fef11c99700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.556+0000 7fef02ffd700 1 -- 172.21.15.16:0/2232585825 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feefc000c10 con 0x7fef0c09bbe0 2019-12-05T06:36:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.556+0000 7fef13efd700 1 -- 172.21.15.16:0/2232585825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef0c09aae0 con 0x7fef0c09bbe0 2019-12-05T06:36:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.556+0000 7fef11c99700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.557+0000 7fef02ffd700 1 -- 172.21.15.16:0/2232585825 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7feefc009790 con 0x7fef0c09bbe0 2019-12-05T06:36:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.557+0000 7fef11c99700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.557+0000 7fef02ffd700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feeec023630 0x7feeec025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.557+0000 7fef02ffd700 1 -- 172.21.15.16:0/2232585825 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7feefc045df0 con 0x7fef0c09bbe0 2019-12-05T06:36:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.557+0000 7fef1249a700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feeec023630 0x7feeec025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:09.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.557+0000 7fef1249a700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feeec023630 0x7feeec025ac0 crc :-1 s=READY pgs=447 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.564+0000 7fef13efd700 1 -- 172.21.15.16:0/2232585825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef0c09aae0 con 0x7fef0c09bbe0 2019-12-05T06:36:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.568+0000 7fef11c99700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.568+0000 7fef02ffd700 1 -- 172.21.15.16:0/2232585825 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7feefc009580 con 0x7fef0c09bbe0 2019-12-05T06:36:09.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.768+0000 7fef13efd700 1 -- 172.21.15.16:0/2232585825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef0c09aae0 con 0x7fef0c09bbe0 2019-12-05T06:36:09.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.769+0000 7fef11c99700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:09.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.769+0000 7fef02ffd700 1 -- 172.21.15.16:0/2232585825 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+1775 (secure 0 0 0) 0x7feefc009790 con 0x7fef0c09bbe0 2019-12-05T06:36:09.776 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 131 2019-12-05T06:36:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.771+0000 7fef13efd700 1 -- 172.21.15.16:0/2232585825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feeec023630 msgr2=0x7feeec025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.771+0000 7fef13efd700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feeec023630 0x7feeec025ac0 crc :-1 s=READY pgs=447 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.771+0000 7fef13efd700 1 -- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 msgr2=0x7fef0c1539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:09.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.771+0000 7fef13efd700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 0x7fef0c1539e0 secure :-1 s=READY pgs=1041 cs=0 l=1 rx=0x7feefc0103a0 tx=0x7feefc010740).stop 2019-12-05T06:36:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.771+0000 7fef13efd700 1 -- 172.21.15.16:0/2232585825 shutdown_connections 2019-12-05T06:36:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.771+0000 7fef13efd700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feeec023630 0x7feeec025ac0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.771+0000 7fef13efd700 1 --2- 172.21.15.16:0/2232585825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fef0c09bbe0 0x7fef0c1539e0 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:09.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.772+0000 7fef13efd700 1 -- 172.21.15.16:0/2232585825 shutdown_connections 2019-12-05T06:36:09.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.772+0000 7fef13efd700 1 -- 172.21.15.16:0/2232585825 wait complete. 2019-12-05T06:36:09.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:09.772+0000 7fef13efd700 1 -- 172.21.15.16:0/2232585825 >> 172.21.15.16:0/2232585825 conn(0x7fef0c0714f0 msgr2=0x7fef0c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:09.779 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:09.779 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:09.783 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:36:11.784 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:12.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.027+0000 7f83d607f700 1 Processor -- start 2019-12-05T06:36:12.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.027+0000 7f83d607f700 1 -- start start 2019-12-05T06:36:12.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.028+0000 7f83d607f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f83d008d2c0 0x7f83d008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:12.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.028+0000 7f83d607f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d00958e0 con 0x7f83d008fef0 2019-12-05T06:36:12.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.028+0000 7f83d607f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d00629e0 con 0x7f83d008c6d0 2019-12-05T06:36:12.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.028+0000 7f83d607f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d008df90 con 0x7f83d008d2c0 2019-12-05T06:36:12.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.028+0000 7f83d4e1d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f83d008fef0 0x7f83d008f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42100/0 (socket says 172.21.15.16:42100) 2019-12-05T06:36:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.028+0000 7f83d4e1d700 1 -- 172.21.15.16:0/260974583 learned_addr learned my addr 172.21.15.16:0/260974583 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.028+0000 7f83c7fff700 1 --2- 172.21.15.16:0/260974583 >> v2:172.21.15.76:3300/0 conn(0x7f83d008d2c0 0x7f83d008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.028+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409320316 0 0) 0x7f83c80014d0 con 0x7f83d008fef0 2019-12-05T06:36:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83bc002e80 con 0x7f83d008fef0 2019-12-05T06:36:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2970161622 0 0) 0x7f83c80014b0 con 0x7f83d008fef0 2019-12-05T06:36:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 >> v2:172.21.15.76:3300/0 conn(0x7f83d008d2c0 msgr2=0x7f83d008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:12.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 --2- 172.21.15.16:0/260974583 >> v2:172.21.15.76:3300/0 conn(0x7f83d008d2c0 0x7f83d008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 >> v1:172.21.15.76:6789/0 conn(0x7f83d008c6d0 legacy=0x7f83d008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d008e620 con 0x7f83d008fef0 2019-12-05T06:36:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f83c8001a40 con 0x7f83d008fef0 2019-12-05T06:36:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 >> v1:172.21.15.16:6789/0 conn(0x7f83d008fef0 legacy=0x7f83d008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:12.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f83bc003a80 0x7f83bc005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 0x7f83bc008b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f83bc0093a0 0x7f83bc00b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc00c020 con 0x7f83bc006690 2019-12-05T06:36:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc050220 con 0x7f83bc0093a0 2019-12-05T06:36:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83bc050450 con 0x7f83bc003a80 2019-12-05T06:36:12.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83c7fff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 0x7f83bc008b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f83bc0093a0 0x7f83bc00b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83c7fff700 1 -- 172.21.15.16:0/260974583 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f83bc003a80 msgr2=0x7f83bc005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83cffff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f83bc003a80 0x7f83bc005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83c7fff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f83bc003a80 0x7f83bc005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:12.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83c7fff700 1 -- 172.21.15.16:0/260974583 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f83bc0093a0 msgr2=0x7f83bc00b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83c7fff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f83bc0093a0 0x7f83bc00b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.029+0000 7f83c7fff700 1 -- 172.21.15.16:0/260974583 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83bc050700 con 0x7f83bc006690 2019-12-05T06:36:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83cffff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f83bc003a80 0x7f83bc005ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:36:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f83bc0093a0 0x7f83bc00b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:36:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83d4e1d700 1 -- 172.21.15.16:0/260974583 reap_dead start 2019-12-05T06:36:12.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83c7fff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 0x7f83bc008b40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f83b800b6e0 tx=0x7f83b800ba80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83c7fff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 0x7f83bc008b40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f83b800b6e0 tx=0x7f83b800ba80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83c7fff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 0x7f83bc008b40 secure :-1 s=READY pgs=1043 cs=0 l=1 rx=0x7f83b800b6e0 tx=0x7f83b800ba80).ready entity=mon.0 client_cookie=41ac7628bc522b2b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83c7fff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 0x7f83bc008b40 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f83b800b6e0 tx=0x7f83b800ba80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83c7fff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 0x7f83bc008b40 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f83b800b6e0 tx=0x7f83b800ba80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83b8000960 con 0x7f83bc006690 2019-12-05T06:36:12.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83c7fff700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 0x7f83bc008b40 secure :-1 s=THROTTLE_DONE pgs=1043 cs=0 l=1 rx=0x7f83b800b6e0 tx=0x7f83b800ba80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83b8009770 con 0x7f83bc006690 2019-12-05T06:36:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83cf7fe700 1 -- 172.21.15.16:0/260974583 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83b8009e00 con 0x7f83bc006690 2019-12-05T06:36:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83d607f700 1 -- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 msgr2=0x7f83bc008b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.030+0000 7f83d607f700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 0x7f83bc008b40 secure :-1 s=READY pgs=1043 cs=0 l=1 rx=0x7f83b800b6e0 tx=0x7f83b800ba80).stop 2019-12-05T06:36:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.031+0000 7f83d607f700 1 -- 172.21.15.16:0/260974583 shutdown_connections 2019-12-05T06:36:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.031+0000 7f83d607f700 1 --2- 172.21.15.16:0/260974583 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83bc006690 0x7f83bc008b40 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:12.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.031+0000 7f83d607f700 1 -- 172.21.15.16:0/260974583 shutdown_connections 2019-12-05T06:36:12.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.031+0000 7f83d607f700 1 -- 172.21.15.16:0/260974583 wait complete. 2019-12-05T06:36:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.031+0000 7f83d607f700 1 -- 172.21.15.16:0/260974583 >> 172.21.15.16:0/260974583 conn(0x7f83d0061940 msgr2=0x7f83d0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.033+0000 7f83d607f700 1 Processor -- start 2019-12-05T06:36:12.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.034+0000 7f83d607f700 1 -- start start 2019-12-05T06:36:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.034+0000 7f83d607f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f83d009e000 0x7f83d015a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.034+0000 7f83d607f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d009e440 con 0x7f83d009e7d0 2019-12-05T06:36:12.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.034+0000 7f83d607f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83d012cb90 con 0x7f83d01302f0 2019-12-05T06:36:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.034+0000 7f83d607f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83d0131ff0 con 0x7f83d009e000 2019-12-05T06:36:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.034+0000 7f83c7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f83d009e7d0 0x7f83d0156e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42110/0 (socket says 172.21.15.16:42110) 2019-12-05T06:36:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.034+0000 7f83c7fff700 1 -- 172.21.15.16:0/1190067554 learned_addr learned my addr 172.21.15.16:0/1190067554 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.034+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> v2:172.21.15.16:3300/0 conn(0x7f83d009e000 0x7f83d015a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d4e1d700 1 -- 172.21.15.16:0/1190067554 >> v1:172.21.15.76:6789/0 conn(0x7f83d01302f0 legacy=0x7f83d009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:12.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d4e1d700 1 -- 172.21.15.16:0/1190067554 >> v1:172.21.15.16:6789/0 conn(0x7f83d009e7d0 legacy=0x7f83d0156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d4e1d700 1 -- 172.21.15.16:0/1190067554 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d0132680 con 0x7f83d009e000 2019-12-05T06:36:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83cd7fa700 1 -- 172.21.15.16:0/1190067554 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415308338 0 0) 0x7f83b800b6b0 con 0x7f83d009e7d0 2019-12-05T06:36:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> v2:172.21.15.16:3300/0 conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> v2:172.21.15.16:3300/0 conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=READY pgs=1045 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:12.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=THROTTLE_DONE pgs=1045 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83cd7fa700 1 -- 172.21.15.16:0/1190067554 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83c8001a70 con 0x7f83d009e000 2019-12-05T06:36:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=THROTTLE_DONE pgs=1045 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=THROTTLE_DONE pgs=1045 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83cd7fa700 1 -- 172.21.15.16:0/1190067554 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f83c80045c0 con 0x7f83d009e000 2019-12-05T06:36:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83cd7fa700 1 -- 172.21.15.16:0/1190067554 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83c8004d10 con 0x7f83d009e000 2019-12-05T06:36:12.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d607f700 1 -- 172.21.15.16:0/1190067554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83d0132680 con 0x7f83d009e000 2019-12-05T06:36:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.035+0000 7f83d607f700 1 -- 172.21.15.16:0/1190067554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83d0132680 con 0x7f83d009e000 2019-12-05T06:36:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.036+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=THROTTLE_DONE pgs=1045 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.036+0000 7f83cd7fa700 1 -- 172.21.15.16:0/1190067554 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f83c80047b0 con 0x7f83d009e000 2019-12-05T06:36:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.036+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=THROTTLE_DONE pgs=1045 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.036+0000 7f83cd7fa700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f83b4023630 0x7f83b4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:12.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.037+0000 7f83cd7fa700 1 -- 172.21.15.16:0/1190067554 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f83c80460d0 con 0x7f83d009e000 2019-12-05T06:36:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.037+0000 7f83cffff700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f83b4023630 0x7f83b4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.037+0000 7f83cffff700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f83b4023630 0x7f83b4025ac0 crc :-1 s=READY pgs=448 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.042+0000 7f83d607f700 1 -- 172.21.15.16:0/1190067554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83d0132680 con 0x7f83d009e000 2019-12-05T06:36:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.046+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=THROTTLE_DONE pgs=1045 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.046+0000 7f83cd7fa700 1 -- 172.21.15.16:0/1190067554 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f83c8032f80 con 0x7f83d009e000 2019-12-05T06:36:12.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.254+0000 7f83d607f700 1 -- 172.21.15.16:0/1190067554 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f83d0132680 con 0x7f83d009e000 2019-12-05T06:36:12.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.254+0000 7f83d4e1d700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=THROTTLE_DONE pgs=1045 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:12.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.254+0000 7f83cd7fa700 1 -- 172.21.15.16:0/1190067554 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 131 v131) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f83c8032f80 con 0x7f83d009e000 2019-12-05T06:36:12.254 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 131 2019-12-05T06:36:12.254 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:12.254 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":131,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.257+0000 7f83d607f700 1 -- 172.21.15.16:0/1190067554 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f83b4023630 msgr2=0x7f83b4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.257+0000 7f83d607f700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f83b4023630 0x7f83b4025ac0 crc :-1 s=READY pgs=448 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.257+0000 7f83d607f700 1 -- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 msgr2=0x7f83d015a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.257+0000 7f83d607f700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 0x7f83d015a690 secure :-1 s=READY pgs=1045 cs=0 l=1 rx=0x7f83c80106c0 tx=0x7f83c8010a60).stop 2019-12-05T06:36:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.257+0000 7f83d607f700 1 -- 172.21.15.16:0/1190067554 shutdown_connections 2019-12-05T06:36:12.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.257+0000 7f83d607f700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f83b4023630 0x7f83b4025ac0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.257+0000 7f83d607f700 1 --2- 172.21.15.16:0/1190067554 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f83d009e000 0x7f83d015a690 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.258+0000 7f83d607f700 1 -- 172.21.15.16:0/1190067554 shutdown_connections 2019-12-05T06:36:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.258+0000 7f83d607f700 1 -- 172.21.15.16:0/1190067554 wait complete. 2019-12-05T06:36:12.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:12.258+0000 7f83d607f700 1 -- 172.21.15.16:0/1190067554 >> 172.21.15.16:0/1190067554 conn(0x7f83d00a2850 msgr2=0x7f83d006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:12.267 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:36:14.267 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:14.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.462+0000 7fed95811700 1 Processor -- start 2019-12-05T06:36:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.462+0000 7fed95811700 1 -- start start 2019-12-05T06:36:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.463+0000 7fed95811700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fed9008d2c0 0x7fed9008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.463+0000 7fed95811700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed900958e0 con 0x7fed9008c6d0 2019-12-05T06:36:14.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.463+0000 7fed95811700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed900629e0 con 0x7fed9008fef0 2019-12-05T06:36:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.463+0000 7fed95811700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed9008df90 con 0x7fed9008d2c0 2019-12-05T06:36:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.463+0000 7fed8f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fed9008c6d0 0x7fed9008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42120/0 (socket says 172.21.15.16:42120) 2019-12-05T06:36:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.463+0000 7fed8f7fe700 1 -- 172.21.15.16:0/3744655615 learned_addr learned my addr 172.21.15.16:0/3744655615 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.463+0000 7fed8effd700 1 --2- 172.21.15.16:0/3744655615 >> v2:172.21.15.76:3300/0 conn(0x7fed9008d2c0 0x7fed9008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.463+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3534867938 0 0) 0x7fed80001150 con 0x7fed9008c6d0 2019-12-05T06:36:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.463+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed78002e80 con 0x7fed9008c6d0 2019-12-05T06:36:14.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 271407370 0 0) 0x7fed80001130 con 0x7fed9008c6d0 2019-12-05T06:36:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 >> v2:172.21.15.76:3300/0 conn(0x7fed9008d2c0 msgr2=0x7fed9008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 --2- 172.21.15.16:0/3744655615 >> v2:172.21.15.76:3300/0 conn(0x7fed9008d2c0 0x7fed9008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 >> v1:172.21.15.76:6789/0 conn(0x7fed9008fef0 legacy=0x7fed9008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed9008e620 con 0x7fed9008c6d0 2019-12-05T06:36:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fed800016c0 con 0x7fed9008c6d0 2019-12-05T06:36:14.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 >> v1:172.21.15.16:6789/0 conn(0x7fed9008c6d0 legacy=0x7fed9008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed78003a80 0x7fed78005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 0x7fed78008b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed780093a0 0x7fed7800b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7800c020 con 0x7fed78006690 2019-12-05T06:36:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed78050220 con 0x7fed78003a80 2019-12-05T06:36:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed78050450 con 0x7fed780093a0 2019-12-05T06:36:14.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8effd700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 0x7fed78008b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:14.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed78003a80 0x7fed78005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:14.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8ffff700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed780093a0 0x7fed7800b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:14.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8effd700 1 -- 172.21.15.16:0/3744655615 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed780093a0 msgr2=0x7fed7800b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:14.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8effd700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed780093a0 0x7fed7800b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:14.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8effd700 1 -- 172.21.15.16:0/3744655615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed78003a80 msgr2=0x7fed78005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:14.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8effd700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed78003a80 0x7fed78005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:14.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8effd700 1 -- 172.21.15.16:0/3744655615 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed78050700 con 0x7fed78006690 2019-12-05T06:36:14.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed78003a80 0x7fed78005ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:36:14.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8ffff700 1 -- 172.21.15.16:0/3744655615 reap_dead start 2019-12-05T06:36:14.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8effd700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 0x7fed78008b40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fed8400b6e0 tx=0x7fed8400b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8effd700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 0x7fed78008b40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fed8400b6e0 tx=0x7fed8400b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.464+0000 7fed8effd700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 0x7fed78008b40 secure :-1 s=READY pgs=1048 cs=0 l=1 rx=0x7fed8400b6e0 tx=0x7fed8400b530).ready entity=mon.0 client_cookie=a1d16e3271f023a0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:14.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.465+0000 7fed8effd700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 0x7fed78008b40 secure :-1 s=THROTTLE_DONE pgs=1048 cs=0 l=1 rx=0x7fed8400b6e0 tx=0x7fed8400b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.465+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed84000960 con 0x7fed78006690 2019-12-05T06:36:14.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.465+0000 7fed8effd700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 0x7fed78008b40 secure :-1 s=THROTTLE_DONE pgs=1048 cs=0 l=1 rx=0x7fed8400b6e0 tx=0x7fed8400b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.465+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed84009770 con 0x7fed78006690 2019-12-05T06:36:14.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.465+0000 7fed8effd700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 0x7fed78008b40 secure :-1 s=THROTTLE_DONE pgs=1048 cs=0 l=1 rx=0x7fed8400b6e0 tx=0x7fed8400b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.465+0000 7fed8e7fc700 1 -- 172.21.15.16:0/3744655615 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed84000960 con 0x7fed78006690 2019-12-05T06:36:14.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.465+0000 7fed95811700 1 -- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 msgr2=0x7fed78008b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:14.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.465+0000 7fed95811700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 0x7fed78008b40 secure :-1 s=READY pgs=1048 cs=0 l=1 rx=0x7fed8400b6e0 tx=0x7fed8400b530).stop 2019-12-05T06:36:14.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.465+0000 7fed95811700 1 -- 172.21.15.16:0/3744655615 shutdown_connections 2019-12-05T06:36:14.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.465+0000 7fed95811700 1 --2- 172.21.15.16:0/3744655615 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed78006690 0x7fed78008b40 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:14.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.466+0000 7fed95811700 1 -- 172.21.15.16:0/3744655615 shutdown_connections 2019-12-05T06:36:14.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.466+0000 7fed95811700 1 -- 172.21.15.16:0/3744655615 wait complete. 2019-12-05T06:36:14.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.466+0000 7fed95811700 1 -- 172.21.15.16:0/3744655615 >> 172.21.15.16:0/3744655615 conn(0x7fed90061940 msgr2=0x7fed90096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:14.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.468+0000 7fed95811700 1 Processor -- start 2019-12-05T06:36:14.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.468+0000 7fed95811700 1 -- start start 2019-12-05T06:36:14.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.469+0000 7fed95811700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9006e840 0x7fed90134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.469+0000 7fed95811700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed90128af0 con 0x7fed9006dc50 2019-12-05T06:36:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.469+0000 7fed95811700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed90128600 con 0x7fed90135450 2019-12-05T06:36:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.469+0000 7fed95811700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed90132830 con 0x7fed9006e840 2019-12-05T06:36:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.469+0000 7fed8f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fed9006dc50 0x7fed9006e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42132/0 (socket says 172.21.15.16:42132) 2019-12-05T06:36:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.469+0000 7fed8f7fe700 1 -- 172.21.15.16:0/841448837 learned_addr learned my addr 172.21.15.16:0/841448837 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:14.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.469+0000 7fed8effd700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9006e840 0x7fed90134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:14.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3823481843 0 0) 0x7fed80000940 con 0x7fed9006dc50 2019-12-05T06:36:14.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed70002e80 con 0x7fed9006dc50 2019-12-05T06:36:14.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623121880 0 0) 0x7fed88000f10 con 0x7fed90135450 2019-12-05T06:36:14.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed700030b0 con 0x7fed90135450 2019-12-05T06:36:14.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2945869147 0 0) 0x7fed800012b0 con 0x7fed9006dc50 2019-12-05T06:36:14.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed700045e0 con 0x7fed9006dc50 2019-12-05T06:36:14.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fed800017c0 con 0x7fed9006dc50 2019-12-05T06:36:14.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9006e840 msgr2=0x7fed90134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:14.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed9006e840 0x7fed90134480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:14.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 >> v1:172.21.15.76:6789/0 conn(0x7fed90135450 legacy=0x7fed901349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:14.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 >> v1:172.21.15.16:6789/0 conn(0x7fed9006dc50 legacy=0x7fed9006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:14.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:14.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed700041e0 0x7fed70004650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:14.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed70004e70 0x7fed7000b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:14.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed70005540 con 0x7fed70003820 2019-12-05T06:36:14.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7000be80 con 0x7fed700041e0 2019-12-05T06:36:14.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed7000c040 con 0x7fed70004e70 2019-12-05T06:36:14.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:14.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.470+0000 7fed8effd700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed700041e0 0x7fed70004650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:14.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 -- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed70004e70 msgr2=0x7fed7000b940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fed70004e70 0x7fed7000b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 -- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed700041e0 msgr2=0x7fed70004650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fed700041e0 0x7fed70004650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 -- 172.21.15.16:0/841448837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7000c2b0 con 0x7fed70003820 2019-12-05T06:36:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed95811700 1 -- 172.21.15.16:0/841448837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed90133b80 con 0x7fed70003820 2019-12-05T06:36:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed95811700 1 -- 172.21.15.16:0/841448837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed90128af0 con 0x7fed70003820 2019-12-05T06:36:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=READY pgs=1050 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed80000940 con 0x7fed70003820 2019-12-05T06:36:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80004240 con 0x7fed70003820 2019-12-05T06:36:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed800048d0 con 0x7fed70003820 2019-12-05T06:36:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.471+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.472+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fed80004430 con 0x7fed70003820 2019-12-05T06:36:14.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.472+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.472+0000 7fed77fff700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed7002ba90 0x7fed7002dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:14.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.472+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fed80045d90 con 0x7fed70003820 2019-12-05T06:36:14.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.472+0000 7fed8effd700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed7002ba90 0x7fed7002dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:14.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.473+0000 7fed8effd700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed7002ba90 0x7fed7002dec0 crc :-1 s=READY pgs=450 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:14.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.477+0000 7fed95811700 1 -- 172.21.15.16:0/841448837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed68002e60 con 0x7fed70003820 2019-12-05T06:36:14.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.481+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.481+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fed80000e10 con 0x7fed70003820 2019-12-05T06:36:14.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.686+0000 7fed95811700 1 -- 172.21.15.16:0/841448837 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fed68001d10 con 0x7fed70003820 2019-12-05T06:36:14.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.686+0000 7fed8f7fe700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=THROTTLE_DONE pgs=1050 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:14.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.686+0000 7fed77fff700 1 -- 172.21.15.16:0/841448837 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fed80004430 con 0x7fed70003820 2019-12-05T06:36:14.685 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 132 2019-12-05T06:36:14.686 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:14.686 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:14.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.689+0000 7fed95811700 1 -- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed7002ba90 msgr2=0x7fed7002dec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:14.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.689+0000 7fed95811700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fed7002ba90 0x7fed7002dec0 crc :-1 s=READY pgs=450 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:14.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.689+0000 7fed95811700 1 -- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 msgr2=0x7fed70003c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:14.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.689+0000 7fed95811700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 secure :-1 s=READY pgs=1050 cs=0 l=1 rx=0x7fed80010460 tx=0x7fed80010310).stop 2019-12-05T06:36:14.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.689+0000 7fed8ffff700 1 -- 172.21.15.16:0/841448837 reap_dead start 2019-12-05T06:36:14.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.689+0000 7fed95811700 1 -- 172.21.15.16:0/841448837 shutdown_connections 2019-12-05T06:36:14.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.689+0000 7fed95811700 1 --2- 172.21.15.16:0/841448837 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fed70003820 0x7fed70003c50 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:14.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.689+0000 7fed95811700 1 -- 172.21.15.16:0/841448837 shutdown_connections 2019-12-05T06:36:14.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.689+0000 7fed95811700 1 -- 172.21.15.16:0/841448837 wait complete. 2019-12-05T06:36:14.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:14.689+0000 7fed95811700 1 -- 172.21.15.16:0/841448837 >> 172.21.15.16:0/841448837 conn(0x7fed900a2850 msgr2=0x7fed9006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:14.701 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:36:14.701 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 38 secs before thrashing 2019-12-05T06:36:38.121 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:36:38.126 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:36:52.702 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.915+0000 7f76c6be4700 1 Processor -- start 2019-12-05T06:36:52.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.915+0000 7f76c6be4700 1 -- start start 2019-12-05T06:36:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c6be4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76c008c6d0 0x7f76c008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c6be4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c008fb80 con 0x7f76c008f720 2019-12-05T06:36:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c6be4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c00958e0 con 0x7f76c008fef0 2019-12-05T06:36:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c6be4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c008df90 con 0x7f76c008c6d0 2019-12-05T06:36:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c5181700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76c008c6d0 0x7f76c008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:52.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c5181700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76c008c6d0 0x7f76c008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57150/0 (socket says 172.21.15.16:57150) 2019-12-05T06:36:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c5181700 1 -- 172.21.15.16:0/1990710221 learned_addr learned my addr 172.21.15.16:0/1990710221 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c5181700 1 -- 172.21.15.16:0/1990710221 >> v1:172.21.15.76:6789/0 conn(0x7f76c008fef0 legacy=0x7f76c008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c5181700 1 -- 172.21.15.16:0/1990710221 >> v1:172.21.15.16:6789/0 conn(0x7f76c008f720 legacy=0x7f76c01533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76b7fff700 1 -- 172.21.15.16:0/1990710221 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039402914 0 0) 0x7f76b8000dc0 con 0x7f76c008f720 2019-12-05T06:36:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c5181700 1 -- 172.21.15.16:0/1990710221 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c008e620 con 0x7f76c008c6d0 2019-12-05T06:36:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.916+0000 7f76c5181700 1 --2- 172.21.15.16:0/1990710221 >> v2:172.21.15.16:3300/0 conn(0x7f76c008c6d0 0x7f76c008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f76b000f740 tx=0x7f76b000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.917+0000 7f76c5181700 1 --2- 172.21.15.16:0/1990710221 >> v2:172.21.15.16:3300/0 conn(0x7f76c008c6d0 0x7f76c008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f76b000f740 tx=0x7f76b000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.917+0000 7f76c5181700 1 --2- 172.21.15.16:0/1990710221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c008c6d0 0x7f76c008cac0 secure :-1 s=READY pgs=1060 cs=0 l=1 rx=0x7f76b000f740 tx=0x7f76b000fae0).ready entity=mon.0 client_cookie=9afffb56cb9e96af server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.917+0000 7f76c5181700 1 --2- 172.21.15.16:0/1990710221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c008c6d0 0x7f76c008cac0 secure :-1 s=THROTTLE_DONE pgs=1060 cs=0 l=1 rx=0x7f76b000f740 tx=0x7f76b000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.917+0000 7f76c5181700 1 --2- 172.21.15.16:0/1990710221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c008c6d0 0x7f76c008cac0 secure :-1 s=THROTTLE_DONE pgs=1060 cs=0 l=1 rx=0x7f76b000f740 tx=0x7f76b000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.917+0000 7f76b7fff700 1 -- 172.21.15.16:0/1990710221 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76b0000960 con 0x7f76c008c6d0 2019-12-05T06:36:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.917+0000 7f76c5181700 1 --2- 172.21.15.16:0/1990710221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c008c6d0 0x7f76c008cac0 secure :-1 s=THROTTLE_DONE pgs=1060 cs=0 l=1 rx=0x7f76b000f740 tx=0x7f76b000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.917+0000 7f76b7fff700 1 -- 172.21.15.16:0/1990710221 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b0000c90 con 0x7f76c008c6d0 2019-12-05T06:36:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.917+0000 7f76b7fff700 1 -- 172.21.15.16:0/1990710221 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76b00013e0 con 0x7f76c008c6d0 2019-12-05T06:36:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.917+0000 7f76c6be4700 1 -- 172.21.15.16:0/1990710221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c008c6d0 msgr2=0x7f76c008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.917+0000 7f76c6be4700 1 --2- 172.21.15.16:0/1990710221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c008c6d0 0x7f76c008cac0 secure :-1 s=READY pgs=1060 cs=0 l=1 rx=0x7f76b000f740 tx=0x7f76b000fae0).stop 2019-12-05T06:36:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.918+0000 7f76c6be4700 1 -- 172.21.15.16:0/1990710221 shutdown_connections 2019-12-05T06:36:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.918+0000 7f76c6be4700 1 --2- 172.21.15.16:0/1990710221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c008c6d0 0x7f76c008cac0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.918+0000 7f76c6be4700 1 -- 172.21.15.16:0/1990710221 shutdown_connections 2019-12-05T06:36:52.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.918+0000 7f76c6be4700 1 -- 172.21.15.16:0/1990710221 wait complete. 2019-12-05T06:36:52.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.918+0000 7f76c6be4700 1 -- 172.21.15.16:0/1990710221 >> 172.21.15.16:0/1990710221 conn(0x7f76c0061940 msgr2=0x7f76c0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:52.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.921+0000 7f76c6be4700 1 Processor -- start 2019-12-05T06:36:52.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.921+0000 7f76c6be4700 1 -- start start 2019-12-05T06:36:52.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.922+0000 7f76c6be4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76c006df10 0x7f76c009d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:52.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.922+0000 7f76c6be4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f76c0152e00 0x7f76c0153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:52.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.922+0000 7f76c6be4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76c006e880 con 0x7f76c009e580 2019-12-05T06:36:52.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.922+0000 7f76c6be4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c009c620 con 0x7f76c006df10 2019-12-05T06:36:52.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.922+0000 7f76c6be4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76c008ada0 con 0x7f76c0152e00 2019-12-05T06:36:52.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.922+0000 7f76c5181700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76c006df10 0x7f76c009d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:52.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.922+0000 7f76c5181700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f76c006df10 0x7f76c009d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57156/0 (socket says 172.21.15.16:57156) 2019-12-05T06:36:52.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.922+0000 7f76c5181700 1 -- 172.21.15.16:0/2962663254 learned_addr learned my addr 172.21.15.16:0/2962663254 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5982700 1 --2- 172.21.15.16:0/2962663254 >> v2:172.21.15.76:3300/0 conn(0x7f76c0152e00 0x7f76c0153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5181700 1 -- 172.21.15.16:0/2962663254 >> v2:172.21.15.76:3300/0 conn(0x7f76c0152e00 msgr2=0x7f76c0153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> v2:172.21.15.76:3300/0 conn(0x7f76c0152e00 0x7f76c0153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5181700 1 -- 172.21.15.16:0/2962663254 >> v1:172.21.15.16:6789/0 conn(0x7f76c009e580 legacy=0x7f76c009daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5181700 1 -- 172.21.15.16:0/2962663254 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c009ccb0 con 0x7f76c006df10 2019-12-05T06:36:52.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5982700 1 --2- 172.21.15.16:0/2962663254 >> v2:172.21.15.76:3300/0 conn(0x7f76c0152e00 0x7f76c0153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:36:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> v2:172.21.15.16:3300/0 conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> v2:172.21.15.16:3300/0 conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=THROTTLE_DONE pgs=1062 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=THROTTLE_DONE pgs=1062 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=THROTTLE_DONE pgs=1062 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76b5ffb700 1 -- 172.21.15.16:0/2962663254 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76b0005560 con 0x7f76c006df10 2019-12-05T06:36:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76b5ffb700 1 -- 172.21.15.16:0/2962663254 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76b0005180 con 0x7f76c006df10 2019-12-05T06:36:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c6be4700 1 -- 172.21.15.16:0/2962663254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c009ccb0 con 0x7f76c006df10 2019-12-05T06:36:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76b5ffb700 1 -- 172.21.15.16:0/2962663254 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76b0005b40 con 0x7f76c006df10 2019-12-05T06:36:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.923+0000 7f76c6be4700 1 -- 172.21.15.16:0/2962663254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c009ccb0 con 0x7f76c006df10 2019-12-05T06:36:52.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.924+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=THROTTLE_DONE pgs=1062 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.925+0000 7f76b5ffb700 1 -- 172.21.15.16:0/2962663254 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f76b0005370 con 0x7f76c006df10 2019-12-05T06:36:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.925+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=THROTTLE_DONE pgs=1062 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.925+0000 7f76b5ffb700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f76a0023630 0x7f76a0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.925+0000 7f76b5ffb700 1 -- 172.21.15.16:0/2962663254 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f76b0045e40 con 0x7f76c006df10 2019-12-05T06:36:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.925+0000 7f76c4980700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f76a0023630 0x7f76a0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:52.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.926+0000 7f76c4980700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f76a0023630 0x7f76a0025ac0 crc :-1 s=READY pgs=455 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:52.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.931+0000 7f76c6be4700 1 -- 172.21.15.16:0/2962663254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c009ccb0 con 0x7f76c006df10 2019-12-05T06:36:52.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.935+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=THROTTLE_DONE pgs=1062 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:52.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:52.935+0000 7f76b5ffb700 1 -- 172.21.15.16:0/2962663254 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f76b00072a0 con 0x7f76c006df10 2019-12-05T06:36:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.162+0000 7f76c6be4700 1 -- 172.21.15.16:0/2962663254 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f76c009ccb0 con 0x7f76c006df10 2019-12-05T06:36:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.162+0000 7f76c5181700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=THROTTLE_DONE pgs=1062 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.162+0000 7f76b5ffb700 1 -- 172.21.15.16:0/2962663254 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f76b0005350 con 0x7f76c006df10 2019-12-05T06:36:53.162 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 132 2019-12-05T06:36:53.162 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:53.162 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:53.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.165+0000 7f76c6be4700 1 -- 172.21.15.16:0/2962663254 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f76a0023630 msgr2=0x7f76a0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:53.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.165+0000 7f76c6be4700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f76a0023630 0x7f76a0025ac0 crc :-1 s=READY pgs=455 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.165+0000 7f76c6be4700 1 -- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 msgr2=0x7f76c009d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.165+0000 7f76c6be4700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 0x7f76c009d5b0 secure :-1 s=READY pgs=1062 cs=0 l=1 rx=0x7f76b0010440 tx=0x7f76b00107e0).stop 2019-12-05T06:36:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.165+0000 7f76c6be4700 1 -- 172.21.15.16:0/2962663254 shutdown_connections 2019-12-05T06:36:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.165+0000 7f76c6be4700 1 --2- 172.21.15.16:0/2962663254 >> v2:172.21.15.76:3300/0 conn(0x7f76c0152e00 0x7f76c0153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.165+0000 7f76c6be4700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f76a0023630 0x7f76a0025ac0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.165+0000 7f76c6be4700 1 --2- 172.21.15.16:0/2962663254 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f76c006df10 0x7f76c009d5b0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.166+0000 7f76c6be4700 1 -- 172.21.15.16:0/2962663254 shutdown_connections 2019-12-05T06:36:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.166+0000 7f76c6be4700 1 -- 172.21.15.16:0/2962663254 wait complete. 2019-12-05T06:36:53.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.166+0000 7f76c6be4700 1 -- 172.21.15.16:0/2962663254 >> 172.21.15.16:0/2962663254 conn(0x7f76c0098480 msgr2=0x7f76c0071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:53.174 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:36:53.175 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:36:53.175 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:36:53.182 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:36:53.182 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:53.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.478+0000 7f09bac08700 1 Processor -- start 2019-12-05T06:36:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.478+0000 7f09bac08700 1 -- start start 2019-12-05T06:36:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.480+0000 7f09bac08700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09b408fef0 0x7f09b408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.480+0000 7f09bac08700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b40958e0 con 0x7f09b408c6d0 2019-12-05T06:36:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.480+0000 7f09bac08700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b40629e0 con 0x7f09b408d2c0 2019-12-05T06:36:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.480+0000 7f09bac08700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b408df90 con 0x7f09b408fef0 2019-12-05T06:36:53.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.480+0000 7f09b99a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09b408fef0 0x7f09b408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.480+0000 7f09b91a5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f09b408c6d0 0x7f09b408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42336/0 (socket says 172.21.15.16:42336) 2019-12-05T06:36:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.480+0000 7f09b99a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09b408fef0 0x7f09b408f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57170/0 (socket says 172.21.15.16:57170) 2019-12-05T06:36:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.480+0000 7f09b91a5700 1 -- 172.21.15.16:0/3897323058 learned_addr learned my addr 172.21.15.16:0/3897323058 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.481+0000 7f09b3fff700 1 -- 172.21.15.16:0/3897323058 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2727207845 0 0) 0x7f09a8001080 con 0x7f09b408d2c0 2019-12-05T06:36:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.481+0000 7f09b99a6700 1 -- 172.21.15.16:0/3897323058 >> v1:172.21.15.76:6789/0 conn(0x7f09b408d2c0 legacy=0x7f09b408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.481+0000 7f09b99a6700 1 -- 172.21.15.16:0/3897323058 >> v1:172.21.15.16:6789/0 conn(0x7f09b408c6d0 legacy=0x7f09b408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:53.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.481+0000 7f09b99a6700 1 -- 172.21.15.16:0/3897323058 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b408e620 con 0x7f09b408fef0 2019-12-05T06:36:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.481+0000 7f09b99a6700 1 --2- 172.21.15.16:0/3897323058 >> v2:172.21.15.16:3300/0 conn(0x7f09b408fef0 0x7f09b408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f09a400b2d0 tx=0x7f09a400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.481+0000 7f09b99a6700 1 --2- 172.21.15.16:0/3897323058 >> v2:172.21.15.16:3300/0 conn(0x7f09b408fef0 0x7f09b408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f09a400b2d0 tx=0x7f09a400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.481+0000 7f09b99a6700 1 --2- 172.21.15.16:0/3897323058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b408fef0 0x7f09b408f720 secure :-1 s=READY pgs=1064 cs=0 l=1 rx=0x7f09a400b2d0 tx=0x7f09a400bae0).ready entity=mon.0 client_cookie=38a0719ff88675a2 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.481+0000 7f09b99a6700 1 --2- 172.21.15.16:0/3897323058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b408fef0 0x7f09b408f720 secure :-1 s=THROTTLE_DONE pgs=1064 cs=0 l=1 rx=0x7f09a400b2d0 tx=0x7f09a400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.481+0000 7f09b99a6700 1 --2- 172.21.15.16:0/3897323058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b408fef0 0x7f09b408f720 secure :-1 s=THROTTLE_DONE pgs=1064 cs=0 l=1 rx=0x7f09a400b2d0 tx=0x7f09a400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.481+0000 7f09b3fff700 1 -- 172.21.15.16:0/3897323058 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a4009f20 con 0x7f09b408fef0 2019-12-05T06:36:53.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.482+0000 7f09b99a6700 1 --2- 172.21.15.16:0/3897323058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b408fef0 0x7f09b408f720 secure :-1 s=THROTTLE_DONE pgs=1064 cs=0 l=1 rx=0x7f09a400b2d0 tx=0x7f09a400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.482+0000 7f09b3fff700 1 -- 172.21.15.16:0/3897323058 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a4000df0 con 0x7f09b408fef0 2019-12-05T06:36:53.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.482+0000 7f09b3fff700 1 -- 172.21.15.16:0/3897323058 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a400a500 con 0x7f09b408fef0 2019-12-05T06:36:53.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.482+0000 7f09bac08700 1 -- 172.21.15.16:0/3897323058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b408fef0 msgr2=0x7f09b408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:53.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.482+0000 7f09bac08700 1 --2- 172.21.15.16:0/3897323058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b408fef0 0x7f09b408f720 secure :-1 s=READY pgs=1064 cs=0 l=1 rx=0x7f09a400b2d0 tx=0x7f09a400bae0).stop 2019-12-05T06:36:53.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.484+0000 7f09bac08700 1 -- 172.21.15.16:0/3897323058 shutdown_connections 2019-12-05T06:36:53.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.484+0000 7f09bac08700 1 --2- 172.21.15.16:0/3897323058 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b408fef0 0x7f09b408f720 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.484+0000 7f09bac08700 1 -- 172.21.15.16:0/3897323058 shutdown_connections 2019-12-05T06:36:53.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.484+0000 7f09bac08700 1 -- 172.21.15.16:0/3897323058 wait complete. 2019-12-05T06:36:53.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.484+0000 7f09bac08700 1 -- 172.21.15.16:0/3897323058 >> 172.21.15.16:0/3897323058 conn(0x7f09b4061940 msgr2=0x7f09b4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:53.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.489+0000 7f09bac08700 1 Processor -- start 2019-12-05T06:36:53.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.489+0000 7f09bac08700 1 -- start start 2019-12-05T06:36:53.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.491+0000 7f09bac08700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f09b4096e90 0x7f09b409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:53.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.491+0000 7f09bac08700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f09b4084f30 0x7f09b409b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:53.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.491+0000 7f09bac08700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09b4097280 0x7f09b4097700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:53.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.491+0000 7f09bac08700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b4099a20 con 0x7f09b4097280 2019-12-05T06:36:53.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.491+0000 7f09bac08700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b4099cf0 con 0x7f09b4084f30 2019-12-05T06:36:53.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.491+0000 7f09bac08700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b409aae0 con 0x7f09b4096e90 2019-12-05T06:36:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.491+0000 7f09b99a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09b4097280 0x7f09b4097700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.491+0000 7f09b99a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f09b4097280 0x7f09b4097700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57176/0 (socket says 172.21.15.16:57176) 2019-12-05T06:36:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.491+0000 7f09b99a6700 1 -- 172.21.15.16:0/425556404 learned_addr learned my addr 172.21.15.16:0/425556404 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.492+0000 7f09b91a5700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f09b4096e90 0x7f09b409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.492+0000 7f09b89a4700 1 --2- 172.21.15.16:0/425556404 >> v2:172.21.15.76:3300/0 conn(0x7f09b4084f30 0x7f09b409b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:53.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.492+0000 7f09b99a6700 1 -- 172.21.15.16:0/425556404 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f09b4096e90 msgr2=0x7f09b409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:53.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.492+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f09b4096e90 0x7f09b409b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.492+0000 7f09b99a6700 1 -- 172.21.15.16:0/425556404 >> v2:172.21.15.76:3300/0 conn(0x7f09b4084f30 msgr2=0x7f09b409b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:53.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.492+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> v2:172.21.15.76:3300/0 conn(0x7f09b4084f30 0x7f09b409b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.492+0000 7f09b99a6700 1 -- 172.21.15.16:0/425556404 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b4084b90 con 0x7f09b4097280 2019-12-05T06:36:53.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.492+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> v2:172.21.15.16:3300/0 conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.492+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> v2:172.21.15.16:3300/0 conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.492+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=READY pgs=1065 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:53.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.493+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.493+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.493+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.493+0000 7f09b1ffb700 1 -- 172.21.15.16:0/425556404 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a400af30 con 0x7f09b4097280 2019-12-05T06:36:53.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.493+0000 7f09b1ffb700 1 -- 172.21.15.16:0/425556404 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a4009d00 con 0x7f09b4097280 2019-12-05T06:36:53.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.493+0000 7f09bac08700 1 -- 172.21.15.16:0/425556404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b4084b90 con 0x7f09b4097280 2019-12-05T06:36:53.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.493+0000 7f09b1ffb700 1 -- 172.21.15.16:0/425556404 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a400a3d0 con 0x7f09b4097280 2019-12-05T06:36:53.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.493+0000 7f09bac08700 1 -- 172.21.15.16:0/425556404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b4084b90 con 0x7f09b4097280 2019-12-05T06:36:53.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.494+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.495+0000 7f09b1ffb700 1 -- 172.21.15.16:0/425556404 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f09a4009900 con 0x7f09b4097280 2019-12-05T06:36:53.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.495+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.495+0000 7f09b1ffb700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0994023630 0x7f0994025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.495+0000 7f09b1ffb700 1 -- 172.21.15.16:0/425556404 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f09a4045f90 con 0x7f09b4097280 2019-12-05T06:36:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.495+0000 7f09b91a5700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0994023630 0x7f0994025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:53.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.496+0000 7f09b91a5700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0994023630 0x7f0994025ac0 crc :-1 s=READY pgs=456 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:53.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.509+0000 7f09bac08700 1 -- 172.21.15.16:0/425556404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b40958e0 con 0x7f09b4097280 2019-12-05T06:36:53.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.515+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.516+0000 7f09b1ffb700 1 -- 172.21.15.16:0/425556404 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f09a400af30 con 0x7f09b4097280 2019-12-05T06:36:53.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.744+0000 7f09bac08700 1 -- 172.21.15.16:0/425556404 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09b40958e0 con 0x7f09b4097280 2019-12-05T06:36:53.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.744+0000 7f09b99a6700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:53.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.744+0000 7f09b1ffb700 1 -- 172.21.15.16:0/425556404 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f09a4009900 con 0x7f09b4097280 2019-12-05T06:36:53.744 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 132 2019-12-05T06:36:53.744 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:53.744 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:53.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.747+0000 7f09bac08700 1 -- 172.21.15.16:0/425556404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0994023630 msgr2=0x7f0994025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:53.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.747+0000 7f09bac08700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0994023630 0x7f0994025ac0 crc :-1 s=READY pgs=456 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.747+0000 7f09bac08700 1 -- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 msgr2=0x7f09b4097700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:53.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.747+0000 7f09bac08700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 0x7f09b4097700 secure :-1 s=READY pgs=1065 cs=0 l=1 rx=0x7f09a4010780 tx=0x7f09a4010630).stop 2019-12-05T06:36:53.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.747+0000 7f09bac08700 1 -- 172.21.15.16:0/425556404 shutdown_connections 2019-12-05T06:36:53.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.747+0000 7f09bac08700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0994023630 0x7f0994025ac0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.747+0000 7f09bac08700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f09b4097280 0x7f09b4097700 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.747+0000 7f09bac08700 1 --2- 172.21.15.16:0/425556404 >> v2:172.21.15.76:3300/0 conn(0x7f09b4084f30 0x7f09b409b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.747+0000 7f09bac08700 1 --2- 172.21.15.16:0/425556404 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f09b4096e90 0x7f09b409b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:53.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.748+0000 7f09bac08700 1 -- 172.21.15.16:0/425556404 shutdown_connections 2019-12-05T06:36:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.748+0000 7f09bac08700 1 -- 172.21.15.16:0/425556404 wait complete. 2019-12-05T06:36:53.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:53.748+0000 7f09bac08700 1 -- 172.21.15.16:0/425556404 >> 172.21.15.16:0/425556404 conn(0x7f09b40714f0 msgr2=0x7f09b409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:53.762 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:36:55.767 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:56.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.054+0000 7fbe4ae36700 1 Processor -- start 2019-12-05T06:36:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.055+0000 7fbe4ae36700 1 -- start start 2019-12-05T06:36:56.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.056+0000 7fbe4ae36700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe4408c6d0 0x7fbe4408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.056+0000 7fbe4ae36700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbe4408fef0 0x7fbe4408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.056+0000 7fbe4ae36700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe440958e0 con 0x7fbe4408ef20 2019-12-05T06:36:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.057+0000 7fbe4ae36700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe4408df90 con 0x7fbe4408c6d0 2019-12-05T06:36:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.057+0000 7fbe4ae36700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbe440a3f70 con 0x7fbe4408fef0 2019-12-05T06:36:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.057+0000 7fbe493d3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe4408c6d0 0x7fbe4408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.057+0000 7fbe48bd2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbe4408fef0 0x7fbe4408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:56.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.057+0000 7fbe493d3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe4408c6d0 0x7fbe4408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57184/0 (socket says 172.21.15.16:57184) 2019-12-05T06:36:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.057+0000 7fbe48bd2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbe4408fef0 0x7fbe4408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:32944/0 (socket says 172.21.15.16:32944) 2019-12-05T06:36:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.057+0000 7fbe49bd4700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbe4408ef20 0x7fbe4408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:35426/0 (socket says 172.21.15.16:35426) 2019-12-05T06:36:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.057+0000 7fbe493d3700 1 -- 172.21.15.16:0/1817177481 learned_addr learned my addr 172.21.15.16:0/1817177481 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.058+0000 7fbe493d3700 1 -- 172.21.15.16:0/1817177481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbe4408fef0 msgr2=0x7fbe4408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.058+0000 7fbe43fff700 1 -- 172.21.15.16:0/1817177481 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1957453844 0 0) 0x7fbe340009b0 con 0x7fbe4408ef20 2019-12-05T06:36:56.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.058+0000 7fbe493d3700 1 --2- 172.21.15.16:0/1817177481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbe4408fef0 0x7fbe4408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.058+0000 7fbe493d3700 1 -- 172.21.15.16:0/1817177481 >> v1:172.21.15.76:6789/0 conn(0x7fbe4408ef20 legacy=0x7fbe4408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.058+0000 7fbe493d3700 1 -- 172.21.15.16:0/1817177481 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4408d540 con 0x7fbe4408c6d0 2019-12-05T06:36:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.058+0000 7fbe48bd2700 1 --2- 172.21.15.16:0/1817177481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbe4408fef0 0x7fbe4408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:36:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.058+0000 7fbe493d3700 1 --2- 172.21.15.16:0/1817177481 >> v2:172.21.15.16:3300/0 conn(0x7fbe4408c6d0 0x7fbe4408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbe3c00fac0 tx=0x7fbe3c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.058+0000 7fbe493d3700 1 --2- 172.21.15.16:0/1817177481 >> v2:172.21.15.16:3300/0 conn(0x7fbe4408c6d0 0x7fbe4408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbe3c00fac0 tx=0x7fbe3c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.058+0000 7fbe493d3700 1 --2- 172.21.15.16:0/1817177481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4408c6d0 0x7fbe4408cac0 secure :-1 s=READY pgs=1066 cs=0 l=1 rx=0x7fbe3c00fac0 tx=0x7fbe3c00fe60).ready entity=mon.0 client_cookie=59fee981f52e687d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.058+0000 7fbe493d3700 1 --2- 172.21.15.16:0/1817177481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4408c6d0 0x7fbe4408cac0 secure :-1 s=THROTTLE_DONE pgs=1066 cs=0 l=1 rx=0x7fbe3c00fac0 tx=0x7fbe3c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.059+0000 7fbe43fff700 1 -- 172.21.15.16:0/1817177481 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe3c000ce0 con 0x7fbe4408c6d0 2019-12-05T06:36:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.059+0000 7fbe493d3700 1 --2- 172.21.15.16:0/1817177481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4408c6d0 0x7fbe4408cac0 secure :-1 s=THROTTLE_DONE pgs=1066 cs=0 l=1 rx=0x7fbe3c00fac0 tx=0x7fbe3c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.059+0000 7fbe493d3700 1 --2- 172.21.15.16:0/1817177481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4408c6d0 0x7fbe4408cac0 secure :-1 s=THROTTLE_DONE pgs=1066 cs=0 l=1 rx=0x7fbe3c00fac0 tx=0x7fbe3c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.059+0000 7fbe43fff700 1 -- 172.21.15.16:0/1817177481 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3c001010 con 0x7fbe4408c6d0 2019-12-05T06:36:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.059+0000 7fbe43fff700 1 -- 172.21.15.16:0/1817177481 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe3c001720 con 0x7fbe4408c6d0 2019-12-05T06:36:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.059+0000 7fbe4ae36700 1 -- 172.21.15.16:0/1817177481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4408c6d0 msgr2=0x7fbe4408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.059+0000 7fbe4ae36700 1 --2- 172.21.15.16:0/1817177481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4408c6d0 0x7fbe4408cac0 secure :-1 s=READY pgs=1066 cs=0 l=1 rx=0x7fbe3c00fac0 tx=0x7fbe3c00fe60).stop 2019-12-05T06:36:56.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.061+0000 7fbe4ae36700 1 -- 172.21.15.16:0/1817177481 shutdown_connections 2019-12-05T06:36:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.061+0000 7fbe4ae36700 1 --2- 172.21.15.16:0/1817177481 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbe4408fef0 0x7fbe4408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.061+0000 7fbe4ae36700 1 --2- 172.21.15.16:0/1817177481 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4408c6d0 0x7fbe4408cac0 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.061+0000 7fbe4ae36700 1 -- 172.21.15.16:0/1817177481 shutdown_connections 2019-12-05T06:36:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.061+0000 7fbe4ae36700 1 -- 172.21.15.16:0/1817177481 wait complete. 2019-12-05T06:36:56.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.061+0000 7fbe4ae36700 1 -- 172.21.15.16:0/1817177481 >> 172.21.15.16:0/1817177481 conn(0x7fbe44061940 msgr2=0x7fbe44096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.066+0000 7fbe4ae36700 1 Processor -- start 2019-12-05T06:36:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.066+0000 7fbe4ae36700 1 -- start start 2019-12-05T06:36:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.068+0000 7fbe4ae36700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe4407bc30 0x7fbe44099ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:56.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.068+0000 7fbe4ae36700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbe4409aea0 0x7fbe4409a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.068+0000 7fbe4ae36700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4407da20 con 0x7fbe4409a950 2019-12-05T06:36:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.068+0000 7fbe4ae36700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe44098f40 con 0x7fbe4407bc30 2019-12-05T06:36:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.068+0000 7fbe4ae36700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbe44098510 con 0x7fbe4409aea0 2019-12-05T06:36:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.069+0000 7fbe493d3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe4407bc30 0x7fbe44099ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.069+0000 7fbe493d3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbe4407bc30 0x7fbe44099ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57190/0 (socket says 172.21.15.16:57190) 2019-12-05T06:36:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.069+0000 7fbe493d3700 1 -- 172.21.15.16:0/940789742 learned_addr learned my addr 172.21.15.16:0/940789742 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:36:56.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.069+0000 7fbe48bd2700 1 --2- 172.21.15.16:0/940789742 >> v2:172.21.15.76:3300/0 conn(0x7fbe4409aea0 0x7fbe4409a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.069+0000 7fbe493d3700 1 -- 172.21.15.16:0/940789742 >> v2:172.21.15.76:3300/0 conn(0x7fbe4409aea0 msgr2=0x7fbe4409a410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.069+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> v2:172.21.15.76:3300/0 conn(0x7fbe4409aea0 0x7fbe4409a410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.069+0000 7fbe493d3700 1 -- 172.21.15.16:0/940789742 >> v1:172.21.15.16:6789/0 conn(0x7fbe4409a950 legacy=0x7fbe4407c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.069+0000 7fbe493d3700 1 -- 172.21.15.16:0/940789742 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe440995d0 con 0x7fbe4407bc30 2019-12-05T06:36:56.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.069+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> v2:172.21.15.16:3300/0 conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.070+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> v2:172.21.15.16:3300/0 conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.070+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=READY pgs=1068 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.070+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=THROTTLE_DONE pgs=1068 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.070+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=THROTTLE_DONE pgs=1068 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.070+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=THROTTLE_DONE pgs=1068 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.070+0000 7fbe41ffb700 1 -- 172.21.15.16:0/940789742 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe3c0058e0 con 0x7fbe4407bc30 2019-12-05T06:36:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.070+0000 7fbe41ffb700 1 -- 172.21.15.16:0/940789742 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe3c005500 con 0x7fbe4407bc30 2019-12-05T06:36:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.070+0000 7fbe41ffb700 1 -- 172.21.15.16:0/940789742 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe3c005ec0 con 0x7fbe4407bc30 2019-12-05T06:36:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.070+0000 7fbe4ae36700 1 -- 172.21.15.16:0/940789742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe440995d0 con 0x7fbe4407bc30 2019-12-05T06:36:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.070+0000 7fbe4ae36700 1 -- 172.21.15.16:0/940789742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe440995d0 con 0x7fbe4407bc30 2019-12-05T06:36:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.071+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=THROTTLE_DONE pgs=1068 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.072+0000 7fbe41ffb700 1 -- 172.21.15.16:0/940789742 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbe3c0056f0 con 0x7fbe4407bc30 2019-12-05T06:36:56.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.072+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=THROTTLE_DONE pgs=1068 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.072+0000 7fbe41ffb700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe24023630 0x7fbe24025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:56.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.072+0000 7fbe41ffb700 1 -- 172.21.15.16:0/940789742 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fbe3c048390 con 0x7fbe4407bc30 2019-12-05T06:36:56.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.073+0000 7fbe48bd2700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe24023630 0x7fbe24025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:56.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.073+0000 7fbe48bd2700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe24023630 0x7fbe24025ac0 crc :-1 s=READY pgs=457 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:56.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.085+0000 7fbe4ae36700 1 -- 172.21.15.16:0/940789742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe440995d0 con 0x7fbe4407bc30 2019-12-05T06:36:56.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.093+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=THROTTLE_DONE pgs=1068 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.094+0000 7fbe41ffb700 1 -- 172.21.15.16:0/940789742 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbe3c007620 con 0x7fbe4407bc30 2019-12-05T06:36:56.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.327+0000 7fbe4ae36700 1 -- 172.21.15.16:0/940789742 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbe440995d0 con 0x7fbe4407bc30 2019-12-05T06:36:56.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.328+0000 7fbe493d3700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=THROTTLE_DONE pgs=1068 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:56.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.328+0000 7fbe41ffb700 1 -- 172.21.15.16:0/940789742 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fbe3c0056c0 con 0x7fbe4407bc30 2019-12-05T06:36:56.329 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:56.329 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:56.330 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 132 2019-12-05T06:36:56.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.332+0000 7fbe4ae36700 1 -- 172.21.15.16:0/940789742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe24023630 msgr2=0x7fbe24025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.332+0000 7fbe4ae36700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe24023630 0x7fbe24025ac0 crc :-1 s=READY pgs=457 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.332+0000 7fbe4ae36700 1 -- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 msgr2=0x7fbe44099ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:56.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.332+0000 7fbe4ae36700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 0x7fbe44099ed0 secure :-1 s=READY pgs=1068 cs=0 l=1 rx=0x7fbe3c010100 tx=0x7fbe3c010670).stop 2019-12-05T06:36:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.333+0000 7fbe4ae36700 1 -- 172.21.15.16:0/940789742 shutdown_connections 2019-12-05T06:36:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.333+0000 7fbe4ae36700 1 --2- 172.21.15.16:0/940789742 >> v2:172.21.15.76:3300/0 conn(0x7fbe4409aea0 0x7fbe4409a410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.333+0000 7fbe4ae36700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe24023630 0x7fbe24025ac0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.333+0000 7fbe4ae36700 1 --2- 172.21.15.16:0/940789742 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbe4407bc30 0x7fbe44099ed0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.333+0000 7fbe4ae36700 1 -- 172.21.15.16:0/940789742 shutdown_connections 2019-12-05T06:36:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.333+0000 7fbe4ae36700 1 -- 172.21.15.16:0/940789742 wait complete. 2019-12-05T06:36:56.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:56.333+0000 7fbe4ae36700 1 -- 172.21.15.16:0/940789742 >> 172.21.15.16:0/940789742 conn(0x7fbe44082220 msgr2=0x7fbe4406d380 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:56.346 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:36:58.348 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:36:58.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.640+0000 7f2d30c5a700 1 Processor -- start 2019-12-05T06:36:58.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.640+0000 7f2d30c5a700 1 -- start start 2019-12-05T06:36:58.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.642+0000 7f2d30c5a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c08c6d0 0x7f2d2c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.642+0000 7f2d30c5a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d2c08f720 0x7f2d2c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:36:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.642+0000 7f2d30c5a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c0958e0 con 0x7f2d2c08fef0 2019-12-05T06:36:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.642+0000 7f2d30c5a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c0a3f70 con 0x7f2d2c08f720 2019-12-05T06:36:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.642+0000 7f2d30c5a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c08d830 con 0x7f2d2c08c6d0 2019-12-05T06:36:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.643+0000 7f2d2b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d2c08f720 0x7f2d2c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:58.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.643+0000 7f2d2a59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2d2c08fef0 0x7f2d2c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42378/0 (socket says 172.21.15.16:42378) 2019-12-05T06:36:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.643+0000 7f2d2a59c700 1 -- 172.21.15.16:0/2232237920 learned_addr learned my addr 172.21.15.16:0/2232237920 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.643+0000 7f2d2ad9d700 1 --2- 172.21.15.16:0/2232237920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c08c6d0 0x7f2d2c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.643+0000 7f2d2b59e700 1 -- 172.21.15.16:0/2232237920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c08c6d0 msgr2=0x7f2d2c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:36:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d2b59e700 1 --2- 172.21.15.16:0/2232237920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c08c6d0 0x7f2d2c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d2b59e700 1 -- 172.21.15.16:0/2232237920 >> v1:172.21.15.16:6789/0 conn(0x7f2d2c08fef0 legacy=0x7f2d2c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d2b59e700 1 -- 172.21.15.16:0/2232237920 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c08fbc0 con 0x7f2d2c08f720 2019-12-05T06:36:58.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d29d9b700 1 -- 172.21.15.16:0/2232237920 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2420674254 0 0) 0x7f2d14001150 con 0x7f2d2c08fef0 2019-12-05T06:36:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d2ad9d700 1 --2- 172.21.15.16:0/2232237920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c08c6d0 0x7f2d2c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:36:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d2b59e700 1 --2- 172.21.15.16:0/2232237920 >> v2:172.21.15.16:3300/0 conn(0x7f2d2c08f720 0x7f2d2c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2d1c00af50 tx=0x7f2d1c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d2b59e700 1 --2- 172.21.15.16:0/2232237920 >> v2:172.21.15.16:3300/0 conn(0x7f2d2c08f720 0x7f2d2c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2d1c00af50 tx=0x7f2d1c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d2b59e700 1 --2- 172.21.15.16:0/2232237920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c08f720 0x7f2d2c08d000 secure :-1 s=READY pgs=1072 cs=0 l=1 rx=0x7f2d1c00af50 tx=0x7f2d1c00b760).ready entity=mon.0 client_cookie=36d0fe8ecaad7ddd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d2b59e700 1 --2- 172.21.15.16:0/2232237920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c08f720 0x7f2d2c08d000 secure :-1 s=THROTTLE_DONE pgs=1072 cs=0 l=1 rx=0x7f2d1c00af50 tx=0x7f2d1c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d29d9b700 1 -- 172.21.15.16:0/2232237920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d1c009ba0 con 0x7f2d2c08f720 2019-12-05T06:36:58.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d2b59e700 1 --2- 172.21.15.16:0/2232237920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c08f720 0x7f2d2c08d000 secure :-1 s=THROTTLE_DONE pgs=1072 cs=0 l=1 rx=0x7f2d1c00af50 tx=0x7f2d1c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d2b59e700 1 --2- 172.21.15.16:0/2232237920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c08f720 0x7f2d2c08d000 secure :-1 s=THROTTLE_DONE pgs=1072 cs=0 l=1 rx=0x7f2d1c00af50 tx=0x7f2d1c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.644+0000 7f2d29d9b700 1 -- 172.21.15.16:0/2232237920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1c0097c0 con 0x7f2d2c08f720 2019-12-05T06:36:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.645+0000 7f2d29d9b700 1 -- 172.21.15.16:0/2232237920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d1c000bc0 con 0x7f2d2c08f720 2019-12-05T06:36:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.645+0000 7f2d30c5a700 1 -- 172.21.15.16:0/2232237920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c08f720 msgr2=0x7f2d2c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.645+0000 7f2d30c5a700 1 --2- 172.21.15.16:0/2232237920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c08f720 0x7f2d2c08d000 secure :-1 s=READY pgs=1072 cs=0 l=1 rx=0x7f2d1c00af50 tx=0x7f2d1c00b760).stop 2019-12-05T06:36:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.647+0000 7f2d30c5a700 1 -- 172.21.15.16:0/2232237920 shutdown_connections 2019-12-05T06:36:58.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.647+0000 7f2d30c5a700 1 --2- 172.21.15.16:0/2232237920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c08f720 0x7f2d2c08d000 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.647+0000 7f2d30c5a700 1 --2- 172.21.15.16:0/2232237920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c08c6d0 0x7f2d2c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:36:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.647+0000 7f2d30c5a700 1 -- 172.21.15.16:0/2232237920 shutdown_connections 2019-12-05T06:36:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.647+0000 7f2d30c5a700 1 -- 172.21.15.16:0/2232237920 wait complete. 2019-12-05T06:36:58.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.647+0000 7f2d30c5a700 1 -- 172.21.15.16:0/2232237920 >> 172.21.15.16:0/2232237920 conn(0x7f2d2c061940 msgr2=0x7f2d2c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.653+0000 7f2d30c5a700 1 Processor -- start 2019-12-05T06:36:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.653+0000 7f2d30c5a700 1 -- start start 2019-12-05T06:36:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.654+0000 7f2d30c5a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d2c0992d0 0x7f2d2c098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.654+0000 7f2d30c5a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c156cb0 0x7f2d2c1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:58.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.654+0000 7f2d30c5a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d2c089fb0 con 0x7f2d2c06df10 2019-12-05T06:36:58.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.654+0000 7f2d30c5a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d2c097370 con 0x7f2d2c0992d0 2019-12-05T06:36:58.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.654+0000 7f2d30c5a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d2c08ad00 con 0x7f2d2c156cb0 2019-12-05T06:36:58.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.654+0000 7f2d2ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2d2c06df10 0x7f2d2c098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42380/0 (socket says 172.21.15.16:42380) 2019-12-05T06:36:58.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.654+0000 7f2d2a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2d2c0992d0 0x7f2d2c098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:58.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.654+0000 7f2d2ad9d700 1 -- 172.21.15.16:0/909439944 learned_addr learned my addr 172.21.15.16:0/909439944 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:36:58.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.655+0000 7f2d2b59e700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c156cb0 0x7f2d2c1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:58.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.655+0000 7f2d2a59c700 1 -- 172.21.15.16:0/909439944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c156cb0 msgr2=0x7f2d2c1590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.655+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c156cb0 0x7f2d2c1590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:58.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.655+0000 7f2d2a59c700 1 -- 172.21.15.16:0/909439944 >> v1:172.21.15.16:6789/0 conn(0x7f2d2c06df10 legacy=0x7f2d2c098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:58.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.655+0000 7f2d237fe700 1 -- 172.21.15.16:0/909439944 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107755780 0 0) 0x7f2d2400e110 con 0x7f2d2c06df10 2019-12-05T06:36:58.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.655+0000 7f2d2a59c700 1 -- 172.21.15.16:0/909439944 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d2c097a00 con 0x7f2d2c0992d0 2019-12-05T06:36:58.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.655+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> v2:172.21.15.16:3300/0 conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.656+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> v2:172.21.15.16:3300/0 conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.656+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=READY pgs=1074 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:58.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.656+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.656+0000 7f2d237fe700 1 -- 172.21.15.16:0/909439944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d14000ef0 con 0x7f2d2c0992d0 2019-12-05T06:36:58.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.656+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.656+0000 7f2d30c5a700 1 -- 172.21.15.16:0/909439944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d2c097a00 con 0x7f2d2c0992d0 2019-12-05T06:36:58.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.656+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.656+0000 7f2d237fe700 1 -- 172.21.15.16:0/909439944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d14001200 con 0x7f2d2c0992d0 2019-12-05T06:36:58.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.656+0000 7f2d237fe700 1 -- 172.21.15.16:0/909439944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d14004430 con 0x7f2d2c0992d0 2019-12-05T06:36:58.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.656+0000 7f2d30c5a700 1 -- 172.21.15.16:0/909439944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d2c097a00 con 0x7f2d2c0992d0 2019-12-05T06:36:58.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.657+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.658+0000 7f2d237fe700 1 -- 172.21.15.16:0/909439944 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2d14004240 con 0x7f2d2c0992d0 2019-12-05T06:36:58.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.658+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.658+0000 7f2d237fe700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2d10023630 0x7f2d10025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:36:58.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.659+0000 7f2d237fe700 1 -- 172.21.15.16:0/909439944 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f2d14045d40 con 0x7f2d2c0992d0 2019-12-05T06:36:58.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.659+0000 7f2d2ad9d700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2d10023630 0x7f2d10025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:36:58.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.659+0000 7f2d2ad9d700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2d10023630 0x7f2d10025ac0 crc :-1 s=READY pgs=458 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:36:58.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.672+0000 7f2d30c5a700 1 -- 172.21.15.16:0/909439944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d2c097a00 con 0x7f2d2c0992d0 2019-12-05T06:36:58.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.680+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.680+0000 7f2d237fe700 1 -- 172.21.15.16:0/909439944 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2d14000c90 con 0x7f2d2c0992d0 2019-12-05T06:36:58.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.939+0000 7f2d30c5a700 1 -- 172.21.15.16:0/909439944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d2c097a00 con 0x7f2d2c0992d0 2019-12-05T06:36:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.940+0000 7f2d2a59c700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=THROTTLE_DONE pgs=1074 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:36:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.940+0000 7f2d237fe700 1 -- 172.21.15.16:0/909439944 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f2d14004240 con 0x7f2d2c0992d0 2019-12-05T06:36:58.940 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 132 2019-12-05T06:36:58.940 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:36:58.941 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:36:58.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 -- 172.21.15.16:0/909439944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2d10023630 msgr2=0x7f2d10025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:58.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2d10023630 0x7f2d10025ac0 crc :-1 s=READY pgs=458 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:58.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 -- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 msgr2=0x7f2d2c098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:36:58.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 0x7f2d2c098b00 secure :-1 s=READY pgs=1074 cs=0 l=1 rx=0x7f2d14010370 tx=0x7f2d140101a0).stop 2019-12-05T06:36:58.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 -- 172.21.15.16:0/909439944 shutdown_connections 2019-12-05T06:36:58.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2d10023630 0x7f2d10025ac0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:58.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2d2c0992d0 0x7f2d2c098b00 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:58.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 --2- 172.21.15.16:0/909439944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2d2c156cb0 0x7f2d2c1590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:36:58.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 -- 172.21.15.16:0/909439944 shutdown_connections 2019-12-05T06:36:58.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 -- 172.21.15.16:0/909439944 wait complete. 2019-12-05T06:36:58.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:36:58.943+0000 7f2d30c5a700 1 -- 172.21.15.16:0/909439944 >> 172.21.15.16:0/909439944 conn(0x7f2d2c0714f0 msgr2=0x7f2d2c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:36:58.953 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:37:00.956 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.222+0000 7fbd02ca4700 1 Processor -- start 2019-12-05T06:37:01.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.223+0000 7fbd02ca4700 1 -- start start 2019-12-05T06:37:01.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.224+0000 7fbd02ca4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbcfc08fef0 0x7fbcfc08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:01.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.224+0000 7fbd02ca4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcfc0958e0 con 0x7fbcfc08d2c0 2019-12-05T06:37:01.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.224+0000 7fbd02ca4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcfc0629e0 con 0x7fbcfc08c6d0 2019-12-05T06:37:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.224+0000 7fbd02ca4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcfc08df90 con 0x7fbcfc08fef0 2019-12-05T06:37:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.225+0000 7fbd00a40700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbcfc08d2c0 0x7fbcfc08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42392/0 (socket says 172.21.15.16:42392) 2019-12-05T06:37:01.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.225+0000 7fbd00a40700 1 -- 172.21.15.16:0/2281515275 learned_addr learned my addr 172.21.15.16:0/2281515275 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:01.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.225+0000 7fbd01a42700 1 --2- 172.21.15.16:0/2281515275 >> v2:172.21.15.76:3300/0 conn(0x7fbcfc08fef0 0x7fbcfc08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:01.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.225+0000 7fbcfbfff700 1 -- 172.21.15.16:0/2281515275 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2301110633 0 0) 0x7fbcf00014d0 con 0x7fbcfc08d2c0 2019-12-05T06:37:01.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.225+0000 7fbcfbfff700 1 -- 172.21.15.16:0/2281515275 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce4002e80 con 0x7fbcfc08d2c0 2019-12-05T06:37:01.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.225+0000 7fbcfbfff700 1 -- 172.21.15.16:0/2281515275 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867160493 0 0) 0x7fbcec001080 con 0x7fbcfc08c6d0 2019-12-05T06:37:01.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.225+0000 7fbcfbfff700 1 -- 172.21.15.16:0/2281515275 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbce4004170 con 0x7fbcfc08c6d0 2019-12-05T06:37:01.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.226+0000 7fbd01a42700 1 -- 172.21.15.16:0/2281515275 >> v1:172.21.15.76:6789/0 conn(0x7fbcfc08c6d0 legacy=0x7fbcfc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:01.237 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.226+0000 7fbd01a42700 1 -- 172.21.15.16:0/2281515275 >> v1:172.21.15.16:6789/0 conn(0x7fbcfc08d2c0 legacy=0x7fbcfc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:01.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.226+0000 7fbcfbfff700 1 -- 172.21.15.16:0/2281515275 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2850675637 0 0) 0x7fbcf00014b0 con 0x7fbcfc08d2c0 2019-12-05T06:37:01.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.226+0000 7fbd01a42700 1 -- 172.21.15.16:0/2281515275 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc08e620 con 0x7fbcfc08fef0 2019-12-05T06:37:01.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.226+0000 7fbd01a42700 1 --2- 172.21.15.16:0/2281515275 >> v2:172.21.15.76:3300/0 conn(0x7fbcfc08fef0 0x7fbcfc08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbcf400af80 tx=0x7fbcf400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.226+0000 7fbd01a42700 1 --2- 172.21.15.16:0/2281515275 >> v2:172.21.15.76:3300/0 conn(0x7fbcfc08fef0 0x7fbcfc08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbcf400af80 tx=0x7fbcf400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.226+0000 7fbd01a42700 1 --2- 172.21.15.16:0/2281515275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbcfc08fef0 0x7fbcfc08f720 secure :-1 s=READY pgs=746 cs=0 l=1 rx=0x7fbcf400af80 tx=0x7fbcf400b790).ready entity=mon.1 client_cookie=336afc4c9c08cba7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:01.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.227+0000 7fbd01a42700 1 --2- 172.21.15.16:0/2281515275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbcfc08fef0 0x7fbcfc08f720 secure :-1 s=THROTTLE_DONE pgs=746 cs=0 l=1 rx=0x7fbcf400af80 tx=0x7fbcf400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.227+0000 7fbd01a42700 1 --2- 172.21.15.16:0/2281515275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbcfc08fef0 0x7fbcfc08f720 secure :-1 s=THROTTLE_DONE pgs=746 cs=0 l=1 rx=0x7fbcf400af80 tx=0x7fbcf400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.227+0000 7fbcfbfff700 1 -- 172.21.15.16:0/2281515275 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbcf4007240 con 0x7fbcfc08fef0 2019-12-05T06:37:01.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.227+0000 7fbd01a42700 1 --2- 172.21.15.16:0/2281515275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbcfc08fef0 0x7fbcfc08f720 secure :-1 s=THROTTLE_DONE pgs=746 cs=0 l=1 rx=0x7fbcf400af80 tx=0x7fbcf400b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.227+0000 7fbcfbfff700 1 -- 172.21.15.16:0/2281515275 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf40099e0 con 0x7fbcfc08fef0 2019-12-05T06:37:01.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.227+0000 7fbcfbfff700 1 -- 172.21.15.16:0/2281515275 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbcf400a070 con 0x7fbcfc08fef0 2019-12-05T06:37:01.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.227+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2281515275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbcfc08fef0 msgr2=0x7fbcfc08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:01.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.227+0000 7fbd02ca4700 1 --2- 172.21.15.16:0/2281515275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbcfc08fef0 0x7fbcfc08f720 secure :-1 s=READY pgs=746 cs=0 l=1 rx=0x7fbcf400af80 tx=0x7fbcf400b790).stop 2019-12-05T06:37:01.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.229+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2281515275 shutdown_connections 2019-12-05T06:37:01.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.229+0000 7fbd02ca4700 1 --2- 172.21.15.16:0/2281515275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbcfc08fef0 0x7fbcfc08f720 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:01.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.229+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2281515275 shutdown_connections 2019-12-05T06:37:01.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.229+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2281515275 wait complete. 2019-12-05T06:37:01.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.229+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2281515275 >> 172.21.15.16:0/2281515275 conn(0x7fbcfc061940 msgr2=0x7fbcfc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.234+0000 7fbd02ca4700 1 Processor -- start 2019-12-05T06:37:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.234+0000 7fbd02ca4700 1 -- start start 2019-12-05T06:37:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.236+0000 7fbd02ca4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbcfc12c740 0x7fbcfc12bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:01.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.236+0000 7fbd02ca4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcfc128b20 con 0x7fbcfc1325e0 2019-12-05T06:37:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.236+0000 7fbd02ca4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcfc128660 con 0x7fbcfc154cb0 2019-12-05T06:37:01.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.236+0000 7fbd02ca4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcfc12a7e0 con 0x7fbcfc12c740 2019-12-05T06:37:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.236+0000 7fbd01241700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbcfc1325e0 0x7fbcfc12b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42396/0 (socket says 172.21.15.16:42396) 2019-12-05T06:37:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.236+0000 7fbd01241700 1 -- 172.21.15.16:0/2464465757 learned_addr learned my addr 172.21.15.16:0/2464465757 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:01.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.236+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> v2:172.21.15.16:3300/0 conn(0x7fbcfc12c740 0x7fbcfc12bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:01.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.237+0000 7fbcf9ffb700 1 -- 172.21.15.16:0/2464465757 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090897442 0 0) 0x7fbcf4009990 con 0x7fbcfc154cb0 2019-12-05T06:37:01.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.237+0000 7fbd00a40700 1 -- 172.21.15.16:0/2464465757 >> v1:172.21.15.76:6789/0 conn(0x7fbcfc154cb0 legacy=0x7fbcfc1570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:01.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.237+0000 7fbd00a40700 1 -- 172.21.15.16:0/2464465757 >> v1:172.21.15.16:6789/0 conn(0x7fbcfc1325e0 legacy=0x7fbcfc12b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:01.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.237+0000 7fbd00a40700 1 -- 172.21.15.16:0/2464465757 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcfc14a820 con 0x7fbcfc12c740 2019-12-05T06:37:01.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.237+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> v2:172.21.15.16:3300/0 conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.237+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> v2:172.21.15.16:3300/0 conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.237+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=READY pgs=1077 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.237+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=THROTTLE_DONE pgs=1077 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.238+0000 7fbcf9ffb700 1 -- 172.21.15.16:0/2464465757 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbcf0001010 con 0x7fbcfc12c740 2019-12-05T06:37:01.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.238+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=THROTTLE_DONE pgs=1077 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.238+0000 7fbcf9ffb700 1 -- 172.21.15.16:0/2464465757 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcf0001760 con 0x7fbcfc12c740 2019-12-05T06:37:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.238+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=THROTTLE_DONE pgs=1077 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.238+0000 7fbcf9ffb700 1 -- 172.21.15.16:0/2464465757 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbcf0001950 con 0x7fbcfc12c740 2019-12-05T06:37:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.238+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2464465757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcfc12ae70 con 0x7fbcfc12c740 2019-12-05T06:37:01.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.238+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2464465757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcfc14a820 con 0x7fbcfc12c740 2019-12-05T06:37:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.239+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=THROTTLE_DONE pgs=1077 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.240+0000 7fbcf9ffb700 1 -- 172.21.15.16:0/2464465757 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbcf0001950 con 0x7fbcfc12c740 2019-12-05T06:37:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.240+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=THROTTLE_DONE pgs=1077 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.240+0000 7fbcf9ffb700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbcdc023630 0x7fbcdc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.240+0000 7fbcf9ffb700 1 -- 172.21.15.16:0/2464465757 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7fbcf0046110 con 0x7fbcfc12c740 2019-12-05T06:37:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.240+0000 7fbd01241700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbcdc023630 0x7fbcdc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:01.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.241+0000 7fbd01241700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbcdc023630 0x7fbcdc025ac0 crc :-1 s=READY pgs=459 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:01.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.254+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2464465757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcfc12ae70 con 0x7fbcfc12c740 2019-12-05T06:37:01.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.263+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=THROTTLE_DONE pgs=1077 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.263+0000 7fbcf9ffb700 1 -- 172.21.15.16:0/2464465757 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbcf0032f90 con 0x7fbcfc12c740 2019-12-05T06:37:01.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.498+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2464465757 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbcfc14a820 con 0x7fbcfc12c740 2019-12-05T06:37:01.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.499+0000 7fbd00a40700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=THROTTLE_DONE pgs=1077 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:01.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.499+0000 7fbcf9ffb700 1 -- 172.21.15.16:0/2464465757 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fbcf0032f90 con 0x7fbcfc12c740 2019-12-05T06:37:01.499 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 132 2019-12-05T06:37:01.500 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:37:01.500 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:01.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.501+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2464465757 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbcdc023630 msgr2=0x7fbcdc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:01.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.501+0000 7fbd02ca4700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbcdc023630 0x7fbcdc025ac0 crc :-1 s=READY pgs=459 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:01.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.501+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 msgr2=0x7fbcfc12bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:01.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.501+0000 7fbd02ca4700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 0x7fbcfc12bf70 secure :-1 s=READY pgs=1077 cs=0 l=1 rx=0x7fbcf0010700 tx=0x7fbcf0010aa0).stop 2019-12-05T06:37:01.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.502+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2464465757 shutdown_connections 2019-12-05T06:37:01.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.502+0000 7fbd02ca4700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbcdc023630 0x7fbcdc025ac0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:01.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.502+0000 7fbd02ca4700 1 --2- 172.21.15.16:0/2464465757 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbcfc12c740 0x7fbcfc12bf70 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:01.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.502+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2464465757 shutdown_connections 2019-12-05T06:37:01.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.502+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2464465757 wait complete. 2019-12-05T06:37:01.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:01.502+0000 7fbd02ca4700 1 -- 172.21.15.16:0/2464465757 >> 172.21.15.16:0/2464465757 conn(0x7fbcfc0a2850 msgr2=0x7fbcfc06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:01.514 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:37:03.515 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:03.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.767+0000 7f528fa90700 1 Processor -- start 2019-12-05T06:37:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.767+0000 7f528fa90700 1 -- start start 2019-12-05T06:37:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.769+0000 7f528fa90700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f528808d2c0 0x7f528808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.769+0000 7f528fa90700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52880958e0 con 0x7f528808c6d0 2019-12-05T06:37:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.769+0000 7f528fa90700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52880629e0 con 0x7f528808fef0 2019-12-05T06:37:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.769+0000 7f528fa90700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f528808df90 con 0x7f528808d2c0 2019-12-05T06:37:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.769+0000 7f528d82c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f528808d2c0 0x7f528808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:03.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.769+0000 7f528d82c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f528808d2c0 0x7f528808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57238/0 (socket says 172.21.15.16:57238) 2019-12-05T06:37:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.769+0000 7f528d82c700 1 -- 172.21.15.16:0/331645411 learned_addr learned my addr 172.21.15.16:0/331645411 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:37:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.769+0000 7f528e02d700 1 --1- 172.21.15.16:0/331645411 >> v1:172.21.15.16:6789/0 conn(0x7f528808c6d0 0x7f528808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42404/0 (socket says 172.21.15.16:42404) 2019-12-05T06:37:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.770+0000 7f528d02b700 1 -- 172.21.15.16:0/331645411 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3765394110 0 0) 0x7f5280000d30 con 0x7f528808fef0 2019-12-05T06:37:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.770+0000 7f528d82c700 1 -- 172.21.15.16:0/331645411 >> v1:172.21.15.76:6789/0 conn(0x7f528808fef0 legacy=0x7f528808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.770+0000 7f528d82c700 1 -- 172.21.15.16:0/331645411 >> v1:172.21.15.16:6789/0 conn(0x7f528808c6d0 legacy=0x7f528808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:03.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.770+0000 7f528d82c700 1 -- 172.21.15.16:0/331645411 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528808e620 con 0x7f528808d2c0 2019-12-05T06:37:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.770+0000 7f528d82c700 1 --2- 172.21.15.16:0/331645411 >> v2:172.21.15.16:3300/0 conn(0x7f528808d2c0 0x7f528808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f527800f740 tx=0x7f527800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.771+0000 7f528d82c700 1 --2- 172.21.15.16:0/331645411 >> v2:172.21.15.16:3300/0 conn(0x7f528808d2c0 0x7f528808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f527800f740 tx=0x7f527800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.771+0000 7f528d82c700 1 --2- 172.21.15.16:0/331645411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528808d2c0 0x7f528808ef20 secure :-1 s=READY pgs=1079 cs=0 l=1 rx=0x7f527800f740 tx=0x7f527800fae0).ready entity=mon.0 client_cookie=b27dd189fa9bfad server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.771+0000 7f528d82c700 1 --2- 172.21.15.16:0/331645411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528808d2c0 0x7f528808ef20 secure :-1 s=THROTTLE_DONE pgs=1079 cs=0 l=1 rx=0x7f527800f740 tx=0x7f527800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.771+0000 7f528d02b700 1 -- 172.21.15.16:0/331645411 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5278000960 con 0x7f528808d2c0 2019-12-05T06:37:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.771+0000 7f528d82c700 1 --2- 172.21.15.16:0/331645411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528808d2c0 0x7f528808ef20 secure :-1 s=THROTTLE_DONE pgs=1079 cs=0 l=1 rx=0x7f527800f740 tx=0x7f527800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.771+0000 7f528d82c700 1 --2- 172.21.15.16:0/331645411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528808d2c0 0x7f528808ef20 secure :-1 s=THROTTLE_DONE pgs=1079 cs=0 l=1 rx=0x7f527800f740 tx=0x7f527800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.771+0000 7f528d02b700 1 -- 172.21.15.16:0/331645411 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5278000c90 con 0x7f528808d2c0 2019-12-05T06:37:03.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.771+0000 7f528d02b700 1 -- 172.21.15.16:0/331645411 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5278001360 con 0x7f528808d2c0 2019-12-05T06:37:03.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.771+0000 7f528fa90700 1 -- 172.21.15.16:0/331645411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528808d2c0 msgr2=0x7f528808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:03.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.771+0000 7f528fa90700 1 --2- 172.21.15.16:0/331645411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528808d2c0 0x7f528808ef20 secure :-1 s=READY pgs=1079 cs=0 l=1 rx=0x7f527800f740 tx=0x7f527800fae0).stop 2019-12-05T06:37:03.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.773+0000 7f528fa90700 1 -- 172.21.15.16:0/331645411 shutdown_connections 2019-12-05T06:37:03.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.773+0000 7f528fa90700 1 --2- 172.21.15.16:0/331645411 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528808d2c0 0x7f528808ef20 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:03.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.773+0000 7f528fa90700 1 -- 172.21.15.16:0/331645411 shutdown_connections 2019-12-05T06:37:03.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.773+0000 7f528fa90700 1 -- 172.21.15.16:0/331645411 wait complete. 2019-12-05T06:37:03.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.773+0000 7f528fa90700 1 -- 172.21.15.16:0/331645411 >> 172.21.15.16:0/331645411 conn(0x7f5288061940 msgr2=0x7f5288096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.777+0000 7f528fa90700 1 Processor -- start 2019-12-05T06:37:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.777+0000 7f528fa90700 1 -- start start 2019-12-05T06:37:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.778+0000 7f528fa90700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f528814d4a0 0x7f528814d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.778+0000 7f528fa90700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f528812eae0 0x7f5288153140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.778+0000 7f528fa90700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f528812ef20 con 0x7f528812f2b0 2019-12-05T06:37:03.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.778+0000 7f528fa90700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5288098550 con 0x7f528812eae0 2019-12-05T06:37:03.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.778+0000 7f528fa90700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5288098820 con 0x7f528814d4a0 2019-12-05T06:37:03.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.778+0000 7f528e82e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f528812eae0 0x7f5288153140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:03.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.778+0000 7f528e82e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f528812eae0 0x7f5288153140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57244/0 (socket says 172.21.15.16:57244) 2019-12-05T06:37:03.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.778+0000 7f528e82e700 1 -- 172.21.15.16:0/3983866944 learned_addr learned my addr 172.21.15.16:0/3983866944 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:37:03.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.778+0000 7f528e02d700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f528814d4a0 0x7f528814d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:03.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.779+0000 7f528e82e700 1 -- 172.21.15.16:0/3983866944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f528814d4a0 msgr2=0x7f528814d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:03.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.779+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f528814d4a0 0x7f528814d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:03.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.779+0000 7f528e82e700 1 -- 172.21.15.16:0/3983866944 >> v1:172.21.15.76:6789/0 conn(0x7f528812f2b0 legacy=0x7f528812e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:03.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.779+0000 7f528e82e700 1 -- 172.21.15.16:0/3983866944 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f528812d9e0 con 0x7f528812eae0 2019-12-05T06:37:03.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.779+0000 7f527effd700 1 -- 172.21.15.16:0/3983866944 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4217644121 0 0) 0x7f5278007920 con 0x7f528812f2b0 2019-12-05T06:37:03.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.779+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> v2:172.21.15.16:3300/0 conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.779+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> v2:172.21.15.16:3300/0 conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.779+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=READY pgs=1080 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:03.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.779+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.779+0000 7f527effd700 1 -- 172.21.15.16:0/3983866944 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5280005af0 con 0x7f528812eae0 2019-12-05T06:37:03.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.780+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.780+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.780+0000 7f527effd700 1 -- 172.21.15.16:0/3983866944 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5280005710 con 0x7f528812eae0 2019-12-05T06:37:03.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.780+0000 7f527effd700 1 -- 172.21.15.16:0/3983866944 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52800060d0 con 0x7f528812eae0 2019-12-05T06:37:03.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.780+0000 7f528fa90700 1 -- 172.21.15.16:0/3983866944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f528812d9e0 con 0x7f528812eae0 2019-12-05T06:37:03.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.780+0000 7f528fa90700 1 -- 172.21.15.16:0/3983866944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f528812d9e0 con 0x7f528812eae0 2019-12-05T06:37:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.781+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.781+0000 7f527effd700 1 -- 172.21.15.16:0/3983866944 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5280005900 con 0x7f528812eae0 2019-12-05T06:37:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.781+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.782+0000 7f527effd700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5270023630 0x7f5270025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.782+0000 7f527effd700 1 -- 172.21.15.16:0/3983866944 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f5280046120 con 0x7f528812eae0 2019-12-05T06:37:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.782+0000 7f528e02d700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5270023630 0x7f5270025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:03.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.783+0000 7f528e02d700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5270023630 0x7f5270025ac0 crc :-1 s=READY pgs=460 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:03.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.784+0000 7f528fa90700 1 -- 172.21.15.16:0/3983866944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f528812d9e0 con 0x7f528812eae0 2019-12-05T06:37:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.790+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:03.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:03.790+0000 7f527effd700 1 -- 172.21.15.16:0/3983866944 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5280007920 con 0x7f528812eae0 2019-12-05T06:37:04.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.032+0000 7f528fa90700 1 -- 172.21.15.16:0/3983866944 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f528812d9e0 con 0x7f528812eae0 2019-12-05T06:37:04.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.032+0000 7f528e82e700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=THROTTLE_DONE pgs=1080 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:04.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.033+0000 7f527effd700 1 -- 172.21.15.16:0/3983866944 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f52800058e0 con 0x7f528812eae0 2019-12-05T06:37:04.032 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 132 2019-12-05T06:37:04.032 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:37:04.032 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:04.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 -- 172.21.15.16:0/3983866944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5270023630 msgr2=0x7f5270025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:04.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5270023630 0x7f5270025ac0 crc :-1 s=READY pgs=460 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:04.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 -- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 msgr2=0x7f5288153140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:04.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 0x7f5288153140 secure :-1 s=READY pgs=1080 cs=0 l=1 rx=0x7f5280010790 tx=0x7f5280010610).stop 2019-12-05T06:37:04.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 -- 172.21.15.16:0/3983866944 shutdown_connections 2019-12-05T06:37:04.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5270023630 0x7f5270025ac0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:04.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f528812eae0 0x7f5288153140 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:04.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 --2- 172.21.15.16:0/3983866944 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f528814d4a0 0x7f528814d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:04.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 -- 172.21.15.16:0/3983866944 shutdown_connections 2019-12-05T06:37:04.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 -- 172.21.15.16:0/3983866944 wait complete. 2019-12-05T06:37:04.036 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:04.035+0000 7f528fa90700 1 -- 172.21.15.16:0/3983866944 >> 172.21.15.16:0/3983866944 conn(0x7f528812d0e0 msgr2=0x7f5288135a40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:04.046 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:37:06.047 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:06.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.309+0000 7f833dc47700 1 Processor -- start 2019-12-05T06:37:06.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.309+0000 7f833dc47700 1 -- start start 2019-12-05T06:37:06.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.311+0000 7f833dc47700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f833808c6d0 0x7f833808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:06.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.311+0000 7f833dc47700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:06.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.311+0000 7f833dc47700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f833808fb70 con 0x7f833808fef0 2019-12-05T06:37:06.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.311+0000 7f833dc47700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833808df90 con 0x7f833808c6d0 2019-12-05T06:37:06.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.311+0000 7f833dc47700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380a3f70 con 0x7f833808f720 2019-12-05T06:37:06.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.312+0000 7f8337fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f833808c6d0 0x7f833808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:06.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.312+0000 7f833c9e5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:06.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.312+0000 7f833c9e5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:33014/0 (socket says 172.21.15.16:33014) 2019-12-05T06:37:06.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.312+0000 7f8337fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f833808c6d0 0x7f833808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:43342/0 (socket says 172.21.15.16:43342) 2019-12-05T06:37:06.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.312+0000 7f833c9e5700 1 -- 172.21.15.16:0/3810512504 learned_addr learned my addr 172.21.15.16:0/3810512504 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:37:06.319 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.312+0000 7f83377fe700 1 --1- 172.21.15.16:0/3810512504 >> v1:172.21.15.76:6789/0 conn(0x7f833808fef0 0x7f833808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:35492/0 (socket says 172.21.15.16:35492) 2019-12-05T06:37:06.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.312+0000 7f833c9e5700 1 -- 172.21.15.16:0/3810512504 >> v2:172.21.15.76:3300/0 conn(0x7f833808c6d0 msgr2=0x7f833808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:06.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.312+0000 7f833c9e5700 1 --2- 172.21.15.16:0/3810512504 >> v2:172.21.15.76:3300/0 conn(0x7f833808c6d0 0x7f833808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:06.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.312+0000 7f833c9e5700 1 -- 172.21.15.16:0/3810512504 >> v1:172.21.15.76:6789/0 conn(0x7f833808fef0 legacy=0x7f833808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:06.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.313+0000 7f8336ffd700 1 -- 172.21.15.16:0/3810512504 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3230652815 0 0) 0x7f8320001080 con 0x7f833808fef0 2019-12-05T06:37:06.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.313+0000 7f833c9e5700 1 -- 172.21.15.16:0/3810512504 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f833808d540 con 0x7f833808f720 2019-12-05T06:37:06.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.313+0000 7f8337fff700 1 --2- 172.21.15.16:0/3810512504 >> v2:172.21.15.76:3300/0 conn(0x7f833808c6d0 0x7f833808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:06.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.313+0000 7f833c9e5700 1 --2- 172.21.15.16:0/3810512504 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f833000f760 tx=0x7f833000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.313+0000 7f833c9e5700 1 --2- 172.21.15.16:0/3810512504 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f833000f760 tx=0x7f833000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.313+0000 7f833c9e5700 1 --2- 172.21.15.16:0/3810512504 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 secure :-1 s=READY pgs=247 cs=0 l=1 rx=0x7f833000f760 tx=0x7f833000fb00).ready entity=mon.2 client_cookie=9663c6b8878a40a3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:06.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.313+0000 7f833c9e5700 1 --2- 172.21.15.16:0/3810512504 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 secure :-1 s=THROTTLE_DONE pgs=247 cs=0 l=1 rx=0x7f833000f760 tx=0x7f833000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.313+0000 7f8336ffd700 1 -- 172.21.15.16:0/3810512504 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8330000a70 con 0x7f833808f720 2019-12-05T06:37:06.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.313+0000 7f833c9e5700 1 --2- 172.21.15.16:0/3810512504 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 secure :-1 s=THROTTLE_DONE pgs=247 cs=0 l=1 rx=0x7f833000f760 tx=0x7f833000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.313+0000 7f833c9e5700 1 --2- 172.21.15.16:0/3810512504 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 secure :-1 s=THROTTLE_DONE pgs=247 cs=0 l=1 rx=0x7f833000f760 tx=0x7f833000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.314+0000 7f8336ffd700 1 -- 172.21.15.16:0/3810512504 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8330000e70 con 0x7f833808f720 2019-12-05T06:37:06.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.314+0000 7f8336ffd700 1 -- 172.21.15.16:0/3810512504 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8330001500 con 0x7f833808f720 2019-12-05T06:37:06.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.314+0000 7f833dc47700 1 -- 172.21.15.16:0/3810512504 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 msgr2=0x7f833808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:06.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.314+0000 7f833dc47700 1 --2- 172.21.15.16:0/3810512504 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 secure :-1 s=READY pgs=247 cs=0 l=1 rx=0x7f833000f760 tx=0x7f833000fb00).stop 2019-12-05T06:37:06.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.315+0000 7f833dc47700 1 -- 172.21.15.16:0/3810512504 shutdown_connections 2019-12-05T06:37:06.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.315+0000 7f833dc47700 1 --2- 172.21.15.16:0/3810512504 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833808f720 0x7f833808d000 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:06.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.315+0000 7f833dc47700 1 --2- 172.21.15.16:0/3810512504 >> v2:172.21.15.76:3300/0 conn(0x7f833808c6d0 0x7f833808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.315+0000 7f833dc47700 1 -- 172.21.15.16:0/3810512504 shutdown_connections 2019-12-05T06:37:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.315+0000 7f833dc47700 1 -- 172.21.15.16:0/3810512504 wait complete. 2019-12-05T06:37:06.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.315+0000 7f833dc47700 1 -- 172.21.15.16:0/3810512504 >> 172.21.15.16:0/3810512504 conn(0x7f8338061940 msgr2=0x7f8338096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.321+0000 7f833dc47700 1 Processor -- start 2019-12-05T06:37:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.322+0000 7f833dc47700 1 -- start start 2019-12-05T06:37:06.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.323+0000 7f833dc47700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833814f390 0x7f833814f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:06.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.323+0000 7f833dc47700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8338086910 0x7f833814fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:06.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.323+0000 7f833dc47700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8338085940 0x7f8338085de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:06.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.323+0000 7f833dc47700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f833809f630 con 0x7f8338085940 2019-12-05T06:37:06.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.323+0000 7f833dc47700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f83380a2290 con 0x7f8338086910 2019-12-05T06:37:06.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.323+0000 7f833dc47700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f83380a2560 con 0x7f833814f390 2019-12-05T06:37:06.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f833c9e5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8338085940 0x7f8338085de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:06.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f833c9e5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8338085940 0x7f8338085de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57262/0 (socket says 172.21.15.16:57262) 2019-12-05T06:37:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f833c9e5700 1 -- 172.21.15.16:0/2102478855 learned_addr learned my addr 172.21.15.16:0/2102478855 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:37:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f83377fe700 1 --2- 172.21.15.16:0/2102478855 >> v2:172.21.15.76:3300/0 conn(0x7f8338086910 0x7f833814fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f8337fff700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833814f390 0x7f833814f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f833c9e5700 1 -- 172.21.15.16:0/2102478855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833814f390 msgr2=0x7f833814f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833814f390 0x7f833814f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:06.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f833c9e5700 1 -- 172.21.15.16:0/2102478855 >> v2:172.21.15.76:3300/0 conn(0x7f8338086910 msgr2=0x7f833814fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> v2:172.21.15.76:3300/0 conn(0x7f8338086910 0x7f833814fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f833c9e5700 1 -- 172.21.15.16:0/2102478855 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8338085040 con 0x7f8338085940 2019-12-05T06:37:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f83377fe700 1 --2- 172.21.15.16:0/2102478855 >> v2:172.21.15.76:3300/0 conn(0x7f8338086910 0x7f833814fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.324+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> v2:172.21.15.16:3300/0 conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.325+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> v2:172.21.15.16:3300/0 conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.325+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=READY pgs=1081 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:06.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.325+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=THROTTLE_DONE pgs=1081 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.325+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=THROTTLE_DONE pgs=1081 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.325+0000 7f8334ff9700 1 -- 172.21.15.16:0/2102478855 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f83300081b0 con 0x7f8338085940 2019-12-05T06:37:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.325+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=THROTTLE_DONE pgs=1081 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.325+0000 7f8334ff9700 1 -- 172.21.15.16:0/2102478855 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8330000c60 con 0x7f8338085940 2019-12-05T06:37:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.325+0000 7f8334ff9700 1 -- 172.21.15.16:0/2102478855 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8330001370 con 0x7f8338085940 2019-12-05T06:37:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.325+0000 7f833dc47700 1 -- 172.21.15.16:0/2102478855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8338085040 con 0x7f8338085940 2019-12-05T06:37:06.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.325+0000 7f833dc47700 1 -- 172.21.15.16:0/2102478855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8338085040 con 0x7f8338085940 2019-12-05T06:37:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.326+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=THROTTLE_DONE pgs=1081 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.327+0000 7f8334ff9700 1 -- 172.21.15.16:0/2102478855 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8330000e50 con 0x7f8338085940 2019-12-05T06:37:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.327+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=THROTTLE_DONE pgs=1081 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.327+0000 7f8334ff9700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f831c023630 0x7f831c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.327+0000 7f8334ff9700 1 -- 172.21.15.16:0/2102478855 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 7751+0+0 (secure 0 0 0) 0x7f8330045d00 con 0x7f8338085940 2019-12-05T06:37:06.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.327+0000 7f8337fff700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f831c023630 0x7f831c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:06.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.328+0000 7f8337fff700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f831c023630 0x7f831c025ac0 crc :-1 s=READY pgs=461 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:06.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.340+0000 7f833dc47700 1 -- 172.21.15.16:0/2102478855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8338085040 con 0x7f8338085940 2019-12-05T06:37:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.346+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=THROTTLE_DONE pgs=1081 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.347+0000 7f8334ff9700 1 -- 172.21.15.16:0/2102478855 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8330007350 con 0x7f8338085940 2019-12-05T06:37:06.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.592+0000 7f833dc47700 1 -- 172.21.15.16:0/2102478855 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8338085040 con 0x7f8338085940 2019-12-05T06:37:06.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.593+0000 7f833c9e5700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=THROTTLE_DONE pgs=1081 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:06.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.593+0000 7f8334ff9700 1 -- 172.21.15.16:0/2102478855 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f8330000e50 con 0x7f8338085940 2019-12-05T06:37:06.593 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 132 2019-12-05T06:37:06.593 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:37:06.593 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6940,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":132}],"filesystems":[{"mdsmap":{"epoch":131,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:36:08.332556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":69,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5483},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5483":{"gid":5483,"name":"a","rank":0,"incarnation":128,"state":"up:active","state_seq":39,"addr":"172.21.15.16:6833/3678674944","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3678674944},{"type":"v1","addr":"172.21.15.16:6833","nonce":3678674944}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 -- 172.21.15.16:0/2102478855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f831c023630 msgr2=0x7f831c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f831c023630 0x7f831c025ac0 crc :-1 s=READY pgs=461 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 -- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 msgr2=0x7f8338085de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 0x7f8338085de0 secure :-1 s=READY pgs=1081 cs=0 l=1 rx=0x7f83300103f0 tx=0x7f8330010790).stop 2019-12-05T06:37:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 -- 172.21.15.16:0/2102478855 shutdown_connections 2019-12-05T06:37:06.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f831c023630 0x7f831c025ac0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:06.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8338085940 0x7f8338085de0 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:06.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 --2- 172.21.15.16:0/2102478855 >> v2:172.21.15.76:3300/0 conn(0x7f8338086910 0x7f833814fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:06.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 --2- 172.21.15.16:0/2102478855 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f833814f390 0x7f833814f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:06.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 -- 172.21.15.16:0/2102478855 shutdown_connections 2019-12-05T06:37:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 -- 172.21.15.16:0/2102478855 wait complete. 2019-12-05T06:37:06.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:06.596+0000 7f833dc47700 1 -- 172.21.15.16:0/2102478855 >> 172.21.15.16:0/2102478855 conn(0x7f833812f780 msgr2=0x7f8338096b90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:06.605 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:37:08.606 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:08.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.901+0000 7fd32d93e700 1 Processor -- start 2019-12-05T06:37:08.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.901+0000 7fd32d93e700 1 -- start start 2019-12-05T06:37:08.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.903+0000 7fd32d93e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd32808c6d0 0x7fd32808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:08.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.903+0000 7fd32d93e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd32808f720 0x7fd32808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:08.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.903+0000 7fd32d93e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3280958e0 con 0x7fd32808fef0 2019-12-05T06:37:08.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.903+0000 7fd32d93e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3280a3f70 con 0x7fd32808f720 2019-12-05T06:37:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.903+0000 7fd32d93e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd32808d830 con 0x7fd32808c6d0 2019-12-05T06:37:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.904+0000 7fd326ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd32808fef0 0x7fd32808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42436/0 (socket says 172.21.15.16:42436) 2019-12-05T06:37:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.904+0000 7fd326ffd700 1 -- 172.21.15.16:0/3762582012 learned_addr learned my addr 172.21.15.16:0/3762582012 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.904+0000 7fd3277fe700 1 --2- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd32808c6d0 0x7fd32808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.904+0000 7fd327fff700 1 --2- 172.21.15.16:0/3762582012 >> v2:172.21.15.76:3300/0 conn(0x7fd32808f720 0x7fd32808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.904+0000 7fd3267fc700 1 -- 172.21.15.16:0/3762582012 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210025286 0 0) 0x7fd3100014d0 con 0x7fd32808fef0 2019-12-05T06:37:08.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.904+0000 7fd3267fc700 1 -- 172.21.15.16:0/3762582012 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd314002e80 con 0x7fd32808fef0 2019-12-05T06:37:08.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.904+0000 7fd327fff700 1 -- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd32808c6d0 msgr2=0x7fd32808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:08.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd327fff700 1 --2- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd32808c6d0 0x7fd32808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:08.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd327fff700 1 -- 172.21.15.16:0/3762582012 >> v1:172.21.15.16:6789/0 conn(0x7fd32808fef0 legacy=0x7fd32808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:08.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd327fff700 1 -- 172.21.15.16:0/3762582012 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd32808fbc0 con 0x7fd32808f720 2019-12-05T06:37:08.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd3277fe700 1 --2- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd32808c6d0 0x7fd32808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:08.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd327fff700 1 --2- 172.21.15.16:0/3762582012 >> v2:172.21.15.76:3300/0 conn(0x7fd32808f720 0x7fd32808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd32000af80 tx=0x7fd32000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd327fff700 1 --2- 172.21.15.16:0/3762582012 >> v2:172.21.15.76:3300/0 conn(0x7fd32808f720 0x7fd32808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd32000af80 tx=0x7fd32000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd327fff700 1 --2- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd32808f720 0x7fd32808d000 secure :-1 s=READY pgs=751 cs=0 l=1 rx=0x7fd32000af80 tx=0x7fd32000b790).ready entity=mon.1 client_cookie=5e0216243df07b14 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:08.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd327fff700 1 --2- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd32808f720 0x7fd32808d000 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7fd32000af80 tx=0x7fd32000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd3267fc700 1 -- 172.21.15.16:0/3762582012 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd320000bc0 con 0x7fd32808f720 2019-12-05T06:37:08.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd327fff700 1 --2- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd32808f720 0x7fd32808d000 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7fd32000af80 tx=0x7fd32000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.905+0000 7fd327fff700 1 --2- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd32808f720 0x7fd32808d000 secure :-1 s=THROTTLE_DONE pgs=751 cs=0 l=1 rx=0x7fd32000af80 tx=0x7fd32000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.906+0000 7fd3267fc700 1 -- 172.21.15.16:0/3762582012 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3200097f0 con 0x7fd32808f720 2019-12-05T06:37:08.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.906+0000 7fd3267fc700 1 -- 172.21.15.16:0/3762582012 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd320009ec0 con 0x7fd32808f720 2019-12-05T06:37:08.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.906+0000 7fd32d93e700 1 -- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd32808f720 msgr2=0x7fd32808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:08.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.906+0000 7fd32d93e700 1 --2- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd32808f720 0x7fd32808d000 secure :-1 s=READY pgs=751 cs=0 l=1 rx=0x7fd32000af80 tx=0x7fd32000b790).stop 2019-12-05T06:37:08.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.907+0000 7fd32d93e700 1 -- 172.21.15.16:0/3762582012 shutdown_connections 2019-12-05T06:37:08.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.907+0000 7fd32d93e700 1 --2- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fd32808f720 0x7fd32808d000 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:08.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.907+0000 7fd32d93e700 1 --2- 172.21.15.16:0/3762582012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd32808c6d0 0x7fd32808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:08.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.907+0000 7fd32d93e700 1 -- 172.21.15.16:0/3762582012 shutdown_connections 2019-12-05T06:37:08.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.907+0000 7fd32d93e700 1 -- 172.21.15.16:0/3762582012 wait complete. 2019-12-05T06:37:08.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.907+0000 7fd32d93e700 1 -- 172.21.15.16:0/3762582012 >> 172.21.15.16:0/3762582012 conn(0x7fd328061940 msgr2=0x7fd328096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:08.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.913+0000 7fd32d93e700 1 Processor -- start 2019-12-05T06:37:08.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.913+0000 7fd32d93e700 1 -- start start 2019-12-05T06:37:08.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.915+0000 7fd32d93e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd32806dc50 0x7fd32806e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:08.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.915+0000 7fd32d93e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.915+0000 7fd32d93e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3281350d0 con 0x7fd328135450 2019-12-05T06:37:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.915+0000 7fd32d93e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd328132830 con 0x7fd32806dc50 2019-12-05T06:37:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.915+0000 7fd32d93e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3281334f0 con 0x7fd328134c80 2019-12-05T06:37:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.915+0000 7fd3277fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd32806dc50 0x7fd32806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.915+0000 7fd3277fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd32806dc50 0x7fd32806e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57272/0 (socket says 172.21.15.16:57272) 2019-12-05T06:37:08.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.915+0000 7fd3277fe700 1 -- 172.21.15.16:0/420220486 learned_addr learned my addr 172.21.15.16:0/420220486 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:37:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.915+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.916+0000 7fd327fff700 1 -- 172.21.15.16:0/420220486 >> v2:172.21.15.16:3300/0 conn(0x7fd32806dc50 msgr2=0x7fd32806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.916+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> v2:172.21.15.16:3300/0 conn(0x7fd32806dc50 0x7fd32806e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.916+0000 7fd327fff700 1 -- 172.21.15.16:0/420220486 >> v1:172.21.15.16:6789/0 conn(0x7fd328135450 legacy=0x7fd328134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.916+0000 7fd327fff700 1 -- 172.21.15.16:0/420220486 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd328133b80 con 0x7fd328134c80 2019-12-05T06:37:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.916+0000 7fd3277fe700 1 --2- 172.21.15.16:0/420220486 >> v2:172.21.15.16:3300/0 conn(0x7fd32806dc50 0x7fd32806e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.916+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.916+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.916+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=READY pgs=248 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.917+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.917+0000 7fd31ffff700 1 -- 172.21.15.16:0/420220486 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd32000b050 con 0x7fd328134c80 2019-12-05T06:37:08.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.917+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.917+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.917+0000 7fd31ffff700 1 -- 172.21.15.16:0/420220486 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3200095b0 con 0x7fd328134c80 2019-12-05T06:37:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.917+0000 7fd31ffff700 1 -- 172.21.15.16:0/420220486 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd320009c80 con 0x7fd328134c80 2019-12-05T06:37:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.917+0000 7fd32d93e700 1 -- 172.21.15.16:0/420220486 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd328133b80 con 0x7fd328134c80 2019-12-05T06:37:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.917+0000 7fd32d93e700 1 -- 172.21.15.16:0/420220486 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd328133b80 con 0x7fd328134c80 2019-12-05T06:37:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.918+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.919+0000 7fd31ffff700 1 -- 172.21.15.16:0/420220486 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd32000ac80 con 0x7fd328134c80 2019-12-05T06:37:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.919+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.919+0000 7fd31ffff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd30c023630 0x7fd30c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.919+0000 7fd31ffff700 1 -- 172.21.15.16:0/420220486 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7fd320089d20 con 0x7fd328134c80 2019-12-05T06:37:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.919+0000 7fd3277fe700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd30c023630 0x7fd30c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.920+0000 7fd3277fe700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd30c023630 0x7fd30c025ac0 crc :-1 s=READY pgs=462 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:08.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.933+0000 7fd32d93e700 1 -- 172.21.15.16:0/420220486 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd328133b80 con 0x7fd328134c80 2019-12-05T06:37:08.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.941+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:08.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:08.941+0000 7fd31ffff700 1 -- 172.21.15.16:0/420220486 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd320002b40 con 0x7fd328134c80 2019-12-05T06:37:09.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.204+0000 7fd32d93e700 1 -- 172.21.15.16:0/420220486 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd328133b80 con 0x7fd328134c80 2019-12-05T06:37:09.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.205+0000 7fd327fff700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=THROTTLE_DONE pgs=248 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.205+0000 7fd31ffff700 1 -- 172.21.15.16:0/420220486 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fd32000abe0 con 0x7fd328134c80 2019-12-05T06:37:09.207 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 133 2019-12-05T06:37:09.208 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:37:09.208 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:08.340682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.210+0000 7fd32d93e700 1 -- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd30c023630 msgr2=0x7fd30c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.210+0000 7fd32d93e700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd30c023630 0x7fd30c025ac0 crc :-1 s=READY pgs=462 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.210+0000 7fd32d93e700 1 -- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 msgr2=0x7fd32806e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.210+0000 7fd32d93e700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 secure :-1 s=READY pgs=248 cs=0 l=1 rx=0x7fd3200543c0 tx=0x7fd320054760).stop 2019-12-05T06:37:09.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.210+0000 7fd32d93e700 1 -- 172.21.15.16:0/420220486 shutdown_connections 2019-12-05T06:37:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.210+0000 7fd32d93e700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd328134c80 0x7fd32806e580 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.210+0000 7fd32d93e700 1 --2- 172.21.15.16:0/420220486 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd30c023630 0x7fd30c025ac0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.210+0000 7fd32d93e700 1 --2- 172.21.15.16:0/420220486 >> v2:172.21.15.16:3300/0 conn(0x7fd32806dc50 0x7fd32806e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.211+0000 7fd32d93e700 1 -- 172.21.15.16:0/420220486 shutdown_connections 2019-12-05T06:37:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.211+0000 7fd32d93e700 1 -- 172.21.15.16:0/420220486 wait complete. 2019-12-05T06:37:09.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.211+0000 7fd32d93e700 1 -- 172.21.15.16:0/420220486 >> 172.21.15.16:0/420220486 conn(0x7fd3280a2850 msgr2=0x7fd32806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:09.221 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:37:09.221 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:37:09.221 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.478+0000 7fb59b22d700 1 Processor -- start 2019-12-05T06:37:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.479+0000 7fb59b22d700 1 -- start start 2019-12-05T06:37:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.482+0000 7fb59b22d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 0x7fb59408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.482+0000 7fb59b22d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5940958e0 con 0x7fb59408c6d0 2019-12-05T06:37:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.482+0000 7fb59b22d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5940629e0 con 0x7fb59408fef0 2019-12-05T06:37:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.482+0000 7fb59b22d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb59408df90 con 0x7fb59408d2c0 2019-12-05T06:37:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.482+0000 7fb5997ca700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb59408c6d0 0x7fb59408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42448/0 (socket says 172.21.15.16:42448) 2019-12-05T06:37:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.482+0000 7fb598fc9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 0x7fb59408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.482+0000 7fb5997ca700 1 -- 172.21.15.16:0/2492526039 learned_addr learned my addr 172.21.15.16:0/2492526039 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.483+0000 7fb58bfff700 1 -- 172.21.15.16:0/2492526039 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298195379 0 0) 0x7fb58c000d30 con 0x7fb59408fef0 2019-12-05T06:37:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.483+0000 7fb58bfff700 1 -- 172.21.15.16:0/2492526039 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb580002e80 con 0x7fb59408fef0 2019-12-05T06:37:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.483+0000 7fb58bfff700 1 -- 172.21.15.16:0/2492526039 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009500644 0 0) 0x7fb590001100 con 0x7fb59408c6d0 2019-12-05T06:37:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.483+0000 7fb598fc9700 1 -- 172.21.15.16:0/2492526039 >> v1:172.21.15.76:6789/0 conn(0x7fb59408fef0 legacy=0x7fb59408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.483+0000 7fb598fc9700 1 -- 172.21.15.16:0/2492526039 >> v1:172.21.15.16:6789/0 conn(0x7fb59408c6d0 legacy=0x7fb59408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.483+0000 7fb598fc9700 1 -- 172.21.15.16:0/2492526039 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59408e620 con 0x7fb59408d2c0 2019-12-05T06:37:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.483+0000 7fb598fc9700 1 --2- 172.21.15.16:0/2492526039 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 0x7fb59408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb58400af50 tx=0x7fb58400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.483+0000 7fb598fc9700 1 --2- 172.21.15.16:0/2492526039 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 0x7fb59408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb58400af50 tx=0x7fb58400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.484+0000 7fb598fc9700 1 --2- 172.21.15.16:0/2492526039 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 0x7fb59408ef20 secure :-1 s=READY pgs=249 cs=0 l=1 rx=0x7fb58400af50 tx=0x7fb58400b760).ready entity=mon.2 client_cookie=70b10cd0acbe219b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.484+0000 7fb598fc9700 1 --2- 172.21.15.16:0/2492526039 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 0x7fb59408ef20 secure :-1 s=THROTTLE_DONE pgs=249 cs=0 l=1 rx=0x7fb58400af50 tx=0x7fb58400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.484+0000 7fb58bfff700 1 -- 172.21.15.16:0/2492526039 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb584000960 con 0x7fb59408d2c0 2019-12-05T06:37:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.484+0000 7fb598fc9700 1 --2- 172.21.15.16:0/2492526039 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 0x7fb59408ef20 secure :-1 s=THROTTLE_DONE pgs=249 cs=0 l=1 rx=0x7fb58400af50 tx=0x7fb58400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.484+0000 7fb598fc9700 1 --2- 172.21.15.16:0/2492526039 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 0x7fb59408ef20 secure :-1 s=THROTTLE_DONE pgs=249 cs=0 l=1 rx=0x7fb58400af50 tx=0x7fb58400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.484+0000 7fb58bfff700 1 -- 172.21.15.16:0/2492526039 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb584009860 con 0x7fb59408d2c0 2019-12-05T06:37:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.484+0000 7fb58bfff700 1 -- 172.21.15.16:0/2492526039 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb584009f30 con 0x7fb59408d2c0 2019-12-05T06:37:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.484+0000 7fb59b22d700 1 -- 172.21.15.16:0/2492526039 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 msgr2=0x7fb59408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.484+0000 7fb59b22d700 1 --2- 172.21.15.16:0/2492526039 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 0x7fb59408ef20 secure :-1 s=READY pgs=249 cs=0 l=1 rx=0x7fb58400af50 tx=0x7fb58400b760).stop 2019-12-05T06:37:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.487+0000 7fb59b22d700 1 -- 172.21.15.16:0/2492526039 shutdown_connections 2019-12-05T06:37:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.487+0000 7fb59b22d700 1 --2- 172.21.15.16:0/2492526039 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb59408d2c0 0x7fb59408ef20 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.487+0000 7fb59b22d700 1 -- 172.21.15.16:0/2492526039 shutdown_connections 2019-12-05T06:37:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.487+0000 7fb59b22d700 1 -- 172.21.15.16:0/2492526039 wait complete. 2019-12-05T06:37:09.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.487+0000 7fb59b22d700 1 -- 172.21.15.16:0/2492526039 >> 172.21.15.16:0/2492526039 conn(0x7fb594061940 msgr2=0x7fb594096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.493+0000 7fb59b22d700 1 Processor -- start 2019-12-05T06:37:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.493+0000 7fb59b22d700 1 -- start start 2019-12-05T06:37:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.495+0000 7fb59b22d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5940a13f0 0x7fb594154c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.495+0000 7fb59b22d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb594155140 0x7fb594157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:09.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.495+0000 7fb59b22d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb59409b3f0 con 0x7fb59406d6c0 2019-12-05T06:37:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.495+0000 7fb59b22d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb59409fc60 con 0x7fb5940a13f0 2019-12-05T06:37:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.495+0000 7fb59b22d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb59409f230 con 0x7fb594155140 2019-12-05T06:37:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb5997ca700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb59406d6c0 0x7fb5940a0bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:35524/0 (socket says 172.21.15.16:35524) 2019-12-05T06:37:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb598fc9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb5940a13f0 0x7fb594154c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb5997ca700 1 -- 172.21.15.16:0/248006768 learned_addr learned my addr 172.21.15.16:0/248006768 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:09.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> v2:172.21.15.76:3300/0 conn(0x7fb594155140 0x7fb594157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb589ffb700 1 -- 172.21.15.16:0/248006768 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4132577519 0 0) 0x7fb590001150 con 0x7fb59406d6c0 2019-12-05T06:37:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb589ffb700 1 -- 172.21.15.16:0/248006768 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb574002e80 con 0x7fb59406d6c0 2019-12-05T06:37:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb599fcb700 1 -- 172.21.15.16:0/248006768 >> v2:172.21.15.16:3300/0 conn(0x7fb5940a13f0 msgr2=0x7fb594154c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> v2:172.21.15.16:3300/0 conn(0x7fb5940a13f0 0x7fb594154c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb599fcb700 1 -- 172.21.15.16:0/248006768 >> v1:172.21.15.76:6789/0 conn(0x7fb59406d6c0 legacy=0x7fb5940a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb599fcb700 1 -- 172.21.15.16:0/248006768 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5940a02f0 con 0x7fb594155140 2019-12-05T06:37:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb598fc9700 1 --2- 172.21.15.16:0/248006768 >> v2:172.21.15.16:3300/0 conn(0x7fb5940a13f0 0x7fb594154c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> v2:172.21.15.76:3300/0 conn(0x7fb594155140 0x7fb594157540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> v2:172.21.15.76:3300/0 conn(0x7fb594155140 0x7fb594157540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.496+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 0x7fb594157540 secure :-1 s=READY pgs=754 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.497+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 0x7fb594157540 secure :-1 s=THROTTLE_DONE pgs=754 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.497+0000 7fb589ffb700 1 -- 172.21.15.16:0/248006768 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb58c009d50 con 0x7fb594155140 2019-12-05T06:37:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.497+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 0x7fb594157540 secure :-1 s=THROTTLE_DONE pgs=754 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.497+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 0x7fb594157540 secure :-1 s=THROTTLE_DONE pgs=754 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.497+0000 7fb589ffb700 1 -- 172.21.15.16:0/248006768 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58c009f50 con 0x7fb594155140 2019-12-05T06:37:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.497+0000 7fb589ffb700 1 -- 172.21.15.16:0/248006768 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb58c00a660 con 0x7fb594155140 2019-12-05T06:37:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.497+0000 7fb59b22d700 1 -- 172.21.15.16:0/248006768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5940a02f0 con 0x7fb594155140 2019-12-05T06:37:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.497+0000 7fb59b22d700 1 -- 172.21.15.16:0/248006768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5940a02f0 con 0x7fb594155140 2019-12-05T06:37:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.498+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 0x7fb594157540 secure :-1 s=THROTTLE_DONE pgs=754 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.499+0000 7fb589ffb700 1 -- 172.21.15.16:0/248006768 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb58c009b60 con 0x7fb594155140 2019-12-05T06:37:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.499+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 0x7fb594157540 secure :-1 s=THROTTLE_DONE pgs=754 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.499+0000 7fb589ffb700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb574023670 0x7fb574025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.499+0000 7fb589ffb700 1 -- 172.21.15.16:0/248006768 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7fb58c0462e0 con 0x7fb594155140 2019-12-05T06:37:09.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.499+0000 7fb5997ca700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb574023670 0x7fb574025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.500+0000 7fb5997ca700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb574023670 0x7fb574025ae0 crc :-1 s=READY pgs=463 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.509+0000 7fb59b22d700 1 -- 172.21.15.16:0/248006768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5940a02f0 con 0x7fb594155140 2019-12-05T06:37:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.512+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 0x7fb594157540 secure :-1 s=THROTTLE_DONE pgs=754 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.513+0000 7fb589ffb700 1 -- 172.21.15.16:0/248006768 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb58c009950 con 0x7fb594155140 2019-12-05T06:37:09.649 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:37:09.654 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:37:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.760+0000 7fb59b22d700 1 -- 172.21.15.16:0/248006768 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb5940a02f0 con 0x7fb594155140 2019-12-05T06:37:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.760+0000 7fb599fcb700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 0x7fb594157540 secure :-1 s=THROTTLE_DONE pgs=754 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.760+0000 7fb589ffb700 1 -- 172.21.15.16:0/248006768 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 133 v133) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fb58c009b60 con 0x7fb594155140 2019-12-05T06:37:09.761 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 133 2019-12-05T06:37:09.761 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:37:09.762 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":133,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":133,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:08.340682+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 -- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb574023670 msgr2=0x7fb574025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:09.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb574023670 0x7fb574025ae0 crc :-1 s=READY pgs=463 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 -- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 msgr2=0x7fb594157540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 0x7fb594157540 secure :-1 s=READY pgs=754 cs=0 l=1 rx=0x7fb58c010720 tx=0x7fb58c010ac0).stop 2019-12-05T06:37:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 -- 172.21.15.16:0/248006768 shutdown_connections 2019-12-05T06:37:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb594155140 0x7fb594157540 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 --2- 172.21.15.16:0/248006768 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb574023670 0x7fb574025ae0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:09.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 --2- 172.21.15.16:0/248006768 >> v2:172.21.15.16:3300/0 conn(0x7fb5940a13f0 0x7fb594154c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 -- 172.21.15.16:0/248006768 shutdown_connections 2019-12-05T06:37:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 -- 172.21.15.16:0/248006768 wait complete. 2019-12-05T06:37:09.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:09.763+0000 7fb59b22d700 1 -- 172.21.15.16:0/248006768 >> 172.21.15.16:0/248006768 conn(0x7fb59408ad10 msgr2=0x7fb59406fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:09.777 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 133, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 133, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6940": { "addr": "172.21.15.76:6835/172302337", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 172302337, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 172302337, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6940, "incarnation": 133, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 70, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:37:08.340682+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6940 } } } ], "standbys": [] } 2019-12-05T06:37:11.778 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:12.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.061+0000 7fa81fa98700 1 Processor -- start 2019-12-05T06:37:12.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.061+0000 7fa81fa98700 1 -- start start 2019-12-05T06:37:12.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.063+0000 7fa81fa98700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 0x7fa81808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:12.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.063+0000 7fa81fa98700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa81808f720 0x7fa81808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:12.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.063+0000 7fa81fa98700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8180958e0 con 0x7fa81808fef0 2019-12-05T06:37:12.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.063+0000 7fa81fa98700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8180a3f70 con 0x7fa81808f720 2019-12-05T06:37:12.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.063+0000 7fa81fa98700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa81808d830 con 0x7fa81808c6d0 2019-12-05T06:37:12.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.063+0000 7fa81d834700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa81808fef0 0x7fa81808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42466/0 (socket says 172.21.15.16:42466) 2019-12-05T06:37:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.063+0000 7fa81e035700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 0x7fa81808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.063+0000 7fa81d834700 1 -- 172.21.15.16:0/3956420519 learned_addr learned my addr 172.21.15.16:0/3956420519 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.063+0000 7fa81e836700 1 --2- 172.21.15.16:0/3956420519 >> v2:172.21.15.76:3300/0 conn(0x7fa81808f720 0x7fa81808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.064+0000 7fa81d033700 1 -- 172.21.15.16:0/3956420519 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2811178445 0 0) 0x7fa80c001150 con 0x7fa81808fef0 2019-12-05T06:37:12.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.064+0000 7fa81d033700 1 -- 172.21.15.16:0/3956420519 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa800002e80 con 0x7fa81808fef0 2019-12-05T06:37:12.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.064+0000 7fa81e035700 1 -- 172.21.15.16:0/3956420519 >> v2:172.21.15.76:3300/0 conn(0x7fa81808f720 msgr2=0x7fa81808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:12.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.064+0000 7fa81e035700 1 --2- 172.21.15.16:0/3956420519 >> v2:172.21.15.76:3300/0 conn(0x7fa81808f720 0x7fa81808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:12.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.064+0000 7fa81e035700 1 -- 172.21.15.16:0/3956420519 >> v1:172.21.15.16:6789/0 conn(0x7fa81808fef0 legacy=0x7fa81808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:12.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.064+0000 7fa81e035700 1 -- 172.21.15.16:0/3956420519 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa81808fbc0 con 0x7fa81808c6d0 2019-12-05T06:37:12.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.064+0000 7fa81e035700 1 --2- 172.21.15.16:0/3956420519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 0x7fa81808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa81000b2d0 tx=0x7fa81000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.064+0000 7fa81e035700 1 --2- 172.21.15.16:0/3956420519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 0x7fa81808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa81000b2d0 tx=0x7fa81000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.065+0000 7fa81e035700 1 --2- 172.21.15.16:0/3956420519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 0x7fa81808cac0 secure :-1 s=READY pgs=250 cs=0 l=1 rx=0x7fa81000b2d0 tx=0x7fa81000b5d0).ready entity=mon.2 client_cookie=9245a5d36bff8b38 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.065+0000 7fa81e035700 1 --2- 172.21.15.16:0/3956420519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 0x7fa81808cac0 secure :-1 s=THROTTLE_DONE pgs=250 cs=0 l=1 rx=0x7fa81000b2d0 tx=0x7fa81000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.065+0000 7fa81e035700 1 --2- 172.21.15.16:0/3956420519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 0x7fa81808cac0 secure :-1 s=THROTTLE_DONE pgs=250 cs=0 l=1 rx=0x7fa81000b2d0 tx=0x7fa81000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.065+0000 7fa81d033700 1 -- 172.21.15.16:0/3956420519 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa810007590 con 0x7fa81808c6d0 2019-12-05T06:37:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.065+0000 7fa81e035700 1 --2- 172.21.15.16:0/3956420519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 0x7fa81808cac0 secure :-1 s=THROTTLE_DONE pgs=250 cs=0 l=1 rx=0x7fa81000b2d0 tx=0x7fa81000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.065+0000 7fa81d033700 1 -- 172.21.15.16:0/3956420519 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa810009d30 con 0x7fa81808c6d0 2019-12-05T06:37:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.065+0000 7fa81d033700 1 -- 172.21.15.16:0/3956420519 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa81000a340 con 0x7fa81808c6d0 2019-12-05T06:37:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.065+0000 7fa81fa98700 1 -- 172.21.15.16:0/3956420519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 msgr2=0x7fa81808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.065+0000 7fa81fa98700 1 --2- 172.21.15.16:0/3956420519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 0x7fa81808cac0 secure :-1 s=READY pgs=250 cs=0 l=1 rx=0x7fa81000b2d0 tx=0x7fa81000b5d0).stop 2019-12-05T06:37:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.067+0000 7fa81fa98700 1 -- 172.21.15.16:0/3956420519 shutdown_connections 2019-12-05T06:37:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.067+0000 7fa81fa98700 1 --2- 172.21.15.16:0/3956420519 >> v2:172.21.15.76:3300/0 conn(0x7fa81808f720 0x7fa81808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:12.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.067+0000 7fa81fa98700 1 --2- 172.21.15.16:0/3956420519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa81808c6d0 0x7fa81808cac0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:12.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.068+0000 7fa81fa98700 1 -- 172.21.15.16:0/3956420519 shutdown_connections 2019-12-05T06:37:12.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.068+0000 7fa81fa98700 1 -- 172.21.15.16:0/3956420519 wait complete. 2019-12-05T06:37:12.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.068+0000 7fa81fa98700 1 -- 172.21.15.16:0/3956420519 >> 172.21.15.16:0/3956420519 conn(0x7fa818061940 msgr2=0x7fa818096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.073+0000 7fa81fa98700 1 Processor -- start 2019-12-05T06:37:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.073+0000 7fa81fa98700 1 -- start start 2019-12-05T06:37:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.075+0000 7fa81fa98700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8181302f0 0x7fa81812a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:12.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.075+0000 7fa81fa98700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa818153b50 0x7fa818155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.075+0000 7fa81fa98700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa818131bf0 con 0x7fa81812b640 2019-12-05T06:37:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.075+0000 7fa81fa98700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa818128cb0 con 0x7fa818153b50 2019-12-05T06:37:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.075+0000 7fa81fa98700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa818128f80 con 0x7fa8181302f0 2019-12-05T06:37:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.075+0000 7fa81d834700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa81812b640 0x7fa81812abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42470/0 (socket says 172.21.15.16:42470) 2019-12-05T06:37:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.075+0000 7fa81d834700 1 -- 172.21.15.16:0/1522132825 learned_addr learned my addr 172.21.15.16:0/1522132825 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:12.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.075+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> v2:172.21.15.16:3300/0 conn(0x7fa818153b50 0x7fa818155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.075+0000 7fa81e035700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8181302f0 0x7fa81812a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.075+0000 7fa81e836700 1 -- 172.21.15.16:0/1522132825 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8181302f0 msgr2=0x7fa81812a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8181302f0 0x7fa81812a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:12.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa81e836700 1 -- 172.21.15.16:0/1522132825 >> v1:172.21.15.16:6789/0 conn(0x7fa81812b640 legacy=0x7fa81812abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa816ffd700 1 -- 172.21.15.16:0/1522132825 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678350813 0 0) 0x7fa80c001170 con 0x7fa81812b640 2019-12-05T06:37:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa81e836700 1 -- 172.21.15.16:0/1522132825 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa818129d70 con 0x7fa818153b50 2019-12-05T06:37:12.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> v2:172.21.15.16:3300/0 conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> v2:172.21.15.16:3300/0 conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=READY pgs=1087 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=THROTTLE_DONE pgs=1087 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa816ffd700 1 -- 172.21.15.16:0/1522132825 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa80800ee80 con 0x7fa818153b50 2019-12-05T06:37:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=THROTTLE_DONE pgs=1087 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa816ffd700 1 -- 172.21.15.16:0/1522132825 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa80800ebe0 con 0x7fa818153b50 2019-12-05T06:37:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa81fa98700 1 -- 172.21.15.16:0/1522132825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa818129d70 con 0x7fa818153b50 2019-12-05T06:37:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=THROTTLE_DONE pgs=1087 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.076+0000 7fa816ffd700 1 -- 172.21.15.16:0/1522132825 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa80800edd0 con 0x7fa818153b50 2019-12-05T06:37:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.077+0000 7fa81fa98700 1 -- 172.21.15.16:0/1522132825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa818129d70 con 0x7fa818153b50 2019-12-05T06:37:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.078+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=THROTTLE_DONE pgs=1087 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.079+0000 7fa816ffd700 1 -- 172.21.15.16:0/1522132825 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa80800edd0 con 0x7fa818153b50 2019-12-05T06:37:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.079+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=THROTTLE_DONE pgs=1087 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.079+0000 7fa816ffd700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7f8025850 0x7fa7f8027ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:12.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.079+0000 7fa816ffd700 1 -- 172.21.15.16:0/1522132825 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7fa808045d40 con 0x7fa818153b50 2019-12-05T06:37:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.079+0000 7fa81e035700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7f8025850 0x7fa7f8027ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:12.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.080+0000 7fa81e035700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7f8025850 0x7fa7f8027ce0 crc :-1 s=READY pgs=464 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:12.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.092+0000 7fa81fa98700 1 -- 172.21.15.16:0/1522132825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa818129d70 con 0x7fa818153b50 2019-12-05T06:37:12.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.100+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=THROTTLE_DONE pgs=1087 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.100+0000 7fa816ffd700 1 -- 172.21.15.16:0/1522132825 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa808000b00 con 0x7fa818153b50 2019-12-05T06:37:12.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.361+0000 7fa81fa98700 1 -- 172.21.15.16:0/1522132825 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa818129d70 con 0x7fa818153b50 2019-12-05T06:37:12.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.361+0000 7fa81e836700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=THROTTLE_DONE pgs=1087 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:12.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.361+0000 7fa816ffd700 1 -- 172.21.15.16:0/1522132825 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 135 v135) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fa80800edd0 con 0x7fa818153b50 2019-12-05T06:37:12.363 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 135 2019-12-05T06:37:12.363 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:37:12.363 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":135,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":135,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:11.668196+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.364+0000 7fa81fa98700 1 -- 172.21.15.16:0/1522132825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7f8025850 msgr2=0x7fa7f8027ce0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.364+0000 7fa81fa98700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7f8025850 0x7fa7f8027ce0 crc :-1 s=READY pgs=464 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.364+0000 7fa81fa98700 1 -- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 msgr2=0x7fa818155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:12.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.364+0000 7fa81fa98700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 0x7fa818155f50 secure :-1 s=READY pgs=1087 cs=0 l=1 rx=0x7fa80800ea40 tx=0x7fa808010240).stop 2019-12-05T06:37:12.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.364+0000 7fa81fa98700 1 -- 172.21.15.16:0/1522132825 shutdown_connections 2019-12-05T06:37:12.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.364+0000 7fa81fa98700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa7f8025850 0x7fa7f8027ce0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:12.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.364+0000 7fa81fa98700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa818153b50 0x7fa818155f50 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:12.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.364+0000 7fa81fa98700 1 --2- 172.21.15.16:0/1522132825 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa8181302f0 0x7fa81812a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:12.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.365+0000 7fa81fa98700 1 -- 172.21.15.16:0/1522132825 shutdown_connections 2019-12-05T06:37:12.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.365+0000 7fa81fa98700 1 -- 172.21.15.16:0/1522132825 wait complete. 2019-12-05T06:37:12.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:12.365+0000 7fa81fa98700 1 -- 172.21.15.16:0/1522132825 >> 172.21.15.16:0/1522132825 conn(0x7fa8180a2850 msgr2=0x7fa81806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:12.376 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:37:14.377 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:14.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.580+0000 7f53a27b3700 1 Processor -- start 2019-12-05T06:37:14.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.580+0000 7f53a27b3700 1 -- start start 2019-12-05T06:37:14.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.581+0000 7f53a27b3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.581+0000 7f53a27b3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f539c08fef0 0x7f539c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.581+0000 7f53a27b3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539c0958e0 con 0x7f539c08c6d0 2019-12-05T06:37:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.581+0000 7f53a27b3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f539c08df90 con 0x7f539c08fef0 2019-12-05T06:37:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.581+0000 7f53a27b3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539c0a3f70 con 0x7f539c08d2c0 2019-12-05T06:37:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.581+0000 7f539bfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:14.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.581+0000 7f53a1551700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f539c08fef0 0x7f539c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f53a1551700 1 -- >> v2:172.21.15.76:3300/0 conn(0x7f539c08fef0 msgr2=0x7f539c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2019-12-05T06:37:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f53a1551700 1 -- >> v2:172.21.15.76:3300/0 conn(0x7f539c08fef0 msgr2=0x7f539c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:37:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f53a1551700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f539c08fef0 0x7f539c08f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:37:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539bfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:33086/0 (socket says 172.21.15.16:33086) 2019-12-05T06:37:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f53a0d50700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f539c08c6d0 0x7f539c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:35564/0 (socket says 172.21.15.16:35564) 2019-12-05T06:37:14.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539bfff700 1 -- 172.21.15.16:0/1339347996 learned_addr learned my addr 172.21.15.16:0/1339347996 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:37:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f53a1551700 1 --2- 172.21.15.16:0/1339347996 >> v2:172.21.15.76:3300/0 conn(0x7f539c08fef0 0x7f539c08f460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:37:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539b7fe700 1 -- 172.21.15.16:0/1339347996 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1195406824 0 0) 0x7f5394001080 con 0x7f539c08c6d0 2019-12-05T06:37:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539b7fe700 1 -- 172.21.15.16:0/1339347996 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5388002e80 con 0x7f539c08c6d0 2019-12-05T06:37:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539bfff700 1 -- 172.21.15.16:0/1339347996 >> v2:172.21.15.76:3300/0 conn(0x7f539c08fef0 msgr2=0x7f539c08f460 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:37:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539bfff700 1 --2- 172.21.15.16:0/1339347996 >> v2:172.21.15.76:3300/0 conn(0x7f539c08fef0 0x7f539c08f460 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:14.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539bfff700 1 -- 172.21.15.16:0/1339347996 >> v1:172.21.15.76:6789/0 conn(0x7f539c08c6d0 legacy=0x7f539c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539bfff700 1 -- 172.21.15.16:0/1339347996 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c08e620 con 0x7f539c08d2c0 2019-12-05T06:37:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539bfff700 1 --2- 172.21.15.16:0/1339347996 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f538c00fa40 tx=0x7f538c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539bfff700 1 --2- 172.21.15.16:0/1339347996 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f538c00fa40 tx=0x7f538c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.582+0000 7f539bfff700 1 --2- 172.21.15.16:0/1339347996 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 secure :-1 s=READY pgs=251 cs=0 l=1 rx=0x7f538c00fa40 tx=0x7f538c00fde0).ready entity=mon.2 client_cookie=6a04c186146937ba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f539bfff700 1 --2- 172.21.15.16:0/1339347996 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 secure :-1 s=THROTTLE_DONE pgs=251 cs=0 l=1 rx=0x7f538c00fa40 tx=0x7f538c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f539b7fe700 1 -- 172.21.15.16:0/1339347996 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f538c000ce0 con 0x7f539c08d2c0 2019-12-05T06:37:14.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f539bfff700 1 --2- 172.21.15.16:0/1339347996 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 secure :-1 s=THROTTLE_DONE pgs=251 cs=0 l=1 rx=0x7f538c00fa40 tx=0x7f538c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f539b7fe700 1 -- 172.21.15.16:0/1339347996 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f538c007590 con 0x7f539c08d2c0 2019-12-05T06:37:14.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f539bfff700 1 --2- 172.21.15.16:0/1339347996 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 secure :-1 s=THROTTLE_DONE pgs=251 cs=0 l=1 rx=0x7f538c00fa40 tx=0x7f538c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f539b7fe700 1 -- 172.21.15.16:0/1339347996 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f538c001200 con 0x7f539c08d2c0 2019-12-05T06:37:14.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f53a27b3700 1 -- 172.21.15.16:0/1339347996 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 msgr2=0x7f539c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:14.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f53a27b3700 1 --2- 172.21.15.16:0/1339347996 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 secure :-1 s=READY pgs=251 cs=0 l=1 rx=0x7f538c00fa40 tx=0x7f538c00fde0).stop 2019-12-05T06:37:14.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f53a27b3700 1 -- 172.21.15.16:0/1339347996 shutdown_connections 2019-12-05T06:37:14.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f53a27b3700 1 --2- 172.21.15.16:0/1339347996 >> v2:172.21.15.76:3300/0 conn(0x7f539c08fef0 0x7f539c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:14.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.583+0000 7f53a27b3700 1 --2- 172.21.15.16:0/1339347996 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c08d2c0 0x7f539c08ef20 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:14.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.584+0000 7f53a27b3700 1 -- 172.21.15.16:0/1339347996 shutdown_connections 2019-12-05T06:37:14.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.584+0000 7f53a27b3700 1 -- 172.21.15.16:0/1339347996 wait complete. 2019-12-05T06:37:14.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.584+0000 7f53a27b3700 1 -- 172.21.15.16:0/1339347996 >> 172.21.15.16:0/1339347996 conn(0x7f539c061940 msgr2=0x7f539c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.586+0000 7f53a27b3700 1 Processor -- start 2019-12-05T06:37:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.586+0000 7f53a27b3700 1 -- start start 2019-12-05T06:37:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.587+0000 7f53a27b3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f539c128360 0x7f539c128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.587+0000 7f53a27b3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c137ea0 0x7f539c128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:14.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.587+0000 7f53a27b3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f539c099720 con 0x7f539c152760 2019-12-05T06:37:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.587+0000 7f53a27b3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f539c12a640 con 0x7f539c128360 2019-12-05T06:37:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.587+0000 7f53a27b3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f539c129c10 con 0x7f539c137ea0 2019-12-05T06:37:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.587+0000 7f53a1551700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f539c152760 0x7f539c152bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42504/0 (socket says 172.21.15.16:42504) 2019-12-05T06:37:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.587+0000 7f53a1551700 1 -- 172.21.15.16:0/1679409758 learned_addr learned my addr 172.21.15.16:0/1679409758 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.587+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> v2:172.21.15.76:3300/0 conn(0x7f539c128360 0x7f539c128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:14.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.587+0000 7f539bfff700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c137ea0 0x7f539c128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53997fa700 1 -- 172.21.15.16:0/1679409758 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2630203452 0 0) 0x7f5384001290 con 0x7f539c152760 2019-12-05T06:37:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53997fa700 1 -- 172.21.15.16:0/1679409758 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5380002e80 con 0x7f539c152760 2019-12-05T06:37:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a0d50700 1 -- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c137ea0 msgr2=0x7f539c128c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c137ea0 0x7f539c128c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a0d50700 1 -- 172.21.15.16:0/1679409758 >> v1:172.21.15.16:6789/0 conn(0x7f539c152760 legacy=0x7f539c152bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a0d50700 1 -- 172.21.15.16:0/1679409758 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f539c12acd0 con 0x7f539c128360 2019-12-05T06:37:14.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> v2:172.21.15.76:3300/0 conn(0x7f539c128360 0x7f539c128750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> v2:172.21.15.76:3300/0 conn(0x7f539c128360 0x7f539c128750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 0x7f539c128750 secure :-1 s=READY pgs=756 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 0x7f539c128750 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53997fa700 1 -- 172.21.15.16:0/1679409758 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5394007210 con 0x7f539c128360 2019-12-05T06:37:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a27b3700 1 -- 172.21.15.16:0/1679409758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f539c12acd0 con 0x7f539c128360 2019-12-05T06:37:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 0x7f539c128750 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 0x7f539c128750 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53997fa700 1 -- 172.21.15.16:0/1679409758 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5394007210 con 0x7f539c128360 2019-12-05T06:37:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53997fa700 1 -- 172.21.15.16:0/1679409758 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5394001460 con 0x7f539c128360 2019-12-05T06:37:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.588+0000 7f53a27b3700 1 -- 172.21.15.16:0/1679409758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f539c099720 con 0x7f539c128360 2019-12-05T06:37:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.589+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 0x7f539c128750 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.590+0000 7f53997fa700 1 -- 172.21.15.16:0/1679409758 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5394000c10 con 0x7f539c128360 2019-12-05T06:37:14.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.590+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 0x7f539c128750 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.590+0000 7f53997fa700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5380023670 0x7f5380025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.590+0000 7f53997fa700 1 -- 172.21.15.16:0/1679409758 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f5394045d70 con 0x7f539c128360 2019-12-05T06:37:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.590+0000 7f539bfff700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5380023670 0x7f5380025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.590+0000 7f539bfff700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5380023670 0x7f5380025ae0 crc :-1 s=READY pgs=465 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:14.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.595+0000 7f53a27b3700 1 -- 172.21.15.16:0/1679409758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f539c12acd0 con 0x7f539c128360 2019-12-05T06:37:14.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.602+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 0x7f539c128750 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.602+0000 7f53997fa700 1 -- 172.21.15.16:0/1679409758 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f53940019f0 con 0x7f539c128360 2019-12-05T06:37:14.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.803+0000 7f53a27b3700 1 -- 172.21.15.16:0/1679409758 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f539c12acd0 con 0x7f539c128360 2019-12-05T06:37:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.803+0000 7f53a0d50700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 0x7f539c128750 secure :-1 s=THROTTLE_DONE pgs=756 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.803+0000 7f53997fa700 1 -- 172.21.15.16:0/1679409758 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f5394000c10 con 0x7f539c128360 2019-12-05T06:37:14.804 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 136 2019-12-05T06:37:14.804 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:37:14.805 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:14.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.806+0000 7f53a27b3700 1 -- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5380023670 msgr2=0x7f5380025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.806+0000 7f53a27b3700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5380023670 0x7f5380025ae0 crc :-1 s=READY pgs=465 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:14.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.806+0000 7f53a27b3700 1 -- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 msgr2=0x7f539c128750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.806+0000 7f53a27b3700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 0x7f539c128750 secure :-1 s=READY pgs=756 cs=0 l=1 rx=0x7f53940103a0 tx=0x7f5394010740).stop 2019-12-05T06:37:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.806+0000 7f53a27b3700 1 -- 172.21.15.16:0/1679409758 shutdown_connections 2019-12-05T06:37:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.806+0000 7f53a27b3700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5380023670 0x7f5380025ae0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.806+0000 7f53a27b3700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f539c137ea0 0x7f539c128c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.806+0000 7f53a27b3700 1 --2- 172.21.15.16:0/1679409758 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f539c128360 0x7f539c128750 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:14.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.806+0000 7f53a27b3700 1 -- 172.21.15.16:0/1679409758 shutdown_connections 2019-12-05T06:37:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.807+0000 7f53a27b3700 1 -- 172.21.15.16:0/1679409758 wait complete. 2019-12-05T06:37:14.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:14.807+0000 7f53a27b3700 1 -- 172.21.15.16:0/1679409758 >> 172.21.15.16:0/1679409758 conn(0x7f539c06d260 msgr2=0x7f539c09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:14.819 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5483 2019-12-05T06:37:14.819 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 4 secs before reviving mds.a 2019-12-05T06:37:18.820 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:37:18.820 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:37:18.821 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:37:18.825 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:37:18.826 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:18.931 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:37:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.966+0000 7f1b923a8700 1 Processor -- start 2019-12-05T06:37:18.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.966+0000 7f1b923a8700 1 -- start start 2019-12-05T06:37:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.967+0000 7f1b923a8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1b8c08fef0 0x7f1b8c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.967+0000 7f1b923a8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0958e0 con 0x7f1b8c08d2c0 2019-12-05T06:37:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.967+0000 7f1b923a8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c0629e0 con 0x7f1b8c08c6d0 2019-12-05T06:37:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.967+0000 7f1b923a8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8c08df90 con 0x7f1b8c08fef0 2019-12-05T06:37:18.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.967+0000 7f1b8bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1b8c08d2c0 0x7f1b8c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42540/0 (socket says 172.21.15.16:42540) 2019-12-05T06:37:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.967+0000 7f1b8bfff700 1 -- 172.21.15.16:0/3201379028 learned_addr learned my addr 172.21.15.16:0/3201379028 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.967+0000 7f1b91146700 1 --2- 172.21.15.16:0/3201379028 >> v2:172.21.15.76:3300/0 conn(0x7f1b8c08fef0 0x7f1b8c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.967+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2806343185 0 0) 0x7f1b7c001100 con 0x7f1b8c08d2c0 2019-12-05T06:37:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.967+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b78002e80 con 0x7f1b8c08d2c0 2019-12-05T06:37:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3491601591 0 0) 0x7f1b7c0010e0 con 0x7f1b8c08d2c0 2019-12-05T06:37:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 >> v2:172.21.15.76:3300/0 conn(0x7f1b8c08fef0 msgr2=0x7f1b8c08f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:18.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 --2- 172.21.15.16:0/3201379028 >> v2:172.21.15.76:3300/0 conn(0x7f1b8c08fef0 0x7f1b8c08f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 >> v1:172.21.15.76:6789/0 conn(0x7f1b8c08c6d0 legacy=0x7f1b8c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c08e620 con 0x7f1b8c08d2c0 2019-12-05T06:37:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f1b7c001610 con 0x7f1b8c08d2c0 2019-12-05T06:37:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 >> v1:172.21.15.16:6789/0 conn(0x7f1b8c08d2c0 legacy=0x7f1b8c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b78003a80 0x7f1b78005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:18.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1b78006690 0x7f1b78008b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 0x7f1b7800b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b7800c020 con 0x7f1b780093a0 2019-12-05T06:37:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b78050220 con 0x7f1b78003a80 2019-12-05T06:37:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b78050450 con 0x7f1b78006690 2019-12-05T06:37:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 0x7f1b7800b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:18.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8bfff700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1b78006690 0x7f1b78008b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:18.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b90945700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b78003a80 0x7f1b78005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:18.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 -- 172.21.15.16:0/3201379028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1b78006690 msgr2=0x7f1b78008b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:18.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1b78006690 0x7f1b78008b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:18.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 -- 172.21.15.16:0/3201379028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b78003a80 msgr2=0x7f1b78005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:18.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1b78003a80 0x7f1b78005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:18.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 -- 172.21.15.16:0/3201379028 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b78050700 con 0x7f1b780093a0 2019-12-05T06:37:18.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 0x7f1b7800b870 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1b8400ec20 tx=0x7f1b8400eb50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 -- 172.21.15.16:0/3201379028 reap_dead start 2019-12-05T06:37:18.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 0x7f1b7800b870 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1b8400ec20 tx=0x7f1b8400eb50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 0x7f1b7800b870 secure :-1 s=READY pgs=1094 cs=0 l=1 rx=0x7f1b8400ec20 tx=0x7f1b8400eb50).ready entity=mon.0 client_cookie=44f84fa8808b8d8d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:18.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 0x7f1b7800b870 secure :-1 s=THROTTLE_DONE pgs=1094 cs=0 l=1 rx=0x7f1b8400ec20 tx=0x7f1b8400eb50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b8400f1a0 con 0x7f1b780093a0 2019-12-05T06:37:18.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 0x7f1b7800b870 secure :-1 s=THROTTLE_DONE pgs=1094 cs=0 l=1 rx=0x7f1b8400ec20 tx=0x7f1b8400eb50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b8400edc0 con 0x7f1b780093a0 2019-12-05T06:37:18.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b91146700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 0x7f1b7800b870 secure :-1 s=THROTTLE_DONE pgs=1094 cs=0 l=1 rx=0x7f1b8400ec20 tx=0x7f1b8400eb50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b8b7fe700 1 -- 172.21.15.16:0/3201379028 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b8400f430 con 0x7f1b780093a0 2019-12-05T06:37:18.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b923a8700 1 -- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 msgr2=0x7f1b7800b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.968+0000 7f1b923a8700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 0x7f1b7800b870 secure :-1 s=READY pgs=1094 cs=0 l=1 rx=0x7f1b8400ec20 tx=0x7f1b8400eb50).stop 2019-12-05T06:37:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.969+0000 7f1b923a8700 1 -- 172.21.15.16:0/3201379028 shutdown_connections 2019-12-05T06:37:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.969+0000 7f1b923a8700 1 --2- 172.21.15.16:0/3201379028 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b780093a0 0x7f1b7800b870 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.969+0000 7f1b923a8700 1 -- 172.21.15.16:0/3201379028 shutdown_connections 2019-12-05T06:37:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.969+0000 7f1b923a8700 1 -- 172.21.15.16:0/3201379028 wait complete. 2019-12-05T06:37:18.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.969+0000 7f1b923a8700 1 -- 172.21.15.16:0/3201379028 >> 172.21.15.16:0/3201379028 conn(0x7f1b8c061940 msgr2=0x7f1b8c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:18.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.972+0000 7f1b923a8700 1 Processor -- start 2019-12-05T06:37:18.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.972+0000 7f1b923a8700 1 -- start start 2019-12-05T06:37:18.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b923a8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1b8c1325e0 0x7f1b8c12b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:18.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b923a8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1b8c12c740 0x7f1b8c12bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:18.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b923a8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b8c128b20 con 0x7f1b8c12c1f0 2019-12-05T06:37:18.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b923a8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b8c12a7e0 con 0x7f1b8c1325e0 2019-12-05T06:37:18.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b923a8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8c129db0 con 0x7f1b8c12c740 2019-12-05T06:37:18.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1b8c1325e0 0x7f1b8c12b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:18.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b91146700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1b8c12c1f0 0x7f1b8c06dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42552/0 (socket says 172.21.15.16:42552) 2019-12-05T06:37:18.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1b8c1325e0 0x7f1b8c12b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57380/0 (socket says 172.21.15.16:57380) 2019-12-05T06:37:18.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b91146700 1 -- 172.21.15.16:0/3502814369 learned_addr learned my addr 172.21.15.16:0/3502814369 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:18.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b8bfff700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1b8c12c740 0x7f1b8c12bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:18.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 -- 172.21.15.16:0/3502814369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1b8c12c740 msgr2=0x7f1b8c12bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:18.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1b8c12c740 0x7f1b8c12bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:18.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 -- 172.21.15.16:0/3502814369 >> v1:172.21.15.16:6789/0 conn(0x7f1b8c12c1f0 legacy=0x7f1b8c06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:18.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b897fa700 1 -- 172.21.15.16:0/3502814369 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144921395 0 0) 0x7f1b84001480 con 0x7f1b8c12c1f0 2019-12-05T06:37:18.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 -- 172.21.15.16:0/3502814369 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b8c14a820 con 0x7f1b8c1325e0 2019-12-05T06:37:18.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> v2:172.21.15.16:3300/0 conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> v2:172.21.15.16:3300/0 conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=READY pgs=1096 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:18.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=THROTTLE_DONE pgs=1096 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b897fa700 1 -- 172.21.15.16:0/3502814369 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b74009c60 con 0x7f1b8c1325e0 2019-12-05T06:37:18.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=THROTTLE_DONE pgs=1096 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b897fa700 1 -- 172.21.15.16:0/3502814369 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b74009880 con 0x7f1b8c1325e0 2019-12-05T06:37:18.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=THROTTLE_DONE pgs=1096 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b923a8700 1 -- 172.21.15.16:0/3502814369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b8c12ae70 con 0x7f1b8c1325e0 2019-12-05T06:37:18.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b897fa700 1 -- 172.21.15.16:0/3502814369 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b74009e50 con 0x7f1b8c1325e0 2019-12-05T06:37:18.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.973+0000 7f1b923a8700 1 -- 172.21.15.16:0/3502814369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b8c14a820 con 0x7f1b8c1325e0 2019-12-05T06:37:18.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.974+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=THROTTLE_DONE pgs=1096 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.974+0000 7f1b897fa700 1 -- 172.21.15.16:0/3502814369 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1b74009a70 con 0x7f1b8c1325e0 2019-12-05T06:37:18.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.974+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=THROTTLE_DONE pgs=1096 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.975+0000 7f1b897fa700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1b70023630 0x7f1b70025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:18.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.975+0000 7f1b897fa700 1 -- 172.21.15.16:0/3502814369 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f1b74045d60 con 0x7f1b8c1325e0 2019-12-05T06:37:18.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.975+0000 7f1b8bfff700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1b70023630 0x7f1b70025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:18.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.975+0000 7f1b8bfff700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1b70023630 0x7f1b70025ac0 crc :-1 s=READY pgs=466 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:18.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.981+0000 7f1b923a8700 1 -- 172.21.15.16:0/3502814369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b8c12ae70 con 0x7f1b8c1325e0 2019-12-05T06:37:18.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.985+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=THROTTLE_DONE pgs=1096 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:18.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:18.985+0000 7f1b897fa700 1 -- 172.21.15.16:0/3502814369 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1b74009670 con 0x7f1b8c1325e0 2019-12-05T06:37:19.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.202+0000 7f1b923a8700 1 -- 172.21.15.16:0/3502814369 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b8c14a820 con 0x7f1b8c1325e0 2019-12-05T06:37:19.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.202+0000 7f1b90945700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=THROTTLE_DONE pgs=1096 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:19.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.202+0000 7f1b897fa700 1 -- 172.21.15.16:0/3502814369 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f1b74009a70 con 0x7f1b8c1325e0 2019-12-05T06:37:19.203 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 136 2019-12-05T06:37:19.203 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:37:19.204 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:19.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 -- 172.21.15.16:0/3502814369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1b70023630 msgr2=0x7f1b70025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:19.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1b70023630 0x7f1b70025ac0 crc :-1 s=READY pgs=466 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:19.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 -- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 msgr2=0x7f1b8c12b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:19.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 0x7f1b8c12b770 secure :-1 s=READY pgs=1096 cs=0 l=1 rx=0x7f1b740103a0 tx=0x7f1b7400b2e0).stop 2019-12-05T06:37:19.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 -- 172.21.15.16:0/3502814369 shutdown_connections 2019-12-05T06:37:19.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1b8c12c740 0x7f1b8c12bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:19.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1b70023630 0x7f1b70025ac0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:19.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 --2- 172.21.15.16:0/3502814369 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1b8c1325e0 0x7f1b8c12b770 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:19.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 -- 172.21.15.16:0/3502814369 shutdown_connections 2019-12-05T06:37:19.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 -- 172.21.15.16:0/3502814369 wait complete. 2019-12-05T06:37:19.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:19.205+0000 7f1b923a8700 1 -- 172.21.15.16:0/3502814369 >> 172.21.15.16:0/3502814369 conn(0x7f1b8c0a2850 msgr2=0x7f1b8c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:19.216 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:37:21.217 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:21.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.509+0000 7f7eeb22a700 1 Processor -- start 2019-12-05T06:37:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.509+0000 7f7eeb22a700 1 -- start start 2019-12-05T06:37:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.511+0000 7f7eeb22a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 0x7f7ee408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.511+0000 7f7eeb22a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7ee408fef0 0x7f7ee408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.511+0000 7f7eeb22a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ee40958e0 con 0x7f7ee408ef20 2019-12-05T06:37:21.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.511+0000 7f7eeb22a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee40a3f70 con 0x7f7ee408fef0 2019-12-05T06:37:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.511+0000 7f7eeb22a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee408d830 con 0x7f7ee408c6d0 2019-12-05T06:37:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.512+0000 7f7ee9fc8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7ee408ef20 0x7f7ee408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42580/0 (socket says 172.21.15.16:42580) 2019-12-05T06:37:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.512+0000 7f7ee9fc8700 1 -- 172.21.15.16:0/473702392 learned_addr learned my addr 172.21.15.16:0/473702392 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.512+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/473702392 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 0x7f7ee408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.512+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/473702392 >> v2:172.21.15.76:3300/0 conn(0x7f7ee408fef0 0x7f7ee408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:21.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.512+0000 7f7ee3fff700 1 -- 172.21.15.16:0/473702392 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1037036923 0 0) 0x7f7ed4001150 con 0x7f7ee408ef20 2019-12-05T06:37:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.512+0000 7f7ee3fff700 1 -- 172.21.15.16:0/473702392 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ecc002e80 con 0x7f7ee408ef20 2019-12-05T06:37:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.512+0000 7f7ee97c7700 1 -- 172.21.15.16:0/473702392 >> v2:172.21.15.76:3300/0 conn(0x7f7ee408fef0 msgr2=0x7f7ee408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.512+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/473702392 >> v2:172.21.15.76:3300/0 conn(0x7f7ee408fef0 0x7f7ee408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:21.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.512+0000 7f7ee97c7700 1 -- 172.21.15.16:0/473702392 >> v1:172.21.15.16:6789/0 conn(0x7f7ee408ef20 legacy=0x7f7ee408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.513+0000 7f7ee97c7700 1 -- 172.21.15.16:0/473702392 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee408d540 con 0x7f7ee408c6d0 2019-12-05T06:37:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.513+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/473702392 >> v2:172.21.15.76:3300/0 conn(0x7f7ee408fef0 0x7f7ee408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.513+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/473702392 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 0x7f7ee408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7edc00b2d0 tx=0x7f7edc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.513+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/473702392 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 0x7f7ee408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7edc00b2d0 tx=0x7f7edc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.513+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/473702392 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 0x7f7ee408cac0 secure :-1 s=READY pgs=254 cs=0 l=1 rx=0x7f7edc00b2d0 tx=0x7f7edc00bae0).ready entity=mon.2 client_cookie=69601482a046698e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.513+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/473702392 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 0x7f7ee408cac0 secure :-1 s=THROTTLE_DONE pgs=254 cs=0 l=1 rx=0x7f7edc00b2d0 tx=0x7f7edc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.513+0000 7f7ee3fff700 1 -- 172.21.15.16:0/473702392 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7edc007590 con 0x7f7ee408c6d0 2019-12-05T06:37:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.513+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/473702392 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 0x7f7ee408cac0 secure :-1 s=THROTTLE_DONE pgs=254 cs=0 l=1 rx=0x7f7edc00b2d0 tx=0x7f7edc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.513+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/473702392 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 0x7f7ee408cac0 secure :-1 s=THROTTLE_DONE pgs=254 cs=0 l=1 rx=0x7f7edc00b2d0 tx=0x7f7edc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.514+0000 7f7ee3fff700 1 -- 172.21.15.16:0/473702392 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7edc009d30 con 0x7f7ee408c6d0 2019-12-05T06:37:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.514+0000 7f7ee3fff700 1 -- 172.21.15.16:0/473702392 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7edc00a400 con 0x7f7ee408c6d0 2019-12-05T06:37:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.514+0000 7f7eeb22a700 1 -- 172.21.15.16:0/473702392 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 msgr2=0x7f7ee408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.514+0000 7f7eeb22a700 1 --2- 172.21.15.16:0/473702392 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 0x7f7ee408cac0 secure :-1 s=READY pgs=254 cs=0 l=1 rx=0x7f7edc00b2d0 tx=0x7f7edc00bae0).stop 2019-12-05T06:37:21.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.515+0000 7f7eeb22a700 1 -- 172.21.15.16:0/473702392 shutdown_connections 2019-12-05T06:37:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.515+0000 7f7eeb22a700 1 --2- 172.21.15.16:0/473702392 >> v2:172.21.15.76:3300/0 conn(0x7f7ee408fef0 0x7f7ee408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.516+0000 7f7eeb22a700 1 --2- 172.21.15.16:0/473702392 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee408c6d0 0x7f7ee408cac0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.516+0000 7f7eeb22a700 1 -- 172.21.15.16:0/473702392 shutdown_connections 2019-12-05T06:37:21.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.516+0000 7f7eeb22a700 1 -- 172.21.15.16:0/473702392 wait complete. 2019-12-05T06:37:21.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.516+0000 7f7eeb22a700 1 -- 172.21.15.16:0/473702392 >> 172.21.15.16:0/473702392 conn(0x7f7ee4061940 msgr2=0x7f7ee4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.521+0000 7f7eeb22a700 1 Processor -- start 2019-12-05T06:37:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.521+0000 7f7eeb22a700 1 -- start start 2019-12-05T06:37:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.523+0000 7f7eeb22a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7ee41302f0 0x7f7ee412a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.523+0000 7f7eeb22a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ee412b640 0x7f7ee412abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:21.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.523+0000 7f7eeb22a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee412b0f0 0x7f7ee41306e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.523+0000 7f7eeb22a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee41296e0 con 0x7f7ee412b640 2019-12-05T06:37:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.523+0000 7f7eeb22a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ee4128cb0 con 0x7f7ee41302f0 2019-12-05T06:37:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.523+0000 7f7eeb22a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ee4128f80 con 0x7f7ee412b0f0 2019-12-05T06:37:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee8fc6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ee412b640 0x7f7ee412abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:21.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee8fc6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7ee412b640 0x7f7ee412abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57416/0 (socket says 172.21.15.16:57416) 2019-12-05T06:37:21.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee8fc6700 1 -- 172.21.15.16:0/472573938 learned_addr learned my addr 172.21.15.16:0/472573938 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:37:21.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/472573938 >> v2:172.21.15.76:3300/0 conn(0x7f7ee41302f0 0x7f7ee412a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:21.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee9fc8700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee412b0f0 0x7f7ee41306e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:21.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee8fc6700 1 -- 172.21.15.16:0/472573938 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee412b0f0 msgr2=0x7f7ee41306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee412b0f0 0x7f7ee41306e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee8fc6700 1 -- 172.21.15.16:0/472573938 >> v2:172.21.15.76:3300/0 conn(0x7f7ee41302f0 msgr2=0x7f7ee412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> v2:172.21.15.76:3300/0 conn(0x7f7ee41302f0 0x7f7ee412a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee8fc6700 1 -- 172.21.15.16:0/472573938 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ee4129d70 con 0x7f7ee412b640 2019-12-05T06:37:21.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.524+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/472573938 >> v2:172.21.15.76:3300/0 conn(0x7f7ee41302f0 0x7f7ee412a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> v2:172.21.15.16:3300/0 conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> v2:172.21.15.16:3300/0 conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=READY pgs=1098 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7ee1ffb700 1 -- 172.21.15.16:0/472573938 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ed800a8c0 con 0x7f7ee412b640 2019-12-05T06:37:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7ee1ffb700 1 -- 172.21.15.16:0/472573938 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ed800db40 con 0x7f7ee412b640 2019-12-05T06:37:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7eeb22a700 1 -- 172.21.15.16:0/472573938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ee4129d70 con 0x7f7ee412b640 2019-12-05T06:37:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7ee1ffb700 1 -- 172.21.15.16:0/472573938 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ed800e190 con 0x7f7ee412b640 2019-12-05T06:37:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.525+0000 7f7eeb22a700 1 -- 172.21.15.16:0/472573938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ee4129d70 con 0x7f7ee412b640 2019-12-05T06:37:21.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.526+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.528+0000 7f7ee1ffb700 1 -- 172.21.15.16:0/472573938 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7ed8009580 con 0x7f7ee412b640 2019-12-05T06:37:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.528+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.528+0000 7f7ee1ffb700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7ec4023630 0x7f7ec4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.528+0000 7f7ee1ffb700 1 -- 172.21.15.16:0/472573938 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f7ed8045d50 con 0x7f7ee412b640 2019-12-05T06:37:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.528+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7ec4023630 0x7f7ec4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:21.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.529+0000 7f7ee97c7700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7ec4023630 0x7f7ec4025ac0 crc :-1 s=READY pgs=467 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:21.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.541+0000 7f7eeb22a700 1 -- 172.21.15.16:0/472573938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ee4129d70 con 0x7f7ee412b640 2019-12-05T06:37:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.550+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.551+0000 7f7ee1ffb700 1 -- 172.21.15.16:0/472573938 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7ed800a590 con 0x7f7ee412b640 2019-12-05T06:37:21.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.813+0000 7f7eeb22a700 1 -- 172.21.15.16:0/472573938 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ee4129d70 con 0x7f7ee412b640 2019-12-05T06:37:21.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.814+0000 7f7ee8fc6700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.814+0000 7f7ee1ffb700 1 -- 172.21.15.16:0/472573938 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 136 v136) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f7ed8009580 con 0x7f7ee412b640 2019-12-05T06:37:21.815 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 136 2019-12-05T06:37:21.815 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:37:21.815 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":136,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:21.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 -- 172.21.15.16:0/472573938 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7ec4023630 msgr2=0x7f7ec4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:21.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7ec4023630 0x7f7ec4025ac0 crc :-1 s=READY pgs=467 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:21.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 -- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 msgr2=0x7f7ee412abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:21.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 0x7f7ee412abb0 secure :-1 s=READY pgs=1098 cs=0 l=1 rx=0x7f7ed8010390 tx=0x7f7ed8010730).stop 2019-12-05T06:37:21.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 -- 172.21.15.16:0/472573938 shutdown_connections 2019-12-05T06:37:21.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ee412b0f0 0x7f7ee41306e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:21.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7ec4023630 0x7f7ec4025ac0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:21.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 --2- 172.21.15.16:0/472573938 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7ee412b640 0x7f7ee412abb0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:21.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 --2- 172.21.15.16:0/472573938 >> v2:172.21.15.76:3300/0 conn(0x7f7ee41302f0 0x7f7ee412a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:21.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 -- 172.21.15.16:0/472573938 shutdown_connections 2019-12-05T06:37:21.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 -- 172.21.15.16:0/472573938 wait complete. 2019-12-05T06:37:21.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:21.817+0000 7f7eeb22a700 1 -- 172.21.15.16:0/472573938 >> 172.21.15.16:0/472573938 conn(0x7f7ee40a2850 msgr2=0x7f7ee406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:21.828 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:37:23.829 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:37:23.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.985+0000 7f20a85e6700 1 Processor -- start 2019-12-05T06:37:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.985+0000 7f20a85e6700 1 -- start start 2019-12-05T06:37:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a85e6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f20a008c6d0 0x7f20a008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a85e6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20a008fef0 0x7f20a008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:37:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a85e6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a008fb80 con 0x7f20a008ef20 2019-12-05T06:37:24.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a85e6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a008df90 con 0x7f20a008fef0 2019-12-05T06:37:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a85e6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a00a3f70 con 0x7f20a008c6d0 2019-12-05T06:37:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6382700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20a008fef0 0x7f20a008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6b83700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f20a008c6d0 0x7f20a008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6382700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20a008fef0 0x7f20a008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57446/0 (socket says 172.21.15.16:57446) 2019-12-05T06:37:24.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6382700 1 -- 172.21.15.16:0/2655537084 learned_addr learned my addr 172.21.15.16:0/2655537084 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:37:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a7384700 1 --1- 172.21.15.16:0/2655537084 >> v1:172.21.15.76:6789/0 conn(0x7f20a008ef20 0x7f20a008f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:35686/0 (socket says 172.21.15.16:35686) 2019-12-05T06:37:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6382700 1 -- 172.21.15.16:0/2655537084 >> v2:172.21.15.76:3300/0 conn(0x7f20a008c6d0 msgr2=0x7f20a008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:37:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6382700 1 --2- 172.21.15.16:0/2655537084 >> v2:172.21.15.76:3300/0 conn(0x7f20a008c6d0 0x7f20a008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6382700 1 -- 172.21.15.16:0/2655537084 >> v1:172.21.15.76:6789/0 conn(0x7f20a008ef20 legacy=0x7f20a008f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6382700 1 -- 172.21.15.16:0/2655537084 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a008d540 con 0x7f20a008fef0 2019-12-05T06:37:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6b83700 1 --2- 172.21.15.16:0/2655537084 >> v2:172.21.15.76:3300/0 conn(0x7f20a008c6d0 0x7f20a008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:37:24.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6382700 1 --2- 172.21.15.16:0/2655537084 >> v2:172.21.15.16:3300/0 conn(0x7f20a008fef0 0x7f20a008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f209000f740 tx=0x7f209000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6382700 1 --2- 172.21.15.16:0/2655537084 >> v2:172.21.15.16:3300/0 conn(0x7f20a008fef0 0x7f20a008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f209000f740 tx=0x7f209000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.987+0000 7f20a6382700 1 --2- 172.21.15.16:0/2655537084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a008fef0 0x7f20a008d000 secure :-1 s=READY pgs=1099 cs=0 l=1 rx=0x7f209000f740 tx=0x7f209000fae0).ready entity=mon.0 client_cookie=bbf91f5c5112343e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.988+0000 7f20a6382700 1 --2- 172.21.15.16:0/2655537084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a008fef0 0x7f20a008d000 secure :-1 s=THROTTLE_DONE pgs=1099 cs=0 l=1 rx=0x7f209000f740 tx=0x7f209000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.988+0000 7f20a5b81700 1 -- 172.21.15.16:0/2655537084 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2090000960 con 0x7f20a008fef0 2019-12-05T06:37:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.988+0000 7f20a6382700 1 --2- 172.21.15.16:0/2655537084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a008fef0 0x7f20a008d000 secure :-1 s=THROTTLE_DONE pgs=1099 cs=0 l=1 rx=0x7f209000f740 tx=0x7f209000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.988+0000 7f20a6382700 1 --2- 172.21.15.16:0/2655537084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a008fef0 0x7f20a008d000 secure :-1 s=THROTTLE_DONE pgs=1099 cs=0 l=1 rx=0x7f209000f740 tx=0x7f209000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.988+0000 7f20a5b81700 1 -- 172.21.15.16:0/2655537084 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2090000c90 con 0x7f20a008fef0 2019-12-05T06:37:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.988+0000 7f20a5b81700 1 -- 172.21.15.16:0/2655537084 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20900013a0 con 0x7f20a008fef0 2019-12-05T06:37:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.988+0000 7f20a85e6700 1 -- 172.21.15.16:0/2655537084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a008fef0 msgr2=0x7f20a008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:24.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.988+0000 7f20a85e6700 1 --2- 172.21.15.16:0/2655537084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a008fef0 0x7f20a008d000 secure :-1 s=READY pgs=1099 cs=0 l=1 rx=0x7f209000f740 tx=0x7f209000fae0).stop 2019-12-05T06:37:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.989+0000 7f20a85e6700 1 -- 172.21.15.16:0/2655537084 shutdown_connections 2019-12-05T06:37:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.989+0000 7f20a85e6700 1 --2- 172.21.15.16:0/2655537084 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a008fef0 0x7f20a008d000 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.989+0000 7f20a85e6700 1 --2- 172.21.15.16:0/2655537084 >> v2:172.21.15.76:3300/0 conn(0x7f20a008c6d0 0x7f20a008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:37:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.989+0000 7f20a85e6700 1 -- 172.21.15.16:0/2655537084 shutdown_connections 2019-12-05T06:37:24.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.989+0000 7f20a85e6700 1 -- 172.21.15.16:0/2655537084 wait complete. 2019-12-05T06:37:24.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.989+0000 7f20a85e6700 1 -- 172.21.15.16:0/2655537084 >> 172.21.15.16:0/2655537084 conn(0x7f20a0061940 msgr2=0x7f20a0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:24.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.992+0000 7f20a85e6700 1 Processor -- start 2019-12-05T06:37:24.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.992+0000 7f20a85e6700 1 -- start start 2019-12-05T06:37:24.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a85e6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20a009bbe0 0x7f20a01539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:24.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a85e6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f20a0084f30 0x7f20a0155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:24.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a85e6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20a0084b90 con 0x7f20a0096e90 2019-12-05T06:37:24.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a85e6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20a009a450 con 0x7f20a009bbe0 2019-12-05T06:37:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a85e6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20a0099a20 con 0x7f20a0084f30 2019-12-05T06:37:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a6382700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f20a009bbe0 0x7f20a01539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a6b83700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f20a0096e90 0x7f20a009b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:42618/0 (socket says 172.21.15.16:42618) 2019-12-05T06:37:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a6b83700 1 -- 172.21.15.16:0/3532654573 learned_addr learned my addr 172.21.15.16:0/3532654573 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:37:24.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a7384700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f20a0084f30 0x7f20a0155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a6382700 1 -- 172.21.15.16:0/3532654573 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f20a0084f30 msgr2=0x7f20a0155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f20a0084f30 0x7f20a0155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a6382700 1 -- 172.21.15.16:0/3532654573 >> v1:172.21.15.16:6789/0 conn(0x7f20a0096e90 legacy=0x7f20a009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a6382700 1 -- 172.21.15.16:0/3532654573 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20a009aae0 con 0x7f20a009bbe0 2019-12-05T06:37:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20977fe700 1 -- 172.21.15.16:0/3532654573 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269506091 0 0) 0x7f209c00dd90 con 0x7f20a0096e90 2019-12-05T06:37:24.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.993+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> v2:172.21.15.16:3300/0 conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> v2:172.21.15.16:3300/0 conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=READY pgs=1101 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:24.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20977fe700 1 -- 172.21.15.16:0/3532654573 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2090000f50 con 0x7f20a009bbe0 2019-12-05T06:37:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20977fe700 1 -- 172.21.15.16:0/3532654573 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2090000940 con 0x7f20a009bbe0 2019-12-05T06:37:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20a85e6700 1 -- 172.21.15.16:0/3532654573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20a009aae0 con 0x7f20a009bbe0 2019-12-05T06:37:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20977fe700 1 -- 172.21.15.16:0/3532654573 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2090001530 con 0x7f20a009bbe0 2019-12-05T06:37:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20a85e6700 1 -- 172.21.15.16:0/3532654573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20a009aae0 con 0x7f20a009bbe0 2019-12-05T06:37:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.994+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.995+0000 7f20977fe700 1 -- 172.21.15.16:0/3532654573 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2090000d60 con 0x7f20a009bbe0 2019-12-05T06:37:24.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.995+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.995+0000 7f20977fe700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2088023630 0x7f2088025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:37:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.995+0000 7f20977fe700 1 -- 172.21.15.16:0/3532654573 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f2090045d20 con 0x7f20a009bbe0 2019-12-05T06:37:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.995+0000 7f20a6b83700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2088023630 0x7f2088025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:37:24.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:23.996+0000 7f20a6b83700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2088023630 0x7f2088025ac0 crc :-1 s=READY pgs=469 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:37:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.001+0000 7f20a85e6700 1 -- 172.21.15.16:0/3532654573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20a009aae0 con 0x7f20a009bbe0 2019-12-05T06:37:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.005+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.005+0000 7f20977fe700 1 -- 172.21.15.16:0/3532654573 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f209002add0 con 0x7f20a009bbe0 2019-12-05T06:37:24.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.207+0000 7f20a85e6700 1 -- 172.21.15.16:0/3532654573 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20a009aae0 con 0x7f20a009bbe0 2019-12-05T06:37:24.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.207+0000 7f20a6382700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=THROTTLE_DONE pgs=1101 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:37:24.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.207+0000 7f20977fe700 1 -- 172.21.15.16:0/3532654573 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f20900013b0 con 0x7f20a009bbe0 2019-12-05T06:37:24.209 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2019-12-05T06:37:24.209 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:37:24.209 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:37:24.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 -- 172.21.15.16:0/3532654573 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2088023630 msgr2=0x7f2088025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:24.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2088023630 0x7f2088025ac0 crc :-1 s=READY pgs=469 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:24.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 -- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 msgr2=0x7f20a01539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:37:24.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 0x7f20a01539e0 secure :-1 s=READY pgs=1101 cs=0 l=1 rx=0x7f2090010390 tx=0x7f2090010210).stop 2019-12-05T06:37:24.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 -- 172.21.15.16:0/3532654573 shutdown_connections 2019-12-05T06:37:24.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2088023630 0x7f2088025ac0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:24.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f20a009bbe0 0x7f20a01539e0 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:24.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 --2- 172.21.15.16:0/3532654573 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f20a0084f30 0x7f20a0155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:37:24.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 -- 172.21.15.16:0/3532654573 shutdown_connections 2019-12-05T06:37:24.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 -- 172.21.15.16:0/3532654573 wait complete. 2019-12-05T06:37:24.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:37:24.210+0000 7f20a85e6700 1 -- 172.21.15.16:0/3532654573 >> 172.21.15.16:0/3532654573 conn(0x7f20a00714f0 msgr2=0x7f20a009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:37:24.222 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:37:24.222 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 99 secs before thrashing 2019-12-05T06:37:39.849 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:37:39.854 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:38:10.030 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:38:10.035 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:38:40.166 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:38:40.171 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:39:03.224 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:03.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.363+0000 7f093c46f700 1 Processor -- start 2019-12-05T06:39:03.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.363+0000 7f093c46f700 1 -- start start 2019-12-05T06:39:03.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.364+0000 7f093c46f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f093408c6d0 0x7f093408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.364+0000 7f093c46f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093408fef0 0x7f093408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.364+0000 7f093c46f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09340958e0 con 0x7f093408ef20 2019-12-05T06:39:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.364+0000 7f093c46f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093408df90 con 0x7f093408c6d0 2019-12-05T06:39:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.364+0000 7f093c46f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09340a3f70 con 0x7f093408fef0 2019-12-05T06:39:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093b20d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f093408ef20 0x7f093408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43068/0 (socket says 172.21.15.16:43068) 2019-12-05T06:39:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093b20d700 1 -- 172.21.15.16:0/3591922063 learned_addr learned my addr 172.21.15.16:0/3591922063 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:03.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093aa0c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f093408c6d0 0x7f093408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093a20b700 1 --2- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093408fef0 0x7f093408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093aa0c700 1 -- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093408fef0 msgr2=0x7f093408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093aa0c700 1 --2- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093408fef0 0x7f093408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093aa0c700 1 -- 172.21.15.16:0/3591922063 >> v1:172.21.15.16:6789/0 conn(0x7f093408ef20 legacy=0x7f093408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093aa0c700 1 -- 172.21.15.16:0/3591922063 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093408d540 con 0x7f093408c6d0 2019-12-05T06:39:03.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093a20b700 1 --2- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093408fef0 0x7f093408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:03.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093aa0c700 1 --2- 172.21.15.16:0/3591922063 >> v2:172.21.15.76:3300/0 conn(0x7f093408c6d0 0x7f093408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f092400b2d0 tx=0x7f092400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093aa0c700 1 --2- 172.21.15.16:0/3591922063 >> v2:172.21.15.76:3300/0 conn(0x7f093408c6d0 0x7f093408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f092400b2d0 tx=0x7f092400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.365+0000 7f093aa0c700 1 --2- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f093408c6d0 0x7f093408cac0 secure :-1 s=READY pgs=760 cs=0 l=1 rx=0x7f092400b2d0 tx=0x7f092400bae0).ready entity=mon.1 client_cookie=509fac310348c30 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.368+0000 7f093aa0c700 1 --2- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f093408c6d0 0x7f093408cac0 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f092400b2d0 tx=0x7f092400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.368+0000 7f0939a0a700 1 -- 172.21.15.16:0/3591922063 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0924007590 con 0x7f093408c6d0 2019-12-05T06:39:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.369+0000 7f093aa0c700 1 --2- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f093408c6d0 0x7f093408cac0 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f092400b2d0 tx=0x7f092400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.369+0000 7f0939a0a700 1 -- 172.21.15.16:0/3591922063 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0924000cc0 con 0x7f093408c6d0 2019-12-05T06:39:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.369+0000 7f093aa0c700 1 --2- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f093408c6d0 0x7f093408cac0 secure :-1 s=THROTTLE_DONE pgs=760 cs=0 l=1 rx=0x7f092400b2d0 tx=0x7f092400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.369+0000 7f0939a0a700 1 -- 172.21.15.16:0/3591922063 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0924009f20 con 0x7f093408c6d0 2019-12-05T06:39:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.369+0000 7f093c46f700 1 -- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f093408c6d0 msgr2=0x7f093408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.369+0000 7f093c46f700 1 --2- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f093408c6d0 0x7f093408cac0 secure :-1 s=READY pgs=760 cs=0 l=1 rx=0x7f092400b2d0 tx=0x7f092400bae0).stop 2019-12-05T06:39:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.369+0000 7f093c46f700 1 -- 172.21.15.16:0/3591922063 shutdown_connections 2019-12-05T06:39:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.369+0000 7f093c46f700 1 --2- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093408fef0 0x7f093408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:03.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.369+0000 7f093c46f700 1 --2- 172.21.15.16:0/3591922063 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f093408c6d0 0x7f093408cac0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.370+0000 7f093c46f700 1 -- 172.21.15.16:0/3591922063 shutdown_connections 2019-12-05T06:39:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.370+0000 7f093c46f700 1 -- 172.21.15.16:0/3591922063 wait complete. 2019-12-05T06:39:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.370+0000 7f093c46f700 1 -- 172.21.15.16:0/3591922063 >> 172.21.15.16:0/3591922063 conn(0x7f0934061940 msgr2=0x7f0934096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:03.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.372+0000 7f093c46f700 1 Processor -- start 2019-12-05T06:39:03.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.372+0000 7f093c46f700 1 -- start start 2019-12-05T06:39:03.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093c46f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:03.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093c46f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0934154c60 0x7f0934157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:03.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093c46f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0934128b20 con 0x7f09341325e0 2019-12-05T06:39:03.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093c46f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f093412a7e0 con 0x7f0934154c60 2019-12-05T06:39:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093c46f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0934129db0 con 0x7f093412c740 2019-12-05T06:39:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093aa0c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f09341325e0 0x7f093412b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43070/0 (socket says 172.21.15.16:43070) 2019-12-05T06:39:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093aa0c700 1 -- 172.21.15.16:0/3485104588 learned_addr learned my addr 172.21.15.16:0/3485104588 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093b20d700 1 --2- 172.21.15.16:0/3485104588 >> v2:172.21.15.76:3300/0 conn(0x7f0934154c60 0x7f0934157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f09337fe700 1 -- 172.21.15.16:0/3485104588 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3209753612 0 0) 0x7f092400b750 con 0x7f09341325e0 2019-12-05T06:39:03.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f09337fe700 1 -- 172.21.15.16:0/3485104588 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0914002e80 con 0x7f09341325e0 2019-12-05T06:39:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093a20b700 1 -- 172.21.15.16:0/3485104588 >> v2:172.21.15.76:3300/0 conn(0x7f0934154c60 msgr2=0x7f0934157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> v2:172.21.15.76:3300/0 conn(0x7f0934154c60 0x7f0934157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093a20b700 1 -- 172.21.15.16:0/3485104588 >> v1:172.21.15.16:6789/0 conn(0x7f09341325e0 legacy=0x7f093412b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093a20b700 1 -- 172.21.15.16:0/3485104588 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f093412ae70 con 0x7f093412c740 2019-12-05T06:39:03.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.373+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=READY pgs=255 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f09337fe700 1 -- 172.21.15.16:0/3485104588 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0928000940 con 0x7f093412c740 2019-12-05T06:39:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f093c46f700 1 -- 172.21.15.16:0/3485104588 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f093412ae70 con 0x7f093412c740 2019-12-05T06:39:03.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f09337fe700 1 -- 172.21.15.16:0/3485104588 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0928000940 con 0x7f093412c740 2019-12-05T06:39:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f09337fe700 1 -- 172.21.15.16:0/3485104588 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f092800e1e0 con 0x7f093412c740 2019-12-05T06:39:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f093c46f700 1 -- 172.21.15.16:0/3485104588 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f093412ae70 con 0x7f093412c740 2019-12-05T06:39:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.374+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.375+0000 7f09337fe700 1 -- 172.21.15.16:0/3485104588 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f092800dba0 con 0x7f093412c740 2019-12-05T06:39:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.375+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.375+0000 7f09337fe700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0914023670 0x7f0914025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:03.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.375+0000 7f09337fe700 1 -- 172.21.15.16:0/3485104588 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f0928045d10 con 0x7f093412c740 2019-12-05T06:39:03.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.375+0000 7f093aa0c700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0914023670 0x7f0914025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.376+0000 7f093aa0c700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0914023670 0x7f0914025ae0 crc :-1 s=READY pgs=474 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:03.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.378+0000 7f093c46f700 1 -- 172.21.15.16:0/3485104588 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f093412ae70 con 0x7f093412c740 2019-12-05T06:39:03.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.385+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.385+0000 7f09337fe700 1 -- 172.21.15.16:0/3485104588 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f092800e770 con 0x7f093412c740 2019-12-05T06:39:03.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.571+0000 7f093c46f700 1 -- 172.21.15.16:0/3485104588 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f093412ae70 con 0x7f093412c740 2019-12-05T06:39:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.573+0000 7f093a20b700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=THROTTLE_DONE pgs=255 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.573+0000 7f09337fe700 1 -- 172.21.15.16:0/3485104588 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f092800e3b0 con 0x7f093412c740 2019-12-05T06:39:03.575 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:03.576 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:03.576 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2019-12-05T06:39:03.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 -- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0914023670 msgr2=0x7f0914025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0914023670 0x7f0914025ae0 crc :-1 s=READY pgs=474 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 -- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 msgr2=0x7f093412bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 secure :-1 s=READY pgs=255 cs=0 l=1 rx=0x7f09280103a0 tx=0x7f09280101d0).stop 2019-12-05T06:39:03.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 -- 172.21.15.16:0/3485104588 shutdown_connections 2019-12-05T06:39:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0914023670 0x7f0914025ae0 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 --2- 172.21.15.16:0/3485104588 >> v2:172.21.15.76:3300/0 conn(0x7f0934154c60 0x7f0934157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 --2- 172.21.15.16:0/3485104588 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f093412c740 0x7f093412bf70 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:03.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 -- 172.21.15.16:0/3485104588 shutdown_connections 2019-12-05T06:39:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 -- 172.21.15.16:0/3485104588 wait complete. 2019-12-05T06:39:03.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.577+0000 7f093c46f700 1 -- 172.21.15.16:0/3485104588 >> 172.21.15.16:0/3485104588 conn(0x7f09340a2850 msgr2=0x7f093406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:03.587 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:39:03.588 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:39:03.588 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:39:03.604 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:39:03.604 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.818+0000 7f98d801d700 1 Processor -- start 2019-12-05T06:39:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.818+0000 7f98d801d700 1 -- start start 2019-12-05T06:39:03.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.819+0000 7f98d801d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f98d008d2c0 0x7f98d008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d801d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98d008fef0 0x7f98d008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d801d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d00958e0 con 0x7f98d008c6d0 2019-12-05T06:39:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d801d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d008df90 con 0x7f98d008d2c0 2019-12-05T06:39:03.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d801d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d00a3f70 con 0x7f98d008fef0 2019-12-05T06:39:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d5db9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f98d008d2c0 0x7f98d008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d5db9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f98d008d2c0 0x7f98d008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57916/0 (socket says 172.21.15.16:57916) 2019-12-05T06:39:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d65ba700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f98d008c6d0 0x7f98d008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43082/0 (socket says 172.21.15.16:43082) 2019-12-05T06:39:03.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d5db9700 1 -- 172.21.15.16:0/182029165 learned_addr learned my addr 172.21.15.16:0/182029165 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d6dbb700 1 --2- 172.21.15.16:0/182029165 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98d008fef0 0x7f98d008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d5db9700 1 -- 172.21.15.16:0/182029165 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98d008fef0 msgr2=0x7f98d008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d5db9700 1 --2- 172.21.15.16:0/182029165 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98d008fef0 0x7f98d008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:03.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d5db9700 1 -- 172.21.15.16:0/182029165 >> v1:172.21.15.16:6789/0 conn(0x7f98d008c6d0 legacy=0x7f98d008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d5db9700 1 -- 172.21.15.16:0/182029165 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d008e620 con 0x7f98d008d2c0 2019-12-05T06:39:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d6dbb700 1 --2- 172.21.15.16:0/182029165 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98d008fef0 0x7f98d008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:39:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d55b8700 1 -- 172.21.15.16:0/182029165 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889398216 0 0) 0x7f98cc001100 con 0x7f98d008c6d0 2019-12-05T06:39:03.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.820+0000 7f98d5db9700 1 --2- 172.21.15.16:0/182029165 >> v2:172.21.15.16:3300/0 conn(0x7f98d008d2c0 0x7f98d008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f98c000f740 tx=0x7f98c000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.821+0000 7f98d5db9700 1 --2- 172.21.15.16:0/182029165 >> v2:172.21.15.16:3300/0 conn(0x7f98d008d2c0 0x7f98d008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f98c000f740 tx=0x7f98c000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.821+0000 7f98d5db9700 1 --2- 172.21.15.16:0/182029165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98d008d2c0 0x7f98d008ef20 secure :-1 s=READY pgs=1108 cs=0 l=1 rx=0x7f98c000f740 tx=0x7f98c000fae0).ready entity=mon.0 client_cookie=e658fddf1fa63de0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.821+0000 7f98d5db9700 1 --2- 172.21.15.16:0/182029165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98d008d2c0 0x7f98d008ef20 secure :-1 s=THROTTLE_DONE pgs=1108 cs=0 l=1 rx=0x7f98c000f740 tx=0x7f98c000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.821+0000 7f98d55b8700 1 -- 172.21.15.16:0/182029165 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98c0000960 con 0x7f98d008d2c0 2019-12-05T06:39:03.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.821+0000 7f98d5db9700 1 --2- 172.21.15.16:0/182029165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98d008d2c0 0x7f98d008ef20 secure :-1 s=THROTTLE_DONE pgs=1108 cs=0 l=1 rx=0x7f98c000f740 tx=0x7f98c000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.821+0000 7f98d5db9700 1 --2- 172.21.15.16:0/182029165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98d008d2c0 0x7f98d008ef20 secure :-1 s=THROTTLE_DONE pgs=1108 cs=0 l=1 rx=0x7f98c000f740 tx=0x7f98c000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.821+0000 7f98d55b8700 1 -- 172.21.15.16:0/182029165 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c0000c90 con 0x7f98d008d2c0 2019-12-05T06:39:03.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.821+0000 7f98d55b8700 1 -- 172.21.15.16:0/182029165 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98c00013a0 con 0x7f98d008d2c0 2019-12-05T06:39:03.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.821+0000 7f98d801d700 1 -- 172.21.15.16:0/182029165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98d008d2c0 msgr2=0x7f98d008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.821+0000 7f98d801d700 1 --2- 172.21.15.16:0/182029165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98d008d2c0 0x7f98d008ef20 secure :-1 s=READY pgs=1108 cs=0 l=1 rx=0x7f98c000f740 tx=0x7f98c000fae0).stop 2019-12-05T06:39:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.822+0000 7f98d801d700 1 -- 172.21.15.16:0/182029165 shutdown_connections 2019-12-05T06:39:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.822+0000 7f98d801d700 1 --2- 172.21.15.16:0/182029165 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98d008d2c0 0x7f98d008ef20 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.822+0000 7f98d801d700 1 --2- 172.21.15.16:0/182029165 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98d008fef0 0x7f98d008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:03.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.822+0000 7f98d801d700 1 -- 172.21.15.16:0/182029165 shutdown_connections 2019-12-05T06:39:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.822+0000 7f98d801d700 1 -- 172.21.15.16:0/182029165 wait complete. 2019-12-05T06:39:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.822+0000 7f98d801d700 1 -- 172.21.15.16:0/182029165 >> 172.21.15.16:0/182029165 conn(0x7f98d0061940 msgr2=0x7f98d0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.826+0000 7f98d801d700 1 Processor -- start 2019-12-05T06:39:03.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.826+0000 7f98d801d700 1 -- start start 2019-12-05T06:39:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.827+0000 7f98d801d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f98d014d4a0 0x7f98d014d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.827+0000 7f98d801d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98d012eae0 0x7f98d019b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.827+0000 7f98d801d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98d012ef40 con 0x7f98d012f2b0 2019-12-05T06:39:03.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.827+0000 7f98d801d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98d0098f80 con 0x7f98d014d4a0 2019-12-05T06:39:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.827+0000 7f98d801d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98d0098550 con 0x7f98d012eae0 2019-12-05T06:39:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.827+0000 7f98d5db9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f98d012f2b0 0x7f98d012e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43090/0 (socket says 172.21.15.16:43090) 2019-12-05T06:39:03.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.827+0000 7f98d5db9700 1 -- 172.21.15.16:0/2294092296 learned_addr learned my addr 172.21.15.16:0/2294092296 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.828+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> v2:172.21.15.76:3300/0 conn(0x7f98d014d4a0 0x7f98d014d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.828+0000 7f98d6dbb700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98d012eae0 0x7f98d019b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.828+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562036622 0 0) 0x7f98c0007da0 con 0x7f98d012f2b0 2019-12-05T06:39:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.828+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b8002e80 con 0x7f98d012f2b0 2019-12-05T06:39:03.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.828+0000 7f98d65ba700 1 -- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98d012eae0 msgr2=0x7f98d019b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.828+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f98d012eae0 0x7f98d019b400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.828+0000 7f98d65ba700 1 -- 172.21.15.16:0/2294092296 >> v1:172.21.15.16:6789/0 conn(0x7f98d012f2b0 legacy=0x7f98d012e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.828+0000 7f98d65ba700 1 -- 172.21.15.16:0/2294092296 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98d012d9e0 con 0x7f98d014d4a0 2019-12-05T06:39:03.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.828+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> v2:172.21.15.76:3300/0 conn(0x7f98d014d4a0 0x7f98d014d890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f98cc010390 tx=0x7f98cc0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.828+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> v2:172.21.15.76:3300/0 conn(0x7f98d014d4a0 0x7f98d014d890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f98cc010390 tx=0x7f98cc0101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98d014d4a0 0x7f98d014d890 secure :-1 s=READY pgs=761 cs=0 l=1 rx=0x7f98cc010390 tx=0x7f98cc0101c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:03.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98d65ba700 1 -- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98d014d4a0 msgr2=0x7f98d014d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:39:03.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98d65ba700 1 -- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98d014d4a0 msgr2=0x7f98d014d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:39:03.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98d014d4a0 0x7f98d014d890 secure :-1 s=READY pgs=761 cs=0 l=1 rx=0x7f98cc010390 tx=0x7f98cc0101c0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:39:03.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98d014d4a0 0x7f98d014d890 secure :-1 s=READY pgs=761 cs=0 l=1 rx=0x7f98cc010390 tx=0x7f98cc0101c0).stop 2019-12-05T06:39:03.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98d014d4a0 msgr2=0x7f98d014d890 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:39:03.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98c6ffd700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98d014d4a0 0x7f98d014d890 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98c6ffd700 1 --2- 172.21.15.16:0/2294092296 >> v2:172.21.15.16:3300/0 conn(0x7f98b8002e50 0x7f98b80052e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98b8009890 con 0x7f98b8005950 2019-12-05T06:39:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98b800aba0 con 0x7f98b8002e50 2019-12-05T06:39:03.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> v2:172.21.15.16:3300/0 conn(0x7f98b8002e50 0x7f98b80052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799831816 0 0) 0x7f98c0007da0 con 0x7f98b8005950 2019-12-05T06:39:03.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98b8009af0 con 0x7f98b8005950 2019-12-05T06:39:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98d65ba700 1 -- 172.21.15.16:0/2294092296 >> v1:172.21.15.76:6789/0 conn(0x7f98b8005950 legacy=0x7f98b8007e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.829+0000 7f98d65ba700 1 -- 172.21.15.16:0/2294092296 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98b800ae50 con 0x7f98b8002e50 2019-12-05T06:39:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> v2:172.21.15.16:3300/0 conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> v2:172.21.15.16:3300/0 conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=THROTTLE_DONE pgs=1110 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98cc001220 con 0x7f98b8002e50 2019-12-05T06:39:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=THROTTLE_DONE pgs=1110 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=THROTTLE_DONE pgs=1110 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98cc0106c0 con 0x7f98b8002e50 2019-12-05T06:39:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98d801d700 1 -- 172.21.15.16:0/2294092296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d012d9e0 con 0x7f98b8002e50 2019-12-05T06:39:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98cc001220 con 0x7f98b8002e50 2019-12-05T06:39:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.830+0000 7f98d801d700 1 -- 172.21.15.16:0/2294092296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d012d9e0 con 0x7f98b8002e50 2019-12-05T06:39:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.831+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=THROTTLE_DONE pgs=1110 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.831+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=THROTTLE_DONE pgs=1110 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.831+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f98cc0102c0 con 0x7f98b8002e50 2019-12-05T06:39:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.832+0000 7f98c6ffd700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98b8029fd0 0x7f98b802c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:03.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.832+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f98cc0475b0 con 0x7f98b8002e50 2019-12-05T06:39:03.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.832+0000 7f98d5db9700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98b8029fd0 0x7f98b802c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:03.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.833+0000 7f98d5db9700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98b8029fd0 0x7f98b802c400 crc :-1 s=READY pgs=475 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:03.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.834+0000 7f98d801d700 1 -- 172.21.15.16:0/2294092296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d012d9e0 con 0x7f98b8002e50 2019-12-05T06:39:03.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.842+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=THROTTLE_DONE pgs=1110 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:03.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:03.842+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f98cc001220 con 0x7f98b8002e50 2019-12-05T06:39:04.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.067+0000 7f98d801d700 1 -- 172.21.15.16:0/2294092296 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98d012d9e0 con 0x7f98b8002e50 2019-12-05T06:39:04.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.068+0000 7f98d65ba700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=THROTTLE_DONE pgs=1110 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:04.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.068+0000 7f98c6ffd700 1 -- 172.21.15.16:0/2294092296 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f98cc0102c0 con 0x7f98b8002e50 2019-12-05T06:39:04.080 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2019-12-05T06:39:04.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d801d700 1 -- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98b8029fd0 msgr2=0x7f98b802c400 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:04.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d801d700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f98b8029fd0 0x7f98b802c400 crc :-1 s=READY pgs=475 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:04.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d801d700 1 -- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 msgr2=0x7f98b80052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:04.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d801d700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 0x7f98b80052e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rx=0x7f98cc011ab0 tx=0x7f98cc011e50).stop 2019-12-05T06:39:04.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d6dbb700 1 -- 172.21.15.16:0/2294092296 reap_dead start 2019-12-05T06:39:04.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d6dbb700 1 -- 172.21.15.16:0/2294092296 reap_dead start 2019-12-05T06:39:04.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d801d700 1 -- 172.21.15.16:0/2294092296 shutdown_connections 2019-12-05T06:39:04.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d801d700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f98d014d4a0 0x7f98d014d890 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:04.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d801d700 1 --2- 172.21.15.16:0/2294092296 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f98b8002e50 0x7f98b80052e0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:04.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d801d700 1 -- 172.21.15.16:0/2294092296 shutdown_connections 2019-12-05T06:39:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d801d700 1 -- 172.21.15.16:0/2294092296 wait complete. 2019-12-05T06:39:04.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:04.073+0000 7f98d801d700 1 -- 172.21.15.16:0/2294092296 >> 172.21.15.16:0/2294092296 conn(0x7f98d012d0e0 msgr2=0x7f98d0135a40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:04.089 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:04.090 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:04.091 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:39:06.092 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.365+0000 7ffa28771700 1 Processor -- start 2019-12-05T06:39:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.365+0000 7ffa28771700 1 -- start start 2019-12-05T06:39:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.367+0000 7ffa28771700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffa2008c6d0 0x7ffa2008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:06.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.367+0000 7ffa28771700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa2008f720 0x7ffa2008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.367+0000 7ffa28771700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa200958e0 con 0x7ffa2008fef0 2019-12-05T06:39:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.367+0000 7ffa28771700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa200a3f70 con 0x7ffa2008f720 2019-12-05T06:39:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.367+0000 7ffa28771700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa2008d830 con 0x7ffa2008c6d0 2019-12-05T06:39:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.367+0000 7ffa2750f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa2008f720 0x7ffa2008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.367+0000 7ffa26d0e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffa2008c6d0 0x7ffa2008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.367+0000 7ffa2750f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ffa2008f720 0x7ffa2008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57938/0 (socket says 172.21.15.16:57938) 2019-12-05T06:39:06.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.367+0000 7ffa2750f700 1 -- 172.21.15.16:0/3685090251 learned_addr learned my addr 172.21.15.16:0/3685090251 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:06.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.368+0000 7ffa2750f700 1 -- 172.21.15.16:0/3685090251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffa2008c6d0 msgr2=0x7ffa2008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:06.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.368+0000 7ffa2750f700 1 --2- 172.21.15.16:0/3685090251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffa2008c6d0 0x7ffa2008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:06.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.368+0000 7ffa2750f700 1 -- 172.21.15.16:0/3685090251 >> v1:172.21.15.76:6789/0 conn(0x7ffa2008fef0 legacy=0x7ffa2008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:06.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.368+0000 7ffa25d0c700 1 -- 172.21.15.16:0/3685090251 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1206911304 0 0) 0x7ffa14001030 con 0x7ffa2008fef0 2019-12-05T06:39:06.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.368+0000 7ffa2750f700 1 -- 172.21.15.16:0/3685090251 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2008fbc0 con 0x7ffa2008f720 2019-12-05T06:39:06.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.368+0000 7ffa26d0e700 1 --2- 172.21.15.16:0/3685090251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffa2008c6d0 0x7ffa2008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.368+0000 7ffa2750f700 1 --2- 172.21.15.16:0/3685090251 >> v2:172.21.15.16:3300/0 conn(0x7ffa2008f720 0x7ffa2008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ffa1000f730 tx=0x7ffa1000fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.368+0000 7ffa2750f700 1 --2- 172.21.15.16:0/3685090251 >> v2:172.21.15.16:3300/0 conn(0x7ffa2008f720 0x7ffa2008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ffa1000f730 tx=0x7ffa1000fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.368+0000 7ffa2750f700 1 --2- 172.21.15.16:0/3685090251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2008f720 0x7ffa2008d000 secure :-1 s=READY pgs=1111 cs=0 l=1 rx=0x7ffa1000f730 tx=0x7ffa1000fad0).ready entity=mon.0 client_cookie=10b42bc3867a03cc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.369+0000 7ffa2750f700 1 --2- 172.21.15.16:0/3685090251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2008f720 0x7ffa2008d000 secure :-1 s=THROTTLE_DONE pgs=1111 cs=0 l=1 rx=0x7ffa1000f730 tx=0x7ffa1000fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.369+0000 7ffa25d0c700 1 -- 172.21.15.16:0/3685090251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa10000b90 con 0x7ffa2008f720 2019-12-05T06:39:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.369+0000 7ffa2750f700 1 --2- 172.21.15.16:0/3685090251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2008f720 0x7ffa2008d000 secure :-1 s=THROTTLE_DONE pgs=1111 cs=0 l=1 rx=0x7ffa1000f730 tx=0x7ffa1000fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.369+0000 7ffa2750f700 1 --2- 172.21.15.16:0/3685090251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2008f720 0x7ffa2008d000 secure :-1 s=THROTTLE_DONE pgs=1111 cs=0 l=1 rx=0x7ffa1000f730 tx=0x7ffa1000fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.369+0000 7ffa25d0c700 1 -- 172.21.15.16:0/3685090251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa100010e0 con 0x7ffa2008f720 2019-12-05T06:39:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.369+0000 7ffa25d0c700 1 -- 172.21.15.16:0/3685090251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa10001830 con 0x7ffa2008f720 2019-12-05T06:39:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.369+0000 7ffa28771700 1 -- 172.21.15.16:0/3685090251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2008f720 msgr2=0x7ffa2008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.369+0000 7ffa28771700 1 --2- 172.21.15.16:0/3685090251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2008f720 0x7ffa2008d000 secure :-1 s=READY pgs=1111 cs=0 l=1 rx=0x7ffa1000f730 tx=0x7ffa1000fad0).stop 2019-12-05T06:39:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.370+0000 7ffa28771700 1 -- 172.21.15.16:0/3685090251 shutdown_connections 2019-12-05T06:39:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.370+0000 7ffa28771700 1 --2- 172.21.15.16:0/3685090251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ffa2008f720 0x7ffa2008d000 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:06.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.370+0000 7ffa28771700 1 --2- 172.21.15.16:0/3685090251 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ffa2008c6d0 0x7ffa2008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:06.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.370+0000 7ffa28771700 1 -- 172.21.15.16:0/3685090251 shutdown_connections 2019-12-05T06:39:06.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.370+0000 7ffa28771700 1 -- 172.21.15.16:0/3685090251 wait complete. 2019-12-05T06:39:06.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.370+0000 7ffa28771700 1 -- 172.21.15.16:0/3685090251 >> 172.21.15.16:0/3685090251 conn(0x7ffa20061940 msgr2=0x7ffa20096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:06.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.375+0000 7ffa28771700 1 Processor -- start 2019-12-05T06:39:06.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.375+0000 7ffa28771700 1 -- start start 2019-12-05T06:39:06.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.377+0000 7ffa28771700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ffa2009bbe0 0x7ffa201539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:06.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.377+0000 7ffa28771700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa20084b90 con 0x7ffa20096e90 2019-12-05T06:39:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.377+0000 7ffa28771700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa20098790 con 0x7ffa20084f30 2019-12-05T06:39:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.377+0000 7ffa28771700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa2009a450 con 0x7ffa2009bbe0 2019-12-05T06:39:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.377+0000 7ffa26d0e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ffa20096e90 0x7ffa2009b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43110/0 (socket says 172.21.15.16:43110) 2019-12-05T06:39:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.377+0000 7ffa26d0e700 1 -- 172.21.15.16:0/1961554999 learned_addr learned my addr 172.21.15.16:0/1961554999 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.377+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> v2:172.21.15.76:3300/0 conn(0x7ffa2009bbe0 0x7ffa201539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:06.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.378+0000 7ffa1f7fe700 1 -- 172.21.15.16:0/1961554999 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862987732 0 0) 0x7ffa180013f0 con 0x7ffa20096e90 2019-12-05T06:39:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.378+0000 7ffa1f7fe700 1 -- 172.21.15.16:0/1961554999 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa00002e80 con 0x7ffa20096e90 2019-12-05T06:39:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.378+0000 7ffa2650d700 1 -- 172.21.15.16:0/1961554999 >> v1:172.21.15.76:6789/0 conn(0x7ffa20084f30 legacy=0x7ffa20155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.378+0000 7ffa1f7fe700 1 -- 172.21.15.16:0/1961554999 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4184820985 0 0) 0x7ffa10000b90 con 0x7ffa20084f30 2019-12-05T06:39:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.378+0000 7ffa2650d700 1 -- 172.21.15.16:0/1961554999 >> v1:172.21.15.16:6789/0 conn(0x7ffa20096e90 legacy=0x7ffa2009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.378+0000 7ffa2650d700 1 -- 172.21.15.16:0/1961554999 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa2009aae0 con 0x7ffa2009bbe0 2019-12-05T06:39:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.378+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> v2:172.21.15.76:3300/0 conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.379+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> v2:172.21.15.76:3300/0 conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.379+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=READY pgs=765 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.379+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=THROTTLE_DONE pgs=765 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.379+0000 7ffa1f7fe700 1 -- 172.21.15.16:0/1961554999 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa1400efe0 con 0x7ffa2009bbe0 2019-12-05T06:39:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.379+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=THROTTLE_DONE pgs=765 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.379+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=THROTTLE_DONE pgs=765 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.379+0000 7ffa1f7fe700 1 -- 172.21.15.16:0/1961554999 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa140009d0 con 0x7ffa2009bbe0 2019-12-05T06:39:06.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.379+0000 7ffa28771700 1 -- 172.21.15.16:0/1961554999 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa2009aae0 con 0x7ffa2009bbe0 2019-12-05T06:39:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.379+0000 7ffa1f7fe700 1 -- 172.21.15.16:0/1961554999 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa1400f5c0 con 0x7ffa2009bbe0 2019-12-05T06:39:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.379+0000 7ffa28771700 1 -- 172.21.15.16:0/1961554999 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa2009aae0 con 0x7ffa2009bbe0 2019-12-05T06:39:06.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.380+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=THROTTLE_DONE pgs=765 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.381+0000 7ffa1f7fe700 1 -- 172.21.15.16:0/1961554999 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ffa1400ee20 con 0x7ffa2009bbe0 2019-12-05T06:39:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.381+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=THROTTLE_DONE pgs=765 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.381+0000 7ffa1f7fe700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffa00023680 0x7ffa00025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.381+0000 7ffa1f7fe700 1 -- 172.21.15.16:0/1961554999 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7ffa14045d90 con 0x7ffa2009bbe0 2019-12-05T06:39:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.382+0000 7ffa26d0e700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffa00023680 0x7ffa00025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:06.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.382+0000 7ffa26d0e700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffa00023680 0x7ffa00025af0 crc :-1 s=READY pgs=476 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:06.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.396+0000 7ffa28771700 1 -- 172.21.15.16:0/1961554999 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa2009aae0 con 0x7ffa2009bbe0 2019-12-05T06:39:06.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.406+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=THROTTLE_DONE pgs=765 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.406+0000 7ffa1f7fe700 1 -- 172.21.15.16:0/1961554999 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ffa1400e8e0 con 0x7ffa2009bbe0 2019-12-05T06:39:06.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.667+0000 7ffa28771700 1 -- 172.21.15.16:0/1961554999 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffa2009aae0 con 0x7ffa2009bbe0 2019-12-05T06:39:06.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.668+0000 7ffa2650d700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=THROTTLE_DONE pgs=765 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:06.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.668+0000 7ffa1f7fe700 1 -- 172.21.15.16:0/1961554999 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+2109 (secure 0 0 0) 0x7ffa1400ed40 con 0x7ffa2009bbe0 2019-12-05T06:39:06.669 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:06.670 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:06.670 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2019-12-05T06:39:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.672+0000 7ffa28771700 1 -- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffa00023680 msgr2=0x7ffa00025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.672+0000 7ffa28771700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffa00023680 0x7ffa00025af0 crc :-1 s=READY pgs=476 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.672+0000 7ffa28771700 1 -- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 msgr2=0x7ffa201539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:06.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.672+0000 7ffa28771700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 0x7ffa201539e0 secure :-1 s=READY pgs=765 cs=0 l=1 rx=0x7ffa140103c0 tx=0x7ffa14010760).stop 2019-12-05T06:39:06.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.673+0000 7ffa28771700 1 -- 172.21.15.16:0/1961554999 shutdown_connections 2019-12-05T06:39:06.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.673+0000 7ffa28771700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ffa00023680 0x7ffa00025af0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:06.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.673+0000 7ffa28771700 1 --2- 172.21.15.16:0/1961554999 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ffa2009bbe0 0x7ffa201539e0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:06.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.673+0000 7ffa28771700 1 -- 172.21.15.16:0/1961554999 shutdown_connections 2019-12-05T06:39:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.673+0000 7ffa28771700 1 -- 172.21.15.16:0/1961554999 wait complete. 2019-12-05T06:39:06.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:06.673+0000 7ffa28771700 1 -- 172.21.15.16:0/1961554999 >> 172.21.15.16:0/1961554999 conn(0x7ffa200714f0 msgr2=0x7ffa2009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:06.686 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:39:08.687 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:08.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.968+0000 7fca970df700 1 Processor -- start 2019-12-05T06:39:08.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.968+0000 7fca970df700 1 -- start start 2019-12-05T06:39:08.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.970+0000 7fca970df700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fca9008fef0 0x7fca9008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:08.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.970+0000 7fca970df700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca900958e0 con 0x7fca9008c6d0 2019-12-05T06:39:08.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.970+0000 7fca970df700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca900629e0 con 0x7fca9008d2c0 2019-12-05T06:39:08.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.970+0000 7fca970df700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca9008df90 con 0x7fca9008fef0 2019-12-05T06:39:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.970+0000 7fca9567c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fca9008c6d0 0x7fca9008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43120/0 (socket says 172.21.15.16:43120) 2019-12-05T06:39:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.970+0000 7fca9567c700 1 -- 172.21.15.16:0/2585108001 learned_addr learned my addr 172.21.15.16:0/2585108001 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.971+0000 7fca95e7d700 1 --2- 172.21.15.16:0/2585108001 >> v2:172.21.15.76:3300/0 conn(0x7fca9008fef0 0x7fca9008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.971+0000 7fca87fff700 1 -- 172.21.15.16:0/2585108001 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762526882 0 0) 0x7fca8c001150 con 0x7fca9008c6d0 2019-12-05T06:39:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.971+0000 7fca87fff700 1 -- 172.21.15.16:0/2585108001 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca7c002e80 con 0x7fca9008c6d0 2019-12-05T06:39:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.971+0000 7fca87fff700 1 -- 172.21.15.16:0/2585108001 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3485958866 0 0) 0x7fca80001080 con 0x7fca9008d2c0 2019-12-05T06:39:08.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.971+0000 7fca87fff700 1 -- 172.21.15.16:0/2585108001 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca7c0030b0 con 0x7fca9008d2c0 2019-12-05T06:39:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.971+0000 7fca95e7d700 1 -- 172.21.15.16:0/2585108001 >> v1:172.21.15.76:6789/0 conn(0x7fca9008d2c0 legacy=0x7fca9008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.972+0000 7fca95e7d700 1 -- 172.21.15.16:0/2585108001 >> v1:172.21.15.16:6789/0 conn(0x7fca9008c6d0 legacy=0x7fca9008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.972+0000 7fca95e7d700 1 -- 172.21.15.16:0/2585108001 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9008e620 con 0x7fca9008fef0 2019-12-05T06:39:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.972+0000 7fca95e7d700 1 --2- 172.21.15.16:0/2585108001 >> v2:172.21.15.76:3300/0 conn(0x7fca9008fef0 0x7fca9008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fca8800b300 tx=0x7fca8800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.972+0000 7fca95e7d700 1 --2- 172.21.15.16:0/2585108001 >> v2:172.21.15.76:3300/0 conn(0x7fca9008fef0 0x7fca9008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fca8800b300 tx=0x7fca8800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.972+0000 7fca95e7d700 1 --2- 172.21.15.16:0/2585108001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca9008fef0 0x7fca9008f720 secure :-1 s=READY pgs=767 cs=0 l=1 rx=0x7fca8800b300 tx=0x7fca8800bb10).ready entity=mon.1 client_cookie=368f0c277892752e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:08.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.972+0000 7fca95e7d700 1 --2- 172.21.15.16:0/2585108001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca9008fef0 0x7fca9008f720 secure :-1 s=THROTTLE_DONE pgs=767 cs=0 l=1 rx=0x7fca8800b300 tx=0x7fca8800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.972+0000 7fca87fff700 1 -- 172.21.15.16:0/2585108001 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca880075c0 con 0x7fca9008fef0 2019-12-05T06:39:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.972+0000 7fca95e7d700 1 --2- 172.21.15.16:0/2585108001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca9008fef0 0x7fca9008f720 secure :-1 s=THROTTLE_DONE pgs=767 cs=0 l=1 rx=0x7fca8800b300 tx=0x7fca8800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.972+0000 7fca95e7d700 1 --2- 172.21.15.16:0/2585108001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca9008fef0 0x7fca9008f720 secure :-1 s=THROTTLE_DONE pgs=767 cs=0 l=1 rx=0x7fca8800b300 tx=0x7fca8800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.973+0000 7fca87fff700 1 -- 172.21.15.16:0/2585108001 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca88000f30 con 0x7fca9008fef0 2019-12-05T06:39:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.973+0000 7fca87fff700 1 -- 172.21.15.16:0/2585108001 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca8800a1e0 con 0x7fca9008fef0 2019-12-05T06:39:08.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.973+0000 7fca970df700 1 -- 172.21.15.16:0/2585108001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca9008fef0 msgr2=0x7fca9008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.973+0000 7fca970df700 1 --2- 172.21.15.16:0/2585108001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca9008fef0 0x7fca9008f720 secure :-1 s=READY pgs=767 cs=0 l=1 rx=0x7fca8800b300 tx=0x7fca8800bb10).stop 2019-12-05T06:39:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.974+0000 7fca970df700 1 -- 172.21.15.16:0/2585108001 shutdown_connections 2019-12-05T06:39:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.974+0000 7fca970df700 1 --2- 172.21.15.16:0/2585108001 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fca9008fef0 0x7fca9008f720 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.975+0000 7fca970df700 1 -- 172.21.15.16:0/2585108001 shutdown_connections 2019-12-05T06:39:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.975+0000 7fca970df700 1 -- 172.21.15.16:0/2585108001 wait complete. 2019-12-05T06:39:08.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.975+0000 7fca970df700 1 -- 172.21.15.16:0/2585108001 >> 172.21.15.16:0/2585108001 conn(0x7fca90061940 msgr2=0x7fca90096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:08.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.980+0000 7fca970df700 1 Processor -- start 2019-12-05T06:39:08.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.980+0000 7fca970df700 1 -- start start 2019-12-05T06:39:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.981+0000 7fca970df700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.982+0000 7fca970df700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9007bfc0 con 0x7fca9007fc20 2019-12-05T06:39:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.982+0000 7fca970df700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca9007bac0 con 0x7fca90153f90 2019-12-05T06:39:08.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.982+0000 7fca970df700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca9007d560 con 0x7fca90150b40 2019-12-05T06:39:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.982+0000 7fca94e7b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fca9007fc20 0x7fca90153790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43128/0 (socket says 172.21.15.16:43128) 2019-12-05T06:39:08.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.982+0000 7fca94e7b700 1 -- 172.21.15.16:0/3635485235 learned_addr learned my addr 172.21.15.16:0/3635485235 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.982+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.982+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850900255 0 0) 0x7fca80001240 con 0x7fca9007fc20 2019-12-05T06:39:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.982+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca70002e80 con 0x7fca9007fc20 2019-12-05T06:39:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.982+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2467546224 0 0) 0x7fca88009c90 con 0x7fca90153f90 2019-12-05T06:39:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.982+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca700030b0 con 0x7fca90153f90 2019-12-05T06:39:08.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca9567c700 1 -- 172.21.15.16:0/3635485235 >> v1:172.21.15.76:6789/0 conn(0x7fca90153f90 legacy=0x7fca9007ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca9567c700 1 -- 172.21.15.16:0/3635485235 >> v1:172.21.15.16:6789/0 conn(0x7fca9007fc20 legacy=0x7fca90153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca9567c700 1 -- 172.21.15.16:0/3635485235 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca9007e350 con 0x7fca90150b40 2019-12-05T06:39:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=READY pgs=256 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca8c000d80 con 0x7fca90150b40 2019-12-05T06:39:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca8c000d60 con 0x7fca90150b40 2019-12-05T06:39:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca970df700 1 -- 172.21.15.16:0/3635485235 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca9007e350 con 0x7fca90150b40 2019-12-05T06:39:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.983+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca8c001610 con 0x7fca90150b40 2019-12-05T06:39:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.984+0000 7fca970df700 1 -- 172.21.15.16:0/3635485235 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca9007e350 con 0x7fca90150b40 2019-12-05T06:39:08.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.984+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.985+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fca8c001230 con 0x7fca90150b40 2019-12-05T06:39:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.985+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.986+0000 7fca85ffb700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fca70023650 0x7fca70025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.986+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7fca8c045da0 con 0x7fca90150b40 2019-12-05T06:39:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.986+0000 7fca94e7b700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fca70023650 0x7fca70025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:08.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:08.987+0000 7fca94e7b700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fca70023650 0x7fca70025ae0 crc :-1 s=READY pgs=477 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:08.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.000+0000 7fca970df700 1 -- 172.21.15.16:0/3635485235 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca9007e350 con 0x7fca90150b40 2019-12-05T06:39:09.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.008+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:09.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.008+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fca8c000f80 con 0x7fca90150b40 2019-12-05T06:39:09.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.261+0000 7fca970df700 1 -- 172.21.15.16:0/3635485235 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fca9007e350 con 0x7fca90150b40 2019-12-05T06:39:09.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.262+0000 7fca9567c700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=THROTTLE_DONE pgs=256 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:09.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.262+0000 7fca85ffb700 1 -- 172.21.15.16:0/3635485235 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fca8c001190 con 0x7fca90150b40 2019-12-05T06:39:09.264 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2019-12-05T06:39:09.265 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:09.265 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:09.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.266+0000 7fca970df700 1 -- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fca70023650 msgr2=0x7fca70025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:09.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.266+0000 7fca970df700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fca70023650 0x7fca70025ae0 crc :-1 s=READY pgs=477 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:09.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.267+0000 7fca970df700 1 -- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 msgr2=0x7fca90150f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:09.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.267+0000 7fca970df700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 secure :-1 s=READY pgs=256 cs=0 l=1 rx=0x7fca8c010380 tx=0x7fca8c010720).stop 2019-12-05T06:39:09.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.267+0000 7fca970df700 1 -- 172.21.15.16:0/3635485235 shutdown_connections 2019-12-05T06:39:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.267+0000 7fca970df700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fca70023650 0x7fca70025ae0 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.267+0000 7fca970df700 1 --2- 172.21.15.16:0/3635485235 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fca90150b40 0x7fca90150f30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.267+0000 7fca970df700 1 -- 172.21.15.16:0/3635485235 shutdown_connections 2019-12-05T06:39:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.267+0000 7fca970df700 1 -- 172.21.15.16:0/3635485235 wait complete. 2019-12-05T06:39:09.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:09.267+0000 7fca970df700 1 -- 172.21.15.16:0/3635485235 >> 172.21.15.16:0/3635485235 conn(0x7fca90084520 msgr2=0x7fca900995a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:09.278 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:39:10.502 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:39:10.507 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:39:11.281 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.571+0000 7fb1ac0bb700 1 Processor -- start 2019-12-05T06:39:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.571+0000 7fb1ac0bb700 1 -- start start 2019-12-05T06:39:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.573+0000 7fb1ac0bb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a408d2c0 0x7fb1a408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:11.581 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.573+0000 7fb1ac0bb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb1a408fef0 0x7fb1a408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.573+0000 7fb1ac0bb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a40958e0 con 0x7fb1a408c6d0 2019-12-05T06:39:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.573+0000 7fb1ac0bb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1a408df90 con 0x7fb1a408fef0 2019-12-05T06:39:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.573+0000 7fb1ac0bb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a40a3f70 con 0x7fb1a408d2c0 2019-12-05T06:39:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.573+0000 7fb1aa658700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb1a408c6d0 0x7fb1a408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43140/0 (socket says 172.21.15.16:43140) 2019-12-05T06:39:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.573+0000 7fb1aa658700 1 -- 172.21.15.16:0/2709231053 learned_addr learned my addr 172.21.15.16:0/2709231053 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:11.582 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.573+0000 7fb1a9e57700 1 --2- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a408d2c0 0x7fb1a408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.573+0000 7fb1aae59700 1 --2- 172.21.15.16:0/2709231053 >> v2:172.21.15.76:3300/0 conn(0x7fb1a408fef0 0x7fb1a408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.574+0000 7fb1a9656700 1 -- 172.21.15.16:0/2709231053 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557884664 0 0) 0x7fb1a00014d0 con 0x7fb1a408c6d0 2019-12-05T06:39:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.574+0000 7fb1a9656700 1 -- 172.21.15.16:0/2709231053 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190002e80 con 0x7fb1a408c6d0 2019-12-05T06:39:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.574+0000 7fb1aae59700 1 -- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a408d2c0 msgr2=0x7fb1a408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.574+0000 7fb1aae59700 1 --2- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a408d2c0 0x7fb1a408ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.574+0000 7fb1aae59700 1 -- 172.21.15.16:0/2709231053 >> v1:172.21.15.16:6789/0 conn(0x7fb1a408c6d0 legacy=0x7fb1a408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:11.583 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.574+0000 7fb1aae59700 1 -- 172.21.15.16:0/2709231053 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a408e620 con 0x7fb1a408fef0 2019-12-05T06:39:11.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.574+0000 7fb1a9e57700 1 --2- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a408d2c0 0x7fb1a408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:11.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1aae59700 1 --2- 172.21.15.16:0/2709231053 >> v2:172.21.15.76:3300/0 conn(0x7fb1a408fef0 0x7fb1a408f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb19c00af80 tx=0x7fb19c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1aae59700 1 --2- 172.21.15.16:0/2709231053 >> v2:172.21.15.76:3300/0 conn(0x7fb1a408fef0 0x7fb1a408f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb19c00af80 tx=0x7fb19c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1aae59700 1 --2- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1a408fef0 0x7fb1a408f460 secure :-1 s=READY pgs=769 cs=0 l=1 rx=0x7fb19c00af80 tx=0x7fb19c00b790).ready entity=mon.1 client_cookie=932669fc3158f953 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:11.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1aae59700 1 --2- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1a408fef0 0x7fb1a408f460 secure :-1 s=THROTTLE_DONE pgs=769 cs=0 l=1 rx=0x7fb19c00af80 tx=0x7fb19c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1aae59700 1 --2- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1a408fef0 0x7fb1a408f460 secure :-1 s=THROTTLE_DONE pgs=769 cs=0 l=1 rx=0x7fb19c00af80 tx=0x7fb19c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1a9656700 1 -- 172.21.15.16:0/2709231053 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb19c000bc0 con 0x7fb1a408fef0 2019-12-05T06:39:11.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1aae59700 1 --2- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1a408fef0 0x7fb1a408f460 secure :-1 s=THROTTLE_DONE pgs=769 cs=0 l=1 rx=0x7fb19c00af80 tx=0x7fb19c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1a9656700 1 -- 172.21.15.16:0/2709231053 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb19c0097f0 con 0x7fb1a408fef0 2019-12-05T06:39:11.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1a9656700 1 -- 172.21.15.16:0/2709231053 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb19c009f60 con 0x7fb1a408fef0 2019-12-05T06:39:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1a408fef0 msgr2=0x7fb1a408f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.575+0000 7fb1ac0bb700 1 --2- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1a408fef0 0x7fb1a408f460 secure :-1 s=READY pgs=769 cs=0 l=1 rx=0x7fb19c00af80 tx=0x7fb19c00b790).stop 2019-12-05T06:39:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.577+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2709231053 shutdown_connections 2019-12-05T06:39:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.578+0000 7fb1ac0bb700 1 --2- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb1a408fef0 0x7fb1a408f460 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.578+0000 7fb1ac0bb700 1 --2- 172.21.15.16:0/2709231053 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a408d2c0 0x7fb1a408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.578+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2709231053 shutdown_connections 2019-12-05T06:39:11.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.578+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2709231053 wait complete. 2019-12-05T06:39:11.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.578+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2709231053 >> 172.21.15.16:0/2709231053 conn(0x7fb1a4061940 msgr2=0x7fb1a4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.583+0000 7fb1ac0bb700 1 Processor -- start 2019-12-05T06:39:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.583+0000 7fb1ac0bb700 1 -- start start 2019-12-05T06:39:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.585+0000 7fb1ac0bb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:11.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.585+0000 7fb1ac0bb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a413ba10 con 0x7fb1a409d8b0 2019-12-05T06:39:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.585+0000 7fb1ac0bb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a413b510 con 0x7fb1a4154b00 2019-12-05T06:39:11.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.585+0000 7fb1ac0bb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a413cca0 con 0x7fb1a40a0920 2019-12-05T06:39:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.585+0000 7fb1a9e57700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb1a409d8b0 0x7fb1a409ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43148/0 (socket says 172.21.15.16:43148) 2019-12-05T06:39:11.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.585+0000 7fb1a9e57700 1 -- 172.21.15.16:0/2559034930 learned_addr learned my addr 172.21.15.16:0/2559034930 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.585+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.586+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1922059539 0 0) 0x7fb19400dd90 con 0x7fb1a409d8b0 2019-12-05T06:39:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.586+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb18c002e80 con 0x7fb1a409d8b0 2019-12-05T06:39:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.586+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2225270851 0 0) 0x7fb19c00acb0 con 0x7fb1a4154b00 2019-12-05T06:39:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.586+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb18c0030b0 con 0x7fb1a4154b00 2019-12-05T06:39:11.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.586+0000 7fb1aa658700 1 -- 172.21.15.16:0/2559034930 >> v1:172.21.15.76:6789/0 conn(0x7fb1a4154b00 legacy=0x7fb1a4156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.586+0000 7fb1aa658700 1 -- 172.21.15.16:0/2559034930 >> v1:172.21.15.16:6789/0 conn(0x7fb1a409d8b0 legacy=0x7fb1a409ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.586+0000 7fb1aa658700 1 -- 172.21.15.16:0/2559034930 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a413bec0 con 0x7fb1a40a0920 2019-12-05T06:39:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.586+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.586+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.586+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=READY pgs=257 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.587+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.587+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1a0001100 con 0x7fb1a40a0920 2019-12-05T06:39:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.587+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.587+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.587+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1a0001700 con 0x7fb1a40a0920 2019-12-05T06:39:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.587+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1a0001dd0 con 0x7fb1a40a0920 2019-12-05T06:39:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.587+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2559034930 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a413bec0 con 0x7fb1a40a0920 2019-12-05T06:39:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.587+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2559034930 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a413bec0 con 0x7fb1a40a0920 2019-12-05T06:39:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.588+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.589+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb1a00018f0 con 0x7fb1a40a0920 2019-12-05T06:39:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.589+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.589+0000 7fb19affd700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb18c023650 0x7fb18c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.589+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7fb1a00460f0 con 0x7fb1a40a0920 2019-12-05T06:39:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.590+0000 7fb1a9e57700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb18c023650 0x7fb18c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:11.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.590+0000 7fb1a9e57700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb18c023650 0x7fb18c025ae0 crc :-1 s=READY pgs=478 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.602+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2559034930 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a413ba10 con 0x7fb1a40a0920 2019-12-05T06:39:11.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.612+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.612+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb1a0001190 con 0x7fb1a40a0920 2019-12-05T06:39:11.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.866+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2559034930 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb1a413ba10 con 0x7fb1a40a0920 2019-12-05T06:39:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.868+0000 7fb1aa658700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=THROTTLE_DONE pgs=257 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:11.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.868+0000 7fb19affd700 1 -- 172.21.15.16:0/2559034930 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fb1a00018f0 con 0x7fb1a40a0920 2019-12-05T06:39:11.869 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:11.869 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:11.870 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2019-12-05T06:39:11.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.872+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb18c023650 msgr2=0x7fb18c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.872+0000 7fb1ac0bb700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb18c023650 0x7fb18c025ae0 crc :-1 s=READY pgs=478 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:11.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.872+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 msgr2=0x7fb1a409c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.872+0000 7fb1ac0bb700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 secure :-1 s=READY pgs=257 cs=0 l=1 rx=0x7fb1a0010700 tx=0x7fb1a0010aa0).stop 2019-12-05T06:39:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.872+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2559034930 shutdown_connections 2019-12-05T06:39:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.872+0000 7fb1ac0bb700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb18c023650 0x7fb18c025ae0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.872+0000 7fb1ac0bb700 1 --2- 172.21.15.16:0/2559034930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb1a40a0920 0x7fb1a409c8e0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.872+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2559034930 shutdown_connections 2019-12-05T06:39:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.872+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2559034930 wait complete. 2019-12-05T06:39:11.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:11.872+0000 7fb1ac0bb700 1 -- 172.21.15.16:0/2559034930 >> 172.21.15.16:0/2559034930 conn(0x7fb1a4130a60 msgr2=0x7fb1a4086470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:11.884 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:39:13.884 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.166+0000 7fb53dcc0700 1 Processor -- start 2019-12-05T06:39:14.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.166+0000 7fb53dcc0700 1 -- start start 2019-12-05T06:39:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.168+0000 7fb53dcc0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb53808c6d0 0x7fb53808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.168+0000 7fb53dcc0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53808fb80 con 0x7fb53808f720 2019-12-05T06:39:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.168+0000 7fb53dcc0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5380958e0 con 0x7fb53808fef0 2019-12-05T06:39:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.168+0000 7fb53dcc0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb53808df90 con 0x7fb53808c6d0 2019-12-05T06:39:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.168+0000 7fb53ca5e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb53808f720 0x7fb5381533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43158/0 (socket says 172.21.15.16:43158) 2019-12-05T06:39:14.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.168+0000 7fb53ca5e700 1 -- 172.21.15.16:0/4170659511 learned_addr learned my addr 172.21.15.16:0/4170659511 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.168+0000 7fb537fff700 1 --2- 172.21.15.16:0/4170659511 >> v2:172.21.15.76:3300/0 conn(0x7fb53808c6d0 0x7fb53808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.169+0000 7fb536ffd700 1 -- 172.21.15.16:0/4170659511 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364119411 0 0) 0x7fb530001150 con 0x7fb53808f720 2019-12-05T06:39:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.169+0000 7fb536ffd700 1 -- 172.21.15.16:0/4170659511 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb524002e80 con 0x7fb53808f720 2019-12-05T06:39:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.169+0000 7fb537fff700 1 -- 172.21.15.16:0/4170659511 >> v1:172.21.15.76:6789/0 conn(0x7fb53808fef0 legacy=0x7fb53808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.169+0000 7fb537fff700 1 -- 172.21.15.16:0/4170659511 >> v1:172.21.15.16:6789/0 conn(0x7fb53808f720 legacy=0x7fb5381533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.169+0000 7fb536ffd700 1 -- 172.21.15.16:0/4170659511 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111118877 0 0) 0x7fb520001080 con 0x7fb53808fef0 2019-12-05T06:39:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.169+0000 7fb537fff700 1 -- 172.21.15.16:0/4170659511 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53808e620 con 0x7fb53808c6d0 2019-12-05T06:39:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.169+0000 7fb537fff700 1 --2- 172.21.15.16:0/4170659511 >> v2:172.21.15.76:3300/0 conn(0x7fb53808c6d0 0x7fb53808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb52800b2d0 tx=0x7fb52800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.170+0000 7fb537fff700 1 --2- 172.21.15.16:0/4170659511 >> v2:172.21.15.76:3300/0 conn(0x7fb53808c6d0 0x7fb53808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb52800b2d0 tx=0x7fb52800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.170+0000 7fb537fff700 1 --2- 172.21.15.16:0/4170659511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb53808c6d0 0x7fb53808cac0 secure :-1 s=READY pgs=772 cs=0 l=1 rx=0x7fb52800b2d0 tx=0x7fb52800bae0).ready entity=mon.1 client_cookie=161b1c7b4369c687 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.170+0000 7fb537fff700 1 --2- 172.21.15.16:0/4170659511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb53808c6d0 0x7fb53808cac0 secure :-1 s=THROTTLE_DONE pgs=772 cs=0 l=1 rx=0x7fb52800b2d0 tx=0x7fb52800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.170+0000 7fb537fff700 1 --2- 172.21.15.16:0/4170659511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb53808c6d0 0x7fb53808cac0 secure :-1 s=THROTTLE_DONE pgs=772 cs=0 l=1 rx=0x7fb52800b2d0 tx=0x7fb52800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.170+0000 7fb536ffd700 1 -- 172.21.15.16:0/4170659511 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb528000ce0 con 0x7fb53808c6d0 2019-12-05T06:39:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.170+0000 7fb537fff700 1 --2- 172.21.15.16:0/4170659511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb53808c6d0 0x7fb53808cac0 secure :-1 s=THROTTLE_DONE pgs=772 cs=0 l=1 rx=0x7fb52800b2d0 tx=0x7fb52800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.170+0000 7fb536ffd700 1 -- 172.21.15.16:0/4170659511 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb528009c60 con 0x7fb53808c6d0 2019-12-05T06:39:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.170+0000 7fb536ffd700 1 -- 172.21.15.16:0/4170659511 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb52800a3d0 con 0x7fb53808c6d0 2019-12-05T06:39:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.170+0000 7fb53dcc0700 1 -- 172.21.15.16:0/4170659511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb53808c6d0 msgr2=0x7fb53808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.170+0000 7fb53dcc0700 1 --2- 172.21.15.16:0/4170659511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb53808c6d0 0x7fb53808cac0 secure :-1 s=READY pgs=772 cs=0 l=1 rx=0x7fb52800b2d0 tx=0x7fb52800bae0).stop 2019-12-05T06:39:14.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.172+0000 7fb53dcc0700 1 -- 172.21.15.16:0/4170659511 shutdown_connections 2019-12-05T06:39:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.172+0000 7fb53dcc0700 1 --2- 172.21.15.16:0/4170659511 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb53808c6d0 0x7fb53808cac0 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.172+0000 7fb53dcc0700 1 -- 172.21.15.16:0/4170659511 shutdown_connections 2019-12-05T06:39:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.172+0000 7fb53dcc0700 1 -- 172.21.15.16:0/4170659511 wait complete. 2019-12-05T06:39:14.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.172+0000 7fb53dcc0700 1 -- 172.21.15.16:0/4170659511 >> 172.21.15.16:0/4170659511 conn(0x7fb538061940 msgr2=0x7fb538096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.178+0000 7fb53dcc0700 1 Processor -- start 2019-12-05T06:39:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.178+0000 7fb53dcc0700 1 -- start start 2019-12-05T06:39:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.180+0000 7fb53dcc0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53806dc50 0x7fb53806e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.180+0000 7fb53dcc0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb538135450 0x7fb53806e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.180+0000 7fb53dcc0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb538128af0 con 0x7fb538134480 2019-12-05T06:39:14.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.180+0000 7fb53dcc0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb538132830 con 0x7fb538135450 2019-12-05T06:39:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.180+0000 7fb53dcc0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb538129d80 con 0x7fb53806dc50 2019-12-05T06:39:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.180+0000 7fb5377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb538135450 0x7fb53806e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.180+0000 7fb5377fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb538135450 0x7fb53806e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:57998/0 (socket says 172.21.15.16:57998) 2019-12-05T06:39:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.180+0000 7fb5377fe700 1 -- 172.21.15.16:0/613530648 learned_addr learned my addr 172.21.15.16:0/613530648 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.180+0000 7fb537fff700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53806dc50 0x7fb53806e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:14.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.181+0000 7fb534ff9700 1 -- 172.21.15.16:0/613530648 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871212740 0 0) 0x7fb5300009b0 con 0x7fb538134480 2019-12-05T06:39:14.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.181+0000 7fb534ff9700 1 -- 172.21.15.16:0/613530648 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb51c002e80 con 0x7fb538134480 2019-12-05T06:39:14.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.181+0000 7fb5377fe700 1 -- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53806dc50 msgr2=0x7fb53806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.181+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53806dc50 0x7fb53806e040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.181+0000 7fb5377fe700 1 -- 172.21.15.16:0/613530648 >> v1:172.21.15.76:6789/0 conn(0x7fb538134480 legacy=0x7fb538134900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.181+0000 7fb5377fe700 1 -- 172.21.15.16:0/613530648 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb538133b80 con 0x7fb538135450 2019-12-05T06:39:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.181+0000 7fb537fff700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53806dc50 0x7fb53806e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:14.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.181+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> v2:172.21.15.16:3300/0 conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.181+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> v2:172.21.15.16:3300/0 conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.181+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=READY pgs=1120 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.182+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=THROTTLE_DONE pgs=1120 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.182+0000 7fb534ff9700 1 -- 172.21.15.16:0/613530648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb520000b50 con 0x7fb538135450 2019-12-05T06:39:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.182+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=THROTTLE_DONE pgs=1120 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.182+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=THROTTLE_DONE pgs=1120 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.182+0000 7fb534ff9700 1 -- 172.21.15.16:0/613530648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb520009a30 con 0x7fb538135450 2019-12-05T06:39:14.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.182+0000 7fb53dcc0700 1 -- 172.21.15.16:0/613530648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb538133b80 con 0x7fb538135450 2019-12-05T06:39:14.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.182+0000 7fb534ff9700 1 -- 172.21.15.16:0/613530648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb52000a100 con 0x7fb538135450 2019-12-05T06:39:14.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.182+0000 7fb53dcc0700 1 -- 172.21.15.16:0/613530648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb538133b80 con 0x7fb538135450 2019-12-05T06:39:14.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.183+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=THROTTLE_DONE pgs=1120 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.184+0000 7fb534ff9700 1 -- 172.21.15.16:0/613530648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb520009630 con 0x7fb538135450 2019-12-05T06:39:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.184+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=THROTTLE_DONE pgs=1120 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.184+0000 7fb534ff9700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb51c023670 0x7fb51c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.184+0000 7fb534ff9700 1 -- 172.21.15.16:0/613530648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7fb520045e60 con 0x7fb538135450 2019-12-05T06:39:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.185+0000 7fb537fff700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb51c023670 0x7fb51c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.185+0000 7fb537fff700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb51c023670 0x7fb51c025ae0 crc :-1 s=READY pgs=479 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.197+0000 7fb53dcc0700 1 -- 172.21.15.16:0/613530648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb538133b80 con 0x7fb538135450 2019-12-05T06:39:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.206+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=THROTTLE_DONE pgs=1120 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.206+0000 7fb534ff9700 1 -- 172.21.15.16:0/613530648 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb52000a5d0 con 0x7fb538135450 2019-12-05T06:39:14.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.468+0000 7fb53dcc0700 1 -- 172.21.15.16:0/613530648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb538133b80 con 0x7fb538135450 2019-12-05T06:39:14.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.469+0000 7fb5377fe700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=THROTTLE_DONE pgs=1120 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.469+0000 7fb534ff9700 1 -- 172.21.15.16:0/613530648 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fb52000a2b0 con 0x7fb538135450 2019-12-05T06:39:14.513 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2019-12-05T06:39:14.513 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:14.513 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:14.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.471+0000 7fb53dcc0700 1 -- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb51c023670 msgr2=0x7fb51c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.471+0000 7fb53dcc0700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb51c023670 0x7fb51c025ae0 crc :-1 s=READY pgs=479 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.471+0000 7fb53dcc0700 1 -- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 msgr2=0x7fb53806e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:14.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.471+0000 7fb53dcc0700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 0x7fb53806e580 secure :-1 s=READY pgs=1120 cs=0 l=1 rx=0x7fb520010450 tx=0x7fb5200107f0).stop 2019-12-05T06:39:14.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.471+0000 7fb53dcc0700 1 -- 172.21.15.16:0/613530648 shutdown_connections 2019-12-05T06:39:14.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.471+0000 7fb53dcc0700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb51c023670 0x7fb51c025ae0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.471+0000 7fb53dcc0700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb538135450 0x7fb53806e580 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.471+0000 7fb53dcc0700 1 --2- 172.21.15.16:0/613530648 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53806dc50 0x7fb53806e040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:14.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.472+0000 7fb53dcc0700 1 -- 172.21.15.16:0/613530648 shutdown_connections 2019-12-05T06:39:14.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.472+0000 7fb53dcc0700 1 -- 172.21.15.16:0/613530648 wait complete. 2019-12-05T06:39:14.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:14.472+0000 7fb53dcc0700 1 -- 172.21.15.16:0/613530648 >> 172.21.15.16:0/613530648 conn(0x7fb5380a2850 msgr2=0x7fb53806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:14.519 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:39:16.520 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:16.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.812+0000 7f93d0fbd700 1 Processor -- start 2019-12-05T06:39:16.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.812+0000 7f93d0fbd700 1 -- start start 2019-12-05T06:39:16.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.814+0000 7f93d0fbd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f93cc08c6d0 0x7f93cc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:16.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.814+0000 7f93d0fbd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93cc08fb80 con 0x7f93cc08fef0 2019-12-05T06:39:16.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.814+0000 7f93d0fbd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93cc0958e0 con 0x7f93cc08f720 2019-12-05T06:39:16.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.814+0000 7f93d0fbd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93cc08df90 con 0x7f93cc08c6d0 2019-12-05T06:39:16.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.814+0000 7f93ca59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f93cc08fef0 0x7f93cc08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43180/0 (socket says 172.21.15.16:43180) 2019-12-05T06:39:16.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.814+0000 7f93ca59c700 1 -- 172.21.15.16:0/2741598111 learned_addr learned my addr 172.21.15.16:0/2741598111 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:16.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.814+0000 7f93cad9d700 1 --2- 172.21.15.16:0/2741598111 >> v2:172.21.15.76:3300/0 conn(0x7f93cc08c6d0 0x7f93cc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:16.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.815+0000 7f93c9d9b700 1 -- 172.21.15.16:0/2741598111 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1651286575 0 0) 0x7f93b40014d0 con 0x7f93cc08fef0 2019-12-05T06:39:16.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.815+0000 7f93c9d9b700 1 -- 172.21.15.16:0/2741598111 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b8002e80 con 0x7f93cc08fef0 2019-12-05T06:39:16.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.815+0000 7f93cad9d700 1 -- 172.21.15.16:0/2741598111 >> v1:172.21.15.76:6789/0 conn(0x7f93cc08f720 legacy=0x7f93cc1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:16.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.815+0000 7f93c9d9b700 1 -- 172.21.15.16:0/2741598111 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3834683252 0 0) 0x7f93c40009b0 con 0x7f93cc08f720 2019-12-05T06:39:16.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.815+0000 7f93cad9d700 1 -- 172.21.15.16:0/2741598111 >> v1:172.21.15.16:6789/0 conn(0x7f93cc08fef0 legacy=0x7f93cc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:16.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.816+0000 7f93cad9d700 1 -- 172.21.15.16:0/2741598111 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc08e620 con 0x7f93cc08c6d0 2019-12-05T06:39:16.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.816+0000 7f93cad9d700 1 --2- 172.21.15.16:0/2741598111 >> v2:172.21.15.76:3300/0 conn(0x7f93cc08c6d0 0x7f93cc08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f93bc00af50 tx=0x7f93bc00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.816+0000 7f93cad9d700 1 --2- 172.21.15.16:0/2741598111 >> v2:172.21.15.76:3300/0 conn(0x7f93cc08c6d0 0x7f93cc08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f93bc00af50 tx=0x7f93bc00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.816+0000 7f93cad9d700 1 --2- 172.21.15.16:0/2741598111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93cc08c6d0 0x7f93cc08cac0 secure :-1 s=READY pgs=775 cs=0 l=1 rx=0x7f93bc00af50 tx=0x7f93bc00b760).ready entity=mon.1 client_cookie=34fa252651522a45 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:16.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.816+0000 7f93cad9d700 1 --2- 172.21.15.16:0/2741598111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93cc08c6d0 0x7f93cc08cac0 secure :-1 s=THROTTLE_DONE pgs=775 cs=0 l=1 rx=0x7f93bc00af50 tx=0x7f93bc00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.816+0000 7f93cad9d700 1 --2- 172.21.15.16:0/2741598111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93cc08c6d0 0x7f93cc08cac0 secure :-1 s=THROTTLE_DONE pgs=775 cs=0 l=1 rx=0x7f93bc00af50 tx=0x7f93bc00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.816+0000 7f93c9d9b700 1 -- 172.21.15.16:0/2741598111 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93bc000960 con 0x7f93cc08c6d0 2019-12-05T06:39:16.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.816+0000 7f93cad9d700 1 --2- 172.21.15.16:0/2741598111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93cc08c6d0 0x7f93cc08cac0 secure :-1 s=THROTTLE_DONE pgs=775 cs=0 l=1 rx=0x7f93bc00af50 tx=0x7f93bc00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.816+0000 7f93c9d9b700 1 -- 172.21.15.16:0/2741598111 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93bc0098e0 con 0x7f93cc08c6d0 2019-12-05T06:39:16.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.816+0000 7f93c9d9b700 1 -- 172.21.15.16:0/2741598111 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93bc00a050 con 0x7f93cc08c6d0 2019-12-05T06:39:16.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.817+0000 7f93d0fbd700 1 -- 172.21.15.16:0/2741598111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93cc08c6d0 msgr2=0x7f93cc08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:16.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.817+0000 7f93d0fbd700 1 --2- 172.21.15.16:0/2741598111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93cc08c6d0 0x7f93cc08cac0 secure :-1 s=READY pgs=775 cs=0 l=1 rx=0x7f93bc00af50 tx=0x7f93bc00b760).stop 2019-12-05T06:39:16.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.819+0000 7f93d0fbd700 1 -- 172.21.15.16:0/2741598111 shutdown_connections 2019-12-05T06:39:16.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.819+0000 7f93d0fbd700 1 --2- 172.21.15.16:0/2741598111 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f93cc08c6d0 0x7f93cc08cac0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:16.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.819+0000 7f93d0fbd700 1 -- 172.21.15.16:0/2741598111 shutdown_connections 2019-12-05T06:39:16.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.819+0000 7f93d0fbd700 1 -- 172.21.15.16:0/2741598111 wait complete. 2019-12-05T06:39:16.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.819+0000 7f93d0fbd700 1 -- 172.21.15.16:0/2741598111 >> 172.21.15.16:0/2741598111 conn(0x7f93cc061940 msgr2=0x7f93cc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:16.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.824+0000 7f93d0fbd700 1 Processor -- start 2019-12-05T06:39:16.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.824+0000 7f93d0fbd700 1 -- start start 2019-12-05T06:39:16.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.825+0000 7f93d0fbd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93cc12b640 0x7f93cc12ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:16.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.825+0000 7f93d0fbd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f93cc153ba0 0x7f93cc155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:16.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.825+0000 7f93d0fbd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93cc131bf0 con 0x7f93cc1302f0 2019-12-05T06:39:16.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.825+0000 7f93d0fbd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93cc1296e0 con 0x7f93cc12b640 2019-12-05T06:39:16.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.825+0000 7f93d0fbd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93cc128cb0 con 0x7f93cc153ba0 2019-12-05T06:39:16.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.825+0000 7f93ca59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93cc12b640 0x7f93cc12ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:16.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.825+0000 7f93ca59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f93cc12b640 0x7f93cc12ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58018/0 (socket says 172.21.15.16:58018) 2019-12-05T06:39:16.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.825+0000 7f93ca59c700 1 -- 172.21.15.16:0/3655472904 learned_addr learned my addr 172.21.15.16:0/3655472904 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:16.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.825+0000 7f93cb59e700 1 --2- 172.21.15.16:0/3655472904 >> v2:172.21.15.76:3300/0 conn(0x7f93cc153ba0 0x7f93cc155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:16.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.826+0000 7f93c37fe700 1 -- 172.21.15.16:0/3655472904 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480690290 0 0) 0x7f93bc000940 con 0x7f93cc1302f0 2019-12-05T06:39:16.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.826+0000 7f93c37fe700 1 -- 172.21.15.16:0/3655472904 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93b0002e80 con 0x7f93cc1302f0 2019-12-05T06:39:16.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.826+0000 7f93ca59c700 1 -- 172.21.15.16:0/3655472904 >> v2:172.21.15.76:3300/0 conn(0x7f93cc153ba0 msgr2=0x7f93cc155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:16.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.826+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> v2:172.21.15.76:3300/0 conn(0x7f93cc153ba0 0x7f93cc155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.826+0000 7f93ca59c700 1 -- 172.21.15.16:0/3655472904 >> v1:172.21.15.76:6789/0 conn(0x7f93cc1302f0 legacy=0x7f93cc12a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:16.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.826+0000 7f93ca59c700 1 -- 172.21.15.16:0/3655472904 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93cc129d70 con 0x7f93cc12b640 2019-12-05T06:39:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.826+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> v2:172.21.15.16:3300/0 conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.826+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> v2:172.21.15.16:3300/0 conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.826+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=READY pgs=1122 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.827+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.827+0000 7f93c37fe700 1 -- 172.21.15.16:0/3655472904 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93b4001010 con 0x7f93cc12b640 2019-12-05T06:39:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.827+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.827+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.827+0000 7f93c37fe700 1 -- 172.21.15.16:0/3655472904 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93b4001440 con 0x7f93cc12b640 2019-12-05T06:39:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.827+0000 7f93d0fbd700 1 -- 172.21.15.16:0/3655472904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93cc129d70 con 0x7f93cc12b640 2019-12-05T06:39:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.827+0000 7f93c37fe700 1 -- 172.21.15.16:0/3655472904 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93b4001ad0 con 0x7f93cc12b640 2019-12-05T06:39:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.827+0000 7f93d0fbd700 1 -- 172.21.15.16:0/3655472904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93cc129d70 con 0x7f93cc12b640 2019-12-05T06:39:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.828+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.829+0000 7f93c37fe700 1 -- 172.21.15.16:0/3655472904 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f93b4001010 con 0x7f93cc12b640 2019-12-05T06:39:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.829+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.829+0000 7f93c37fe700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f93b0023670 0x7f93b0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.829+0000 7f93c37fe700 1 -- 172.21.15.16:0/3655472904 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 7837+0+0 (secure 0 0 0) 0x7f93b40460f0 con 0x7f93cc12b640 2019-12-05T06:39:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.830+0000 7f93cad9d700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f93b0023670 0x7f93b0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.830+0000 7f93cad9d700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f93b0023670 0x7f93b0025ae0 crc :-1 s=READY pgs=480 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.844+0000 7f93d0fbd700 1 -- 172.21.15.16:0/3655472904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93cc129d70 con 0x7f93cc12b640 2019-12-05T06:39:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.853+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:16.853+0000 7f93c37fe700 1 -- 172.21.15.16:0/3655472904 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f93b4007950 con 0x7f93cc12b640 2019-12-05T06:39:17.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.118+0000 7f93d0fbd700 1 -- 172.21.15.16:0/3655472904 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f93cc129d70 con 0x7f93cc12b640 2019-12-05T06:39:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.119+0000 7f93ca59c700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:17.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.119+0000 7f93c37fe700 1 -- 172.21.15.16:0/3655472904 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 137 v137) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f93b4001010 con 0x7f93cc12b640 2019-12-05T06:39:17.121 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 137 2019-12-05T06:39:17.121 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:17.122 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":137,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7011,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":137}],"filesystems":[{"mdsmap":{"epoch":136,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:37:12.675375+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":70,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6940},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6940":{"gid":6940,"name":"b","rank":0,"incarnation":133,"state":"up:active","state_seq":19,"addr":"172.21.15.76:6835/172302337","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":172302337},{"type":"v1","addr":"172.21.15.76:6835","nonce":172302337}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:17.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.124+0000 7f93d0fbd700 1 -- 172.21.15.16:0/3655472904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f93b0023670 msgr2=0x7f93b0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:17.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.124+0000 7f93d0fbd700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f93b0023670 0x7f93b0025ae0 crc :-1 s=READY pgs=480 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:17.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.124+0000 7f93d0fbd700 1 -- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 msgr2=0x7f93cc12ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:17.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.124+0000 7f93d0fbd700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 0x7f93cc12ae70 secure :-1 s=READY pgs=1122 cs=0 l=1 rx=0x7f93b4010760 tx=0x7f93b40105e0).stop 2019-12-05T06:39:17.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.125+0000 7f93d0fbd700 1 -- 172.21.15.16:0/3655472904 shutdown_connections 2019-12-05T06:39:17.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.125+0000 7f93d0fbd700 1 --2- 172.21.15.16:0/3655472904 >> v2:172.21.15.76:3300/0 conn(0x7f93cc153ba0 0x7f93cc155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:17.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.125+0000 7f93d0fbd700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f93b0023670 0x7f93b0025ae0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:17.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.125+0000 7f93d0fbd700 1 --2- 172.21.15.16:0/3655472904 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f93cc12b640 0x7f93cc12ae70 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:17.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.125+0000 7f93d0fbd700 1 -- 172.21.15.16:0/3655472904 shutdown_connections 2019-12-05T06:39:17.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.125+0000 7f93d0fbd700 1 -- 172.21.15.16:0/3655472904 wait complete. 2019-12-05T06:39:17.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:17.125+0000 7f93d0fbd700 1 -- 172.21.15.16:0/3655472904 >> 172.21.15.16:0/3655472904 conn(0x7f93cc0a2850 msgr2=0x7f93cc06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:17.136 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:39:19.138 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:19.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.333+0000 7fe48b4cf700 1 Processor -- start 2019-12-05T06:39:19.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.333+0000 7fe48b4cf700 1 -- start start 2019-12-05T06:39:19.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe48b4cf700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe48408fef0 0x7fe48408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:19.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe48b4cf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4840958e0 con 0x7fe48408c6d0 2019-12-05T06:39:19.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe48b4cf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4840629e0 con 0x7fe48408d2c0 2019-12-05T06:39:19.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe48b4cf700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe48408df90 con 0x7fe48408fef0 2019-12-05T06:39:19.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe489a6c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe48408c6d0 0x7fe48408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43206/0 (socket says 172.21.15.16:43206) 2019-12-05T06:39:19.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe489a6c700 1 -- 172.21.15.16:0/2205337105 learned_addr learned my addr 172.21.15.16:0/2205337105 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:19.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe48a26d700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe48408fef0 0x7fe48408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 403495451 0 0) 0x7fe47c0014d0 con 0x7fe48408c6d0 2019-12-05T06:39:19.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe46c002e80 con 0x7fe48408c6d0 2019-12-05T06:39:19.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2234482823 0 0) 0x7fe47c0014b0 con 0x7fe48408c6d0 2019-12-05T06:39:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe48408fef0 msgr2=0x7fe48408f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe488a6a700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe48408fef0 0x7fe48408f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 >> v1:172.21.15.76:6789/0 conn(0x7fe48408d2c0 legacy=0x7fe48408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48408e620 con 0x7fe48408c6d0 2019-12-05T06:39:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fe47c001a20 con 0x7fe48408c6d0 2019-12-05T06:39:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.334+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 >> v1:172.21.15.16:6789/0 conn(0x7fe48408c6d0 legacy=0x7fe48408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe488a6a700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe46c003a80 0x7fe46c005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe488a6a700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe46c006690 0x7fe46c008b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe488a6a700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe46c009310 0x7fe46c00b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c00bff0 con 0x7fe46c009310 2019-12-05T06:39:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c0501f0 con 0x7fe46c003a80 2019-12-05T06:39:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe46c050420 con 0x7fe46c006690 2019-12-05T06:39:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe46c009310 0x7fe46c00b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe489a6c700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe46c003a80 0x7fe46c005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48926b700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe46c006690 0x7fe46c008b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 -- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe46c006690 msgr2=0x7fe46c008b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe46c006690 0x7fe46c008b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 -- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe46c003a80 msgr2=0x7fe46c005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe46c003a80 0x7fe46c005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 -- 172.21.15.16:0/2205337105 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe46c0506d0 con 0x7fe46c009310 2019-12-05T06:39:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48926b700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe46c006690 0x7fe46c008b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:39:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe46c009310 0x7fe46c00b7e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe474002a20 tx=0x7fe474002920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe46c009310 0x7fe46c00b7e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe474002a20 tx=0x7fe474002920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe46c009310 0x7fe46c00b7e0 secure :-1 s=READY pgs=1124 cs=0 l=1 rx=0x7fe474002a20 tx=0x7fe474002920).ready entity=mon.0 client_cookie=b5f5258f0b17d02b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe46c009310 0x7fe46c00b7e0 secure :-1 s=THROTTLE_DONE pgs=1124 cs=0 l=1 rx=0x7fe474002a20 tx=0x7fe474002920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe474000a70 con 0x7fe46c009310 2019-12-05T06:39:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe46c009310 0x7fe46c00b7e0 secure :-1 s=THROTTLE_DONE pgs=1124 cs=0 l=1 rx=0x7fe474002a20 tx=0x7fe474002920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe46c009310 0x7fe46c00b7e0 secure :-1 s=THROTTLE_DONE pgs=1124 cs=0 l=1 rx=0x7fe474002a20 tx=0x7fe474002920).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47400ec90 con 0x7fe46c009310 2019-12-05T06:39:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe488a6a700 1 -- 172.21.15.16:0/2205337105 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe47400ee80 con 0x7fe46c009310 2019-12-05T06:39:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48b4cf700 1 -- 172.21.15.16:0/2205337105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe46c009310 msgr2=0x7fe46c00b7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48b4cf700 1 --2- 172.21.15.16:0/2205337105 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe46c009310 0x7fe46c00b7e0 secure :-1 s=READY pgs=1124 cs=0 l=1 rx=0x7fe474002a20 tx=0x7fe474002920).stop 2019-12-05T06:39:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.335+0000 7fe48a26d700 1 -- 172.21.15.16:0/2205337105 reap_dead start 2019-12-05T06:39:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.336+0000 7fe48b4cf700 1 -- 172.21.15.16:0/2205337105 shutdown_connections 2019-12-05T06:39:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.336+0000 7fe48b4cf700 1 -- 172.21.15.16:0/2205337105 shutdown_connections 2019-12-05T06:39:19.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.336+0000 7fe48b4cf700 1 -- 172.21.15.16:0/2205337105 wait complete. 2019-12-05T06:39:19.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.336+0000 7fe48b4cf700 1 -- 172.21.15.16:0/2205337105 >> 172.21.15.16:0/2205337105 conn(0x7fe484061940 msgr2=0x7fe484096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:19.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.339+0000 7fe48b4cf700 1 Processor -- start 2019-12-05T06:39:19.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.339+0000 7fe48b4cf700 1 -- start start 2019-12-05T06:39:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48b4cf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe48412c740 0x7fe48412bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48b4cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe484154cb0 0x7fe4841570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48b4cf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe484128b20 con 0x7fe4841325e0 2019-12-05T06:39:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48b4cf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe48412a7e0 con 0x7fe484154cb0 2019-12-05T06:39:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48b4cf700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe484129db0 con 0x7fe48412c740 2019-12-05T06:39:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe484154cb0 0x7fe4841570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe484154cb0 0x7fe4841570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58052/0 (socket says 172.21.15.16:58052) 2019-12-05T06:39:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 -- 172.21.15.16:0/1992578112 learned_addr learned my addr 172.21.15.16:0/1992578112 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48926b700 1 --2- 172.21.15.16:0/1992578112 >> v2:172.21.15.76:3300/0 conn(0x7fe48412c740 0x7fe48412bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 -- 172.21.15.16:0/1992578112 >> v2:172.21.15.76:3300/0 conn(0x7fe48412c740 msgr2=0x7fe48412bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> v2:172.21.15.76:3300/0 conn(0x7fe48412c740 0x7fe48412bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 -- 172.21.15.16:0/1992578112 >> v1:172.21.15.76:6789/0 conn(0x7fe4841325e0 legacy=0x7fe48412b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 -- 172.21.15.16:0/1992578112 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48414a820 con 0x7fe484154cb0 2019-12-05T06:39:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> v2:172.21.15.16:3300/0 conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> v2:172.21.15.16:3300/0 conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=READY pgs=1125 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.340+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=THROTTLE_DONE pgs=1125 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.341+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=THROTTLE_DONE pgs=1125 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.341+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=THROTTLE_DONE pgs=1125 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.341+0000 7fe47a7fc700 1 -- 172.21.15.16:0/1992578112 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4740025c0 con 0x7fe484154cb0 2019-12-05T06:39:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.341+0000 7fe47a7fc700 1 -- 172.21.15.16:0/1992578112 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe474000c60 con 0x7fe484154cb0 2019-12-05T06:39:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.341+0000 7fe48b4cf700 1 -- 172.21.15.16:0/1992578112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48412ae70 con 0x7fe484154cb0 2019-12-05T06:39:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.341+0000 7fe47a7fc700 1 -- 172.21.15.16:0/1992578112 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe474001330 con 0x7fe484154cb0 2019-12-05T06:39:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.341+0000 7fe48b4cf700 1 -- 172.21.15.16:0/1992578112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48414a820 con 0x7fe484154cb0 2019-12-05T06:39:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.341+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=THROTTLE_DONE pgs=1125 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.342+0000 7fe47a7fc700 1 -- 172.21.15.16:0/1992578112 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe4740025c0 con 0x7fe484154cb0 2019-12-05T06:39:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.342+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=THROTTLE_DONE pgs=1125 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.342+0000 7fe47a7fc700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe464023630 0x7fe464025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:19.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.342+0000 7fe47a7fc700 1 -- 172.21.15.16:0/1992578112 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7fe474045ef0 con 0x7fe484154cb0 2019-12-05T06:39:19.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.342+0000 7fe489a6c700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe464023630 0x7fe464025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.342+0000 7fe489a6c700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe464023630 0x7fe464025ac0 crc :-1 s=READY pgs=481 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:19.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.348+0000 7fe48b4cf700 1 -- 172.21.15.16:0/1992578112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48412ae70 con 0x7fe484154cb0 2019-12-05T06:39:19.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.355+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=THROTTLE_DONE pgs=1125 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.355+0000 7fe47a7fc700 1 -- 172.21.15.16:0/1992578112 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe474001800 con 0x7fe484154cb0 2019-12-05T06:39:19.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.554+0000 7fe48b4cf700 1 -- 172.21.15.16:0/1992578112 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe48414a820 con 0x7fe484154cb0 2019-12-05T06:39:19.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.554+0000 7fe48a26d700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=THROTTLE_DONE pgs=1125 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.554+0000 7fe47a7fc700 1 -- 172.21.15.16:0/1992578112 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fe4740025c0 con 0x7fe484154cb0 2019-12-05T06:39:19.555 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 138 2019-12-05T06:39:19.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:19.555 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:18.369256+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:19.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 -- 172.21.15.16:0/1992578112 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe464023630 msgr2=0x7fe464025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe464023630 0x7fe464025ac0 crc :-1 s=READY pgs=481 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 -- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 msgr2=0x7fe4841570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 0x7fe4841570b0 secure :-1 s=READY pgs=1125 cs=0 l=1 rx=0x7fe474010460 tx=0x7fe474010290).stop 2019-12-05T06:39:19.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 -- 172.21.15.16:0/1992578112 shutdown_connections 2019-12-05T06:39:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe464023630 0x7fe464025ac0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 --2- 172.21.15.16:0/1992578112 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe484154cb0 0x7fe4841570b0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 --2- 172.21.15.16:0/1992578112 >> v2:172.21.15.76:3300/0 conn(0x7fe48412c740 0x7fe48412bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 -- 172.21.15.16:0/1992578112 shutdown_connections 2019-12-05T06:39:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 -- 172.21.15.16:0/1992578112 wait complete. 2019-12-05T06:39:19.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.557+0000 7fe48b4cf700 1 -- 172.21.15.16:0/1992578112 >> 172.21.15.16:0/1992578112 conn(0x7fe4840a2850 msgr2=0x7fe48406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:19.567 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:39:19.567 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:39:19.568 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:19.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.708+0000 7f8d75620700 1 Processor -- start 2019-12-05T06:39:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.708+0000 7f8d75620700 1 -- start start 2019-12-05T06:39:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.709+0000 7f8d75620700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7008c6d0 0x7f8d7008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:19.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.709+0000 7f8d75620700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8d7008fef0 0x7f8d7008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.709+0000 7f8d75620700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d7008ef20 0x7f8d7008f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.709+0000 7f8d75620700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d700a3f70 con 0x7f8d7008ef20 2019-12-05T06:39:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.709+0000 7f8d75620700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7008d830 con 0x7f8d7008fef0 2019-12-05T06:39:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.709+0000 7f8d75620700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d7008d540 con 0x7f8d7008c6d0 2019-12-05T06:39:19.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.709+0000 7f8d6ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d7008ef20 0x7f8d7008f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.709+0000 7f8d6ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8d7008ef20 0x7f8d7008f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58056/0 (socket says 172.21.15.16:58056) 2019-12-05T06:39:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.709+0000 7f8d6ffff700 1 -- 172.21.15.16:0/926975997 learned_addr learned my addr 172.21.15.16:0/926975997 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:19.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.709+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/926975997 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7008c6d0 0x7f8d7008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6effd700 1 --2- 172.21.15.16:0/926975997 >> v2:172.21.15.76:3300/0 conn(0x7f8d7008fef0 0x7f8d7008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 -- 172.21.15.16:0/926975997 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7008c6d0 msgr2=0x7f8d7008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 --2- 172.21.15.16:0/926975997 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7008c6d0 0x7f8d7008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:19.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 -- 172.21.15.16:0/926975997 >> v2:172.21.15.76:3300/0 conn(0x7f8d7008fef0 msgr2=0x7f8d7008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 --2- 172.21.15.16:0/926975997 >> v2:172.21.15.76:3300/0 conn(0x7f8d7008fef0 0x7f8d7008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 -- 172.21.15.16:0/926975997 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d7008e620 con 0x7f8d7008ef20 2019-12-05T06:39:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6effd700 1 --2- 172.21.15.16:0/926975997 >> v2:172.21.15.76:3300/0 conn(0x7f8d7008fef0 0x7f8d7008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:39:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 --2- 172.21.15.16:0/926975997 >> v2:172.21.15.16:3300/0 conn(0x7f8d7008ef20 0x7f8d7008f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8d6000f6b0 tx=0x7f8d6000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 --2- 172.21.15.16:0/926975997 >> v2:172.21.15.16:3300/0 conn(0x7f8d7008ef20 0x7f8d7008f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8d6000f6b0 tx=0x7f8d6000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 --2- 172.21.15.16:0/926975997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d7008ef20 0x7f8d7008f3c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rx=0x7f8d6000f6b0 tx=0x7f8d6000fa50).ready entity=mon.0 client_cookie=a262a81aec8cdaea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 --2- 172.21.15.16:0/926975997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d7008ef20 0x7f8d7008f3c0 secure :-1 s=THROTTLE_DONE pgs=1126 cs=0 l=1 rx=0x7f8d6000f6b0 tx=0x7f8d6000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 --2- 172.21.15.16:0/926975997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d7008ef20 0x7f8d7008f3c0 secure :-1 s=THROTTLE_DONE pgs=1126 cs=0 l=1 rx=0x7f8d6000f6b0 tx=0x7f8d6000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6e7fc700 1 -- 172.21.15.16:0/926975997 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d600010e0 con 0x7f8d7008ef20 2019-12-05T06:39:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6ffff700 1 --2- 172.21.15.16:0/926975997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d7008ef20 0x7f8d7008f3c0 secure :-1 s=THROTTLE_DONE pgs=1126 cs=0 l=1 rx=0x7f8d6000f6b0 tx=0x7f8d6000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6e7fc700 1 -- 172.21.15.16:0/926975997 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d60000b90 con 0x7f8d7008ef20 2019-12-05T06:39:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d6e7fc700 1 -- 172.21.15.16:0/926975997 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d600016c0 con 0x7f8d7008ef20 2019-12-05T06:39:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d75620700 1 -- 172.21.15.16:0/926975997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d7008ef20 msgr2=0x7f8d7008f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.710+0000 7f8d75620700 1 --2- 172.21.15.16:0/926975997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d7008ef20 0x7f8d7008f3c0 secure :-1 s=READY pgs=1126 cs=0 l=1 rx=0x7f8d6000f6b0 tx=0x7f8d6000fa50).stop 2019-12-05T06:39:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.711+0000 7f8d75620700 1 -- 172.21.15.16:0/926975997 shutdown_connections 2019-12-05T06:39:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.711+0000 7f8d75620700 1 --2- 172.21.15.16:0/926975997 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8d7008ef20 0x7f8d7008f3c0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.711+0000 7f8d75620700 1 --2- 172.21.15.16:0/926975997 >> v2:172.21.15.76:3300/0 conn(0x7f8d7008fef0 0x7f8d7008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.711+0000 7f8d75620700 1 --2- 172.21.15.16:0/926975997 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7008c6d0 0x7f8d7008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:19.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.711+0000 7f8d75620700 1 -- 172.21.15.16:0/926975997 shutdown_connections 2019-12-05T06:39:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.711+0000 7f8d75620700 1 -- 172.21.15.16:0/926975997 wait complete. 2019-12-05T06:39:19.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.711+0000 7f8d75620700 1 -- 172.21.15.16:0/926975997 >> 172.21.15.16:0/926975997 conn(0x7f8d70061940 msgr2=0x7f8d70096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:19.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.713+0000 7f8d75620700 1 Processor -- start 2019-12-05T06:39:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.714+0000 7f8d75620700 1 -- start start 2019-12-05T06:39:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.714+0000 7f8d75620700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.714+0000 7f8d75620700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8d700992d0 0x7f8d70098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.714+0000 7f8d75620700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d7006e300 con 0x7f8d70098d80 2019-12-05T06:39:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.714+0000 7f8d75620700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d7008ad00 con 0x7f8d700992d0 2019-12-05T06:39:19.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.714+0000 7f8d75620700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d70096c10 con 0x7f8d7006df10 2019-12-05T06:39:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6f7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6effd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8d700992d0 0x7f8d70098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6f7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:33818/0 (socket says 172.21.15.16:33818) 2019-12-05T06:39:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6f7fe700 1 -- 172.21.15.16:0/3368518396 learned_addr learned my addr 172.21.15.16:0/3368518396 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6f7fe700 1 -- 172.21.15.16:0/3368518396 >> v2:172.21.15.76:3300/0 conn(0x7f8d700992d0 msgr2=0x7f8d70098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> v2:172.21.15.76:3300/0 conn(0x7f8d700992d0 0x7f8d70098840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6f7fe700 1 -- 172.21.15.16:0/3368518396 >> v1:172.21.15.76:6789/0 conn(0x7f8d70098d80 legacy=0x7f8d70088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6f7fe700 1 -- 172.21.15.16:0/3368518396 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d70097a00 con 0x7f8d7006df10 2019-12-05T06:39:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6effd700 1 --2- 172.21.15.16:0/3368518396 >> v2:172.21.15.76:3300/0 conn(0x7f8d700992d0 0x7f8d70098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.715+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=READY pgs=258 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:19.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.716+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.716+0000 7f8d67fff700 1 -- 172.21.15.16:0/3368518396 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d68000cc0 con 0x7f8d7006df10 2019-12-05T06:39:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.716+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.716+0000 7f8d75620700 1 -- 172.21.15.16:0/3368518396 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d70097a00 con 0x7f8d7006df10 2019-12-05T06:39:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.716+0000 7f8d67fff700 1 -- 172.21.15.16:0/3368518396 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d68000cc0 con 0x7f8d7006df10 2019-12-05T06:39:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.716+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.716+0000 7f8d67fff700 1 -- 172.21.15.16:0/3368518396 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d68007970 con 0x7f8d7006df10 2019-12-05T06:39:19.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.716+0000 7f8d75620700 1 -- 172.21.15.16:0/3368518396 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d70097a00 con 0x7f8d7006df10 2019-12-05T06:39:19.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.716+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.717+0000 7f8d67fff700 1 -- 172.21.15.16:0/3368518396 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8d68001220 con 0x7f8d7006df10 2019-12-05T06:39:19.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.717+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.717+0000 7f8d67fff700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8d48023630 0x7f8d48025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:19.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.717+0000 7f8d67fff700 1 -- 172.21.15.16:0/3368518396 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f8d68046100 con 0x7f8d7006df10 2019-12-05T06:39:19.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.717+0000 7f8d6effd700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8d48023630 0x7f8d48025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:19.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.718+0000 7f8d6effd700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8d48023630 0x7f8d48025ac0 crc :-1 s=READY pgs=482 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:19.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.724+0000 7f8d75620700 1 -- 172.21.15.16:0/3368518396 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d70097a00 con 0x7f8d7006df10 2019-12-05T06:39:19.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.731+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.731+0000 7f8d67fff700 1 -- 172.21.15.16:0/3368518396 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8d68001010 con 0x7f8d7006df10 2019-12-05T06:39:19.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.929+0000 7f8d75620700 1 -- 172.21.15.16:0/3368518396 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8d70097a00 con 0x7f8d7006df10 2019-12-05T06:39:19.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.931+0000 7f8d6f7fe700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=THROTTLE_DONE pgs=258 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:19.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.931+0000 7f8d67fff700 1 -- 172.21.15.16:0/3368518396 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 138 v138) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f8d68001220 con 0x7f8d7006df10 2019-12-05T06:39:19.932 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 138 2019-12-05T06:39:19.932 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:19.933 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":138,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":138,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:18.369256+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:19.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.935+0000 7f8d75620700 1 -- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8d48023630 msgr2=0x7f8d48025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.935+0000 7f8d75620700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8d48023630 0x7f8d48025ac0 crc :-1 s=READY pgs=482 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.935+0000 7f8d75620700 1 -- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 msgr2=0x7f8d70098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:19.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.935+0000 7f8d75620700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 secure :-1 s=READY pgs=258 cs=0 l=1 rx=0x7f8d68010780 tx=0x7f8d68010b20).stop 2019-12-05T06:39:19.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.935+0000 7f8d75620700 1 -- 172.21.15.16:0/3368518396 shutdown_connections 2019-12-05T06:39:19.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.935+0000 7f8d75620700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8d48023630 0x7f8d48025ac0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.935+0000 7f8d75620700 1 --2- 172.21.15.16:0/3368518396 >> v2:172.21.15.76:3300/0 conn(0x7f8d700992d0 0x7f8d70098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.935+0000 7f8d75620700 1 --2- 172.21.15.16:0/3368518396 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8d7006df10 0x7f8d70098300 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:19.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.936+0000 7f8d75620700 1 -- 172.21.15.16:0/3368518396 shutdown_connections 2019-12-05T06:39:19.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.936+0000 7f8d75620700 1 -- 172.21.15.16:0/3368518396 wait complete. 2019-12-05T06:39:19.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:19.936+0000 7f8d75620700 1 -- 172.21.15.16:0/3368518396 >> 172.21.15.16:0/3368518396 conn(0x7f8d700714f0 msgr2=0x7f8d7009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:19.945 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 138, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 138, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7011": { "addr": "172.21.15.16:6833/3506755338", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 3506755338, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 3506755338, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7011, "incarnation": 138, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 71, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:39:18.369256+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7011 } } } ], "standbys": [] } 2019-12-05T06:39:21.949 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:22.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.197+0000 7f589514d700 1 Processor -- start 2019-12-05T06:39:22.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.197+0000 7f589514d700 1 -- start start 2019-12-05T06:39:22.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.199+0000 7f589514d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f589008c6d0 0x7f589008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:22.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.199+0000 7f589514d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:22.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.199+0000 7f589514d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589008fb70 con 0x7f589008fef0 2019-12-05T06:39:22.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.199+0000 7f589514d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f589008df90 con 0x7f589008c6d0 2019-12-05T06:39:22.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.199+0000 7f589514d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58900a3f70 con 0x7f589008f720 2019-12-05T06:39:22.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.199+0000 7f588f59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f589008c6d0 0x7f589008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.199+0000 7f589494c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.200+0000 7f588ed9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f589008fef0 0x7f589008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:36310/0 (socket says 172.21.15.16:36310) 2019-12-05T06:39:22.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.200+0000 7f589494c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:33832/0 (socket says 172.21.15.16:33832) 2019-12-05T06:39:22.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.200+0000 7f588ed9d700 1 -- 172.21.15.16:0/3637597557 learned_addr learned my addr 172.21.15.16:0/3637597557 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:22.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.200+0000 7f588e59c700 1 -- 172.21.15.16:0/3637597557 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1375270648 0 0) 0x7f5884001190 con 0x7f589008fef0 2019-12-05T06:39:22.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.200+0000 7f588e59c700 1 -- 172.21.15.16:0/3637597557 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5878002e80 con 0x7f589008fef0 2019-12-05T06:39:22.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.201+0000 7f589494c700 1 -- 172.21.15.16:0/3637597557 >> v2:172.21.15.16:3300/0 conn(0x7f589008c6d0 msgr2=0x7f589008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:22.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.201+0000 7f589494c700 1 --2- 172.21.15.16:0/3637597557 >> v2:172.21.15.16:3300/0 conn(0x7f589008c6d0 0x7f589008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:22.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.201+0000 7f589494c700 1 -- 172.21.15.16:0/3637597557 >> v1:172.21.15.76:6789/0 conn(0x7f589008fef0 legacy=0x7f589008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.201+0000 7f589494c700 1 -- 172.21.15.16:0/3637597557 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f589008d540 con 0x7f589008f720 2019-12-05T06:39:22.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.201+0000 7f589494c700 1 --2- 172.21.15.16:0/3637597557 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f588800b300 tx=0x7f588800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.201+0000 7f589494c700 1 --2- 172.21.15.16:0/3637597557 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f588800b300 tx=0x7f588800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.201+0000 7f589494c700 1 --2- 172.21.15.16:0/3637597557 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 secure :-1 s=READY pgs=259 cs=0 l=1 rx=0x7f588800b300 tx=0x7f588800bb10).ready entity=mon.2 client_cookie=6870a860d6bbc0c4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:22.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.202+0000 7f589494c700 1 --2- 172.21.15.16:0/3637597557 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 secure :-1 s=THROTTLE_DONE pgs=259 cs=0 l=1 rx=0x7f588800b300 tx=0x7f588800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.202+0000 7f588e59c700 1 -- 172.21.15.16:0/3637597557 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5888000f40 con 0x7f589008f720 2019-12-05T06:39:22.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.202+0000 7f589494c700 1 --2- 172.21.15.16:0/3637597557 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 secure :-1 s=THROTTLE_DONE pgs=259 cs=0 l=1 rx=0x7f588800b300 tx=0x7f588800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.202+0000 7f589494c700 1 --2- 172.21.15.16:0/3637597557 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 secure :-1 s=THROTTLE_DONE pgs=259 cs=0 l=1 rx=0x7f588800b300 tx=0x7f588800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.202+0000 7f588e59c700 1 -- 172.21.15.16:0/3637597557 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5888009b70 con 0x7f589008f720 2019-12-05T06:39:22.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.202+0000 7f588e59c700 1 -- 172.21.15.16:0/3637597557 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f588800a240 con 0x7f589008f720 2019-12-05T06:39:22.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.202+0000 7f589514d700 1 -- 172.21.15.16:0/3637597557 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 msgr2=0x7f589008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.202+0000 7f589514d700 1 --2- 172.21.15.16:0/3637597557 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 secure :-1 s=READY pgs=259 cs=0 l=1 rx=0x7f588800b300 tx=0x7f588800bb10).stop 2019-12-05T06:39:22.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.206+0000 7f589514d700 1 -- 172.21.15.16:0/3637597557 shutdown_connections 2019-12-05T06:39:22.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.206+0000 7f589514d700 1 --2- 172.21.15.16:0/3637597557 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589008f720 0x7f589008d000 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.206+0000 7f589514d700 1 --2- 172.21.15.16:0/3637597557 >> v2:172.21.15.16:3300/0 conn(0x7f589008c6d0 0x7f589008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:22.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.206+0000 7f589514d700 1 -- 172.21.15.16:0/3637597557 shutdown_connections 2019-12-05T06:39:22.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.206+0000 7f589514d700 1 -- 172.21.15.16:0/3637597557 wait complete. 2019-12-05T06:39:22.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.206+0000 7f589514d700 1 -- 172.21.15.16:0/3637597557 >> 172.21.15.16:0/3637597557 conn(0x7f5890061940 msgr2=0x7f5890096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.210+0000 7f589514d700 1 Processor -- start 2019-12-05T06:39:22.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.210+0000 7f589514d700 1 -- start start 2019-12-05T06:39:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.211+0000 7f589514d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:22.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.211+0000 7f589514d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5890156d50 0x7f5890159150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:22.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.211+0000 7f589514d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f589008a1b0 con 0x7f58900a1fe0 2019-12-05T06:39:22.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.211+0000 7f589514d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f589009f650 con 0x7f5890156d50 2019-12-05T06:39:22.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.211+0000 7f589514d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f589009f920 con 0x7f589006df30 2019-12-05T06:39:22.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.212+0000 7f588f59e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:22.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.212+0000 7f589494c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5890156d50 0x7f5890159150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.212+0000 7f588f59e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:33834/0 (socket says 172.21.15.16:33834) 2019-12-05T06:39:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.212+0000 7f589494c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5890156d50 0x7f5890159150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:44170/0 (socket says 172.21.15.16:44170) 2019-12-05T06:39:22.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.212+0000 7f588f59e700 1 -- 172.21.15.16:0/1586498463 learned_addr learned my addr 172.21.15.16:0/1586498463 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:22.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.212+0000 7f588f59e700 1 -- 172.21.15.16:0/1586498463 >> v2:172.21.15.76:3300/0 conn(0x7f5890156d50 msgr2=0x7f5890159150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.212+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> v2:172.21.15.76:3300/0 conn(0x7f5890156d50 0x7f5890159150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.212+0000 7f588f59e700 1 -- 172.21.15.16:0/1586498463 >> v1:172.21.15.76:6789/0 conn(0x7f58900a1fe0 legacy=0x7f58900a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.212+0000 7f588f59e700 1 -- 172.21.15.16:0/1586498463 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58900a0710 con 0x7f589006df30 2019-12-05T06:39:22.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f589494c700 1 --2- 172.21.15.16:0/1586498463 >> v2:172.21.15.76:3300/0 conn(0x7f5890156d50 0x7f5890159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:22.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=READY pgs=260 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:22.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f5877fff700 1 -- 172.21.15.16:0/1586498463 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5880000940 con 0x7f589006df30 2019-12-05T06:39:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f5877fff700 1 -- 172.21.15.16:0/1586498463 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5880000ea0 con 0x7f589006df30 2019-12-05T06:39:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f5877fff700 1 -- 172.21.15.16:0/1586498463 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5880001470 con 0x7f589006df30 2019-12-05T06:39:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f589514d700 1 -- 172.21.15.16:0/1586498463 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58900a0710 con 0x7f589006df30 2019-12-05T06:39:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.213+0000 7f589514d700 1 -- 172.21.15.16:0/1586498463 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f589008a1b0 con 0x7f589006df30 2019-12-05T06:39:22.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.215+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.216+0000 7f5877fff700 1 -- 172.21.15.16:0/1586498463 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5880000c90 con 0x7f589006df30 2019-12-05T06:39:22.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.216+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.216+0000 7f5877fff700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5870023630 0x7f5870025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:22.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.217+0000 7f5877fff700 1 -- 172.21.15.16:0/1586498463 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f5880045dd0 con 0x7f589006df30 2019-12-05T06:39:22.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.217+0000 7f588ed9d700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5870023630 0x7f5870025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:22.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.217+0000 7f588ed9d700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5870023630 0x7f5870025ac0 crc :-1 s=READY pgs=483 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:22.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.227+0000 7f589514d700 1 -- 172.21.15.16:0/1586498463 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58900a0710 con 0x7f589006df30 2019-12-05T06:39:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.235+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.235+0000 7f5877fff700 1 -- 172.21.15.16:0/1586498463 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f588002adf0 con 0x7f589006df30 2019-12-05T06:39:22.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.465+0000 7f589514d700 1 -- 172.21.15.16:0/1586498463 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58900a0710 con 0x7f589006df30 2019-12-05T06:39:22.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.466+0000 7f588f59e700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=THROTTLE_DONE pgs=260 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:22.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.466+0000 7f5877fff700 1 -- 172.21.15.16:0/1586498463 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 140 v140) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f58800017f0 con 0x7f589006df30 2019-12-05T06:39:22.491 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 140 2019-12-05T06:39:22.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.469+0000 7f589514d700 1 -- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5870023630 msgr2=0x7f5870025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.469+0000 7f589514d700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5870023630 0x7f5870025ac0 crc :-1 s=READY pgs=483 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.469+0000 7f589514d700 1 -- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 msgr2=0x7f58900a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:22.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.469+0000 7f589514d700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 secure :-1 s=READY pgs=260 cs=0 l=1 rx=0x7f5880010400 tx=0x7f58800107a0).stop 2019-12-05T06:39:22.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.470+0000 7f589514d700 1 -- 172.21.15.16:0/1586498463 shutdown_connections 2019-12-05T06:39:22.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.470+0000 7f589514d700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5870023630 0x7f5870025ac0 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.493 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:22.494 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":140,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":140,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:22.186623+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:rejoin","state_seq":33,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:22.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.470+0000 7f589514d700 1 --2- 172.21.15.16:0/1586498463 >> v2:172.21.15.76:3300/0 conn(0x7f5890156d50 0x7f5890159150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.470+0000 7f589514d700 1 --2- 172.21.15.16:0/1586498463 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f589006df30 0x7f58900a1010 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:22.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.470+0000 7f589514d700 1 -- 172.21.15.16:0/1586498463 shutdown_connections 2019-12-05T06:39:22.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.470+0000 7f589514d700 1 -- 172.21.15.16:0/1586498463 wait complete. 2019-12-05T06:39:22.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:22.470+0000 7f589514d700 1 -- 172.21.15.16:0/1586498463 >> 172.21.15.16:0/1586498463 conn(0x7f589012f780 msgr2=0x7f5890096b90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:22.496 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:39:24.498 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.655+0000 7f3c28fe1700 1 Processor -- start 2019-12-05T06:39:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.655+0000 7f3c28fe1700 1 -- start start 2019-12-05T06:39:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.657+0000 7f3c28fe1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c2408d2c0 0x7f3c2408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.657+0000 7f3c28fe1700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c2408fef0 0x7f3c2408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:24.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.657+0000 7f3c28fe1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c240958e0 con 0x7f3c2408c6d0 2019-12-05T06:39:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.657+0000 7f3c28fe1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c2408df90 con 0x7f3c2408d2c0 2019-12-05T06:39:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.657+0000 7f3c28fe1700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c240a3f70 con 0x7f3c2408fef0 2019-12-05T06:39:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.657+0000 7f3c2259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c2408d2c0 0x7f3c2408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.657+0000 7f3c22d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3c2408c6d0 0x7f3c2408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43270/0 (socket says 172.21.15.16:43270) 2019-12-05T06:39:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.657+0000 7f3c22d9d700 1 -- 172.21.15.16:0/2235505221 learned_addr learned my addr 172.21.15.16:0/2235505221 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:24.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.657+0000 7f3c2359e700 1 --2- 172.21.15.16:0/2235505221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c2408fef0 0x7f3c2408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:24.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c2259c700 1 -- 172.21.15.16:0/2235505221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c2408fef0 msgr2=0x7f3c2408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:24.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c2259c700 1 --2- 172.21.15.16:0/2235505221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c2408fef0 0x7f3c2408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:24.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c2259c700 1 -- 172.21.15.16:0/2235505221 >> v1:172.21.15.16:6789/0 conn(0x7f3c2408c6d0 legacy=0x7f3c2408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c2259c700 1 -- 172.21.15.16:0/2235505221 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2408e620 con 0x7f3c2408d2c0 2019-12-05T06:39:24.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c21d9b700 1 -- 172.21.15.16:0/2235505221 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559316653 0 0) 0x7f3c1c001150 con 0x7f3c2408c6d0 2019-12-05T06:39:24.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c2259c700 1 --2- 172.21.15.16:0/2235505221 >> v2:172.21.15.16:3300/0 conn(0x7f3c2408d2c0 0x7f3c2408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3c0c00b2d0 tx=0x7f3c0c00ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c2259c700 1 --2- 172.21.15.16:0/2235505221 >> v2:172.21.15.16:3300/0 conn(0x7f3c2408d2c0 0x7f3c2408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3c0c00b2d0 tx=0x7f3c0c00ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c2259c700 1 --2- 172.21.15.16:0/2235505221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c2408d2c0 0x7f3c2408ef20 secure :-1 s=READY pgs=1128 cs=0 l=1 rx=0x7f3c0c00b2d0 tx=0x7f3c0c00ba60).ready entity=mon.0 client_cookie=6ec18c110ea0e9c4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:24.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c2259c700 1 --2- 172.21.15.16:0/2235505221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c2408d2c0 0x7f3c2408ef20 secure :-1 s=THROTTLE_DONE pgs=1128 cs=0 l=1 rx=0x7f3c0c00b2d0 tx=0x7f3c0c00ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c2259c700 1 --2- 172.21.15.16:0/2235505221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c2408d2c0 0x7f3c2408ef20 secure :-1 s=THROTTLE_DONE pgs=1128 cs=0 l=1 rx=0x7f3c0c00b2d0 tx=0x7f3c0c00ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c21d9b700 1 -- 172.21.15.16:0/2235505221 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c0c007590 con 0x7f3c2408d2c0 2019-12-05T06:39:24.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c2259c700 1 --2- 172.21.15.16:0/2235505221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c2408d2c0 0x7f3c2408ef20 secure :-1 s=THROTTLE_DONE pgs=1128 cs=0 l=1 rx=0x7f3c0c00b2d0 tx=0x7f3c0c00ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c21d9b700 1 -- 172.21.15.16:0/2235505221 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0c009d30 con 0x7f3c2408d2c0 2019-12-05T06:39:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c21d9b700 1 -- 172.21.15.16:0/2235505221 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c0c00a400 con 0x7f3c2408d2c0 2019-12-05T06:39:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2235505221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c2408d2c0 msgr2=0x7f3c2408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.658+0000 7f3c28fe1700 1 --2- 172.21.15.16:0/2235505221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c2408d2c0 0x7f3c2408ef20 secure :-1 s=READY pgs=1128 cs=0 l=1 rx=0x7f3c0c00b2d0 tx=0x7f3c0c00ba60).stop 2019-12-05T06:39:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.660+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2235505221 shutdown_connections 2019-12-05T06:39:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.660+0000 7f3c28fe1700 1 --2- 172.21.15.16:0/2235505221 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c2408d2c0 0x7f3c2408ef20 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.660+0000 7f3c28fe1700 1 --2- 172.21.15.16:0/2235505221 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c2408fef0 0x7f3c2408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.661+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2235505221 shutdown_connections 2019-12-05T06:39:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.661+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2235505221 wait complete. 2019-12-05T06:39:24.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.661+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2235505221 >> 172.21.15.16:0/2235505221 conn(0x7f3c24061940 msgr2=0x7f3c24096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:24.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.664+0000 7f3c28fe1700 1 Processor -- start 2019-12-05T06:39:24.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.664+0000 7f3c28fe1700 1 -- start start 2019-12-05T06:39:24.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.665+0000 7f3c28fe1700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:24.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.665+0000 7f3c28fe1700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3c24084f30 0x7f3c2409b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:24.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.665+0000 7f3c28fe1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c24084b90 con 0x7f3c24097280 2019-12-05T06:39:24.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.665+0000 7f3c28fe1700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c24099a20 con 0x7f3c24084f30 2019-12-05T06:39:24.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.665+0000 7f3c28fe1700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c24099cf0 con 0x7f3c24096e90 2019-12-05T06:39:24.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.665+0000 7f3c22d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:24.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.665+0000 7f3c2259c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3c24084f30 0x7f3c2409b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:24.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.665+0000 7f3c22d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:33864/0 (socket says 172.21.15.16:33864) 2019-12-05T06:39:24.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c2259c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3c24084f30 0x7f3c2409b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:44198/0 (socket says 172.21.15.16:44198) 2019-12-05T06:39:24.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c2359e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3c24097280 0x7f3c24097700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:36346/0 (socket says 172.21.15.16:36346) 2019-12-05T06:39:24.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c22d9d700 1 -- 172.21.15.16:0/2044858005 learned_addr learned my addr 172.21.15.16:0/2044858005 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:24.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c22d9d700 1 -- 172.21.15.16:0/2044858005 >> v2:172.21.15.76:3300/0 conn(0x7f3c24084f30 msgr2=0x7f3c2409b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> v2:172.21.15.76:3300/0 conn(0x7f3c24084f30 0x7f3c2409b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:24.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c22d9d700 1 -- 172.21.15.16:0/2044858005 >> v1:172.21.15.76:6789/0 conn(0x7f3c24097280 legacy=0x7f3c24097700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c22d9d700 1 -- 172.21.15.16:0/2044858005 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c2409aae0 con 0x7f3c24096e90 2019-12-05T06:39:24.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c2259c700 1 --2- 172.21.15.16:0/2044858005 >> v2:172.21.15.76:3300/0 conn(0x7f3c24084f30 0x7f3c2409b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:24.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c1b7fe700 1 -- 172.21.15.16:0/2044858005 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3192790089 0 0) 0x7f3c14001080 con 0x7f3c24097280 2019-12-05T06:39:24.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.666+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.667+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=READY pgs=261 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:24.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.667+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.667+0000 7f3c1b7fe700 1 -- 172.21.15.16:0/2044858005 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c1c000ed0 con 0x7f3c24096e90 2019-12-05T06:39:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.667+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.667+0000 7f3c1b7fe700 1 -- 172.21.15.16:0/2044858005 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c1c004430 con 0x7f3c24096e90 2019-12-05T06:39:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.667+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2044858005 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c2409aae0 con 0x7f3c24096e90 2019-12-05T06:39:24.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.667+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.667+0000 7f3c1b7fe700 1 -- 172.21.15.16:0/2044858005 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c1c004620 con 0x7f3c24096e90 2019-12-05T06:39:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.667+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2044858005 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c2409aae0 con 0x7f3c24096e90 2019-12-05T06:39:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.668+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.669+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.669+0000 7f3c1b7fe700 1 -- 172.21.15.16:0/2044858005 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3c1c004240 con 0x7f3c24096e90 2019-12-05T06:39:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.669+0000 7f3c1b7fe700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3c08023630 0x7f3c08025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.669+0000 7f3c1b7fe700 1 -- 172.21.15.16:0/2044858005 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f3c1c045dc0 con 0x7f3c24096e90 2019-12-05T06:39:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.669+0000 7f3c2259c700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3c08023630 0x7f3c08025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.670+0000 7f3c2259c700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3c08023630 0x7f3c08025ac0 crc :-1 s=READY pgs=484 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.672+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2044858005 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c2409aae0 con 0x7f3c24096e90 2019-12-05T06:39:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.681+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.681+0000 7f3c1b7fe700 1 -- 172.21.15.16:0/2044858005 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3c1c0012e0 con 0x7f3c24096e90 2019-12-05T06:39:24.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.949+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2044858005 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c2409aae0 con 0x7f3c24096e90 2019-12-05T06:39:24.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.951+0000 7f3c22d9d700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=THROTTLE_DONE pgs=261 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:24.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.951+0000 7f3c1b7fe700 1 -- 172.21.15.16:0/2044858005 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f3c1c004240 con 0x7f3c24096e90 2019-12-05T06:39:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 141 2019-12-05T06:39:24.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3c08023630 msgr2=0x7f3c08025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3c08023630 0x7f3c08025ac0 crc :-1 s=READY pgs=484 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 msgr2=0x7f3c2409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 secure :-1 s=READY pgs=261 cs=0 l=1 rx=0x7f3c1c010430 tx=0x7f3c1c0107d0).stop 2019-12-05T06:39:24.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2044858005 shutdown_connections 2019-12-05T06:39:24.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3c08023630 0x7f3c08025ac0 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 --2- 172.21.15.16:0/2044858005 >> v2:172.21.15.76:3300/0 conn(0x7f3c24084f30 0x7f3c2409b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 --2- 172.21.15.16:0/2044858005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3c24096e90 0x7f3c2409b3e0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:24.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2044858005 shutdown_connections 2019-12-05T06:39:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2044858005 wait complete. 2019-12-05T06:39:24.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:24.955+0000 7f3c28fe1700 1 -- 172.21.15.16:0/2044858005 >> 172.21.15.16:0/2044858005 conn(0x7f3c240714f0 msgr2=0x7f3c2409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:24.967 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:24.968 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:24.974 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6940 2019-12-05T06:39:24.974 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.b 2019-12-05T06:39:25.979 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:39:25.979 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:39:25.979 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:39:25.987 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:39:25.987 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:26.181 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:39:26.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.242+0000 7f05bb3b7700 1 Processor -- start 2019-12-05T06:39:26.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.242+0000 7f05bb3b7700 1 -- start start 2019-12-05T06:39:26.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.244+0000 7f05bb3b7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f05b408d2c0 0x7f05b408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:26.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.244+0000 7f05bb3b7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b40958e0 con 0x7f05b408c6d0 2019-12-05T06:39:26.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.244+0000 7f05bb3b7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b40629e0 con 0x7f05b408fef0 2019-12-05T06:39:26.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.244+0000 7f05bb3b7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b408df90 con 0x7f05b408d2c0 2019-12-05T06:39:26.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.244+0000 7f05b9954700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f05b408c6d0 0x7f05b408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43288/0 (socket says 172.21.15.16:43288) 2019-12-05T06:39:26.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.244+0000 7f05b9954700 1 -- 172.21.15.16:0/2510002701 learned_addr learned my addr 172.21.15.16:0/2510002701 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:26.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.244+0000 7f05b9153700 1 --2- 172.21.15.16:0/2510002701 >> v2:172.21.15.76:3300/0 conn(0x7f05b408d2c0 0x7f05b408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:26.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.245+0000 7f05b8952700 1 -- 172.21.15.16:0/2510002701 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005679636 0 0) 0x7f05ac001100 con 0x7f05b408c6d0 2019-12-05T06:39:26.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.245+0000 7f05b8952700 1 -- 172.21.15.16:0/2510002701 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f059c002e80 con 0x7f05b408c6d0 2019-12-05T06:39:26.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.245+0000 7f05b8952700 1 -- 172.21.15.16:0/2510002701 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003491558 0 0) 0x7f05a4001080 con 0x7f05b408fef0 2019-12-05T06:39:26.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.245+0000 7f05b8952700 1 -- 172.21.15.16:0/2510002701 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f059c0030b0 con 0x7f05b408fef0 2019-12-05T06:39:26.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.245+0000 7f05b9153700 1 -- 172.21.15.16:0/2510002701 >> v1:172.21.15.76:6789/0 conn(0x7f05b408fef0 legacy=0x7f05b408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.245+0000 7f05b9153700 1 -- 172.21.15.16:0/2510002701 >> v1:172.21.15.16:6789/0 conn(0x7f05b408c6d0 legacy=0x7f05b408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.245+0000 7f05b9153700 1 -- 172.21.15.16:0/2510002701 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b408e620 con 0x7f05b408d2c0 2019-12-05T06:39:26.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.245+0000 7f05b9153700 1 --2- 172.21.15.16:0/2510002701 >> v2:172.21.15.76:3300/0 conn(0x7f05b408d2c0 0x7f05b408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f05a800b2d0 tx=0x7f05a800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.245+0000 7f05b9153700 1 --2- 172.21.15.16:0/2510002701 >> v2:172.21.15.76:3300/0 conn(0x7f05b408d2c0 0x7f05b408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f05a800b2d0 tx=0x7f05a800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.245+0000 7f05b9153700 1 --2- 172.21.15.16:0/2510002701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f05b408d2c0 0x7f05b408ef20 secure :-1 s=READY pgs=787 cs=0 l=1 rx=0x7f05a800b2d0 tx=0x7f05a800bae0).ready entity=mon.1 client_cookie=a617cb993012dcc9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.246+0000 7f05b9153700 1 --2- 172.21.15.16:0/2510002701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f05b408d2c0 0x7f05b408ef20 secure :-1 s=THROTTLE_DONE pgs=787 cs=0 l=1 rx=0x7f05a800b2d0 tx=0x7f05a800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.246+0000 7f05b8952700 1 -- 172.21.15.16:0/2510002701 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05a8000ce0 con 0x7f05b408d2c0 2019-12-05T06:39:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.246+0000 7f05b9153700 1 --2- 172.21.15.16:0/2510002701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f05b408d2c0 0x7f05b408ef20 secure :-1 s=THROTTLE_DONE pgs=787 cs=0 l=1 rx=0x7f05a800b2d0 tx=0x7f05a800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.246+0000 7f05b9153700 1 --2- 172.21.15.16:0/2510002701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f05b408d2c0 0x7f05b408ef20 secure :-1 s=THROTTLE_DONE pgs=787 cs=0 l=1 rx=0x7f05a800b2d0 tx=0x7f05a800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.246+0000 7f05b8952700 1 -- 172.21.15.16:0/2510002701 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a8007590 con 0x7f05b408d2c0 2019-12-05T06:39:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.246+0000 7f05b8952700 1 -- 172.21.15.16:0/2510002701 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05a8009f20 con 0x7f05b408d2c0 2019-12-05T06:39:26.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.246+0000 7f05bb3b7700 1 -- 172.21.15.16:0/2510002701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f05b408d2c0 msgr2=0x7f05b408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.246+0000 7f05bb3b7700 1 --2- 172.21.15.16:0/2510002701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f05b408d2c0 0x7f05b408ef20 secure :-1 s=READY pgs=787 cs=0 l=1 rx=0x7f05a800b2d0 tx=0x7f05a800bae0).stop 2019-12-05T06:39:26.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.247+0000 7f05bb3b7700 1 -- 172.21.15.16:0/2510002701 shutdown_connections 2019-12-05T06:39:26.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.247+0000 7f05bb3b7700 1 --2- 172.21.15.16:0/2510002701 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f05b408d2c0 0x7f05b408ef20 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.247+0000 7f05bb3b7700 1 -- 172.21.15.16:0/2510002701 shutdown_connections 2019-12-05T06:39:26.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.247+0000 7f05bb3b7700 1 -- 172.21.15.16:0/2510002701 wait complete. 2019-12-05T06:39:26.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.247+0000 7f05bb3b7700 1 -- 172.21.15.16:0/2510002701 >> 172.21.15.16:0/2510002701 conn(0x7f05b4061940 msgr2=0x7f05b4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:26.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.252+0000 7f05bb3b7700 1 Processor -- start 2019-12-05T06:39:26.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.252+0000 7f05bb3b7700 1 -- start start 2019-12-05T06:39:26.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05bb3b7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f05b41302f0 0x7f05b412a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:26.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05bb3b7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f05b412b640 0x7f05b412abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05bb3b7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05b4131bf0 con 0x7f05b412b0f0 2019-12-05T06:39:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05bb3b7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05b4128cb0 con 0x7f05b412b640 2019-12-05T06:39:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05bb3b7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05b4128f80 con 0x7f05b41302f0 2019-12-05T06:39:26.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05b9153700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f05b412b640 0x7f05b412abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05b9153700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f05b412b640 0x7f05b412abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58126/0 (socket says 172.21.15.16:58126) 2019-12-05T06:39:26.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05b9153700 1 -- 172.21.15.16:0/3114091114 learned_addr learned my addr 172.21.15.16:0/3114091114 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:26.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05b9954700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f05b41302f0 0x7f05b412a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05b9153700 1 -- 172.21.15.16:0/3114091114 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f05b41302f0 msgr2=0x7f05b412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f05b41302f0 0x7f05b412a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05b9153700 1 -- 172.21.15.16:0/3114091114 >> v1:172.21.15.76:6789/0 conn(0x7f05b412b0f0 legacy=0x7f05b41306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.254+0000 7f05b9153700 1 -- 172.21.15.16:0/3114091114 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05b4129d70 con 0x7f05b412b640 2019-12-05T06:39:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.255+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> v2:172.21.15.16:3300/0 conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.255+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> v2:172.21.15.16:3300/0 conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.255+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:26.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.255+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=THROTTLE_DONE pgs=1132 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.255+0000 7f05b27fc700 1 -- 172.21.15.16:0/3114091114 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05a8000cc0 con 0x7f05b412b640 2019-12-05T06:39:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.255+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=THROTTLE_DONE pgs=1132 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.255+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=THROTTLE_DONE pgs=1132 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.255+0000 7f05b27fc700 1 -- 172.21.15.16:0/3114091114 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f05a8009d00 con 0x7f05b412b640 2019-12-05T06:39:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.255+0000 7f05bb3b7700 1 -- 172.21.15.16:0/3114091114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05b4129d70 con 0x7f05b412b640 2019-12-05T06:39:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.255+0000 7f05b27fc700 1 -- 172.21.15.16:0/3114091114 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05a800a450 con 0x7f05b412b640 2019-12-05T06:39:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.256+0000 7f05bb3b7700 1 -- 172.21.15.16:0/3114091114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05b4129d70 con 0x7f05b412b640 2019-12-05T06:39:26.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.256+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=THROTTLE_DONE pgs=1132 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.257+0000 7f05b27fc700 1 -- 172.21.15.16:0/3114091114 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f05a8009ef0 con 0x7f05b412b640 2019-12-05T06:39:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.257+0000 7f05b27fc700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0594023630 0x7f0594025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.258+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=THROTTLE_DONE pgs=1132 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.258+0000 7f05b9954700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0594023630 0x7f0594025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:26.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.258+0000 7f05b27fc700 1 -- 172.21.15.16:0/3114091114 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f05a8009ed0 con 0x7f05b412b640 2019-12-05T06:39:26.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.258+0000 7f05b9954700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0594023630 0x7f0594025ac0 crc :-1 s=READY pgs=485 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:26.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.273+0000 7f05bb3b7700 1 -- 172.21.15.16:0/3114091114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05b4129d70 con 0x7f05b412b640 2019-12-05T06:39:26.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.280+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=THROTTLE_DONE pgs=1132 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.280+0000 7f05b27fc700 1 -- 172.21.15.16:0/3114091114 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f05a8009900 con 0x7f05b412b640 2019-12-05T06:39:26.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.504+0000 7f05bb3b7700 1 -- 172.21.15.16:0/3114091114 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05b4129d70 con 0x7f05b412b640 2019-12-05T06:39:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.505+0000 7f05b9153700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=THROTTLE_DONE pgs=1132 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:26.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.505+0000 7f05b27fc700 1 -- 172.21.15.16:0/3114091114 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f05a802cf60 con 0x7f05b412b640 2019-12-05T06:39:26.507 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 141 2019-12-05T06:39:26.507 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:26.507 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:26.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 -- 172.21.15.16:0/3114091114 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0594023630 msgr2=0x7f0594025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0594023630 0x7f0594025ac0 crc :-1 s=READY pgs=485 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 -- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 msgr2=0x7f05b412abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:26.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 0x7f05b412abb0 secure :-1 s=READY pgs=1132 cs=0 l=1 rx=0x7f05a80107f0 tx=0x7f05a80105f0).stop 2019-12-05T06:39:26.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 -- 172.21.15.16:0/3114091114 shutdown_connections 2019-12-05T06:39:26.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0594023630 0x7f0594025ac0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f05b412b640 0x7f05b412abb0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 --2- 172.21.15.16:0/3114091114 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f05b41302f0 0x7f05b412a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:26.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 -- 172.21.15.16:0/3114091114 shutdown_connections 2019-12-05T06:39:26.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 -- 172.21.15.16:0/3114091114 wait complete. 2019-12-05T06:39:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:26.508+0000 7f05bb3b7700 1 -- 172.21.15.16:0/3114091114 >> 172.21.15.16:0/3114091114 conn(0x7f05b40a2850 msgr2=0x7f05b406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:26.522 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:39:28.525 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:28.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.774+0000 7fb270c3e700 1 Processor -- start 2019-12-05T06:39:28.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.774+0000 7fb270c3e700 1 -- start start 2019-12-05T06:39:28.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.776+0000 7fb270c3e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb26c08d2c0 0x7fb26c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:28.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.776+0000 7fb270c3e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb26c08fef0 0x7fb26c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:28.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.776+0000 7fb270c3e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c0958e0 con 0x7fb26c08c6d0 2019-12-05T06:39:28.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.776+0000 7fb270c3e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c08df90 con 0x7fb26c08fef0 2019-12-05T06:39:28.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.776+0000 7fb270c3e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb26c0a3f70 con 0x7fb26c08d2c0 2019-12-05T06:39:28.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.776+0000 7fb26ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb26c08c6d0 0x7fb26c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43306/0 (socket says 172.21.15.16:43306) 2019-12-05T06:39:28.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.776+0000 7fb26ad9d700 1 -- 172.21.15.16:0/3072463547 learned_addr learned my addr 172.21.15.16:0/3072463547 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:28.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.776+0000 7fb26b59e700 1 --2- 172.21.15.16:0/3072463547 >> v2:172.21.15.76:3300/0 conn(0x7fb26c08fef0 0x7fb26c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:28.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.776+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb26c08d2c0 0x7fb26c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:28.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.777+0000 7fb269d9b700 1 -- 172.21.15.16:0/3072463547 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668644543 0 0) 0x7fb25c001150 con 0x7fb26c08c6d0 2019-12-05T06:39:28.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.777+0000 7fb269d9b700 1 -- 172.21.15.16:0/3072463547 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb258002e80 con 0x7fb26c08c6d0 2019-12-05T06:39:28.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.777+0000 7fb26b59e700 1 -- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb26c08d2c0 msgr2=0x7fb26c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:28.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.777+0000 7fb26b59e700 1 --2- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb26c08d2c0 0x7fb26c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:28.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.777+0000 7fb26b59e700 1 -- 172.21.15.16:0/3072463547 >> v1:172.21.15.16:6789/0 conn(0x7fb26c08c6d0 legacy=0x7fb26c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:28.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.777+0000 7fb26b59e700 1 -- 172.21.15.16:0/3072463547 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c08e620 con 0x7fb26c08fef0 2019-12-05T06:39:28.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.777+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb26c08d2c0 0x7fb26c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:39:28.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.777+0000 7fb26b59e700 1 --2- 172.21.15.16:0/3072463547 >> v2:172.21.15.76:3300/0 conn(0x7fb26c08fef0 0x7fb26c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb26400b300 tx=0x7fb26400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.777+0000 7fb26b59e700 1 --2- 172.21.15.16:0/3072463547 >> v2:172.21.15.76:3300/0 conn(0x7fb26c08fef0 0x7fb26c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb26400b300 tx=0x7fb26400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.777+0000 7fb26b59e700 1 --2- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c08fef0 0x7fb26c08f460 secure :-1 s=READY pgs=789 cs=0 l=1 rx=0x7fb26400b300 tx=0x7fb26400bb10).ready entity=mon.1 client_cookie=6a1815f57e34ae89 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:28.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.778+0000 7fb26b59e700 1 --2- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c08fef0 0x7fb26c08f460 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7fb26400b300 tx=0x7fb26400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.778+0000 7fb269d9b700 1 -- 172.21.15.16:0/3072463547 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb264000f40 con 0x7fb26c08fef0 2019-12-05T06:39:28.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.778+0000 7fb26b59e700 1 --2- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c08fef0 0x7fb26c08f460 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7fb26400b300 tx=0x7fb26400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.778+0000 7fb26b59e700 1 --2- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c08fef0 0x7fb26c08f460 secure :-1 s=THROTTLE_DONE pgs=789 cs=0 l=1 rx=0x7fb26400b300 tx=0x7fb26400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.778+0000 7fb269d9b700 1 -- 172.21.15.16:0/3072463547 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb264009b70 con 0x7fb26c08fef0 2019-12-05T06:39:28.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.778+0000 7fb269d9b700 1 -- 172.21.15.16:0/3072463547 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb264000df0 con 0x7fb26c08fef0 2019-12-05T06:39:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.778+0000 7fb270c3e700 1 -- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c08fef0 msgr2=0x7fb26c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:28.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.778+0000 7fb270c3e700 1 --2- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c08fef0 0x7fb26c08f460 secure :-1 s=READY pgs=789 cs=0 l=1 rx=0x7fb26400b300 tx=0x7fb26400bb10).stop 2019-12-05T06:39:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.780+0000 7fb270c3e700 1 -- 172.21.15.16:0/3072463547 shutdown_connections 2019-12-05T06:39:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.780+0000 7fb270c3e700 1 --2- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c08fef0 0x7fb26c08f460 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.780+0000 7fb270c3e700 1 --2- 172.21.15.16:0/3072463547 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb26c08d2c0 0x7fb26c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:28.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.780+0000 7fb270c3e700 1 -- 172.21.15.16:0/3072463547 shutdown_connections 2019-12-05T06:39:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.780+0000 7fb270c3e700 1 -- 172.21.15.16:0/3072463547 wait complete. 2019-12-05T06:39:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.780+0000 7fb270c3e700 1 -- 172.21.15.16:0/3072463547 >> 172.21.15.16:0/3072463547 conn(0x7fb26c061940 msgr2=0x7fb26c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:28.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.784+0000 7fb270c3e700 1 Processor -- start 2019-12-05T06:39:28.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.785+0000 7fb270c3e700 1 -- start start 2019-12-05T06:39:28.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.786+0000 7fb270c3e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb26c09e7d0 0x7fb26c09e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:28.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.786+0000 7fb270c3e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c12cb90 con 0x7fb26c156e70 2019-12-05T06:39:28.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.786+0000 7fb270c3e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb26c12d040 con 0x7fb26c1302f0 2019-12-05T06:39:28.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.786+0000 7fb270c3e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb26c131ff0 con 0x7fb26c09e7d0 2019-12-05T06:39:28.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.786+0000 7fb26b59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb26c156e70 0x7fb26c159270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43316/0 (socket says 172.21.15.16:43316) 2019-12-05T06:39:28.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.786+0000 7fb26b59e700 1 -- 172.21.15.16:0/3929460615 learned_addr learned my addr 172.21.15.16:0/3929460615 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:39:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.786+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> v2:172.21.15.76:3300/0 conn(0x7fb26c09e7d0 0x7fb26c09e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.787+0000 7fb2637fe700 1 -- 172.21.15.16:0/3929460615 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230964162 0 0) 0x7fb264009930 con 0x7fb26c156e70 2019-12-05T06:39:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.787+0000 7fb2637fe700 1 -- 172.21.15.16:0/3929460615 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb250002e80 con 0x7fb26c156e70 2019-12-05T06:39:28.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.787+0000 7fb26a59c700 1 -- 172.21.15.16:0/3929460615 >> v1:172.21.15.76:6789/0 conn(0x7fb26c1302f0 legacy=0x7fb26c09d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:28.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.787+0000 7fb26a59c700 1 -- 172.21.15.16:0/3929460615 >> v1:172.21.15.16:6789/0 conn(0x7fb26c156e70 legacy=0x7fb26c159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:28.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.787+0000 7fb26a59c700 1 -- 172.21.15.16:0/3929460615 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb26c132680 con 0x7fb26c09e7d0 2019-12-05T06:39:28.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.787+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> v2:172.21.15.76:3300/0 conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.787+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> v2:172.21.15.76:3300/0 conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.787+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=READY pgs=791 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:28.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.788+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=THROTTLE_DONE pgs=791 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.788+0000 7fb2637fe700 1 -- 172.21.15.16:0/3929460615 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb254000940 con 0x7fb26c09e7d0 2019-12-05T06:39:28.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.788+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=THROTTLE_DONE pgs=791 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.788+0000 7fb2637fe700 1 -- 172.21.15.16:0/3929460615 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb25400dff0 con 0x7fb26c09e7d0 2019-12-05T06:39:28.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.788+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=THROTTLE_DONE pgs=791 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.788+0000 7fb270c3e700 1 -- 172.21.15.16:0/3929460615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb26c132680 con 0x7fb26c09e7d0 2019-12-05T06:39:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.788+0000 7fb2637fe700 1 -- 172.21.15.16:0/3929460615 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb25400e1e0 con 0x7fb26c09e7d0 2019-12-05T06:39:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.788+0000 7fb270c3e700 1 -- 172.21.15.16:0/3929460615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb26c132680 con 0x7fb26c09e7d0 2019-12-05T06:39:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.789+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=THROTTLE_DONE pgs=791 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.790+0000 7fb2637fe700 1 -- 172.21.15.16:0/3929460615 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb25400e1e0 con 0x7fb26c09e7d0 2019-12-05T06:39:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.790+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=THROTTLE_DONE pgs=791 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.790+0000 7fb2637fe700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb250023670 0x7fb250025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:28.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.790+0000 7fb2637fe700 1 -- 172.21.15.16:0/3929460615 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7fb254045de0 con 0x7fb26c09e7d0 2019-12-05T06:39:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.790+0000 7fb26ad9d700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb250023670 0x7fb250025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.791+0000 7fb26ad9d700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb250023670 0x7fb250025ae0 crc :-1 s=READY pgs=486 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.805+0000 7fb270c3e700 1 -- 172.21.15.16:0/3929460615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb26c132680 con 0x7fb26c09e7d0 2019-12-05T06:39:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.815+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=THROTTLE_DONE pgs=791 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:28.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:28.815+0000 7fb2637fe700 1 -- 172.21.15.16:0/3929460615 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb25402ae00 con 0x7fb26c09e7d0 2019-12-05T06:39:29.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.060+0000 7fb270c3e700 1 -- 172.21.15.16:0/3929460615 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb26c132680 con 0x7fb26c09e7d0 2019-12-05T06:39:29.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.060+0000 7fb26a59c700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=THROTTLE_DONE pgs=791 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:29.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.060+0000 7fb2637fe700 1 -- 172.21.15.16:0/3929460615 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 141 v141) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fb25400ee30 con 0x7fb26c09e7d0 2019-12-05T06:39:29.065 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 141 2019-12-05T06:39:29.066 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:29.066 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":141,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.065+0000 7fb270c3e700 1 -- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb250023670 msgr2=0x7fb250025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.065+0000 7fb270c3e700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb250023670 0x7fb250025ae0 crc :-1 s=READY pgs=486 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.065+0000 7fb270c3e700 1 -- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 msgr2=0x7fb26c09e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:29.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.065+0000 7fb270c3e700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 0x7fb26c09e000 secure :-1 s=READY pgs=791 cs=0 l=1 rx=0x7fb2540103a0 tx=0x7fb254010740).stop 2019-12-05T06:39:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.065+0000 7fb270c3e700 1 -- 172.21.15.16:0/3929460615 shutdown_connections 2019-12-05T06:39:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.065+0000 7fb270c3e700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb250023670 0x7fb250025ae0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.065+0000 7fb270c3e700 1 --2- 172.21.15.16:0/3929460615 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb26c09e7d0 0x7fb26c09e000 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.065+0000 7fb270c3e700 1 -- 172.21.15.16:0/3929460615 shutdown_connections 2019-12-05T06:39:29.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.065+0000 7fb270c3e700 1 -- 172.21.15.16:0/3929460615 wait complete. 2019-12-05T06:39:29.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:29.065+0000 7fb270c3e700 1 -- 172.21.15.16:0/3929460615 >> 172.21.15.16:0/3929460615 conn(0x7fb26c0a2850 msgr2=0x7fb26c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:29.096 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:39:31.097 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:39:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.257+0000 7fb43f02a700 1 Processor -- start 2019-12-05T06:39:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.257+0000 7fb43f02a700 1 -- start start 2019-12-05T06:39:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43f02a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb43808c6d0 0x7fb43808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43f02a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb43808fef0 0x7fb43808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43f02a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb43808ef20 0x7fb43808f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:39:31.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43f02a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4380a3f70 con 0x7fb43808ef20 2019-12-05T06:39:31.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43f02a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43808d830 con 0x7fb43808fef0 2019-12-05T06:39:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43f02a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb43808d540 con 0x7fb43808c6d0 2019-12-05T06:39:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43ddc8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb43808ef20 0x7fb43808f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43d5c7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb43808c6d0 0x7fb43808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43ddc8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb43808ef20 0x7fb43808f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58166/0 (socket says 172.21.15.16:58166) 2019-12-05T06:39:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43ddc8700 1 -- 172.21.15.16:0/2354139995 learned_addr learned my addr 172.21.15.16:0/2354139995 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.259+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/2354139995 >> v2:172.21.15.76:3300/0 conn(0x7fb43808fef0 0x7fb43808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.260+0000 7fb43ddc8700 1 -- 172.21.15.16:0/2354139995 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb43808c6d0 msgr2=0x7fb43808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.260+0000 7fb43ddc8700 1 --2- 172.21.15.16:0/2354139995 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb43808c6d0 0x7fb43808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:31.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.260+0000 7fb43ddc8700 1 -- 172.21.15.16:0/2354139995 >> v2:172.21.15.76:3300/0 conn(0x7fb43808fef0 msgr2=0x7fb43808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:39:31.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.260+0000 7fb43ddc8700 1 --2- 172.21.15.16:0/2354139995 >> v2:172.21.15.76:3300/0 conn(0x7fb43808fef0 0x7fb43808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:31.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.260+0000 7fb43ddc8700 1 -- 172.21.15.16:0/2354139995 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43808e620 con 0x7fb43808ef20 2019-12-05T06:39:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.260+0000 7fb43ddc8700 1 --2- 172.21.15.16:0/2354139995 >> v2:172.21.15.16:3300/0 conn(0x7fb43808ef20 0x7fb43808f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb42800f6b0 tx=0x7fb428007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.260+0000 7fb43ddc8700 1 --2- 172.21.15.16:0/2354139995 >> v2:172.21.15.16:3300/0 conn(0x7fb43808ef20 0x7fb43808f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb42800f6b0 tx=0x7fb428007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.260+0000 7fb43ddc8700 1 --2- 172.21.15.16:0/2354139995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43808ef20 0x7fb43808f3c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rx=0x7fb42800f6b0 tx=0x7fb428007d50).ready entity=mon.0 client_cookie=1f424339f18e7e7b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:31.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.260+0000 7fb43ddc8700 1 --2- 172.21.15.16:0/2354139995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43808ef20 0x7fb43808f3c0 secure :-1 s=THROTTLE_DONE pgs=1135 cs=0 l=1 rx=0x7fb42800f6b0 tx=0x7fb428007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.261+0000 7fb43ddc8700 1 --2- 172.21.15.16:0/2354139995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43808ef20 0x7fb43808f3c0 secure :-1 s=THROTTLE_DONE pgs=1135 cs=0 l=1 rx=0x7fb42800f6b0 tx=0x7fb428007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.261+0000 7fb437fff700 1 -- 172.21.15.16:0/2354139995 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb428000a70 con 0x7fb43808ef20 2019-12-05T06:39:31.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.261+0000 7fb43ddc8700 1 --2- 172.21.15.16:0/2354139995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43808ef20 0x7fb43808f3c0 secure :-1 s=THROTTLE_DONE pgs=1135 cs=0 l=1 rx=0x7fb42800f6b0 tx=0x7fb428007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.261+0000 7fb437fff700 1 -- 172.21.15.16:0/2354139995 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4280010e0 con 0x7fb43808ef20 2019-12-05T06:39:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.261+0000 7fb437fff700 1 -- 172.21.15.16:0/2354139995 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb428001830 con 0x7fb43808ef20 2019-12-05T06:39:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.261+0000 7fb43f02a700 1 -- 172.21.15.16:0/2354139995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43808ef20 msgr2=0x7fb43808f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.261+0000 7fb43f02a700 1 --2- 172.21.15.16:0/2354139995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43808ef20 0x7fb43808f3c0 secure :-1 s=READY pgs=1135 cs=0 l=1 rx=0x7fb42800f6b0 tx=0x7fb428007d50).stop 2019-12-05T06:39:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.262+0000 7fb43f02a700 1 -- 172.21.15.16:0/2354139995 shutdown_connections 2019-12-05T06:39:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.262+0000 7fb43f02a700 1 --2- 172.21.15.16:0/2354139995 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb43808ef20 0x7fb43808f3c0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.262+0000 7fb43f02a700 1 --2- 172.21.15.16:0/2354139995 >> v2:172.21.15.76:3300/0 conn(0x7fb43808fef0 0x7fb43808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.262+0000 7fb43f02a700 1 --2- 172.21.15.16:0/2354139995 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb43808c6d0 0x7fb43808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:39:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.262+0000 7fb43f02a700 1 -- 172.21.15.16:0/2354139995 shutdown_connections 2019-12-05T06:39:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.262+0000 7fb43f02a700 1 -- 172.21.15.16:0/2354139995 wait complete. 2019-12-05T06:39:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.262+0000 7fb43f02a700 1 -- 172.21.15.16:0/2354139995 >> 172.21.15.16:0/2354139995 conn(0x7fb438061940 msgr2=0x7fb438096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.266+0000 7fb43f02a700 1 Processor -- start 2019-12-05T06:39:31.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.266+0000 7fb43f02a700 1 -- start start 2019-12-05T06:39:31.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43f02a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb43809b850 0x7fb438131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:31.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43f02a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb438132d00 0x7fb438132270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:31.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43f02a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb438098da0 con 0x7fb4381327b0 2019-12-05T06:39:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43f02a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43809aa60 con 0x7fb438132d00 2019-12-05T06:39:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43f02a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb43809a030 con 0x7fb43809b850 2019-12-05T06:39:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43cdc6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb438132d00 0x7fb438132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43cdc6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb438132d00 0x7fb438132270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58170/0 (socket says 172.21.15.16:58170) 2019-12-05T06:39:31.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43cdc6700 1 -- 172.21.15.16:0/1939131071 learned_addr learned my addr 172.21.15.16:0/1939131071 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:39:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43d5c7700 1 --2- 172.21.15.16:0/1939131071 >> v2:172.21.15.76:3300/0 conn(0x7fb43809b850 0x7fb438131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43cdc6700 1 -- 172.21.15.16:0/1939131071 >> v2:172.21.15.76:3300/0 conn(0x7fb43809b850 msgr2=0x7fb438131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> v2:172.21.15.76:3300/0 conn(0x7fb43809b850 0x7fb438131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43cdc6700 1 -- 172.21.15.16:0/1939131071 >> v1:172.21.15.76:6789/0 conn(0x7fb4381327b0 legacy=0x7fb43806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43cdc6700 1 -- 172.21.15.16:0/1939131071 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb43809b0f0 con 0x7fb438132d00 2019-12-05T06:39:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> v2:172.21.15.16:3300/0 conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> v2:172.21.15.16:3300/0 conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.268+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=READY pgs=1136 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.269+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=THROTTLE_DONE pgs=1136 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.269+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=THROTTLE_DONE pgs=1136 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.269+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=THROTTLE_DONE pgs=1136 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.269+0000 7fb435ffb700 1 -- 172.21.15.16:0/1939131071 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb42c000940 con 0x7fb438132d00 2019-12-05T06:39:31.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.269+0000 7fb435ffb700 1 -- 172.21.15.16:0/1939131071 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb42c000e80 con 0x7fb438132d00 2019-12-05T06:39:31.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.269+0000 7fb43f02a700 1 -- 172.21.15.16:0/1939131071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb43809b0f0 con 0x7fb438132d00 2019-12-05T06:39:31.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.269+0000 7fb435ffb700 1 -- 172.21.15.16:0/1939131071 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb42c0014d0 con 0x7fb438132d00 2019-12-05T06:39:31.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.269+0000 7fb43f02a700 1 -- 172.21.15.16:0/1939131071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb438098da0 con 0x7fb438132d00 2019-12-05T06:39:31.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.270+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=THROTTLE_DONE pgs=1136 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.270+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=THROTTLE_DONE pgs=1136 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.287 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.270+0000 7fb435ffb700 1 -- 172.21.15.16:0/1939131071 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb42c0019a0 con 0x7fb438132d00 2019-12-05T06:39:31.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.271+0000 7fb435ffb700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb418023630 0x7fb418025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:39:31.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.271+0000 7fb435ffb700 1 -- 172.21.15.16:0/1939131071 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7fb42c045e70 con 0x7fb438132d00 2019-12-05T06:39:31.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.271+0000 7fb43d5c7700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb418023630 0x7fb418025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:39:31.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.271+0000 7fb43d5c7700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb418023630 0x7fb418025ac0 crc :-1 s=READY pgs=488 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:39:31.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.273+0000 7fb43f02a700 1 -- 172.21.15.16:0/1939131071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb43809b0f0 con 0x7fb438132d00 2019-12-05T06:39:31.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.282+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=THROTTLE_DONE pgs=1136 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.282+0000 7fb435ffb700 1 -- 172.21.15.16:0/1939131071 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb42c001790 con 0x7fb438132d00 2019-12-05T06:39:31.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.517+0000 7fb43f02a700 1 -- 172.21.15.16:0/1939131071 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb43809b0f0 con 0x7fb438132d00 2019-12-05T06:39:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.518+0000 7fb43cdc6700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=THROTTLE_DONE pgs=1136 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:39:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.518+0000 7fb435ffb700 1 -- 172.21.15.16:0/1939131071 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb42c0019a0 con 0x7fb438132d00 2019-12-05T06:39:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 142 2019-12-05T06:39:31.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 -- 172.21.15.16:0/1939131071 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb418023630 msgr2=0x7fb418025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb418023630 0x7fb418025ac0 crc :-1 s=READY pgs=488 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 -- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 msgr2=0x7fb438132270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:39:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 0x7fb438132270 secure :-1 s=READY pgs=1136 cs=0 l=1 rx=0x7fb42c010450 tx=0x7fb42c010280).stop 2019-12-05T06:39:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 -- 172.21.15.16:0/1939131071 shutdown_connections 2019-12-05T06:39:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb418023630 0x7fb418025ac0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 --2- 172.21.15.16:0/1939131071 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb438132d00 0x7fb438132270 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 --2- 172.21.15.16:0/1939131071 >> v2:172.21.15.76:3300/0 conn(0x7fb43809b850 0x7fb438131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:39:31.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 -- 172.21.15.16:0/1939131071 shutdown_connections 2019-12-05T06:39:31.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 -- 172.21.15.16:0/1939131071 wait complete. 2019-12-05T06:39:31.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:39:31.521+0000 7fb43f02a700 1 -- 172.21.15.16:0/1939131071 >> 172.21.15.16:0/1939131071 conn(0x7fb4380714f0 msgr2=0x7fb43809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:39:31.554 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:39:31.554 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:39:31.556 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:39:31.556 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 86 secs before thrashing 2019-12-05T06:39:40.667 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:39:40.672 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:40:10.854 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:40:10.883 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:40:41.053 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:40:41.058 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:40:57.557 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:57.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.848+0000 7f4234d8d700 1 Processor -- start 2019-12-05T06:40:57.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.848+0000 7f4234d8d700 1 -- start start 2019-12-05T06:40:57.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.850+0000 7f4234d8d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f423008fef0 0x7f423008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:57.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.850+0000 7f4234d8d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300958e0 con 0x7f423008d2c0 2019-12-05T06:40:57.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.850+0000 7f4234d8d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42300629e0 con 0x7f423008c6d0 2019-12-05T06:40:57.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.850+0000 7f4234d8d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f423008df90 con 0x7f423008fef0 2019-12-05T06:40:57.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.850+0000 7f422e59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f423008d2c0 0x7f423008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43740/0 (socket says 172.21.15.16:43740) 2019-12-05T06:40:57.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.850+0000 7f422e59c700 1 -- 172.21.15.16:0/3579693795 learned_addr learned my addr 172.21.15.16:0/3579693795 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:40:57.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.850+0000 7f422f59e700 1 --2- 172.21.15.16:0/3579693795 >> v2:172.21.15.76:3300/0 conn(0x7f423008fef0 0x7f423008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.851+0000 7f422dd9b700 1 -- 172.21.15.16:0/3579693795 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1361837939 0 0) 0x7f4218001150 con 0x7f423008d2c0 2019-12-05T06:40:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.851+0000 7f422dd9b700 1 -- 172.21.15.16:0/3579693795 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f421c002e80 con 0x7f423008d2c0 2019-12-05T06:40:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.851+0000 7f422dd9b700 1 -- 172.21.15.16:0/3579693795 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005725903 0 0) 0x7f4228001400 con 0x7f423008c6d0 2019-12-05T06:40:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.851+0000 7f422dd9b700 1 -- 172.21.15.16:0/3579693795 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f421c004170 con 0x7f423008c6d0 2019-12-05T06:40:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.851+0000 7f422f59e700 1 -- 172.21.15.16:0/3579693795 >> v1:172.21.15.76:6789/0 conn(0x7f423008c6d0 legacy=0x7f423008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:57.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.851+0000 7f422f59e700 1 -- 172.21.15.16:0/3579693795 >> v1:172.21.15.16:6789/0 conn(0x7f423008d2c0 legacy=0x7f423008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:57.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.851+0000 7f422f59e700 1 -- 172.21.15.16:0/3579693795 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423008e620 con 0x7f423008fef0 2019-12-05T06:40:57.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.852+0000 7f422f59e700 1 --2- 172.21.15.16:0/3579693795 >> v2:172.21.15.76:3300/0 conn(0x7f423008fef0 0x7f423008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f422000af80 tx=0x7f422000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.852+0000 7f422f59e700 1 --2- 172.21.15.16:0/3579693795 >> v2:172.21.15.76:3300/0 conn(0x7f423008fef0 0x7f423008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f422000af80 tx=0x7f422000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.852+0000 7f422f59e700 1 --2- 172.21.15.16:0/3579693795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f423008fef0 0x7f423008f720 secure :-1 s=READY pgs=794 cs=0 l=1 rx=0x7f422000af80 tx=0x7f422000b790).ready entity=mon.1 client_cookie=404357cb8965cbd7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.852+0000 7f422f59e700 1 --2- 172.21.15.16:0/3579693795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f423008fef0 0x7f423008f720 secure :-1 s=THROTTLE_DONE pgs=794 cs=0 l=1 rx=0x7f422000af80 tx=0x7f422000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.852+0000 7f422f59e700 1 --2- 172.21.15.16:0/3579693795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f423008fef0 0x7f423008f720 secure :-1 s=THROTTLE_DONE pgs=794 cs=0 l=1 rx=0x7f422000af80 tx=0x7f422000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.852+0000 7f422f59e700 1 --2- 172.21.15.16:0/3579693795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f423008fef0 0x7f423008f720 secure :-1 s=THROTTLE_DONE pgs=794 cs=0 l=1 rx=0x7f422000af80 tx=0x7f422000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.852+0000 7f422dd9b700 1 -- 172.21.15.16:0/3579693795 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4220007240 con 0x7f423008fef0 2019-12-05T06:40:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.852+0000 7f422dd9b700 1 -- 172.21.15.16:0/3579693795 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4220000c30 con 0x7f423008fef0 2019-12-05T06:40:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.852+0000 7f422dd9b700 1 -- 172.21.15.16:0/3579693795 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4220009f10 con 0x7f423008fef0 2019-12-05T06:40:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.853+0000 7f4234d8d700 1 -- 172.21.15.16:0/3579693795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f423008fef0 msgr2=0x7f423008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.853+0000 7f4234d8d700 1 --2- 172.21.15.16:0/3579693795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f423008fef0 0x7f423008f720 secure :-1 s=READY pgs=794 cs=0 l=1 rx=0x7f422000af80 tx=0x7f422000b790).stop 2019-12-05T06:40:57.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.855+0000 7f4234d8d700 1 -- 172.21.15.16:0/3579693795 shutdown_connections 2019-12-05T06:40:57.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.855+0000 7f4234d8d700 1 --2- 172.21.15.16:0/3579693795 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f423008fef0 0x7f423008f720 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:57.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.855+0000 7f4234d8d700 1 -- 172.21.15.16:0/3579693795 shutdown_connections 2019-12-05T06:40:57.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.855+0000 7f4234d8d700 1 -- 172.21.15.16:0/3579693795 wait complete. 2019-12-05T06:40:57.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.855+0000 7f4234d8d700 1 -- 172.21.15.16:0/3579693795 >> 172.21.15.16:0/3579693795 conn(0x7f4230061940 msgr2=0x7f4230096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:57.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.860+0000 7f4234d8d700 1 Processor -- start 2019-12-05T06:40:57.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.861+0000 7f4234d8d700 1 -- start start 2019-12-05T06:40:57.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.862+0000 7f4234d8d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:57.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.862+0000 7f4234d8d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f423007b730 0x7f4230153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:57.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.862+0000 7f4234d8d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423008ab00 con 0x7f423007bf00 2019-12-05T06:40:57.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.862+0000 7f4234d8d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4230089f00 con 0x7f423007b730 2019-12-05T06:40:57.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.862+0000 7f4234d8d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4230088f90 con 0x7f423014d410 2019-12-05T06:40:57.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.862+0000 7f422f59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f423007b730 0x7f4230153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:57.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.863+0000 7f422f59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f423007b730 0x7f4230153150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:44670/0 (socket says 172.21.15.16:44670) 2019-12-05T06:40:57.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.862+0000 7f422ed9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:57.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.863+0000 7f422f59e700 1 -- 172.21.15.16:0/2188023169 learned_addr learned my addr 172.21.15.16:0/2188023169 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:40:57.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.863+0000 7f422ed9d700 1 -- 172.21.15.16:0/2188023169 >> v2:172.21.15.76:3300/0 conn(0x7f423007b730 msgr2=0x7f4230153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:57.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.863+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> v2:172.21.15.76:3300/0 conn(0x7f423007b730 0x7f4230153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:57.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.863+0000 7f422ed9d700 1 -- 172.21.15.16:0/2188023169 >> v1:172.21.15.76:6789/0 conn(0x7f423007bf00 legacy=0x7f423007af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:57.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.863+0000 7f422ed9d700 1 -- 172.21.15.16:0/2188023169 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423007bbd0 con 0x7f423014d410 2019-12-05T06:40:57.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.863+0000 7f422f59e700 1 --2- 172.21.15.16:0/2188023169 >> v2:172.21.15.76:3300/0 conn(0x7f423007b730 0x7f4230153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:40:57.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.863+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.864+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.864+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=READY pgs=266 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:57.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.864+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.864+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.864+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.864+0000 7f42277fe700 1 -- 172.21.15.16:0/2188023169 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4228000e20 con 0x7f423014d410 2019-12-05T06:40:57.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.864+0000 7f42277fe700 1 -- 172.21.15.16:0/2188023169 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4228009d00 con 0x7f423014d410 2019-12-05T06:40:57.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.864+0000 7f42277fe700 1 -- 172.21.15.16:0/2188023169 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f422800a410 con 0x7f423014d410 2019-12-05T06:40:57.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.864+0000 7f4234d8d700 1 -- 172.21.15.16:0/2188023169 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423007bbd0 con 0x7f423014d410 2019-12-05T06:40:57.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.864+0000 7f4234d8d700 1 -- 172.21.15.16:0/2188023169 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f423007a630 con 0x7f423014d410 2019-12-05T06:40:57.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.865+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.866+0000 7f42277fe700 1 -- 172.21.15.16:0/2188023169 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4228000e20 con 0x7f423014d410 2019-12-05T06:40:57.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.866+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.867+0000 7f42277fe700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4214023630 0x7f4214025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:57.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.867+0000 7f42277fe700 1 -- 172.21.15.16:0/2188023169 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f42280460d0 con 0x7f423014d410 2019-12-05T06:40:57.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.867+0000 7f422e59c700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4214023630 0x7f4214025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:57.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.868+0000 7f422e59c700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4214023630 0x7f4214025ac0 crc :-1 s=READY pgs=490 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:57.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.879+0000 7f4234d8d700 1 -- 172.21.15.16:0/2188023169 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f423007a630 con 0x7f423014d410 2019-12-05T06:40:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.889+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:57.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:57.889+0000 7f42277fe700 1 -- 172.21.15.16:0/2188023169 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4228009900 con 0x7f423014d410 2019-12-05T06:40:58.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.176+0000 7f4234d8d700 1 -- 172.21.15.16:0/2188023169 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f423007a630 con 0x7f423014d410 2019-12-05T06:40:58.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.177+0000 7f422ed9d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=THROTTLE_DONE pgs=266 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.177+0000 7f42277fe700 1 -- 172.21.15.16:0/2188023169 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4228000e20 con 0x7f423014d410 2019-12-05T06:40:58.179 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 142 2019-12-05T06:40:58.179 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:40:58.179 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:58.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 -- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4214023630 msgr2=0x7f4214025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:58.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4214023630 0x7f4214025ac0 crc :-1 s=READY pgs=490 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:58.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 -- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 msgr2=0x7f423014d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 secure :-1 s=READY pgs=266 cs=0 l=1 rx=0x7f4228010720 tx=0x7f4228010ac0).stop 2019-12-05T06:40:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 -- 172.21.15.16:0/2188023169 shutdown_connections 2019-12-05T06:40:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4214023630 0x7f4214025ac0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 --2- 172.21.15.16:0/2188023169 >> v2:172.21.15.76:3300/0 conn(0x7f423007b730 0x7f4230153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:58.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 --2- 172.21.15.16:0/2188023169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f423014d410 0x7f423014d800 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:58.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 -- 172.21.15.16:0/2188023169 shutdown_connections 2019-12-05T06:40:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 -- 172.21.15.16:0/2188023169 wait complete. 2019-12-05T06:40:58.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.180+0000 7f4234d8d700 1 -- 172.21.15.16:0/2188023169 >> 172.21.15.16:0/2188023169 conn(0x7f423006dfd0 msgr2=0x7f4230070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:58.195 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:40:58.196 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:40:58.196 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:40:58.204 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:40:58.205 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:40:58.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.536+0000 7f5d0fc79700 1 Processor -- start 2019-12-05T06:40:58.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.536+0000 7f5d0fc79700 1 -- start start 2019-12-05T06:40:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.538+0000 7f5d0fc79700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 0x7f5d0808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:40:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.538+0000 7f5d0fc79700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080958e0 con 0x7f5d0808d2c0 2019-12-05T06:40:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.538+0000 7f5d0fc79700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d080629e0 con 0x7f5d0808c6d0 2019-12-05T06:40:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.538+0000 7f5d0fc79700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d0808df90 con 0x7f5d0808fef0 2019-12-05T06:40:58.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.538+0000 7f5d0da15700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5d0808d2c0 0x7f5d0808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43760/0 (socket says 172.21.15.16:43760) 2019-12-05T06:40:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.538+0000 7f5d0da15700 1 -- 172.21.15.16:0/1042336206 learned_addr learned my addr 172.21.15.16:0/1042336206 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:40:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.538+0000 7f5d0ea17700 1 --2- 172.21.15.16:0/1042336206 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 0x7f5d0808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.539+0000 7f5d0d214700 1 -- 172.21.15.16:0/1042336206 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3253595538 0 0) 0x7f5cf8001080 con 0x7f5d0808c6d0 2019-12-05T06:40:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.539+0000 7f5d0d214700 1 -- 172.21.15.16:0/1042336206 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf0002e80 con 0x7f5d0808c6d0 2019-12-05T06:40:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.539+0000 7f5d0d214700 1 -- 172.21.15.16:0/1042336206 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263279592 0 0) 0x7f5cfc001100 con 0x7f5d0808d2c0 2019-12-05T06:40:58.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.539+0000 7f5d0d214700 1 -- 172.21.15.16:0/1042336206 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf0002e80 con 0x7f5d0808d2c0 2019-12-05T06:40:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.539+0000 7f5d0ea17700 1 -- 172.21.15.16:0/1042336206 >> v1:172.21.15.76:6789/0 conn(0x7f5d0808c6d0 legacy=0x7f5d0808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.539+0000 7f5d0ea17700 1 -- 172.21.15.16:0/1042336206 >> v1:172.21.15.16:6789/0 conn(0x7f5d0808d2c0 legacy=0x7f5d0808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.539+0000 7f5d0ea17700 1 -- 172.21.15.16:0/1042336206 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0808e620 con 0x7f5d0808fef0 2019-12-05T06:40:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.539+0000 7f5d0ea17700 1 --2- 172.21.15.16:0/1042336206 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 0x7f5d0808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5d0000b300 tx=0x7f5d0000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.539+0000 7f5d0ea17700 1 --2- 172.21.15.16:0/1042336206 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 0x7f5d0808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5d0000b300 tx=0x7f5d0000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.539+0000 7f5d0ea17700 1 --2- 172.21.15.16:0/1042336206 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 0x7f5d0808f720 secure :-1 s=READY pgs=268 cs=0 l=1 rx=0x7f5d0000b300 tx=0x7f5d0000bb10).ready entity=mon.2 client_cookie=d35b8c8cf166310c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:58.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.540+0000 7f5d0ea17700 1 --2- 172.21.15.16:0/1042336206 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 0x7f5d0808f720 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7f5d0000b300 tx=0x7f5d0000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.540+0000 7f5d0d214700 1 -- 172.21.15.16:0/1042336206 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d000075c0 con 0x7f5d0808fef0 2019-12-05T06:40:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.540+0000 7f5d0ea17700 1 --2- 172.21.15.16:0/1042336206 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 0x7f5d0808f720 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7f5d0000b300 tx=0x7f5d0000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.540+0000 7f5d0ea17700 1 --2- 172.21.15.16:0/1042336206 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 0x7f5d0808f720 secure :-1 s=THROTTLE_DONE pgs=268 cs=0 l=1 rx=0x7f5d0000b300 tx=0x7f5d0000bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.540+0000 7f5d0d214700 1 -- 172.21.15.16:0/1042336206 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d00000f30 con 0x7f5d0808fef0 2019-12-05T06:40:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.540+0000 7f5d0d214700 1 -- 172.21.15.16:0/1042336206 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d00009f50 con 0x7f5d0808fef0 2019-12-05T06:40:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.540+0000 7f5d0fc79700 1 -- 172.21.15.16:0/1042336206 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 msgr2=0x7f5d0808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.540+0000 7f5d0fc79700 1 --2- 172.21.15.16:0/1042336206 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 0x7f5d0808f720 secure :-1 s=READY pgs=268 cs=0 l=1 rx=0x7f5d0000b300 tx=0x7f5d0000bb10).stop 2019-12-05T06:40:58.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.541+0000 7f5d0fc79700 1 -- 172.21.15.16:0/1042336206 shutdown_connections 2019-12-05T06:40:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.541+0000 7f5d0fc79700 1 --2- 172.21.15.16:0/1042336206 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0808fef0 0x7f5d0808f720 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.541+0000 7f5d0fc79700 1 -- 172.21.15.16:0/1042336206 shutdown_connections 2019-12-05T06:40:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.541+0000 7f5d0fc79700 1 -- 172.21.15.16:0/1042336206 wait complete. 2019-12-05T06:40:58.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.541+0000 7f5d0fc79700 1 -- 172.21.15.16:0/1042336206 >> 172.21.15.16:0/1042336206 conn(0x7f5d08061940 msgr2=0x7f5d08096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.546+0000 7f5d0fc79700 1 Processor -- start 2019-12-05T06:40:58.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.546+0000 7f5d0fc79700 1 -- start start 2019-12-05T06:40:58.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.548+0000 7f5d0fc79700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5d08128360 0x7f5d08128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:58.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.548+0000 7f5d0fc79700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0812bdd0 0x7f5d08128c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.548+0000 7f5d0fc79700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d08099720 con 0x7f5d08137ea0 2019-12-05T06:40:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.548+0000 7f5d0fc79700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d0812a640 con 0x7f5d08128360 2019-12-05T06:40:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.548+0000 7f5d0fc79700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d08129c10 con 0x7f5d0812bdd0 2019-12-05T06:40:58.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.548+0000 7f5d0ea17700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0812bdd0 0x7f5d08128c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.548+0000 7f5d0e216700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5d08128360 0x7f5d08128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.548+0000 7f5d0ea17700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0812bdd0 0x7f5d08128c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:34356/0 (socket says 172.21.15.16:34356) 2019-12-05T06:40:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.548+0000 7f5d0e216700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5d08128360 0x7f5d08128750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:44686/0 (socket says 172.21.15.16:44686) 2019-12-05T06:40:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.548+0000 7f5d0ea17700 1 -- 172.21.15.16:0/459830592 learned_addr learned my addr 172.21.15.16:0/459830592 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:40:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.549+0000 7f5d0e216700 1 -- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0812bdd0 msgr2=0x7f5d08128c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.549+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0812bdd0 0x7f5d08128c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:58.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.549+0000 7f5d0e216700 1 -- 172.21.15.16:0/459830592 >> v1:172.21.15.76:6789/0 conn(0x7f5d08137ea0 legacy=0x7f5d0812b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:58.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.549+0000 7f5d0e216700 1 -- 172.21.15.16:0/459830592 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d0812acd0 con 0x7f5d08128360 2019-12-05T06:40:58.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.549+0000 7f5d0ea17700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0812bdd0 0x7f5d08128c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:40:58.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.549+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> v2:172.21.15.76:3300/0 conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.549+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> v2:172.21.15.76:3300/0 conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.549+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=READY pgs=799 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:58.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.550+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=THROTTLE_DONE pgs=799 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.550+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=THROTTLE_DONE pgs=799 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.550+0000 7f5d06ffd700 1 -- 172.21.15.16:0/459830592 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cf8009580 con 0x7f5d08128360 2019-12-05T06:40:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.550+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=THROTTLE_DONE pgs=799 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.550+0000 7f5d06ffd700 1 -- 172.21.15.16:0/459830592 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cf8009bf0 con 0x7f5d08128360 2019-12-05T06:40:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.550+0000 7f5d06ffd700 1 -- 172.21.15.16:0/459830592 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cf800a2c0 con 0x7f5d08128360 2019-12-05T06:40:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.550+0000 7f5d0fc79700 1 -- 172.21.15.16:0/459830592 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d0812acd0 con 0x7f5d08128360 2019-12-05T06:40:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.550+0000 7f5d0fc79700 1 -- 172.21.15.16:0/459830592 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d0812acd0 con 0x7f5d08128360 2019-12-05T06:40:58.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.551+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=THROTTLE_DONE pgs=799 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.551+0000 7f5d06ffd700 1 -- 172.21.15.16:0/459830592 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5cf8009790 con 0x7f5d08128360 2019-12-05T06:40:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.551+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=THROTTLE_DONE pgs=799 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.552+0000 7f5d06ffd700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5ce8023630 0x7f5ce8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:40:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.552+0000 7f5d06ffd700 1 -- 172.21.15.16:0/459830592 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7f5cf8045db0 con 0x7f5d08128360 2019-12-05T06:40:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.552+0000 7f5d0da15700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5ce8023630 0x7f5ce8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:40:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.553+0000 7f5d0da15700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5ce8023630 0x7f5ce8025ac0 crc :-1 s=READY pgs=491 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:40:58.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.568+0000 7f5d0fc79700 1 -- 172.21.15.16:0/459830592 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d0812acd0 con 0x7f5d08128360 2019-12-05T06:40:58.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.576+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=THROTTLE_DONE pgs=799 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.576+0000 7f5d06ffd700 1 -- 172.21.15.16:0/459830592 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5cf8009580 con 0x7f5d08128360 2019-12-05T06:40:58.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.848+0000 7f5d0fc79700 1 -- 172.21.15.16:0/459830592 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d0812acd0 con 0x7f5d08128360 2019-12-05T06:40:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.849+0000 7f5d0e216700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=THROTTLE_DONE pgs=799 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:40:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.849+0000 7f5d06ffd700 1 -- 172.21.15.16:0/459830592 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f5cf8009790 con 0x7f5d08128360 2019-12-05T06:40:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 142 2019-12-05T06:40:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.853+0000 7f5d0fc79700 1 -- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5ce8023630 msgr2=0x7f5ce8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:58.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.853+0000 7f5d0fc79700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5ce8023630 0x7f5ce8025ac0 crc :-1 s=READY pgs=491 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.853+0000 7f5d0fc79700 1 -- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 msgr2=0x7f5d08128750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:40:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.853+0000 7f5d0fc79700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 0x7f5d08128750 secure :-1 s=READY pgs=799 cs=0 l=1 rx=0x7f5cf80103a0 tx=0x7f5cf8010740).stop 2019-12-05T06:40:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.853+0000 7f5d0fc79700 1 -- 172.21.15.16:0/459830592 shutdown_connections 2019-12-05T06:40:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.853+0000 7f5d0fc79700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5ce8023630 0x7f5ce8025ac0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.853+0000 7f5d0fc79700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5d0812bdd0 0x7f5d08128c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:58.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.853+0000 7f5d0fc79700 1 --2- 172.21.15.16:0/459830592 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d08128360 0x7f5d08128750 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:40:58.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.853+0000 7f5d0fc79700 1 -- 172.21.15.16:0/459830592 shutdown_connections 2019-12-05T06:40:58.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.854+0000 7f5d0fc79700 1 -- 172.21.15.16:0/459830592 wait complete. 2019-12-05T06:40:58.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:40:58.854+0000 7f5d0fc79700 1 -- 172.21.15.16:0/459830592 >> 172.21.15.16:0/459830592 conn(0x7f5d0806d260 msgr2=0x7f5d0809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:40:58.860 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:40:58.860 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:40:58.873 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:41:00.874 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:01.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.156+0000 7fcc8093b700 1 Processor -- start 2019-12-05T06:41:01.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.156+0000 7fcc8093b700 1 -- start start 2019-12-05T06:41:01.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.158+0000 7fcc8093b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 0x7fcc7c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:01.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.159+0000 7fcc8093b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c0958e0 con 0x7fcc7c08c6d0 2019-12-05T06:41:01.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.159+0000 7fcc8093b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c0629e0 con 0x7fcc7c08d2c0 2019-12-05T06:41:01.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.159+0000 7fcc8093b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc7c08df90 con 0x7fcc7c08fef0 2019-12-05T06:41:01.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.159+0000 7fcc7ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc7c08c6d0 0x7fcc7c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43776/0 (socket says 172.21.15.16:43776) 2019-12-05T06:41:01.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.159+0000 7fcc7ad9d700 1 -- 172.21.15.16:0/570695700 learned_addr learned my addr 172.21.15.16:0/570695700 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.159+0000 7fcc7b59e700 1 --2- 172.21.15.16:0/570695700 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 0x7fcc7c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.159+0000 7fcc79d9b700 1 -- 172.21.15.16:0/570695700 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 991507233 0 0) 0x7fcc740014f0 con 0x7fcc7c08c6d0 2019-12-05T06:41:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.160+0000 7fcc79d9b700 1 -- 172.21.15.16:0/570695700 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc68002e80 con 0x7fcc7c08c6d0 2019-12-05T06:41:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.160+0000 7fcc79d9b700 1 -- 172.21.15.16:0/570695700 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610990421 0 0) 0x7fcc64001080 con 0x7fcc7c08d2c0 2019-12-05T06:41:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.160+0000 7fcc79d9b700 1 -- 172.21.15.16:0/570695700 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc68004170 con 0x7fcc7c08d2c0 2019-12-05T06:41:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.160+0000 7fcc7b59e700 1 -- 172.21.15.16:0/570695700 >> v1:172.21.15.76:6789/0 conn(0x7fcc7c08d2c0 legacy=0x7fcc7c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:01.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.160+0000 7fcc7b59e700 1 -- 172.21.15.16:0/570695700 >> v1:172.21.15.16:6789/0 conn(0x7fcc7c08c6d0 legacy=0x7fcc7c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.160+0000 7fcc7b59e700 1 -- 172.21.15.16:0/570695700 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c08e620 con 0x7fcc7c08fef0 2019-12-05T06:41:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.160+0000 7fcc7b59e700 1 --2- 172.21.15.16:0/570695700 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 0x7fcc7c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc6c00af80 tx=0x7fcc6c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.160+0000 7fcc7b59e700 1 --2- 172.21.15.16:0/570695700 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 0x7fcc7c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc6c00af80 tx=0x7fcc6c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.160+0000 7fcc7b59e700 1 --2- 172.21.15.16:0/570695700 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 0x7fcc7c08f720 secure :-1 s=READY pgs=269 cs=0 l=1 rx=0x7fcc6c00af80 tx=0x7fcc6c00b790).ready entity=mon.2 client_cookie=48b7bf1c8bea24ed server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.161+0000 7fcc7b59e700 1 --2- 172.21.15.16:0/570695700 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 0x7fcc7c08f720 secure :-1 s=THROTTLE_DONE pgs=269 cs=0 l=1 rx=0x7fcc6c00af80 tx=0x7fcc6c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.161+0000 7fcc7b59e700 1 --2- 172.21.15.16:0/570695700 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 0x7fcc7c08f720 secure :-1 s=THROTTLE_DONE pgs=269 cs=0 l=1 rx=0x7fcc6c00af80 tx=0x7fcc6c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.161+0000 7fcc7b59e700 1 --2- 172.21.15.16:0/570695700 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 0x7fcc7c08f720 secure :-1 s=THROTTLE_DONE pgs=269 cs=0 l=1 rx=0x7fcc6c00af80 tx=0x7fcc6c00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.161+0000 7fcc79d9b700 1 -- 172.21.15.16:0/570695700 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc6c007240 con 0x7fcc7c08fef0 2019-12-05T06:41:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.161+0000 7fcc79d9b700 1 -- 172.21.15.16:0/570695700 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc6c000c30 con 0x7fcc7c08fef0 2019-12-05T06:41:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.161+0000 7fcc79d9b700 1 -- 172.21.15.16:0/570695700 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc6c009f50 con 0x7fcc7c08fef0 2019-12-05T06:41:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.161+0000 7fcc8093b700 1 -- 172.21.15.16:0/570695700 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 msgr2=0x7fcc7c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.161+0000 7fcc8093b700 1 --2- 172.21.15.16:0/570695700 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 0x7fcc7c08f720 secure :-1 s=READY pgs=269 cs=0 l=1 rx=0x7fcc6c00af80 tx=0x7fcc6c00b790).stop 2019-12-05T06:41:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.163+0000 7fcc8093b700 1 -- 172.21.15.16:0/570695700 shutdown_connections 2019-12-05T06:41:01.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.163+0000 7fcc8093b700 1 --2- 172.21.15.16:0/570695700 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc7c08fef0 0x7fcc7c08f720 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.163+0000 7fcc8093b700 1 -- 172.21.15.16:0/570695700 shutdown_connections 2019-12-05T06:41:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.163+0000 7fcc8093b700 1 -- 172.21.15.16:0/570695700 wait complete. 2019-12-05T06:41:01.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.163+0000 7fcc8093b700 1 -- 172.21.15.16:0/570695700 >> 172.21.15.16:0/570695700 conn(0x7fcc7c061940 msgr2=0x7fcc7c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.168+0000 7fcc8093b700 1 Processor -- start 2019-12-05T06:41:01.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.168+0000 7fcc8093b700 1 -- start start 2019-12-05T06:41:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.170+0000 7fcc8093b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.170+0000 7fcc8093b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcc7c14d9c0 0x7fcc7c153320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.170+0000 7fcc8093b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc7c14de80 con 0x7fcc7c14d5d0 2019-12-05T06:41:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.170+0000 7fcc8093b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c12b3f0 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.170+0000 7fcc8093b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc7c12a9c0 con 0x7fcc7c14d9c0 2019-12-05T06:41:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.170+0000 7fcc7a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:01.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.170+0000 7fcc7a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58616/0 (socket says 172.21.15.16:58616) 2019-12-05T06:41:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.170+0000 7fcc7a59c700 1 -- 172.21.15.16:0/1589541831 learned_addr learned my addr 172.21.15.16:0/1589541831 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.170+0000 7fcc7b59e700 1 --2- 172.21.15.16:0/1589541831 >> v2:172.21.15.76:3300/0 conn(0x7fcc7c14d9c0 0x7fcc7c153320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.171+0000 7fcc7a59c700 1 -- 172.21.15.16:0/1589541831 >> v2:172.21.15.76:3300/0 conn(0x7fcc7c14d9c0 msgr2=0x7fcc7c153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.171+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> v2:172.21.15.76:3300/0 conn(0x7fcc7c14d9c0 0x7fcc7c153320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.171+0000 7fcc7a59c700 1 -- 172.21.15.16:0/1589541831 >> v1:172.21.15.76:6789/0 conn(0x7fcc7c14d5d0 legacy=0x7fcc7c09d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.171+0000 7fcc7a59c700 1 -- 172.21.15.16:0/1589541831 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc7c12ba80 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.171+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> v2:172.21.15.16:3300/0 conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.171+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> v2:172.21.15.16:3300/0 conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.171+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=READY pgs=1145 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.172+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=THROTTLE_DONE pgs=1145 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.172+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=THROTTLE_DONE pgs=1145 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.172+0000 7fcc737fe700 1 -- 172.21.15.16:0/1589541831 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc64005770 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.172+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=THROTTLE_DONE pgs=1145 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.172+0000 7fcc737fe700 1 -- 172.21.15.16:0/1589541831 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc64005390 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.172+0000 7fcc8093b700 1 -- 172.21.15.16:0/1589541831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc7c12ba80 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.172+0000 7fcc737fe700 1 -- 172.21.15.16:0/1589541831 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc64005d50 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.172+0000 7fcc8093b700 1 -- 172.21.15.16:0/1589541831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc7c12ba80 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.173+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=THROTTLE_DONE pgs=1145 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.174+0000 7fcc737fe700 1 -- 172.21.15.16:0/1589541831 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcc64005580 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.174+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=THROTTLE_DONE pgs=1145 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.174+0000 7fcc737fe700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc60023630 0x7fcc60025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.174+0000 7fcc737fe700 1 -- 172.21.15.16:0/1589541831 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7fcc64045d80 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.175+0000 7fcc7ad9d700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc60023630 0x7fcc60025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:01.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.175+0000 7fcc7ad9d700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc60023630 0x7fcc60025ac0 crc :-1 s=READY pgs=492 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:01.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.188+0000 7fcc8093b700 1 -- 172.21.15.16:0/1589541831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc7c12ba80 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.197+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=THROTTLE_DONE pgs=1145 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.197+0000 7fcc737fe700 1 -- 172.21.15.16:0/1589541831 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcc640075a0 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.439+0000 7fcc8093b700 1 -- 172.21.15.16:0/1589541831 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc7c12ba80 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.440+0000 7fcc7a59c700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=THROTTLE_DONE pgs=1145 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.440+0000 7fcc737fe700 1 -- 172.21.15.16:0/1589541831 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fcc64005560 con 0x7fcc7c09e2c0 2019-12-05T06:41:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 142 2019-12-05T06:41:01.452 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 -- 172.21.15.16:0/1589541831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc60023630 msgr2=0x7fcc60025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc60023630 0x7fcc60025ac0 crc :-1 s=READY pgs=492 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 -- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 msgr2=0x7fcc7c09dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:01.453 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 secure :-1 s=READY pgs=1145 cs=0 l=1 rx=0x7fcc64010410 tx=0x7fcc64010290).stop 2019-12-05T06:41:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 -- 172.21.15.16:0/1589541831 shutdown_connections 2019-12-05T06:41:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 --2- 172.21.15.16:0/1589541831 >> v2:172.21.15.76:3300/0 conn(0x7fcc7c14d9c0 0x7fcc7c153320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc60023630 0x7fcc60025ac0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 --2- 172.21.15.16:0/1589541831 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc7c09e2c0 0x7fcc7c09dba0 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 -- 172.21.15.16:0/1589541831 shutdown_connections 2019-12-05T06:41:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 -- 172.21.15.16:0/1589541831 wait complete. 2019-12-05T06:41:01.454 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:01.443+0000 7fcc8093b700 1 -- 172.21.15.16:0/1589541831 >> 172.21.15.16:0/1589541831 conn(0x7fcc7c06d260 msgr2=0x7fcc7c09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:01.455 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:01.455 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:01.456 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:41:03.457 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:03.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.711+0000 7fb30ac9a700 1 Processor -- start 2019-12-05T06:41:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.712+0000 7fb30ac9a700 1 -- start start 2019-12-05T06:41:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.713+0000 7fb30ac9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb30408c6d0 0x7fb30408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.713+0000 7fb30ac9a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb30408f720 0x7fb30408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.713+0000 7fb30ac9a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb30408fb80 con 0x7fb30408fef0 2019-12-05T06:41:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.713+0000 7fb30ac9a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb30408df90 con 0x7fb30408c6d0 2019-12-05T06:41:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.713+0000 7fb30ac9a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3040a3f70 con 0x7fb30408f720 2019-12-05T06:41:03.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.713+0000 7fb309237700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb30408c6d0 0x7fb30408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.713+0000 7fb309237700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb30408c6d0 0x7fb30408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58630/0 (socket says 172.21.15.16:58630) 2019-12-05T06:41:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.713+0000 7fb309237700 1 -- 172.21.15.16:0/3602015412 learned_addr learned my addr 172.21.15.16:0/3602015412 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.713+0000 7fb308a36700 1 --1- 172.21.15.16:0/3602015412 >> v1:172.21.15.76:6789/0 conn(0x7fb30408fef0 0x7fb30408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:36870/0 (socket says 172.21.15.16:36870) 2019-12-05T06:41:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.713+0000 7fb309a38700 1 --2- 172.21.15.16:0/3602015412 >> v2:172.21.15.76:3300/0 conn(0x7fb30408f720 0x7fb30408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb309237700 1 -- 172.21.15.16:0/3602015412 >> v2:172.21.15.76:3300/0 conn(0x7fb30408f720 msgr2=0x7fb30408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:41:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb309237700 1 --2- 172.21.15.16:0/3602015412 >> v2:172.21.15.76:3300/0 conn(0x7fb30408f720 0x7fb30408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:03.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb309237700 1 -- 172.21.15.16:0/3602015412 >> v1:172.21.15.76:6789/0 conn(0x7fb30408fef0 legacy=0x7fb30408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb309237700 1 -- 172.21.15.16:0/3602015412 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30408d540 con 0x7fb30408c6d0 2019-12-05T06:41:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb309237700 1 --2- 172.21.15.16:0/3602015412 >> v2:172.21.15.16:3300/0 conn(0x7fb30408c6d0 0x7fb30408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb2fc00fac0 tx=0x7fb2fc00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb309237700 1 --2- 172.21.15.16:0/3602015412 >> v2:172.21.15.16:3300/0 conn(0x7fb30408c6d0 0x7fb30408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb2fc00fac0 tx=0x7fb2fc00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb309237700 1 --2- 172.21.15.16:0/3602015412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb30408c6d0 0x7fb30408cac0 secure :-1 s=READY pgs=1146 cs=0 l=1 rx=0x7fb2fc00fac0 tx=0x7fb2fc00f8f0).ready entity=mon.0 client_cookie=ca5e61f84b6d99c1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb309237700 1 --2- 172.21.15.16:0/3602015412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb30408c6d0 0x7fb30408cac0 secure :-1 s=THROTTLE_DONE pgs=1146 cs=0 l=1 rx=0x7fb2fc00fac0 tx=0x7fb2fc00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb309237700 1 --2- 172.21.15.16:0/3602015412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb30408c6d0 0x7fb30408cac0 secure :-1 s=THROTTLE_DONE pgs=1146 cs=0 l=1 rx=0x7fb2fc00fac0 tx=0x7fb2fc00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb303fff700 1 -- 172.21.15.16:0/3602015412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2fc000ce0 con 0x7fb30408c6d0 2019-12-05T06:41:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.714+0000 7fb309237700 1 --2- 172.21.15.16:0/3602015412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb30408c6d0 0x7fb30408cac0 secure :-1 s=THROTTLE_DONE pgs=1146 cs=0 l=1 rx=0x7fb2fc00fac0 tx=0x7fb2fc00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.715+0000 7fb303fff700 1 -- 172.21.15.16:0/3602015412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2fc001010 con 0x7fb30408c6d0 2019-12-05T06:41:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.715+0000 7fb303fff700 1 -- 172.21.15.16:0/3602015412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2fc001720 con 0x7fb30408c6d0 2019-12-05T06:41:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.715+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3602015412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb30408c6d0 msgr2=0x7fb30408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:03.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.715+0000 7fb30ac9a700 1 --2- 172.21.15.16:0/3602015412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb30408c6d0 0x7fb30408cac0 secure :-1 s=READY pgs=1146 cs=0 l=1 rx=0x7fb2fc00fac0 tx=0x7fb2fc00f8f0).stop 2019-12-05T06:41:03.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.716+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3602015412 shutdown_connections 2019-12-05T06:41:03.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.716+0000 7fb30ac9a700 1 --2- 172.21.15.16:0/3602015412 >> v2:172.21.15.76:3300/0 conn(0x7fb30408f720 0x7fb30408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:03.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.716+0000 7fb30ac9a700 1 --2- 172.21.15.16:0/3602015412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb30408c6d0 0x7fb30408cac0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:03.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.716+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3602015412 shutdown_connections 2019-12-05T06:41:03.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.716+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3602015412 wait complete. 2019-12-05T06:41:03.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.716+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3602015412 >> 172.21.15.16:0/3602015412 conn(0x7fb304061940 msgr2=0x7fb304096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.720+0000 7fb30ac9a700 1 Processor -- start 2019-12-05T06:41:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.720+0000 7fb30ac9a700 1 -- start start 2019-12-05T06:41:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.722+0000 7fb30ac9a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb30414d890 0x7fb30412eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.722+0000 7fb30ac9a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb30412c070 con 0x7fb30414d4a0 2019-12-05T06:41:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.722+0000 7fb30ac9a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb304153140 con 0x7fb30412f2b0 2019-12-05T06:41:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.722+0000 7fb30ac9a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb304098f80 con 0x7fb30414d890 2019-12-05T06:41:03.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.722+0000 7fb309237700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb30414d4a0 0x7fb30412e2e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43804/0 (socket says 172.21.15.16:43804) 2019-12-05T06:41:03.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.722+0000 7fb309237700 1 -- 172.21.15.16:0/3535827753 learned_addr learned my addr 172.21.15.16:0/3535827753 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:03.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.722+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> v2:172.21.15.76:3300/0 conn(0x7fb30414d890 0x7fb30412eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:03.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.722+0000 7fb301ffb700 1 -- 172.21.15.16:0/3535827753 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638006826 0 0) 0x7fb2fc007d20 con 0x7fb30414d4a0 2019-12-05T06:41:03.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.723+0000 7fb301ffb700 1 -- 172.21.15.16:0/3535827753 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2e4002e80 con 0x7fb30414d4a0 2019-12-05T06:41:03.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.723+0000 7fb309a38700 1 -- 172.21.15.16:0/3535827753 >> v1:172.21.15.76:6789/0 conn(0x7fb30412f2b0 legacy=0x7fb3041993f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:03.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.723+0000 7fb309a38700 1 -- 172.21.15.16:0/3535827753 >> v1:172.21.15.16:6789/0 conn(0x7fb30414d4a0 legacy=0x7fb30412e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:03.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.723+0000 7fb309a38700 1 -- 172.21.15.16:0/3535827753 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb30412d9e0 con 0x7fb30414d890 2019-12-05T06:41:03.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.723+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> v2:172.21.15.76:3300/0 conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.723+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> v2:172.21.15.76:3300/0 conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.723+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=READY pgs=804 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:03.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.724+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.724+0000 7fb301ffb700 1 -- 172.21.15.16:0/3535827753 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2f4009990 con 0x7fb30414d890 2019-12-05T06:41:03.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.724+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.724+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.724+0000 7fb301ffb700 1 -- 172.21.15.16:0/3535827753 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2f40095b0 con 0x7fb30414d890 2019-12-05T06:41:03.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.724+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3535827753 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb30414dcd0 con 0x7fb30414d890 2019-12-05T06:41:03.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.724+0000 7fb301ffb700 1 -- 172.21.15.16:0/3535827753 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2f400a010 con 0x7fb30414d890 2019-12-05T06:41:03.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.724+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3535827753 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb30412d9e0 con 0x7fb30414d890 2019-12-05T06:41:03.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.725+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.725+0000 7fb301ffb700 1 -- 172.21.15.16:0/3535827753 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb2f40099b0 con 0x7fb30414d890 2019-12-05T06:41:03.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.725+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.726+0000 7fb301ffb700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2e4023670 0x7fb2e4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:03.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.726+0000 7fb301ffb700 1 -- 172.21.15.16:0/3535827753 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7fb2f4045df0 con 0x7fb30414d890 2019-12-05T06:41:03.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.726+0000 7fb309237700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2e4023670 0x7fb2e4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:03.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.727+0000 7fb309237700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2e4023670 0x7fb2e4025ae0 crc :-1 s=READY pgs=493 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:03.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.727+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3535827753 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb30414dcd0 con 0x7fb30414d890 2019-12-05T06:41:03.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.736+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.736+0000 7fb301ffb700 1 -- 172.21.15.16:0/3535827753 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb2f40097a0 con 0x7fb30414d890 2019-12-05T06:41:03.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.956+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3535827753 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb30412d9e0 con 0x7fb30414d890 2019-12-05T06:41:03.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.957+0000 7fb309a38700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=THROTTLE_DONE pgs=804 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:03.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.957+0000 7fb301ffb700 1 -- 172.21.15.16:0/3535827753 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb2f400a0f0 con 0x7fb30414d890 2019-12-05T06:41:03.959 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 142 2019-12-05T06:41:03.960 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:03.960 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:03.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.961+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2e4023670 msgr2=0x7fb2e4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:03.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.961+0000 7fb30ac9a700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2e4023670 0x7fb2e4025ae0 crc :-1 s=READY pgs=493 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:03.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.961+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 msgr2=0x7fb30412eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:03.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.961+0000 7fb30ac9a700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 0x7fb30412eae0 secure :-1 s=READY pgs=804 cs=0 l=1 rx=0x7fb2f4010390 tx=0x7fb2f4010730).stop 2019-12-05T06:41:03.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.961+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3535827753 shutdown_connections 2019-12-05T06:41:03.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.961+0000 7fb30ac9a700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb2e4023670 0x7fb2e4025ae0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:03.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.961+0000 7fb30ac9a700 1 --2- 172.21.15.16:0/3535827753 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb30414d890 0x7fb30412eae0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:03.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.962+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3535827753 shutdown_connections 2019-12-05T06:41:03.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.962+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3535827753 wait complete. 2019-12-05T06:41:03.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:03.962+0000 7fb30ac9a700 1 -- 172.21.15.16:0/3535827753 >> 172.21.15.16:0/3535827753 conn(0x7fb30412d0e0 msgr2=0x7fb304135a40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:03.977 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:41:05.978 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.256+0000 7fafae912700 1 Processor -- start 2019-12-05T06:41:06.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.256+0000 7fafae912700 1 -- start start 2019-12-05T06:41:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.258+0000 7fafae912700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fafa808c6d0 0x7fafa808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.258+0000 7fafae912700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fafa808f720 0x7fafa808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.258+0000 7fafae912700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa808fb80 con 0x7fafa808fef0 2019-12-05T06:41:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.258+0000 7fafae912700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa808df90 con 0x7fafa808c6d0 2019-12-05T06:41:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.258+0000 7fafae912700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa80a3f70 con 0x7fafa808f720 2019-12-05T06:41:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.258+0000 7fafaceaf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fafa808c6d0 0x7fafa808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:06.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.258+0000 7fafaceaf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fafa808c6d0 0x7fafa808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58646/0 (socket says 172.21.15.16:58646) 2019-12-05T06:41:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.258+0000 7fafaceaf700 1 -- 172.21.15.16:0/2529519531 learned_addr learned my addr 172.21.15.16:0/2529519531 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:06.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.258+0000 7fafad6b0700 1 --2- 172.21.15.16:0/2529519531 >> v2:172.21.15.76:3300/0 conn(0x7fafa808f720 0x7fafa808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:06.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.259+0000 7fafaceaf700 1 -- 172.21.15.16:0/2529519531 >> v2:172.21.15.76:3300/0 conn(0x7fafa808f720 msgr2=0x7fafa808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:41:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.259+0000 7fafaceaf700 1 --2- 172.21.15.16:0/2529519531 >> v2:172.21.15.76:3300/0 conn(0x7fafa808f720 0x7fafa808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.259+0000 7fafaceaf700 1 -- 172.21.15.16:0/2529519531 >> v1:172.21.15.16:6789/0 conn(0x7fafa808fef0 legacy=0x7fafa808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.259+0000 7fafaceaf700 1 -- 172.21.15.16:0/2529519531 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa808d540 con 0x7fafa808c6d0 2019-12-05T06:41:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.259+0000 7fafad6b0700 1 --2- 172.21.15.16:0/2529519531 >> v2:172.21.15.76:3300/0 conn(0x7fafa808f720 0x7fafa808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:41:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.259+0000 7fafaceaf700 1 --2- 172.21.15.16:0/2529519531 >> v2:172.21.15.16:3300/0 conn(0x7fafa808c6d0 0x7fafa808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faf9000f740 tx=0x7faf9000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.259+0000 7fafaceaf700 1 --2- 172.21.15.16:0/2529519531 >> v2:172.21.15.16:3300/0 conn(0x7fafa808c6d0 0x7fafa808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faf9000f740 tx=0x7faf9000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.259+0000 7fafaceaf700 1 --2- 172.21.15.16:0/2529519531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa808c6d0 0x7fafa808cac0 secure :-1 s=READY pgs=1149 cs=0 l=1 rx=0x7faf9000f740 tx=0x7faf9000fae0).ready entity=mon.0 client_cookie=4dbc28504105ed75 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.260+0000 7fafaceaf700 1 --2- 172.21.15.16:0/2529519531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa808c6d0 0x7fafa808cac0 secure :-1 s=THROTTLE_DONE pgs=1149 cs=0 l=1 rx=0x7faf9000f740 tx=0x7faf9000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.260+0000 7fafaceaf700 1 --2- 172.21.15.16:0/2529519531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa808c6d0 0x7fafa808cac0 secure :-1 s=THROTTLE_DONE pgs=1149 cs=0 l=1 rx=0x7faf9000f740 tx=0x7faf9000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.260+0000 7fafa77fe700 1 -- 172.21.15.16:0/2529519531 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf90000960 con 0x7fafa808c6d0 2019-12-05T06:41:06.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.260+0000 7fafaceaf700 1 --2- 172.21.15.16:0/2529519531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa808c6d0 0x7fafa808cac0 secure :-1 s=THROTTLE_DONE pgs=1149 cs=0 l=1 rx=0x7faf9000f740 tx=0x7faf9000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.260+0000 7fafa77fe700 1 -- 172.21.15.16:0/2529519531 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf90000c90 con 0x7fafa808c6d0 2019-12-05T06:41:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.260+0000 7fafa77fe700 1 -- 172.21.15.16:0/2529519531 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf900013a0 con 0x7fafa808c6d0 2019-12-05T06:41:06.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.260+0000 7fafae912700 1 -- 172.21.15.16:0/2529519531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa808c6d0 msgr2=0x7fafa808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.260+0000 7fafae912700 1 --2- 172.21.15.16:0/2529519531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa808c6d0 0x7fafa808cac0 secure :-1 s=READY pgs=1149 cs=0 l=1 rx=0x7faf9000f740 tx=0x7faf9000fae0).stop 2019-12-05T06:41:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.262+0000 7fafae912700 1 -- 172.21.15.16:0/2529519531 shutdown_connections 2019-12-05T06:41:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.262+0000 7fafae912700 1 --2- 172.21.15.16:0/2529519531 >> v2:172.21.15.76:3300/0 conn(0x7fafa808f720 0x7fafa808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:06.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.262+0000 7fafae912700 1 --2- 172.21.15.16:0/2529519531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fafa808c6d0 0x7fafa808cac0 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.262+0000 7fafae912700 1 -- 172.21.15.16:0/2529519531 shutdown_connections 2019-12-05T06:41:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.262+0000 7fafae912700 1 -- 172.21.15.16:0/2529519531 wait complete. 2019-12-05T06:41:06.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.262+0000 7fafae912700 1 -- 172.21.15.16:0/2529519531 >> 172.21.15.16:0/2529519531 conn(0x7fafa8061940 msgr2=0x7fafa8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.266+0000 7fafae912700 1 Processor -- start 2019-12-05T06:41:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.266+0000 7fafae912700 1 -- start start 2019-12-05T06:41:06.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.268+0000 7fafae912700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fafa806dc10 0x7fafa806e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.268+0000 7fafae912700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa8130870 con 0x7fafa8130be0 2019-12-05T06:41:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.268+0000 7fafae912700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafa8128690 con 0x7fafa8130410 2019-12-05T06:41:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.268+0000 7fafae912700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafa809e690 con 0x7fafa806dc10 2019-12-05T06:41:06.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.268+0000 7fafa7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fafa8130be0 0x7fafa812fc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43822/0 (socket says 172.21.15.16:43822) 2019-12-05T06:41:06.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.269+0000 7fafa7fff700 1 -- 172.21.15.16:0/1521865182 learned_addr learned my addr 172.21.15.16:0/1521865182 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:06.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.269+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> v2:172.21.15.76:3300/0 conn(0x7fafa806dc10 0x7fafa806e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:06.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.269+0000 7fafa57fa700 1 -- 172.21.15.16:0/1521865182 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2723581959 0 0) 0x7faf98001100 con 0x7fafa8130be0 2019-12-05T06:41:06.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.269+0000 7fafa57fa700 1 -- 172.21.15.16:0/1521865182 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf8c002e80 con 0x7fafa8130be0 2019-12-05T06:41:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.269+0000 7fafaceaf700 1 -- 172.21.15.16:0/1521865182 >> v1:172.21.15.76:6789/0 conn(0x7fafa8130410 legacy=0x7fafa81517c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.269+0000 7fafaceaf700 1 -- 172.21.15.16:0/1521865182 >> v1:172.21.15.16:6789/0 conn(0x7fafa8130be0 legacy=0x7fafa812fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.269+0000 7fafa57fa700 1 -- 172.21.15.16:0/1521865182 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547524288 0 0) 0x7fafa0009b40 con 0x7fafa8130410 2019-12-05T06:41:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.269+0000 7fafaceaf700 1 -- 172.21.15.16:0/1521865182 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa812f310 con 0x7fafa806dc10 2019-12-05T06:41:06.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> v2:172.21.15.76:3300/0 conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> v2:172.21.15.76:3300/0 conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=READY pgs=806 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:06.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=THROTTLE_DONE pgs=806 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafa57fa700 1 -- 172.21.15.16:0/1521865182 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf90005560 con 0x7fafa806dc10 2019-12-05T06:41:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=THROTTLE_DONE pgs=806 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=THROTTLE_DONE pgs=806 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafa57fa700 1 -- 172.21.15.16:0/1521865182 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf90005180 con 0x7fafa806dc10 2019-12-05T06:41:06.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafae912700 1 -- 172.21.15.16:0/1521865182 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafa812f310 con 0x7fafa806dc10 2019-12-05T06:41:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafa57fa700 1 -- 172.21.15.16:0/1521865182 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf90005b40 con 0x7fafa806dc10 2019-12-05T06:41:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.270+0000 7fafae912700 1 -- 172.21.15.16:0/1521865182 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafa812f310 con 0x7fafa806dc10 2019-12-05T06:41:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.271+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=THROTTLE_DONE pgs=806 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.272+0000 7fafa57fa700 1 -- 172.21.15.16:0/1521865182 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faf90005370 con 0x7fafa806dc10 2019-12-05T06:41:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.272+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=THROTTLE_DONE pgs=806 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.273+0000 7fafa57fa700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf8c023680 0x7faf8c025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:06.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.273+0000 7fafa57fa700 1 -- 172.21.15.16:0/1521865182 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7faf90045da0 con 0x7fafa806dc10 2019-12-05T06:41:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.273+0000 7fafa7fff700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf8c023680 0x7faf8c025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:06.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.273+0000 7fafa7fff700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf8c023680 0x7faf8c025af0 crc :-1 s=READY pgs=494 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:06.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.288+0000 7fafae912700 1 -- 172.21.15.16:0/1521865182 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafa812f310 con 0x7fafa806dc10 2019-12-05T06:41:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.297+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=THROTTLE_DONE pgs=806 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.297+0000 7fafa57fa700 1 -- 172.21.15.16:0/1521865182 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faf90007310 con 0x7fafa806dc10 2019-12-05T06:41:06.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.550+0000 7fafae912700 1 -- 172.21.15.16:0/1521865182 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fafa812f310 con 0x7fafa806dc10 2019-12-05T06:41:06.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.551+0000 7fafaceaf700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=THROTTLE_DONE pgs=806 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:06.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.551+0000 7fafa57fa700 1 -- 172.21.15.16:0/1521865182 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+2112 (secure 0 0 0) 0x7faf90005370 con 0x7fafa806dc10 2019-12-05T06:41:06.555 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 142 2019-12-05T06:41:06.555 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:06.556 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:06.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.556+0000 7fafae912700 1 -- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf8c023680 msgr2=0x7faf8c025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:06.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.556+0000 7fafae912700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf8c023680 0x7faf8c025af0 crc :-1 s=READY pgs=494 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:06.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.556+0000 7fafae912700 1 -- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 msgr2=0x7fafa806e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:06.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.556+0000 7fafae912700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 0x7fafa806e000 secure :-1 s=READY pgs=806 cs=0 l=1 rx=0x7faf900103a0 tx=0x7faf90010740).stop 2019-12-05T06:41:06.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.556+0000 7fafae912700 1 -- 172.21.15.16:0/1521865182 shutdown_connections 2019-12-05T06:41:06.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.556+0000 7fafae912700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf8c023680 0x7faf8c025af0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:06.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.556+0000 7fafae912700 1 --2- 172.21.15.16:0/1521865182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fafa806dc10 0x7fafa806e000 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:06.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.556+0000 7fafae912700 1 -- 172.21.15.16:0/1521865182 shutdown_connections 2019-12-05T06:41:06.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.556+0000 7fafae912700 1 -- 172.21.15.16:0/1521865182 wait complete. 2019-12-05T06:41:06.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:06.556+0000 7fafae912700 1 -- 172.21.15.16:0/1521865182 >> 172.21.15.16:0/1521865182 conn(0x7fafa8098480 msgr2=0x7fafa8071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:06.566 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:41:08.567 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:08.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.828+0000 7fb70df18700 1 Processor -- start 2019-12-05T06:41:08.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.828+0000 7fb70df18700 1 -- start start 2019-12-05T06:41:08.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.830+0000 7fb70df18700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 0x7fb70808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:08.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.830+0000 7fb70df18700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7080958e0 con 0x7fb70808fef0 2019-12-05T06:41:08.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.830+0000 7fb70df18700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7080629e0 con 0x7fb70808f720 2019-12-05T06:41:08.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.830+0000 7fb70df18700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7080a3f70 con 0x7fb70808c6d0 2019-12-05T06:41:08.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.830+0000 7fb7077fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb70808fef0 0x7fb70808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43830/0 (socket says 172.21.15.16:43830) 2019-12-05T06:41:08.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.830+0000 7fb7077fe700 1 -- 172.21.15.16:0/2357861596 learned_addr learned my addr 172.21.15.16:0/2357861596 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:08.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.830+0000 7fb707fff700 1 --2- 172.21.15.16:0/2357861596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 0x7fb70808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:08.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.831+0000 7fb706ffd700 1 -- 172.21.15.16:0/2357861596 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495760474 0 0) 0x7fb6f0001150 con 0x7fb70808fef0 2019-12-05T06:41:08.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.831+0000 7fb706ffd700 1 -- 172.21.15.16:0/2357861596 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f4002e80 con 0x7fb70808fef0 2019-12-05T06:41:08.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.831+0000 7fb706ffd700 1 -- 172.21.15.16:0/2357861596 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 517444403 0 0) 0x7fb700001080 con 0x7fb70808f720 2019-12-05T06:41:08.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.831+0000 7fb706ffd700 1 -- 172.21.15.16:0/2357861596 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6f4004170 con 0x7fb70808f720 2019-12-05T06:41:08.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.831+0000 7fb707fff700 1 -- 172.21.15.16:0/2357861596 >> v1:172.21.15.76:6789/0 conn(0x7fb70808f720 legacy=0x7fb7081533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:08.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.831+0000 7fb707fff700 1 -- 172.21.15.16:0/2357861596 >> v1:172.21.15.16:6789/0 conn(0x7fb70808fef0 legacy=0x7fb70808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.831+0000 7fb707fff700 1 -- 172.21.15.16:0/2357861596 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70808fbc0 con 0x7fb70808c6d0 2019-12-05T06:41:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.831+0000 7fb707fff700 1 --2- 172.21.15.16:0/2357861596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 0x7fb70808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb6f800b2d0 tx=0x7fb6f800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.831+0000 7fb707fff700 1 --2- 172.21.15.16:0/2357861596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 0x7fb70808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb6f800b2d0 tx=0x7fb6f800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.831+0000 7fb707fff700 1 --2- 172.21.15.16:0/2357861596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 0x7fb70808cac0 secure :-1 s=READY pgs=271 cs=0 l=1 rx=0x7fb6f800b2d0 tx=0x7fb6f800bae0).ready entity=mon.2 client_cookie=9519d8fc538452f9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.832+0000 7fb707fff700 1 --2- 172.21.15.16:0/2357861596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 0x7fb70808cac0 secure :-1 s=THROTTLE_DONE pgs=271 cs=0 l=1 rx=0x7fb6f800b2d0 tx=0x7fb6f800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.832+0000 7fb706ffd700 1 -- 172.21.15.16:0/2357861596 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6f8000ce0 con 0x7fb70808c6d0 2019-12-05T06:41:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.832+0000 7fb707fff700 1 --2- 172.21.15.16:0/2357861596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 0x7fb70808cac0 secure :-1 s=THROTTLE_DONE pgs=271 cs=0 l=1 rx=0x7fb6f800b2d0 tx=0x7fb6f800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.832+0000 7fb707fff700 1 --2- 172.21.15.16:0/2357861596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 0x7fb70808cac0 secure :-1 s=THROTTLE_DONE pgs=271 cs=0 l=1 rx=0x7fb6f800b2d0 tx=0x7fb6f800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.832+0000 7fb706ffd700 1 -- 172.21.15.16:0/2357861596 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f8009be0 con 0x7fb70808c6d0 2019-12-05T06:41:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.832+0000 7fb706ffd700 1 -- 172.21.15.16:0/2357861596 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6f800a2b0 con 0x7fb70808c6d0 2019-12-05T06:41:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.832+0000 7fb70df18700 1 -- 172.21.15.16:0/2357861596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 msgr2=0x7fb70808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.832+0000 7fb70df18700 1 --2- 172.21.15.16:0/2357861596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 0x7fb70808cac0 secure :-1 s=READY pgs=271 cs=0 l=1 rx=0x7fb6f800b2d0 tx=0x7fb6f800bae0).stop 2019-12-05T06:41:08.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.833+0000 7fb70df18700 1 -- 172.21.15.16:0/2357861596 shutdown_connections 2019-12-05T06:41:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.833+0000 7fb70df18700 1 --2- 172.21.15.16:0/2357861596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70808c6d0 0x7fb70808cac0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.833+0000 7fb70df18700 1 -- 172.21.15.16:0/2357861596 shutdown_connections 2019-12-05T06:41:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.833+0000 7fb70df18700 1 -- 172.21.15.16:0/2357861596 wait complete. 2019-12-05T06:41:08.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.834+0000 7fb70df18700 1 -- 172.21.15.16:0/2357861596 >> 172.21.15.16:0/2357861596 conn(0x7fb708061940 msgr2=0x7fb708096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.838+0000 7fb70df18700 1 Processor -- start 2019-12-05T06:41:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.839+0000 7fb70df18700 1 -- start start 2019-12-05T06:41:08.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.840+0000 7fb70df18700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb708150b40 0x7fb708150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.840+0000 7fb70df18700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70807fc20 0x7fb70807ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.840+0000 7fb70df18700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb70807bfc0 con 0x7fb70807f190 2019-12-05T06:41:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.840+0000 7fb70df18700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb70807dcc0 con 0x7fb708150b40 2019-12-05T06:41:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.840+0000 7fb70df18700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb70807d560 con 0x7fb70807fc20 2019-12-05T06:41:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.840+0000 7fb70ccb6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb70807f190 0x7fb708153790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43838/0 (socket says 172.21.15.16:43838) 2019-12-05T06:41:08.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.840+0000 7fb70ccb6700 1 -- 172.21.15.16:0/3538275692 learned_addr learned my addr 172.21.15.16:0/3538275692 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:08.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.841+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> v2:172.21.15.76:3300/0 conn(0x7fb708150b40 0x7fb708150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:08.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.841+0000 7fb7077fe700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70807fc20 0x7fb70807ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:08.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.841+0000 7fb704ff9700 1 -- 172.21.15.16:0/3538275692 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 971288413 0 0) 0x7fb700000d90 con 0x7fb70807f190 2019-12-05T06:41:08.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.841+0000 7fb704ff9700 1 -- 172.21.15.16:0/3538275692 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6ec002e80 con 0x7fb70807f190 2019-12-05T06:41:08.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.841+0000 7fb707fff700 1 -- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70807fc20 msgr2=0x7fb70807ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:08.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.841+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70807fc20 0x7fb70807ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:08.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.841+0000 7fb707fff700 1 -- 172.21.15.16:0/3538275692 >> v1:172.21.15.16:6789/0 conn(0x7fb70807f190 legacy=0x7fb708153790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:08.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.841+0000 7fb707fff700 1 -- 172.21.15.16:0/3538275692 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb70807e350 con 0x7fb708150b40 2019-12-05T06:41:08.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.841+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> v2:172.21.15.76:3300/0 conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.841+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> v2:172.21.15.76:3300/0 conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.842+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=READY pgs=808 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:08.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.842+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.842+0000 7fb704ff9700 1 -- 172.21.15.16:0/3538275692 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6f8000cc0 con 0x7fb708150b40 2019-12-05T06:41:08.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.842+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.842+0000 7fb70df18700 1 -- 172.21.15.16:0/3538275692 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb70807e350 con 0x7fb708150b40 2019-12-05T06:41:08.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.842+0000 7fb704ff9700 1 -- 172.21.15.16:0/3538275692 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb6f8000cc0 con 0x7fb708150b40 2019-12-05T06:41:08.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.842+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.842+0000 7fb704ff9700 1 -- 172.21.15.16:0/3538275692 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb6f8009ef0 con 0x7fb708150b40 2019-12-05T06:41:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.842+0000 7fb70df18700 1 -- 172.21.15.16:0/3538275692 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb70807e350 con 0x7fb708150b40 2019-12-05T06:41:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.843+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.844+0000 7fb704ff9700 1 -- 172.21.15.16:0/3538275692 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb6f800a650 con 0x7fb708150b40 2019-12-05T06:41:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.844+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.845+0000 7fb704ff9700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb6ec023670 0x7fb6ec025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.845+0000 7fb704ff9700 1 -- 172.21.15.16:0/3538275692 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7fb6f8046120 con 0x7fb708150b40 2019-12-05T06:41:08.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.845+0000 7fb7077fe700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb6ec023670 0x7fb6ec025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:08.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.845+0000 7fb7077fe700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb6ec023670 0x7fb6ec025ae0 crc :-1 s=READY pgs=495 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:08.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.859+0000 7fb70df18700 1 -- 172.21.15.16:0/3538275692 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb70807e350 con 0x7fb708150b40 2019-12-05T06:41:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.869+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:08.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:08.869+0000 7fb704ff9700 1 -- 172.21.15.16:0/3538275692 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb6f800a440 con 0x7fb708150b40 2019-12-05T06:41:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.097+0000 7fb70df18700 1 -- 172.21.15.16:0/3538275692 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb70807e350 con 0x7fb708150b40 2019-12-05T06:41:09.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.098+0000 7fb707fff700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=THROTTLE_DONE pgs=808 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.098+0000 7fb704ff9700 1 -- 172.21.15.16:0/3538275692 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb6f800a650 con 0x7fb708150b40 2019-12-05T06:41:09.101 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 142 2019-12-05T06:41:09.101 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:09.101 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:09.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.102+0000 7fb70df18700 1 -- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb6ec023670 msgr2=0x7fb6ec025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.102+0000 7fb70df18700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb6ec023670 0x7fb6ec025ae0 crc :-1 s=READY pgs=495 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.102+0000 7fb70df18700 1 -- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 msgr2=0x7fb708150f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:09.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.102+0000 7fb70df18700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 0x7fb708150f30 secure :-1 s=READY pgs=808 cs=0 l=1 rx=0x7fb6f8010710 tx=0x7fb6f8010ab0).stop 2019-12-05T06:41:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.103+0000 7fb70df18700 1 -- 172.21.15.16:0/3538275692 shutdown_connections 2019-12-05T06:41:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.103+0000 7fb70df18700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb6ec023670 0x7fb6ec025ae0 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.103+0000 7fb70df18700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb70807fc20 0x7fb70807ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:09.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.103+0000 7fb70df18700 1 --2- 172.21.15.16:0/3538275692 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb708150b40 0x7fb708150f30 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.103+0000 7fb70df18700 1 -- 172.21.15.16:0/3538275692 shutdown_connections 2019-12-05T06:41:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.103+0000 7fb70df18700 1 -- 172.21.15.16:0/3538275692 wait complete. 2019-12-05T06:41:09.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:09.103+0000 7fb70df18700 1 -- 172.21.15.16:0/3538275692 >> 172.21.15.16:0/3538275692 conn(0x7fb708084520 msgr2=0x7fb7080995a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:09.118 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:41:11.120 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:11.226 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:41:11.231 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:41:11.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.367+0000 7fd85a4a5700 1 Processor -- start 2019-12-05T06:41:11.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.368+0000 7fd85a4a5700 1 -- start start 2019-12-05T06:41:11.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd85a4a5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 0x7fd85408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd85a4a5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd85408fef0 0x7fd85408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd85a4a5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8540958e0 con 0x7fd85408ef20 2019-12-05T06:41:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd85a4a5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8540a3f70 con 0x7fd85408fef0 2019-12-05T06:41:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd85a4a5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd85408d830 con 0x7fd85408c6d0 2019-12-05T06:41:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd859243700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd85408ef20 0x7fd85408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43848/0 (socket says 172.21.15.16:43848) 2019-12-05T06:41:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd859243700 1 -- 172.21.15.16:0/3238991788 learned_addr learned my addr 172.21.15.16:0/3238991788 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:11.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd858a42700 1 --2- 172.21.15.16:0/3238991788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 0x7fd85408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd853fff700 1 --2- 172.21.15.16:0/3238991788 >> v2:172.21.15.76:3300/0 conn(0x7fd85408fef0 0x7fd85408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd8537fe700 1 -- 172.21.15.16:0/3238991788 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454092389 0 0) 0x7fd84c0014d0 con 0x7fd85408ef20 2019-12-05T06:41:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.369+0000 7fd8537fe700 1 -- 172.21.15.16:0/3238991788 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd83c002e80 con 0x7fd85408ef20 2019-12-05T06:41:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.370+0000 7fd8537fe700 1 -- 172.21.15.16:0/3238991788 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4028313150 0 0) 0x7fd84c0014b0 con 0x7fd85408ef20 2019-12-05T06:41:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.370+0000 7fd858a42700 1 -- 172.21.15.16:0/3238991788 >> v2:172.21.15.76:3300/0 conn(0x7fd85408fef0 msgr2=0x7fd85408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:41:11.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.370+0000 7fd858a42700 1 --2- 172.21.15.16:0/3238991788 >> v2:172.21.15.76:3300/0 conn(0x7fd85408fef0 0x7fd85408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.370+0000 7fd858a42700 1 -- 172.21.15.16:0/3238991788 >> v1:172.21.15.16:6789/0 conn(0x7fd85408ef20 legacy=0x7fd85408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.370+0000 7fd858a42700 1 -- 172.21.15.16:0/3238991788 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd85408d540 con 0x7fd85408c6d0 2019-12-05T06:41:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.370+0000 7fd858a42700 1 --2- 172.21.15.16:0/3238991788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 0x7fd85408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd84400af50 tx=0x7fd84400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.370+0000 7fd858a42700 1 --2- 172.21.15.16:0/3238991788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 0x7fd85408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd84400af50 tx=0x7fd84400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.370+0000 7fd858a42700 1 --2- 172.21.15.16:0/3238991788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 0x7fd85408cac0 secure :-1 s=READY pgs=272 cs=0 l=1 rx=0x7fd84400af50 tx=0x7fd84400b250).ready entity=mon.2 client_cookie=ba0b0bfe6e577f17 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.370+0000 7fd858a42700 1 --2- 172.21.15.16:0/3238991788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 0x7fd85408cac0 secure :-1 s=THROTTLE_DONE pgs=272 cs=0 l=1 rx=0x7fd84400af50 tx=0x7fd84400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.371+0000 7fd8537fe700 1 -- 172.21.15.16:0/3238991788 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd844007210 con 0x7fd85408c6d0 2019-12-05T06:41:11.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.371+0000 7fd858a42700 1 --2- 172.21.15.16:0/3238991788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 0x7fd85408cac0 secure :-1 s=THROTTLE_DONE pgs=272 cs=0 l=1 rx=0x7fd84400af50 tx=0x7fd84400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.371+0000 7fd858a42700 1 --2- 172.21.15.16:0/3238991788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 0x7fd85408cac0 secure :-1 s=THROTTLE_DONE pgs=272 cs=0 l=1 rx=0x7fd84400af50 tx=0x7fd84400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.371+0000 7fd8537fe700 1 -- 172.21.15.16:0/3238991788 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8440099b0 con 0x7fd85408c6d0 2019-12-05T06:41:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.371+0000 7fd8537fe700 1 -- 172.21.15.16:0/3238991788 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd84400a080 con 0x7fd85408c6d0 2019-12-05T06:41:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.371+0000 7fd85a4a5700 1 -- 172.21.15.16:0/3238991788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 msgr2=0x7fd85408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.371+0000 7fd85a4a5700 1 --2- 172.21.15.16:0/3238991788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 0x7fd85408cac0 secure :-1 s=READY pgs=272 cs=0 l=1 rx=0x7fd84400af50 tx=0x7fd84400b250).stop 2019-12-05T06:41:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.372+0000 7fd85a4a5700 1 -- 172.21.15.16:0/3238991788 shutdown_connections 2019-12-05T06:41:11.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.372+0000 7fd85a4a5700 1 --2- 172.21.15.16:0/3238991788 >> v2:172.21.15.76:3300/0 conn(0x7fd85408fef0 0x7fd85408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:11.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.372+0000 7fd85a4a5700 1 --2- 172.21.15.16:0/3238991788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd85408c6d0 0x7fd85408cac0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:11.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.372+0000 7fd85a4a5700 1 -- 172.21.15.16:0/3238991788 shutdown_connections 2019-12-05T06:41:11.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.372+0000 7fd85a4a5700 1 -- 172.21.15.16:0/3238991788 wait complete. 2019-12-05T06:41:11.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.372+0000 7fd85a4a5700 1 -- 172.21.15.16:0/3238991788 >> 172.21.15.16:0/3238991788 conn(0x7fd854061940 msgr2=0x7fd854096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:11.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.376+0000 7fd85a4a5700 1 Processor -- start 2019-12-05T06:41:11.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.376+0000 7fd85a4a5700 1 -- start start 2019-12-05T06:41:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.377+0000 7fd85a4a5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd85412b640 0x7fd85412ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.377+0000 7fd85a4a5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd854153b50 0x7fd854155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.377+0000 7fd85a4a5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd854131bf0 con 0x7fd8541302f0 2019-12-05T06:41:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.377+0000 7fd85a4a5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd8541296e0 con 0x7fd85412b640 2019-12-05T06:41:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.377+0000 7fd85a4a5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd854128cb0 con 0x7fd854153b50 2019-12-05T06:41:11.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.377+0000 7fd853fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd85412b640 0x7fd85412ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd85412b640 0x7fd85412ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58684/0 (socket says 172.21.15.16:58684) 2019-12-05T06:41:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 -- 172.21.15.16:0/54062524 learned_addr learned my addr 172.21.15.16:0/54062524 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd859243700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd854153b50 0x7fd854155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 -- 172.21.15.16:0/54062524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd854153b50 msgr2=0x7fd854155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd854153b50 0x7fd854155f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 -- 172.21.15.16:0/54062524 >> v1:172.21.15.76:6789/0 conn(0x7fd8541302f0 legacy=0x7fd85412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:11.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 -- 172.21.15.16:0/54062524 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd854129d70 con 0x7fd85412b640 2019-12-05T06:41:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> v2:172.21.15.16:3300/0 conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> v2:172.21.15.16:3300/0 conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=READY pgs=1154 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.378+0000 7fd8517fa700 1 -- 172.21.15.16:0/54062524 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd848000940 con 0x7fd85412b640 2019-12-05T06:41:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.379+0000 7fd8517fa700 1 -- 172.21.15.16:0/54062524 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd848000e80 con 0x7fd85412b640 2019-12-05T06:41:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.379+0000 7fd8517fa700 1 -- 172.21.15.16:0/54062524 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd8480014d0 con 0x7fd85412b640 2019-12-05T06:41:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.379+0000 7fd85a4a5700 1 -- 172.21.15.16:0/54062524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd854129d70 con 0x7fd85412b640 2019-12-05T06:41:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.379+0000 7fd85a4a5700 1 -- 172.21.15.16:0/54062524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd854129d70 con 0x7fd85412b640 2019-12-05T06:41:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.379+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.380+0000 7fd8517fa700 1 -- 172.21.15.16:0/54062524 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd848001070 con 0x7fd85412b640 2019-12-05T06:41:11.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.380+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.380+0000 7fd8517fa700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd834023630 0x7fd834025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:11.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.380+0000 7fd8517fa700 1 -- 172.21.15.16:0/54062524 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 7923+0+0 (secure 0 0 0) 0x7fd848045de0 con 0x7fd85412b640 2019-12-05T06:41:11.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.381+0000 7fd858a42700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd834023630 0x7fd834025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:11.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.381+0000 7fd858a42700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd834023630 0x7fd834025ac0 crc :-1 s=READY pgs=496 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:11.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.382+0000 7fd85a4a5700 1 -- 172.21.15.16:0/54062524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd854129d70 con 0x7fd85412b640 2019-12-05T06:41:11.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.387+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.388+0000 7fd8517fa700 1 -- 172.21.15.16:0/54062524 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd848000d80 con 0x7fd85412b640 2019-12-05T06:41:11.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.600+0000 7fd85a4a5700 1 -- 172.21.15.16:0/54062524 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd854129d70 con 0x7fd85412b640 2019-12-05T06:41:11.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.601+0000 7fd853fff700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=THROTTLE_DONE pgs=1154 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:11.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.601+0000 7fd8517fa700 1 -- 172.21.15.16:0/54062524 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 142 v142) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fd848000d80 con 0x7fd85412b640 2019-12-05T06:41:11.602 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 142 2019-12-05T06:41:11.602 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:11.602 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":142,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7132,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":142}],"filesystems":[{"mdsmap":{"epoch":141,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:39:23.195456+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":71,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7011},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7011":{"gid":7011,"name":"a","rank":0,"incarnation":138,"state":"up:active","state_seq":34,"addr":"172.21.15.16:6833/3506755338","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3506755338},{"type":"v1","addr":"172.21.15.16:6833","nonce":3506755338}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.603+0000 7fd85a4a5700 1 -- 172.21.15.16:0/54062524 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd834023630 msgr2=0x7fd834025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.603+0000 7fd85a4a5700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd834023630 0x7fd834025ac0 crc :-1 s=READY pgs=496 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:11.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.603+0000 7fd85a4a5700 1 -- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 msgr2=0x7fd85412ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:11.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.603+0000 7fd85a4a5700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 0x7fd85412ae70 secure :-1 s=READY pgs=1154 cs=0 l=1 rx=0x7fd848010450 tx=0x7fd848010280).stop 2019-12-05T06:41:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.604+0000 7fd85a4a5700 1 -- 172.21.15.16:0/54062524 shutdown_connections 2019-12-05T06:41:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.604+0000 7fd85a4a5700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd854153b50 0x7fd854155f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.604+0000 7fd85a4a5700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd834023630 0x7fd834025ac0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.604+0000 7fd85a4a5700 1 --2- 172.21.15.16:0/54062524 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd85412b640 0x7fd85412ae70 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.604+0000 7fd85a4a5700 1 -- 172.21.15.16:0/54062524 shutdown_connections 2019-12-05T06:41:11.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.604+0000 7fd85a4a5700 1 -- 172.21.15.16:0/54062524 wait complete. 2019-12-05T06:41:11.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:11.604+0000 7fd85a4a5700 1 -- 172.21.15.16:0/54062524 >> 172.21.15.16:0/54062524 conn(0x7fd8540a2850 msgr2=0x7fd85406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:11.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:41:13.623 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:14.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.882+0000 7f59d4154700 1 Processor -- start 2019-12-05T06:41:14.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.882+0000 7f59d4154700 1 -- start start 2019-12-05T06:41:14.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.884+0000 7f59d4154700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59cc08c6d0 0x7f59cc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:14.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.884+0000 7f59d4154700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f59cc08fef0 0x7f59cc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:14.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.884+0000 7f59d4154700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc08fb80 con 0x7f59cc08ef20 2019-12-05T06:41:14.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.884+0000 7f59d4154700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59cc08df90 con 0x7f59cc08c6d0 2019-12-05T06:41:14.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.884+0000 7f59d4154700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59cc0a3f70 con 0x7f59cc08fef0 2019-12-05T06:41:14.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.884+0000 7f59d26f1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59cc08c6d0 0x7f59cc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:14.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.884+0000 7f59d26f1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59cc08c6d0 0x7f59cc08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58690/0 (socket says 172.21.15.16:58690) 2019-12-05T06:41:14.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.884+0000 7f59d26f1700 1 -- 172.21.15.16:0/3056182758 learned_addr learned my addr 172.21.15.16:0/3056182758 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:14.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.884+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/3056182758 >> v2:172.21.15.76:3300/0 conn(0x7f59cc08fef0 0x7f59cc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:14.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.885+0000 7f59d26f1700 1 -- 172.21.15.16:0/3056182758 >> v2:172.21.15.76:3300/0 conn(0x7f59cc08fef0 msgr2=0x7f59cc08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:41:14.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.885+0000 7f59d26f1700 1 --2- 172.21.15.16:0/3056182758 >> v2:172.21.15.76:3300/0 conn(0x7f59cc08fef0 0x7f59cc08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:14.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.885+0000 7f59d26f1700 1 -- 172.21.15.16:0/3056182758 >> v1:172.21.15.16:6789/0 conn(0x7f59cc08ef20 legacy=0x7f59cc08f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.885+0000 7f59d26f1700 1 -- 172.21.15.16:0/3056182758 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc08d540 con 0x7f59cc08c6d0 2019-12-05T06:41:14.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.885+0000 7f59d16ef700 1 -- 172.21.15.16:0/3056182758 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 328157766 0 0) 0x7f59bc000a40 con 0x7f59cc08ef20 2019-12-05T06:41:14.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.885+0000 7f59d26f1700 1 --2- 172.21.15.16:0/3056182758 >> v2:172.21.15.16:3300/0 conn(0x7f59cc08c6d0 0x7f59cc08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f59c400fa40 tx=0x7f59c400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.885+0000 7f59d26f1700 1 --2- 172.21.15.16:0/3056182758 >> v2:172.21.15.16:3300/0 conn(0x7f59cc08c6d0 0x7f59cc08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f59c400fa40 tx=0x7f59c400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.885+0000 7f59d26f1700 1 --2- 172.21.15.16:0/3056182758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc08c6d0 0x7f59cc08cac0 secure :-1 s=READY pgs=1156 cs=0 l=1 rx=0x7f59c400fa40 tx=0x7f59c400fde0).ready entity=mon.0 client_cookie=dab7be65bf8eac53 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:14.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.886+0000 7f59d26f1700 1 --2- 172.21.15.16:0/3056182758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc08c6d0 0x7f59cc08cac0 secure :-1 s=THROTTLE_DONE pgs=1156 cs=0 l=1 rx=0x7f59c400fa40 tx=0x7f59c400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.886+0000 7f59d16ef700 1 -- 172.21.15.16:0/3056182758 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c4000ce0 con 0x7f59cc08c6d0 2019-12-05T06:41:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.886+0000 7f59d26f1700 1 --2- 172.21.15.16:0/3056182758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc08c6d0 0x7f59cc08cac0 secure :-1 s=THROTTLE_DONE pgs=1156 cs=0 l=1 rx=0x7f59c400fa40 tx=0x7f59c400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.886+0000 7f59d26f1700 1 --2- 172.21.15.16:0/3056182758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc08c6d0 0x7f59cc08cac0 secure :-1 s=THROTTLE_DONE pgs=1156 cs=0 l=1 rx=0x7f59c400fa40 tx=0x7f59c400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.886+0000 7f59d16ef700 1 -- 172.21.15.16:0/3056182758 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c4001010 con 0x7f59cc08c6d0 2019-12-05T06:41:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.886+0000 7f59d16ef700 1 -- 172.21.15.16:0/3056182758 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c4001720 con 0x7f59cc08c6d0 2019-12-05T06:41:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.886+0000 7f59d4154700 1 -- 172.21.15.16:0/3056182758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc08c6d0 msgr2=0x7f59cc08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.886+0000 7f59d4154700 1 --2- 172.21.15.16:0/3056182758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc08c6d0 0x7f59cc08cac0 secure :-1 s=READY pgs=1156 cs=0 l=1 rx=0x7f59c400fa40 tx=0x7f59c400fde0).stop 2019-12-05T06:41:14.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.888+0000 7f59d4154700 1 -- 172.21.15.16:0/3056182758 shutdown_connections 2019-12-05T06:41:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.888+0000 7f59d4154700 1 --2- 172.21.15.16:0/3056182758 >> v2:172.21.15.76:3300/0 conn(0x7f59cc08fef0 0x7f59cc08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:14.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.888+0000 7f59d4154700 1 --2- 172.21.15.16:0/3056182758 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc08c6d0 0x7f59cc08cac0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.888+0000 7f59d4154700 1 -- 172.21.15.16:0/3056182758 shutdown_connections 2019-12-05T06:41:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.888+0000 7f59d4154700 1 -- 172.21.15.16:0/3056182758 wait complete. 2019-12-05T06:41:14.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:13.888+0000 7f59d4154700 1 -- 172.21.15.16:0/3056182758 >> 172.21.15.16:0/3056182758 conn(0x7f59cc061940 msgr2=0x7f59cc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:14.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.116+0000 7f59d4154700 1 Processor -- start 2019-12-05T06:41:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.116+0000 7f59d4154700 1 -- start start 2019-12-05T06:41:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.118+0000 7f59d4154700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f59cc096b90 0x7f59cc096f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:14.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.118+0000 7f59d4154700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59cc089790 0x7f59cc0974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:14.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.118+0000 7f59d4154700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59cc06d980 con 0x7f59cc0887c0 2019-12-05T06:41:14.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.118+0000 7f59d4154700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59cc087830 con 0x7f59cc089790 2019-12-05T06:41:14.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.118+0000 7f59d4154700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59cc098a80 con 0x7f59cc096b90 2019-12-05T06:41:14.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.119+0000 7f59d1ef0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59cc089790 0x7f59cc0974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:14.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.119+0000 7f59d1ef0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59cc089790 0x7f59cc0974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58698/0 (socket says 172.21.15.16:58698) 2019-12-05T06:41:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.119+0000 7f59d1ef0700 1 -- 172.21.15.16:0/598459232 learned_addr learned my addr 172.21.15.16:0/598459232 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:14.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.119+0000 7f59d26f1700 1 --2- 172.21.15.16:0/598459232 >> v2:172.21.15.76:3300/0 conn(0x7f59cc096b90 0x7f59cc096f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:14.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.119+0000 7f59caffd700 1 -- 172.21.15.16:0/598459232 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495553411 0 0) 0x7f59bc000e20 con 0x7f59cc0887c0 2019-12-05T06:41:14.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.119+0000 7f59caffd700 1 -- 172.21.15.16:0/598459232 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ac002e80 con 0x7f59cc0887c0 2019-12-05T06:41:14.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.119+0000 7f59d1ef0700 1 -- 172.21.15.16:0/598459232 >> v2:172.21.15.76:3300/0 conn(0x7f59cc096b90 msgr2=0x7f59cc096f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.119+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> v2:172.21.15.76:3300/0 conn(0x7f59cc096b90 0x7f59cc096f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:14.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59d1ef0700 1 -- 172.21.15.16:0/598459232 >> v1:172.21.15.76:6789/0 conn(0x7f59cc0887c0 legacy=0x7f59cc088c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59d1ef0700 1 -- 172.21.15.16:0/598459232 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59cc087ec0 con 0x7f59cc089790 2019-12-05T06:41:14.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> v2:172.21.15.16:3300/0 conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> v2:172.21.15.16:3300/0 conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59caffd700 1 -- 172.21.15.16:0/598459232 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c0000940 con 0x7f59cc089790 2019-12-05T06:41:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59caffd700 1 -- 172.21.15.16:0/598459232 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c0000e80 con 0x7f59cc089790 2019-12-05T06:41:14.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59d4154700 1 -- 172.21.15.16:0/598459232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59cc087ec0 con 0x7f59cc089790 2019-12-05T06:41:14.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.120+0000 7f59caffd700 1 -- 172.21.15.16:0/598459232 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59c00014d0 con 0x7f59cc089790 2019-12-05T06:41:14.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.121+0000 7f59d4154700 1 -- 172.21.15.16:0/598459232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc087ec0 con 0x7f59cc089790 2019-12-05T06:41:14.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.122+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.123+0000 7f59caffd700 1 -- 172.21.15.16:0/598459232 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f59c0001070 con 0x7f59cc089790 2019-12-05T06:41:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.123+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.123+0000 7f59caffd700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59ac023670 0x7f59ac025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.123+0000 7f59caffd700 1 -- 172.21.15.16:0/598459232 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f59c0045df0 con 0x7f59cc089790 2019-12-05T06:41:14.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.123+0000 7f59d26f1700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59ac023670 0x7f59ac025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:14.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.124+0000 7f59d26f1700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59ac023670 0x7f59ac025ae0 crc :-1 s=READY pgs=497 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:14.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.146+0000 7f59d4154700 1 -- 172.21.15.16:0/598459232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59cc087ec0 con 0x7f59cc089790 2019-12-05T06:41:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.156+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.157+0000 7f59caffd700 1 -- 172.21.15.16:0/598459232 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59c003cfe0 con 0x7f59cc089790 2019-12-05T06:41:14.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.414+0000 7f59d4154700 1 -- 172.21.15.16:0/598459232 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59cc087ec0 con 0x7f59cc089790 2019-12-05T06:41:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.414+0000 7f59d1ef0700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=THROTTLE_DONE pgs=1157 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.414+0000 7f59caffd700 1 -- 172.21.15.16:0/598459232 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f59c0009250 con 0x7f59cc089790 2019-12-05T06:41:14.416 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 143 2019-12-05T06:41:14.416 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:14.417 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:13.399920+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:14.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.417+0000 7f59d4154700 1 -- 172.21.15.16:0/598459232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59ac023670 msgr2=0x7f59ac025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.417+0000 7f59d4154700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59ac023670 0x7f59ac025ae0 crc :-1 s=READY pgs=497 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:14.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.417+0000 7f59d4154700 1 -- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 msgr2=0x7f59cc0974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.417+0000 7f59d4154700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 0x7f59cc0974c0 secure :-1 s=READY pgs=1157 cs=0 l=1 rx=0x7f59c0010450 tx=0x7f59c0010280).stop 2019-12-05T06:41:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.417+0000 7f59d4154700 1 -- 172.21.15.16:0/598459232 shutdown_connections 2019-12-05T06:41:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.417+0000 7f59d4154700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59ac023670 0x7f59ac025ae0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.417+0000 7f59d4154700 1 --2- 172.21.15.16:0/598459232 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59cc089790 0x7f59cc0974c0 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:14.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.417+0000 7f59d4154700 1 --2- 172.21.15.16:0/598459232 >> v2:172.21.15.76:3300/0 conn(0x7f59cc096b90 0x7f59cc096f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.417+0000 7f59d4154700 1 -- 172.21.15.16:0/598459232 shutdown_connections 2019-12-05T06:41:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.417+0000 7f59d4154700 1 -- 172.21.15.16:0/598459232 wait complete. 2019-12-05T06:41:14.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.418+0000 7f59d4154700 1 -- 172.21.15.16:0/598459232 >> 172.21.15.16:0/598459232 conn(0x7f59cc08a470 msgr2=0x7f59cc09df90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:14.430 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:41:14.430 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:41:14.430 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.621+0000 7ff998aa2700 1 Processor -- start 2019-12-05T06:41:14.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.621+0000 7ff998aa2700 1 -- start start 2019-12-05T06:41:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.623+0000 7ff998aa2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 0x7ff99408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.623+0000 7ff998aa2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9940958e0 con 0x7ff99408d2c0 2019-12-05T06:41:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.623+0000 7ff998aa2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9940629e0 con 0x7ff99408c6d0 2019-12-05T06:41:14.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.623+0000 7ff998aa2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff99408df90 con 0x7ff99408fef0 2019-12-05T06:41:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.623+0000 7ff99259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff99408d2c0 0x7ff99408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43874/0 (socket says 172.21.15.16:43874) 2019-12-05T06:41:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.623+0000 7ff99259c700 1 -- 172.21.15.16:0/4250901609 learned_addr learned my addr 172.21.15.16:0/4250901609 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.623+0000 7ff99359e700 1 --2- 172.21.15.16:0/4250901609 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 0x7ff99408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:14.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.624+0000 7ff991d9b700 1 -- 172.21.15.16:0/4250901609 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759129500 0 0) 0x7ff97c001150 con 0x7ff99408d2c0 2019-12-05T06:41:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.624+0000 7ff991d9b700 1 -- 172.21.15.16:0/4250901609 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff980002e80 con 0x7ff99408d2c0 2019-12-05T06:41:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.624+0000 7ff99359e700 1 -- 172.21.15.16:0/4250901609 >> v1:172.21.15.76:6789/0 conn(0x7ff99408c6d0 legacy=0x7ff99408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:41:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.624+0000 7ff99359e700 1 -- 172.21.15.16:0/4250901609 >> v1:172.21.15.16:6789/0 conn(0x7ff99408d2c0 legacy=0x7ff99408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.624+0000 7ff99359e700 1 -- 172.21.15.16:0/4250901609 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99408e620 con 0x7ff99408fef0 2019-12-05T06:41:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.624+0000 7ff99359e700 1 --2- 172.21.15.16:0/4250901609 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 0x7ff99408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff98400b300 tx=0x7ff98400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.624+0000 7ff99359e700 1 --2- 172.21.15.16:0/4250901609 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 0x7ff99408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff98400b300 tx=0x7ff98400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.624+0000 7ff99359e700 1 --2- 172.21.15.16:0/4250901609 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 0x7ff99408f720 secure :-1 s=READY pgs=273 cs=0 l=1 rx=0x7ff98400b300 tx=0x7ff98400bb10).ready entity=mon.2 client_cookie=b8233cc3c24aa6ba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.624+0000 7ff99359e700 1 --2- 172.21.15.16:0/4250901609 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 0x7ff99408f720 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7ff98400b300 tx=0x7ff98400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.625+0000 7ff991d9b700 1 -- 172.21.15.16:0/4250901609 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff9840075c0 con 0x7ff99408fef0 2019-12-05T06:41:14.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.625+0000 7ff99359e700 1 --2- 172.21.15.16:0/4250901609 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 0x7ff99408f720 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7ff98400b300 tx=0x7ff98400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.625+0000 7ff99359e700 1 --2- 172.21.15.16:0/4250901609 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 0x7ff99408f720 secure :-1 s=THROTTLE_DONE pgs=273 cs=0 l=1 rx=0x7ff98400b300 tx=0x7ff98400bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.625+0000 7ff991d9b700 1 -- 172.21.15.16:0/4250901609 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff984000f30 con 0x7ff99408fef0 2019-12-05T06:41:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.625+0000 7ff991d9b700 1 -- 172.21.15.16:0/4250901609 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff98400a1e0 con 0x7ff99408fef0 2019-12-05T06:41:14.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.625+0000 7ff998aa2700 1 -- 172.21.15.16:0/4250901609 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 msgr2=0x7ff99408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.625+0000 7ff998aa2700 1 --2- 172.21.15.16:0/4250901609 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 0x7ff99408f720 secure :-1 s=READY pgs=273 cs=0 l=1 rx=0x7ff98400b300 tx=0x7ff98400bb10).stop 2019-12-05T06:41:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.626+0000 7ff998aa2700 1 -- 172.21.15.16:0/4250901609 shutdown_connections 2019-12-05T06:41:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.626+0000 7ff998aa2700 1 --2- 172.21.15.16:0/4250901609 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff99408fef0 0x7ff99408f720 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.627+0000 7ff998aa2700 1 -- 172.21.15.16:0/4250901609 shutdown_connections 2019-12-05T06:41:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.627+0000 7ff998aa2700 1 -- 172.21.15.16:0/4250901609 wait complete. 2019-12-05T06:41:14.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.627+0000 7ff998aa2700 1 -- 172.21.15.16:0/4250901609 >> 172.21.15.16:0/4250901609 conn(0x7ff994061940 msgr2=0x7ff994096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:14.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.631+0000 7ff998aa2700 1 Processor -- start 2019-12-05T06:41:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.632+0000 7ff998aa2700 1 -- start start 2019-12-05T06:41:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.633+0000 7ff998aa2700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff99409e2c0 0x7ff99409dba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:14.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.633+0000 7ff998aa2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99414de80 con 0x7ff99414d5d0 2019-12-05T06:41:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.633+0000 7ff998aa2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff99406e250 con 0x7ff99414d9c0 2019-12-05T06:41:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.633+0000 7ff998aa2700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff99412b3f0 con 0x7ff99409e2c0 2019-12-05T06:41:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.633+0000 7ff992d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff99414d5d0 0x7ff99409d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43878/0 (socket says 172.21.15.16:43878) 2019-12-05T06:41:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.633+0000 7ff992d9d700 1 -- 172.21.15.16:0/1810302056 learned_addr learned my addr 172.21.15.16:0/1810302056 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:14.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.633+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> v2:172.21.15.76:3300/0 conn(0x7ff99409e2c0 0x7ff99409dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.633+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 536745396 0 0) 0x7ff98c001150 con 0x7ff99414d5d0 2019-12-05T06:41:14.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.634+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff978002e80 con 0x7ff99414d5d0 2019-12-05T06:41:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.634+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177395584 0 0) 0x7ff984009d10 con 0x7ff99414d9c0 2019-12-05T06:41:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.634+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff9780030b0 con 0x7ff99414d9c0 2019-12-05T06:41:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.634+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1094792472 0 0) 0x7ff98c0012c0 con 0x7ff99414d5d0 2019-12-05T06:41:14.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.634+0000 7ff99259c700 1 -- 172.21.15.16:0/1810302056 >> v1:172.21.15.76:6789/0 conn(0x7ff99414d9c0 legacy=0x7ff994153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.634+0000 7ff99259c700 1 -- 172.21.15.16:0/1810302056 >> v1:172.21.15.16:6789/0 conn(0x7ff99414d5d0 legacy=0x7ff99409d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.634+0000 7ff99259c700 1 -- 172.21.15.16:0/1810302056 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff99412ba80 con 0x7ff99409e2c0 2019-12-05T06:41:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.634+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> v2:172.21.15.76:3300/0 conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.635+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> v2:172.21.15.76:3300/0 conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.635+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=READY pgs=813 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.635+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.635+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff97c00f130 con 0x7ff99409e2c0 2019-12-05T06:41:14.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.635+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.635+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.635+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff97c00ed50 con 0x7ff99409e2c0 2019-12-05T06:41:14.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.635+0000 7ff998aa2700 1 -- 172.21.15.16:0/1810302056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff99412ba80 con 0x7ff99409e2c0 2019-12-05T06:41:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.635+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff97c00f7b0 con 0x7ff99409e2c0 2019-12-05T06:41:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.636+0000 7ff998aa2700 1 -- 172.21.15.16:0/1810302056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff99412ba80 con 0x7ff99409e2c0 2019-12-05T06:41:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.636+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.638+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff97c00ed50 con 0x7ff99409e2c0 2019-12-05T06:41:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.638+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.638+0000 7ff98b7fe700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff978023650 0x7ff978025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.638+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7ff97c045dc0 con 0x7ff99409e2c0 2019-12-05T06:41:14.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.638+0000 7ff992d9d700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff978023650 0x7ff978025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:14.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.639+0000 7ff992d9d700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff978023650 0x7ff978025ae0 crc :-1 s=READY pgs=498 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:14.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.652+0000 7ff998aa2700 1 -- 172.21.15.16:0/1810302056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff99412ba80 con 0x7ff99409e2c0 2019-12-05T06:41:14.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.659+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.659+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff97c00eb40 con 0x7ff99409e2c0 2019-12-05T06:41:14.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.902+0000 7ff998aa2700 1 -- 172.21.15.16:0/1810302056 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff99412ba80 con 0x7ff99409e2c0 2019-12-05T06:41:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.903+0000 7ff99259c700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=THROTTLE_DONE pgs=813 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.903+0000 7ff98b7fe700 1 -- 172.21.15.16:0/1810302056 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 143 v143) v1 ==== 95+0+1774 (secure 0 0 0) 0x7ff97c00eb40 con 0x7ff99409e2c0 2019-12-05T06:41:14.905 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 143 2019-12-05T06:41:14.905 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:14.906 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":143,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":143,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:13.399920+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:14.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.907+0000 7ff998aa2700 1 -- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff978023650 msgr2=0x7ff978025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.907+0000 7ff998aa2700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff978023650 0x7ff978025ae0 crc :-1 s=READY pgs=498 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:14.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.907+0000 7ff998aa2700 1 -- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 msgr2=0x7ff99409dba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:14.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.907+0000 7ff998aa2700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 0x7ff99409dba0 secure :-1 s=READY pgs=813 cs=0 l=1 rx=0x7ff97c010380 tx=0x7ff97c010720).stop 2019-12-05T06:41:14.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.908+0000 7ff998aa2700 1 -- 172.21.15.16:0/1810302056 shutdown_connections 2019-12-05T06:41:14.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.908+0000 7ff998aa2700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff978023650 0x7ff978025ae0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:14.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.908+0000 7ff998aa2700 1 --2- 172.21.15.16:0/1810302056 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff99409e2c0 0x7ff99409dba0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:14.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.908+0000 7ff998aa2700 1 -- 172.21.15.16:0/1810302056 shutdown_connections 2019-12-05T06:41:14.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.908+0000 7ff998aa2700 1 -- 172.21.15.16:0/1810302056 wait complete. 2019-12-05T06:41:14.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:14.908+0000 7ff998aa2700 1 -- 172.21.15.16:0/1810302056 >> 172.21.15.16:0/1810302056 conn(0x7ff99406d260 msgr2=0x7ff99409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:14.926 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 143, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 143, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7132": { "addr": "172.21.15.76:6835/2989717660", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 2989717660, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 2989717660, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7132, "incarnation": 143, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 72, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:41:13.399920+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7132 } } } ], "standbys": [] } 2019-12-05T06:41:16.929 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:17.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.195+0000 7f77a4e70700 1 Processor -- start 2019-12-05T06:41:17.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.195+0000 7f77a4e70700 1 -- start start 2019-12-05T06:41:17.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.197+0000 7f77a4e70700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77a008d2c0 0x7f77a008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:17.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.197+0000 7f77a4e70700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f77a008fef0 0x7f77a008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:17.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.197+0000 7f77a4e70700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a00958e0 con 0x7f77a008c6d0 2019-12-05T06:41:17.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.197+0000 7f77a4e70700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a008df90 con 0x7f77a008d2c0 2019-12-05T06:41:17.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.197+0000 7f77a4e70700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a00a3f70 con 0x7f77a008fef0 2019-12-05T06:41:17.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.197+0000 7f779e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77a008d2c0 0x7f77a008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:17.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.198+0000 7f779e59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77a008d2c0 0x7f77a008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58722/0 (socket says 172.21.15.16:58722) 2019-12-05T06:41:17.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.198+0000 7f779e59c700 1 -- 172.21.15.16:0/3745878763 learned_addr learned my addr 172.21.15.16:0/3745878763 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:17.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.198+0000 7f779f59e700 1 --2- 172.21.15.16:0/3745878763 >> v2:172.21.15.76:3300/0 conn(0x7f77a008fef0 0x7f77a008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.198+0000 7f779e59c700 1 -- 172.21.15.16:0/3745878763 >> v2:172.21.15.76:3300/0 conn(0x7f77a008fef0 msgr2=0x7f77a008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:41:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.198+0000 7f779e59c700 1 --2- 172.21.15.16:0/3745878763 >> v2:172.21.15.76:3300/0 conn(0x7f77a008fef0 0x7f77a008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.198+0000 7f779e59c700 1 -- 172.21.15.16:0/3745878763 >> v1:172.21.15.16:6789/0 conn(0x7f77a008c6d0 legacy=0x7f77a008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.198+0000 7f779e59c700 1 -- 172.21.15.16:0/3745878763 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a008e620 con 0x7f77a008d2c0 2019-12-05T06:41:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.198+0000 7f779dd9b700 1 -- 172.21.15.16:0/3745878763 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2706194835 0 0) 0x7f7798001100 con 0x7f77a008c6d0 2019-12-05T06:41:17.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.198+0000 7f779e59c700 1 --2- 172.21.15.16:0/3745878763 >> v2:172.21.15.16:3300/0 conn(0x7f77a008d2c0 0x7f77a008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f778800fa40 tx=0x7f778800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.199+0000 7f779e59c700 1 --2- 172.21.15.16:0/3745878763 >> v2:172.21.15.16:3300/0 conn(0x7f77a008d2c0 0x7f77a008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f778800fa40 tx=0x7f778800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.199+0000 7f779e59c700 1 --2- 172.21.15.16:0/3745878763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a008d2c0 0x7f77a008ef20 secure :-1 s=READY pgs=1161 cs=0 l=1 rx=0x7f778800fa40 tx=0x7f778800fde0).ready entity=mon.0 client_cookie=b0a8358ea40a2916 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.199+0000 7f779e59c700 1 --2- 172.21.15.16:0/3745878763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a008d2c0 0x7f77a008ef20 secure :-1 s=THROTTLE_DONE pgs=1161 cs=0 l=1 rx=0x7f778800fa40 tx=0x7f778800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.199+0000 7f779dd9b700 1 -- 172.21.15.16:0/3745878763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7788000ce0 con 0x7f77a008d2c0 2019-12-05T06:41:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.199+0000 7f779e59c700 1 --2- 172.21.15.16:0/3745878763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a008d2c0 0x7f77a008ef20 secure :-1 s=THROTTLE_DONE pgs=1161 cs=0 l=1 rx=0x7f778800fa40 tx=0x7f778800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.199+0000 7f779e59c700 1 --2- 172.21.15.16:0/3745878763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a008d2c0 0x7f77a008ef20 secure :-1 s=THROTTLE_DONE pgs=1161 cs=0 l=1 rx=0x7f778800fa40 tx=0x7f778800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.199+0000 7f779dd9b700 1 -- 172.21.15.16:0/3745878763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7788001010 con 0x7f77a008d2c0 2019-12-05T06:41:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.199+0000 7f779dd9b700 1 -- 172.21.15.16:0/3745878763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7788000ce0 con 0x7f77a008d2c0 2019-12-05T06:41:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.199+0000 7f77a4e70700 1 -- 172.21.15.16:0/3745878763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a008d2c0 msgr2=0x7f77a008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.199+0000 7f77a4e70700 1 --2- 172.21.15.16:0/3745878763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a008d2c0 0x7f77a008ef20 secure :-1 s=READY pgs=1161 cs=0 l=1 rx=0x7f778800fa40 tx=0x7f778800fde0).stop 2019-12-05T06:41:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.201+0000 7f77a4e70700 1 -- 172.21.15.16:0/3745878763 shutdown_connections 2019-12-05T06:41:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.201+0000 7f77a4e70700 1 --2- 172.21.15.16:0/3745878763 >> v2:172.21.15.76:3300/0 conn(0x7f77a008fef0 0x7f77a008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:17.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.201+0000 7f77a4e70700 1 --2- 172.21.15.16:0/3745878763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a008d2c0 0x7f77a008ef20 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:17.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.201+0000 7f77a4e70700 1 -- 172.21.15.16:0/3745878763 shutdown_connections 2019-12-05T06:41:17.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.202+0000 7f77a4e70700 1 -- 172.21.15.16:0/3745878763 wait complete. 2019-12-05T06:41:17.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.202+0000 7f77a4e70700 1 -- 172.21.15.16:0/3745878763 >> 172.21.15.16:0/3745878763 conn(0x7f77a0061940 msgr2=0x7f77a0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:17.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.207+0000 7f77a4e70700 1 Processor -- start 2019-12-05T06:41:17.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.207+0000 7f77a4e70700 1 -- start start 2019-12-05T06:41:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.208+0000 7f77a4e70700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77a01333a0 0x7f77a0128360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.208+0000 7f77a4e70700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f77a0128fe0 0x7f77a01288a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.208+0000 7f77a4e70700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a0070330 con 0x7f77a006ed20 2019-12-05T06:41:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.208+0000 7f77a4e70700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a0088730 con 0x7f77a01333a0 2019-12-05T06:41:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.208+0000 7f77a4e70700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a0089810 con 0x7f77a0128fe0 2019-12-05T06:41:17.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.209+0000 7f779ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77a01333a0 0x7f77a0128360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.209+0000 7f779ed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f77a01333a0 0x7f77a0128360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58726/0 (socket says 172.21.15.16:58726) 2019-12-05T06:41:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.209+0000 7f779ed9d700 1 -- 172.21.15.16:0/3286642881 learned_addr learned my addr 172.21.15.16:0/3286642881 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.209+0000 7f779e59c700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f77a0128fe0 0x7f77a01288a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.209+0000 7f779ed9d700 1 -- 172.21.15.16:0/3286642881 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f77a0128fe0 msgr2=0x7f77a01288a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.209+0000 7f77977fe700 1 -- 172.21.15.16:0/3286642881 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205552716 0 0) 0x7f7790001080 con 0x7f77a006ed20 2019-12-05T06:41:17.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.209+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f77a0128fe0 0x7f77a01288a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.209+0000 7f779ed9d700 1 -- 172.21.15.16:0/3286642881 >> v1:172.21.15.76:6789/0 conn(0x7f77a006ed20 legacy=0x7f77a006f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.209+0000 7f779ed9d700 1 -- 172.21.15.16:0/3286642881 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a0089ea0 con 0x7f77a01333a0 2019-12-05T06:41:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> v2:172.21.15.16:3300/0 conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> v2:172.21.15.16:3300/0 conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=READY pgs=1162 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f77977fe700 1 -- 172.21.15.16:0/3286642881 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7798000ce0 con 0x7f77a01333a0 2019-12-05T06:41:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f77977fe700 1 -- 172.21.15.16:0/3286642881 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7798000ce0 con 0x7f77a01333a0 2019-12-05T06:41:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f77a4e70700 1 -- 172.21.15.16:0/3286642881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a0089ea0 con 0x7f77a01333a0 2019-12-05T06:41:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f77977fe700 1 -- 172.21.15.16:0/3286642881 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f779800f160 con 0x7f77a01333a0 2019-12-05T06:41:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.210+0000 7f77a4e70700 1 -- 172.21.15.16:0/3286642881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a0089ea0 con 0x7f77a01333a0 2019-12-05T06:41:17.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.211+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.212+0000 7f77977fe700 1 -- 172.21.15.16:0/3286642881 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7798009630 con 0x7f77a01333a0 2019-12-05T06:41:17.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.212+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.212+0000 7f77977fe700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7784023630 0x7f7784025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:17.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.212+0000 7f77977fe700 1 -- 172.21.15.16:0/3286642881 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f7798045e10 con 0x7f77a01333a0 2019-12-05T06:41:17.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.212+0000 7f779e59c700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7784023630 0x7f7784025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:17.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.213+0000 7f779e59c700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7784023630 0x7f7784025ac0 crc :-1 s=READY pgs=499 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:17.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.227+0000 7f77a4e70700 1 -- 172.21.15.16:0/3286642881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a0089ea0 con 0x7f77a01333a0 2019-12-05T06:41:17.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.235+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.235+0000 7f77977fe700 1 -- 172.21.15.16:0/3286642881 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f779800f670 con 0x7f77a01333a0 2019-12-05T06:41:17.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.475+0000 7f77a4e70700 1 -- 172.21.15.16:0/3286642881 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77a0089ea0 con 0x7f77a01333a0 2019-12-05T06:41:17.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.478+0000 7f779ed9d700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=THROTTLE_DONE pgs=1162 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:17.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.478+0000 7f77977fe700 1 -- 172.21.15.16:0/3286642881 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 144 v144) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f7798009630 con 0x7f77a01333a0 2019-12-05T06:41:17.483 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 144 2019-12-05T06:41:17.483 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:17.483 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":144,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":144,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:16.471663+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:reconnect","state_seq":29,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:17.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.486+0000 7f77a4e70700 1 -- 172.21.15.16:0/3286642881 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7784023630 msgr2=0x7f7784025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:17.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.486+0000 7f77a4e70700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7784023630 0x7f7784025ac0 crc :-1 s=READY pgs=499 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:17.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.486+0000 7f77a4e70700 1 -- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 msgr2=0x7f77a0128360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:17.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.486+0000 7f77a4e70700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 0x7f77a0128360 secure :-1 s=READY pgs=1162 cs=0 l=1 rx=0x7f7798010450 tx=0x7f779800b390).stop 2019-12-05T06:41:17.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.487+0000 7f77a4e70700 1 -- 172.21.15.16:0/3286642881 shutdown_connections 2019-12-05T06:41:17.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.487+0000 7f77a4e70700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f77a0128fe0 0x7f77a01288a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:17.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.487+0000 7f77a4e70700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7784023630 0x7f7784025ac0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:17.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.487+0000 7f77a4e70700 1 --2- 172.21.15.16:0/3286642881 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f77a01333a0 0x7f77a0128360 unknown :-1 s=CLOSED pgs=1162 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:17.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.487+0000 7f77a4e70700 1 -- 172.21.15.16:0/3286642881 shutdown_connections 2019-12-05T06:41:17.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.487+0000 7f77a4e70700 1 -- 172.21.15.16:0/3286642881 wait complete. 2019-12-05T06:41:17.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:17.487+0000 7f77a4e70700 1 -- 172.21.15.16:0/3286642881 >> 172.21.15.16:0/3286642881 conn(0x7f77a009de30 msgr2=0x7f77a0070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:17.503 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:41:19.510 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.767+0000 7f7a2699a700 1 Processor -- start 2019-12-05T06:41:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.767+0000 7f7a2699a700 1 -- start start 2019-12-05T06:41:19.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.768+0000 7f7a2699a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a2008c6d0 0x7f7a2008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.769+0000 7f7a2699a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7a2008f720 0x7f7a2008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.769+0000 7f7a2699a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a200958e0 con 0x7f7a2008fef0 2019-12-05T06:41:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.769+0000 7f7a2699a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a200a3f70 con 0x7f7a2008f720 2019-12-05T06:41:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.769+0000 7f7a2699a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a2008d830 con 0x7f7a2008c6d0 2019-12-05T06:41:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.769+0000 7f7a1ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a2008fef0 0x7f7a2008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43908/0 (socket says 172.21.15.16:43908) 2019-12-05T06:41:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.769+0000 7f7a1ffff700 1 -- 172.21.15.16:0/3994526551 learned_addr learned my addr 172.21.15.16:0/3994526551 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:19.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.769+0000 7f7a24f37700 1 --2- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a2008c6d0 0x7f7a2008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.769+0000 7f7a25738700 1 --2- 172.21.15.16:0/3994526551 >> v2:172.21.15.76:3300/0 conn(0x7f7a2008f720 0x7f7a2008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.769+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/3994526551 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3172648615 0 0) 0x7f7a10001150 con 0x7f7a2008fef0 2019-12-05T06:41:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.769+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/3994526551 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a0c002e80 con 0x7f7a2008fef0 2019-12-05T06:41:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.770+0000 7f7a25738700 1 -- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a2008c6d0 msgr2=0x7f7a2008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:41:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.770+0000 7f7a25738700 1 --2- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a2008c6d0 0x7f7a2008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:19.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.770+0000 7f7a25738700 1 -- 172.21.15.16:0/3994526551 >> v1:172.21.15.16:6789/0 conn(0x7f7a2008fef0 legacy=0x7f7a2008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.770+0000 7f7a25738700 1 -- 172.21.15.16:0/3994526551 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a2008fbc0 con 0x7f7a2008f720 2019-12-05T06:41:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.770+0000 7f7a24f37700 1 --2- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a2008c6d0 0x7f7a2008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:41:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.770+0000 7f7a25738700 1 --2- 172.21.15.16:0/3994526551 >> v2:172.21.15.76:3300/0 conn(0x7f7a2008f720 0x7f7a2008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7a0800af80 tx=0x7f7a0800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.770+0000 7f7a25738700 1 --2- 172.21.15.16:0/3994526551 >> v2:172.21.15.76:3300/0 conn(0x7f7a2008f720 0x7f7a2008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7a0800af80 tx=0x7f7a0800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.770+0000 7f7a25738700 1 --2- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2008f720 0x7f7a2008d000 secure :-1 s=READY pgs=816 cs=0 l=1 rx=0x7f7a0800af80 tx=0x7f7a0800b790).ready entity=mon.1 client_cookie=62ddaa19821fe587 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.771+0000 7f7a25738700 1 --2- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2008f720 0x7f7a2008d000 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f7a0800af80 tx=0x7f7a0800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.771+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/3994526551 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a08000bc0 con 0x7f7a2008f720 2019-12-05T06:41:19.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.771+0000 7f7a25738700 1 --2- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2008f720 0x7f7a2008d000 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f7a0800af80 tx=0x7f7a0800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.771+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/3994526551 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a08007240 con 0x7f7a2008f720 2019-12-05T06:41:19.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.771+0000 7f7a25738700 1 --2- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2008f720 0x7f7a2008d000 secure :-1 s=THROTTLE_DONE pgs=816 cs=0 l=1 rx=0x7f7a0800af80 tx=0x7f7a0800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.771+0000 7f7a1f7fe700 1 -- 172.21.15.16:0/3994526551 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a08000bf0 con 0x7f7a2008f720 2019-12-05T06:41:19.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.771+0000 7f7a2699a700 1 -- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2008f720 msgr2=0x7f7a2008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:19.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.771+0000 7f7a2699a700 1 --2- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2008f720 0x7f7a2008d000 secure :-1 s=READY pgs=816 cs=0 l=1 rx=0x7f7a0800af80 tx=0x7f7a0800b790).stop 2019-12-05T06:41:19.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.772+0000 7f7a2699a700 1 -- 172.21.15.16:0/3994526551 shutdown_connections 2019-12-05T06:41:19.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.772+0000 7f7a2699a700 1 --2- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2008f720 0x7f7a2008d000 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:19.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.772+0000 7f7a2699a700 1 --2- 172.21.15.16:0/3994526551 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7a2008c6d0 0x7f7a2008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:19.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.773+0000 7f7a2699a700 1 -- 172.21.15.16:0/3994526551 shutdown_connections 2019-12-05T06:41:19.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.773+0000 7f7a2699a700 1 -- 172.21.15.16:0/3994526551 wait complete. 2019-12-05T06:41:19.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.773+0000 7f7a2699a700 1 -- 172.21.15.16:0/3994526551 >> 172.21.15.16:0/3994526551 conn(0x7f7a20061940 msgr2=0x7f7a20096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:19.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.777+0000 7f7a2699a700 1 Processor -- start 2019-12-05T06:41:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.777+0000 7f7a2699a700 1 -- start start 2019-12-05T06:41:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.778+0000 7f7a2699a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7a2006e840 0x7f7a20134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.779+0000 7f7a2699a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a20128af0 con 0x7f7a20135450 2019-12-05T06:41:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.779+0000 7f7a2699a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a20128600 con 0x7f7a2006dc50 2019-12-05T06:41:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.779+0000 7f7a2699a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a20132830 con 0x7f7a2006e840 2019-12-05T06:41:19.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.779+0000 7f7a25738700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7a20135450 0x7f7a201349c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43918/0 (socket says 172.21.15.16:43918) 2019-12-05T06:41:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.779+0000 7f7a25738700 1 -- 172.21.15.16:0/2257730943 learned_addr learned my addr 172.21.15.16:0/2257730943 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.779+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> v2:172.21.15.76:3300/0 conn(0x7f7a2006e840 0x7f7a20134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.779+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/2257730943 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3416867484 0 0) 0x7f7a0800af70 con 0x7f7a20135450 2019-12-05T06:41:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.779+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/2257730943 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a04002e80 con 0x7f7a20135450 2019-12-05T06:41:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1ffff700 1 -- 172.21.15.16:0/2257730943 >> v1:172.21.15.76:6789/0 conn(0x7f7a2006dc50 legacy=0x7f7a2006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/2257730943 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3880596269 0 0) 0x7f7a1800df00 con 0x7f7a2006dc50 2019-12-05T06:41:19.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1ffff700 1 -- 172.21.15.16:0/2257730943 >> v1:172.21.15.16:6789/0 conn(0x7f7a20135450 legacy=0x7f7a201349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1ffff700 1 -- 172.21.15.16:0/2257730943 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a20133b80 con 0x7f7a2006e840 2019-12-05T06:41:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> v2:172.21.15.76:3300/0 conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> v2:172.21.15.76:3300/0 conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=READY pgs=818 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=THROTTLE_DONE pgs=818 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/2257730943 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a1000f130 con 0x7f7a2006e840 2019-12-05T06:41:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=THROTTLE_DONE pgs=818 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=THROTTLE_DONE pgs=818 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/2257730943 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a10000940 con 0x7f7a2006e840 2019-12-05T06:41:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a2699a700 1 -- 172.21.15.16:0/2257730943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a20133b80 con 0x7f7a2006e840 2019-12-05T06:41:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/2257730943 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a1000f710 con 0x7f7a2006e840 2019-12-05T06:41:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.780+0000 7f7a2699a700 1 -- 172.21.15.16:0/2257730943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a20133b80 con 0x7f7a2006e840 2019-12-05T06:41:19.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.781+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=THROTTLE_DONE pgs=818 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.782+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/2257730943 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7a1000ee90 con 0x7f7a2006e840 2019-12-05T06:41:19.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.782+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=THROTTLE_DONE pgs=818 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.782+0000 7f7a1d7fa700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a04023680 0x7f7a04025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:19.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.782+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/2257730943 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f7a10045fe0 con 0x7f7a2006e840 2019-12-05T06:41:19.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.782+0000 7f7a24f37700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a04023680 0x7f7a04025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:19.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.783+0000 7f7a24f37700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a04023680 0x7f7a04025af0 crc :-1 s=READY pgs=500 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:19.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.798+0000 7f7a2699a700 1 -- 172.21.15.16:0/2257730943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a20133b80 con 0x7f7a2006e840 2019-12-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.807+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=THROTTLE_DONE pgs=818 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:19.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:19.807+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/2257730943 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7a1000eb10 con 0x7f7a2006e840 2019-12-05T06:41:20.033 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.032+0000 7f7a2699a700 1 -- 172.21.15.16:0/2257730943 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7a20133b80 con 0x7f7a2006e840 2019-12-05T06:41:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.033+0000 7f7a1ffff700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=THROTTLE_DONE pgs=818 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:20.034 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.033+0000 7f7a1d7fa700 1 -- 172.21.15.16:0/2257730943 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f7a10000940 con 0x7f7a2006e840 2019-12-05T06:41:20.036 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 146 2019-12-05T06:41:20.037 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:20.037 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.039+0000 7f7a2699a700 1 -- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a04023680 msgr2=0x7f7a04025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:20.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.039+0000 7f7a2699a700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a04023680 0x7f7a04025af0 crc :-1 s=READY pgs=500 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.039+0000 7f7a2699a700 1 -- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 msgr2=0x7f7a20134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:20.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.039+0000 7f7a2699a700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 0x7f7a20134480 secure :-1 s=READY pgs=818 cs=0 l=1 rx=0x7f7a10010380 tx=0x7f7a10010720).stop 2019-12-05T06:41:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.040+0000 7f7a2699a700 1 -- 172.21.15.16:0/2257730943 shutdown_connections 2019-12-05T06:41:20.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.040+0000 7f7a2699a700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7a04023680 0x7f7a04025af0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:20.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.040+0000 7f7a2699a700 1 --2- 172.21.15.16:0/2257730943 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7a2006e840 0x7f7a20134480 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:20.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.040+0000 7f7a2699a700 1 -- 172.21.15.16:0/2257730943 shutdown_connections 2019-12-05T06:41:20.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.040+0000 7f7a2699a700 1 -- 172.21.15.16:0/2257730943 wait complete. 2019-12-05T06:41:20.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:20.040+0000 7f7a2699a700 1 -- 172.21.15.16:0/2257730943 >> 172.21.15.16:0/2257730943 conn(0x7f7a200a2850 msgr2=0x7f7a2006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:20.062 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=7011 2019-12-05T06:41:20.062 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.a 2019-12-05T06:41:29.063 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:41:29.064 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:41:29.064 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:41:29.069 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:41:29.069 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:29.240 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:41:29.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.287+0000 7f2f22471700 1 Processor -- start 2019-12-05T06:41:29.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.287+0000 7f2f22471700 1 -- start start 2019-12-05T06:41:29.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.288+0000 7f2f22471700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2f1c08c6d0 0x7f2f1c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:29.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.288+0000 7f2f22471700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f1c08fef0 0x7f2f1c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:29.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.288+0000 7f2f22471700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2f1c08ef20 0x7f2f1c08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:29.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.288+0000 7f2f22471700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f1c0a3f70 con 0x7f2f1c08fef0 2019-12-05T06:41:29.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.288+0000 7f2f22471700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f1c08d830 con 0x7f2f1c08ef20 2019-12-05T06:41:29.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.288+0000 7f2f22471700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2f1c08d540 con 0x7f2f1c08c6d0 2019-12-05T06:41:29.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.288+0000 7f2f1bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f1c08fef0 0x7f2f1c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:29.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.288+0000 7f2f1bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2f1c08fef0 0x7f2f1c08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58808/0 (socket says 172.21.15.16:58808) 2019-12-05T06:41:29.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f1bfff700 1 -- 172.21.15.16:0/2045095244 learned_addr learned my addr 172.21.15.16:0/2045095244 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:29.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2045095244 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2f1c08c6d0 0x7f2f1c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:29.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f2120f700 1 --2- 172.21.15.16:0/2045095244 >> v2:172.21.15.76:3300/0 conn(0x7f2f1c08ef20 0x7f2f1c08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:29.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f1bfff700 1 -- 172.21.15.16:0/2045095244 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2f1c08c6d0 msgr2=0x7f2f1c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:41:29.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f1bfff700 1 --2- 172.21.15.16:0/2045095244 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2f1c08c6d0 0x7f2f1c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:29.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f1bfff700 1 -- 172.21.15.16:0/2045095244 >> v2:172.21.15.76:3300/0 conn(0x7f2f1c08ef20 msgr2=0x7f2f1c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:41:29.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f1bfff700 1 --2- 172.21.15.16:0/2045095244 >> v2:172.21.15.76:3300/0 conn(0x7f2f1c08ef20 0x7f2f1c08f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:29.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f1bfff700 1 -- 172.21.15.16:0/2045095244 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c08e620 con 0x7f2f1c08fef0 2019-12-05T06:41:29.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f1bfff700 1 --2- 172.21.15.16:0/2045095244 >> v2:172.21.15.16:3300/0 conn(0x7f2f1c08fef0 0x7f2f1c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2f0c00fa40 tx=0x7f2f0c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f1bfff700 1 --2- 172.21.15.16:0/2045095244 >> v2:172.21.15.16:3300/0 conn(0x7f2f1c08fef0 0x7f2f1c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2f0c00fa40 tx=0x7f2f0c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.289+0000 7f2f1bfff700 1 --2- 172.21.15.16:0/2045095244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c08fef0 0x7f2f1c08d000 secure :-1 s=READY pgs=1171 cs=0 l=1 rx=0x7f2f0c00fa40 tx=0x7f2f0c00f8f0).ready entity=mon.0 client_cookie=ce15619583211998 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:29.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.290+0000 7f2f1bfff700 1 --2- 172.21.15.16:0/2045095244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c08fef0 0x7f2f1c08d000 secure :-1 s=THROTTLE_DONE pgs=1171 cs=0 l=1 rx=0x7f2f0c00fa40 tx=0x7f2f0c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.290+0000 7f2f1bfff700 1 --2- 172.21.15.16:0/2045095244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c08fef0 0x7f2f1c08d000 secure :-1 s=THROTTLE_DONE pgs=1171 cs=0 l=1 rx=0x7f2f0c00fa40 tx=0x7f2f0c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.290+0000 7f2f1b7fe700 1 -- 172.21.15.16:0/2045095244 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f0c000ce0 con 0x7f2f1c08fef0 2019-12-05T06:41:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.290+0000 7f2f1bfff700 1 --2- 172.21.15.16:0/2045095244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c08fef0 0x7f2f1c08d000 secure :-1 s=THROTTLE_DONE pgs=1171 cs=0 l=1 rx=0x7f2f0c00fa40 tx=0x7f2f0c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.290+0000 7f2f1b7fe700 1 -- 172.21.15.16:0/2045095244 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f0c001010 con 0x7f2f1c08fef0 2019-12-05T06:41:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.290+0000 7f2f1b7fe700 1 -- 172.21.15.16:0/2045095244 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f0c001720 con 0x7f2f1c08fef0 2019-12-05T06:41:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.290+0000 7f2f22471700 1 -- 172.21.15.16:0/2045095244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c08fef0 msgr2=0x7f2f1c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.290+0000 7f2f22471700 1 --2- 172.21.15.16:0/2045095244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c08fef0 0x7f2f1c08d000 secure :-1 s=READY pgs=1171 cs=0 l=1 rx=0x7f2f0c00fa40 tx=0x7f2f0c00f8f0).stop 2019-12-05T06:41:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.292+0000 7f2f22471700 1 -- 172.21.15.16:0/2045095244 shutdown_connections 2019-12-05T06:41:29.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.292+0000 7f2f22471700 1 --2- 172.21.15.16:0/2045095244 >> v2:172.21.15.76:3300/0 conn(0x7f2f1c08ef20 0x7f2f1c08f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:29.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.292+0000 7f2f22471700 1 --2- 172.21.15.16:0/2045095244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2f1c08fef0 0x7f2f1c08d000 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:29.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.292+0000 7f2f22471700 1 --2- 172.21.15.16:0/2045095244 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2f1c08c6d0 0x7f2f1c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:41:29.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.292+0000 7f2f22471700 1 -- 172.21.15.16:0/2045095244 shutdown_connections 2019-12-05T06:41:29.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.292+0000 7f2f22471700 1 -- 172.21.15.16:0/2045095244 wait complete. 2019-12-05T06:41:29.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.292+0000 7f2f22471700 1 -- 172.21.15.16:0/2045095244 >> 172.21.15.16:0/2045095244 conn(0x7f2f1c061940 msgr2=0x7f2f1c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:29.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.295+0000 7f2f22471700 1 Processor -- start 2019-12-05T06:41:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.295+0000 7f2f22471700 1 -- start start 2019-12-05T06:41:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.296+0000 7f2f22471700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2f1c06df10 0x7f2f1c098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.296+0000 7f2f22471700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1c089fb0 con 0x7f2f1c0992d0 2019-12-05T06:41:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.296+0000 7f2f22471700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2f1c06e300 con 0x7f2f1c156cb0 2019-12-05T06:41:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.296+0000 7f2f22471700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2f1c097370 con 0x7f2f1c06df10 2019-12-05T06:41:29.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.296+0000 7f2f1bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2f1c0992d0 0x7f2f1c098840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:43982/0 (socket says 172.21.15.16:43982) 2019-12-05T06:41:29.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.296+0000 7f2f1bfff700 1 -- 172.21.15.16:0/2590073314 learned_addr learned my addr 172.21.15.16:0/2590073314 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:29.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.297+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> v2:172.21.15.76:3300/0 conn(0x7f2f1c06df10 0x7f2f1c098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:29.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.297+0000 7f2f197fa700 1 -- 172.21.15.16:0/2590073314 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3735949693 0 0) 0x7f2f0c007900 con 0x7f2f1c0992d0 2019-12-05T06:41:29.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.297+0000 7f2f197fa700 1 -- 172.21.15.16:0/2590073314 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f00002e80 con 0x7f2f1c0992d0 2019-12-05T06:41:29.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.297+0000 7f2f20a0e700 1 -- 172.21.15.16:0/2590073314 >> v1:172.21.15.76:6789/0 conn(0x7f2f1c156cb0 legacy=0x7f2f1c1590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:29.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.297+0000 7f2f197fa700 1 -- 172.21.15.16:0/2590073314 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4182253745 0 0) 0x7f2f1400a8a0 con 0x7f2f1c156cb0 2019-12-05T06:41:29.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.297+0000 7f2f20a0e700 1 -- 172.21.15.16:0/2590073314 >> v1:172.21.15.16:6789/0 conn(0x7f2f1c0992d0 legacy=0x7f2f1c098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.297+0000 7f2f20a0e700 1 -- 172.21.15.16:0/2590073314 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f1c097a00 con 0x7f2f1c06df10 2019-12-05T06:41:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.297+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> v2:172.21.15.76:3300/0 conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.298+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> v2:172.21.15.76:3300/0 conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.298+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=READY pgs=822 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.298+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=THROTTLE_DONE pgs=822 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.298+0000 7f2f197fa700 1 -- 172.21.15.16:0/2590073314 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f04000940 con 0x7f2f1c06df10 2019-12-05T06:41:29.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.298+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=THROTTLE_DONE pgs=822 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.298+0000 7f2f22471700 1 -- 172.21.15.16:0/2590073314 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f1c097a00 con 0x7f2f1c06df10 2019-12-05T06:41:29.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.298+0000 7f2f197fa700 1 -- 172.21.15.16:0/2590073314 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2f04000940 con 0x7f2f1c06df10 2019-12-05T06:41:29.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.298+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=THROTTLE_DONE pgs=822 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.298+0000 7f2f197fa700 1 -- 172.21.15.16:0/2590073314 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2f04005770 con 0x7f2f1c06df10 2019-12-05T06:41:29.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.298+0000 7f2f22471700 1 -- 172.21.15.16:0/2590073314 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f1c097a00 con 0x7f2f1c06df10 2019-12-05T06:41:29.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.299+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=THROTTLE_DONE pgs=822 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.300+0000 7f2f197fa700 1 -- 172.21.15.16:0/2590073314 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2f04005180 con 0x7f2f1c06df10 2019-12-05T06:41:29.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.300+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=THROTTLE_DONE pgs=822 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.300+0000 7f2f197fa700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f00023680 0x7f2f00025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:29.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.300+0000 7f2f197fa700 1 -- 172.21.15.16:0/2590073314 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f2f04045df0 con 0x7f2f1c06df10 2019-12-05T06:41:29.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.300+0000 7f2f1bfff700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f00023680 0x7f2f00025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:29.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.301+0000 7f2f1bfff700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f00023680 0x7f2f00025af0 crc :-1 s=READY pgs=502 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:29.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.302+0000 7f2f22471700 1 -- 172.21.15.16:0/2590073314 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f1c097a00 con 0x7f2f1c06df10 2019-12-05T06:41:29.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.310+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=THROTTLE_DONE pgs=822 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.311+0000 7f2f197fa700 1 -- 172.21.15.16:0/2590073314 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2f04005b80 con 0x7f2f1c06df10 2019-12-05T06:41:29.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.523+0000 7f2f22471700 1 -- 172.21.15.16:0/2590073314 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2f1c097a00 con 0x7f2f1c06df10 2019-12-05T06:41:29.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.524+0000 7f2f20a0e700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=THROTTLE_DONE pgs=822 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:29.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.524+0000 7f2f197fa700 1 -- 172.21.15.16:0/2590073314 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f2f04005180 con 0x7f2f1c06df10 2019-12-05T06:41:29.526 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 146 2019-12-05T06:41:29.526 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:29.527 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:29.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.526+0000 7f2f22471700 1 -- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f00023680 msgr2=0x7f2f00025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:29.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.527+0000 7f2f22471700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f00023680 0x7f2f00025af0 crc :-1 s=READY pgs=502 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:29.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.527+0000 7f2f22471700 1 -- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 msgr2=0x7f2f1c098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:29.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.527+0000 7f2f22471700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 0x7f2f1c098300 secure :-1 s=READY pgs=822 cs=0 l=1 rx=0x7f2f040103b0 tx=0x7f2f04010750).stop 2019-12-05T06:41:29.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.527+0000 7f2f22471700 1 -- 172.21.15.16:0/2590073314 shutdown_connections 2019-12-05T06:41:29.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.527+0000 7f2f22471700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2f00023680 0x7f2f00025af0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:29.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.527+0000 7f2f22471700 1 --2- 172.21.15.16:0/2590073314 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2f1c06df10 0x7f2f1c098300 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:29.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.527+0000 7f2f22471700 1 -- 172.21.15.16:0/2590073314 shutdown_connections 2019-12-05T06:41:29.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.527+0000 7f2f22471700 1 -- 172.21.15.16:0/2590073314 wait complete. 2019-12-05T06:41:29.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:29.527+0000 7f2f22471700 1 -- 172.21.15.16:0/2590073314 >> 172.21.15.16:0/2590073314 conn(0x7f2f1c0714f0 msgr2=0x7f2f1c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:29.539 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:41:31.541 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:31.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.699+0000 7f679cb3a700 1 Processor -- start 2019-12-05T06:41:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.699+0000 7f679cb3a700 1 -- start start 2019-12-05T06:41:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.700+0000 7f679cb3a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f679808d2c0 0x7f679808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.700+0000 7f679cb3a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67980958e0 con 0x7f679808c6d0 2019-12-05T06:41:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.700+0000 7f679cb3a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67980629e0 con 0x7f679808fef0 2019-12-05T06:41:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.700+0000 7f679cb3a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f679808df90 con 0x7f679808d2c0 2019-12-05T06:41:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.700+0000 7f679659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f679808d2c0 0x7f679808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:31.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.700+0000 7f679659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f679808d2c0 0x7f679808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58832/0 (socket says 172.21.15.16:58832) 2019-12-05T06:41:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.700+0000 7f679659c700 1 -- 172.21.15.16:0/4058421453 learned_addr learned my addr 172.21.15.16:0/4058421453 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679659c700 1 -- 172.21.15.16:0/4058421453 >> v1:172.21.15.76:6789/0 conn(0x7f679808fef0 legacy=0x7f679808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679659c700 1 -- 172.21.15.16:0/4058421453 >> v1:172.21.15.16:6789/0 conn(0x7f679808c6d0 legacy=0x7f679808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:31.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679659c700 1 -- 172.21.15.16:0/4058421453 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679808e620 con 0x7f679808d2c0 2019-12-05T06:41:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679659c700 1 --2- 172.21.15.16:0/4058421453 >> v2:172.21.15.16:3300/0 conn(0x7f679808d2c0 0x7f679808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f678000f740 tx=0x7f678000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679659c700 1 --2- 172.21.15.16:0/4058421453 >> v2:172.21.15.16:3300/0 conn(0x7f679808d2c0 0x7f679808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f678000f740 tx=0x7f678000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679659c700 1 --2- 172.21.15.16:0/4058421453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679808d2c0 0x7f679808ef20 secure :-1 s=READY pgs=1174 cs=0 l=1 rx=0x7f678000f740 tx=0x7f678000fae0).ready entity=mon.0 client_cookie=1f1946beec8291c3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:31.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679659c700 1 --2- 172.21.15.16:0/4058421453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679808d2c0 0x7f679808ef20 secure :-1 s=THROTTLE_DONE pgs=1174 cs=0 l=1 rx=0x7f678000f740 tx=0x7f678000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679659c700 1 --2- 172.21.15.16:0/4058421453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679808d2c0 0x7f679808ef20 secure :-1 s=THROTTLE_DONE pgs=1174 cs=0 l=1 rx=0x7f678000f740 tx=0x7f678000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f6795d9b700 1 -- 172.21.15.16:0/4058421453 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6780000960 con 0x7f679808d2c0 2019-12-05T06:41:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679659c700 1 --2- 172.21.15.16:0/4058421453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679808d2c0 0x7f679808ef20 secure :-1 s=THROTTLE_DONE pgs=1174 cs=0 l=1 rx=0x7f678000f740 tx=0x7f678000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f6795d9b700 1 -- 172.21.15.16:0/4058421453 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6780000c90 con 0x7f679808d2c0 2019-12-05T06:41:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f6795d9b700 1 -- 172.21.15.16:0/4058421453 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6780001360 con 0x7f679808d2c0 2019-12-05T06:41:31.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679cb3a700 1 -- 172.21.15.16:0/4058421453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679808d2c0 msgr2=0x7f679808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.701+0000 7f679cb3a700 1 --2- 172.21.15.16:0/4058421453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679808d2c0 0x7f679808ef20 secure :-1 s=READY pgs=1174 cs=0 l=1 rx=0x7f678000f740 tx=0x7f678000fae0).stop 2019-12-05T06:41:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.703+0000 7f679cb3a700 1 -- 172.21.15.16:0/4058421453 shutdown_connections 2019-12-05T06:41:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.703+0000 7f679cb3a700 1 --2- 172.21.15.16:0/4058421453 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f679808d2c0 0x7f679808ef20 unknown :-1 s=CLOSED pgs=1174 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.703+0000 7f679cb3a700 1 -- 172.21.15.16:0/4058421453 shutdown_connections 2019-12-05T06:41:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.703+0000 7f679cb3a700 1 -- 172.21.15.16:0/4058421453 wait complete. 2019-12-05T06:41:31.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.703+0000 7f679cb3a700 1 -- 172.21.15.16:0/4058421453 >> 172.21.15.16:0/4058421453 conn(0x7f6798061940 msgr2=0x7f6798096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:31.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.706+0000 7f679cb3a700 1 Processor -- start 2019-12-05T06:41:31.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.706+0000 7f679cb3a700 1 -- start start 2019-12-05T06:41:31.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.707+0000 7f679cb3a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.707+0000 7f679cb3a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6798154170 con 0x7f679809e580 2019-12-05T06:41:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.707+0000 7f679cb3a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f679806e880 con 0x7f679814c3a0 2019-12-05T06:41:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.707+0000 7f679cb3a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f679809c620 con 0x7f679806e210 2019-12-05T06:41:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.707+0000 7f679759e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f679809e580 0x7f6798152e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44008/0 (socket says 172.21.15.16:44008) 2019-12-05T06:41:31.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.707+0000 7f679759e700 1 -- 172.21.15.16:0/2809043145 learned_addr learned my addr 172.21.15.16:0/2809043145 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:31.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.707+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:31.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.707+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 552350669 0 0) 0x7f6788001030 con 0x7f679809e580 2019-12-05T06:41:31.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.707+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6770002e80 con 0x7f679809e580 2019-12-05T06:41:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760081020 0 0) 0x7f6790001480 con 0x7f679814c3a0 2019-12-05T06:41:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67700030b0 con 0x7f679814c3a0 2019-12-05T06:41:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f679659c700 1 -- 172.21.15.16:0/2809043145 >> v1:172.21.15.76:6789/0 conn(0x7f679814c3a0 legacy=0x7f679809d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f679659c700 1 -- 172.21.15.16:0/2809043145 >> v1:172.21.15.16:6789/0 conn(0x7f679809e580 legacy=0x7f6798152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:31.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4250222599 0 0) 0x7f6788001160 con 0x7f679809e580 2019-12-05T06:41:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f679659c700 1 -- 172.21.15.16:0/2809043145 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679809ccb0 con 0x7f679806e210 2019-12-05T06:41:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=READY pgs=274 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6780007700 con 0x7f679806e210 2019-12-05T06:41:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f679cb3a700 1 -- 172.21.15.16:0/2809043145 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679809ccb0 con 0x7f679806e210 2019-12-05T06:41:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6780007700 con 0x7f679806e210 2019-12-05T06:41:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.708+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.709+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6780005770 con 0x7f679806e210 2019-12-05T06:41:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.709+0000 7f679cb3a700 1 -- 172.21.15.16:0/2809043145 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679809ccb0 con 0x7f679806e210 2019-12-05T06:41:31.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.709+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.710+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6780005390 con 0x7f679806e210 2019-12-05T06:41:31.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.710+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.710+0000 7f678f7fe700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6770023650 0x7f6770025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:31.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.710+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f6780045dc0 con 0x7f679806e210 2019-12-05T06:41:31.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.711+0000 7f6796d9d700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6770023650 0x7f6770025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:31.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.711+0000 7f6796d9d700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6770023650 0x7f6770025ae0 crc :-1 s=READY pgs=503 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:31.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.714+0000 7f679cb3a700 1 -- 172.21.15.16:0/2809043145 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679809ccb0 con 0x7f679806e210 2019-12-05T06:41:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.724+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.724+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6780005180 con 0x7f679806e210 2019-12-05T06:41:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.955+0000 7f679cb3a700 1 -- 172.21.15.16:0/2809043145 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f679809ccb0 con 0x7f679806e210 2019-12-05T06:41:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.956+0000 7f679659c700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=THROTTLE_DONE pgs=274 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:31.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.957+0000 7f678f7fe700 1 -- 172.21.15.16:0/2809043145 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 146 v146) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f6780005390 con 0x7f679806e210 2019-12-05T06:41:31.959 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 146 2019-12-05T06:41:31.960 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:31.960 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":146,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:31.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.961+0000 7f679cb3a700 1 -- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6770023650 msgr2=0x7f6770025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:31.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.961+0000 7f679cb3a700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6770023650 0x7f6770025ae0 crc :-1 s=READY pgs=503 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:31.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.961+0000 7f679cb3a700 1 -- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 msgr2=0x7f679809ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:31.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.961+0000 7f679cb3a700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 secure :-1 s=READY pgs=274 cs=0 l=1 rx=0x7f6780010390 tx=0x7f6780010730).stop 2019-12-05T06:41:31.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.961+0000 7f679cb3a700 1 -- 172.21.15.16:0/2809043145 shutdown_connections 2019-12-05T06:41:31.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.961+0000 7f679cb3a700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6770023650 0x7f6770025ae0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:31.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.961+0000 7f679cb3a700 1 --2- 172.21.15.16:0/2809043145 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f679806e210 0x7f679809ddb0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:31.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.962+0000 7f679cb3a700 1 -- 172.21.15.16:0/2809043145 shutdown_connections 2019-12-05T06:41:31.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.962+0000 7f679cb3a700 1 -- 172.21.15.16:0/2809043145 wait complete. 2019-12-05T06:41:31.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:31.962+0000 7f679cb3a700 1 -- 172.21.15.16:0/2809043145 >> 172.21.15.16:0/2809043145 conn(0x7f6798098480 msgr2=0x7f6798071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:31.975 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:41:33.975 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:41:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.245+0000 7fa9c4a0e700 1 Processor -- start 2019-12-05T06:41:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.245+0000 7fa9c4a0e700 1 -- start start 2019-12-05T06:41:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.247+0000 7fa9c4a0e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa9c008c6d0 0x7fa9c008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:41:34.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.247+0000 7fa9c4a0e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c008fb80 con 0x7fa9c008f720 2019-12-05T06:41:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.247+0000 7fa9c4a0e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c00958e0 con 0x7fa9c008fef0 2019-12-05T06:41:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.247+0000 7fa9c4a0e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9c008df90 con 0x7fa9c008c6d0 2019-12-05T06:41:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.247+0000 7fa9bed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa9c008c6d0 0x7fa9c008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.248+0000 7fa9bed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa9c008c6d0 0x7fa9c008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:58856/0 (socket says 172.21.15.16:58856) 2019-12-05T06:41:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.248+0000 7fa9bed9d700 1 -- 172.21.15.16:0/4027456097 learned_addr learned my addr 172.21.15.16:0/4027456097 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:41:34.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.248+0000 7fa9bf59e700 1 --1- 172.21.15.16:0/4027456097 >> v1:172.21.15.16:6789/0 conn(0x7fa9c008f720 0x7fa9c01533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44028/0 (socket says 172.21.15.16:44028) 2019-12-05T06:41:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.248+0000 7fa9bed9d700 1 -- 172.21.15.16:0/4027456097 >> v1:172.21.15.76:6789/0 conn(0x7fa9c008fef0 legacy=0x7fa9c008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.248+0000 7fa9bed9d700 1 -- 172.21.15.16:0/4027456097 >> v1:172.21.15.16:6789/0 conn(0x7fa9c008f720 legacy=0x7fa9c01533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.248+0000 7fa9bed9d700 1 -- 172.21.15.16:0/4027456097 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c008e620 con 0x7fa9c008c6d0 2019-12-05T06:41:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.248+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/4027456097 >> v2:172.21.15.16:3300/0 conn(0x7fa9c008c6d0 0x7fa9c008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa9b000fac0 tx=0x7fa9b000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.249+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/4027456097 >> v2:172.21.15.16:3300/0 conn(0x7fa9c008c6d0 0x7fa9c008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa9b000fac0 tx=0x7fa9b000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.249+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/4027456097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c008c6d0 0x7fa9c008cac0 secure :-1 s=READY pgs=1177 cs=0 l=1 rx=0x7fa9b000fac0 tx=0x7fa9b000fe60).ready entity=mon.0 client_cookie=87a773e2e6b6ff4f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.251+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/4027456097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c008c6d0 0x7fa9c008cac0 secure :-1 s=THROTTLE_DONE pgs=1177 cs=0 l=1 rx=0x7fa9b000fac0 tx=0x7fa9b000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.251+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/4027456097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c008c6d0 0x7fa9c008cac0 secure :-1 s=THROTTLE_DONE pgs=1177 cs=0 l=1 rx=0x7fa9b000fac0 tx=0x7fa9b000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.251+0000 7fa9bdd9b700 1 -- 172.21.15.16:0/4027456097 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9b0000ce0 con 0x7fa9c008c6d0 2019-12-05T06:41:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.251+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/4027456097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c008c6d0 0x7fa9c008cac0 secure :-1 s=THROTTLE_DONE pgs=1177 cs=0 l=1 rx=0x7fa9b000fac0 tx=0x7fa9b000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.252+0000 7fa9bdd9b700 1 -- 172.21.15.16:0/4027456097 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b0001010 con 0x7fa9c008c6d0 2019-12-05T06:41:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.252+0000 7fa9bdd9b700 1 -- 172.21.15.16:0/4027456097 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9b00016e0 con 0x7fa9c008c6d0 2019-12-05T06:41:34.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.252+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/4027456097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c008c6d0 msgr2=0x7fa9c008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:34.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.252+0000 7fa9c4a0e700 1 --2- 172.21.15.16:0/4027456097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c008c6d0 0x7fa9c008cac0 secure :-1 s=READY pgs=1177 cs=0 l=1 rx=0x7fa9b000fac0 tx=0x7fa9b000fe60).stop 2019-12-05T06:41:34.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.254+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/4027456097 shutdown_connections 2019-12-05T06:41:34.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.254+0000 7fa9c4a0e700 1 --2- 172.21.15.16:0/4027456097 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa9c008c6d0 0x7fa9c008cac0 unknown :-1 s=CLOSED pgs=1177 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:34.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.254+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/4027456097 shutdown_connections 2019-12-05T06:41:34.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.254+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/4027456097 wait complete. 2019-12-05T06:41:34.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.254+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/4027456097 >> 172.21.15.16:0/4027456097 conn(0x7fa9c0061940 msgr2=0x7fa9c0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:34.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.259+0000 7fa9c4a0e700 1 Processor -- start 2019-12-05T06:41:34.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.259+0000 7fa9c4a0e700 1 -- start start 2019-12-05T06:41:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.260+0000 7fa9c4a0e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.260+0000 7fa9c4a0e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c009e030 con 0x7fa9c00879b0 2019-12-05T06:41:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.260+0000 7fa9c4a0e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9c0154970 con 0x7fa9c009ce90 2019-12-05T06:41:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.260+0000 7fa9c4a0e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9c0085020 con 0x7fa9c014ca10 2019-12-05T06:41:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.261+0000 7fa9be59c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa9c00879b0 0x7fa9c0086f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44032/0 (socket says 172.21.15.16:44032) 2019-12-05T06:41:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.261+0000 7fa9be59c700 1 -- 172.21.15.16:0/3292564940 learned_addr learned my addr 172.21.15.16:0/3292564940 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:41:34.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.261+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.261+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 392009405 0 0) 0x7fa9a8001150 con 0x7fa9c00879b0 2019-12-05T06:41:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.261+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a4002e80 con 0x7fa9c00879b0 2019-12-05T06:41:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.261+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1538937562 0 0) 0x7fa9b8000ef0 con 0x7fa9c009ce90 2019-12-05T06:41:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.261+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9a4002f30 con 0x7fa9c009ce90 2019-12-05T06:41:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.262+0000 7fa9bed9d700 1 -- 172.21.15.16:0/3292564940 >> v1:172.21.15.76:6789/0 conn(0x7fa9c009ce90 legacy=0x7fa9c0154170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:34.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.262+0000 7fa9bed9d700 1 -- 172.21.15.16:0/3292564940 >> v1:172.21.15.16:6789/0 conn(0x7fa9c00879b0 legacy=0x7fa9c0086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.262+0000 7fa9bed9d700 1 -- 172.21.15.16:0/3292564940 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9c00860e0 con 0x7fa9c014ca10 2019-12-05T06:41:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.262+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.262+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.262+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=READY pgs=275 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.262+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.262+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9b0000cc0 con 0x7fa9c014ca10 2019-12-05T06:41:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.262+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.263+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.263+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa9b00012d0 con 0x7fa9c014ca10 2019-12-05T06:41:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.263+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9b00019a0 con 0x7fa9c014ca10 2019-12-05T06:41:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.263+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/3292564940 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9c00860e0 con 0x7fa9c014ca10 2019-12-05T06:41:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.263+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/3292564940 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9c00860e0 con 0x7fa9c014ca10 2019-12-05T06:41:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.264+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.265+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa9b00014c0 con 0x7fa9c014ca10 2019-12-05T06:41:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.265+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.265+0000 7fa9b77fe700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa9a4023670 0x7fa9a4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:41:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.265+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7fa9b0046160 con 0x7fa9c014ca10 2019-12-05T06:41:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.265+0000 7fa9be59c700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa9a4023670 0x7fa9a4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:41:34.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.266+0000 7fa9be59c700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa9a4023670 0x7fa9a4025ae0 crc :-1 s=READY pgs=505 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:41:34.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.277+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/3292564940 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9c00860e0 con 0x7fa9c014ca10 2019-12-05T06:41:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.287+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.287+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa9b0000ed0 con 0x7fa9c014ca10 2019-12-05T06:41:34.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.537+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/3292564940 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa9c00860e0 con 0x7fa9c014ca10 2019-12-05T06:41:34.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.537+0000 7fa9bed9d700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=THROTTLE_DONE pgs=275 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:41:34.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.537+0000 7fa9b77fe700 1 -- 172.21.15.16:0/3292564940 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa9b00014c0 con 0x7fa9c014ca10 2019-12-05T06:41:34.539 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 147 2019-12-05T06:41:34.540 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:41:34.540 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7200,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":147}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:41:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.540+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa9a4023670 msgr2=0x7fa9a4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.540+0000 7fa9c4a0e700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa9a4023670 0x7fa9a4025ae0 crc :-1 s=READY pgs=505 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.540+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 msgr2=0x7fa9c00869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:41:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.540+0000 7fa9c4a0e700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 secure :-1 s=READY pgs=275 cs=0 l=1 rx=0x7fa9b00106f0 tx=0x7fa9b0010a90).stop 2019-12-05T06:41:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.540+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/3292564940 shutdown_connections 2019-12-05T06:41:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.540+0000 7fa9c4a0e700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa9a4023670 0x7fa9a4025ae0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.540+0000 7fa9c4a0e700 1 --2- 172.21.15.16:0/3292564940 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa9c014ca10 0x7fa9c00869e0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:41:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.540+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/3292564940 shutdown_connections 2019-12-05T06:41:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.540+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/3292564940 wait complete. 2019-12-05T06:41:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:41:34.540+0000 7fa9c4a0e700 1 -- 172.21.15.16:0/3292564940 >> 172.21.15.16:0/3292564940 conn(0x7fa9c0098480 msgr2=0x7fa9c0071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:41:34.551 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:41:34.552 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 94 secs before thrashing 2019-12-05T06:41:41.429 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:41:41.434 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:42:11.556 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:42:11.562 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:42:41.702 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:42:41.707 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:43:08.552 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.698+0000 7f7f6222f700 1 Processor -- start 2019-12-05T06:43:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.698+0000 7f7f6222f700 1 -- start start 2019-12-05T06:43:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.698+0000 7f7f6222f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 0x7f7f5c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:08.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.698+0000 7f7f6222f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c0958e0 con 0x7f7f5c08fef0 2019-12-05T06:43:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.698+0000 7f7f6222f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c0629e0 con 0x7f7f5c08f720 2019-12-05T06:43:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.698+0000 7f7f6222f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f5c0a3f70 con 0x7f7f5c08c6d0 2019-12-05T06:43:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.699+0000 7f7f537fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7f5c08fef0 0x7f7f5c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44412/0 (socket says 172.21.15.16:44412) 2019-12-05T06:43:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.699+0000 7f7f537fe700 1 -- 172.21.15.16:0/205285787 learned_addr learned my addr 172.21.15.16:0/205285787 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:08.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.699+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/205285787 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 0x7f7f5c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.699+0000 7f7f5b7fe700 1 -- 172.21.15.16:0/205285787 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3819696354 0 0) 0x7f7f44001150 con 0x7f7f5c08fef0 2019-12-05T06:43:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.699+0000 7f7f5b7fe700 1 -- 172.21.15.16:0/205285787 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f48002e80 con 0x7f7f5c08fef0 2019-12-05T06:43:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.699+0000 7f7f5b7fe700 1 -- 172.21.15.16:0/205285787 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2825305472 0 0) 0x7f7f54000d30 con 0x7f7f5c08f720 2019-12-05T06:43:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.699+0000 7f7f5bfff700 1 -- 172.21.15.16:0/205285787 >> v1:172.21.15.76:6789/0 conn(0x7f7f5c08f720 legacy=0x7f7f5c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.699+0000 7f7f5bfff700 1 -- 172.21.15.16:0/205285787 >> v1:172.21.15.16:6789/0 conn(0x7f7f5c08fef0 legacy=0x7f7f5c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:08.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.699+0000 7f7f5bfff700 1 -- 172.21.15.16:0/205285787 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c08fbc0 con 0x7f7f5c08c6d0 2019-12-05T06:43:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.699+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/205285787 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 0x7f7f5c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7f4c00af50 tx=0x7f7f4c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.700+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/205285787 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 0x7f7f5c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7f4c00af50 tx=0x7f7f4c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.700+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/205285787 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 0x7f7f5c08cac0 secure :-1 s=READY pgs=284 cs=0 l=1 rx=0x7f7f4c00af50 tx=0x7f7f4c00b760).ready entity=mon.2 client_cookie=2018fa2f0f3f02cf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.700+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/205285787 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 0x7f7f5c08cac0 secure :-1 s=THROTTLE_DONE pgs=284 cs=0 l=1 rx=0x7f7f4c00af50 tx=0x7f7f4c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.700+0000 7f7f5b7fe700 1 -- 172.21.15.16:0/205285787 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f4c000960 con 0x7f7f5c08c6d0 2019-12-05T06:43:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.700+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/205285787 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 0x7f7f5c08cac0 secure :-1 s=THROTTLE_DONE pgs=284 cs=0 l=1 rx=0x7f7f4c00af50 tx=0x7f7f4c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.700+0000 7f7f5b7fe700 1 -- 172.21.15.16:0/205285787 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c000940 con 0x7f7f5c08c6d0 2019-12-05T06:43:08.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.700+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/205285787 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 0x7f7f5c08cac0 secure :-1 s=THROTTLE_DONE pgs=284 cs=0 l=1 rx=0x7f7f4c00af50 tx=0x7f7f4c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.700+0000 7f7f6222f700 1 -- 172.21.15.16:0/205285787 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 msgr2=0x7f7f5c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.700+0000 7f7f5b7fe700 1 -- 172.21.15.16:0/205285787 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f4c009ba0 con 0x7f7f5c08c6d0 2019-12-05T06:43:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.700+0000 7f7f6222f700 1 --2- 172.21.15.16:0/205285787 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 0x7f7f5c08cac0 secure :-1 s=READY pgs=284 cs=0 l=1 rx=0x7f7f4c00af50 tx=0x7f7f4c00b760).stop 2019-12-05T06:43:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.701+0000 7f7f6222f700 1 -- 172.21.15.16:0/205285787 shutdown_connections 2019-12-05T06:43:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.701+0000 7f7f6222f700 1 --2- 172.21.15.16:0/205285787 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f5c08c6d0 0x7f7f5c08cac0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.701+0000 7f7f6222f700 1 -- 172.21.15.16:0/205285787 shutdown_connections 2019-12-05T06:43:08.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.701+0000 7f7f6222f700 1 -- 172.21.15.16:0/205285787 wait complete. 2019-12-05T06:43:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.701+0000 7f7f6222f700 1 -- 172.21.15.16:0/205285787 >> 172.21.15.16:0/205285787 conn(0x7f7f5c061940 msgr2=0x7f7f5c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.704+0000 7f7f6222f700 1 Processor -- start 2019-12-05T06:43:08.709 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.704+0000 7f7f6222f700 1 -- start start 2019-12-05T06:43:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f6222f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f5c06df50 0x7f7f5c14f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:08.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f6222f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c129940 con 0x7f7f5c129cb0 2019-12-05T06:43:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f6222f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f5c06e500 con 0x7f7f5c128ce0 2019-12-05T06:43:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f6222f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f5c0997e0 con 0x7f7f5c06df50 2019-12-05T06:43:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f5c06df50 0x7f7f5c14f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f5c06df50 0x7f7f5c14f2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59248/0 (socket says 172.21.15.16:59248) 2019-12-05T06:43:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 -- 172.21.15.16:0/1491564902 learned_addr learned my addr 172.21.15.16:0/1491564902 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:08.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 -- 172.21.15.16:0/1491564902 >> v1:172.21.15.76:6789/0 conn(0x7f7f5c128ce0 legacy=0x7f7f5c129140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 -- 172.21.15.16:0/1491564902 >> v1:172.21.15.16:6789/0 conn(0x7f7f5c129cb0 legacy=0x7f7f5c14f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 -- 172.21.15.16:0/1491564902 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c1549f0 con 0x7f7f5c06df50 2019-12-05T06:43:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/1491564902 >> v2:172.21.15.16:3300/0 conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/1491564902 >> v2:172.21.15.16:3300/0 conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:08.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=THROTTLE_DONE pgs=1190 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f597fa700 1 -- 172.21.15.16:0/1491564902 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f4c000940 con 0x7f7f5c06df50 2019-12-05T06:43:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=THROTTLE_DONE pgs=1190 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=THROTTLE_DONE pgs=1190 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f597fa700 1 -- 172.21.15.16:0/1491564902 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f4c009980 con 0x7f7f5c06df50 2019-12-05T06:43:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.705+0000 7f7f6222f700 1 -- 172.21.15.16:0/1491564902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f5c1549f0 con 0x7f7f5c06df50 2019-12-05T06:43:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.706+0000 7f7f597fa700 1 -- 172.21.15.16:0/1491564902 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f4c00a0d0 con 0x7f7f5c06df50 2019-12-05T06:43:08.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.706+0000 7f7f6222f700 1 -- 172.21.15.16:0/1491564902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f5c1549f0 con 0x7f7f5c06df50 2019-12-05T06:43:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.706+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=THROTTLE_DONE pgs=1190 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.706+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=THROTTLE_DONE pgs=1190 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.706+0000 7f7f597fa700 1 -- 172.21.15.16:0/1491564902 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7f4c009b70 con 0x7f7f5c06df50 2019-12-05T06:43:08.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.707+0000 7f7f597fa700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f40023630 0x7f7f40025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.707+0000 7f7f597fa700 1 -- 172.21.15.16:0/1491564902 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f7f4c0460c0 con 0x7f7f5c06df50 2019-12-05T06:43:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.707+0000 7f7f537fe700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f40023630 0x7f7f40025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.708+0000 7f7f537fe700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f40023630 0x7f7f40025ae0 crc :-1 s=READY pgs=508 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:08.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.713+0000 7f7f6222f700 1 -- 172.21.15.16:0/1491564902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f5c1283e0 con 0x7f7f5c06df50 2019-12-05T06:43:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.719+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=THROTTLE_DONE pgs=1190 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.719+0000 7f7f597fa700 1 -- 172.21.15.16:0/1491564902 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7f4c009580 con 0x7f7f5c06df50 2019-12-05T06:43:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.927+0000 7f7f6222f700 1 -- 172.21.15.16:0/1491564902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f5c1283e0 con 0x7f7f5c06df50 2019-12-05T06:43:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.928+0000 7f7f5bfff700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=THROTTLE_DONE pgs=1190 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.928+0000 7f7f597fa700 1 -- 172.21.15.16:0/1491564902 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7f4c009980 con 0x7f7f5c06df50 2019-12-05T06:43:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 147 2019-12-05T06:43:08.928 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:08.929 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7200,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":147}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:08.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.931+0000 7f7f6222f700 1 -- 172.21.15.16:0/1491564902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f40023630 msgr2=0x7f7f40025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:08.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.931+0000 7f7f6222f700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f40023630 0x7f7f40025ae0 crc :-1 s=READY pgs=508 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.931+0000 7f7f6222f700 1 -- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 msgr2=0x7f7f5c14f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:08.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.931+0000 7f7f6222f700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 0x7f7f5c14f2a0 secure :-1 s=READY pgs=1190 cs=0 l=1 rx=0x7f7f4c010400 tx=0x7f7f4c0107a0).stop 2019-12-05T06:43:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.931+0000 7f7f6222f700 1 -- 172.21.15.16:0/1491564902 shutdown_connections 2019-12-05T06:43:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.931+0000 7f7f6222f700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f40023630 0x7f7f40025ae0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.931+0000 7f7f6222f700 1 --2- 172.21.15.16:0/1491564902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f5c06df50 0x7f7f5c14f2a0 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.931+0000 7f7f6222f700 1 -- 172.21.15.16:0/1491564902 shutdown_connections 2019-12-05T06:43:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.931+0000 7f7f6222f700 1 -- 172.21.15.16:0/1491564902 wait complete. 2019-12-05T06:43:08.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:08.931+0000 7f7f6222f700 1 -- 172.21.15.16:0/1491564902 >> 172.21.15.16:0/1491564902 conn(0x7f7f5c06d260 msgr2=0x7f7f5c09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:08.943 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:43:08.944 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:43:08.944 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:43:08.955 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:43:08.955 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:09.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.116+0000 7f28bc4d0700 1 Processor -- start 2019-12-05T06:43:09.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.116+0000 7f28bc4d0700 1 -- start start 2019-12-05T06:43:09.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.118+0000 7f28bc4d0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28b408c6d0 0x7f28b408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:09.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.118+0000 7f28bc4d0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f28b408f720 0x7f28b408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.118+0000 7f28bc4d0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b408fb70 con 0x7f28b408fef0 2019-12-05T06:43:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.118+0000 7f28bc4d0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b408df90 con 0x7f28b408c6d0 2019-12-05T06:43:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.118+0000 7f28bc4d0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28b40a3f70 con 0x7f28b408f720 2019-12-05T06:43:09.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.118+0000 7f28baa6d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28b408c6d0 0x7f28b408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.118+0000 7f28baa6d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f28b408c6d0 0x7f28b408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59260/0 (socket says 172.21.15.16:59260) 2019-12-05T06:43:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.118+0000 7f28baa6d700 1 -- 172.21.15.16:0/1532335169 learned_addr learned my addr 172.21.15.16:0/1532335169 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.118+0000 7f28bb26e700 1 --2- 172.21.15.16:0/1532335169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f28b408f720 0x7f28b408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28baa6d700 1 -- 172.21.15.16:0/1532335169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f28b408f720 msgr2=0x7f28b408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:09.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28baa6d700 1 --2- 172.21.15.16:0/1532335169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f28b408f720 0x7f28b408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28baa6d700 1 -- 172.21.15.16:0/1532335169 >> v1:172.21.15.76:6789/0 conn(0x7f28b408fef0 legacy=0x7f28b408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28baa6d700 1 -- 172.21.15.16:0/1532335169 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b408d540 con 0x7f28b408c6d0 2019-12-05T06:43:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28baa6d700 1 --2- 172.21.15.16:0/1532335169 >> v2:172.21.15.16:3300/0 conn(0x7f28b408c6d0 0x7f28b408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f28ac00fac0 tx=0x7f28ac00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28baa6d700 1 --2- 172.21.15.16:0/1532335169 >> v2:172.21.15.16:3300/0 conn(0x7f28b408c6d0 0x7f28b408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f28ac00fac0 tx=0x7f28ac00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28baa6d700 1 --2- 172.21.15.16:0/1532335169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28b408c6d0 0x7f28b408cac0 secure :-1 s=READY pgs=1191 cs=0 l=1 rx=0x7f28ac00fac0 tx=0x7f28ac00f8f0).ready entity=mon.0 client_cookie=8742f6985d8d6639 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28baa6d700 1 --2- 172.21.15.16:0/1532335169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28b408c6d0 0x7f28b408cac0 secure :-1 s=THROTTLE_DONE pgs=1191 cs=0 l=1 rx=0x7f28ac00fac0 tx=0x7f28ac00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28baa6d700 1 --2- 172.21.15.16:0/1532335169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28b408c6d0 0x7f28b408cac0 secure :-1 s=THROTTLE_DONE pgs=1191 cs=0 l=1 rx=0x7f28ac00fac0 tx=0x7f28ac00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28b9a6b700 1 -- 172.21.15.16:0/1532335169 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28ac000ce0 con 0x7f28b408c6d0 2019-12-05T06:43:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28baa6d700 1 --2- 172.21.15.16:0/1532335169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28b408c6d0 0x7f28b408cac0 secure :-1 s=THROTTLE_DONE pgs=1191 cs=0 l=1 rx=0x7f28ac00fac0 tx=0x7f28ac00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28b9a6b700 1 -- 172.21.15.16:0/1532335169 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28ac001010 con 0x7f28b408c6d0 2019-12-05T06:43:09.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28b9a6b700 1 -- 172.21.15.16:0/1532335169 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f28ac001720 con 0x7f28b408c6d0 2019-12-05T06:43:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28bc4d0700 1 -- 172.21.15.16:0/1532335169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28b408c6d0 msgr2=0x7f28b408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.119+0000 7f28bc4d0700 1 --2- 172.21.15.16:0/1532335169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28b408c6d0 0x7f28b408cac0 secure :-1 s=READY pgs=1191 cs=0 l=1 rx=0x7f28ac00fac0 tx=0x7f28ac00f8f0).stop 2019-12-05T06:43:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.121+0000 7f28bc4d0700 1 -- 172.21.15.16:0/1532335169 shutdown_connections 2019-12-05T06:43:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.121+0000 7f28bc4d0700 1 --2- 172.21.15.16:0/1532335169 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f28b408f720 0x7f28b408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.121+0000 7f28bc4d0700 1 --2- 172.21.15.16:0/1532335169 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f28b408c6d0 0x7f28b408cac0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.121+0000 7f28bc4d0700 1 -- 172.21.15.16:0/1532335169 shutdown_connections 2019-12-05T06:43:09.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.121+0000 7f28bc4d0700 1 -- 172.21.15.16:0/1532335169 wait complete. 2019-12-05T06:43:09.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.121+0000 7f28bc4d0700 1 -- 172.21.15.16:0/1532335169 >> 172.21.15.16:0/1532335169 conn(0x7f28b4061940 msgr2=0x7f28b4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:09.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.124+0000 7f28bc4d0700 1 Processor -- start 2019-12-05T06:43:09.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.125+0000 7f28bc4d0700 1 -- start start 2019-12-05T06:43:09.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.125+0000 7f28bc4d0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f28b41309e0 0x7f28b4130210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:09.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.125+0000 7f28bc4d0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b4088c30 con 0x7f28b4081220 2019-12-05T06:43:09.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.125+0000 7f28bc4d0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b4088730 con 0x7f28b414da50 2019-12-05T06:43:09.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.125+0000 7f28bc4d0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28b408a8f0 con 0x7f28b41309e0 2019-12-05T06:43:09.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.126+0000 7f28bb26e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f28b4081220 0x7f28b41571d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44436/0 (socket says 172.21.15.16:44436) 2019-12-05T06:43:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.126+0000 7f28bb26e700 1 -- 172.21.15.16:0/3803981984 learned_addr learned my addr 172.21.15.16:0/3803981984 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.126+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> v2:172.21.15.76:3300/0 conn(0x7f28b41309e0 0x7f28b4130210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.126+0000 7f28b37fe700 1 -- 172.21.15.16:0/3803981984 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686464060 0 0) 0x7f28a40095b0 con 0x7f28b4081220 2019-12-05T06:43:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.126+0000 7f28b37fe700 1 -- 172.21.15.16:0/3803981984 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2898002e80 con 0x7f28b4081220 2019-12-05T06:43:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.126+0000 7f28ba26c700 1 -- 172.21.15.16:0/3803981984 >> v1:172.21.15.76:6789/0 conn(0x7f28b414da50 legacy=0x7f28b412fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.126+0000 7f28b37fe700 1 -- 172.21.15.16:0/3803981984 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2646292776 0 0) 0x7f28a4000a70 con 0x7f28b4081220 2019-12-05T06:43:09.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.126+0000 7f28ba26c700 1 -- 172.21.15.16:0/3803981984 >> v1:172.21.15.16:6789/0 conn(0x7f28b4081220 legacy=0x7f28b41571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:09.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.126+0000 7f28ba26c700 1 -- 172.21.15.16:0/3803981984 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b412f110 con 0x7f28b41309e0 2019-12-05T06:43:09.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> v2:172.21.15.76:3300/0 conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> v2:172.21.15.76:3300/0 conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=READY pgs=832 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:09.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=THROTTLE_DONE pgs=832 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28b37fe700 1 -- 172.21.15.16:0/3803981984 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f289c000960 con 0x7f28b41309e0 2019-12-05T06:43:09.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=THROTTLE_DONE pgs=832 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=THROTTLE_DONE pgs=832 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28b37fe700 1 -- 172.21.15.16:0/3803981984 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f289c00eef0 con 0x7f28b41309e0 2019-12-05T06:43:09.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28bc4d0700 1 -- 172.21.15.16:0/3803981984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28b412f110 con 0x7f28b41309e0 2019-12-05T06:43:09.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28b37fe700 1 -- 172.21.15.16:0/3803981984 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f289c000960 con 0x7f28b41309e0 2019-12-05T06:43:09.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.127+0000 7f28bc4d0700 1 -- 172.21.15.16:0/3803981984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28b412f110 con 0x7f28b41309e0 2019-12-05T06:43:09.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.128+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=THROTTLE_DONE pgs=832 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.129+0000 7f28b37fe700 1 -- 172.21.15.16:0/3803981984 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f289c00f500 con 0x7f28b41309e0 2019-12-05T06:43:09.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.129+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=THROTTLE_DONE pgs=832 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.129+0000 7f28b37fe700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2898023680 0x7f2898025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:09.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.129+0000 7f28b37fe700 1 -- 172.21.15.16:0/3803981984 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f289c045db0 con 0x7f28b41309e0 2019-12-05T06:43:09.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.130+0000 7f28baa6d700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2898023680 0x7f2898025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:09.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.130+0000 7f28baa6d700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2898023680 0x7f2898025af0 crc :-1 s=READY pgs=509 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:09.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.132+0000 7f28bc4d0700 1 -- 172.21.15.16:0/3803981984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28b412f110 con 0x7f28b41309e0 2019-12-05T06:43:09.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.141+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=THROTTLE_DONE pgs=832 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.141+0000 7f28b37fe700 1 -- 172.21.15.16:0/3803981984 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f289c00f2f0 con 0x7f28b41309e0 2019-12-05T06:43:09.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.384+0000 7f28bc4d0700 1 -- 172.21.15.16:0/3803981984 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f28b412f110 con 0x7f28b41309e0 2019-12-05T06:43:09.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.385+0000 7f28ba26c700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=THROTTLE_DONE pgs=832 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:09.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.386+0000 7f28b37fe700 1 -- 172.21.15.16:0/3803981984 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f289c00f500 con 0x7f28b41309e0 2019-12-05T06:43:09.388 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 147 2019-12-05T06:43:09.388 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:09.388 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7200,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":147}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:09.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.390+0000 7f28bc4d0700 1 -- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2898023680 msgr2=0x7f2898025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.390+0000 7f28bc4d0700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2898023680 0x7f2898025af0 crc :-1 s=READY pgs=509 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.390+0000 7f28bc4d0700 1 -- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 msgr2=0x7f28b4130210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.390+0000 7f28bc4d0700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 0x7f28b4130210 secure :-1 s=READY pgs=832 cs=0 l=1 rx=0x7f289c0103a0 tx=0x7f289c010740).stop 2019-12-05T06:43:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.391+0000 7f28bc4d0700 1 -- 172.21.15.16:0/3803981984 shutdown_connections 2019-12-05T06:43:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.391+0000 7f28bc4d0700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2898023680 0x7f2898025af0 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:09.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.391+0000 7f28bc4d0700 1 --2- 172.21.15.16:0/3803981984 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f28b41309e0 0x7f28b4130210 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.391+0000 7f28bc4d0700 1 -- 172.21.15.16:0/3803981984 shutdown_connections 2019-12-05T06:43:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.391+0000 7f28bc4d0700 1 -- 172.21.15.16:0/3803981984 wait complete. 2019-12-05T06:43:09.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:09.391+0000 7f28bc4d0700 1 -- 172.21.15.16:0/3803981984 >> 172.21.15.16:0/3803981984 conn(0x7f28b409de30 msgr2=0x7f28b4070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:09.401 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:43:11.404 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:11.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.675+0000 7f548885b700 1 Processor -- start 2019-12-05T06:43:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.675+0000 7f548885b700 1 -- start start 2019-12-05T06:43:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.678+0000 7f548885b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f548408d2c0 0x7f548408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.678+0000 7f548885b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f548408fef0 0x7f548408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.678+0000 7f548885b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54840958e0 con 0x7f548408c6d0 2019-12-05T06:43:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.678+0000 7f548885b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548408df90 con 0x7f548408d2c0 2019-12-05T06:43:11.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.678+0000 7f548885b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54840a3f70 con 0x7f548408fef0 2019-12-05T06:43:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.678+0000 7f548259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f548408d2c0 0x7f548408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.678+0000 7f548259c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f548408d2c0 0x7f548408ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59282/0 (socket says 172.21.15.16:59282) 2019-12-05T06:43:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.678+0000 7f548259c700 1 -- 172.21.15.16:0/1216331856 learned_addr learned my addr 172.21.15.16:0/1216331856 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:11.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.678+0000 7f5482d9d700 1 --1- 172.21.15.16:0/1216331856 >> v1:172.21.15.76:6789/0 conn(0x7f548408c6d0 0x7f548408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:37518/0 (socket says 172.21.15.16:37518) 2019-12-05T06:43:11.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.678+0000 7f548359e700 1 --2- 172.21.15.16:0/1216331856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f548408fef0 0x7f548408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:11.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.679+0000 7f5481d9b700 1 -- 172.21.15.16:0/1216331856 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2516638042 0 0) 0x7f54740010c0 con 0x7f548408c6d0 2019-12-05T06:43:11.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.679+0000 7f548259c700 1 -- 172.21.15.16:0/1216331856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f548408fef0 msgr2=0x7f548408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:11.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.679+0000 7f548259c700 1 --2- 172.21.15.16:0/1216331856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f548408fef0 0x7f548408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:11.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.679+0000 7f548259c700 1 -- 172.21.15.16:0/1216331856 >> v1:172.21.15.76:6789/0 conn(0x7f548408c6d0 legacy=0x7f548408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:11.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.679+0000 7f548259c700 1 -- 172.21.15.16:0/1216331856 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548408e620 con 0x7f548408d2c0 2019-12-05T06:43:11.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.679+0000 7f548359e700 1 --2- 172.21.15.16:0/1216331856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f548408fef0 0x7f548408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:11.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.679+0000 7f548259c700 1 --2- 172.21.15.16:0/1216331856 >> v2:172.21.15.16:3300/0 conn(0x7f548408d2c0 0x7f548408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f547800f740 tx=0x7f547800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.679+0000 7f548259c700 1 --2- 172.21.15.16:0/1216331856 >> v2:172.21.15.16:3300/0 conn(0x7f548408d2c0 0x7f548408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f547800f740 tx=0x7f547800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.679+0000 7f548259c700 1 --2- 172.21.15.16:0/1216331856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548408d2c0 0x7f548408ef20 secure :-1 s=READY pgs=1193 cs=0 l=1 rx=0x7f547800f740 tx=0x7f547800fae0).ready entity=mon.0 client_cookie=425e41266a9691a8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:11.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.680+0000 7f548259c700 1 --2- 172.21.15.16:0/1216331856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548408d2c0 0x7f548408ef20 secure :-1 s=THROTTLE_DONE pgs=1193 cs=0 l=1 rx=0x7f547800f740 tx=0x7f547800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.680+0000 7f5481d9b700 1 -- 172.21.15.16:0/1216331856 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5478000960 con 0x7f548408d2c0 2019-12-05T06:43:11.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.680+0000 7f548259c700 1 --2- 172.21.15.16:0/1216331856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548408d2c0 0x7f548408ef20 secure :-1 s=THROTTLE_DONE pgs=1193 cs=0 l=1 rx=0x7f547800f740 tx=0x7f547800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.680+0000 7f548259c700 1 --2- 172.21.15.16:0/1216331856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548408d2c0 0x7f548408ef20 secure :-1 s=THROTTLE_DONE pgs=1193 cs=0 l=1 rx=0x7f547800f740 tx=0x7f547800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.680+0000 7f5481d9b700 1 -- 172.21.15.16:0/1216331856 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5478000c90 con 0x7f548408d2c0 2019-12-05T06:43:11.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.680+0000 7f548885b700 1 -- 172.21.15.16:0/1216331856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548408d2c0 msgr2=0x7f548408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:11.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.680+0000 7f548885b700 1 --2- 172.21.15.16:0/1216331856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548408d2c0 0x7f548408ef20 secure :-1 s=READY pgs=1193 cs=0 l=1 rx=0x7f547800f740 tx=0x7f547800fae0).stop 2019-12-05T06:43:11.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.680+0000 7f5481d9b700 1 -- 172.21.15.16:0/1216331856 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54780013a0 con 0x7f548408d2c0 2019-12-05T06:43:11.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.682+0000 7f548885b700 1 -- 172.21.15.16:0/1216331856 shutdown_connections 2019-12-05T06:43:11.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.682+0000 7f548885b700 1 --2- 172.21.15.16:0/1216331856 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f548408fef0 0x7f548408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:11.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.682+0000 7f548885b700 1 --2- 172.21.15.16:0/1216331856 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f548408d2c0 0x7f548408ef20 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:11.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.682+0000 7f548885b700 1 -- 172.21.15.16:0/1216331856 shutdown_connections 2019-12-05T06:43:11.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.682+0000 7f548885b700 1 -- 172.21.15.16:0/1216331856 wait complete. 2019-12-05T06:43:11.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.682+0000 7f548885b700 1 -- 172.21.15.16:0/1216331856 >> 172.21.15.16:0/1216331856 conn(0x7f5484061940 msgr2=0x7f5484096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.685+0000 7f548885b700 1 Processor -- start 2019-12-05T06:43:11.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.685+0000 7f548885b700 1 -- start start 2019-12-05T06:43:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.686+0000 7f548885b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f548414da50 0x7f548412fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.686+0000 7f548885b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5484081220 0x7f54841571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.686+0000 7f548885b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5484088c30 con 0x7f54841309e0 2019-12-05T06:43:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.686+0000 7f548885b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f548408a8f0 con 0x7f548414da50 2019-12-05T06:43:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.686+0000 7f548885b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5484089ec0 con 0x7f5484081220 2019-12-05T06:43:11.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.686+0000 7f5482d9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f548414da50 0x7f548412fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.686+0000 7f548359e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5484081220 0x7f54841571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.686+0000 7f5482d9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f548414da50 0x7f548412fa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:45376/0 (socket says 172.21.15.16:45376) 2019-12-05T06:43:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.687+0000 7f5482d9d700 1 -- 172.21.15.16:0/940065514 learned_addr learned my addr 172.21.15.16:0/940065514 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:11.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.687+0000 7f5482d9d700 1 -- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5484081220 msgr2=0x7f54841571d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.687+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5484081220 0x7f54841571d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.687+0000 7f5482d9d700 1 -- 172.21.15.16:0/940065514 >> v1:172.21.15.76:6789/0 conn(0x7f54841309e0 legacy=0x7f548412ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.687+0000 7f5482d9d700 1 -- 172.21.15.16:0/940065514 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548412f110 con 0x7f548414da50 2019-12-05T06:43:11.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.687+0000 7f54737fe700 1 -- 172.21.15.16:0/940065514 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 359121257 0 0) 0x7f5478007e60 con 0x7f54841309e0 2019-12-05T06:43:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.687+0000 7f548359e700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5484081220 0x7f54841571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.687+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> v2:172.21.15.76:3300/0 conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.688+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> v2:172.21.15.76:3300/0 conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.688+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=READY pgs=835 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.688+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=THROTTLE_DONE pgs=835 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.688+0000 7f54737fe700 1 -- 172.21.15.16:0/940065514 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5474000e20 con 0x7f548414da50 2019-12-05T06:43:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.688+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=THROTTLE_DONE pgs=835 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.688+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=THROTTLE_DONE pgs=835 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.688+0000 7f54737fe700 1 -- 172.21.15.16:0/940065514 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5474003ff0 con 0x7f548414da50 2019-12-05T06:43:11.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.688+0000 7f548885b700 1 -- 172.21.15.16:0/940065514 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f548412f110 con 0x7f548414da50 2019-12-05T06:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.688+0000 7f54737fe700 1 -- 172.21.15.16:0/940065514 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5474004640 con 0x7f548414da50 2019-12-05T06:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.688+0000 7f548885b700 1 -- 172.21.15.16:0/940065514 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548412f110 con 0x7f548414da50 2019-12-05T06:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.689+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=THROTTLE_DONE pgs=835 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.690+0000 7f54737fe700 1 -- 172.21.15.16:0/940065514 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f54740041e0 con 0x7f548414da50 2019-12-05T06:43:11.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.690+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=THROTTLE_DONE pgs=835 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.690+0000 7f54737fe700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5468023630 0x7f5468025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.690+0000 7f54737fe700 1 -- 172.21.15.16:0/940065514 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f5474045e30 con 0x7f548414da50 2019-12-05T06:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.690+0000 7f548259c700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5468023630 0x7f5468025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.691+0000 7f548259c700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5468023630 0x7f5468025ac0 crc :-1 s=READY pgs=510 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.706+0000 7f548885b700 1 -- 172.21.15.16:0/940065514 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548412f110 con 0x7f548414da50 2019-12-05T06:43:11.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.716+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=THROTTLE_DONE pgs=835 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.717+0000 7f54737fe700 1 -- 172.21.15.16:0/940065514 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5474000940 con 0x7f548414da50 2019-12-05T06:43:11.848 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:43:11.853 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:43:11.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.922+0000 7f548885b700 1 -- 172.21.15.16:0/940065514 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f548412f110 con 0x7f548414da50 2019-12-05T06:43:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.923+0000 7f5482d9d700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=THROTTLE_DONE pgs=835 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:11.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.923+0000 7f54737fe700 1 -- 172.21.15.16:0/940065514 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f54740041e0 con 0x7f548414da50 2019-12-05T06:43:11.926 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 147 2019-12-05T06:43:11.926 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:11.927 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7200,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":147}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:11.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 -- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5468023630 msgr2=0x7f5468025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:11.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5468023630 0x7f5468025ac0 crc :-1 s=READY pgs=510 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:11.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 -- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 msgr2=0x7f548412fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:11.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 0x7f548412fa10 secure :-1 s=READY pgs=835 cs=0 l=1 rx=0x7f5474010450 tx=0x7f54740107f0).stop 2019-12-05T06:43:11.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 -- 172.21.15.16:0/940065514 shutdown_connections 2019-12-05T06:43:11.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5468023630 0x7f5468025ac0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:11.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5484081220 0x7f54841571d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:11.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 --2- 172.21.15.16:0/940065514 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f548414da50 0x7f548412fa10 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:11.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 -- 172.21.15.16:0/940065514 shutdown_connections 2019-12-05T06:43:11.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 -- 172.21.15.16:0/940065514 wait complete. 2019-12-05T06:43:11.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:11.929+0000 7f548885b700 1 -- 172.21.15.16:0/940065514 >> 172.21.15.16:0/940065514 conn(0x7f548409de30 msgr2=0x7f5484070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:11.952 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:43:13.953 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:14.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.243+0000 7f46a954c700 1 Processor -- start 2019-12-05T06:43:14.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.243+0000 7f46a954c700 1 -- start start 2019-12-05T06:43:14.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.245+0000 7f46a954c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46a408fef0 0x7f46a408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:14.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.245+0000 7f46a954c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40958e0 con 0x7f46a408d2c0 2019-12-05T06:43:14.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.245+0000 7f46a954c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a40629e0 con 0x7f46a408c6d0 2019-12-05T06:43:14.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.245+0000 7f46a954c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a408df90 con 0x7f46a408fef0 2019-12-05T06:43:14.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.245+0000 7f46a2ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f46a408d2c0 0x7f46a408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44466/0 (socket says 172.21.15.16:44466) 2019-12-05T06:43:14.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.245+0000 7f46a3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f46a408fef0 0x7f46a408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:14.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.245+0000 7f46a2ffd700 1 -- 172.21.15.16:0/3267985149 learned_addr learned my addr 172.21.15.16:0/3267985149 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:14.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.246+0000 7f46a27fc700 1 -- 172.21.15.16:0/3267985149 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 339376033 0 0) 0x7f468c001150 con 0x7f46a408d2c0 2019-12-05T06:43:14.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.246+0000 7f46a27fc700 1 -- 172.21.15.16:0/3267985149 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4690002e80 con 0x7f46a408d2c0 2019-12-05T06:43:14.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.246+0000 7f46a27fc700 1 -- 172.21.15.16:0/3267985149 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 760929121 0 0) 0x7f469c001400 con 0x7f46a408c6d0 2019-12-05T06:43:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.246+0000 7f46a27fc700 1 -- 172.21.15.16:0/3267985149 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46900030b0 con 0x7f46a408c6d0 2019-12-05T06:43:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.246+0000 7f46a3fff700 1 -- 172.21.15.16:0/3267985149 >> v1:172.21.15.76:6789/0 conn(0x7f46a408c6d0 legacy=0x7f46a408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:14.261 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.246+0000 7f46a27fc700 1 -- 172.21.15.16:0/3267985149 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 194960670 0 0) 0x7f468c001130 con 0x7f46a408d2c0 2019-12-05T06:43:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.246+0000 7f46a3fff700 1 -- 172.21.15.16:0/3267985149 >> v1:172.21.15.16:6789/0 conn(0x7f46a408d2c0 legacy=0x7f46a408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.246+0000 7f46a3fff700 1 -- 172.21.15.16:0/3267985149 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a408e620 con 0x7f46a408fef0 2019-12-05T06:43:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.246+0000 7f46a3fff700 1 --2- 172.21.15.16:0/3267985149 >> v2:172.21.15.16:3300/0 conn(0x7f46a408fef0 0x7f46a408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f469400af50 tx=0x7f469400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.247+0000 7f46a3fff700 1 --2- 172.21.15.16:0/3267985149 >> v2:172.21.15.16:3300/0 conn(0x7f46a408fef0 0x7f46a408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f469400af50 tx=0x7f469400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.262 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.247+0000 7f46a3fff700 1 --2- 172.21.15.16:0/3267985149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a408fef0 0x7f46a408f720 secure :-1 s=READY pgs=1195 cs=0 l=1 rx=0x7f469400af50 tx=0x7f469400b760).ready entity=mon.0 client_cookie=1bec2460e8e08555 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.247+0000 7f46a3fff700 1 --2- 172.21.15.16:0/3267985149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a408fef0 0x7f46a408f720 secure :-1 s=THROTTLE_DONE pgs=1195 cs=0 l=1 rx=0x7f469400af50 tx=0x7f469400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.247+0000 7f46a3fff700 1 --2- 172.21.15.16:0/3267985149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a408fef0 0x7f46a408f720 secure :-1 s=THROTTLE_DONE pgs=1195 cs=0 l=1 rx=0x7f469400af50 tx=0x7f469400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.247+0000 7f46a27fc700 1 -- 172.21.15.16:0/3267985149 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4694009ba0 con 0x7f46a408fef0 2019-12-05T06:43:14.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.247+0000 7f46a3fff700 1 --2- 172.21.15.16:0/3267985149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a408fef0 0x7f46a408f720 secure :-1 s=THROTTLE_DONE pgs=1195 cs=0 l=1 rx=0x7f469400af50 tx=0x7f469400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.247+0000 7f46a27fc700 1 -- 172.21.15.16:0/3267985149 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4694000a70 con 0x7f46a408fef0 2019-12-05T06:43:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.247+0000 7f46a27fc700 1 -- 172.21.15.16:0/3267985149 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f469400a180 con 0x7f46a408fef0 2019-12-05T06:43:14.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.247+0000 7f46a954c700 1 -- 172.21.15.16:0/3267985149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a408fef0 msgr2=0x7f46a408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.247+0000 7f46a954c700 1 --2- 172.21.15.16:0/3267985149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a408fef0 0x7f46a408f720 secure :-1 s=READY pgs=1195 cs=0 l=1 rx=0x7f469400af50 tx=0x7f469400b760).stop 2019-12-05T06:43:14.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.249+0000 7f46a954c700 1 -- 172.21.15.16:0/3267985149 shutdown_connections 2019-12-05T06:43:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.249+0000 7f46a954c700 1 --2- 172.21.15.16:0/3267985149 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f46a408fef0 0x7f46a408f720 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.249+0000 7f46a954c700 1 -- 172.21.15.16:0/3267985149 shutdown_connections 2019-12-05T06:43:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.249+0000 7f46a954c700 1 -- 172.21.15.16:0/3267985149 wait complete. 2019-12-05T06:43:14.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.249+0000 7f46a954c700 1 -- 172.21.15.16:0/3267985149 >> 172.21.15.16:0/3267985149 conn(0x7f46a4061940 msgr2=0x7f46a4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:14.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.254+0000 7f46a954c700 1 Processor -- start 2019-12-05T06:43:14.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.255+0000 7f46a954c700 1 -- start start 2019-12-05T06:43:14.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.256+0000 7f46a954c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46a4153fd0 0x7f46a406e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:14.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.256+0000 7f46a954c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f46a4130be0 0x7f46a406e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:14.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.257+0000 7f46a954c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46a4128690 con 0x7f46a406dc10 2019-12-05T06:43:14.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.257+0000 7f46a954c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46a409e690 con 0x7f46a4130be0 2019-12-05T06:43:14.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.257+0000 7f46a954c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46a412ec80 con 0x7f46a4153fd0 2019-12-05T06:43:14.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.257+0000 7f46a37fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f46a406dc10 0x7f46a41537d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:37540/0 (socket says 172.21.15.16:37540) 2019-12-05T06:43:14.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.257+0000 7f46a3fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f46a4130be0 0x7f46a406e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:14.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.257+0000 7f46a2ffd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46a4153fd0 0x7f46a406e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:14.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.257+0000 7f46a37fe700 1 -- 172.21.15.16:0/1441373275 learned_addr learned my addr 172.21.15.16:0/1441373275 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:14.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.258+0000 7f46a3fff700 1 -- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46a4153fd0 msgr2=0x7f46a406e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:14.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.258+0000 7f469bfff700 1 -- 172.21.15.16:0/1441373275 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974256496 0 0) 0x7f469c0015c0 con 0x7f46a406dc10 2019-12-05T06:43:14.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.258+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46a4153fd0 0x7f46a406e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:14.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.258+0000 7f46a3fff700 1 -- 172.21.15.16:0/1441373275 >> v1:172.21.15.76:6789/0 conn(0x7f46a406dc10 legacy=0x7f46a41537d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:14.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.258+0000 7f46a3fff700 1 -- 172.21.15.16:0/1441373275 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46a412f310 con 0x7f46a4130be0 2019-12-05T06:43:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.258+0000 7f46a2ffd700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46a4153fd0 0x7f46a406e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.258+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> v2:172.21.15.76:3300/0 conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.258+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> v2:172.21.15.76:3300/0 conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.258+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=READY pgs=838 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.258+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.259+0000 7f469bfff700 1 -- 172.21.15.16:0/1441373275 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f469400ac20 con 0x7f46a4130be0 2019-12-05T06:43:14.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.259+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.259+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.259+0000 7f469bfff700 1 -- 172.21.15.16:0/1441373275 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4694009770 con 0x7f46a4130be0 2019-12-05T06:43:14.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.259+0000 7f469bfff700 1 -- 172.21.15.16:0/1441373275 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4694009e40 con 0x7f46a4130be0 2019-12-05T06:43:14.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.259+0000 7f46a954c700 1 -- 172.21.15.16:0/1441373275 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46a412f310 con 0x7f46a4130be0 2019-12-05T06:43:14.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.259+0000 7f46a954c700 1 -- 172.21.15.16:0/1441373275 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46a412f310 con 0x7f46a4130be0 2019-12-05T06:43:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.260+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.261+0000 7f469bfff700 1 -- 172.21.15.16:0/1441373275 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4694009960 con 0x7f46a4130be0 2019-12-05T06:43:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.261+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.261+0000 7f469bfff700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4688023630 0x7f4688025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.261+0000 7f469bfff700 1 -- 172.21.15.16:0/1441373275 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f4694045d40 con 0x7f46a4130be0 2019-12-05T06:43:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.261+0000 7f46a37fe700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4688023630 0x7f4688025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:14.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.262+0000 7f46a37fe700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4688023630 0x7f4688025ac0 crc :-1 s=READY pgs=511 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.274+0000 7f46a954c700 1 -- 172.21.15.16:0/1441373275 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46a412f310 con 0x7f46a4130be0 2019-12-05T06:43:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.284+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.284+0000 7f469bfff700 1 -- 172.21.15.16:0/1441373275 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4694000940 con 0x7f46a4130be0 2019-12-05T06:43:14.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.553+0000 7f46a954c700 1 -- 172.21.15.16:0/1441373275 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f46a412f310 con 0x7f46a4130be0 2019-12-05T06:43:14.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.554+0000 7f46a3fff700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=THROTTLE_DONE pgs=838 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:14.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.554+0000 7f469bfff700 1 -- 172.21.15.16:0/1441373275 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f4694009960 con 0x7f46a4130be0 2019-12-05T06:43:14.557 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:14.557 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7200,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":147}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:14.558 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 147 2019-12-05T06:43:14.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 -- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4688023630 msgr2=0x7f4688025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:14.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4688023630 0x7f4688025ac0 crc :-1 s=READY pgs=511 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:14.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 -- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 msgr2=0x7f46a406e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:14.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 0x7f46a406e540 secure :-1 s=READY pgs=838 cs=0 l=1 rx=0x7f4694010380 tx=0x7f4694010720).stop 2019-12-05T06:43:14.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 -- 172.21.15.16:0/1441373275 shutdown_connections 2019-12-05T06:43:14.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4688023630 0x7f4688025ac0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:14.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f46a4130be0 0x7f46a406e540 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:14.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 --2- 172.21.15.16:0/1441373275 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f46a4153fd0 0x7f46a406e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:14.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 -- 172.21.15.16:0/1441373275 shutdown_connections 2019-12-05T06:43:14.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 -- 172.21.15.16:0/1441373275 wait complete. 2019-12-05T06:43:14.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:14.559+0000 7f46a954c700 1 -- 172.21.15.16:0/1441373275 >> 172.21.15.16:0/1441373275 conn(0x7f46a4098480 msgr2=0x7f46a4071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:14.571 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:43:16.575 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:16.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.835+0000 7f9ede713700 1 Processor -- start 2019-12-05T06:43:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.835+0000 7f9ede713700 1 -- start start 2019-12-05T06:43:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.837+0000 7f9ede713700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9ed808d2c0 0x7f9ed808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.837+0000 7f9ede713700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ed808fef0 0x7f9ed808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.837+0000 7f9ede713700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed80958e0 con 0x7f9ed808c6d0 2019-12-05T06:43:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.837+0000 7f9ede713700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed808df90 con 0x7f9ed808fef0 2019-12-05T06:43:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.837+0000 7f9ede713700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed80a3f70 con 0x7f9ed808d2c0 2019-12-05T06:43:16.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.837+0000 7f9edd4b1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ed808fef0 0x7f9ed808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.837+0000 7f9edd4b1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9ed808fef0 0x7f9ed808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59318/0 (socket says 172.21.15.16:59318) 2019-12-05T06:43:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.837+0000 7f9edd4b1700 1 -- 172.21.15.16:0/864284120 learned_addr learned my addr 172.21.15.16:0/864284120 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.837+0000 7f9ed7fff700 1 --2- 172.21.15.16:0/864284120 >> v2:172.21.15.76:3300/0 conn(0x7f9ed808d2c0 0x7f9ed808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.838+0000 7f9edd4b1700 1 -- 172.21.15.16:0/864284120 >> v2:172.21.15.76:3300/0 conn(0x7f9ed808d2c0 msgr2=0x7f9ed808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.838+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/864284120 >> v2:172.21.15.76:3300/0 conn(0x7f9ed808d2c0 0x7f9ed808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.838+0000 7f9edd4b1700 1 -- 172.21.15.16:0/864284120 >> v1:172.21.15.76:6789/0 conn(0x7f9ed808c6d0 legacy=0x7f9ed808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:16.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.838+0000 7f9edd4b1700 1 -- 172.21.15.16:0/864284120 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed808e620 con 0x7f9ed808fef0 2019-12-05T06:43:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.838+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/864284120 >> v2:172.21.15.16:3300/0 conn(0x7f9ed808fef0 0x7f9ed808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9ec000f730 tx=0x7f9ec0007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.839+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/864284120 >> v2:172.21.15.16:3300/0 conn(0x7f9ed808fef0 0x7f9ed808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9ec000f730 tx=0x7f9ec0007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.839+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/864284120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed808fef0 0x7f9ed808f460 secure :-1 s=READY pgs=1198 cs=0 l=1 rx=0x7f9ec000f730 tx=0x7f9ec0007d50).ready entity=mon.0 client_cookie=991a457db329f903 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.839+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/864284120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed808fef0 0x7f9ed808f460 secure :-1 s=THROTTLE_DONE pgs=1198 cs=0 l=1 rx=0x7f9ec000f730 tx=0x7f9ec0007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.839+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/864284120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed808fef0 0x7f9ed808f460 secure :-1 s=THROTTLE_DONE pgs=1198 cs=0 l=1 rx=0x7f9ec000f730 tx=0x7f9ec0007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.839+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/864284120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed808fef0 0x7f9ed808f460 secure :-1 s=THROTTLE_DONE pgs=1198 cs=0 l=1 rx=0x7f9ec000f730 tx=0x7f9ec0007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.839+0000 7f9ed77fe700 1 -- 172.21.15.16:0/864284120 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ec0000a70 con 0x7f9ed808fef0 2019-12-05T06:43:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.839+0000 7f9ed77fe700 1 -- 172.21.15.16:0/864284120 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec00010e0 con 0x7f9ed808fef0 2019-12-05T06:43:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.839+0000 7f9ed77fe700 1 -- 172.21.15.16:0/864284120 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ec00017b0 con 0x7f9ed808fef0 2019-12-05T06:43:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.839+0000 7f9ede713700 1 -- 172.21.15.16:0/864284120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed808fef0 msgr2=0x7f9ed808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.839+0000 7f9ede713700 1 --2- 172.21.15.16:0/864284120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed808fef0 0x7f9ed808f460 secure :-1 s=READY pgs=1198 cs=0 l=1 rx=0x7f9ec000f730 tx=0x7f9ec0007d50).stop 2019-12-05T06:43:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.841+0000 7f9ede713700 1 -- 172.21.15.16:0/864284120 shutdown_connections 2019-12-05T06:43:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.841+0000 7f9ede713700 1 --2- 172.21.15.16:0/864284120 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9ed808fef0 0x7f9ed808f460 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:16.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.841+0000 7f9ede713700 1 --2- 172.21.15.16:0/864284120 >> v2:172.21.15.76:3300/0 conn(0x7f9ed808d2c0 0x7f9ed808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.841+0000 7f9ede713700 1 -- 172.21.15.16:0/864284120 shutdown_connections 2019-12-05T06:43:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.841+0000 7f9ede713700 1 -- 172.21.15.16:0/864284120 wait complete. 2019-12-05T06:43:16.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.841+0000 7f9ede713700 1 -- 172.21.15.16:0/864284120 >> 172.21.15.16:0/864284120 conn(0x7f9ed8061940 msgr2=0x7f9ed8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:16.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.847+0000 7f9ede713700 1 Processor -- start 2019-12-05T06:43:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.847+0000 7f9ede713700 1 -- start start 2019-12-05T06:43:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.849+0000 7f9ede713700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9ed80879b0 0x7f9ed80871e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.849+0000 7f9ede713700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.849+0000 7f9ede713700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ed8153170 con 0x7f9ed814ca10 2019-12-05T06:43:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.849+0000 7f9ede713700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ed8085a50 con 0x7f9ed80879b0 2019-12-05T06:43:16.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.849+0000 7f9ede713700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ed8085020 con 0x7f9ed809be80 2019-12-05T06:43:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.850+0000 7f9edccb0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9ed814ca10 0x7f9ed80869e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44488/0 (socket says 172.21.15.16:44488) 2019-12-05T06:43:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.850+0000 7f9edccb0700 1 -- 172.21.15.16:0/2491516675 learned_addr learned my addr 172.21.15.16:0/2491516675 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.850+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.850+0000 7f9ed7fff700 1 --2- 172.21.15.16:0/2491516675 >> v2:172.21.15.76:3300/0 conn(0x7f9ed80879b0 0x7f9ed80871e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.850+0000 7f9ed57fa700 1 -- 172.21.15.16:0/2491516675 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29153941 0 0) 0x7f9ed00014d0 con 0x7f9ed814ca10 2019-12-05T06:43:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.850+0000 7f9ed57fa700 1 -- 172.21.15.16:0/2491516675 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ebc002e80 con 0x7f9ed814ca10 2019-12-05T06:43:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.850+0000 7f9edd4b1700 1 -- 172.21.15.16:0/2491516675 >> v2:172.21.15.76:3300/0 conn(0x7f9ed80879b0 msgr2=0x7f9ed80871e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:16.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.850+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> v2:172.21.15.76:3300/0 conn(0x7f9ed80879b0 0x7f9ed80871e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:16.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9edd4b1700 1 -- 172.21.15.16:0/2491516675 >> v1:172.21.15.16:6789/0 conn(0x7f9ed814ca10 legacy=0x7f9ed80869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:16.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9edd4b1700 1 -- 172.21.15.16:0/2491516675 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ed80860e0 con 0x7f9ed809be80 2019-12-05T06:43:16.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=READY pgs=285 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:16.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9ed57fa700 1 -- 172.21.15.16:0/2491516675 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ec00014a0 con 0x7f9ed809be80 2019-12-05T06:43:16.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9ed57fa700 1 -- 172.21.15.16:0/2491516675 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ec00010c0 con 0x7f9ed809be80 2019-12-05T06:43:16.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9ede713700 1 -- 172.21.15.16:0/2491516675 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ed80860e0 con 0x7f9ed809be80 2019-12-05T06:43:16.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9ed57fa700 1 -- 172.21.15.16:0/2491516675 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ec0001a80 con 0x7f9ed809be80 2019-12-05T06:43:16.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.851+0000 7f9ede713700 1 -- 172.21.15.16:0/2491516675 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ed80860e0 con 0x7f9ed809be80 2019-12-05T06:43:16.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.852+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.853+0000 7f9ed57fa700 1 -- 172.21.15.16:0/2491516675 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9ec00012b0 con 0x7f9ed809be80 2019-12-05T06:43:16.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.853+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.853+0000 7f9ed57fa700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ebc023670 0x7f9ebc025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:16.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.853+0000 7f9ed57fa700 1 -- 172.21.15.16:0/2491516675 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f9ec0045ce0 con 0x7f9ed809be80 2019-12-05T06:43:16.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.854+0000 7f9edccb0700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ebc023670 0x7f9ebc025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:16.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.854+0000 7f9edccb0700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ebc023670 0x7f9ebc025ae0 crc :-1 s=READY pgs=512 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:16.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.866+0000 7f9ede713700 1 -- 172.21.15.16:0/2491516675 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ed80860e0 con 0x7f9ed809be80 2019-12-05T06:43:16.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.876+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:16.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:16.876+0000 7f9ed57fa700 1 -- 172.21.15.16:0/2491516675 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9ec0000eb0 con 0x7f9ed809be80 2019-12-05T06:43:17.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.135+0000 7f9ede713700 1 -- 172.21.15.16:0/2491516675 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9ed80860e0 con 0x7f9ed809be80 2019-12-05T06:43:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.136+0000 7f9edd4b1700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=THROTTLE_DONE pgs=285 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:17.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.136+0000 7f9ed57fa700 1 -- 172.21.15.16:0/2491516675 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9ec00012b0 con 0x7f9ed809be80 2019-12-05T06:43:17.139 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 147 2019-12-05T06:43:17.140 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:17.140 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7200,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":147}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 -- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ebc023670 msgr2=0x7f9ebc025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ebc023670 0x7f9ebc025ae0 crc :-1 s=READY pgs=512 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 -- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 msgr2=0x7f9ed809c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 secure :-1 s=READY pgs=285 cs=0 l=1 rx=0x7f9ec00103c0 tx=0x7f9ec00101f0).stop 2019-12-05T06:43:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 -- 172.21.15.16:0/2491516675 shutdown_connections 2019-12-05T06:43:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9ebc023670 0x7f9ebc025ae0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 --2- 172.21.15.16:0/2491516675 >> v2:172.21.15.76:3300/0 conn(0x7f9ed80879b0 0x7f9ed80871e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 --2- 172.21.15.16:0/2491516675 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9ed809be80 0x7f9ed809c2d0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 -- 172.21.15.16:0/2491516675 shutdown_connections 2019-12-05T06:43:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 -- 172.21.15.16:0/2491516675 wait complete. 2019-12-05T06:43:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:17.141+0000 7f9ede713700 1 -- 172.21.15.16:0/2491516675 >> 172.21.15.16:0/2491516675 conn(0x7f9ed8098480 msgr2=0x7f9ed8071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:17.154 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:43:19.155 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:19.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.414+0000 7fab0d993700 1 Processor -- start 2019-12-05T06:43:19.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.414+0000 7fab0d993700 1 -- start start 2019-12-05T06:43:19.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.416+0000 7fab0d993700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fab0808c6d0 0x7fab0808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:19.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.416+0000 7fab0d993700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab0808fef0 0x7fab0808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:19.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.416+0000 7fab0d993700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab080958e0 con 0x7fab0808ef20 2019-12-05T06:43:19.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.416+0000 7fab0d993700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0808df90 con 0x7fab0808c6d0 2019-12-05T06:43:19.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.416+0000 7fab0d993700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab080a3f70 con 0x7fab0808fef0 2019-12-05T06:43:19.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.417+0000 7fab077fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fab0808c6d0 0x7fab0808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.417+0000 7fab06ffd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab0808fef0 0x7fab0808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.417+0000 7fab06ffd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab0808fef0 0x7fab0808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:35090/0 (socket says 172.21.15.16:35090) 2019-12-05T06:43:19.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.417+0000 7fab077fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fab0808c6d0 0x7fab0808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:45418/0 (socket says 172.21.15.16:45418) 2019-12-05T06:43:19.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.417+0000 7fab06ffd700 1 -- 172.21.15.16:0/2353873387 learned_addr learned my addr 172.21.15.16:0/2353873387 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:19.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.417+0000 7fab077fe700 1 -- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab0808fef0 msgr2=0x7fab0808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:19.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.417+0000 7fab077fe700 1 --2- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab0808fef0 0x7fab0808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:19.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.417+0000 7fab077fe700 1 -- 172.21.15.16:0/2353873387 >> v1:172.21.15.76:6789/0 conn(0x7fab0808ef20 legacy=0x7fab0808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.418+0000 7fab077fe700 1 -- 172.21.15.16:0/2353873387 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0808d540 con 0x7fab0808c6d0 2019-12-05T06:43:19.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.417+0000 7fab067fc700 1 -- 172.21.15.16:0/2353873387 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3424364126 0 0) 0x7faaf80009b0 con 0x7fab0808ef20 2019-12-05T06:43:19.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.418+0000 7fab06ffd700 1 --2- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab0808fef0 0x7fab0808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:19.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.418+0000 7fab077fe700 1 --2- 172.21.15.16:0/2353873387 >> v2:172.21.15.76:3300/0 conn(0x7fab0808c6d0 0x7fab0808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fab0000b2d0 tx=0x7fab0000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.418+0000 7fab077fe700 1 --2- 172.21.15.16:0/2353873387 >> v2:172.21.15.76:3300/0 conn(0x7fab0808c6d0 0x7fab0808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fab0000b2d0 tx=0x7fab0000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.418+0000 7fab077fe700 1 --2- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab0808c6d0 0x7fab0808cac0 secure :-1 s=READY pgs=841 cs=0 l=1 rx=0x7fab0000b2d0 tx=0x7fab0000bae0).ready entity=mon.1 client_cookie=f79107b6c6e2affa server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:19.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.418+0000 7fab077fe700 1 --2- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab0808c6d0 0x7fab0808cac0 secure :-1 s=THROTTLE_DONE pgs=841 cs=0 l=1 rx=0x7fab0000b2d0 tx=0x7fab0000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.418+0000 7fab067fc700 1 -- 172.21.15.16:0/2353873387 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab00007590 con 0x7fab0808c6d0 2019-12-05T06:43:19.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.418+0000 7fab077fe700 1 --2- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab0808c6d0 0x7fab0808cac0 secure :-1 s=THROTTLE_DONE pgs=841 cs=0 l=1 rx=0x7fab0000b2d0 tx=0x7fab0000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.419+0000 7fab077fe700 1 --2- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab0808c6d0 0x7fab0808cac0 secure :-1 s=THROTTLE_DONE pgs=841 cs=0 l=1 rx=0x7fab0000b2d0 tx=0x7fab0000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.419+0000 7fab067fc700 1 -- 172.21.15.16:0/2353873387 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab00009d30 con 0x7fab0808c6d0 2019-12-05T06:43:19.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.419+0000 7fab067fc700 1 -- 172.21.15.16:0/2353873387 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab0000a400 con 0x7fab0808c6d0 2019-12-05T06:43:19.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.419+0000 7fab0d993700 1 -- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab0808c6d0 msgr2=0x7fab0808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.419+0000 7fab0d993700 1 --2- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab0808c6d0 0x7fab0808cac0 secure :-1 s=READY pgs=841 cs=0 l=1 rx=0x7fab0000b2d0 tx=0x7fab0000bae0).stop 2019-12-05T06:43:19.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.421+0000 7fab0d993700 1 -- 172.21.15.16:0/2353873387 shutdown_connections 2019-12-05T06:43:19.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.421+0000 7fab0d993700 1 --2- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fab0808fef0 0x7fab0808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:19.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.421+0000 7fab0d993700 1 --2- 172.21.15.16:0/2353873387 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fab0808c6d0 0x7fab0808cac0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.421+0000 7fab0d993700 1 -- 172.21.15.16:0/2353873387 shutdown_connections 2019-12-05T06:43:19.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.421+0000 7fab0d993700 1 -- 172.21.15.16:0/2353873387 wait complete. 2019-12-05T06:43:19.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.421+0000 7fab0d993700 1 -- 172.21.15.16:0/2353873387 >> 172.21.15.16:0/2353873387 conn(0x7fab08061940 msgr2=0x7fab08096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:19.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.426+0000 7fab0d993700 1 Processor -- start 2019-12-05T06:43:19.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.426+0000 7fab0d993700 1 -- start start 2019-12-05T06:43:19.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.428+0000 7fab0d993700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab0812c740 0x7fab0812bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:19.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.428+0000 7fab0d993700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fab08154c60 0x7fab08157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:19.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.428+0000 7fab0d993700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab08128b20 con 0x7fab081325e0 2019-12-05T06:43:19.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.428+0000 7fab0d993700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab0812a7e0 con 0x7fab0812c740 2019-12-05T06:43:19.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.428+0000 7fab0d993700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab08129db0 con 0x7fab08154c60 2019-12-05T06:43:19.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.428+0000 7fab06ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab0812c740 0x7fab0812bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.428+0000 7fab06ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fab0812c740 0x7fab0812bf70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59336/0 (socket says 172.21.15.16:59336) 2019-12-05T06:43:19.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.428+0000 7fab06ffd700 1 -- 172.21.15.16:0/1505841055 learned_addr learned my addr 172.21.15.16:0/1505841055 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:19.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.428+0000 7fab07fff700 1 --2- 172.21.15.16:0/1505841055 >> v2:172.21.15.76:3300/0 conn(0x7fab08154c60 0x7fab08157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab06ffd700 1 -- 172.21.15.16:0/1505841055 >> v2:172.21.15.76:3300/0 conn(0x7fab08154c60 msgr2=0x7fab08157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> v2:172.21.15.76:3300/0 conn(0x7fab08154c60 0x7fab08157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab06ffd700 1 -- 172.21.15.16:0/1505841055 >> v1:172.21.15.16:6789/0 conn(0x7fab081325e0 legacy=0x7fab0812b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab06ffd700 1 -- 172.21.15.16:0/1505841055 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0812ae70 con 0x7fab0812c740 2019-12-05T06:43:19.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> v2:172.21.15.16:3300/0 conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> v2:172.21.15.16:3300/0 conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=READY pgs=1201 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:19.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=THROTTLE_DONE pgs=1201 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7faafffff700 1 -- 172.21.15.16:0/1505841055 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaf0000940 con 0x7fab0812c740 2019-12-05T06:43:19.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=THROTTLE_DONE pgs=1201 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7faafffff700 1 -- 172.21.15.16:0/1505841055 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faaf0000e80 con 0x7fab0812c740 2019-12-05T06:43:19.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=THROTTLE_DONE pgs=1201 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.429+0000 7fab0d993700 1 -- 172.21.15.16:0/1505841055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab0812ae70 con 0x7fab0812c740 2019-12-05T06:43:19.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.430+0000 7faafffff700 1 -- 172.21.15.16:0/1505841055 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faaf0001550 con 0x7fab0812c740 2019-12-05T06:43:19.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.430+0000 7fab0d993700 1 -- 172.21.15.16:0/1505841055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab0812ae70 con 0x7fab0812c740 2019-12-05T06:43:19.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.431+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=THROTTLE_DONE pgs=1201 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.433+0000 7faafffff700 1 -- 172.21.15.16:0/1505841055 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faaf0001070 con 0x7fab0812c740 2019-12-05T06:43:19.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.433+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=THROTTLE_DONE pgs=1201 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.433+0000 7faafffff700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaec023630 0x7faaec025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:19.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.433+0000 7faafffff700 1 -- 172.21.15.16:0/1505841055 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7faaf0045df0 con 0x7fab0812c740 2019-12-05T06:43:19.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.433+0000 7fab077fe700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaec023630 0x7faaec025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:19.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.434+0000 7fab077fe700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaec023630 0x7faaec025ac0 crc :-1 s=READY pgs=513 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:19.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.446+0000 7fab0d993700 1 -- 172.21.15.16:0/1505841055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab0812ae70 con 0x7fab0812c740 2019-12-05T06:43:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.454+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=THROTTLE_DONE pgs=1201 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.455 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.454+0000 7faafffff700 1 -- 172.21.15.16:0/1505841055 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faaf0007610 con 0x7fab0812c740 2019-12-05T06:43:19.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.693+0000 7fab0d993700 1 -- 172.21.15.16:0/1505841055 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab0812ae70 con 0x7fab0812c740 2019-12-05T06:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.694+0000 7fab06ffd700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=THROTTLE_DONE pgs=1201 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.694+0000 7faafffff700 1 -- 172.21.15.16:0/1505841055 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+2112 (secure 0 0 0) 0x7faaf0007610 con 0x7fab0812c740 2019-12-05T06:43:19.695 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 147 2019-12-05T06:43:19.696 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:19.696 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7200,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":147}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:19.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.696+0000 7fab0d993700 1 -- 172.21.15.16:0/1505841055 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaec023630 msgr2=0x7faaec025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.696+0000 7fab0d993700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaec023630 0x7faaec025ac0 crc :-1 s=READY pgs=513 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.696+0000 7fab0d993700 1 -- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 msgr2=0x7fab0812bf70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.696+0000 7fab0d993700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 0x7fab0812bf70 secure :-1 s=READY pgs=1201 cs=0 l=1 rx=0x7faaf0010460 tx=0x7faaf0010290).stop 2019-12-05T06:43:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.696+0000 7fab0d993700 1 -- 172.21.15.16:0/1505841055 shutdown_connections 2019-12-05T06:43:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.696+0000 7fab0d993700 1 --2- 172.21.15.16:0/1505841055 >> v2:172.21.15.76:3300/0 conn(0x7fab08154c60 0x7fab08157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.697+0000 7fab0d993700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faaec023630 0x7faaec025ac0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.697+0000 7fab0d993700 1 --2- 172.21.15.16:0/1505841055 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fab0812c740 0x7fab0812bf70 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.697+0000 7fab0d993700 1 -- 172.21.15.16:0/1505841055 shutdown_connections 2019-12-05T06:43:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.697+0000 7fab0d993700 1 -- 172.21.15.16:0/1505841055 wait complete. 2019-12-05T06:43:19.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:19.697+0000 7fab0d993700 1 -- 172.21.15.16:0/1505841055 >> 172.21.15.16:0/1505841055 conn(0x7fab080a2850 msgr2=0x7fab0806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:19.708 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:43:21.708 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:21.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.972+0000 7f247537d700 1 Processor -- start 2019-12-05T06:43:21.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.973+0000 7f247537d700 1 -- start start 2019-12-05T06:43:21.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.974+0000 7f247537d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f247008c6d0 0x7f247008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:21.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.974+0000 7f247537d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f247008fef0 0x7f247008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.974+0000 7f247537d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24700958e0 con 0x7f247008ef20 2019-12-05T06:43:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.974+0000 7f247537d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f247008df90 con 0x7f247008c6d0 2019-12-05T06:43:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.974+0000 7f247537d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24700a3f70 con 0x7f247008fef0 2019-12-05T06:43:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.975+0000 7f246f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f247008c6d0 0x7f247008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:21.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.975+0000 7f246ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f247008ef20 0x7f247008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44522/0 (socket says 172.21.15.16:44522) 2019-12-05T06:43:21.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.975+0000 7f246f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f247008c6d0 0x7f247008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59352/0 (socket says 172.21.15.16:59352) 2019-12-05T06:43:21.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.975+0000 7f246ffff700 1 -- 172.21.15.16:0/1752413099 learned_addr learned my addr 172.21.15.16:0/1752413099 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.975+0000 7f246effd700 1 --2- 172.21.15.16:0/1752413099 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f247008fef0 0x7f247008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246f7fe700 1 -- 172.21.15.16:0/1752413099 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f247008fef0 msgr2=0x7f247008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1752413099 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f247008fef0 0x7f247008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246f7fe700 1 -- 172.21.15.16:0/1752413099 >> v1:172.21.15.16:6789/0 conn(0x7f247008ef20 legacy=0x7f247008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246f7fe700 1 -- 172.21.15.16:0/1752413099 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f247008d540 con 0x7f247008c6d0 2019-12-05T06:43:21.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246effd700 1 --2- 172.21.15.16:0/1752413099 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f247008fef0 0x7f247008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1752413099 >> v2:172.21.15.16:3300/0 conn(0x7f247008c6d0 0x7f247008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f246800b2d0 tx=0x7f246800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1752413099 >> v2:172.21.15.16:3300/0 conn(0x7f247008c6d0 0x7f247008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f246800b2d0 tx=0x7f246800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1752413099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247008c6d0 0x7f247008cac0 secure :-1 s=READY pgs=1203 cs=0 l=1 rx=0x7f246800b2d0 tx=0x7f246800bae0).ready entity=mon.0 client_cookie=571bdba2ef734611 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1752413099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247008c6d0 0x7f247008cac0 secure :-1 s=THROTTLE_DONE pgs=1203 cs=0 l=1 rx=0x7f246800b2d0 tx=0x7f246800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1752413099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247008c6d0 0x7f247008cac0 secure :-1 s=THROTTLE_DONE pgs=1203 cs=0 l=1 rx=0x7f246800b2d0 tx=0x7f246800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:21.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1752413099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247008c6d0 0x7f247008cac0 secure :-1 s=THROTTLE_DONE pgs=1203 cs=0 l=1 rx=0x7f246800b2d0 tx=0x7f246800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.976+0000 7f246e7fc700 1 -- 172.21.15.16:0/1752413099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2468007590 con 0x7f247008c6d0 2019-12-05T06:43:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.977+0000 7f246e7fc700 1 -- 172.21.15.16:0/1752413099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2468009d30 con 0x7f247008c6d0 2019-12-05T06:43:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.977+0000 7f246e7fc700 1 -- 172.21.15.16:0/1752413099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f246800a400 con 0x7f247008c6d0 2019-12-05T06:43:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.977+0000 7f247537d700 1 -- 172.21.15.16:0/1752413099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247008c6d0 msgr2=0x7f247008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.977+0000 7f247537d700 1 --2- 172.21.15.16:0/1752413099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247008c6d0 0x7f247008cac0 secure :-1 s=READY pgs=1203 cs=0 l=1 rx=0x7f246800b2d0 tx=0x7f246800bae0).stop 2019-12-05T06:43:21.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.979+0000 7f247537d700 1 -- 172.21.15.16:0/1752413099 shutdown_connections 2019-12-05T06:43:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.979+0000 7f247537d700 1 --2- 172.21.15.16:0/1752413099 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f247008fef0 0x7f247008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:21.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.979+0000 7f247537d700 1 --2- 172.21.15.16:0/1752413099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f247008c6d0 0x7f247008cac0 unknown :-1 s=CLOSED pgs=1203 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.979+0000 7f247537d700 1 -- 172.21.15.16:0/1752413099 shutdown_connections 2019-12-05T06:43:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.979+0000 7f247537d700 1 -- 172.21.15.16:0/1752413099 wait complete. 2019-12-05T06:43:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.979+0000 7f247537d700 1 -- 172.21.15.16:0/1752413099 >> 172.21.15.16:0/1752413099 conn(0x7f2470061940 msgr2=0x7f2470096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.984+0000 7f247537d700 1 Processor -- start 2019-12-05T06:43:21.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.984+0000 7f247537d700 1 -- start start 2019-12-05T06:43:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.986+0000 7f247537d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.986+0000 7f247537d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f247006e7c0 con 0x7f247009ae80 2019-12-05T06:43:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.986+0000 7f247537d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2470133b30 con 0x7f2470099eb0 2019-12-05T06:43:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.986+0000 7f247537d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2470134390 con 0x7f2470132630 2019-12-05T06:43:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.986+0000 7f246effd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f247009ae80 0x7f2470132f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44528/0 (socket says 172.21.15.16:44528) 2019-12-05T06:43:22.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.986+0000 7f246effd700 1 -- 172.21.15.16:0/1621478426 learned_addr learned my addr 172.21.15.16:0/1621478426 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:22.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.986+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:22.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.987+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891239461 0 0) 0x7f245800dd90 con 0x7f247009ae80 2019-12-05T06:43:22.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.987+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2454002e80 con 0x7f247009ae80 2019-12-05T06:43:22.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.987+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 374146728 0 0) 0x7f2460000f40 con 0x7f2470099eb0 2019-12-05T06:43:22.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.987+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f24540030b0 con 0x7f2470099eb0 2019-12-05T06:43:22.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.987+0000 7f246f7fe700 1 -- 172.21.15.16:0/1621478426 >> v1:172.21.15.76:6789/0 conn(0x7f2470099eb0 legacy=0x7f247009a330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:22.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.987+0000 7f246f7fe700 1 -- 172.21.15.16:0/1621478426 >> v1:172.21.15.16:6789/0 conn(0x7f247009ae80 legacy=0x7f2470132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:22.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.987+0000 7f246f7fe700 1 -- 172.21.15.16:0/1621478426 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24700995b0 con 0x7f2470132630 2019-12-05T06:43:22.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.987+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.988+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.988+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=READY pgs=286 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:22.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.988+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=THROTTLE_DONE pgs=286 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.988+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2468000cc0 con 0x7f2470132630 2019-12-05T06:43:22.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.988+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=THROTTLE_DONE pgs=286 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.988+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=THROTTLE_DONE pgs=286 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.988+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2468009af0 con 0x7f2470132630 2019-12-05T06:43:22.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.988+0000 7f247537d700 1 -- 172.21.15.16:0/1621478426 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24700995b0 con 0x7f2470132630 2019-12-05T06:43:22.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.988+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f246800a1c0 con 0x7f2470132630 2019-12-05T06:43:22.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.988+0000 7f247537d700 1 -- 172.21.15.16:0/1621478426 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24700995b0 con 0x7f2470132630 2019-12-05T06:43:22.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.989+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=THROTTLE_DONE pgs=286 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.990+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2468009ce0 con 0x7f2470132630 2019-12-05T06:43:22.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.990+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=THROTTLE_DONE pgs=286 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.990+0000 7f2467fff700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2454023650 0x7f2454025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:22.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.990+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 8009+0+0 (secure 0 0 0) 0x7f2468046120 con 0x7f2470132630 2019-12-05T06:43:22.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.991+0000 7f246effd700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2454023650 0x7f2454025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:22.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:21.991+0000 7f246effd700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2454023650 0x7f2454025ae0 crc :-1 s=READY pgs=514 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:22.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.004+0000 7f247537d700 1 -- 172.21.15.16:0/1621478426 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24700995b0 con 0x7f2470132630 2019-12-05T06:43:22.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.013+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=THROTTLE_DONE pgs=286 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.013+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2468036ee0 con 0x7f2470132630 2019-12-05T06:43:22.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.278+0000 7f247537d700 1 -- 172.21.15.16:0/1621478426 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f24700995b0 con 0x7f2470132630 2019-12-05T06:43:22.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.279+0000 7f246f7fe700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=THROTTLE_DONE pgs=286 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:22.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.279+0000 7f2467fff700 1 -- 172.21.15.16:0/1621478426 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 147 v147) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f2468036ee0 con 0x7f2470132630 2019-12-05T06:43:22.282 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 147 2019-12-05T06:43:22.282 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:22.282 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":147,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7200,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":147}],"filesystems":[{"mdsmap":{"epoch":146,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:41:18.483890+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":72,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7132},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7132":{"gid":7132,"name":"b","rank":0,"incarnation":143,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/2989717660","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2989717660},{"type":"v1","addr":"172.21.15.76:6835","nonce":2989717660}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:22.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.283+0000 7f247537d700 1 -- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2454023650 msgr2=0x7f2454025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:22.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.283+0000 7f247537d700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2454023650 0x7f2454025ae0 crc :-1 s=READY pgs=514 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:22.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.283+0000 7f247537d700 1 -- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 msgr2=0x7f2470132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:22.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.283+0000 7f247537d700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 secure :-1 s=READY pgs=286 cs=0 l=1 rx=0x7f2468010730 tx=0x7f2468010ad0).stop 2019-12-05T06:43:22.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.283+0000 7f247537d700 1 -- 172.21.15.16:0/1621478426 shutdown_connections 2019-12-05T06:43:22.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.283+0000 7f247537d700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2454023650 0x7f2454025ae0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:22.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.283+0000 7f247537d700 1 --2- 172.21.15.16:0/1621478426 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2470132630 0x7f2470132a20 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:22.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.284+0000 7f247537d700 1 -- 172.21.15.16:0/1621478426 shutdown_connections 2019-12-05T06:43:22.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.284+0000 7f247537d700 1 -- 172.21.15.16:0/1621478426 wait complete. 2019-12-05T06:43:22.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:22.284+0000 7f247537d700 1 -- 172.21.15.16:0/1621478426 >> 172.21.15.16:0/1621478426 conn(0x7f24700714f0 msgr2=0x7f247009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:22.300 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:43:24.302 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.480+0000 7f2a80a20700 1 Processor -- start 2019-12-05T06:43:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.480+0000 7f2a80a20700 1 -- start start 2019-12-05T06:43:24.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a80a20700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2a7c08c6d0 0x7f2a7c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a80a20700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a7c08fef0 0x7f2a7c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a80a20700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a80a20700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a7c08df90 con 0x7f2a7c08ef20 2019-12-05T06:43:24.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a80a20700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a7c0a3f70 con 0x7f2a7c08c6d0 2019-12-05T06:43:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a80a20700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a7c08d830 con 0x7f2a7c08fef0 2019-12-05T06:43:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59390/0 (socket says 172.21.15.16:59390) 2019-12-05T06:43:24.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7b59e700 1 -- 172.21.15.16:0/2445973403 learned_addr learned my addr 172.21.15.16:0/2445973403 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:24.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7a59c700 1 --2- 172.21.15.16:0/2445973403 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a7c08fef0 0x7f2a7c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7ad9d700 1 --2- 172.21.15.16:0/2445973403 >> v2:172.21.15.76:3300/0 conn(0x7f2a7c08c6d0 0x7f2a7c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7b59e700 1 -- 172.21.15.16:0/2445973403 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a7c08fef0 msgr2=0x7f2a7c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:24.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/2445973403 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a7c08fef0 0x7f2a7c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7b59e700 1 -- 172.21.15.16:0/2445973403 >> v2:172.21.15.76:3300/0 conn(0x7f2a7c08c6d0 msgr2=0x7f2a7c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/2445973403 >> v2:172.21.15.76:3300/0 conn(0x7f2a7c08c6d0 0x7f2a7c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7b59e700 1 -- 172.21.15.16:0/2445973403 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a7c08d540 con 0x7f2a7c08ef20 2019-12-05T06:43:24.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.482+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/2445973403 >> v2:172.21.15.16:3300/0 conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2a7400f6b0 tx=0x7f2a74007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.483+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/2445973403 >> v2:172.21.15.16:3300/0 conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2a7400f6b0 tx=0x7f2a74007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.483+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/2445973403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rx=0x7f2a7400f6b0 tx=0x7f2a74007d50).ready entity=mon.0 client_cookie=369d2e38b81742cc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.483+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/2445973403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 secure :-1 s=THROTTLE_DONE pgs=1207 cs=0 l=1 rx=0x7f2a7400f6b0 tx=0x7f2a74007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.483+0000 7f2a79d9b700 1 -- 172.21.15.16:0/2445973403 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a74000a70 con 0x7f2a7c08ef20 2019-12-05T06:43:24.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.483+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/2445973403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 secure :-1 s=THROTTLE_DONE pgs=1207 cs=0 l=1 rx=0x7f2a7400f6b0 tx=0x7f2a74007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.483+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/2445973403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 secure :-1 s=THROTTLE_DONE pgs=1207 cs=0 l=1 rx=0x7f2a7400f6b0 tx=0x7f2a74007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.483+0000 7f2a79d9b700 1 -- 172.21.15.16:0/2445973403 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a740010e0 con 0x7f2a7c08ef20 2019-12-05T06:43:24.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.483+0000 7f2a79d9b700 1 -- 172.21.15.16:0/2445973403 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a74001830 con 0x7f2a7c08ef20 2019-12-05T06:43:24.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.483+0000 7f2a80a20700 1 -- 172.21.15.16:0/2445973403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a7c08ef20 msgr2=0x7f2a7c08f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.483+0000 7f2a80a20700 1 --2- 172.21.15.16:0/2445973403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 secure :-1 s=READY pgs=1207 cs=0 l=1 rx=0x7f2a7400f6b0 tx=0x7f2a74007d50).stop 2019-12-05T06:43:24.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.484+0000 7f2a80a20700 1 -- 172.21.15.16:0/2445973403 shutdown_connections 2019-12-05T06:43:24.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.484+0000 7f2a80a20700 1 --2- 172.21.15.16:0/2445973403 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a7c08ef20 0x7f2a7c08f3c0 unknown :-1 s=CLOSED pgs=1207 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.484+0000 7f2a80a20700 1 --2- 172.21.15.16:0/2445973403 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a7c08fef0 0x7f2a7c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.484+0000 7f2a80a20700 1 --2- 172.21.15.16:0/2445973403 >> v2:172.21.15.76:3300/0 conn(0x7f2a7c08c6d0 0x7f2a7c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.484+0000 7f2a80a20700 1 -- 172.21.15.16:0/2445973403 shutdown_connections 2019-12-05T06:43:24.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.484+0000 7f2a80a20700 1 -- 172.21.15.16:0/2445973403 wait complete. 2019-12-05T06:43:24.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.484+0000 7f2a80a20700 1 -- 172.21.15.16:0/2445973403 >> 172.21.15.16:0/2445973403 conn(0x7f2a7c061940 msgr2=0x7f2a7c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:24.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.488+0000 7f2a80a20700 1 Processor -- start 2019-12-05T06:43:24.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.488+0000 7f2a80a20700 1 -- start start 2019-12-05T06:43:24.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a80a20700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2a7c084f30 0x7f2a7c157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a80a20700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7c084b90 con 0x7f2a7c096e90 2019-12-05T06:43:24.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a80a20700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a7c098790 con 0x7f2a7c09bbe0 2019-12-05T06:43:24.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a80a20700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2a7c09a450 con 0x7f2a7c084f30 2019-12-05T06:43:24.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a7ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2a7c096e90 0x7f2a7c09b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44560/0 (socket says 172.21.15.16:44560) 2019-12-05T06:43:24.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a7ad9d700 1 -- 172.21.15.16:0/4268082704 learned_addr learned my addr 172.21.15.16:0/4268082704 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> v2:172.21.15.76:3300/0 conn(0x7f2a7c084f30 0x7f2a7c157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882576260 0 0) 0x7f2a6c000cc0 con 0x7f2a7c096e90 2019-12-05T06:43:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a60002e80 con 0x7f2a7c096e90 2019-12-05T06:43:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1894234923 0 0) 0x7f2a64000f30 con 0x7f2a7c09bbe0 2019-12-05T06:43:24.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a600030b0 con 0x7f2a7c09bbe0 2019-12-05T06:43:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 622908356 0 0) 0x7f2a6c001010 con 0x7f2a7c096e90 2019-12-05T06:43:24.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2a60003850 con 0x7f2a7c096e90 2019-12-05T06:43:24.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.490+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f2a6c001980 con 0x7f2a7c096e90 2019-12-05T06:43:24.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 >> v2:172.21.15.76:3300/0 conn(0x7f2a7c084f30 msgr2=0x7f2a7c157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a737fe700 1 --2- 172.21.15.16:0/4268082704 >> v2:172.21.15.76:3300/0 conn(0x7f2a7c084f30 0x7f2a7c157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 >> v1:172.21.15.76:6789/0 conn(0x7f2a7c09bbe0 legacy=0x7f2a7c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 >> v1:172.21.15.16:6789/0 conn(0x7f2a7c096e90 legacy=0x7f2a7c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a737fe700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a600062d0 0x7f2a600086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a737fe700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a60008c10 0x7f2a6000b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a737fe700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6000de90 con 0x7f2a6000b550 2019-12-05T06:43:24.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6000e050 con 0x7f2a60008c10 2019-12-05T06:43:24.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6000e280 con 0x7f2a600062d0 2019-12-05T06:43:24.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7a59c700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a60008c10 0x7f2a6000b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7ad9d700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a600062d0 0x7f2a600086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7b59e700 1 -- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a600062d0 msgr2=0x7f2a600086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2a600062d0 0x7f2a600086d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7b59e700 1 -- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a60008c10 msgr2=0x7f2a6000b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f2a60008c10 0x7f2a6000b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7b59e700 1 -- 172.21.15.16:0/4268082704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6000e530 con 0x7f2a6000b550 2019-12-05T06:43:24.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a80a20700 1 -- 172.21.15.16:0/4268082704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a7c09aae0 con 0x7f2a6000b550 2019-12-05T06:43:24.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a80a20700 1 -- 172.21.15.16:0/4268082704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a7c084b90 con 0x7f2a6000b550 2019-12-05T06:43:24.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7b59e700 1 -- 172.21.15.16:0/4268082704 reap_dead start 2019-12-05T06:43:24.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.491+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=READY pgs=1209 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:24.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.492+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.492+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a740010a0 con 0x7f2a6000b550 2019-12-05T06:43:24.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.492+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.492+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a74000eb0 con 0x7f2a6000b550 2019-12-05T06:43:24.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.492+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.492+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a740010a0 con 0x7f2a6000b550 2019-12-05T06:43:24.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.492+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.493+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2a740010a0 con 0x7f2a6000b550 2019-12-05T06:43:24.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.493+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.493+0000 7f2a737fe700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a600285f0 0x7f2a6002aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.493+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f2a74089e60 con 0x7f2a6000b550 2019-12-05T06:43:24.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.493+0000 7f2a7ad9d700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a600285f0 0x7f2a6002aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.494+0000 7f2a7ad9d700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a600285f0 0x7f2a6002aa20 crc :-1 s=READY pgs=515 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:24.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.495+0000 7f2a80a20700 1 -- 172.21.15.16:0/4268082704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a7c09aae0 con 0x7f2a6000b550 2019-12-05T06:43:24.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.501+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.501+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2a74000c40 con 0x7f2a6000b550 2019-12-05T06:43:24.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.697+0000 7f2a80a20700 1 -- 172.21.15.16:0/4268082704 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a7c09aae0 con 0x7f2a6000b550 2019-12-05T06:43:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.698+0000 7f2a7b59e700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=THROTTLE_DONE pgs=1209 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.698+0000 7f2a737fe700 1 -- 172.21.15.16:0/4268082704 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f2a74000eb0 con 0x7f2a6000b550 2019-12-05T06:43:24.700 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 148 2019-12-05T06:43:24.700 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:24.700 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:23.429092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.701+0000 7f2a80a20700 1 -- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a600285f0 msgr2=0x7f2a6002aa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.701+0000 7f2a80a20700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a600285f0 0x7f2a6002aa20 crc :-1 s=READY pgs=515 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.701+0000 7f2a80a20700 1 -- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 msgr2=0x7f2a6000d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.701+0000 7f2a80a20700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 secure :-1 s=READY pgs=1209 cs=0 l=1 rx=0x7f2a740545f0 tx=0x7f2a740544f0).stop 2019-12-05T06:43:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.701+0000 7f2a80a20700 1 -- 172.21.15.16:0/4268082704 shutdown_connections 2019-12-05T06:43:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.701+0000 7f2a80a20700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2a600285f0 0x7f2a6002aa20 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.701+0000 7f2a80a20700 1 --2- 172.21.15.16:0/4268082704 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2a6000b550 0x7f2a6000d950 unknown :-1 s=CLOSED pgs=1209 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.701+0000 7f2a80a20700 1 -- 172.21.15.16:0/4268082704 shutdown_connections 2019-12-05T06:43:24.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.701+0000 7f2a80a20700 1 -- 172.21.15.16:0/4268082704 wait complete. 2019-12-05T06:43:24.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.701+0000 7f2a80a20700 1 -- 172.21.15.16:0/4268082704 >> 172.21.15.16:0/4268082704 conn(0x7f2a7c0714f0 msgr2=0x7f2a7c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:24.712 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:43:24.712 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:43:24.712 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:24.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.844+0000 7fee104a4700 1 Processor -- start 2019-12-05T06:43:24.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.844+0000 7fee104a4700 1 -- start start 2019-12-05T06:43:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.845+0000 7fee104a4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fee0808d2c0 0x7fee0808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.845+0000 7fee104a4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fee0808fef0 0x7fee0808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.845+0000 7fee104a4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee080958e0 con 0x7fee0808c6d0 2019-12-05T06:43:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.845+0000 7fee104a4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee0808df90 con 0x7fee0808d2c0 2019-12-05T06:43:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.845+0000 7fee104a4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee080a3f70 con 0x7fee0808fef0 2019-12-05T06:43:24.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.845+0000 7fee0ea41700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fee0808c6d0 0x7fee0808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44574/0 (socket says 172.21.15.16:44574) 2019-12-05T06:43:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.845+0000 7fee0ea41700 1 -- 172.21.15.16:0/3056759655 learned_addr learned my addr 172.21.15.16:0/3056759655 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.845+0000 7fee0e240700 1 --2- 172.21.15.16:0/3056759655 >> v2:172.21.15.76:3300/0 conn(0x7fee0808d2c0 0x7fee0808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.845+0000 7fee0f242700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fee0808fef0 0x7fee0808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335370234 0 0) 0x7fee000014f0 con 0x7fee0808c6d0 2019-12-05T06:43:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf4002e80 con 0x7fee0808c6d0 2019-12-05T06:43:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1721611220 0 0) 0x7fee000014b0 con 0x7fee0808c6d0 2019-12-05T06:43:24.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fee0808fef0 msgr2=0x7fee0808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fee0808fef0 0x7fee0808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 >> v2:172.21.15.76:3300/0 conn(0x7fee0808d2c0 msgr2=0x7fee0808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 --2- 172.21.15.16:0/3056759655 >> v2:172.21.15.76:3300/0 conn(0x7fee0808d2c0 0x7fee0808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee0808e620 con 0x7fee0808c6d0 2019-12-05T06:43:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fee000019e0 con 0x7fee0808c6d0 2019-12-05T06:43:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 >> v1:172.21.15.16:6789/0 conn(0x7fee0808c6d0 legacy=0x7fee0808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedf40479e0 0x7fedf4049e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf404a5d0 0x7fedf404caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fedf404d300 0x7fedf404f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf404ff50 con 0x7fedf404a5d0 2019-12-05T06:43:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf4050140 con 0x7fedf40479e0 2019-12-05T06:43:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf4050370 con 0x7fedf404d300 2019-12-05T06:43:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.846+0000 7fee0e240700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf404a5d0 0x7fedf404caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0ea41700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedf40479e0 0x7fedf4049e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0f242700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fedf404d300 0x7fedf404f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 -- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fedf404d300 msgr2=0x7fedf404f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:24.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fedf404d300 0x7fedf404f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 -- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedf40479e0 msgr2=0x7fedf4049e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedf40479e0 0x7fedf4049e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 -- 172.21.15.16:0/3056759655 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf4050620 con 0x7fedf404a5d0 2019-12-05T06:43:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf404a5d0 0x7fedf404caa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fee0400b660 tx=0x7fee0400b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf404a5d0 0x7fedf404caa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fee0400b660 tx=0x7fee0400b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf404a5d0 0x7fedf404caa0 secure :-1 s=READY pgs=1211 cs=0 l=1 rx=0x7fee0400b660 tx=0x7fee0400b530).ready entity=mon.0 client_cookie=ff286223ebe8a245 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf404a5d0 0x7fedf404caa0 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7fee0400b660 tx=0x7fee0400b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf404a5d0 0x7fedf404caa0 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7fee0400b660 tx=0x7fee0400b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee04000960 con 0x7fedf404a5d0 2019-12-05T06:43:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0e240700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf404a5d0 0x7fedf404caa0 secure :-1 s=THROTTLE_DONE pgs=1211 cs=0 l=1 rx=0x7fee0400b660 tx=0x7fee0400b530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee04009980 con 0x7fedf404a5d0 2019-12-05T06:43:24.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0da3f700 1 -- 172.21.15.16:0/3056759655 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee0400a0d0 con 0x7fedf404a5d0 2019-12-05T06:43:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee104a4700 1 -- 172.21.15.16:0/3056759655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf404a5d0 msgr2=0x7fedf404caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee104a4700 1 --2- 172.21.15.16:0/3056759655 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf404a5d0 0x7fedf404caa0 secure :-1 s=READY pgs=1211 cs=0 l=1 rx=0x7fee0400b660 tx=0x7fee0400b530).stop 2019-12-05T06:43:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.847+0000 7fee0f242700 1 -- 172.21.15.16:0/3056759655 reap_dead start 2019-12-05T06:43:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.848+0000 7fee104a4700 1 -- 172.21.15.16:0/3056759655 shutdown_connections 2019-12-05T06:43:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.848+0000 7fee104a4700 1 -- 172.21.15.16:0/3056759655 shutdown_connections 2019-12-05T06:43:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.848+0000 7fee104a4700 1 -- 172.21.15.16:0/3056759655 wait complete. 2019-12-05T06:43:24.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.848+0000 7fee104a4700 1 -- 172.21.15.16:0/3056759655 >> 172.21.15.16:0/3056759655 conn(0x7fee08061940 msgr2=0x7fee08096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:24.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.852+0000 7fee104a4700 1 Processor -- start 2019-12-05T06:43:24.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.852+0000 7fee104a4700 1 -- start start 2019-12-05T06:43:24.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.853+0000 7fee104a4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fee080a0910 0x7fee0814d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.853+0000 7fee104a4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee08070770 con 0x7fee080a0140 2019-12-05T06:43:24.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.853+0000 7fee104a4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee0809d1a0 con 0x7fee0814d410 2019-12-05T06:43:24.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.853+0000 7fee104a4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee08070400 con 0x7fee080a0910 2019-12-05T06:43:24.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.853+0000 7fee0f242700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fee080a0140 0x7fee08157100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44590/0 (socket says 172.21.15.16:44590) 2019-12-05T06:43:24.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.853+0000 7fee0f242700 1 -- 172.21.15.16:0/3125494262 learned_addr learned my addr 172.21.15.16:0/3125494262 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:24.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fee0e240700 1 --2- 172.21.15.16:0/3125494262 >> v2:172.21.15.76:3300/0 conn(0x7fee080a0910 0x7fee0814d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457369133 0 0) 0x7fedfc001290 con 0x7fee080a0140 2019-12-05T06:43:24.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fedf0002e80 con 0x7fee080a0140 2019-12-05T06:43:24.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1313970587 0 0) 0x7fedfc000aa0 con 0x7fee080a0140 2019-12-05T06:43:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fedf00032e0 con 0x7fee080a0140 2019-12-05T06:43:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fedfc0016e0 con 0x7fee080a0140 2019-12-05T06:43:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 >> v2:172.21.15.76:3300/0 conn(0x7fee080a0910 msgr2=0x7fee0814d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 --2- 172.21.15.16:0/3125494262 >> v2:172.21.15.76:3300/0 conn(0x7fee080a0910 0x7fee0814d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 >> v1:172.21.15.76:6789/0 conn(0x7fee0814d410 legacy=0x7fee0809f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 >> v1:172.21.15.16:6789/0 conn(0x7fee080a0140 legacy=0x7fee08157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedf00036d0 0x7fedf0003ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fedf0004020 0x7fedf00044e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf000bdd0 con 0x7fedf0004ca0 2019-12-05T06:43:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedf000bf90 con 0x7fedf00036d0 2019-12-05T06:43:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedf000c150 con 0x7fedf0004020 2019-12-05T06:43:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fee0e240700 1 --2- 172.21.15.16:0/3125494262 >> v2:172.21.15.76:3300/0 conn(0x7fee080a0910 0x7fee0814d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.854+0000 7fee0ea41700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedf00036d0 0x7fedf0003ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0e240700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fedf0004020 0x7fedf00044e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 -- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fedf0004020 msgr2=0x7fedf00044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fedf0004020 0x7fedf00044e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 -- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedf00036d0 msgr2=0x7fedf0003ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedf00036d0 0x7fedf0003ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 -- 172.21.15.16:0/3125494262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedf000c390 con 0x7fedf0004ca0 2019-12-05T06:43:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0ea41700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fedf00036d0 0x7fedf0003ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:43:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 -- 172.21.15.16:0/3125494262 reap_dead start 2019-12-05T06:43:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee104a4700 1 -- 172.21.15.16:0/3125494262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee0809f040 con 0x7fedf0004ca0 2019-12-05T06:43:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=READY pgs=1213 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee104a4700 1 -- 172.21.15.16:0/3125494262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee0809f040 con 0x7fedf0004ca0 2019-12-05T06:43:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedfc000e70 con 0x7fedf0004ca0 2019-12-05T06:43:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedfc001aa0 con 0x7fedf0004ca0 2019-12-05T06:43:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.855+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedfc002170 con 0x7fedf0004ca0 2019-12-05T06:43:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.856+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.856+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fedfc001c90 con 0x7fedf0004ca0 2019-12-05T06:43:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.856+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.856+0000 7fedfb7fe700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedf00263b0 0x7fedf00287e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.856+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fedfc045f90 con 0x7fedf0004ca0 2019-12-05T06:43:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.857+0000 7fee0ea41700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedf00263b0 0x7fedf00287e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.857+0000 7fee0ea41700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedf00263b0 0x7fedf00287e0 crc :-1 s=READY pgs=516 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.860+0000 7fee104a4700 1 -- 172.21.15.16:0/3125494262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee0809f040 con 0x7fedf0004ca0 2019-12-05T06:43:24.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.863+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:24.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:24.863+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fedfc0016a0 con 0x7fedf0004ca0 2019-12-05T06:43:25.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.050+0000 7fee104a4700 1 -- 172.21.15.16:0/3125494262 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee0809f040 con 0x7fedf0004ca0 2019-12-05T06:43:25.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.050+0000 7fee0f242700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=THROTTLE_DONE pgs=1213 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.050+0000 7fedfb7fe700 1 -- 172.21.15.16:0/3125494262 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 148 v148) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fedfc001aa0 con 0x7fedf0004ca0 2019-12-05T06:43:25.052 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 148 2019-12-05T06:43:25.052 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:25.052 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":148,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":148,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:23.429092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.053+0000 7fee104a4700 1 -- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedf00263b0 msgr2=0x7fedf00287e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:25.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.053+0000 7fee104a4700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedf00263b0 0x7fedf00287e0 crc :-1 s=READY pgs=516 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.053+0000 7fee104a4700 1 -- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 msgr2=0x7fedf000b890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.053+0000 7fee104a4700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 secure :-1 s=READY pgs=1213 cs=0 l=1 rx=0x7fedfc010400 tx=0x7fedfc0107a0).stop 2019-12-05T06:43:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.053+0000 7fee104a4700 1 -- 172.21.15.16:0/3125494262 shutdown_connections 2019-12-05T06:43:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.053+0000 7fee104a4700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fedf00263b0 0x7fedf00287e0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.053+0000 7fee104a4700 1 --2- 172.21.15.16:0/3125494262 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fedf0004ca0 0x7fedf000b890 unknown :-1 s=CLOSED pgs=1213 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.053+0000 7fee104a4700 1 -- 172.21.15.16:0/3125494262 shutdown_connections 2019-12-05T06:43:25.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.053+0000 7fee104a4700 1 -- 172.21.15.16:0/3125494262 wait complete. 2019-12-05T06:43:25.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:25.053+0000 7fee104a4700 1 -- 172.21.15.16:0/3125494262 >> 172.21.15.16:0/3125494262 conn(0x7fee0808acd0 msgr2=0x7fee08099650 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:25.064 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 148, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 148, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7200": { "addr": "172.21.15.16:6833/1298592917", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 1298592917, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 1298592917, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7200, "incarnation": 148, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 73, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:43:23.429092+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7200 } } } ], "standbys": [] } 2019-12-05T06:43:27.065 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:27.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.326+0000 7fa347aee700 1 Processor -- start 2019-12-05T06:43:27.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.326+0000 7fa347aee700 1 -- start start 2019-12-05T06:43:27.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.327+0000 7fa347aee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa34008d2c0 0x7fa34008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:27.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.328+0000 7fa347aee700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa34008fef0 0x7fa34008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:27.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.328+0000 7fa347aee700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3400958e0 con 0x7fa34008c6d0 2019-12-05T06:43:27.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.328+0000 7fa347aee700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34008df90 con 0x7fa34008d2c0 2019-12-05T06:43:27.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.328+0000 7fa347aee700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3400a3f70 con 0x7fa34008fef0 2019-12-05T06:43:27.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.328+0000 7fa34588a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa34008d2c0 0x7fa34008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.328+0000 7fa34588a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa34008d2c0 0x7fa34008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59436/0 (socket says 172.21.15.16:59436) 2019-12-05T06:43:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.328+0000 7fa34588a700 1 -- 172.21.15.16:0/2515186238 learned_addr learned my addr 172.21.15.16:0/2515186238 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:27.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.328+0000 7fa34688c700 1 --2- 172.21.15.16:0/2515186238 >> v2:172.21.15.76:3300/0 conn(0x7fa34008fef0 0x7fa34008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.329+0000 7fa34588a700 1 -- 172.21.15.16:0/2515186238 >> v2:172.21.15.76:3300/0 conn(0x7fa34008fef0 msgr2=0x7fa34008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.329+0000 7fa34588a700 1 --2- 172.21.15.16:0/2515186238 >> v2:172.21.15.76:3300/0 conn(0x7fa34008fef0 0x7fa34008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:27.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.329+0000 7fa34588a700 1 -- 172.21.15.16:0/2515186238 >> v1:172.21.15.76:6789/0 conn(0x7fa34008c6d0 legacy=0x7fa34008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.329+0000 7fa34588a700 1 -- 172.21.15.16:0/2515186238 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34008e620 con 0x7fa34008d2c0 2019-12-05T06:43:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.329+0000 7fa34588a700 1 --2- 172.21.15.16:0/2515186238 >> v2:172.21.15.16:3300/0 conn(0x7fa34008d2c0 0x7fa34008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa33400f740 tx=0x7fa33400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.329+0000 7fa34588a700 1 --2- 172.21.15.16:0/2515186238 >> v2:172.21.15.16:3300/0 conn(0x7fa34008d2c0 0x7fa34008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa33400f740 tx=0x7fa33400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.329+0000 7fa34588a700 1 --2- 172.21.15.16:0/2515186238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34008d2c0 0x7fa34008ef20 secure :-1 s=READY pgs=1214 cs=0 l=1 rx=0x7fa33400f740 tx=0x7fa33400f570).ready entity=mon.0 client_cookie=ce6ba9fe431f81a5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:27.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.329+0000 7fa34588a700 1 --2- 172.21.15.16:0/2515186238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34008d2c0 0x7fa34008ef20 secure :-1 s=THROTTLE_DONE pgs=1214 cs=0 l=1 rx=0x7fa33400f740 tx=0x7fa33400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.330+0000 7fa34588a700 1 --2- 172.21.15.16:0/2515186238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34008d2c0 0x7fa34008ef20 secure :-1 s=THROTTLE_DONE pgs=1214 cs=0 l=1 rx=0x7fa33400f740 tx=0x7fa33400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.330+0000 7fa345089700 1 -- 172.21.15.16:0/2515186238 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa334000960 con 0x7fa34008d2c0 2019-12-05T06:43:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.330+0000 7fa34588a700 1 --2- 172.21.15.16:0/2515186238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34008d2c0 0x7fa34008ef20 secure :-1 s=THROTTLE_DONE pgs=1214 cs=0 l=1 rx=0x7fa33400f740 tx=0x7fa33400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.330+0000 7fa345089700 1 -- 172.21.15.16:0/2515186238 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa334000c90 con 0x7fa34008d2c0 2019-12-05T06:43:27.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.330+0000 7fa345089700 1 -- 172.21.15.16:0/2515186238 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3340013a0 con 0x7fa34008d2c0 2019-12-05T06:43:27.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.330+0000 7fa347aee700 1 -- 172.21.15.16:0/2515186238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34008d2c0 msgr2=0x7fa34008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.330+0000 7fa347aee700 1 --2- 172.21.15.16:0/2515186238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34008d2c0 0x7fa34008ef20 secure :-1 s=READY pgs=1214 cs=0 l=1 rx=0x7fa33400f740 tx=0x7fa33400f570).stop 2019-12-05T06:43:27.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.332+0000 7fa347aee700 1 -- 172.21.15.16:0/2515186238 shutdown_connections 2019-12-05T06:43:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.332+0000 7fa347aee700 1 --2- 172.21.15.16:0/2515186238 >> v2:172.21.15.76:3300/0 conn(0x7fa34008fef0 0x7fa34008f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.332+0000 7fa347aee700 1 --2- 172.21.15.16:0/2515186238 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa34008d2c0 0x7fa34008ef20 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.332+0000 7fa347aee700 1 -- 172.21.15.16:0/2515186238 shutdown_connections 2019-12-05T06:43:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.332+0000 7fa347aee700 1 -- 172.21.15.16:0/2515186238 wait complete. 2019-12-05T06:43:27.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.332+0000 7fa347aee700 1 -- 172.21.15.16:0/2515186238 >> 172.21.15.16:0/2515186238 conn(0x7fa340061940 msgr2=0x7fa340096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:27.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.337+0000 7fa347aee700 1 Processor -- start 2019-12-05T06:43:27.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.337+0000 7fa347aee700 1 -- start start 2019-12-05T06:43:27.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.338+0000 7fa347aee700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa3401309e0 0x7fa340130210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:27.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.338+0000 7fa347aee700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa340081220 0x7fa3401571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:27.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.338+0000 7fa347aee700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa340088c30 con 0x7fa34014da50 2019-12-05T06:43:27.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.338+0000 7fa347aee700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa34008a8f0 con 0x7fa340081220 2019-12-05T06:43:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.338+0000 7fa347aee700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa340089ec0 con 0x7fa3401309e0 2019-12-05T06:43:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.338+0000 7fa34608b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa34014da50 0x7fa34012fa10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:37678/0 (socket says 172.21.15.16:37678) 2019-12-05T06:43:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.338+0000 7fa34608b700 1 -- 172.21.15.16:0/977676884 learned_addr learned my addr 172.21.15.16:0/977676884 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.338+0000 7fa34588a700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa3401309e0 0x7fa340130210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.339+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> v2:172.21.15.76:3300/0 conn(0x7fa340081220 0x7fa3401571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.339+0000 7fa33effd700 1 -- 172.21.15.16:0/977676884 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970196212 0 0) 0x7fa338000fe0 con 0x7fa34014da50 2019-12-05T06:43:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.339+0000 7fa33effd700 1 -- 172.21.15.16:0/977676884 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa320002e80 con 0x7fa34014da50 2019-12-05T06:43:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.339+0000 7fa34688c700 1 -- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa3401309e0 msgr2=0x7fa340130210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.339+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa3401309e0 0x7fa340130210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.339+0000 7fa34688c700 1 -- 172.21.15.16:0/977676884 >> v1:172.21.15.76:6789/0 conn(0x7fa34014da50 legacy=0x7fa34012fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.339+0000 7fa34688c700 1 -- 172.21.15.16:0/977676884 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34012f110 con 0x7fa340081220 2019-12-05T06:43:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.339+0000 7fa34588a700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa3401309e0 0x7fa340130210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.339+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> v2:172.21.15.76:3300/0 conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.340+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> v2:172.21.15.76:3300/0 conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.340+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=READY pgs=847 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.340+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=THROTTLE_DONE pgs=847 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.340+0000 7fa33effd700 1 -- 172.21.15.16:0/977676884 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa330009990 con 0x7fa340081220 2019-12-05T06:43:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.340+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=THROTTLE_DONE pgs=847 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.340+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=THROTTLE_DONE pgs=847 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.340+0000 7fa33effd700 1 -- 172.21.15.16:0/977676884 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3300095b0 con 0x7fa340081220 2019-12-05T06:43:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.340+0000 7fa33effd700 1 -- 172.21.15.16:0/977676884 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa330009f70 con 0x7fa340081220 2019-12-05T06:43:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.340+0000 7fa347aee700 1 -- 172.21.15.16:0/977676884 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa34012f110 con 0x7fa340081220 2019-12-05T06:43:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.340+0000 7fa347aee700 1 -- 172.21.15.16:0/977676884 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa34012f110 con 0x7fa340081220 2019-12-05T06:43:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.341+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=THROTTLE_DONE pgs=847 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.342+0000 7fa33effd700 1 -- 172.21.15.16:0/977676884 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa33000ea80 con 0x7fa340081220 2019-12-05T06:43:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.342+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=THROTTLE_DONE pgs=847 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.343+0000 7fa33effd700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa320023670 0x7fa320025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.343+0000 7fa33effd700 1 -- 172.21.15.16:0/977676884 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fa330045d10 con 0x7fa340081220 2019-12-05T06:43:27.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.343+0000 7fa34608b700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa320023670 0x7fa320025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:27.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.343+0000 7fa34608b700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa320023670 0x7fa320025ae0 crc :-1 s=READY pgs=517 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:27.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.358+0000 7fa347aee700 1 -- 172.21.15.16:0/977676884 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa34012f110 con 0x7fa340081220 2019-12-05T06:43:27.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.368+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=THROTTLE_DONE pgs=847 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.368+0000 7fa33effd700 1 -- 172.21.15.16:0/977676884 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa3300097a0 con 0x7fa340081220 2019-12-05T06:43:27.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.625+0000 7fa347aee700 1 -- 172.21.15.16:0/977676884 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa34012f110 con 0x7fa340081220 2019-12-05T06:43:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.626+0000 7fa34688c700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=THROTTLE_DONE pgs=847 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:27.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.626+0000 7fa33effd700 1 -- 172.21.15.16:0/977676884 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 150 v150) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fa33000ea80 con 0x7fa340081220 2019-12-05T06:43:27.629 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:27.629 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":150,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":150,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:27.448608+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:rejoin","state_seq":32,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 150 2019-12-05T06:43:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.630+0000 7fa347aee700 1 -- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa320023670 msgr2=0x7fa320025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.630+0000 7fa347aee700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa320023670 0x7fa320025ae0 crc :-1 s=READY pgs=517 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.630+0000 7fa347aee700 1 -- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 msgr2=0x7fa3401571d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.630+0000 7fa347aee700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 0x7fa3401571d0 secure :-1 s=READY pgs=847 cs=0 l=1 rx=0x7fa330010390 tx=0x7fa330010730).stop 2019-12-05T06:43:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.630+0000 7fa347aee700 1 -- 172.21.15.16:0/977676884 shutdown_connections 2019-12-05T06:43:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.631+0000 7fa347aee700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa320023670 0x7fa320025ae0 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.631+0000 7fa347aee700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa340081220 0x7fa3401571d0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.631+0000 7fa347aee700 1 --2- 172.21.15.16:0/977676884 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa3401309e0 0x7fa340130210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.631+0000 7fa347aee700 1 -- 172.21.15.16:0/977676884 shutdown_connections 2019-12-05T06:43:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.631+0000 7fa347aee700 1 -- 172.21.15.16:0/977676884 wait complete. 2019-12-05T06:43:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:27.631+0000 7fa347aee700 1 -- 172.21.15.16:0/977676884 >> 172.21.15.16:0/977676884 conn(0x7fa34009de30 msgr2=0x7fa340070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:27.641 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:43:29.648 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:29.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.791+0000 7f8d004fd700 1 Processor -- start 2019-12-05T06:43:29.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.792+0000 7f8d004fd700 1 -- start start 2019-12-05T06:43:29.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.793+0000 7f8d004fd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8cf808fef0 0x7f8cf808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:29.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.793+0000 7f8d004fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf80958e0 con 0x7f8cf808d2c0 2019-12-05T06:43:29.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.793+0000 7f8d004fd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf80629e0 con 0x7f8cf808c6d0 2019-12-05T06:43:29.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.793+0000 7f8d004fd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf808df90 con 0x7f8cf808fef0 2019-12-05T06:43:29.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.793+0000 7f8cfe299700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8cf808d2c0 0x7f8cf808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44630/0 (socket says 172.21.15.16:44630) 2019-12-05T06:43:29.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.793+0000 7f8cfe299700 1 -- 172.21.15.16:0/130204774 learned_addr learned my addr 172.21.15.16:0/130204774 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:29.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.793+0000 7f8cff29b700 1 --2- 172.21.15.16:0/130204774 >> v2:172.21.15.76:3300/0 conn(0x7f8cf808fef0 0x7f8cf808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.793+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755212496 0 0) 0x7f8ce8001150 con 0x7f8cf808d2c0 2019-12-05T06:43:29.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.793+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ce4002e80 con 0x7f8cf808d2c0 2019-12-05T06:43:29.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 323300307 0 0) 0x7f8ce8001130 con 0x7f8cf808d2c0 2019-12-05T06:43:29.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 >> v2:172.21.15.76:3300/0 conn(0x7f8cf808fef0 msgr2=0x7f8cf808f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:29.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 --2- 172.21.15.16:0/130204774 >> v2:172.21.15.76:3300/0 conn(0x7f8cf808fef0 0x7f8cf808f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:29.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 >> v1:172.21.15.76:6789/0 conn(0x7f8cf808c6d0 legacy=0x7f8cf808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf808e620 con 0x7f8cf808d2c0 2019-12-05T06:43:29.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f8ce80016e0 con 0x7f8cf808d2c0 2019-12-05T06:43:29.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 >> v1:172.21.15.16:6789/0 conn(0x7f8cf808d2c0 legacy=0x7f8cf808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce4003a80 0x7f8ce4005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:29.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 0x7f8ce4008b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:29.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ce40093a0 0x7f8ce400b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:29.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce400c020 con 0x7f8ce4006690 2019-12-05T06:43:29.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce4050220 con 0x7f8ce40093a0 2019-12-05T06:43:29.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce4050450 con 0x7f8ce4003a80 2019-12-05T06:43:29.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 0x7f8ce4008b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce4003a80 0x7f8ce4005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cff29b700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ce40093a0 0x7f8ce400b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 -- 172.21.15.16:0/130204774 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce4003a80 msgr2=0x7f8ce4005ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ce4003a80 0x7f8ce4005ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 -- 172.21.15.16:0/130204774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ce40093a0 msgr2=0x7f8ce400b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:29.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8ce40093a0 0x7f8ce400b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 -- 172.21.15.16:0/130204774 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce4050700 con 0x7f8ce4006690 2019-12-05T06:43:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cff29b700 1 -- 172.21.15.16:0/130204774 reap_dead start 2019-12-05T06:43:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 0x7f8ce4008b40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8ce800f890 tx=0x7f8ce8001dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 0x7f8ce4008b40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8ce800f890 tx=0x7f8ce8001dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 0x7f8ce4008b40 secure :-1 s=READY pgs=1216 cs=0 l=1 rx=0x7f8ce800f890 tx=0x7f8ce8001dd0).ready entity=mon.0 client_cookie=9c957dfb42b0895c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 0x7f8ce4008b40 secure :-1 s=THROTTLE_DONE pgs=1216 cs=0 l=1 rx=0x7f8ce800f890 tx=0x7f8ce8001dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ce8004640 con 0x7f8ce4006690 2019-12-05T06:43:29.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.794+0000 7f8cfe299700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 0x7f8ce4008b40 secure :-1 s=THROTTLE_DONE pgs=1216 cs=0 l=1 rx=0x7f8ce800f890 tx=0x7f8ce8001dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.795+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ce8001130 con 0x7f8ce4006690 2019-12-05T06:43:29.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.795+0000 7f8cfe299700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 0x7f8ce4008b40 secure :-1 s=THROTTLE_DONE pgs=1216 cs=0 l=1 rx=0x7f8ce800f890 tx=0x7f8ce8001dd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.795+0000 7f8cfda98700 1 -- 172.21.15.16:0/130204774 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ce8004830 con 0x7f8ce4006690 2019-12-05T06:43:29.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.795+0000 7f8d004fd700 1 -- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 msgr2=0x7f8ce4008b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.795+0000 7f8d004fd700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 0x7f8ce4008b40 secure :-1 s=READY pgs=1216 cs=0 l=1 rx=0x7f8ce800f890 tx=0x7f8ce8001dd0).stop 2019-12-05T06:43:29.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.795+0000 7f8d004fd700 1 -- 172.21.15.16:0/130204774 shutdown_connections 2019-12-05T06:43:29.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.795+0000 7f8d004fd700 1 --2- 172.21.15.16:0/130204774 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ce4006690 0x7f8ce4008b40 unknown :-1 s=CLOSED pgs=1216 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:29.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.796+0000 7f8d004fd700 1 -- 172.21.15.16:0/130204774 shutdown_connections 2019-12-05T06:43:29.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.796+0000 7f8d004fd700 1 -- 172.21.15.16:0/130204774 wait complete. 2019-12-05T06:43:29.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.796+0000 7f8d004fd700 1 -- 172.21.15.16:0/130204774 >> 172.21.15.16:0/130204774 conn(0x7f8cf8061940 msgr2=0x7f8cf8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:29.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.798+0000 7f8d004fd700 1 Processor -- start 2019-12-05T06:43:29.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.798+0000 7f8d004fd700 1 -- start start 2019-12-05T06:43:29.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8d004fd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cf81302f0 0x7f8cf812a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:29.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8d004fd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8cf812b640 0x7f8cf812abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:29.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8d004fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cf8131bf0 con 0x7f8cf812b0f0 2019-12-05T06:43:29.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8d004fd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cf81296e0 con 0x7f8cf81302f0 2019-12-05T06:43:29.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8d004fd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cf8128cb0 con 0x7f8cf812b640 2019-12-05T06:43:29.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cfea9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cf81302f0 0x7f8cf812a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cfea9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8cf81302f0 0x7f8cf812a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59472/0 (socket says 172.21.15.16:59472) 2019-12-05T06:43:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cff29b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8cf812b0f0 0x7f8cf81306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44644/0 (socket says 172.21.15.16:44644) 2019-12-05T06:43:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cfea9a700 1 -- 172.21.15.16:0/830501297 learned_addr learned my addr 172.21.15.16:0/830501297 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:29.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cfea9a700 1 -- 172.21.15.16:0/830501297 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8cf812b640 msgr2=0x7f8cf812abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8cf812b640 0x7f8cf812abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:29.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cfea9a700 1 -- 172.21.15.16:0/830501297 >> v1:172.21.15.16:6789/0 conn(0x7f8cf812b0f0 legacy=0x7f8cf81306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:29.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cfea9a700 1 -- 172.21.15.16:0/830501297 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf8129d70 con 0x7f8cf81302f0 2019-12-05T06:43:29.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cef7fe700 1 -- 172.21.15.16:0/830501297 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103717752 0 0) 0x7f8cf0002470 con 0x7f8cf812b0f0 2019-12-05T06:43:29.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> v2:172.21.15.16:3300/0 conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> v2:172.21.15.16:3300/0 conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.799+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=READY pgs=1218 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:29.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.800+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.800+0000 7f8cef7fe700 1 -- 172.21.15.16:0/830501297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cf4009e40 con 0x7f8cf81302f0 2019-12-05T06:43:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.800+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.800+0000 7f8cef7fe700 1 -- 172.21.15.16:0/830501297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cf4009a60 con 0x7f8cf81302f0 2019-12-05T06:43:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.800+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.800+0000 7f8d004fd700 1 -- 172.21.15.16:0/830501297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cf8129d70 con 0x7f8cf81302f0 2019-12-05T06:43:29.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.800+0000 7f8cef7fe700 1 -- 172.21.15.16:0/830501297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cf400a0d0 con 0x7f8cf81302f0 2019-12-05T06:43:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.800+0000 7f8d004fd700 1 -- 172.21.15.16:0/830501297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cf8129d70 con 0x7f8cf81302f0 2019-12-05T06:43:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.800+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.801+0000 7f8cef7fe700 1 -- 172.21.15.16:0/830501297 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8cf400a0f0 con 0x7f8cf81302f0 2019-12-05T06:43:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.801+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.801+0000 7f8cef7fe700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8ce0023630 0x7f8ce0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:29.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.801+0000 7f8cef7fe700 1 -- 172.21.15.16:0/830501297 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f8cf4045e10 con 0x7f8cf81302f0 2019-12-05T06:43:29.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.801+0000 7f8cfe299700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8ce0023630 0x7f8ce0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:29.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.802+0000 7f8cfe299700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8ce0023630 0x7f8ce0025ac0 crc :-1 s=READY pgs=518 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:29.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.807+0000 7f8d004fd700 1 -- 172.21.15.16:0/830501297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cf8129d70 con 0x7f8cf81302f0 2019-12-05T06:43:29.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.810+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:29.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:29.811+0000 7f8cef7fe700 1 -- 172.21.15.16:0/830501297 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8cf4009ee0 con 0x7f8cf81302f0 2019-12-05T06:43:30.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.007+0000 7f8d004fd700 1 -- 172.21.15.16:0/830501297 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8cf8129d70 con 0x7f8cf81302f0 2019-12-05T06:43:30.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.007+0000 7f8cfea9a700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=THROTTLE_DONE pgs=1218 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:30.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.008+0000 7f8cef7fe700 1 -- 172.21.15.16:0/830501297 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 151 v151) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f8cf403aeb0 con 0x7f8cf81302f0 2019-12-05T06:43:30.009 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 151 2019-12-05T06:43:30.010 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:30.010 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":151,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:28.456001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":33,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:30.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.010+0000 7f8d004fd700 1 -- 172.21.15.16:0/830501297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8ce0023630 msgr2=0x7f8ce0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:30.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.010+0000 7f8d004fd700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8ce0023630 0x7f8ce0025ac0 crc :-1 s=READY pgs=518 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:30.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.010+0000 7f8d004fd700 1 -- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 msgr2=0x7f8cf812a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:30.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.011+0000 7f8d004fd700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 0x7f8cf812a670 secure :-1 s=READY pgs=1218 cs=0 l=1 rx=0x7f8cf40103f0 tx=0x7f8cf4010790).stop 2019-12-05T06:43:30.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.011+0000 7f8d004fd700 1 -- 172.21.15.16:0/830501297 shutdown_connections 2019-12-05T06:43:30.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.011+0000 7f8d004fd700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8cf812b640 0x7f8cf812abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:30.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.011+0000 7f8d004fd700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8ce0023630 0x7f8ce0025ac0 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:30.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.011+0000 7f8d004fd700 1 --2- 172.21.15.16:0/830501297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8cf81302f0 0x7f8cf812a670 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:30.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.011+0000 7f8d004fd700 1 -- 172.21.15.16:0/830501297 shutdown_connections 2019-12-05T06:43:30.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.011+0000 7f8d004fd700 1 -- 172.21.15.16:0/830501297 wait complete. 2019-12-05T06:43:30.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:30.011+0000 7f8d004fd700 1 -- 172.21.15.16:0/830501297 >> 172.21.15.16:0/830501297 conn(0x7f8cf80a2850 msgr2=0x7f8cf806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:30.023 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7132 2019-12-05T06:43:30.023 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.b 2019-12-05T06:43:33.024 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:43:33.024 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:43:33.025 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:43:33.030 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:43:33.030 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:43:33.222 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:43:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.281+0000 7fb9c80a7700 1 Processor -- start 2019-12-05T06:43:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.281+0000 7fb9c80a7700 1 -- start start 2019-12-05T06:43:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.282+0000 7fb9c80a7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb9c008c6d0 0x7fb9c008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:33.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.282+0000 7fb9c80a7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9c008fef0 0x7fb9c008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:33.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.282+0000 7fb9c80a7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9c008ef20 0x7fb9c008f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:43:33.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.282+0000 7fb9c80a7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c008df90 con 0x7fb9c008fef0 2019-12-05T06:43:33.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.282+0000 7fb9c80a7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c00a3f70 con 0x7fb9c008c6d0 2019-12-05T06:43:33.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.282+0000 7fb9c80a7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c008d830 con 0x7fb9c008ef20 2019-12-05T06:43:33.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c5e43700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9c008fef0 0x7fb9c008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:33.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c5e43700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9c008fef0 0x7fb9c008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59490/0 (socket says 172.21.15.16:59490) 2019-12-05T06:43:33.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c5e43700 1 -- 172.21.15.16:0/1012429317 learned_addr learned my addr 172.21.15.16:0/1012429317 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:43:33.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c6644700 1 --2- 172.21.15.16:0/1012429317 >> v2:172.21.15.76:3300/0 conn(0x7fb9c008c6d0 0x7fb9c008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:33.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c6e45700 1 --2- 172.21.15.16:0/1012429317 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9c008ef20 0x7fb9c008f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:33.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c5e43700 1 -- 172.21.15.16:0/1012429317 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9c008ef20 msgr2=0x7fb9c008f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:33.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/1012429317 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9c008ef20 0x7fb9c008f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:33.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c5e43700 1 -- 172.21.15.16:0/1012429317 >> v2:172.21.15.76:3300/0 conn(0x7fb9c008c6d0 msgr2=0x7fb9c008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:43:33.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/1012429317 >> v2:172.21.15.76:3300/0 conn(0x7fb9c008c6d0 0x7fb9c008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:33.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c5e43700 1 -- 172.21.15.16:0/1012429317 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c008d540 con 0x7fb9c008fef0 2019-12-05T06:43:33.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.283+0000 7fb9c6644700 1 --2- 172.21.15.16:0/1012429317 >> v2:172.21.15.76:3300/0 conn(0x7fb9c008c6d0 0x7fb9c008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:43:33.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/1012429317 >> v2:172.21.15.16:3300/0 conn(0x7fb9c008fef0 0x7fb9c008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb9bc00f740 tx=0x7fb9bc00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/1012429317 >> v2:172.21.15.16:3300/0 conn(0x7fb9c008fef0 0x7fb9c008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb9bc00f740 tx=0x7fb9bc00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/1012429317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c008fef0 0x7fb9c008d000 secure :-1 s=READY pgs=1220 cs=0 l=1 rx=0x7fb9bc00f740 tx=0x7fb9bc00f570).ready entity=mon.0 client_cookie=8e3770567a8519ed server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:33.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/1012429317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c008fef0 0x7fb9c008d000 secure :-1 s=THROTTLE_DONE pgs=1220 cs=0 l=1 rx=0x7fb9bc00f740 tx=0x7fb9bc00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/1012429317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c008fef0 0x7fb9c008d000 secure :-1 s=THROTTLE_DONE pgs=1220 cs=0 l=1 rx=0x7fb9bc00f740 tx=0x7fb9bc00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/1012429317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c008fef0 0x7fb9c008d000 secure :-1 s=THROTTLE_DONE pgs=1220 cs=0 l=1 rx=0x7fb9bc00f740 tx=0x7fb9bc00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c5642700 1 -- 172.21.15.16:0/1012429317 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9bc000960 con 0x7fb9c008fef0 2019-12-05T06:43:33.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c5642700 1 -- 172.21.15.16:0/1012429317 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9bc000c90 con 0x7fb9c008fef0 2019-12-05T06:43:33.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c5642700 1 -- 172.21.15.16:0/1012429317 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9bc0012e0 con 0x7fb9c008fef0 2019-12-05T06:43:33.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c80a7700 1 -- 172.21.15.16:0/1012429317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c008fef0 msgr2=0x7fb9c008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:33.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.284+0000 7fb9c80a7700 1 --2- 172.21.15.16:0/1012429317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c008fef0 0x7fb9c008d000 secure :-1 s=READY pgs=1220 cs=0 l=1 rx=0x7fb9bc00f740 tx=0x7fb9bc00f570).stop 2019-12-05T06:43:33.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.286+0000 7fb9c80a7700 1 -- 172.21.15.16:0/1012429317 shutdown_connections 2019-12-05T06:43:33.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.286+0000 7fb9c80a7700 1 --2- 172.21.15.16:0/1012429317 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9c008ef20 0x7fb9c008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:33.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.286+0000 7fb9c80a7700 1 --2- 172.21.15.16:0/1012429317 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c008fef0 0x7fb9c008d000 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:33.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.286+0000 7fb9c80a7700 1 --2- 172.21.15.16:0/1012429317 >> v2:172.21.15.76:3300/0 conn(0x7fb9c008c6d0 0x7fb9c008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:43:33.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.286+0000 7fb9c80a7700 1 -- 172.21.15.16:0/1012429317 shutdown_connections 2019-12-05T06:43:33.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.286+0000 7fb9c80a7700 1 -- 172.21.15.16:0/1012429317 wait complete. 2019-12-05T06:43:33.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.286+0000 7fb9c80a7700 1 -- 172.21.15.16:0/1012429317 >> 172.21.15.16:0/1012429317 conn(0x7fb9c0061940 msgr2=0x7fb9c0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:33.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.289+0000 7fb9c80a7700 1 Processor -- start 2019-12-05T06:43:33.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.290+0000 7fb9c80a7700 1 -- start start 2019-12-05T06:43:33.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c80a7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9c00880a0 0x7fb9c013bcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:33.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c80a7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9c013c210 0x7fb9c0088490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c80a7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9c0089580 con 0x7fb9c013c6d0 2019-12-05T06:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c80a7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9c008ab20 con 0x7fb9c00880a0 2019-12-05T06:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c80a7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9c013ad40 con 0x7fb9c013c210 2019-12-05T06:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c6644700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb9c00880a0 0x7fb9c013bcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c6e45700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb9c013c6d0 0x7fb9c00889d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44666/0 (socket says 172.21.15.16:44666) 2019-12-05T06:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c6e45700 1 -- 172.21.15.16:0/2073410412 learned_addr learned my addr 172.21.15.16:0/2073410412 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:43:33.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9c013c210 0x7fb9c0088490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c6644700 1 -- 172.21.15.16:0/2073410412 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9c013c210 msgr2=0x7fb9c0088490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9c013c210 0x7fb9c0088490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c6644700 1 -- 172.21.15.16:0/2073410412 >> v1:172.21.15.16:6789/0 conn(0x7fb9c013c6d0 legacy=0x7fb9c00889d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9b6ffd700 1 -- 172.21.15.16:0/2073410412 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1694035950 0 0) 0x7fb9b0001150 con 0x7fb9c013c6d0 2019-12-05T06:43:33.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.291+0000 7fb9c6644700 1 -- 172.21.15.16:0/2073410412 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9c013b3d0 con 0x7fb9c00880a0 2019-12-05T06:43:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> v2:172.21.15.16:3300/0 conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> v2:172.21.15.16:3300/0 conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=READY pgs=1222 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=THROTTLE_DONE pgs=1222 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9b6ffd700 1 -- 172.21.15.16:0/2073410412 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9b8000cc0 con 0x7fb9c00880a0 2019-12-05T06:43:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=THROTTLE_DONE pgs=1222 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9b6ffd700 1 -- 172.21.15.16:0/2073410412 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9b8008a90 con 0x7fb9c00880a0 2019-12-05T06:43:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=THROTTLE_DONE pgs=1222 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9c80a7700 1 -- 172.21.15.16:0/2073410412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9c013b3d0 con 0x7fb9c00880a0 2019-12-05T06:43:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9b6ffd700 1 -- 172.21.15.16:0/2073410412 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9b8008c80 con 0x7fb9c00880a0 2019-12-05T06:43:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.292+0000 7fb9c80a7700 1 -- 172.21.15.16:0/2073410412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9c013b3d0 con 0x7fb9c00880a0 2019-12-05T06:43:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.293+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=THROTTLE_DONE pgs=1222 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.294+0000 7fb9b6ffd700 1 -- 172.21.15.16:0/2073410412 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb9b8008c80 con 0x7fb9c00880a0 2019-12-05T06:43:33.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.294+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=THROTTLE_DONE pgs=1222 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.294+0000 7fb9b6ffd700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9a0023630 0x7fb9a0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.294+0000 7fb9b6ffd700 1 -- 172.21.15.16:0/2073410412 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fb9b8046070 con 0x7fb9c00880a0 2019-12-05T06:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.294+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9a0023630 0x7fb9a0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.295+0000 7fb9c5e43700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9a0023630 0x7fb9a0025ac0 crc :-1 s=READY pgs=520 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.296+0000 7fb9c80a7700 1 -- 172.21.15.16:0/2073410412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9c013b3d0 con 0x7fb9c00880a0 2019-12-05T06:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.301+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=THROTTLE_DONE pgs=1222 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.301+0000 7fb9b6ffd700 1 -- 172.21.15.16:0/2073410412 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb9b800ccd0 con 0x7fb9c00880a0 2019-12-05T06:43:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.506+0000 7fb9c80a7700 1 -- 172.21.15.16:0/2073410412 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb9c013b3d0 con 0x7fb9c00880a0 2019-12-05T06:43:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.507+0000 7fb9c6644700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=THROTTLE_DONE pgs=1222 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:43:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.507+0000 7fb9b6ffd700 1 -- 172.21.15.16:0/2073410412 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb9b800ccd0 con 0x7fb9c00880a0 2019-12-05T06:43:33.509 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 152 2019-12-05T06:43:33.509 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:43:33.510 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5744,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:28.456001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":33,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:43:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.510+0000 7fb9c80a7700 1 -- 172.21.15.16:0/2073410412 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9a0023630 msgr2=0x7fb9a0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.510+0000 7fb9c80a7700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9a0023630 0x7fb9a0025ac0 crc :-1 s=READY pgs=520 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.510+0000 7fb9c80a7700 1 -- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 msgr2=0x7fb9c013bcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:43:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.510+0000 7fb9c80a7700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 0x7fb9c013bcd0 secure :-1 s=READY pgs=1222 cs=0 l=1 rx=0x7fb9b8010710 tx=0x7fb9b8006260).stop 2019-12-05T06:43:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.511+0000 7fb9c80a7700 1 -- 172.21.15.16:0/2073410412 shutdown_connections 2019-12-05T06:43:33.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.511+0000 7fb9c80a7700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb9c013c210 0x7fb9c0088490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.511+0000 7fb9c80a7700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9a0023630 0x7fb9a0025ac0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.511+0000 7fb9c80a7700 1 --2- 172.21.15.16:0/2073410412 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb9c00880a0 0x7fb9c013bcd0 unknown :-1 s=CLOSED pgs=1222 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:43:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.511+0000 7fb9c80a7700 1 -- 172.21.15.16:0/2073410412 shutdown_connections 2019-12-05T06:43:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.511+0000 7fb9c80a7700 1 -- 172.21.15.16:0/2073410412 wait complete. 2019-12-05T06:43:33.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:43:33.511+0000 7fb9c80a7700 1 -- 172.21.15.16:0/2073410412 >> 172.21.15.16:0/2073410412 conn(0x7fb9c0084520 msgr2=0x7fb9c0097680 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:43:33.522 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:43:33.523 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 48 secs before thrashing 2019-12-05T06:43:41.980 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:43:41.985 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:44:12.239 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:44:12.244 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:44:21.523 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:21.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.750+0000 7f678ce0e700 1 Processor -- start 2019-12-05T06:44:21.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.751+0000 7f678ce0e700 1 -- start start 2019-12-05T06:44:21.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.752+0000 7f678ce0e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f678808c6d0 0x7f678808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:21.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.752+0000 7f678ce0e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678808fb80 con 0x7f678808fef0 2019-12-05T06:44:21.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.752+0000 7f678ce0e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67880958e0 con 0x7f678808f720 2019-12-05T06:44:21.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.752+0000 7f678ce0e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f678808df90 con 0x7f678808c6d0 2019-12-05T06:44:21.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.752+0000 7f678659c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f678808fef0 0x7f678808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44910/0 (socket says 172.21.15.16:44910) 2019-12-05T06:44:21.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.752+0000 7f678659c700 1 -- 172.21.15.16:0/722377324 learned_addr learned my addr 172.21.15.16:0/722377324 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:21.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.752+0000 7f6786d9d700 1 --2- 172.21.15.16:0/722377324 >> v2:172.21.15.76:3300/0 conn(0x7f678808c6d0 0x7f678808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.752+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2645829558 0 0) 0x7f6770001150 con 0x7f678808fef0 2019-12-05T06:44:21.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.752+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6774002e80 con 0x7f678808fef0 2019-12-05T06:44:21.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3630580243 0 0) 0x7f6770001130 con 0x7f678808fef0 2019-12-05T06:44:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 >> v2:172.21.15.76:3300/0 conn(0x7f678808c6d0 msgr2=0x7f678808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 --2- 172.21.15.16:0/722377324 >> v2:172.21.15.76:3300/0 conn(0x7f678808c6d0 0x7f678808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 >> v1:172.21.15.76:6789/0 conn(0x7f678808f720 legacy=0x7f67881533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f678808e620 con 0x7f678808fef0 2019-12-05T06:44:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f6770001660 con 0x7f678808fef0 2019-12-05T06:44:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 >> v1:172.21.15.16:6789/0 conn(0x7f678808fef0 legacy=0x7f678808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 0x7f6774005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:21.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6774006690 0x7f6774008b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67740093a0 0x7f677400b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f677400c020 con 0x7f6774003a80 2019-12-05T06:44:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6774050220 con 0x7f67740093a0 2019-12-05T06:44:21.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6774050450 con 0x7f6774006690 2019-12-05T06:44:21.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6786d9d700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 0x7f6774005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f678759e700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67740093a0 0x7f677400b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f678659c700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6774006690 0x7f6774008b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6786d9d700 1 -- 172.21.15.16:0/722377324 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6774006690 msgr2=0x7f6774008b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:21.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6786d9d700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6774006690 0x7f6774008b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:21.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6786d9d700 1 -- 172.21.15.16:0/722377324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67740093a0 msgr2=0x7f677400b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:21.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6786d9d700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67740093a0 0x7f677400b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:21.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6786d9d700 1 -- 172.21.15.16:0/722377324 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6774050700 con 0x7f6774003a80 2019-12-05T06:44:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f678759e700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f67740093a0 0x7f677400b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:44:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f678659c700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6774006690 0x7f6774008b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:44:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f678759e700 1 -- 172.21.15.16:0/722377324 reap_dead start 2019-12-05T06:44:21.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6786d9d700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 0x7f6774005ee0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f677800b6e0 tx=0x7f677800ba80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6786d9d700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 0x7f6774005ee0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f677800b6e0 tx=0x7f677800ba80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.753+0000 7f6786d9d700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 0x7f6774005ee0 secure :-1 s=READY pgs=1226 cs=0 l=1 rx=0x7f677800b6e0 tx=0x7f677800ba80).ready entity=mon.0 client_cookie=5746e93c83dac11c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:21.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.754+0000 7f6786d9d700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 0x7f6774005ee0 secure :-1 s=THROTTLE_DONE pgs=1226 cs=0 l=1 rx=0x7f677800b6e0 tx=0x7f677800ba80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.754+0000 7f6786d9d700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 0x7f6774005ee0 secure :-1 s=THROTTLE_DONE pgs=1226 cs=0 l=1 rx=0x7f677800b6e0 tx=0x7f677800ba80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.754+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6778000960 con 0x7f6774003a80 2019-12-05T06:44:21.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.754+0000 7f6786d9d700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 0x7f6774005ee0 secure :-1 s=THROTTLE_DONE pgs=1226 cs=0 l=1 rx=0x7f677800b6e0 tx=0x7f677800ba80).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.754+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6778009770 con 0x7f6774003a80 2019-12-05T06:44:21.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.754+0000 7f6785d9b700 1 -- 172.21.15.16:0/722377324 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6778009ec0 con 0x7f6774003a80 2019-12-05T06:44:21.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.754+0000 7f678ce0e700 1 -- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 msgr2=0x7f6774005ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.754+0000 7f678ce0e700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 0x7f6774005ee0 secure :-1 s=READY pgs=1226 cs=0 l=1 rx=0x7f677800b6e0 tx=0x7f677800ba80).stop 2019-12-05T06:44:21.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.755+0000 7f678ce0e700 1 -- 172.21.15.16:0/722377324 shutdown_connections 2019-12-05T06:44:21.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.755+0000 7f678ce0e700 1 --2- 172.21.15.16:0/722377324 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6774003a80 0x7f6774005ee0 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:21.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.755+0000 7f678ce0e700 1 -- 172.21.15.16:0/722377324 shutdown_connections 2019-12-05T06:44:21.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.755+0000 7f678ce0e700 1 -- 172.21.15.16:0/722377324 wait complete. 2019-12-05T06:44:21.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.755+0000 7f678ce0e700 1 -- 172.21.15.16:0/722377324 >> 172.21.15.16:0/722377324 conn(0x7f6788061940 msgr2=0x7f6788096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:21.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.757+0000 7f678ce0e700 1 Processor -- start 2019-12-05T06:44:21.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.757+0000 7f678ce0e700 1 -- start start 2019-12-05T06:44:21.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.758+0000 7f678ce0e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.758+0000 7f678ce0e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6788156e70 0x7f6788159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.758+0000 7f678ce0e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f678812cb90 con 0x7f678809e7d0 2019-12-05T06:44:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.758+0000 7f678ce0e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67881315c0 con 0x7f6788156e70 2019-12-05T06:44:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.758+0000 7f678ce0e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6788131890 con 0x7f67881302f0 2019-12-05T06:44:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.758+0000 7f6786d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.759+0000 7f6786d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:35510/0 (socket says 172.21.15.16:35510) 2019-12-05T06:44:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.759+0000 7f6786d9d700 1 -- 172.21.15.16:0/2759874971 learned_addr learned my addr 172.21.15.16:0/2759874971 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.759+0000 7f678759e700 1 --2- 172.21.15.16:0/2759874971 >> v2:172.21.15.76:3300/0 conn(0x7f6788156e70 0x7f6788159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.759+0000 7f6786d9d700 1 -- 172.21.15.16:0/2759874971 >> v2:172.21.15.76:3300/0 conn(0x7f6788156e70 msgr2=0x7f6788159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.759+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> v2:172.21.15.76:3300/0 conn(0x7f6788156e70 0x7f6788159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:21.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.759+0000 7f6786d9d700 1 -- 172.21.15.16:0/2759874971 >> v1:172.21.15.76:6789/0 conn(0x7f678809e7d0 legacy=0x7f678809dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.759+0000 7f6786d9d700 1 -- 172.21.15.16:0/2759874971 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6788132680 con 0x7f67881302f0 2019-12-05T06:44:21.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.759+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.759+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.759+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=READY pgs=288 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:21.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.760+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.760+0000 7f677f7fe700 1 -- 172.21.15.16:0/2759874971 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f677800af00 con 0x7f67881302f0 2019-12-05T06:44:21.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.760+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.760+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.760+0000 7f677f7fe700 1 -- 172.21.15.16:0/2759874971 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6778009980 con 0x7f67881302f0 2019-12-05T06:44:21.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.760+0000 7f677f7fe700 1 -- 172.21.15.16:0/2759874971 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f677800af00 con 0x7f67881302f0 2019-12-05T06:44:21.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.760+0000 7f678ce0e700 1 -- 172.21.15.16:0/2759874971 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6788132680 con 0x7f67881302f0 2019-12-05T06:44:21.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.760+0000 7f678ce0e700 1 -- 172.21.15.16:0/2759874971 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6788132680 con 0x7f67881302f0 2019-12-05T06:44:21.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.761+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.761+0000 7f677f7fe700 1 -- 172.21.15.16:0/2759874971 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f677800af00 con 0x7f67881302f0 2019-12-05T06:44:21.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.761+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.761+0000 7f677f7fe700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f676c023630 0x7f676c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:21.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.761+0000 7f677f7fe700 1 -- 172.21.15.16:0/2759874971 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f6778045de0 con 0x7f67881302f0 2019-12-05T06:44:21.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.762+0000 7f678659c700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f676c023630 0x7f676c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:21.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.762+0000 7f678659c700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f676c023630 0x7f676c025ac0 crc :-1 s=READY pgs=521 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:21.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.766+0000 7f678ce0e700 1 -- 172.21.15.16:0/2759874971 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6788132680 con 0x7f67881302f0 2019-12-05T06:44:21.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.776+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.776+0000 7f677f7fe700 1 -- 172.21.15.16:0/2759874971 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f677802ae00 con 0x7f67881302f0 2019-12-05T06:44:21.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.984+0000 7f678ce0e700 1 -- 172.21.15.16:0/2759874971 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6788132680 con 0x7f67881302f0 2019-12-05T06:44:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.985+0000 7f6786d9d700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=THROTTLE_DONE pgs=288 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.985+0000 7f677f7fe700 1 -- 172.21.15.16:0/2759874971 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f6778009e30 con 0x7f67881302f0 2019-12-05T06:44:21.986 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 152 2019-12-05T06:44:21.986 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:21.987 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5744,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:28.456001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":33,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:21.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 -- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f676c023630 msgr2=0x7f676c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f676c023630 0x7f676c025ac0 crc :-1 s=READY pgs=521 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 -- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 msgr2=0x7f678809d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:21.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 secure :-1 s=READY pgs=288 cs=0 l=1 rx=0x7f6778010450 tx=0x7f6778010280).stop 2019-12-05T06:44:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 -- 172.21.15.16:0/2759874971 shutdown_connections 2019-12-05T06:44:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f676c023630 0x7f676c025ac0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 --2- 172.21.15.16:0/2759874971 >> v2:172.21.15.76:3300/0 conn(0x7f6788156e70 0x7f6788159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:21.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 --2- 172.21.15.16:0/2759874971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f67881302f0 0x7f678809d800 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:21.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 -- 172.21.15.16:0/2759874971 shutdown_connections 2019-12-05T06:44:21.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 -- 172.21.15.16:0/2759874971 wait complete. 2019-12-05T06:44:21.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:21.987+0000 7f678ce0e700 1 -- 172.21.15.16:0/2759874971 >> 172.21.15.16:0/2759874971 conn(0x7f67880a2850 msgr2=0x7f678806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:22.001 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:44:22.001 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:44:22.001 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:44:22.009 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:44:22.010 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.281+0000 7fe537346700 1 Processor -- start 2019-12-05T06:44:22.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.281+0000 7fe537346700 1 -- start start 2019-12-05T06:44:22.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.282+0000 7fe537346700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe53008fef0 0x7fe53008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:22.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.283+0000 7fe537346700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300958e0 con 0x7fe53008d2c0 2019-12-05T06:44:22.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.283+0000 7fe537346700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5300629e0 con 0x7fe53008c6d0 2019-12-05T06:44:22.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.283+0000 7fe537346700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53008df90 con 0x7fe53008fef0 2019-12-05T06:44:22.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.283+0000 7fe5350e2700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe53008d2c0 0x7fe53008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44942/0 (socket says 172.21.15.16:44942) 2019-12-05T06:44:22.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.283+0000 7fe5350e2700 1 -- 172.21.15.16:0/600489323 learned_addr learned my addr 172.21.15.16:0/600489323 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.283+0000 7fe5360e4700 1 --2- 172.21.15.16:0/600489323 >> v2:172.21.15.76:3300/0 conn(0x7fe53008fef0 0x7fe53008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.283+0000 7fe5348e1700 1 -- 172.21.15.16:0/600489323 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968435869 0 0) 0x7fe524001150 con 0x7fe53008d2c0 2019-12-05T06:44:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.283+0000 7fe5348e1700 1 -- 172.21.15.16:0/600489323 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe518002e80 con 0x7fe53008d2c0 2019-12-05T06:44:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.284+0000 7fe5348e1700 1 -- 172.21.15.16:0/600489323 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1776335802 0 0) 0x7fe528001400 con 0x7fe53008c6d0 2019-12-05T06:44:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.284+0000 7fe5348e1700 1 -- 172.21.15.16:0/600489323 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe518002ec0 con 0x7fe53008c6d0 2019-12-05T06:44:22.294 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.284+0000 7fe5360e4700 1 -- 172.21.15.16:0/600489323 >> v1:172.21.15.76:6789/0 conn(0x7fe53008c6d0 legacy=0x7fe53008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.284+0000 7fe5360e4700 1 -- 172.21.15.16:0/600489323 >> v1:172.21.15.16:6789/0 conn(0x7fe53008d2c0 legacy=0x7fe53008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.284+0000 7fe5360e4700 1 -- 172.21.15.16:0/600489323 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53008e620 con 0x7fe53008fef0 2019-12-05T06:44:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.284+0000 7fe5360e4700 1 --2- 172.21.15.16:0/600489323 >> v2:172.21.15.76:3300/0 conn(0x7fe53008fef0 0x7fe53008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe52000af80 tx=0x7fe52000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.284+0000 7fe5360e4700 1 --2- 172.21.15.16:0/600489323 >> v2:172.21.15.76:3300/0 conn(0x7fe53008fef0 0x7fe53008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe52000af80 tx=0x7fe52000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.284+0000 7fe5360e4700 1 --2- 172.21.15.16:0/600489323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe53008fef0 0x7fe53008f720 secure :-1 s=READY pgs=854 cs=0 l=1 rx=0x7fe52000af80 tx=0x7fe52000b790).ready entity=mon.1 client_cookie=d48449b01ff5c75a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.285+0000 7fe5360e4700 1 --2- 172.21.15.16:0/600489323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe53008fef0 0x7fe53008f720 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7fe52000af80 tx=0x7fe52000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.285+0000 7fe5348e1700 1 -- 172.21.15.16:0/600489323 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe520007240 con 0x7fe53008fef0 2019-12-05T06:44:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.285+0000 7fe5360e4700 1 --2- 172.21.15.16:0/600489323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe53008fef0 0x7fe53008f720 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7fe52000af80 tx=0x7fe52000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.285+0000 7fe5360e4700 1 --2- 172.21.15.16:0/600489323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe53008fef0 0x7fe53008f720 secure :-1 s=THROTTLE_DONE pgs=854 cs=0 l=1 rx=0x7fe52000af80 tx=0x7fe52000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.285+0000 7fe5348e1700 1 -- 172.21.15.16:0/600489323 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe520000bb0 con 0x7fe53008fef0 2019-12-05T06:44:22.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.285+0000 7fe5348e1700 1 -- 172.21.15.16:0/600489323 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe520009e60 con 0x7fe53008fef0 2019-12-05T06:44:22.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.285+0000 7fe537346700 1 -- 172.21.15.16:0/600489323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe53008fef0 msgr2=0x7fe53008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:22.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.285+0000 7fe537346700 1 --2- 172.21.15.16:0/600489323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe53008fef0 0x7fe53008f720 secure :-1 s=READY pgs=854 cs=0 l=1 rx=0x7fe52000af80 tx=0x7fe52000b790).stop 2019-12-05T06:44:22.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.287+0000 7fe537346700 1 -- 172.21.15.16:0/600489323 shutdown_connections 2019-12-05T06:44:22.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.287+0000 7fe537346700 1 --2- 172.21.15.16:0/600489323 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe53008fef0 0x7fe53008f720 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:22.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.287+0000 7fe537346700 1 -- 172.21.15.16:0/600489323 shutdown_connections 2019-12-05T06:44:22.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.287+0000 7fe537346700 1 -- 172.21.15.16:0/600489323 wait complete. 2019-12-05T06:44:22.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.287+0000 7fe537346700 1 -- 172.21.15.16:0/600489323 >> 172.21.15.16:0/600489323 conn(0x7fe530061940 msgr2=0x7fe530096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:22.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.292+0000 7fe537346700 1 Processor -- start 2019-12-05T06:44:22.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.292+0000 7fe537346700 1 -- start start 2019-12-05T06:44:22.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.294+0000 7fe537346700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:22.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.294+0000 7fe537346700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53012d950 con 0x7fe53012a200 2019-12-05T06:44:22.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.294+0000 7fe537346700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53012de00 con 0x7fe53014f5e0 2019-12-05T06:44:22.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.294+0000 7fe537346700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe530082d80 con 0x7fe53014f9d0 2019-12-05T06:44:22.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.294+0000 7fe5350e2700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe53012a200 0x7fe53012c5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44948/0 (socket says 172.21.15.16:44948) 2019-12-05T06:44:22.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.294+0000 7fe5350e2700 1 -- 172.21.15.16:0/3032503083 learned_addr learned my addr 172.21.15.16:0/3032503083 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:22.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.294+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.294+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 133271512 0 0) 0x7fe524001150 con 0x7fe53012a200 2019-12-05T06:44:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.294+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe510002e80 con 0x7fe53012a200 2019-12-05T06:44:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1355518926 0 0) 0x7fe5280011a0 con 0x7fe53014f5e0 2019-12-05T06:44:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5100030b0 con 0x7fe53014f5e0 2019-12-05T06:44:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe5360e4700 1 -- 172.21.15.16:0/3032503083 >> v1:172.21.15.76:6789/0 conn(0x7fe53014f5e0 legacy=0x7fe530129230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:22.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe5360e4700 1 -- 172.21.15.16:0/3032503083 >> v1:172.21.15.16:6789/0 conn(0x7fe53012a200 legacy=0x7fe53012c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe5360e4700 1 -- 172.21.15.16:0/3032503083 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe530128930 con 0x7fe53014f9d0 2019-12-05T06:44:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=READY pgs=289 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5200099b0 con 0x7fe53014f9d0 2019-12-05T06:44:22.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe520009bb0 con 0x7fe53014f9d0 2019-12-05T06:44:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.295+0000 7fe537346700 1 -- 172.21.15.16:0/3032503083 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe530128930 con 0x7fe53014f9d0 2019-12-05T06:44:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.296+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe52000a2c0 con 0x7fe53014f9d0 2019-12-05T06:44:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.296+0000 7fe537346700 1 -- 172.21.15.16:0/3032503083 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe53012d0d0 con 0x7fe53014f9d0 2019-12-05T06:44:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.297+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.298+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe5200097c0 con 0x7fe53014f9d0 2019-12-05T06:44:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.298+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.298+0000 7fe52e7fc700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023650 0x7fe510025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.298+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fe520045db0 con 0x7fe53014f9d0 2019-12-05T06:44:22.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.298+0000 7fe5358e3700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023650 0x7fe510025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:22.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.299+0000 7fe5358e3700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023650 0x7fe510025ae0 crc :-1 s=READY pgs=522 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:22.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.312+0000 7fe537346700 1 -- 172.21.15.16:0/3032503083 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe530128930 con 0x7fe53014f9d0 2019-12-05T06:44:22.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.321+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.321+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe5200095b0 con 0x7fe53014f9d0 2019-12-05T06:44:22.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.547+0000 7fe537346700 1 -- 172.21.15.16:0/3032503083 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe530128930 con 0x7fe53014f9d0 2019-12-05T06:44:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.549+0000 7fe5360e4700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=THROTTLE_DONE pgs=289 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:22.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.549+0000 7fe52e7fc700 1 -- 172.21.15.16:0/3032503083 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe5200097c0 con 0x7fe53014f9d0 2019-12-05T06:44:22.551 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:22.551 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5744,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:28.456001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":33,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:22.552 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 152 2019-12-05T06:44:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.553+0000 7fe537346700 1 -- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023650 msgr2=0x7fe510025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:22.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.553+0000 7fe537346700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023650 0x7fe510025ae0 crc :-1 s=READY pgs=522 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.553+0000 7fe537346700 1 -- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 msgr2=0x7fe530129a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.553+0000 7fe537346700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 secure :-1 s=READY pgs=289 cs=0 l=1 rx=0x7fe5200103c0 tx=0x7fe520010760).stop 2019-12-05T06:44:22.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.553+0000 7fe537346700 1 -- 172.21.15.16:0/3032503083 shutdown_connections 2019-12-05T06:44:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.553+0000 7fe537346700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023650 0x7fe510025ae0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.553+0000 7fe537346700 1 --2- 172.21.15.16:0/3032503083 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53014f9d0 0x7fe530129a30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.553+0000 7fe537346700 1 -- 172.21.15.16:0/3032503083 shutdown_connections 2019-12-05T06:44:22.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.553+0000 7fe537346700 1 -- 172.21.15.16:0/3032503083 wait complete. 2019-12-05T06:44:22.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:22.553+0000 7fe537346700 1 -- 172.21.15.16:0/3032503083 >> 172.21.15.16:0/3032503083 conn(0x7fe5300a1ac0 msgr2=0x7fe53008ab30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:22.567 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:44:24.569 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.858+0000 7f1c2d329700 1 Processor -- start 2019-12-05T06:44:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.858+0000 7f1c2d329700 1 -- start start 2019-12-05T06:44:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.860+0000 7f1c2d329700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c2808c6d0 0x7f1c2808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:24.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.860+0000 7f1c2d329700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c2808f720 0x7f1c2808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.860+0000 7f1c2d329700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c280958e0 con 0x7f1c2808fef0 2019-12-05T06:44:24.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.860+0000 7f1c2d329700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c280a3f70 con 0x7f1c2808f720 2019-12-05T06:44:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.860+0000 7f1c2d329700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2808d830 con 0x7f1c2808c6d0 2019-12-05T06:44:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.860+0000 7f1c27fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c2808f720 0x7f1c2808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.860+0000 7f1c27fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c2808f720 0x7f1c2808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59798/0 (socket says 172.21.15.16:59798) 2019-12-05T06:44:24.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.860+0000 7f1c27fff700 1 -- 172.21.15.16:0/215365493 learned_addr learned my addr 172.21.15.16:0/215365493 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.860+0000 7f1c26ffd700 1 --1- 172.21.15.16:0/215365493 >> v1:172.21.15.16:6789/0 conn(0x7f1c2808fef0 0x7f1c2808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44964/0 (socket says 172.21.15.16:44964) 2019-12-05T06:44:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.860+0000 7f1c277fe700 1 --2- 172.21.15.16:0/215365493 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c2808c6d0 0x7f1c2808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.861+0000 7f1c27fff700 1 -- 172.21.15.16:0/215365493 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c2808c6d0 msgr2=0x7f1c2808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:24.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.861+0000 7f1c27fff700 1 --2- 172.21.15.16:0/215365493 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c2808c6d0 0x7f1c2808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.861+0000 7f1c27fff700 1 -- 172.21.15.16:0/215365493 >> v1:172.21.15.16:6789/0 conn(0x7f1c2808fef0 legacy=0x7f1c2808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.861+0000 7f1c27fff700 1 -- 172.21.15.16:0/215365493 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c2808fbc0 con 0x7f1c2808f720 2019-12-05T06:44:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.861+0000 7f1c27fff700 1 --2- 172.21.15.16:0/215365493 >> v2:172.21.15.16:3300/0 conn(0x7f1c2808f720 0x7f1c2808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1c2000fa30 tx=0x7f1c2000fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.861+0000 7f1c27fff700 1 --2- 172.21.15.16:0/215365493 >> v2:172.21.15.16:3300/0 conn(0x7f1c2808f720 0x7f1c2808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1c2000fa30 tx=0x7f1c2000fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.861+0000 7f1c27fff700 1 --2- 172.21.15.16:0/215365493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2808f720 0x7f1c2808d000 secure :-1 s=READY pgs=1230 cs=0 l=1 rx=0x7f1c2000fa30 tx=0x7f1c2000fdd0).ready entity=mon.0 client_cookie=a6a66fb15746944b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.862+0000 7f1c27fff700 1 --2- 172.21.15.16:0/215365493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2808f720 0x7f1c2808d000 secure :-1 s=THROTTLE_DONE pgs=1230 cs=0 l=1 rx=0x7f1c2000fa30 tx=0x7f1c2000fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.862+0000 7f1c27fff700 1 --2- 172.21.15.16:0/215365493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2808f720 0x7f1c2808d000 secure :-1 s=THROTTLE_DONE pgs=1230 cs=0 l=1 rx=0x7f1c2000fa30 tx=0x7f1c2000fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.862+0000 7f1c27fff700 1 --2- 172.21.15.16:0/215365493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2808f720 0x7f1c2808d000 secure :-1 s=THROTTLE_DONE pgs=1230 cs=0 l=1 rx=0x7f1c2000fa30 tx=0x7f1c2000fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.862+0000 7f1c267fc700 1 -- 172.21.15.16:0/215365493 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c20000df0 con 0x7f1c2808f720 2019-12-05T06:44:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.862+0000 7f1c267fc700 1 -- 172.21.15.16:0/215365493 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c20001460 con 0x7f1c2808f720 2019-12-05T06:44:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.862+0000 7f1c267fc700 1 -- 172.21.15.16:0/215365493 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c20001b30 con 0x7f1c2808f720 2019-12-05T06:44:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.862+0000 7f1c2d329700 1 -- 172.21.15.16:0/215365493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2808f720 msgr2=0x7f1c2808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.862+0000 7f1c2d329700 1 --2- 172.21.15.16:0/215365493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2808f720 0x7f1c2808d000 secure :-1 s=READY pgs=1230 cs=0 l=1 rx=0x7f1c2000fa30 tx=0x7f1c2000fdd0).stop 2019-12-05T06:44:24.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.864+0000 7f1c2d329700 1 -- 172.21.15.16:0/215365493 shutdown_connections 2019-12-05T06:44:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.864+0000 7f1c2d329700 1 --2- 172.21.15.16:0/215365493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2808f720 0x7f1c2808d000 unknown :-1 s=CLOSED pgs=1230 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.864+0000 7f1c2d329700 1 --2- 172.21.15.16:0/215365493 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1c2808c6d0 0x7f1c2808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.864+0000 7f1c2d329700 1 -- 172.21.15.16:0/215365493 shutdown_connections 2019-12-05T06:44:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.864+0000 7f1c2d329700 1 -- 172.21.15.16:0/215365493 wait complete. 2019-12-05T06:44:24.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.864+0000 7f1c2d329700 1 -- 172.21.15.16:0/215365493 >> 172.21.15.16:0/215365493 conn(0x7f1c28061940 msgr2=0x7f1c28096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:24.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.870+0000 7f1c2d329700 1 Processor -- start 2019-12-05T06:44:24.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.870+0000 7f1c2d329700 1 -- start start 2019-12-05T06:44:24.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.872+0000 7f1c2d329700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1c28152dd0 0x7f1c28089f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:24.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.872+0000 7f1c2d329700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c2819b400 0x7f1c2819d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.872+0000 7f1c2d329700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c2809be80 con 0x7f1c281529e0 2019-12-05T06:44:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.872+0000 7f1c2d329700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2809cff0 con 0x7f1c2819b400 2019-12-05T06:44:24.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.872+0000 7f1c2d329700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c2809c5c0 con 0x7f1c28152dd0 2019-12-05T06:44:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.872+0000 7f1c27fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c2819b400 0x7f1c2819d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.872+0000 7f1c27fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1c2819b400 0x7f1c2819d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59804/0 (socket says 172.21.15.16:59804) 2019-12-05T06:44:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.872+0000 7f1c27fff700 1 -- 172.21.15.16:0/893994512 learned_addr learned my addr 172.21.15.16:0/893994512 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:24.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.872+0000 7f1c26ffd700 1 --2- 172.21.15.16:0/893994512 >> v2:172.21.15.76:3300/0 conn(0x7f1c28152dd0 0x7f1c28089f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.873+0000 7f1c27fff700 1 -- 172.21.15.16:0/893994512 >> v2:172.21.15.76:3300/0 conn(0x7f1c28152dd0 msgr2=0x7f1c28089f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:24.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.873+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> v2:172.21.15.76:3300/0 conn(0x7f1c28152dd0 0x7f1c28089f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.873+0000 7f1c27fff700 1 -- 172.21.15.16:0/893994512 >> v1:172.21.15.76:6789/0 conn(0x7f1c281529e0 legacy=0x7f1c28089700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:24.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.873+0000 7f1c27fff700 1 -- 172.21.15.16:0/893994512 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c28079ea0 con 0x7f1c2819b400 2019-12-05T06:44:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.873+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> v2:172.21.15.16:3300/0 conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.873+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> v2:172.21.15.16:3300/0 conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.873+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=READY pgs=1231 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.874+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=THROTTLE_DONE pgs=1231 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.874+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=THROTTLE_DONE pgs=1231 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.874+0000 7f1c1ffff700 1 -- 172.21.15.16:0/893994512 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c20005cb0 con 0x7f1c2819b400 2019-12-05T06:44:24.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.874+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=THROTTLE_DONE pgs=1231 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.874+0000 7f1c1ffff700 1 -- 172.21.15.16:0/893994512 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c200058d0 con 0x7f1c2819b400 2019-12-05T06:44:24.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.874+0000 7f1c1ffff700 1 -- 172.21.15.16:0/893994512 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c20006290 con 0x7f1c2819b400 2019-12-05T06:44:24.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.874+0000 7f1c2d329700 1 -- 172.21.15.16:0/893994512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c28079ea0 con 0x7f1c2819b400 2019-12-05T06:44:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.874+0000 7f1c2d329700 1 -- 172.21.15.16:0/893994512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c28079ea0 con 0x7f1c2819b400 2019-12-05T06:44:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.875+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=THROTTLE_DONE pgs=1231 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.876+0000 7f1c1ffff700 1 -- 172.21.15.16:0/893994512 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1c20005ac0 con 0x7f1c2819b400 2019-12-05T06:44:24.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.876+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=THROTTLE_DONE pgs=1231 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.876+0000 7f1c1ffff700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 0x7f1c0c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:24.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.876+0000 7f1c1ffff700 1 -- 172.21.15.16:0/893994512 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f1c20046140 con 0x7f1c2819b400 2019-12-05T06:44:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.876+0000 7f1c277fe700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 0x7f1c0c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.877+0000 7f1c277fe700 1 -- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 msgr2=0x7f1c0c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:44:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.877+0000 7f1c277fe700 1 -- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 msgr2=0x7f1c0c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:44:24.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.877+0000 7f1c277fe700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 0x7f1c0c025ac0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:44:24.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.877+0000 7f1c277fe700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 0x7f1c0c025ac0 crc :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:44:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.889+0000 7f1c2d329700 1 -- 172.21.15.16:0/893994512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c2809be80 con 0x7f1c2819b400 2019-12-05T06:44:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.896+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=THROTTLE_DONE pgs=1231 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:24.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:24.896+0000 7f1c1ffff700 1 -- 172.21.15.16:0/893994512 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1c200056c0 con 0x7f1c2819b400 2019-12-05T06:44:25.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.078+0000 7f1c277fe700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 0x7f1c0c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:25.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.078+0000 7f1c277fe700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 0x7f1c0c025ac0 crc :-1 s=READY pgs=524 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:25.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.163+0000 7f1c2d329700 1 -- 172.21.15.16:0/893994512 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c280958e0 con 0x7f1c2819b400 2019-12-05T06:44:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.164+0000 7f1c27fff700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=THROTTLE_DONE pgs=1231 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.164+0000 7f1c1ffff700 1 -- 172.21.15.16:0/893994512 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f1c200056c0 con 0x7f1c2819b400 2019-12-05T06:44:25.165 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 152 2019-12-05T06:44:25.165 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:25.165 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5744,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:28.456001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":33,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.166+0000 7f1c2d329700 1 -- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 msgr2=0x7f1c0c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.166+0000 7f1c2d329700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 0x7f1c0c025ac0 crc :-1 s=READY pgs=524 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.166+0000 7f1c2d329700 1 -- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 msgr2=0x7f1c2819d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:25.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.166+0000 7f1c2d329700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 0x7f1c2819d800 secure :-1 s=READY pgs=1231 cs=0 l=1 rx=0x7f1c200107e0 tx=0x7f1c20010610).stop 2019-12-05T06:44:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.166+0000 7f1c2d329700 1 -- 172.21.15.16:0/893994512 shutdown_connections 2019-12-05T06:44:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.166+0000 7f1c2d329700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c0c023630 0x7f1c0c025ac0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.166+0000 7f1c2d329700 1 --2- 172.21.15.16:0/893994512 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1c2819b400 0x7f1c2819d800 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.166+0000 7f1c2d329700 1 --2- 172.21.15.16:0/893994512 >> v2:172.21.15.76:3300/0 conn(0x7f1c28152dd0 0x7f1c28089f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.167+0000 7f1c2d329700 1 -- 172.21.15.16:0/893994512 shutdown_connections 2019-12-05T06:44:25.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.167+0000 7f1c2d329700 1 -- 172.21.15.16:0/893994512 wait complete. 2019-12-05T06:44:25.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:25.167+0000 7f1c2d329700 1 -- 172.21.15.16:0/893994512 >> 172.21.15.16:0/893994512 conn(0x7f1c28098480 msgr2=0x7f1c28071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:25.179 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:44:27.180 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:27.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.431+0000 7fb540d89700 1 Processor -- start 2019-12-05T06:44:27.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.431+0000 7fb540d89700 1 -- start start 2019-12-05T06:44:27.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.433+0000 7fb540d89700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 0x7fb53c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:27.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.433+0000 7fb540d89700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53c0958e0 con 0x7fb53c08f720 2019-12-05T06:44:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.433+0000 7fb540d89700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53c0629e0 con 0x7fb53c08fef0 2019-12-05T06:44:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.433+0000 7fb540d89700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c0a3f70 con 0x7fb53c08c6d0 2019-12-05T06:44:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.433+0000 7fb53b59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb53c08f720 0x7fb53c1533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44982/0 (socket says 172.21.15.16:44982) 2019-12-05T06:44:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.433+0000 7fb53b59e700 1 -- 172.21.15.16:0/1540302066 learned_addr learned my addr 172.21.15.16:0/1540302066 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.433+0000 7fb53a59c700 1 --1- 172.21.15.16:0/1540302066 >> v1:172.21.15.76:6789/0 conn(0x7fb53c08fef0 0x7fb53c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:38054/0 (socket says 172.21.15.16:38054) 2019-12-05T06:44:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.433+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/1540302066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 0x7fb53c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:27.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.434+0000 7fb539d9b700 1 -- 172.21.15.16:0/1540302066 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1392947311 0 0) 0x7fb524001080 con 0x7fb53c08fef0 2019-12-05T06:44:27.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.434+0000 7fb539d9b700 1 -- 172.21.15.16:0/1540302066 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb528002e80 con 0x7fb53c08fef0 2019-12-05T06:44:27.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.434+0000 7fb539d9b700 1 -- 172.21.15.16:0/1540302066 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54821534 0 0) 0x7fb52c001150 con 0x7fb53c08f720 2019-12-05T06:44:27.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.434+0000 7fb539d9b700 1 -- 172.21.15.16:0/1540302066 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb528004170 con 0x7fb53c08f720 2019-12-05T06:44:27.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.434+0000 7fb53ad9d700 1 -- 172.21.15.16:0/1540302066 >> v1:172.21.15.76:6789/0 conn(0x7fb53c08fef0 legacy=0x7fb53c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:27.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.434+0000 7fb53ad9d700 1 -- 172.21.15.16:0/1540302066 >> v1:172.21.15.16:6789/0 conn(0x7fb53c08f720 legacy=0x7fb53c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:27.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.434+0000 7fb53ad9d700 1 -- 172.21.15.16:0/1540302066 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c08fbc0 con 0x7fb53c08c6d0 2019-12-05T06:44:27.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.434+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/1540302066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 0x7fb53c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb53400b2d0 tx=0x7fb53400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.434+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/1540302066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 0x7fb53c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb53400b2d0 tx=0x7fb53400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.435+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/1540302066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 0x7fb53c08cac0 secure :-1 s=READY pgs=290 cs=0 l=1 rx=0x7fb53400b2d0 tx=0x7fb53400bae0).ready entity=mon.2 client_cookie=86fb1d54849b0ded server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:27.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.435+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/1540302066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 0x7fb53c08cac0 secure :-1 s=THROTTLE_DONE pgs=290 cs=0 l=1 rx=0x7fb53400b2d0 tx=0x7fb53400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.435+0000 7fb539d9b700 1 -- 172.21.15.16:0/1540302066 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb534000ce0 con 0x7fb53c08c6d0 2019-12-05T06:44:27.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.435+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/1540302066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 0x7fb53c08cac0 secure :-1 s=THROTTLE_DONE pgs=290 cs=0 l=1 rx=0x7fb53400b2d0 tx=0x7fb53400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.435+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/1540302066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 0x7fb53c08cac0 secure :-1 s=THROTTLE_DONE pgs=290 cs=0 l=1 rx=0x7fb53400b2d0 tx=0x7fb53400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.435+0000 7fb539d9b700 1 -- 172.21.15.16:0/1540302066 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb534009be0 con 0x7fb53c08c6d0 2019-12-05T06:44:27.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.435+0000 7fb539d9b700 1 -- 172.21.15.16:0/1540302066 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53400a2b0 con 0x7fb53c08c6d0 2019-12-05T06:44:27.446 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.435+0000 7fb540d89700 1 -- 172.21.15.16:0/1540302066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 msgr2=0x7fb53c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.435+0000 7fb540d89700 1 --2- 172.21.15.16:0/1540302066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 0x7fb53c08cac0 secure :-1 s=READY pgs=290 cs=0 l=1 rx=0x7fb53400b2d0 tx=0x7fb53400bae0).stop 2019-12-05T06:44:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.437+0000 7fb540d89700 1 -- 172.21.15.16:0/1540302066 shutdown_connections 2019-12-05T06:44:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.437+0000 7fb540d89700 1 --2- 172.21.15.16:0/1540302066 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c08c6d0 0x7fb53c08cac0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.437+0000 7fb540d89700 1 -- 172.21.15.16:0/1540302066 shutdown_connections 2019-12-05T06:44:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.437+0000 7fb540d89700 1 -- 172.21.15.16:0/1540302066 wait complete. 2019-12-05T06:44:27.447 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.437+0000 7fb540d89700 1 -- 172.21.15.16:0/1540302066 >> 172.21.15.16:0/1540302066 conn(0x7fb53c061940 msgr2=0x7fb53c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:27.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.443+0000 7fb540d89700 1 Processor -- start 2019-12-05T06:44:27.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.443+0000 7fb540d89700 1 -- start start 2019-12-05T06:44:27.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.445+0000 7fb540d89700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:27.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.445+0000 7fb540d89700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53c128b20 con 0x7fb53c154cb0 2019-12-05T06:44:27.456 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.445+0000 7fb540d89700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb53c128660 con 0x7fb53c12c740 2019-12-05T06:44:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.445+0000 7fb540d89700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb53c129db0 con 0x7fb53c1325e0 2019-12-05T06:44:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.445+0000 7fb53b59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb53c154cb0 0x7fb53c1570b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:44990/0 (socket says 172.21.15.16:44990) 2019-12-05T06:44:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.445+0000 7fb53b59e700 1 -- 172.21.15.16:0/2746912534 learned_addr learned my addr 172.21.15.16:0/2746912534 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.445+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.445+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3332302076 0 0) 0x7fb52c000a40 con 0x7fb53c154cb0 2019-12-05T06:44:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.446+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb520002e80 con 0x7fb53c154cb0 2019-12-05T06:44:27.457 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.446+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2641894814 0 0) 0x7fb524000e70 con 0x7fb53c12c740 2019-12-05T06:44:27.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.446+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb520004170 con 0x7fb53c12c740 2019-12-05T06:44:27.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.446+0000 7fb53ad9d700 1 -- 172.21.15.16:0/2746912534 >> v1:172.21.15.76:6789/0 conn(0x7fb53c12c740 legacy=0x7fb53c12bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:27.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.446+0000 7fb53ad9d700 1 -- 172.21.15.16:0/2746912534 >> v1:172.21.15.16:6789/0 conn(0x7fb53c154cb0 legacy=0x7fb53c1570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:27.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.446+0000 7fb53ad9d700 1 -- 172.21.15.16:0/2746912534 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb53c14a820 con 0x7fb53c1325e0 2019-12-05T06:44:27.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.446+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.446+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.446+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=READY pgs=291 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:27.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.447+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.447+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb534000cc0 con 0x7fb53c1325e0 2019-12-05T06:44:27.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.447+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.447+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.447+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb534009af0 con 0x7fb53c1325e0 2019-12-05T06:44:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.447+0000 7fb540d89700 1 -- 172.21.15.16:0/2746912534 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb53c12ae70 con 0x7fb53c1325e0 2019-12-05T06:44:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.447+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb53400a240 con 0x7fb53c1325e0 2019-12-05T06:44:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.447+0000 7fb540d89700 1 -- 172.21.15.16:0/2746912534 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb53c14a820 con 0x7fb53c1325e0 2019-12-05T06:44:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.448+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.449+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb534009ce0 con 0x7fb53c1325e0 2019-12-05T06:44:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.449+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.449+0000 7fb5337fe700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb520023650 0x7fb520025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.449+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fb534046130 con 0x7fb53c1325e0 2019-12-05T06:44:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.450+0000 7fb53a59c700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb520023650 0x7fb520025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:27.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.450+0000 7fb53a59c700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb520023650 0x7fb520025ae0 crc :-1 s=READY pgs=525 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:27.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.463+0000 7fb540d89700 1 -- 172.21.15.16:0/2746912534 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb53c12ae70 con 0x7fb53c1325e0 2019-12-05T06:44:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.472+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.472+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb53400b8c0 con 0x7fb53c1325e0 2019-12-05T06:44:27.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.740+0000 7fb540d89700 1 -- 172.21.15.16:0/2746912534 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb53c14a820 con 0x7fb53c1325e0 2019-12-05T06:44:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.742+0000 7fb53ad9d700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=THROTTLE_DONE pgs=291 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:27.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.742+0000 7fb5337fe700 1 -- 172.21.15.16:0/2746912534 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fb534009ce0 con 0x7fb53c1325e0 2019-12-05T06:44:27.744 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 152 2019-12-05T06:44:27.744 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:27.744 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5744,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:28.456001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":33,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.746+0000 7fb540d89700 1 -- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb520023650 msgr2=0x7fb520025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.746+0000 7fb540d89700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb520023650 0x7fb520025ae0 crc :-1 s=READY pgs=525 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.746+0000 7fb540d89700 1 -- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 msgr2=0x7fb53c12b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.746+0000 7fb540d89700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 secure :-1 s=READY pgs=291 cs=0 l=1 rx=0x7fb534010700 tx=0x7fb534010aa0).stop 2019-12-05T06:44:27.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.746+0000 7fb540d89700 1 -- 172.21.15.16:0/2746912534 shutdown_connections 2019-12-05T06:44:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.746+0000 7fb540d89700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb520023650 0x7fb520025ae0 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.746+0000 7fb540d89700 1 --2- 172.21.15.16:0/2746912534 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb53c1325e0 0x7fb53c12b770 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.746+0000 7fb540d89700 1 -- 172.21.15.16:0/2746912534 shutdown_connections 2019-12-05T06:44:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.746+0000 7fb540d89700 1 -- 172.21.15.16:0/2746912534 wait complete. 2019-12-05T06:44:27.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:27.746+0000 7fb540d89700 1 -- 172.21.15.16:0/2746912534 >> 172.21.15.16:0/2746912534 conn(0x7fb53c0a2850 msgr2=0x7fb53c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:27.757 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:44:29.758 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:30.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.026+0000 7f0b39b6c700 1 Processor -- start 2019-12-05T06:44:30.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.027+0000 7f0b39b6c700 1 -- start start 2019-12-05T06:44:30.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.028+0000 7f0b39b6c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0b3408c6d0 0x7f0b3408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.028+0000 7f0b39b6c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3408fef0 0x7f0b3408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.028+0000 7f0b39b6c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b340958e0 con 0x7f0b3408ef20 2019-12-05T06:44:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.028+0000 7f0b39b6c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b3408df90 con 0x7f0b3408c6d0 2019-12-05T06:44:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.028+0000 7f0b39b6c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b340a3f70 con 0x7f0b3408fef0 2019-12-05T06:44:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.029+0000 7f0b337fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3408fef0 0x7f0b3408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.029+0000 7f0b33fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0b3408c6d0 0x7f0b3408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:30.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.029+0000 7f0b337fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3408fef0 0x7f0b3408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:35590/0 (socket says 172.21.15.16:35590) 2019-12-05T06:44:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.029+0000 7f0b33fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0b3408c6d0 0x7f0b3408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:45918/0 (socket says 172.21.15.16:45918) 2019-12-05T06:44:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.029+0000 7f0b337fe700 1 -- 172.21.15.16:0/273912776 learned_addr learned my addr 172.21.15.16:0/273912776 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.030+0000 7f0b33fff700 1 -- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3408fef0 msgr2=0x7f0b3408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.030+0000 7f0b33fff700 1 --2- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3408fef0 0x7f0b3408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.030+0000 7f0b33fff700 1 -- 172.21.15.16:0/273912776 >> v1:172.21.15.76:6789/0 conn(0x7f0b3408ef20 legacy=0x7f0b3408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.030+0000 7f0b32ffd700 1 -- 172.21.15.16:0/273912776 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789695089 0 0) 0x7f0b2c0009b0 con 0x7f0b3408ef20 2019-12-05T06:44:30.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.030+0000 7f0b33fff700 1 -- 172.21.15.16:0/273912776 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3408d540 con 0x7f0b3408c6d0 2019-12-05T06:44:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.030+0000 7f0b337fe700 1 --2- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3408fef0 0x7f0b3408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.030+0000 7f0b33fff700 1 --2- 172.21.15.16:0/273912776 >> v2:172.21.15.76:3300/0 conn(0x7f0b3408c6d0 0x7f0b3408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0b2400b2d0 tx=0x7f0b2400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.030+0000 7f0b33fff700 1 --2- 172.21.15.16:0/273912776 >> v2:172.21.15.76:3300/0 conn(0x7f0b3408c6d0 0x7f0b3408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0b2400b2d0 tx=0x7f0b2400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.030+0000 7f0b33fff700 1 --2- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3408c6d0 0x7f0b3408cac0 secure :-1 s=READY pgs=860 cs=0 l=1 rx=0x7f0b2400b2d0 tx=0x7f0b2400bae0).ready entity=mon.1 client_cookie=87281f359993bc25 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.030+0000 7f0b33fff700 1 --2- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3408c6d0 0x7f0b3408cac0 secure :-1 s=THROTTLE_DONE pgs=860 cs=0 l=1 rx=0x7f0b2400b2d0 tx=0x7f0b2400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.031+0000 7f0b32ffd700 1 -- 172.21.15.16:0/273912776 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b24007590 con 0x7f0b3408c6d0 2019-12-05T06:44:30.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.031+0000 7f0b33fff700 1 --2- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3408c6d0 0x7f0b3408cac0 secure :-1 s=THROTTLE_DONE pgs=860 cs=0 l=1 rx=0x7f0b2400b2d0 tx=0x7f0b2400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.031+0000 7f0b33fff700 1 --2- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3408c6d0 0x7f0b3408cac0 secure :-1 s=THROTTLE_DONE pgs=860 cs=0 l=1 rx=0x7f0b2400b2d0 tx=0x7f0b2400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.031+0000 7f0b32ffd700 1 -- 172.21.15.16:0/273912776 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b24009d30 con 0x7f0b3408c6d0 2019-12-05T06:44:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.031+0000 7f0b32ffd700 1 -- 172.21.15.16:0/273912776 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b2400a400 con 0x7f0b3408c6d0 2019-12-05T06:44:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.031+0000 7f0b39b6c700 1 -- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3408c6d0 msgr2=0x7f0b3408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.031+0000 7f0b39b6c700 1 --2- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3408c6d0 0x7f0b3408cac0 secure :-1 s=READY pgs=860 cs=0 l=1 rx=0x7f0b2400b2d0 tx=0x7f0b2400bae0).stop 2019-12-05T06:44:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.033+0000 7f0b39b6c700 1 -- 172.21.15.16:0/273912776 shutdown_connections 2019-12-05T06:44:30.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.033+0000 7f0b39b6c700 1 --2- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3408fef0 0x7f0b3408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.033+0000 7f0b39b6c700 1 --2- 172.21.15.16:0/273912776 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f0b3408c6d0 0x7f0b3408cac0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.033+0000 7f0b39b6c700 1 -- 172.21.15.16:0/273912776 shutdown_connections 2019-12-05T06:44:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.033+0000 7f0b39b6c700 1 -- 172.21.15.16:0/273912776 wait complete. 2019-12-05T06:44:30.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.033+0000 7f0b39b6c700 1 -- 172.21.15.16:0/273912776 >> 172.21.15.16:0/273912776 conn(0x7f0b34061940 msgr2=0x7f0b34096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:30.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.038+0000 7f0b39b6c700 1 Processor -- start 2019-12-05T06:44:30.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.038+0000 7f0b39b6c700 1 -- start start 2019-12-05T06:44:30.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b39b6c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b3412ca70 0x7f0b3409d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:30.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b39b6c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0b3409e7c0 0x7f0b3409dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:30.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b39b6c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3409e270 0x7f0b3406dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:30.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b39b6c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b34131c20 con 0x7f0b3412ca70 2019-12-05T06:44:30.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b39b6c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b341311f0 con 0x7f0b3409e7c0 2019-12-05T06:44:30.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b39b6c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b341314c0 con 0x7f0b3409e270 2019-12-05T06:44:30.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b33fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b3412ca70 0x7f0b3409d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:30.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b33fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b3412ca70 0x7f0b3409d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59834/0 (socket says 172.21.15.16:59834) 2019-12-05T06:44:30.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b33fff700 1 -- 172.21.15.16:0/2364333794 learned_addr learned my addr 172.21.15.16:0/2364333794 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:30.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b337fe700 1 --2- 172.21.15.16:0/2364333794 >> v2:172.21.15.76:3300/0 conn(0x7f0b3409e7c0 0x7f0b3409dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:30.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.040+0000 7f0b3890a700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3409e270 0x7f0b3406dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:30.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b33fff700 1 -- 172.21.15.16:0/2364333794 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3409e270 msgr2=0x7f0b3406dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:30.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3409e270 0x7f0b3406dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:30.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b33fff700 1 -- 172.21.15.16:0/2364333794 >> v2:172.21.15.76:3300/0 conn(0x7f0b3409e7c0 msgr2=0x7f0b3409dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:30.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> v2:172.21.15.76:3300/0 conn(0x7f0b3409e7c0 0x7f0b3409dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:30.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b33fff700 1 -- 172.21.15.16:0/2364333794 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b341322b0 con 0x7f0b3412ca70 2019-12-05T06:44:30.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> v2:172.21.15.16:3300/0 conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> v2:172.21.15.16:3300/0 conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=READY pgs=1234 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:30.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=THROTTLE_DONE pgs=1234 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b30ff9700 1 -- 172.21.15.16:0/2364333794 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b2400aff0 con 0x7f0b3412ca70 2019-12-05T06:44:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.041+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=THROTTLE_DONE pgs=1234 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.042+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=THROTTLE_DONE pgs=1234 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.042+0000 7f0b30ff9700 1 -- 172.21.15.16:0/2364333794 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b24009d00 con 0x7f0b3412ca70 2019-12-05T06:44:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.042+0000 7f0b30ff9700 1 -- 172.21.15.16:0/2364333794 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b2400a410 con 0x7f0b3412ca70 2019-12-05T06:44:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.042+0000 7f0b39b6c700 1 -- 172.21.15.16:0/2364333794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b341322b0 con 0x7f0b3412ca70 2019-12-05T06:44:30.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.042+0000 7f0b39b6c700 1 -- 172.21.15.16:0/2364333794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b341322b0 con 0x7f0b3412ca70 2019-12-05T06:44:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.043+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=THROTTLE_DONE pgs=1234 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.044+0000 7f0b30ff9700 1 -- 172.21.15.16:0/2364333794 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0b24009900 con 0x7f0b3412ca70 2019-12-05T06:44:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.044+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=THROTTLE_DONE pgs=1234 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.044+0000 7f0b30ff9700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b18023630 0x7f0b18025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.045+0000 7f0b30ff9700 1 -- 172.21.15.16:0/2364333794 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f0b2408a140 con 0x7f0b3412ca70 2019-12-05T06:44:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.045+0000 7f0b337fe700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b18023630 0x7f0b18025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:30.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.045+0000 7f0b337fe700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b18023630 0x7f0b18025ac0 crc :-1 s=READY pgs=526 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:30.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.058+0000 7f0b39b6c700 1 -- 172.21.15.16:0/2364333794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b341322b0 con 0x7f0b3412ca70 2019-12-05T06:44:30.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.067+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=THROTTLE_DONE pgs=1234 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.067+0000 7f0b30ff9700 1 -- 172.21.15.16:0/2364333794 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0b2400afa0 con 0x7f0b3412ca70 2019-12-05T06:44:30.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.331+0000 7f0b39b6c700 1 -- 172.21.15.16:0/2364333794 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b341322b0 con 0x7f0b3412ca70 2019-12-05T06:44:30.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.331+0000 7f0b33fff700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=THROTTLE_DONE pgs=1234 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:30.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.331+0000 7f0b30ff9700 1 -- 172.21.15.16:0/2364333794 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0b24009900 con 0x7f0b3412ca70 2019-12-05T06:44:30.333 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 152 2019-12-05T06:44:30.333 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:30.333 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5744,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:28.456001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":33,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:30.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 -- 172.21.15.16:0/2364333794 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b18023630 msgr2=0x7f0b18025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:30.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b18023630 0x7f0b18025ac0 crc :-1 s=READY pgs=526 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:30.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 -- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 msgr2=0x7f0b3409d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:30.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 0x7f0b3409d7f0 secure :-1 s=READY pgs=1234 cs=0 l=1 rx=0x7f0b24054840 tx=0x7f0b240546f0).stop 2019-12-05T06:44:30.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 -- 172.21.15.16:0/2364333794 shutdown_connections 2019-12-05T06:44:30.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b3409e270 0x7f0b3406dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:30.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 --2- 172.21.15.16:0/2364333794 >> v2:172.21.15.76:3300/0 conn(0x7f0b3409e7c0 0x7f0b3409dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:30.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b18023630 0x7f0b18025ac0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:30.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 --2- 172.21.15.16:0/2364333794 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b3412ca70 0x7f0b3409d7f0 unknown :-1 s=CLOSED pgs=1234 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:30.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 -- 172.21.15.16:0/2364333794 shutdown_connections 2019-12-05T06:44:30.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 -- 172.21.15.16:0/2364333794 wait complete. 2019-12-05T06:44:30.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:30.334+0000 7f0b39b6c700 1 -- 172.21.15.16:0/2364333794 >> 172.21.15.16:0/2364333794 conn(0x7f0b340a2850 msgr2=0x7f0b3406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:30.351 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:44:32.351 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:32.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.630+0000 7fda594c8700 1 Processor -- start 2019-12-05T06:44:32.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.630+0000 7fda594c8700 1 -- start start 2019-12-05T06:44:32.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.632+0000 7fda594c8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fda5408c6d0 0x7fda5408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:32.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.632+0000 7fda594c8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda5408fef0 0x7fda5408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:32.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.632+0000 7fda594c8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda5408fb80 con 0x7fda5408ef20 2019-12-05T06:44:32.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.632+0000 7fda594c8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda5408df90 con 0x7fda5408fef0 2019-12-05T06:44:32.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.632+0000 7fda594c8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda540a3f70 con 0x7fda5408c6d0 2019-12-05T06:44:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.632+0000 7fda52ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fda5408fef0 0x7fda5408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.632+0000 7fda537fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fda5408c6d0 0x7fda5408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.632+0000 7fda53fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fda5408ef20 0x7fda5408f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45016/0 (socket says 172.21.15.16:45016) 2019-12-05T06:44:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.633+0000 7fda53fff700 1 -- 172.21.15.16:0/1121102515 learned_addr learned my addr 172.21.15.16:0/1121102515 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.633+0000 7fda537fe700 1 -- 172.21.15.16:0/1121102515 >> v2:172.21.15.16:3300/0 conn(0x7fda5408fef0 msgr2=0x7fda5408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:32.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.633+0000 7fda537fe700 1 --2- 172.21.15.16:0/1121102515 >> v2:172.21.15.16:3300/0 conn(0x7fda5408fef0 0x7fda5408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.633+0000 7fda537fe700 1 -- 172.21.15.16:0/1121102515 >> v1:172.21.15.16:6789/0 conn(0x7fda5408ef20 legacy=0x7fda5408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.633+0000 7fda527fc700 1 -- 172.21.15.16:0/1121102515 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3399506103 0 0) 0x7fda44001150 con 0x7fda5408ef20 2019-12-05T06:44:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.633+0000 7fda537fe700 1 -- 172.21.15.16:0/1121102515 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda5408d540 con 0x7fda5408c6d0 2019-12-05T06:44:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.633+0000 7fda52ffd700 1 --2- 172.21.15.16:0/1121102515 >> v2:172.21.15.16:3300/0 conn(0x7fda5408fef0 0x7fda5408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.634+0000 7fda537fe700 1 --2- 172.21.15.16:0/1121102515 >> v2:172.21.15.76:3300/0 conn(0x7fda5408c6d0 0x7fda5408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fda4c00b2d0 tx=0x7fda4c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.634+0000 7fda537fe700 1 --2- 172.21.15.16:0/1121102515 >> v2:172.21.15.76:3300/0 conn(0x7fda5408c6d0 0x7fda5408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fda4c00b2d0 tx=0x7fda4c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.634+0000 7fda537fe700 1 --2- 172.21.15.16:0/1121102515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda5408c6d0 0x7fda5408cac0 secure :-1 s=READY pgs=861 cs=0 l=1 rx=0x7fda4c00b2d0 tx=0x7fda4c00bae0).ready entity=mon.1 client_cookie=3c36577a3679b9f9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:32.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.635+0000 7fda537fe700 1 --2- 172.21.15.16:0/1121102515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda5408c6d0 0x7fda5408cac0 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7fda4c00b2d0 tx=0x7fda4c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.635+0000 7fda537fe700 1 --2- 172.21.15.16:0/1121102515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda5408c6d0 0x7fda5408cac0 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7fda4c00b2d0 tx=0x7fda4c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.635+0000 7fda527fc700 1 -- 172.21.15.16:0/1121102515 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda4c007590 con 0x7fda5408c6d0 2019-12-05T06:44:32.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.635+0000 7fda537fe700 1 --2- 172.21.15.16:0/1121102515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda5408c6d0 0x7fda5408cac0 secure :-1 s=THROTTLE_DONE pgs=861 cs=0 l=1 rx=0x7fda4c00b2d0 tx=0x7fda4c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.635+0000 7fda527fc700 1 -- 172.21.15.16:0/1121102515 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda4c009d30 con 0x7fda5408c6d0 2019-12-05T06:44:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.635+0000 7fda527fc700 1 -- 172.21.15.16:0/1121102515 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda4c00a400 con 0x7fda5408c6d0 2019-12-05T06:44:32.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.635+0000 7fda594c8700 1 -- 172.21.15.16:0/1121102515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda5408c6d0 msgr2=0x7fda5408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.635+0000 7fda594c8700 1 --2- 172.21.15.16:0/1121102515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda5408c6d0 0x7fda5408cac0 secure :-1 s=READY pgs=861 cs=0 l=1 rx=0x7fda4c00b2d0 tx=0x7fda4c00bae0).stop 2019-12-05T06:44:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.637+0000 7fda594c8700 1 -- 172.21.15.16:0/1121102515 shutdown_connections 2019-12-05T06:44:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.637+0000 7fda594c8700 1 --2- 172.21.15.16:0/1121102515 >> v2:172.21.15.16:3300/0 conn(0x7fda5408fef0 0x7fda5408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.637+0000 7fda594c8700 1 --2- 172.21.15.16:0/1121102515 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fda5408c6d0 0x7fda5408cac0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.637+0000 7fda594c8700 1 -- 172.21.15.16:0/1121102515 shutdown_connections 2019-12-05T06:44:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.637+0000 7fda594c8700 1 -- 172.21.15.16:0/1121102515 wait complete. 2019-12-05T06:44:32.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.637+0000 7fda594c8700 1 -- 172.21.15.16:0/1121102515 >> 172.21.15.16:0/1121102515 conn(0x7fda54061940 msgr2=0x7fda54096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:32.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.643+0000 7fda594c8700 1 Processor -- start 2019-12-05T06:44:32.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.643+0000 7fda594c8700 1 -- start start 2019-12-05T06:44:32.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.645+0000 7fda594c8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:32.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.645+0000 7fda594c8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda54128af0 con 0x7fda5406e840 2019-12-05T06:44:32.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.645+0000 7fda594c8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda54128600 con 0x7fda5406dc50 2019-12-05T06:44:32.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.645+0000 7fda594c8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda54132830 con 0x7fda54135450 2019-12-05T06:44:32.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.645+0000 7fda52ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fda5406e840 0x7fda54134480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45022/0 (socket says 172.21.15.16:45022) 2019-12-05T06:44:32.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.645+0000 7fda52ffd700 1 -- 172.21.15.16:0/2753526007 learned_addr learned my addr 172.21.15.16:0/2753526007 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:32.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.645+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:32.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.646+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946192495 0 0) 0x7fda3c00dd90 con 0x7fda5406e840 2019-12-05T06:44:32.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.646+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda38002e80 con 0x7fda5406e840 2019-12-05T06:44:32.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.646+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1088712558 0 0) 0x7fda4c00af30 con 0x7fda5406dc50 2019-12-05T06:44:32.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.646+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda38004170 con 0x7fda5406dc50 2019-12-05T06:44:32.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.646+0000 7fda53fff700 1 -- 172.21.15.16:0/2753526007 >> v1:172.21.15.76:6789/0 conn(0x7fda5406dc50 legacy=0x7fda5406e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:32.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.646+0000 7fda53fff700 1 -- 172.21.15.16:0/2753526007 >> v1:172.21.15.16:6789/0 conn(0x7fda5406e840 legacy=0x7fda54134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:32.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.646+0000 7fda53fff700 1 -- 172.21.15.16:0/2753526007 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda54133b80 con 0x7fda54135450 2019-12-05T06:44:32.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.646+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.647+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.647+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=READY pgs=292 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:32.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.647+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.647+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda44001570 con 0x7fda54135450 2019-12-05T06:44:32.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.647+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.647+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.647+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda44000940 con 0x7fda54135450 2019-12-05T06:44:32.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.647+0000 7fda594c8700 1 -- 172.21.15.16:0/2753526007 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda54133b80 con 0x7fda54135450 2019-12-05T06:44:32.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.647+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda44001b50 con 0x7fda54135450 2019-12-05T06:44:32.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.647+0000 7fda594c8700 1 -- 172.21.15.16:0/2753526007 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda54133b80 con 0x7fda54135450 2019-12-05T06:44:32.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.648+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.649+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fda44001190 con 0x7fda54135450 2019-12-05T06:44:32.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.649+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.649+0000 7fda4bfff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda38023650 0x7fda38025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:32.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.650+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7fda44045d70 con 0x7fda54135450 2019-12-05T06:44:32.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.650+0000 7fda537fe700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda38023650 0x7fda38025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:32.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.650+0000 7fda537fe700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda38023650 0x7fda38025ae0 crc :-1 s=READY pgs=527 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:32.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.662+0000 7fda594c8700 1 -- 172.21.15.16:0/2753526007 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda54133b80 con 0x7fda54135450 2019-12-05T06:44:32.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.671+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.671+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fda4402c610 con 0x7fda54135450 2019-12-05T06:44:32.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.943+0000 7fda594c8700 1 -- 172.21.15.16:0/2753526007 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda54133b80 con 0x7fda54135450 2019-12-05T06:44:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.945+0000 7fda53fff700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=THROTTLE_DONE pgs=292 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:32.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.945+0000 7fda4bfff700 1 -- 172.21.15.16:0/2753526007 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fda4402c610 con 0x7fda54135450 2019-12-05T06:44:32.947 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:32.947 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5744,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:28.456001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":33,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:32.948 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 152 2019-12-05T06:44:32.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.949+0000 7fda594c8700 1 -- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda38023650 msgr2=0x7fda38025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:32.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.949+0000 7fda594c8700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda38023650 0x7fda38025ae0 crc :-1 s=READY pgs=527 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:32.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.949+0000 7fda594c8700 1 -- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 msgr2=0x7fda54134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.949+0000 7fda594c8700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 secure :-1 s=READY pgs=292 cs=0 l=1 rx=0x7fda440103a0 tx=0x7fda44010740).stop 2019-12-05T06:44:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.949+0000 7fda594c8700 1 -- 172.21.15.16:0/2753526007 shutdown_connections 2019-12-05T06:44:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.949+0000 7fda594c8700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fda38023650 0x7fda38025ae0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.949+0000 7fda594c8700 1 --2- 172.21.15.16:0/2753526007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fda54135450 0x7fda54134c80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.949+0000 7fda594c8700 1 -- 172.21.15.16:0/2753526007 shutdown_connections 2019-12-05T06:44:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.949+0000 7fda594c8700 1 -- 172.21.15.16:0/2753526007 wait complete. 2019-12-05T06:44:32.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:32.949+0000 7fda594c8700 1 -- 172.21.15.16:0/2753526007 >> 172.21.15.16:0/2753526007 conn(0x7fda540a2850 msgr2=0x7fda5406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:32.961 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:44:34.962 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:35.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.221+0000 7f4073f1f700 1 Processor -- start 2019-12-05T06:44:35.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.221+0000 7f4073f1f700 1 -- start start 2019-12-05T06:44:35.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.222+0000 7f4073f1f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f406c08d2c0 0x7f406c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:35.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.222+0000 7f4073f1f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f406c08fef0 0x7f406c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:35.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.222+0000 7f4073f1f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f406c0958e0 con 0x7f406c08c6d0 2019-12-05T06:44:35.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.222+0000 7f4073f1f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406c08df90 con 0x7f406c08d2c0 2019-12-05T06:44:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.222+0000 7f4073f1f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c0a3f70 con 0x7f406c08fef0 2019-12-05T06:44:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.223+0000 7f4071cbb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f406c08d2c0 0x7f406c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.223+0000 7f4071cbb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f406c08d2c0 0x7f406c08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59864/0 (socket says 172.21.15.16:59864) 2019-12-05T06:44:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.223+0000 7f4071cbb700 1 -- 172.21.15.16:0/1155095922 learned_addr learned my addr 172.21.15.16:0/1155095922 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.223+0000 7f4072cbd700 1 --2- 172.21.15.16:0/1155095922 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f406c08fef0 0x7f406c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:35.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.223+0000 7f4071cbb700 1 -- 172.21.15.16:0/1155095922 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f406c08fef0 msgr2=0x7f406c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.223+0000 7f4071cbb700 1 --2- 172.21.15.16:0/1155095922 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f406c08fef0 0x7f406c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f4071cbb700 1 -- 172.21.15.16:0/1155095922 >> v1:172.21.15.16:6789/0 conn(0x7f406c08c6d0 legacy=0x7f406c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f40714ba700 1 -- 172.21.15.16:0/1155095922 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3225933199 0 0) 0x7f4064001480 con 0x7f406c08c6d0 2019-12-05T06:44:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f4071cbb700 1 -- 172.21.15.16:0/1155095922 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c08e620 con 0x7f406c08d2c0 2019-12-05T06:44:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f4071cbb700 1 --2- 172.21.15.16:0/1155095922 >> v2:172.21.15.16:3300/0 conn(0x7f406c08d2c0 0x7f406c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f405c00f6c0 tx=0x7f405c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f4071cbb700 1 --2- 172.21.15.16:0/1155095922 >> v2:172.21.15.16:3300/0 conn(0x7f406c08d2c0 0x7f406c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f405c00f6c0 tx=0x7f405c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f4071cbb700 1 --2- 172.21.15.16:0/1155095922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c08d2c0 0x7f406c08ef20 secure :-1 s=READY pgs=1238 cs=0 l=1 rx=0x7f405c00f6c0 tx=0x7f405c00fa60).ready entity=mon.0 client_cookie=c0725098204c69d4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:35.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f4071cbb700 1 --2- 172.21.15.16:0/1155095922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c08d2c0 0x7f406c08ef20 secure :-1 s=THROTTLE_DONE pgs=1238 cs=0 l=1 rx=0x7f405c00f6c0 tx=0x7f405c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f4071cbb700 1 --2- 172.21.15.16:0/1155095922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c08d2c0 0x7f406c08ef20 secure :-1 s=THROTTLE_DONE pgs=1238 cs=0 l=1 rx=0x7f405c00f6c0 tx=0x7f405c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f40714ba700 1 -- 172.21.15.16:0/1155095922 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405c000960 con 0x7f406c08d2c0 2019-12-05T06:44:35.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f4071cbb700 1 --2- 172.21.15.16:0/1155095922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c08d2c0 0x7f406c08ef20 secure :-1 s=THROTTLE_DONE pgs=1238 cs=0 l=1 rx=0x7f405c00f6c0 tx=0x7f405c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.224+0000 7f40714ba700 1 -- 172.21.15.16:0/1155095922 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405c000c90 con 0x7f406c08d2c0 2019-12-05T06:44:35.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.225+0000 7f40714ba700 1 -- 172.21.15.16:0/1155095922 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405c0013a0 con 0x7f406c08d2c0 2019-12-05T06:44:35.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.225+0000 7f4073f1f700 1 -- 172.21.15.16:0/1155095922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c08d2c0 msgr2=0x7f406c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.225+0000 7f4073f1f700 1 --2- 172.21.15.16:0/1155095922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c08d2c0 0x7f406c08ef20 secure :-1 s=READY pgs=1238 cs=0 l=1 rx=0x7f405c00f6c0 tx=0x7f405c00fa60).stop 2019-12-05T06:44:35.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.227+0000 7f4073f1f700 1 -- 172.21.15.16:0/1155095922 shutdown_connections 2019-12-05T06:44:35.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.227+0000 7f4073f1f700 1 --2- 172.21.15.16:0/1155095922 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c08d2c0 0x7f406c08ef20 unknown :-1 s=CLOSED pgs=1238 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.227+0000 7f4073f1f700 1 --2- 172.21.15.16:0/1155095922 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f406c08fef0 0x7f406c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:35.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.227+0000 7f4073f1f700 1 -- 172.21.15.16:0/1155095922 shutdown_connections 2019-12-05T06:44:35.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.227+0000 7f4073f1f700 1 -- 172.21.15.16:0/1155095922 wait complete. 2019-12-05T06:44:35.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.227+0000 7f4073f1f700 1 -- 172.21.15.16:0/1155095922 >> 172.21.15.16:0/1155095922 conn(0x7f406c061940 msgr2=0x7f406c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:35.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.231+0000 7f4073f1f700 1 Processor -- start 2019-12-05T06:44:35.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.231+0000 7f4073f1f700 1 -- start start 2019-12-05T06:44:35.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.232+0000 7f4073f1f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f406c14d400 0x7f406c14d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:35.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.232+0000 7f4073f1f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f406c132d00 0x7f406c131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:35.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.232+0000 7f4073f1f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f406c132270 0x7f406c06dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:35.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.232+0000 7f4073f1f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406c09bb50 con 0x7f406c132d00 2019-12-05T06:44:35.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.232+0000 7f4073f1f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f406c09aa60 con 0x7f406c14d400 2019-12-05T06:44:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.232+0000 7f4073f1f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f406c09a030 con 0x7f406c132270 2019-12-05T06:44:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.232+0000 7f4071cbb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f406c132d00 0x7f406c131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.233+0000 7f4071cbb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f406c132d00 0x7f406c131d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59872/0 (socket says 172.21.15.16:59872) 2019-12-05T06:44:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.233+0000 7f4071cbb700 1 -- 172.21.15.16:0/3860678090 learned_addr learned my addr 172.21.15.16:0/3860678090 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.233+0000 7f4072cbd700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f406c132270 0x7f406c06dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:35.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.233+0000 7f40724bc700 1 --2- 172.21.15.16:0/3860678090 >> v2:172.21.15.76:3300/0 conn(0x7f406c14d400 0x7f406c14d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:35.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.233+0000 7f4071cbb700 1 -- 172.21.15.16:0/3860678090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f406c132270 msgr2=0x7f406c06dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.233+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f406c132270 0x7f406c06dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.233+0000 7f4071cbb700 1 -- 172.21.15.16:0/3860678090 >> v2:172.21.15.76:3300/0 conn(0x7f406c14d400 msgr2=0x7f406c14d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.233+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> v2:172.21.15.76:3300/0 conn(0x7f406c14d400 0x7f406c14d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.233+0000 7f4071cbb700 1 -- 172.21.15.16:0/3860678090 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f406c09b0f0 con 0x7f406c132d00 2019-12-05T06:44:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.233+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> v2:172.21.15.16:3300/0 conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.234+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> v2:172.21.15.16:3300/0 conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.234+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=READY pgs=1239 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:35.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.234+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=THROTTLE_DONE pgs=1239 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.234+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=THROTTLE_DONE pgs=1239 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.234+0000 7f406affd700 1 -- 172.21.15.16:0/3860678090 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405c0078e0 con 0x7f406c132d00 2019-12-05T06:44:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.234+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=THROTTLE_DONE pgs=1239 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.234+0000 7f406affd700 1 -- 172.21.15.16:0/3860678090 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f405c000e40 con 0x7f406c132d00 2019-12-05T06:44:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.234+0000 7f4073f1f700 1 -- 172.21.15.16:0/3860678090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f406c09b0f0 con 0x7f406c132d00 2019-12-05T06:44:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.234+0000 7f406affd700 1 -- 172.21.15.16:0/3860678090 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f405c001510 con 0x7f406c132d00 2019-12-05T06:44:35.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.234+0000 7f4073f1f700 1 -- 172.21.15.16:0/3860678090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f406c09b0f0 con 0x7f406c132d00 2019-12-05T06:44:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.236+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=THROTTLE_DONE pgs=1239 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.237+0000 7f406affd700 1 -- 172.21.15.16:0/3860678090 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f405c0078e0 con 0x7f406c132d00 2019-12-05T06:44:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.237+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=THROTTLE_DONE pgs=1239 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.237+0000 7f406affd700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4050023630 0x7f4050025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.237+0000 7f406affd700 1 -- 172.21.15.16:0/3860678090 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f405c045d50 con 0x7f406c132d00 2019-12-05T06:44:35.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.237+0000 7f40724bc700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4050023630 0x7f4050025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:35.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.238+0000 7f40724bc700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4050023630 0x7f4050025ac0 crc :-1 s=READY pgs=528 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:35.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.253+0000 7f4073f1f700 1 -- 172.21.15.16:0/3860678090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f406c09b0f0 con 0x7f406c132d00 2019-12-05T06:44:35.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.259+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=THROTTLE_DONE pgs=1239 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.259+0000 7f406affd700 1 -- 172.21.15.16:0/3860678090 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f405c007580 con 0x7f406c132d00 2019-12-05T06:44:35.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.509+0000 7f4073f1f700 1 -- 172.21.15.16:0/3860678090 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f406c1329c0 con 0x7f406c132d00 2019-12-05T06:44:35.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.511+0000 7f4071cbb700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=THROTTLE_DONE pgs=1239 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:35.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.511+0000 7f406affd700 1 -- 172.21.15.16:0/3860678090 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f405c0078c0 con 0x7f406c132d00 2019-12-05T06:44:35.513 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 152 2019-12-05T06:44:35.513 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:35.513 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5744,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:28.456001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":33,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.514+0000 7f4073f1f700 1 -- 172.21.15.16:0/3860678090 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4050023630 msgr2=0x7f4050025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.514+0000 7f4073f1f700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4050023630 0x7f4050025ac0 crc :-1 s=READY pgs=528 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.514+0000 7f4073f1f700 1 -- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 msgr2=0x7f406c131d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:35.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.514+0000 7f4073f1f700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 0x7f406c131d30 secure :-1 s=READY pgs=1239 cs=0 l=1 rx=0x7f405c0103e0 tx=0x7f405c010290).stop 2019-12-05T06:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.515+0000 7f4073f1f700 1 -- 172.21.15.16:0/3860678090 shutdown_connections 2019-12-05T06:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.515+0000 7f4073f1f700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f406c132270 0x7f406c06dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.515+0000 7f4073f1f700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4050023630 0x7f4050025ac0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.515+0000 7f4073f1f700 1 --2- 172.21.15.16:0/3860678090 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f406c132d00 0x7f406c131d30 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.515+0000 7f4073f1f700 1 --2- 172.21.15.16:0/3860678090 >> v2:172.21.15.76:3300/0 conn(0x7f406c14d400 0x7f406c14d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:35.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.515+0000 7f4073f1f700 1 -- 172.21.15.16:0/3860678090 shutdown_connections 2019-12-05T06:44:35.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.515+0000 7f4073f1f700 1 -- 172.21.15.16:0/3860678090 wait complete. 2019-12-05T06:44:35.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:35.515+0000 7f4073f1f700 1 -- 172.21.15.16:0/3860678090 >> 172.21.15.16:0/3860678090 conn(0x7f406c0714f0 msgr2=0x7f406c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:35.532 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:44:37.533 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.815+0000 7f0130a05700 1 Processor -- start 2019-12-05T06:44:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.815+0000 7f0130a05700 1 -- start start 2019-12-05T06:44:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.817+0000 7f0130a05700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f012c08d2c0 0x7f012c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.817+0000 7f0130a05700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c08fef0 0x7f012c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.817+0000 7f0130a05700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f012c0958e0 con 0x7f012c08c6d0 2019-12-05T06:44:37.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.817+0000 7f0130a05700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f012c08df90 con 0x7f012c08d2c0 2019-12-05T06:44:37.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.817+0000 7f0130a05700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012c0a3f70 con 0x7f012c08fef0 2019-12-05T06:44:37.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.817+0000 7f012a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f012c08d2c0 0x7f012c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:37.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.817+0000 7f012a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f012c08d2c0 0x7f012c08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59878/0 (socket says 172.21.15.16:59878) 2019-12-05T06:44:37.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.817+0000 7f012a59c700 1 -- 172.21.15.16:0/246791791 learned_addr learned my addr 172.21.15.16:0/246791791 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:37.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.817+0000 7f012b59e700 1 --2- 172.21.15.16:0/246791791 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c08fef0 0x7f012c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:37.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.818+0000 7f012a59c700 1 -- 172.21.15.16:0/246791791 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c08fef0 msgr2=0x7f012c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:37.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.818+0000 7f012a59c700 1 --2- 172.21.15.16:0/246791791 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c08fef0 0x7f012c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:37.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.818+0000 7f012a59c700 1 -- 172.21.15.16:0/246791791 >> v1:172.21.15.76:6789/0 conn(0x7f012c08c6d0 legacy=0x7f012c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:37.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.818+0000 7f012a59c700 1 -- 172.21.15.16:0/246791791 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c08e620 con 0x7f012c08d2c0 2019-12-05T06:44:37.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.818+0000 7f012b59e700 1 --2- 172.21.15.16:0/246791791 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c08fef0 0x7f012c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:44:37.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.818+0000 7f012a59c700 1 --2- 172.21.15.16:0/246791791 >> v2:172.21.15.16:3300/0 conn(0x7f012c08d2c0 0x7f012c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f011400fac0 tx=0x7f011400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.818+0000 7f012a59c700 1 --2- 172.21.15.16:0/246791791 >> v2:172.21.15.16:3300/0 conn(0x7f012c08d2c0 0x7f012c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f011400fac0 tx=0x7f011400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.818+0000 7f012a59c700 1 --2- 172.21.15.16:0/246791791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c08d2c0 0x7f012c08ef20 secure :-1 s=READY pgs=1240 cs=0 l=1 rx=0x7f011400fac0 tx=0x7f011400fe60).ready entity=mon.0 client_cookie=fc207a43fff7f869 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:37.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.819+0000 7f012a59c700 1 --2- 172.21.15.16:0/246791791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c08d2c0 0x7f012c08ef20 secure :-1 s=THROTTLE_DONE pgs=1240 cs=0 l=1 rx=0x7f011400fac0 tx=0x7f011400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.819+0000 7f012a59c700 1 --2- 172.21.15.16:0/246791791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c08d2c0 0x7f012c08ef20 secure :-1 s=THROTTLE_DONE pgs=1240 cs=0 l=1 rx=0x7f011400fac0 tx=0x7f011400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.819+0000 7f0129d9b700 1 -- 172.21.15.16:0/246791791 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0114000ce0 con 0x7f012c08d2c0 2019-12-05T06:44:37.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.819+0000 7f012a59c700 1 --2- 172.21.15.16:0/246791791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c08d2c0 0x7f012c08ef20 secure :-1 s=THROTTLE_DONE pgs=1240 cs=0 l=1 rx=0x7f011400fac0 tx=0x7f011400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.819+0000 7f0129d9b700 1 -- 172.21.15.16:0/246791791 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0114001010 con 0x7f012c08d2c0 2019-12-05T06:44:37.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.819+0000 7f0129d9b700 1 -- 172.21.15.16:0/246791791 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0114001720 con 0x7f012c08d2c0 2019-12-05T06:44:37.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.819+0000 7f0130a05700 1 -- 172.21.15.16:0/246791791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c08d2c0 msgr2=0x7f012c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:37.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.819+0000 7f0130a05700 1 --2- 172.21.15.16:0/246791791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c08d2c0 0x7f012c08ef20 secure :-1 s=READY pgs=1240 cs=0 l=1 rx=0x7f011400fac0 tx=0x7f011400fe60).stop 2019-12-05T06:44:37.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.821+0000 7f0130a05700 1 -- 172.21.15.16:0/246791791 shutdown_connections 2019-12-05T06:44:37.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.821+0000 7f0130a05700 1 --2- 172.21.15.16:0/246791791 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c08fef0 0x7f012c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:37.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.821+0000 7f0130a05700 1 --2- 172.21.15.16:0/246791791 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c08d2c0 0x7f012c08ef20 unknown :-1 s=CLOSED pgs=1240 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:37.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.821+0000 7f0130a05700 1 -- 172.21.15.16:0/246791791 shutdown_connections 2019-12-05T06:44:37.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.821+0000 7f0130a05700 1 -- 172.21.15.16:0/246791791 wait complete. 2019-12-05T06:44:37.831 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.821+0000 7f0130a05700 1 -- 172.21.15.16:0/246791791 >> 172.21.15.16:0/246791791 conn(0x7f012c061940 msgr2=0x7f012c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:37.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.827+0000 7f0130a05700 1 Processor -- start 2019-12-05T06:44:37.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.827+0000 7f0130a05700 1 -- start start 2019-12-05T06:44:37.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.828+0000 7f0130a05700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c096e90 0x7f012c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:37.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.828+0000 7f0130a05700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f012c084f30 0x7f012c09b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.828+0000 7f0130a05700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f012c097280 0x7f012c097700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.828+0000 7f0130a05700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f012c099a20 con 0x7f012c097280 2019-12-05T06:44:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.828+0000 7f0130a05700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f012c099cf0 con 0x7f012c084f30 2019-12-05T06:44:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.828+0000 7f0130a05700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f012c09aae0 con 0x7f012c096e90 2019-12-05T06:44:37.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f012c097280 0x7f012c097700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f012c097280 0x7f012c097700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59886/0 (socket says 172.21.15.16:59886) 2019-12-05T06:44:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012b59e700 1 -- 172.21.15.16:0/520622315 learned_addr learned my addr 172.21.15.16:0/520622315 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012a59c700 1 --2- 172.21.15.16:0/520622315 >> v2:172.21.15.76:3300/0 conn(0x7f012c084f30 0x7f012c09b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012ad9d700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c096e90 0x7f012c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012b59e700 1 -- 172.21.15.16:0/520622315 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c096e90 msgr2=0x7f012c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c096e90 0x7f012c09b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:37.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012b59e700 1 -- 172.21.15.16:0/520622315 >> v2:172.21.15.76:3300/0 conn(0x7f012c084f30 msgr2=0x7f012c09b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> v2:172.21.15.76:3300/0 conn(0x7f012c084f30 0x7f012c09b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012b59e700 1 -- 172.21.15.16:0/520622315 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f012c084b90 con 0x7f012c097280 2019-12-05T06:44:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.829+0000 7f012a59c700 1 --2- 172.21.15.16:0/520622315 >> v2:172.21.15.76:3300/0 conn(0x7f012c084f30 0x7f012c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> v2:172.21.15.16:3300/0 conn(0x7f012c097280 0x7f012c097700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> v2:172.21.15.16:3300/0 conn(0x7f012c097280 0x7f012c097700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 0x7f012c097700 secure :-1 s=READY pgs=1241 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:37.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 0x7f012c097700 secure :-1 s=THROTTLE_DONE pgs=1241 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f01237fe700 1 -- 172.21.15.16:0/520622315 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f011c009bd0 con 0x7f012c097280 2019-12-05T06:44:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 0x7f012c097700 secure :-1 s=THROTTLE_DONE pgs=1241 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 0x7f012c097700 secure :-1 s=THROTTLE_DONE pgs=1241 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f01237fe700 1 -- 172.21.15.16:0/520622315 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f011c0097f0 con 0x7f012c097280 2019-12-05T06:44:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f01237fe700 1 -- 172.21.15.16:0/520622315 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f011c00a1b0 con 0x7f012c097280 2019-12-05T06:44:37.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f0130a05700 1 -- 172.21.15.16:0/520622315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f012c084b90 con 0x7f012c097280 2019-12-05T06:44:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.830+0000 7f0130a05700 1 -- 172.21.15.16:0/520622315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f012c084b90 con 0x7f012c097280 2019-12-05T06:44:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.831+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 0x7f012c097700 secure :-1 s=THROTTLE_DONE pgs=1241 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.832+0000 7f01237fe700 1 -- 172.21.15.16:0/520622315 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f011c000b10 con 0x7f012c097280 2019-12-05T06:44:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.832+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 0x7f012c097700 secure :-1 s=THROTTLE_DONE pgs=1241 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.832+0000 7f01237fe700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0110023630 0x7f0110025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:37.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.833+0000 7f01237fe700 1 -- 172.21.15.16:0/520622315 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 8095+0+0 (secure 0 0 0) 0x7f011c045d50 con 0x7f012c097280 2019-12-05T06:44:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.833+0000 7f012ad9d700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0110023630 0x7f0110025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:37.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.833+0000 7f012ad9d700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0110023630 0x7f0110025ac0 crc :-1 s=READY pgs=529 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:37.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.847+0000 7f0130a05700 1 -- 172.21.15.16:0/520622315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f012c0958e0 con 0x7f012c097280 2019-12-05T06:44:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.855+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 0x7f012c097700 secure :-1 s=THROTTLE_DONE pgs=1241 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:37.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:37.855+0000 7f01237fe700 1 -- 172.21.15.16:0/520622315 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f011c0095e0 con 0x7f012c097280 2019-12-05T06:44:38.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.107+0000 7f0130a05700 1 -- 172.21.15.16:0/520622315 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f012c0958e0 con 0x7f012c097280 2019-12-05T06:44:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.107+0000 7f012b59e700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 0x7f012c097700 secure :-1 s=THROTTLE_DONE pgs=1241 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.107+0000 7f01237fe700 1 -- 172.21.15.16:0/520622315 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 152 v152) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f011c001320 con 0x7f012c097280 2019-12-05T06:44:38.109 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 152 2019-12-05T06:44:38.109 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:38.110 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":152,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5744,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":152}],"filesystems":[{"mdsmap":{"epoch":151,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:43:28.456001+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7200},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7200":{"gid":7200,"name":"a","rank":0,"incarnation":148,"state":"up:active","state_seq":33,"addr":"172.21.15.16:6833/1298592917","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":1298592917},{"type":"v1","addr":"172.21.15.16:6833","nonce":1298592917}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.110+0000 7f0130a05700 1 -- 172.21.15.16:0/520622315 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0110023630 msgr2=0x7f0110025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.110+0000 7f0130a05700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0110023630 0x7f0110025ac0 crc :-1 s=READY pgs=529 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.110+0000 7f0130a05700 1 -- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 msgr2=0x7f012c097700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.110+0000 7f0130a05700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 0x7f012c097700 secure :-1 s=READY pgs=1241 cs=0 l=1 rx=0x7f011c010480 tx=0x7f011c010280).stop 2019-12-05T06:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.110+0000 7f0130a05700 1 -- 172.21.15.16:0/520622315 shutdown_connections 2019-12-05T06:44:38.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.110+0000 7f0130a05700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0110023630 0x7f0110025ac0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.110+0000 7f0130a05700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f012c097280 0x7f012c097700 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.110+0000 7f0130a05700 1 --2- 172.21.15.16:0/520622315 >> v2:172.21.15.76:3300/0 conn(0x7f012c084f30 0x7f012c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.110+0000 7f0130a05700 1 --2- 172.21.15.16:0/520622315 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f012c096e90 0x7f012c09b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.111+0000 7f0130a05700 1 -- 172.21.15.16:0/520622315 shutdown_connections 2019-12-05T06:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.111+0000 7f0130a05700 1 -- 172.21.15.16:0/520622315 wait complete. 2019-12-05T06:44:38.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:38.111+0000 7f0130a05700 1 -- 172.21.15.16:0/520622315 >> 172.21.15.16:0/520622315 conn(0x7f012c0714f0 msgr2=0x7f012c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:38.122 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:44:40.123 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.403+0000 7f22bb8e8700 1 Processor -- start 2019-12-05T06:44:40.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.403+0000 7f22bb8e8700 1 -- start start 2019-12-05T06:44:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.405+0000 7f22bb8e8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f22b408c6d0 0x7f22b408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.405+0000 7f22bb8e8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.405+0000 7f22bb8e8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b408ef20 0x7f22b408f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.405+0000 7f22bb8e8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b408df90 con 0x7f22b408ef20 2019-12-05T06:44:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.405+0000 7f22bb8e8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b40a3f70 con 0x7f22b408c6d0 2019-12-05T06:44:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.405+0000 7f22bb8e8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b408d830 con 0x7f22b408fef0 2019-12-05T06:44:40.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22ba686700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b408ef20 0x7f22b408f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22b9684700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22b9e85700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f22b408c6d0 0x7f22b408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22ba686700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b408ef20 0x7f22b408f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59898/0 (socket says 172.21.15.16:59898) 2019-12-05T06:44:40.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22b9684700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:35654/0 (socket says 172.21.15.16:35654) 2019-12-05T06:44:40.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22ba686700 1 -- 172.21.15.16:0/598278115 learned_addr learned my addr 172.21.15.16:0/598278115 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:40.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22b9684700 1 -- 172.21.15.16:0/598278115 >> v2:172.21.15.76:3300/0 conn(0x7f22b408c6d0 msgr2=0x7f22b408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22b9684700 1 --2- 172.21.15.16:0/598278115 >> v2:172.21.15.76:3300/0 conn(0x7f22b408c6d0 0x7f22b408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22b9684700 1 -- 172.21.15.16:0/598278115 >> v2:172.21.15.16:3300/0 conn(0x7f22b408ef20 msgr2=0x7f22b408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22b9684700 1 --2- 172.21.15.16:0/598278115 >> v2:172.21.15.16:3300/0 conn(0x7f22b408ef20 0x7f22b408f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22b9684700 1 -- 172.21.15.16:0/598278115 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b408d540 con 0x7f22b408fef0 2019-12-05T06:44:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22b9e85700 1 --2- 172.21.15.16:0/598278115 >> v2:172.21.15.76:3300/0 conn(0x7f22b408c6d0 0x7f22b408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.406+0000 7f22ba686700 1 --2- 172.21.15.16:0/598278115 >> v2:172.21.15.16:3300/0 conn(0x7f22b408ef20 0x7f22b408f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:40.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.407+0000 7f22b9684700 1 --2- 172.21.15.16:0/598278115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f22a800af50 tx=0x7f22a800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.407+0000 7f22b9684700 1 --2- 172.21.15.16:0/598278115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f22a800af50 tx=0x7f22a800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.407+0000 7f22b9684700 1 --2- 172.21.15.16:0/598278115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 secure :-1 s=READY pgs=293 cs=0 l=1 rx=0x7f22a800af50 tx=0x7f22a800b760).ready entity=mon.2 client_cookie=c174fe86ca13bd4e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.407+0000 7f22b9684700 1 --2- 172.21.15.16:0/598278115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7f22a800af50 tx=0x7f22a800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.407+0000 7f22b9684700 1 --2- 172.21.15.16:0/598278115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7f22a800af50 tx=0x7f22a800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.407+0000 7f22b9684700 1 --2- 172.21.15.16:0/598278115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 secure :-1 s=THROTTLE_DONE pgs=293 cs=0 l=1 rx=0x7f22a800af50 tx=0x7f22a800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.407+0000 7f22b8e83700 1 -- 172.21.15.16:0/598278115 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a8000960 con 0x7f22b408fef0 2019-12-05T06:44:40.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.407+0000 7f22b8e83700 1 -- 172.21.15.16:0/598278115 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a8009840 con 0x7f22b408fef0 2019-12-05T06:44:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.407+0000 7f22b8e83700 1 -- 172.21.15.16:0/598278115 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a8009ed0 con 0x7f22b408fef0 2019-12-05T06:44:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.407+0000 7f22bb8e8700 1 -- 172.21.15.16:0/598278115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 msgr2=0x7f22b408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.408+0000 7f22bb8e8700 1 --2- 172.21.15.16:0/598278115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 secure :-1 s=READY pgs=293 cs=0 l=1 rx=0x7f22a800af50 tx=0x7f22a800b760).stop 2019-12-05T06:44:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.409+0000 7f22bb8e8700 1 -- 172.21.15.16:0/598278115 shutdown_connections 2019-12-05T06:44:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.409+0000 7f22bb8e8700 1 --2- 172.21.15.16:0/598278115 >> v2:172.21.15.16:3300/0 conn(0x7f22b408ef20 0x7f22b408f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.409+0000 7f22bb8e8700 1 --2- 172.21.15.16:0/598278115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b408fef0 0x7f22b408d000 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.409+0000 7f22bb8e8700 1 --2- 172.21.15.16:0/598278115 >> v2:172.21.15.76:3300/0 conn(0x7f22b408c6d0 0x7f22b408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:40.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.410+0000 7f22bb8e8700 1 -- 172.21.15.16:0/598278115 shutdown_connections 2019-12-05T06:44:40.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.410+0000 7f22bb8e8700 1 -- 172.21.15.16:0/598278115 wait complete. 2019-12-05T06:44:40.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.410+0000 7f22bb8e8700 1 -- 172.21.15.16:0/598278115 >> 172.21.15.16:0/598278115 conn(0x7f22b4061940 msgr2=0x7f22b4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:40.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.415+0000 7f22bb8e8700 1 Processor -- start 2019-12-05T06:44:40.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.415+0000 7f22bb8e8700 1 -- start start 2019-12-05T06:44:40.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22bb8e8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f22b409b850 0x7f22b4131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:40.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22bb8e8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b4132d00 0x7f22b4132270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:40.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22bb8e8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:40.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22bb8e8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b409aa60 con 0x7f22b4132d00 2019-12-05T06:44:40.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22bb8e8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22b409a030 con 0x7f22b409b850 2019-12-05T06:44:40.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22bb8e8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22b409a300 con 0x7f22b41327b0 2019-12-05T06:44:40.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22b9684700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b4132d00 0x7f22b4132270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22b9684700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f22b4132d00 0x7f22b4132270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59904/0 (socket says 172.21.15.16:59904) 2019-12-05T06:44:40.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22b9684700 1 -- 172.21.15.16:0/1197382957 learned_addr learned my addr 172.21.15.16:0/1197382957 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:40.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22ba686700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.417+0000 7f22b9e85700 1 --2- 172.21.15.16:0/1197382957 >> v2:172.21.15.76:3300/0 conn(0x7f22b409b850 0x7f22b4131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 -- 172.21.15.16:0/1197382957 >> v2:172.21.15.76:3300/0 conn(0x7f22b409b850 msgr2=0x7f22b4131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> v2:172.21.15.76:3300/0 conn(0x7f22b409b850 0x7f22b4131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 -- 172.21.15.16:0/1197382957 >> v2:172.21.15.16:3300/0 conn(0x7f22b4132d00 msgr2=0x7f22b4132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> v2:172.21.15.16:3300/0 conn(0x7f22b4132d00 0x7f22b4132270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 -- 172.21.15.16:0/1197382957 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b409b0f0 con 0x7f22b41327b0 2019-12-05T06:44:40.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22b9e85700 1 --2- 172.21.15.16:0/1197382957 >> v2:172.21.15.76:3300/0 conn(0x7f22b409b850 0x7f22b4131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:40.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=READY pgs=294 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:40.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22b27fc700 1 -- 172.21.15.16:0/1197382957 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a40010e0 con 0x7f22b41327b0 2019-12-05T06:44:40.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22b27fc700 1 -- 172.21.15.16:0/1197382957 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22a4007d50 con 0x7f22b41327b0 2019-12-05T06:44:40.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22bb8e8700 1 -- 172.21.15.16:0/1197382957 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b409b0f0 con 0x7f22b41327b0 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.418+0000 7f22b27fc700 1 -- 172.21.15.16:0/1197382957 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22a40016c0 con 0x7f22b41327b0 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.419+0000 7f22bb8e8700 1 -- 172.21.15.16:0/1197382957 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b409b0f0 con 0x7f22b41327b0 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.419+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.421+0000 7f22b27fc700 1 -- 172.21.15.16:0/1197382957 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f22a4000ef0 con 0x7f22b41327b0 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.421+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.421+0000 7f22b27fc700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2294023630 0x7f2294025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:40.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.421+0000 7f22b27fc700 1 -- 172.21.15.16:0/1197382957 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f22a4089c90 con 0x7f22b41327b0 2019-12-05T06:44:40.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.421+0000 7f22b9e85700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2294023630 0x7f2294025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.422+0000 7f22b9e85700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2294023630 0x7f2294025ac0 crc :-1 s=READY pgs=530 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:40.436 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.434+0000 7f22bb8e8700 1 -- 172.21.15.16:0/1197382957 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22b409b0f0 con 0x7f22b41327b0 2019-12-05T06:44:40.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.441+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.441+0000 7f22b27fc700 1 -- 172.21.15.16:0/1197382957 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f22a40840b0 con 0x7f22b41327b0 2019-12-05T06:44:40.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.690+0000 7f22bb8e8700 1 -- 172.21.15.16:0/1197382957 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f22b409b0f0 con 0x7f22b41327b0 2019-12-05T06:44:40.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.690+0000 7f22ba686700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=THROTTLE_DONE pgs=294 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.690+0000 7f22b27fc700 1 -- 172.21.15.16:0/1197382957 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f22a40840b0 con 0x7f22b41327b0 2019-12-05T06:44:40.692 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 154 2019-12-05T06:44:40.692 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:40.692 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:40.675621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 -- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2294023630 msgr2=0x7f2294025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2294023630 0x7f2294025ac0 crc :-1 s=READY pgs=530 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 -- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 msgr2=0x7f22b406dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 secure :-1 s=READY pgs=294 cs=0 l=1 rx=0x7f22a4054390 tx=0x7f22a4054240).stop 2019-12-05T06:44:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 -- 172.21.15.16:0/1197382957 shutdown_connections 2019-12-05T06:44:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22b41327b0 0x7f22b406dc10 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 --2- 172.21.15.16:0/1197382957 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2294023630 0x7f2294025ac0 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 --2- 172.21.15.16:0/1197382957 >> v2:172.21.15.16:3300/0 conn(0x7f22b4132d00 0x7f22b4132270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 --2- 172.21.15.16:0/1197382957 >> v2:172.21.15.76:3300/0 conn(0x7f22b409b850 0x7f22b4131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 -- 172.21.15.16:0/1197382957 shutdown_connections 2019-12-05T06:44:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 -- 172.21.15.16:0/1197382957 wait complete. 2019-12-05T06:44:40.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.694+0000 7f22bb8e8700 1 -- 172.21.15.16:0/1197382957 >> 172.21.15.16:0/1197382957 conn(0x7f22b40714f0 msgr2=0x7f22b409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:40.705 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:44:40.705 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:44:40.706 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.930+0000 7f41b3dbe700 1 Processor -- start 2019-12-05T06:44:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.930+0000 7f41b3dbe700 1 -- start start 2019-12-05T06:44:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.932+0000 7f41b3dbe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac08c6d0 0x7f41ac08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.932+0000 7f41b3dbe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f41ac08f720 0x7f41ac08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.932+0000 7f41b3dbe700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac0958e0 con 0x7f41ac08fef0 2019-12-05T06:44:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.932+0000 7f41b3dbe700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ac0a3f70 con 0x7f41ac08f720 2019-12-05T06:44:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.932+0000 7f41b3dbe700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac08d830 con 0x7f41ac08c6d0 2019-12-05T06:44:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.932+0000 7f41b2b5c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f41ac08f720 0x7f41ac08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.932+0000 7f41b1b5a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41ac08fef0 0x7f41ac08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45080/0 (socket says 172.21.15.16:45080) 2019-12-05T06:44:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.932+0000 7f41b1b5a700 1 -- 172.21.15.16:0/2450812054 learned_addr learned my addr 172.21.15.16:0/2450812054 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.932+0000 7f41b235b700 1 --2- 172.21.15.16:0/2450812054 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac08c6d0 0x7f41ac08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.933+0000 7f41b1359700 1 -- 172.21.15.16:0/2450812054 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1015457127 0 0) 0x7f41a00014f0 con 0x7f41ac08fef0 2019-12-05T06:44:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.933+0000 7f41b1359700 1 -- 172.21.15.16:0/2450812054 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4194002e80 con 0x7f41ac08fef0 2019-12-05T06:44:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.933+0000 7f41b2b5c700 1 -- 172.21.15.16:0/2450812054 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac08c6d0 msgr2=0x7f41ac08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.933+0000 7f41b2b5c700 1 --2- 172.21.15.16:0/2450812054 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac08c6d0 0x7f41ac08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.933+0000 7f41b2b5c700 1 -- 172.21.15.16:0/2450812054 >> v1:172.21.15.16:6789/0 conn(0x7f41ac08fef0 legacy=0x7f41ac08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.933+0000 7f41b2b5c700 1 -- 172.21.15.16:0/2450812054 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac08fbc0 con 0x7f41ac08f720 2019-12-05T06:44:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.933+0000 7f41b2b5c700 1 --2- 172.21.15.16:0/2450812054 >> v2:172.21.15.16:3300/0 conn(0x7f41ac08f720 0x7f41ac08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f41a400af50 tx=0x7f41a400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.933+0000 7f41b2b5c700 1 --2- 172.21.15.16:0/2450812054 >> v2:172.21.15.16:3300/0 conn(0x7f41ac08f720 0x7f41ac08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f41a400af50 tx=0x7f41a400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.933+0000 7f41b2b5c700 1 --2- 172.21.15.16:0/2450812054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41ac08f720 0x7f41ac08d000 secure :-1 s=READY pgs=1245 cs=0 l=1 rx=0x7f41a400af50 tx=0x7f41a400b250).ready entity=mon.0 client_cookie=174e59c954986a4e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.934+0000 7f41b2b5c700 1 --2- 172.21.15.16:0/2450812054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41ac08f720 0x7f41ac08d000 secure :-1 s=THROTTLE_DONE pgs=1245 cs=0 l=1 rx=0x7f41a400af50 tx=0x7f41a400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.934+0000 7f41b1359700 1 -- 172.21.15.16:0/2450812054 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41a4009ba0 con 0x7f41ac08f720 2019-12-05T06:44:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.934+0000 7f41b2b5c700 1 --2- 172.21.15.16:0/2450812054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41ac08f720 0x7f41ac08d000 secure :-1 s=THROTTLE_DONE pgs=1245 cs=0 l=1 rx=0x7f41a400af50 tx=0x7f41a400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.934+0000 7f41b2b5c700 1 --2- 172.21.15.16:0/2450812054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41ac08f720 0x7f41ac08d000 secure :-1 s=THROTTLE_DONE pgs=1245 cs=0 l=1 rx=0x7f41a400af50 tx=0x7f41a400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.934+0000 7f41b1359700 1 -- 172.21.15.16:0/2450812054 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a40097c0 con 0x7f41ac08f720 2019-12-05T06:44:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.934+0000 7f41b1359700 1 -- 172.21.15.16:0/2450812054 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41a400a180 con 0x7f41ac08f720 2019-12-05T06:44:40.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.934+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2450812054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41ac08f720 msgr2=0x7f41ac08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.934+0000 7f41b3dbe700 1 --2- 172.21.15.16:0/2450812054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41ac08f720 0x7f41ac08d000 secure :-1 s=READY pgs=1245 cs=0 l=1 rx=0x7f41a400af50 tx=0x7f41a400b250).stop 2019-12-05T06:44:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.936+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2450812054 shutdown_connections 2019-12-05T06:44:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.936+0000 7f41b3dbe700 1 --2- 172.21.15.16:0/2450812054 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f41ac08f720 0x7f41ac08d000 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.936+0000 7f41b3dbe700 1 --2- 172.21.15.16:0/2450812054 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac08c6d0 0x7f41ac08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:40.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.936+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2450812054 shutdown_connections 2019-12-05T06:44:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.936+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2450812054 wait complete. 2019-12-05T06:44:40.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.936+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2450812054 >> 172.21.15.16:0/2450812054 conn(0x7f41ac061940 msgr2=0x7f41ac096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.941+0000 7f41b3dbe700 1 Processor -- start 2019-12-05T06:44:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.941+0000 7f41b3dbe700 1 -- start start 2019-12-05T06:44:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41b3dbe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f41ac096e90 0x7f41ac09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41b3dbe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:40.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41b3dbe700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41ac084b90 con 0x7f41ac097280 2019-12-05T06:44:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41b3dbe700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f41ac09a450 con 0x7f41ac096e90 2019-12-05T06:44:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41b3dbe700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f41ac099a20 con 0x7f41ac084f30 2019-12-05T06:44:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41b2b5c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f41ac097280 0x7f41ac0976c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45088/0 (socket says 172.21.15.16:45088) 2019-12-05T06:44:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41b2b5c700 1 -- 172.21.15.16:0/2937889356 learned_addr learned my addr 172.21.15.16:0/2937889356 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41b235b700 1 --2- 172.21.15.16:0/2937889356 >> v2:172.21.15.76:3300/0 conn(0x7f41ac096e90 0x7f41ac09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41aaffd700 1 -- 172.21.15.16:0/2937889356 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531356350 0 0) 0x7f41a4009580 con 0x7f41ac097280 2019-12-05T06:44:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.943+0000 7f41aaffd700 1 -- 172.21.15.16:0/2937889356 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f418c002e80 con 0x7f41ac097280 2019-12-05T06:44:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.944+0000 7f41b1b5a700 1 -- 172.21.15.16:0/2937889356 >> v2:172.21.15.76:3300/0 conn(0x7f41ac096e90 msgr2=0x7f41ac09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.944+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> v2:172.21.15.76:3300/0 conn(0x7f41ac096e90 0x7f41ac09b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.944+0000 7f41b1b5a700 1 -- 172.21.15.16:0/2937889356 >> v1:172.21.15.16:6789/0 conn(0x7f41ac097280 legacy=0x7f41ac0976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:40.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.944+0000 7f41b1b5a700 1 -- 172.21.15.16:0/2937889356 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41ac09aae0 con 0x7f41ac084f30 2019-12-05T06:44:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.944+0000 7f41b235b700 1 --2- 172.21.15.16:0/2937889356 >> v2:172.21.15.76:3300/0 conn(0x7f41ac096e90 0x7f41ac09b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:44:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.944+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.944+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.944+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=READY pgs=295 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.945+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.945+0000 7f41aaffd700 1 -- 172.21.15.16:0/2937889356 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41a0001010 con 0x7f41ac084f30 2019-12-05T06:44:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.945+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.945+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.945+0000 7f41aaffd700 1 -- 172.21.15.16:0/2937889356 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f41a00045c0 con 0x7f41ac084f30 2019-12-05T06:44:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.945+0000 7f41aaffd700 1 -- 172.21.15.16:0/2937889356 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f41a0004c90 con 0x7f41ac084f30 2019-12-05T06:44:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.945+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2937889356 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41ac09aae0 con 0x7f41ac084f30 2019-12-05T06:44:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.945+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2937889356 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41ac09aae0 con 0x7f41ac084f30 2019-12-05T06:44:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.946+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.946+0000 7f41aaffd700 1 -- 172.21.15.16:0/2937889356 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f41a00047b0 con 0x7f41ac084f30 2019-12-05T06:44:40.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.946+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.947+0000 7f41aaffd700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f418c023670 0x7f418c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:40.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.947+0000 7f41aaffd700 1 -- 172.21.15.16:0/2937889356 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f41a0046080 con 0x7f41ac084f30 2019-12-05T06:44:40.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.947+0000 7f41b235b700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f418c023670 0x7f418c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:40.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.948+0000 7f41b235b700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f418c023670 0x7f418c025ae0 crc :-1 s=READY pgs=531 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:40.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.961+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2937889356 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41ac09aae0 con 0x7f41ac084f30 2019-12-05T06:44:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.971+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:40.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:40.971+0000 7f41aaffd700 1 -- 172.21.15.16:0/2937889356 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f41a0000cc0 con 0x7f41ac084f30 2019-12-05T06:44:41.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.200+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2937889356 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41ac09aae0 con 0x7f41ac084f30 2019-12-05T06:44:41.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.202+0000 7f41b1b5a700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=THROTTLE_DONE pgs=295 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:41.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.202+0000 7f41aaffd700 1 -- 172.21.15.16:0/2937889356 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 154 v154) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f41a00047b0 con 0x7f41ac084f30 2019-12-05T06:44:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 154 2019-12-05T06:44:41.204 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:41.204 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":154,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":154,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:40.675621+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f418c023670 msgr2=0x7f418c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f418c023670 0x7f418c025ae0 crc :-1 s=READY pgs=531 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 msgr2=0x7f41ac09b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 secure :-1 s=READY pgs=295 cs=0 l=1 rx=0x7f41a0010700 tx=0x7f41a0010aa0).stop 2019-12-05T06:44:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2937889356 shutdown_connections 2019-12-05T06:44:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f418c023670 0x7f418c025ae0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 --2- 172.21.15.16:0/2937889356 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f41ac084f30 0x7f41ac09b920 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 --2- 172.21.15.16:0/2937889356 >> v2:172.21.15.76:3300/0 conn(0x7f41ac096e90 0x7f41ac09b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2937889356 shutdown_connections 2019-12-05T06:44:41.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2937889356 wait complete. 2019-12-05T06:44:41.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:41.206+0000 7f41b3dbe700 1 -- 172.21.15.16:0/2937889356 >> 172.21.15.16:0/2937889356 conn(0x7f41ac0714f0 msgr2=0x7f41ac09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:41.220 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 154, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 154, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5744": { "addr": "172.21.15.76:6835/3914386168", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 3914386168, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 3914386168, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5744, "incarnation": 153, "name": "b", "rank": 0, "state": "up:reconnect", "state_seq": 18 } }, "last_failure": 0, "last_failure_osd_epoch": 74, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:44:40.675621+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5744 } } } ], "standbys": [] } 2019-12-05T06:44:42.384 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:44:42.389 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:44:43.222 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.486+0000 7fdb1c6e0700 1 Processor -- start 2019-12-05T06:44:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.486+0000 7fdb1c6e0700 1 -- start start 2019-12-05T06:44:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.488+0000 7fdb1c6e0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 0x7fdb1408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.488+0000 7fdb1c6e0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdb1408fef0 0x7fdb1408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:43.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.488+0000 7fdb1c6e0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb140958e0 con 0x7fdb1408c6d0 2019-12-05T06:44:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.488+0000 7fdb1c6e0700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb1408df90 con 0x7fdb1408fef0 2019-12-05T06:44:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.488+0000 7fdb1c6e0700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb140a3f70 con 0x7fdb1408d2c0 2019-12-05T06:44:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.488+0000 7fdb1ac7d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdb1408c6d0 0x7fdb1408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45096/0 (socket says 172.21.15.16:45096) 2019-12-05T06:44:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.489+0000 7fdb1ac7d700 1 -- 172.21.15.16:0/3598773326 learned_addr learned my addr 172.21.15.16:0/3598773326 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.489+0000 7fdb1a47c700 1 --2- 172.21.15.16:0/3598773326 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 0x7fdb1408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.489+0000 7fdb1b47e700 1 --2- 172.21.15.16:0/3598773326 >> v2:172.21.15.76:3300/0 conn(0x7fdb1408fef0 0x7fdb1408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:43.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.489+0000 7fdb19c7b700 1 -- 172.21.15.16:0/3598773326 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994353161 0 0) 0x7fdb0c0014d0 con 0x7fdb1408c6d0 2019-12-05T06:44:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.489+0000 7fdb19c7b700 1 -- 172.21.15.16:0/3598773326 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdafc002e80 con 0x7fdb1408c6d0 2019-12-05T06:44:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.489+0000 7fdb1a47c700 1 -- 172.21.15.16:0/3598773326 >> v2:172.21.15.76:3300/0 conn(0x7fdb1408fef0 msgr2=0x7fdb1408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.489+0000 7fdb1a47c700 1 --2- 172.21.15.16:0/3598773326 >> v2:172.21.15.76:3300/0 conn(0x7fdb1408fef0 0x7fdb1408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.490+0000 7fdb1a47c700 1 -- 172.21.15.16:0/3598773326 >> v1:172.21.15.16:6789/0 conn(0x7fdb1408c6d0 legacy=0x7fdb1408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.490+0000 7fdb1a47c700 1 -- 172.21.15.16:0/3598773326 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1408e620 con 0x7fdb1408d2c0 2019-12-05T06:44:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.490+0000 7fdb1a47c700 1 --2- 172.21.15.16:0/3598773326 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 0x7fdb1408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdb0800af50 tx=0x7fdb0800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.490+0000 7fdb1a47c700 1 --2- 172.21.15.16:0/3598773326 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 0x7fdb1408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdb0800af50 tx=0x7fdb0800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.490+0000 7fdb1a47c700 1 --2- 172.21.15.16:0/3598773326 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 0x7fdb1408ef20 secure :-1 s=READY pgs=297 cs=0 l=1 rx=0x7fdb0800af50 tx=0x7fdb0800b250).ready entity=mon.2 client_cookie=bf43cc3b7b1e4d14 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.490+0000 7fdb1a47c700 1 --2- 172.21.15.16:0/3598773326 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 0x7fdb1408ef20 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7fdb0800af50 tx=0x7fdb0800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.490+0000 7fdb1a47c700 1 --2- 172.21.15.16:0/3598773326 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 0x7fdb1408ef20 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7fdb0800af50 tx=0x7fdb0800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.490+0000 7fdb19c7b700 1 -- 172.21.15.16:0/3598773326 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb08007210 con 0x7fdb1408d2c0 2019-12-05T06:44:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.490+0000 7fdb1a47c700 1 --2- 172.21.15.16:0/3598773326 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 0x7fdb1408ef20 secure :-1 s=THROTTLE_DONE pgs=297 cs=0 l=1 rx=0x7fdb0800af50 tx=0x7fdb0800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.491+0000 7fdb19c7b700 1 -- 172.21.15.16:0/3598773326 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb080099b0 con 0x7fdb1408d2c0 2019-12-05T06:44:43.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.491+0000 7fdb19c7b700 1 -- 172.21.15.16:0/3598773326 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb0800a080 con 0x7fdb1408d2c0 2019-12-05T06:44:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.491+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/3598773326 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 msgr2=0x7fdb1408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.491+0000 7fdb1c6e0700 1 --2- 172.21.15.16:0/3598773326 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 0x7fdb1408ef20 secure :-1 s=READY pgs=297 cs=0 l=1 rx=0x7fdb0800af50 tx=0x7fdb0800b250).stop 2019-12-05T06:44:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.493+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/3598773326 shutdown_connections 2019-12-05T06:44:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.493+0000 7fdb1c6e0700 1 --2- 172.21.15.16:0/3598773326 >> v2:172.21.15.76:3300/0 conn(0x7fdb1408fef0 0x7fdb1408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.493+0000 7fdb1c6e0700 1 --2- 172.21.15.16:0/3598773326 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdb1408d2c0 0x7fdb1408ef20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.493+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/3598773326 shutdown_connections 2019-12-05T06:44:43.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.493+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/3598773326 wait complete. 2019-12-05T06:44:43.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.493+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/3598773326 >> 172.21.15.16:0/3598773326 conn(0x7fdb14061940 msgr2=0x7fdb14096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:43.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.498+0000 7fdb1c6e0700 1 Processor -- start 2019-12-05T06:44:43.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.498+0000 7fdb1c6e0700 1 -- start start 2019-12-05T06:44:43.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.500+0000 7fdb1c6e0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdb141302f0 0x7fdb1412a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:43.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.500+0000 7fdb1c6e0700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb14131bf0 con 0x7fdb1412b640 2019-12-05T06:44:43.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.500+0000 7fdb1c6e0700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb141316f0 con 0x7fdb14153b50 2019-12-05T06:44:43.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.500+0000 7fdb1c6e0700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb141296e0 con 0x7fdb141302f0 2019-12-05T06:44:43.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.500+0000 7fdb1ac7d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdb141302f0 0x7fdb1412a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:43.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.500+0000 7fdb1ac7d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fdb141302f0 0x7fdb1412a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:59934/0 (socket says 172.21.15.16:59934) 2019-12-05T06:44:43.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.500+0000 7fdb1ac7d700 1 -- 172.21.15.16:0/2550056956 learned_addr learned my addr 172.21.15.16:0/2550056956 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.500+0000 7fdb137fe700 1 -- 172.21.15.16:0/2550056956 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2442825796 0 0) 0x7fdb04001080 con 0x7fdb14153b50 2019-12-05T06:44:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb137fe700 1 -- 172.21.15.16:0/2550056956 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdaf4002e80 con 0x7fdb14153b50 2019-12-05T06:44:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb1ac7d700 1 -- 172.21.15.16:0/2550056956 >> v1:172.21.15.76:6789/0 conn(0x7fdb14153b50 legacy=0x7fdb14155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb1ac7d700 1 -- 172.21.15.16:0/2550056956 >> v1:172.21.15.16:6789/0 conn(0x7fdb1412b640 legacy=0x7fdb1412abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb137fe700 1 -- 172.21.15.16:0/2550056956 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 266358911 0 0) 0x7fdb0800ac40 con 0x7fdb1412b640 2019-12-05T06:44:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb1ac7d700 1 -- 172.21.15.16:0/2550056956 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb14129d70 con 0x7fdb141302f0 2019-12-05T06:44:43.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb1ac7d700 1 --2- 172.21.15.16:0/2550056956 >> v2:172.21.15.16:3300/0 conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb1ac7d700 1 --2- 172.21.15.16:0/2550056956 >> v2:172.21.15.16:3300/0 conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb1ac7d700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=READY pgs=1249 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:43.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb1ac7d700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=THROTTLE_DONE pgs=1249 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb137fe700 1 -- 172.21.15.16:0/2550056956 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb0c001010 con 0x7fdb141302f0 2019-12-05T06:44:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb1ac7d700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=THROTTLE_DONE pgs=1249 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.501+0000 7fdb1ac7d700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=THROTTLE_DONE pgs=1249 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.502+0000 7fdb137fe700 1 -- 172.21.15.16:0/2550056956 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb0c001440 con 0x7fdb141302f0 2019-12-05T06:44:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.502+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/2550056956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb14129d70 con 0x7fdb141302f0 2019-12-05T06:44:43.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.502+0000 7fdb137fe700 1 -- 172.21.15.16:0/2550056956 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb0c001b10 con 0x7fdb141302f0 2019-12-05T06:44:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.502+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/2550056956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb14129d70 con 0x7fdb141302f0 2019-12-05T06:44:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.503+0000 7fdb1ac7d700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=THROTTLE_DONE pgs=1249 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.504+0000 7fdb137fe700 1 -- 172.21.15.16:0/2550056956 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdb0c001010 con 0x7fdb141302f0 2019-12-05T06:44:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.504+0000 7fdb1ac7d700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=THROTTLE_DONE pgs=1249 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.504+0000 7fdb137fe700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdaf4023680 0x7fdaf4025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.504+0000 7fdb137fe700 1 -- 172.21.15.16:0/2550056956 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7fdb0c0461d0 con 0x7fdb141302f0 2019-12-05T06:44:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.504+0000 7fdb1a47c700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdaf4023680 0x7fdaf4025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:43.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.505+0000 7fdb1a47c700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdaf4023680 0x7fdaf4025af0 crc :-1 s=READY pgs=532 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:43.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.518+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/2550056956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb14129d70 con 0x7fdb141302f0 2019-12-05T06:44:43.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.525+0000 7fdb1ac7d700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=THROTTLE_DONE pgs=1249 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.525+0000 7fdb137fe700 1 -- 172.21.15.16:0/2550056956 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdb0c007920 con 0x7fdb141302f0 2019-12-05T06:44:43.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.776+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/2550056956 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdb14129d70 con 0x7fdb141302f0 2019-12-05T06:44:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.776+0000 7fdb1ac7d700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=THROTTLE_DONE pgs=1249 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.776+0000 7fdb137fe700 1 -- 172.21.15.16:0/2550056956 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fdb0c001010 con 0x7fdb141302f0 2019-12-05T06:44:43.778 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 156 2019-12-05T06:44:43.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:43.778 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:43.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.779+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/2550056956 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdaf4023680 msgr2=0x7fdaf4025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:43.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.779+0000 7fdb1c6e0700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdaf4023680 0x7fdaf4025af0 crc :-1 s=READY pgs=532 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:43.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.779+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 msgr2=0x7fdb1412a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:43.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.779+0000 7fdb1c6e0700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 0x7fdb1412a670 secure :-1 s=READY pgs=1249 cs=0 l=1 rx=0x7fdb0c010760 tx=0x7fdb0c010b00).stop 2019-12-05T06:44:43.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.779+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/2550056956 shutdown_connections 2019-12-05T06:44:43.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.779+0000 7fdb1c6e0700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdaf4023680 0x7fdaf4025af0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:43.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.780+0000 7fdb1c6e0700 1 --2- 172.21.15.16:0/2550056956 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdb141302f0 0x7fdb1412a670 unknown :-1 s=CLOSED pgs=1249 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:43.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.780+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/2550056956 shutdown_connections 2019-12-05T06:44:43.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.780+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/2550056956 wait complete. 2019-12-05T06:44:43.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:43.780+0000 7fdb1c6e0700 1 -- 172.21.15.16:0/2550056956 >> 172.21.15.16:0/2550056956 conn(0x7fdb140a2850 msgr2=0x7fdb1406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:43.792 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=7200 2019-12-05T06:44:43.792 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 5 secs before reviving mds.a 2019-12-05T06:44:48.793 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:44:48.793 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:44:48.793 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:44:48.797 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:44:48.798 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:48.897 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:44:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.931+0000 7f54fd1d8700 1 Processor -- start 2019-12-05T06:44:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.931+0000 7f54fd1d8700 1 -- start start 2019-12-05T06:44:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54fd1d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54f808d2c0 0x7f54f808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54fd1d8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f54f808fef0 0x7f54f808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54fd1d8700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f80958e0 con 0x7f54f808c6d0 2019-12-05T06:44:48.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54fd1d8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f808df90 con 0x7f54f808d2c0 2019-12-05T06:44:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54fd1d8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f80a3f70 con 0x7f54f808fef0 2019-12-05T06:44:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54f808d2c0 0x7f54f808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54f808d2c0 0x7f54f808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60000/0 (socket says 172.21.15.16:60000) 2019-12-05T06:44:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f6d9d700 1 -- 172.21.15.16:0/3621652383 learned_addr learned my addr 172.21.15.16:0/3621652383 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54fc9d7700 1 --2- 172.21.15.16:0/3621652383 >> v2:172.21.15.76:3300/0 conn(0x7f54f808fef0 0x7f54f808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:48.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f6d9d700 1 -- 172.21.15.16:0/3621652383 >> v2:172.21.15.76:3300/0 conn(0x7f54f808fef0 msgr2=0x7f54f808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3621652383 >> v2:172.21.15.76:3300/0 conn(0x7f54f808fef0 0x7f54f808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f6d9d700 1 -- 172.21.15.16:0/3621652383 >> v1:172.21.15.16:6789/0 conn(0x7f54f808c6d0 legacy=0x7f54f808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f659c700 1 -- 172.21.15.16:0/3621652383 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649546214 0 0) 0x7f54e8001480 con 0x7f54f808c6d0 2019-12-05T06:44:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f6d9d700 1 -- 172.21.15.16:0/3621652383 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f808e620 con 0x7f54f808d2c0 2019-12-05T06:44:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3621652383 >> v2:172.21.15.16:3300/0 conn(0x7f54f808d2c0 0x7f54f808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f54ec00f6c0 tx=0x7f54ec00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3621652383 >> v2:172.21.15.16:3300/0 conn(0x7f54f808d2c0 0x7f54f808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f54ec00f6c0 tx=0x7f54ec00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.932+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3621652383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f808d2c0 0x7f54f808ef20 secure :-1 s=READY pgs=1255 cs=0 l=1 rx=0x7f54ec00f6c0 tx=0x7f54ec00fa60).ready entity=mon.0 client_cookie=4879ef1558982c02 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3621652383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f808d2c0 0x7f54f808ef20 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7f54ec00f6c0 tx=0x7f54ec00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54f659c700 1 -- 172.21.15.16:0/3621652383 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54ec000960 con 0x7f54f808d2c0 2019-12-05T06:44:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3621652383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f808d2c0 0x7f54f808ef20 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7f54ec00f6c0 tx=0x7f54ec00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54f659c700 1 -- 172.21.15.16:0/3621652383 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ec000960 con 0x7f54f808d2c0 2019-12-05T06:44:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3621652383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f808d2c0 0x7f54f808ef20 secure :-1 s=THROTTLE_DONE pgs=1255 cs=0 l=1 rx=0x7f54ec00f6c0 tx=0x7f54ec00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54f659c700 1 -- 172.21.15.16:0/3621652383 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54ec001070 con 0x7f54f808d2c0 2019-12-05T06:44:48.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3621652383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f808d2c0 msgr2=0x7f54f808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:48.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54fd1d8700 1 --2- 172.21.15.16:0/3621652383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f808d2c0 0x7f54f808ef20 secure :-1 s=READY pgs=1255 cs=0 l=1 rx=0x7f54ec00f6c0 tx=0x7f54ec00fa60).stop 2019-12-05T06:44:48.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3621652383 shutdown_connections 2019-12-05T06:44:48.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54fd1d8700 1 --2- 172.21.15.16:0/3621652383 >> v2:172.21.15.76:3300/0 conn(0x7f54f808fef0 0x7f54f808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:48.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.933+0000 7f54fd1d8700 1 --2- 172.21.15.16:0/3621652383 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f808d2c0 0x7f54f808ef20 unknown :-1 s=CLOSED pgs=1255 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:48.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.934+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3621652383 shutdown_connections 2019-12-05T06:44:48.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.934+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3621652383 wait complete. 2019-12-05T06:44:48.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.934+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3621652383 >> 172.21.15.16:0/3621652383 conn(0x7f54f8061940 msgr2=0x7f54f8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:48.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.936+0000 7f54fd1d8700 1 Processor -- start 2019-12-05T06:44:48.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.936+0000 7f54fd1d8700 1 -- start start 2019-12-05T06:44:48.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54fd1d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54f8132d00 0x7f54f814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:48.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54fd1d8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f54f8132530 0x7f54f8157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:48.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54fd1d8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54f8132980 con 0x7f54f814d400 2019-12-05T06:44:48.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54fd1d8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54f809bb50 con 0x7f54f8132d00 2019-12-05T06:44:48.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54fd1d8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54f809aa60 con 0x7f54f8132530 2019-12-05T06:44:48.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54f6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54f8132d00 0x7f54f814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:48.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54f6d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f54f8132d00 0x7f54f814d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60004/0 (socket says 172.21.15.16:60004) 2019-12-05T06:44:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54f6d9d700 1 -- 172.21.15.16:0/3153169382 learned_addr learned my addr 172.21.15.16:0/3153169382 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54fc9d7700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f54f8132530 0x7f54f8157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54f6d9d700 1 -- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f54f8132530 msgr2=0x7f54f8157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f54f8132530 0x7f54f8157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54f6d9d700 1 -- 172.21.15.16:0/3153169382 >> v1:172.21.15.76:6789/0 conn(0x7f54f814d400 legacy=0x7f54f8131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54f6d9d700 1 -- 172.21.15.16:0/3153169382 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f809b0f0 con 0x7f54f8132d00 2019-12-05T06:44:48.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54fc9d7700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f54f8132530 0x7f54f8157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:44:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> v2:172.21.15.16:3300/0 conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> v2:172.21.15.16:3300/0 conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.937+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=READY pgs=1256 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.938+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=THROTTLE_DONE pgs=1256 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.938+0000 7f54dffff700 1 -- 172.21.15.16:0/3153169382 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54ec0079c0 con 0x7f54f8132d00 2019-12-05T06:44:48.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.938+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=THROTTLE_DONE pgs=1256 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.938+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=THROTTLE_DONE pgs=1256 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.938+0000 7f54dffff700 1 -- 172.21.15.16:0/3153169382 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54ec000e40 con 0x7f54f8132d00 2019-12-05T06:44:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.938+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3153169382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54f809b0f0 con 0x7f54f8132d00 2019-12-05T06:44:48.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.938+0000 7f54dffff700 1 -- 172.21.15.16:0/3153169382 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54ec001510 con 0x7f54f8132d00 2019-12-05T06:44:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.938+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3153169382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54f809b0f0 con 0x7f54f8132d00 2019-12-05T06:44:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.938+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=THROTTLE_DONE pgs=1256 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.939+0000 7f54dffff700 1 -- 172.21.15.16:0/3153169382 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f54ec001030 con 0x7f54f8132d00 2019-12-05T06:44:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.939+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=THROTTLE_DONE pgs=1256 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.939+0000 7f54dffff700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f54d8023630 0x7f54d8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.939+0000 7f54dffff700 1 -- 172.21.15.16:0/3153169382 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f54ec045dd0 con 0x7f54f8132d00 2019-12-05T06:44:48.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.939+0000 7f54f759e700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f54d8023630 0x7f54d8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:48.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.940+0000 7f54f759e700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f54d8023630 0x7f54d8025ac0 crc :-1 s=READY pgs=533 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:48.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.947+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3153169382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54f809b0f0 con 0x7f54f8132d00 2019-12-05T06:44:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.952+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=THROTTLE_DONE pgs=1256 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:48.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:48.952+0000 7f54dffff700 1 -- 172.21.15.16:0/3153169382 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f54ec007580 con 0x7f54f8132d00 2019-12-05T06:44:49.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.166+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3153169382 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f54f809b0f0 con 0x7f54f8132d00 2019-12-05T06:44:49.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.166+0000 7f54f6d9d700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=THROTTLE_DONE pgs=1256 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:49.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.167+0000 7f54dffff700 1 -- 172.21.15.16:0/3153169382 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f54ec001000 con 0x7f54f8132d00 2019-12-05T06:44:49.168 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 156 2019-12-05T06:44:49.168 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:49.168 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.169+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f54d8023630 msgr2=0x7f54d8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.169+0000 7f54fd1d8700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f54d8023630 0x7f54d8025ac0 crc :-1 s=READY pgs=533 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:49.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.169+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 msgr2=0x7f54f814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.169+0000 7f54fd1d8700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 0x7f54f814d7f0 secure :-1 s=READY pgs=1256 cs=0 l=1 rx=0x7f54ec010460 tx=0x7f54ec010800).stop 2019-12-05T06:44:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.169+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3153169382 shutdown_connections 2019-12-05T06:44:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.169+0000 7f54fd1d8700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f54f8132530 0x7f54f8157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.169+0000 7f54fd1d8700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f54d8023630 0x7f54d8025ac0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:49.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.169+0000 7f54fd1d8700 1 --2- 172.21.15.16:0/3153169382 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f54f8132d00 0x7f54f814d7f0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.170+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3153169382 shutdown_connections 2019-12-05T06:44:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.170+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3153169382 wait complete. 2019-12-05T06:44:49.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:49.170+0000 7f54fd1d8700 1 -- 172.21.15.16:0/3153169382 >> 172.21.15.16:0/3153169382 conn(0x7f54f80714f0 msgr2=0x7f54f809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:49.181 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:44:51.182 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.441+0000 7fe762b6f700 1 Processor -- start 2019-12-05T06:44:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.441+0000 7fe762b6f700 1 -- start start 2019-12-05T06:44:51.458 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe762b6f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe75c08c6d0 0x7fe75c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe762b6f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c08fef0 0x7fe75c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe762b6f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe75c08ef20 0x7fe75c08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe762b6f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75c08df90 con 0x7fe75c08c6d0 2019-12-05T06:44:51.459 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe762b6f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe75c0a3f70 con 0x7fe75c08ef20 2019-12-05T06:44:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe762b6f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c08d830 con 0x7fe75c08fef0 2019-12-05T06:44:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe76110c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe75c08c6d0 0x7fe75c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe76110c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe75c08c6d0 0x7fe75c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60028/0 (socket says 172.21.15.16:60028) 2019-12-05T06:44:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe76190d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe75c08ef20 0x7fe75c08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe76110c700 1 -- 172.21.15.16:0/2998935838 learned_addr learned my addr 172.21.15.16:0/2998935838 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:51.460 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.443+0000 7fe76090b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c08fef0 0x7fe75c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.444+0000 7fe76110c700 1 -- 172.21.15.16:0/2998935838 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c08fef0 msgr2=0x7fe75c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.444+0000 7fe76110c700 1 --2- 172.21.15.16:0/2998935838 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c08fef0 0x7fe75c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.444+0000 7fe76110c700 1 -- 172.21.15.16:0/2998935838 >> v2:172.21.15.76:3300/0 conn(0x7fe75c08ef20 msgr2=0x7fe75c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.444+0000 7fe76110c700 1 --2- 172.21.15.16:0/2998935838 >> v2:172.21.15.76:3300/0 conn(0x7fe75c08ef20 0x7fe75c08f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.444+0000 7fe76110c700 1 -- 172.21.15.16:0/2998935838 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c08d540 con 0x7fe75c08c6d0 2019-12-05T06:44:51.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.444+0000 7fe76110c700 1 --2- 172.21.15.16:0/2998935838 >> v2:172.21.15.16:3300/0 conn(0x7fe75c08c6d0 0x7fe75c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe75400f6c0 tx=0x7fe75400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.444+0000 7fe76110c700 1 --2- 172.21.15.16:0/2998935838 >> v2:172.21.15.16:3300/0 conn(0x7fe75c08c6d0 0x7fe75c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe75400f6c0 tx=0x7fe75400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.444+0000 7fe76110c700 1 --2- 172.21.15.16:0/2998935838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75c08c6d0 0x7fe75c08cac0 secure :-1 s=READY pgs=1257 cs=0 l=1 rx=0x7fe75400f6c0 tx=0x7fe75400f570).ready entity=mon.0 client_cookie=6f08273d44718a6a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.445+0000 7fe76110c700 1 --2- 172.21.15.16:0/2998935838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75c08c6d0 0x7fe75c08cac0 secure :-1 s=THROTTLE_DONE pgs=1257 cs=0 l=1 rx=0x7fe75400f6c0 tx=0x7fe75400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.445+0000 7fe76110c700 1 --2- 172.21.15.16:0/2998935838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75c08c6d0 0x7fe75c08cac0 secure :-1 s=THROTTLE_DONE pgs=1257 cs=0 l=1 rx=0x7fe75400f6c0 tx=0x7fe75400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.445+0000 7fe76110c700 1 --2- 172.21.15.16:0/2998935838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75c08c6d0 0x7fe75c08cac0 secure :-1 s=THROTTLE_DONE pgs=1257 cs=0 l=1 rx=0x7fe75400f6c0 tx=0x7fe75400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.445+0000 7fe75bfff700 1 -- 172.21.15.16:0/2998935838 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe754000960 con 0x7fe75c08c6d0 2019-12-05T06:44:51.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.445+0000 7fe75bfff700 1 -- 172.21.15.16:0/2998935838 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe754000c90 con 0x7fe75c08c6d0 2019-12-05T06:44:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.445+0000 7fe75bfff700 1 -- 172.21.15.16:0/2998935838 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe754001320 con 0x7fe75c08c6d0 2019-12-05T06:44:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.445+0000 7fe762b6f700 1 -- 172.21.15.16:0/2998935838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75c08c6d0 msgr2=0x7fe75c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.445+0000 7fe762b6f700 1 --2- 172.21.15.16:0/2998935838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75c08c6d0 0x7fe75c08cac0 secure :-1 s=READY pgs=1257 cs=0 l=1 rx=0x7fe75400f6c0 tx=0x7fe75400f570).stop 2019-12-05T06:44:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.447+0000 7fe762b6f700 1 -- 172.21.15.16:0/2998935838 shutdown_connections 2019-12-05T06:44:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.447+0000 7fe762b6f700 1 --2- 172.21.15.16:0/2998935838 >> v2:172.21.15.76:3300/0 conn(0x7fe75c08ef20 0x7fe75c08f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.447+0000 7fe762b6f700 1 --2- 172.21.15.16:0/2998935838 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c08fef0 0x7fe75c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:44:51.463 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.447+0000 7fe762b6f700 1 --2- 172.21.15.16:0/2998935838 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe75c08c6d0 0x7fe75c08cac0 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.447+0000 7fe762b6f700 1 -- 172.21.15.16:0/2998935838 shutdown_connections 2019-12-05T06:44:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.448+0000 7fe762b6f700 1 -- 172.21.15.16:0/2998935838 wait complete. 2019-12-05T06:44:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.448+0000 7fe762b6f700 1 -- 172.21.15.16:0/2998935838 >> 172.21.15.16:0/2998935838 conn(0x7fe75c061940 msgr2=0x7fe75c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.452+0000 7fe762b6f700 1 Processor -- start 2019-12-05T06:44:51.464 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.452+0000 7fe762b6f700 1 -- start start 2019-12-05T06:44:51.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.454+0000 7fe762b6f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:51.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.454+0000 7fe762b6f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c084b90 con 0x7fe75c096e90 2019-12-05T06:44:51.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.454+0000 7fe762b6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe75c098790 con 0x7fe75c09bbe0 2019-12-05T06:44:51.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.454+0000 7fe762b6f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe75c09a450 con 0x7fe75c084f30 2019-12-05T06:44:51.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.454+0000 7fe76110c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe75c096e90 0x7fe75c09b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45202/0 (socket says 172.21.15.16:45202) 2019-12-05T06:44:51.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.454+0000 7fe76110c700 1 -- 172.21.15.16:0/1150508919 learned_addr learned my addr 172.21.15.16:0/1150508919 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:44:51.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.454+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:51.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.455+0000 7fe759ffb700 1 -- 172.21.15.16:0/1150508919 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298043902 0 0) 0x7fe754007910 con 0x7fe75c096e90 2019-12-05T06:44:51.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.455+0000 7fe759ffb700 1 -- 172.21.15.16:0/1150508919 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe73c002e80 con 0x7fe75c096e90 2019-12-05T06:44:51.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.455+0000 7fe759ffb700 1 -- 172.21.15.16:0/1150508919 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1494307011 0 0) 0x7fe7500012b0 con 0x7fe75c09bbe0 2019-12-05T06:44:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.455+0000 7fe76190d700 1 -- 172.21.15.16:0/1150508919 >> v1:172.21.15.76:6789/0 conn(0x7fe75c09bbe0 legacy=0x7fe75c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.455+0000 7fe76190d700 1 -- 172.21.15.16:0/1150508919 >> v1:172.21.15.16:6789/0 conn(0x7fe75c096e90 legacy=0x7fe75c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.455+0000 7fe76190d700 1 -- 172.21.15.16:0/1150508919 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe75c09aae0 con 0x7fe75c084f30 2019-12-05T06:44:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.455+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.455+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.455+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=READY pgs=298 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:51.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.455+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.456+0000 7fe759ffb700 1 -- 172.21.15.16:0/1150508919 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe74c009990 con 0x7fe75c084f30 2019-12-05T06:44:51.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.456+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.456+0000 7fe759ffb700 1 -- 172.21.15.16:0/1150508919 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe74c0095b0 con 0x7fe75c084f30 2019-12-05T06:44:51.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.456+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.456+0000 7fe762b6f700 1 -- 172.21.15.16:0/1150508919 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe75c09aae0 con 0x7fe75c084f30 2019-12-05T06:44:51.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.456+0000 7fe759ffb700 1 -- 172.21.15.16:0/1150508919 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe74c009b80 con 0x7fe75c084f30 2019-12-05T06:44:51.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.456+0000 7fe762b6f700 1 -- 172.21.15.16:0/1150508919 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe75c09aae0 con 0x7fe75c084f30 2019-12-05T06:44:51.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.457+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.458+0000 7fe759ffb700 1 -- 172.21.15.16:0/1150508919 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe74c0097a0 con 0x7fe75c084f30 2019-12-05T06:44:51.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.458+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.458+0000 7fe759ffb700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe73c023650 0x7fe73c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:51.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.458+0000 7fe759ffb700 1 -- 172.21.15.16:0/1150508919 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7fe74c047fd0 con 0x7fe75c084f30 2019-12-05T06:44:51.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.458+0000 7fe76110c700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe73c023650 0x7fe73c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:51.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.459+0000 7fe76110c700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe73c023650 0x7fe73c025ae0 crc :-1 s=READY pgs=534 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:51.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.473+0000 7fe762b6f700 1 -- 172.21.15.16:0/1150508919 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe75c09aae0 con 0x7fe75c084f30 2019-12-05T06:44:51.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.482+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.482+0000 7fe759ffb700 1 -- 172.21.15.16:0/1150508919 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe74c015180 con 0x7fe75c084f30 2019-12-05T06:44:51.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.751+0000 7fe762b6f700 1 -- 172.21.15.16:0/1150508919 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe75c09aae0 con 0x7fe75c084f30 2019-12-05T06:44:51.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.752+0000 7fe76190d700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=THROTTLE_DONE pgs=298 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:51.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.752+0000 7fe759ffb700 1 -- 172.21.15.16:0/1150508919 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 156 v156) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fe74c015180 con 0x7fe75c084f30 2019-12-05T06:44:51.753 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 156 2019-12-05T06:44:51.753 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:51.753 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":156,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.755+0000 7fe762b6f700 1 -- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe73c023650 msgr2=0x7fe73c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:51.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.755+0000 7fe762b6f700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe73c023650 0x7fe73c025ae0 crc :-1 s=READY pgs=534 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.755+0000 7fe762b6f700 1 -- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 msgr2=0x7fe75c157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.755+0000 7fe762b6f700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 secure :-1 s=READY pgs=298 cs=0 l=1 rx=0x7fe74c012560 tx=0x7fe74c012900).stop 2019-12-05T06:44:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.755+0000 7fe762b6f700 1 -- 172.21.15.16:0/1150508919 shutdown_connections 2019-12-05T06:44:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.755+0000 7fe762b6f700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe73c023650 0x7fe73c025ae0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.755+0000 7fe762b6f700 1 --2- 172.21.15.16:0/1150508919 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe75c084f30 0x7fe75c157200 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:51.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.755+0000 7fe762b6f700 1 -- 172.21.15.16:0/1150508919 shutdown_connections 2019-12-05T06:44:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.755+0000 7fe762b6f700 1 -- 172.21.15.16:0/1150508919 wait complete. 2019-12-05T06:44:51.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:51.755+0000 7fe762b6f700 1 -- 172.21.15.16:0/1150508919 >> 172.21.15.16:0/1150508919 conn(0x7fe75c0714f0 msgr2=0x7fe75c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:51.765 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:44:53.767 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:44:53.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.916+0000 7f6ef475a700 1 Processor -- start 2019-12-05T06:44:53.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.916+0000 7f6ef475a700 1 -- start start 2019-12-05T06:44:53.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.917+0000 7f6ef475a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6eec08c6d0 0x7f6eec08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:44:53.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.917+0000 7f6ef475a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec08fb80 con 0x7f6eec08f720 2019-12-05T06:44:53.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.917+0000 7f6ef475a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec0958e0 con 0x7f6eec08fef0 2019-12-05T06:44:53.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.917+0000 7f6ef475a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eec08df90 con 0x7f6eec08c6d0 2019-12-05T06:44:53.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.917+0000 7f6ef2cf7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6eec08c6d0 0x7f6eec08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:53.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.917+0000 7f6ef2cf7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6eec08c6d0 0x7f6eec08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60064/0 (socket says 172.21.15.16:60064) 2019-12-05T06:44:53.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.917+0000 7f6ef2cf7700 1 -- 172.21.15.16:0/68643639 learned_addr learned my addr 172.21.15.16:0/68643639 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:53.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.917+0000 7f6ef34f8700 1 --1- 172.21.15.16:0/68643639 >> v1:172.21.15.16:6789/0 conn(0x7f6eec08f720 0x7f6eec1533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45234/0 (socket says 172.21.15.16:45234) 2019-12-05T06:44:53.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef2cf7700 1 -- 172.21.15.16:0/68643639 >> v1:172.21.15.76:6789/0 conn(0x7f6eec08fef0 legacy=0x7f6eec08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:44:53.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef2cf7700 1 -- 172.21.15.16:0/68643639 >> v1:172.21.15.16:6789/0 conn(0x7f6eec08f720 legacy=0x7f6eec1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:53.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef2cf7700 1 -- 172.21.15.16:0/68643639 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec08e620 con 0x7f6eec08c6d0 2019-12-05T06:44:53.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef1cf5700 1 -- 172.21.15.16:0/68643639 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1794279506 0 0) 0x7f6ee40009b0 con 0x7f6eec08f720 2019-12-05T06:44:53.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/68643639 >> v2:172.21.15.16:3300/0 conn(0x7f6eec08c6d0 0x7f6eec08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6edc00f740 tx=0x7f6edc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/68643639 >> v2:172.21.15.16:3300/0 conn(0x7f6eec08c6d0 0x7f6eec08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6edc00f740 tx=0x7f6edc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/68643639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec08c6d0 0x7f6eec08cac0 secure :-1 s=READY pgs=1260 cs=0 l=1 rx=0x7f6edc00f740 tx=0x7f6edc00fae0).ready entity=mon.0 client_cookie=d59775523ed4cea server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:53.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/68643639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec08c6d0 0x7f6eec08cac0 secure :-1 s=THROTTLE_DONE pgs=1260 cs=0 l=1 rx=0x7f6edc00f740 tx=0x7f6edc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef1cf5700 1 -- 172.21.15.16:0/68643639 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6edc000960 con 0x7f6eec08c6d0 2019-12-05T06:44:53.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/68643639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec08c6d0 0x7f6eec08cac0 secure :-1 s=THROTTLE_DONE pgs=1260 cs=0 l=1 rx=0x7f6edc00f740 tx=0x7f6edc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/68643639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec08c6d0 0x7f6eec08cac0 secure :-1 s=THROTTLE_DONE pgs=1260 cs=0 l=1 rx=0x7f6edc00f740 tx=0x7f6edc00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef1cf5700 1 -- 172.21.15.16:0/68643639 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6edc000c90 con 0x7f6eec08c6d0 2019-12-05T06:44:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef1cf5700 1 -- 172.21.15.16:0/68643639 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6edc001360 con 0x7f6eec08c6d0 2019-12-05T06:44:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef475a700 1 -- 172.21.15.16:0/68643639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec08c6d0 msgr2=0x7f6eec08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.918+0000 7f6ef475a700 1 --2- 172.21.15.16:0/68643639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec08c6d0 0x7f6eec08cac0 secure :-1 s=READY pgs=1260 cs=0 l=1 rx=0x7f6edc00f740 tx=0x7f6edc00fae0).stop 2019-12-05T06:44:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.919+0000 7f6ef475a700 1 -- 172.21.15.16:0/68643639 shutdown_connections 2019-12-05T06:44:53.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.919+0000 7f6ef475a700 1 --2- 172.21.15.16:0/68643639 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec08c6d0 0x7f6eec08cac0 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.919+0000 7f6ef475a700 1 -- 172.21.15.16:0/68643639 shutdown_connections 2019-12-05T06:44:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.919+0000 7f6ef475a700 1 -- 172.21.15.16:0/68643639 wait complete. 2019-12-05T06:44:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.919+0000 7f6ef475a700 1 -- 172.21.15.16:0/68643639 >> 172.21.15.16:0/68643639 conn(0x7f6eec061940 msgr2=0x7f6eec096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.921+0000 7f6ef475a700 1 Processor -- start 2019-12-05T06:44:53.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.921+0000 7f6ef475a700 1 -- start start 2019-12-05T06:44:53.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef475a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6eec09bbe0 0x7f6eec1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:53.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef475a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6eec084f30 0x7f6eec155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:53.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef475a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6eec084b90 con 0x7f6eec096e90 2019-12-05T06:44:53.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef475a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6eec09a450 con 0x7f6eec09bbe0 2019-12-05T06:44:53.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef475a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6eec099a20 con 0x7f6eec084f30 2019-12-05T06:44:53.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef24f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6eec09bbe0 0x7f6eec1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:53.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef24f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6eec09bbe0 0x7f6eec1539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60072/0 (socket says 172.21.15.16:60072) 2019-12-05T06:44:53.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef2cf7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6eec096e90 0x7f6eec09b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45238/0 (socket says 172.21.15.16:45238) 2019-12-05T06:44:53.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef24f6700 1 -- 172.21.15.16:0/3021898902 learned_addr learned my addr 172.21.15.16:0/3021898902 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:44:53.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef34f8700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6eec084f30 0x7f6eec155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:53.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef24f6700 1 -- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6eec084f30 msgr2=0x7f6eec155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:53.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6eec084f30 0x7f6eec155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:53.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef24f6700 1 -- 172.21.15.16:0/3021898902 >> v1:172.21.15.16:6789/0 conn(0x7f6eec096e90 legacy=0x7f6eec09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:53.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef24f6700 1 -- 172.21.15.16:0/3021898902 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6eec09aae0 con 0x7f6eec09bbe0 2019-12-05T06:44:53.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1167747738 0 0) 0x7f6edc0077e0 con 0x7f6eec096e90 2019-12-05T06:44:53.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef34f8700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6eec084f30 0x7f6eec155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:44:53.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> v2:172.21.15.16:3300/0 conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> v2:172.21.15.16:3300/0 conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.922+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:53.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.923+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=THROTTLE_DONE pgs=1262 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.923+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ee0001420 con 0x7f6eec09bbe0 2019-12-05T06:44:53.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.923+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=THROTTLE_DONE pgs=1262 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.923+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=THROTTLE_DONE pgs=1262 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.923+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ee0000f90 con 0x7f6eec09bbe0 2019-12-05T06:44:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.923+0000 7f6ef475a700 1 -- 172.21.15.16:0/3021898902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6eec09aae0 con 0x7f6eec09bbe0 2019-12-05T06:44:53.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.923+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ee0001a00 con 0x7f6eec09bbe0 2019-12-05T06:44:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.923+0000 7f6ef475a700 1 -- 172.21.15.16:0/3021898902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6eec09aae0 con 0x7f6eec09bbe0 2019-12-05T06:44:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.923+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=THROTTLE_DONE pgs=1262 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.924+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6ee0001230 con 0x7f6eec09bbe0 2019-12-05T06:44:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.924+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=THROTTLE_DONE pgs=1262 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.924+0000 7f6eeb7fe700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ecc023630 0x7f6ecc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.924+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f6ee00461a0 con 0x7f6eec09bbe0 2019-12-05T06:44:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.924+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ecc023630 0x7f6ecc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:53.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.924+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ecc023630 0x7f6ecc025ac0 crc :-1 s=READY pgs=536 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:53.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.932+0000 7f6ef475a700 1 -- 172.21.15.16:0/3021898902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6eec09aae0 con 0x7f6eec09bbe0 2019-12-05T06:44:53.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.936+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=THROTTLE_DONE pgs=1262 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:53.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:53.936+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6ee00027a0 con 0x7f6eec09bbe0 2019-12-05T06:44:54.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6ef475a700 1 -- 172.21.15.16:0/3021898902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6eec09aae0 con 0x7f6eec09bbe0 2019-12-05T06:44:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6ef24f6700 1 -- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 msgr2=0x7f6eec1539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:44:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6ef24f6700 1 -- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 msgr2=0x7f6eec1539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:44:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:44:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 secure :-1 s=READY pgs=1262 cs=0 l=1 rx=0x7f6ee0010790 tx=0x7f6ee0010b30).stop 2019-12-05T06:44:54.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 msgr2=0x7f6eec1539e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:44:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6eeb7fe700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6eeb7fe700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ecc02bfe0 0x7f6ecc02e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6eeb7fe700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc02ed10 0x7f6ecc0311e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6eeb7fe700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:44:54.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc034660 con 0x7f6ecc0319e0 2019-12-05T06:44:54.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc034890 con 0x7f6ecc02bfe0 2019-12-05T06:44:54.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ecc034ac0 con 0x7f6ecc02ed10 2019-12-05T06:44:54.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6ef24f6700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ecc02bfe0 0x7f6ecc02e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.131+0000 7f6ef34f8700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc02ed10 0x7f6ecc0311e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:44:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 -- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc02ed10 msgr2=0x7f6ecc0311e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6ecc02ed10 0x7f6ecc0311e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 -- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ecc02bfe0 msgr2=0x7f6ecc02e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6ecc02bfe0 0x7f6ecc02e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 -- 172.21.15.16:0/3021898902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7f6ecc034db0 con 0x7f6ecc0319e0 2019-12-05T06:44:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 -- 172.21.15.16:0/3021898902 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6edc0119f0 con 0x7f6ecc0319e0 2019-12-05T06:44:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6edc011c20 tx=0x7f6edc0118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6edc011c20 tx=0x7f6edc0118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 secure :-1 s=READY pgs=1263 cs=0 l=1 rx=0x7f6edc011c20 tx=0x7f6edc0118f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:44:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 secure :-1 s=THROTTLE_DONE pgs=1263 cs=0 l=1 rx=0x7f6edc011c20 tx=0x7f6edc0118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6edc008020 con 0x7f6ecc0319e0 2019-12-05T06:44:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 secure :-1 s=THROTTLE_DONE pgs=1263 cs=0 l=1 rx=0x7f6edc011c20 tx=0x7f6edc0118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.132+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6edc008020 con 0x7f6ecc0319e0 2019-12-05T06:44:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.133+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 secure :-1 s=THROTTLE_DONE pgs=1263 cs=0 l=1 rx=0x7f6edc011c20 tx=0x7f6edc0118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.134+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6edc000b50 con 0x7f6ecc0319e0 2019-12-05T06:44:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.134+0000 7f6ef2cf7700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 secure :-1 s=THROTTLE_DONE pgs=1263 cs=0 l=1 rx=0x7f6edc011c20 tx=0x7f6edc0118f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:44:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.134+0000 7f6eeb7fe700 1 -- 172.21.15.16:0/3021898902 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f6edc047c90 con 0x7f6ecc0319e0 2019-12-05T06:44:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 157 2019-12-05T06:44:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef475a700 1 -- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ecc023630 msgr2=0x7f6ecc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef475a700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6ecc023630 0x7f6ecc025ac0 crc :-1 s=READY pgs=536 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef34f8700 1 -- 172.21.15.16:0/3021898902 reap_dead start 2019-12-05T06:44:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef475a700 1 -- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 msgr2=0x7f6ecc033eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:44:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef475a700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 secure :-1 s=READY pgs=1263 cs=0 l=1 rx=0x7f6edc011c20 tx=0x7f6edc0118f0).stop 2019-12-05T06:44:54.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef475a700 1 -- 172.21.15.16:0/3021898902 shutdown_connections 2019-12-05T06:44:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef475a700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6ecc0319e0 0x7f6ecc033eb0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef475a700 1 --2- 172.21.15.16:0/3021898902 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6eec09bbe0 0x7f6eec1539e0 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:44:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef475a700 1 -- 172.21.15.16:0/3021898902 shutdown_connections 2019-12-05T06:44:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef475a700 1 -- 172.21.15.16:0/3021898902 wait complete. 2019-12-05T06:44:54.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:44:54.141+0000 7f6ef475a700 1 -- 172.21.15.16:0/3021898902 >> 172.21.15.16:0/3021898902 conn(0x7f6eec0714f0 msgr2=0x7f6eec09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:44:54.143 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:44:54.144 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:44:54.154 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:44:54.154 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 36 secs before thrashing 2019-12-05T06:45:12.566 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:45:12.571 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:45:30.155 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.416+0000 7f21a1502700 1 Processor -- start 2019-12-05T06:45:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.416+0000 7f21a1502700 1 -- start start 2019-12-05T06:45:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.418+0000 7f21a1502700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f219c08c6d0 0x7f219c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:30.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.418+0000 7f21a1502700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f219c08f720 0x7f219c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.418+0000 7f21a1502700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c08fb80 con 0x7f219c08fef0 2019-12-05T06:45:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.418+0000 7f21a1502700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219c08df90 con 0x7f219c08c6d0 2019-12-05T06:45:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.418+0000 7f21a1502700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219c0a3f70 con 0x7f219c08f720 2019-12-05T06:45:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.418+0000 7f219b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f219c08c6d0 0x7f219c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.418+0000 7f219b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f219c08c6d0 0x7f219c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60236/0 (socket says 172.21.15.16:60236) 2019-12-05T06:45:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.418+0000 7f219b7fe700 1 -- 172.21.15.16:0/3737449778 learned_addr learned my addr 172.21.15.16:0/3737449778 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:30.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.418+0000 7f219bfff700 1 --2- 172.21.15.16:0/3737449778 >> v2:172.21.15.76:3300/0 conn(0x7f219c08f720 0x7f219c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:30.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.419+0000 7f219b7fe700 1 -- 172.21.15.16:0/3737449778 >> v2:172.21.15.76:3300/0 conn(0x7f219c08f720 msgr2=0x7f219c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:30.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.419+0000 7f219b7fe700 1 --2- 172.21.15.16:0/3737449778 >> v2:172.21.15.76:3300/0 conn(0x7f219c08f720 0x7f219c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:30.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.419+0000 7f219b7fe700 1 -- 172.21.15.16:0/3737449778 >> v1:172.21.15.76:6789/0 conn(0x7f219c08fef0 legacy=0x7f219c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:30.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.419+0000 7f219b7fe700 1 -- 172.21.15.16:0/3737449778 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c08d540 con 0x7f219c08c6d0 2019-12-05T06:45:30.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.419+0000 7f219a7fc700 1 -- 172.21.15.16:0/3737449778 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3260573285 0 0) 0x7f2184001080 con 0x7f219c08fef0 2019-12-05T06:45:30.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.419+0000 7f219b7fe700 1 --2- 172.21.15.16:0/3737449778 >> v2:172.21.15.16:3300/0 conn(0x7f219c08c6d0 0x7f219c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f219400fa40 tx=0x7f219400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.419+0000 7f219b7fe700 1 --2- 172.21.15.16:0/3737449778 >> v2:172.21.15.16:3300/0 conn(0x7f219c08c6d0 0x7f219c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f219400fa40 tx=0x7f219400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.419+0000 7f219b7fe700 1 --2- 172.21.15.16:0/3737449778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c08c6d0 0x7f219c08cac0 secure :-1 s=READY pgs=1265 cs=0 l=1 rx=0x7f219400fa40 tx=0x7f219400fde0).ready entity=mon.0 client_cookie=56584cdb24dba778 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:30.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.419+0000 7f219b7fe700 1 --2- 172.21.15.16:0/3737449778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c08c6d0 0x7f219c08cac0 secure :-1 s=THROTTLE_DONE pgs=1265 cs=0 l=1 rx=0x7f219400fa40 tx=0x7f219400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.420+0000 7f219b7fe700 1 --2- 172.21.15.16:0/3737449778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c08c6d0 0x7f219c08cac0 secure :-1 s=THROTTLE_DONE pgs=1265 cs=0 l=1 rx=0x7f219400fa40 tx=0x7f219400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.420+0000 7f219a7fc700 1 -- 172.21.15.16:0/3737449778 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2194000ce0 con 0x7f219c08c6d0 2019-12-05T06:45:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.420+0000 7f219b7fe700 1 --2- 172.21.15.16:0/3737449778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c08c6d0 0x7f219c08cac0 secure :-1 s=THROTTLE_DONE pgs=1265 cs=0 l=1 rx=0x7f219400fa40 tx=0x7f219400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.420+0000 7f219a7fc700 1 -- 172.21.15.16:0/3737449778 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2194001010 con 0x7f219c08c6d0 2019-12-05T06:45:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.420+0000 7f219a7fc700 1 -- 172.21.15.16:0/3737449778 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2194001720 con 0x7f219c08c6d0 2019-12-05T06:45:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.420+0000 7f21a1502700 1 -- 172.21.15.16:0/3737449778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c08c6d0 msgr2=0x7f219c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.420+0000 7f21a1502700 1 --2- 172.21.15.16:0/3737449778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c08c6d0 0x7f219c08cac0 secure :-1 s=READY pgs=1265 cs=0 l=1 rx=0x7f219400fa40 tx=0x7f219400fde0).stop 2019-12-05T06:45:30.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.422+0000 7f21a1502700 1 -- 172.21.15.16:0/3737449778 shutdown_connections 2019-12-05T06:45:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.422+0000 7f21a1502700 1 --2- 172.21.15.16:0/3737449778 >> v2:172.21.15.76:3300/0 conn(0x7f219c08f720 0x7f219c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.422+0000 7f21a1502700 1 --2- 172.21.15.16:0/3737449778 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c08c6d0 0x7f219c08cac0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.422+0000 7f21a1502700 1 -- 172.21.15.16:0/3737449778 shutdown_connections 2019-12-05T06:45:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.422+0000 7f21a1502700 1 -- 172.21.15.16:0/3737449778 wait complete. 2019-12-05T06:45:30.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.422+0000 7f21a1502700 1 -- 172.21.15.16:0/3737449778 >> 172.21.15.16:0/3737449778 conn(0x7f219c061940 msgr2=0x7f219c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:30.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.426+0000 7f21a1502700 1 Processor -- start 2019-12-05T06:45:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.426+0000 7f21a1502700 1 -- start start 2019-12-05T06:45:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.427+0000 7f21a1502700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f219c09cf40 0x7f219c09c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:30.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.427+0000 7f21a1502700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f219c14d840 0x7f219c1571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.427+0000 7f21a1502700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f219c14dca0 con 0x7f219c14d450 2019-12-05T06:45:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.427+0000 7f21a1502700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f219c138b80 con 0x7f219c14d840 2019-12-05T06:45:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.427+0000 7f21a1502700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f219c09afe0 con 0x7f219c09cf40 2019-12-05T06:45:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f219c14d840 0x7f219c1571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f219c14d840 0x7f219c1571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60246/0 (socket says 172.21.15.16:60246) 2019-12-05T06:45:30.439 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219bfff700 1 -- 172.21.15.16:0/916179761 learned_addr learned my addr 172.21.15.16:0/916179761 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219affd700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f219c09cf40 0x7f219c09c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219bfff700 1 -- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f219c09cf40 msgr2=0x7f219c09c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f219c09cf40 0x7f219c09c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219bfff700 1 -- 172.21.15.16:0/916179761 >> v1:172.21.15.76:6789/0 conn(0x7f219c14d450 legacy=0x7f219c09bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:30.440 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219bfff700 1 -- 172.21.15.16:0/916179761 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f219c09b670 con 0x7f219c14d840 2019-12-05T06:45:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219affd700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f219c09cf40 0x7f219c09c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:45:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> v2:172.21.15.16:3300/0 conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> v2:172.21.15.16:3300/0 conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.441 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.428+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.429+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=THROTTLE_DONE pgs=1266 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.429+0000 7f2193fff700 1 -- 172.21.15.16:0/916179761 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f218c000b10 con 0x7f219c14d840 2019-12-05T06:45:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.429+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=THROTTLE_DONE pgs=1266 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.429+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=THROTTLE_DONE pgs=1266 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.429+0000 7f2193fff700 1 -- 172.21.15.16:0/916179761 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f218c00ef40 con 0x7f219c14d840 2019-12-05T06:45:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.429+0000 7f2193fff700 1 -- 172.21.15.16:0/916179761 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f218c00f610 con 0x7f219c14d840 2019-12-05T06:45:30.442 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.429+0000 7f21a1502700 1 -- 172.21.15.16:0/916179761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f219c09b670 con 0x7f219c14d840 2019-12-05T06:45:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.429+0000 7f21a1502700 1 -- 172.21.15.16:0/916179761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f219c09b670 con 0x7f219c14d840 2019-12-05T06:45:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.430+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=THROTTLE_DONE pgs=1266 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.431+0000 7f2193fff700 1 -- 172.21.15.16:0/916179761 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f218c000a70 con 0x7f219c14d840 2019-12-05T06:45:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.431+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=THROTTLE_DONE pgs=1266 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.431+0000 7f2193fff700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2180023630 0x7f2180025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.431+0000 7f2193fff700 1 -- 172.21.15.16:0/916179761 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f218c045df0 con 0x7f219c14d840 2019-12-05T06:45:30.443 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.432+0000 7f219b7fe700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2180023630 0x7f2180025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.432+0000 7f21a1502700 1 -- 172.21.15.16:0/916179761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f219c09b670 con 0x7f219c14d840 2019-12-05T06:45:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.432+0000 7f219b7fe700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2180023630 0x7f2180025ac0 crc :-1 s=READY pgs=538 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:30.444 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.441+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=THROTTLE_DONE pgs=1266 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.445 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.441+0000 7f2193fff700 1 -- 172.21.15.16:0/916179761 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f218c00ea80 con 0x7f219c14d840 2019-12-05T06:45:30.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.690+0000 7f21a1502700 1 -- 172.21.15.16:0/916179761 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f219c09b670 con 0x7f219c14d840 2019-12-05T06:45:30.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.690+0000 7f219bfff700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=THROTTLE_DONE pgs=1266 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.690+0000 7f2193fff700 1 -- 172.21.15.16:0/916179761 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f218c000a70 con 0x7f219c14d840 2019-12-05T06:45:30.694 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 157 2019-12-05T06:45:30.694 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:30.694 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.693+0000 7f21a1502700 1 -- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2180023630 msgr2=0x7f2180025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.693+0000 7f21a1502700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2180023630 0x7f2180025ac0 crc :-1 s=READY pgs=538 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.693+0000 7f21a1502700 1 -- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 msgr2=0x7f219c1571d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.693+0000 7f21a1502700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 0x7f219c1571d0 secure :-1 s=READY pgs=1266 cs=0 l=1 rx=0x7f218c010450 tx=0x7f218c0107f0).stop 2019-12-05T06:45:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.693+0000 7f21a1502700 1 -- 172.21.15.16:0/916179761 shutdown_connections 2019-12-05T06:45:30.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.693+0000 7f21a1502700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2180023630 0x7f2180025ac0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:30.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.693+0000 7f21a1502700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f219c14d840 0x7f219c1571d0 unknown :-1 s=CLOSED pgs=1266 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:30.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.693+0000 7f21a1502700 1 --2- 172.21.15.16:0/916179761 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f219c09cf40 0x7f219c09c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:30.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.694+0000 7f21a1502700 1 -- 172.21.15.16:0/916179761 shutdown_connections 2019-12-05T06:45:30.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.694+0000 7f21a1502700 1 -- 172.21.15.16:0/916179761 wait complete. 2019-12-05T06:45:30.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.694+0000 7f21a1502700 1 -- 172.21.15.16:0/916179761 >> 172.21.15.16:0/916179761 conn(0x7f219c07aeb0 msgr2=0x7f219c0a2040 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:30.706 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:45:30.706 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:45:30.706 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:45:30.722 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:45:30.722 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:30.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.909+0000 7fb38c2b2700 1 Processor -- start 2019-12-05T06:45:30.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.909+0000 7fb38c2b2700 1 -- start start 2019-12-05T06:45:30.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.910+0000 7fb38c2b2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb38408d2c0 0x7fb38408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:30.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.910+0000 7fb38c2b2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb38408fef0 0x7fb38408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:30.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.910+0000 7fb38c2b2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3840958e0 con 0x7fb38408c6d0 2019-12-05T06:45:30.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.910+0000 7fb38c2b2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb38408df90 con 0x7fb38408fef0 2019-12-05T06:45:30.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.910+0000 7fb38c2b2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3840a3f70 con 0x7fb38408d2c0 2019-12-05T06:45:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.911+0000 7fb38b050700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb38408fef0 0x7fb38408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.911+0000 7fb38a84f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb38408c6d0 0x7fb38408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45420/0 (socket says 172.21.15.16:45420) 2019-12-05T06:45:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.911+0000 7fb38a84f700 1 -- 172.21.15.16:0/1675730616 learned_addr learned my addr 172.21.15.16:0/1675730616 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.911+0000 7fb38a04e700 1 --2- 172.21.15.16:0/1675730616 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb38408d2c0 0x7fb38408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.911+0000 7fb38b050700 1 -- 172.21.15.16:0/1675730616 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb38408d2c0 msgr2=0x7fb38408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.911+0000 7fb38b050700 1 --2- 172.21.15.16:0/1675730616 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb38408d2c0 0x7fb38408ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:30.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.911+0000 7fb38b050700 1 -- 172.21.15.16:0/1675730616 >> v1:172.21.15.16:6789/0 conn(0x7fb38408c6d0 legacy=0x7fb38408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:30.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.911+0000 7fb38984d700 1 -- 172.21.15.16:0/1675730616 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120703446 0 0) 0x7fb37c0014d0 con 0x7fb38408c6d0 2019-12-05T06:45:30.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.911+0000 7fb38b050700 1 -- 172.21.15.16:0/1675730616 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb38408e620 con 0x7fb38408fef0 2019-12-05T06:45:30.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38b050700 1 --2- 172.21.15.16:0/1675730616 >> v2:172.21.15.16:3300/0 conn(0x7fb38408fef0 0x7fb38408f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb37400af50 tx=0x7fb37400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38b050700 1 --2- 172.21.15.16:0/1675730616 >> v2:172.21.15.16:3300/0 conn(0x7fb38408fef0 0x7fb38408f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb37400af50 tx=0x7fb37400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38b050700 1 --2- 172.21.15.16:0/1675730616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38408fef0 0x7fb38408f460 secure :-1 s=READY pgs=1268 cs=0 l=1 rx=0x7fb37400af50 tx=0x7fb37400b6e0).ready entity=mon.0 client_cookie=1c48bc9c658af967 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:30.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38b050700 1 --2- 172.21.15.16:0/1675730616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38408fef0 0x7fb38408f460 secure :-1 s=THROTTLE_DONE pgs=1268 cs=0 l=1 rx=0x7fb37400af50 tx=0x7fb37400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38984d700 1 -- 172.21.15.16:0/1675730616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb374009ba0 con 0x7fb38408fef0 2019-12-05T06:45:30.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38b050700 1 --2- 172.21.15.16:0/1675730616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38408fef0 0x7fb38408f460 secure :-1 s=THROTTLE_DONE pgs=1268 cs=0 l=1 rx=0x7fb37400af50 tx=0x7fb37400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38984d700 1 -- 172.21.15.16:0/1675730616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3740097c0 con 0x7fb38408fef0 2019-12-05T06:45:30.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38b050700 1 --2- 172.21.15.16:0/1675730616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38408fef0 0x7fb38408f460 secure :-1 s=THROTTLE_DONE pgs=1268 cs=0 l=1 rx=0x7fb37400af50 tx=0x7fb37400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1675730616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38408fef0 msgr2=0x7fb38408f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:30.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38984d700 1 -- 172.21.15.16:0/1675730616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb374009ba0 con 0x7fb38408fef0 2019-12-05T06:45:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.912+0000 7fb38c2b2700 1 --2- 172.21.15.16:0/1675730616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38408fef0 0x7fb38408f460 secure :-1 s=READY pgs=1268 cs=0 l=1 rx=0x7fb37400af50 tx=0x7fb37400b6e0).stop 2019-12-05T06:45:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.915+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1675730616 shutdown_connections 2019-12-05T06:45:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.915+0000 7fb38c2b2700 1 --2- 172.21.15.16:0/1675730616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38408fef0 0x7fb38408f460 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.915+0000 7fb38c2b2700 1 --2- 172.21.15.16:0/1675730616 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb38408d2c0 0x7fb38408ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.915+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1675730616 shutdown_connections 2019-12-05T06:45:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.915+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1675730616 wait complete. 2019-12-05T06:45:30.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.915+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1675730616 >> 172.21.15.16:0/1675730616 conn(0x7fb384061940 msgr2=0x7fb384096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:30.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.919+0000 7fb38c2b2700 1 Processor -- start 2019-12-05T06:45:30.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.919+0000 7fb38c2b2700 1 -- start start 2019-12-05T06:45:30.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.920+0000 7fb38c2b2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb38414ca10 0x7fb3840869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:30.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.920+0000 7fb38c2b2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb38409be80 0x7fb38409c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:30.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.920+0000 7fb38c2b2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb384153170 con 0x7fb3840879b0 2019-12-05T06:45:30.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.920+0000 7fb38c2b2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb384085a50 con 0x7fb38414ca10 2019-12-05T06:45:30.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.920+0000 7fb38c2b2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb384085020 con 0x7fb38409be80 2019-12-05T06:45:30.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.920+0000 7fb38a84f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb38414ca10 0x7fb3840869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:30.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb38a84f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb38414ca10 0x7fb3840869e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60260/0 (socket says 172.21.15.16:60260) 2019-12-05T06:45:30.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb38a84f700 1 -- 172.21.15.16:0/1081367234 learned_addr learned my addr 172.21.15.16:0/1081367234 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:30.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb38b050700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb38409be80 0x7fb38409c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:30.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb38a84f700 1 -- 172.21.15.16:0/1081367234 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb38409be80 msgr2=0x7fb38409c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:30.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb38409be80 0x7fb38409c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:30.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb38a84f700 1 -- 172.21.15.16:0/1081367234 >> v1:172.21.15.76:6789/0 conn(0x7fb3840879b0 legacy=0x7fb384086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:30.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb38a84f700 1 -- 172.21.15.16:0/1081367234 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3840860e0 con 0x7fb38414ca10 2019-12-05T06:45:30.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb3837fe700 1 -- 172.21.15.16:0/1081367234 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 984521254 0 0) 0x7fb378000f30 con 0x7fb3840879b0 2019-12-05T06:45:30.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> v2:172.21.15.16:3300/0 conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> v2:172.21.15.16:3300/0 conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.921+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=READY pgs=1269 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.922+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=THROTTLE_DONE pgs=1269 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.922+0000 7fb3837fe700 1 -- 172.21.15.16:0/1081367234 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb37c001250 con 0x7fb38414ca10 2019-12-05T06:45:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.922+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=THROTTLE_DONE pgs=1269 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.922+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=THROTTLE_DONE pgs=1269 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.922+0000 7fb3837fe700 1 -- 172.21.15.16:0/1081367234 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb37c0047b0 con 0x7fb38414ca10 2019-12-05T06:45:30.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.922+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1081367234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3840860e0 con 0x7fb38414ca10 2019-12-05T06:45:30.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.922+0000 7fb3837fe700 1 -- 172.21.15.16:0/1081367234 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb37c0049a0 con 0x7fb38414ca10 2019-12-05T06:45:30.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.922+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1081367234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3840860e0 con 0x7fb38414ca10 2019-12-05T06:45:30.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.923+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=THROTTLE_DONE pgs=1269 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.923+0000 7fb3837fe700 1 -- 172.21.15.16:0/1081367234 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb37c0049a0 con 0x7fb38414ca10 2019-12-05T06:45:30.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.923+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=THROTTLE_DONE pgs=1269 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.924+0000 7fb3837fe700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb364023630 0x7fb364025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:30.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.924+0000 7fb3837fe700 1 -- 172.21.15.16:0/1081367234 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7fb37c0461a0 con 0x7fb38414ca10 2019-12-05T06:45:30.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.924+0000 7fb38a04e700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb364023630 0x7fb364025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:30.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.925+0000 7fb38a04e700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb364023630 0x7fb364025ac0 crc :-1 s=READY pgs=539 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:30.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.940+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1081367234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3840860e0 con 0x7fb38414ca10 2019-12-05T06:45:30.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.946+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=THROTTLE_DONE pgs=1269 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:30.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:30.946+0000 7fb3837fe700 1 -- 172.21.15.16:0/1081367234 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb37c01a7d0 con 0x7fb38414ca10 2019-12-05T06:45:31.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.174+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1081367234 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3840860e0 con 0x7fb38414ca10 2019-12-05T06:45:31.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.175+0000 7fb38a84f700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=THROTTLE_DONE pgs=1269 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:31.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.175+0000 7fb3837fe700 1 -- 172.21.15.16:0/1081367234 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fb37c01a7d0 con 0x7fb38414ca10 2019-12-05T06:45:31.178 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 157 2019-12-05T06:45:31.178 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:31.179 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1081367234 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb364023630 msgr2=0x7fb364025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb364023630 0x7fb364025ac0 crc :-1 s=READY pgs=539 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 msgr2=0x7fb3840869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 0x7fb3840869e0 secure :-1 s=READY pgs=1269 cs=0 l=1 rx=0x7fb37c0107b0 tx=0x7fb37c0105e0).stop 2019-12-05T06:45:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1081367234 shutdown_connections 2019-12-05T06:45:31.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb38409be80 0x7fb38409c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb364023630 0x7fb364025ac0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 --2- 172.21.15.16:0/1081367234 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb38414ca10 0x7fb3840869e0 unknown :-1 s=CLOSED pgs=1269 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1081367234 shutdown_connections 2019-12-05T06:45:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1081367234 wait complete. 2019-12-05T06:45:31.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:31.180+0000 7fb38c2b2700 1 -- 172.21.15.16:0/1081367234 >> 172.21.15.16:0/1081367234 conn(0x7fb384098480 msgr2=0x7fb384071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:31.197 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:45:33.198 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:33.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.470+0000 7f0b80bf5700 1 Processor -- start 2019-12-05T06:45:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.471+0000 7f0b80bf5700 1 -- start start 2019-12-05T06:45:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.473+0000 7f0b80bf5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.473+0000 7f0b80bf5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0b7c08fef0 0x7f0b7c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:33.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.473+0000 7f0b80bf5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c0958e0 con 0x7f0b7c08c6d0 2019-12-05T06:45:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.473+0000 7f0b80bf5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c08df90 con 0x7f0b7c08d2c0 2019-12-05T06:45:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.473+0000 7f0b80bf5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c0a3f70 con 0x7f0b7c08fef0 2019-12-05T06:45:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.473+0000 7f0b7a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.473+0000 7f0b7b59e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0b7c08fef0 0x7f0b7c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.473+0000 7f0b7a59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60282/0 (socket says 172.21.15.16:60282) 2019-12-05T06:45:33.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.473+0000 7f0b7a59c700 1 -- 172.21.15.16:0/2970580813 learned_addr learned my addr 172.21.15.16:0/2970580813 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.474+0000 7f0b7a59c700 1 -- 172.21.15.16:0/2970580813 >> v2:172.21.15.76:3300/0 conn(0x7f0b7c08fef0 msgr2=0x7f0b7c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.474+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/2970580813 >> v2:172.21.15.76:3300/0 conn(0x7f0b7c08fef0 0x7f0b7c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.474+0000 7f0b7a59c700 1 -- 172.21.15.16:0/2970580813 >> v1:172.21.15.76:6789/0 conn(0x7f0b7c08c6d0 legacy=0x7f0b7c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.474+0000 7f0b7a59c700 1 -- 172.21.15.16:0/2970580813 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c08e620 con 0x7f0b7c08d2c0 2019-12-05T06:45:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.474+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/2970580813 >> v2:172.21.15.16:3300/0 conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0b6400f6c0 tx=0x7f0b6400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.474+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/2970580813 >> v2:172.21.15.16:3300/0 conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0b6400f6c0 tx=0x7f0b6400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.474+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/2970580813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 secure :-1 s=READY pgs=1272 cs=0 l=1 rx=0x7f0b6400f6c0 tx=0x7f0b6400fa60).ready entity=mon.0 client_cookie=d1f9755f592f33fa server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.475+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/2970580813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 secure :-1 s=THROTTLE_DONE pgs=1272 cs=0 l=1 rx=0x7f0b6400f6c0 tx=0x7f0b6400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.475+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/2970580813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 secure :-1 s=THROTTLE_DONE pgs=1272 cs=0 l=1 rx=0x7f0b6400f6c0 tx=0x7f0b6400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.475+0000 7f0b79d9b700 1 -- 172.21.15.16:0/2970580813 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b64000960 con 0x7f0b7c08d2c0 2019-12-05T06:45:33.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.475+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/2970580813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 secure :-1 s=THROTTLE_DONE pgs=1272 cs=0 l=1 rx=0x7f0b6400f6c0 tx=0x7f0b6400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.475+0000 7f0b79d9b700 1 -- 172.21.15.16:0/2970580813 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b64000c90 con 0x7f0b7c08d2c0 2019-12-05T06:45:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.475+0000 7f0b79d9b700 1 -- 172.21.15.16:0/2970580813 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b640013a0 con 0x7f0b7c08d2c0 2019-12-05T06:45:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.475+0000 7f0b80bf5700 1 -- 172.21.15.16:0/2970580813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b7c08d2c0 msgr2=0x7f0b7c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:33.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.475+0000 7f0b80bf5700 1 --2- 172.21.15.16:0/2970580813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 secure :-1 s=READY pgs=1272 cs=0 l=1 rx=0x7f0b6400f6c0 tx=0x7f0b6400fa60).stop 2019-12-05T06:45:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.477+0000 7f0b80bf5700 1 -- 172.21.15.16:0/2970580813 shutdown_connections 2019-12-05T06:45:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.477+0000 7f0b80bf5700 1 --2- 172.21.15.16:0/2970580813 >> v2:172.21.15.76:3300/0 conn(0x7f0b7c08fef0 0x7f0b7c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:33.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.477+0000 7f0b80bf5700 1 --2- 172.21.15.16:0/2970580813 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0b7c08d2c0 0x7f0b7c08ef20 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.477+0000 7f0b80bf5700 1 -- 172.21.15.16:0/2970580813 shutdown_connections 2019-12-05T06:45:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.477+0000 7f0b80bf5700 1 -- 172.21.15.16:0/2970580813 wait complete. 2019-12-05T06:45:33.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.477+0000 7f0b80bf5700 1 -- 172.21.15.16:0/2970580813 >> 172.21.15.16:0/2970580813 conn(0x7f0b7c061940 msgr2=0x7f0b7c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.483+0000 7f0b80bf5700 1 Processor -- start 2019-12-05T06:45:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.483+0000 7f0b80bf5700 1 -- start start 2019-12-05T06:45:33.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.484+0000 7f0b80bf5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.484+0000 7f0b80bf5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0b7c084f30 0x7f0b7c155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.484+0000 7f0b80bf5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b7c084b90 con 0x7f0b7c096e90 2019-12-05T06:45:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.484+0000 7f0b80bf5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b7c09a450 con 0x7f0b7c084f30 2019-12-05T06:45:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.484+0000 7f0b80bf5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7c099a20 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.485+0000 7f0b7ad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f0b7c096e90 0x7f0b7c09b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45452/0 (socket says 172.21.15.16:45452) 2019-12-05T06:45:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.485+0000 7f0b7ad9d700 1 -- 172.21.15.16:0/3773548674 learned_addr learned my addr 172.21.15.16:0/3773548674 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:33.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.485+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:33.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.485+0000 7f0b7b59e700 1 --2- 172.21.15.16:0/3773548674 >> v2:172.21.15.76:3300/0 conn(0x7f0b7c084f30 0x7f0b7c155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.485+0000 7f0b737fe700 1 -- 172.21.15.16:0/3773548674 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3639181403 0 0) 0x7f0b740014d0 con 0x7f0b7c096e90 2019-12-05T06:45:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.485+0000 7f0b737fe700 1 -- 172.21.15.16:0/3773548674 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b60002e80 con 0x7f0b7c096e90 2019-12-05T06:45:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7a59c700 1 -- 172.21.15.16:0/3773548674 >> v2:172.21.15.76:3300/0 conn(0x7f0b7c084f30 msgr2=0x7f0b7c155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> v2:172.21.15.76:3300/0 conn(0x7f0b7c084f30 0x7f0b7c155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7a59c700 1 -- 172.21.15.16:0/3773548674 >> v1:172.21.15.16:6789/0 conn(0x7f0b7c096e90 legacy=0x7f0b7c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:33.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7a59c700 1 -- 172.21.15.16:0/3773548674 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b7c09aae0 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7b59e700 1 --2- 172.21.15.16:0/3773548674 >> v2:172.21.15.76:3300/0 conn(0x7f0b7c084f30 0x7f0b7c155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=READY pgs=299 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b737fe700 1 -- 172.21.15.16:0/3773548674 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b64000940 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b737fe700 1 -- 172.21.15.16:0/3773548674 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b64000f50 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b80bf5700 1 -- 172.21.15.16:0/3773548674 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7c09aae0 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.486+0000 7f0b737fe700 1 -- 172.21.15.16:0/3773548674 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b640016a0 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.487+0000 7f0b80bf5700 1 -- 172.21.15.16:0/3773548674 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7c09aae0 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.487+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.488+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.488+0000 7f0b737fe700 1 -- 172.21.15.16:0/3773548674 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0b64001140 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.489+0000 7f0b737fe700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b60023670 0x7f0b60025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.489+0000 7f0b737fe700 1 -- 172.21.15.16:0/3773548674 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f0b64045db0 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.489+0000 7f0b7ad9d700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b60023670 0x7f0b60025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:33.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.490+0000 7f0b7ad9d700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b60023670 0x7f0b60025ae0 crc :-1 s=READY pgs=540 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:33.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.502+0000 7f0b80bf5700 1 -- 172.21.15.16:0/3773548674 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b7c09aae0 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.511+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.511+0000 7f0b737fe700 1 -- 172.21.15.16:0/3773548674 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0b64000b30 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.779+0000 7f0b80bf5700 1 -- 172.21.15.16:0/3773548674 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0b7c09aae0 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.781+0000 7f0b7a59c700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=THROTTLE_DONE pgs=299 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:33.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.781+0000 7f0b737fe700 1 -- 172.21.15.16:0/3773548674 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f0b640019d0 con 0x7f0b7c09bbe0 2019-12-05T06:45:33.783 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 157 2019-12-05T06:45:33.783 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:33.784 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.785+0000 7f0b80bf5700 1 -- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b60023670 msgr2=0x7f0b60025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.785+0000 7f0b80bf5700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b60023670 0x7f0b60025ae0 crc :-1 s=READY pgs=540 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.785+0000 7f0b80bf5700 1 -- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 msgr2=0x7f0b7c1539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.785+0000 7f0b80bf5700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 secure :-1 s=READY pgs=299 cs=0 l=1 rx=0x7f0b640103a0 tx=0x7f0b64010740).stop 2019-12-05T06:45:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.785+0000 7f0b80bf5700 1 -- 172.21.15.16:0/3773548674 shutdown_connections 2019-12-05T06:45:33.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.785+0000 7f0b80bf5700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0b60023670 0x7f0b60025ae0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:33.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.785+0000 7f0b80bf5700 1 --2- 172.21.15.16:0/3773548674 >> v2:172.21.15.76:3300/0 conn(0x7f0b7c084f30 0x7f0b7c155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:33.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.785+0000 7f0b80bf5700 1 --2- 172.21.15.16:0/3773548674 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0b7c09bbe0 0x7f0b7c1539e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:33.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.786+0000 7f0b80bf5700 1 -- 172.21.15.16:0/3773548674 shutdown_connections 2019-12-05T06:45:33.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.786+0000 7f0b80bf5700 1 -- 172.21.15.16:0/3773548674 wait complete. 2019-12-05T06:45:33.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:33.786+0000 7f0b80bf5700 1 -- 172.21.15.16:0/3773548674 >> 172.21.15.16:0/3773548674 conn(0x7f0b7c0714f0 msgr2=0x7f0b7c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:33.797 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:45:35.799 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.060+0000 7fe959029700 1 Processor -- start 2019-12-05T06:45:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.060+0000 7fe959029700 1 -- start start 2019-12-05T06:45:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.062+0000 7fe959029700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe95408c6d0 0x7fe95408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:36.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.062+0000 7fe959029700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95408f720 0x7fe95408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.062+0000 7fe959029700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95408fb70 con 0x7fe95408fef0 2019-12-05T06:45:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.062+0000 7fe959029700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95408df90 con 0x7fe95408c6d0 2019-12-05T06:45:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.062+0000 7fe959029700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe9540a3f70 con 0x7fe95408f720 2019-12-05T06:45:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.062+0000 7fe952d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe95408fef0 0x7fe95408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45462/0 (socket says 172.21.15.16:45462) 2019-12-05T06:45:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.062+0000 7fe952d9d700 1 -- 172.21.15.16:0/2074538779 learned_addr learned my addr 172.21.15.16:0/2074538779 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.062+0000 7fe95359e700 1 --2- 172.21.15.16:0/2074538779 >> v2:172.21.15.76:3300/0 conn(0x7fe95408c6d0 0x7fe95408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:36.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.062+0000 7fe958828700 1 --2- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95408f720 0x7fe95408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.063+0000 7fe95259c700 1 -- 172.21.15.16:0/2074538779 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079755522 0 0) 0x7fe93c001170 con 0x7fe95408fef0 2019-12-05T06:45:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.063+0000 7fe95259c700 1 -- 172.21.15.16:0/2074538779 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe934002e80 con 0x7fe95408fef0 2019-12-05T06:45:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.063+0000 7fe95359e700 1 -- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95408f720 msgr2=0x7fe95408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.063+0000 7fe95359e700 1 --2- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95408f720 0x7fe95408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.063+0000 7fe95359e700 1 -- 172.21.15.16:0/2074538779 >> v1:172.21.15.16:6789/0 conn(0x7fe95408fef0 legacy=0x7fe95408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.063+0000 7fe95359e700 1 -- 172.21.15.16:0/2074538779 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95408d540 con 0x7fe95408c6d0 2019-12-05T06:45:36.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.063+0000 7fe958828700 1 --2- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95408f720 0x7fe95408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:36.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.063+0000 7fe95359e700 1 --2- 172.21.15.16:0/2074538779 >> v2:172.21.15.76:3300/0 conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe94c00af50 tx=0x7fe94c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.064+0000 7fe95359e700 1 --2- 172.21.15.16:0/2074538779 >> v2:172.21.15.76:3300/0 conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe94c00af50 tx=0x7fe94c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.064+0000 7fe95359e700 1 --2- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=READY pgs=877 cs=0 l=1 rx=0x7fe94c00af50 tx=0x7fe94c00b760).ready entity=mon.1 client_cookie=daad1fb8cd4a4746 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:36.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.064+0000 7fe95359e700 1 --2- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=THROTTLE_DONE pgs=877 cs=0 l=1 rx=0x7fe94c00af50 tx=0x7fe94c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.064+0000 7fe95259c700 1 -- 172.21.15.16:0/2074538779 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe94c007210 con 0x7fe95408c6d0 2019-12-05T06:45:36.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.064+0000 7fe95359e700 1 --2- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=THROTTLE_DONE pgs=877 cs=0 l=1 rx=0x7fe94c00af50 tx=0x7fe94c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.064+0000 7fe95359e700 1 --2- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=THROTTLE_DONE pgs=877 cs=0 l=1 rx=0x7fe94c00af50 tx=0x7fe94c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.064+0000 7fe95259c700 1 -- 172.21.15.16:0/2074538779 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c0099b0 con 0x7fe95408c6d0 2019-12-05T06:45:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.064+0000 7fe95259c700 1 -- 172.21.15.16:0/2074538779 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe94c00a080 con 0x7fe95408c6d0 2019-12-05T06:45:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.064+0000 7fe959029700 1 -- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe95408c6d0 msgr2=0x7fe95408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.064+0000 7fe959029700 1 --2- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=READY pgs=877 cs=0 l=1 rx=0x7fe94c00af50 tx=0x7fe94c00b760).stop 2019-12-05T06:45:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.066+0000 7fe959029700 1 -- 172.21.15.16:0/2074538779 shutdown_connections 2019-12-05T06:45:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.066+0000 7fe959029700 1 --2- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95408f720 0x7fe95408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:36.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.066+0000 7fe959029700 1 --2- 172.21.15.16:0/2074538779 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:36.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.066+0000 7fe959029700 1 -- 172.21.15.16:0/2074538779 shutdown_connections 2019-12-05T06:45:36.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.066+0000 7fe959029700 1 -- 172.21.15.16:0/2074538779 wait complete. 2019-12-05T06:45:36.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.066+0000 7fe959029700 1 -- 172.21.15.16:0/2074538779 >> 172.21.15.16:0/2074538779 conn(0x7fe954061940 msgr2=0x7fe954096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:36.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.071+0000 7fe959029700 1 Processor -- start 2019-12-05T06:45:36.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.071+0000 7fe959029700 1 -- start start 2019-12-05T06:45:36.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.073+0000 7fe959029700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:36.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.073+0000 7fe959029700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9540a0db0 con 0x7fe95414fd10 2019-12-05T06:45:36.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.073+0000 7fe959029700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe954070800 con 0x7fe95413b130 2019-12-05T06:45:36.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.073+0000 7fe959029700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe95409f800 con 0x7fe95413a9d0 2019-12-05T06:45:36.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.073+0000 7fe95359e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe95414fd10 0x7fe95413a1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45468/0 (socket says 172.21.15.16:45468) 2019-12-05T06:45:36.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.073+0000 7fe95359e700 1 -- 172.21.15.16:0/2523695 learned_addr learned my addr 172.21.15.16:0/2523695 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:36.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.073+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:36.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.073+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2282134708 0 0) 0x7fe94c00abb0 con 0x7fe95414fd10 2019-12-05T06:45:36.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.073+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c002e80 con 0x7fe95414fd10 2019-12-05T06:45:36.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.074+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4286574986 0 0) 0x7fe93c001170 con 0x7fe95413b130 2019-12-05T06:45:36.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.074+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe92c0030b0 con 0x7fe95413b130 2019-12-05T06:45:36.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.074+0000 7fe958828700 1 -- 172.21.15.16:0/2523695 >> v1:172.21.15.76:6789/0 conn(0x7fe95413b130 legacy=0x7fe95406fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:36.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.074+0000 7fe958828700 1 -- 172.21.15.16:0/2523695 >> v1:172.21.15.16:6789/0 conn(0x7fe95414fd10 legacy=0x7fe95413a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:36.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.074+0000 7fe958828700 1 -- 172.21.15.16:0/2523695 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95409fe90 con 0x7fe95413a9d0 2019-12-05T06:45:36.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.074+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.074+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.074+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=READY pgs=300 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:36.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.075+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.075+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe94400af10 con 0x7fe95413a9d0 2019-12-05T06:45:36.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.075+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.075+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.075+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe944009930 con 0x7fe95413a9d0 2019-12-05T06:45:36.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.075+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe94400a080 con 0x7fe95413a9d0 2019-12-05T06:45:36.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.075+0000 7fe959029700 1 -- 172.21.15.16:0/2523695 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95413adf0 con 0x7fe95413a9d0 2019-12-05T06:45:36.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.075+0000 7fe959029700 1 -- 172.21.15.16:0/2523695 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95409fe90 con 0x7fe95413a9d0 2019-12-05T06:45:36.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.076+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.077+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe944009b20 con 0x7fe95413a9d0 2019-12-05T06:45:36.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.077+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.077+0000 7fe94bfff700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe92c023650 0x7fe92c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:36.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.077+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7fe944046130 con 0x7fe95413a9d0 2019-12-05T06:45:36.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.077+0000 7fe95359e700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe92c023650 0x7fe92c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:36.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.078+0000 7fe95359e700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe92c023650 0x7fe92c025ae0 crc :-1 s=READY pgs=541 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:36.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.092+0000 7fe959029700 1 -- 172.21.15.16:0/2523695 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95413adf0 con 0x7fe95413a9d0 2019-12-05T06:45:36.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.101+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.102+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe94401b520 con 0x7fe95413a9d0 2019-12-05T06:45:36.320 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.319+0000 7fe959029700 1 -- 172.21.15.16:0/2523695 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe95409fe90 con 0x7fe95413a9d0 2019-12-05T06:45:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.321+0000 7fe958828700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=THROTTLE_DONE pgs=300 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:36.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.321+0000 7fe94bfff700 1 -- 172.21.15.16:0/2523695 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fe94401b520 con 0x7fe95413a9d0 2019-12-05T06:45:36.323 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:36.323 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:36.323 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 157 2019-12-05T06:45:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.324+0000 7fe959029700 1 -- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe92c023650 msgr2=0x7fe92c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:36.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.324+0000 7fe959029700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe92c023650 0x7fe92c025ae0 crc :-1 s=READY pgs=541 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.324+0000 7fe959029700 1 -- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 msgr2=0x7fe95415a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.324+0000 7fe959029700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 secure :-1 s=READY pgs=300 cs=0 l=1 rx=0x7fe944010720 tx=0x7fe944010ac0).stop 2019-12-05T06:45:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.324+0000 7fe959029700 1 -- 172.21.15.16:0/2523695 shutdown_connections 2019-12-05T06:45:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.324+0000 7fe959029700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe92c023650 0x7fe92c025ae0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.324+0000 7fe959029700 1 --2- 172.21.15.16:0/2523695 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe95413a9d0 0x7fe95415a440 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.325+0000 7fe959029700 1 -- 172.21.15.16:0/2523695 shutdown_connections 2019-12-05T06:45:36.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.325+0000 7fe959029700 1 -- 172.21.15.16:0/2523695 wait complete. 2019-12-05T06:45:36.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:36.325+0000 7fe959029700 1 -- 172.21.15.16:0/2523695 >> 172.21.15.16:0/2523695 conn(0x7fe954130a60 msgr2=0x7fe954086470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:36.337 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:45:38.340 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:38.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.621+0000 7f647dad2700 1 Processor -- start 2019-12-05T06:45:38.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.621+0000 7f647dad2700 1 -- start start 2019-12-05T06:45:38.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.623+0000 7f647dad2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f647808c6d0 0x7f647808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:38.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.623+0000 7f647dad2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647808fef0 0x7f647808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:38.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.623+0000 7f647dad2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64780958e0 con 0x7f647808ef20 2019-12-05T06:45:38.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.623+0000 7f647dad2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647808df90 con 0x7f647808c6d0 2019-12-05T06:45:38.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.623+0000 7f647dad2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64780a3f70 con 0x7f647808fef0 2019-12-05T06:45:38.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.623+0000 7f6477fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f647808c6d0 0x7f647808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:38.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.623+0000 7f6477fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f647808c6d0 0x7f647808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60310/0 (socket says 172.21.15.16:60310) 2019-12-05T06:45:38.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.623+0000 7f6477fff700 1 -- 172.21.15.16:0/917528531 learned_addr learned my addr 172.21.15.16:0/917528531 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:38.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.623+0000 7f64777fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647808fef0 0x7f647808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:38.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6477fff700 1 -- 172.21.15.16:0/917528531 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647808fef0 msgr2=0x7f647808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:38.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6477fff700 1 --2- 172.21.15.16:0/917528531 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647808fef0 0x7f647808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:38.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6477fff700 1 -- 172.21.15.16:0/917528531 >> v1:172.21.15.76:6789/0 conn(0x7f647808ef20 legacy=0x7f647808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:38.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6477fff700 1 -- 172.21.15.16:0/917528531 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647808d540 con 0x7f647808c6d0 2019-12-05T06:45:38.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6477fff700 1 --2- 172.21.15.16:0/917528531 >> v2:172.21.15.16:3300/0 conn(0x7f647808c6d0 0x7f647808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f646800f6c0 tx=0x7f646800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6477fff700 1 --2- 172.21.15.16:0/917528531 >> v2:172.21.15.16:3300/0 conn(0x7f647808c6d0 0x7f647808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f646800f6c0 tx=0x7f646800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6477fff700 1 --2- 172.21.15.16:0/917528531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647808c6d0 0x7f647808cac0 secure :-1 s=READY pgs=1276 cs=0 l=1 rx=0x7f646800f6c0 tx=0x7f646800fa60).ready entity=mon.0 client_cookie=658664d91e78310 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:38.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6477fff700 1 --2- 172.21.15.16:0/917528531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647808c6d0 0x7f647808cac0 secure :-1 s=THROTTLE_DONE pgs=1276 cs=0 l=1 rx=0x7f646800f6c0 tx=0x7f646800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6477fff700 1 --2- 172.21.15.16:0/917528531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647808c6d0 0x7f647808cac0 secure :-1 s=THROTTLE_DONE pgs=1276 cs=0 l=1 rx=0x7f646800f6c0 tx=0x7f646800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6476ffd700 1 -- 172.21.15.16:0/917528531 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6468000960 con 0x7f647808c6d0 2019-12-05T06:45:38.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.624+0000 7f6477fff700 1 --2- 172.21.15.16:0/917528531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647808c6d0 0x7f647808cac0 secure :-1 s=THROTTLE_DONE pgs=1276 cs=0 l=1 rx=0x7f646800f6c0 tx=0x7f646800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.625+0000 7f6476ffd700 1 -- 172.21.15.16:0/917528531 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6468000c90 con 0x7f647808c6d0 2019-12-05T06:45:38.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.625+0000 7f6476ffd700 1 -- 172.21.15.16:0/917528531 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64680013a0 con 0x7f647808c6d0 2019-12-05T06:45:38.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.625+0000 7f647dad2700 1 -- 172.21.15.16:0/917528531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647808c6d0 msgr2=0x7f647808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:38.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.625+0000 7f647dad2700 1 --2- 172.21.15.16:0/917528531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647808c6d0 0x7f647808cac0 secure :-1 s=READY pgs=1276 cs=0 l=1 rx=0x7f646800f6c0 tx=0x7f646800fa60).stop 2019-12-05T06:45:38.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.626+0000 7f647dad2700 1 -- 172.21.15.16:0/917528531 shutdown_connections 2019-12-05T06:45:38.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.626+0000 7f647dad2700 1 --2- 172.21.15.16:0/917528531 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647808fef0 0x7f647808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:38.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.626+0000 7f647dad2700 1 --2- 172.21.15.16:0/917528531 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647808c6d0 0x7f647808cac0 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:38.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.627+0000 7f647dad2700 1 -- 172.21.15.16:0/917528531 shutdown_connections 2019-12-05T06:45:38.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.627+0000 7f647dad2700 1 -- 172.21.15.16:0/917528531 wait complete. 2019-12-05T06:45:38.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.627+0000 7f647dad2700 1 -- 172.21.15.16:0/917528531 >> 172.21.15.16:0/917528531 conn(0x7f6478061940 msgr2=0x7f6478096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:38.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.631+0000 7f647dad2700 1 Processor -- start 2019-12-05T06:45:38.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.631+0000 7f647dad2700 1 -- start start 2019-12-05T06:45:38.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.633+0000 7f647dad2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6478098b00 0x7f647815a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:38.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.633+0000 7f647dad2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6478089fb0 con 0x7f647806df10 2019-12-05T06:45:38.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.633+0000 7f647dad2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647806e300 con 0x7f64780992d0 2019-12-05T06:45:38.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.633+0000 7f647dad2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6478097370 con 0x7f6478098b00 2019-12-05T06:45:38.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.633+0000 7f6477fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f647806df10 0x7f6478098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45484/0 (socket says 172.21.15.16:45484) 2019-12-05T06:45:38.642 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.633+0000 7f6477fff700 1 -- 172.21.15.16:0/1671128589 learned_addr learned my addr 172.21.15.16:0/1671128589 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.633+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> v2:172.21.15.16:3300/0 conn(0x7f6478098b00 0x7f647815a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.633+0000 7f6474ff9700 1 -- 172.21.15.16:0/1671128589 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3156184248 0 0) 0x7f6468007e10 con 0x7f647806df10 2019-12-05T06:45:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f6474ff9700 1 -- 172.21.15.16:0/1671128589 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6458002e80 con 0x7f647806df10 2019-12-05T06:45:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f6474ff9700 1 -- 172.21.15.16:0/1671128589 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365285860 0 0) 0x7f646c000cc0 con 0x7f64780992d0 2019-12-05T06:45:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f647c870700 1 -- 172.21.15.16:0/1671128589 >> v1:172.21.15.76:6789/0 conn(0x7f64780992d0 legacy=0x7f6478156cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:38.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f647c870700 1 -- 172.21.15.16:0/1671128589 >> v1:172.21.15.16:6789/0 conn(0x7f647806df10 legacy=0x7f6478098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:38.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f647c870700 1 -- 172.21.15.16:0/1671128589 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6478097a00 con 0x7f6478098b00 2019-12-05T06:45:38.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> v2:172.21.15.16:3300/0 conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> v2:172.21.15.16:3300/0 conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:38.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=THROTTLE_DONE pgs=1278 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f6474ff9700 1 -- 172.21.15.16:0/1671128589 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6470000bc0 con 0x7f6478098b00 2019-12-05T06:45:38.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=THROTTLE_DONE pgs=1278 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=THROTTLE_DONE pgs=1278 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.634+0000 7f6474ff9700 1 -- 172.21.15.16:0/1671128589 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f647000ede0 con 0x7f6478098b00 2019-12-05T06:45:38.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.635+0000 7f647dad2700 1 -- 172.21.15.16:0/1671128589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6478097a00 con 0x7f6478098b00 2019-12-05T06:45:38.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.635+0000 7f6474ff9700 1 -- 172.21.15.16:0/1671128589 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f647000f530 con 0x7f6478098b00 2019-12-05T06:45:38.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.635+0000 7f647dad2700 1 -- 172.21.15.16:0/1671128589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6478097a00 con 0x7f6478098b00 2019-12-05T06:45:38.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.635+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=THROTTLE_DONE pgs=1278 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.636+0000 7f6474ff9700 1 -- 172.21.15.16:0/1671128589 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f647000efd0 con 0x7f6478098b00 2019-12-05T06:45:38.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.636+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=THROTTLE_DONE pgs=1278 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.637+0000 7f6474ff9700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6458023680 0x7f6458025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:38.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.637+0000 7f6474ff9700 1 -- 172.21.15.16:0/1671128589 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f6470045e40 con 0x7f6478098b00 2019-12-05T06:45:38.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.637+0000 7f6477fff700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6458023680 0x7f6458025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:38.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.637+0000 7f6477fff700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6458023680 0x7f6458025af0 crc :-1 s=READY pgs=542 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:38.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.652+0000 7f647dad2700 1 -- 172.21.15.16:0/1671128589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6478097a00 con 0x7f6478098b00 2019-12-05T06:45:38.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.658+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=THROTTLE_DONE pgs=1278 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.658+0000 7f6474ff9700 1 -- 172.21.15.16:0/1671128589 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f647000e9c0 con 0x7f6478098b00 2019-12-05T06:45:38.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.886+0000 7f647dad2700 1 -- 172.21.15.16:0/1671128589 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6478097a00 con 0x7f6478098b00 2019-12-05T06:45:38.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.887+0000 7f647c870700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=THROTTLE_DONE pgs=1278 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:38.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.887+0000 7f6474ff9700 1 -- 172.21.15.16:0/1671128589 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f647000efb0 con 0x7f6478098b00 2019-12-05T06:45:38.888 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 157 2019-12-05T06:45:38.889 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:38.889 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.889+0000 7f647dad2700 1 -- 172.21.15.16:0/1671128589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6458023680 msgr2=0x7f6458025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:38.890 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.889+0000 7f647dad2700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6458023680 0x7f6458025af0 crc :-1 s=READY pgs=542 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.889+0000 7f647dad2700 1 -- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 msgr2=0x7f647815a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.889+0000 7f647dad2700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 0x7f647815a4d0 secure :-1 s=READY pgs=1278 cs=0 l=1 rx=0x7f64700103b0 tx=0x7f6470010750).stop 2019-12-05T06:45:38.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.889+0000 7f647dad2700 1 -- 172.21.15.16:0/1671128589 shutdown_connections 2019-12-05T06:45:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.889+0000 7f647dad2700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6458023680 0x7f6458025af0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:38.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.889+0000 7f647dad2700 1 --2- 172.21.15.16:0/1671128589 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6478098b00 0x7f647815a4d0 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:38.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.890+0000 7f647dad2700 1 -- 172.21.15.16:0/1671128589 shutdown_connections 2019-12-05T06:45:38.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.890+0000 7f647dad2700 1 -- 172.21.15.16:0/1671128589 wait complete. 2019-12-05T06:45:38.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:38.890+0000 7f647dad2700 1 -- 172.21.15.16:0/1671128589 >> 172.21.15.16:0/1671128589 conn(0x7f64780714f0 msgr2=0x7f647809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:38.905 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:45:40.916 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:41.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.191+0000 7f366ef2e700 1 Processor -- start 2019-12-05T06:45:41.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.191+0000 7f366ef2e700 1 -- start start 2019-12-05T06:45:41.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.193+0000 7f366ef2e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f366808fef0 0x7f366808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:41.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.193+0000 7f366ef2e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36680958e0 con 0x7f366808c6d0 2019-12-05T06:45:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.193+0000 7f366ef2e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36680629e0 con 0x7f366808d2c0 2019-12-05T06:45:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.193+0000 7f366ef2e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366808df90 con 0x7f366808fef0 2019-12-05T06:45:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.193+0000 7f366d4cb700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f366808c6d0 0x7f366808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45496/0 (socket says 172.21.15.16:45496) 2019-12-05T06:45:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.193+0000 7f366d4cb700 1 -- 172.21.15.16:0/1844701717 learned_addr learned my addr 172.21.15.16:0/1844701717 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.193+0000 7f366dccc700 1 --2- 172.21.15.16:0/1844701717 >> v2:172.21.15.76:3300/0 conn(0x7f366808fef0 0x7f366808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.194+0000 7f3667fff700 1 -- 172.21.15.16:0/1844701717 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072315965 0 0) 0x7f3660001150 con 0x7f366808c6d0 2019-12-05T06:45:41.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.194+0000 7f3667fff700 1 -- 172.21.15.16:0/1844701717 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3654002e80 con 0x7f366808c6d0 2019-12-05T06:45:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.194+0000 7f3667fff700 1 -- 172.21.15.16:0/1844701717 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398759251 0 0) 0x7f3650001400 con 0x7f366808d2c0 2019-12-05T06:45:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.194+0000 7f366dccc700 1 -- 172.21.15.16:0/1844701717 >> v1:172.21.15.76:6789/0 conn(0x7f366808d2c0 legacy=0x7f366808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.194+0000 7f366dccc700 1 -- 172.21.15.16:0/1844701717 >> v1:172.21.15.16:6789/0 conn(0x7f366808c6d0 legacy=0x7f366808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.194+0000 7f366dccc700 1 -- 172.21.15.16:0/1844701717 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366808e620 con 0x7f366808fef0 2019-12-05T06:45:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.195+0000 7f366dccc700 1 --2- 172.21.15.16:0/1844701717 >> v2:172.21.15.76:3300/0 conn(0x7f366808fef0 0x7f366808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f365800af80 tx=0x7f365800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.195+0000 7f366dccc700 1 --2- 172.21.15.16:0/1844701717 >> v2:172.21.15.76:3300/0 conn(0x7f366808fef0 0x7f366808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f365800af80 tx=0x7f365800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.195+0000 7f366dccc700 1 --2- 172.21.15.16:0/1844701717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366808fef0 0x7f366808f720 secure :-1 s=READY pgs=882 cs=0 l=1 rx=0x7f365800af80 tx=0x7f365800b790).ready entity=mon.1 client_cookie=bf49aa1c7d5a9324 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.195+0000 7f366dccc700 1 --2- 172.21.15.16:0/1844701717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366808fef0 0x7f366808f720 secure :-1 s=THROTTLE_DONE pgs=882 cs=0 l=1 rx=0x7f365800af80 tx=0x7f365800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.195+0000 7f366dccc700 1 --2- 172.21.15.16:0/1844701717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366808fef0 0x7f366808f720 secure :-1 s=THROTTLE_DONE pgs=882 cs=0 l=1 rx=0x7f365800af80 tx=0x7f365800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.196+0000 7f366dccc700 1 --2- 172.21.15.16:0/1844701717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366808fef0 0x7f366808f720 secure :-1 s=THROTTLE_DONE pgs=882 cs=0 l=1 rx=0x7f365800af80 tx=0x7f365800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.196+0000 7f3667fff700 1 -- 172.21.15.16:0/1844701717 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3658007240 con 0x7f366808fef0 2019-12-05T06:45:41.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.196+0000 7f3667fff700 1 -- 172.21.15.16:0/1844701717 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3658000c30 con 0x7f366808fef0 2019-12-05T06:45:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.196+0000 7f3667fff700 1 -- 172.21.15.16:0/1844701717 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3658009f10 con 0x7f366808fef0 2019-12-05T06:45:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.196+0000 7f366ef2e700 1 -- 172.21.15.16:0/1844701717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366808fef0 msgr2=0x7f366808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.196+0000 7f366ef2e700 1 --2- 172.21.15.16:0/1844701717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366808fef0 0x7f366808f720 secure :-1 s=READY pgs=882 cs=0 l=1 rx=0x7f365800af80 tx=0x7f365800b790).stop 2019-12-05T06:45:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.198+0000 7f366ef2e700 1 -- 172.21.15.16:0/1844701717 shutdown_connections 2019-12-05T06:45:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.198+0000 7f366ef2e700 1 --2- 172.21.15.16:0/1844701717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f366808fef0 0x7f366808f720 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.198+0000 7f366ef2e700 1 -- 172.21.15.16:0/1844701717 shutdown_connections 2019-12-05T06:45:41.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.198+0000 7f366ef2e700 1 -- 172.21.15.16:0/1844701717 wait complete. 2019-12-05T06:45:41.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.198+0000 7f366ef2e700 1 -- 172.21.15.16:0/1844701717 >> 172.21.15.16:0/1844701717 conn(0x7f3668061940 msgr2=0x7f3668096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.203+0000 7f366ef2e700 1 Processor -- start 2019-12-05T06:45:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.203+0000 7f366ef2e700 1 -- start start 2019-12-05T06:45:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.205+0000 7f366ef2e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f36681325e0 0x7f366812b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.205+0000 7f366ef2e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3668154cb0 0x7f36681570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.205+0000 7f366ef2e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3668128b20 con 0x7f366812c740 2019-12-05T06:45:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.205+0000 7f366ef2e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f366812a7e0 con 0x7f3668154cb0 2019-12-05T06:45:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.205+0000 7f366ef2e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3668129db0 con 0x7f36681325e0 2019-12-05T06:45:41.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.206+0000 7f366dccc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3668154cb0 0x7f36681570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.206+0000 7f366ccca700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f366812c740 0x7f366812bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45504/0 (socket says 172.21.15.16:45504) 2019-12-05T06:45:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.206+0000 7f366dccc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3668154cb0 0x7f36681570b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60338/0 (socket says 172.21.15.16:60338) 2019-12-05T06:45:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.206+0000 7f366ccca700 1 -- 172.21.15.16:0/3589439005 learned_addr learned my addr 172.21.15.16:0/3589439005 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:41.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.206+0000 7f366d4cb700 1 --2- 172.21.15.16:0/3589439005 >> v2:172.21.15.76:3300/0 conn(0x7f36681325e0 0x7f366812b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.206+0000 7f366dccc700 1 -- 172.21.15.16:0/3589439005 >> v2:172.21.15.76:3300/0 conn(0x7f36681325e0 msgr2=0x7f366812b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.206+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> v2:172.21.15.76:3300/0 conn(0x7f36681325e0 0x7f366812b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.206+0000 7f366dccc700 1 -- 172.21.15.16:0/3589439005 >> v1:172.21.15.16:6789/0 conn(0x7f366812c740 legacy=0x7f366812bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.206+0000 7f3665ffb700 1 -- 172.21.15.16:0/3589439005 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3075369121 0 0) 0x7f3650001640 con 0x7f366812c740 2019-12-05T06:45:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f366dccc700 1 -- 172.21.15.16:0/3589439005 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f366814a820 con 0x7f3668154cb0 2019-12-05T06:45:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f366d4cb700 1 --2- 172.21.15.16:0/3589439005 >> v2:172.21.15.76:3300/0 conn(0x7f36681325e0 0x7f366812b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> v2:172.21.15.16:3300/0 conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> v2:172.21.15.16:3300/0 conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=READY pgs=1281 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f3665ffb700 1 -- 172.21.15.16:0/3589439005 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36580097c0 con 0x7f3668154cb0 2019-12-05T06:45:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f3665ffb700 1 -- 172.21.15.16:0/3589439005 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f365800ae60 con 0x7f3668154cb0 2019-12-05T06:45:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f3665ffb700 1 -- 172.21.15.16:0/3589439005 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f365800b050 con 0x7f3668154cb0 2019-12-05T06:45:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f366ef2e700 1 -- 172.21.15.16:0/3589439005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f366812ae70 con 0x7f3668154cb0 2019-12-05T06:45:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.207+0000 7f366ef2e700 1 -- 172.21.15.16:0/3589439005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f366814a820 con 0x7f3668154cb0 2019-12-05T06:45:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.209+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.210+0000 7f3665ffb700 1 -- 172.21.15.16:0/3589439005 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f36580095b0 con 0x7f3668154cb0 2019-12-05T06:45:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.210+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.210+0000 7f3665ffb700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f364c023630 0x7f364c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.210+0000 7f3665ffb700 1 -- 172.21.15.16:0/3589439005 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f3658045d90 con 0x7f3668154cb0 2019-12-05T06:45:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.210+0000 7f366d4cb700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f364c023630 0x7f364c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:41.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.211+0000 7f366d4cb700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f364c023630 0x7f364c025ac0 crc :-1 s=READY pgs=543 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.222+0000 7f366ef2e700 1 -- 172.21.15.16:0/3589439005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f366812ae70 con 0x7f3668154cb0 2019-12-05T06:45:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.231+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.231+0000 7f3665ffb700 1 -- 172.21.15.16:0/3589439005 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f365800b050 con 0x7f3668154cb0 2019-12-05T06:45:41.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.474+0000 7f366ef2e700 1 -- 172.21.15.16:0/3589439005 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f366814a820 con 0x7f3668154cb0 2019-12-05T06:45:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.475+0000 7f366dccc700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=THROTTLE_DONE pgs=1281 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:41.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.475+0000 7f3665ffb700 1 -- 172.21.15.16:0/3589439005 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f36580095b0 con 0x7f3668154cb0 2019-12-05T06:45:41.478 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 157 2019-12-05T06:45:41.478 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:41.479 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.481+0000 7f366ef2e700 1 -- 172.21.15.16:0/3589439005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f364c023630 msgr2=0x7f364c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.481+0000 7f366ef2e700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f364c023630 0x7f364c025ac0 crc :-1 s=READY pgs=543 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.481+0000 7f366ef2e700 1 -- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 msgr2=0x7f36681570b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:41.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.481+0000 7f366ef2e700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 0x7f36681570b0 secure :-1 s=READY pgs=1281 cs=0 l=1 rx=0x7f3658010380 tx=0x7f3658010720).stop 2019-12-05T06:45:41.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.481+0000 7f366ef2e700 1 -- 172.21.15.16:0/3589439005 shutdown_connections 2019-12-05T06:45:41.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.481+0000 7f366ef2e700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f364c023630 0x7f364c025ac0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.481+0000 7f366ef2e700 1 --2- 172.21.15.16:0/3589439005 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3668154cb0 0x7f36681570b0 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.481+0000 7f366ef2e700 1 --2- 172.21.15.16:0/3589439005 >> v2:172.21.15.76:3300/0 conn(0x7f36681325e0 0x7f366812b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.482+0000 7f366ef2e700 1 -- 172.21.15.16:0/3589439005 shutdown_connections 2019-12-05T06:45:41.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.482+0000 7f366ef2e700 1 -- 172.21.15.16:0/3589439005 wait complete. 2019-12-05T06:45:41.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:41.482+0000 7f366ef2e700 1 -- 172.21.15.16:0/3589439005 >> 172.21.15.16:0/3589439005 conn(0x7f36680a2850 msgr2=0x7f366806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:41.500 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:45:42.697 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:45:42.705 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:45:43.503 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.737+0000 7febc14ef700 1 Processor -- start 2019-12-05T06:45:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.737+0000 7febc14ef700 1 -- start start 2019-12-05T06:45:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febc14ef700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7febbc08c6d0 0x7febbc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febc14ef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7febbc08f720 0x7febbc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:43.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febc14ef700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc08fb80 con 0x7febbc08fef0 2019-12-05T06:45:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febc14ef700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febbc08df90 con 0x7febbc08f720 2019-12-05T06:45:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febc14ef700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febbc0a3f70 con 0x7febbc08c6d0 2019-12-05T06:45:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febbbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7febbc08f720 0x7febbc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febbaffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7febbc08fef0 0x7febbc08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45516/0 (socket says 172.21.15.16:45516) 2019-12-05T06:45:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febbbfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7febbc08f720 0x7febbc08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60346/0 (socket says 172.21.15.16:60346) 2019-12-05T06:45:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febbaffd700 1 -- 172.21.15.16:0/4136977771 learned_addr learned my addr 172.21.15.16:0/4136977771 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febbb7fe700 1 --2- 172.21.15.16:0/4136977771 >> v2:172.21.15.76:3300/0 conn(0x7febbc08c6d0 0x7febbc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:43.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febbbfff700 1 -- 172.21.15.16:0/4136977771 >> v2:172.21.15.76:3300/0 conn(0x7febbc08c6d0 msgr2=0x7febbc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febbbfff700 1 --2- 172.21.15.16:0/4136977771 >> v2:172.21.15.76:3300/0 conn(0x7febbc08c6d0 0x7febbc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febbbfff700 1 -- 172.21.15.16:0/4136977771 >> v1:172.21.15.16:6789/0 conn(0x7febbc08fef0 legacy=0x7febbc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febbbfff700 1 -- 172.21.15.16:0/4136977771 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc08d540 con 0x7febbc08f720 2019-12-05T06:45:43.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.739+0000 7febba7fc700 1 -- 172.21.15.16:0/4136977771 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576095379 0 0) 0x7febb0001150 con 0x7febbc08fef0 2019-12-05T06:45:43.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febbbfff700 1 --2- 172.21.15.16:0/4136977771 >> v2:172.21.15.16:3300/0 conn(0x7febbc08f720 0x7febbc08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7febb400b2d0 tx=0x7febb400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febbbfff700 1 --2- 172.21.15.16:0/4136977771 >> v2:172.21.15.16:3300/0 conn(0x7febbc08f720 0x7febbc08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7febb400b2d0 tx=0x7febb400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febbbfff700 1 --2- 172.21.15.16:0/4136977771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc08f720 0x7febbc08d000 secure :-1 s=READY pgs=1283 cs=0 l=1 rx=0x7febb400b2d0 tx=0x7febb400ba60).ready entity=mon.0 client_cookie=65aa3bbbffc828df server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:43.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febbbfff700 1 --2- 172.21.15.16:0/4136977771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc08f720 0x7febbc08d000 secure :-1 s=THROTTLE_DONE pgs=1283 cs=0 l=1 rx=0x7febb400b2d0 tx=0x7febb400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febba7fc700 1 -- 172.21.15.16:0/4136977771 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febb4009f20 con 0x7febbc08f720 2019-12-05T06:45:43.751 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febbbfff700 1 --2- 172.21.15.16:0/4136977771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc08f720 0x7febbc08d000 secure :-1 s=THROTTLE_DONE pgs=1283 cs=0 l=1 rx=0x7febb400b2d0 tx=0x7febb400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febba7fc700 1 -- 172.21.15.16:0/4136977771 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febb4000f40 con 0x7febbc08f720 2019-12-05T06:45:43.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febbbfff700 1 --2- 172.21.15.16:0/4136977771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc08f720 0x7febbc08d000 secure :-1 s=THROTTLE_DONE pgs=1283 cs=0 l=1 rx=0x7febb400b2d0 tx=0x7febb400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febba7fc700 1 -- 172.21.15.16:0/4136977771 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febb4009f20 con 0x7febbc08f720 2019-12-05T06:45:43.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febc14ef700 1 -- 172.21.15.16:0/4136977771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc08f720 msgr2=0x7febbc08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:43.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.740+0000 7febc14ef700 1 --2- 172.21.15.16:0/4136977771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc08f720 0x7febbc08d000 secure :-1 s=READY pgs=1283 cs=0 l=1 rx=0x7febb400b2d0 tx=0x7febb400ba60).stop 2019-12-05T06:45:43.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.742+0000 7febc14ef700 1 -- 172.21.15.16:0/4136977771 shutdown_connections 2019-12-05T06:45:43.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.742+0000 7febc14ef700 1 --2- 172.21.15.16:0/4136977771 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7febbc08f720 0x7febbc08d000 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:43.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.742+0000 7febc14ef700 1 --2- 172.21.15.16:0/4136977771 >> v2:172.21.15.76:3300/0 conn(0x7febbc08c6d0 0x7febbc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:43.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.742+0000 7febc14ef700 1 -- 172.21.15.16:0/4136977771 shutdown_connections 2019-12-05T06:45:43.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.742+0000 7febc14ef700 1 -- 172.21.15.16:0/4136977771 wait complete. 2019-12-05T06:45:43.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.742+0000 7febc14ef700 1 -- 172.21.15.16:0/4136977771 >> 172.21.15.16:0/4136977771 conn(0x7febbc061940 msgr2=0x7febbc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:43.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.746+0000 7febc14ef700 1 Processor -- start 2019-12-05T06:45:43.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.746+0000 7febc14ef700 1 -- start start 2019-12-05T06:45:43.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.747+0000 7febc14ef700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febbc06e210 0x7febbc09ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:43.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.747+0000 7febc14ef700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7febbc09e580 0x7febbc19d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:43.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.747+0000 7febc14ef700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febbc153160 con 0x7febbc14c3a0 2019-12-05T06:45:43.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.747+0000 7febc14ef700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febbc09c620 con 0x7febbc09e580 2019-12-05T06:45:43.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.747+0000 7febc14ef700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febbc08ada0 con 0x7febbc06e210 2019-12-05T06:45:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.747+0000 7febbb7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7febbc14c3a0 0x7febbc09d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45518/0 (socket says 172.21.15.16:45518) 2019-12-05T06:45:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.748+0000 7febbb7fe700 1 -- 172.21.15.16:0/603245625 learned_addr learned my addr 172.21.15.16:0/603245625 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.748+0000 7febbbfff700 1 --2- 172.21.15.16:0/603245625 >> v2:172.21.15.76:3300/0 conn(0x7febbc09e580 0x7febbc19d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.748+0000 7febbaffd700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febbc06e210 0x7febbc09ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.748+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443178871 0 0) 0x7febac0010b0 con 0x7febbc14c3a0 2019-12-05T06:45:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.748+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb9c002e80 con 0x7febbc14c3a0 2019-12-05T06:45:43.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.748+0000 7febbbfff700 1 -- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febbc06e210 msgr2=0x7febbc09ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.748+0000 7febbbfff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febbc06e210 0x7febbc09ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.748+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3134590858 0 0) 0x7febac0012b0 con 0x7febbc14c3a0 2019-12-05T06:45:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.748+0000 7febbbfff700 1 -- 172.21.15.16:0/603245625 >> v1:172.21.15.16:6789/0 conn(0x7febbc14c3a0 legacy=0x7febbc09d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbbfff700 1 -- 172.21.15.16:0/603245625 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febbc09ccb0 con 0x7febbc09e580 2019-12-05T06:45:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbaffd700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7febbc06e210 0x7febbc09ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbbfff700 1 --2- 172.21.15.16:0/603245625 >> v2:172.21.15.76:3300/0 conn(0x7febbc09e580 0x7febbc19d410 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7febb4010700 tx=0x7febb4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.761 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbbfff700 1 --2- 172.21.15.16:0/603245625 >> v2:172.21.15.76:3300/0 conn(0x7febbc09e580 0x7febbc19d410 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7febb4010700 tx=0x7febb4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbbfff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febbc09e580 0x7febbc19d410 secure :-1 s=READY pgs=883 cs=0 l=1 rx=0x7febb4010700 tx=0x7febb4010aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbbfff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febbc09e580 0x7febbc19d410 secure :-1 s=THROTTLE_DONE pgs=883 cs=0 l=1 rx=0x7febb4010700 tx=0x7febb4010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febb400b330 con 0x7febbc09e580 2019-12-05T06:45:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febc14ef700 1 -- 172.21.15.16:0/603245625 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febbc09ccb0 con 0x7febbc09e580 2019-12-05T06:45:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbbfff700 1 -- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febbc09e580 msgr2=0x7febbc19d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2019-12-05T06:45:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbbfff700 1 -- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febbc09e580 msgr2=0x7febbc19d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:45:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbbfff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febbc09e580 0x7febbc19d410 secure :-1 s=READY pgs=883 cs=0 l=1 rx=0x7febb4010700 tx=0x7febb4010aa0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:45:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbbfff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febbc09e580 0x7febbc19d410 secure :-1 s=READY pgs=883 cs=0 l=1 rx=0x7febb4010700 tx=0x7febb4010aa0).stop 2019-12-05T06:45:43.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febbc09e580 msgr2=0x7febbc19d410 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:45:43.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7feba3fff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febbc09e580 0x7febbc19d410 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:43.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7feba3fff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:43.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7feba3fff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feb9c006010 0x7feb9c0084e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:43.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7feba3fff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb9c008ce0 0x7feb9c00b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:43.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c00b9c0 con 0x7feb9c003400 2019-12-05T06:45:43.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c00bbf0 con 0x7feb9c008ce0 2019-12-05T06:45:43.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c00be20 con 0x7feb9c006010 2019-12-05T06:45:43.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.749+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:43.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbbfff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb9c008ce0 0x7feb9c00b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:43.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbaffd700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feb9c006010 0x7feb9c0084e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:43.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbb7fe700 1 -- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feb9c006010 msgr2=0x7feb9c0084e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:43.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7feb9c006010 0x7feb9c0084e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:43.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbb7fe700 1 -- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb9c008ce0 msgr2=0x7feb9c00b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:43.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7feb9c008ce0 0x7feb9c00b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:43.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbb7fe700 1 -- 172.21.15.16:0/603245625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) v3 -- 0x7feb9c00c110 con 0x7feb9c003400 2019-12-05T06:45:43.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbb7fe700 1 -- 172.21.15.16:0/603245625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febbc09ccb0 con 0x7feb9c003400 2019-12-05T06:45:43.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7febac010470 tx=0x7febac010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7febac010470 tx=0x7febac010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.750+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 secure :-1 s=READY pgs=1285 cs=0 l=1 rx=0x7febac010470 tx=0x7febac010340).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.751+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7febac010470 tx=0x7febac010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.751+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febac001070 con 0x7feb9c003400 2019-12-05T06:45:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.751+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7febac010470 tx=0x7febac010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.751+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febac000c90 con 0x7feb9c003400 2019-12-05T06:45:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.751+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7febac010470 tx=0x7febac010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.752+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7febac000e80 con 0x7feb9c003400 2019-12-05T06:45:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.752+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7febac010470 tx=0x7febac010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.752+0000 7feba3fff700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feb9c02b8c0 0x7feb9c02dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.752+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7febac045f90 con 0x7feb9c003400 2019-12-05T06:45:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.752+0000 7febc14ef700 1 -- 172.21.15.16:0/603245625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb94002e60 con 0x7feb9c003400 2019-12-05T06:45:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.752+0000 7febbaffd700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feb9c02b8c0 0x7feb9c02dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:43.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.753+0000 7febbaffd700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feb9c02b8c0 0x7feb9c02dcf0 crc :-1 s=READY pgs=545 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.761+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7febac010470 tx=0x7febac010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:43.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:43.761+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 <== mon.0 v2:172.21.15.16:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7febac000ea0 con 0x7feb9c003400 2019-12-05T06:45:44.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.004+0000 7febc14ef700 1 -- 172.21.15.16:0/603245625 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb94001d10 con 0x7feb9c003400 2019-12-05T06:45:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.004+0000 7febbb7fe700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 secure :-1 s=THROTTLE_DONE pgs=1285 cs=0 l=1 rx=0x7febac010470 tx=0x7febac010340).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:44.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.004+0000 7feba3fff700 1 -- 172.21.15.16:0/603245625 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2109 (secure 0 0 0) 0x7febac000c90 con 0x7feb9c003400 2019-12-05T06:45:44.006 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 157 2019-12-05T06:45:44.006 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:44.006 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febc14ef700 1 -- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feb9c02b8c0 msgr2=0x7feb9c02dcf0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febc14ef700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feb9c02b8c0 0x7feb9c02dcf0 crc :-1 s=READY pgs=545 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febc14ef700 1 -- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 msgr2=0x7feb9c005860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:44.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febc14ef700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 secure :-1 s=READY pgs=1285 cs=0 l=1 rx=0x7febac010470 tx=0x7febac010340).stop 2019-12-05T06:45:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febbbfff700 1 -- 172.21.15.16:0/603245625 reap_dead start 2019-12-05T06:45:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febc14ef700 1 -- 172.21.15.16:0/603245625 shutdown_connections 2019-12-05T06:45:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febc14ef700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7febbc09e580 0x7febbc19d410 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febc14ef700 1 --2- 172.21.15.16:0/603245625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7feb9c003400 0x7feb9c005860 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:44.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febc14ef700 1 -- 172.21.15.16:0/603245625 shutdown_connections 2019-12-05T06:45:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febc14ef700 1 -- 172.21.15.16:0/603245625 wait complete. 2019-12-05T06:45:44.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:44.007+0000 7febc14ef700 1 -- 172.21.15.16:0/603245625 >> 172.21.15.16:0/603245625 conn(0x7febbc098480 msgr2=0x7febbc071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:44.019 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:45:46.020 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:46.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.311+0000 7f60cb4e7700 1 Processor -- start 2019-12-05T06:45:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.311+0000 7f60cb4e7700 1 -- start start 2019-12-05T06:45:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.313+0000 7f60cb4e7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60c408c6d0 0x7f60c408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.313+0000 7f60cb4e7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f60c408fef0 0x7f60c408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:46.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.313+0000 7f60cb4e7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c408fb80 con 0x7f60c408ef20 2019-12-05T06:45:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.313+0000 7f60cb4e7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c408df90 con 0x7f60c408c6d0 2019-12-05T06:45:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.313+0000 7f60cb4e7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c40a3f70 con 0x7f60c408fef0 2019-12-05T06:45:46.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.314+0000 7f60c9a84700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60c408c6d0 0x7f60c408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:46.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.314+0000 7f60ca285700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f60c408ef20 0x7f60c408f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45534/0 (socket says 172.21.15.16:45534) 2019-12-05T06:45:46.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.314+0000 7f60ca285700 1 -- 172.21.15.16:0/1808462515 learned_addr learned my addr 172.21.15.16:0/1808462515 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:46.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.314+0000 7f60c9283700 1 --2- 172.21.15.16:0/1808462515 >> v2:172.21.15.76:3300/0 conn(0x7f60c408fef0 0x7f60c408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:46.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.314+0000 7f60c8a82700 1 -- 172.21.15.16:0/1808462515 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2675627794 0 0) 0x7f60b4001150 con 0x7f60c408ef20 2019-12-05T06:45:46.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.314+0000 7f60c8a82700 1 -- 172.21.15.16:0/1808462515 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60ac002e80 con 0x7f60c408ef20 2019-12-05T06:45:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.314+0000 7f60c9a84700 1 -- 172.21.15.16:0/1808462515 >> v2:172.21.15.76:3300/0 conn(0x7f60c408fef0 msgr2=0x7f60c408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.314+0000 7f60c9a84700 1 --2- 172.21.15.16:0/1808462515 >> v2:172.21.15.76:3300/0 conn(0x7f60c408fef0 0x7f60c408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.315+0000 7f60c9a84700 1 -- 172.21.15.16:0/1808462515 >> v1:172.21.15.16:6789/0 conn(0x7f60c408ef20 legacy=0x7f60c408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.315+0000 7f60c9a84700 1 -- 172.21.15.16:0/1808462515 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c408d540 con 0x7f60c408c6d0 2019-12-05T06:45:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.315+0000 7f60c9283700 1 --2- 172.21.15.16:0/1808462515 >> v2:172.21.15.76:3300/0 conn(0x7f60c408fef0 0x7f60c408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:46.328 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.315+0000 7f60c9a84700 1 --2- 172.21.15.16:0/1808462515 >> v2:172.21.15.16:3300/0 conn(0x7f60c408c6d0 0x7f60c408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f60bc00b2d0 tx=0x7f60bc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.315+0000 7f60c9a84700 1 --2- 172.21.15.16:0/1808462515 >> v2:172.21.15.16:3300/0 conn(0x7f60c408c6d0 0x7f60c408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f60bc00b2d0 tx=0x7f60bc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.315+0000 7f60c9a84700 1 --2- 172.21.15.16:0/1808462515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c408c6d0 0x7f60c408cac0 secure :-1 s=READY pgs=1287 cs=0 l=1 rx=0x7f60bc00b2d0 tx=0x7f60bc00bae0).ready entity=mon.0 client_cookie=293cfd4e2140d025 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:46.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.315+0000 7f60c9a84700 1 --2- 172.21.15.16:0/1808462515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c408c6d0 0x7f60c408cac0 secure :-1 s=THROTTLE_DONE pgs=1287 cs=0 l=1 rx=0x7f60bc00b2d0 tx=0x7f60bc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.315+0000 7f60c8a82700 1 -- 172.21.15.16:0/1808462515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60bc007590 con 0x7f60c408c6d0 2019-12-05T06:45:46.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.315+0000 7f60c9a84700 1 --2- 172.21.15.16:0/1808462515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c408c6d0 0x7f60c408cac0 secure :-1 s=THROTTLE_DONE pgs=1287 cs=0 l=1 rx=0x7f60bc00b2d0 tx=0x7f60bc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.329 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.315+0000 7f60c9a84700 1 --2- 172.21.15.16:0/1808462515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c408c6d0 0x7f60c408cac0 secure :-1 s=THROTTLE_DONE pgs=1287 cs=0 l=1 rx=0x7f60bc00b2d0 tx=0x7f60bc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.316+0000 7f60c8a82700 1 -- 172.21.15.16:0/1808462515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60bc009d30 con 0x7f60c408c6d0 2019-12-05T06:45:46.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.316+0000 7f60c8a82700 1 -- 172.21.15.16:0/1808462515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60bc00a400 con 0x7f60c408c6d0 2019-12-05T06:45:46.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.316+0000 7f60cb4e7700 1 -- 172.21.15.16:0/1808462515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c408c6d0 msgr2=0x7f60c408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:46.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.316+0000 7f60cb4e7700 1 --2- 172.21.15.16:0/1808462515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c408c6d0 0x7f60c408cac0 secure :-1 s=READY pgs=1287 cs=0 l=1 rx=0x7f60bc00b2d0 tx=0x7f60bc00bae0).stop 2019-12-05T06:45:46.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.317+0000 7f60cb4e7700 1 -- 172.21.15.16:0/1808462515 shutdown_connections 2019-12-05T06:45:46.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.317+0000 7f60cb4e7700 1 --2- 172.21.15.16:0/1808462515 >> v2:172.21.15.76:3300/0 conn(0x7f60c408fef0 0x7f60c408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:46.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.317+0000 7f60cb4e7700 1 --2- 172.21.15.16:0/1808462515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c408c6d0 0x7f60c408cac0 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:46.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.318+0000 7f60cb4e7700 1 -- 172.21.15.16:0/1808462515 shutdown_connections 2019-12-05T06:45:46.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.318+0000 7f60cb4e7700 1 -- 172.21.15.16:0/1808462515 wait complete. 2019-12-05T06:45:46.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.318+0000 7f60cb4e7700 1 -- 172.21.15.16:0/1808462515 >> 172.21.15.16:0/1808462515 conn(0x7f60c4061940 msgr2=0x7f60c4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:46.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.323+0000 7f60cb4e7700 1 Processor -- start 2019-12-05T06:45:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.323+0000 7f60cb4e7700 1 -- start start 2019-12-05T06:45:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.324+0000 7f60cb4e7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60c414d400 0x7f60c414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.324+0000 7f60cb4e7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f60c4132d00 0x7f60c4131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:46.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.324+0000 7f60cb4e7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60c4098da0 con 0x7f60c4132270 2019-12-05T06:45:46.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.325+0000 7f60cb4e7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c409bb50 con 0x7f60c414d400 2019-12-05T06:45:46.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.325+0000 7f60cb4e7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60c409aa60 con 0x7f60c4132d00 2019-12-05T06:45:46.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.325+0000 7f60c9a84700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60c414d400 0x7f60c414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:46.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.325+0000 7f60c9a84700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f60c414d400 0x7f60c414d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60372/0 (socket says 172.21.15.16:60372) 2019-12-05T06:45:46.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.325+0000 7f60c9a84700 1 -- 172.21.15.16:0/2174448670 learned_addr learned my addr 172.21.15.16:0/2174448670 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:46.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.325+0000 7f60c9283700 1 --2- 172.21.15.16:0/2174448670 >> v2:172.21.15.76:3300/0 conn(0x7f60c4132d00 0x7f60c4131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:46.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.325+0000 7f60c9a84700 1 -- 172.21.15.16:0/2174448670 >> v2:172.21.15.76:3300/0 conn(0x7f60c4132d00 msgr2=0x7f60c4131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:46.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.325+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> v2:172.21.15.76:3300/0 conn(0x7f60c4132d00 0x7f60c4131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:46.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.325+0000 7f60c9a84700 1 -- 172.21.15.16:0/2174448670 >> v1:172.21.15.16:6789/0 conn(0x7f60c4132270 legacy=0x7f60c406dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:46.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c9a84700 1 -- 172.21.15.16:0/2174448670 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60c409b0f0 con 0x7f60c414d400 2019-12-05T06:45:46.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> v2:172.21.15.16:3300/0 conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c9283700 1 --2- 172.21.15.16:0/2174448670 >> v2:172.21.15.76:3300/0 conn(0x7f60c4132d00 0x7f60c4131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:46.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> v2:172.21.15.16:3300/0 conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:46.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c27fc700 1 -- 172.21.15.16:0/2174448670 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60bc00b700 con 0x7f60c414d400 2019-12-05T06:45:46.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c27fc700 1 -- 172.21.15.16:0/2174448670 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60bc009b10 con 0x7f60c414d400 2019-12-05T06:45:46.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60c27fc700 1 -- 172.21.15.16:0/2174448670 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60bc00a1e0 con 0x7f60c414d400 2019-12-05T06:45:46.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.326+0000 7f60cb4e7700 1 -- 172.21.15.16:0/2174448670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60c409b0f0 con 0x7f60c414d400 2019-12-05T06:45:46.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.327+0000 7f60cb4e7700 1 -- 172.21.15.16:0/2174448670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60c409b0f0 con 0x7f60c414d400 2019-12-05T06:45:46.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.328+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.329+0000 7f60c27fc700 1 -- 172.21.15.16:0/2174448670 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f60bc009d00 con 0x7f60c414d400 2019-12-05T06:45:46.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.329+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.329+0000 7f60c27fc700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60a4023630 0x7f60a4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:46.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.329+0000 7f60c27fc700 1 -- 172.21.15.16:0/2174448670 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 8181+0+0 (secure 0 0 0) 0x7f60bc0461a0 con 0x7f60c414d400 2019-12-05T06:45:46.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.329+0000 7f60c9283700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60a4023630 0x7f60a4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:46.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.330+0000 7f60c9283700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60a4023630 0x7f60a4025ac0 crc :-1 s=READY pgs=546 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:46.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.342+0000 7f60cb4e7700 1 -- 172.21.15.16:0/2174448670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60c409b0f0 con 0x7f60c414d400 2019-12-05T06:45:46.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.350+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.350+0000 7f60c27fc700 1 -- 172.21.15.16:0/2174448670 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f60bc027f40 con 0x7f60c414d400 2019-12-05T06:45:46.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.605+0000 7f60cb4e7700 1 -- 172.21.15.16:0/2174448670 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60c409b0f0 con 0x7f60c414d400 2019-12-05T06:45:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.606+0000 7f60c9a84700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=THROTTLE_DONE pgs=1289 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.606+0000 7f60c27fc700 1 -- 172.21.15.16:0/2174448670 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 157 v157) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f60bc027f40 con 0x7f60c414d400 2019-12-05T06:45:46.607 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 157 2019-12-05T06:45:46.608 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:46.608 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":157,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7401,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":157}],"filesystems":[{"mdsmap":{"epoch":156,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:44:42.685991+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":74,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5744},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5744":{"gid":5744,"name":"b","rank":0,"incarnation":153,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/3914386168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3914386168},{"type":"v1","addr":"172.21.15.76:6835","nonce":3914386168}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:46.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 -- 172.21.15.16:0/2174448670 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60a4023630 msgr2=0x7f60a4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60a4023630 0x7f60a4025ac0 crc :-1 s=READY pgs=546 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 -- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 msgr2=0x7f60c414d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 0x7f60c414d7f0 secure :-1 s=READY pgs=1289 cs=0 l=1 rx=0x7f60bc0107c0 tx=0x7f60bc010b60).stop 2019-12-05T06:45:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 -- 172.21.15.16:0/2174448670 shutdown_connections 2019-12-05T06:45:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 --2- 172.21.15.16:0/2174448670 >> v2:172.21.15.76:3300/0 conn(0x7f60c4132d00 0x7f60c4131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f60a4023630 0x7f60a4025ac0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:46.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 --2- 172.21.15.16:0/2174448670 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f60c414d400 0x7f60c414d7f0 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 -- 172.21.15.16:0/2174448670 shutdown_connections 2019-12-05T06:45:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 -- 172.21.15.16:0/2174448670 wait complete. 2019-12-05T06:45:46.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:46.608+0000 7f60cb4e7700 1 -- 172.21.15.16:0/2174448670 >> 172.21.15.16:0/2174448670 conn(0x7f60c40714f0 msgr2=0x7f60c409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:46.620 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:45:48.622 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:48.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.826+0000 7f1caba02700 1 Processor -- start 2019-12-05T06:45:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.827+0000 7f1caba02700 1 -- start start 2019-12-05T06:45:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.827+0000 7f1caba02700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1ca408c6d0 0x7f1ca408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.827+0000 7f1caba02700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca408fef0 0x7f1ca408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.827+0000 7f1caba02700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1ca408ef20 0x7f1ca408f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:48.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.827+0000 7f1caba02700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca408df90 con 0x7f1ca408fef0 2019-12-05T06:45:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.827+0000 7f1caba02700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca40a3f70 con 0x7f1ca408c6d0 2019-12-05T06:45:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.827+0000 7f1caba02700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca408d830 con 0x7f1ca408ef20 2019-12-05T06:45:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca408fef0 0x7f1ca408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca408fef0 0x7f1ca408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60392/0 (socket says 172.21.15.16:60392) 2019-12-05T06:45:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 -- 172.21.15.16:0/2300150877 learned_addr learned my addr 172.21.15.16:0/2300150877 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/2300150877 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1ca408ef20 0x7f1ca408f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:48.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca9f9f700 1 --2- 172.21.15.16:0/2300150877 >> v2:172.21.15.76:3300/0 conn(0x7f1ca408c6d0 0x7f1ca408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 -- 172.21.15.16:0/2300150877 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1ca408ef20 msgr2=0x7f1ca408f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 --2- 172.21.15.16:0/2300150877 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1ca408ef20 0x7f1ca408f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 -- 172.21.15.16:0/2300150877 >> v2:172.21.15.76:3300/0 conn(0x7f1ca408c6d0 msgr2=0x7f1ca408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 --2- 172.21.15.16:0/2300150877 >> v2:172.21.15.76:3300/0 conn(0x7f1ca408c6d0 0x7f1ca408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 -- 172.21.15.16:0/2300150877 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca408d540 con 0x7f1ca408fef0 2019-12-05T06:45:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca9f9f700 1 --2- 172.21.15.16:0/2300150877 >> v2:172.21.15.76:3300/0 conn(0x7f1ca408c6d0 0x7f1ca408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:45:48.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 --2- 172.21.15.16:0/2300150877 >> v2:172.21.15.16:3300/0 conn(0x7f1ca408fef0 0x7f1ca408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1ca000f6c0 tx=0x7f1ca000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 --2- 172.21.15.16:0/2300150877 >> v2:172.21.15.16:3300/0 conn(0x7f1ca408fef0 0x7f1ca408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1ca000f6c0 tx=0x7f1ca000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 --2- 172.21.15.16:0/2300150877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca408fef0 0x7f1ca408d000 secure :-1 s=READY pgs=1290 cs=0 l=1 rx=0x7f1ca000f6c0 tx=0x7f1ca000fa60).ready entity=mon.0 client_cookie=6b96b4deb81fda6b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.828+0000 7f1ca979e700 1 --2- 172.21.15.16:0/2300150877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca408fef0 0x7f1ca408d000 secure :-1 s=THROTTLE_DONE pgs=1290 cs=0 l=1 rx=0x7f1ca000f6c0 tx=0x7f1ca000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.829+0000 7f1ca979e700 1 --2- 172.21.15.16:0/2300150877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca408fef0 0x7f1ca408d000 secure :-1 s=THROTTLE_DONE pgs=1290 cs=0 l=1 rx=0x7f1ca000f6c0 tx=0x7f1ca000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.829+0000 7f1ca8f9d700 1 -- 172.21.15.16:0/2300150877 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ca0000960 con 0x7f1ca408fef0 2019-12-05T06:45:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.829+0000 7f1ca979e700 1 --2- 172.21.15.16:0/2300150877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca408fef0 0x7f1ca408d000 secure :-1 s=THROTTLE_DONE pgs=1290 cs=0 l=1 rx=0x7f1ca000f6c0 tx=0x7f1ca000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.829+0000 7f1ca8f9d700 1 -- 172.21.15.16:0/2300150877 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ca0000c90 con 0x7f1ca408fef0 2019-12-05T06:45:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.829+0000 7f1ca8f9d700 1 -- 172.21.15.16:0/2300150877 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ca0001440 con 0x7f1ca408fef0 2019-12-05T06:45:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.829+0000 7f1caba02700 1 -- 172.21.15.16:0/2300150877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca408fef0 msgr2=0x7f1ca408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.829+0000 7f1caba02700 1 --2- 172.21.15.16:0/2300150877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca408fef0 0x7f1ca408d000 secure :-1 s=READY pgs=1290 cs=0 l=1 rx=0x7f1ca000f6c0 tx=0x7f1ca000fa60).stop 2019-12-05T06:45:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.830+0000 7f1caba02700 1 -- 172.21.15.16:0/2300150877 shutdown_connections 2019-12-05T06:45:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.830+0000 7f1caba02700 1 --2- 172.21.15.16:0/2300150877 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1ca408ef20 0x7f1ca408f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.830+0000 7f1caba02700 1 --2- 172.21.15.16:0/2300150877 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca408fef0 0x7f1ca408d000 unknown :-1 s=CLOSED pgs=1290 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:48.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.830+0000 7f1caba02700 1 --2- 172.21.15.16:0/2300150877 >> v2:172.21.15.76:3300/0 conn(0x7f1ca408c6d0 0x7f1ca408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:48.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.831+0000 7f1caba02700 1 -- 172.21.15.16:0/2300150877 shutdown_connections 2019-12-05T06:45:48.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.831+0000 7f1caba02700 1 -- 172.21.15.16:0/2300150877 wait complete. 2019-12-05T06:45:48.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.831+0000 7f1caba02700 1 -- 172.21.15.16:0/2300150877 >> 172.21.15.16:0/2300150877 conn(0x7f1ca4061940 msgr2=0x7f1ca4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.834+0000 7f1caba02700 1 Processor -- start 2019-12-05T06:45:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.834+0000 7f1caba02700 1 -- start start 2019-12-05T06:45:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1caba02700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1ca406dc30 0x7f1ca406e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1caba02700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca4087230 0x7f1ca406e560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:48.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1caba02700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ca4087670 con 0x7f1ca4087a00 2019-12-05T06:45:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1caba02700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca4084810 con 0x7f1ca4087230 2019-12-05T06:45:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1caba02700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ca4085aa0 con 0x7f1ca406dc30 2019-12-05T06:45:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1caa7a0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca4087230 0x7f1ca406e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1ca9f9f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1ca406dc30 0x7f1ca406e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1ca9f9f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1ca406dc30 0x7f1ca406e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:46488/0 (socket says 172.21.15.16:46488) 2019-12-05T06:45:48.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1ca979e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1ca4087a00 0x7f1ca4086a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:38638/0 (socket says 172.21.15.16:38638) 2019-12-05T06:45:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1ca9f9f700 1 -- 172.21.15.16:0/431890540 learned_addr learned my addr 172.21.15.16:0/431890540 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.835+0000 7f1caa7a0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1ca4087230 0x7f1ca406e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60402/0 (socket says 172.21.15.16:60402) 2019-12-05T06:45:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.836+0000 7f1caa7a0700 1 -- 172.21.15.16:0/431890540 >> v2:172.21.15.76:3300/0 conn(0x7f1ca406dc30 msgr2=0x7f1ca406e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.836+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> v2:172.21.15.76:3300/0 conn(0x7f1ca406dc30 0x7f1ca406e020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.836+0000 7f1caa7a0700 1 -- 172.21.15.16:0/431890540 >> v1:172.21.15.76:6789/0 conn(0x7f1ca4087a00 legacy=0x7f1ca4086a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.836+0000 7f1caa7a0700 1 -- 172.21.15.16:0/431890540 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca4086130 con 0x7f1ca4087230 2019-12-05T06:45:48.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.836+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> v2:172.21.15.16:3300/0 conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.836+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> v2:172.21.15.16:3300/0 conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.836+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=READY pgs=1291 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.836+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=THROTTLE_DONE pgs=1291 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.837+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=THROTTLE_DONE pgs=1291 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.837+0000 7f1c967fc700 1 -- 172.21.15.16:0/431890540 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c98000a70 con 0x7f1ca4087230 2019-12-05T06:45:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.837+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=THROTTLE_DONE pgs=1291 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.837+0000 7f1c967fc700 1 -- 172.21.15.16:0/431890540 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c980055d0 con 0x7f1ca4087230 2019-12-05T06:45:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.837+0000 7f1c967fc700 1 -- 172.21.15.16:0/431890540 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c98005ca0 con 0x7f1ca4087230 2019-12-05T06:45:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.837+0000 7f1caba02700 1 -- 172.21.15.16:0/431890540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca4086130 con 0x7f1ca4087230 2019-12-05T06:45:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.837+0000 7f1caba02700 1 -- 172.21.15.16:0/431890540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca4086130 con 0x7f1ca4087230 2019-12-05T06:45:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.838+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=THROTTLE_DONE pgs=1291 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.838+0000 7f1c967fc700 1 -- 172.21.15.16:0/431890540 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1c98000a70 con 0x7f1ca4087230 2019-12-05T06:45:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.838+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=THROTTLE_DONE pgs=1291 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.838+0000 7f1c967fc700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c8c023630 0x7f1c8c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.838+0000 7f1c967fc700 1 -- 172.21.15.16:0/431890540 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f1c98045f90 con 0x7f1ca4087230 2019-12-05T06:45:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.839+0000 7f1ca9f9f700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c8c023630 0x7f1c8c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.839+0000 7f1ca9f9f700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c8c023630 0x7f1c8c025ac0 crc :-1 s=READY pgs=547 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.842+0000 7f1caba02700 1 -- 172.21.15.16:0/431890540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca4086130 con 0x7f1ca4087230 2019-12-05T06:45:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.848+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=THROTTLE_DONE pgs=1291 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:48.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:48.848+0000 7f1c967fc700 1 -- 172.21.15.16:0/431890540 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1c980051d0 con 0x7f1ca4087230 2019-12-05T06:45:49.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.042+0000 7f1caba02700 1 -- 172.21.15.16:0/431890540 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ca4086130 con 0x7f1ca4087230 2019-12-05T06:45:49.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.043+0000 7f1caa7a0700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=THROTTLE_DONE pgs=1291 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.043+0000 7f1c967fc700 1 -- 172.21.15.16:0/431890540 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f1c98000a70 con 0x7f1ca4087230 2019-12-05T06:45:49.044 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 158 2019-12-05T06:45:49.044 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:49.045 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:48.463397+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:49.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 -- 172.21.15.16:0/431890540 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c8c023630 msgr2=0x7f1c8c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c8c023630 0x7f1c8c025ac0 crc :-1 s=READY pgs=547 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 -- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 msgr2=0x7f1ca406e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 0x7f1ca406e560 secure :-1 s=READY pgs=1291 cs=0 l=1 rx=0x7f1c98010390 tx=0x7f1c98010730).stop 2019-12-05T06:45:49.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 -- 172.21.15.16:0/431890540 shutdown_connections 2019-12-05T06:45:49.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1c8c023630 0x7f1c8c025ac0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 --2- 172.21.15.16:0/431890540 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1ca4087230 0x7f1ca406e560 unknown :-1 s=CLOSED pgs=1291 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 --2- 172.21.15.16:0/431890540 >> v2:172.21.15.76:3300/0 conn(0x7f1ca406dc30 0x7f1ca406e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 -- 172.21.15.16:0/431890540 shutdown_connections 2019-12-05T06:45:49.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 -- 172.21.15.16:0/431890540 wait complete. 2019-12-05T06:45:49.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.046+0000 7f1caba02700 1 -- 172.21.15.16:0/431890540 >> 172.21.15.16:0/431890540 conn(0x7f1ca412f780 msgr2=0x7f1ca4096b90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:49.057 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:45:49.057 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:45:49.057 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.198+0000 7fb4a69eb700 1 Processor -- start 2019-12-05T06:45:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.198+0000 7fb4a69eb700 1 -- start start 2019-12-05T06:45:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.199+0000 7fb4a69eb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4a008c6d0 0x7fb4a008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.199+0000 7fb4a69eb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb4a008f720 0x7fb4a008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.199+0000 7fb4a69eb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a008fb70 con 0x7fb4a008fef0 2019-12-05T06:45:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.199+0000 7fb4a69eb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a008df90 con 0x7fb4a008c6d0 2019-12-05T06:45:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.199+0000 7fb4a69eb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a00a3f70 con 0x7fb4a008f720 2019-12-05T06:45:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.199+0000 7fb4a4f88700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4a008c6d0 0x7fb4a008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.199+0000 7fb4a4f88700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4a008c6d0 0x7fb4a008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60406/0 (socket says 172.21.15.16:60406) 2019-12-05T06:45:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.199+0000 7fb4a4f88700 1 -- 172.21.15.16:0/4088806060 learned_addr learned my addr 172.21.15.16:0/4088806060 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.199+0000 7fb4a5789700 1 --2- 172.21.15.16:0/4088806060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb4a008f720 0x7fb4a008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a4f88700 1 -- 172.21.15.16:0/4088806060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb4a008f720 msgr2=0x7fb4a008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a4f88700 1 --2- 172.21.15.16:0/4088806060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb4a008f720 0x7fb4a008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a4f88700 1 -- 172.21.15.16:0/4088806060 >> v1:172.21.15.76:6789/0 conn(0x7fb4a008fef0 legacy=0x7fb4a008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a4f88700 1 -- 172.21.15.16:0/4088806060 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a008d540 con 0x7fb4a008c6d0 2019-12-05T06:45:49.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a5789700 1 --2- 172.21.15.16:0/4088806060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb4a008f720 0x7fb4a008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:45:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a4f88700 1 --2- 172.21.15.16:0/4088806060 >> v2:172.21.15.16:3300/0 conn(0x7fb4a008c6d0 0x7fb4a008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb48800f740 tx=0x7fb48800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a4f88700 1 --2- 172.21.15.16:0/4088806060 >> v2:172.21.15.16:3300/0 conn(0x7fb4a008c6d0 0x7fb4a008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb48800f740 tx=0x7fb48800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a4f88700 1 --2- 172.21.15.16:0/4088806060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a008c6d0 0x7fb4a008cac0 secure :-1 s=READY pgs=1292 cs=0 l=1 rx=0x7fb48800f740 tx=0x7fb48800fae0).ready entity=mon.0 client_cookie=22f339bb10da0834 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a4f88700 1 --2- 172.21.15.16:0/4088806060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a008c6d0 0x7fb4a008cac0 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7fb48800f740 tx=0x7fb48800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a4f88700 1 --2- 172.21.15.16:0/4088806060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a008c6d0 0x7fb4a008cac0 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7fb48800f740 tx=0x7fb48800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb49f7fe700 1 -- 172.21.15.16:0/4088806060 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb488000960 con 0x7fb4a008c6d0 2019-12-05T06:45:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a4f88700 1 --2- 172.21.15.16:0/4088806060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a008c6d0 0x7fb4a008cac0 secure :-1 s=THROTTLE_DONE pgs=1292 cs=0 l=1 rx=0x7fb48800f740 tx=0x7fb48800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb49f7fe700 1 -- 172.21.15.16:0/4088806060 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb488000c90 con 0x7fb4a008c6d0 2019-12-05T06:45:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb49f7fe700 1 -- 172.21.15.16:0/4088806060 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4880013a0 con 0x7fb4a008c6d0 2019-12-05T06:45:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a69eb700 1 -- 172.21.15.16:0/4088806060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a008c6d0 msgr2=0x7fb4a008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.200+0000 7fb4a69eb700 1 --2- 172.21.15.16:0/4088806060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a008c6d0 0x7fb4a008cac0 secure :-1 s=READY pgs=1292 cs=0 l=1 rx=0x7fb48800f740 tx=0x7fb48800fae0).stop 2019-12-05T06:45:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.201+0000 7fb4a69eb700 1 -- 172.21.15.16:0/4088806060 shutdown_connections 2019-12-05T06:45:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.201+0000 7fb4a69eb700 1 --2- 172.21.15.16:0/4088806060 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb4a008f720 0x7fb4a008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.201+0000 7fb4a69eb700 1 --2- 172.21.15.16:0/4088806060 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a008c6d0 0x7fb4a008cac0 unknown :-1 s=CLOSED pgs=1292 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.201+0000 7fb4a69eb700 1 -- 172.21.15.16:0/4088806060 shutdown_connections 2019-12-05T06:45:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.201+0000 7fb4a69eb700 1 -- 172.21.15.16:0/4088806060 wait complete. 2019-12-05T06:45:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.201+0000 7fb4a69eb700 1 -- 172.21.15.16:0/4088806060 >> 172.21.15.16:0/4088806060 conn(0x7fb4a0061940 msgr2=0x7fb4a0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:49.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.204+0000 7fb4a69eb700 1 Processor -- start 2019-12-05T06:45:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.204+0000 7fb4a69eb700 1 -- start start 2019-12-05T06:45:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.205+0000 7fb4a69eb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb4a00879b0 0x7fb4a00871e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.205+0000 7fb4a69eb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb4a009be80 0x7fb4a009c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.205+0000 7fb4a69eb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a0153170 con 0x7fb4a014ca10 2019-12-05T06:45:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.205+0000 7fb4a69eb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a0085a50 con 0x7fb4a00879b0 2019-12-05T06:45:49.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.205+0000 7fb4a69eb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a0085020 con 0x7fb4a009be80 2019-12-05T06:45:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.205+0000 7fb4a4f88700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb4a014ca10 0x7fb4a00869e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45580/0 (socket says 172.21.15.16:45580) 2019-12-05T06:45:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.205+0000 7fb4a4f88700 1 -- 172.21.15.16:0/2726486493 learned_addr learned my addr 172.21.15.16:0/2726486493 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.205+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> v2:172.21.15.16:3300/0 conn(0x7fb4a00879b0 0x7fb4a00871e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.205+0000 7fb4a5789700 1 --2- 172.21.15.16:0/2726486493 >> v2:172.21.15.76:3300/0 conn(0x7fb4a009be80 0x7fb4a009c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.205+0000 7fb49d7fa700 1 -- 172.21.15.16:0/2726486493 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034048160 0 0) 0x7fb4880075d0 con 0x7fb4a014ca10 2019-12-05T06:45:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49d7fa700 1 -- 172.21.15.16:0/2726486493 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb484002e80 con 0x7fb4a014ca10 2019-12-05T06:45:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49ffff700 1 -- 172.21.15.16:0/2726486493 >> v2:172.21.15.76:3300/0 conn(0x7fb4a009be80 msgr2=0x7fb4a009c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> v2:172.21.15.76:3300/0 conn(0x7fb4a009be80 0x7fb4a009c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49ffff700 1 -- 172.21.15.16:0/2726486493 >> v1:172.21.15.16:6789/0 conn(0x7fb4a014ca10 legacy=0x7fb4a00869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49ffff700 1 -- 172.21.15.16:0/2726486493 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a00860e0 con 0x7fb4a00879b0 2019-12-05T06:45:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> v2:172.21.15.16:3300/0 conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> v2:172.21.15.16:3300/0 conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=READY pgs=1294 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=THROTTLE_DONE pgs=1294 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49d7fa700 1 -- 172.21.15.16:0/2726486493 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb490000ce0 con 0x7fb4a00879b0 2019-12-05T06:45:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=THROTTLE_DONE pgs=1294 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=THROTTLE_DONE pgs=1294 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49d7fa700 1 -- 172.21.15.16:0/2726486493 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb490009de0 con 0x7fb4a00879b0 2019-12-05T06:45:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.206+0000 7fb49d7fa700 1 -- 172.21.15.16:0/2726486493 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb49000a4f0 con 0x7fb4a00879b0 2019-12-05T06:45:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.207+0000 7fb4a69eb700 1 -- 172.21.15.16:0/2726486493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a00860e0 con 0x7fb4a00879b0 2019-12-05T06:45:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.207+0000 7fb4a69eb700 1 -- 172.21.15.16:0/2726486493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a00860e0 con 0x7fb4a00879b0 2019-12-05T06:45:49.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.208+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=THROTTLE_DONE pgs=1294 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.208+0000 7fb49d7fa700 1 -- 172.21.15.16:0/2726486493 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb4900099e0 con 0x7fb4a00879b0 2019-12-05T06:45:49.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.208+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=THROTTLE_DONE pgs=1294 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.208+0000 7fb49d7fa700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb484023670 0x7fb484025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:49.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.208+0000 7fb49d7fa700 1 -- 172.21.15.16:0/2726486493 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fb4900460c0 con 0x7fb4a00879b0 2019-12-05T06:45:49.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.209+0000 7fb4a4f88700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb484023670 0x7fb484025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:49.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.209+0000 7fb4a4f88700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb484023670 0x7fb484025ae0 crc :-1 s=READY pgs=548 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:49.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.213+0000 7fb4a69eb700 1 -- 172.21.15.16:0/2726486493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a00860e0 con 0x7fb4a00879b0 2019-12-05T06:45:49.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.217+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=THROTTLE_DONE pgs=1294 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.217+0000 7fb49d7fa700 1 -- 172.21.15.16:0/2726486493 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb490000f90 con 0x7fb4a00879b0 2019-12-05T06:45:49.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.418+0000 7fb4a69eb700 1 -- 172.21.15.16:0/2726486493 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb4a00860e0 con 0x7fb4a00879b0 2019-12-05T06:45:49.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.418+0000 7fb49ffff700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=THROTTLE_DONE pgs=1294 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:49.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.418+0000 7fb49d7fa700 1 -- 172.21.15.16:0/2726486493 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 158 v158) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fb490009990 con 0x7fb4a00879b0 2019-12-05T06:45:49.420 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 158 2019-12-05T06:45:49.420 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:49.421 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":158,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":158,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:48.463397+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:49.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 -- 172.21.15.16:0/2726486493 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb484023670 msgr2=0x7fb484025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb484023670 0x7fb484025ae0 crc :-1 s=READY pgs=548 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 -- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 msgr2=0x7fb4a00871e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:49.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 0x7fb4a00871e0 secure :-1 s=READY pgs=1294 cs=0 l=1 rx=0x7fb490010720 tx=0x7fb490010ac0).stop 2019-12-05T06:45:49.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 -- 172.21.15.16:0/2726486493 shutdown_connections 2019-12-05T06:45:49.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 --2- 172.21.15.16:0/2726486493 >> v2:172.21.15.76:3300/0 conn(0x7fb4a009be80 0x7fb4a009c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb484023670 0x7fb484025ae0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 --2- 172.21.15.16:0/2726486493 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb4a00879b0 0x7fb4a00871e0 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 -- 172.21.15.16:0/2726486493 shutdown_connections 2019-12-05T06:45:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 -- 172.21.15.16:0/2726486493 wait complete. 2019-12-05T06:45:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:49.421+0000 7fb4a69eb700 1 -- 172.21.15.16:0/2726486493 >> 172.21.15.16:0/2726486493 conn(0x7fb4a0098480 msgr2=0x7fb4a0071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:49.432 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 158, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 158, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7401": { "addr": "172.21.15.16:6833/330412031", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 330412031, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 330412031, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7401, "incarnation": 158, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 75, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:45:48.463397+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7401 } } } ], "standbys": [] } 2019-12-05T06:45:51.434 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:51.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.707+0000 7fc512429700 1 Processor -- start 2019-12-05T06:45:51.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.707+0000 7fc512429700 1 -- start start 2019-12-05T06:45:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.708+0000 7fc512429700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc50c08c6d0 0x7fc50c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.708+0000 7fc512429700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c08fef0 0x7fc50c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:51.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.708+0000 7fc512429700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c0958e0 con 0x7fc50c08ef20 2019-12-05T06:45:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.708+0000 7fc512429700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc50c08df90 con 0x7fc50c08c6d0 2019-12-05T06:45:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.708+0000 7fc5109c6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc50c08c6d0 0x7fc50c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:51.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.708+0000 7fc512429700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c0a3f70 con 0x7fc50c08fef0 2019-12-05T06:45:51.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.709+0000 7fc5109c6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc50c08c6d0 0x7fc50c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60426/0 (socket says 172.21.15.16:60426) 2019-12-05T06:45:51.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.709+0000 7fc5109c6700 1 -- 172.21.15.16:0/226181994 learned_addr learned my addr 172.21.15.16:0/226181994 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:51.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.709+0000 7fc50bfff700 1 --2- 172.21.15.16:0/226181994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c08fef0 0x7fc50c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:51.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.709+0000 7fc5109c6700 1 -- 172.21.15.16:0/226181994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c08fef0 msgr2=0x7fc50c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.709+0000 7fc5109c6700 1 --2- 172.21.15.16:0/226181994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c08fef0 0x7fc50c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.709+0000 7fc50b7fe700 1 -- 172.21.15.16:0/226181994 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4109972029 0 0) 0x7fc4fc000a40 con 0x7fc50c08ef20 2019-12-05T06:45:51.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.709+0000 7fc5109c6700 1 -- 172.21.15.16:0/226181994 >> v1:172.21.15.16:6789/0 conn(0x7fc50c08ef20 legacy=0x7fc50c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:51.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.709+0000 7fc5109c6700 1 -- 172.21.15.16:0/226181994 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c08d540 con 0x7fc50c08c6d0 2019-12-05T06:45:51.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc5109c6700 1 --2- 172.21.15.16:0/226181994 >> v2:172.21.15.16:3300/0 conn(0x7fc50c08c6d0 0x7fc50c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc50400fa40 tx=0x7fc50400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc5109c6700 1 --2- 172.21.15.16:0/226181994 >> v2:172.21.15.16:3300/0 conn(0x7fc50c08c6d0 0x7fc50c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc50400fa40 tx=0x7fc50400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc5109c6700 1 --2- 172.21.15.16:0/226181994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c08c6d0 0x7fc50c08cac0 secure :-1 s=READY pgs=1296 cs=0 l=1 rx=0x7fc50400fa40 tx=0x7fc50400fde0).ready entity=mon.0 client_cookie=4bc258f259b4ed8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:51.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc5109c6700 1 --2- 172.21.15.16:0/226181994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c08c6d0 0x7fc50c08cac0 secure :-1 s=THROTTLE_DONE pgs=1296 cs=0 l=1 rx=0x7fc50400fa40 tx=0x7fc50400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.723 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc50b7fe700 1 -- 172.21.15.16:0/226181994 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc504000ce0 con 0x7fc50c08c6d0 2019-12-05T06:45:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc5109c6700 1 --2- 172.21.15.16:0/226181994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c08c6d0 0x7fc50c08cac0 secure :-1 s=THROTTLE_DONE pgs=1296 cs=0 l=1 rx=0x7fc50400fa40 tx=0x7fc50400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc50b7fe700 1 -- 172.21.15.16:0/226181994 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc504001010 con 0x7fc50c08c6d0 2019-12-05T06:45:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc5109c6700 1 --2- 172.21.15.16:0/226181994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c08c6d0 0x7fc50c08cac0 secure :-1 s=THROTTLE_DONE pgs=1296 cs=0 l=1 rx=0x7fc50400fa40 tx=0x7fc50400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc50b7fe700 1 -- 172.21.15.16:0/226181994 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc504007590 con 0x7fc50c08c6d0 2019-12-05T06:45:51.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc512429700 1 -- 172.21.15.16:0/226181994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c08c6d0 msgr2=0x7fc50c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:51.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.710+0000 7fc512429700 1 --2- 172.21.15.16:0/226181994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c08c6d0 0x7fc50c08cac0 secure :-1 s=READY pgs=1296 cs=0 l=1 rx=0x7fc50400fa40 tx=0x7fc50400fde0).stop 2019-12-05T06:45:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.711+0000 7fc512429700 1 -- 172.21.15.16:0/226181994 shutdown_connections 2019-12-05T06:45:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.711+0000 7fc512429700 1 --2- 172.21.15.16:0/226181994 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c08fef0 0x7fc50c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.711+0000 7fc512429700 1 --2- 172.21.15.16:0/226181994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc50c08c6d0 0x7fc50c08cac0 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.711+0000 7fc512429700 1 -- 172.21.15.16:0/226181994 shutdown_connections 2019-12-05T06:45:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.711+0000 7fc512429700 1 -- 172.21.15.16:0/226181994 wait complete. 2019-12-05T06:45:51.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.711+0000 7fc512429700 1 -- 172.21.15.16:0/226181994 >> 172.21.15.16:0/226181994 conn(0x7fc50c061940 msgr2=0x7fc50c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.718+0000 7fc512429700 1 Processor -- start 2019-12-05T06:45:51.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.718+0000 7fc512429700 1 -- start start 2019-12-05T06:45:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.721+0000 7fc512429700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.721+0000 7fc512429700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc50c0879b0 0x7fc50c086f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.721+0000 7fc512429700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc50c09e030 con 0x7fc50c087460 2019-12-05T06:45:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.721+0000 7fc512429700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc50c085020 con 0x7fc50c0879b0 2019-12-05T06:45:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.721+0000 7fc512429700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc50c0852f0 con 0x7fc50c14ca10 2019-12-05T06:45:51.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.721+0000 7fc5109c6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.721+0000 7fc5111c7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc50c087460 0x7fc50c152160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45604/0 (socket says 172.21.15.16:45604) 2019-12-05T06:45:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.721+0000 7fc5111c7700 1 -- 172.21.15.16:0/1445382049 learned_addr learned my addr 172.21.15.16:0/1445382049 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.721+0000 7fc50bfff700 1 --2- 172.21.15.16:0/1445382049 >> v2:172.21.15.76:3300/0 conn(0x7fc50c0879b0 0x7fc50c086f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5109c6700 1 -- 172.21.15.16:0/1445382049 >> v2:172.21.15.76:3300/0 conn(0x7fc50c0879b0 msgr2=0x7fc50c086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> v2:172.21.15.76:3300/0 conn(0x7fc50c0879b0 0x7fc50c086f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5109c6700 1 -- 172.21.15.16:0/1445382049 >> v1:172.21.15.16:6789/0 conn(0x7fc50c087460 legacy=0x7fc50c152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:51.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5097fa700 1 -- 172.21.15.16:0/1445382049 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345077379 0 0) 0x7fc4fc0010f0 con 0x7fc50c087460 2019-12-05T06:45:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5109c6700 1 -- 172.21.15.16:0/1445382049 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc50c0860e0 con 0x7fc50c14ca10 2019-12-05T06:45:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc50bfff700 1 --2- 172.21.15.16:0/1445382049 >> v2:172.21.15.76:3300/0 conn(0x7fc50c0879b0 0x7fc50c086f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:45:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=READY pgs=301 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:51.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5097fa700 1 -- 172.21.15.16:0/1445382049 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5040075b0 con 0x7fc50c14ca10 2019-12-05T06:45:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.722+0000 7fc5097fa700 1 -- 172.21.15.16:0/1445382049 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5040084c0 con 0x7fc50c14ca10 2019-12-05T06:45:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.723+0000 7fc512429700 1 -- 172.21.15.16:0/1445382049 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc50c0860e0 con 0x7fc50c14ca10 2019-12-05T06:45:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.723+0000 7fc5097fa700 1 -- 172.21.15.16:0/1445382049 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5040075b0 con 0x7fc50c14ca10 2019-12-05T06:45:51.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.723+0000 7fc512429700 1 -- 172.21.15.16:0/1445382049 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc50c0860e0 con 0x7fc50c14ca10 2019-12-05T06:45:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.723+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.724+0000 7fc5097fa700 1 -- 172.21.15.16:0/1445382049 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc504005710 con 0x7fc50c14ca10 2019-12-05T06:45:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.724+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.724+0000 7fc5097fa700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc4f0023630 0x7fc4f0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.724+0000 7fc5097fa700 1 -- 172.21.15.16:0/1445382049 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fc5040460b0 con 0x7fc50c14ca10 2019-12-05T06:45:51.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.724+0000 7fc50bfff700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc4f0023630 0x7fc4f0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:51.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.725+0000 7fc50bfff700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc4f0023630 0x7fc4f0025ac0 crc :-1 s=READY pgs=549 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:51.739 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.739+0000 7fc512429700 1 -- 172.21.15.16:0/1445382049 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc50c0860e0 con 0x7fc50c14ca10 2019-12-05T06:45:51.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.746+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.746+0000 7fc5097fa700 1 -- 172.21.15.16:0/1445382049 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc504005500 con 0x7fc50c14ca10 2019-12-05T06:45:51.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.945+0000 7fc512429700 1 -- 172.21.15.16:0/1445382049 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc50c0860e0 con 0x7fc50c14ca10 2019-12-05T06:45:51.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.946+0000 7fc5109c6700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=THROTTLE_DONE pgs=301 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:51.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.946+0000 7fc5097fa700 1 -- 172.21.15.16:0/1445382049 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 160 v160) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fc504005710 con 0x7fc50c14ca10 2019-12-05T06:45:51.949 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 160 2019-12-05T06:45:51.949 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:51.950 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":160,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":160,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:51.676384+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:rejoin","state_seq":18,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:51.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 -- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc4f0023630 msgr2=0x7fc4f0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:51.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc4f0023630 0x7fc4f0025ac0 crc :-1 s=READY pgs=549 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:51.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 -- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 msgr2=0x7fc50c0869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:51.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 secure :-1 s=READY pgs=301 cs=0 l=1 rx=0x7fc504010700 tx=0x7fc504010aa0).stop 2019-12-05T06:45:51.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 -- 172.21.15.16:0/1445382049 shutdown_connections 2019-12-05T06:45:51.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc4f0023630 0x7fc4f0025ac0 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:51.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 --2- 172.21.15.16:0/1445382049 >> v2:172.21.15.76:3300/0 conn(0x7fc50c0879b0 0x7fc50c086f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:51.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 --2- 172.21.15.16:0/1445382049 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc50c14ca10 0x7fc50c0869e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:51.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 -- 172.21.15.16:0/1445382049 shutdown_connections 2019-12-05T06:45:51.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 -- 172.21.15.16:0/1445382049 wait complete. 2019-12-05T06:45:51.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:51.952+0000 7fc512429700 1 -- 172.21.15.16:0/1445382049 >> 172.21.15.16:0/1445382049 conn(0x7fc50c098480 msgr2=0x7fc50c071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:51.970 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:45:53.971 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:45:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.125+0000 7fbb71ea7700 1 Processor -- start 2019-12-05T06:45:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.125+0000 7fbb71ea7700 1 -- start start 2019-12-05T06:45:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.127+0000 7fbb71ea7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbb6c08d2c0 0x7fbb6c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:54.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.127+0000 7fbb71ea7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb6c08fef0 0x7fbb6c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:45:54.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.127+0000 7fbb71ea7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6c0958e0 con 0x7fbb6c08c6d0 2019-12-05T06:45:54.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.127+0000 7fbb71ea7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c08df90 con 0x7fbb6c08fef0 2019-12-05T06:45:54.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.127+0000 7fbb71ea7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c0a3f70 con 0x7fbb6c08d2c0 2019-12-05T06:45:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.127+0000 7fbb70c45700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb6c08fef0 0x7fbb6c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.127+0000 7fbb70c45700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbb6c08fef0 0x7fbb6c08f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60458/0 (socket says 172.21.15.16:60458) 2019-12-05T06:45:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.127+0000 7fbb70c45700 1 -- 172.21.15.16:0/3534206180 learned_addr learned my addr 172.21.15.16:0/3534206180 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:45:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.127+0000 7fbb637fe700 1 --2- 172.21.15.16:0/3534206180 >> v2:172.21.15.76:3300/0 conn(0x7fbb6c08d2c0 0x7fbb6c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb6b7fe700 1 -- 172.21.15.16:0/3534206180 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1867871136 0 0) 0x7fbb5c001400 con 0x7fbb6c08c6d0 2019-12-05T06:45:54.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb70c45700 1 -- 172.21.15.16:0/3534206180 >> v2:172.21.15.76:3300/0 conn(0x7fbb6c08d2c0 msgr2=0x7fbb6c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:45:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb70c45700 1 --2- 172.21.15.16:0/3534206180 >> v2:172.21.15.76:3300/0 conn(0x7fbb6c08d2c0 0x7fbb6c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb70c45700 1 -- 172.21.15.16:0/3534206180 >> v1:172.21.15.76:6789/0 conn(0x7fbb6c08c6d0 legacy=0x7fbb6c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb70c45700 1 -- 172.21.15.16:0/3534206180 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c08e620 con 0x7fbb6c08fef0 2019-12-05T06:45:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb70c45700 1 --2- 172.21.15.16:0/3534206180 >> v2:172.21.15.16:3300/0 conn(0x7fbb6c08fef0 0x7fbb6c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbb6400f730 tx=0x7fbb64007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb70c45700 1 --2- 172.21.15.16:0/3534206180 >> v2:172.21.15.16:3300/0 conn(0x7fbb6c08fef0 0x7fbb6c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbb6400f730 tx=0x7fbb64007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb70c45700 1 --2- 172.21.15.16:0/3534206180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c08fef0 0x7fbb6c08f460 secure :-1 s=READY pgs=1298 cs=0 l=1 rx=0x7fbb6400f730 tx=0x7fbb64007d50).ready entity=mon.0 client_cookie=ac20d49c8e4c487d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb70c45700 1 --2- 172.21.15.16:0/3534206180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c08fef0 0x7fbb6c08f460 secure :-1 s=THROTTLE_DONE pgs=1298 cs=0 l=1 rx=0x7fbb6400f730 tx=0x7fbb64007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb6b7fe700 1 -- 172.21.15.16:0/3534206180 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb64000a70 con 0x7fbb6c08fef0 2019-12-05T06:45:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb70c45700 1 --2- 172.21.15.16:0/3534206180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c08fef0 0x7fbb6c08f460 secure :-1 s=THROTTLE_DONE pgs=1298 cs=0 l=1 rx=0x7fbb6400f730 tx=0x7fbb64007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb6b7fe700 1 -- 172.21.15.16:0/3534206180 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb640010e0 con 0x7fbb6c08fef0 2019-12-05T06:45:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.128+0000 7fbb70c45700 1 --2- 172.21.15.16:0/3534206180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c08fef0 0x7fbb6c08f460 secure :-1 s=THROTTLE_DONE pgs=1298 cs=0 l=1 rx=0x7fbb6400f730 tx=0x7fbb64007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.129+0000 7fbb6b7fe700 1 -- 172.21.15.16:0/3534206180 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb640012d0 con 0x7fbb6c08fef0 2019-12-05T06:45:54.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.129+0000 7fbb71ea7700 1 -- 172.21.15.16:0/3534206180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c08fef0 msgr2=0x7fbb6c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.129+0000 7fbb71ea7700 1 --2- 172.21.15.16:0/3534206180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c08fef0 0x7fbb6c08f460 secure :-1 s=READY pgs=1298 cs=0 l=1 rx=0x7fbb6400f730 tx=0x7fbb64007d50).stop 2019-12-05T06:45:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.129+0000 7fbb71ea7700 1 -- 172.21.15.16:0/3534206180 shutdown_connections 2019-12-05T06:45:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.129+0000 7fbb71ea7700 1 --2- 172.21.15.16:0/3534206180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbb6c08fef0 0x7fbb6c08f460 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.129+0000 7fbb71ea7700 1 --2- 172.21.15.16:0/3534206180 >> v2:172.21.15.76:3300/0 conn(0x7fbb6c08d2c0 0x7fbb6c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:45:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.129+0000 7fbb71ea7700 1 -- 172.21.15.16:0/3534206180 shutdown_connections 2019-12-05T06:45:54.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.129+0000 7fbb71ea7700 1 -- 172.21.15.16:0/3534206180 wait complete. 2019-12-05T06:45:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.129+0000 7fbb71ea7700 1 -- 172.21.15.16:0/3534206180 >> 172.21.15.16:0/3534206180 conn(0x7fbb6c061940 msgr2=0x7fbb6c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.133+0000 7fbb71ea7700 1 Processor -- start 2019-12-05T06:45:54.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.133+0000 7fbb71ea7700 1 -- start start 2019-12-05T06:45:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb71ea7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbb6c14d4a0 0x7fbb6c14d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb71ea7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6c12ef20 con 0x7fbb6c12eae0 2019-12-05T06:45:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb71ea7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb6c12c070 con 0x7fbb6c12f2b0 2019-12-05T06:45:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb71ea7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb6c098f80 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb70c45700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbb6c12eae0 0x7fbb6c19b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45630/0 (socket says 172.21.15.16:45630) 2019-12-05T06:45:54.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb70c45700 1 -- 172.21.15.16:0/66398161 learned_addr learned my addr 172.21.15.16:0/66398161 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:45:54.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> v2:172.21.15.76:3300/0 conn(0x7fbb6c14d4a0 0x7fbb6c14d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:54.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb637fe700 1 -- 172.21.15.16:0/66398161 >> v1:172.21.15.76:6789/0 conn(0x7fbb6c12f2b0 legacy=0x7fbb6c12e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2019-12-05T06:45:54.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb637fe700 1 -- 172.21.15.16:0/66398161 >> v1:172.21.15.76:6789/0 conn(0x7fbb6c12f2b0 legacy=0x7fbb6c12e2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:45:54.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb637fe700 1 --1- 172.21.15.16:0/66398161 >> v1:172.21.15.76:6789/0 conn(0x7fbb6c12f2b0 0x7fbb6c12e2e0 :-1 s=OPENED pgs=887 cs=1 l=1).handle_message read tag failed 2019-12-05T06:45:54.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb637fe700 1 --1- 172.21.15.16:0/66398161 >> v1:172.21.15.76:6789/0 conn(0x7fbb6c12f2b0 0x7fbb6c12e2e0 :-1 s=OPENED pgs=887 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:45:54.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb697fa700 1 -- 172.21.15.16:0/66398161 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661459280 0 0) 0x7fbb64000940 con 0x7fbb6c12eae0 2019-12-05T06:45:54.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.135+0000 7fbb697fa700 1 -- 172.21.15.16:0/66398161 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50002e80 con 0x7fbb6c12eae0 2019-12-05T06:45:54.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb6bfff700 1 -- 172.21.15.16:0/66398161 >> v1:172.21.15.76:6789/0 conn(0x7fbb6c12f2b0 legacy=0x7fbb6c12e2e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:45:54.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb6bfff700 1 -- 172.21.15.16:0/66398161 >> v1:172.21.15.16:6789/0 conn(0x7fbb6c12eae0 legacy=0x7fbb6c19b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb6bfff700 1 -- 172.21.15.16:0/66398161 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb6c12d9e0 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> v2:172.21.15.76:3300/0 conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> v2:172.21.15.76:3300/0 conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=READY pgs=888 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:54.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb697fa700 1 -- 172.21.15.16:0/66398161 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb5c00f640 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb697fa700 1 -- 172.21.15.16:0/66398161 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb5c000df0 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb71ea7700 1 -- 172.21.15.16:0/66398161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb6c12d9e0 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb697fa700 1 -- 172.21.15.16:0/66398161 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb5c00f800 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.136+0000 7fbb71ea7700 1 -- 172.21.15.16:0/66398161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb6c12d9e0 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.137+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.138+0000 7fbb697fa700 1 -- 172.21.15.16:0/66398161 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbb5c00f4f0 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.138+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.138+0000 7fbb697fa700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb50023670 0x7fbb50025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:45:54.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.138+0000 7fbb697fa700 1 -- 172.21.15.16:0/66398161 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fbb5c046040 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.138+0000 7fbb637fe700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb50023670 0x7fbb50025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:45:54.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.139+0000 7fbb637fe700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb50023670 0x7fbb50025ae0 crc :-1 s=READY pgs=550 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:45:54.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.141+0000 7fbb71ea7700 1 -- 172.21.15.16:0/66398161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb6c12d9e0 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.151+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.151+0000 7fbb697fa700 1 -- 172.21.15.16:0/66398161 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbb5c000dd0 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.427+0000 7fbb71ea7700 1 -- 172.21.15.16:0/66398161 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb6c12d9e0 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.428+0000 7fbb6bfff700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=THROTTLE_DONE pgs=888 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:45:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.428+0000 7fbb697fa700 1 -- 172.21.15.16:0/66398161 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fbb5c00f1d0 con 0x7fbb6c14d4a0 2019-12-05T06:45:54.429 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 161 2019-12-05T06:45:54.430 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:45:54.430 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:45:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.431+0000 7fbb71ea7700 1 -- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb50023670 msgr2=0x7fbb50025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.431+0000 7fbb71ea7700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb50023670 0x7fbb50025ae0 crc :-1 s=READY pgs=550 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.431+0000 7fbb71ea7700 1 -- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 msgr2=0x7fbb6c14d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:45:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.431+0000 7fbb71ea7700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 0x7fbb6c14d890 secure :-1 s=READY pgs=888 cs=0 l=1 rx=0x7fbb5c010670 tx=0x7fbb5c010a10).stop 2019-12-05T06:45:54.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.431+0000 7fbb71ea7700 1 -- 172.21.15.16:0/66398161 shutdown_connections 2019-12-05T06:45:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.431+0000 7fbb71ea7700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbb50023670 0x7fbb50025ae0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.431+0000 7fbb71ea7700 1 --2- 172.21.15.16:0/66398161 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb6c14d4a0 0x7fbb6c14d890 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:45:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.431+0000 7fbb71ea7700 1 -- 172.21.15.16:0/66398161 shutdown_connections 2019-12-05T06:45:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.431+0000 7fbb71ea7700 1 -- 172.21.15.16:0/66398161 wait complete. 2019-12-05T06:45:54.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:45:54.431+0000 7fbb71ea7700 1 -- 172.21.15.16:0/66398161 >> 172.21.15.16:0/66398161 conn(0x7fbb6c12d0e0 msgr2=0x7fbb6c135a40 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:45:54.450 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5744 2019-12-05T06:45:54.451 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.b 2019-12-05T06:46:00.452 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:46:00.452 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:46:00.453 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:46:00.466 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:46:00.467 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:46:00.640 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:46:00.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.754+0000 7f294d489700 1 Processor -- start 2019-12-05T06:46:00.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.754+0000 7f294d489700 1 -- start start 2019-12-05T06:46:00.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.756+0000 7f294d489700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f294808c6d0 0x7f294808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:00.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.756+0000 7f294d489700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f294808fef0 0x7f294808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:00.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.756+0000 7f294d489700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294808fb80 con 0x7f294808ef20 2019-12-05T06:46:00.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.756+0000 7f294d489700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f294808df90 con 0x7f294808fef0 2019-12-05T06:46:00.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.756+0000 7f294d489700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29480a3f70 con 0x7f294808c6d0 2019-12-05T06:46:00.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.756+0000 7f2946ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f294808fef0 0x7f294808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:00.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.756+0000 7f2946ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f294808fef0 0x7f294808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60494/0 (socket says 172.21.15.16:60494) 2019-12-05T06:46:00.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.756+0000 7f2946ffd700 1 -- 172.21.15.16:0/1450299886 learned_addr learned my addr 172.21.15.16:0/1450299886 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:46:00.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.756+0000 7f29477fe700 1 --2- 172.21.15.16:0/1450299886 >> v2:172.21.15.76:3300/0 conn(0x7f294808c6d0 0x7f294808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:00.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.757+0000 7f2946ffd700 1 -- 172.21.15.16:0/1450299886 >> v2:172.21.15.76:3300/0 conn(0x7f294808c6d0 msgr2=0x7f294808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:46:00.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.757+0000 7f29467fc700 1 -- 172.21.15.16:0/1450299886 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518926858 0 0) 0x7f2940000d30 con 0x7f294808ef20 2019-12-05T06:46:00.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.757+0000 7f2946ffd700 1 --2- 172.21.15.16:0/1450299886 >> v2:172.21.15.76:3300/0 conn(0x7f294808c6d0 0x7f294808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:00.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.757+0000 7f2946ffd700 1 -- 172.21.15.16:0/1450299886 >> v1:172.21.15.76:6789/0 conn(0x7f294808ef20 legacy=0x7f294808f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:00.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.757+0000 7f2946ffd700 1 -- 172.21.15.16:0/1450299886 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294808d540 con 0x7f294808fef0 2019-12-05T06:46:00.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.757+0000 7f29477fe700 1 --2- 172.21.15.16:0/1450299886 >> v2:172.21.15.76:3300/0 conn(0x7f294808c6d0 0x7f294808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:46:00.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.757+0000 7f2946ffd700 1 --2- 172.21.15.16:0/1450299886 >> v2:172.21.15.16:3300/0 conn(0x7f294808fef0 0x7f294808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f293400f740 tx=0x7f293400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.757+0000 7f2946ffd700 1 --2- 172.21.15.16:0/1450299886 >> v2:172.21.15.16:3300/0 conn(0x7f294808fef0 0x7f294808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f293400f740 tx=0x7f293400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.757+0000 7f2946ffd700 1 --2- 172.21.15.16:0/1450299886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294808fef0 0x7f294808d000 secure :-1 s=READY pgs=1302 cs=0 l=1 rx=0x7f293400f740 tx=0x7f293400fae0).ready entity=mon.0 client_cookie=d9d6b400562fb995 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:00.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.758+0000 7f2946ffd700 1 --2- 172.21.15.16:0/1450299886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294808fef0 0x7f294808d000 secure :-1 s=THROTTLE_DONE pgs=1302 cs=0 l=1 rx=0x7f293400f740 tx=0x7f293400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.758+0000 7f2946ffd700 1 --2- 172.21.15.16:0/1450299886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294808fef0 0x7f294808d000 secure :-1 s=THROTTLE_DONE pgs=1302 cs=0 l=1 rx=0x7f293400f740 tx=0x7f293400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.758+0000 7f29467fc700 1 -- 172.21.15.16:0/1450299886 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2934000960 con 0x7f294808fef0 2019-12-05T06:46:00.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.758+0000 7f2946ffd700 1 --2- 172.21.15.16:0/1450299886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294808fef0 0x7f294808d000 secure :-1 s=THROTTLE_DONE pgs=1302 cs=0 l=1 rx=0x7f293400f740 tx=0x7f293400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.758+0000 7f29467fc700 1 -- 172.21.15.16:0/1450299886 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2934000c90 con 0x7f294808fef0 2019-12-05T06:46:00.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.758+0000 7f29467fc700 1 -- 172.21.15.16:0/1450299886 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29340013a0 con 0x7f294808fef0 2019-12-05T06:46:00.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.758+0000 7f294d489700 1 -- 172.21.15.16:0/1450299886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294808fef0 msgr2=0x7f294808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:00.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.758+0000 7f294d489700 1 --2- 172.21.15.16:0/1450299886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294808fef0 0x7f294808d000 secure :-1 s=READY pgs=1302 cs=0 l=1 rx=0x7f293400f740 tx=0x7f293400fae0).stop 2019-12-05T06:46:00.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.760+0000 7f294d489700 1 -- 172.21.15.16:0/1450299886 shutdown_connections 2019-12-05T06:46:00.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.760+0000 7f294d489700 1 --2- 172.21.15.16:0/1450299886 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f294808fef0 0x7f294808d000 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:00.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.760+0000 7f294d489700 1 --2- 172.21.15.16:0/1450299886 >> v2:172.21.15.76:3300/0 conn(0x7f294808c6d0 0x7f294808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:46:00.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.760+0000 7f294d489700 1 -- 172.21.15.16:0/1450299886 shutdown_connections 2019-12-05T06:46:00.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.760+0000 7f294d489700 1 -- 172.21.15.16:0/1450299886 wait complete. 2019-12-05T06:46:00.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.760+0000 7f294d489700 1 -- 172.21.15.16:0/1450299886 >> 172.21.15.16:0/1450299886 conn(0x7f2948061940 msgr2=0x7f2948096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:00.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.766+0000 7f294d489700 1 Processor -- start 2019-12-05T06:46:00.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.766+0000 7f294d489700 1 -- start start 2019-12-05T06:46:00.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f294d489700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2948084e90 0x7f2948088790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:00.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f294d489700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2948154c50 0x7f2948157050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:00.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f294d489700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2948085ae0 con 0x7f2948089760 2019-12-05T06:46:00.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f294d489700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2948087800 con 0x7f2948084e90 2019-12-05T06:46:00.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f294d489700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2948086dd0 con 0x7f2948154c50 2019-12-05T06:46:00.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f29477fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2948084e90 0x7f2948088790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:00.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f29477fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2948084e90 0x7f2948088790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60496/0 (socket says 172.21.15.16:60496) 2019-12-05T06:46:00.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f29477fe700 1 -- 172.21.15.16:0/802372895 learned_addr learned my addr 172.21.15.16:0/802372895 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:46:00.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f2947fff700 1 --2- 172.21.15.16:0/802372895 >> v2:172.21.15.76:3300/0 conn(0x7f2948154c50 0x7f2948157050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:00.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f29477fe700 1 -- 172.21.15.16:0/802372895 >> v2:172.21.15.76:3300/0 conn(0x7f2948154c50 msgr2=0x7f2948157050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:00.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> v2:172.21.15.76:3300/0 conn(0x7f2948154c50 0x7f2948157050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:00.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f29477fe700 1 -- 172.21.15.16:0/802372895 >> v1:172.21.15.76:6789/0 conn(0x7f2948089760 legacy=0x7f2948088cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:00.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.768+0000 7f29477fe700 1 -- 172.21.15.16:0/802372895 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948087e90 con 0x7f2948084e90 2019-12-05T06:46:00.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f2947fff700 1 --2- 172.21.15.16:0/802372895 >> v2:172.21.15.76:3300/0 conn(0x7f2948154c50 0x7f2948157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:46:00.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> v2:172.21.15.16:3300/0 conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> v2:172.21.15.16:3300/0 conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=READY pgs=1303 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:00.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=THROTTLE_DONE pgs=1303 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=THROTTLE_DONE pgs=1303 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=THROTTLE_DONE pgs=1303 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f293ffff700 1 -- 172.21.15.16:0/802372895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f293800a8c0 con 0x7f2948084e90 2019-12-05T06:46:00.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f293ffff700 1 -- 172.21.15.16:0/802372895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2938009770 con 0x7f2948084e90 2019-12-05T06:46:00.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f293ffff700 1 -- 172.21.15.16:0/802372895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2938009e40 con 0x7f2948084e90 2019-12-05T06:46:00.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f294d489700 1 -- 172.21.15.16:0/802372895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2948087e90 con 0x7f2948084e90 2019-12-05T06:46:00.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.769+0000 7f294d489700 1 -- 172.21.15.16:0/802372895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2948087e90 con 0x7f2948084e90 2019-12-05T06:46:00.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.770+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=THROTTLE_DONE pgs=1303 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.771+0000 7f293ffff700 1 -- 172.21.15.16:0/802372895 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2938009960 con 0x7f2948084e90 2019-12-05T06:46:00.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.771+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=THROTTLE_DONE pgs=1303 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.772+0000 7f293ffff700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2924023630 0x7f2924025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:00.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.772+0000 7f293ffff700 1 -- 172.21.15.16:0/802372895 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f2938045df0 con 0x7f2948084e90 2019-12-05T06:46:00.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.772+0000 7f2946ffd700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2924023630 0x7f2924025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:00.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.772+0000 7f2946ffd700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2924023630 0x7f2924025ac0 crc :-1 s=READY pgs=551 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:00.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.785+0000 7f294d489700 1 -- 172.21.15.16:0/802372895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2948087e90 con 0x7f2948084e90 2019-12-05T06:46:00.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.793+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=THROTTLE_DONE pgs=1303 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:00.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:00.794+0000 7f293ffff700 1 -- 172.21.15.16:0/802372895 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f293800a560 con 0x7f2948084e90 2019-12-05T06:46:01.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.037+0000 7f294d489700 1 -- 172.21.15.16:0/802372895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2948087e90 con 0x7f2948084e90 2019-12-05T06:46:01.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.037+0000 7f29477fe700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=THROTTLE_DONE pgs=1303 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:01.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.038+0000 7f293ffff700 1 -- 172.21.15.16:0/802372895 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f2938009930 con 0x7f2948084e90 2019-12-05T06:46:01.038 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 161 2019-12-05T06:46:01.038 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:46:01.039 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:46:01.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.040+0000 7f294d489700 1 -- 172.21.15.16:0/802372895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2924023630 msgr2=0x7f2924025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:01.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.040+0000 7f294d489700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2924023630 0x7f2924025ac0 crc :-1 s=READY pgs=551 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:01.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.040+0000 7f294d489700 1 -- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 msgr2=0x7f2948088790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:01.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.040+0000 7f294d489700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 0x7f2948088790 secure :-1 s=READY pgs=1303 cs=0 l=1 rx=0x7f2938010410 tx=0x7f29380107b0).stop 2019-12-05T06:46:01.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.040+0000 7f294d489700 1 -- 172.21.15.16:0/802372895 shutdown_connections 2019-12-05T06:46:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.040+0000 7f294d489700 1 --2- 172.21.15.16:0/802372895 >> v2:172.21.15.76:3300/0 conn(0x7f2948154c50 0x7f2948157050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.040+0000 7f294d489700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2924023630 0x7f2924025ac0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.040+0000 7f294d489700 1 --2- 172.21.15.16:0/802372895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2948084e90 0x7f2948088790 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.041+0000 7f294d489700 1 -- 172.21.15.16:0/802372895 shutdown_connections 2019-12-05T06:46:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.041+0000 7f294d489700 1 -- 172.21.15.16:0/802372895 wait complete. 2019-12-05T06:46:01.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:01.041+0000 7f294d489700 1 -- 172.21.15.16:0/802372895 >> 172.21.15.16:0/802372895 conn(0x7f294806ecf0 msgr2=0x7f29480706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:01.051 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:46:03.064 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:46:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.332+0000 7fc56b73d700 1 Processor -- start 2019-12-05T06:46:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.333+0000 7fc56b73d700 1 -- start start 2019-12-05T06:46:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.334+0000 7fc56b73d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc56408d2c0 0x7fc56408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:03.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.334+0000 7fc56b73d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5640958e0 con 0x7fc56408c6d0 2019-12-05T06:46:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.334+0000 7fc56b73d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5640629e0 con 0x7fc56408fef0 2019-12-05T06:46:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.334+0000 7fc56b73d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56408df90 con 0x7fc56408d2c0 2019-12-05T06:46:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.334+0000 7fc5694d9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc56408d2c0 0x7fc56408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.334+0000 7fc569cda700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc56408c6d0 0x7fc56408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45688/0 (socket says 172.21.15.16:45688) 2019-12-05T06:46:03.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.334+0000 7fc5694d9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc56408d2c0 0x7fc56408ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60524/0 (socket says 172.21.15.16:60524) 2019-12-05T06:46:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.335+0000 7fc569cda700 1 -- 172.21.15.16:0/2053960843 learned_addr learned my addr 172.21.15.16:0/2053960843 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:46:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.335+0000 7fc5694d9700 1 -- 172.21.15.16:0/2053960843 >> v1:172.21.15.76:6789/0 conn(0x7fc56408fef0 legacy=0x7fc56408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.335+0000 7fc5694d9700 1 -- 172.21.15.16:0/2053960843 >> v1:172.21.15.16:6789/0 conn(0x7fc56408c6d0 legacy=0x7fc56408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.335+0000 7fc5694d9700 1 -- 172.21.15.16:0/2053960843 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56408e620 con 0x7fc56408d2c0 2019-12-05T06:46:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.335+0000 7fc568cd8700 1 -- 172.21.15.16:0/2053960843 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4241222694 0 0) 0x7fc55c000d30 con 0x7fc56408fef0 2019-12-05T06:46:03.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.335+0000 7fc5694d9700 1 --2- 172.21.15.16:0/2053960843 >> v2:172.21.15.16:3300/0 conn(0x7fc56408d2c0 0x7fc56408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc55800af50 tx=0x7fc55800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.336+0000 7fc5694d9700 1 --2- 172.21.15.16:0/2053960843 >> v2:172.21.15.16:3300/0 conn(0x7fc56408d2c0 0x7fc56408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc55800af50 tx=0x7fc55800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.336+0000 7fc5694d9700 1 --2- 172.21.15.16:0/2053960843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56408d2c0 0x7fc56408ef20 secure :-1 s=READY pgs=1305 cs=0 l=1 rx=0x7fc55800af50 tx=0x7fc55800b760).ready entity=mon.0 client_cookie=4afdf6eda2c63cac server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.336+0000 7fc5694d9700 1 --2- 172.21.15.16:0/2053960843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56408d2c0 0x7fc56408ef20 secure :-1 s=THROTTLE_DONE pgs=1305 cs=0 l=1 rx=0x7fc55800af50 tx=0x7fc55800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.336+0000 7fc5694d9700 1 --2- 172.21.15.16:0/2053960843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56408d2c0 0x7fc56408ef20 secure :-1 s=THROTTLE_DONE pgs=1305 cs=0 l=1 rx=0x7fc55800af50 tx=0x7fc55800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.336+0000 7fc568cd8700 1 -- 172.21.15.16:0/2053960843 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc558000960 con 0x7fc56408d2c0 2019-12-05T06:46:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.336+0000 7fc5694d9700 1 --2- 172.21.15.16:0/2053960843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56408d2c0 0x7fc56408ef20 secure :-1 s=THROTTLE_DONE pgs=1305 cs=0 l=1 rx=0x7fc55800af50 tx=0x7fc55800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.336+0000 7fc568cd8700 1 -- 172.21.15.16:0/2053960843 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5580098e0 con 0x7fc56408d2c0 2019-12-05T06:46:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.336+0000 7fc568cd8700 1 -- 172.21.15.16:0/2053960843 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc55800a050 con 0x7fc56408d2c0 2019-12-05T06:46:03.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.336+0000 7fc56b73d700 1 -- 172.21.15.16:0/2053960843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56408d2c0 msgr2=0x7fc56408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.336+0000 7fc56b73d700 1 --2- 172.21.15.16:0/2053960843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56408d2c0 0x7fc56408ef20 secure :-1 s=READY pgs=1305 cs=0 l=1 rx=0x7fc55800af50 tx=0x7fc55800b760).stop 2019-12-05T06:46:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.338+0000 7fc56b73d700 1 -- 172.21.15.16:0/2053960843 shutdown_connections 2019-12-05T06:46:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.338+0000 7fc56b73d700 1 --2- 172.21.15.16:0/2053960843 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc56408d2c0 0x7fc56408ef20 unknown :-1 s=CLOSED pgs=1305 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.338+0000 7fc56b73d700 1 -- 172.21.15.16:0/2053960843 shutdown_connections 2019-12-05T06:46:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.338+0000 7fc56b73d700 1 -- 172.21.15.16:0/2053960843 wait complete. 2019-12-05T06:46:03.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.338+0000 7fc56b73d700 1 -- 172.21.15.16:0/2053960843 >> 172.21.15.16:0/2053960843 conn(0x7fc564061940 msgr2=0x7fc564096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:03.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.342+0000 7fc56b73d700 1 Processor -- start 2019-12-05T06:46:03.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.342+0000 7fc56b73d700 1 -- start start 2019-12-05T06:46:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc56b73d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc564096e90 0x7fc56409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:03.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc56b73d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc56b73d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc564084b90 con 0x7fc564084f30 2019-12-05T06:46:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc56b73d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc56409a450 con 0x7fc564096e90 2019-12-05T06:46:03.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc56b73d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc564099a20 con 0x7fc56409b920 2019-12-05T06:46:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc5694d9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc564084f30 0x7fc564097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45698/0 (socket says 172.21.15.16:45698) 2019-12-05T06:46:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc5694d9700 1 -- 172.21.15.16:0/2088476269 learned_addr learned my addr 172.21.15.16:0/2088476269 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:46:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:03.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc569cda700 1 --2- 172.21.15.16:0/2088476269 >> v2:172.21.15.76:3300/0 conn(0x7fc564096e90 0x7fc56409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:03.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc5627fc700 1 -- 172.21.15.16:0/2088476269 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3984756413 0 0) 0x7fc55800abb0 con 0x7fc564084f30 2019-12-05T06:46:03.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.344+0000 7fc5627fc700 1 -- 172.21.15.16:0/2088476269 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc544002e80 con 0x7fc564084f30 2019-12-05T06:46:03.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.345+0000 7fc56a4db700 1 -- 172.21.15.16:0/2088476269 >> v2:172.21.15.76:3300/0 conn(0x7fc564096e90 msgr2=0x7fc56409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:03.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.345+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> v2:172.21.15.76:3300/0 conn(0x7fc564096e90 0x7fc56409b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:03.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.345+0000 7fc5627fc700 1 -- 172.21.15.16:0/2088476269 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1573977179 0 0) 0x7fc558000940 con 0x7fc564084f30 2019-12-05T06:46:03.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.345+0000 7fc56a4db700 1 -- 172.21.15.16:0/2088476269 >> v1:172.21.15.16:6789/0 conn(0x7fc564084f30 legacy=0x7fc564097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:03.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.345+0000 7fc56a4db700 1 -- 172.21.15.16:0/2088476269 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc56409aae0 con 0x7fc56409b920 2019-12-05T06:46:03.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.345+0000 7fc569cda700 1 --2- 172.21.15.16:0/2088476269 >> v2:172.21.15.76:3300/0 conn(0x7fc564096e90 0x7fc56409b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:46:03.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.345+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.345+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.345+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=READY pgs=303 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.345+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.346+0000 7fc5627fc700 1 -- 172.21.15.16:0/2088476269 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc55c000cc0 con 0x7fc56409b920 2019-12-05T06:46:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.346+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.346+0000 7fc5627fc700 1 -- 172.21.15.16:0/2088476269 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc55c009d10 con 0x7fc56409b920 2019-12-05T06:46:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.346+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.346+0000 7fc56b73d700 1 -- 172.21.15.16:0/2088476269 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc56409aae0 con 0x7fc56409b920 2019-12-05T06:46:03.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.346+0000 7fc5627fc700 1 -- 172.21.15.16:0/2088476269 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc55c009f00 con 0x7fc56409b920 2019-12-05T06:46:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.346+0000 7fc56b73d700 1 -- 172.21.15.16:0/2088476269 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc56409aae0 con 0x7fc56409b920 2019-12-05T06:46:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.347+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.347+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.347+0000 7fc5627fc700 1 -- 172.21.15.16:0/2088476269 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc55c009f00 con 0x7fc56409b920 2019-12-05T06:46:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.347+0000 7fc5627fc700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc544023680 0x7fc544025b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:03.400 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.347+0000 7fc5627fc700 1 -- 172.21.15.16:0/2088476269 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fc55c046330 con 0x7fc56409b920 2019-12-05T06:46:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.348+0000 7fc569cda700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc544023680 0x7fc544025b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.348+0000 7fc569cda700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc544023680 0x7fc544025b10 crc :-1 s=READY pgs=552 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:03.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.364+0000 7fc56b73d700 1 -- 172.21.15.16:0/2088476269 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc56409aae0 con 0x7fc56409b920 2019-12-05T06:46:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.374+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.374+0000 7fc5627fc700 1 -- 172.21.15.16:0/2088476269 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc55c009910 con 0x7fc56409b920 2019-12-05T06:46:03.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.626+0000 7fc56b73d700 1 -- 172.21.15.16:0/2088476269 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc56409aae0 con 0x7fc56409b920 2019-12-05T06:46:03.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.627+0000 7fc56a4db700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=THROTTLE_DONE pgs=303 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:03.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.627+0000 7fc5627fc700 1 -- 172.21.15.16:0/2088476269 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 161 v161) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fc55c009cd0 con 0x7fc56409b920 2019-12-05T06:46:03.629 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:46:03.629 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":161,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:46:03.630 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 161 2019-12-05T06:46:03.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.632+0000 7fc56b73d700 1 -- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc544023680 msgr2=0x7fc544025b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:03.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.632+0000 7fc56b73d700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc544023680 0x7fc544025b10 crc :-1 s=READY pgs=552 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:03.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.632+0000 7fc56b73d700 1 -- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 msgr2=0x7fc564097a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:03.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.632+0000 7fc56b73d700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 secure :-1 s=READY pgs=303 cs=0 l=1 rx=0x7fc55c010730 tx=0x7fc55c010ad0).stop 2019-12-05T06:46:03.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.632+0000 7fc56b73d700 1 -- 172.21.15.16:0/2088476269 shutdown_connections 2019-12-05T06:46:03.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.632+0000 7fc56b73d700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc544023680 0x7fc544025b10 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:03.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.632+0000 7fc56b73d700 1 --2- 172.21.15.16:0/2088476269 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc56409b920 0x7fc564097a80 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:03.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.632+0000 7fc56b73d700 1 --2- 172.21.15.16:0/2088476269 >> v2:172.21.15.76:3300/0 conn(0x7fc564096e90 0x7fc56409b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:03.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.632+0000 7fc56b73d700 1 -- 172.21.15.16:0/2088476269 shutdown_connections 2019-12-05T06:46:03.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.632+0000 7fc56b73d700 1 -- 172.21.15.16:0/2088476269 wait complete. 2019-12-05T06:46:03.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:03.633+0000 7fc56b73d700 1 -- 172.21.15.16:0/2088476269 >> 172.21.15.16:0/2088476269 conn(0x7fc5640714f0 msgr2=0x7fc56409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:03.642 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:46:05.643 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:46:05.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.911+0000 7f2eeff28700 1 Processor -- start 2019-12-05T06:46:05.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.911+0000 7f2eeff28700 1 -- start start 2019-12-05T06:46:05.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.913+0000 7f2eeff28700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ee808d2c0 0x7f2ee808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:46:05.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.913+0000 7f2eeff28700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee80958e0 con 0x7f2ee808c6d0 2019-12-05T06:46:05.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.913+0000 7f2eeff28700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee80629e0 con 0x7f2ee808fef0 2019-12-05T06:46:05.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.913+0000 7f2eeff28700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee808df90 con 0x7f2ee808d2c0 2019-12-05T06:46:05.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.913+0000 7f2eedcc4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ee808d2c0 0x7f2ee808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:05.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.913+0000 7f2eee4c5700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2ee808c6d0 0x7f2ee808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:45720/0 (socket says 172.21.15.16:45720) 2019-12-05T06:46:05.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.913+0000 7f2eee4c5700 1 -- 172.21.15.16:0/3875465556 learned_addr learned my addr 172.21.15.16:0/3875465556 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:46:05.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.914+0000 7f2eedcc4700 1 -- 172.21.15.16:0/3875465556 >> v1:172.21.15.76:6789/0 conn(0x7f2ee808fef0 legacy=0x7f2ee808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:05.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.914+0000 7f2eedcc4700 1 -- 172.21.15.16:0/3875465556 >> v1:172.21.15.16:6789/0 conn(0x7f2ee808c6d0 legacy=0x7f2ee808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:05.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.914+0000 7f2eed4c3700 1 -- 172.21.15.16:0/3875465556 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 651928698 0 0) 0x7f2ee00009b0 con 0x7f2ee808fef0 2019-12-05T06:46:05.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.914+0000 7f2eedcc4700 1 -- 172.21.15.16:0/3875465556 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee808e620 con 0x7f2ee808d2c0 2019-12-05T06:46:05.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.914+0000 7f2eedcc4700 1 --2- 172.21.15.16:0/3875465556 >> v2:172.21.15.16:3300/0 conn(0x7f2ee808d2c0 0x7f2ee808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2edc00b2d0 tx=0x7f2edc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.915+0000 7f2eedcc4700 1 --2- 172.21.15.16:0/3875465556 >> v2:172.21.15.16:3300/0 conn(0x7f2ee808d2c0 0x7f2ee808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2edc00b2d0 tx=0x7f2edc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.915+0000 7f2eedcc4700 1 --2- 172.21.15.16:0/3875465556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee808d2c0 0x7f2ee808ef20 secure :-1 s=READY pgs=1308 cs=0 l=1 rx=0x7f2edc00b2d0 tx=0x7f2edc00bae0).ready entity=mon.0 client_cookie=421fe9f471f2c6f4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:05.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.915+0000 7f2eedcc4700 1 --2- 172.21.15.16:0/3875465556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee808d2c0 0x7f2ee808ef20 secure :-1 s=THROTTLE_DONE pgs=1308 cs=0 l=1 rx=0x7f2edc00b2d0 tx=0x7f2edc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.915+0000 7f2eedcc4700 1 --2- 172.21.15.16:0/3875465556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee808d2c0 0x7f2ee808ef20 secure :-1 s=THROTTLE_DONE pgs=1308 cs=0 l=1 rx=0x7f2edc00b2d0 tx=0x7f2edc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.915+0000 7f2eed4c3700 1 -- 172.21.15.16:0/3875465556 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2edc000ce0 con 0x7f2ee808d2c0 2019-12-05T06:46:05.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.915+0000 7f2eedcc4700 1 --2- 172.21.15.16:0/3875465556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee808d2c0 0x7f2ee808ef20 secure :-1 s=THROTTLE_DONE pgs=1308 cs=0 l=1 rx=0x7f2edc00b2d0 tx=0x7f2edc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.915+0000 7f2eed4c3700 1 -- 172.21.15.16:0/3875465556 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2edc009be0 con 0x7f2ee808d2c0 2019-12-05T06:46:05.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.915+0000 7f2eed4c3700 1 -- 172.21.15.16:0/3875465556 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2edc00a2b0 con 0x7f2ee808d2c0 2019-12-05T06:46:05.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.915+0000 7f2eeff28700 1 -- 172.21.15.16:0/3875465556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee808d2c0 msgr2=0x7f2ee808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:05.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.915+0000 7f2eeff28700 1 --2- 172.21.15.16:0/3875465556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee808d2c0 0x7f2ee808ef20 secure :-1 s=READY pgs=1308 cs=0 l=1 rx=0x7f2edc00b2d0 tx=0x7f2edc00bae0).stop 2019-12-05T06:46:05.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.917+0000 7f2eeff28700 1 -- 172.21.15.16:0/3875465556 shutdown_connections 2019-12-05T06:46:05.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.917+0000 7f2eeff28700 1 --2- 172.21.15.16:0/3875465556 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee808d2c0 0x7f2ee808ef20 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:05.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.917+0000 7f2eeff28700 1 -- 172.21.15.16:0/3875465556 shutdown_connections 2019-12-05T06:46:05.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.918+0000 7f2eeff28700 1 -- 172.21.15.16:0/3875465556 wait complete. 2019-12-05T06:46:05.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.918+0000 7f2eeff28700 1 -- 172.21.15.16:0/3875465556 >> 172.21.15.16:0/3875465556 conn(0x7f2ee8061940 msgr2=0x7f2ee8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:05.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.923+0000 7f2eeff28700 1 Processor -- start 2019-12-05T06:46:05.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.923+0000 7f2eeff28700 1 -- start start 2019-12-05T06:46:05.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.924+0000 7f2eeff28700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ee814f5b0 0x7f2ee814f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:05.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.924+0000 7f2eeff28700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2ee8098b00 0x7f2ee81592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:05.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2eeff28700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ee8089fb0 con 0x7f2ee80992d0 2019-12-05T06:46:05.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2eeff28700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ee8097370 con 0x7f2ee814f5b0 2019-12-05T06:46:05.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2eeff28700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ee808ad00 con 0x7f2ee8098b00 2019-12-05T06:46:05.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2eee4c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ee814f5b0 0x7f2ee814f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2eee4c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2ee814f5b0 0x7f2ee814f9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:60558/0 (socket says 172.21.15.16:60558) 2019-12-05T06:46:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2eee4c5700 1 -- 172.21.15.16:0/64144748 learned_addr learned my addr 172.21.15.16:0/64144748 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:46:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2eeecc6700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2ee8098b00 0x7f2ee81592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2eee4c5700 1 -- 172.21.15.16:0/64144748 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2ee8098b00 msgr2=0x7f2ee81592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2ee8098b00 0x7f2ee81592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:05.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2eee4c5700 1 -- 172.21.15.16:0/64144748 >> v1:172.21.15.76:6789/0 conn(0x7f2ee80992d0 legacy=0x7f2ee8098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:05.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.925+0000 7f2ee6ffd700 1 -- 172.21.15.16:0/64144748 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1471247806 0 0) 0x7f2edc00b750 con 0x7f2ee80992d0 2019-12-05T06:46:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2eee4c5700 1 -- 172.21.15.16:0/64144748 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ee8097a00 con 0x7f2ee814f5b0 2019-12-05T06:46:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> v2:172.21.15.16:3300/0 conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> v2:172.21.15.16:3300/0 conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:05.965 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2ee6ffd700 1 -- 172.21.15.16:0/64144748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ed8000ed0 con 0x7f2ee814f5b0 2019-12-05T06:46:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2ee6ffd700 1 -- 172.21.15.16:0/64144748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ed8004430 con 0x7f2ee814f5b0 2019-12-05T06:46:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2eeff28700 1 -- 172.21.15.16:0/64144748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ee8097a00 con 0x7f2ee814f5b0 2019-12-05T06:46:05.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2ee6ffd700 1 -- 172.21.15.16:0/64144748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ed8004620 con 0x7f2ee814f5b0 2019-12-05T06:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.926+0000 7f2eeff28700 1 -- 172.21.15.16:0/64144748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ee8097a00 con 0x7f2ee814f5b0 2019-12-05T06:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.927+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.928+0000 7f2ee6ffd700 1 -- 172.21.15.16:0/64144748 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2ed8004620 con 0x7f2ee814f5b0 2019-12-05T06:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.928+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.929+0000 7f2ee6ffd700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ec8023630 0x7f2ec8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.929+0000 7f2ee6ffd700 1 -- 172.21.15.16:0/64144748 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f2ed8045e40 con 0x7f2ee814f5b0 2019-12-05T06:46:05.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.929+0000 7f2eedcc4700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ec8023630 0x7f2ec8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:46:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.930+0000 7f2eedcc4700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ec8023630 0x7f2ec8025ac0 crc :-1 s=READY pgs=554 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:46:05.968 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.943+0000 7f2eeff28700 1 -- 172.21.15.16:0/64144748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ee8097a00 con 0x7f2ee814f5b0 2019-12-05T06:46:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.952+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:05.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:05.952+0000 7f2ee6ffd700 1 -- 172.21.15.16:0/64144748 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2ed80012b0 con 0x7f2ee814f5b0 2019-12-05T06:46:06.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.211+0000 7f2eeff28700 1 -- 172.21.15.16:0/64144748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ee8097a00 con 0x7f2ee814f5b0 2019-12-05T06:46:06.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.212+0000 7f2eee4c5700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=THROTTLE_DONE pgs=1309 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:46:06.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.212+0000 7f2ee6ffd700 1 -- 172.21.15.16:0/64144748 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f2ed8004620 con 0x7f2ee814f5b0 2019-12-05T06:46:06.213 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 162 2019-12-05T06:46:06.213 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:46:06.213 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:46:06.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.214+0000 7f2eeff28700 1 -- 172.21.15.16:0/64144748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ec8023630 msgr2=0x7f2ec8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:06.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.214+0000 7f2eeff28700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ec8023630 0x7f2ec8025ac0 crc :-1 s=READY pgs=554 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.214+0000 7f2eeff28700 1 -- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 msgr2=0x7f2ee814f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:46:06.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.214+0000 7f2eeff28700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 0x7f2ee814f9a0 secure :-1 s=READY pgs=1309 cs=0 l=1 rx=0x7f2ed8010430 tx=0x7f2ed8010260).stop 2019-12-05T06:46:06.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.215+0000 7f2eeff28700 1 -- 172.21.15.16:0/64144748 shutdown_connections 2019-12-05T06:46:06.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.215+0000 7f2eeff28700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2ee8098b00 0x7f2ee81592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.215+0000 7f2eeff28700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2ec8023630 0x7f2ec8025ac0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.215+0000 7f2eeff28700 1 --2- 172.21.15.16:0/64144748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2ee814f5b0 0x7f2ee814f9a0 unknown :-1 s=CLOSED pgs=1309 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:46:06.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.215+0000 7f2eeff28700 1 -- 172.21.15.16:0/64144748 shutdown_connections 2019-12-05T06:46:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.215+0000 7f2eeff28700 1 -- 172.21.15.16:0/64144748 wait complete. 2019-12-05T06:46:06.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:46:06.215+0000 7f2eeff28700 1 -- 172.21.15.16:0/64144748 >> 172.21.15.16:0/64144748 conn(0x7f2ee80714f0 msgr2=0x7f2ee809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:46:06.225 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:46:06.225 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 118 secs before thrashing 2019-12-05T06:46:12.855 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:46:12.860 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:46:43.130 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:46:43.147 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:47:14.070 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:47:14.075 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:47:44.221 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:47:44.240 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:48:04.226 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:04.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.384+0000 7f8558418700 1 Processor -- start 2019-12-05T06:48:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.385+0000 7f8558418700 1 -- start start 2019-12-05T06:48:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.386+0000 7f8558418700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f855008c6d0 0x7f855008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:04.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.386+0000 7f8558418700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f855008f720 0x7f855008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.386+0000 7f8558418700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f855008fb80 con 0x7f855008fef0 2019-12-05T06:48:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.386+0000 7f8558418700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f855008df90 con 0x7f855008c6d0 2019-12-05T06:48:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.386+0000 7f8558418700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85500a3f70 con 0x7f855008f720 2019-12-05T06:48:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.386+0000 7f85569b5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f855008c6d0 0x7f855008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.386+0000 7f85561b4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f855008fef0 0x7f855008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46308/0 (socket says 172.21.15.16:46308) 2019-12-05T06:48:04.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.386+0000 7f85561b4700 1 -- 172.21.15.16:0/2314165227 learned_addr learned my addr 172.21.15.16:0/2314165227 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85571b6700 1 --2- 172.21.15.16:0/2314165227 >> v2:172.21.15.76:3300/0 conn(0x7f855008f720 0x7f855008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85559b3700 1 -- 172.21.15.16:0/2314165227 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281575749 0 0) 0x7f8540001150 con 0x7f855008fef0 2019-12-05T06:48:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85559b3700 1 -- 172.21.15.16:0/2314165227 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8538002e80 con 0x7f855008fef0 2019-12-05T06:48:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85569b5700 1 -- 172.21.15.16:0/2314165227 >> v2:172.21.15.76:3300/0 conn(0x7f855008f720 msgr2=0x7f855008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85569b5700 1 --2- 172.21.15.16:0/2314165227 >> v2:172.21.15.76:3300/0 conn(0x7f855008f720 0x7f855008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85569b5700 1 -- 172.21.15.16:0/2314165227 >> v1:172.21.15.16:6789/0 conn(0x7f855008fef0 legacy=0x7f855008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:04.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85569b5700 1 -- 172.21.15.16:0/2314165227 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f855008d540 con 0x7f855008c6d0 2019-12-05T06:48:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85571b6700 1 --2- 172.21.15.16:0/2314165227 >> v2:172.21.15.76:3300/0 conn(0x7f855008f720 0x7f855008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:48:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85569b5700 1 --2- 172.21.15.16:0/2314165227 >> v2:172.21.15.16:3300/0 conn(0x7f855008c6d0 0x7f855008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f854400af50 tx=0x7f854400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85569b5700 1 --2- 172.21.15.16:0/2314165227 >> v2:172.21.15.16:3300/0 conn(0x7f855008c6d0 0x7f855008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f854400af50 tx=0x7f854400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85569b5700 1 --2- 172.21.15.16:0/2314165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855008c6d0 0x7f855008cac0 secure :-1 s=READY pgs=1319 cs=0 l=1 rx=0x7f854400af50 tx=0x7f854400b760).ready entity=mon.0 client_cookie=317979d47675aaf9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85569b5700 1 --2- 172.21.15.16:0/2314165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855008c6d0 0x7f855008cac0 secure :-1 s=THROTTLE_DONE pgs=1319 cs=0 l=1 rx=0x7f854400af50 tx=0x7f854400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.387+0000 7f85559b3700 1 -- 172.21.15.16:0/2314165227 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8544007210 con 0x7f855008c6d0 2019-12-05T06:48:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.388+0000 7f85569b5700 1 --2- 172.21.15.16:0/2314165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855008c6d0 0x7f855008cac0 secure :-1 s=THROTTLE_DONE pgs=1319 cs=0 l=1 rx=0x7f854400af50 tx=0x7f854400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.388+0000 7f85559b3700 1 -- 172.21.15.16:0/2314165227 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8544007210 con 0x7f855008c6d0 2019-12-05T06:48:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.388+0000 7f85569b5700 1 --2- 172.21.15.16:0/2314165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855008c6d0 0x7f855008cac0 secure :-1 s=THROTTLE_DONE pgs=1319 cs=0 l=1 rx=0x7f854400af50 tx=0x7f854400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.388+0000 7f85559b3700 1 -- 172.21.15.16:0/2314165227 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8544009ba0 con 0x7f855008c6d0 2019-12-05T06:48:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.388+0000 7f8558418700 1 -- 172.21.15.16:0/2314165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855008c6d0 msgr2=0x7f855008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.388+0000 7f8558418700 1 --2- 172.21.15.16:0/2314165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855008c6d0 0x7f855008cac0 secure :-1 s=READY pgs=1319 cs=0 l=1 rx=0x7f854400af50 tx=0x7f854400b760).stop 2019-12-05T06:48:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.389+0000 7f8558418700 1 -- 172.21.15.16:0/2314165227 shutdown_connections 2019-12-05T06:48:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.389+0000 7f8558418700 1 --2- 172.21.15.16:0/2314165227 >> v2:172.21.15.76:3300/0 conn(0x7f855008f720 0x7f855008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:04.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.389+0000 7f8558418700 1 --2- 172.21.15.16:0/2314165227 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f855008c6d0 0x7f855008cac0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.389+0000 7f8558418700 1 -- 172.21.15.16:0/2314165227 shutdown_connections 2019-12-05T06:48:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.389+0000 7f8558418700 1 -- 172.21.15.16:0/2314165227 wait complete. 2019-12-05T06:48:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.389+0000 7f8558418700 1 -- 172.21.15.16:0/2314165227 >> 172.21.15.16:0/2314165227 conn(0x7f8550061940 msgr2=0x7f8550096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.393+0000 7f8558418700 1 Processor -- start 2019-12-05T06:48:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.393+0000 7f8558418700 1 -- start start 2019-12-05T06:48:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.394+0000 7f8558418700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f855014ca10 0x7f85500869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:04.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.394+0000 7f8558418700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f85500879b0 0x7f8550086f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.394+0000 7f8558418700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f855009e030 con 0x7f8550087460 2019-12-05T06:48:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.394+0000 7f8558418700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8550085a50 con 0x7f855014ca10 2019-12-05T06:48:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.394+0000 7f8558418700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8550085020 con 0x7f85500879b0 2019-12-05T06:48:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.395+0000 7f85569b5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f855014ca10 0x7f85500869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.395+0000 7f85561b4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f85500879b0 0x7f8550086f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:04.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.395+0000 7f85571b6700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f8550087460 0x7f8550152160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:39386/0 (socket says 172.21.15.16:39386) 2019-12-05T06:48:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.395+0000 7f85571b6700 1 -- 172.21.15.16:0/3587671986 learned_addr learned my addr 172.21.15.16:0/3587671986 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85569b5700 1 -- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f85500879b0 msgr2=0x7f8550086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f85500879b0 0x7f8550086f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:04.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85569b5700 1 -- 172.21.15.16:0/3587671986 >> v1:172.21.15.76:6789/0 conn(0x7f8550087460 legacy=0x7f8550152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85569b5700 1 -- 172.21.15.16:0/3587671986 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85500860e0 con 0x7f855014ca10 2019-12-05T06:48:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f854f7fe700 1 -- 172.21.15.16:0/3587671986 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2976473404 0 0) 0x7f8548009e90 con 0x7f8550087460 2019-12-05T06:48:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85561b4700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f85500879b0 0x7f8550086f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> v2:172.21.15.76:3300/0 conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> v2:172.21.15.76:3300/0 conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=READY pgs=898 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f854f7fe700 1 -- 172.21.15.16:0/3587671986 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8544000940 con 0x7f855014ca10 2019-12-05T06:48:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f854f7fe700 1 -- 172.21.15.16:0/3587671986 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8544000940 con 0x7f855014ca10 2019-12-05T06:48:04.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f8558418700 1 -- 172.21.15.16:0/3587671986 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85500860e0 con 0x7f855014ca10 2019-12-05T06:48:04.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.396+0000 7f854f7fe700 1 -- 172.21.15.16:0/3587671986 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8544009960 con 0x7f855014ca10 2019-12-05T06:48:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.397+0000 7f8558418700 1 -- 172.21.15.16:0/3587671986 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85500860e0 con 0x7f855014ca10 2019-12-05T06:48:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.397+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.398+0000 7f854f7fe700 1 -- 172.21.15.16:0/3587671986 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8544009580 con 0x7f855014ca10 2019-12-05T06:48:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.398+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.398+0000 7f854f7fe700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8530023630 0x7f8530025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:04.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.398+0000 7f854f7fe700 1 -- 172.21.15.16:0/3587671986 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f8544045d50 con 0x7f855014ca10 2019-12-05T06:48:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.399+0000 7f85561b4700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8530023630 0x7f8530025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.399+0000 7f85561b4700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8530023630 0x7f8530025ac0 crc :-1 s=READY pgs=557 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:04.433 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.400+0000 7f8558418700 1 -- 172.21.15.16:0/3587671986 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85500860e0 con 0x7f855014ca10 2019-12-05T06:48:04.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.410+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.434 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.410+0000 7f854f7fe700 1 -- 172.21.15.16:0/3587671986 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f854400af90 con 0x7f855014ca10 2019-12-05T06:48:04.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.646+0000 7f8558418700 1 -- 172.21.15.16:0/3587671986 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85500860e0 con 0x7f855014ca10 2019-12-05T06:48:04.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.647+0000 7f85569b5700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=THROTTLE_DONE pgs=898 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:04.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.647+0000 7f854f7fe700 1 -- 172.21.15.16:0/3587671986 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f8544009580 con 0x7f855014ca10 2019-12-05T06:48:04.649 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 162 2019-12-05T06:48:04.649 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:04.650 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 -- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8530023630 msgr2=0x7f8530025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8530023630 0x7f8530025ac0 crc :-1 s=READY pgs=557 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 -- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 msgr2=0x7f85500869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:04.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 0x7f85500869e0 secure :-1 s=READY pgs=898 cs=0 l=1 rx=0x7f8544010380 tx=0x7f8544010720).stop 2019-12-05T06:48:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 -- 172.21.15.16:0/3587671986 shutdown_connections 2019-12-05T06:48:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8530023630 0x7f8530025ac0 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f85500879b0 0x7f8550086f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 --2- 172.21.15.16:0/3587671986 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f855014ca10 0x7f85500869e0 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 -- 172.21.15.16:0/3587671986 shutdown_connections 2019-12-05T06:48:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 -- 172.21.15.16:0/3587671986 wait complete. 2019-12-05T06:48:04.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.649+0000 7f8558418700 1 -- 172.21.15.16:0/3587671986 >> 172.21.15.16:0/3587671986 conn(0x7f8550098480 msgr2=0x7f8550071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:04.663 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:48:04.664 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:48:04.664 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:48:04.673 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:48:04.674 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:04.978 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2019-12-05T06:48:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.992+0000 7f8514402700 1 Processor -- start 2019-12-05T06:48:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.992+0000 7f8514402700 1 -- start start 2019-12-05T06:48:05.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.994+0000 7f8514402700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f850c08c6d0 0x7f850c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.994+0000 7f8514402700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.994+0000 7f8514402700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c08fb70 con 0x7f850c08fef0 2019-12-05T06:48:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.994+0000 7f8514402700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c08df90 con 0x7f850c08c6d0 2019-12-05T06:48:05.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.994+0000 7f8514402700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0a3f70 con 0x7f850c08f720 2019-12-05T06:48:05.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.995+0000 7f85131a0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:05.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.995+0000 7f851299f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f850c08c6d0 0x7f850c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:05.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.995+0000 7f85131a0700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:36918/0 (socket says 172.21.15.16:36918) 2019-12-05T06:48:05.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.995+0000 7f851299f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f850c08c6d0 0x7f850c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:47248/0 (socket says 172.21.15.16:47248) 2019-12-05T06:48:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.995+0000 7f85131a0700 1 -- 172.21.15.16:0/445390992 learned_addr learned my addr 172.21.15.16:0/445390992 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.995+0000 7f85131a0700 1 -- 172.21.15.16:0/445390992 >> v2:172.21.15.76:3300/0 conn(0x7f850c08c6d0 msgr2=0x7f850c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:05.032 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.995+0000 7f85131a0700 1 --2- 172.21.15.16:0/445390992 >> v2:172.21.15.76:3300/0 conn(0x7f850c08c6d0 0x7f850c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:05.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.995+0000 7f85131a0700 1 -- 172.21.15.16:0/445390992 >> v1:172.21.15.76:6789/0 conn(0x7f850c08fef0 legacy=0x7f850c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.995+0000 7f85131a0700 1 -- 172.21.15.16:0/445390992 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c08d540 con 0x7f850c08f720 2019-12-05T06:48:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.995+0000 7f85131a0700 1 --2- 172.21.15.16:0/445390992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f850400f760 tx=0x7f850400f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.996+0000 7f85131a0700 1 --2- 172.21.15.16:0/445390992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f850400f760 tx=0x7f850400f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.996+0000 7f85131a0700 1 --2- 172.21.15.16:0/445390992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 secure :-1 s=READY pgs=312 cs=0 l=1 rx=0x7f850400f760 tx=0x7f850400f590).ready entity=mon.2 client_cookie=d299a5ab80340e96 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.996+0000 7f85131a0700 1 --2- 172.21.15.16:0/445390992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 secure :-1 s=THROTTLE_DONE pgs=312 cs=0 l=1 rx=0x7f850400f760 tx=0x7f850400f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.996+0000 7f85131a0700 1 --2- 172.21.15.16:0/445390992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 secure :-1 s=THROTTLE_DONE pgs=312 cs=0 l=1 rx=0x7f850400f760 tx=0x7f850400f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.996+0000 7f85131a0700 1 --2- 172.21.15.16:0/445390992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 secure :-1 s=THROTTLE_DONE pgs=312 cs=0 l=1 rx=0x7f850400f760 tx=0x7f850400f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.996+0000 7f851199d700 1 -- 172.21.15.16:0/445390992 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8504000a70 con 0x7f850c08f720 2019-12-05T06:48:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.996+0000 7f851199d700 1 -- 172.21.15.16:0/445390992 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8504000e70 con 0x7f850c08f720 2019-12-05T06:48:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.996+0000 7f851199d700 1 -- 172.21.15.16:0/445390992 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8504001500 con 0x7f850c08f720 2019-12-05T06:48:05.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.996+0000 7f8514402700 1 -- 172.21.15.16:0/445390992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 msgr2=0x7f850c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:05.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.996+0000 7f8514402700 1 --2- 172.21.15.16:0/445390992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 secure :-1 s=READY pgs=312 cs=0 l=1 rx=0x7f850400f760 tx=0x7f850400f590).stop 2019-12-05T06:48:05.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.998+0000 7f8514402700 1 -- 172.21.15.16:0/445390992 shutdown_connections 2019-12-05T06:48:05.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.998+0000 7f8514402700 1 --2- 172.21.15.16:0/445390992 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c08f720 0x7f850c08d000 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:05.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.998+0000 7f8514402700 1 --2- 172.21.15.16:0/445390992 >> v2:172.21.15.76:3300/0 conn(0x7f850c08c6d0 0x7f850c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:05.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.998+0000 7f8514402700 1 -- 172.21.15.16:0/445390992 shutdown_connections 2019-12-05T06:48:05.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.998+0000 7f8514402700 1 -- 172.21.15.16:0/445390992 wait complete. 2019-12-05T06:48:05.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:04.998+0000 7f8514402700 1 -- 172.21.15.16:0/445390992 >> 172.21.15.16:0/445390992 conn(0x7f850c061940 msgr2=0x7f850c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:05.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.003+0000 7f8514402700 1 Processor -- start 2019-12-05T06:48:05.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.003+0000 7f8514402700 1 -- start start 2019-12-05T06:48:05.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.004+0000 7f8514402700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f850c14f390 0x7f850c14f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:05.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.005+0000 7f8514402700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c086140 0x7f850c14fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:05.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.005+0000 7f8514402700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f850c09fcd0 con 0x7f850c086910 2019-12-05T06:48:05.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.005+0000 7f8514402700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f850c09f630 con 0x7f850c14f390 2019-12-05T06:48:05.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.005+0000 7f8514402700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f850c0849b0 con 0x7f850c086140 2019-12-05T06:48:05.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.005+0000 7f851219e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f850c086910 0x7f850c085940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46334/0 (socket says 172.21.15.16:46334) 2019-12-05T06:48:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.005+0000 7f851219e700 1 -- 172.21.15.16:0/1029275032 learned_addr learned my addr 172.21.15.16:0/1029275032 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.005+0000 7f85131a0700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c086140 0x7f850c14fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.005+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> v2:172.21.15.76:3300/0 conn(0x7f850c14f390 0x7f850c14f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.005+0000 7f85037fe700 1 -- 172.21.15.16:0/1029275032 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2329808310 0 0) 0x7f84fc001170 con 0x7f850c086910 2019-12-05T06:48:05.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.005+0000 7f85037fe700 1 -- 172.21.15.16:0/1029275032 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f84f4002e80 con 0x7f850c086910 2019-12-05T06:48:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f851299f700 1 -- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c086140 msgr2=0x7f850c14fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c086140 0x7f850c14fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f851299f700 1 -- 172.21.15.16:0/1029275032 >> v1:172.21.15.16:6789/0 conn(0x7f850c086910 legacy=0x7f850c085940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:05.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f851299f700 1 -- 172.21.15.16:0/1029275032 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f850c085040 con 0x7f850c14f390 2019-12-05T06:48:05.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f85131a0700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c086140 0x7f850c14fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:05.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> v2:172.21.15.76:3300/0 conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> v2:172.21.15.76:3300/0 conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=READY pgs=900 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:05.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f85037fe700 1 -- 172.21.15.16:0/1029275032 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8508000cc0 con 0x7f850c14f390 2019-12-05T06:48:05.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f8514402700 1 -- 172.21.15.16:0/1029275032 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f850c085040 con 0x7f850c14f390 2019-12-05T06:48:05.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f85037fe700 1 -- 172.21.15.16:0/1029275032 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8508000cc0 con 0x7f850c14f390 2019-12-05T06:48:05.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f85037fe700 1 -- 172.21.15.16:0/1029275032 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f850800a140 con 0x7f850c14f390 2019-12-05T06:48:05.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.006+0000 7f8514402700 1 -- 172.21.15.16:0/1029275032 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f850c085040 con 0x7f850c14f390 2019-12-05T06:48:05.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.007+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.008+0000 7f85037fe700 1 -- 172.21.15.16:0/1029275032 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f850800a880 con 0x7f850c14f390 2019-12-05T06:48:05.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.008+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.008+0000 7f85037fe700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84f4023670 0x7f84f4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:05.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.008+0000 7f85037fe700 1 -- 172.21.15.16:0/1029275032 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f85080460c0 con 0x7f850c14f390 2019-12-05T06:48:05.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.009+0000 7f851219e700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84f4023670 0x7f84f4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:05.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.009+0000 7f851219e700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84f4023670 0x7f84f4025ae0 crc :-1 s=READY pgs=558 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:05.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.024+0000 7f8514402700 1 -- 172.21.15.16:0/1029275032 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f850c085040 con 0x7f850c14f390 2019-12-05T06:48:05.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.033+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.033+0000 7f85037fe700 1 -- 172.21.15.16:0/1029275032 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8508032f70 con 0x7f850c14f390 2019-12-05T06:48:05.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.301+0000 7f8514402700 1 -- 172.21.15.16:0/1029275032 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f850c085040 con 0x7f850c14f390 2019-12-05T06:48:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.302+0000 7f851299f700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=THROTTLE_DONE pgs=900 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:05.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.302+0000 7f85037fe700 1 -- 172.21.15.16:0/1029275032 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f8508032f70 con 0x7f850c14f390 2019-12-05T06:48:05.305 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 162 2019-12-05T06:48:05.305 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:05.305 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:05.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.308+0000 7f8514402700 1 -- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84f4023670 msgr2=0x7f84f4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.308+0000 7f8514402700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84f4023670 0x7f84f4025ae0 crc :-1 s=READY pgs=558 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.308+0000 7f8514402700 1 -- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 msgr2=0x7f850c14f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.308+0000 7f8514402700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 0x7f850c14f780 secure :-1 s=READY pgs=900 cs=0 l=1 rx=0x7f8508010710 tx=0x7f8508010ab0).stop 2019-12-05T06:48:05.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.308+0000 7f8514402700 1 -- 172.21.15.16:0/1029275032 shutdown_connections 2019-12-05T06:48:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.308+0000 7f8514402700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f84f4023670 0x7f84f4025ae0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.308+0000 7f8514402700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f850c086140 0x7f850c14fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.309+0000 7f8514402700 1 --2- 172.21.15.16:0/1029275032 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f850c14f390 0x7f850c14f780 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:05.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.309+0000 7f8514402700 1 -- 172.21.15.16:0/1029275032 shutdown_connections 2019-12-05T06:48:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.309+0000 7f8514402700 1 -- 172.21.15.16:0/1029275032 wait complete. 2019-12-05T06:48:05.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:05.309+0000 7f8514402700 1 -- 172.21.15.16:0/1029275032 >> 172.21.15.16:0/1029275032 conn(0x7f850c12f780 msgr2=0x7f850c096b90 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:05.332 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:07.333 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.595+0000 7f2c9d276700 1 Processor -- start 2019-12-05T06:48:07.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.595+0000 7f2c9d276700 1 -- start start 2019-12-05T06:48:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.596+0000 7f2c9d276700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c9808c6d0 0x7f2c9808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.596+0000 7f2c9d276700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9808f720 0x7f2c9808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.596+0000 7f2c9d276700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c9808fb70 con 0x7f2c9808fef0 2019-12-05T06:48:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.596+0000 7f2c9d276700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c9808df90 con 0x7f2c9808c6d0 2019-12-05T06:48:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.596+0000 7f2c9d276700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c980a3f70 con 0x7f2c9808f720 2019-12-05T06:48:07.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.596+0000 7f2c977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c9808c6d0 0x7f2c9808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.596+0000 7f2c96ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2c9808fef0 0x7f2c9808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46346/0 (socket says 172.21.15.16:46346) 2019-12-05T06:48:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.596+0000 7f2c977fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c9808c6d0 0x7f2c9808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32944/0 (socket says 172.21.15.16:32944) 2019-12-05T06:48:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.596+0000 7f2c96ffd700 1 -- 172.21.15.16:0/2190133 learned_addr learned my addr 172.21.15.16:0/2190133 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.596+0000 7f2c97fff700 1 --2- 172.21.15.16:0/2190133 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9808f720 0x7f2c9808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c977fe700 1 -- 172.21.15.16:0/2190133 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9808f720 msgr2=0x7f2c9808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:07.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c977fe700 1 --2- 172.21.15.16:0/2190133 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9808f720 0x7f2c9808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c977fe700 1 -- 172.21.15.16:0/2190133 >> v1:172.21.15.16:6789/0 conn(0x7f2c9808fef0 legacy=0x7f2c9808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c977fe700 1 -- 172.21.15.16:0/2190133 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c9808d540 con 0x7f2c9808c6d0 2019-12-05T06:48:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c967fc700 1 -- 172.21.15.16:0/2190133 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3790670810 0 0) 0x7f2c80001070 con 0x7f2c9808fef0 2019-12-05T06:48:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c97fff700 1 --2- 172.21.15.16:0/2190133 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9808f720 0x7f2c9808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:48:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c977fe700 1 --2- 172.21.15.16:0/2190133 >> v2:172.21.15.16:3300/0 conn(0x7f2c9808c6d0 0x7f2c9808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2c8800af50 tx=0x7f2c8800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c977fe700 1 --2- 172.21.15.16:0/2190133 >> v2:172.21.15.16:3300/0 conn(0x7f2c9808c6d0 0x7f2c9808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2c8800af50 tx=0x7f2c8800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c977fe700 1 --2- 172.21.15.16:0/2190133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9808c6d0 0x7f2c9808cac0 secure :-1 s=READY pgs=1322 cs=0 l=1 rx=0x7f2c8800af50 tx=0x7f2c8800b760).ready entity=mon.0 client_cookie=31e3ea890ce69d13 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:07.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c977fe700 1 --2- 172.21.15.16:0/2190133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9808c6d0 0x7f2c9808cac0 secure :-1 s=THROTTLE_DONE pgs=1322 cs=0 l=1 rx=0x7f2c8800af50 tx=0x7f2c8800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c967fc700 1 -- 172.21.15.16:0/2190133 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c88007210 con 0x7f2c9808c6d0 2019-12-05T06:48:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c977fe700 1 --2- 172.21.15.16:0/2190133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9808c6d0 0x7f2c9808cac0 secure :-1 s=THROTTLE_DONE pgs=1322 cs=0 l=1 rx=0x7f2c8800af50 tx=0x7f2c8800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c977fe700 1 --2- 172.21.15.16:0/2190133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9808c6d0 0x7f2c9808cac0 secure :-1 s=THROTTLE_DONE pgs=1322 cs=0 l=1 rx=0x7f2c8800af50 tx=0x7f2c8800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c967fc700 1 -- 172.21.15.16:0/2190133 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c880099b0 con 0x7f2c9808c6d0 2019-12-05T06:48:07.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.597+0000 7f2c967fc700 1 -- 172.21.15.16:0/2190133 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c8800a080 con 0x7f2c9808c6d0 2019-12-05T06:48:07.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.598+0000 7f2c9d276700 1 -- 172.21.15.16:0/2190133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9808c6d0 msgr2=0x7f2c9808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.598+0000 7f2c9d276700 1 --2- 172.21.15.16:0/2190133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9808c6d0 0x7f2c9808cac0 secure :-1 s=READY pgs=1322 cs=0 l=1 rx=0x7f2c8800af50 tx=0x7f2c8800b760).stop 2019-12-05T06:48:07.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.599+0000 7f2c9d276700 1 -- 172.21.15.16:0/2190133 shutdown_connections 2019-12-05T06:48:07.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.599+0000 7f2c9d276700 1 --2- 172.21.15.16:0/2190133 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2c9808f720 0x7f2c9808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:07.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.599+0000 7f2c9d276700 1 --2- 172.21.15.16:0/2190133 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c9808c6d0 0x7f2c9808cac0 unknown :-1 s=CLOSED pgs=1322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.599+0000 7f2c9d276700 1 -- 172.21.15.16:0/2190133 shutdown_connections 2019-12-05T06:48:07.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.599+0000 7f2c9d276700 1 -- 172.21.15.16:0/2190133 wait complete. 2019-12-05T06:48:07.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.599+0000 7f2c9d276700 1 -- 172.21.15.16:0/2190133 >> 172.21.15.16:0/2190133 conn(0x7f2c98061940 msgr2=0x7f2c98096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:07.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.602+0000 7f2c9d276700 1 Processor -- start 2019-12-05T06:48:07.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.603+0000 7f2c9d276700 1 -- start start 2019-12-05T06:48:07.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.603+0000 7f2c9d276700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f2c98088da0 0x7f2c9814d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:07.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c9d276700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c980885d0 0x7f2c9819b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:07.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c9d276700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c98153190 con 0x7f2c9814d4f0 2019-12-05T06:48:07.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c9d276700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c98086e40 con 0x7f2c980885d0 2019-12-05T06:48:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c9d276700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c98086410 con 0x7f2c98088da0 2019-12-05T06:48:07.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c97fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c980885d0 0x7f2c9819b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c97fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2c980885d0 0x7f2c9819b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32950/0 (socket says 172.21.15.16:32950) 2019-12-05T06:48:07.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c97fff700 1 -- 172.21.15.16:0/3382733408 learned_addr learned my addr 172.21.15.16:0/3382733408 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:07.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c96ffd700 1 --2- 172.21.15.16:0/3382733408 >> v2:172.21.15.76:3300/0 conn(0x7f2c98088da0 0x7f2c9814d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c97fff700 1 -- 172.21.15.16:0/3382733408 >> v2:172.21.15.76:3300/0 conn(0x7f2c98088da0 msgr2=0x7f2c9814d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> v2:172.21.15.76:3300/0 conn(0x7f2c98088da0 0x7f2c9814d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c97fff700 1 -- 172.21.15.16:0/3382733408 >> v1:172.21.15.76:6789/0 conn(0x7f2c9814d4f0 legacy=0x7f2c98087dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c97fff700 1 -- 172.21.15.16:0/3382733408 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c980874d0 con 0x7f2c980885d0 2019-12-05T06:48:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c96ffd700 1 --2- 172.21.15.16:0/3382733408 >> v2:172.21.15.76:3300/0 conn(0x7f2c98088da0 0x7f2c9814d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:48:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.604+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> v2:172.21.15.16:3300/0 conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.605+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> v2:172.21.15.16:3300/0 conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.605+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=READY pgs=1323 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.605+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.605+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.605+0000 7f2c8ffff700 1 -- 172.21.15.16:0/3382733408 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c90008800 con 0x7f2c980885d0 2019-12-05T06:48:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.605+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.605+0000 7f2c8ffff700 1 -- 172.21.15.16:0/3382733408 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c90009e00 con 0x7f2c980885d0 2019-12-05T06:48:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.605+0000 7f2c8ffff700 1 -- 172.21.15.16:0/3382733408 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c9000a490 con 0x7f2c980885d0 2019-12-05T06:48:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.605+0000 7f2c9d276700 1 -- 172.21.15.16:0/3382733408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c980874d0 con 0x7f2c980885d0 2019-12-05T06:48:07.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.605+0000 7f2c9d276700 1 -- 172.21.15.16:0/3382733408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c98153190 con 0x7f2c980885d0 2019-12-05T06:48:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.606+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.607+0000 7f2c8ffff700 1 -- 172.21.15.16:0/3382733408 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2c90000df0 con 0x7f2c980885d0 2019-12-05T06:48:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.607+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.607+0000 7f2c8ffff700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023630 0x7f2c7c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:07.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.607+0000 7f2c8ffff700 1 -- 172.21.15.16:0/3382733408 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f2c900461e0 con 0x7f2c980885d0 2019-12-05T06:48:07.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.607+0000 7f2c977fe700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023630 0x7f2c7c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:07.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.608+0000 7f2c977fe700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023630 0x7f2c7c025ac0 crc :-1 s=READY pgs=559 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:07.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.611+0000 7f2c9d276700 1 -- 172.21.15.16:0/3382733408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c980874d0 con 0x7f2c980885d0 2019-12-05T06:48:07.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.617+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.617+0000 7f2c8ffff700 1 -- 172.21.15.16:0/3382733408 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2c90009960 con 0x7f2c980885d0 2019-12-05T06:48:07.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.860+0000 7f2c9d276700 1 -- 172.21.15.16:0/3382733408 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2c980874d0 con 0x7f2c980885d0 2019-12-05T06:48:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.861+0000 7f2c97fff700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=THROTTLE_DONE pgs=1323 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:07.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.861+0000 7f2c8ffff700 1 -- 172.21.15.16:0/3382733408 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f2c900016a0 con 0x7f2c980885d0 2019-12-05T06:48:07.864 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:07.865 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:07.865 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 162 2019-12-05T06:48:07.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.865+0000 7f2c9d276700 1 -- 172.21.15.16:0/3382733408 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023630 msgr2=0x7f2c7c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.865+0000 7f2c9d276700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023630 0x7f2c7c025ac0 crc :-1 s=READY pgs=559 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.865+0000 7f2c9d276700 1 -- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 msgr2=0x7f2c9819b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.865+0000 7f2c9d276700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 0x7f2c9819b400 secure :-1 s=READY pgs=1323 cs=0 l=1 rx=0x7f2c90010800 tx=0x7f2c90010ba0).stop 2019-12-05T06:48:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.866+0000 7f2c9d276700 1 -- 172.21.15.16:0/3382733408 shutdown_connections 2019-12-05T06:48:07.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.866+0000 7f2c9d276700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2c7c023630 0x7f2c7c025ac0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.866+0000 7f2c9d276700 1 --2- 172.21.15.16:0/3382733408 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2c980885d0 0x7f2c9819b400 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.866+0000 7f2c9d276700 1 --2- 172.21.15.16:0/3382733408 >> v2:172.21.15.76:3300/0 conn(0x7f2c98088da0 0x7f2c9814d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.866+0000 7f2c9d276700 1 -- 172.21.15.16:0/3382733408 shutdown_connections 2019-12-05T06:48:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.866+0000 7f2c9d276700 1 -- 172.21.15.16:0/3382733408 wait complete. 2019-12-05T06:48:07.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:07.866+0000 7f2c9d276700 1 -- 172.21.15.16:0/3382733408 >> 172.21.15.16:0/3382733408 conn(0x7f2c98099490 msgr2=0x7f2c98132ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:07.878 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:09.879 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:10.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.172+0000 7fbbf7f6a700 1 Processor -- start 2019-12-05T06:48:10.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.172+0000 7fbbf7f6a700 1 -- start start 2019-12-05T06:48:10.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.175+0000 7fbbf7f6a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf008fef0 0x7fbbf008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.175+0000 7fbbf7f6a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf00958e0 con 0x7fbbf008c6d0 2019-12-05T06:48:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.175+0000 7fbbf7f6a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf00629e0 con 0x7fbbf008d2c0 2019-12-05T06:48:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.175+0000 7fbbf7f6a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf008df90 con 0x7fbbf008fef0 2019-12-05T06:48:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.175+0000 7fbbf6507700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbbf008c6d0 0x7fbbf008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46360/0 (socket says 172.21.15.16:46360) 2019-12-05T06:48:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.175+0000 7fbbf6507700 1 -- 172.21.15.16:0/1158122007 learned_addr learned my addr 172.21.15.16:0/1158122007 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.175+0000 7fbbf6d08700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf008fef0 0x7fbbf008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:10.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132109333 0 0) 0x7fbbe80014d0 con 0x7fbbf008c6d0 2019-12-05T06:48:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf6d08700 1 -- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf008fef0 msgr2=0x7fbbf008f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 16 2019-12-05T06:48:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf6d08700 1 -- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf008fef0 msgr2=0x7fbbf008f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:48:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf6d08700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf008fef0 0x7fbbf008f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:48:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd8002e80 con 0x7fbbf008c6d0 2019-12-05T06:48:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1458018073 0 0) 0x7fbbe4001080 con 0x7fbbf008d2c0 2019-12-05T06:48:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf6d08700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf008fef0 0x7fbbf008f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:48:10.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbbd8002ec0 con 0x7fbbf008d2c0 2019-12-05T06:48:10.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5d06700 1 -- 172.21.15.16:0/1158122007 >> v1:172.21.15.76:6789/0 conn(0x7fbbf008d2c0 legacy=0x7fbbf008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2019-12-05T06:48:10.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3344639727 0 0) 0x7fbbe80014b0 con 0x7fbbf008c6d0 2019-12-05T06:48:10.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5d06700 1 -- 172.21.15.16:0/1158122007 >> v1:172.21.15.76:6789/0 conn(0x7fbbf008d2c0 legacy=0x7fbbf008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:48:10.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5d06700 1 --1- 172.21.15.16:0/1158122007 >> v1:172.21.15.76:6789/0 conn(0x7fbbf008d2c0 0x7fbbf008ef20 :-1 s=OPENED pgs=902 cs=1 l=1).handle_message read tag failed 2019-12-05T06:48:10.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5d06700 1 --1- 172.21.15.16:0/1158122007 >> v1:172.21.15.76:6789/0 conn(0x7fbbf008d2c0 0x7fbbf008ef20 :-1 s=OPENED pgs=902 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:48:10.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf008fef0 msgr2=0x7fbbf008f720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:48:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.176+0000 7fbbf5505700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf008fef0 0x7fbbf008f720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 >> v1:172.21.15.76:6789/0 conn(0x7fbbf008d2c0 legacy=0x7fbbf008ef20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:48:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf008e620 con 0x7fbbf008c6d0 2019-12-05T06:48:10.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fbbe80019e0 con 0x7fbbf008c6d0 2019-12-05T06:48:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 >> v1:172.21.15.16:6789/0 conn(0x7fbbf008c6d0 legacy=0x7fbbf008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5505700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd80479b0 0x7fbbd8049e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:10.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5505700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbd804a5c0 0x7fbbd804ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5505700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbd804d2d0 0x7fbbd804f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:10.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd804ff20 con 0x7fbbd80479b0 2019-12-05T06:48:10.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd8050110 con 0x7fbbd804a5c0 2019-12-05T06:48:10.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbd8050340 con 0x7fbbd804d2d0 2019-12-05T06:48:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf6507700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd80479b0 0x7fbbd8049e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf6d08700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbd804d2d0 0x7fbbd804f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.177+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbd804a5c0 0x7fbbd804ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:10.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf6507700 1 -- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbd804d2d0 msgr2=0x7fbbd804f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf6507700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbd804d2d0 0x7fbbd804f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf6507700 1 -- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbd804a5c0 msgr2=0x7fbbd804ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf6507700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbbd804a5c0 0x7fbbd804ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:10.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf6507700 1 -- 172.21.15.16:0/1158122007 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbd80505f0 con 0x7fbbd80479b0 2019-12-05T06:48:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf6507700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd80479b0 0x7fbbd8049e10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbbe800fc60 tx=0x7fbbe800fb90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf6507700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd80479b0 0x7fbbd8049e10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbbe800fc60 tx=0x7fbbe800fb90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf6507700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd80479b0 0x7fbbd8049e10 secure :-1 s=READY pgs=1325 cs=0 l=1 rx=0x7fbbe800fc60 tx=0x7fbbe800fb90).ready entity=mon.0 client_cookie=a5948564dd33b395 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:10.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf6507700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd80479b0 0x7fbbd8049e10 secure :-1 s=THROTTLE_DONE pgs=1325 cs=0 l=1 rx=0x7fbbe800fc60 tx=0x7fbbe800fb90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbe80047b0 con 0x7fbbd80479b0 2019-12-05T06:48:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.178+0000 7fbbf6507700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd80479b0 0x7fbbd8049e10 secure :-1 s=THROTTLE_DONE pgs=1325 cs=0 l=1 rx=0x7fbbe800fc60 tx=0x7fbbe800fb90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.179+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe80014b0 con 0x7fbbd80479b0 2019-12-05T06:48:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.179+0000 7fbbf6507700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd80479b0 0x7fbbd8049e10 secure :-1 s=THROTTLE_DONE pgs=1325 cs=0 l=1 rx=0x7fbbe800fc60 tx=0x7fbbe800fb90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.179+0000 7fbbf5505700 1 -- 172.21.15.16:0/1158122007 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbe80049a0 con 0x7fbbd80479b0 2019-12-05T06:48:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.179+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/1158122007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd80479b0 msgr2=0x7fbbd8049e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.179+0000 7fbbf7f6a700 1 --2- 172.21.15.16:0/1158122007 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbd80479b0 0x7fbbd8049e10 secure :-1 s=READY pgs=1325 cs=0 l=1 rx=0x7fbbe800fc60 tx=0x7fbbe800fb90).stop 2019-12-05T06:48:10.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.179+0000 7fbbf6d08700 1 -- 172.21.15.16:0/1158122007 reap_dead start 2019-12-05T06:48:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.181+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/1158122007 shutdown_connections 2019-12-05T06:48:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.181+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/1158122007 shutdown_connections 2019-12-05T06:48:10.211 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.181+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/1158122007 wait complete. 2019-12-05T06:48:10.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.181+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/1158122007 >> 172.21.15.16:0/1158122007 conn(0x7fbbf0061940 msgr2=0x7fbbf0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:10.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.186+0000 7fbbf7f6a700 1 Processor -- start 2019-12-05T06:48:10.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.187+0000 7fbbf7f6a700 1 -- start start 2019-12-05T06:48:10.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.188+0000 7fbbf7f6a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf01302f0 0x7fbbf012a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.188+0000 7fbbf7f6a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbbf012b640 0x7fbbf012abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.188+0000 7fbbf7f6a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbbf0131bf0 con 0x7fbbf012b0f0 2019-12-05T06:48:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.188+0000 7fbbf7f6a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbbf0128cb0 con 0x7fbbf012b640 2019-12-05T06:48:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.188+0000 7fbbf7f6a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbbf0128f80 con 0x7fbbf01302f0 2019-12-05T06:48:10.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.188+0000 7fbbf5d06700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbbf012b640 0x7fbbf012abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.189+0000 7fbbf5d06700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbbf012b640 0x7fbbf012abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32974/0 (socket says 172.21.15.16:32974) 2019-12-05T06:48:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.189+0000 7fbbf5d06700 1 -- 172.21.15.16:0/3617093263 learned_addr learned my addr 172.21.15.16:0/3617093263 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.189+0000 7fbbf6507700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf01302f0 0x7fbbf012a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.189+0000 7fbbf5d06700 1 -- 172.21.15.16:0/3617093263 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf01302f0 msgr2=0x7fbbf012a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.189+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf01302f0 0x7fbbf012a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:10.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.189+0000 7fbbeeffd700 1 -- 172.21.15.16:0/3617093263 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005540634 0 0) 0x7fbbe000e100 con 0x7fbbf012b0f0 2019-12-05T06:48:10.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.189+0000 7fbbf5d06700 1 -- 172.21.15.16:0/3617093263 >> v1:172.21.15.76:6789/0 conn(0x7fbbf012b0f0 legacy=0x7fbbf01306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.189+0000 7fbbf5d06700 1 -- 172.21.15.16:0/3617093263 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf0129d70 con 0x7fbbf012b640 2019-12-05T06:48:10.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.189+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> v2:172.21.15.16:3300/0 conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.190+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> v2:172.21.15.16:3300/0 conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.190+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=READY pgs=1326 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:10.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.190+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=THROTTLE_DONE pgs=1326 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.190+0000 7fbbeeffd700 1 -- 172.21.15.16:0/3617093263 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbe400eb40 con 0x7fbbf012b640 2019-12-05T06:48:10.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.190+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=THROTTLE_DONE pgs=1326 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.190+0000 7fbbeeffd700 1 -- 172.21.15.16:0/3617093263 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbbe4000c10 con 0x7fbbf012b640 2019-12-05T06:48:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.190+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/3617093263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbf0129d70 con 0x7fbbf012b640 2019-12-05T06:48:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.190+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=THROTTLE_DONE pgs=1326 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.190+0000 7fbbeeffd700 1 -- 172.21.15.16:0/3617093263 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbbe4000e00 con 0x7fbbf012b640 2019-12-05T06:48:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.190+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/3617093263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbf0129d70 con 0x7fbbf012b640 2019-12-05T06:48:10.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.192+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=THROTTLE_DONE pgs=1326 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.193+0000 7fbbeeffd700 1 -- 172.21.15.16:0/3617093263 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbbe4000c10 con 0x7fbbf012b640 2019-12-05T06:48:10.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.193+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=THROTTLE_DONE pgs=1326 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.193+0000 7fbbeeffd700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbbd0023630 0x7fbbd0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:10.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.193+0000 7fbbeeffd700 1 -- 172.21.15.16:0/3617093263 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fbbe4045e20 con 0x7fbbf012b640 2019-12-05T06:48:10.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.193+0000 7fbbf6507700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbbd0023630 0x7fbbd0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:10.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.194+0000 7fbbf6507700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbbd0023630 0x7fbbd0025ac0 crc :-1 s=READY pgs=560 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:10.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.204+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/3617093263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbf0129d70 con 0x7fbbf012b640 2019-12-05T06:48:10.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.213+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=THROTTLE_DONE pgs=1326 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.214+0000 7fbbeeffd700 1 -- 172.21.15.16:0/3617093263 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbbe400dbd0 con 0x7fbbf012b640 2019-12-05T06:48:10.465 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.466+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/3617093263 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbbf0129d70 con 0x7fbbf012b640 2019-12-05T06:48:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.467+0000 7fbbf5d06700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=THROTTLE_DONE pgs=1326 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:10.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.467+0000 7fbbeeffd700 1 -- 172.21.15.16:0/3617093263 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fbbe4000c10 con 0x7fbbf012b640 2019-12-05T06:48:10.469 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:10.469 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:10.469 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 162 2019-12-05T06:48:10.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.472+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/3617093263 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbbd0023630 msgr2=0x7fbbd0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.472+0000 7fbbf7f6a700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbbd0023630 0x7fbbd0025ac0 crc :-1 s=READY pgs=560 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:10.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.473+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 msgr2=0x7fbbf012abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.473+0000 7fbbf7f6a700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 0x7fbbf012abb0 secure :-1 s=READY pgs=1326 cs=0 l=1 rx=0x7fbbe4010460 tx=0x7fbbe4010260).stop 2019-12-05T06:48:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.473+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/3617093263 shutdown_connections 2019-12-05T06:48:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.473+0000 7fbbf7f6a700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbbd0023630 0x7fbbd0025ac0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:10.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.473+0000 7fbbf7f6a700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbbf012b640 0x7fbbf012abb0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:10.476 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.473+0000 7fbbf7f6a700 1 --2- 172.21.15.16:0/3617093263 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbbf01302f0 0x7fbbf012a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:10.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.473+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/3617093263 shutdown_connections 2019-12-05T06:48:10.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.473+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/3617093263 wait complete. 2019-12-05T06:48:10.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:10.473+0000 7fbbf7f6a700 1 -- 172.21.15.16:0/3617093263 >> 172.21.15.16:0/3617093263 conn(0x7fbbf00a2850 msgr2=0x7fbbf006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:10.485 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:11.392 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2019-12-05T06:48:12.486 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:12.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.755+0000 7f6905101700 1 Processor -- start 2019-12-05T06:48:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.755+0000 7f6905101700 1 -- start start 2019-12-05T06:48:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.756+0000 7f6905101700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f690008c6d0 0x7f690008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.756+0000 7f6905101700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f690008fef0 0x7f690008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.757+0000 7f6905101700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69000958e0 con 0x7f690008ef20 2019-12-05T06:48:12.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.757+0000 7f6905101700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f690008df90 con 0x7f690008c6d0 2019-12-05T06:48:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.757+0000 7f6905101700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69000a3f70 con 0x7f690008fef0 2019-12-05T06:48:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.757+0000 7f6904900700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f690008ef20 0x7f690008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46388/0 (socket says 172.21.15.16:46388) 2019-12-05T06:48:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.757+0000 7f6904900700 1 -- 172.21.15.16:0/3121898751 learned_addr learned my addr 172.21.15.16:0/3121898751 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.757+0000 7f68ff59e700 1 --2- 172.21.15.16:0/3121898751 >> v2:172.21.15.76:3300/0 conn(0x7f690008c6d0 0x7f690008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.757+0000 7f68fed9d700 1 --2- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f690008fef0 0x7f690008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:12.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.757+0000 7f68fe59c700 1 -- 172.21.15.16:0/3121898751 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904604803 0 0) 0x7f68f0001100 con 0x7f690008ef20 2019-12-05T06:48:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.758+0000 7f68fe59c700 1 -- 172.21.15.16:0/3121898751 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68ec002e80 con 0x7f690008ef20 2019-12-05T06:48:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.758+0000 7f68ff59e700 1 -- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f690008fef0 msgr2=0x7f690008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.758+0000 7f68ff59e700 1 --2- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f690008fef0 0x7f690008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.758+0000 7f68ff59e700 1 -- 172.21.15.16:0/3121898751 >> v1:172.21.15.16:6789/0 conn(0x7f690008ef20 legacy=0x7f690008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.758+0000 7f68ff59e700 1 -- 172.21.15.16:0/3121898751 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f690008d540 con 0x7f690008c6d0 2019-12-05T06:48:12.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.758+0000 7f68fed9d700 1 --2- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f690008fef0 0x7f690008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:12.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.758+0000 7f68ff59e700 1 --2- 172.21.15.16:0/3121898751 >> v2:172.21.15.76:3300/0 conn(0x7f690008c6d0 0x7f690008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f68f800b2d0 tx=0x7f68f800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.758+0000 7f68ff59e700 1 --2- 172.21.15.16:0/3121898751 >> v2:172.21.15.76:3300/0 conn(0x7f690008c6d0 0x7f690008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f68f800b2d0 tx=0x7f68f800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.758+0000 7f68ff59e700 1 --2- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690008c6d0 0x7f690008cac0 secure :-1 s=READY pgs=904 cs=0 l=1 rx=0x7f68f800b2d0 tx=0x7f68f800bae0).ready entity=mon.1 client_cookie=9167c943a40018b4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:12.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.759+0000 7f68ff59e700 1 --2- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690008c6d0 0x7f690008cac0 secure :-1 s=THROTTLE_DONE pgs=904 cs=0 l=1 rx=0x7f68f800b2d0 tx=0x7f68f800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.759+0000 7f68fe59c700 1 -- 172.21.15.16:0/3121898751 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68f8007590 con 0x7f690008c6d0 2019-12-05T06:48:12.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.759+0000 7f68ff59e700 1 --2- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690008c6d0 0x7f690008cac0 secure :-1 s=THROTTLE_DONE pgs=904 cs=0 l=1 rx=0x7f68f800b2d0 tx=0x7f68f800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.759+0000 7f68ff59e700 1 --2- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690008c6d0 0x7f690008cac0 secure :-1 s=THROTTLE_DONE pgs=904 cs=0 l=1 rx=0x7f68f800b2d0 tx=0x7f68f800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.759+0000 7f68fe59c700 1 -- 172.21.15.16:0/3121898751 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68f8009d30 con 0x7f690008c6d0 2019-12-05T06:48:12.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.759+0000 7f68fe59c700 1 -- 172.21.15.16:0/3121898751 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68f800a400 con 0x7f690008c6d0 2019-12-05T06:48:12.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.759+0000 7f6905101700 1 -- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690008c6d0 msgr2=0x7f690008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:12.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.759+0000 7f6905101700 1 --2- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690008c6d0 0x7f690008cac0 secure :-1 s=READY pgs=904 cs=0 l=1 rx=0x7f68f800b2d0 tx=0x7f68f800bae0).stop 2019-12-05T06:48:12.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.761+0000 7f6905101700 1 -- 172.21.15.16:0/3121898751 shutdown_connections 2019-12-05T06:48:12.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.761+0000 7f6905101700 1 --2- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f690008fef0 0x7f690008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:12.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.761+0000 7f6905101700 1 --2- 172.21.15.16:0/3121898751 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f690008c6d0 0x7f690008cac0 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:12.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.761+0000 7f6905101700 1 -- 172.21.15.16:0/3121898751 shutdown_connections 2019-12-05T06:48:12.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.761+0000 7f6905101700 1 -- 172.21.15.16:0/3121898751 wait complete. 2019-12-05T06:48:12.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.761+0000 7f6905101700 1 -- 172.21.15.16:0/3121898751 >> 172.21.15.16:0/3121898751 conn(0x7f6900061940 msgr2=0x7f6900096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:12.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.766+0000 7f6905101700 1 Processor -- start 2019-12-05T06:48:12.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.766+0000 7f6905101700 1 -- start start 2019-12-05T06:48:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.768+0000 7f6905101700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f69001302f0 0x7f690009d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.768+0000 7f6905101700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f690009e7d0 0x7f690009dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.768+0000 7f6905101700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f690012cb90 con 0x7f690009e280 2019-12-05T06:48:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.768+0000 7f6905101700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6900131ff0 con 0x7f690009e7d0 2019-12-05T06:48:12.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.768+0000 7f6905101700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69001315c0 con 0x7f69001302f0 2019-12-05T06:48:12.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.768+0000 7f68fed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f690009e7d0 0x7f690009dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.768+0000 7f68fed9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f690009e7d0 0x7f690009dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:32990/0 (socket says 172.21.15.16:32990) 2019-12-05T06:48:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.768+0000 7f68fed9d700 1 -- 172.21.15.16:0/2835514983 learned_addr learned my addr 172.21.15.16:0/2835514983 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.768+0000 7f68ff59e700 1 --2- 172.21.15.16:0/2835514983 >> v2:172.21.15.76:3300/0 conn(0x7f69001302f0 0x7f690009d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68fed9d700 1 -- 172.21.15.16:0/2835514983 >> v2:172.21.15.76:3300/0 conn(0x7f69001302f0 msgr2=0x7f690009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> v2:172.21.15.76:3300/0 conn(0x7f69001302f0 0x7f690009d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:12.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68fed9d700 1 -- 172.21.15.16:0/2835514983 >> v1:172.21.15.76:6789/0 conn(0x7f690009e280 legacy=0x7f690006dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:12.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68fed9d700 1 -- 172.21.15.16:0/2835514983 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6900132680 con 0x7f690009e7d0 2019-12-05T06:48:12.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> v2:172.21.15.16:3300/0 conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> v2:172.21.15.16:3300/0 conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=READY pgs=1329 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:12.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=THROTTLE_DONE pgs=1329 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68f7fff700 1 -- 172.21.15.16:0/2835514983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68e8000940 con 0x7f690009e7d0 2019-12-05T06:48:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=THROTTLE_DONE pgs=1329 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=THROTTLE_DONE pgs=1329 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68f7fff700 1 -- 172.21.15.16:0/2835514983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68e8009770 con 0x7f690009e7d0 2019-12-05T06:48:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f68f7fff700 1 -- 172.21.15.16:0/2835514983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68e8009e40 con 0x7f690009e7d0 2019-12-05T06:48:12.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f6905101700 1 -- 172.21.15.16:0/2835514983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6900132680 con 0x7f690009e7d0 2019-12-05T06:48:12.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.769+0000 7f6905101700 1 -- 172.21.15.16:0/2835514983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6900132680 con 0x7f690009e7d0 2019-12-05T06:48:12.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.770+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=THROTTLE_DONE pgs=1329 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.771+0000 7f68f7fff700 1 -- 172.21.15.16:0/2835514983 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f68e8009580 con 0x7f690009e7d0 2019-12-05T06:48:12.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.771+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=THROTTLE_DONE pgs=1329 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.771+0000 7f68f7fff700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f68e4023630 0x7f68e4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:12.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.771+0000 7f68f7fff700 1 -- 172.21.15.16:0/2835514983 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f68e8045da0 con 0x7f690009e7d0 2019-12-05T06:48:12.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.772+0000 7f68ff59e700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f68e4023630 0x7f68e4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:12.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.772+0000 7f68ff59e700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f68e4023630 0x7f68e4025ac0 crc :-1 s=READY pgs=561 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:12.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.787+0000 7f6905101700 1 -- 172.21.15.16:0/2835514983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6900132680 con 0x7f690009e7d0 2019-12-05T06:48:12.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.796+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=THROTTLE_DONE pgs=1329 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:12.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:12.796+0000 7f68f7fff700 1 -- 172.21.15.16:0/2835514983 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f68e800a5b0 con 0x7f690009e7d0 2019-12-05T06:48:13.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.055+0000 7f6905101700 1 -- 172.21.15.16:0/2835514983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6900132680 con 0x7f690009e7d0 2019-12-05T06:48:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.056+0000 7f68fed9d700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=THROTTLE_DONE pgs=1329 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:13.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.056+0000 7f68f7fff700 1 -- 172.21.15.16:0/2835514983 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f68e8009e30 con 0x7f690009e7d0 2019-12-05T06:48:13.057 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:13.058 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:13.058 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 162 2019-12-05T06:48:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.060+0000 7f6905101700 1 -- 172.21.15.16:0/2835514983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f68e4023630 msgr2=0x7f68e4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.060+0000 7f6905101700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f68e4023630 0x7f68e4025ac0 crc :-1 s=READY pgs=561 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:13.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.060+0000 7f6905101700 1 -- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 msgr2=0x7f690009dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.060+0000 7f6905101700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 0x7f690009dd40 secure :-1 s=READY pgs=1329 cs=0 l=1 rx=0x7f68e8010410 tx=0x7f68e8010290).stop 2019-12-05T06:48:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.060+0000 7f6905101700 1 -- 172.21.15.16:0/2835514983 shutdown_connections 2019-12-05T06:48:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.060+0000 7f6905101700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f68e4023630 0x7f68e4025ac0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.060+0000 7f6905101700 1 --2- 172.21.15.16:0/2835514983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f690009e7d0 0x7f690009dd40 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.060+0000 7f6905101700 1 --2- 172.21.15.16:0/2835514983 >> v2:172.21.15.76:3300/0 conn(0x7f69001302f0 0x7f690009d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:13.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.061+0000 7f6905101700 1 -- 172.21.15.16:0/2835514983 shutdown_connections 2019-12-05T06:48:13.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.061+0000 7f6905101700 1 -- 172.21.15.16:0/2835514983 wait complete. 2019-12-05T06:48:13.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:13.061+0000 7f6905101700 1 -- 172.21.15.16:0/2835514983 >> 172.21.15.16:0/2835514983 conn(0x7f69000a2850 msgr2=0x7f690006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:13.071 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:14.377 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:48:14.383 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:48:15.073 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.348+0000 7f59ec37c700 1 Processor -- start 2019-12-05T06:48:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.348+0000 7f59ec37c700 1 -- start start 2019-12-05T06:48:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.350+0000 7f59ec37c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59e408d2c0 0x7f59e408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:15.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.350+0000 7f59ec37c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f59e408fef0 0x7f59e408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.350+0000 7f59ec37c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e40958e0 con 0x7f59e408c6d0 2019-12-05T06:48:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.350+0000 7f59ec37c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e408df90 con 0x7f59e408d2c0 2019-12-05T06:48:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.350+0000 7f59ec37c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e40a3f70 con 0x7f59e408fef0 2019-12-05T06:48:15.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.351+0000 7f59ea118700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59e408d2c0 0x7f59e408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:15.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.351+0000 7f59ea118700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59e408d2c0 0x7f59e408ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33004/0 (socket says 172.21.15.16:33004) 2019-12-05T06:48:15.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.351+0000 7f59ea118700 1 -- 172.21.15.16:0/3226797180 learned_addr learned my addr 172.21.15.16:0/3226797180 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:15.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.351+0000 7f59eb11a700 1 --2- 172.21.15.16:0/3226797180 >> v2:172.21.15.76:3300/0 conn(0x7f59e408fef0 0x7f59e408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:15.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.351+0000 7f59e9917700 1 -- 172.21.15.16:0/3226797180 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1110538369 0 0) 0x7f59dc001400 con 0x7f59e408c6d0 2019-12-05T06:48:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.351+0000 7f59e9917700 1 -- 172.21.15.16:0/3226797180 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc002e80 con 0x7f59e408c6d0 2019-12-05T06:48:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.351+0000 7f59ea118700 1 -- 172.21.15.16:0/3226797180 >> v2:172.21.15.76:3300/0 conn(0x7f59e408fef0 msgr2=0x7f59e408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.351+0000 7f59ea118700 1 --2- 172.21.15.16:0/3226797180 >> v2:172.21.15.76:3300/0 conn(0x7f59e408fef0 0x7f59e408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:15.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.351+0000 7f59ea118700 1 -- 172.21.15.16:0/3226797180 >> v1:172.21.15.76:6789/0 conn(0x7f59e408c6d0 legacy=0x7f59e408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59ea118700 1 -- 172.21.15.16:0/3226797180 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e408e620 con 0x7f59e408d2c0 2019-12-05T06:48:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59eb11a700 1 --2- 172.21.15.16:0/3226797180 >> v2:172.21.15.76:3300/0 conn(0x7f59e408fef0 0x7f59e408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59ea118700 1 --2- 172.21.15.16:0/3226797180 >> v2:172.21.15.16:3300/0 conn(0x7f59e408d2c0 0x7f59e408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f59d800f740 tx=0x7f59d800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59ea118700 1 --2- 172.21.15.16:0/3226797180 >> v2:172.21.15.16:3300/0 conn(0x7f59e408d2c0 0x7f59e408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f59d800f740 tx=0x7f59d800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59ea118700 1 --2- 172.21.15.16:0/3226797180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e408d2c0 0x7f59e408ef20 secure :-1 s=READY pgs=1330 cs=0 l=1 rx=0x7f59d800f740 tx=0x7f59d800fae0).ready entity=mon.0 client_cookie=7324566ae0bb7a48 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59ea118700 1 --2- 172.21.15.16:0/3226797180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e408d2c0 0x7f59e408ef20 secure :-1 s=THROTTLE_DONE pgs=1330 cs=0 l=1 rx=0x7f59d800f740 tx=0x7f59d800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59ea118700 1 --2- 172.21.15.16:0/3226797180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e408d2c0 0x7f59e408ef20 secure :-1 s=THROTTLE_DONE pgs=1330 cs=0 l=1 rx=0x7f59d800f740 tx=0x7f59d800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59e9917700 1 -- 172.21.15.16:0/3226797180 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59d8000960 con 0x7f59e408d2c0 2019-12-05T06:48:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59ea118700 1 --2- 172.21.15.16:0/3226797180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e408d2c0 0x7f59e408ef20 secure :-1 s=THROTTLE_DONE pgs=1330 cs=0 l=1 rx=0x7f59d800f740 tx=0x7f59d800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59e9917700 1 -- 172.21.15.16:0/3226797180 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d8000c90 con 0x7f59e408d2c0 2019-12-05T06:48:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.352+0000 7f59e9917700 1 -- 172.21.15.16:0/3226797180 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59d80013e0 con 0x7f59e408d2c0 2019-12-05T06:48:15.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.353+0000 7f59ec37c700 1 -- 172.21.15.16:0/3226797180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e408d2c0 msgr2=0x7f59e408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:15.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.353+0000 7f59ec37c700 1 --2- 172.21.15.16:0/3226797180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e408d2c0 0x7f59e408ef20 secure :-1 s=READY pgs=1330 cs=0 l=1 rx=0x7f59d800f740 tx=0x7f59d800fae0).stop 2019-12-05T06:48:15.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.355+0000 7f59ec37c700 1 -- 172.21.15.16:0/3226797180 shutdown_connections 2019-12-05T06:48:15.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.355+0000 7f59ec37c700 1 --2- 172.21.15.16:0/3226797180 >> v2:172.21.15.76:3300/0 conn(0x7f59e408fef0 0x7f59e408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:15.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.355+0000 7f59ec37c700 1 --2- 172.21.15.16:0/3226797180 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e408d2c0 0x7f59e408ef20 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:15.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.355+0000 7f59ec37c700 1 -- 172.21.15.16:0/3226797180 shutdown_connections 2019-12-05T06:48:15.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.355+0000 7f59ec37c700 1 -- 172.21.15.16:0/3226797180 wait complete. 2019-12-05T06:48:15.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.355+0000 7f59ec37c700 1 -- 172.21.15.16:0/3226797180 >> 172.21.15.16:0/3226797180 conn(0x7f59e4061940 msgr2=0x7f59e4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:15.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.361+0000 7f59ec37c700 1 Processor -- start 2019-12-05T06:48:15.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.361+0000 7f59ec37c700 1 -- start start 2019-12-05T06:48:15.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.363+0000 7f59ec37c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f59e414d400 0x7f59e414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:15.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.363+0000 7f59ec37c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59e4132530 0x7f59e4157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:15.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.363+0000 7f59ec37c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e4098da0 con 0x7f59e4132d00 2019-12-05T06:48:15.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.363+0000 7f59ec37c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e409bb50 con 0x7f59e4132530 2019-12-05T06:48:15.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.363+0000 7f59ec37c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59e409aa60 con 0x7f59e414d400 2019-12-05T06:48:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.363+0000 7f59ea118700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f59e4132d00 0x7f59e4131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46410/0 (socket says 172.21.15.16:46410) 2019-12-05T06:48:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.363+0000 7f59ea118700 1 -- 172.21.15.16:0/683528367 learned_addr learned my addr 172.21.15.16:0/683528367 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.364+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> v2:172.21.15.16:3300/0 conn(0x7f59e4132530 0x7f59e4157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:15.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.364+0000 7f59ea919700 1 --2- 172.21.15.16:0/683528367 >> v2:172.21.15.76:3300/0 conn(0x7f59e414d400 0x7f59e414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.364+0000 7f59e37fe700 1 -- 172.21.15.16:0/683528367 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728026010 0 0) 0x7f59d8007da0 con 0x7f59e4132d00 2019-12-05T06:48:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.364+0000 7f59e37fe700 1 -- 172.21.15.16:0/683528367 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c4002e80 con 0x7f59e4132d00 2019-12-05T06:48:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.364+0000 7f59eb11a700 1 -- 172.21.15.16:0/683528367 >> v2:172.21.15.76:3300/0 conn(0x7f59e414d400 msgr2=0x7f59e414d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.364+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> v2:172.21.15.76:3300/0 conn(0x7f59e414d400 0x7f59e414d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:15.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.364+0000 7f59eb11a700 1 -- 172.21.15.16:0/683528367 >> v1:172.21.15.16:6789/0 conn(0x7f59e4132d00 legacy=0x7f59e4131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.364+0000 7f59eb11a700 1 -- 172.21.15.16:0/683528367 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e409b0f0 con 0x7f59e4132530 2019-12-05T06:48:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59ea919700 1 --2- 172.21.15.16:0/683528367 >> v2:172.21.15.76:3300/0 conn(0x7f59e414d400 0x7f59e414d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> v2:172.21.15.16:3300/0 conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> v2:172.21.15.16:3300/0 conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=READY pgs=1332 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=THROTTLE_DONE pgs=1332 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59e37fe700 1 -- 172.21.15.16:0/683528367 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59d400ab90 con 0x7f59e4132530 2019-12-05T06:48:15.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=THROTTLE_DONE pgs=1332 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=THROTTLE_DONE pgs=1332 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59e37fe700 1 -- 172.21.15.16:0/683528367 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d40097a0 con 0x7f59e4132530 2019-12-05T06:48:15.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59ec37c700 1 -- 172.21.15.16:0/683528367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e409b0f0 con 0x7f59e4132530 2019-12-05T06:48:15.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59e37fe700 1 -- 172.21.15.16:0/683528367 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59d4009eb0 con 0x7f59e4132530 2019-12-05T06:48:15.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.365+0000 7f59ec37c700 1 -- 172.21.15.16:0/683528367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e409b0f0 con 0x7f59e4132530 2019-12-05T06:48:15.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.366+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=THROTTLE_DONE pgs=1332 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.367+0000 7f59e37fe700 1 -- 172.21.15.16:0/683528367 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f59d4009990 con 0x7f59e4132530 2019-12-05T06:48:15.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.367+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=THROTTLE_DONE pgs=1332 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.368+0000 7f59e37fe700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59c4023670 0x7f59c4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:15.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.368+0000 7f59e37fe700 1 -- 172.21.15.16:0/683528367 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f59d4045db0 con 0x7f59e4132530 2019-12-05T06:48:15.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.368+0000 7f59ea919700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59c4023670 0x7f59c4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:15.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.369+0000 7f59ea919700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59c4023670 0x7f59c4025ae0 crc :-1 s=READY pgs=562 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:15.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.380+0000 7f59ec37c700 1 -- 172.21.15.16:0/683528367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e409b0f0 con 0x7f59e4132530 2019-12-05T06:48:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.387+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=THROTTLE_DONE pgs=1332 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.387+0000 7f59e37fe700 1 -- 172.21.15.16:0/683528367 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59d4009970 con 0x7f59e4132530 2019-12-05T06:48:15.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.631+0000 7f59ec37c700 1 -- 172.21.15.16:0/683528367 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59e409b0f0 con 0x7f59e4132530 2019-12-05T06:48:15.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.632+0000 7f59eb11a700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=THROTTLE_DONE pgs=1332 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:15.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.632+0000 7f59e37fe700 1 -- 172.21.15.16:0/683528367 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f59d4009970 con 0x7f59e4132530 2019-12-05T06:48:15.632 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 162 2019-12-05T06:48:15.633 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:15.633 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:15.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.635+0000 7f59ec37c700 1 -- 172.21.15.16:0/683528367 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59c4023670 msgr2=0x7f59c4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:15.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.635+0000 7f59ec37c700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59c4023670 0x7f59c4025ae0 crc :-1 s=READY pgs=562 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:15.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.635+0000 7f59ec37c700 1 -- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 msgr2=0x7f59e4157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.635+0000 7f59ec37c700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 0x7f59e4157100 secure :-1 s=READY pgs=1332 cs=0 l=1 rx=0x7f59d4010380 tx=0x7f59d4010720).stop 2019-12-05T06:48:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.635+0000 7f59ec37c700 1 -- 172.21.15.16:0/683528367 shutdown_connections 2019-12-05T06:48:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.635+0000 7f59ec37c700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59c4023670 0x7f59c4025ae0 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.635+0000 7f59ec37c700 1 --2- 172.21.15.16:0/683528367 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f59e4132530 0x7f59e4157100 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.635+0000 7f59ec37c700 1 --2- 172.21.15.16:0/683528367 >> v2:172.21.15.76:3300/0 conn(0x7f59e414d400 0x7f59e414d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:15.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.636+0000 7f59ec37c700 1 -- 172.21.15.16:0/683528367 shutdown_connections 2019-12-05T06:48:15.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.636+0000 7f59ec37c700 1 -- 172.21.15.16:0/683528367 wait complete. 2019-12-05T06:48:15.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:15.636+0000 7f59ec37c700 1 -- 172.21.15.16:0/683528367 >> 172.21.15.16:0/683528367 conn(0x7f59e40714f0 msgr2=0x7f59e409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:15.645 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:17.646 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:17.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.940+0000 7f6822e95700 1 Processor -- start 2019-12-05T06:48:17.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.941+0000 7f6822e95700 1 -- start start 2019-12-05T06:48:17.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6822e95700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f681c08d2c0 0x7f681c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:17.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6822e95700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c08fef0 0x7f681c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:17.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6822e95700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c0958e0 con 0x7f681c08c6d0 2019-12-05T06:48:17.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6822e95700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c08df90 con 0x7f681c08d2c0 2019-12-05T06:48:17.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6822e95700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c0a3f70 con 0x7f681c08fef0 2019-12-05T06:48:17.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6820c31700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f681c08d2c0 0x7f681c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:17.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6821c33700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c08fef0 0x7f681c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:17.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6821432700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f681c08c6d0 0x7f681c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:39488/0 (socket says 172.21.15.16:39488) 2019-12-05T06:48:17.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6820c31700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f681c08d2c0 0x7f681c08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:47342/0 (socket says 172.21.15.16:47342) 2019-12-05T06:48:17.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6821c33700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c08fef0 0x7f681c08f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:37010/0 (socket says 172.21.15.16:37010) 2019-12-05T06:48:17.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.943+0000 7f6821432700 1 -- 172.21.15.16:0/1225149216 learned_addr learned my addr 172.21.15.16:0/1225149216 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:17.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.944+0000 7f6820c31700 1 -- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c08fef0 msgr2=0x7f681c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:17.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.944+0000 7f6820c31700 1 --2- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c08fef0 0x7f681c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:17.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.944+0000 7f6820c31700 1 -- 172.21.15.16:0/1225149216 >> v1:172.21.15.76:6789/0 conn(0x7f681c08c6d0 legacy=0x7f681c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:17.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.944+0000 7f6813fff700 1 -- 172.21.15.16:0/1225149216 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1345721136 0 0) 0x7f6818001100 con 0x7f681c08c6d0 2019-12-05T06:48:17.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.944+0000 7f6820c31700 1 -- 172.21.15.16:0/1225149216 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c08e620 con 0x7f681c08d2c0 2019-12-05T06:48:17.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.944+0000 7f6821c33700 1 --2- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c08fef0 0x7f681c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:17.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.944+0000 7f6820c31700 1 --2- 172.21.15.16:0/1225149216 >> v2:172.21.15.76:3300/0 conn(0x7f681c08d2c0 0x7f681c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f680c00af50 tx=0x7f680c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.944+0000 7f6820c31700 1 --2- 172.21.15.16:0/1225149216 >> v2:172.21.15.76:3300/0 conn(0x7f681c08d2c0 0x7f681c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f680c00af50 tx=0x7f680c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.945+0000 7f6820c31700 1 --2- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f681c08d2c0 0x7f681c08ef20 secure :-1 s=READY pgs=908 cs=0 l=1 rx=0x7f680c00af50 tx=0x7f680c00b760).ready entity=mon.1 client_cookie=126dcf5484d6807b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:17.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.945+0000 7f6820c31700 1 --2- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f681c08d2c0 0x7f681c08ef20 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7f680c00af50 tx=0x7f680c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:17.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.945+0000 7f6813fff700 1 -- 172.21.15.16:0/1225149216 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f680c007210 con 0x7f681c08d2c0 2019-12-05T06:48:17.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.945+0000 7f6820c31700 1 --2- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f681c08d2c0 0x7f681c08ef20 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7f680c00af50 tx=0x7f680c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.945+0000 7f6813fff700 1 -- 172.21.15.16:0/1225149216 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c0099b0 con 0x7f681c08d2c0 2019-12-05T06:48:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.945+0000 7f6820c31700 1 --2- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f681c08d2c0 0x7f681c08ef20 secure :-1 s=THROTTLE_DONE pgs=908 cs=0 l=1 rx=0x7f680c00af50 tx=0x7f680c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.945+0000 7f6813fff700 1 -- 172.21.15.16:0/1225149216 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f680c007210 con 0x7f681c08d2c0 2019-12-05T06:48:18.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.945+0000 7f6822e95700 1 -- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f681c08d2c0 msgr2=0x7f681c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.945+0000 7f6822e95700 1 --2- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f681c08d2c0 0x7f681c08ef20 secure :-1 s=READY pgs=908 cs=0 l=1 rx=0x7f680c00af50 tx=0x7f680c00b760).stop 2019-12-05T06:48:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.947+0000 7f6822e95700 1 -- 172.21.15.16:0/1225149216 shutdown_connections 2019-12-05T06:48:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.947+0000 7f6822e95700 1 --2- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c08fef0 0x7f681c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:18.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.947+0000 7f6822e95700 1 --2- 172.21.15.16:0/1225149216 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f681c08d2c0 0x7f681c08ef20 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.948+0000 7f6822e95700 1 -- 172.21.15.16:0/1225149216 shutdown_connections 2019-12-05T06:48:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.948+0000 7f6822e95700 1 -- 172.21.15.16:0/1225149216 wait complete. 2019-12-05T06:48:18.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.948+0000 7f6822e95700 1 -- 172.21.15.16:0/1225149216 >> 172.21.15.16:0/1225149216 conn(0x7f681c061940 msgr2=0x7f681c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:18.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.953+0000 7f6822e95700 1 Processor -- start 2019-12-05T06:48:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.953+0000 7f6822e95700 1 -- start start 2019-12-05T06:48:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.955+0000 7f6822e95700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f681c06e840 0x7f681c134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.955+0000 7f6822e95700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c135450 0x7f681c1349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:18.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.955+0000 7f6822e95700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f681c128af0 con 0x7f681c06dc50 2019-12-05T06:48:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.955+0000 7f6822e95700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f681c132830 con 0x7f681c06e840 2019-12-05T06:48:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.955+0000 7f6822e95700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f681c1334f0 con 0x7f681c135450 2019-12-05T06:48:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.956+0000 7f6820c31700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f681c06e840 0x7f681c134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:18.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.956+0000 7f6821432700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f681c06dc50 0x7f681c06e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:39492/0 (socket says 172.21.15.16:39492) 2019-12-05T06:48:18.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.956+0000 7f6821432700 1 -- 172.21.15.16:0/3396163179 learned_addr learned my addr 172.21.15.16:0/3396163179 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:18.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.956+0000 7f6821c33700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c135450 0x7f681c1349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:18.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.956+0000 7f6811ffb700 1 -- 172.21.15.16:0/3396163179 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1771219646 0 0) 0x7f6818000c60 con 0x7f681c06dc50 2019-12-05T06:48:18.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.956+0000 7f6820c31700 1 -- 172.21.15.16:0/3396163179 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c135450 msgr2=0x7f681c1349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:18.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.956+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c135450 0x7f681c1349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:18.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.956+0000 7f6820c31700 1 -- 172.21.15.16:0/3396163179 >> v1:172.21.15.76:6789/0 conn(0x7f681c06dc50 legacy=0x7f681c06e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:18.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6820c31700 1 -- 172.21.15.16:0/3396163179 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f681c133b80 con 0x7f681c06e840 2019-12-05T06:48:18.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> v2:172.21.15.16:3300/0 conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> v2:172.21.15.16:3300/0 conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=READY pgs=1333 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:18.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=THROTTLE_DONE pgs=1333 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6811ffb700 1 -- 172.21.15.16:0/3396163179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f680c00b380 con 0x7f681c06e840 2019-12-05T06:48:18.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=THROTTLE_DONE pgs=1333 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=THROTTLE_DONE pgs=1333 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6811ffb700 1 -- 172.21.15.16:0/3396163179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f680c00aca0 con 0x7f681c06e840 2019-12-05T06:48:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6822e95700 1 -- 172.21.15.16:0/3396163179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f681c133b80 con 0x7f681c06e840 2019-12-05T06:48:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.957+0000 7f6811ffb700 1 -- 172.21.15.16:0/3396163179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f680c009d60 con 0x7f681c06e840 2019-12-05T06:48:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.958+0000 7f6822e95700 1 -- 172.21.15.16:0/3396163179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f681c133b80 con 0x7f681c06e840 2019-12-05T06:48:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.958+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=THROTTLE_DONE pgs=1333 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.959+0000 7f6811ffb700 1 -- 172.21.15.16:0/3396163179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f680c009790 con 0x7f681c06e840 2019-12-05T06:48:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.959+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=THROTTLE_DONE pgs=1333 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.960+0000 7f6811ffb700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f67fc023630 0x7f67fc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.960+0000 7f6811ffb700 1 -- 172.21.15.16:0/3396163179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f680c089d40 con 0x7f681c06e840 2019-12-05T06:48:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.960+0000 7f6821432700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f67fc023630 0x7f67fc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.961+0000 7f6821432700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f67fc023630 0x7f67fc025ac0 crc :-1 s=READY pgs=563 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.972+0000 7f6822e95700 1 -- 172.21.15.16:0/3396163179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f681c133b80 con 0x7f681c06e840 2019-12-05T06:48:18.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.981+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=THROTTLE_DONE pgs=1333 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:17.981+0000 7f6811ffb700 1 -- 172.21.15.16:0/3396163179 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f680c009580 con 0x7f681c06e840 2019-12-05T06:48:18.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.242+0000 7f6822e95700 1 -- 172.21.15.16:0/3396163179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f681c133b80 con 0x7f681c06e840 2019-12-05T06:48:18.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.243+0000 7f6820c31700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=THROTTLE_DONE pgs=1333 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:18.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.243+0000 7f6811ffb700 1 -- 172.21.15.16:0/3396163179 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f680c009790 con 0x7f681c06e840 2019-12-05T06:48:18.246 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 162 2019-12-05T06:48:18.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.245+0000 7f6822e95700 1 -- 172.21.15.16:0/3396163179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f67fc023630 msgr2=0x7f67fc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:18.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.245+0000 7f6822e95700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f67fc023630 0x7f67fc025ac0 crc :-1 s=READY pgs=563 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:18.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.245+0000 7f6822e95700 1 -- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 msgr2=0x7f681c134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:18.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.245+0000 7f6822e95700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 0x7f681c134480 secure :-1 s=READY pgs=1333 cs=0 l=1 rx=0x7f680c0543e0 tx=0x7f680c054210).stop 2019-12-05T06:48:18.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.245+0000 7f6822e95700 1 -- 172.21.15.16:0/3396163179 shutdown_connections 2019-12-05T06:48:18.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.245+0000 7f6822e95700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f681c135450 0x7f681c1349c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:18.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.246+0000 7f6822e95700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f67fc023630 0x7f67fc025ac0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:18.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.246+0000 7f6822e95700 1 --2- 172.21.15.16:0/3396163179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f681c06e840 0x7f681c134480 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:18.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.246+0000 7f6822e95700 1 -- 172.21.15.16:0/3396163179 shutdown_connections 2019-12-05T06:48:18.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.246+0000 7f6822e95700 1 -- 172.21.15.16:0/3396163179 wait complete. 2019-12-05T06:48:18.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:18.246+0000 7f6822e95700 1 -- 172.21.15.16:0/3396163179 >> 172.21.15.16:0/3396163179 conn(0x7f681c0a2850 msgr2=0x7f681c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:18.248 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:18.248 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:18.264 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:20.266 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.529+0000 7f435e983700 1 Processor -- start 2019-12-05T06:48:20.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.529+0000 7f435e983700 1 -- start start 2019-12-05T06:48:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f435e983700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f435808c6d0 0x7f435808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:20.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f435e983700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f435808fef0 0x7f435808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:20.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f435e983700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f435808ef20 0x7f435808f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:20.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f435e983700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435808df90 con 0x7f435808c6d0 2019-12-05T06:48:20.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f435e983700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43580a3f70 con 0x7f435808fef0 2019-12-05T06:48:20.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f435e983700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f435808d830 con 0x7f435808ef20 2019-12-05T06:48:20.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f435cf20700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f435808c6d0 0x7f435808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:20.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f435cf20700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f435808c6d0 0x7f435808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33030/0 (socket says 172.21.15.16:33030) 2019-12-05T06:48:20.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f435cf20700 1 -- 172.21.15.16:0/3421306710 learned_addr learned my addr 172.21.15.16:0/3421306710 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:20.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f435d721700 1 --2- 172.21.15.16:0/3421306710 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f435808ef20 0x7f435808f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.531+0000 7f4357fff700 1 --2- 172.21.15.16:0/3421306710 >> v2:172.21.15.76:3300/0 conn(0x7f435808fef0 0x7f435808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.532+0000 7f435cf20700 1 -- 172.21.15.16:0/3421306710 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f435808ef20 msgr2=0x7f435808f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:20.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.532+0000 7f435cf20700 1 --2- 172.21.15.16:0/3421306710 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f435808ef20 0x7f435808f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.532+0000 7f435cf20700 1 -- 172.21.15.16:0/3421306710 >> v2:172.21.15.76:3300/0 conn(0x7f435808fef0 msgr2=0x7f435808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.532+0000 7f435cf20700 1 --2- 172.21.15.16:0/3421306710 >> v2:172.21.15.76:3300/0 conn(0x7f435808fef0 0x7f435808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.532+0000 7f435cf20700 1 -- 172.21.15.16:0/3421306710 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435808d540 con 0x7f435808c6d0 2019-12-05T06:48:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.532+0000 7f435d721700 1 --2- 172.21.15.16:0/3421306710 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f435808ef20 0x7f435808f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.532+0000 7f4357fff700 1 --2- 172.21.15.16:0/3421306710 >> v2:172.21.15.76:3300/0 conn(0x7f435808fef0 0x7f435808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:20.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.532+0000 7f435cf20700 1 --2- 172.21.15.16:0/3421306710 >> v2:172.21.15.16:3300/0 conn(0x7f435808c6d0 0x7f435808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f435000fac0 tx=0x7f435000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.532+0000 7f435cf20700 1 --2- 172.21.15.16:0/3421306710 >> v2:172.21.15.16:3300/0 conn(0x7f435808c6d0 0x7f435808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f435000fac0 tx=0x7f435000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.532+0000 7f435cf20700 1 --2- 172.21.15.16:0/3421306710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f435808c6d0 0x7f435808cac0 secure :-1 s=READY pgs=1334 cs=0 l=1 rx=0x7f435000fac0 tx=0x7f435000fe60).ready entity=mon.0 client_cookie=4d9ebf2b2bb799a5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.533+0000 7f435cf20700 1 --2- 172.21.15.16:0/3421306710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f435808c6d0 0x7f435808cac0 secure :-1 s=THROTTLE_DONE pgs=1334 cs=0 l=1 rx=0x7f435000fac0 tx=0x7f435000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.533+0000 7f435cf20700 1 --2- 172.21.15.16:0/3421306710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f435808c6d0 0x7f435808cac0 secure :-1 s=THROTTLE_DONE pgs=1334 cs=0 l=1 rx=0x7f435000fac0 tx=0x7f435000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.533+0000 7f435cf20700 1 --2- 172.21.15.16:0/3421306710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f435808c6d0 0x7f435808cac0 secure :-1 s=THROTTLE_DONE pgs=1334 cs=0 l=1 rx=0x7f435000fac0 tx=0x7f435000fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.533+0000 7f43577fe700 1 -- 172.21.15.16:0/3421306710 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4350000ce0 con 0x7f435808c6d0 2019-12-05T06:48:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.533+0000 7f43577fe700 1 -- 172.21.15.16:0/3421306710 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4350001200 con 0x7f435808c6d0 2019-12-05T06:48:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.533+0000 7f43577fe700 1 -- 172.21.15.16:0/3421306710 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4350001950 con 0x7f435808c6d0 2019-12-05T06:48:20.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.533+0000 7f435e983700 1 -- 172.21.15.16:0/3421306710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f435808c6d0 msgr2=0x7f435808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:20.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.533+0000 7f435e983700 1 --2- 172.21.15.16:0/3421306710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f435808c6d0 0x7f435808cac0 secure :-1 s=READY pgs=1334 cs=0 l=1 rx=0x7f435000fac0 tx=0x7f435000fe60).stop 2019-12-05T06:48:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.534+0000 7f435e983700 1 -- 172.21.15.16:0/3421306710 shutdown_connections 2019-12-05T06:48:20.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.534+0000 7f435e983700 1 --2- 172.21.15.16:0/3421306710 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f435808ef20 0x7f435808f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.534+0000 7f435e983700 1 --2- 172.21.15.16:0/3421306710 >> v2:172.21.15.76:3300/0 conn(0x7f435808fef0 0x7f435808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.534+0000 7f435e983700 1 --2- 172.21.15.16:0/3421306710 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f435808c6d0 0x7f435808cac0 unknown :-1 s=CLOSED pgs=1334 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.534+0000 7f435e983700 1 -- 172.21.15.16:0/3421306710 shutdown_connections 2019-12-05T06:48:20.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.534+0000 7f435e983700 1 -- 172.21.15.16:0/3421306710 wait complete. 2019-12-05T06:48:20.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.534+0000 7f435e983700 1 -- 172.21.15.16:0/3421306710 >> 172.21.15.16:0/3421306710 conn(0x7f4358061940 msgr2=0x7f4358096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:20.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.539+0000 7f435e983700 1 Processor -- start 2019-12-05T06:48:20.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.539+0000 7f435e983700 1 -- start start 2019-12-05T06:48:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.541+0000 7f435e983700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4358096e90 0x7f435809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:20.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.541+0000 7f435e983700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4358084b90 con 0x7f4358084f30 2019-12-05T06:48:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.541+0000 7f435e983700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4358098790 con 0x7f435809b920 2019-12-05T06:48:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.541+0000 7f435e983700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f435809a450 con 0x7f4358096e90 2019-12-05T06:48:20.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.541+0000 7f4357fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4358084f30 0x7f4358097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46438/0 (socket says 172.21.15.16:46438) 2019-12-05T06:48:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.541+0000 7f435cf20700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4358096e90 0x7f435809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.542+0000 7f4357fff700 1 -- 172.21.15.16:0/1392303542 learned_addr learned my addr 172.21.15.16:0/1392303542 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.542+0000 7f435cf20700 1 -- 172.21.15.16:0/1392303542 >> v1:172.21.15.76:6789/0 conn(0x7f435809b920 legacy=0x7f4358097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:20.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.542+0000 7f435cf20700 1 -- 172.21.15.16:0/1392303542 >> v1:172.21.15.16:6789/0 conn(0x7f4358084f30 legacy=0x7f4358097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.542+0000 7f435cf20700 1 -- 172.21.15.16:0/1392303542 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f435809aae0 con 0x7f4358096e90 2019-12-05T06:48:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.542+0000 7f43557fa700 1 -- 172.21.15.16:0/1392303542 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3113786856 0 0) 0x7f434800dd90 con 0x7f4358084f30 2019-12-05T06:48:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.542+0000 7f435cf20700 1 --2- 172.21.15.16:0/1392303542 >> v2:172.21.15.16:3300/0 conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.543+0000 7f435cf20700 1 --2- 172.21.15.16:0/1392303542 >> v2:172.21.15.16:3300/0 conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.543+0000 7f435cf20700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=READY pgs=1336 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.543+0000 7f435cf20700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.543+0000 7f43557fa700 1 -- 172.21.15.16:0/1392303542 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43500084e0 con 0x7f4358096e90 2019-12-05T06:48:20.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.543+0000 7f435cf20700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.543+0000 7f435cf20700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.543+0000 7f43557fa700 1 -- 172.21.15.16:0/1392303542 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4350005900 con 0x7f4358096e90 2019-12-05T06:48:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.543+0000 7f435e983700 1 -- 172.21.15.16:0/1392303542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f435809aae0 con 0x7f4358096e90 2019-12-05T06:48:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.543+0000 7f43557fa700 1 -- 172.21.15.16:0/1392303542 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4350006050 con 0x7f4358096e90 2019-12-05T06:48:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.543+0000 7f435e983700 1 -- 172.21.15.16:0/1392303542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f435809aae0 con 0x7f4358096e90 2019-12-05T06:48:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.544+0000 7f435cf20700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.545+0000 7f43557fa700 1 -- 172.21.15.16:0/1392303542 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4350008460 con 0x7f4358096e90 2019-12-05T06:48:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.545+0000 7f435cf20700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.545+0000 7f43557fa700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f433c023630 0x7f433c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.545+0000 7f43557fa700 1 -- 172.21.15.16:0/1392303542 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7f435008a150 con 0x7f4358096e90 2019-12-05T06:48:20.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.546+0000 7f4357fff700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f433c023630 0x7f433c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:20.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.546+0000 7f4357fff700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f433c023630 0x7f433c025ac0 crc :-1 s=READY pgs=564 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:20.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.561+0000 7f435e983700 1 -- 172.21.15.16:0/1392303542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f435809aae0 con 0x7f4358096e90 2019-12-05T06:48:20.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.567+0000 7f435cf20700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.578 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.567+0000 7f43557fa700 1 -- 172.21.15.16:0/1392303542 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f435006f170 con 0x7f4358096e90 2019-12-05T06:48:20.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.812+0000 7f435e983700 1 -- 172.21.15.16:0/1392303542 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f435809aae0 con 0x7f4358096e90 2019-12-05T06:48:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.813+0000 7f435cf20700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=THROTTLE_DONE pgs=1336 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.813+0000 7f43557fa700 1 -- 172.21.15.16:0/1392303542 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f4350005db0 con 0x7f4358096e90 2019-12-05T06:48:20.813 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 162 2019-12-05T06:48:20.813 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:20.814 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.815+0000 7f435e983700 1 -- 172.21.15.16:0/1392303542 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f433c023630 msgr2=0x7f433c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.815+0000 7f435e983700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f433c023630 0x7f433c025ac0 crc :-1 s=READY pgs=564 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.815+0000 7f435e983700 1 -- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 msgr2=0x7f435809b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:20.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.815+0000 7f435e983700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 0x7f435809b3e0 secure :-1 s=READY pgs=1336 cs=0 l=1 rx=0x7f4350054740 tx=0x7f4350054ae0).stop 2019-12-05T06:48:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.815+0000 7f435e983700 1 -- 172.21.15.16:0/1392303542 shutdown_connections 2019-12-05T06:48:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.815+0000 7f435e983700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f433c023630 0x7f433c025ac0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.815+0000 7f435e983700 1 --2- 172.21.15.16:0/1392303542 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4358096e90 0x7f435809b3e0 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.816+0000 7f435e983700 1 -- 172.21.15.16:0/1392303542 shutdown_connections 2019-12-05T06:48:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.816+0000 7f435e983700 1 -- 172.21.15.16:0/1392303542 wait complete. 2019-12-05T06:48:20.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:20.816+0000 7f435e983700 1 -- 172.21.15.16:0/1392303542 >> 172.21.15.16:0/1392303542 conn(0x7f43580714f0 msgr2=0x7f435809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:20.827 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:22.828 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:23.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.086+0000 7fde41dc3700 1 Processor -- start 2019-12-05T06:48:23.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.086+0000 7fde41dc3700 1 -- start start 2019-12-05T06:48:23.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.088+0000 7fde41dc3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde3c08d2c0 0x7fde3c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:23.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.088+0000 7fde41dc3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3c0958e0 con 0x7fde3c08c6d0 2019-12-05T06:48:23.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.088+0000 7fde41dc3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3c0629e0 con 0x7fde3c08fef0 2019-12-05T06:48:23.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.088+0000 7fde41dc3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde3c08df90 con 0x7fde3c08d2c0 2019-12-05T06:48:23.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.088+0000 7fde3b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde3c08d2c0 0x7fde3c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.089+0000 7fde3b7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde3c08d2c0 0x7fde3c08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33046/0 (socket says 172.21.15.16:33046) 2019-12-05T06:48:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.089+0000 7fde3b7fe700 1 -- 172.21.15.16:0/3809917709 learned_addr learned my addr 172.21.15.16:0/3809917709 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.089+0000 7fde3bfff700 1 --1- 172.21.15.16:0/3809917709 >> v1:172.21.15.16:6789/0 conn(0x7fde3c08c6d0 0x7fde3c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46444/0 (socket says 172.21.15.16:46444) 2019-12-05T06:48:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.089+0000 7fde3b7fe700 1 -- 172.21.15.16:0/3809917709 >> v1:172.21.15.76:6789/0 conn(0x7fde3c08fef0 legacy=0x7fde3c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.089+0000 7fde3b7fe700 1 -- 172.21.15.16:0/3809917709 >> v1:172.21.15.16:6789/0 conn(0x7fde3c08c6d0 legacy=0x7fde3c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:23.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.089+0000 7fde3affd700 1 -- 172.21.15.16:0/3809917709 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703291722 0 0) 0x7fde340009b0 con 0x7fde3c08fef0 2019-12-05T06:48:23.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.089+0000 7fde3b7fe700 1 -- 172.21.15.16:0/3809917709 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c08e620 con 0x7fde3c08d2c0 2019-12-05T06:48:23.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde3b7fe700 1 --2- 172.21.15.16:0/3809917709 >> v2:172.21.15.16:3300/0 conn(0x7fde3c08d2c0 0x7fde3c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fde2400f740 tx=0x7fde2400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde3b7fe700 1 --2- 172.21.15.16:0/3809917709 >> v2:172.21.15.16:3300/0 conn(0x7fde3c08d2c0 0x7fde3c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fde2400f740 tx=0x7fde2400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde3b7fe700 1 --2- 172.21.15.16:0/3809917709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c08d2c0 0x7fde3c08ef20 secure :-1 s=READY pgs=1338 cs=0 l=1 rx=0x7fde2400f740 tx=0x7fde2400fae0).ready entity=mon.0 client_cookie=5a3adf98b5715061 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:23.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde3b7fe700 1 --2- 172.21.15.16:0/3809917709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c08d2c0 0x7fde3c08ef20 secure :-1 s=THROTTLE_DONE pgs=1338 cs=0 l=1 rx=0x7fde2400f740 tx=0x7fde2400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde3affd700 1 -- 172.21.15.16:0/3809917709 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde24000960 con 0x7fde3c08d2c0 2019-12-05T06:48:23.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde3b7fe700 1 --2- 172.21.15.16:0/3809917709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c08d2c0 0x7fde3c08ef20 secure :-1 s=THROTTLE_DONE pgs=1338 cs=0 l=1 rx=0x7fde2400f740 tx=0x7fde2400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde3b7fe700 1 --2- 172.21.15.16:0/3809917709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c08d2c0 0x7fde3c08ef20 secure :-1 s=THROTTLE_DONE pgs=1338 cs=0 l=1 rx=0x7fde2400f740 tx=0x7fde2400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde3affd700 1 -- 172.21.15.16:0/3809917709 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde24000c90 con 0x7fde3c08d2c0 2019-12-05T06:48:23.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde3affd700 1 -- 172.21.15.16:0/3809917709 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde24001360 con 0x7fde3c08d2c0 2019-12-05T06:48:23.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde41dc3700 1 -- 172.21.15.16:0/3809917709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c08d2c0 msgr2=0x7fde3c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:23.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.090+0000 7fde41dc3700 1 --2- 172.21.15.16:0/3809917709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c08d2c0 0x7fde3c08ef20 secure :-1 s=READY pgs=1338 cs=0 l=1 rx=0x7fde2400f740 tx=0x7fde2400fae0).stop 2019-12-05T06:48:23.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.092+0000 7fde41dc3700 1 -- 172.21.15.16:0/3809917709 shutdown_connections 2019-12-05T06:48:23.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.092+0000 7fde41dc3700 1 --2- 172.21.15.16:0/3809917709 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c08d2c0 0x7fde3c08ef20 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:23.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.092+0000 7fde41dc3700 1 -- 172.21.15.16:0/3809917709 shutdown_connections 2019-12-05T06:48:23.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.092+0000 7fde41dc3700 1 -- 172.21.15.16:0/3809917709 wait complete. 2019-12-05T06:48:23.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.092+0000 7fde41dc3700 1 -- 172.21.15.16:0/3809917709 >> 172.21.15.16:0/3809917709 conn(0x7fde3c061940 msgr2=0x7fde3c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:23.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.097+0000 7fde41dc3700 1 Processor -- start 2019-12-05T06:48:23.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.098+0000 7fde41dc3700 1 -- start start 2019-12-05T06:48:23.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.099+0000 7fde41dc3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde3c14f5b0 0x7fde3c14f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:23.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.099+0000 7fde41dc3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fde3c0992d0 0x7fde3c098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:23.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.099+0000 7fde41dc3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde3c098840 0x7fde3c088fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:23.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.099+0000 7fde41dc3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde3c08ad00 con 0x7fde3c098840 2019-12-05T06:48:23.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.099+0000 7fde41dc3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde3c096c10 con 0x7fde3c0992d0 2019-12-05T06:48:23.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.099+0000 7fde41dc3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde3c097a00 con 0x7fde3c14f5b0 2019-12-05T06:48:23.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde40b61700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde3c098840 0x7fde3c088fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:23.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde3b7fe700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fde3c0992d0 0x7fde3c098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:23.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde40b61700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde3c098840 0x7fde3c088fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33054/0 (socket says 172.21.15.16:33054) 2019-12-05T06:48:23.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde40b61700 1 -- 172.21.15.16:0/1147394509 learned_addr learned my addr 172.21.15.16:0/1147394509 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:23.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde3bfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde3c14f5b0 0x7fde3c14f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:23.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde40b61700 1 -- 172.21.15.16:0/1147394509 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde3c14f5b0 msgr2=0x7fde3c14f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:23.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde3c14f5b0 0x7fde3c14f9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:23.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde40b61700 1 -- 172.21.15.16:0/1147394509 >> v2:172.21.15.76:3300/0 conn(0x7fde3c0992d0 msgr2=0x7fde3c098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:23.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> v2:172.21.15.76:3300/0 conn(0x7fde3c0992d0 0x7fde3c098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:23.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde40b61700 1 -- 172.21.15.16:0/1147394509 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde3c06e7c0 con 0x7fde3c098840 2019-12-05T06:48:23.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.100+0000 7fde3b7fe700 1 --2- 172.21.15.16:0/1147394509 >> v2:172.21.15.76:3300/0 conn(0x7fde3c0992d0 0x7fde3c098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:23.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> v2:172.21.15.16:3300/0 conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> v2:172.21.15.16:3300/0 conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=READY pgs=1339 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:23.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=THROTTLE_DONE pgs=1339 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=THROTTLE_DONE pgs=1339 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=THROTTLE_DONE pgs=1339 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde38ff9700 1 -- 172.21.15.16:0/1147394509 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde34000940 con 0x7fde3c098840 2019-12-05T06:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde38ff9700 1 -- 172.21.15.16:0/1147394509 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde34005580 con 0x7fde3c098840 2019-12-05T06:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde41dc3700 1 -- 172.21.15.16:0/1147394509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde3c06e7c0 con 0x7fde3c098840 2019-12-05T06:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde38ff9700 1 -- 172.21.15.16:0/1147394509 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde34005b50 con 0x7fde3c098840 2019-12-05T06:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.101+0000 7fde41dc3700 1 -- 172.21.15.16:0/1147394509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde3c06e7c0 con 0x7fde3c098840 2019-12-05T06:48:23.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.102+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=THROTTLE_DONE pgs=1339 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.103+0000 7fde38ff9700 1 -- 172.21.15.16:0/1147394509 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fde34005770 con 0x7fde3c098840 2019-12-05T06:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.103+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=THROTTLE_DONE pgs=1339 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.104+0000 7fde38ff9700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde20023630 0x7fde20025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.104+0000 7fde38ff9700 1 -- 172.21.15.16:0/1147394509 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 8267+0+0 (secure 0 0 0) 0x7fde34045d10 con 0x7fde3c098840 2019-12-05T06:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.104+0000 7fde3bfff700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde20023630 0x7fde20025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:23.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.104+0000 7fde3bfff700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde20023630 0x7fde20025ac0 crc :-1 s=READY pgs=565 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:23.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.118+0000 7fde41dc3700 1 -- 172.21.15.16:0/1147394509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde3c098f70 con 0x7fde3c098840 2019-12-05T06:48:23.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.128+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=THROTTLE_DONE pgs=1339 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.128+0000 7fde38ff9700 1 -- 172.21.15.16:0/1147394509 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fde340075a0 con 0x7fde3c098840 2019-12-05T06:48:23.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.370+0000 7fde41dc3700 1 -- 172.21.15.16:0/1147394509 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde3c098f70 con 0x7fde3c098840 2019-12-05T06:48:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.371+0000 7fde40b61700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=THROTTLE_DONE pgs=1339 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.371+0000 7fde38ff9700 1 -- 172.21.15.16:0/1147394509 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 162 v162) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fde34005750 con 0x7fde3c098840 2019-12-05T06:48:23.371 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 162 2019-12-05T06:48:23.372 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:23.372 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":162,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7546,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":162}],"filesystems":[{"mdsmap":{"epoch":161,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:45:52.682968+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":75,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7401},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7401":{"gid":7401,"name":"a","rank":0,"incarnation":158,"state":"up:active","state_seq":19,"addr":"172.21.15.16:6833/330412031","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":330412031},{"type":"v1","addr":"172.21.15.16:6833","nonce":330412031}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:23.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 -- 172.21.15.16:0/1147394509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde20023630 msgr2=0x7fde20025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:23.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde20023630 0x7fde20025ac0 crc :-1 s=READY pgs=565 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:23.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 -- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 msgr2=0x7fde3c088fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:23.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 0x7fde3c088fa0 secure :-1 s=READY pgs=1339 cs=0 l=1 rx=0x7fde34010410 tx=0x7fde34010290).stop 2019-12-05T06:48:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 -- 172.21.15.16:0/1147394509 shutdown_connections 2019-12-05T06:48:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde20023630 0x7fde20025ac0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde3c098840 0x7fde3c088fa0 unknown :-1 s=CLOSED pgs=1339 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 --2- 172.21.15.16:0/1147394509 >> v2:172.21.15.76:3300/0 conn(0x7fde3c0992d0 0x7fde3c098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:23.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 --2- 172.21.15.16:0/1147394509 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde3c14f5b0 0x7fde3c14f9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 -- 172.21.15.16:0/1147394509 shutdown_connections 2019-12-05T06:48:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 -- 172.21.15.16:0/1147394509 wait complete. 2019-12-05T06:48:23.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:23.374+0000 7fde41dc3700 1 -- 172.21.15.16:0/1147394509 >> 172.21.15.16:0/1147394509 conn(0x7fde3c0714f0 msgr2=0x7fde3c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:23.385 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:48:25.386 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.646+0000 7fba26a29700 1 Processor -- start 2019-12-05T06:48:25.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.646+0000 7fba26a29700 1 -- start start 2019-12-05T06:48:25.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.648+0000 7fba26a29700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba2008c6d0 0x7fba2008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.648+0000 7fba26a29700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fba2008f720 0x7fba2008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.648+0000 7fba26a29700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba200958e0 con 0x7fba2008fef0 2019-12-05T06:48:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.648+0000 7fba26a29700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba200a3f70 con 0x7fba2008f720 2019-12-05T06:48:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.648+0000 7fba26a29700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba2008d830 con 0x7fba2008c6d0 2019-12-05T06:48:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.648+0000 7fba257c7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fba2008f720 0x7fba2008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.648+0000 7fba1ffff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fba2008fef0 0x7fba2008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:39528/0 (socket says 172.21.15.16:39528) 2019-12-05T06:48:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.648+0000 7fba1ffff700 1 -- 172.21.15.16:0/2235224465 learned_addr learned my addr 172.21.15.16:0/2235224465 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:25.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.648+0000 7fba24fc6700 1 --2- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba2008c6d0 0x7fba2008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:25.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.648+0000 7fba1f7fe700 1 -- 172.21.15.16:0/2235224465 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039642182 0 0) 0x7fba10000fe0 con 0x7fba2008fef0 2019-12-05T06:48:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba1f7fe700 1 -- 172.21.15.16:0/2235224465 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba0c002e80 con 0x7fba2008fef0 2019-12-05T06:48:25.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba257c7700 1 -- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba2008c6d0 msgr2=0x7fba2008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba257c7700 1 --2- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba2008c6d0 0x7fba2008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba257c7700 1 -- 172.21.15.16:0/2235224465 >> v1:172.21.15.76:6789/0 conn(0x7fba2008fef0 legacy=0x7fba2008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba257c7700 1 -- 172.21.15.16:0/2235224465 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba2008fbc0 con 0x7fba2008f720 2019-12-05T06:48:25.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba24fc6700 1 --2- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba2008c6d0 0x7fba2008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba257c7700 1 --2- 172.21.15.16:0/2235224465 >> v2:172.21.15.76:3300/0 conn(0x7fba2008f720 0x7fba2008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fba1800af80 tx=0x7fba1800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba257c7700 1 --2- 172.21.15.16:0/2235224465 >> v2:172.21.15.76:3300/0 conn(0x7fba2008f720 0x7fba2008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fba1800af80 tx=0x7fba1800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba257c7700 1 --2- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba2008f720 0x7fba2008d000 secure :-1 s=READY pgs=913 cs=0 l=1 rx=0x7fba1800af80 tx=0x7fba1800b790).ready entity=mon.1 client_cookie=2ca7f7d05c5e04e0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba257c7700 1 --2- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba2008f720 0x7fba2008d000 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fba1800af80 tx=0x7fba1800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba1f7fe700 1 -- 172.21.15.16:0/2235224465 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba18000bc0 con 0x7fba2008f720 2019-12-05T06:48:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba257c7700 1 --2- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba2008f720 0x7fba2008d000 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fba1800af80 tx=0x7fba1800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.649+0000 7fba257c7700 1 --2- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba2008f720 0x7fba2008d000 secure :-1 s=THROTTLE_DONE pgs=913 cs=0 l=1 rx=0x7fba1800af80 tx=0x7fba1800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.650+0000 7fba1f7fe700 1 -- 172.21.15.16:0/2235224465 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba180097f0 con 0x7fba2008f720 2019-12-05T06:48:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.650+0000 7fba1f7fe700 1 -- 172.21.15.16:0/2235224465 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba18009ec0 con 0x7fba2008f720 2019-12-05T06:48:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.650+0000 7fba26a29700 1 -- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba2008f720 msgr2=0x7fba2008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.650+0000 7fba26a29700 1 --2- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba2008f720 0x7fba2008d000 secure :-1 s=READY pgs=913 cs=0 l=1 rx=0x7fba1800af80 tx=0x7fba1800b790).stop 2019-12-05T06:48:25.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.651+0000 7fba26a29700 1 -- 172.21.15.16:0/2235224465 shutdown_connections 2019-12-05T06:48:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.651+0000 7fba26a29700 1 --2- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba2008f720 0x7fba2008d000 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.651+0000 7fba26a29700 1 --2- 172.21.15.16:0/2235224465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba2008c6d0 0x7fba2008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.651+0000 7fba26a29700 1 -- 172.21.15.16:0/2235224465 shutdown_connections 2019-12-05T06:48:25.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.651+0000 7fba26a29700 1 -- 172.21.15.16:0/2235224465 wait complete. 2019-12-05T06:48:25.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.651+0000 7fba26a29700 1 -- 172.21.15.16:0/2235224465 >> 172.21.15.16:0/2235224465 conn(0x7fba20061940 msgr2=0x7fba20096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.655+0000 7fba26a29700 1 Processor -- start 2019-12-05T06:48:25.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.655+0000 7fba26a29700 1 -- start start 2019-12-05T06:48:25.675 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.657+0000 7fba26a29700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fba201302f0 0x7fba2012a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.657+0000 7fba26a29700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba2012b640 0x7fba2012abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:25.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.657+0000 7fba26a29700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba20131bf0 con 0x7fba2012b0f0 2019-12-05T06:48:25.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.657+0000 7fba26a29700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba201296e0 con 0x7fba2012b640 2019-12-05T06:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.657+0000 7fba26a29700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba20128cb0 con 0x7fba201302f0 2019-12-05T06:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.657+0000 7fba24fc6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fba201302f0 0x7fba2012a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.657+0000 7fba1ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba2012b640 0x7fba2012abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.657+0000 7fba257c7700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fba2012b0f0 0x7fba201306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46468/0 (socket says 172.21.15.16:46468) 2019-12-05T06:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.657+0000 7fba24fc6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fba201302f0 0x7fba2012a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:47386/0 (socket says 172.21.15.16:47386) 2019-12-05T06:48:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.657+0000 7fba257c7700 1 -- 172.21.15.16:0/1595338182 learned_addr learned my addr 172.21.15.16:0/1595338182 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.658+0000 7fba24fc6700 1 -- 172.21.15.16:0/1595338182 >> v2:172.21.15.16:3300/0 conn(0x7fba2012b640 msgr2=0x7fba2012abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.658+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> v2:172.21.15.16:3300/0 conn(0x7fba2012b640 0x7fba2012abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.658+0000 7fba24fc6700 1 -- 172.21.15.16:0/1595338182 >> v1:172.21.15.16:6789/0 conn(0x7fba2012b0f0 legacy=0x7fba201306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.658+0000 7fba24fc6700 1 -- 172.21.15.16:0/1595338182 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba20129d70 con 0x7fba201302f0 2019-12-05T06:48:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.658+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> v2:172.21.15.76:3300/0 conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.658+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> v2:172.21.15.76:3300/0 conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.658+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=READY pgs=914 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.658+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=THROTTLE_DONE pgs=914 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.659+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=THROTTLE_DONE pgs=914 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.659+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=THROTTLE_DONE pgs=914 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.659+0000 7fba1d7fa700 1 -- 172.21.15.16:0/1595338182 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba08000940 con 0x7fba201302f0 2019-12-05T06:48:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.659+0000 7fba1d7fa700 1 -- 172.21.15.16:0/1595338182 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba0800f040 con 0x7fba201302f0 2019-12-05T06:48:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.659+0000 7fba26a29700 1 -- 172.21.15.16:0/1595338182 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba20129d70 con 0x7fba201302f0 2019-12-05T06:48:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.659+0000 7fba1d7fa700 1 -- 172.21.15.16:0/1595338182 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fba0800f6d0 con 0x7fba201302f0 2019-12-05T06:48:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.659+0000 7fba26a29700 1 -- 172.21.15.16:0/1595338182 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba20129d70 con 0x7fba201302f0 2019-12-05T06:48:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.660+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=THROTTLE_DONE pgs=914 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.660+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=THROTTLE_DONE pgs=914 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.660+0000 7fba1d7fa700 1 -- 172.21.15.16:0/1595338182 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fba0800f230 con 0x7fba201302f0 2019-12-05T06:48:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.660+0000 7fba1d7fa700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fba04023630 0x7fba04025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:25.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.660+0000 7fba1d7fa700 1 -- 172.21.15.16:0/1595338182 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7fba0804e750 con 0x7fba201302f0 2019-12-05T06:48:25.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.661+0000 7fba1ffff700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fba04023630 0x7fba04025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:25.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.661+0000 7fba1ffff700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fba04023630 0x7fba04025ac0 crc :-1 s=READY pgs=566 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:25.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.662+0000 7fba26a29700 1 -- 172.21.15.16:0/1595338182 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba20129d70 con 0x7fba201302f0 2019-12-05T06:48:25.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.667+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=THROTTLE_DONE pgs=914 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.667+0000 7fba1d7fa700 1 -- 172.21.15.16:0/1595338182 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fba0800eaa0 con 0x7fba201302f0 2019-12-05T06:48:25.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.922+0000 7fba26a29700 1 -- 172.21.15.16:0/1595338182 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba20129d70 con 0x7fba201302f0 2019-12-05T06:48:25.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.923+0000 7fba24fc6700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=THROTTLE_DONE pgs=914 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:25.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.923+0000 7fba1d7fa700 1 -- 172.21.15.16:0/1595338182 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fba0800efd0 con 0x7fba201302f0 2019-12-05T06:48:25.925 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 163 2019-12-05T06:48:25.926 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:25.926 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:23.502436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:25.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.927+0000 7fba26a29700 1 -- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fba04023630 msgr2=0x7fba04025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:25.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.927+0000 7fba26a29700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fba04023630 0x7fba04025ac0 crc :-1 s=READY pgs=566 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:25.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.927+0000 7fba26a29700 1 -- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 msgr2=0x7fba2012a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:25.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.927+0000 7fba26a29700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 0x7fba2012a670 secure :-1 s=READY pgs=914 cs=0 l=1 rx=0x7fba08018b80 tx=0x7fba08018f20).stop 2019-12-05T06:48:25.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.927+0000 7fba26a29700 1 -- 172.21.15.16:0/1595338182 shutdown_connections 2019-12-05T06:48:25.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.927+0000 7fba26a29700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fba04023630 0x7fba04025ac0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:25.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.927+0000 7fba26a29700 1 --2- 172.21.15.16:0/1595338182 >> v2:172.21.15.16:3300/0 conn(0x7fba2012b640 0x7fba2012abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:25.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.927+0000 7fba26a29700 1 --2- 172.21.15.16:0/1595338182 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fba201302f0 0x7fba2012a670 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:25.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.928+0000 7fba26a29700 1 -- 172.21.15.16:0/1595338182 shutdown_connections 2019-12-05T06:48:25.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.928+0000 7fba26a29700 1 -- 172.21.15.16:0/1595338182 wait complete. 2019-12-05T06:48:25.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:25.928+0000 7fba26a29700 1 -- 172.21.15.16:0/1595338182 >> 172.21.15.16:0/1595338182 conn(0x7fba200a2850 msgr2=0x7fba2006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:25.938 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:48:25.938 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:48:25.938 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:26.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.177+0000 7f61f188c700 1 Processor -- start 2019-12-05T06:48:26.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.177+0000 7f61f188c700 1 -- start start 2019-12-05T06:48:26.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.179+0000 7f61f188c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61ec08c6d0 0x7f61ec08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:26.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.179+0000 7f61f188c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f61ec08fef0 0x7f61ec08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:26.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.179+0000 7f61f188c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ec0958e0 con 0x7f61ec08ef20 2019-12-05T06:48:26.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.179+0000 7f61f188c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61ec08df90 con 0x7f61ec08c6d0 2019-12-05T06:48:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.179+0000 7f61f188c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61ec0a3f70 con 0x7f61ec08fef0 2019-12-05T06:48:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.179+0000 7f61eb7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61ec08c6d0 0x7f61ec08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.179+0000 7f61eb7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f61ec08c6d0 0x7f61ec08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33072/0 (socket says 172.21.15.16:33072) 2019-12-05T06:48:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.179+0000 7f61eb7fe700 1 -- 172.21.15.16:0/2250589294 learned_addr learned my addr 172.21.15.16:0/2250589294 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.179+0000 7f61eaffd700 1 --2- 172.21.15.16:0/2250589294 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f61ec08fef0 0x7f61ec08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:26.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.180+0000 7f61eb7fe700 1 -- 172.21.15.16:0/2250589294 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f61ec08fef0 msgr2=0x7f61ec08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.180+0000 7f61ea7fc700 1 -- 172.21.15.16:0/2250589294 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1003172291 0 0) 0x7f61dc0009b0 con 0x7f61ec08ef20 2019-12-05T06:48:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.180+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/2250589294 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f61ec08fef0 0x7f61ec08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.180+0000 7f61eb7fe700 1 -- 172.21.15.16:0/2250589294 >> v1:172.21.15.76:6789/0 conn(0x7f61ec08ef20 legacy=0x7f61ec08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.180+0000 7f61eb7fe700 1 -- 172.21.15.16:0/2250589294 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec08d540 con 0x7f61ec08c6d0 2019-12-05T06:48:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.180+0000 7f61eaffd700 1 --2- 172.21.15.16:0/2250589294 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f61ec08fef0 0x7f61ec08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:48:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/2250589294 >> v2:172.21.15.16:3300/0 conn(0x7f61ec08c6d0 0x7f61ec08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f61e400fac0 tx=0x7f61e400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/2250589294 >> v2:172.21.15.16:3300/0 conn(0x7f61ec08c6d0 0x7f61ec08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f61e400fac0 tx=0x7f61e400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/2250589294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61ec08c6d0 0x7f61ec08cac0 secure :-1 s=READY pgs=1341 cs=0 l=1 rx=0x7f61e400fac0 tx=0x7f61e400fe60).ready entity=mon.0 client_cookie=94fab7e75c99945 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/2250589294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61ec08c6d0 0x7f61ec08cac0 secure :-1 s=THROTTLE_DONE pgs=1341 cs=0 l=1 rx=0x7f61e400fac0 tx=0x7f61e400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61ea7fc700 1 -- 172.21.15.16:0/2250589294 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61e4000ce0 con 0x7f61ec08c6d0 2019-12-05T06:48:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/2250589294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61ec08c6d0 0x7f61ec08cac0 secure :-1 s=THROTTLE_DONE pgs=1341 cs=0 l=1 rx=0x7f61e400fac0 tx=0x7f61e400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/2250589294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61ec08c6d0 0x7f61ec08cac0 secure :-1 s=THROTTLE_DONE pgs=1341 cs=0 l=1 rx=0x7f61e400fac0 tx=0x7f61e400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61ea7fc700 1 -- 172.21.15.16:0/2250589294 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e4001010 con 0x7f61ec08c6d0 2019-12-05T06:48:26.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61ea7fc700 1 -- 172.21.15.16:0/2250589294 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61e4001720 con 0x7f61ec08c6d0 2019-12-05T06:48:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61f188c700 1 -- 172.21.15.16:0/2250589294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61ec08c6d0 msgr2=0x7f61ec08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.181+0000 7f61f188c700 1 --2- 172.21.15.16:0/2250589294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61ec08c6d0 0x7f61ec08cac0 secure :-1 s=READY pgs=1341 cs=0 l=1 rx=0x7f61e400fac0 tx=0x7f61e400fe60).stop 2019-12-05T06:48:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.183+0000 7f61f188c700 1 -- 172.21.15.16:0/2250589294 shutdown_connections 2019-12-05T06:48:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.183+0000 7f61f188c700 1 --2- 172.21.15.16:0/2250589294 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f61ec08fef0 0x7f61ec08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.183+0000 7f61f188c700 1 --2- 172.21.15.16:0/2250589294 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61ec08c6d0 0x7f61ec08cac0 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.183+0000 7f61f188c700 1 -- 172.21.15.16:0/2250589294 shutdown_connections 2019-12-05T06:48:26.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.183+0000 7f61f188c700 1 -- 172.21.15.16:0/2250589294 wait complete. 2019-12-05T06:48:26.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.183+0000 7f61f188c700 1 -- 172.21.15.16:0/2250589294 >> 172.21.15.16:0/2250589294 conn(0x7f61ec061940 msgr2=0x7f61ec096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.188+0000 7f61f188c700 1 Processor -- start 2019-12-05T06:48:26.195 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.188+0000 7f61f188c700 1 -- start start 2019-12-05T06:48:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.190+0000 7f61f188c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f61ec14d400 0x7f61ec14d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.190+0000 7f61f188c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ec098da0 con 0x7f61ec132d00 2019-12-05T06:48:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.190+0000 7f61f188c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61ec09f0a0 con 0x7f61ec132530 2019-12-05T06:48:26.196 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.190+0000 7f61f188c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61ec09bb50 con 0x7f61ec14d400 2019-12-05T06:48:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.190+0000 7f61eaffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f61ec132d00 0x7f61ec131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46478/0 (socket says 172.21.15.16:46478) 2019-12-05T06:48:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.190+0000 7f61eaffd700 1 -- 172.21.15.16:0/617364039 learned_addr learned my addr 172.21.15.16:0/617364039 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.190+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> v2:172.21.15.76:3300/0 conn(0x7f61ec14d400 0x7f61ec14d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.191+0000 7f61e3fff700 1 -- 172.21.15.16:0/617364039 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1323201919 0 0) 0x7f61d4000fb0 con 0x7f61ec132d00 2019-12-05T06:48:26.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.191+0000 7f61e3fff700 1 -- 172.21.15.16:0/617364039 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61d0002e80 con 0x7f61ec132d00 2019-12-05T06:48:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.191+0000 7f61eb7fe700 1 -- 172.21.15.16:0/617364039 >> v1:172.21.15.76:6789/0 conn(0x7f61ec132530 legacy=0x7f61ec157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.191+0000 7f61eb7fe700 1 -- 172.21.15.16:0/617364039 >> v1:172.21.15.16:6789/0 conn(0x7f61ec132d00 legacy=0x7f61ec131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.191+0000 7f61eb7fe700 1 -- 172.21.15.16:0/617364039 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61ec09b0f0 con 0x7f61ec14d400 2019-12-05T06:48:26.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.191+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> v2:172.21.15.76:3300/0 conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.192+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> v2:172.21.15.76:3300/0 conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.192+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=READY pgs=917 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.192+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.192+0000 7f61e3fff700 1 -- 172.21.15.16:0/617364039 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61e40058e0 con 0x7f61ec14d400 2019-12-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.192+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.192+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.192+0000 7f61e3fff700 1 -- 172.21.15.16:0/617364039 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61e4005500 con 0x7f61ec14d400 2019-12-05T06:48:26.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.192+0000 7f61f188c700 1 -- 172.21.15.16:0/617364039 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61ec09b0f0 con 0x7f61ec14d400 2019-12-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.192+0000 7f61e3fff700 1 -- 172.21.15.16:0/617364039 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f61e4005ec0 con 0x7f61ec14d400 2019-12-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.192+0000 7f61f188c700 1 -- 172.21.15.16:0/617364039 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61ec09b0f0 con 0x7f61ec14d400 2019-12-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.193+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.194+0000 7f61e3fff700 1 -- 172.21.15.16:0/617364039 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f61e40056f0 con 0x7f61ec14d400 2019-12-05T06:48:26.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.194+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.194+0000 7f61e3fff700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61d0023670 0x7f61d0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.194+0000 7f61e3fff700 1 -- 172.21.15.16:0/617364039 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f61e4046160 con 0x7f61ec14d400 2019-12-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.195+0000 7f61eaffd700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61d0023670 0x7f61d0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:26.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.195+0000 7f61eaffd700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61d0023670 0x7f61d0025ae0 crc :-1 s=READY pgs=567 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:26.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.209+0000 7f61f188c700 1 -- 172.21.15.16:0/617364039 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61ec09b0f0 con 0x7f61ec14d400 2019-12-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.217+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.218+0000 7f61e3fff700 1 -- 172.21.15.16:0/617364039 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f61e4007690 con 0x7f61ec14d400 2019-12-05T06:48:26.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.479+0000 7f61f188c700 1 -- 172.21.15.16:0/617364039 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f61ec09b0f0 con 0x7f61ec14d400 2019-12-05T06:48:26.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.480+0000 7f61eb7fe700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=THROTTLE_DONE pgs=917 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:26.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.481+0000 7f61e3fff700 1 -- 172.21.15.16:0/617364039 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 163 v163) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f61e40056f0 con 0x7f61ec14d400 2019-12-05T06:48:26.483 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 163 2019-12-05T06:48:26.483 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:26.484 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":163,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":163,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:23.502436+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:26.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.485+0000 7f61f188c700 1 -- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61d0023670 msgr2=0x7f61d0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.485+0000 7f61f188c700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61d0023670 0x7f61d0025ae0 crc :-1 s=READY pgs=567 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.485+0000 7f61f188c700 1 -- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 msgr2=0x7f61ec14d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:26.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.485+0000 7f61f188c700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 0x7f61ec14d7f0 secure :-1 s=READY pgs=917 cs=0 l=1 rx=0x7f61e4010720 tx=0x7f61e4010ac0).stop 2019-12-05T06:48:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.485+0000 7f61f188c700 1 -- 172.21.15.16:0/617364039 shutdown_connections 2019-12-05T06:48:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.485+0000 7f61f188c700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61d0023670 0x7f61d0025ae0 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.485+0000 7f61f188c700 1 --2- 172.21.15.16:0/617364039 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61ec14d400 0x7f61ec14d7f0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.486+0000 7f61f188c700 1 -- 172.21.15.16:0/617364039 shutdown_connections 2019-12-05T06:48:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.486+0000 7f61f188c700 1 -- 172.21.15.16:0/617364039 wait complete. 2019-12-05T06:48:26.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:26.486+0000 7f61f188c700 1 -- 172.21.15.16:0/617364039 >> 172.21.15.16:0/617364039 conn(0x7f61ec0714f0 msgr2=0x7f61ec09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:26.522 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 163, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 163, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7546": { "addr": "172.21.15.76:6835/3562825781", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 3562825781, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 3562825781, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7546, "incarnation": 163, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 76, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:48:23.502436+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7546 } } } ], "standbys": [] } 2019-12-05T06:48:28.524 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.759+0000 7f38d56c5700 1 Processor -- start 2019-12-05T06:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.759+0000 7f38d56c5700 1 -- start start 2019-12-05T06:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.760+0000 7f38d56c5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f38d008d2c0 0x7f38d008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:28.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.760+0000 7f38d56c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38d008fef0 0x7f38d008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.760+0000 7f38d56c5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d00958e0 con 0x7f38d008c6d0 2019-12-05T06:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.760+0000 7f38d56c5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d008df90 con 0x7f38d008fef0 2019-12-05T06:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.760+0000 7f38d56c5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d00a3f70 con 0x7f38d008d2c0 2019-12-05T06:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.760+0000 7f38cffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38d008fef0 0x7f38d008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.760+0000 7f38cf7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38d008c6d0 0x7f38d008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46488/0 (socket says 172.21.15.16:46488) 2019-12-05T06:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.760+0000 7f38cf7fe700 1 -- 172.21.15.16:0/525058137 learned_addr learned my addr 172.21.15.16:0/525058137 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:28.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38ceffd700 1 --2- 172.21.15.16:0/525058137 >> v2:172.21.15.76:3300/0 conn(0x7f38d008d2c0 0x7f38d008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38cffff700 1 -- 172.21.15.16:0/525058137 >> v2:172.21.15.76:3300/0 conn(0x7f38d008d2c0 msgr2=0x7f38d008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38cffff700 1 --2- 172.21.15.16:0/525058137 >> v2:172.21.15.76:3300/0 conn(0x7f38d008d2c0 0x7f38d008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38cffff700 1 -- 172.21.15.16:0/525058137 >> v1:172.21.15.16:6789/0 conn(0x7f38d008c6d0 legacy=0x7f38d008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38cffff700 1 -- 172.21.15.16:0/525058137 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d008e620 con 0x7f38d008fef0 2019-12-05T06:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38ce7fc700 1 -- 172.21.15.16:0/525058137 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 63043413 0 0) 0x7f38c0001150 con 0x7f38d008c6d0 2019-12-05T06:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38cffff700 1 --2- 172.21.15.16:0/525058137 >> v2:172.21.15.16:3300/0 conn(0x7f38d008fef0 0x7f38d008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f38c800af50 tx=0x7f38c800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38cffff700 1 --2- 172.21.15.16:0/525058137 >> v2:172.21.15.16:3300/0 conn(0x7f38d008fef0 0x7f38d008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f38c800af50 tx=0x7f38c800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38cffff700 1 --2- 172.21.15.16:0/525058137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d008fef0 0x7f38d008f460 secure :-1 s=READY pgs=1346 cs=0 l=1 rx=0x7f38c800af50 tx=0x7f38c800b6e0).ready entity=mon.0 client_cookie=6351e91f9ffa2fba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38cffff700 1 --2- 172.21.15.16:0/525058137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d008fef0 0x7f38d008f460 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7f38c800af50 tx=0x7f38c800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38ce7fc700 1 -- 172.21.15.16:0/525058137 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38c8009ba0 con 0x7f38d008fef0 2019-12-05T06:48:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38cffff700 1 --2- 172.21.15.16:0/525058137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d008fef0 0x7f38d008f460 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7f38c800af50 tx=0x7f38c800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38ce7fc700 1 -- 172.21.15.16:0/525058137 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c80097c0 con 0x7f38d008fef0 2019-12-05T06:48:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38cffff700 1 --2- 172.21.15.16:0/525058137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d008fef0 0x7f38d008f460 secure :-1 s=THROTTLE_DONE pgs=1346 cs=0 l=1 rx=0x7f38c800af50 tx=0x7f38c800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38ce7fc700 1 -- 172.21.15.16:0/525058137 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38c8009d90 con 0x7f38d008fef0 2019-12-05T06:48:28.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38d56c5700 1 -- 172.21.15.16:0/525058137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d008fef0 msgr2=0x7f38d008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:28.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.761+0000 7f38d56c5700 1 --2- 172.21.15.16:0/525058137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d008fef0 0x7f38d008f460 secure :-1 s=READY pgs=1346 cs=0 l=1 rx=0x7f38c800af50 tx=0x7f38c800b6e0).stop 2019-12-05T06:48:28.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.762+0000 7f38d56c5700 1 -- 172.21.15.16:0/525058137 shutdown_connections 2019-12-05T06:48:28.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.762+0000 7f38d56c5700 1 --2- 172.21.15.16:0/525058137 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38d008fef0 0x7f38d008f460 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:28.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.762+0000 7f38d56c5700 1 --2- 172.21.15.16:0/525058137 >> v2:172.21.15.76:3300/0 conn(0x7f38d008d2c0 0x7f38d008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:28.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.762+0000 7f38d56c5700 1 -- 172.21.15.16:0/525058137 shutdown_connections 2019-12-05T06:48:28.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.762+0000 7f38d56c5700 1 -- 172.21.15.16:0/525058137 wait complete. 2019-12-05T06:48:28.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.762+0000 7f38d56c5700 1 -- 172.21.15.16:0/525058137 >> 172.21.15.16:0/525058137 conn(0x7f38d0061940 msgr2=0x7f38d0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:28.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.765+0000 7f38d56c5700 1 Processor -- start 2019-12-05T06:48:28.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.765+0000 7f38d56c5700 1 -- start start 2019-12-05T06:48:28.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.766+0000 7f38d56c5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f38d0084f30 0x7f38d0157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:28.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.766+0000 7f38d56c5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d0084b90 con 0x7f38d0096e90 2019-12-05T06:48:28.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.766+0000 7f38d56c5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d0098790 con 0x7f38d009bbe0 2019-12-05T06:48:28.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.766+0000 7f38d56c5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38d009a450 con 0x7f38d0084f30 2019-12-05T06:48:28.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.766+0000 7f38cf7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38d0096e90 0x7f38d009b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46494/0 (socket says 172.21.15.16:46494) 2019-12-05T06:48:28.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.766+0000 7f38cf7fe700 1 -- 172.21.15.16:0/1003627255 learned_addr learned my addr 172.21.15.16:0/1003627255 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:28.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> v2:172.21.15.76:3300/0 conn(0x7f38d0084f30 0x7f38d0157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:28.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836800531 0 0) 0x7f38c0001150 con 0x7f38d0096e90 2019-12-05T06:48:28.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b4002e80 con 0x7f38d0096e90 2019-12-05T06:48:28.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3669222596 0 0) 0x7f38c0001300 con 0x7f38d0096e90 2019-12-05T06:48:28.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38b40032e0 con 0x7f38d0096e90 2019-12-05T06:48:28.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f38c00018b0 con 0x7f38d0096e90 2019-12-05T06:48:28.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 >> v2:172.21.15.76:3300/0 conn(0x7f38d0084f30 msgr2=0x7f38d0157200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:28.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 --2- 172.21.15.16:0/1003627255 >> v2:172.21.15.76:3300/0 conn(0x7f38d0084f30 0x7f38d0157200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:28.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 >> v1:172.21.15.76:6789/0 conn(0x7f38d009bbe0 legacy=0x7f38d01539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:28.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 >> v1:172.21.15.16:6789/0 conn(0x7f38d0096e90 legacy=0x7f38d009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:28.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38b40033b0 0x7f38b4003800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:28.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f38b4003d40 0x7f38b40041c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:28.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:28.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b400bbf0 con 0x7f38b4004950 2019-12-05T06:48:28.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b400be20 con 0x7f38b4003d40 2019-12-05T06:48:28.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38b400c050 con 0x7f38b40033b0 2019-12-05T06:48:28.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.767+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:28.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38ceffd700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f38b4003d40 0x7f38b40041c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:28.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cf7fe700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38b40033b0 0x7f38b4003800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:28.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 -- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38b40033b0 msgr2=0x7f38b4003800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:28.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38b40033b0 0x7f38b4003800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:28.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 -- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f38b4003d40 msgr2=0x7f38b40041c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:28.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f38b4003d40 0x7f38b40041c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:28.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 -- 172.21.15.16:0/1003627255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38b400c300 con 0x7f38b4004950 2019-12-05T06:48:28.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38d56c5700 1 -- 172.21.15.16:0/1003627255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d009aae0 con 0x7f38b4004950 2019-12-05T06:48:28.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cf7fe700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38b40033b0 0x7f38b4003800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:48:28.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38d56c5700 1 -- 172.21.15.16:0/1003627255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d0084b90 con 0x7f38b4004950 2019-12-05T06:48:28.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 -- 172.21.15.16:0/1003627255 reap_dead start 2019-12-05T06:48:28.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=READY pgs=1348 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:28.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=THROTTLE_DONE pgs=1348 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38c8009960 con 0x7f38b4004950 2019-12-05T06:48:28.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=THROTTLE_DONE pgs=1348 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c8009580 con 0x7f38b4004950 2019-12-05T06:48:28.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=THROTTLE_DONE pgs=1348 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.768+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38c8009770 con 0x7f38b4004950 2019-12-05T06:48:28.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.769+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=THROTTLE_DONE pgs=1348 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.769+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f38c8009770 con 0x7f38b4004950 2019-12-05T06:48:28.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.769+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=THROTTLE_DONE pgs=1348 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.769+0000 7f38c7fff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38b4026340 0x7f38b40287b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:28.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.769+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f38c8045c40 con 0x7f38b4004950 2019-12-05T06:48:28.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.770+0000 7f38cf7fe700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38b4026340 0x7f38b40287b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:28.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.770+0000 7f38cf7fe700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38b4026340 0x7f38b40287b0 crc :-1 s=READY pgs=568 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:28.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.774+0000 7f38d56c5700 1 -- 172.21.15.16:0/1003627255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38a8002e60 con 0x7f38b4004950 2019-12-05T06:48:28.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.779+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=THROTTLE_DONE pgs=1348 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.779+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f38c8000940 con 0x7f38b4004950 2019-12-05T06:48:28.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.979+0000 7f38d56c5700 1 -- 172.21.15.16:0/1003627255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38a8001d10 con 0x7f38b4004950 2019-12-05T06:48:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.979+0000 7f38cffff700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=THROTTLE_DONE pgs=1348 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:28.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.979+0000 7f38c7fff700 1 -- 172.21.15.16:0/1003627255 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f38c8009770 con 0x7f38b4004950 2019-12-05T06:48:28.980 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 166 2019-12-05T06:48:28.980 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:28.980 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.982+0000 7f38d56c5700 1 -- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38b4026340 msgr2=0x7f38b40287b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.982+0000 7f38d56c5700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38b4026340 0x7f38b40287b0 crc :-1 s=READY pgs=568 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.982+0000 7f38d56c5700 1 -- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 msgr2=0x7f38b4004e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.982+0000 7f38d56c5700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 secure :-1 s=READY pgs=1348 cs=0 l=1 rx=0x7f38c8010620 tx=0x7f38c8010520).stop 2019-12-05T06:48:28.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.982+0000 7f38d56c5700 1 -- 172.21.15.16:0/1003627255 shutdown_connections 2019-12-05T06:48:28.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.982+0000 7f38d56c5700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38b4026340 0x7f38b40287b0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:28.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.982+0000 7f38d56c5700 1 --2- 172.21.15.16:0/1003627255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38b4004950 0x7f38b4004e10 unknown :-1 s=CLOSED pgs=1348 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:28.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.982+0000 7f38d56c5700 1 -- 172.21.15.16:0/1003627255 shutdown_connections 2019-12-05T06:48:28.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.982+0000 7f38d56c5700 1 -- 172.21.15.16:0/1003627255 wait complete. 2019-12-05T06:48:28.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:28.982+0000 7f38d56c5700 1 -- 172.21.15.16:0/1003627255 >> 172.21.15.16:0/1003627255 conn(0x7f38d00714f0 msgr2=0x7f38d009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:28.997 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=7401 2019-12-05T06:48:28.997 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.a 2019-12-05T06:48:32.002 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:48:32.002 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:48:32.003 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:48:32.007 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:48:32.007 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:32.112 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:48:32.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.153+0000 7fc877daf700 1 Processor -- start 2019-12-05T06:48:32.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.153+0000 7fc877daf700 1 -- start start 2019-12-05T06:48:32.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.154+0000 7fc877daf700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc87008c6d0 0x7fc87008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:32.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.154+0000 7fc877daf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc87008fef0 0x7fc87008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:32.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.154+0000 7fc877daf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc87008ef20 0x7fc87008f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:32.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc877daf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8700a3f70 con 0x7fc87008ef20 2019-12-05T06:48:32.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc877daf700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc87008d830 con 0x7fc87008fef0 2019-12-05T06:48:32.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc877daf700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87008d540 con 0x7fc87008c6d0 2019-12-05T06:48:32.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc87008ef20 0x7fc87008f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:32.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc87008ef20 0x7fc87008f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33126/0 (socket says 172.21.15.16:33126) 2019-12-05T06:48:32.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 -- 172.21.15.16:0/1765716378 learned_addr learned my addr 172.21.15.16:0/1765716378 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:32.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc875b4b700 1 --2- 172.21.15.16:0/1765716378 >> v2:172.21.15.76:3300/0 conn(0x7fc87008fef0 0x7fc87008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:32.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc87634c700 1 --2- 172.21.15.16:0/1765716378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc87008c6d0 0x7fc87008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:32.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 -- 172.21.15.16:0/1765716378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc87008c6d0 msgr2=0x7fc87008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:32.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 --2- 172.21.15.16:0/1765716378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc87008c6d0 0x7fc87008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:32.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 -- 172.21.15.16:0/1765716378 >> v2:172.21.15.76:3300/0 conn(0x7fc87008fef0 msgr2=0x7fc87008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:32.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 --2- 172.21.15.16:0/1765716378 >> v2:172.21.15.76:3300/0 conn(0x7fc87008fef0 0x7fc87008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:32.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 -- 172.21.15.16:0/1765716378 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87008e620 con 0x7fc87008ef20 2019-12-05T06:48:32.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc875b4b700 1 --2- 172.21.15.16:0/1765716378 >> v2:172.21.15.76:3300/0 conn(0x7fc87008fef0 0x7fc87008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:48:32.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 --2- 172.21.15.16:0/1765716378 >> v2:172.21.15.16:3300/0 conn(0x7fc87008ef20 0x7fc87008f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc86800fa30 tx=0x7fc86800fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 --2- 172.21.15.16:0/1765716378 >> v2:172.21.15.16:3300/0 conn(0x7fc87008ef20 0x7fc87008f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc86800fa30 tx=0x7fc86800fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 --2- 172.21.15.16:0/1765716378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87008ef20 0x7fc87008f3c0 secure :-1 s=READY pgs=1354 cs=0 l=1 rx=0x7fc86800fa30 tx=0x7fc86800fdd0).ready entity=mon.0 client_cookie=f95e2a4844df2462 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:32.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 --2- 172.21.15.16:0/1765716378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87008ef20 0x7fc87008f3c0 secure :-1 s=THROTTLE_DONE pgs=1354 cs=0 l=1 rx=0x7fc86800fa30 tx=0x7fc86800fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 --2- 172.21.15.16:0/1765716378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87008ef20 0x7fc87008f3c0 secure :-1 s=THROTTLE_DONE pgs=1354 cs=0 l=1 rx=0x7fc86800fa30 tx=0x7fc86800fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc876b4d700 1 --2- 172.21.15.16:0/1765716378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87008ef20 0x7fc87008f3c0 secure :-1 s=THROTTLE_DONE pgs=1354 cs=0 l=1 rx=0x7fc86800fa30 tx=0x7fc86800fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.155+0000 7fc87534a700 1 -- 172.21.15.16:0/1765716378 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc868001460 con 0x7fc87008ef20 2019-12-05T06:48:32.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.156+0000 7fc87534a700 1 -- 172.21.15.16:0/1765716378 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc868000f10 con 0x7fc87008ef20 2019-12-05T06:48:32.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.156+0000 7fc87534a700 1 -- 172.21.15.16:0/1765716378 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc868001bc0 con 0x7fc87008ef20 2019-12-05T06:48:32.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.156+0000 7fc877daf700 1 -- 172.21.15.16:0/1765716378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87008ef20 msgr2=0x7fc87008f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:32.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.156+0000 7fc877daf700 1 --2- 172.21.15.16:0/1765716378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87008ef20 0x7fc87008f3c0 secure :-1 s=READY pgs=1354 cs=0 l=1 rx=0x7fc86800fa30 tx=0x7fc86800fdd0).stop 2019-12-05T06:48:32.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.156+0000 7fc877daf700 1 -- 172.21.15.16:0/1765716378 shutdown_connections 2019-12-05T06:48:32.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.156+0000 7fc877daf700 1 --2- 172.21.15.16:0/1765716378 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87008ef20 0x7fc87008f3c0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:32.163 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.156+0000 7fc877daf700 1 --2- 172.21.15.16:0/1765716378 >> v2:172.21.15.76:3300/0 conn(0x7fc87008fef0 0x7fc87008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:32.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.156+0000 7fc877daf700 1 --2- 172.21.15.16:0/1765716378 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc87008c6d0 0x7fc87008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:32.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.157+0000 7fc877daf700 1 -- 172.21.15.16:0/1765716378 shutdown_connections 2019-12-05T06:48:32.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.157+0000 7fc877daf700 1 -- 172.21.15.16:0/1765716378 wait complete. 2019-12-05T06:48:32.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.157+0000 7fc877daf700 1 -- 172.21.15.16:0/1765716378 >> 172.21.15.16:0/1765716378 conn(0x7fc870061940 msgr2=0x7fc870096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:32.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.159+0000 7fc877daf700 1 Processor -- start 2019-12-05T06:48:32.164 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.159+0000 7fc877daf700 1 -- start start 2019-12-05T06:48:32.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.160+0000 7fc877daf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc870084e90 0x7fc870088790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:32.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.160+0000 7fc877daf700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc870089760 0x7fc870088cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:32.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.160+0000 7fc877daf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc870085ae0 con 0x7fc870089210 2019-12-05T06:48:32.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.160+0000 7fc877daf700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc870087800 con 0x7fc870084e90 2019-12-05T06:48:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.160+0000 7fc877daf700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc870086dd0 con 0x7fc870089760 2019-12-05T06:48:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.160+0000 7fc875b4b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc870089760 0x7fc870088cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.160+0000 7fc87634c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc870084e90 0x7fc870088790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.160+0000 7fc87634c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc870084e90 0x7fc870088790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:47452/0 (socket says 172.21.15.16:47452) 2019-12-05T06:48:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.160+0000 7fc875b4b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc870089760 0x7fc870088cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:37122/0 (socket says 172.21.15.16:37122) 2019-12-05T06:48:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.160+0000 7fc87634c700 1 -- 172.21.15.16:0/2464632319 learned_addr learned my addr 172.21.15.16:0/2464632319 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:32.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc87634c700 1 -- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc870089760 msgr2=0x7fc870088cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc870089760 0x7fc870088cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc87634c700 1 -- 172.21.15.16:0/2464632319 >> v1:172.21.15.76:6789/0 conn(0x7fc870089210 legacy=0x7fc870098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc87634c700 1 -- 172.21.15.16:0/2464632319 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc870087e90 con 0x7fc870084e90 2019-12-05T06:48:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> v2:172.21.15.76:3300/0 conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> v2:172.21.15.76:3300/0 conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=READY pgs=920 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:32.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc86effd700 1 -- 172.21.15.16:0/2464632319 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc860008890 con 0x7fc870084e90 2019-12-05T06:48:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc877daf700 1 -- 172.21.15.16:0/2464632319 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc870087e90 con 0x7fc870084e90 2019-12-05T06:48:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc86effd700 1 -- 172.21.15.16:0/2464632319 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8600084b0 con 0x7fc870084e90 2019-12-05T06:48:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc86effd700 1 -- 172.21.15.16:0/2464632319 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc860008b80 con 0x7fc870084e90 2019-12-05T06:48:32.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.161+0000 7fc877daf700 1 -- 172.21.15.16:0/2464632319 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc870087e90 con 0x7fc870084e90 2019-12-05T06:48:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.162+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.163+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.163+0000 7fc86effd700 1 -- 172.21.15.16:0/2464632319 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc8600084b0 con 0x7fc870084e90 2019-12-05T06:48:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.163+0000 7fc86effd700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc850023630 0x7fc850025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.163+0000 7fc86effd700 1 -- 172.21.15.16:0/2464632319 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7fc860045e10 con 0x7fc870084e90 2019-12-05T06:48:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.163+0000 7fc875b4b700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc850023630 0x7fc850025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:32.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.164+0000 7fc875b4b700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc850023630 0x7fc850025ac0 crc :-1 s=READY pgs=569 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:32.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.169+0000 7fc877daf700 1 -- 172.21.15.16:0/2464632319 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc870087e90 con 0x7fc870084e90 2019-12-05T06:48:32.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.178+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.178+0000 7fc86effd700 1 -- 172.21.15.16:0/2464632319 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc8600029d0 con 0x7fc870084e90 2019-12-05T06:48:32.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.375+0000 7fc877daf700 1 -- 172.21.15.16:0/2464632319 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc870087e90 con 0x7fc870084e90 2019-12-05T06:48:32.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.376+0000 7fc87634c700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=THROTTLE_DONE pgs=920 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:32.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.376+0000 7fc86effd700 1 -- 172.21.15.16:0/2464632319 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fc8600084b0 con 0x7fc870084e90 2019-12-05T06:48:32.378 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:32.379 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:32.379 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 166 2019-12-05T06:48:32.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 -- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc850023630 msgr2=0x7fc850025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:32.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc850023630 0x7fc850025ac0 crc :-1 s=READY pgs=569 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:32.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 -- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 msgr2=0x7fc870088790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:32.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 0x7fc870088790 secure :-1 s=READY pgs=920 cs=0 l=1 rx=0x7fc860010440 tx=0x7fc860010270).stop 2019-12-05T06:48:32.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 -- 172.21.15.16:0/2464632319 shutdown_connections 2019-12-05T06:48:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc850023630 0x7fc850025ac0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc870089760 0x7fc870088cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 --2- 172.21.15.16:0/2464632319 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc870084e90 0x7fc870088790 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 -- 172.21.15.16:0/2464632319 shutdown_connections 2019-12-05T06:48:32.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 -- 172.21.15.16:0/2464632319 wait complete. 2019-12-05T06:48:32.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:32.381+0000 7fc877daf700 1 -- 172.21.15.16:0/2464632319 >> 172.21.15.16:0/2464632319 conn(0x7fc87006ecf0 msgr2=0x7fc8700706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:32.397 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:48:34.398 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:34.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.536+0000 7f674fe52700 1 Processor -- start 2019-12-05T06:48:34.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.536+0000 7f674fe52700 1 -- start start 2019-12-05T06:48:34.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.537+0000 7f674fe52700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f674808c6d0 0x7f674808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.537+0000 7f674fe52700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f674808fef0 0x7f674808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.537+0000 7f674fe52700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67480958e0 con 0x7f674808ef20 2019-12-05T06:48:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.537+0000 7f674fe52700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67480a3f70 con 0x7f674808fef0 2019-12-05T06:48:34.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.537+0000 7f674fe52700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674808d830 con 0x7f674808c6d0 2019-12-05T06:48:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.537+0000 7f674dbee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f674808fef0 0x7f674808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.537+0000 7f674dbee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f674808fef0 0x7f674808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33154/0 (socket says 172.21.15.16:33154) 2019-12-05T06:48:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.537+0000 7f674dbee700 1 -- 172.21.15.16:0/2491021693 learned_addr learned my addr 172.21.15.16:0/2491021693 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:34.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674e3ef700 1 --2- 172.21.15.16:0/2491021693 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f674808c6d0 0x7f674808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674dbee700 1 -- 172.21.15.16:0/2491021693 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f674808c6d0 msgr2=0x7f674808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674dbee700 1 --2- 172.21.15.16:0/2491021693 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f674808c6d0 0x7f674808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674dbee700 1 -- 172.21.15.16:0/2491021693 >> v1:172.21.15.76:6789/0 conn(0x7f674808ef20 legacy=0x7f674808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:34.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674dbee700 1 -- 172.21.15.16:0/2491021693 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674808d540 con 0x7f674808fef0 2019-12-05T06:48:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674dbee700 1 --2- 172.21.15.16:0/2491021693 >> v2:172.21.15.16:3300/0 conn(0x7f674808fef0 0x7f674808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f674400fa40 tx=0x7f674400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674dbee700 1 --2- 172.21.15.16:0/2491021693 >> v2:172.21.15.16:3300/0 conn(0x7f674808fef0 0x7f674808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f674400fa40 tx=0x7f674400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674dbee700 1 --2- 172.21.15.16:0/2491021693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674808fef0 0x7f674808d000 secure :-1 s=READY pgs=1355 cs=0 l=1 rx=0x7f674400fa40 tx=0x7f674400fde0).ready entity=mon.0 client_cookie=a206bdb149071b8d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:34.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674dbee700 1 --2- 172.21.15.16:0/2491021693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674808fef0 0x7f674808d000 secure :-1 s=THROTTLE_DONE pgs=1355 cs=0 l=1 rx=0x7f674400fa40 tx=0x7f674400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674d3ed700 1 -- 172.21.15.16:0/2491021693 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6744000ce0 con 0x7f674808fef0 2019-12-05T06:48:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674dbee700 1 --2- 172.21.15.16:0/2491021693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674808fef0 0x7f674808d000 secure :-1 s=THROTTLE_DONE pgs=1355 cs=0 l=1 rx=0x7f674400fa40 tx=0x7f674400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674dbee700 1 --2- 172.21.15.16:0/2491021693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674808fef0 0x7f674808d000 secure :-1 s=THROTTLE_DONE pgs=1355 cs=0 l=1 rx=0x7f674400fa40 tx=0x7f674400fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674d3ed700 1 -- 172.21.15.16:0/2491021693 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6744001010 con 0x7f674808fef0 2019-12-05T06:48:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674d3ed700 1 -- 172.21.15.16:0/2491021693 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6744000ce0 con 0x7f674808fef0 2019-12-05T06:48:34.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674fe52700 1 -- 172.21.15.16:0/2491021693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674808fef0 msgr2=0x7f674808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:34.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.538+0000 7f674fe52700 1 --2- 172.21.15.16:0/2491021693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674808fef0 0x7f674808d000 secure :-1 s=READY pgs=1355 cs=0 l=1 rx=0x7f674400fa40 tx=0x7f674400fde0).stop 2019-12-05T06:48:34.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.539+0000 7f674fe52700 1 -- 172.21.15.16:0/2491021693 shutdown_connections 2019-12-05T06:48:34.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.539+0000 7f674fe52700 1 --2- 172.21.15.16:0/2491021693 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f674808fef0 0x7f674808d000 unknown :-1 s=CLOSED pgs=1355 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:34.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.539+0000 7f674fe52700 1 --2- 172.21.15.16:0/2491021693 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f674808c6d0 0x7f674808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:34.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.539+0000 7f674fe52700 1 -- 172.21.15.16:0/2491021693 shutdown_connections 2019-12-05T06:48:34.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.539+0000 7f674fe52700 1 -- 172.21.15.16:0/2491021693 wait complete. 2019-12-05T06:48:34.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.539+0000 7f674fe52700 1 -- 172.21.15.16:0/2491021693 >> 172.21.15.16:0/2491021693 conn(0x7f6748061940 msgr2=0x7f6748096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.542+0000 7f674fe52700 1 Processor -- start 2019-12-05T06:48:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.542+0000 7f674fe52700 1 -- start start 2019-12-05T06:48:34.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.542+0000 7f674fe52700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6748132630 0x7f6748132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:34.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674fe52700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f674809ae80 0x7f6748132f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:34.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674fe52700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674809ab10 con 0x7f6748099eb0 2019-12-05T06:48:34.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674fe52700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6748098f20 con 0x7f6748132630 2019-12-05T06:48:34.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674fe52700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6748134390 con 0x7f674809ae80 2019-12-05T06:48:34.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6748132630 0x7f6748132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:34.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6748132630 0x7f6748132a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33158/0 (socket says 172.21.15.16:33158) 2019-12-05T06:48:34.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 -- 172.21.15.16:0/1336644515 learned_addr learned my addr 172.21.15.16:0/1336644515 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:34.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674dbee700 1 --2- 172.21.15.16:0/1336644515 >> v2:172.21.15.76:3300/0 conn(0x7f674809ae80 0x7f6748132f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:34.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 -- 172.21.15.16:0/1336644515 >> v2:172.21.15.76:3300/0 conn(0x7f674809ae80 msgr2=0x7f6748132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:34.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> v2:172.21.15.76:3300/0 conn(0x7f674809ae80 0x7f6748132f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:34.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 -- 172.21.15.16:0/1336644515 >> v1:172.21.15.16:6789/0 conn(0x7f6748099eb0 legacy=0x7f674809a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:34.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 -- 172.21.15.16:0/1336644515 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67480995b0 con 0x7f6748132630 2019-12-05T06:48:34.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674dbee700 1 --2- 172.21.15.16:0/1336644515 >> v2:172.21.15.76:3300/0 conn(0x7f674809ae80 0x7f6748132f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:48:34.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f673effd700 1 -- 172.21.15.16:0/1336644515 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955208713 0 0) 0x7f6740001200 con 0x7f6748099eb0 2019-12-05T06:48:34.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> v2:172.21.15.16:3300/0 conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> v2:172.21.15.16:3300/0 conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=READY pgs=1357 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:34.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=THROTTLE_DONE pgs=1357 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f673effd700 1 -- 172.21.15.16:0/1336644515 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6738005180 con 0x7f6748132630 2019-12-05T06:48:34.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=THROTTLE_DONE pgs=1357 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=THROTTLE_DONE pgs=1357 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.543+0000 7f673effd700 1 -- 172.21.15.16:0/1336644515 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6738005880 con 0x7f6748132630 2019-12-05T06:48:34.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.544+0000 7f673effd700 1 -- 172.21.15.16:0/1336644515 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6738005a70 con 0x7f6748132630 2019-12-05T06:48:34.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.544+0000 7f674fe52700 1 -- 172.21.15.16:0/1336644515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67480995b0 con 0x7f6748132630 2019-12-05T06:48:34.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.544+0000 7f674fe52700 1 -- 172.21.15.16:0/1336644515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67480995b0 con 0x7f6748132630 2019-12-05T06:48:34.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.544+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=THROTTLE_DONE pgs=1357 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.545+0000 7f673effd700 1 -- 172.21.15.16:0/1336644515 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6738005220 con 0x7f6748132630 2019-12-05T06:48:34.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.545+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=THROTTLE_DONE pgs=1357 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.545+0000 7f673effd700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6730023630 0x7f6730025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:34.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.545+0000 7f673effd700 1 -- 172.21.15.16:0/1336644515 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f6738045e50 con 0x7f6748132630 2019-12-05T06:48:34.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.545+0000 7f674dbee700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6730023630 0x7f6730025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:34.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.546+0000 7f674dbee700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6730023630 0x7f6730025ac0 crc :-1 s=READY pgs=570 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:34.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.552+0000 7f674fe52700 1 -- 172.21.15.16:0/1336644515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67480995b0 con 0x7f6748132630 2019-12-05T06:48:34.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.556+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=THROTTLE_DONE pgs=1357 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.556+0000 7f673effd700 1 -- 172.21.15.16:0/1336644515 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6738012180 con 0x7f6748132630 2019-12-05T06:48:34.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.752+0000 7f674fe52700 1 -- 172.21.15.16:0/1336644515 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f67480995b0 con 0x7f6748132630 2019-12-05T06:48:34.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.753+0000 7f674e3ef700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=THROTTLE_DONE pgs=1357 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:34.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.753+0000 7f673effd700 1 -- 172.21.15.16:0/1336644515 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 166 v166) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f6738012180 con 0x7f6748132630 2019-12-05T06:48:34.758 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 166 2019-12-05T06:48:34.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 -- 172.21.15.16:0/1336644515 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6730023630 msgr2=0x7f6730025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:34.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6730023630 0x7f6730025ac0 crc :-1 s=READY pgs=570 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:34.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 -- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 msgr2=0x7f6748132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:34.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 0x7f6748132a20 secure :-1 s=READY pgs=1357 cs=0 l=1 rx=0x7f6738010470 tx=0x7f6738010810).stop 2019-12-05T06:48:34.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 -- 172.21.15.16:0/1336644515 shutdown_connections 2019-12-05T06:48:34.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 --2- 172.21.15.16:0/1336644515 >> v2:172.21.15.76:3300/0 conn(0x7f674809ae80 0x7f6748132f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:34.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6730023630 0x7f6730025ac0 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:34.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 --2- 172.21.15.16:0/1336644515 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6748132630 0x7f6748132a20 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:34.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 -- 172.21.15.16:0/1336644515 shutdown_connections 2019-12-05T06:48:34.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 -- 172.21.15.16:0/1336644515 wait complete. 2019-12-05T06:48:34.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:34.756+0000 7f674fe52700 1 -- 172.21.15.16:0/1336644515 >> 172.21.15.16:0/1336644515 conn(0x7f67480714f0 msgr2=0x7f674809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:34.761 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:34.761 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":166,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:34.771 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:48:36.772 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:48:37.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.033+0000 7f37b0bd3700 1 Processor -- start 2019-12-05T06:48:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.033+0000 7f37b0bd3700 1 -- start start 2019-12-05T06:48:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37b0bd3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37ac08d2c0 0x7f37ac08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37b0bd3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f37ac08fef0 0x7f37ac08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:48:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37b0bd3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac0958e0 con 0x7f37ac08c6d0 2019-12-05T06:48:37.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37b0bd3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ac08df90 con 0x7f37ac08d2c0 2019-12-05T06:48:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37b0bd3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac0a3f70 con 0x7f37ac08fef0 2019-12-05T06:48:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37aa59c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37ac08d2c0 0x7f37ac08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37aad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f37ac08c6d0 0x7f37ac08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46578/0 (socket says 172.21.15.16:46578) 2019-12-05T06:48:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37aad9d700 1 -- 172.21.15.16:0/88366519 learned_addr learned my addr 172.21.15.16:0/88366519 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:48:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37ab59e700 1 --2- 172.21.15.16:0/88366519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f37ac08fef0 0x7f37ac08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:37.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37a9d9b700 1 -- 172.21.15.16:0/88366519 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2386641242 0 0) 0x7f37a40014d0 con 0x7f37ac08c6d0 2019-12-05T06:48:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.035+0000 7f37a9d9b700 1 -- 172.21.15.16:0/88366519 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3798002e80 con 0x7f37ac08c6d0 2019-12-05T06:48:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37aa59c700 1 -- 172.21.15.16:0/88366519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f37ac08fef0 msgr2=0x7f37ac08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:48:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37aa59c700 1 --2- 172.21.15.16:0/88366519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f37ac08fef0 0x7f37ac08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37aa59c700 1 -- 172.21.15.16:0/88366519 >> v1:172.21.15.16:6789/0 conn(0x7f37ac08c6d0 legacy=0x7f37ac08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37aa59c700 1 -- 172.21.15.16:0/88366519 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac08e620 con 0x7f37ac08d2c0 2019-12-05T06:48:37.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37aa59c700 1 --2- 172.21.15.16:0/88366519 >> v2:172.21.15.16:3300/0 conn(0x7f37ac08d2c0 0x7f37ac08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f379400af50 tx=0x7f379400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37aa59c700 1 --2- 172.21.15.16:0/88366519 >> v2:172.21.15.16:3300/0 conn(0x7f37ac08d2c0 0x7f37ac08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f379400af50 tx=0x7f379400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37aa59c700 1 --2- 172.21.15.16:0/88366519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac08d2c0 0x7f37ac08ef20 secure :-1 s=READY pgs=1359 cs=0 l=1 rx=0x7f379400af50 tx=0x7f379400b6e0).ready entity=mon.0 client_cookie=94c0ca97a7687a6b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37aa59c700 1 --2- 172.21.15.16:0/88366519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac08d2c0 0x7f37ac08ef20 secure :-1 s=THROTTLE_DONE pgs=1359 cs=0 l=1 rx=0x7f379400af50 tx=0x7f379400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37a9d9b700 1 -- 172.21.15.16:0/88366519 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3794007210 con 0x7f37ac08d2c0 2019-12-05T06:48:37.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37aa59c700 1 --2- 172.21.15.16:0/88366519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac08d2c0 0x7f37ac08ef20 secure :-1 s=THROTTLE_DONE pgs=1359 cs=0 l=1 rx=0x7f379400af50 tx=0x7f379400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.036+0000 7f37aa59c700 1 --2- 172.21.15.16:0/88366519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac08d2c0 0x7f37ac08ef20 secure :-1 s=THROTTLE_DONE pgs=1359 cs=0 l=1 rx=0x7f379400af50 tx=0x7f379400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.037+0000 7f37a9d9b700 1 -- 172.21.15.16:0/88366519 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37940099b0 con 0x7f37ac08d2c0 2019-12-05T06:48:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.037+0000 7f37a9d9b700 1 -- 172.21.15.16:0/88366519 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379400a080 con 0x7f37ac08d2c0 2019-12-05T06:48:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.037+0000 7f37b0bd3700 1 -- 172.21.15.16:0/88366519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac08d2c0 msgr2=0x7f37ac08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.037+0000 7f37b0bd3700 1 --2- 172.21.15.16:0/88366519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac08d2c0 0x7f37ac08ef20 secure :-1 s=READY pgs=1359 cs=0 l=1 rx=0x7f379400af50 tx=0x7f379400b6e0).stop 2019-12-05T06:48:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.039+0000 7f37b0bd3700 1 -- 172.21.15.16:0/88366519 shutdown_connections 2019-12-05T06:48:37.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.039+0000 7f37b0bd3700 1 --2- 172.21.15.16:0/88366519 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac08d2c0 0x7f37ac08ef20 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.039+0000 7f37b0bd3700 1 --2- 172.21.15.16:0/88366519 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f37ac08fef0 0x7f37ac08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:48:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.039+0000 7f37b0bd3700 1 -- 172.21.15.16:0/88366519 shutdown_connections 2019-12-05T06:48:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.039+0000 7f37b0bd3700 1 -- 172.21.15.16:0/88366519 wait complete. 2019-12-05T06:48:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.039+0000 7f37b0bd3700 1 -- 172.21.15.16:0/88366519 >> 172.21.15.16:0/88366519 conn(0x7f37ac061940 msgr2=0x7f37ac096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.043+0000 7f37b0bd3700 1 Processor -- start 2019-12-05T06:48:37.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.043+0000 7f37b0bd3700 1 -- start start 2019-12-05T06:48:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37b0bd3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f37ac09cf40 0x7f37ac09c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37b0bd3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37ac14d840 0x7f37ac1571d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:37.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37b0bd3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37ac14dca0 con 0x7f37ac14d450 2019-12-05T06:48:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37b0bd3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37ac138b80 con 0x7f37ac14d840 2019-12-05T06:48:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37b0bd3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37ac09afe0 con 0x7f37ac09cf40 2019-12-05T06:48:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37ab59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37ac14d840 0x7f37ac1571d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37ab59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f37ac14d840 0x7f37ac1571d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33186/0 (socket says 172.21.15.16:33186) 2019-12-05T06:48:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37ab59e700 1 -- 172.21.15.16:0/3314692076 learned_addr learned my addr 172.21.15.16:0/3314692076 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:48:37.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37aa59c700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f37ac09cf40 0x7f37ac09c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37ab59e700 1 -- 172.21.15.16:0/3314692076 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f37ac09cf40 msgr2=0x7f37ac09c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f37ac09cf40 0x7f37ac09c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37ab59e700 1 -- 172.21.15.16:0/3314692076 >> v1:172.21.15.76:6789/0 conn(0x7f37ac14d450 legacy=0x7f37ac09bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.044+0000 7f37ab59e700 1 -- 172.21.15.16:0/3314692076 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac09b670 con 0x7f37ac14d840 2019-12-05T06:48:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.045+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> v2:172.21.15.16:3300/0 conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.045+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> v2:172.21.15.16:3300/0 conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.045+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.045+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.045+0000 7f37a37fe700 1 -- 172.21.15.16:0/3314692076 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379c00f090 con 0x7f37ac14d840 2019-12-05T06:48:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.045+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.045+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.045+0000 7f37a37fe700 1 -- 172.21.15.16:0/3314692076 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379c000a70 con 0x7f37ac14d840 2019-12-05T06:48:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.045+0000 7f37b0bd3700 1 -- 172.21.15.16:0/3314692076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37ac09b670 con 0x7f37ac14d840 2019-12-05T06:48:37.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.045+0000 7f37a37fe700 1 -- 172.21.15.16:0/3314692076 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379c00f670 con 0x7f37ac14d840 2019-12-05T06:48:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.046+0000 7f37b0bd3700 1 -- 172.21.15.16:0/3314692076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37ac09b670 con 0x7f37ac14d840 2019-12-05T06:48:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.046+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.046+0000 7f37a37fe700 1 -- 172.21.15.16:0/3314692076 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f379c00edf0 con 0x7f37ac14d840 2019-12-05T06:48:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.046+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.047+0000 7f37a37fe700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3790023630 0x7f3790025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:48:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.047+0000 7f37a37fe700 1 -- 172.21.15.16:0/3314692076 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f379c045ee0 con 0x7f37ac14d840 2019-12-05T06:48:37.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.047+0000 7f37aad9d700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3790023630 0x7f3790025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:48:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.048+0000 7f37aad9d700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3790023630 0x7f3790025ae0 crc :-1 s=READY pgs=572 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:48:37.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.050+0000 7f37b0bd3700 1 -- 172.21.15.16:0/3314692076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37ac09b670 con 0x7f37ac14d840 2019-12-05T06:48:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.058+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.058+0000 7f37a37fe700 1 -- 172.21.15.16:0/3314692076 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f379c00ea80 con 0x7f37ac14d840 2019-12-05T06:48:37.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.324+0000 7f37b0bd3700 1 -- 172.21.15.16:0/3314692076 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37ac09b670 con 0x7f37ac14d840 2019-12-05T06:48:37.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.325+0000 7f37ab59e700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=THROTTLE_DONE pgs=1360 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:48:37.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.325+0000 7f37a37fe700 1 -- 172.21.15.16:0/3314692076 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f379c000a70 con 0x7f37ac14d840 2019-12-05T06:48:37.331 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 167 2019-12-05T06:48:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 -- 172.21.15.16:0/3314692076 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3790023630 msgr2=0x7f3790025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3790023630 0x7f3790025ae0 crc :-1 s=READY pgs=572 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 -- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 msgr2=0x7f37ac1571d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:48:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 0x7f37ac1571d0 secure :-1 s=READY pgs=1360 cs=0 l=1 rx=0x7f379c010450 tx=0x7f379c010280).stop 2019-12-05T06:48:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 -- 172.21.15.16:0/3314692076 shutdown_connections 2019-12-05T06:48:37.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3790023630 0x7f3790025ae0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f37ac14d840 0x7f37ac1571d0 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 --2- 172.21.15.16:0/3314692076 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f37ac09cf40 0x7f37ac09c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:48:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 -- 172.21.15.16:0/3314692076 shutdown_connections 2019-12-05T06:48:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 -- 172.21.15.16:0/3314692076 wait complete. 2019-12-05T06:48:37.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:48:37.330+0000 7f37b0bd3700 1 -- 172.21.15.16:0/3314692076 >> 172.21.15.16:0/3314692076 conn(0x7f37ac07aeb0 msgr2=0x7f37ac0a2040 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:48:37.333 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:48:37.334 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5897,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:48:37.342 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:48:37.342 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 85 secs before thrashing 2019-12-05T06:48:44.541 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:48:44.546 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:49:14.699 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:49:14.709 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:49:44.829 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:49:44.835 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:50:02.344 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:02.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.621+0000 7f87d6262700 1 Processor -- start 2019-12-05T06:50:02.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.622+0000 7f87d6262700 1 -- start start 2019-12-05T06:50:02.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.623+0000 7f87d6262700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f87d008d2c0 0x7f87d008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:02.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.623+0000 7f87d6262700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87d008fef0 0x7f87d008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:02.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.623+0000 7f87d6262700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d00958e0 con 0x7f87d008c6d0 2019-12-05T06:50:02.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.623+0000 7f87d6262700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d008df90 con 0x7f87d008fef0 2019-12-05T06:50:02.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.623+0000 7f87d6262700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d00a3f70 con 0x7f87d008d2c0 2019-12-05T06:50:02.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.624+0000 7f87d5000700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87d008fef0 0x7f87d008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:02.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.624+0000 7f87cffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f87d008c6d0 0x7f87d008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:46974/0 (socket says 172.21.15.16:46974) 2019-12-05T06:50:02.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.624+0000 7f87cffff700 1 -- 172.21.15.16:0/535557913 learned_addr learned my addr 172.21.15.16:0/535557913 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:02.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.624+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/535557913 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f87d008d2c0 0x7f87d008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:02.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87ceffd700 1 -- 172.21.15.16:0/535557913 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193873469 0 0) 0x7f87c00014d0 con 0x7f87d008c6d0 2019-12-05T06:50:02.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87ceffd700 1 -- 172.21.15.16:0/535557913 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87bc002e80 con 0x7f87d008c6d0 2019-12-05T06:50:02.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87d5000700 1 -- 172.21.15.16:0/535557913 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f87d008d2c0 msgr2=0x7f87d008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:02.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87d5000700 1 --2- 172.21.15.16:0/535557913 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f87d008d2c0 0x7f87d008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:02.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87d5000700 1 -- 172.21.15.16:0/535557913 >> v1:172.21.15.16:6789/0 conn(0x7f87d008c6d0 legacy=0x7f87d008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:02.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87d5000700 1 -- 172.21.15.16:0/535557913 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d008e620 con 0x7f87d008fef0 2019-12-05T06:50:02.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87d5000700 1 --2- 172.21.15.16:0/535557913 >> v2:172.21.15.16:3300/0 conn(0x7f87d008fef0 0x7f87d008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f87c800af50 tx=0x7f87c800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87d5000700 1 --2- 172.21.15.16:0/535557913 >> v2:172.21.15.16:3300/0 conn(0x7f87d008fef0 0x7f87d008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f87c800af50 tx=0x7f87c800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87d5000700 1 --2- 172.21.15.16:0/535557913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d008fef0 0x7f87d008f460 secure :-1 s=READY pgs=1368 cs=0 l=1 rx=0x7f87c800af50 tx=0x7f87c800b250).ready entity=mon.0 client_cookie=9c382458e2d67883 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:02.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87d5000700 1 --2- 172.21.15.16:0/535557913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d008fef0 0x7f87d008f460 secure :-1 s=THROTTLE_DONE pgs=1368 cs=0 l=1 rx=0x7f87c800af50 tx=0x7f87c800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.625+0000 7f87ceffd700 1 -- 172.21.15.16:0/535557913 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87c8009ba0 con 0x7f87d008fef0 2019-12-05T06:50:02.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.626+0000 7f87d5000700 1 --2- 172.21.15.16:0/535557913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d008fef0 0x7f87d008f460 secure :-1 s=THROTTLE_DONE pgs=1368 cs=0 l=1 rx=0x7f87c800af50 tx=0x7f87c800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.626+0000 7f87d5000700 1 --2- 172.21.15.16:0/535557913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d008fef0 0x7f87d008f460 secure :-1 s=THROTTLE_DONE pgs=1368 cs=0 l=1 rx=0x7f87c800af50 tx=0x7f87c800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.626+0000 7f87ceffd700 1 -- 172.21.15.16:0/535557913 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87c80097c0 con 0x7f87d008fef0 2019-12-05T06:50:02.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.626+0000 7f87ceffd700 1 -- 172.21.15.16:0/535557913 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87c800a180 con 0x7f87d008fef0 2019-12-05T06:50:02.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.626+0000 7f87d6262700 1 -- 172.21.15.16:0/535557913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d008fef0 msgr2=0x7f87d008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:02.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.626+0000 7f87d6262700 1 --2- 172.21.15.16:0/535557913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d008fef0 0x7f87d008f460 secure :-1 s=READY pgs=1368 cs=0 l=1 rx=0x7f87c800af50 tx=0x7f87c800b250).stop 2019-12-05T06:50:02.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.628+0000 7f87d6262700 1 -- 172.21.15.16:0/535557913 shutdown_connections 2019-12-05T06:50:02.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.628+0000 7f87d6262700 1 --2- 172.21.15.16:0/535557913 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d008fef0 0x7f87d008f460 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:02.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.628+0000 7f87d6262700 1 --2- 172.21.15.16:0/535557913 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f87d008d2c0 0x7f87d008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:02.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.628+0000 7f87d6262700 1 -- 172.21.15.16:0/535557913 shutdown_connections 2019-12-05T06:50:02.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.628+0000 7f87d6262700 1 -- 172.21.15.16:0/535557913 wait complete. 2019-12-05T06:50:02.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.628+0000 7f87d6262700 1 -- 172.21.15.16:0/535557913 >> 172.21.15.16:0/535557913 conn(0x7f87d0061940 msgr2=0x7f87d0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:02.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.633+0000 7f87d6262700 1 Processor -- start 2019-12-05T06:50:02.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.634+0000 7f87d6262700 1 -- start start 2019-12-05T06:50:02.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.635+0000 7f87d6262700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87d009bbe0 0x7f87d01539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:02.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.635+0000 7f87d6262700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f87d0084f30 0x7f87d0155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:02.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.635+0000 7f87d6262700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87d0084b90 con 0x7f87d0096e90 2019-12-05T06:50:02.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.635+0000 7f87d6262700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87d009a450 con 0x7f87d009bbe0 2019-12-05T06:50:02.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.635+0000 7f87d6262700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f87d0099a20 con 0x7f87d0084f30 2019-12-05T06:50:02.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.635+0000 7f87cf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87d009bbe0 0x7f87d01539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:02.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.635+0000 7f87cf7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f87d009bbe0 0x7f87d01539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33580/0 (socket says 172.21.15.16:33580) 2019-12-05T06:50:02.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.635+0000 7f87cf7fe700 1 -- 172.21.15.16:0/3400745210 learned_addr learned my addr 172.21.15.16:0/3400745210 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:02.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.636+0000 7f87d5000700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f87d0084f30 0x7f87d0155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:02.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.636+0000 7f87cf7fe700 1 -- 172.21.15.16:0/3400745210 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f87d0084f30 msgr2=0x7f87d0155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:02.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.636+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f87d0084f30 0x7f87d0155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:02.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.636+0000 7f87cf7fe700 1 -- 172.21.15.16:0/3400745210 >> v1:172.21.15.76:6789/0 conn(0x7f87d0096e90 legacy=0x7f87d009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:02.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.636+0000 7f87cf7fe700 1 -- 172.21.15.16:0/3400745210 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87d009aae0 con 0x7f87d009bbe0 2019-12-05T06:50:02.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.636+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> v2:172.21.15.16:3300/0 conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.636+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> v2:172.21.15.16:3300/0 conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.636+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:02.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.637+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=THROTTLE_DONE pgs=1369 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.637+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=THROTTLE_DONE pgs=1369 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.637+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=THROTTLE_DONE pgs=1369 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.637+0000 7f87ccff9700 1 -- 172.21.15.16:0/3400745210 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87b8000940 con 0x7f87d009bbe0 2019-12-05T06:50:02.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.637+0000 7f87ccff9700 1 -- 172.21.15.16:0/3400745210 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f87b8000e80 con 0x7f87d009bbe0 2019-12-05T06:50:02.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.637+0000 7f87d6262700 1 -- 172.21.15.16:0/3400745210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d009aae0 con 0x7f87d009bbe0 2019-12-05T06:50:02.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.637+0000 7f87ccff9700 1 -- 172.21.15.16:0/3400745210 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f87b80014d0 con 0x7f87d009bbe0 2019-12-05T06:50:02.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.637+0000 7f87d6262700 1 -- 172.21.15.16:0/3400745210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d009aae0 con 0x7f87d009bbe0 2019-12-05T06:50:02.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.638+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=THROTTLE_DONE pgs=1369 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.639+0000 7f87ccff9700 1 -- 172.21.15.16:0/3400745210 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f87b8001070 con 0x7f87d009bbe0 2019-12-05T06:50:02.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.639+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=THROTTLE_DONE pgs=1369 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.639+0000 7f87ccff9700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87b4023630 0x7f87b4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:02.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.639+0000 7f87ccff9700 1 -- 172.21.15.16:0/3400745210 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f87b8045de0 con 0x7f87d009bbe0 2019-12-05T06:50:02.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.639+0000 7f87cffff700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87b4023630 0x7f87b4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:02.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.640+0000 7f87cffff700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87b4023630 0x7f87b4025ac0 crc :-1 s=READY pgs=574 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:02.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.653+0000 7f87d6262700 1 -- 172.21.15.16:0/3400745210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d009aae0 con 0x7f87d009bbe0 2019-12-05T06:50:02.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.660+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=THROTTLE_DONE pgs=1369 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.660+0000 7f87ccff9700 1 -- 172.21.15.16:0/3400745210 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f87b8033940 con 0x7f87d009bbe0 2019-12-05T06:50:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.926+0000 7f87d6262700 1 -- 172.21.15.16:0/3400745210 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f87d009aae0 con 0x7f87d009bbe0 2019-12-05T06:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.926+0000 7f87cf7fe700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=THROTTLE_DONE pgs=1369 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.926+0000 7f87ccff9700 1 -- 172.21.15.16:0/3400745210 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f87b8033940 con 0x7f87d009bbe0 2019-12-05T06:50:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 167 2019-12-05T06:50:02.929 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:02.929 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5897,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 -- 172.21.15.16:0/3400745210 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87b4023630 msgr2=0x7f87b4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87b4023630 0x7f87b4025ac0 crc :-1 s=READY pgs=574 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:02.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 -- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 msgr2=0x7f87d01539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:02.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 0x7f87d01539e0 secure :-1 s=READY pgs=1369 cs=0 l=1 rx=0x7f87b8010450 tx=0x7f87b8010280).stop 2019-12-05T06:50:02.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 -- 172.21.15.16:0/3400745210 shutdown_connections 2019-12-05T06:50:02.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f87d0084f30 0x7f87d0155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:02.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f87b4023630 0x7f87b4025ac0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:02.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 --2- 172.21.15.16:0/3400745210 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f87d009bbe0 0x7f87d01539e0 unknown :-1 s=CLOSED pgs=1369 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:02.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 -- 172.21.15.16:0/3400745210 shutdown_connections 2019-12-05T06:50:02.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 -- 172.21.15.16:0/3400745210 wait complete. 2019-12-05T06:50:02.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:02.929+0000 7f87d6262700 1 -- 172.21.15.16:0/3400745210 >> 172.21.15.16:0/3400745210 conn(0x7f87d00714f0 msgr2=0x7f87d009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:02.947 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:50:02.947 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:50:02.947 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:50:02.963 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:50:02.963 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:03.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.191+0000 7f5015b69700 1 Processor -- start 2019-12-05T06:50:03.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.191+0000 7f5015b69700 1 -- start start 2019-12-05T06:50:03.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.193+0000 7f5015b69700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f501008c6d0 0x7f501008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:03.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.193+0000 7f5015b69700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f501008fef0 0x7f501008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:03.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.193+0000 7f5015b69700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501008fb80 con 0x7f501008ef20 2019-12-05T06:50:03.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.193+0000 7f5015b69700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f501008df90 con 0x7f501008c6d0 2019-12-05T06:50:03.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.193+0000 7f5015b69700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50100a3f70 con 0x7f501008fef0 2019-12-05T06:50:03.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.193+0000 7f500ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f501008c6d0 0x7f501008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:03.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.193+0000 7f500ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f501008c6d0 0x7f501008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33590/0 (socket says 172.21.15.16:33590) 2019-12-05T06:50:03.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.193+0000 7f500ffff700 1 -- 172.21.15.16:0/529591697 learned_addr learned my addr 172.21.15.16:0/529591697 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:03.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.194+0000 7f500f7fe700 1 --2- 172.21.15.16:0/529591697 >> v2:172.21.15.76:3300/0 conn(0x7f501008fef0 0x7f501008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:03.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.194+0000 7f500ffff700 1 -- 172.21.15.16:0/529591697 >> v2:172.21.15.76:3300/0 conn(0x7f501008fef0 msgr2=0x7f501008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:03.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.194+0000 7f500ffff700 1 --2- 172.21.15.16:0/529591697 >> v2:172.21.15.76:3300/0 conn(0x7f501008fef0 0x7f501008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:03.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.194+0000 7f500ffff700 1 -- 172.21.15.16:0/529591697 >> v1:172.21.15.76:6789/0 conn(0x7f501008ef20 legacy=0x7f501008f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.194+0000 7f500effd700 1 -- 172.21.15.16:0/529591697 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954667768 0 0) 0x7f50080009b0 con 0x7f501008ef20 2019-12-05T06:50:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.194+0000 7f500ffff700 1 -- 172.21.15.16:0/529591697 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501008d540 con 0x7f501008c6d0 2019-12-05T06:50:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.194+0000 7f500ffff700 1 --2- 172.21.15.16:0/529591697 >> v2:172.21.15.16:3300/0 conn(0x7f501008c6d0 0x7f501008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f500000fa40 tx=0x7f500000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.194+0000 7f500ffff700 1 --2- 172.21.15.16:0/529591697 >> v2:172.21.15.16:3300/0 conn(0x7f501008c6d0 0x7f501008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f500000fa40 tx=0x7f500000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.194+0000 7f500ffff700 1 --2- 172.21.15.16:0/529591697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501008c6d0 0x7f501008cac0 secure :-1 s=READY pgs=1370 cs=0 l=1 rx=0x7f500000fa40 tx=0x7f500000fde0).ready entity=mon.0 client_cookie=feb95a5c7ad54ba8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.195+0000 7f500ffff700 1 --2- 172.21.15.16:0/529591697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501008c6d0 0x7f501008cac0 secure :-1 s=THROTTLE_DONE pgs=1370 cs=0 l=1 rx=0x7f500000fa40 tx=0x7f500000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.195+0000 7f500effd700 1 -- 172.21.15.16:0/529591697 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5000000ce0 con 0x7f501008c6d0 2019-12-05T06:50:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.195+0000 7f500ffff700 1 --2- 172.21.15.16:0/529591697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501008c6d0 0x7f501008cac0 secure :-1 s=THROTTLE_DONE pgs=1370 cs=0 l=1 rx=0x7f500000fa40 tx=0x7f500000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.195+0000 7f500ffff700 1 --2- 172.21.15.16:0/529591697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501008c6d0 0x7f501008cac0 secure :-1 s=THROTTLE_DONE pgs=1370 cs=0 l=1 rx=0x7f500000fa40 tx=0x7f500000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.195+0000 7f500effd700 1 -- 172.21.15.16:0/529591697 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5000001010 con 0x7f501008c6d0 2019-12-05T06:50:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.195+0000 7f500effd700 1 -- 172.21.15.16:0/529591697 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5000000ce0 con 0x7f501008c6d0 2019-12-05T06:50:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.195+0000 7f5015b69700 1 -- 172.21.15.16:0/529591697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501008c6d0 msgr2=0x7f501008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.195+0000 7f5015b69700 1 --2- 172.21.15.16:0/529591697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501008c6d0 0x7f501008cac0 secure :-1 s=READY pgs=1370 cs=0 l=1 rx=0x7f500000fa40 tx=0x7f500000fde0).stop 2019-12-05T06:50:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.196+0000 7f5015b69700 1 -- 172.21.15.16:0/529591697 shutdown_connections 2019-12-05T06:50:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.196+0000 7f5015b69700 1 --2- 172.21.15.16:0/529591697 >> v2:172.21.15.76:3300/0 conn(0x7f501008fef0 0x7f501008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.196+0000 7f5015b69700 1 --2- 172.21.15.16:0/529591697 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f501008c6d0 0x7f501008cac0 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.197+0000 7f5015b69700 1 -- 172.21.15.16:0/529591697 shutdown_connections 2019-12-05T06:50:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.197+0000 7f5015b69700 1 -- 172.21.15.16:0/529591697 wait complete. 2019-12-05T06:50:03.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.197+0000 7f5015b69700 1 -- 172.21.15.16:0/529591697 >> 172.21.15.16:0/529591697 conn(0x7f5010061940 msgr2=0x7f5010096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.201+0000 7f5015b69700 1 Processor -- start 2019-12-05T06:50:03.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.202+0000 7f5015b69700 1 -- start start 2019-12-05T06:50:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.203+0000 7f5015b69700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5010132630 0x7f5010132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.203+0000 7f5015b69700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5010099eb0 0x7f501009a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.203+0000 7f5015b69700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f501006e7c0 con 0x7f501009ae80 2019-12-05T06:50:03.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.203+0000 7f5015b69700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5010098f20 con 0x7f5010132630 2019-12-05T06:50:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.203+0000 7f5015b69700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5010134390 con 0x7f5010099eb0 2019-12-05T06:50:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.203+0000 7f5014907700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5010099eb0 0x7f501009a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f500ffff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5010132630 0x7f5010132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f500f7fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f501009ae80 0x7f5010132f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:40066/0 (socket says 172.21.15.16:40066) 2019-12-05T06:50:03.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f500ffff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5010132630 0x7f5010132a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:47918/0 (socket says 172.21.15.16:47918) 2019-12-05T06:50:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f5014907700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5010099eb0 0x7f501009a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:37588/0 (socket says 172.21.15.16:37588) 2019-12-05T06:50:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f500f7fe700 1 -- 172.21.15.16:0/2239936075 learned_addr learned my addr 172.21.15.16:0/2239936075 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f500ffff700 1 -- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5010099eb0 msgr2=0x7f501009a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5010099eb0 0x7f501009a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f500ffff700 1 -- 172.21.15.16:0/2239936075 >> v1:172.21.15.76:6789/0 conn(0x7f501009ae80 legacy=0x7f5010132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f500ffff700 1 -- 172.21.15.16:0/2239936075 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50100995b0 con 0x7f5010132630 2019-12-05T06:50:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f500cff9700 1 -- 172.21.15.16:0/2239936075 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808689540 0 0) 0x7f4ff8000c90 con 0x7f501009ae80 2019-12-05T06:50:03.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f5014907700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5010099eb0 0x7f501009a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.204+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> v2:172.21.15.76:3300/0 conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.205+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> v2:172.21.15.76:3300/0 conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.205+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=READY pgs=933 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:03.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.205+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=THROTTLE_DONE pgs=933 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.205+0000 7f500cff9700 1 -- 172.21.15.16:0/2239936075 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5000000cc0 con 0x7f5010132630 2019-12-05T06:50:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.205+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=THROTTLE_DONE pgs=933 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.205+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=THROTTLE_DONE pgs=933 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.205+0000 7f500cff9700 1 -- 172.21.15.16:0/2239936075 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50000081d0 con 0x7f5010132630 2019-12-05T06:50:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.205+0000 7f500cff9700 1 -- 172.21.15.16:0/2239936075 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5000005980 con 0x7f5010132630 2019-12-05T06:50:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.205+0000 7f5015b69700 1 -- 172.21.15.16:0/2239936075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50100995b0 con 0x7f5010132630 2019-12-05T06:50:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.205+0000 7f5015b69700 1 -- 172.21.15.16:0/2239936075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50100995b0 con 0x7f5010132630 2019-12-05T06:50:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.206+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=THROTTLE_DONE pgs=933 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.207+0000 7f500cff9700 1 -- 172.21.15.16:0/2239936075 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5000005500 con 0x7f5010132630 2019-12-05T06:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.207+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=THROTTLE_DONE pgs=933 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.207+0000 7f500cff9700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ff4023630 0x7f4ff4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.208+0000 7f500cff9700 1 -- 172.21.15.16:0/2239936075 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f500008a080 con 0x7f5010132630 2019-12-05T06:50:03.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.208+0000 7f500f7fe700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ff4023630 0x7f4ff4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.208+0000 7f500f7fe700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ff4023630 0x7f4ff4025ac0 crc :-1 s=READY pgs=575 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:03.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.223+0000 7f5015b69700 1 -- 172.21.15.16:0/2239936075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50100995b0 con 0x7f5010132630 2019-12-05T06:50:03.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.232+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=THROTTLE_DONE pgs=933 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.232+0000 7f500cff9700 1 -- 172.21.15.16:0/2239936075 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5000008570 con 0x7f5010132630 2019-12-05T06:50:03.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.501+0000 7f5015b69700 1 -- 172.21.15.16:0/2239936075 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50100995b0 con 0x7f5010132630 2019-12-05T06:50:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.502+0000 7f500ffff700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=THROTTLE_DONE pgs=933 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.502+0000 7f500cff9700 1 -- 172.21.15.16:0/2239936075 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f5000005500 con 0x7f5010132630 2019-12-05T06:50:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 167 2019-12-05T06:50:03.505 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:03.505 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5897,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.507+0000 7f5015b69700 1 -- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ff4023630 msgr2=0x7f4ff4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.507+0000 7f5015b69700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ff4023630 0x7f4ff4025ac0 crc :-1 s=READY pgs=575 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.507+0000 7f5015b69700 1 -- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 msgr2=0x7f5010132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.507+0000 7f5015b69700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 0x7f5010132a20 secure :-1 s=READY pgs=933 cs=0 l=1 rx=0x7f5000054740 tx=0x7f5000054ae0).stop 2019-12-05T06:50:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.508+0000 7f5015b69700 1 -- 172.21.15.16:0/2239936075 shutdown_connections 2019-12-05T06:50:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.508+0000 7f5015b69700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4ff4023630 0x7f4ff4025ac0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.508+0000 7f5015b69700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5010099eb0 0x7f501009a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.508+0000 7f5015b69700 1 --2- 172.21.15.16:0/2239936075 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5010132630 0x7f5010132a20 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:03.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.508+0000 7f5015b69700 1 -- 172.21.15.16:0/2239936075 shutdown_connections 2019-12-05T06:50:03.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.508+0000 7f5015b69700 1 -- 172.21.15.16:0/2239936075 wait complete. 2019-12-05T06:50:03.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:03.508+0000 7f5015b69700 1 -- 172.21.15.16:0/2239936075 >> 172.21.15.16:0/2239936075 conn(0x7f50100714f0 msgr2=0x7f501009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:03.528 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:05.530 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:05.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.771+0000 7f59bbdcc700 1 Processor -- start 2019-12-05T06:50:05.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.771+0000 7f59bbdcc700 1 -- start start 2019-12-05T06:50:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.773+0000 7f59bbdcc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 0x7f59b408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.774+0000 7f59bbdcc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b40958e0 con 0x7f59b408c6d0 2019-12-05T06:50:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.774+0000 7f59bbdcc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b40629e0 con 0x7f59b408d2c0 2019-12-05T06:50:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.774+0000 7f59bbdcc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b408df90 con 0x7f59b408fef0 2019-12-05T06:50:05.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.774+0000 7f59ba369700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f59b408c6d0 0x7f59b408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47006/0 (socket says 172.21.15.16:47006) 2019-12-05T06:50:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.774+0000 7f59ba369700 1 -- 172.21.15.16:0/394499213 learned_addr learned my addr 172.21.15.16:0/394499213 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.774+0000 7f59bab6a700 1 --2- 172.21.15.16:0/394499213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 0x7f59b408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.774+0000 7f59b9367700 1 -- 172.21.15.16:0/394499213 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425010439 0 0) 0x7f59a4001150 con 0x7f59b408c6d0 2019-12-05T06:50:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.775+0000 7f59b9367700 1 -- 172.21.15.16:0/394499213 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c002e80 con 0x7f59b408c6d0 2019-12-05T06:50:05.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.775+0000 7f59b9367700 1 -- 172.21.15.16:0/394499213 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3107391681 0 0) 0x7f59b0001400 con 0x7f59b408d2c0 2019-12-05T06:50:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.775+0000 7f59b9367700 1 -- 172.21.15.16:0/394499213 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f599c004170 con 0x7f59b408d2c0 2019-12-05T06:50:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.775+0000 7f59bab6a700 1 -- 172.21.15.16:0/394499213 >> v1:172.21.15.76:6789/0 conn(0x7f59b408d2c0 legacy=0x7f59b408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.775+0000 7f59b9367700 1 -- 172.21.15.16:0/394499213 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 762508843 0 0) 0x7f59a4001130 con 0x7f59b408c6d0 2019-12-05T06:50:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.775+0000 7f59bab6a700 1 -- 172.21.15.16:0/394499213 >> v1:172.21.15.16:6789/0 conn(0x7f59b408c6d0 legacy=0x7f59b408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.775+0000 7f59bab6a700 1 -- 172.21.15.16:0/394499213 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b408e620 con 0x7f59b408fef0 2019-12-05T06:50:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.775+0000 7f59bab6a700 1 --2- 172.21.15.16:0/394499213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 0x7f59b408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f59ac00af80 tx=0x7f59ac00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.776+0000 7f59bab6a700 1 --2- 172.21.15.16:0/394499213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 0x7f59b408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f59ac00af80 tx=0x7f59ac00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.776+0000 7f59bab6a700 1 --2- 172.21.15.16:0/394499213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 0x7f59b408f720 secure :-1 s=READY pgs=317 cs=0 l=1 rx=0x7f59ac00af80 tx=0x7f59ac00b790).ready entity=mon.2 client_cookie=38830900637c98e9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.776+0000 7f59bab6a700 1 --2- 172.21.15.16:0/394499213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 0x7f59b408f720 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f59ac00af80 tx=0x7f59ac00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.776+0000 7f59bab6a700 1 --2- 172.21.15.16:0/394499213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 0x7f59b408f720 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f59ac00af80 tx=0x7f59ac00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.776+0000 7f59b9367700 1 -- 172.21.15.16:0/394499213 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59ac007240 con 0x7f59b408fef0 2019-12-05T06:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.776+0000 7f59bab6a700 1 --2- 172.21.15.16:0/394499213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 0x7f59b408f720 secure :-1 s=THROTTLE_DONE pgs=317 cs=0 l=1 rx=0x7f59ac00af80 tx=0x7f59ac00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.776+0000 7f59b9367700 1 -- 172.21.15.16:0/394499213 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59ac000c30 con 0x7f59b408fef0 2019-12-05T06:50:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.776+0000 7f59b9367700 1 -- 172.21.15.16:0/394499213 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59ac009e60 con 0x7f59b408fef0 2019-12-05T06:50:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.776+0000 7f59bbdcc700 1 -- 172.21.15.16:0/394499213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 msgr2=0x7f59b408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.776+0000 7f59bbdcc700 1 --2- 172.21.15.16:0/394499213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 0x7f59b408f720 secure :-1 s=READY pgs=317 cs=0 l=1 rx=0x7f59ac00af80 tx=0x7f59ac00b790).stop 2019-12-05T06:50:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.778+0000 7f59bbdcc700 1 -- 172.21.15.16:0/394499213 shutdown_connections 2019-12-05T06:50:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.778+0000 7f59bbdcc700 1 --2- 172.21.15.16:0/394499213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b408fef0 0x7f59b408f720 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.778+0000 7f59bbdcc700 1 -- 172.21.15.16:0/394499213 shutdown_connections 2019-12-05T06:50:05.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.779+0000 7f59bbdcc700 1 -- 172.21.15.16:0/394499213 wait complete. 2019-12-05T06:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.779+0000 7f59bbdcc700 1 -- 172.21.15.16:0/394499213 >> 172.21.15.16:0/394499213 conn(0x7f59b4061940 msgr2=0x7f59b4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.783+0000 7f59bbdcc700 1 Processor -- start 2019-12-05T06:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.783+0000 7f59bbdcc700 1 -- start start 2019-12-05T06:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.784+0000 7f59bbdcc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.784+0000 7f59bbdcc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f59b4154cb0 0x7f59b41570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.784+0000 7f59bbdcc700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59b4128b20 con 0x7f59b412c740 2019-12-05T06:50:05.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.784+0000 7f59bbdcc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59b4129db0 con 0x7f59b4154cb0 2019-12-05T06:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.784+0000 7f59bbdcc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59b412a080 con 0x7f59b41325e0 2019-12-05T06:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.785+0000 7f59b9b68700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f59b412c740 0x7f59b412bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47014/0 (socket says 172.21.15.16:47014) 2019-12-05T06:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.785+0000 7f59b9b68700 1 -- 172.21.15.16:0/445492752 learned_addr learned my addr 172.21.15.16:0/445492752 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.785+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.785+0000 7f59bab6a700 1 --2- 172.21.15.16:0/445492752 >> v2:172.21.15.76:3300/0 conn(0x7f59b4154cb0 0x7f59b41570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.785+0000 7f59aaffd700 1 -- 172.21.15.16:0/445492752 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076337737 0 0) 0x7f59b00015c0 con 0x7f59b412c740 2019-12-05T06:50:05.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.785+0000 7f59aaffd700 1 -- 172.21.15.16:0/445492752 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5994002e80 con 0x7f59b412c740 2019-12-05T06:50:05.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59ba369700 1 -- 172.21.15.16:0/445492752 >> v2:172.21.15.76:3300/0 conn(0x7f59b4154cb0 msgr2=0x7f59b41570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:05.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> v2:172.21.15.76:3300/0 conn(0x7f59b4154cb0 0x7f59b41570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:05.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59ba369700 1 -- 172.21.15.16:0/445492752 >> v1:172.21.15.16:6789/0 conn(0x7f59b412c740 legacy=0x7f59b412bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:05.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59aaffd700 1 -- 172.21.15.16:0/445492752 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2532035818 0 0) 0x7f59b00017c0 con 0x7f59b412c740 2019-12-05T06:50:05.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59ba369700 1 -- 172.21.15.16:0/445492752 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b414a820 con 0x7f59b41325e0 2019-12-05T06:50:05.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59bab6a700 1 --2- 172.21.15.16:0/445492752 >> v2:172.21.15.76:3300/0 conn(0x7f59b4154cb0 0x7f59b41570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:05.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=READY pgs=318 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=THROTTLE_DONE pgs=318 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.786+0000 7f59aaffd700 1 -- 172.21.15.16:0/445492752 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59a4000d80 con 0x7f59b41325e0 2019-12-05T06:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.787+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=THROTTLE_DONE pgs=318 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.787+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=THROTTLE_DONE pgs=318 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.787+0000 7f59aaffd700 1 -- 172.21.15.16:0/445492752 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59a40013e0 con 0x7f59b41325e0 2019-12-05T06:50:05.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.787+0000 7f59aaffd700 1 -- 172.21.15.16:0/445492752 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59a40015d0 con 0x7f59b41325e0 2019-12-05T06:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.787+0000 7f59bbdcc700 1 -- 172.21.15.16:0/445492752 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59b412ae70 con 0x7f59b41325e0 2019-12-05T06:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.787+0000 7f59bbdcc700 1 -- 172.21.15.16:0/445492752 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59b414a820 con 0x7f59b41325e0 2019-12-05T06:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.788+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=THROTTLE_DONE pgs=318 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.789+0000 7f59aaffd700 1 -- 172.21.15.16:0/445492752 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f59a40015d0 con 0x7f59b41325e0 2019-12-05T06:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.789+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=THROTTLE_DONE pgs=318 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.789+0000 7f59aaffd700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5994023680 0x7f5994025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:05.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.789+0000 7f59aaffd700 1 -- 172.21.15.16:0/445492752 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f59a4045d00 con 0x7f59b41325e0 2019-12-05T06:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.789+0000 7f59b9b68700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5994023680 0x7f5994025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.790+0000 7f59b9b68700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5994023680 0x7f5994025af0 crc :-1 s=READY pgs=576 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:05.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.803+0000 7f59bbdcc700 1 -- 172.21.15.16:0/445492752 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59b412ae70 con 0x7f59b41325e0 2019-12-05T06:50:05.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.813+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=THROTTLE_DONE pgs=318 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:05.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:05.813+0000 7f59aaffd700 1 -- 172.21.15.16:0/445492752 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59a4000fe0 con 0x7f59b41325e0 2019-12-05T06:50:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.070+0000 7f59bbdcc700 1 -- 172.21.15.16:0/445492752 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59b414a820 con 0x7f59b41325e0 2019-12-05T06:50:06.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.072+0000 7f59ba369700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=THROTTLE_DONE pgs=318 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:06.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.072+0000 7f59aaffd700 1 -- 172.21.15.16:0/445492752 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f59a40015d0 con 0x7f59b41325e0 2019-12-05T06:50:06.074 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 167 2019-12-05T06:50:06.074 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:06.074 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5897,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 -- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5994023680 msgr2=0x7f5994025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5994023680 0x7f5994025af0 crc :-1 s=READY pgs=576 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 -- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 msgr2=0x7f59b412b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:06.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 secure :-1 s=READY pgs=318 cs=0 l=1 rx=0x7f59a4010380 tx=0x7f59a4010720).stop 2019-12-05T06:50:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 -- 172.21.15.16:0/445492752 shutdown_connections 2019-12-05T06:50:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5994023680 0x7f5994025af0 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 --2- 172.21.15.16:0/445492752 >> v2:172.21.15.76:3300/0 conn(0x7f59b4154cb0 0x7f59b41570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 --2- 172.21.15.16:0/445492752 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59b41325e0 0x7f59b412b770 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:06.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 -- 172.21.15.16:0/445492752 shutdown_connections 2019-12-05T06:50:06.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 -- 172.21.15.16:0/445492752 wait complete. 2019-12-05T06:50:06.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:06.076+0000 7f59bbdcc700 1 -- 172.21.15.16:0/445492752 >> 172.21.15.16:0/445492752 conn(0x7f59b40a2850 msgr2=0x7f59b406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:06.088 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:08.090 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.345+0000 7f3b0e799700 1 Processor -- start 2019-12-05T06:50:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.345+0000 7f3b0e799700 1 -- start start 2019-12-05T06:50:08.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.346+0000 7f3b0e799700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b0808d2c0 0x7f3b0808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.346+0000 7f3b0e799700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b080958e0 con 0x7f3b0808fef0 2019-12-05T06:50:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.347+0000 7f3b0e799700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b080629e0 con 0x7f3b0808c6d0 2019-12-05T06:50:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.347+0000 7f3b0e799700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0808df90 con 0x7f3b0808d2c0 2019-12-05T06:50:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.347+0000 7f3b07fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b0808d2c0 0x7f3b0808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.347+0000 7f3b0d537700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3b0808fef0 0x7f3b0808f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47026/0 (socket says 172.21.15.16:47026) 2019-12-05T06:50:08.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.347+0000 7f3b07fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b0808d2c0 0x7f3b0808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33624/0 (socket says 172.21.15.16:33624) 2019-12-05T06:50:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.347+0000 7f3b0d537700 1 -- 172.21.15.16:0/2877030741 learned_addr learned my addr 172.21.15.16:0/2877030741 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.347+0000 7f3b0cd36700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3b0808c6d0 0x7f3b0808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:40092/0 (socket says 172.21.15.16:40092) 2019-12-05T06:50:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.347+0000 7f3b07fff700 1 -- 172.21.15.16:0/2877030741 >> v1:172.21.15.76:6789/0 conn(0x7f3b0808c6d0 legacy=0x7f3b0808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b07fff700 1 -- 172.21.15.16:0/2877030741 >> v1:172.21.15.16:6789/0 conn(0x7f3b0808fef0 legacy=0x7f3b0808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.347+0000 7f3b077fe700 1 -- 172.21.15.16:0/2877030741 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860750 0 0) 0x7f3af0001150 con 0x7f3b0808fef0 2019-12-05T06:50:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b07fff700 1 -- 172.21.15.16:0/2877030741 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0808e620 con 0x7f3b0808d2c0 2019-12-05T06:50:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b07fff700 1 --2- 172.21.15.16:0/2877030741 >> v2:172.21.15.16:3300/0 conn(0x7f3b0808d2c0 0x7f3b0808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3af800af50 tx=0x7f3af800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b07fff700 1 --2- 172.21.15.16:0/2877030741 >> v2:172.21.15.16:3300/0 conn(0x7f3b0808d2c0 0x7f3b0808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3af800af50 tx=0x7f3af800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b07fff700 1 --2- 172.21.15.16:0/2877030741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0808d2c0 0x7f3b0808ef20 secure :-1 s=READY pgs=1374 cs=0 l=1 rx=0x7f3af800af50 tx=0x7f3af800b760).ready entity=mon.0 client_cookie=ea6650909cd0fdd7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b07fff700 1 --2- 172.21.15.16:0/2877030741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0808d2c0 0x7f3b0808ef20 secure :-1 s=THROTTLE_DONE pgs=1374 cs=0 l=1 rx=0x7f3af800af50 tx=0x7f3af800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b077fe700 1 -- 172.21.15.16:0/2877030741 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3af8000960 con 0x7f3b0808d2c0 2019-12-05T06:50:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b07fff700 1 --2- 172.21.15.16:0/2877030741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0808d2c0 0x7f3b0808ef20 secure :-1 s=THROTTLE_DONE pgs=1374 cs=0 l=1 rx=0x7f3af800af50 tx=0x7f3af800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b07fff700 1 --2- 172.21.15.16:0/2877030741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0808d2c0 0x7f3b0808ef20 secure :-1 s=THROTTLE_DONE pgs=1374 cs=0 l=1 rx=0x7f3af800af50 tx=0x7f3af800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b077fe700 1 -- 172.21.15.16:0/2877030741 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af8009860 con 0x7f3b0808d2c0 2019-12-05T06:50:08.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.348+0000 7f3b077fe700 1 -- 172.21.15.16:0/2877030741 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3af8009f30 con 0x7f3b0808d2c0 2019-12-05T06:50:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.349+0000 7f3b0e799700 1 -- 172.21.15.16:0/2877030741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0808d2c0 msgr2=0x7f3b0808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.349+0000 7f3b0e799700 1 --2- 172.21.15.16:0/2877030741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0808d2c0 0x7f3b0808ef20 secure :-1 s=READY pgs=1374 cs=0 l=1 rx=0x7f3af800af50 tx=0x7f3af800b760).stop 2019-12-05T06:50:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.350+0000 7f3b0e799700 1 -- 172.21.15.16:0/2877030741 shutdown_connections 2019-12-05T06:50:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.350+0000 7f3b0e799700 1 --2- 172.21.15.16:0/2877030741 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0808d2c0 0x7f3b0808ef20 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.350+0000 7f3b0e799700 1 -- 172.21.15.16:0/2877030741 shutdown_connections 2019-12-05T06:50:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.350+0000 7f3b0e799700 1 -- 172.21.15.16:0/2877030741 wait complete. 2019-12-05T06:50:08.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.350+0000 7f3b0e799700 1 -- 172.21.15.16:0/2877030741 >> 172.21.15.16:0/2877030741 conn(0x7f3b08061940 msgr2=0x7f3b08096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.355+0000 7f3b0e799700 1 Processor -- start 2019-12-05T06:50:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.355+0000 7f3b0e799700 1 -- start start 2019-12-05T06:50:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.356+0000 7f3b0e799700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3b0814c3a0 0x7f3b0809d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.356+0000 7f3b0e799700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b0806e210 0x7f3b0809daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:08.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.357+0000 7f3b0e799700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3b0809e030 0x7f3b08088fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:08.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.357+0000 7f3b0e799700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0809c620 con 0x7f3b0806e210 2019-12-05T06:50:08.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.357+0000 7f3b0e799700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b0808ada0 con 0x7f3b0814c3a0 2019-12-05T06:50:08.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.357+0000 7f3b0e799700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b0809bec0 con 0x7f3b0809e030 2019-12-05T06:50:08.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.357+0000 7f3b07fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b0806e210 0x7f3b0809daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:08.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.357+0000 7f3b07fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3b0806e210 0x7f3b0809daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33632/0 (socket says 172.21.15.16:33632) 2019-12-05T06:50:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.357+0000 7f3b07fff700 1 -- 172.21.15.16:0/4286160260 learned_addr learned my addr 172.21.15.16:0/4286160260 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.357+0000 7f3b0cd36700 1 --2- 172.21.15.16:0/4286160260 >> v2:172.21.15.76:3300/0 conn(0x7f3b0814c3a0 0x7f3b0809d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.357+0000 7f3b0d537700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3b0809e030 0x7f3b08088fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 -- 172.21.15.16:0/4286160260 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3b0809e030 msgr2=0x7f3b08088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3b0809e030 0x7f3b08088fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 -- 172.21.15.16:0/4286160260 >> v2:172.21.15.76:3300/0 conn(0x7f3b0814c3a0 msgr2=0x7f3b0809d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> v2:172.21.15.76:3300/0 conn(0x7f3b0814c3a0 0x7f3b0809d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 -- 172.21.15.16:0/4286160260 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b0809ccb0 con 0x7f3b0806e210 2019-12-05T06:50:08.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> v2:172.21.15.16:3300/0 conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> v2:172.21.15.16:3300/0 conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=READY pgs=1375 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:08.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=THROTTLE_DONE pgs=1375 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=THROTTLE_DONE pgs=1375 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=THROTTLE_DONE pgs=1375 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b057fa700 1 -- 172.21.15.16:0/4286160260 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3af8000940 con 0x7f3b0806e210 2019-12-05T06:50:08.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b057fa700 1 -- 172.21.15.16:0/4286160260 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3af8009980 con 0x7f3b0806e210 2019-12-05T06:50:08.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b0e799700 1 -- 172.21.15.16:0/4286160260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b0809ccb0 con 0x7f3b0806e210 2019-12-05T06:50:08.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.358+0000 7f3b057fa700 1 -- 172.21.15.16:0/4286160260 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3af8009fd0 con 0x7f3b0806e210 2019-12-05T06:50:08.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.359+0000 7f3b0e799700 1 -- 172.21.15.16:0/4286160260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b0809ccb0 con 0x7f3b0806e210 2019-12-05T06:50:08.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.360+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=THROTTLE_DONE pgs=1375 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.361+0000 7f3b057fa700 1 -- 172.21.15.16:0/4286160260 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3af8009b70 con 0x7f3b0806e210 2019-12-05T06:50:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.361+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=THROTTLE_DONE pgs=1375 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.361+0000 7f3b057fa700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3aec023630 0x7f3aec025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.361+0000 7f3b057fa700 1 -- 172.21.15.16:0/4286160260 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f3af8045d50 con 0x7f3b0806e210 2019-12-05T06:50:08.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.362+0000 7f3b0cd36700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3aec023630 0x7f3aec025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:08.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.362+0000 7f3b0cd36700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3aec023630 0x7f3aec025ac0 crc :-1 s=READY pgs=577 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:08.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.377+0000 7f3b0e799700 1 -- 172.21.15.16:0/4286160260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b0809ccb0 con 0x7f3b0806e210 2019-12-05T06:50:08.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.383+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=THROTTLE_DONE pgs=1375 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.383+0000 7f3b057fa700 1 -- 172.21.15.16:0/4286160260 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3af802bf60 con 0x7f3b0806e210 2019-12-05T06:50:08.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.627+0000 7f3b0e799700 1 -- 172.21.15.16:0/4286160260 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b0809ccb0 con 0x7f3b0806e210 2019-12-05T06:50:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.628+0000 7f3b07fff700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=THROTTLE_DONE pgs=1375 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:08.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.628+0000 7f3b057fa700 1 -- 172.21.15.16:0/4286160260 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f3af802bf60 con 0x7f3b0806e210 2019-12-05T06:50:08.629 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 167 2019-12-05T06:50:08.629 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:08.630 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5897,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:08.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.630+0000 7f3b0e799700 1 -- 172.21.15.16:0/4286160260 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3aec023630 msgr2=0x7f3aec025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.630+0000 7f3b0e799700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3aec023630 0x7f3aec025ac0 crc :-1 s=READY pgs=577 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.630+0000 7f3b0e799700 1 -- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 msgr2=0x7f3b0809daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.630+0000 7f3b0e799700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 0x7f3b0809daf0 secure :-1 s=READY pgs=1375 cs=0 l=1 rx=0x7f3af80103f0 tx=0x7f3af80102a0).stop 2019-12-05T06:50:08.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.631+0000 7f3b0e799700 1 -- 172.21.15.16:0/4286160260 shutdown_connections 2019-12-05T06:50:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.631+0000 7f3b0e799700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3b0809e030 0x7f3b08088fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.631+0000 7f3b0e799700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3aec023630 0x7f3aec025ac0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.631+0000 7f3b0e799700 1 --2- 172.21.15.16:0/4286160260 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3b0806e210 0x7f3b0809daf0 unknown :-1 s=CLOSED pgs=1375 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.631+0000 7f3b0e799700 1 --2- 172.21.15.16:0/4286160260 >> v2:172.21.15.76:3300/0 conn(0x7f3b0814c3a0 0x7f3b0809d5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.631+0000 7f3b0e799700 1 -- 172.21.15.16:0/4286160260 shutdown_connections 2019-12-05T06:50:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.631+0000 7f3b0e799700 1 -- 172.21.15.16:0/4286160260 wait complete. 2019-12-05T06:50:08.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:08.631+0000 7f3b0e799700 1 -- 172.21.15.16:0/4286160260 >> 172.21.15.16:0/4286160260 conn(0x7f3b08098480 msgr2=0x7f3b08071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:08.641 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:10.643 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:10.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.912+0000 7fe537d53700 1 Processor -- start 2019-12-05T06:50:10.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.912+0000 7fe537d53700 1 -- start start 2019-12-05T06:50:10.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.914+0000 7fe537d53700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe53008c6d0 0x7fe53008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:10.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.914+0000 7fe537d53700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53008fef0 0x7fe53008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:10.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.914+0000 7fe537d53700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe53008ef20 0x7fe53008f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:10.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.914+0000 7fe537d53700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe53008df90 con 0x7fe53008c6d0 2019-12-05T06:50:10.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.914+0000 7fe537d53700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5300a3f70 con 0x7fe53008ef20 2019-12-05T06:50:10.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.914+0000 7fe537d53700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe53008d830 con 0x7fe53008fef0 2019-12-05T06:50:10.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.914+0000 7fe5362f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe53008c6d0 0x7fe53008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.914+0000 7fe5362f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe53008c6d0 0x7fe53008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33642/0 (socket says 172.21.15.16:33642) 2019-12-05T06:50:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.914+0000 7fe5362f0700 1 -- 172.21.15.16:0/3590226892 learned_addr learned my addr 172.21.15.16:0/3590226892 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.914+0000 7fe536af1700 1 --2- 172.21.15.16:0/3590226892 >> v2:172.21.15.76:3300/0 conn(0x7fe53008ef20 0x7fe53008f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.915+0000 7fe535aef700 1 --2- 172.21.15.16:0/3590226892 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53008fef0 0x7fe53008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.915+0000 7fe5362f0700 1 -- 172.21.15.16:0/3590226892 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53008fef0 msgr2=0x7fe53008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:10.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.915+0000 7fe5362f0700 1 --2- 172.21.15.16:0/3590226892 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53008fef0 0x7fe53008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:10.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.915+0000 7fe5362f0700 1 -- 172.21.15.16:0/3590226892 >> v2:172.21.15.76:3300/0 conn(0x7fe53008ef20 msgr2=0x7fe53008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:10.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.915+0000 7fe5362f0700 1 --2- 172.21.15.16:0/3590226892 >> v2:172.21.15.76:3300/0 conn(0x7fe53008ef20 0x7fe53008f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:10.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.915+0000 7fe5362f0700 1 -- 172.21.15.16:0/3590226892 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe53008d540 con 0x7fe53008c6d0 2019-12-05T06:50:10.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.915+0000 7fe5362f0700 1 --2- 172.21.15.16:0/3590226892 >> v2:172.21.15.16:3300/0 conn(0x7fe53008c6d0 0x7fe53008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe52800fa40 tx=0x7fe52800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.915+0000 7fe5362f0700 1 --2- 172.21.15.16:0/3590226892 >> v2:172.21.15.16:3300/0 conn(0x7fe53008c6d0 0x7fe53008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe52800fa40 tx=0x7fe52800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.915+0000 7fe5362f0700 1 --2- 172.21.15.16:0/3590226892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe53008c6d0 0x7fe53008cac0 secure :-1 s=READY pgs=1376 cs=0 l=1 rx=0x7fe52800fa40 tx=0x7fe52800f8f0).ready entity=mon.0 client_cookie=16894b11c51a41f9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:10.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.916+0000 7fe5362f0700 1 --2- 172.21.15.16:0/3590226892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe53008c6d0 0x7fe53008cac0 secure :-1 s=THROTTLE_DONE pgs=1376 cs=0 l=1 rx=0x7fe52800fa40 tx=0x7fe52800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.916+0000 7fe5362f0700 1 --2- 172.21.15.16:0/3590226892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe53008c6d0 0x7fe53008cac0 secure :-1 s=THROTTLE_DONE pgs=1376 cs=0 l=1 rx=0x7fe52800fa40 tx=0x7fe52800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.916+0000 7fe5352ee700 1 -- 172.21.15.16:0/3590226892 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe528000ce0 con 0x7fe53008c6d0 2019-12-05T06:50:10.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.916+0000 7fe5362f0700 1 --2- 172.21.15.16:0/3590226892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe53008c6d0 0x7fe53008cac0 secure :-1 s=THROTTLE_DONE pgs=1376 cs=0 l=1 rx=0x7fe52800fa40 tx=0x7fe52800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.916+0000 7fe5352ee700 1 -- 172.21.15.16:0/3590226892 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe528001010 con 0x7fe53008c6d0 2019-12-05T06:50:10.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.916+0000 7fe5352ee700 1 -- 172.21.15.16:0/3590226892 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe528001720 con 0x7fe53008c6d0 2019-12-05T06:50:10.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.916+0000 7fe537d53700 1 -- 172.21.15.16:0/3590226892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe53008c6d0 msgr2=0x7fe53008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:10.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.916+0000 7fe537d53700 1 --2- 172.21.15.16:0/3590226892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe53008c6d0 0x7fe53008cac0 secure :-1 s=READY pgs=1376 cs=0 l=1 rx=0x7fe52800fa40 tx=0x7fe52800f8f0).stop 2019-12-05T06:50:10.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.918+0000 7fe537d53700 1 -- 172.21.15.16:0/3590226892 shutdown_connections 2019-12-05T06:50:10.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.918+0000 7fe537d53700 1 --2- 172.21.15.16:0/3590226892 >> v2:172.21.15.76:3300/0 conn(0x7fe53008ef20 0x7fe53008f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:10.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.918+0000 7fe537d53700 1 --2- 172.21.15.16:0/3590226892 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53008fef0 0x7fe53008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:10.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.918+0000 7fe537d53700 1 --2- 172.21.15.16:0/3590226892 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe53008c6d0 0x7fe53008cac0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:10.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.918+0000 7fe537d53700 1 -- 172.21.15.16:0/3590226892 shutdown_connections 2019-12-05T06:50:10.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.918+0000 7fe537d53700 1 -- 172.21.15.16:0/3590226892 wait complete. 2019-12-05T06:50:10.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.918+0000 7fe537d53700 1 -- 172.21.15.16:0/3590226892 >> 172.21.15.16:0/3590226892 conn(0x7fe530061940 msgr2=0x7fe530096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:10.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.923+0000 7fe537d53700 1 Processor -- start 2019-12-05T06:50:10.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.923+0000 7fe537d53700 1 -- start start 2019-12-05T06:50:10.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe537d53700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53009ae80 0x7fe530133220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:10.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe537d53700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe530099eb0 0x7fe53009a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:10.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe537d53700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe53006e7c0 con 0x7fe530132630 2019-12-05T06:50:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe537d53700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe530098f20 con 0x7fe530099eb0 2019-12-05T06:50:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe537d53700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe530134390 con 0x7fe53009ae80 2019-12-05T06:50:10.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe536af1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe530099eb0 0x7fe53009a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe536af1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe530099eb0 0x7fe53009a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33652/0 (socket says 172.21.15.16:33652) 2019-12-05T06:50:10.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe536af1700 1 -- 172.21.15.16:0/2991976465 learned_addr learned my addr 172.21.15.16:0/2991976465 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:10.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe535aef700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53009ae80 0x7fe530133220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe536af1700 1 -- 172.21.15.16:0/2991976465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53009ae80 msgr2=0x7fe530133220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:10.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53009ae80 0x7fe530133220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:10.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe536af1700 1 -- 172.21.15.16:0/2991976465 >> v1:172.21.15.76:6789/0 conn(0x7fe530132630 legacy=0x7fe530132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:10.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.925+0000 7fe536af1700 1 -- 172.21.15.16:0/2991976465 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5300995b0 con 0x7fe530099eb0 2019-12-05T06:50:10.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> v2:172.21.15.16:3300/0 conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> v2:172.21.15.16:3300/0 conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=READY pgs=1377 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe52effd700 1 -- 172.21.15.16:0/2991976465 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe52000f090 con 0x7fe530099eb0 2019-12-05T06:50:10.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe52effd700 1 -- 172.21.15.16:0/2991976465 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe520000a70 con 0x7fe530099eb0 2019-12-05T06:50:10.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe537d53700 1 -- 172.21.15.16:0/2991976465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5300995b0 con 0x7fe530099eb0 2019-12-05T06:50:10.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe52effd700 1 -- 172.21.15.16:0/2991976465 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe52000f670 con 0x7fe530099eb0 2019-12-05T06:50:10.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.926+0000 7fe537d53700 1 -- 172.21.15.16:0/2991976465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5300995b0 con 0x7fe530099eb0 2019-12-05T06:50:10.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.927+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.928+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.928+0000 7fe52effd700 1 -- 172.21.15.16:0/2991976465 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe52000ed50 con 0x7fe530099eb0 2019-12-05T06:50:10.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.929+0000 7fe52effd700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023630 0x7fe510025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:10.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.929+0000 7fe52effd700 1 -- 172.21.15.16:0/2991976465 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7fe520045e10 con 0x7fe530099eb0 2019-12-05T06:50:10.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.929+0000 7fe5362f0700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023630 0x7fe510025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:10.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.929+0000 7fe5362f0700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023630 0x7fe510025ac0 crc :-1 s=READY pgs=578 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:10.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.944+0000 7fe537d53700 1 -- 172.21.15.16:0/2991976465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5300995b0 con 0x7fe530099eb0 2019-12-05T06:50:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.953+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:10.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:10.953+0000 7fe52effd700 1 -- 172.21.15.16:0/2991976465 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe52000eaa0 con 0x7fe530099eb0 2019-12-05T06:50:11.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.214+0000 7fe537d53700 1 -- 172.21.15.16:0/2991976465 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe5300995b0 con 0x7fe530099eb0 2019-12-05T06:50:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.214+0000 7fe536af1700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=THROTTLE_DONE pgs=1377 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.214+0000 7fe52effd700 1 -- 172.21.15.16:0/2991976465 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fe52000ec90 con 0x7fe530099eb0 2019-12-05T06:50:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 167 2019-12-05T06:50:11.217 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:11.217 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5897,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.217+0000 7fe537d53700 1 -- 172.21.15.16:0/2991976465 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023630 msgr2=0x7fe510025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.217+0000 7fe537d53700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023630 0x7fe510025ac0 crc :-1 s=READY pgs=578 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.217+0000 7fe537d53700 1 -- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 msgr2=0x7fe53009a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.217+0000 7fe537d53700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 0x7fe53009a370 secure :-1 s=READY pgs=1377 cs=0 l=1 rx=0x7fe520010450 tx=0x7fe520010280).stop 2019-12-05T06:50:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.217+0000 7fe537d53700 1 -- 172.21.15.16:0/2991976465 shutdown_connections 2019-12-05T06:50:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.217+0000 7fe537d53700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe510023630 0x7fe510025ac0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.217+0000 7fe537d53700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe530099eb0 0x7fe53009a370 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.217+0000 7fe537d53700 1 --2- 172.21.15.16:0/2991976465 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe53009ae80 0x7fe530133220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.218+0000 7fe537d53700 1 -- 172.21.15.16:0/2991976465 shutdown_connections 2019-12-05T06:50:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.218+0000 7fe537d53700 1 -- 172.21.15.16:0/2991976465 wait complete. 2019-12-05T06:50:11.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:11.218+0000 7fe537d53700 1 -- 172.21.15.16:0/2991976465 >> 172.21.15.16:0/2991976465 conn(0x7fe5300714f0 msgr2=0x7fe53009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:11.229 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:13.232 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:13.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.501+0000 7f182a627700 1 Processor -- start 2019-12-05T06:50:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.501+0000 7f182a627700 1 -- start start 2019-12-05T06:50:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.502+0000 7f182a627700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f182408fef0 0x7f182408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:13.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.502+0000 7f182a627700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18240958e0 con 0x7f182408c6d0 2019-12-05T06:50:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.502+0000 7f182a627700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18240629e0 con 0x7f182408d2c0 2019-12-05T06:50:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.502+0000 7f182a627700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182408df90 con 0x7f182408fef0 2019-12-05T06:50:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.503+0000 7f18293c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f182408fef0 0x7f182408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.503+0000 7f18293c5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f182408fef0 0x7f182408f720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33660/0 (socket says 172.21.15.16:33660) 2019-12-05T06:50:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.503+0000 7f1828bc4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f182408c6d0 0x7f182408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47056/0 (socket says 172.21.15.16:47056) 2019-12-05T06:50:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.503+0000 7f18293c5700 1 -- 172.21.15.16:0/4011933127 learned_addr learned my addr 172.21.15.16:0/4011933127 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:13.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.503+0000 7f18293c5700 1 -- 172.21.15.16:0/4011933127 >> v1:172.21.15.76:6789/0 conn(0x7f182408d2c0 legacy=0x7f182408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:13.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.503+0000 7f18293c5700 1 -- 172.21.15.16:0/4011933127 >> v1:172.21.15.16:6789/0 conn(0x7f182408c6d0 legacy=0x7f182408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:13.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.503+0000 7f18293c5700 1 -- 172.21.15.16:0/4011933127 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f182408e620 con 0x7f182408fef0 2019-12-05T06:50:13.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.503+0000 7f18293c5700 1 --2- 172.21.15.16:0/4011933127 >> v2:172.21.15.16:3300/0 conn(0x7f182408fef0 0x7f182408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f180c00f730 tx=0x7f180c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.503+0000 7f18293c5700 1 --2- 172.21.15.16:0/4011933127 >> v2:172.21.15.16:3300/0 conn(0x7f182408fef0 0x7f182408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f180c00f730 tx=0x7f180c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.503+0000 7f18293c5700 1 --2- 172.21.15.16:0/4011933127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f182408fef0 0x7f182408f720 secure :-1 s=READY pgs=1379 cs=0 l=1 rx=0x7f180c00f730 tx=0x7f180c00fad0).ready entity=mon.0 client_cookie=d79d6429ddc3dfcc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:13.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.504+0000 7f18293c5700 1 --2- 172.21.15.16:0/4011933127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f182408fef0 0x7f182408f720 secure :-1 s=THROTTLE_DONE pgs=1379 cs=0 l=1 rx=0x7f180c00f730 tx=0x7f180c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.504+0000 7f18293c5700 1 --2- 172.21.15.16:0/4011933127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f182408fef0 0x7f182408f720 secure :-1 s=THROTTLE_DONE pgs=1379 cs=0 l=1 rx=0x7f180c00f730 tx=0x7f180c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.504+0000 7f18237fe700 1 -- 172.21.15.16:0/4011933127 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f180c0010e0 con 0x7f182408fef0 2019-12-05T06:50:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.504+0000 7f18293c5700 1 --2- 172.21.15.16:0/4011933127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f182408fef0 0x7f182408f720 secure :-1 s=THROTTLE_DONE pgs=1379 cs=0 l=1 rx=0x7f180c00f730 tx=0x7f180c00fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.504+0000 7f18237fe700 1 -- 172.21.15.16:0/4011933127 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f180c000a70 con 0x7f182408fef0 2019-12-05T06:50:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.504+0000 7f18237fe700 1 -- 172.21.15.16:0/4011933127 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f180c0016c0 con 0x7f182408fef0 2019-12-05T06:50:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.504+0000 7f182a627700 1 -- 172.21.15.16:0/4011933127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f182408fef0 msgr2=0x7f182408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:13.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.504+0000 7f182a627700 1 --2- 172.21.15.16:0/4011933127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f182408fef0 0x7f182408f720 secure :-1 s=READY pgs=1379 cs=0 l=1 rx=0x7f180c00f730 tx=0x7f180c00fad0).stop 2019-12-05T06:50:13.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.506+0000 7f182a627700 1 -- 172.21.15.16:0/4011933127 shutdown_connections 2019-12-05T06:50:13.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.506+0000 7f182a627700 1 --2- 172.21.15.16:0/4011933127 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f182408fef0 0x7f182408f720 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:13.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.506+0000 7f182a627700 1 -- 172.21.15.16:0/4011933127 shutdown_connections 2019-12-05T06:50:13.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.506+0000 7f182a627700 1 -- 172.21.15.16:0/4011933127 wait complete. 2019-12-05T06:50:13.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.506+0000 7f182a627700 1 -- 172.21.15.16:0/4011933127 >> 172.21.15.16:0/4011933127 conn(0x7f1824061940 msgr2=0x7f1824096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:13.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.510+0000 7f182a627700 1 Processor -- start 2019-12-05T06:50:13.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.510+0000 7f182a627700 1 -- start start 2019-12-05T06:50:13.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.512+0000 7f182a627700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1824088b50 0x7f1824088380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:13.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.512+0000 7f182a627700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1824152c20 0x7f1824153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.512+0000 7f182a627700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1824084ed0 con 0x7f182409e440 2019-12-05T06:50:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.512+0000 7f182a627700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f182414a7b0 con 0x7f1824088b50 2019-12-05T06:50:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.512+0000 7f182a627700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1824086bf0 con 0x7f1824152c20 2019-12-05T06:50:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.512+0000 7f1823fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1824088b50 0x7f1824088380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.513+0000 7f1823fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1824088b50 0x7f1824088380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33664/0 (socket says 172.21.15.16:33664) 2019-12-05T06:50:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.513+0000 7f1823fff700 1 -- 172.21.15.16:0/3688694718 learned_addr learned my addr 172.21.15.16:0/3688694718 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:13.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.513+0000 7f18293c5700 1 --2- 172.21.15.16:0/3688694718 >> v2:172.21.15.76:3300/0 conn(0x7f1824152c20 0x7f1824153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.513+0000 7f1823fff700 1 -- 172.21.15.16:0/3688694718 >> v2:172.21.15.76:3300/0 conn(0x7f1824152c20 msgr2=0x7f1824153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.513+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> v2:172.21.15.76:3300/0 conn(0x7f1824152c20 0x7f1824153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.513+0000 7f1823fff700 1 -- 172.21.15.16:0/3688694718 >> v1:172.21.15.76:6789/0 conn(0x7f182409e440 legacy=0x7f1824087b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.513+0000 7f1823fff700 1 -- 172.21.15.16:0/3688694718 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1824087280 con 0x7f1824088b50 2019-12-05T06:50:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.513+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> v2:172.21.15.16:3300/0 conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.513+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> v2:172.21.15.16:3300/0 conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.513+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=READY pgs=1380 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:13.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.514+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.514+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.514+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.514+0000 7f18217fa700 1 -- 172.21.15.16:0/3688694718 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1814005b40 con 0x7f1824088b50 2019-12-05T06:50:13.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.514+0000 7f18217fa700 1 -- 172.21.15.16:0/3688694718 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1814005760 con 0x7f1824088b50 2019-12-05T06:50:13.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.514+0000 7f18217fa700 1 -- 172.21.15.16:0/3688694718 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18140061c0 con 0x7f1824088b50 2019-12-05T06:50:13.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.514+0000 7f182a627700 1 -- 172.21.15.16:0/3688694718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1824087280 con 0x7f1824088b50 2019-12-05T06:50:13.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.514+0000 7f182a627700 1 -- 172.21.15.16:0/3688694718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1824087280 con 0x7f1824088b50 2019-12-05T06:50:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.515+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.515+0000 7f18217fa700 1 -- 172.21.15.16:0/3688694718 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f18140059f0 con 0x7f1824088b50 2019-12-05T06:50:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.516+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.516+0000 7f18217fa700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1808023630 0x7f1808025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.516+0000 7f18217fa700 1 -- 172.21.15.16:0/3688694718 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f1814005760 con 0x7f1824088b50 2019-12-05T06:50:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.516+0000 7f1828bc4700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1808023630 0x7f1808025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:13.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.517+0000 7f1828bc4700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1808023630 0x7f1808025ac0 crc :-1 s=READY pgs=579 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:13.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.532+0000 7f182a627700 1 -- 172.21.15.16:0/3688694718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1824087280 con 0x7f1824088b50 2019-12-05T06:50:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.538+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.539+0000 7f18217fa700 1 -- 172.21.15.16:0/3688694718 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1814007980 con 0x7f1824088b50 2019-12-05T06:50:13.763 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.762+0000 7f182a627700 1 -- 172.21.15.16:0/3688694718 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1824087280 con 0x7f1824088b50 2019-12-05T06:50:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.763+0000 7f1823fff700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=THROTTLE_DONE pgs=1380 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.763+0000 7f18217fa700 1 -- 172.21.15.16:0/3688694718 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f1814005760 con 0x7f1824088b50 2019-12-05T06:50:13.764 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 167 2019-12-05T06:50:13.765 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:13.765 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5897,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:13.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.765+0000 7f182a627700 1 -- 172.21.15.16:0/3688694718 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1808023630 msgr2=0x7f1808025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:13.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.765+0000 7f182a627700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1808023630 0x7f1808025ac0 crc :-1 s=READY pgs=579 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:13.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.765+0000 7f182a627700 1 -- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 msgr2=0x7f1824088380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:13.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.765+0000 7f182a627700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 0x7f1824088380 secure :-1 s=READY pgs=1380 cs=0 l=1 rx=0x7f1814010790 tx=0x7f1814010610).stop 2019-12-05T06:50:13.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.766+0000 7f182a627700 1 -- 172.21.15.16:0/3688694718 shutdown_connections 2019-12-05T06:50:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.766+0000 7f182a627700 1 --2- 172.21.15.16:0/3688694718 >> v2:172.21.15.76:3300/0 conn(0x7f1824152c20 0x7f1824153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.766+0000 7f182a627700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1808023630 0x7f1808025ac0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.766+0000 7f182a627700 1 --2- 172.21.15.16:0/3688694718 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1824088b50 0x7f1824088380 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.766+0000 7f182a627700 1 -- 172.21.15.16:0/3688694718 shutdown_connections 2019-12-05T06:50:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.766+0000 7f182a627700 1 -- 172.21.15.16:0/3688694718 wait complete. 2019-12-05T06:50:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:13.766+0000 7f182a627700 1 -- 172.21.15.16:0/3688694718 >> 172.21.15.16:0/3688694718 conn(0x7f1824098480 msgr2=0x7f1824071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:13.791 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:15.044 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:50:15.058 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:50:15.796 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.062+0000 7f8b30740700 1 Processor -- start 2019-12-05T06:50:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.062+0000 7f8b30740700 1 -- start start 2019-12-05T06:50:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b30740700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b2808c6d0 0x7f8b2808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b30740700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8b2808f720 0x7f8b2808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:16.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b30740700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b2808fb80 con 0x7f8b2808fef0 2019-12-05T06:50:16.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b30740700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b2808df90 con 0x7f8b2808c6d0 2019-12-05T06:50:16.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b30740700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b280a3f70 con 0x7f8b2808f720 2019-12-05T06:50:16.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b2ecdd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b2808c6d0 0x7f8b2808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:16.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b2e4dc700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8b2808fef0 0x7f8b2808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47074/0 (socket says 172.21.15.16:47074) 2019-12-05T06:50:16.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b2ecdd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b2808c6d0 0x7f8b2808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33672/0 (socket says 172.21.15.16:33672) 2019-12-05T06:50:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b2e4dc700 1 -- 172.21.15.16:0/3839540010 learned_addr learned my addr 172.21.15.16:0/3839540010 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/3839540010 >> v2:172.21.15.76:3300/0 conn(0x7f8b2808f720 0x7f8b2808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b2ecdd700 1 -- 172.21.15.16:0/3839540010 >> v2:172.21.15.76:3300/0 conn(0x7f8b2808f720 msgr2=0x7f8b2808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b2ecdd700 1 --2- 172.21.15.16:0/3839540010 >> v2:172.21.15.76:3300/0 conn(0x7f8b2808f720 0x7f8b2808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.065+0000 7f8b2ecdd700 1 -- 172.21.15.16:0/3839540010 >> v1:172.21.15.16:6789/0 conn(0x7f8b2808fef0 legacy=0x7f8b2808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b2ecdd700 1 -- 172.21.15.16:0/3839540010 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2808d540 con 0x7f8b2808c6d0 2019-12-05T06:50:16.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b2ecdd700 1 --2- 172.21.15.16:0/3839540010 >> v2:172.21.15.16:3300/0 conn(0x7f8b2808c6d0 0x7f8b2808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8b2000b2d0 tx=0x7f8b2000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b2ecdd700 1 --2- 172.21.15.16:0/3839540010 >> v2:172.21.15.16:3300/0 conn(0x7f8b2808c6d0 0x7f8b2808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8b2000b2d0 tx=0x7f8b2000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b2ecdd700 1 --2- 172.21.15.16:0/3839540010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b2808c6d0 0x7f8b2808cac0 secure :-1 s=READY pgs=1382 cs=0 l=1 rx=0x7f8b2000b2d0 tx=0x7f8b2000b5d0).ready entity=mon.0 client_cookie=c58d85578627acec server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b2ecdd700 1 --2- 172.21.15.16:0/3839540010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b2808c6d0 0x7f8b2808cac0 secure :-1 s=THROTTLE_DONE pgs=1382 cs=0 l=1 rx=0x7f8b2000b2d0 tx=0x7f8b2000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b2ecdd700 1 --2- 172.21.15.16:0/3839540010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b2808c6d0 0x7f8b2808cac0 secure :-1 s=THROTTLE_DONE pgs=1382 cs=0 l=1 rx=0x7f8b2000b2d0 tx=0x7f8b2000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b2ecdd700 1 --2- 172.21.15.16:0/3839540010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b2808c6d0 0x7f8b2808cac0 secure :-1 s=THROTTLE_DONE pgs=1382 cs=0 l=1 rx=0x7f8b2000b2d0 tx=0x7f8b2000b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b2dcdb700 1 -- 172.21.15.16:0/3839540010 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b20007590 con 0x7f8b2808c6d0 2019-12-05T06:50:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b2dcdb700 1 -- 172.21.15.16:0/3839540010 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b20009d30 con 0x7f8b2808c6d0 2019-12-05T06:50:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b2dcdb700 1 -- 172.21.15.16:0/3839540010 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b2000a400 con 0x7f8b2808c6d0 2019-12-05T06:50:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b30740700 1 -- 172.21.15.16:0/3839540010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b2808c6d0 msgr2=0x7f8b2808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:16.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.066+0000 7f8b30740700 1 --2- 172.21.15.16:0/3839540010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b2808c6d0 0x7f8b2808cac0 secure :-1 s=READY pgs=1382 cs=0 l=1 rx=0x7f8b2000b2d0 tx=0x7f8b2000b5d0).stop 2019-12-05T06:50:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.068+0000 7f8b30740700 1 -- 172.21.15.16:0/3839540010 shutdown_connections 2019-12-05T06:50:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.068+0000 7f8b30740700 1 --2- 172.21.15.16:0/3839540010 >> v2:172.21.15.76:3300/0 conn(0x7f8b2808f720 0x7f8b2808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.068+0000 7f8b30740700 1 --2- 172.21.15.16:0/3839540010 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b2808c6d0 0x7f8b2808cac0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.069+0000 7f8b30740700 1 -- 172.21.15.16:0/3839540010 shutdown_connections 2019-12-05T06:50:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.069+0000 7f8b30740700 1 -- 172.21.15.16:0/3839540010 wait complete. 2019-12-05T06:50:16.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.069+0000 7f8b30740700 1 -- 172.21.15.16:0/3839540010 >> 172.21.15.16:0/3839540010 conn(0x7f8b28061940 msgr2=0x7f8b28096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.074+0000 7f8b30740700 1 Processor -- start 2019-12-05T06:50:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.074+0000 7f8b30740700 1 -- start start 2019-12-05T06:50:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.075+0000 7f8b30740700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8b2808ac20 0x7f8b28152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:16.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.075+0000 7f8b30740700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8b28086700 0x7f8b28085730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.075+0000 7f8b30740700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b28085c70 0x7f8b2807a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:16.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.075+0000 7f8b30740700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b2807cd30 con 0x7f8b28085c70 2019-12-05T06:50:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.075+0000 7f8b30740700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b2807d000 con 0x7f8b28086700 2019-12-05T06:50:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.075+0000 7f8b30740700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b2807ddf0 con 0x7f8b2808ac20 2019-12-05T06:50:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.075+0000 7f8b2f4de700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b28085c70 0x7f8b2807a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2f4de700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b28085c70 0x7f8b2807a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33680/0 (socket says 172.21.15.16:33680) 2019-12-05T06:50:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2f4de700 1 -- 172.21.15.16:0/2726246036 learned_addr learned my addr 172.21.15.16:0/2726246036 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:16.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2e4dc700 1 --2- 172.21.15.16:0/2726246036 >> v2:172.21.15.76:3300/0 conn(0x7f8b28086700 0x7f8b28085730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2ecdd700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8b2808ac20 0x7f8b28152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2f4de700 1 -- 172.21.15.16:0/2726246036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8b2808ac20 msgr2=0x7f8b28152bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8b2808ac20 0x7f8b28152bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2f4de700 1 -- 172.21.15.16:0/2726246036 >> v2:172.21.15.76:3300/0 conn(0x7f8b28086700 msgr2=0x7f8b28085730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:16.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> v2:172.21.15.76:3300/0 conn(0x7f8b28086700 0x7f8b28085730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:16.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2f4de700 1 -- 172.21.15.16:0/2726246036 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b2807baa0 con 0x7f8b28085c70 2019-12-05T06:50:16.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> v2:172.21.15.16:3300/0 conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> v2:172.21.15.16:3300/0 conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.076+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=READY pgs=1383 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.077+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=THROTTLE_DONE pgs=1383 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.077+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=THROTTLE_DONE pgs=1383 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.077+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=THROTTLE_DONE pgs=1383 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.077+0000 7f8b277fe700 1 -- 172.21.15.16:0/2726246036 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b18000b10 con 0x7f8b28085c70 2019-12-05T06:50:16.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.077+0000 7f8b277fe700 1 -- 172.21.15.16:0/2726246036 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b1800ef40 con 0x7f8b28085c70 2019-12-05T06:50:16.102 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.077+0000 7f8b277fe700 1 -- 172.21.15.16:0/2726246036 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b1800f5d0 con 0x7f8b28085c70 2019-12-05T06:50:16.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.077+0000 7f8b30740700 1 -- 172.21.15.16:0/2726246036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b2807baa0 con 0x7f8b28085c70 2019-12-05T06:50:16.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.077+0000 7f8b30740700 1 -- 172.21.15.16:0/2726246036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b2807baa0 con 0x7f8b28085c70 2019-12-05T06:50:16.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.078+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=THROTTLE_DONE pgs=1383 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.079+0000 7f8b277fe700 1 -- 172.21.15.16:0/2726246036 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8b1800eb20 con 0x7f8b28085c70 2019-12-05T06:50:16.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.079+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=THROTTLE_DONE pgs=1383 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.079+0000 7f8b277fe700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b08023630 0x7f8b08025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:16.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.079+0000 7f8b277fe700 1 -- 172.21.15.16:0/2726246036 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 8353+0+0 (secure 0 0 0) 0x7f8b18045d40 con 0x7f8b28085c70 2019-12-05T06:50:16.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.080+0000 7f8b2ecdd700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b08023630 0x7f8b08025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:16.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.080+0000 7f8b2ecdd700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b08023630 0x7f8b08025ac0 crc :-1 s=READY pgs=580 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:16.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.093+0000 7f8b30740700 1 -- 172.21.15.16:0/2726246036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b280958e0 con 0x7f8b28085c70 2019-12-05T06:50:16.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.098+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=THROTTLE_DONE pgs=1383 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.098+0000 7f8b277fe700 1 -- 172.21.15.16:0/2726246036 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8b18000b10 con 0x7f8b28085c70 2019-12-05T06:50:16.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.353+0000 7f8b30740700 1 -- 172.21.15.16:0/2726246036 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b280958e0 con 0x7f8b28085c70 2019-12-05T06:50:16.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.353+0000 7f8b2f4de700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=THROTTLE_DONE pgs=1383 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:16.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.353+0000 7f8b277fe700 1 -- 172.21.15.16:0/2726246036 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 167 v167) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f8b1800ea80 con 0x7f8b28085c70 2019-12-05T06:50:16.354 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 167 2019-12-05T06:50:16.355 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:16.355 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":167,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5897,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":167}],"filesystems":[{"mdsmap":{"epoch":166,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:48:28.592516+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":76,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7546},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7546":{"gid":7546,"name":"b","rank":0,"incarnation":163,"state":"up:active","state_seq":40,"addr":"172.21.15.76:6835/3562825781","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":3562825781},{"type":"v1","addr":"172.21.15.76:6835","nonce":3562825781}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:16.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.356+0000 7f8b30740700 1 -- 172.21.15.16:0/2726246036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b08023630 msgr2=0x7f8b08025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:16.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.356+0000 7f8b30740700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b08023630 0x7f8b08025ac0 crc :-1 s=READY pgs=580 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:16.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.356+0000 7f8b30740700 1 -- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 msgr2=0x7f8b2807a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:16.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.356+0000 7f8b30740700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 0x7f8b2807a200 secure :-1 s=READY pgs=1383 cs=0 l=1 rx=0x7f8b18010450 tx=0x7f8b18010300).stop 2019-12-05T06:50:16.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.356+0000 7f8b30740700 1 -- 172.21.15.16:0/2726246036 shutdown_connections 2019-12-05T06:50:16.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.356+0000 7f8b30740700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b08023630 0x7f8b08025ac0 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:16.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.356+0000 7f8b30740700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b28085c70 0x7f8b2807a200 unknown :-1 s=CLOSED pgs=1383 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:16.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.356+0000 7f8b30740700 1 --2- 172.21.15.16:0/2726246036 >> v2:172.21.15.76:3300/0 conn(0x7f8b28086700 0x7f8b28085730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:16.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.356+0000 7f8b30740700 1 --2- 172.21.15.16:0/2726246036 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8b2808ac20 0x7f8b28152bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:16.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.357+0000 7f8b30740700 1 -- 172.21.15.16:0/2726246036 shutdown_connections 2019-12-05T06:50:16.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.357+0000 7f8b30740700 1 -- 172.21.15.16:0/2726246036 wait complete. 2019-12-05T06:50:16.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:16.357+0000 7f8b30740700 1 -- 172.21.15.16:0/2726246036 >> 172.21.15.16:0/2726246036 conn(0x7f8b28071fb0 msgr2=0x7f8b28084210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:16.366 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:50:18.368 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:18.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.615+0000 7fcb5741f700 1 Processor -- start 2019-12-05T06:50:18.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.615+0000 7fcb5741f700 1 -- start start 2019-12-05T06:50:18.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.616+0000 7fcb5741f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 0x7fcb5008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:18.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.616+0000 7fcb5741f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500958e0 con 0x7fcb5008f720 2019-12-05T06:50:18.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.616+0000 7fcb5741f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb500629e0 con 0x7fcb5008fef0 2019-12-05T06:50:18.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.616+0000 7fcb5741f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb500a3f70 con 0x7fcb5008c6d0 2019-12-05T06:50:18.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.617+0000 7fcb561bd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcb5008f720 0x7fcb501533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47106/0 (socket says 172.21.15.16:47106) 2019-12-05T06:50:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.617+0000 7fcb561bd700 1 -- 172.21.15.16:0/2795488300 learned_addr learned my addr 172.21.15.16:0/2795488300 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.617+0000 7fcb551bb700 1 --1- 172.21.15.16:0/2795488300 >> v1:172.21.15.76:6789/0 conn(0x7fcb5008fef0 0x7fcb5008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:40172/0 (socket says 172.21.15.16:40172) 2019-12-05T06:50:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.617+0000 7fcb559bc700 1 --2- 172.21.15.16:0/2795488300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 0x7fcb5008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:18.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.617+0000 7fcb549ba700 1 -- 172.21.15.16:0/2795488300 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258689193 0 0) 0x7fcb48001480 con 0x7fcb5008f720 2019-12-05T06:50:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.617+0000 7fcb549ba700 1 -- 172.21.15.16:0/2795488300 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb38002e80 con 0x7fcb5008f720 2019-12-05T06:50:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.617+0000 7fcb549ba700 1 -- 172.21.15.16:0/2795488300 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 242519707 0 0) 0x7fcb48001460 con 0x7fcb5008f720 2019-12-05T06:50:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.617+0000 7fcb559bc700 1 -- 172.21.15.16:0/2795488300 >> v1:172.21.15.76:6789/0 conn(0x7fcb5008fef0 legacy=0x7fcb5008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.617+0000 7fcb559bc700 1 -- 172.21.15.16:0/2795488300 >> v1:172.21.15.16:6789/0 conn(0x7fcb5008f720 legacy=0x7fcb501533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.617+0000 7fcb559bc700 1 -- 172.21.15.16:0/2795488300 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5008fbc0 con 0x7fcb5008c6d0 2019-12-05T06:50:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.618+0000 7fcb559bc700 1 --2- 172.21.15.16:0/2795488300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 0x7fcb5008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcb4000af50 tx=0x7fcb4000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.618+0000 7fcb559bc700 1 --2- 172.21.15.16:0/2795488300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 0x7fcb5008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcb4000af50 tx=0x7fcb4000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.618+0000 7fcb559bc700 1 --2- 172.21.15.16:0/2795488300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 0x7fcb5008cac0 secure :-1 s=READY pgs=319 cs=0 l=1 rx=0x7fcb4000af50 tx=0x7fcb4000b760).ready entity=mon.2 client_cookie=5433731caac838b4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.637+0000 7fcb559bc700 1 --2- 172.21.15.16:0/2795488300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 0x7fcb5008cac0 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7fcb4000af50 tx=0x7fcb4000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.638+0000 7fcb559bc700 1 --2- 172.21.15.16:0/2795488300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 0x7fcb5008cac0 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7fcb4000af50 tx=0x7fcb4000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.638+0000 7fcb549ba700 1 -- 172.21.15.16:0/2795488300 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb40000960 con 0x7fcb5008c6d0 2019-12-05T06:50:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.638+0000 7fcb559bc700 1 --2- 172.21.15.16:0/2795488300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 0x7fcb5008cac0 secure :-1 s=THROTTLE_DONE pgs=319 cs=0 l=1 rx=0x7fcb4000af50 tx=0x7fcb4000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.638+0000 7fcb549ba700 1 -- 172.21.15.16:0/2795488300 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb400098e0 con 0x7fcb5008c6d0 2019-12-05T06:50:18.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.638+0000 7fcb549ba700 1 -- 172.21.15.16:0/2795488300 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb4000a050 con 0x7fcb5008c6d0 2019-12-05T06:50:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.638+0000 7fcb5741f700 1 -- 172.21.15.16:0/2795488300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 msgr2=0x7fcb5008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.638+0000 7fcb5741f700 1 --2- 172.21.15.16:0/2795488300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 0x7fcb5008cac0 secure :-1 s=READY pgs=319 cs=0 l=1 rx=0x7fcb4000af50 tx=0x7fcb4000b760).stop 2019-12-05T06:50:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.639+0000 7fcb5741f700 1 -- 172.21.15.16:0/2795488300 shutdown_connections 2019-12-05T06:50:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.639+0000 7fcb5741f700 1 --2- 172.21.15.16:0/2795488300 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcb5008c6d0 0x7fcb5008cac0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.639+0000 7fcb5741f700 1 -- 172.21.15.16:0/2795488300 shutdown_connections 2019-12-05T06:50:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.639+0000 7fcb5741f700 1 -- 172.21.15.16:0/2795488300 wait complete. 2019-12-05T06:50:18.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.639+0000 7fcb5741f700 1 -- 172.21.15.16:0/2795488300 >> 172.21.15.16:0/2795488300 conn(0x7fcb50061940 msgr2=0x7fcb50096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.642+0000 7fcb5741f700 1 Processor -- start 2019-12-05T06:50:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.643+0000 7fcb5741f700 1 -- start start 2019-12-05T06:50:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb5741f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb5014d680 0x7fcb5014da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:18.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb5741f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcb5009dc10 0x7fcb5009d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb5741f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb5008a840 con 0x7fcb5012bce0 2019-12-05T06:50:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb5741f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb5012ad50 con 0x7fcb5014d680 2019-12-05T06:50:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb5741f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcb50099bb0 con 0x7fcb5009dc10 2019-12-05T06:50:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb559bc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb5014d680 0x7fcb5014da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:18.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb559bc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcb5014d680 0x7fcb5014da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33712/0 (socket says 172.21.15.16:33712) 2019-12-05T06:50:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb559bc700 1 -- 172.21.15.16:0/4280420687 learned_addr learned my addr 172.21.15.16:0/4280420687 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb551bb700 1 --2- 172.21.15.16:0/4280420687 >> v2:172.21.15.76:3300/0 conn(0x7fcb5009dc10 0x7fcb5009d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:18.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb559bc700 1 -- 172.21.15.16:0/4280420687 >> v2:172.21.15.76:3300/0 conn(0x7fcb5009dc10 msgr2=0x7fcb5009d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> v2:172.21.15.76:3300/0 conn(0x7fcb5009dc10 0x7fcb5009d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb559bc700 1 -- 172.21.15.16:0/4280420687 >> v1:172.21.15.16:6789/0 conn(0x7fcb5012bce0 legacy=0x7fcb50098180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb559bc700 1 -- 172.21.15.16:0/4280420687 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5009d8e0 con 0x7fcb5014d680 2019-12-05T06:50:18.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> v2:172.21.15.16:3300/0 conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.644+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> v2:172.21.15.16:3300/0 conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.645+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=READY pgs=1386 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.645+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.645+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.645+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.645+0000 7fcb4e7fc700 1 -- 172.21.15.16:0/4280420687 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb40000940 con 0x7fcb5014d680 2019-12-05T06:50:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.645+0000 7fcb4e7fc700 1 -- 172.21.15.16:0/4280420687 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb40009980 con 0x7fcb5014d680 2019-12-05T06:50:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.645+0000 7fcb5741f700 1 -- 172.21.15.16:0/4280420687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5009d8e0 con 0x7fcb5014d680 2019-12-05T06:50:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.645+0000 7fcb4e7fc700 1 -- 172.21.15.16:0/4280420687 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb4000a010 con 0x7fcb5014d680 2019-12-05T06:50:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.646+0000 7fcb5741f700 1 -- 172.21.15.16:0/4280420687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb5009d8e0 con 0x7fcb5014d680 2019-12-05T06:50:18.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.646+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.647+0000 7fcb4e7fc700 1 -- 172.21.15.16:0/4280420687 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcb40009b70 con 0x7fcb5014d680 2019-12-05T06:50:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.647+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.647+0000 7fcb4e7fc700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcb30023630 0x7fcb30025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.647+0000 7fcb4e7fc700 1 -- 172.21.15.16:0/4280420687 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7fcb40045ed0 con 0x7fcb5014d680 2019-12-05T06:50:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.647+0000 7fcb551bb700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcb30023630 0x7fcb30025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:18.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.647+0000 7fcb551bb700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcb30023630 0x7fcb30025ac0 crc :-1 s=READY pgs=581 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:18.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.679+0000 7fcb5741f700 1 -- 172.21.15.16:0/4280420687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb5012b3e0 con 0x7fcb5014d680 2019-12-05T06:50:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.687+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.687+0000 7fcb4e7fc700 1 -- 172.21.15.16:0/4280420687 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcb40009580 con 0x7fcb5014d680 2019-12-05T06:50:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.888+0000 7fcb5741f700 1 -- 172.21.15.16:0/4280420687 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcb5012b3e0 con 0x7fcb5014d680 2019-12-05T06:50:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.889+0000 7fcb559bc700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=THROTTLE_DONE pgs=1386 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.889+0000 7fcb4e7fc700 1 -- 172.21.15.16:0/4280420687 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fcb40009980 con 0x7fcb5014d680 2019-12-05T06:50:18.901 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 168 2019-12-05T06:50:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 -- 172.21.15.16:0/4280420687 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcb30023630 msgr2=0x7fcb30025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcb30023630 0x7fcb30025ac0 crc :-1 s=READY pgs=581 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 -- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 msgr2=0x7fcb5014da70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 0x7fcb5014da70 secure :-1 s=READY pgs=1386 cs=0 l=1 rx=0x7fcb40010440 tx=0x7fcb400102c0).stop 2019-12-05T06:50:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 -- 172.21.15.16:0/4280420687 shutdown_connections 2019-12-05T06:50:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 --2- 172.21.15.16:0/4280420687 >> v2:172.21.15.76:3300/0 conn(0x7fcb5009dc10 0x7fcb5009d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:18.902 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcb30023630 0x7fcb30025ac0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 --2- 172.21.15.16:0/4280420687 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcb5014d680 0x7fcb5014da70 unknown :-1 s=CLOSED pgs=1386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 -- 172.21.15.16:0/4280420687 shutdown_connections 2019-12-05T06:50:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 -- 172.21.15.16:0/4280420687 wait complete. 2019-12-05T06:50:18.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:18.892+0000 7fcb5741f700 1 -- 172.21.15.16:0/4280420687 >> 172.21.15.16:0/4280420687 conn(0x7fcb5006d260 msgr2=0x7fcb5009aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:18.903 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:18.903 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:18.528955+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:18.905 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:50:18.905 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:50:18.905 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.055+0000 7f785836b700 1 Processor -- start 2019-12-05T06:50:19.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.055+0000 7f785836b700 1 -- start start 2019-12-05T06:50:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.056+0000 7f785836b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f785008c6d0 0x7f785008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.056+0000 7f785836b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785008fef0 0x7f785008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.056+0000 7f785836b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f785008ef20 0x7f785008f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.056+0000 7f785836b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785008df90 con 0x7f785008c6d0 2019-12-05T06:50:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.056+0000 7f785836b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78500a3f70 con 0x7f785008ef20 2019-12-05T06:50:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.056+0000 7f785836b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f785008d830 con 0x7f785008fef0 2019-12-05T06:50:19.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.056+0000 7f7856908700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f785008c6d0 0x7f785008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:19.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.056+0000 7f7856908700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f785008c6d0 0x7f785008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33720/0 (socket says 172.21.15.16:33720) 2019-12-05T06:50:19.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.056+0000 7f7856908700 1 -- 172.21.15.16:0/2866118637 learned_addr learned my addr 172.21.15.16:0/2866118637 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:19.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7857109700 1 --2- 172.21.15.16:0/2866118637 >> v2:172.21.15.76:3300/0 conn(0x7f785008ef20 0x7f785008f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:19.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856107700 1 --2- 172.21.15.16:0/2866118637 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785008fef0 0x7f785008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 -- 172.21.15.16:0/2866118637 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785008fef0 msgr2=0x7f785008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 --2- 172.21.15.16:0/2866118637 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785008fef0 0x7f785008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 -- 172.21.15.16:0/2866118637 >> v2:172.21.15.76:3300/0 conn(0x7f785008ef20 msgr2=0x7f785008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 --2- 172.21.15.16:0/2866118637 >> v2:172.21.15.76:3300/0 conn(0x7f785008ef20 0x7f785008f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 -- 172.21.15.16:0/2866118637 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785008d540 con 0x7f785008c6d0 2019-12-05T06:50:19.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7857109700 1 --2- 172.21.15.16:0/2866118637 >> v2:172.21.15.76:3300/0 conn(0x7f785008ef20 0x7f785008f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:50:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 --2- 172.21.15.16:0/2866118637 >> v2:172.21.15.16:3300/0 conn(0x7f785008c6d0 0x7f785008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f784800fa40 tx=0x7f784800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 --2- 172.21.15.16:0/2866118637 >> v2:172.21.15.16:3300/0 conn(0x7f785008c6d0 0x7f785008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f784800fa40 tx=0x7f784800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 --2- 172.21.15.16:0/2866118637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f785008c6d0 0x7f785008cac0 secure :-1 s=READY pgs=1387 cs=0 l=1 rx=0x7f784800fa40 tx=0x7f784800fde0).ready entity=mon.0 client_cookie=fa5f740637fab056 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 --2- 172.21.15.16:0/2866118637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f785008c6d0 0x7f785008cac0 secure :-1 s=THROTTLE_DONE pgs=1387 cs=0 l=1 rx=0x7f784800fa40 tx=0x7f784800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7855906700 1 -- 172.21.15.16:0/2866118637 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7848000ce0 con 0x7f785008c6d0 2019-12-05T06:50:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 --2- 172.21.15.16:0/2866118637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f785008c6d0 0x7f785008cac0 secure :-1 s=THROTTLE_DONE pgs=1387 cs=0 l=1 rx=0x7f784800fa40 tx=0x7f784800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7855906700 1 -- 172.21.15.16:0/2866118637 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7848001010 con 0x7f785008c6d0 2019-12-05T06:50:19.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7856908700 1 --2- 172.21.15.16:0/2866118637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f785008c6d0 0x7f785008cac0 secure :-1 s=THROTTLE_DONE pgs=1387 cs=0 l=1 rx=0x7f784800fa40 tx=0x7f784800fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f7855906700 1 -- 172.21.15.16:0/2866118637 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7848007590 con 0x7f785008c6d0 2019-12-05T06:50:19.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f785836b700 1 -- 172.21.15.16:0/2866118637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f785008c6d0 msgr2=0x7f785008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:19.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.057+0000 7f785836b700 1 --2- 172.21.15.16:0/2866118637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f785008c6d0 0x7f785008cac0 secure :-1 s=READY pgs=1387 cs=0 l=1 rx=0x7f784800fa40 tx=0x7f784800fde0).stop 2019-12-05T06:50:19.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.058+0000 7f785836b700 1 -- 172.21.15.16:0/2866118637 shutdown_connections 2019-12-05T06:50:19.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.058+0000 7f785836b700 1 --2- 172.21.15.16:0/2866118637 >> v2:172.21.15.76:3300/0 conn(0x7f785008ef20 0x7f785008f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:19.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.058+0000 7f785836b700 1 --2- 172.21.15.16:0/2866118637 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785008fef0 0x7f785008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:19.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.058+0000 7f785836b700 1 --2- 172.21.15.16:0/2866118637 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f785008c6d0 0x7f785008cac0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:19.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.058+0000 7f785836b700 1 -- 172.21.15.16:0/2866118637 shutdown_connections 2019-12-05T06:50:19.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.058+0000 7f785836b700 1 -- 172.21.15.16:0/2866118637 wait complete. 2019-12-05T06:50:19.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.058+0000 7f785836b700 1 -- 172.21.15.16:0/2866118637 >> 172.21.15.16:0/2866118637 conn(0x7f7850061940 msgr2=0x7f7850096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:19.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.062+0000 7f785836b700 1 Processor -- start 2019-12-05T06:50:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.062+0000 7f785836b700 1 -- start start 2019-12-05T06:50:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f785836b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f785836b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7850084f30 0x7f7850155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f785836b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7850084b90 con 0x7f7850096e90 2019-12-05T06:50:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f785836b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f785009a450 con 0x7f7850084f30 2019-12-05T06:50:19.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f785836b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7850099a20 con 0x7f785009bbe0 2019-12-05T06:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f7856107700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f7857109700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7850084f30 0x7f7850155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f7857109700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7850084f30 0x7f7850155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:48050/0 (socket says 172.21.15.16:48050) 2019-12-05T06:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f7856107700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:37720/0 (socket says 172.21.15.16:37720) 2019-12-05T06:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f7856908700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7850096e90 0x7f785009b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:40196/0 (socket says 172.21.15.16:40196) 2019-12-05T06:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.063+0000 7f7857109700 1 -- 172.21.15.16:0/1296052041 learned_addr learned my addr 172.21.15.16:0/1296052041 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:19.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7856107700 1 -- 172.21.15.16:0/1296052041 >> v2:172.21.15.76:3300/0 conn(0x7f7850084f30 msgr2=0x7f7850155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> v2:172.21.15.76:3300/0 conn(0x7f7850084f30 0x7f7850155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7856107700 1 -- 172.21.15.16:0/1296052041 >> v1:172.21.15.76:6789/0 conn(0x7f7850096e90 legacy=0x7f785009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f78477fe700 1 -- 172.21.15.16:0/1296052041 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134400276 0 0) 0x7f7848000cc0 con 0x7f7850096e90 2019-12-05T06:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7856107700 1 -- 172.21.15.16:0/1296052041 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f785009aae0 con 0x7f785009bbe0 2019-12-05T06:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7857109700 1 --2- 172.21.15.16:0/1296052041 >> v2:172.21.15.76:3300/0 conn(0x7f7850084f30 0x7f7850155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:19.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=READY pgs=320 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f78477fe700 1 -- 172.21.15.16:0/1296052041 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f784c009620 con 0x7f785009bbe0 2019-12-05T06:50:19.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f785836b700 1 -- 172.21.15.16:0/1296052041 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f785009aae0 con 0x7f785009bbe0 2019-12-05T06:50:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f78477fe700 1 -- 172.21.15.16:0/1296052041 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f784c009620 con 0x7f785009bbe0 2019-12-05T06:50:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f78477fe700 1 -- 172.21.15.16:0/1296052041 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f784c009e30 con 0x7f785009bbe0 2019-12-05T06:50:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.064+0000 7f785836b700 1 -- 172.21.15.16:0/1296052041 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7850084b90 con 0x7f785009bbe0 2019-12-05T06:50:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.065+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.066+0000 7f78477fe700 1 -- 172.21.15.16:0/1296052041 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f784c0098e0 con 0x7f785009bbe0 2019-12-05T06:50:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.066+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.066+0000 7f78477fe700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7838023630 0x7f7838025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:19.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.066+0000 7f78477fe700 1 -- 172.21.15.16:0/1296052041 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f784c045d10 con 0x7f785009bbe0 2019-12-05T06:50:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.066+0000 7f7856908700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7838023630 0x7f7838025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.067+0000 7f7856908700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7838023630 0x7f7838025ac0 crc :-1 s=READY pgs=582 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:19.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.071+0000 7f785836b700 1 -- 172.21.15.16:0/1296052041 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f785009aae0 con 0x7f785009bbe0 2019-12-05T06:50:19.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.078+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.078+0000 7f78477fe700 1 -- 172.21.15.16:0/1296052041 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f784c000980 con 0x7f785009bbe0 2019-12-05T06:50:19.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.287+0000 7f785836b700 1 -- 172.21.15.16:0/1296052041 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f785009aae0 con 0x7f785009bbe0 2019-12-05T06:50:19.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.288+0000 7f7856107700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=THROTTLE_DONE pgs=320 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:19.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.288+0000 7f78477fe700 1 -- 172.21.15.16:0/1296052041 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 168 v168) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f784c009810 con 0x7f785009bbe0 2019-12-05T06:50:19.302 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 168 2019-12-05T06:50:19.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 -- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7838023630 msgr2=0x7f7838025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:19.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7838023630 0x7f7838025ac0 crc :-1 s=READY pgs=582 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:19.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 -- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 msgr2=0x7f78501539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:19.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 secure :-1 s=READY pgs=320 cs=0 l=1 rx=0x7f784c0103a0 tx=0x7f784c010740).stop 2019-12-05T06:50:19.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 -- 172.21.15.16:0/1296052041 shutdown_connections 2019-12-05T06:50:19.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7838023630 0x7f7838025ac0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:19.303 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 --2- 172.21.15.16:0/1296052041 >> v2:172.21.15.76:3300/0 conn(0x7f7850084f30 0x7f7850155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:19.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 --2- 172.21.15.16:0/1296052041 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f785009bbe0 0x7f78501539e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:19.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 -- 172.21.15.16:0/1296052041 shutdown_connections 2019-12-05T06:50:19.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 -- 172.21.15.16:0/1296052041 wait complete. 2019-12-05T06:50:19.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:19.294+0000 7f785836b700 1 -- 172.21.15.16:0/1296052041 >> 172.21.15.16:0/1296052041 conn(0x7f78500714f0 msgr2=0x7f785009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:19.304 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:19.305 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":168,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":168,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:18.528955+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:19.314 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 168, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 168, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5897": { "addr": "172.21.15.16:6833/321778108", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 321778108, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 321778108, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5897, "incarnation": 168, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 77, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:50:18.528955+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5897 } } } ], "standbys": [] } 2019-12-05T06:50:21.315 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:21.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.582+0000 7f44f7d1c700 1 Processor -- start 2019-12-05T06:50:21.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.582+0000 7f44f7d1c700 1 -- start start 2019-12-05T06:50:21.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.584+0000 7f44f7d1c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f008c6d0 0x7f44f008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:21.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.585+0000 7f44f7d1c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f44f008f720 0x7f44f008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.585+0000 7f44f7d1c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f00958e0 con 0x7f44f008fef0 2019-12-05T06:50:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.585+0000 7f44f7d1c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f00a3f70 con 0x7f44f008f720 2019-12-05T06:50:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.585+0000 7f44f7d1c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f008d830 con 0x7f44f008c6d0 2019-12-05T06:50:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.585+0000 7f44f6aba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f44f008f720 0x7f44f008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.585+0000 7f44f5ab8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f44f008fef0 0x7f44f008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47138/0 (socket says 172.21.15.16:47138) 2019-12-05T06:50:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.585+0000 7f44f5ab8700 1 -- 172.21.15.16:0/3105559082 learned_addr learned my addr 172.21.15.16:0/3105559082 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:21.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.585+0000 7f44f6aba700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f44f008f720 0x7f44f008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33740/0 (socket says 172.21.15.16:33740) 2019-12-05T06:50:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.585+0000 7f44f62b9700 1 --2- 172.21.15.16:0/3105559082 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f008c6d0 0x7f44f008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f6aba700 1 -- 172.21.15.16:0/3105559082 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f008c6d0 msgr2=0x7f44f008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f6aba700 1 --2- 172.21.15.16:0/3105559082 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f008c6d0 0x7f44f008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f6aba700 1 -- 172.21.15.16:0/3105559082 >> v1:172.21.15.16:6789/0 conn(0x7f44f008fef0 legacy=0x7f44f008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f52b7700 1 -- 172.21.15.16:0/3105559082 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2308250805 0 0) 0x7f44e0001150 con 0x7f44f008fef0 2019-12-05T06:50:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f6aba700 1 -- 172.21.15.16:0/3105559082 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f008fbc0 con 0x7f44f008f720 2019-12-05T06:50:21.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f6aba700 1 --2- 172.21.15.16:0/3105559082 >> v2:172.21.15.16:3300/0 conn(0x7f44f008f720 0x7f44f008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f44e800b2d0 tx=0x7f44e800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f6aba700 1 --2- 172.21.15.16:0/3105559082 >> v2:172.21.15.16:3300/0 conn(0x7f44f008f720 0x7f44f008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f44e800b2d0 tx=0x7f44e800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f6aba700 1 --2- 172.21.15.16:0/3105559082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f008f720 0x7f44f008d000 secure :-1 s=READY pgs=1389 cs=0 l=1 rx=0x7f44e800b2d0 tx=0x7f44e800ba60).ready entity=mon.0 client_cookie=57e4248726a5466c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f6aba700 1 --2- 172.21.15.16:0/3105559082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f008f720 0x7f44f008d000 secure :-1 s=THROTTLE_DONE pgs=1389 cs=0 l=1 rx=0x7f44e800b2d0 tx=0x7f44e800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f52b7700 1 -- 172.21.15.16:0/3105559082 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e8009f20 con 0x7f44f008f720 2019-12-05T06:50:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f6aba700 1 --2- 172.21.15.16:0/3105559082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f008f720 0x7f44f008d000 secure :-1 s=THROTTLE_DONE pgs=1389 cs=0 l=1 rx=0x7f44e800b2d0 tx=0x7f44e800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.586+0000 7f44f6aba700 1 --2- 172.21.15.16:0/3105559082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f008f720 0x7f44f008d000 secure :-1 s=THROTTLE_DONE pgs=1389 cs=0 l=1 rx=0x7f44e800b2d0 tx=0x7f44e800ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.587+0000 7f44f52b7700 1 -- 172.21.15.16:0/3105559082 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e8009b40 con 0x7f44f008f720 2019-12-05T06:50:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.587+0000 7f44f52b7700 1 -- 172.21.15.16:0/3105559082 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e8009d30 con 0x7f44f008f720 2019-12-05T06:50:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.587+0000 7f44f7d1c700 1 -- 172.21.15.16:0/3105559082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f008f720 msgr2=0x7f44f008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.587+0000 7f44f7d1c700 1 --2- 172.21.15.16:0/3105559082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f008f720 0x7f44f008d000 secure :-1 s=READY pgs=1389 cs=0 l=1 rx=0x7f44e800b2d0 tx=0x7f44e800ba60).stop 2019-12-05T06:50:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.589+0000 7f44f7d1c700 1 -- 172.21.15.16:0/3105559082 shutdown_connections 2019-12-05T06:50:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.589+0000 7f44f7d1c700 1 --2- 172.21.15.16:0/3105559082 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f44f008f720 0x7f44f008d000 unknown :-1 s=CLOSED pgs=1389 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.589+0000 7f44f7d1c700 1 --2- 172.21.15.16:0/3105559082 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f008c6d0 0x7f44f008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:21.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.589+0000 7f44f7d1c700 1 -- 172.21.15.16:0/3105559082 shutdown_connections 2019-12-05T06:50:21.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.589+0000 7f44f7d1c700 1 -- 172.21.15.16:0/3105559082 wait complete. 2019-12-05T06:50:21.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.589+0000 7f44f7d1c700 1 -- 172.21.15.16:0/3105559082 >> 172.21.15.16:0/3105559082 conn(0x7f44f0061940 msgr2=0x7f44f0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.594+0000 7f44f7d1c700 1 Processor -- start 2019-12-05T06:50:21.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.594+0000 7f44f7d1c700 1 -- start start 2019-12-05T06:50:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.596+0000 7f44f7d1c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f0096e90 0x7f44f009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.596+0000 7f44f7d1c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f44f0084f30 0x7f44f009b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.596+0000 7f44f7d1c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44f0084b90 con 0x7f44f0097280 2019-12-05T06:50:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.596+0000 7f44f7d1c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44f0099a20 con 0x7f44f0084f30 2019-12-05T06:50:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.596+0000 7f44f7d1c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44f0099cf0 con 0x7f44f0096e90 2019-12-05T06:50:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.596+0000 7f44f6aba700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f44f0097280 0x7f44f0097700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47146/0 (socket says 172.21.15.16:47146) 2019-12-05T06:50:21.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.596+0000 7f44f6aba700 1 -- 172.21.15.16:0/513486902 learned_addr learned my addr 172.21.15.16:0/513486902 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.596+0000 7f44f62b9700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f0096e90 0x7f44f009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.596+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> v2:172.21.15.76:3300/0 conn(0x7f44f0084f30 0x7f44f009b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.597+0000 7f44e6ffd700 1 -- 172.21.15.16:0/513486902 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1806370259 0 0) 0x7f44e8000ed0 con 0x7f44f0097280 2019-12-05T06:50:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.597+0000 7f44e6ffd700 1 -- 172.21.15.16:0/513486902 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44d8002e80 con 0x7f44f0097280 2019-12-05T06:50:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.597+0000 7f44f5ab8700 1 -- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f0096e90 msgr2=0x7f44f009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.597+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f0096e90 0x7f44f009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:21.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.597+0000 7f44f5ab8700 1 -- 172.21.15.16:0/513486902 >> v1:172.21.15.16:6789/0 conn(0x7f44f0097280 legacy=0x7f44f0097700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.597+0000 7f44f5ab8700 1 -- 172.21.15.16:0/513486902 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f009aae0 con 0x7f44f0084f30 2019-12-05T06:50:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.597+0000 7f44f62b9700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f0096e90 0x7f44f009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.597+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> v2:172.21.15.76:3300/0 conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.597+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> v2:172.21.15.76:3300/0 conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.597+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=READY pgs=941 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:21.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.598+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=THROTTLE_DONE pgs=941 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.598+0000 7f44e6ffd700 1 -- 172.21.15.16:0/513486902 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e0000ef0 con 0x7f44f0084f30 2019-12-05T06:50:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.598+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=THROTTLE_DONE pgs=941 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.598+0000 7f44e6ffd700 1 -- 172.21.15.16:0/513486902 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e0004240 con 0x7f44f0084f30 2019-12-05T06:50:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.598+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=THROTTLE_DONE pgs=941 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.598+0000 7f44f7d1c700 1 -- 172.21.15.16:0/513486902 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f009aae0 con 0x7f44f0084f30 2019-12-05T06:50:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.598+0000 7f44e6ffd700 1 -- 172.21.15.16:0/513486902 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e0001200 con 0x7f44f0084f30 2019-12-05T06:50:21.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.598+0000 7f44f7d1c700 1 -- 172.21.15.16:0/513486902 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f009aae0 con 0x7f44f0084f30 2019-12-05T06:50:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.599+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=THROTTLE_DONE pgs=941 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.599+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=THROTTLE_DONE pgs=941 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.599+0000 7f44e6ffd700 1 -- 172.21.15.16:0/513486902 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f44e0004430 con 0x7f44f0084f30 2019-12-05T06:50:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.600+0000 7f44e6ffd700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f44d8023670 0x7f44d8025b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:21.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.600+0000 7f44e6ffd700 1 -- 172.21.15.16:0/513486902 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f44e0045f70 con 0x7f44f0084f30 2019-12-05T06:50:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.600+0000 7f44f62b9700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f44d8023670 0x7f44d8025b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:21.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.601+0000 7f44f62b9700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f44d8023670 0x7f44d8025b00 crc :-1 s=READY pgs=583 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:21.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.614+0000 7f44f7d1c700 1 -- 172.21.15.16:0/513486902 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f009aae0 con 0x7f44f0084f30 2019-12-05T06:50:21.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.623+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=THROTTLE_DONE pgs=941 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.623+0000 7f44e6ffd700 1 -- 172.21.15.16:0/513486902 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f44e0000c90 con 0x7f44f0084f30 2019-12-05T06:50:21.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.869+0000 7f44f7d1c700 1 -- 172.21.15.16:0/513486902 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f44f009aae0 con 0x7f44f0084f30 2019-12-05T06:50:21.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.871+0000 7f44f5ab8700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=THROTTLE_DONE pgs=941 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:21.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.871+0000 7f44e6ffd700 1 -- 172.21.15.16:0/513486902 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 170 v170) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f44e0004240 con 0x7f44f0084f30 2019-12-05T06:50:21.872 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:21.873 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":170,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":170,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:21.724515+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:rejoin","state_seq":30,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:21.873 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 170 2019-12-05T06:50:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 -- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f44d8023670 msgr2=0x7f44d8025b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f44d8023670 0x7f44d8025b00 crc :-1 s=READY pgs=583 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 -- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 msgr2=0x7f44f009b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 0x7f44f009b920 secure :-1 s=READY pgs=941 cs=0 l=1 rx=0x7f44e0010370 tx=0x7f44e0010710).stop 2019-12-05T06:50:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 -- 172.21.15.16:0/513486902 shutdown_connections 2019-12-05T06:50:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f44d8023670 0x7f44d8025b00 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f44f0084f30 0x7f44f009b920 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 --2- 172.21.15.16:0/513486902 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f44f0096e90 0x7f44f009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 -- 172.21.15.16:0/513486902 shutdown_connections 2019-12-05T06:50:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 -- 172.21.15.16:0/513486902 wait complete. 2019-12-05T06:50:21.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:21.875+0000 7f44f7d1c700 1 -- 172.21.15.16:0/513486902 >> 172.21.15.16:0/513486902 conn(0x7f44f00714f0 msgr2=0x7f44f009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:21.885 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:50:23.886 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.053+0000 7f318b723700 1 Processor -- start 2019-12-05T06:50:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.054+0000 7f318b723700 1 -- start start 2019-12-05T06:50:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.055+0000 7f318b723700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f318408c6d0 0x7f318408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.055+0000 7f318b723700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:24.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.055+0000 7f318b723700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31840958e0 con 0x7f318408ef20 2019-12-05T06:50:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.055+0000 7f318b723700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f318408df90 con 0x7f318408c6d0 2019-12-05T06:50:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.055+0000 7f318b723700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31840a3f70 con 0x7f318408fef0 2019-12-05T06:50:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.055+0000 7f31894bf700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f3189cc0700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f318408c6d0 0x7f318408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f318a4c1700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f318408ef20 0x7f318408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:40232/0 (socket says 172.21.15.16:40232) 2019-12-05T06:50:24.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f31894bf700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:37756/0 (socket says 172.21.15.16:37756) 2019-12-05T06:50:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f318a4c1700 1 -- 172.21.15.16:0/178967581 learned_addr learned my addr 172.21.15.16:0/178967581 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f3188cbe700 1 -- 172.21.15.16:0/178967581 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 143452814 0 0) 0x7f317c000d30 con 0x7f318408ef20 2019-12-05T06:50:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f3188cbe700 1 -- 172.21.15.16:0/178967581 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3170002e80 con 0x7f318408ef20 2019-12-05T06:50:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f31894bf700 1 -- 172.21.15.16:0/178967581 >> v2:172.21.15.76:3300/0 conn(0x7f318408c6d0 msgr2=0x7f318408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f31894bf700 1 --2- 172.21.15.16:0/178967581 >> v2:172.21.15.76:3300/0 conn(0x7f318408c6d0 0x7f318408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f31894bf700 1 -- 172.21.15.16:0/178967581 >> v1:172.21.15.76:6789/0 conn(0x7f318408ef20 legacy=0x7f318408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:24.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f31894bf700 1 -- 172.21.15.16:0/178967581 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318408d540 con 0x7f318408fef0 2019-12-05T06:50:24.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.056+0000 7f3189cc0700 1 --2- 172.21.15.16:0/178967581 >> v2:172.21.15.76:3300/0 conn(0x7f318408c6d0 0x7f318408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:24.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f31894bf700 1 --2- 172.21.15.16:0/178967581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f317400af50 tx=0x7f317400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f31894bf700 1 --2- 172.21.15.16:0/178967581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f317400af50 tx=0x7f317400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f31894bf700 1 --2- 172.21.15.16:0/178967581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 secure :-1 s=READY pgs=321 cs=0 l=1 rx=0x7f317400af50 tx=0x7f317400b760).ready entity=mon.2 client_cookie=e0cab075b2e6db2e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:24.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f31894bf700 1 --2- 172.21.15.16:0/178967581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7f317400af50 tx=0x7f317400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f3188cbe700 1 -- 172.21.15.16:0/178967581 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3174007210 con 0x7f318408fef0 2019-12-05T06:50:24.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f31894bf700 1 --2- 172.21.15.16:0/178967581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7f317400af50 tx=0x7f317400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f31894bf700 1 --2- 172.21.15.16:0/178967581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 secure :-1 s=THROTTLE_DONE pgs=321 cs=0 l=1 rx=0x7f317400af50 tx=0x7f317400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f3188cbe700 1 -- 172.21.15.16:0/178967581 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31740099b0 con 0x7f318408fef0 2019-12-05T06:50:24.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f3188cbe700 1 -- 172.21.15.16:0/178967581 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f317400a080 con 0x7f318408fef0 2019-12-05T06:50:24.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f318b723700 1 -- 172.21.15.16:0/178967581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 msgr2=0x7f318408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:24.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.057+0000 7f318b723700 1 --2- 172.21.15.16:0/178967581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 secure :-1 s=READY pgs=321 cs=0 l=1 rx=0x7f317400af50 tx=0x7f317400b760).stop 2019-12-05T06:50:24.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.058+0000 7f318b723700 1 -- 172.21.15.16:0/178967581 shutdown_connections 2019-12-05T06:50:24.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.058+0000 7f318b723700 1 --2- 172.21.15.16:0/178967581 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f318408fef0 0x7f318408d000 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:24.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.058+0000 7f318b723700 1 --2- 172.21.15.16:0/178967581 >> v2:172.21.15.76:3300/0 conn(0x7f318408c6d0 0x7f318408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:24.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.059+0000 7f318b723700 1 -- 172.21.15.16:0/178967581 shutdown_connections 2019-12-05T06:50:24.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.059+0000 7f318b723700 1 -- 172.21.15.16:0/178967581 wait complete. 2019-12-05T06:50:24.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.059+0000 7f318b723700 1 -- 172.21.15.16:0/178967581 >> 172.21.15.16:0/178967581 conn(0x7f3184061940 msgr2=0x7f3184096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.063+0000 7f318b723700 1 Processor -- start 2019-12-05T06:50:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.063+0000 7f318b723700 1 -- start start 2019-12-05T06:50:24.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.064+0000 7f318b723700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.064+0000 7f318b723700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3184128b20 con 0x7f3184154c60 2019-12-05T06:50:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.064+0000 7f318b723700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3184128660 con 0x7f318412c740 2019-12-05T06:50:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.064+0000 7f318b723700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3184129db0 con 0x7f31841325e0 2019-12-05T06:50:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.064+0000 7f318a4c1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3184154c60 0x7f3184157060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47170/0 (socket says 172.21.15.16:47170) 2019-12-05T06:50:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.064+0000 7f318a4c1700 1 -- 172.21.15.16:0/3038116966 learned_addr learned my addr 172.21.15.16:0/3038116966 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.065+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:24.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.065+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721964364 0 0) 0x7f317c001480 con 0x7f3184154c60 2019-12-05T06:50:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.065+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f316c002e80 con 0x7f3184154c60 2019-12-05T06:50:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.065+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137228882 0 0) 0x7f317400adc0 con 0x7f318412c740 2019-12-05T06:50:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.065+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f316c0030b0 con 0x7f318412c740 2019-12-05T06:50:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.065+0000 7f3189cc0700 1 -- 172.21.15.16:0/3038116966 >> v1:172.21.15.76:6789/0 conn(0x7f318412c740 legacy=0x7f318412bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.065+0000 7f3189cc0700 1 -- 172.21.15.16:0/3038116966 >> v1:172.21.15.16:6789/0 conn(0x7f3184154c60 legacy=0x7f3184157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:24.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.065+0000 7f3189cc0700 1 -- 172.21.15.16:0/3038116966 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f318412ae70 con 0x7f31841325e0 2019-12-05T06:50:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.065+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.066+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.066+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=READY pgs=322 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.066+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.066+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3180000940 con 0x7f31841325e0 2019-12-05T06:50:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.066+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.066+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.066+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f318000dff0 con 0x7f31841325e0 2019-12-05T06:50:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.066+0000 7f318b723700 1 -- 172.21.15.16:0/3038116966 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f318412ae70 con 0x7f31841325e0 2019-12-05T06:50:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.066+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3180000940 con 0x7f31841325e0 2019-12-05T06:50:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.066+0000 7f318b723700 1 -- 172.21.15.16:0/3038116966 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f318412ae70 con 0x7f31841325e0 2019-12-05T06:50:24.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.067+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.068+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f318000db50 con 0x7f31841325e0 2019-12-05T06:50:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.068+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.068+0000 7f317a7fc700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f316c023650 0x7f316c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.068+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f3180045de0 con 0x7f31841325e0 2019-12-05T06:50:24.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.068+0000 7f31894bf700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f316c023650 0x7f316c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.069+0000 7f318b723700 1 -- 172.21.15.16:0/3038116966 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f318412ae70 con 0x7f31841325e0 2019-12-05T06:50:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.069+0000 7f31894bf700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f316c023650 0x7f316c025ae0 crc :-1 s=READY pgs=584 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:24.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.079+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.079+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3180000940 con 0x7f31841325e0 2019-12-05T06:50:24.312 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.312+0000 7f318b723700 1 -- 172.21.15.16:0/3038116966 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f318412ae70 con 0x7f31841325e0 2019-12-05T06:50:24.313 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.312+0000 7f3189cc0700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=THROTTLE_DONE pgs=322 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.313+0000 7f317a7fc700 1 -- 172.21.15.16:0/3038116966 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f318000da60 con 0x7f31841325e0 2019-12-05T06:50:24.314 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 171 2019-12-05T06:50:24.314 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:24.314 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:24.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.315+0000 7f318b723700 1 -- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f316c023650 msgr2=0x7f316c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.315+0000 7f318b723700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f316c023650 0x7f316c025ae0 crc :-1 s=READY pgs=584 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.315+0000 7f318b723700 1 -- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 msgr2=0x7f318412b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.315+0000 7f318b723700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 secure :-1 s=READY pgs=322 cs=0 l=1 rx=0x7f31800103a0 tx=0x7f3180010740).stop 2019-12-05T06:50:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.316+0000 7f318b723700 1 -- 172.21.15.16:0/3038116966 shutdown_connections 2019-12-05T06:50:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.316+0000 7f318b723700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f316c023650 0x7f316c025ae0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:24.317 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.316+0000 7f318b723700 1 --2- 172.21.15.16:0/3038116966 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31841325e0 0x7f318412b770 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.316+0000 7f318b723700 1 -- 172.21.15.16:0/3038116966 shutdown_connections 2019-12-05T06:50:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.316+0000 7f318b723700 1 -- 172.21.15.16:0/3038116966 wait complete. 2019-12-05T06:50:24.318 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:24.316+0000 7f318b723700 1 -- 172.21.15.16:0/3038116966 >> 172.21.15.16:0/3038116966 conn(0x7f31840a2850 msgr2=0x7f318406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:24.329 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=7546 2019-12-05T06:50:24.329 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.b 2019-12-05T06:50:27.330 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:50:27.330 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:50:27.331 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:50:27.335 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:50:27.336 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:27.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.528+0000 7f1312403700 1 Processor -- start 2019-12-05T06:50:27.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.528+0000 7f1312403700 1 -- start start 2019-12-05T06:50:27.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.528+0000 7f1312403700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c08c6d0 0x7f130c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:27.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.528+0000 7f1312403700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f130c08f720 0x7f130c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:27.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.528+0000 7f1312403700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c0958e0 con 0x7f130c08fef0 2019-12-05T06:50:27.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.529+0000 7f1312403700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130c0a3f70 con 0x7f130c08f720 2019-12-05T06:50:27.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.529+0000 7f1312403700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c08d830 con 0x7f130c08c6d0 2019-12-05T06:50:27.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.529+0000 7f13111a1700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f130c08f720 0x7f130c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:27.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.529+0000 7f13111a1700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f130c08f720 0x7f130c08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:48110/0 (socket says 172.21.15.16:48110) 2019-12-05T06:50:27.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.529+0000 7f130bfff700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f130c08fef0 0x7f130c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:40260/0 (socket says 172.21.15.16:40260) 2019-12-05T06:50:27.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.529+0000 7f13111a1700 1 -- 172.21.15.16:0/2066852769 learned_addr learned my addr 172.21.15.16:0/2066852769 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:27.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.529+0000 7f13109a0700 1 --2- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c08c6d0 0x7f130c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:27.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f13111a1700 1 -- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c08c6d0 msgr2=0x7f130c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:27.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f13111a1700 1 --2- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c08c6d0 0x7f130c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:27.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f13111a1700 1 -- 172.21.15.16:0/2066852769 >> v1:172.21.15.76:6789/0 conn(0x7f130c08fef0 legacy=0x7f130c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f13111a1700 1 -- 172.21.15.16:0/2066852769 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c08fbc0 con 0x7f130c08f720 2019-12-05T06:50:27.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f13111a1700 1 --2- 172.21.15.16:0/2066852769 >> v2:172.21.15.76:3300/0 conn(0x7f130c08f720 0x7f130c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f12f400f760 tx=0x7f12f400f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f13111a1700 1 --2- 172.21.15.16:0/2066852769 >> v2:172.21.15.76:3300/0 conn(0x7f130c08f720 0x7f130c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f12f400f760 tx=0x7f12f400f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f13111a1700 1 --2- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f130c08f720 0x7f130c08d000 secure :-1 s=READY pgs=945 cs=0 l=1 rx=0x7f12f400f760 tx=0x7f12f400f590).ready entity=mon.1 client_cookie=d1c4ff2ef18a383f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:27.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f13111a1700 1 --2- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f130c08f720 0x7f130c08d000 secure :-1 s=THROTTLE_DONE pgs=945 cs=0 l=1 rx=0x7f12f400f760 tx=0x7f12f400f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f13111a1700 1 --2- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f130c08f720 0x7f130c08d000 secure :-1 s=THROTTLE_DONE pgs=945 cs=0 l=1 rx=0x7f12f400f760 tx=0x7f12f400f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f130b7fe700 1 -- 172.21.15.16:0/2066852769 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12f4000a70 con 0x7f130c08f720 2019-12-05T06:50:27.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f13111a1700 1 --2- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f130c08f720 0x7f130c08d000 secure :-1 s=THROTTLE_DONE pgs=945 cs=0 l=1 rx=0x7f12f400f760 tx=0x7f12f400f590).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f130b7fe700 1 -- 172.21.15.16:0/2066852769 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f4000e70 con 0x7f130c08f720 2019-12-05T06:50:27.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f130b7fe700 1 -- 172.21.15.16:0/2066852769 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12f4001500 con 0x7f130c08f720 2019-12-05T06:50:27.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f1312403700 1 -- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f130c08f720 msgr2=0x7f130c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.530+0000 7f1312403700 1 --2- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f130c08f720 0x7f130c08d000 secure :-1 s=READY pgs=945 cs=0 l=1 rx=0x7f12f400f760 tx=0x7f12f400f590).stop 2019-12-05T06:50:27.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.532+0000 7f1312403700 1 -- 172.21.15.16:0/2066852769 shutdown_connections 2019-12-05T06:50:27.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.532+0000 7f1312403700 1 --2- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f130c08f720 0x7f130c08d000 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.532+0000 7f1312403700 1 --2- 172.21.15.16:0/2066852769 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c08c6d0 0x7f130c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.532+0000 7f1312403700 1 -- 172.21.15.16:0/2066852769 shutdown_connections 2019-12-05T06:50:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.532+0000 7f1312403700 1 -- 172.21.15.16:0/2066852769 wait complete. 2019-12-05T06:50:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.532+0000 7f1312403700 1 -- 172.21.15.16:0/2066852769 >> 172.21.15.16:0/2066852769 conn(0x7f130c061940 msgr2=0x7f130c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.535+0000 7f1312403700 1 Processor -- start 2019-12-05T06:50:27.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.535+0000 7f1312403700 1 -- start start 2019-12-05T06:50:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.536+0000 7f1312403700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.536+0000 7f1312403700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c0709e0 con 0x7f130c0813b0 2019-12-05T06:50:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.536+0000 7f1312403700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f130c07e6d0 con 0x7f130c081870 2019-12-05T06:50:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.536+0000 7f1312403700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130c07f4b0 con 0x7f130c152fa0 2019-12-05T06:50:27.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.536+0000 7f130bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f130c0813b0 0x7f130c06ed60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47198/0 (socket says 172.21.15.16:47198) 2019-12-05T06:50:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.536+0000 7f130bfff700 1 -- 172.21.15.16:0/2396294868 learned_addr learned my addr 172.21.15.16:0/2396294868 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.536+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 316348081 0 0) 0x7f12fc001480 con 0x7f130c0813b0 2019-12-05T06:50:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f0002e80 con 0x7f130c0813b0 2019-12-05T06:50:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866821 0 0) 0x7f12f4007e40 con 0x7f130c081870 2019-12-05T06:50:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12f00041b0 con 0x7f130c081870 2019-12-05T06:50:27.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13109a0700 1 -- 172.21.15.16:0/2396294868 >> v1:172.21.15.76:6789/0 conn(0x7f130c081870 legacy=0x7f130c06f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13109a0700 1 -- 172.21.15.16:0/2396294868 >> v1:172.21.15.16:6789/0 conn(0x7f130c0813b0 legacy=0x7f130c06ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13109a0700 1 -- 172.21.15.16:0/2396294868 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f130c080570 con 0x7f130c152fa0 2019-12-05T06:50:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=READY pgs=324 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1304009640 con 0x7f130c152fa0 2019-12-05T06:50:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.537+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.538+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1304000940 con 0x7f130c152fa0 2019-12-05T06:50:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.538+0000 7f1312403700 1 -- 172.21.15.16:0/2396294868 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130c080570 con 0x7f130c152fa0 2019-12-05T06:50:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.538+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f130400a0e0 con 0x7f130c152fa0 2019-12-05T06:50:27.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.538+0000 7f1312403700 1 -- 172.21.15.16:0/2396294868 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f130c080570 con 0x7f130c152fa0 2019-12-05T06:50:27.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.539+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.539+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.539+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1304009a70 con 0x7f130c152fa0 2019-12-05T06:50:27.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.539+0000 7f13097fa700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f12f0023650 0x7f12f0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:27.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.539+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f1304046030 con 0x7f130c152fa0 2019-12-05T06:50:27.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.540+0000 7f130bfff700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f12f0023650 0x7f12f0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:27.551 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:50:27.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.540+0000 7f130bfff700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f12f0023650 0x7f12f0025ae0 crc :-1 s=READY pgs=585 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:27.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.543+0000 7f1312403700 1 -- 172.21.15.16:0/2396294868 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130c080570 con 0x7f130c152fa0 2019-12-05T06:50:27.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.549+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.549+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1304009840 con 0x7f130c152fa0 2019-12-05T06:50:27.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.770+0000 7f1312403700 1 -- 172.21.15.16:0/2396294868 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f130c080570 con 0x7f130c152fa0 2019-12-05T06:50:27.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.771+0000 7f13109a0700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=THROTTLE_DONE pgs=324 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:27.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.771+0000 7f13097fa700 1 -- 172.21.15.16:0/2396294868 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f1304009a50 con 0x7f130c152fa0 2019-12-05T06:50:27.773 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 171 2019-12-05T06:50:27.773 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:27.773 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.776+0000 7f1312403700 1 -- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f12f0023650 msgr2=0x7f12f0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.776+0000 7f1312403700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f12f0023650 0x7f12f0025ae0 crc :-1 s=READY pgs=585 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:27.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.776+0000 7f1312403700 1 -- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 msgr2=0x7f130c080e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:27.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.776+0000 7f1312403700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 secure :-1 s=READY pgs=324 cs=0 l=1 rx=0x7f1304010390 tx=0x7f1304010730).stop 2019-12-05T06:50:27.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.776+0000 7f1312403700 1 -- 172.21.15.16:0/2396294868 shutdown_connections 2019-12-05T06:50:27.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.776+0000 7f1312403700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f12f0023650 0x7f12f0025ae0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.776+0000 7f1312403700 1 --2- 172.21.15.16:0/2396294868 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f130c152fa0 0x7f130c080e70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.777+0000 7f1312403700 1 -- 172.21.15.16:0/2396294868 shutdown_connections 2019-12-05T06:50:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.777+0000 7f1312403700 1 -- 172.21.15.16:0/2396294868 wait complete. 2019-12-05T06:50:27.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:27.777+0000 7f1312403700 1 -- 172.21.15.16:0/2396294868 >> 172.21.15.16:0/2396294868 conn(0x7f130c084520 msgr2=0x7f130c0995a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:27.787 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:50:29.788 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.044+0000 7fb793f6f700 1 Processor -- start 2019-12-05T06:50:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.044+0000 7fb793f6f700 1 -- start start 2019-12-05T06:50:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.046+0000 7fb793f6f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c08d2c0 0x7fb78c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.046+0000 7fb793f6f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb78c08fef0 0x7fb78c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.046+0000 7fb793f6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c0958e0 con 0x7fb78c08c6d0 2019-12-05T06:50:30.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.046+0000 7fb793f6f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c08df90 con 0x7fb78c08fef0 2019-12-05T06:50:30.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.046+0000 7fb793f6f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c0a3f70 con 0x7fb78c08d2c0 2019-12-05T06:50:30.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.047+0000 7fb792d0d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb78c08fef0 0x7fb78c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:30.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.047+0000 7fb791d0b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c08d2c0 0x7fb78c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:30.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.047+0000 7fb792d0d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb78c08fef0 0x7fb78c08f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:48136/0 (socket says 172.21.15.16:48136) 2019-12-05T06:50:30.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.047+0000 7fb79250c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb78c08c6d0 0x7fb78c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:40280/0 (socket says 172.21.15.16:40280) 2019-12-05T06:50:30.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.047+0000 7fb792d0d700 1 -- 172.21.15.16:0/1483103150 learned_addr learned my addr 172.21.15.16:0/1483103150 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:30.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.047+0000 7fb792d0d700 1 -- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c08d2c0 msgr2=0x7fb78c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:50:30.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.047+0000 7fb792d0d700 1 --2- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c08d2c0 0x7fb78c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:30.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.047+0000 7fb792d0d700 1 -- 172.21.15.16:0/1483103150 >> v1:172.21.15.76:6789/0 conn(0x7fb78c08c6d0 legacy=0x7fb78c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:30.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.047+0000 7fb792d0d700 1 -- 172.21.15.16:0/1483103150 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c08e620 con 0x7fb78c08fef0 2019-12-05T06:50:30.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.047+0000 7fb79150a700 1 -- 172.21.15.16:0/1483103150 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262415736 0 0) 0x7fb784001400 con 0x7fb78c08c6d0 2019-12-05T06:50:30.055 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb791d0b700 1 --2- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c08d2c0 0x7fb78c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:50:30.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb792d0d700 1 --2- 172.21.15.16:0/1483103150 >> v2:172.21.15.76:3300/0 conn(0x7fb78c08fef0 0x7fb78c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb78000f760 tx=0x7fb78000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.056 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb792d0d700 1 --2- 172.21.15.16:0/1483103150 >> v2:172.21.15.76:3300/0 conn(0x7fb78c08fef0 0x7fb78c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb78000f760 tx=0x7fb78000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb792d0d700 1 --2- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb78c08fef0 0x7fb78c08f460 secure :-1 s=READY pgs=948 cs=0 l=1 rx=0x7fb78000f760 tx=0x7fb78000fb00).ready entity=mon.1 client_cookie=4761a7c89b176946 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:30.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb792d0d700 1 --2- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb78c08fef0 0x7fb78c08f460 secure :-1 s=THROTTLE_DONE pgs=948 cs=0 l=1 rx=0x7fb78000f760 tx=0x7fb78000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb79150a700 1 -- 172.21.15.16:0/1483103150 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb780000a70 con 0x7fb78c08fef0 2019-12-05T06:50:30.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb792d0d700 1 --2- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb78c08fef0 0x7fb78c08f460 secure :-1 s=THROTTLE_DONE pgs=948 cs=0 l=1 rx=0x7fb78000f760 tx=0x7fb78000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb79150a700 1 -- 172.21.15.16:0/1483103150 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb780007240 con 0x7fb78c08fef0 2019-12-05T06:50:30.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb792d0d700 1 --2- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb78c08fef0 0x7fb78c08f460 secure :-1 s=THROTTLE_DONE pgs=948 cs=0 l=1 rx=0x7fb78000f760 tx=0x7fb78000fb00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb79150a700 1 -- 172.21.15.16:0/1483103150 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb780001060 con 0x7fb78c08fef0 2019-12-05T06:50:30.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb793f6f700 1 -- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb78c08fef0 msgr2=0x7fb78c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:30.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.048+0000 7fb793f6f700 1 --2- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb78c08fef0 0x7fb78c08f460 secure :-1 s=READY pgs=948 cs=0 l=1 rx=0x7fb78000f760 tx=0x7fb78000fb00).stop 2019-12-05T06:50:30.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.050+0000 7fb793f6f700 1 -- 172.21.15.16:0/1483103150 shutdown_connections 2019-12-05T06:50:30.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.050+0000 7fb793f6f700 1 --2- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb78c08fef0 0x7fb78c08f460 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:30.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.050+0000 7fb793f6f700 1 --2- 172.21.15.16:0/1483103150 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c08d2c0 0x7fb78c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:50:30.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.050+0000 7fb793f6f700 1 -- 172.21.15.16:0/1483103150 shutdown_connections 2019-12-05T06:50:30.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.050+0000 7fb793f6f700 1 -- 172.21.15.16:0/1483103150 wait complete. 2019-12-05T06:50:30.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.050+0000 7fb793f6f700 1 -- 172.21.15.16:0/1483103150 >> 172.21.15.16:0/1483103150 conn(0x7fb78c061940 msgr2=0x7fb78c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:30.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.054+0000 7fb793f6f700 1 Processor -- start 2019-12-05T06:50:30.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.054+0000 7fb793f6f700 1 -- start start 2019-12-05T06:50:30.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.056+0000 7fb793f6f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb78c129ae0 0x7fb78c129310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:30.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.056+0000 7fb793f6f700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c15b170 0x7fb78c15d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:30.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.056+0000 7fb793f6f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb78c132930 con 0x7fb78c0829b0 2019-12-05T06:50:30.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.056+0000 7fb793f6f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb78c12fff0 con 0x7fb78c129ae0 2019-12-05T06:50:30.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.056+0000 7fb793f6f700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c131040 con 0x7fb78c15b170 2019-12-05T06:50:30.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.056+0000 7fb791d0b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb78c129ae0 0x7fb78c129310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:30.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.056+0000 7fb791d0b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb78c129ae0 0x7fb78c129310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33818/0 (socket says 172.21.15.16:33818) 2019-12-05T06:50:30.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.056+0000 7fb791d0b700 1 -- 172.21.15.16:0/2657302136 learned_addr learned my addr 172.21.15.16:0/2657302136 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:30.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.056+0000 7fb792d0d700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c15b170 0x7fb78c15d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:30.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb791d0b700 1 -- 172.21.15.16:0/2657302136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c15b170 msgr2=0x7fb78c15d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:30.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c15b170 0x7fb78c15d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:30.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb791d0b700 1 -- 172.21.15.16:0/2657302136 >> v1:172.21.15.76:6789/0 conn(0x7fb78c0829b0 legacy=0x7fb78c128b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:30.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb791d0b700 1 -- 172.21.15.16:0/2657302136 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c131f00 con 0x7fb78c129ae0 2019-12-05T06:50:30.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> v2:172.21.15.16:3300/0 conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> v2:172.21.15.16:3300/0 conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=READY pgs=1396 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:30.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=THROTTLE_DONE pgs=1396 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=THROTTLE_DONE pgs=1396 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=THROTTLE_DONE pgs=1396 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb77effd700 1 -- 172.21.15.16:0/2657302136 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb788000940 con 0x7fb78c129ae0 2019-12-05T06:50:30.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.057+0000 7fb77effd700 1 -- 172.21.15.16:0/2657302136 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb788009770 con 0x7fb78c129ae0 2019-12-05T06:50:30.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.058+0000 7fb793f6f700 1 -- 172.21.15.16:0/2657302136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c131f00 con 0x7fb78c129ae0 2019-12-05T06:50:30.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.058+0000 7fb77effd700 1 -- 172.21.15.16:0/2657302136 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb788009e40 con 0x7fb78c129ae0 2019-12-05T06:50:30.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.058+0000 7fb793f6f700 1 -- 172.21.15.16:0/2657302136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c131f00 con 0x7fb78c129ae0 2019-12-05T06:50:30.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.058+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=THROTTLE_DONE pgs=1396 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.059+0000 7fb77effd700 1 -- 172.21.15.16:0/2657302136 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb788009960 con 0x7fb78c129ae0 2019-12-05T06:50:30.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.059+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=THROTTLE_DONE pgs=1396 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.059+0000 7fb77effd700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb774023630 0x7fb774025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:30.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.059+0000 7fb77effd700 1 -- 172.21.15.16:0/2657302136 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7fb788045dd0 con 0x7fb78c129ae0 2019-12-05T06:50:30.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.060+0000 7fb79250c700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb774023630 0x7fb774025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:30.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.060+0000 7fb79250c700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb774023630 0x7fb774025ac0 crc :-1 s=READY pgs=586 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:30.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.076+0000 7fb793f6f700 1 -- 172.21.15.16:0/2657302136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c132930 con 0x7fb78c129ae0 2019-12-05T06:50:30.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.081+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=THROTTLE_DONE pgs=1396 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.081+0000 7fb77effd700 1 -- 172.21.15.16:0/2657302136 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb78802b040 con 0x7fb78c129ae0 2019-12-05T06:50:30.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.302+0000 7fb793f6f700 1 -- 172.21.15.16:0/2657302136 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb78c132930 con 0x7fb78c129ae0 2019-12-05T06:50:30.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.302+0000 7fb791d0b700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=THROTTLE_DONE pgs=1396 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:30.302 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.302+0000 7fb77effd700 1 -- 172.21.15.16:0/2657302136 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 171 v171) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fb78802b250 con 0x7fb78c129ae0 2019-12-05T06:50:30.302 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 171 2019-12-05T06:50:30.302 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:30.303 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":171,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.305+0000 7fb793f6f700 1 -- 172.21.15.16:0/2657302136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb774023630 msgr2=0x7fb774025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:30.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.305+0000 7fb793f6f700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb774023630 0x7fb774025ac0 crc :-1 s=READY pgs=586 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.305+0000 7fb793f6f700 1 -- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 msgr2=0x7fb78c129310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.305+0000 7fb793f6f700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 0x7fb78c129310 secure :-1 s=READY pgs=1396 cs=0 l=1 rx=0x7fb788010410 tx=0x7fb788010290).stop 2019-12-05T06:50:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.305+0000 7fb793f6f700 1 -- 172.21.15.16:0/2657302136 shutdown_connections 2019-12-05T06:50:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.305+0000 7fb793f6f700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb78c15b170 0x7fb78c15d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.305+0000 7fb793f6f700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb774023630 0x7fb774025ac0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:30.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.305+0000 7fb793f6f700 1 --2- 172.21.15.16:0/2657302136 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb78c129ae0 0x7fb78c129310 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:30.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.306+0000 7fb793f6f700 1 -- 172.21.15.16:0/2657302136 shutdown_connections 2019-12-05T06:50:30.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.306+0000 7fb793f6f700 1 -- 172.21.15.16:0/2657302136 wait complete. 2019-12-05T06:50:30.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:30.306+0000 7fb793f6f700 1 -- 172.21.15.16:0/2657302136 >> 172.21.15.16:0/2657302136 conn(0x7fb78c0a2850 msgr2=0x7fb78c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:30.315 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:50:32.316 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:50:32.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.529+0000 7f2e7df39700 1 Processor -- start 2019-12-05T06:50:32.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.529+0000 7f2e7df39700 1 -- start start 2019-12-05T06:50:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.531+0000 7f2e7df39700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e7808fef0 0x7f2e7808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:50:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.531+0000 7f2e7df39700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e780958e0 con 0x7f2e7808d2c0 2019-12-05T06:50:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.531+0000 7f2e7df39700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e780629e0 con 0x7f2e7808c6d0 2019-12-05T06:50:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.531+0000 7f2e7df39700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e7808df90 con 0x7f2e7808fef0 2019-12-05T06:50:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.531+0000 7f2e7ccd7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e7808fef0 0x7f2e7808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.531+0000 7f2e777fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2e7808d2c0 0x7f2e7808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47250/0 (socket says 172.21.15.16:47250) 2019-12-05T06:50:32.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.531+0000 7f2e777fe700 1 -- 172.21.15.16:0/47449198 learned_addr learned my addr 172.21.15.16:0/47449198 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:50:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.532+0000 7f2e76ffd700 1 -- 172.21.15.16:0/47449198 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 139959715 0 0) 0x7f2e60001150 con 0x7f2e7808d2c0 2019-12-05T06:50:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.532+0000 7f2e76ffd700 1 -- 172.21.15.16:0/47449198 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e64002e80 con 0x7f2e7808d2c0 2019-12-05T06:50:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.532+0000 7f2e76ffd700 1 -- 172.21.15.16:0/47449198 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877278468 0 0) 0x7f2e68001400 con 0x7f2e7808c6d0 2019-12-05T06:50:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.532+0000 7f2e7ccd7700 1 -- 172.21.15.16:0/47449198 >> v1:172.21.15.76:6789/0 conn(0x7f2e7808c6d0 legacy=0x7f2e7808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.532+0000 7f2e7ccd7700 1 -- 172.21.15.16:0/47449198 >> v1:172.21.15.16:6789/0 conn(0x7f2e7808d2c0 legacy=0x7f2e7808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.532+0000 7f2e7ccd7700 1 -- 172.21.15.16:0/47449198 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e7808e620 con 0x7f2e7808fef0 2019-12-05T06:50:32.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.532+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/47449198 >> v2:172.21.15.16:3300/0 conn(0x7f2e7808fef0 0x7f2e7808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f2e7000af50 tx=0x7f2e7000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.532+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/47449198 >> v2:172.21.15.16:3300/0 conn(0x7f2e7808fef0 0x7f2e7808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f2e7000af50 tx=0x7f2e7000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.533+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/47449198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808f720 secure :-1 s=READY pgs=1398 cs=0 l=1 rx=0x7f2e7000af50 tx=0x7f2e7000b760).ready entity=mon.0 client_cookie=6e52ef7ed488f11f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.533+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/47449198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808f720 secure :-1 s=THROTTLE_DONE pgs=1398 cs=0 l=1 rx=0x7f2e7000af50 tx=0x7f2e7000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.533+0000 7f2e76ffd700 1 -- 172.21.15.16:0/47449198 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e70009ba0 con 0x7f2e7808fef0 2019-12-05T06:50:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.533+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/47449198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808f720 secure :-1 s=THROTTLE_DONE pgs=1398 cs=0 l=1 rx=0x7f2e7000af50 tx=0x7f2e7000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.533+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/47449198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808f720 secure :-1 s=THROTTLE_DONE pgs=1398 cs=0 l=1 rx=0x7f2e7000af50 tx=0x7f2e7000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.533+0000 7f2e76ffd700 1 -- 172.21.15.16:0/47449198 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e70000a70 con 0x7f2e7808fef0 2019-12-05T06:50:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.533+0000 7f2e76ffd700 1 -- 172.21.15.16:0/47449198 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e7000a180 con 0x7f2e7808fef0 2019-12-05T06:50:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.533+0000 7f2e7df39700 1 -- 172.21.15.16:0/47449198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 msgr2=0x7f2e7808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.533+0000 7f2e7df39700 1 --2- 172.21.15.16:0/47449198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808f720 secure :-1 s=READY pgs=1398 cs=0 l=1 rx=0x7f2e7000af50 tx=0x7f2e7000b760).stop 2019-12-05T06:50:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.535+0000 7f2e7df39700 1 -- 172.21.15.16:0/47449198 shutdown_connections 2019-12-05T06:50:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.535+0000 7f2e7df39700 1 --2- 172.21.15.16:0/47449198 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e7808fef0 0x7f2e7808f720 unknown :-1 s=CLOSED pgs=1398 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.535+0000 7f2e7df39700 1 -- 172.21.15.16:0/47449198 shutdown_connections 2019-12-05T06:50:32.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.535+0000 7f2e7df39700 1 -- 172.21.15.16:0/47449198 wait complete. 2019-12-05T06:50:32.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.535+0000 7f2e7df39700 1 -- 172.21.15.16:0/47449198 >> 172.21.15.16:0/47449198 conn(0x7f2e78061940 msgr2=0x7f2e78096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:32.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.540+0000 7f2e7df39700 1 Processor -- start 2019-12-05T06:50:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.540+0000 7f2e7df39700 1 -- start start 2019-12-05T06:50:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7df39700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7809ae80 0x7f2e78133220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7df39700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e78099eb0 0x7f2e7809a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7df39700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e7806e7c0 con 0x7f2e78132630 2019-12-05T06:50:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7df39700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2e78098f20 con 0x7f2e78099eb0 2019-12-05T06:50:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7df39700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2e78134390 con 0x7f2e7809ae80 2019-12-05T06:50:32.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7ccd7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e78099eb0 0x7f2e7809a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:32.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7ccd7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f2e78099eb0 0x7f2e7809a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:33858/0 (socket says 172.21.15.16:33858) 2019-12-05T06:50:32.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7ccd7700 1 -- 172.21.15.16:0/67592659 learned_addr learned my addr 172.21.15.16:0/67592659 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:50:32.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e777fe700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7809ae80 0x7f2e78133220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7ccd7700 1 -- 172.21.15.16:0/67592659 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7809ae80 msgr2=0x7f2e78133220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7809ae80 0x7f2e78133220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7ccd7700 1 -- 172.21.15.16:0/67592659 >> v1:172.21.15.76:6789/0 conn(0x7f2e78132630 legacy=0x7f2e78132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7ccd7700 1 -- 172.21.15.16:0/67592659 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e780995b0 con 0x7f2e78099eb0 2019-12-05T06:50:32.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.542+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> v2:172.21.15.16:3300/0 conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.543+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> v2:172.21.15.16:3300/0 conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.543+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=READY pgs=1399 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:32.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.543+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=THROTTLE_DONE pgs=1399 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.543+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=THROTTLE_DONE pgs=1399 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.543+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=THROTTLE_DONE pgs=1399 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.543+0000 7f2e74ff9700 1 -- 172.21.15.16:0/67592659 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e7000aec0 con 0x7f2e78099eb0 2019-12-05T06:50:32.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.543+0000 7f2e74ff9700 1 -- 172.21.15.16:0/67592659 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2e70009770 con 0x7f2e78099eb0 2019-12-05T06:50:32.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.543+0000 7f2e74ff9700 1 -- 172.21.15.16:0/67592659 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2e70009e40 con 0x7f2e78099eb0 2019-12-05T06:50:32.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.543+0000 7f2e7df39700 1 -- 172.21.15.16:0/67592659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e780995b0 con 0x7f2e78099eb0 2019-12-05T06:50:32.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.543+0000 7f2e7df39700 1 -- 172.21.15.16:0/67592659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e780995b0 con 0x7f2e78099eb0 2019-12-05T06:50:32.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.544+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=THROTTLE_DONE pgs=1399 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.545+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=THROTTLE_DONE pgs=1399 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.545+0000 7f2e74ff9700 1 -- 172.21.15.16:0/67592659 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2e70009960 con 0x7f2e78099eb0 2019-12-05T06:50:32.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.545+0000 7f2e74ff9700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 0x7f2e5c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:50:32.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.545+0000 7f2e74ff9700 1 -- 172.21.15.16:0/67592659 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f2e70045e40 con 0x7f2e78099eb0 2019-12-05T06:50:32.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.546+0000 7f2e77fff700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 0x7f2e5c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:50:32.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.546+0000 7f2e77fff700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 0x7f2e5c025ac0 crc :-1 s=READY pgs=588 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:50:32.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.560+0000 7f2e7df39700 1 -- 172.21.15.16:0/67592659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e780995b0 con 0x7f2e78099eb0 2019-12-05T06:50:32.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.567+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=THROTTLE_DONE pgs=1399 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.567+0000 7f2e74ff9700 1 -- 172.21.15.16:0/67592659 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2e7000acb0 con 0x7f2e78099eb0 2019-12-05T06:50:32.776 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.776+0000 7f2e7df39700 1 -- 172.21.15.16:0/67592659 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2e780995b0 con 0x7f2e78099eb0 2019-12-05T06:50:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.777+0000 7f2e7ccd7700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=THROTTLE_DONE pgs=1399 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:50:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.777+0000 7f2e74ff9700 1 -- 172.21.15.16:0/67592659 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f2e7000a1f0 con 0x7f2e78099eb0 2019-12-05T06:50:32.777 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 172 2019-12-05T06:50:32.778 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:50:32.778 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:50:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 -- 172.21.15.16:0/67592659 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 msgr2=0x7f2e5c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 0x7f2e5c025ac0 crc :-1 s=READY pgs=588 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 -- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 msgr2=0x7f2e7809a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:50:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 0x7f2e7809a370 secure :-1 s=READY pgs=1399 cs=0 l=1 rx=0x7f2e70010450 tx=0x7f2e70010280).stop 2019-12-05T06:50:32.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 -- 172.21.15.16:0/67592659 shutdown_connections 2019-12-05T06:50:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2e5c023630 0x7f2e5c025ac0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f2e78099eb0 0x7f2e7809a370 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 --2- 172.21.15.16:0/67592659 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2e7809ae80 0x7f2e78133220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:50:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 -- 172.21.15.16:0/67592659 shutdown_connections 2019-12-05T06:50:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 -- 172.21.15.16:0/67592659 wait complete. 2019-12-05T06:50:32.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:50:32.780+0000 7f2e7df39700 1 -- 172.21.15.16:0/67592659 >> 172.21.15.16:0/67592659 conn(0x7f2e780714f0 msgr2=0x7f2e7809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:50:32.789 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:50:32.790 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 84 secs before thrashing 2019-12-05T06:50:45.208 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:50:45.213 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:51:15.355 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:51:15.360 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:51:45.520 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:51:45.524 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:51:56.790 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:51:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.942+0000 7f808207e700 1 Processor -- start 2019-12-05T06:51:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.942+0000 7f808207e700 1 -- start start 2019-12-05T06:51:56.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.943+0000 7f808207e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f807c08c6d0 0x7f807c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.943+0000 7f808207e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f807c08fef0 0x7f807c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.943+0000 7f808207e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f807c08ef20 0x7f807c08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.943+0000 7f808207e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c0a3f70 con 0x7f807c08ef20 2019-12-05T06:51:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.943+0000 7f808207e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c08d830 con 0x7f807c08fef0 2019-12-05T06:51:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.943+0000 7f808207e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c08d540 con 0x7f807c08c6d0 2019-12-05T06:51:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.943+0000 7f8080e1c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f807c08ef20 0x7f807c08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:56.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.943+0000 7f8080e1c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f807c08ef20 0x7f807c08f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34264/0 (socket says 172.21.15.16:34264) 2019-12-05T06:51:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.943+0000 7f8080e1c700 1 -- 172.21.15.16:0/3652985126 learned_addr learned my addr 172.21.15.16:0/3652985126 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:51:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f80737fe700 1 --2- 172.21.15.16:0/3652985126 >> v2:172.21.15.76:3300/0 conn(0x7f807c08fef0 0x7f807c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f807bfff700 1 --2- 172.21.15.16:0/3652985126 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f807c08c6d0 0x7f807c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 -- 172.21.15.16:0/3652985126 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f807c08c6d0 msgr2=0x7f807c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 --2- 172.21.15.16:0/3652985126 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f807c08c6d0 0x7f807c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 -- 172.21.15.16:0/3652985126 >> v2:172.21.15.76:3300/0 conn(0x7f807c08fef0 msgr2=0x7f807c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:56.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 --2- 172.21.15.16:0/3652985126 >> v2:172.21.15.76:3300/0 conn(0x7f807c08fef0 0x7f807c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 -- 172.21.15.16:0/3652985126 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c08e620 con 0x7f807c08ef20 2019-12-05T06:51:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 --2- 172.21.15.16:0/3652985126 >> v2:172.21.15.16:3300/0 conn(0x7f807c08ef20 0x7f807c08f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f807400fa30 tx=0x7f80740080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 --2- 172.21.15.16:0/3652985126 >> v2:172.21.15.16:3300/0 conn(0x7f807c08ef20 0x7f807c08f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f807400fa30 tx=0x7f80740080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 --2- 172.21.15.16:0/3652985126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c08ef20 0x7f807c08f3c0 secure :-1 s=READY pgs=1402 cs=0 l=1 rx=0x7f807400fa30 tx=0x7f80740080d0).ready entity=mon.0 client_cookie=dcee993b756530cf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 --2- 172.21.15.16:0/3652985126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c08ef20 0x7f807c08f3c0 secure :-1 s=THROTTLE_DONE pgs=1402 cs=0 l=1 rx=0x7f807400fa30 tx=0x7f80740080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 --2- 172.21.15.16:0/3652985126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c08ef20 0x7f807c08f3c0 secure :-1 s=THROTTLE_DONE pgs=1402 cs=0 l=1 rx=0x7f807400fa30 tx=0x7f80740080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f807b7fe700 1 -- 172.21.15.16:0/3652985126 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8074000f20 con 0x7f807c08ef20 2019-12-05T06:51:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f8080e1c700 1 --2- 172.21.15.16:0/3652985126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c08ef20 0x7f807c08f3c0 secure :-1 s=THROTTLE_DONE pgs=1402 cs=0 l=1 rx=0x7f807400fa30 tx=0x7f80740080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f807b7fe700 1 -- 172.21.15.16:0/3652985126 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8074001460 con 0x7f807c08ef20 2019-12-05T06:51:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f807b7fe700 1 -- 172.21.15.16:0/3652985126 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8074001b30 con 0x7f807c08ef20 2019-12-05T06:51:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f808207e700 1 -- 172.21.15.16:0/3652985126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c08ef20 msgr2=0x7f807c08f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.944+0000 7f808207e700 1 --2- 172.21.15.16:0/3652985126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c08ef20 0x7f807c08f3c0 secure :-1 s=READY pgs=1402 cs=0 l=1 rx=0x7f807400fa30 tx=0x7f80740080d0).stop 2019-12-05T06:51:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.945+0000 7f808207e700 1 -- 172.21.15.16:0/3652985126 shutdown_connections 2019-12-05T06:51:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.945+0000 7f808207e700 1 --2- 172.21.15.16:0/3652985126 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c08ef20 0x7f807c08f3c0 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:56.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.945+0000 7f808207e700 1 --2- 172.21.15.16:0/3652985126 >> v2:172.21.15.76:3300/0 conn(0x7f807c08fef0 0x7f807c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.945+0000 7f808207e700 1 --2- 172.21.15.16:0/3652985126 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f807c08c6d0 0x7f807c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.945+0000 7f808207e700 1 -- 172.21.15.16:0/3652985126 shutdown_connections 2019-12-05T06:51:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.945+0000 7f808207e700 1 -- 172.21.15.16:0/3652985126 wait complete. 2019-12-05T06:51:56.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.945+0000 7f808207e700 1 -- 172.21.15.16:0/3652985126 >> 172.21.15.16:0/3652985126 conn(0x7f807c061940 msgr2=0x7f807c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:56.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.948+0000 7f808207e700 1 Processor -- start 2019-12-05T06:51:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.948+0000 7f808207e700 1 -- start start 2019-12-05T06:51:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f808207e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f807c096e90 0x7f807c09b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f808207e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f807c084f30 0x7f807c09b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f808207e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f807c097280 0x7f807c0976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f808207e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c09a450 con 0x7f807c096e90 2019-12-05T06:51:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f808207e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f807c099a20 con 0x7f807c084f30 2019-12-05T06:51:56.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f808207e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f807c099cf0 con 0x7f807c097280 2019-12-05T06:51:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f807c096e90 0x7f807c09b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f807c096e90 0x7f807c09b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34266/0 (socket says 172.21.15.16:34266) 2019-12-05T06:51:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 -- 172.21.15.16:0/1130477355 learned_addr learned my addr 172.21.15.16:0/1130477355 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:51:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f80737fe700 1 --2- 172.21.15.16:0/1130477355 >> v2:172.21.15.76:3300/0 conn(0x7f807c084f30 0x7f807c09b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f8080e1c700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f807c097280 0x7f807c0976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 -- 172.21.15.16:0/1130477355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f807c097280 msgr2=0x7f807c0976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:56.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f807c097280 0x7f807c0976c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 -- 172.21.15.16:0/1130477355 >> v2:172.21.15.76:3300/0 conn(0x7f807c084f30 msgr2=0x7f807c09b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> v2:172.21.15.76:3300/0 conn(0x7f807c084f30 0x7f807c09b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 -- 172.21.15.16:0/1130477355 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f807c09aae0 con 0x7f807c096e90 2019-12-05T06:51:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f80737fe700 1 --2- 172.21.15.16:0/1130477355 >> v2:172.21.15.76:3300/0 conn(0x7f807c084f30 0x7f807c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:51:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> v2:172.21.15.16:3300/0 conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> v2:172.21.15.16:3300/0 conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=READY pgs=1403 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=THROTTLE_DONE pgs=1403 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=THROTTLE_DONE pgs=1403 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=THROTTLE_DONE pgs=1403 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.949+0000 7f80797fa700 1 -- 172.21.15.16:0/1130477355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f806c005180 con 0x7f807c096e90 2019-12-05T06:51:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.950+0000 7f80797fa700 1 -- 172.21.15.16:0/1130477355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f806c005880 con 0x7f807c096e90 2019-12-05T06:51:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.950+0000 7f808207e700 1 -- 172.21.15.16:0/1130477355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f807c09aae0 con 0x7f807c096e90 2019-12-05T06:51:56.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.950+0000 7f80797fa700 1 -- 172.21.15.16:0/1130477355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f806c005f90 con 0x7f807c096e90 2019-12-05T06:51:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.950+0000 7f808207e700 1 -- 172.21.15.16:0/1130477355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f807c09aae0 con 0x7f807c096e90 2019-12-05T06:51:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.950+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=THROTTLE_DONE pgs=1403 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.950+0000 7f80797fa700 1 -- 172.21.15.16:0/1130477355 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f806c005a70 con 0x7f807c096e90 2019-12-05T06:51:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.951+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=THROTTLE_DONE pgs=1403 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.951+0000 7f80797fa700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8060023630 0x7f8060025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.951+0000 7f80797fa700 1 -- 172.21.15.16:0/1130477355 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f806c04a150 con 0x7f807c096e90 2019-12-05T06:51:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.951+0000 7f80737fe700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8060023630 0x7f8060025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:56.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.951+0000 7f80737fe700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8060023630 0x7f8060025ac0 crc :-1 s=READY pgs=591 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:56.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.957+0000 7f808207e700 1 -- 172.21.15.16:0/1130477355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f807c09aae0 con 0x7f807c096e90 2019-12-05T06:51:56.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.962+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=THROTTLE_DONE pgs=1403 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:56.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:56.962+0000 7f80797fa700 1 -- 172.21.15.16:0/1130477355 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f806c005390 con 0x7f807c096e90 2019-12-05T06:51:57.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.156+0000 7f808207e700 1 -- 172.21.15.16:0/1130477355 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f807c09aae0 con 0x7f807c096e90 2019-12-05T06:51:57.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.156+0000 7f807bfff700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=THROTTLE_DONE pgs=1403 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.156+0000 7f80797fa700 1 -- 172.21.15.16:0/1130477355 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f806c006320 con 0x7f807c096e90 2019-12-05T06:51:57.157 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 172 2019-12-05T06:51:57.157 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:51:57.157 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:51:57.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 -- 172.21.15.16:0/1130477355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8060023630 msgr2=0x7f8060025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:57.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8060023630 0x7f8060025ac0 crc :-1 s=READY pgs=591 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 -- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 msgr2=0x7f807c09b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:57.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 0x7f807c09b3e0 secure :-1 s=READY pgs=1403 cs=0 l=1 rx=0x7f806c014880 tx=0x7f806c014680).stop 2019-12-05T06:51:57.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 -- 172.21.15.16:0/1130477355 shutdown_connections 2019-12-05T06:51:57.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f807c097280 0x7f807c0976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 --2- 172.21.15.16:0/1130477355 >> v2:172.21.15.76:3300/0 conn(0x7f807c084f30 0x7f807c09b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8060023630 0x7f8060025ac0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 --2- 172.21.15.16:0/1130477355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f807c096e90 0x7f807c09b3e0 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 -- 172.21.15.16:0/1130477355 shutdown_connections 2019-12-05T06:51:57.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 -- 172.21.15.16:0/1130477355 wait complete. 2019-12-05T06:51:57.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.159+0000 7f808207e700 1 -- 172.21.15.16:0/1130477355 >> 172.21.15.16:0/1130477355 conn(0x7f807c0714f0 msgr2=0x7f807c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:57.170 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:51:57.170 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:51:57.171 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:51:57.181 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:51:57.181 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:51:57.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.507+0000 7f32bb9e6700 1 Processor -- start 2019-12-05T06:51:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.508+0000 7f32bb9e6700 1 -- start start 2019-12-05T06:51:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.509+0000 7f32bb9e6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f32b408c6d0 0x7f32b408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.509+0000 7f32bb9e6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f32b408fef0 0x7f32b408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:51:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.509+0000 7f32bb9e6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b40958e0 con 0x7f32b408ef20 2019-12-05T06:51:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.509+0000 7f32bb9e6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b408df90 con 0x7f32b408c6d0 2019-12-05T06:51:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.509+0000 7f32bb9e6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32b40a3f70 con 0x7f32b408fef0 2019-12-05T06:51:57.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.510+0000 7f32b9f83700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f32b408c6d0 0x7f32b408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.510+0000 7f32ba784700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f32b408ef20 0x7f32b408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47686/0 (socket says 172.21.15.16:47686) 2019-12-05T06:51:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.510+0000 7f32ba784700 1 -- 172.21.15.16:0/522528788 learned_addr learned my addr 172.21.15.16:0/522528788 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:51:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.510+0000 7f32b9782700 1 --2- 172.21.15.16:0/522528788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f32b408fef0 0x7f32b408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.510+0000 7f32b9f83700 1 -- 172.21.15.16:0/522528788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f32b408fef0 msgr2=0x7f32b408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:51:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.510+0000 7f32b9f83700 1 --2- 172.21.15.16:0/522528788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f32b408fef0 0x7f32b408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.510+0000 7f32b9f83700 1 -- 172.21.15.16:0/522528788 >> v1:172.21.15.16:6789/0 conn(0x7f32b408ef20 legacy=0x7f32b408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:57.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.511+0000 7f32b9f83700 1 -- 172.21.15.16:0/522528788 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b408d540 con 0x7f32b408c6d0 2019-12-05T06:51:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.510+0000 7f32b8f81700 1 -- 172.21.15.16:0/522528788 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553424591 0 0) 0x7f32ac001150 con 0x7f32b408ef20 2019-12-05T06:51:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.511+0000 7f32b9f83700 1 --2- 172.21.15.16:0/522528788 >> v2:172.21.15.16:3300/0 conn(0x7f32b408c6d0 0x7f32b408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f32b000b2d0 tx=0x7f32b000ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.511+0000 7f32b9f83700 1 --2- 172.21.15.16:0/522528788 >> v2:172.21.15.16:3300/0 conn(0x7f32b408c6d0 0x7f32b408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f32b000b2d0 tx=0x7f32b000ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.511+0000 7f32b9f83700 1 --2- 172.21.15.16:0/522528788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b408c6d0 0x7f32b408cac0 secure :-1 s=READY pgs=1405 cs=0 l=1 rx=0x7f32b000b2d0 tx=0x7f32b000ba60).ready entity=mon.0 client_cookie=2acf43bc6f93b933 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.511+0000 7f32b9f83700 1 --2- 172.21.15.16:0/522528788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b408c6d0 0x7f32b408cac0 secure :-1 s=THROTTLE_DONE pgs=1405 cs=0 l=1 rx=0x7f32b000b2d0 tx=0x7f32b000ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.511+0000 7f32b8f81700 1 -- 172.21.15.16:0/522528788 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b0007590 con 0x7f32b408c6d0 2019-12-05T06:51:57.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.511+0000 7f32b9f83700 1 --2- 172.21.15.16:0/522528788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b408c6d0 0x7f32b408cac0 secure :-1 s=THROTTLE_DONE pgs=1405 cs=0 l=1 rx=0x7f32b000b2d0 tx=0x7f32b000ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.511+0000 7f32b9f83700 1 --2- 172.21.15.16:0/522528788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b408c6d0 0x7f32b408cac0 secure :-1 s=THROTTLE_DONE pgs=1405 cs=0 l=1 rx=0x7f32b000b2d0 tx=0x7f32b000ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.512+0000 7f32b8f81700 1 -- 172.21.15.16:0/522528788 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0009d30 con 0x7f32b408c6d0 2019-12-05T06:51:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.512+0000 7f32b8f81700 1 -- 172.21.15.16:0/522528788 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b000a400 con 0x7f32b408c6d0 2019-12-05T06:51:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.512+0000 7f32bb9e6700 1 -- 172.21.15.16:0/522528788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b408c6d0 msgr2=0x7f32b408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.512+0000 7f32bb9e6700 1 --2- 172.21.15.16:0/522528788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b408c6d0 0x7f32b408cac0 secure :-1 s=READY pgs=1405 cs=0 l=1 rx=0x7f32b000b2d0 tx=0x7f32b000ba60).stop 2019-12-05T06:51:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.513+0000 7f32bb9e6700 1 -- 172.21.15.16:0/522528788 shutdown_connections 2019-12-05T06:51:57.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.513+0000 7f32bb9e6700 1 --2- 172.21.15.16:0/522528788 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f32b408fef0 0x7f32b408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:51:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.513+0000 7f32bb9e6700 1 --2- 172.21.15.16:0/522528788 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b408c6d0 0x7f32b408cac0 unknown :-1 s=CLOSED pgs=1405 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.514+0000 7f32bb9e6700 1 -- 172.21.15.16:0/522528788 shutdown_connections 2019-12-05T06:51:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.514+0000 7f32bb9e6700 1 -- 172.21.15.16:0/522528788 wait complete. 2019-12-05T06:51:57.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.514+0000 7f32bb9e6700 1 -- 172.21.15.16:0/522528788 >> 172.21.15.16:0/522528788 conn(0x7f32b4061940 msgr2=0x7f32b4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:57.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.519+0000 7f32bb9e6700 1 Processor -- start 2019-12-05T06:51:57.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.519+0000 7f32bb9e6700 1 -- start start 2019-12-05T06:51:57.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.520+0000 7f32bb9e6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f32b41333a0 0x7f32b4128360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:57.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.520+0000 7f32bb9e6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f32b41288a0 0x7f32b406f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:57.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.520+0000 7f32bb9e6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32b4070330 con 0x7f32b4128fe0 2019-12-05T06:51:57.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.520+0000 7f32bb9e6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b4088730 con 0x7f32b41333a0 2019-12-05T06:51:57.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.520+0000 7f32bb9e6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32b4089810 con 0x7f32b41288a0 2019-12-05T06:51:57.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.520+0000 7f32b9f83700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f32b41333a0 0x7f32b4128360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:57.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.520+0000 7f32b9782700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f32b4128fe0 0x7f32b406ed20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47690/0 (socket says 172.21.15.16:47690) 2019-12-05T06:51:57.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.520+0000 7f32b9f83700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f32b41333a0 0x7f32b4128360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34290/0 (socket says 172.21.15.16:34290) 2019-12-05T06:51:57.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.520+0000 7f32b9782700 1 -- 172.21.15.16:0/837628203 learned_addr learned my addr 172.21.15.16:0/837628203 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:51:57.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.521+0000 7f32ba784700 1 --2- 172.21.15.16:0/837628203 >> v2:172.21.15.76:3300/0 conn(0x7f32b41288a0 0x7f32b406f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:57.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.521+0000 7f32b9f83700 1 -- 172.21.15.16:0/837628203 >> v2:172.21.15.76:3300/0 conn(0x7f32b41288a0 msgr2=0x7f32b406f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:57.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.521+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> v2:172.21.15.76:3300/0 conn(0x7f32b41288a0 0x7f32b406f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.521+0000 7f32b9f83700 1 -- 172.21.15.16:0/837628203 >> v1:172.21.15.16:6789/0 conn(0x7f32b4128fe0 legacy=0x7f32b406ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:57.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.521+0000 7f32b9f83700 1 -- 172.21.15.16:0/837628203 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b4089ea0 con 0x7f32b41333a0 2019-12-05T06:51:57.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.521+0000 7f32aa7fc700 1 -- 172.21.15.16:0/837628203 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 64265964 0 0) 0x7f32a40010b0 con 0x7f32b4128fe0 2019-12-05T06:51:57.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.521+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> v2:172.21.15.16:3300/0 conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.521+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> v2:172.21.15.16:3300/0 conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.521+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=READY pgs=1407 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:57.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.522+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=THROTTLE_DONE pgs=1407 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.522+0000 7f32aa7fc700 1 -- 172.21.15.16:0/837628203 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b0000cc0 con 0x7f32b41333a0 2019-12-05T06:51:57.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.522+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=THROTTLE_DONE pgs=1407 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.522+0000 7f32aa7fc700 1 -- 172.21.15.16:0/837628203 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32b0009af0 con 0x7f32b41333a0 2019-12-05T06:51:57.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.522+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=THROTTLE_DONE pgs=1407 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.522+0000 7f32bb9e6700 1 -- 172.21.15.16:0/837628203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32b4089ea0 con 0x7f32b41333a0 2019-12-05T06:51:57.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.522+0000 7f32aa7fc700 1 -- 172.21.15.16:0/837628203 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32b0009ce0 con 0x7f32b41333a0 2019-12-05T06:51:57.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.522+0000 7f32bb9e6700 1 -- 172.21.15.16:0/837628203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32b4089ea0 con 0x7f32b41333a0 2019-12-05T06:51:57.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.523+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=THROTTLE_DONE pgs=1407 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.524+0000 7f32aa7fc700 1 -- 172.21.15.16:0/837628203 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f32b0009ce0 con 0x7f32b41333a0 2019-12-05T06:51:57.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.524+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=THROTTLE_DONE pgs=1407 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.524+0000 7f32aa7fc700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f329c023630 0x7f329c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:51:57.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.524+0000 7f32aa7fc700 1 -- 172.21.15.16:0/837628203 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f32b0046080 con 0x7f32b41333a0 2019-12-05T06:51:57.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.525+0000 7f32b9782700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f329c023630 0x7f329c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:51:57.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.525+0000 7f32b9782700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f329c023630 0x7f329c025ac0 crc :-1 s=READY pgs=592 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:51:57.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.540+0000 7f32bb9e6700 1 -- 172.21.15.16:0/837628203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32b4089ea0 con 0x7f32b41333a0 2019-12-05T06:51:57.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.548+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=THROTTLE_DONE pgs=1407 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.548+0000 7f32aa7fc700 1 -- 172.21.15.16:0/837628203 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f32b0040340 con 0x7f32b41333a0 2019-12-05T06:51:57.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.810+0000 7f32bb9e6700 1 -- 172.21.15.16:0/837628203 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f32b4089ea0 con 0x7f32b41333a0 2019-12-05T06:51:57.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.811+0000 7f32b9f83700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=THROTTLE_DONE pgs=1407 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:51:57.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.811+0000 7f32aa7fc700 1 -- 172.21.15.16:0/837628203 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f32b0040340 con 0x7f32b41333a0 2019-12-05T06:51:57.812 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 172 2019-12-05T06:51:57.812 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:51:57.812 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:51:57.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 -- 172.21.15.16:0/837628203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f329c023630 msgr2=0x7f329c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:57.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f329c023630 0x7f329c025ac0 crc :-1 s=READY pgs=592 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 -- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 msgr2=0x7f32b4128360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:51:57.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 0x7f32b4128360 secure :-1 s=READY pgs=1407 cs=0 l=1 rx=0x7f32b0010670 tx=0x7f32b0010a10).stop 2019-12-05T06:51:57.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 -- 172.21.15.16:0/837628203 shutdown_connections 2019-12-05T06:51:57.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 --2- 172.21.15.16:0/837628203 >> v2:172.21.15.76:3300/0 conn(0x7f32b41288a0 0x7f32b406f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f329c023630 0x7f329c025ac0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 --2- 172.21.15.16:0/837628203 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f32b41333a0 0x7f32b4128360 unknown :-1 s=CLOSED pgs=1407 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:51:57.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 -- 172.21.15.16:0/837628203 shutdown_connections 2019-12-05T06:51:57.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 -- 172.21.15.16:0/837628203 wait complete. 2019-12-05T06:51:57.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:51:57.814+0000 7f32bb9e6700 1 -- 172.21.15.16:0/837628203 >> 172.21.15.16:0/837628203 conn(0x7f32b409de30 msgr2=0x7f32b4070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:51:57.825 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:51:59.826 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:00.103 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.093+0000 7f1d95253700 1 Processor -- start 2019-12-05T06:52:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.093+0000 7f1d95253700 1 -- start start 2019-12-05T06:52:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.095+0000 7f1d95253700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1d9008d2c0 0x7f1d9008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.095+0000 7f1d95253700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 0x7f1d9008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.095+0000 7f1d95253700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d900958e0 con 0x7f1d9008c6d0 2019-12-05T06:52:00.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.095+0000 7f1d95253700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d9008df90 con 0x7f1d9008d2c0 2019-12-05T06:52:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.095+0000 7f1d95253700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d900a3f70 con 0x7f1d9008fef0 2019-12-05T06:52:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.095+0000 7f1d8ed9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1d9008d2c0 0x7f1d9008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.095+0000 7f1d94a52700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 0x7f1d9008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.095+0000 7f1d8f59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f1d9008c6d0 0x7f1d9008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:40772/0 (socket says 172.21.15.16:40772) 2019-12-05T06:52:00.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.095+0000 7f1d8f59e700 1 -- 172.21.15.16:0/3388051090 learned_addr learned my addr 172.21.15.16:0/3388051090 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:00.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.096+0000 7f1d94a52700 1 -- 172.21.15.16:0/3388051090 >> v2:172.21.15.76:3300/0 conn(0x7f1d9008d2c0 msgr2=0x7f1d9008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.096+0000 7f1d94a52700 1 --2- 172.21.15.16:0/3388051090 >> v2:172.21.15.76:3300/0 conn(0x7f1d9008d2c0 0x7f1d9008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.096+0000 7f1d94a52700 1 -- 172.21.15.16:0/3388051090 >> v1:172.21.15.76:6789/0 conn(0x7f1d9008c6d0 legacy=0x7f1d9008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:00.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.096+0000 7f1d94a52700 1 -- 172.21.15.16:0/3388051090 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9008e620 con 0x7f1d9008fef0 2019-12-05T06:52:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.096+0000 7f1d8e59c700 1 -- 172.21.15.16:0/3388051090 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064551371 0 0) 0x7f1d88001480 con 0x7f1d9008c6d0 2019-12-05T06:52:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.096+0000 7f1d8ed9d700 1 --2- 172.21.15.16:0/3388051090 >> v2:172.21.15.76:3300/0 conn(0x7f1d9008d2c0 0x7f1d9008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:52:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.096+0000 7f1d94a52700 1 --2- 172.21.15.16:0/3388051090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 0x7f1d9008f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1d8000af80 tx=0x7f1d8000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.096+0000 7f1d94a52700 1 --2- 172.21.15.16:0/3388051090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 0x7f1d9008f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1d8000af80 tx=0x7f1d8000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.096+0000 7f1d94a52700 1 --2- 172.21.15.16:0/3388051090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 0x7f1d9008f460 secure :-1 s=READY pgs=328 cs=0 l=1 rx=0x7f1d8000af80 tx=0x7f1d8000b790).ready entity=mon.2 client_cookie=24ff585b5c06c48d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.097+0000 7f1d94a52700 1 --2- 172.21.15.16:0/3388051090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 0x7f1d9008f460 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7f1d8000af80 tx=0x7f1d8000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.097+0000 7f1d8e59c700 1 -- 172.21.15.16:0/3388051090 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d80000bc0 con 0x7f1d9008fef0 2019-12-05T06:52:00.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.097+0000 7f1d94a52700 1 --2- 172.21.15.16:0/3388051090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 0x7f1d9008f460 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7f1d8000af80 tx=0x7f1d8000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.097+0000 7f1d94a52700 1 --2- 172.21.15.16:0/3388051090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 0x7f1d9008f460 secure :-1 s=THROTTLE_DONE pgs=328 cs=0 l=1 rx=0x7f1d8000af80 tx=0x7f1d8000b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.097+0000 7f1d8e59c700 1 -- 172.21.15.16:0/3388051090 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d800097f0 con 0x7f1d9008fef0 2019-12-05T06:52:00.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.097+0000 7f1d8e59c700 1 -- 172.21.15.16:0/3388051090 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d80000bc0 con 0x7f1d9008fef0 2019-12-05T06:52:00.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.097+0000 7f1d95253700 1 -- 172.21.15.16:0/3388051090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 msgr2=0x7f1d9008f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:00.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.097+0000 7f1d95253700 1 --2- 172.21.15.16:0/3388051090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 0x7f1d9008f460 secure :-1 s=READY pgs=328 cs=0 l=1 rx=0x7f1d8000af80 tx=0x7f1d8000b790).stop 2019-12-05T06:52:00.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.099+0000 7f1d95253700 1 -- 172.21.15.16:0/3388051090 shutdown_connections 2019-12-05T06:52:00.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.099+0000 7f1d95253700 1 --2- 172.21.15.16:0/3388051090 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1d9008fef0 0x7f1d9008f460 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:00.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.099+0000 7f1d95253700 1 --2- 172.21.15.16:0/3388051090 >> v2:172.21.15.76:3300/0 conn(0x7f1d9008d2c0 0x7f1d9008ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:00.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.099+0000 7f1d95253700 1 -- 172.21.15.16:0/3388051090 shutdown_connections 2019-12-05T06:52:00.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.099+0000 7f1d95253700 1 -- 172.21.15.16:0/3388051090 wait complete. 2019-12-05T06:52:00.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.099+0000 7f1d95253700 1 -- 172.21.15.16:0/3388051090 >> 172.21.15.16:0/3388051090 conn(0x7f1d90061940 msgr2=0x7f1d90096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:00.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.104+0000 7f1d95253700 1 Processor -- start 2019-12-05T06:52:00.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.105+0000 7f1d95253700 1 -- start start 2019-12-05T06:52:00.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.106+0000 7f1d95253700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1d90135450 0x7f1d90134c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:00.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.106+0000 7f1d95253700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d90128af0 con 0x7f1d9006dc50 2019-12-05T06:52:00.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.106+0000 7f1d95253700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1d90128600 con 0x7f1d9006e840 2019-12-05T06:52:00.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.106+0000 7f1d95253700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d90132830 con 0x7f1d90135450 2019-12-05T06:52:00.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.106+0000 7f1d8f59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1d9006dc50 0x7f1d9006e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47708/0 (socket says 172.21.15.16:47708) 2019-12-05T06:52:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.106+0000 7f1d8f59e700 1 -- 172.21.15.16:0/2493282147 learned_addr learned my addr 172.21.15.16:0/2493282147 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.106+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> v2:172.21.15.76:3300/0 conn(0x7f1d90135450 0x7f1d90134c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.107+0000 7f1d87fff700 1 -- 172.21.15.16:0/2493282147 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369981573 0 0) 0x7f1d88000fe0 con 0x7f1d9006dc50 2019-12-05T06:52:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.107+0000 7f1d87fff700 1 -- 172.21.15.16:0/2493282147 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1d74002e80 con 0x7f1d9006dc50 2019-12-05T06:52:00.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.107+0000 7f1d94a52700 1 -- 172.21.15.16:0/2493282147 >> v1:172.21.15.76:6789/0 conn(0x7f1d9006e840 legacy=0x7f1d90134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.107+0000 7f1d94a52700 1 -- 172.21.15.16:0/2493282147 >> v1:172.21.15.16:6789/0 conn(0x7f1d9006dc50 legacy=0x7f1d9006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:00.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.107+0000 7f1d94a52700 1 -- 172.21.15.16:0/2493282147 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d90133b80 con 0x7f1d90135450 2019-12-05T06:52:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.107+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> v2:172.21.15.76:3300/0 conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.108+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> v2:172.21.15.76:3300/0 conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.108+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=READY pgs=958 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.108+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.108+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.108+0000 7f1d87fff700 1 -- 172.21.15.16:0/2493282147 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d80009990 con 0x7f1d90135450 2019-12-05T06:52:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.108+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.123 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.108+0000 7f1d87fff700 1 -- 172.21.15.16:0/2493282147 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d800095b0 con 0x7f1d90135450 2019-12-05T06:52:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.108+0000 7f1d95253700 1 -- 172.21.15.16:0/2493282147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d90133b80 con 0x7f1d90135450 2019-12-05T06:52:00.124 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.108+0000 7f1d87fff700 1 -- 172.21.15.16:0/2493282147 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d80009f70 con 0x7f1d90135450 2019-12-05T06:52:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.108+0000 7f1d95253700 1 -- 172.21.15.16:0/2493282147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d90133b80 con 0x7f1d90135450 2019-12-05T06:52:00.126 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.109+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.110+0000 7f1d87fff700 1 -- 172.21.15.16:0/2493282147 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1d80009840 con 0x7f1d90135450 2019-12-05T06:52:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.110+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.111+0000 7f1d87fff700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023670 0x7f1d74025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.111+0000 7f1d87fff700 1 -- 172.21.15.16:0/2493282147 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f1d80045d90 con 0x7f1d90135450 2019-12-05T06:52:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.111+0000 7f1d8f59e700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023670 0x7f1d74025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:00.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.111+0000 7f1d8f59e700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023670 0x7f1d74025ae0 crc :-1 s=READY pgs=593 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:00.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.125+0000 7f1d95253700 1 -- 172.21.15.16:0/2493282147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d90133b80 con 0x7f1d90135450 2019-12-05T06:52:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.134+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.134+0000 7f1d87fff700 1 -- 172.21.15.16:0/2493282147 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1d80002c30 con 0x7f1d90135450 2019-12-05T06:52:00.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.389+0000 7f1d95253700 1 -- 172.21.15.16:0/2493282147 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1d90133b80 con 0x7f1d90135450 2019-12-05T06:52:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.390+0000 7f1d94a52700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=THROTTLE_DONE pgs=958 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:00.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.390+0000 7f1d87fff700 1 -- 172.21.15.16:0/2493282147 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f1d800097a0 con 0x7f1d90135450 2019-12-05T06:52:00.392 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:00.393 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:00.393 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 172 2019-12-05T06:52:00.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.394+0000 7f1d95253700 1 -- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023670 msgr2=0x7f1d74025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:00.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.394+0000 7f1d95253700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023670 0x7f1d74025ae0 crc :-1 s=READY pgs=593 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:00.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.394+0000 7f1d95253700 1 -- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 msgr2=0x7f1d90134c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:00.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.394+0000 7f1d95253700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 0x7f1d90134c80 secure :-1 s=READY pgs=958 cs=0 l=1 rx=0x7f1d800103f0 tx=0x7f1d80010790).stop 2019-12-05T06:52:00.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.394+0000 7f1d95253700 1 -- 172.21.15.16:0/2493282147 shutdown_connections 2019-12-05T06:52:00.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.394+0000 7f1d95253700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1d74023670 0x7f1d74025ae0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:00.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.394+0000 7f1d95253700 1 --2- 172.21.15.16:0/2493282147 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1d90135450 0x7f1d90134c80 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:00.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.394+0000 7f1d95253700 1 -- 172.21.15.16:0/2493282147 shutdown_connections 2019-12-05T06:52:00.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.394+0000 7f1d95253700 1 -- 172.21.15.16:0/2493282147 wait complete. 2019-12-05T06:52:00.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:00.394+0000 7f1d95253700 1 -- 172.21.15.16:0/2493282147 >> 172.21.15.16:0/2493282147 conn(0x7f1d900a2850 msgr2=0x7f1d9006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:00.406 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:52:02.407 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:02.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.667+0000 7f5001145700 1 Processor -- start 2019-12-05T06:52:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.667+0000 7f5001145700 1 -- start start 2019-12-05T06:52:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.669+0000 7f5001145700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 0x7f4ffc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.669+0000 7f5001145700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc0958e0 con 0x7f4ffc08fef0 2019-12-05T06:52:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.669+0000 7f5001145700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc0629e0 con 0x7f4ffc08f720 2019-12-05T06:52:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.669+0000 7f5001145700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc0a3f70 con 0x7f4ffc08c6d0 2019-12-05T06:52:02.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.669+0000 7f4ffad9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4ffc08fef0 0x7f4ffc08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47720/0 (socket says 172.21.15.16:47720) 2019-12-05T06:52:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.669+0000 7f4ffad9d700 1 -- 172.21.15.16:0/2383195318 learned_addr learned my addr 172.21.15.16:0/2383195318 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.669+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/2383195318 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 0x7f4ffc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.670+0000 7f4ffa59c700 1 -- 172.21.15.16:0/2383195318 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3896438288 0 0) 0x7f4fe4001150 con 0x7f4ffc08fef0 2019-12-05T06:52:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.670+0000 7f4ffa59c700 1 -- 172.21.15.16:0/2383195318 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe8002e80 con 0x7f4ffc08fef0 2019-12-05T06:52:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.670+0000 7f4ffa59c700 1 -- 172.21.15.16:0/2383195318 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 569432358 0 0) 0x7f4fec0009b0 con 0x7f4ffc08f720 2019-12-05T06:52:02.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.670+0000 7f4ffa59c700 1 -- 172.21.15.16:0/2383195318 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe80030b0 con 0x7f4ffc08f720 2019-12-05T06:52:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.670+0000 7f4ffb59e700 1 -- 172.21.15.16:0/2383195318 >> v1:172.21.15.76:6789/0 conn(0x7f4ffc08f720 legacy=0x7f4ffc1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.670+0000 7f4ffb59e700 1 -- 172.21.15.16:0/2383195318 >> v1:172.21.15.16:6789/0 conn(0x7f4ffc08fef0 legacy=0x7f4ffc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.670+0000 7f4ffb59e700 1 -- 172.21.15.16:0/2383195318 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc08fbc0 con 0x7f4ffc08c6d0 2019-12-05T06:52:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.670+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/2383195318 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 0x7f4ffc08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f4ff400b2d0 tx=0x7f4ff400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.671+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/2383195318 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 0x7f4ffc08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f4ff400b2d0 tx=0x7f4ff400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.671+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/2383195318 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 0x7f4ffc08cac0 secure :-1 s=READY pgs=329 cs=0 l=1 rx=0x7f4ff400b2d0 tx=0x7f4ff400bae0).ready entity=mon.2 client_cookie=145e8352408583e5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.671+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/2383195318 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 0x7f4ffc08cac0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f4ff400b2d0 tx=0x7f4ff400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.671+0000 7f4ffa59c700 1 -- 172.21.15.16:0/2383195318 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ff4000ce0 con 0x7f4ffc08c6d0 2019-12-05T06:52:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.671+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/2383195318 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 0x7f4ffc08cac0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f4ff400b2d0 tx=0x7f4ff400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.671+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/2383195318 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 0x7f4ffc08cac0 secure :-1 s=THROTTLE_DONE pgs=329 cs=0 l=1 rx=0x7f4ff400b2d0 tx=0x7f4ff400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.671+0000 7f4ffa59c700 1 -- 172.21.15.16:0/2383195318 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff4009be0 con 0x7f4ffc08c6d0 2019-12-05T06:52:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.671+0000 7f4ffa59c700 1 -- 172.21.15.16:0/2383195318 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ff400a2b0 con 0x7f4ffc08c6d0 2019-12-05T06:52:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.671+0000 7f5001145700 1 -- 172.21.15.16:0/2383195318 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 msgr2=0x7f4ffc08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:02.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.671+0000 7f5001145700 1 --2- 172.21.15.16:0/2383195318 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 0x7f4ffc08cac0 secure :-1 s=READY pgs=329 cs=0 l=1 rx=0x7f4ff400b2d0 tx=0x7f4ff400bae0).stop 2019-12-05T06:52:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.672+0000 7f5001145700 1 -- 172.21.15.16:0/2383195318 shutdown_connections 2019-12-05T06:52:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.672+0000 7f5001145700 1 --2- 172.21.15.16:0/2383195318 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc08c6d0 0x7f4ffc08cac0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.672+0000 7f5001145700 1 -- 172.21.15.16:0/2383195318 shutdown_connections 2019-12-05T06:52:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.672+0000 7f5001145700 1 -- 172.21.15.16:0/2383195318 wait complete. 2019-12-05T06:52:02.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.672+0000 7f5001145700 1 -- 172.21.15.16:0/2383195318 >> 172.21.15.16:0/2383195318 conn(0x7f4ffc061940 msgr2=0x7f4ffc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.677+0000 7f5001145700 1 Processor -- start 2019-12-05T06:52:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.678+0000 7f5001145700 1 -- start start 2019-12-05T06:52:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.679+0000 7f5001145700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:02.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.679+0000 7f5001145700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc09e2c0 0x7f4ffc09d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.679+0000 7f5001145700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ffc06e250 con 0x7f4ffc09d8e0 2019-12-05T06:52:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.679+0000 7f5001145700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ffc12b3f0 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.679+0000 7f5001145700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ffc12a9c0 con 0x7f4ffc09e2c0 2019-12-05T06:52:02.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.679+0000 7f4ffb59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.679+0000 7f4ffb59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34324/0 (socket says 172.21.15.16:34324) 2019-12-05T06:52:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.679+0000 7f4ffb59e700 1 -- 172.21.15.16:0/1764903894 learned_addr learned my addr 172.21.15.16:0/1764903894 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.679+0000 7f4ffad9d700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc09e2c0 0x7f4ffc09d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ffb59e700 1 -- 172.21.15.16:0/1764903894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc09e2c0 msgr2=0x7f4ffc09d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:02.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc09e2c0 0x7f4ffc09d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ffb59e700 1 -- 172.21.15.16:0/1764903894 >> v1:172.21.15.76:6789/0 conn(0x7f4ffc09d8e0 legacy=0x7f4ffc098180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ffb59e700 1 -- 172.21.15.16:0/1764903894 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ffc12ba80 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> v2:172.21.15.16:3300/0 conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> v2:172.21.15.16:3300/0 conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=READY pgs=1410 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:02.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=THROTTLE_DONE pgs=1410 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=THROTTLE_DONE pgs=1410 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ff3fff700 1 -- 172.21.15.16:0/1764903894 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ff4000cc0 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.680+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=THROTTLE_DONE pgs=1410 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.681+0000 7f4ff3fff700 1 -- 172.21.15.16:0/1764903894 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ff4009d00 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.681+0000 7f5001145700 1 -- 172.21.15.16:0/1764903894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ffc12ba80 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.681+0000 7f4ff3fff700 1 -- 172.21.15.16:0/1764903894 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ff400a450 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.681+0000 7f5001145700 1 -- 172.21.15.16:0/1764903894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ffc12ba80 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.682+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=THROTTLE_DONE pgs=1410 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.682+0000 7f4ff3fff700 1 -- 172.21.15.16:0/1764903894 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4ff4009900 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.682+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=THROTTLE_DONE pgs=1410 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.682+0000 7f4ff3fff700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fe0023630 0x7f4fe0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:02.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.682+0000 7f4ff3fff700 1 -- 172.21.15.16:0/1764903894 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f4ff4046160 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.683+0000 7f4ffad9d700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fe0023630 0x7f4fe0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:02.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.683+0000 7f4ffad9d700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fe0023630 0x7f4fe0025ac0 crc :-1 s=READY pgs=594 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:02.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.699+0000 7f5001145700 1 -- 172.21.15.16:0/1764903894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ffc12ba80 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.705+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=THROTTLE_DONE pgs=1410 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.705+0000 7f4ff3fff700 1 -- 172.21.15.16:0/1764903894 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4ff400a730 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.925+0000 7f5001145700 1 -- 172.21.15.16:0/1764903894 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ffc12ba80 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.926+0000 7f4ffb59e700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=THROTTLE_DONE pgs=1410 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.926+0000 7f4ff3fff700 1 -- 172.21.15.16:0/1764903894 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f4ff4009900 con 0x7f4ffc14d5d0 2019-12-05T06:52:02.926 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 172 2019-12-05T06:52:02.927 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:02.927 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:02.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.928+0000 7f5001145700 1 -- 172.21.15.16:0/1764903894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fe0023630 msgr2=0x7f4fe0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:02.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.928+0000 7f5001145700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fe0023630 0x7f4fe0025ac0 crc :-1 s=READY pgs=594 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:02.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.928+0000 7f5001145700 1 -- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 msgr2=0x7f4ffc14d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:02.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.928+0000 7f5001145700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 secure :-1 s=READY pgs=1410 cs=0 l=1 rx=0x7f4ff40107d0 tx=0x7f4ff4010650).stop 2019-12-05T06:52:02.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.928+0000 7f5001145700 1 -- 172.21.15.16:0/1764903894 shutdown_connections 2019-12-05T06:52:02.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.928+0000 7f5001145700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4ffc09e2c0 0x7f4ffc09d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:02.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.928+0000 7f5001145700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f4fe0023630 0x7f4fe0025ac0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:02.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.928+0000 7f5001145700 1 --2- 172.21.15.16:0/1764903894 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f4ffc14d5d0 0x7f4ffc14d9c0 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.929+0000 7f5001145700 1 -- 172.21.15.16:0/1764903894 shutdown_connections 2019-12-05T06:52:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.929+0000 7f5001145700 1 -- 172.21.15.16:0/1764903894 wait complete. 2019-12-05T06:52:02.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:02.929+0000 7f5001145700 1 -- 172.21.15.16:0/1764903894 >> 172.21.15.16:0/1764903894 conn(0x7f4ffc06d260 msgr2=0x7f4ffc09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:02.939 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:52:04.940 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:05.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.201+0000 7f4768663700 1 Processor -- start 2019-12-05T06:52:05.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.201+0000 7f4768663700 1 -- start start 2019-12-05T06:52:05.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.203+0000 7f4768663700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f476008d2c0 0x7f476008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:05.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.203+0000 7f4768663700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47600958e0 con 0x7f476008fef0 2019-12-05T06:52:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.203+0000 7f4768663700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47600629e0 con 0x7f476008c6d0 2019-12-05T06:52:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.203+0000 7f4768663700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f476008df90 con 0x7f476008d2c0 2019-12-05T06:52:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.203+0000 7f47663ff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f476008d2c0 0x7f476008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.204+0000 7f4767401700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f476008fef0 0x7f476008f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47738/0 (socket says 172.21.15.16:47738) 2019-12-05T06:52:05.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.204+0000 7f4767401700 1 -- 172.21.15.16:0/521947555 learned_addr learned my addr 172.21.15.16:0/521947555 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.204+0000 7f4766c00700 1 --1- 172.21.15.16:0/521947555 >> v1:172.21.15.76:6789/0 conn(0x7f476008c6d0 0x7f476008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:40804/0 (socket says 172.21.15.16:40804) 2019-12-05T06:52:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.204+0000 7f4765bfe700 1 -- 172.21.15.16:0/521947555 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891722702 0 0) 0x7f4758001100 con 0x7f476008fef0 2019-12-05T06:52:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.204+0000 7f4765bfe700 1 -- 172.21.15.16:0/521947555 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4744002e80 con 0x7f476008fef0 2019-12-05T06:52:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.204+0000 7f47663ff700 1 -- 172.21.15.16:0/521947555 >> v1:172.21.15.76:6789/0 conn(0x7f476008c6d0 legacy=0x7f476008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.204+0000 7f4765bfe700 1 -- 172.21.15.16:0/521947555 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 998067362 0 0) 0x7f4750001080 con 0x7f476008c6d0 2019-12-05T06:52:05.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.205+0000 7f47663ff700 1 -- 172.21.15.16:0/521947555 >> v1:172.21.15.16:6789/0 conn(0x7f476008fef0 legacy=0x7f476008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.205+0000 7f47663ff700 1 -- 172.21.15.16:0/521947555 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f476008e620 con 0x7f476008d2c0 2019-12-05T06:52:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.205+0000 7f47663ff700 1 --2- 172.21.15.16:0/521947555 >> v2:172.21.15.76:3300/0 conn(0x7f476008d2c0 0x7f476008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f474c00b2d0 tx=0x7f474c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.205+0000 7f47663ff700 1 --2- 172.21.15.16:0/521947555 >> v2:172.21.15.76:3300/0 conn(0x7f476008d2c0 0x7f476008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f474c00b2d0 tx=0x7f474c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.205+0000 7f47663ff700 1 --2- 172.21.15.16:0/521947555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f476008d2c0 0x7f476008ef20 secure :-1 s=READY pgs=962 cs=0 l=1 rx=0x7f474c00b2d0 tx=0x7f474c00bae0).ready entity=mon.1 client_cookie=d3aacef9974cea3a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.205+0000 7f47663ff700 1 --2- 172.21.15.16:0/521947555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f476008d2c0 0x7f476008ef20 secure :-1 s=THROTTLE_DONE pgs=962 cs=0 l=1 rx=0x7f474c00b2d0 tx=0x7f474c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.205+0000 7f4765bfe700 1 -- 172.21.15.16:0/521947555 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f474c000ce0 con 0x7f476008d2c0 2019-12-05T06:52:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.205+0000 7f47663ff700 1 --2- 172.21.15.16:0/521947555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f476008d2c0 0x7f476008ef20 secure :-1 s=THROTTLE_DONE pgs=962 cs=0 l=1 rx=0x7f474c00b2d0 tx=0x7f474c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.205+0000 7f47663ff700 1 --2- 172.21.15.16:0/521947555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f476008d2c0 0x7f476008ef20 secure :-1 s=THROTTLE_DONE pgs=962 cs=0 l=1 rx=0x7f474c00b2d0 tx=0x7f474c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.206+0000 7f4765bfe700 1 -- 172.21.15.16:0/521947555 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f474c009be0 con 0x7f476008d2c0 2019-12-05T06:52:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.206+0000 7f4765bfe700 1 -- 172.21.15.16:0/521947555 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f474c00a2b0 con 0x7f476008d2c0 2019-12-05T06:52:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.206+0000 7f4768663700 1 -- 172.21.15.16:0/521947555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f476008d2c0 msgr2=0x7f476008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.206+0000 7f4768663700 1 --2- 172.21.15.16:0/521947555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f476008d2c0 0x7f476008ef20 secure :-1 s=READY pgs=962 cs=0 l=1 rx=0x7f474c00b2d0 tx=0x7f474c00bae0).stop 2019-12-05T06:52:05.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.208+0000 7f4768663700 1 -- 172.21.15.16:0/521947555 shutdown_connections 2019-12-05T06:52:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.208+0000 7f4768663700 1 --2- 172.21.15.16:0/521947555 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f476008d2c0 0x7f476008ef20 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.208+0000 7f4768663700 1 -- 172.21.15.16:0/521947555 shutdown_connections 2019-12-05T06:52:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.208+0000 7f4768663700 1 -- 172.21.15.16:0/521947555 wait complete. 2019-12-05T06:52:05.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.208+0000 7f4768663700 1 -- 172.21.15.16:0/521947555 >> 172.21.15.16:0/521947555 conn(0x7f4760061940 msgr2=0x7f4760096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.213+0000 7f4768663700 1 Processor -- start 2019-12-05T06:52:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.213+0000 7f4768663700 1 -- start start 2019-12-05T06:52:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.215+0000 7f4768663700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47601302f0 0x7f476012a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:05.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.215+0000 7f4768663700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f4760153ba0 0x7f4760155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:05.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.215+0000 7f4768663700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4760131bf0 con 0x7f476012b640 2019-12-05T06:52:05.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.215+0000 7f4768663700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4760128cb0 con 0x7f4760153ba0 2019-12-05T06:52:05.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.215+0000 7f4768663700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4760128f80 con 0x7f47601302f0 2019-12-05T06:52:05.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.215+0000 7f47663ff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f476012b640 0x7f476012abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47744/0 (socket says 172.21.15.16:47744) 2019-12-05T06:52:05.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.215+0000 7f47663ff700 1 -- 172.21.15.16:0/1522736653 learned_addr learned my addr 172.21.15.16:0/1522736653 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:05.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.215+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> v2:172.21.15.76:3300/0 conn(0x7f4760153ba0 0x7f4760155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:05.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.215+0000 7f4766c00700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47601302f0 0x7f476012a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:05.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.216+0000 7f475f7fe700 1 -- 172.21.15.16:0/1522736653 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 728242476 0 0) 0x7f474c00afa0 con 0x7f476012b640 2019-12-05T06:52:05.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.216+0000 7f475f7fe700 1 -- 172.21.15.16:0/1522736653 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f473c002e80 con 0x7f476012b640 2019-12-05T06:52:05.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.216+0000 7f4767401700 1 -- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47601302f0 msgr2=0x7f476012a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:05.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.216+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47601302f0 0x7f476012a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:05.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.216+0000 7f4767401700 1 -- 172.21.15.16:0/1522736653 >> v1:172.21.15.16:6789/0 conn(0x7f476012b640 legacy=0x7f476012abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:05.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.216+0000 7f4767401700 1 -- 172.21.15.16:0/1522736653 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4760129d70 con 0x7f4760153ba0 2019-12-05T06:52:05.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.216+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> v2:172.21.15.76:3300/0 conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.216+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> v2:172.21.15.76:3300/0 conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.216+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=READY pgs=963 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:05.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.217+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=THROTTLE_DONE pgs=963 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.217+0000 7f475f7fe700 1 -- 172.21.15.16:0/1522736653 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4758000ef0 con 0x7f4760153ba0 2019-12-05T06:52:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.217+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=THROTTLE_DONE pgs=963 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.217+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=THROTTLE_DONE pgs=963 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.217+0000 7f475f7fe700 1 -- 172.21.15.16:0/1522736653 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4758000ef0 con 0x7f4760153ba0 2019-12-05T06:52:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.217+0000 7f4768663700 1 -- 172.21.15.16:0/1522736653 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4760129d70 con 0x7f4760153ba0 2019-12-05T06:52:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.217+0000 7f475f7fe700 1 -- 172.21.15.16:0/1522736653 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4758004430 con 0x7f4760153ba0 2019-12-05T06:52:05.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.217+0000 7f4768663700 1 -- 172.21.15.16:0/1522736653 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4760129d70 con 0x7f4760153ba0 2019-12-05T06:52:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.218+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=THROTTLE_DONE pgs=963 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.218+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=THROTTLE_DONE pgs=963 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.218+0000 7f475f7fe700 1 -- 172.21.15.16:0/1522736653 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4758004240 con 0x7f4760153ba0 2019-12-05T06:52:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.219+0000 7f475f7fe700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f473c023670 0x7f473c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.219+0000 7f475f7fe700 1 -- 172.21.15.16:0/1522736653 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f4758045f90 con 0x7f4760153ba0 2019-12-05T06:52:05.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.219+0000 7f4766c00700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f473c023670 0x7f473c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:05.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.220+0000 7f4766c00700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f473c023670 0x7f473c025ae0 crc :-1 s=READY pgs=595 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:05.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.231+0000 7f4768663700 1 -- 172.21.15.16:0/1522736653 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4760129d70 con 0x7f4760153ba0 2019-12-05T06:52:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.241+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=THROTTLE_DONE pgs=963 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.241+0000 7f475f7fe700 1 -- 172.21.15.16:0/1522736653 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4758004240 con 0x7f4760153ba0 2019-12-05T06:52:05.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.483+0000 7f4768663700 1 -- 172.21.15.16:0/1522736653 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4760129d70 con 0x7f4760153ba0 2019-12-05T06:52:05.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.484+0000 7f4767401700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=THROTTLE_DONE pgs=963 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:05.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.484+0000 7f475f7fe700 1 -- 172.21.15.16:0/1522736653 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f4758000ef0 con 0x7f4760153ba0 2019-12-05T06:52:05.485 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 172 2019-12-05T06:52:05.486 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:05.486 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:05.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 -- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f473c023670 msgr2=0x7f473c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:05.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f473c023670 0x7f473c025ae0 crc :-1 s=READY pgs=595 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:05.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 -- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 msgr2=0x7f4760155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:05.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 0x7f4760155fa0 secure :-1 s=READY pgs=963 cs=0 l=1 rx=0x7f4758010390 tx=0x7f47580101c0).stop 2019-12-05T06:52:05.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 -- 172.21.15.16:0/1522736653 shutdown_connections 2019-12-05T06:52:05.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f473c023670 0x7f473c025ae0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:05.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f4760153ba0 0x7f4760155fa0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:05.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 --2- 172.21.15.16:0/1522736653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f47601302f0 0x7f476012a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:05.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 -- 172.21.15.16:0/1522736653 shutdown_connections 2019-12-05T06:52:05.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 -- 172.21.15.16:0/1522736653 wait complete. 2019-12-05T06:52:05.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:05.487+0000 7f4768663700 1 -- 172.21.15.16:0/1522736653 >> 172.21.15.16:0/1522736653 conn(0x7f47600a2850 msgr2=0x7f476006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:05.502 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:52:07.503 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:07.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.743+0000 7f64829c6700 1 Processor -- start 2019-12-05T06:52:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.743+0000 7f64829c6700 1 -- start start 2019-12-05T06:52:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.744+0000 7f64829c6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f647c08d2c0 0x7f647c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.744+0000 7f64829c6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f647c08fef0 0x7f647c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f64829c6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c0958e0 con 0x7f647c08c6d0 2019-12-05T06:52:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f64829c6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647c08df90 con 0x7f647c08fef0 2019-12-05T06:52:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f64829c6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647c0a3f70 con 0x7f647c08d2c0 2019-12-05T06:52:07.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f6481764700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f647c08fef0 0x7f647c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:07.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f6481764700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f647c08fef0 0x7f647c08f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34356/0 (socket says 172.21.15.16:34356) 2019-12-05T06:52:07.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f6481764700 1 -- 172.21.15.16:0/3961585668 learned_addr learned my addr 172.21.15.16:0/3961585668 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:07.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f647bfff700 1 --2- 172.21.15.16:0/3961585668 >> v2:172.21.15.76:3300/0 conn(0x7f647c08d2c0 0x7f647c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:07.760 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f6481764700 1 -- 172.21.15.16:0/3961585668 >> v2:172.21.15.76:3300/0 conn(0x7f647c08d2c0 msgr2=0x7f647c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:07.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f6481764700 1 --2- 172.21.15.16:0/3961585668 >> v2:172.21.15.76:3300/0 conn(0x7f647c08d2c0 0x7f647c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:07.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f6481764700 1 -- 172.21.15.16:0/3961585668 >> v1:172.21.15.76:6789/0 conn(0x7f647c08c6d0 legacy=0x7f647c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:07.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.745+0000 7f6481764700 1 -- 172.21.15.16:0/3961585668 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c08e620 con 0x7f647c08fef0 2019-12-05T06:52:07.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f6481764700 1 --2- 172.21.15.16:0/3961585668 >> v2:172.21.15.16:3300/0 conn(0x7f647c08fef0 0x7f647c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6474008950 tx=0x7f64740087a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f6481764700 1 --2- 172.21.15.16:0/3961585668 >> v2:172.21.15.16:3300/0 conn(0x7f647c08fef0 0x7f647c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6474008950 tx=0x7f64740087a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f6481764700 1 --2- 172.21.15.16:0/3961585668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c08fef0 0x7f647c08f460 secure :-1 s=READY pgs=1413 cs=0 l=1 rx=0x7f6474008950 tx=0x7f64740087a0).ready entity=mon.0 client_cookie=df135aba63091eb8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:07.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f6481764700 1 --2- 172.21.15.16:0/3961585668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c08fef0 0x7f647c08f460 secure :-1 s=THROTTLE_DONE pgs=1413 cs=0 l=1 rx=0x7f6474008950 tx=0x7f64740087a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f6481764700 1 --2- 172.21.15.16:0/3961585668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c08fef0 0x7f647c08f460 secure :-1 s=THROTTLE_DONE pgs=1413 cs=0 l=1 rx=0x7f6474008950 tx=0x7f64740087a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f647b7fe700 1 -- 172.21.15.16:0/3961585668 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6474000f40 con 0x7f647c08fef0 2019-12-05T06:52:07.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f6481764700 1 --2- 172.21.15.16:0/3961585668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c08fef0 0x7f647c08f460 secure :-1 s=THROTTLE_DONE pgs=1413 cs=0 l=1 rx=0x7f6474008950 tx=0x7f64740087a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f647b7fe700 1 -- 172.21.15.16:0/3961585668 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6474001460 con 0x7f647c08fef0 2019-12-05T06:52:07.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f647b7fe700 1 -- 172.21.15.16:0/3961585668 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6474001bb0 con 0x7f647c08fef0 2019-12-05T06:52:07.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f64829c6700 1 -- 172.21.15.16:0/3961585668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c08fef0 msgr2=0x7f647c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:07.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.746+0000 7f64829c6700 1 --2- 172.21.15.16:0/3961585668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c08fef0 0x7f647c08f460 secure :-1 s=READY pgs=1413 cs=0 l=1 rx=0x7f6474008950 tx=0x7f64740087a0).stop 2019-12-05T06:52:07.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.748+0000 7f64829c6700 1 -- 172.21.15.16:0/3961585668 shutdown_connections 2019-12-05T06:52:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.748+0000 7f64829c6700 1 --2- 172.21.15.16:0/3961585668 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f647c08fef0 0x7f647c08f460 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.748+0000 7f64829c6700 1 --2- 172.21.15.16:0/3961585668 >> v2:172.21.15.76:3300/0 conn(0x7f647c08d2c0 0x7f647c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.748+0000 7f64829c6700 1 -- 172.21.15.16:0/3961585668 shutdown_connections 2019-12-05T06:52:07.767 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.748+0000 7f64829c6700 1 -- 172.21.15.16:0/3961585668 wait complete. 2019-12-05T06:52:07.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.748+0000 7f64829c6700 1 -- 172.21.15.16:0/3961585668 >> 172.21.15.16:0/3961585668 conn(0x7f647c061940 msgr2=0x7f647c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:07.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.752+0000 7f64829c6700 1 Processor -- start 2019-12-05T06:52:07.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.752+0000 7f64829c6700 1 -- start start 2019-12-05T06:52:07.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.753+0000 7f64829c6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f647c1570c0 0x7f647c1574b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:07.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.753+0000 7f64829c6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:07.768 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.753+0000 7f64829c6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f647c0887e0 con 0x7f647c088b70 2019-12-05T06:52:07.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.753+0000 7f64829c6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f647c086c10 con 0x7f647c1570c0 2019-12-05T06:52:07.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.753+0000 7f64829c6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f647c0861e0 con 0x7f647c0883a0 2019-12-05T06:52:07.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.753+0000 7f647bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f647c088b70 0x7f647c087ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47764/0 (socket says 172.21.15.16:47764) 2019-12-05T06:52:07.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.753+0000 7f647bfff700 1 -- 172.21.15.16:0/3732940991 learned_addr learned my addr 172.21.15.16:0/3732940991 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:07.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.753+0000 7f6480f63700 1 --2- 172.21.15.16:0/3732940991 >> v2:172.21.15.76:3300/0 conn(0x7f647c1570c0 0x7f647c1574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:07.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.753+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:07.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.754+0000 7f64797fa700 1 -- 172.21.15.16:0/3732940991 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 980452153 0 0) 0x7f64640010b0 con 0x7f647c088b70 2019-12-05T06:52:07.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.754+0000 7f64797fa700 1 -- 172.21.15.16:0/3732940991 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6460002e80 con 0x7f647c088b70 2019-12-05T06:52:07.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.754+0000 7f6481764700 1 -- 172.21.15.16:0/3732940991 >> v2:172.21.15.76:3300/0 conn(0x7f647c1570c0 msgr2=0x7f647c1574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:07.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.754+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> v2:172.21.15.76:3300/0 conn(0x7f647c1570c0 0x7f647c1574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:07.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.754+0000 7f6481764700 1 -- 172.21.15.16:0/3732940991 >> v1:172.21.15.16:6789/0 conn(0x7f647c088b70 legacy=0x7f647c087ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:07.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.754+0000 7f6481764700 1 -- 172.21.15.16:0/3732940991 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f647c0872a0 con 0x7f647c0883a0 2019-12-05T06:52:07.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.754+0000 7f6480f63700 1 --2- 172.21.15.16:0/3732940991 >> v2:172.21.15.76:3300/0 conn(0x7f647c1570c0 0x7f647c1574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:52:07.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.754+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.754+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.754+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=READY pgs=330 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:07.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.755+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.755+0000 7f64797fa700 1 -- 172.21.15.16:0/3732940991 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6474000f40 con 0x7f647c0883a0 2019-12-05T06:52:07.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.755+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.755+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.755+0000 7f64797fa700 1 -- 172.21.15.16:0/3732940991 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6474001270 con 0x7f647c0883a0 2019-12-05T06:52:07.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.755+0000 7f64829c6700 1 -- 172.21.15.16:0/3732940991 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f647c0872a0 con 0x7f647c0883a0 2019-12-05T06:52:07.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.755+0000 7f64797fa700 1 -- 172.21.15.16:0/3732940991 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6474001900 con 0x7f647c0883a0 2019-12-05T06:52:07.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.755+0000 7f64829c6700 1 -- 172.21.15.16:0/3732940991 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f647c0872a0 con 0x7f647c0883a0 2019-12-05T06:52:07.773 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.756+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.756+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.756+0000 7f64797fa700 1 -- 172.21.15.16:0/3732940991 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6474001460 con 0x7f647c0883a0 2019-12-05T06:52:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.757+0000 7f64797fa700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6460023670 0x7f6460025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.757+0000 7f64797fa700 1 -- 172.21.15.16:0/3732940991 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f6474046330 con 0x7f647c0883a0 2019-12-05T06:52:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.757+0000 7f6480f63700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6460023670 0x7f6460025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:07.774 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.758+0000 7f6480f63700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6460023670 0x7f6460025ae0 crc :-1 s=READY pgs=596 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.759+0000 7f64829c6700 1 -- 172.21.15.16:0/3732940991 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f647c0872a0 con 0x7f647c0883a0 2019-12-05T06:52:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.768+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.775 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.768+0000 7f64797fa700 1 -- 172.21.15.16:0/3732940991 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6474000cc0 con 0x7f647c0883a0 2019-12-05T06:52:07.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.977+0000 7f64829c6700 1 -- 172.21.15.16:0/3732940991 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f647c0872a0 con 0x7f647c0883a0 2019-12-05T06:52:07.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.978+0000 7f6481764700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=THROTTLE_DONE pgs=330 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:07.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.978+0000 7f64797fa700 1 -- 172.21.15.16:0/3732940991 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f6474001b20 con 0x7f647c0883a0 2019-12-05T06:52:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 172 2019-12-05T06:52:07.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.984+0000 7f64829c6700 1 -- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6460023670 msgr2=0x7f6460025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.984+0000 7f64829c6700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6460023670 0x7f6460025ae0 crc :-1 s=READY pgs=596 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.984+0000 7f64829c6700 1 -- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 msgr2=0x7f647c12d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:07.985 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.984+0000 7f64829c6700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 secure :-1 s=READY pgs=330 cs=0 l=1 rx=0x7f6474010730 tx=0x7f6474010ad0).stop 2019-12-05T06:52:07.985 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:07.986 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.984+0000 7f64829c6700 1 -- 172.21.15.16:0/3732940991 shutdown_connections 2019-12-05T06:52:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.984+0000 7f64829c6700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6460023670 0x7f6460025ae0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.984+0000 7f64829c6700 1 --2- 172.21.15.16:0/3732940991 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f647c0883a0 0x7f647c12d730 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.984+0000 7f64829c6700 1 --2- 172.21.15.16:0/3732940991 >> v2:172.21.15.76:3300/0 conn(0x7f647c1570c0 0x7f647c1574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.985+0000 7f64829c6700 1 -- 172.21.15.16:0/3732940991 shutdown_connections 2019-12-05T06:52:07.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.985+0000 7f64829c6700 1 -- 172.21.15.16:0/3732940991 wait complete. 2019-12-05T06:52:07.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:07.985+0000 7f64829c6700 1 -- 172.21.15.16:0/3732940991 >> 172.21.15.16:0/3732940991 conn(0x7f647c0829c0 msgr2=0x7f647c0706d0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:08.000 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:52:10.001 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.254+0000 7f9b6dfff700 1 Processor -- start 2019-12-05T06:52:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.254+0000 7f9b6dfff700 1 -- start start 2019-12-05T06:52:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.256+0000 7f9b6dfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 0x7f9b6808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.256+0000 7f9b6dfff700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b680958e0 con 0x7f9b6808fef0 2019-12-05T06:52:10.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.256+0000 7f9b6dfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b680629e0 con 0x7f9b6808f720 2019-12-05T06:52:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.256+0000 7f9b6dfff700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b680a3f70 con 0x7f9b6808c6d0 2019-12-05T06:52:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.256+0000 7f9b677fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9b6808fef0 0x7f9b6808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47776/0 (socket says 172.21.15.16:47776) 2019-12-05T06:52:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.256+0000 7f9b677fe700 1 -- 172.21.15.16:0/3738947590 learned_addr learned my addr 172.21.15.16:0/3738947590 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.256+0000 7f9b67fff700 1 --2- 172.21.15.16:0/3738947590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 0x7f9b6808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.257+0000 7f9b66ffd700 1 -- 172.21.15.16:0/3738947590 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667036675 0 0) 0x7f9b50001150 con 0x7f9b6808fef0 2019-12-05T06:52:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.257+0000 7f9b66ffd700 1 -- 172.21.15.16:0/3738947590 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b54002e80 con 0x7f9b6808fef0 2019-12-05T06:52:10.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.257+0000 7f9b66ffd700 1 -- 172.21.15.16:0/3738947590 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 302001315 0 0) 0x7f9b600009b0 con 0x7f9b6808f720 2019-12-05T06:52:10.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.257+0000 7f9b66ffd700 1 -- 172.21.15.16:0/3738947590 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b54002e80 con 0x7f9b6808f720 2019-12-05T06:52:10.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.257+0000 7f9b67fff700 1 -- 172.21.15.16:0/3738947590 >> v1:172.21.15.76:6789/0 conn(0x7f9b6808f720 legacy=0x7f9b681533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:10.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.257+0000 7f9b67fff700 1 -- 172.21.15.16:0/3738947590 >> v1:172.21.15.16:6789/0 conn(0x7f9b6808fef0 legacy=0x7f9b6808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:10.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.257+0000 7f9b67fff700 1 -- 172.21.15.16:0/3738947590 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6808fbc0 con 0x7f9b6808c6d0 2019-12-05T06:52:10.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.257+0000 7f9b67fff700 1 --2- 172.21.15.16:0/3738947590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 0x7f9b6808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9b5800b2d0 tx=0x7f9b5800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.258+0000 7f9b67fff700 1 --2- 172.21.15.16:0/3738947590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 0x7f9b6808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9b5800b2d0 tx=0x7f9b5800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.258+0000 7f9b67fff700 1 --2- 172.21.15.16:0/3738947590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 0x7f9b6808cac0 secure :-1 s=READY pgs=332 cs=0 l=1 rx=0x7f9b5800b2d0 tx=0x7f9b5800bae0).ready entity=mon.2 client_cookie=93217322a59851ef server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:10.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.258+0000 7f9b67fff700 1 --2- 172.21.15.16:0/3738947590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 0x7f9b6808cac0 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7f9b5800b2d0 tx=0x7f9b5800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.258+0000 7f9b66ffd700 1 -- 172.21.15.16:0/3738947590 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b58000ce0 con 0x7f9b6808c6d0 2019-12-05T06:52:10.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.258+0000 7f9b67fff700 1 --2- 172.21.15.16:0/3738947590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 0x7f9b6808cac0 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7f9b5800b2d0 tx=0x7f9b5800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.258+0000 7f9b67fff700 1 --2- 172.21.15.16:0/3738947590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 0x7f9b6808cac0 secure :-1 s=THROTTLE_DONE pgs=332 cs=0 l=1 rx=0x7f9b5800b2d0 tx=0x7f9b5800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.258+0000 7f9b66ffd700 1 -- 172.21.15.16:0/3738947590 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b58009be0 con 0x7f9b6808c6d0 2019-12-05T06:52:10.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.258+0000 7f9b66ffd700 1 -- 172.21.15.16:0/3738947590 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b5800a2b0 con 0x7f9b6808c6d0 2019-12-05T06:52:10.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.258+0000 7f9b6dfff700 1 -- 172.21.15.16:0/3738947590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 msgr2=0x7f9b6808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:10.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.258+0000 7f9b6dfff700 1 --2- 172.21.15.16:0/3738947590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 0x7f9b6808cac0 secure :-1 s=READY pgs=332 cs=0 l=1 rx=0x7f9b5800b2d0 tx=0x7f9b5800bae0).stop 2019-12-05T06:52:10.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.260+0000 7f9b6dfff700 1 -- 172.21.15.16:0/3738947590 shutdown_connections 2019-12-05T06:52:10.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.260+0000 7f9b6dfff700 1 --2- 172.21.15.16:0/3738947590 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b6808c6d0 0x7f9b6808cac0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:10.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.260+0000 7f9b6dfff700 1 -- 172.21.15.16:0/3738947590 shutdown_connections 2019-12-05T06:52:10.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.260+0000 7f9b6dfff700 1 -- 172.21.15.16:0/3738947590 wait complete. 2019-12-05T06:52:10.268 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.260+0000 7f9b6dfff700 1 -- 172.21.15.16:0/3738947590 >> 172.21.15.16:0/3738947590 conn(0x7f9b68061940 msgr2=0x7f9b68096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.265+0000 7f9b6dfff700 1 Processor -- start 2019-12-05T06:52:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.265+0000 7f9b6dfff700 1 -- start start 2019-12-05T06:52:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.267+0000 7f9b6dfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.267+0000 7f9b6dfff700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b6806e5c0 con 0x7f9b6812a1d0 2019-12-05T06:52:10.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.267+0000 7f9b6dfff700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b68098500 con 0x7f9b68129a00 2019-12-05T06:52:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.267+0000 7f9b6dfff700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9b6806e250 con 0x7f9b680a17b0 2019-12-05T06:52:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.267+0000 7f9b67fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.267+0000 7f9b6cd9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f9b68129a00 0x7f9b68153a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:40854/0 (socket says 172.21.15.16:40854) 2019-12-05T06:52:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.267+0000 7f9b677fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9b6812a1d0 0x7f9b68129200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47786/0 (socket says 172.21.15.16:47786) 2019-12-05T06:52:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.267+0000 7f9b67fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:38372/0 (socket says 172.21.15.16:38372) 2019-12-05T06:52:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.267+0000 7f9b6cd9d700 1 -- 172.21.15.16:0/1058169001 learned_addr learned my addr 172.21.15.16:0/1058169001 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:10.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.268+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480696552 0 0) 0x7f9b50001150 con 0x7f9b6812a1d0 2019-12-05T06:52:10.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.268+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b4c002e80 con 0x7f9b6812a1d0 2019-12-05T06:52:10.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.268+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607695867 0 0) 0x7f9b600010b0 con 0x7f9b68129a00 2019-12-05T06:52:10.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.268+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b4c004220 con 0x7f9b68129a00 2019-12-05T06:52:10.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.268+0000 7f9b67fff700 1 -- 172.21.15.16:0/1058169001 >> v1:172.21.15.76:6789/0 conn(0x7f9b68129a00 legacy=0x7f9b68153a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:10.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.268+0000 7f9b67fff700 1 -- 172.21.15.16:0/1058169001 >> v1:172.21.15.16:6789/0 conn(0x7f9b6812a1d0 legacy=0x7f9b68129200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:10.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.268+0000 7f9b67fff700 1 -- 172.21.15.16:0/1058169001 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b68128900 con 0x7f9b680a17b0 2019-12-05T06:52:10.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.268+0000 7f9b67fff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.268+0000 7f9b67fff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.269+0000 7f9b67fff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=READY pgs=333 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:10.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.269+0000 7f9b67fff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=THROTTLE_DONE pgs=333 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.269+0000 7f9b67fff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=THROTTLE_DONE pgs=333 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.269+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b58000cc0 con 0x7f9b680a17b0 2019-12-05T06:52:10.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.269+0000 7f9b67fff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=THROTTLE_DONE pgs=333 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.274 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.269+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9b58009d00 con 0x7f9b680a17b0 2019-12-05T06:52:10.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.269+0000 7f9b6dfff700 1 -- 172.21.15.16:0/1058169001 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b68128900 con 0x7f9b680a17b0 2019-12-05T06:52:10.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.269+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9b5800a450 con 0x7f9b680a17b0 2019-12-05T06:52:10.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.269+0000 7f9b6dfff700 1 -- 172.21.15.16:0/1058169001 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b68128900 con 0x7f9b680a17b0 2019-12-05T06:52:10.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.270+0000 7f9b67fff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=THROTTLE_DONE pgs=333 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.271+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9b58009900 con 0x7f9b680a17b0 2019-12-05T06:52:10.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.271+0000 7f9b67fff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=THROTTLE_DONE pgs=333 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.272+0000 7f9b64ff9700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b4c023650 0x7f9b4c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:10.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.272+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f9b58046150 con 0x7f9b680a17b0 2019-12-05T06:52:10.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.272+0000 7f9b677fe700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b4c023650 0x7f9b4c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:10.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.272+0000 7f9b677fe700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b4c023650 0x7f9b4c025ae0 crc :-1 s=READY pgs=597 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:10.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.286+0000 7f9b6dfff700 1 -- 172.21.15.16:0/1058169001 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b68128900 con 0x7f9b680a17b0 2019-12-05T06:52:10.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.295+0000 7f9b67fff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=THROTTLE_DONE pgs=333 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.295+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9b58000cc0 con 0x7f9b680a17b0 2019-12-05T06:52:10.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.537+0000 7f9b6dfff700 1 -- 172.21.15.16:0/1058169001 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9b68128900 con 0x7f9b680a17b0 2019-12-05T06:52:10.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.538+0000 7f9b67fff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=THROTTLE_DONE pgs=333 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:10.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.538+0000 7f9b64ff9700 1 -- 172.21.15.16:0/1058169001 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f9b58009900 con 0x7f9b680a17b0 2019-12-05T06:52:10.540 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:10.540 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:10.541 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 172 2019-12-05T06:52:10.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.542+0000 7f9b6dfff700 1 -- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b4c023650 msgr2=0x7f9b4c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.542+0000 7f9b6dfff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b4c023650 0x7f9b4c025ae0 crc :-1 s=READY pgs=597 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.542+0000 7f9b6dfff700 1 -- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 msgr2=0x7f9b680a1ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.542+0000 7f9b6dfff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 secure :-1 s=READY pgs=333 cs=0 l=1 rx=0x7f9b58010710 tx=0x7f9b58010ab0).stop 2019-12-05T06:52:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.542+0000 7f9b6dfff700 1 -- 172.21.15.16:0/1058169001 shutdown_connections 2019-12-05T06:52:10.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.542+0000 7f9b6dfff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9b4c023650 0x7f9b4c025ae0 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.542+0000 7f9b6dfff700 1 --2- 172.21.15.16:0/1058169001 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9b680a17b0 0x7f9b680a1ba0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.543+0000 7f9b6dfff700 1 -- 172.21.15.16:0/1058169001 shutdown_connections 2019-12-05T06:52:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.543+0000 7f9b6dfff700 1 -- 172.21.15.16:0/1058169001 wait complete. 2019-12-05T06:52:10.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:10.543+0000 7f9b6dfff700 1 -- 172.21.15.16:0/1058169001 >> 172.21.15.16:0/1058169001 conn(0x7f9b6806d260 msgr2=0x7f9b6809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:10.558 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:52:12.559 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.823+0000 7f227d2fd700 1 Processor -- start 2019-12-05T06:52:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.824+0000 7f227d2fd700 1 -- start start 2019-12-05T06:52:12.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.825+0000 7f227d2fd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f227808c6d0 0x7f227808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.825+0000 7f227d2fd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f227808f720 0x7f227808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.825+0000 7f227d2fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22780958e0 con 0x7f227808fef0 2019-12-05T06:52:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.825+0000 7f227d2fd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22780a3f70 con 0x7f227808f720 2019-12-05T06:52:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.825+0000 7f227d2fd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227808d830 con 0x7f227808c6d0 2019-12-05T06:52:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.825+0000 7f2277fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f227808f720 0x7f227808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:12.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.825+0000 7f2277fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f227808f720 0x7f227808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34392/0 (socket says 172.21.15.16:34392) 2019-12-05T06:52:12.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.825+0000 7f2277fff700 1 -- 172.21.15.16:0/4289691839 learned_addr learned my addr 172.21.15.16:0/4289691839 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:12.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.825+0000 7f22777fe700 1 --2- 172.21.15.16:0/4289691839 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f227808c6d0 0x7f227808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:12.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.826+0000 7f2277fff700 1 -- 172.21.15.16:0/4289691839 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f227808c6d0 msgr2=0x7f227808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:12.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.826+0000 7f2277fff700 1 --2- 172.21.15.16:0/4289691839 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f227808c6d0 0x7f227808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:12.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.826+0000 7f2277fff700 1 -- 172.21.15.16:0/4289691839 >> v1:172.21.15.16:6789/0 conn(0x7f227808fef0 legacy=0x7f227808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:12.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.826+0000 7f22767fc700 1 -- 172.21.15.16:0/4289691839 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 495792551 0 0) 0x7f2260001030 con 0x7f227808fef0 2019-12-05T06:52:12.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.826+0000 7f2277fff700 1 -- 172.21.15.16:0/4289691839 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f227808fbc0 con 0x7f227808f720 2019-12-05T06:52:12.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.826+0000 7f22777fe700 1 --2- 172.21.15.16:0/4289691839 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f227808c6d0 0x7f227808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:52:12.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.826+0000 7f2277fff700 1 --2- 172.21.15.16:0/4289691839 >> v2:172.21.15.16:3300/0 conn(0x7f227808f720 0x7f227808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f227000fab0 tx=0x7f227000fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.826+0000 7f2277fff700 1 --2- 172.21.15.16:0/4289691839 >> v2:172.21.15.16:3300/0 conn(0x7f227808f720 0x7f227808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f227000fab0 tx=0x7f227000fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.827+0000 7f2277fff700 1 --2- 172.21.15.16:0/4289691839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f227808f720 0x7f227808d000 secure :-1 s=READY pgs=1418 cs=0 l=1 rx=0x7f227000fab0 tx=0x7f227000fe50).ready entity=mon.0 client_cookie=b8c8c6f327dae79f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:12.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.827+0000 7f2277fff700 1 --2- 172.21.15.16:0/4289691839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f227808f720 0x7f227808d000 secure :-1 s=THROTTLE_DONE pgs=1418 cs=0 l=1 rx=0x7f227000fab0 tx=0x7f227000fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.827+0000 7f22767fc700 1 -- 172.21.15.16:0/4289691839 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2270000f10 con 0x7f227808f720 2019-12-05T06:52:12.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.827+0000 7f2277fff700 1 --2- 172.21.15.16:0/4289691839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f227808f720 0x7f227808d000 secure :-1 s=THROTTLE_DONE pgs=1418 cs=0 l=1 rx=0x7f227000fab0 tx=0x7f227000fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.827+0000 7f2277fff700 1 --2- 172.21.15.16:0/4289691839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f227808f720 0x7f227808d000 secure :-1 s=THROTTLE_DONE pgs=1418 cs=0 l=1 rx=0x7f227000fab0 tx=0x7f227000fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.827+0000 7f22767fc700 1 -- 172.21.15.16:0/4289691839 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2270001460 con 0x7f227808f720 2019-12-05T06:52:12.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.827+0000 7f22767fc700 1 -- 172.21.15.16:0/4289691839 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2270001bb0 con 0x7f227808f720 2019-12-05T06:52:12.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.827+0000 7f227d2fd700 1 -- 172.21.15.16:0/4289691839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f227808f720 msgr2=0x7f227808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:12.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.827+0000 7f227d2fd700 1 --2- 172.21.15.16:0/4289691839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f227808f720 0x7f227808d000 secure :-1 s=READY pgs=1418 cs=0 l=1 rx=0x7f227000fab0 tx=0x7f227000fe50).stop 2019-12-05T06:52:12.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.829+0000 7f227d2fd700 1 -- 172.21.15.16:0/4289691839 shutdown_connections 2019-12-05T06:52:12.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.829+0000 7f227d2fd700 1 --2- 172.21.15.16:0/4289691839 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f227808f720 0x7f227808d000 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:12.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.829+0000 7f227d2fd700 1 --2- 172.21.15.16:0/4289691839 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f227808c6d0 0x7f227808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:12.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.829+0000 7f227d2fd700 1 -- 172.21.15.16:0/4289691839 shutdown_connections 2019-12-05T06:52:12.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.829+0000 7f227d2fd700 1 -- 172.21.15.16:0/4289691839 wait complete. 2019-12-05T06:52:12.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.829+0000 7f227d2fd700 1 -- 172.21.15.16:0/4289691839 >> 172.21.15.16:0/4289691839 conn(0x7f2278061940 msgr2=0x7f2278096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:12.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.833+0000 7f227d2fd700 1 Processor -- start 2019-12-05T06:52:12.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.834+0000 7f227d2fd700 1 -- start start 2019-12-05T06:52:12.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.835+0000 7f227d2fd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f227806df10 0x7f2278098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.835+0000 7f227d2fd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22780992d0 0x7f2278098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.835+0000 7f227d2fd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f227806e300 con 0x7f2278098d80 2019-12-05T06:52:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.835+0000 7f227d2fd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2278097370 con 0x7f227806df10 2019-12-05T06:52:12.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.835+0000 7f227d2fd700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f227808ad00 con 0x7f22780992d0 2019-12-05T06:52:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.835+0000 7f2277fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f2278098d80 0x7f2278088fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47800/0 (socket says 172.21.15.16:47800) 2019-12-05T06:52:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.835+0000 7f2277fff700 1 -- 172.21.15.16:0/1746636633 learned_addr learned my addr 172.21.15.16:0/1746636633 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:12.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.835+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> v2:172.21.15.76:3300/0 conn(0x7f227806df10 0x7f2278098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.835+0000 7f2276ffd700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22780992d0 0x7f2278098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.835+0000 7f226ffff700 1 -- 172.21.15.16:0/1746636633 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 851509072 0 0) 0x7f2270000ed0 con 0x7f2278098d80 2019-12-05T06:52:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.836+0000 7f226ffff700 1 -- 172.21.15.16:0/1746636633 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f225c002e80 con 0x7f2278098d80 2019-12-05T06:52:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.836+0000 7f22777fe700 1 -- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22780992d0 msgr2=0x7f2278098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.836+0000 7f226ffff700 1 -- 172.21.15.16:0/1746636633 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1121770771 0 0) 0x7f22700081d0 con 0x7f2278098d80 2019-12-05T06:52:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.836+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22780992d0 0x7f2278098840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:12.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.836+0000 7f22777fe700 1 -- 172.21.15.16:0/1746636633 >> v1:172.21.15.16:6789/0 conn(0x7f2278098d80 legacy=0x7f2278088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.836+0000 7f22777fe700 1 -- 172.21.15.16:0/1746636633 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2278097a00 con 0x7f227806df10 2019-12-05T06:52:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.836+0000 7f2276ffd700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22780992d0 0x7f2278098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:52:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.836+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> v2:172.21.15.76:3300/0 conn(0x7f227806df10 0x7f2278098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.836+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> v2:172.21.15.76:3300/0 conn(0x7f227806df10 0x7f2278098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.837+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 0x7f2278098300 secure :-1 s=READY pgs=967 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.837+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 0x7f2278098300 secure :-1 s=THROTTLE_DONE pgs=967 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.837+0000 7f226ffff700 1 -- 172.21.15.16:0/1746636633 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2268000940 con 0x7f227806df10 2019-12-05T06:52:12.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.837+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 0x7f2278098300 secure :-1 s=THROTTLE_DONE pgs=967 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.837+0000 7f226ffff700 1 -- 172.21.15.16:0/1746636633 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f226800dff0 con 0x7f227806df10 2019-12-05T06:52:12.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.837+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 0x7f2278098300 secure :-1 s=THROTTLE_DONE pgs=967 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.837+0000 7f227d2fd700 1 -- 172.21.15.16:0/1746636633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2278097a00 con 0x7f227806df10 2019-12-05T06:52:12.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.837+0000 7f226ffff700 1 -- 172.21.15.16:0/1746636633 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2268000940 con 0x7f227806df10 2019-12-05T06:52:12.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.837+0000 7f227d2fd700 1 -- 172.21.15.16:0/1746636633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2278097a00 con 0x7f227806df10 2019-12-05T06:52:12.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.838+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 0x7f2278098300 secure :-1 s=THROTTLE_DONE pgs=967 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.839+0000 7f226ffff700 1 -- 172.21.15.16:0/1746636633 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f226800dc70 con 0x7f227806df10 2019-12-05T06:52:12.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.839+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 0x7f2278098300 secure :-1 s=THROTTLE_DONE pgs=967 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.839+0000 7f226ffff700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f225c023680 0x7f225c025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:12.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.839+0000 7f226ffff700 1 -- 172.21.15.16:0/1746636633 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 8439+0+0 (secure 0 0 0) 0x7f2268045dc0 con 0x7f227806df10 2019-12-05T06:52:12.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.840+0000 7f2276ffd700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f225c023680 0x7f225c025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:12.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.840+0000 7f2276ffd700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f225c023680 0x7f225c025af0 crc :-1 s=READY pgs=598 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:12.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.855+0000 7f227d2fd700 1 -- 172.21.15.16:0/1746636633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2278097a00 con 0x7f227806df10 2019-12-05T06:52:12.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.865+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 0x7f2278098300 secure :-1 s=THROTTLE_DONE pgs=967 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:12.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:12.865+0000 7f226ffff700 1 -- 172.21.15.16:0/1746636633 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f226800da10 con 0x7f227806df10 2019-12-05T06:52:13.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.086+0000 7f227d2fd700 1 -- 172.21.15.16:0/1746636633 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2278097a00 con 0x7f227806df10 2019-12-05T06:52:13.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.087+0000 7f22777fe700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 0x7f2278098300 secure :-1 s=THROTTLE_DONE pgs=967 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:13.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.087+0000 7f226ffff700 1 -- 172.21.15.16:0/1746636633 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 172 v172) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f226800dc20 con 0x7f227806df10 2019-12-05T06:52:13.089 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:13.089 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":172,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5972,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":172}],"filesystems":[{"mdsmap":{"epoch":171,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:50:22.733701+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":77,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5897},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5897":{"gid":5897,"name":"a","rank":0,"incarnation":168,"state":"up:active","state_seq":31,"addr":"172.21.15.16:6833/321778108","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":321778108},{"type":"v1","addr":"172.21.15.16:6833","nonce":321778108}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:13.089 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 172 2019-12-05T06:52:13.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 -- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f225c023680 msgr2=0x7f225c025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:13.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f225c023680 0x7f225c025af0 crc :-1 s=READY pgs=598 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:13.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 -- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 msgr2=0x7f2278098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:13.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 0x7f2278098300 secure :-1 s=READY pgs=967 cs=0 l=1 rx=0x7f22680103a0 tx=0x7f2268010740).stop 2019-12-05T06:52:13.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 -- 172.21.15.16:0/1746636633 shutdown_connections 2019-12-05T06:52:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f225c023680 0x7f225c025af0 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f22780992d0 0x7f2278098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 --2- 172.21.15.16:0/1746636633 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f227806df10 0x7f2278098300 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 -- 172.21.15.16:0/1746636633 shutdown_connections 2019-12-05T06:52:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 -- 172.21.15.16:0/1746636633 wait complete. 2019-12-05T06:52:13.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:13.091+0000 7f227d2fd700 1 -- 172.21.15.16:0/1746636633 >> 172.21.15.16:0/1746636633 conn(0x7f22780714f0 msgr2=0x7f227809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:13.106 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:52:15.107 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.376+0000 7fd0a1039700 1 Processor -- start 2019-12-05T06:52:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.376+0000 7fd0a1039700 1 -- start start 2019-12-05T06:52:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.377+0000 7fd0a1039700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 0x7fd09c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:15.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.377+0000 7fd0a1039700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd09c08fef0 0x7fd09c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.377+0000 7fd0a1039700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd09c0958e0 con 0x7fd09c08c6d0 2019-12-05T06:52:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.378+0000 7fd0a1039700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c08df90 con 0x7fd09c08fef0 2019-12-05T06:52:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.378+0000 7fd0a1039700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c0a3f70 con 0x7fd09c08d2c0 2019-12-05T06:52:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.378+0000 7fd0a0838700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd09c08fef0 0x7fd09c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.378+0000 7fd09b59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd09c08c6d0 0x7fd09c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47804/0 (socket says 172.21.15.16:47804) 2019-12-05T06:52:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.378+0000 7fd09b59e700 1 -- 172.21.15.16:0/440647771 learned_addr learned my addr 172.21.15.16:0/440647771 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:15.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.378+0000 7fd09ad9d700 1 --2- 172.21.15.16:0/440647771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 0x7fd09c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.378+0000 7fd09ad9d700 1 -- 172.21.15.16:0/440647771 >> v2:172.21.15.16:3300/0 conn(0x7fd09c08fef0 msgr2=0x7fd09c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.378+0000 7fd09ad9d700 1 --2- 172.21.15.16:0/440647771 >> v2:172.21.15.16:3300/0 conn(0x7fd09c08fef0 0x7fd09c08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.378+0000 7fd09ad9d700 1 -- 172.21.15.16:0/440647771 >> v1:172.21.15.16:6789/0 conn(0x7fd09c08c6d0 legacy=0x7fd09c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd09ad9d700 1 -- 172.21.15.16:0/440647771 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c08e620 con 0x7fd09c08d2c0 2019-12-05T06:52:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.378+0000 7fd09a59c700 1 -- 172.21.15.16:0/440647771 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358884406 0 0) 0x7fd08c001150 con 0x7fd09c08c6d0 2019-12-05T06:52:15.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd0a0838700 1 --2- 172.21.15.16:0/440647771 >> v2:172.21.15.16:3300/0 conn(0x7fd09c08fef0 0x7fd09c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:52:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd09ad9d700 1 --2- 172.21.15.16:0/440647771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 0x7fd09c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd09000af50 tx=0x7fd09000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd09ad9d700 1 --2- 172.21.15.16:0/440647771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 0x7fd09c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd09000af50 tx=0x7fd09000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd09ad9d700 1 --2- 172.21.15.16:0/440647771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 0x7fd09c08ef20 secure :-1 s=READY pgs=334 cs=0 l=1 rx=0x7fd09000af50 tx=0x7fd09000b760).ready entity=mon.2 client_cookie=7fc15670dd209af4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd09ad9d700 1 --2- 172.21.15.16:0/440647771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 0x7fd09c08ef20 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fd09000af50 tx=0x7fd09000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd09a59c700 1 -- 172.21.15.16:0/440647771 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd090007210 con 0x7fd09c08d2c0 2019-12-05T06:52:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd09ad9d700 1 --2- 172.21.15.16:0/440647771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 0x7fd09c08ef20 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fd09000af50 tx=0x7fd09000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd09ad9d700 1 --2- 172.21.15.16:0/440647771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 0x7fd09c08ef20 secure :-1 s=THROTTLE_DONE pgs=334 cs=0 l=1 rx=0x7fd09000af50 tx=0x7fd09000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd09a59c700 1 -- 172.21.15.16:0/440647771 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0900099b0 con 0x7fd09c08d2c0 2019-12-05T06:52:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd09a59c700 1 -- 172.21.15.16:0/440647771 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd09000a080 con 0x7fd09c08d2c0 2019-12-05T06:52:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.379+0000 7fd0a1039700 1 -- 172.21.15.16:0/440647771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 msgr2=0x7fd09c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.380+0000 7fd0a1039700 1 --2- 172.21.15.16:0/440647771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 0x7fd09c08ef20 secure :-1 s=READY pgs=334 cs=0 l=1 rx=0x7fd09000af50 tx=0x7fd09000b760).stop 2019-12-05T06:52:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.381+0000 7fd0a1039700 1 -- 172.21.15.16:0/440647771 shutdown_connections 2019-12-05T06:52:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.381+0000 7fd0a1039700 1 --2- 172.21.15.16:0/440647771 >> v2:172.21.15.16:3300/0 conn(0x7fd09c08fef0 0x7fd09c08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:15.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.381+0000 7fd0a1039700 1 --2- 172.21.15.16:0/440647771 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c08d2c0 0x7fd09c08ef20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.381+0000 7fd0a1039700 1 -- 172.21.15.16:0/440647771 shutdown_connections 2019-12-05T06:52:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.381+0000 7fd0a1039700 1 -- 172.21.15.16:0/440647771 wait complete. 2019-12-05T06:52:15.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.381+0000 7fd0a1039700 1 -- 172.21.15.16:0/440647771 >> 172.21.15.16:0/440647771 conn(0x7fd09c061940 msgr2=0x7fd09c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:15.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.386+0000 7fd0a1039700 1 Processor -- start 2019-12-05T06:52:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.387+0000 7fd0a1039700 1 -- start start 2019-12-05T06:52:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd0a1039700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd09c1302f0 0x7fd09c09d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd0a1039700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd09c09e7d0 0x7fd09c09dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd0a1039700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:15.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd0a1039700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c131ff0 con 0x7fd09c1302f0 2019-12-05T06:52:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd0a1039700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd09c1315c0 con 0x7fd09c09e7d0 2019-12-05T06:52:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd0a1039700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c131890 con 0x7fd09c09e280 2019-12-05T06:52:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd09b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd09c1302f0 0x7fd09c09d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd09b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd09c1302f0 0x7fd09c09d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34410/0 (socket says 172.21.15.16:34410) 2019-12-05T06:52:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd0a0838700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:15.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd09b59e700 1 -- 172.21.15.16:0/2631468267 learned_addr learned my addr 172.21.15.16:0/2631468267 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.388+0000 7fd09ad9d700 1 --2- 172.21.15.16:0/2631468267 >> v2:172.21.15.76:3300/0 conn(0x7fd09c09e7d0 0x7fd09c09dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd0a0838700 1 -- 172.21.15.16:0/2631468267 >> v2:172.21.15.76:3300/0 conn(0x7fd09c09e7d0 msgr2=0x7fd09c09dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> v2:172.21.15.76:3300/0 conn(0x7fd09c09e7d0 0x7fd09c09dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd0a0838700 1 -- 172.21.15.16:0/2631468267 >> v2:172.21.15.16:3300/0 conn(0x7fd09c1302f0 msgr2=0x7fd09c09d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> v2:172.21.15.16:3300/0 conn(0x7fd09c1302f0 0x7fd09c09d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd0a0838700 1 -- 172.21.15.16:0/2631468267 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd09c132680 con 0x7fd09c09e280 2019-12-05T06:52:15.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd09b59e700 1 --2- 172.21.15.16:0/2631468267 >> v2:172.21.15.16:3300/0 conn(0x7fd09c1302f0 0x7fd09c09d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:52:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=READY pgs=335 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.390+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.389+0000 7fd083fff700 1 -- 172.21.15.16:0/2631468267 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd094000cc0 con 0x7fd09c09e280 2019-12-05T06:52:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.390+0000 7fd083fff700 1 -- 172.21.15.16:0/2631468267 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd09400f1f0 con 0x7fd09c09e280 2019-12-05T06:52:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.390+0000 7fd0a1039700 1 -- 172.21.15.16:0/2631468267 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09c132680 con 0x7fd09c09e280 2019-12-05T06:52:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.390+0000 7fd083fff700 1 -- 172.21.15.16:0/2631468267 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd09400f8c0 con 0x7fd09c09e280 2019-12-05T06:52:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.390+0000 7fd0a1039700 1 -- 172.21.15.16:0/2631468267 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09c132680 con 0x7fd09c09e280 2019-12-05T06:52:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.390+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.391+0000 7fd083fff700 1 -- 172.21.15.16:0/2631468267 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd094000cc0 con 0x7fd09c09e280 2019-12-05T06:52:15.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.391+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.392+0000 7fd083fff700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd07c023630 0x7fd07c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:15.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.392+0000 7fd083fff700 1 -- 172.21.15.16:0/2631468267 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7fd094046040 con 0x7fd09c09e280 2019-12-05T06:52:15.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.392+0000 7fd09b59e700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd07c023630 0x7fd07c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:15.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.392+0000 7fd09b59e700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd07c023630 0x7fd07c025ac0 crc :-1 s=READY pgs=599 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:15.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.407+0000 7fd0a1039700 1 -- 172.21.15.16:0/2631468267 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c132680 con 0x7fd09c09e280 2019-12-05T06:52:15.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.413+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.413+0000 7fd083fff700 1 -- 172.21.15.16:0/2631468267 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd09403c640 con 0x7fd09c09e280 2019-12-05T06:52:15.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.639+0000 7fd0a1039700 1 -- 172.21.15.16:0/2631468267 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd09c132680 con 0x7fd09c09e280 2019-12-05T06:52:15.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.641+0000 7fd0a0838700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=THROTTLE_DONE pgs=335 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.641+0000 7fd083fff700 1 -- 172.21.15.16:0/2631468267 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fd094000cc0 con 0x7fd09c09e280 2019-12-05T06:52:15.642 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 173 2019-12-05T06:52:15.643 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:15.643 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:13.567867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.646+0000 7fd0a1039700 1 -- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd07c023630 msgr2=0x7fd07c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.646+0000 7fd0a1039700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd07c023630 0x7fd07c025ac0 crc :-1 s=READY pgs=599 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.646+0000 7fd0a1039700 1 -- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 msgr2=0x7fd09c06dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.647 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.646+0000 7fd0a1039700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 secure :-1 s=READY pgs=335 cs=0 l=1 rx=0x7fd0940106d0 tx=0x7fd094010a70).stop 2019-12-05T06:52:15.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.646+0000 7fd0a1039700 1 -- 172.21.15.16:0/2631468267 shutdown_connections 2019-12-05T06:52:15.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.646+0000 7fd0a1039700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd09c09e280 0x7fd09c06dc50 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:15.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.646+0000 7fd0a1039700 1 --2- 172.21.15.16:0/2631468267 >> v2:172.21.15.76:3300/0 conn(0x7fd09c09e7d0 0x7fd09c09dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:15.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.646+0000 7fd0a1039700 1 --2- 172.21.15.16:0/2631468267 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd07c023630 0x7fd07c025ac0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:15.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.646+0000 7fd0a1039700 1 --2- 172.21.15.16:0/2631468267 >> v2:172.21.15.16:3300/0 conn(0x7fd09c1302f0 0x7fd09c09d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:15.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.647+0000 7fd0a1039700 1 -- 172.21.15.16:0/2631468267 shutdown_connections 2019-12-05T06:52:15.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.647+0000 7fd0a1039700 1 -- 172.21.15.16:0/2631468267 wait complete. 2019-12-05T06:52:15.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.647+0000 7fd0a1039700 1 -- 172.21.15.16:0/2631468267 >> 172.21.15.16:0/2631468267 conn(0x7fd09c0a2850 msgr2=0x7fd09c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:15.664 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:52:15.664 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:52:15.664 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:15.677 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:15.712 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.885+0000 7f28682be700 1 Processor -- start 2019-12-05T06:52:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.885+0000 7f28682be700 1 -- start start 2019-12-05T06:52:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.887+0000 7f28682be700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 0x7f286008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:15.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.887+0000 7f28682be700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600958e0 con 0x7f286008c6d0 2019-12-05T06:52:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.887+0000 7f28682be700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600629e0 con 0x7f286008d2c0 2019-12-05T06:52:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.887+0000 7f28682be700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286008df90 con 0x7f286008fef0 2019-12-05T06:52:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.888+0000 7f286685b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f286008c6d0 0x7f286008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47820/0 (socket says 172.21.15.16:47820) 2019-12-05T06:52:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.888+0000 7f286685b700 1 -- 172.21.15.16:0/665930115 learned_addr learned my addr 172.21.15.16:0/665930115 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.888+0000 7f286705c700 1 --2- 172.21.15.16:0/665930115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 0x7f286008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:15.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.888+0000 7f2865859700 1 -- 172.21.15.16:0/665930115 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 178871980 0 0) 0x7f2850001150 con 0x7f286008c6d0 2019-12-05T06:52:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.888+0000 7f2865859700 1 -- 172.21.15.16:0/665930115 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2848002e80 con 0x7f286008c6d0 2019-12-05T06:52:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.888+0000 7f2865859700 1 -- 172.21.15.16:0/665930115 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759773768 0 0) 0x7f2854001400 con 0x7f286008d2c0 2019-12-05T06:52:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.888+0000 7f2865859700 1 -- 172.21.15.16:0/665930115 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28480030b0 con 0x7f286008d2c0 2019-12-05T06:52:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f286705c700 1 -- 172.21.15.16:0/665930115 >> v1:172.21.15.76:6789/0 conn(0x7f286008d2c0 legacy=0x7f286008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f286705c700 1 -- 172.21.15.16:0/665930115 >> v1:172.21.15.16:6789/0 conn(0x7f286008c6d0 legacy=0x7f286008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f286705c700 1 -- 172.21.15.16:0/665930115 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286008e620 con 0x7f286008fef0 2019-12-05T06:52:15.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f286705c700 1 --2- 172.21.15.16:0/665930115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 0x7f286008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f285800af80 tx=0x7f285800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f286705c700 1 --2- 172.21.15.16:0/665930115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 0x7f286008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f285800af80 tx=0x7f285800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f286705c700 1 --2- 172.21.15.16:0/665930115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 0x7f286008f720 secure :-1 s=READY pgs=336 cs=0 l=1 rx=0x7f285800af80 tx=0x7f285800b790).ready entity=mon.2 client_cookie=2bc53a5093dabcf7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f286705c700 1 --2- 172.21.15.16:0/665930115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 0x7f286008f720 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7f285800af80 tx=0x7f285800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f286705c700 1 --2- 172.21.15.16:0/665930115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 0x7f286008f720 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7f285800af80 tx=0x7f285800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f2865859700 1 -- 172.21.15.16:0/665930115 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2858007240 con 0x7f286008fef0 2019-12-05T06:52:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f286705c700 1 --2- 172.21.15.16:0/665930115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 0x7f286008f720 secure :-1 s=THROTTLE_DONE pgs=336 cs=0 l=1 rx=0x7f285800af80 tx=0x7f285800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.889+0000 7f2865859700 1 -- 172.21.15.16:0/665930115 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2858000c30 con 0x7f286008fef0 2019-12-05T06:52:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.890+0000 7f2865859700 1 -- 172.21.15.16:0/665930115 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2858009e60 con 0x7f286008fef0 2019-12-05T06:52:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.890+0000 7f28682be700 1 -- 172.21.15.16:0/665930115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 msgr2=0x7f286008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.890+0000 7f28682be700 1 --2- 172.21.15.16:0/665930115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 0x7f286008f720 secure :-1 s=READY pgs=336 cs=0 l=1 rx=0x7f285800af80 tx=0x7f285800b790).stop 2019-12-05T06:52:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.892+0000 7f28682be700 1 -- 172.21.15.16:0/665930115 shutdown_connections 2019-12-05T06:52:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.892+0000 7f28682be700 1 --2- 172.21.15.16:0/665930115 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286008fef0 0x7f286008f720 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.892+0000 7f28682be700 1 -- 172.21.15.16:0/665930115 shutdown_connections 2019-12-05T06:52:15.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.892+0000 7f28682be700 1 -- 172.21.15.16:0/665930115 wait complete. 2019-12-05T06:52:15.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.892+0000 7f28682be700 1 -- 172.21.15.16:0/665930115 >> 172.21.15.16:0/665930115 conn(0x7f2860061940 msgr2=0x7f2860096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:15.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.896+0000 7f28682be700 1 Processor -- start 2019-12-05T06:52:15.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.896+0000 7f28682be700 1 -- start start 2019-12-05T06:52:15.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.897+0000 7f28682be700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f286014fd10 0x7f286013a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:15.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.897+0000 7f28682be700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286006fd70 0x7f2860159430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:15.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.897+0000 7f28682be700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28600a0db0 con 0x7f286013b130 2019-12-05T06:52:15.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.897+0000 7f28682be700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f286009f800 con 0x7f286014fd10 2019-12-05T06:52:15.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.897+0000 7f28682be700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f286009edd0 con 0x7f286006fd70 2019-12-05T06:52:15.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f286605a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f286013b130 0x7f286013a710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47828/0 (socket says 172.21.15.16:47828) 2019-12-05T06:52:15.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f286605a700 1 -- 172.21.15.16:0/2618591808 learned_addr learned my addr 172.21.15.16:0/2618591808 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:15.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> v2:172.21.15.76:3300/0 conn(0x7f286014fd10 0x7f286013a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:15.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f286705c700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286006fd70 0x7f2860159430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f285f7fe700 1 -- 172.21.15.16:0/2618591808 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4160047609 0 0) 0x7f28540015c0 con 0x7f286013b130 2019-12-05T06:52:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f285f7fe700 1 -- 172.21.15.16:0/2618591808 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2840002e80 con 0x7f286013b130 2019-12-05T06:52:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f286685b700 1 -- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286006fd70 msgr2=0x7f2860159430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286006fd70 0x7f2860159430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f286685b700 1 -- 172.21.15.16:0/2618591808 >> v1:172.21.15.16:6789/0 conn(0x7f286013b130 legacy=0x7f286013a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:15.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f286685b700 1 -- 172.21.15.16:0/2618591808 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f286009fe90 con 0x7f286014fd10 2019-12-05T06:52:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f286705c700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286006fd70 0x7f2860159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:52:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.898+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> v2:172.21.15.76:3300/0 conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.899+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> v2:172.21.15.76:3300/0 conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.899+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=READY pgs=969 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.899+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=THROTTLE_DONE pgs=969 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.899+0000 7f285f7fe700 1 -- 172.21.15.16:0/2618591808 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2850000c90 con 0x7f286014fd10 2019-12-05T06:52:15.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.899+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=THROTTLE_DONE pgs=969 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.899+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=THROTTLE_DONE pgs=969 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.899+0000 7f285f7fe700 1 -- 172.21.15.16:0/2618591808 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28500013e0 con 0x7f286014fd10 2019-12-05T06:52:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.899+0000 7f28682be700 1 -- 172.21.15.16:0/2618591808 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f286009fe90 con 0x7f286014fd10 2019-12-05T06:52:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.899+0000 7f285f7fe700 1 -- 172.21.15.16:0/2618591808 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2850001a30 con 0x7f286014fd10 2019-12-05T06:52:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.899+0000 7f28682be700 1 -- 172.21.15.16:0/2618591808 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f286009fe90 con 0x7f286014fd10 2019-12-05T06:52:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.900+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=THROTTLE_DONE pgs=969 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.900+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=THROTTLE_DONE pgs=969 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.900+0000 7f285f7fe700 1 -- 172.21.15.16:0/2618591808 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2850000c90 con 0x7f286014fd10 2019-12-05T06:52:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.901+0000 7f285f7fe700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2840023670 0x7f2840025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.901+0000 7f285f7fe700 1 -- 172.21.15.16:0/2618591808 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f2850045d20 con 0x7f286014fd10 2019-12-05T06:52:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.901+0000 7f286605a700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2840023670 0x7f2840025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:15.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.901+0000 7f286605a700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2840023670 0x7f2840025ae0 crc :-1 s=READY pgs=600 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:15.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.917+0000 7f28682be700 1 -- 172.21.15.16:0/2618591808 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f286009fe90 con 0x7f286014fd10 2019-12-05T06:52:15.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.926+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=THROTTLE_DONE pgs=969 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:15.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:15.926+0000 7f285f7fe700 1 -- 172.21.15.16:0/2618591808 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2850000fe0 con 0x7f286014fd10 2019-12-05T06:52:16.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.174+0000 7f28682be700 1 -- 172.21.15.16:0/2618591808 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f286009fe90 con 0x7f286014fd10 2019-12-05T06:52:16.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.175+0000 7f286685b700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=THROTTLE_DONE pgs=969 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:16.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.176+0000 7f285f7fe700 1 -- 172.21.15.16:0/2618591808 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 173 v173) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f2850000c90 con 0x7f286014fd10 2019-12-05T06:52:16.178 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 173 2019-12-05T06:52:16.179 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:16.179 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":173,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":173,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:13.567867+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.180+0000 7f28682be700 1 -- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2840023670 msgr2=0x7f2840025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.180+0000 7f28682be700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2840023670 0x7f2840025ae0 crc :-1 s=READY pgs=600 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:16.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.180+0000 7f28682be700 1 -- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 msgr2=0x7f286013a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.180+0000 7f28682be700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 0x7f286013a1d0 secure :-1 s=READY pgs=969 cs=0 l=1 rx=0x7f2850010380 tx=0x7f2850010720).stop 2019-12-05T06:52:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.181+0000 7f28682be700 1 -- 172.21.15.16:0/2618591808 shutdown_connections 2019-12-05T06:52:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.181+0000 7f28682be700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2840023670 0x7f2840025ae0 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.181+0000 7f28682be700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f286006fd70 0x7f2860159430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.181+0000 7f28682be700 1 --2- 172.21.15.16:0/2618591808 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f286014fd10 0x7f286013a1d0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.181+0000 7f28682be700 1 -- 172.21.15.16:0/2618591808 shutdown_connections 2019-12-05T06:52:16.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.181+0000 7f28682be700 1 -- 172.21.15.16:0/2618591808 wait complete. 2019-12-05T06:52:16.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:16.181+0000 7f28682be700 1 -- 172.21.15.16:0/2618591808 >> 172.21.15.16:0/2618591808 conn(0x7f2860130a60 msgr2=0x7f2860086470 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:16.199 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 173, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 173, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_5972": { "addr": "172.21.15.76:6835/727762895", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 727762895, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 727762895, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 5972, "incarnation": 173, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 78, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:52:13.567867+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 5972 } } } ], "standbys": [] } 2019-12-05T06:52:18.200 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:18.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.456+0000 7fcae7253700 1 Processor -- start 2019-12-05T06:52:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.456+0000 7fcae7253700 1 -- start start 2019-12-05T06:52:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.458+0000 7fcae7253700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcae008c6d0 0x7fcae008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.458+0000 7fcae7253700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcae008fef0 0x7fcae008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.458+0000 7fcae7253700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcae008ef20 0x7fcae008f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:18.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.458+0000 7fcae7253700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae00a3f70 con 0x7fcae008fef0 2019-12-05T06:52:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.458+0000 7fcae7253700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae008d830 con 0x7fcae008ef20 2019-12-05T06:52:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.458+0000 7fcae7253700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae008d540 con 0x7fcae008c6d0 2019-12-05T06:52:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.458+0000 7fcae4fef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcae008fef0 0x7fcae008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.459+0000 7fcae4fef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcae008fef0 0x7fcae008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34442/0 (socket says 172.21.15.16:34442) 2019-12-05T06:52:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.459+0000 7fcae4fef700 1 -- 172.21.15.16:0/2550345457 learned_addr learned my addr 172.21.15.16:0/2550345457 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:18.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.459+0000 7fcae5ff1700 1 --2- 172.21.15.16:0/2550345457 >> v2:172.21.15.76:3300/0 conn(0x7fcae008ef20 0x7fcae008f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.459+0000 7fcae57f0700 1 --2- 172.21.15.16:0/2550345457 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcae008c6d0 0x7fcae008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.459+0000 7fcae4fef700 1 -- 172.21.15.16:0/2550345457 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcae008c6d0 msgr2=0x7fcae008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.459+0000 7fcae4fef700 1 --2- 172.21.15.16:0/2550345457 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcae008c6d0 0x7fcae008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.459+0000 7fcae4fef700 1 -- 172.21.15.16:0/2550345457 >> v2:172.21.15.76:3300/0 conn(0x7fcae008ef20 msgr2=0x7fcae008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.459+0000 7fcae4fef700 1 --2- 172.21.15.16:0/2550345457 >> v2:172.21.15.76:3300/0 conn(0x7fcae008ef20 0x7fcae008f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.459+0000 7fcae4fef700 1 -- 172.21.15.16:0/2550345457 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae008e620 con 0x7fcae008fef0 2019-12-05T06:52:18.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.459+0000 7fcae4fef700 1 --2- 172.21.15.16:0/2550345457 >> v2:172.21.15.16:3300/0 conn(0x7fcae008fef0 0x7fcae008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcad400f6c0 tx=0x7fcad400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.460+0000 7fcae4fef700 1 --2- 172.21.15.16:0/2550345457 >> v2:172.21.15.16:3300/0 conn(0x7fcae008fef0 0x7fcae008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcad400f6c0 tx=0x7fcad400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.460+0000 7fcae4fef700 1 --2- 172.21.15.16:0/2550345457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae008fef0 0x7fcae008d000 secure :-1 s=READY pgs=1424 cs=0 l=1 rx=0x7fcad400f6c0 tx=0x7fcad400f570).ready entity=mon.0 client_cookie=95780ef5a45e5853 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.460+0000 7fcae4fef700 1 --2- 172.21.15.16:0/2550345457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae008fef0 0x7fcae008d000 secure :-1 s=THROTTLE_DONE pgs=1424 cs=0 l=1 rx=0x7fcad400f6c0 tx=0x7fcad400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.460+0000 7fcae4fef700 1 --2- 172.21.15.16:0/2550345457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae008fef0 0x7fcae008d000 secure :-1 s=THROTTLE_DONE pgs=1424 cs=0 l=1 rx=0x7fcad400f6c0 tx=0x7fcad400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.460+0000 7fcae4fef700 1 --2- 172.21.15.16:0/2550345457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae008fef0 0x7fcae008d000 secure :-1 s=THROTTLE_DONE pgs=1424 cs=0 l=1 rx=0x7fcad400f6c0 tx=0x7fcad400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.460+0000 7fcadffff700 1 -- 172.21.15.16:0/2550345457 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcad4000960 con 0x7fcae008fef0 2019-12-05T06:52:18.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.460+0000 7fcadffff700 1 -- 172.21.15.16:0/2550345457 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad4000c90 con 0x7fcae008fef0 2019-12-05T06:52:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.460+0000 7fcadffff700 1 -- 172.21.15.16:0/2550345457 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcad40013a0 con 0x7fcae008fef0 2019-12-05T06:52:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.460+0000 7fcae7253700 1 -- 172.21.15.16:0/2550345457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae008fef0 msgr2=0x7fcae008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.461+0000 7fcae7253700 1 --2- 172.21.15.16:0/2550345457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae008fef0 0x7fcae008d000 secure :-1 s=READY pgs=1424 cs=0 l=1 rx=0x7fcad400f6c0 tx=0x7fcad400f570).stop 2019-12-05T06:52:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.462+0000 7fcae7253700 1 -- 172.21.15.16:0/2550345457 shutdown_connections 2019-12-05T06:52:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.462+0000 7fcae7253700 1 --2- 172.21.15.16:0/2550345457 >> v2:172.21.15.76:3300/0 conn(0x7fcae008ef20 0x7fcae008f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.462+0000 7fcae7253700 1 --2- 172.21.15.16:0/2550345457 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae008fef0 0x7fcae008d000 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:18.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.462+0000 7fcae7253700 1 --2- 172.21.15.16:0/2550345457 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcae008c6d0 0x7fcae008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.463+0000 7fcae7253700 1 -- 172.21.15.16:0/2550345457 shutdown_connections 2019-12-05T06:52:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.463+0000 7fcae7253700 1 -- 172.21.15.16:0/2550345457 wait complete. 2019-12-05T06:52:18.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.463+0000 7fcae7253700 1 -- 172.21.15.16:0/2550345457 >> 172.21.15.16:0/2550345457 conn(0x7fcae0061940 msgr2=0x7fcae0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.468+0000 7fcae7253700 1 Processor -- start 2019-12-05T06:52:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.468+0000 7fcae7253700 1 -- start start 2019-12-05T06:52:18.477 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae7253700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcae0096e90 0x7fcae009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae7253700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcae0084f30 0x7fcae009b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae7253700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcae0097280 0x7fcae00976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae7253700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae009a450 con 0x7fcae0096e90 2019-12-05T06:52:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae7253700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcae0099a20 con 0x7fcae0084f30 2019-12-05T06:52:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae7253700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcae0099cf0 con 0x7fcae0097280 2019-12-05T06:52:18.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae57f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcae0096e90 0x7fcae009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae57f0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcae0096e90 0x7fcae009b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34444/0 (socket says 172.21.15.16:34444) 2019-12-05T06:52:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae57f0700 1 -- 172.21.15.16:0/3555028407 learned_addr learned my addr 172.21.15.16:0/3555028407 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae5ff1700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcae0097280 0x7fcae00976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.470+0000 7fcae4fef700 1 --2- 172.21.15.16:0/3555028407 >> v2:172.21.15.76:3300/0 conn(0x7fcae0084f30 0x7fcae009b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:18.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.471+0000 7fcae57f0700 1 -- 172.21.15.16:0/3555028407 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcae0097280 msgr2=0x7fcae00976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.471+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcae0097280 0x7fcae00976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.471+0000 7fcae57f0700 1 -- 172.21.15.16:0/3555028407 >> v2:172.21.15.76:3300/0 conn(0x7fcae0084f30 msgr2=0x7fcae009b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.471+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> v2:172.21.15.76:3300/0 conn(0x7fcae0084f30 0x7fcae009b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.471+0000 7fcae57f0700 1 -- 172.21.15.16:0/3555028407 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcae009aae0 con 0x7fcae0096e90 2019-12-05T06:52:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.471+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> v2:172.21.15.16:3300/0 conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.471+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> v2:172.21.15.16:3300/0 conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.471+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=READY pgs=1425 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.471+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=THROTTLE_DONE pgs=1425 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.472+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=THROTTLE_DONE pgs=1425 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.472+0000 7fcaddffb700 1 -- 172.21.15.16:0/3555028407 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcad8005af0 con 0x7fcae0096e90 2019-12-05T06:52:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.472+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=THROTTLE_DONE pgs=1425 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.472+0000 7fcaddffb700 1 -- 172.21.15.16:0/3555028407 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcad8005710 con 0x7fcae0096e90 2019-12-05T06:52:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.472+0000 7fcaddffb700 1 -- 172.21.15.16:0/3555028407 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcad80060d0 con 0x7fcae0096e90 2019-12-05T06:52:18.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.472+0000 7fcae7253700 1 -- 172.21.15.16:0/3555028407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcae009aae0 con 0x7fcae0096e90 2019-12-05T06:52:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.472+0000 7fcae7253700 1 -- 172.21.15.16:0/3555028407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcae009aae0 con 0x7fcae0096e90 2019-12-05T06:52:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.473+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=THROTTLE_DONE pgs=1425 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.474+0000 7fcaddffb700 1 -- 172.21.15.16:0/3555028407 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcad8005900 con 0x7fcae0096e90 2019-12-05T06:52:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.474+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=THROTTLE_DONE pgs=1425 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.474+0000 7fcaddffb700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcac0023630 0x7fcac0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.475+0000 7fcaddffb700 1 -- 172.21.15.16:0/3555028407 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7fcad80460d0 con 0x7fcae0096e90 2019-12-05T06:52:18.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.475+0000 7fcae4fef700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcac0023630 0x7fcac0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:18.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.475+0000 7fcae4fef700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcac0023630 0x7fcac0025ac0 crc :-1 s=READY pgs=601 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:18.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.488+0000 7fcae7253700 1 -- 172.21.15.16:0/3555028407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcae009aae0 con 0x7fcae0096e90 2019-12-05T06:52:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.497+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=THROTTLE_DONE pgs=1425 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.497+0000 7fcaddffb700 1 -- 172.21.15.16:0/3555028407 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcad8005500 con 0x7fcae0096e90 2019-12-05T06:52:18.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.753+0000 7fcae7253700 1 -- 172.21.15.16:0/3555028407 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcae009aae0 con 0x7fcae0096e90 2019-12-05T06:52:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.754+0000 7fcae57f0700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=THROTTLE_DONE pgs=1425 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.754+0000 7fcaddffb700 1 -- 172.21.15.16:0/3555028407 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fcad8005900 con 0x7fcae0096e90 2019-12-05T06:52:18.755 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 176 2019-12-05T06:52:18.756 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:18.756 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:18.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 -- 172.21.15.16:0/3555028407 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcac0023630 msgr2=0x7fcac0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcac0023630 0x7fcac0025ac0 crc :-1 s=READY pgs=601 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 -- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 msgr2=0x7fcae009b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 0x7fcae009b3e0 secure :-1 s=READY pgs=1425 cs=0 l=1 rx=0x7fcad80107d0 tx=0x7fcad8010650).stop 2019-12-05T06:52:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 -- 172.21.15.16:0/3555028407 shutdown_connections 2019-12-05T06:52:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcae0097280 0x7fcae00976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:18.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 --2- 172.21.15.16:0/3555028407 >> v2:172.21.15.76:3300/0 conn(0x7fcae0084f30 0x7fcae009b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcac0023630 0x7fcac0025ac0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 --2- 172.21.15.16:0/3555028407 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcae0096e90 0x7fcae009b3e0 unknown :-1 s=CLOSED pgs=1425 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 -- 172.21.15.16:0/3555028407 shutdown_connections 2019-12-05T06:52:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 -- 172.21.15.16:0/3555028407 wait complete. 2019-12-05T06:52:18.759 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:18.757+0000 7fcae7253700 1 -- 172.21.15.16:0/3555028407 >> 172.21.15.16:0/3555028407 conn(0x7fcae00714f0 msgr2=0x7fcae009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:18.769 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=5897 2019-12-05T06:52:18.769 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 8 secs before reviving mds.a 2019-12-05T06:52:26.770 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:52:26.770 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:52:26.770 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:52:26.775 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:52:26.776 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:26.960 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:52:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.993+0000 7fb782206700 1 Processor -- start 2019-12-05T06:52:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.993+0000 7fb782206700 1 -- start start 2019-12-05T06:52:26.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.994+0000 7fb782206700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb77c08fef0 0x7fb77c08f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.994+0000 7fb782206700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77c0958e0 con 0x7fb77c08c6d0 2019-12-05T06:52:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.994+0000 7fb782206700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77c0629e0 con 0x7fb77c08d2c0 2019-12-05T06:52:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.994+0000 7fb782206700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77c08df90 con 0x7fb77c08fef0 2019-12-05T06:52:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.994+0000 7fb780fa4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb77c08fef0 0x7fb77c08f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.994+0000 7fb77bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb77c08c6d0 0x7fb77c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47888/0 (socket says 172.21.15.16:47888) 2019-12-05T06:52:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.994+0000 7fb77bfff700 1 -- 172.21.15.16:0/1181369109 learned_addr learned my addr 172.21.15.16:0/1181369109 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:27.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.994+0000 7fb780fa4700 1 -- 172.21.15.16:0/1181369109 >> v1:172.21.15.76:6789/0 conn(0x7fb77c08d2c0 legacy=0x7fb77c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb780fa4700 1 -- 172.21.15.16:0/1181369109 >> v1:172.21.15.16:6789/0 conn(0x7fb77c08c6d0 legacy=0x7fb77c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb780fa4700 1 -- 172.21.15.16:0/1181369109 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c08e620 con 0x7fb77c08fef0 2019-12-05T06:52:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb77affd700 1 -- 172.21.15.16:0/1181369109 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 937859962 0 0) 0x7fb76c001150 con 0x7fb77c08c6d0 2019-12-05T06:52:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb780fa4700 1 --2- 172.21.15.16:0/1181369109 >> v2:172.21.15.16:3300/0 conn(0x7fb77c08fef0 0x7fb77c08f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb77400b2d0 tx=0x7fb77400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb780fa4700 1 --2- 172.21.15.16:0/1181369109 >> v2:172.21.15.16:3300/0 conn(0x7fb77c08fef0 0x7fb77c08f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb77400b2d0 tx=0x7fb77400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb780fa4700 1 --2- 172.21.15.16:0/1181369109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c08fef0 0x7fb77c08f720 secure :-1 s=READY pgs=1431 cs=0 l=1 rx=0x7fb77400b2d0 tx=0x7fb77400bae0).ready entity=mon.0 client_cookie=4de23be2480b4bd0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:27.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb780fa4700 1 --2- 172.21.15.16:0/1181369109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c08fef0 0x7fb77c08f720 secure :-1 s=THROTTLE_DONE pgs=1431 cs=0 l=1 rx=0x7fb77400b2d0 tx=0x7fb77400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb780fa4700 1 --2- 172.21.15.16:0/1181369109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c08fef0 0x7fb77c08f720 secure :-1 s=THROTTLE_DONE pgs=1431 cs=0 l=1 rx=0x7fb77400b2d0 tx=0x7fb77400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb77affd700 1 -- 172.21.15.16:0/1181369109 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb774009f20 con 0x7fb77c08fef0 2019-12-05T06:52:27.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb780fa4700 1 --2- 172.21.15.16:0/1181369109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c08fef0 0x7fb77c08f720 secure :-1 s=THROTTLE_DONE pgs=1431 cs=0 l=1 rx=0x7fb77400b2d0 tx=0x7fb77400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb77affd700 1 -- 172.21.15.16:0/1181369109 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb774000df0 con 0x7fb77c08fef0 2019-12-05T06:52:27.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb77affd700 1 -- 172.21.15.16:0/1181369109 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb77400a500 con 0x7fb77c08fef0 2019-12-05T06:52:27.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb782206700 1 -- 172.21.15.16:0/1181369109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c08fef0 msgr2=0x7fb77c08f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:27.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.995+0000 7fb782206700 1 --2- 172.21.15.16:0/1181369109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c08fef0 0x7fb77c08f720 secure :-1 s=READY pgs=1431 cs=0 l=1 rx=0x7fb77400b2d0 tx=0x7fb77400bae0).stop 2019-12-05T06:52:27.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.996+0000 7fb782206700 1 -- 172.21.15.16:0/1181369109 shutdown_connections 2019-12-05T06:52:27.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.996+0000 7fb782206700 1 --2- 172.21.15.16:0/1181369109 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb77c08fef0 0x7fb77c08f720 unknown :-1 s=CLOSED pgs=1431 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:27.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.996+0000 7fb782206700 1 -- 172.21.15.16:0/1181369109 shutdown_connections 2019-12-05T06:52:27.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.996+0000 7fb782206700 1 -- 172.21.15.16:0/1181369109 wait complete. 2019-12-05T06:52:27.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.996+0000 7fb782206700 1 -- 172.21.15.16:0/1181369109 >> 172.21.15.16:0/1181369109 conn(0x7fb77c061940 msgr2=0x7fb77c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.999+0000 7fb782206700 1 Processor -- start 2019-12-05T06:52:27.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:26.999+0000 7fb782206700 1 -- start start 2019-12-05T06:52:27.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.000+0000 7fb782206700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb77c158610 0x7fb77c15aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:27.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.000+0000 7fb782206700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77c070330 con 0x7fb77c1333a0 2019-12-05T06:52:27.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.000+0000 7fb782206700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb77c06fe30 con 0x7fb77c128fe0 2019-12-05T06:52:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.000+0000 7fb782206700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb77c088730 con 0x7fb77c158610 2019-12-05T06:52:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.000+0000 7fb77bfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb77c1333a0 0x7fb77c128360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47896/0 (socket says 172.21.15.16:47896) 2019-12-05T06:52:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.000+0000 7fb77bfff700 1 -- 172.21.15.16:0/3706543218 learned_addr learned my addr 172.21.15.16:0/3706543218 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.001+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> v2:172.21.15.76:3300/0 conn(0x7fb77c158610 0x7fb77c15aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.001+0000 7fb778ff9700 1 -- 172.21.15.16:0/3706543218 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2258157559 0 0) 0x7fb76c001150 con 0x7fb77c1333a0 2019-12-05T06:52:27.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.001+0000 7fb778ff9700 1 -- 172.21.15.16:0/3706543218 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb758002e80 con 0x7fb77c1333a0 2019-12-05T06:52:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.001+0000 7fb778ff9700 1 -- 172.21.15.16:0/3706543218 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 835542716 0 0) 0x7fb76c001300 con 0x7fb77c1333a0 2019-12-05T06:52:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.001+0000 7fb780fa4700 1 -- 172.21.15.16:0/3706543218 >> v1:172.21.15.76:6789/0 conn(0x7fb77c128fe0 legacy=0x7fb77c156e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.001+0000 7fb780fa4700 1 -- 172.21.15.16:0/3706543218 >> v1:172.21.15.16:6789/0 conn(0x7fb77c1333a0 legacy=0x7fb77c128360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.001+0000 7fb780fa4700 1 -- 172.21.15.16:0/3706543218 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb77c089ea0 con 0x7fb77c158610 2019-12-05T06:52:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.001+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> v2:172.21.15.76:3300/0 conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.002+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> v2:172.21.15.76:3300/0 conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.002+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=READY pgs=973 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.002+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=THROTTLE_DONE pgs=973 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.002+0000 7fb778ff9700 1 -- 172.21.15.16:0/3706543218 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb77400b330 con 0x7fb77c158610 2019-12-05T06:52:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.002+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=THROTTLE_DONE pgs=973 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.002+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=THROTTLE_DONE pgs=973 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.002+0000 7fb778ff9700 1 -- 172.21.15.16:0/3706543218 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb774009af0 con 0x7fb77c158610 2019-12-05T06:52:27.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.002+0000 7fb782206700 1 -- 172.21.15.16:0/3706543218 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb77c089ea0 con 0x7fb77c158610 2019-12-05T06:52:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.002+0000 7fb778ff9700 1 -- 172.21.15.16:0/3706543218 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb77400a1c0 con 0x7fb77c158610 2019-12-05T06:52:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.002+0000 7fb782206700 1 -- 172.21.15.16:0/3706543218 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb77c089ea0 con 0x7fb77c158610 2019-12-05T06:52:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.003+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=THROTTLE_DONE pgs=973 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.004+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=THROTTLE_DONE pgs=973 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.004+0000 7fb778ff9700 1 -- 172.21.15.16:0/3706543218 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb774009ce0 con 0x7fb77c158610 2019-12-05T06:52:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.004+0000 7fb778ff9700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb758023680 0x7fb758025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:27.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.004+0000 7fb778ff9700 1 -- 172.21.15.16:0/3706543218 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7fb774046130 con 0x7fb77c158610 2019-12-05T06:52:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.005+0000 7fb77bfff700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb758023680 0x7fb758025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.005+0000 7fb77bfff700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb758023680 0x7fb758025af0 crc :-1 s=READY pgs=602 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:27.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.009+0000 7fb782206700 1 -- 172.21.15.16:0/3706543218 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb77c089ea0 con 0x7fb77c158610 2019-12-05T06:52:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.017+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=THROTTLE_DONE pgs=973 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.018+0000 7fb778ff9700 1 -- 172.21.15.16:0/3706543218 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb77400b010 con 0x7fb77c158610 2019-12-05T06:52:27.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.265+0000 7fb782206700 1 -- 172.21.15.16:0/3706543218 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb77c089ea0 con 0x7fb77c158610 2019-12-05T06:52:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.265+0000 7fb780fa4700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=THROTTLE_DONE pgs=973 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.265+0000 7fb778ff9700 1 -- 172.21.15.16:0/3706543218 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fb774009ce0 con 0x7fb77c158610 2019-12-05T06:52:27.265 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 176 2019-12-05T06:52:27.265 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:27.266 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.268+0000 7fb782206700 1 -- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb758023680 msgr2=0x7fb758025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:27.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.268+0000 7fb782206700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb758023680 0x7fb758025af0 crc :-1 s=READY pgs=602 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.268+0000 7fb782206700 1 -- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 msgr2=0x7fb77c15aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.268+0000 7fb782206700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 0x7fb77c15aa10 secure :-1 s=READY pgs=973 cs=0 l=1 rx=0x7fb7740106c0 tx=0x7fb774010a60).stop 2019-12-05T06:52:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.268+0000 7fb782206700 1 -- 172.21.15.16:0/3706543218 shutdown_connections 2019-12-05T06:52:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.268+0000 7fb782206700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb758023680 0x7fb758025af0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.268+0000 7fb782206700 1 --2- 172.21.15.16:0/3706543218 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fb77c158610 0x7fb77c15aa10 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.269+0000 7fb782206700 1 -- 172.21.15.16:0/3706543218 shutdown_connections 2019-12-05T06:52:27.270 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.269+0000 7fb782206700 1 -- 172.21.15.16:0/3706543218 wait complete. 2019-12-05T06:52:27.271 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:27.269+0000 7fb782206700 1 -- 172.21.15.16:0/3706543218 >> 172.21.15.16:0/3706543218 conn(0x7fb77c09de30 msgr2=0x7fb77c070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:27.279 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:52:29.280 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:29.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.545+0000 7f862d356700 1 Processor -- start 2019-12-05T06:52:29.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.545+0000 7f862d356700 1 -- start start 2019-12-05T06:52:29.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f862d356700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f862808c6d0 0x7f862808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:29.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f862d356700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f862808fef0 0x7f862808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:29.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f862d356700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f862808ef20 0x7f862808f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:29.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f862d356700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862808df90 con 0x7f862808c6d0 2019-12-05T06:52:29.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f862d356700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86280a3f70 con 0x7f862808fef0 2019-12-05T06:52:29.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f862d356700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f862808d830 con 0x7f862808ef20 2019-12-05T06:52:29.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f86277fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f862808c6d0 0x7f862808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:29.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f86277fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f862808c6d0 0x7f862808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34518/0 (socket says 172.21.15.16:34518) 2019-12-05T06:52:29.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f86277fe700 1 -- 172.21.15.16:0/1497605689 learned_addr learned my addr 172.21.15.16:0/1497605689 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:29.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f8627fff700 1 --2- 172.21.15.16:0/1497605689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f862808ef20 0x7f862808f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:29.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.547+0000 7f8626ffd700 1 --2- 172.21.15.16:0/1497605689 >> v2:172.21.15.76:3300/0 conn(0x7f862808fef0 0x7f862808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:29.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.548+0000 7f86277fe700 1 -- 172.21.15.16:0/1497605689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f862808ef20 msgr2=0x7f862808f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:29.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.548+0000 7f86277fe700 1 --2- 172.21.15.16:0/1497605689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f862808ef20 0x7f862808f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:29.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.548+0000 7f86277fe700 1 -- 172.21.15.16:0/1497605689 >> v2:172.21.15.76:3300/0 conn(0x7f862808fef0 msgr2=0x7f862808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:29.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.548+0000 7f86277fe700 1 --2- 172.21.15.16:0/1497605689 >> v2:172.21.15.76:3300/0 conn(0x7f862808fef0 0x7f862808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:29.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.548+0000 7f86277fe700 1 -- 172.21.15.16:0/1497605689 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862808d540 con 0x7f862808c6d0 2019-12-05T06:52:29.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.548+0000 7f86277fe700 1 --2- 172.21.15.16:0/1497605689 >> v2:172.21.15.16:3300/0 conn(0x7f862808c6d0 0x7f862808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f862000fa40 tx=0x7f862000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.548+0000 7f86277fe700 1 --2- 172.21.15.16:0/1497605689 >> v2:172.21.15.16:3300/0 conn(0x7f862808c6d0 0x7f862808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f862000fa40 tx=0x7f862000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.548+0000 7f86277fe700 1 --2- 172.21.15.16:0/1497605689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f862808c6d0 0x7f862808cac0 secure :-1 s=READY pgs=1433 cs=0 l=1 rx=0x7f862000fa40 tx=0x7f862000f8f0).ready entity=mon.0 client_cookie=d714054dea83dfa7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:29.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.548+0000 7f86277fe700 1 --2- 172.21.15.16:0/1497605689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f862808c6d0 0x7f862808cac0 secure :-1 s=THROTTLE_DONE pgs=1433 cs=0 l=1 rx=0x7f862000fa40 tx=0x7f862000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.549+0000 7f86277fe700 1 --2- 172.21.15.16:0/1497605689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f862808c6d0 0x7f862808cac0 secure :-1 s=THROTTLE_DONE pgs=1433 cs=0 l=1 rx=0x7f862000fa40 tx=0x7f862000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.549+0000 7f86277fe700 1 --2- 172.21.15.16:0/1497605689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f862808c6d0 0x7f862808cac0 secure :-1 s=THROTTLE_DONE pgs=1433 cs=0 l=1 rx=0x7f862000fa40 tx=0x7f862000f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.549+0000 7f86267fc700 1 -- 172.21.15.16:0/1497605689 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8620000ce0 con 0x7f862808c6d0 2019-12-05T06:52:29.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.549+0000 7f86267fc700 1 -- 172.21.15.16:0/1497605689 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8620001010 con 0x7f862808c6d0 2019-12-05T06:52:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.549+0000 7f86267fc700 1 -- 172.21.15.16:0/1497605689 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8620001720 con 0x7f862808c6d0 2019-12-05T06:52:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.549+0000 7f862d356700 1 -- 172.21.15.16:0/1497605689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f862808c6d0 msgr2=0x7f862808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.549+0000 7f862d356700 1 --2- 172.21.15.16:0/1497605689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f862808c6d0 0x7f862808cac0 secure :-1 s=READY pgs=1433 cs=0 l=1 rx=0x7f862000fa40 tx=0x7f862000f8f0).stop 2019-12-05T06:52:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.551+0000 7f862d356700 1 -- 172.21.15.16:0/1497605689 shutdown_connections 2019-12-05T06:52:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.551+0000 7f862d356700 1 --2- 172.21.15.16:0/1497605689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f862808ef20 0x7f862808f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:29.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.551+0000 7f862d356700 1 --2- 172.21.15.16:0/1497605689 >> v2:172.21.15.76:3300/0 conn(0x7f862808fef0 0x7f862808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:29.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.551+0000 7f862d356700 1 --2- 172.21.15.16:0/1497605689 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f862808c6d0 0x7f862808cac0 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:29.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.551+0000 7f862d356700 1 -- 172.21.15.16:0/1497605689 shutdown_connections 2019-12-05T06:52:29.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.551+0000 7f862d356700 1 -- 172.21.15.16:0/1497605689 wait complete. 2019-12-05T06:52:29.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.551+0000 7f862d356700 1 -- 172.21.15.16:0/1497605689 >> 172.21.15.16:0/1497605689 conn(0x7f8628061940 msgr2=0x7f8628096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:29.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.556+0000 7f862d356700 1 Processor -- start 2019-12-05T06:52:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.556+0000 7f862d356700 1 -- start start 2019-12-05T06:52:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.558+0000 7f862d356700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8628080290 0x7f862812f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.558+0000 7f862d356700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8628130750 0x7f862812fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.558+0000 7f862d356700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86280889b0 con 0x7f8628130200 2019-12-05T06:52:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.558+0000 7f862d356700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f862808a670 con 0x7f8628080290 2019-12-05T06:52:29.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.558+0000 7f862d356700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8628089c40 con 0x7f8628130750 2019-12-05T06:52:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.558+0000 7f86277fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8628080290 0x7f862812f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.558+0000 7f86277fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8628080290 0x7f862812f780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34524/0 (socket says 172.21.15.16:34524) 2019-12-05T06:52:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.558+0000 7f86277fe700 1 -- 172.21.15.16:0/1292584390 learned_addr learned my addr 172.21.15.16:0/1292584390 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.558+0000 7f8626ffd700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8628130750 0x7f862812fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.559+0000 7f86277fe700 1 -- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8628130750 msgr2=0x7f862812fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.559+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8628130750 0x7f862812fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:29.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.559+0000 7f86277fe700 1 -- 172.21.15.16:0/1292584390 >> v1:172.21.15.16:6789/0 conn(0x7f8628130200 legacy=0x7f8628080680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:29.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.559+0000 7f86277fe700 1 -- 172.21.15.16:0/1292584390 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862812ee80 con 0x7f8628080290 2019-12-05T06:52:29.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.559+0000 7f861ffff700 1 -- 172.21.15.16:0/1292584390 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 756567782 0 0) 0x7f8618001080 con 0x7f8628130200 2019-12-05T06:52:29.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.559+0000 7f8626ffd700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8628130750 0x7f862812fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:52:29.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.559+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> v2:172.21.15.16:3300/0 conn(0x7f8628080290 0x7f862812f780 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.559+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> v2:172.21.15.16:3300/0 conn(0x7f8628080290 0x7f862812f780 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.559+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 0x7f862812f780 secure :-1 s=READY pgs=1435 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:29.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.560+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 0x7f862812f780 secure :-1 s=THROTTLE_DONE pgs=1435 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.560+0000 7f861ffff700 1 -- 172.21.15.16:0/1292584390 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86200080c0 con 0x7f8628080290 2019-12-05T06:52:29.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.560+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 0x7f862812f780 secure :-1 s=THROTTLE_DONE pgs=1435 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.560+0000 7f861ffff700 1 -- 172.21.15.16:0/1292584390 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86200056f0 con 0x7f8628080290 2019-12-05T06:52:29.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.560+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 0x7f862812f780 secure :-1 s=THROTTLE_DONE pgs=1435 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.560+0000 7f862d356700 1 -- 172.21.15.16:0/1292584390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f862812ee80 con 0x7f8628080290 2019-12-05T06:52:29.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.560+0000 7f861ffff700 1 -- 172.21.15.16:0/1292584390 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86200058e0 con 0x7f8628080290 2019-12-05T06:52:29.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.560+0000 7f862d356700 1 -- 172.21.15.16:0/1292584390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f862812ee80 con 0x7f8628080290 2019-12-05T06:52:29.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.561+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 0x7f862812f780 secure :-1 s=THROTTLE_DONE pgs=1435 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.562+0000 7f861ffff700 1 -- 172.21.15.16:0/1292584390 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8620008050 con 0x7f8628080290 2019-12-05T06:52:29.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.562+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 0x7f862812f780 secure :-1 s=THROTTLE_DONE pgs=1435 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.562+0000 7f861ffff700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f860c023630 0x7f860c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:29.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.562+0000 7f861ffff700 1 -- 172.21.15.16:0/1292584390 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f862008a110 con 0x7f8628080290 2019-12-05T06:52:29.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.563+0000 7f8626ffd700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f860c023630 0x7f860c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:29.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.563+0000 7f8626ffd700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f860c023630 0x7f860c025ac0 crc :-1 s=READY pgs=603 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:29.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.576+0000 7f862d356700 1 -- 172.21.15.16:0/1292584390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f862812ee80 con 0x7f8628080290 2019-12-05T06:52:29.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.587+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 0x7f862812f780 secure :-1 s=THROTTLE_DONE pgs=1435 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.587+0000 7f861ffff700 1 -- 172.21.15.16:0/1292584390 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8620000cc0 con 0x7f8628080290 2019-12-05T06:52:29.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.839+0000 7f862d356700 1 -- 172.21.15.16:0/1292584390 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f862812ee80 con 0x7f8628080290 2019-12-05T06:52:29.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.840+0000 7f86277fe700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 0x7f862812f780 secure :-1 s=THROTTLE_DONE pgs=1435 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:29.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.840+0000 7f861ffff700 1 -- 172.21.15.16:0/1292584390 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 176 v176) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f8620008050 con 0x7f8628080290 2019-12-05T06:52:29.841 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 176 2019-12-05T06:52:29.841 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:29.841 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":176,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.842+0000 7f862d356700 1 -- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f860c023630 msgr2=0x7f860c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.842+0000 7f862d356700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f860c023630 0x7f860c025ac0 crc :-1 s=READY pgs=603 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.842+0000 7f862d356700 1 -- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 msgr2=0x7f862812f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.842+0000 7f862d356700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 0x7f862812f780 secure :-1 s=READY pgs=1435 cs=0 l=1 rx=0x7f86200547f0 tx=0x7f8620054b90).stop 2019-12-05T06:52:29.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.843+0000 7f862d356700 1 -- 172.21.15.16:0/1292584390 shutdown_connections 2019-12-05T06:52:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.843+0000 7f862d356700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8628130750 0x7f862812fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.843+0000 7f862d356700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f860c023630 0x7f860c025ac0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.843+0000 7f862d356700 1 --2- 172.21.15.16:0/1292584390 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8628080290 0x7f862812f780 unknown :-1 s=CLOSED pgs=1435 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.843+0000 7f862d356700 1 -- 172.21.15.16:0/1292584390 shutdown_connections 2019-12-05T06:52:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.843+0000 7f862d356700 1 -- 172.21.15.16:0/1292584390 wait complete. 2019-12-05T06:52:29.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:29.843+0000 7f862d356700 1 -- 172.21.15.16:0/1292584390 >> 172.21.15.16:0/1292584390 conn(0x7f862809de30 msgr2=0x7f8628070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:29.853 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:52:31.854 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:32.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.004+0000 7fa6ba794700 1 Processor -- start 2019-12-05T06:52:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.004+0000 7fa6ba794700 1 -- start start 2019-12-05T06:52:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.005+0000 7fa6ba794700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6b408c6d0 0x7fa6b408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.005+0000 7fa6ba794700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa6b408f720 0x7fa6b408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:32.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.005+0000 7fa6ba794700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b408fb80 con 0x7fa6b408fef0 2019-12-05T06:52:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.005+0000 7fa6ba794700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b408df90 con 0x7fa6b408c6d0 2019-12-05T06:52:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.005+0000 7fa6ba794700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b40a3f70 con 0x7fa6b408f720 2019-12-05T06:52:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.005+0000 7fa6b8d31700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6b408c6d0 0x7fa6b408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.005+0000 7fa6b8d31700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa6b408c6d0 0x7fa6b408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34554/0 (socket says 172.21.15.16:34554) 2019-12-05T06:52:32.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.005+0000 7fa6b8d31700 1 -- 172.21.15.16:0/878206143 learned_addr learned my addr 172.21.15.16:0/878206143 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.005+0000 7fa6b9532700 1 --2- 172.21.15.16:0/878206143 >> v2:172.21.15.76:3300/0 conn(0x7fa6b408f720 0x7fa6b408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b8d31700 1 -- 172.21.15.16:0/878206143 >> v2:172.21.15.76:3300/0 conn(0x7fa6b408f720 msgr2=0x7fa6b408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/878206143 >> v2:172.21.15.76:3300/0 conn(0x7fa6b408f720 0x7fa6b408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b8d31700 1 -- 172.21.15.16:0/878206143 >> v1:172.21.15.16:6789/0 conn(0x7fa6b408fef0 legacy=0x7fa6b408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b37fe700 1 -- 172.21.15.16:0/878206143 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699248317 0 0) 0x7fa6a4001100 con 0x7fa6b408fef0 2019-12-05T06:52:32.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b8d31700 1 -- 172.21.15.16:0/878206143 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b408d540 con 0x7fa6b408c6d0 2019-12-05T06:52:32.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b9532700 1 --2- 172.21.15.16:0/878206143 >> v2:172.21.15.76:3300/0 conn(0x7fa6b408f720 0x7fa6b408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:52:32.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/878206143 >> v2:172.21.15.16:3300/0 conn(0x7fa6b408c6d0 0x7fa6b408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa69c00f740 tx=0x7fa69c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/878206143 >> v2:172.21.15.16:3300/0 conn(0x7fa6b408c6d0 0x7fa6b408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa69c00f740 tx=0x7fa69c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/878206143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6b408c6d0 0x7fa6b408cac0 secure :-1 s=READY pgs=1437 cs=0 l=1 rx=0x7fa69c00f740 tx=0x7fa69c00fae0).ready entity=mon.0 client_cookie=dc178351a7025df4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:32.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/878206143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6b408c6d0 0x7fa6b408cac0 secure :-1 s=THROTTLE_DONE pgs=1437 cs=0 l=1 rx=0x7fa69c00f740 tx=0x7fa69c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b37fe700 1 -- 172.21.15.16:0/878206143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa69c000960 con 0x7fa6b408c6d0 2019-12-05T06:52:32.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/878206143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6b408c6d0 0x7fa6b408cac0 secure :-1 s=THROTTLE_DONE pgs=1437 cs=0 l=1 rx=0x7fa69c00f740 tx=0x7fa69c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/878206143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6b408c6d0 0x7fa6b408cac0 secure :-1 s=THROTTLE_DONE pgs=1437 cs=0 l=1 rx=0x7fa69c00f740 tx=0x7fa69c00fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b37fe700 1 -- 172.21.15.16:0/878206143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa69c000c90 con 0x7fa6b408c6d0 2019-12-05T06:52:32.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6b37fe700 1 -- 172.21.15.16:0/878206143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa69c000960 con 0x7fa6b408c6d0 2019-12-05T06:52:32.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6ba794700 1 -- 172.21.15.16:0/878206143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6b408c6d0 msgr2=0x7fa6b408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:32.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.006+0000 7fa6ba794700 1 --2- 172.21.15.16:0/878206143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6b408c6d0 0x7fa6b408cac0 secure :-1 s=READY pgs=1437 cs=0 l=1 rx=0x7fa69c00f740 tx=0x7fa69c00fae0).stop 2019-12-05T06:52:32.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.007+0000 7fa6ba794700 1 -- 172.21.15.16:0/878206143 shutdown_connections 2019-12-05T06:52:32.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.007+0000 7fa6ba794700 1 --2- 172.21.15.16:0/878206143 >> v2:172.21.15.76:3300/0 conn(0x7fa6b408f720 0x7fa6b408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:32.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.007+0000 7fa6ba794700 1 --2- 172.21.15.16:0/878206143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa6b408c6d0 0x7fa6b408cac0 unknown :-1 s=CLOSED pgs=1437 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:32.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.007+0000 7fa6ba794700 1 -- 172.21.15.16:0/878206143 shutdown_connections 2019-12-05T06:52:32.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.007+0000 7fa6ba794700 1 -- 172.21.15.16:0/878206143 wait complete. 2019-12-05T06:52:32.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.007+0000 7fa6ba794700 1 -- 172.21.15.16:0/878206143 >> 172.21.15.16:0/878206143 conn(0x7fa6b4061940 msgr2=0x7fa6b4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:32.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.009+0000 7fa6ba794700 1 Processor -- start 2019-12-05T06:52:32.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.009+0000 7fa6ba794700 1 -- start start 2019-12-05T06:52:32.021 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.010+0000 7fa6ba794700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa6b4132d00 0x7fa6b414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:32.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.010+0000 7fa6ba794700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6b4132530 0x7fa6b4157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:32.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.010+0000 7fa6ba794700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b4132980 con 0x7fa6b414d400 2019-12-05T06:52:32.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.010+0000 7fa6ba794700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b409bb50 con 0x7fa6b4132d00 2019-12-05T06:52:32.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.010+0000 7fa6ba794700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b409aa60 con 0x7fa6b4132530 2019-12-05T06:52:32.022 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b8d31700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa6b414d400 0x7fa6b4131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:47960/0 (socket says 172.21.15.16:47960) 2019-12-05T06:52:32.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b8d31700 1 -- 172.21.15.16:0/4036339631 learned_addr learned my addr 172.21.15.16:0/4036339631 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:32.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b3fff700 1 --2- 172.21.15.16:0/4036339631 >> v2:172.21.15.76:3300/0 conn(0x7fa6b4132d00 0x7fa6b414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:32.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6b4132530 0x7fa6b4157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:32.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643361464 0 0) 0x7fa69c0075d0 con 0x7fa6b414d400 2019-12-05T06:52:32.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa698002e80 con 0x7fa6b414d400 2019-12-05T06:52:32.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 753314169 0 0) 0x7fa69c007da0 con 0x7fa6b414d400 2019-12-05T06:52:32.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6980032e0 con 0x7fa6b414d400 2019-12-05T06:52:32.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fa69c000940 con 0x7fa6b414d400 2019-12-05T06:52:32.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6b4132530 msgr2=0x7fa6b4157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:32.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa6b4132530 0x7fa6b4157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:32.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 >> v2:172.21.15.76:3300/0 conn(0x7fa6b4132d00 msgr2=0x7fa6b414d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:32.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 --2- 172.21.15.16:0/4036339631 >> v2:172.21.15.76:3300/0 conn(0x7fa6b4132d00 0x7fa6b414d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:32.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 >> v1:172.21.15.16:6789/0 conn(0x7fa6b414d400 legacy=0x7fa6b4131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:32.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6980033a0 0x7fa6980037f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:32.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa698003d30 0x7fa698004120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:32.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:32.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69800bb60 con 0x7fa698004800 2019-12-05T06:52:32.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69800bd50 con 0x7fa6980033a0 2019-12-05T06:52:32.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa698014790 con 0x7fa698003d30 2019-12-05T06:52:32.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:32.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.011+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6980033a0 0x7fa6980037f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:32.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b3fff700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa698003d30 0x7fa698004120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:32.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 -- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa698003d30 msgr2=0x7fa698004120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa698003d30 0x7fa698004120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 -- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6980033a0 msgr2=0x7fa6980037f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6980033a0 0x7fa6980037f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 -- 172.21.15.16:0/4036339631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa698014a00 con 0x7fa698004800 2019-12-05T06:52:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b3fff700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa698003d30 0x7fa698004120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:52:32.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa6980033a0 0x7fa6980037f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:52:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6ba794700 1 -- 172.21.15.16:0/4036339631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b409b0f0 con 0x7fa698004800 2019-12-05T06:52:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=READY pgs=1439 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6ba794700 1 -- 172.21.15.16:0/4036339631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b4132980 con 0x7fa698004800 2019-12-05T06:52:32.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=THROTTLE_DONE pgs=1439 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=THROTTLE_DONE pgs=1439 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6ac006580 con 0x7fa698004800 2019-12-05T06:52:32.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=THROTTLE_DONE pgs=1439 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6ac005950 con 0x7fa698004800 2019-12-05T06:52:32.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6ac006020 con 0x7fa698004800 2019-12-05T06:52:32.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.012+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=THROTTLE_DONE pgs=1439 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.013+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa6ac006240 con 0x7fa698004800 2019-12-05T06:52:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.013+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=THROTTLE_DONE pgs=1439 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.013+0000 7fa6b17fa700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa698034200 0x7fa698036630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.013+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7fa6ac046050 con 0x7fa698004800 2019-12-05T06:52:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.013+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa698034200 0x7fa698036630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:32.030 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.014+0000 7fa6b8d31700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa698034200 0x7fa698036630 crc :-1 s=READY pgs=605 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:32.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.019+0000 7fa6ba794700 1 -- 172.21.15.16:0/4036339631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa68c002e60 con 0x7fa698004800 2019-12-05T06:52:32.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.023+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=THROTTLE_DONE pgs=1439 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.031 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.023+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa6ac005550 con 0x7fa698004800 2019-12-05T06:52:32.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.224+0000 7fa6ba794700 1 -- 172.21.15.16:0/4036339631 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa68c001d10 con 0x7fa698004800 2019-12-05T06:52:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.224+0000 7fa6b9532700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=THROTTLE_DONE pgs=1439 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.224+0000 7fa6b17fa700 1 -- 172.21.15.16:0/4036339631 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+2106 (secure 0 0 0) 0x7fa6ac006200 con 0x7fa698004800 2019-12-05T06:52:32.224 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 177 2019-12-05T06:52:32.225 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:32.225 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7857,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.227+0000 7fa6ba794700 1 -- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa698034200 msgr2=0x7fa698036630 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.227+0000 7fa6ba794700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa698034200 0x7fa698036630 crc :-1 s=READY pgs=605 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.227+0000 7fa6ba794700 1 -- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 msgr2=0x7fa698004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.227+0000 7fa6ba794700 1 --2- 172.21.15.16:0/4036339631 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa698004800 0x7fa698004cc0 secure :-1 s=READY pgs=1439 cs=0 l=1 rx=0x7fa6ac0106f0 tx=0x7fa6ac010a90).stop 2019-12-05T06:52:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.227+0000 7fa6b9532700 1 -- 172.21.15.16:0/4036339631 reap_dead start 2019-12-05T06:52:32.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.227+0000 7fa6b9532700 1 -- 172.21.15.16:0/4036339631 reap_dead start 2019-12-05T06:52:32.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.227+0000 7fa6ba794700 1 -- 172.21.15.16:0/4036339631 shutdown_connections 2019-12-05T06:52:32.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.227+0000 7fa6ba794700 1 -- 172.21.15.16:0/4036339631 shutdown_connections 2019-12-05T06:52:32.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.227+0000 7fa6ba794700 1 -- 172.21.15.16:0/4036339631 wait complete. 2019-12-05T06:52:32.236 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:32.227+0000 7fa6ba794700 1 -- 172.21.15.16:0/4036339631 >> 172.21.15.16:0/4036339631 conn(0x7fa6b40714f0 msgr2=0x7fa6b409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:32.238 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:52:32.238 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2019-12-05T06:52:45.821 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:45.826 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:52:49.239 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.519+0000 7f351ce03700 1 Processor -- start 2019-12-05T06:52:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.519+0000 7f351ce03700 1 -- start start 2019-12-05T06:52:49.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.521+0000 7f351ce03700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f351808c6d0 0x7f351808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.521+0000 7f351ce03700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f351808fef0 0x7f351808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.521+0000 7f351ce03700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f351808fb80 con 0x7f351808ef20 2019-12-05T06:52:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.521+0000 7f351ce03700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351808df90 con 0x7f351808c6d0 2019-12-05T06:52:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.521+0000 7f351ce03700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f35180a3f70 con 0x7f351808fef0 2019-12-05T06:52:49.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.522+0000 7f3516d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f351808c6d0 0x7f351808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.522+0000 7f3516d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f351808c6d0 0x7f351808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34636/0 (socket says 172.21.15.16:34636) 2019-12-05T06:52:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.522+0000 7f351659c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f351808fef0 0x7f351808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.522+0000 7f3516d9d700 1 -- 172.21.15.16:0/1054543214 learned_addr learned my addr 172.21.15.16:0/1054543214 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.522+0000 7f3515d9b700 1 -- 172.21.15.16:0/1054543214 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340358485 0 0) 0x7f35100009b0 con 0x7f351808ef20 2019-12-05T06:52:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.522+0000 7f3516d9d700 1 -- 172.21.15.16:0/1054543214 >> v2:172.21.15.76:3300/0 conn(0x7f351808fef0 msgr2=0x7f351808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.522+0000 7f3516d9d700 1 --2- 172.21.15.16:0/1054543214 >> v2:172.21.15.76:3300/0 conn(0x7f351808fef0 0x7f351808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:49.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.523+0000 7f3516d9d700 1 -- 172.21.15.16:0/1054543214 >> v1:172.21.15.76:6789/0 conn(0x7f351808ef20 legacy=0x7f351808f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:49.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.523+0000 7f3516d9d700 1 -- 172.21.15.16:0/1054543214 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351808d540 con 0x7f351808c6d0 2019-12-05T06:52:49.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.523+0000 7f351659c700 1 --2- 172.21.15.16:0/1054543214 >> v2:172.21.15.76:3300/0 conn(0x7f351808fef0 0x7f351808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:52:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.523+0000 7f3516d9d700 1 --2- 172.21.15.16:0/1054543214 >> v2:172.21.15.16:3300/0 conn(0x7f351808c6d0 0x7f351808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f350800f740 tx=0x7f350800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.523+0000 7f3516d9d700 1 --2- 172.21.15.16:0/1054543214 >> v2:172.21.15.16:3300/0 conn(0x7f351808c6d0 0x7f351808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f350800f740 tx=0x7f350800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.523+0000 7f3516d9d700 1 --2- 172.21.15.16:0/1054543214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f351808c6d0 0x7f351808cac0 secure :-1 s=READY pgs=1441 cs=0 l=1 rx=0x7f350800f740 tx=0x7f350800fae0).ready entity=mon.0 client_cookie=71e849f74b00c521 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.523+0000 7f3516d9d700 1 --2- 172.21.15.16:0/1054543214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f351808c6d0 0x7f351808cac0 secure :-1 s=THROTTLE_DONE pgs=1441 cs=0 l=1 rx=0x7f350800f740 tx=0x7f350800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.523+0000 7f3515d9b700 1 -- 172.21.15.16:0/1054543214 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3508000960 con 0x7f351808c6d0 2019-12-05T06:52:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.523+0000 7f3516d9d700 1 --2- 172.21.15.16:0/1054543214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f351808c6d0 0x7f351808cac0 secure :-1 s=THROTTLE_DONE pgs=1441 cs=0 l=1 rx=0x7f350800f740 tx=0x7f350800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.523+0000 7f3516d9d700 1 --2- 172.21.15.16:0/1054543214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f351808c6d0 0x7f351808cac0 secure :-1 s=THROTTLE_DONE pgs=1441 cs=0 l=1 rx=0x7f350800f740 tx=0x7f350800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.524+0000 7f3515d9b700 1 -- 172.21.15.16:0/1054543214 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3508000c90 con 0x7f351808c6d0 2019-12-05T06:52:49.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.524+0000 7f3515d9b700 1 -- 172.21.15.16:0/1054543214 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35080013a0 con 0x7f351808c6d0 2019-12-05T06:52:49.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.524+0000 7f351ce03700 1 -- 172.21.15.16:0/1054543214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f351808c6d0 msgr2=0x7f351808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:49.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.524+0000 7f351ce03700 1 --2- 172.21.15.16:0/1054543214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f351808c6d0 0x7f351808cac0 secure :-1 s=READY pgs=1441 cs=0 l=1 rx=0x7f350800f740 tx=0x7f350800fae0).stop 2019-12-05T06:52:49.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.526+0000 7f351ce03700 1 -- 172.21.15.16:0/1054543214 shutdown_connections 2019-12-05T06:52:49.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.526+0000 7f351ce03700 1 --2- 172.21.15.16:0/1054543214 >> v2:172.21.15.76:3300/0 conn(0x7f351808fef0 0x7f351808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:49.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.526+0000 7f351ce03700 1 --2- 172.21.15.16:0/1054543214 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f351808c6d0 0x7f351808cac0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:49.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.526+0000 7f351ce03700 1 -- 172.21.15.16:0/1054543214 shutdown_connections 2019-12-05T06:52:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.526+0000 7f351ce03700 1 -- 172.21.15.16:0/1054543214 wait complete. 2019-12-05T06:52:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.526+0000 7f351ce03700 1 -- 172.21.15.16:0/1054543214 >> 172.21.15.16:0/1054543214 conn(0x7f3518061940 msgr2=0x7f3518096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.531+0000 7f351ce03700 1 Processor -- start 2019-12-05T06:52:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.531+0000 7f351ce03700 1 -- start start 2019-12-05T06:52:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f351ce03700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f351806df10 0x7f3518098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:49.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f351ce03700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f35180992d0 0x7f3518098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f351ce03700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3518098d80 0x7f3518088fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f351ce03700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f351808ad00 con 0x7f35180992d0 2019-12-05T06:52:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f351ce03700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3518096c10 con 0x7f3518098d80 2019-12-05T06:52:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f351ce03700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3518097a00 con 0x7f351806df10 2019-12-05T06:52:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f351659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f35180992d0 0x7f3518098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f351659c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f35180992d0 0x7f3518098840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34646/0 (socket says 172.21.15.16:34646) 2019-12-05T06:52:49.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f351659c700 1 -- 172.21.15.16:0/3064464874 learned_addr learned my addr 172.21.15.16:0/3064464874 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f3516d9d700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f351806df10 0x7f3518098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.533+0000 7f351759e700 1 --2- 172.21.15.16:0/3064464874 >> v2:172.21.15.76:3300/0 conn(0x7f3518098d80 0x7f3518088fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.534+0000 7f351659c700 1 -- 172.21.15.16:0/3064464874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f351806df10 msgr2=0x7f3518098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.534+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f351806df10 0x7f3518098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.534+0000 7f351659c700 1 -- 172.21.15.16:0/3064464874 >> v2:172.21.15.76:3300/0 conn(0x7f3518098d80 msgr2=0x7f3518088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.534+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> v2:172.21.15.76:3300/0 conn(0x7f3518098d80 0x7f3518088fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:49.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.534+0000 7f351659c700 1 -- 172.21.15.16:0/3064464874 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f351806e300 con 0x7f35180992d0 2019-12-05T06:52:49.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.534+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> v2:172.21.15.16:3300/0 conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.534+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> v2:172.21.15.16:3300/0 conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.534+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=READY pgs=1442 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:49.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.535+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=THROTTLE_DONE pgs=1442 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.535+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=THROTTLE_DONE pgs=1442 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.535+0000 7f350f7fe700 1 -- 172.21.15.16:0/3064464874 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3500000cc0 con 0x7f35180992d0 2019-12-05T06:52:49.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.535+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=THROTTLE_DONE pgs=1442 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.535+0000 7f350f7fe700 1 -- 172.21.15.16:0/3064464874 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3500009af0 con 0x7f35180992d0 2019-12-05T06:52:49.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.535+0000 7f351ce03700 1 -- 172.21.15.16:0/3064464874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f351806e300 con 0x7f35180992d0 2019-12-05T06:52:49.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.535+0000 7f350f7fe700 1 -- 172.21.15.16:0/3064464874 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f350000a180 con 0x7f35180992d0 2019-12-05T06:52:49.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.535+0000 7f351ce03700 1 -- 172.21.15.16:0/3064464874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f351806e300 con 0x7f35180992d0 2019-12-05T06:52:49.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.536+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=THROTTLE_DONE pgs=1442 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.537+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=THROTTLE_DONE pgs=1442 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.537+0000 7f350f7fe700 1 -- 172.21.15.16:0/3064464874 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3500009ce0 con 0x7f35180992d0 2019-12-05T06:52:49.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.537+0000 7f350f7fe700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34fc023630 0x7f34fc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:49.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.537+0000 7f350f7fe700 1 -- 172.21.15.16:0/3064464874 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f35000460b0 con 0x7f35180992d0 2019-12-05T06:52:49.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.538+0000 7f3516d9d700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34fc023630 0x7f34fc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:49.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.538+0000 7f3516d9d700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34fc023630 0x7f34fc025ac0 crc :-1 s=READY pgs=606 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:49.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.551+0000 7f351ce03700 1 -- 172.21.15.16:0/3064464874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35180958e0 con 0x7f35180992d0 2019-12-05T06:52:49.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.560+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=THROTTLE_DONE pgs=1442 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.560+0000 7f350f7fe700 1 -- 172.21.15.16:0/3064464874 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f350000a930 con 0x7f35180992d0 2019-12-05T06:52:49.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.815+0000 7f351ce03700 1 -- 172.21.15.16:0/3064464874 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f35180958e0 con 0x7f35180992d0 2019-12-05T06:52:49.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.816+0000 7f351659c700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=THROTTLE_DONE pgs=1442 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.816+0000 7f350f7fe700 1 -- 172.21.15.16:0/3064464874 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f3500009be0 con 0x7f35180992d0 2019-12-05T06:52:49.817 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 177 2019-12-05T06:52:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:49.817 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7857,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:49.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 -- 172.21.15.16:0/3064464874 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34fc023630 msgr2=0x7f34fc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:49.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34fc023630 0x7f34fc025ac0 crc :-1 s=READY pgs=606 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:49.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 -- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 msgr2=0x7f3518098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:49.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 0x7f3518098840 secure :-1 s=READY pgs=1442 cs=0 l=1 rx=0x7f3500010790 tx=0x7f3500010610).stop 2019-12-05T06:52:49.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 -- 172.21.15.16:0/3064464874 shutdown_connections 2019-12-05T06:52:49.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 --2- 172.21.15.16:0/3064464874 >> v2:172.21.15.76:3300/0 conn(0x7f3518098d80 0x7f3518088fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:49.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f34fc023630 0x7f34fc025ac0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:49.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f35180992d0 0x7f3518098840 unknown :-1 s=CLOSED pgs=1442 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:49.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 --2- 172.21.15.16:0/3064464874 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f351806df10 0x7f3518098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:49.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 -- 172.21.15.16:0/3064464874 shutdown_connections 2019-12-05T06:52:49.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 -- 172.21.15.16:0/3064464874 wait complete. 2019-12-05T06:52:49.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.819+0000 7f351ce03700 1 -- 172.21.15.16:0/3064464874 >> 172.21.15.16:0/3064464874 conn(0x7f35180714f0 msgr2=0x7f351809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:49.830 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:52:49.831 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:52:49.831 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:52:49.842 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:52:49.843 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:49.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.989+0000 7f0f75b7a700 1 Processor -- start 2019-12-05T06:52:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.989+0000 7f0f75b7a700 1 -- start start 2019-12-05T06:52:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.990+0000 7f0f75b7a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7008c6d0 0x7f0f7008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.990+0000 7f0f75b7a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f7008fb80 con 0x7f0f7008f720 2019-12-05T06:52:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.990+0000 7f0f75b7a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f700958e0 con 0x7f0f7008fef0 2019-12-05T06:52:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.990+0000 7f0f75b7a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7008df90 con 0x7f0f7008c6d0 2019-12-05T06:52:49.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.990+0000 7f0f6ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7008c6d0 0x7f0f7008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.990+0000 7f0f6ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7008c6d0 0x7f0f7008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34654/0 (socket says 172.21.15.16:34654) 2019-12-05T06:52:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.990+0000 7f0f6ffff700 1 -- 172.21.15.16:0/106889490 learned_addr learned my addr 172.21.15.16:0/106889490 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6ffff700 1 -- 172.21.15.16:0/106889490 >> v1:172.21.15.76:6789/0 conn(0x7f0f7008fef0 legacy=0x7f0f7008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6ffff700 1 -- 172.21.15.16:0/106889490 >> v1:172.21.15.16:6789/0 conn(0x7f0f7008f720 legacy=0x7f0f701533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6f7fe700 1 -- 172.21.15.16:0/106889490 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 239362952 0 0) 0x7f0f68001480 con 0x7f0f7008f720 2019-12-05T06:52:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6ffff700 1 -- 172.21.15.16:0/106889490 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f7008e620 con 0x7f0f7008c6d0 2019-12-05T06:52:49.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/106889490 >> v2:172.21.15.16:3300/0 conn(0x7f0f7008c6d0 0x7f0f7008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0f6000f740 tx=0x7f0f6000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/106889490 >> v2:172.21.15.16:3300/0 conn(0x7f0f7008c6d0 0x7f0f7008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0f6000f740 tx=0x7f0f6000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/106889490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7008c6d0 0x7f0f7008cac0 secure :-1 s=READY pgs=1444 cs=0 l=1 rx=0x7f0f6000f740 tx=0x7f0f6000fae0).ready entity=mon.0 client_cookie=13e5022bf051466 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/106889490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7008c6d0 0x7f0f7008cac0 secure :-1 s=THROTTLE_DONE pgs=1444 cs=0 l=1 rx=0x7f0f6000f740 tx=0x7f0f6000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6f7fe700 1 -- 172.21.15.16:0/106889490 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f60000960 con 0x7f0f7008c6d0 2019-12-05T06:52:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/106889490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7008c6d0 0x7f0f7008cac0 secure :-1 s=THROTTLE_DONE pgs=1444 cs=0 l=1 rx=0x7f0f6000f740 tx=0x7f0f6000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/106889490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7008c6d0 0x7f0f7008cac0 secure :-1 s=THROTTLE_DONE pgs=1444 cs=0 l=1 rx=0x7f0f6000f740 tx=0x7f0f6000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:49.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6f7fe700 1 -- 172.21.15.16:0/106889490 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f60000c90 con 0x7f0f7008c6d0 2019-12-05T06:52:49.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f6f7fe700 1 -- 172.21.15.16:0/106889490 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f60000960 con 0x7f0f7008c6d0 2019-12-05T06:52:49.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f75b7a700 1 -- 172.21.15.16:0/106889490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7008c6d0 msgr2=0x7f0f7008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:49.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.991+0000 7f0f75b7a700 1 --2- 172.21.15.16:0/106889490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7008c6d0 0x7f0f7008cac0 secure :-1 s=READY pgs=1444 cs=0 l=1 rx=0x7f0f6000f740 tx=0x7f0f6000fae0).stop 2019-12-05T06:52:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.992+0000 7f0f75b7a700 1 -- 172.21.15.16:0/106889490 shutdown_connections 2019-12-05T06:52:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.992+0000 7f0f75b7a700 1 --2- 172.21.15.16:0/106889490 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7008c6d0 0x7f0f7008cac0 unknown :-1 s=CLOSED pgs=1444 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.992+0000 7f0f75b7a700 1 -- 172.21.15.16:0/106889490 shutdown_connections 2019-12-05T06:52:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.992+0000 7f0f75b7a700 1 -- 172.21.15.16:0/106889490 wait complete. 2019-12-05T06:52:50.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.992+0000 7f0f75b7a700 1 -- 172.21.15.16:0/106889490 >> 172.21.15.16:0/106889490 conn(0x7f0f70061940 msgr2=0x7f0f70096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.995+0000 7f0f75b7a700 1 Processor -- start 2019-12-05T06:52:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.995+0000 7f0f75b7a700 1 -- start start 2019-12-05T06:52:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.996+0000 7f0f75b7a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f70150970 0x7f0f70150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.996+0000 7f0f75b7a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0f701309e0 0x7f0f7012fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.996+0000 7f0f75b7a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7012ff50 0x7f0f70080210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.996+0000 7f0f75b7a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f70089ec0 con 0x7f0f7012ff50 2019-12-05T06:52:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.996+0000 7f0f75b7a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f7008a190 con 0x7f0f701309e0 2019-12-05T06:52:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.996+0000 7f0f75b7a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f7012f110 con 0x7f0f70150970 2019-12-05T06:52:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.996+0000 7f0f74918700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7012ff50 0x7f0f70080210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f74918700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0f7012ff50 0x7f0f70080210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34664/0 (socket says 172.21.15.16:34664) 2019-12-05T06:52:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f74918700 1 -- 172.21.15.16:0/41880523 learned_addr learned my addr 172.21.15.16:0/41880523 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f70150970 0x7f0f70150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:50.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f677fe700 1 --2- 172.21.15.16:0/41880523 >> v2:172.21.15.76:3300/0 conn(0x7f0f701309e0 0x7f0f7012fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f74918700 1 -- 172.21.15.16:0/41880523 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f70150970 msgr2=0x7f0f70150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f70150970 0x7f0f70150d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f74918700 1 -- 172.21.15.16:0/41880523 >> v2:172.21.15.76:3300/0 conn(0x7f0f701309e0 msgr2=0x7f0f7012fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:50.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> v2:172.21.15.76:3300/0 conn(0x7f0f701309e0 0x7f0f7012fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:50.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f74918700 1 -- 172.21.15.16:0/41880523 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f70088c30 con 0x7f0f7012ff50 2019-12-05T06:52:50.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> v2:172.21.15.16:3300/0 conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:50.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> v2:172.21.15.16:3300/0 conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:50.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.997+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=READY pgs=1445 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:50.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.998+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=THROTTLE_DONE pgs=1445 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:50.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.998+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=THROTTLE_DONE pgs=1445 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:50.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.998+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/41880523 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f68005b90 con 0x7f0f7012ff50 2019-12-05T06:52:50.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.998+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=THROTTLE_DONE pgs=1445 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:50.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.998+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/41880523 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f680057b0 con 0x7f0f7012ff50 2019-12-05T06:52:50.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.998+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/41880523 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f680059a0 con 0x7f0f7012ff50 2019-12-05T06:52:50.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.998+0000 7f0f75b7a700 1 -- 172.21.15.16:0/41880523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f70088c30 con 0x7f0f7012ff50 2019-12-05T06:52:50.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.998+0000 7f0f75b7a700 1 -- 172.21.15.16:0/41880523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f70088730 con 0x7f0f7012ff50 2019-12-05T06:52:50.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:49.999+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=THROTTLE_DONE pgs=1445 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:50.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.000+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/41880523 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0f680059a0 con 0x7f0f7012ff50 2019-12-05T06:52:50.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.000+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=THROTTLE_DONE pgs=1445 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:50.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.000+0000 7f0f6d7fa700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f54023630 0x7f0f54025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:50.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.000+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/41880523 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f0f680460d0 con 0x7f0f7012ff50 2019-12-05T06:52:50.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.000+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f54023630 0x7f0f54025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:50.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.001+0000 7f0f6ffff700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f54023630 0x7f0f54025ac0 crc :-1 s=READY pgs=607 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:50.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.005+0000 7f0f75b7a700 1 -- 172.21.15.16:0/41880523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f7012f110 con 0x7f0f7012ff50 2019-12-05T06:52:50.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.009+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=THROTTLE_DONE pgs=1445 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:50.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.009+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/41880523 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0f68007920 con 0x7f0f7012ff50 2019-12-05T06:52:50.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.224+0000 7f0f75b7a700 1 -- 172.21.15.16:0/41880523 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0f7012f110 con 0x7f0f7012ff50 2019-12-05T06:52:50.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.225+0000 7f0f74918700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=THROTTLE_DONE pgs=1445 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:50.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.225+0000 7f0f6d7fa700 1 -- 172.21.15.16:0/41880523 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f0f68005980 con 0x7f0f7012ff50 2019-12-05T06:52:50.226 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 177 2019-12-05T06:52:50.227 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:50.227 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7857,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:50.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 -- 172.21.15.16:0/41880523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f54023630 msgr2=0x7f0f54025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:50.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f54023630 0x7f0f54025ac0 crc :-1 s=READY pgs=607 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:50.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 -- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 msgr2=0x7f0f70080210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:50.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 0x7f0f70080210 secure :-1 s=READY pgs=1445 cs=0 l=1 rx=0x7f0f68010710 tx=0x7f0f68010610).stop 2019-12-05T06:52:50.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 -- 172.21.15.16:0/41880523 shutdown_connections 2019-12-05T06:52:50.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0f54023630 0x7f0f54025ac0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:50.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0f7012ff50 0x7f0f70080210 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:50.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 --2- 172.21.15.16:0/41880523 >> v2:172.21.15.76:3300/0 conn(0x7f0f701309e0 0x7f0f7012fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:50.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 --2- 172.21.15.16:0/41880523 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0f70150970 0x7f0f70150d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:50.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 -- 172.21.15.16:0/41880523 shutdown_connections 2019-12-05T06:52:50.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 -- 172.21.15.16:0/41880523 wait complete. 2019-12-05T06:52:50.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:50.227+0000 7f0f75b7a700 1 -- 172.21.15.16:0/41880523 >> 172.21.15.16:0/41880523 conn(0x7f0f7009de30 msgr2=0x7f0f70070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:50.239 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:52:52.240 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.506+0000 7f6a3ef3c700 1 Processor -- start 2019-12-05T06:52:52.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.506+0000 7f6a3ef3c700 1 -- start start 2019-12-05T06:52:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.508+0000 7f6a3ef3c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6a3808d2c0 0x7f6a3808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.508+0000 7f6a3ef3c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a3808fef0 0x7f6a3808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.508+0000 7f6a3ef3c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a380958e0 con 0x7f6a3808c6d0 2019-12-05T06:52:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.508+0000 7f6a3ef3c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a3808df90 con 0x7f6a3808fef0 2019-12-05T06:52:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.508+0000 7f6a3ef3c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a380a3f70 con 0x7f6a3808d2c0 2019-12-05T06:52:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.509+0000 7f6a3dcda700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a3808fef0 0x7f6a3808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:52.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.509+0000 7f6a3dcda700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6a3808fef0 0x7f6a3808f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34676/0 (socket says 172.21.15.16:34676) 2019-12-05T06:52:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.509+0000 7f6a3dcda700 1 -- 172.21.15.16:0/912676212 learned_addr learned my addr 172.21.15.16:0/912676212 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.509+0000 7f6a3ccd8700 1 --2- 172.21.15.16:0/912676212 >> v2:172.21.15.76:3300/0 conn(0x7f6a3808d2c0 0x7f6a3808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.510+0000 7f6a3dcda700 1 -- 172.21.15.16:0/912676212 >> v2:172.21.15.76:3300/0 conn(0x7f6a3808d2c0 msgr2=0x7f6a3808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.510+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/912676212 >> v2:172.21.15.76:3300/0 conn(0x7f6a3808d2c0 0x7f6a3808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.510+0000 7f6a3dcda700 1 -- 172.21.15.16:0/912676212 >> v1:172.21.15.76:6789/0 conn(0x7f6a3808c6d0 legacy=0x7f6a3808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.510+0000 7f6a37fff700 1 -- 172.21.15.16:0/912676212 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812618273 0 0) 0x7f6a30001400 con 0x7f6a3808c6d0 2019-12-05T06:52:52.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.510+0000 7f6a3dcda700 1 -- 172.21.15.16:0/912676212 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3808e620 con 0x7f6a3808fef0 2019-12-05T06:52:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.510+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/912676212 >> v2:172.21.15.16:3300/0 conn(0x7f6a3808fef0 0x7f6a3808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6a2800f730 tx=0x7f6a28007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.510+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/912676212 >> v2:172.21.15.16:3300/0 conn(0x7f6a3808fef0 0x7f6a3808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6a2800f730 tx=0x7f6a28007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.510+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/912676212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3808fef0 0x7f6a3808f460 secure :-1 s=READY pgs=1446 cs=0 l=1 rx=0x7f6a2800f730 tx=0x7f6a28007d50).ready entity=mon.0 client_cookie=d95c67781dac01c8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.510+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/912676212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3808fef0 0x7f6a3808f460 secure :-1 s=THROTTLE_DONE pgs=1446 cs=0 l=1 rx=0x7f6a2800f730 tx=0x7f6a28007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.510+0000 7f6a37fff700 1 -- 172.21.15.16:0/912676212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a28000a70 con 0x7f6a3808fef0 2019-12-05T06:52:52.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.511+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/912676212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3808fef0 0x7f6a3808f460 secure :-1 s=THROTTLE_DONE pgs=1446 cs=0 l=1 rx=0x7f6a2800f730 tx=0x7f6a28007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.511+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/912676212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3808fef0 0x7f6a3808f460 secure :-1 s=THROTTLE_DONE pgs=1446 cs=0 l=1 rx=0x7f6a2800f730 tx=0x7f6a28007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.511+0000 7f6a37fff700 1 -- 172.21.15.16:0/912676212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a280010e0 con 0x7f6a3808fef0 2019-12-05T06:52:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.511+0000 7f6a37fff700 1 -- 172.21.15.16:0/912676212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a280012d0 con 0x7f6a3808fef0 2019-12-05T06:52:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.511+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/912676212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3808fef0 msgr2=0x7f6a3808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.511+0000 7f6a3ef3c700 1 --2- 172.21.15.16:0/912676212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3808fef0 0x7f6a3808f460 secure :-1 s=READY pgs=1446 cs=0 l=1 rx=0x7f6a2800f730 tx=0x7f6a28007d50).stop 2019-12-05T06:52:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.513+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/912676212 shutdown_connections 2019-12-05T06:52:52.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.513+0000 7f6a3ef3c700 1 --2- 172.21.15.16:0/912676212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6a3808fef0 0x7f6a3808f460 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.513+0000 7f6a3ef3c700 1 --2- 172.21.15.16:0/912676212 >> v2:172.21.15.76:3300/0 conn(0x7f6a3808d2c0 0x7f6a3808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.513+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/912676212 shutdown_connections 2019-12-05T06:52:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.513+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/912676212 wait complete. 2019-12-05T06:52:52.522 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.513+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/912676212 >> 172.21.15.16:0/912676212 conn(0x7f6a38061940 msgr2=0x7f6a38096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:52.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.518+0000 7f6a3ef3c700 1 Processor -- start 2019-12-05T06:52:52.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.519+0000 7f6a3ef3c700 1 -- start start 2019-12-05T06:52:52.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.521+0000 7f6a3ef3c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6a38084f30 0x7f6a38157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:52.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.521+0000 7f6a3ef3c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a38084b90 con 0x7f6a38096e90 2019-12-05T06:52:52.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.521+0000 7f6a3ef3c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a38098790 con 0x7f6a3809bbe0 2019-12-05T06:52:52.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.521+0000 7f6a3ef3c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a3809a450 con 0x7f6a38084f30 2019-12-05T06:52:52.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.521+0000 7f6a3d4d9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6a38096e90 0x7f6a3809b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48080/0 (socket says 172.21.15.16:48080) 2019-12-05T06:52:52.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.521+0000 7f6a3d4d9700 1 -- 172.21.15.16:0/4211615908 learned_addr learned my addr 172.21.15.16:0/4211615908 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:52.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.521+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> v2:172.21.15.76:3300/0 conn(0x7f6a38084f30 0x7f6a38157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:52.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.521+0000 7f6a35ffb700 1 -- 172.21.15.16:0/4211615908 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2720121797 0 0) 0x7f6a30001640 con 0x7f6a38096e90 2019-12-05T06:52:52.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.521+0000 7f6a35ffb700 1 -- 172.21.15.16:0/4211615908 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a18002e80 con 0x7f6a38096e90 2019-12-05T06:52:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.522+0000 7f6a3dcda700 1 -- 172.21.15.16:0/4211615908 >> v1:172.21.15.76:6789/0 conn(0x7f6a3809bbe0 legacy=0x7f6a381539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.522+0000 7f6a3dcda700 1 -- 172.21.15.16:0/4211615908 >> v1:172.21.15.16:6789/0 conn(0x7f6a38096e90 legacy=0x7f6a3809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.522+0000 7f6a35ffb700 1 -- 172.21.15.16:0/4211615908 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3181288217 0 0) 0x7f6a30000ec0 con 0x7f6a38096e90 2019-12-05T06:52:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.522+0000 7f6a3dcda700 1 -- 172.21.15.16:0/4211615908 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a3809aae0 con 0x7f6a38084f30 2019-12-05T06:52:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.522+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> v2:172.21.15.76:3300/0 conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.522+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> v2:172.21.15.76:3300/0 conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.522+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=READY pgs=980 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.523+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=THROTTLE_DONE pgs=980 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.523+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=THROTTLE_DONE pgs=980 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.523+0000 7f6a35ffb700 1 -- 172.21.15.16:0/4211615908 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a28007bb0 con 0x7f6a38084f30 2019-12-05T06:52:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.523+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=THROTTLE_DONE pgs=980 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.523+0000 7f6a35ffb700 1 -- 172.21.15.16:0/4211615908 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a280010c0 con 0x7f6a38084f30 2019-12-05T06:52:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.523+0000 7f6a35ffb700 1 -- 172.21.15.16:0/4211615908 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a28001880 con 0x7f6a38084f30 2019-12-05T06:52:52.529 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.523+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/4211615908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3809aae0 con 0x7f6a38084f30 2019-12-05T06:52:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.523+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/4211615908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3809aae0 con 0x7f6a38084f30 2019-12-05T06:52:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.524+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=THROTTLE_DONE pgs=980 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.525+0000 7f6a35ffb700 1 -- 172.21.15.16:0/4211615908 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6a280012b0 con 0x7f6a38084f30 2019-12-05T06:52:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.525+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=THROTTLE_DONE pgs=980 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.526+0000 7f6a35ffb700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a18023680 0x7f6a18025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.526+0000 7f6a35ffb700 1 -- 172.21.15.16:0/4211615908 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f6a28089e80 con 0x7f6a38084f30 2019-12-05T06:52:52.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.526+0000 7f6a3d4d9700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a18023680 0x7f6a18025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:52.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.527+0000 7f6a3d4d9700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a18023680 0x7f6a18025af0 crc :-1 s=READY pgs=608 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:52.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.538+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/4211615908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a3809aae0 con 0x7f6a38084f30 2019-12-05T06:52:52.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.548+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=THROTTLE_DONE pgs=980 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.548+0000 7f6a35ffb700 1 -- 172.21.15.16:0/4211615908 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6a280077e0 con 0x7f6a38084f30 2019-12-05T06:52:52.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.805+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/4211615908 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6a3809aae0 con 0x7f6a38084f30 2019-12-05T06:52:52.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.807+0000 7f6a3dcda700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=THROTTLE_DONE pgs=980 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:52.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.807+0000 7f6a35ffb700 1 -- 172.21.15.16:0/4211615908 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f6a28000eb0 con 0x7f6a38084f30 2019-12-05T06:52:52.809 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:52.809 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7857,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:52.809 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 177 2019-12-05T06:52:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.811+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a18023680 msgr2=0x7f6a18025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.811+0000 7f6a3ef3c700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a18023680 0x7f6a18025af0 crc :-1 s=READY pgs=608 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.811+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 msgr2=0x7f6a38157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.811+0000 7f6a3ef3c700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 0x7f6a38157200 secure :-1 s=READY pgs=980 cs=0 l=1 rx=0x7f6a2800fdf0 tx=0x7f6a28054740).stop 2019-12-05T06:52:52.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.811+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/4211615908 shutdown_connections 2019-12-05T06:52:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.811+0000 7f6a3ef3c700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6a18023680 0x7f6a18025af0 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.811+0000 7f6a3ef3c700 1 --2- 172.21.15.16:0/4211615908 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6a38084f30 0x7f6a38157200 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.811+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/4211615908 shutdown_connections 2019-12-05T06:52:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.811+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/4211615908 wait complete. 2019-12-05T06:52:52.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:52.811+0000 7f6a3ef3c700 1 -- 172.21.15.16:0/4211615908 >> 172.21.15.16:0/4211615908 conn(0x7f6a380714f0 msgr2=0x7f6a3809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:52.822 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:52:54.824 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.076+0000 7f7f9d1da700 1 Processor -- start 2019-12-05T06:52:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.076+0000 7f7f9d1da700 1 -- start start 2019-12-05T06:52:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.078+0000 7f7f9d1da700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7f9808c6d0 0x7f7f9808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.078+0000 7f7f9d1da700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f9808f720 0x7f7f9808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:55.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.078+0000 7f7f9d1da700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9808fb80 con 0x7f7f9808fef0 2019-12-05T06:52:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.078+0000 7f7f9d1da700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f9808df90 con 0x7f7f9808f720 2019-12-05T06:52:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.078+0000 7f7f9d1da700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f980a3f70 con 0x7f7f9808c6d0 2019-12-05T06:52:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.079+0000 7f7f9c9d9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f9808f720 0x7f7f9808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.079+0000 7f7f9c9d9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f9808f720 0x7f7f9808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34694/0 (socket says 172.21.15.16:34694) 2019-12-05T06:52:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.079+0000 7f7f9c9d9700 1 -- 172.21.15.16:0/1439999143 learned_addr learned my addr 172.21.15.16:0/1439999143 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:55.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.079+0000 7f7f9759e700 1 --2- 172.21.15.16:0/1439999143 >> v2:172.21.15.76:3300/0 conn(0x7f7f9808c6d0 0x7f7f9808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:55.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.079+0000 7f7f9c9d9700 1 -- 172.21.15.16:0/1439999143 >> v2:172.21.15.76:3300/0 conn(0x7f7f9808c6d0 msgr2=0x7f7f9808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:55.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.079+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1439999143 >> v2:172.21.15.76:3300/0 conn(0x7f7f9808c6d0 0x7f7f9808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:55.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.079+0000 7f7f9c9d9700 1 -- 172.21.15.16:0/1439999143 >> v1:172.21.15.76:6789/0 conn(0x7f7f9808fef0 legacy=0x7f7f9808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:55.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.079+0000 7f7f9c9d9700 1 -- 172.21.15.16:0/1439999143 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9808d540 con 0x7f7f9808f720 2019-12-05T06:52:55.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.079+0000 7f7f9759e700 1 --2- 172.21.15.16:0/1439999143 >> v2:172.21.15.76:3300/0 conn(0x7f7f9808c6d0 0x7f7f9808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:52:55.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1439999143 >> v2:172.21.15.16:3300/0 conn(0x7f7f9808f720 0x7f7f9808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7f8800f730 tx=0x7f7f8800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1439999143 >> v2:172.21.15.16:3300/0 conn(0x7f7f9808f720 0x7f7f9808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7f8800f730 tx=0x7f7f8800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1439999143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f9808f720 0x7f7f9808d000 secure :-1 s=READY pgs=1448 cs=0 l=1 rx=0x7f7f8800f730 tx=0x7f7f8800fad0).ready entity=mon.0 client_cookie=7c613084afeddd46 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1439999143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f9808f720 0x7f7f9808d000 secure :-1 s=THROTTLE_DONE pgs=1448 cs=0 l=1 rx=0x7f7f8800f730 tx=0x7f7f8800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1439999143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f9808f720 0x7f7f9808d000 secure :-1 s=THROTTLE_DONE pgs=1448 cs=0 l=1 rx=0x7f7f8800f730 tx=0x7f7f8800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9659c700 1 -- 172.21.15.16:0/1439999143 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f88000b90 con 0x7f7f9808f720 2019-12-05T06:52:55.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1439999143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f9808f720 0x7f7f9808d000 secure :-1 s=THROTTLE_DONE pgs=1448 cs=0 l=1 rx=0x7f7f8800f730 tx=0x7f7f8800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9659c700 1 -- 172.21.15.16:0/1439999143 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f880010e0 con 0x7f7f9808f720 2019-12-05T06:52:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9659c700 1 -- 172.21.15.16:0/1439999143 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f88001830 con 0x7f7f9808f720 2019-12-05T06:52:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1439999143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f9808f720 msgr2=0x7f7f9808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.080+0000 7f7f9d1da700 1 --2- 172.21.15.16:0/1439999143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f9808f720 0x7f7f9808d000 secure :-1 s=READY pgs=1448 cs=0 l=1 rx=0x7f7f8800f730 tx=0x7f7f8800fad0).stop 2019-12-05T06:52:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.082+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1439999143 shutdown_connections 2019-12-05T06:52:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.082+0000 7f7f9d1da700 1 --2- 172.21.15.16:0/1439999143 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f9808f720 0x7f7f9808d000 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:55.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.082+0000 7f7f9d1da700 1 --2- 172.21.15.16:0/1439999143 >> v2:172.21.15.76:3300/0 conn(0x7f7f9808c6d0 0x7f7f9808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.082+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1439999143 shutdown_connections 2019-12-05T06:52:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.082+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1439999143 wait complete. 2019-12-05T06:52:55.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.082+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1439999143 >> 172.21.15.16:0/1439999143 conn(0x7f7f98061940 msgr2=0x7f7f98096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.087+0000 7f7f9d1da700 1 Processor -- start 2019-12-05T06:52:55.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.088+0000 7f7f9d1da700 1 -- start start 2019-12-05T06:52:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.089+0000 7f7f9d1da700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.089+0000 7f7f9d1da700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f98153160 con 0x7f7f9806e210 2019-12-05T06:52:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.089+0000 7f7f9d1da700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f9806e880 con 0x7f7f9814c3a0 2019-12-05T06:52:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.089+0000 7f7f9d1da700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f9809c620 con 0x7f7f9809e580 2019-12-05T06:52:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.089+0000 7f7f96d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7f9806e210 0x7f7f9819b400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48100/0 (socket says 172.21.15.16:48100) 2019-12-05T06:52:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.089+0000 7f7f9759e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f7f9814c3a0 0x7f7f9809d5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:41168/0 (socket says 172.21.15.16:41168) 2019-12-05T06:52:55.095 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.089+0000 7f7f96d9d700 1 -- 172.21.15.16:0/1875729886 learned_addr learned my addr 172.21.15.16:0/1875729886 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.090+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.090+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998734831 0 0) 0x7f7f9000df00 con 0x7f7f9814c3a0 2019-12-05T06:52:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.090+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f7c002e80 con 0x7f7f9814c3a0 2019-12-05T06:52:55.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.090+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3363018693 0 0) 0x7f7f800010c0 con 0x7f7f9806e210 2019-12-05T06:52:55.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.090+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f7c002f30 con 0x7f7f9806e210 2019-12-05T06:52:55.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.090+0000 7f7f9c9d9700 1 -- 172.21.15.16:0/1875729886 >> v1:172.21.15.76:6789/0 conn(0x7f7f9814c3a0 legacy=0x7f7f9809d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:55.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.090+0000 7f7f9c9d9700 1 -- 172.21.15.16:0/1875729886 >> v1:172.21.15.16:6789/0 conn(0x7f7f9806e210 legacy=0x7f7f9819b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:55.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.090+0000 7f7f9c9d9700 1 -- 172.21.15.16:0/1875729886 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f9809ccb0 con 0x7f7f9809e580 2019-12-05T06:52:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.090+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.091+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.091+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=READY pgs=342 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.091+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.091+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f88001290 con 0x7f7f9809e580 2019-12-05T06:52:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.091+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.091+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.091+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f88000eb0 con 0x7f7f9809e580 2019-12-05T06:52:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.091+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1875729886 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f9809ccb0 con 0x7f7f9809e580 2019-12-05T06:52:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.091+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f88001870 con 0x7f7f9809e580 2019-12-05T06:52:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.091+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1875729886 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f9809ccb0 con 0x7f7f9809e580 2019-12-05T06:52:55.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.092+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.093+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7f88000eb0 con 0x7f7f9809e580 2019-12-05T06:52:55.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.093+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.093+0000 7f7f8ffff700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f7c023670 0x7f7f7c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:55.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.094+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f7f88045d90 con 0x7f7f9809e580 2019-12-05T06:52:55.112 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.094+0000 7f7f9759e700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f7c023670 0x7f7f7c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:55.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.094+0000 7f7f9759e700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f7c023670 0x7f7f7c025ae0 crc :-1 s=READY pgs=609 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:55.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.108+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1875729886 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f9809ccb0 con 0x7f7f9809e580 2019-12-05T06:52:55.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.117+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.125 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.118+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7f88007df0 con 0x7f7f9809e580 2019-12-05T06:52:55.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.363+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1875729886 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f9809ccb0 con 0x7f7f9809e580 2019-12-05T06:52:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.364+0000 7f7f9c9d9700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=THROTTLE_DONE pgs=342 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:55.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.364+0000 7f7f8ffff700 1 -- 172.21.15.16:0/1875729886 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f7f88000eb0 con 0x7f7f9809e580 2019-12-05T06:52:55.366 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 177 2019-12-05T06:52:55.367 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:55.367 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7857,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:55.369 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.368+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f7c023670 msgr2=0x7f7f7c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:55.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.368+0000 7f7f9d1da700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f7c023670 0x7f7f7c025ae0 crc :-1 s=READY pgs=609 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:55.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.368+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 msgr2=0x7f7f9809ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:55.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.368+0000 7f7f9d1da700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 secure :-1 s=READY pgs=342 cs=0 l=1 rx=0x7f7f880103a0 tx=0x7f7f88010740).stop 2019-12-05T06:52:55.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.368+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1875729886 shutdown_connections 2019-12-05T06:52:55.370 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.368+0000 7f7f9d1da700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f7c023670 0x7f7f7c025ae0 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:55.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.368+0000 7f7f9d1da700 1 --2- 172.21.15.16:0/1875729886 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f9809e580 0x7f7f9809ddb0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:55.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.369+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1875729886 shutdown_connections 2019-12-05T06:52:55.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.369+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1875729886 wait complete. 2019-12-05T06:52:55.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:55.369+0000 7f7f9d1da700 1 -- 172.21.15.16:0/1875729886 >> 172.21.15.16:0/1875729886 conn(0x7f7f98098480 msgr2=0x7f7f98071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:55.379 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:52:57.380 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:52:57.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.659+0000 7f7d52461700 1 Processor -- start 2019-12-05T06:52:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.659+0000 7f7d52461700 1 -- start start 2019-12-05T06:52:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.660+0000 7f7d52461700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7d4c08c6d0 0x7f7d4c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.660+0000 7f7d52461700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d4c08fef0 0x7f7d4c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:52:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.661+0000 7f7d52461700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c08fb80 con 0x7f7d4c08ef20 2019-12-05T06:52:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.661+0000 7f7d52461700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c08df90 con 0x7f7d4c08fef0 2019-12-05T06:52:57.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.661+0000 7f7d52461700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c0a3f70 con 0x7f7d4c08c6d0 2019-12-05T06:52:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.661+0000 7f7d4bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d4c08fef0 0x7f7d4c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.661+0000 7f7d511ff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7d4c08ef20 0x7f7d4c08f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48112/0 (socket says 172.21.15.16:48112) 2019-12-05T06:52:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.661+0000 7f7d511ff700 1 -- 172.21.15.16:0/2904283714 learned_addr learned my addr 172.21.15.16:0/2904283714 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:52:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.661+0000 7f7d509fe700 1 --2- 172.21.15.16:0/2904283714 >> v2:172.21.15.76:3300/0 conn(0x7f7d4c08c6d0 0x7f7d4c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.661+0000 7f7d4bfff700 1 -- 172.21.15.16:0/2904283714 >> v2:172.21.15.76:3300/0 conn(0x7f7d4c08c6d0 msgr2=0x7f7d4c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:52:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.661+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2904283714 >> v2:172.21.15.76:3300/0 conn(0x7f7d4c08c6d0 0x7f7d4c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:57.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4bfff700 1 -- 172.21.15.16:0/2904283714 >> v1:172.21.15.16:6789/0 conn(0x7f7d4c08ef20 legacy=0x7f7d4c08f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:57.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4b7fe700 1 -- 172.21.15.16:0/2904283714 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635647794 0 0) 0x7f7d440014d0 con 0x7f7d4c08ef20 2019-12-05T06:52:57.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4bfff700 1 -- 172.21.15.16:0/2904283714 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c08d540 con 0x7f7d4c08fef0 2019-12-05T06:52:57.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2904283714 >> v2:172.21.15.16:3300/0 conn(0x7f7d4c08fef0 0x7f7d4c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7d3800af50 tx=0x7f7d3800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2904283714 >> v2:172.21.15.16:3300/0 conn(0x7f7d4c08fef0 0x7f7d4c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7d3800af50 tx=0x7f7d3800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2904283714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c08fef0 0x7f7d4c08d000 secure :-1 s=READY pgs=1451 cs=0 l=1 rx=0x7f7d3800af50 tx=0x7f7d3800b6e0).ready entity=mon.0 client_cookie=50d06e1f049b0b74 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:57.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2904283714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c08fef0 0x7f7d4c08d000 secure :-1 s=THROTTLE_DONE pgs=1451 cs=0 l=1 rx=0x7f7d3800af50 tx=0x7f7d3800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4b7fe700 1 -- 172.21.15.16:0/2904283714 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d38007210 con 0x7f7d4c08fef0 2019-12-05T06:52:57.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2904283714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c08fef0 0x7f7d4c08d000 secure :-1 s=THROTTLE_DONE pgs=1451 cs=0 l=1 rx=0x7f7d3800af50 tx=0x7f7d3800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2904283714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c08fef0 0x7f7d4c08d000 secure :-1 s=THROTTLE_DONE pgs=1451 cs=0 l=1 rx=0x7f7d3800af50 tx=0x7f7d3800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4b7fe700 1 -- 172.21.15.16:0/2904283714 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d380099b0 con 0x7f7d4c08fef0 2019-12-05T06:52:57.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.662+0000 7f7d4b7fe700 1 -- 172.21.15.16:0/2904283714 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d3800a080 con 0x7f7d4c08fef0 2019-12-05T06:52:57.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.663+0000 7f7d52461700 1 -- 172.21.15.16:0/2904283714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c08fef0 msgr2=0x7f7d4c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:57.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.663+0000 7f7d52461700 1 --2- 172.21.15.16:0/2904283714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c08fef0 0x7f7d4c08d000 secure :-1 s=READY pgs=1451 cs=0 l=1 rx=0x7f7d3800af50 tx=0x7f7d3800b6e0).stop 2019-12-05T06:52:57.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.664+0000 7f7d52461700 1 -- 172.21.15.16:0/2904283714 shutdown_connections 2019-12-05T06:52:57.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.664+0000 7f7d52461700 1 --2- 172.21.15.16:0/2904283714 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c08fef0 0x7f7d4c08d000 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:57.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.664+0000 7f7d52461700 1 --2- 172.21.15.16:0/2904283714 >> v2:172.21.15.76:3300/0 conn(0x7f7d4c08c6d0 0x7f7d4c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:52:57.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.664+0000 7f7d52461700 1 -- 172.21.15.16:0/2904283714 shutdown_connections 2019-12-05T06:52:57.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.664+0000 7f7d52461700 1 -- 172.21.15.16:0/2904283714 wait complete. 2019-12-05T06:52:57.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.664+0000 7f7d52461700 1 -- 172.21.15.16:0/2904283714 >> 172.21.15.16:0/2904283714 conn(0x7f7d4c061940 msgr2=0x7f7d4c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:57.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.669+0000 7f7d52461700 1 Processor -- start 2019-12-05T06:52:57.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.669+0000 7f7d52461700 1 -- start start 2019-12-05T06:52:57.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.670+0000 7f7d52461700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7d4c06df10 0x7f7d4c098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:57.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.670+0000 7f7d52461700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d4c0992d0 0x7f7d4c098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:57.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.670+0000 7f7d52461700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d4c06e300 con 0x7f7d4c098d80 2019-12-05T06:52:57.676 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.670+0000 7f7d52461700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c097370 con 0x7f7d4c0992d0 2019-12-05T06:52:57.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.670+0000 7f7d52461700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d4c08ad00 con 0x7f7d4c06df10 2019-12-05T06:52:57.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.671+0000 7f7d4bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d4c0992d0 0x7f7d4c098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:57.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.671+0000 7f7d4bfff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7d4c0992d0 0x7f7d4c098840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34716/0 (socket says 172.21.15.16:34716) 2019-12-05T06:52:57.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.671+0000 7f7d4bfff700 1 -- 172.21.15.16:0/2423817344 learned_addr learned my addr 172.21.15.16:0/2423817344 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:52:57.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.671+0000 7f7d509fe700 1 --2- 172.21.15.16:0/2423817344 >> v2:172.21.15.76:3300/0 conn(0x7f7d4c06df10 0x7f7d4c098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:57.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.671+0000 7f7d4bfff700 1 -- 172.21.15.16:0/2423817344 >> v2:172.21.15.76:3300/0 conn(0x7f7d4c06df10 msgr2=0x7f7d4c098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:57.677 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.671+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> v2:172.21.15.76:3300/0 conn(0x7f7d4c06df10 0x7f7d4c098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.671+0000 7f7d4bfff700 1 -- 172.21.15.16:0/2423817344 >> v1:172.21.15.76:6789/0 conn(0x7f7d4c098d80 legacy=0x7f7d4c088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.671+0000 7f7d4bfff700 1 -- 172.21.15.16:0/2423817344 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d4c097a00 con 0x7f7d4c0992d0 2019-12-05T06:52:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.671+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> v2:172.21.15.16:3300/0 conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.672+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> v2:172.21.15.16:3300/0 conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.672+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=READY pgs=1452 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.672+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.672+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.672+0000 7f7d497fa700 1 -- 172.21.15.16:0/2423817344 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d38000940 con 0x7f7d4c0992d0 2019-12-05T06:52:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.672+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.672+0000 7f7d497fa700 1 -- 172.21.15.16:0/2423817344 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d38009980 con 0x7f7d4c0992d0 2019-12-05T06:52:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.672+0000 7f7d52461700 1 -- 172.21.15.16:0/2423817344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d4c097a00 con 0x7f7d4c0992d0 2019-12-05T06:52:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.672+0000 7f7d497fa700 1 -- 172.21.15.16:0/2423817344 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d3800a090 con 0x7f7d4c0992d0 2019-12-05T06:52:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.672+0000 7f7d52461700 1 -- 172.21.15.16:0/2423817344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d4c097a00 con 0x7f7d4c0992d0 2019-12-05T06:52:57.679 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.673+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.674+0000 7f7d497fa700 1 -- 172.21.15.16:0/2423817344 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7d38009b70 con 0x7f7d4c0992d0 2019-12-05T06:52:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.674+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.674+0000 7f7d497fa700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d28023630 0x7f7d28025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:52:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.674+0000 7f7d497fa700 1 -- 172.21.15.16:0/2423817344 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f7d38045de0 con 0x7f7d4c0992d0 2019-12-05T06:52:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.674+0000 7f7d509fe700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d28023630 0x7f7d28025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:52:57.680 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.675+0000 7f7d509fe700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d28023630 0x7f7d28025ac0 crc :-1 s=READY pgs=610 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:52:57.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.689+0000 7f7d52461700 1 -- 172.21.15.16:0/2423817344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d4c097a00 con 0x7f7d4c0992d0 2019-12-05T06:52:57.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.698+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.698+0000 7f7d497fa700 1 -- 172.21.15.16:0/2423817344 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7d3800f310 con 0x7f7d4c0992d0 2019-12-05T06:52:57.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.940+0000 7f7d52461700 1 -- 172.21.15.16:0/2423817344 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d4c097a00 con 0x7f7d4c0992d0 2019-12-05T06:52:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.941+0000 7f7d4bfff700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=THROTTLE_DONE pgs=1452 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:52:57.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.941+0000 7f7d497fa700 1 -- 172.21.15.16:0/2423817344 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f7d38009580 con 0x7f7d4c0992d0 2019-12-05T06:52:57.944 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 177 2019-12-05T06:52:57.944 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:52:57.944 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7857,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:52:57.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 -- 172.21.15.16:0/2423817344 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d28023630 msgr2=0x7f7d28025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:57.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d28023630 0x7f7d28025ac0 crc :-1 s=READY pgs=610 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:57.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 -- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 msgr2=0x7f7d4c098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:52:57.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 0x7f7d4c098840 secure :-1 s=READY pgs=1452 cs=0 l=1 rx=0x7f7d380103d0 tx=0x7f7d38010770).stop 2019-12-05T06:52:57.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 -- 172.21.15.16:0/2423817344 shutdown_connections 2019-12-05T06:52:57.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d28023630 0x7f7d28025ac0 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:57.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 --2- 172.21.15.16:0/2423817344 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d4c0992d0 0x7f7d4c098840 unknown :-1 s=CLOSED pgs=1452 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:57.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 --2- 172.21.15.16:0/2423817344 >> v2:172.21.15.76:3300/0 conn(0x7f7d4c06df10 0x7f7d4c098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:52:57.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 -- 172.21.15.16:0/2423817344 shutdown_connections 2019-12-05T06:52:57.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 -- 172.21.15.16:0/2423817344 wait complete. 2019-12-05T06:52:57.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:52:57.946+0000 7f7d52461700 1 -- 172.21.15.16:0/2423817344 >> 172.21.15.16:0/2423817344 conn(0x7f7d4c0714f0 msgr2=0x7f7d4c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:52:57.957 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:52:59.959 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:00.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.244+0000 7f7dc0712700 1 Processor -- start 2019-12-05T06:53:00.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.244+0000 7f7dc0712700 1 -- start start 2019-12-05T06:53:00.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.246+0000 7f7dc0712700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7db808d2c0 0x7f7db808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:00.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.246+0000 7f7dc0712700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7db808fef0 0x7f7db808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.246+0000 7f7dc0712700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db80958e0 con 0x7f7db808c6d0 2019-12-05T06:53:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.246+0000 7f7dc0712700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db808df90 con 0x7f7db808d2c0 2019-12-05T06:53:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.246+0000 7f7dc0712700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db80a3f70 con 0x7f7db808fef0 2019-12-05T06:53:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbecaf700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7db808c6d0 0x7f7db808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48128/0 (socket says 172.21.15.16:48128) 2019-12-05T06:53:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbecaf700 1 -- 172.21.15.16:0/3583694632 learned_addr learned my addr 172.21.15.16:0/3583694632 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:00.255 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbe4ae700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7db808d2c0 0x7f7db808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3583694632 >> v2:172.21.15.76:3300/0 conn(0x7f7db808fef0 0x7f7db808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbdcad700 1 -- 172.21.15.16:0/3583694632 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2860761215 0 0) 0x7f7db00014f0 con 0x7f7db808c6d0 2019-12-05T06:53:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbdcad700 1 -- 172.21.15.16:0/3583694632 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7da0002e80 con 0x7f7db808c6d0 2019-12-05T06:53:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbe4ae700 1 -- 172.21.15.16:0/3583694632 >> v2:172.21.15.76:3300/0 conn(0x7f7db808fef0 msgr2=0x7f7db808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbe4ae700 1 --2- 172.21.15.16:0/3583694632 >> v2:172.21.15.76:3300/0 conn(0x7f7db808fef0 0x7f7db808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:00.256 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbe4ae700 1 -- 172.21.15.16:0/3583694632 >> v1:172.21.15.16:6789/0 conn(0x7f7db808c6d0 legacy=0x7f7db808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:00.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbe4ae700 1 -- 172.21.15.16:0/3583694632 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db808e620 con 0x7f7db808d2c0 2019-12-05T06:53:00.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.247+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3583694632 >> v2:172.21.15.76:3300/0 conn(0x7f7db808fef0 0x7f7db808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:53:00.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dbe4ae700 1 --2- 172.21.15.16:0/3583694632 >> v2:172.21.15.16:3300/0 conn(0x7f7db808d2c0 0x7f7db808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7dac00af50 tx=0x7f7dac00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dbe4ae700 1 --2- 172.21.15.16:0/3583694632 >> v2:172.21.15.16:3300/0 conn(0x7f7db808d2c0 0x7f7db808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7dac00af50 tx=0x7f7dac00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dbe4ae700 1 --2- 172.21.15.16:0/3583694632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db808d2c0 0x7f7db808ef20 secure :-1 s=READY pgs=1455 cs=0 l=1 rx=0x7f7dac00af50 tx=0x7f7dac00b760).ready entity=mon.0 client_cookie=5bfe808b5359efc9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:00.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dbe4ae700 1 --2- 172.21.15.16:0/3583694632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db808d2c0 0x7f7db808ef20 secure :-1 s=THROTTLE_DONE pgs=1455 cs=0 l=1 rx=0x7f7dac00af50 tx=0x7f7dac00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.257 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dbe4ae700 1 --2- 172.21.15.16:0/3583694632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db808d2c0 0x7f7db808ef20 secure :-1 s=THROTTLE_DONE pgs=1455 cs=0 l=1 rx=0x7f7dac00af50 tx=0x7f7dac00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dbdcad700 1 -- 172.21.15.16:0/3583694632 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dac007210 con 0x7f7db808d2c0 2019-12-05T06:53:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dbe4ae700 1 --2- 172.21.15.16:0/3583694632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db808d2c0 0x7f7db808ef20 secure :-1 s=THROTTLE_DONE pgs=1455 cs=0 l=1 rx=0x7f7dac00af50 tx=0x7f7dac00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dbdcad700 1 -- 172.21.15.16:0/3583694632 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dac0099b0 con 0x7f7db808d2c0 2019-12-05T06:53:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dbdcad700 1 -- 172.21.15.16:0/3583694632 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dac00a080 con 0x7f7db808d2c0 2019-12-05T06:53:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dc0712700 1 -- 172.21.15.16:0/3583694632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db808d2c0 msgr2=0x7f7db808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:00.258 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.248+0000 7f7dc0712700 1 --2- 172.21.15.16:0/3583694632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db808d2c0 0x7f7db808ef20 secure :-1 s=READY pgs=1455 cs=0 l=1 rx=0x7f7dac00af50 tx=0x7f7dac00b760).stop 2019-12-05T06:53:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.249+0000 7f7dc0712700 1 -- 172.21.15.16:0/3583694632 shutdown_connections 2019-12-05T06:53:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.249+0000 7f7dc0712700 1 --2- 172.21.15.16:0/3583694632 >> v2:172.21.15.76:3300/0 conn(0x7f7db808fef0 0x7f7db808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.250+0000 7f7dc0712700 1 --2- 172.21.15.16:0/3583694632 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db808d2c0 0x7f7db808ef20 unknown :-1 s=CLOSED pgs=1455 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.250+0000 7f7dc0712700 1 -- 172.21.15.16:0/3583694632 shutdown_connections 2019-12-05T06:53:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.250+0000 7f7dc0712700 1 -- 172.21.15.16:0/3583694632 wait complete. 2019-12-05T06:53:00.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.250+0000 7f7dc0712700 1 -- 172.21.15.16:0/3583694632 >> 172.21.15.16:0/3583694632 conn(0x7f7db8061940 msgr2=0x7f7db8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.255+0000 7f7dc0712700 1 Processor -- start 2019-12-05T06:53:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.255+0000 7f7dc0712700 1 -- start start 2019-12-05T06:53:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.256+0000 7f7dc0712700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7db8084f30 0x7f7db8157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.256+0000 7f7dc0712700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db8084b90 con 0x7f7db8096e90 2019-12-05T06:53:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.256+0000 7f7dc0712700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db8098790 con 0x7f7db809bbe0 2019-12-05T06:53:00.263 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.256+0000 7f7dc0712700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db809a450 con 0x7f7db8084f30 2019-12-05T06:53:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7dbecaf700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7db8096e90 0x7f7db809b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48134/0 (socket says 172.21.15.16:48134) 2019-12-05T06:53:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7dbecaf700 1 -- 172.21.15.16:0/3328085099 learned_addr learned my addr 172.21.15.16:0/3328085099 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> v2:172.21.15.16:3300/0 conn(0x7f7db8084f30 0x7f7db8157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7db77fe700 1 -- 172.21.15.16:0/3328085099 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3294497247 0 0) 0x7f7db00014d0 con 0x7f7db8096e90 2019-12-05T06:53:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7db77fe700 1 -- 172.21.15.16:0/3328085099 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d98002e80 con 0x7f7db8096e90 2019-12-05T06:53:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7dbf4b0700 1 -- 172.21.15.16:0/3328085099 >> v1:172.21.15.76:6789/0 conn(0x7f7db809bbe0 legacy=0x7f7db81539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:00.264 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7db77fe700 1 -- 172.21.15.16:0/3328085099 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 341142548 0 0) 0x7f7dac00adc0 con 0x7f7db809bbe0 2019-12-05T06:53:00.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7dbf4b0700 1 -- 172.21.15.16:0/3328085099 >> v1:172.21.15.16:6789/0 conn(0x7f7db8096e90 legacy=0x7f7db809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:00.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7dbf4b0700 1 -- 172.21.15.16:0/3328085099 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db809aae0 con 0x7f7db8084f30 2019-12-05T06:53:00.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> v2:172.21.15.16:3300/0 conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.257+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> v2:172.21.15.16:3300/0 conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.258+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=READY pgs=1457 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:00.265 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.258+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=THROTTLE_DONE pgs=1457 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.258+0000 7f7db77fe700 1 -- 172.21.15.16:0/3328085099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7da8009bd0 con 0x7f7db8084f30 2019-12-05T06:53:00.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.258+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=THROTTLE_DONE pgs=1457 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.258+0000 7f7dc0712700 1 -- 172.21.15.16:0/3328085099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db809aae0 con 0x7f7db8084f30 2019-12-05T06:53:00.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.258+0000 7f7db77fe700 1 -- 172.21.15.16:0/3328085099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7da8000a70 con 0x7f7db8084f30 2019-12-05T06:53:00.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.258+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=THROTTLE_DONE pgs=1457 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.258+0000 7f7db77fe700 1 -- 172.21.15.16:0/3328085099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7da8000c60 con 0x7f7db8084f30 2019-12-05T06:53:00.266 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.258+0000 7f7dc0712700 1 -- 172.21.15.16:0/3328085099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db809aae0 con 0x7f7db8084f30 2019-12-05T06:53:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.259+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=THROTTLE_DONE pgs=1457 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.260+0000 7f7db77fe700 1 -- 172.21.15.16:0/3328085099 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7da8000e70 con 0x7f7db8084f30 2019-12-05T06:53:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.260+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=THROTTLE_DONE pgs=1457 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.260+0000 7f7db77fe700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98023680 0x7f7d98025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.260+0000 7f7db77fe700 1 -- 172.21.15.16:0/3328085099 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f7da8045dd0 con 0x7f7db8084f30 2019-12-05T06:53:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.260+0000 7f7dbecaf700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98023680 0x7f7d98025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:00.267 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.261+0000 7f7dbecaf700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98023680 0x7f7d98025af0 crc :-1 s=READY pgs=611 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:00.275 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.275+0000 7f7dc0712700 1 -- 172.21.15.16:0/3328085099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db809aae0 con 0x7f7db8084f30 2019-12-05T06:53:00.284 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.284+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=THROTTLE_DONE pgs=1457 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.284+0000 7f7db77fe700 1 -- 172.21.15.16:0/3328085099 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7da8000c60 con 0x7f7db8084f30 2019-12-05T06:53:00.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.488+0000 7f7dc0712700 1 -- 172.21.15.16:0/3328085099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7db809aae0 con 0x7f7db8084f30 2019-12-05T06:53:00.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.489+0000 7f7dbf4b0700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=THROTTLE_DONE pgs=1457 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:00.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.489+0000 7f7db77fe700 1 -- 172.21.15.16:0/3328085099 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f7da8000e70 con 0x7f7db8084f30 2019-12-05T06:53:00.492 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:00.492 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7857,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:00.492 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 177 2019-12-05T06:53:00.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.494+0000 7f7dc0712700 1 -- 172.21.15.16:0/3328085099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98023680 msgr2=0x7f7d98025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:00.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.494+0000 7f7dc0712700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98023680 0x7f7d98025af0 crc :-1 s=READY pgs=611 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:00.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.494+0000 7f7dc0712700 1 -- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 msgr2=0x7f7db8157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:00.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.494+0000 7f7dc0712700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 0x7f7db8157200 secure :-1 s=READY pgs=1457 cs=0 l=1 rx=0x7f7da80103c0 tx=0x7f7da8010760).stop 2019-12-05T06:53:00.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.494+0000 7f7dc0712700 1 -- 172.21.15.16:0/3328085099 shutdown_connections 2019-12-05T06:53:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.494+0000 7f7dc0712700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98023680 0x7f7d98025af0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.494+0000 7f7dc0712700 1 --2- 172.21.15.16:0/3328085099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db8084f30 0x7f7db8157200 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.494+0000 7f7dc0712700 1 -- 172.21.15.16:0/3328085099 shutdown_connections 2019-12-05T06:53:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.494+0000 7f7dc0712700 1 -- 172.21.15.16:0/3328085099 wait complete. 2019-12-05T06:53:00.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:00.494+0000 7f7dc0712700 1 -- 172.21.15.16:0/3328085099 >> 172.21.15.16:0/3328085099 conn(0x7f7db80714f0 msgr2=0x7f7db809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:00.505 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:53:02.506 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.776+0000 7f7f824d5700 1 Processor -- start 2019-12-05T06:53:02.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.776+0000 7f7f824d5700 1 -- start start 2019-12-05T06:53:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.778+0000 7f7f824d5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 0x7f7f7c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.778+0000 7f7f824d5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f7c0958e0 con 0x7f7f7c08fef0 2019-12-05T06:53:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.778+0000 7f7f824d5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f7c0629e0 con 0x7f7f7c08c6d0 2019-12-05T06:53:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.778+0000 7f7f824d5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f7c08df90 con 0x7f7f7c08d2c0 2019-12-05T06:53:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.778+0000 7f7f81273700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7f7c08fef0 0x7f7f7c08f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48148/0 (socket says 172.21.15.16:48148) 2019-12-05T06:53:02.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.778+0000 7f7f81273700 1 -- 172.21.15.16:0/3928263005 learned_addr learned my addr 172.21.15.16:0/3928263005 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.778+0000 7f7f7bfff700 1 --2- 172.21.15.16:0/3928263005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 0x7f7f7c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.779+0000 7f7f7b7fe700 1 -- 172.21.15.16:0/3928263005 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2778153078 0 0) 0x7f7f64001150 con 0x7f7f7c08fef0 2019-12-05T06:53:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.779+0000 7f7f7b7fe700 1 -- 172.21.15.16:0/3928263005 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f68002e80 con 0x7f7f7c08fef0 2019-12-05T06:53:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.779+0000 7f7f7b7fe700 1 -- 172.21.15.16:0/3928263005 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3103467135 0 0) 0x7f7f74001400 con 0x7f7f7c08c6d0 2019-12-05T06:53:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.779+0000 7f7f7b7fe700 1 -- 172.21.15.16:0/3928263005 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f68004170 con 0x7f7f7c08c6d0 2019-12-05T06:53:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.779+0000 7f7f7bfff700 1 -- 172.21.15.16:0/3928263005 >> v1:172.21.15.76:6789/0 conn(0x7f7f7c08c6d0 legacy=0x7f7f7c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:02.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.779+0000 7f7f7bfff700 1 -- 172.21.15.16:0/3928263005 >> v1:172.21.15.16:6789/0 conn(0x7f7f7c08fef0 legacy=0x7f7f7c08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:02.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.779+0000 7f7f7bfff700 1 -- 172.21.15.16:0/3928263005 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c08e620 con 0x7f7f7c08d2c0 2019-12-05T06:53:02.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.779+0000 7f7f7bfff700 1 --2- 172.21.15.16:0/3928263005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 0x7f7f7c08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7f6c00af50 tx=0x7f7f6c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.779+0000 7f7f7bfff700 1 --2- 172.21.15.16:0/3928263005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 0x7f7f7c08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7f6c00af50 tx=0x7f7f6c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.779+0000 7f7f7bfff700 1 --2- 172.21.15.16:0/3928263005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 0x7f7f7c08ef20 secure :-1 s=READY pgs=343 cs=0 l=1 rx=0x7f7f6c00af50 tx=0x7f7f6c00b760).ready entity=mon.2 client_cookie=2078fa944b32cf63 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:02.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.780+0000 7f7f7bfff700 1 --2- 172.21.15.16:0/3928263005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 0x7f7f7c08ef20 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7f7f6c00af50 tx=0x7f7f6c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.780+0000 7f7f7b7fe700 1 -- 172.21.15.16:0/3928263005 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f6c000960 con 0x7f7f7c08d2c0 2019-12-05T06:53:02.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.780+0000 7f7f7bfff700 1 --2- 172.21.15.16:0/3928263005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 0x7f7f7c08ef20 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7f7f6c00af50 tx=0x7f7f6c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.780+0000 7f7f7bfff700 1 --2- 172.21.15.16:0/3928263005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 0x7f7f7c08ef20 secure :-1 s=THROTTLE_DONE pgs=343 cs=0 l=1 rx=0x7f7f6c00af50 tx=0x7f7f6c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.780+0000 7f7f7b7fe700 1 -- 172.21.15.16:0/3928263005 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f6c009860 con 0x7f7f7c08d2c0 2019-12-05T06:53:02.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.780+0000 7f7f7b7fe700 1 -- 172.21.15.16:0/3928263005 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f6c000960 con 0x7f7f7c08d2c0 2019-12-05T06:53:02.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.780+0000 7f7f824d5700 1 -- 172.21.15.16:0/3928263005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 msgr2=0x7f7f7c08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:02.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.780+0000 7f7f824d5700 1 --2- 172.21.15.16:0/3928263005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 0x7f7f7c08ef20 secure :-1 s=READY pgs=343 cs=0 l=1 rx=0x7f7f6c00af50 tx=0x7f7f6c00b760).stop 2019-12-05T06:53:02.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.781+0000 7f7f824d5700 1 -- 172.21.15.16:0/3928263005 shutdown_connections 2019-12-05T06:53:02.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.781+0000 7f7f824d5700 1 --2- 172.21.15.16:0/3928263005 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7f7c08d2c0 0x7f7f7c08ef20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:02.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.782+0000 7f7f824d5700 1 -- 172.21.15.16:0/3928263005 shutdown_connections 2019-12-05T06:53:02.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.782+0000 7f7f824d5700 1 -- 172.21.15.16:0/3928263005 wait complete. 2019-12-05T06:53:02.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.782+0000 7f7f824d5700 1 -- 172.21.15.16:0/3928263005 >> 172.21.15.16:0/3928263005 conn(0x7f7f7c061940 msgr2=0x7f7f7c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.786+0000 7f7f824d5700 1 Processor -- start 2019-12-05T06:53:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.786+0000 7f7f824d5700 1 -- start start 2019-12-05T06:53:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.788+0000 7f7f824d5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7f7c128f50 0x7f7f7c12b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.788+0000 7f7f824d5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f7c137ea0 0x7f7f7c12bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.788+0000 7f7f824d5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f7c099720 con 0x7f7f7c128360 2019-12-05T06:53:02.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.788+0000 7f7f824d5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f7c12a640 con 0x7f7f7c137ea0 2019-12-05T06:53:02.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.788+0000 7f7f824d5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f7c129c10 con 0x7f7f7c128f50 2019-12-05T06:53:02.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.788+0000 7f7f81273700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f7c137ea0 0x7f7f7c12bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:02.803 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.788+0000 7f7f81273700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7f7c137ea0 0x7f7f7c12bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34754/0 (socket says 172.21.15.16:34754) 2019-12-05T06:53:02.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.788+0000 7f7f81273700 1 -- 172.21.15.16:0/4272081215 learned_addr learned my addr 172.21.15.16:0/4272081215 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:02.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.788+0000 7f7f7bfff700 1 --2- 172.21.15.16:0/4272081215 >> v2:172.21.15.76:3300/0 conn(0x7f7f7c128f50 0x7f7f7c12b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:02.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f81273700 1 -- 172.21.15.16:0/4272081215 >> v2:172.21.15.76:3300/0 conn(0x7f7f7c128f50 msgr2=0x7f7f7c12b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:02.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> v2:172.21.15.76:3300/0 conn(0x7f7f7c128f50 0x7f7f7c12b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f81273700 1 -- 172.21.15.16:0/4272081215 >> v1:172.21.15.76:6789/0 conn(0x7f7f7c128360 legacy=0x7f7f7c128750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f81273700 1 -- 172.21.15.16:0/4272081215 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f7c12acd0 con 0x7f7f7c137ea0 2019-12-05T06:53:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> v2:172.21.15.16:3300/0 conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> v2:172.21.15.16:3300/0 conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=READY pgs=1459 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:02.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=THROTTLE_DONE pgs=1459 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=THROTTLE_DONE pgs=1459 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f797fa700 1 -- 172.21.15.16:0/4272081215 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f64000ef0 con 0x7f7f7c137ea0 2019-12-05T06:53:02.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=THROTTLE_DONE pgs=1459 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.789+0000 7f7f797fa700 1 -- 172.21.15.16:0/4272081215 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f64004240 con 0x7f7f7c137ea0 2019-12-05T06:53:02.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.790+0000 7f7f797fa700 1 -- 172.21.15.16:0/4272081215 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f640048d0 con 0x7f7f7c137ea0 2019-12-05T06:53:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.790+0000 7f7f824d5700 1 -- 172.21.15.16:0/4272081215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f7c12acd0 con 0x7f7f7c137ea0 2019-12-05T06:53:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.790+0000 7f7f824d5700 1 -- 172.21.15.16:0/4272081215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f7c12acd0 con 0x7f7f7c137ea0 2019-12-05T06:53:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.791+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=THROTTLE_DONE pgs=1459 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.791+0000 7f7f797fa700 1 -- 172.21.15.16:0/4272081215 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7f64004430 con 0x7f7f7c137ea0 2019-12-05T06:53:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.791+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=THROTTLE_DONE pgs=1459 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.792+0000 7f7f797fa700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f60023630 0x7f7f60025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:02.807 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.792+0000 7f7f797fa700 1 -- 172.21.15.16:0/4272081215 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f7f64045e30 con 0x7f7f7c137ea0 2019-12-05T06:53:02.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.792+0000 7f7f80a72700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f60023630 0x7f7f60025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:02.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.792+0000 7f7f80a72700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f60023630 0x7f7f60025ac0 crc :-1 s=READY pgs=612 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:02.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.808+0000 7f7f824d5700 1 -- 172.21.15.16:0/4272081215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f7c12acd0 con 0x7f7f7c137ea0 2019-12-05T06:53:02.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.815+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=THROTTLE_DONE pgs=1459 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:02.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:02.815+0000 7f7f797fa700 1 -- 172.21.15.16:0/4272081215 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7f64031ed0 con 0x7f7f7c137ea0 2019-12-05T06:53:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.067+0000 7f7f824d5700 1 -- 172.21.15.16:0/4272081215 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7f7c12acd0 con 0x7f7f7c137ea0 2019-12-05T06:53:03.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.068+0000 7f7f81273700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=THROTTLE_DONE pgs=1459 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:03.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.068+0000 7f7f797fa700 1 -- 172.21.15.16:0/4272081215 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f7f64031ed0 con 0x7f7f7c137ea0 2019-12-05T06:53:03.069 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:03.069 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7857,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:03.070 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 177 2019-12-05T06:53:03.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.070+0000 7f7f824d5700 1 -- 172.21.15.16:0/4272081215 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f60023630 msgr2=0x7f7f60025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:03.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.070+0000 7f7f824d5700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f60023630 0x7f7f60025ac0 crc :-1 s=READY pgs=612 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:03.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.070+0000 7f7f824d5700 1 -- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 msgr2=0x7f7f7c12bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:03.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.070+0000 7f7f824d5700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 0x7f7f7c12bb10 secure :-1 s=READY pgs=1459 cs=0 l=1 rx=0x7f7f64010430 tx=0x7f7f64010260).stop 2019-12-05T06:53:03.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.071+0000 7f7f824d5700 1 -- 172.21.15.16:0/4272081215 shutdown_connections 2019-12-05T06:53:03.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.071+0000 7f7f824d5700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7f60023630 0x7f7f60025ac0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:03.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.071+0000 7f7f824d5700 1 --2- 172.21.15.16:0/4272081215 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7f7c137ea0 0x7f7f7c12bb10 unknown :-1 s=CLOSED pgs=1459 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.071+0000 7f7f824d5700 1 --2- 172.21.15.16:0/4272081215 >> v2:172.21.15.76:3300/0 conn(0x7f7f7c128f50 0x7f7f7c12b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.071+0000 7f7f824d5700 1 -- 172.21.15.16:0/4272081215 shutdown_connections 2019-12-05T06:53:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.071+0000 7f7f824d5700 1 -- 172.21.15.16:0/4272081215 wait complete. 2019-12-05T06:53:03.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:03.071+0000 7f7f824d5700 1 -- 172.21.15.16:0/4272081215 >> 172.21.15.16:0/4272081215 conn(0x7f7f7c06d260 msgr2=0x7f7f7c09aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:03.082 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:53:05.083 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:05.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.361+0000 7f38ca292700 1 Processor -- start 2019-12-05T06:53:05.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.361+0000 7f38ca292700 1 -- start start 2019-12-05T06:53:05.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.363+0000 7f38ca292700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38c408d2c0 0x7f38c408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:05.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.363+0000 7f38ca292700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c408fef0 0x7f38c408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.363+0000 7f38ca292700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c40958e0 con 0x7f38c408c6d0 2019-12-05T06:53:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.363+0000 7f38ca292700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c408df90 con 0x7f38c408d2c0 2019-12-05T06:53:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.363+0000 7f38ca292700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c40a3f70 con 0x7f38c408fef0 2019-12-05T06:53:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.364+0000 7f38c3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38c408d2c0 0x7f38c408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.364+0000 7f38c3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38c408d2c0 0x7f38c408ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34760/0 (socket says 172.21.15.16:34760) 2019-12-05T06:53:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.364+0000 7f38c882f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38c408c6d0 0x7f38c408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48158/0 (socket says 172.21.15.16:48158) 2019-12-05T06:53:05.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.364+0000 7f38c9030700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c408fef0 0x7f38c408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.364+0000 7f38c3fff700 1 -- 172.21.15.16:0/1538415117 learned_addr learned my addr 172.21.15.16:0/1538415117 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.364+0000 7f38c3fff700 1 -- 172.21.15.16:0/1538415117 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c408fef0 msgr2=0x7f38c408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.364+0000 7f38c3fff700 1 --2- 172.21.15.16:0/1538415117 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c408fef0 0x7f38c408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.364+0000 7f38c3fff700 1 -- 172.21.15.16:0/1538415117 >> v1:172.21.15.16:6789/0 conn(0x7f38c408c6d0 legacy=0x7f38c408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.364+0000 7f38c37fe700 1 -- 172.21.15.16:0/1538415117 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4268212199 0 0) 0x7f38bc001480 con 0x7f38c408c6d0 2019-12-05T06:53:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.364+0000 7f38c3fff700 1 -- 172.21.15.16:0/1538415117 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c408e620 con 0x7f38c408d2c0 2019-12-05T06:53:05.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.365+0000 7f38c9030700 1 --2- 172.21.15.16:0/1538415117 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c408fef0 0x7f38c408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:53:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.365+0000 7f38c3fff700 1 --2- 172.21.15.16:0/1538415117 >> v2:172.21.15.16:3300/0 conn(0x7f38c408d2c0 0x7f38c408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f38b400f740 tx=0x7f38b400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.365+0000 7f38c3fff700 1 --2- 172.21.15.16:0/1538415117 >> v2:172.21.15.16:3300/0 conn(0x7f38c408d2c0 0x7f38c408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f38b400f740 tx=0x7f38b400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.365+0000 7f38c3fff700 1 --2- 172.21.15.16:0/1538415117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c408d2c0 0x7f38c408ef20 secure :-1 s=READY pgs=1461 cs=0 l=1 rx=0x7f38b400f740 tx=0x7f38b400fae0).ready entity=mon.0 client_cookie=187f7d67d206d2d8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.365+0000 7f38c3fff700 1 --2- 172.21.15.16:0/1538415117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c408d2c0 0x7f38c408ef20 secure :-1 s=THROTTLE_DONE pgs=1461 cs=0 l=1 rx=0x7f38b400f740 tx=0x7f38b400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.365+0000 7f38c37fe700 1 -- 172.21.15.16:0/1538415117 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38b4000960 con 0x7f38c408d2c0 2019-12-05T06:53:05.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.365+0000 7f38c3fff700 1 --2- 172.21.15.16:0/1538415117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c408d2c0 0x7f38c408ef20 secure :-1 s=THROTTLE_DONE pgs=1461 cs=0 l=1 rx=0x7f38b400f740 tx=0x7f38b400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.365+0000 7f38c3fff700 1 --2- 172.21.15.16:0/1538415117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c408d2c0 0x7f38c408ef20 secure :-1 s=THROTTLE_DONE pgs=1461 cs=0 l=1 rx=0x7f38b400f740 tx=0x7f38b400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.365+0000 7f38c37fe700 1 -- 172.21.15.16:0/1538415117 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b4000c90 con 0x7f38c408d2c0 2019-12-05T06:53:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.365+0000 7f38c37fe700 1 -- 172.21.15.16:0/1538415117 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38b40013a0 con 0x7f38c408d2c0 2019-12-05T06:53:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.366+0000 7f38ca292700 1 -- 172.21.15.16:0/1538415117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c408d2c0 msgr2=0x7f38c408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.366+0000 7f38ca292700 1 --2- 172.21.15.16:0/1538415117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c408d2c0 0x7f38c408ef20 secure :-1 s=READY pgs=1461 cs=0 l=1 rx=0x7f38b400f740 tx=0x7f38b400fae0).stop 2019-12-05T06:53:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.367+0000 7f38ca292700 1 -- 172.21.15.16:0/1538415117 shutdown_connections 2019-12-05T06:53:05.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.367+0000 7f38ca292700 1 --2- 172.21.15.16:0/1538415117 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c408d2c0 0x7f38c408ef20 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:05.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.367+0000 7f38ca292700 1 --2- 172.21.15.16:0/1538415117 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c408fef0 0x7f38c408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:05.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.367+0000 7f38ca292700 1 -- 172.21.15.16:0/1538415117 shutdown_connections 2019-12-05T06:53:05.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.367+0000 7f38ca292700 1 -- 172.21.15.16:0/1538415117 wait complete. 2019-12-05T06:53:05.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.367+0000 7f38ca292700 1 -- 172.21.15.16:0/1538415117 >> 172.21.15.16:0/1538415117 conn(0x7f38c4061940 msgr2=0x7f38c4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.373+0000 7f38ca292700 1 Processor -- start 2019-12-05T06:53:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.373+0000 7f38ca292700 1 -- start start 2019-12-05T06:53:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.375+0000 7f38ca292700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38c414da50 0x7f38c412fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.375+0000 7f38ca292700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c41309e0 0x7f38c412ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:05.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.375+0000 7f38ca292700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c4088c30 con 0x7f38c4130490 2019-12-05T06:53:05.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.375+0000 7f38ca292700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c408a8f0 con 0x7f38c414da50 2019-12-05T06:53:05.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.375+0000 7f38ca292700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c4089ec0 con 0x7f38c41309e0 2019-12-05T06:53:05.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.375+0000 7f38c882f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f38c414da50 0x7f38c412fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:05.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.375+0000 7f38c9030700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f38c4130490 0x7f38c4080210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48168/0 (socket says 172.21.15.16:48168) 2019-12-05T06:53:05.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.375+0000 7f38c9030700 1 -- 172.21.15.16:0/3424037100 learned_addr learned my addr 172.21.15.16:0/3424037100 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:05.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.376+0000 7f38c3fff700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c41309e0 0x7f38c412ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.376+0000 7f38c882f700 1 -- 172.21.15.16:0/3424037100 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c41309e0 msgr2=0x7f38c412ff50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.376+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c41309e0 0x7f38c412ff50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.376+0000 7f38c882f700 1 -- 172.21.15.16:0/3424037100 >> v1:172.21.15.16:6789/0 conn(0x7f38c4130490 legacy=0x7f38c4080210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.376+0000 7f38c882f700 1 -- 172.21.15.16:0/3424037100 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c412f110 con 0x7f38c414da50 2019-12-05T06:53:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.376+0000 7f38c17fa700 1 -- 172.21.15.16:0/3424037100 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1941152719 0 0) 0x7f38ac00a8c0 con 0x7f38c4130490 2019-12-05T06:53:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.376+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> v2:172.21.15.16:3300/0 conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.376+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> v2:172.21.15.16:3300/0 conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.376+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=READY pgs=1463 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:05.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.377+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=THROTTLE_DONE pgs=1463 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.377+0000 7f38c17fa700 1 -- 172.21.15.16:0/3424037100 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38bc0045c0 con 0x7f38c414da50 2019-12-05T06:53:05.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.377+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=THROTTLE_DONE pgs=1463 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.377+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=THROTTLE_DONE pgs=1463 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.377+0000 7f38c17fa700 1 -- 172.21.15.16:0/3424037100 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38bc001010 con 0x7f38c414da50 2019-12-05T06:53:05.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.377+0000 7f38ca292700 1 -- 172.21.15.16:0/3424037100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c412f110 con 0x7f38c414da50 2019-12-05T06:53:05.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.377+0000 7f38c17fa700 1 -- 172.21.15.16:0/3424037100 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38bc004ba0 con 0x7f38c414da50 2019-12-05T06:53:05.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.377+0000 7f38ca292700 1 -- 172.21.15.16:0/3424037100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c412f110 con 0x7f38c414da50 2019-12-05T06:53:05.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.378+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=THROTTLE_DONE pgs=1463 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.379+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=THROTTLE_DONE pgs=1463 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.379+0000 7f38c17fa700 1 -- 172.21.15.16:0/3424037100 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f38bc0045c0 con 0x7f38c414da50 2019-12-05T06:53:05.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.379+0000 7f38c17fa700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38a8023630 0x7f38a8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:05.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.379+0000 7f38c17fa700 1 -- 172.21.15.16:0/3424037100 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f38bc0460e0 con 0x7f38c414da50 2019-12-05T06:53:05.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.380+0000 7f38c3fff700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38a8023630 0x7f38a8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:05.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.380+0000 7f38c3fff700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38a8023630 0x7f38a8025ac0 crc :-1 s=READY pgs=613 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:05.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.392+0000 7f38ca292700 1 -- 172.21.15.16:0/3424037100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c412f110 con 0x7f38c414da50 2019-12-05T06:53:05.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.401+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=THROTTLE_DONE pgs=1463 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.401+0000 7f38c17fa700 1 -- 172.21.15.16:0/3424037100 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f38bc0015a0 con 0x7f38c414da50 2019-12-05T06:53:05.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.644+0000 7f38ca292700 1 -- 172.21.15.16:0/3424037100 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f38c412f110 con 0x7f38c414da50 2019-12-05T06:53:05.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.645+0000 7f38c882f700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=THROTTLE_DONE pgs=1463 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:05.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.645+0000 7f38c17fa700 1 -- 172.21.15.16:0/3424037100 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f38bc0045c0 con 0x7f38c414da50 2019-12-05T06:53:05.663 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 177 2019-12-05T06:53:05.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.647+0000 7f38ca292700 1 -- 172.21.15.16:0/3424037100 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38a8023630 msgr2=0x7f38a8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:05.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.647+0000 7f38ca292700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38a8023630 0x7f38a8025ac0 crc :-1 s=READY pgs=613 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:05.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.647+0000 7f38ca292700 1 -- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 msgr2=0x7f38c412fa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.647+0000 7f38ca292700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 0x7f38c412fa10 secure :-1 s=READY pgs=1463 cs=0 l=1 rx=0x7f38bc0106f0 tx=0x7f38bc010520).stop 2019-12-05T06:53:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.647+0000 7f38ca292700 1 -- 172.21.15.16:0/3424037100 shutdown_connections 2019-12-05T06:53:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.647+0000 7f38ca292700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f38c41309e0 0x7f38c412ff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.647+0000 7f38ca292700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f38a8023630 0x7f38a8025ac0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:05.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.647+0000 7f38ca292700 1 --2- 172.21.15.16:0/3424037100 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f38c414da50 0x7f38c412fa10 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.647+0000 7f38ca292700 1 -- 172.21.15.16:0/3424037100 shutdown_connections 2019-12-05T06:53:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.648+0000 7f38ca292700 1 -- 172.21.15.16:0/3424037100 wait complete. 2019-12-05T06:53:05.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:05.648+0000 7f38ca292700 1 -- 172.21.15.16:0/3424037100 >> 172.21.15.16:0/3424037100 conn(0x7f38c409de30 msgr2=0x7f38c4070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:05.665 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:05.665 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7857,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:05.666 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:53:07.667 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:07.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.934+0000 7f6383e7d700 1 Processor -- start 2019-12-05T06:53:07.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.934+0000 7f6383e7d700 1 -- start start 2019-12-05T06:53:07.945 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.936+0000 7f6383e7d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f637c08c6d0 0x7f637c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:07.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.936+0000 7f6383e7d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c08f720 0x7f637c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:07.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.937+0000 7f6383e7d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c08fb70 con 0x7f637c08fef0 2019-12-05T06:53:07.946 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.937+0000 7f6383e7d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637c08df90 con 0x7f637c08c6d0 2019-12-05T06:53:07.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.937+0000 7f6383e7d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637c0a3f70 con 0x7f637c08f720 2019-12-05T06:53:07.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.937+0000 7f638241a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f637c08c6d0 0x7f637c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:07.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.937+0000 7f638241a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f637c08c6d0 0x7f637c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34772/0 (socket says 172.21.15.16:34772) 2019-12-05T06:53:07.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.937+0000 7f638241a700 1 -- 172.21.15.16:0/2277438785 learned_addr learned my addr 172.21.15.16:0/2277438785 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:07.947 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.937+0000 7f6382c1b700 1 --2- 172.21.15.16:0/2277438785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c08f720 0x7f637c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:07.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f638241a700 1 -- 172.21.15.16:0/2277438785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c08f720 msgr2=0x7f637c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f638241a700 1 --2- 172.21.15.16:0/2277438785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c08f720 0x7f637c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f638241a700 1 -- 172.21.15.16:0/2277438785 >> v1:172.21.15.76:6789/0 conn(0x7f637c08fef0 legacy=0x7f637c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f6381418700 1 -- 172.21.15.16:0/2277438785 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954948080 0 0) 0x7f6370001080 con 0x7f637c08fef0 2019-12-05T06:53:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f638241a700 1 -- 172.21.15.16:0/2277438785 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c08d540 con 0x7f637c08c6d0 2019-12-05T06:53:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f6382c1b700 1 --2- 172.21.15.16:0/2277438785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c08f720 0x7f637c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:53:07.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f638241a700 1 --2- 172.21.15.16:0/2277438785 >> v2:172.21.15.16:3300/0 conn(0x7f637c08c6d0 0x7f637c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f637400fac0 tx=0x7f637400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f638241a700 1 --2- 172.21.15.16:0/2277438785 >> v2:172.21.15.16:3300/0 conn(0x7f637c08c6d0 0x7f637c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f637400fac0 tx=0x7f637400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f638241a700 1 --2- 172.21.15.16:0/2277438785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c08c6d0 0x7f637c08cac0 secure :-1 s=READY pgs=1464 cs=0 l=1 rx=0x7f637400fac0 tx=0x7f637400fe60).ready entity=mon.0 client_cookie=f208837451be6794 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f638241a700 1 --2- 172.21.15.16:0/2277438785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c08c6d0 0x7f637c08cac0 secure :-1 s=THROTTLE_DONE pgs=1464 cs=0 l=1 rx=0x7f637400fac0 tx=0x7f637400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f6381418700 1 -- 172.21.15.16:0/2277438785 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6374000ce0 con 0x7f637c08c6d0 2019-12-05T06:53:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f638241a700 1 --2- 172.21.15.16:0/2277438785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c08c6d0 0x7f637c08cac0 secure :-1 s=THROTTLE_DONE pgs=1464 cs=0 l=1 rx=0x7f637400fac0 tx=0x7f637400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.938+0000 7f638241a700 1 --2- 172.21.15.16:0/2277438785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c08c6d0 0x7f637c08cac0 secure :-1 s=THROTTLE_DONE pgs=1464 cs=0 l=1 rx=0x7f637400fac0 tx=0x7f637400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.939+0000 7f6381418700 1 -- 172.21.15.16:0/2277438785 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6374001010 con 0x7f637c08c6d0 2019-12-05T06:53:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.939+0000 7f6381418700 1 -- 172.21.15.16:0/2277438785 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6374001720 con 0x7f637c08c6d0 2019-12-05T06:53:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.939+0000 7f6383e7d700 1 -- 172.21.15.16:0/2277438785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c08c6d0 msgr2=0x7f637c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.939+0000 7f6383e7d700 1 --2- 172.21.15.16:0/2277438785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c08c6d0 0x7f637c08cac0 secure :-1 s=READY pgs=1464 cs=0 l=1 rx=0x7f637400fac0 tx=0x7f637400fe60).stop 2019-12-05T06:53:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.941+0000 7f6383e7d700 1 -- 172.21.15.16:0/2277438785 shutdown_connections 2019-12-05T06:53:07.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.941+0000 7f6383e7d700 1 --2- 172.21.15.16:0/2277438785 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c08f720 0x7f637c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:07.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.941+0000 7f6383e7d700 1 --2- 172.21.15.16:0/2277438785 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f637c08c6d0 0x7f637c08cac0 unknown :-1 s=CLOSED pgs=1464 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.941+0000 7f6383e7d700 1 -- 172.21.15.16:0/2277438785 shutdown_connections 2019-12-05T06:53:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.941+0000 7f6383e7d700 1 -- 172.21.15.16:0/2277438785 wait complete. 2019-12-05T06:53:07.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.941+0000 7f6383e7d700 1 -- 172.21.15.16:0/2277438785 >> 172.21.15.16:0/2277438785 conn(0x7f637c061940 msgr2=0x7f637c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:07.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.946+0000 7f6383e7d700 1 Processor -- start 2019-12-05T06:53:07.955 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.946+0000 7f6383e7d700 1 -- start start 2019-12-05T06:53:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f6383e7d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c09bbe0 0x7f637c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f6383e7d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f637c084f30 0x7f637c155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f6383e7d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f637c084b90 con 0x7f637c096e90 2019-12-05T06:53:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f6383e7d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f637c09a450 con 0x7f637c084f30 2019-12-05T06:53:07.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f6383e7d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f637c099a20 con 0x7f637c09bbe0 2019-12-05T06:53:07.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f638241a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f637c096e90 0x7f637c09b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48178/0 (socket says 172.21.15.16:48178) 2019-12-05T06:53:07.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f638241a700 1 -- 172.21.15.16:0/1305093377 learned_addr learned my addr 172.21.15.16:0/1305093377 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:07.959 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> v2:172.21.15.76:3300/0 conn(0x7f637c084f30 0x7f637c155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f6381c19700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c09bbe0 0x7f637c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f637affd700 1 -- 172.21.15.16:0/1305093377 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914174568 0 0) 0x7f6374008190 con 0x7f637c096e90 2019-12-05T06:53:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.948+0000 7f637affd700 1 -- 172.21.15.16:0/1305093377 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f635c002e80 con 0x7f637c096e90 2019-12-05T06:53:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.949+0000 7f6382c1b700 1 -- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c09bbe0 msgr2=0x7f637c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.949+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c09bbe0 0x7f637c1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.949+0000 7f6382c1b700 1 -- 172.21.15.16:0/1305093377 >> v1:172.21.15.16:6789/0 conn(0x7f637c096e90 legacy=0x7f637c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:07.960 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.949+0000 7f6382c1b700 1 -- 172.21.15.16:0/1305093377 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f637c09aae0 con 0x7f637c084f30 2019-12-05T06:53:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.949+0000 7f6381c19700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c09bbe0 0x7f637c1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:53:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.949+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> v2:172.21.15.76:3300/0 conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.949+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> v2:172.21.15.76:3300/0 conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.949+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=READY pgs=988 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:07.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.950+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.950+0000 7f637affd700 1 -- 172.21.15.16:0/1305093377 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f636c00a670 con 0x7f637c084f30 2019-12-05T06:53:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.950+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.950+0000 7f6383e7d700 1 -- 172.21.15.16:0/1305093377 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f637c09aae0 con 0x7f637c084f30 2019-12-05T06:53:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.950+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.950+0000 7f637affd700 1 -- 172.21.15.16:0/1305093377 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f636c00a9a0 con 0x7f637c084f30 2019-12-05T06:53:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.950+0000 7f637affd700 1 -- 172.21.15.16:0/1305093377 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f636c00ab90 con 0x7f637c084f30 2019-12-05T06:53:07.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.950+0000 7f6383e7d700 1 -- 172.21.15.16:0/1305093377 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f637c09aae0 con 0x7f637c084f30 2019-12-05T06:53:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.951+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.952+0000 7f637affd700 1 -- 172.21.15.16:0/1305093377 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f636c0095b0 con 0x7f637c084f30 2019-12-05T06:53:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.952+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.952+0000 7f637affd700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f635c023670 0x7f635c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.952+0000 7f637affd700 1 -- 172.21.15.16:0/1305093377 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 8525+0+0 (secure 0 0 0) 0x7f636c045d20 con 0x7f637c084f30 2019-12-05T06:53:07.963 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.952+0000 7f638241a700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f635c023670 0x7f635c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:07.964 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.953+0000 7f638241a700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f635c023670 0x7f635c025ae0 crc :-1 s=READY pgs=614 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:07.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.966+0000 7f6383e7d700 1 -- 172.21.15.16:0/1305093377 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f637c09aae0 con 0x7f637c084f30 2019-12-05T06:53:07.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.975+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:07.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:07.975+0000 7f637affd700 1 -- 172.21.15.16:0/1305093377 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f636c00ab90 con 0x7f637c084f30 2019-12-05T06:53:08.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.232+0000 7f6383e7d700 1 -- 172.21.15.16:0/1305093377 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f637c09aae0 con 0x7f637c084f30 2019-12-05T06:53:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.234+0000 7f6382c1b700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=THROTTLE_DONE pgs=988 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:08.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.234+0000 7f637affd700 1 -- 172.21.15.16:0/1305093377 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 177 v177) v1 ==== 95+0+2106 (secure 0 0 0) 0x7f636c0095b0 con 0x7f637c084f30 2019-12-05T06:53:08.237 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 177 2019-12-05T06:53:08.237 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:08.237 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":177,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7857,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":177}],"filesystems":[{"mdsmap":{"epoch":176,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:52:18.715151+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":78,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5972},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5972":{"gid":5972,"name":"b","rank":0,"incarnation":173,"state":"up:active","state_seq":31,"addr":"172.21.15.76:6835/727762895","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":727762895},{"type":"v1","addr":"172.21.15.76:6835","nonce":727762895}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.238+0000 7f6383e7d700 1 -- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f635c023670 msgr2=0x7f635c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.238+0000 7f6383e7d700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f635c023670 0x7f635c025ae0 crc :-1 s=READY pgs=614 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.239+0000 7f6383e7d700 1 -- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 msgr2=0x7f637c155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.239+0000 7f6383e7d700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 0x7f637c155f30 secure :-1 s=READY pgs=988 cs=0 l=1 rx=0x7f636c0103a0 tx=0x7f636c010740).stop 2019-12-05T06:53:08.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.239+0000 7f6383e7d700 1 -- 172.21.15.16:0/1305093377 shutdown_connections 2019-12-05T06:53:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.239+0000 7f6383e7d700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f635c023670 0x7f635c025ae0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.239+0000 7f6383e7d700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f637c084f30 0x7f637c155f30 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.239+0000 7f6383e7d700 1 --2- 172.21.15.16:0/1305093377 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f637c09bbe0 0x7f637c1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.239+0000 7f6383e7d700 1 -- 172.21.15.16:0/1305093377 shutdown_connections 2019-12-05T06:53:08.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.239+0000 7f6383e7d700 1 -- 172.21.15.16:0/1305093377 wait complete. 2019-12-05T06:53:08.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:08.239+0000 7f6383e7d700 1 -- 172.21.15.16:0/1305093377 >> 172.21.15.16:0/1305093377 conn(0x7f637c0714f0 msgr2=0x7f637c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:08.249 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:53:10.251 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:10.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.411+0000 7ff903142700 1 Processor -- start 2019-12-05T06:53:10.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.412+0000 7ff903142700 1 -- start start 2019-12-05T06:53:10.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.413+0000 7ff903142700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff8fc08c6d0 0x7ff8fc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:10.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.413+0000 7ff903142700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff8fc08fef0 0x7ff8fc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:10.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.413+0000 7ff903142700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff8fc08ef20 0x7ff8fc08f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:10.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.413+0000 7ff903142700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8fc08df90 con 0x7ff8fc08fef0 2019-12-05T06:53:10.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.413+0000 7ff903142700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8fc0a3f70 con 0x7ff8fc08c6d0 2019-12-05T06:53:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.413+0000 7ff903142700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8fc08d830 con 0x7ff8fc08ef20 2019-12-05T06:53:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.413+0000 7ff900ede700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff8fc08fef0 0x7ff8fc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.413+0000 7ff900ede700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff8fc08fef0 0x7ff8fc08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34802/0 (socket says 172.21.15.16:34802) 2019-12-05T06:53:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.413+0000 7ff900ede700 1 -- 172.21.15.16:0/4027733366 learned_addr learned my addr 172.21.15.16:0/4027733366 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.413+0000 7ff9016df700 1 --2- 172.21.15.16:0/4027733366 >> v2:172.21.15.76:3300/0 conn(0x7ff8fc08c6d0 0x7ff8fc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff901ee0700 1 --2- 172.21.15.16:0/4027733366 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff8fc08ef20 0x7ff8fc08f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:10.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 -- 172.21.15.16:0/4027733366 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff8fc08ef20 msgr2=0x7ff8fc08f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 --2- 172.21.15.16:0/4027733366 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff8fc08ef20 0x7ff8fc08f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 -- 172.21.15.16:0/4027733366 >> v2:172.21.15.76:3300/0 conn(0x7ff8fc08c6d0 msgr2=0x7ff8fc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 --2- 172.21.15.16:0/4027733366 >> v2:172.21.15.76:3300/0 conn(0x7ff8fc08c6d0 0x7ff8fc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 -- 172.21.15.16:0/4027733366 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc08d540 con 0x7ff8fc08fef0 2019-12-05T06:53:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff901ee0700 1 --2- 172.21.15.16:0/4027733366 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff8fc08ef20 0x7ff8fc08f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:53:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 --2- 172.21.15.16:0/4027733366 >> v2:172.21.15.16:3300/0 conn(0x7ff8fc08fef0 0x7ff8fc08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff8ec00f740 tx=0x7ff8ec00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.422 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 --2- 172.21.15.16:0/4027733366 >> v2:172.21.15.16:3300/0 conn(0x7ff8fc08fef0 0x7ff8fc08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff8ec00f740 tx=0x7ff8ec00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 --2- 172.21.15.16:0/4027733366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc08fef0 0x7ff8fc08d000 secure :-1 s=READY pgs=1466 cs=0 l=1 rx=0x7ff8ec00f740 tx=0x7ff8ec00f5f0).ready entity=mon.0 client_cookie=26054d51341976b6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 --2- 172.21.15.16:0/4027733366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc08fef0 0x7ff8fc08d000 secure :-1 s=THROTTLE_DONE pgs=1466 cs=0 l=1 rx=0x7ff8ec00f740 tx=0x7ff8ec00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 --2- 172.21.15.16:0/4027733366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc08fef0 0x7ff8fc08d000 secure :-1 s=THROTTLE_DONE pgs=1466 cs=0 l=1 rx=0x7ff8ec00f740 tx=0x7ff8ec00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff8f3fff700 1 -- 172.21.15.16:0/4027733366 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8ec000960 con 0x7ff8fc08fef0 2019-12-05T06:53:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff900ede700 1 --2- 172.21.15.16:0/4027733366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc08fef0 0x7ff8fc08d000 secure :-1 s=THROTTLE_DONE pgs=1466 cs=0 l=1 rx=0x7ff8ec00f740 tx=0x7ff8ec00f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff8f3fff700 1 -- 172.21.15.16:0/4027733366 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8ec000c90 con 0x7ff8fc08fef0 2019-12-05T06:53:10.423 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.414+0000 7ff8f3fff700 1 -- 172.21.15.16:0/4027733366 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8ec0013a0 con 0x7ff8fc08fef0 2019-12-05T06:53:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.415+0000 7ff903142700 1 -- 172.21.15.16:0/4027733366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc08fef0 msgr2=0x7ff8fc08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.415+0000 7ff903142700 1 --2- 172.21.15.16:0/4027733366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc08fef0 0x7ff8fc08d000 secure :-1 s=READY pgs=1466 cs=0 l=1 rx=0x7ff8ec00f740 tx=0x7ff8ec00f5f0).stop 2019-12-05T06:53:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.416+0000 7ff903142700 1 -- 172.21.15.16:0/4027733366 shutdown_connections 2019-12-05T06:53:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.416+0000 7ff903142700 1 --2- 172.21.15.16:0/4027733366 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff8fc08ef20 0x7ff8fc08f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.416+0000 7ff903142700 1 --2- 172.21.15.16:0/4027733366 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc08fef0 0x7ff8fc08d000 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.416+0000 7ff903142700 1 --2- 172.21.15.16:0/4027733366 >> v2:172.21.15.76:3300/0 conn(0x7ff8fc08c6d0 0x7ff8fc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:10.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.416+0000 7ff903142700 1 -- 172.21.15.16:0/4027733366 shutdown_connections 2019-12-05T06:53:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.416+0000 7ff903142700 1 -- 172.21.15.16:0/4027733366 wait complete. 2019-12-05T06:53:10.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.416+0000 7ff903142700 1 -- 172.21.15.16:0/4027733366 >> 172.21.15.16:0/4027733366 conn(0x7ff8fc061940 msgr2=0x7ff8fc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:10.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.420+0000 7ff903142700 1 Processor -- start 2019-12-05T06:53:10.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.420+0000 7ff903142700 1 -- start start 2019-12-05T06:53:10.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.421+0000 7ff903142700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff8fc06d680 0x7ff8fc085aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff903142700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff8fc154b70 0x7ff8fc156f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff903142700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8fc134550 con 0x7ff8fc086a70 2019-12-05T06:53:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff903142700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8fc084b10 con 0x7ff8fc154b70 2019-12-05T06:53:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff903142700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8fc135820 con 0x7ff8fc06d680 2019-12-05T06:53:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff901ee0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff8fc154b70 0x7ff8fc156f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff901ee0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff8fc154b70 0x7ff8fc156f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34810/0 (socket says 172.21.15.16:34810) 2019-12-05T06:53:10.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff901ee0700 1 -- 172.21.15.16:0/951074195 learned_addr learned my addr 172.21.15.16:0/951074195 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:10.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff9016df700 1 --2- 172.21.15.16:0/951074195 >> v2:172.21.15.76:3300/0 conn(0x7ff8fc06d680 0x7ff8fc085aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:10.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff901ee0700 1 -- 172.21.15.16:0/951074195 >> v2:172.21.15.76:3300/0 conn(0x7ff8fc06d680 msgr2=0x7ff8fc085aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:10.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> v2:172.21.15.76:3300/0 conn(0x7ff8fc06d680 0x7ff8fc085aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:10.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff901ee0700 1 -- 172.21.15.16:0/951074195 >> v1:172.21.15.76:6789/0 conn(0x7ff8fc086a70 legacy=0x7ff8fc085fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff901ee0700 1 -- 172.21.15.16:0/951074195 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8fc0851a0 con 0x7ff8fc154b70 2019-12-05T06:53:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> v2:172.21.15.16:3300/0 conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> v2:172.21.15.16:3300/0 conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.422+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=READY pgs=1467 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.423+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=THROTTLE_DONE pgs=1467 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.423+0000 7ff8f1ffb700 1 -- 172.21.15.16:0/951074195 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8f4008800 con 0x7ff8fc154b70 2019-12-05T06:53:10.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.423+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=THROTTLE_DONE pgs=1467 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.423+0000 7ff8f1ffb700 1 -- 172.21.15.16:0/951074195 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8f4009e00 con 0x7ff8fc154b70 2019-12-05T06:53:10.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.423+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=THROTTLE_DONE pgs=1467 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.423+0000 7ff903142700 1 -- 172.21.15.16:0/951074195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8fc0851a0 con 0x7ff8fc154b70 2019-12-05T06:53:10.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.423+0000 7ff8f1ffb700 1 -- 172.21.15.16:0/951074195 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8f4009b70 con 0x7ff8fc154b70 2019-12-05T06:53:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.423+0000 7ff903142700 1 -- 172.21.15.16:0/951074195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8fc0851a0 con 0x7ff8fc154b70 2019-12-05T06:53:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.423+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=THROTTLE_DONE pgs=1467 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.424+0000 7ff8f1ffb700 1 -- 172.21.15.16:0/951074195 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff8f4009ff0 con 0x7ff8fc154b70 2019-12-05T06:53:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.424+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=THROTTLE_DONE pgs=1467 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.424+0000 7ff8f1ffb700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff8dc023630 0x7ff8dc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:10.435 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.424+0000 7ff8f1ffb700 1 -- 172.21.15.16:0/951074195 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7ff8f4046190 con 0x7ff8fc154b70 2019-12-05T06:53:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.425+0000 7ff9016df700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff8dc023630 0x7ff8dc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.425+0000 7ff9016df700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff8dc023630 0x7ff8dc025ac0 crc :-1 s=READY pgs=615 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:10.437 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.426+0000 7ff903142700 1 -- 172.21.15.16:0/951074195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8fc0851a0 con 0x7ff8fc154b70 2019-12-05T06:53:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.432+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=THROTTLE_DONE pgs=1467 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.432+0000 7ff8f1ffb700 1 -- 172.21.15.16:0/951074195 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff8f4009910 con 0x7ff8fc154b70 2019-12-05T06:53:10.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.623+0000 7ff903142700 1 -- 172.21.15.16:0/951074195 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8fc0851a0 con 0x7ff8fc154b70 2019-12-05T06:53:10.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.623+0000 7ff901ee0700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=THROTTLE_DONE pgs=1467 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.624+0000 7ff8f1ffb700 1 -- 172.21.15.16:0/951074195 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+1771 (secure 0 0 0) 0x7ff8f4009ff0 con 0x7ff8fc154b70 2019-12-05T06:53:10.625 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 178 2019-12-05T06:53:10.625 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:10.625 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:08.580870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:10.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.626+0000 7ff903142700 1 -- 172.21.15.16:0/951074195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff8dc023630 msgr2=0x7ff8dc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:10.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.626+0000 7ff903142700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff8dc023630 0x7ff8dc025ac0 crc :-1 s=READY pgs=615 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:10.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.626+0000 7ff903142700 1 -- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 msgr2=0x7ff8fc156f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:10.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.626+0000 7ff903142700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 0x7ff8fc156f70 secure :-1 s=READY pgs=1467 cs=0 l=1 rx=0x7ff8f4010800 tx=0x7ff8f4010600).stop 2019-12-05T06:53:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.627+0000 7ff903142700 1 -- 172.21.15.16:0/951074195 shutdown_connections 2019-12-05T06:53:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.627+0000 7ff903142700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff8dc023630 0x7ff8dc025ac0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.627+0000 7ff903142700 1 --2- 172.21.15.16:0/951074195 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff8fc154b70 0x7ff8fc156f70 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:10.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.627+0000 7ff903142700 1 --2- 172.21.15.16:0/951074195 >> v2:172.21.15.76:3300/0 conn(0x7ff8fc06d680 0x7ff8fc085aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:10.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.627+0000 7ff903142700 1 -- 172.21.15.16:0/951074195 shutdown_connections 2019-12-05T06:53:10.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.627+0000 7ff903142700 1 -- 172.21.15.16:0/951074195 wait complete. 2019-12-05T06:53:10.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.627+0000 7ff903142700 1 -- 172.21.15.16:0/951074195 >> 172.21.15.16:0/951074195 conn(0x7ff8fc09be40 msgr2=0x7ff8fc06fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:10.638 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:53:10.638 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:53:10.638 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:10.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.809+0000 7fc077660700 1 Processor -- start 2019-12-05T06:53:10.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.809+0000 7fc077660700 1 -- start start 2019-12-05T06:53:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.811+0000 7fc077660700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc07008d2c0 0x7fc07008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.811+0000 7fc077660700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0700958e0 con 0x7fc07008c6d0 2019-12-05T06:53:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.811+0000 7fc077660700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0700629e0 con 0x7fc07008fef0 2019-12-05T06:53:10.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc0753fc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc07008d2c0 0x7fc07008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc075bfd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc07008c6d0 0x7fc07008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48214/0 (socket says 172.21.15.16:48214) 2019-12-05T06:53:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc075bfd700 1 -- 172.21.15.16:0/374900811 learned_addr learned my addr 172.21.15.16:0/374900811 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc077660700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc07008df90 con 0x7fc07008d2c0 2019-12-05T06:53:10.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc074bfb700 1 -- 172.21.15.16:0/374900811 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572287285 0 0) 0x7fc06c001150 con 0x7fc07008c6d0 2019-12-05T06:53:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc074bfb700 1 -- 172.21.15.16:0/374900811 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc05c002e80 con 0x7fc07008c6d0 2019-12-05T06:53:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc0753fc700 1 -- 172.21.15.16:0/374900811 >> v1:172.21.15.76:6789/0 conn(0x7fc07008fef0 legacy=0x7fc07008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc074bfb700 1 -- 172.21.15.16:0/374900811 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977416572 0 0) 0x7fc068001400 con 0x7fc07008fef0 2019-12-05T06:53:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc0753fc700 1 -- 172.21.15.16:0/374900811 >> v1:172.21.15.16:6789/0 conn(0x7fc07008c6d0 legacy=0x7fc07008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:10.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc0753fc700 1 -- 172.21.15.16:0/374900811 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc07008e620 con 0x7fc07008d2c0 2019-12-05T06:53:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.812+0000 7fc0753fc700 1 --2- 172.21.15.16:0/374900811 >> v2:172.21.15.16:3300/0 conn(0x7fc07008d2c0 0x7fc07008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc06000af50 tx=0x7fc06000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.813+0000 7fc0753fc700 1 --2- 172.21.15.16:0/374900811 >> v2:172.21.15.16:3300/0 conn(0x7fc07008d2c0 0x7fc07008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc06000af50 tx=0x7fc06000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.813+0000 7fc0753fc700 1 --2- 172.21.15.16:0/374900811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07008d2c0 0x7fc07008ef20 secure :-1 s=READY pgs=1469 cs=0 l=1 rx=0x7fc06000af50 tx=0x7fc06000b760).ready entity=mon.0 client_cookie=c97cfe77345bd290 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:10.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.813+0000 7fc0753fc700 1 --2- 172.21.15.16:0/374900811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07008d2c0 0x7fc07008ef20 secure :-1 s=THROTTLE_DONE pgs=1469 cs=0 l=1 rx=0x7fc06000af50 tx=0x7fc06000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.813+0000 7fc074bfb700 1 -- 172.21.15.16:0/374900811 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc060007210 con 0x7fc07008d2c0 2019-12-05T06:53:10.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.813+0000 7fc0753fc700 1 --2- 172.21.15.16:0/374900811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07008d2c0 0x7fc07008ef20 secure :-1 s=THROTTLE_DONE pgs=1469 cs=0 l=1 rx=0x7fc06000af50 tx=0x7fc06000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.813+0000 7fc0753fc700 1 --2- 172.21.15.16:0/374900811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07008d2c0 0x7fc07008ef20 secure :-1 s=THROTTLE_DONE pgs=1469 cs=0 l=1 rx=0x7fc06000af50 tx=0x7fc06000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.813+0000 7fc074bfb700 1 -- 172.21.15.16:0/374900811 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0600099b0 con 0x7fc07008d2c0 2019-12-05T06:53:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.813+0000 7fc074bfb700 1 -- 172.21.15.16:0/374900811 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc060009ba0 con 0x7fc07008d2c0 2019-12-05T06:53:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.813+0000 7fc077660700 1 -- 172.21.15.16:0/374900811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07008d2c0 msgr2=0x7fc07008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.813+0000 7fc077660700 1 --2- 172.21.15.16:0/374900811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07008d2c0 0x7fc07008ef20 secure :-1 s=READY pgs=1469 cs=0 l=1 rx=0x7fc06000af50 tx=0x7fc06000b760).stop 2019-12-05T06:53:10.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.814+0000 7fc077660700 1 -- 172.21.15.16:0/374900811 shutdown_connections 2019-12-05T06:53:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.814+0000 7fc077660700 1 --2- 172.21.15.16:0/374900811 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc07008d2c0 0x7fc07008ef20 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.814+0000 7fc077660700 1 -- 172.21.15.16:0/374900811 shutdown_connections 2019-12-05T06:53:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.814+0000 7fc077660700 1 -- 172.21.15.16:0/374900811 wait complete. 2019-12-05T06:53:10.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.814+0000 7fc077660700 1 -- 172.21.15.16:0/374900811 >> 172.21.15.16:0/374900811 conn(0x7fc070061940 msgr2=0x7fc070096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:10.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.819+0000 7fc077660700 1 Processor -- start 2019-12-05T06:53:10.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.820+0000 7fc077660700 1 -- start start 2019-12-05T06:53:10.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.821+0000 7fc077660700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:10.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.821+0000 7fc077660700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc070084ad0 0x7fc070135230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:10.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.821+0000 7fc077660700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc070080830 con 0x7fc070132140 2019-12-05T06:53:10.827 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.821+0000 7fc077660700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc07006d980 con 0x7fc070084ad0 2019-12-05T06:53:10.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.821+0000 7fc077660700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc070133870 con 0x7fc070131d50 2019-12-05T06:53:10.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc075bfd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:10.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc0753fc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc070084ad0 0x7fc070135230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:10.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc075bfd700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:38810/0 (socket says 172.21.15.16:38810) 2019-12-05T06:53:10.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc075bfd700 1 -- 172.21.15.16:0/1187992950 learned_addr learned my addr 172.21.15.16:0/1187992950 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:10.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc075bfd700 1 -- 172.21.15.16:0/1187992950 >> v2:172.21.15.76:3300/0 conn(0x7fc070084ad0 msgr2=0x7fc070135230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:10.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> v2:172.21.15.76:3300/0 conn(0x7fc070084ad0 0x7fc070135230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:10.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc075bfd700 1 -- 172.21.15.16:0/1187992950 >> v1:172.21.15.76:6789/0 conn(0x7fc070132140 legacy=0x7fc07014f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:10.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc075bfd700 1 -- 172.21.15.16:0/1187992950 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc070134930 con 0x7fc070131d50 2019-12-05T06:53:10.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc0753fc700 1 --2- 172.21.15.16:0/1187992950 >> v2:172.21.15.76:3300/0 conn(0x7fc070084ad0 0x7fc070135230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:53:10.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.822+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=READY pgs=345 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:10.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.823+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.823+0000 7fc0667fc700 1 -- 172.21.15.16:0/1187992950 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc06c000c90 con 0x7fc070131d50 2019-12-05T06:53:10.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.823+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.823+0000 7fc0667fc700 1 -- 172.21.15.16:0/1187992950 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc06c000c90 con 0x7fc070131d50 2019-12-05T06:53:10.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.823+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.823+0000 7fc077660700 1 -- 172.21.15.16:0/1187992950 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc070134930 con 0x7fc070131d50 2019-12-05T06:53:10.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.823+0000 7fc0667fc700 1 -- 172.21.15.16:0/1187992950 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc06c000e80 con 0x7fc070131d50 2019-12-05T06:53:10.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.823+0000 7fc077660700 1 -- 172.21.15.16:0/1187992950 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc070134930 con 0x7fc070131d50 2019-12-05T06:53:10.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.824+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.824+0000 7fc0667fc700 1 -- 172.21.15.16:0/1187992950 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc06c000e80 con 0x7fc070131d50 2019-12-05T06:53:10.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.824+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.824+0000 7fc0667fc700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc058023630 0x7fc058025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.824+0000 7fc0667fc700 1 -- 172.21.15.16:0/1187992950 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7fc06c045df0 con 0x7fc070131d50 2019-12-05T06:53:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.825+0000 7fc0753fc700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc058023630 0x7fc058025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:10.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.825+0000 7fc0753fc700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc058023630 0x7fc058025ac0 crc :-1 s=READY pgs=616 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:10.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.839+0000 7fc077660700 1 -- 172.21.15.16:0/1187992950 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc070134930 con 0x7fc070131d50 2019-12-05T06:53:10.847 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.847+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:10.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:10.847+0000 7fc0667fc700 1 -- 172.21.15.16:0/1187992950 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc06c0075d0 con 0x7fc070131d50 2019-12-05T06:53:11.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.042+0000 7fc077660700 1 -- 172.21.15.16:0/1187992950 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc070134930 con 0x7fc070131d50 2019-12-05T06:53:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.043+0000 7fc075bfd700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=THROTTLE_DONE pgs=345 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:11.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.043+0000 7fc0667fc700 1 -- 172.21.15.16:0/1187992950 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 178 v178) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fc06c000d70 con 0x7fc070131d50 2019-12-05T06:53:11.046 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 178 2019-12-05T06:53:11.046 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:11.046 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":178,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":178,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:08.580870+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 -- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc058023630 msgr2=0x7fc058025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc058023630 0x7fc058025ac0 crc :-1 s=READY pgs=616 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:11.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 -- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 msgr2=0x7fc070084320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 secure :-1 s=READY pgs=345 cs=0 l=1 rx=0x7fc06c0103e0 tx=0x7fc06c010780).stop 2019-12-05T06:53:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 -- 172.21.15.16:0/1187992950 shutdown_connections 2019-12-05T06:53:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc058023630 0x7fc058025ac0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 --2- 172.21.15.16:0/1187992950 >> v2:172.21.15.76:3300/0 conn(0x7fc070084ad0 0x7fc070135230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 --2- 172.21.15.16:0/1187992950 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc070131d50 0x7fc070084320 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 -- 172.21.15.16:0/1187992950 shutdown_connections 2019-12-05T06:53:11.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 -- 172.21.15.16:0/1187992950 wait complete. 2019-12-05T06:53:11.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:11.049+0000 7fc077660700 1 -- 172.21.15.16:0/1187992950 >> 172.21.15.16:0/1187992950 conn(0x7fc07009be40 msgr2=0x7fc07006fdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:11.060 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 178, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 178, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_7857": { "addr": "172.21.15.16:6833/359492061", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 359492061, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 359492061, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 7857, "incarnation": 178, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 79, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:53:08.580870+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 7857 } } } ], "standbys": [] } 2019-12-05T06:53:13.061 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:13.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.328+0000 7fa99f046700 1 Processor -- start 2019-12-05T06:53:13.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.328+0000 7fa99f046700 1 -- start start 2019-12-05T06:53:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.330+0000 7fa99f046700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa99808d2c0 0x7fa99808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.330+0000 7fa99f046700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa99808fef0 0x7fa99808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.330+0000 7fa99f046700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9980958e0 con 0x7fa99808c6d0 2019-12-05T06:53:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.330+0000 7fa99f046700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99808df90 con 0x7fa99808d2c0 2019-12-05T06:53:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.330+0000 7fa99f046700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9980a3f70 con 0x7fa99808fef0 2019-12-05T06:53:13.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.330+0000 7fa99cde2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa99808d2c0 0x7fa99808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.330+0000 7fa99cde2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa99808d2c0 0x7fa99808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34836/0 (socket says 172.21.15.16:34836) 2019-12-05T06:53:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.330+0000 7fa99cde2700 1 -- 172.21.15.16:0/2379933216 learned_addr learned my addr 172.21.15.16:0/2379933216 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.330+0000 7fa99d5e3700 1 --1- 172.21.15.16:0/2379933216 >> v1:172.21.15.16:6789/0 conn(0x7fa99808c6d0 0x7fa99808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48234/0 (socket says 172.21.15.16:48234) 2019-12-05T06:53:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.331+0000 7fa99dde4700 1 --2- 172.21.15.16:0/2379933216 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa99808fef0 0x7fa99808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.331+0000 7fa99cde2700 1 -- 172.21.15.16:0/2379933216 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa99808fef0 msgr2=0x7fa99808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:13.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.331+0000 7fa99cde2700 1 --2- 172.21.15.16:0/2379933216 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa99808fef0 0x7fa99808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.331+0000 7fa99cde2700 1 -- 172.21.15.16:0/2379933216 >> v1:172.21.15.16:6789/0 conn(0x7fa99808c6d0 legacy=0x7fa99808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.331+0000 7fa99cde2700 1 -- 172.21.15.16:0/2379933216 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99808e620 con 0x7fa99808d2c0 2019-12-05T06:53:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.331+0000 7fa99cde2700 1 --2- 172.21.15.16:0/2379933216 >> v2:172.21.15.16:3300/0 conn(0x7fa99808d2c0 0x7fa99808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa99400f740 tx=0x7fa99400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.331+0000 7fa99cde2700 1 --2- 172.21.15.16:0/2379933216 >> v2:172.21.15.16:3300/0 conn(0x7fa99808d2c0 0x7fa99808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa99400f740 tx=0x7fa99400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.331+0000 7fa99cde2700 1 --2- 172.21.15.16:0/2379933216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa99808d2c0 0x7fa99808ef20 secure :-1 s=READY pgs=1473 cs=0 l=1 rx=0x7fa99400f740 tx=0x7fa99400f570).ready entity=mon.0 client_cookie=6f06d564172d9509 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.332+0000 7fa99cde2700 1 --2- 172.21.15.16:0/2379933216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa99808d2c0 0x7fa99808ef20 secure :-1 s=THROTTLE_DONE pgs=1473 cs=0 l=1 rx=0x7fa99400f740 tx=0x7fa99400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.332+0000 7fa99cde2700 1 --2- 172.21.15.16:0/2379933216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa99808d2c0 0x7fa99808ef20 secure :-1 s=THROTTLE_DONE pgs=1473 cs=0 l=1 rx=0x7fa99400f740 tx=0x7fa99400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.332+0000 7fa99cde2700 1 --2- 172.21.15.16:0/2379933216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa99808d2c0 0x7fa99808ef20 secure :-1 s=THROTTLE_DONE pgs=1473 cs=0 l=1 rx=0x7fa99400f740 tx=0x7fa99400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.332+0000 7fa98bfff700 1 -- 172.21.15.16:0/2379933216 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa994000960 con 0x7fa99808d2c0 2019-12-05T06:53:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.332+0000 7fa98bfff700 1 -- 172.21.15.16:0/2379933216 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa994000c90 con 0x7fa99808d2c0 2019-12-05T06:53:13.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.332+0000 7fa98bfff700 1 -- 172.21.15.16:0/2379933216 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa9940012a0 con 0x7fa99808d2c0 2019-12-05T06:53:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.332+0000 7fa99f046700 1 -- 172.21.15.16:0/2379933216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa99808d2c0 msgr2=0x7fa99808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.332+0000 7fa99f046700 1 --2- 172.21.15.16:0/2379933216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa99808d2c0 0x7fa99808ef20 secure :-1 s=READY pgs=1473 cs=0 l=1 rx=0x7fa99400f740 tx=0x7fa99400f570).stop 2019-12-05T06:53:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.333+0000 7fa99f046700 1 -- 172.21.15.16:0/2379933216 shutdown_connections 2019-12-05T06:53:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.333+0000 7fa99f046700 1 --2- 172.21.15.16:0/2379933216 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa99808d2c0 0x7fa99808ef20 unknown :-1 s=CLOSED pgs=1473 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.333+0000 7fa99f046700 1 --2- 172.21.15.16:0/2379933216 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa99808fef0 0x7fa99808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:13.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.334+0000 7fa99f046700 1 -- 172.21.15.16:0/2379933216 shutdown_connections 2019-12-05T06:53:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.334+0000 7fa99f046700 1 -- 172.21.15.16:0/2379933216 wait complete. 2019-12-05T06:53:13.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.334+0000 7fa99f046700 1 -- 172.21.15.16:0/2379933216 >> 172.21.15.16:0/2379933216 conn(0x7fa998061940 msgr2=0x7fa998096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.339+0000 7fa99f046700 1 Processor -- start 2019-12-05T06:53:13.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.339+0000 7fa99f046700 1 -- start start 2019-12-05T06:53:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.340+0000 7fa99f046700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa998132d00 0x7fa99814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa99f046700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa998098da0 con 0x7fa99814d400 2019-12-05T06:53:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa99f046700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa99809f0a0 con 0x7fa998132530 2019-12-05T06:53:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa99f046700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa99809bb50 con 0x7fa998132d00 2019-12-05T06:53:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa99cde2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa998132d00 0x7fa99814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:13.347 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa99d5e3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa99814d400 0x7fa998131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48240/0 (socket says 172.21.15.16:48240) 2019-12-05T06:53:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa99d5e3700 1 -- 172.21.15.16:0/1110626988 learned_addr learned my addr 172.21.15.16:0/1110626988 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa99cde2700 1 -- 172.21.15.16:0/1110626988 >> v1:172.21.15.76:6789/0 conn(0x7fa998132530 legacy=0x7fa998157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa99cde2700 1 -- 172.21.15.16:0/1110626988 >> v1:172.21.15.16:6789/0 conn(0x7fa99814d400 legacy=0x7fa998131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa99cde2700 1 -- 172.21.15.16:0/1110626988 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa99809b0f0 con 0x7fa998132d00 2019-12-05T06:53:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa989ffb700 1 -- 172.21.15.16:0/1110626988 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4113876994 0 0) 0x7fa98c001150 con 0x7fa99814d400 2019-12-05T06:53:13.348 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.341+0000 7fa99cde2700 1 --2- 172.21.15.16:0/1110626988 >> v2:172.21.15.16:3300/0 conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.342+0000 7fa99cde2700 1 --2- 172.21.15.16:0/1110626988 >> v2:172.21.15.16:3300/0 conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.342+0000 7fa99cde2700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.342+0000 7fa99cde2700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=THROTTLE_DONE pgs=1475 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.342+0000 7fa989ffb700 1 -- 172.21.15.16:0/1110626988 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa994000f50 con 0x7fa998132d00 2019-12-05T06:53:13.349 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.342+0000 7fa99cde2700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=THROTTLE_DONE pgs=1475 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.342+0000 7fa99cde2700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=THROTTLE_DONE pgs=1475 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.342+0000 7fa989ffb700 1 -- 172.21.15.16:0/1110626988 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa994000940 con 0x7fa998132d00 2019-12-05T06:53:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.342+0000 7fa99f046700 1 -- 172.21.15.16:0/1110626988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99809b0f0 con 0x7fa998132d00 2019-12-05T06:53:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.342+0000 7fa989ffb700 1 -- 172.21.15.16:0/1110626988 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa994001530 con 0x7fa998132d00 2019-12-05T06:53:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.342+0000 7fa99f046700 1 -- 172.21.15.16:0/1110626988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99809b0f0 con 0x7fa998132d00 2019-12-05T06:53:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.343+0000 7fa99cde2700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=THROTTLE_DONE pgs=1475 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.344+0000 7fa99cde2700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=THROTTLE_DONE pgs=1475 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.344+0000 7fa989ffb700 1 -- 172.21.15.16:0/1110626988 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa994000d60 con 0x7fa998132d00 2019-12-05T06:53:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.344+0000 7fa989ffb700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa978023630 0x7fa978025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.344+0000 7fa989ffb700 1 -- 172.21.15.16:0/1110626988 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7fa994045db0 con 0x7fa998132d00 2019-12-05T06:53:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.344+0000 7fa99d5e3700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa978023630 0x7fa978025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:13.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.345+0000 7fa99d5e3700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa978023630 0x7fa978025ac0 crc :-1 s=READY pgs=617 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:13.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.359+0000 7fa99f046700 1 -- 172.21.15.16:0/1110626988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa99809b0f0 con 0x7fa998132d00 2019-12-05T06:53:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.366+0000 7fa99cde2700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=THROTTLE_DONE pgs=1475 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.366+0000 7fa989ffb700 1 -- 172.21.15.16:0/1110626988 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa994000b50 con 0x7fa998132d00 2019-12-05T06:53:13.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.590+0000 7fa99f046700 1 -- 172.21.15.16:0/1110626988 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa99809b0f0 con 0x7fa998132d00 2019-12-05T06:53:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.591+0000 7fa99cde2700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=THROTTLE_DONE pgs=1475 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:13.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.591+0000 7fa989ffb700 1 -- 172.21.15.16:0/1110626988 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 180 v180) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fa994000d60 con 0x7fa998132d00 2019-12-05T06:53:13.594 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:13.594 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":180,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":180,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:12.678018+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:13.594 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 180 2019-12-05T06:53:13.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.594+0000 7fa99f046700 1 -- 172.21.15.16:0/1110626988 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa978023630 msgr2=0x7fa978025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:13.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.594+0000 7fa99f046700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa978023630 0x7fa978025ac0 crc :-1 s=READY pgs=617 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:13.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.594+0000 7fa99f046700 1 -- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 msgr2=0x7fa99814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:13.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.594+0000 7fa99f046700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 0x7fa99814d7f0 secure :-1 s=READY pgs=1475 cs=0 l=1 rx=0x7fa994010390 tx=0x7fa994010730).stop 2019-12-05T06:53:13.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.594+0000 7fa99f046700 1 -- 172.21.15.16:0/1110626988 shutdown_connections 2019-12-05T06:53:13.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.594+0000 7fa99f046700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa978023630 0x7fa978025ac0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:13.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.594+0000 7fa99f046700 1 --2- 172.21.15.16:0/1110626988 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa998132d00 0x7fa99814d7f0 unknown :-1 s=CLOSED pgs=1475 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:13.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.594+0000 7fa99f046700 1 -- 172.21.15.16:0/1110626988 shutdown_connections 2019-12-05T06:53:13.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.594+0000 7fa99f046700 1 -- 172.21.15.16:0/1110626988 wait complete. 2019-12-05T06:53:13.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:13.594+0000 7fa99f046700 1 -- 172.21.15.16:0/1110626988 >> 172.21.15.16:0/1110626988 conn(0x7fa9980714f0 msgr2=0x7fa99809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:13.605 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:53:15.606 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:15.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.848+0000 7f5c3e2c4700 1 Processor -- start 2019-12-05T06:53:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.848+0000 7f5c3e2c4700 1 -- start start 2019-12-05T06:53:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.849+0000 7f5c3e2c4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5c3808c6d0 0x7f5c3808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.849+0000 7f5c3e2c4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5c3808f720 0x7f5c3808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.849+0000 7f5c3e2c4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3808fb70 con 0x7f5c3808fef0 2019-12-05T06:53:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.849+0000 7f5c3e2c4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c3808df90 con 0x7f5c3808c6d0 2019-12-05T06:53:15.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.849+0000 7f5c3e2c4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c380a3f70 con 0x7f5c3808f720 2019-12-05T06:53:15.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.850+0000 7f5c3c861700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5c3808c6d0 0x7f5c3808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:15.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.850+0000 7f5c3d062700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5c3808f720 0x7f5c3808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:15.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.850+0000 7f5c3c861700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5c3808c6d0 0x7f5c3808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:49186/0 (socket says 172.21.15.16:49186) 2019-12-05T06:53:15.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.850+0000 7f5c3c861700 1 -- 172.21.15.16:0/1155211923 learned_addr learned my addr 172.21.15.16:0/1155211923 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:15.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.850+0000 7f5c3c861700 1 -- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5c3808f720 msgr2=0x7f5c3808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.850+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5c3808f720 0x7f5c3808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.850+0000 7f5c3c861700 1 -- 172.21.15.16:0/1155211923 >> v1:172.21.15.76:6789/0 conn(0x7f5c3808fef0 legacy=0x7f5c3808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.850+0000 7f5c3c861700 1 -- 172.21.15.16:0/1155211923 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3808d540 con 0x7f5c3808c6d0 2019-12-05T06:53:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.850+0000 7f5c3d062700 1 --2- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5c3808f720 0x7f5c3808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:53:15.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1155211923 >> v2:172.21.15.76:3300/0 conn(0x7f5c3808c6d0 0x7f5c3808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5c2000f740 tx=0x7f5c2000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1155211923 >> v2:172.21.15.76:3300/0 conn(0x7f5c3808c6d0 0x7f5c3808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5c2000f740 tx=0x7f5c2000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c3808c6d0 0x7f5c3808cac0 secure :-1 s=READY pgs=994 cs=0 l=1 rx=0x7f5c2000f740 tx=0x7f5c2000fae0).ready entity=mon.1 client_cookie=a7e1b39a8628b38e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c3808c6d0 0x7f5c3808cac0 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7f5c2000f740 tx=0x7f5c2000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c377fe700 1 -- 172.21.15.16:0/1155211923 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c20000960 con 0x7f5c3808c6d0 2019-12-05T06:53:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c3808c6d0 0x7f5c3808cac0 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7f5c2000f740 tx=0x7f5c2000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c3808c6d0 0x7f5c3808cac0 secure :-1 s=THROTTLE_DONE pgs=994 cs=0 l=1 rx=0x7f5c2000f740 tx=0x7f5c2000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c377fe700 1 -- 172.21.15.16:0/1155211923 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c20000c90 con 0x7f5c3808c6d0 2019-12-05T06:53:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c377fe700 1 -- 172.21.15.16:0/1155211923 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c200013a0 con 0x7f5c3808c6d0 2019-12-05T06:53:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c3808c6d0 msgr2=0x7f5c3808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.851+0000 7f5c3e2c4700 1 --2- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c3808c6d0 0x7f5c3808cac0 secure :-1 s=READY pgs=994 cs=0 l=1 rx=0x7f5c2000f740 tx=0x7f5c2000fae0).stop 2019-12-05T06:53:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.853+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1155211923 shutdown_connections 2019-12-05T06:53:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.853+0000 7f5c3e2c4700 1 --2- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5c3808f720 0x7f5c3808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:15.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.853+0000 7f5c3e2c4700 1 --2- 172.21.15.16:0/1155211923 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c3808c6d0 0x7f5c3808cac0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.853+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1155211923 shutdown_connections 2019-12-05T06:53:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.853+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1155211923 wait complete. 2019-12-05T06:53:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.853+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1155211923 >> 172.21.15.16:0/1155211923 conn(0x7f5c38061940 msgr2=0x7f5c38096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.856+0000 7f5c3e2c4700 1 Processor -- start 2019-12-05T06:53:15.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.856+0000 7f5c3e2c4700 1 -- start start 2019-12-05T06:53:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.857+0000 7f5c3e2c4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5c3812ae70 0x7f5c38157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.857+0000 7f5c3e2c4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c3812b2d0 con 0x7f5c381302f0 2019-12-05T06:53:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.857+0000 7f5c3e2c4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c38131bf0 con 0x7f5c3812b640 2019-12-05T06:53:15.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.857+0000 7f5c3e2c4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c381296e0 con 0x7f5c3812ae70 2019-12-05T06:53:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.857+0000 7f5c3c861700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5c381302f0 0x7f5c3812a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48270/0 (socket says 172.21.15.16:48270) 2019-12-05T06:53:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.857+0000 7f5c3c861700 1 -- 172.21.15.16:0/1668111775 learned_addr learned my addr 172.21.15.16:0/1668111775 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.857+0000 7f5c3d062700 1 --2- 172.21.15.16:0/1668111775 >> v2:172.21.15.76:3300/0 conn(0x7f5c3812ae70 0x7f5c38157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.857+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2798796069 0 0) 0x7f5c20007e10 con 0x7f5c381302f0 2019-12-05T06:53:15.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.857+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c1c002e80 con 0x7f5c381302f0 2019-12-05T06:53:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3318582647 0 0) 0x7f5c200079e0 con 0x7f5c381302f0 2019-12-05T06:53:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5c1c0032e0 con 0x7f5c381302f0 2019-12-05T06:53:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f5c20000940 con 0x7f5c381302f0 2019-12-05T06:53:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 >> v2:172.21.15.76:3300/0 conn(0x7f5c3812ae70 msgr2=0x7f5c38157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 --2- 172.21.15.16:0/1668111775 >> v2:172.21.15.76:3300/0 conn(0x7f5c3812ae70 0x7f5c38157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 >> v1:172.21.15.76:6789/0 conn(0x7f5c3812b640 legacy=0x7f5c38153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:15.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 >> v1:172.21.15.16:6789/0 conn(0x7f5c381302f0 legacy=0x7f5c3812a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:15.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:15.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5c1c003cd0 0x7f5c1c004170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:15.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c1c004930 0x7f5c1c004df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:15.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c00bbf0 con 0x7f5c1c003380 2019-12-05T06:53:15.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c00be20 con 0x7f5c1c004930 2019-12-05T06:53:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c1c00c050 con 0x7f5c1c003cd0 2019-12-05T06:53:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3d062700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c1c004930 0x7f5c1c004df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:15.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c37fff700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5c1c003cd0 0x7f5c1c004170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3c861700 1 -- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5c1c003cd0 msgr2=0x7f5c1c004170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5c1c003cd0 0x7f5c1c004170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3c861700 1 -- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c1c004930 msgr2=0x7f5c1c004df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5c1c004930 0x7f5c1c004df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3c861700 1 -- 172.21.15.16:0/1668111775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c1c00c300 con 0x7f5c1c003380 2019-12-05T06:53:15.885 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3d062700 1 -- 172.21.15.16:0/1668111775 reap_dead start 2019-12-05T06:53:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1668111775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c38129d70 con 0x7f5c1c003380 2019-12-05T06:53:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.858+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1668111775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c3812b2d0 con 0x7f5c1c003380 2019-12-05T06:53:15.886 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.859+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.859+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=READY pgs=1480 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.859+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=THROTTLE_DONE pgs=1480 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.859+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c20005580 con 0x7f5c1c003380 2019-12-05T06:53:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.859+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=THROTTLE_DONE pgs=1480 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.887 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.859+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=THROTTLE_DONE pgs=1480 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.859+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c200075d0 con 0x7f5c1c003380 2019-12-05T06:53:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.859+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c20005770 con 0x7f5c1c003380 2019-12-05T06:53:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.859+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=THROTTLE_DONE pgs=1480 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.860+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5c20005390 con 0x7f5c1c003380 2019-12-05T06:53:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.860+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=THROTTLE_DONE pgs=1480 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.888 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.860+0000 7f5c357fa700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5c1c0263b0 0x7f5c1c0287e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:15.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.860+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f5c20045ce0 con 0x7f5c1c003380 2019-12-05T06:53:15.889 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.860+0000 7f5c37fff700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5c1c0263b0 0x7f5c1c0287e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:15.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.861+0000 7f5c37fff700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5c1c0263b0 0x7f5c1c0287e0 crc :-1 s=READY pgs=618 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:15.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.864+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1668111775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c10002e60 con 0x7f5c1c003380 2019-12-05T06:53:15.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.868+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=THROTTLE_DONE pgs=1480 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:15.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:15.868+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5c20027f00 con 0x7f5c1c003380 2019-12-05T06:53:16.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.067+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1668111775 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5c10001d10 con 0x7f5c1c003380 2019-12-05T06:53:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.068+0000 7f5c3c861700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=THROTTLE_DONE pgs=1480 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:16.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.068+0000 7f5c357fa700 1 -- 172.21.15.16:0/1668111775 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f5c20027f00 con 0x7f5c1c003380 2019-12-05T06:53:16.069 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 181 2019-12-05T06:53:16.069 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:16.069 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:16.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.071+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5c1c0263b0 msgr2=0x7f5c1c0287e0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:16.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.071+0000 7f5c3e2c4700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5c1c0263b0 0x7f5c1c0287e0 crc :-1 s=READY pgs=618 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:16.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.071+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 msgr2=0x7f5c1c003790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.071+0000 7f5c3e2c4700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 secure :-1 s=READY pgs=1480 cs=0 l=1 rx=0x7f5c200103e0 tx=0x7f5c20010290).stop 2019-12-05T06:53:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.071+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1668111775 shutdown_connections 2019-12-05T06:53:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.071+0000 7f5c3e2c4700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5c1c0263b0 0x7f5c1c0287e0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.071+0000 7f5c3e2c4700 1 --2- 172.21.15.16:0/1668111775 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5c1c003380 0x7f5c1c003790 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.071+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1668111775 shutdown_connections 2019-12-05T06:53:16.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.071+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1668111775 wait complete. 2019-12-05T06:53:16.073 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:16.071+0000 7f5c3e2c4700 1 -- 172.21.15.16:0/1668111775 >> 172.21.15.16:0/1668111775 conn(0x7f5c380a2850 msgr2=0x7f5c3806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:16.092 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=5972 2019-12-05T06:53:16.093 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 3 secs before reviving mds.b 2019-12-05T06:53:16.114 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:53:16.120 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:53:19.093 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:53:19.093 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:53:19.094 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:53:19.099 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:53:19.099 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:19.269 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:53:19.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.327+0000 7fba071f6700 1 Processor -- start 2019-12-05T06:53:19.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.327+0000 7fba071f6700 1 -- start start 2019-12-05T06:53:19.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba071f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba0008c6d0 0x7fba0008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:19.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba071f6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba0008fef0 0x7fba0008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:19.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba071f6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fba0008ef20 0x7fba0008f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:19.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba071f6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0008df90 con 0x7fba0008c6d0 2019-12-05T06:53:19.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba071f6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba000a3f70 con 0x7fba0008ef20 2019-12-05T06:53:19.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba071f6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fba0008d830 con 0x7fba0008fef0 2019-12-05T06:53:19.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba05793700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba0008c6d0 0x7fba0008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:19.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba05793700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba0008c6d0 0x7fba0008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34902/0 (socket says 172.21.15.16:34902) 2019-12-05T06:53:19.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba05793700 1 -- 172.21.15.16:0/3224159243 learned_addr learned my addr 172.21.15.16:0/3224159243 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:19.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba04f92700 1 --2- 172.21.15.16:0/3224159243 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba0008fef0 0x7fba0008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:19.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.329+0000 7fba05f94700 1 --2- 172.21.15.16:0/3224159243 >> v2:172.21.15.76:3300/0 conn(0x7fba0008ef20 0x7fba0008f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:19.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 -- 172.21.15.16:0/3224159243 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba0008fef0 msgr2=0x7fba0008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:19.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 --2- 172.21.15.16:0/3224159243 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba0008fef0 0x7fba0008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:19.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 -- 172.21.15.16:0/3224159243 >> v2:172.21.15.76:3300/0 conn(0x7fba0008ef20 msgr2=0x7fba0008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:19.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 --2- 172.21.15.16:0/3224159243 >> v2:172.21.15.76:3300/0 conn(0x7fba0008ef20 0x7fba0008f3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:19.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 -- 172.21.15.16:0/3224159243 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba0008d540 con 0x7fba0008c6d0 2019-12-05T06:53:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 --2- 172.21.15.16:0/3224159243 >> v2:172.21.15.16:3300/0 conn(0x7fba0008c6d0 0x7fba0008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb9f800fa40 tx=0x7fb9f800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 --2- 172.21.15.16:0/3224159243 >> v2:172.21.15.16:3300/0 conn(0x7fba0008c6d0 0x7fba0008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb9f800fa40 tx=0x7fb9f800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 --2- 172.21.15.16:0/3224159243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0008c6d0 0x7fba0008cac0 secure :-1 s=READY pgs=1483 cs=0 l=1 rx=0x7fb9f800fa40 tx=0x7fb9f800f8f0).ready entity=mon.0 client_cookie=a53fdfb3f3f27cba server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 --2- 172.21.15.16:0/3224159243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0008c6d0 0x7fba0008cac0 secure :-1 s=THROTTLE_DONE pgs=1483 cs=0 l=1 rx=0x7fb9f800fa40 tx=0x7fb9f800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 --2- 172.21.15.16:0/3224159243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0008c6d0 0x7fba0008cac0 secure :-1 s=THROTTLE_DONE pgs=1483 cs=0 l=1 rx=0x7fb9f800fa40 tx=0x7fb9f800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba05793700 1 --2- 172.21.15.16:0/3224159243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0008c6d0 0x7fba0008cac0 secure :-1 s=THROTTLE_DONE pgs=1483 cs=0 l=1 rx=0x7fb9f800fa40 tx=0x7fb9f800f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fb9fffff700 1 -- 172.21.15.16:0/3224159243 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9f8000ce0 con 0x7fba0008c6d0 2019-12-05T06:53:19.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fb9fffff700 1 -- 172.21.15.16:0/3224159243 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f8001010 con 0x7fba0008c6d0 2019-12-05T06:53:19.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fb9fffff700 1 -- 172.21.15.16:0/3224159243 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9f8001720 con 0x7fba0008c6d0 2019-12-05T06:53:19.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba071f6700 1 -- 172.21.15.16:0/3224159243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0008c6d0 msgr2=0x7fba0008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:19.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.330+0000 7fba071f6700 1 --2- 172.21.15.16:0/3224159243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0008c6d0 0x7fba0008cac0 secure :-1 s=READY pgs=1483 cs=0 l=1 rx=0x7fb9f800fa40 tx=0x7fb9f800f8f0).stop 2019-12-05T06:53:19.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.332+0000 7fba071f6700 1 -- 172.21.15.16:0/3224159243 shutdown_connections 2019-12-05T06:53:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.332+0000 7fba071f6700 1 --2- 172.21.15.16:0/3224159243 >> v2:172.21.15.76:3300/0 conn(0x7fba0008ef20 0x7fba0008f3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.332+0000 7fba071f6700 1 --2- 172.21.15.16:0/3224159243 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fba0008fef0 0x7fba0008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:19.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.332+0000 7fba071f6700 1 --2- 172.21.15.16:0/3224159243 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba0008c6d0 0x7fba0008cac0 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:19.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.332+0000 7fba071f6700 1 -- 172.21.15.16:0/3224159243 shutdown_connections 2019-12-05T06:53:19.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.332+0000 7fba071f6700 1 -- 172.21.15.16:0/3224159243 wait complete. 2019-12-05T06:53:19.346 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.332+0000 7fba071f6700 1 -- 172.21.15.16:0/3224159243 >> 172.21.15.16:0/3224159243 conn(0x7fba00061940 msgr2=0x7fba00096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.336+0000 7fba071f6700 1 Processor -- start 2019-12-05T06:53:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.336+0000 7fba071f6700 1 -- start start 2019-12-05T06:53:19.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.338+0000 7fba071f6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fba0006df10 0x7fba00098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.338+0000 7fba071f6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba000992d0 0x7fba00098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:19.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.338+0000 7fba071f6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba0006e300 con 0x7fba00098d80 2019-12-05T06:53:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.338+0000 7fba071f6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba00097370 con 0x7fba000992d0 2019-12-05T06:53:19.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.338+0000 7fba071f6700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fba0008ad00 con 0x7fba0006df10 2019-12-05T06:53:19.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.338+0000 7fba04f92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba000992d0 0x7fba00098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:19.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.338+0000 7fba04f92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fba000992d0 0x7fba00098840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34910/0 (socket says 172.21.15.16:34910) 2019-12-05T06:53:19.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.338+0000 7fba05f94700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fba00098d80 0x7fba00088fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48312/0 (socket says 172.21.15.16:48312) 2019-12-05T06:53:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.338+0000 7fba04f92700 1 -- 172.21.15.16:0/2799139473 learned_addr learned my addr 172.21.15.16:0/2799139473 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.338+0000 7fba05793700 1 --2- 172.21.15.16:0/2799139473 >> v2:172.21.15.76:3300/0 conn(0x7fba0006df10 0x7fba00098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba04f92700 1 -- 172.21.15.16:0/2799139473 >> v2:172.21.15.76:3300/0 conn(0x7fba0006df10 msgr2=0x7fba00098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> v2:172.21.15.76:3300/0 conn(0x7fba0006df10 0x7fba00098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:19.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba04f92700 1 -- 172.21.15.16:0/2799139473 >> v1:172.21.15.16:6789/0 conn(0x7fba00098d80 legacy=0x7fba00088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba04f92700 1 -- 172.21.15.16:0/2799139473 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba00097a00 con 0x7fba000992d0 2019-12-05T06:53:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fb9fdffb700 1 -- 172.21.15.16:0/2799139473 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4282396340 0 0) 0x7fb9f0001080 con 0x7fba00098d80 2019-12-05T06:53:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> v2:172.21.15.16:3300/0 conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> v2:172.21.15.16:3300/0 conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=READY pgs=1485 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fb9fdffb700 1 -- 172.21.15.16:0/2799139473 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9f4000940 con 0x7fba000992d0 2019-12-05T06:53:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba071f6700 1 -- 172.21.15.16:0/2799139473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba00097a00 con 0x7fba000992d0 2019-12-05T06:53:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fb9fdffb700 1 -- 172.21.15.16:0/2799139473 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb9f4000940 con 0x7fba000992d0 2019-12-05T06:53:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.359 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.339+0000 7fb9fdffb700 1 -- 172.21.15.16:0/2799139473 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb9f4001070 con 0x7fba000992d0 2019-12-05T06:53:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.340+0000 7fba071f6700 1 -- 172.21.15.16:0/2799139473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba00097a00 con 0x7fba000992d0 2019-12-05T06:53:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.341+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.341+0000 7fb9fdffb700 1 -- 172.21.15.16:0/2799139473 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb9f4000d30 con 0x7fba000992d0 2019-12-05T06:53:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.341+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.342+0000 7fb9fdffb700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9e0023630 0x7fb9e0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:19.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.342+0000 7fb9fdffb700 1 -- 172.21.15.16:0/2799139473 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7fb9f4045de0 con 0x7fba000992d0 2019-12-05T06:53:19.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.342+0000 7fba05793700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9e0023630 0x7fb9e0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:19.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.343+0000 7fba05793700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9e0023630 0x7fb9e0025ac0 crc :-1 s=READY pgs=619 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:19.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.343+0000 7fba071f6700 1 -- 172.21.15.16:0/2799139473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba00097a00 con 0x7fba000992d0 2019-12-05T06:53:19.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.351+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.352+0000 7fb9fdffb700 1 -- 172.21.15.16:0/2799139473 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb9f4009250 con 0x7fba000992d0 2019-12-05T06:53:19.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.609+0000 7fba071f6700 1 -- 172.21.15.16:0/2799139473 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fba00097a00 con 0x7fba000992d0 2019-12-05T06:53:19.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.610+0000 7fba04f92700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=THROTTLE_DONE pgs=1485 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:19.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.610+0000 7fb9fdffb700 1 -- 172.21.15.16:0/2799139473 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+1772 (secure 0 0 0) 0x7fb9f4000c90 con 0x7fba000992d0 2019-12-05T06:53:19.612 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 181 2019-12-05T06:53:19.612 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:19.612 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:19.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.612+0000 7fba071f6700 1 -- 172.21.15.16:0/2799139473 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9e0023630 msgr2=0x7fb9e0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:19.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.612+0000 7fba071f6700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9e0023630 0x7fb9e0025ac0 crc :-1 s=READY pgs=619 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:19.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.612+0000 7fba071f6700 1 -- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 msgr2=0x7fba00098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:19.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.612+0000 7fba071f6700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 0x7fba00098840 secure :-1 s=READY pgs=1485 cs=0 l=1 rx=0x7fb9f40103d0 tx=0x7fb9f4010770).stop 2019-12-05T06:53:19.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.613+0000 7fba071f6700 1 -- 172.21.15.16:0/2799139473 shutdown_connections 2019-12-05T06:53:19.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.613+0000 7fba071f6700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb9e0023630 0x7fb9e0025ac0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:19.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.613+0000 7fba071f6700 1 --2- 172.21.15.16:0/2799139473 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fba000992d0 0x7fba00098840 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:19.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.613+0000 7fba071f6700 1 --2- 172.21.15.16:0/2799139473 >> v2:172.21.15.76:3300/0 conn(0x7fba0006df10 0x7fba00098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:19.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.613+0000 7fba071f6700 1 -- 172.21.15.16:0/2799139473 shutdown_connections 2019-12-05T06:53:19.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.613+0000 7fba071f6700 1 -- 172.21.15.16:0/2799139473 wait complete. 2019-12-05T06:53:19.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:19.613+0000 7fba071f6700 1 -- 172.21.15.16:0/2799139473 >> 172.21.15.16:0/2799139473 conn(0x7fba000714f0 msgr2=0x7fba0009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:19.622 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:53:21.623 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:21.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.851+0000 7ff80ae9b700 1 Processor -- start 2019-12-05T06:53:21.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.852+0000 7ff80ae9b700 1 -- start start 2019-12-05T06:53:21.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.854+0000 7ff80ae9b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:21.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.854+0000 7ff80ae9b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff80408fef0 0x7ff80408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:21.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.854+0000 7ff80ae9b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8040958e0 con 0x7ff80408c6d0 2019-12-05T06:53:21.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.854+0000 7ff80ae9b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff80408df90 con 0x7ff80408fef0 2019-12-05T06:53:21.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.854+0000 7ff80ae9b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8040a3f70 con 0x7ff80408d2c0 2019-12-05T06:53:21.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.854+0000 7ff808c37700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:21.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.854+0000 7ff809c39700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff80408fef0 0x7ff80408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:21.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.854+0000 7ff809438700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7ff80408c6d0 0x7ff80408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:41388/0 (socket says 172.21.15.16:41388) 2019-12-05T06:53:21.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.854+0000 7ff808c37700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:38910/0 (socket says 172.21.15.16:38910) 2019-12-05T06:53:21.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.854+0000 7ff809438700 1 -- 172.21.15.16:0/928154097 learned_addr learned my addr 172.21.15.16:0/928154097 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:21.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.855+0000 7ff808c37700 1 -- 172.21.15.16:0/928154097 >> v2:172.21.15.76:3300/0 conn(0x7ff80408fef0 msgr2=0x7ff80408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:21.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.855+0000 7ff808c37700 1 --2- 172.21.15.16:0/928154097 >> v2:172.21.15.76:3300/0 conn(0x7ff80408fef0 0x7ff80408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:21.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.855+0000 7ff808c37700 1 -- 172.21.15.16:0/928154097 >> v1:172.21.15.76:6789/0 conn(0x7ff80408c6d0 legacy=0x7ff80408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:21.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.855+0000 7ff808c37700 1 -- 172.21.15.16:0/928154097 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff80408e620 con 0x7ff80408d2c0 2019-12-05T06:53:21.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.855+0000 7ff7fbfff700 1 -- 172.21.15.16:0/928154097 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204798578 0 0) 0x7ff800001100 con 0x7ff80408c6d0 2019-12-05T06:53:21.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.855+0000 7ff809c39700 1 --2- 172.21.15.16:0/928154097 >> v2:172.21.15.76:3300/0 conn(0x7ff80408fef0 0x7ff80408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:53:21.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.855+0000 7ff808c37700 1 --2- 172.21.15.16:0/928154097 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff7f400af50 tx=0x7ff7f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.855+0000 7ff808c37700 1 --2- 172.21.15.16:0/928154097 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff7f400af50 tx=0x7ff7f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.856+0000 7ff808c37700 1 --2- 172.21.15.16:0/928154097 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 secure :-1 s=READY pgs=347 cs=0 l=1 rx=0x7ff7f400af50 tx=0x7ff7f400b760).ready entity=mon.2 client_cookie=d41e46c84efc7987 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:21.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.856+0000 7ff808c37700 1 --2- 172.21.15.16:0/928154097 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7ff7f400af50 tx=0x7ff7f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.856+0000 7ff808c37700 1 --2- 172.21.15.16:0/928154097 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7ff7f400af50 tx=0x7ff7f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.856+0000 7ff808c37700 1 --2- 172.21.15.16:0/928154097 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 secure :-1 s=THROTTLE_DONE pgs=347 cs=0 l=1 rx=0x7ff7f400af50 tx=0x7ff7f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.856+0000 7ff7fbfff700 1 -- 172.21.15.16:0/928154097 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7f4007210 con 0x7ff80408d2c0 2019-12-05T06:53:21.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.856+0000 7ff7fbfff700 1 -- 172.21.15.16:0/928154097 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7f40099b0 con 0x7ff80408d2c0 2019-12-05T06:53:21.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.856+0000 7ff7fbfff700 1 -- 172.21.15.16:0/928154097 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7f4009f80 con 0x7ff80408d2c0 2019-12-05T06:53:21.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.856+0000 7ff80ae9b700 1 -- 172.21.15.16:0/928154097 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 msgr2=0x7ff80408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:21.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.856+0000 7ff80ae9b700 1 --2- 172.21.15.16:0/928154097 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 secure :-1 s=READY pgs=347 cs=0 l=1 rx=0x7ff7f400af50 tx=0x7ff7f400b760).stop 2019-12-05T06:53:21.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.858+0000 7ff80ae9b700 1 -- 172.21.15.16:0/928154097 shutdown_connections 2019-12-05T06:53:21.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.858+0000 7ff80ae9b700 1 --2- 172.21.15.16:0/928154097 >> v2:172.21.15.76:3300/0 conn(0x7ff80408fef0 0x7ff80408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:21.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.858+0000 7ff80ae9b700 1 --2- 172.21.15.16:0/928154097 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff80408d2c0 0x7ff80408ef20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:21.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.858+0000 7ff80ae9b700 1 -- 172.21.15.16:0/928154097 shutdown_connections 2019-12-05T06:53:21.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.858+0000 7ff80ae9b700 1 -- 172.21.15.16:0/928154097 wait complete. 2019-12-05T06:53:21.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.858+0000 7ff80ae9b700 1 -- 172.21.15.16:0/928154097 >> 172.21.15.16:0/928154097 conn(0x7ff804061940 msgr2=0x7ff804096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:21.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.864+0000 7ff80ae9b700 1 Processor -- start 2019-12-05T06:53:21.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.864+0000 7ff80ae9b700 1 -- start start 2019-12-05T06:53:21.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.866+0000 7ff80ae9b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff80412b640 0x7ff80412ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:21.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.866+0000 7ff80ae9b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff804153b50 0x7ff804155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:21.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.866+0000 7ff80ae9b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff804131bf0 con 0x7ff8041302f0 2019-12-05T06:53:21.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.866+0000 7ff80ae9b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8041296e0 con 0x7ff804153b50 2019-12-05T06:53:21.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.866+0000 7ff80ae9b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff804128cb0 con 0x7ff80412b640 2019-12-05T06:53:21.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.866+0000 7ff809c39700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff804153b50 0x7ff804155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:21.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.867+0000 7ff809c39700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff804153b50 0x7ff804155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34928/0 (socket says 172.21.15.16:34928) 2019-12-05T06:53:21.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.867+0000 7ff809c39700 1 -- 172.21.15.16:0/4293065663 learned_addr learned my addr 172.21.15.16:0/4293065663 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:21.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.867+0000 7ff808c37700 1 --2- 172.21.15.16:0/4293065663 >> v2:172.21.15.76:3300/0 conn(0x7ff80412b640 0x7ff80412ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:21.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.867+0000 7ff809c39700 1 -- 172.21.15.16:0/4293065663 >> v2:172.21.15.76:3300/0 conn(0x7ff80412b640 msgr2=0x7ff80412ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:21.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.867+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> v2:172.21.15.76:3300/0 conn(0x7ff80412b640 0x7ff80412ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:21.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.867+0000 7ff809c39700 1 -- 172.21.15.16:0/4293065663 >> v1:172.21.15.76:6789/0 conn(0x7ff8041302f0 legacy=0x7ff80412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:21.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.867+0000 7ff809c39700 1 -- 172.21.15.16:0/4293065663 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff804129d70 con 0x7ff804153b50 2019-12-05T06:53:21.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.867+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> v2:172.21.15.16:3300/0 conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.867+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> v2:172.21.15.16:3300/0 conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.868+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=READY pgs=1486 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:21.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.868+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=THROTTLE_DONE pgs=1486 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.868+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=THROTTLE_DONE pgs=1486 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.868+0000 7ff7f9ffb700 1 -- 172.21.15.16:0/4293065663 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7fc00aad0 con 0x7ff804153b50 2019-12-05T06:53:21.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.868+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=THROTTLE_DONE pgs=1486 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.868+0000 7ff7f9ffb700 1 -- 172.21.15.16:0/4293065663 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff7fc009930 con 0x7ff804153b50 2019-12-05T06:53:21.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.868+0000 7ff80ae9b700 1 -- 172.21.15.16:0/4293065663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff804129d70 con 0x7ff804153b50 2019-12-05T06:53:21.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.868+0000 7ff7f9ffb700 1 -- 172.21.15.16:0/4293065663 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff7fc009fc0 con 0x7ff804153b50 2019-12-05T06:53:21.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.868+0000 7ff80ae9b700 1 -- 172.21.15.16:0/4293065663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff804129d70 con 0x7ff804153b50 2019-12-05T06:53:21.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.869+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=THROTTLE_DONE pgs=1486 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.870+0000 7ff7f9ffb700 1 -- 172.21.15.16:0/4293065663 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff7fc00ade0 con 0x7ff804153b50 2019-12-05T06:53:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.870+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=THROTTLE_DONE pgs=1486 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.871+0000 7ff7f9ffb700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff7e4023630 0x7ff7e4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.871+0000 7ff7f9ffb700 1 -- 172.21.15.16:0/4293065663 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7ff7fc046180 con 0x7ff804153b50 2019-12-05T06:53:21.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.871+0000 7ff809438700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff7e4023630 0x7ff7e4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:21.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.872+0000 7ff809438700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff7e4023630 0x7ff7e4025ac0 crc :-1 s=READY pgs=620 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:21.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.882+0000 7ff80ae9b700 1 -- 172.21.15.16:0/4293065663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff804129d70 con 0x7ff804153b50 2019-12-05T06:53:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.891+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=THROTTLE_DONE pgs=1486 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:21.891 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:21.891+0000 7ff7f9ffb700 1 -- 172.21.15.16:0/4293065663 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff7fc000df0 con 0x7ff804153b50 2019-12-05T06:53:22.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.133+0000 7ff80ae9b700 1 -- 172.21.15.16:0/4293065663 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff804129d70 con 0x7ff804153b50 2019-12-05T06:53:22.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.134+0000 7ff809c39700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=THROTTLE_DONE pgs=1486 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:22.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.134+0000 7ff7f9ffb700 1 -- 172.21.15.16:0/4293065663 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 181 v181) v1 ==== 95+0+1772 (secure 0 0 0) 0x7ff7fc00ade0 con 0x7ff804153b50 2019-12-05T06:53:22.134 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 181 2019-12-05T06:53:22.134 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:22.135 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":181,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:22.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.136+0000 7ff80ae9b700 1 -- 172.21.15.16:0/4293065663 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff7e4023630 msgr2=0x7ff7e4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:22.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.136+0000 7ff80ae9b700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff7e4023630 0x7ff7e4025ac0 crc :-1 s=READY pgs=620 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:22.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.136+0000 7ff80ae9b700 1 -- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 msgr2=0x7ff804155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:22.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.136+0000 7ff80ae9b700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 0x7ff804155f50 secure :-1 s=READY pgs=1486 cs=0 l=1 rx=0x7ff7fc0107b0 tx=0x7ff7fc010630).stop 2019-12-05T06:53:22.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.137+0000 7ff80ae9b700 1 -- 172.21.15.16:0/4293065663 shutdown_connections 2019-12-05T06:53:22.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.137+0000 7ff80ae9b700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff7e4023630 0x7ff7e4025ac0 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:22.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.137+0000 7ff80ae9b700 1 --2- 172.21.15.16:0/4293065663 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff804153b50 0x7ff804155f50 unknown :-1 s=CLOSED pgs=1486 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.137+0000 7ff80ae9b700 1 --2- 172.21.15.16:0/4293065663 >> v2:172.21.15.76:3300/0 conn(0x7ff80412b640 0x7ff80412ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.137+0000 7ff80ae9b700 1 -- 172.21.15.16:0/4293065663 shutdown_connections 2019-12-05T06:53:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.137+0000 7ff80ae9b700 1 -- 172.21.15.16:0/4293065663 wait complete. 2019-12-05T06:53:22.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:22.137+0000 7ff80ae9b700 1 -- 172.21.15.16:0/4293065663 >> 172.21.15.16:0/4293065663 conn(0x7ff8040a2850 msgr2=0x7ff80406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:22.147 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:53:24.148 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.395+0000 7f7cc30a7700 1 Processor -- start 2019-12-05T06:53:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.396+0000 7f7cc30a7700 1 -- start start 2019-12-05T06:53:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc30a7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7cbc08c6d0 0x7f7cbc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc30a7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7cbc08fef0 0x7f7cbc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:24.401 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc30a7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc30a7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cbc08df90 con 0x7f7cbc08ef20 2019-12-05T06:53:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc30a7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cbc0a3f70 con 0x7f7cbc08c6d0 2019-12-05T06:53:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc30a7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cbc08d830 con 0x7f7cbc08fef0 2019-12-05T06:53:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34946/0 (socket says 172.21.15.16:34946) 2019-12-05T06:53:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 -- 172.21.15.16:0/1821052930 learned_addr learned my addr 172.21.15.16:0/1821052930 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:24.402 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc0e43700 1 --2- 172.21.15.16:0/1821052930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7cbc08fef0 0x7f7cbc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1644700 1 --2- 172.21.15.16:0/1821052930 >> v2:172.21.15.76:3300/0 conn(0x7f7cbc08c6d0 0x7f7cbc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 -- 172.21.15.16:0/1821052930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7cbc08fef0 msgr2=0x7f7cbc08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 --2- 172.21.15.16:0/1821052930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7cbc08fef0 0x7f7cbc08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 -- 172.21.15.16:0/1821052930 >> v2:172.21.15.76:3300/0 conn(0x7f7cbc08c6d0 msgr2=0x7f7cbc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 --2- 172.21.15.16:0/1821052930 >> v2:172.21.15.76:3300/0 conn(0x7f7cbc08c6d0 0x7f7cbc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 -- 172.21.15.16:0/1821052930 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc08d540 con 0x7f7cbc08ef20 2019-12-05T06:53:24.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc0e43700 1 --2- 172.21.15.16:0/1821052930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7cbc08fef0 0x7f7cbc08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:53:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1644700 1 --2- 172.21.15.16:0/1821052930 >> v2:172.21.15.76:3300/0 conn(0x7f7cbc08c6d0 0x7f7cbc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:53:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 --2- 172.21.15.16:0/1821052930 >> v2:172.21.15.16:3300/0 conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7cb400f730 tx=0x7f7cb400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 --2- 172.21.15.16:0/1821052930 >> v2:172.21.15.16:3300/0 conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7cb400f730 tx=0x7f7cb400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.397+0000 7f7cc1e45700 1 --2- 172.21.15.16:0/1821052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 secure :-1 s=READY pgs=1489 cs=0 l=1 rx=0x7f7cb400f730 tx=0x7f7cb400fad0).ready entity=mon.0 client_cookie=8d660ed1191797ff server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:24.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.398+0000 7f7cc1e45700 1 --2- 172.21.15.16:0/1821052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 secure :-1 s=THROTTLE_DONE pgs=1489 cs=0 l=1 rx=0x7f7cb400f730 tx=0x7f7cb400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.398+0000 7f7cc1e45700 1 --2- 172.21.15.16:0/1821052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 secure :-1 s=THROTTLE_DONE pgs=1489 cs=0 l=1 rx=0x7f7cb400f730 tx=0x7f7cb400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.398+0000 7f7cbbfff700 1 -- 172.21.15.16:0/1821052930 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cb40010e0 con 0x7f7cbc08ef20 2019-12-05T06:53:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.398+0000 7f7cc1e45700 1 --2- 172.21.15.16:0/1821052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 secure :-1 s=THROTTLE_DONE pgs=1489 cs=0 l=1 rx=0x7f7cb400f730 tx=0x7f7cb400fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.398+0000 7f7cbbfff700 1 -- 172.21.15.16:0/1821052930 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cb4000b90 con 0x7f7cbc08ef20 2019-12-05T06:53:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.398+0000 7f7cbbfff700 1 -- 172.21.15.16:0/1821052930 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cb40016c0 con 0x7f7cbc08ef20 2019-12-05T06:53:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.398+0000 7f7cc30a7700 1 -- 172.21.15.16:0/1821052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc08ef20 msgr2=0x7f7cbc08f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:24.405 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.398+0000 7f7cc30a7700 1 --2- 172.21.15.16:0/1821052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 secure :-1 s=READY pgs=1489 cs=0 l=1 rx=0x7f7cb400f730 tx=0x7f7cb400fad0).stop 2019-12-05T06:53:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.399+0000 7f7cc30a7700 1 -- 172.21.15.16:0/1821052930 shutdown_connections 2019-12-05T06:53:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.399+0000 7f7cc30a7700 1 --2- 172.21.15.16:0/1821052930 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc08ef20 0x7f7cbc08f3c0 unknown :-1 s=CLOSED pgs=1489 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.399+0000 7f7cc30a7700 1 --2- 172.21.15.16:0/1821052930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7cbc08fef0 0x7f7cbc08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.399+0000 7f7cc30a7700 1 --2- 172.21.15.16:0/1821052930 >> v2:172.21.15.76:3300/0 conn(0x7f7cbc08c6d0 0x7f7cbc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.399+0000 7f7cc30a7700 1 -- 172.21.15.16:0/1821052930 shutdown_connections 2019-12-05T06:53:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.399+0000 7f7cc30a7700 1 -- 172.21.15.16:0/1821052930 wait complete. 2019-12-05T06:53:24.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.399+0000 7f7cc30a7700 1 -- 172.21.15.16:0/1821052930 >> 172.21.15.16:0/1821052930 conn(0x7f7cbc061940 msgr2=0x7f7cbc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:24.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.403+0000 7f7cc30a7700 1 Processor -- start 2019-12-05T06:53:24.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.403+0000 7f7cc30a7700 1 -- start start 2019-12-05T06:53:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc30a7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7cbc09b850 0x7f7cbc131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc30a7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7cbc154b00 0x7f7cbc156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc30a7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cbc098da0 con 0x7f7cbc132d00 2019-12-05T06:53:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc30a7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cbc09aa60 con 0x7f7cbc09b850 2019-12-05T06:53:24.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc30a7700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cbc09a030 con 0x7f7cbc154b00 2019-12-05T06:53:24.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc1644700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7cbc09b850 0x7f7cbc131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc1644700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7cbc09b850 0x7f7cbc131d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34952/0 (socket says 172.21.15.16:34952) 2019-12-05T06:53:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc1644700 1 -- 172.21.15.16:0/801499648 learned_addr learned my addr 172.21.15.16:0/801499648 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc1e45700 1 --2- 172.21.15.16:0/801499648 >> v2:172.21.15.76:3300/0 conn(0x7f7cbc154b00 0x7f7cbc156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc1644700 1 -- 172.21.15.16:0/801499648 >> v2:172.21.15.76:3300/0 conn(0x7f7cbc154b00 msgr2=0x7f7cbc156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:24.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> v2:172.21.15.76:3300/0 conn(0x7f7cbc154b00 0x7f7cbc156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc1644700 1 -- 172.21.15.16:0/801499648 >> v1:172.21.15.76:6789/0 conn(0x7f7cbc132d00 legacy=0x7f7cbc132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc1644700 1 -- 172.21.15.16:0/801499648 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cbc09b0f0 con 0x7f7cbc09b850 2019-12-05T06:53:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.404+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> v2:172.21.15.16:3300/0 conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.405+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> v2:172.21.15.16:3300/0 conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.405+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=READY pgs=1490 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:24.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.405+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=THROTTLE_DONE pgs=1490 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.405+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=THROTTLE_DONE pgs=1490 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.405+0000 7f7cb9ffb700 1 -- 172.21.15.16:0/801499648 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cac009830 con 0x7f7cbc09b850 2019-12-05T06:53:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.405+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=THROTTLE_DONE pgs=1490 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.405+0000 7f7cb9ffb700 1 -- 172.21.15.16:0/801499648 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cac009a30 con 0x7f7cbc09b850 2019-12-05T06:53:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.405+0000 7f7cc30a7700 1 -- 172.21.15.16:0/801499648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cbc09b0f0 con 0x7f7cbc09b850 2019-12-05T06:53:24.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.405+0000 7f7cb9ffb700 1 -- 172.21.15.16:0/801499648 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cac001650 con 0x7f7cbc09b850 2019-12-05T06:53:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.405+0000 7f7cc30a7700 1 -- 172.21.15.16:0/801499648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cbc09b0f0 con 0x7f7cbc09b850 2019-12-05T06:53:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.406+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=THROTTLE_DONE pgs=1490 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.406+0000 7f7cb9ffb700 1 -- 172.21.15.16:0/801499648 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7cac000f40 con 0x7f7cbc09b850 2019-12-05T06:53:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.406+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=THROTTLE_DONE pgs=1490 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.406+0000 7f7cb9ffb700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c9c023630 0x7f7c9c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.406+0000 7f7cb9ffb700 1 -- 172.21.15.16:0/801499648 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f7cac04a7d0 con 0x7f7cbc09b850 2019-12-05T06:53:24.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.406+0000 7f7cc0e43700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c9c023630 0x7f7c9c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:24.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.407+0000 7f7cc0e43700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c9c023630 0x7f7c9c025ac0 crc :-1 s=READY pgs=622 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:24.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.411+0000 7f7cc30a7700 1 -- 172.21.15.16:0/801499648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cbc09b0f0 con 0x7f7cbc09b850 2019-12-05T06:53:24.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.415+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=THROTTLE_DONE pgs=1490 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.415+0000 7f7cb9ffb700 1 -- 172.21.15.16:0/801499648 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7cac000d10 con 0x7f7cbc09b850 2019-12-05T06:53:24.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.640+0000 7f7cc30a7700 1 -- 172.21.15.16:0/801499648 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7cbc09b0f0 con 0x7f7cbc09b850 2019-12-05T06:53:24.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.641+0000 7f7cc1644700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=THROTTLE_DONE pgs=1490 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:24.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.641+0000 7f7cb9ffb700 1 -- 172.21.15.16:0/801499648 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f7cac009a30 con 0x7f7cbc09b850 2019-12-05T06:53:24.642 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 182 2019-12-05T06:53:24.642 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:24.642 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6092,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":182}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:24.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 -- 172.21.15.16:0/801499648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c9c023630 msgr2=0x7f7c9c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:24.643 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c9c023630 0x7f7c9c025ac0 crc :-1 s=READY pgs=622 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:24.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 -- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 msgr2=0x7f7cbc131d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:24.644 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 0x7f7cbc131d30 secure :-1 s=READY pgs=1490 cs=0 l=1 rx=0x7f7cac014b40 tx=0x7f7cac014ee0).stop 2019-12-05T06:53:24.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 -- 172.21.15.16:0/801499648 shutdown_connections 2019-12-05T06:53:24.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 --2- 172.21.15.16:0/801499648 >> v2:172.21.15.76:3300/0 conn(0x7f7cbc154b00 0x7f7cbc156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:24.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c9c023630 0x7f7c9c025ac0 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:24.645 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 --2- 172.21.15.16:0/801499648 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7cbc09b850 0x7f7cbc131d30 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:24.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 -- 172.21.15.16:0/801499648 shutdown_connections 2019-12-05T06:53:24.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 -- 172.21.15.16:0/801499648 wait complete. 2019-12-05T06:53:24.646 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:24.644+0000 7f7cc30a7700 1 -- 172.21.15.16:0/801499648 >> 172.21.15.16:0/801499648 conn(0x7f7cbc0714f0 msgr2=0x7f7cbc09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:24.654 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:53:24.655 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before thrashing 2019-12-05T06:53:26.656 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:26.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.806+0000 7f1f76c9e700 1 Processor -- start 2019-12-05T06:53:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.806+0000 7f1f76c9e700 1 -- start start 2019-12-05T06:53:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f76c9e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1f7008c6d0 0x7f1f7008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:26.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f76c9e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f7008fef0 0x7f1f7008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f76c9e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f7008ef20 0x7f1f7008f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f76c9e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f7008df90 con 0x7f1f7008fef0 2019-12-05T06:53:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f76c9e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f700a3f70 con 0x7f1f7008c6d0 2019-12-05T06:53:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f76c9e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7008d830 con 0x7f1f7008ef20 2019-12-05T06:53:26.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f74a3a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f7008fef0 0x7f1f7008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f74a3a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1f7008fef0 0x7f1f7008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:34982/0 (socket says 172.21.15.16:34982) 2019-12-05T06:53:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f74a3a700 1 -- 172.21.15.16:0/2590466538 learned_addr learned my addr 172.21.15.16:0/2590466538 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f7523b700 1 --2- 172.21.15.16:0/2590466538 >> v2:172.21.15.76:3300/0 conn(0x7f1f7008c6d0 0x7f1f7008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:26.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f75a3c700 1 --2- 172.21.15.16:0/2590466538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f7008ef20 0x7f1f7008f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f7523b700 1 -- 172.21.15.16:0/2590466538 >> v2:172.21.15.76:3300/0 conn(0x7f1f7008c6d0 msgr2=0x7f1f7008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T06:53:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f7523b700 1 -- 172.21.15.16:0/2590466538 >> v2:172.21.15.76:3300/0 conn(0x7f1f7008c6d0 msgr2=0x7f1f7008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:53:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f7523b700 1 --2- 172.21.15.16:0/2590466538 >> v2:172.21.15.76:3300/0 conn(0x7f1f7008c6d0 0x7f1f7008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:53:26.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f7523b700 1 --2- 172.21.15.16:0/2590466538 >> v2:172.21.15.76:3300/0 conn(0x7f1f7008c6d0 0x7f1f7008cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:53:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f74a3a700 1 -- 172.21.15.16:0/2590466538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f7008ef20 msgr2=0x7f1f7008f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/2590466538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f7008ef20 0x7f1f7008f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f74a3a700 1 -- 172.21.15.16:0/2590466538 >> v2:172.21.15.76:3300/0 conn(0x7f1f7008c6d0 msgr2=0x7f1f7008cac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:53:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/2590466538 >> v2:172.21.15.76:3300/0 conn(0x7f1f7008c6d0 0x7f1f7008cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:26.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f74a3a700 1 -- 172.21.15.16:0/2590466538 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f7008d540 con 0x7f1f7008fef0 2019-12-05T06:53:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f75a3c700 1 --2- 172.21.15.16:0/2590466538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f7008ef20 0x7f1f7008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:53:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/2590466538 >> v2:172.21.15.16:3300/0 conn(0x7f1f7008fef0 0x7f1f7008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1f6400f740 tx=0x7f1f6400f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.807+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/2590466538 >> v2:172.21.15.16:3300/0 conn(0x7f1f7008fef0 0x7f1f7008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1f6400f740 tx=0x7f1f6400f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.808+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/2590466538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7008fef0 0x7f1f7008d000 secure :-1 s=READY pgs=1491 cs=0 l=1 rx=0x7f1f6400f740 tx=0x7f1f6400f5f0).ready entity=mon.0 client_cookie=a29835d6e38f0779 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.808+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/2590466538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7008fef0 0x7f1f7008d000 secure :-1 s=THROTTLE_DONE pgs=1491 cs=0 l=1 rx=0x7f1f6400f740 tx=0x7f1f6400f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.808+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/2590466538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7008fef0 0x7f1f7008d000 secure :-1 s=THROTTLE_DONE pgs=1491 cs=0 l=1 rx=0x7f1f6400f740 tx=0x7f1f6400f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.808+0000 7f1f6ffff700 1 -- 172.21.15.16:0/2590466538 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f64000960 con 0x7f1f7008fef0 2019-12-05T06:53:26.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.808+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/2590466538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7008fef0 0x7f1f7008d000 secure :-1 s=THROTTLE_DONE pgs=1491 cs=0 l=1 rx=0x7f1f6400f740 tx=0x7f1f6400f5f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.820 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.808+0000 7f1f6ffff700 1 -- 172.21.15.16:0/2590466538 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f64000c90 con 0x7f1f7008fef0 2019-12-05T06:53:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.808+0000 7f1f6ffff700 1 -- 172.21.15.16:0/2590466538 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f640013a0 con 0x7f1f7008fef0 2019-12-05T06:53:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.808+0000 7f1f76c9e700 1 -- 172.21.15.16:0/2590466538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7008fef0 msgr2=0x7f1f7008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.808+0000 7f1f76c9e700 1 --2- 172.21.15.16:0/2590466538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7008fef0 0x7f1f7008d000 secure :-1 s=READY pgs=1491 cs=0 l=1 rx=0x7f1f6400f740 tx=0x7f1f6400f5f0).stop 2019-12-05T06:53:26.821 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.809+0000 7f1f76c9e700 1 -- 172.21.15.16:0/2590466538 shutdown_connections 2019-12-05T06:53:26.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.809+0000 7f1f76c9e700 1 --2- 172.21.15.16:0/2590466538 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f7008ef20 0x7f1f7008f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:26.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.809+0000 7f1f76c9e700 1 --2- 172.21.15.16:0/2590466538 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f7008fef0 0x7f1f7008d000 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:26.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.809+0000 7f1f76c9e700 1 --2- 172.21.15.16:0/2590466538 >> v2:172.21.15.76:3300/0 conn(0x7f1f7008c6d0 0x7f1f7008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:26.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.809+0000 7f1f76c9e700 1 -- 172.21.15.16:0/2590466538 shutdown_connections 2019-12-05T06:53:26.822 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.809+0000 7f1f76c9e700 1 -- 172.21.15.16:0/2590466538 wait complete. 2019-12-05T06:53:26.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.809+0000 7f1f76c9e700 1 -- 172.21.15.16:0/2590466538 >> 172.21.15.16:0/2590466538 conn(0x7f1f70061940 msgr2=0x7f1f70096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:26.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.811+0000 7f1f76c9e700 1 Processor -- start 2019-12-05T06:53:26.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.812+0000 7f1f76c9e700 1 -- start start 2019-12-05T06:53:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.812+0000 7f1f76c9e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f70131d50 0x7f1f7012ca30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f76c9e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7012d270 con 0x7f1f7012be40 2019-12-05T06:53:26.832 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f76c9e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f7012d720 con 0x7f1f70130d80 2019-12-05T06:53:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f76c9e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f7012e050 con 0x7f1f70131d50 2019-12-05T06:53:26.833 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f7523b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1f7012be40 0x7f1f7012c230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48384/0 (socket says 172.21.15.16:48384) 2019-12-05T06:53:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f7523b700 1 -- 172.21.15.16:0/1603874912 learned_addr learned my addr 172.21.15.16:0/1603874912 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f70131d50 0x7f1f7012ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372786146 0 0) 0x7f1f68001150 con 0x7f1f7012be40 2019-12-05T06:53:26.834 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f50002e80 con 0x7f1f7012be40 2019-12-05T06:53:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470239675 0 0) 0x7f1f60001080 con 0x7f1f70130d80 2019-12-05T06:53:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f500030b0 con 0x7f1f70130d80 2019-12-05T06:53:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2988858331 0 0) 0x7f1f68001350 con 0x7f1f7012be40 2019-12-05T06:53:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f500033c0 con 0x7f1f7012be40 2019-12-05T06:53:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f1f68001880 con 0x7f1f7012be40 2019-12-05T06:53:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f70131d50 msgr2=0x7f1f7012ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:26.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f70131d50 0x7f1f7012ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:26.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 >> v1:172.21.15.76:6789/0 conn(0x7f1f70130d80 legacy=0x7f1f70131240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:26.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 >> v1:172.21.15.16:6789/0 conn(0x7f1f7012be40 legacy=0x7f1f7012c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:26.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:26.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f500046f0 0x7f1f50004b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:26.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f50005040 0x7f1f500033c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:26.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f500039c0 con 0x7f1f50005dc0 2019-12-05T06:53:26.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f50003bf0 con 0x7f1f50005040 2019-12-05T06:53:26.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.813+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f50003e20 con 0x7f1f500046f0 2019-12-05T06:53:26.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:26.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f500046f0 0x7f1f50004b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:26.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f75a3c700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f50005040 0x7f1f500033c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:26.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 -- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f500046f0 msgr2=0x7f1f50004b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:26.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f500046f0 0x7f1f50004b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:26.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 -- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f50005040 msgr2=0x7f1f500033c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:26.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f1f50005040 0x7f1f500033c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:26.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 -- 172.21.15.16:0/1603874912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f500040d0 con 0x7f1f50005dc0 2019-12-05T06:53:26.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f76c9e700 1 -- 172.21.15.16:0/1603874912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f70130480 con 0x7f1f50005dc0 2019-12-05T06:53:26.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1f500046f0 0x7f1f50004b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:53:26.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f76c9e700 1 -- 172.21.15.16:0/1603874912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f7012d270 con 0x7f1f50005dc0 2019-12-05T06:53:26.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=THROTTLE_DONE pgs=1493 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f68001220 con 0x7f1f50005dc0 2019-12-05T06:53:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=THROTTLE_DONE pgs=1493 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=THROTTLE_DONE pgs=1493 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f68001570 con 0x7f1f50005dc0 2019-12-05T06:53:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.814+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1f68001c40 con 0x7f1f50005dc0 2019-12-05T06:53:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.815+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=THROTTLE_DONE pgs=1493 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.815+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1f68000cc0 con 0x7f1f50005dc0 2019-12-05T06:53:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.815+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=THROTTLE_DONE pgs=1493 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.815+0000 7f1f6dffb700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1f5002bad0 0x7f1f5002df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.815+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f1f68046310 con 0x7f1f50005dc0 2019-12-05T06:53:26.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.816+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1f5002bad0 0x7f1f5002df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.816+0000 7f1f74a3a700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1f5002bad0 0x7f1f5002df00 crc :-1 s=READY pgs=623 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.821+0000 7f1f76c9e700 1 -- 172.21.15.16:0/1603874912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f48002e60 con 0x7f1f50005dc0 2019-12-05T06:53:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.826+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=THROTTLE_DONE pgs=1493 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:26.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:26.826+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1f68005500 con 0x7f1f50005dc0 2019-12-05T06:53:27.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.048+0000 7f1f76c9e700 1 -- 172.21.15.16:0/1603874912 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1f48001d10 con 0x7f1f50005dc0 2019-12-05T06:53:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.049+0000 7f1f7523b700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=THROTTLE_DONE pgs=1493 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.049+0000 7f1f6dffb700 1 -- 172.21.15.16:0/1603874912 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f1f68001e20 con 0x7f1f50005dc0 2019-12-05T06:53:27.050 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 182 2019-12-05T06:53:27.050 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:27.050 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6092,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":182}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:27.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.051+0000 7f1f76c9e700 1 -- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1f5002bad0 msgr2=0x7f1f5002df00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:27.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.051+0000 7f1f76c9e700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1f5002bad0 0x7f1f5002df00 crc :-1 s=READY pgs=623 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:27.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.051+0000 7f1f76c9e700 1 -- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 msgr2=0x7f1f500061d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:27.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.051+0000 7f1f76c9e700 1 --2- 172.21.15.16:0/1603874912 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f1f50005dc0 0x7f1f500061d0 secure :-1 s=READY pgs=1493 cs=0 l=1 rx=0x7f1f680107c0 tx=0x7f1f680106c0).stop 2019-12-05T06:53:27.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.051+0000 7f1f75a3c700 1 -- 172.21.15.16:0/1603874912 reap_dead start 2019-12-05T06:53:27.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.052+0000 7f1f76c9e700 1 -- 172.21.15.16:0/1603874912 shutdown_connections 2019-12-05T06:53:27.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.052+0000 7f1f76c9e700 1 -- 172.21.15.16:0/1603874912 shutdown_connections 2019-12-05T06:53:27.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.052+0000 7f1f76c9e700 1 -- 172.21.15.16:0/1603874912 wait complete. 2019-12-05T06:53:27.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.052+0000 7f1f76c9e700 1 -- 172.21.15.16:0/1603874912 >> 172.21.15.16:0/1603874912 conn(0x7f1f700709e0 msgr2=0x7f1f7006eae0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:27.063 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:53:27.063 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:53:27.063 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:53:27.070 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:53:27.071 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.343+0000 7f051d014700 1 Processor -- start 2019-12-05T06:53:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.343+0000 7f051d014700 1 -- start start 2019-12-05T06:53:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.345+0000 7f051d014700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f051808c6d0 0x7f051808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.345+0000 7f051d014700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f051808f720 0x7f051808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.345+0000 7f051d014700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f051808fb70 con 0x7f051808fef0 2019-12-05T06:53:27.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.345+0000 7f051d014700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051808df90 con 0x7f051808c6d0 2019-12-05T06:53:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.345+0000 7f051d014700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05180a3f70 con 0x7f051808f720 2019-12-05T06:53:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.345+0000 7f051759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f051808c6d0 0x7f051808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.345+0000 7f051759e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f051808c6d0 0x7f051808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35006/0 (socket says 172.21.15.16:35006) 2019-12-05T06:53:27.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.345+0000 7f051759e700 1 -- 172.21.15.16:0/1634477763 learned_addr learned my addr 172.21.15.16:0/1634477763 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.345+0000 7f051c813700 1 --2- 172.21.15.16:0/1634477763 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f051808f720 0x7f051808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051759e700 1 -- 172.21.15.16:0/1634477763 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f051808f720 msgr2=0x7f051808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051759e700 1 --2- 172.21.15.16:0/1634477763 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f051808f720 0x7f051808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051759e700 1 -- 172.21.15.16:0/1634477763 >> v1:172.21.15.16:6789/0 conn(0x7f051808fef0 legacy=0x7f051808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051759e700 1 -- 172.21.15.16:0/1634477763 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051808d540 con 0x7f051808c6d0 2019-12-05T06:53:27.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051659c700 1 -- 172.21.15.16:0/1634477763 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150105478 0 0) 0x7f0500001100 con 0x7f051808fef0 2019-12-05T06:53:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051759e700 1 --2- 172.21.15.16:0/1634477763 >> v2:172.21.15.16:3300/0 conn(0x7f051808c6d0 0x7f051808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f051000fa40 tx=0x7f051000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051759e700 1 --2- 172.21.15.16:0/1634477763 >> v2:172.21.15.16:3300/0 conn(0x7f051808c6d0 0x7f051808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f051000fa40 tx=0x7f051000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051759e700 1 --2- 172.21.15.16:0/1634477763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051808c6d0 0x7f051808cac0 secure :-1 s=READY pgs=1495 cs=0 l=1 rx=0x7f051000fa40 tx=0x7f051000fde0).ready entity=mon.0 client_cookie=9e5fa128b71efed9 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051759e700 1 --2- 172.21.15.16:0/1634477763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051808c6d0 0x7f051808cac0 secure :-1 s=THROTTLE_DONE pgs=1495 cs=0 l=1 rx=0x7f051000fa40 tx=0x7f051000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051659c700 1 -- 172.21.15.16:0/1634477763 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0510000ce0 con 0x7f051808c6d0 2019-12-05T06:53:27.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051759e700 1 --2- 172.21.15.16:0/1634477763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051808c6d0 0x7f051808cac0 secure :-1 s=THROTTLE_DONE pgs=1495 cs=0 l=1 rx=0x7f051000fa40 tx=0x7f051000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.346+0000 7f051759e700 1 --2- 172.21.15.16:0/1634477763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051808c6d0 0x7f051808cac0 secure :-1 s=THROTTLE_DONE pgs=1495 cs=0 l=1 rx=0x7f051000fa40 tx=0x7f051000fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.347+0000 7f051659c700 1 -- 172.21.15.16:0/1634477763 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0510001010 con 0x7f051808c6d0 2019-12-05T06:53:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.347+0000 7f051659c700 1 -- 172.21.15.16:0/1634477763 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0510001720 con 0x7f051808c6d0 2019-12-05T06:53:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.347+0000 7f051d014700 1 -- 172.21.15.16:0/1634477763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051808c6d0 msgr2=0x7f051808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.347+0000 7f051d014700 1 --2- 172.21.15.16:0/1634477763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051808c6d0 0x7f051808cac0 secure :-1 s=READY pgs=1495 cs=0 l=1 rx=0x7f051000fa40 tx=0x7f051000fde0).stop 2019-12-05T06:53:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.348+0000 7f051d014700 1 -- 172.21.15.16:0/1634477763 shutdown_connections 2019-12-05T06:53:27.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.348+0000 7f051d014700 1 --2- 172.21.15.16:0/1634477763 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f051808f720 0x7f051808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.348+0000 7f051d014700 1 --2- 172.21.15.16:0/1634477763 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051808c6d0 0x7f051808cac0 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.348+0000 7f051d014700 1 -- 172.21.15.16:0/1634477763 shutdown_connections 2019-12-05T06:53:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.348+0000 7f051d014700 1 -- 172.21.15.16:0/1634477763 wait complete. 2019-12-05T06:53:27.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.348+0000 7f051d014700 1 -- 172.21.15.16:0/1634477763 >> 172.21.15.16:0/1634477763 conn(0x7f0518061940 msgr2=0x7f0518096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:27.361 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.353+0000 7f051d014700 1 Processor -- start 2019-12-05T06:53:27.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.354+0000 7f051d014700 1 -- start start 2019-12-05T06:53:27.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.355+0000 7f051d014700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f051809bbe0 0x7f05181539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:27.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.355+0000 7f051d014700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f0518084f30 0x7f0518155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:27.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.355+0000 7f051d014700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0518084b90 con 0x7f0518096e90 2019-12-05T06:53:27.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.355+0000 7f051d014700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f051809a450 con 0x7f051809bbe0 2019-12-05T06:53:27.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.355+0000 7f051d014700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0518099a20 con 0x7f0518084f30 2019-12-05T06:53:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.355+0000 7f0516d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f051809bbe0 0x7f05181539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.355+0000 7f0516d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f051809bbe0 0x7f05181539e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35012/0 (socket says 172.21.15.16:35012) 2019-12-05T06:53:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.355+0000 7f0516d9d700 1 -- 172.21.15.16:0/378341400 learned_addr learned my addr 172.21.15.16:0/378341400 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:27.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f051c813700 1 --2- 172.21.15.16:0/378341400 >> v2:172.21.15.76:3300/0 conn(0x7f0518084f30 0x7f0518155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:27.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f0516d9d700 1 -- 172.21.15.16:0/378341400 >> v2:172.21.15.76:3300/0 conn(0x7f0518084f30 msgr2=0x7f0518155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:27.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> v2:172.21.15.76:3300/0 conn(0x7f0518084f30 0x7f0518155f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:27.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f0516d9d700 1 -- 172.21.15.16:0/378341400 >> v1:172.21.15.76:6789/0 conn(0x7f0518096e90 legacy=0x7f051809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:27.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f0516d9d700 1 -- 172.21.15.16:0/378341400 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f051809aae0 con 0x7f051809bbe0 2019-12-05T06:53:27.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f051c813700 1 --2- 172.21.15.16:0/378341400 >> v2:172.21.15.76:3300/0 conn(0x7f0518084f30 0x7f0518155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:53:27.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> v2:172.21.15.16:3300/0 conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> v2:172.21.15.16:3300/0 conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=READY pgs=1496 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=THROTTLE_DONE pgs=1496 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=THROTTLE_DONE pgs=1496 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f050ffff700 1 -- 172.21.15.16:0/378341400 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0500000f90 con 0x7f051809bbe0 2019-12-05T06:53:27.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.356+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=THROTTLE_DONE pgs=1496 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.357+0000 7f050ffff700 1 -- 172.21.15.16:0/378341400 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0500000940 con 0x7f051809bbe0 2019-12-05T06:53:27.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.357+0000 7f050ffff700 1 -- 172.21.15.16:0/378341400 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0500001570 con 0x7f051809bbe0 2019-12-05T06:53:27.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.357+0000 7f051d014700 1 -- 172.21.15.16:0/378341400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f051809aae0 con 0x7f051809bbe0 2019-12-05T06:53:27.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.357+0000 7f051d014700 1 -- 172.21.15.16:0/378341400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f051809aae0 con 0x7f051809bbe0 2019-12-05T06:53:27.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.357+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=THROTTLE_DONE pgs=1496 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.358+0000 7f050ffff700 1 -- 172.21.15.16:0/378341400 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f05000053e0 con 0x7f051809bbe0 2019-12-05T06:53:27.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.358+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=THROTTLE_DONE pgs=1496 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.359+0000 7f050ffff700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04fc023630 0x7f04fc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:27.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.359+0000 7f050ffff700 1 -- 172.21.15.16:0/378341400 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f0500045e00 con 0x7f051809bbe0 2019-12-05T06:53:27.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.359+0000 7f051759e700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04fc023630 0x7f04fc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:27.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.359+0000 7f051759e700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04fc023630 0x7f04fc025ac0 crc :-1 s=READY pgs=626 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:27.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.374+0000 7f051d014700 1 -- 172.21.15.16:0/378341400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f051809aae0 con 0x7f051809bbe0 2019-12-05T06:53:27.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.383+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=THROTTLE_DONE pgs=1496 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.383+0000 7f050ffff700 1 -- 172.21.15.16:0/378341400 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f05000075b0 con 0x7f051809bbe0 2019-12-05T06:53:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.628+0000 7f051d014700 1 -- 172.21.15.16:0/378341400 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f051809aae0 con 0x7f051809bbe0 2019-12-05T06:53:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.629+0000 7f0516d9d700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=THROTTLE_DONE pgs=1496 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:27.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.629+0000 7f050ffff700 1 -- 172.21.15.16:0/378341400 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f0500005340 con 0x7f051809bbe0 2019-12-05T06:53:27.631 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 182 2019-12-05T06:53:27.631 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:27.631 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6092,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":182}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:27.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 -- 172.21.15.16:0/378341400 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04fc023630 msgr2=0x7f04fc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04fc023630 0x7f04fc025ac0 crc :-1 s=READY pgs=626 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 -- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 msgr2=0x7f05181539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 0x7f05181539e0 secure :-1 s=READY pgs=1496 cs=0 l=1 rx=0x7f0500010410 tx=0x7f05000107b0).stop 2019-12-05T06:53:27.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 -- 172.21.15.16:0/378341400 shutdown_connections 2019-12-05T06:53:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 --2- 172.21.15.16:0/378341400 >> v2:172.21.15.76:3300/0 conn(0x7f0518084f30 0x7f0518155f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04fc023630 0x7f04fc025ac0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 --2- 172.21.15.16:0/378341400 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f051809bbe0 0x7f05181539e0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:27.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 -- 172.21.15.16:0/378341400 shutdown_connections 2019-12-05T06:53:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 -- 172.21.15.16:0/378341400 wait complete. 2019-12-05T06:53:27.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:27.632+0000 7f051d014700 1 -- 172.21.15.16:0/378341400 >> 172.21.15.16:0/378341400 conn(0x7f05180714f0 msgr2=0x7f051809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:27.643 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:53:29.644 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:29.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.897+0000 7ff194115700 1 Processor -- start 2019-12-05T06:53:29.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.897+0000 7ff194115700 1 -- start start 2019-12-05T06:53:29.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.899+0000 7ff194115700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff18c08c6d0 0x7ff18c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:29.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.899+0000 7ff194115700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c08f720 0x7ff18c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.899+0000 7ff194115700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c08fb70 con 0x7ff18c08fef0 2019-12-05T06:53:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.899+0000 7ff194115700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff18c08df90 con 0x7ff18c08c6d0 2019-12-05T06:53:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.899+0000 7ff194115700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c0a3f70 con 0x7ff18c08f720 2019-12-05T06:53:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.900+0000 7ff1926b2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff18c08c6d0 0x7ff18c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.900+0000 7ff191eb1700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff18c08fef0 0x7ff18c08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48428/0 (socket says 172.21.15.16:48428) 2019-12-05T06:53:29.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.900+0000 7ff191eb1700 1 -- 172.21.15.16:0/4012854355 learned_addr learned my addr 172.21.15.16:0/4012854355 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.900+0000 7ff192eb3700 1 --2- 172.21.15.16:0/4012854355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c08f720 0x7ff18c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.900+0000 7ff1926b2700 1 -- 172.21.15.16:0/4012854355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c08f720 msgr2=0x7ff18c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.900+0000 7ff1926b2700 1 --2- 172.21.15.16:0/4012854355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c08f720 0x7ff18c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.900+0000 7ff1926b2700 1 -- 172.21.15.16:0/4012854355 >> v1:172.21.15.16:6789/0 conn(0x7ff18c08fef0 legacy=0x7ff18c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.900+0000 7ff1926b2700 1 -- 172.21.15.16:0/4012854355 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c08d540 con 0x7ff18c08c6d0 2019-12-05T06:53:29.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.900+0000 7ff1916b0700 1 -- 172.21.15.16:0/4012854355 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618971440 0 0) 0x7ff180001070 con 0x7ff18c08fef0 2019-12-05T06:53:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff1926b2700 1 --2- 172.21.15.16:0/4012854355 >> v2:172.21.15.16:3300/0 conn(0x7ff18c08c6d0 0x7ff18c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff18400b2d0 tx=0x7ff18400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff1926b2700 1 --2- 172.21.15.16:0/4012854355 >> v2:172.21.15.16:3300/0 conn(0x7ff18c08c6d0 0x7ff18c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff18400b2d0 tx=0x7ff18400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff1926b2700 1 --2- 172.21.15.16:0/4012854355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18c08c6d0 0x7ff18c08cac0 secure :-1 s=READY pgs=1498 cs=0 l=1 rx=0x7ff18400b2d0 tx=0x7ff18400ba60).ready entity=mon.0 client_cookie=f32f4d228028bf0e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff1926b2700 1 --2- 172.21.15.16:0/4012854355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18c08c6d0 0x7ff18c08cac0 secure :-1 s=THROTTLE_DONE pgs=1498 cs=0 l=1 rx=0x7ff18400b2d0 tx=0x7ff18400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff1916b0700 1 -- 172.21.15.16:0/4012854355 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff184007590 con 0x7ff18c08c6d0 2019-12-05T06:53:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff1926b2700 1 --2- 172.21.15.16:0/4012854355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18c08c6d0 0x7ff18c08cac0 secure :-1 s=THROTTLE_DONE pgs=1498 cs=0 l=1 rx=0x7ff18400b2d0 tx=0x7ff18400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff1926b2700 1 --2- 172.21.15.16:0/4012854355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18c08c6d0 0x7ff18c08cac0 secure :-1 s=THROTTLE_DONE pgs=1498 cs=0 l=1 rx=0x7ff18400b2d0 tx=0x7ff18400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff1916b0700 1 -- 172.21.15.16:0/4012854355 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff184009d30 con 0x7ff18c08c6d0 2019-12-05T06:53:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff1916b0700 1 -- 172.21.15.16:0/4012854355 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff18400a400 con 0x7ff18c08c6d0 2019-12-05T06:53:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff194115700 1 -- 172.21.15.16:0/4012854355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18c08c6d0 msgr2=0x7ff18c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.901+0000 7ff194115700 1 --2- 172.21.15.16:0/4012854355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18c08c6d0 0x7ff18c08cac0 secure :-1 s=READY pgs=1498 cs=0 l=1 rx=0x7ff18400b2d0 tx=0x7ff18400ba60).stop 2019-12-05T06:53:29.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.904+0000 7ff194115700 1 -- 172.21.15.16:0/4012854355 shutdown_connections 2019-12-05T06:53:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.904+0000 7ff194115700 1 --2- 172.21.15.16:0/4012854355 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c08f720 0x7ff18c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.904+0000 7ff194115700 1 --2- 172.21.15.16:0/4012854355 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff18c08c6d0 0x7ff18c08cac0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:29.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.904+0000 7ff194115700 1 -- 172.21.15.16:0/4012854355 shutdown_connections 2019-12-05T06:53:29.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.904+0000 7ff194115700 1 -- 172.21.15.16:0/4012854355 wait complete. 2019-12-05T06:53:29.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.904+0000 7ff194115700 1 -- 172.21.15.16:0/4012854355 >> 172.21.15.16:0/4012854355 conn(0x7ff18c061940 msgr2=0x7ff18c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:29.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.908+0000 7ff194115700 1 Processor -- start 2019-12-05T06:53:29.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.908+0000 7ff194115700 1 -- start start 2019-12-05T06:53:29.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.909+0000 7ff194115700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:29.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.909+0000 7ff194115700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c132970 con 0x7ff18c14d400 2019-12-05T06:53:29.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.910+0000 7ff194115700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff18c098da0 con 0x7ff18c132530 2019-12-05T06:53:29.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.910+0000 7ff194115700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c09bb50 con 0x7ff18c132d00 2019-12-05T06:53:29.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.910+0000 7ff1926b2700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff18c14d400 0x7ff18c131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48430/0 (socket says 172.21.15.16:48430) 2019-12-05T06:53:29.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.910+0000 7ff1926b2700 1 -- 172.21.15.16:0/3270767520 learned_addr learned my addr 172.21.15.16:0/3270767520 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:29.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.910+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:29.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.910+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435063331 0 0) 0x7ff184009990 con 0x7ff18c14d400 2019-12-05T06:53:29.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.910+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff16c002e80 con 0x7ff18c14d400 2019-12-05T06:53:29.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.910+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 287123653 0 0) 0x7ff17c000e70 con 0x7ff18c132530 2019-12-05T06:53:29.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.911+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff16c0030b0 con 0x7ff18c132530 2019-12-05T06:53:29.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.911+0000 7ff191eb1700 1 -- 172.21.15.16:0/3270767520 >> v1:172.21.15.76:6789/0 conn(0x7ff18c132530 legacy=0x7ff18c157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:29.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.911+0000 7ff191eb1700 1 -- 172.21.15.16:0/3270767520 >> v1:172.21.15.16:6789/0 conn(0x7ff18c14d400 legacy=0x7ff18c131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:29.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.911+0000 7ff191eb1700 1 -- 172.21.15.16:0/3270767520 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c09b0f0 con 0x7ff18c132d00 2019-12-05T06:53:29.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.911+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.911+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.911+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=READY pgs=348 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:29.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.911+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=THROTTLE_DONE pgs=348 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.911+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1800041d0 con 0x7ff18c132d00 2019-12-05T06:53:29.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.911+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=THROTTLE_DONE pgs=348 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.912+0000 7ff194115700 1 -- 172.21.15.16:0/3270767520 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c09b0f0 con 0x7ff18c132d00 2019-12-05T06:53:29.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.912+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=THROTTLE_DONE pgs=348 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.912+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff180000990 con 0x7ff18c132d00 2019-12-05T06:53:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.912+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1800041d0 con 0x7ff18c132d00 2019-12-05T06:53:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.912+0000 7ff194115700 1 -- 172.21.15.16:0/3270767520 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c09b0f0 con 0x7ff18c132d00 2019-12-05T06:53:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.913+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=THROTTLE_DONE pgs=348 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.914+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff180000df0 con 0x7ff18c132d00 2019-12-05T06:53:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.914+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=THROTTLE_DONE pgs=348 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.914+0000 7ff18affd700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff16c023650 0x7ff16c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:29.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.914+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7ff180045d60 con 0x7ff18c132d00 2019-12-05T06:53:29.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.914+0000 7ff1926b2700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff16c023650 0x7ff16c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:29.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.915+0000 7ff1926b2700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff16c023650 0x7ff16c025ae0 crc :-1 s=READY pgs=627 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:29.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.930+0000 7ff194115700 1 -- 172.21.15.16:0/3270767520 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff18c09b0f0 con 0x7ff18c132d00 2019-12-05T06:53:29.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.939+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=THROTTLE_DONE pgs=348 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:29.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:29.939+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff1800062e0 con 0x7ff18c132d00 2019-12-05T06:53:30.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.207+0000 7ff194115700 1 -- 172.21.15.16:0/3270767520 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff18c09b0f0 con 0x7ff18c132d00 2019-12-05T06:53:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.208+0000 7ff191eb1700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=THROTTLE_DONE pgs=348 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.208+0000 7ff18affd700 1 -- 172.21.15.16:0/3270767520 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+2109 (secure 0 0 0) 0x7ff1800062e0 con 0x7ff18c132d00 2019-12-05T06:53:30.208 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 182 2019-12-05T06:53:30.209 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:30.209 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6092,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":182}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.212+0000 7ff194115700 1 -- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff16c023650 msgr2=0x7ff16c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:30.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.212+0000 7ff194115700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff16c023650 0x7ff16c025ae0 crc :-1 s=READY pgs=627 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.212+0000 7ff194115700 1 -- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 msgr2=0x7ff18c14d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.212+0000 7ff194115700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 secure :-1 s=READY pgs=348 cs=0 l=1 rx=0x7ff180010330 tx=0x7ff1800106d0).stop 2019-12-05T06:53:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.212+0000 7ff194115700 1 -- 172.21.15.16:0/3270767520 shutdown_connections 2019-12-05T06:53:30.214 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.212+0000 7ff194115700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff16c023650 0x7ff16c025ae0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.212+0000 7ff194115700 1 --2- 172.21.15.16:0/3270767520 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff18c132d00 0x7ff18c14d7f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.212+0000 7ff194115700 1 -- 172.21.15.16:0/3270767520 shutdown_connections 2019-12-05T06:53:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.212+0000 7ff194115700 1 -- 172.21.15.16:0/3270767520 wait complete. 2019-12-05T06:53:30.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:30.212+0000 7ff194115700 1 -- 172.21.15.16:0/3270767520 >> 172.21.15.16:0/3270767520 conn(0x7ff18c0714f0 msgr2=0x7ff18c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:30.229 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:53:32.231 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:32.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.503+0000 7f3047322700 1 Processor -- start 2019-12-05T06:53:32.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.504+0000 7f3047322700 1 -- start start 2019-12-05T06:53:32.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.505+0000 7f3047322700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f304008c6d0 0x7f304008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:32.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.505+0000 7f3047322700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304008fb80 con 0x7f304008fef0 2019-12-05T06:53:32.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.505+0000 7f3047322700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30400958e0 con 0x7f304008f720 2019-12-05T06:53:32.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.505+0000 7f3047322700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f304008df90 con 0x7f304008c6d0 2019-12-05T06:53:32.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.505+0000 7f30450be700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f304008fef0 0x7f304008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48446/0 (socket says 172.21.15.16:48446) 2019-12-05T06:53:32.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.505+0000 7f30450be700 1 -- 172.21.15.16:0/3058609160 learned_addr learned my addr 172.21.15.16:0/3058609160 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:32.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.506+0000 7f30458bf700 1 --2- 172.21.15.16:0/3058609160 >> v2:172.21.15.76:3300/0 conn(0x7f304008c6d0 0x7f304008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:32.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.506+0000 7f30448bd700 1 -- 172.21.15.16:0/3058609160 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1945879481 0 0) 0x7f3034001070 con 0x7f304008fef0 2019-12-05T06:53:32.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.506+0000 7f30448bd700 1 -- 172.21.15.16:0/3058609160 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3028002e80 con 0x7f304008fef0 2019-12-05T06:53:32.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.506+0000 7f30458bf700 1 -- 172.21.15.16:0/3058609160 >> v1:172.21.15.76:6789/0 conn(0x7f304008f720 legacy=0x7f30401533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:32.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.506+0000 7f30448bd700 1 -- 172.21.15.16:0/3058609160 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722246693 0 0) 0x7f30300009b0 con 0x7f304008f720 2019-12-05T06:53:32.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.506+0000 7f30458bf700 1 -- 172.21.15.16:0/3058609160 >> v1:172.21.15.16:6789/0 conn(0x7f304008fef0 legacy=0x7f304008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:32.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.507+0000 7f30458bf700 1 -- 172.21.15.16:0/3058609160 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304008e620 con 0x7f304008c6d0 2019-12-05T06:53:32.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.507+0000 7f30458bf700 1 --2- 172.21.15.16:0/3058609160 >> v2:172.21.15.76:3300/0 conn(0x7f304008c6d0 0x7f304008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f303800b2d0 tx=0x7f303800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.507+0000 7f30458bf700 1 --2- 172.21.15.16:0/3058609160 >> v2:172.21.15.76:3300/0 conn(0x7f304008c6d0 0x7f304008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f303800b2d0 tx=0x7f303800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.507+0000 7f30458bf700 1 --2- 172.21.15.16:0/3058609160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304008c6d0 0x7f304008cac0 secure :-1 s=READY pgs=1006 cs=0 l=1 rx=0x7f303800b2d0 tx=0x7f303800bae0).ready entity=mon.1 client_cookie=f838f549e5ff9624 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:32.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.507+0000 7f30458bf700 1 --2- 172.21.15.16:0/3058609160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304008c6d0 0x7f304008cac0 secure :-1 s=THROTTLE_DONE pgs=1006 cs=0 l=1 rx=0x7f303800b2d0 tx=0x7f303800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.507+0000 7f30448bd700 1 -- 172.21.15.16:0/3058609160 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3038000ce0 con 0x7f304008c6d0 2019-12-05T06:53:32.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.507+0000 7f30458bf700 1 --2- 172.21.15.16:0/3058609160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304008c6d0 0x7f304008cac0 secure :-1 s=THROTTLE_DONE pgs=1006 cs=0 l=1 rx=0x7f303800b2d0 tx=0x7f303800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.508+0000 7f30458bf700 1 --2- 172.21.15.16:0/3058609160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304008c6d0 0x7f304008cac0 secure :-1 s=THROTTLE_DONE pgs=1006 cs=0 l=1 rx=0x7f303800b2d0 tx=0x7f303800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.508+0000 7f30448bd700 1 -- 172.21.15.16:0/3058609160 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3038009be0 con 0x7f304008c6d0 2019-12-05T06:53:32.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.508+0000 7f30448bd700 1 -- 172.21.15.16:0/3058609160 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f303800a2b0 con 0x7f304008c6d0 2019-12-05T06:53:32.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.508+0000 7f3047322700 1 -- 172.21.15.16:0/3058609160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304008c6d0 msgr2=0x7f304008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:32.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.508+0000 7f3047322700 1 --2- 172.21.15.16:0/3058609160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304008c6d0 0x7f304008cac0 secure :-1 s=READY pgs=1006 cs=0 l=1 rx=0x7f303800b2d0 tx=0x7f303800bae0).stop 2019-12-05T06:53:32.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.510+0000 7f3047322700 1 -- 172.21.15.16:0/3058609160 shutdown_connections 2019-12-05T06:53:32.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.510+0000 7f3047322700 1 --2- 172.21.15.16:0/3058609160 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304008c6d0 0x7f304008cac0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:32.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.510+0000 7f3047322700 1 -- 172.21.15.16:0/3058609160 shutdown_connections 2019-12-05T06:53:32.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.510+0000 7f3047322700 1 -- 172.21.15.16:0/3058609160 wait complete. 2019-12-05T06:53:32.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.510+0000 7f3047322700 1 -- 172.21.15.16:0/3058609160 >> 172.21.15.16:0/3058609160 conn(0x7f3040061940 msgr2=0x7f3040096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:32.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.515+0000 7f3047322700 1 Processor -- start 2019-12-05T06:53:32.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.515+0000 7f3047322700 1 -- start start 2019-12-05T06:53:32.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.516+0000 7f3047322700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f304009e000 0x7f304015a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:32.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.516+0000 7f3047322700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304009e440 con 0x7f304009e7d0 2019-12-05T06:53:32.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.516+0000 7f3047322700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f304012cb90 con 0x7f30401302f0 2019-12-05T06:53:32.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.516+0000 7f3047322700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3040131ff0 con 0x7f304009e000 2019-12-05T06:53:32.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.516+0000 7f30450be700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f304009e7d0 0x7f3040156e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48450/0 (socket says 172.21.15.16:48450) 2019-12-05T06:53:32.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.516+0000 7f30450be700 1 -- 172.21.15.16:0/3839826713 learned_addr learned my addr 172.21.15.16:0/3839826713 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:32.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.516+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> v2:172.21.15.76:3300/0 conn(0x7f304009e000 0x7f304015a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:32.520 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.517+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338327802 0 0) 0x7f3034001280 con 0x7f304009e7d0 2019-12-05T06:53:32.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.517+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3020002e80 con 0x7f304009e7d0 2019-12-05T06:53:32.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.517+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1931144013 0 0) 0x7f303800b750 con 0x7f30401302f0 2019-12-05T06:53:32.521 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.517+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3020002ec0 con 0x7f30401302f0 2019-12-05T06:53:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.517+0000 7f30460c0700 1 -- 172.21.15.16:0/3839826713 >> v1:172.21.15.76:6789/0 conn(0x7f30401302f0 legacy=0x7f304009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.517+0000 7f30460c0700 1 -- 172.21.15.16:0/3839826713 >> v1:172.21.15.16:6789/0 conn(0x7f304009e7d0 legacy=0x7f3040156e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.517+0000 7f30460c0700 1 -- 172.21.15.16:0/3839826713 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3040132680 con 0x7f304009e000 2019-12-05T06:53:32.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.517+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> v2:172.21.15.76:3300/0 conn(0x7f304009e000 0x7f304015a690 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.518+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> v2:172.21.15.76:3300/0 conn(0x7f304009e000 0x7f304015a690 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.518+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 0x7f304015a690 secure :-1 s=READY pgs=1008 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.518+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 0x7f304015a690 secure :-1 s=THROTTLE_DONE pgs=1008 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.518+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3030000940 con 0x7f304009e000 2019-12-05T06:53:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.518+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 0x7f304015a690 secure :-1 s=THROTTLE_DONE pgs=1008 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.524 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.518+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 0x7f304015a690 secure :-1 s=THROTTLE_DONE pgs=1008 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.518+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3030009990 con 0x7f304009e000 2019-12-05T06:53:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.518+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f303000a060 con 0x7f304009e000 2019-12-05T06:53:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.518+0000 7f3047322700 1 -- 172.21.15.16:0/3839826713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3040132680 con 0x7f304009e000 2019-12-05T06:53:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.519+0000 7f3047322700 1 -- 172.21.15.16:0/3839826713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3040132680 con 0x7f304009e000 2019-12-05T06:53:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.519+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 0x7f304015a690 secure :-1 s=THROTTLE_DONE pgs=1008 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.520+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3030009590 con 0x7f304009e000 2019-12-05T06:53:32.525 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.520+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 0x7f304015a690 secure :-1 s=THROTTLE_DONE pgs=1008 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.521+0000 7f303e7fc700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3020023680 0x7f3020025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:32.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.521+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f3030045dc0 con 0x7f304009e000 2019-12-05T06:53:32.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.521+0000 7f30458bf700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3020023680 0x7f3020025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:32.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.521+0000 7f30458bf700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3020023680 0x7f3020025af0 crc :-1 s=READY pgs=628 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:32.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.535+0000 7f3047322700 1 -- 172.21.15.16:0/3839826713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3040132680 con 0x7f304009e000 2019-12-05T06:53:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.544+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 0x7f304015a690 secure :-1 s=THROTTLE_DONE pgs=1008 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.544+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3030000940 con 0x7f304009e000 2019-12-05T06:53:32.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.805+0000 7f3047322700 1 -- 172.21.15.16:0/3839826713 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3040132680 con 0x7f304009e000 2019-12-05T06:53:32.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.806+0000 7f30460c0700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 0x7f304015a690 secure :-1 s=THROTTLE_DONE pgs=1008 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:32.805 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.806+0000 7f303e7fc700 1 -- 172.21.15.16:0/3839826713 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f3030009590 con 0x7f304009e000 2019-12-05T06:53:32.807 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:32.808 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6092,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":182}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:32.808 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 182 2019-12-05T06:53:32.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.810+0000 7f3047322700 1 -- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3020023680 msgr2=0x7f3020025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:32.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.810+0000 7f3047322700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3020023680 0x7f3020025af0 crc :-1 s=READY pgs=628 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:32.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.810+0000 7f3047322700 1 -- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 msgr2=0x7f304015a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:32.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.810+0000 7f3047322700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 0x7f304015a690 secure :-1 s=READY pgs=1008 cs=0 l=1 rx=0x7f30300103b0 tx=0x7f3030010750).stop 2019-12-05T06:53:32.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.810+0000 7f3047322700 1 -- 172.21.15.16:0/3839826713 shutdown_connections 2019-12-05T06:53:32.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.810+0000 7f3047322700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3020023680 0x7f3020025af0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:32.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.810+0000 7f3047322700 1 --2- 172.21.15.16:0/3839826713 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f304009e000 0x7f304015a690 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:32.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.810+0000 7f3047322700 1 -- 172.21.15.16:0/3839826713 shutdown_connections 2019-12-05T06:53:32.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.810+0000 7f3047322700 1 -- 172.21.15.16:0/3839826713 wait complete. 2019-12-05T06:53:32.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:32.811+0000 7f3047322700 1 -- 172.21.15.16:0/3839826713 >> 172.21.15.16:0/3839826713 conn(0x7f30400a2850 msgr2=0x7f304006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:32.821 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:53:34.822 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.107+0000 7f489d382700 1 Processor -- start 2019-12-05T06:53:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.108+0000 7f489d382700 1 -- start start 2019-12-05T06:53:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.109+0000 7f489d382700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f489808c6d0 0x7f489808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.109+0000 7f489d382700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f489808f720 0x7f489808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.109+0000 7f489d382700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f489808fb80 con 0x7f489808fef0 2019-12-05T06:53:35.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.109+0000 7f489d382700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489808df90 con 0x7f489808f720 2019-12-05T06:53:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.109+0000 7f489d382700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48980a3f70 con 0x7f489808c6d0 2019-12-05T06:53:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.110+0000 7f4897fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f489808f720 0x7f489808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.110+0000 7f4897fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f489808f720 0x7f489808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35078/0 (socket says 172.21.15.16:35078) 2019-12-05T06:53:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.110+0000 7f4897fff700 1 -- 172.21.15.16:0/2013781645 learned_addr learned my addr 172.21.15.16:0/2013781645 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.110+0000 7f4896ffd700 1 --1- 172.21.15.16:0/2013781645 >> v1:172.21.15.16:6789/0 conn(0x7f489808fef0 0x7f489808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48478/0 (socket says 172.21.15.16:48478) 2019-12-05T06:53:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.110+0000 7f48977fe700 1 --2- 172.21.15.16:0/2013781645 >> v2:172.21.15.76:3300/0 conn(0x7f489808c6d0 0x7f489808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:35.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.110+0000 7f4897fff700 1 -- 172.21.15.16:0/2013781645 >> v2:172.21.15.76:3300/0 conn(0x7f489808c6d0 msgr2=0x7f489808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:35.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.110+0000 7f4897fff700 1 --2- 172.21.15.16:0/2013781645 >> v2:172.21.15.76:3300/0 conn(0x7f489808c6d0 0x7f489808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:35.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.110+0000 7f4897fff700 1 -- 172.21.15.16:0/2013781645 >> v1:172.21.15.16:6789/0 conn(0x7f489808fef0 legacy=0x7f489808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:35.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.111+0000 7f4897fff700 1 -- 172.21.15.16:0/2013781645 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489808d540 con 0x7f489808f720 2019-12-05T06:53:35.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.110+0000 7f48967fc700 1 -- 172.21.15.16:0/2013781645 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137163991 0 0) 0x7f4880001100 con 0x7f489808fef0 2019-12-05T06:53:35.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.111+0000 7f4897fff700 1 --2- 172.21.15.16:0/2013781645 >> v2:172.21.15.16:3300/0 conn(0x7f489808f720 0x7f489808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f489000fab0 tx=0x7f48900080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.111+0000 7f4897fff700 1 --2- 172.21.15.16:0/2013781645 >> v2:172.21.15.16:3300/0 conn(0x7f489808f720 0x7f489808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f489000fab0 tx=0x7f48900080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.111+0000 7f4897fff700 1 --2- 172.21.15.16:0/2013781645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489808f720 0x7f489808d000 secure :-1 s=READY pgs=1503 cs=0 l=1 rx=0x7f489000fab0 tx=0x7f48900080d0).ready entity=mon.0 client_cookie=9d313b6ca1a6614b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:35.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.111+0000 7f4897fff700 1 --2- 172.21.15.16:0/2013781645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489808f720 0x7f489808d000 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7f489000fab0 tx=0x7f48900080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.111+0000 7f4897fff700 1 --2- 172.21.15.16:0/2013781645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489808f720 0x7f489808d000 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7f489000fab0 tx=0x7f48900080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.111+0000 7f48967fc700 1 -- 172.21.15.16:0/2013781645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4890000df0 con 0x7f489808f720 2019-12-05T06:53:35.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.111+0000 7f4897fff700 1 --2- 172.21.15.16:0/2013781645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489808f720 0x7f489808d000 secure :-1 s=THROTTLE_DONE pgs=1503 cs=0 l=1 rx=0x7f489000fab0 tx=0x7f48900080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.112+0000 7f48967fc700 1 -- 172.21.15.16:0/2013781645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4890001460 con 0x7f489808f720 2019-12-05T06:53:35.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.112+0000 7f48967fc700 1 -- 172.21.15.16:0/2013781645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4890001b30 con 0x7f489808f720 2019-12-05T06:53:35.121 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.112+0000 7f489d382700 1 -- 172.21.15.16:0/2013781645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489808f720 msgr2=0x7f489808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.112+0000 7f489d382700 1 --2- 172.21.15.16:0/2013781645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489808f720 0x7f489808d000 secure :-1 s=READY pgs=1503 cs=0 l=1 rx=0x7f489000fab0 tx=0x7f48900080d0).stop 2019-12-05T06:53:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.113+0000 7f489d382700 1 -- 172.21.15.16:0/2013781645 shutdown_connections 2019-12-05T06:53:35.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.113+0000 7f489d382700 1 --2- 172.21.15.16:0/2013781645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f489808f720 0x7f489808d000 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:35.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.113+0000 7f489d382700 1 --2- 172.21.15.16:0/2013781645 >> v2:172.21.15.76:3300/0 conn(0x7f489808c6d0 0x7f489808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:35.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.113+0000 7f489d382700 1 -- 172.21.15.16:0/2013781645 shutdown_connections 2019-12-05T06:53:35.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.114+0000 7f489d382700 1 -- 172.21.15.16:0/2013781645 wait complete. 2019-12-05T06:53:35.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.114+0000 7f489d382700 1 -- 172.21.15.16:0/2013781645 >> 172.21.15.16:0/2013781645 conn(0x7f4898061940 msgr2=0x7f4898096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.119+0000 7f489d382700 1 Processor -- start 2019-12-05T06:53:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.119+0000 7f489d382700 1 -- start start 2019-12-05T06:53:35.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.121+0000 7f489d382700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f489809bbe0 0x7f48981539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:35.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.121+0000 7f489d382700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:35.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.121+0000 7f489d382700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4898084b90 con 0x7f4898096e90 2019-12-05T06:53:35.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.121+0000 7f489d382700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f489809a450 con 0x7f489809bbe0 2019-12-05T06:53:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.121+0000 7f489d382700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4898099a20 con 0x7f4898084f30 2019-12-05T06:53:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.121+0000 7f4897fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.121+0000 7f4896ffd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f489809bbe0 0x7f48981539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.121+0000 7f4897fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:39072/0 (socket says 172.21.15.16:39072) 2019-12-05T06:53:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.121+0000 7f4897fff700 1 -- 172.21.15.16:0/4032544449 learned_addr learned my addr 172.21.15.16:0/4032544449 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.122+0000 7f4897fff700 1 -- 172.21.15.16:0/4032544449 >> v2:172.21.15.76:3300/0 conn(0x7f489809bbe0 msgr2=0x7f48981539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:35.154 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.122+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> v2:172.21.15.76:3300/0 conn(0x7f489809bbe0 0x7f48981539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.122+0000 7f4897fff700 1 -- 172.21.15.16:0/4032544449 >> v1:172.21.15.76:6789/0 conn(0x7f4898096e90 legacy=0x7f489809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.122+0000 7f4897fff700 1 -- 172.21.15.16:0/4032544449 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489809aae0 con 0x7f4898084f30 2019-12-05T06:53:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.122+0000 7f4896ffd700 1 --2- 172.21.15.16:0/4032544449 >> v2:172.21.15.76:3300/0 conn(0x7f489809bbe0 0x7f48981539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:53:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.122+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.122+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.155 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.122+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=READY pgs=349 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.123+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.123+0000 7f488ffff700 1 -- 172.21.15.16:0/4032544449 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4890005aa0 con 0x7f4898084f30 2019-12-05T06:53:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.123+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.123+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.123+0000 7f488ffff700 1 -- 172.21.15.16:0/4032544449 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48900056c0 con 0x7f4898084f30 2019-12-05T06:53:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.123+0000 7f488ffff700 1 -- 172.21.15.16:0/4032544449 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4890006080 con 0x7f4898084f30 2019-12-05T06:53:35.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.123+0000 7f489d382700 1 -- 172.21.15.16:0/4032544449 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f489809aae0 con 0x7f4898084f30 2019-12-05T06:53:35.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.123+0000 7f489d382700 1 -- 172.21.15.16:0/4032544449 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f489809aae0 con 0x7f4898084f30 2019-12-05T06:53:35.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.124+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.125+0000 7f488ffff700 1 -- 172.21.15.16:0/4032544449 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f48900058b0 con 0x7f4898084f30 2019-12-05T06:53:35.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.125+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.126+0000 7f488ffff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f487c023630 0x7f487c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:35.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.126+0000 7f488ffff700 1 -- 172.21.15.16:0/4032544449 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f4890046120 con 0x7f4898084f30 2019-12-05T06:53:35.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.126+0000 7f48977fe700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f487c023630 0x7f487c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:35.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.127+0000 7f48977fe700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f487c023630 0x7f487c025ac0 crc :-1 s=READY pgs=629 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.139+0000 7f489d382700 1 -- 172.21.15.16:0/4032544449 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f489809aae0 con 0x7f4898084f30 2019-12-05T06:53:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.148+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.148+0000 7f488ffff700 1 -- 172.21.15.16:0/4032544449 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4890007db0 con 0x7f4898084f30 2019-12-05T06:53:35.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.409+0000 7f489d382700 1 -- 172.21.15.16:0/4032544449 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f489809aae0 con 0x7f4898084f30 2019-12-05T06:53:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.410+0000 7f4897fff700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=THROTTLE_DONE pgs=349 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:35.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.410+0000 7f488ffff700 1 -- 172.21.15.16:0/4032544449 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f4890006160 con 0x7f4898084f30 2019-12-05T06:53:35.412 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 182 2019-12-05T06:53:35.413 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:35.413 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6092,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":182}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:35.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.415+0000 7f489d382700 1 -- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f487c023630 msgr2=0x7f487c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:35.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.415+0000 7f489d382700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f487c023630 0x7f487c025ac0 crc :-1 s=READY pgs=629 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.415+0000 7f489d382700 1 -- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 msgr2=0x7f4898155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.415+0000 7f489d382700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 secure :-1 s=READY pgs=349 cs=0 l=1 rx=0x7f48900107b0 tx=0x7f4890010b50).stop 2019-12-05T06:53:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.415+0000 7f489d382700 1 -- 172.21.15.16:0/4032544449 shutdown_connections 2019-12-05T06:53:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.415+0000 7f489d382700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f487c023630 0x7f487c025ac0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.415+0000 7f489d382700 1 --2- 172.21.15.16:0/4032544449 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4898084f30 0x7f4898155f30 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:35.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.415+0000 7f489d382700 1 --2- 172.21.15.16:0/4032544449 >> v2:172.21.15.76:3300/0 conn(0x7f489809bbe0 0x7f48981539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.416+0000 7f489d382700 1 -- 172.21.15.16:0/4032544449 shutdown_connections 2019-12-05T06:53:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.416+0000 7f489d382700 1 -- 172.21.15.16:0/4032544449 wait complete. 2019-12-05T06:53:35.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:35.416+0000 7f489d382700 1 -- 172.21.15.16:0/4032544449 >> 172.21.15.16:0/4032544449 conn(0x7f48980714f0 msgr2=0x7f489809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:35.427 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:53:37.428 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:37.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.682+0000 7faf7c397700 1 Processor -- start 2019-12-05T06:53:37.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.682+0000 7faf7c397700 1 -- start start 2019-12-05T06:53:37.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.684+0000 7faf7c397700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 0x7faf7408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:37.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.684+0000 7faf7c397700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf740958e0 con 0x7faf7408f720 2019-12-05T06:53:37.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.684+0000 7faf7c397700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf740629e0 con 0x7faf7408fef0 2019-12-05T06:53:37.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.684+0000 7faf7c397700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf740a3f70 con 0x7faf7408c6d0 2019-12-05T06:53:37.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.685+0000 7faf7b135700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7faf7408f720 0x7faf741533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48490/0 (socket says 172.21.15.16:48490) 2019-12-05T06:53:37.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.685+0000 7faf7b135700 1 -- 172.21.15.16:0/146834596 learned_addr learned my addr 172.21.15.16:0/146834596 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:37.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.685+0000 7faf7a934700 1 --2- 172.21.15.16:0/146834596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 0x7faf7408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:37.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.685+0000 7faf79932700 1 -- 172.21.15.16:0/146834596 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 655514599 0 0) 0x7faf6c0014d0 con 0x7faf7408f720 2019-12-05T06:53:37.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.685+0000 7faf79932700 1 -- 172.21.15.16:0/146834596 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf5c002e80 con 0x7faf7408f720 2019-12-05T06:53:37.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.685+0000 7faf79932700 1 -- 172.21.15.16:0/146834596 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3539544081 0 0) 0x7faf68001080 con 0x7faf7408fef0 2019-12-05T06:53:37.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.685+0000 7faf79932700 1 -- 172.21.15.16:0/146834596 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf5c004220 con 0x7faf7408fef0 2019-12-05T06:53:37.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7a934700 1 -- 172.21.15.16:0/146834596 >> v1:172.21.15.76:6789/0 conn(0x7faf7408fef0 legacy=0x7faf7408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:37.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7a934700 1 -- 172.21.15.16:0/146834596 >> v1:172.21.15.16:6789/0 conn(0x7faf7408f720 legacy=0x7faf741533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:37.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7a934700 1 -- 172.21.15.16:0/146834596 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf7408fbc0 con 0x7faf7408c6d0 2019-12-05T06:53:37.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7a934700 1 --2- 172.21.15.16:0/146834596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 0x7faf7408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7faf6400af50 tx=0x7faf6400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7a934700 1 --2- 172.21.15.16:0/146834596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 0x7faf7408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7faf6400af50 tx=0x7faf6400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7a934700 1 --2- 172.21.15.16:0/146834596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 0x7faf7408cac0 secure :-1 s=READY pgs=350 cs=0 l=1 rx=0x7faf6400af50 tx=0x7faf6400b760).ready entity=mon.2 client_cookie=26d1b7560970abfa server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:37.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7a934700 1 --2- 172.21.15.16:0/146834596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 0x7faf7408cac0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7faf6400af50 tx=0x7faf6400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf79932700 1 -- 172.21.15.16:0/146834596 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf64000960 con 0x7faf7408c6d0 2019-12-05T06:53:37.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7a934700 1 --2- 172.21.15.16:0/146834596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 0x7faf7408cac0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7faf6400af50 tx=0x7faf6400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7a934700 1 --2- 172.21.15.16:0/146834596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 0x7faf7408cac0 secure :-1 s=THROTTLE_DONE pgs=350 cs=0 l=1 rx=0x7faf6400af50 tx=0x7faf6400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf79932700 1 -- 172.21.15.16:0/146834596 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf64009860 con 0x7faf7408c6d0 2019-12-05T06:53:37.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf79932700 1 -- 172.21.15.16:0/146834596 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf64009f30 con 0x7faf7408c6d0 2019-12-05T06:53:37.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7c397700 1 -- 172.21.15.16:0/146834596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 msgr2=0x7faf7408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:37.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.686+0000 7faf7c397700 1 --2- 172.21.15.16:0/146834596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 0x7faf7408cac0 secure :-1 s=READY pgs=350 cs=0 l=1 rx=0x7faf6400af50 tx=0x7faf6400b760).stop 2019-12-05T06:53:37.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.688+0000 7faf7c397700 1 -- 172.21.15.16:0/146834596 shutdown_connections 2019-12-05T06:53:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.688+0000 7faf7c397700 1 --2- 172.21.15.16:0/146834596 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7408c6d0 0x7faf7408cac0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.688+0000 7faf7c397700 1 -- 172.21.15.16:0/146834596 shutdown_connections 2019-12-05T06:53:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.688+0000 7faf7c397700 1 -- 172.21.15.16:0/146834596 wait complete. 2019-12-05T06:53:37.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.688+0000 7faf7c397700 1 -- 172.21.15.16:0/146834596 >> 172.21.15.16:0/146834596 conn(0x7faf74061940 msgr2=0x7faf74096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:37.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.693+0000 7faf7c397700 1 Processor -- start 2019-12-05T06:53:37.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.693+0000 7faf7c397700 1 -- start start 2019-12-05T06:53:37.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.694+0000 7faf7c397700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7faf741302f0 0x7faf7412a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:37.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.694+0000 7faf7c397700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7412b640 0x7faf7412abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:37.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.694+0000 7faf7c397700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf7412b0f0 0x7faf741306e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:37.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.694+0000 7faf7c397700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf741296e0 con 0x7faf7412b0f0 2019-12-05T06:53:37.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.694+0000 7faf7c397700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faf74128cb0 con 0x7faf741302f0 2019-12-05T06:53:37.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.694+0000 7faf7c397700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faf74128f80 con 0x7faf7412b640 2019-12-05T06:53:37.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7b135700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf7412b0f0 0x7faf741306e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:37.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7b135700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7faf7412b0f0 0x7faf741306e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35094/0 (socket says 172.21.15.16:35094) 2019-12-05T06:53:37.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7b135700 1 -- 172.21.15.16:0/900890099 learned_addr learned my addr 172.21.15.16:0/900890099 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:37.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7a934700 1 --2- 172.21.15.16:0/900890099 >> v2:172.21.15.76:3300/0 conn(0x7faf741302f0 0x7faf7412a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:37.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7a133700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7412b640 0x7faf7412abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:37.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7b135700 1 -- 172.21.15.16:0/900890099 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7412b640 msgr2=0x7faf7412abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:37.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7412b640 0x7faf7412abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7b135700 1 -- 172.21.15.16:0/900890099 >> v2:172.21.15.76:3300/0 conn(0x7faf741302f0 msgr2=0x7faf7412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> v2:172.21.15.76:3300/0 conn(0x7faf741302f0 0x7faf7412a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:37.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7b135700 1 -- 172.21.15.16:0/900890099 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf74129d70 con 0x7faf7412b0f0 2019-12-05T06:53:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> v2:172.21.15.16:3300/0 conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.695+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> v2:172.21.15.16:3300/0 conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.696+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.696+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.696+0000 7faf737fe700 1 -- 172.21.15.16:0/900890099 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf6c001270 con 0x7faf7412b0f0 2019-12-05T06:53:37.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.696+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.696+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.696+0000 7faf737fe700 1 -- 172.21.15.16:0/900890099 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf6c0045c0 con 0x7faf7412b0f0 2019-12-05T06:53:37.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.696+0000 7faf7c397700 1 -- 172.21.15.16:0/900890099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf74129d70 con 0x7faf7412b0f0 2019-12-05T06:53:37.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.696+0000 7faf737fe700 1 -- 172.21.15.16:0/900890099 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf6c004c10 con 0x7faf7412b0f0 2019-12-05T06:53:37.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.696+0000 7faf7c397700 1 -- 172.21.15.16:0/900890099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf74129d70 con 0x7faf7412b0f0 2019-12-05T06:53:37.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.697+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.698+0000 7faf737fe700 1 -- 172.21.15.16:0/900890099 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7faf6c0106e0 con 0x7faf7412b0f0 2019-12-05T06:53:37.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.698+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.698+0000 7faf737fe700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf54023630 0x7faf54025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:37.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.698+0000 7faf737fe700 1 -- 172.21.15.16:0/900890099 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7faf6c0460e0 con 0x7faf7412b0f0 2019-12-05T06:53:37.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.698+0000 7faf7a934700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf54023630 0x7faf54025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:37.732 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.699+0000 7faf7a934700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf54023630 0x7faf54025ac0 crc :-1 s=READY pgs=630 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:37.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.714+0000 7faf7c397700 1 -- 172.21.15.16:0/900890099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf74129d70 con 0x7faf7412b0f0 2019-12-05T06:53:37.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.722+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.722+0000 7faf737fe700 1 -- 172.21.15.16:0/900890099 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7faf6c000cc0 con 0x7faf7412b0f0 2019-12-05T06:53:37.966 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.965+0000 7faf7c397700 1 -- 172.21.15.16:0/900890099 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7faf74129d70 con 0x7faf7412b0f0 2019-12-05T06:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.966+0000 7faf7b135700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=THROTTLE_DONE pgs=1505 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:37.967 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.966+0000 7faf737fe700 1 -- 172.21.15.16:0/900890099 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+2109 (secure 0 0 0) 0x7faf6c0106e0 con 0x7faf7412b0f0 2019-12-05T06:53:37.968 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 182 2019-12-05T06:53:37.968 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:37.968 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6092,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":182}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:37.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.969+0000 7faf7c397700 1 -- 172.21.15.16:0/900890099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf54023630 msgr2=0x7faf54025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:37.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.969+0000 7faf7c397700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf54023630 0x7faf54025ac0 crc :-1 s=READY pgs=630 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:37.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.969+0000 7faf7c397700 1 -- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 msgr2=0x7faf741306e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:37.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.969+0000 7faf7c397700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 0x7faf741306e0 secure :-1 s=READY pgs=1505 cs=0 l=1 rx=0x7faf6c0129c0 tx=0x7faf6c010660).stop 2019-12-05T06:53:37.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.969+0000 7faf7c397700 1 -- 172.21.15.16:0/900890099 shutdown_connections 2019-12-05T06:53:37.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.969+0000 7faf7c397700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7faf54023630 0x7faf54025ac0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:37.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.969+0000 7faf7c397700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7faf7412b0f0 0x7faf741306e0 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:37.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.969+0000 7faf7c397700 1 --2- 172.21.15.16:0/900890099 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7faf7412b640 0x7faf7412abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:37.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.969+0000 7faf7c397700 1 --2- 172.21.15.16:0/900890099 >> v2:172.21.15.76:3300/0 conn(0x7faf741302f0 0x7faf7412a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:37.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.970+0000 7faf7c397700 1 -- 172.21.15.16:0/900890099 shutdown_connections 2019-12-05T06:53:37.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.970+0000 7faf7c397700 1 -- 172.21.15.16:0/900890099 wait complete. 2019-12-05T06:53:37.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:37.970+0000 7faf7c397700 1 -- 172.21.15.16:0/900890099 >> 172.21.15.16:0/900890099 conn(0x7faf740a2850 msgr2=0x7faf7406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:37.986 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:53:39.998 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:40.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.284+0000 7f8bd717c700 1 Processor -- start 2019-12-05T06:53:40.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.284+0000 7f8bd717c700 1 -- start start 2019-12-05T06:53:40.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.286+0000 7f8bd717c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bd008c6d0 0x7f8bd008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:40.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.286+0000 7f8bd717c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd008fb80 con 0x7f8bd008f720 2019-12-05T06:53:40.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.286+0000 7f8bd717c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd00958e0 con 0x7f8bd008fef0 2019-12-05T06:53:40.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.286+0000 7f8bd717c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd008df90 con 0x7f8bd008c6d0 2019-12-05T06:53:40.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.286+0000 7f8bd5719700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bd008c6d0 0x7f8bd008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:40.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.286+0000 7f8bd5f1a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8bd008f720 0x7f8bd01533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48504/0 (socket says 172.21.15.16:48504) 2019-12-05T06:53:40.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.286+0000 7f8bd5f1a700 1 -- 172.21.15.16:0/2575152375 learned_addr learned my addr 172.21.15.16:0/2575152375 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.287+0000 7f8bc7fff700 1 -- 172.21.15.16:0/2575152375 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2615608865 0 0) 0x7f8bc8001150 con 0x7f8bd008f720 2019-12-05T06:53:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.287+0000 7f8bc7fff700 1 -- 172.21.15.16:0/2575152375 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bbc002e80 con 0x7f8bd008f720 2019-12-05T06:53:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.287+0000 7f8bc7fff700 1 -- 172.21.15.16:0/2575152375 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3614971369 0 0) 0x7f8bc0001080 con 0x7f8bd008fef0 2019-12-05T06:53:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.287+0000 7f8bd5719700 1 -- 172.21.15.16:0/2575152375 >> v1:172.21.15.76:6789/0 conn(0x7f8bd008fef0 legacy=0x7f8bd008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:40.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.287+0000 7f8bd5719700 1 -- 172.21.15.16:0/2575152375 >> v1:172.21.15.16:6789/0 conn(0x7f8bd008f720 legacy=0x7f8bd01533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:40.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.287+0000 7f8bd5719700 1 -- 172.21.15.16:0/2575152375 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd008e620 con 0x7f8bd008c6d0 2019-12-05T06:53:40.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.287+0000 7f8bd5719700 1 --2- 172.21.15.16:0/2575152375 >> v2:172.21.15.16:3300/0 conn(0x7f8bd008c6d0 0x7f8bd008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8bcc00b2d0 tx=0x7f8bcc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.288+0000 7f8bd5719700 1 --2- 172.21.15.16:0/2575152375 >> v2:172.21.15.16:3300/0 conn(0x7f8bd008c6d0 0x7f8bd008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8bcc00b2d0 tx=0x7f8bcc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.288+0000 7f8bd5719700 1 --2- 172.21.15.16:0/2575152375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd008c6d0 0x7f8bd008cac0 secure :-1 s=READY pgs=1507 cs=0 l=1 rx=0x7f8bcc00b2d0 tx=0x7f8bcc00bae0).ready entity=mon.0 client_cookie=e1b0bf972f256049 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:40.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.288+0000 7f8bd5719700 1 --2- 172.21.15.16:0/2575152375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd008c6d0 0x7f8bd008cac0 secure :-1 s=THROTTLE_DONE pgs=1507 cs=0 l=1 rx=0x7f8bcc00b2d0 tx=0x7f8bcc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.288+0000 7f8bc7fff700 1 -- 172.21.15.16:0/2575152375 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bcc000ce0 con 0x7f8bd008c6d0 2019-12-05T06:53:40.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.288+0000 7f8bd5719700 1 --2- 172.21.15.16:0/2575152375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd008c6d0 0x7f8bd008cac0 secure :-1 s=THROTTLE_DONE pgs=1507 cs=0 l=1 rx=0x7f8bcc00b2d0 tx=0x7f8bcc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.288+0000 7f8bd5719700 1 --2- 172.21.15.16:0/2575152375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd008c6d0 0x7f8bd008cac0 secure :-1 s=THROTTLE_DONE pgs=1507 cs=0 l=1 rx=0x7f8bcc00b2d0 tx=0x7f8bcc00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.288+0000 7f8bc7fff700 1 -- 172.21.15.16:0/2575152375 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bcc009be0 con 0x7f8bd008c6d0 2019-12-05T06:53:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.288+0000 7f8bc7fff700 1 -- 172.21.15.16:0/2575152375 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bcc00a2b0 con 0x7f8bd008c6d0 2019-12-05T06:53:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.288+0000 7f8bd717c700 1 -- 172.21.15.16:0/2575152375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd008c6d0 msgr2=0x7f8bd008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.288+0000 7f8bd717c700 1 --2- 172.21.15.16:0/2575152375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd008c6d0 0x7f8bd008cac0 secure :-1 s=READY pgs=1507 cs=0 l=1 rx=0x7f8bcc00b2d0 tx=0x7f8bcc00bae0).stop 2019-12-05T06:53:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.290+0000 7f8bd717c700 1 -- 172.21.15.16:0/2575152375 shutdown_connections 2019-12-05T06:53:40.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.290+0000 7f8bd717c700 1 --2- 172.21.15.16:0/2575152375 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd008c6d0 0x7f8bd008cac0 unknown :-1 s=CLOSED pgs=1507 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:40.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.291+0000 7f8bd717c700 1 -- 172.21.15.16:0/2575152375 shutdown_connections 2019-12-05T06:53:40.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.291+0000 7f8bd717c700 1 -- 172.21.15.16:0/2575152375 wait complete. 2019-12-05T06:53:40.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.291+0000 7f8bd717c700 1 -- 172.21.15.16:0/2575152375 >> 172.21.15.16:0/2575152375 conn(0x7f8bd0061940 msgr2=0x7f8bd0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:40.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.296+0000 7f8bd717c700 1 Processor -- start 2019-12-05T06:53:40.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.296+0000 7f8bd717c700 1 -- start start 2019-12-05T06:53:40.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.298+0000 7f8bd717c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bd009bbe0 0x7f8bd01539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:40.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.298+0000 7f8bd717c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd0084b90 con 0x7f8bd0084f30 2019-12-05T06:53:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.298+0000 7f8bd717c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bd0098790 con 0x7f8bd0096e90 2019-12-05T06:53:40.304 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.298+0000 7f8bd717c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bd009a450 con 0x7f8bd009bbe0 2019-12-05T06:53:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.298+0000 7f8bd4f18700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8bd009bbe0 0x7f8bd01539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.298+0000 7f8bd5f1a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8bd0084f30 0x7f8bd0155f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48512/0 (socket says 172.21.15.16:48512) 2019-12-05T06:53:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.298+0000 7f8bd5f1a700 1 -- 172.21.15.16:0/1506505917 learned_addr learned my addr 172.21.15.16:0/1506505917 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.298+0000 7f8bd4f18700 1 -- 172.21.15.16:0/1506505917 >> v1:172.21.15.76:6789/0 conn(0x7f8bd0096e90 legacy=0x7f8bd009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:40.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.298+0000 7f8bd4f18700 1 -- 172.21.15.16:0/1506505917 >> v1:172.21.15.16:6789/0 conn(0x7f8bd0084f30 legacy=0x7f8bd0155f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.298+0000 7f8bd4f18700 1 -- 172.21.15.16:0/1506505917 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bd009aae0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bd4f18700 1 --2- 172.21.15.16:0/1506505917 >> v2:172.21.15.16:3300/0 conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bd4f18700 1 --2- 172.21.15.16:0/1506505917 >> v2:172.21.15.16:3300/0 conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bd4f18700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=READY pgs=1509 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bd4f18700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=THROTTLE_DONE pgs=1509 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.306 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bc5ffb700 1 -- 172.21.15.16:0/1506505917 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bc0000ea0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bd4f18700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=THROTTLE_DONE pgs=1509 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bd4f18700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=THROTTLE_DONE pgs=1509 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bc5ffb700 1 -- 172.21.15.16:0/1506505917 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bc00058e0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bd717c700 1 -- 172.21.15.16:0/1506505917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bd009aae0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bc5ffb700 1 -- 172.21.15.16:0/1506505917 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bc0005f70 con 0x7f8bd009bbe0 2019-12-05T06:53:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.299+0000 7f8bd717c700 1 -- 172.21.15.16:0/1506505917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bd009aae0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.300+0000 7f8bd4f18700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=THROTTLE_DONE pgs=1509 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.302+0000 7f8bc5ffb700 1 -- 172.21.15.16:0/1506505917 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8bc00054e0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.302+0000 7f8bd4f18700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=THROTTLE_DONE pgs=1509 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.308 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.302+0000 7f8bc5ffb700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8bb0023630 0x7f8bb0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.302+0000 7f8bc5ffb700 1 -- 172.21.15.16:0/1506505917 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f8bc0045da0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.302+0000 7f8bd5719700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8bb0023630 0x7f8bb0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:40.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.303+0000 7f8bd5719700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8bb0023630 0x7f8bb0025ac0 crc :-1 s=READY pgs=631 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:40.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.316+0000 7f8bd717c700 1 -- 172.21.15.16:0/1506505917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bd009aae0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.325+0000 7f8bd4f18700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=THROTTLE_DONE pgs=1509 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.325+0000 7f8bc5ffb700 1 -- 172.21.15.16:0/1506505917 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8bc0000ea0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.560+0000 7f8bd717c700 1 -- 172.21.15.16:0/1506505917 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8bd009aae0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.561+0000 7f8bd4f18700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=THROTTLE_DONE pgs=1509 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.561+0000 7f8bc5ffb700 1 -- 172.21.15.16:0/1506505917 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f8bc00054e0 con 0x7f8bd009bbe0 2019-12-05T06:53:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 182 2019-12-05T06:53:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.563+0000 7f8bd717c700 1 -- 172.21.15.16:0/1506505917 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8bb0023630 msgr2=0x7f8bb0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.563+0000 7f8bd717c700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8bb0023630 0x7f8bb0025ac0 crc :-1 s=READY pgs=631 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.563+0000 7f8bd717c700 1 -- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 msgr2=0x7f8bd01539e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.563+0000 7f8bd717c700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 0x7f8bd01539e0 secure :-1 s=READY pgs=1509 cs=0 l=1 rx=0x7f8bc0010380 tx=0x7f8bc0010720).stop 2019-12-05T06:53:40.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.564+0000 7f8bd717c700 1 -- 172.21.15.16:0/1506505917 shutdown_connections 2019-12-05T06:53:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.564+0000 7f8bd717c700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8bb0023630 0x7f8bb0025ac0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.564+0000 7f8bd717c700 1 --2- 172.21.15.16:0/1506505917 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8bd009bbe0 0x7f8bd01539e0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.564+0000 7f8bd717c700 1 -- 172.21.15.16:0/1506505917 shutdown_connections 2019-12-05T06:53:40.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.564+0000 7f8bd717c700 1 -- 172.21.15.16:0/1506505917 wait complete. 2019-12-05T06:53:40.567 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:40.567 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6092,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":182}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:40.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:40.564+0000 7f8bd717c700 1 -- 172.21.15.16:0/1506505917 >> 172.21.15.16:0/1506505917 conn(0x7f8bd00714f0 msgr2=0x7f8bd009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:40.575 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:53:42.576 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:42.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.824+0000 7f0c46309700 1 Processor -- start 2019-12-05T06:53:42.835 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.825+0000 7f0c46309700 1 -- start start 2019-12-05T06:53:42.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.826+0000 7f0c46309700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c4008c6d0 0x7f0c4008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:42.836 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.826+0000 7f0c46309700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4008fb80 con 0x7f0c4008f720 2019-12-05T06:53:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.826+0000 7f0c46309700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c400958e0 con 0x7f0c4008fef0 2019-12-05T06:53:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.826+0000 7f0c46309700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4008df90 con 0x7f0c4008c6d0 2019-12-05T06:53:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.827+0000 7f0c448a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c4008c6d0 0x7f0c4008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:42.837 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.827+0000 7f0c448a6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c4008c6d0 0x7f0c4008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35124/0 (socket says 172.21.15.16:35124) 2019-12-05T06:53:42.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.827+0000 7f0c448a6700 1 -- 172.21.15.16:0/1254288314 learned_addr learned my addr 172.21.15.16:0/1254288314 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:42.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.827+0000 7f0c450a7700 1 --1- 172.21.15.16:0/1254288314 >> v1:172.21.15.16:6789/0 conn(0x7f0c4008f720 0x7f0c401533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48524/0 (socket says 172.21.15.16:48524) 2019-12-05T06:53:42.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.827+0000 7f0c448a6700 1 -- 172.21.15.16:0/1254288314 >> v1:172.21.15.76:6789/0 conn(0x7f0c4008fef0 legacy=0x7f0c4008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:42.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.827+0000 7f0c448a6700 1 -- 172.21.15.16:0/1254288314 >> v1:172.21.15.16:6789/0 conn(0x7f0c4008f720 legacy=0x7f0c401533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:42.838 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.827+0000 7f0c3f7fe700 1 -- 172.21.15.16:0/1254288314 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 827319985 0 0) 0x7f0c380009b0 con 0x7f0c4008f720 2019-12-05T06:53:42.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.827+0000 7f0c448a6700 1 -- 172.21.15.16:0/1254288314 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4008e620 con 0x7f0c4008c6d0 2019-12-05T06:53:42.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c448a6700 1 --2- 172.21.15.16:0/1254288314 >> v2:172.21.15.16:3300/0 conn(0x7f0c4008c6d0 0x7f0c4008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f0c28008400 tx=0x7f0c2800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c448a6700 1 --2- 172.21.15.16:0/1254288314 >> v2:172.21.15.16:3300/0 conn(0x7f0c4008c6d0 0x7f0c4008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f0c28008400 tx=0x7f0c2800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.839 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c448a6700 1 --2- 172.21.15.16:0/1254288314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c4008c6d0 0x7f0c4008cac0 secure :-1 s=READY pgs=1511 cs=0 l=1 rx=0x7f0c28008400 tx=0x7f0c2800fad0).ready entity=mon.0 client_cookie=b87c9552946f2bc6 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:42.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c448a6700 1 --2- 172.21.15.16:0/1254288314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c4008c6d0 0x7f0c4008cac0 secure :-1 s=THROTTLE_DONE pgs=1511 cs=0 l=1 rx=0x7f0c28008400 tx=0x7f0c2800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c3f7fe700 1 -- 172.21.15.16:0/1254288314 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c28000960 con 0x7f0c4008c6d0 2019-12-05T06:53:42.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c448a6700 1 --2- 172.21.15.16:0/1254288314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c4008c6d0 0x7f0c4008cac0 secure :-1 s=THROTTLE_DONE pgs=1511 cs=0 l=1 rx=0x7f0c28008400 tx=0x7f0c2800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c448a6700 1 --2- 172.21.15.16:0/1254288314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c4008c6d0 0x7f0c4008cac0 secure :-1 s=THROTTLE_DONE pgs=1511 cs=0 l=1 rx=0x7f0c28008400 tx=0x7f0c2800fad0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.840 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c3f7fe700 1 -- 172.21.15.16:0/1254288314 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c28000c90 con 0x7f0c4008c6d0 2019-12-05T06:53:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c3f7fe700 1 -- 172.21.15.16:0/1254288314 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c28001360 con 0x7f0c4008c6d0 2019-12-05T06:53:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c46309700 1 -- 172.21.15.16:0/1254288314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c4008c6d0 msgr2=0x7f0c4008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.828+0000 7f0c46309700 1 --2- 172.21.15.16:0/1254288314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c4008c6d0 0x7f0c4008cac0 secure :-1 s=READY pgs=1511 cs=0 l=1 rx=0x7f0c28008400 tx=0x7f0c2800fad0).stop 2019-12-05T06:53:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.830+0000 7f0c46309700 1 -- 172.21.15.16:0/1254288314 shutdown_connections 2019-12-05T06:53:42.841 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.830+0000 7f0c46309700 1 --2- 172.21.15.16:0/1254288314 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c4008c6d0 0x7f0c4008cac0 unknown :-1 s=CLOSED pgs=1511 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:42.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.830+0000 7f0c46309700 1 -- 172.21.15.16:0/1254288314 shutdown_connections 2019-12-05T06:53:42.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.830+0000 7f0c46309700 1 -- 172.21.15.16:0/1254288314 wait complete. 2019-12-05T06:53:42.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.830+0000 7f0c46309700 1 -- 172.21.15.16:0/1254288314 >> 172.21.15.16:0/1254288314 conn(0x7f0c40061940 msgr2=0x7f0c40096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:42.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.835+0000 7f0c46309700 1 Processor -- start 2019-12-05T06:53:42.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.835+0000 7f0c46309700 1 -- start start 2019-12-05T06:53:42.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.837+0000 7f0c46309700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0c40096e90 0x7f0c4009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:42.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.837+0000 7f0c46309700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c40084f30 0x7f0c4009b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:42.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.837+0000 7f0c46309700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c40084b90 con 0x7f0c40097280 2019-12-05T06:53:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.837+0000 7f0c46309700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c40099a20 con 0x7f0c40084f30 2019-12-05T06:53:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.837+0000 7f0c46309700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c40099cf0 con 0x7f0c40096e90 2019-12-05T06:53:42.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.837+0000 7f0c3ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c40084f30 0x7f0c4009b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.837+0000 7f0c3ffff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f0c40084f30 0x7f0c4009b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35130/0 (socket says 172.21.15.16:35130) 2019-12-05T06:53:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.837+0000 7f0c3ffff700 1 -- 172.21.15.16:0/2248330553 learned_addr learned my addr 172.21.15.16:0/2248330553 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.837+0000 7f0c448a6700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0c40096e90 0x7f0c4009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3d7fa700 1 -- 172.21.15.16:0/2248330553 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3079690280 0 0) 0x7f0c38000ef0 con 0x7f0c40097280 2019-12-05T06:53:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3ffff700 1 -- 172.21.15.16:0/2248330553 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0c40096e90 msgr2=0x7f0c4009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:42.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0c40096e90 0x7f0c4009b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3ffff700 1 -- 172.21.15.16:0/2248330553 >> v1:172.21.15.76:6789/0 conn(0x7f0c40097280 legacy=0x7f0c40097700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3ffff700 1 -- 172.21.15.16:0/2248330553 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4009aae0 con 0x7f0c40084f30 2019-12-05T06:53:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> v2:172.21.15.16:3300/0 conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> v2:172.21.15.16:3300/0 conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.852 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=READY pgs=1512 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3d7fa700 1 -- 172.21.15.16:0/2248330553 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c30005aa0 con 0x7f0c40084f30 2019-12-05T06:53:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.838+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.839+0000 7f0c3d7fa700 1 -- 172.21.15.16:0/2248330553 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c300056c0 con 0x7f0c40084f30 2019-12-05T06:53:42.853 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.839+0000 7f0c46309700 1 -- 172.21.15.16:0/2248330553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c4009aae0 con 0x7f0c40084f30 2019-12-05T06:53:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.839+0000 7f0c3d7fa700 1 -- 172.21.15.16:0/2248330553 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c30006080 con 0x7f0c40084f30 2019-12-05T06:53:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.839+0000 7f0c46309700 1 -- 172.21.15.16:0/2248330553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4009aae0 con 0x7f0c40084f30 2019-12-05T06:53:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.839+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.841+0000 7f0c3d7fa700 1 -- 172.21.15.16:0/2248330553 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0c300058b0 con 0x7f0c40084f30 2019-12-05T06:53:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.841+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.841+0000 7f0c3d7fa700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0c24023630 0x7f0c24025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:42.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.841+0000 7f0c3d7fa700 1 -- 172.21.15.16:0/2248330553 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 8611+0+0 (secure 0 0 0) 0x7f0c30046180 con 0x7f0c40084f30 2019-12-05T06:53:42.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.841+0000 7f0c448a6700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0c24023630 0x7f0c24025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:42.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.842+0000 7f0c448a6700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0c24023630 0x7f0c24025ac0 crc :-1 s=READY pgs=632 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:42.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.856+0000 7f0c46309700 1 -- 172.21.15.16:0/2248330553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c4009aae0 con 0x7f0c40084f30 2019-12-05T06:53:42.863 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.863+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:42.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:42.863+0000 7f0c3d7fa700 1 -- 172.21.15.16:0/2248330553 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0c30007930 con 0x7f0c40084f30 2019-12-05T06:53:43.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.105+0000 7f0c46309700 1 -- 172.21.15.16:0/2248330553 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0c4009aae0 con 0x7f0c40084f30 2019-12-05T06:53:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.105+0000 7f0c3ffff700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=THROTTLE_DONE pgs=1512 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.105+0000 7f0c3d7fa700 1 -- 172.21.15.16:0/2248330553 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 182 v182) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f0c30005880 con 0x7f0c40084f30 2019-12-05T06:53:43.107 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 182 2019-12-05T06:53:43.107 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:43.108 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":182,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6092,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":182}],"filesystems":[{"mdsmap":{"epoch":181,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:13.686092+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":79,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7857},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7857":{"gid":7857,"name":"a","rank":0,"incarnation":178,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/359492061","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":359492061},{"type":"v1","addr":"172.21.15.16:6833","nonce":359492061}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:43.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.108+0000 7f0c46309700 1 -- 172.21.15.16:0/2248330553 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0c24023630 msgr2=0x7f0c24025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:43.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.108+0000 7f0c46309700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0c24023630 0x7f0c24025ac0 crc :-1 s=READY pgs=632 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:43.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.108+0000 7f0c46309700 1 -- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 msgr2=0x7f0c4009b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:43.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.108+0000 7f0c46309700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 0x7f0c4009b920 secure :-1 s=READY pgs=1512 cs=0 l=1 rx=0x7f0c30010790 tx=0x7f0c300105f0).stop 2019-12-05T06:53:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.108+0000 7f0c46309700 1 -- 172.21.15.16:0/2248330553 shutdown_connections 2019-12-05T06:53:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.108+0000 7f0c46309700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0c24023630 0x7f0c24025ac0 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:43.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.108+0000 7f0c46309700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f0c40084f30 0x7f0c4009b920 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:43.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.108+0000 7f0c46309700 1 --2- 172.21.15.16:0/2248330553 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f0c40096e90 0x7f0c4009b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:43.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.109+0000 7f0c46309700 1 -- 172.21.15.16:0/2248330553 shutdown_connections 2019-12-05T06:53:43.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.109+0000 7f0c46309700 1 -- 172.21.15.16:0/2248330553 wait complete. 2019-12-05T06:53:43.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:43.109+0000 7f0c46309700 1 -- 172.21.15.16:0/2248330553 >> 172.21.15.16:0/2248330553 conn(0x7f0c400714f0 msgr2=0x7f0c4009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:43.120 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:53:45.121 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.374+0000 7f6184038700 1 Processor -- start 2019-12-05T06:53:45.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.374+0000 7f6184038700 1 -- start start 2019-12-05T06:53:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.376+0000 7f6184038700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f617c08c6d0 0x7f617c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.376+0000 7f6184038700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f617c08fef0 0x7f617c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.376+0000 7f6184038700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617c08fb80 con 0x7f617c08ef20 2019-12-05T06:53:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.376+0000 7f6184038700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617c08df90 con 0x7f617c08c6d0 2019-12-05T06:53:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.376+0000 7f6184038700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617c0a3f70 con 0x7f617c08fef0 2019-12-05T06:53:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.376+0000 7f61825d5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f617c08c6d0 0x7f617c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:45.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.376+0000 7f61825d5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f617c08c6d0 0x7f617c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35140/0 (socket says 172.21.15.16:35140) 2019-12-05T06:53:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.376+0000 7f61825d5700 1 -- 172.21.15.16:0/3652146533 learned_addr learned my addr 172.21.15.16:0/3652146533 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.376+0000 7f6181dd4700 1 --2- 172.21.15.16:0/3652146533 >> v2:172.21.15.76:3300/0 conn(0x7f617c08fef0 0x7f617c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.377+0000 7f61825d5700 1 -- 172.21.15.16:0/3652146533 >> v2:172.21.15.76:3300/0 conn(0x7f617c08fef0 msgr2=0x7f617c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.377+0000 7f61825d5700 1 --2- 172.21.15.16:0/3652146533 >> v2:172.21.15.76:3300/0 conn(0x7f617c08fef0 0x7f617c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.377+0000 7f61825d5700 1 -- 172.21.15.16:0/3652146533 >> v1:172.21.15.76:6789/0 conn(0x7f617c08ef20 legacy=0x7f617c08f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.377+0000 7f61825d5700 1 -- 172.21.15.16:0/3652146533 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c08d540 con 0x7f617c08c6d0 2019-12-05T06:53:45.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.377+0000 7f61815d3700 1 -- 172.21.15.16:0/3652146533 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 762215046 0 0) 0x7f616c0009b0 con 0x7f617c08ef20 2019-12-05T06:53:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.377+0000 7f6181dd4700 1 --2- 172.21.15.16:0/3652146533 >> v2:172.21.15.76:3300/0 conn(0x7f617c08fef0 0x7f617c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:53:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.377+0000 7f61825d5700 1 --2- 172.21.15.16:0/3652146533 >> v2:172.21.15.16:3300/0 conn(0x7f617c08c6d0 0x7f617c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f617400fac0 tx=0x7f617400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.377+0000 7f61825d5700 1 --2- 172.21.15.16:0/3652146533 >> v2:172.21.15.16:3300/0 conn(0x7f617c08c6d0 0x7f617c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f617400fac0 tx=0x7f617400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.377+0000 7f61825d5700 1 --2- 172.21.15.16:0/3652146533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c08c6d0 0x7f617c08cac0 secure :-1 s=READY pgs=1514 cs=0 l=1 rx=0x7f617400fac0 tx=0x7f617400fe60).ready entity=mon.0 client_cookie=d1787621b148b189 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.378+0000 7f61825d5700 1 --2- 172.21.15.16:0/3652146533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c08c6d0 0x7f617c08cac0 secure :-1 s=THROTTLE_DONE pgs=1514 cs=0 l=1 rx=0x7f617400fac0 tx=0x7f617400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.378+0000 7f61825d5700 1 --2- 172.21.15.16:0/3652146533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c08c6d0 0x7f617c08cac0 secure :-1 s=THROTTLE_DONE pgs=1514 cs=0 l=1 rx=0x7f617400fac0 tx=0x7f617400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.378+0000 7f61815d3700 1 -- 172.21.15.16:0/3652146533 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6174000ce0 con 0x7f617c08c6d0 2019-12-05T06:53:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.378+0000 7f61825d5700 1 --2- 172.21.15.16:0/3652146533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c08c6d0 0x7f617c08cac0 secure :-1 s=THROTTLE_DONE pgs=1514 cs=0 l=1 rx=0x7f617400fac0 tx=0x7f617400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.378+0000 7f61815d3700 1 -- 172.21.15.16:0/3652146533 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6174001010 con 0x7f617c08c6d0 2019-12-05T06:53:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.378+0000 7f61815d3700 1 -- 172.21.15.16:0/3652146533 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6174001760 con 0x7f617c08c6d0 2019-12-05T06:53:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.378+0000 7f6184038700 1 -- 172.21.15.16:0/3652146533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c08c6d0 msgr2=0x7f617c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:45.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.378+0000 7f6184038700 1 --2- 172.21.15.16:0/3652146533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c08c6d0 0x7f617c08cac0 secure :-1 s=READY pgs=1514 cs=0 l=1 rx=0x7f617400fac0 tx=0x7f617400fe60).stop 2019-12-05T06:53:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.380+0000 7f6184038700 1 -- 172.21.15.16:0/3652146533 shutdown_connections 2019-12-05T06:53:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.380+0000 7f6184038700 1 --2- 172.21.15.16:0/3652146533 >> v2:172.21.15.76:3300/0 conn(0x7f617c08fef0 0x7f617c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.380+0000 7f6184038700 1 --2- 172.21.15.16:0/3652146533 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c08c6d0 0x7f617c08cac0 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.380+0000 7f6184038700 1 -- 172.21.15.16:0/3652146533 shutdown_connections 2019-12-05T06:53:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.380+0000 7f6184038700 1 -- 172.21.15.16:0/3652146533 wait complete. 2019-12-05T06:53:45.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.380+0000 7f6184038700 1 -- 172.21.15.16:0/3652146533 >> 172.21.15.16:0/3652146533 conn(0x7f617c061940 msgr2=0x7f617c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:45.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.385+0000 7f6184038700 1 Processor -- start 2019-12-05T06:53:45.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.385+0000 7f6184038700 1 -- start start 2019-12-05T06:53:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.387+0000 7f6184038700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f617c132630 0x7f617c132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.387+0000 7f6184038700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f617c09ae80 0x7f617c132f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.387+0000 7f6184038700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f617c06e7c0 con 0x7f617c099eb0 2019-12-05T06:53:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.387+0000 7f6184038700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f617c098f20 con 0x7f617c132630 2019-12-05T06:53:45.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.387+0000 7f6184038700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f617c134390 con 0x7f617c09ae80 2019-12-05T06:53:45.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.387+0000 7f61825d5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f617c132630 0x7f617c132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:45.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.387+0000 7f61825d5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f617c132630 0x7f617c132a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35146/0 (socket says 172.21.15.16:35146) 2019-12-05T06:53:45.394 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.387+0000 7f61825d5700 1 -- 172.21.15.16:0/2657095737 learned_addr learned my addr 172.21.15.16:0/2657095737 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:45.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.387+0000 7f6181dd4700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f617c09ae80 0x7f617c132f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:45.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f61825d5700 1 -- 172.21.15.16:0/2657095737 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f617c09ae80 msgr2=0x7f617c132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:45.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f617c09ae80 0x7f617c132f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:45.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f61825d5700 1 -- 172.21.15.16:0/2657095737 >> v1:172.21.15.76:6789/0 conn(0x7f617c099eb0 legacy=0x7f617c09a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:45.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f61825d5700 1 -- 172.21.15.16:0/2657095737 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f617c0995b0 con 0x7f617c132630 2019-12-05T06:53:45.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f617affd700 1 -- 172.21.15.16:0/2657095737 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837626617 0 0) 0x7f616c001240 con 0x7f617c099eb0 2019-12-05T06:53:45.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> v2:172.21.15.16:3300/0 conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> v2:172.21.15.16:3300/0 conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=READY pgs=1515 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:45.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f617affd700 1 -- 172.21.15.16:0/2657095737 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6174005af0 con 0x7f617c132630 2019-12-05T06:53:45.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f617affd700 1 -- 172.21.15.16:0/2657095737 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6174005710 con 0x7f617c132630 2019-12-05T06:53:45.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f6184038700 1 -- 172.21.15.16:0/2657095737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f617c0995b0 con 0x7f617c132630 2019-12-05T06:53:45.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f617affd700 1 -- 172.21.15.16:0/2657095737 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6174005900 con 0x7f617c132630 2019-12-05T06:53:45.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.388+0000 7f6184038700 1 -- 172.21.15.16:0/2657095737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f617c0995b0 con 0x7f617c132630 2019-12-05T06:53:45.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.389+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.390+0000 7f617affd700 1 -- 172.21.15.16:0/2657095737 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6174005900 con 0x7f617c132630 2019-12-05T06:53:45.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.390+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.391+0000 7f617affd700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f615c023630 0x7f615c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:45.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.391+0000 7f617affd700 1 -- 172.21.15.16:0/2657095737 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f6174046180 con 0x7f617c132630 2019-12-05T06:53:45.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.391+0000 7f6181dd4700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f615c023630 0x7f615c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:45.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.392+0000 7f6181dd4700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f615c023630 0x7f615c025ac0 crc :-1 s=READY pgs=633 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:45.406 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.405+0000 7f6184038700 1 -- 172.21.15.16:0/2657095737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f617c0995b0 con 0x7f617c132630 2019-12-05T06:53:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.413+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.413+0000 7f617affd700 1 -- 172.21.15.16:0/2657095737 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6174005500 con 0x7f617c132630 2019-12-05T06:53:45.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.656+0000 7f6184038700 1 -- 172.21.15.16:0/2657095737 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f617c0995b0 con 0x7f617c132630 2019-12-05T06:53:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.657+0000 7f61825d5700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=THROTTLE_DONE pgs=1515 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.657+0000 7f617affd700 1 -- 172.21.15.16:0/2657095737 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f61740058e0 con 0x7f617c132630 2019-12-05T06:53:45.660 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 183 2019-12-05T06:53:45.660 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:45.660 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:43.590072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:45.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.662+0000 7f6184038700 1 -- 172.21.15.16:0/2657095737 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f615c023630 msgr2=0x7f615c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:45.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.662+0000 7f6184038700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f615c023630 0x7f615c025ac0 crc :-1 s=READY pgs=633 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:45.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.662+0000 7f6184038700 1 -- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 msgr2=0x7f617c132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:45.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.662+0000 7f6184038700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 0x7f617c132a20 secure :-1 s=READY pgs=1515 cs=0 l=1 rx=0x7f6174010750 tx=0x7f6174010af0).stop 2019-12-05T06:53:45.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.662+0000 7f6184038700 1 -- 172.21.15.16:0/2657095737 shutdown_connections 2019-12-05T06:53:45.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.662+0000 7f6184038700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f617c09ae80 0x7f617c132f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:45.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.662+0000 7f6184038700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f615c023630 0x7f615c025ac0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:45.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.662+0000 7f6184038700 1 --2- 172.21.15.16:0/2657095737 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f617c132630 0x7f617c132a20 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:45.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.662+0000 7f6184038700 1 -- 172.21.15.16:0/2657095737 shutdown_connections 2019-12-05T06:53:45.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.663+0000 7f6184038700 1 -- 172.21.15.16:0/2657095737 wait complete. 2019-12-05T06:53:45.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.663+0000 7f6184038700 1 -- 172.21.15.16:0/2657095737 >> 172.21.15.16:0/2657095737 conn(0x7f617c0714f0 msgr2=0x7f617c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:45.674 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:53:45.674 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:53:45.674 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:45.903 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.893+0000 7f1dea16b700 1 Processor -- start 2019-12-05T06:53:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.893+0000 7f1dea16b700 1 -- start start 2019-12-05T06:53:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.895+0000 7f1dea16b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1de408c6d0 0x7f1de408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.895+0000 7f1dea16b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 0x7f1de408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.895+0000 7f1dea16b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de40958e0 con 0x7f1de408ef20 2019-12-05T06:53:45.904 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.895+0000 7f1dea16b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de408df90 con 0x7f1de408c6d0 2019-12-05T06:53:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.895+0000 7f1dea16b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de40a3f70 con 0x7f1de408fef0 2019-12-05T06:53:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.895+0000 7f1de37fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 0x7f1de408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.896+0000 7f1de3fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f1de408c6d0 0x7f1de408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:45.905 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.896+0000 7f1de8f09700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f1de408ef20 0x7f1de408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48558/0 (socket says 172.21.15.16:48558) 2019-12-05T06:53:45.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.896+0000 7f1de8f09700 1 -- 172.21.15.16:0/2160319658 learned_addr learned my addr 172.21.15.16:0/2160319658 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:45.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.896+0000 7f1de2ffd700 1 -- 172.21.15.16:0/2160319658 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2188830572 0 0) 0x7f1ddc001150 con 0x7f1de408ef20 2019-12-05T06:53:45.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.896+0000 7f1de37fe700 1 -- 172.21.15.16:0/2160319658 >> v2:172.21.15.76:3300/0 conn(0x7f1de408c6d0 msgr2=0x7f1de408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:45.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.896+0000 7f1de37fe700 1 --2- 172.21.15.16:0/2160319658 >> v2:172.21.15.76:3300/0 conn(0x7f1de408c6d0 0x7f1de408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:45.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.896+0000 7f1de37fe700 1 -- 172.21.15.16:0/2160319658 >> v1:172.21.15.16:6789/0 conn(0x7f1de408ef20 legacy=0x7f1de408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.896+0000 7f1de37fe700 1 -- 172.21.15.16:0/2160319658 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de408d540 con 0x7f1de408fef0 2019-12-05T06:53:45.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.896+0000 7f1de3fff700 1 --2- 172.21.15.16:0/2160319658 >> v2:172.21.15.76:3300/0 conn(0x7f1de408c6d0 0x7f1de408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:53:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.897+0000 7f1de37fe700 1 --2- 172.21.15.16:0/2160319658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 0x7f1de408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f1dd000b2d0 tx=0x7f1dd000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.897+0000 7f1de37fe700 1 --2- 172.21.15.16:0/2160319658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 0x7f1de408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f1dd000b2d0 tx=0x7f1dd000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.897+0000 7f1de37fe700 1 --2- 172.21.15.16:0/2160319658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 0x7f1de408d000 secure :-1 s=READY pgs=351 cs=0 l=1 rx=0x7f1dd000b2d0 tx=0x7f1dd000bae0).ready entity=mon.2 client_cookie=657d50d6f699137f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.897+0000 7f1de37fe700 1 --2- 172.21.15.16:0/2160319658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 0x7f1de408d000 secure :-1 s=THROTTLE_DONE pgs=351 cs=0 l=1 rx=0x7f1dd000b2d0 tx=0x7f1dd000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.897+0000 7f1de37fe700 1 --2- 172.21.15.16:0/2160319658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 0x7f1de408d000 secure :-1 s=THROTTLE_DONE pgs=351 cs=0 l=1 rx=0x7f1dd000b2d0 tx=0x7f1dd000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.897+0000 7f1de2ffd700 1 -- 172.21.15.16:0/2160319658 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dd0007590 con 0x7f1de408fef0 2019-12-05T06:53:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.897+0000 7f1de37fe700 1 --2- 172.21.15.16:0/2160319658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 0x7f1de408d000 secure :-1 s=THROTTLE_DONE pgs=351 cs=0 l=1 rx=0x7f1dd000b2d0 tx=0x7f1dd000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.897+0000 7f1de2ffd700 1 -- 172.21.15.16:0/2160319658 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd0009d30 con 0x7f1de408fef0 2019-12-05T06:53:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.897+0000 7f1de2ffd700 1 -- 172.21.15.16:0/2160319658 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dd000a400 con 0x7f1de408fef0 2019-12-05T06:53:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.898+0000 7f1dea16b700 1 -- 172.21.15.16:0/2160319658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 msgr2=0x7f1de408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.898+0000 7f1dea16b700 1 --2- 172.21.15.16:0/2160319658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 0x7f1de408d000 secure :-1 s=READY pgs=351 cs=0 l=1 rx=0x7f1dd000b2d0 tx=0x7f1dd000bae0).stop 2019-12-05T06:53:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.899+0000 7f1dea16b700 1 -- 172.21.15.16:0/2160319658 shutdown_connections 2019-12-05T06:53:45.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.899+0000 7f1dea16b700 1 --2- 172.21.15.16:0/2160319658 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de408fef0 0x7f1de408d000 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.899+0000 7f1dea16b700 1 --2- 172.21.15.16:0/2160319658 >> v2:172.21.15.76:3300/0 conn(0x7f1de408c6d0 0x7f1de408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.899+0000 7f1dea16b700 1 -- 172.21.15.16:0/2160319658 shutdown_connections 2019-12-05T06:53:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.899+0000 7f1dea16b700 1 -- 172.21.15.16:0/2160319658 wait complete. 2019-12-05T06:53:45.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.899+0000 7f1dea16b700 1 -- 172.21.15.16:0/2160319658 >> 172.21.15.16:0/2160319658 conn(0x7f1de4061940 msgr2=0x7f1de4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.905+0000 7f1dea16b700 1 Processor -- start 2019-12-05T06:53:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.905+0000 7f1dea16b700 1 -- start start 2019-12-05T06:53:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.907+0000 7f1dea16b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:45.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.907+0000 7f1dea16b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1de4154c60 0x7f1de4157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.907+0000 7f1dea16b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1de4128b20 con 0x7f1de412c740 2019-12-05T06:53:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.907+0000 7f1dea16b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1de4129db0 con 0x7f1de4154c60 2019-12-05T06:53:45.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.907+0000 7f1dea16b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1de412a080 con 0x7f1de41325e0 2019-12-05T06:53:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.907+0000 7f1de8f09700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1de4154c60 0x7f1de4157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.907+0000 7f1de3fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.907+0000 7f1de3fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:39150/0 (socket says 172.21.15.16:39150) 2019-12-05T06:53:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.907+0000 7f1de8f09700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f1de4154c60 0x7f1de4157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35162/0 (socket says 172.21.15.16:35162) 2019-12-05T06:53:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.907+0000 7f1de3fff700 1 -- 172.21.15.16:0/1393458078 learned_addr learned my addr 172.21.15.16:0/1393458078 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:45.917 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de3fff700 1 -- 172.21.15.16:0/1393458078 >> v2:172.21.15.16:3300/0 conn(0x7f1de4154c60 msgr2=0x7f1de4157060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:45.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> v2:172.21.15.16:3300/0 conn(0x7f1de4154c60 0x7f1de4157060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:45.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de3fff700 1 -- 172.21.15.16:0/1393458078 >> v1:172.21.15.16:6789/0 conn(0x7f1de412c740 legacy=0x7f1de412bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:45.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de3fff700 1 -- 172.21.15.16:0/1393458078 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de412ae70 con 0x7f1de41325e0 2019-12-05T06:53:45.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.918 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=READY pgs=352 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de0ff9700 1 -- 172.21.15.16:0/1393458078 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dd4007510 con 0x7f1de41325e0 2019-12-05T06:53:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de0ff9700 1 -- 172.21.15.16:0/1393458078 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1dd400edd0 con 0x7f1de41325e0 2019-12-05T06:53:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1de0ff9700 1 -- 172.21.15.16:0/1393458078 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1dd400f4e0 con 0x7f1de41325e0 2019-12-05T06:53:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.908+0000 7f1dea16b700 1 -- 172.21.15.16:0/1393458078 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de412ae70 con 0x7f1de41325e0 2019-12-05T06:53:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.909+0000 7f1dea16b700 1 -- 172.21.15.16:0/1393458078 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de412ae70 con 0x7f1de41325e0 2019-12-05T06:53:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.909+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.910+0000 7f1de0ff9700 1 -- 172.21.15.16:0/1393458078 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f1dd400e9b0 con 0x7f1de41325e0 2019-12-05T06:53:45.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.910+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.911+0000 7f1de0ff9700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1dc0023630 0x7f1dc0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:45.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.911+0000 7f1de0ff9700 1 -- 172.21.15.16:0/1393458078 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f1dd4045df0 con 0x7f1de41325e0 2019-12-05T06:53:45.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.911+0000 7f1de37fe700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1dc0023630 0x7f1dc0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:45.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.912+0000 7f1de37fe700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1dc0023630 0x7f1dc0025ac0 crc :-1 s=READY pgs=634 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:45.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.925+0000 7f1dea16b700 1 -- 172.21.15.16:0/1393458078 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de412ae70 con 0x7f1de41325e0 2019-12-05T06:53:45.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.934+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:45.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:45.934+0000 7f1de0ff9700 1 -- 172.21.15.16:0/1393458078 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f1dd4007510 con 0x7f1de41325e0 2019-12-05T06:53:46.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.197+0000 7f1dea16b700 1 -- 172.21.15.16:0/1393458078 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1de412ae70 con 0x7f1de41325e0 2019-12-05T06:53:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.198+0000 7f1de3fff700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=THROTTLE_DONE pgs=352 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.198+0000 7f1de0ff9700 1 -- 172.21.15.16:0/1393458078 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 183 v183) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f1dd400e9b0 con 0x7f1de41325e0 2019-12-05T06:53:46.199 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 183 2019-12-05T06:53:46.199 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:46.200 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":183,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":183,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:43.590072+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:replay","state_seq":2,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:46.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 -- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1dc0023630 msgr2=0x7f1dc0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:46.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1dc0023630 0x7f1dc0025ac0 crc :-1 s=READY pgs=634 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:46.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 -- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 msgr2=0x7f1de412b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:46.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 secure :-1 s=READY pgs=352 cs=0 l=1 rx=0x7f1dd4010430 tx=0x7f1dd40102b0).stop 2019-12-05T06:53:46.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 -- 172.21.15.16:0/1393458078 shutdown_connections 2019-12-05T06:53:46.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f1dc0023630 0x7f1dc0025ac0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:46.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 --2- 172.21.15.16:0/1393458078 >> v2:172.21.15.16:3300/0 conn(0x7f1de4154c60 0x7f1de4157060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:46.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 --2- 172.21.15.16:0/1393458078 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f1de41325e0 0x7f1de412b770 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:46.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 -- 172.21.15.16:0/1393458078 shutdown_connections 2019-12-05T06:53:46.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 -- 172.21.15.16:0/1393458078 wait complete. 2019-12-05T06:53:46.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:46.201+0000 7f1dea16b700 1 -- 172.21.15.16:0/1393458078 >> 172.21.15.16:0/1393458078 conn(0x7f1de40a2850 msgr2=0x7f1de406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:46.213 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 183, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 183, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6092": { "addr": "172.21.15.76:6835/2551129089", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 2551129089, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 2551129089, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6092, "incarnation": 183, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 80, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:53:43.590072+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6092 } } } ], "standbys": [] } 2019-12-05T06:53:46.321 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:53:46.326 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:53:48.214 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:48.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.460+0000 7f220f14a700 1 Processor -- start 2019-12-05T06:53:48.470 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.460+0000 7f220f14a700 1 -- start start 2019-12-05T06:53:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.462+0000 7f220f14a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f220808c6d0 0x7f220808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.462+0000 7f220f14a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f220808f720 0x7f220808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.462+0000 7f220f14a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22080958e0 con 0x7f220808fef0 2019-12-05T06:53:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.462+0000 7f220f14a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22080a3f70 con 0x7f220808f720 2019-12-05T06:53:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.462+0000 7f220f14a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220808d830 con 0x7f220808c6d0 2019-12-05T06:53:48.471 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.462+0000 7f220dee8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f220808f720 0x7f220808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.462+0000 7f220dee8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f220808f720 0x7f220808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35174/0 (socket says 172.21.15.16:35174) 2019-12-05T06:53:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.462+0000 7f220dee8700 1 -- 172.21.15.16:0/3997393672 learned_addr learned my addr 172.21.15.16:0/3997393672 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.462+0000 7f220d6e7700 1 --2- 172.21.15.16:0/3997393672 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f220808c6d0 0x7f220808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.463+0000 7f220dee8700 1 -- 172.21.15.16:0/3997393672 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f220808c6d0 msgr2=0x7f220808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.463+0000 7f220dee8700 1 --2- 172.21.15.16:0/3997393672 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f220808c6d0 0x7f220808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.463+0000 7f220dee8700 1 -- 172.21.15.16:0/3997393672 >> v1:172.21.15.76:6789/0 conn(0x7f220808fef0 legacy=0x7f220808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:48.472 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.463+0000 7f220dee8700 1 -- 172.21.15.16:0/3997393672 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220808fbc0 con 0x7f220808f720 2019-12-05T06:53:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.463+0000 7f220dee8700 1 --2- 172.21.15.16:0/3997393672 >> v2:172.21.15.16:3300/0 conn(0x7f220808f720 0x7f220808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f21f800f730 tx=0x7f21f8007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.463+0000 7f220dee8700 1 --2- 172.21.15.16:0/3997393672 >> v2:172.21.15.16:3300/0 conn(0x7f220808f720 0x7f220808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f21f800f730 tx=0x7f21f8007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.463+0000 7f220dee8700 1 --2- 172.21.15.16:0/3997393672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220808f720 0x7f220808d000 secure :-1 s=READY pgs=1518 cs=0 l=1 rx=0x7f21f800f730 tx=0x7f21f8007d50).ready entity=mon.0 client_cookie=74db89f89f11cb71 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.464+0000 7f220dee8700 1 --2- 172.21.15.16:0/3997393672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220808f720 0x7f220808d000 secure :-1 s=THROTTLE_DONE pgs=1518 cs=0 l=1 rx=0x7f21f800f730 tx=0x7f21f8007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.464+0000 7f220dee8700 1 --2- 172.21.15.16:0/3997393672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220808f720 0x7f220808d000 secure :-1 s=THROTTLE_DONE pgs=1518 cs=0 l=1 rx=0x7f21f800f730 tx=0x7f21f8007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.464+0000 7f2207fff700 1 -- 172.21.15.16:0/3997393672 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21f8000a70 con 0x7f220808f720 2019-12-05T06:53:48.473 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.464+0000 7f220dee8700 1 --2- 172.21.15.16:0/3997393672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220808f720 0x7f220808d000 secure :-1 s=THROTTLE_DONE pgs=1518 cs=0 l=1 rx=0x7f21f800f730 tx=0x7f21f8007d50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.464+0000 7f2207fff700 1 -- 172.21.15.16:0/3997393672 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21f80010e0 con 0x7f220808f720 2019-12-05T06:53:48.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.464+0000 7f2207fff700 1 -- 172.21.15.16:0/3997393672 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21f80017b0 con 0x7f220808f720 2019-12-05T06:53:48.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.464+0000 7f220f14a700 1 -- 172.21.15.16:0/3997393672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220808f720 msgr2=0x7f220808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:48.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.464+0000 7f220f14a700 1 --2- 172.21.15.16:0/3997393672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220808f720 0x7f220808d000 secure :-1 s=READY pgs=1518 cs=0 l=1 rx=0x7f21f800f730 tx=0x7f21f8007d50).stop 2019-12-05T06:53:48.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.465+0000 7f220f14a700 1 -- 172.21.15.16:0/3997393672 shutdown_connections 2019-12-05T06:53:48.474 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.465+0000 7f220f14a700 1 --2- 172.21.15.16:0/3997393672 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220808f720 0x7f220808d000 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.465+0000 7f220f14a700 1 --2- 172.21.15.16:0/3997393672 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f220808c6d0 0x7f220808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.466+0000 7f220f14a700 1 -- 172.21.15.16:0/3997393672 shutdown_connections 2019-12-05T06:53:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.466+0000 7f220f14a700 1 -- 172.21.15.16:0/3997393672 wait complete. 2019-12-05T06:53:48.475 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.466+0000 7f220f14a700 1 -- 172.21.15.16:0/3997393672 >> 172.21.15.16:0/3997393672 conn(0x7f2208061940 msgr2=0x7f2208096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:48.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.472+0000 7f220f14a700 1 Processor -- start 2019-12-05T06:53:48.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.472+0000 7f220f14a700 1 -- start start 2019-12-05T06:53:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.474+0000 7f220f14a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f220806e800 0x7f220812fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.474+0000 7f220f14a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2208130be0 0x7f2208130150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.474+0000 7f220f14a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2208128690 con 0x7f220806dc10 2019-12-05T06:53:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.474+0000 7f220f14a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f220809e690 con 0x7f220806e800 2019-12-05T06:53:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.474+0000 7f220f14a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f220812ec80 con 0x7f2208130be0 2019-12-05T06:53:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.474+0000 7f220cee6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f220806e800 0x7f220812fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:48.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.474+0000 7f220cee6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f220806e800 0x7f220812fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35178/0 (socket says 172.21.15.16:35178) 2019-12-05T06:53:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.474+0000 7f220cee6700 1 -- 172.21.15.16:0/440515485 learned_addr learned my addr 172.21.15.16:0/440515485 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.474+0000 7f220d6e7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f220806dc10 0x7f220806e000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:41646/0 (socket says 172.21.15.16:41646) 2019-12-05T06:53:48.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.475+0000 7f220dee8700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2208130be0 0x7f2208130150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.475+0000 7f2205ffb700 1 -- 172.21.15.16:0/440515485 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269489073 0 0) 0x7f22000012b0 con 0x7f220806dc10 2019-12-05T06:53:48.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.475+0000 7f220cee6700 1 -- 172.21.15.16:0/440515485 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2208130be0 msgr2=0x7f2208130150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.475+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2208130be0 0x7f2208130150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.475+0000 7f220cee6700 1 -- 172.21.15.16:0/440515485 >> v1:172.21.15.76:6789/0 conn(0x7f220806dc10 legacy=0x7f220806e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.475+0000 7f220cee6700 1 -- 172.21.15.16:0/440515485 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f220812f310 con 0x7f220806e800 2019-12-05T06:53:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.475+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> v2:172.21.15.16:3300/0 conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.475+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> v2:172.21.15.16:3300/0 conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.475+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=READY pgs=1519 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:48.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.476+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=THROTTLE_DONE pgs=1519 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.476+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=THROTTLE_DONE pgs=1519 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.476+0000 7f2205ffb700 1 -- 172.21.15.16:0/440515485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21fc000960 con 0x7f220806e800 2019-12-05T06:53:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.476+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=THROTTLE_DONE pgs=1519 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.476+0000 7f2205ffb700 1 -- 172.21.15.16:0/440515485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21fc0010a0 con 0x7f220806e800 2019-12-05T06:53:48.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.476+0000 7f2205ffb700 1 -- 172.21.15.16:0/440515485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21fc0017f0 con 0x7f220806e800 2019-12-05T06:53:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.476+0000 7f220f14a700 1 -- 172.21.15.16:0/440515485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f220812f310 con 0x7f220806e800 2019-12-05T06:53:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.476+0000 7f220f14a700 1 -- 172.21.15.16:0/440515485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f220812f310 con 0x7f220806e800 2019-12-05T06:53:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.477+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=THROTTLE_DONE pgs=1519 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.478+0000 7f2205ffb700 1 -- 172.21.15.16:0/440515485 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f21fc001290 con 0x7f220806e800 2019-12-05T06:53:48.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.478+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=THROTTLE_DONE pgs=1519 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.497 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.478+0000 7f2205ffb700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f21e8023630 0x7f21e8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:48.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.479+0000 7f2205ffb700 1 -- 172.21.15.16:0/440515485 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f21fc045e00 con 0x7f220806e800 2019-12-05T06:53:48.498 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.479+0000 7f220d6e7700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f21e8023630 0x7f21e8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:48.499 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.479+0000 7f220d6e7700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f21e8023630 0x7f21e8025ac0 crc :-1 s=READY pgs=635 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:48.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.491+0000 7f220f14a700 1 -- 172.21.15.16:0/440515485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f220812f310 con 0x7f220806e800 2019-12-05T06:53:48.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.500+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=THROTTLE_DONE pgs=1519 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.500+0000 7f2205ffb700 1 -- 172.21.15.16:0/440515485 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f21fc0075b0 con 0x7f220806e800 2019-12-05T06:53:48.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.730+0000 7f220f14a700 1 -- 172.21.15.16:0/440515485 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f220812f310 con 0x7f220806e800 2019-12-05T06:53:48.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.731+0000 7f220cee6700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=THROTTLE_DONE pgs=1519 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.731+0000 7f2205ffb700 1 -- 172.21.15.16:0/440515485 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f21fc000c10 con 0x7f220806e800 2019-12-05T06:53:48.735 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 186 2019-12-05T06:53:48.735 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:48.735 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:48.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 -- 172.21.15.16:0/440515485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f21e8023630 msgr2=0x7f21e8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:48.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f21e8023630 0x7f21e8025ac0 crc :-1 s=READY pgs=635 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:48.741 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 -- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 msgr2=0x7f220812fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 0x7f220812fc10 secure :-1 s=READY pgs=1519 cs=0 l=1 rx=0x7f21fc010410 tx=0x7f21fc010270).stop 2019-12-05T06:53:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 -- 172.21.15.16:0/440515485 shutdown_connections 2019-12-05T06:53:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f2208130be0 0x7f2208130150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f21e8023630 0x7f21e8025ac0 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 --2- 172.21.15.16:0/440515485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f220806e800 0x7f220812fc10 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 -- 172.21.15.16:0/440515485 shutdown_connections 2019-12-05T06:53:48.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 -- 172.21.15.16:0/440515485 wait complete. 2019-12-05T06:53:48.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:48.734+0000 7f220f14a700 1 -- 172.21.15.16:0/440515485 >> 172.21.15.16:0/440515485 conn(0x7f2208098480 msgr2=0x7f2208071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:48.746 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=7857 2019-12-05T06:53:48.746 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 6 secs before reviving mds.a 2019-12-05T06:53:54.747 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:53:54.748 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:53:54.748 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:53:54.752 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:53:54.753 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:54.851 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:53:54.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.886+0000 7fc8a5421700 1 Processor -- start 2019-12-05T06:53:54.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.886+0000 7fc8a5421700 1 -- start start 2019-12-05T06:53:54.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.887+0000 7fc8a5421700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc8a008c6d0 0x7fc8a008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:54.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.887+0000 7fc8a5421700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a008fb80 con 0x7fc8a008f720 2019-12-05T06:53:54.892 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.887+0000 7fc8a5421700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a00958e0 con 0x7fc8a008fef0 2019-12-05T06:53:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.887+0000 7fc8a5421700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a008df90 con 0x7fc8a008c6d0 2019-12-05T06:53:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.887+0000 7fc89ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc8a008f720 0x7fc8a01533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48636/0 (socket says 172.21.15.16:48636) 2019-12-05T06:53:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.887+0000 7fc89ffff700 1 -- 172.21.15.16:0/3943127245 learned_addr learned my addr 172.21.15.16:0/3943127245 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.887+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/3943127245 >> v2:172.21.15.76:3300/0 conn(0x7fc8a008c6d0 0x7fc8a008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.887+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390606882 0 0) 0x7fc8980014d0 con 0x7fc8a008f720 2019-12-05T06:53:54.893 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.887+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc888002e80 con 0x7fc8a008f720 2019-12-05T06:53:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 3336026129 0 0) 0x7fc8980014b0 con 0x7fc8a008f720 2019-12-05T06:53:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 >> v2:172.21.15.76:3300/0 conn(0x7fc8a008c6d0 msgr2=0x7fc8a008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 --2- 172.21.15.16:0/3943127245 >> v2:172.21.15.76:3300/0 conn(0x7fc8a008c6d0 0x7fc8a008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 >> v1:172.21.15.76:6789/0 conn(0x7fc8a008fef0 legacy=0x7fc8a008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a008e620 con 0x7fc8a008f720 2019-12-05T06:53:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fc8980019e0 con 0x7fc8a008f720 2019-12-05T06:53:54.894 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 >> v1:172.21.15.16:6789/0 conn(0x7fc8a008f720 legacy=0x7fc8a01533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 0x7fc888005ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc888006690 0x7fc888008b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc8880093a0 0x7fc88800b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88800c020 con 0x7fc888003a80 2019-12-05T06:53:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc888050220 con 0x7fc888006690 2019-12-05T06:53:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc888050450 con 0x7fc8880093a0 2019-12-05T06:53:54.895 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 0x7fc888005ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89ffff700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc8880093a0 0x7fc88800b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:54.896 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89f7fe700 1 -- 172.21.15.16:0/3943127245 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc8880093a0 msgr2=0x7fc88800b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:54.897 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc8880093a0 0x7fc88800b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:54.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89f7fe700 1 -- 172.21.15.16:0/3943127245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc888006690 msgr2=0x7fc888008b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:54.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc888006690 0x7fc888008b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:54.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89f7fe700 1 -- 172.21.15.16:0/3943127245 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc888050700 con 0x7fc888003a80 2019-12-05T06:53:54.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89ffff700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc8880093a0 0x7fc88800b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:53:54.898 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89ffff700 1 -- 172.21.15.16:0/3943127245 reap_dead start 2019-12-05T06:53:54.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 0x7fc888005ee0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc89000b6e0 tx=0x7fc89000b5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 0x7fc888005ee0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc89000b6e0 tx=0x7fc89000b5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.888+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 0x7fc888005ee0 secure :-1 s=READY pgs=1525 cs=0 l=1 rx=0x7fc89000b6e0 tx=0x7fc89000b5b0).ready entity=mon.0 client_cookie=1764b8664d7a1ac7 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:54.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 0x7fc888005ee0 secure :-1 s=THROTTLE_DONE pgs=1525 cs=0 l=1 rx=0x7fc89000b6e0 tx=0x7fc89000b5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc890000960 con 0x7fc888003a80 2019-12-05T06:53:54.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 0x7fc888005ee0 secure :-1 s=THROTTLE_DONE pgs=1525 cs=0 l=1 rx=0x7fc89000b6e0 tx=0x7fc89000b5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.899 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 0x7fc888005ee0 secure :-1 s=THROTTLE_DONE pgs=1525 cs=0 l=1 rx=0x7fc89000b6e0 tx=0x7fc89000b5b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc890009770 con 0x7fc888003a80 2019-12-05T06:53:54.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc89e7fc700 1 -- 172.21.15.16:0/3943127245 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc890000960 con 0x7fc888003a80 2019-12-05T06:53:54.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc8a5421700 1 -- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 msgr2=0x7fc888005ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:54.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc8a5421700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 0x7fc888005ee0 secure :-1 s=READY pgs=1525 cs=0 l=1 rx=0x7fc89000b6e0 tx=0x7fc89000b5b0).stop 2019-12-05T06:53:54.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc8a5421700 1 -- 172.21.15.16:0/3943127245 shutdown_connections 2019-12-05T06:53:54.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc8a5421700 1 --2- 172.21.15.16:0/3943127245 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc888003a80 0x7fc888005ee0 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:54.900 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc8a5421700 1 -- 172.21.15.16:0/3943127245 shutdown_connections 2019-12-05T06:53:54.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc8a5421700 1 -- 172.21.15.16:0/3943127245 wait complete. 2019-12-05T06:53:54.901 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.889+0000 7fc8a5421700 1 -- 172.21.15.16:0/3943127245 >> 172.21.15.16:0/3943127245 conn(0x7fc8a0061940 msgr2=0x7fc8a0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:54.906 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.892+0000 7fc8a5421700 1 Processor -- start 2019-12-05T06:53:54.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.892+0000 7fc8a5421700 1 -- start start 2019-12-05T06:53:54.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc8a5421700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc8a01302f0 0x7fc8a012a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:54.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc8a5421700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc8a0153ba0 0x7fc8a0155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:54.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc8a5421700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8a0131bf0 con 0x7fc8a012b640 2019-12-05T06:53:54.908 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc8a5421700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8a01296e0 con 0x7fc8a01302f0 2019-12-05T06:53:54.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc8a5421700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8a0128cb0 con 0x7fc8a0153ba0 2019-12-05T06:53:54.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc89effd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc8a012b640 0x7fc8a012abb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48646/0 (socket says 172.21.15.16:48646) 2019-12-05T06:53:54.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc89effd700 1 -- 172.21.15.16:0/268438606 learned_addr learned my addr 172.21.15.16:0/268438606 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:53:54.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc8a0153ba0 0x7fc8a0155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:54.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/268438606 >> v2:172.21.15.76:3300/0 conn(0x7fc8a01302f0 0x7fc8a012a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:54.909 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635462613 0 0) 0x7fc894000ca0 con 0x7fc8a012b640 2019-12-05T06:53:54.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc880002e80 con 0x7fc8a012b640 2019-12-05T06:53:54.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 20102626 0 0) 0x7fc894001480 con 0x7fc8a012b640 2019-12-05T06:53:54.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc8800032e0 con 0x7fc8a012b640 2019-12-05T06:53:54.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fc8940019f0 con 0x7fc8a012b640 2019-12-05T06:53:54.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc8a0153ba0 msgr2=0x7fc8a0155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:54.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc887fff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc8a0153ba0 0x7fc8a0155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:54.910 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 >> v2:172.21.15.76:3300/0 conn(0x7fc8a01302f0 msgr2=0x7fc8a012a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:54.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc887fff700 1 --2- 172.21.15.16:0/268438606 >> v2:172.21.15.76:3300/0 conn(0x7fc8a01302f0 0x7fc8a012a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:54.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.893+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 >> v1:172.21.15.16:6789/0 conn(0x7fc8a012b640 legacy=0x7fc8a012abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:54.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc887fff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880003330 0x7fc880003760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:54.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc887fff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc880003ca0 0x7fc8800040d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:54.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc887fff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:54.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88000bb60 con 0x7fc880004800 2019-12-05T06:53:54.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc88000bd50 con 0x7fc880003330 2019-12-05T06:53:54.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc88000bf80 con 0x7fc880003ca0 2019-12-05T06:53:54.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:54.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 -- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc880003ca0 msgr2=0x7fc8800040d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:54.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc880003ca0 0x7fc8800040d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:54.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 -- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880003330 msgr2=0x7fc880003760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:54.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc880003330 0x7fc880003760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:54.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 -- 172.21.15.16:0/268438606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc88000c230 con 0x7fc880004800 2019-12-05T06:53:54.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc8a5421700 1 -- 172.21.15.16:0/268438606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8a0129d70 con 0x7fc880004800 2019-12-05T06:53:54.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc8a5421700 1 -- 172.21.15.16:0/268438606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8a01496f0 con 0x7fc880004800 2019-12-05T06:53:54.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=READY pgs=1527 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:54.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=THROTTLE_DONE pgs=1527 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc898001970 con 0x7fc880004800 2019-12-05T06:53:54.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=THROTTLE_DONE pgs=1527 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc898001970 con 0x7fc880004800 2019-12-05T06:53:54.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=THROTTLE_DONE pgs=1527 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.894+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc898004bb0 con 0x7fc880004800 2019-12-05T06:53:54.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.895+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=THROTTLE_DONE pgs=1527 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.895+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc8980047d0 con 0x7fc880004800 2019-12-05T06:53:54.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.895+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=THROTTLE_DONE pgs=1527 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.896+0000 7fc887fff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc88002b970 0x7fc88002dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:54.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.896+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fc898046080 con 0x7fc880004800 2019-12-05T06:53:54.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.896+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc88002b970 0x7fc88002dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:54.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.896+0000 7fc89f7fe700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc88002b970 0x7fc88002dde0 crc :-1 s=READY pgs=636 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:54.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.902+0000 7fc8a5421700 1 -- 172.21.15.16:0/268438606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc878002e60 con 0x7fc880004800 2019-12-05T06:53:54.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.907+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=THROTTLE_DONE pgs=1527 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:54.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:54.907+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc8980045c0 con 0x7fc880004800 2019-12-05T06:53:55.100 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.100+0000 7fc8a5421700 1 -- 172.21.15.16:0/268438606 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc878001d10 con 0x7fc880004800 2019-12-05T06:53:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.100+0000 7fc89ffff700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=THROTTLE_DONE pgs=1527 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:55.101 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.100+0000 7fc887fff700 1 -- 172.21.15.16:0/268438606 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fc8980047d0 con 0x7fc880004800 2019-12-05T06:53:55.102 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 186 2019-12-05T06:53:55.102 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:55.102 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:55.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.103+0000 7fc8a5421700 1 -- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc88002b970 msgr2=0x7fc88002dde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:55.104 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.103+0000 7fc8a5421700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc88002b970 0x7fc88002dde0 crc :-1 s=READY pgs=636 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.103+0000 7fc8a5421700 1 -- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 msgr2=0x7fc880004cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.103+0000 7fc89ffff700 1 -- 172.21.15.16:0/268438606 reap_dead start 2019-12-05T06:53:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.103+0000 7fc8a5421700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 secure :-1 s=READY pgs=1527 cs=0 l=1 rx=0x7fc8980106f0 tx=0x7fc8980105a0).stop 2019-12-05T06:53:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.103+0000 7fc8a5421700 1 -- 172.21.15.16:0/268438606 shutdown_connections 2019-12-05T06:53:55.105 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.103+0000 7fc8a5421700 1 --2- 172.21.15.16:0/268438606 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc880004800 0x7fc880004cc0 unknown :-1 s=CLOSED pgs=1527 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:55.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.103+0000 7fc8a5421700 1 -- 172.21.15.16:0/268438606 shutdown_connections 2019-12-05T06:53:55.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.103+0000 7fc8a5421700 1 -- 172.21.15.16:0/268438606 wait complete. 2019-12-05T06:53:55.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:55.103+0000 7fc8a5421700 1 -- 172.21.15.16:0/268438606 >> 172.21.15.16:0/268438606 conn(0x7fc8a00a2850 msgr2=0x7fc8a006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:55.115 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:53:57.116 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.270+0000 7f7c2c120700 1 Processor -- start 2019-12-05T06:53:57.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.270+0000 7f7c2c120700 1 -- start start 2019-12-05T06:53:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c2c120700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c2408d2c0 0x7f7c2408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c2c120700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7c2408fef0 0x7f7c2408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c2c120700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c240958e0 con 0x7f7c2408c6d0 2019-12-05T06:53:57.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c2c120700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2408df90 con 0x7f7c2408d2c0 2019-12-05T06:53:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c2c120700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c240a3f70 con 0x7f7c2408fef0 2019-12-05T06:53:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c29ebc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c2408d2c0 0x7f7c2408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c29ebc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7c2408d2c0 0x7f7c2408ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35272/0 (socket says 172.21.15.16:35272) 2019-12-05T06:53:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c29ebc700 1 -- 172.21.15.16:0/998314650 learned_addr learned my addr 172.21.15.16:0/998314650 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c2aebe700 1 --2- 172.21.15.16:0/998314650 >> v2:172.21.15.76:3300/0 conn(0x7f7c2408fef0 0x7f7c2408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c29ebc700 1 -- 172.21.15.16:0/998314650 >> v2:172.21.15.76:3300/0 conn(0x7f7c2408fef0 msgr2=0x7f7c2408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:57.279 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c29ebc700 1 --2- 172.21.15.16:0/998314650 >> v2:172.21.15.76:3300/0 conn(0x7f7c2408fef0 0x7f7c2408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c29ebc700 1 -- 172.21.15.16:0/998314650 >> v1:172.21.15.76:6789/0 conn(0x7f7c2408c6d0 legacy=0x7f7c2408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c29ebc700 1 -- 172.21.15.16:0/998314650 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2408e620 con 0x7f7c2408d2c0 2019-12-05T06:53:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c29ebc700 1 --2- 172.21.15.16:0/998314650 >> v2:172.21.15.16:3300/0 conn(0x7f7c2408d2c0 0x7f7c2408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7c1400f740 tx=0x7f7c1400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c29ebc700 1 --2- 172.21.15.16:0/998314650 >> v2:172.21.15.16:3300/0 conn(0x7f7c2408d2c0 0x7f7c2408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7c1400f740 tx=0x7f7c1400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.271+0000 7f7c29ebc700 1 --2- 172.21.15.16:0/998314650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c2408d2c0 0x7f7c2408ef20 secure :-1 s=READY pgs=1528 cs=0 l=1 rx=0x7f7c1400f740 tx=0x7f7c1400f570).ready entity=mon.0 client_cookie=9971a4f20243ece4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.272+0000 7f7c29ebc700 1 --2- 172.21.15.16:0/998314650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c2408d2c0 0x7f7c2408ef20 secure :-1 s=THROTTLE_DONE pgs=1528 cs=0 l=1 rx=0x7f7c1400f740 tx=0x7f7c1400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.272+0000 7f7c29ebc700 1 --2- 172.21.15.16:0/998314650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c2408d2c0 0x7f7c2408ef20 secure :-1 s=THROTTLE_DONE pgs=1528 cs=0 l=1 rx=0x7f7c1400f740 tx=0x7f7c1400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.272+0000 7f7c296bb700 1 -- 172.21.15.16:0/998314650 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c14000960 con 0x7f7c2408d2c0 2019-12-05T06:53:57.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.272+0000 7f7c29ebc700 1 --2- 172.21.15.16:0/998314650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c2408d2c0 0x7f7c2408ef20 secure :-1 s=THROTTLE_DONE pgs=1528 cs=0 l=1 rx=0x7f7c1400f740 tx=0x7f7c1400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.272+0000 7f7c296bb700 1 -- 172.21.15.16:0/998314650 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c14000c90 con 0x7f7c2408d2c0 2019-12-05T06:53:57.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.272+0000 7f7c296bb700 1 -- 172.21.15.16:0/998314650 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c140013a0 con 0x7f7c2408d2c0 2019-12-05T06:53:57.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.272+0000 7f7c2c120700 1 -- 172.21.15.16:0/998314650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c2408d2c0 msgr2=0x7f7c2408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:57.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.272+0000 7f7c2c120700 1 --2- 172.21.15.16:0/998314650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c2408d2c0 0x7f7c2408ef20 secure :-1 s=READY pgs=1528 cs=0 l=1 rx=0x7f7c1400f740 tx=0x7f7c1400f570).stop 2019-12-05T06:53:57.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.273+0000 7f7c2c120700 1 -- 172.21.15.16:0/998314650 shutdown_connections 2019-12-05T06:53:57.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.273+0000 7f7c2c120700 1 --2- 172.21.15.16:0/998314650 >> v2:172.21.15.76:3300/0 conn(0x7f7c2408fef0 0x7f7c2408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:57.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.273+0000 7f7c2c120700 1 --2- 172.21.15.16:0/998314650 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7c2408d2c0 0x7f7c2408ef20 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:57.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.273+0000 7f7c2c120700 1 -- 172.21.15.16:0/998314650 shutdown_connections 2019-12-05T06:53:57.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.273+0000 7f7c2c120700 1 -- 172.21.15.16:0/998314650 wait complete. 2019-12-05T06:53:57.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.273+0000 7f7c2c120700 1 -- 172.21.15.16:0/998314650 >> 172.21.15.16:0/998314650 conn(0x7f7c24061940 msgr2=0x7f7c24096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:57.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.275+0000 7f7c2c120700 1 Processor -- start 2019-12-05T06:53:57.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.275+0000 7f7c2c120700 1 -- start start 2019-12-05T06:53:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.276+0000 7f7c2c120700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7c2414c3a0 0x7f7c2409d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.276+0000 7f7c2c120700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c2406e210 0x7f7c2409daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.276+0000 7f7c2c120700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c24153160 con 0x7f7c2409e030 2019-12-05T06:53:57.283 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.276+0000 7f7c2c120700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c2409c620 con 0x7f7c2414c3a0 2019-12-05T06:53:57.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.276+0000 7f7c2c120700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c2408ada0 con 0x7f7c2406e210 2019-12-05T06:53:57.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c2a6bd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7c2414c3a0 0x7f7c2409d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:57.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c2a6bd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7c2414c3a0 0x7f7c2409d5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:49598/0 (socket says 172.21.15.16:49598) 2019-12-05T06:53:57.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c2a6bd700 1 -- 172.21.15.16:0/2071943619 learned_addr learned my addr 172.21.15.16:0/2071943619 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:57.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c29ebc700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c2406e210 0x7f7c2409daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:57.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c2a6bd700 1 -- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c2406e210 msgr2=0x7f7c2409daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:57.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c2406e210 0x7f7c2409daf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:57.299 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c2a6bd700 1 -- 172.21.15.16:0/2071943619 >> v1:172.21.15.76:6789/0 conn(0x7f7c2409e030 legacy=0x7f7c24088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c2a6bd700 1 -- 172.21.15.16:0/2071943619 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2409ccb0 con 0x7f7c2414c3a0 2019-12-05T06:53:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c1affd700 1 -- 172.21.15.16:0/2071943619 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2816899998 0 0) 0x7f7c1c0011f0 con 0x7f7c2409e030 2019-12-05T06:53:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> v2:172.21.15.76:3300/0 conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> v2:172.21.15.76:3300/0 conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.277+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=READY pgs=1028 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:57.300 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.278+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=THROTTLE_DONE pgs=1028 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.278+0000 7f7c1affd700 1 -- 172.21.15.16:0/2071943619 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c20007210 con 0x7f7c2414c3a0 2019-12-05T06:53:57.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.278+0000 7f7c2c120700 1 -- 172.21.15.16:0/2071943619 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2409ccb0 con 0x7f7c2414c3a0 2019-12-05T06:53:57.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.278+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=THROTTLE_DONE pgs=1028 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.278+0000 7f7c1affd700 1 -- 172.21.15.16:0/2071943619 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c20007210 con 0x7f7c2414c3a0 2019-12-05T06:53:57.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.278+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=THROTTLE_DONE pgs=1028 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.301 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.278+0000 7f7c2c120700 1 -- 172.21.15.16:0/2071943619 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c24153160 con 0x7f7c2414c3a0 2019-12-05T06:53:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.279+0000 7f7c1affd700 1 -- 172.21.15.16:0/2071943619 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c200055b0 con 0x7f7c2414c3a0 2019-12-05T06:53:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.279+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=THROTTLE_DONE pgs=1028 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.279+0000 7f7c1affd700 1 -- 172.21.15.16:0/2071943619 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7c20005270 con 0x7f7c2414c3a0 2019-12-05T06:53:57.314 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.279+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=THROTTLE_DONE pgs=1028 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.279+0000 7f7c1affd700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c0c023630 0x7f7c0c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.279+0000 7f7c1affd700 1 -- 172.21.15.16:0/2071943619 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f7c20045de0 con 0x7f7c2414c3a0 2019-12-05T06:53:57.315 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.279+0000 7f7c29ebc700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c0c023630 0x7f7c0c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.280+0000 7f7c29ebc700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c0c023630 0x7f7c0c025ac0 crc :-1 s=READY pgs=637 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.285+0000 7f7c2c120700 1 -- 172.21.15.16:0/2071943619 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c2409ccb0 con 0x7f7c2414c3a0 2019-12-05T06:53:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.291+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=THROTTLE_DONE pgs=1028 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.316 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.291+0000 7f7c1affd700 1 -- 172.21.15.16:0/2071943619 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7c2002ae00 con 0x7f7c2414c3a0 2019-12-05T06:53:57.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.483+0000 7f7c2c120700 1 -- 172.21.15.16:0/2071943619 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7c2409ccb0 con 0x7f7c2414c3a0 2019-12-05T06:53:57.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.483+0000 7f7c2a6bd700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=THROTTLE_DONE pgs=1028 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.483+0000 7f7c1affd700 1 -- 172.21.15.16:0/2071943619 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 186 v186) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f7c2002ae00 con 0x7f7c2414c3a0 2019-12-05T06:53:57.485 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 186 2019-12-05T06:53:57.485 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:57.485 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":186,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 -- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c0c023630 msgr2=0x7f7c0c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c0c023630 0x7f7c0c025ac0 crc :-1 s=READY pgs=637 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 -- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 msgr2=0x7f7c2409d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 0x7f7c2409d5b0 secure :-1 s=READY pgs=1028 cs=0 l=1 rx=0x7f7c20010370 tx=0x7f7c20010710).stop 2019-12-05T06:53:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 -- 172.21.15.16:0/2071943619 shutdown_connections 2019-12-05T06:53:57.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7c0c023630 0x7f7c0c025ac0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7c2406e210 0x7f7c2409daf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 --2- 172.21.15.16:0/2071943619 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7c2414c3a0 0x7f7c2409d5b0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 -- 172.21.15.16:0/2071943619 shutdown_connections 2019-12-05T06:53:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 -- 172.21.15.16:0/2071943619 wait complete. 2019-12-05T06:53:57.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:57.486+0000 7f7c2c120700 1 -- 172.21.15.16:0/2071943619 >> 172.21.15.16:0/2071943619 conn(0x7f7c24098480 msgr2=0x7f7c24071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:57.498 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:53:59.499 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:53:59.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.675+0000 7f94bee20700 1 Processor -- start 2019-12-05T06:53:59.683 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.676+0000 7f94bee20700 1 -- start start 2019-12-05T06:53:59.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bee20700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f94b808c6d0 0x7f94b808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:59.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bee20700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94b808fef0 0x7f94b808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:59.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bee20700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94b808ef20 0x7f94b808f370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:53:59.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bee20700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b808df90 con 0x7f94b808fef0 2019-12-05T06:53:59.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bee20700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b80a3f70 con 0x7f94b808c6d0 2019-12-05T06:53:59.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bee20700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b808d830 con 0x7f94b808ef20 2019-12-05T06:53:59.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bcbbc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94b808fef0 0x7f94b808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bcbbc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94b808fef0 0x7f94b808d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35292/0 (socket says 172.21.15.16:35292) 2019-12-05T06:53:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bcbbc700 1 -- 172.21.15.16:0/462753964 learned_addr learned my addr 172.21.15.16:0/462753964 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bd3bd700 1 --2- 172.21.15.16:0/462753964 >> v2:172.21.15.76:3300/0 conn(0x7f94b808c6d0 0x7f94b808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.677+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/462753964 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94b808ef20 0x7f94b808f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:59.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 -- 172.21.15.16:0/462753964 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94b808ef20 msgr2=0x7f94b808f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 --2- 172.21.15.16:0/462753964 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94b808ef20 0x7f94b808f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:59.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 -- 172.21.15.16:0/462753964 >> v2:172.21.15.76:3300/0 conn(0x7f94b808c6d0 msgr2=0x7f94b808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:53:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 --2- 172.21.15.16:0/462753964 >> v2:172.21.15.76:3300/0 conn(0x7f94b808c6d0 0x7f94b808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 -- 172.21.15.16:0/462753964 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b808d540 con 0x7f94b808fef0 2019-12-05T06:53:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bd3bd700 1 --2- 172.21.15.16:0/462753964 >> v2:172.21.15.76:3300/0 conn(0x7f94b808c6d0 0x7f94b808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:53:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 --2- 172.21.15.16:0/462753964 >> v2:172.21.15.16:3300/0 conn(0x7f94b808fef0 0x7f94b808d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f94ac00f740 tx=0x7f94ac00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 --2- 172.21.15.16:0/462753964 >> v2:172.21.15.16:3300/0 conn(0x7f94b808fef0 0x7f94b808d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f94ac00f740 tx=0x7f94ac00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 --2- 172.21.15.16:0/462753964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b808fef0 0x7f94b808d000 secure :-1 s=READY pgs=1529 cs=0 l=1 rx=0x7f94ac00f740 tx=0x7f94ac00f570).ready entity=mon.0 client_cookie=7a97a25707e93824 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 --2- 172.21.15.16:0/462753964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b808fef0 0x7f94b808d000 secure :-1 s=THROTTLE_DONE pgs=1529 cs=0 l=1 rx=0x7f94ac00f740 tx=0x7f94ac00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94b7fff700 1 -- 172.21.15.16:0/462753964 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94ac000960 con 0x7f94b808fef0 2019-12-05T06:53:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 --2- 172.21.15.16:0/462753964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b808fef0 0x7f94b808d000 secure :-1 s=THROTTLE_DONE pgs=1529 cs=0 l=1 rx=0x7f94ac00f740 tx=0x7f94ac00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bcbbc700 1 --2- 172.21.15.16:0/462753964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b808fef0 0x7f94b808d000 secure :-1 s=THROTTLE_DONE pgs=1529 cs=0 l=1 rx=0x7f94ac00f740 tx=0x7f94ac00f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94b7fff700 1 -- 172.21.15.16:0/462753964 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94ac000c90 con 0x7f94b808fef0 2019-12-05T06:53:59.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94b7fff700 1 -- 172.21.15.16:0/462753964 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94ac0013a0 con 0x7f94b808fef0 2019-12-05T06:53:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bee20700 1 -- 172.21.15.16:0/462753964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b808fef0 msgr2=0x7f94b808d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.678+0000 7f94bee20700 1 --2- 172.21.15.16:0/462753964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b808fef0 0x7f94b808d000 secure :-1 s=READY pgs=1529 cs=0 l=1 rx=0x7f94ac00f740 tx=0x7f94ac00f570).stop 2019-12-05T06:53:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.680+0000 7f94bee20700 1 -- 172.21.15.16:0/462753964 shutdown_connections 2019-12-05T06:53:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.680+0000 7f94bee20700 1 --2- 172.21.15.16:0/462753964 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94b808ef20 0x7f94b808f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.680+0000 7f94bee20700 1 --2- 172.21.15.16:0/462753964 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b808fef0 0x7f94b808d000 unknown :-1 s=CLOSED pgs=1529 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.680+0000 7f94bee20700 1 --2- 172.21.15.16:0/462753964 >> v2:172.21.15.76:3300/0 conn(0x7f94b808c6d0 0x7f94b808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:53:59.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.680+0000 7f94bee20700 1 -- 172.21.15.16:0/462753964 shutdown_connections 2019-12-05T06:53:59.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.680+0000 7f94bee20700 1 -- 172.21.15.16:0/462753964 wait complete. 2019-12-05T06:53:59.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.680+0000 7f94bee20700 1 -- 172.21.15.16:0/462753964 >> 172.21.15.16:0/462753964 conn(0x7f94b8061940 msgr2=0x7f94b8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:59.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.683+0000 7f94bee20700 1 Processor -- start 2019-12-05T06:53:59.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.683+0000 7f94bee20700 1 -- start start 2019-12-05T06:53:59.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.684+0000 7f94bee20700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94b8096e90 0x7f94b809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:59.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.684+0000 7f94bee20700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f94b8084f30 0x7f94b809b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:59.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.684+0000 7f94bee20700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94b8097280 0x7f94b8097700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:59.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.684+0000 7f94bee20700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b8099a20 con 0x7f94b8097280 2019-12-05T06:53:59.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bee20700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94b8099cf0 con 0x7f94b8084f30 2019-12-05T06:53:59.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bee20700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94b809aae0 con 0x7f94b8096e90 2019-12-05T06:53:59.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bdbbe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94b8097280 0x7f94b8097700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:59.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bdbbe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f94b8097280 0x7f94b8097700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35302/0 (socket says 172.21.15.16:35302) 2019-12-05T06:53:59.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bdbbe700 1 -- 172.21.15.16:0/1141638545 learned_addr learned my addr 172.21.15.16:0/1141638545 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:53:59.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bd3bd700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94b8096e90 0x7f94b809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:59.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bcbbc700 1 --2- 172.21.15.16:0/1141638545 >> v2:172.21.15.76:3300/0 conn(0x7f94b8084f30 0x7f94b809b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:59.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bdbbe700 1 -- 172.21.15.16:0/1141638545 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94b8096e90 msgr2=0x7f94b809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:59.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94b8096e90 0x7f94b809b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:59.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bdbbe700 1 -- 172.21.15.16:0/1141638545 >> v2:172.21.15.76:3300/0 conn(0x7f94b8084f30 msgr2=0x7f94b809b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:59.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> v2:172.21.15.76:3300/0 conn(0x7f94b8084f30 0x7f94b809b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:59.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bdbbe700 1 -- 172.21.15.16:0/1141638545 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b8084b90 con 0x7f94b8097280 2019-12-05T06:53:59.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.685+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> v2:172.21.15.16:3300/0 conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.686+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> v2:172.21.15.16:3300/0 conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.686+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=READY pgs=1530 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:59.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.686+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=THROTTLE_DONE pgs=1530 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.686+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=THROTTLE_DONE pgs=1530 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.686+0000 7f94b5ffb700 1 -- 172.21.15.16:0/1141638545 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94a800a6b0 con 0x7f94b8097280 2019-12-05T06:53:59.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.686+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=THROTTLE_DONE pgs=1530 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.686+0000 7f94b5ffb700 1 -- 172.21.15.16:0/1141638545 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94a8001060 con 0x7f94b8097280 2019-12-05T06:53:59.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.686+0000 7f94b5ffb700 1 -- 172.21.15.16:0/1141638545 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f94a8001670 con 0x7f94b8097280 2019-12-05T06:53:59.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.686+0000 7f94bee20700 1 -- 172.21.15.16:0/1141638545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94b8084b90 con 0x7f94b8097280 2019-12-05T06:53:59.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.686+0000 7f94bee20700 1 -- 172.21.15.16:0/1141638545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94b8098790 con 0x7f94b8097280 2019-12-05T06:53:59.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.687+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=THROTTLE_DONE pgs=1530 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.688+0000 7f94b5ffb700 1 -- 172.21.15.16:0/1141638545 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f94a800a6b0 con 0x7f94b8097280 2019-12-05T06:53:59.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.688+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=THROTTLE_DONE pgs=1530 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.688+0000 7f94b5ffb700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9498023630 0x7f9498025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:53:59.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.688+0000 7f94b5ffb700 1 -- 172.21.15.16:0/1141638545 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f94a8045d50 con 0x7f94b8097280 2019-12-05T06:53:59.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.689+0000 7f94bd3bd700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9498023630 0x7f9498025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:53:59.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.689+0000 7f94bd3bd700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9498023630 0x7f9498025ac0 crc :-1 s=READY pgs=639 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:53:59.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.691+0000 7f94bee20700 1 -- 172.21.15.16:0/1141638545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94b80958e0 con 0x7f94b8097280 2019-12-05T06:53:59.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.695+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=THROTTLE_DONE pgs=1530 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.695+0000 7f94b5ffb700 1 -- 172.21.15.16:0/1141638545 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f94a8034f80 con 0x7f94b8097280 2019-12-05T06:53:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.906+0000 7f94bee20700 1 -- 172.21.15.16:0/1141638545 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f94b80958e0 con 0x7f94b8097280 2019-12-05T06:53:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.907+0000 7f94bdbbe700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=THROTTLE_DONE pgs=1530 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:53:59.907 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.907+0000 7f94b5ffb700 1 -- 172.21.15.16:0/1141638545 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f94a8034f80 con 0x7f94b8097280 2019-12-05T06:53:59.908 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 187 2019-12-05T06:53:59.908 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:53:59.909 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:53:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.909+0000 7f94bee20700 1 -- 172.21.15.16:0/1141638545 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9498023630 msgr2=0x7f9498025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:59.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.909+0000 7f94bee20700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9498023630 0x7f9498025ac0 crc :-1 s=READY pgs=639 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.909+0000 7f94bee20700 1 -- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 msgr2=0x7f94b8097700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:53:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.909+0000 7f94bee20700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 0x7f94b8097700 secure :-1 s=READY pgs=1530 cs=0 l=1 rx=0x7f94a80103e0 tx=0x7f94a8010260).stop 2019-12-05T06:53:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.910+0000 7f94bee20700 1 -- 172.21.15.16:0/1141638545 shutdown_connections 2019-12-05T06:53:59.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.910+0000 7f94bee20700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9498023630 0x7f9498025ac0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.910+0000 7f94bee20700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f94b8097280 0x7f94b8097700 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.910+0000 7f94bee20700 1 --2- 172.21.15.16:0/1141638545 >> v2:172.21.15.76:3300/0 conn(0x7f94b8084f30 0x7f94b809b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:59.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.910+0000 7f94bee20700 1 --2- 172.21.15.16:0/1141638545 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f94b8096e90 0x7f94b809b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:53:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.910+0000 7f94bee20700 1 -- 172.21.15.16:0/1141638545 shutdown_connections 2019-12-05T06:53:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.910+0000 7f94bee20700 1 -- 172.21.15.16:0/1141638545 wait complete. 2019-12-05T06:53:59.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:53:59.910+0000 7f94bee20700 1 -- 172.21.15.16:0/1141638545 >> 172.21.15.16:0/1141638545 conn(0x7f94b80714f0 msgr2=0x7f94b809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:53:59.922 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:53:59.922 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 17 secs before thrashing 2019-12-05T06:54:16.494 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:54:16.499 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:54:16.924 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.137+0000 7f521e1e3700 1 Processor -- start 2019-12-05T06:54:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.137+0000 7f521e1e3700 1 -- start start 2019-12-05T06:54:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f521e1e3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f521808c6d0 0x7f521808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f521e1e3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f521808f720 0x7f521808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:17.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f521e1e3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521808fb70 con 0x7f521808fef0 2019-12-05T06:54:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f521e1e3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f521808df90 con 0x7f521808c6d0 2019-12-05T06:54:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f521e1e3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52180a3f70 con 0x7f521808f720 2019-12-05T06:54:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f5217fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f521808c6d0 0x7f521808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:17.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f5217fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f521808c6d0 0x7f521808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35364/0 (socket says 172.21.15.16:35364) 2019-12-05T06:54:17.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f5217fff700 1 -- 172.21.15.16:0/4028569759 learned_addr learned my addr 172.21.15.16:0/4028569759 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:17.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f521cf81700 1 --2- 172.21.15.16:0/4028569759 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f521808f720 0x7f521808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:17.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f5217fff700 1 -- 172.21.15.16:0/4028569759 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f521808f720 msgr2=0x7f521808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:17.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f5217fff700 1 --2- 172.21.15.16:0/4028569759 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f521808f720 0x7f521808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:17.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f5217fff700 1 -- 172.21.15.16:0/4028569759 >> v1:172.21.15.76:6789/0 conn(0x7f521808fef0 legacy=0x7f521808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f5217fff700 1 -- 172.21.15.16:0/4028569759 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521808d540 con 0x7f521808c6d0 2019-12-05T06:54:17.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f521cf81700 1 --2- 172.21.15.16:0/4028569759 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f521808f720 0x7f521808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:54:17.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.138+0000 7f5217fff700 1 --2- 172.21.15.16:0/4028569759 >> v2:172.21.15.16:3300/0 conn(0x7f521808c6d0 0x7f521808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f520800f740 tx=0x7f520800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.139+0000 7f5217fff700 1 --2- 172.21.15.16:0/4028569759 >> v2:172.21.15.16:3300/0 conn(0x7f521808c6d0 0x7f521808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f520800f740 tx=0x7f520800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.139+0000 7f5217fff700 1 --2- 172.21.15.16:0/4028569759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f521808c6d0 0x7f521808cac0 secure :-1 s=READY pgs=1531 cs=0 l=1 rx=0x7f520800f740 tx=0x7f520800fae0).ready entity=mon.0 client_cookie=6b55b1e9e0d1e1fc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:17.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.139+0000 7f5217fff700 1 --2- 172.21.15.16:0/4028569759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f521808c6d0 0x7f521808cac0 secure :-1 s=THROTTLE_DONE pgs=1531 cs=0 l=1 rx=0x7f520800f740 tx=0x7f520800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.139+0000 7f5217fff700 1 --2- 172.21.15.16:0/4028569759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f521808c6d0 0x7f521808cac0 secure :-1 s=THROTTLE_DONE pgs=1531 cs=0 l=1 rx=0x7f520800f740 tx=0x7f520800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.139+0000 7f52177fe700 1 -- 172.21.15.16:0/4028569759 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5208000960 con 0x7f521808c6d0 2019-12-05T06:54:17.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.139+0000 7f5217fff700 1 --2- 172.21.15.16:0/4028569759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f521808c6d0 0x7f521808cac0 secure :-1 s=THROTTLE_DONE pgs=1531 cs=0 l=1 rx=0x7f520800f740 tx=0x7f520800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.139+0000 7f52177fe700 1 -- 172.21.15.16:0/4028569759 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5208000c90 con 0x7f521808c6d0 2019-12-05T06:54:17.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.139+0000 7f52177fe700 1 -- 172.21.15.16:0/4028569759 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52080013a0 con 0x7f521808c6d0 2019-12-05T06:54:17.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.139+0000 7f521e1e3700 1 -- 172.21.15.16:0/4028569759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f521808c6d0 msgr2=0x7f521808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.139+0000 7f521e1e3700 1 --2- 172.21.15.16:0/4028569759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f521808c6d0 0x7f521808cac0 secure :-1 s=READY pgs=1531 cs=0 l=1 rx=0x7f520800f740 tx=0x7f520800fae0).stop 2019-12-05T06:54:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.140+0000 7f521e1e3700 1 -- 172.21.15.16:0/4028569759 shutdown_connections 2019-12-05T06:54:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.140+0000 7f521e1e3700 1 --2- 172.21.15.16:0/4028569759 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f521808f720 0x7f521808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.140+0000 7f521e1e3700 1 --2- 172.21.15.16:0/4028569759 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f521808c6d0 0x7f521808cac0 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.140+0000 7f521e1e3700 1 -- 172.21.15.16:0/4028569759 shutdown_connections 2019-12-05T06:54:17.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.140+0000 7f521e1e3700 1 -- 172.21.15.16:0/4028569759 wait complete. 2019-12-05T06:54:17.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.140+0000 7f521e1e3700 1 -- 172.21.15.16:0/4028569759 >> 172.21.15.16:0/4028569759 conn(0x7f5218061940 msgr2=0x7f5218096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.143+0000 7f521e1e3700 1 Processor -- start 2019-12-05T06:54:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.143+0000 7f521e1e3700 1 -- start start 2019-12-05T06:54:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.144+0000 7f521e1e3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5218132630 0x7f5218132a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:17.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.144+0000 7f521e1e3700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5218099eb0 0x7f521809a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:17.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.144+0000 7f521e1e3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f521806e7c0 con 0x7f521809ae80 2019-12-05T06:54:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.144+0000 7f521e1e3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5218098f20 con 0x7f5218132630 2019-12-05T06:54:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.144+0000 7f521e1e3700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5218134390 con 0x7f5218099eb0 2019-12-05T06:54:17.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5218132630 0x7f5218132a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f5218132630 0x7f5218132a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35370/0 (socket says 172.21.15.16:35370) 2019-12-05T06:54:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 -- 172.21.15.16:0/3055233179 learned_addr learned my addr 172.21.15.16:0/3055233179 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f521cf81700 1 --2- 172.21.15.16:0/3055233179 >> v2:172.21.15.76:3300/0 conn(0x7f5218099eb0 0x7f521809a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 -- 172.21.15.16:0/3055233179 >> v2:172.21.15.76:3300/0 conn(0x7f5218099eb0 msgr2=0x7f521809a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> v2:172.21.15.76:3300/0 conn(0x7f5218099eb0 0x7f521809a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 -- 172.21.15.16:0/3055233179 >> v1:172.21.15.76:6789/0 conn(0x7f521809ae80 legacy=0x7f5218132f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 -- 172.21.15.16:0/3055233179 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52180995b0 con 0x7f5218132630 2019-12-05T06:54:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> v2:172.21.15.16:3300/0 conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> v2:172.21.15.16:3300/0 conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=READY pgs=1532 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=THROTTLE_DONE pgs=1532 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=THROTTLE_DONE pgs=1532 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f52157fa700 1 -- 172.21.15.16:0/3055233179 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5208005770 con 0x7f5218132630 2019-12-05T06:54:17.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=THROTTLE_DONE pgs=1532 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f52157fa700 1 -- 172.21.15.16:0/3055233179 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5208005390 con 0x7f5218132630 2019-12-05T06:54:17.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f52157fa700 1 -- 172.21.15.16:0/3055233179 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5208005d50 con 0x7f5218132630 2019-12-05T06:54:17.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f521e1e3700 1 -- 172.21.15.16:0/3055233179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52180995b0 con 0x7f5218132630 2019-12-05T06:54:17.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.145+0000 7f521e1e3700 1 -- 172.21.15.16:0/3055233179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52180995b0 con 0x7f5218132630 2019-12-05T06:54:17.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.146+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=THROTTLE_DONE pgs=1532 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.146+0000 7f52157fa700 1 -- 172.21.15.16:0/3055233179 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5208005580 con 0x7f5218132630 2019-12-05T06:54:17.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.146+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=THROTTLE_DONE pgs=1532 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.147+0000 7f52157fa700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51fc023630 0x7f51fc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.147+0000 7f52157fa700 1 -- 172.21.15.16:0/3055233179 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f5208045dd0 con 0x7f5218132630 2019-12-05T06:54:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.147+0000 7f520f7fe700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51fc023630 0x7f51fc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.147+0000 7f520f7fe700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51fc023630 0x7f51fc025ac0 crc :-1 s=READY pgs=640 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:17.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.151+0000 7f521e1e3700 1 -- 172.21.15.16:0/3055233179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52180995b0 con 0x7f5218132630 2019-12-05T06:54:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.156+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=THROTTLE_DONE pgs=1532 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.156+0000 7f52157fa700 1 -- 172.21.15.16:0/3055233179 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5208005180 con 0x7f5218132630 2019-12-05T06:54:17.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.352+0000 7f521e1e3700 1 -- 172.21.15.16:0/3055233179 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f52180995b0 con 0x7f5218132630 2019-12-05T06:54:17.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.352+0000 7f5217fff700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=THROTTLE_DONE pgs=1532 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.352 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.352+0000 7f52157fa700 1 -- 172.21.15.16:0/3055233179 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f5208005560 con 0x7f5218132630 2019-12-05T06:54:17.353 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 187 2019-12-05T06:54:17.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:17.353 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:17.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 -- 172.21.15.16:0/3055233179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51fc023630 msgr2=0x7f51fc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51fc023630 0x7f51fc025ac0 crc :-1 s=READY pgs=640 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 -- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 msgr2=0x7f5218132a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.355 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 0x7f5218132a20 secure :-1 s=READY pgs=1532 cs=0 l=1 rx=0x7f5208010410 tx=0x7f5208010290).stop 2019-12-05T06:54:17.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 -- 172.21.15.16:0/3055233179 shutdown_connections 2019-12-05T06:54:17.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 --2- 172.21.15.16:0/3055233179 >> v2:172.21.15.76:3300/0 conn(0x7f5218099eb0 0x7f521809a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51fc023630 0x7f51fc025ac0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 --2- 172.21.15.16:0/3055233179 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5218132630 0x7f5218132a20 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 -- 172.21.15.16:0/3055233179 shutdown_connections 2019-12-05T06:54:17.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 -- 172.21.15.16:0/3055233179 wait complete. 2019-12-05T06:54:17.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.355+0000 7f521e1e3700 1 -- 172.21.15.16:0/3055233179 >> 172.21.15.16:0/3055233179 conn(0x7f52180714f0 msgr2=0x7f521809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:17.372 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:54:17.373 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:54:17.373 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:54:17.388 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:54:17.389 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:17.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.616+0000 7fd29ede3700 1 Processor -- start 2019-12-05T06:54:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.616+0000 7fd29ede3700 1 -- start start 2019-12-05T06:54:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29ede3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd29808d2c0 0x7fd29808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29ede3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd29808fef0 0x7fd29808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29ede3700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2980958e0 con 0x7fd29808c6d0 2019-12-05T06:54:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29ede3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd29808df90 con 0x7fd29808d2c0 2019-12-05T06:54:17.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29ede3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2980a3f70 con 0x7fd29808fef0 2019-12-05T06:54:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29d380700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd29808c6d0 0x7fd29808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48782/0 (socket says 172.21.15.16:48782) 2019-12-05T06:54:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29d380700 1 -- 172.21.15.16:0/2812316920 learned_addr learned my addr 172.21.15.16:0/2812316920 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2812316920 >> v2:172.21.15.16:3300/0 conn(0x7fd29808d2c0 0x7fd29808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29d380700 1 -- 172.21.15.16:0/2812316920 >> v1:172.21.15.16:6789/0 conn(0x7fd29808c6d0 legacy=0x7fd29808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2019-12-05T06:54:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29d380700 1 -- 172.21.15.16:0/2812316920 >> v1:172.21.15.16:6789/0 conn(0x7fd29808c6d0 legacy=0x7fd29808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:54:17.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29d380700 1 --1- 172.21.15.16:0/2812316920 >> v1:172.21.15.16:6789/0 conn(0x7fd29808c6d0 0x7fd29808cac0 :-1 s=OPENED pgs=1533 cs=1 l=1).handle_message read tag failed 2019-12-05T06:54:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29d380700 1 --1- 172.21.15.16:0/2812316920 >> v1:172.21.15.16:6789/0 conn(0x7fd29808c6d0 0x7fd29808cac0 :-1 s=OPENED pgs=1533 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:54:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.618+0000 7fd29db81700 1 --2- 172.21.15.16:0/2812316920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd29808fef0 0x7fd29808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.619+0000 7fd29cb7f700 1 -- 172.21.15.16:0/2812316920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd29808fef0 msgr2=0x7fd29808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.619+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2812316920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd29808fef0 0x7fd29808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.619+0000 7fd29cb7f700 1 -- 172.21.15.16:0/2812316920 >> v1:172.21.15.16:6789/0 conn(0x7fd29808c6d0 legacy=0x7fd29808cac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:54:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.619+0000 7fd29cb7f700 1 -- 172.21.15.16:0/2812316920 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd29808e620 con 0x7fd29808d2c0 2019-12-05T06:54:17.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.619+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2812316920 >> v2:172.21.15.16:3300/0 conn(0x7fd29808d2c0 0x7fd29808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd28c00af50 tx=0x7fd28c00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.619+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2812316920 >> v2:172.21.15.16:3300/0 conn(0x7fd29808d2c0 0x7fd29808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd28c00af50 tx=0x7fd28c00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.619+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2812316920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29808d2c0 0x7fd29808ef20 secure :-1 s=READY pgs=1534 cs=0 l=1 rx=0x7fd28c00af50 tx=0x7fd28c00b250).ready entity=mon.0 client_cookie=629614c94c068a78 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.620+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2812316920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29808d2c0 0x7fd29808ef20 secure :-1 s=THROTTLE_DONE pgs=1534 cs=0 l=1 rx=0x7fd28c00af50 tx=0x7fd28c00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.620+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2812316920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29808d2c0 0x7fd29808ef20 secure :-1 s=THROTTLE_DONE pgs=1534 cs=0 l=1 rx=0x7fd28c00af50 tx=0x7fd28c00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.620+0000 7fd297fff700 1 -- 172.21.15.16:0/2812316920 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd28c007210 con 0x7fd29808d2c0 2019-12-05T06:54:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.620+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2812316920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29808d2c0 0x7fd29808ef20 secure :-1 s=THROTTLE_DONE pgs=1534 cs=0 l=1 rx=0x7fd28c00af50 tx=0x7fd28c00b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.620+0000 7fd297fff700 1 -- 172.21.15.16:0/2812316920 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c0099b0 con 0x7fd29808d2c0 2019-12-05T06:54:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.620+0000 7fd297fff700 1 -- 172.21.15.16:0/2812316920 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd28c000960 con 0x7fd29808d2c0 2019-12-05T06:54:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.620+0000 7fd29ede3700 1 -- 172.21.15.16:0/2812316920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29808d2c0 msgr2=0x7fd29808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.620+0000 7fd29ede3700 1 --2- 172.21.15.16:0/2812316920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29808d2c0 0x7fd29808ef20 secure :-1 s=READY pgs=1534 cs=0 l=1 rx=0x7fd28c00af50 tx=0x7fd28c00b250).stop 2019-12-05T06:54:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.622+0000 7fd29ede3700 1 -- 172.21.15.16:0/2812316920 shutdown_connections 2019-12-05T06:54:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.622+0000 7fd29ede3700 1 --2- 172.21.15.16:0/2812316920 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd29808d2c0 0x7fd29808ef20 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.622+0000 7fd29ede3700 1 --2- 172.21.15.16:0/2812316920 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd29808fef0 0x7fd29808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:17.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.622+0000 7fd29ede3700 1 -- 172.21.15.16:0/2812316920 shutdown_connections 2019-12-05T06:54:17.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.622+0000 7fd29ede3700 1 -- 172.21.15.16:0/2812316920 wait complete. 2019-12-05T06:54:17.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.622+0000 7fd29ede3700 1 -- 172.21.15.16:0/2812316920 >> 172.21.15.16:0/2812316920 conn(0x7fd298061940 msgr2=0x7fd298096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:17.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.627+0000 7fd29ede3700 1 Processor -- start 2019-12-05T06:54:17.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.627+0000 7fd29ede3700 1 -- start start 2019-12-05T06:54:17.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.629+0000 7fd29ede3700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd2980992d0 0x7fd298098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.629+0000 7fd29ede3700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd298156cb0 0x7fd2981590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.629+0000 7fd29ede3700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd298089fb0 con 0x7fd29806df10 2019-12-05T06:54:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.629+0000 7fd29ede3700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd298097370 con 0x7fd2980992d0 2019-12-05T06:54:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.629+0000 7fd29ede3700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29808ad00 con 0x7fd298156cb0 2019-12-05T06:54:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.629+0000 7fd29cb7f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd2980992d0 0x7fd298098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.629+0000 7fd29cb7f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd2980992d0 0x7fd298098b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35390/0 (socket says 172.21.15.16:35390) 2019-12-05T06:54:17.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.629+0000 7fd29d380700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fd29806df10 0x7fd298098300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:41858/0 (socket says 172.21.15.16:41858) 2019-12-05T06:54:17.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.629+0000 7fd29cb7f700 1 -- 172.21.15.16:0/2427442585 learned_addr learned my addr 172.21.15.16:0/2427442585 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:17.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.629+0000 7fd29db81700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd298156cb0 0x7fd2981590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:17.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29cb7f700 1 -- 172.21.15.16:0/2427442585 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd298156cb0 msgr2=0x7fd2981590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd298156cb0 0x7fd2981590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29cb7f700 1 -- 172.21.15.16:0/2427442585 >> v1:172.21.15.76:6789/0 conn(0x7fd29806df10 legacy=0x7fd298098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29cb7f700 1 -- 172.21.15.16:0/2427442585 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd298097a00 con 0x7fd2980992d0 2019-12-05T06:54:17.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd295ffb700 1 -- 172.21.15.16:0/2427442585 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2204785044 0 0) 0x7fd2900016e0 con 0x7fd29806df10 2019-12-05T06:54:17.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> v2:172.21.15.16:3300/0 conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> v2:172.21.15.16:3300/0 conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=READY pgs=1535 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:17.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=THROTTLE_DONE pgs=1535 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd295ffb700 1 -- 172.21.15.16:0/2427442585 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd28c00abb0 con 0x7fd2980992d0 2019-12-05T06:54:17.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=THROTTLE_DONE pgs=1535 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=THROTTLE_DONE pgs=1535 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd295ffb700 1 -- 172.21.15.16:0/2427442585 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28c009770 con 0x7fd2980992d0 2019-12-05T06:54:17.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd29ede3700 1 -- 172.21.15.16:0/2427442585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd298097a00 con 0x7fd2980992d0 2019-12-05T06:54:17.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.630+0000 7fd295ffb700 1 -- 172.21.15.16:0/2427442585 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd28c009e00 con 0x7fd2980992d0 2019-12-05T06:54:17.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.631+0000 7fd29ede3700 1 -- 172.21.15.16:0/2427442585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd298097a00 con 0x7fd2980992d0 2019-12-05T06:54:17.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.632+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=THROTTLE_DONE pgs=1535 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.632+0000 7fd295ffb700 1 -- 172.21.15.16:0/2427442585 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd28c00abb0 con 0x7fd2980992d0 2019-12-05T06:54:17.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.632+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=THROTTLE_DONE pgs=1535 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.633+0000 7fd295ffb700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd278023630 0x7fd278025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:17.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.633+0000 7fd295ffb700 1 -- 172.21.15.16:0/2427442585 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fd28c089e40 con 0x7fd2980992d0 2019-12-05T06:54:17.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.633+0000 7fd29d380700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd278023630 0x7fd278025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:17.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.634+0000 7fd29d380700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd278023630 0x7fd278025ac0 crc :-1 s=READY pgs=641 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:17.648 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.648+0000 7fd29ede3700 1 -- 172.21.15.16:0/2427442585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd298097a00 con 0x7fd2980992d0 2019-12-05T06:54:17.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.655+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=THROTTLE_DONE pgs=1535 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.655+0000 7fd295ffb700 1 -- 172.21.15.16:0/2427442585 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd28c00b050 con 0x7fd2980992d0 2019-12-05T06:54:17.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.924+0000 7fd29ede3700 1 -- 172.21.15.16:0/2427442585 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd298097a00 con 0x7fd2980992d0 2019-12-05T06:54:17.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.925+0000 7fd29cb7f700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=THROTTLE_DONE pgs=1535 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:17.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.925+0000 7fd295ffb700 1 -- 172.21.15.16:0/2427442585 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fd28c00abb0 con 0x7fd2980992d0 2019-12-05T06:54:17.926 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 187 2019-12-05T06:54:17.926 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:17.926 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:17.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 -- 172.21.15.16:0/2427442585 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd278023630 msgr2=0x7fd278025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd278023630 0x7fd278025ac0 crc :-1 s=READY pgs=641 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 -- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 msgr2=0x7fd298098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 0x7fd298098b00 secure :-1 s=READY pgs=1535 cs=0 l=1 rx=0x7fd28c054440 tx=0x7fd28c0542c0).stop 2019-12-05T06:54:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 -- 172.21.15.16:0/2427442585 shutdown_connections 2019-12-05T06:54:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd298156cb0 0x7fd2981590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd278023630 0x7fd278025ac0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 --2- 172.21.15.16:0/2427442585 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd2980992d0 0x7fd298098b00 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 -- 172.21.15.16:0/2427442585 shutdown_connections 2019-12-05T06:54:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 -- 172.21.15.16:0/2427442585 wait complete. 2019-12-05T06:54:17.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:17.927+0000 7fd29ede3700 1 -- 172.21.15.16:0/2427442585 >> 172.21.15.16:0/2427442585 conn(0x7fd2980714f0 msgr2=0x7fd29809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:17.944 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:54:19.945 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:20.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.194+0000 7fa067152700 1 Processor -- start 2019-12-05T06:54:20.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.195+0000 7fa067152700 1 -- start start 2019-12-05T06:54:20.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.196+0000 7fa067152700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa06008c6d0 0x7fa06008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:20.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.196+0000 7fa067152700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06008f720 0x7fa06008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:20.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.196+0000 7fa067152700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa06008fb70 con 0x7fa06008fef0 2019-12-05T06:54:20.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.197+0000 7fa067152700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06008df90 con 0x7fa06008c6d0 2019-12-05T06:54:20.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.197+0000 7fa067152700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa0600a3f70 con 0x7fa06008f720 2019-12-05T06:54:20.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.197+0000 7fa0656ef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa06008c6d0 0x7fa06008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:20.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.197+0000 7fa064eee700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa06008fef0 0x7fa06008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48804/0 (socket says 172.21.15.16:48804) 2019-12-05T06:54:20.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.197+0000 7fa064eee700 1 -- 172.21.15.16:0/193882870 learned_addr learned my addr 172.21.15.16:0/193882870 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:20.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.197+0000 7fa065ef0700 1 --2- 172.21.15.16:0/193882870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06008f720 0x7fa06008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:20.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.197+0000 7fa057fff700 1 -- 172.21.15.16:0/193882870 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2846176844 0 0) 0x7fa05c001150 con 0x7fa06008fef0 2019-12-05T06:54:20.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa057fff700 1 -- 172.21.15.16:0/193882870 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa048002e80 con 0x7fa06008fef0 2019-12-05T06:54:20.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa0656ef700 1 -- 172.21.15.16:0/193882870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06008f720 msgr2=0x7fa06008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa0656ef700 1 --2- 172.21.15.16:0/193882870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06008f720 0x7fa06008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa0656ef700 1 -- 172.21.15.16:0/193882870 >> v1:172.21.15.16:6789/0 conn(0x7fa06008fef0 legacy=0x7fa06008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa0656ef700 1 -- 172.21.15.16:0/193882870 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06008d540 con 0x7fa06008c6d0 2019-12-05T06:54:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa065ef0700 1 --2- 172.21.15.16:0/193882870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06008f720 0x7fa06008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:54:20.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa0656ef700 1 --2- 172.21.15.16:0/193882870 >> v2:172.21.15.16:3300/0 conn(0x7fa06008c6d0 0x7fa06008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa05800b2d0 tx=0x7fa05800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa0656ef700 1 --2- 172.21.15.16:0/193882870 >> v2:172.21.15.16:3300/0 conn(0x7fa06008c6d0 0x7fa06008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa05800b2d0 tx=0x7fa05800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa0656ef700 1 --2- 172.21.15.16:0/193882870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06008c6d0 0x7fa06008cac0 secure :-1 s=READY pgs=1537 cs=0 l=1 rx=0x7fa05800b2d0 tx=0x7fa05800bae0).ready entity=mon.0 client_cookie=23b4418c0fa63e64 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:20.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa0656ef700 1 --2- 172.21.15.16:0/193882870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06008c6d0 0x7fa06008cac0 secure :-1 s=THROTTLE_DONE pgs=1537 cs=0 l=1 rx=0x7fa05800b2d0 tx=0x7fa05800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.198+0000 7fa057fff700 1 -- 172.21.15.16:0/193882870 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa058007590 con 0x7fa06008c6d0 2019-12-05T06:54:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.199+0000 7fa0656ef700 1 --2- 172.21.15.16:0/193882870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06008c6d0 0x7fa06008cac0 secure :-1 s=THROTTLE_DONE pgs=1537 cs=0 l=1 rx=0x7fa05800b2d0 tx=0x7fa05800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.199+0000 7fa0656ef700 1 --2- 172.21.15.16:0/193882870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06008c6d0 0x7fa06008cac0 secure :-1 s=THROTTLE_DONE pgs=1537 cs=0 l=1 rx=0x7fa05800b2d0 tx=0x7fa05800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.199+0000 7fa057fff700 1 -- 172.21.15.16:0/193882870 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa058009d30 con 0x7fa06008c6d0 2019-12-05T06:54:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.199+0000 7fa057fff700 1 -- 172.21.15.16:0/193882870 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa05800a400 con 0x7fa06008c6d0 2019-12-05T06:54:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.199+0000 7fa067152700 1 -- 172.21.15.16:0/193882870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06008c6d0 msgr2=0x7fa06008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:20.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.199+0000 7fa067152700 1 --2- 172.21.15.16:0/193882870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06008c6d0 0x7fa06008cac0 secure :-1 s=READY pgs=1537 cs=0 l=1 rx=0x7fa05800b2d0 tx=0x7fa05800bae0).stop 2019-12-05T06:54:20.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.200+0000 7fa067152700 1 -- 172.21.15.16:0/193882870 shutdown_connections 2019-12-05T06:54:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.200+0000 7fa067152700 1 --2- 172.21.15.16:0/193882870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06008f720 0x7fa06008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.201+0000 7fa067152700 1 --2- 172.21.15.16:0/193882870 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa06008c6d0 0x7fa06008cac0 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.201+0000 7fa067152700 1 -- 172.21.15.16:0/193882870 shutdown_connections 2019-12-05T06:54:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.201+0000 7fa067152700 1 -- 172.21.15.16:0/193882870 wait complete. 2019-12-05T06:54:20.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.201+0000 7fa067152700 1 -- 172.21.15.16:0/193882870 >> 172.21.15.16:0/193882870 conn(0x7fa060061940 msgr2=0x7fa060096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.206+0000 7fa067152700 1 Processor -- start 2019-12-05T06:54:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.206+0000 7fa067152700 1 -- start start 2019-12-05T06:54:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.208+0000 7fa067152700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa060096e90 0x7fa06009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.208+0000 7fa067152700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06009b920 0x7fa060097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.208+0000 7fa067152700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa060084b90 con 0x7fa060084f30 2019-12-05T06:54:20.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.208+0000 7fa067152700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa06009a450 con 0x7fa060096e90 2019-12-05T06:54:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.208+0000 7fa067152700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa060099a20 con 0x7fa06009b920 2019-12-05T06:54:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.208+0000 7fa0656ef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa060096e90 0x7fa06009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.208+0000 7fa064eee700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fa060084f30 0x7fa060097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:41878/0 (socket says 172.21.15.16:41878) 2019-12-05T06:54:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.208+0000 7fa0656ef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa060096e90 0x7fa06009b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35406/0 (socket says 172.21.15.16:35406) 2019-12-05T06:54:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.208+0000 7fa064eee700 1 -- 172.21.15.16:0/3551937629 learned_addr learned my addr 172.21.15.16:0/3551937629 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:20.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.208+0000 7fa065ef0700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06009b920 0x7fa060097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa055ffb700 1 -- 172.21.15.16:0/3551937629 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2932116843 0 0) 0x7fa05c000f40 con 0x7fa060084f30 2019-12-05T06:54:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa055ffb700 1 -- 172.21.15.16:0/3551937629 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa040002e80 con 0x7fa060084f30 2019-12-05T06:54:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa0656ef700 1 -- 172.21.15.16:0/3551937629 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06009b920 msgr2=0x7fa060097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06009b920 0x7fa060097a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa0656ef700 1 -- 172.21.15.16:0/3551937629 >> v1:172.21.15.76:6789/0 conn(0x7fa060084f30 legacy=0x7fa060097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa0656ef700 1 -- 172.21.15.16:0/3551937629 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa06009aae0 con 0x7fa060096e90 2019-12-05T06:54:20.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> v2:172.21.15.16:3300/0 conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> v2:172.21.15.16:3300/0 conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=READY pgs=1538 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:20.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=THROTTLE_DONE pgs=1538 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=THROTTLE_DONE pgs=1538 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa055ffb700 1 -- 172.21.15.16:0/3551937629 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa058000cc0 con 0x7fa060096e90 2019-12-05T06:54:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.209+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=THROTTLE_DONE pgs=1538 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.210+0000 7fa055ffb700 1 -- 172.21.15.16:0/3551937629 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa058009af0 con 0x7fa060096e90 2019-12-05T06:54:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.210+0000 7fa067152700 1 -- 172.21.15.16:0/3551937629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa06009aae0 con 0x7fa060096e90 2019-12-05T06:54:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.210+0000 7fa055ffb700 1 -- 172.21.15.16:0/3551937629 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa05800a260 con 0x7fa060096e90 2019-12-05T06:54:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.210+0000 7fa067152700 1 -- 172.21.15.16:0/3551937629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa06009aae0 con 0x7fa060096e90 2019-12-05T06:54:20.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.211+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=THROTTLE_DONE pgs=1538 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.211+0000 7fa055ffb700 1 -- 172.21.15.16:0/3551937629 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa058009ce0 con 0x7fa060096e90 2019-12-05T06:54:20.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.211+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=THROTTLE_DONE pgs=1538 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.212+0000 7fa055ffb700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa040023670 0x7fa040025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:20.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.212+0000 7fa055ffb700 1 -- 172.21.15.16:0/3551937629 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fa0580460f0 con 0x7fa060096e90 2019-12-05T06:54:20.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.212+0000 7fa064eee700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa040023670 0x7fa040025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:20.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.213+0000 7fa064eee700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa040023670 0x7fa040025ae0 crc :-1 s=READY pgs=642 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:20.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.226+0000 7fa067152700 1 -- 172.21.15.16:0/3551937629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa06009aae0 con 0x7fa060096e90 2019-12-05T06:54:20.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.232+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=THROTTLE_DONE pgs=1538 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.233+0000 7fa055ffb700 1 -- 172.21.15.16:0/3551937629 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa05800b750 con 0x7fa060096e90 2019-12-05T06:54:20.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.480+0000 7fa067152700 1 -- 172.21.15.16:0/3551937629 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa06009aae0 con 0x7fa060096e90 2019-12-05T06:54:20.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.481+0000 7fa0656ef700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=THROTTLE_DONE pgs=1538 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:20.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.481+0000 7fa055ffb700 1 -- 172.21.15.16:0/3551937629 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fa058009cc0 con 0x7fa060096e90 2019-12-05T06:54:20.482 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 187 2019-12-05T06:54:20.482 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:20.482 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:20.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 -- 172.21.15.16:0/3551937629 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa040023670 msgr2=0x7fa040025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:20.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa040023670 0x7fa040025ae0 crc :-1 s=READY pgs=642 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:20.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 -- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 msgr2=0x7fa06009b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:20.484 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 0x7fa06009b3e0 secure :-1 s=READY pgs=1538 cs=0 l=1 rx=0x7fa058010730 tx=0x7fa058010530).stop 2019-12-05T06:54:20.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 -- 172.21.15.16:0/3551937629 shutdown_connections 2019-12-05T06:54:20.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa06009b920 0x7fa060097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:20.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa040023670 0x7fa040025ae0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:20.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 --2- 172.21.15.16:0/3551937629 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa060096e90 0x7fa06009b3e0 unknown :-1 s=CLOSED pgs=1538 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:20.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 -- 172.21.15.16:0/3551937629 shutdown_connections 2019-12-05T06:54:20.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 -- 172.21.15.16:0/3551937629 wait complete. 2019-12-05T06:54:20.485 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:20.484+0000 7fa067152700 1 -- 172.21.15.16:0/3551937629 >> 172.21.15.16:0/3551937629 conn(0x7fa0600714f0 msgr2=0x7fa06009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:20.494 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:54:22.495 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.768+0000 7f431d436700 1 Processor -- start 2019-12-05T06:54:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.768+0000 7f431d436700 1 -- start start 2019-12-05T06:54:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.770+0000 7f431d436700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f431808d2c0 0x7f431808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.770+0000 7f431d436700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 0x7f431808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.770+0000 7f431d436700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43180958e0 con 0x7f431808c6d0 2019-12-05T06:54:22.778 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.770+0000 7f431d436700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f431808df90 con 0x7f431808d2c0 2019-12-05T06:54:22.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.770+0000 7f431d436700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43180a3f70 con 0x7f431808fef0 2019-12-05T06:54:22.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.770+0000 7f43177fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f431808c6d0 0x7f431808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48816/0 (socket says 172.21.15.16:48816) 2019-12-05T06:54:22.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.770+0000 7f43177fe700 1 -- 172.21.15.16:0/3702981000 learned_addr learned my addr 172.21.15.16:0/3702981000 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:22.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.770+0000 7f4316ffd700 1 --2- 172.21.15.16:0/3702981000 >> v2:172.21.15.76:3300/0 conn(0x7f431808d2c0 0x7f431808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:22.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.770+0000 7f4317fff700 1 --2- 172.21.15.16:0/3702981000 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 0x7f431808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:22.779 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.771+0000 7f43167fc700 1 -- 172.21.15.16:0/3702981000 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3890534094 0 0) 0x7f43100014f0 con 0x7f431808c6d0 2019-12-05T06:54:22.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.771+0000 7f43167fc700 1 -- 172.21.15.16:0/3702981000 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4304002e80 con 0x7f431808c6d0 2019-12-05T06:54:22.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.771+0000 7f4317fff700 1 -- 172.21.15.16:0/3702981000 >> v2:172.21.15.76:3300/0 conn(0x7f431808d2c0 msgr2=0x7f431808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:22.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.771+0000 7f4317fff700 1 --2- 172.21.15.16:0/3702981000 >> v2:172.21.15.76:3300/0 conn(0x7f431808d2c0 0x7f431808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:22.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.771+0000 7f4317fff700 1 -- 172.21.15.16:0/3702981000 >> v1:172.21.15.16:6789/0 conn(0x7f431808c6d0 legacy=0x7f431808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:22.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.771+0000 7f4317fff700 1 -- 172.21.15.16:0/3702981000 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431808e620 con 0x7f431808fef0 2019-12-05T06:54:22.780 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.771+0000 7f4316ffd700 1 --2- 172.21.15.16:0/3702981000 >> v2:172.21.15.76:3300/0 conn(0x7f431808d2c0 0x7f431808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:54:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.772+0000 7f4317fff700 1 --2- 172.21.15.16:0/3702981000 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 0x7f431808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f430800af80 tx=0x7f430800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.772+0000 7f4317fff700 1 --2- 172.21.15.16:0/3702981000 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 0x7f431808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f430800af80 tx=0x7f430800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.772+0000 7f4317fff700 1 --2- 172.21.15.16:0/3702981000 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 0x7f431808f460 secure :-1 s=READY pgs=362 cs=0 l=1 rx=0x7f430800af80 tx=0x7f430800b790).ready entity=mon.2 client_cookie=b22affdf8160190d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.772+0000 7f4317fff700 1 --2- 172.21.15.16:0/3702981000 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 0x7f431808f460 secure :-1 s=THROTTLE_DONE pgs=362 cs=0 l=1 rx=0x7f430800af80 tx=0x7f430800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.772+0000 7f4317fff700 1 --2- 172.21.15.16:0/3702981000 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 0x7f431808f460 secure :-1 s=THROTTLE_DONE pgs=362 cs=0 l=1 rx=0x7f430800af80 tx=0x7f430800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.772+0000 7f4317fff700 1 --2- 172.21.15.16:0/3702981000 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 0x7f431808f460 secure :-1 s=THROTTLE_DONE pgs=362 cs=0 l=1 rx=0x7f430800af80 tx=0x7f430800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.781 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.772+0000 7f43167fc700 1 -- 172.21.15.16:0/3702981000 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4308000bc0 con 0x7f431808fef0 2019-12-05T06:54:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.773+0000 7f43167fc700 1 -- 172.21.15.16:0/3702981000 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43080097f0 con 0x7f431808fef0 2019-12-05T06:54:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.773+0000 7f43167fc700 1 -- 172.21.15.16:0/3702981000 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4308009e60 con 0x7f431808fef0 2019-12-05T06:54:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.773+0000 7f431d436700 1 -- 172.21.15.16:0/3702981000 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 msgr2=0x7f431808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.773+0000 7f431d436700 1 --2- 172.21.15.16:0/3702981000 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 0x7f431808f460 secure :-1 s=READY pgs=362 cs=0 l=1 rx=0x7f430800af80 tx=0x7f430800b790).stop 2019-12-05T06:54:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.774+0000 7f431d436700 1 -- 172.21.15.16:0/3702981000 shutdown_connections 2019-12-05T06:54:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.774+0000 7f431d436700 1 --2- 172.21.15.16:0/3702981000 >> v2:172.21.15.76:3300/0 conn(0x7f431808d2c0 0x7f431808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:22.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.774+0000 7f431d436700 1 --2- 172.21.15.16:0/3702981000 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f431808fef0 0x7f431808f460 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.775+0000 7f431d436700 1 -- 172.21.15.16:0/3702981000 shutdown_connections 2019-12-05T06:54:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.775+0000 7f431d436700 1 -- 172.21.15.16:0/3702981000 wait complete. 2019-12-05T06:54:22.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.775+0000 7f431d436700 1 -- 172.21.15.16:0/3702981000 >> 172.21.15.16:0/3702981000 conn(0x7f4318061940 msgr2=0x7f4318096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:22.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.780+0000 7f431d436700 1 Processor -- start 2019-12-05T06:54:22.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.780+0000 7f431d436700 1 -- start start 2019-12-05T06:54:22.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.782+0000 7f431d436700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:22.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.782+0000 7f431d436700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4318084b90 con 0x7f4318096e90 2019-12-05T06:54:22.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.782+0000 7f431d436700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4318098790 con 0x7f431809bbe0 2019-12-05T06:54:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.782+0000 7f431d436700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f431809a450 con 0x7f4318084f30 2019-12-05T06:54:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.782+0000 7f43177fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f4318096e90 0x7f431809b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48822/0 (socket says 172.21.15.16:48822) 2019-12-05T06:54:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.782+0000 7f43177fe700 1 -- 172.21.15.16:0/2787109689 learned_addr learned my addr 172.21.15.16:0/2787109689 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.782+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.783+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452480928 0 0) 0x7f43100014f0 con 0x7f4318096e90 2019-12-05T06:54:22.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.783+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42fc002e80 con 0x7f4318096e90 2019-12-05T06:54:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.783+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518295054 0 0) 0x7f430000db80 con 0x7f431809bbe0 2019-12-05T06:54:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.783+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42fc004220 con 0x7f431809bbe0 2019-12-05T06:54:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.783+0000 7f4317fff700 1 -- 172.21.15.16:0/2787109689 >> v1:172.21.15.76:6789/0 conn(0x7f431809bbe0 legacy=0x7f43181539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.783+0000 7f4317fff700 1 -- 172.21.15.16:0/2787109689 >> v1:172.21.15.16:6789/0 conn(0x7f4318096e90 legacy=0x7f431809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:22.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.783+0000 7f4317fff700 1 -- 172.21.15.16:0/2787109689 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f431809aae0 con 0x7f4318084f30 2019-12-05T06:54:22.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.783+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.783+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.783+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=READY pgs=363 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:22.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.784+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.784+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4308009990 con 0x7f4318084f30 2019-12-05T06:54:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.784+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.784+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.784+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f43080095b0 con 0x7f4318084f30 2019-12-05T06:54:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.784+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4308009f70 con 0x7f4318084f30 2019-12-05T06:54:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.784+0000 7f431d436700 1 -- 172.21.15.16:0/2787109689 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f431809aae0 con 0x7f4318084f30 2019-12-05T06:54:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.784+0000 7f431d436700 1 -- 172.21.15.16:0/2787109689 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f431809aae0 con 0x7f4318084f30 2019-12-05T06:54:22.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.785+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.786+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f4308009840 con 0x7f4318084f30 2019-12-05T06:54:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.786+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.787+0000 7f430ffff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023650 0x7f42fc025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.787+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f4308045dd0 con 0x7f4318084f30 2019-12-05T06:54:22.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.787+0000 7f43177fe700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023650 0x7f42fc025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:22.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.788+0000 7f43177fe700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023650 0x7f42fc025ae0 crc :-1 s=READY pgs=643 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:22.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.799+0000 7f431d436700 1 -- 172.21.15.16:0/2787109689 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f431809aae0 con 0x7f4318084f30 2019-12-05T06:54:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.808+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:22.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:22.809+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f4308002c30 con 0x7f4318084f30 2019-12-05T06:54:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.076+0000 7f431d436700 1 -- 172.21.15.16:0/2787109689 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f431809aae0 con 0x7f4318084f30 2019-12-05T06:54:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.078+0000 7f4317fff700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=THROTTLE_DONE pgs=363 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:23.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.078+0000 7f430ffff700 1 -- 172.21.15.16:0/2787109689 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f43080097a0 con 0x7f4318084f30 2019-12-05T06:54:23.079 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 187 2019-12-05T06:54:23.080 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:23.080 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:23.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.082+0000 7f431d436700 1 -- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023650 msgr2=0x7f42fc025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.082+0000 7f431d436700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023650 0x7f42fc025ae0 crc :-1 s=READY pgs=643 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.082+0000 7f431d436700 1 -- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 msgr2=0x7f4318157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.082+0000 7f431d436700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 secure :-1 s=READY pgs=363 cs=0 l=1 rx=0x7f43080103f0 tx=0x7f4308010790).stop 2019-12-05T06:54:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.082+0000 7f431d436700 1 -- 172.21.15.16:0/2787109689 shutdown_connections 2019-12-05T06:54:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.082+0000 7f431d436700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f42fc023650 0x7f42fc025ae0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.082+0000 7f431d436700 1 --2- 172.21.15.16:0/2787109689 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f4318084f30 0x7f4318157200 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:23.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.083+0000 7f431d436700 1 -- 172.21.15.16:0/2787109689 shutdown_connections 2019-12-05T06:54:23.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.083+0000 7f431d436700 1 -- 172.21.15.16:0/2787109689 wait complete. 2019-12-05T06:54:23.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:23.083+0000 7f431d436700 1 -- 172.21.15.16:0/2787109689 >> 172.21.15.16:0/2787109689 conn(0x7f43180714f0 msgr2=0x7f431809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:23.094 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:54:25.096 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:25.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.364+0000 7f951f35b700 1 Processor -- start 2019-12-05T06:54:25.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.365+0000 7f951f35b700 1 -- start start 2019-12-05T06:54:25.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.367+0000 7f951f35b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f951808c6d0 0x7f951808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:25.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.367+0000 7f951f35b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f951808fef0 0x7f951808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:25.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.367+0000 7f951f35b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95180958e0 con 0x7f951808ef20 2019-12-05T06:54:25.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.367+0000 7f951f35b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951808df90 con 0x7f951808c6d0 2019-12-05T06:54:25.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.367+0000 7f951f35b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95180a3f70 con 0x7f951808fef0 2019-12-05T06:54:25.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.367+0000 7f951d8f8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f951808c6d0 0x7f951808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:25.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.367+0000 7f951e0f9700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f951808ef20 0x7f951808f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48836/0 (socket says 172.21.15.16:48836) 2019-12-05T06:54:25.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.367+0000 7f951e0f9700 1 -- 172.21.15.16:0/110224321 learned_addr learned my addr 172.21.15.16:0/110224321 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:25.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.367+0000 7f951d0f7700 1 --2- 172.21.15.16:0/110224321 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f951808fef0 0x7f951808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.368+0000 7f951d8f8700 1 -- 172.21.15.16:0/110224321 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f951808fef0 msgr2=0x7f951808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.368+0000 7f951d8f8700 1 --2- 172.21.15.16:0/110224321 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f951808fef0 0x7f951808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.368+0000 7f951d8f8700 1 -- 172.21.15.16:0/110224321 >> v1:172.21.15.16:6789/0 conn(0x7f951808ef20 legacy=0x7f951808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.368+0000 7f951c8f6700 1 -- 172.21.15.16:0/110224321 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958598727 0 0) 0x7f9510001150 con 0x7f951808ef20 2019-12-05T06:54:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.368+0000 7f951d8f8700 1 -- 172.21.15.16:0/110224321 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951808d540 con 0x7f951808c6d0 2019-12-05T06:54:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.368+0000 7f951d8f8700 1 --2- 172.21.15.16:0/110224321 >> v2:172.21.15.16:3300/0 conn(0x7f951808c6d0 0x7f951808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f950800af50 tx=0x7f950800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.368+0000 7f951d8f8700 1 --2- 172.21.15.16:0/110224321 >> v2:172.21.15.16:3300/0 conn(0x7f951808c6d0 0x7f951808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f950800af50 tx=0x7f950800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.368+0000 7f951d8f8700 1 --2- 172.21.15.16:0/110224321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951808c6d0 0x7f951808cac0 secure :-1 s=READY pgs=1543 cs=0 l=1 rx=0x7f950800af50 tx=0x7f950800b6e0).ready entity=mon.0 client_cookie=f4e2985f3d7cb76f server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.369+0000 7f951d8f8700 1 --2- 172.21.15.16:0/110224321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951808c6d0 0x7f951808cac0 secure :-1 s=THROTTLE_DONE pgs=1543 cs=0 l=1 rx=0x7f950800af50 tx=0x7f950800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.369+0000 7f951d8f8700 1 --2- 172.21.15.16:0/110224321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951808c6d0 0x7f951808cac0 secure :-1 s=THROTTLE_DONE pgs=1543 cs=0 l=1 rx=0x7f950800af50 tx=0x7f950800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.369+0000 7f951c8f6700 1 -- 172.21.15.16:0/110224321 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9508007210 con 0x7f951808c6d0 2019-12-05T06:54:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.369+0000 7f951d8f8700 1 --2- 172.21.15.16:0/110224321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951808c6d0 0x7f951808cac0 secure :-1 s=THROTTLE_DONE pgs=1543 cs=0 l=1 rx=0x7f950800af50 tx=0x7f950800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.369+0000 7f951c8f6700 1 -- 172.21.15.16:0/110224321 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95080099b0 con 0x7f951808c6d0 2019-12-05T06:54:25.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.369+0000 7f951c8f6700 1 -- 172.21.15.16:0/110224321 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f950800a080 con 0x7f951808c6d0 2019-12-05T06:54:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.369+0000 7f951f35b700 1 -- 172.21.15.16:0/110224321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951808c6d0 msgr2=0x7f951808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.369+0000 7f951f35b700 1 --2- 172.21.15.16:0/110224321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951808c6d0 0x7f951808cac0 secure :-1 s=READY pgs=1543 cs=0 l=1 rx=0x7f950800af50 tx=0x7f950800b6e0).stop 2019-12-05T06:54:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.371+0000 7f951f35b700 1 -- 172.21.15.16:0/110224321 shutdown_connections 2019-12-05T06:54:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.371+0000 7f951f35b700 1 --2- 172.21.15.16:0/110224321 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f951808fef0 0x7f951808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.371+0000 7f951f35b700 1 --2- 172.21.15.16:0/110224321 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f951808c6d0 0x7f951808cac0 unknown :-1 s=CLOSED pgs=1543 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.371+0000 7f951f35b700 1 -- 172.21.15.16:0/110224321 shutdown_connections 2019-12-05T06:54:25.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.371+0000 7f951f35b700 1 -- 172.21.15.16:0/110224321 wait complete. 2019-12-05T06:54:25.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.371+0000 7f951f35b700 1 -- 172.21.15.16:0/110224321 >> 172.21.15.16:0/110224321 conn(0x7f9518061940 msgr2=0x7f9518096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:25.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.377+0000 7f951f35b700 1 Processor -- start 2019-12-05T06:54:25.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.377+0000 7f951f35b700 1 -- start start 2019-12-05T06:54:25.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.378+0000 7f951f35b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f951814d400 0x7f951814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:25.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.378+0000 7f951f35b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9518132530 0x7f9518157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:25.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.378+0000 7f951f35b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9518098da0 con 0x7f9518132d00 2019-12-05T06:54:25.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.378+0000 7f951f35b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951809bb50 con 0x7f9518132530 2019-12-05T06:54:25.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.378+0000 7f951f35b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f951809aa60 con 0x7f951814d400 2019-12-05T06:54:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.379+0000 7f951e0f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9518132530 0x7f9518157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.379+0000 7f951e0f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9518132530 0x7f9518157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35440/0 (socket says 172.21.15.16:35440) 2019-12-05T06:54:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.379+0000 7f951e0f9700 1 -- 172.21.15.16:0/105874895 learned_addr learned my addr 172.21.15.16:0/105874895 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.379+0000 7f951d8f8700 1 --2- 172.21.15.16:0/105874895 >> v2:172.21.15.76:3300/0 conn(0x7f951814d400 0x7f951814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.379+0000 7f951e0f9700 1 -- 172.21.15.16:0/105874895 >> v2:172.21.15.76:3300/0 conn(0x7f951814d400 msgr2=0x7f951814d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:25.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.379+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> v2:172.21.15.76:3300/0 conn(0x7f951814d400 0x7f951814d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:25.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.379+0000 7f951e0f9700 1 -- 172.21.15.16:0/105874895 >> v1:172.21.15.76:6789/0 conn(0x7f9518132d00 legacy=0x7f9518131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:25.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.379+0000 7f951e0f9700 1 -- 172.21.15.16:0/105874895 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f951809b0f0 con 0x7f9518132530 2019-12-05T06:54:25.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.379+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> v2:172.21.15.16:3300/0 conn(0x7f9518132530 0x7f9518157100 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.380+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> v2:172.21.15.16:3300/0 conn(0x7f9518132530 0x7f9518157100 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.380+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 0x7f9518157100 secure :-1 s=READY pgs=1544 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:25.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.380+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 0x7f9518157100 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.385 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.380+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 0x7f9518157100 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.380+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 0x7f9518157100 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.380+0000 7f95167fc700 1 -- 172.21.15.16:0/105874895 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9510000960 con 0x7f9518132530 2019-12-05T06:54:25.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.380+0000 7f95167fc700 1 -- 172.21.15.16:0/105874895 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f95100010e0 con 0x7f9518132530 2019-12-05T06:54:25.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.380+0000 7f951f35b700 1 -- 172.21.15.16:0/105874895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f951809b0f0 con 0x7f9518132530 2019-12-05T06:54:25.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.380+0000 7f95167fc700 1 -- 172.21.15.16:0/105874895 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f95100017f0 con 0x7f9518132530 2019-12-05T06:54:25.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.381+0000 7f951f35b700 1 -- 172.21.15.16:0/105874895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f951809b0f0 con 0x7f9518132530 2019-12-05T06:54:25.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.381+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 0x7f9518157100 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.382+0000 7f95167fc700 1 -- 172.21.15.16:0/105874895 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9510000960 con 0x7f9518132530 2019-12-05T06:54:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.383+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 0x7f9518157100 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.383+0000 7f95167fc700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94f8023630 0x7f94f8025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.383+0000 7f95167fc700 1 -- 172.21.15.16:0/105874895 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7f9510045e00 con 0x7f9518132530 2019-12-05T06:54:25.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.383+0000 7f951d8f8700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94f8023630 0x7f94f8025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:25.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.384+0000 7f951d8f8700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94f8023630 0x7f94f8025ac0 crc :-1 s=READY pgs=644 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:25.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.396+0000 7f951f35b700 1 -- 172.21.15.16:0/105874895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f951809b0f0 con 0x7f9518132530 2019-12-05T06:54:25.403 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.404+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 0x7f9518157100 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.404 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.404+0000 7f95167fc700 1 -- 172.21.15.16:0/105874895 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f95100075b0 con 0x7f9518132530 2019-12-05T06:54:25.655 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.655+0000 7f951f35b700 1 -- 172.21.15.16:0/105874895 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f951809b0f0 con 0x7f9518132530 2019-12-05T06:54:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.656+0000 7f951e0f9700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 0x7f9518157100 secure :-1 s=THROTTLE_DONE pgs=1544 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.656+0000 7f95167fc700 1 -- 172.21.15.16:0/105874895 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9510001990 con 0x7f9518132530 2019-12-05T06:54:25.656 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 187 2019-12-05T06:54:25.656 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:25.656 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:25.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.658+0000 7f951f35b700 1 -- 172.21.15.16:0/105874895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94f8023630 msgr2=0x7f94f8025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.658+0000 7f951f35b700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94f8023630 0x7f94f8025ac0 crc :-1 s=READY pgs=644 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.658+0000 7f951f35b700 1 -- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 msgr2=0x7f9518157100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.658+0000 7f951f35b700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 0x7f9518157100 secure :-1 s=READY pgs=1544 cs=0 l=1 rx=0x7f9510010410 tx=0x7f9510010270).stop 2019-12-05T06:54:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.658+0000 7f951f35b700 1 -- 172.21.15.16:0/105874895 shutdown_connections 2019-12-05T06:54:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.658+0000 7f951f35b700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f94f8023630 0x7f94f8025ac0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:25.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.658+0000 7f951f35b700 1 --2- 172.21.15.16:0/105874895 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9518132530 0x7f9518157100 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.658+0000 7f951f35b700 1 --2- 172.21.15.16:0/105874895 >> v2:172.21.15.76:3300/0 conn(0x7f951814d400 0x7f951814d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.658+0000 7f951f35b700 1 -- 172.21.15.16:0/105874895 shutdown_connections 2019-12-05T06:54:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.659+0000 7f951f35b700 1 -- 172.21.15.16:0/105874895 wait complete. 2019-12-05T06:54:25.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:25.659+0000 7f951f35b700 1 -- 172.21.15.16:0/105874895 >> 172.21.15.16:0/105874895 conn(0x7f95180714f0 msgr2=0x7f951809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:25.670 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:54:27.671 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.962+0000 7fcc49da4700 1 Processor -- start 2019-12-05T06:54:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.962+0000 7fcc49da4700 1 -- start start 2019-12-05T06:54:27.972 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.964+0000 7fcc49da4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fcc4408c6d0 0x7fcc4408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.964+0000 7fcc49da4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc4408f720 0x7fcc4408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.964+0000 7fcc49da4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4408fb80 con 0x7fcc4408fef0 2019-12-05T06:54:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.964+0000 7fcc49da4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4408df90 con 0x7fcc4408f720 2019-12-05T06:54:27.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.964+0000 7fcc49da4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc440a3f70 con 0x7fcc4408c6d0 2019-12-05T06:54:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.964+0000 7fcc48b42700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc4408f720 0x7fcc4408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.964+0000 7fcc437fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcc4408fef0 0x7fcc4408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48846/0 (socket says 172.21.15.16:48846) 2019-12-05T06:54:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.964+0000 7fcc437fe700 1 -- 172.21.15.16:0/2524726405 learned_addr learned my addr 172.21.15.16:0/2524726405 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:27.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.964+0000 7fcc43fff700 1 --2- 172.21.15.16:0/2524726405 >> v2:172.21.15.76:3300/0 conn(0x7fcc4408c6d0 0x7fcc4408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.965+0000 7fcc42ffd700 1 -- 172.21.15.16:0/2524726405 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3427107196 0 0) 0x7fcc30001150 con 0x7fcc4408fef0 2019-12-05T06:54:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.965+0000 7fcc42ffd700 1 -- 172.21.15.16:0/2524726405 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc28002e80 con 0x7fcc4408fef0 2019-12-05T06:54:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.965+0000 7fcc48b42700 1 -- 172.21.15.16:0/2524726405 >> v2:172.21.15.76:3300/0 conn(0x7fcc4408c6d0 msgr2=0x7fcc4408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:27.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.965+0000 7fcc48b42700 1 --2- 172.21.15.16:0/2524726405 >> v2:172.21.15.76:3300/0 conn(0x7fcc4408c6d0 0x7fcc4408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:27.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.965+0000 7fcc48b42700 1 -- 172.21.15.16:0/2524726405 >> v1:172.21.15.16:6789/0 conn(0x7fcc4408fef0 legacy=0x7fcc4408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:27.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.965+0000 7fcc48b42700 1 -- 172.21.15.16:0/2524726405 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc4408d540 con 0x7fcc4408f720 2019-12-05T06:54:27.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.965+0000 7fcc43fff700 1 --2- 172.21.15.16:0/2524726405 >> v2:172.21.15.76:3300/0 conn(0x7fcc4408c6d0 0x7fcc4408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:54:27.976 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.965+0000 7fcc48b42700 1 --2- 172.21.15.16:0/2524726405 >> v2:172.21.15.16:3300/0 conn(0x7fcc4408f720 0x7fcc4408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fcc3c00b2d0 tx=0x7fcc3c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.965+0000 7fcc48b42700 1 --2- 172.21.15.16:0/2524726405 >> v2:172.21.15.16:3300/0 conn(0x7fcc4408f720 0x7fcc4408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fcc3c00b2d0 tx=0x7fcc3c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.965+0000 7fcc48b42700 1 --2- 172.21.15.16:0/2524726405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4408f720 0x7fcc4408d000 secure :-1 s=READY pgs=1546 cs=0 l=1 rx=0x7fcc3c00b2d0 tx=0x7fcc3c00bae0).ready entity=mon.0 client_cookie=df3e7c6d01bf33fd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.966+0000 7fcc48b42700 1 --2- 172.21.15.16:0/2524726405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4408f720 0x7fcc4408d000 secure :-1 s=THROTTLE_DONE pgs=1546 cs=0 l=1 rx=0x7fcc3c00b2d0 tx=0x7fcc3c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.966+0000 7fcc48b42700 1 --2- 172.21.15.16:0/2524726405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4408f720 0x7fcc4408d000 secure :-1 s=THROTTLE_DONE pgs=1546 cs=0 l=1 rx=0x7fcc3c00b2d0 tx=0x7fcc3c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.977 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.966+0000 7fcc42ffd700 1 -- 172.21.15.16:0/2524726405 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc3c009f20 con 0x7fcc4408f720 2019-12-05T06:54:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.966+0000 7fcc48b42700 1 --2- 172.21.15.16:0/2524726405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4408f720 0x7fcc4408d000 secure :-1 s=THROTTLE_DONE pgs=1546 cs=0 l=1 rx=0x7fcc3c00b2d0 tx=0x7fcc3c00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.978 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.966+0000 7fcc42ffd700 1 -- 172.21.15.16:0/2524726405 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc3c009b40 con 0x7fcc4408f720 2019-12-05T06:54:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.966+0000 7fcc42ffd700 1 -- 172.21.15.16:0/2524726405 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc3c000f40 con 0x7fcc4408f720 2019-12-05T06:54:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.966+0000 7fcc49da4700 1 -- 172.21.15.16:0/2524726405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4408f720 msgr2=0x7fcc4408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.966+0000 7fcc49da4700 1 --2- 172.21.15.16:0/2524726405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4408f720 0x7fcc4408d000 secure :-1 s=READY pgs=1546 cs=0 l=1 rx=0x7fcc3c00b2d0 tx=0x7fcc3c00bae0).stop 2019-12-05T06:54:27.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.968+0000 7fcc49da4700 1 -- 172.21.15.16:0/2524726405 shutdown_connections 2019-12-05T06:54:27.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.968+0000 7fcc49da4700 1 --2- 172.21.15.16:0/2524726405 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc4408f720 0x7fcc4408d000 unknown :-1 s=CLOSED pgs=1546 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:27.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.968+0000 7fcc49da4700 1 --2- 172.21.15.16:0/2524726405 >> v2:172.21.15.76:3300/0 conn(0x7fcc4408c6d0 0x7fcc4408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:27.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.968+0000 7fcc49da4700 1 -- 172.21.15.16:0/2524726405 shutdown_connections 2019-12-05T06:54:27.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.968+0000 7fcc49da4700 1 -- 172.21.15.16:0/2524726405 wait complete. 2019-12-05T06:54:27.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.968+0000 7fcc49da4700 1 -- 172.21.15.16:0/2524726405 >> 172.21.15.16:0/2524726405 conn(0x7fcc44061940 msgr2=0x7fcc44096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.974+0000 7fcc49da4700 1 Processor -- start 2019-12-05T06:54:27.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.974+0000 7fcc49da4700 1 -- start start 2019-12-05T06:54:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.976+0000 7fcc49da4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc4406df10 0x7fcc44098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.976+0000 7fcc49da4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc440992d0 0x7fcc44098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.976+0000 7fcc49da4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc4406e300 con 0x7fcc44098d80 2019-12-05T06:54:27.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.976+0000 7fcc49da4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcc4408ad00 con 0x7fcc440992d0 2019-12-05T06:54:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.976+0000 7fcc49da4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcc44096c10 con 0x7fcc4406df10 2019-12-05T06:54:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.976+0000 7fcc437fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc440992d0 0x7fcc44098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.976+0000 7fcc437fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcc440992d0 0x7fcc44098840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35452/0 (socket says 172.21.15.16:35452) 2019-12-05T06:54:27.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.976+0000 7fcc437fe700 1 -- 172.21.15.16:0/270278946 learned_addr learned my addr 172.21.15.16:0/270278946 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:27.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.976+0000 7fcc43fff700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc4406df10 0x7fcc44098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:27.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.977+0000 7fcc437fe700 1 -- 172.21.15.16:0/270278946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc4406df10 msgr2=0x7fcc44098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:27.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.977+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc4406df10 0x7fcc44098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:27.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.977+0000 7fcc437fe700 1 -- 172.21.15.16:0/270278946 >> v1:172.21.15.76:6789/0 conn(0x7fcc44098d80 legacy=0x7fcc44088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:27.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.977+0000 7fcc437fe700 1 -- 172.21.15.16:0/270278946 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc44097a00 con 0x7fcc440992d0 2019-12-05T06:54:27.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.977+0000 7fcc40ff9700 1 -- 172.21.15.16:0/270278946 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3429040008 0 0) 0x7fcc3c000cc0 con 0x7fcc44098d80 2019-12-05T06:54:27.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.977+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> v2:172.21.15.16:3300/0 conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.977+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> v2:172.21.15.16:3300/0 conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.977+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=READY pgs=1547 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.977+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.977+0000 7fcc40ff9700 1 -- 172.21.15.16:0/270278946 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc30000c90 con 0x7fcc440992d0 2019-12-05T06:54:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.978+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.978+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.978+0000 7fcc40ff9700 1 -- 172.21.15.16:0/270278946 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcc300010c0 con 0x7fcc440992d0 2019-12-05T06:54:27.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.978+0000 7fcc49da4700 1 -- 172.21.15.16:0/270278946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc44097a00 con 0x7fcc440992d0 2019-12-05T06:54:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.978+0000 7fcc40ff9700 1 -- 172.21.15.16:0/270278946 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcc30001710 con 0x7fcc440992d0 2019-12-05T06:54:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.978+0000 7fcc49da4700 1 -- 172.21.15.16:0/270278946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc44097a00 con 0x7fcc440992d0 2019-12-05T06:54:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.979+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.980+0000 7fcc40ff9700 1 -- 172.21.15.16:0/270278946 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fcc30000c90 con 0x7fcc440992d0 2019-12-05T06:54:27.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.980+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:27.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.980+0000 7fcc40ff9700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc20023630 0x7fcc20025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:27.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.980+0000 7fcc40ff9700 1 -- 172.21.15.16:0/270278946 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fcc30045da0 con 0x7fcc440992d0 2019-12-05T06:54:27.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.981+0000 7fcc43fff700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc20023630 0x7fcc20025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:27.994 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.981+0000 7fcc43fff700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc20023630 0x7fcc20025ac0 crc :-1 s=READY pgs=645 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:27.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:27.993+0000 7fcc49da4700 1 -- 172.21.15.16:0/270278946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc44097a00 con 0x7fcc440992d0 2019-12-05T06:54:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.001+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:28.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.001+0000 7fcc40ff9700 1 -- 172.21.15.16:0/270278946 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fcc300075d0 con 0x7fcc440992d0 2019-12-05T06:54:28.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.248+0000 7fcc49da4700 1 -- 172.21.15.16:0/270278946 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc44097a00 con 0x7fcc440992d0 2019-12-05T06:54:28.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.249+0000 7fcc437fe700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=THROTTLE_DONE pgs=1547 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:28.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.249+0000 7fcc40ff9700 1 -- 172.21.15.16:0/270278946 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fcc30000c90 con 0x7fcc440992d0 2019-12-05T06:54:28.249 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:28.250 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:28.250 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 187 2019-12-05T06:54:28.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.251+0000 7fcc49da4700 1 -- 172.21.15.16:0/270278946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc20023630 msgr2=0x7fcc20025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:28.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.251+0000 7fcc49da4700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc20023630 0x7fcc20025ac0 crc :-1 s=READY pgs=645 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:28.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.251+0000 7fcc49da4700 1 -- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 msgr2=0x7fcc44098840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:28.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.251+0000 7fcc49da4700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 0x7fcc44098840 secure :-1 s=READY pgs=1547 cs=0 l=1 rx=0x7fcc30010360 tx=0x7fcc30010700).stop 2019-12-05T06:54:28.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.252+0000 7fcc49da4700 1 -- 172.21.15.16:0/270278946 shutdown_connections 2019-12-05T06:54:28.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.252+0000 7fcc49da4700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fcc20023630 0x7fcc20025ac0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:28.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.252+0000 7fcc49da4700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcc440992d0 0x7fcc44098840 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:28.253 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.252+0000 7fcc49da4700 1 --2- 172.21.15.16:0/270278946 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcc4406df10 0x7fcc44098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:28.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.252+0000 7fcc49da4700 1 -- 172.21.15.16:0/270278946 shutdown_connections 2019-12-05T06:54:28.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.252+0000 7fcc49da4700 1 -- 172.21.15.16:0/270278946 wait complete. 2019-12-05T06:54:28.254 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:28.252+0000 7fcc49da4700 1 -- 172.21.15.16:0/270278946 >> 172.21.15.16:0/270278946 conn(0x7fcc440714f0 msgr2=0x7fcc4409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:28.273 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:54:30.275 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:30.530 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.521+0000 7fdcd20e4700 1 Processor -- start 2019-12-05T06:54:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.521+0000 7fdcd20e4700 1 -- start start 2019-12-05T06:54:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.522+0000 7fdcd20e4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdccc08c6d0 0x7fdccc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.522+0000 7fdcd20e4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdccc08fb80 con 0x7fdccc08fef0 2019-12-05T06:54:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.523+0000 7fdcd20e4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdccc0958e0 con 0x7fdccc08f720 2019-12-05T06:54:30.531 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.523+0000 7fdcd20e4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdccc08df90 con 0x7fdccc08c6d0 2019-12-05T06:54:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.523+0000 7fdccb7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdccc08fef0 0x7fdccc08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48870/0 (socket says 172.21.15.16:48870) 2019-12-05T06:54:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.523+0000 7fdccb7fe700 1 -- 172.21.15.16:0/3400624175 learned_addr learned my addr 172.21.15.16:0/3400624175 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.523+0000 7fdccbfff700 1 --2- 172.21.15.16:0/3400624175 >> v2:172.21.15.76:3300/0 conn(0x7fdccc08c6d0 0x7fdccc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.523+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4098576051 0 0) 0x7fdcb4001170 con 0x7fdccc08fef0 2019-12-05T06:54:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.523+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcb8002e80 con 0x7fdccc08fef0 2019-12-05T06:54:30.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.523+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 539641798 0 0) 0x7fdcb4001130 con 0x7fdccc08fef0 2019-12-05T06:54:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 >> v2:172.21.15.76:3300/0 conn(0x7fdccc08c6d0 msgr2=0x7fdccc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 --2- 172.21.15.16:0/3400624175 >> v2:172.21.15.76:3300/0 conn(0x7fdccc08c6d0 0x7fdccc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 >> v1:172.21.15.76:6789/0 conn(0x7fdccc08f720 legacy=0x7fdccc1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc08e620 con 0x7fdccc08fef0 2019-12-05T06:54:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fdcb40016e0 con 0x7fdccc08fef0 2019-12-05T06:54:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 >> v1:172.21.15.16:6789/0 conn(0x7fdccc08fef0 legacy=0x7fdccc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:30.533 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 0x7fdcb8005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcb8006690 0x7fdcb8008b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdcb8009380 0x7fdcb800b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb800c000 con 0x7fdcb8003ab0 2019-12-05T06:54:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb8050200 con 0x7fdcb8006690 2019-12-05T06:54:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb8050430 con 0x7fdcb8009380 2019-12-05T06:54:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccbfff700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 0x7fdcb8005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:30.534 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcb8006690 0x7fdcb8008b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdcd0e82700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdcb8009380 0x7fdcb800b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccbfff700 1 -- 172.21.15.16:0/3400624175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdcb8009380 msgr2=0x7fdcb800b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccbfff700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdcb8009380 0x7fdcb800b850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccbfff700 1 -- 172.21.15.16:0/3400624175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcb8006690 msgr2=0x7fdcb8008b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccbfff700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdcb8006690 0x7fdcb8008b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdccbfff700 1 -- 172.21.15.16:0/3400624175 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb80506e0 con 0x7fdcb8003ab0 2019-12-05T06:54:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.524+0000 7fdcd0e82700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdcb8009380 0x7fdcb800b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:54:30.535 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdcd0e82700 1 -- 172.21.15.16:0/3400624175 reap_dead start 2019-12-05T06:54:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdccbfff700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 0x7fdcb8005f10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdcbc00ba60 tx=0x7fdcbc00b930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdccbfff700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 0x7fdcb8005f10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdcbc00ba60 tx=0x7fdcbc00b930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdccbfff700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 0x7fdcb8005f10 secure :-1 s=READY pgs=1549 cs=0 l=1 rx=0x7fdcbc00ba60 tx=0x7fdcbc00b930).ready entity=mon.0 client_cookie=caf11e318c296f77 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdccbfff700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 0x7fdcb8005f10 secure :-1 s=THROTTLE_DONE pgs=1549 cs=0 l=1 rx=0x7fdcbc00ba60 tx=0x7fdcbc00b930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.536 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcbc000ce0 con 0x7fdcb8003ab0 2019-12-05T06:54:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdccbfff700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 0x7fdcb8005f10 secure :-1 s=THROTTLE_DONE pgs=1549 cs=0 l=1 rx=0x7fdcbc00ba60 tx=0x7fdcbc00b930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdccbfff700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 0x7fdcb8005f10 secure :-1 s=THROTTLE_DONE pgs=1549 cs=0 l=1 rx=0x7fdcbc00ba60 tx=0x7fdcbc00b930).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcbc009af0 con 0x7fdcb8003ab0 2019-12-05T06:54:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdccaffd700 1 -- 172.21.15.16:0/3400624175 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcbc000ce0 con 0x7fdcb8003ab0 2019-12-05T06:54:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdcd20e4700 1 -- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 msgr2=0x7fdcb8005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.525+0000 7fdcd20e4700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 0x7fdcb8005f10 secure :-1 s=READY pgs=1549 cs=0 l=1 rx=0x7fdcbc00ba60 tx=0x7fdcbc00b930).stop 2019-12-05T06:54:30.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.526+0000 7fdcd20e4700 1 -- 172.21.15.16:0/3400624175 shutdown_connections 2019-12-05T06:54:30.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.526+0000 7fdcd20e4700 1 --2- 172.21.15.16:0/3400624175 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fdcb8003ab0 0x7fdcb8005f10 unknown :-1 s=CLOSED pgs=1549 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:30.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.527+0000 7fdcd20e4700 1 -- 172.21.15.16:0/3400624175 shutdown_connections 2019-12-05T06:54:30.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.527+0000 7fdcd20e4700 1 -- 172.21.15.16:0/3400624175 wait complete. 2019-12-05T06:54:30.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.527+0000 7fdcd20e4700 1 -- 172.21.15.16:0/3400624175 >> 172.21.15.16:0/3400624175 conn(0x7fdccc061940 msgr2=0x7fdccc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:30.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.531+0000 7fdcd20e4700 1 Processor -- start 2019-12-05T06:54:30.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.532+0000 7fdcd20e4700 1 -- start start 2019-12-05T06:54:30.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.533+0000 7fdcd20e4700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdccc1325e0 0x7fdccc12b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:30.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.533+0000 7fdcd20e4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:30.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.533+0000 7fdcd20e4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdccc128b20 con 0x7fdccc12c1f0 2019-12-05T06:54:30.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.533+0000 7fdcd20e4700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdccc12a7e0 con 0x7fdccc1325e0 2019-12-05T06:54:30.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.533+0000 7fdcd20e4700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdccc129db0 con 0x7fdccc12c740 2019-12-05T06:54:30.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.533+0000 7fdccb7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:30.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.534+0000 7fdccbfff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdccc1325e0 0x7fdccc12b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:30.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.534+0000 7fdccb7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:39472/0 (socket says 172.21.15.16:39472) 2019-12-05T06:54:30.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.534+0000 7fdccb7fe700 1 -- 172.21.15.16:0/2321460384 learned_addr learned my addr 172.21.15.16:0/2321460384 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:30.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.534+0000 7fdccb7fe700 1 -- 172.21.15.16:0/2321460384 >> v2:172.21.15.76:3300/0 conn(0x7fdccc1325e0 msgr2=0x7fdccc12b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:30.543 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.534+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> v2:172.21.15.76:3300/0 conn(0x7fdccc1325e0 0x7fdccc12b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:30.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.534+0000 7fdccb7fe700 1 -- 172.21.15.16:0/2321460384 >> v1:172.21.15.76:6789/0 conn(0x7fdccc12c1f0 legacy=0x7fdccc06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:30.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.534+0000 7fdccb7fe700 1 -- 172.21.15.16:0/2321460384 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdccc14a820 con 0x7fdccc12c740 2019-12-05T06:54:30.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.534+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.544 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.534+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.534+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=READY pgs=366 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.535+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=THROTTLE_DONE pgs=366 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.535+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=THROTTLE_DONE pgs=366 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.545 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.535+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=THROTTLE_DONE pgs=366 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.535+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/2321460384 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb4001ba0 con 0x7fdccc12c740 2019-12-05T06:54:30.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.535+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/2321460384 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb4004430 con 0x7fdccc12c740 2019-12-05T06:54:30.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.535+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/2321460384 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb4004a00 con 0x7fdccc12c740 2019-12-05T06:54:30.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.535+0000 7fdcd20e4700 1 -- 172.21.15.16:0/2321460384 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdccc12ae70 con 0x7fdccc12c740 2019-12-05T06:54:30.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.535+0000 7fdcd20e4700 1 -- 172.21.15.16:0/2321460384 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdccc14a820 con 0x7fdccc12c740 2019-12-05T06:54:30.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.536+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=THROTTLE_DONE pgs=366 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.536+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=THROTTLE_DONE pgs=366 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.536+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/2321460384 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdcb4001ba0 con 0x7fdccc12c740 2019-12-05T06:54:30.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.537+0000 7fdcc8ff9700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdcb0023630 0x7fdcb0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:30.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.537+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/2321460384 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fdcb4045da0 con 0x7fdccc12c740 2019-12-05T06:54:30.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.537+0000 7fdccbfff700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdcb0023630 0x7fdcb0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:30.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.538+0000 7fdccbfff700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdcb0023630 0x7fdcb0025ac0 crc :-1 s=READY pgs=646 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:30.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.552+0000 7fdcd20e4700 1 -- 172.21.15.16:0/2321460384 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdccc12ae70 con 0x7fdccc12c740 2019-12-05T06:54:30.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.561+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=THROTTLE_DONE pgs=366 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.561+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/2321460384 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdcb4001990 con 0x7fdccc12c740 2019-12-05T06:54:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.770+0000 7fdcd20e4700 1 -- 172.21.15.16:0/2321460384 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdccc14a820 con 0x7fdccc12c740 2019-12-05T06:54:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.771+0000 7fdccb7fe700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=THROTTLE_DONE pgs=366 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:30.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.771+0000 7fdcc8ff9700 1 -- 172.21.15.16:0/2321460384 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fdcb4001ba0 con 0x7fdccc12c740 2019-12-05T06:54:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 187 2019-12-05T06:54:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.775+0000 7fdcd20e4700 1 -- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdcb0023630 msgr2=0x7fdcb0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.775+0000 7fdcd20e4700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdcb0023630 0x7fdcb0025ac0 crc :-1 s=READY pgs=646 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.775+0000 7fdcd20e4700 1 -- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 msgr2=0x7fdccc12bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.775+0000 7fdcd20e4700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 secure :-1 s=READY pgs=366 cs=0 l=1 rx=0x7fdcb4010430 tx=0x7fdcb4010260).stop 2019-12-05T06:54:30.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.775+0000 7fdcd20e4700 1 -- 172.21.15.16:0/2321460384 shutdown_connections 2019-12-05T06:54:30.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.775+0000 7fdcd20e4700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdcb0023630 0x7fdcb0025ac0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:30.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.775+0000 7fdcd20e4700 1 --2- 172.21.15.16:0/2321460384 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdccc12c740 0x7fdccc12bcb0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:30.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.775+0000 7fdcd20e4700 1 --2- 172.21.15.16:0/2321460384 >> v2:172.21.15.76:3300/0 conn(0x7fdccc1325e0 0x7fdccc12b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:30.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.776+0000 7fdcd20e4700 1 -- 172.21.15.16:0/2321460384 shutdown_connections 2019-12-05T06:54:30.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.776+0000 7fdcd20e4700 1 -- 172.21.15.16:0/2321460384 wait complete. 2019-12-05T06:54:30.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:30.776+0000 7fdcd20e4700 1 -- 172.21.15.16:0/2321460384 >> 172.21.15.16:0/2321460384 conn(0x7fdccc0a2850 msgr2=0x7fdccc06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:30.790 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:30.790 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:30.791 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:54:32.792 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:33.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.078+0000 7fe95bc5c700 1 Processor -- start 2019-12-05T06:54:33.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.078+0000 7fe95bc5c700 1 -- start start 2019-12-05T06:54:33.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.080+0000 7fe95bc5c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe95408c6d0 0x7fe95408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:33.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.080+0000 7fe95bc5c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe95408fef0 0x7fe95408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.081+0000 7fe95bc5c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe95408fb80 con 0x7fe95408ef20 2019-12-05T06:54:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.081+0000 7fe95bc5c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95408df90 con 0x7fe95408c6d0 2019-12-05T06:54:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.081+0000 7fe95bc5c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9540a3f70 con 0x7fe95408fef0 2019-12-05T06:54:33.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.081+0000 7fe95a1f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe95408c6d0 0x7fe95408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.081+0000 7fe95a1f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe95408c6d0 0x7fe95408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35490/0 (socket says 172.21.15.16:35490) 2019-12-05T06:54:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.081+0000 7fe95a1f9700 1 -- 172.21.15.16:0/3826640297 learned_addr learned my addr 172.21.15.16:0/3826640297 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.081+0000 7fe9599f8700 1 --2- 172.21.15.16:0/3826640297 >> v2:172.21.15.76:3300/0 conn(0x7fe95408fef0 0x7fe95408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe95a1f9700 1 -- 172.21.15.16:0/3826640297 >> v2:172.21.15.76:3300/0 conn(0x7fe95408fef0 msgr2=0x7fe95408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:33.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe95a1f9700 1 --2- 172.21.15.16:0/3826640297 >> v2:172.21.15.76:3300/0 conn(0x7fe95408fef0 0x7fe95408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe95a1f9700 1 -- 172.21.15.16:0/3826640297 >> v1:172.21.15.76:6789/0 conn(0x7fe95408ef20 legacy=0x7fe95408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe95a1f9700 1 -- 172.21.15.16:0/3826640297 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95408d540 con 0x7fe95408c6d0 2019-12-05T06:54:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe9599f8700 1 --2- 172.21.15.16:0/3826640297 >> v2:172.21.15.76:3300/0 conn(0x7fe95408fef0 0x7fe95408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:54:33.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe95a1f9700 1 --2- 172.21.15.16:0/3826640297 >> v2:172.21.15.16:3300/0 conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe94c00fac0 tx=0x7fe94c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe95a1f9700 1 --2- 172.21.15.16:0/3826640297 >> v2:172.21.15.16:3300/0 conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe94c00fac0 tx=0x7fe94c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.094 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe95a1f9700 1 --2- 172.21.15.16:0/3826640297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=READY pgs=1550 cs=0 l=1 rx=0x7fe94c00fac0 tx=0x7fe94c00fe60).ready entity=mon.0 client_cookie=47b56c951102f239 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe95a1f9700 1 --2- 172.21.15.16:0/3826640297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=THROTTLE_DONE pgs=1550 cs=0 l=1 rx=0x7fe94c00fac0 tx=0x7fe94c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe95a1f9700 1 --2- 172.21.15.16:0/3826640297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=THROTTLE_DONE pgs=1550 cs=0 l=1 rx=0x7fe94c00fac0 tx=0x7fe94c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe9591f7700 1 -- 172.21.15.16:0/3826640297 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe94c000ce0 con 0x7fe95408c6d0 2019-12-05T06:54:33.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.082+0000 7fe95a1f9700 1 --2- 172.21.15.16:0/3826640297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=THROTTLE_DONE pgs=1550 cs=0 l=1 rx=0x7fe94c00fac0 tx=0x7fe94c00fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.083+0000 7fe9591f7700 1 -- 172.21.15.16:0/3826640297 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94c001010 con 0x7fe95408c6d0 2019-12-05T06:54:33.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.083+0000 7fe9591f7700 1 -- 172.21.15.16:0/3826640297 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe94c001720 con 0x7fe95408c6d0 2019-12-05T06:54:33.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.083+0000 7fe95bc5c700 1 -- 172.21.15.16:0/3826640297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95408c6d0 msgr2=0x7fe95408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:33.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.083+0000 7fe95bc5c700 1 --2- 172.21.15.16:0/3826640297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 secure :-1 s=READY pgs=1550 cs=0 l=1 rx=0x7fe94c00fac0 tx=0x7fe94c00fe60).stop 2019-12-05T06:54:33.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.084+0000 7fe95bc5c700 1 -- 172.21.15.16:0/3826640297 shutdown_connections 2019-12-05T06:54:33.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.084+0000 7fe95bc5c700 1 --2- 172.21.15.16:0/3826640297 >> v2:172.21.15.76:3300/0 conn(0x7fe95408fef0 0x7fe95408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:33.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.084+0000 7fe95bc5c700 1 --2- 172.21.15.16:0/3826640297 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe95408c6d0 0x7fe95408cac0 unknown :-1 s=CLOSED pgs=1550 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:33.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.085+0000 7fe95bc5c700 1 -- 172.21.15.16:0/3826640297 shutdown_connections 2019-12-05T06:54:33.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.085+0000 7fe95bc5c700 1 -- 172.21.15.16:0/3826640297 wait complete. 2019-12-05T06:54:33.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.085+0000 7fe95bc5c700 1 -- 172.21.15.16:0/3826640297 >> 172.21.15.16:0/3826640297 conn(0x7fe954061940 msgr2=0x7fe954096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:33.106 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.090+0000 7fe95bc5c700 1 Processor -- start 2019-12-05T06:54:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.090+0000 7fe95bc5c700 1 -- start start 2019-12-05T06:54:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe95bc5c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe954096e90 0x7fe95409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe95bc5c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fe954084f30 0x7fe95409b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:33.107 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe95bc5c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe954097280 0x7fe9540976c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe95bc5c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe95409a450 con 0x7fe954096e90 2019-12-05T06:54:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe95bc5c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe954099a20 con 0x7fe954084f30 2019-12-05T06:54:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe95bc5c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe954099cf0 con 0x7fe954097280 2019-12-05T06:54:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe95a1f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe954096e90 0x7fe95409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:33.108 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe95a1f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe954096e90 0x7fe95409b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35496/0 (socket says 172.21.15.16:35496) 2019-12-05T06:54:33.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe95a1f9700 1 -- 172.21.15.16:0/1109736559 learned_addr learned my addr 172.21.15.16:0/1109736559 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:33.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> v2:172.21.15.76:3300/0 conn(0x7fe954084f30 0x7fe95409b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:33.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.092+0000 7fe95a9fa700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe954097280 0x7fe9540976c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:33.109 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.093+0000 7fe9599f8700 1 -- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe954097280 msgr2=0x7fe9540976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.093+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe954097280 0x7fe9540976c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.093+0000 7fe9599f8700 1 -- 172.21.15.16:0/1109736559 >> v2:172.21.15.16:3300/0 conn(0x7fe954096e90 msgr2=0x7fe95409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.093+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> v2:172.21.15.16:3300/0 conn(0x7fe954096e90 0x7fe95409b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.093+0000 7fe9599f8700 1 -- 172.21.15.16:0/1109736559 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe95409aae0 con 0x7fe954084f30 2019-12-05T06:54:33.110 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.093+0000 7fe95a1f9700 1 --2- 172.21.15.16:0/1109736559 >> v2:172.21.15.16:3300/0 conn(0x7fe954096e90 0x7fe95409b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:54:33.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.093+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> v2:172.21.15.76:3300/0 conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.093+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> v2:172.21.15.76:3300/0 conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.093+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=READY pgs=1041 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:33.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.094+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.094+0000 7fe952ffd700 1 -- 172.21.15.16:0/1109736559 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe948000940 con 0x7fe954084f30 2019-12-05T06:54:33.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.094+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.094+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.094+0000 7fe952ffd700 1 -- 172.21.15.16:0/1109736559 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe94800de60 con 0x7fe954084f30 2019-12-05T06:54:33.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.094+0000 7fe95bc5c700 1 -- 172.21.15.16:0/1109736559 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe95409aae0 con 0x7fe954084f30 2019-12-05T06:54:33.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.094+0000 7fe952ffd700 1 -- 172.21.15.16:0/1109736559 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe94800e590 con 0x7fe954084f30 2019-12-05T06:54:33.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.094+0000 7fe95bc5c700 1 -- 172.21.15.16:0/1109736559 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe95409aae0 con 0x7fe954084f30 2019-12-05T06:54:33.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.095+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.096+0000 7fe952ffd700 1 -- 172.21.15.16:0/1109736559 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe94800da60 con 0x7fe954084f30 2019-12-05T06:54:33.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.096+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.096+0000 7fe952ffd700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe934023630 0x7fe934025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:33.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.097+0000 7fe952ffd700 1 -- 172.21.15.16:0/1109736559 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 8697+0+0 (secure 0 0 0) 0x7fe948045c70 con 0x7fe954084f30 2019-12-05T06:54:33.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.097+0000 7fe95a1f9700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe934023630 0x7fe934025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:33.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.097+0000 7fe95a1f9700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe934023630 0x7fe934025ac0 crc :-1 s=READY pgs=647 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:33.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.110+0000 7fe95bc5c700 1 -- 172.21.15.16:0/1109736559 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe95409aae0 con 0x7fe954084f30 2019-12-05T06:54:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.119+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.119+0000 7fe952ffd700 1 -- 172.21.15.16:0/1109736559 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe94800e360 con 0x7fe954084f30 2019-12-05T06:54:33.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.380+0000 7fe95bc5c700 1 -- 172.21.15.16:0/1109736559 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe95409aae0 con 0x7fe954084f30 2019-12-05T06:54:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.382+0000 7fe9599f8700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=THROTTLE_DONE pgs=1041 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:33.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.382+0000 7fe952ffd700 1 -- 172.21.15.16:0/1109736559 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 187 v187) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe94800da10 con 0x7fe954084f30 2019-12-05T06:54:33.384 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 187 2019-12-05T06:54:33.384 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:33.385 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":187,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8106,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":187}],"filesystems":[{"mdsmap":{"epoch":186,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:53:48.726207+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":80,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6092},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6092":{"gid":6092,"name":"b","rank":0,"incarnation":183,"state":"up:active","state_seq":10,"addr":"172.21.15.76:6835/2551129089","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2551129089},{"type":"v1","addr":"172.21.15.76:6835","nonce":2551129089}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:33.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 -- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe934023630 msgr2=0x7fe934025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:33.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe934023630 0x7fe934025ac0 crc :-1 s=READY pgs=647 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:33.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 -- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 msgr2=0x7fe95409b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:33.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 0x7fe95409b920 secure :-1 s=READY pgs=1041 cs=0 l=1 rx=0x7fe9480103a0 tx=0x7fe9480101d0).stop 2019-12-05T06:54:33.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 -- 172.21.15.16:0/1109736559 shutdown_connections 2019-12-05T06:54:33.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe954097280 0x7fe9540976c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:33.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fe954084f30 0x7fe95409b920 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:33.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 --2- 172.21.15.16:0/1109736559 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe934023630 0x7fe934025ac0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:33.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 --2- 172.21.15.16:0/1109736559 >> v2:172.21.15.16:3300/0 conn(0x7fe954096e90 0x7fe95409b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:33.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 -- 172.21.15.16:0/1109736559 shutdown_connections 2019-12-05T06:54:33.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 -- 172.21.15.16:0/1109736559 wait complete. 2019-12-05T06:54:33.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:33.386+0000 7fe95bc5c700 1 -- 172.21.15.16:0/1109736559 >> 172.21.15.16:0/1109736559 conn(0x7fe9540714f0 msgr2=0x7fe95409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:33.398 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:54:35.399 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:35.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.543+0000 7fb162749700 1 Processor -- start 2019-12-05T06:54:35.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.543+0000 7fb162749700 1 -- start start 2019-12-05T06:54:35.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.544+0000 7fb162749700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb15c08c6d0 0x7fb15c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:35.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.544+0000 7fb162749700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 0x7fb15c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:35.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.544+0000 7fb162749700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15c0958e0 con 0x7fb15c08ef20 2019-12-05T06:54:35.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.544+0000 7fb162749700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15c08df90 con 0x7fb15c08c6d0 2019-12-05T06:54:35.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.544+0000 7fb162749700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c0a3f70 con 0x7fb15c08fef0 2019-12-05T06:54:35.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.544+0000 7fb160ce6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb15c08c6d0 0x7fb15c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:35.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.544+0000 7fb15bfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 0x7fb15c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:35.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.544+0000 7fb160ce6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb15c08c6d0 0x7fb15c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:49842/0 (socket says 172.21.15.16:49842) 2019-12-05T06:54:35.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.544+0000 7fb1614e7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fb15c08ef20 0x7fb15c08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:41994/0 (socket says 172.21.15.16:41994) 2019-12-05T06:54:35.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.544+0000 7fb160ce6700 1 -- 172.21.15.16:0/1105207735 learned_addr learned my addr 172.21.15.16:0/1105207735 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:35.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15bfff700 1 -- 172.21.15.16:0/1105207735 >> v2:172.21.15.76:3300/0 conn(0x7fb15c08c6d0 msgr2=0x7fb15c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:35.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15bfff700 1 --2- 172.21.15.16:0/1105207735 >> v2:172.21.15.76:3300/0 conn(0x7fb15c08c6d0 0x7fb15c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:35.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15bfff700 1 -- 172.21.15.16:0/1105207735 >> v1:172.21.15.76:6789/0 conn(0x7fb15c08ef20 legacy=0x7fb15c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15bfff700 1 -- 172.21.15.16:0/1105207735 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c08d540 con 0x7fb15c08fef0 2019-12-05T06:54:35.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15bfff700 1 --2- 172.21.15.16:0/1105207735 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 0x7fb15c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb14400af50 tx=0x7fb14400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15bfff700 1 --2- 172.21.15.16:0/1105207735 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 0x7fb15c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb14400af50 tx=0x7fb14400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15bfff700 1 --2- 172.21.15.16:0/1105207735 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 0x7fb15c08d000 secure :-1 s=READY pgs=367 cs=0 l=1 rx=0x7fb14400af50 tx=0x7fb14400b250).ready entity=mon.2 client_cookie=78dcdfe4766c69c5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:35.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15bfff700 1 --2- 172.21.15.16:0/1105207735 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 0x7fb15c08d000 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7fb14400af50 tx=0x7fb14400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15b7fe700 1 -- 172.21.15.16:0/1105207735 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb144007210 con 0x7fb15c08fef0 2019-12-05T06:54:35.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15bfff700 1 --2- 172.21.15.16:0/1105207735 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 0x7fb15c08d000 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7fb14400af50 tx=0x7fb14400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15b7fe700 1 -- 172.21.15.16:0/1105207735 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1440097c0 con 0x7fb15c08fef0 2019-12-05T06:54:35.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15bfff700 1 --2- 172.21.15.16:0/1105207735 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 0x7fb15c08d000 secure :-1 s=THROTTLE_DONE pgs=367 cs=0 l=1 rx=0x7fb14400af50 tx=0x7fb14400b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb15b7fe700 1 -- 172.21.15.16:0/1105207735 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb1440099b0 con 0x7fb15c08fef0 2019-12-05T06:54:35.552 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb162749700 1 -- 172.21.15.16:0/1105207735 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 msgr2=0x7fb15c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.545+0000 7fb162749700 1 --2- 172.21.15.16:0/1105207735 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 0x7fb15c08d000 secure :-1 s=READY pgs=367 cs=0 l=1 rx=0x7fb14400af50 tx=0x7fb14400b250).stop 2019-12-05T06:54:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.546+0000 7fb162749700 1 -- 172.21.15.16:0/1105207735 shutdown_connections 2019-12-05T06:54:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.546+0000 7fb162749700 1 --2- 172.21.15.16:0/1105207735 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c08fef0 0x7fb15c08d000 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.546+0000 7fb162749700 1 --2- 172.21.15.16:0/1105207735 >> v2:172.21.15.76:3300/0 conn(0x7fb15c08c6d0 0x7fb15c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.546+0000 7fb162749700 1 -- 172.21.15.16:0/1105207735 shutdown_connections 2019-12-05T06:54:35.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.546+0000 7fb162749700 1 -- 172.21.15.16:0/1105207735 wait complete. 2019-12-05T06:54:35.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.546+0000 7fb162749700 1 -- 172.21.15.16:0/1105207735 >> 172.21.15.16:0/1105207735 conn(0x7fb15c061940 msgr2=0x7fb15c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:35.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.549+0000 7fb162749700 1 Processor -- start 2019-12-05T06:54:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.549+0000 7fb162749700 1 -- start start 2019-12-05T06:54:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb162749700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c12b640 0x7fb15c12ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb162749700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb15c153b50 0x7fb15c155f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb162749700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb15c131bf0 con 0x7fb15c1302f0 2019-12-05T06:54:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb162749700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb15c1296e0 con 0x7fb15c153b50 2019-12-05T06:54:35.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb162749700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb15c128cb0 con 0x7fb15c12b640 2019-12-05T06:54:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb15c153b50 0x7fb15c155f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb15c153b50 0x7fb15c155f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35530/0 (socket says 172.21.15.16:35530) 2019-12-05T06:54:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 -- 172.21.15.16:0/254784750 learned_addr learned my addr 172.21.15.16:0/254784750 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb15bfff700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c12b640 0x7fb15c12ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 -- 172.21.15.16:0/254784750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c12b640 msgr2=0x7fb15c12ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c12b640 0x7fb15c12ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:35.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 -- 172.21.15.16:0/254784750 >> v1:172.21.15.76:6789/0 conn(0x7fb15c1302f0 legacy=0x7fb15c12a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 -- 172.21.15.16:0/254784750 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb15c129d70 con 0x7fb15c153b50 2019-12-05T06:54:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> v2:172.21.15.16:3300/0 conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> v2:172.21.15.16:3300/0 conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=READY pgs=1551 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:35.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.550+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.551+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.551+0000 7fb1597fa700 1 -- 172.21.15.16:0/254784750 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb154000ce0 con 0x7fb15c153b50 2019-12-05T06:54:35.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.551+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.551+0000 7fb1597fa700 1 -- 172.21.15.16:0/254784750 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb154004540 con 0x7fb15c153b50 2019-12-05T06:54:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.551+0000 7fb1597fa700 1 -- 172.21.15.16:0/254784750 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb154004c10 con 0x7fb15c153b50 2019-12-05T06:54:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.551+0000 7fb162749700 1 -- 172.21.15.16:0/254784750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb15c129d70 con 0x7fb15c153b50 2019-12-05T06:54:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.551+0000 7fb162749700 1 -- 172.21.15.16:0/254784750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb15c129d70 con 0x7fb15c153b50 2019-12-05T06:54:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.551+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.562 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.552+0000 7fb1597fa700 1 -- 172.21.15.16:0/254784750 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb154004730 con 0x7fb15c153b50 2019-12-05T06:54:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.552+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.552+0000 7fb1597fa700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb140023630 0x7fb140025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.552+0000 7fb1597fa700 1 -- 172.21.15.16:0/254784750 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7fb154046190 con 0x7fb15c153b50 2019-12-05T06:54:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.552+0000 7fb160ce6700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb140023630 0x7fb140025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:35.563 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.552+0000 7fb160ce6700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb140023630 0x7fb140025ac0 crc :-1 s=READY pgs=648 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.557+0000 7fb162749700 1 -- 172.21.15.16:0/254784750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb15c129d70 con 0x7fb15c153b50 2019-12-05T06:54:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.561+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.561+0000 7fb1597fa700 1 -- 172.21.15.16:0/254784750 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb154001670 con 0x7fb15c153b50 2019-12-05T06:54:35.752 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.751+0000 7fb162749700 1 -- 172.21.15.16:0/254784750 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb15c129d70 con 0x7fb15c153b50 2019-12-05T06:54:35.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.752+0000 7fb1614e7700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=THROTTLE_DONE pgs=1551 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.752+0000 7fb1597fa700 1 -- 172.21.15.16:0/254784750 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fb154004730 con 0x7fb15c153b50 2019-12-05T06:54:35.753 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 188 2019-12-05T06:54:35.753 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:35.754 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:33.602224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:35.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.754+0000 7fb162749700 1 -- 172.21.15.16:0/254784750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb140023630 msgr2=0x7fb140025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.754+0000 7fb162749700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb140023630 0x7fb140025ac0 crc :-1 s=READY pgs=648 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.754+0000 7fb162749700 1 -- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 msgr2=0x7fb15c155f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.754+0000 7fb162749700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 0x7fb15c155f50 secure :-1 s=READY pgs=1551 cs=0 l=1 rx=0x7fb154010790 tx=0x7fb154010610).stop 2019-12-05T06:54:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.755+0000 7fb162749700 1 -- 172.21.15.16:0/254784750 shutdown_connections 2019-12-05T06:54:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.755+0000 7fb162749700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb140023630 0x7fb140025ac0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:35.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.755+0000 7fb162749700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb15c153b50 0x7fb15c155f50 unknown :-1 s=CLOSED pgs=1551 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.755+0000 7fb162749700 1 --2- 172.21.15.16:0/254784750 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb15c12b640 0x7fb15c12ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.755+0000 7fb162749700 1 -- 172.21.15.16:0/254784750 shutdown_connections 2019-12-05T06:54:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.755+0000 7fb162749700 1 -- 172.21.15.16:0/254784750 wait complete. 2019-12-05T06:54:35.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.755+0000 7fb162749700 1 -- 172.21.15.16:0/254784750 >> 172.21.15.16:0/254784750 conn(0x7fb15c0a2850 msgr2=0x7fb15c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:35.766 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:54:35.766 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:54:35.766 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.905+0000 7fde9c753700 1 Processor -- start 2019-12-05T06:54:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.905+0000 7fde9c753700 1 -- start start 2019-12-05T06:54:35.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.906+0000 7fde9c753700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde9408fef0 0x7fde9408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.906+0000 7fde9c753700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde940958e0 con 0x7fde9408d2c0 2019-12-05T06:54:35.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.906+0000 7fde9c753700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde940629e0 con 0x7fde9408c6d0 2019-12-05T06:54:35.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.906+0000 7fde9c753700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9408df90 con 0x7fde9408fef0 2019-12-05T06:54:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.906+0000 7fde9a4ef700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fde9408d2c0 0x7fde9408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48936/0 (socket says 172.21.15.16:48936) 2019-12-05T06:54:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.906+0000 7fde9a4ef700 1 -- 172.21.15.16:0/3176125487 learned_addr learned my addr 172.21.15.16:0/3176125487 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.906+0000 7fde9b4f1700 1 --2- 172.21.15.16:0/3176125487 >> v2:172.21.15.16:3300/0 conn(0x7fde9408fef0 0x7fde9408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.906+0000 7fde99cee700 1 -- 172.21.15.16:0/3176125487 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 755556232 0 0) 0x7fde84001170 con 0x7fde9408d2c0 2019-12-05T06:54:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.906+0000 7fde99cee700 1 -- 172.21.15.16:0/3176125487 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde7c002e80 con 0x7fde9408d2c0 2019-12-05T06:54:35.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9b4f1700 1 -- 172.21.15.16:0/3176125487 >> v1:172.21.15.76:6789/0 conn(0x7fde9408c6d0 legacy=0x7fde9408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9b4f1700 1 -- 172.21.15.16:0/3176125487 >> v1:172.21.15.16:6789/0 conn(0x7fde9408d2c0 legacy=0x7fde9408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9b4f1700 1 -- 172.21.15.16:0/3176125487 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9408e620 con 0x7fde9408fef0 2019-12-05T06:54:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9b4f1700 1 --2- 172.21.15.16:0/3176125487 >> v2:172.21.15.16:3300/0 conn(0x7fde9408fef0 0x7fde9408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fde8800af50 tx=0x7fde8800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9b4f1700 1 --2- 172.21.15.16:0/3176125487 >> v2:172.21.15.16:3300/0 conn(0x7fde9408fef0 0x7fde9408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fde8800af50 tx=0x7fde8800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9b4f1700 1 --2- 172.21.15.16:0/3176125487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9408fef0 0x7fde9408f720 secure :-1 s=READY pgs=1553 cs=0 l=1 rx=0x7fde8800af50 tx=0x7fde8800b760).ready entity=mon.0 client_cookie=5b92454551bbfb1e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9b4f1700 1 --2- 172.21.15.16:0/3176125487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9408fef0 0x7fde9408f720 secure :-1 s=THROTTLE_DONE pgs=1553 cs=0 l=1 rx=0x7fde8800af50 tx=0x7fde8800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde99cee700 1 -- 172.21.15.16:0/3176125487 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde88009ba0 con 0x7fde9408fef0 2019-12-05T06:54:35.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9b4f1700 1 --2- 172.21.15.16:0/3176125487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9408fef0 0x7fde9408f720 secure :-1 s=THROTTLE_DONE pgs=1553 cs=0 l=1 rx=0x7fde8800af50 tx=0x7fde8800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9b4f1700 1 --2- 172.21.15.16:0/3176125487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9408fef0 0x7fde9408f720 secure :-1 s=THROTTLE_DONE pgs=1553 cs=0 l=1 rx=0x7fde8800af50 tx=0x7fde8800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde99cee700 1 -- 172.21.15.16:0/3176125487 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde88000b10 con 0x7fde9408fef0 2019-12-05T06:54:35.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde99cee700 1 -- 172.21.15.16:0/3176125487 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde88009ba0 con 0x7fde9408fef0 2019-12-05T06:54:35.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9c753700 1 -- 172.21.15.16:0/3176125487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9408fef0 msgr2=0x7fde9408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.907+0000 7fde9c753700 1 --2- 172.21.15.16:0/3176125487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9408fef0 0x7fde9408f720 secure :-1 s=READY pgs=1553 cs=0 l=1 rx=0x7fde8800af50 tx=0x7fde8800b760).stop 2019-12-05T06:54:35.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.909+0000 7fde9c753700 1 -- 172.21.15.16:0/3176125487 shutdown_connections 2019-12-05T06:54:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.909+0000 7fde9c753700 1 --2- 172.21.15.16:0/3176125487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9408fef0 0x7fde9408f720 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.909+0000 7fde9c753700 1 -- 172.21.15.16:0/3176125487 shutdown_connections 2019-12-05T06:54:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.909+0000 7fde9c753700 1 -- 172.21.15.16:0/3176125487 wait complete. 2019-12-05T06:54:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.909+0000 7fde9c753700 1 -- 172.21.15.16:0/3176125487 >> 172.21.15.16:0/3176125487 conn(0x7fde94061940 msgr2=0x7fde94096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:35.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.913+0000 7fde9c753700 1 Processor -- start 2019-12-05T06:54:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.913+0000 7fde9c753700 1 -- start start 2019-12-05T06:54:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9c753700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde9406ded0 0x7fde94129e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9c753700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fde94156cd0 0x7fde941590d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:35.948 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9c753700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde9412d8f0 con 0x7fde9412ae50 2019-12-05T06:54:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9c753700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde94089b80 con 0x7fde9406ded0 2019-12-05T06:54:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9c753700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde94089e50 con 0x7fde94156cd0 2019-12-05T06:54:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9acf0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde9406ded0 0x7fde94129e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9acf0700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde9406ded0 0x7fde94129e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35540/0 (socket says 172.21.15.16:35540) 2019-12-05T06:54:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9acf0700 1 -- 172.21.15.16:0/2916319973 learned_addr learned my addr 172.21.15.16:0/2916319973 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9b4f1700 1 --2- 172.21.15.16:0/2916319973 >> v2:172.21.15.76:3300/0 conn(0x7fde94156cd0 0x7fde941590d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9acf0700 1 -- 172.21.15.16:0/2916319973 >> v2:172.21.15.76:3300/0 conn(0x7fde94156cd0 msgr2=0x7fde941590d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.949 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> v2:172.21.15.76:3300/0 conn(0x7fde94156cd0 0x7fde941590d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9acf0700 1 -- 172.21.15.16:0/2916319973 >> v1:172.21.15.16:6789/0 conn(0x7fde9412ae50 legacy=0x7fde9412a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde937fe700 1 -- 172.21.15.16:0/2916319973 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 375765399 0 0) 0x7fde84001150 con 0x7fde9412ae50 2019-12-05T06:54:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9acf0700 1 -- 172.21.15.16:0/2916319973 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde94129580 con 0x7fde9406ded0 2019-12-05T06:54:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.914+0000 7fde9b4f1700 1 --2- 172.21.15.16:0/2916319973 >> v2:172.21.15.76:3300/0 conn(0x7fde94156cd0 0x7fde941590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:54:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> v2:172.21.15.16:3300/0 conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> v2:172.21.15.16:3300/0 conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.950 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=READY pgs=1555 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde937fe700 1 -- 172.21.15.16:0/2916319973 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde8c005b40 con 0x7fde9406ded0 2019-12-05T06:54:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde937fe700 1 -- 172.21.15.16:0/2916319973 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde8c005760 con 0x7fde9406ded0 2019-12-05T06:54:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde9c753700 1 -- 172.21.15.16:0/2916319973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde94129580 con 0x7fde9406ded0 2019-12-05T06:54:35.951 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde937fe700 1 -- 172.21.15.16:0/2916319973 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde8c005d30 con 0x7fde9406ded0 2019-12-05T06:54:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.915+0000 7fde9c753700 1 -- 172.21.15.16:0/2916319973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde94129580 con 0x7fde9406ded0 2019-12-05T06:54:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.916+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.916+0000 7fde937fe700 1 -- 172.21.15.16:0/2916319973 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fde8c005760 con 0x7fde9406ded0 2019-12-05T06:54:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.916+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.917+0000 7fde937fe700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde74023630 0x7fde74025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.917+0000 7fde937fe700 1 -- 172.21.15.16:0/2916319973 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7fde8c0461a0 con 0x7fde9406ded0 2019-12-05T06:54:35.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.917+0000 7fde9a4ef700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde74023630 0x7fde74025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:35.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.918+0000 7fde9a4ef700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde74023630 0x7fde74025ac0 crc :-1 s=READY pgs=649 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:35.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.920+0000 7fde9c753700 1 -- 172.21.15.16:0/2916319973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde94129580 con 0x7fde9406ded0 2019-12-05T06:54:35.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.925+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:35.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:35.925+0000 7fde937fe700 1 -- 172.21.15.16:0/2916319973 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fde8c007980 con 0x7fde9406ded0 2019-12-05T06:54:36.111 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.111+0000 7fde9c753700 1 -- 172.21.15.16:0/2916319973 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde94129580 con 0x7fde9406ded0 2019-12-05T06:54:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.111+0000 7fde9acf0700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=THROTTLE_DONE pgs=1555 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.112+0000 7fde937fe700 1 -- 172.21.15.16:0/2916319973 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fde8c005760 con 0x7fde9406ded0 2019-12-05T06:54:36.113 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 188 2019-12-05T06:54:36.113 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:36.113 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":188,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:33.602224+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:36.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 -- 172.21.15.16:0/2916319973 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde74023630 msgr2=0x7fde74025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:36.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde74023630 0x7fde74025ac0 crc :-1 s=READY pgs=649 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:36.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 -- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 msgr2=0x7fde94129e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 0x7fde94129e80 secure :-1 s=READY pgs=1555 cs=0 l=1 rx=0x7fde8c010790 tx=0x7fde8c010b30).stop 2019-12-05T06:54:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 -- 172.21.15.16:0/2916319973 shutdown_connections 2019-12-05T06:54:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 --2- 172.21.15.16:0/2916319973 >> v2:172.21.15.76:3300/0 conn(0x7fde94156cd0 0x7fde941590d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde74023630 0x7fde74025ac0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 --2- 172.21.15.16:0/2916319973 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9406ded0 0x7fde94129e80 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:36.116 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 -- 172.21.15.16:0/2916319973 shutdown_connections 2019-12-05T06:54:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 -- 172.21.15.16:0/2916319973 wait complete. 2019-12-05T06:54:36.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:36.114+0000 7fde9c753700 1 -- 172.21.15.16:0/2916319973 >> 172.21.15.16:0/2916319973 conn(0x7fde94071440 msgr2=0x7fde94096e00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:36.126 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 188, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 188, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8106": { "addr": "172.21.15.16:6833/4247952173", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 4247952173, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 4247952173, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8106, "incarnation": 188, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 81, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:54:33.602224+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8106 } } } ], "standbys": [] } 2019-12-05T06:54:38.126 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:38.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.405+0000 7f5d88570700 1 Processor -- start 2019-12-05T06:54:38.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.405+0000 7f5d88570700 1 -- start start 2019-12-05T06:54:38.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.407+0000 7f5d88570700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5d8008c6d0 0x7f5d8008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:38.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.407+0000 7f5d88570700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8008fb80 con 0x7f5d8008f720 2019-12-05T06:54:38.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.407+0000 7f5d88570700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d800958e0 con 0x7f5d8008fef0 2019-12-05T06:54:38.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.407+0000 7f5d88570700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d8008df90 con 0x7f5d8008c6d0 2019-12-05T06:54:38.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.408+0000 7f5d8730e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5d8008f720 0x7f5d801533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48958/0 (socket says 172.21.15.16:48958) 2019-12-05T06:54:38.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.408+0000 7f5d8730e700 1 -- 172.21.15.16:0/2558257641 learned_addr learned my addr 172.21.15.16:0/2558257641 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:38.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.408+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/2558257641 >> v2:172.21.15.76:3300/0 conn(0x7f5d8008c6d0 0x7f5d8008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:38.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.408+0000 7f5d85b0b700 1 -- 172.21.15.16:0/2558257641 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772023281 0 0) 0x7f5d70001150 con 0x7f5d8008f720 2019-12-05T06:54:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.408+0000 7f5d85b0b700 1 -- 172.21.15.16:0/2558257641 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d68002e80 con 0x7f5d8008f720 2019-12-05T06:54:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d86b0d700 1 -- 172.21.15.16:0/2558257641 >> v1:172.21.15.76:6789/0 conn(0x7f5d8008fef0 legacy=0x7f5d8008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d85b0b700 1 -- 172.21.15.16:0/2558257641 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 868912343 0 0) 0x7f5d74001080 con 0x7f5d8008fef0 2019-12-05T06:54:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d86b0d700 1 -- 172.21.15.16:0/2558257641 >> v1:172.21.15.16:6789/0 conn(0x7f5d8008f720 legacy=0x7f5d801533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d86b0d700 1 -- 172.21.15.16:0/2558257641 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8008e620 con 0x7f5d8008c6d0 2019-12-05T06:54:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/2558257641 >> v2:172.21.15.76:3300/0 conn(0x7f5d8008c6d0 0x7f5d8008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f5d7800b2d0 tx=0x7f5d7800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/2558257641 >> v2:172.21.15.76:3300/0 conn(0x7f5d8008c6d0 0x7f5d8008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f5d7800b2d0 tx=0x7f5d7800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/2558257641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8008c6d0 0x7f5d8008cac0 secure :-1 s=READY pgs=1046 cs=0 l=1 rx=0x7f5d7800b2d0 tx=0x7f5d7800bae0).ready entity=mon.1 client_cookie=d418833e5a42559d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:38.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/2558257641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8008c6d0 0x7f5d8008cac0 secure :-1 s=THROTTLE_DONE pgs=1046 cs=0 l=1 rx=0x7f5d7800b2d0 tx=0x7f5d7800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d85b0b700 1 -- 172.21.15.16:0/2558257641 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d78000ce0 con 0x7f5d8008c6d0 2019-12-05T06:54:38.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/2558257641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8008c6d0 0x7f5d8008cac0 secure :-1 s=THROTTLE_DONE pgs=1046 cs=0 l=1 rx=0x7f5d7800b2d0 tx=0x7f5d7800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/2558257641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8008c6d0 0x7f5d8008cac0 secure :-1 s=THROTTLE_DONE pgs=1046 cs=0 l=1 rx=0x7f5d7800b2d0 tx=0x7f5d7800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d85b0b700 1 -- 172.21.15.16:0/2558257641 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78009be0 con 0x7f5d8008c6d0 2019-12-05T06:54:38.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.409+0000 7f5d85b0b700 1 -- 172.21.15.16:0/2558257641 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d7800a2b0 con 0x7f5d8008c6d0 2019-12-05T06:54:38.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.410+0000 7f5d88570700 1 -- 172.21.15.16:0/2558257641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8008c6d0 msgr2=0x7f5d8008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:38.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.410+0000 7f5d88570700 1 --2- 172.21.15.16:0/2558257641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8008c6d0 0x7f5d8008cac0 secure :-1 s=READY pgs=1046 cs=0 l=1 rx=0x7f5d7800b2d0 tx=0x7f5d7800bae0).stop 2019-12-05T06:54:38.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.412+0000 7f5d88570700 1 -- 172.21.15.16:0/2558257641 shutdown_connections 2019-12-05T06:54:38.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.412+0000 7f5d88570700 1 --2- 172.21.15.16:0/2558257641 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8008c6d0 0x7f5d8008cac0 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:38.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.412+0000 7f5d88570700 1 -- 172.21.15.16:0/2558257641 shutdown_connections 2019-12-05T06:54:38.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.412+0000 7f5d88570700 1 -- 172.21.15.16:0/2558257641 wait complete. 2019-12-05T06:54:38.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.412+0000 7f5d88570700 1 -- 172.21.15.16:0/2558257641 >> 172.21.15.16:0/2558257641 conn(0x7f5d80061940 msgr2=0x7f5d80096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:38.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.417+0000 7f5d88570700 1 Processor -- start 2019-12-05T06:54:38.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.417+0000 7f5d88570700 1 -- start start 2019-12-05T06:54:38.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.418+0000 7f5d88570700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f5d8014d5d0 0x7f5d8014d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:38.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.418+0000 7f5d88570700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d8006e250 con 0x7f5d8009e2c0 2019-12-05T06:54:38.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.418+0000 7f5d88570700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d80099460 con 0x7f5d8009dba0 2019-12-05T06:54:38.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.418+0000 7f5d88570700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d8012b3f0 con 0x7f5d8014d5d0 2019-12-05T06:54:38.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.418+0000 7f5d8630c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f5d8009e2c0 0x7f5d8009d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48962/0 (socket says 172.21.15.16:48962) 2019-12-05T06:54:38.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.418+0000 7f5d8630c700 1 -- 172.21.15.16:0/1614114892 learned_addr learned my addr 172.21.15.16:0/1614114892 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:38.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.419+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> v2:172.21.15.76:3300/0 conn(0x7f5d8014d5d0 0x7f5d8014d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:38.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.419+0000 7f5d7f7fe700 1 -- 172.21.15.16:0/1614114892 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845460100 0 0) 0x7f5d740012c0 con 0x7f5d8009e2c0 2019-12-05T06:54:38.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.419+0000 7f5d7f7fe700 1 -- 172.21.15.16:0/1614114892 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d60002e80 con 0x7f5d8009e2c0 2019-12-05T06:54:38.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.419+0000 7f5d86b0d700 1 -- 172.21.15.16:0/1614114892 >> v1:172.21.15.76:6789/0 conn(0x7f5d8009dba0 legacy=0x7f5d80153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:38.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.419+0000 7f5d86b0d700 1 -- 172.21.15.16:0/1614114892 >> v1:172.21.15.16:6789/0 conn(0x7f5d8009e2c0 legacy=0x7f5d8009d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:38.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.419+0000 7f5d86b0d700 1 -- 172.21.15.16:0/1614114892 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8012ba80 con 0x7f5d8014d5d0 2019-12-05T06:54:38.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.419+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> v2:172.21.15.76:3300/0 conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.419+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> v2:172.21.15.76:3300/0 conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.420+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:38.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.420+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=THROTTLE_DONE pgs=1048 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.420+0000 7f5d7f7fe700 1 -- 172.21.15.16:0/1614114892 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d78000cc0 con 0x7f5d8014d5d0 2019-12-05T06:54:38.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.420+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=THROTTLE_DONE pgs=1048 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.420+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=THROTTLE_DONE pgs=1048 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.420+0000 7f5d7f7fe700 1 -- 172.21.15.16:0/1614114892 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78009af0 con 0x7f5d8014d5d0 2019-12-05T06:54:38.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.420+0000 7f5d7f7fe700 1 -- 172.21.15.16:0/1614114892 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d78000cc0 con 0x7f5d8014d5d0 2019-12-05T06:54:38.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.420+0000 7f5d88570700 1 -- 172.21.15.16:0/1614114892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8012ba80 con 0x7f5d8014d5d0 2019-12-05T06:54:38.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.420+0000 7f5d88570700 1 -- 172.21.15.16:0/1614114892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8006e250 con 0x7f5d8014d5d0 2019-12-05T06:54:38.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.421+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=THROTTLE_DONE pgs=1048 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.422+0000 7f5d7f7fe700 1 -- 172.21.15.16:0/1614114892 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5d78009900 con 0x7f5d8014d5d0 2019-12-05T06:54:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.422+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=THROTTLE_DONE pgs=1048 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.422+0000 7f5d7f7fe700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5d60023670 0x7f5d60025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.422+0000 7f5d7f7fe700 1 -- 172.21.15.16:0/1614114892 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f5d78046140 con 0x7f5d8014d5d0 2019-12-05T06:54:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.422+0000 7f5d8630c700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5d60023670 0x7f5d60025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:38.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.423+0000 7f5d8630c700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5d60023670 0x7f5d60025ae0 crc :-1 s=READY pgs=650 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:38.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.437+0000 7f5d88570700 1 -- 172.21.15.16:0/1614114892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d8012ba80 con 0x7f5d8014d5d0 2019-12-05T06:54:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.447+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=THROTTLE_DONE pgs=1048 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.448 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.447+0000 7f5d7f7fe700 1 -- 172.21.15.16:0/1614114892 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f5d7800b8c0 con 0x7f5d8014d5d0 2019-12-05T06:54:38.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.704+0000 7f5d88570700 1 -- 172.21.15.16:0/1614114892 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5d8012ba80 con 0x7f5d8014d5d0 2019-12-05T06:54:38.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.705+0000 7f5d86b0d700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=THROTTLE_DONE pgs=1048 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:38.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.705+0000 7f5d7f7fe700 1 -- 172.21.15.16:0/1614114892 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 190 v190) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f5d78009900 con 0x7f5d8014d5d0 2019-12-05T06:54:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 190 2019-12-05T06:54:38.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.711+0000 7f5d88570700 1 -- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5d60023670 msgr2=0x7f5d60025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.711+0000 7f5d88570700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5d60023670 0x7f5d60025ae0 crc :-1 s=READY pgs=650 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.711+0000 7f5d88570700 1 -- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 msgr2=0x7f5d8014d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.711+0000 7f5d88570700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 0x7f5d8014d9c0 secure :-1 s=READY pgs=1048 cs=0 l=1 rx=0x7f5d78010700 tx=0x7f5d78010aa0).stop 2019-12-05T06:54:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.711+0000 7f5d88570700 1 -- 172.21.15.16:0/1614114892 shutdown_connections 2019-12-05T06:54:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.711+0000 7f5d88570700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f5d60023670 0x7f5d60025ae0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:38.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.712+0000 7f5d88570700 1 --2- 172.21.15.16:0/1614114892 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5d8014d5d0 0x7f5d8014d9c0 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:38.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.712+0000 7f5d88570700 1 -- 172.21.15.16:0/1614114892 shutdown_connections 2019-12-05T06:54:38.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.712+0000 7f5d88570700 1 -- 172.21.15.16:0/1614114892 wait complete. 2019-12-05T06:54:38.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:38.712+0000 7f5d88570700 1 -- 172.21.15.16:0/1614114892 >> 172.21.15.16:0/1614114892 conn(0x7f5d8006d260 msgr2=0x7f5d8009aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:38.715 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:38.715 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":190,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":190,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:37.695784+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:rejoin","state_seq":13,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:38.732 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:54:40.733 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:40.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.920+0000 7fd6e1f9d700 1 Processor -- start 2019-12-05T06:54:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.920+0000 7fd6e1f9d700 1 -- start start 2019-12-05T06:54:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.922+0000 7fd6e1f9d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fd6dc08c6d0 0x7fd6dc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.922+0000 7fd6e1f9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6dc08fef0 0x7fd6dc08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.922+0000 7fd6e1f9d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc08fb80 con 0x7fd6dc08ef20 2019-12-05T06:54:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.922+0000 7fd6e1f9d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6dc08df90 con 0x7fd6dc08fef0 2019-12-05T06:54:40.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.922+0000 7fd6e1f9d700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6dc0a3f70 con 0x7fd6dc08c6d0 2019-12-05T06:54:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.922+0000 7fd6db7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6dc08fef0 0x7fd6dc08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.922+0000 7fd6db7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd6dc08fef0 0x7fd6dc08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35584/0 (socket says 172.21.15.16:35584) 2019-12-05T06:54:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.922+0000 7fd6db7fe700 1 -- 172.21.15.16:0/1941350968 learned_addr learned my addr 172.21.15.16:0/1941350968 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.922+0000 7fd6dbfff700 1 --2- 172.21.15.16:0/1941350968 >> v2:172.21.15.76:3300/0 conn(0x7fd6dc08c6d0 0x7fd6dc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.923+0000 7fd6db7fe700 1 -- 172.21.15.16:0/1941350968 >> v2:172.21.15.76:3300/0 conn(0x7fd6dc08c6d0 msgr2=0x7fd6dc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.923+0000 7fd6db7fe700 1 --2- 172.21.15.16:0/1941350968 >> v2:172.21.15.76:3300/0 conn(0x7fd6dc08c6d0 0x7fd6dc08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:40.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.923+0000 7fd6db7fe700 1 -- 172.21.15.16:0/1941350968 >> v1:172.21.15.76:6789/0 conn(0x7fd6dc08ef20 legacy=0x7fd6dc08f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.923+0000 7fd6db7fe700 1 -- 172.21.15.16:0/1941350968 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc08d540 con 0x7fd6dc08fef0 2019-12-05T06:54:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.923+0000 7fd6dbfff700 1 --2- 172.21.15.16:0/1941350968 >> v2:172.21.15.76:3300/0 conn(0x7fd6dc08c6d0 0x7fd6dc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:54:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.923+0000 7fd6db7fe700 1 --2- 172.21.15.16:0/1941350968 >> v2:172.21.15.16:3300/0 conn(0x7fd6dc08fef0 0x7fd6dc08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd6c400f740 tx=0x7fd6c400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.923+0000 7fd6db7fe700 1 --2- 172.21.15.16:0/1941350968 >> v2:172.21.15.16:3300/0 conn(0x7fd6dc08fef0 0x7fd6dc08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd6c400f740 tx=0x7fd6c400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.923+0000 7fd6db7fe700 1 --2- 172.21.15.16:0/1941350968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6dc08fef0 0x7fd6dc08d000 secure :-1 s=READY pgs=1560 cs=0 l=1 rx=0x7fd6c400f740 tx=0x7fd6c400fae0).ready entity=mon.0 client_cookie=e6cca8bbf184b939 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.924+0000 7fd6db7fe700 1 --2- 172.21.15.16:0/1941350968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6dc08fef0 0x7fd6dc08d000 secure :-1 s=THROTTLE_DONE pgs=1560 cs=0 l=1 rx=0x7fd6c400f740 tx=0x7fd6c400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.924+0000 7fd6db7fe700 1 --2- 172.21.15.16:0/1941350968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6dc08fef0 0x7fd6dc08d000 secure :-1 s=THROTTLE_DONE pgs=1560 cs=0 l=1 rx=0x7fd6c400f740 tx=0x7fd6c400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.924+0000 7fd6db7fe700 1 --2- 172.21.15.16:0/1941350968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6dc08fef0 0x7fd6dc08d000 secure :-1 s=THROTTLE_DONE pgs=1560 cs=0 l=1 rx=0x7fd6c400f740 tx=0x7fd6c400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.924+0000 7fd6daffd700 1 -- 172.21.15.16:0/1941350968 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6c4000960 con 0x7fd6dc08fef0 2019-12-05T06:54:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.924+0000 7fd6daffd700 1 -- 172.21.15.16:0/1941350968 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6c4000c90 con 0x7fd6dc08fef0 2019-12-05T06:54:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.924+0000 7fd6daffd700 1 -- 172.21.15.16:0/1941350968 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6c40013a0 con 0x7fd6dc08fef0 2019-12-05T06:54:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.924+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/1941350968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6dc08fef0 msgr2=0x7fd6dc08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.924+0000 7fd6e1f9d700 1 --2- 172.21.15.16:0/1941350968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6dc08fef0 0x7fd6dc08d000 secure :-1 s=READY pgs=1560 cs=0 l=1 rx=0x7fd6c400f740 tx=0x7fd6c400fae0).stop 2019-12-05T06:54:40.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.925+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/1941350968 shutdown_connections 2019-12-05T06:54:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.925+0000 7fd6e1f9d700 1 --2- 172.21.15.16:0/1941350968 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd6dc08fef0 0x7fd6dc08d000 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.925+0000 7fd6e1f9d700 1 --2- 172.21.15.16:0/1941350968 >> v2:172.21.15.76:3300/0 conn(0x7fd6dc08c6d0 0x7fd6dc08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.926+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/1941350968 shutdown_connections 2019-12-05T06:54:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.926+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/1941350968 wait complete. 2019-12-05T06:54:40.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.926+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/1941350968 >> 172.21.15.16:0/1941350968 conn(0x7fd6dc061940 msgr2=0x7fd6dc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.930+0000 7fd6e1f9d700 1 Processor -- start 2019-12-05T06:54:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.931+0000 7fd6e1f9d700 1 -- start start 2019-12-05T06:54:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.932+0000 7fd6e1f9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.932+0000 7fd6e1f9d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc085c60 con 0x7fd6dc0826b0 2019-12-05T06:54:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.932+0000 7fd6e1f9d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6dc09c680 con 0x7fd6dc085ff0 2019-12-05T06:54:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.932+0000 7fd6e1f9d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6dc09e340 con 0x7fd6dc085820 2019-12-05T06:54:40.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.932+0000 7fd6dbfff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd6dc0826b0 0x7fd6dc085020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:48988/0 (socket says 172.21.15.16:48988) 2019-12-05T06:54:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.932+0000 7fd6dbfff700 1 -- 172.21.15.16:0/2390635539 learned_addr learned my addr 172.21.15.16:0/2390635539 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.932+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.933+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3813892461 0 0) 0x7fd6cc00e110 con 0x7fd6dc0826b0 2019-12-05T06:54:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.933+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c0002e80 con 0x7fd6dc0826b0 2019-12-05T06:54:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.933+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066295527 0 0) 0x7fd6c4007900 con 0x7fd6dc085ff0 2019-12-05T06:54:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.933+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd6c00030b0 con 0x7fd6dc085ff0 2019-12-05T06:54:40.939 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.933+0000 7fd6e0d3b700 1 -- 172.21.15.16:0/2390635539 >> v1:172.21.15.76:6789/0 conn(0x7fd6dc085ff0 legacy=0x7fd6dc19f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.933+0000 7fd6e0d3b700 1 -- 172.21.15.16:0/2390635539 >> v1:172.21.15.16:6789/0 conn(0x7fd6dc0826b0 legacy=0x7fd6dc085020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.933+0000 7fd6e0d3b700 1 -- 172.21.15.16:0/2390635539 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6dc09e9d0 con 0x7fd6dc085820 2019-12-05T06:54:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.933+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.934+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.934+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=READY pgs=368 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.934+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.940 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.934+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.934+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6d4000960 con 0x7fd6dc085820 2019-12-05T06:54:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.934+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.934+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6d4009820 con 0x7fd6dc085820 2019-12-05T06:54:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.934+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/2390635539 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6dc09e9d0 con 0x7fd6dc085820 2019-12-05T06:54:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.934+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6d4009f70 con 0x7fd6dc085820 2019-12-05T06:54:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.934+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/2390635539 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6dc09e9d0 con 0x7fd6dc085820 2019-12-05T06:54:40.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.935+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.936+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.936+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd6d4009a10 con 0x7fd6dc085820 2019-12-05T06:54:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.936+0000 7fd6d8ff9700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6c0023650 0x7fd6c0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.936+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7fd6d4045dc0 con 0x7fd6dc085820 2019-12-05T06:54:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.936+0000 7fd6dbfff700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6c0023650 0x7fd6c0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:40.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.937+0000 7fd6dbfff700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6c0023650 0x7fd6c0025ae0 crc :-1 s=READY pgs=651 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:40.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.951+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/2390635539 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6dc09e9d0 con 0x7fd6dc085820 2019-12-05T06:54:40.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.961+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:40.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:40.961+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd6d4007220 con 0x7fd6dc085820 2019-12-05T06:54:41.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.181+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/2390635539 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd6dc09e9d0 con 0x7fd6dc085820 2019-12-05T06:54:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.182+0000 7fd6e0d3b700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=THROTTLE_DONE pgs=368 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:41.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.182+0000 7fd6d8ff9700 1 -- 172.21.15.16:0/2390635539 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fd6d40099f0 con 0x7fd6dc085820 2019-12-05T06:54:41.186 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 191 2019-12-05T06:54:41.187 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:41.187 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:38.705960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:active","state_seq":14,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:41.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.186+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6c0023650 msgr2=0x7fd6c0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:41.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.186+0000 7fd6e1f9d700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6c0023650 0x7fd6c0025ae0 crc :-1 s=READY pgs=651 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:41.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.186+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 msgr2=0x7fd6dc1a2c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:41.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.186+0000 7fd6e1f9d700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 secure :-1 s=READY pgs=368 cs=0 l=1 rx=0x7fd6d40103b0 tx=0x7fd6d4010750).stop 2019-12-05T06:54:41.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.187+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/2390635539 shutdown_connections 2019-12-05T06:54:41.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.187+0000 7fd6e1f9d700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd6c0023650 0x7fd6c0025ae0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:41.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.187+0000 7fd6e1f9d700 1 --2- 172.21.15.16:0/2390635539 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd6dc085820 0x7fd6dc1a2c60 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:41.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.187+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/2390635539 shutdown_connections 2019-12-05T06:54:41.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.187+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/2390635539 wait complete. 2019-12-05T06:54:41.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:41.187+0000 7fd6e1f9d700 1 -- 172.21.15.16:0/2390635539 >> 172.21.15.16:0/2390635539 conn(0x7fd6dc098480 msgr2=0x7fd6dc071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:41.198 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6092 2019-12-05T06:54:41.199 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 2 secs before reviving mds.b 2019-12-05T06:54:43.199 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:54:43.200 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:54:43.200 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:54:43.213 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:54:43.214 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:43.378 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.373+0000 7f80f6d68700 1 Processor -- start 2019-12-05T06:54:43.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.373+0000 7f80f6d68700 1 -- start start 2019-12-05T06:54:43.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.374+0000 7f80f6d68700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f008c6d0 0x7f80f008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:43.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.374+0000 7f80f6d68700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80f008f720 0x7f80f008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:43.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.374+0000 7f80f6d68700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f00958e0 con 0x7f80f008fef0 2019-12-05T06:54:43.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.374+0000 7f80f6d68700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f00a3f70 con 0x7f80f008f720 2019-12-05T06:54:43.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.374+0000 7f80f6d68700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f008d830 con 0x7f80f008c6d0 2019-12-05T06:54:43.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.374+0000 7f80f5b06700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80f008f720 0x7f80f008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:43.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.374+0000 7f80f5b06700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f80f008f720 0x7f80f008d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35606/0 (socket says 172.21.15.16:35606) 2019-12-05T06:54:43.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.374+0000 7f80f5b06700 1 -- 172.21.15.16:0/949177746 learned_addr learned my addr 172.21.15.16:0/949177746 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:43.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.374+0000 7f80f5305700 1 --2- 172.21.15.16:0/949177746 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f008c6d0 0x7f80f008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:43.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5b06700 1 -- 172.21.15.16:0/949177746 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f008c6d0 msgr2=0x7f80f008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:43.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5b06700 1 --2- 172.21.15.16:0/949177746 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f008c6d0 0x7f80f008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5b06700 1 -- 172.21.15.16:0/949177746 >> v1:172.21.15.76:6789/0 conn(0x7f80f008fef0 legacy=0x7f80f008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5b06700 1 -- 172.21.15.16:0/949177746 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f008fbc0 con 0x7f80f008f720 2019-12-05T06:54:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5305700 1 --2- 172.21.15.16:0/949177746 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f008c6d0 0x7f80f008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:54:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5b06700 1 --2- 172.21.15.16:0/949177746 >> v2:172.21.15.16:3300/0 conn(0x7f80f008f720 0x7f80f008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f80e800fab0 tx=0x7f80e800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5b06700 1 --2- 172.21.15.16:0/949177746 >> v2:172.21.15.16:3300/0 conn(0x7f80f008f720 0x7f80f008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f80e800fab0 tx=0x7f80e800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5b06700 1 --2- 172.21.15.16:0/949177746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f008f720 0x7f80f008d000 secure :-1 s=READY pgs=1562 cs=0 l=1 rx=0x7f80e800fab0 tx=0x7f80e800fe50).ready entity=mon.0 client_cookie=78bac11a76ce253e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:43.381 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5b06700 1 --2- 172.21.15.16:0/949177746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f008f720 0x7f80f008d000 secure :-1 s=THROTTLE_DONE pgs=1562 cs=0 l=1 rx=0x7f80e800fab0 tx=0x7f80e800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80effff700 1 -- 172.21.15.16:0/949177746 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e8000f10 con 0x7f80f008f720 2019-12-05T06:54:43.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5b06700 1 --2- 172.21.15.16:0/949177746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f008f720 0x7f80f008d000 secure :-1 s=THROTTLE_DONE pgs=1562 cs=0 l=1 rx=0x7f80e800fab0 tx=0x7f80e800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f5b06700 1 --2- 172.21.15.16:0/949177746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f008f720 0x7f80f008d000 secure :-1 s=THROTTLE_DONE pgs=1562 cs=0 l=1 rx=0x7f80e800fab0 tx=0x7f80e800fe50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80effff700 1 -- 172.21.15.16:0/949177746 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e8001460 con 0x7f80f008f720 2019-12-05T06:54:43.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80effff700 1 -- 172.21.15.16:0/949177746 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e8001bb0 con 0x7f80f008f720 2019-12-05T06:54:43.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f6d68700 1 -- 172.21.15.16:0/949177746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f008f720 msgr2=0x7f80f008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:43.382 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.375+0000 7f80f6d68700 1 --2- 172.21.15.16:0/949177746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f008f720 0x7f80f008d000 secure :-1 s=READY pgs=1562 cs=0 l=1 rx=0x7f80e800fab0 tx=0x7f80e800fe50).stop 2019-12-05T06:54:43.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.376+0000 7f80f6d68700 1 -- 172.21.15.16:0/949177746 shutdown_connections 2019-12-05T06:54:43.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.376+0000 7f80f6d68700 1 --2- 172.21.15.16:0/949177746 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f80f008f720 0x7f80f008d000 unknown :-1 s=CLOSED pgs=1562 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:43.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.376+0000 7f80f6d68700 1 --2- 172.21.15.16:0/949177746 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f008c6d0 0x7f80f008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:43.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.376+0000 7f80f6d68700 1 -- 172.21.15.16:0/949177746 shutdown_connections 2019-12-05T06:54:43.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.376+0000 7f80f6d68700 1 -- 172.21.15.16:0/949177746 wait complete. 2019-12-05T06:54:43.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.376+0000 7f80f6d68700 1 -- 172.21.15.16:0/949177746 >> 172.21.15.16:0/949177746 conn(0x7f80f0061940 msgr2=0x7f80f0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:43.386 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.378+0000 7f80f6d68700 1 Processor -- start 2019-12-05T06:54:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.378+0000 7f80f6d68700 1 -- start start 2019-12-05T06:54:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.379+0000 7f80f6d68700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f80f014ca10 0x7f80f00869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.379+0000 7f80f6d68700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f00879b0 0x7f80f0086f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.379+0000 7f80f6d68700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80f009e030 con 0x7f80f0087460 2019-12-05T06:54:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.379+0000 7f80f6d68700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80f0085a50 con 0x7f80f014ca10 2019-12-05T06:54:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.379+0000 7f80f6d68700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f80f0085020 con 0x7f80f00879b0 2019-12-05T06:54:43.387 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.379+0000 7f80f5b06700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f80f0087460 0x7f80f0152160 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49014/0 (socket says 172.21.15.16:49014) 2019-12-05T06:54:43.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.379+0000 7f80f5b06700 1 -- 172.21.15.16:0/2738099360 learned_addr learned my addr 172.21.15.16:0/2738099360 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:43.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.379+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> v2:172.21.15.76:3300/0 conn(0x7f80f014ca10 0x7f80f00869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:43.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.379+0000 7f80f4b04700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f00879b0 0x7f80f0086f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:43.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80edffb700 1 -- 172.21.15.16:0/2738099360 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679322061 0 0) 0x7f80e8007900 con 0x7f80f0087460 2019-12-05T06:54:43.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80edffb700 1 -- 172.21.15.16:0/2738099360 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d0002e80 con 0x7f80f0087460 2019-12-05T06:54:43.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f5305700 1 -- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f00879b0 msgr2=0x7f80f0086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:43.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f00879b0 0x7f80f0086f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:43.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f5305700 1 -- 172.21.15.16:0/2738099360 >> v1:172.21.15.16:6789/0 conn(0x7f80f0087460 legacy=0x7f80f0152160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:43.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f5305700 1 -- 172.21.15.16:0/2738099360 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f00860e0 con 0x7f80f014ca10 2019-12-05T06:54:43.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> v2:172.21.15.76:3300/0 conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> v2:172.21.15.76:3300/0 conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:43.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80edffb700 1 -- 172.21.15.16:0/2738099360 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e00091e0 con 0x7f80f014ca10 2019-12-05T06:54:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80edffb700 1 -- 172.21.15.16:0/2738099360 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80e0000db0 con 0x7f80f014ca10 2019-12-05T06:54:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f6d68700 1 -- 172.21.15.16:0/2738099360 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f00860e0 con 0x7f80f014ca10 2019-12-05T06:54:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80edffb700 1 -- 172.21.15.16:0/2738099360 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80e00097a0 con 0x7f80f014ca10 2019-12-05T06:54:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.380+0000 7f80f6d68700 1 -- 172.21.15.16:0/2738099360 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f00860e0 con 0x7f80f014ca10 2019-12-05T06:54:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.381+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.381+0000 7f80edffb700 1 -- 172.21.15.16:0/2738099360 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f80e0000940 con 0x7f80f014ca10 2019-12-05T06:54:43.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.381+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.381+0000 7f80edffb700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f80d0023670 0x7f80d0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:43.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.382+0000 7f80edffb700 1 -- 172.21.15.16:0/2738099360 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f80e0047ef0 con 0x7f80f014ca10 2019-12-05T06:54:43.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.382+0000 7f80f4b04700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f80d0023670 0x7f80d0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:43.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.382+0000 7f80f4b04700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f80d0023670 0x7f80d0025ae0 crc :-1 s=READY pgs=652 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:43.392 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:54:43.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.389+0000 7f80f6d68700 1 -- 172.21.15.16:0/2738099360 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f00860e0 con 0x7f80f014ca10 2019-12-05T06:54:43.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.393+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.393+0000 7f80edffb700 1 -- 172.21.15.16:0/2738099360 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f80e00097a0 con 0x7f80f014ca10 2019-12-05T06:54:43.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.587+0000 7f80f6d68700 1 -- 172.21.15.16:0/2738099360 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f80f00860e0 con 0x7f80f014ca10 2019-12-05T06:54:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.588+0000 7f80f5305700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=THROTTLE_DONE pgs=1052 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:43.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.588+0000 7f80edffb700 1 -- 172.21.15.16:0/2738099360 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 191 v191) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f80e0000940 con 0x7f80f014ca10 2019-12-05T06:54:43.591 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:43.591 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":191,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:38.705960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:active","state_seq":14,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:43.592 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 191 2019-12-05T06:54:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.592+0000 7f80f6d68700 1 -- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f80d0023670 msgr2=0x7f80d0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.592+0000 7f80f6d68700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f80d0023670 0x7f80d0025ae0 crc :-1 s=READY pgs=652 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.592+0000 7f80f6d68700 1 -- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 msgr2=0x7f80f00869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.592+0000 7f80f6d68700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 0x7f80f00869e0 secure :-1 s=READY pgs=1052 cs=0 l=1 rx=0x7f80e000a030 tx=0x7f80e00101d0).stop 2019-12-05T06:54:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.593+0000 7f80f6d68700 1 -- 172.21.15.16:0/2738099360 shutdown_connections 2019-12-05T06:54:43.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.593+0000 7f80f6d68700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f80d0023670 0x7f80d0025ae0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:43.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.593+0000 7f80f6d68700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f80f00879b0 0x7f80f0086f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:43.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.593+0000 7f80f6d68700 1 --2- 172.21.15.16:0/2738099360 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f80f014ca10 0x7f80f00869e0 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:43.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.593+0000 7f80f6d68700 1 -- 172.21.15.16:0/2738099360 shutdown_connections 2019-12-05T06:54:43.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.593+0000 7f80f6d68700 1 -- 172.21.15.16:0/2738099360 wait complete. 2019-12-05T06:54:43.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:43.593+0000 7f80f6d68700 1 -- 172.21.15.16:0/2738099360 >> 172.21.15.16:0/2738099360 conn(0x7f80f0098480 msgr2=0x7f80f0071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:43.604 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:54:45.608 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:54:45.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.836+0000 7f21b0079700 1 Processor -- start 2019-12-05T06:54:45.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.836+0000 7f21b0079700 1 -- start start 2019-12-05T06:54:45.842 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.837+0000 7f21b0079700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f21a808d2c0 0x7f21a808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.837+0000 7f21b0079700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21a808fef0 0x7f21a808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:54:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.837+0000 7f21b0079700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a80958e0 con 0x7f21a808c6d0 2019-12-05T06:54:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.837+0000 7f21b0079700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a808df90 con 0x7f21a808fef0 2019-12-05T06:54:45.843 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.837+0000 7f21b0079700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a80a3f70 con 0x7f21a808d2c0 2019-12-05T06:54:45.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.837+0000 7f21aee17700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21a808fef0 0x7f21a808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:45.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.837+0000 7f21ae616700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f21a808c6d0 0x7f21a808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49030/0 (socket says 172.21.15.16:49030) 2019-12-05T06:54:45.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.837+0000 7f21ae616700 1 -- 172.21.15.16:0/1283043438 learned_addr learned my addr 172.21.15.16:0/1283043438 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:54:45.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21ad614700 1 -- 172.21.15.16:0/1283043438 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1678451989 0 0) 0x7f21980014d0 con 0x7f21a808c6d0 2019-12-05T06:54:45.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21ad614700 1 -- 172.21.15.16:0/1283043438 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2190002e80 con 0x7f21a808c6d0 2019-12-05T06:54:45.844 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21ade15700 1 --2- 172.21.15.16:0/1283043438 >> v2:172.21.15.76:3300/0 conn(0x7f21a808d2c0 0x7f21a808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21aee17700 1 -- 172.21.15.16:0/1283043438 >> v2:172.21.15.76:3300/0 conn(0x7f21a808d2c0 msgr2=0x7f21a808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:54:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21aee17700 1 --2- 172.21.15.16:0/1283043438 >> v2:172.21.15.76:3300/0 conn(0x7f21a808d2c0 0x7f21a808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21aee17700 1 -- 172.21.15.16:0/1283043438 >> v1:172.21.15.16:6789/0 conn(0x7f21a808c6d0 legacy=0x7f21a808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21aee17700 1 -- 172.21.15.16:0/1283043438 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a808e620 con 0x7f21a808fef0 2019-12-05T06:54:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21ade15700 1 --2- 172.21.15.16:0/1283043438 >> v2:172.21.15.76:3300/0 conn(0x7f21a808d2c0 0x7f21a808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:54:45.845 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21aee17700 1 --2- 172.21.15.16:0/1283043438 >> v2:172.21.15.16:3300/0 conn(0x7f21a808fef0 0x7f21a808f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f21a000af50 tx=0x7f21a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21aee17700 1 --2- 172.21.15.16:0/1283043438 >> v2:172.21.15.16:3300/0 conn(0x7f21a808fef0 0x7f21a808f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f21a000af50 tx=0x7f21a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21aee17700 1 --2- 172.21.15.16:0/1283043438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a808fef0 0x7f21a808f460 secure :-1 s=READY pgs=1565 cs=0 l=1 rx=0x7f21a000af50 tx=0x7f21a000b760).ready entity=mon.0 client_cookie=dbac49b029861057 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21aee17700 1 --2- 172.21.15.16:0/1283043438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a808fef0 0x7f21a808f460 secure :-1 s=THROTTLE_DONE pgs=1565 cs=0 l=1 rx=0x7f21a000af50 tx=0x7f21a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21ad614700 1 -- 172.21.15.16:0/1283043438 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a0009ba0 con 0x7f21a808fef0 2019-12-05T06:54:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21aee17700 1 --2- 172.21.15.16:0/1283043438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a808fef0 0x7f21a808f460 secure :-1 s=THROTTLE_DONE pgs=1565 cs=0 l=1 rx=0x7f21a000af50 tx=0x7f21a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.846 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21ad614700 1 -- 172.21.15.16:0/1283043438 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a00097c0 con 0x7f21a808fef0 2019-12-05T06:54:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21aee17700 1 --2- 172.21.15.16:0/1283043438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a808fef0 0x7f21a808f460 secure :-1 s=THROTTLE_DONE pgs=1565 cs=0 l=1 rx=0x7f21a000af50 tx=0x7f21a000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21ad614700 1 -- 172.21.15.16:0/1283043438 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a0000bc0 con 0x7f21a808fef0 2019-12-05T06:54:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21b0079700 1 -- 172.21.15.16:0/1283043438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a808fef0 msgr2=0x7f21a808f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.838+0000 7f21b0079700 1 --2- 172.21.15.16:0/1283043438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a808fef0 0x7f21a808f460 secure :-1 s=READY pgs=1565 cs=0 l=1 rx=0x7f21a000af50 tx=0x7f21a000b760).stop 2019-12-05T06:54:45.848 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.839+0000 7f21b0079700 1 -- 172.21.15.16:0/1283043438 shutdown_connections 2019-12-05T06:54:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.839+0000 7f21b0079700 1 --2- 172.21.15.16:0/1283043438 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a808fef0 0x7f21a808f460 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.839+0000 7f21b0079700 1 --2- 172.21.15.16:0/1283043438 >> v2:172.21.15.76:3300/0 conn(0x7f21a808d2c0 0x7f21a808ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:54:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.839+0000 7f21b0079700 1 -- 172.21.15.16:0/1283043438 shutdown_connections 2019-12-05T06:54:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.839+0000 7f21b0079700 1 -- 172.21.15.16:0/1283043438 wait complete. 2019-12-05T06:54:45.849 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.839+0000 7f21b0079700 1 -- 172.21.15.16:0/1283043438 >> 172.21.15.16:0/1283043438 conn(0x7f21a8061940 msgr2=0x7f21a8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.842+0000 7f21b0079700 1 Processor -- start 2019-12-05T06:54:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.842+0000 7f21b0079700 1 -- start start 2019-12-05T06:54:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21b0079700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f21a8096e90 0x7f21a809b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21b0079700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21a809b920 0x7f21a8097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21b0079700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21a8084b90 con 0x7f21a8084f30 2019-12-05T06:54:45.850 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21b0079700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a809a450 con 0x7f21a809b920 2019-12-05T06:54:45.851 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21b0079700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21a8099a20 con 0x7f21a8096e90 2019-12-05T06:54:45.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21aee17700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21a809b920 0x7f21a8097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21aee17700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f21a809b920 0x7f21a8097a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35640/0 (socket says 172.21.15.16:35640) 2019-12-05T06:54:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21ade15700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f21a8084f30 0x7f21a8097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49038/0 (socket says 172.21.15.16:49038) 2019-12-05T06:54:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21aee17700 1 -- 172.21.15.16:0/3750512325 learned_addr learned my addr 172.21.15.16:0/3750512325 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:54:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21ae616700 1 --2- 172.21.15.16:0/3750512325 >> v2:172.21.15.76:3300/0 conn(0x7f21a8096e90 0x7f21a809b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21aee17700 1 -- 172.21.15.16:0/3750512325 >> v2:172.21.15.76:3300/0 conn(0x7f21a8096e90 msgr2=0x7f21a809b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:45.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> v2:172.21.15.76:3300/0 conn(0x7f21a8096e90 0x7f21a809b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:45.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21aee17700 1 -- 172.21.15.16:0/3750512325 >> v1:172.21.15.16:6789/0 conn(0x7f21a8084f30 legacy=0x7f21a8097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:45.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21aee17700 1 -- 172.21.15.16:0/3750512325 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21a809aae0 con 0x7f21a809b920 2019-12-05T06:54:45.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> v2:172.21.15.16:3300/0 conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.843+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> v2:172.21.15.16:3300/0 conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.844+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=READY pgs=1567 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:45.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.844+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=THROTTLE_DONE pgs=1567 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.844+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=THROTTLE_DONE pgs=1567 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.844+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=THROTTLE_DONE pgs=1567 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.844+0000 7f21a6ffd700 1 -- 172.21.15.16:0/3750512325 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a000af60 con 0x7f21a809b920 2019-12-05T06:54:45.871 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.844+0000 7f21a6ffd700 1 -- 172.21.15.16:0/3750512325 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21a0009770 con 0x7f21a809b920 2019-12-05T06:54:45.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.844+0000 7f21a6ffd700 1 -- 172.21.15.16:0/3750512325 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21a0009e00 con 0x7f21a809b920 2019-12-05T06:54:45.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.844+0000 7f21b0079700 1 -- 172.21.15.16:0/3750512325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21a809aae0 con 0x7f21a809b920 2019-12-05T06:54:45.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.844+0000 7f21b0079700 1 -- 172.21.15.16:0/3750512325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21a809aae0 con 0x7f21a809b920 2019-12-05T06:54:45.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.844+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=THROTTLE_DONE pgs=1567 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.845+0000 7f21a6ffd700 1 -- 172.21.15.16:0/3750512325 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f21a0009960 con 0x7f21a809b920 2019-12-05T06:54:45.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.845+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=THROTTLE_DONE pgs=1567 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.845+0000 7f21a6ffd700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2188023630 0x7f2188025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:54:45.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.845+0000 7f21a6ffd700 1 -- 172.21.15.16:0/3750512325 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f21a0089e40 con 0x7f21a809b920 2019-12-05T06:54:45.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.845+0000 7f21ae616700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2188023630 0x7f2188025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:54:45.873 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.846+0000 7f21ae616700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2188023630 0x7f2188025ac0 crc :-1 s=READY pgs=654 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:54:45.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.852+0000 7f21b0079700 1 -- 172.21.15.16:0/3750512325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21a809aae0 con 0x7f21a809b920 2019-12-05T06:54:45.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.857+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=THROTTLE_DONE pgs=1567 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:45.874 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:45.857+0000 7f21a6ffd700 1 -- 172.21.15.16:0/3750512325 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f21a000ac40 con 0x7f21a809b920 2019-12-05T06:54:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.085+0000 7f21b0079700 1 -- 172.21.15.16:0/3750512325 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21a809aae0 con 0x7f21a809b920 2019-12-05T06:54:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.086+0000 7f21aee17700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=THROTTLE_DONE pgs=1567 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:54:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.086+0000 7f21a6ffd700 1 -- 172.21.15.16:0/3750512325 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f21a0009930 con 0x7f21a809b920 2019-12-05T06:54:46.089 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 192 2019-12-05T06:54:46.090 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:54:46.090 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:38.705960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:active","state_seq":14,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:54:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 -- 172.21.15.16:0/3750512325 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2188023630 msgr2=0x7f2188025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2188023630 0x7f2188025ac0 crc :-1 s=READY pgs=654 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:46.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 -- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 msgr2=0x7f21a8097a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:54:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 0x7f21a8097a80 secure :-1 s=READY pgs=1567 cs=0 l=1 rx=0x7f21a00543f0 tx=0x7f21a0054790).stop 2019-12-05T06:54:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 -- 172.21.15.16:0/3750512325 shutdown_connections 2019-12-05T06:54:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f2188023630 0x7f2188025ac0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 --2- 172.21.15.16:0/3750512325 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f21a809b920 0x7f21a8097a80 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 --2- 172.21.15.16:0/3750512325 >> v2:172.21.15.76:3300/0 conn(0x7f21a8096e90 0x7f21a809b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:54:46.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 -- 172.21.15.16:0/3750512325 shutdown_connections 2019-12-05T06:54:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 -- 172.21.15.16:0/3750512325 wait complete. 2019-12-05T06:54:46.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:54:46.089+0000 7f21b0079700 1 -- 172.21.15.16:0/3750512325 >> 172.21.15.16:0/3750512325 conn(0x7f21a80714f0 msgr2=0x7f21a809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:54:46.101 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:54:46.101 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 47 secs before thrashing 2019-12-05T06:54:46.681 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:54:46.687 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:55:16.855 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:55:16.860 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:55:33.102 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:33.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.353+0000 7f53bd678700 1 Processor -- start 2019-12-05T06:55:33.362 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.353+0000 7f53bd678700 1 -- start start 2019-12-05T06:55:33.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53bd678700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53b808d2c0 0x7f53b808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:33.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53bd678700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f53b808fef0 0x7f53b808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:33.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53bd678700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b80958e0 con 0x7f53b808c6d0 2019-12-05T06:55:33.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53bd678700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b808df90 con 0x7f53b808d2c0 2019-12-05T06:55:33.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53bd678700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f53b80a3f70 con 0x7f53b808fef0 2019-12-05T06:55:33.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53aeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53b808d2c0 0x7f53b808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:33.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53aeffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f53b808d2c0 0x7f53b808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35872/0 (socket says 172.21.15.16:35872) 2019-12-05T06:55:33.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53aeffd700 1 -- 172.21.15.16:0/2052279673 learned_addr learned my addr 172.21.15.16:0/2052279673 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:55:33.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53b7fff700 1 --2- 172.21.15.16:0/2052279673 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f53b808fef0 0x7f53b808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:33.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53aeffd700 1 -- 172.21.15.16:0/2052279673 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f53b808fef0 msgr2=0x7f53b808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:55:33.364 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.355+0000 7f53aeffd700 1 --2- 172.21.15.16:0/2052279673 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f53b808fef0 0x7f53b808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:33.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53aeffd700 1 -- 172.21.15.16:0/2052279673 >> v1:172.21.15.76:6789/0 conn(0x7f53b808c6d0 legacy=0x7f53b808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:33.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53aeffd700 1 -- 172.21.15.16:0/2052279673 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b808e620 con 0x7f53b808d2c0 2019-12-05T06:55:33.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53b6ffd700 1 -- 172.21.15.16:0/2052279673 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415902857 0 0) 0x7f53a8000ff0 con 0x7f53b808c6d0 2019-12-05T06:55:33.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53aeffd700 1 --2- 172.21.15.16:0/2052279673 >> v2:172.21.15.16:3300/0 conn(0x7f53b808d2c0 0x7f53b808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f53a000f740 tx=0x7f53a000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53aeffd700 1 --2- 172.21.15.16:0/2052279673 >> v2:172.21.15.16:3300/0 conn(0x7f53b808d2c0 0x7f53b808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f53a000f740 tx=0x7f53a000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53aeffd700 1 --2- 172.21.15.16:0/2052279673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b808d2c0 0x7f53b808ef20 secure :-1 s=READY pgs=1570 cs=0 l=1 rx=0x7f53a000f740 tx=0x7f53a000f570).ready entity=mon.0 client_cookie=587c9e2fd8f9ab3d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:33.365 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53aeffd700 1 --2- 172.21.15.16:0/2052279673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b808d2c0 0x7f53b808ef20 secure :-1 s=THROTTLE_DONE pgs=1570 cs=0 l=1 rx=0x7f53a000f740 tx=0x7f53a000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53b6ffd700 1 -- 172.21.15.16:0/2052279673 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53a0000960 con 0x7f53b808d2c0 2019-12-05T06:55:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53aeffd700 1 --2- 172.21.15.16:0/2052279673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b808d2c0 0x7f53b808ef20 secure :-1 s=THROTTLE_DONE pgs=1570 cs=0 l=1 rx=0x7f53a000f740 tx=0x7f53a000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53aeffd700 1 --2- 172.21.15.16:0/2052279673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b808d2c0 0x7f53b808ef20 secure :-1 s=THROTTLE_DONE pgs=1570 cs=0 l=1 rx=0x7f53a000f740 tx=0x7f53a000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53b6ffd700 1 -- 172.21.15.16:0/2052279673 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a0000c90 con 0x7f53b808d2c0 2019-12-05T06:55:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53b6ffd700 1 -- 172.21.15.16:0/2052279673 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53a0000960 con 0x7f53b808d2c0 2019-12-05T06:55:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53bd678700 1 -- 172.21.15.16:0/2052279673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b808d2c0 msgr2=0x7f53b808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:33.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.356+0000 7f53bd678700 1 --2- 172.21.15.16:0/2052279673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b808d2c0 0x7f53b808ef20 secure :-1 s=READY pgs=1570 cs=0 l=1 rx=0x7f53a000f740 tx=0x7f53a000f570).stop 2019-12-05T06:55:33.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.357+0000 7f53bd678700 1 -- 172.21.15.16:0/2052279673 shutdown_connections 2019-12-05T06:55:33.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.357+0000 7f53bd678700 1 --2- 172.21.15.16:0/2052279673 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f53b808fef0 0x7f53b808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:33.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.357+0000 7f53bd678700 1 --2- 172.21.15.16:0/2052279673 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f53b808d2c0 0x7f53b808ef20 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:33.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.357+0000 7f53bd678700 1 -- 172.21.15.16:0/2052279673 shutdown_connections 2019-12-05T06:55:33.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.357+0000 7f53bd678700 1 -- 172.21.15.16:0/2052279673 wait complete. 2019-12-05T06:55:33.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.357+0000 7f53bd678700 1 -- 172.21.15.16:0/2052279673 >> 172.21.15.16:0/2052279673 conn(0x7f53b8061940 msgr2=0x7f53b8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.362+0000 7f53bd678700 1 Processor -- start 2019-12-05T06:55:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.363+0000 7f53bd678700 1 -- start start 2019-12-05T06:55:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.364+0000 7f53bd678700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f53b814d400 0x7f53b814d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.364+0000 7f53bd678700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b8098da0 con 0x7f53b8132d00 2019-12-05T06:55:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.364+0000 7f53bd678700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53b809f0a0 con 0x7f53b8132530 2019-12-05T06:55:33.371 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.364+0000 7f53bd678700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f53b809bb50 con 0x7f53b814d400 2019-12-05T06:55:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.364+0000 7f53aeffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f53b8132d00 0x7f53b8131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49284/0 (socket says 172.21.15.16:49284) 2019-12-05T06:55:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.364+0000 7f53aeffd700 1 -- 172.21.15.16:0/4237002261 learned_addr learned my addr 172.21.15.16:0/4237002261 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.364+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> v2:172.21.15.76:3300/0 conn(0x7f53b814d400 0x7f53b814d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.365+0000 7f53b4ff9700 1 -- 172.21.15.16:0/4237002261 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063506780 0 0) 0x7f53a0007900 con 0x7f53b8132d00 2019-12-05T06:55:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.365+0000 7f53b4ff9700 1 -- 172.21.15.16:0/4237002261 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f539c002e80 con 0x7f53b8132d00 2019-12-05T06:55:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.365+0000 7f53b77fe700 1 -- 172.21.15.16:0/4237002261 >> v1:172.21.15.76:6789/0 conn(0x7f53b8132530 legacy=0x7f53b8157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:33.372 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.365+0000 7f53b4ff9700 1 -- 172.21.15.16:0/4237002261 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 343849314 0 0) 0x7f53a00081d0 con 0x7f53b8132d00 2019-12-05T06:55:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.365+0000 7f53b77fe700 1 -- 172.21.15.16:0/4237002261 >> v1:172.21.15.16:6789/0 conn(0x7f53b8132d00 legacy=0x7f53b8131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:33.373 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.365+0000 7f53b77fe700 1 -- 172.21.15.16:0/4237002261 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53b809b0f0 con 0x7f53b814d400 2019-12-05T06:55:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.365+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> v2:172.21.15.76:3300/0 conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.365+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> v2:172.21.15.76:3300/0 conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.365+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=READY pgs=1058 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:33.374 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.366+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=THROTTLE_DONE pgs=1058 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.366+0000 7f53b4ff9700 1 -- 172.21.15.16:0/4237002261 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53a800f070 con 0x7f53b814d400 2019-12-05T06:55:33.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.366+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=THROTTLE_DONE pgs=1058 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.366+0000 7f53b4ff9700 1 -- 172.21.15.16:0/4237002261 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f53a800ec90 con 0x7f53b814d400 2019-12-05T06:55:33.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.366+0000 7f53bd678700 1 -- 172.21.15.16:0/4237002261 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53b809b0f0 con 0x7f53b814d400 2019-12-05T06:55:33.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.366+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=THROTTLE_DONE pgs=1058 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.375 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.366+0000 7f53b4ff9700 1 -- 172.21.15.16:0/4237002261 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f53a800ee80 con 0x7f53b814d400 2019-12-05T06:55:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.366+0000 7f53bd678700 1 -- 172.21.15.16:0/4237002261 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53b809b0f0 con 0x7f53b814d400 2019-12-05T06:55:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.367+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=THROTTLE_DONE pgs=1058 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.367+0000 7f53b4ff9700 1 -- 172.21.15.16:0/4237002261 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f53a800ef20 con 0x7f53b814d400 2019-12-05T06:55:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.367+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=THROTTLE_DONE pgs=1058 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.368+0000 7f53b4ff9700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f539c023680 0x7f539c025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.368+0000 7f53b4ff9700 1 -- 172.21.15.16:0/4237002261 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f53a8045de0 con 0x7f53b814d400 2019-12-05T06:55:33.376 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.368+0000 7f53aeffd700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f539c023680 0x7f539c025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:33.377 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.368+0000 7f53aeffd700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f539c023680 0x7f539c025af0 crc :-1 s=READY pgs=657 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:33.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.383+0000 7f53bd678700 1 -- 172.21.15.16:0/4237002261 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53b809b0f0 con 0x7f53b814d400 2019-12-05T06:55:33.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.392+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=THROTTLE_DONE pgs=1058 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.392+0000 7f53b4ff9700 1 -- 172.21.15.16:0/4237002261 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f53a800ea80 con 0x7f53b814d400 2019-12-05T06:55:33.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.627+0000 7f53bd678700 1 -- 172.21.15.16:0/4237002261 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f53b809b0f0 con 0x7f53b814d400 2019-12-05T06:55:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.628+0000 7f53b77fe700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=THROTTLE_DONE pgs=1058 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.628+0000 7f53b4ff9700 1 -- 172.21.15.16:0/4237002261 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f53a800ee80 con 0x7f53b814d400 2019-12-05T06:55:33.629 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 192 2019-12-05T06:55:33.629 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:33.630 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:38.705960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:active","state_seq":14,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.630+0000 7f53bd678700 1 -- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f539c023680 msgr2=0x7f539c025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.630+0000 7f53bd678700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f539c023680 0x7f539c025af0 crc :-1 s=READY pgs=657 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.630+0000 7f53bd678700 1 -- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 msgr2=0x7f53b814d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.630+0000 7f53bd678700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 0x7f53b814d7f0 secure :-1 s=READY pgs=1058 cs=0 l=1 rx=0x7f53a80103a0 tx=0x7f53a8010740).stop 2019-12-05T06:55:33.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.630+0000 7f53bd678700 1 -- 172.21.15.16:0/4237002261 shutdown_connections 2019-12-05T06:55:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.630+0000 7f53bd678700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f539c023680 0x7f539c025af0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.630+0000 7f53bd678700 1 --2- 172.21.15.16:0/4237002261 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f53b814d400 0x7f53b814d7f0 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.631+0000 7f53bd678700 1 -- 172.21.15.16:0/4237002261 shutdown_connections 2019-12-05T06:55:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.631+0000 7f53bd678700 1 -- 172.21.15.16:0/4237002261 wait complete. 2019-12-05T06:55:33.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.631+0000 7f53bd678700 1 -- 172.21.15.16:0/4237002261 >> 172.21.15.16:0/4237002261 conn(0x7f53b80714f0 msgr2=0x7f53b809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:33.649 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:55:33.650 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:55:33.650 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:55:33.659 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:55:33.660 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.968+0000 7f008fb63700 1 Processor -- start 2019-12-05T06:55:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.968+0000 7f008fb63700 1 -- start start 2019-12-05T06:55:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.970+0000 7f008fb63700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f008808d2c0 0x7f008808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.970+0000 7f008fb63700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008808fef0 0x7f008808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.970+0000 7f008fb63700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00880958e0 con 0x7f008808c6d0 2019-12-05T06:55:33.979 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.970+0000 7f008fb63700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f008808df90 con 0x7f008808d2c0 2019-12-05T06:55:33.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.970+0000 7f008fb63700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00880a3f70 con 0x7f008808fef0 2019-12-05T06:55:33.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.970+0000 7f008e100700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f008808c6d0 0x7f008808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49294/0 (socket says 172.21.15.16:49294) 2019-12-05T06:55:33.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.970+0000 7f008e100700 1 -- 172.21.15.16:0/3692690012 learned_addr learned my addr 172.21.15.16:0/3692690012 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:33.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.970+0000 7f008d8ff700 1 --2- 172.21.15.16:0/3692690012 >> v2:172.21.15.76:3300/0 conn(0x7f008808d2c0 0x7f008808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:33.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.970+0000 7f008e901700 1 --2- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008808fef0 0x7f008808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:33.980 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.971+0000 7f008d0fe700 1 -- 172.21.15.16:0/3692690012 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181584339 0 0) 0x7f0080001150 con 0x7f008808c6d0 2019-12-05T06:55:33.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.971+0000 7f008d0fe700 1 -- 172.21.15.16:0/3692690012 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0070002e80 con 0x7f008808c6d0 2019-12-05T06:55:33.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.971+0000 7f008d8ff700 1 -- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008808fef0 msgr2=0x7f008808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:55:33.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.971+0000 7f008d8ff700 1 --2- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008808fef0 0x7f008808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:33.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.971+0000 7f008d8ff700 1 -- 172.21.15.16:0/3692690012 >> v1:172.21.15.16:6789/0 conn(0x7f008808c6d0 legacy=0x7f008808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:33.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.971+0000 7f008d8ff700 1 -- 172.21.15.16:0/3692690012 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f008808e620 con 0x7f008808d2c0 2019-12-05T06:55:33.981 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.971+0000 7f008e901700 1 --2- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008808fef0 0x7f008808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:55:33.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.971+0000 7f008d8ff700 1 --2- 172.21.15.16:0/3692690012 >> v2:172.21.15.76:3300/0 conn(0x7f008808d2c0 0x7f008808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f008400b2d0 tx=0x7f008400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.972+0000 7f008d8ff700 1 --2- 172.21.15.16:0/3692690012 >> v2:172.21.15.76:3300/0 conn(0x7f008808d2c0 0x7f008808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f008400b2d0 tx=0x7f008400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.972+0000 7f008d8ff700 1 --2- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f008808d2c0 0x7f008808ef20 secure :-1 s=READY pgs=1059 cs=0 l=1 rx=0x7f008400b2d0 tx=0x7f008400bae0).ready entity=mon.1 client_cookie=3778dfa2e361ca51 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:33.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.972+0000 7f008d8ff700 1 --2- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f008808d2c0 0x7f008808ef20 secure :-1 s=THROTTLE_DONE pgs=1059 cs=0 l=1 rx=0x7f008400b2d0 tx=0x7f008400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.972+0000 7f008d0fe700 1 -- 172.21.15.16:0/3692690012 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0084007590 con 0x7f008808d2c0 2019-12-05T06:55:33.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.972+0000 7f008d8ff700 1 --2- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f008808d2c0 0x7f008808ef20 secure :-1 s=THROTTLE_DONE pgs=1059 cs=0 l=1 rx=0x7f008400b2d0 tx=0x7f008400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.982 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.972+0000 7f008d8ff700 1 --2- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f008808d2c0 0x7f008808ef20 secure :-1 s=THROTTLE_DONE pgs=1059 cs=0 l=1 rx=0x7f008400b2d0 tx=0x7f008400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.972+0000 7f008d0fe700 1 -- 172.21.15.16:0/3692690012 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0084009d30 con 0x7f008808d2c0 2019-12-05T06:55:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.972+0000 7f008d0fe700 1 -- 172.21.15.16:0/3692690012 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f008400a400 con 0x7f008808d2c0 2019-12-05T06:55:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.972+0000 7f008fb63700 1 -- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f008808d2c0 msgr2=0x7f008808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.972+0000 7f008fb63700 1 --2- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f008808d2c0 0x7f008808ef20 secure :-1 s=READY pgs=1059 cs=0 l=1 rx=0x7f008400b2d0 tx=0x7f008400bae0).stop 2019-12-05T06:55:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.974+0000 7f008fb63700 1 -- 172.21.15.16:0/3692690012 shutdown_connections 2019-12-05T06:55:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.974+0000 7f008fb63700 1 --2- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f008808d2c0 0x7f008808ef20 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:33.983 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.974+0000 7f008fb63700 1 --2- 172.21.15.16:0/3692690012 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008808fef0 0x7f008808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:33.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.974+0000 7f008fb63700 1 -- 172.21.15.16:0/3692690012 shutdown_connections 2019-12-05T06:55:33.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.974+0000 7f008fb63700 1 -- 172.21.15.16:0/3692690012 wait complete. 2019-12-05T06:55:33.984 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.974+0000 7f008fb63700 1 -- 172.21.15.16:0/3692690012 >> 172.21.15.16:0/3692690012 conn(0x7f0088061940 msgr2=0x7f0088096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.979+0000 7f008fb63700 1 Processor -- start 2019-12-05T06:55:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.979+0000 7f008fb63700 1 -- start start 2019-12-05T06:55:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008fb63700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00881302f0 0x7f008809d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008fb63700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f008809e7d0 0x7f008809dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008fb63700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008809e280 0x7f008806dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008fb63700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0088131ff0 con 0x7f00881302f0 2019-12-05T06:55:33.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008fb63700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00881315c0 con 0x7f008809e7d0 2019-12-05T06:55:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008fb63700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0088131890 con 0x7f008809e280 2019-12-05T06:55:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008e100700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00881302f0 0x7f008809d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008e100700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f00881302f0 0x7f008809d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35900/0 (socket says 172.21.15.16:35900) 2019-12-05T06:55:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008e100700 1 -- 172.21.15.16:0/4282949676 learned_addr learned my addr 172.21.15.16:0/4282949676 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:55:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008e901700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008809e280 0x7f008806dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.980+0000 7f008d8ff700 1 --2- 172.21.15.16:0/4282949676 >> v2:172.21.15.76:3300/0 conn(0x7f008809e7d0 0x7f008809dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:33.989 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.981+0000 7f008e100700 1 -- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008809e280 msgr2=0x7f008806dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.981+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008809e280 0x7f008806dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.981+0000 7f008e100700 1 -- 172.21.15.16:0/4282949676 >> v2:172.21.15.76:3300/0 conn(0x7f008809e7d0 msgr2=0x7f008809dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.981+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> v2:172.21.15.76:3300/0 conn(0x7f008809e7d0 0x7f008809dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.981+0000 7f008e100700 1 -- 172.21.15.16:0/4282949676 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0088132680 con 0x7f00881302f0 2019-12-05T06:55:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.981+0000 7f008e901700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008809e280 0x7f008806dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:55:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.981+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> v2:172.21.15.16:3300/0 conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.990 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.981+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> v2:172.21.15.16:3300/0 conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.981+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=READY pgs=1573 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.982+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=THROTTLE_DONE pgs=1573 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.982+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=THROTTLE_DONE pgs=1573 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.982+0000 7f007effd700 1 -- 172.21.15.16:0/4282949676 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0080000c90 con 0x7f00881302f0 2019-12-05T06:55:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.982+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=THROTTLE_DONE pgs=1573 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.982+0000 7f007effd700 1 -- 172.21.15.16:0/4282949676 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00800010c0 con 0x7f00881302f0 2019-12-05T06:55:33.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.982+0000 7f008fb63700 1 -- 172.21.15.16:0/4282949676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0088132680 con 0x7f00881302f0 2019-12-05T06:55:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.982+0000 7f007effd700 1 -- 172.21.15.16:0/4282949676 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00800016d0 con 0x7f00881302f0 2019-12-05T06:55:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.982+0000 7f008fb63700 1 -- 172.21.15.16:0/4282949676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008812cb90 con 0x7f00881302f0 2019-12-05T06:55:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.983+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=THROTTLE_DONE pgs=1573 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.984+0000 7f007effd700 1 -- 172.21.15.16:0/4282949676 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0080000c90 con 0x7f00881302f0 2019-12-05T06:55:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.984+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=THROTTLE_DONE pgs=1573 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.984+0000 7f007effd700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0068023630 0x7f0068025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:33.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.984+0000 7f007effd700 1 -- 172.21.15.16:0/4282949676 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f0080045cd0 con 0x7f00881302f0 2019-12-05T06:55:33.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.985+0000 7f008d8ff700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0068023630 0x7f0068025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:33.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:33.985+0000 7f008d8ff700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0068023630 0x7f0068025ac0 crc :-1 s=READY pgs=658 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:34.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.000+0000 7f008fb63700 1 -- 172.21.15.16:0/4282949676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0088132680 con 0x7f00881302f0 2019-12-05T06:55:34.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.009+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=THROTTLE_DONE pgs=1573 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:34.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.009+0000 7f007effd700 1 -- 172.21.15.16:0/4282949676 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f0080010ae0 con 0x7f00881302f0 2019-12-05T06:55:34.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.275+0000 7f008fb63700 1 -- 172.21.15.16:0/4282949676 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0088132680 con 0x7f00881302f0 2019-12-05T06:55:34.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.276+0000 7f008e100700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=THROTTLE_DONE pgs=1573 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:34.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.276+0000 7f007effd700 1 -- 172.21.15.16:0/4282949676 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f0080010ae0 con 0x7f00881302f0 2019-12-05T06:55:34.278 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 192 2019-12-05T06:55:34.278 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:34.278 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:38.705960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:active","state_seq":14,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:34.280 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 -- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0068023630 msgr2=0x7f0068025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0068023630 0x7f0068025ac0 crc :-1 s=READY pgs=658 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 -- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 msgr2=0x7f008809d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 0x7f008809d800 secure :-1 s=READY pgs=1573 cs=0 l=1 rx=0x7f00800103e0 tx=0x7f00800102e0).stop 2019-12-05T06:55:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 -- 172.21.15.16:0/4282949676 shutdown_connections 2019-12-05T06:55:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f008809e280 0x7f008806dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:34.281 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 --2- 172.21.15.16:0/4282949676 >> v2:172.21.15.76:3300/0 conn(0x7f008809e7d0 0x7f008809dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0068023630 0x7f0068025ac0 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 --2- 172.21.15.16:0/4282949676 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f00881302f0 0x7f008809d800 unknown :-1 s=CLOSED pgs=1573 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 -- 172.21.15.16:0/4282949676 shutdown_connections 2019-12-05T06:55:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 -- 172.21.15.16:0/4282949676 wait complete. 2019-12-05T06:55:34.282 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:34.279+0000 7f008fb63700 1 -- 172.21.15.16:0/4282949676 >> 172.21.15.16:0/4282949676 conn(0x7f00880a2850 msgr2=0x7f008806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:34.295 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:55:36.296 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:36.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.577+0000 7f7dba4cd700 1 Processor -- start 2019-12-05T06:55:36.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.577+0000 7f7dba4cd700 1 -- start start 2019-12-05T06:55:36.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.579+0000 7f7dba4cd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7db408c6d0 0x7f7db408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:36.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.579+0000 7f7dba4cd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db408fb80 con 0x7f7db408fef0 2019-12-05T06:55:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.579+0000 7f7dba4cd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db40958e0 con 0x7f7db408f720 2019-12-05T06:55:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.579+0000 7f7dba4cd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db408df90 con 0x7f7db408c6d0 2019-12-05T06:55:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.579+0000 7f7db8a6a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7db408c6d0 0x7f7db408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.579+0000 7f7db3fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7db408fef0 0x7f7db408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49312/0 (socket says 172.21.15.16:49312) 2019-12-05T06:55:36.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.579+0000 7f7db3fff700 1 -- 172.21.15.16:0/4041128251 learned_addr learned my addr 172.21.15.16:0/4041128251 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:36.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.580+0000 7f7db8a6a700 1 -- 172.21.15.16:0/4041128251 >> v1:172.21.15.76:6789/0 conn(0x7f7db408f720 legacy=0x7f7db41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:36.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.580+0000 7f7db8a6a700 1 -- 172.21.15.16:0/4041128251 >> v1:172.21.15.16:6789/0 conn(0x7f7db408fef0 legacy=0x7f7db408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:36.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.580+0000 7f7db37fe700 1 -- 172.21.15.16:0/4041128251 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3535998821 0 0) 0x7f7dac000d30 con 0x7f7db408f720 2019-12-05T06:55:36.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.580+0000 7f7db8a6a700 1 -- 172.21.15.16:0/4041128251 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db408e620 con 0x7f7db408c6d0 2019-12-05T06:55:36.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.580+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/4041128251 >> v2:172.21.15.16:3300/0 conn(0x7f7db408c6d0 0x7f7db408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7d9c00af50 tx=0x7f7d9c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.580+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/4041128251 >> v2:172.21.15.16:3300/0 conn(0x7f7db408c6d0 0x7f7db408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7d9c00af50 tx=0x7f7d9c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.580+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/4041128251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db408c6d0 0x7f7db408cac0 secure :-1 s=READY pgs=1575 cs=0 l=1 rx=0x7f7d9c00af50 tx=0x7f7d9c00b760).ready entity=mon.0 client_cookie=76ecbb0b04267469 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:36.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.580+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/4041128251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db408c6d0 0x7f7db408cac0 secure :-1 s=THROTTLE_DONE pgs=1575 cs=0 l=1 rx=0x7f7d9c00af50 tx=0x7f7d9c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.580+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/4041128251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db408c6d0 0x7f7db408cac0 secure :-1 s=THROTTLE_DONE pgs=1575 cs=0 l=1 rx=0x7f7d9c00af50 tx=0x7f7d9c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.580+0000 7f7db37fe700 1 -- 172.21.15.16:0/4041128251 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d9c000960 con 0x7f7db408c6d0 2019-12-05T06:55:36.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.581+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/4041128251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db408c6d0 0x7f7db408cac0 secure :-1 s=THROTTLE_DONE pgs=1575 cs=0 l=1 rx=0x7f7d9c00af50 tx=0x7f7d9c00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.581+0000 7f7db37fe700 1 -- 172.21.15.16:0/4041128251 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c009860 con 0x7f7db408c6d0 2019-12-05T06:55:36.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.581+0000 7f7db37fe700 1 -- 172.21.15.16:0/4041128251 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d9c009f30 con 0x7f7db408c6d0 2019-12-05T06:55:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.581+0000 7f7dba4cd700 1 -- 172.21.15.16:0/4041128251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db408c6d0 msgr2=0x7f7db408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.581+0000 7f7dba4cd700 1 --2- 172.21.15.16:0/4041128251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db408c6d0 0x7f7db408cac0 secure :-1 s=READY pgs=1575 cs=0 l=1 rx=0x7f7d9c00af50 tx=0x7f7d9c00b760).stop 2019-12-05T06:55:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.582+0000 7f7dba4cd700 1 -- 172.21.15.16:0/4041128251 shutdown_connections 2019-12-05T06:55:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.582+0000 7f7dba4cd700 1 --2- 172.21.15.16:0/4041128251 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7db408c6d0 0x7f7db408cac0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:36.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.583+0000 7f7dba4cd700 1 -- 172.21.15.16:0/4041128251 shutdown_connections 2019-12-05T06:55:36.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.583+0000 7f7dba4cd700 1 -- 172.21.15.16:0/4041128251 wait complete. 2019-12-05T06:55:36.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.583+0000 7f7dba4cd700 1 -- 172.21.15.16:0/4041128251 >> 172.21.15.16:0/4041128251 conn(0x7f7db4061940 msgr2=0x7f7db4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:36.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.587+0000 7f7dba4cd700 1 Processor -- start 2019-12-05T06:55:36.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.588+0000 7f7dba4cd700 1 -- start start 2019-12-05T06:55:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.589+0000 7f7dba4cd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7db4096e90 0x7f7db409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.589+0000 7f7dba4cd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db4084b90 con 0x7f7db409b920 2019-12-05T06:55:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.589+0000 7f7dba4cd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7db4098790 con 0x7f7db4084f30 2019-12-05T06:55:36.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.589+0000 7f7dba4cd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7db409a450 con 0x7f7db4096e90 2019-12-05T06:55:36.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.590+0000 7f7db926b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7db409b920 0x7f7db4097a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49322/0 (socket says 172.21.15.16:49322) 2019-12-05T06:55:36.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.590+0000 7f7db926b700 1 -- 172.21.15.16:0/3401980265 learned_addr learned my addr 172.21.15.16:0/3401980265 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:36.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.590+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> v2:172.21.15.76:3300/0 conn(0x7f7db4096e90 0x7f7db409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:36.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.590+0000 7f7db8a6a700 1 -- 172.21.15.16:0/3401980265 >> v2:172.21.15.76:3300/0 conn(0x7f7db4096e90 msgr2=0x7f7db409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2019-12-05T06:55:36.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.590+0000 7f7db8a6a700 1 -- 172.21.15.16:0/3401980265 >> v2:172.21.15.76:3300/0 conn(0x7f7db4096e90 msgr2=0x7f7db409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:55:36.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.590+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> v2:172.21.15.76:3300/0 conn(0x7f7db4096e90 0x7f7db409b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:55:36.601 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.590+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> v2:172.21.15.76:3300/0 conn(0x7f7db4096e90 0x7f7db409b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:55:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.590+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808203798 0 0) 0x7f7dac0015c0 con 0x7f7db409b920 2019-12-05T06:55:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.590+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d98002e80 con 0x7f7db409b920 2019-12-05T06:55:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2744469241 0 0) 0x7f7da4001100 con 0x7f7db4084f30 2019-12-05T06:55:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d980030b0 con 0x7f7db4084f30 2019-12-05T06:55:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 4015270802 0 0) 0x7f7dac0017c0 con 0x7f7db409b920 2019-12-05T06:55:36.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7d980033c0 con 0x7f7db409b920 2019-12-05T06:55:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f7dac001d10 con 0x7f7db409b920 2019-12-05T06:55:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 >> v2:172.21.15.76:3300/0 conn(0x7f7db4096e90 msgr2=0x7f7db409b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2019-12-05T06:55:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 --2- 172.21.15.16:0/3401980265 >> v2:172.21.15.76:3300/0 conn(0x7f7db4096e90 0x7f7db409b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 >> v1:172.21.15.76:6789/0 conn(0x7f7db4084f30 legacy=0x7f7db4097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 >> v1:172.21.15.16:6789/0 conn(0x7f7db409b920 legacy=0x7f7db4097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:36.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d980046f0 0x7f7d98004bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d980050f0 0x7f7d980032e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98003940 con 0x7f7d98005d40 2019-12-05T06:55:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98003b70 con 0x7f7d980050f0 2019-12-05T06:55:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.591+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d98003da0 con 0x7f7d980046f0 2019-12-05T06:55:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7db926b700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d980050f0 0x7f7d980032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:36.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7db3fff700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d980046f0 0x7f7d98004bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7db8a6a700 1 -- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d980046f0 msgr2=0x7f7d98004bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7d980046f0 0x7f7d98004bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7db8a6a700 1 -- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d980050f0 msgr2=0x7f7d980032e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f7d980050f0 0x7f7d980032e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7db926b700 1 -- 172.21.15.16:0/3401980265 reap_dead start 2019-12-05T06:55:36.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7db8a6a700 1 -- 172.21.15.16:0/3401980265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d98004050 con 0x7f7d98005d40 2019-12-05T06:55:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7dba4cd700 1 -- 172.21.15.16:0/3401980265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db409aae0 con 0x7f7d98005d40 2019-12-05T06:55:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.592+0000 7f7dba4cd700 1 -- 172.21.15.16:0/3401980265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db4084b90 con 0x7f7d98005d40 2019-12-05T06:55:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.593+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.593+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=READY pgs=1577 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.593+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=THROTTLE_DONE pgs=1577 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.593+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d9c00b4e0 con 0x7f7d98005d40 2019-12-05T06:55:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.593+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=THROTTLE_DONE pgs=1577 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.593+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d9c009790 con 0x7f7d98005d40 2019-12-05T06:55:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.593+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=THROTTLE_DONE pgs=1577 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.593+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d9c000940 con 0x7f7d98005d40 2019-12-05T06:55:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.594+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=THROTTLE_DONE pgs=1577 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.595+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7d9c009980 con 0x7f7d98005d40 2019-12-05T06:55:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.595+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=THROTTLE_DONE pgs=1577 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.595+0000 7f7db17fa700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98026350 0x7f7d980287a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.595+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f7d9c045c50 con 0x7f7d98005d40 2019-12-05T06:55:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.596+0000 7f7db3fff700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98026350 0x7f7d980287a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:36.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.596+0000 7f7db3fff700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98026350 0x7f7d980287a0 crc :-1 s=READY pgs=659 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:36.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.608+0000 7f7dba4cd700 1 -- 172.21.15.16:0/3401980265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8c002e60 con 0x7f7d98005d40 2019-12-05T06:55:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.616+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=THROTTLE_DONE pgs=1577 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.616+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7d9c009580 con 0x7f7d98005d40 2019-12-05T06:55:36.854 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.853+0000 7f7dba4cd700 1 -- 172.21.15.16:0/3401980265 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d8c001d10 con 0x7f7d98005d40 2019-12-05T06:55:36.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.854+0000 7f7db8a6a700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=THROTTLE_DONE pgs=1577 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:36.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.854+0000 7f7db17fa700 1 -- 172.21.15.16:0/3401980265 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7d9c009960 con 0x7f7d98005d40 2019-12-05T06:55:36.857 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:36.858 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:38.705960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:active","state_seq":14,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:36.858 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 192 2019-12-05T06:55:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.859+0000 7f7dba4cd700 1 -- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98026350 msgr2=0x7f7d980287a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.859+0000 7f7dba4cd700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98026350 0x7f7d980287a0 crc :-1 s=READY pgs=659 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.859+0000 7f7dba4cd700 1 -- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 msgr2=0x7f7d98006130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:36.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.859+0000 7f7dba4cd700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 secure :-1 s=READY pgs=1577 cs=0 l=1 rx=0x7f7d9c00b3d0 tx=0x7f7d9c00b460).stop 2019-12-05T06:55:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.859+0000 7f7dba4cd700 1 -- 172.21.15.16:0/3401980265 shutdown_connections 2019-12-05T06:55:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.859+0000 7f7dba4cd700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7d98026350 0x7f7d980287a0 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.859+0000 7f7dba4cd700 1 --2- 172.21.15.16:0/3401980265 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7d98005d40 0x7f7d98006130 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:36.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.859+0000 7f7dba4cd700 1 -- 172.21.15.16:0/3401980265 shutdown_connections 2019-12-05T06:55:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.859+0000 7f7dba4cd700 1 -- 172.21.15.16:0/3401980265 wait complete. 2019-12-05T06:55:36.862 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:36.859+0000 7f7dba4cd700 1 -- 172.21.15.16:0/3401980265 >> 172.21.15.16:0/3401980265 conn(0x7f7db40714f0 msgr2=0x7f7db409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:36.877 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:55:38.878 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:39.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.166+0000 7f7e2ab92700 1 Processor -- start 2019-12-05T06:55:39.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.167+0000 7f7e2ab92700 1 -- start start 2019-12-05T06:55:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.169+0000 7f7e2ab92700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 0x7f7e2408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.169+0000 7f7e2ab92700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e240958e0 con 0x7f7e2408fef0 2019-12-05T06:55:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.169+0000 7f7e2ab92700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e240629e0 con 0x7f7e2408c6d0 2019-12-05T06:55:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.169+0000 7f7e2ab92700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e2408df90 con 0x7f7e2408d2c0 2019-12-05T06:55:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.169+0000 7f7e29930700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7e2408fef0 0x7f7e2408f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49338/0 (socket says 172.21.15.16:49338) 2019-12-05T06:55:39.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.170+0000 7f7e29930700 1 -- 172.21.15.16:0/3497631809 learned_addr learned my addr 172.21.15.16:0/3497631809 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.170+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3497631809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 0x7f7e2408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.170+0000 7f7e23fff700 1 -- 172.21.15.16:0/3497631809 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2344136895 0 0) 0x7f7e14001150 con 0x7f7e2408fef0 2019-12-05T06:55:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.170+0000 7f7e23fff700 1 -- 172.21.15.16:0/3497631809 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e0c002e80 con 0x7f7e2408fef0 2019-12-05T06:55:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.170+0000 7f7e23fff700 1 -- 172.21.15.16:0/3497631809 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 59611086 0 0) 0x7f7e1c001400 con 0x7f7e2408c6d0 2019-12-05T06:55:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.170+0000 7f7e2892e700 1 -- 172.21.15.16:0/3497631809 >> v1:172.21.15.76:6789/0 conn(0x7f7e2408c6d0 legacy=0x7f7e2408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.170+0000 7f7e2892e700 1 -- 172.21.15.16:0/3497631809 >> v1:172.21.15.16:6789/0 conn(0x7f7e2408fef0 legacy=0x7f7e2408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:39.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.170+0000 7f7e2892e700 1 -- 172.21.15.16:0/3497631809 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2408e620 con 0x7f7e2408d2c0 2019-12-05T06:55:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.171+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3497631809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 0x7f7e2408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7e1800af50 tx=0x7f7e1800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.171+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3497631809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 0x7f7e2408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7e1800af50 tx=0x7f7e1800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.171+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3497631809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 0x7f7e2408ef20 secure :-1 s=READY pgs=376 cs=0 l=1 rx=0x7f7e1800af50 tx=0x7f7e1800b760).ready entity=mon.2 client_cookie=5950f386be05dc3d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.171+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3497631809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 0x7f7e2408ef20 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f7e1800af50 tx=0x7f7e1800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.171+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3497631809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 0x7f7e2408ef20 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f7e1800af50 tx=0x7f7e1800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.171+0000 7f7e23fff700 1 -- 172.21.15.16:0/3497631809 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e18000960 con 0x7f7e2408d2c0 2019-12-05T06:55:39.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.171+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3497631809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 0x7f7e2408ef20 secure :-1 s=THROTTLE_DONE pgs=376 cs=0 l=1 rx=0x7f7e1800af50 tx=0x7f7e1800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.171+0000 7f7e23fff700 1 -- 172.21.15.16:0/3497631809 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e180098e0 con 0x7f7e2408d2c0 2019-12-05T06:55:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.171+0000 7f7e23fff700 1 -- 172.21.15.16:0/3497631809 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e18009eb0 con 0x7f7e2408d2c0 2019-12-05T06:55:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.172+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3497631809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 msgr2=0x7f7e2408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.172+0000 7f7e2ab92700 1 --2- 172.21.15.16:0/3497631809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 0x7f7e2408ef20 secure :-1 s=READY pgs=376 cs=0 l=1 rx=0x7f7e1800af50 tx=0x7f7e1800b760).stop 2019-12-05T06:55:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.173+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3497631809 shutdown_connections 2019-12-05T06:55:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.173+0000 7f7e2ab92700 1 --2- 172.21.15.16:0/3497631809 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7e2408d2c0 0x7f7e2408ef20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:39.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.173+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3497631809 shutdown_connections 2019-12-05T06:55:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.174+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3497631809 wait complete. 2019-12-05T06:55:39.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.174+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3497631809 >> 172.21.15.16:0/3497631809 conn(0x7f7e24061940 msgr2=0x7f7e24096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.179+0000 7f7e2ab92700 1 Processor -- start 2019-12-05T06:55:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.180+0000 7f7e2ab92700 1 -- start start 2019-12-05T06:55:39.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.182+0000 7f7e2ab92700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7e2414d680 0x7f7e2414da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.182+0000 7f7e2ab92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e2409dc10 0x7f7e2409d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.182+0000 7f7e2ab92700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e2408a840 con 0x7f7e2412bce0 2019-12-05T06:55:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.182+0000 7f7e2ab92700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e2412ad50 con 0x7f7e2409dc10 2019-12-05T06:55:39.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.182+0000 7f7e2ab92700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7e24099bb0 con 0x7f7e2414d680 2019-12-05T06:55:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.182+0000 7f7e2892e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e2409dc10 0x7f7e2409d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:39.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.182+0000 7f7e2892e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f7e2409dc10 0x7f7e2409d3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35942/0 (socket says 172.21.15.16:35942) 2019-12-05T06:55:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.182+0000 7f7e2892e700 1 -- 172.21.15.16:0/3489579505 learned_addr learned my addr 172.21.15.16:0/3489579505 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:55:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.182+0000 7f7e2912f700 1 --2- 172.21.15.16:0/3489579505 >> v2:172.21.15.76:3300/0 conn(0x7f7e2414d680 0x7f7e2414da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e2892e700 1 -- 172.21.15.16:0/3489579505 >> v2:172.21.15.76:3300/0 conn(0x7f7e2414d680 msgr2=0x7f7e2414da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> v2:172.21.15.76:3300/0 conn(0x7f7e2414d680 0x7f7e2414da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e2892e700 1 -- 172.21.15.16:0/3489579505 >> v1:172.21.15.16:6789/0 conn(0x7f7e2412bce0 legacy=0x7f7e24098180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:39.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e2892e700 1 -- 172.21.15.16:0/3489579505 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e2409d8e0 con 0x7f7e2409dc10 2019-12-05T06:55:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> v2:172.21.15.16:3300/0 conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> v2:172.21.15.16:3300/0 conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=READY pgs=1580 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=THROTTLE_DONE pgs=1580 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=THROTTLE_DONE pgs=1580 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e21ffb700 1 -- 172.21.15.16:0/3489579505 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e18000940 con 0x7f7e2409dc10 2019-12-05T06:55:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.183+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=THROTTLE_DONE pgs=1580 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.184+0000 7f7e21ffb700 1 -- 172.21.15.16:0/3489579505 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e18009980 con 0x7f7e2409dc10 2019-12-05T06:55:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.184+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3489579505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e2409d8e0 con 0x7f7e2409dc10 2019-12-05T06:55:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.184+0000 7f7e21ffb700 1 -- 172.21.15.16:0/3489579505 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e1800a0d0 con 0x7f7e2409dc10 2019-12-05T06:55:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.184+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3489579505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e2409d8e0 con 0x7f7e2409dc10 2019-12-05T06:55:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.185+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=THROTTLE_DONE pgs=1580 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.186+0000 7f7e21ffb700 1 -- 172.21.15.16:0/3489579505 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7e18009b70 con 0x7f7e2409dc10 2019-12-05T06:55:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.186+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=THROTTLE_DONE pgs=1580 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.186+0000 7f7e21ffb700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e04023630 0x7f7e04025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.186+0000 7f7e21ffb700 1 -- 172.21.15.16:0/3489579505 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f7e18045df0 con 0x7f7e2409dc10 2019-12-05T06:55:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.186+0000 7f7e2912f700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e04023630 0x7f7e04025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:39.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.187+0000 7f7e2912f700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e04023630 0x7f7e04025ac0 crc :-1 s=READY pgs=660 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:39.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.198+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3489579505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e2412b3e0 con 0x7f7e2409dc10 2019-12-05T06:55:39.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.207+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=THROTTLE_DONE pgs=1580 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.207+0000 7f7e21ffb700 1 -- 172.21.15.16:0/3489579505 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7e18009580 con 0x7f7e2409dc10 2019-12-05T06:55:39.461 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.460+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3489579505 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7e2412b3e0 con 0x7f7e2409dc10 2019-12-05T06:55:39.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.461+0000 7f7e2892e700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=THROTTLE_DONE pgs=1580 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:39.462 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.461+0000 7f7e21ffb700 1 -- 172.21.15.16:0/3489579505 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f7e18009b70 con 0x7f7e2409dc10 2019-12-05T06:55:39.464 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 192 2019-12-05T06:55:39.464 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:39.465 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:38.705960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:active","state_seq":14,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:39.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.466+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3489579505 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e04023630 msgr2=0x7f7e04025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:39.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.466+0000 7f7e2ab92700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e04023630 0x7f7e04025ac0 crc :-1 s=READY pgs=660 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:39.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.466+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 msgr2=0x7f7e2409d3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:39.467 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.466+0000 7f7e2ab92700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 0x7f7e2409d3a0 secure :-1 s=READY pgs=1580 cs=0 l=1 rx=0x7f7e18010450 tx=0x7f7e18010280).stop 2019-12-05T06:55:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.466+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3489579505 shutdown_connections 2019-12-05T06:55:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.466+0000 7f7e2ab92700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7e04023630 0x7f7e04025ac0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:39.468 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.466+0000 7f7e2ab92700 1 --2- 172.21.15.16:0/3489579505 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f7e2409dc10 0x7f7e2409d3a0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:39.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.466+0000 7f7e2ab92700 1 --2- 172.21.15.16:0/3489579505 >> v2:172.21.15.76:3300/0 conn(0x7f7e2414d680 0x7f7e2414da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:39.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.467+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3489579505 shutdown_connections 2019-12-05T06:55:39.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.467+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3489579505 wait complete. 2019-12-05T06:55:39.469 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:39.467+0000 7f7e2ab92700 1 -- 172.21.15.16:0/3489579505 >> 172.21.15.16:0/3489579505 conn(0x7f7e2406d260 msgr2=0x7f7e2409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:39.478 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:55:41.479 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.732+0000 7f3c1c26f700 1 Processor -- start 2019-12-05T06:55:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.732+0000 7f3c1c26f700 1 -- start start 2019-12-05T06:55:41.742 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.734+0000 7f3c1c26f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3c1408c6d0 0x7f3c1408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.734+0000 7f3c1c26f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1408fb80 con 0x7f3c1408f720 2019-12-05T06:55:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.734+0000 7f3c1c26f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c140958e0 con 0x7f3c1408fef0 2019-12-05T06:55:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.734+0000 7f3c1c26f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1408df90 con 0x7f3c1408c6d0 2019-12-05T06:55:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.734+0000 7f3c1b00d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3c1408f720 0x7f3c141533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49354/0 (socket says 172.21.15.16:49354) 2019-12-05T06:55:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.734+0000 7f3c1b00d700 1 -- 172.21.15.16:0/1055264286 learned_addr learned my addr 172.21.15.16:0/1055264286 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:41.743 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.734+0000 7f3c1a80c700 1 --2- 172.21.15.16:0/1055264286 >> v2:172.21.15.76:3300/0 conn(0x7f3c1408c6d0 0x7f3c1408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.735+0000 7f3c1980a700 1 -- 172.21.15.16:0/1055264286 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724084325 0 0) 0x7f3c0c0014d0 con 0x7f3c1408f720 2019-12-05T06:55:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.735+0000 7f3c1980a700 1 -- 172.21.15.16:0/1055264286 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c00002e80 con 0x7f3c1408f720 2019-12-05T06:55:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.735+0000 7f3c1980a700 1 -- 172.21.15.16:0/1055264286 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674076314 0 0) 0x7f3c04001080 con 0x7f3c1408fef0 2019-12-05T06:55:41.744 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.735+0000 7f3c1980a700 1 -- 172.21.15.16:0/1055264286 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c000030b0 con 0x7f3c1408fef0 2019-12-05T06:55:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.735+0000 7f3c1a80c700 1 -- 172.21.15.16:0/1055264286 >> v1:172.21.15.76:6789/0 conn(0x7f3c1408fef0 legacy=0x7f3c1408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.735+0000 7f3c1a80c700 1 -- 172.21.15.16:0/1055264286 >> v1:172.21.15.16:6789/0 conn(0x7f3c1408f720 legacy=0x7f3c141533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.735+0000 7f3c1980a700 1 -- 172.21.15.16:0/1055264286 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2382833747 0 0) 0x7f3c0c0014b0 con 0x7f3c1408f720 2019-12-05T06:55:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.735+0000 7f3c1a80c700 1 -- 172.21.15.16:0/1055264286 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1408e620 con 0x7f3c1408c6d0 2019-12-05T06:55:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.735+0000 7f3c1a80c700 1 --2- 172.21.15.16:0/1055264286 >> v2:172.21.15.76:3300/0 conn(0x7f3c1408c6d0 0x7f3c1408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3c1000af50 tx=0x7f3c1000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.736+0000 7f3c1a80c700 1 --2- 172.21.15.16:0/1055264286 >> v2:172.21.15.76:3300/0 conn(0x7f3c1408c6d0 0x7f3c1408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3c1000af50 tx=0x7f3c1000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.745 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.736+0000 7f3c1a80c700 1 --2- 172.21.15.16:0/1055264286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c1408c6d0 0x7f3c1408cac0 secure :-1 s=READY pgs=1065 cs=0 l=1 rx=0x7f3c1000af50 tx=0x7f3c1000b760).ready entity=mon.1 client_cookie=643a966373e61251 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.736+0000 7f3c1a80c700 1 --2- 172.21.15.16:0/1055264286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c1408c6d0 0x7f3c1408cac0 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f3c1000af50 tx=0x7f3c1000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.736+0000 7f3c1980a700 1 -- 172.21.15.16:0/1055264286 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c10000960 con 0x7f3c1408c6d0 2019-12-05T06:55:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.736+0000 7f3c1a80c700 1 --2- 172.21.15.16:0/1055264286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c1408c6d0 0x7f3c1408cac0 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f3c1000af50 tx=0x7f3c1000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.746 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.736+0000 7f3c1a80c700 1 --2- 172.21.15.16:0/1055264286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c1408c6d0 0x7f3c1408cac0 secure :-1 s=THROTTLE_DONE pgs=1065 cs=0 l=1 rx=0x7f3c1000af50 tx=0x7f3c1000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.736+0000 7f3c1980a700 1 -- 172.21.15.16:0/1055264286 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c10009860 con 0x7f3c1408c6d0 2019-12-05T06:55:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.736+0000 7f3c1980a700 1 -- 172.21.15.16:0/1055264286 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c10000960 con 0x7f3c1408c6d0 2019-12-05T06:55:41.747 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.736+0000 7f3c1c26f700 1 -- 172.21.15.16:0/1055264286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c1408c6d0 msgr2=0x7f3c1408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.736+0000 7f3c1c26f700 1 --2- 172.21.15.16:0/1055264286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c1408c6d0 0x7f3c1408cac0 secure :-1 s=READY pgs=1065 cs=0 l=1 rx=0x7f3c1000af50 tx=0x7f3c1000b760).stop 2019-12-05T06:55:41.748 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.737+0000 7f3c1c26f700 1 -- 172.21.15.16:0/1055264286 shutdown_connections 2019-12-05T06:55:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.737+0000 7f3c1c26f700 1 --2- 172.21.15.16:0/1055264286 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3c1408c6d0 0x7f3c1408cac0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.738+0000 7f3c1c26f700 1 -- 172.21.15.16:0/1055264286 shutdown_connections 2019-12-05T06:55:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.738+0000 7f3c1c26f700 1 -- 172.21.15.16:0/1055264286 wait complete. 2019-12-05T06:55:41.749 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.738+0000 7f3c1c26f700 1 -- 172.21.15.16:0/1055264286 >> 172.21.15.16:0/1055264286 conn(0x7f3c14061940 msgr2=0x7f3c14096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.743+0000 7f3c1c26f700 1 Processor -- start 2019-12-05T06:55:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.743+0000 7f3c1c26f700 1 -- start start 2019-12-05T06:55:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.744+0000 7f3c1c26f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c1412b640 0x7f3c1412ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.744+0000 7f3c1c26f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3c14153ba0 0x7f3c14155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:41.753 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.744+0000 7f3c1c26f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c14131bf0 con 0x7f3c141302f0 2019-12-05T06:55:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.744+0000 7f3c1c26f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c141296e0 con 0x7f3c1412b640 2019-12-05T06:55:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.744+0000 7f3c1c26f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c14128cb0 con 0x7f3c14153ba0 2019-12-05T06:55:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.745+0000 7f3c1a80c700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f3c141302f0 0x7f3c1412a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:42428/0 (socket says 172.21.15.16:42428) 2019-12-05T06:55:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.745+0000 7f3c1a80c700 1 -- 172.21.15.16:0/2294234889 learned_addr learned my addr 172.21.15.16:0/2294234889 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:41.754 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.745+0000 7f3c1a00b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3c1412b640 0x7f3c1412ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.745+0000 7f3c1b00d700 1 --2- 172.21.15.16:0/2294234889 >> v2:172.21.15.76:3300/0 conn(0x7f3c14153ba0 0x7f3c14155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.745+0000 7f3c0b7fe700 1 -- 172.21.15.16:0/2294234889 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2383343841 0 0) 0x7f3c1000ac20 con 0x7f3c141302f0 2019-12-05T06:55:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.745+0000 7f3c0b7fe700 1 -- 172.21.15.16:0/2294234889 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3bfc002e80 con 0x7f3c141302f0 2019-12-05T06:55:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.745+0000 7f3c1a00b700 1 -- 172.21.15.16:0/2294234889 >> v2:172.21.15.76:3300/0 conn(0x7f3c14153ba0 msgr2=0x7f3c14155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.745+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> v2:172.21.15.76:3300/0 conn(0x7f3c14153ba0 0x7f3c14155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.745+0000 7f3c1a00b700 1 -- 172.21.15.16:0/2294234889 >> v1:172.21.15.76:6789/0 conn(0x7f3c141302f0 legacy=0x7f3c1412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:41.755 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.745+0000 7f3c1a00b700 1 -- 172.21.15.16:0/2294234889 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c14129d70 con 0x7f3c1412b640 2019-12-05T06:55:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> v2:172.21.15.16:3300/0 conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> v2:172.21.15.16:3300/0 conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=READY pgs=1584 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c0b7fe700 1 -- 172.21.15.16:0/2294234889 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c04000aa0 con 0x7f3c1412b640 2019-12-05T06:55:41.756 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c0b7fe700 1 -- 172.21.15.16:0/2294234889 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0400efd0 con 0x7f3c1412b640 2019-12-05T06:55:41.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c1c26f700 1 -- 172.21.15.16:0/2294234889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c14129d70 con 0x7f3c1412b640 2019-12-05T06:55:41.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c0b7fe700 1 -- 172.21.15.16:0/2294234889 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c0400f6e0 con 0x7f3c1412b640 2019-12-05T06:55:41.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.746+0000 7f3c1c26f700 1 -- 172.21.15.16:0/2294234889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c14129d70 con 0x7f3c1412b640 2019-12-05T06:55:41.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.747+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.757 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.748+0000 7f3c0b7fe700 1 -- 172.21.15.16:0/2294234889 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3c0400eca0 con 0x7f3c1412b640 2019-12-05T06:55:41.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.748+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.749+0000 7f3c0b7fe700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3bfc023670 0x7f3bfc025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:41.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.749+0000 7f3c0b7fe700 1 -- 172.21.15.16:0/2294234889 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7f3c04045d60 con 0x7f3c1412b640 2019-12-05T06:55:41.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.749+0000 7f3c1a80c700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3bfc023670 0x7f3bfc025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:41.758 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.749+0000 7f3c1a80c700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3bfc023670 0x7f3bfc025ae0 crc :-1 s=READY pgs=661 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:41.764 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.763+0000 7f3c1c26f700 1 -- 172.21.15.16:0/2294234889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c14129d70 con 0x7f3c1412b640 2019-12-05T06:55:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.770+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:41.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:41.770+0000 7f3c0b7fe700 1 -- 172.21.15.16:0/2294234889 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3c0400ea70 con 0x7f3c1412b640 2019-12-05T06:55:42.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.012+0000 7f3c1c26f700 1 -- 172.21.15.16:0/2294234889 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c14129d70 con 0x7f3c1412b640 2019-12-05T06:55:42.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.013+0000 7f3c1a00b700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=THROTTLE_DONE pgs=1584 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.013+0000 7f3c0b7fe700 1 -- 172.21.15.16:0/2294234889 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f3c0400ec80 con 0x7f3c1412b640 2019-12-05T06:55:42.015 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 192 2019-12-05T06:55:42.015 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:42.015 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:38.705960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:active","state_seq":14,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.015+0000 7f3c1c26f700 1 -- 172.21.15.16:0/2294234889 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3bfc023670 msgr2=0x7f3bfc025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.015+0000 7f3c1c26f700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3bfc023670 0x7f3bfc025ae0 crc :-1 s=READY pgs=661 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:42.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.015+0000 7f3c1c26f700 1 -- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 msgr2=0x7f3c1412ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.015+0000 7f3c1c26f700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 0x7f3c1412ae70 secure :-1 s=READY pgs=1584 cs=0 l=1 rx=0x7f3c040103c0 tx=0x7f3c0400b300).stop 2019-12-05T06:55:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.016+0000 7f3c1c26f700 1 -- 172.21.15.16:0/2294234889 shutdown_connections 2019-12-05T06:55:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.016+0000 7f3c1c26f700 1 --2- 172.21.15.16:0/2294234889 >> v2:172.21.15.76:3300/0 conn(0x7f3c14153ba0 0x7f3c14155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:42.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.016+0000 7f3c1c26f700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3bfc023670 0x7f3bfc025ae0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.016+0000 7f3c1c26f700 1 --2- 172.21.15.16:0/2294234889 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3c1412b640 0x7f3c1412ae70 unknown :-1 s=CLOSED pgs=1584 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.016+0000 7f3c1c26f700 1 -- 172.21.15.16:0/2294234889 shutdown_connections 2019-12-05T06:55:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.016+0000 7f3c1c26f700 1 -- 172.21.15.16:0/2294234889 wait complete. 2019-12-05T06:55:42.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:42.016+0000 7f3c1c26f700 1 -- 172.21.15.16:0/2294234889 >> 172.21.15.16:0/2294234889 conn(0x7f3c140a2850 msgr2=0x7f3c1406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:42.028 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:55:44.030 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.310+0000 7fe516a9a700 1 Processor -- start 2019-12-05T06:55:44.321 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.311+0000 7fe516a9a700 1 -- start start 2019-12-05T06:55:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.313+0000 7fe516a9a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe51008c6d0 0x7fe51008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.313+0000 7fe516a9a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe51008f720 0x7fe51008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.313+0000 7fe516a9a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5100958e0 con 0x7fe51008fef0 2019-12-05T06:55:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.313+0000 7fe516a9a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5100a3f70 con 0x7fe51008f720 2019-12-05T06:55:44.322 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.313+0000 7fe516a9a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51008d830 con 0x7fe51008c6d0 2019-12-05T06:55:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.313+0000 7fe515838700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fe51008f720 0x7fe51008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.313+0000 7fe514836700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe51008fef0 0x7fe51008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49368/0 (socket says 172.21.15.16:49368) 2019-12-05T06:55:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.313+0000 7fe515037700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe51008c6d0 0x7fe51008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.313+0000 7fe514836700 1 -- 172.21.15.16:0/3163412582 learned_addr learned my addr 172.21.15.16:0/3163412582 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.314+0000 7fe515838700 1 -- 172.21.15.16:0/3163412582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe51008c6d0 msgr2=0x7fe51008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:55:44.323 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.314+0000 7fe515838700 1 --2- 172.21.15.16:0/3163412582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe51008c6d0 0x7fe51008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.314+0000 7fe515838700 1 -- 172.21.15.16:0/3163412582 >> v1:172.21.15.16:6789/0 conn(0x7fe51008fef0 legacy=0x7fe51008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.314+0000 7fe507fff700 1 -- 172.21.15.16:0/3163412582 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 909418203 0 0) 0x7fe500001150 con 0x7fe51008fef0 2019-12-05T06:55:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.314+0000 7fe515838700 1 -- 172.21.15.16:0/3163412582 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51008fbc0 con 0x7fe51008f720 2019-12-05T06:55:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.314+0000 7fe515037700 1 --2- 172.21.15.16:0/3163412582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe51008c6d0 0x7fe51008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:55:44.324 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.314+0000 7fe515838700 1 --2- 172.21.15.16:0/3163412582 >> v2:172.21.15.16:3300/0 conn(0x7fe51008f720 0x7fe51008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fe50800b2d0 tx=0x7fe50800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.314+0000 7fe515838700 1 --2- 172.21.15.16:0/3163412582 >> v2:172.21.15.16:3300/0 conn(0x7fe51008f720 0x7fe51008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fe50800b2d0 tx=0x7fe50800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.314+0000 7fe515838700 1 --2- 172.21.15.16:0/3163412582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51008f720 0x7fe51008d000 secure :-1 s=READY pgs=1586 cs=0 l=1 rx=0x7fe50800b2d0 tx=0x7fe50800bae0).ready entity=mon.0 client_cookie=7cbb6517f2eb1b5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.314+0000 7fe515838700 1 --2- 172.21.15.16:0/3163412582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51008f720 0x7fe51008d000 secure :-1 s=THROTTLE_DONE pgs=1586 cs=0 l=1 rx=0x7fe50800b2d0 tx=0x7fe50800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.315+0000 7fe515838700 1 --2- 172.21.15.16:0/3163412582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51008f720 0x7fe51008d000 secure :-1 s=THROTTLE_DONE pgs=1586 cs=0 l=1 rx=0x7fe50800b2d0 tx=0x7fe50800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.315+0000 7fe507fff700 1 -- 172.21.15.16:0/3163412582 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe508009f20 con 0x7fe51008f720 2019-12-05T06:55:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.315+0000 7fe515838700 1 --2- 172.21.15.16:0/3163412582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51008f720 0x7fe51008d000 secure :-1 s=THROTTLE_DONE pgs=1586 cs=0 l=1 rx=0x7fe50800b2d0 tx=0x7fe50800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.325 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.315+0000 7fe507fff700 1 -- 172.21.15.16:0/3163412582 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe508009b40 con 0x7fe51008f720 2019-12-05T06:55:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.315+0000 7fe507fff700 1 -- 172.21.15.16:0/3163412582 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe508000f40 con 0x7fe51008f720 2019-12-05T06:55:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.315+0000 7fe516a9a700 1 -- 172.21.15.16:0/3163412582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51008f720 msgr2=0x7fe51008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.315+0000 7fe516a9a700 1 --2- 172.21.15.16:0/3163412582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51008f720 0x7fe51008d000 secure :-1 s=READY pgs=1586 cs=0 l=1 rx=0x7fe50800b2d0 tx=0x7fe50800bae0).stop 2019-12-05T06:55:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.316+0000 7fe516a9a700 1 -- 172.21.15.16:0/3163412582 shutdown_connections 2019-12-05T06:55:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.316+0000 7fe516a9a700 1 --2- 172.21.15.16:0/3163412582 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fe51008f720 0x7fe51008d000 unknown :-1 s=CLOSED pgs=1586 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.316+0000 7fe516a9a700 1 --2- 172.21.15.16:0/3163412582 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe51008c6d0 0x7fe51008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:44.326 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.317+0000 7fe516a9a700 1 -- 172.21.15.16:0/3163412582 shutdown_connections 2019-12-05T06:55:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.317+0000 7fe516a9a700 1 -- 172.21.15.16:0/3163412582 wait complete. 2019-12-05T06:55:44.327 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.317+0000 7fe516a9a700 1 -- 172.21.15.16:0/3163412582 >> 172.21.15.16:0/3163412582 conn(0x7fe510061940 msgr2=0x7fe510096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:44.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.322+0000 7fe516a9a700 1 Processor -- start 2019-12-05T06:55:44.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.322+0000 7fe516a9a700 1 -- start start 2019-12-05T06:55:44.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.324+0000 7fe516a9a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:44.330 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.324+0000 7fe516a9a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe510084b90 con 0x7fe51009bbe0 2019-12-05T06:55:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.324+0000 7fe516a9a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe510098790 con 0x7fe510096e90 2019-12-05T06:55:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.324+0000 7fe516a9a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe51009a450 con 0x7fe510084f30 2019-12-05T06:55:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.324+0000 7fe514836700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fe51009bbe0 0x7fe5101539e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49374/0 (socket says 172.21.15.16:49374) 2019-12-05T06:55:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.324+0000 7fe514836700 1 -- 172.21.15.16:0/529564369 learned_addr learned my addr 172.21.15.16:0/529564369 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.324+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:44.331 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.325+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 92282561 0 0) 0x7fe500001150 con 0x7fe51009bbe0 2019-12-05T06:55:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.325+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f0002e80 con 0x7fe51009bbe0 2019-12-05T06:55:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.325+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077925696 0 0) 0x7fe50c000f30 con 0x7fe510096e90 2019-12-05T06:55:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.325+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe4f0004220 con 0x7fe510096e90 2019-12-05T06:55:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.325+0000 7fe515838700 1 -- 172.21.15.16:0/529564369 >> v1:172.21.15.76:6789/0 conn(0x7fe510096e90 legacy=0x7fe51009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.325+0000 7fe515838700 1 -- 172.21.15.16:0/529564369 >> v1:172.21.15.16:6789/0 conn(0x7fe51009bbe0 legacy=0x7fe5101539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:44.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.325+0000 7fe515838700 1 -- 172.21.15.16:0/529564369 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51009aae0 con 0x7fe510084f30 2019-12-05T06:55:44.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.325+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.325+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.326+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=READY pgs=377 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:44.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.326+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.326+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe508000cc0 con 0x7fe510084f30 2019-12-05T06:55:44.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.326+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.326+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.326+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe508000cc0 con 0x7fe510084f30 2019-12-05T06:55:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.326+0000 7fe516a9a700 1 -- 172.21.15.16:0/529564369 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe51009aae0 con 0x7fe510084f30 2019-12-05T06:55:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.326+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe508009ef0 con 0x7fe510084f30 2019-12-05T06:55:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.326+0000 7fe516a9a700 1 -- 172.21.15.16:0/529564369 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe51009aae0 con 0x7fe510084f30 2019-12-05T06:55:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.327+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.328+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fe50800a690 con 0x7fe510084f30 2019-12-05T06:55:44.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.328+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.328+0000 7fe505ffb700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4f0023650 0x7fe4f0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.328+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7fe508046160 con 0x7fe510084f30 2019-12-05T06:55:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.329+0000 7fe515037700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4f0023650 0x7fe4f0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:44.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.329+0000 7fe515037700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4f0023650 0x7fe4f0025ae0 crc :-1 s=READY pgs=662 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:44.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.342+0000 7fe516a9a700 1 -- 172.21.15.16:0/529564369 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe51009aae0 con 0x7fe510084f30 2019-12-05T06:55:44.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.352+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.353 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.352+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fe50800a480 con 0x7fe510084f30 2019-12-05T06:55:44.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.607+0000 7fe516a9a700 1 -- 172.21.15.16:0/529564369 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe51009aae0 con 0x7fe510084f30 2019-12-05T06:55:44.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.609+0000 7fe515838700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=THROTTLE_DONE pgs=377 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:44.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.609+0000 7fe505ffb700 1 -- 172.21.15.16:0/529564369 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fe50800a690 con 0x7fe510084f30 2019-12-05T06:55:44.611 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 192 2019-12-05T06:55:44.611 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:44.612 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:38.705960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:active","state_seq":14,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:44.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.612+0000 7fe516a9a700 1 -- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4f0023650 msgr2=0x7fe4f0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:44.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.612+0000 7fe516a9a700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4f0023650 0x7fe4f0025ae0 crc :-1 s=READY pgs=662 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:44.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.612+0000 7fe516a9a700 1 -- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 msgr2=0x7fe510157200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:44.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.612+0000 7fe516a9a700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 secure :-1 s=READY pgs=377 cs=0 l=1 rx=0x7fe508010750 tx=0x7fe508010af0).stop 2019-12-05T06:55:44.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.613+0000 7fe516a9a700 1 -- 172.21.15.16:0/529564369 shutdown_connections 2019-12-05T06:55:44.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.613+0000 7fe516a9a700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fe4f0023650 0x7fe4f0025ae0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:44.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.613+0000 7fe516a9a700 1 --2- 172.21.15.16:0/529564369 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fe510084f30 0x7fe510157200 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:44.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.613+0000 7fe516a9a700 1 -- 172.21.15.16:0/529564369 shutdown_connections 2019-12-05T06:55:44.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.613+0000 7fe516a9a700 1 -- 172.21.15.16:0/529564369 wait complete. 2019-12-05T06:55:44.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:44.613+0000 7fe516a9a700 1 -- 172.21.15.16:0/529564369 >> 172.21.15.16:0/529564369 conn(0x7fe5100714f0 msgr2=0x7fe51009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:44.624 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:55:46.625 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:46.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.900+0000 7ff83fecb700 1 Processor -- start 2019-12-05T06:55:46.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.900+0000 7ff83fecb700 1 -- start start 2019-12-05T06:55:46.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83fecb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83808c6d0 0x7ff83808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:46.911 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83fecb700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff83808fef0 0x7ff83808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:46.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83fecb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff83808ef20 0x7ff83808f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:46.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83fecb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8380a3f70 con 0x7ff83808ef20 2019-12-05T06:55:46.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83fecb700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff83808d830 con 0x7ff83808fef0 2019-12-05T06:55:46.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83fecb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83808d540 con 0x7ff83808c6d0 2019-12-05T06:55:46.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83ec69700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff83808ef20 0x7ff83808f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:46.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83ec69700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff83808ef20 0x7ff83808f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:35986/0 (socket says 172.21.15.16:35986) 2019-12-05T06:55:46.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83ec69700 1 -- 172.21.15.16:0/1014177443 learned_addr learned my addr 172.21.15.16:0/1014177443 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:55:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83dc67700 1 --2- 172.21.15.16:0/1014177443 >> v2:172.21.15.76:3300/0 conn(0x7ff83808fef0 0x7ff83808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.902+0000 7ff83e468700 1 --2- 172.21.15.16:0/1014177443 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83808c6d0 0x7ff83808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.903+0000 7ff83ec69700 1 -- 172.21.15.16:0/1014177443 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83808c6d0 msgr2=0x7ff83808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:55:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.903+0000 7ff83ec69700 1 --2- 172.21.15.16:0/1014177443 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83808c6d0 0x7ff83808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.903+0000 7ff83ec69700 1 -- 172.21.15.16:0/1014177443 >> v2:172.21.15.76:3300/0 conn(0x7ff83808fef0 msgr2=0x7ff83808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:55:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.903+0000 7ff83ec69700 1 --2- 172.21.15.16:0/1014177443 >> v2:172.21.15.76:3300/0 conn(0x7ff83808fef0 0x7ff83808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:46.913 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.903+0000 7ff83ec69700 1 -- 172.21.15.16:0/1014177443 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83808e620 con 0x7ff83808ef20 2019-12-05T06:55:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.903+0000 7ff83ec69700 1 --2- 172.21.15.16:0/1014177443 >> v2:172.21.15.16:3300/0 conn(0x7ff83808ef20 0x7ff83808f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff83000fa30 tx=0x7ff8300080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.903+0000 7ff83ec69700 1 --2- 172.21.15.16:0/1014177443 >> v2:172.21.15.16:3300/0 conn(0x7ff83808ef20 0x7ff83808f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff83000fa30 tx=0x7ff8300080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.903+0000 7ff83ec69700 1 --2- 172.21.15.16:0/1014177443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83808ef20 0x7ff83808f3c0 secure :-1 s=READY pgs=1588 cs=0 l=1 rx=0x7ff83000fa30 tx=0x7ff8300080d0).ready entity=mon.0 client_cookie=b45fd136abc42376 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.904+0000 7ff83ec69700 1 --2- 172.21.15.16:0/1014177443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83808ef20 0x7ff83808f3c0 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7ff83000fa30 tx=0x7ff8300080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.904+0000 7ff83ec69700 1 --2- 172.21.15.16:0/1014177443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83808ef20 0x7ff83808f3c0 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7ff83000fa30 tx=0x7ff8300080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.914 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.904+0000 7ff83ec69700 1 --2- 172.21.15.16:0/1014177443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83808ef20 0x7ff83808f3c0 secure :-1 s=THROTTLE_DONE pgs=1588 cs=0 l=1 rx=0x7ff83000fa30 tx=0x7ff8300080d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.904+0000 7ff83d466700 1 -- 172.21.15.16:0/1014177443 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff830000df0 con 0x7ff83808ef20 2019-12-05T06:55:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.904+0000 7ff83d466700 1 -- 172.21.15.16:0/1014177443 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff830001460 con 0x7ff83808ef20 2019-12-05T06:55:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.904+0000 7ff83d466700 1 -- 172.21.15.16:0/1014177443 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff830001bb0 con 0x7ff83808ef20 2019-12-05T06:55:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.904+0000 7ff83fecb700 1 -- 172.21.15.16:0/1014177443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83808ef20 msgr2=0x7ff83808f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.904+0000 7ff83fecb700 1 --2- 172.21.15.16:0/1014177443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83808ef20 0x7ff83808f3c0 secure :-1 s=READY pgs=1588 cs=0 l=1 rx=0x7ff83000fa30 tx=0x7ff8300080d0).stop 2019-12-05T06:55:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.906+0000 7ff83fecb700 1 -- 172.21.15.16:0/1014177443 shutdown_connections 2019-12-05T06:55:46.915 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.906+0000 7ff83fecb700 1 --2- 172.21.15.16:0/1014177443 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff83808ef20 0x7ff83808f3c0 unknown :-1 s=CLOSED pgs=1588 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.906+0000 7ff83fecb700 1 --2- 172.21.15.16:0/1014177443 >> v2:172.21.15.76:3300/0 conn(0x7ff83808fef0 0x7ff83808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.906+0000 7ff83fecb700 1 --2- 172.21.15.16:0/1014177443 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83808c6d0 0x7ff83808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.906+0000 7ff83fecb700 1 -- 172.21.15.16:0/1014177443 shutdown_connections 2019-12-05T06:55:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.906+0000 7ff83fecb700 1 -- 172.21.15.16:0/1014177443 wait complete. 2019-12-05T06:55:46.916 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.906+0000 7ff83fecb700 1 -- 172.21.15.16:0/1014177443 >> 172.21.15.16:0/1014177443 conn(0x7ff838061940 msgr2=0x7ff838096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:46.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.912+0000 7ff83fecb700 1 Processor -- start 2019-12-05T06:55:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.912+0000 7ff83fecb700 1 -- start start 2019-12-05T06:55:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.914+0000 7ff83fecb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.914+0000 7ff83fecb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff838098da0 con 0x7ff838132d00 2019-12-05T06:55:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.914+0000 7ff83fecb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff83809f0a0 con 0x7ff838154b00 2019-12-05T06:55:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.914+0000 7ff83fecb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff83809a030 con 0x7ff83809b850 2019-12-05T06:55:46.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.914+0000 7ff83dc67700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff838132d00 0x7ff838132270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49390/0 (socket says 172.21.15.16:49390) 2019-12-05T06:55:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.914+0000 7ff83dc67700 1 -- 172.21.15.16:0/1724264745 learned_addr learned my addr 172.21.15.16:0/1724264745 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.915+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.915+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120671936 0 0) 0x7ff8340010b0 con 0x7ff838132d00 2019-12-05T06:55:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.915+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff820002e80 con 0x7ff838132d00 2019-12-05T06:55:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.915+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039707346 0 0) 0x7ff830008190 con 0x7ff838154b00 2019-12-05T06:55:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.915+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff820002e80 con 0x7ff838154b00 2019-12-05T06:55:46.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.915+0000 7ff83e468700 1 -- 172.21.15.16:0/1724264745 >> v1:172.21.15.76:6789/0 conn(0x7ff838154b00 legacy=0x7ff838156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:46.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.915+0000 7ff83e468700 1 -- 172.21.15.16:0/1724264745 >> v1:172.21.15.16:6789/0 conn(0x7ff838132d00 legacy=0x7ff838132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:46.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.915+0000 7ff83e468700 1 -- 172.21.15.16:0/1724264745 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff83809b0f0 con 0x7ff83809b850 2019-12-05T06:55:46.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=READY pgs=378 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:46.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff828000940 con 0x7ff83809b850 2019-12-05T06:55:46.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff828005580 con 0x7ff83809b850 2019-12-05T06:55:46.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff83fecb700 1 -- 172.21.15.16:0/1724264745 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff83809b0f0 con 0x7ff83809b850 2019-12-05T06:55:46.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff828005770 con 0x7ff83809b850 2019-12-05T06:55:46.923 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.916+0000 7ff83fecb700 1 -- 172.21.15.16:0/1724264745 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff83809b0f0 con 0x7ff83809b850 2019-12-05T06:55:46.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.917+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.918+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff828005180 con 0x7ff83809b850 2019-12-05T06:55:46.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.918+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.919+0000 7ff82effd700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff820023650 0x7ff820025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:46.924 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.919+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 8783+0+0 (secure 0 0 0) 0x7ff828045d90 con 0x7ff83809b850 2019-12-05T06:55:46.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.919+0000 7ff83dc67700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff820023650 0x7ff820025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:46.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.920+0000 7ff83dc67700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff820023650 0x7ff820025ae0 crc :-1 s=READY pgs=663 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:46.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.931+0000 7ff83fecb700 1 -- 172.21.15.16:0/1724264745 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff83809b0f0 con 0x7ff83809b850 2019-12-05T06:55:46.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.941+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:46.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:46.941+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff82802adb0 con 0x7ff83809b850 2019-12-05T06:55:47.004 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:55:47.009 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:55:47.198 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.197+0000 7ff83fecb700 1 -- 172.21.15.16:0/1724264745 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff83809b0f0 con 0x7ff83809b850 2019-12-05T06:55:47.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.199+0000 7ff83e468700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=THROTTLE_DONE pgs=378 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:47.199 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.199+0000 7ff82effd700 1 -- 172.21.15.16:0/1724264745 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 192 v192) v1 ==== 95+0+2112 (secure 0 0 0) 0x7ff828005180 con 0x7ff83809b850 2019-12-05T06:55:47.201 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:47.202 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":192,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6212,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":192}],"filesystems":[{"mdsmap":{"epoch":191,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:54:38.705960+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":81,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8106},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8106":{"gid":8106,"name":"a","rank":0,"incarnation":188,"state":"up:active","state_seq":14,"addr":"172.21.15.16:6833/4247952173","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":4247952173},{"type":"v1","addr":"172.21.15.16:6833","nonce":4247952173}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:47.202 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 192 2019-12-05T06:55:47.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.203+0000 7ff83fecb700 1 -- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff820023650 msgr2=0x7ff820025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.203+0000 7ff83fecb700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff820023650 0x7ff820025ae0 crc :-1 s=READY pgs=663 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.203+0000 7ff83fecb700 1 -- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 msgr2=0x7ff838131d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.203+0000 7ff83fecb700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 secure :-1 s=READY pgs=378 cs=0 l=1 rx=0x7ff8280103b0 tx=0x7ff828010750).stop 2019-12-05T06:55:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.203+0000 7ff83fecb700 1 -- 172.21.15.16:0/1724264745 shutdown_connections 2019-12-05T06:55:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.203+0000 7ff83fecb700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff820023650 0x7ff820025ae0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:47.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.203+0000 7ff83fecb700 1 --2- 172.21.15.16:0/1724264745 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff83809b850 0x7ff838131d30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.203+0000 7ff83fecb700 1 -- 172.21.15.16:0/1724264745 shutdown_connections 2019-12-05T06:55:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.203+0000 7ff83fecb700 1 -- 172.21.15.16:0/1724264745 wait complete. 2019-12-05T06:55:47.206 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:47.203+0000 7ff83fecb700 1 -- 172.21.15.16:0/1724264745 >> 172.21.15.16:0/1724264745 conn(0x7ff8380714f0 msgr2=0x7ff83809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:47.221 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:55:49.222 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:49.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.469+0000 7fd976731700 1 Processor -- start 2019-12-05T06:55:49.478 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.469+0000 7fd976731700 1 -- start start 2019-12-05T06:55:49.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.470+0000 7fd976731700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd97008c6d0 0x7fd97008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:49.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.470+0000 7fd976731700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97008f720 0x7fd97008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:49.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.470+0000 7fd976731700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97008fb70 con 0x7fd97008fef0 2019-12-05T06:55:49.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.470+0000 7fd976731700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd97008df90 con 0x7fd97008c6d0 2019-12-05T06:55:49.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.470+0000 7fd976731700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9700a3f70 con 0x7fd97008f720 2019-12-05T06:55:49.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.470+0000 7fd974cce700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd97008c6d0 0x7fd97008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:49.479 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.470+0000 7fd96ffff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd97008fef0 0x7fd97008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49400/0 (socket says 172.21.15.16:49400) 2019-12-05T06:55:49.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.470+0000 7fd96ffff700 1 -- 172.21.15.16:0/3824234906 learned_addr learned my addr 172.21.15.16:0/3824234906 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:49.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.471+0000 7fd9754cf700 1 --2- 172.21.15.16:0/3824234906 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97008f720 0x7fd97008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:49.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.471+0000 7fd96f7fe700 1 -- 172.21.15.16:0/3824234906 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 74444459 0 0) 0x7fd9680014d0 con 0x7fd97008fef0 2019-12-05T06:55:49.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.471+0000 7fd974cce700 1 -- 172.21.15.16:0/3824234906 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97008f720 msgr2=0x7fd97008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:55:49.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.471+0000 7fd974cce700 1 --2- 172.21.15.16:0/3824234906 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97008f720 0x7fd97008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:49.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.471+0000 7fd974cce700 1 -- 172.21.15.16:0/3824234906 >> v1:172.21.15.16:6789/0 conn(0x7fd97008fef0 legacy=0x7fd97008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:49.480 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.471+0000 7fd974cce700 1 -- 172.21.15.16:0/3824234906 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97008d540 con 0x7fd97008c6d0 2019-12-05T06:55:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.471+0000 7fd974cce700 1 --2- 172.21.15.16:0/3824234906 >> v2:172.21.15.16:3300/0 conn(0x7fd97008c6d0 0x7fd97008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd95800af50 tx=0x7fd95800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.471+0000 7fd974cce700 1 --2- 172.21.15.16:0/3824234906 >> v2:172.21.15.16:3300/0 conn(0x7fd97008c6d0 0x7fd97008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd95800af50 tx=0x7fd95800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.471+0000 7fd974cce700 1 --2- 172.21.15.16:0/3824234906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97008c6d0 0x7fd97008cac0 secure :-1 s=READY pgs=1593 cs=0 l=1 rx=0x7fd95800af50 tx=0x7fd95800b250).ready entity=mon.0 client_cookie=a86aea6b64df7f85 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.472+0000 7fd974cce700 1 --2- 172.21.15.16:0/3824234906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97008c6d0 0x7fd97008cac0 secure :-1 s=THROTTLE_DONE pgs=1593 cs=0 l=1 rx=0x7fd95800af50 tx=0x7fd95800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.472+0000 7fd96f7fe700 1 -- 172.21.15.16:0/3824234906 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd958007210 con 0x7fd97008c6d0 2019-12-05T06:55:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.472+0000 7fd974cce700 1 --2- 172.21.15.16:0/3824234906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97008c6d0 0x7fd97008cac0 secure :-1 s=THROTTLE_DONE pgs=1593 cs=0 l=1 rx=0x7fd95800af50 tx=0x7fd95800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.481 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.472+0000 7fd96f7fe700 1 -- 172.21.15.16:0/3824234906 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9580097c0 con 0x7fd97008c6d0 2019-12-05T06:55:49.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.472+0000 7fd974cce700 1 --2- 172.21.15.16:0/3824234906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97008c6d0 0x7fd97008cac0 secure :-1 s=THROTTLE_DONE pgs=1593 cs=0 l=1 rx=0x7fd95800af50 tx=0x7fd95800b250).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.472+0000 7fd96f7fe700 1 -- 172.21.15.16:0/3824234906 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9580099b0 con 0x7fd97008c6d0 2019-12-05T06:55:49.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.472+0000 7fd976731700 1 -- 172.21.15.16:0/3824234906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97008c6d0 msgr2=0x7fd97008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:49.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.472+0000 7fd976731700 1 --2- 172.21.15.16:0/3824234906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97008c6d0 0x7fd97008cac0 secure :-1 s=READY pgs=1593 cs=0 l=1 rx=0x7fd95800af50 tx=0x7fd95800b250).stop 2019-12-05T06:55:49.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.474+0000 7fd976731700 1 -- 172.21.15.16:0/3824234906 shutdown_connections 2019-12-05T06:55:49.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.474+0000 7fd976731700 1 --2- 172.21.15.16:0/3824234906 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97008f720 0x7fd97008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:49.482 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.474+0000 7fd976731700 1 --2- 172.21.15.16:0/3824234906 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd97008c6d0 0x7fd97008cac0 unknown :-1 s=CLOSED pgs=1593 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:49.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.474+0000 7fd976731700 1 -- 172.21.15.16:0/3824234906 shutdown_connections 2019-12-05T06:55:49.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.474+0000 7fd976731700 1 -- 172.21.15.16:0/3824234906 wait complete. 2019-12-05T06:55:49.483 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.474+0000 7fd976731700 1 -- 172.21.15.16:0/3824234906 >> 172.21.15.16:0/3824234906 conn(0x7fd970061940 msgr2=0x7fd970096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:49.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.478+0000 7fd976731700 1 Processor -- start 2019-12-05T06:55:49.486 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.479+0000 7fd976731700 1 -- start start 2019-12-05T06:55:49.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.480+0000 7fd976731700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:49.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.480+0000 7fd976731700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd970153160 con 0x7fd97009e580 2019-12-05T06:55:49.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.480+0000 7fd976731700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd97006e880 con 0x7fd97006e210 2019-12-05T06:55:49.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.480+0000 7fd976731700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd97008ada0 con 0x7fd97014c3a0 2019-12-05T06:55:49.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.480+0000 7fd9754cf700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd97009e580 0x7fd97019d410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49406/0 (socket says 172.21.15.16:49406) 2019-12-05T06:55:49.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.480+0000 7fd9754cf700 1 -- 172.21.15.16:0/542840990 learned_addr learned my addr 172.21.15.16:0/542840990 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:49.487 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.480+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.480+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 308980123 0 0) 0x7fd960002180 con 0x7fd97009e580 2019-12-05T06:55:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.480+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd954002e80 con 0x7fd97009e580 2019-12-05T06:55:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.480+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741900872 0 0) 0x7fd968001480 con 0x7fd97006e210 2019-12-05T06:55:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd9540030b0 con 0x7fd97006e210 2019-12-05T06:55:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd974cce700 1 -- 172.21.15.16:0/542840990 >> v1:172.21.15.76:6789/0 conn(0x7fd97006e210 legacy=0x7fd97009daf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd974cce700 1 -- 172.21.15.16:0/542840990 >> v1:172.21.15.16:6789/0 conn(0x7fd97009e580 legacy=0x7fd97019d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:49.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd974cce700 1 -- 172.21.15.16:0/542840990 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97009ccb0 con 0x7fd97014c3a0 2019-12-05T06:55:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=READY pgs=380 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd958000960 con 0x7fd97014c3a0 2019-12-05T06:55:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd958009790 con 0x7fd97014c3a0 2019-12-05T06:55:49.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd976731700 1 -- 172.21.15.16:0/542840990 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd97009ccb0 con 0x7fd97014c3a0 2019-12-05T06:55:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd958000960 con 0x7fd97014c3a0 2019-12-05T06:55:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.481+0000 7fd976731700 1 -- 172.21.15.16:0/542840990 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd97009ccb0 con 0x7fd97014c3a0 2019-12-05T06:55:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.482+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.483+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd958009980 con 0x7fd97014c3a0 2019-12-05T06:55:49.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.483+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.483+0000 7fd96d7fa700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd954023650 0x7fd954025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.483+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fd958046050 con 0x7fd97014c3a0 2019-12-05T06:55:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.484+0000 7fd96ffff700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd954023650 0x7fd954025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:49.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.484+0000 7fd96ffff700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd954023650 0x7fd954025ae0 crc :-1 s=READY pgs=664 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:49.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.499+0000 7fd976731700 1 -- 172.21.15.16:0/542840990 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd97009ccb0 con 0x7fd97014c3a0 2019-12-05T06:55:49.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.505+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.505+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd958009580 con 0x7fd97014c3a0 2019-12-05T06:55:49.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.701+0000 7fd976731700 1 -- 172.21.15.16:0/542840990 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd97009ccb0 con 0x7fd97014c3a0 2019-12-05T06:55:49.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.703+0000 7fd974cce700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=THROTTLE_DONE pgs=380 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:49.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.703+0000 7fd96d7fa700 1 -- 172.21.15.16:0/542840990 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fd958009790 con 0x7fd97014c3a0 2019-12-05T06:55:49.815 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 193 2019-12-05T06:55:49.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.705+0000 7fd976731700 1 -- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd954023650 msgr2=0x7fd954025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:49.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.705+0000 7fd976731700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd954023650 0x7fd954025ae0 crc :-1 s=READY pgs=664 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:49.815 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.705+0000 7fd976731700 1 -- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 msgr2=0x7fd97009d5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:49.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.705+0000 7fd976731700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 secure :-1 s=READY pgs=380 cs=0 l=1 rx=0x7fd958010390 tx=0x7fd958010730).stop 2019-12-05T06:55:49.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.705+0000 7fd976731700 1 -- 172.21.15.16:0/542840990 shutdown_connections 2019-12-05T06:55:49.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.705+0000 7fd976731700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd954023650 0x7fd954025ae0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:49.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.705+0000 7fd976731700 1 --2- 172.21.15.16:0/542840990 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd97014c3a0 0x7fd97009d5b0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:49.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.705+0000 7fd976731700 1 -- 172.21.15.16:0/542840990 shutdown_connections 2019-12-05T06:55:49.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.705+0000 7fd976731700 1 -- 172.21.15.16:0/542840990 wait complete. 2019-12-05T06:55:49.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:49.705+0000 7fd976731700 1 -- 172.21.15.16:0/542840990 >> 172.21.15.16:0/542840990 conn(0x7fd970098480 msgr2=0x7fd970071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:49.817 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:49.818 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:48.617877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:replay","state_seq":1,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:49.818 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:55:49.818 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:55:49.819 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.027+0000 7f929b7f5700 1 Processor -- start 2019-12-05T06:55:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.027+0000 7f929b7f5700 1 -- start start 2019-12-05T06:55:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.028+0000 7f929b7f5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f929408d2c0 0x7f929408ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:50.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.028+0000 7f929b7f5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f929408fef0 0x7f929408f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.028+0000 7f929b7f5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92940958e0 con 0x7f929408c6d0 2019-12-05T06:55:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.028+0000 7f929b7f5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929408df90 con 0x7f929408d2c0 2019-12-05T06:55:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.028+0000 7f929b7f5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92940a3f70 con 0x7f929408fef0 2019-12-05T06:55:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.029+0000 7f9299591700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f929408d2c0 0x7f929408ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.029+0000 7f9299591700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f929408d2c0 0x7f929408ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36014/0 (socket says 172.21.15.16:36014) 2019-12-05T06:55:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.029+0000 7f9299591700 1 -- 172.21.15.16:0/743097281 learned_addr learned my addr 172.21.15.16:0/743097281 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:55:50.038 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.029+0000 7f929a593700 1 --2- 172.21.15.16:0/743097281 >> v2:172.21.15.76:3300/0 conn(0x7f929408fef0 0x7f929408f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.029+0000 7f9299591700 1 -- 172.21.15.16:0/743097281 >> v2:172.21.15.76:3300/0 conn(0x7f929408fef0 msgr2=0x7f929408f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:55:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.029+0000 7f9299591700 1 --2- 172.21.15.16:0/743097281 >> v2:172.21.15.76:3300/0 conn(0x7f929408fef0 0x7f929408f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.029+0000 7f9299591700 1 -- 172.21.15.16:0/743097281 >> v1:172.21.15.16:6789/0 conn(0x7f929408c6d0 legacy=0x7f929408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.029+0000 7f9299591700 1 -- 172.21.15.16:0/743097281 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929408e620 con 0x7f929408d2c0 2019-12-05T06:55:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f9299591700 1 --2- 172.21.15.16:0/743097281 >> v2:172.21.15.16:3300/0 conn(0x7f929408d2c0 0x7f929408ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f929000f740 tx=0x7f929000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.039 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f9299591700 1 --2- 172.21.15.16:0/743097281 >> v2:172.21.15.16:3300/0 conn(0x7f929408d2c0 0x7f929408ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f929000f740 tx=0x7f929000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f9299591700 1 --2- 172.21.15.16:0/743097281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f929408d2c0 0x7f929408ef20 secure :-1 s=READY pgs=1596 cs=0 l=1 rx=0x7f929000f740 tx=0x7f929000f570).ready entity=mon.0 client_cookie=fc5155a68e5cd7d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:50.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f9299591700 1 --2- 172.21.15.16:0/743097281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f929408d2c0 0x7f929408ef20 secure :-1 s=THROTTLE_DONE pgs=1596 cs=0 l=1 rx=0x7f929000f740 tx=0x7f929000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f9299591700 1 --2- 172.21.15.16:0/743097281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f929408d2c0 0x7f929408ef20 secure :-1 s=THROTTLE_DONE pgs=1596 cs=0 l=1 rx=0x7f929000f740 tx=0x7f929000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f9298d90700 1 -- 172.21.15.16:0/743097281 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9290000960 con 0x7f929408d2c0 2019-12-05T06:55:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f9299591700 1 --2- 172.21.15.16:0/743097281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f929408d2c0 0x7f929408ef20 secure :-1 s=THROTTLE_DONE pgs=1596 cs=0 l=1 rx=0x7f929000f740 tx=0x7f929000f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f9298d90700 1 -- 172.21.15.16:0/743097281 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9290000c90 con 0x7f929408d2c0 2019-12-05T06:55:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f9298d90700 1 -- 172.21.15.16:0/743097281 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f92900013a0 con 0x7f929408d2c0 2019-12-05T06:55:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f929b7f5700 1 -- 172.21.15.16:0/743097281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f929408d2c0 msgr2=0x7f929408ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.030+0000 7f929b7f5700 1 --2- 172.21.15.16:0/743097281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f929408d2c0 0x7f929408ef20 secure :-1 s=READY pgs=1596 cs=0 l=1 rx=0x7f929000f740 tx=0x7f929000f570).stop 2019-12-05T06:55:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.031+0000 7f929b7f5700 1 -- 172.21.15.16:0/743097281 shutdown_connections 2019-12-05T06:55:50.043 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.031+0000 7f929b7f5700 1 --2- 172.21.15.16:0/743097281 >> v2:172.21.15.76:3300/0 conn(0x7f929408fef0 0x7f929408f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.031+0000 7f929b7f5700 1 --2- 172.21.15.16:0/743097281 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f929408d2c0 0x7f929408ef20 unknown :-1 s=CLOSED pgs=1596 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.032+0000 7f929b7f5700 1 -- 172.21.15.16:0/743097281 shutdown_connections 2019-12-05T06:55:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.032+0000 7f929b7f5700 1 -- 172.21.15.16:0/743097281 wait complete. 2019-12-05T06:55:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.032+0000 7f929b7f5700 1 -- 172.21.15.16:0/743097281 >> 172.21.15.16:0/743097281 conn(0x7f9294061940 msgr2=0x7f9294096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.036+0000 7f929b7f5700 1 Processor -- start 2019-12-05T06:55:50.044 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.036+0000 7f929b7f5700 1 -- start start 2019-12-05T06:55:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.037+0000 7f929b7f5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9294096e90 0x7f929409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.037+0000 7f929b7f5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f929409b920 0x7f9294097a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.037+0000 7f929b7f5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9294084b90 con 0x7f9294084f30 2019-12-05T06:55:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.037+0000 7f929b7f5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f929409a450 con 0x7f9294096e90 2019-12-05T06:55:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.037+0000 7f929b7f5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9294099a20 con 0x7f929409b920 2019-12-05T06:55:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.037+0000 7f9299d92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9294096e90 0x7f929409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.037+0000 7f9299d92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9294096e90 0x7f929409b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36018/0 (socket says 172.21.15.16:36018) 2019-12-05T06:55:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.037+0000 7f9299d92700 1 -- 172.21.15.16:0/1833486391 learned_addr learned my addr 172.21.15.16:0/1833486391 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:55:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.037+0000 7f929a593700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f929409b920 0x7f9294097a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f928a7fc700 1 -- 172.21.15.16:0/1833486391 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186922129 0 0) 0x7f9290007d20 con 0x7f9294084f30 2019-12-05T06:55:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f9299d92700 1 -- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f929409b920 msgr2=0x7f9294097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f929409b920 0x7f9294097a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f9299d92700 1 -- 172.21.15.16:0/1833486391 >> v1:172.21.15.76:6789/0 conn(0x7f9294084f30 legacy=0x7f9294097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f9299d92700 1 -- 172.21.15.16:0/1833486391 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f929409aae0 con 0x7f9294096e90 2019-12-05T06:55:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f929a593700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f929409b920 0x7f9294097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:55:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> v2:172.21.15.16:3300/0 conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> v2:172.21.15.16:3300/0 conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=READY pgs=1597 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=THROTTLE_DONE pgs=1597 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.050 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f928a7fc700 1 -- 172.21.15.16:0/1833486391 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f928c005bc0 con 0x7f9294096e90 2019-12-05T06:55:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=THROTTLE_DONE pgs=1597 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=THROTTLE_DONE pgs=1597 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f928a7fc700 1 -- 172.21.15.16:0/1833486391 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f928c0057e0 con 0x7f9294096e90 2019-12-05T06:55:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f928a7fc700 1 -- 172.21.15.16:0/1833486391 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f928c0061a0 con 0x7f9294096e90 2019-12-05T06:55:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.038+0000 7f929b7f5700 1 -- 172.21.15.16:0/1833486391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f929409aae0 con 0x7f9294096e90 2019-12-05T06:55:50.051 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.039+0000 7f929b7f5700 1 -- 172.21.15.16:0/1833486391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f929409aae0 con 0x7f9294096e90 2019-12-05T06:55:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.039+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=THROTTLE_DONE pgs=1597 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.040+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=THROTTLE_DONE pgs=1597 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.052 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.040+0000 7f928a7fc700 1 -- 172.21.15.16:0/1833486391 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f928c0059d0 con 0x7f9294096e90 2019-12-05T06:55:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.040+0000 7f928a7fc700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9274023630 0x7f9274025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.040+0000 7f928a7fc700 1 -- 172.21.15.16:0/1833486391 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f928c046170 con 0x7f9294096e90 2019-12-05T06:55:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.041+0000 7f9299591700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9274023630 0x7f9274025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.041+0000 7f9299591700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9274023630 0x7f9274025ac0 crc :-1 s=READY pgs=665 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:50.053 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.042+0000 7f929b7f5700 1 -- 172.21.15.16:0/1833486391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929409aae0 con 0x7f9294096e90 2019-12-05T06:55:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.050+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=THROTTLE_DONE pgs=1597 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.054 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.050+0000 7f928a7fc700 1 -- 172.21.15.16:0/1833486391 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f928c007980 con 0x7f9294096e90 2019-12-05T06:55:50.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.271+0000 7f929b7f5700 1 -- 172.21.15.16:0/1833486391 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f929409aae0 con 0x7f9294096e90 2019-12-05T06:55:50.272 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.271+0000 7f9299d92700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=THROTTLE_DONE pgs=1597 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:50.273 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.271+0000 7f928a7fc700 1 -- 172.21.15.16:0/1833486391 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 193 v193) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f928c0059d0 con 0x7f9294096e90 2019-12-05T06:55:50.273 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 193 2019-12-05T06:55:50.273 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:50.273 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":193,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":193,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:48.617877+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:replay","state_seq":1,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:50.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.274+0000 7f929b7f5700 1 -- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9274023630 msgr2=0x7f9274025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:50.276 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.274+0000 7f929b7f5700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9274023630 0x7f9274025ac0 crc :-1 s=READY pgs=665 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:50.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.274+0000 7f929b7f5700 1 -- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 msgr2=0x7f929409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:50.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.274+0000 7f929b7f5700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 0x7f929409b3e0 secure :-1 s=READY pgs=1597 cs=0 l=1 rx=0x7f928c010790 tx=0x7f928c010b30).stop 2019-12-05T06:55:50.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.275+0000 7f929b7f5700 1 -- 172.21.15.16:0/1833486391 shutdown_connections 2019-12-05T06:55:50.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.275+0000 7f929b7f5700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f929409b920 0x7f9294097a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:50.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.275+0000 7f929b7f5700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9274023630 0x7f9274025ac0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:50.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.275+0000 7f929b7f5700 1 --2- 172.21.15.16:0/1833486391 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9294096e90 0x7f929409b3e0 unknown :-1 s=CLOSED pgs=1597 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:50.277 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.275+0000 7f929b7f5700 1 -- 172.21.15.16:0/1833486391 shutdown_connections 2019-12-05T06:55:50.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.275+0000 7f929b7f5700 1 -- 172.21.15.16:0/1833486391 wait complete. 2019-12-05T06:55:50.278 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:50.275+0000 7f929b7f5700 1 -- 172.21.15.16:0/1833486391 >> 172.21.15.16:0/1833486391 conn(0x7f92940714f0 msgr2=0x7f929409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:50.287 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 193, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 193, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_6212": { "addr": "172.21.15.76:6835/2545654850", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 2545654850, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 2545654850, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 6212, "incarnation": 193, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 82, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:55:48.617877+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 6212 } } } ], "standbys": [] } 2019-12-05T06:55:52.288 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.544+0000 7f8b6cf9e700 1 Processor -- start 2019-12-05T06:55:52.553 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.544+0000 7f8b6cf9e700 1 -- start start 2019-12-05T06:55:52.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.545+0000 7f8b6cf9e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b6808c6d0 0x7f8b6808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:52.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.545+0000 7f8b6cf9e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8b6808fef0 0x7f8b6808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:52.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.545+0000 7f8b6cf9e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6808fb80 con 0x7f8b6808ef20 2019-12-05T06:55:52.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.545+0000 7f8b6cf9e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b6808df90 con 0x7f8b6808c6d0 2019-12-05T06:55:52.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.545+0000 7f8b6cf9e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b680a3f70 con 0x7f8b6808fef0 2019-12-05T06:55:52.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.546+0000 7f8b66d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b6808c6d0 0x7f8b6808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:52.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.546+0000 7f8b66d9d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8b6808c6d0 0x7f8b6808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36028/0 (socket says 172.21.15.16:36028) 2019-12-05T06:55:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.546+0000 7f8b6759e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8b6808ef20 0x7f8b6808f380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49432/0 (socket says 172.21.15.16:49432) 2019-12-05T06:55:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.546+0000 7f8b66d9d700 1 -- 172.21.15.16:0/3905305485 learned_addr learned my addr 172.21.15.16:0/3905305485 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:55:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.546+0000 7f8b6659c700 1 --2- 172.21.15.16:0/3905305485 >> v2:172.21.15.76:3300/0 conn(0x7f8b6808fef0 0x7f8b6808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.546+0000 7f8b66d9d700 1 -- 172.21.15.16:0/3905305485 >> v2:172.21.15.76:3300/0 conn(0x7f8b6808fef0 msgr2=0x7f8b6808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:55:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.546+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/3905305485 >> v2:172.21.15.76:3300/0 conn(0x7f8b6808fef0 0x7f8b6808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.546+0000 7f8b66d9d700 1 -- 172.21.15.16:0/3905305485 >> v1:172.21.15.16:6789/0 conn(0x7f8b6808ef20 legacy=0x7f8b6808f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:52.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.547+0000 7f8b65d9b700 1 -- 172.21.15.16:0/3905305485 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722785616 0 0) 0x7f8b60000d30 con 0x7f8b6808ef20 2019-12-05T06:55:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.547+0000 7f8b66d9d700 1 -- 172.21.15.16:0/3905305485 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b6808d540 con 0x7f8b6808c6d0 2019-12-05T06:55:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.547+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/3905305485 >> v2:172.21.15.16:3300/0 conn(0x7f8b6808c6d0 0x7f8b6808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8b5800f6c0 tx=0x7f8b5800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.547+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/3905305485 >> v2:172.21.15.16:3300/0 conn(0x7f8b6808c6d0 0x7f8b6808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8b5800f6c0 tx=0x7f8b5800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.547+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/3905305485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6808c6d0 0x7f8b6808cac0 secure :-1 s=READY pgs=1599 cs=0 l=1 rx=0x7f8b5800f6c0 tx=0x7f8b5800fa60).ready entity=mon.0 client_cookie=dbe9e0dd3fafca98 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.547+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/3905305485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6808c6d0 0x7f8b6808cac0 secure :-1 s=THROTTLE_DONE pgs=1599 cs=0 l=1 rx=0x7f8b5800f6c0 tx=0x7f8b5800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.547+0000 7f8b65d9b700 1 -- 172.21.15.16:0/3905305485 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b58000960 con 0x7f8b6808c6d0 2019-12-05T06:55:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.547+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/3905305485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6808c6d0 0x7f8b6808cac0 secure :-1 s=THROTTLE_DONE pgs=1599 cs=0 l=1 rx=0x7f8b5800f6c0 tx=0x7f8b5800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.547+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/3905305485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6808c6d0 0x7f8b6808cac0 secure :-1 s=THROTTLE_DONE pgs=1599 cs=0 l=1 rx=0x7f8b5800f6c0 tx=0x7f8b5800fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.548+0000 7f8b65d9b700 1 -- 172.21.15.16:0/3905305485 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b58000c90 con 0x7f8b6808c6d0 2019-12-05T06:55:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.548+0000 7f8b65d9b700 1 -- 172.21.15.16:0/3905305485 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b580013a0 con 0x7f8b6808c6d0 2019-12-05T06:55:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.548+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/3905305485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6808c6d0 msgr2=0x7f8b6808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.548+0000 7f8b6cf9e700 1 --2- 172.21.15.16:0/3905305485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6808c6d0 0x7f8b6808cac0 secure :-1 s=READY pgs=1599 cs=0 l=1 rx=0x7f8b5800f6c0 tx=0x7f8b5800fa60).stop 2019-12-05T06:55:52.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.550+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/3905305485 shutdown_connections 2019-12-05T06:55:52.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.550+0000 7f8b6cf9e700 1 --2- 172.21.15.16:0/3905305485 >> v2:172.21.15.76:3300/0 conn(0x7f8b6808fef0 0x7f8b6808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:52.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.550+0000 7f8b6cf9e700 1 --2- 172.21.15.16:0/3905305485 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8b6808c6d0 0x7f8b6808cac0 unknown :-1 s=CLOSED pgs=1599 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:52.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.550+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/3905305485 shutdown_connections 2019-12-05T06:55:52.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.550+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/3905305485 wait complete. 2019-12-05T06:55:52.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.550+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/3905305485 >> 172.21.15.16:0/3905305485 conn(0x7f8b68061940 msgr2=0x7f8b68096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:52.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.554+0000 7f8b6cf9e700 1 Processor -- start 2019-12-05T06:55:52.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.554+0000 7f8b6cf9e700 1 -- start start 2019-12-05T06:55:52.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.555+0000 7f8b6cf9e700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8b6806df10 0x7f8b68098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:52.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.555+0000 7f8b6cf9e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8b680992d0 0x7f8b68098840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:52.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.555+0000 7f8b6cf9e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b6806e300 con 0x7f8b68098d80 2019-12-05T06:55:52.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.555+0000 7f8b6cf9e700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b68097370 con 0x7f8b6806df10 2019-12-05T06:55:52.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.555+0000 7f8b6cf9e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b6808ad00 con 0x7f8b680992d0 2019-12-05T06:55:52.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.555+0000 7f8b6759e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8b68098d80 0x7f8b68088fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49436/0 (socket says 172.21.15.16:49436) 2019-12-05T06:55:52.565 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.555+0000 7f8b6759e700 1 -- 172.21.15.16:0/4239601404 learned_addr learned my addr 172.21.15.16:0/4239601404 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:52.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.555+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> v2:172.21.15.76:3300/0 conn(0x7f8b6806df10 0x7f8b68098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:52.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.555+0000 7f8b6659c700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8b680992d0 0x7f8b68098840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:52.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.556+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/4239601404 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592052010 0 0) 0x7f8b60001690 con 0x7f8b68098d80 2019-12-05T06:55:52.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.556+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/4239601404 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8b4c002e80 con 0x7f8b68098d80 2019-12-05T06:55:52.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.556+0000 7f8b66d9d700 1 -- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8b680992d0 msgr2=0x7f8b68098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:52.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.556+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8b680992d0 0x7f8b68098840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:52.566 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.556+0000 7f8b66d9d700 1 -- 172.21.15.16:0/4239601404 >> v1:172.21.15.16:6789/0 conn(0x7f8b68098d80 legacy=0x7f8b68088fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:52.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.556+0000 7f8b66d9d700 1 -- 172.21.15.16:0/4239601404 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b68097a00 con 0x7f8b6806df10 2019-12-05T06:55:52.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.556+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> v2:172.21.15.76:3300/0 conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.556+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> v2:172.21.15.76:3300/0 conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.556+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=READY pgs=1071 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:52.567 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.557+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=THROTTLE_DONE pgs=1071 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.557+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/4239601404 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b58007a90 con 0x7f8b6806df10 2019-12-05T06:55:52.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.557+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=THROTTLE_DONE pgs=1071 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.557+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=THROTTLE_DONE pgs=1071 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.557+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/4239601404 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b58007250 con 0x7f8b6806df10 2019-12-05T06:55:52.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.557+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/4239601404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b68097a00 con 0x7f8b6806df10 2019-12-05T06:55:52.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.557+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/4239601404 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b58005770 con 0x7f8b6806df10 2019-12-05T06:55:52.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.557+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/4239601404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b68097a00 con 0x7f8b6806df10 2019-12-05T06:55:52.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.558+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=THROTTLE_DONE pgs=1071 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.559+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/4239601404 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8b58005390 con 0x7f8b6806df10 2019-12-05T06:55:52.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.559+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=THROTTLE_DONE pgs=1071 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.560+0000 7f8b5f7fe700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b4c023670 0x7f8b4c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:52.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.560+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/4239601404 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f8b58045d20 con 0x7f8b6806df10 2019-12-05T06:55:52.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.560+0000 7f8b6659c700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b4c023670 0x7f8b4c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:52.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.561+0000 7f8b6659c700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b4c023670 0x7f8b4c025ae0 crc :-1 s=READY pgs=666 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:52.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.575+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/4239601404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b68097a00 con 0x7f8b6806df10 2019-12-05T06:55:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.586+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=THROTTLE_DONE pgs=1071 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.586+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/4239601404 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8b58005180 con 0x7f8b6806df10 2019-12-05T06:55:52.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.816+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/4239601404 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8b68097a00 con 0x7f8b6806df10 2019-12-05T06:55:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.818+0000 7f8b66d9d700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=THROTTLE_DONE pgs=1071 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:52.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.818+0000 7f8b5f7fe700 1 -- 172.21.15.16:0/4239601404 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 195 v195) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f8b58005390 con 0x7f8b6806df10 2019-12-05T06:55:52.820 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 195 2019-12-05T06:55:52.820 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:52.820 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":195,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":195,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:52.271500+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:rejoin","state_seq":19,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:52.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b4c023670 msgr2=0x7f8b4c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:52.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b4c023670 0x7f8b4c025ae0 crc :-1 s=READY pgs=666 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:52.823 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 msgr2=0x7f8b68098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 0x7f8b68098300 secure :-1 s=READY pgs=1071 cs=0 l=1 rx=0x7f8b58010370 tx=0x7f8b58010710).stop 2019-12-05T06:55:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/4239601404 shutdown_connections 2019-12-05T06:55:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8b4c023670 0x7f8b4c025ae0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8b680992d0 0x7f8b68098840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 --2- 172.21.15.16:0/4239601404 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f8b6806df10 0x7f8b68098300 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/4239601404 shutdown_connections 2019-12-05T06:55:52.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/4239601404 wait complete. 2019-12-05T06:55:52.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:52.822+0000 7f8b6cf9e700 1 -- 172.21.15.16:0/4239601404 >> 172.21.15.16:0/4239601404 conn(0x7f8b680714f0 msgr2=0x7f8b6809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:52.836 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:55:54.837 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:55:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.119+0000 7f08b87ce700 1 Processor -- start 2019-12-05T06:55:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.119+0000 7f08b87ce700 1 -- start start 2019-12-05T06:55:55.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.121+0000 7f08b87ce700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08b008c6d0 0x7f08b008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:55.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.121+0000 7f08b87ce700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f08b008fef0 0x7f08b008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:55.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.121+0000 7f08b87ce700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08b008ef20 0x7f08b008f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:55:55.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.121+0000 7f08b87ce700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b00a3f70 con 0x7f08b008ef20 2019-12-05T06:55:55.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.121+0000 7f08b87ce700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b008d830 con 0x7f08b008fef0 2019-12-05T06:55:55.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.121+0000 7f08b87ce700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08b008d540 con 0x7f08b008c6d0 2019-12-05T06:55:55.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.121+0000 7f08b756c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08b008ef20 0x7f08b008f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:55.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.122+0000 7f08b756c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08b008ef20 0x7f08b008f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36052/0 (socket says 172.21.15.16:36052) 2019-12-05T06:55:55.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.121+0000 7f08b6d6b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08b008c6d0 0x7f08b008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:55.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.122+0000 7f08b756c700 1 -- 172.21.15.16:0/2155339580 learned_addr learned my addr 172.21.15.16:0/2155339580 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:55:55.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.122+0000 7f08b656a700 1 --2- 172.21.15.16:0/2155339580 >> v2:172.21.15.76:3300/0 conn(0x7f08b008fef0 0x7f08b008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:55.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.122+0000 7f08b756c700 1 -- 172.21.15.16:0/2155339580 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08b008c6d0 msgr2=0x7f08b008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:55:55.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.122+0000 7f08b756c700 1 --2- 172.21.15.16:0/2155339580 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08b008c6d0 0x7f08b008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:55.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.122+0000 7f08b756c700 1 -- 172.21.15.16:0/2155339580 >> v2:172.21.15.76:3300/0 conn(0x7f08b008fef0 msgr2=0x7f08b008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:55:55.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.122+0000 7f08b756c700 1 --2- 172.21.15.16:0/2155339580 >> v2:172.21.15.76:3300/0 conn(0x7f08b008fef0 0x7f08b008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:55.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.122+0000 7f08b756c700 1 -- 172.21.15.16:0/2155339580 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b008e620 con 0x7f08b008ef20 2019-12-05T06:55:55.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.122+0000 7f08b756c700 1 --2- 172.21.15.16:0/2155339580 >> v2:172.21.15.16:3300/0 conn(0x7f08b008ef20 0x7f08b008f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f08a800f6b0 tx=0x7f08a800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.123+0000 7f08b756c700 1 --2- 172.21.15.16:0/2155339580 >> v2:172.21.15.16:3300/0 conn(0x7f08b008ef20 0x7f08b008f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f08a800f6b0 tx=0x7f08a800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.123+0000 7f08b756c700 1 --2- 172.21.15.16:0/2155339580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b008ef20 0x7f08b008f3c0 secure :-1 s=READY pgs=1602 cs=0 l=1 rx=0x7f08a800f6b0 tx=0x7f08a800fa50).ready entity=mon.0 client_cookie=29abe385811dc322 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:55.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.123+0000 7f08b756c700 1 --2- 172.21.15.16:0/2155339580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b008ef20 0x7f08b008f3c0 secure :-1 s=THROTTLE_DONE pgs=1602 cs=0 l=1 rx=0x7f08a800f6b0 tx=0x7f08a800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.123+0000 7f08b756c700 1 --2- 172.21.15.16:0/2155339580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b008ef20 0x7f08b008f3c0 secure :-1 s=THROTTLE_DONE pgs=1602 cs=0 l=1 rx=0x7f08a800f6b0 tx=0x7f08a800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.123+0000 7f08b756c700 1 --2- 172.21.15.16:0/2155339580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b008ef20 0x7f08b008f3c0 secure :-1 s=THROTTLE_DONE pgs=1602 cs=0 l=1 rx=0x7f08a800f6b0 tx=0x7f08a800fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.123+0000 7f08b5d69700 1 -- 172.21.15.16:0/2155339580 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a8000a70 con 0x7f08b008ef20 2019-12-05T06:55:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.123+0000 7f08b5d69700 1 -- 172.21.15.16:0/2155339580 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a80010e0 con 0x7f08b008ef20 2019-12-05T06:55:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.123+0000 7f08b5d69700 1 -- 172.21.15.16:0/2155339580 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a80017b0 con 0x7f08b008ef20 2019-12-05T06:55:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.123+0000 7f08b87ce700 1 -- 172.21.15.16:0/2155339580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b008ef20 msgr2=0x7f08b008f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:55.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.123+0000 7f08b87ce700 1 --2- 172.21.15.16:0/2155339580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b008ef20 0x7f08b008f3c0 secure :-1 s=READY pgs=1602 cs=0 l=1 rx=0x7f08a800f6b0 tx=0x7f08a800fa50).stop 2019-12-05T06:55:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.126+0000 7f08b87ce700 1 -- 172.21.15.16:0/2155339580 shutdown_connections 2019-12-05T06:55:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.126+0000 7f08b87ce700 1 --2- 172.21.15.16:0/2155339580 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b008ef20 0x7f08b008f3c0 unknown :-1 s=CLOSED pgs=1602 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.126+0000 7f08b87ce700 1 --2- 172.21.15.16:0/2155339580 >> v2:172.21.15.76:3300/0 conn(0x7f08b008fef0 0x7f08b008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.126+0000 7f08b87ce700 1 --2- 172.21.15.16:0/2155339580 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f08b008c6d0 0x7f08b008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:55:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.126+0000 7f08b87ce700 1 -- 172.21.15.16:0/2155339580 shutdown_connections 2019-12-05T06:55:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.126+0000 7f08b87ce700 1 -- 172.21.15.16:0/2155339580 wait complete. 2019-12-05T06:55:55.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.126+0000 7f08b87ce700 1 -- 172.21.15.16:0/2155339580 >> 172.21.15.16:0/2155339580 conn(0x7f08b0061940 msgr2=0x7f08b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:55.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.131+0000 7f08b87ce700 1 Processor -- start 2019-12-05T06:55:55.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.131+0000 7f08b87ce700 1 -- start start 2019-12-05T06:55:55.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.133+0000 7f08b87ce700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f08b009bbe0 0x7f08b01539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:55.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.133+0000 7f08b87ce700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08b0084f30 0x7f08b0155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:55.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.133+0000 7f08b87ce700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08b0084b90 con 0x7f08b0096e90 2019-12-05T06:55:55.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.133+0000 7f08b87ce700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b009a450 con 0x7f08b0084f30 2019-12-05T06:55:55.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.133+0000 7f08b87ce700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08b0099a20 con 0x7f08b009bbe0 2019-12-05T06:55:55.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.133+0000 7f08b756c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08b0084f30 0x7f08b0155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:55.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.133+0000 7f08b656a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f08b009bbe0 0x7f08b01539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:55.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.133+0000 7f08b6d6b700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f08b0096e90 0x7f08b009b3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:42524/0 (socket says 172.21.15.16:42524) 2019-12-05T06:55:55.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.133+0000 7f08b6d6b700 1 -- 172.21.15.16:0/2736746616 learned_addr learned my addr 172.21.15.16:0/2736746616 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:55:55.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.133+0000 7f08b756c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f08b0084f30 0x7f08b0155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36058/0 (socket says 172.21.15.16:36058) 2019-12-05T06:55:55.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.134+0000 7f08b756c700 1 -- 172.21.15.16:0/2736746616 >> v2:172.21.15.76:3300/0 conn(0x7f08b009bbe0 msgr2=0x7f08b01539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:55.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.134+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> v2:172.21.15.76:3300/0 conn(0x7f08b009bbe0 0x7f08b01539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:55.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.134+0000 7f08b756c700 1 -- 172.21.15.16:0/2736746616 >> v1:172.21.15.76:6789/0 conn(0x7f08b0096e90 legacy=0x7f08b009b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:55.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.134+0000 7f08b756c700 1 -- 172.21.15.16:0/2736746616 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08b009aae0 con 0x7f08b0084f30 2019-12-05T06:55:55.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.134+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> v2:172.21.15.16:3300/0 conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.134+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> v2:172.21.15.16:3300/0 conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.134+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=READY pgs=1603 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:55.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.134+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=THROTTLE_DONE pgs=1603 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.135+0000 7f08af7fe700 1 -- 172.21.15.16:0/2736746616 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a8054030 con 0x7f08b0084f30 2019-12-05T06:55:55.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.135+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=THROTTLE_DONE pgs=1603 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.135+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=THROTTLE_DONE pgs=1603 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.135+0000 7f08af7fe700 1 -- 172.21.15.16:0/2736746616 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08a80081f0 con 0x7f08b0084f30 2019-12-05T06:55:55.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.135+0000 7f08af7fe700 1 -- 172.21.15.16:0/2736746616 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08a8053e40 con 0x7f08b0084f30 2019-12-05T06:55:55.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.135+0000 7f08b87ce700 1 -- 172.21.15.16:0/2736746616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08b009aae0 con 0x7f08b0084f30 2019-12-05T06:55:55.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.135+0000 7f08b87ce700 1 -- 172.21.15.16:0/2736746616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08b0084b90 con 0x7f08b0084f30 2019-12-05T06:55:55.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.136+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=THROTTLE_DONE pgs=1603 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.137+0000 7f08af7fe700 1 -- 172.21.15.16:0/2736746616 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f08a80542d0 con 0x7f08b0084f30 2019-12-05T06:55:55.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.137+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=THROTTLE_DONE pgs=1603 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.138+0000 7f08af7fe700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0894023630 0x7f0894025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:55:55.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.138+0000 7f08af7fe700 1 -- 172.21.15.16:0/2736746616 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f08a808bf40 con 0x7f08b0084f30 2019-12-05T06:55:55.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.138+0000 7f08b6d6b700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0894023630 0x7f0894025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:55:55.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.139+0000 7f08b6d6b700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0894023630 0x7f0894025ac0 crc :-1 s=READY pgs=667 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:55:55.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.150+0000 7f08b87ce700 1 -- 172.21.15.16:0/2736746616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08b009aae0 con 0x7f08b0084f30 2019-12-05T06:55:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.160+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=THROTTLE_DONE pgs=1603 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.160+0000 7f08af7fe700 1 -- 172.21.15.16:0/2736746616 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f08a8054090 con 0x7f08b0084f30 2019-12-05T06:55:55.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.414+0000 7f08b87ce700 1 -- 172.21.15.16:0/2736746616 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f08b009aae0 con 0x7f08b0084f30 2019-12-05T06:55:55.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.414+0000 7f08b756c700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=THROTTLE_DONE pgs=1603 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:55:55.415 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.414+0000 7f08af7fe700 1 -- 172.21.15.16:0/2736746616 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f08a80542a0 con 0x7f08b0084f30 2019-12-05T06:55:55.416 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 196 2019-12-05T06:55:55.416 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:55:55.416 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:55:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 -- 172.21.15.16:0/2736746616 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0894023630 msgr2=0x7f0894025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:55.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0894023630 0x7f0894025ac0 crc :-1 s=READY pgs=667 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 -- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 msgr2=0x7f08b0155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:55:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 0x7f08b0155f30 secure :-1 s=READY pgs=1603 cs=0 l=1 rx=0x7f08a80565b0 tx=0x7f08a80563e0).stop 2019-12-05T06:55:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 -- 172.21.15.16:0/2736746616 shutdown_connections 2019-12-05T06:55:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f0894023630 0x7f0894025ac0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 --2- 172.21.15.16:0/2736746616 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f08b0084f30 0x7f08b0155f30 unknown :-1 s=CLOSED pgs=1603 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 --2- 172.21.15.16:0/2736746616 >> v2:172.21.15.76:3300/0 conn(0x7f08b009bbe0 0x7f08b01539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:55:55.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 -- 172.21.15.16:0/2736746616 shutdown_connections 2019-12-05T06:55:55.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 -- 172.21.15.16:0/2736746616 wait complete. 2019-12-05T06:55:55.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:55:55.417+0000 7f08b87ce700 1 -- 172.21.15.16:0/2736746616 >> 172.21.15.16:0/2736746616 conn(0x7f08b00714f0 msgr2=0x7f08b009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:55:55.429 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=8106 2019-12-05T06:55:55.429 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 9 secs before reviving mds.a 2019-12-05T06:56:04.431 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:56:04.432 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:56:04.432 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:56:04.437 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:56:04.437 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:04.599 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:56:04.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.608+0000 7fde9d9d9700 1 Processor -- start 2019-12-05T06:56:04.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.609+0000 7fde9d9d9700 1 -- start start 2019-12-05T06:56:04.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.609+0000 7fde9d9d9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fde9808c6d0 0x7fde9808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:04.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.609+0000 7fde9d9d9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde9808fef0 0x7fde9808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:04.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.609+0000 7fde9d9d9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde9808ef20 0x7fde9808f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:04.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.609+0000 7fde9d9d9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9808df90 con 0x7fde9808ef20 2019-12-05T06:56:04.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.609+0000 7fde9d9d9700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde980a3f70 con 0x7fde9808c6d0 2019-12-05T06:56:04.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.609+0000 7fde9d9d9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde9808d830 con 0x7fde9808fef0 2019-12-05T06:56:04.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.609+0000 7fde97fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde9808ef20 0x7fde9808f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:04.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fde9808ef20 0x7fde9808f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36130/0 (socket says 172.21.15.16:36130) 2019-12-05T06:56:04.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 -- 172.21.15.16:0/2401714563 learned_addr learned my addr 172.21.15.16:0/2401714563 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:04.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde977fe700 1 --2- 172.21.15.16:0/2401714563 >> v2:172.21.15.76:3300/0 conn(0x7fde9808c6d0 0x7fde9808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:04.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde96ffd700 1 --2- 172.21.15.16:0/2401714563 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde9808fef0 0x7fde9808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:04.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 -- 172.21.15.16:0/2401714563 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde9808fef0 msgr2=0x7fde9808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:04.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 --2- 172.21.15.16:0/2401714563 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde9808fef0 0x7fde9808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:04.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 -- 172.21.15.16:0/2401714563 >> v2:172.21.15.76:3300/0 conn(0x7fde9808c6d0 msgr2=0x7fde9808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:04.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 --2- 172.21.15.16:0/2401714563 >> v2:172.21.15.76:3300/0 conn(0x7fde9808c6d0 0x7fde9808cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:04.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 -- 172.21.15.16:0/2401714563 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde9808d540 con 0x7fde9808ef20 2019-12-05T06:56:04.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde977fe700 1 --2- 172.21.15.16:0/2401714563 >> v2:172.21.15.76:3300/0 conn(0x7fde9808c6d0 0x7fde9808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:56:04.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 --2- 172.21.15.16:0/2401714563 >> v2:172.21.15.16:3300/0 conn(0x7fde9808ef20 0x7fde9808f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fde9000f6b0 tx=0x7fde9000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 --2- 172.21.15.16:0/2401714563 >> v2:172.21.15.16:3300/0 conn(0x7fde9808ef20 0x7fde9808f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fde9000f6b0 tx=0x7fde9000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 --2- 172.21.15.16:0/2401714563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9808ef20 0x7fde9808f3c0 secure :-1 s=READY pgs=1606 cs=0 l=1 rx=0x7fde9000f6b0 tx=0x7fde9000fa50).ready entity=mon.0 client_cookie=8f29ba1c3c4e7e7c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:04.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 --2- 172.21.15.16:0/2401714563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9808ef20 0x7fde9808f3c0 secure :-1 s=THROTTLE_DONE pgs=1606 cs=0 l=1 rx=0x7fde9000f6b0 tx=0x7fde9000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 --2- 172.21.15.16:0/2401714563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9808ef20 0x7fde9808f3c0 secure :-1 s=THROTTLE_DONE pgs=1606 cs=0 l=1 rx=0x7fde9000f6b0 tx=0x7fde9000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde967fc700 1 -- 172.21.15.16:0/2401714563 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde900010e0 con 0x7fde9808ef20 2019-12-05T06:56:04.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.610+0000 7fde97fff700 1 --2- 172.21.15.16:0/2401714563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9808ef20 0x7fde9808f3c0 secure :-1 s=THROTTLE_DONE pgs=1606 cs=0 l=1 rx=0x7fde9000f6b0 tx=0x7fde9000fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.611+0000 7fde967fc700 1 -- 172.21.15.16:0/2401714563 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde90000b90 con 0x7fde9808ef20 2019-12-05T06:56:04.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.611+0000 7fde967fc700 1 -- 172.21.15.16:0/2401714563 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde900016c0 con 0x7fde9808ef20 2019-12-05T06:56:04.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.611+0000 7fde9d9d9700 1 -- 172.21.15.16:0/2401714563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9808ef20 msgr2=0x7fde9808f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:04.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.611+0000 7fde9d9d9700 1 --2- 172.21.15.16:0/2401714563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9808ef20 0x7fde9808f3c0 secure :-1 s=READY pgs=1606 cs=0 l=1 rx=0x7fde9000f6b0 tx=0x7fde9000fa50).stop 2019-12-05T06:56:04.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.612+0000 7fde9d9d9700 1 -- 172.21.15.16:0/2401714563 shutdown_connections 2019-12-05T06:56:04.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.612+0000 7fde9d9d9700 1 --2- 172.21.15.16:0/2401714563 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde9808ef20 0x7fde9808f3c0 unknown :-1 s=CLOSED pgs=1606 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:04.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.612+0000 7fde9d9d9700 1 --2- 172.21.15.16:0/2401714563 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde9808fef0 0x7fde9808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:04.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.612+0000 7fde9d9d9700 1 --2- 172.21.15.16:0/2401714563 >> v2:172.21.15.76:3300/0 conn(0x7fde9808c6d0 0x7fde9808cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:04.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.612+0000 7fde9d9d9700 1 -- 172.21.15.16:0/2401714563 shutdown_connections 2019-12-05T06:56:04.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.612+0000 7fde9d9d9700 1 -- 172.21.15.16:0/2401714563 wait complete. 2019-12-05T06:56:04.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.612+0000 7fde9d9d9700 1 -- 172.21.15.16:0/2401714563 >> 172.21.15.16:0/2401714563 conn(0x7fde98061940 msgr2=0x7fde98096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.614+0000 7fde9d9d9700 1 Processor -- start 2019-12-05T06:56:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.615+0000 7fde9d9d9700 1 -- start start 2019-12-05T06:56:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.615+0000 7fde9d9d9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde98130750 0x7fde9812ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:04.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.615+0000 7fde9d9d9700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fde98154af0 0x7fde98156ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.615+0000 7fde9d9d9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde980889b0 con 0x7fde98080290 2019-12-05T06:56:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.615+0000 7fde9d9d9700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde9808a670 con 0x7fde98154af0 2019-12-05T06:56:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.615+0000 7fde9d9d9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde98089c40 con 0x7fde98130750 2019-12-05T06:56:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.615+0000 7fde977fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fde98080290 0x7fde9812f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49532/0 (socket says 172.21.15.16:49532) 2019-12-05T06:56:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.615+0000 7fde977fe700 1 -- 172.21.15.16:0/1470828274 learned_addr learned my addr 172.21.15.16:0/1470828274 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:04.630 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde98130750 0x7fde9812ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde97fff700 1 --2- 172.21.15.16:0/1470828274 >> v2:172.21.15.76:3300/0 conn(0x7fde98154af0 0x7fde98156ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1190192928 0 0) 0x7fde8000a010 con 0x7fde98080290 2019-12-05T06:56:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde7c002e80 con 0x7fde98080290 2019-12-05T06:56:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 797533794 0 0) 0x7fde8000a210 con 0x7fde98080290 2019-12-05T06:56:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde7c0032e0 con 0x7fde98080290 2019-12-05T06:56:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fde80009b20 con 0x7fde98080290 2019-12-05T06:56:04.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde98130750 msgr2=0x7fde9812ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:04.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde98130750 0x7fde9812ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:04.632 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 >> v2:172.21.15.76:3300/0 conn(0x7fde98154af0 msgr2=0x7fde98156ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 --2- 172.21.15.16:0/1470828274 >> v2:172.21.15.76:3300/0 conn(0x7fde98154af0 0x7fde98156ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 >> v1:172.21.15.16:6789/0 conn(0x7fde98080290 legacy=0x7fde9812f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde7c0033f0 0x7fde7c003840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde7c004890 0x7fde7c004d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:04.633 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c00bb90 con 0x7fde7c003d80 2019-12-05T06:56:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c00bdc0 con 0x7fde7c004890 2019-12-05T06:56:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c00bff0 con 0x7fde7c0033f0 2019-12-05T06:56:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.616+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde98130750 0x7fde9812ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:56:04.634 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde977fe700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde7c0033f0 0x7fde7c003840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde97fff700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde7c004890 0x7fde7c004d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 -- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde7c0033f0 msgr2=0x7fde7c003840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde7c0033f0 0x7fde7c003840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 -- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde7c004890 msgr2=0x7fde7c004d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:04.635 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde7c004890 0x7fde7c004d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 -- 172.21.15.16:0/1470828274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c00c2a0 con 0x7fde7c003d80 2019-12-05T06:56:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde9d9d9700 1 -- 172.21.15.16:0/1470828274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde9812ee80 con 0x7fde7c003d80 2019-12-05T06:56:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.636 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde977fe700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fde7c0033f0 0x7fde7c003840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:56:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde97fff700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fde7c004890 0x7fde7c004d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:56:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde9d9d9700 1 -- 172.21.15.16:0/1470828274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde980889b0 con 0x7fde7c003d80 2019-12-05T06:56:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=READY pgs=1608 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde88009980 con 0x7fde7c003d80 2019-12-05T06:56:04.637 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde880095a0 con 0x7fde7c003d80 2019-12-05T06:56:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.617+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde88009b70 con 0x7fde7c003d80 2019-12-05T06:56:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.618+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.618+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fde88009790 con 0x7fde7c003d80 2019-12-05T06:56:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.618+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.638 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.618+0000 7fde8ffff700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde7c02b9e0 0x7fde7c02de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.619+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fde88045cc0 con 0x7fde7c003d80 2019-12-05T06:56:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.619+0000 7fde977fe700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde7c02b9e0 0x7fde7c02de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.619+0000 7fde977fe700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde7c02b9e0 0x7fde7c02de10 crc :-1 s=READY pgs=668 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.624+0000 7fde9d9d9700 1 -- 172.21.15.16:0/1470828274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde70002e60 con 0x7fde7c003d80 2019-12-05T06:56:04.639 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.630+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:04.640 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.630+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fde88000980 con 0x7fde7c003d80 2019-12-05T06:56:04.824 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.825+0000 7fde9d9d9700 1 -- 172.21.15.16:0/1470828274 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fde70001d10 con 0x7fde7c003d80 2019-12-05T06:56:04.825 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.825+0000 7fde96ffd700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=THROTTLE_DONE pgs=1608 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).handle_read_frame_epilogue_main read frame epilogue bytes=32dumped fsmap epoch 196 2019-12-05T06:56:04.826 INFO:teuthology.orchestra.run.smithi016.stderr: 2019-12-05T06:56:04.826 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.825+0000 7fde8ffff700 1 -- 172.21.15.16:0/1470828274 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fde88009790 con 0x7fde7c003d80 2019-12-05T06:56:04.826 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:04.826 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:04.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.828+0000 7fde9d9d9700 1 -- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde7c02b9e0 msgr2=0x7fde7c02de10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:04.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.828+0000 7fde9d9d9700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fde7c02b9e0 0x7fde7c02de10 crc :-1 s=READY pgs=668 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:04.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.828+0000 7fde9d9d9700 1 -- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 msgr2=0x7fde7c004220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:04.828 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.828+0000 7fde9d9d9700 1 --2- 172.21.15.16:0/1470828274 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fde7c003d80 0x7fde7c004220 secure :-1 s=READY pgs=1608 cs=0 l=1 rx=0x7fde88010370 tx=0x7fde88010710).stop 2019-12-05T06:56:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.828+0000 7fde97fff700 1 -- 172.21.15.16:0/1470828274 reap_dead start 2019-12-05T06:56:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.828+0000 7fde9d9d9700 1 -- 172.21.15.16:0/1470828274 shutdown_connections 2019-12-05T06:56:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.829+0000 7fde9d9d9700 1 -- 172.21.15.16:0/1470828274 shutdown_connections 2019-12-05T06:56:04.829 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.829+0000 7fde9d9d9700 1 -- 172.21.15.16:0/1470828274 wait complete. 2019-12-05T06:56:04.830 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:04.829+0000 7fde9d9d9700 1 -- 172.21.15.16:0/1470828274 >> 172.21.15.16:0/1470828274 conn(0x7fde9809de30 msgr2=0x7fde98070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:04.838 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:56:06.839 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.072+0000 7fb1771f4700 1 Processor -- start 2019-12-05T06:56:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.072+0000 7fb1771f4700 1 -- start start 2019-12-05T06:56:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.073+0000 7fb1771f4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb17008fef0 0x7fb17008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.073+0000 7fb1771f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1700958e0 con 0x7fb17008c6d0 2019-12-05T06:56:07.078 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.073+0000 7fb1771f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1700629e0 con 0x7fb17008d2c0 2019-12-05T06:56:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.073+0000 7fb1771f4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb17008df90 con 0x7fb17008fef0 2019-12-05T06:56:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.073+0000 7fb175f92700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb17008fef0 0x7fb17008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.073+0000 7fb175791700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fb17008c6d0 0x7fb17008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49550/0 (socket says 172.21.15.16:49550) 2019-12-05T06:56:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.073+0000 7fb175791700 1 -- 172.21.15.16:0/3972532508 learned_addr learned my addr 172.21.15.16:0/3972532508 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb175f92700 1 -- 172.21.15.16:0/3972532508 >> v1:172.21.15.76:6789/0 conn(0x7fb17008d2c0 legacy=0x7fb17008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb175f92700 1 -- 172.21.15.16:0/3972532508 >> v1:172.21.15.16:6789/0 conn(0x7fb17008c6d0 legacy=0x7fb17008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:07.079 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb175f92700 1 -- 172.21.15.16:0/3972532508 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17008e620 con 0x7fb17008fef0 2019-12-05T06:56:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb167fff700 1 -- 172.21.15.16:0/3972532508 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 976119728 0 0) 0x7fb16c001150 con 0x7fb17008c6d0 2019-12-05T06:56:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb175f92700 1 --2- 172.21.15.16:0/3972532508 >> v2:172.21.15.16:3300/0 conn(0x7fb17008fef0 0x7fb17008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb16800b2d0 tx=0x7fb16800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb175f92700 1 --2- 172.21.15.16:0/3972532508 >> v2:172.21.15.16:3300/0 conn(0x7fb17008fef0 0x7fb17008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb16800b2d0 tx=0x7fb16800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb175f92700 1 --2- 172.21.15.16:0/3972532508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17008fef0 0x7fb17008f720 secure :-1 s=READY pgs=1610 cs=0 l=1 rx=0x7fb16800b2d0 tx=0x7fb16800bae0).ready entity=mon.0 client_cookie=ab64c82c9d935f5c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb175f92700 1 --2- 172.21.15.16:0/3972532508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17008fef0 0x7fb17008f720 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7fb16800b2d0 tx=0x7fb16800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb167fff700 1 -- 172.21.15.16:0/3972532508 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb168009f20 con 0x7fb17008fef0 2019-12-05T06:56:07.080 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb175f92700 1 --2- 172.21.15.16:0/3972532508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17008fef0 0x7fb17008f720 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7fb16800b2d0 tx=0x7fb16800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb175f92700 1 --2- 172.21.15.16:0/3972532508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17008fef0 0x7fb17008f720 secure :-1 s=THROTTLE_DONE pgs=1610 cs=0 l=1 rx=0x7fb16800b2d0 tx=0x7fb16800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb167fff700 1 -- 172.21.15.16:0/3972532508 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb168000df0 con 0x7fb17008fef0 2019-12-05T06:56:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.074+0000 7fb167fff700 1 -- 172.21.15.16:0/3972532508 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb16800a310 con 0x7fb17008fef0 2019-12-05T06:56:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.075+0000 7fb1771f4700 1 -- 172.21.15.16:0/3972532508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17008fef0 msgr2=0x7fb17008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.075+0000 7fb1771f4700 1 --2- 172.21.15.16:0/3972532508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17008fef0 0x7fb17008f720 secure :-1 s=READY pgs=1610 cs=0 l=1 rx=0x7fb16800b2d0 tx=0x7fb16800bae0).stop 2019-12-05T06:56:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.075+0000 7fb1771f4700 1 -- 172.21.15.16:0/3972532508 shutdown_connections 2019-12-05T06:56:07.081 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.075+0000 7fb1771f4700 1 --2- 172.21.15.16:0/3972532508 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17008fef0 0x7fb17008f720 unknown :-1 s=CLOSED pgs=1610 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.076+0000 7fb1771f4700 1 -- 172.21.15.16:0/3972532508 shutdown_connections 2019-12-05T06:56:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.076+0000 7fb1771f4700 1 -- 172.21.15.16:0/3972532508 wait complete. 2019-12-05T06:56:07.082 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.076+0000 7fb1771f4700 1 -- 172.21.15.16:0/3972532508 >> 172.21.15.16:0/3972532508 conn(0x7fb170061940 msgr2=0x7fb170096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:07.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.079+0000 7fb1771f4700 1 Processor -- start 2019-12-05T06:56:07.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.079+0000 7fb1771f4700 1 -- start start 2019-12-05T06:56:07.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb1771f4700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb17009b850 0x7fb170131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:07.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb1771f4700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb170098da0 con 0x7fb170132d00 2019-12-05T06:56:07.083 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb1771f4700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb17009f0a0 con 0x7fb170154b00 2019-12-05T06:56:07.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb1771f4700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb17009aa60 con 0x7fb17009b850 2019-12-05T06:56:07.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb175791700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb17009b850 0x7fb170131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb175791700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb17009b850 0x7fb170131d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36156/0 (socket says 172.21.15.16:36156) 2019-12-05T06:56:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb175791700 1 -- 172.21.15.16:0/3468534255 learned_addr learned my addr 172.21.15.16:0/3468534255 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:07.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb175791700 1 -- 172.21.15.16:0/3468534255 >> v1:172.21.15.76:6789/0 conn(0x7fb170154b00 legacy=0x7fb170156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb175791700 1 -- 172.21.15.16:0/3468534255 >> v1:172.21.15.16:6789/0 conn(0x7fb170132d00 legacy=0x7fb170132270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:07.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb175791700 1 -- 172.21.15.16:0/3468534255 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb17009b0f0 con 0x7fb17009b850 2019-12-05T06:56:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb165ffb700 1 -- 172.21.15.16:0/3468534255 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498749402 0 0) 0x7fb160001080 con 0x7fb170132d00 2019-12-05T06:56:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.080+0000 7fb175791700 1 --2- 172.21.15.16:0/3468534255 >> v2:172.21.15.16:3300/0 conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.081+0000 7fb175791700 1 --2- 172.21.15.16:0/3468534255 >> v2:172.21.15.16:3300/0 conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.081+0000 7fb175791700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=READY pgs=1612 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.081+0000 7fb175791700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=THROTTLE_DONE pgs=1612 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.081+0000 7fb175791700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=THROTTLE_DONE pgs=1612 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.081+0000 7fb165ffb700 1 -- 172.21.15.16:0/3468534255 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb16c000ed0 con 0x7fb17009b850 2019-12-05T06:56:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.081+0000 7fb175791700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=THROTTLE_DONE pgs=1612 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.081+0000 7fb165ffb700 1 -- 172.21.15.16:0/3468534255 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb16c004430 con 0x7fb17009b850 2019-12-05T06:56:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.081+0000 7fb1771f4700 1 -- 172.21.15.16:0/3468534255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb17009b0f0 con 0x7fb17009b850 2019-12-05T06:56:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.081+0000 7fb165ffb700 1 -- 172.21.15.16:0/3468534255 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb16c004a80 con 0x7fb17009b850 2019-12-05T06:56:07.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.081+0000 7fb1771f4700 1 -- 172.21.15.16:0/3468534255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb17009b0f0 con 0x7fb17009b850 2019-12-05T06:56:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.082+0000 7fb175791700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=THROTTLE_DONE pgs=1612 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.082+0000 7fb165ffb700 1 -- 172.21.15.16:0/3468534255 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb16c004620 con 0x7fb17009b850 2019-12-05T06:56:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.082+0000 7fb175791700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=THROTTLE_DONE pgs=1612 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.082+0000 7fb165ffb700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb150023630 0x7fb150025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.082+0000 7fb165ffb700 1 -- 172.21.15.16:0/3468534255 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fb16c045ed0 con 0x7fb17009b850 2019-12-05T06:56:07.092 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.082+0000 7fb174f90700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb150023630 0x7fb150025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:07.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.083+0000 7fb174f90700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb150023630 0x7fb150025ac0 crc :-1 s=READY pgs=669 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:07.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.087+0000 7fb1771f4700 1 -- 172.21.15.16:0/3468534255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb17009b0f0 con 0x7fb17009b850 2019-12-05T06:56:07.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.092+0000 7fb175791700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=THROTTLE_DONE pgs=1612 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.092+0000 7fb165ffb700 1 -- 172.21.15.16:0/3468534255 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb16c0012b0 con 0x7fb17009b850 2019-12-05T06:56:07.305 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.305+0000 7fb1771f4700 1 -- 172.21.15.16:0/3468534255 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb17009b0f0 con 0x7fb17009b850 2019-12-05T06:56:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.306+0000 7fb175791700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=THROTTLE_DONE pgs=1612 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.306+0000 7fb165ffb700 1 -- 172.21.15.16:0/3468534255 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 196 v196) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fb16c004430 con 0x7fb17009b850 2019-12-05T06:56:07.307 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 196 2019-12-05T06:56:07.307 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:07.308 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":196,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.309+0000 7fb1771f4700 1 -- 172.21.15.16:0/3468534255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb150023630 msgr2=0x7fb150025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.309+0000 7fb1771f4700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb150023630 0x7fb150025ac0 crc :-1 s=READY pgs=669 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:07.309 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.309+0000 7fb1771f4700 1 -- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 msgr2=0x7fb170131d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.309+0000 7fb1771f4700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 0x7fb170131d30 secure :-1 s=READY pgs=1612 cs=0 l=1 rx=0x7fb16c010430 tx=0x7fb16c0107d0).stop 2019-12-05T06:56:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.309+0000 7fb1771f4700 1 -- 172.21.15.16:0/3468534255 shutdown_connections 2019-12-05T06:56:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.309+0000 7fb1771f4700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb150023630 0x7fb150025ac0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.309+0000 7fb1771f4700 1 --2- 172.21.15.16:0/3468534255 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb17009b850 0x7fb170131d30 unknown :-1 s=CLOSED pgs=1612 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.309+0000 7fb1771f4700 1 -- 172.21.15.16:0/3468534255 shutdown_connections 2019-12-05T06:56:07.310 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.309+0000 7fb1771f4700 1 -- 172.21.15.16:0/3468534255 wait complete. 2019-12-05T06:56:07.311 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:07.309+0000 7fb1771f4700 1 -- 172.21.15.16:0/3468534255 >> 172.21.15.16:0/3468534255 conn(0x7fb1700714f0 msgr2=0x7fb17009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:07.320 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:56:09.320 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.480+0000 7f621e3c2700 1 Processor -- start 2019-12-05T06:56:09.488 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.480+0000 7f621e3c2700 1 -- start start 2019-12-05T06:56:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.481+0000 7f621e3c2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f621808d2c0 0x7f621808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.481+0000 7f621e3c2700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180958e0 con 0x7f621808c6d0 2019-12-05T06:56:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.481+0000 7f621e3c2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62180629e0 con 0x7f621808fef0 2019-12-05T06:56:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.481+0000 7f621e3c2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f621808df90 con 0x7f621808d2c0 2019-12-05T06:56:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f621c95f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f621808c6d0 0x7f621808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49598/0 (socket says 172.21.15.16:49598) 2019-12-05T06:56:09.489 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f621c95f700 1 -- 172.21.15.16:0/1098411241 learned_addr learned my addr 172.21.15.16:0/1098411241 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f6217fff700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f621808d2c0 0x7f621808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1762446650 0 0) 0x7f6200001170 con 0x7f621808c6d0 2019-12-05T06:56:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6204002e80 con 0x7f621808c6d0 2019-12-05T06:56:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f621d160700 1 -- 172.21.15.16:0/1098411241 >> v1:172.21.15.76:6789/0 conn(0x7f621808fef0 legacy=0x7f621808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2019-12-05T06:56:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f621d160700 1 -- 172.21.15.16:0/1098411241 >> v1:172.21.15.76:6789/0 conn(0x7f621808fef0 legacy=0x7f621808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:56:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f621d160700 1 --1- 172.21.15.16:0/1098411241 >> v1:172.21.15.76:6789/0 conn(0x7f621808fef0 0x7f621808f460 :-1 s=OPENED pgs=1081 cs=1 l=1).handle_message read tag failed 2019-12-05T06:56:09.490 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f621d160700 1 --1- 172.21.15.16:0/1098411241 >> v1:172.21.15.76:6789/0 conn(0x7f621808fef0 0x7f621808f460 :-1 s=OPENED pgs=1081 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:56:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 4112637970 0 0) 0x7f6200001130 con 0x7f621808c6d0 2019-12-05T06:56:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f621808d2c0 msgr2=0x7f621808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f62177fe700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f621808d2c0 0x7f621808ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.482+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 >> v1:172.21.15.76:6789/0 conn(0x7f621808fef0 legacy=0x7f621808f460 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:56:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f621808e620 con 0x7f621808c6d0 2019-12-05T06:56:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f6200001660 con 0x7f621808c6d0 2019-12-05T06:56:09.491 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 >> v1:172.21.15.16:6789/0 conn(0x7f621808c6d0 legacy=0x7f621808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f62177fe700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62040039b0 0x7f6204005e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f62177fe700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f62040065c0 0x7f6204008a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f62177fe700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620404d250 0x7f620404f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f620404ff30 con 0x7f620404d250 2019-12-05T06:56:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6204050120 con 0x7f62040039b0 2019-12-05T06:56:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6204050350 con 0x7f62040065c0 2019-12-05T06:56:09.492 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f621d160700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620404d250 0x7f620404f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f621c95f700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62040039b0 0x7f6204005e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f6217fff700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f62040065c0 0x7f6204008a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f621d160700 1 -- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f62040065c0 msgr2=0x7f6204008a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f621d160700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f62040065c0 0x7f6204008a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.483+0000 7f621d160700 1 -- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62040039b0 msgr2=0x7f6204005e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:09.493 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621d160700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f62040039b0 0x7f6204005e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621d160700 1 -- 172.21.15.16:0/1098411241 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6204050600 con 0x7f620404d250 2019-12-05T06:56:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621d160700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620404d250 0x7f620404f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f620800b3e0 tx=0x7f620800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621d160700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620404d250 0x7f620404f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f620800b3e0 tx=0x7f620800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621d160700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620404d250 0x7f620404f720 secure :-1 s=READY pgs=1614 cs=0 l=1 rx=0x7f620800b3e0 tx=0x7f620800b6e0).ready entity=mon.0 client_cookie=4b71db511227bcf server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621d160700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620404d250 0x7f620404f720 secure :-1 s=THROTTLE_DONE pgs=1614 cs=0 l=1 rx=0x7f620800b3e0 tx=0x7f620800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621d160700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620404d250 0x7f620404f720 secure :-1 s=THROTTLE_DONE pgs=1614 cs=0 l=1 rx=0x7f620800b3e0 tx=0x7f620800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62080011f0 con 0x7f620404d250 2019-12-05T06:56:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621d160700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620404d250 0x7f620404f720 secure :-1 s=THROTTLE_DONE pgs=1614 cs=0 l=1 rx=0x7f620800b3e0 tx=0x7f620800b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6208000de0 con 0x7f620404d250 2019-12-05T06:56:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f62177fe700 1 -- 172.21.15.16:0/1098411241 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62080082c0 con 0x7f620404d250 2019-12-05T06:56:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621e3c2700 1 -- 172.21.15.16:0/1098411241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620404d250 msgr2=0x7f620404f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621e3c2700 1 --2- 172.21.15.16:0/1098411241 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f620404d250 0x7f620404f720 secure :-1 s=READY pgs=1614 cs=0 l=1 rx=0x7f620800b3e0 tx=0x7f620800b6e0).stop 2019-12-05T06:56:09.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.484+0000 7f621d160700 1 -- 172.21.15.16:0/1098411241 reap_dead start 2019-12-05T06:56:09.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.485+0000 7f621e3c2700 1 -- 172.21.15.16:0/1098411241 shutdown_connections 2019-12-05T06:56:09.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.485+0000 7f621e3c2700 1 -- 172.21.15.16:0/1098411241 shutdown_connections 2019-12-05T06:56:09.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.485+0000 7f621e3c2700 1 -- 172.21.15.16:0/1098411241 wait complete. 2019-12-05T06:56:09.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.485+0000 7f621e3c2700 1 -- 172.21.15.16:0/1098411241 >> 172.21.15.16:0/1098411241 conn(0x7f6218061940 msgr2=0x7f6218096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.489+0000 7f621e3c2700 1 Processor -- start 2019-12-05T06:56:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.489+0000 7f621e3c2700 1 -- start start 2019-12-05T06:56:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f621e3c2700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f621806e840 0x7f6218134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f621e3c2700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6218135450 0x7f62181349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f621e3c2700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6218128af0 con 0x7f621806dc50 2019-12-05T06:56:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f621e3c2700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6218132830 con 0x7f621806e840 2019-12-05T06:56:09.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f621e3c2700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62181334f0 con 0x7f6218135450 2019-12-05T06:56:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f6217fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f621806e840 0x7f6218134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f6217fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f621806e840 0x7f6218134480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36208/0 (socket says 172.21.15.16:36208) 2019-12-05T06:56:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f6217fff700 1 -- 172.21.15.16:0/3990031914 learned_addr learned my addr 172.21.15.16:0/3990031914 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f621d160700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6218135450 0x7f62181349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f6217fff700 1 -- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6218135450 msgr2=0x7f62181349c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6218135450 0x7f62181349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:09.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f6217fff700 1 -- 172.21.15.16:0/3990031914 >> v1:172.21.15.76:6789/0 conn(0x7f621806dc50 legacy=0x7f621806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f6217fff700 1 -- 172.21.15.16:0/3990031914 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6218133b80 con 0x7f621806e840 2019-12-05T06:56:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.491+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> v2:172.21.15.16:3300/0 conn(0x7f621806e840 0x7f6218134480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6210010400 tx=0x7f62100102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> v2:172.21.15.16:3300/0 conn(0x7f621806e840 0x7f6218134480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6210010400 tx=0x7f62100102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 0x7f6218134480 secure :-1 s=READY pgs=1615 cs=0 l=1 rx=0x7f6210010400 tx=0x7f62100102b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 0x7f6218134480 secure :-1 s=THROTTLE_DONE pgs=1615 cs=0 l=1 rx=0x7f6210010400 tx=0x7f62100102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 0x7f6218134480 secure :-1 s=THROTTLE_DONE pgs=1615 cs=0 l=1 rx=0x7f6210010400 tx=0x7f62100102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f621000a7f0 con 0x7f621806e840 2019-12-05T06:56:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 0x7f6218134480 secure :-1 s=THROTTLE_DONE pgs=1615 cs=0 l=1 rx=0x7f6210010400 tx=0x7f62100102b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6210009580 con 0x7f621806e840 2019-12-05T06:56:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f621e3c2700 1 -- 172.21.15.16:0/3990031914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6218133b80 con 0x7f621806e840 2019-12-05T06:56:09.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6210009de0 con 0x7f621806e840 2019-12-05T06:56:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f621e3c2700 1 -- 172.21.15.16:0/3990031914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6218133b80 con 0x7f621806e840 2019-12-05T06:56:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f6217fff700 1 -- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 msgr2=0x7f6218134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 19 2019-12-05T06:56:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f6217fff700 1 -- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 msgr2=0x7f6218134480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:56:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 0x7f6218134480 secure :-1 s=READY pgs=1615 cs=0 l=1 rx=0x7f6210010400 tx=0x7f62100102b0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:56:09.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 0x7f6218134480 secure :-1 s=READY pgs=1615 cs=0 l=1 rx=0x7f6210010400 tx=0x7f62100102b0).stop 2019-12-05T06:56:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 msgr2=0x7f6218134480 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:56:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 0x7f6218134480 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f61fc006020 0x7f61fc0084f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61fc008cf0 0x7f61fc00b1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc00b9d0 con 0x7f61fc003410 2019-12-05T06:56:09.508 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc00bc00 con 0x7f61fc008cf0 2019-12-05T06:56:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.492+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc00be30 con 0x7f61fc006020 2019-12-05T06:56:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.493+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.493+0000 7f621d160700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61fc008cf0 0x7f61fc00b1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.493+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f61fc006020 0x7f61fc0084f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:09.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.493+0000 7f621c95f700 1 -- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f61fc006020 msgr2=0x7f61fc0084f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.493+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f61fc006020 0x7f61fc0084f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.493+0000 7f621c95f700 1 -- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61fc008cf0 msgr2=0x7f61fc00b1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.493+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f61fc008cf0 0x7f61fc00b1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.493+0000 7f621c95f700 1 -- 172.21.15.16:0/3990031914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f61fc00c160 con 0x7f61fc003410 2019-12-05T06:56:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.493+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6200010370 tx=0x7f6200010220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.494+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6200010370 tx=0x7f6200010220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.494+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 secure :-1 s=READY pgs=1616 cs=0 l=1 rx=0x7f6200010370 tx=0x7f6200010220).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.494+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7f6200010370 tx=0x7f6200010220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.494+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6200004430 con 0x7f61fc003410 2019-12-05T06:56:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.494+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7f6200010370 tx=0x7f6200010220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.494+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6200001640 con 0x7f61fc003410 2019-12-05T06:56:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.494+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7f6200010370 tx=0x7f6200010220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.495+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6200004240 con 0x7f61fc003410 2019-12-05T06:56:09.511 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.495+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7f6200010370 tx=0x7f6200010220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.495+0000 7f621e3c2700 1 -- 172.21.15.16:0/3990031914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6218133b80 con 0x7f61fc003410 2019-12-05T06:56:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.495+0000 7f62157fa700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61fc02b990 0x7f61fc02dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.495+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 <== mon.0 v2:172.21.15.16:3300/0 4 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f6200045cc0 con 0x7f61fc003410 2019-12-05T06:56:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.495+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61fc02b990 0x7f61fc02dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.496+0000 7f6217fff700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61fc02b990 0x7f61fc02dde0 crc :-1 s=READY pgs=671 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.500+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7f6200010370 tx=0x7f6200010220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.500+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 <== mon.0 v2:172.21.15.16:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6200001b00 con 0x7f61fc003410 2019-12-05T06:56:09.733 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.732+0000 7f621e3c2700 1 -- 172.21.15.16:0/3990031914 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6218133b80 con 0x7f61fc003410 2019-12-05T06:56:09.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.733+0000 7f621c95f700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 secure :-1 s=THROTTLE_DONE pgs=1616 cs=0 l=1 rx=0x7f6200010370 tx=0x7f6200010220).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:09.734 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.733+0000 7f62157fa700 1 -- 172.21.15.16:0/3990031914 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f6200004af0 con 0x7f61fc003410 2019-12-05T06:56:09.734 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 197 2019-12-05T06:56:09.734 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:09.735 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8308,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":197}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.735+0000 7f621e3c2700 1 -- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61fc02b990 msgr2=0x7f61fc02dde0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.736+0000 7f621e3c2700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f61fc02b990 0x7f61fc02dde0 crc :-1 s=READY pgs=671 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:09.736 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.736+0000 7f621e3c2700 1 -- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 msgr2=0x7f61fc005870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.736+0000 7f621e3c2700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f61fc003410 0x7f61fc005870 secure :-1 s=READY pgs=1616 cs=0 l=1 rx=0x7f6200010370 tx=0x7f6200010220).stop 2019-12-05T06:56:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.736+0000 7f621d160700 1 -- 172.21.15.16:0/3990031914 reap_dead start 2019-12-05T06:56:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.736+0000 7f621d160700 1 -- 172.21.15.16:0/3990031914 reap_dead start 2019-12-05T06:56:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.736+0000 7f621e3c2700 1 -- 172.21.15.16:0/3990031914 shutdown_connections 2019-12-05T06:56:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.736+0000 7f621e3c2700 1 --2- 172.21.15.16:0/3990031914 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f621806e840 0x7f6218134480 unknown :-1 s=CLOSED pgs=1615 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.736+0000 7f621e3c2700 1 -- 172.21.15.16:0/3990031914 shutdown_connections 2019-12-05T06:56:09.737 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.736+0000 7f621e3c2700 1 -- 172.21.15.16:0/3990031914 wait complete. 2019-12-05T06:56:09.738 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:09.736+0000 7f621e3c2700 1 -- 172.21.15.16:0/3990031914 >> 172.21.15.16:0/3990031914 conn(0x7f62180a2850 msgr2=0x7f621806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:09.747 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:56:09.747 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 23 secs before thrashing 2019-12-05T06:56:17.149 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:56:17.154 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:56:32.748 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.984+0000 7fc54ea08700 1 Processor -- start 2019-12-05T06:56:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.984+0000 7fc54ea08700 1 -- start start 2019-12-05T06:56:32.995 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.986+0000 7fc54ea08700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc54808d2c0 0x7fc54808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.986+0000 7fc54ea08700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5480958e0 con 0x7fc54808fef0 2019-12-05T06:56:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.986+0000 7fc54ea08700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5480629e0 con 0x7fc54808c6d0 2019-12-05T06:56:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.986+0000 7fc54ea08700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54808df90 con 0x7fc54808d2c0 2019-12-05T06:56:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.987+0000 7fc54d7a6700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc54808fef0 0x7fc54808f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49708/0 (socket says 172.21.15.16:49708) 2019-12-05T06:56:32.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.987+0000 7fc54d7a6700 1 -- 172.21.15.16:0/2320020437 learned_addr learned my addr 172.21.15.16:0/2320020437 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.987+0000 7fc547fff700 1 --2- 172.21.15.16:0/2320020437 >> v2:172.21.15.76:3300/0 conn(0x7fc54808d2c0 0x7fc54808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.987+0000 7fc5477fe700 1 -- 172.21.15.16:0/2320020437 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 510461166 0 0) 0x7fc5380014d0 con 0x7fc54808fef0 2019-12-05T06:56:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.987+0000 7fc5477fe700 1 -- 172.21.15.16:0/2320020437 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc534002e80 con 0x7fc54808fef0 2019-12-05T06:56:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.987+0000 7fc5477fe700 1 -- 172.21.15.16:0/2320020437 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3959053177 0 0) 0x7fc540001080 con 0x7fc54808c6d0 2019-12-05T06:56:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.987+0000 7fc5477fe700 1 -- 172.21.15.16:0/2320020437 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc534004170 con 0x7fc54808c6d0 2019-12-05T06:56:32.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.988+0000 7fc547fff700 1 -- 172.21.15.16:0/2320020437 >> v1:172.21.15.76:6789/0 conn(0x7fc54808c6d0 legacy=0x7fc54808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.988+0000 7fc547fff700 1 -- 172.21.15.16:0/2320020437 >> v1:172.21.15.16:6789/0 conn(0x7fc54808fef0 legacy=0x7fc54808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.988+0000 7fc547fff700 1 -- 172.21.15.16:0/2320020437 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54808e620 con 0x7fc54808d2c0 2019-12-05T06:56:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.988+0000 7fc547fff700 1 --2- 172.21.15.16:0/2320020437 >> v2:172.21.15.76:3300/0 conn(0x7fc54808d2c0 0x7fc54808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc53000af50 tx=0x7fc53000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.988+0000 7fc547fff700 1 --2- 172.21.15.16:0/2320020437 >> v2:172.21.15.76:3300/0 conn(0x7fc54808d2c0 0x7fc54808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc53000af50 tx=0x7fc53000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.988+0000 7fc547fff700 1 --2- 172.21.15.16:0/2320020437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc54808d2c0 0x7fc54808ef20 secure :-1 s=READY pgs=1086 cs=0 l=1 rx=0x7fc53000af50 tx=0x7fc53000b760).ready entity=mon.1 client_cookie=df9a923fbb887c5a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:32.998 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.988+0000 7fc547fff700 1 --2- 172.21.15.16:0/2320020437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc54808d2c0 0x7fc54808ef20 secure :-1 s=THROTTLE_DONE pgs=1086 cs=0 l=1 rx=0x7fc53000af50 tx=0x7fc53000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.988+0000 7fc547fff700 1 --2- 172.21.15.16:0/2320020437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc54808d2c0 0x7fc54808ef20 secure :-1 s=THROTTLE_DONE pgs=1086 cs=0 l=1 rx=0x7fc53000af50 tx=0x7fc53000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.988+0000 7fc5477fe700 1 -- 172.21.15.16:0/2320020437 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc530000960 con 0x7fc54808d2c0 2019-12-05T06:56:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.988+0000 7fc547fff700 1 --2- 172.21.15.16:0/2320020437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc54808d2c0 0x7fc54808ef20 secure :-1 s=THROTTLE_DONE pgs=1086 cs=0 l=1 rx=0x7fc53000af50 tx=0x7fc53000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.989+0000 7fc5477fe700 1 -- 172.21.15.16:0/2320020437 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5300098e0 con 0x7fc54808d2c0 2019-12-05T06:56:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.989+0000 7fc5477fe700 1 -- 172.21.15.16:0/2320020437 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc530009ef0 con 0x7fc54808d2c0 2019-12-05T06:56:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.989+0000 7fc54ea08700 1 -- 172.21.15.16:0/2320020437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc54808d2c0 msgr2=0x7fc54808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:32.999 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.989+0000 7fc54ea08700 1 --2- 172.21.15.16:0/2320020437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc54808d2c0 0x7fc54808ef20 secure :-1 s=READY pgs=1086 cs=0 l=1 rx=0x7fc53000af50 tx=0x7fc53000b760).stop 2019-12-05T06:56:33.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.990+0000 7fc54ea08700 1 -- 172.21.15.16:0/2320020437 shutdown_connections 2019-12-05T06:56:33.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.990+0000 7fc54ea08700 1 --2- 172.21.15.16:0/2320020437 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc54808d2c0 0x7fc54808ef20 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:33.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.990+0000 7fc54ea08700 1 -- 172.21.15.16:0/2320020437 shutdown_connections 2019-12-05T06:56:33.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.991+0000 7fc54ea08700 1 -- 172.21.15.16:0/2320020437 wait complete. 2019-12-05T06:56:33.000 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.991+0000 7fc54ea08700 1 -- 172.21.15.16:0/2320020437 >> 172.21.15.16:0/2320020437 conn(0x7fc548061940 msgr2=0x7fc548096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:33.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.996+0000 7fc54ea08700 1 Processor -- start 2019-12-05T06:56:33.001 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.996+0000 7fc54ea08700 1 -- start start 2019-12-05T06:56:33.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.998+0000 7fc54ea08700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc548128360 0x7fc548128750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:33.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.998+0000 7fc54ea08700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc548099720 con 0x7fc548137ea0 2019-12-05T06:56:33.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.998+0000 7fc54ea08700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5480986d0 con 0x7fc54812bdd0 2019-12-05T06:56:33.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.998+0000 7fc54ea08700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc54812a640 con 0x7fc548128360 2019-12-05T06:56:33.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.998+0000 7fc54cfa5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc548128360 0x7fc548128750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.999+0000 7fc54cfa5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc548128360 0x7fc548128750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36310/0 (socket says 172.21.15.16:36310) 2019-12-05T06:56:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.999+0000 7fc54cfa5700 1 -- 172.21.15.16:0/1093519152 learned_addr learned my addr 172.21.15.16:0/1093519152 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:33.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.999+0000 7fc5457fa700 1 -- 172.21.15.16:0/1093519152 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 804323432 0 0) 0x7fc538000d30 con 0x7fc54812bdd0 2019-12-05T06:56:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.999+0000 7fc54cfa5700 1 -- 172.21.15.16:0/1093519152 >> v1:172.21.15.76:6789/0 conn(0x7fc54812bdd0 legacy=0x7fc548152760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.999+0000 7fc54cfa5700 1 -- 172.21.15.16:0/1093519152 >> v1:172.21.15.16:6789/0 conn(0x7fc548137ea0 legacy=0x7fc54812b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:32.999+0000 7fc54cfa5700 1 -- 172.21.15.16:0/1093519152 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc54812acd0 con 0x7fc548128360 2019-12-05T06:56:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc54cfa5700 1 --2- 172.21.15.16:0/1093519152 >> v2:172.21.15.16:3300/0 conn(0x7fc548128360 0x7fc548128750 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc54cfa5700 1 --2- 172.21.15.16:0/1093519152 >> v2:172.21.15.16:3300/0 conn(0x7fc548128360 0x7fc548128750 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc54cfa5700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 0x7fc548128750 secure :-1 s=READY pgs=1619 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:33.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc54cfa5700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 0x7fc548128750 secure :-1 s=THROTTLE_DONE pgs=1619 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc5457fa700 1 -- 172.21.15.16:0/1093519152 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc540005810 con 0x7fc548128360 2019-12-05T06:56:33.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc54cfa5700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 0x7fc548128750 secure :-1 s=THROTTLE_DONE pgs=1619 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc54cfa5700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 0x7fc548128750 secure :-1 s=THROTTLE_DONE pgs=1619 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc5457fa700 1 -- 172.21.15.16:0/1093519152 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc540005390 con 0x7fc548128360 2019-12-05T06:56:33.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc54ea08700 1 -- 172.21.15.16:0/1093519152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc54812acd0 con 0x7fc548128360 2019-12-05T06:56:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc5457fa700 1 -- 172.21.15.16:0/1093519152 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc540005df0 con 0x7fc548128360 2019-12-05T06:56:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.000+0000 7fc54ea08700 1 -- 172.21.15.16:0/1093519152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc54812acd0 con 0x7fc548128360 2019-12-05T06:56:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.001+0000 7fc54cfa5700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 0x7fc548128750 secure :-1 s=THROTTLE_DONE pgs=1619 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.002+0000 7fc5457fa700 1 -- 172.21.15.16:0/1093519152 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc540005580 con 0x7fc548128360 2019-12-05T06:56:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.002+0000 7fc54cfa5700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 0x7fc548128750 secure :-1 s=THROTTLE_DONE pgs=1619 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.008 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.003+0000 7fc5457fa700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc52c023630 0x7fc52c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:33.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.003+0000 7fc5457fa700 1 -- 172.21.15.16:0/1093519152 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fc540045e20 con 0x7fc548128360 2019-12-05T06:56:33.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.003+0000 7fc547fff700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc52c023630 0x7fc52c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:33.009 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.004+0000 7fc547fff700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc52c023630 0x7fc52c025ac0 crc :-1 s=READY pgs=673 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:33.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.016+0000 7fc54ea08700 1 -- 172.21.15.16:0/1093519152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc54812acd0 con 0x7fc548128360 2019-12-05T06:56:33.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.024+0000 7fc54cfa5700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 0x7fc548128750 secure :-1 s=THROTTLE_DONE pgs=1619 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.024+0000 7fc5457fa700 1 -- 172.21.15.16:0/1093519152 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc540019170 con 0x7fc548128360 2019-12-05T06:56:33.292 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.291+0000 7fc54ea08700 1 -- 172.21.15.16:0/1093519152 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc54812c1f0 con 0x7fc548128360 2019-12-05T06:56:33.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.292+0000 7fc54cfa5700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 0x7fc548128750 secure :-1 s=THROTTLE_DONE pgs=1619 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.293 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.292+0000 7fc5457fa700 1 -- 172.21.15.16:0/1093519152 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fc540019170 con 0x7fc548128360 2019-12-05T06:56:33.294 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 197 2019-12-05T06:56:33.294 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:33.294 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8308,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":197}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:33.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.294+0000 7fc54ea08700 1 -- 172.21.15.16:0/1093519152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc52c023630 msgr2=0x7fc52c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:33.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.294+0000 7fc54ea08700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc52c023630 0x7fc52c025ac0 crc :-1 s=READY pgs=673 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:33.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.294+0000 7fc54ea08700 1 -- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 msgr2=0x7fc548128750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:33.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.295+0000 7fc54ea08700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 0x7fc548128750 secure :-1 s=READY pgs=1619 cs=0 l=1 rx=0x7fc540010410 tx=0x7fc5400107b0).stop 2019-12-05T06:56:33.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.295+0000 7fc54ea08700 1 -- 172.21.15.16:0/1093519152 shutdown_connections 2019-12-05T06:56:33.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.295+0000 7fc54ea08700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc52c023630 0x7fc52c025ac0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:33.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.295+0000 7fc54ea08700 1 --2- 172.21.15.16:0/1093519152 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc548128360 0x7fc548128750 unknown :-1 s=CLOSED pgs=1619 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:33.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.295+0000 7fc54ea08700 1 -- 172.21.15.16:0/1093519152 shutdown_connections 2019-12-05T06:56:33.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.295+0000 7fc54ea08700 1 -- 172.21.15.16:0/1093519152 wait complete. 2019-12-05T06:56:33.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.295+0000 7fc54ea08700 1 -- 172.21.15.16:0/1093519152 >> 172.21.15.16:0/1093519152 conn(0x7fc54806d260 msgr2=0x7fc54809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:33.306 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:56:33.307 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:56:33.307 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:56:33.322 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:56:33.322 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:33.537 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.527+0000 7f73109ef700 1 Processor -- start 2019-12-05T06:56:33.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.527+0000 7f73109ef700 1 -- start start 2019-12-05T06:56:33.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.529+0000 7f73109ef700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f730c08c6d0 0x7f730c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:33.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.529+0000 7f73109ef700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 0x7f730c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:33.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.529+0000 7f73109ef700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c0958e0 con 0x7f730c08ef20 2019-12-05T06:56:33.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.529+0000 7f73109ef700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730c08df90 con 0x7f730c08c6d0 2019-12-05T06:56:33.538 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.529+0000 7f73109ef700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c0a3f70 con 0x7f730c08fef0 2019-12-05T06:56:33.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.530+0000 7f730b59e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f730c08ef20 0x7f730c08f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49724/0 (socket says 172.21.15.16:49724) 2019-12-05T06:56:33.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.530+0000 7f730b59e700 1 -- 172.21.15.16:0/2137897930 learned_addr learned my addr 172.21.15.16:0/2137897930 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:33.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.530+0000 7f730a59c700 1 --2- 172.21.15.16:0/2137897930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 0x7f730c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:33.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.530+0000 7f730ad9d700 1 --2- 172.21.15.16:0/2137897930 >> v2:172.21.15.76:3300/0 conn(0x7f730c08c6d0 0x7f730c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:33.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.530+0000 7f7309d9b700 1 -- 172.21.15.16:0/2137897930 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2438060116 0 0) 0x7f73040014d0 con 0x7f730c08ef20 2019-12-05T06:56:33.539 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.530+0000 7f7309d9b700 1 -- 172.21.15.16:0/2137897930 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f8002e80 con 0x7f730c08ef20 2019-12-05T06:56:33.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f730a59c700 1 -- 172.21.15.16:0/2137897930 >> v2:172.21.15.76:3300/0 conn(0x7f730c08c6d0 msgr2=0x7f730c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:33.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f730a59c700 1 --2- 172.21.15.16:0/2137897930 >> v2:172.21.15.76:3300/0 conn(0x7f730c08c6d0 0x7f730c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:33.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f730a59c700 1 -- 172.21.15.16:0/2137897930 >> v1:172.21.15.16:6789/0 conn(0x7f730c08ef20 legacy=0x7f730c08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:33.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f730a59c700 1 -- 172.21.15.16:0/2137897930 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c08d540 con 0x7f730c08fef0 2019-12-05T06:56:33.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f730a59c700 1 --2- 172.21.15.16:0/2137897930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 0x7f730c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f72f400af50 tx=0x7f72f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f730a59c700 1 --2- 172.21.15.16:0/2137897930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 0x7f730c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f72f400af50 tx=0x7f72f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.540 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f730a59c700 1 --2- 172.21.15.16:0/2137897930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 0x7f730c08d000 secure :-1 s=READY pgs=383 cs=0 l=1 rx=0x7f72f400af50 tx=0x7f72f400b760).ready entity=mon.2 client_cookie=c73daf2b287b004a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:33.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f730a59c700 1 --2- 172.21.15.16:0/2137897930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 0x7f730c08d000 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7f72f400af50 tx=0x7f72f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f7309d9b700 1 -- 172.21.15.16:0/2137897930 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72f4007210 con 0x7f730c08fef0 2019-12-05T06:56:33.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f730a59c700 1 --2- 172.21.15.16:0/2137897930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 0x7f730c08d000 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7f72f400af50 tx=0x7f72f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.531+0000 7f730a59c700 1 --2- 172.21.15.16:0/2137897930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 0x7f730c08d000 secure :-1 s=THROTTLE_DONE pgs=383 cs=0 l=1 rx=0x7f72f400af50 tx=0x7f72f400b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.532+0000 7f7309d9b700 1 -- 172.21.15.16:0/2137897930 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f40099b0 con 0x7f730c08fef0 2019-12-05T06:56:33.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.532+0000 7f7309d9b700 1 -- 172.21.15.16:0/2137897930 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72f400a080 con 0x7f730c08fef0 2019-12-05T06:56:33.541 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.532+0000 7f73109ef700 1 -- 172.21.15.16:0/2137897930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 msgr2=0x7f730c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:33.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.532+0000 7f73109ef700 1 --2- 172.21.15.16:0/2137897930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 0x7f730c08d000 secure :-1 s=READY pgs=383 cs=0 l=1 rx=0x7f72f400af50 tx=0x7f72f400b760).stop 2019-12-05T06:56:33.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.533+0000 7f73109ef700 1 -- 172.21.15.16:0/2137897930 shutdown_connections 2019-12-05T06:56:33.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.533+0000 7f73109ef700 1 --2- 172.21.15.16:0/2137897930 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c08fef0 0x7f730c08d000 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:33.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.533+0000 7f73109ef700 1 --2- 172.21.15.16:0/2137897930 >> v2:172.21.15.76:3300/0 conn(0x7f730c08c6d0 0x7f730c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:33.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.533+0000 7f73109ef700 1 -- 172.21.15.16:0/2137897930 shutdown_connections 2019-12-05T06:56:33.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.533+0000 7f73109ef700 1 -- 172.21.15.16:0/2137897930 wait complete. 2019-12-05T06:56:33.542 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.533+0000 7f73109ef700 1 -- 172.21.15.16:0/2137897930 >> 172.21.15.16:0/2137897930 conn(0x7f730c061940 msgr2=0x7f730c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.538+0000 7f73109ef700 1 Processor -- start 2019-12-05T06:56:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.538+0000 7f73109ef700 1 -- start start 2019-12-05T06:56:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.540+0000 7f73109ef700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c1302f0 0x7f730c12a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.540+0000 7f73109ef700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f730c12b640 0x7f730c12abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.540+0000 7f73109ef700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f730c131bf0 con 0x7f730c12b0f0 2019-12-05T06:56:33.546 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.540+0000 7f73109ef700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f730c128cb0 con 0x7f730c12b640 2019-12-05T06:56:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.540+0000 7f73109ef700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f730c128f80 con 0x7f730c1302f0 2019-12-05T06:56:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.540+0000 7f730a59c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f730c12b640 0x7f730c12abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.540+0000 7f730ad9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c1302f0 0x7f730c12a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.540+0000 7f730a59c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f730c12b640 0x7f730c12abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:50652/0 (socket says 172.21.15.16:50652) 2019-12-05T06:56:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.540+0000 7f730b59e700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f730c12b0f0 0x7f730c1306e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:42802/0 (socket says 172.21.15.16:42802) 2019-12-05T06:56:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.540+0000 7f730a59c700 1 -- 172.21.15.16:0/1993954337 learned_addr learned my addr 172.21.15.16:0/1993954337 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:33.547 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.541+0000 7f730a59c700 1 -- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c1302f0 msgr2=0x7f730c12a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.541+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c1302f0 0x7f730c12a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.541+0000 7f730a59c700 1 -- 172.21.15.16:0/1993954337 >> v1:172.21.15.76:6789/0 conn(0x7f730c12b0f0 legacy=0x7f730c1306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.541+0000 7f73037fe700 1 -- 172.21.15.16:0/1993954337 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1364519737 0 0) 0x7f7304000d30 con 0x7f730c12b0f0 2019-12-05T06:56:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.541+0000 7f730a59c700 1 -- 172.21.15.16:0/1993954337 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f730c129d70 con 0x7f730c12b640 2019-12-05T06:56:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.541+0000 7f730ad9d700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c1302f0 0x7f730c12a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:56:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.541+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> v2:172.21.15.76:3300/0 conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.548 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.541+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> v2:172.21.15.76:3300/0 conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.541+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=READY pgs=1089 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:33.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.542+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.542+0000 7f73037fe700 1 -- 172.21.15.16:0/1993954337 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72f4000940 con 0x7f730c12b640 2019-12-05T06:56:33.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.542+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.542+0000 7f73109ef700 1 -- 172.21.15.16:0/1993954337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f730c129d70 con 0x7f730c12b640 2019-12-05T06:56:33.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.542+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.549 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.542+0000 7f73037fe700 1 -- 172.21.15.16:0/1993954337 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72f4000940 con 0x7f730c12b640 2019-12-05T06:56:33.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.542+0000 7f73037fe700 1 -- 172.21.15.16:0/1993954337 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72f400b380 con 0x7f730c12b640 2019-12-05T06:56:33.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.542+0000 7f73109ef700 1 -- 172.21.15.16:0/1993954337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f730c129d70 con 0x7f730c12b640 2019-12-05T06:56:33.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.543+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.543+0000 7f73037fe700 1 -- 172.21.15.16:0/1993954337 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f72f4009790 con 0x7f730c12b640 2019-12-05T06:56:33.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.543+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.544+0000 7f73037fe700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72f0023630 0x7f72f0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:33.550 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.544+0000 7f73037fe700 1 -- 172.21.15.16:0/1993954337 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f72f4047fe0 con 0x7f730c12b640 2019-12-05T06:56:33.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.544+0000 7f730ad9d700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72f0023630 0x7f72f0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:33.551 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.545+0000 7f730ad9d700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72f0023630 0x7f72f0025ac0 crc :-1 s=READY pgs=674 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:33.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.559+0000 7f73109ef700 1 -- 172.21.15.16:0/1993954337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f730c129d70 con 0x7f730c12b640 2019-12-05T06:56:33.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.567+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.567+0000 7f73037fe700 1 -- 172.21.15.16:0/1993954337 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f72f4009580 con 0x7f730c12b640 2019-12-05T06:56:33.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.810+0000 7f73109ef700 1 -- 172.21.15.16:0/1993954337 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f730c129d70 con 0x7f730c12b640 2019-12-05T06:56:33.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.812+0000 7f730a59c700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=THROTTLE_DONE pgs=1089 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:33.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.812+0000 7f73037fe700 1 -- 172.21.15.16:0/1993954337 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f72f4009790 con 0x7f730c12b640 2019-12-05T06:56:33.814 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:33.814 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8308,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":197}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:33.815 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 197 2019-12-05T06:56:33.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 -- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72f0023630 msgr2=0x7f72f0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:33.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72f0023630 0x7f72f0025ac0 crc :-1 s=READY pgs=674 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:33.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 -- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 msgr2=0x7f730c12abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:33.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 0x7f730c12abb0 secure :-1 s=READY pgs=1089 cs=0 l=1 rx=0x7f72f4012650 tx=0x7f72f40129f0).stop 2019-12-05T06:56:33.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 -- 172.21.15.16:0/1993954337 shutdown_connections 2019-12-05T06:56:33.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f72f0023630 0x7f72f0025ac0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:33.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f730c12b640 0x7f730c12abb0 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:33.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 --2- 172.21.15.16:0/1993954337 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f730c1302f0 0x7f730c12a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:33.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 -- 172.21.15.16:0/1993954337 shutdown_connections 2019-12-05T06:56:33.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 -- 172.21.15.16:0/1993954337 wait complete. 2019-12-05T06:56:33.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:33.816+0000 7f73109ef700 1 -- 172.21.15.16:0/1993954337 >> 172.21.15.16:0/1993954337 conn(0x7f730c0a2850 msgr2=0x7f730c06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:33.828 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:56:35.829 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:36.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.118+0000 7f40f185d700 1 Processor -- start 2019-12-05T06:56:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.118+0000 7f40f185d700 1 -- start start 2019-12-05T06:56:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.119+0000 7f40f185d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40ec08d2c0 0x7f40ec08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.119+0000 7f40f185d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec08fef0 0x7f40ec08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.120+0000 7f40f185d700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec0958e0 con 0x7f40ec08c6d0 2019-12-05T06:56:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.120+0000 7f40f185d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ec08df90 con 0x7f40ec08d2c0 2019-12-05T06:56:36.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.120+0000 7f40f185d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec0a3f70 con 0x7f40ec08fef0 2019-12-05T06:56:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.120+0000 7f40eaffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40ec08d2c0 0x7f40ec08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.120+0000 7f40eaffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40ec08d2c0 0x7f40ec08ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36342/0 (socket says 172.21.15.16:36342) 2019-12-05T06:56:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.120+0000 7f40ebfff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec08fef0 0x7f40ec08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.120+0000 7f40eaffd700 1 -- 172.21.15.16:0/3691875916 learned_addr learned my addr 172.21.15.16:0/3691875916 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.120+0000 7f40eb7fe700 1 --1- 172.21.15.16:0/3691875916 >> v1:172.21.15.76:6789/0 conn(0x7f40ec08c6d0 0x7f40ec08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:42810/0 (socket says 172.21.15.16:42810) 2019-12-05T06:56:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40eaffd700 1 -- 172.21.15.16:0/3691875916 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec08fef0 msgr2=0x7f40ec08f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:36.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40eaffd700 1 --2- 172.21.15.16:0/3691875916 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec08fef0 0x7f40ec08f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40eaffd700 1 -- 172.21.15.16:0/3691875916 >> v1:172.21.15.76:6789/0 conn(0x7f40ec08c6d0 legacy=0x7f40ec08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40eaffd700 1 -- 172.21.15.16:0/3691875916 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec08e620 con 0x7f40ec08d2c0 2019-12-05T06:56:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40ea7fc700 1 -- 172.21.15.16:0/3691875916 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401196512 0 0) 0x7f40e4001400 con 0x7f40ec08c6d0 2019-12-05T06:56:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40eaffd700 1 --2- 172.21.15.16:0/3691875916 >> v2:172.21.15.16:3300/0 conn(0x7f40ec08d2c0 0x7f40ec08ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f40d400f6c0 tx=0x7f40d400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40eaffd700 1 --2- 172.21.15.16:0/3691875916 >> v2:172.21.15.16:3300/0 conn(0x7f40ec08d2c0 0x7f40ec08ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f40d400f6c0 tx=0x7f40d400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40eaffd700 1 --2- 172.21.15.16:0/3691875916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08d2c0 0x7f40ec08ef20 secure :-1 s=READY pgs=1621 cs=0 l=1 rx=0x7f40d400f6c0 tx=0x7f40d400fa60).ready entity=mon.0 client_cookie=f911a9a208ecc072 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40eaffd700 1 --2- 172.21.15.16:0/3691875916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08d2c0 0x7f40ec08ef20 secure :-1 s=THROTTLE_DONE pgs=1621 cs=0 l=1 rx=0x7f40d400f6c0 tx=0x7f40d400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40eaffd700 1 --2- 172.21.15.16:0/3691875916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08d2c0 0x7f40ec08ef20 secure :-1 s=THROTTLE_DONE pgs=1621 cs=0 l=1 rx=0x7f40d400f6c0 tx=0x7f40d400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40ea7fc700 1 -- 172.21.15.16:0/3691875916 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40d4000960 con 0x7f40ec08d2c0 2019-12-05T06:56:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.121+0000 7f40eaffd700 1 --2- 172.21.15.16:0/3691875916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08d2c0 0x7f40ec08ef20 secure :-1 s=THROTTLE_DONE pgs=1621 cs=0 l=1 rx=0x7f40d400f6c0 tx=0x7f40d400fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.122+0000 7f40ea7fc700 1 -- 172.21.15.16:0/3691875916 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40d4000c90 con 0x7f40ec08d2c0 2019-12-05T06:56:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.122+0000 7f40ea7fc700 1 -- 172.21.15.16:0/3691875916 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40d40013a0 con 0x7f40ec08d2c0 2019-12-05T06:56:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.122+0000 7f40f185d700 1 -- 172.21.15.16:0/3691875916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08d2c0 msgr2=0x7f40ec08ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.122+0000 7f40f185d700 1 --2- 172.21.15.16:0/3691875916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08d2c0 0x7f40ec08ef20 secure :-1 s=READY pgs=1621 cs=0 l=1 rx=0x7f40d400f6c0 tx=0x7f40d400fa60).stop 2019-12-05T06:56:36.131 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.123+0000 7f40f185d700 1 -- 172.21.15.16:0/3691875916 shutdown_connections 2019-12-05T06:56:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.123+0000 7f40f185d700 1 --2- 172.21.15.16:0/3691875916 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec08fef0 0x7f40ec08f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.123+0000 7f40f185d700 1 --2- 172.21.15.16:0/3691875916 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec08d2c0 0x7f40ec08ef20 unknown :-1 s=CLOSED pgs=1621 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.123+0000 7f40f185d700 1 -- 172.21.15.16:0/3691875916 shutdown_connections 2019-12-05T06:56:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.123+0000 7f40f185d700 1 -- 172.21.15.16:0/3691875916 wait complete. 2019-12-05T06:56:36.132 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.123+0000 7f40f185d700 1 -- 172.21.15.16:0/3691875916 >> 172.21.15.16:0/3691875916 conn(0x7f40ec061940 msgr2=0x7f40ec096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:36.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.128+0000 7f40f185d700 1 Processor -- start 2019-12-05T06:56:36.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.129+0000 7f40f185d700 1 -- start start 2019-12-05T06:56:36.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.130+0000 7f40f185d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40ec06df10 0x7f40ec098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:36.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.130+0000 7f40f185d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec156cb0 0x7f40ec1590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:36.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.130+0000 7f40f185d700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40ec089fb0 con 0x7f40ec0992d0 2019-12-05T06:56:36.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.130+0000 7f40f185d700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40ec097370 con 0x7f40ec06df10 2019-12-05T06:56:36.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.130+0000 7f40f185d700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40ec08ad00 con 0x7f40ec156cb0 2019-12-05T06:56:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.130+0000 7f40eb7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40ec06df10 0x7f40ec098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.131+0000 7f40eb7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f40ec06df10 0x7f40ec098300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36346/0 (socket says 172.21.15.16:36346) 2019-12-05T06:56:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.131+0000 7f40eb7fe700 1 -- 172.21.15.16:0/1602983724 learned_addr learned my addr 172.21.15.16:0/1602983724 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.131+0000 7f40ebfff700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec156cb0 0x7f40ec1590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.131+0000 7f40eb7fe700 1 -- 172.21.15.16:0/1602983724 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec156cb0 msgr2=0x7f40ec1590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.131+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec156cb0 0x7f40ec1590b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:36.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.131+0000 7f40eb7fe700 1 -- 172.21.15.16:0/1602983724 >> v1:172.21.15.16:6789/0 conn(0x7f40ec0992d0 legacy=0x7f40ec098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.131+0000 7f40eb7fe700 1 -- 172.21.15.16:0/1602983724 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40ec097a00 con 0x7f40ec06df10 2019-12-05T06:56:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.131+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> v2:172.21.15.16:3300/0 conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.132+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> v2:172.21.15.16:3300/0 conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.132+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=READY pgs=1623 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.132+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=THROTTLE_DONE pgs=1623 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.132+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=THROTTLE_DONE pgs=1623 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.132+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=THROTTLE_DONE pgs=1623 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.132+0000 7f40e3fff700 1 -- 172.21.15.16:0/1602983724 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e4004540 con 0x7f40ec06df10 2019-12-05T06:56:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.132+0000 7f40e3fff700 1 -- 172.21.15.16:0/1602983724 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f40e4000fe0 con 0x7f40ec06df10 2019-12-05T06:56:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.132+0000 7f40e3fff700 1 -- 172.21.15.16:0/1602983724 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40e4004b20 con 0x7f40ec06df10 2019-12-05T06:56:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.132+0000 7f40f185d700 1 -- 172.21.15.16:0/1602983724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40ec097a00 con 0x7f40ec06df10 2019-12-05T06:56:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.132+0000 7f40f185d700 1 -- 172.21.15.16:0/1602983724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40ec097a00 con 0x7f40ec06df10 2019-12-05T06:56:36.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.133+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=THROTTLE_DONE pgs=1623 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.134+0000 7f40e3fff700 1 -- 172.21.15.16:0/1602983724 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f40e4000fe0 con 0x7f40ec06df10 2019-12-05T06:56:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.134+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=THROTTLE_DONE pgs=1623 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.134+0000 7f40e3fff700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023630 0x7f40d0025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.134+0000 7f40e3fff700 1 -- 172.21.15.16:0/1602983724 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f40e40461a0 con 0x7f40ec06df10 2019-12-05T06:56:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.135+0000 7f40eaffd700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023630 0x7f40d0025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:36.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.135+0000 7f40eaffd700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023630 0x7f40d0025ac0 crc :-1 s=READY pgs=675 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:36.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.149+0000 7f40f185d700 1 -- 172.21.15.16:0/1602983724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40ec097a00 con 0x7f40ec06df10 2019-12-05T06:56:36.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.158+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=THROTTLE_DONE pgs=1623 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.158+0000 7f40e3fff700 1 -- 172.21.15.16:0/1602983724 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f40e4001680 con 0x7f40ec06df10 2019-12-05T06:56:36.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.412+0000 7f40f185d700 1 -- 172.21.15.16:0/1602983724 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f40ec097a00 con 0x7f40ec06df10 2019-12-05T06:56:36.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.413+0000 7f40eb7fe700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=THROTTLE_DONE pgs=1623 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:36.414 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.413+0000 7f40e3fff700 1 -- 172.21.15.16:0/1602983724 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f40e4000fe0 con 0x7f40ec06df10 2019-12-05T06:56:36.414 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 197 2019-12-05T06:56:36.414 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:36.415 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8308,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":197}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:36.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.415+0000 7f40f185d700 1 -- 172.21.15.16:0/1602983724 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023630 msgr2=0x7f40d0025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:36.416 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.415+0000 7f40f185d700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023630 0x7f40d0025ac0 crc :-1 s=READY pgs=675 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.415+0000 7f40f185d700 1 -- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 msgr2=0x7f40ec098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.415+0000 7f40f185d700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 0x7f40ec098300 secure :-1 s=READY pgs=1623 cs=0 l=1 rx=0x7f40e40107c0 tx=0x7f40e40105f0).stop 2019-12-05T06:56:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.415+0000 7f40f185d700 1 -- 172.21.15.16:0/1602983724 shutdown_connections 2019-12-05T06:56:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.415+0000 7f40f185d700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f40ec156cb0 0x7f40ec1590b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.415+0000 7f40f185d700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f40d0023630 0x7f40d0025ac0 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.415+0000 7f40f185d700 1 --2- 172.21.15.16:0/1602983724 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f40ec06df10 0x7f40ec098300 unknown :-1 s=CLOSED pgs=1623 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:36.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.415+0000 7f40f185d700 1 -- 172.21.15.16:0/1602983724 shutdown_connections 2019-12-05T06:56:36.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.416+0000 7f40f185d700 1 -- 172.21.15.16:0/1602983724 wait complete. 2019-12-05T06:56:36.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:36.416+0000 7f40f185d700 1 -- 172.21.15.16:0/1602983724 >> 172.21.15.16:0/1602983724 conn(0x7f40ec0714f0 msgr2=0x7f40ec09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:36.427 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:56:38.428 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.675+0000 7fdf5f071700 1 Processor -- start 2019-12-05T06:56:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.675+0000 7fdf5f071700 1 -- start start 2019-12-05T06:56:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.676+0000 7fdf5f071700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:38.684 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.676+0000 7fdf5f071700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdf5808fef0 0x7fdf5808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:38.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.676+0000 7fdf5f071700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf580958e0 con 0x7fdf5808ef20 2019-12-05T06:56:38.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.676+0000 7fdf5f071700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf580a3f70 con 0x7fdf5808fef0 2019-12-05T06:56:38.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.676+0000 7fdf5f071700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5808d830 con 0x7fdf5808c6d0 2019-12-05T06:56:38.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.677+0000 7fdf5ce0d700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdf5808fef0 0x7fdf5808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:38.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.677+0000 7fdf5d60e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:38.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.677+0000 7fdf5de0f700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fdf5808ef20 0x7fdf5808f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:42830/0 (socket says 172.21.15.16:42830) 2019-12-05T06:56:38.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.677+0000 7fdf5d60e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:40348/0 (socket says 172.21.15.16:40348) 2019-12-05T06:56:38.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.677+0000 7fdf5de0f700 1 -- 172.21.15.16:0/430487124 learned_addr learned my addr 172.21.15.16:0/430487124 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:38.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.677+0000 7fdf5d60e700 1 -- 172.21.15.16:0/430487124 >> v2:172.21.15.76:3300/0 conn(0x7fdf5808fef0 msgr2=0x7fdf5808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:38.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.677+0000 7fdf5d60e700 1 --2- 172.21.15.16:0/430487124 >> v2:172.21.15.76:3300/0 conn(0x7fdf5808fef0 0x7fdf5808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:38.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.677+0000 7fdf5d60e700 1 -- 172.21.15.16:0/430487124 >> v1:172.21.15.76:6789/0 conn(0x7fdf5808ef20 legacy=0x7fdf5808f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:38.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.677+0000 7fdf5d60e700 1 -- 172.21.15.16:0/430487124 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf5808d540 con 0x7fdf5808c6d0 2019-12-05T06:56:38.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.677+0000 7fdf5ce0d700 1 --2- 172.21.15.16:0/430487124 >> v2:172.21.15.76:3300/0 conn(0x7fdf5808fef0 0x7fdf5808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:56:38.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf5d60e700 1 --2- 172.21.15.16:0/430487124 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fdf4800af50 tx=0x7fdf4800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf5d60e700 1 --2- 172.21.15.16:0/430487124 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fdf4800af50 tx=0x7fdf4800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf5d60e700 1 --2- 172.21.15.16:0/430487124 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 secure :-1 s=READY pgs=385 cs=0 l=1 rx=0x7fdf4800af50 tx=0x7fdf4800b760).ready entity=mon.2 client_cookie=79f5b9af7f0e4093 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:38.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf5d60e700 1 --2- 172.21.15.16:0/430487124 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fdf4800af50 tx=0x7fdf4800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf57fff700 1 -- 172.21.15.16:0/430487124 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf48007210 con 0x7fdf5808c6d0 2019-12-05T06:56:38.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf5d60e700 1 --2- 172.21.15.16:0/430487124 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fdf4800af50 tx=0x7fdf4800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf5d60e700 1 --2- 172.21.15.16:0/430487124 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 secure :-1 s=THROTTLE_DONE pgs=385 cs=0 l=1 rx=0x7fdf4800af50 tx=0x7fdf4800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf57fff700 1 -- 172.21.15.16:0/430487124 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf480099b0 con 0x7fdf5808c6d0 2019-12-05T06:56:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf57fff700 1 -- 172.21.15.16:0/430487124 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf48007210 con 0x7fdf5808c6d0 2019-12-05T06:56:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf5f071700 1 -- 172.21.15.16:0/430487124 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 msgr2=0x7fdf5808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.678+0000 7fdf5f071700 1 --2- 172.21.15.16:0/430487124 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 secure :-1 s=READY pgs=385 cs=0 l=1 rx=0x7fdf4800af50 tx=0x7fdf4800b760).stop 2019-12-05T06:56:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.679+0000 7fdf5f071700 1 -- 172.21.15.16:0/430487124 shutdown_connections 2019-12-05T06:56:38.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.679+0000 7fdf5f071700 1 --2- 172.21.15.16:0/430487124 >> v2:172.21.15.76:3300/0 conn(0x7fdf5808fef0 0x7fdf5808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:38.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.679+0000 7fdf5f071700 1 --2- 172.21.15.16:0/430487124 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fdf5808c6d0 0x7fdf5808cac0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:38.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.680+0000 7fdf5f071700 1 -- 172.21.15.16:0/430487124 shutdown_connections 2019-12-05T06:56:38.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.680+0000 7fdf5f071700 1 -- 172.21.15.16:0/430487124 wait complete. 2019-12-05T06:56:38.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.680+0000 7fdf5f071700 1 -- 172.21.15.16:0/430487124 >> 172.21.15.16:0/430487124 conn(0x7fdf58061940 msgr2=0x7fdf58096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:38.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.683+0000 7fdf5f071700 1 Processor -- start 2019-12-05T06:56:38.689 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.683+0000 7fdf5f071700 1 -- start start 2019-12-05T06:56:38.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.684+0000 7fdf5f071700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fdf5806fd70 0x7fdf5819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:38.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.684+0000 7fdf5f071700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf580709e0 con 0x7fdf58152fa0 2019-12-05T06:56:38.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.684+0000 7fdf5f071700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf5807e6d0 con 0x7fdf58081670 2019-12-05T06:56:38.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.684+0000 7fdf5f071700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5807fee0 con 0x7fdf5806fd70 2019-12-05T06:56:38.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.684+0000 7fdf5d60e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fdf58152fa0 0x7fdf58080e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49764/0 (socket says 172.21.15.16:49764) 2019-12-05T06:56:38.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.684+0000 7fdf5d60e700 1 -- 172.21.15.16:0/971432935 learned_addr learned my addr 172.21.15.16:0/971432935 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:38.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.685+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> v2:172.21.15.76:3300/0 conn(0x7fdf5806fd70 0x7fdf5819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:38.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.685+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376328506 0 0) 0x7fdf500012b0 con 0x7fdf58081670 2019-12-05T06:56:38.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.685+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf3c002e80 con 0x7fdf58081670 2019-12-05T06:56:38.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.685+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 338564431 0 0) 0x7fdf4800abb0 con 0x7fdf58152fa0 2019-12-05T06:56:38.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.685+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf3c0030b0 con 0x7fdf58152fa0 2019-12-05T06:56:38.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.685+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1794219877 0 0) 0x7fdf48000940 con 0x7fdf58152fa0 2019-12-05T06:56:38.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.685+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf3c0033c0 con 0x7fdf58152fa0 2019-12-05T06:56:38.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fdf4800b4f0 con 0x7fdf58152fa0 2019-12-05T06:56:38.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5de0f700 1 -- 172.21.15.16:0/971432935 >> v1:172.21.15.76:6789/0 conn(0x7fdf58081670 legacy=0x7fdf5819b400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:38.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5de0f700 1 -- 172.21.15.16:0/971432935 >> v1:172.21.15.16:6789/0 conn(0x7fdf58152fa0 legacy=0x7fdf58080e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:38.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5de0f700 1 -- 172.21.15.16:0/971432935 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf58080570 con 0x7fdf5806fd70 2019-12-05T06:56:38.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> v2:172.21.15.76:3300/0 conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> v2:172.21.15.76:3300/0 conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=READY pgs=1093 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:38.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf40007210 con 0x7fdf5806fd70 2019-12-05T06:56:38.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5f071700 1 -- 172.21.15.16:0/971432935 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf58080570 con 0x7fdf5806fd70 2019-12-05T06:56:38.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf40007210 con 0x7fdf5806fd70 2019-12-05T06:56:38.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf40001460 con 0x7fdf5806fd70 2019-12-05T06:56:38.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.686+0000 7fdf5f071700 1 -- 172.21.15.16:0/971432935 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf58080570 con 0x7fdf5806fd70 2019-12-05T06:56:38.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.688+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.689+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fdf40000e70 con 0x7fdf5806fd70 2019-12-05T06:56:38.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.689+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.689+0000 7fdf55ffb700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf3c023680 0x7fdf3c025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:38.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.689+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fdf40045de0 con 0x7fdf5806fd70 2019-12-05T06:56:38.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.689+0000 7fdf5d60e700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf3c023680 0x7fdf3c025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:38.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.690+0000 7fdf5d60e700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf3c023680 0x7fdf3c025af0 crc :-1 s=READY pgs=676 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:38.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.691+0000 7fdf5f071700 1 -- 172.21.15.16:0/971432935 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf58080570 con 0x7fdf5806fd70 2019-12-05T06:56:38.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.698+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.698+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fdf40000e70 con 0x7fdf5806fd70 2019-12-05T06:56:38.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.920+0000 7fdf5f071700 1 -- 172.21.15.16:0/971432935 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdf58080570 con 0x7fdf5806fd70 2019-12-05T06:56:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.922+0000 7fdf5de0f700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=THROTTLE_DONE pgs=1093 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:38.922 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.922+0000 7fdf55ffb700 1 -- 172.21.15.16:0/971432935 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fdf40001960 con 0x7fdf5806fd70 2019-12-05T06:56:38.924 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 197 2019-12-05T06:56:38.924 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:38.924 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8308,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":197}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.925+0000 7fdf5f071700 1 -- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf3c023680 msgr2=0x7fdf3c025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.925+0000 7fdf5f071700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf3c023680 0x7fdf3c025af0 crc :-1 s=READY pgs=676 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.925+0000 7fdf5f071700 1 -- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 msgr2=0x7fdf5819ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:38.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.925+0000 7fdf5f071700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 0x7fdf5819ec20 secure :-1 s=READY pgs=1093 cs=0 l=1 rx=0x7fdf400103a0 tx=0x7fdf40010740).stop 2019-12-05T06:56:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.926+0000 7fdf5f071700 1 -- 172.21.15.16:0/971432935 shutdown_connections 2019-12-05T06:56:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.926+0000 7fdf5f071700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fdf3c023680 0x7fdf3c025af0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.926+0000 7fdf5f071700 1 --2- 172.21.15.16:0/971432935 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fdf5806fd70 0x7fdf5819ec20 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.926+0000 7fdf5f071700 1 -- 172.21.15.16:0/971432935 shutdown_connections 2019-12-05T06:56:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.926+0000 7fdf5f071700 1 -- 172.21.15.16:0/971432935 wait complete. 2019-12-05T06:56:38.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:38.926+0000 7fdf5f071700 1 -- 172.21.15.16:0/971432935 >> 172.21.15.16:0/971432935 conn(0x7fdf58084520 msgr2=0x7fdf580995a0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:38.938 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:56:40.939 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.208+0000 7ff022b52700 1 Processor -- start 2019-12-05T06:56:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.209+0000 7ff022b52700 1 -- start start 2019-12-05T06:56:41.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.210+0000 7ff022b52700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff01c08c6d0 0x7ff01c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.210+0000 7ff022b52700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01c08fb80 con 0x7ff01c08f720 2019-12-05T06:56:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.210+0000 7ff022b52700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01c0958e0 con 0x7ff01c08fef0 2019-12-05T06:56:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.210+0000 7ff022b52700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01c08df90 con 0x7ff01c08c6d0 2019-12-05T06:56:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.211+0000 7ff0210ef700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff01c08c6d0 0x7ff01c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.211+0000 7ff0218f0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff01c08f720 0x7ff01c1533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49774/0 (socket says 172.21.15.16:49774) 2019-12-05T06:56:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.211+0000 7ff0218f0700 1 -- 172.21.15.16:0/2893564891 learned_addr learned my addr 172.21.15.16:0/2893564891 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:41.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.211+0000 7ff0210ef700 1 -- 172.21.15.16:0/2893564891 >> v1:172.21.15.76:6789/0 conn(0x7ff01c08fef0 legacy=0x7ff01c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.211+0000 7ff0210ef700 1 -- 172.21.15.16:0/2893564891 >> v1:172.21.15.16:6789/0 conn(0x7ff01c08f720 legacy=0x7ff01c1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.211+0000 7ff00ffff700 1 -- 172.21.15.16:0/2893564891 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1516869150 0 0) 0x7ff010001080 con 0x7ff01c08fef0 2019-12-05T06:56:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.211+0000 7ff0210ef700 1 -- 172.21.15.16:0/2893564891 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c08e620 con 0x7ff01c08c6d0 2019-12-05T06:56:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.212+0000 7ff0210ef700 1 --2- 172.21.15.16:0/2893564891 >> v2:172.21.15.16:3300/0 conn(0x7ff01c08c6d0 0x7ff01c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff01800b2d0 tx=0x7ff01800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.212+0000 7ff0210ef700 1 --2- 172.21.15.16:0/2893564891 >> v2:172.21.15.16:3300/0 conn(0x7ff01c08c6d0 0x7ff01c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff01800b2d0 tx=0x7ff01800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.212+0000 7ff0210ef700 1 --2- 172.21.15.16:0/2893564891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c08c6d0 0x7ff01c08cac0 secure :-1 s=READY pgs=1626 cs=0 l=1 rx=0x7ff01800b2d0 tx=0x7ff01800bae0).ready entity=mon.0 client_cookie=38f5965922ee0019 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:41.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.212+0000 7ff0210ef700 1 --2- 172.21.15.16:0/2893564891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c08c6d0 0x7ff01c08cac0 secure :-1 s=THROTTLE_DONE pgs=1626 cs=0 l=1 rx=0x7ff01800b2d0 tx=0x7ff01800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.212+0000 7ff00ffff700 1 -- 172.21.15.16:0/2893564891 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff018000ce0 con 0x7ff01c08c6d0 2019-12-05T06:56:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.212+0000 7ff0210ef700 1 --2- 172.21.15.16:0/2893564891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c08c6d0 0x7ff01c08cac0 secure :-1 s=THROTTLE_DONE pgs=1626 cs=0 l=1 rx=0x7ff01800b2d0 tx=0x7ff01800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.212+0000 7ff0210ef700 1 --2- 172.21.15.16:0/2893564891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c08c6d0 0x7ff01c08cac0 secure :-1 s=THROTTLE_DONE pgs=1626 cs=0 l=1 rx=0x7ff01800b2d0 tx=0x7ff01800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.212+0000 7ff00ffff700 1 -- 172.21.15.16:0/2893564891 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff018009be0 con 0x7ff01c08c6d0 2019-12-05T06:56:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.212+0000 7ff00ffff700 1 -- 172.21.15.16:0/2893564891 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff01800a2b0 con 0x7ff01c08c6d0 2019-12-05T06:56:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.212+0000 7ff022b52700 1 -- 172.21.15.16:0/2893564891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c08c6d0 msgr2=0x7ff01c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:41.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.213+0000 7ff022b52700 1 --2- 172.21.15.16:0/2893564891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c08c6d0 0x7ff01c08cac0 secure :-1 s=READY pgs=1626 cs=0 l=1 rx=0x7ff01800b2d0 tx=0x7ff01800bae0).stop 2019-12-05T06:56:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.214+0000 7ff022b52700 1 -- 172.21.15.16:0/2893564891 shutdown_connections 2019-12-05T06:56:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.214+0000 7ff022b52700 1 --2- 172.21.15.16:0/2893564891 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c08c6d0 0x7ff01c08cac0 unknown :-1 s=CLOSED pgs=1626 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.215+0000 7ff022b52700 1 -- 172.21.15.16:0/2893564891 shutdown_connections 2019-12-05T06:56:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.215+0000 7ff022b52700 1 -- 172.21.15.16:0/2893564891 wait complete. 2019-12-05T06:56:41.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.215+0000 7ff022b52700 1 -- 172.21.15.16:0/2893564891 >> 172.21.15.16:0/2893564891 conn(0x7ff01c061940 msgr2=0x7ff01c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.220+0000 7ff022b52700 1 Processor -- start 2019-12-05T06:56:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.220+0000 7ff022b52700 1 -- start start 2019-12-05T06:56:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.222+0000 7ff022b52700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff01c07b2a0 0x7ff01c07aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.222+0000 7ff022b52700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff01c14f9c0 0x7ff01c12a870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.222+0000 7ff022b52700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff01c12ca50 con 0x7ff01c14f5d0 2019-12-05T06:56:41.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.222+0000 7ff022b52700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff01c154a20 con 0x7ff01c07b2a0 2019-12-05T06:56:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.222+0000 7ff022b52700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff01c12e7d0 con 0x7ff01c14f9c0 2019-12-05T06:56:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.222+0000 7ff0208ee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff01c07b2a0 0x7ff01c07aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.222+0000 7ff0208ee700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff01c07b2a0 0x7ff01c07aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36380/0 (socket says 172.21.15.16:36380) 2019-12-05T06:56:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.222+0000 7ff0208ee700 1 -- 172.21.15.16:0/2529342095 learned_addr learned my addr 172.21.15.16:0/2529342095 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.222+0000 7ff0218f0700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff01c14f9c0 0x7ff01c12a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff0208ee700 1 -- 172.21.15.16:0/2529342095 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff01c14f9c0 msgr2=0x7ff01c12a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:41.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff01c14f9c0 0x7ff01c12a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff0208ee700 1 -- 172.21.15.16:0/2529342095 >> v1:172.21.15.16:6789/0 conn(0x7ff01c14f5d0 legacy=0x7ff01c07a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff0208ee700 1 -- 172.21.15.16:0/2529342095 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c12ee60 con 0x7ff01c07b2a0 2019-12-05T06:56:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> v2:172.21.15.16:3300/0 conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> v2:172.21.15.16:3300/0 conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=READY pgs=1628 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=THROTTLE_DONE pgs=1628 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=THROTTLE_DONE pgs=1628 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=THROTTLE_DONE pgs=1628 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff00dffb700 1 -- 172.21.15.16:0/2529342095 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff010000b50 con 0x7ff01c07b2a0 2019-12-05T06:56:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff00dffb700 1 -- 172.21.15.16:0/2529342095 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff010009a30 con 0x7ff01c07b2a0 2019-12-05T06:56:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff022b52700 1 -- 172.21.15.16:0/2529342095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff01c12ee60 con 0x7ff01c07b2a0 2019-12-05T06:56:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.223+0000 7ff00dffb700 1 -- 172.21.15.16:0/2529342095 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff01000a100 con 0x7ff01c07b2a0 2019-12-05T06:56:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.224+0000 7ff022b52700 1 -- 172.21.15.16:0/2529342095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff01c12ee60 con 0x7ff01c07b2a0 2019-12-05T06:56:41.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.225+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=THROTTLE_DONE pgs=1628 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.225+0000 7ff00dffb700 1 -- 172.21.15.16:0/2529342095 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff010009630 con 0x7ff01c07b2a0 2019-12-05T06:56:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.225+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=THROTTLE_DONE pgs=1628 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.226+0000 7ff00dffb700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feffc023630 0x7feffc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.226+0000 7ff00dffb700 1 -- 172.21.15.16:0/2529342095 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7ff010045df0 con 0x7ff01c07b2a0 2019-12-05T06:56:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.226+0000 7ff0210ef700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feffc023630 0x7feffc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:41.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.227+0000 7ff0210ef700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feffc023630 0x7feffc025ac0 crc :-1 s=READY pgs=677 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:41.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.240+0000 7ff022b52700 1 -- 172.21.15.16:0/2529342095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff01c12ee60 con 0x7ff01c07b2a0 2019-12-05T06:56:41.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.247+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=THROTTLE_DONE pgs=1628 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.247+0000 7ff00dffb700 1 -- 172.21.15.16:0/2529342095 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff010039450 con 0x7ff01c07b2a0 2019-12-05T06:56:41.494 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.493+0000 7ff022b52700 1 -- 172.21.15.16:0/2529342095 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff01c12ee60 con 0x7ff01c07b2a0 2019-12-05T06:56:41.495 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.494+0000 7ff0208ee700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=THROTTLE_DONE pgs=1628 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:41.496 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.494+0000 7ff00dffb700 1 -- 172.21.15.16:0/2529342095 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+2112 (secure 0 0 0) 0x7ff010039450 con 0x7ff01c07b2a0 2019-12-05T06:56:41.497 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:41.498 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8308,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":197}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:41.498 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 197 2019-12-05T06:56:41.500 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 -- 172.21.15.16:0/2529342095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feffc023630 msgr2=0x7feffc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feffc023630 0x7feffc025ac0 crc :-1 s=READY pgs=677 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:41.501 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 -- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 msgr2=0x7ff01c07aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 0x7ff01c07aad0 secure :-1 s=READY pgs=1628 cs=0 l=1 rx=0x7ff010010450 tx=0x7ff01000b390).stop 2019-12-05T06:56:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 -- 172.21.15.16:0/2529342095 shutdown_connections 2019-12-05T06:56:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7feffc023630 0x7feffc025ac0 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:41.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff01c07b2a0 0x7ff01c07aad0 unknown :-1 s=CLOSED pgs=1628 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:41.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 --2- 172.21.15.16:0/2529342095 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff01c14f9c0 0x7ff01c12a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 -- 172.21.15.16:0/2529342095 shutdown_connections 2019-12-05T06:56:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 -- 172.21.15.16:0/2529342095 wait complete. 2019-12-05T06:56:41.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:41.499+0000 7ff022b52700 1 -- 172.21.15.16:0/2529342095 >> 172.21.15.16:0/2529342095 conn(0x7ff01c132ac0 msgr2=0x7ff01c070bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:41.511 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:56:43.512 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:43.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.773+0000 7f9f35e46700 1 Processor -- start 2019-12-05T06:56:43.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.773+0000 7f9f35e46700 1 -- start start 2019-12-05T06:56:43.782 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.774+0000 7f9f35e46700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9f3008c6d0 0x7f9f3008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:43.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.774+0000 7f9f35e46700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f9f3008f720 0x7f9f3008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:43.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.774+0000 7f9f35e46700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f300958e0 con 0x7f9f3008fef0 2019-12-05T06:56:43.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.774+0000 7f9f35e46700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f300a3f70 con 0x7f9f3008f720 2019-12-05T06:56:43.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.774+0000 7f9f35e46700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f3008d830 con 0x7f9f3008c6d0 2019-12-05T06:56:43.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.775+0000 7f9f2f7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9f3008fef0 0x7f9f3008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49788/0 (socket says 172.21.15.16:49788) 2019-12-05T06:56:43.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.775+0000 7f9f2f7fe700 1 -- 172.21.15.16:0/1604976177 learned_addr learned my addr 172.21.15.16:0/1604976177 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:43.783 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.775+0000 7f9f2ffff700 1 --2- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9f3008c6d0 0x7f9f3008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:43.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.775+0000 7f9f34be4700 1 --2- 172.21.15.16:0/1604976177 >> v2:172.21.15.76:3300/0 conn(0x7f9f3008f720 0x7f9f3008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:43.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.775+0000 7f9f2effd700 1 -- 172.21.15.16:0/1604976177 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4192741903 0 0) 0x7f9f18001100 con 0x7f9f3008fef0 2019-12-05T06:56:43.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.775+0000 7f9f2effd700 1 -- 172.21.15.16:0/1604976177 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f1c002e80 con 0x7f9f3008fef0 2019-12-05T06:56:43.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f34be4700 1 -- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9f3008c6d0 msgr2=0x7f9f3008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:43.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f34be4700 1 --2- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9f3008c6d0 0x7f9f3008cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:43.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f34be4700 1 -- 172.21.15.16:0/1604976177 >> v1:172.21.15.16:6789/0 conn(0x7f9f3008fef0 legacy=0x7f9f3008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:43.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f34be4700 1 -- 172.21.15.16:0/1604976177 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3008fbc0 con 0x7f9f3008f720 2019-12-05T06:56:43.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f2ffff700 1 --2- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9f3008c6d0 0x7f9f3008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:56:43.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f34be4700 1 --2- 172.21.15.16:0/1604976177 >> v2:172.21.15.76:3300/0 conn(0x7f9f3008f720 0x7f9f3008d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f9f2800af80 tx=0x7f9f2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f34be4700 1 --2- 172.21.15.16:0/1604976177 >> v2:172.21.15.76:3300/0 conn(0x7f9f3008f720 0x7f9f3008d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f9f2800af80 tx=0x7f9f2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f34be4700 1 --2- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f3008f720 0x7f9f3008d000 secure :-1 s=READY pgs=1095 cs=0 l=1 rx=0x7f9f2800af80 tx=0x7f9f2800b790).ready entity=mon.1 client_cookie=75348e4fc605d398 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:43.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f34be4700 1 --2- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f3008f720 0x7f9f3008d000 secure :-1 s=THROTTLE_DONE pgs=1095 cs=0 l=1 rx=0x7f9f2800af80 tx=0x7f9f2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f2effd700 1 -- 172.21.15.16:0/1604976177 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f28000bc0 con 0x7f9f3008f720 2019-12-05T06:56:43.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f34be4700 1 --2- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f3008f720 0x7f9f3008d000 secure :-1 s=THROTTLE_DONE pgs=1095 cs=0 l=1 rx=0x7f9f2800af80 tx=0x7f9f2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.776+0000 7f9f34be4700 1 --2- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f3008f720 0x7f9f3008d000 secure :-1 s=THROTTLE_DONE pgs=1095 cs=0 l=1 rx=0x7f9f2800af80 tx=0x7f9f2800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.777+0000 7f9f2effd700 1 -- 172.21.15.16:0/1604976177 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f280097f0 con 0x7f9f3008f720 2019-12-05T06:56:43.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.777+0000 7f9f2effd700 1 -- 172.21.15.16:0/1604976177 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f28000bc0 con 0x7f9f3008f720 2019-12-05T06:56:43.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.777+0000 7f9f35e46700 1 -- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f3008f720 msgr2=0x7f9f3008d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:43.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.777+0000 7f9f35e46700 1 --2- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f3008f720 0x7f9f3008d000 secure :-1 s=READY pgs=1095 cs=0 l=1 rx=0x7f9f2800af80 tx=0x7f9f2800b790).stop 2019-12-05T06:56:43.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.778+0000 7f9f35e46700 1 -- 172.21.15.16:0/1604976177 shutdown_connections 2019-12-05T06:56:43.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.778+0000 7f9f35e46700 1 --2- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f9f3008f720 0x7f9f3008d000 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:43.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.778+0000 7f9f35e46700 1 --2- 172.21.15.16:0/1604976177 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f9f3008c6d0 0x7f9f3008cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:43.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.778+0000 7f9f35e46700 1 -- 172.21.15.16:0/1604976177 shutdown_connections 2019-12-05T06:56:43.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.778+0000 7f9f35e46700 1 -- 172.21.15.16:0/1604976177 wait complete. 2019-12-05T06:56:43.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.778+0000 7f9f35e46700 1 -- 172.21.15.16:0/1604976177 >> 172.21.15.16:0/1604976177 conn(0x7f9f30061940 msgr2=0x7f9f30096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:43.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.782+0000 7f9f35e46700 1 Processor -- start 2019-12-05T06:56:43.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.783+0000 7f9f35e46700 1 -- start start 2019-12-05T06:56:43.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.784+0000 7f9f35e46700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f3012b640 0x7f9f3012ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:43.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.784+0000 7f9f35e46700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f30131bf0 con 0x7f9f30153b50 2019-12-05T06:56:43.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.784+0000 7f9f35e46700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f301316f0 con 0x7f9f301302f0 2019-12-05T06:56:43.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.784+0000 7f9f35e46700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f301296e0 con 0x7f9f3012b640 2019-12-05T06:56:43.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.784+0000 7f9f2f7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f9f3012b640 0x7f9f3012ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.784+0000 7f9f34be4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f9f30153b50 0x7f9f30155f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49796/0 (socket says 172.21.15.16:49796) 2019-12-05T06:56:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.784+0000 7f9f34be4700 1 -- 172.21.15.16:0/2021053066 learned_addr learned my addr 172.21.15.16:0/2021053066 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.785+0000 7f9f2cff9700 1 -- 172.21.15.16:0/2021053066 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 198298355 0 0) 0x7f9f2000df00 con 0x7f9f301302f0 2019-12-05T06:56:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.785+0000 7f9f2f7fe700 1 -- 172.21.15.16:0/2021053066 >> v1:172.21.15.76:6789/0 conn(0x7f9f301302f0 legacy=0x7f9f3012a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.785+0000 7f9f2f7fe700 1 -- 172.21.15.16:0/2021053066 >> v1:172.21.15.16:6789/0 conn(0x7f9f30153b50 legacy=0x7f9f30155f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:43.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.785+0000 7f9f2f7fe700 1 -- 172.21.15.16:0/2021053066 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f30129d70 con 0x7f9f3012b640 2019-12-05T06:56:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.785+0000 7f9f2f7fe700 1 --2- 172.21.15.16:0/2021053066 >> v2:172.21.15.16:3300/0 conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.785+0000 7f9f2f7fe700 1 --2- 172.21.15.16:0/2021053066 >> v2:172.21.15.16:3300/0 conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.785+0000 7f9f2f7fe700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=READY pgs=1631 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.786+0000 7f9f2f7fe700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=THROTTLE_DONE pgs=1631 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.786+0000 7f9f2cff9700 1 -- 172.21.15.16:0/2021053066 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f18000d80 con 0x7f9f3012b640 2019-12-05T06:56:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.786+0000 7f9f2f7fe700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=THROTTLE_DONE pgs=1631 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.786+0000 7f9f2f7fe700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=THROTTLE_DONE pgs=1631 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.786+0000 7f9f2cff9700 1 -- 172.21.15.16:0/2021053066 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18001470 con 0x7f9f3012b640 2019-12-05T06:56:43.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.786+0000 7f9f35e46700 1 -- 172.21.15.16:0/2021053066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f30129d70 con 0x7f9f3012b640 2019-12-05T06:56:43.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.786+0000 7f9f2cff9700 1 -- 172.21.15.16:0/2021053066 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f18001b40 con 0x7f9f3012b640 2019-12-05T06:56:43.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.786+0000 7f9f35e46700 1 -- 172.21.15.16:0/2021053066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f30129d70 con 0x7f9f3012b640 2019-12-05T06:56:43.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.787+0000 7f9f2f7fe700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=THROTTLE_DONE pgs=1631 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.788+0000 7f9f2cff9700 1 -- 172.21.15.16:0/2021053066 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f9f18000c90 con 0x7f9f3012b640 2019-12-05T06:56:43.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.789+0000 7f9f2f7fe700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=THROTTLE_DONE pgs=1631 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.789+0000 7f9f2cff9700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9f14023630 0x7f9f14025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.789+0000 7f9f2cff9700 1 -- 172.21.15.16:0/2021053066 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7f9f18045d60 con 0x7f9f3012b640 2019-12-05T06:56:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.789+0000 7f9f2ffff700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9f14023630 0x7f9f14025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.790+0000 7f9f2ffff700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9f14023630 0x7f9f14025ac0 crc :-1 s=READY pgs=678 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:43.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.790+0000 7f9f35e46700 1 -- 172.21.15.16:0/2021053066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f30129d70 con 0x7f9f3012b640 2019-12-05T06:56:43.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.798+0000 7f9f2f7fe700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=THROTTLE_DONE pgs=1631 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:43.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:43.799+0000 7f9f2cff9700 1 -- 172.21.15.16:0/2021053066 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f9f18001070 con 0x7f9f3012b640 2019-12-05T06:56:44.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.057+0000 7f9f35e46700 1 -- 172.21.15.16:0/2021053066 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9f30129d70 con 0x7f9f3012b640 2019-12-05T06:56:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.058+0000 7f9f2f7fe700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=THROTTLE_DONE pgs=1631 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:44.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.058+0000 7f9f2cff9700 1 -- 172.21.15.16:0/2021053066 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f9f18001070 con 0x7f9f3012b640 2019-12-05T06:56:44.061 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 197 2019-12-05T06:56:44.062 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:44.062 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8308,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":197}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.064+0000 7f9f35e46700 1 -- 172.21.15.16:0/2021053066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9f14023630 msgr2=0x7f9f14025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.064+0000 7f9f35e46700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9f14023630 0x7f9f14025ac0 crc :-1 s=READY pgs=678 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.064+0000 7f9f35e46700 1 -- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 msgr2=0x7f9f3012ae70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:44.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.064+0000 7f9f35e46700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 0x7f9f3012ae70 secure :-1 s=READY pgs=1631 cs=0 l=1 rx=0x7f9f18010380 tx=0x7f9f18010720).stop 2019-12-05T06:56:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.064+0000 7f9f35e46700 1 -- 172.21.15.16:0/2021053066 shutdown_connections 2019-12-05T06:56:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.064+0000 7f9f35e46700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f9f14023630 0x7f9f14025ac0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:44.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.064+0000 7f9f35e46700 1 --2- 172.21.15.16:0/2021053066 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f9f3012b640 0x7f9f3012ae70 unknown :-1 s=CLOSED pgs=1631 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.065+0000 7f9f35e46700 1 -- 172.21.15.16:0/2021053066 shutdown_connections 2019-12-05T06:56:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.065+0000 7f9f35e46700 1 -- 172.21.15.16:0/2021053066 wait complete. 2019-12-05T06:56:44.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:44.065+0000 7f9f35e46700 1 -- 172.21.15.16:0/2021053066 >> 172.21.15.16:0/2021053066 conn(0x7f9f300a2850 msgr2=0x7f9f3006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:44.083 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:56:46.084 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:46.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.683+0000 7fbf1f427700 1 Processor -- start 2019-12-05T06:56:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.684+0000 7fbf1f427700 1 -- start start 2019-12-05T06:56:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.685+0000 7fbf1f427700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 0x7fbf1808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.685+0000 7fbf1f427700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbf1808fef0 0x7fbf1808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.685+0000 7fbf1f427700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf180958e0 con 0x7fbf1808c6d0 2019-12-05T06:56:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.685+0000 7fbf1f427700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf1808df90 con 0x7fbf1808fef0 2019-12-05T06:56:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.685+0000 7fbf1f427700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf180a3f70 con 0x7fbf1808d2c0 2019-12-05T06:56:46.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.686+0000 7fbf1d9c4700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbf1808c6d0 0x7fbf1808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49802/0 (socket says 172.21.15.16:49802) 2019-12-05T06:56:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.686+0000 7fbf1d9c4700 1 -- 172.21.15.16:0/2944119870 learned_addr learned my addr 172.21.15.16:0/2944119870 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.686+0000 7fbf1d1c3700 1 --2- 172.21.15.16:0/2944119870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 0x7fbf1808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.686+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/2944119870 >> v2:172.21.15.76:3300/0 conn(0x7fbf1808fef0 0x7fbf1808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.686+0000 7fbf1c9c2700 1 -- 172.21.15.16:0/2944119870 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 131350466 0 0) 0x7fbf10001150 con 0x7fbf1808c6d0 2019-12-05T06:56:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.686+0000 7fbf1c9c2700 1 -- 172.21.15.16:0/2944119870 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf04002e80 con 0x7fbf1808c6d0 2019-12-05T06:56:46.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1d1c3700 1 -- 172.21.15.16:0/2944119870 >> v2:172.21.15.76:3300/0 conn(0x7fbf1808fef0 msgr2=0x7fbf1808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1d1c3700 1 --2- 172.21.15.16:0/2944119870 >> v2:172.21.15.76:3300/0 conn(0x7fbf1808fef0 0x7fbf1808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1d1c3700 1 -- 172.21.15.16:0/2944119870 >> v1:172.21.15.16:6789/0 conn(0x7fbf1808c6d0 legacy=0x7fbf1808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1d1c3700 1 -- 172.21.15.16:0/2944119870 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1808e620 con 0x7fbf1808d2c0 2019-12-05T06:56:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1d1c3700 1 --2- 172.21.15.16:0/2944119870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 0x7fbf1808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf0800b2d0 tx=0x7fbf0800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1d1c3700 1 --2- 172.21.15.16:0/2944119870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 0x7fbf1808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf0800b2d0 tx=0x7fbf0800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1d1c3700 1 --2- 172.21.15.16:0/2944119870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 0x7fbf1808ef20 secure :-1 s=READY pgs=386 cs=0 l=1 rx=0x7fbf0800b2d0 tx=0x7fbf0800b5d0).ready entity=mon.2 client_cookie=998450211542758d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1d1c3700 1 --2- 172.21.15.16:0/2944119870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 0x7fbf1808ef20 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7fbf0800b2d0 tx=0x7fbf0800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1d1c3700 1 --2- 172.21.15.16:0/2944119870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 0x7fbf1808ef20 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7fbf0800b2d0 tx=0x7fbf0800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1c9c2700 1 -- 172.21.15.16:0/2944119870 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf08007590 con 0x7fbf1808d2c0 2019-12-05T06:56:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.687+0000 7fbf1d1c3700 1 --2- 172.21.15.16:0/2944119870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 0x7fbf1808ef20 secure :-1 s=THROTTLE_DONE pgs=386 cs=0 l=1 rx=0x7fbf0800b2d0 tx=0x7fbf0800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.688+0000 7fbf1c9c2700 1 -- 172.21.15.16:0/2944119870 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf08009d30 con 0x7fbf1808d2c0 2019-12-05T06:56:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.688+0000 7fbf1c9c2700 1 -- 172.21.15.16:0/2944119870 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf0800a400 con 0x7fbf1808d2c0 2019-12-05T06:56:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.688+0000 7fbf1f427700 1 -- 172.21.15.16:0/2944119870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 msgr2=0x7fbf1808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.688+0000 7fbf1f427700 1 --2- 172.21.15.16:0/2944119870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 0x7fbf1808ef20 secure :-1 s=READY pgs=386 cs=0 l=1 rx=0x7fbf0800b2d0 tx=0x7fbf0800b5d0).stop 2019-12-05T06:56:46.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.689+0000 7fbf1f427700 1 -- 172.21.15.16:0/2944119870 shutdown_connections 2019-12-05T06:56:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.689+0000 7fbf1f427700 1 --2- 172.21.15.16:0/2944119870 >> v2:172.21.15.76:3300/0 conn(0x7fbf1808fef0 0x7fbf1808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.689+0000 7fbf1f427700 1 --2- 172.21.15.16:0/2944119870 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf1808d2c0 0x7fbf1808ef20 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.690+0000 7fbf1f427700 1 -- 172.21.15.16:0/2944119870 shutdown_connections 2019-12-05T06:56:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.690+0000 7fbf1f427700 1 -- 172.21.15.16:0/2944119870 wait complete. 2019-12-05T06:56:46.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.690+0000 7fbf1f427700 1 -- 172.21.15.16:0/2944119870 >> 172.21.15.16:0/2944119870 conn(0x7fbf18061940 msgr2=0x7fbf18096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.695+0000 7fbf1f427700 1 Processor -- start 2019-12-05T06:56:46.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.695+0000 7fbf1f427700 1 -- start start 2019-12-05T06:56:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.696+0000 7fbf1f427700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf181325e0 0x7fbf1812b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.696+0000 7fbf1f427700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbf18154c60 0x7fbf18157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.696+0000 7fbf1f427700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf18128b20 con 0x7fbf1812c740 2019-12-05T06:56:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.696+0000 7fbf1f427700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf18129db0 con 0x7fbf18154c60 2019-12-05T06:56:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.696+0000 7fbf1f427700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1812a080 con 0x7fbf181325e0 2019-12-05T06:56:46.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.697+0000 7fbf1e1c5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbf18154c60 0x7fbf18157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.697+0000 7fbf1d9c4700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf181325e0 0x7fbf1812b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.697+0000 7fbf1e1c5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbf18154c60 0x7fbf18157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:50732/0 (socket says 172.21.15.16:50732) 2019-12-05T06:56:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.697+0000 7fbf1d1c3700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbf1812c740 0x7fbf1812bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:42882/0 (socket says 172.21.15.16:42882) 2019-12-05T06:56:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.697+0000 7fbf1e1c5700 1 -- 172.21.15.16:0/1471742226 learned_addr learned my addr 172.21.15.16:0/1471742226 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.697+0000 7fbf1e1c5700 1 -- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf181325e0 msgr2=0x7fbf1812b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf181325e0 0x7fbf1812b770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:46.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf167fc700 1 -- 172.21.15.16:0/1471742226 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3697229844 0 0) 0x7fbf0800afa0 con 0x7fbf1812c740 2019-12-05T06:56:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1e1c5700 1 -- 172.21.15.16:0/1471742226 >> v1:172.21.15.76:6789/0 conn(0x7fbf1812c740 legacy=0x7fbf1812bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1e1c5700 1 -- 172.21.15.16:0/1471742226 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1812ae70 con 0x7fbf18154c60 2019-12-05T06:56:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1d9c4700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf181325e0 0x7fbf1812b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:56:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> v2:172.21.15.76:3300/0 conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> v2:172.21.15.76:3300/0 conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=READY pgs=1098 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:46.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf167fc700 1 -- 172.21.15.16:0/1471742226 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf00009990 con 0x7fbf18154c60 2019-12-05T06:56:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf167fc700 1 -- 172.21.15.16:0/1471742226 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf000095b0 con 0x7fbf18154c60 2019-12-05T06:56:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.698+0000 7fbf1f427700 1 -- 172.21.15.16:0/1471742226 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1812ae70 con 0x7fbf18154c60 2019-12-05T06:56:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.699+0000 7fbf167fc700 1 -- 172.21.15.16:0/1471742226 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf000097a0 con 0x7fbf18154c60 2019-12-05T06:56:46.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.699+0000 7fbf1f427700 1 -- 172.21.15.16:0/1471742226 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1812ae70 con 0x7fbf18154c60 2019-12-05T06:56:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.700+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.701+0000 7fbf167fc700 1 -- 172.21.15.16:0/1471742226 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbf0000ecb0 con 0x7fbf18154c60 2019-12-05T06:56:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.701+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.701+0000 7fbf167fc700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbefc023630 0x7fbefc025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.701+0000 7fbf167fc700 1 -- 172.21.15.16:0/1471742226 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 8869+0+0 (secure 0 0 0) 0x7fbf00045dc0 con 0x7fbf18154c60 2019-12-05T06:56:46.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.701+0000 7fbf1d9c4700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbefc023630 0x7fbefc025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:46.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.702+0000 7fbf1d9c4700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbefc023630 0x7fbefc025ac0 crc :-1 s=READY pgs=679 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:46.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.715+0000 7fbf1f427700 1 -- 172.21.15.16:0/1471742226 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf1812ae70 con 0x7fbf18154c60 2019-12-05T06:56:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.725+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.725+0000 7fbf167fc700 1 -- 172.21.15.16:0/1471742226 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbf000426f0 con 0x7fbf18154c60 2019-12-05T06:56:46.986 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.985+0000 7fbf1f427700 1 -- 172.21.15.16:0/1471742226 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf1812ae70 con 0x7fbf18154c60 2019-12-05T06:56:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.986+0000 7fbf1e1c5700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=THROTTLE_DONE pgs=1098 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:46.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.986+0000 7fbf167fc700 1 -- 172.21.15.16:0/1471742226 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 197 v197) v1 ==== 95+0+2112 (secure 0 0 0) 0x7fbf000426f0 con 0x7fbf18154c60 2019-12-05T06:56:46.989 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 197 2019-12-05T06:56:46.989 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:46.989 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":197,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8308,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":197}],"filesystems":[{"mdsmap":{"epoch":196,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:55:53.279947+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6212},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6212":{"gid":6212,"name":"b","rank":0,"incarnation":193,"state":"up:active","state_seq":20,"addr":"172.21.15.76:6835/2545654850","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":2545654850},{"type":"v1","addr":"172.21.15.76:6835","nonce":2545654850}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:46.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.990+0000 7fbf1f427700 1 -- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbefc023630 msgr2=0x7fbefc025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:46.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.990+0000 7fbf1f427700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbefc023630 0x7fbefc025ac0 crc :-1 s=READY pgs=679 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:46.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.990+0000 7fbf1f427700 1 -- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 msgr2=0x7fbf18157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:46.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.990+0000 7fbf1f427700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 0x7fbf18157060 secure :-1 s=READY pgs=1098 cs=0 l=1 rx=0x7fbf00010440 tx=0x7fbf000107e0).stop 2019-12-05T06:56:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.990+0000 7fbf1f427700 1 -- 172.21.15.16:0/1471742226 shutdown_connections 2019-12-05T06:56:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.990+0000 7fbf1f427700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbefc023630 0x7fbefc025ac0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.990+0000 7fbf1f427700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf18154c60 0x7fbf18157060 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.990+0000 7fbf1f427700 1 --2- 172.21.15.16:0/1471742226 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbf181325e0 0x7fbf1812b770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.991+0000 7fbf1f427700 1 -- 172.21.15.16:0/1471742226 shutdown_connections 2019-12-05T06:56:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.991+0000 7fbf1f427700 1 -- 172.21.15.16:0/1471742226 wait complete. 2019-12-05T06:56:46.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:46.991+0000 7fbf1f427700 1 -- 172.21.15.16:0/1471742226 >> 172.21.15.16:0/1471742226 conn(0x7fbf180a2850 msgr2=0x7fbf1806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:47.003 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:56:47.334 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:56:47.339 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:56:49.004 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:49.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.192+0000 7fbf5e641700 1 Processor -- start 2019-12-05T06:56:49.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.192+0000 7fbf5e641700 1 -- start start 2019-12-05T06:56:49.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.193+0000 7fbf5e641700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbf5808c6d0 0x7fbf5808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:49.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf5e641700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5808fb80 con 0x7fbf5808f720 2019-12-05T06:56:49.200 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf5e641700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf580958e0 con 0x7fbf5808fef0 2019-12-05T06:56:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf5e641700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf5808df90 con 0x7fbf5808c6d0 2019-12-05T06:56:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf5d3df700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbf5808f720 0x7fbf581533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49834/0 (socket says 172.21.15.16:49834) 2019-12-05T06:56:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf5d3df700 1 -- 172.21.15.16:0/1333416479 learned_addr learned my addr 172.21.15.16:0/1333416479 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/1333416479 >> v2:172.21.15.76:3300/0 conn(0x7fbf5808c6d0 0x7fbf5808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf577fe700 1 -- 172.21.15.16:0/1333416479 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339581073 0 0) 0x7fbf40001170 con 0x7fbf5808f720 2019-12-05T06:56:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf577fe700 1 -- 172.21.15.16:0/1333416479 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf44002e80 con 0x7fbf5808f720 2019-12-05T06:56:49.201 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf5cbde700 1 -- 172.21.15.16:0/1333416479 >> v1:172.21.15.76:6789/0 conn(0x7fbf5808fef0 legacy=0x7fbf5808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf5cbde700 1 -- 172.21.15.16:0/1333416479 >> v1:172.21.15.16:6789/0 conn(0x7fbf5808f720 legacy=0x7fbf581533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.194+0000 7fbf5cbde700 1 -- 172.21.15.16:0/1333416479 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5808e620 con 0x7fbf5808c6d0 2019-12-05T06:56:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/1333416479 >> v2:172.21.15.76:3300/0 conn(0x7fbf5808c6d0 0x7fbf5808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbf5000af50 tx=0x7fbf5000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/1333416479 >> v2:172.21.15.76:3300/0 conn(0x7fbf5808c6d0 0x7fbf5808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbf5000af50 tx=0x7fbf5000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/1333416479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf5808c6d0 0x7fbf5808cac0 secure :-1 s=READY pgs=1100 cs=0 l=1 rx=0x7fbf5000af50 tx=0x7fbf5000b760).ready entity=mon.1 client_cookie=886fcfd027abd99e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/1333416479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf5808c6d0 0x7fbf5808cac0 secure :-1 s=THROTTLE_DONE pgs=1100 cs=0 l=1 rx=0x7fbf5000af50 tx=0x7fbf5000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.202 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf577fe700 1 -- 172.21.15.16:0/1333416479 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf50000960 con 0x7fbf5808c6d0 2019-12-05T06:56:49.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/1333416479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf5808c6d0 0x7fbf5808cac0 secure :-1 s=THROTTLE_DONE pgs=1100 cs=0 l=1 rx=0x7fbf5000af50 tx=0x7fbf5000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/1333416479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf5808c6d0 0x7fbf5808cac0 secure :-1 s=THROTTLE_DONE pgs=1100 cs=0 l=1 rx=0x7fbf5000af50 tx=0x7fbf5000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf577fe700 1 -- 172.21.15.16:0/1333416479 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf50009860 con 0x7fbf5808c6d0 2019-12-05T06:56:49.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf577fe700 1 -- 172.21.15.16:0/1333416479 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf50007210 con 0x7fbf5808c6d0 2019-12-05T06:56:49.203 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf5e641700 1 -- 172.21.15.16:0/1333416479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf5808c6d0 msgr2=0x7fbf5808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.195+0000 7fbf5e641700 1 --2- 172.21.15.16:0/1333416479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf5808c6d0 0x7fbf5808cac0 secure :-1 s=READY pgs=1100 cs=0 l=1 rx=0x7fbf5000af50 tx=0x7fbf5000b760).stop 2019-12-05T06:56:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.196+0000 7fbf5e641700 1 -- 172.21.15.16:0/1333416479 shutdown_connections 2019-12-05T06:56:49.204 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.196+0000 7fbf5e641700 1 --2- 172.21.15.16:0/1333416479 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf5808c6d0 0x7fbf5808cac0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.196+0000 7fbf5e641700 1 -- 172.21.15.16:0/1333416479 shutdown_connections 2019-12-05T06:56:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.196+0000 7fbf5e641700 1 -- 172.21.15.16:0/1333416479 wait complete. 2019-12-05T06:56:49.205 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.196+0000 7fbf5e641700 1 -- 172.21.15.16:0/1333416479 >> 172.21.15.16:0/1333416479 conn(0x7fbf58061940 msgr2=0x7fbf58096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.200+0000 7fbf5e641700 1 Processor -- start 2019-12-05T06:56:49.207 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.200+0000 7fbf5e641700 1 -- start start 2019-12-05T06:56:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf5e641700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbf581302f0 0x7fbf5812a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf5e641700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf58131bf0 con 0x7fbf58153ba0 2019-12-05T06:56:49.208 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf5e641700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf581316f0 con 0x7fbf5812b640 2019-12-05T06:56:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf5d3df700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbf58153ba0 0x7fbf58155fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49838/0 (socket says 172.21.15.16:49838) 2019-12-05T06:56:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbf581296e0 con 0x7fbf581302f0 2019-12-05T06:56:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf5d3df700 1 -- 172.21.15.16:0/2096485518 learned_addr learned my addr 172.21.15.16:0/2096485518 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> v2:172.21.15.76:3300/0 conn(0x7fbf581302f0 0x7fbf5812a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:49.209 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3737653 0 0) 0x7fbf40000f80 con 0x7fbf58153ba0 2019-12-05T06:56:49.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf3c002e80 con 0x7fbf58153ba0 2019-12-05T06:56:49.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 194849736 0 0) 0x7fbf48001510 con 0x7fbf5812b640 2019-12-05T06:56:49.210 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.202+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf3c0030b0 con 0x7fbf5812b640 2019-12-05T06:56:49.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 712793109 0 0) 0x7fbf40000940 con 0x7fbf58153ba0 2019-12-05T06:56:49.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5cbde700 1 -- 172.21.15.16:0/2096485518 >> v1:172.21.15.76:6789/0 conn(0x7fbf5812b640 legacy=0x7fbf5812abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5cbde700 1 -- 172.21.15.16:0/2096485518 >> v1:172.21.15.16:6789/0 conn(0x7fbf58153ba0 legacy=0x7fbf58155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5cbde700 1 -- 172.21.15.16:0/2096485518 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf58129d70 con 0x7fbf581302f0 2019-12-05T06:56:49.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> v2:172.21.15.76:3300/0 conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> v2:172.21.15.76:3300/0 conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=READY pgs=1102 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=THROTTLE_DONE pgs=1102 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf50000940 con 0x7fbf581302f0 2019-12-05T06:56:49.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=THROTTLE_DONE pgs=1102 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbf50009770 con 0x7fbf581302f0 2019-12-05T06:56:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=THROTTLE_DONE pgs=1102 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbf50009960 con 0x7fbf581302f0 2019-12-05T06:56:49.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf58129d70 con 0x7fbf581302f0 2019-12-05T06:56:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.203+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf58129d70 con 0x7fbf581302f0 2019-12-05T06:56:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.204+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=THROTTLE_DONE pgs=1102 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.205+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbf50009960 con 0x7fbf581302f0 2019-12-05T06:56:49.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.205+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=THROTTLE_DONE pgs=1102 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.205+0000 7fbf557fa700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbf3c023650 0x7fbf3c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.205+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7fbf50045dc0 con 0x7fbf581302f0 2019-12-05T06:56:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.205+0000 7fbf57fff700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbf3c023650 0x7fbf3c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:49.223 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.206+0000 7fbf57fff700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbf3c023650 0x7fbf3c025ae0 crc :-1 s=READY pgs=680 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:49.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.207+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf58129d70 con 0x7fbf581302f0 2019-12-05T06:56:49.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.215+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=THROTTLE_DONE pgs=1102 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.215+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbf5000b5b0 con 0x7fbf581302f0 2019-12-05T06:56:49.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.423+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbf58129d70 con 0x7fbf581302f0 2019-12-05T06:56:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.424+0000 7fbf5cbde700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=THROTTLE_DONE pgs=1102 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.424+0000 7fbf557fa700 1 -- 172.21.15.16:0/2096485518 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+1774 (secure 0 0 0) 0x7fbf50009960 con 0x7fbf581302f0 2019-12-05T06:56:49.427 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 198 2019-12-05T06:56:49.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:49.428 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:48.633646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:49.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.428+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbf3c023650 msgr2=0x7fbf3c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.428+0000 7fbf5e641700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbf3c023650 0x7fbf3c025ae0 crc :-1 s=READY pgs=680 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.428+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 msgr2=0x7fbf5812a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.428+0000 7fbf5e641700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 0x7fbf5812a670 secure :-1 s=READY pgs=1102 cs=0 l=1 rx=0x7fbf50010350 tx=0x7fbf500106f0).stop 2019-12-05T06:56:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.429+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 shutdown_connections 2019-12-05T06:56:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.429+0000 7fbf5e641700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbf3c023650 0x7fbf3c025ae0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.429+0000 7fbf5e641700 1 --2- 172.21.15.16:0/2096485518 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbf581302f0 0x7fbf5812a670 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.429+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 shutdown_connections 2019-12-05T06:56:49.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.429+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 wait complete. 2019-12-05T06:56:49.432 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.429+0000 7fbf5e641700 1 -- 172.21.15.16:0/2096485518 >> 172.21.15.16:0/2096485518 conn(0x7fbf580a2850 msgr2=0x7fbf5806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:49.441 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:56:49.441 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:56:49.442 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:49.584 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.578+0000 7f51ec17a700 1 Processor -- start 2019-12-05T06:56:49.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.578+0000 7f51ec17a700 1 -- start start 2019-12-05T06:56:49.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.579+0000 7f51ec17a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f51e408c6d0 0x7f51e408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:49.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.579+0000 7f51ec17a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51e408fef0 0x7f51e408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:49.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.579+0000 7f51ec17a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e408fb80 con 0x7f51e408ef20 2019-12-05T06:56:49.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.579+0000 7f51ec17a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e408df90 con 0x7f51e408fef0 2019-12-05T06:56:49.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.579+0000 7f51ec17a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e40a3f70 con 0x7f51e408c6d0 2019-12-05T06:56:49.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.579+0000 7f51e9f16700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51e408fef0 0x7f51e408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:49.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.579+0000 7f51e9f16700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f51e408fef0 0x7f51e408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36444/0 (socket says 172.21.15.16:36444) 2019-12-05T06:56:49.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.579+0000 7f51e9f16700 1 -- 172.21.15.16:0/559096521 learned_addr learned my addr 172.21.15.16:0/559096521 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:49.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.579+0000 7f51ea717700 1 --2- 172.21.15.16:0/559096521 >> v2:172.21.15.76:3300/0 conn(0x7f51e408c6d0 0x7f51e408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:49.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9f16700 1 -- 172.21.15.16:0/559096521 >> v2:172.21.15.76:3300/0 conn(0x7f51e408c6d0 msgr2=0x7f51e408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:49.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9f16700 1 --2- 172.21.15.16:0/559096521 >> v2:172.21.15.76:3300/0 conn(0x7f51e408c6d0 0x7f51e408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:49.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9f16700 1 -- 172.21.15.16:0/559096521 >> v1:172.21.15.76:6789/0 conn(0x7f51e408ef20 legacy=0x7f51e408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9f16700 1 -- 172.21.15.16:0/559096521 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e408d540 con 0x7f51e408fef0 2019-12-05T06:56:49.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51ea717700 1 --2- 172.21.15.16:0/559096521 >> v2:172.21.15.76:3300/0 conn(0x7f51e408c6d0 0x7f51e408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:56:49.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9f16700 1 --2- 172.21.15.16:0/559096521 >> v2:172.21.15.16:3300/0 conn(0x7f51e408fef0 0x7f51e408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f51d800f740 tx=0x7f51d800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9f16700 1 --2- 172.21.15.16:0/559096521 >> v2:172.21.15.16:3300/0 conn(0x7f51e408fef0 0x7f51e408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f51d800f740 tx=0x7f51d800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9f16700 1 --2- 172.21.15.16:0/559096521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51e408fef0 0x7f51e408d000 secure :-1 s=READY pgs=1635 cs=0 l=1 rx=0x7f51d800f740 tx=0x7f51d800fae0).ready entity=mon.0 client_cookie=ecd2966737d9f3d4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:49.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9f16700 1 --2- 172.21.15.16:0/559096521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51e408fef0 0x7f51e408d000 secure :-1 s=THROTTLE_DONE pgs=1635 cs=0 l=1 rx=0x7f51d800f740 tx=0x7f51d800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9f16700 1 --2- 172.21.15.16:0/559096521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51e408fef0 0x7f51e408d000 secure :-1 s=THROTTLE_DONE pgs=1635 cs=0 l=1 rx=0x7f51d800f740 tx=0x7f51d800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9715700 1 -- 172.21.15.16:0/559096521 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51d8000960 con 0x7f51e408fef0 2019-12-05T06:56:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9f16700 1 --2- 172.21.15.16:0/559096521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51e408fef0 0x7f51e408d000 secure :-1 s=THROTTLE_DONE pgs=1635 cs=0 l=1 rx=0x7f51d800f740 tx=0x7f51d800fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9715700 1 -- 172.21.15.16:0/559096521 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d8000c90 con 0x7f51e408fef0 2019-12-05T06:56:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51e9715700 1 -- 172.21.15.16:0/559096521 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51d80013a0 con 0x7f51e408fef0 2019-12-05T06:56:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51ec17a700 1 -- 172.21.15.16:0/559096521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51e408fef0 msgr2=0x7f51e408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.580+0000 7f51ec17a700 1 --2- 172.21.15.16:0/559096521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51e408fef0 0x7f51e408d000 secure :-1 s=READY pgs=1635 cs=0 l=1 rx=0x7f51d800f740 tx=0x7f51d800fae0).stop 2019-12-05T06:56:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.581+0000 7f51ec17a700 1 -- 172.21.15.16:0/559096521 shutdown_connections 2019-12-05T06:56:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.581+0000 7f51ec17a700 1 --2- 172.21.15.16:0/559096521 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f51e408fef0 0x7f51e408d000 unknown :-1 s=CLOSED pgs=1635 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:49.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.581+0000 7f51ec17a700 1 --2- 172.21.15.16:0/559096521 >> v2:172.21.15.76:3300/0 conn(0x7f51e408c6d0 0x7f51e408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.582+0000 7f51ec17a700 1 -- 172.21.15.16:0/559096521 shutdown_connections 2019-12-05T06:56:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.582+0000 7f51ec17a700 1 -- 172.21.15.16:0/559096521 wait complete. 2019-12-05T06:56:49.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.582+0000 7f51ec17a700 1 -- 172.21.15.16:0/559096521 >> 172.21.15.16:0/559096521 conn(0x7f51e4061940 msgr2=0x7f51e4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:49.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.584+0000 7f51ec17a700 1 Processor -- start 2019-12-05T06:56:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.584+0000 7f51ec17a700 1 -- start start 2019-12-05T06:56:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.585+0000 7f51ec17a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f51e414d400 0x7f51e414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.585+0000 7f51ec17a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e4098da0 con 0x7f51e4132530 2019-12-05T06:56:49.593 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.585+0000 7f51ec17a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51e409f0a0 con 0x7f51e4132d00 2019-12-05T06:56:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.585+0000 7f51ec17a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51e409bb50 con 0x7f51e414d400 2019-12-05T06:56:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.585+0000 7f51eaf18700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f51e4132530 0x7f51e4157100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49854/0 (socket says 172.21.15.16:49854) 2019-12-05T06:56:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.585+0000 7f51eaf18700 1 -- 172.21.15.16:0/2336202934 learned_addr learned my addr 172.21.15.16:0/2336202934 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.585+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> v2:172.21.15.76:3300/0 conn(0x7f51e414d400 0x7f51e414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:49.594 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424386244 0 0) 0x7f51dc001580 con 0x7f51e4132530 2019-12-05T06:56:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51c4002e80 con 0x7f51e4132530 2019-12-05T06:56:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3529550870 0 0) 0x7f51dc001780 con 0x7f51e4132530 2019-12-05T06:56:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f51c40032e0 con 0x7f51e4132530 2019-12-05T06:56:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f51dc001d10 con 0x7f51e4132530 2019-12-05T06:56:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51ea717700 1 -- 172.21.15.16:0/2336202934 >> v1:172.21.15.76:6789/0 conn(0x7f51e4132d00 legacy=0x7f51e4131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51ea717700 1 -- 172.21.15.16:0/2336202934 >> v1:172.21.15.16:6789/0 conn(0x7f51e4132530 legacy=0x7f51e4157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51ea717700 1 -- 172.21.15.16:0/2336202934 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51e409b0f0 con 0x7f51e414d400 2019-12-05T06:56:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> v2:172.21.15.76:3300/0 conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> v2:172.21.15.76:3300/0 conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.586+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=READY pgs=1105 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.587+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.587+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51d4000db0 con 0x7f51e414d400 2019-12-05T06:56:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.587+0000 7f51ec17a700 1 -- 172.21.15.16:0/2336202934 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51e409b0f0 con 0x7f51e414d400 2019-12-05T06:56:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.587+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.587+0000 7f51ec17a700 1 -- 172.21.15.16:0/2336202934 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51e409b0f0 con 0x7f51e414d400 2019-12-05T06:56:49.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.587+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51d40091e0 con 0x7f51e414d400 2019-12-05T06:56:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.587+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.587+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51d4000f30 con 0x7f51e414d400 2019-12-05T06:56:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.588+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.588+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f51d4000b50 con 0x7f51e414d400 2019-12-05T06:56:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.588+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.588+0000 7f51e2ffd700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51c4023670 0x7f51c4025b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:49.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.588+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f51d4045df0 con 0x7f51e414d400 2019-12-05T06:56:49.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.588+0000 7f51e9f16700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51c4023670 0x7f51c4025b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:49.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.589+0000 7f51e9f16700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51c4023670 0x7f51c4025b00 crc :-1 s=READY pgs=681 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:49.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.593+0000 7f51ec17a700 1 -- 172.21.15.16:0/2336202934 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51e409b0f0 con 0x7f51e414d400 2019-12-05T06:56:49.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.601+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.602 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.601+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f51d4000940 con 0x7f51e414d400 2019-12-05T06:56:49.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.805+0000 7f51ec17a700 1 -- 172.21.15.16:0/2336202934 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f51e409b0f0 con 0x7f51e414d400 2019-12-05T06:56:49.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.806+0000 7f51ea717700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=THROTTLE_DONE pgs=1105 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:49.808 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.806+0000 7f51e2ffd700 1 -- 172.21.15.16:0/2336202934 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 198 v198) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f51d4000940 con 0x7f51e414d400 2019-12-05T06:56:49.809 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 198 2019-12-05T06:56:49.810 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:49.810 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":198,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":198,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:48.633646+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:49.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.811+0000 7f51ec17a700 1 -- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51c4023670 msgr2=0x7f51c4025b00 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.811+0000 7f51ec17a700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51c4023670 0x7f51c4025b00 crc :-1 s=READY pgs=681 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:49.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.811+0000 7f51ec17a700 1 -- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 msgr2=0x7f51e414d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:49.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.811+0000 7f51ec17a700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 0x7f51e414d7f0 secure :-1 s=READY pgs=1105 cs=0 l=1 rx=0x7f51d4010350 tx=0x7f51d40106f0).stop 2019-12-05T06:56:49.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.811+0000 7f51ec17a700 1 -- 172.21.15.16:0/2336202934 shutdown_connections 2019-12-05T06:56:49.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.811+0000 7f51ec17a700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f51c4023670 0x7f51c4025b00 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:49.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.811+0000 7f51ec17a700 1 --2- 172.21.15.16:0/2336202934 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f51e414d400 0x7f51e414d7f0 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:49.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.811+0000 7f51ec17a700 1 -- 172.21.15.16:0/2336202934 shutdown_connections 2019-12-05T06:56:49.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.811+0000 7f51ec17a700 1 -- 172.21.15.16:0/2336202934 wait complete. 2019-12-05T06:56:49.814 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:49.811+0000 7f51ec17a700 1 -- 172.21.15.16:0/2336202934 >> 172.21.15.16:0/2336202934 conn(0x7f51e40714f0 msgr2=0x7f51e409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:49.822 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 198, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 198, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8308": { "addr": "172.21.15.16:6833/2914676212", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 2914676212, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 2914676212, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8308, "incarnation": 198, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 83, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:56:48.633646+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8308 } } } ], "standbys": [] } 2019-12-05T06:56:51.823 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:52.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.092+0000 7f81b126a700 1 Processor -- start 2019-12-05T06:56:52.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.092+0000 7f81b126a700 1 -- start start 2019-12-05T06:56:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.094+0000 7f81b126a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac08c6d0 0x7f81ac08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.094+0000 7f81b126a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f81ac08fef0 0x7f81ac08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.094+0000 7f81b126a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81ac0958e0 con 0x7f81ac08ef20 2019-12-05T06:56:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.094+0000 7f81b126a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81ac0a3f70 con 0x7f81ac08fef0 2019-12-05T06:56:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.094+0000 7f81b126a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac08d830 con 0x7f81ac08c6d0 2019-12-05T06:56:52.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.094+0000 7f81aaffd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f81ac08fef0 0x7f81ac08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.094+0000 7f81ab7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac08c6d0 0x7f81ac08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.094+0000 7f81aaffd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f81ac08fef0 0x7f81ac08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:50782/0 (socket says 172.21.15.16:50782) 2019-12-05T06:56:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.094+0000 7f81ab7fe700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac08c6d0 0x7f81ac08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:40450/0 (socket says 172.21.15.16:40450) 2019-12-05T06:56:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.094+0000 7f81aaffd700 1 -- 172.21.15.16:0/1890250344 learned_addr learned my addr 172.21.15.16:0/1890250344 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.095+0000 7f81aaffd700 1 -- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac08c6d0 msgr2=0x7f81ac08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.095+0000 7f81aaffd700 1 --2- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac08c6d0 0x7f81ac08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:52.138 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.095+0000 7f81aaffd700 1 -- 172.21.15.16:0/1890250344 >> v1:172.21.15.76:6789/0 conn(0x7f81ac08ef20 legacy=0x7f81ac08f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.095+0000 7f81aaffd700 1 -- 172.21.15.16:0/1890250344 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac08d540 con 0x7f81ac08fef0 2019-12-05T06:56:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.095+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac08c6d0 0x7f81ac08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:56:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.095+0000 7f81aaffd700 1 --2- 172.21.15.16:0/1890250344 >> v2:172.21.15.76:3300/0 conn(0x7f81ac08fef0 0x7f81ac08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f819400f740 tx=0x7f819400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.095+0000 7f81aaffd700 1 --2- 172.21.15.16:0/1890250344 >> v2:172.21.15.76:3300/0 conn(0x7f81ac08fef0 0x7f81ac08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f819400f740 tx=0x7f819400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.095+0000 7f81aaffd700 1 --2- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81ac08fef0 0x7f81ac08d000 secure :-1 s=READY pgs=1107 cs=0 l=1 rx=0x7f819400f740 tx=0x7f819400fae0).ready entity=mon.1 client_cookie=5c4bc9f776436d50 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.096+0000 7f81aaffd700 1 --2- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81ac08fef0 0x7f81ac08d000 secure :-1 s=THROTTLE_DONE pgs=1107 cs=0 l=1 rx=0x7f819400f740 tx=0x7f819400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.096+0000 7f81aaffd700 1 --2- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81ac08fef0 0x7f81ac08d000 secure :-1 s=THROTTLE_DONE pgs=1107 cs=0 l=1 rx=0x7f819400f740 tx=0x7f819400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.096+0000 7f81aa7fc700 1 -- 172.21.15.16:0/1890250344 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8194000960 con 0x7f81ac08fef0 2019-12-05T06:56:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.096+0000 7f81aaffd700 1 --2- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81ac08fef0 0x7f81ac08d000 secure :-1 s=THROTTLE_DONE pgs=1107 cs=0 l=1 rx=0x7f819400f740 tx=0x7f819400fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.096+0000 7f81aa7fc700 1 -- 172.21.15.16:0/1890250344 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8194000c90 con 0x7f81ac08fef0 2019-12-05T06:56:52.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.096+0000 7f81aa7fc700 1 -- 172.21.15.16:0/1890250344 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81940013a0 con 0x7f81ac08fef0 2019-12-05T06:56:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.096+0000 7f81b126a700 1 -- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81ac08fef0 msgr2=0x7f81ac08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.096+0000 7f81b126a700 1 --2- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81ac08fef0 0x7f81ac08d000 secure :-1 s=READY pgs=1107 cs=0 l=1 rx=0x7f819400f740 tx=0x7f819400fae0).stop 2019-12-05T06:56:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.097+0000 7f81b126a700 1 -- 172.21.15.16:0/1890250344 shutdown_connections 2019-12-05T06:56:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.097+0000 7f81b126a700 1 --2- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f81ac08fef0 0x7f81ac08d000 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:52.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.097+0000 7f81b126a700 1 --2- 172.21.15.16:0/1890250344 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac08c6d0 0x7f81ac08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.097+0000 7f81b126a700 1 -- 172.21.15.16:0/1890250344 shutdown_connections 2019-12-05T06:56:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.097+0000 7f81b126a700 1 -- 172.21.15.16:0/1890250344 wait complete. 2019-12-05T06:56:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.097+0000 7f81b126a700 1 -- 172.21.15.16:0/1890250344 >> 172.21.15.16:0/1890250344 conn(0x7f81ac061940 msgr2=0x7f81ac096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.102+0000 7f81b126a700 1 Processor -- start 2019-12-05T06:56:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.102+0000 7f81b126a700 1 -- start start 2019-12-05T06:56:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.103+0000 7f81b126a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81ac1325e0 0x7f81ac12b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.103+0000 7f81b126a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f81ac12c740 0x7f81ac12bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:52.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.103+0000 7f81b126a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac12c1f0 0x7f81ac06dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.103+0000 7f81b126a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81ac12a7e0 con 0x7f81ac1325e0 2019-12-05T06:56:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.103+0000 7f81b126a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81ac129db0 con 0x7f81ac12c740 2019-12-05T06:56:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.103+0000 7f81b126a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81ac12a080 con 0x7f81ac12c1f0 2019-12-05T06:56:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81ab7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81ac1325e0 0x7f81ac12b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81ab7fe700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f81ac1325e0 0x7f81ac12b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36466/0 (socket says 172.21.15.16:36466) 2019-12-05T06:56:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81ab7fe700 1 -- 172.21.15.16:0/4119206806 learned_addr learned my addr 172.21.15.16:0/4119206806 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:52.143 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81aaffd700 1 --2- 172.21.15.16:0/4119206806 >> v2:172.21.15.76:3300/0 conn(0x7f81ac12c740 0x7f81ac12bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:52.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81abfff700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac12c1f0 0x7f81ac06dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:52.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81ab7fe700 1 -- 172.21.15.16:0/4119206806 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac12c1f0 msgr2=0x7f81ac06dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:52.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac12c1f0 0x7f81ac06dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:52.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81ab7fe700 1 -- 172.21.15.16:0/4119206806 >> v2:172.21.15.76:3300/0 conn(0x7f81ac12c740 msgr2=0x7f81ac12bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:52.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> v2:172.21.15.76:3300/0 conn(0x7f81ac12c740 0x7f81ac12bcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:52.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81ab7fe700 1 -- 172.21.15.16:0/4119206806 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81ac12ae70 con 0x7f81ac1325e0 2019-12-05T06:56:52.144 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.104+0000 7f81aaffd700 1 --2- 172.21.15.16:0/4119206806 >> v2:172.21.15.76:3300/0 conn(0x7f81ac12c740 0x7f81ac12bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:56:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.105+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> v2:172.21.15.16:3300/0 conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.105+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> v2:172.21.15.16:3300/0 conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.105+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=READY pgs=1637 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.105+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=THROTTLE_DONE pgs=1637 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.105+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=THROTTLE_DONE pgs=1637 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.105+0000 7f81a3fff700 1 -- 172.21.15.16:0/4119206806 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81a4000cc0 con 0x7f81ac1325e0 2019-12-05T06:56:52.145 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.105+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=THROTTLE_DONE pgs=1637 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.105+0000 7f81a3fff700 1 -- 172.21.15.16:0/4119206806 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f81a4009af0 con 0x7f81ac1325e0 2019-12-05T06:56:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.105+0000 7f81a3fff700 1 -- 172.21.15.16:0/4119206806 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f81a400a180 con 0x7f81ac1325e0 2019-12-05T06:56:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.105+0000 7f81b126a700 1 -- 172.21.15.16:0/4119206806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81ac12ae70 con 0x7f81ac1325e0 2019-12-05T06:56:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.106+0000 7f81b126a700 1 -- 172.21.15.16:0/4119206806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81ac12ae70 con 0x7f81ac1325e0 2019-12-05T06:56:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.106+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=THROTTLE_DONE pgs=1637 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.107+0000 7f81a3fff700 1 -- 172.21.15.16:0/4119206806 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f81a4009ce0 con 0x7f81ac1325e0 2019-12-05T06:56:52.146 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.107+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=THROTTLE_DONE pgs=1637 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.107+0000 7f81a3fff700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8184023630 0x7f8184025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:52.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.107+0000 7f81a3fff700 1 -- 172.21.15.16:0/4119206806 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f81a40460c0 con 0x7f81ac1325e0 2019-12-05T06:56:52.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.108+0000 7f81aaffd700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8184023630 0x7f8184025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:52.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.108+0000 7f81aaffd700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8184023630 0x7f8184025ac0 crc :-1 s=READY pgs=682 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:52.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.123+0000 7f81b126a700 1 -- 172.21.15.16:0/4119206806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81ac12ae70 con 0x7f81ac1325e0 2019-12-05T06:56:52.147 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.127+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=THROTTLE_DONE pgs=1637 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.127+0000 7f81a3fff700 1 -- 172.21.15.16:0/4119206806 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f81a400a930 con 0x7f81ac1325e0 2019-12-05T06:56:52.350 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.349+0000 7f81b126a700 1 -- 172.21.15.16:0/4119206806 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f81ac12ae70 con 0x7f81ac1325e0 2019-12-05T06:56:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.350+0000 7f81ab7fe700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=THROTTLE_DONE pgs=1637 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:52.351 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.350+0000 7f81a3fff700 1 -- 172.21.15.16:0/4119206806 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 200 v200) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f81a4009cb0 con 0x7f81ac1325e0 2019-12-05T06:56:52.353 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:52.353 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":200,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":200,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:52.056311+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:rejoin","state_seq":14,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:52.353 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 200 2019-12-05T06:56:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 -- 172.21.15.16:0/4119206806 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8184023630 msgr2=0x7f8184025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8184023630 0x7f8184025ac0 crc :-1 s=READY pgs=682 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 -- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 msgr2=0x7f81ac12b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:52.356 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 0x7f81ac12b770 secure :-1 s=READY pgs=1637 cs=0 l=1 rx=0x7f81a4010790 tx=0x7f81a4010610).stop 2019-12-05T06:56:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 -- 172.21.15.16:0/4119206806 shutdown_connections 2019-12-05T06:56:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f81ac12c1f0 0x7f81ac06dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 --2- 172.21.15.16:0/4119206806 >> v2:172.21.15.76:3300/0 conn(0x7f81ac12c740 0x7f81ac12bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8184023630 0x7f8184025ac0 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 --2- 172.21.15.16:0/4119206806 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f81ac1325e0 0x7f81ac12b770 unknown :-1 s=CLOSED pgs=1637 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 -- 172.21.15.16:0/4119206806 shutdown_connections 2019-12-05T06:56:52.357 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 -- 172.21.15.16:0/4119206806 wait complete. 2019-12-05T06:56:52.358 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:52.354+0000 7f81b126a700 1 -- 172.21.15.16:0/4119206806 >> 172.21.15.16:0/4119206806 conn(0x7f81ac0a2850 msgr2=0x7f81ac06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:52.366 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:56:54.367 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:56:54.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.639+0000 7fd197dcb700 1 Processor -- start 2019-12-05T06:56:54.649 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.640+0000 7fd197dcb700 1 -- start start 2019-12-05T06:56:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd197dcb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd19008c6d0 0x7fd19008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd197dcb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd19008f720 0x7fd19008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:56:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd197dcb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd19008fb70 con 0x7fd19008fef0 2019-12-05T06:56:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd197dcb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd19008df90 con 0x7fd19008c6d0 2019-12-05T06:56:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd197dcb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1900a3f70 con 0x7fd19008f720 2019-12-05T06:56:54.650 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd196368700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd19008c6d0 0x7fd19008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd196368700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd19008c6d0 0x7fd19008cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36478/0 (socket says 172.21.15.16:36478) 2019-12-05T06:56:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd196368700 1 -- 172.21.15.16:0/2644795350 learned_addr learned my addr 172.21.15.16:0/2644795350 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:56:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd196b69700 1 --2- 172.21.15.16:0/2644795350 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd19008f720 0x7fd19008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd196368700 1 -- 172.21.15.16:0/2644795350 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd19008f720 msgr2=0x7fd19008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:56:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd196368700 1 --2- 172.21.15.16:0/2644795350 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd19008f720 0x7fd19008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd196368700 1 -- 172.21.15.16:0/2644795350 >> v1:172.21.15.76:6789/0 conn(0x7fd19008fef0 legacy=0x7fd19008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:54.651 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.642+0000 7fd196368700 1 -- 172.21.15.16:0/2644795350 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19008d540 con 0x7fd19008c6d0 2019-12-05T06:56:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd196368700 1 --2- 172.21.15.16:0/2644795350 >> v2:172.21.15.16:3300/0 conn(0x7fd19008c6d0 0x7fd19008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fd18c00fac0 tx=0x7fd18c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd196368700 1 --2- 172.21.15.16:0/2644795350 >> v2:172.21.15.16:3300/0 conn(0x7fd19008c6d0 0x7fd19008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fd18c00fac0 tx=0x7fd18c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd196368700 1 --2- 172.21.15.16:0/2644795350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19008c6d0 0x7fd19008cac0 secure :-1 s=READY pgs=1638 cs=0 l=1 rx=0x7fd18c00fac0 tx=0x7fd18c00f8f0).ready entity=mon.0 client_cookie=11d2cd4605fa7c2b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd196368700 1 --2- 172.21.15.16:0/2644795350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19008c6d0 0x7fd19008cac0 secure :-1 s=THROTTLE_DONE pgs=1638 cs=0 l=1 rx=0x7fd18c00fac0 tx=0x7fd18c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd196368700 1 --2- 172.21.15.16:0/2644795350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19008c6d0 0x7fd19008cac0 secure :-1 s=THROTTLE_DONE pgs=1638 cs=0 l=1 rx=0x7fd18c00fac0 tx=0x7fd18c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd196368700 1 --2- 172.21.15.16:0/2644795350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19008c6d0 0x7fd19008cac0 secure :-1 s=THROTTLE_DONE pgs=1638 cs=0 l=1 rx=0x7fd18c00fac0 tx=0x7fd18c00f8f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.652 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd195366700 1 -- 172.21.15.16:0/2644795350 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18c000ce0 con 0x7fd19008c6d0 2019-12-05T06:56:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd195366700 1 -- 172.21.15.16:0/2644795350 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18c001010 con 0x7fd19008c6d0 2019-12-05T06:56:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd195366700 1 -- 172.21.15.16:0/2644795350 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18c001620 con 0x7fd19008c6d0 2019-12-05T06:56:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd197dcb700 1 -- 172.21.15.16:0/2644795350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19008c6d0 msgr2=0x7fd19008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.643+0000 7fd197dcb700 1 --2- 172.21.15.16:0/2644795350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19008c6d0 0x7fd19008cac0 secure :-1 s=READY pgs=1638 cs=0 l=1 rx=0x7fd18c00fac0 tx=0x7fd18c00f8f0).stop 2019-12-05T06:56:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.644+0000 7fd197dcb700 1 -- 172.21.15.16:0/2644795350 shutdown_connections 2019-12-05T06:56:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.644+0000 7fd197dcb700 1 --2- 172.21.15.16:0/2644795350 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd19008f720 0x7fd19008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:56:54.653 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.644+0000 7fd197dcb700 1 --2- 172.21.15.16:0/2644795350 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd19008c6d0 0x7fd19008cac0 unknown :-1 s=CLOSED pgs=1638 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.645+0000 7fd197dcb700 1 -- 172.21.15.16:0/2644795350 shutdown_connections 2019-12-05T06:56:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.645+0000 7fd197dcb700 1 -- 172.21.15.16:0/2644795350 wait complete. 2019-12-05T06:56:54.654 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.645+0000 7fd197dcb700 1 -- 172.21.15.16:0/2644795350 >> 172.21.15.16:0/2644795350 conn(0x7fd190061940 msgr2=0x7fd190096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:54.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.649+0000 7fd197dcb700 1 Processor -- start 2019-12-05T06:56:54.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.649+0000 7fd197dcb700 1 -- start start 2019-12-05T06:56:54.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.651+0000 7fd197dcb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd190096e90 0x7fd19009b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:54.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.651+0000 7fd197dcb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd190084f30 0x7fd19009b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:54.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.651+0000 7fd197dcb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd190084b90 con 0x7fd190097280 2019-12-05T06:56:54.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.651+0000 7fd197dcb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd19009a450 con 0x7fd190096e90 2019-12-05T06:56:54.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.651+0000 7fd197dcb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd190099a20 con 0x7fd190084f30 2019-12-05T06:56:54.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.651+0000 7fd196368700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fd190096e90 0x7fd19009b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:54.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.651+0000 7fd196b69700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fd190097280 0x7fd1900976c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49888/0 (socket says 172.21.15.16:49888) 2019-12-05T06:56:54.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.651+0000 7fd196b69700 1 -- 172.21.15.16:0/569119955 learned_addr learned my addr 172.21.15.16:0/569119955 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:56:54.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.651+0000 7fd195b67700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd190084f30 0x7fd19009b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:54.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd196368700 1 -- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd190084f30 msgr2=0x7fd19009b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:54.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd190084f30 0x7fd19009b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:54.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd196368700 1 -- 172.21.15.16:0/569119955 >> v1:172.21.15.16:6789/0 conn(0x7fd190097280 legacy=0x7fd1900976c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:54.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd196368700 1 -- 172.21.15.16:0/569119955 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19009aae0 con 0x7fd190096e90 2019-12-05T06:56:54.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd195b67700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd190084f30 0x7fd19009b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:56:54.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> v2:172.21.15.16:3300/0 conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> v2:172.21.15.16:3300/0 conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:54.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=THROTTLE_DONE pgs=1640 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd186ffd700 1 -- 172.21.15.16:0/569119955 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18c0058e0 con 0x7fd190096e90 2019-12-05T06:56:54.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=THROTTLE_DONE pgs=1640 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=THROTTLE_DONE pgs=1640 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd186ffd700 1 -- 172.21.15.16:0/569119955 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd18c005500 con 0x7fd190096e90 2019-12-05T06:56:54.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.652+0000 7fd197dcb700 1 -- 172.21.15.16:0/569119955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd19009aae0 con 0x7fd190096e90 2019-12-05T06:56:54.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.653+0000 7fd186ffd700 1 -- 172.21.15.16:0/569119955 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18c005ec0 con 0x7fd190096e90 2019-12-05T06:56:54.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.653+0000 7fd197dcb700 1 -- 172.21.15.16:0/569119955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd19009aae0 con 0x7fd190096e90 2019-12-05T06:56:54.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.654+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=THROTTLE_DONE pgs=1640 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.654+0000 7fd186ffd700 1 -- 172.21.15.16:0/569119955 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fd18c0056f0 con 0x7fd190096e90 2019-12-05T06:56:54.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.654+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=THROTTLE_DONE pgs=1640 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.655+0000 7fd186ffd700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd178023630 0x7fd178025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:56:54.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.655+0000 7fd186ffd700 1 -- 172.21.15.16:0/569119955 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7fd18c045ef0 con 0x7fd190096e90 2019-12-05T06:56:54.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.655+0000 7fd195b67700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd178023630 0x7fd178025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:56:54.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.655+0000 7fd195b67700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd178023630 0x7fd178025ac0 crc :-1 s=READY pgs=683 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:56:54.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.670+0000 7fd197dcb700 1 -- 172.21.15.16:0/569119955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd19009aae0 con 0x7fd190096e90 2019-12-05T06:56:54.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.677+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=THROTTLE_DONE pgs=1640 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.678 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.677+0000 7fd186ffd700 1 -- 172.21.15.16:0/569119955 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fd18c007690 con 0x7fd190096e90 2019-12-05T06:56:54.878 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.877+0000 7fd197dcb700 1 -- 172.21.15.16:0/569119955 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd19009aae0 con 0x7fd190096e90 2019-12-05T06:56:54.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.878+0000 7fd196368700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=THROTTLE_DONE pgs=1640 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:56:54.879 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.878+0000 7fd186ffd700 1 -- 172.21.15.16:0/569119955 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fd18c0056f0 con 0x7fd190096e90 2019-12-05T06:56:54.879 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 201 2019-12-05T06:56:54.879 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:56:54.880 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":201,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:53.063332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:56:54.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 -- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd178023630 msgr2=0x7fd178025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:54.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd178023630 0x7fd178025ac0 crc :-1 s=READY pgs=683 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:54.882 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 -- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 msgr2=0x7fd19009b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:56:54.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 0x7fd19009b3e0 secure :-1 s=READY pgs=1640 cs=0 l=1 rx=0x7fd18c0106f0 tx=0x7fd18c010a90).stop 2019-12-05T06:56:54.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 -- 172.21.15.16:0/569119955 shutdown_connections 2019-12-05T06:56:54.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fd190084f30 0x7fd19009b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:54.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fd178023630 0x7fd178025ac0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:54.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 --2- 172.21.15.16:0/569119955 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fd190096e90 0x7fd19009b3e0 unknown :-1 s=CLOSED pgs=1640 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:56:54.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 -- 172.21.15.16:0/569119955 shutdown_connections 2019-12-05T06:56:54.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 -- 172.21.15.16:0/569119955 wait complete. 2019-12-05T06:56:54.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:56:54.881+0000 7fd197dcb700 1 -- 172.21.15.16:0/569119955 >> 172.21.15.16:0/569119955 conn(0x7fd1900714f0 msgr2=0x7fd19009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:56:54.893 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=6212 2019-12-05T06:56:54.893 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.b 2019-12-05T06:57:01.894 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:57:01.894 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:57:01.895 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:57:01.900 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:57:01.900 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:02.057 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.046+0000 7fc898226700 1 Processor -- start 2019-12-05T06:57:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.046+0000 7fc898226700 1 -- start start 2019-12-05T06:57:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.047+0000 7fc898226700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc89008d2c0 0x7fc89008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.047+0000 7fc898226700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc89008fef0 0x7fc89008f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.047+0000 7fc898226700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8900958e0 con 0x7fc89008c6d0 2019-12-05T06:57:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.047+0000 7fc898226700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc89008df90 con 0x7fc89008fef0 2019-12-05T06:57:02.058 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.047+0000 7fc898226700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8900a3f70 con 0x7fc89008d2c0 2019-12-05T06:57:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.047+0000 7fc8967c3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc89008c6d0 0x7fc89008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49942/0 (socket says 172.21.15.16:49942) 2019-12-05T06:57:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.047+0000 7fc8967c3700 1 -- 172.21.15.16:0/2321707188 learned_addr learned my addr 172.21.15.16:0/2321707188 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.047+0000 7fc895fc2700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc89008d2c0 0x7fc89008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.047+0000 7fc896fc4700 1 --2- 172.21.15.16:0/2321707188 >> v2:172.21.15.76:3300/0 conn(0x7fc89008fef0 0x7fc89008f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:02.059 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702965637 0 0) 0x7fc88c001150 con 0x7fc89008c6d0 2019-12-05T06:57:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc87c002e80 con 0x7fc89008c6d0 2019-12-05T06:57:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2214931838 0 0) 0x7fc88c001130 con 0x7fc89008c6d0 2019-12-05T06:57:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc89008d2c0 msgr2=0x7fc89008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc89008d2c0 0x7fc89008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 >> v2:172.21.15.76:3300/0 conn(0x7fc89008fef0 msgr2=0x7fc89008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:02.060 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 --2- 172.21.15.16:0/2321707188 >> v2:172.21.15.76:3300/0 conn(0x7fc89008fef0 0x7fc89008f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:02.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc89008e620 con 0x7fc89008c6d0 2019-12-05T06:57:02.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7fc88c0016c0 con 0x7fc89008c6d0 2019-12-05T06:57:02.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 >> v1:172.21.15.16:6789/0 conn(0x7fc89008c6d0 legacy=0x7fc89008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:02.061 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc87c0479e0 0x7fc87c049e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:02.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87c04a5a0 0x7fc87c04ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:02.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc87c04d270 0x7fc87c04f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:02.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c04ff50 con 0x7fc87c04a5a0 2019-12-05T06:57:02.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c050140 con 0x7fc87c04d270 2019-12-05T06:57:02.062 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc87c050370 con 0x7fc87c0479e0 2019-12-05T06:57:02.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc895fc2700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87c04a5a0 0x7fc87c04ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:02.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc8967c3700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc87c0479e0 0x7fc87c049e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:02.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc896fc4700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc87c04d270 0x7fc87c04f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:02.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc895fc2700 1 -- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc87c0479e0 msgr2=0x7fc87c049e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:02.063 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc895fc2700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc87c0479e0 0x7fc87c049e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc895fc2700 1 -- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc87c04d270 msgr2=0x7fc87c04f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc895fc2700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc87c04d270 0x7fc87c04f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc895fc2700 1 -- 172.21.15.16:0/2321707188 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc87c050620 con 0x7fc87c04a5a0 2019-12-05T06:57:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.048+0000 7fc896fc4700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc87c04d270 0x7fc87c04f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:57:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc895fc2700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87c04a5a0 0x7fc87c04ca70 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc88000f660 tx=0x7fc880002b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.064 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc895fc2700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87c04a5a0 0x7fc87c04ca70 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc88000f660 tx=0x7fc880002b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc895fc2700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87c04a5a0 0x7fc87c04ca70 secure :-1 s=READY pgs=1643 cs=0 l=1 rx=0x7fc88000f660 tx=0x7fc880002b90).ready entity=mon.0 client_cookie=6726aabd9a8e0554 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc895fc2700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87c04a5a0 0x7fc87c04ca70 secure :-1 s=THROTTLE_DONE pgs=1643 cs=0 l=1 rx=0x7fc88000f660 tx=0x7fc880002b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc880000940 con 0x7fc87c04a5a0 2019-12-05T06:57:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc895fc2700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87c04a5a0 0x7fc87c04ca70 secure :-1 s=THROTTLE_DONE pgs=1643 cs=0 l=1 rx=0x7fc88000f660 tx=0x7fc880002b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.065 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc895fc2700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87c04a5a0 0x7fc87c04ca70 secure :-1 s=THROTTLE_DONE pgs=1643 cs=0 l=1 rx=0x7fc88000f660 tx=0x7fc880002b90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc880000ea0 con 0x7fc87c04a5a0 2019-12-05T06:57:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc8957c1700 1 -- 172.21.15.16:0/2321707188 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc880001530 con 0x7fc87c04a5a0 2019-12-05T06:57:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc898226700 1 -- 172.21.15.16:0/2321707188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87c04a5a0 msgr2=0x7fc87c04ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc898226700 1 --2- 172.21.15.16:0/2321707188 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc87c04a5a0 0x7fc87c04ca70 secure :-1 s=READY pgs=1643 cs=0 l=1 rx=0x7fc88000f660 tx=0x7fc880002b90).stop 2019-12-05T06:57:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.049+0000 7fc896fc4700 1 -- 172.21.15.16:0/2321707188 reap_dead start 2019-12-05T06:57:02.066 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.050+0000 7fc898226700 1 -- 172.21.15.16:0/2321707188 shutdown_connections 2019-12-05T06:57:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.050+0000 7fc898226700 1 -- 172.21.15.16:0/2321707188 shutdown_connections 2019-12-05T06:57:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.050+0000 7fc898226700 1 -- 172.21.15.16:0/2321707188 wait complete. 2019-12-05T06:57:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.050+0000 7fc898226700 1 -- 172.21.15.16:0/2321707188 >> 172.21.15.16:0/2321707188 conn(0x7fc890061940 msgr2=0x7fc890096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.052+0000 7fc898226700 1 Processor -- start 2019-12-05T06:57:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.052+0000 7fc898226700 1 -- start start 2019-12-05T06:57:02.067 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc898226700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc89009e7d0 0x7fc89009e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc898226700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc890156e70 0x7fc890159270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc898226700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc89012cb90 con 0x7fc8901302f0 2019-12-05T06:57:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc898226700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc890131ff0 con 0x7fc89009e7d0 2019-12-05T06:57:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc898226700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8901315c0 con 0x7fc890156e70 2019-12-05T06:57:02.068 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc8967c3700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc8901302f0 0x7fc89009d800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49952/0 (socket says 172.21.15.16:49952) 2019-12-05T06:57:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc8967c3700 1 -- 172.21.15.16:0/3215179767 learned_addr learned my addr 172.21.15.16:0/3215179767 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> v2:172.21.15.76:3300/0 conn(0x7fc89009e7d0 0x7fc89009e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc896fc4700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc890156e70 0x7fc890159270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:02.069 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc886ffd700 1 -- 172.21.15.16:0/3215179767 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844857947 0 0) 0x7fc88c000c90 con 0x7fc8901302f0 2019-12-05T06:57:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.053+0000 7fc886ffd700 1 -- 172.21.15.16:0/3215179767 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc878002e80 con 0x7fc8901302f0 2019-12-05T06:57:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc895fc2700 1 -- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc890156e70 msgr2=0x7fc890159270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc890156e70 0x7fc890159270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc895fc2700 1 -- 172.21.15.16:0/3215179767 >> v1:172.21.15.16:6789/0 conn(0x7fc8901302f0 legacy=0x7fc89009d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:02.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc895fc2700 1 -- 172.21.15.16:0/3215179767 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc890132680 con 0x7fc89009e7d0 2019-12-05T06:57:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> v2:172.21.15.76:3300/0 conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> v2:172.21.15.76:3300/0 conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=READY pgs=1109 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:02.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc886ffd700 1 -- 172.21.15.16:0/3215179767 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc880002260 con 0x7fc89009e7d0 2019-12-05T06:57:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc886ffd700 1 -- 172.21.15.16:0/3215179767 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc880000b30 con 0x7fc89009e7d0 2019-12-05T06:57:02.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.074 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc898226700 1 -- 172.21.15.16:0/3215179767 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc890132680 con 0x7fc89009e7d0 2019-12-05T06:57:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc886ffd700 1 -- 172.21.15.16:0/3215179767 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc880000d20 con 0x7fc89009e7d0 2019-12-05T06:57:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.054+0000 7fc898226700 1 -- 172.21.15.16:0/3215179767 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc890132680 con 0x7fc89009e7d0 2019-12-05T06:57:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.055+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.056+0000 7fc886ffd700 1 -- 172.21.15.16:0/3215179767 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc880000940 con 0x7fc89009e7d0 2019-12-05T06:57:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.056+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.075 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.056+0000 7fc886ffd700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc878023670 0x7fc878025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.056+0000 7fc886ffd700 1 -- 172.21.15.16:0/3215179767 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7fc880045c90 con 0x7fc89009e7d0 2019-12-05T06:57:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.056+0000 7fc8967c3700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc878023670 0x7fc878025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:02.076 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.057+0000 7fc8967c3700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc878023670 0x7fc878025ae0 crc :-1 s=READY pgs=684 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.061+0000 7fc898226700 1 -- 172.21.15.16:0/3215179767 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc890132680 con 0x7fc89009e7d0 2019-12-05T06:57:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.067+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.077 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.067+0000 7fc886ffd700 1 -- 172.21.15.16:0/3215179767 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc880002620 con 0x7fc89009e7d0 2019-12-05T06:57:02.081 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:57:02.288 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.288+0000 7fc898226700 1 -- 172.21.15.16:0/3215179767 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc890132680 con 0x7fc89009e7d0 2019-12-05T06:57:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.289+0000 7fc895fc2700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=THROTTLE_DONE pgs=1109 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:02.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.289+0000 7fc886ffd700 1 -- 172.21.15.16:0/3215179767 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 201 v201) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fc880002620 con 0x7fc89009e7d0 2019-12-05T06:57:02.292 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 201 2019-12-05T06:57:02.293 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:02.293 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":201,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:53.063332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:02.295 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 -- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc878023670 msgr2=0x7fc878025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:02.296 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc878023670 0x7fc878025ae0 crc :-1 s=READY pgs=684 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:02.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 -- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 msgr2=0x7fc89009e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:02.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 0x7fc89009e000 secure :-1 s=READY pgs=1109 cs=0 l=1 rx=0x7fc8800103b0 tx=0x7fc8800101e0).stop 2019-12-05T06:57:02.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 -- 172.21.15.16:0/3215179767 shutdown_connections 2019-12-05T06:57:02.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc878023670 0x7fc878025ae0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:02.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc89009e7d0 0x7fc89009e000 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:02.297 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 --2- 172.21.15.16:0/3215179767 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc890156e70 0x7fc890159270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:02.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 -- 172.21.15.16:0/3215179767 shutdown_connections 2019-12-05T06:57:02.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 -- 172.21.15.16:0/3215179767 wait complete. 2019-12-05T06:57:02.298 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:02.295+0000 7fc898226700 1 -- 172.21.15.16:0/3215179767 >> 172.21.15.16:0/3215179767 conn(0x7fc8900a2850 msgr2=0x7fc89006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:02.310 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:57:04.311 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.548+0000 7fa98380b700 1 Processor -- start 2019-12-05T06:57:04.554 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.548+0000 7fa98380b700 1 -- start start 2019-12-05T06:57:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.549+0000 7fa98380b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa97c08c6d0 0x7fa97c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.549+0000 7fa98380b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa97c08f720 0x7fa97c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.549+0000 7fa98380b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa97c08fb80 con 0x7fa97c08fef0 2019-12-05T06:57:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.549+0000 7fa98380b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97c08df90 con 0x7fa97c08c6d0 2019-12-05T06:57:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.549+0000 7fa98380b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97c0a3f70 con 0x7fa97c08f720 2019-12-05T06:57:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa981da8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa97c08c6d0 0x7fa97c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:04.555 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa981da8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fa97c08c6d0 0x7fa97c08cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36566/0 (socket says 172.21.15.16:36566) 2019-12-05T06:57:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa981da8700 1 -- 172.21.15.16:0/2938217994 learned_addr learned my addr 172.21.15.16:0/2938217994 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:57:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa9825a9700 1 --2- 172.21.15.16:0/2938217994 >> v2:172.21.15.76:3300/0 conn(0x7fa97c08f720 0x7fa97c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa981da8700 1 -- 172.21.15.16:0/2938217994 >> v2:172.21.15.76:3300/0 conn(0x7fa97c08f720 msgr2=0x7fa97c08d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:04.556 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa981da8700 1 --2- 172.21.15.16:0/2938217994 >> v2:172.21.15.76:3300/0 conn(0x7fa97c08f720 0x7fa97c08d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa981da8700 1 -- 172.21.15.16:0/2938217994 >> v1:172.21.15.76:6789/0 conn(0x7fa97c08fef0 legacy=0x7fa97c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:04.557 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa981da8700 1 -- 172.21.15.16:0/2938217994 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c08d540 con 0x7fa97c08c6d0 2019-12-05T06:57:04.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa981da8700 1 --2- 172.21.15.16:0/2938217994 >> v2:172.21.15.16:3300/0 conn(0x7fa97c08c6d0 0x7fa97c08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fa96c00f6c0 tx=0x7fa96c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa981da8700 1 --2- 172.21.15.16:0/2938217994 >> v2:172.21.15.16:3300/0 conn(0x7fa97c08c6d0 0x7fa97c08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fa96c00f6c0 tx=0x7fa96c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.550+0000 7fa981da8700 1 --2- 172.21.15.16:0/2938217994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa97c08c6d0 0x7fa97c08cac0 secure :-1 s=READY pgs=1648 cs=0 l=1 rx=0x7fa96c00f6c0 tx=0x7fa96c00fa60).ready entity=mon.0 client_cookie=49894028fd82d8e1 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:04.558 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.551+0000 7fa981da8700 1 --2- 172.21.15.16:0/2938217994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa97c08c6d0 0x7fa97c08cac0 secure :-1 s=THROTTLE_DONE pgs=1648 cs=0 l=1 rx=0x7fa96c00f6c0 tx=0x7fa96c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.551+0000 7fa981da8700 1 --2- 172.21.15.16:0/2938217994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa97c08c6d0 0x7fa97c08cac0 secure :-1 s=THROTTLE_DONE pgs=1648 cs=0 l=1 rx=0x7fa96c00f6c0 tx=0x7fa96c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.551+0000 7fa981da8700 1 --2- 172.21.15.16:0/2938217994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa97c08c6d0 0x7fa97c08cac0 secure :-1 s=THROTTLE_DONE pgs=1648 cs=0 l=1 rx=0x7fa96c00f6c0 tx=0x7fa96c00fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.551+0000 7fa980da6700 1 -- 172.21.15.16:0/2938217994 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96c000960 con 0x7fa97c08c6d0 2019-12-05T06:57:04.559 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.551+0000 7fa980da6700 1 -- 172.21.15.16:0/2938217994 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c000c90 con 0x7fa97c08c6d0 2019-12-05T06:57:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.551+0000 7fa980da6700 1 -- 172.21.15.16:0/2938217994 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96c0013a0 con 0x7fa97c08c6d0 2019-12-05T06:57:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.551+0000 7fa98380b700 1 -- 172.21.15.16:0/2938217994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa97c08c6d0 msgr2=0x7fa97c08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.551+0000 7fa98380b700 1 --2- 172.21.15.16:0/2938217994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa97c08c6d0 0x7fa97c08cac0 secure :-1 s=READY pgs=1648 cs=0 l=1 rx=0x7fa96c00f6c0 tx=0x7fa96c00fa60).stop 2019-12-05T06:57:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.552+0000 7fa98380b700 1 -- 172.21.15.16:0/2938217994 shutdown_connections 2019-12-05T06:57:04.560 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.552+0000 7fa98380b700 1 --2- 172.21.15.16:0/2938217994 >> v2:172.21.15.76:3300/0 conn(0x7fa97c08f720 0x7fa97c08d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.552+0000 7fa98380b700 1 --2- 172.21.15.16:0/2938217994 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fa97c08c6d0 0x7fa97c08cac0 unknown :-1 s=CLOSED pgs=1648 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:04.561 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.552+0000 7fa98380b700 1 -- 172.21.15.16:0/2938217994 shutdown_connections 2019-12-05T06:57:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.552+0000 7fa98380b700 1 -- 172.21.15.16:0/2938217994 wait complete. 2019-12-05T06:57:04.564 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.552+0000 7fa98380b700 1 -- 172.21.15.16:0/2938217994 >> 172.21.15.16:0/2938217994 conn(0x7fa97c061940 msgr2=0x7fa97c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.556+0000 7fa98380b700 1 Processor -- start 2019-12-05T06:57:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.556+0000 7fa98380b700 1 -- start start 2019-12-05T06:57:04.568 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.557+0000 7fa98380b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fa97c0855a0 0x7fa97c084dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.557+0000 7fa98380b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa97c151ac0 0x7fa97c151eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.557+0000 7fa98380b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa97c087c40 con 0x7fa97c0a09a0 2019-12-05T06:57:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.557+0000 7fa98380b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa97c089900 con 0x7fa97c0855a0 2019-12-05T06:57:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.557+0000 7fa98380b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa97c088ed0 con 0x7fa97c151ac0 2019-12-05T06:57:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.557+0000 7fa981da8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fa97c0a09a0 0x7fa97c0845d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:49972/0 (socket says 172.21.15.16:49972) 2019-12-05T06:57:04.569 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.557+0000 7fa981da8700 1 -- 172.21.15.16:0/3798465307 learned_addr learned my addr 172.21.15.16:0/3798465307 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.557+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> v2:172.21.15.76:3300/0 conn(0x7fa97c0855a0 0x7fa97c084dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.558+0000 7fa9825a9700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa97c151ac0 0x7fa97c151eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.558+0000 7fa97a7fc700 1 -- 172.21.15.16:0/3798465307 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 427189378 0 0) 0x7fa96c0075d0 con 0x7fa97c0a09a0 2019-12-05T06:57:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.558+0000 7fa97a7fc700 1 -- 172.21.15.16:0/3798465307 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa95c002e80 con 0x7fa97c0a09a0 2019-12-05T06:57:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.558+0000 7fa9815a7700 1 -- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa97c151ac0 msgr2=0x7fa97c151eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.558+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa97c151ac0 0x7fa97c151eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:04.570 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.558+0000 7fa9815a7700 1 -- 172.21.15.16:0/3798465307 >> v1:172.21.15.16:6789/0 conn(0x7fa97c0a09a0 legacy=0x7fa97c0845d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.558+0000 7fa9815a7700 1 -- 172.21.15.16:0/3798465307 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa97c089f90 con 0x7fa97c0855a0 2019-12-05T06:57:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.558+0000 7fa9825a9700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa97c151ac0 0x7fa97c151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:57:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.558+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> v2:172.21.15.76:3300/0 conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.571 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.559+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> v2:172.21.15.76:3300/0 conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.559+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=READY pgs=1113 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.559+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=THROTTLE_DONE pgs=1113 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.559+0000 7fa97a7fc700 1 -- 172.21.15.16:0/3798465307 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa970009b70 con 0x7fa97c0855a0 2019-12-05T06:57:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.559+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=THROTTLE_DONE pgs=1113 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.572 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.559+0000 7fa98380b700 1 -- 172.21.15.16:0/3798465307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa97c089f90 con 0x7fa97c0855a0 2019-12-05T06:57:04.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.559+0000 7fa97a7fc700 1 -- 172.21.15.16:0/3798465307 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa970009790 con 0x7fa97c0855a0 2019-12-05T06:57:04.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.559+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=THROTTLE_DONE pgs=1113 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.573 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.559+0000 7fa97a7fc700 1 -- 172.21.15.16:0/3798465307 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa970009980 con 0x7fa97c0855a0 2019-12-05T06:57:04.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.559+0000 7fa98380b700 1 -- 172.21.15.16:0/3798465307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97c089f90 con 0x7fa97c0855a0 2019-12-05T06:57:04.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.560+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=THROTTLE_DONE pgs=1113 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.561+0000 7fa97a7fc700 1 -- 172.21.15.16:0/3798465307 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fa970009a20 con 0x7fa97c0855a0 2019-12-05T06:57:04.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.561+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=THROTTLE_DONE pgs=1113 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.574 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.561+0000 7fa97a7fc700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa95c023670 0x7fa95c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:04.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.561+0000 7fa97a7fc700 1 -- 172.21.15.16:0/3798465307 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7fa970045d70 con 0x7fa97c0855a0 2019-12-05T06:57:04.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.562+0000 7fa981da8700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa95c023670 0x7fa95c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:04.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.562+0000 7fa981da8700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa95c023670 0x7fa95c025ae0 crc :-1 s=READY pgs=687 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:04.575 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.564+0000 7fa98380b700 1 -- 172.21.15.16:0/3798465307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97c089f90 con 0x7fa97c0855a0 2019-12-05T06:57:04.576 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.573+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=THROTTLE_DONE pgs=1113 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.577 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.573+0000 7fa97a7fc700 1 -- 172.21.15.16:0/3798465307 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fa970009580 con 0x7fa97c0855a0 2019-12-05T06:57:04.804 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.804+0000 7fa98380b700 1 -- 172.21.15.16:0/3798465307 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa97c089f90 con 0x7fa97c0855a0 2019-12-05T06:57:04.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.805+0000 7fa9815a7700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=THROTTLE_DONE pgs=1113 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:04.806 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.805+0000 7fa97a7fc700 1 -- 172.21.15.16:0/3798465307 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fa970009930 con 0x7fa97c0855a0 2019-12-05T06:57:04.806 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 202 2019-12-05T06:57:04.806 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:04.815 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:53.063332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 -- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa95c023670 msgr2=0x7fa95c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa95c023670 0x7fa95c025ae0 crc :-1 s=READY pgs=687 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 -- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 msgr2=0x7fa97c084dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:04.816 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 0x7fa97c084dd0 secure :-1 s=READY pgs=1113 cs=0 l=1 rx=0x7fa9700103a0 tx=0x7fa970010740).stop 2019-12-05T06:57:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 -- 172.21.15.16:0/3798465307 shutdown_connections 2019-12-05T06:57:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fa95c023670 0x7fa95c025ae0 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:04.817 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fa97c0855a0 0x7fa97c084dd0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 --2- 172.21.15.16:0/3798465307 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fa97c151ac0 0x7fa97c151eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 -- 172.21.15.16:0/3798465307 shutdown_connections 2019-12-05T06:57:04.818 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 -- 172.21.15.16:0/3798465307 wait complete. 2019-12-05T06:57:04.819 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:04.808+0000 7fa98380b700 1 -- 172.21.15.16:0/3798465307 >> 172.21.15.16:0/3798465307 conn(0x7fa97c0710b0 msgr2=0x7fa97c12d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:04.823 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:57:04.823 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 34 secs before thrashing 2019-12-05T06:57:17.516 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:57:17.521 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:57:38.824 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:39.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.004+0000 7ff62868c700 1 Processor -- start 2019-12-05T06:57:39.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.004+0000 7ff62868c700 1 -- start start 2019-12-05T06:57:39.010 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.004+0000 7ff62868c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff62008d2c0 0x7ff62008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:39.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff62868c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6200958e0 con 0x7ff62008fef0 2019-12-05T06:57:39.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff62868c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6200629e0 con 0x7ff62008c6d0 2019-12-05T06:57:39.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff62868c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62008df90 con 0x7ff62008d2c0 2019-12-05T06:57:39.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff62742a700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7ff62008fef0 0x7ff62008f460 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50138/0 (socket says 172.21.15.16:50138) 2019-12-05T06:57:39.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff62742a700 1 -- 172.21.15.16:0/629892191 learned_addr learned my addr 172.21.15.16:0/629892191 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:39.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff626428700 1 --2- 172.21.15.16:0/629892191 >> v2:172.21.15.76:3300/0 conn(0x7ff62008d2c0 0x7ff62008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.011 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3232647979 0 0) 0x7ff618001150 con 0x7ff62008fef0 2019-12-05T06:57:39.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff60c002e80 con 0x7ff62008fef0 2019-12-05T06:57:39.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 558862523 0 0) 0x7ff618001130 con 0x7ff62008fef0 2019-12-05T06:57:39.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 >> v2:172.21.15.76:3300/0 conn(0x7ff62008d2c0 msgr2=0x7ff62008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:39.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 --2- 172.21.15.16:0/629892191 >> v2:172.21.15.76:3300/0 conn(0x7ff62008d2c0 0x7ff62008ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:39.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 >> v1:172.21.15.76:6789/0 conn(0x7ff62008c6d0 legacy=0x7ff62008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.012 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62008e620 con 0x7ff62008fef0 2019-12-05T06:57:39.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7ff618001660 con 0x7ff62008fef0 2019-12-05T06:57:39.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 >> v1:172.21.15.16:6789/0 conn(0x7ff62008fef0 legacy=0x7ff62008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 0x7ff60c005f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:39.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff60c006670 0x7ff60c008b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:39.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff60c0093a0 0x7ff60c00b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:39.013 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c00c020 con 0x7ff60c003ab0 2019-12-05T06:57:39.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff626c29700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 0x7ff60c005f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.005+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c050220 con 0x7ff60c006670 2019-12-05T06:57:39.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c050450 con 0x7ff60c0093a0 2019-12-05T06:57:39.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626428700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff60c006670 0x7ff60c008b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.014 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff62742a700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff60c0093a0 0x7ff60c00b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 -- 172.21.15.16:0/629892191 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff60c0093a0 msgr2=0x7ff60c00b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:39.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff60c0093a0 0x7ff60c00b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:39.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 -- 172.21.15.16:0/629892191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff60c006670 msgr2=0x7ff60c008b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:39.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff60c006670 0x7ff60c008b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:39.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 -- 172.21.15.16:0/629892191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c050700 con 0x7ff60c003ab0 2019-12-05T06:57:39.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff62742a700 1 -- 172.21.15.16:0/629892191 reap_dead start 2019-12-05T06:57:39.015 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626428700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff60c006670 0x7ff60c008b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:57:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 0x7ff60c005f10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff61c00b2d0 tx=0x7ff61c00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 0x7ff60c005f10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff61c00b2d0 tx=0x7ff61c00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 0x7ff60c005f10 secure :-1 s=READY pgs=1651 cs=0 l=1 rx=0x7ff61c00b2d0 tx=0x7ff61c00b5d0).ready entity=mon.0 client_cookie=4a7f5adc34502a9b server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 0x7ff60c005f10 secure :-1 s=THROTTLE_DONE pgs=1651 cs=0 l=1 rx=0x7ff61c00b2d0 tx=0x7ff61c00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff61c000ce0 con 0x7ff60c003ab0 2019-12-05T06:57:39.016 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 0x7ff60c005f10 secure :-1 s=THROTTLE_DONE pgs=1651 cs=0 l=1 rx=0x7ff61c00b2d0 tx=0x7ff61c00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff626c29700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 0x7ff60c005f10 secure :-1 s=THROTTLE_DONE pgs=1651 cs=0 l=1 rx=0x7ff61c00b2d0 tx=0x7ff61c00b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c009d00 con 0x7ff60c003ab0 2019-12-05T06:57:39.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff625c27700 1 -- 172.21.15.16:0/629892191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff61c009ef0 con 0x7ff60c003ab0 2019-12-05T06:57:39.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff62868c700 1 -- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 msgr2=0x7ff60c005f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.006+0000 7ff62868c700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 0x7ff60c005f10 secure :-1 s=READY pgs=1651 cs=0 l=1 rx=0x7ff61c00b2d0 tx=0x7ff61c00b5d0).stop 2019-12-05T06:57:39.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.007+0000 7ff62868c700 1 -- 172.21.15.16:0/629892191 shutdown_connections 2019-12-05T06:57:39.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.007+0000 7ff62868c700 1 --2- 172.21.15.16:0/629892191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff60c003ab0 0x7ff60c005f10 unknown :-1 s=CLOSED pgs=1651 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:39.017 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.007+0000 7ff62868c700 1 -- 172.21.15.16:0/629892191 shutdown_connections 2019-12-05T06:57:39.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.007+0000 7ff62868c700 1 -- 172.21.15.16:0/629892191 wait complete. 2019-12-05T06:57:39.018 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.007+0000 7ff62868c700 1 -- 172.21.15.16:0/629892191 >> 172.21.15.16:0/629892191 conn(0x7ff620061940 msgr2=0x7ff620096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:39.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.009+0000 7ff62868c700 1 Processor -- start 2019-12-05T06:57:39.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.009+0000 7ff62868c700 1 -- start start 2019-12-05T06:57:39.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.010+0000 7ff62868c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff6201325e0 0x7ff62012b770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:39.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.010+0000 7ff62868c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff620154cb0 0x7ff6201570b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:39.019 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.010+0000 7ff62868c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff620128b20 con 0x7ff62012c740 2019-12-05T06:57:39.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.010+0000 7ff62868c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff62012a7e0 con 0x7ff6201325e0 2019-12-05T06:57:39.020 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.010+0000 7ff62868c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff620129db0 con 0x7ff620154cb0 2019-12-05T06:57:39.023 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.010+0000 7ff62742a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff620154cb0 0x7ff6201570b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.010+0000 7ff626c29700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff6201325e0 0x7ff62012b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.011+0000 7ff626c29700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff6201325e0 0x7ff62012b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:51068/0 (socket says 172.21.15.16:51068) 2019-12-05T06:57:39.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.011+0000 7ff626c29700 1 -- 172.21.15.16:0/2947715915 learned_addr learned my addr 172.21.15.16:0/2947715915 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:57:39.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.011+0000 7ff626c29700 1 -- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff620154cb0 msgr2=0x7ff6201570b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.024 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.011+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff620154cb0 0x7ff6201570b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.011+0000 7ff626c29700 1 -- 172.21.15.16:0/2947715915 >> v1:172.21.15.76:6789/0 conn(0x7ff62012c740 legacy=0x7ff62012bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.011+0000 7ff626c29700 1 -- 172.21.15.16:0/2947715915 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62014a820 con 0x7ff6201325e0 2019-12-05T06:57:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.011+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> v2:172.21.15.76:3300/0 conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.011+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> v2:172.21.15.76:3300/0 conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.011+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=READY pgs=1116 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.012+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=THROTTLE_DONE pgs=1116 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.025 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.012+0000 7ff6177fe700 1 -- 172.21.15.16:0/2947715915 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff61c00ad60 con 0x7ff6201325e0 2019-12-05T06:57:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.012+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=THROTTLE_DONE pgs=1116 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.012+0000 7ff6177fe700 1 -- 172.21.15.16:0/2947715915 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff61c00ad60 con 0x7ff6201325e0 2019-12-05T06:57:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.012+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=THROTTLE_DONE pgs=1116 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.012+0000 7ff62868c700 1 -- 172.21.15.16:0/2947715915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff62012ae70 con 0x7ff6201325e0 2019-12-05T06:57:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.012+0000 7ff6177fe700 1 -- 172.21.15.16:0/2947715915 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff61c000d60 con 0x7ff6201325e0 2019-12-05T06:57:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.012+0000 7ff62868c700 1 -- 172.21.15.16:0/2947715915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff62014a820 con 0x7ff6201325e0 2019-12-05T06:57:39.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.013+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=THROTTLE_DONE pgs=1116 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.013+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=THROTTLE_DONE pgs=1116 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.013+0000 7ff6177fe700 1 -- 172.21.15.16:0/2947715915 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff61c009900 con 0x7ff6201325e0 2019-12-05T06:57:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.013+0000 7ff6177fe700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff608023630 0x7ff608025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.013+0000 7ff6177fe700 1 -- 172.21.15.16:0/2947715915 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7ff61c046160 con 0x7ff6201325e0 2019-12-05T06:57:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.013+0000 7ff626428700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff608023630 0x7ff608025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.014+0000 7ff626428700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff608023630 0x7ff608025ac0 crc :-1 s=READY pgs=690 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:39.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.019+0000 7ff62868c700 1 -- 172.21.15.16:0/2947715915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff62012ae70 con 0x7ff6201325e0 2019-12-05T06:57:39.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.028+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=THROTTLE_DONE pgs=1116 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.029 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.028+0000 7ff6177fe700 1 -- 172.21.15.16:0/2947715915 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff61c000d60 con 0x7ff6201325e0 2019-12-05T06:57:39.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.238+0000 7ff62868c700 1 -- 172.21.15.16:0/2947715915 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff62014a820 con 0x7ff6201325e0 2019-12-05T06:57:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.239+0000 7ff626c29700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=THROTTLE_DONE pgs=1116 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.239+0000 7ff6177fe700 1 -- 172.21.15.16:0/2947715915 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+2109 (secure 0 0 0) 0x7ff61c009900 con 0x7ff6201325e0 2019-12-05T06:57:39.240 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 202 2019-12-05T06:57:39.241 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:39.241 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:53.063332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:39.244 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 -- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff608023630 msgr2=0x7ff608025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff608023630 0x7ff608025ac0 crc :-1 s=READY pgs=690 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:39.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 -- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 msgr2=0x7ff62012b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 0x7ff62012b770 secure :-1 s=READY pgs=1116 cs=0 l=1 rx=0x7ff61c0107c0 tx=0x7ff61c010b60).stop 2019-12-05T06:57:39.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 -- 172.21.15.16:0/2947715915 shutdown_connections 2019-12-05T06:57:39.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff608023630 0x7ff608025ac0 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:39.245 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7ff620154cb0 0x7ff6201570b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 --2- 172.21.15.16:0/2947715915 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff6201325e0 0x7ff62012b770 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 -- 172.21.15.16:0/2947715915 shutdown_connections 2019-12-05T06:57:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 -- 172.21.15.16:0/2947715915 wait complete. 2019-12-05T06:57:39.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.243+0000 7ff62868c700 1 -- 172.21.15.16:0/2947715915 >> 172.21.15.16:0/2947715915 conn(0x7ff6200a2850 msgr2=0x7ff62006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:39.259 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.a (rank=0) 2019-12-05T06:57:39.259 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2019-12-05T06:57:39.259 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:57:39.268 INFO:tasks.ceph.mds.a:Stopped 2019-12-05T06:57:39.269 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:39.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.599+0000 7f31be34c700 1 Processor -- start 2019-12-05T06:57:39.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.599+0000 7f31be34c700 1 -- start start 2019-12-05T06:57:39.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.601+0000 7f31be34c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31b808d2c0 0x7f31b808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:39.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.601+0000 7f31be34c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31b808fef0 0x7f31b808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:39.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.601+0000 7f31be34c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b80958e0 con 0x7f31b808c6d0 2019-12-05T06:57:39.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.601+0000 7f31be34c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b808df90 con 0x7f31b808d2c0 2019-12-05T06:57:39.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.601+0000 7f31be34c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b80a3f70 con 0x7f31b808fef0 2019-12-05T06:57:39.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.601+0000 7f31b7fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31b808d2c0 0x7f31b808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.601+0000 7f31bc8e9700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f31b808c6d0 0x7f31b808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:43232/0 (socket says 172.21.15.16:43232) 2019-12-05T06:57:39.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.601+0000 7f31bc8e9700 1 -- 172.21.15.16:0/1333224487 learned_addr learned my addr 172.21.15.16:0/1333224487 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:39.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.601+0000 7f31bd0ea700 1 --2- 172.21.15.16:0/1333224487 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31b808fef0 0x7f31b808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.602+0000 7f31b77fe700 1 -- 172.21.15.16:0/1333224487 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214179742 0 0) 0x7f31b0001480 con 0x7f31b808c6d0 2019-12-05T06:57:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.602+0000 7f31b77fe700 1 -- 172.21.15.16:0/1333224487 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31a4002e80 con 0x7f31b808c6d0 2019-12-05T06:57:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.602+0000 7f31b7fff700 1 -- 172.21.15.16:0/1333224487 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31b808fef0 msgr2=0x7f31b808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.602+0000 7f31b7fff700 1 --2- 172.21.15.16:0/1333224487 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31b808fef0 0x7f31b808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.602+0000 7f31b7fff700 1 -- 172.21.15.16:0/1333224487 >> v1:172.21.15.76:6789/0 conn(0x7f31b808c6d0 legacy=0x7f31b808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.602+0000 7f31b7fff700 1 -- 172.21.15.16:0/1333224487 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b808e620 con 0x7f31b808d2c0 2019-12-05T06:57:39.612 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.602+0000 7f31bd0ea700 1 --2- 172.21.15.16:0/1333224487 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31b808fef0 0x7f31b808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:57:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.602+0000 7f31b7fff700 1 --2- 172.21.15.16:0/1333224487 >> v2:172.21.15.16:3300/0 conn(0x7f31b808d2c0 0x7f31b808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f31a800af50 tx=0x7f31a800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.602+0000 7f31b7fff700 1 --2- 172.21.15.16:0/1333224487 >> v2:172.21.15.16:3300/0 conn(0x7f31b808d2c0 0x7f31b808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f31a800af50 tx=0x7f31a800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.602+0000 7f31b7fff700 1 --2- 172.21.15.16:0/1333224487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31b808d2c0 0x7f31b808ef20 secure :-1 s=READY pgs=1652 cs=0 l=1 rx=0x7f31a800af50 tx=0x7f31a800b760).ready entity=mon.0 client_cookie=2809c14b10bbed0c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.603+0000 7f31b7fff700 1 --2- 172.21.15.16:0/1333224487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31b808d2c0 0x7f31b808ef20 secure :-1 s=THROTTLE_DONE pgs=1652 cs=0 l=1 rx=0x7f31a800af50 tx=0x7f31a800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.603+0000 7f31b7fff700 1 --2- 172.21.15.16:0/1333224487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31b808d2c0 0x7f31b808ef20 secure :-1 s=THROTTLE_DONE pgs=1652 cs=0 l=1 rx=0x7f31a800af50 tx=0x7f31a800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.613 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.603+0000 7f31b77fe700 1 -- 172.21.15.16:0/1333224487 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31a8007210 con 0x7f31b808d2c0 2019-12-05T06:57:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.603+0000 7f31b7fff700 1 --2- 172.21.15.16:0/1333224487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31b808d2c0 0x7f31b808ef20 secure :-1 s=THROTTLE_DONE pgs=1652 cs=0 l=1 rx=0x7f31a800af50 tx=0x7f31a800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.603+0000 7f31b77fe700 1 -- 172.21.15.16:0/1333224487 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31a80099b0 con 0x7f31b808d2c0 2019-12-05T06:57:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.603+0000 7f31b77fe700 1 -- 172.21.15.16:0/1333224487 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31a800a080 con 0x7f31b808d2c0 2019-12-05T06:57:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.603+0000 7f31be34c700 1 -- 172.21.15.16:0/1333224487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31b808d2c0 msgr2=0x7f31b808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.603+0000 7f31be34c700 1 --2- 172.21.15.16:0/1333224487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31b808d2c0 0x7f31b808ef20 secure :-1 s=READY pgs=1652 cs=0 l=1 rx=0x7f31a800af50 tx=0x7f31a800b760).stop 2019-12-05T06:57:39.614 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.605+0000 7f31be34c700 1 -- 172.21.15.16:0/1333224487 shutdown_connections 2019-12-05T06:57:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.605+0000 7f31be34c700 1 --2- 172.21.15.16:0/1333224487 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31b808fef0 0x7f31b808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.605+0000 7f31be34c700 1 --2- 172.21.15.16:0/1333224487 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31b808d2c0 0x7f31b808ef20 unknown :-1 s=CLOSED pgs=1652 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.605+0000 7f31be34c700 1 -- 172.21.15.16:0/1333224487 shutdown_connections 2019-12-05T06:57:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.605+0000 7f31be34c700 1 -- 172.21.15.16:0/1333224487 wait complete. 2019-12-05T06:57:39.615 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.605+0000 7f31be34c700 1 -- 172.21.15.16:0/1333224487 >> 172.21.15.16:0/1333224487 conn(0x7f31b8061940 msgr2=0x7f31b8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:39.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.610+0000 7f31be34c700 1 Processor -- start 2019-12-05T06:57:39.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.610+0000 7f31be34c700 1 -- start start 2019-12-05T06:57:39.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.612+0000 7f31be34c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f31b806df10 0x7f31b8098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:39.619 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.612+0000 7f31be34c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b8089fb0 con 0x7f31b8156cb0 2019-12-05T06:57:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.612+0000 7f31be34c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b806e300 con 0x7f31b80992d0 2019-12-05T06:57:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.612+0000 7f31be34c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b8097370 con 0x7f31b806df10 2019-12-05T06:57:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.613+0000 7f31bd0ea700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f31b8156cb0 0x7f31b81590b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50172/0 (socket says 172.21.15.16:50172) 2019-12-05T06:57:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.613+0000 7f31bd0ea700 1 -- 172.21.15.16:0/2557534495 learned_addr learned my addr 172.21.15.16:0/2557534495 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.613+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> v2:172.21.15.76:3300/0 conn(0x7f31b806df10 0x7f31b8098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.613+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3043933445 0 0) 0x7f31a0009890 con 0x7f31b8156cb0 2019-12-05T06:57:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.613+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f319c002e80 con 0x7f31b8156cb0 2019-12-05T06:57:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.613+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186439153 0 0) 0x7f31a8009580 con 0x7f31b80992d0 2019-12-05T06:57:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.613+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f319c004170 con 0x7f31b80992d0 2019-12-05T06:57:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.614+0000 7f31bc8e9700 1 -- 172.21.15.16:0/2557534495 >> v1:172.21.15.76:6789/0 conn(0x7f31b80992d0 legacy=0x7f31b8098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.614+0000 7f31bc8e9700 1 -- 172.21.15.16:0/2557534495 >> v1:172.21.15.16:6789/0 conn(0x7f31b8156cb0 legacy=0x7f31b81590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:39.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.614+0000 7f31bc8e9700 1 -- 172.21.15.16:0/2557534495 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b8097a00 con 0x7f31b806df10 2019-12-05T06:57:39.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.614+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> v2:172.21.15.76:3300/0 conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.614+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> v2:172.21.15.76:3300/0 conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.614+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=READY pgs=1119 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:39.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.614+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=THROTTLE_DONE pgs=1119 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.615+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31b0000f90 con 0x7f31b806df10 2019-12-05T06:57:39.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.615+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=THROTTLE_DONE pgs=1119 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.615+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=THROTTLE_DONE pgs=1119 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.615+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b0004730 con 0x7f31b806df10 2019-12-05T06:57:39.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.615+0000 7f31be34c700 1 -- 172.21.15.16:0/2557534495 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b8097a00 con 0x7f31b806df10 2019-12-05T06:57:39.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.615+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31b0004e00 con 0x7f31b806df10 2019-12-05T06:57:39.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.615+0000 7f31be34c700 1 -- 172.21.15.16:0/2557534495 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b8097a00 con 0x7f31b806df10 2019-12-05T06:57:39.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.616+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=THROTTLE_DONE pgs=1119 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.617+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=THROTTLE_DONE pgs=1119 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.617+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f31b0004920 con 0x7f31b806df10 2019-12-05T06:57:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.617+0000 7f31b57fa700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f319c023650 0x7f319c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.617+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f31b0046160 con 0x7f31b806df10 2019-12-05T06:57:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.618+0000 7f31b7fff700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f319c023650 0x7f319c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:39.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.618+0000 7f31b7fff700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f319c023650 0x7f319c025ae0 crc :-1 s=READY pgs=691 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:39.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.630+0000 7f31be34c700 1 -- 172.21.15.16:0/2557534495 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b8097a00 con 0x7f31b806df10 2019-12-05T06:57:39.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.640+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=THROTTLE_DONE pgs=1119 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:39.641 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.640+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f31b00015b0 con 0x7f31b806df10 2019-12-05T06:57:39.912 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.910+0000 7f31be34c700 1 -- 172.21.15.16:0/2557534495 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31b8097a00 con 0x7f31b806df10 2019-12-05T06:57:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.912+0000 7f31bc8e9700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=THROTTLE_DONE pgs=1119 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.912+0000 7f31b57fa700 1 -- 172.21.15.16:0/2557534495 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f31b0004920 con 0x7f31b806df10 2019-12-05T06:57:40.070 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 202 2019-12-05T06:57:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.918+0000 7f31be34c700 1 -- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f319c023650 msgr2=0x7f319c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.918+0000 7f31be34c700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f319c023650 0x7f319c025ae0 crc :-1 s=READY pgs=691 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.918+0000 7f31be34c700 1 -- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 msgr2=0x7f31b8098300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.918+0000 7f31be34c700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 0x7f31b8098300 secure :-1 s=READY pgs=1119 cs=0 l=1 rx=0x7f31b00106f0 tx=0x7f31b0010a90).stop 2019-12-05T06:57:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.918+0000 7f31be34c700 1 -- 172.21.15.16:0/2557534495 shutdown_connections 2019-12-05T06:57:40.071 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.918+0000 7f31be34c700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f319c023650 0x7f319c025ae0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.918+0000 7f31be34c700 1 --2- 172.21.15.16:0/2557534495 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f31b806df10 0x7f31b8098300 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.918+0000 7f31be34c700 1 -- 172.21.15.16:0/2557534495 shutdown_connections 2019-12-05T06:57:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.918+0000 7f31be34c700 1 -- 172.21.15.16:0/2557534495 wait complete. 2019-12-05T06:57:40.072 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:39.918+0000 7f31be34c700 1 -- 172.21.15.16:0/2557534495 >> 172.21.15.16:0/2557534495 conn(0x7f31b80714f0 msgr2=0x7f31b809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:40.072 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:40.073 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:53.063332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:40.073 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:57:42.074 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:42.332 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.322+0000 7fc52e3b5700 1 Processor -- start 2019-12-05T06:57:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.322+0000 7fc52e3b5700 1 -- start start 2019-12-05T06:57:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.324+0000 7fc52e3b5700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc52808fef0 0x7fc52808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.324+0000 7fc52e3b5700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5280958e0 con 0x7fc52808d2c0 2019-12-05T06:57:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.324+0000 7fc52e3b5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5280629e0 con 0x7fc52808c6d0 2019-12-05T06:57:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.324+0000 7fc52e3b5700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52808df90 con 0x7fc52808fef0 2019-12-05T06:57:42.333 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.324+0000 7fc527fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fc52808d2c0 0x7fc52808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50188/0 (socket says 172.21.15.16:50188) 2019-12-05T06:57:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.324+0000 7fc52d153700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc52808fef0 0x7fc52808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.324+0000 7fc527fff700 1 -- 172.21.15.16:0/197571046 learned_addr learned my addr 172.21.15.16:0/197571046 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.325+0000 7fc52d153700 1 -- 172.21.15.16:0/197571046 >> v1:172.21.15.76:6789/0 conn(0x7fc52808c6d0 legacy=0x7fc52808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.325+0000 7fc52d153700 1 -- 172.21.15.16:0/197571046 >> v1:172.21.15.16:6789/0 conn(0x7fc52808d2c0 legacy=0x7fc52808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.325+0000 7fc5277fe700 1 -- 172.21.15.16:0/197571046 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2642981299 0 0) 0x7fc510001080 con 0x7fc52808c6d0 2019-12-05T06:57:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.325+0000 7fc52d153700 1 -- 172.21.15.16:0/197571046 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52808e620 con 0x7fc52808fef0 2019-12-05T06:57:42.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.325+0000 7fc52d153700 1 --2- 172.21.15.16:0/197571046 >> v2:172.21.15.16:3300/0 conn(0x7fc52808fef0 0x7fc52808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc52000b2d0 tx=0x7fc52000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.326+0000 7fc52d153700 1 --2- 172.21.15.16:0/197571046 >> v2:172.21.15.16:3300/0 conn(0x7fc52808fef0 0x7fc52808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc52000b2d0 tx=0x7fc52000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.326+0000 7fc52d153700 1 --2- 172.21.15.16:0/197571046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52808fef0 0x7fc52808f720 secure :-1 s=READY pgs=1655 cs=0 l=1 rx=0x7fc52000b2d0 tx=0x7fc52000bae0).ready entity=mon.0 client_cookie=478eeee8d39358b4 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.326+0000 7fc52d153700 1 --2- 172.21.15.16:0/197571046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52808fef0 0x7fc52808f720 secure :-1 s=THROTTLE_DONE pgs=1655 cs=0 l=1 rx=0x7fc52000b2d0 tx=0x7fc52000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.326+0000 7fc5277fe700 1 -- 172.21.15.16:0/197571046 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc520009f20 con 0x7fc52808fef0 2019-12-05T06:57:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.326+0000 7fc52d153700 1 --2- 172.21.15.16:0/197571046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52808fef0 0x7fc52808f720 secure :-1 s=THROTTLE_DONE pgs=1655 cs=0 l=1 rx=0x7fc52000b2d0 tx=0x7fc52000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.326+0000 7fc52d153700 1 --2- 172.21.15.16:0/197571046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52808fef0 0x7fc52808f720 secure :-1 s=THROTTLE_DONE pgs=1655 cs=0 l=1 rx=0x7fc52000b2d0 tx=0x7fc52000bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.326+0000 7fc5277fe700 1 -- 172.21.15.16:0/197571046 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc520000df0 con 0x7fc52808fef0 2019-12-05T06:57:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.326+0000 7fc5277fe700 1 -- 172.21.15.16:0/197571046 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc52000a310 con 0x7fc52808fef0 2019-12-05T06:57:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.326+0000 7fc52e3b5700 1 -- 172.21.15.16:0/197571046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52808fef0 msgr2=0x7fc52808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.326+0000 7fc52e3b5700 1 --2- 172.21.15.16:0/197571046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52808fef0 0x7fc52808f720 secure :-1 s=READY pgs=1655 cs=0 l=1 rx=0x7fc52000b2d0 tx=0x7fc52000bae0).stop 2019-12-05T06:57:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.328+0000 7fc52e3b5700 1 -- 172.21.15.16:0/197571046 shutdown_connections 2019-12-05T06:57:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.328+0000 7fc52e3b5700 1 --2- 172.21.15.16:0/197571046 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc52808fef0 0x7fc52808f720 unknown :-1 s=CLOSED pgs=1655 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.328+0000 7fc52e3b5700 1 -- 172.21.15.16:0/197571046 shutdown_connections 2019-12-05T06:57:42.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.328+0000 7fc52e3b5700 1 -- 172.21.15.16:0/197571046 wait complete. 2019-12-05T06:57:42.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.328+0000 7fc52e3b5700 1 -- 172.21.15.16:0/197571046 >> 172.21.15.16:0/197571046 conn(0x7fc528061940 msgr2=0x7fc528096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.333+0000 7fc52e3b5700 1 Processor -- start 2019-12-05T06:57:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.333+0000 7fc52e3b5700 1 -- start start 2019-12-05T06:57:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.335+0000 7fc52e3b5700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc52809e440 0x7fc528087b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.335+0000 7fc52e3b5700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc528088b50 0x7fc5280880c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.335+0000 7fc52e3b5700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc528084ed0 con 0x7fc528088600 2019-12-05T06:57:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.335+0000 7fc52e3b5700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc52814a7b0 con 0x7fc528088b50 2019-12-05T06:57:42.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.335+0000 7fc52e3b5700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5280861c0 con 0x7fc52809e440 2019-12-05T06:57:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.335+0000 7fc527fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc528088b50 0x7fc5280880c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:42.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.335+0000 7fc52c952700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc52809e440 0x7fc528087b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.335+0000 7fc527fff700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fc528088b50 0x7fc5280880c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3300/0 says I am v2:172.21.15.16:51116/0 (socket says 172.21.15.16:51116) 2019-12-05T06:57:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.335+0000 7fc527fff700 1 -- 172.21.15.16:0/2868195907 learned_addr learned my addr 172.21.15.16:0/2868195907 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:57:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.336+0000 7fc527fff700 1 -- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc52809e440 msgr2=0x7fc528087b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.336+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc52809e440 0x7fc528087b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.336+0000 7fc527fff700 1 -- 172.21.15.16:0/2868195907 >> v1:172.21.15.76:6789/0 conn(0x7fc528088600 legacy=0x7fc52806dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.336+0000 7fc527fff700 1 -- 172.21.15.16:0/2868195907 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc528087280 con 0x7fc528088b50 2019-12-05T06:57:42.342 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.336+0000 7fc52c952700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc52809e440 0x7fc528087b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:57:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.336+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> v2:172.21.15.76:3300/0 conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.336+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> v2:172.21.15.76:3300/0 conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.336+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.337+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.337+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.337+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.343 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.337+0000 7fc5257fa700 1 -- 172.21.15.16:0/2868195907 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc518000ed0 con 0x7fc528088b50 2019-12-05T06:57:42.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.337+0000 7fc5257fa700 1 -- 172.21.15.16:0/2868195907 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc518004430 con 0x7fc528088b50 2019-12-05T06:57:42.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.337+0000 7fc52e3b5700 1 -- 172.21.15.16:0/2868195907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc528087280 con 0x7fc528088b50 2019-12-05T06:57:42.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.337+0000 7fc5257fa700 1 -- 172.21.15.16:0/2868195907 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc518004a80 con 0x7fc528088b50 2019-12-05T06:57:42.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.337+0000 7fc52e3b5700 1 -- 172.21.15.16:0/2868195907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc528087280 con 0x7fc528088b50 2019-12-05T06:57:42.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.338+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.339+0000 7fc5257fa700 1 -- 172.21.15.16:0/2868195907 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc518004620 con 0x7fc528088b50 2019-12-05T06:57:42.344 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.339+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.340+0000 7fc5257fa700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc50c023630 0x7fc50c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:42.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.340+0000 7fc5257fa700 1 -- 172.21.15.16:0/2868195907 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7fc518045e10 con 0x7fc528088b50 2019-12-05T06:57:42.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.340+0000 7fc52c952700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc50c023630 0x7fc50c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:42.345 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.341+0000 7fc52c952700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc50c023630 0x7fc50c025ac0 crc :-1 s=READY pgs=692 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:42.354 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.353+0000 7fc52e3b5700 1 -- 172.21.15.16:0/2868195907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc528087280 con 0x7fc528088b50 2019-12-05T06:57:42.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.362+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.363 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.362+0000 7fc5257fa700 1 -- 172.21.15.16:0/2868195907 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc5180012b0 con 0x7fc528088b50 2019-12-05T06:57:42.603 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.602+0000 7fc52e3b5700 1 -- 172.21.15.16:0/2868195907 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc528087280 con 0x7fc528088b50 2019-12-05T06:57:42.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.603+0000 7fc527fff700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=THROTTLE_DONE pgs=1122 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:42.604 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.603+0000 7fc5257fa700 1 -- 172.21.15.16:0/2868195907 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fc5180045f0 con 0x7fc528088b50 2019-12-05T06:57:42.606 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:42.606 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:53.063332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:42.606 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 202 2019-12-05T06:57:42.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 -- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc50c023630 msgr2=0x7fc50c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:42.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc50c023630 0x7fc50c025ac0 crc :-1 s=READY pgs=692 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:42.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 -- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 msgr2=0x7fc5280880c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:42.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 0x7fc5280880c0 secure :-1 s=READY pgs=1122 cs=0 l=1 rx=0x7fc518010430 tx=0x7fc5180107d0).stop 2019-12-05T06:57:42.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 -- 172.21.15.16:0/2868195907 shutdown_connections 2019-12-05T06:57:42.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc50c023630 0x7fc50c025ac0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:42.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fc528088b50 0x7fc5280880c0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:42.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 --2- 172.21.15.16:0/2868195907 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc52809e440 0x7fc528087b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:42.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 -- 172.21.15.16:0/2868195907 shutdown_connections 2019-12-05T06:57:42.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 -- 172.21.15.16:0/2868195907 wait complete. 2019-12-05T06:57:42.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:42.607+0000 7fc52e3b5700 1 -- 172.21.15.16:0/2868195907 >> 172.21.15.16:0/2868195907 conn(0x7fc528098480 msgr2=0x7fc528071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:42.620 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:57:44.621 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:44.855 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.846+0000 7f294dcbc700 1 Processor -- start 2019-12-05T06:57:44.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.846+0000 7f294dcbc700 1 -- start start 2019-12-05T06:57:44.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.847+0000 7f294dcbc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:44.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.847+0000 7f294dcbc700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f294808f720 0x7f294808d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:44.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.847+0000 7f294dcbc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29480958e0 con 0x7f294808fef0 2019-12-05T06:57:44.856 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.847+0000 7f294dcbc700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29480a3f70 con 0x7f294808f720 2019-12-05T06:57:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.847+0000 7f294dcbc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f294808d830 con 0x7f294808c6d0 2019-12-05T06:57:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f2947fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f294ca5a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f294808f720 0x7f294808d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f2947fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.76:3301/0 says I am v2:172.21.15.16:40790/0 (socket says 172.21.15.16:40790) 2019-12-05T06:57:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f2947fff700 1 -- 172.21.15.16:0/3272791600 learned_addr learned my addr 172.21.15.16:0/3272791600 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:57:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f29477fe700 1 --1- 172.21.15.16:0/3272791600 >> v1:172.21.15.76:6789/0 conn(0x7f294808fef0 0x7f294808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:43272/0 (socket says 172.21.15.16:43272) 2019-12-05T06:57:44.857 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f2947fff700 1 -- 172.21.15.16:0/3272791600 >> v2:172.21.15.76:3300/0 conn(0x7f294808f720 msgr2=0x7f294808d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f2947fff700 1 --2- 172.21.15.16:0/3272791600 >> v2:172.21.15.76:3300/0 conn(0x7f294808f720 0x7f294808d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f2947fff700 1 -- 172.21.15.16:0/3272791600 >> v1:172.21.15.76:6789/0 conn(0x7f294808fef0 legacy=0x7f294808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f2947fff700 1 -- 172.21.15.16:0/3272791600 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294808fbc0 con 0x7f294808c6d0 2019-12-05T06:57:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f294ca5a700 1 --2- 172.21.15.16:0/3272791600 >> v2:172.21.15.76:3300/0 conn(0x7f294808f720 0x7f294808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:57:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.848+0000 7f2947fff700 1 --2- 172.21.15.16:0/3272791600 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f293800fac0 tx=0x7f293800fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.849+0000 7f2947fff700 1 --2- 172.21.15.16:0/3272791600 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f293800fac0 tx=0x7f293800fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.858 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.849+0000 7f2947fff700 1 --2- 172.21.15.16:0/3272791600 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 secure :-1 s=READY pgs=389 cs=0 l=1 rx=0x7f293800fac0 tx=0x7f293800fe60).ready entity=mon.2 client_cookie=a27b9e56b9c4bc5c server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.849+0000 7f2947fff700 1 --2- 172.21.15.16:0/3272791600 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 secure :-1 s=THROTTLE_DONE pgs=389 cs=0 l=1 rx=0x7f293800fac0 tx=0x7f293800fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.849+0000 7f2947fff700 1 --2- 172.21.15.16:0/3272791600 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 secure :-1 s=THROTTLE_DONE pgs=389 cs=0 l=1 rx=0x7f293800fac0 tx=0x7f293800fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.849+0000 7f2947fff700 1 --2- 172.21.15.16:0/3272791600 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 secure :-1 s=THROTTLE_DONE pgs=389 cs=0 l=1 rx=0x7f293800fac0 tx=0x7f293800fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.849+0000 7f2946ffd700 1 -- 172.21.15.16:0/3272791600 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2938000ce0 con 0x7f294808c6d0 2019-12-05T06:57:44.859 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.849+0000 7f2946ffd700 1 -- 172.21.15.16:0/3272791600 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2938001010 con 0x7f294808c6d0 2019-12-05T06:57:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.849+0000 7f2946ffd700 1 -- 172.21.15.16:0/3272791600 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29380016e0 con 0x7f294808c6d0 2019-12-05T06:57:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.849+0000 7f294dcbc700 1 -- 172.21.15.16:0/3272791600 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 msgr2=0x7f294808cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.849+0000 7f294dcbc700 1 --2- 172.21.15.16:0/3272791600 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 secure :-1 s=READY pgs=389 cs=0 l=1 rx=0x7f293800fac0 tx=0x7f293800fe60).stop 2019-12-05T06:57:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.851+0000 7f294dcbc700 1 -- 172.21.15.16:0/3272791600 shutdown_connections 2019-12-05T06:57:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.851+0000 7f294dcbc700 1 --2- 172.21.15.16:0/3272791600 >> v2:172.21.15.76:3300/0 conn(0x7f294808f720 0x7f294808d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.851+0000 7f294dcbc700 1 --2- 172.21.15.16:0/3272791600 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294808c6d0 0x7f294808cac0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:44.860 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.851+0000 7f294dcbc700 1 -- 172.21.15.16:0/3272791600 shutdown_connections 2019-12-05T06:57:44.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.851+0000 7f294dcbc700 1 -- 172.21.15.16:0/3272791600 wait complete. 2019-12-05T06:57:44.861 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.851+0000 7f294dcbc700 1 -- 172.21.15.16:0/3272791600 >> 172.21.15.16:0/3272791600 conn(0x7f2948061940 msgr2=0x7f2948096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:44.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.855+0000 7f294dcbc700 1 Processor -- start 2019-12-05T06:57:44.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.855+0000 7f294dcbc700 1 -- start start 2019-12-05T06:57:44.864 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.857+0000 7f294dcbc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29481302f0 0x7f294809d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:44.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.857+0000 7f294dcbc700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294809e7d0 0x7f294809dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:44.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.857+0000 7f294dcbc700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f294812cb90 con 0x7f294809e280 2019-12-05T06:57:44.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.857+0000 7f294dcbc700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2948131ff0 con 0x7f29481302f0 2019-12-05T06:57:44.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.857+0000 7f294dcbc700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29481315c0 con 0x7f294809e7d0 2019-12-05T06:57:44.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.857+0000 7f2947fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29481302f0 0x7f294809d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:44.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.857+0000 7f2947fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f29481302f0 0x7f294809d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36806/0 (socket says 172.21.15.16:36806) 2019-12-05T06:57:44.865 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.857+0000 7f2947fff700 1 -- 172.21.15.16:0/275698983 learned_addr learned my addr 172.21.15.16:0/275698983 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:57:44.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.857+0000 7f29477fe700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294809e7d0 0x7f294809dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:44.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2947fff700 1 -- 172.21.15.16:0/275698983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294809e7d0 msgr2=0x7f294809dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:44.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294809e7d0 0x7f294809dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:44.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2947fff700 1 -- 172.21.15.16:0/275698983 >> v1:172.21.15.76:6789/0 conn(0x7f294809e280 legacy=0x7f294806dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:44.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2947fff700 1 -- 172.21.15.16:0/275698983 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2948132680 con 0x7f29481302f0 2019-12-05T06:57:44.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2944ff9700 1 -- 172.21.15.16:0/275698983 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3480459418 0 0) 0x7f29400097c0 con 0x7f294809e280 2019-12-05T06:57:44.866 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> v2:172.21.15.16:3300/0 conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> v2:172.21.15.16:3300/0 conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=READY pgs=1656 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:44.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=THROTTLE_DONE pgs=1656 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2944ff9700 1 -- 172.21.15.16:0/275698983 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2938005af0 con 0x7f29481302f0 2019-12-05T06:57:44.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=THROTTLE_DONE pgs=1656 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=THROTTLE_DONE pgs=1656 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.867 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2944ff9700 1 -- 172.21.15.16:0/275698983 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2938005710 con 0x7f29481302f0 2019-12-05T06:57:44.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f294dcbc700 1 -- 172.21.15.16:0/275698983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2948132680 con 0x7f29481302f0 2019-12-05T06:57:44.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f2944ff9700 1 -- 172.21.15.16:0/275698983 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29380060d0 con 0x7f29481302f0 2019-12-05T06:57:44.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.858+0000 7f294dcbc700 1 -- 172.21.15.16:0/275698983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2948132680 con 0x7f29481302f0 2019-12-05T06:57:44.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.859+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=THROTTLE_DONE pgs=1656 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.860+0000 7f2944ff9700 1 -- 172.21.15.16:0/275698983 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f2938005900 con 0x7f29481302f0 2019-12-05T06:57:44.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.860+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=THROTTLE_DONE pgs=1656 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.868 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.860+0000 7f2944ff9700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f292c023630 0x7f292c025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:44.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.860+0000 7f2944ff9700 1 -- 172.21.15.16:0/275698983 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f29380463a0 con 0x7f29481302f0 2019-12-05T06:57:44.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.860+0000 7f29477fe700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f292c023630 0x7f292c025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:44.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.861+0000 7f29477fe700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f292c023630 0x7f292c025ae0 crc :-1 s=READY pgs=693 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:44.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.861+0000 7f294dcbc700 1 -- 172.21.15.16:0/275698983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2948132680 con 0x7f29481302f0 2019-12-05T06:57:44.869 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.867+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=THROTTLE_DONE pgs=1656 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:44.870 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:44.867+0000 7f2944ff9700 1 -- 172.21.15.16:0/275698983 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f2938005500 con 0x7f29481302f0 2019-12-05T06:57:45.114 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.113+0000 7f294dcbc700 1 -- 172.21.15.16:0/275698983 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2948132680 con 0x7f29481302f0 2019-12-05T06:57:45.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.114+0000 7f2947fff700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=THROTTLE_DONE pgs=1656 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:45.115 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.114+0000 7f2944ff9700 1 -- 172.21.15.16:0/275698983 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f2938005710 con 0x7f29481302f0 2019-12-05T06:57:45.115 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 202 2019-12-05T06:57:45.115 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:45.116 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:53.063332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.116+0000 7f294dcbc700 1 -- 172.21.15.16:0/275698983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f292c023630 msgr2=0x7f292c025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:45.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.116+0000 7f294dcbc700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f292c023630 0x7f292c025ae0 crc :-1 s=READY pgs=693 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.116+0000 7f294dcbc700 1 -- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 msgr2=0x7f294809d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.116+0000 7f294dcbc700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 0x7f294809d800 secure :-1 s=READY pgs=1656 cs=0 l=1 rx=0x7f2938010790 tx=0x7f2938010610).stop 2019-12-05T06:57:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.116+0000 7f294dcbc700 1 -- 172.21.15.16:0/275698983 shutdown_connections 2019-12-05T06:57:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.116+0000 7f294dcbc700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f294809e7d0 0x7f294809dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:45.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.116+0000 7f294dcbc700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f292c023630 0x7f292c025ae0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.116+0000 7f294dcbc700 1 --2- 172.21.15.16:0/275698983 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f29481302f0 0x7f294809d800 unknown :-1 s=CLOSED pgs=1656 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:45.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.116+0000 7f294dcbc700 1 -- 172.21.15.16:0/275698983 shutdown_connections 2019-12-05T06:57:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.117+0000 7f294dcbc700 1 -- 172.21.15.16:0/275698983 wait complete. 2019-12-05T06:57:45.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:45.117+0000 7f294dcbc700 1 -- 172.21.15.16:0/275698983 >> 172.21.15.16:0/275698983 conn(0x7f29480a2850 msgr2=0x7f294806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:45.128 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:57:47.129 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:47.407 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.397+0000 7fbead8cb700 1 Processor -- start 2019-12-05T06:57:47.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.397+0000 7fbead8cb700 1 -- start start 2019-12-05T06:57:47.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.399+0000 7fbead8cb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbea808d2c0 0x7fbea808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:47.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.399+0000 7fbead8cb700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbea808fef0 0x7fbea808f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:47.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.399+0000 7fbead8cb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea80958e0 con 0x7fbea808c6d0 2019-12-05T06:57:47.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.399+0000 7fbead8cb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea808df90 con 0x7fbea808d2c0 2019-12-05T06:57:47.408 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.399+0000 7fbead8cb700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea80a3f70 con 0x7fbea808fef0 2019-12-05T06:57:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.400+0000 7fbea6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbea808d2c0 0x7fbea808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.400+0000 7fbea6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbea808d2c0 0x7fbea808ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36820/0 (socket says 172.21.15.16:36820) 2019-12-05T06:57:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.400+0000 7fbea77fe700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fbea808c6d0 0x7fbea808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:43286/0 (socket says 172.21.15.16:43286) 2019-12-05T06:57:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.400+0000 7fbea6ffd700 1 -- 172.21.15.16:0/3964606653 learned_addr learned my addr 172.21.15.16:0/3964606653 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:57:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.400+0000 7fbea7fff700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbea808fef0 0x7fbea808f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:47.409 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.400+0000 7fbea67fc700 1 -- 172.21.15.16:0/3964606653 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167960014 0 0) 0x7fbea0001400 con 0x7fbea808c6d0 2019-12-05T06:57:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.400+0000 7fbea6ffd700 1 -- 172.21.15.16:0/3964606653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbea808fef0 msgr2=0x7fbea808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.400+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/3964606653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbea808fef0 0x7fbea808f460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.400+0000 7fbea6ffd700 1 -- 172.21.15.16:0/3964606653 >> v1:172.21.15.76:6789/0 conn(0x7fbea808c6d0 legacy=0x7fbea808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.400+0000 7fbea6ffd700 1 -- 172.21.15.16:0/3964606653 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea808e620 con 0x7fbea808d2c0 2019-12-05T06:57:47.410 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.401+0000 7fbea7fff700 1 --2- 172.21.15.16:0/3964606653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbea808fef0 0x7fbea808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:57:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.401+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/3964606653 >> v2:172.21.15.16:3300/0 conn(0x7fbea808d2c0 0x7fbea808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbe9000f740 tx=0x7fbe9000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.401+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/3964606653 >> v2:172.21.15.16:3300/0 conn(0x7fbea808d2c0 0x7fbea808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbe9000f740 tx=0x7fbe9000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.401+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/3964606653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea808d2c0 0x7fbea808ef20 secure :-1 s=READY pgs=1657 cs=0 l=1 rx=0x7fbe9000f740 tx=0x7fbe9000fae0).ready entity=mon.0 client_cookie=8ba6a2b7237108b0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.401+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/3964606653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea808d2c0 0x7fbea808ef20 secure :-1 s=THROTTLE_DONE pgs=1657 cs=0 l=1 rx=0x7fbe9000f740 tx=0x7fbe9000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.401+0000 7fbea67fc700 1 -- 172.21.15.16:0/3964606653 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe90000960 con 0x7fbea808d2c0 2019-12-05T06:57:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.401+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/3964606653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea808d2c0 0x7fbea808ef20 secure :-1 s=THROTTLE_DONE pgs=1657 cs=0 l=1 rx=0x7fbe9000f740 tx=0x7fbe9000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.411 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.401+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/3964606653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea808d2c0 0x7fbea808ef20 secure :-1 s=THROTTLE_DONE pgs=1657 cs=0 l=1 rx=0x7fbe9000f740 tx=0x7fbe9000fae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.401+0000 7fbea67fc700 1 -- 172.21.15.16:0/3964606653 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe90000c90 con 0x7fbea808d2c0 2019-12-05T06:57:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.402+0000 7fbea67fc700 1 -- 172.21.15.16:0/3964606653 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe900013a0 con 0x7fbea808d2c0 2019-12-05T06:57:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.402+0000 7fbead8cb700 1 -- 172.21.15.16:0/3964606653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea808d2c0 msgr2=0x7fbea808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.402+0000 7fbead8cb700 1 --2- 172.21.15.16:0/3964606653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea808d2c0 0x7fbea808ef20 secure :-1 s=READY pgs=1657 cs=0 l=1 rx=0x7fbe9000f740 tx=0x7fbe9000fae0).stop 2019-12-05T06:57:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.403+0000 7fbead8cb700 1 -- 172.21.15.16:0/3964606653 shutdown_connections 2019-12-05T06:57:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.403+0000 7fbead8cb700 1 --2- 172.21.15.16:0/3964606653 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbea808fef0 0x7fbea808f460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:47.412 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.403+0000 7fbead8cb700 1 --2- 172.21.15.16:0/3964606653 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea808d2c0 0x7fbea808ef20 unknown :-1 s=CLOSED pgs=1657 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.403+0000 7fbead8cb700 1 -- 172.21.15.16:0/3964606653 shutdown_connections 2019-12-05T06:57:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.403+0000 7fbead8cb700 1 -- 172.21.15.16:0/3964606653 wait complete. 2019-12-05T06:57:47.413 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.403+0000 7fbead8cb700 1 -- 172.21.15.16:0/3964606653 >> 172.21.15.16:0/3964606653 conn(0x7fbea8061940 msgr2=0x7fbea8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.408+0000 7fbead8cb700 1 Processor -- start 2019-12-05T06:57:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.409+0000 7fbead8cb700 1 -- start start 2019-12-05T06:57:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.410+0000 7fbead8cb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbea80992d0 0x7fbea8098b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.410+0000 7fbead8cb700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea8089fb0 con 0x7fbea8156cb0 2019-12-05T06:57:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.411+0000 7fbead8cb700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea806e300 con 0x7fbea806df10 2019-12-05T06:57:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.411+0000 7fbead8cb700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea8097370 con 0x7fbea80992d0 2019-12-05T06:57:47.417 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.411+0000 7fbea6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fbea80992d0 0x7fbea8098b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.411+0000 7fbea7fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbea8156cb0 0x7fbea81590b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50226/0 (socket says 172.21.15.16:50226) 2019-12-05T06:57:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.411+0000 7fbea7fff700 1 -- 172.21.15.16:0/177290815 learned_addr learned my addr 172.21.15.16:0/177290815 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.411+0000 7fbe9ffff700 1 -- 172.21.15.16:0/177290815 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194351035 0 0) 0x7fbea0001020 con 0x7fbea806df10 2019-12-05T06:57:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.411+0000 7fbe9ffff700 1 -- 172.21.15.16:0/177290815 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe8c002e80 con 0x7fbea806df10 2019-12-05T06:57:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.411+0000 7fbe9ffff700 1 -- 172.21.15.16:0/177290815 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121222839 0 0) 0x7fbe980097c0 con 0x7fbea8156cb0 2019-12-05T06:57:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.411+0000 7fbea6ffd700 1 -- 172.21.15.16:0/177290815 >> v1:172.21.15.76:6789/0 conn(0x7fbea806df10 legacy=0x7fbea8098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:47.418 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbea6ffd700 1 -- 172.21.15.16:0/177290815 >> v1:172.21.15.16:6789/0 conn(0x7fbea8156cb0 legacy=0x7fbea81590b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbea6ffd700 1 -- 172.21.15.16:0/177290815 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea8097a00 con 0x7fbea80992d0 2019-12-05T06:57:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/177290815 >> v2:172.21.15.16:3300/0 conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/177290815 >> v2:172.21.15.16:3300/0 conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=READY pgs=1659 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=THROTTLE_DONE pgs=1659 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbe9ffff700 1 -- 172.21.15.16:0/177290815 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe90000f50 con 0x7fbea80992d0 2019-12-05T06:57:47.419 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=THROTTLE_DONE pgs=1659 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbe9ffff700 1 -- 172.21.15.16:0/177290815 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe90000940 con 0x7fbea80992d0 2019-12-05T06:57:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=THROTTLE_DONE pgs=1659 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbead8cb700 1 -- 172.21.15.16:0/177290815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea8097a00 con 0x7fbea80992d0 2019-12-05T06:57:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbe9ffff700 1 -- 172.21.15.16:0/177290815 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe90000d60 con 0x7fbea80992d0 2019-12-05T06:57:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.412+0000 7fbead8cb700 1 -- 172.21.15.16:0/177290815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea8097a00 con 0x7fbea80992d0 2019-12-05T06:57:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.413+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=THROTTLE_DONE pgs=1659 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.420 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.414+0000 7fbe9ffff700 1 -- 172.21.15.16:0/177290815 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbe90000d60 con 0x7fbea80992d0 2019-12-05T06:57:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.414+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=THROTTLE_DONE pgs=1659 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.415+0000 7fbe9ffff700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe8c023680 0x7fbe8c025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.415+0000 7fbe9ffff700 1 -- 172.21.15.16:0/177290815 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7fbe90045e10 con 0x7fbea80992d0 2019-12-05T06:57:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.415+0000 7fbea77fe700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe8c023680 0x7fbe8c025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:47.421 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.415+0000 7fbea77fe700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe8c023680 0x7fbe8c025af0 crc :-1 s=READY pgs=694 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:47.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.429+0000 7fbead8cb700 1 -- 172.21.15.16:0/177290815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea8097a00 con 0x7fbea80992d0 2019-12-05T06:57:47.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.437+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=THROTTLE_DONE pgs=1659 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.438 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.437+0000 7fbe9ffff700 1 -- 172.21.15.16:0/177290815 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbe9002ae30 con 0x7fbea80992d0 2019-12-05T06:57:47.686 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:57:47.691 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:57:47.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.709+0000 7fbead8cb700 1 -- 172.21.15.16:0/177290815 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbea8097a00 con 0x7fbea80992d0 2019-12-05T06:57:47.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.709+0000 7fbea6ffd700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=THROTTLE_DONE pgs=1659 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:47.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.709+0000 7fbe9ffff700 1 -- 172.21.15.16:0/177290815 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fbe900012e0 con 0x7fbea80992d0 2019-12-05T06:57:47.711 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 202 2019-12-05T06:57:47.711 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:47.711 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:53.063332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:47.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.712+0000 7fbead8cb700 1 -- 172.21.15.16:0/177290815 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe8c023680 msgr2=0x7fbe8c025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:47.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.712+0000 7fbead8cb700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe8c023680 0x7fbe8c025af0 crc :-1 s=READY pgs=694 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:47.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.712+0000 7fbead8cb700 1 -- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 msgr2=0x7fbea8098b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:47.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.712+0000 7fbead8cb700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 0x7fbea8098b00 secure :-1 s=READY pgs=1659 cs=0 l=1 rx=0x7fbe900103a0 tx=0x7fbe90010740).stop 2019-12-05T06:57:47.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.713+0000 7fbead8cb700 1 -- 172.21.15.16:0/177290815 shutdown_connections 2019-12-05T06:57:47.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.713+0000 7fbead8cb700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbe8c023680 0x7fbe8c025af0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:47.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.713+0000 7fbead8cb700 1 --2- 172.21.15.16:0/177290815 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fbea80992d0 0x7fbea8098b00 unknown :-1 s=CLOSED pgs=1659 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:47.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.713+0000 7fbead8cb700 1 -- 172.21.15.16:0/177290815 shutdown_connections 2019-12-05T06:57:47.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.713+0000 7fbead8cb700 1 -- 172.21.15.16:0/177290815 wait complete. 2019-12-05T06:57:47.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:47.713+0000 7fbead8cb700 1 -- 172.21.15.16:0/177290815 >> 172.21.15.16:0/177290815 conn(0x7fbea80714f0 msgr2=0x7fbea809ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:47.725 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:57:49.726 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.993+0000 7f02ad800700 1 Processor -- start 2019-12-05T06:57:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.993+0000 7f02ad800700 1 -- start start 2019-12-05T06:57:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.995+0000 7f02ad800700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 0x7f02a808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:50.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.995+0000 7f02ad800700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a80958e0 con 0x7f02a808d2c0 2019-12-05T06:57:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.995+0000 7f02ad800700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a80629e0 con 0x7f02a808c6d0 2019-12-05T06:57:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.995+0000 7f02ad800700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a808df90 con 0x7f02a808fef0 2019-12-05T06:57:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.995+0000 7f02a6ffd700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f02a808d2c0 0x7f02a808ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50232/0 (socket says 172.21.15.16:50232) 2019-12-05T06:57:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.995+0000 7f02a6ffd700 1 -- 172.21.15.16:0/54790942 learned_addr learned my addr 172.21.15.16:0/54790942 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.995+0000 7f02a7fff700 1 --2- 172.21.15.16:0/54790942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 0x7f02a808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.996+0000 7f02a67fc700 1 -- 172.21.15.16:0/54790942 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775558877 0 0) 0x7f0290001150 con 0x7f02a808d2c0 2019-12-05T06:57:50.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.996+0000 7f02a67fc700 1 -- 172.21.15.16:0/54790942 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0294002e80 con 0x7f02a808d2c0 2019-12-05T06:57:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.996+0000 7f02a67fc700 1 -- 172.21.15.16:0/54790942 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908796477 0 0) 0x7f02a0001400 con 0x7f02a808c6d0 2019-12-05T06:57:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.996+0000 7f02a67fc700 1 -- 172.21.15.16:0/54790942 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0294004170 con 0x7f02a808c6d0 2019-12-05T06:57:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.996+0000 7f02a7fff700 1 -- 172.21.15.16:0/54790942 >> v1:172.21.15.76:6789/0 conn(0x7f02a808c6d0 legacy=0x7f02a808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.996+0000 7f02a7fff700 1 -- 172.21.15.16:0/54790942 >> v1:172.21.15.16:6789/0 conn(0x7f02a808d2c0 legacy=0x7f02a808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.996+0000 7f02a7fff700 1 -- 172.21.15.16:0/54790942 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a808e620 con 0x7f02a808fef0 2019-12-05T06:57:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.997+0000 7f02a7fff700 1 --2- 172.21.15.16:0/54790942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 0x7f02a808f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f029800af80 tx=0x7f029800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.005 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.997+0000 7f02a7fff700 1 --2- 172.21.15.16:0/54790942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 0x7f02a808f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f029800af80 tx=0x7f029800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.997+0000 7f02a7fff700 1 --2- 172.21.15.16:0/54790942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 0x7f02a808f720 secure :-1 s=READY pgs=390 cs=0 l=1 rx=0x7f029800af80 tx=0x7f029800b790).ready entity=mon.2 client_cookie=8b4bac9c7d36552a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.997+0000 7f02a7fff700 1 --2- 172.21.15.16:0/54790942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 0x7f02a808f720 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f029800af80 tx=0x7f029800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.997+0000 7f02a67fc700 1 -- 172.21.15.16:0/54790942 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0298007240 con 0x7f02a808fef0 2019-12-05T06:57:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.997+0000 7f02a7fff700 1 --2- 172.21.15.16:0/54790942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 0x7f02a808f720 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f029800af80 tx=0x7f029800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.997+0000 7f02a7fff700 1 --2- 172.21.15.16:0/54790942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 0x7f02a808f720 secure :-1 s=THROTTLE_DONE pgs=390 cs=0 l=1 rx=0x7f029800af80 tx=0x7f029800b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.997+0000 7f02a67fc700 1 -- 172.21.15.16:0/54790942 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0298000bb0 con 0x7f02a808fef0 2019-12-05T06:57:50.006 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.997+0000 7f02a67fc700 1 -- 172.21.15.16:0/54790942 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0298009e60 con 0x7f02a808fef0 2019-12-05T06:57:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.998+0000 7f02ad800700 1 -- 172.21.15.16:0/54790942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 msgr2=0x7f02a808f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.998+0000 7f02ad800700 1 --2- 172.21.15.16:0/54790942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 0x7f02a808f720 secure :-1 s=READY pgs=390 cs=0 l=1 rx=0x7f029800af80 tx=0x7f029800b790).stop 2019-12-05T06:57:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.999+0000 7f02ad800700 1 -- 172.21.15.16:0/54790942 shutdown_connections 2019-12-05T06:57:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.999+0000 7f02ad800700 1 --2- 172.21.15.16:0/54790942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a808fef0 0x7f02a808f720 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.999+0000 7f02ad800700 1 -- 172.21.15.16:0/54790942 shutdown_connections 2019-12-05T06:57:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.999+0000 7f02ad800700 1 -- 172.21.15.16:0/54790942 wait complete. 2019-12-05T06:57:50.007 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:49.999+0000 7f02ad800700 1 -- 172.21.15.16:0/54790942 >> 172.21.15.16:0/54790942 conn(0x7f02a8061940 msgr2=0x7f02a8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:50.026 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.003+0000 7f02ad800700 1 Processor -- start 2019-12-05T06:57:50.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.004+0000 7f02ad800700 1 -- start start 2019-12-05T06:57:50.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.005+0000 7f02ad800700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a8153a50 0x7f02a8153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:50.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.005+0000 7f02ad800700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02a812a1d0 0x7f02a8129200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:50.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.005+0000 7f02ad800700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02a806e5c0 con 0x7f02a8129740 2019-12-05T06:57:50.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.005+0000 7f02ad800700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f02a806e250 con 0x7f02a812a1d0 2019-12-05T06:57:50.027 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.005+0000 7f02ad800700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f02a806eac0 con 0x7f02a8153a50 2019-12-05T06:57:50.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.006+0000 7f02a6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02a812a1d0 0x7f02a8129200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:50.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.006+0000 7f02a6ffd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f02a812a1d0 0x7f02a8129200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36838/0 (socket says 172.21.15.16:36838) 2019-12-05T06:57:50.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.006+0000 7f02a6ffd700 1 -- 172.21.15.16:0/3972087645 learned_addr learned my addr 172.21.15.16:0/3972087645 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:57:50.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.006+0000 7f02a77fe700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a8153a50 0x7f02a8153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:50.028 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.006+0000 7f02a6ffd700 1 -- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a8153a50 msgr2=0x7f02a8153e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.006+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a8153a50 0x7f02a8153e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:50.045 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.006+0000 7f02a6ffd700 1 -- 172.21.15.16:0/3972087645 >> v1:172.21.15.16:6789/0 conn(0x7f02a8129740 legacy=0x7f02a80a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.006+0000 7f02a6ffd700 1 -- 172.21.15.16:0/3972087645 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02a8128900 con 0x7f02a812a1d0 2019-12-05T06:57:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f02a77fe700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a8153a50 0x7f02a8153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:57:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> v2:172.21.15.16:3300/0 conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> v2:172.21.15.16:3300/0 conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=READY pgs=1662 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.046 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f029ffff700 1 -- 172.21.15.16:0/3972087645 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0290000c90 con 0x7f02a812a1d0 2019-12-05T06:57:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f029ffff700 1 -- 172.21.15.16:0/3972087645 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02900010c0 con 0x7f02a812a1d0 2019-12-05T06:57:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f02ad800700 1 -- 172.21.15.16:0/3972087645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02a8128900 con 0x7f02a812a1d0 2019-12-05T06:57:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f029ffff700 1 -- 172.21.15.16:0/3972087645 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0290001710 con 0x7f02a812a1d0 2019-12-05T06:57:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.007+0000 7f02ad800700 1 -- 172.21.15.16:0/3972087645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02a8128900 con 0x7f02a812a1d0 2019-12-05T06:57:50.047 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.008+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.009+0000 7f029ffff700 1 -- 172.21.15.16:0/3972087645 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f0290000c90 con 0x7f02a812a1d0 2019-12-05T06:57:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.009+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.010+0000 7f029ffff700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f028c023630 0x7f028c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.010+0000 7f029ffff700 1 -- 172.21.15.16:0/3972087645 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f0290045df0 con 0x7f02a812a1d0 2019-12-05T06:57:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.010+0000 7f02a77fe700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f028c023630 0x7f028c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.011+0000 7f02a77fe700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f028c023630 0x7f028c025ac0 crc :-1 s=READY pgs=695 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:50.048 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.025+0000 7f02ad800700 1 -- 172.21.15.16:0/3972087645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02a8128900 con 0x7f02a812a1d0 2019-12-05T06:57:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.031+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.049 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.031+0000 7f029ffff700 1 -- 172.21.15.16:0/3972087645 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f02900075a0 con 0x7f02a812a1d0 2019-12-05T06:57:50.285 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.284+0000 7f02ad800700 1 -- 172.21.15.16:0/3972087645 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f02a8128900 con 0x7f02a812a1d0 2019-12-05T06:57:50.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.285+0000 7f02a6ffd700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=THROTTLE_DONE pgs=1662 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:50.286 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.285+0000 7f029ffff700 1 -- 172.21.15.16:0/3972087645 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f0290000fb0 con 0x7f02a812a1d0 2019-12-05T06:57:50.287 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 202 2019-12-05T06:57:50.287 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:50.287 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:53.063332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.287+0000 7f02ad800700 1 -- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f028c023630 msgr2=0x7f028c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.287+0000 7f02ad800700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f028c023630 0x7f028c025ac0 crc :-1 s=READY pgs=695 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:50.289 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.287+0000 7f02ad800700 1 -- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 msgr2=0x7f02a8129200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.287+0000 7f02ad800700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 0x7f02a8129200 secure :-1 s=READY pgs=1662 cs=0 l=1 rx=0x7f02900103e0 tx=0x7f0290010780).stop 2019-12-05T06:57:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.288+0000 7f02ad800700 1 -- 172.21.15.16:0/3972087645 shutdown_connections 2019-12-05T06:57:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.288+0000 7f02ad800700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f028c023630 0x7f028c025ac0 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:50.290 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.288+0000 7f02ad800700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f02a812a1d0 0x7f02a8129200 unknown :-1 s=CLOSED pgs=1662 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.288+0000 7f02ad800700 1 --2- 172.21.15.16:0/3972087645 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f02a8153a50 0x7f02a8153e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.288+0000 7f02ad800700 1 -- 172.21.15.16:0/3972087645 shutdown_connections 2019-12-05T06:57:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.288+0000 7f02ad800700 1 -- 172.21.15.16:0/3972087645 wait complete. 2019-12-05T06:57:50.291 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:50.288+0000 7f02ad800700 1 -- 172.21.15.16:0/3972087645 >> 172.21.15.16:0/3972087645 conn(0x7f02a806d260 msgr2=0x7f02a809aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:50.304 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:57:52.305 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.575+0000 7f3a9366c700 1 Processor -- start 2019-12-05T06:57:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.575+0000 7f3a9366c700 1 -- start start 2019-12-05T06:57:52.585 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.577+0000 7f3a9366c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3a8c08c6d0 0x7f3a8c08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.577+0000 7f3a9366c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3a8c08fef0 0x7f3a8c08d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.577+0000 7f3a9366c700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8c08fb80 con 0x7f3a8c08ef20 2019-12-05T06:57:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.577+0000 7f3a9366c700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a8c08df90 con 0x7f3a8c08fef0 2019-12-05T06:57:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.577+0000 7f3a9366c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a8c0a3f70 con 0x7f3a8c08c6d0 2019-12-05T06:57:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.577+0000 7f3a91408700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3a8c08fef0 0x7f3a8c08d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.577+0000 7f3a91408700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3a8c08fef0 0x7f3a8c08d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36848/0 (socket says 172.21.15.16:36848) 2019-12-05T06:57:52.586 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.577+0000 7f3a91c09700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3a8c08c6d0 0x7f3a8c08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.577+0000 7f3a91408700 1 -- 172.21.15.16:0/2083908562 learned_addr learned my addr 172.21.15.16:0/2083908562 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:57:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.578+0000 7f3a91408700 1 -- 172.21.15.16:0/2083908562 >> v2:172.21.15.76:3300/0 conn(0x7f3a8c08c6d0 msgr2=0x7f3a8c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:57:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.578+0000 7f3a91408700 1 --2- 172.21.15.16:0/2083908562 >> v2:172.21.15.76:3300/0 conn(0x7f3a8c08c6d0 0x7f3a8c08cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.578+0000 7f3a91408700 1 -- 172.21.15.16:0/2083908562 >> v1:172.21.15.76:6789/0 conn(0x7f3a8c08ef20 legacy=0x7f3a8c08f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.578+0000 7f3a91408700 1 -- 172.21.15.16:0/2083908562 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c08d540 con 0x7f3a8c08fef0 2019-12-05T06:57:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.578+0000 7f3a91408700 1 --2- 172.21.15.16:0/2083908562 >> v2:172.21.15.16:3300/0 conn(0x7f3a8c08fef0 0x7f3a8c08d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3a8000f6c0 tx=0x7f3a8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.587 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.578+0000 7f3a91408700 1 --2- 172.21.15.16:0/2083908562 >> v2:172.21.15.16:3300/0 conn(0x7f3a8c08fef0 0x7f3a8c08d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3a8000f6c0 tx=0x7f3a8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.578+0000 7f3a91408700 1 --2- 172.21.15.16:0/2083908562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c08fef0 0x7f3a8c08d000 secure :-1 s=READY pgs=1663 cs=0 l=1 rx=0x7f3a8000f6c0 tx=0x7f3a8000fa60).ready entity=mon.0 client_cookie=4e9a3397a0fac784 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.578+0000 7f3a91408700 1 --2- 172.21.15.16:0/2083908562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c08fef0 0x7f3a8c08d000 secure :-1 s=THROTTLE_DONE pgs=1663 cs=0 l=1 rx=0x7f3a8000f6c0 tx=0x7f3a8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.579+0000 7f3a91408700 1 --2- 172.21.15.16:0/2083908562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c08fef0 0x7f3a8c08d000 secure :-1 s=THROTTLE_DONE pgs=1663 cs=0 l=1 rx=0x7f3a8000f6c0 tx=0x7f3a8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.579+0000 7f3a91408700 1 --2- 172.21.15.16:0/2083908562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c08fef0 0x7f3a8c08d000 secure :-1 s=THROTTLE_DONE pgs=1663 cs=0 l=1 rx=0x7f3a8000f6c0 tx=0x7f3a8000fa60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.579+0000 7f3a90c07700 1 -- 172.21.15.16:0/2083908562 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a80000960 con 0x7f3a8c08fef0 2019-12-05T06:57:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.579+0000 7f3a90c07700 1 -- 172.21.15.16:0/2083908562 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a80000c90 con 0x7f3a8c08fef0 2019-12-05T06:57:52.588 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.579+0000 7f3a90c07700 1 -- 172.21.15.16:0/2083908562 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a800013a0 con 0x7f3a8c08fef0 2019-12-05T06:57:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.579+0000 7f3a9366c700 1 -- 172.21.15.16:0/2083908562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c08fef0 msgr2=0x7f3a8c08d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:52.589 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.579+0000 7f3a9366c700 1 --2- 172.21.15.16:0/2083908562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c08fef0 0x7f3a8c08d000 secure :-1 s=READY pgs=1663 cs=0 l=1 rx=0x7f3a8000f6c0 tx=0x7f3a8000fa60).stop 2019-12-05T06:57:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.580+0000 7f3a9366c700 1 -- 172.21.15.16:0/2083908562 shutdown_connections 2019-12-05T06:57:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.580+0000 7f3a9366c700 1 --2- 172.21.15.16:0/2083908562 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3a8c08fef0 0x7f3a8c08d000 unknown :-1 s=CLOSED pgs=1663 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.580+0000 7f3a9366c700 1 --2- 172.21.15.16:0/2083908562 >> v2:172.21.15.76:3300/0 conn(0x7f3a8c08c6d0 0x7f3a8c08cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:57:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.581+0000 7f3a9366c700 1 -- 172.21.15.16:0/2083908562 shutdown_connections 2019-12-05T06:57:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.581+0000 7f3a9366c700 1 -- 172.21.15.16:0/2083908562 wait complete. 2019-12-05T06:57:52.590 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.581+0000 7f3a9366c700 1 -- 172.21.15.16:0/2083908562 >> 172.21.15.16:0/2083908562 conn(0x7f3a8c061940 msgr2=0x7f3a8c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.586+0000 7f3a9366c700 1 Processor -- start 2019-12-05T06:57:52.591 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.586+0000 7f3a9366c700 1 -- start start 2019-12-05T06:57:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.587+0000 7f3a9366c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.587+0000 7f3a9366c700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3a8c099eb0 0x7f3a8c09a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.587+0000 7f3a9366c700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8c06e7c0 con 0x7f3a8c132630 2019-12-05T06:57:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.587+0000 7f3a9366c700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a8c098f20 con 0x7f3a8c099eb0 2019-12-05T06:57:52.592 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.587+0000 7f3a9366c700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c134390 con 0x7f3a8c09ae80 2019-12-05T06:57:52.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.588+0000 7f3a91c09700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3a8c132630 0x7f3a8c132a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50252/0 (socket says 172.21.15.16:50252) 2019-12-05T06:57:52.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.588+0000 7f3a91c09700 1 -- 172.21.15.16:0/221920601 learned_addr learned my addr 172.21.15.16:0/221920601 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:52.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.588+0000 7f3a9240a700 1 --2- 172.21.15.16:0/221920601 >> v2:172.21.15.76:3300/0 conn(0x7f3a8c099eb0 0x7f3a8c09a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:52.595 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.588+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:52.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.588+0000 7f3a8a7fc700 1 -- 172.21.15.16:0/221920601 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1346628149 0 0) 0x7f3a84001430 con 0x7f3a8c132630 2019-12-05T06:57:52.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.588+0000 7f3a8a7fc700 1 -- 172.21.15.16:0/221920601 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a6c002e80 con 0x7f3a8c132630 2019-12-05T06:57:52.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.589+0000 7f3a91408700 1 -- 172.21.15.16:0/221920601 >> v2:172.21.15.76:3300/0 conn(0x7f3a8c099eb0 msgr2=0x7f3a8c09a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:52.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.589+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> v2:172.21.15.76:3300/0 conn(0x7f3a8c099eb0 0x7f3a8c09a370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:52.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.589+0000 7f3a91408700 1 -- 172.21.15.16:0/221920601 >> v1:172.21.15.16:6789/0 conn(0x7f3a8c132630 legacy=0x7f3a8c132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:52.596 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.589+0000 7f3a8a7fc700 1 -- 172.21.15.16:0/221920601 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 3132058888 0 0) 0x7f3a84001630 con 0x7f3a8c132630 2019-12-05T06:57:52.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.589+0000 7f3a91408700 1 -- 172.21.15.16:0/221920601 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c0995b0 con 0x7f3a8c09ae80 2019-12-05T06:57:52.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.589+0000 7f3a9240a700 1 --2- 172.21.15.16:0/221920601 >> v2:172.21.15.76:3300/0 conn(0x7f3a8c099eb0 0x7f3a8c09a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:57:52.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.589+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.589+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.589+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=READY pgs=391 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:52.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.589+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.597 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.590+0000 7f3a8a7fc700 1 -- 172.21.15.16:0/221920601 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a80000940 con 0x7f3a8c09ae80 2019-12-05T06:57:52.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.590+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.590+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.590+0000 7f3a8a7fc700 1 -- 172.21.15.16:0/221920601 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a80000f50 con 0x7f3a8c09ae80 2019-12-05T06:57:52.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.590+0000 7f3a9366c700 1 -- 172.21.15.16:0/221920601 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8c0995b0 con 0x7f3a8c09ae80 2019-12-05T06:57:52.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.590+0000 7f3a8a7fc700 1 -- 172.21.15.16:0/221920601 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a800016a0 con 0x7f3a8c09ae80 2019-12-05T06:57:52.598 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.590+0000 7f3a9366c700 1 -- 172.21.15.16:0/221920601 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8c0995b0 con 0x7f3a8c09ae80 2019-12-05T06:57:52.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.591+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.592+0000 7f3a8a7fc700 1 -- 172.21.15.16:0/221920601 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3a80001140 con 0x7f3a8c09ae80 2019-12-05T06:57:52.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.592+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.592+0000 7f3a8a7fc700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a6c023680 0x7f3a6c025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:52.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.592+0000 7f3a8a7fc700 1 -- 172.21.15.16:0/221920601 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 8955+0+0 (secure 0 0 0) 0x7f3a80045d50 con 0x7f3a8c09ae80 2019-12-05T06:57:52.599 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.592+0000 7f3a91c09700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a6c023680 0x7f3a6c025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:52.600 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.593+0000 7f3a91c09700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a6c023680 0x7f3a6c025af0 crc :-1 s=READY pgs=696 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:52.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.606+0000 7f3a9366c700 1 -- 172.21.15.16:0/221920601 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8c0995b0 con 0x7f3a8c09ae80 2019-12-05T06:57:52.616 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.616+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.616+0000 7f3a8a7fc700 1 -- 172.21.15.16:0/221920601 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3a80000b50 con 0x7f3a8c09ae80 2019-12-05T06:57:52.875 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.874+0000 7f3a9366c700 1 -- 172.21.15.16:0/221920601 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3a8c0995b0 con 0x7f3a8c09ae80 2019-12-05T06:57:52.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.876+0000 7f3a91408700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=THROTTLE_DONE pgs=391 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:52.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.876+0000 7f3a8a7fc700 1 -- 172.21.15.16:0/221920601 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 202 v202) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f3a800019f0 con 0x7f3a8c09ae80 2019-12-05T06:57:52.878 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 202 2019-12-05T06:57:52.878 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:52.879 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":202,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8430,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":202}],"filesystems":[{"mdsmap":{"epoch":201,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:56:53.063332+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":83,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8308},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8308":{"gid":8308,"name":"a","rank":0,"incarnation":198,"state":"up:active","state_seq":15,"addr":"172.21.15.16:6833/2914676212","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":2914676212},{"type":"v1","addr":"172.21.15.16:6833","nonce":2914676212}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:52.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 -- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a6c023680 msgr2=0x7f3a6c025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:52.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a6c023680 0x7f3a6c025af0 crc :-1 s=READY pgs=696 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:52.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 -- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 msgr2=0x7f3a8c133220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:52.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 secure :-1 s=READY pgs=391 cs=0 l=1 rx=0x7f3a800103a0 tx=0x7f3a80010740).stop 2019-12-05T06:57:52.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 -- 172.21.15.16:0/221920601 shutdown_connections 2019-12-05T06:57:52.883 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3a6c023680 0x7f3a6c025af0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:52.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 --2- 172.21.15.16:0/221920601 >> v2:172.21.15.76:3300/0 conn(0x7f3a8c099eb0 0x7f3a8c09a370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:52.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 --2- 172.21.15.16:0/221920601 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f3a8c09ae80 0x7f3a8c133220 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:52.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 -- 172.21.15.16:0/221920601 shutdown_connections 2019-12-05T06:57:52.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 -- 172.21.15.16:0/221920601 wait complete. 2019-12-05T06:57:52.884 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:52.881+0000 7f3a9366c700 1 -- 172.21.15.16:0/221920601 >> 172.21.15.16:0/221920601 conn(0x7f3a8c0714f0 msgr2=0x7f3a8c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:52.900 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is laggy/crashed, in failed state, or mds.a is removed from mdsmap 2019-12-05T06:57:54.901 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:55.148 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.138+0000 7fcceae5e700 1 Processor -- start 2019-12-05T06:57:55.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.139+0000 7fcceae5e700 1 -- start start 2019-12-05T06:57:55.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.140+0000 7fcceae5e700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 0x7fcce408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:55.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.140+0000 7fcceae5e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce40958e0 con 0x7fcce408d2c0 2019-12-05T06:57:55.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.140+0000 7fcceae5e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce40629e0 con 0x7fcce408c6d0 2019-12-05T06:57:55.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.140+0000 7fcceae5e700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcce408df90 con 0x7fcce408fef0 2019-12-05T06:57:55.149 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.141+0000 7fcce8bfa700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcce408d2c0 0x7fcce408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50260/0 (socket says 172.21.15.16:50260) 2019-12-05T06:57:55.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.141+0000 7fcce8bfa700 1 -- 172.21.15.16:0/1157900213 learned_addr learned my addr 172.21.15.16:0/1157900213 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:55.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.141+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/1157900213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 0x7fcce408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:55.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.141+0000 7fcce3fff700 1 -- 172.21.15.16:0/1157900213 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508529826 0 0) 0x7fccd4001400 con 0x7fcce408c6d0 2019-12-05T06:57:55.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.141+0000 7fcce3fff700 1 -- 172.21.15.16:0/1157900213 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcccc002e80 con 0x7fcce408c6d0 2019-12-05T06:57:55.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.141+0000 7fcce3fff700 1 -- 172.21.15.16:0/1157900213 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227775651 0 0) 0x7fccd8001100 con 0x7fcce408d2c0 2019-12-05T06:57:55.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.141+0000 7fcce3fff700 1 -- 172.21.15.16:0/1157900213 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcccc004170 con 0x7fcce408d2c0 2019-12-05T06:57:55.150 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce9bfc700 1 -- 172.21.15.16:0/1157900213 >> v1:172.21.15.76:6789/0 conn(0x7fcce408c6d0 legacy=0x7fcce408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce9bfc700 1 -- 172.21.15.16:0/1157900213 >> v1:172.21.15.16:6789/0 conn(0x7fcce408d2c0 legacy=0x7fcce408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce9bfc700 1 -- 172.21.15.16:0/1157900213 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce408e620 con 0x7fcce408fef0 2019-12-05T06:57:55.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/1157900213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 0x7fcce408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fccdc00af80 tx=0x7fccdc00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/1157900213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 0x7fcce408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fccdc00af80 tx=0x7fccdc00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/1157900213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 0x7fcce408f720 secure :-1 s=READY pgs=392 cs=0 l=1 rx=0x7fccdc00af80 tx=0x7fccdc00b790).ready entity=mon.2 client_cookie=a5b5c89a6c35c354 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:55.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/1157900213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 0x7fcce408f720 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7fccdc00af80 tx=0x7fccdc00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.151 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/1157900213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 0x7fcce408f720 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7fccdc00af80 tx=0x7fccdc00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce3fff700 1 -- 172.21.15.16:0/1157900213 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccdc007240 con 0x7fcce408fef0 2019-12-05T06:57:55.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/1157900213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 0x7fcce408f720 secure :-1 s=THROTTLE_DONE pgs=392 cs=0 l=1 rx=0x7fccdc00af80 tx=0x7fccdc00b790).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.142+0000 7fcce3fff700 1 -- 172.21.15.16:0/1157900213 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc000c30 con 0x7fcce408fef0 2019-12-05T06:57:55.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.143+0000 7fcce3fff700 1 -- 172.21.15.16:0/1157900213 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccdc009f10 con 0x7fcce408fef0 2019-12-05T06:57:55.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.143+0000 7fcceae5e700 1 -- 172.21.15.16:0/1157900213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 msgr2=0x7fcce408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.152 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.143+0000 7fcceae5e700 1 --2- 172.21.15.16:0/1157900213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 0x7fcce408f720 secure :-1 s=READY pgs=392 cs=0 l=1 rx=0x7fccdc00af80 tx=0x7fccdc00b790).stop 2019-12-05T06:57:55.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.144+0000 7fcceae5e700 1 -- 172.21.15.16:0/1157900213 shutdown_connections 2019-12-05T06:57:55.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.144+0000 7fcceae5e700 1 --2- 172.21.15.16:0/1157900213 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fcce408fef0 0x7fcce408f720 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.145+0000 7fcceae5e700 1 -- 172.21.15.16:0/1157900213 shutdown_connections 2019-12-05T06:57:55.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.145+0000 7fcceae5e700 1 -- 172.21.15.16:0/1157900213 wait complete. 2019-12-05T06:57:55.153 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.145+0000 7fcceae5e700 1 -- 172.21.15.16:0/1157900213 >> 172.21.15.16:0/1157900213 conn(0x7fcce4061940 msgr2=0x7fcce4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:55.156 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.150+0000 7fcceae5e700 1 Processor -- start 2019-12-05T06:57:55.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.150+0000 7fcceae5e700 1 -- start start 2019-12-05T06:57:55.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.152+0000 7fcceae5e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fcce409dba0 0x7fcce414d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:55.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.152+0000 7fcceae5e700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce406e250 con 0x7fcce409e2c0 2019-12-05T06:57:55.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.152+0000 7fcceae5e700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcce4099460 con 0x7fcce414d5d0 2019-12-05T06:57:55.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.152+0000 7fcceae5e700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcce412b3f0 con 0x7fcce409dba0 2019-12-05T06:57:55.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.153+0000 7fcce8bfa700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fcce409e2c0 0x7fcce4153320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50268/0 (socket says 172.21.15.16:50268) 2019-12-05T06:57:55.157 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.153+0000 7fcce8bfa700 1 -- 172.21.15.16:0/481735018 learned_addr learned my addr 172.21.15.16:0/481735018 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:55.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.153+0000 7fcce93fb700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fcce414d5d0 0x7fcce409d3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:43336/0 (socket says 172.21.15.16:43336) 2019-12-05T06:57:55.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.153+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> v2:172.21.15.16:3300/0 conn(0x7fcce409dba0 0x7fcce414d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:55.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.153+0000 7fcce1ffb700 1 -- 172.21.15.16:0/481735018 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1397212857 0 0) 0x7fccd4001230 con 0x7fcce414d5d0 2019-12-05T06:57:55.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.153+0000 7fcce1ffb700 1 -- 172.21.15.16:0/481735018 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fccc4002e80 con 0x7fcce414d5d0 2019-12-05T06:57:55.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.153+0000 7fcce9bfc700 1 -- 172.21.15.16:0/481735018 >> v1:172.21.15.76:6789/0 conn(0x7fcce414d5d0 legacy=0x7fcce409d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.153+0000 7fcce9bfc700 1 -- 172.21.15.16:0/481735018 >> v1:172.21.15.16:6789/0 conn(0x7fcce409e2c0 legacy=0x7fcce4153320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.158 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.153+0000 7fcce1ffb700 1 -- 172.21.15.16:0/481735018 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792543683 0 0) 0x7fccd8000f80 con 0x7fcce409e2c0 2019-12-05T06:57:55.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.153+0000 7fcce9bfc700 1 -- 172.21.15.16:0/481735018 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcce412ba80 con 0x7fcce409dba0 2019-12-05T06:57:55.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> v2:172.21.15.16:3300/0 conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> v2:172.21.15.16:3300/0 conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:55.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.159 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcce1ffb700 1 -- 172.21.15.16:0/481735018 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccdc0099b0 con 0x7fcce409dba0 2019-12-05T06:57:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcce1ffb700 1 -- 172.21.15.16:0/481735018 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fccdc009bb0 con 0x7fcce409dba0 2019-12-05T06:57:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcce1ffb700 1 -- 172.21.15.16:0/481735018 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fccdc009da0 con 0x7fcce409dba0 2019-12-05T06:57:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcceae5e700 1 -- 172.21.15.16:0/481735018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcce412ba80 con 0x7fcce409dba0 2019-12-05T06:57:55.160 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.154+0000 7fcceae5e700 1 -- 172.21.15.16:0/481735018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcce4099460 con 0x7fcce409dba0 2019-12-05T06:57:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.155+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.156+0000 7fcce1ffb700 1 -- 172.21.15.16:0/481735018 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fccdc009e40 con 0x7fcce409dba0 2019-12-05T06:57:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.156+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.156+0000 7fcce1ffb700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fccc4023680 0x7fccc4025b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:55.161 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.157+0000 7fcce1ffb700 1 -- 172.21.15.16:0/481735018 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7fccdc046070 con 0x7fcce409dba0 2019-12-05T06:57:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.157+0000 7fcce93fb700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fccc4023680 0x7fccc4025b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:55.162 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.157+0000 7fcce93fb700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fccc4023680 0x7fccc4025b10 crc :-1 s=READY pgs=697 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:55.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.170+0000 7fcceae5e700 1 -- 172.21.15.16:0/481735018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcce412ba80 con 0x7fcce409dba0 2019-12-05T06:57:55.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.177+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.177+0000 7fcce1ffb700 1 -- 172.21.15.16:0/481735018 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fccdc00a330 con 0x7fcce409dba0 2019-12-05T06:57:55.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.422+0000 7fcceae5e700 1 -- 172.21.15.16:0/481735018 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcce412ba80 con 0x7fcce409dba0 2019-12-05T06:57:55.424 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.423+0000 7fcce9bfc700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=THROTTLE_DONE pgs=1667 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.423+0000 7fcce1ffb700 1 -- 172.21.15.16:0/481735018 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+1771 (secure 0 0 0) 0x7fccdc009bb0 con 0x7fcce409dba0 2019-12-05T06:57:55.425 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 203 2019-12-05T06:57:55.426 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:55.426 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:53.649148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:replay","state_seq":1,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:55.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.425+0000 7fcceae5e700 1 -- 172.21.15.16:0/481735018 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fccc4023680 msgr2=0x7fccc4025b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.425+0000 7fcceae5e700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fccc4023680 0x7fccc4025b10 crc :-1 s=READY pgs=697 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.425+0000 7fcceae5e700 1 -- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 msgr2=0x7fcce414d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.425+0000 7fcceae5e700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 0x7fcce414d9c0 secure :-1 s=READY pgs=1667 cs=0 l=1 rx=0x7fccdc0103c0 tx=0x7fccdc010760).stop 2019-12-05T06:57:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.426+0000 7fcceae5e700 1 -- 172.21.15.16:0/481735018 shutdown_connections 2019-12-05T06:57:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.426+0000 7fcceae5e700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fccc4023680 0x7fccc4025b10 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.426+0000 7fcceae5e700 1 --2- 172.21.15.16:0/481735018 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fcce409dba0 0x7fcce414d9c0 unknown :-1 s=CLOSED pgs=1667 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.427 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.426+0000 7fcceae5e700 1 -- 172.21.15.16:0/481735018 shutdown_connections 2019-12-05T06:57:55.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.426+0000 7fcceae5e700 1 -- 172.21.15.16:0/481735018 wait complete. 2019-12-05T06:57:55.428 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.426+0000 7fcceae5e700 1 -- 172.21.15.16:0/481735018 >> 172.21.15.16:0/481735018 conn(0x7fcce406d260 msgr2=0x7fcce409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:55.437 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a down, removed from mdsmap 2019-12-05T06:57:55.437 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:57:55.437 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:55.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.647+0000 7f393873b700 1 Processor -- start 2019-12-05T06:57:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.647+0000 7f393873b700 1 -- start start 2019-12-05T06:57:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.649+0000 7f393873b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f393008d2c0 0x7f393008ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.649+0000 7f393873b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39300958e0 con 0x7f393008fef0 2019-12-05T06:57:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.649+0000 7f393873b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39300629e0 con 0x7f393008c6d0 2019-12-05T06:57:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.649+0000 7f393873b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f393008df90 con 0x7f393008d2c0 2019-12-05T06:57:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.649+0000 7f39364d7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f393008d2c0 0x7f393008ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:55.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.649+0000 7f39364d7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f393008d2c0 0x7f393008ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36876/0 (socket says 172.21.15.16:36876) 2019-12-05T06:57:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.649+0000 7f39364d7700 1 -- 172.21.15.16:0/1629185890 learned_addr learned my addr 172.21.15.16:0/1629185890 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:57:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.650+0000 7f3935cd6700 1 -- 172.21.15.16:0/1629185890 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4000593616 0 0) 0x7f3920001080 con 0x7f393008c6d0 2019-12-05T06:57:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.650+0000 7f39364d7700 1 -- 172.21.15.16:0/1629185890 >> v1:172.21.15.76:6789/0 conn(0x7f393008c6d0 legacy=0x7f393008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.650+0000 7f39364d7700 1 -- 172.21.15.16:0/1629185890 >> v1:172.21.15.16:6789/0 conn(0x7f393008fef0 legacy=0x7f393008f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.650+0000 7f39364d7700 1 -- 172.21.15.16:0/1629185890 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393008e620 con 0x7f393008d2c0 2019-12-05T06:57:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.650+0000 7f39364d7700 1 --2- 172.21.15.16:0/1629185890 >> v2:172.21.15.16:3300/0 conn(0x7f393008d2c0 0x7f393008ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f392400fac0 tx=0x7f392400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.650+0000 7f39364d7700 1 --2- 172.21.15.16:0/1629185890 >> v2:172.21.15.16:3300/0 conn(0x7f393008d2c0 0x7f393008ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f392400fac0 tx=0x7f392400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.650+0000 7f39364d7700 1 --2- 172.21.15.16:0/1629185890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393008d2c0 0x7f393008ef20 secure :-1 s=READY pgs=1669 cs=0 l=1 rx=0x7f392400fac0 tx=0x7f392400fe60).ready entity=mon.0 client_cookie=13128f2c02a13bef server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.651+0000 7f39364d7700 1 --2- 172.21.15.16:0/1629185890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393008d2c0 0x7f393008ef20 secure :-1 s=THROTTLE_DONE pgs=1669 cs=0 l=1 rx=0x7f392400fac0 tx=0x7f392400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.651+0000 7f3935cd6700 1 -- 172.21.15.16:0/1629185890 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3924000ce0 con 0x7f393008d2c0 2019-12-05T06:57:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.651+0000 7f39364d7700 1 --2- 172.21.15.16:0/1629185890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393008d2c0 0x7f393008ef20 secure :-1 s=THROTTLE_DONE pgs=1669 cs=0 l=1 rx=0x7f392400fac0 tx=0x7f392400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.651+0000 7f39364d7700 1 --2- 172.21.15.16:0/1629185890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393008d2c0 0x7f393008ef20 secure :-1 s=THROTTLE_DONE pgs=1669 cs=0 l=1 rx=0x7f392400fac0 tx=0x7f392400fe60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.651+0000 7f3935cd6700 1 -- 172.21.15.16:0/1629185890 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3924001010 con 0x7f393008d2c0 2019-12-05T06:57:55.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.651+0000 7f3935cd6700 1 -- 172.21.15.16:0/1629185890 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39240016e0 con 0x7f393008d2c0 2019-12-05T06:57:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.651+0000 7f393873b700 1 -- 172.21.15.16:0/1629185890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393008d2c0 msgr2=0x7f393008ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.651+0000 7f393873b700 1 --2- 172.21.15.16:0/1629185890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393008d2c0 0x7f393008ef20 secure :-1 s=READY pgs=1669 cs=0 l=1 rx=0x7f392400fac0 tx=0x7f392400fe60).stop 2019-12-05T06:57:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.653+0000 7f393873b700 1 -- 172.21.15.16:0/1629185890 shutdown_connections 2019-12-05T06:57:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.653+0000 7f393873b700 1 --2- 172.21.15.16:0/1629185890 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f393008d2c0 0x7f393008ef20 unknown :-1 s=CLOSED pgs=1669 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.653+0000 7f393873b700 1 -- 172.21.15.16:0/1629185890 shutdown_connections 2019-12-05T06:57:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.653+0000 7f393873b700 1 -- 172.21.15.16:0/1629185890 wait complete. 2019-12-05T06:57:55.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.653+0000 7f393873b700 1 -- 172.21.15.16:0/1629185890 >> 172.21.15.16:0/1629185890 conn(0x7f3930061940 msgr2=0x7f3930096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:55.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.657+0000 7f393873b700 1 Processor -- start 2019-12-05T06:57:55.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.657+0000 7f393873b700 1 -- start start 2019-12-05T06:57:55.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.658+0000 7f393873b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:55.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.658+0000 7f393873b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f39300879b0 0x7f3930086f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.658+0000 7f393873b700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3930087460 0x7f393009be80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.658+0000 7f393873b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3930085020 con 0x7f39300879b0 2019-12-05T06:57:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.658+0000 7f393873b700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f39300852f0 con 0x7f3930087460 2019-12-05T06:57:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.658+0000 7f393873b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f39300860e0 con 0x7f393014ca10 2019-12-05T06:57:55.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.658+0000 7f39364d7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f39300879b0 0x7f3930086f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:55.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f3936cd8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:55.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f39364d7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f39300879b0 0x7f3930086f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36882/0 (socket says 172.21.15.16:36882) 2019-12-05T06:57:55.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f39364d7700 1 -- 172.21.15.16:0/133650783 learned_addr learned my addr 172.21.15.16:0/133650783 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:57:55.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f39374d9700 1 --2- 172.21.15.16:0/133650783 >> v2:172.21.15.76:3300/0 conn(0x7f3930087460 0x7f393009be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:55.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f3936cd8700 1 -- 172.21.15.16:0/133650783 >> v2:172.21.15.76:3300/0 conn(0x7f3930087460 msgr2=0x7f393009be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> v2:172.21.15.76:3300/0 conn(0x7f3930087460 0x7f393009be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.666 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f3936cd8700 1 -- 172.21.15.16:0/133650783 >> v2:172.21.15.16:3300/0 conn(0x7f39300879b0 msgr2=0x7f3930086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> v2:172.21.15.16:3300/0 conn(0x7f39300879b0 0x7f3930086f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f3936cd8700 1 -- 172.21.15.16:0/133650783 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f393009e030 con 0x7f393014ca10 2019-12-05T06:57:55.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f39364d7700 1 --2- 172.21.15.16:0/133650783 >> v2:172.21.15.16:3300/0 conn(0x7f39300879b0 0x7f3930086f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:57:55.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.659+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.660+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.660+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=READY pgs=393 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:55.667 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.660+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.660+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.660+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.660+0000 7f392f7fe700 1 -- 172.21.15.16:0/133650783 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3920000af0 con 0x7f393014ca10 2019-12-05T06:57:55.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.660+0000 7f392f7fe700 1 -- 172.21.15.16:0/133650783 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3920009980 con 0x7f393014ca10 2019-12-05T06:57:55.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.660+0000 7f392f7fe700 1 -- 172.21.15.16:0/133650783 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f392000a0d0 con 0x7f393014ca10 2019-12-05T06:57:55.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.660+0000 7f393873b700 1 -- 172.21.15.16:0/133650783 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f393009e030 con 0x7f393014ca10 2019-12-05T06:57:55.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.660+0000 7f393873b700 1 -- 172.21.15.16:0/133650783 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f393009e030 con 0x7f393014ca10 2019-12-05T06:57:55.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.661+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.662+0000 7f392f7fe700 1 -- 172.21.15.16:0/133650783 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3920000ad0 con 0x7f393014ca10 2019-12-05T06:57:55.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.663+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.663+0000 7f392f7fe700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3910025840 0x7f3910027cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:55.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.663+0000 7f392f7fe700 1 -- 172.21.15.16:0/133650783 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f3920045cc0 con 0x7f393014ca10 2019-12-05T06:57:55.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.663+0000 7f39364d7700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3910025840 0x7f3910027cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:55.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.664+0000 7f39364d7700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3910025840 0x7f3910027cd0 crc :-1 s=READY pgs=698 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:55.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.664+0000 7f393873b700 1 -- 172.21.15.16:0/133650783 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39300860e0 con 0x7f393014ca10 2019-12-05T06:57:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.674+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.674 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.674+0000 7f392f7fe700 1 -- 172.21.15.16:0/133650783 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f39200056f0 con 0x7f393014ca10 2019-12-05T06:57:55.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.920+0000 7f393873b700 1 -- 172.21.15.16:0/133650783 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f39300860e0 con 0x7f393014ca10 2019-12-05T06:57:55.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.921+0000 7f3936cd8700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=THROTTLE_DONE pgs=393 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:55.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.921+0000 7f392f7fe700 1 -- 172.21.15.16:0/133650783 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 203 v203) v1 ==== 95+0+1771 (secure 0 0 0) 0x7f39200056f0 con 0x7f393014ca10 2019-12-05T06:57:55.923 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:55.923 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":203,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":203,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:53.649148+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:replay","state_seq":1,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:55.924 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 203 2019-12-05T06:57:55.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 -- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3910025840 msgr2=0x7f3910027cd0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3910025840 0x7f3910027cd0 crc :-1 s=READY pgs=698 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 -- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 msgr2=0x7f39300869e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:55.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 secure :-1 s=READY pgs=393 cs=0 l=1 rx=0x7f39200103a0 tx=0x7f392000b2e0).stop 2019-12-05T06:57:55.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 -- 172.21.15.16:0/133650783 shutdown_connections 2019-12-05T06:57:55.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 --2- 172.21.15.16:0/133650783 >> v2:172.21.15.76:3300/0 conn(0x7f3930087460 0x7f393009be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3910025840 0x7f3910027cd0 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 --2- 172.21.15.16:0/133650783 >> v2:172.21.15.16:3300/0 conn(0x7f39300879b0 0x7f3930086f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 --2- 172.21.15.16:0/133650783 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f393014ca10 0x7f39300869e0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:55.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 -- 172.21.15.16:0/133650783 shutdown_connections 2019-12-05T06:57:55.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 -- 172.21.15.16:0/133650783 wait complete. 2019-12-05T06:57:55.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:55.926+0000 7f393873b700 1 -- 172.21.15.16:0/133650783 >> 172.21.15.16:0/133650783 conn(0x7f3930098480 msgr2=0x7f3930071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:55.944 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 203, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 203, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8430": { "addr": "172.21.15.76:6835/448242135", "addrs": { "addrvec": [ { "addr": "172.21.15.76:6834", "nonce": 448242135, "type": "v2" }, { "addr": "172.21.15.76:6835", "nonce": 448242135, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8430, "incarnation": 203, "name": "b", "rank": 0, "state": "up:replay", "state_seq": 1 } }, "last_failure": 0, "last_failure_osd_epoch": 84, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:57:53.649148+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8430 } } } ], "standbys": [] } 2019-12-05T06:57:57.945 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:57:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.228+0000 7f6e01c38700 1 Processor -- start 2019-12-05T06:57:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.228+0000 7f6e01c38700 1 -- start start 2019-12-05T06:57:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.230+0000 7f6e01c38700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 0x7f6dfc08cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:57:58.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.230+0000 7f6e01c38700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc0958e0 con 0x7f6dfc08fef0 2019-12-05T06:57:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.230+0000 7f6e01c38700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc0629e0 con 0x7f6dfc08f720 2019-12-05T06:57:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.230+0000 7f6e01c38700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dfc0a3f70 con 0x7f6dfc08c6d0 2019-12-05T06:57:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.231+0000 7f6dfb7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6dfc08fef0 0x7f6dfc08ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50292/0 (socket says 172.21.15.16:50292) 2019-12-05T06:57:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.231+0000 7f6dfb7fe700 1 -- 172.21.15.16:0/1979591301 learned_addr learned my addr 172.21.15.16:0/1979591301 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.231+0000 7f6dfbfff700 1 --2- 172.21.15.16:0/1979591301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 0x7f6dfc08cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:58.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.231+0000 7f6dfaffd700 1 -- 172.21.15.16:0/1979591301 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3875810390 0 0) 0x7f6df40009b0 con 0x7f6dfc08f720 2019-12-05T06:57:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.231+0000 7f6dfaffd700 1 -- 172.21.15.16:0/1979591301 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de8002e80 con 0x7f6dfc08f720 2019-12-05T06:57:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.231+0000 7f6dfaffd700 1 -- 172.21.15.16:0/1979591301 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3184243551 0 0) 0x7f6de4001100 con 0x7f6dfc08fef0 2019-12-05T06:57:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.231+0000 7f6dfaffd700 1 -- 172.21.15.16:0/1979591301 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de8002e80 con 0x7f6dfc08fef0 2019-12-05T06:57:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.232+0000 7f6dfbfff700 1 -- 172.21.15.16:0/1979591301 >> v1:172.21.15.76:6789/0 conn(0x7f6dfc08f720 legacy=0x7f6dfc1533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.232+0000 7f6dfbfff700 1 -- 172.21.15.16:0/1979591301 >> v1:172.21.15.16:6789/0 conn(0x7f6dfc08fef0 legacy=0x7f6dfc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.232+0000 7f6dfbfff700 1 -- 172.21.15.16:0/1979591301 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc08fbc0 con 0x7f6dfc08c6d0 2019-12-05T06:57:58.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.232+0000 7f6dfbfff700 1 --2- 172.21.15.16:0/1979591301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 0x7f6dfc08cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6dec00b2d0 tx=0x7f6dec00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.232+0000 7f6dfbfff700 1 --2- 172.21.15.16:0/1979591301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 0x7f6dfc08cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6dec00b2d0 tx=0x7f6dec00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.232+0000 7f6dfbfff700 1 --2- 172.21.15.16:0/1979591301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 0x7f6dfc08cac0 secure :-1 s=READY pgs=394 cs=0 l=1 rx=0x7f6dec00b2d0 tx=0x7f6dec00bae0).ready entity=mon.2 client_cookie=daa12368a7b6681a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.232+0000 7f6dfbfff700 1 --2- 172.21.15.16:0/1979591301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 0x7f6dfc08cac0 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f6dec00b2d0 tx=0x7f6dec00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.233+0000 7f6dfaffd700 1 -- 172.21.15.16:0/1979591301 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dec000ce0 con 0x7f6dfc08c6d0 2019-12-05T06:57:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.233+0000 7f6dfbfff700 1 --2- 172.21.15.16:0/1979591301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 0x7f6dfc08cac0 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f6dec00b2d0 tx=0x7f6dec00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.233+0000 7f6dfbfff700 1 --2- 172.21.15.16:0/1979591301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 0x7f6dfc08cac0 secure :-1 s=THROTTLE_DONE pgs=394 cs=0 l=1 rx=0x7f6dec00b2d0 tx=0x7f6dec00bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.233+0000 7f6dfaffd700 1 -- 172.21.15.16:0/1979591301 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dec009be0 con 0x7f6dfc08c6d0 2019-12-05T06:57:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.233+0000 7f6dfaffd700 1 -- 172.21.15.16:0/1979591301 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dec00a2b0 con 0x7f6dfc08c6d0 2019-12-05T06:57:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.233+0000 7f6e01c38700 1 -- 172.21.15.16:0/1979591301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 msgr2=0x7f6dfc08cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.233+0000 7f6e01c38700 1 --2- 172.21.15.16:0/1979591301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 0x7f6dfc08cac0 secure :-1 s=READY pgs=394 cs=0 l=1 rx=0x7f6dec00b2d0 tx=0x7f6dec00bae0).stop 2019-12-05T06:57:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.235+0000 7f6e01c38700 1 -- 172.21.15.16:0/1979591301 shutdown_connections 2019-12-05T06:57:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.235+0000 7f6e01c38700 1 --2- 172.21.15.16:0/1979591301 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6dfc08c6d0 0x7f6dfc08cac0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.235+0000 7f6e01c38700 1 -- 172.21.15.16:0/1979591301 shutdown_connections 2019-12-05T06:57:58.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.235+0000 7f6e01c38700 1 -- 172.21.15.16:0/1979591301 wait complete. 2019-12-05T06:57:58.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.235+0000 7f6e01c38700 1 -- 172.21.15.16:0/1979591301 >> 172.21.15.16:0/1979591301 conn(0x7f6dfc061940 msgr2=0x7f6dfc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:58.246 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.240+0000 7f6e01c38700 1 Processor -- start 2019-12-05T06:57:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.240+0000 7f6e01c38700 1 -- start start 2019-12-05T06:57:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.242+0000 7f6e01c38700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f6dfc19b600 0x7f6dfc142050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.242+0000 7f6e01c38700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc12ace0 con 0x7f6dfc142820 2019-12-05T06:57:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.242+0000 7f6e01c38700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dfc12a7e0 con 0x7f6dfc19b210 2019-12-05T06:57:58.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.242+0000 7f6e01c38700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dfc19ba60 con 0x7f6dfc19b600 2019-12-05T06:57:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.242+0000 7f6dfb7fe700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6dfc142820 0x7f6dfc19fe50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50298/0 (socket says 172.21.15.16:50298) 2019-12-05T06:57:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.242+0000 7f6dfb7fe700 1 -- 172.21.15.16:0/2637357942 learned_addr learned my addr 172.21.15.16:0/2637357942 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:57:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.243+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> v2:172.21.15.76:3300/0 conn(0x7f6dfc19b600 0x7f6dfc142050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.243+0000 7f6df8ff9700 1 -- 172.21.15.16:0/2637357942 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3387536081 0 0) 0x7f6de4000d20 con 0x7f6dfc142820 2019-12-05T06:57:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.243+0000 7f6df8ff9700 1 -- 172.21.15.16:0/2637357942 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6de0002e80 con 0x7f6dfc142820 2019-12-05T06:57:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.243+0000 7f6e009d6700 1 -- 172.21.15.16:0/2637357942 >> v1:172.21.15.76:6789/0 conn(0x7f6dfc19b210 legacy=0x7f6dfc141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:58.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.243+0000 7f6df8ff9700 1 -- 172.21.15.16:0/2637357942 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3345659820 0 0) 0x7f6dec00b750 con 0x7f6dfc19b210 2019-12-05T06:57:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.243+0000 7f6e009d6700 1 -- 172.21.15.16:0/2637357942 >> v1:172.21.15.16:6789/0 conn(0x7f6dfc142820 legacy=0x7f6dfc19fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.243+0000 7f6e009d6700 1 -- 172.21.15.16:0/2637357942 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dfc140f50 con 0x7f6dfc19b600 2019-12-05T06:57:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> v2:172.21.15.76:3300/0 conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> v2:172.21.15.76:3300/0 conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=READY pgs=1134 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=THROTTLE_DONE pgs=1134 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.249 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6df8ff9700 1 -- 172.21.15.16:0/2637357942 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6df40099d0 con 0x7f6dfc19b600 2019-12-05T06:57:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=THROTTLE_DONE pgs=1134 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=THROTTLE_DONE pgs=1134 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6df8ff9700 1 -- 172.21.15.16:0/2637357942 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6df4009bd0 con 0x7f6dfc19b600 2019-12-05T06:57:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6e01c38700 1 -- 172.21.15.16:0/2637357942 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dfc140f50 con 0x7f6dfc19b600 2019-12-05T06:57:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6df8ff9700 1 -- 172.21.15.16:0/2637357942 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6df400a320 con 0x7f6dfc19b600 2019-12-05T06:57:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.244+0000 7f6e01c38700 1 -- 172.21.15.16:0/2637357942 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dfc140f50 con 0x7f6dfc19b600 2019-12-05T06:57:58.250 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.245+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=THROTTLE_DONE pgs=1134 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.246+0000 7f6df8ff9700 1 -- 172.21.15.16:0/2637357942 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6df40095d0 con 0x7f6dfc19b600 2019-12-05T06:57:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.246+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=THROTTLE_DONE pgs=1134 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.246+0000 7f6df8ff9700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6de0023680 0x7f6de0025af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:57:58.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.247+0000 7f6df8ff9700 1 -- 172.21.15.16:0/2637357942 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f6df4045da0 con 0x7f6dfc19b600 2019-12-05T06:57:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.247+0000 7f6dfbfff700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6de0023680 0x7f6de0025af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:57:58.252 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.247+0000 7f6dfbfff700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6de0023680 0x7f6de0025af0 crc :-1 s=READY pgs=699 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:57:58.259 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.259+0000 7f6e01c38700 1 -- 172.21.15.16:0/2637357942 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dfc140f50 con 0x7f6dfc19b600 2019-12-05T06:57:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.269+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=THROTTLE_DONE pgs=1134 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.269 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.269+0000 7f6df8ff9700 1 -- 172.21.15.16:0/2637357942 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6df4009fd0 con 0x7f6dfc19b600 2019-12-05T06:57:58.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.512+0000 7f6e01c38700 1 -- 172.21.15.16:0/2637357942 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dfc140f50 con 0x7f6dfc19b600 2019-12-05T06:57:58.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.513+0000 7f6e009d6700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=THROTTLE_DONE pgs=1134 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:57:58.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.513+0000 7f6df8ff9700 1 -- 172.21.15.16:0/2637357942 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 205 v205) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f6df40095d0 con 0x7f6dfc19b600 2019-12-05T06:57:58.515 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:57:58.516 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":205,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":205,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:57.671201+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:rejoin","state_seq":16,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:57:58.516 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 205 2019-12-05T06:57:58.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.517+0000 7f6e01c38700 1 -- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6de0023680 msgr2=0x7f6de0025af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:58.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.517+0000 7f6e01c38700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6de0023680 0x7f6de0025af0 crc :-1 s=READY pgs=699 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:58.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.517+0000 7f6e01c38700 1 -- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 msgr2=0x7f6dfc142050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:57:58.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.517+0000 7f6e01c38700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 0x7f6dfc142050 secure :-1 s=READY pgs=1134 cs=0 l=1 rx=0x7f6df40103a0 tx=0x7f6df4010740).stop 2019-12-05T06:57:58.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.517+0000 7f6e01c38700 1 -- 172.21.15.16:0/2637357942 shutdown_connections 2019-12-05T06:57:58.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.518+0000 7f6e01c38700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6de0023680 0x7f6de0025af0 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:58.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.518+0000 7f6e01c38700 1 --2- 172.21.15.16:0/2637357942 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f6dfc19b600 0x7f6dfc142050 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:57:58.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.518+0000 7f6e01c38700 1 -- 172.21.15.16:0/2637357942 shutdown_connections 2019-12-05T06:57:58.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.518+0000 7f6e01c38700 1 -- 172.21.15.16:0/2637357942 wait complete. 2019-12-05T06:57:58.519 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:57:58.518+0000 7f6e01c38700 1 -- 172.21.15.16:0/2637357942 >> 172.21.15.16:0/2637357942 conn(0x7f6dfc0829c0 msgr2=0x7f6dfc098960 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:57:58.529 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:58:00.530 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:58:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.708+0000 7f3d7fb91700 1 Processor -- start 2019-12-05T06:58:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.708+0000 7f3d7fb91700 1 -- start start 2019-12-05T06:58:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.709+0000 7f3d7fb91700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3d7808d2c0 0x7f3d7808ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:58:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.709+0000 7f3d7fb91700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d780958e0 con 0x7f3d7808c6d0 2019-12-05T06:58:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.709+0000 7f3d7fb91700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d780629e0 con 0x7f3d7808fef0 2019-12-05T06:58:00.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.709+0000 7f3d7fb91700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d7808df90 con 0x7f3d7808d2c0 2019-12-05T06:58:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7e12e700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f3d7808c6d0 0x7f3d7808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50320/0 (socket says 172.21.15.16:50320) 2019-12-05T06:58:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7e12e700 1 -- 172.21.15.16:0/3438921524 learned_addr learned my addr 172.21.15.16:0/3438921524 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:58:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d92d700 1 --2- 172.21.15.16:0/3438921524 >> v2:172.21.15.76:3300/0 conn(0x7f3d7808d2c0 0x7f3d7808ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d12c700 1 -- 172.21.15.16:0/3438921524 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906809058 0 0) 0x7f3d74001150 con 0x7f3d7808c6d0 2019-12-05T06:58:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d12c700 1 -- 172.21.15.16:0/3438921524 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d64002e80 con 0x7f3d7808c6d0 2019-12-05T06:58:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d12c700 1 -- 172.21.15.16:0/3438921524 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1211230456 0 0) 0x7f3d70000d30 con 0x7f3d7808fef0 2019-12-05T06:58:00.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d12c700 1 -- 172.21.15.16:0/3438921524 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d640030b0 con 0x7f3d7808fef0 2019-12-05T06:58:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d92d700 1 -- 172.21.15.16:0/3438921524 >> v1:172.21.15.76:6789/0 conn(0x7f3d7808fef0 legacy=0x7f3d7808f460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d12c700 1 -- 172.21.15.16:0/3438921524 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1217357326 0 0) 0x7f3d74001130 con 0x7f3d7808c6d0 2019-12-05T06:58:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d92d700 1 -- 172.21.15.16:0/3438921524 >> v1:172.21.15.16:6789/0 conn(0x7f3d7808c6d0 legacy=0x7f3d7808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d92d700 1 -- 172.21.15.16:0/3438921524 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d7808e620 con 0x7f3d7808d2c0 2019-12-05T06:58:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d92d700 1 --2- 172.21.15.16:0/3438921524 >> v2:172.21.15.76:3300/0 conn(0x7f3d7808d2c0 0x7f3d7808ef20 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f3d6800af50 tx=0x7f3d6800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d92d700 1 --2- 172.21.15.16:0/3438921524 >> v2:172.21.15.76:3300/0 conn(0x7f3d7808d2c0 0x7f3d7808ef20 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f3d6800af50 tx=0x7f3d6800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.710+0000 7f3d7d92d700 1 --2- 172.21.15.16:0/3438921524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d7808d2c0 0x7f3d7808ef20 secure :-1 s=READY pgs=1136 cs=0 l=1 rx=0x7f3d6800af50 tx=0x7f3d6800b760).ready entity=mon.1 client_cookie=3b8fb37f4064e1c0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.711+0000 7f3d7d92d700 1 --2- 172.21.15.16:0/3438921524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d7808d2c0 0x7f3d7808ef20 secure :-1 s=THROTTLE_DONE pgs=1136 cs=0 l=1 rx=0x7f3d6800af50 tx=0x7f3d6800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.711+0000 7f3d7d12c700 1 -- 172.21.15.16:0/3438921524 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d68000960 con 0x7f3d7808d2c0 2019-12-05T06:58:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.711+0000 7f3d7d92d700 1 --2- 172.21.15.16:0/3438921524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d7808d2c0 0x7f3d7808ef20 secure :-1 s=THROTTLE_DONE pgs=1136 cs=0 l=1 rx=0x7f3d6800af50 tx=0x7f3d6800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.717 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.711+0000 7f3d7d92d700 1 --2- 172.21.15.16:0/3438921524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d7808d2c0 0x7f3d7808ef20 secure :-1 s=THROTTLE_DONE pgs=1136 cs=0 l=1 rx=0x7f3d6800af50 tx=0x7f3d6800b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.711+0000 7f3d7d12c700 1 -- 172.21.15.16:0/3438921524 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d68000940 con 0x7f3d7808d2c0 2019-12-05T06:58:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.711+0000 7f3d7d12c700 1 -- 172.21.15.16:0/3438921524 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d68009ba0 con 0x7f3d7808d2c0 2019-12-05T06:58:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.711+0000 7f3d7fb91700 1 -- 172.21.15.16:0/3438921524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d7808d2c0 msgr2=0x7f3d7808ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:00.718 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.711+0000 7f3d7fb91700 1 --2- 172.21.15.16:0/3438921524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d7808d2c0 0x7f3d7808ef20 secure :-1 s=READY pgs=1136 cs=0 l=1 rx=0x7f3d6800af50 tx=0x7f3d6800b760).stop 2019-12-05T06:58:00.719 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.712+0000 7f3d7fb91700 1 -- 172.21.15.16:0/3438921524 shutdown_connections 2019-12-05T06:58:00.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.712+0000 7f3d7fb91700 1 --2- 172.21.15.16:0/3438921524 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f3d7808d2c0 0x7f3d7808ef20 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:00.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.712+0000 7f3d7fb91700 1 -- 172.21.15.16:0/3438921524 shutdown_connections 2019-12-05T06:58:00.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.712+0000 7f3d7fb91700 1 -- 172.21.15.16:0/3438921524 wait complete. 2019-12-05T06:58:00.720 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.712+0000 7f3d7fb91700 1 -- 172.21.15.16:0/3438921524 >> 172.21.15.16:0/3438921524 conn(0x7f3d78061940 msgr2=0x7f3d78096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.714+0000 7f3d7fb91700 1 Processor -- start 2019-12-05T06:58:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.714+0000 7f3d7fb91700 1 -- start start 2019-12-05T06:58:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.715+0000 7f3d7fb91700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f3d7806e840 0x7f3d78134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:00.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.715+0000 7f3d7fb91700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3d78135450 0x7f3d781349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:00.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.715+0000 7f3d7fb91700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d78128af0 con 0x7f3d7806dc50 2019-12-05T06:58:00.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.715+0000 7f3d7fb91700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d78132830 con 0x7f3d78135450 2019-12-05T06:58:00.722 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.715+0000 7f3d7fb91700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d781334f0 con 0x7f3d7806e840 2019-12-05T06:58:00.724 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.715+0000 7f3d7e92f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3d78135450 0x7f3d781349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.715+0000 7f3d7e92f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f3d78135450 0x7f3d781349c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36928/0 (socket says 172.21.15.16:36928) 2019-12-05T06:58:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.715+0000 7f3d7e92f700 1 -- 172.21.15.16:0/2315397171 learned_addr learned my addr 172.21.15.16:0/2315397171 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:58:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.715+0000 7f3d7d92d700 1 --2- 172.21.15.16:0/2315397171 >> v2:172.21.15.76:3300/0 conn(0x7f3d7806e840 0x7f3d78134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7e92f700 1 -- 172.21.15.16:0/2315397171 >> v2:172.21.15.76:3300/0 conn(0x7f3d7806e840 msgr2=0x7f3d78134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> v2:172.21.15.76:3300/0 conn(0x7f3d7806e840 0x7f3d78134480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7e92f700 1 -- 172.21.15.16:0/2315397171 >> v1:172.21.15.16:6789/0 conn(0x7f3d7806dc50 legacy=0x7f3d7806e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:00.725 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7e92f700 1 -- 172.21.15.16:0/2315397171 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d78133b80 con 0x7f3d78135450 2019-12-05T06:58:00.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> v2:172.21.15.16:3300/0 conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> v2:172.21.15.16:3300/0 conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=READY pgs=1675 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:00.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=THROTTLE_DONE pgs=1675 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d6effd700 1 -- 172.21.15.16:0/2315397171 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d70005af0 con 0x7f3d78135450 2019-12-05T06:58:00.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=THROTTLE_DONE pgs=1675 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=THROTTLE_DONE pgs=1675 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d6effd700 1 -- 172.21.15.16:0/2315397171 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d70005710 con 0x7f3d78135450 2019-12-05T06:58:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7fb91700 1 -- 172.21.15.16:0/2315397171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d78133b80 con 0x7f3d78135450 2019-12-05T06:58:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d6effd700 1 -- 172.21.15.16:0/2315397171 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d700060d0 con 0x7f3d78135450 2019-12-05T06:58:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.716+0000 7f3d7fb91700 1 -- 172.21.15.16:0/2315397171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d78133b80 con 0x7f3d78135450 2019-12-05T06:58:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.717+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=THROTTLE_DONE pgs=1675 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.717+0000 7f3d6effd700 1 -- 172.21.15.16:0/2315397171 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f3d70005900 con 0x7f3d78135450 2019-12-05T06:58:00.727 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.717+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=THROTTLE_DONE pgs=1675 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.717+0000 7f3d6effd700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3d60023630 0x7f3d60025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.717+0000 7f3d6effd700 1 -- 172.21.15.16:0/2315397171 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f3d70046140 con 0x7f3d78135450 2019-12-05T06:58:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.718+0000 7f3d7e12e700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3d60023630 0x7f3d60025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.718+0000 7f3d7e12e700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3d60023630 0x7f3d60025ac0 crc :-1 s=READY pgs=700 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:00.728 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.724+0000 7f3d7fb91700 1 -- 172.21.15.16:0/2315397171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d78133b80 con 0x7f3d78135450 2019-12-05T06:58:00.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.728+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=THROTTLE_DONE pgs=1675 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.729 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.728+0000 7f3d6effd700 1 -- 172.21.15.16:0/2315397171 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f3d70000cc0 con 0x7f3d78135450 2019-12-05T06:58:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.933+0000 7f3d7fb91700 1 -- 172.21.15.16:0/2315397171 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d78133b80 con 0x7f3d78135450 2019-12-05T06:58:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.933+0000 7f3d7e92f700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=THROTTLE_DONE pgs=1675 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.934+0000 7f3d6effd700 1 -- 172.21.15.16:0/2315397171 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f3d70005900 con 0x7f3d78135450 2019-12-05T06:58:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 206 2019-12-05T06:58:00.934 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:58:00.934 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:58:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.936+0000 7f3d7fb91700 1 -- 172.21.15.16:0/2315397171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3d60023630 msgr2=0x7f3d60025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.936+0000 7f3d7fb91700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3d60023630 0x7f3d60025ac0 crc :-1 s=READY pgs=700 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.936+0000 7f3d7fb91700 1 -- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 msgr2=0x7f3d781349c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.936+0000 7f3d7fb91700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 0x7f3d781349c0 secure :-1 s=READY pgs=1675 cs=0 l=1 rx=0x7f3d70010790 tx=0x7f3d70010b30).stop 2019-12-05T06:58:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.936+0000 7f3d7fb91700 1 -- 172.21.15.16:0/2315397171 shutdown_connections 2019-12-05T06:58:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.936+0000 7f3d7fb91700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f3d60023630 0x7f3d60025ac0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.936+0000 7f3d7fb91700 1 --2- 172.21.15.16:0/2315397171 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f3d78135450 0x7f3d781349c0 unknown :-1 s=CLOSED pgs=1675 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.936+0000 7f3d7fb91700 1 --2- 172.21.15.16:0/2315397171 >> v2:172.21.15.76:3300/0 conn(0x7f3d7806e840 0x7f3d78134480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.937+0000 7f3d7fb91700 1 -- 172.21.15.16:0/2315397171 shutdown_connections 2019-12-05T06:58:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.937+0000 7f3d7fb91700 1 -- 172.21.15.16:0/2315397171 wait complete. 2019-12-05T06:58:00.938 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:00.937+0000 7f3d7fb91700 1 -- 172.21.15.16:0/2315397171 >> 172.21.15.16:0/2315397171 conn(0x7f3d780a2850 msgr2=0x7f3d7806ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:00.952 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b has gained rank=0, replacing gid=8308 2019-12-05T06:58:00.952 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 7 secs before reviving mds.a 2019-12-05T06:58:07.953 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.a 2019-12-05T06:58:07.953 INFO:tasks.ceph.mds.a:Restarting daemon 2019-12-05T06:58:07.953 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2019-12-05T06:58:07.958 INFO:tasks.ceph.mds.a:Started 2019-12-05T06:58:07.958 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:58:08.112 INFO:tasks.ceph.mds.a.smithi016.stdout:starting mds.a at 2019-12-05T06:58:08.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.161+0000 7efc3b2b1700 1 Processor -- start 2019-12-05T06:58:08.166 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.161+0000 7efc3b2b1700 1 -- start start 2019-12-05T06:58:08.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3b2b1700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efc3408c6d0 0x7efc3408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:58:08.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3b2b1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efc3408fef0 0x7efc3408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:58:08.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3b2b1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc340958e0 con 0x7efc3408ef20 2019-12-05T06:58:08.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3b2b1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc340a3f70 con 0x7efc3408fef0 2019-12-05T06:58:08.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3b2b1700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efc3408d830 con 0x7efc3408c6d0 2019-12-05T06:58:08.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3904d700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efc3408fef0 0x7efc3408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:08.167 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3a04f700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efc3408ef20 0x7efc3408f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50362/0 (socket says 172.21.15.16:50362) 2019-12-05T06:58:08.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3a04f700 1 -- 172.21.15.16:0/193819212 learned_addr learned my addr 172.21.15.16:0/193819212 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:58:08.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3984e700 1 --2- 172.21.15.16:0/193819212 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efc3408c6d0 0x7efc3408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:08.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3904d700 1 -- 172.21.15.16:0/193819212 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efc3408c6d0 msgr2=0x7efc3408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:58:08.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.162+0000 7efc3904d700 1 --2- 172.21.15.16:0/193819212 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efc3408c6d0 0x7efc3408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:58:08.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3904d700 1 -- 172.21.15.16:0/193819212 >> v1:172.21.15.16:6789/0 conn(0x7efc3408ef20 legacy=0x7efc3408f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:08.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3884c700 1 -- 172.21.15.16:0/193819212 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2994284383 0 0) 0x7efc2c0014d0 con 0x7efc3408ef20 2019-12-05T06:58:08.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3904d700 1 -- 172.21.15.16:0/193819212 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3408d540 con 0x7efc3408fef0 2019-12-05T06:58:08.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3904d700 1 --2- 172.21.15.16:0/193819212 >> v2:172.21.15.16:3300/0 conn(0x7efc3408fef0 0x7efc3408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7efc2400af50 tx=0x7efc2400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3904d700 1 --2- 172.21.15.16:0/193819212 >> v2:172.21.15.16:3300/0 conn(0x7efc3408fef0 0x7efc3408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7efc2400af50 tx=0x7efc2400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3904d700 1 --2- 172.21.15.16:0/193819212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3408fef0 0x7efc3408d000 secure :-1 s=READY pgs=1680 cs=0 l=1 rx=0x7efc2400af50 tx=0x7efc2400b6e0).ready entity=mon.0 client_cookie=d8b6bb3a26a47d32 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:08.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3904d700 1 --2- 172.21.15.16:0/193819212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3408fef0 0x7efc3408d000 secure :-1 s=THROTTLE_DONE pgs=1680 cs=0 l=1 rx=0x7efc2400af50 tx=0x7efc2400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3884c700 1 -- 172.21.15.16:0/193819212 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc24007210 con 0x7efc3408fef0 2019-12-05T06:58:08.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3904d700 1 --2- 172.21.15.16:0/193819212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3408fef0 0x7efc3408d000 secure :-1 s=THROTTLE_DONE pgs=1680 cs=0 l=1 rx=0x7efc2400af50 tx=0x7efc2400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3884c700 1 -- 172.21.15.16:0/193819212 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc24000960 con 0x7efc3408fef0 2019-12-05T06:58:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3904d700 1 --2- 172.21.15.16:0/193819212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3408fef0 0x7efc3408d000 secure :-1 s=THROTTLE_DONE pgs=1680 cs=0 l=1 rx=0x7efc2400af50 tx=0x7efc2400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3884c700 1 -- 172.21.15.16:0/193819212 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc24009ba0 con 0x7efc3408fef0 2019-12-05T06:58:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3b2b1700 1 -- 172.21.15.16:0/193819212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3408fef0 msgr2=0x7efc3408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.163+0000 7efc3b2b1700 1 --2- 172.21.15.16:0/193819212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3408fef0 0x7efc3408d000 secure :-1 s=READY pgs=1680 cs=0 l=1 rx=0x7efc2400af50 tx=0x7efc2400b6e0).stop 2019-12-05T06:58:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.164+0000 7efc3b2b1700 1 -- 172.21.15.16:0/193819212 shutdown_connections 2019-12-05T06:58:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.164+0000 7efc3b2b1700 1 --2- 172.21.15.16:0/193819212 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3408fef0 0x7efc3408d000 unknown :-1 s=CLOSED pgs=1680 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.164+0000 7efc3b2b1700 1 --2- 172.21.15.16:0/193819212 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7efc3408c6d0 0x7efc3408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:58:08.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.164+0000 7efc3b2b1700 1 -- 172.21.15.16:0/193819212 shutdown_connections 2019-12-05T06:58:08.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.164+0000 7efc3b2b1700 1 -- 172.21.15.16:0/193819212 wait complete. 2019-12-05T06:58:08.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.164+0000 7efc3b2b1700 1 -- 172.21.15.16:0/193819212 >> 172.21.15.16:0/193819212 conn(0x7efc34061940 msgr2=0x7efc34096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:08.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.167+0000 7efc3b2b1700 1 Processor -- start 2019-12-05T06:58:08.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.167+0000 7efc3b2b1700 1 -- start start 2019-12-05T06:58:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.168+0000 7efc3b2b1700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efc3414d570 0x7efc3414d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.168+0000 7efc3b2b1700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3406ea70 con 0x7efc3409d7e0 2019-12-05T06:58:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.168+0000 7efc3b2b1700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc3406e570 con 0x7efc341552b0 2019-12-05T06:58:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.168+0000 7efc3b2b1700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efc3406e210 con 0x7efc3414d570 2019-12-05T06:58:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.168+0000 7efc3984e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7efc3414d570 0x7efc3414d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.168+0000 7efc3904d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7efc3409d7e0 0x7efc34081c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50366/0 (socket says 172.21.15.16:50366) 2019-12-05T06:58:08.175 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.168+0000 7efc3904d700 1 -- 172.21.15.16:0/1282510567 learned_addr learned my addr 172.21.15.16:0/1282510567 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:58:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.168+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> v2:172.21.15.16:3300/0 conn(0x7efc3414d570 0x7efc3414d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:36964/0 (socket says 172.21.15.16:36964) 2019-12-05T06:58:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc3984e700 1 -- 172.21.15.16:0/1282510567 >> v1:172.21.15.76:6789/0 conn(0x7efc341552b0 legacy=0x7efc341576b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc3984e700 1 -- 172.21.15.16:0/1282510567 >> v1:172.21.15.16:6789/0 conn(0x7efc3409d7e0 legacy=0x7efc34081c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc2a7fc700 1 -- 172.21.15.16:0/1282510567 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950479649 0 0) 0x7efc2400ae50 con 0x7efc3409d7e0 2019-12-05T06:58:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc3984e700 1 -- 172.21.15.16:0/1282510567 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc34081320 con 0x7efc3414d570 2019-12-05T06:58:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> v2:172.21.15.16:3300/0 conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.176 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> v2:172.21.15.16:3300/0 conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=READY pgs=1682 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:08.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc2a7fc700 1 -- 172.21.15.16:0/1282510567 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc30000940 con 0x7efc3414d570 2019-12-05T06:58:08.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc3b2b1700 1 -- 172.21.15.16:0/1282510567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc34081320 con 0x7efc3414d570 2019-12-05T06:58:08.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.177 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc2a7fc700 1 -- 172.21.15.16:0/1282510567 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efc300009c0 con 0x7efc3414d570 2019-12-05T06:58:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.169+0000 7efc2a7fc700 1 -- 172.21.15.16:0/1282510567 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efc30009330 con 0x7efc3414d570 2019-12-05T06:58:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.170+0000 7efc3b2b1700 1 -- 172.21.15.16:0/1282510567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc34081320 con 0x7efc3414d570 2019-12-05T06:58:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.170+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.171+0000 7efc2a7fc700 1 -- 172.21.15.16:0/1282510567 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7efc30000c90 con 0x7efc3414d570 2019-12-05T06:58:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.171+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.178 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.171+0000 7efc2a7fc700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efc1c023630 0x7efc1c025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.171+0000 7efc2a7fc700 1 -- 172.21.15.16:0/1282510567 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7efc3004a470 con 0x7efc3414d570 2019-12-05T06:58:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.171+0000 7efc3904d700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efc1c023630 0x7efc1c025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.171+0000 7efc3904d700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efc1c023630 0x7efc1c025ac0 crc :-1 s=READY pgs=701 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:08.179 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.176+0000 7efc3b2b1700 1 -- 172.21.15.16:0/1282510567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc34081320 con 0x7efc3414d570 2019-12-05T06:58:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.180+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.180 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.180+0000 7efc2a7fc700 1 -- 172.21.15.16:0/1282510567 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7efc30009310 con 0x7efc3414d570 2019-12-05T06:58:08.379 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.379+0000 7efc3b2b1700 1 -- 172.21.15.16:0/1282510567 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efc34081320 con 0x7efc3414d570 2019-12-05T06:58:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.379+0000 7efc3984e700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=THROTTLE_DONE pgs=1682 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.379+0000 7efc2a7fc700 1 -- 172.21.15.16:0/1282510567 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+1772 (secure 0 0 0) 0x7efc30000940 con 0x7efc3414d570 2019-12-05T06:58:08.380 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 206 2019-12-05T06:58:08.381 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:58:08.381 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:58:08.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.382+0000 7efc3b2b1700 1 -- 172.21.15.16:0/1282510567 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efc1c023630 msgr2=0x7efc1c025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:08.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.382+0000 7efc3b2b1700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efc1c023630 0x7efc1c025ac0 crc :-1 s=READY pgs=701 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:08.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.382+0000 7efc3b2b1700 1 -- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 msgr2=0x7efc3414d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:08.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.382+0000 7efc3b2b1700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 0x7efc3414d960 secure :-1 s=READY pgs=1682 cs=0 l=1 rx=0x7efc30014770 tx=0x7efc30014b10).stop 2019-12-05T06:58:08.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.382+0000 7efc3b2b1700 1 -- 172.21.15.16:0/1282510567 shutdown_connections 2019-12-05T06:58:08.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.382+0000 7efc3b2b1700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7efc1c023630 0x7efc1c025ac0 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:08.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.382+0000 7efc3b2b1700 1 --2- 172.21.15.16:0/1282510567 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7efc3414d570 0x7efc3414d960 unknown :-1 s=CLOSED pgs=1682 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:08.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.382+0000 7efc3b2b1700 1 -- 172.21.15.16:0/1282510567 shutdown_connections 2019-12-05T06:58:08.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.382+0000 7efc3b2b1700 1 -- 172.21.15.16:0/1282510567 wait complete. 2019-12-05T06:58:08.384 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:08.382+0000 7efc3b2b1700 1 -- 172.21.15.16:0/1282510567 >> 172.21.15.16:0/1282510567 conn(0x7efc3408ae70 msgr2=0x7efc3409c330 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:08.406 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:58:10.408 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:58:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.684+0000 7f18ca74a700 1 Processor -- start 2019-12-05T06:58:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.684+0000 7f18ca74a700 1 -- start start 2019-12-05T06:58:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.686+0000 7f18ca74a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f18c408c6d0 0x7f18c408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:58:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.686+0000 7f18ca74a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18c408fef0 0x7f18c408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:58:10.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.686+0000 7f18ca74a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c408fb80 con 0x7f18c408ef20 2019-12-05T06:58:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.686+0000 7f18ca74a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c408df90 con 0x7f18c408fef0 2019-12-05T06:58:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.686+0000 7f18ca74a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18c40a3f70 con 0x7f18c408c6d0 2019-12-05T06:58:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.686+0000 7f18c3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18c408fef0 0x7f18c408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.686+0000 7f18c3fff700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f18c408fef0 0x7f18c408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37004/0 (socket says 172.21.15.16:37004) 2019-12-05T06:58:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.686+0000 7f18c3fff700 1 -- 172.21.15.16:0/873630078 learned_addr learned my addr 172.21.15.16:0/873630078 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:58:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.686+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/873630078 >> v2:172.21.15.76:3300/0 conn(0x7f18c408c6d0 0x7f18c408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:10.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.687+0000 7f18c3fff700 1 -- 172.21.15.16:0/873630078 >> v2:172.21.15.76:3300/0 conn(0x7f18c408c6d0 msgr2=0x7f18c408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:58:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.687+0000 7f18c3fff700 1 --2- 172.21.15.16:0/873630078 >> v2:172.21.15.76:3300/0 conn(0x7f18c408c6d0 0x7f18c408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:58:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.687+0000 7f18c3fff700 1 -- 172.21.15.16:0/873630078 >> v1:172.21.15.76:6789/0 conn(0x7f18c408ef20 legacy=0x7f18c408f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.687+0000 7f18c3fff700 1 -- 172.21.15.16:0/873630078 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c408d540 con 0x7f18c408fef0 2019-12-05T06:58:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.687+0000 7f18c3fff700 1 --2- 172.21.15.16:0/873630078 >> v2:172.21.15.16:3300/0 conn(0x7f18c408fef0 0x7f18c408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f18b400f740 tx=0x7f18b400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.688+0000 7f18c3fff700 1 --2- 172.21.15.16:0/873630078 >> v2:172.21.15.16:3300/0 conn(0x7f18c408fef0 0x7f18c408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f18b400f740 tx=0x7f18b400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.688+0000 7f18c3fff700 1 --2- 172.21.15.16:0/873630078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c408fef0 0x7f18c408d000 secure :-1 s=READY pgs=1683 cs=0 l=1 rx=0x7f18b400f740 tx=0x7f18b400f570).ready entity=mon.0 client_cookie=37427e9d0003ac5d server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:10.696 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.688+0000 7f18c3fff700 1 --2- 172.21.15.16:0/873630078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c408fef0 0x7f18c408d000 secure :-1 s=THROTTLE_DONE pgs=1683 cs=0 l=1 rx=0x7f18b400f740 tx=0x7f18b400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.688+0000 7f18c3fff700 1 --2- 172.21.15.16:0/873630078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c408fef0 0x7f18c408d000 secure :-1 s=THROTTLE_DONE pgs=1683 cs=0 l=1 rx=0x7f18b400f740 tx=0x7f18b400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.688+0000 7f18c37fe700 1 -- 172.21.15.16:0/873630078 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18b4000960 con 0x7f18c408fef0 2019-12-05T06:58:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.688+0000 7f18c3fff700 1 --2- 172.21.15.16:0/873630078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c408fef0 0x7f18c408d000 secure :-1 s=THROTTLE_DONE pgs=1683 cs=0 l=1 rx=0x7f18b400f740 tx=0x7f18b400f570).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.688+0000 7f18c37fe700 1 -- 172.21.15.16:0/873630078 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18b4000c90 con 0x7f18c408fef0 2019-12-05T06:58:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.688+0000 7f18c37fe700 1 -- 172.21.15.16:0/873630078 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18b40013a0 con 0x7f18c408fef0 2019-12-05T06:58:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.688+0000 7f18ca74a700 1 -- 172.21.15.16:0/873630078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c408fef0 msgr2=0x7f18c408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:10.697 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.688+0000 7f18ca74a700 1 --2- 172.21.15.16:0/873630078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c408fef0 0x7f18c408d000 secure :-1 s=READY pgs=1683 cs=0 l=1 rx=0x7f18b400f740 tx=0x7f18b400f570).stop 2019-12-05T06:58:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.690+0000 7f18ca74a700 1 -- 172.21.15.16:0/873630078 shutdown_connections 2019-12-05T06:58:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.690+0000 7f18ca74a700 1 --2- 172.21.15.16:0/873630078 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f18c408fef0 0x7f18c408d000 unknown :-1 s=CLOSED pgs=1683 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.690+0000 7f18ca74a700 1 --2- 172.21.15.16:0/873630078 >> v2:172.21.15.76:3300/0 conn(0x7f18c408c6d0 0x7f18c408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:58:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.690+0000 7f18ca74a700 1 -- 172.21.15.16:0/873630078 shutdown_connections 2019-12-05T06:58:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.690+0000 7f18ca74a700 1 -- 172.21.15.16:0/873630078 wait complete. 2019-12-05T06:58:10.698 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.690+0000 7f18ca74a700 1 -- 172.21.15.16:0/873630078 >> 172.21.15.16:0/873630078 conn(0x7f18c4061940 msgr2=0x7f18c4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.696+0000 7f18ca74a700 1 Processor -- start 2019-12-05T06:58:10.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.696+0000 7f18ca74a700 1 -- start start 2019-12-05T06:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.698+0000 7f18ca74a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.698+0000 7f18ca74a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c4084b90 con 0x7f18c4084f30 2019-12-05T06:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.698+0000 7f18ca74a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18c4098790 con 0x7f18c409b920 2019-12-05T06:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.698+0000 7f18ca74a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18c4099a20 con 0x7f18c4096e90 2019-12-05T06:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.698+0000 7f18c3fff700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f18c4084f30 0x7f18c4097280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50410/0 (socket says 172.21.15.16:50410) 2019-12-05T06:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.698+0000 7f18c3fff700 1 -- 172.21.15.16:0/1822280623 learned_addr learned my addr 172.21.15.16:0/1822280623 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:58:10.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.698+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.699+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 304654561 0 0) 0x7f18b4008030 con 0x7f18c4084f30 2019-12-05T06:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.699+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a8002e80 con 0x7f18c4084f30 2019-12-05T06:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.699+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818308981 0 0) 0x7f18ac001070 con 0x7f18c409b920 2019-12-05T06:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.699+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18a8004170 con 0x7f18c409b920 2019-12-05T06:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.699+0000 7f18c8ce7700 1 -- 172.21.15.16:0/1822280623 >> v1:172.21.15.76:6789/0 conn(0x7f18c409b920 legacy=0x7f18c4097a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.699+0000 7f18c8ce7700 1 -- 172.21.15.16:0/1822280623 >> v1:172.21.15.16:6789/0 conn(0x7f18c4084f30 legacy=0x7f18c4097280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:10.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.699+0000 7f18c8ce7700 1 -- 172.21.15.16:0/1822280623 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18c409aae0 con 0x7f18c4096e90 2019-12-05T06:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.699+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.699+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.699+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=READY pgs=396 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.700+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.700+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18bc000cc0 con 0x7f18c4096e90 2019-12-05T06:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.700+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.700+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.700+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18bc009fe0 con 0x7f18c4096e90 2019-12-05T06:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.700+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18bc00a630 con 0x7f18c4096e90 2019-12-05T06:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.700+0000 7f18ca74a700 1 -- 172.21.15.16:0/1822280623 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18c409aae0 con 0x7f18c4096e90 2019-12-05T06:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.700+0000 7f18ca74a700 1 -- 172.21.15.16:0/1822280623 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18c409aae0 con 0x7f18c4096e90 2019-12-05T06:58:10.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.701+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.702+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f18bc009a50 con 0x7f18c4096e90 2019-12-05T06:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.702+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.702+0000 7f18c17fa700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f18a8023650 0x7f18a8025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:10.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.703+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f18bc046180 con 0x7f18c4096e90 2019-12-05T06:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.703+0000 7f18c3fff700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f18a8023650 0x7f18a8025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:10.708 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.703+0000 7f18c3fff700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f18a8023650 0x7f18a8025ae0 crc :-1 s=READY pgs=702 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:10.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.715+0000 7f18ca74a700 1 -- 172.21.15.16:0/1822280623 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18c409aae0 con 0x7f18c4096e90 2019-12-05T06:58:10.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.724+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.726 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.724+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f18bc000cc0 con 0x7f18c4096e90 2019-12-05T06:58:10.987 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.987+0000 7f18ca74a700 1 -- 172.21.15.16:0/1822280623 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18c409aae0 con 0x7f18c4096e90 2019-12-05T06:58:10.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.988+0000 7f18c8ce7700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=THROTTLE_DONE pgs=396 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:10.988 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.988+0000 7f18c17fa700 1 -- 172.21.15.16:0/1822280623 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 206 v206) v1 ==== 95+0+1772 (secure 0 0 0) 0x7f18bc000cc0 con 0x7f18c4096e90 2019-12-05T06:58:10.988 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 206 2019-12-05T06:58:10.989 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:58:10.989 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":206,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:58:10.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.990+0000 7f18ca74a700 1 -- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f18a8023650 msgr2=0x7f18a8025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:10.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.990+0000 7f18ca74a700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f18a8023650 0x7f18a8025ae0 crc :-1 s=READY pgs=702 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:10.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.990+0000 7f18ca74a700 1 -- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 msgr2=0x7f18c409b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:10.991 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.990+0000 7f18ca74a700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 secure :-1 s=READY pgs=396 cs=0 l=1 rx=0x7f18bc010710 tx=0x7f18bc010ab0).stop 2019-12-05T06:58:10.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.991+0000 7f18ca74a700 1 -- 172.21.15.16:0/1822280623 shutdown_connections 2019-12-05T06:58:10.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.991+0000 7f18ca74a700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f18a8023650 0x7f18a8025ae0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:10.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.991+0000 7f18ca74a700 1 --2- 172.21.15.16:0/1822280623 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f18c4096e90 0x7f18c409b3e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:10.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.991+0000 7f18ca74a700 1 -- 172.21.15.16:0/1822280623 shutdown_connections 2019-12-05T06:58:10.992 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.991+0000 7f18ca74a700 1 -- 172.21.15.16:0/1822280623 wait complete. 2019-12-05T06:58:10.993 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:10.991+0000 7f18ca74a700 1 -- 172.21.15.16:0/1822280623 >> 172.21.15.16:0/1822280623 conn(0x7f18c40714f0 msgr2=0x7f18c409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:11.001 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.a is in active, standby or standby-replay 2019-12-05T06:58:13.006 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:58:13.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.161+0000 7f6b641f9700 1 Processor -- start 2019-12-05T06:58:13.168 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.161+0000 7f6b641f9700 1 -- start start 2019-12-05T06:58:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.162+0000 7f6b641f9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c08d2c0 0x7f6b5c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:58:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.162+0000 7f6b641f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b5c08fef0 0x7f6b5c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:58:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.162+0000 7f6b641f9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5c0958e0 con 0x7f6b5c08c6d0 2019-12-05T06:58:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.162+0000 7f6b641f9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5c08df90 con 0x7f6b5c08fef0 2019-12-05T06:58:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.162+0000 7f6b641f9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c0a3f70 con 0x7f6b5c08d2c0 2019-12-05T06:58:13.169 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.162+0000 7f6b62796700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f6b5c08c6d0 0x7f6b5c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50430/0 (socket says 172.21.15.16:50430) 2019-12-05T06:58:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.162+0000 7f6b62f97700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b5c08fef0 0x7f6b5c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.162+0000 7f6b62796700 1 -- 172.21.15.16:0/126785067 learned_addr learned my addr 172.21.15.16:0/126785067 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:58:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.163+0000 7f6b61f95700 1 --2- 172.21.15.16:0/126785067 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c08d2c0 0x7f6b5c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.163+0000 7f6b62f97700 1 -- 172.21.15.16:0/126785067 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c08d2c0 msgr2=0x7f6b5c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:58:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.163+0000 7f6b62f97700 1 --2- 172.21.15.16:0/126785067 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c08d2c0 0x7f6b5c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:58:13.170 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.163+0000 7f6b62f97700 1 -- 172.21.15.16:0/126785067 >> v1:172.21.15.16:6789/0 conn(0x7f6b5c08c6d0 legacy=0x7f6b5c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.163+0000 7f6b62f97700 1 -- 172.21.15.16:0/126785067 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c08e620 con 0x7f6b5c08fef0 2019-12-05T06:58:13.171 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.163+0000 7f6b61794700 1 -- 172.21.15.16:0/126785067 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965549011 0 0) 0x7f6b58001150 con 0x7f6b5c08c6d0 2019-12-05T06:58:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.163+0000 7f6b62f97700 1 --2- 172.21.15.16:0/126785067 >> v2:172.21.15.16:3300/0 conn(0x7f6b5c08fef0 0x7f6b5c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f6b5400b2d0 tx=0x7f6b5400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.163+0000 7f6b62f97700 1 --2- 172.21.15.16:0/126785067 >> v2:172.21.15.16:3300/0 conn(0x7f6b5c08fef0 0x7f6b5c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f6b5400b2d0 tx=0x7f6b5400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.163+0000 7f6b62f97700 1 --2- 172.21.15.16:0/126785067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5c08fef0 0x7f6b5c08f460 secure :-1 s=READY pgs=1687 cs=0 l=1 rx=0x7f6b5400b2d0 tx=0x7f6b5400ba60).ready entity=mon.0 client_cookie=2ee975ff06de4c3 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.164+0000 7f6b62f97700 1 --2- 172.21.15.16:0/126785067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5c08fef0 0x7f6b5c08f460 secure :-1 s=THROTTLE_DONE pgs=1687 cs=0 l=1 rx=0x7f6b5400b2d0 tx=0x7f6b5400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.172 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.164+0000 7f6b61794700 1 -- 172.21.15.16:0/126785067 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b54009f20 con 0x7f6b5c08fef0 2019-12-05T06:58:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.164+0000 7f6b62f97700 1 --2- 172.21.15.16:0/126785067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5c08fef0 0x7f6b5c08f460 secure :-1 s=THROTTLE_DONE pgs=1687 cs=0 l=1 rx=0x7f6b5400b2d0 tx=0x7f6b5400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.164+0000 7f6b61794700 1 -- 172.21.15.16:0/126785067 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b54000f40 con 0x7f6b5c08fef0 2019-12-05T06:58:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.164+0000 7f6b62f97700 1 --2- 172.21.15.16:0/126785067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5c08fef0 0x7f6b5c08f460 secure :-1 s=THROTTLE_DONE pgs=1687 cs=0 l=1 rx=0x7f6b5400b2d0 tx=0x7f6b5400ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.164+0000 7f6b61794700 1 -- 172.21.15.16:0/126785067 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b54009fc0 con 0x7f6b5c08fef0 2019-12-05T06:58:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.164+0000 7f6b641f9700 1 -- 172.21.15.16:0/126785067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5c08fef0 msgr2=0x7f6b5c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:13.173 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.164+0000 7f6b641f9700 1 --2- 172.21.15.16:0/126785067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5c08fef0 0x7f6b5c08f460 secure :-1 s=READY pgs=1687 cs=0 l=1 rx=0x7f6b5400b2d0 tx=0x7f6b5400ba60).stop 2019-12-05T06:58:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.166+0000 7f6b641f9700 1 -- 172.21.15.16:0/126785067 shutdown_connections 2019-12-05T06:58:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.166+0000 7f6b641f9700 1 --2- 172.21.15.16:0/126785067 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6b5c08fef0 0x7f6b5c08f460 unknown :-1 s=CLOSED pgs=1687 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.166+0000 7f6b641f9700 1 --2- 172.21.15.16:0/126785067 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c08d2c0 0x7f6b5c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:58:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.166+0000 7f6b641f9700 1 -- 172.21.15.16:0/126785067 shutdown_connections 2019-12-05T06:58:13.174 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.166+0000 7f6b641f9700 1 -- 172.21.15.16:0/126785067 wait complete. 2019-12-05T06:58:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.166+0000 7f6b641f9700 1 -- 172.21.15.16:0/126785067 >> 172.21.15.16:0/126785067 conn(0x7f6b5c061940 msgr2=0x7f6b5c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:13.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.169+0000 7f6b641f9700 1 Processor -- start 2019-12-05T06:58:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.169+0000 7f6b641f9700 1 -- start start 2019-12-05T06:58:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.170+0000 7f6b641f9700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b5c14ca10 0x7f6b5c0869e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.170+0000 7f6b641f9700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.170+0000 7f6b641f9700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b5c153170 con 0x7f6b5c0879b0 2019-12-05T06:58:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.170+0000 7f6b641f9700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b5c085a50 con 0x7f6b5c14ca10 2019-12-05T06:58:13.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.170+0000 7f6b641f9700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b5c085020 con 0x7f6b5c09be80 2019-12-05T06:58:13.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.170+0000 7f6b62796700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b5c14ca10 0x7f6b5c0869e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:13.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.170+0000 7f6b62796700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6b5c14ca10 0x7f6b5c0869e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37036/0 (socket says 172.21.15.16:37036) 2019-12-05T06:58:13.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.170+0000 7f6b62796700 1 -- 172.21.15.16:0/2634684544 learned_addr learned my addr 172.21.15.16:0/2634684544 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:58:13.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.170+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b4effd700 1 -- 172.21.15.16:0/2634684544 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305837309 0 0) 0x7f6b50000f30 con 0x7f6b5c0879b0 2019-12-05T06:58:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b4effd700 1 -- 172.21.15.16:0/2634684544 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6b44002e80 con 0x7f6b5c0879b0 2019-12-05T06:58:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62f97700 1 -- 172.21.15.16:0/2634684544 >> v2:172.21.15.16:3300/0 conn(0x7f6b5c14ca10 msgr2=0x7f6b5c0869e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> v2:172.21.15.16:3300/0 conn(0x7f6b5c14ca10 0x7f6b5c0869e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:13.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62f97700 1 -- 172.21.15.16:0/2634684544 >> v1:172.21.15.16:6789/0 conn(0x7f6b5c0879b0 legacy=0x7f6b5c086f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62f97700 1 -- 172.21.15.16:0/2634684544 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b5c0860e0 con 0x7f6b5c09be80 2019-12-05T06:58:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62796700 1 --2- 172.21.15.16:0/2634684544 >> v2:172.21.15.16:3300/0 conn(0x7f6b5c14ca10 0x7f6b5c0869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:58:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=READY pgs=400 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b4effd700 1 -- 172.21.15.16:0/2634684544 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b5400afa0 con 0x7f6b5c09be80 2019-12-05T06:58:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b4effd700 1 -- 172.21.15.16:0/2634684544 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b54009b10 con 0x7f6b5c09be80 2019-12-05T06:58:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b641f9700 1 -- 172.21.15.16:0/2634684544 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b5c0860e0 con 0x7f6b5c09be80 2019-12-05T06:58:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b4effd700 1 -- 172.21.15.16:0/2634684544 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b5400afa0 con 0x7f6b5c09be80 2019-12-05T06:58:13.187 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.171+0000 7f6b641f9700 1 -- 172.21.15.16:0/2634684544 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b5c0860e0 con 0x7f6b5c09be80 2019-12-05T06:58:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.172+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.173+0000 7f6b4effd700 1 -- 172.21.15.16:0/2634684544 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f6b54000cc0 con 0x7f6b5c09be80 2019-12-05T06:58:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.173+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.173+0000 7f6b4effd700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b44023670 0x7f6b44025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.173+0000 7f6b4effd700 1 -- 172.21.15.16:0/2634684544 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f6b54046300 con 0x7f6b5c09be80 2019-12-05T06:58:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.173+0000 7f6b62796700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b44023670 0x7f6b44025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:13.188 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.173+0000 7f6b62796700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b44023670 0x7f6b44025ae0 crc :-1 s=READY pgs=704 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:13.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.177+0000 7f6b641f9700 1 -- 172.21.15.16:0/2634684544 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b5c0860e0 con 0x7f6b5c09be80 2019-12-05T06:58:13.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.182+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.182+0000 7f6b4effd700 1 -- 172.21.15.16:0/2634684544 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f6b5402b320 con 0x7f6b5c09be80 2019-12-05T06:58:13.390 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.390+0000 7f6b641f9700 1 -- 172.21.15.16:0/2634684544 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6b5c0860e0 con 0x7f6b5c09be80 2019-12-05T06:58:13.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.391+0000 7f6b62f97700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=THROTTLE_DONE pgs=400 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:13.391 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.391+0000 7f6b4effd700 1 -- 172.21.15.16:0/2634684544 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f6b5402b320 con 0x7f6b5c09be80 2019-12-05T06:58:13.393 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 207 2019-12-05T06:58:13.393 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:58:13.393 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8523,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":207}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:58:13.397 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.396+0000 7f6b641f9700 1 -- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b44023670 msgr2=0x7f6b44025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.396+0000 7f6b641f9700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b44023670 0x7f6b44025ae0 crc :-1 s=READY pgs=704 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.396+0000 7f6b641f9700 1 -- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 msgr2=0x7f6b5c09c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.396+0000 7f6b641f9700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 secure :-1 s=READY pgs=400 cs=0 l=1 rx=0x7f6b54010720 tx=0x7f6b54010ac0).stop 2019-12-05T06:58:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.396+0000 7f6b641f9700 1 -- 172.21.15.16:0/2634684544 shutdown_connections 2019-12-05T06:58:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.396+0000 7f6b641f9700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f6b5c09be80 0x7f6b5c09c2d0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.396+0000 7f6b641f9700 1 --2- 172.21.15.16:0/2634684544 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6b44023670 0x7f6b44025ae0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:13.398 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.396+0000 7f6b641f9700 1 --2- 172.21.15.16:0/2634684544 >> v2:172.21.15.16:3300/0 conn(0x7f6b5c14ca10 0x7f6b5c0869e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:13.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.397+0000 7f6b641f9700 1 -- 172.21.15.16:0/2634684544 shutdown_connections 2019-12-05T06:58:13.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.397+0000 7f6b641f9700 1 -- 172.21.15.16:0/2634684544 wait complete. 2019-12-05T06:58:13.399 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:13.397+0000 7f6b641f9700 1 -- 172.21.15.16:0/2634684544 >> 172.21.15.16:0/2634684544 conn(0x7f6b5c098480 msgr2=0x7f6b5c071210 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:13.413 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a reported in up:standby state 2019-12-05T06:58:13.413 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 44 secs before thrashing 2019-12-05T06:58:17.858 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:58:17.863 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:58:48.014 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:58:48.020 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:58:57.414 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:58:57.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.597+0000 7f5747f88700 1 Processor -- start 2019-12-05T06:58:57.605 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.597+0000 7f5747f88700 1 -- start start 2019-12-05T06:58:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.599+0000 7f5747f88700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f574008fef0 0x7f574008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:58:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.599+0000 7f5747f88700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57400958e0 con 0x7f574008d2c0 2019-12-05T06:58:57.606 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.599+0000 7f5747f88700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57400629e0 con 0x7f574008c6d0 2019-12-05T06:58:57.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.599+0000 7f5747f88700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f574008df90 con 0x7f574008fef0 2019-12-05T06:58:57.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.599+0000 7f5746d26700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f574008fef0 0x7f574008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:57.607 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.599+0000 7f5746525700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f574008c6d0 0x7f574008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:43724/0 (socket says 172.21.15.16:43724) 2019-12-05T06:58:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.600+0000 7f5746525700 1 -- 172.21.15.16:0/2526657298 learned_addr learned my addr 172.21.15.16:0/2526657298 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:58:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.599+0000 7f5745d24700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f574008d2c0 0x7f574008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50656/0 (socket says 172.21.15.16:50656) 2019-12-05T06:58:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.600+0000 7f5746d26700 1 -- 172.21.15.16:0/2526657298 >> v1:172.21.15.76:6789/0 conn(0x7f574008c6d0 legacy=0x7f574008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.600+0000 7f5746d26700 1 -- 172.21.15.16:0/2526657298 >> v1:172.21.15.16:6789/0 conn(0x7f574008d2c0 legacy=0x7f574008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.600+0000 7f5745523700 1 -- 172.21.15.16:0/2526657298 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 298056833 0 0) 0x7f573c001100 con 0x7f574008c6d0 2019-12-05T06:58:57.608 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.600+0000 7f5746d26700 1 -- 172.21.15.16:0/2526657298 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574008e620 con 0x7f574008fef0 2019-12-05T06:58:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.600+0000 7f5746d26700 1 --2- 172.21.15.16:0/2526657298 >> v2:172.21.15.16:3300/0 conn(0x7f574008fef0 0x7f574008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f573800b2d0 tx=0x7f573800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.600+0000 7f5746d26700 1 --2- 172.21.15.16:0/2526657298 >> v2:172.21.15.16:3300/0 conn(0x7f574008fef0 0x7f574008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f573800b2d0 tx=0x7f573800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.600+0000 7f5746d26700 1 --2- 172.21.15.16:0/2526657298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574008fef0 0x7f574008f720 secure :-1 s=READY pgs=1691 cs=0 l=1 rx=0x7f573800b2d0 tx=0x7f573800bae0).ready entity=mon.0 client_cookie=c90c54eced093239 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.600+0000 7f5746d26700 1 --2- 172.21.15.16:0/2526657298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574008fef0 0x7f574008f720 secure :-1 s=THROTTLE_DONE pgs=1691 cs=0 l=1 rx=0x7f573800b2d0 tx=0x7f573800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.609 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.600+0000 7f5745523700 1 -- 172.21.15.16:0/2526657298 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5738009f20 con 0x7f574008fef0 2019-12-05T06:58:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.601+0000 7f5746d26700 1 --2- 172.21.15.16:0/2526657298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574008fef0 0x7f574008f720 secure :-1 s=THROTTLE_DONE pgs=1691 cs=0 l=1 rx=0x7f573800b2d0 tx=0x7f573800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.601+0000 7f5745523700 1 -- 172.21.15.16:0/2526657298 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5738000ce0 con 0x7f574008fef0 2019-12-05T06:58:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.601+0000 7f5746d26700 1 --2- 172.21.15.16:0/2526657298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574008fef0 0x7f574008f720 secure :-1 s=THROTTLE_DONE pgs=1691 cs=0 l=1 rx=0x7f573800b2d0 tx=0x7f573800bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.610 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.601+0000 7f5745523700 1 -- 172.21.15.16:0/2526657298 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5738009f20 con 0x7f574008fef0 2019-12-05T06:58:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.601+0000 7f5747f88700 1 -- 172.21.15.16:0/2526657298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574008fef0 msgr2=0x7f574008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.601+0000 7f5747f88700 1 --2- 172.21.15.16:0/2526657298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574008fef0 0x7f574008f720 secure :-1 s=READY pgs=1691 cs=0 l=1 rx=0x7f573800b2d0 tx=0x7f573800bae0).stop 2019-12-05T06:58:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.602+0000 7f5747f88700 1 -- 172.21.15.16:0/2526657298 shutdown_connections 2019-12-05T06:58:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.602+0000 7f5747f88700 1 --2- 172.21.15.16:0/2526657298 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f574008fef0 0x7f574008f720 unknown :-1 s=CLOSED pgs=1691 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.602+0000 7f5747f88700 1 -- 172.21.15.16:0/2526657298 shutdown_connections 2019-12-05T06:58:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.602+0000 7f5747f88700 1 -- 172.21.15.16:0/2526657298 wait complete. 2019-12-05T06:58:57.611 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.602+0000 7f5747f88700 1 -- 172.21.15.16:0/2526657298 >> 172.21.15.16:0/2526657298 conn(0x7f5740061940 msgr2=0x7f5740096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:57.617 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.606+0000 7f5747f88700 1 Processor -- start 2019-12-05T06:58:57.618 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.606+0000 7f5747f88700 1 -- start start 2019-12-05T06:58:57.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5747f88700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5740150970 0x7f5740150d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:57.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5747f88700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f57401309e0 0x7f574012fa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:57.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5747f88700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5740088c30 con 0x7f574012ff50 2019-12-05T06:58:57.620 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5747f88700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5740089ec0 con 0x7f57401309e0 2019-12-05T06:58:57.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5747f88700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f574008a190 con 0x7f5740150970 2019-12-05T06:58:57.621 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5746d26700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f574012ff50 0x7f5740080210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50662/0 (socket says 172.21.15.16:50662) 2019-12-05T06:58:57.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5746d26700 1 -- 172.21.15.16:0/473544717 learned_addr learned my addr 172.21.15.16:0/473544717 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:58:57.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5745d24700 1 --2- 172.21.15.16:0/473544717 >> v2:172.21.15.76:3300/0 conn(0x7f57401309e0 0x7f574012fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:57.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5746525700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5740150970 0x7f5740150d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:57.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 144244435 0 0) 0x7f5738000d50 con 0x7f574012ff50 2019-12-05T06:58:57.622 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.607+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5728002e80 con 0x7f574012ff50 2019-12-05T06:58:57.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1721401294 0 0) 0x7f5738000ed0 con 0x7f574012ff50 2019-12-05T06:58:57.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57280032e0 con 0x7f574012ff50 2019-12-05T06:58:57.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f573800b1f0 con 0x7f574012ff50 2019-12-05T06:58:57.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5740150970 msgr2=0x7f5740150d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:57.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5740150970 0x7f5740150d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:57.623 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 >> v2:172.21.15.76:3300/0 conn(0x7f57401309e0 msgr2=0x7f574012fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:57.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 --2- 172.21.15.16:0/473544717 >> v2:172.21.15.76:3300/0 conn(0x7f57401309e0 0x7f574012fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:57.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 >> v1:172.21.15.16:6789/0 conn(0x7f574012ff50 legacy=0x7f5740080210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:57.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5728003330 0x7f5728003760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:57.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5728003ca0 0x7f5728004110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:57.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:57.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572800bb60 con 0x7f5728004840 2019-12-05T06:58:57.624 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f572800bd90 con 0x7f5728003330 2019-12-05T06:58:57.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f572800bfc0 con 0x7f5728003ca0 2019-12-05T06:58:57.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.608+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:57.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746525700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5728003330 0x7f5728003760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:57.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5745d24700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5728003ca0 0x7f5728004110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:57.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 -- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5728003ca0 msgr2=0x7f5728004110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:57.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f5728003ca0 0x7f5728004110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:57.625 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 -- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5728003330 msgr2=0x7f5728003760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:57.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5728003330 0x7f5728003760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:57.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 -- 172.21.15.16:0/473544717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572800c270 con 0x7f5728004840 2019-12-05T06:58:57.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746525700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f5728003330 0x7f5728003760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:58:57.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5747f88700 1 -- 172.21.15.16:0/473544717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574012f110 con 0x7f5728004840 2019-12-05T06:58:57.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.626 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5747f88700 1 -- 172.21.15.16:0/473544717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5740088c30 con 0x7f5728004840 2019-12-05T06:58:57.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=READY pgs=1693 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:57.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=THROTTLE_DONE pgs=1693 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=THROTTLE_DONE pgs=1693 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.627 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5738000ed0 con 0x7f5728004840 2019-12-05T06:58:57.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=THROTTLE_DONE pgs=1693 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5738009af0 con 0x7f5728004840 2019-12-05T06:58:57.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.609+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f573800a200 con 0x7f5728004840 2019-12-05T06:58:57.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.610+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=THROTTLE_DONE pgs=1693 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.610+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f5738009ce0 con 0x7f5728004840 2019-12-05T06:58:57.628 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.610+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=THROTTLE_DONE pgs=1693 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.610+0000 7f5736ffd700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f572802b9f0 0x7f572802de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:57.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.611+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f5738046310 con 0x7f5728004840 2019-12-05T06:58:57.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.611+0000 7f5746525700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f572802b9f0 0x7f572802de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:57.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.611+0000 7f5746525700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f572802b9f0 0x7f572802de20 crc :-1 s=READY pgs=707 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:57.629 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.614+0000 7f5747f88700 1 -- 172.21.15.16:0/473544717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f574012f110 con 0x7f5728004840 2019-12-05T06:58:57.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.619+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=THROTTLE_DONE pgs=1693 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.631 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.619+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f573800b940 con 0x7f5728004840 2019-12-05T06:58:57.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.871+0000 7f5747f88700 1 -- 172.21.15.16:0/473544717 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f574012f110 con 0x7f5728004840 2019-12-05T06:58:57.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.872+0000 7f5746d26700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=THROTTLE_DONE pgs=1693 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:57.872 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.872+0000 7f5736ffd700 1 -- 172.21.15.16:0/473544717 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f5738009af0 con 0x7f5728004840 2019-12-05T06:58:57.873 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 207 2019-12-05T06:58:57.873 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:58:57.874 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8523,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":207}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:58:57.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.875+0000 7f5747f88700 1 -- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f572802b9f0 msgr2=0x7f572802de20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:57.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.875+0000 7f5747f88700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f572802b9f0 0x7f572802de20 crc :-1 s=READY pgs=707 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:57.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.875+0000 7f5747f88700 1 -- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 msgr2=0x7f5728004d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:57.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.875+0000 7f5747f88700 1 --2- 172.21.15.16:0/473544717 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f5728004840 0x7f5728004d00 secure :-1 s=READY pgs=1693 cs=0 l=1 rx=0x7f57380107c0 tx=0x7f5738010640).stop 2019-12-05T06:58:57.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.875+0000 7f5746d26700 1 -- 172.21.15.16:0/473544717 reap_dead start 2019-12-05T06:58:57.876 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.875+0000 7f5746d26700 1 -- 172.21.15.16:0/473544717 reap_dead start 2019-12-05T06:58:57.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.875+0000 7f5747f88700 1 -- 172.21.15.16:0/473544717 shutdown_connections 2019-12-05T06:58:57.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.875+0000 7f5747f88700 1 -- 172.21.15.16:0/473544717 shutdown_connections 2019-12-05T06:58:57.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.875+0000 7f5747f88700 1 -- 172.21.15.16:0/473544717 wait complete. 2019-12-05T06:58:57.877 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:57.875+0000 7f5747f88700 1 -- 172.21.15.16:0/473544717 >> 172.21.15.16:0/473544717 conn(0x7f574009de30 msgr2=0x7f5740070fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:57.886 INFO:tasks.mds_thrash.fs.[cephfs]:kill mds.b (rank=0) 2019-12-05T06:58:57.887 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2019-12-05T06:58:57.887 INFO:teuthology.orchestra.run:waiting for 300 2019-12-05T06:58:57.899 INFO:tasks.ceph.mds.b:Stopped 2019-12-05T06:58:57.902 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:58:58.127 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.117+0000 7f59c743b700 1 Processor -- start 2019-12-05T06:58:58.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.117+0000 7f59c743b700 1 -- start start 2019-12-05T06:58:58.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.119+0000 7f59c743b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 0x7f59c008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:58:58.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.119+0000 7f59c743b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c00958e0 con 0x7f59c008c6d0 2019-12-05T06:58:58.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.119+0000 7f59c743b700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c00629e0 con 0x7f59c008d2c0 2019-12-05T06:58:58.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.119+0000 7f59c743b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c008df90 con 0x7f59c008fef0 2019-12-05T06:58:58.128 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.119+0000 7f59c59d8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f59c008c6d0 0x7f59c008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50678/0 (socket says 172.21.15.16:50678) 2019-12-05T06:58:58.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.119+0000 7f59c59d8700 1 -- 172.21.15.16:0/2232247228 learned_addr learned my addr 172.21.15.16:0/2232247228 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:58:58.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.119+0000 7f59c61d9700 1 --2- 172.21.15.16:0/2232247228 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 0x7f59c008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:58.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.120+0000 7f59c49d6700 1 -- 172.21.15.16:0/2232247228 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2908789079 0 0) 0x7f59bc001150 con 0x7f59c008c6d0 2019-12-05T06:58:58.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.120+0000 7f59c49d6700 1 -- 172.21.15.16:0/2232247228 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ac002e80 con 0x7f59c008c6d0 2019-12-05T06:58:58.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.120+0000 7f59c49d6700 1 -- 172.21.15.16:0/2232247228 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022515813 0 0) 0x7f59b0001080 con 0x7f59c008d2c0 2019-12-05T06:58:58.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.120+0000 7f59c49d6700 1 -- 172.21.15.16:0/2232247228 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ac0030b0 con 0x7f59c008d2c0 2019-12-05T06:58:58.129 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.120+0000 7f59c61d9700 1 -- 172.21.15.16:0/2232247228 >> v1:172.21.15.76:6789/0 conn(0x7f59c008d2c0 legacy=0x7f59c008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:58.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.120+0000 7f59c61d9700 1 -- 172.21.15.16:0/2232247228 >> v1:172.21.15.16:6789/0 conn(0x7f59c008c6d0 legacy=0x7f59c008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:58.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.120+0000 7f59c49d6700 1 -- 172.21.15.16:0/2232247228 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1768240346 0 0) 0x7f59bc001130 con 0x7f59c008c6d0 2019-12-05T06:58:58.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.120+0000 7f59c61d9700 1 -- 172.21.15.16:0/2232247228 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c008e620 con 0x7f59c008fef0 2019-12-05T06:58:58.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.121+0000 7f59c61d9700 1 --2- 172.21.15.16:0/2232247228 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 0x7f59c008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f59b800b300 tx=0x7f59b800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.130 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.121+0000 7f59c61d9700 1 --2- 172.21.15.16:0/2232247228 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 0x7f59c008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f59b800b300 tx=0x7f59b800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.121+0000 7f59c61d9700 1 --2- 172.21.15.16:0/2232247228 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 0x7f59c008f720 secure :-1 s=READY pgs=401 cs=0 l=1 rx=0x7f59b800b300 tx=0x7f59b800bb10).ready entity=mon.2 client_cookie=c9fff3f9ddd8d3f5 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:58.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.121+0000 7f59c61d9700 1 --2- 172.21.15.16:0/2232247228 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 0x7f59c008f720 secure :-1 s=THROTTLE_DONE pgs=401 cs=0 l=1 rx=0x7f59b800b300 tx=0x7f59b800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.121+0000 7f59c61d9700 1 --2- 172.21.15.16:0/2232247228 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 0x7f59c008f720 secure :-1 s=THROTTLE_DONE pgs=401 cs=0 l=1 rx=0x7f59b800b300 tx=0x7f59b800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.133 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.121+0000 7f59c49d6700 1 -- 172.21.15.16:0/2232247228 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59b80075c0 con 0x7f59c008fef0 2019-12-05T06:58:58.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.121+0000 7f59c61d9700 1 --2- 172.21.15.16:0/2232247228 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 0x7f59c008f720 secure :-1 s=THROTTLE_DONE pgs=401 cs=0 l=1 rx=0x7f59b800b300 tx=0x7f59b800bb10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.121+0000 7f59c49d6700 1 -- 172.21.15.16:0/2232247228 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b8000fb0 con 0x7f59c008fef0 2019-12-05T06:58:58.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.122+0000 7f59c49d6700 1 -- 172.21.15.16:0/2232247228 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59b800a1e0 con 0x7f59c008fef0 2019-12-05T06:58:58.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.122+0000 7f59c743b700 1 -- 172.21.15.16:0/2232247228 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 msgr2=0x7f59c008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:58.134 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.122+0000 7f59c743b700 1 --2- 172.21.15.16:0/2232247228 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 0x7f59c008f720 secure :-1 s=READY pgs=401 cs=0 l=1 rx=0x7f59b800b300 tx=0x7f59b800bb10).stop 2019-12-05T06:58:58.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.123+0000 7f59c743b700 1 -- 172.21.15.16:0/2232247228 shutdown_connections 2019-12-05T06:58:58.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.123+0000 7f59c743b700 1 --2- 172.21.15.16:0/2232247228 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c008fef0 0x7f59c008f720 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:58.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.124+0000 7f59c743b700 1 -- 172.21.15.16:0/2232247228 shutdown_connections 2019-12-05T06:58:58.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.124+0000 7f59c743b700 1 -- 172.21.15.16:0/2232247228 wait complete. 2019-12-05T06:58:58.135 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.124+0000 7f59c743b700 1 -- 172.21.15.16:0/2232247228 >> 172.21.15.16:0/2232247228 conn(0x7f59c0061940 msgr2=0x7f59c0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:58.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.128+0000 7f59c743b700 1 Processor -- start 2019-12-05T06:58:58.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.128+0000 7f59c743b700 1 -- start start 2019-12-05T06:58:58.136 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.129+0000 7f59c743b700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c006e840 0x7f59c0134480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:58.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.130+0000 7f59c743b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 0x7f59c01349c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:58.137 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.130+0000 7f59c743b700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c0128af0 con 0x7f59c006dc50 2019-12-05T06:58:58.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.130+0000 7f59c743b700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c0132830 con 0x7f59c0135450 2019-12-05T06:58:58.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.130+0000 7f59c743b700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c01334f0 con 0x7f59c006e840 2019-12-05T06:58:58.139 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.130+0000 7f59c59d8700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f59c006dc50 0x7f59c006e040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50684/0 (socket says 172.21.15.16:50684) 2019-12-05T06:58:58.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.130+0000 7f59c59d8700 1 -- 172.21.15.16:0/1428278127 learned_addr learned my addr 172.21.15.16:0/1428278127 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:58:58.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.130+0000 7f59c61d9700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 0x7f59c01349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:58.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.130+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c006e840 0x7f59c0134480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:58.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.130+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809901361 0 0) 0x7f59bc000f40 con 0x7f59c006dc50 2019-12-05T06:58:58.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.130+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59a8002e80 con 0x7f59c006dc50 2019-12-05T06:58:58.140 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.131+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1192980584 0 0) 0x7f59bc000c90 con 0x7f59c006dc50 2019-12-05T06:58:58.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.131+0000 7f59c61d9700 1 -- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c006e840 msgr2=0x7f59c0134480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:58.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.131+0000 7f59c61d9700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f59c006e840 0x7f59c0134480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:58.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.131+0000 7f59c61d9700 1 -- 172.21.15.16:0/1428278127 >> v1:172.21.15.16:6789/0 conn(0x7f59c006dc50 legacy=0x7f59c006e040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:58.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.131+0000 7f59c61d9700 1 -- 172.21.15.16:0/1428278127 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0133b80 con 0x7f59c0135450 2019-12-05T06:58:58.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.131+0000 7f59c61d9700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 0x7f59c01349c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f59b8010700 tx=0x7f59b8010580).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.131+0000 7f59c61d9700 1 -- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 msgr2=0x7f59c01349c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2019-12-05T06:58:58.141 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.131+0000 7f59c61d9700 1 -- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 msgr2=0x7f59c01349c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:58:58.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.131+0000 7f59c61d9700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 0x7f59c01349c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f59b8010700 tx=0x7f59b8010580).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:58:58.142 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.131+0000 7f59c61d9700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 0x7f59c01349c0 secure :-1 s=START_CONNECT pgs=0 cs=0 l=1 rx=0x7f59b8010700 tx=0x7f59b8010580)._fault waiting 0.200000 2019-12-05T06:58:58.334 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.332+0000 7f59c61d9700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 0x7f59c01349c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:58.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.332+0000 7f59c61d9700 0 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 0x7f59c01349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2019-12-05T06:58:58.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.332+0000 7f59c61d9700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 0x7f59c01349c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:58.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.332+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 msgr2=0x7f59c01349c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:58:58.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.332+0000 7f59b67fc700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.16:3300/0 conn(0x7f59c0135450 0x7f59c01349c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:58.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.332+0000 7f59b67fc700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.76:3300/0 conn(0x7f59a8006d50 0x7f59a8009150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:58.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.332+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59a80098a0 con 0x7f59a80030b0 2019-12-05T06:58:58.335 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.332+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59a800abb0 con 0x7f59a8006d50 2019-12-05T06:58:58.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.333+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.76:3300/0 conn(0x7f59a8006d50 0x7f59a8009150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:58.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.333+0000 7f59c51d7700 1 -- 172.21.15.16:0/1428278127 >> v1:172.21.15.76:6789/0 conn(0x7f59a80030b0 legacy=0x7f59a8005540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:58.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.333+0000 7f59c51d7700 1 -- 172.21.15.16:0/1428278127 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a800ae60 con 0x7f59a8006d50 2019-12-05T06:58:58.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.333+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.76:3300/0 conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.334+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> v2:172.21.15.76:3300/0 conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.334+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=READY pgs=1144 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:58.336 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.334+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=THROTTLE_DONE pgs=1144 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.337 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.334+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=THROTTLE_DONE pgs=1144 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.334+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59b0000bc0 con 0x7f59a8006d50 2019-12-05T06:58:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.334+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=THROTTLE_DONE pgs=1144 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.338 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.334+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b0009770 con 0x7f59a8006d50 2019-12-05T06:58:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.334+0000 7f59c743b700 1 -- 172.21.15.16:0/1428278127 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c0133b80 con 0x7f59a8006d50 2019-12-05T06:58:58.339 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.334+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59b0009dc0 con 0x7f59a8006d50 2019-12-05T06:58:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.335+0000 7f59c743b700 1 -- 172.21.15.16:0/1428278127 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c0133b80 con 0x7f59a8006d50 2019-12-05T06:58:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.336+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=THROTTLE_DONE pgs=1144 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.337+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 <== mon.1 v2:172.21.15.76:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f59b0009960 con 0x7f59a8006d50 2019-12-05T06:58:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.337+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=THROTTLE_DONE pgs=1144 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.337+0000 7f59b67fc700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59a8029fe0 0x7f59a802c430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:58:58.340 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.337+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 <== mon.1 v2:172.21.15.76:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f59b0045f60 con 0x7f59a8006d50 2019-12-05T06:58:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.337+0000 7f59c59d8700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59a8029fe0 0x7f59a802c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:58:58.341 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.338+0000 7f59c59d8700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59a8029fe0 0x7f59a802c430 crc :-1 s=READY pgs=708 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:58:58.383 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.382+0000 7f59c743b700 1 -- 172.21.15.16:0/1428278127 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c0133b80 con 0x7f59a8006d50 2019-12-05T06:58:58.392 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.392+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=THROTTLE_DONE pgs=1144 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.393 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.392+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 <== mon.1 v2:172.21.15.76:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f59b000a8a0 con 0x7f59a8006d50 2019-12-05T06:58:58.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.656+0000 7f59c743b700 1 -- 172.21.15.16:0/1428278127 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f59c0133b80 con 0x7f59a8006d50 2019-12-05T06:58:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.657+0000 7f59c51d7700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=THROTTLE_DONE pgs=1144 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:58:58.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.657+0000 7f59b67fc700 1 -- 172.21.15.16:0/1428278127 <== mon.1 v2:172.21.15.76:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f59b0009960 con 0x7f59a8006d50 2019-12-05T06:58:58.660 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 207 2019-12-05T06:58:58.660 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:58:58.660 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8523,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":207}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:58:58.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.662+0000 7f59c743b700 1 -- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59a8029fe0 msgr2=0x7f59a802c430 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:58.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.662+0000 7f59c743b700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f59a8029fe0 0x7f59a802c430 crc :-1 s=READY pgs=708 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:58.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.662+0000 7f59c743b700 1 -- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 msgr2=0x7f59a8009150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:58:58.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.662+0000 7f59c743b700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 0x7f59a8009150 secure :-1 s=READY pgs=1144 cs=0 l=1 rx=0x7f59b0010570 tx=0x7f59b0010910).stop 2019-12-05T06:58:58.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.662+0000 7f59c61d9700 1 -- 172.21.15.16:0/1428278127 reap_dead start 2019-12-05T06:58:58.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.662+0000 7f59c743b700 1 -- 172.21.15.16:0/1428278127 shutdown_connections 2019-12-05T06:58:58.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.662+0000 7f59c743b700 1 --2- 172.21.15.16:0/1428278127 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f59a8006d50 0x7f59a8009150 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:58:58.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.663+0000 7f59c743b700 1 -- 172.21.15.16:0/1428278127 shutdown_connections 2019-12-05T06:58:58.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.663+0000 7f59c743b700 1 -- 172.21.15.16:0/1428278127 wait complete. 2019-12-05T06:58:58.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:58:58.663+0000 7f59c743b700 1 -- 172.21.15.16:0/1428278127 >> 172.21.15.16:0/1428278127 conn(0x7f59c00a2850 msgr2=0x7f59c006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:58:58.683 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:59:00.684 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:00.929 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.920+0000 7f31d6ef6700 1 Processor -- start 2019-12-05T06:59:00.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.920+0000 7f31d6ef6700 1 -- start start 2019-12-05T06:59:00.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.922+0000 7f31d6ef6700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31d008fef0 0x7f31d008f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:00.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.922+0000 7f31d6ef6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d00958e0 con 0x7f31d008d2c0 2019-12-05T06:59:00.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.922+0000 7f31d6ef6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d00629e0 con 0x7f31d008c6d0 2019-12-05T06:59:00.930 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.922+0000 7f31d6ef6700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31d008df90 con 0x7f31d008fef0 2019-12-05T06:59:00.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.922+0000 7f31d5c94700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f31d008fef0 0x7f31d008f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:00.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.922+0000 7f31d5493700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f31d008c6d0 0x7f31d008cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:43774/0 (socket says 172.21.15.16:43774) 2019-12-05T06:59:00.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.922+0000 7f31d4c92700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f31d008d2c0 0x7f31d008ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50708/0 (socket says 172.21.15.16:50708) 2019-12-05T06:59:00.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.922+0000 7f31d5493700 1 -- 172.21.15.16:0/2586299625 learned_addr learned my addr 172.21.15.16:0/2586299625 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:00.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31cffff700 1 -- 172.21.15.16:0/2586299625 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2148175467 0 0) 0x7f31c80013c0 con 0x7f31d008c6d0 2019-12-05T06:59:00.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31d5c94700 1 -- 172.21.15.16:0/2586299625 >> v1:172.21.15.76:6789/0 conn(0x7f31d008c6d0 legacy=0x7f31d008cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:00.931 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31d5c94700 1 -- 172.21.15.16:0/2586299625 >> v1:172.21.15.16:6789/0 conn(0x7f31d008d2c0 legacy=0x7f31d008ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:00.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31d5c94700 1 -- 172.21.15.16:0/2586299625 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d008e620 con 0x7f31d008fef0 2019-12-05T06:59:00.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31d5c94700 1 --2- 172.21.15.16:0/2586299625 >> v2:172.21.15.16:3300/0 conn(0x7f31d008fef0 0x7f31d008f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f31c000af50 tx=0x7f31c000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31d5c94700 1 --2- 172.21.15.16:0/2586299625 >> v2:172.21.15.16:3300/0 conn(0x7f31d008fef0 0x7f31d008f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f31c000af50 tx=0x7f31c000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31d5c94700 1 --2- 172.21.15.16:0/2586299625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d008fef0 0x7f31d008f720 secure :-1 s=READY pgs=1698 cs=0 l=1 rx=0x7f31c000af50 tx=0x7f31c000b760).ready entity=mon.0 client_cookie=1af6c7fb13ab9cdd server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:00.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31d5c94700 1 --2- 172.21.15.16:0/2586299625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d008fef0 0x7f31d008f720 secure :-1 s=THROTTLE_DONE pgs=1698 cs=0 l=1 rx=0x7f31c000af50 tx=0x7f31c000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31d5c94700 1 --2- 172.21.15.16:0/2586299625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d008fef0 0x7f31d008f720 secure :-1 s=THROTTLE_DONE pgs=1698 cs=0 l=1 rx=0x7f31c000af50 tx=0x7f31c000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.932 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31cffff700 1 -- 172.21.15.16:0/2586299625 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31c0009ba0 con 0x7f31d008fef0 2019-12-05T06:59:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31d5c94700 1 --2- 172.21.15.16:0/2586299625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d008fef0 0x7f31d008f720 secure :-1 s=THROTTLE_DONE pgs=1698 cs=0 l=1 rx=0x7f31c000af50 tx=0x7f31c000b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.923+0000 7f31cffff700 1 -- 172.21.15.16:0/2586299625 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c0000a70 con 0x7f31d008fef0 2019-12-05T06:59:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.924+0000 7f31cffff700 1 -- 172.21.15.16:0/2586299625 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31c000a180 con 0x7f31d008fef0 2019-12-05T06:59:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.924+0000 7f31d6ef6700 1 -- 172.21.15.16:0/2586299625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d008fef0 msgr2=0x7f31d008f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.924+0000 7f31d6ef6700 1 --2- 172.21.15.16:0/2586299625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d008fef0 0x7f31d008f720 secure :-1 s=READY pgs=1698 cs=0 l=1 rx=0x7f31c000af50 tx=0x7f31c000b760).stop 2019-12-05T06:59:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.925+0000 7f31d6ef6700 1 -- 172.21.15.16:0/2586299625 shutdown_connections 2019-12-05T06:59:00.933 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.925+0000 7f31d6ef6700 1 --2- 172.21.15.16:0/2586299625 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f31d008fef0 0x7f31d008f720 unknown :-1 s=CLOSED pgs=1698 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.926+0000 7f31d6ef6700 1 -- 172.21.15.16:0/2586299625 shutdown_connections 2019-12-05T06:59:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.926+0000 7f31d6ef6700 1 -- 172.21.15.16:0/2586299625 wait complete. 2019-12-05T06:59:00.934 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.926+0000 7f31d6ef6700 1 -- 172.21.15.16:0/2586299625 >> 172.21.15.16:0/2586299625 conn(0x7f31d0061940 msgr2=0x7f31d0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:00.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.931+0000 7f31d6ef6700 1 Processor -- start 2019-12-05T06:59:00.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.931+0000 7f31d6ef6700 1 -- start start 2019-12-05T06:59:00.935 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.933+0000 7f31d6ef6700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:00.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.933+0000 7f31d6ef6700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d006e7c0 con 0x7f31d0132630 2019-12-05T06:59:00.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.933+0000 7f31d6ef6700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31d0133b30 con 0x7f31d0099eb0 2019-12-05T06:59:00.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.933+0000 7f31d6ef6700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31d0098f20 con 0x7f31d009ae80 2019-12-05T06:59:00.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.933+0000 7f31d5493700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f31d0132630 0x7f31d0132a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50710/0 (socket says 172.21.15.16:50710) 2019-12-05T06:59:00.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.933+0000 7f31d5493700 1 -- 172.21.15.16:0/325947983 learned_addr learned my addr 172.21.15.16:0/325947983 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:00.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.933+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:00.936 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.934+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 882867336 0 0) 0x7f31c80014f0 con 0x7f31d0132630 2019-12-05T06:59:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.934+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b0002e80 con 0x7f31d0132630 2019-12-05T06:59:00.937 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.934+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326519889 0 0) 0x7f31c0000940 con 0x7f31d0099eb0 2019-12-05T06:59:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.934+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31b00030b0 con 0x7f31d0099eb0 2019-12-05T06:59:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.934+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 2848983490 0 0) 0x7f31c80015c0 con 0x7f31d0132630 2019-12-05T06:59:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.934+0000 7f31d4c92700 1 -- 172.21.15.16:0/325947983 >> v1:172.21.15.76:6789/0 conn(0x7f31d0099eb0 legacy=0x7f31d009a370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.934+0000 7f31d4c92700 1 -- 172.21.15.16:0/325947983 >> v1:172.21.15.16:6789/0 conn(0x7f31d0132630 legacy=0x7f31d0132a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.934+0000 7f31d4c92700 1 -- 172.21.15.16:0/325947983 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31d00995b0 con 0x7f31d009ae80 2019-12-05T06:59:00.941 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.934+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.934+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.935+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=READY pgs=402 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.935+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.935+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31c4004240 con 0x7f31d009ae80 2019-12-05T06:59:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.935+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.935+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.942 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.935+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31c4000c90 con 0x7f31d009ae80 2019-12-05T06:59:00.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.935+0000 7f31d6ef6700 1 -- 172.21.15.16:0/325947983 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31d00995b0 con 0x7f31d009ae80 2019-12-05T06:59:00.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.935+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f31c4004820 con 0x7f31d009ae80 2019-12-05T06:59:00.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.935+0000 7f31d6ef6700 1 -- 172.21.15.16:0/325947983 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31d00995b0 con 0x7f31d009ae80 2019-12-05T06:59:00.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.936+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.937+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f31c4004240 con 0x7f31d009ae80 2019-12-05T06:59:00.943 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.937+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.937+0000 7f31cdffb700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f31b0023650 0x7f31b0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.937+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f31c4045dc0 con 0x7f31d009ae80 2019-12-05T06:59:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.937+0000 7f31d5493700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f31b0023650 0x7f31b0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:00.944 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.938+0000 7f31d5493700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f31b0023650 0x7f31b0025ae0 crc :-1 s=READY pgs=709 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:00.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.951+0000 7f31d6ef6700 1 -- 172.21.15.16:0/325947983 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31d00995b0 con 0x7f31d009ae80 2019-12-05T06:59:00.961 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.961+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:00.962 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:00.961+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f31c4033100 con 0x7f31d009ae80 2019-12-05T06:59:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.211+0000 7f31d6ef6700 1 -- 172.21.15.16:0/325947983 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f31d00995b0 con 0x7f31d009ae80 2019-12-05T06:59:01.212 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.212+0000 7f31d4c92700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=THROTTLE_DONE pgs=402 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:01.213 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.212+0000 7f31cdffb700 1 -- 172.21.15.16:0/325947983 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f31c4033100 con 0x7f31d009ae80 2019-12-05T06:59:01.215 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 207 2019-12-05T06:59:01.215 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:01.215 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8523,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":207}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:01.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.217+0000 7f31d6ef6700 1 -- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f31b0023650 msgr2=0x7f31b0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:01.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.217+0000 7f31d6ef6700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f31b0023650 0x7f31b0025ae0 crc :-1 s=READY pgs=709 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:01.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.217+0000 7f31d6ef6700 1 -- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 msgr2=0x7f31d0133220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:01.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.217+0000 7f31d6ef6700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 secure :-1 s=READY pgs=402 cs=0 l=1 rx=0x7f31c4010370 tx=0x7f31c4010710).stop 2019-12-05T06:59:01.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.218+0000 7f31d6ef6700 1 -- 172.21.15.16:0/325947983 shutdown_connections 2019-12-05T06:59:01.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.218+0000 7f31d6ef6700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f31b0023650 0x7f31b0025ae0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:01.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.218+0000 7f31d6ef6700 1 --2- 172.21.15.16:0/325947983 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f31d009ae80 0x7f31d0133220 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:01.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.218+0000 7f31d6ef6700 1 -- 172.21.15.16:0/325947983 shutdown_connections 2019-12-05T06:59:01.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.218+0000 7f31d6ef6700 1 -- 172.21.15.16:0/325947983 wait complete. 2019-12-05T06:59:01.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:01.218+0000 7f31d6ef6700 1 -- 172.21.15.16:0/325947983 >> 172.21.15.16:0/325947983 conn(0x7f31d00714f0 msgr2=0x7f31d009ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:01.231 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:59:03.232 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.492+0000 7f75fb8dd700 1 Processor -- start 2019-12-05T06:59:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.492+0000 7f75fb8dd700 1 -- start start 2019-12-05T06:59:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.494+0000 7f75fb8dd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f75f408c6d0 0x7f75f408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.494+0000 7f75fb8dd700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f408fb80 con 0x7f75f408f720 2019-12-05T06:59:03.502 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.494+0000 7f75fb8dd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f40958e0 con 0x7f75f408fef0 2019-12-05T06:59:03.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.494+0000 7f75fb8dd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f408df90 con 0x7f75f408c6d0 2019-12-05T06:59:03.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.494+0000 7f75fa67b700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f75f408f720 0x7f75f41533d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50726/0 (socket says 172.21.15.16:50726) 2019-12-05T06:59:03.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.494+0000 7f75fa67b700 1 -- 172.21.15.16:0/669497873 learned_addr learned my addr 172.21.15.16:0/669497873 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:03.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.494+0000 7f75f9e7a700 1 --2- 172.21.15.16:0/669497873 >> v2:172.21.15.76:3300/0 conn(0x7f75f408c6d0 0x7f75f408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:03.503 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.495+0000 7f75f8e78700 1 -- 172.21.15.16:0/669497873 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595438254 0 0) 0x7f75e4001150 con 0x7f75f408f720 2019-12-05T06:59:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.495+0000 7f75f8e78700 1 -- 172.21.15.16:0/669497873 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75dc002e80 con 0x7f75f408f720 2019-12-05T06:59:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.495+0000 7f75f9e7a700 1 -- 172.21.15.16:0/669497873 >> v1:172.21.15.76:6789/0 conn(0x7f75f408fef0 legacy=0x7f75f408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.495+0000 7f75f8e78700 1 -- 172.21.15.16:0/669497873 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2515974982 0 0) 0x7f75e8001400 con 0x7f75f408fef0 2019-12-05T06:59:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.495+0000 7f75f9e7a700 1 -- 172.21.15.16:0/669497873 >> v1:172.21.15.16:6789/0 conn(0x7f75f408f720 legacy=0x7f75f41533d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:03.504 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.495+0000 7f75f9e7a700 1 -- 172.21.15.16:0/669497873 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f408e620 con 0x7f75f408c6d0 2019-12-05T06:59:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.495+0000 7f75f9e7a700 1 --2- 172.21.15.16:0/669497873 >> v2:172.21.15.76:3300/0 conn(0x7f75f408c6d0 0x7f75f408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f75ec00af50 tx=0x7f75ec00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.495+0000 7f75f9e7a700 1 --2- 172.21.15.16:0/669497873 >> v2:172.21.15.76:3300/0 conn(0x7f75f408c6d0 0x7f75f408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f75ec00af50 tx=0x7f75ec00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.495+0000 7f75f9e7a700 1 --2- 172.21.15.16:0/669497873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75f408c6d0 0x7f75f408cac0 secure :-1 s=READY pgs=1148 cs=0 l=1 rx=0x7f75ec00af50 tx=0x7f75ec00b760).ready entity=mon.1 client_cookie=fdbbdfe57be4ff3e server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.496+0000 7f75f9e7a700 1 --2- 172.21.15.16:0/669497873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75f408c6d0 0x7f75f408cac0 secure :-1 s=THROTTLE_DONE pgs=1148 cs=0 l=1 rx=0x7f75ec00af50 tx=0x7f75ec00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.496+0000 7f75f8e78700 1 -- 172.21.15.16:0/669497873 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75ec000960 con 0x7f75f408c6d0 2019-12-05T06:59:03.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.496+0000 7f75f9e7a700 1 --2- 172.21.15.16:0/669497873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75f408c6d0 0x7f75f408cac0 secure :-1 s=THROTTLE_DONE pgs=1148 cs=0 l=1 rx=0x7f75ec00af50 tx=0x7f75ec00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.496+0000 7f75f9e7a700 1 --2- 172.21.15.16:0/669497873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75f408c6d0 0x7f75f408cac0 secure :-1 s=THROTTLE_DONE pgs=1148 cs=0 l=1 rx=0x7f75ec00af50 tx=0x7f75ec00b760).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.496+0000 7f75f8e78700 1 -- 172.21.15.16:0/669497873 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75ec009860 con 0x7f75f408c6d0 2019-12-05T06:59:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.496+0000 7f75f8e78700 1 -- 172.21.15.16:0/669497873 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75ec000960 con 0x7f75f408c6d0 2019-12-05T06:59:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.496+0000 7f75fb8dd700 1 -- 172.21.15.16:0/669497873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75f408c6d0 msgr2=0x7f75f408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.496+0000 7f75fb8dd700 1 --2- 172.21.15.16:0/669497873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75f408c6d0 0x7f75f408cac0 secure :-1 s=READY pgs=1148 cs=0 l=1 rx=0x7f75ec00af50 tx=0x7f75ec00b760).stop 2019-12-05T06:59:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.498+0000 7f75fb8dd700 1 -- 172.21.15.16:0/669497873 shutdown_connections 2019-12-05T06:59:03.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.498+0000 7f75fb8dd700 1 --2- 172.21.15.16:0/669497873 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f75f408c6d0 0x7f75f408cac0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.498+0000 7f75fb8dd700 1 -- 172.21.15.16:0/669497873 shutdown_connections 2019-12-05T06:59:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.498+0000 7f75fb8dd700 1 -- 172.21.15.16:0/669497873 wait complete. 2019-12-05T06:59:03.507 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.498+0000 7f75fb8dd700 1 -- 172.21.15.16:0/669497873 >> 172.21.15.16:0/669497873 conn(0x7f75f4061940 msgr2=0x7f75f4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.503+0000 7f75fb8dd700 1 Processor -- start 2019-12-05T06:59:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.503+0000 7f75fb8dd700 1 -- start start 2019-12-05T06:59:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.504+0000 7f75fb8dd700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f75f412b640 0x7f75f412ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.504+0000 7f75fb8dd700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75f4153ba0 0x7f75f4155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:03.509 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.504+0000 7f75fb8dd700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75f4131bf0 con 0x7f75f41302f0 2019-12-05T06:59:03.510 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.504+0000 7f75fb8dd700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f41296e0 con 0x7f75f4153ba0 2019-12-05T06:59:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.504+0000 7f75fb8dd700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75f4128cb0 con 0x7f75f412b640 2019-12-05T06:59:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.504+0000 7f75fa67b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75f4153ba0 0x7f75f4155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.504+0000 7f75fa67b700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f75f4153ba0 0x7f75f4155fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37332/0 (socket says 172.21.15.16:37332) 2019-12-05T06:59:03.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.504+0000 7f75fa67b700 1 -- 172.21.15.16:0/1032942244 learned_addr learned my addr 172.21.15.16:0/1032942244 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:59:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.504+0000 7f75f9679700 1 --2- 172.21.15.16:0/1032942244 >> v2:172.21.15.76:3300/0 conn(0x7f75f412b640 0x7f75f412ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.505+0000 7f75fa67b700 1 -- 172.21.15.16:0/1032942244 >> v2:172.21.15.76:3300/0 conn(0x7f75f412b640 msgr2=0x7f75f412ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.505+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> v2:172.21.15.76:3300/0 conn(0x7f75f412b640 0x7f75f412ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:03.514 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.505+0000 7f75fa67b700 1 -- 172.21.15.16:0/1032942244 >> v1:172.21.15.76:6789/0 conn(0x7f75f41302f0 legacy=0x7f75f412a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.505+0000 7f75fa67b700 1 -- 172.21.15.16:0/1032942244 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f4129d70 con 0x7f75f4153ba0 2019-12-05T06:59:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.505+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> v2:172.21.15.16:3300/0 conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.505+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> v2:172.21.15.16:3300/0 conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.505+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=READY pgs=1701 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.506+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=THROTTLE_DONE pgs=1701 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.515 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.506+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=THROTTLE_DONE pgs=1701 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.506+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=THROTTLE_DONE pgs=1701 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.506+0000 7f75f27fc700 1 -- 172.21.15.16:0/1032942244 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e4000ef0 con 0x7f75f4153ba0 2019-12-05T06:59:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.506+0000 7f75f27fc700 1 -- 172.21.15.16:0/1032942244 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75e4004240 con 0x7f75f4153ba0 2019-12-05T06:59:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.506+0000 7f75fb8dd700 1 -- 172.21.15.16:0/1032942244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75f4129d70 con 0x7f75f4153ba0 2019-12-05T06:59:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.506+0000 7f75f27fc700 1 -- 172.21.15.16:0/1032942244 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75e4004950 con 0x7f75f4153ba0 2019-12-05T06:59:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.506+0000 7f75fb8dd700 1 -- 172.21.15.16:0/1032942244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75f4129d70 con 0x7f75f4153ba0 2019-12-05T06:59:03.516 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.507+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=THROTTLE_DONE pgs=1701 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.508+0000 7f75f27fc700 1 -- 172.21.15.16:0/1032942244 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f75e4004430 con 0x7f75f4153ba0 2019-12-05T06:59:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.508+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=THROTTLE_DONE pgs=1701 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.508+0000 7f75f27fc700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d4023630 0x7f75d4025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:03.517 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.508+0000 7f75f27fc700 1 -- 172.21.15.16:0/1032942244 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f75e4045e00 con 0x7f75f4153ba0 2019-12-05T06:59:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.508+0000 7f75f9e7a700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d4023630 0x7f75d4025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:03.518 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.509+0000 7f75f9e7a700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d4023630 0x7f75d4025ac0 crc :-1 s=READY pgs=710 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:03.523 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.523+0000 7f75fb8dd700 1 -- 172.21.15.16:0/1032942244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75f4129d70 con 0x7f75f4153ba0 2019-12-05T06:59:03.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.532+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=THROTTLE_DONE pgs=1701 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.532 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.532+0000 7f75f27fc700 1 -- 172.21.15.16:0/1032942244 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f75e4000940 con 0x7f75f4153ba0 2019-12-05T06:59:03.765 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.765+0000 7f75fb8dd700 1 -- 172.21.15.16:0/1032942244 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f75f4129d70 con 0x7f75f4153ba0 2019-12-05T06:59:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.766+0000 7f75fa67b700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=THROTTLE_DONE pgs=1701 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.766+0000 7f75f27fc700 1 -- 172.21.15.16:0/1032942244 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f75e4004430 con 0x7f75f4153ba0 2019-12-05T06:59:03.766 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 207 2019-12-05T06:59:03.768 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:03.768 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8523,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":207}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:03.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.769+0000 7f75fb8dd700 1 -- 172.21.15.16:0/1032942244 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d4023630 msgr2=0x7f75d4025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:03.769 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.769+0000 7f75fb8dd700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d4023630 0x7f75d4025ac0 crc :-1 s=READY pgs=710 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:03.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.769+0000 7f75fb8dd700 1 -- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 msgr2=0x7f75f4155fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:03.770 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.769+0000 7f75fb8dd700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 0x7f75f4155fa0 secure :-1 s=READY pgs=1701 cs=0 l=1 rx=0x7f75e4010430 tx=0x7f75e4010260).stop 2019-12-05T06:59:03.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.770+0000 7f75fb8dd700 1 -- 172.21.15.16:0/1032942244 shutdown_connections 2019-12-05T06:59:03.771 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.770+0000 7f75fb8dd700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f75d4023630 0x7f75d4025ac0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.770+0000 7f75fb8dd700 1 --2- 172.21.15.16:0/1032942244 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f75f4153ba0 0x7f75f4155fa0 unknown :-1 s=CLOSED pgs=1701 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.770+0000 7f75fb8dd700 1 --2- 172.21.15.16:0/1032942244 >> v2:172.21.15.76:3300/0 conn(0x7f75f412b640 0x7f75f412ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.770+0000 7f75fb8dd700 1 -- 172.21.15.16:0/1032942244 shutdown_connections 2019-12-05T06:59:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.770+0000 7f75fb8dd700 1 -- 172.21.15.16:0/1032942244 wait complete. 2019-12-05T06:59:03.772 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:03.770+0000 7f75fb8dd700 1 -- 172.21.15.16:0/1032942244 >> 172.21.15.16:0/1032942244 conn(0x7f75f40a2850 msgr2=0x7f75f406ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:03.789 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:59:05.790 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:06.084 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.073+0000 7fc420828700 1 Processor -- start 2019-12-05T06:59:06.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.073+0000 7fc420828700 1 -- start start 2019-12-05T06:59:06.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.075+0000 7fc420828700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c08d2c0 0x7fc41c08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:06.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.075+0000 7fc420828700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc41c08fef0 0x7fc41c08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:06.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.075+0000 7fc420828700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c0958e0 con 0x7fc41c08c6d0 2019-12-05T06:59:06.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.075+0000 7fc420828700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41c08df90 con 0x7fc41c08fef0 2019-12-05T06:59:06.085 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.075+0000 7fc420828700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c0a3f70 con 0x7fc41c08d2c0 2019-12-05T06:59:06.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.076+0000 7fc41b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc41c08fef0 0x7fc41c08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:06.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.076+0000 7fc41a59c700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c08d2c0 0x7fc41c08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:06.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.076+0000 7fc41ad9d700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7fc41c08c6d0 0x7fc41c08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:43822/0 (socket says 172.21.15.16:43822) 2019-12-05T06:59:06.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.076+0000 7fc41ad9d700 1 -- 172.21.15.16:0/2734781730 learned_addr learned my addr 172.21.15.16:0/2734781730 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:06.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.076+0000 7fc419d9b700 1 -- 172.21.15.16:0/2734781730 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597903139 0 0) 0x7fc40c001100 con 0x7fc41c08c6d0 2019-12-05T06:59:06.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.076+0000 7fc419d9b700 1 -- 172.21.15.16:0/2734781730 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc408002e80 con 0x7fc41c08c6d0 2019-12-05T06:59:06.086 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41b59e700 1 -- 172.21.15.16:0/2734781730 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c08d2c0 msgr2=0x7fc41c08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:06.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2734781730 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c08d2c0 0x7fc41c08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:06.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41b59e700 1 -- 172.21.15.16:0/2734781730 >> v1:172.21.15.76:6789/0 conn(0x7fc41c08c6d0 legacy=0x7fc41c08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:06.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41b59e700 1 -- 172.21.15.16:0/2734781730 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c08e620 con 0x7fc41c08fef0 2019-12-05T06:59:06.087 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41a59c700 1 --2- 172.21.15.16:0/2734781730 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c08d2c0 0x7fc41c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:59:06.088 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2734781730 >> v2:172.21.15.16:3300/0 conn(0x7fc41c08fef0 0x7fc41c08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fc41400b2d0 tx=0x7fc41400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2734781730 >> v2:172.21.15.16:3300/0 conn(0x7fc41c08fef0 0x7fc41c08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fc41400b2d0 tx=0x7fc41400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2734781730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c08fef0 0x7fc41c08f460 secure :-1 s=READY pgs=1702 cs=0 l=1 rx=0x7fc41400b2d0 tx=0x7fc41400bae0).ready entity=mon.0 client_cookie=a6c7380327427221 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:06.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2734781730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c08fef0 0x7fc41c08f460 secure :-1 s=THROTTLE_DONE pgs=1702 cs=0 l=1 rx=0x7fc41400b2d0 tx=0x7fc41400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc419d9b700 1 -- 172.21.15.16:0/2734781730 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc414009f20 con 0x7fc41c08fef0 2019-12-05T06:59:06.089 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2734781730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c08fef0 0x7fc41c08f460 secure :-1 s=THROTTLE_DONE pgs=1702 cs=0 l=1 rx=0x7fc41400b2d0 tx=0x7fc41400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.077+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2734781730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c08fef0 0x7fc41c08f460 secure :-1 s=THROTTLE_DONE pgs=1702 cs=0 l=1 rx=0x7fc41400b2d0 tx=0x7fc41400bae0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.078+0000 7fc419d9b700 1 -- 172.21.15.16:0/2734781730 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc414009b40 con 0x7fc41c08fef0 2019-12-05T06:59:06.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.078+0000 7fc419d9b700 1 -- 172.21.15.16:0/2734781730 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc414000f40 con 0x7fc41c08fef0 2019-12-05T06:59:06.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.078+0000 7fc420828700 1 -- 172.21.15.16:0/2734781730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c08fef0 msgr2=0x7fc41c08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:06.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.078+0000 7fc420828700 1 --2- 172.21.15.16:0/2734781730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c08fef0 0x7fc41c08f460 secure :-1 s=READY pgs=1702 cs=0 l=1 rx=0x7fc41400b2d0 tx=0x7fc41400bae0).stop 2019-12-05T06:59:06.090 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.080+0000 7fc420828700 1 -- 172.21.15.16:0/2734781730 shutdown_connections 2019-12-05T06:59:06.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.080+0000 7fc420828700 1 --2- 172.21.15.16:0/2734781730 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c08fef0 0x7fc41c08f460 unknown :-1 s=CLOSED pgs=1702 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:06.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.080+0000 7fc420828700 1 --2- 172.21.15.16:0/2734781730 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c08d2c0 0x7fc41c08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:06.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.080+0000 7fc420828700 1 -- 172.21.15.16:0/2734781730 shutdown_connections 2019-12-05T06:59:06.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.080+0000 7fc420828700 1 -- 172.21.15.16:0/2734781730 wait complete. 2019-12-05T06:59:06.091 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.080+0000 7fc420828700 1 -- 172.21.15.16:0/2734781730 >> 172.21.15.16:0/2734781730 conn(0x7fc41c061940 msgr2=0x7fc41c096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:06.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.085+0000 7fc420828700 1 Processor -- start 2019-12-05T06:59:06.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.085+0000 7fc420828700 1 -- start start 2019-12-05T06:59:06.093 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.087+0000 7fc420828700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c09bbe0 0x7fc41c1539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.087+0000 7fc420828700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc41c084f30 0x7fc41c155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.087+0000 7fc420828700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc41c084b90 con 0x7fc41c096e90 2019-12-05T06:59:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.087+0000 7fc420828700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc41c09a450 con 0x7fc41c084f30 2019-12-05T06:59:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.087+0000 7fc420828700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc41c099a20 con 0x7fc41c09bbe0 2019-12-05T06:59:06.096 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.087+0000 7fc41b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc41c084f30 0x7fc41c155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.087+0000 7fc41b59e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fc41c084f30 0x7fc41c155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37362/0 (socket says 172.21.15.16:37362) 2019-12-05T06:59:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.087+0000 7fc41b59e700 1 -- 172.21.15.16:0/2592697986 learned_addr learned my addr 172.21.15.16:0/2592697986 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:59:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.087+0000 7fc41a59c700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c09bbe0 0x7fc41c1539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.088+0000 7fc4137fe700 1 -- 172.21.15.16:0/2592697986 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 523801088 0 0) 0x7fc40c001080 con 0x7fc41c096e90 2019-12-05T06:59:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.088+0000 7fc41b59e700 1 -- 172.21.15.16:0/2592697986 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c09bbe0 msgr2=0x7fc41c1539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.088+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c09bbe0 0x7fc41c1539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:06.097 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.088+0000 7fc41b59e700 1 -- 172.21.15.16:0/2592697986 >> v1:172.21.15.76:6789/0 conn(0x7fc41c096e90 legacy=0x7fc41c09b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:06.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.088+0000 7fc41b59e700 1 -- 172.21.15.16:0/2592697986 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc41c09aae0 con 0x7fc41c084f30 2019-12-05T06:59:06.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.088+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> v2:172.21.15.16:3300/0 conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.089+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> v2:172.21.15.16:3300/0 conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.089+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=READY pgs=1703 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:06.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.089+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=THROTTLE_DONE pgs=1703 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.089+0000 7fc4137fe700 1 -- 172.21.15.16:0/2592697986 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc414009900 con 0x7fc41c084f30 2019-12-05T06:59:06.098 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.089+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=THROTTLE_DONE pgs=1703 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.089+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=THROTTLE_DONE pgs=1703 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.099 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.089+0000 7fc4137fe700 1 -- 172.21.15.16:0/2592697986 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc414000cc0 con 0x7fc41c084f30 2019-12-05T06:59:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.089+0000 7fc420828700 1 -- 172.21.15.16:0/2592697986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc41c09aae0 con 0x7fc41c084f30 2019-12-05T06:59:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.089+0000 7fc4137fe700 1 -- 172.21.15.16:0/2592697986 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc414009f80 con 0x7fc41c084f30 2019-12-05T06:59:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.089+0000 7fc420828700 1 -- 172.21.15.16:0/2592697986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc41c09aae0 con 0x7fc41c084f30 2019-12-05T06:59:06.117 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.090+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=THROTTLE_DONE pgs=1703 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.091+0000 7fc4137fe700 1 -- 172.21.15.16:0/2592697986 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fc414009900 con 0x7fc41c084f30 2019-12-05T06:59:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.091+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=THROTTLE_DONE pgs=1703 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.092+0000 7fc4137fe700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc400023630 0x7fc400025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.092+0000 7fc4137fe700 1 -- 172.21.15.16:0/2592697986 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7fc414046150 con 0x7fc41c084f30 2019-12-05T06:59:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.092+0000 7fc41ad9d700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc400023630 0x7fc400025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:06.118 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.093+0000 7fc41ad9d700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc400023630 0x7fc400025ac0 crc :-1 s=READY pgs=711 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:06.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.105+0000 7fc420828700 1 -- 172.21.15.16:0/2592697986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc41c09aae0 con 0x7fc41c084f30 2019-12-05T06:59:06.119 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.113+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=THROTTLE_DONE pgs=1703 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.120 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.113+0000 7fc4137fe700 1 -- 172.21.15.16:0/2592697986 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fc41400aa60 con 0x7fc41c084f30 2019-12-05T06:59:06.360 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.359+0000 7fc420828700 1 -- 172.21.15.16:0/2592697986 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc41c09aae0 con 0x7fc41c084f30 2019-12-05T06:59:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.360+0000 7fc41b59e700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=THROTTLE_DONE pgs=1703 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.360+0000 7fc4137fe700 1 -- 172.21.15.16:0/2592697986 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fc41400a1b0 con 0x7fc41c084f30 2019-12-05T06:59:06.366 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 207 2019-12-05T06:59:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 -- 172.21.15.16:0/2592697986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc400023630 msgr2=0x7fc400025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc400023630 0x7fc400025ac0 crc :-1 s=READY pgs=711 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 -- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 msgr2=0x7fc41c155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 0x7fc41c155f30 secure :-1 s=READY pgs=1703 cs=0 l=1 rx=0x7fc414010770 tx=0x7fc414010b10).stop 2019-12-05T06:59:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 -- 172.21.15.16:0/2592697986 shutdown_connections 2019-12-05T06:59:06.367 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fc400023630 0x7fc400025ac0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fc41c084f30 0x7fc41c155f30 unknown :-1 s=CLOSED pgs=1703 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 --2- 172.21.15.16:0/2592697986 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fc41c09bbe0 0x7fc41c1539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 -- 172.21.15.16:0/2592697986 shutdown_connections 2019-12-05T06:59:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 -- 172.21.15.16:0/2592697986 wait complete. 2019-12-05T06:59:06.368 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:06.363+0000 7fc420828700 1 -- 172.21.15.16:0/2592697986 >> 172.21.15.16:0/2592697986 conn(0x7fc41c0714f0 msgr2=0x7fc41c09ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:06.368 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:06.368 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8523,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":207}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:06.374 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:59:08.375 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:08.656 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.639+0000 7f8ffbe65700 1 Processor -- start 2019-12-05T06:59:08.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.640+0000 7f8ffbe65700 1 -- start start 2019-12-05T06:59:08.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.642+0000 7f8ffbe65700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ff408c6d0 0x7f8ff408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:08.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.642+0000 7f8ffbe65700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff408f720 0x7f8ff408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:08.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.642+0000 7f8ffbe65700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff408fb70 con 0x7f8ff408fef0 2019-12-05T06:59:08.657 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.642+0000 7f8ffbe65700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff408df90 con 0x7f8ff408c6d0 2019-12-05T06:59:08.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.642+0000 7f8ffbe65700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff40a3f70 con 0x7f8ff408f720 2019-12-05T06:59:08.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.643+0000 7f8ffa402700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f8ff408c6d0 0x7f8ff408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:08.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.643+0000 7f8ff9c01700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8ff408fef0 0x7f8ff408ef20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50772/0 (socket says 172.21.15.16:50772) 2019-12-05T06:59:08.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.643+0000 7f8ff9c01700 1 -- 172.21.15.16:0/1805113657 learned_addr learned my addr 172.21.15.16:0/1805113657 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:08.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.643+0000 7f8ffac03700 1 --2- 172.21.15.16:0/1805113657 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff408f720 0x7f8ff408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:08.658 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.644+0000 7f8ffa402700 1 -- 172.21.15.16:0/1805113657 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff408f720 msgr2=0x7f8ff408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.644+0000 7f8ffa402700 1 --2- 172.21.15.16:0/1805113657 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff408f720 0x7f8ff408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.644+0000 7f8ffa402700 1 -- 172.21.15.16:0/1805113657 >> v1:172.21.15.16:6789/0 conn(0x7f8ff408fef0 legacy=0x7f8ff408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.644+0000 7f8ff9400700 1 -- 172.21.15.16:0/1805113657 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1275438785 0 0) 0x7f8fe8001100 con 0x7f8ff408fef0 2019-12-05T06:59:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.644+0000 7f8ffa402700 1 -- 172.21.15.16:0/1805113657 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff408d540 con 0x7f8ff408c6d0 2019-12-05T06:59:08.659 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.644+0000 7f8ffa402700 1 --2- 172.21.15.16:0/1805113657 >> v2:172.21.15.16:3300/0 conn(0x7f8ff408c6d0 0x7f8ff408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f8fec00b2d0 tx=0x7f8fec00ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.644+0000 7f8ffa402700 1 --2- 172.21.15.16:0/1805113657 >> v2:172.21.15.16:3300/0 conn(0x7f8ff408c6d0 0x7f8ff408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f8fec00b2d0 tx=0x7f8fec00ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.644+0000 7f8ffa402700 1 --2- 172.21.15.16:0/1805113657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff408c6d0 0x7f8ff408cac0 secure :-1 s=READY pgs=1705 cs=0 l=1 rx=0x7f8fec00b2d0 tx=0x7f8fec00ba60).ready entity=mon.0 client_cookie=66936d4517899668 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.645+0000 7f8ffa402700 1 --2- 172.21.15.16:0/1805113657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff408c6d0 0x7f8ff408cac0 secure :-1 s=THROTTLE_DONE pgs=1705 cs=0 l=1 rx=0x7f8fec00b2d0 tx=0x7f8fec00ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.645+0000 7f8ffa402700 1 --2- 172.21.15.16:0/1805113657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff408c6d0 0x7f8ff408cac0 secure :-1 s=THROTTLE_DONE pgs=1705 cs=0 l=1 rx=0x7f8fec00b2d0 tx=0x7f8fec00ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.660 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.645+0000 7f8ff9400700 1 -- 172.21.15.16:0/1805113657 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fec007590 con 0x7f8ff408c6d0 2019-12-05T06:59:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.645+0000 7f8ffa402700 1 --2- 172.21.15.16:0/1805113657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff408c6d0 0x7f8ff408cac0 secure :-1 s=THROTTLE_DONE pgs=1705 cs=0 l=1 rx=0x7f8fec00b2d0 tx=0x7f8fec00ba60).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.645+0000 7f8ff9400700 1 -- 172.21.15.16:0/1805113657 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fec009d30 con 0x7f8ff408c6d0 2019-12-05T06:59:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.645+0000 7f8ff9400700 1 -- 172.21.15.16:0/1805113657 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fec00a400 con 0x7f8ff408c6d0 2019-12-05T06:59:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.645+0000 7f8ffbe65700 1 -- 172.21.15.16:0/1805113657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff408c6d0 msgr2=0x7f8ff408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:08.661 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.645+0000 7f8ffbe65700 1 --2- 172.21.15.16:0/1805113657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff408c6d0 0x7f8ff408cac0 secure :-1 s=READY pgs=1705 cs=0 l=1 rx=0x7f8fec00b2d0 tx=0x7f8fec00ba60).stop 2019-12-05T06:59:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.647+0000 7f8ffbe65700 1 -- 172.21.15.16:0/1805113657 shutdown_connections 2019-12-05T06:59:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.647+0000 7f8ffbe65700 1 --2- 172.21.15.16:0/1805113657 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff408f720 0x7f8ff408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.647+0000 7f8ffbe65700 1 --2- 172.21.15.16:0/1805113657 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f8ff408c6d0 0x7f8ff408cac0 unknown :-1 s=CLOSED pgs=1705 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:08.662 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.648+0000 7f8ffbe65700 1 -- 172.21.15.16:0/1805113657 shutdown_connections 2019-12-05T06:59:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.648+0000 7f8ffbe65700 1 -- 172.21.15.16:0/1805113657 wait complete. 2019-12-05T06:59:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.648+0000 7f8ffbe65700 1 -- 172.21.15.16:0/1805113657 >> 172.21.15.16:0/1805113657 conn(0x7f8ff4061940 msgr2=0x7f8ff4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:08.663 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.654+0000 7f8ffbe65700 1 Processor -- start 2019-12-05T06:59:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.654+0000 7f8ffbe65700 1 -- start start 2019-12-05T06:59:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.655+0000 7f8ffbe65700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f8ff406df10 0x7f8ff4098300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.655+0000 7f8ffbe65700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:08.664 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.655+0000 7f8ffbe65700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ff4089fb0 con 0x7f8ff40992d0 2019-12-05T06:59:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.655+0000 7f8ffbe65700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ff4097370 con 0x7f8ff406df10 2019-12-05T06:59:08.665 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.655+0000 7f8ffbe65700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ff408ad00 con 0x7f8ff4156cb0 2019-12-05T06:59:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.655+0000 7f8ff9c01700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f8ff40992d0 0x7f8ff4098840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50774/0 (socket says 172.21.15.16:50774) 2019-12-05T06:59:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.655+0000 7f8ff9c01700 1 -- 172.21.15.16:0/2425204257 learned_addr learned my addr 172.21.15.16:0/2425204257 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.656+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.656+0000 7f8ffa402700 1 --2- 172.21.15.16:0/2425204257 >> v2:172.21.15.76:3300/0 conn(0x7f8ff406df10 0x7f8ff4098300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:08.668 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.656+0000 7f8ff2ffd700 1 -- 172.21.15.16:0/2425204257 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2169442860 0 0) 0x7f8fe8001150 con 0x7f8ff40992d0 2019-12-05T06:59:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.656+0000 7f8ff2ffd700 1 -- 172.21.15.16:0/2425204257 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fd4002e80 con 0x7f8ff40992d0 2019-12-05T06:59:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.656+0000 7f8ffac03700 1 -- 172.21.15.16:0/2425204257 >> v2:172.21.15.76:3300/0 conn(0x7f8ff406df10 msgr2=0x7f8ff4098300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.656+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> v2:172.21.15.76:3300/0 conn(0x7f8ff406df10 0x7f8ff4098300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.656+0000 7f8ffac03700 1 -- 172.21.15.16:0/2425204257 >> v1:172.21.15.16:6789/0 conn(0x7f8ff40992d0 legacy=0x7f8ff4098840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:08.669 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.656+0000 7f8ffac03700 1 -- 172.21.15.16:0/2425204257 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ff4097a00 con 0x7f8ff4156cb0 2019-12-05T06:59:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.656+0000 7f8ffa402700 1 --2- 172.21.15.16:0/2425204257 >> v2:172.21.15.76:3300/0 conn(0x7f8ff406df10 0x7f8ff4098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:59:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=READY pgs=403 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.670 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ff2ffd700 1 -- 172.21.15.16:0/2425204257 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fe4009990 con 0x7f8ff4156cb0 2019-12-05T06:59:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ff2ffd700 1 -- 172.21.15.16:0/2425204257 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fe40095b0 con 0x7f8ff4156cb0 2019-12-05T06:59:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ffbe65700 1 -- 172.21.15.16:0/2425204257 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ff4097a00 con 0x7f8ff4156cb0 2019-12-05T06:59:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ff2ffd700 1 -- 172.21.15.16:0/2425204257 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fe4009f70 con 0x7f8ff4156cb0 2019-12-05T06:59:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.657+0000 7f8ffbe65700 1 -- 172.21.15.16:0/2425204257 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ff4097a00 con 0x7f8ff4156cb0 2019-12-05T06:59:08.671 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.658+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.659+0000 7f8ff2ffd700 1 -- 172.21.15.16:0/2425204257 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f8fe400ea80 con 0x7f8ff4156cb0 2019-12-05T06:59:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.659+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.660+0000 7f8ff2ffd700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8fd4023670 0x7f8fd4025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.660+0000 7f8ff2ffd700 1 -- 172.21.15.16:0/2425204257 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7f8fe4045cd0 con 0x7f8ff4156cb0 2019-12-05T06:59:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.660+0000 7f8ffa402700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8fd4023670 0x7f8fd4025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:08.672 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.661+0000 7f8ffa402700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8fd4023670 0x7f8fd4025ae0 crc :-1 s=READY pgs=712 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:08.673 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.671+0000 7f8ffbe65700 1 -- 172.21.15.16:0/2425204257 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ff4097a00 con 0x7f8ff4156cb0 2019-12-05T06:59:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.681+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.681 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.681+0000 7f8ff2ffd700 1 -- 172.21.15.16:0/2425204257 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f8fe4009760 con 0x7f8ff4156cb0 2019-12-05T06:59:08.919 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.919+0000 7f8ffbe65700 1 -- 172.21.15.16:0/2425204257 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8ff4097a00 con 0x7f8ff4156cb0 2019-12-05T06:59:08.920 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.920+0000 7f8ffac03700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=THROTTLE_DONE pgs=403 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:08.921 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.920+0000 7f8ff2ffd700 1 -- 172.21.15.16:0/2425204257 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+2109 (secure 0 0 0) 0x7f8fe400f330 con 0x7f8ff4156cb0 2019-12-05T06:59:08.922 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:08.922 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8523,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":207}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:08.923 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 207 2019-12-05T06:59:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.924+0000 7f8ffbe65700 1 -- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8fd4023670 msgr2=0x7f8fd4025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:08.925 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.924+0000 7f8ffbe65700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8fd4023670 0x7f8fd4025ae0 crc :-1 s=READY pgs=712 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.924+0000 7f8ffbe65700 1 -- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 msgr2=0x7f8ff41590b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.924+0000 7f8ffbe65700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 secure :-1 s=READY pgs=403 cs=0 l=1 rx=0x7f8fe4010390 tx=0x7f8fe4010730).stop 2019-12-05T06:59:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.925+0000 7f8ffbe65700 1 -- 172.21.15.16:0/2425204257 shutdown_connections 2019-12-05T06:59:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.925+0000 7f8ffbe65700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8fd4023670 0x7f8fd4025ae0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:08.926 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.925+0000 7f8ffbe65700 1 --2- 172.21.15.16:0/2425204257 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f8ff4156cb0 0x7f8ff41590b0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.925+0000 7f8ffbe65700 1 --2- 172.21.15.16:0/2425204257 >> v2:172.21.15.76:3300/0 conn(0x7f8ff406df10 0x7f8ff4098300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.925+0000 7f8ffbe65700 1 -- 172.21.15.16:0/2425204257 shutdown_connections 2019-12-05T06:59:08.927 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.925+0000 7f8ffbe65700 1 -- 172.21.15.16:0/2425204257 wait complete. 2019-12-05T06:59:08.928 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:08.925+0000 7f8ffbe65700 1 -- 172.21.15.16:0/2425204257 >> 172.21.15.16:0/2425204257 conn(0x7f8ff40714f0 msgr2=0x7f8ff409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:08.940 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:59:10.941 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:11.215 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.206+0000 7fb8888d8700 1 Processor -- start 2019-12-05T06:59:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.206+0000 7fb8888d8700 1 -- start start 2019-12-05T06:59:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.208+0000 7fb8888d8700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb88408c6d0 0x7fb88408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.208+0000 7fb8888d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb88408f720 0x7fb88408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.208+0000 7fb8888d8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8840958e0 con 0x7fb88408fef0 2019-12-05T06:59:11.216 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.208+0000 7fb8888d8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8840a3f70 con 0x7fb88408f720 2019-12-05T06:59:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.208+0000 7fb8888d8700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88408d830 con 0x7fb88408c6d0 2019-12-05T06:59:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.208+0000 7fb88359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb88408f720 0x7fb88408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.208+0000 7fb88359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb88408f720 0x7fb88408d000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37386/0 (socket says 172.21.15.16:37386) 2019-12-05T06:59:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.208+0000 7fb882d9d700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb88408c6d0 0x7fb88408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:11.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.208+0000 7fb88359e700 1 -- 172.21.15.16:0/199501834 learned_addr learned my addr 172.21.15.16:0/199501834 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:59:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb88359e700 1 -- 172.21.15.16:0/199501834 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb88408c6d0 msgr2=0x7fb88408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb88359e700 1 --2- 172.21.15.16:0/199501834 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb88408c6d0 0x7fb88408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb88359e700 1 -- 172.21.15.16:0/199501834 >> v1:172.21.15.76:6789/0 conn(0x7fb88408fef0 legacy=0x7fb88408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb88359e700 1 -- 172.21.15.16:0/199501834 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88408fbc0 con 0x7fb88408f720 2019-12-05T06:59:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb881d9b700 1 -- 172.21.15.16:0/199501834 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686302159 0 0) 0x7fb86c001080 con 0x7fb88408fef0 2019-12-05T06:59:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb88359e700 1 --2- 172.21.15.16:0/199501834 >> v2:172.21.15.16:3300/0 conn(0x7fb88408f720 0x7fb88408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fb87c00f6b0 tx=0x7fb87c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb88359e700 1 --2- 172.21.15.16:0/199501834 >> v2:172.21.15.16:3300/0 conn(0x7fb88408f720 0x7fb88408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fb87c00f6b0 tx=0x7fb87c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb88359e700 1 --2- 172.21.15.16:0/199501834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb88408f720 0x7fb88408d000 secure :-1 s=READY pgs=1707 cs=0 l=1 rx=0x7fb87c00f6b0 tx=0x7fb87c00fa50).ready entity=mon.0 client_cookie=5492d2c8144ea329 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb88359e700 1 --2- 172.21.15.16:0/199501834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb88408f720 0x7fb88408d000 secure :-1 s=THROTTLE_DONE pgs=1707 cs=0 l=1 rx=0x7fb87c00f6b0 tx=0x7fb87c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb881d9b700 1 -- 172.21.15.16:0/199501834 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb87c000a70 con 0x7fb88408f720 2019-12-05T06:59:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb88359e700 1 --2- 172.21.15.16:0/199501834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb88408f720 0x7fb88408d000 secure :-1 s=THROTTLE_DONE pgs=1707 cs=0 l=1 rx=0x7fb87c00f6b0 tx=0x7fb87c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.209+0000 7fb88359e700 1 --2- 172.21.15.16:0/199501834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb88408f720 0x7fb88408d000 secure :-1 s=THROTTLE_DONE pgs=1707 cs=0 l=1 rx=0x7fb87c00f6b0 tx=0x7fb87c00fa50).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.210+0000 7fb881d9b700 1 -- 172.21.15.16:0/199501834 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c0010e0 con 0x7fb88408f720 2019-12-05T06:59:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.210+0000 7fb881d9b700 1 -- 172.21.15.16:0/199501834 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb87c0017b0 con 0x7fb88408f720 2019-12-05T06:59:11.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.210+0000 7fb8888d8700 1 -- 172.21.15.16:0/199501834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb88408f720 msgr2=0x7fb88408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.210+0000 7fb8888d8700 1 --2- 172.21.15.16:0/199501834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb88408f720 0x7fb88408d000 secure :-1 s=READY pgs=1707 cs=0 l=1 rx=0x7fb87c00f6b0 tx=0x7fb87c00fa50).stop 2019-12-05T06:59:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.211+0000 7fb8888d8700 1 -- 172.21.15.16:0/199501834 shutdown_connections 2019-12-05T06:59:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.211+0000 7fb8888d8700 1 --2- 172.21.15.16:0/199501834 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb88408f720 0x7fb88408d000 unknown :-1 s=CLOSED pgs=1707 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.211+0000 7fb8888d8700 1 --2- 172.21.15.16:0/199501834 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fb88408c6d0 0x7fb88408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.211+0000 7fb8888d8700 1 -- 172.21.15.16:0/199501834 shutdown_connections 2019-12-05T06:59:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.211+0000 7fb8888d8700 1 -- 172.21.15.16:0/199501834 wait complete. 2019-12-05T06:59:11.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.211+0000 7fb8888d8700 1 -- 172.21.15.16:0/199501834 >> 172.21.15.16:0/199501834 conn(0x7fb884061940 msgr2=0x7fb884096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.216+0000 7fb8888d8700 1 Processor -- start 2019-12-05T06:59:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.216+0000 7fb8888d8700 1 -- start start 2019-12-05T06:59:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.217+0000 7fb8888d8700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fb88409bbe0 0x7fb8841539e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb8888d8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb884084f30 0x7fb884155f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:11.224 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb8888d8700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb884084b90 con 0x7fb884096e90 2019-12-05T06:59:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb8888d8700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb88409a450 con 0x7fb884084f30 2019-12-05T06:59:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb8888d8700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb884099a20 con 0x7fb88409bbe0 2019-12-05T06:59:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb88359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb884084f30 0x7fb884155f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb88359e700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7fb884084f30 0x7fb884155f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37392/0 (socket says 172.21.15.16:37392) 2019-12-05T06:59:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb88359e700 1 -- 172.21.15.16:0/4251149027 learned_addr learned my addr 172.21.15.16:0/4251149027 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:59:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb88259c700 1 --2- 172.21.15.16:0/4251149027 >> v2:172.21.15.76:3300/0 conn(0x7fb88409bbe0 0x7fb8841539e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:11.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb88359e700 1 -- 172.21.15.16:0/4251149027 >> v2:172.21.15.76:3300/0 conn(0x7fb88409bbe0 msgr2=0x7fb8841539e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> v2:172.21.15.76:3300/0 conn(0x7fb88409bbe0 0x7fb8841539e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb88359e700 1 -- 172.21.15.16:0/4251149027 >> v1:172.21.15.76:6789/0 conn(0x7fb884096e90 legacy=0x7fb88409b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.218+0000 7fb88359e700 1 -- 172.21.15.16:0/4251149027 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb88409aae0 con 0x7fb884084f30 2019-12-05T06:59:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> v2:172.21.15.16:3300/0 conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> v2:172.21.15.16:3300/0 conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=READY pgs=1708 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:11.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb87b7fe700 1 -- 172.21.15.16:0/4251149027 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb87c005930 con 0x7fb884084f30 2019-12-05T06:59:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb87b7fe700 1 -- 172.21.15.16:0/4251149027 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb87c005550 con 0x7fb884084f30 2019-12-05T06:59:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb87b7fe700 1 -- 172.21.15.16:0/4251149027 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb87c005f10 con 0x7fb884084f30 2019-12-05T06:59:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb8888d8700 1 -- 172.21.15.16:0/4251149027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb88409aae0 con 0x7fb884084f30 2019-12-05T06:59:11.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.219+0000 7fb8888d8700 1 -- 172.21.15.16:0/4251149027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb88409aae0 con 0x7fb884084f30 2019-12-05T06:59:11.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.220+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.221+0000 7fb87b7fe700 1 -- 172.21.15.16:0/4251149027 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fb87c005740 con 0x7fb884084f30 2019-12-05T06:59:11.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.221+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.222+0000 7fb87b7fe700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 0x7fb868025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:11.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.222+0000 7fb87b7fe700 1 -- 172.21.15.16:0/4251149027 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 9041+0+0 (secure 0 0 0) 0x7fb87c045db0 con 0x7fb884084f30 2019-12-05T06:59:11.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.222+0000 7fb882d9d700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 0x7fb868025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:11.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.223+0000 7fb882d9d700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 0x7fb868025ac0 crc :-1 s=READY pgs=713 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:11.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.238+0000 7fb8888d8700 1 -- 172.21.15.16:0/4251149027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb88409aae0 con 0x7fb884084f30 2019-12-05T06:59:11.247 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.246+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.248 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.246+0000 7fb87b7fe700 1 -- 172.21.15.16:0/4251149027 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fb87c005340 con 0x7fb884084f30 2019-12-05T06:59:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.518+0000 7fb8888d8700 1 -- 172.21.15.16:0/4251149027 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb88409aae0 con 0x7fb884084f30 2019-12-05T06:59:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.519+0000 7fb88359e700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=THROTTLE_DONE pgs=1708 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.519+0000 7fb87b7fe700 1 -- 172.21.15.16:0/4251149027 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 207 v207) v1 ==== 95+0+2109 (secure 0 0 0) 0x7fb87c005710 con 0x7fb884084f30 2019-12-05T06:59:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 207 2019-12-05T06:59:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.521+0000 7fb8888d8700 1 -- 172.21.15.16:0/4251149027 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 msgr2=0x7fb868025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:11.526 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.521+0000 7fb8888d8700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 0x7fb868025ac0 crc :-1 s=READY pgs=713 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.521+0000 7fb8888d8700 1 -- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 msgr2=0x7fb884155f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.521+0000 7fb8888d8700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 0x7fb884155f30 secure :-1 s=READY pgs=1708 cs=0 l=1 rx=0x7fb87c010460 tx=0x7fb87c010290).stop 2019-12-05T06:59:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.522+0000 7fb8888d8700 1 -- 172.21.15.16:0/4251149027 shutdown_connections 2019-12-05T06:59:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.522+0000 7fb8888d8700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fb868023630 0x7fb868025ac0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.522+0000 7fb8888d8700 1 --2- 172.21.15.16:0/4251149027 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7fb884084f30 0x7fb884155f30 unknown :-1 s=CLOSED pgs=1708 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.522+0000 7fb8888d8700 1 --2- 172.21.15.16:0/4251149027 >> v2:172.21.15.76:3300/0 conn(0x7fb88409bbe0 0x7fb8841539e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:11.527 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.522+0000 7fb8888d8700 1 -- 172.21.15.16:0/4251149027 shutdown_connections 2019-12-05T06:59:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.522+0000 7fb8888d8700 1 -- 172.21.15.16:0/4251149027 wait complete. 2019-12-05T06:59:11.528 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:11.522+0000 7fb8888d8700 1 -- 172.21.15.16:0/4251149027 >> 172.21.15.16:0/4251149027 conn(0x7fb8840714f0 msgr2=0x7fb88409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:11.528 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:11.528 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":207,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8523,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":207}],"filesystems":[{"mdsmap":{"epoch":206,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:57:58.678934+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":84,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8430},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8430":{"gid":8430,"name":"b","rank":0,"incarnation":203,"state":"up:active","state_seq":17,"addr":"172.21.15.76:6835/448242135","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":448242135},{"type":"v1","addr":"172.21.15.76:6835","nonce":448242135}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:11.533 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is laggy/crashed, in failed state, or mds.b is removed from mdsmap 2019-12-05T06:59:13.534 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.780+0000 7ff33c2cf700 1 Processor -- start 2019-12-05T06:59:13.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.780+0000 7ff33c2cf700 1 -- start start 2019-12-05T06:59:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33c2cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff33408c6d0 0x7ff33408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33c2cf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff33408f720 0x7ff33408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:13.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33c2cf700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff33408fb80 con 0x7ff33408fef0 2019-12-05T06:59:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33c2cf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33408df90 con 0x7ff33408c6d0 2019-12-05T06:59:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33c2cf700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3340a3f70 con 0x7ff33408f720 2019-12-05T06:59:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33a86c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff33408c6d0 0x7ff33408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33a86c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff33408c6d0 0x7ff33408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37416/0 (socket says 172.21.15.16:37416) 2019-12-05T06:59:13.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33a86c700 1 -- 172.21.15.16:0/2114955473 learned_addr learned my addr 172.21.15.16:0/2114955473 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:59:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33a86c700 1 -- 172.21.15.16:0/2114955473 >> v2:172.21.15.16:3300/0 conn(0x7ff33408c6d0 msgr2=0x7ff33408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2019-12-05T06:59:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33a86c700 1 -- 172.21.15.16:0/2114955473 >> v2:172.21.15.16:3300/0 conn(0x7ff33408c6d0 msgr2=0x7ff33408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2019-12-05T06:59:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2114955473 >> v2:172.21.15.16:3300/0 conn(0x7ff33408c6d0 0x7ff33408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_read_frame_preamble_main read frame length and tag failed r=-1 ((1) Operation not permitted) 2019-12-05T06:59:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2114955473 >> v2:172.21.15.16:3300/0 conn(0x7ff33408c6d0 0x7ff33408cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0)._fault waiting 0.200000 2019-12-05T06:59:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.781+0000 7ff33b06d700 1 --2- 172.21.15.16:0/2114955473 >> v2:172.21.15.76:3300/0 conn(0x7ff33408f720 0x7ff33408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:13.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.782+0000 7ff33986a700 1 -- 172.21.15.16:0/2114955473 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463215549 0 0) 0x7ff328001080 con 0x7ff33408fef0 2019-12-05T06:59:13.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.782+0000 7ff33986a700 1 -- 172.21.15.16:0/2114955473 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff31c002e80 con 0x7ff33408fef0 2019-12-05T06:59:13.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.782+0000 7ff33b06d700 1 -- 172.21.15.16:0/2114955473 >> v2:172.21.15.16:3300/0 conn(0x7ff33408c6d0 msgr2=0x7ff33408cac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2019-12-05T06:59:13.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.782+0000 7ff33b06d700 1 --2- 172.21.15.16:0/2114955473 >> v2:172.21.15.16:3300/0 conn(0x7ff33408c6d0 0x7ff33408cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:13.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.782+0000 7ff33b06d700 1 -- 172.21.15.16:0/2114955473 >> v1:172.21.15.76:6789/0 conn(0x7ff33408fef0 legacy=0x7ff33408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:13.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.782+0000 7ff33b06d700 1 -- 172.21.15.16:0/2114955473 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33408d540 con 0x7ff33408f720 2019-12-05T06:59:13.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.782+0000 7ff33b06d700 1 --2- 172.21.15.16:0/2114955473 >> v2:172.21.15.76:3300/0 conn(0x7ff33408f720 0x7ff33408d000 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7ff32c00af80 tx=0x7ff32c00b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.782+0000 7ff33b06d700 1 --2- 172.21.15.16:0/2114955473 >> v2:172.21.15.76:3300/0 conn(0x7ff33408f720 0x7ff33408d000 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7ff32c00af80 tx=0x7ff32c00b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.782+0000 7ff33b06d700 1 --2- 172.21.15.16:0/2114955473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33408f720 0x7ff33408d000 secure :-1 s=READY pgs=1155 cs=0 l=1 rx=0x7ff32c00af80 tx=0x7ff32c00b710).ready entity=mon.1 client_cookie=f9faa49384aedfb0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:13.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.783+0000 7ff33b06d700 1 --2- 172.21.15.16:0/2114955473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33408f720 0x7ff33408d000 secure :-1 s=THROTTLE_DONE pgs=1155 cs=0 l=1 rx=0x7ff32c00af80 tx=0x7ff32c00b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.783+0000 7ff33986a700 1 -- 172.21.15.16:0/2114955473 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff32c007240 con 0x7ff33408f720 2019-12-05T06:59:13.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.783+0000 7ff33b06d700 1 --2- 172.21.15.16:0/2114955473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33408f720 0x7ff33408d000 secure :-1 s=THROTTLE_DONE pgs=1155 cs=0 l=1 rx=0x7ff32c00af80 tx=0x7ff32c00b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.783+0000 7ff33986a700 1 -- 172.21.15.16:0/2114955473 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32c0099e0 con 0x7ff33408f720 2019-12-05T06:59:13.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.783+0000 7ff33b06d700 1 --2- 172.21.15.16:0/2114955473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33408f720 0x7ff33408d000 secure :-1 s=THROTTLE_DONE pgs=1155 cs=0 l=1 rx=0x7ff32c00af80 tx=0x7ff32c00b710).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.783+0000 7ff33986a700 1 -- 172.21.15.16:0/2114955473 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff32c000bc0 con 0x7ff33408f720 2019-12-05T06:59:13.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.783+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2114955473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33408f720 msgr2=0x7ff33408d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:13.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.783+0000 7ff33c2cf700 1 --2- 172.21.15.16:0/2114955473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33408f720 0x7ff33408d000 secure :-1 s=READY pgs=1155 cs=0 l=1 rx=0x7ff32c00af80 tx=0x7ff32c00b710).stop 2019-12-05T06:59:13.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.784+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2114955473 shutdown_connections 2019-12-05T06:59:13.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.784+0000 7ff33c2cf700 1 --2- 172.21.15.16:0/2114955473 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7ff33408f720 0x7ff33408d000 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:13.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.784+0000 7ff33c2cf700 1 --2- 172.21.15.16:0/2114955473 >> v2:172.21.15.16:3300/0 conn(0x7ff33408c6d0 0x7ff33408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:13.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.784+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2114955473 shutdown_connections 2019-12-05T06:59:13.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.784+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2114955473 wait complete. 2019-12-05T06:59:13.794 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.784+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2114955473 >> 172.21.15.16:0/2114955473 conn(0x7ff334061940 msgr2=0x7ff334096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:13.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.786+0000 7ff33c2cf700 1 Processor -- start 2019-12-05T06:59:13.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.786+0000 7ff33c2cf700 1 -- start start 2019-12-05T06:59:13.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.787+0000 7ff33c2cf700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff33414d400 0x7ff33414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:13.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.787+0000 7ff33c2cf700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7ff334132d00 0x7ff334131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:13.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.787+0000 7ff33c2cf700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff334098da0 con 0x7ff334132270 2019-12-05T06:59:13.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.787+0000 7ff33c2cf700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33409bb50 con 0x7ff33414d400 2019-12-05T06:59:13.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.787+0000 7ff33c2cf700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33409aa60 con 0x7ff334132d00 2019-12-05T06:59:13.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.787+0000 7ff33a86c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff33414d400 0x7ff33414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:13.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.787+0000 7ff33a86c700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7ff33414d400 0x7ff33414d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37422/0 (socket says 172.21.15.16:37422) 2019-12-05T06:59:13.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.787+0000 7ff33a86c700 1 -- 172.21.15.16:0/2872254945 learned_addr learned my addr 172.21.15.16:0/2872254945 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:59:13.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a86c700 1 -- 172.21.15.16:0/2872254945 >> v2:172.21.15.76:3300/0 conn(0x7ff334132d00 msgr2=0x7ff334131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:13.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.787+0000 7ff33a06b700 1 --2- 172.21.15.16:0/2872254945 >> v2:172.21.15.76:3300/0 conn(0x7ff334132d00 0x7ff334131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:13.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> v2:172.21.15.76:3300/0 conn(0x7ff334132d00 0x7ff334131d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:13.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a86c700 1 -- 172.21.15.16:0/2872254945 >> v1:172.21.15.16:6789/0 conn(0x7ff334132270 legacy=0x7ff33406dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:13.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a86c700 1 -- 172.21.15.16:0/2872254945 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33409b0f0 con 0x7ff33414d400 2019-12-05T06:59:13.801 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff3337fe700 1 -- 172.21.15.16:0/2872254945 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176354394 0 0) 0x7ff32c00af00 con 0x7ff334132270 2019-12-05T06:59:13.802 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a06b700 1 --2- 172.21.15.16:0/2872254945 >> v2:172.21.15.76:3300/0 conn(0x7ff334132d00 0x7ff334131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).send_auth_request state changed! 2019-12-05T06:59:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> v2:172.21.15.16:3300/0 conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> v2:172.21.15.16:3300/0 conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=READY pgs=1713 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=THROTTLE_DONE pgs=1713 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.809 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=THROTTLE_DONE pgs=1713 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=THROTTLE_DONE pgs=1713 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff3337fe700 1 -- 172.21.15.16:0/2872254945 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff324007ae0 con 0x7ff33414d400 2019-12-05T06:59:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff3337fe700 1 -- 172.21.15.16:0/2872254945 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff32400bf70 con 0x7ff33414d400 2019-12-05T06:59:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2872254945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff33409b0f0 con 0x7ff33414d400 2019-12-05T06:59:13.810 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff3337fe700 1 -- 172.21.15.16:0/2872254945 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff32400c600 con 0x7ff33414d400 2019-12-05T06:59:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.788+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2872254945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33409b0f0 con 0x7ff33414d400 2019-12-05T06:59:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.789+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=THROTTLE_DONE pgs=1713 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.790+0000 7ff3337fe700 1 -- 172.21.15.16:0/2872254945 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7ff32400c160 con 0x7ff33414d400 2019-12-05T06:59:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.790+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=THROTTLE_DONE pgs=1713 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.790+0000 7ff3337fe700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff314023630 0x7ff314025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.790+0000 7ff3337fe700 1 -- 172.21.15.16:0/2872254945 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7ff3240461b0 con 0x7ff33414d400 2019-12-05T06:59:13.811 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.790+0000 7ff33a06b700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff314023630 0x7ff314025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:13.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.790+0000 7ff33a06b700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff314023630 0x7ff314025ac0 crc :-1 s=READY pgs=714 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:13.812 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.796+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2872254945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff33409b0f0 con 0x7ff33414d400 2019-12-05T06:59:13.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.803+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=THROTTLE_DONE pgs=1713 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:13.813 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:13.803+0000 7ff3337fe700 1 -- 172.21.15.16:0/2872254945 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7ff324007660 con 0x7ff33414d400 2019-12-05T06:59:14.035 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.034+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2872254945 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff33409b0f0 con 0x7ff33414d400 2019-12-05T06:59:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.034+0000 7ff33a86c700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=THROTTLE_DONE pgs=1713 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.034+0000 7ff3337fe700 1 -- 172.21.15.16:0/2872254945 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+1774 (secure 0 0 0) 0x7ff32400c160 con 0x7ff33414d400 2019-12-05T06:59:14.037 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 208 2019-12-05T06:59:14.037 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:14.037 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:59:13.667290+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8523":{"gid":8523,"name":"a","rank":0,"incarnation":208,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:14.040 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2872254945 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff314023630 msgr2=0x7ff314025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff314023630 0x7ff314025ac0 crc :-1 s=READY pgs=714 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 msgr2=0x7ff33414d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 0x7ff33414d7f0 secure :-1 s=READY pgs=1713 cs=0 l=1 rx=0x7ff3240107d0 tx=0x7ff324010b70).stop 2019-12-05T06:59:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2872254945 shutdown_connections 2019-12-05T06:59:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 --2- 172.21.15.16:0/2872254945 >> v2:172.21.15.76:3300/0 conn(0x7ff334132d00 0x7ff334131d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7ff314023630 0x7ff314025ac0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:14.041 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 --2- 172.21.15.16:0/2872254945 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7ff33414d400 0x7ff33414d7f0 unknown :-1 s=CLOSED pgs=1713 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2872254945 shutdown_connections 2019-12-05T06:59:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2872254945 wait complete. 2019-12-05T06:59:14.042 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.038+0000 7ff33c2cf700 1 -- 172.21.15.16:0/2872254945 >> 172.21.15.16:0/2872254945 conn(0x7ff3340714f0 msgr2=0x7ff33409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:14.049 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b down, removed from mdsmap 2019-12-05T06:59:14.050 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for mds cluster to stabilize... 2019-12-05T06:59:14.050 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.210+0000 7f619ba1f700 1 Processor -- start 2019-12-05T06:59:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.211+0000 7f619ba1f700 1 -- start start 2019-12-05T06:59:14.217 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.211+0000 7f619ba1f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f619408c6d0 0x7f619408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.211+0000 7f619ba1f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f619408f720 0x7f619408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.211+0000 7f619ba1f700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619408fb80 con 0x7f619408fef0 2019-12-05T06:59:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.211+0000 7f619ba1f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619408df90 con 0x7f619408c6d0 2019-12-05T06:59:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.211+0000 7f619ba1f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61940a3f70 con 0x7f619408f720 2019-12-05T06:59:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.211+0000 7f6199fbc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f619408c6d0 0x7f619408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:14.218 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.211+0000 7f6199fbc700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f619408c6d0 0x7f619408cac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37430/0 (socket says 172.21.15.16:37430) 2019-12-05T06:59:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.211+0000 7f6199fbc700 1 -- 172.21.15.16:0/3211411747 learned_addr learned my addr 172.21.15.16:0/3211411747 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:59:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f619a7bd700 1 --2- 172.21.15.16:0/3211411747 >> v2:172.21.15.76:3300/0 conn(0x7f619408f720 0x7f619408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6199fbc700 1 -- 172.21.15.16:0/3211411747 >> v2:172.21.15.76:3300/0 conn(0x7f619408f720 msgr2=0x7f619408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3211411747 >> v2:172.21.15.76:3300/0 conn(0x7f619408f720 0x7f619408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6199fbc700 1 -- 172.21.15.16:0/3211411747 >> v1:172.21.15.16:6789/0 conn(0x7f619408fef0 legacy=0x7f619408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:14.219 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6198fba700 1 -- 172.21.15.16:0/3211411747 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689320600 0 0) 0x7f6190001100 con 0x7f619408fef0 2019-12-05T06:59:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6199fbc700 1 -- 172.21.15.16:0/3211411747 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619408d540 con 0x7f619408c6d0 2019-12-05T06:59:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3211411747 >> v2:172.21.15.16:3300/0 conn(0x7f619408c6d0 0x7f619408cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f618c00fa40 tx=0x7f618c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3211411747 >> v2:172.21.15.16:3300/0 conn(0x7f619408c6d0 0x7f619408cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f618c00fa40 tx=0x7f618c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3211411747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619408c6d0 0x7f619408cac0 secure :-1 s=READY pgs=1715 cs=0 l=1 rx=0x7f618c00fa40 tx=0x7f618c00fde0).ready entity=mon.0 client_cookie=70b4831ef3487fd0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3211411747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619408c6d0 0x7f619408cac0 secure :-1 s=THROTTLE_DONE pgs=1715 cs=0 l=1 rx=0x7f618c00fa40 tx=0x7f618c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6198fba700 1 -- 172.21.15.16:0/3211411747 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f618c000ce0 con 0x7f619408c6d0 2019-12-05T06:59:14.220 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3211411747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619408c6d0 0x7f619408cac0 secure :-1 s=THROTTLE_DONE pgs=1715 cs=0 l=1 rx=0x7f618c00fa40 tx=0x7f618c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6198fba700 1 -- 172.21.15.16:0/3211411747 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f618c001010 con 0x7f619408c6d0 2019-12-05T06:59:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3211411747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619408c6d0 0x7f619408cac0 secure :-1 s=THROTTLE_DONE pgs=1715 cs=0 l=1 rx=0x7f618c00fa40 tx=0x7f618c00fde0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f6198fba700 1 -- 172.21.15.16:0/3211411747 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f618c007590 con 0x7f619408c6d0 2019-12-05T06:59:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f619ba1f700 1 -- 172.21.15.16:0/3211411747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619408c6d0 msgr2=0x7f619408cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.212+0000 7f619ba1f700 1 --2- 172.21.15.16:0/3211411747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619408c6d0 0x7f619408cac0 secure :-1 s=READY pgs=1715 cs=0 l=1 rx=0x7f618c00fa40 tx=0x7f618c00fde0).stop 2019-12-05T06:59:14.221 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.213+0000 7f619ba1f700 1 -- 172.21.15.16:0/3211411747 shutdown_connections 2019-12-05T06:59:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.213+0000 7f619ba1f700 1 --2- 172.21.15.16:0/3211411747 >> v2:172.21.15.76:3300/0 conn(0x7f619408f720 0x7f619408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.213+0000 7f619ba1f700 1 --2- 172.21.15.16:0/3211411747 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f619408c6d0 0x7f619408cac0 unknown :-1 s=CLOSED pgs=1715 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.213+0000 7f619ba1f700 1 -- 172.21.15.16:0/3211411747 shutdown_connections 2019-12-05T06:59:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.213+0000 7f619ba1f700 1 -- 172.21.15.16:0/3211411747 wait complete. 2019-12-05T06:59:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.213+0000 7f619ba1f700 1 -- 172.21.15.16:0/3211411747 >> 172.21.15.16:0/3211411747 conn(0x7f6194061940 msgr2=0x7f6194096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.216+0000 7f619ba1f700 1 Processor -- start 2019-12-05T06:59:14.222 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.216+0000 7f619ba1f700 1 -- start start 2019-12-05T06:59:14.225 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.216+0000 7f619ba1f700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f619414d400 0x7f619414d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.216+0000 7f619ba1f700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6194132d00 0x7f6194131d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.216+0000 7f619ba1f700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6194098da0 con 0x7f6194132270 2019-12-05T06:59:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.216+0000 7f619ba1f700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619409bb50 con 0x7f6194132d00 2019-12-05T06:59:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.216+0000 7f619ba1f700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619409aa60 con 0x7f619414d400 2019-12-05T06:59:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.216+0000 7f61997bb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6194132d00 0x7f6194131d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.216+0000 7f61997bb700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f6194132d00 0x7f6194131d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37438/0 (socket says 172.21.15.16:37438) 2019-12-05T06:59:14.226 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.216+0000 7f61997bb700 1 -- 172.21.15.16:0/3753816359 learned_addr learned my addr 172.21.15.16:0/3753816359 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:59:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3753816359 >> v2:172.21.15.76:3300/0 conn(0x7f619414d400 0x7f619414d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f61997bb700 1 -- 172.21.15.16:0/3753816359 >> v2:172.21.15.76:3300/0 conn(0x7f619414d400 msgr2=0x7f619414d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> v2:172.21.15.76:3300/0 conn(0x7f619414d400 0x7f619414d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f61997bb700 1 -- 172.21.15.16:0/3753816359 >> v1:172.21.15.76:6789/0 conn(0x7f6194132270 legacy=0x7f619406dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f61997bb700 1 -- 172.21.15.16:0/3753816359 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619409b0f0 con 0x7f6194132d00 2019-12-05T06:59:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3753816359 >> v2:172.21.15.76:3300/0 conn(0x7f619414d400 0x7f619414d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_reply_more state changed! 2019-12-05T06:59:14.227 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> v2:172.21.15.16:3300/0 conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> v2:172.21.15.16:3300/0 conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=READY pgs=1716 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=THROTTLE_DONE pgs=1716 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=THROTTLE_DONE pgs=1716 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=THROTTLE_DONE pgs=1716 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f618a7fc700 1 -- 172.21.15.16:0/3753816359 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6190000cc0 con 0x7f6194132d00 2019-12-05T06:59:14.228 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f618a7fc700 1 -- 172.21.15.16:0/3753816359 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f619000ee80 con 0x7f6194132d00 2019-12-05T06:59:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f619ba1f700 1 -- 172.21.15.16:0/3753816359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619409b0f0 con 0x7f6194132d00 2019-12-05T06:59:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.217+0000 7f618a7fc700 1 -- 172.21.15.16:0/3753816359 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f619000f550 con 0x7f6194132d00 2019-12-05T06:59:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.218+0000 7f619ba1f700 1 -- 172.21.15.16:0/3753816359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619409b0f0 con 0x7f6194132d00 2019-12-05T06:59:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.218+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=THROTTLE_DONE pgs=1716 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.218+0000 7f618a7fc700 1 -- 172.21.15.16:0/3753816359 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f619000f070 con 0x7f6194132d00 2019-12-05T06:59:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.218+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=THROTTLE_DONE pgs=1716 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.219+0000 7f618a7fc700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6174023630 0x7f6174025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:14.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.219+0000 7f618a7fc700 1 -- 172.21.15.16:0/3753816359 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f6190045e20 con 0x7f6194132d00 2019-12-05T06:59:14.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.219+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6174023630 0x7f6174025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:14.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.219+0000 7f6199fbc700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6174023630 0x7f6174025ac0 crc :-1 s=READY pgs=715 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:14.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.226+0000 7f619ba1f700 1 -- 172.21.15.16:0/3753816359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619409b0f0 con 0x7f6194132d00 2019-12-05T06:59:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.229+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=THROTTLE_DONE pgs=1716 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.229+0000 7f618a7fc700 1 -- 172.21.15.16:0/3753816359 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f619000f280 con 0x7f6194132d00 2019-12-05T06:59:14.425 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.424+0000 7f619ba1f700 1 -- 172.21.15.16:0/3753816359 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f619409b0f0 con 0x7f6194132d00 2019-12-05T06:59:14.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.425+0000 7f61997bb700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=THROTTLE_DONE pgs=1716 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:14.426 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.425+0000 7f618a7fc700 1 -- 172.21.15.16:0/3753816359 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 208 v208) v1 ==== 95+0+1774 (secure 0 0 0) 0x7f619000f070 con 0x7f6194132d00 2019-12-05T06:59:14.426 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 208 2019-12-05T06:59:14.427 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:14.427 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":208,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":208,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:59:13.667290+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8523":{"gid":8523,"name":"a","rank":0,"incarnation":208,"state":"up:replay","state_seq":2,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:14.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.427+0000 7f619ba1f700 1 -- 172.21.15.16:0/3753816359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6174023630 msgr2=0x7f6174025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:14.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.427+0000 7f619ba1f700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6174023630 0x7f6174025ac0 crc :-1 s=READY pgs=715 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:14.429 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.427+0000 7f619ba1f700 1 -- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 msgr2=0x7f6194131d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:14.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.427+0000 7f619ba1f700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 0x7f6194131d30 secure :-1 s=READY pgs=1716 cs=0 l=1 rx=0x7f6190010450 tx=0x7f61900107f0).stop 2019-12-05T06:59:14.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.428+0000 7f619ba1f700 1 -- 172.21.15.16:0/3753816359 shutdown_connections 2019-12-05T06:59:14.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.428+0000 7f619ba1f700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f6174023630 0x7f6174025ac0 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:14.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.428+0000 7f619ba1f700 1 --2- 172.21.15.16:0/3753816359 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f6194132d00 0x7f6194131d30 unknown :-1 s=CLOSED pgs=1716 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:14.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.428+0000 7f619ba1f700 1 --2- 172.21.15.16:0/3753816359 >> v2:172.21.15.76:3300/0 conn(0x7f619414d400 0x7f619414d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:14.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.428+0000 7f619ba1f700 1 -- 172.21.15.16:0/3753816359 shutdown_connections 2019-12-05T06:59:14.430 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.428+0000 7f619ba1f700 1 -- 172.21.15.16:0/3753816359 wait complete. 2019-12-05T06:59:14.431 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:14.428+0000 7f619ba1f700 1 -- 172.21.15.16:0/3753816359 >> 172.21.15.16:0/3753816359 conn(0x7f61940714f0 msgr2=0x7f619409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:14.439 INFO:tasks.mds_thrash.fs.[cephfs]:mds map: { "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "default_fscid": 1, "epoch": 208, "feature_flags": { "enable_multiple": false, "ever_enabled_multiple": false }, "filesystems": [ { "id": 1, "mdsmap": { "balancer": "", "compat": { "compat": {}, "incompat": { "feature_1": "base v0.20", "feature_10": "snaprealm v2", "feature_2": "client writeable ranges", "feature_3": "default file layouts on dirs", "feature_4": "dir inode in separate object", "feature_5": "mds uses versioned encoding", "feature_6": "dirfrag is stored in omap", "feature_8": "no anchor table", "feature_9": "file layout v2" }, "ro_compat": {} }, "created": "2019-12-05T06:06:40.256556+0000", "damaged": [], "data_pools": [ 3 ], "enabled": true, "epoch": 208, "ever_allowed_features": 2, "explicitly_allowed_features": 2, "failed": [], "flags": 18, "fs_name": "cephfs", "in": [ 0 ], "info": { "gid_8523": { "addr": "172.21.15.16:6833/3211386191", "addrs": { "addrvec": [ { "addr": "172.21.15.16:6832", "nonce": 3211386191, "type": "v2" }, { "addr": "172.21.15.16:6833", "nonce": 3211386191, "type": "v1" } ] }, "export_targets": [], "features": 4540138292836696063, "flags": 0, "gid": 8523, "incarnation": 208, "name": "a", "rank": 0, "state": "up:replay", "state_seq": 2 } }, "last_failure": 0, "last_failure_osd_epoch": 85, "max_file_size": 1099511627776, "max_mds": 1, "metadata_pool": 2, "min_compat_client": "0 (unknown)", "modified": "2019-12-05T06:59:13.667290+0000", "root": 0, "session_autoclose": 300, "session_timeout": 60, "standby_count_wanted": 0, "stopped": [], "tableserver": 0, "up": { "mds_0": 8523 } } } ], "standbys": [] } 2019-12-05T06:59:16.440 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:16.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.689+0000 7f84b6a8a700 1 Processor -- start 2019-12-05T06:59:16.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.689+0000 7f84b6a8a700 1 -- start start 2019-12-05T06:59:16.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.691+0000 7f84b6a8a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 0x7f84b008cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:16.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.691+0000 7f84b6a8a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f84b008fef0 0x7f84b008d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:16.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.691+0000 7f84b6a8a700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b00958e0 con 0x7f84b008ef20 2019-12-05T06:59:16.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.691+0000 7f84b6a8a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b00a3f70 con 0x7f84b008fef0 2019-12-05T06:59:16.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.691+0000 7f84b6a8a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84b008d830 con 0x7f84b008c6d0 2019-12-05T06:59:16.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.691+0000 7f84b5828700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f84b008ef20 0x7f84b008f3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50846/0 (socket says 172.21.15.16:50846) 2019-12-05T06:59:16.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.691+0000 7f84b5828700 1 -- 172.21.15.16:0/3398086942 learned_addr learned my addr 172.21.15.16:0/3398086942 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:16.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.691+0000 7f84b5027700 1 --2- 172.21.15.16:0/3398086942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 0x7f84b008cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:16.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.691+0000 7f84b4826700 1 --2- 172.21.15.16:0/3398086942 >> v2:172.21.15.76:3300/0 conn(0x7f84b008fef0 0x7f84b008d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:16.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.692+0000 7f84affff700 1 -- 172.21.15.16:0/3398086942 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282798980 0 0) 0x7f84a0001100 con 0x7f84b008ef20 2019-12-05T06:59:16.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.692+0000 7f84affff700 1 -- 172.21.15.16:0/3398086942 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8498002e80 con 0x7f84b008ef20 2019-12-05T06:59:16.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.692+0000 7f84b5027700 1 -- 172.21.15.16:0/3398086942 >> v2:172.21.15.76:3300/0 conn(0x7f84b008fef0 msgr2=0x7f84b008d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:16.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.692+0000 7f84b5027700 1 --2- 172.21.15.16:0/3398086942 >> v2:172.21.15.76:3300/0 conn(0x7f84b008fef0 0x7f84b008d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:16.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.692+0000 7f84b5027700 1 -- 172.21.15.16:0/3398086942 >> v1:172.21.15.16:6789/0 conn(0x7f84b008ef20 legacy=0x7f84b008f3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:16.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.692+0000 7f84b5027700 1 -- 172.21.15.16:0/3398086942 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b008d540 con 0x7f84b008c6d0 2019-12-05T06:59:16.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.692+0000 7f84b5027700 1 --2- 172.21.15.16:0/3398086942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 0x7f84b008cac0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f84a800b2d0 tx=0x7f84a800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.693+0000 7f84b5027700 1 --2- 172.21.15.16:0/3398086942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 0x7f84b008cac0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f84a800b2d0 tx=0x7f84a800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.693+0000 7f84b5027700 1 --2- 172.21.15.16:0/3398086942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 0x7f84b008cac0 secure :-1 s=READY pgs=409 cs=0 l=1 rx=0x7f84a800b2d0 tx=0x7f84a800b5d0).ready entity=mon.2 client_cookie=ccbfec82658487bc server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:16.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.693+0000 7f84b5027700 1 --2- 172.21.15.16:0/3398086942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 0x7f84b008cac0 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7f84a800b2d0 tx=0x7f84a800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.693+0000 7f84b5027700 1 --2- 172.21.15.16:0/3398086942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 0x7f84b008cac0 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7f84a800b2d0 tx=0x7f84a800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.693+0000 7f84affff700 1 -- 172.21.15.16:0/3398086942 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84a8007590 con 0x7f84b008c6d0 2019-12-05T06:59:16.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.693+0000 7f84b5027700 1 --2- 172.21.15.16:0/3398086942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 0x7f84b008cac0 secure :-1 s=THROTTLE_DONE pgs=409 cs=0 l=1 rx=0x7f84a800b2d0 tx=0x7f84a800b5d0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.693+0000 7f84affff700 1 -- 172.21.15.16:0/3398086942 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a8009d30 con 0x7f84b008c6d0 2019-12-05T06:59:16.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.693+0000 7f84affff700 1 -- 172.21.15.16:0/3398086942 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84a800a400 con 0x7f84b008c6d0 2019-12-05T06:59:16.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.693+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3398086942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 msgr2=0x7f84b008cac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:16.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.693+0000 7f84b6a8a700 1 --2- 172.21.15.16:0/3398086942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 0x7f84b008cac0 secure :-1 s=READY pgs=409 cs=0 l=1 rx=0x7f84a800b2d0 tx=0x7f84a800b5d0).stop 2019-12-05T06:59:16.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.695+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3398086942 shutdown_connections 2019-12-05T06:59:16.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.695+0000 7f84b6a8a700 1 --2- 172.21.15.16:0/3398086942 >> v2:172.21.15.76:3300/0 conn(0x7f84b008fef0 0x7f84b008d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:16.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.695+0000 7f84b6a8a700 1 --2- 172.21.15.16:0/3398086942 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f84b008c6d0 0x7f84b008cac0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:16.706 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.695+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3398086942 shutdown_connections 2019-12-05T06:59:16.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.696+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3398086942 wait complete. 2019-12-05T06:59:16.707 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.696+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3398086942 >> 172.21.15.16:0/3398086942 conn(0x7f84b0061940 msgr2=0x7f84b0096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:16.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.701+0000 7f84b6a8a700 1 Processor -- start 2019-12-05T06:59:16.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.701+0000 7f84b6a8a700 1 -- start start 2019-12-05T06:59:16.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b6a8a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84b01302f0 0x7f84b012a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:16.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b6a8a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f84b012b640 0x7f84b012abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:16.710 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b6a8a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84b0131bf0 con 0x7f84b012b0f0 2019-12-05T06:59:16.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b6a8a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b01296e0 con 0x7f84b01302f0 2019-12-05T06:59:16.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b6a8a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84b0128cb0 con 0x7f84b012b640 2019-12-05T06:59:16.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b5027700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84b01302f0 0x7f84b012a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:16.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b5027700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f84b01302f0 0x7f84b012a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37450/0 (socket says 172.21.15.16:37450) 2019-12-05T06:59:16.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b5027700 1 -- 172.21.15.16:0/3461319748 learned_addr learned my addr 172.21.15.16:0/3461319748 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:59:16.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b4826700 1 --2- 172.21.15.16:0/3461319748 >> v2:172.21.15.76:3300/0 conn(0x7f84b012b640 0x7f84b012abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:16.711 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b5027700 1 -- 172.21.15.16:0/3461319748 >> v2:172.21.15.76:3300/0 conn(0x7f84b012b640 msgr2=0x7f84b012abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:16.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> v2:172.21.15.76:3300/0 conn(0x7f84b012b640 0x7f84b012abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:16.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b5027700 1 -- 172.21.15.16:0/3461319748 >> v1:172.21.15.76:6789/0 conn(0x7f84b012b0f0 legacy=0x7f84b01306e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:16.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.703+0000 7f84b5027700 1 -- 172.21.15.16:0/3461319748 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84b0129d70 con 0x7f84b01302f0 2019-12-05T06:59:16.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> v2:172.21.15.16:3300/0 conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> v2:172.21.15.16:3300/0 conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.712 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=READY pgs=1718 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:16.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=THROTTLE_DONE pgs=1718 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=THROTTLE_DONE pgs=1718 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=THROTTLE_DONE pgs=1718 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84adffb700 1 -- 172.21.15.16:0/3461319748 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84a800af30 con 0x7f84b01302f0 2019-12-05T06:59:16.713 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84adffb700 1 -- 172.21.15.16:0/3461319748 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84a8009af0 con 0x7f84b01302f0 2019-12-05T06:59:16.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3461319748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84b0129d70 con 0x7f84b01302f0 2019-12-05T06:59:16.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84adffb700 1 -- 172.21.15.16:0/3461319748 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84a800a140 con 0x7f84b01302f0 2019-12-05T06:59:16.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.704+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3461319748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84b0129d70 con 0x7f84b01302f0 2019-12-05T06:59:16.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.705+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=THROTTLE_DONE pgs=1718 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.706+0000 7f84adffb700 1 -- 172.21.15.16:0/3461319748 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f84a8009ce0 con 0x7f84b01302f0 2019-12-05T06:59:16.714 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.706+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=THROTTLE_DONE pgs=1718 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.707+0000 7f84adffb700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8490023630 0x7f8490025ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:16.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.707+0000 7f84adffb700 1 -- 172.21.15.16:0/3461319748 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f84a80461b0 con 0x7f84b01302f0 2019-12-05T06:59:16.715 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.707+0000 7f84b4826700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8490023630 0x7f8490025ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:16.716 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.708+0000 7f84b4826700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8490023630 0x7f8490025ac0 crc :-1 s=READY pgs=716 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:16.721 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.720+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3461319748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84b0129d70 con 0x7f84b01302f0 2019-12-05T06:59:16.730 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.729+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=THROTTLE_DONE pgs=1718 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.731 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.729+0000 7f84adffb700 1 -- 172.21.15.16:0/3461319748 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f84a8000cc0 con 0x7f84b01302f0 2019-12-05T06:59:16.969 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.968+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3461319748 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84b0129d70 con 0x7f84b01302f0 2019-12-05T06:59:16.970 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.969+0000 7f84b5027700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=THROTTLE_DONE pgs=1718 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:16.971 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.969+0000 7f84adffb700 1 -- 172.21.15.16:0/3461319748 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 209 v209) v1 ==== 95+0+1778 (secure 0 0 0) 0x7f84a8009ce0 con 0x7f84b01302f0 2019-12-05T06:59:16.971 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 209 2019-12-05T06:59:16.971 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:16.973 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":209,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":209,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:59:16.772807+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8523":{"gid":8523,"name":"a","rank":0,"incarnation":208,"state":"up:reconnect","state_seq":18,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:16.973 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3461319748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8490023630 msgr2=0x7f8490025ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8490023630 0x7f8490025ac0 crc :-1 s=READY pgs=716 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 msgr2=0x7f84b012a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 0x7f84b012a670 secure :-1 s=READY pgs=1718 cs=0 l=1 rx=0x7f84a80107e0 tx=0x7f84a8010610).stop 2019-12-05T06:59:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3461319748 shutdown_connections 2019-12-05T06:59:16.974 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 --2- 172.21.15.16:0/3461319748 >> v2:172.21.15.76:3300/0 conn(0x7f84b012b640 0x7f84b012abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:16.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f8490023630 0x7f8490025ac0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:16.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 --2- 172.21.15.16:0/3461319748 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f84b01302f0 0x7f84b012a670 unknown :-1 s=CLOSED pgs=1718 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:16.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3461319748 shutdown_connections 2019-12-05T06:59:16.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3461319748 wait complete. 2019-12-05T06:59:16.975 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:16.971+0000 7f84b6a8a700 1 -- 172.21.15.16:0/3461319748 >> 172.21.15.16:0/3461319748 conn(0x7f84b00a2850 msgr2=0x7f84b006ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:16.989 INFO:tasks.mds_thrash.fs.[cephfs]:no change 2019-12-05T06:59:18.163 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:59:18.171 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:59:18.992 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:19.229 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.218+0000 7fbb18aaa700 1 Processor -- start 2019-12-05T06:59:19.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.218+0000 7fbb18aaa700 1 -- start start 2019-12-05T06:59:19.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.220+0000 7fbb18aaa700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7fbb1408fef0 0x7fbb1408f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:19.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.220+0000 7fbb18aaa700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb140958e0 con 0x7fbb1408c6d0 2019-12-05T06:59:19.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.220+0000 7fbb18aaa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb140629e0 con 0x7fbb1408d2c0 2019-12-05T06:59:19.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.220+0000 7fbb18aaa700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb1408df90 con 0x7fbb1408fef0 2019-12-05T06:59:19.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.221+0000 7fbb12d9d700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbb1408c6d0 0x7fbb1408cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50864/0 (socket says 172.21.15.16:50864) 2019-12-05T06:59:19.230 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.221+0000 7fbb12d9d700 1 -- 172.21.15.16:0/1537754123 learned_addr learned my addr 172.21.15.16:0/1537754123 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:19.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.221+0000 7fbb1359e700 1 --2- 172.21.15.16:0/1537754123 >> v2:172.21.15.76:3300/0 conn(0x7fbb1408fef0 0x7fbb1408f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:19.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.221+0000 7fbb12d9d700 1 -- 172.21.15.16:0/1537754123 >> v1:172.21.15.16:6789/0 conn(0x7fbb1408c6d0 legacy=0x7fbb1408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2019-12-05T06:59:19.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.221+0000 7fbb12d9d700 1 -- 172.21.15.16:0/1537754123 >> v1:172.21.15.16:6789/0 conn(0x7fbb1408c6d0 legacy=0x7fbb1408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2019-12-05T06:59:19.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.221+0000 7fbb12d9d700 1 --1- 172.21.15.16:0/1537754123 >> v1:172.21.15.16:6789/0 conn(0x7fbb1408c6d0 0x7fbb1408cac0 :-1 s=OPENED pgs=1719 cs=1 l=1).handle_message read tag failed 2019-12-05T06:59:19.231 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.221+0000 7fbb12d9d700 1 --1- 172.21.15.16:0/1537754123 >> v1:172.21.15.16:6789/0 conn(0x7fbb1408c6d0 0x7fbb1408cac0 :-1 s=OPENED pgs=1719 cs=1 l=1).fault on lossy channel, failing 2019-12-05T06:59:19.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.221+0000 7fbb1359e700 1 -- 172.21.15.16:0/1537754123 >> v1:172.21.15.76:6789/0 conn(0x7fbb1408d2c0 legacy=0x7fbb1408ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:19.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.221+0000 7fbb1359e700 1 -- 172.21.15.16:0/1537754123 >> v1:172.21.15.16:6789/0 conn(0x7fbb1408c6d0 legacy=0x7fbb1408cac0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2019-12-05T06:59:19.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.221+0000 7fbb11d9b700 1 -- 172.21.15.16:0/1537754123 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511772610 0 0) 0x7fbafc001080 con 0x7fbb1408d2c0 2019-12-05T06:59:19.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.222+0000 7fbb1359e700 1 -- 172.21.15.16:0/1537754123 --> v2:172.21.15.76:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb1408e620 con 0x7fbb1408fef0 2019-12-05T06:59:19.232 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.222+0000 7fbb1359e700 1 --2- 172.21.15.16:0/1537754123 >> v2:172.21.15.76:3300/0 conn(0x7fbb1408fef0 0x7fbb1408f720 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7fbb0400af80 tx=0x7fbb0400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.222+0000 7fbb1359e700 1 --2- 172.21.15.16:0/1537754123 >> v2:172.21.15.76:3300/0 conn(0x7fbb1408fef0 0x7fbb1408f720 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7fbb0400af80 tx=0x7fbb0400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.222+0000 7fbb1359e700 1 --2- 172.21.15.16:0/1537754123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb1408fef0 0x7fbb1408f720 secure :-1 s=READY pgs=1159 cs=0 l=1 rx=0x7fbb0400af80 tx=0x7fbb0400b6e0).ready entity=mon.1 client_cookie=b87acb34a97da6be server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:19.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.222+0000 7fbb1359e700 1 --2- 172.21.15.16:0/1537754123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb1408fef0 0x7fbb1408f720 secure :-1 s=THROTTLE_DONE pgs=1159 cs=0 l=1 rx=0x7fbb0400af80 tx=0x7fbb0400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.222+0000 7fbb11d9b700 1 -- 172.21.15.16:0/1537754123 <== mon.1 v2:172.21.15.76:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb04007240 con 0x7fbb1408fef0 2019-12-05T06:59:19.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.222+0000 7fbb1359e700 1 --2- 172.21.15.16:0/1537754123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb1408fef0 0x7fbb1408f720 secure :-1 s=THROTTLE_DONE pgs=1159 cs=0 l=1 rx=0x7fbb0400af80 tx=0x7fbb0400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.233 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.222+0000 7fbb1359e700 1 --2- 172.21.15.16:0/1537754123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb1408fef0 0x7fbb1408f720 secure :-1 s=THROTTLE_DONE pgs=1159 cs=0 l=1 rx=0x7fbb0400af80 tx=0x7fbb0400b6e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.222+0000 7fbb11d9b700 1 -- 172.21.15.16:0/1537754123 <== mon.1 v2:172.21.15.76:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb040099e0 con 0x7fbb1408fef0 2019-12-05T06:59:19.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.222+0000 7fbb11d9b700 1 -- 172.21.15.16:0/1537754123 <== mon.1 v2:172.21.15.76:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb0400a070 con 0x7fbb1408fef0 2019-12-05T06:59:19.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.223+0000 7fbb18aaa700 1 -- 172.21.15.16:0/1537754123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb1408fef0 msgr2=0x7fbb1408f720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:19.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.223+0000 7fbb18aaa700 1 --2- 172.21.15.16:0/1537754123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb1408fef0 0x7fbb1408f720 secure :-1 s=READY pgs=1159 cs=0 l=1 rx=0x7fbb0400af80 tx=0x7fbb0400b6e0).stop 2019-12-05T06:59:19.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.225+0000 7fbb18aaa700 1 -- 172.21.15.16:0/1537754123 shutdown_connections 2019-12-05T06:59:19.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.225+0000 7fbb18aaa700 1 --2- 172.21.15.16:0/1537754123 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7fbb1408fef0 0x7fbb1408f720 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:19.234 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.225+0000 7fbb18aaa700 1 -- 172.21.15.16:0/1537754123 shutdown_connections 2019-12-05T06:59:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.225+0000 7fbb18aaa700 1 -- 172.21.15.16:0/1537754123 wait complete. 2019-12-05T06:59:19.235 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.225+0000 7fbb18aaa700 1 -- 172.21.15.16:0/1537754123 >> 172.21.15.16:0/1537754123 conn(0x7fbb14061940 msgr2=0x7fbb14096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.229+0000 7fbb18aaa700 1 Processor -- start 2019-12-05T06:59:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.230+0000 7fbb18aaa700 1 -- start start 2019-12-05T06:59:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.231+0000 7fbb18aaa700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.231+0000 7fbb18aaa700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb1406e5c0 con 0x7fbb1412a1d0 2019-12-05T06:59:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.231+0000 7fbb18aaa700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb14098500 con 0x7fbb14153a50 2019-12-05T06:59:19.238 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.231+0000 7fbb18aaa700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1406e250 con 0x7fbb14153e40 2019-12-05T06:59:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.231+0000 7fbb1259c700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7fbb1412a1d0 0x7fbb14152770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50872/0 (socket says 172.21.15.16:50872) 2019-12-05T06:59:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.232+0000 7fbb1259c700 1 -- 172.21.15.16:0/4268512971 learned_addr learned my addr 172.21.15.16:0/4268512971 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.232+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.232+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782174368 0 0) 0x7fbafc000b00 con 0x7fbb1412a1d0 2019-12-05T06:59:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.232+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf8002e80 con 0x7fbb1412a1d0 2019-12-05T06:59:19.239 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.232+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591141868 0 0) 0x7fbb0c0012f0 con 0x7fbb14153a50 2019-12-05T06:59:19.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.232+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbaf80030b0 con 0x7fbb14153a50 2019-12-05T06:59:19.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.232+0000 7fbb1359e700 1 -- 172.21.15.16:0/4268512971 >> v1:172.21.15.76:6789/0 conn(0x7fbb14153a50 legacy=0x7fbb14129200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:19.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.232+0000 7fbb1359e700 1 -- 172.21.15.16:0/4268512971 >> v1:172.21.15.16:6789/0 conn(0x7fbb1412a1d0 legacy=0x7fbb14152770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:19.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.232+0000 7fbb1359e700 1 -- 172.21.15.16:0/4268512971 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb14128900 con 0x7fbb14153e40 2019-12-05T06:59:19.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.240 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=READY pgs=410 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:19.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=THROTTLE_DONE pgs=410 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb040099b0 con 0x7fbb14153e40 2019-12-05T06:59:19.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=THROTTLE_DONE pgs=410 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=THROTTLE_DONE pgs=410 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb04009bb0 con 0x7fbb14153e40 2019-12-05T06:59:19.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb04009da0 con 0x7fbb14153e40 2019-12-05T06:59:19.241 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb18aaa700 1 -- 172.21.15.16:0/4268512971 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb14128900 con 0x7fbb14153e40 2019-12-05T06:59:19.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.233+0000 7fbb18aaa700 1 -- 172.21.15.16:0/4268512971 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb14128900 con 0x7fbb14153e40 2019-12-05T06:59:19.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.234+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=THROTTLE_DONE pgs=410 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.235+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7fbb04009da0 con 0x7fbb14153e40 2019-12-05T06:59:19.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.235+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=THROTTLE_DONE pgs=410 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.242 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.235+0000 7fbb0b7fe700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbaf8023650 0x7fbaf8025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.235+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7fbb04045db0 con 0x7fbb14153e40 2019-12-05T06:59:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.236+0000 7fbb12d9d700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbaf8023650 0x7fbaf8025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:19.243 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.236+0000 7fbb12d9d700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbaf8023650 0x7fbaf8025ae0 crc :-1 s=READY pgs=717 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:19.251 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.250+0000 7fbb18aaa700 1 -- 172.21.15.16:0/4268512971 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb14128900 con 0x7fbb14153e40 2019-12-05T06:59:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.259+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=THROTTLE_DONE pgs=410 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.260 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.259+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7fbb0400a170 con 0x7fbb14153e40 2019-12-05T06:59:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.503+0000 7fbb18aaa700 1 -- 172.21.15.16:0/4268512971 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb14128900 con 0x7fbb14153e40 2019-12-05T06:59:19.505 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.504+0000 7fbb1359e700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=THROTTLE_DONE pgs=410 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:19.506 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.504+0000 7fbb0b7fe700 1 -- 172.21.15.16:0/4268512971 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+1775 (secure 0 0 0) 0x7fbb0400a380 con 0x7fbb14153e40 2019-12-05T06:59:19.509 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 211 2019-12-05T06:59:19.510 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:19.510 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:59:18.782686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8523":{"gid":8523,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":20,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.509+0000 7fbb18aaa700 1 -- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbaf8023650 msgr2=0x7fbaf8025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.509+0000 7fbb18aaa700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbaf8023650 0x7fbaf8025ae0 crc :-1 s=READY pgs=717 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.509+0000 7fbb18aaa700 1 -- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 msgr2=0x7fbb14129a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:19.512 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.509+0000 7fbb18aaa700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 secure :-1 s=READY pgs=410 cs=0 l=1 rx=0x7fbb04010390 tx=0x7fbb04010730).stop 2019-12-05T06:59:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.509+0000 7fbb18aaa700 1 -- 172.21.15.16:0/4268512971 shutdown_connections 2019-12-05T06:59:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.509+0000 7fbb18aaa700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7fbaf8023650 0x7fbaf8025ae0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.509+0000 7fbb18aaa700 1 --2- 172.21.15.16:0/4268512971 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7fbb14153e40 0x7fbb14129a00 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.509+0000 7fbb18aaa700 1 -- 172.21.15.16:0/4268512971 shutdown_connections 2019-12-05T06:59:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.509+0000 7fbb18aaa700 1 -- 172.21.15.16:0/4268512971 wait complete. 2019-12-05T06:59:19.513 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:19.509+0000 7fbb18aaa700 1 -- 172.21.15.16:0/4268512971 >> 172.21.15.16:0/4268512971 conn(0x7fbb1406d260 msgr2=0x7fbb1409aa30 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:19.521 INFO:tasks.mds_thrash.fs.[cephfs]:mds.a has gained rank=0, replacing gid=8430 2019-12-05T06:59:19.521 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 1 secs before reviving mds.b 2019-12-05T06:59:20.522 INFO:tasks.mds_thrash.fs.[cephfs]:reviving mds.b 2019-12-05T06:59:20.522 INFO:tasks.ceph.mds.b:Restarting daemon 2019-12-05T06:59:20.522 INFO:teuthology.orchestra.run.smithi076:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2019-12-05T06:59:20.528 INFO:tasks.ceph.mds.b:Started 2019-12-05T06:59:20.529 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:20.696 INFO:tasks.ceph.mds.b.smithi076.stdout:starting mds.b at 2019-12-05T06:59:20.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.776+0000 7f04df5a7700 1 Processor -- start 2019-12-05T06:59:20.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.776+0000 7f04df5a7700 1 -- start start 2019-12-05T06:59:20.784 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.777+0000 7f04df5a7700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d808fef0 0x7f04d808f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:20.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.777+0000 7f04df5a7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d80958e0 con 0x7f04d808c6d0 2019-12-05T06:59:20.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.777+0000 7f04df5a7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d80629e0 con 0x7f04d808d2c0 2019-12-05T06:59:20.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.777+0000 7f04df5a7700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04d808df90 con 0x7f04d808fef0 2019-12-05T06:59:20.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.777+0000 7f04ddb44700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04d808c6d0 0x7f04d808cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50880/0 (socket says 172.21.15.16:50880) 2019-12-05T06:59:20.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.777+0000 7f04ddb44700 1 -- 172.21.15.16:0/1188704550 learned_addr learned my addr 172.21.15.16:0/1188704550 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:20.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04de345700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d808fef0 0x7f04d808f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:20.785 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4005428840 0 0) 0x7f04d40014d0 con 0x7f04d808c6d0 2019-12-05T06:59:20.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c4002e80 con 0x7f04d808c6d0 2019-12-05T06:59:20.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3520108836 0 0) 0x7f04c8001040 con 0x7f04d808d2c0 2019-12-05T06:59:20.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 --> v1:172.21.15.76:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c40030b0 con 0x7f04d808d2c0 2019-12-05T06:59:20.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 1267782769 0 0) 0x7f04d40014b0 con 0x7f04d808c6d0 2019-12-05T06:59:20.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d808fef0 msgr2=0x7f04d808f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:20.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04d808fef0 0x7f04d808f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:20.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 >> v1:172.21.15.76:6789/0 conn(0x7f04d808d2c0 legacy=0x7f04d808ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:20.786 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 --> v1:172.21.15.16:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d808e620 con 0x7f04d808c6d0 2019-12-05T06:59:20.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f04d40019c0 con 0x7f04d808c6d0 2019-12-05T06:59:20.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 >> v1:172.21.15.16:6789/0 conn(0x7f04d808c6d0 legacy=0x7f04d808cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:20.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04c4003b30 0x7f04c4005f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:20.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c4006660 0x7f04c4008b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:20.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f04c404d320 0x7f04c404f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:20.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c404ffd0 con 0x7f04c4006660 2019-12-05T06:59:20.787 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dd343700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c4006660 0x7f04c4008b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:20.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 --> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4050200 con 0x7f04c404d320 2019-12-05T06:59:20.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.778+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04c4050430 con 0x7f04c4003b30 2019-12-05T06:59:20.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04ddb44700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04c4003b30 0x7f04c4005f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:20.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04de345700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f04c404d320 0x7f04c404f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:20.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 -- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04c4003b30 msgr2=0x7f04c4005f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:20.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f04c4003b30 0x7f04c4005f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:20.788 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 -- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f04c404d320 msgr2=0x7f04c404f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:20.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] conn(0x7f04c404d320 0x7f04c404f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:20.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 -- 172.21.15.16:0/1188704550 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04c40506e0 con 0x7f04c4006660 2019-12-05T06:59:20.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c4006660 0x7f04c4008b10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f04c800af60 tx=0x7f04c800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c4006660 0x7f04c4008b10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f04c800af60 tx=0x7f04c800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c4006660 0x7f04c4008b10 secure :-1 s=READY pgs=1725 cs=0 l=1 rx=0x7f04c800af60 tx=0x7f04c800b2e0).ready entity=mon.0 client_cookie=76a0d5ac97d6cb4a server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:20.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c4006660 0x7f04c4008b10 secure :-1 s=THROTTLE_DONE pgs=1725 cs=0 l=1 rx=0x7f04c800af60 tx=0x7f04c800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.789 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c4006660 0x7f04c4008b10 secure :-1 s=THROTTLE_DONE pgs=1725 cs=0 l=1 rx=0x7f04c800af60 tx=0x7f04c800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04c8000990 con 0x7f04c4006660 2019-12-05T06:59:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dd343700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c4006660 0x7f04c4008b10 secure :-1 s=THROTTLE_DONE pgs=1725 cs=0 l=1 rx=0x7f04c800af60 tx=0x7f04c800b2e0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04c8009b30 con 0x7f04c4006660 2019-12-05T06:59:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04dcb42700 1 -- 172.21.15.16:0/1188704550 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04c800a140 con 0x7f04c4006660 2019-12-05T06:59:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04df5a7700 1 -- 172.21.15.16:0/1188704550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c4006660 msgr2=0x7f04c4008b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04df5a7700 1 --2- 172.21.15.16:0/1188704550 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04c4006660 0x7f04c4008b10 secure :-1 s=READY pgs=1725 cs=0 l=1 rx=0x7f04c800af60 tx=0x7f04c800b2e0).stop 2019-12-05T06:59:20.790 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.779+0000 7f04de345700 1 -- 172.21.15.16:0/1188704550 reap_dead start 2019-12-05T06:59:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.780+0000 7f04df5a7700 1 -- 172.21.15.16:0/1188704550 shutdown_connections 2019-12-05T06:59:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.780+0000 7f04df5a7700 1 -- 172.21.15.16:0/1188704550 shutdown_connections 2019-12-05T06:59:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.780+0000 7f04df5a7700 1 -- 172.21.15.16:0/1188704550 wait complete. 2019-12-05T06:59:20.791 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.780+0000 7f04df5a7700 1 -- 172.21.15.16:0/1188704550 >> 172.21.15.16:0/1188704550 conn(0x7f04d8061940 msgr2=0x7f04d8096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:20.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.784+0000 7f04df5a7700 1 Processor -- start 2019-12-05T06:59:20.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.784+0000 7f04df5a7700 1 -- start start 2019-12-05T06:59:20.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04df5a7700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f04d81289c0 0x7f04d8084c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:20.792 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04df5a7700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d812a170 con 0x7f04d812d870 2019-12-05T06:59:20.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04df5a7700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04d8129c70 con 0x7f04d81295b0 2019-12-05T06:59:20.793 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04df5a7700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04d812abf0 con 0x7f04d81289c0 2019-12-05T06:59:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04dd343700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f04d812d870 0x7f04d8128db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50894/0 (socket says 172.21.15.16:50894) 2019-12-05T06:59:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04dd343700 1 -- 172.21.15.16:0/294723467 learned_addr learned my addr 172.21.15.16:0/294723467 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:20.795 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> v2:172.21.15.16:3300/0 conn(0x7f04d81289c0 0x7f04d8084c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04ce7fc700 1 -- 172.21.15.16:0/294723467 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148343597 0 0) 0x7f04c800abc0 con 0x7f04d812d870 2019-12-05T06:59:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04ce7fc700 1 -- 172.21.15.16:0/294723467 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04c0002e80 con 0x7f04d812d870 2019-12-05T06:59:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04ddb44700 1 -- 172.21.15.16:0/294723467 >> v1:172.21.15.76:6789/0 conn(0x7f04d81295b0 legacy=0x7f04d81566a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04ddb44700 1 -- 172.21.15.16:0/294723467 >> v1:172.21.15.16:6789/0 conn(0x7f04d812d870 legacy=0x7f04d8128db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04ddb44700 1 -- 172.21.15.16:0/294723467 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04d812cf70 con 0x7f04d81289c0 2019-12-05T06:59:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> v2:172.21.15.16:3300/0 conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.796 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.785+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> v2:172.21.15.16:3300/0 conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.786+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=READY pgs=1727 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.786+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=THROTTLE_DONE pgs=1727 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.786+0000 7f04ce7fc700 1 -- 172.21.15.16:0/294723467 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04d4001010 con 0x7f04d81289c0 2019-12-05T06:59:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.786+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=THROTTLE_DONE pgs=1727 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.786+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=THROTTLE_DONE pgs=1727 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.786+0000 7f04ce7fc700 1 -- 172.21.15.16:0/294723467 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04d4001460 con 0x7f04d81289c0 2019-12-05T06:59:20.797 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.786+0000 7f04ce7fc700 1 -- 172.21.15.16:0/294723467 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04d4004b90 con 0x7f04d81289c0 2019-12-05T06:59:20.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.786+0000 7f04df5a7700 1 -- 172.21.15.16:0/294723467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04d812cf70 con 0x7f04d81289c0 2019-12-05T06:59:20.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.786+0000 7f04df5a7700 1 -- 172.21.15.16:0/294723467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04d812cf70 con 0x7f04d81289c0 2019-12-05T06:59:20.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.786+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=THROTTLE_DONE pgs=1727 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.787+0000 7f04ce7fc700 1 -- 172.21.15.16:0/294723467 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f04d40045c0 con 0x7f04d81289c0 2019-12-05T06:59:20.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.787+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=THROTTLE_DONE pgs=1727 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.787+0000 7f04ce7fc700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04c0023670 0x7f04c0025ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:20.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.787+0000 7f04ce7fc700 1 -- 172.21.15.16:0/294723467 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f04d4046140 con 0x7f04d81289c0 2019-12-05T06:59:20.798 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.788+0000 7f04dd343700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04c0023670 0x7f04c0025ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:20.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.788+0000 7f04dd343700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04c0023670 0x7f04c0025ae0 crc :-1 s=READY pgs=718 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:20.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.792+0000 7f04df5a7700 1 -- 172.21.15.16:0/294723467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04d812cf70 con 0x7f04d81289c0 2019-12-05T06:59:20.799 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.796+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=THROTTLE_DONE pgs=1727 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.800 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.796+0000 7f04ce7fc700 1 -- 172.21.15.16:0/294723467 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f04d4034f20 con 0x7f04d81289c0 2019-12-05T06:59:20.996 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.994+0000 7f04df5a7700 1 -- 172.21.15.16:0/294723467 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f04d812cf70 con 0x7f04d81289c0 2019-12-05T06:59:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.995+0000 7f04ddb44700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=THROTTLE_DONE pgs=1727 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:20.997 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:20.995+0000 7f04ce7fc700 1 -- 172.21.15.16:0/294723467 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f04d4034f20 con 0x7f04d81289c0 2019-12-05T06:59:20.999 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 211 2019-12-05T06:59:20.999 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:20.999 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:59:18.782686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8523":{"gid":8523,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":20,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:21.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:21.000+0000 7f04df5a7700 1 -- 172.21.15.16:0/294723467 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04c0023670 msgr2=0x7f04c0025ae0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:21.002 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:21.000+0000 7f04df5a7700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04c0023670 0x7f04c0025ae0 crc :-1 s=READY pgs=718 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:21.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:21.000+0000 7f04df5a7700 1 -- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 msgr2=0x7f04d8084c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:21.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:21.000+0000 7f04df5a7700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 0x7f04d8084c10 secure :-1 s=READY pgs=1727 cs=0 l=1 rx=0x7f04d40106f0 tx=0x7f04d4010a90).stop 2019-12-05T06:59:21.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:21.001+0000 7f04df5a7700 1 -- 172.21.15.16:0/294723467 shutdown_connections 2019-12-05T06:59:21.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:21.001+0000 7f04df5a7700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f04c0023670 0x7f04c0025ae0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:21.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:21.001+0000 7f04df5a7700 1 --2- 172.21.15.16:0/294723467 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f04d81289c0 0x7f04d8084c10 unknown :-1 s=CLOSED pgs=1727 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:21.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:21.001+0000 7f04df5a7700 1 -- 172.21.15.16:0/294723467 shutdown_connections 2019-12-05T06:59:21.003 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:21.001+0000 7f04df5a7700 1 -- 172.21.15.16:0/294723467 wait complete. 2019-12-05T06:59:21.004 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:21.001+0000 7f04df5a7700 1 -- 172.21.15.16:0/294723467 >> 172.21.15.16:0/294723467 conn(0x7f04d80a1e80 msgr2=0x7f04d806cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:21.012 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:59:23.014 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.174+0000 7f7cd4413700 1 Processor -- start 2019-12-05T06:59:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.174+0000 7f7cd4413700 1 -- start start 2019-12-05T06:59:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd4413700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f7ccc08d2c0 0x7f7ccc08ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd4413700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 0x7f7ccc08f460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:23.181 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd4413700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc0958e0 con 0x7f7ccc08c6d0 2019-12-05T06:59:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd4413700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ccc08df90 con 0x7f7ccc08d2c0 2019-12-05T06:59:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd4413700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc0a3f70 con 0x7f7ccc08fef0 2019-12-05T06:59:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd29b0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7ccc08c6d0 0x7f7ccc08cac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50912/0 (socket says 172.21.15.16:50912) 2019-12-05T06:59:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd29b0700 1 -- 172.21.15.16:0/110838432 learned_addr learned my addr 172.21.15.16:0/110838432 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/110838432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 0x7f7ccc08f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd21af700 1 --2- 172.21.15.16:0/110838432 >> v2:172.21.15.76:3300/0 conn(0x7f7ccc08d2c0 0x7f7ccc08ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:23.182 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd19ae700 1 -- 172.21.15.16:0/110838432 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4283072762 0 0) 0x7f7cbc001150 con 0x7f7ccc08c6d0 2019-12-05T06:59:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.175+0000 7f7cd19ae700 1 -- 172.21.15.16:0/110838432 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cb4002e80 con 0x7f7ccc08c6d0 2019-12-05T06:59:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd19ae700 1 -- 172.21.15.16:0/110838432 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 390+0+0 (unknown 2812871082 0 0) 0x7f7cbc001130 con 0x7f7ccc08c6d0 2019-12-05T06:59:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd31b1700 1 -- 172.21.15.16:0/110838432 >> v2:172.21.15.76:3300/0 conn(0x7f7ccc08d2c0 msgr2=0x7f7ccc08ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/110838432 >> v2:172.21.15.76:3300/0 conn(0x7f7ccc08d2c0 0x7f7ccc08ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd31b1700 1 -- 172.21.15.16:0/110838432 >> v1:172.21.15.16:6789/0 conn(0x7f7ccc08c6d0 legacy=0x7f7ccc08cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd31b1700 1 -- 172.21.15.16:0/110838432 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc08e620 con 0x7f7ccc08fef0 2019-12-05T06:59:23.183 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/110838432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 0x7f7ccc08f460 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f7cc400af80 tx=0x7f7cc400b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/110838432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 0x7f7ccc08f460 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f7cc400af80 tx=0x7f7cc400b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/110838432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 0x7f7ccc08f460 secure :-1 s=READY pgs=411 cs=0 l=1 rx=0x7f7cc400af80 tx=0x7f7cc400b280).ready entity=mon.2 client_cookie=f9e6063971b10b97 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/110838432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 0x7f7ccc08f460 secure :-1 s=THROTTLE_DONE pgs=411 cs=0 l=1 rx=0x7f7cc400af80 tx=0x7f7cc400b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd19ae700 1 -- 172.21.15.16:0/110838432 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cc4007240 con 0x7f7ccc08fef0 2019-12-05T06:59:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/110838432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 0x7f7ccc08f460 secure :-1 s=THROTTLE_DONE pgs=411 cs=0 l=1 rx=0x7f7cc400af80 tx=0x7f7cc400b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/110838432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 0x7f7ccc08f460 secure :-1 s=THROTTLE_DONE pgs=411 cs=0 l=1 rx=0x7f7cc400af80 tx=0x7f7cc400b280).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.184 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd19ae700 1 -- 172.21.15.16:0/110838432 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc4000b10 con 0x7f7ccc08fef0 2019-12-05T06:59:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd19ae700 1 -- 172.21.15.16:0/110838432 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cc40097f0 con 0x7f7ccc08fef0 2019-12-05T06:59:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd4413700 1 -- 172.21.15.16:0/110838432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 msgr2=0x7f7ccc08f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.176+0000 7f7cd4413700 1 --2- 172.21.15.16:0/110838432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 0x7f7ccc08f460 secure :-1 s=READY pgs=411 cs=0 l=1 rx=0x7f7cc400af80 tx=0x7f7cc400b280).stop 2019-12-05T06:59:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.177+0000 7f7cd4413700 1 -- 172.21.15.16:0/110838432 shutdown_connections 2019-12-05T06:59:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.177+0000 7f7cd4413700 1 --2- 172.21.15.16:0/110838432 >> v2:172.21.15.76:3300/0 conn(0x7f7ccc08d2c0 0x7f7ccc08ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.177+0000 7f7cd4413700 1 --2- 172.21.15.16:0/110838432 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc08fef0 0x7f7ccc08f460 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:23.185 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.177+0000 7f7cd4413700 1 -- 172.21.15.16:0/110838432 shutdown_connections 2019-12-05T06:59:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.177+0000 7f7cd4413700 1 -- 172.21.15.16:0/110838432 wait complete. 2019-12-05T06:59:23.186 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.177+0000 7f7cd4413700 1 -- 172.21.15.16:0/110838432 >> 172.21.15.16:0/110838432 conn(0x7f7ccc061940 msgr2=0x7f7ccc096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.181+0000 7f7cd4413700 1 Processor -- start 2019-12-05T06:59:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.181+0000 7f7cd4413700 1 -- start start 2019-12-05T06:59:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.181+0000 7f7cd4413700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.181+0000 7f7cd4413700 1 -- --> v1:172.21.15.16:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc12c3a0 con 0x7f7ccc1325e0 2019-12-05T06:59:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.181+0000 7f7cd4413700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ccc128b20 con 0x7f7ccc12c740 2019-12-05T06:59:23.189 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.181+0000 7f7cd4413700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ccc12a7e0 con 0x7f7ccc12bf70 2019-12-05T06:59:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.181+0000 7f7cd29b0700 1 --1- >> v1:172.21.15.16:6789/0 conn(0x7f7ccc1325e0 0x7f7ccc12b770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.16:6789/0 says I am v1:172.21.15.16:50918/0 (socket says 172.21.15.16:50918) 2019-12-05T06:59:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.181+0000 7f7cd29b0700 1 -- 172.21.15.16:0/2888205894 learned_addr learned my addr 172.21.15.16:0/2888205894 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 <== mon.0 v1:172.21.15.16:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426655315 0 0) 0x7f7cbc000f40 con 0x7f7ccc1325e0 2019-12-05T06:59:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 --> v1:172.21.15.16:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cac002e80 con 0x7f7ccc1325e0 2019-12-05T06:59:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 <== mon.0 v1:172.21.15.16:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 935+0+0 (unknown 1012063268 0 0) 0x7f7cbc000c90 con 0x7f7ccc1325e0 2019-12-05T06:59:23.190 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 --> v1:172.21.15.16:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cac0032e0 con 0x7f7ccc1325e0 2019-12-05T06:59:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 <== mon.0 v1:172.21.15.16:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1040655821 0 0) 0x7f7cbc001760 con 0x7f7ccc1325e0 2019-12-05T06:59:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7cd31b1700 1 -- 172.21.15.16:0/2888205894 >> v1:172.21.15.76:6789/0 conn(0x7f7ccc12c740 legacy=0x7f7ccc154c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7cd31b1700 1 -- 172.21.15.16:0/2888205894 >> v1:172.21.15.16:6789/0 conn(0x7f7ccc1325e0 legacy=0x7f7ccc12b770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7cd31b1700 1 -- 172.21.15.16:0/2888205894 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ccc12ae70 con 0x7f7ccc12bf70 2019-12-05T06:59:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.191 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.182+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=READY pgs=412 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.183+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.183+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 <== mon.2 v2:172.21.15.76:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cc400abe0 con 0x7f7ccc12bf70 2019-12-05T06:59:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.183+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.183+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.183+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 <== mon.2 v2:172.21.15.76:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cc4009900 con 0x7f7ccc12bf70 2019-12-05T06:59:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.183+0000 7f7cd4413700 1 -- 172.21.15.16:0/2888205894 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ccc12ae70 con 0x7f7ccc12bf70 2019-12-05T06:59:23.192 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.183+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 <== mon.2 v2:172.21.15.76:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cc400a050 con 0x7f7ccc12bf70 2019-12-05T06:59:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.183+0000 7f7cd4413700 1 -- 172.21.15.16:0/2888205894 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ccc12ae70 con 0x7f7ccc12bf70 2019-12-05T06:59:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.184+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.184+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 <== mon.2 v2:172.21.15.76:3301/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f7cc400ac80 con 0x7f7ccc12bf70 2019-12-05T06:59:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.184+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.184+0000 7f7ccb7fe700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cac027a80 0x7f7cac029f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.185+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 <== mon.2 v2:172.21.15.76:3301/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f7cc4045dc0 con 0x7f7ccc12bf70 2019-12-05T06:59:23.193 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.185+0000 7f7cd29b0700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cac027a80 0x7f7cac029f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.185+0000 7f7cd29b0700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cac027a80 0x7f7cac029f10 crc :-1 s=READY pgs=719 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:23.194 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.189+0000 7f7cd4413700 1 -- 172.21.15.16:0/2888205894 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ccc12ae70 con 0x7f7ccc12bf70 2019-12-05T06:59:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.196+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.197 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.196+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 <== mon.2 v2:172.21.15.76:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f7cc402ade0 con 0x7f7ccc12bf70 2019-12-05T06:59:23.388 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.387+0000 7f7cd4413700 1 -- 172.21.15.16:0/2888205894 --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7ccc12ae70 con 0x7f7ccc12bf70 2019-12-05T06:59:23.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.388+0000 7f7cd31b1700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=THROTTLE_DONE pgs=412 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:23.389 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.388+0000 7f7ccb7fe700 1 -- 172.21.15.16:0/2888205894 <== mon.2 v2:172.21.15.76:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 211 v211) v1 ==== 95+0+1775 (secure 0 0 0) 0x7f7cc4009d10 con 0x7f7ccc12bf70 2019-12-05T06:59:23.392 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:23.392 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":211,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:59:18.782686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8523":{"gid":8523,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":20,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:23.392 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 211 2019-12-05T06:59:23.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.393+0000 7f7cd4413700 1 -- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cac027a80 msgr2=0x7f7cac029f10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:23.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.393+0000 7f7cd4413700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cac027a80 0x7f7cac029f10 crc :-1 s=READY pgs=719 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:23.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.393+0000 7f7cd4413700 1 -- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 msgr2=0x7f7ccc158480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:23.395 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.393+0000 7f7cd4413700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 secure :-1 s=READY pgs=412 cs=0 l=1 rx=0x7f7cc4010380 tx=0x7f7cc4010720).stop 2019-12-05T06:59:23.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.394+0000 7f7cd4413700 1 -- 172.21.15.16:0/2888205894 shutdown_connections 2019-12-05T06:59:23.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.394+0000 7f7cd4413700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f7cac027a80 0x7f7cac029f10 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:23.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.394+0000 7f7cd4413700 1 --2- 172.21.15.16:0/2888205894 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f7ccc12bf70 0x7f7ccc158480 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:23.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.394+0000 7f7cd4413700 1 -- 172.21.15.16:0/2888205894 shutdown_connections 2019-12-05T06:59:23.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.394+0000 7f7cd4413700 1 -- 172.21.15.16:0/2888205894 wait complete. 2019-12-05T06:59:23.396 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:23.394+0000 7f7cd4413700 1 -- 172.21.15.16:0/2888205894 >> 172.21.15.16:0/2888205894 conn(0x7f7ccc0a2850 msgr2=0x7f7ccc06ede0 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:23.411 INFO:tasks.mds_thrash.fs.[cephfs]:waiting till mds map indicates mds.b is in active, standby or standby-replay 2019-12-05T06:59:25.412 INFO:teuthology.orchestra.run.smithi016:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2019-12-05T06:59:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.674+0000 7f50ab95a700 1 Processor -- start 2019-12-05T06:59:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.675+0000 7f50ab95a700 1 -- start start 2019-12-05T06:59:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50ab95a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f50a408c6d0 0x7f50a408cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:25.685 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50ab95a700 1 --2- >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f50a408fef0 0x7f50a408d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50ab95a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50a408ef20 0x7f50a408f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rx=0 tx=0).connect 2019-12-05T06:59:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50ab95a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a408df90 con 0x7f50a408ef20 2019-12-05T06:59:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50ab95a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a40a3f70 con 0x7f50a408c6d0 2019-12-05T06:59:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50ab95a700 1 -- --> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50a408d830 con 0x7f50a408fef0 2019-12-05T06:59:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50aa6f8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50a408ef20 0x7f50a408f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:25.686 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50a9ef7700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f50a408c6d0 0x7f50a408cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50aa6f8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50a408ef20 0x7f50a408f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).handle_hello peer v2:172.21.15.16:3300/0 says I am v2:172.21.15.16:37550/0 (socket says 172.21.15.16:37550) 2019-12-05T06:59:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50aa6f8700 1 -- 172.21.15.16:0/710366475 learned_addr learned my addr 172.21.15.16:0/710366475 (peer_addr_for_me v2:172.21.15.16:0/0) 2019-12-05T06:59:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.676+0000 7f50a96f6700 1 --2- 172.21.15.16:0/710366475 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f50a408fef0 0x7f50a408d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 -- 172.21.15.16:0/710366475 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f50a408fef0 msgr2=0x7f50a408d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:25.687 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/710366475 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f50a408fef0 0x7f50a408d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 -- 172.21.15.16:0/710366475 >> v2:172.21.15.76:3300/0 conn(0x7f50a408c6d0 msgr2=0x7f50a408cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2019-12-05T06:59:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/710366475 >> v2:172.21.15.76:3300/0 conn(0x7f50a408c6d0 0x7f50a408cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 -- 172.21.15.16:0/710366475 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a408d540 con 0x7f50a408ef20 2019-12-05T06:59:25.688 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50a9ef7700 1 --2- 172.21.15.16:0/710366475 >> v2:172.21.15.76:3300/0 conn(0x7f50a408c6d0 0x7f50a408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:59:25.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/710366475 >> v2:172.21.15.16:3300/0 conn(0x7f50a408ef20 0x7f50a408f3c0 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=0 rx=0x7f509c00fa30 tx=0x7f509c00fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/710366475 >> v2:172.21.15.16:3300/0 conn(0x7f50a408ef20 0x7f50a408f3c0 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rx=0x7f509c00fa30 tx=0x7f509c00fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.690 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/710366475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a408ef20 0x7f50a408f3c0 secure :-1 s=READY pgs=1730 cs=0 l=1 rx=0x7f509c00fa30 tx=0x7f509c00fdd0).ready entity=mon.0 client_cookie=da36bf8017d58cf8 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/710366475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a408ef20 0x7f50a408f3c0 secure :-1 s=THROTTLE_DONE pgs=1730 cs=0 l=1 rx=0x7f509c00fa30 tx=0x7f509c00fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/710366475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a408ef20 0x7f50a408f3c0 secure :-1 s=THROTTLE_DONE pgs=1730 cs=0 l=1 rx=0x7f509c00fa30 tx=0x7f509c00fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/710366475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a408ef20 0x7f50a408f3c0 secure :-1 s=THROTTLE_DONE pgs=1730 cs=0 l=1 rx=0x7f509c00fa30 tx=0x7f509c00fdd0).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.677+0000 7f50a8ef5700 1 -- 172.21.15.16:0/710366475 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f509c001460 con 0x7f50a408ef20 2019-12-05T06:59:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.678+0000 7f50a8ef5700 1 -- 172.21.15.16:0/710366475 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c000f10 con 0x7f50a408ef20 2019-12-05T06:59:25.691 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.678+0000 7f50a8ef5700 1 -- 172.21.15.16:0/710366475 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f509c001a40 con 0x7f50a408ef20 2019-12-05T06:59:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.678+0000 7f50ab95a700 1 -- 172.21.15.16:0/710366475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a408ef20 msgr2=0x7f50a408f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.678+0000 7f50ab95a700 1 --2- 172.21.15.16:0/710366475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a408ef20 0x7f50a408f3c0 secure :-1 s=READY pgs=1730 cs=0 l=1 rx=0x7f509c00fa30 tx=0x7f509c00fdd0).stop 2019-12-05T06:59:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.679+0000 7f50ab95a700 1 -- 172.21.15.16:0/710366475 shutdown_connections 2019-12-05T06:59:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.679+0000 7f50ab95a700 1 --2- 172.21.15.16:0/710366475 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a408ef20 0x7f50a408f3c0 unknown :-1 s=CLOSED pgs=1730 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.679+0000 7f50ab95a700 1 --2- 172.21.15.16:0/710366475 >> [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] conn(0x7f50a408fef0 0x7f50a408d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:25.692 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.679+0000 7f50ab95a700 1 --2- 172.21.15.16:0/710366475 >> v2:172.21.15.76:3300/0 conn(0x7f50a408c6d0 0x7f50a408cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rx=0 tx=0).stop 2019-12-05T06:59:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.679+0000 7f50ab95a700 1 -- 172.21.15.16:0/710366475 shutdown_connections 2019-12-05T06:59:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.679+0000 7f50ab95a700 1 -- 172.21.15.16:0/710366475 wait complete. 2019-12-05T06:59:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.679+0000 7f50ab95a700 1 -- 172.21.15.16:0/710366475 >> 172.21.15.16:0/710366475 conn(0x7f50a4061940 msgr2=0x7f50a4096020 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.683+0000 7f50ab95a700 1 Processor -- start 2019-12-05T06:59:25.693 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.683+0000 7f50ab95a700 1 -- start start 2019-12-05T06:59:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.685+0000 7f50ab95a700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f50a4132d00 0x7f50a4132530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:25.694 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.685+0000 7f50ab95a700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50a4154b00 0x7f50a4156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.685+0000 7f50ab95a700 1 -- --> v1:172.21.15.76:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50a4098da0 con 0x7f50a409b850 2019-12-05T06:59:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.685+0000 7f50ab95a700 1 -- --> v2:172.21.15.16:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a409aa60 con 0x7f50a4154b00 2019-12-05T06:59:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.685+0000 7f50ab95a700 1 -- --> v2:172.21.15.76:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50a409a030 con 0x7f50a4132d00 2019-12-05T06:59:25.695 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.685+0000 7f50aa6f8700 1 --2- >> v2:172.21.15.16:3300/0 conn(0x7f50a4154b00 0x7f50a4156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:25.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.685+0000 7f50a96f6700 1 --2- >> v2:172.21.15.76:3300/0 conn(0x7f50a4132d00 0x7f50a4132530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:25.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.685+0000 7f50a9ef7700 1 --1- >> v1:172.21.15.76:6789/0 conn(0x7f50a409b850 0x7f50a4131d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.76:6789/0 says I am v1:172.21.15.16:44024/0 (socket says 172.21.15.16:44024) 2019-12-05T06:59:25.699 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.685+0000 7f50a9ef7700 1 -- 172.21.15.16:0/3242693191 learned_addr learned my addr 172.21.15.16:0/3242693191 (peer_addr_for_me v1:172.21.15.16:0/0) 2019-12-05T06:59:25.700 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.686+0000 7f50aa6f8700 1 -- 172.21.15.16:0/3242693191 >> v2:172.21.15.76:3300/0 conn(0x7f50a4132d00 msgr2=0x7f50a4132530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:25.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.686+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> v2:172.21.15.76:3300/0 conn(0x7f50a4132d00 0x7f50a4132530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:25.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.686+0000 7f50aa6f8700 1 -- 172.21.15.16:0/3242693191 >> v1:172.21.15.76:6789/0 conn(0x7f50a409b850 legacy=0x7f50a4131d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:25.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.686+0000 7f50aa6f8700 1 -- 172.21.15.16:0/3242693191 --> v2:172.21.15.16:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50a409b0f0 con 0x7f50a4154b00 2019-12-05T06:59:25.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.686+0000 7f50a27fc700 1 -- 172.21.15.16:0/3242693191 <== mon.1 v1:172.21.15.76:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018094356 0 0) 0x7f50940010b0 con 0x7f50a409b850 2019-12-05T06:59:25.701 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.686+0000 7f50a96f6700 1 --2- 172.21.15.16:0/3242693191 >> v2:172.21.15.76:3300/0 conn(0x7f50a4132d00 0x7f50a4132530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).handle_auth_done state changed! 2019-12-05T06:59:25.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.686+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> v2:172.21.15.16:3300/0 conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.687+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> v2:172.21.15.16:3300/0 conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.687+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=READY pgs=1731 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:25.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.687+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=THROTTLE_DONE pgs=1731 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.687+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=THROTTLE_DONE pgs=1731 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.702 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.687+0000 7f50a27fc700 1 -- 172.21.15.16:0/3242693191 <== mon.0 v2:172.21.15.16:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f509c001630 con 0x7f50a4154b00 2019-12-05T06:59:25.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.687+0000 7f50a27fc700 1 -- 172.21.15.16:0/3242693191 <== mon.0 v2:172.21.15.16:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f509c000cc0 con 0x7f50a4154b00 2019-12-05T06:59:25.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.687+0000 7f50ab95a700 1 -- 172.21.15.16:0/3242693191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50a409b0f0 con 0x7f50a4154b00 2019-12-05T06:59:25.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.687+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=THROTTLE_DONE pgs=1731 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.687+0000 7f50a27fc700 1 -- 172.21.15.16:0/3242693191 <== mon.0 v2:172.21.15.16:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f509c001840 con 0x7f50a4154b00 2019-12-05T06:59:25.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.688+0000 7f50ab95a700 1 -- 172.21.15.16:0/3242693191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50a409b0f0 con 0x7f50a4154b00 2019-12-05T06:59:25.703 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.688+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=THROTTLE_DONE pgs=1731 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.689+0000 7f50a27fc700 1 -- 172.21.15.16:0/3242693191 <== mon.0 v2:172.21.15.16:3300/0 4 ==== mgrmap(e 3) v1 ==== 45131+0+0 (secure 0 0 0) 0x7f509c001440 con 0x7f50a4154b00 2019-12-05T06:59:25.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.689+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=THROTTLE_DONE pgs=1731 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.689+0000 7f50a27fc700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f508402be50 0x7f508402e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rx=0 tx=0).connect 2019-12-05T06:59:25.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.689+0000 7f50a27fc700 1 -- 172.21.15.16:0/3242693191 <== mon.0 v2:172.21.15.16:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 9127+0+0 (secure 0 0 0) 0x7f509c08a330 con 0x7f50a4154b00 2019-12-05T06:59:25.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.689+0000 7f50a9ef7700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f508402be50 0x7f508402e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rx=0 tx=0)._handle_peer_banner_payload supported=0 required=0 2019-12-05T06:59:25.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.690+0000 7f50a9ef7700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f508402be50 0x7f508402e300 crc :-1 s=READY pgs=721 cs=0 l=1 rx=0 tx=0).ready entity=mgr.4100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2019-12-05T06:59:25.704 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.691+0000 7f50ab95a700 1 -- 172.21.15.16:0/3242693191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50a409b0f0 con 0x7f50a4154b00 2019-12-05T06:59:25.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.701+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=THROTTLE_DONE pgs=1731 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.705 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.701+0000 7f50a27fc700 1 -- 172.21.15.16:0/3242693191 <== mon.0 v2:172.21.15.16:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+111599 (secure 0 0 0) 0x7f509c001230 con 0x7f50a4154b00 2019-12-05T06:59:25.952 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.951+0000 7f50ab95a700 1 -- 172.21.15.16:0/3242693191 --> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f50a409b0f0 con 0x7f50a4154b00 2019-12-05T06:59:25.953 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.952+0000 7f50aa6f8700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=THROTTLE_DONE pgs=1731 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).handle_read_frame_epilogue_main read frame epilogue bytes=32 2019-12-05T06:59:25.954 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.952+0000 7f50a27fc700 1 -- 172.21.15.16:0/3242693191 <== mon.0 v2:172.21.15.16:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 212 v212) v1 ==== 95+0+2112 (secure 0 0 0) 0x7f509c001440 con 0x7f50a4154b00 2019-12-05T06:59:25.954 INFO:teuthology.orchestra.run.smithi016.stderr:dumped fsmap epoch 212 2019-12-05T06:59:25.954 INFO:teuthology.orchestra.run.smithi016.stdout: 2019-12-05T06:59:25.954 INFO:teuthology.orchestra.run.smithi016.stdout:{"epoch":212,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":8643,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.76:6835/1486320672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6834","nonce":1486320672},{"type":"v1","addr":"172.21.15.76:6835","nonce":1486320672}]},"export_targets":[],"features":4540138292836696063,"flags":0,"epoch":212}],"filesystems":[{"mdsmap":{"epoch":211,"flags":18,"ever_allowed_features":2,"explicitly_allowed_features":2,"created":"2019-12-05T06:06:40.256556+0000","modified":"2019-12-05T06:59:18.782686+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":85,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8523},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8523":{"gid":8523,"name":"a","rank":0,"incarnation":208,"state":"up:active","state_seq":20,"addr":"172.21.15.16:6833/3211386191","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.16:6832","nonce":3211386191},{"type":"v1","addr":"172.21.15.16:6833","nonce":3211386191}]},"export_targets":[],"features":4540138292836696063,"flags":0}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2019-12-05T06:59:25.956 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 -- 172.21.15.16:0/3242693191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f508402be50 msgr2=0x7f508402e300 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:25.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f508402be50 0x7f508402e300 crc :-1 s=READY pgs=721 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:25.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 -- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 msgr2=0x7f50a4156f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2019-12-05T06:59:25.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 0x7f50a4156f00 secure :-1 s=READY pgs=1731 cs=0 l=1 rx=0x7f509c054760 tx=0x7f509c054b00).stop 2019-12-05T06:59:25.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 -- 172.21.15.16:0/3242693191 shutdown_connections 2019-12-05T06:59:25.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.76:6800/9546,v1:172.21.15.76:6801/9546] conn(0x7f508402be50 0x7f508402e300 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:25.957 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 --2- 172.21.15.16:0/3242693191 >> [v2:172.21.15.16:3300/0,v1:172.21.15.16:6789/0] conn(0x7f50a4154b00 0x7f50a4156f00 unknown :-1 s=CLOSED pgs=1731 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 --2- 172.21.15.16:0/3242693191 >> v2:172.21.15.76:3300/0 conn(0x7f50a4132d00 0x7f50a4132530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rx=0 tx=0).stop 2019-12-05T06:59:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 -- 172.21.15.16:0/3242693191 shutdown_connections 2019-12-05T06:59:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 -- 172.21.15.16:0/3242693191 wait complete. 2019-12-05T06:59:25.958 INFO:teuthology.orchestra.run.smithi016.stderr:2019-12-05T06:59:25.955+0000 7f50ab95a700 1 -- 172.21.15.16:0/3242693191 >> 172.21.15.16:0/3242693191 conn(0x7f50a40714f0 msgr2=0x7f50a409ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2019-12-05T06:59:25.966 INFO:tasks.mds_thrash.fs.[cephfs]:mds.b reported in up:standby state 2019-12-05T06:59:25.966 INFO:tasks.mds_thrash.fs.[cephfs]:waiting for 91 secs before thrashing 2019-12-05T06:59:48.370 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T06:59:48.376 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T07:00:18.525 INFO:teuthology.orchestra.run.smithi016:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T07:00:18.530 INFO:teuthology.orchestra.run.smithi076:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2019-12-05T07:00:38.101 DEBUG:teuthology.exit:Got signal 15; running 2 handlers... 2019-12-05T07:00:38.101 DEBUG:teuthology.task.console_log:Killing console logger for smithi076 2019-12-05T07:00:38.101 DEBUG:teuthology.task.console_log:Killing console logger for smithi016 2019-12-05T07:00:38.102 DEBUG:teuthology.task.console_log:Killing console logger for smithi076 2019-12-05T07:00:38.102 DEBUG:teuthology.task.console_log:Killing console logger for smithi016 2019-12-05T07:00:38.103 DEBUG:teuthology.exit:Finished running handlers